VISTAS 2023 - Drive the Momentum - October 18-20, Caesars Palace, Las Vegas - Learn More
what is web crawling
Looking for a needle in a haystack?
by: Fabrizio Bianchi | January 24, 2023

Discover the 5 benefits of crawling global web data

In a recent blog, we talked about the 9 types of data to consider gathering for Revenue Growth Management and their importance in enabling and enhancing analytics capabilities at the different stages of the digital transformation journey.

Web data is the data and any content made available on publicly accessible networks
Web data crawling is the act of collecting all data from websites, detecting contents and storing them

There is a category of data that deserves an additional deep dive, and this is web data. Gathering, storing and crunching this data is no easy feat. Since there are a variety of ways that data and analytics can be leveraged to solve a problem, let’s explore 5 of them from the lens of sales and marketing.

  1. Optimizing the focus of your trade spend plan. It’s no secret that consumer behavior has drastically changed in the last few years. The lockdown, supply chain disruption and inflation have caused changes in where, how and what online shoppers buy. In this context, historical data is still the foundation of any trade spending optimization and forecast but it has to be integrated with fresh information which help AI/ML distinguish between persistent trends and those that will probably change. Testing and learning from a panel of consumers in collaboration with retailers is one tactic that leading companies are adopting. Another important data source is the utilization of web data scraping or crawling, to detect daily and up-to-date market data.
  2. Getting ahead of your competitors. Either as input for optimization or to run market analysis, crawling web data enables you to observe competitors in the short term. New promotional and pricing strategies adopted by competitors can be immediately identified by area, retailer, channel, product category and target consumer, and your response can be planned accordingly. Similarly, web data extracted from competitor and retailer sites provides valuable information about decisions on assortments, new launches and pack architecture policies of competitors. Having this kind of visibility is fundamental information to forecast sales, plan production and correctly evaluate results.
  3. Strengthening retail negotiation. From optimization and competitor monitoring, manufacturers can get the perfect outputs to formulate their negotiation with retailers in the best possible way. When informed about competitive behaviors, market evolutions, pricing and promotional effectiveness, manufacturers can negotiate based on valuable insights and identify wins with retailers by taking a more collaborative approach.
  4. Setting a dynamic pricing strategy. Consumer Product companies operating in the Direct-to-Consumer channel, need to have tools in place and information to properly set dynamic pricing. Whatever your pricing strategy, information about competitive pricing for comparable products and the prices of your own products sold by other retailers, online or in-store, is necessary to set the right price. The right price means maintaining consistency among DTC and other channels but at the same time, detecting all the market opportunities to gain share or improve margins.
  5. Aligning with consumer sentiment. Leveraging more marketing-oriented benefits, and crawling customer reviews and feedback from retailer portals or from social media, can provide data related to customer satisfaction. This enables manufacturers to proactively address customer concerns, identify consumer requests for product enhancements and plan more specific campaigns.

Be sure to consider all 5 components of your strategy individually and in combination to obtain optimal results. Here’s what you need:

  • Adequate technical infrastructure has to be in place. This infrastructure must be able to manage a huge amount of data to be crawled, stored, cleaned and harmonized from different sources and in heterogeneous formats.
  • Advanced analytics capabilities are required to enable optimization programs in pricing, promotions and assortments leveraging data science to support an immediate understanding of data, trends and KPIs through reports and dashboards.
  • Integration of data with insights and recommendations into the solution is needed to help avoid disconnection between analysis and planning for concrete results.
  • Preparation of a change management program to assist people with this new way of working is important.

Also, see: 9 types of CPG data to consider for Revenue Growth Management