Overview

Our client, a leading public sector authority in the UK, aimed to collect pricing data from over 20,000 store outlets across the country. Their objective was to gather vast amounts of daily pricing data across multiple product and service categories to accurately calculate the Consumer Price Index (CPI). This data would play a crucial role in shaping government policies and regulations, enabling a deeper understanding of market trends and consumer spending patterns.

Given the scale and complexity of the project, the client required a highly automated solution with minimal human intervention. The data needed to be sourced from diverse and disparate channels, ensuring comprehensive coverage while maintaining accuracy. Additionally, they sought to aggregate multiple data attributes per product category and implement robust anomaly detection mechanisms to eliminate inconsistencies, ensuring the CPI remained precise and reliable.

Challenges

The client approached Xtract.io as they encountered several challenges with tracking prices for a huge volume of stores. 

• Manual data extraction required significant time and effort, making the process inefficient and costly.
• Human errors in data entry led to inaccuracies, affecting the reliability of consumer price index calculations.
• Detecting and correcting data anomalies was slow and required extensive manual intervention.
• Frequent price fluctuations demanded real-time updates, which were difficult to achieve with manual processes.
• Without an automated solution, gathering and integrating accurate pricing data across multiple sources was challenging. 

Solutions

To tackle the client’s challenges, we implemented a fully automated, scalable solution that ensured accurate, real-time pricing data collection with minimal human intervention.

Custom Automation for Data Extraction

We developed site-specific custom bots to gather pricing data from retail websites, ensuring precise and efficient data collection. We automated data retrieval using Perl and Python scripts and presented it in an easily accessible format. Selenium-based browser automation was implemented to streamline the entire extraction process.

Intelligent Data Processing and Quality Assurance

Extracted data was stored in a structured database, where it underwent deduplication to remove redundancies. A two-level quality check ensured accuracy—first, an ML-based anomaly detection system identified inconsistencies, followed by manual random sampling for final validation. This rigorous process guaranteed high-quality, reliable data.

Continuous Data Collection for Market Coverage

To provide a comprehensive pricing dataset, we executed data extraction and integration throughout the year, including holidays. This ensured uninterrupted access to real-time pricing information, allowing for accurate consumer price index calculations.

Scalability and High-Volume Data Handling

Our solution gathered over one million records daily across diverse product and service categories, from aviation to clothing. The extracted data was formatted according to client specifications, making it easy to integrate into their analysis workflows.

Reliable Technical and Human Oversight

To maintain smooth operations, we combined technical expertise with dedicated human support, proactively monitoring the bots and addressing any challenges to ensure consistent, error-free data delivery.

Results

With our advanced data solutions, the client achieved seamless automation, real-time accuracy, and unmatched efficiency in price data collection.

Exclusive Data Partner

Our success led to Xtract.io becoming the sole offshore data partner for this prestigious organization, enabling them to calculate the consumer price index with unmatched precision.

Real-Time Price Tracking

With continuous data updates, our solution helped the client reflect real-time price changes in their calculations, ensuring accurate economic insights.

Unmatched Data Accuracy

Leveraging our ML-driven approach, we delivered high-quality data with over 98% accuracy, eliminating inconsistencies in the client’s pricing datasets.

Automated, Targeted Data Extraction

Custom-built bots extracted precise data points from massive datasets, ensuring the client received only the most relevant pricing information.

Significant Time Savings

Our automation reduced data collection time by two-thirds, allowing the client to focus on analysis and decision-making rather than manual data gathering.