We are seeking a highly skilled professional to develop and maintain a comprehensive database of stores and their associated products. This project involves extracting data from nine APIs with specific constraints and using advanced methods to validate and structure the data into a usable format. The data sources only provide the following fields: store name, address, and latitude/longitude. Your task will include finding phone numbers and emails for these stores by cross-referencing and validating data.
Project Scope:
– Data Extraction:
– Integrate with nine distinct APIs, each with its own data structure and a limitation of 100 responses per query.
– Use strategic latitude/longitude searches to collect approximately 2,000 records per API while avoiding rate limits.
– Ensure geographic coverage across the U.S. through efficient query management.
– Data Validation and Deduplication:
– Clean and normalize poorly formatted data, correcting typos, inconsistencies, and invalid entries.
– Use AI-powered tools to cross-verify store names, addresses, phone numbers, emails, and websites where applicable.
– Match and merge records where slight variations in store name or address may refer to the same store, ensuring deduplication.
– Build a confidence rating system using AI to evaluate the accuracy and reliability of phone numbers, emails, and other derived data.
– Relational Data Structuring:
– Establish a relational database that associates each store with its specific products.
– Link validated store records (name, address, contact information, latitude/longitude) to their respective products.
– Cross-associate records to identify and group stores with similar details that might belong to the same entity.
– Deliverables:
– A relational database with the following features:
– Stores: Includes name, address, latitude/longitude, and validated phone number and email.
– Products: Relationally linked to the correct stores.
– Confidence Ratings: AI-generated scores indicating the accuracy of phone numbers, emails, and any other derived details.
– Duplicate Resolution: Ensures that stores with similar names or addresses are linked appropriately.
– AI Integration:
– Use AI tools to:
– Validate and verify phone numbers and emails.
– Cross-reference store details for duplicate resolution.
– Generate confidence ratings for derived data.
– Implement adaptive learning to improve validation and data accuracy over time.
– Ongoing Maintenance:
– Refresh the database quarterly with updated data.
– Monitor API changes and adapt to new structures or limitations.
– Maintain consistent data quality across all updates.
Preferred Skills:
– Proficiency in data scraping tools (e.g., Scrapy, Beautiful Soup, Selenium).
– Expertise in handling APIs with rate limits and diverse data structures.
– Experience with AI-driven tools for data validation and confidence scoring.
– Strong relational database management skills (e.g., PostgreSQL, MySQL, SQLite).
– Advanced data cleaning and normalization techniques.
– Familiarity with geographic data and search APIs (e.g., Google Search API, Bing API).
To Apply:
Please include the following in your proposal:
– Your experience with similar projects involving data scraping and validation.
– Examples of how you’ve used AI for data validation and confidence scoring.
– Your approach to handling duplicate records and poor formatting in datasets.
– A timeline and cost estimate for the initial project and ongoing quarterly updates.
Note: The only data points provided by the APIs are store name, address, and latitude/longitude. Your role is to derive and validate phone numbers and emails for these stores using external validation methods. This is a critical aspect of the project and requires attention to detail and a high level of accuracy.
