# Skill: Build a data-scraping bot with AI enrichment, deduplication, and a free schedule

> Specify and build a three-phase web scraper (collect -> enrich -> store) that runs on a free cloud schedule, calls AI in batches to avoid rate limits, and learns from user feedback. Applies when you need to automate recurring data collection from public websites.

Source: sodigi·learn — vibecoding/bot-sobiraet-dannye · https://sodigi.io/learn/vibecoding/bot-sobiraet-dannye

## When to use
- You are manually visiting multiple sites to collect prices, job postings, listings, or news on a schedule.
- You need AI to rank or filter the collected items for relevance.
- You want the bot to run for free without a dedicated server.
- You want the bot to improve its scoring based on your accept/reject decisions.

## Core rules
- Structure every scraper as three phases: collect (scrape HTML), enrich (AI scores and summarizes), store (write to table, skip duplicates).
- Call the AI in batches of 5 items, never one at a time -- one-at-a-time calls exhaust free-tier limits before the first page finishes.
- Before storing, check whether the item's URL already exists in the table -- if yes, skip it.
- Put all keywords, filters, and configuration in a separate settings file, never hardcoded in the source.
- Store credentials and API keys in a secure environment variable file, never in the code.
- Add polite delays between site visits and respect robots.txt. If the site has an official API, use it instead.
- For dynamic sites (infinite-scroll feeds), use a headless browser tool that waits for content to load.
- Implement a feedback journal: record user keep/skip decisions and feed them to the AI on the next run for smarter scoring.

## Procedure
1. Define the three floors: which URLs to scrape (collect), what AI scoring criteria to apply (enrich), and where to store results (store -- Notion/Sheets/DB).
2. Identify the data fields to extract per item: title, URL, key attributes.
3. Write the scraper specifying batch size (5), duplicate check (by URL), delay between requests, robots.txt respect.
4. Wire in a free AI model (e.g., Gemini Flash free tier) to score each batch 0-100 and write a one-line summary.
5. Write the storage layer with an "already seen" URL check before insert.
6. Add a GitHub Actions workflow (or equivalent free scheduler) to trigger the bot on a cron schedule.
7. Move all keywords/filters to a settings file and all secrets to env vars.
8. Optionally add a feedback log that the enrichment step reads to improve scores over time.

## Ready-to-use prompt
```
Build a bot that tracks job postings automatically and stores them in a table.

1. Once a day visit the target job sites and collect new postings: title, link, company, salary.
2. Before processing, filter out items that clearly do not match keywords. Put all settings (keywords, filters) in a separate config file so they can be changed without touching code.
3. Connect a free AI model. Score each posting 0-100 for fit and write a short explanation. Call the AI in batches of 5, not one at a time.
4. Store results in a table. Before adding, check whether that link is already there -- no duplicates.
5. Run the bot on a free schedule once a day (GitHub Actions or equivalent).
6. Do not write passwords or API keys in the code -- use environment variables.

Show the plan and folder structure first, then write the code.
```

## Pitfalls
- Calling AI for each item individually -- hits free-tier rate limits by the third page; always batch (5 per call).
- Hammering the site with no delay -- the site bans your IP; add pauses between requests.
- Not checking for duplicates -- within a week the table fills with hundreds of copies of the same item.
- Hardcoding keywords in the source -- requires editing code to change a search term; use a config file.
- Putting API keys in the code -- they leak when the project is pushed to a public repo.
- Using a heavy scraper when the site provides an official API -- prefer the API; it is faster and more reliable.