Build an automation script with AI: bots and scrapers

To build an automation script with AI, you describe a repetitive job to a coding agent and test the small Python script it writes. Renaming files, merging Excel and CSV files, a weekly email report or a scraper that tracks prices on a site usually needs 50–200 lines. The hard part isn’t the prompt: it’s describing the input precisely, checking the output, scheduling the script and hearing about it when it breaks.

This post goes from nothing to a working automation: what to install, which tool to pick when, what to tell the agent first, how to test, and how to run it every day with cron. I also cover the legal and ethical side of scraping, because the AI won’t think about that for you.

What “automation with AI” means here

I don’t mean a model doing the job itself every time. The agent writes a script once; from then on the script runs the same way every day, without any AI. That distinction matters: a script is cheap, repeatable and you can read what it does. A setup that calls a model every day costs more and is harder to predict.

This is vibe coding: you don’t write the code line by line, you describe what you want and steer the result. Automations suit it well because the goal is clear and the result is easy to check: were the files named correctly, does the sheet have the right rows?

Which jobs are worth automating?

A good candidate repeats often, has rules you can write down and can be undone if it goes wrong. Everyday examples:

• Files: rename invoices in your Downloads folder by date and move them into year/month folders.
• Excel and CSV: merge the sales files each branch sends, drop empty rows, build a summary sheet.
• Email reports: pull data from a folder or a database every week and send a summary to yourself or your team.
• Watching the web: track a product’s price, new entries on a listings page or an announcements page, and get told when something changes.

For anything where a person should stay in charge (paying someone, sending a contract), stop the automation at “prepare it, I’ll approve”.

Step 1: install the tools

You need a coding agent and Python. Any terminal-based coding agent works: Claude Code, Codex, Gemini CLI or Aider can all read and write files and run commands.

# Is Python 3 installed?
python3 --version

# project folder and virtual environment
mkdir invoice-sorter && cd invoice-sorter
git init
python3 -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate

Why Python? The automation libraries are mature: pandas and openpyxl for spreadsheets, requests for HTTP, beautifulsoup4 for parsing HTML, playwright for a browser. Agents have seen plenty of code using them, so they make fewer mistakes. Making the folder a git repository means you can undo any change the agent makes.

Step 2: the first prompt

A vague request gets you a vague script. A good first message has the input, the output, the rules and the edge cases. Put two or three real sample files in the folder (fake ones if they contain personal data) and point the agent at them:

Write a Python script: rename_invoices.py

Input: PDF invoices in ./inbox. Examples are in ./samples.
Page one of each PDF has an "Invoice date: DD/MM/YYYY" line and a "Company:" line.

What it does:
- Rename the file to YYYY-MM-DD_company-name.pdf (lowercase, hyphens instead of spaces).
- Move it to ./archive/YYYY/MM/.
- If the date or company can’t be read, leave the file alone, copy it to ./failed and log it.

Rules:
- Run as --dry-run by default: print what it would do, touch nothing.
- Require an --apply flag for the real run.
- Never overwrite an existing file; append -2 instead.
- List dependencies in requirements.txt.
- When done, run it on ./samples with --dry-run and show me the output.

Two rules here are worth copying into every automation: dry run by default and never overwrite. If the agent misreads a date, you get a list to review instead of hundreds of files moved to the wrong place.

Step 3: iterate and test

The first version rarely handles every file. The loop:

1. Read the --dry-run output. Is anything matched wrongly?
2. Show the agent the problem file: “samples/invoice_7.pdf came out as 1970-01-01; that file uses the label ‘Issue date’. Support both.”
3. Ask it to add a small test for that case. A few sample files with pytest are enough.
4. When everything looks right, run --apply on a copy first, then on the real folder.

For Excel and CSV jobs the check is simpler: compare the row count of the input with the totals in the output. Have the script itself assert something like “the merged file should have 1,240 rows and this revenue total”. Code that runs is not proof that it calculates correctly.

Step 4: the scraper — requests or Playwright?

The first decision in scraping is how the page is built. Right-click, choose “View page source” and look for the data you want (the price, the title) in it.

If it’s in the source, requests and BeautifulSoup are enough. They’re fast, open no browser and put little load on the server.

import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "price-tracker/1.0 (contact@example.com)"}

resp = requests.get("https://example.com/product/123", headers=HEADERS, timeout=20)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
price = soup.select_one(".product-price")
if price is None:
    raise RuntimeError("Price element not found; the page layout may have changed")
print(price.get_text(strip=True))

If the content arrives later via JavaScript, or you need to log in, click or scroll, you need a real browser. Playwright is a common choice:

pip install playwright
playwright install chromium

Don’t tell the agent which to use; ask it to check first:

I want the product names and prices from https://example.com/category/headphones.
First check robots.txt and the page source: is the data in the HTML, or loaded
by JavaScript? If the site has a public API or a JSON endpoint, prefer it.
Then pick the simplest approach and explain why.
Wait at least 3 seconds between requests and visit at most 5 pages.
Write the result to products.csv.

A tip: on many sites you open with Playwright, the browser’s network tab shows the data actually comes from a JSON request. Calling that endpoint directly can be sturdier and lighter, though it is still subject to the site’s terms.

Legal and ethical limits: robots.txt, terms, GDPR

An AI will happily write code that scrapes a site; whether you should is your call. This isn’t legal advice, but these are my working rules:

• Official API first. Many sites, exchanges, couriers and public bodies offer their data through an API. It’s more stable, its rules are explicit and it doesn’t break when the page design changes.
• Read robots.txt and the terms of service. If the terms forbid automated access or robots.txt disallows the path, don’t. Scraping pages behind a login usually breaks the terms.
• Stay away from personal data. Collecting names, phone numbers or emails falls under GDPR (and KVKK in Turkey); without a purpose, a legal basis and a retention period, don’t collect it.
• Go slow. Wait a few seconds between requests, send one at a time, don’t refetch pages you already have. Use a User-Agent that identifies you.
• Mind copyright. Tracking a price for yourself is different from copying someone’s content and republishing it.
• Don’t fight the defences. Solving CAPTCHAs, rotating IPs or getting around login protection are signs the site doesn’t want you there.

Step 5: errors, retries and alerts

A script that works today breaks quietly next week: the site changes its layout, someone adds a column to the spreadsheet, the network drops. Asking for this up front saves you trouble:

Harden the script for daily use:
- Add timeouts to network requests; on transient errors (connection error, 429, 5xx)
  retry up to 3 times with increasing waits. Don’t retry 4xx errors.
- Log every run to logs/YYYY-MM-DD.log.
- Treat zero results, or a missing expected field, as an error.
- On error, email me using the SMTP settings in .env, with a summary of the error
  and the path of the log file.
- Send nothing on success.

“Zero results is an error” is the rule that pays off most. Most scrapers don’t crash when they break; they produce an empty CSV. Instead of email you can use Slack or Telegram webhooks and bot APIs; pick whatever you’ll actually notice.

Step 6: schedule it with cron

Once the script is ready, the simplest way to run it regularly on macOS and Linux is cron. Run crontab -e and add:

# Run every day at 08:30
30 8 * * * cd /Users/you/price-tracker && .venv/bin/python track.py >> logs/cron.log 2>&1

Things to watch:

• cron doesn’t load your shell environment. Use the full path to Python (the one inside the virtual environment) and cd into the folder.
• cron doesn’t run while the machine sleeps. For jobs that must run every day, a small Linux server or a scheduled GitHub Actions workflow is more reliable.
• On Windows, Task Scheduler does the same job; ask the agent for a .bat file and the setup steps.
• Check the logs every day for the first week.

What the AI commonly gets wrong

• Inventing selectors. Without seeing the page, the agent guesses a CSS class like .price. Give it the real HTML, or ask it to download and inspect the page.
• Swallowing errors. An except: pass block makes the script look like it works while doing nothing. Read every except line.
• Date and number formats. European-style 1.234,56 numbers and DD/MM/YYYY dates get misread often. Compare a few output values against the source by hand.
• Encoding. If a CSV with accented characters looks garbled, ask it to try utf-8-sig or the right legacy encoding.
• Overconfident success messages. “The script is done and tested” is not evidence on its own. Look at the output yourself.
• Too many dependencies. Spinning up browser automation or a framework for a ten-line job. Ask for the simplest solution.

Security

• Secrets go in .env. SMTP passwords, API keys and logins never go in the code; read them from .env with python-dotenv and add .env to .gitignore. Commit a .env.example with dummy values.
• Never paste a key into the chat. Don’t type “my key is …” to the agent. Conversations can be stored and end up in logs. Give it only the variable name: “read it from the SMTP_PASSWORD environment variable.”
• Read the commands the agent runs. Especially rm, bulk mv, pip install of packages you don’t know, and anything that sends data out. If you’re not sure, don’t approve it.
• Least privilege. Use an app password for an email account, and a read-only key for an API.

Doing it with AgentVera

All of this works in a plain terminal with one agent. When I run several automations at once I use AgentVera, because it smooths a few steps:

• Separate agents, separate folders. In the workspace, Claude Code can write the scraper in one pane while Codex builds the Excel report in another. If they share a repository, give each its own git worktree to avoid collisions.
• Agent browser. Agents using Playwright drive a Chrome with its own profile, and you watch what they do live in the agent browser. It makes it easier to see why a selector doesn’t match.
• Running on a server. From the SSH and SFTP page you can copy the script to a server and set up the crontab there.
• Scheduled agent messages. Scheduled tasks schedule a message to an agent, not your script. For example, once a week: “read the logs and summarise any failures”. The daily run should still be cron’s job.
• Review the diff. Code review checks each finished turn, which helps catch swallowed errors during the hardening step.

If you want to chain agents together, see AI agent workflow pipelines; to connect an agent to outside tools, see what is an MCP server.

Quick checklist

• Input, output and edge cases written in the first prompt
• Dry run by default, no overwriting
• Tested on sample files, a few results checked by hand
• Official API preferred where one exists; robots.txt and terms read
• Delays between requests, timeouts and limited retries
• Zero results count as failure, failures send an alert
• Secrets in .env, .env out of git
• Full paths in the cron line, logs checked in the first week

Conclusion

The trick to building automations with AI is to give the agent a small, well-described job and then treat the script it writes like software: dry runs, tests, logs, alerts. On the scraping side the technical choice is easy (requests if it’s in the source, Playwright if not); the harder part is respecting the limits and choosing the official route when there is one. For your first automation, pick a task you did by hand three times this week, adapt the prompt above and start with a dry run.