Your scraper breaks every time the target site changes a div or adds a CAPTCHA? And when content loads via JavaScript, you just get empty pages. We see this often with clients who come to us with fragile data extraction pipelines built on static selectors and regex. The problem isn't the technique—it's the approach. You're asking a program to do what a human does naturally: understand where the data is and adapt. With generative AI, you can turn scraping into intelligent browsing, where the agent decides how to interact with the page. Let's see how it works and how to build one.
Why traditional web scraping is becoming obsolete?
If you've ever built a scraper with Requests + BeautifulSoup or Scrapy, you know the pain: the site restructures its CSS, changes IDs, or introduces async loading, and your script goes blank. Add modern CAPTCHAs, Cloudflare challenges, and aggressive rate limiting, and the game is over. Sites are increasingly dynamic: React, Vue, SPAs. The traditional scraper doesn't 'see' the rendered DOM—you need a headless browser. But even with Puppeteer or Playwright, the extraction logic remains rigid: you have to manually update selectors every time something changes. It's a war of attrition your budget can't win.
The hidden cost of manual scraping
We at Meteora Web have followed clients spending hours each week fixing scraping scripts. That translates to €500+ per month in maintenance, for data that arrives late or incomplete. The solution is not a more robust script—it's an agent that understands what to take and where to go.
Key difference: a classic scraper executes fixed instructions. An AI agent interprets a goal (e.g., 'get the price of all products on this page') and moves on its own, clicking, scrolling, and extracting what's needed.
How does an AI agent for scraping work?
The idea is simple: instead of writing driver.find_element(By.CLASS_NAME, 'price'), you give the agent a natural language command. The agent uses a language model (LLM) to decide which actions to perform in the browser—click a button, read text, navigate to a URL. It then executes with an automation tool (Playwright, Puppeteer) and returns the result to the LLM to evaluate whether the goal is met.
Sponsored Protocol
Typical agent architecture
- LLM orchestrator (GPT-4, Claude 3.5, or local models like Llama 3) – receives the task and decides steps.
- Headless browser (Playwright, Puppeteer) – executes actions (click, scroll, extract).
- Tool functions – each action is a callable function from the LLM (e.g.,
click_element(selector),extract_text(selector)). - Memory – the agent keeps track of page state and already collected data to avoid repeating actions.
The agent runs a loop: it observes the DOM (often simplified into a text summary), decides the next action, executes it, analyzes the result, and continues until completion.
Which tools to use for building an AI scraping agent?
There are already ready-made frameworks, but often it's better to build a custom solution to control costs and latency. Here are the three main paths.
1. Crawl4AI – fastest to start
Crawl4AI is an open-source Python library designed to extract data from sites using AI. It supports local and remote LLMs, structured extraction, and automatic JavaScript handling. Perfect if you want an 'intelligent' scraper without writing the decision loop from scratch.
import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler(verbose=True) as crawler:
result = await crawler.arun(
url="https://example.com/products",
bypass_cache=True,
extraction_strategy="llm",
llm_config={"provider": "openai/gpt-4", "api_token": "sk-..."}
)
print(result.extracted_content)
asyncio.run(main())Caution: Crawl4AI uses API tokens. Estimate cost per page: a powerful LLM can consume $0.01–0.03 per request. For large volumes, use smaller models (Llama 3 8B) on a local GPU.
Sponsored Protocol
2. OpenAI Functions + Playwright – granular control
If you need complex logic (e.g., login, dynamic pagination, modals), build your own loop with OpenAI's function calling.
from openai import OpenAI
from playwright.async_api import async_playwright
import json
client = OpenAI(api_key="sk-...")
async def navigate(url: str) -> str:
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page()
await page.goto(url)
# Get page summary (titles, links, brief text)
summary = await page.evaluate('''() => {
return JSON.stringify({
title: document.title,
links: Array.from(document.querySelectorAll('a')).slice(0,10).map(a => a.href),
text: document.body.innerText.substring(0,2000)
});
}''')
await browser.close()
return summary
functions = [
{
"name": "navigate",
"description": "Navigate to a URL and return a summary of the page",
"parameters": {
"type": "object",
"properties": {
"url": {"type": "string", "description": "Full URL"}
},
"required": ["url"]
}
}
]
# Agent decision loop
messages = [{"role": "system", "content": "You are an assistant that extracts data from websites. Use navigate to explore pages."},
{"role": "user", "content": "Find the price of product 'iPhone 16' on an e-commerce site."}]
# Initial call
response = client.chat.completions.create(
model="gpt-4",
messages=messages,
functions=functions,
function_call="auto"
)Then handle the response, execute the function, and continue until the LLM returns the final data.
Sponsored Protocol
3. Browser-use – preconfigured agent
Browser-use is a project that integrates an LLM and Playwright into a ready-to-use agent. Just describe the task in natural language and it executes. Useful for prototypes, but less flexible for sites with unique interaction patterns.
How to avoid blocks and manage rate limiting with an agent?
Even the smartest agent gets blocked if it behaves like a bot. The difference is that AI can mimic a human: move the mouse, scroll naturally, wait random times. Here are the practices we use in our projects.
Proxy rotation and user-agent
Never use a single IP at scale. Integrate residential proxies (e.g., BrightData, Oxylabs) and change user-agent on each request. With Playwright you can set a proxy per session:
browser = await p.chromium.launch(proxy={"server": "http://proxy:8080"})Also rotate viewport, browser language, and initial cookies to look like a real user.
Human delays and random behaviors
The agent must introduce random delays between actions (1–3 seconds). Use await page.wait_for_timeout(random.uniform(1000, 3000)). Also simulate gradual scrolling instead of jumping directly to the bottom.
Handling CAPTCHAs
If the site presents reCAPTCHA, you have three options:
- Avoid detection: use residential proxies and human-like behavior – many sites don't trigger CAPTCHA unless they perceive suspicious activity.
- Paid solving services: 2Captcha, Capsolver – integrable via API, cost ~$0.001 per CAPTCHA.
- Vision AI: for simple CAPTCHAs (distorted text), use an OCR model + LLM. Not recommended for reCAPTCHA v3.
Practical example: an agent that extracts price lists from e-commerce competitors
Let's put it all together: an agent that navigates an e-commerce clothing site (e.g., Zalando) and collects name, price, and availability of all winter jackets. We use Playwright + OpenAI with an extraction function.
Sponsored Protocol
import asyncio
from playwright.async_api import async_playwright
from openai import OpenAI
import json
import random
client = OpenAI(api_key="sk-...")
async def extract_products(url: str) -> list:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
context = await browser.new_context(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
locale="en-US",
viewport={"width": 1920, "height": 1080}
)
page = await context.new_page()
await page.goto(url, wait_until="networkidle")
# Slow human-like scroll
for _ in range(5):
await page.evaluate("window.scrollBy(0, window.innerHeight * 0.2)")
await asyncio.sleep(random.uniform(0.5, 1.5))
# Extract reduced DOM for the agent
dom_snapshot = await page.evaluate('''() => {
function extract(element, depth=0) {
if (depth > 3) return '';
let items = [];
for (let child of element.children) {
let tag = child.tagName.toLowerCase();
if (['script', 'style', 'noscript'].includes(tag)) continue;
let text = child.innerText?.substring(0,200);
if (text) items.push({tag, text, children: extract(child, depth+1)});
}
return items;
}
return JSON.stringify(extract(document.body));
}''')
await browser.close()
# Ask the LLM to extract products
response = client.chat.completions.create(
model="gpt-4o-mini", # cheaper
messages=[
{"role": "system", "content": "Extract all products from the page. Return a JSON array with objects {name, price, available: true/false}."},
{"role": "user", "content": f"Here is the page structure: {dom_snapshot[:10000]}"}
]
)
content = response.choices[0].message.content
# Extract JSON from response (may contain markdown)
import re
json_match = re.search(r'\[.*\]', content, re.DOTALL)
if json_match:
return json.loads(json_match.group())
return []
asyncio.run(extract_products("https://www.zalando.com/winter-jackets/"))Caution: This code is demonstrative. In production you need to handle errors, pagination, and limit the DOM size sent to the API. We use a selective 'cut and paste' strategy to reduce costs.
Sponsored Protocol
What to do next
We've covered the problem, concepts, tools, and a concrete example. Now it's your turn.
- Build a minimal agent with a cheap LLM (GPT-4o-mini or Claude 3 Haiku) and a simple site that doesn't block. Try extracting titles and links from a static page.
- Test human-like behavior: add proxies and delays, verify that CAPTCHA doesn't trigger.
- Measure costs: calculate how many tokens you consume per page. With GPT-4o-mini, a page with ~2000 input tokens and 200 output tokens costs about $0.0006. For 1000 pages, $0.60. Much less than a developer fixing scripts every week.
- Apply the pattern to a real case: extract competitor prices for your industry (clothing, electronics, groceries). Remember: respect terms of use and frequency.
- Read the documentation for Playwright Python and OpenAI Function Calling to go deeper.
We at Meteora Web use these agents to monitor competitor prices for our e-commerce clients. If you need a robust pipeline or a custom agent, check our pillar on AI agentics or talk to us.
