WebPerception API integrates as a custom function in AutoGen — scrape any page, take screenshots, and extract structured data with AI.
pip install pyautogen requestsimport requests
from typing import Annotated
def scrape_webpage(
url: Annotated[str, "The URL to scrape"],
extract_schema: Annotated[dict | None, "Optional JSON schema for AI extraction"] = None
) -> str:
"""Scrape a webpage and optionally extract structured data using AI."""
headers = {"x-api-key": "YOUR_API_KEY"}
if extract_schema:
# AI-powered structured extraction
resp = requests.post("https://api.mantisapi.com/v1/extract",
headers=headers,
json={"url": url, "schema": extract_schema}
)
else:
# Raw HTML/text scraping
resp = requests.post("https://api.mantisapi.com/v1/scrape",
headers=headers,
json={"url": url, "render_js": True}
)
return resp.json()import autogen
config_list = [{"model": "gpt-4", "api_key": "YOUR_OPENAI_KEY"}]
assistant = autogen.AssistantAgent(
name="researcher",
llm_config={"config_list": config_list},
system_message="You are a research assistant. Use scrape_webpage to gather information from the web."
)
user_proxy = autogen.UserProxyAgent(
name="user",
human_input_mode="NEVER",
code_execution_config={"use_docker": False}
)
# Register the function with both agents
assistant.register_for_llm(name="scrape_webpage", description="Scrape a webpage and extract data")(scrape_webpage)
user_proxy.register_for_execution(name="scrape_webpage")(scrape_webpage)
# Start the conversation
user_proxy.initiate_chat(
assistant,
message="Research the top 5 trending AI startups. Scrape TechCrunch and summarize findings."
)Any agent in your AutoGen group chat can call web scraping functions. Researcher agents gather data while analyst agents process it.
Don't parse HTML manually. Define a JSON schema and get structured data back. Prices, names, dates — whatever you need.
Take screenshots and feed them to GPT-4V. Your agents can "see" web pages, not just read their HTML.
No Selenium, no Playwright, no browser management. One API call handles JavaScript rendering, proxies, and anti-bot bypasses.
AutoGen's function registration system works perfectly with WebPerception. Register once, use across all agents in your group.
No monthly browser costs. 100 free API calls/month to start. Scale to millions when your agents need it.
Build an AutoGen group chat with a researcher agent that scrapes sources, a fact-checker that verifies claims, and a writer that produces reports. WebPerception gives the researcher real-time web access.
# Group chat with web-enabled researcher
groupchat = autogen.GroupChat(
agents=[user_proxy, researcher, fact_checker, writer],
messages=[], max_round=12
)
manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)
user_proxy.initiate_chat(manager, message="Write a report on the state of AI agents in 2026")Agents that monitor competitor websites daily, extract pricing changes, and alert your team. AutoGen orchestrates the workflow; WebPerception handles the scraping.
# Extract competitor pricing with AI
result = scrape_webpage(
url="https://competitor.com/pricing",
extract_schema={
"plans": [{
"name": "string",
"price": "string",
"features": ["string"]
}]
}
)Build agents that track product prices across multiple stores, compare deals, and notify when prices drop below a threshold. AutoGen manages the multi-step workflow automatically.
# Track prices across stores
stores = ["https://store-a.com/product/123", "https://store-b.com/item/456"]
for store in stores:
data = scrape_webpage(store, extract_schema={
"product_name": "string",
"price": "number",
"in_stock": "boolean"
})100 free API calls/month. No credit card required.
Get Your API Key →