WebPerception API integrates with LangChain in 5 lines of code. Scrape any website, take screenshots, and extract structured data with AI.
pip install requests langchainfrom langchain.tools import Tool
import requests
def web_perceive(url: str) -> str:
"""Scrape a URL and extract structured data using AI."""
response = requests.post(
"https://api.mantisapi.com/v1/scrape",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"url": url,
"extract": {
"schema": {
"title": "string",
"main_content": "string",
"links": ["string"]
}
}
}
)
return str(response.json())
scrape_tool = Tool(
name="web_scraper",
description="Scrape a website and extract structured data. Input: URL",
func=web_perceive
)from langchain.agents import initialize_agent, AgentType
from langchain.chat_models import ChatOpenAI
llm = ChatOpenAI(model="gpt-4", temperature=0)
agent = initialize_agent(
tools=[scrape_tool],
llm=llm,
agent=AgentType.OPENAI_FUNCTIONS,
verbose=True
)
# Your agent can now browse the web!
result = agent.run("What's the top story on Hacker News right now?")
print(result)No CSS selectors. No XPath. Define a JSON schema and get structured data back. Works even when websites change their HTML.
Full browser rendering handles SPAs, infinite scroll, and dynamic content. Your agent sees exactly what a human sees.
Built-in residential proxies, browser fingerprinting, and CAPTCHA handling. No blocked requests.
Take screenshots for your vision-enabled agents. Combine with GPT-4V for visual understanding of any webpage.
Designed as a LangChain Tool from day one. Works with OpenAI Functions, ReAct, and any agent architecture.
100 requests/month free. No credit card required. Start building your web-enabled agent in minutes.
Build an agent that researches any topic by crawling multiple sources, extracting key facts, and synthesizing a report.
result = agent.run(
"Research the top 5 AI startups that raised Series A in 2026. "
"For each, find their product, funding amount, and key investors."
)Track competitor pricing, features, and messaging changes in real-time. Get alerts when something changes.
result = agent.run(
"Check competitor.com/pricing and extract all plan names, "
"prices, and included features. Compare with our current pricing."
)Build an agent that monitors news sources and curates relevant articles for your team, complete with summaries.
result = agent.run(
"Check TechCrunch, The Verge, and Hacker News for AI-related "
"news from today. Summarize the top 3 stories."
)Free tier. No credit card. 100 requests/month to get started.
Get Free API Key → Read the Docs