Track daily

-----|-------------|------|-------------|-------|

| Raw HTTP | ~5% | Free | High | Fast |

| Headless + Proxies | ~60-70% | $100+/mo | Very High | Slow |

| Google API | 100% | $5/1K queries | Low | Fast |

| WebPerception | 99%+ | $0.005/call | None | Fast |

Building a SERP Tracker

Here's a practical example — tracking your keyword rankings:

import requests
import json
from datetime import datetime

def track_rankings(keywords, domain):
    rankings = {}
    
    for keyword in keywords:
        response = requests.get(
            "https://api.mantisapi.com/v1/scrape",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            params={
                "url": f"https://www.google.com/search?q={keyword}&num=20",
                "format": "structured"
            }
        )
        
        data = response.json()
        position = None
        
        for i, result in enumerate(data["data"]["organic_results"], 1):
            if domain in result["url"]:
                position = i
                break
        
        rankings[keyword] = {
            "position": position,
            "date": datetime.now().isoformat()
        }
    
    return rankings

# Track daily
keywords = ["web scraping api", "ai data extraction", "scrape website api"]
results = track_rankings(keywords, "mantisapi.com")
print(json.dumps(results, indent=2))

Giving AI Agents Search Access

The most powerful use case: letting your AI agent search the web:

def agent_search_tool(query: str) -> str:
    """Search Google and return top results for the agent."""
    response = requests.get(
        "https://api.mantisapi.com/v1/scrape",
        headers={"Authorization": "Bearer YOUR_API_KEY"},
        params={
            "url": f"https://www.google.com/search?q={query}",
            "format": "markdown"
        }
    )
    return response.json()["content"]

This is a single function that gives any AI agent real-time web search capabilities. No browser setup, no proxy management, no CAPTCHA solving.

Legal Considerations

Before scraping Google:

Google's ToS prohibit automated access to search results

SERP APIs (like WebPerception) operate in a legal gray area — they're widely used and accepted by the industry

Rate limiting — Even with an API, be respectful of query volume

Data usage — Don't redistribute raw SERP data commercially

Consider alternatives — For simple needs, Google's official API may suffice

Getting Started

Sign up at mantisapi.com — 100 free API calls/month

Test a query — Scrape a Google search result page

Parse the results — Use structured JSON output

Build your pipeline — SERP tracking, research, or agent tools

Conclusion

Scraping Google in 2026 is an arms race you don't want to fight alone. Between CAPTCHAs, IP bans, and constantly changing HTML, maintaining a Google scraper is a full-time job.

WebPerception API abstracts away the entire problem. One API call = structured search results, every time.

Stop fighting Google's anti-bot systems. Try WebPerception API free →