Web Scraping for Construction & Infrastructure: How AI Agents Track Bids, Permits, Materials & Project Data in 2026

The global construction industry generates over $13 trillion annually, making it one of the largest economic sectors on the planet. In the United States alone, construction spending exceeded $2.1 trillion in 2025, spanning residential, commercial, industrial, and public infrastructure projects. The 2021 Infrastructure Investment and Jobs Act injected an additional $1.2 trillion into roads, bridges, broadband, water systems, and energy grid modernization — creating the largest infrastructure pipeline in a generation.

Yet construction remains one of the least digitized industries. Most contractors still discover bid opportunities through personal networks, track material prices via phone calls to suppliers, and learn about new permits weeks after they're filed. Market intelligence is fragmented across thousands of municipal portals, dozens of bid aggregators, and volatile commodity exchanges.

The opportunity: AI agents that continuously scrape, structure, and analyze construction data from hundreds of sources — permit databases, bid portals, commodity indices, OSHA enforcement records, and project tracking sites — to give contractors, suppliers, and developers the real-time intelligence that traditionally required $3K-$20K/month platforms.

Why AI Agents Need Construction Data

Construction intelligence requires data from sources that are fragmented by design — every city, county, and state runs its own systems:

• Permit tracking: Building permits, zoning applications, plan reviews, and certificates of occupancy from thousands of municipal portals — each with different formats, search interfaces, and update schedules
• Bid opportunities: Public RFPs, ITBs (Invitations to Bid), and RFQs from federal (SAM.gov), state, county, and municipal procurement portals, plus private bid boards and plan rooms
• Material pricing: Steel (HRC, rebar), lumber (framing, plywood), concrete (ready-mix), copper, PVC, asphalt — tracked across commodity exchanges, distributor portals, and regional supplier quotes
• Project pipeline: Projects moving through planning → permitted → under construction → completed, with contractor assignments, subcontractor opportunities, and timeline changes
• Safety & compliance: OSHA inspection results, violation citations, penalty assessments, and safety records for contractors, subcontractors, and project sites
• Labor market: Wage rates, workforce availability, union agreements, and prevailing wage determinations from DOL and state agencies

An AI agent using the Mantis WebPerception API can monitor all of these sources simultaneously, extract structured data regardless of format, and deliver actionable intelligence — new bids matching your trade, permit spikes signaling market opportunities, material price movements affecting your estimates, and competitor activity in your territory.

Step 1: Scraping Building Permits from Municipal Portals

Building permits are the earliest signal of construction activity. A new commercial permit filing can indicate a project 6-12 months before ground breaks — giving contractors, suppliers, and developers a massive head start on competitors who wait for projects to appear on bid boards.

The challenge: permit data lives across thousands of municipal portals, each with its own search interface, data format, and update schedule. Some use Accela, others use Tyler Technologies (EnerGov), and many smaller jurisdictions still use custom-built systems.

import httpx
from pydantic import BaseModel, Field
from datetime import date
from typing import Optional

class BuildingPermit(BaseModel):
    permit_number: str = Field(description="Official permit ID")
    permit_type: str = Field(description="Type: residential, commercial, industrial, demolition")
    status: str = Field(description="Status: submitted, under review, approved, issued, final")
    description: str = Field(description="Project description from filing")
    address: str = Field(description="Project site address")
    city: str = Field(description="City/municipality")
    county: str = Field(description="County")
    state: str = Field(description="State abbreviation")
    applicant: str = Field(description="Permit applicant name")
    contractor: Optional[str] = Field(description="Licensed contractor if listed")
    estimated_value: Optional[float] = Field(description="Declared project valuation in USD")
    square_footage: Optional[int] = Field(description="Square footage if listed")
    filing_date: date = Field(description="Date permit was filed")
    issue_date: Optional[date] = Field(description="Date permit was issued")

MANTIS_API_KEY = "your-api-key"
MANTIS_URL = "https://api.mantisapi.com/v1/scrape"

# Example: Scrape permits from a county portal
async def scrape_permits(jurisdiction_url: str, permit_type: str = "commercial"):
    async with httpx.AsyncClient(timeout=30) as client:
        response = await client.post(
            MANTIS_URL,
            headers={"x-api-key": MANTIS_API_KEY},
            json={
                "url": jurisdiction_url,
                "prompt": f"""Extract all {permit_type} building permits from this page.
                For each permit, extract: permit number, type, status, description,
                full address, applicant name, contractor (if listed),
                estimated project value, square footage, filing date, and issue date.
                Focus on permits filed in the last 30 days.""",
                "schema": BuildingPermit.model_json_schema()
            }
        )
        return response.json()

# Monitor multiple jurisdictions
JURISDICTIONS = [
    {"name": "Miami-Dade County", "url": "https://www.miamidade.gov/permits/"},
    {"name": "Maricopa County", "url": "https://eservices.maricopa.gov/building/"},
    {"name": "Harris County", "url": "https://permittingportal.harriscountytx.gov/"},
    {"name": "Clark County NV", "url": "https://citizenaccess.clarkcountynv.gov/"},
]

async def monitor_all_permits():
    for jurisdiction in JURISDICTIONS:
        permits = await scrape_permits(jurisdiction["url"])
        for permit in permits.get("extracted_data", []):
            p = BuildingPermit(**permit)
            if p.estimated_value and p.estimated_value > 500_000:
                print(f"🏗️ HIGH-VALUE PERMIT: {p.permit_number}")
                print(f"   {p.description}")
                print(f"   {p.address}, {p.city}, {p.state}")
                print(f"   Value: ${p.estimated_value:,.0f}")
                print(f"   Filed: {p.filing_date}")
                print(f"   Applicant: {p.applicant}")
                if p.contractor:
                    print(f"   Contractor: {p.contractor}")

Step 2: Tracking Bid Opportunities Across Portals

Bid opportunities are the lifeblood of construction companies. Missing a bid deadline by a single day means losing a potential multi-million-dollar project. The problem: bids are posted across dozens of platforms — SAM.gov for federal, state procurement portals, county purchasing departments, school district boards, and private plan rooms.

class BidOpportunity(BaseModel):
    bid_id: str = Field(description="Bid/solicitation number")
    title: str = Field(description="Project title")
    description: str = Field(description="Full project description")
    agency: str = Field(description="Issuing agency/owner")
    bid_type: str = Field(description="Type: ITB, RFP, RFQ, design-build, CMAR")
    trade_categories: list[str] = Field(description="CSI divisions or trade categories")
    estimated_budget: Optional[float] = Field(description="Estimated budget if disclosed")
    location: str = Field(description="Project location")
    post_date: date = Field(description="Date bid was posted")
    due_date: date = Field(description="Bid submission deadline")
    pre_bid_meeting: Optional[str] = Field(description="Pre-bid meeting date/location")
    bonding_required: Optional[bool] = Field(description="Whether bid/performance bond required")
    prevailing_wage: Optional[bool] = Field(description="Subject to prevailing wage requirements")
    set_aside: Optional[str] = Field(description="Set-aside: SBE, MBE, WBE, DBE, SDVOSB, 8(a)")
    plan_room_url: Optional[str] = Field(description="URL to download plans/specs")
    contact_name: Optional[str] = Field(description="Bid contact person")
    contact_email: Optional[str] = Field(description="Bid contact email")

BID_SOURCES = [
    {"name": "SAM.gov", "url": "https://sam.gov/search/?index=opp&sort=-modifiedDate&sfm%5Bstatus%5D%5Bis_active%5D=true&sfm%5BsimpleSearch%5D%5BkeywordRadio%5D=ALL&page=1"},
    {"name": "BidNet", "url": "https://www.bidnet.com/bids/construction"},
    {"name": "Texas SmartBuy", "url": "https://comptroller.texas.gov/purchasing/open-bids/"},
    {"name": "California eProcure", "url": "https://caleprocure.ca.gov/pages/Events-BS3/event-search.aspx"},
]

async def find_matching_bids(trade: str, max_distance_miles: int = 100):
    """Find bids matching a specific trade within a geographic radius."""
    all_bids = []
    for source in BID_SOURCES:
        async with httpx.AsyncClient(timeout=30) as client:
            response = await client.post(
                MANTIS_URL,
                headers={"x-api-key": MANTIS_API_KEY},
                json={
                    "url": source["url"],
                    "prompt": f"""Extract all active construction bid opportunities
                    related to {trade}. Include bids for general construction
                    that would require {trade} subcontractors.
                    Only include bids with deadlines in the future.""",
                    "schema": BidOpportunity.model_json_schema()
                }
            )
            bids = response.json().get("extracted_data", [])
            for bid in bids:
                b = BidOpportunity(**bid)
                all_bids.append(b)
                days_until_due = (b.due_date - date.today()).days
                print(f"📋 {b.title}")
                print(f"   Agency: {b.agency}")
                print(f"   Budget: ${b.estimated_budget:,.0f}" if b.estimated_budget else "   Budget: Not disclosed")
                print(f"   Due: {b.due_date} ({days_until_due} days)")
                print(f"   Set-aside: {b.set_aside}" if b.set_aside else "")
    return all_bids

Step 3: Monitoring Material Prices in Real Time

Construction material costs can swing dramatically. Lumber prices famously tripled during the 2021 supply chain crisis. Steel prices fluctuate with tariff announcements. Ready-mix concrete varies regionally based on aggregate availability and fuel costs. For a contractor submitting a bid today on a project that breaks ground in 6 months, material price forecasting can make or break profitability.

class MaterialPrice(BaseModel):
    material: str = Field(description="Material name: HRC steel, rebar, lumber, concrete, copper, PVC")
    grade_spec: Optional[str] = Field(description="Grade/spec: #4 rebar, 2x4 SPF, 4000 PSI ready-mix")
    price: float = Field(description="Current price per unit")
    unit: str = Field(description="Unit: per ton, per MBF, per cubic yard, per pound, per 100ft")
    price_change_pct: Optional[float] = Field(description="Price change percentage vs prior period")
    region: Optional[str] = Field(description="Geographic region if regional pricing")
    source: str = Field(description="Data source: CME, distributor, index")
    date: date = Field(description="Price date")
    lead_time_days: Optional[int] = Field(description="Current lead time in days if available")

MATERIAL_SOURCES = [
    {"name": "CME Steel HRC", "url": "https://www.cmegroup.com/markets/metals/ferrous/hrc-steel.html"},
    {"name": "Random Lengths Lumber", "url": "https://www.randomlengths.com/In-Depth/Monthly-Composite-Prices/"},
    {"name": "FRED Producer Price Index", "url": "https://fred.stlouisfed.org/series/PCU32733273"},
    {"name": "Engineering News-Record", "url": "https://www.enr.com/economics/"},
    {"name": "Metal Miner", "url": "https://agmetalminer.com/metal-prices/"},
]

import sqlite3

def store_material_price(db_path: str, price: MaterialPrice):
    conn = sqlite3.connect(db_path)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS material_prices (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            material TEXT, grade_spec TEXT, price REAL, unit TEXT,
            price_change_pct REAL, region TEXT, source TEXT,
            date TEXT, lead_time_days INTEGER,
            scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)
    conn.execute(
        "INSERT INTO material_prices (material, grade_spec, price, unit, price_change_pct, region, source, date, lead_time_days) VALUES (?,?,?,?,?,?,?,?,?)",
        (price.material, price.grade_spec, price.price, price.unit,
         price.price_change_pct, price.region, price.source,
         str(price.date), price.lead_time_days)
    )
    conn.commit()
    conn.close()

def detect_price_anomalies(db_path: str, threshold_pct: float = 5.0):
    """Alert when material prices spike beyond threshold."""
    conn = sqlite3.connect(db_path)
    cursor = conn.execute("""
        SELECT material, grade_spec, price, date,
               LAG(price) OVER (PARTITION BY material, grade_spec ORDER BY date) as prev_price
        FROM material_prices
        WHERE date >= date('now', '-30 days')
        ORDER BY material, date
    """)
    alerts = []
    for row in cursor:
        material, spec, price, dt, prev_price = row
        if prev_price and prev_price > 0:
            change_pct = ((price - prev_price) / prev_price) * 100
            if abs(change_pct) > threshold_pct:
                direction = "📈 SPIKE" if change_pct > 0 else "📉 DROP"
                alerts.append({
                    "material": material,
                    "spec": spec,
                    "change_pct": change_pct,
                    "price": price,
                    "prev_price": prev_price,
                    "direction": direction,
                    "date": dt
                })
                print(f"{direction}: {material} ({spec}) moved {change_pct:+.1f}% — ${prev_price:.2f} → ${price:.2f}")
    conn.close()
    return alerts

Step 4: OSHA Safety & Compliance Monitoring

Safety records are critical intelligence in construction. Before hiring a subcontractor, smart GCs check their OSHA history. Before bidding a project, savvy contractors research the owner's safety expectations. OSHA publishes inspection data, violation details, and penalty amounts — all public record, all scrapable.

class OSHAInspection(BaseModel):
    activity_number: str = Field(description="OSHA inspection activity number")
    establishment: str = Field(description="Company/establishment name")
    site_address: str = Field(description="Inspection site address")
    sic_code: Optional[str] = Field(description="SIC code")
    naics_code: Optional[str] = Field(description="NAICS code")
    inspection_type: str = Field(description="Type: planned, complaint, referral, fatality, follow-up")
    open_date: date = Field(description="Inspection open date")
    close_date: Optional[date] = Field(description="Inspection close date")
    violations_serious: int = Field(description="Number of serious violations", default=0)
    violations_willful: int = Field(description="Number of willful violations", default=0)
    violations_repeat: int = Field(description="Number of repeat violations", default=0)
    violations_other: int = Field(description="Number of other violations", default=0)
    total_penalty: float = Field(description="Total initial penalty amount in USD", default=0)
    violation_descriptions: list[str] = Field(description="List of violation standard descriptions")

async def check_contractor_safety(contractor_name: str):
    """Check a contractor's OSHA inspection history."""
    async with httpx.AsyncClient(timeout=30) as client:
        response = await client.post(
            MANTIS_URL,
            headers={"x-api-key": MANTIS_API_KEY},
            json={
                "url": f"https://www.osha.gov/cgi-bin/est/est1?sic=&p_es_name={contractor_name}&p_state=&p_zip=&p_city=&p_county=",
                "prompt": f"""Extract all OSHA inspection records for '{contractor_name}'.
                For each inspection, extract: activity number, establishment name,
                site address, inspection type, dates, violation counts by severity
                (serious, willful, repeat, other), total penalties, and
                descriptions of each violation standard cited.""",
                "schema": OSHAInspection.model_json_schema()
            }
        )
        inspections = response.json().get("extracted_data", [])
        
        total_serious = sum(i.get("violations_serious", 0) for i in inspections)
        total_penalties = sum(i.get("total_penalty", 0) for i in inspections)
        has_willful = any(i.get("violations_willful", 0) > 0 for i in inspections)
        has_fatality = any(i.get("inspection_type") == "fatality" for i in inspections)
        
        print(f"🔍 OSHA History for: {contractor_name}")
        print(f"   Total inspections: {len(inspections)}")
        print(f"   Serious violations: {total_serious}")
        print(f"   Total penalties: ${total_penalties:,.0f}")
        if has_willful:
            print(f"   ⚠️ WARNING: Has willful violations")
        if has_fatality:
            print(f"   🚨 ALERT: Has fatality-related inspections")
        
        return inspections

Step 5: AI-Powered Construction Intelligence

Raw data becomes actionable intelligence when you add AI analysis. GPT-4o can correlate permit filings with bid opportunities, assess material cost impact on active estimates, evaluate subcontractor risk profiles, and predict market trends from permit volume data.

from openai import OpenAI

openai_client = OpenAI()

async def analyze_market_opportunity(permits: list, bids: list, material_prices: list):
    """Use GPT-4o to generate a construction market intelligence briefing."""
    
    context = f"""
CONSTRUCTION MARKET DATA — {date.today().isoformat()}

RECENT PERMITS (Last 30 Days):
{chr(10).join(f"- {p.get('description', 'N/A')} | {p.get('address', 'N/A')} | Value: ${p.get('estimated_value', 0):,.0f} | Filed: {p.get('filing_date', 'N/A')}" for p in permits[:20])}

ACTIVE BIDS:
{chr(10).join(f"- {b.get('title', 'N/A')} | Agency: {b.get('agency', 'N/A')} | Budget: ${b.get('estimated_budget', 0):,.0f} | Due: {b.get('due_date', 'N/A')}" for b in bids[:20])}

MATERIAL PRICES:
{chr(10).join(f"- {m.get('material', 'N/A')} ({m.get('grade_spec', '')}): ${m.get('price', 0):.2f}/{m.get('unit', '')} | Change: {m.get('price_change_pct', 0):+.1f}%" for m in material_prices[:15])}
"""
    
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": """You are a construction market intelligence analyst.
            Analyze the data and provide:
            1. MARKET OVERVIEW: Permit volume trends, hot geographic areas, growing sectors
            2. BID RECOMMENDATIONS: Which bids to prioritize based on budget, competition likelihood, and alignment
            3. MATERIAL COST IMPACT: How current price trends affect active estimates and upcoming bids
            4. RISK ALERTS: OSHA trends, supply chain risks, labor market concerns
            5. 30-DAY OUTLOOK: What to expect in the next month based on permit pipeline and material trends
            Be specific with numbers. Flag actionable opportunities."""},
            {"role": "user", "content": context}
        ]
    )
    
    briefing = response.choices[0].message.content
    print("📊 CONSTRUCTION MARKET INTELLIGENCE BRIEFING")
    print("=" * 60)
    print(briefing)
    return briefing

async def estimate_bid_competitiveness(bid: dict, material_db: str):
    """Assess bid competitiveness based on current material costs and market conditions."""
    conn = sqlite3.connect(material_db)
    recent_prices = conn.execute("""
        SELECT material, grade_spec, price, unit, price_change_pct
        FROM material_prices
        WHERE date >= date('now', '-7 days')
        ORDER BY date DESC
    """).fetchall()
    conn.close()
    
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": """You are a construction estimating advisor.
            Given a bid opportunity and current material prices, assess:
            1. Material cost risk (are key materials trending up or down?)
            2. Recommended escalation clause percentage
            3. Suggested bid strategy (aggressive, competitive, premium)
            4. Key risks to flag in the bid
            5. Estimated material cost as % of total budget"""},
            {"role": "user", "content": f"BID: {bid}\n\nCURRENT MATERIAL PRICES:\n{recent_prices}"}
        ]
    )
    return response.choices[0].message.content

Step 6: Automated Alerts & Reporting

The final piece: automated delivery of intelligence to the people who need it. Bid coordinators get new opportunity alerts. Estimators get material price warnings. Safety directors get OSHA activity near their projects. Executives get weekly market briefings.

import httpx

async def send_slack_alert(webhook_url: str, alert_type: str, data: dict):
    """Send construction intelligence alerts to Slack."""
    
    if alert_type == "new_permit":
        message = {
            "blocks": [
                {"type": "header", "text": {"type": "plain_text", "text": "🏗️ New High-Value Permit Filed"}},
                {"type": "section", "text": {"type": "mrkdwn", "text": (
                    f"*{data['description']}*\n"
                    f"📍 {data['address']}, {data['city']}, {data['state']}\n"
                    f"💰 Estimated Value: ${data['estimated_value']:,.0f}\n"
                    f"📅 Filed: {data['filing_date']}\n"
                    f"👷 Applicant: {data['applicant']}\n"
                    f"{'🔨 Contractor: ' + data['contractor'] if data.get('contractor') else '⚡ No contractor assigned yet — OPPORTUNITY'}"
                )}}
            ]
        }
    elif alert_type == "bid_deadline":
        days_left = data.get("days_until_due", 0)
        urgency = "🔴" if days_left <= 3 else "🟡" if days_left <= 7 else "🟢"
        message = {
            "blocks": [
                {"type": "header", "text": {"type": "plain_text", "text": f"{urgency} Bid Deadline Alert — {days_left} Days Left"}},
                {"type": "section", "text": {"type": "mrkdwn", "text": (
                    f"*{data['title']}*\n"
                    f"🏛️ Agency: {data['agency']}\n"
                    f"📍 Location: {data['location']}\n"
                    f"💰 Budget: ${data.get('estimated_budget', 0):,.0f}\n"
                    f"📅 Due: {data['due_date']}\n"
                    f"{'📋 Pre-bid: ' + data['pre_bid_meeting'] if data.get('pre_bid_meeting') else ''}"
                )}}
            ]
        }
    elif alert_type == "material_spike":
        message = {
            "blocks": [
                {"type": "header", "text": {"type": "plain_text", "text": f"{data['direction']}: {data['material']} Price Alert"}},
                {"type": "section", "text": {"type": "mrkdwn", "text": (
                    f"*{data['material']}* ({data.get('spec', 'N/A')})\n"
                    f"Price moved *{data['change_pct']:+.1f}%*\n"
                    f"${data['prev_price']:.2f} → ${data['price']:.2f}\n"
                    f"⚠️ Review active estimates with significant {data['material'].lower()} exposure"
                )}}
            ]
        }
    
    async with httpx.AsyncClient() as client:
        await client.post(webhook_url, json=message)

# Run the full pipeline
async def construction_intelligence_pipeline():
    """Complete construction intelligence pipeline — run daily."""
    
    # 1. Scrape permits from target jurisdictions
    all_permits = []
    for jurisdiction in JURISDICTIONS:
        permits = await scrape_permits(jurisdiction["url"])
        for p in permits.get("extracted_data", []):
            store_permit(DB_PATH, p)
            if p.get("estimated_value", 0) > 500_000:
                await send_slack_alert(WEBHOOK_URL, "new_permit", p)
            all_permits.append(p)
    
    # 2. Check bid opportunities
    bids = await find_matching_bids(trade="electrical", max_distance_miles=150)
    for b in bids:
        days_left = (b.due_date - date.today()).days
        if 0 < days_left <= 7:
            await send_slack_alert(WEBHOOK_URL, "bid_deadline", {**b.model_dump(), "days_until_due": days_left})
    
    # 3. Track material prices
    prices = await scrape_material_prices()
    for p in prices:
        store_material_price(DB_PATH, p)
    anomalies = detect_price_anomalies(DB_PATH, threshold_pct=5.0)
    for a in anomalies:
        await send_slack_alert(WEBHOOK_URL, "material_spike", a)
    
    # 4. Generate AI briefing
    briefing = await analyze_market_opportunity(all_permits, [b.model_dump() for b in bids], [p.model_dump() for p in prices])
    
    print(f"\n✅ Pipeline complete: {len(all_permits)} permits, {len(bids)} bids, {len(prices)} prices tracked")

Advanced: Permit-to-Project Pipeline Tracking

The most powerful construction intelligence comes from tracking the full lifecycle: permit filed → plans submitted → permit issued → construction started → inspections → certificate of occupancy. Each stage represents a different opportunity for different players in the construction ecosystem.

class ProjectLifecycle(BaseModel):
    permit_number: str
    address: str
    project_type: str  # residential, commercial, industrial, infrastructure
    owner: str
    estimated_value: float
    stages: list[dict]  # [{stage, date, notes}]
    current_stage: str
    general_contractor: Optional[str]
    subcontractors: list[str]
    days_in_current_stage: int
    is_stalled: bool  # True if no progress in 60+ days

async def track_project_lifecycle(permit_number: str, jurisdiction_url: str):
    """Track a specific project through its full lifecycle."""
    async with httpx.AsyncClient(timeout=30) as client:
        response = await client.post(
            MANTIS_URL,
            headers={"x-api-key": MANTIS_API_KEY},
            json={
                "url": f"{jurisdiction_url}/permit/{permit_number}",
                "prompt": f"""Track the complete lifecycle of permit {permit_number}.
                Extract every stage the project has gone through:
                application, plan review, revisions, approval, issuance,
                inspection requests, inspection results, final inspection, CO.
                Include dates, inspector names, any notes or conditions.
                Identify the current stage and whether the project appears stalled.
                List all contractors and subcontractors associated with the permit.""",
                "schema": ProjectLifecycle.model_json_schema()
            }
        )
        project = response.json()
        
        lifecycle = ProjectLifecycle(**project.get("extracted_data", {}))
        
        if lifecycle.is_stalled:
            print(f"⚠️ STALLED PROJECT: {lifecycle.address}")
            print(f"   Stuck in '{lifecycle.current_stage}' for {lifecycle.days_in_current_stage} days")
            print(f"   Value: ${lifecycle.estimated_value:,.0f}")
            print(f"   Owner: {lifecycle.owner}")
            print(f"   This may indicate: financing issues, design changes, or contractor problems")
        
        if lifecycle.current_stage == "approved" and not lifecycle.general_contractor:
            print(f"⚡ OPPORTUNITY: {lifecycle.address} — approved but no GC assigned")
            print(f"   Value: ${lifecycle.estimated_value:,.0f}")
            print(f"   Owner: {lifecycle.owner}")
        
        return lifecycle

# Aggregate permit volume for market analysis
def analyze_permit_trends(db_path: str, geography: str):
    """Analyze permit filing trends to predict market direction."""
    conn = sqlite3.connect(db_path)
    trends = conn.execute("""
        SELECT 
            strftime('%Y-%m', filing_date) as month,
            permit_type,
            COUNT(*) as permit_count,
            SUM(estimated_value) as total_value,
            AVG(estimated_value) as avg_value
        FROM building_permits
        WHERE city = ? AND filing_date >= date('now', '-12 months')
        GROUP BY month, permit_type
        ORDER BY month DESC
    """, (geography,)).fetchall()
    conn.close()
    
    print(f"📊 Permit Trends for {geography} (Last 12 Months)")
    print("-" * 60)
    for month, ptype, count, total, avg in trends:
        print(f"  {month} | {ptype:12s} | {count:4d} permits | ${total:>12,.0f} total | ${avg:>10,.0f} avg")
    
    return trends

Construction Intelligence: Cost Comparison

Traditional construction data platforms charge premium prices for intelligence that's largely derived from public records:

The key difference: commercial platforms aggregate and repackage public data with polished interfaces and sales teams. An AI agent gives you the same data, customized to your exact trade, geography, and project criteria, at a fraction of the cost. You also own the data and the intelligence pipeline.

Use Cases by Company Type

General Contractors

GCs need the broadest view: track permits in target markets to identify upcoming projects early, monitor all relevant bid portals automatically, assess subcontractor safety records before qualification, and track material prices across active estimates. A single AI agent can replace 3-4 analysts manually checking portals every morning.

Subcontractors & Specialty Trades

Electrical, mechanical, plumbing, and specialty subs need bid opportunities filtered by their specific CSI division. An agent can monitor 50+ bid portals, flag opportunities in their trade and geography, track GC relationships (who wins what), and alert when large permits are filed that will eventually need their trade — months before the bid hits the street.

Material Suppliers & Distributors

Suppliers need permit data to forecast demand. A spike in residential permits in a market means increased demand for framing lumber, concrete, and electrical supplies 3-6 months out. Commodity price tracking helps with inventory purchasing decisions. Monitoring competitor pricing on distributor portals protects margins.

Real Estate Developers

Developers use permit data to gauge competition, identify zoning changes creating new opportunities, track infrastructure investments that increase land values, and monitor the permitting timeline in jurisdictions they're considering. Cross-referencing permit data with real estate market data creates a powerful development intelligence system.

Compliance & Ethical Considerations

Construction data scraping has a strong legal foundation:

• Building permits are public records — Filed with government agencies and subject to FOIA/public records laws in all 50 states. There is no legal barrier to collecting and analyzing permit data.
• Government bid postings are public by law — Federal, state, and local procurement opportunities must be publicly posted. Scraping these is not just legal, it's the intended purpose of publishing them.
• OSHA data is public — Inspection results, violations, and penalties are published by OSHA and available through their online search tools.
• Commodity prices on exchanges are published data — CME, LME, and other exchanges publish settlement prices. Redistribution terms vary, so check licensing.
• Private bid platforms have ToS — BidNet, PlanHub, iSqFt, and similar private platforms have terms of service. Respect rate limits and consider whether a partnership or API access is more appropriate.
• Rate limit everything — Municipal portals often run on limited infrastructure. Be a good citizen: space requests, cache aggressively, and scrape during off-peak hours.

Getting Started

Building a construction intelligence agent follows a clear progression:

1. Start with permits — Pick your top 5 target jurisdictions and scrape their permit portals daily. Filter for your market (commercial, residential, industrial) and minimum project value.
2. Add bid monitoring — Connect SAM.gov + your state procurement portal + 2-3 private bid boards. Filter by trade, geography, and project size.
3. Layer in materials — Track the 3-5 materials most critical to your trade. Set alerts for price movements >5% that would affect your estimates.
4. Build safety intelligence — Check OSHA records for every subcontractor before qualification. Monitor your own company name for new inspections.
5. Scale with AI — GPT-4o turns raw data into market briefings, bid recommendations, cost impact assessments, and competitive intelligence that would take analysts hours to compile.

Further Reading

• The Complete Guide to Web Scraping with AI Agents in 2026
• Web Scraping for Real Estate: Track Properties, Prices & Market Data
• Web Scraping for Government & Public Sector: Track Contracts, Grants & Policy Changes
• Web Scraping for Supply Chain & Logistics: Track Shipments, Inventory & Supplier Data
• Web Scraping for Price Monitoring: Build an AI-Powered Price Tracker
• Web Scraping for Market Research: Analyze Competitors, Trends & Opportunities