LoginSign Up
← Back to all sparks
F

Firecrawl

AI-ASSISTANTS
Velocity6.3

Web scraping and crawling API that turns websites into clean, LLM-ready markdown and structured data.

firecrawl.dev ↗

Firecrawl is rebuilding web scraping as token-cheap, grounded infrastructure for agents.

web-scrapingagent-nativetoken-efficiencygroundingmonitoringresearch-index
◆Current state
Firecrawl has moved well past 'turn a page into Markdown.' Nearly every recent release optimizes for the two things agents care about: minimal tokens and provable grounding. Question and Highlights formats, an excerpt-returning /search, and the arXiv/GitHub Research Index all hand back just the relevant lines with citations instead of whole pages, repeatedly claiming benchmark wins and '10-100x fewer tokens.' A parallel security track (Lockdown Mode, PII redaction, prompt-injection hardening) and a monitoring track that watches first pages, then the whole web, round it out.
◆Where it's heading
The product is consolidating into an agent-native web-data platform where every endpoint is judged on accuracy-per-token. The benchmark-and-efficiency framing — SimpleQA, arXivQA, token counts — is now the through-line of releases, and the search, monitor, and research surfaces are converging toward a single 'give an agent a goal, get grounded results' interface.
◆Prediction
Next moves likely extend the custom relevance model to more endpoints and broaden the Research Index past arXiv, with continued emphasis on published benchmark wins over rival search and scrape APIs.

◆Recent moves

  1. 17d ago

    Introducing our most accurate /search yet

    ⚡ SPARK

    The token-efficiency thesis applied to search: a custom relevance model scores every paragraph, list, and table and returns only the excerpts that answer the query. Same API shape, so every existing /search call gets the upgrade — and Firecrawl attaches a SOTA benchmark claim to it.

  2. 1mo ago

    Web-scale /monitor

    The /monitor endpoint grows from watching URLs you name to searching the entire web against a query and goal, alerting you or your agent when something new matches. An expansion of the change-detection line rather than a new direction, but it meaningfully widens what a single monitor can cover.

  3. 1mo ago

    v2.11.0: Research Index, keyless access, PII redaction

    A dense roll-up release that packages the standalone Research Index alongside keyless access to core endpoints, automatic PII redaction, a cache-per-schema deterministicJson format, and universal video discovery. Each item advances the same token-efficiency-and-grounding agenda; the Research Index is the headline, called out separately below.

  4. 1mo ago

    Firecrawl Research Index

    ⚡ SPARK

    Firecrawl builds a purpose-specific index for research agents — 3M+ arXiv papers plus the GitHub code behind them, refreshed daily — and claims category-leading arXivQA recall. It extends the platform from general web scraping into a curated vertical corpus agents can reason and cite against.

  5. 2mo ago

    Introducing /monitor

    ⚡ SPARK

    The original /monitor launch: describe in plain English what to watch, and Firecrawl configures the URLs, schema, and schedule and fires a signed webhook when a page changes — ingesting only the diff for up to 90% fewer tokens. This opened the change-detection line the later web-scale version expands.

  6. 2mo ago

    v2.10: /parse endpoint, Lockdown Mode, four new SDKs

    A roll-up release adding the Rust-based /parse endpoint, cache-only Lockdown Mode, the Question and Highlights formats, and four new official SDKs (Go, Ruby, PHP, .NET). It's the release where the token-efficient formats and the security posture both land at once — foundational, but a bundle rather than a single directional move.