Turn Search Results Into Clean RAG Page Content Without Maintaining a Scraper

You can turn search results into clean page content for a RAG index without maintaining your own scraper. Request contents options directly on Exa's /search endpoint, or call the Contents API with URLs you already hold. One request returns full page text, highlights, summaries, and metadata, with boilerplate, ads, and navigation already stripped.

Introduction

Every RAG pipeline hits the same wall: retrieval gives you URLs, but your model needs clean text. The usual answer is a homemade scraper, and that answer ages badly. Selectors rot when a site redesigns its markup, JavaScript-rendered pages come back empty, PDFs fail silently, and every new domain means another round of proxy plumbing, rate-limit handling, and boilerplate stripping. You end up maintaining crawling infrastructure instead of building your product.

Exa's Contents API removes that layer. You hand it a list of URLs or IDs, and it returns full page text, targeted highlights, LLM-generated summaries, and metadata for each page, using cached content when available and live crawling as a fallback for uncached pages. If you also need to find the pages in the first place, the same extraction can be requested through the Search API, so search and cleaning live in one call path instead of two systems.

Key Takeaways

• A scraper is not a feature of your product; it is a liability you maintain. Exa's Contents API replaces it with one call that returns clean, LLM-ready text.
• /contents takes 1 to 100 URLs or IDs per request and returns text, highlights, summaries, and metadata, with cached content served instantly and live crawling as fallback.
• If you need to find pages first, use /search with contents options; use /contents when you already hold the URLs, per Exa's own guidance in the Contents API quickstart.
• On /search, text and highlights are included in the base price for up to 10 results, and the extra charges are results above 10 ($1 per 1,000 results) and AI page summaries ($1 per 1,000 pages); on /contents, pricing is $1 per 1,000 URLs per requested content type (Exa pricing).
• Freshness is a parameter, not a project: maxAgeHours limits how old cached page content may be (from -1 to 720 hours); cached copies older than the limit are re-fetched, and 0 requests a fresh fetch.

Why This Solution Fits

The question is not "can I scrape?" It is "should scraping be my problem?" For a RAG index, the answer is almost always no, for three reasons.

First, RAG quality is a cleaning problem. Raw HTML carries navigation, cookie banners, ads, and footer boilerplate that dilute embeddings and waste context tokens. Exa's Contents API returns clean Markdown and handles JS-rendered pages, PDFs, and complex layouts, so the text you embed is the text a reader would see. Exa's WebCode benchmark scores extraction against a reference that retains substantive prose, code blocks, API signatures and tables while removing navigation and other surrounding material; in Exa's evaluation across 250 URLs, Exa scored 96.7 on code recall and 91.9 on table recall (WebCode).

Second, scrapers fail unpredictably. A single site redesign can silently empty your index. The Contents API returns per-URL statuses keyed by id, with status: "success" | "error", a source of cached or crawled, and structured error tags such as CRAWL_NOT_FOUND with an HTTP status code, so failures are explicit and observable instead of silent (see the error code documentation).

Third, the economics favor delegation. On /search, text and highlights are included in the base price for up to 10 results, billed once; the extra charges are results above 10 ($1 per 1,000 results) and AI page summaries ($1 per 1,000 pages). On /contents, pricing is $1 per 1,000 URLs per requested content type, with no charge for results above 10 (Exa's pricing docs). Compare that with the engineering hours spent on proxies, headless browsers, and selector maintenance, and the build-versus-buy question answers itself.

Key Capabilities

• One call for fetch, clean, and trim. Request text, highlights, or summary per page. Exa's highlights documentation reports that 500 characters of highlights match the retrieval accuracy of the first 8,000 characters of page text, a 16x reduction in indexed characters (Search highlights docs), which keeps your index token-efficient.
• Batch retrieval. Pass 1 to 100 URLs or document IDs per request, with content options at the top level: text, highlights, summary, subpages, subpageTarget, extras, and maxAgeHours (Contents reference).
• Subpage crawling. The subpages parameter follows up to N linked subpages (0 to 100) from each starting URL in the same request and may return fewer, useful for ingesting pages from a docs section or an investor-relations area without writing a crawler. Each returned URL is billed per content type.
• Freshness control. maxAgeHours sets how old cached page content may be, from -1 (cache only) to 720 hours, with 0 requesting a fresh fetch. Cached copies younger than the limit can still be served and older ones are re-fetched; when it is omitted, cached content is used where available and uncached pages are fetched live.
• Structured extraction. The summary option accepts a JSON schema, turning pages into structured objects rather than free text.
• Search plus contents in one step. When URLs are unknown, /search with contents options finds and extracts in a single call. Exa's docs recommend /search with contents options for web search tool use cases, with text and highlights included in the base price for up to 10 results per search (Contents API quickstart).

Example: Search With Contents in One Call

Ask /search for highlights on each result:

curl -X POST https://api.exa.ai/search -H "x-api-key: $EXA_API_KEY" -H "Content-Type: application/json" -d '{"query": "<topic query>", "type": "auto", "numResults": 10, "contents": {"highlights": true}}'

A response excerpt:

{
  "results": [
    {
      "title": "<page title>",
      "url": "<result url>",
      "publishedDate": "<YYYY-MM-DD>",
      "highlights": ["<relevant passage>"]
    }
  ]
}

Use "contents": {"text": true} to get search plus full page text in one call. When your pipeline already holds URLs from a prior search, a sitemap, or user input, call /contents instead with exa.get_contents([urls], ...) in Python or exa.getContents([urls], {...}) in JavaScript. REST bodies use camelCase; the Python SDK uses snake_case keyword arguments. See the Search reference for the full schema.

Proof & Evidence

The strongest evidence is the API contract itself. Exa's Contents reference documents batch requests of 1 to 100 URLs or IDs, top-level content options, and per-item statuses that distinguish cached from crawled results and surface explicit error tags. That is the shape of a system designed for pipelines, not for one-off scripts.

Exa's Contents API quickstart states the intended division of labor plainly: prefer /search with contents options for search-tool use cases, and use /contents when you already hold the URLs. For RAG ingestion, that means your retrieval step and your cleaning step can be the same request.

On cost, Exa's pricing documentation lists Exa Contents at $1 per 1,000 pages per content type on /contents, with text and highlights included in the base Search price for up to 10 results (pricing). And Exa's WebCode write-up frames why this matters at all: search matters "precisely where parametric memory fails," and stale or noisy results "can poison or even derail" long-running agents. Exa also publishes its open-source retrieval benchmarks on GitHub, including the WebCode evals behind the extraction claims above.

Buyer Considerations

• Know which endpoint you need. If your pipeline already has URLs (from a prior search, a sitemap, or user input), /contents is the right call. If you need discovery plus extraction, use /search with contents options.
• Plan for cache misses. Pages with no stored version are handled by live crawling, and failed fetches are reported in statuses[] with an error tag. Build your ingestion to log and retry those.
• Match freshness to cost. A low maxAgeHours adds a fresh page fetch for any page whose cached copy is older than the limit. For a nightly index rebuild, a generous window is cheaper; for a live agent, tighten it.
• Check current pricing and limits before committing. Prices and plan features change; verify against Exa's pricing docs at decision time rather than relying on any article, including this one.
• Compliance at a high level. If your deployment has data-handling requirements, review Exa's current documentation and terms directly rather than assuming controls; do not treat any third-party summary as a compliance certification.

Frequently Asked Questions

Do I still need my own scraper for JavaScript-heavy sites or PDFs?

No. Exa's Contents API handles JS-rendered pages, PDFs, and complex layouts, returning clean Markdown instead of raw HTML, per the Contents API quickstart.

How many pages can I fetch in one request?

Between 1 and 100 URLs or document IDs per /contents request, with content options such as text, highlights, summary, subpages, and maxAgeHours set at the top level (Contents reference).

What does it cost compared with running my own crawler?

On /search, text and highlights are included in the base price for up to 10 results; results above 10 cost $1 per 1,000 results, and summaries add $1 per 1,000 pages. On /contents, pricing is $1 per 1,000 URLs per requested content type (Exa's pricing docs). Your own crawler's cost is mostly engineering time and proxy infrastructure, which is exactly what this replaces.

How fresh is the content Exa returns?

Cached pages are served from Exa's index, and maxAgeHours limits how old cached page content may be, from -1 (cache only) to 720 hours, with 0 requesting a fresh fetch. When it is omitted, cached content is used where available. Pages with no cached version are fetched live, subject to livecrawlTimeout (default 10,000 ms), so timing on those depends on page size (Contents reference).

Conclusion

A RAG index is only as good as the text you put in it, and maintaining a scraper to produce that text is a tax on every other thing your team could be building. Exa's Contents API collapses fetch, clean, and trim into one call, gives you explicit per-URL statuses instead of silent failures, and prices extraction at $1 per 1,000 URLs per requested content type. Pair it with the Search API when you need discovery, wire maxAgeHours to your freshness requirements, and spend the reclaimed engineering time on retrieval quality and your actual product. The scraper you were about to write is the part of the pipeline you will never miss.

Related Articles

• Which Website Content APIs Return LLM-Ready Text Instead of Raw HTML?
• Turn a List of URLs into LLM-Ready Text with One API Call
• Read a Pricing Page, Get Only the Details You Asked For