Turn a List of URLs into LLM-Ready Text with One API Call

The Exa Contents API is built for exactly this job: you send it a list of URLs, and it returns clean, LLM-ready text or Markdown for each page, with HTML boilerplate, navigation, ads, and pop-ups stripped out. One request handles fetching, rendering, cleaning, and formatting, so you never maintain a scraper stack again.

Introduction

If you have a list of URLs and need clean markdown or plain text for each one, the naive path is a DIY pipeline: an HTTP client, a headless browser for JavaScript-heavy pages, a readability library for boilerplate removal, a Markdown converter, and a proxy layer for the sites that block plain requests. Every piece breaks on its own schedule, and you spend your time maintaining plumbing instead of building your product.

The Exa Contents API replaces that entire stack with a single call. You pass up to 100 URLs per request, and Exa returns full page text, highlights, summaries, and metadata for each one, using cached content when available and live crawling as the fallback for uncached pages. The output arrives already cleaned and structured for LLM consumption, including for JavaScript-rendered pages, PDFs, and complex layouts.

Key Takeaways

• The Exa Contents API takes a list of URLs (1 to 100 per request) and returns clean text or Markdown per page, with boilerplate, navigation, and ads removed.
• One call covers fetching, rendering, cleaning, and formatting, including JS-rendered pages and PDFs, so there is no scraper stack to maintain.
• You control output shape and cost: full text, query-targeted highlights, schema-driven summary, and subpages crawling from a single entry URL.
• Cached content with live-crawl fallback keeps latency and freshness in your hands via the maxAgeHours parameter.
• Contents is priced at $1 per 1,000 pages per content type on /contents, per Exa's pricing docs, a fraction of what a maintained scraping pipeline costs in engineering time.

Why This Solution Fits

Your problem statement is precise: you already know the URLs, and you need clean text for each. That is exactly the boundary Exa draws between its own endpoints. Exa's own guidance in the Contents API guide is to use /contents when you already hold the URLs, and to use /search with contents options when Exa needs to find the pages first. Since your list is in hand, /contents is the direct fit with no wasted search spend.

It also fits because the cleaning happens server-side, against parsers Exa maintains and updates. A readability library tuned for blog posts will mangle a docs site, a forum thread, or a PDF annual report. Exa's WebCode benchmark scores extraction against a reference that retains substantive prose, code blocks, API signatures and tables while removing navigation and other surrounding material; in Exa's evaluation across 250 URLs, Exa scored 96.7 on code recall and 91.9 on table recall. Contents also handles JavaScript-rendered pages and PDFs without you adding a headless browser. The result is text you can drop straight into a RAG pipeline, an agent context window, or a fine-tuning dataset.

Key Capabilities

• Batch URL fetching. Send urls (or document ids) in a single request, from 1 to 100 items. The response reports a per-URL statuses[] entry, so you know exactly which pages succeeded, which came from cache versus a live crawl, and which failed with a specific error tag.
• Clean text and Markdown output. Set text: true to get full page content with boilerplate removed, ready for an LLM context window.
• Token-efficient highlights. Pass a highlights query and get back only the passages that answer it, instead of whole pages. This cuts token costs when you only need the relevant slice of each document.
• Structured summaries. Use summary with a JSON schema to extract specific fields (names, dates, prices, attributes) from each page as structured objects rather than free text.
• Subpage crawling. Point at one entry URL with subpages (0 to 100) and an optional subpageTarget (for example ["docs", "api"]) to follow linked pages from a site in one request; it may return fewer than you request.
• Freshness control. The maxAgeHours parameter decides the cache-versus-livecrawl tradeoff: omit it to livecrawl only when no cache exists, set 24 to accept cache under a day old, set 0 to always livecrawl, or set -1 for cache-only speed (maximum supported value is 720 hours).
• Simple SDK calls. In Python it is exa.get_contents(urls, text=True), and in JavaScript it is exa.getContents(urls, {...}), with your API key from the Exa dashboard.

Example Request

curl -X POST https://api.exa.ai/contents -H "x-api-key: $EXA_API_KEY" -H "Content-Type: application/json" -d '{"urls": ["<url1>", "<url2>"], "text": true}'

A response excerpt (one result, plus one statuses[] entry for the URL that failed):

{
  "results": [
    {
      "title": "<page title>",
      "url": "<url1>",
      "publishedDate": "<YYYY-MM-DD>",
      "author": "<author, if available>",
      "text": "<clean extracted text>"
    }
  ],
  "statuses": [
    {"id": "<url2>", "status": "error", "source": "crawled", "error": {"tag": "CRAWL_TIMEOUT", "httpStatusCode": null}}
  ]
}

SDK forms: in Python, exa.get_contents(["<url>"], text=True). In JavaScript, await exa.getContents(["<url>"], { text: true }).

Proof & Evidence

The mechanics above are documented, not marketing claims. The Contents API reference specifies the request shape (urls or ids, 1 to 100), the content options (text, highlights, summary, subpages, extras, maxAgeHours), and the per-item statuses[] response with source: "cached" | "crawled" and structured error tags such as CRAWL_NOT_FOUND. The Contents quickstart walks through the common patterns, including highlights with a query, schema-driven summaries, and subpage crawling with freshness settings.

On cost, Exa's pricing page lists the Contents endpoint at $1 per 1,000 pages per content type. Compare that with the engineering hours a self-hosted fetch-and-clean pipeline consumes every month, and the economics are hard to argue with. Exa also publishes its own open-source retrieval benchmarks, including the WebCode extraction evals, on GitHub.

Buyer Considerations

• Batch size. Requests accept 1 to 100 URLs. For larger lists, chunk your URLs and issue parallel requests. For batch planning, /contents allows 100 QPS (rate limits).
• Freshness versus speed. Decide your maxAgeHours policy up front. Cache-only (-1) is the fastest option but may return stale content; always-livecrawl (0) is the freshest but slowest. For a full-text or live-crawl fetch, timing depends on page size and on livecrawlTimeout (default 10,000 ms), so benchmark on your own URLs. Most RAG pipelines are well served by a modest window such as 24 hours.
• Timeouts. livecrawlTimeout (1 to 90,000 ms, default 10,000) sets the timeout for a live fetch, and the CRAWL_LIVECRAWL_TIMEOUT and CRAWL_TIMEOUT error tags in statuses[] tell you when a fetch timed out (Contents reference).
• Output volume. Full text on every page can flood your context window. If you only need answers, prefer highlights with a query, or cap characters on text extraction.
• Error handling. Some pages will fail (removed, blocked, or never crawled). Design for the per-URL statuses[] response rather than assuming every item succeeds.

Frequently Asked Questions

What API turns a list of URLs into clean text for an LLM?

The Exa Contents API. Send your URLs to /contents (up to 100 per request) and it returns full page text with boilerplate, navigation, and ads removed, using cached content when available and live crawling as fallback.

Does it handle JavaScript-rendered pages and PDFs?

Yes. Exa's extraction handles JS-rendered pages, PDFs, and complex layouts, so you do not need a headless browser or separate PDF tooling in your pipeline.

Can I get Markdown instead of plain text?

Yes. Exa Contents returns clean Markdown suited for LLM ingestion, with HTML boilerplate, navigation, and ads removed.

How much does it cost?

Per Exa's pricing page, the Contents endpoint is $1 per 1,000 pages per content type. Exa's free tier gives $10 in credits every month (up to 2,500 Lite searches), plus a one-time $10 onboarding bonus, with no payment method required, so you can start testing from the Exa dashboard.

Conclusion

You do not need a scraping stack to feed an LLM. You need one endpoint that takes your URL list and hands back clean, structured text. The Exa Contents API does exactly that, with batch fetching, highlights, schema-driven summaries, subpage crawling, and explicit freshness control, at $1 per 1,000 pages per content type. Get an API key from the Exa dashboard, run your first batch today, and retire the scraper maintenance for good.

Related Articles

• Which Website Content APIs Return LLM-Ready Text Instead of Raw HTML?
• What’s the Most Cost-Effective API for Turning Known Webpages Into Clean Text?
• One API Call to Fetch JavaScript-Rendered Pages as Clean, Readable Text