Which URL-to-Text Services Are Reliable Enough to Run Inside an Agent Loop?

For an agent loop, the reliable choice is a URL-to-text API that returns clean page text from cache in one call, falls back to live crawling when needed, and reports per-URL status instead of throwing. Exa's Contents API is built that way, at $1 per 1,000 pages per content type.

Introduction

An agent loop is only as reliable as its weakest tool call, and fetching web pages is often exactly that. A raw HTTP request returns HTML soup. A DIY scraper breaks when a site changes its markup. A headless browser adds seconds of latency and a whole class of timeouts. The result is the pattern every agent developer knows: wrap the fetch in retries, add a timeout, add a fallback, and still watch the loop stall on the one page that will not cooperate.

The better question is not "which fetch library has the best retry policy" but "which URL-to-text service can I call once and trust." Exa's Contents API is designed as that single call: you hand it a list of URLs you already know, and it returns full page text, summaries, highlights, and metadata, serving cached content when a stored version exists and live crawling as the fallback for pages that do not.

Key Takeaways

Reliability in an agent loop comes from design, not retries: cached-first delivery with live-crawl fallback keeps loop iterations short and predictable.
Exa Contents returns clean, LLM-ready text with HTML boilerplate, ads, and navigation removed, so the model's context window is not wasted on page chrome.
Per-URL status responses (success or error, with source: cached or crawled) let your agent degrade gracefully instead of crashing on one bad page.
At $1 per 1,000 pages per content type (Exa pricing), replacing a brittle scraping stack has a forecastable cost.

Why This Solution Fits

Agent loops fail on web fetches for three reasons: slow responses, dirty output, and all-or-nothing error handling. Exa Contents addresses each one directly.

Speed. Cached pages come back straight from Exa's cache, and Exa reports that the highlights extraction step runs in under 100 ms. For a full-text or live-crawl fetch, timing depends on page size and on livecrawlTimeout (default 10,000 ms), so benchmark on your own URLs; the structural win is that stored pages never pay the live-crawl cost at all. When your agent fetches a page on every iteration, that difference compounds: the loop does not spend its budget waiting on headless browsers or proxy rotations.

Clean output. The API returns text that is ready for a model, with HTML boilerplate, ads, and navigation removed. Exa's WebCode benchmark scores extraction against a reference that retains substantive prose, code blocks, API signatures and tables while removing navigation and other surrounding material; in Exa's evaluation across 250 URLs, Exa scored 96.7 on code recall and 91.9 on table recall. That matters for reliability too, because noisy input is what pushes context windows over the limit and forces your loop into truncation and re-fetching.

Graceful failure. The Contents reference specifies that each response includes a statuses[] array keyed by the requested URL or ID, with a status of success or error, the source (cached or crawled), and on failures an error.tag such as CRAWL_NOT_FOUND plus an HTTP status code. One unreachable page does not take down a batch of one hundred. Your agent reads the status and moves on, which is precisely the behavior a retry-wrapped scraper cannot give you.

Key Capabilities

Batch retrieval of known URLs: pass 1 to 100 URLs or document IDs in a single exa.getContents([...], {...}) call (or exa.get_contents in Python).
Cached-first with live-crawl fallback: stored versions are served from cache; uncached pages are crawled live, so you never maintain your own crawl infrastructure.
Multiple content modes per call: full text, highlights, summary, and metadata, so you can trim tokens without a second request.
Subpage crawling: follow up to N linked subpages (0 to 100) from each URL in the same request when a task needs more than one page; it may return fewer, and each returned URL is billed per content type.
Freshness control: maxAgeHours lets you decide how stale a cached version may be, trading latency for freshness per call.
Timeout controls: livecrawlTimeout (1 to 90,000 ms, default 10,000) sets how long a live crawl may run; Exa's guidance is to raise it to 12,000 to 15,000 ms for slow sites (Contents API quickstart). The CRAWL_LIVECRAWL_TIMEOUT and CRAWL_TIMEOUT error tags appear in statuses[] when a crawl times out (Error codes: content fetch status tags).
Rate limit: /contents allows 100 QPS (Billing and rate limits); over the limit the API returns 429, so honor Retry-After or back off exponentially.
JS-rendered pages and PDFs handled: the extraction layer deals with JavaScript rendering and complex layouts, including PDFs such as filings and reports.

A minimal batch call, using the exact request shape from the Contents reference:

curl -X POST https://api.exa.ai/contents -H "x-api-key: $EXA_API_KEY" -H "Content-Type: application/json" -d

curl -X POST https://api.exa.ai/contents -H "x-api-key: $EXA_API_KEY" -H "Content-Type: application/json" -d '{"urls": ["<url1>", "<url2>"], "text": true}'

The response carries results[] (title, url, publishedDate, author, text) and a statuses[] array. A failed URL looks like this:

{"id": "<url>", "status": "error", "source": "crawled", "error": {"tag": "CRAWL_TIMEOUT", "httpStatusCode":

{"id": "<url>", "status": "error", "source": "crawled", "error": {"tag": "CRAWL_TIMEOUT", "httpStatusCode": null}}

SDK forms: Python exa.get_contents(["<url>"], text=True); JavaScript await exa.getContents(["<url>"], { text: true }).

Then read statuses[] and retry only the entries whose error.tag is CRAWL_TIMEOUT or CRAWL_LIVECRAWL_TIMEOUT, with a raised livecrawlTimeout. Everything else in the batch is already in results[], so the loop retries one URL instead of the whole batch.

One scoping note from Exa's own guidance: use /contents when you already hold the URLs, and use /search with contents options when the agent needs to find the pages first. Keeping that boundary clean is part of keeping the loop predictable.

Proof & Evidence

The reliability claims above are documented, not aspirational. Exa's Contents API quickstart describes the one-call fetch, clean, and trim workflow, including text, highlights, and summary modes, JS-rendered page and PDF handling, subpage crawling, and maxAgeHours freshness control. The Contents reference specifies the per-URL statuses[] response contract that makes partial failure survivable, and the error codes page enumerates the documented status tags.

The pages agents fetch with a tool like this are often primary sources themselves: SEC filings pulled straight from EDGAR, for example, where a Contents call on a filing URL returns the document's text without a scraping pipeline. Exa's open-source retrieval benchmarks are published at github.com/exa-labs/benchmarks; they are Exa's own evals, not an independent check, but they are checkable. The $1 per 1,000 pages per content type price is a published Contents figure (Exa pricing).

Buyer Considerations

Do you already know the URLs? Contents is the right surface when your agent has a URL in hand (a link extracted from a prior step, a docs page, a filing). If the agent must discover pages first, start with Search and request contents there.
How fresh must the text be? Cached delivery is the speed win, but if a page changes constantly, set a low maxAgeHours and accept that a fresh fetch adds latency on cache misses.
How does your loop handle partial failure? Plan for the error.tag values in statuses[]: skip, log, and continue rather than retrying the whole batch.
What is your token budget? Prefer highlights or summary over full text when the agent only needs the passage that answers its current question; it is faster and cheaper in context.
Volume forecasting: at $1 per 1,000 pages per content type, cost scales linearly with fetch volume, which makes per-task budgets straightforward to model.

Frequently Asked Questions

What makes a URL-to-text service reliable enough for an agent loop?

Three things: fast responses (cached delivery on Exa Contents), clean output that does not blow up the context window, and structured per-URL error reporting so one bad page does not fail the batch. A service with those properties needs far less retry scaffolding than a raw fetch.

Does Exa Contents handle JavaScript-heavy pages and PDFs?

Yes. Exa's Contents API quickstart states that the API handles JS-rendered pages, PDFs, and complex layouts, returning clean Markdown-style text rather than raw HTML.

What happens when a page cannot be retrieved?

The response's statuses[] entry for that URL carries status: "error" with an error.tag (for example CRAWL_NOT_FOUND) and an HTTP status code, while other URLs in the same batch still return successfully.

How much does it cost?

Exa Contents is priced at $1 per 1,000 pages per content type on /contents (Exa pricing), so a loop making a few thousand single-content-type fetches per day costs a few dollars a day, not an infrastructure line item.

Conclusion

Constant retries and timeouts are a symptom of using a fetch tool that was never designed for agents. Exa's Contents API replaces that scaffolding with one dependable call: cached text with live-crawl fallback, clean model-ready output, and per-URL status reporting, at $1 per 1,000 pages per content type. If your agent loop is spending more time recovering from fetches than reasoning, it is time to try Exa Contents and let the retry code go.