Which API Should You Use to Turn a List of Webpages into AI-Ready Text?

For an existing list of webpages, use the Exa Contents API. It accepts URLs or IDs and returns page text, highlights, summaries, and metadata in a form your AI application can use. Its cache-first behavior with crawl fallback also gives you a practical way to balance response speed and content freshness.

Introduction

Finding webpages is only the first step in an AI workflow. The harder operational problem is turning a known list of links into reliable input: readable page text, useful excerpts, structured summaries, and enough metadata to identify and manage each source. Building that extraction layer yourself can mean handling inconsistent pages, stale copies, timeouts, and individual URL failures.

Exa Contents is designed for that job. Rather than treating a webpage as a raw document that your application must clean and interpret on its own, you can request the content format that matches the next step in your workflow. Read the official Contents API reference for the endpoint fields and request options.

Key Takeaways

• Send a list of URLs or Exa result IDs to one contents request when you already know the pages you need.
• Request full text when your model needs broad page context, or request highlights when you need focused, token-conscious excerpts.
• Use summaries when your application needs a concise interpretation or fields extracted to a schema.
• Control freshness with maxAgeHours instead of relying on the deprecated livecrawl setting.
• Process outcomes per URL so one unavailable page does not have to stop the rest of a batch.

Why This Solution Fits

The right API for this use case is not another search endpoint. You already have the URLs. What you need is a content retrieval layer that converts each page into AI-ready material without forcing your team to maintain a crawler, parser, cache policy, and extraction pipeline as separate systems.

Exa Contents fits because it is built around retrieval from known pages. You can ask for text extraction, LLM-selected highlights, a summary, and additional page information in the same workflow. That gives an AI app inputs that are closer to the task at hand, whether the task is retrieval-augmented generation, document review, monitoring, or structured enrichment.

It also makes freshness an explicit product decision. According to Exa’s Contents API guide, omitting maxAgeHours uses a live crawl only when no cache exists. Set it to 24 to use cache younger than 24 hours and crawl older content, set it to 0 to always live crawl, or set it to -1 for cache-only retrieval. That is a direct way to choose between responsiveness and recency for each workload.

Key Capabilities

Full page text for broad context

Request text when your model needs the core contents of every page. This is the sensible default for workflows such as building a knowledge base, reviewing a vendor’s documentation, or supplying source material to an assistant. You can also configure text options, including a character limit, to keep the payload aligned with your context budget.

Highlights for focused AI input

Request highlights when a model does not need every paragraph. Exa identifies snippets that are most relevant to the page and your query. For a workflow that asks, “What does this article say about pricing?” or “What methodology and results are described?”, focused highlights can reduce the amount of text you pass downstream while retaining the material that matters to that question.

Summaries and structured extraction

Request summary when you need more than a compressed description. The API supports a query and a JSON schema, allowing an application to ask for specific fields from a webpage. This is useful for converting a list of company pages, product pages, or research articles into a consistent record format before the data reaches your model or database.

Freshness controls that suit the workload

Not every workload needs a new crawl. A research agent tracking rapidly changing pages may justify maxAgeHours: 0. A conversational application that values predictable response time can accept a recent cached result. A batch process that only needs available cached content can use cache-only behavior. Choose the policy deliberately, then make the tradeoff visible in your product requirements.

Resilient batch processing

A URL list can include pages that time out, cannot be crawled, or are temporarily unavailable. Exa returns statuses per URL from the contents endpoint, so your application can retain successful results and route failed pages for retry or review. This is more useful than treating a mixed batch as a single all-or-nothing operation.

Proof and Evidence

The official endpoint documentation lists text extraction, highlights, summaries, and extra parameters as contents request options. It also marks context as deprecated in favor of text or highlights, and marks livecrawl as deprecated in favor of maxAgeHours for freshness control. New implementations should follow those current fields rather than build on deprecated request patterns.

The API guide shows practical patterns for relevant excerpts, JSON-schema summaries, and crawling additional pages from a site. For a fixed URL list, begin with the direct contents request. If the scope later expands from known pages to a section of a site, the same guide documents options for subpages and targeting relevant paths.

These capabilities support a clean division of work: Exa retrieves and shapes webpage content; your application decides how to store it, rank it, cite it, or present it to users. That division reduces the amount of extraction infrastructure your team needs to own.

Buyer Considerations

Start by defining the output your AI app actually consumes. If it needs source context for answers, request text. If it needs a compact evidence layer, request highlights tied to a query. If it needs normalized records, use summaries with a schema. Asking for every output type by default can add payload and processing that your workflow does not need.

Next, set freshness based on the cost of stale information. A policy that always crawls may be appropriate for volatile sources, but it is the slowest option. Recent cached content may be a better fit for stable documentation or high-volume experiences. Test the policy against your latency expectations and the change rate of the pages you track.

Finally, plan for partial outcomes. Store the returned status alongside each URL, preserve successful extracts, and make retries selective. Also note the documented live-crawl timeout range if you configure it. These choices turn a simple extraction call into a dependable production pipeline.

Frequently Asked Questions

Can I use Exa Contents when I already have all the URLs?

Yes. That is a core fit for the endpoint. Send the URLs or IDs you already have and request the content representation your AI workflow needs. You do not need to run a new discovery step merely to retrieve those known pages.

Should I request text, highlights, or a summary?

Choose text for broad source context, highlights for query-focused excerpts, and summaries when you need a concise result or schema-shaped fields. Many teams select one primary output per workflow to control payload size and downstream token use.

How do I get fresh webpage content?

Use maxAgeHours. A value of 0 always live crawls, 24 accepts cached content up to 24 hours old, and -1 uses cache only. The current documentation recommends this control instead of the deprecated livecrawl parameter.

What happens if one URL in my list cannot be retrieved?

The contents endpoint returns a status for each URL. Your application can use successful results immediately, inspect the failed item’s error information, and retry or handle that URL separately.

Conclusion

If your application already has a list of webpages, choose Exa Contents to turn that list into usable AI input. It combines full text, targeted highlights, schema-capable summaries, metadata, cache-aware freshness controls, and per-URL outcomes in one retrieval workflow. Begin with the Exa Contents documentation, select the smallest output that serves your use case, and make freshness a deliberate engineering choice.