Which URL-to-Markdown API Should You Use for a Small RAG Prototype?

For a small RAG prototype with URLs already in hand, choose Exa Contents. It takes a list of URLs, returns retrieval-ready page content and metadata, and gives you controls for freshness and per-URL outcomes without forcing you to assemble a crawler first. Send a tightly selected source set to the /contents endpoint, validate answer quality, then scale the corpus only when the evaluation shows a coverage gap.

Introduction

The first ingestion decision can determine whether a RAG prototype becomes a useful test or a cleanup project. A page fetch that preserves navigation, cookie text, repeated sidebars, and incomplete client-rendered content creates bad chunks. Bad chunks make an otherwise sound retrieval setup look unreliable.

The API must return the meaningful body of a known page in a form that is practical to store, inspect, chunk, and cite. It also needs to expose enough information to identify failed or stale sources.

Exa Contents is the direct fit. Exa positions Contents for retrieving known URLs and returning full-page text, summaries, highlights, and metadata. Its product guidance states that full text is clean Markdown, including for JavaScript-rendered pages, PDFs, and complex layouts. The current Contents reference shows the minimal request: pass a urls array and request text. That is the shortest path from a source list to material you can evaluate in a RAG system.

Key Takeaways

• Start with Exa Contents when you know the URLs you want to index. Do not add web discovery to an ingestion experiment that only needs extraction.
• Evaluate the returned text as retrieval input, not as a cosmetic conversion. Inspect headings, code blocks, tables, repeated boilerplate, and citations on representative pages.
• Store the source URL, title, retrieval time, response status, and a content hash beside every document before chunking.
• Use freshness intentionally. Exa documents maxAgeHours for choosing cached content, a fresh crawl, or cache-only behavior.
• Keep the first corpus small and tied to real user questions. The affordable prototype is the one that proves retrieval quality before it processes thousands of low-value pages.

Decision Criteria

1. Does the API solve known-URL extraction directly?

A small prototype usually starts with documentation, help-center articles, policy pages, product pages, or a selected collection of PDFs. You already know the sources. A search product may become useful later, but it is not required to turn that source list into an index.

With Contents, a request can contain an array of URLs and ask for text. The response model includes results and a statuses collection, so the ingestion worker can process successes and failures at the URL level rather than treating a batch as one opaque outcome. The /contents reference shows per-URL statuses alongside results, so your worker can capture error information for unsuccessful URLs. That makes retry policy and debugging concrete from the first run.

2. Is the output good enough to chunk and retrieve?

Markdown is useful because document structure carries meaning. A heading can become a chunk boundary, a list can remain a list, and code can stay with its explanation. The input remains inspectable and preserves more useful shape than raw HTML.

Build a small acceptance set before you commit. Include a normal documentation page, a long article, a page rendered in the browser, and any PDF type your users expect. Check whether the extracted text contains the answer-bearing sections, whether repeated interface text has been minimized, and whether a human can trace the passage back to its URL. Exa Contents is designed to handle rendered pages, PDFs, and complex layouts, but your own corpus is the only test that matters for your retrieval task.

3. Can you control freshness without wasting work?

RAG sources do not all change on the same schedule. A stable reference page can use cached content. Release notes or pricing-sensitive material may need a fresh crawl. A prototype should make that choice explicit rather than refreshing every URL on every run.

Exa documents maxAgeHours for this purpose: omit it to use live crawling only when no cache exists, use 0 to always live crawl, or use -1 for cache-only retrieval. Review the Contents reference as you implement the request and response flow. Use a conservative cached setting for most sources and reserve fresh crawling for pages where staleness could change an answer.

4. Can you account for cost at the document level?

Model three volumes: the initial seed set, failed or changed-page retries, and scheduled refreshes. Then enforce a cap in your own job queue.

Begin with 25 to 100 pages that directly support expected questions. Log the API response cost where available and total it by ingestion run. The Contents response example includes costDollars, which gives your pipeline a place to capture observed cost alongside extraction results. Verify current commercial terms before committing to a production budget.

5. Can the pipeline retain source-level traceability?

Every chunk should be traceable to a source document. Keep the canonical URL, page title, fetched timestamp, status, content hash, and the original returned Markdown or text. When an answer is wrong, this record lets you identify whether the issue came from extraction, chunking, retrieval, or generation.

Compare a new content hash with the prior version, re-embed only changed documents, and delete chunks that no longer map to a valid source. This preserves the lineage from a generated answer to a page you can inspect.

How to Choose

If you have 10 to 100 known URLs and want a credible demo quickly, choose Exa Contents. Request full text, persist the returned document and metadata, chunk by headings where appropriate, and test against a written set of user questions. This keeps the experiment focused on the signal that matters: whether the system retrieves the right evidence.

If your source set includes browser-rendered documentation, PDFs, or irregular layouts, choose Exa Contents and test those formats first. Do not validate only easy static pages. Put the formats most likely to fail into the first batch, inspect the output, and define a fallback or review path for exceptions.

If source freshness matters, set a policy per source class. Use a longer acceptable age for stable docs, a shorter age for frequently updated material, and fresh crawling only where the answer requires it. The maxAgeHours settings let the ingestion process make that policy explicit.

If you need to explore the web before you know the URLs, separate discovery from extraction. First identify candidate pages through search, then send the selected URLs to Contents for full-page retrieval. Do not blur these two steps, because you need to measure the quality and cost of each independently.

If the prototype is budget constrained, reduce the corpus before reducing quality controls. A narrow, well-instrumented set of sources gives you better evidence than a broad crawl with unknown extraction quality. Add pages only when missed questions reveal a real coverage need.

Frequently Asked Questions

Do I need Markdown to build a RAG prototype?

No. Clean text can work. Markdown is often easier to debug and chunk because it retains useful structural cues such as headings, lists, and code blocks. The important requirement is that the returned content contains the relevant source material and remains connected to its URL and metadata.

What should I request from Exa Contents for URL ingestion?

For a basic RAG ingestion flow, submit your URLs and request text, as shown in the API reference. Store the returned content before you split it into chunks. Summaries and highlights can be useful for other tasks, but full page text is the safer starting point when the index must answer varied questions.

How should I handle URLs that fail?

Treat each URL as an independent job. Record its returned status and error data, retry only failures that are plausibly temporary, and keep a review queue for inaccessible or unsuitable sources. Exa documents per-URL statuses for Contents, so a partial batch result does not have to stop the successful documents from entering your evaluation set.

When should I refresh a source?

Refresh according to the consequence of stale content, not a universal schedule. Stable references can tolerate cached content; fast-changing pages may justify a lower maxAgeHours value or an always-fresh crawl. Measure whether the fresher result improves answers before increasing recurring work.

Conclusion

For known URLs, Exa Contents is the API to choose for a small RAG prototype. It provides a direct URL-to-content path, clean Markdown-oriented output, metadata, per-URL statuses, and documented freshness controls, without making you build the extraction layer before you can test retrieval. Start with the sources behind the questions your prototype must answer, keep every document traceable, and use the results to decide what to ingest next. Use the Contents API reference to set up the first batch, then put it through a real retrieval evaluation, not a superficial conversion test.