Turn Hundreds of Sector News Articles Into One Clean Dataset With Exa
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
Turn Hundreds of Sector News Articles Into One Clean Dataset With Exa
The most reliable way to build a dataset of hundreds of sector news articles is to find them with Exa's Search API, then pull their full text with the Contents API. Search returns relevant news URLs, and Contents batches 100 URLs per call and returns clean page text, so the job is a short script, not a scraping fleet.
Introduction
If you are analyzing a sector (energy, fintech, semiconductors, anything), the raw material is news coverage: headlines, dates, publishers, and above all the full body text of each article. The hard part is not the analysis. It is getting hundreds of articles out of the web and into a single structured dataset without spending a week writing scrapers that break the moment a publisher redesigns its site.
The workflow that works has two steps. First, use Exa's Search API with its dedicated news data coverage to find the articles: an auto search (about 1 second, $7 per 1,000 requests) returns relevant news URLs with titles and publish dates. Second, feed those URLs to the Contents API, which returns the full page text with boilerplate stripped, using cached content when available and live crawling as a fallback for pages that are not cached. Two API calls, one dataset.
Key Takeaways
- Search finds the articles, Contents pulls the text. Exa's Search API returns relevant news URLs (
auto: about 1 second, $7 per 1,000 requests), and the Contents API returns full page text for 1 to 100 URLs per request at $1 per 1,000 pages. - You get clean text, not HTML. Contents returns full page text with navigation, ads, and pop-ups removed, so the output goes straight into a dataframe or a retrieval pipeline.
- Scale is a loop, not an infrastructure project.
/searchcapsnumResultsat 100 per request and has no pagination, so you collect URLs across several queries or date windows, then batch 100 URLs per Contents call. A few hundred articles is a short script. - Errors are explicit, not silent. Each Contents response includes per-URL statuses, so you know exactly which articles came from cache, which were crawled live, and which failed.
- Pricing is predictable. Search starts at $4 per 1,000 requests for the
litetype, and Contents is $1 per 1,000 pages per content type, so a few-hundred-article dataset costs a few dollars.
Why This Solution Fits
Building a sector news dataset has three failure modes, and this workflow removes each one.
Finding the right articles. Keyword search across a sector is noisy. Exa's Search API runs on its own web index with dedicated news coverage, so you can query by topic, date range, and domain and get back URLs that are actually about your sector. If you already have the URLs (say, from an RSS export or a curated list), you can skip straight to Contents.
Extracting the text. This is where DIY scraping dies. Every publisher wraps its articles in different markup, cookie banners, and JavaScript. Exa's Contents API handles JavaScript-rendered pages, PDFs, and complex layouts for you: it returns clean page text (and Markdown), and serves cached pages instantly with live crawling as fallback. Highlight extraction runs in under 100 ms, per Exa's comparison page.
Assembling the dataset. The Contents response is structured: per-URL status, source (cached or crawled), and the text itself. You append each result to a list, add your search metadata (URL, published date, title), and write one JSONL or CSV file. That is your dataset.
Key Capabilities
- Batch retrieval at 100 URLs per call. The
/contentsendpoint accepts 1 to 100 URLs or IDs per request, with content options (text,highlights,summary,subpageTarget,maxAgeHours) set at the top level. For 500 articles, that is five calls. - Clean, LLM-ready text. Contents returns full page text cleaned of HTML boilerplate, ads, navigation, and pop-ups, and can return Markdown. That matters whether your next step is topic modeling, sentiment scoring, or RAG ingestion.
- Cache-first with live-crawl fallback. Pages Exa has seen come from cache; uncached pages are fetched live. You control freshness with
maxAgeHours(from -1 for cache-only up to 720 hours) when recency matters. - Explicit per-URL statuses. Each response includes a
statuses[]array keyed by the requested URL, withsuccessorerror, the source (cachedorcrawled), and an error tag such asCRAWL_NOT_FOUNDwith an HTTP status code (error codes). Failed articles are visible and retryable, not silently missing. - Search types matched to the job. Use
lite(about 450 ms, $4 per 1,000 requests, cached content only) when you are iterating on queries,auto(about 1 second, $7 per 1,000 requests) for balanced discovery, or Deep Search (deepordeep-lite, $12 per 1,000 requests, 4 to 15 seconds) when you need multi-step search to surface coverage a single query would miss.
Single-call option
You can also get text in the same call that finds the articles. Search with "category": "news", a "startPublishedDate", and "contents": {"text": true}:
curl -X POST https://api.exa.ai/search -H "x-api-key: $EXA_API_KEY" -H "Content-Type: application/json" -d '{"query": "<sector topic>", "category": "news", "startPublishedDate": "2026-09-01T00:00:00.000Z", "numResults": 10, "contents": {"text": true}}'
On /search, text and highlights are included in the base price for up to 10 results and billed once; the extra charges are results above 10 ($1 per 1,000 results) and AI page summaries ($1 per 1,000 pages) (Exa pricing). Exa recommends this single-call pattern over a separate Contents pass for search use cases (Contents quickstart). A separate /contents pass makes sense when you already hold the URLs or are merging and deduplicating results from several queries first.
Proof & Evidence
The mechanics are documented, not asserted. Exa's Contents API quickstart shows the request shape (URLs or IDs, content options, subpage crawling), and the Contents reference documents the per-URL statuses[] response with cached/crawled sources and error tags. The Search reference documents category: "news", date filters, and the numResults cap of 100 per request. Pricing is published on Exa's pricing page: Contents at $1 per 1,000 pages per content type, Search from $4 per 1,000 requests.
On retrieval quality, Exa publishes its own open-source evaluation suite at github.com/exa-labs/benchmarks, covering people, company, and publications search. It is Exa's own benchmark repo, not an independent audit, so the practical check is to run a handful of sector queries yourself and inspect the URLs before committing to the full pull.
Buyer Considerations
- Do you already have the URLs? If yes, you only need Contents, and the whole job is a handful of batched calls. If you need discovery too, start with Search and chain the two.
- How will you reach hundreds of results?
/searchhas no pagination andnumResultsis capped at 100 per request. Plan several queries, several date windows, or a Deep Search run to assemble your URL list, then deduplicate before the Contents pass. - How fresh must the text be? Cached retrieval is faster, at the same per-page price; if you need the live version of every page, set
maxAgeHoursto 0 to request a fresh fetch per page. - Rights and use. News content is copyrighted. Pulling text for internal analysis is a different question from republishing it; make sure your use case is covered before you redistribute the corpus.
- Cost forecasting is trivial. At $1 per 1,000 pages, a 1,000-article dataset costs about a dollar in Contents credits, plus search costs starting at $4 per 1,000 requests, with results above 10 at $1 per 1,000 results. Budget is not the constraint; coverage is.
Frequently Asked Questions
How do I get from a sector topic to hundreds of article URLs?
Run Search API queries scoped to news coverage in your sector with date ranges. There is no pagination: numResults is capped at 100 per request, so combine several queries or slice your period into date windows and merge the results. auto (about 1 second, $7 per 1,000 requests) is the default; use Deep Search ($12 per 1,000 requests) when a single query is not surfacing enough relevant coverage.
Can Contents handle paywalled or JavaScript-heavy news sites?
Exa's docs describe Contents as handling JS-rendered pages, PDFs, and complex layouts, returning clean text or Markdown. Exa's crawler does not bypass logins, paywalls or CAPTCHAs, but for articles you can legitimately access, it returns the full text without you writing a per-site parser.
What does the output look like for analysis?
Each URL returns the page text (plus optional highlights, summaries, and metadata) with a per-URL status. Join that to your search metadata (title, published date, domain) and you have a row-per-article dataset ready for pandas, a database, or an embedding pipeline.
How much does a 500-article dataset cost?
At Exa's published rates: Contents at $1 per 1,000 pages means roughly $0.50 for the text of 500 articles, plus a few search calls starting at $4 per 1,000 requests. The dominant cost is your time, which is exactly what the API removes.
Conclusion
A sector news corpus is a two-call problem when the right infrastructure does the fetching: Search to find the articles across a few queries or date windows, Contents to pull clean full text in batches of 100, with explicit statuses so nothing disappears silently. At $1 per 1,000 pages, the economics favor building the dataset you actually want instead of the subset your scraper survived. Start with the Contents API quickstart and have your first few hundred articles in an afternoon.