How to Fetch the Live Text of a Competitor's Pricing or Product Page (Not a Stale Cached Copy)

Set maxAgeHours: 0 on Exa's Contents API and the endpoint fetches the page fresh at request time instead of serving a stored copy. One call returns the current, cleaned text of a URL you already hold, with JavaScript-rendered pages and PDFs handled for you, and per-URL status entries telling you exactly what came back.

Introduction

If you have ever pulled a competitor's pricing page and gotten numbers that no longer match what the site actually shows, you know the problem: many retrieval tools serve a cached copy whose age you cannot see or control. For competitive monitoring, that is worse than useless. A stale price or plan tier can flow straight into a comparison table, a sales battlecard, or an automated report, and nobody catches the error until a customer does.

The fix is not a bigger cache. It is explicit freshness control. Exa's Contents API is built for exactly this job: you pass the URLs you already know, and you decide how old the returned content is allowed to be. Setting maxAgeHours to 0 forces a fresh page fetch on every request, so what you get back is the page as it exists right now, cleaned of navigation, ads, and boilerplate, and ready to feed into an LLM or a dashboard. This article explains how the freshness controls work, why this approach fits competitive page monitoring, and what to check before you build on it.

Key Takeaways

• maxAgeHours: 0 on the Contents API means always fetch fresh; -1 means cache only; the maximum is 720 hours (30 days). You choose the tradeoff per request.
• Omitting the parameter uses the default behavior: a live crawl only when no cached version exists.
• The cache is what makes routine lookups fast (cached and fresh pages cost the same); the fresh-fetch option is there for the moments when only live data will do.
• One call handles fetch, clean, and trim: full text, query-targeted highlights, or a structured summary, with JavaScript-rendered pages and PDFs supported.
• Per-URL statuses[] entries report success or error with a tagged error code, so a failed fetch is a visible event rather than silently missing data.

Why This Solution Fits

Fetching a competitor's pricing page sounds trivial until you try to do it reliably. A DIY scraper breaks the moment the site changes its markup, needs proxy infrastructure to avoid blocks, and still hands you raw HTML you have to strip down yourself. A generic search API returns ranked links, not the current text of a specific URL. And a plain HTTP GET often returns a JavaScript shell with no pricing content in it at all.

The Contents API fits this job because it removes each of those failure modes in one place. You already hold the URL, which is exactly the case Exa's docs say /contents is for; when you need to discover pages first, the same extraction is available through /search. You get clean text or Markdown back, not HTML soup. Pages that render their pricing tables client-side are handled, as are PDFs like rate cards or plan documents. And most importantly for this use case, freshness is a parameter you control, not a black box: maxAgeHours: 0 always fetches fresh, a higher value lets the API serve a recent cached copy when that is good enough, and -1 restricts the response to cache only.

That per-request control matters because competitive monitoring has two distinct modes. For continuous watch jobs, cached content keeps latency down and you only need to know when something changed. For the moment you detect a change, or for a report going to a decision maker, you re-fetch with maxAgeHours: 0 and confirm the live number before you publish it.

Key Capabilities

Explicit freshness control. maxAgeHours accepts 0 (always fetch fresh), -1 (cache only), or any value up to 720 hours, per the Contents reference. A low value adds a fresh page fetch to the request. Note that the older livecrawl parameter is deprecated; freshness is managed through maxAgeHours, and you should not send both.

One call for fetch, clean, and trim. The same request can return full page text, query-targeted highlights, or a summary with a JSON schema for structured fields, all cleaned of navigation, ads, and boilerplate. Exa's Contents API quickstart describes the endpoint as handling JavaScript-rendered pages, PDFs, and complex layouts, which covers most modern pricing and product pages.

Batch requests. Pass 1 to 100 URLs or IDs in a single call, so a watchlist of competitor pages is one request, not one hundred.

Subpage crawling. The subpages parameter (with subpageTarget) pulls linked subpages, useful when a pricing page links out to per-plan detail pages or an FAQ you also want in the same pass.

Per-URL status reporting. Each entry in the response's statuses[] array is keyed by the requested URL or ID and reports status (success or error) with a tagged error code such as CRAWL_NOT_FOUND, CRAWL_TIMEOUT, or SOURCE_NOT_AVAILABLE (error code reference). Your pipeline can treat a failed fetch as a visible, tagged event instead of silently missing data.

For a developer, the whole workflow is one request. The Python SDK call below fetches a pricing page fresh, trimmed to a token-friendly length:

result = exa.get_contents(
  ["https://example.com"],
  text={"max_characters": 5000},
  max_age_hours=0
)

Proof & Evidence

The freshness semantics come straight from Exa's documentation: maxAgeHours is documented with 0 meaning always fetch fresh, -1 meaning cache only, and a maximum of 720 hours, with livecrawl marked deprecated (Contents reference). The response contract, including statuses[] entries keyed by URL or ID with tagged errors such as CRAWL_NOT_FOUND and SOURCE_NOT_AVAILABLE, is specified in the error code reference and the changelog. Contents costs $1 per 1,000 pages per content type (Exa pricing), so a daily fresh fetch of a ten-page watchlist costs about one cent per content type per day.

Buyer Considerations

• Match the freshness setting to the job. A common design is cached reads for routine checks and a fresh fetch when a change is detected or before anything is published.
• Handle the error cases. Pages move, paywalls appear, and bot defenses change. Build on the statuses[] contract so a failed fetch is a visible, tagged event rather than silently missing data.
• Gated pages return an error, not text. Login-gated or paywalled pages come back as an error entry in statuses[] (for example SOURCE_NOT_AVAILABLE) rather than text (Contents reference).
• Mind the rate limits. /contents allows 100 requests per second; over the limit the API returns 429, so honor Retry-After or back off (billing and rate limits).
• Pricing. Contents costs $1 per 1,000 pages per content type (Exa pricing), so a daily fresh fetch of a ten-page watchlist costs about one cent per content type per day.
• Confirm current limits in the docs. Parameter ranges and prices can change; check Exa's docs before you hardcode them.
• Know the boundary of the product. Contents is for pages whose URLs you hold. If your workflow starts from "find every competitor pricing page," use /search with contents options instead.

Frequently Asked Questions

How do I force a live fetch instead of a cached copy?

Set maxAgeHours: 0 on the request. Exa's docs define 0 as always fetch fresh, so the API retrieves the page at request time rather than serving a stored version. Use -1 when you want cache-only behavior and a value up to 720 when a recent copy is acceptable.

What if I omit maxAgeHours entirely?

The default behavior livecrawls only when no cached version exists for the URL. That is convenient for one-off lookups, but for competitive monitoring you should set the parameter explicitly so freshness is a deliberate choice rather than an accident of cache state.

What about pages that load their pricing with JavaScript or publish it as a PDF?

The Contents API handles JavaScript-rendered pages, PDFs, and complex layouts, returning cleaned text or Markdown. You do not need a headless browser or a separate rendering service in the pipeline.

How do I know if a fetch failed?

Check the statuses[] array in the response. Each requested URL or ID gets its own entry with a status of success or error, and errors carry a tag such as CRAWL_NOT_FOUND, CRAWL_TIMEOUT, or SOURCE_NOT_AVAILABLE so your code can react to each failure specifically.

Conclusion

Stale competitive data is a silent failure: nothing breaks, the report just quietly says the wrong thing. The remedy is to make freshness an explicit, verifiable part of the fetch. With Exa's Contents API, maxAgeHours: 0 gets you the live page, statuses[] tells you whether every URL in the batch actually came back, and the same call returns clean, LLM-ready text whether the page is HTML, JavaScript-rendered, or a PDF. Set freshness deliberately, handle the error tags, and your pricing and product intelligence will be as current as the pages themselves. Start with the Contents API quickstart and see Exa's pricing for the $1 per 1,000 pages per content type rate.

Related Articles

• How to Get Fresh Page Content Instead of a Cached Copy with the Exa Contents API