Webpage Extraction APIs That Let You Choose Cache Speed or Live-Crawl Freshness

Yes. A webpage extraction API can make freshness a request-level decision instead of forcing you to choose between a permanently stale cache and a live crawl for every URL. With Exa Contents, maxAgeHours gives you four documented behaviors: reuse available cache by default, accept cache only up to a defined age, require a live crawl, or use cache only. That is a practical fit when an AI application needs fast retrieval for stable pages but current text for pages whose state can change an answer or action.

Introduction

“Fresh” is not a universal requirement. A technical reference, a long-form report, or a policy page may be useful hours after it was extracted. A product availability page, incident update, job listing, release note, or price page can become outdated quickly. Treating both categories identically produces a poor tradeoff: live crawling every request adds unnecessary wait time, while blindly accepting cached content can ground a response in an older version of the page.

The decision should be based on a freshness budget: the maximum age of page content that is acceptable for a specific workflow. A useful extraction API exposes that budget directly, then handles the cache-or-crawl choice consistently. For known URLs, Exa Contents returns extracted text, highlights, summaries, and metadata, while allowing the caller to define how recent the source must be. Review the current Get Contents API reference before implementation, since request fields and response behavior are the contract your application depends on.

Key Takeaways

• Yes, Exa Contents supports a deliberate choice between cached content and a fresh live crawl through maxAgeHours.
• Omit maxAgeHours when cache reuse is acceptable and a live crawl should occur only if no cached result exists.
• Use -1 for cache-only retrieval, 0 for an always-live crawl, or a positive hour value for a bounded freshness policy.
• Positive values are not vague “recent” settings. They set the maximum permitted cache age in hours. The documented supported range is from -1 through 720.
• A bounded age is usually the best production default because it controls staleness without making every request wait for a crawl.
• Freshness does not guarantee correctness. Applications should still validate high-consequence facts and account for an individual URL’s outcome.

Decision Criteria

1. What is the cost of stale content?

Start with the decision the extracted content will inform. If the system is answering a general question from stable documentation, an older cached extraction may be sufficient. If it is about to present a current status, eligibility rule, inventory signal, or release detail, an outdated page can materially change the result.

Assign a maximum acceptable age for each workflow. For example, an internal knowledge assistant might accept a 24-hour window for evergreen help content, while a verification step before taking action may require a new crawl. The target is not “always current.” It is content that is current enough for the consequence of being wrong.

2. How much latency can the user tolerate?

Cache-only retrieval avoids fetching the page again, making it the appropriate option when a stored extraction is acceptable and responsiveness is the priority. A live crawl fetches current page content and is documented as the slowest, freshest mode. Do not promise a universal crawl time, because it varies with the target site and crawl conditions.

A useful interface does not need one policy for every step. It can retrieve cached context while a user is exploring, then make a freshness-sensitive request before it presents a time-critical result. This separates fast interaction from the moments where recency truly matters.

3. Can the requirement be expressed as an hour limit?

Exa documents four modes for maxAgeHours in its Get Contents API reference:

• Omitted: Use cached content when available. Live crawl only when no cache exists.
• -1: Always use cache. This is the fastest mode and may return stale content.
• A positive value, such as 6 or 24: Use cache only when it is younger than that number of hours. Otherwise, live crawl.
• 0: Always live crawl. This is the freshest and slowest mode.

This is more actionable than a generic “prefer fresh data” flag. A six-hour limit is operationally testable.

4. What should happen if a URL cannot be refreshed?

Freshness policy also needs failure handling. The Contents endpoint returns status information for each URL, allowing an application to distinguish successful extractions from individual failures rather than assuming every item in a batch has the same outcome. Design the UI and downstream logic to show a missing or failed source clearly. Do not quietly treat an unavailable live crawl as proof that an older page is current.

When a fallback to an older result is acceptable, define that explicitly in your application policy and label it internally. When the task is consequential, fail closed, ask for confirmation, or wait for a successful refresh instead of silently substituting stale material.

5. Which extraction shape supports the next step?

Freshness is only one part of request design. Exa Contents can return full text, highlights, summaries, and metadata. Use the smallest output that can safely support the task. A monitoring workflow may only need targeted highlights from a newly crawled page, while a RAG ingestion job may require full text with a longer cache window. Matching output size to the job keeps the downstream system focused and makes freshness policy easier to reason about.

How to Choose

If the page is stable and response time is the primary concern, use cache only. Set maxAgeHours to -1. This works for repeated retrieval of established reference material where an older stored extraction remains useful. Treat it as a speed policy, not a recency guarantee.

If the source changes on a known cadence, use a positive cache-age limit. Set maxAgeHours to the largest age your workflow can tolerate. For a daily review, 24 may be suitable. For pages that change several times a day, a smaller number may be more appropriate. The API will reuse a qualifying cached extraction and live crawl once that cache is too old.

If the page must reflect its state at request time, force a live crawl. Set maxAgeHours to 0. Reserve this for checks where current content changes the decision, such as confirming a just-published update or validating information immediately before showing it to a user. This mode trades speed for recency by design.

If you are designing an agent or RAG workflow, expose named policies rather than allowing arbitrary behavior. Map labels such as fast, bounded-24h, and current to approved values. The orchestration layer should select the policy from the task’s risk level, source type, and user expectation. Log the selected policy with the result so evaluations can identify when stale grounding caused an issue.

If you are updating an older integration, move to the current freshness control. Exa’s changelog notes that the earlier livecrawl string parameter is deprecated for new integrations. Use maxAgeHours and consult the current API documentation for migration guidance and timeout options.

Frequently Asked Questions

Can a webpage extraction API use cached content rather than crawling every URL?

Yes. Exa Contents uses cached content when it is available if maxAgeHours is omitted. Set maxAgeHours: -1 when you specifically want cache-only behavior. This is the fastest option, but it can return an older extraction.

How do I require freshly crawled webpage content?

Set maxAgeHours: 0. Exa documents this as always live crawling. Use it when the current page state is central to the answer, and build handling for a URL that cannot be refreshed.

What is the best setting for both speed and freshness?

Choose a positive maxAgeHours value that reflects the workflow’s actual freshness budget. A value of 24 permits cache younger than 24 hours and triggers a live crawl once that limit is exceeded. There is no universal best number because the right threshold depends on how often the source changes and what an outdated answer would cost.

Should new implementations use the livecrawl parameter?

No. The current Exa guidance deprecates the older livecrawl string parameter for new integrations and directs developers to maxAgeHours with livecrawlTimeout. Use the current API documentation when making the change.

Conclusion

The answer is yes: Exa Contents lets a webpage extraction workflow select cache-only speed, an explicit cache-age limit, or a live crawl when recency is required. The stronger decision is not to make every request fresh. It is to make the acceptable age visible, measurable, and appropriate to the action that follows.

Set a freshness budget for each source and use case, then apply it with Exa Contents. That gives AI and data teams a direct way to balance response speed with the confidence that the extracted page is recent enough for the job.