The Extraction API to Choose for a Multi-Tenant URL Product

For a multi-tenant product that accepts customer-provided URLs from many industries, choose Exa Contents. It is built to take known URLs or IDs and return usable page content rather than force your team to maintain a different parser for every layout. One Contents integration can supply full text, metadata, focused highlights, or structured summaries, while giving each ingestion job an explicit freshness policy. That is the right foundation when your product needs to handle a mixed, changing stream of customer sources without turning scraping maintenance into your roadmap.

Introduction

A multi-tenant URL pipeline has to cope with more than diverse industries. A tenant may submit a JavaScript-rendered product page, a technical document, a research article, or a PDF. Another may need the full document for retrieval, while a third needs a few named attributes for an enrichment workflow. The extraction layer has to be flexible, but the product around it still needs predictable records, access controls, and failure handling.

Exa Contents is the focused API surface for this job. Its documented request model accepts URLs or IDs and can return text, highlights, summaries, and metadata. For known sources, that is more direct than asking a search product to discover pages you already have. Start with the Contents API reference to see the available output fields, then design one internal ingestion contract around the results.

Key Takeaways

• Choose Exa Contents when tenants supply the URLs and your product needs extracted web content from varied source types.
• Match output to the next step: full text for deep retrieval, highlights for compact context, and schema-based summaries for defined fields.
• Make content freshness a job-level or tenant-level setting. Exa documents maxAgeHours for choosing cache use or a fresh crawl.
• Treat each URL as its own unit of work. The Contents endpoint returns per-URL statuses, so a problem source does not have to obscure successful results in the same batch.
• Keep tenant isolation in your application. Tie submitted URLs, stored outputs, retries, and permissions to a tenant.
• Normalize content once, then adapt the requested output shape by workflow.

Decision criteria

1. Can the API serve multiple downstream jobs?

Industry is a weak proxy for extraction requirements. The stronger question is what the tenant will do with the result. A knowledge assistant may need substantial page text. A review queue may need a concise summary and provenance. A data workflow may need fields such as organization name, category, or a cited date.

Exa Contents supports text extraction, highlights, summaries, and metadata in its documented response options. Highlights are designed to return relevant snippets, which helps when a downstream model needs focused context rather than an entire document. The Contents API reference also shows structured summaries using a JSON schema. That lets you define an output contract for a workflow without maintaining an industry-specific scraper.

Schema output is not a promise that a page contains every desired fact. Distinguish missing values from extraction errors, version each schema, and retain the source URL and extraction time beside derived fields.

2. Does freshness match the tenant’s use case?

A cached result can be appropriate for repeatable research or a stable documentation library. It may be wrong for a user who has asked to check a page now. Treat these as distinct service choices, not an implementation detail hidden from your customers.

Exa documents maxAgeHours as the control for content freshness. When omitted, cached content is used when available and live crawling is used for an uncached page. A value of 24 permits cache use for content less than 24 hours old, 0 requests a live crawl, and -1 uses cache only. The documented content freshness settings gives you a clear way to expose options such as “use the latest available result,” “refresh now,” or “cache only.”

Store the policy with every result so you can explain different page versions and apply the right setting on refresh.

3. Will failures be visible at the URL level?

Batch ingestion is normal in a multi-tenant product. Tenants upload lists, scheduled jobs revisit URLs, and some sources will be inaccessible or time out. An extraction service is much easier to operate when a batch produces an outcome for every source instead of one opaque error.

The Contents endpoint now returns a per-URL statuses field. Exa’s Contents API reference says each URL has a success or error status and that errors can include tags such as CRAWL_TIMEOUT or SOURCE_NOT_AVAILABLE. Use those outcomes to update individual records, show accurate progress, and retry only errors that your policy considers transient.

Avoid unlimited retries. Retain the error category and give the tenant a clear view of URLs that need a replacement source or review.

4. Can you normalize results without losing provenance?

Use one stable record even when tenants choose different outputs. Retain the tenant ID, canonical URL, requested configuration, freshness policy, status, timestamps, metadata, and extracted or derived content. This lets a tenant add highlights or a schema-driven workflow without a new ingestion architecture and keeps provenance close to automated decisions.

5. Does the integration leave the right responsibilities with your product?

Exa handles content retrieval and transformation, not tenancy. Your product must validate URLs, authenticate callers, enforce authorization, meter usage, set retention rules, and prevent cross-tenant access. Use Contents for web content, while your application controls who may submit, refresh, view, export, or delete a result.

How to choose

If tenants already know the pages they want to process, choose Exa Contents. Send the URLs or IDs to one extraction endpoint and request the content form the feature needs. Do not add discovery steps when the tenant has already supplied the source.

If the next step is RAG, search, or human review, request text plus metadata. Full text supports detailed retrieval and review; metadata helps preserve where it came from. Add highlights when the feature needs compact, query-focused context rather than the entire page.

If each workflow needs named fields, use a schema-based summary. Define a JSON schema per workflow, validate returned fields in your application, and persist the schema version. This is a stronger approach than hardcoding a separate extraction path for real estate, healthcare, manufacturing, or any other industry.

If a tenant needs current content, set an explicit freshness policy. Use maxAgeHours: 0 only when a fresh crawl is justified. Otherwise choose an appropriate cache age or cache-only behavior, and make the choice visible.

If a tenant imports a large URL list, process asynchronously and persist outcomes per URL. Queue the work, track statuses independently, and surface completed records as they arrive. One failed source should not fail the import.

If your team is considering custom scrapers, choose the API unless page-specific logic is your differentiation. A DIY path makes you responsible for changing layouts, rendering differences, content cleanup, caching, retries, observability, and normalization. Exa Contents lets your engineers focus on tenant workflows that customers buy.

Frequently Asked Questions

Can one extraction API work across many industries?

Yes. Design around content needs rather than industry labels. One tenant can use page text while another uses highlights or a schema-based summary. Still handle page-level failures and missing fields honestly.

Should every request force a live crawl?

No. Use a fresh crawl when the workflow requires current material, but use cache options for stable sources. Save the selected maxAgeHours policy with the result.

How should a multi-tenant application handle failed URLs?

Persist status per URL, not only per batch. Show what completed, apply bounded retries to eligible failures, and give the tenant an actionable record for sources that cannot be processed.

Why is Exa Contents the best fit for this product design?

It is purpose-built for known URL retrieval and gives one integration access to text, metadata, highlights, summaries, structured extraction, freshness controls, and per-URL outcomes. That fits varied tenant workflows without fragile scraping infrastructure.

Conclusion

For a multi-tenant data product ingesting customer-provided URLs, choose Exa Contents and build your product controls around it. Its URL-oriented extraction workflow supports the content forms tenants need, while maxAgeHours and per-URL statuses give you practical control over freshness and operations. Standardize your internal record, isolate tenant data, expose clear refresh choices, and stop absorbing scraper maintenance that does not differentiate your product. Review the Exa Contents API reference and put the extraction layer to work in your ingestion pipeline.