Skip to content

Ingesting pages

There is one ingest endpoint, POST /v1/pages. Every new page — from a browser extension, a history importer, or a manual add — goes through it. The caller supplies the body text; Refindery does the rest asynchronously.

For the exact request/response contract (fields, status codes, examples), see the Upstream ingest API. This guide explains what happens to a page once it arrives.

The three body modes

The request carries the page body in one of three ways:

Mode Field What Refindery does
Pre-extracted (preferred) body_extracted Stores the plain text directly.
HTML body_html Extracts main-body markdown (requires the html extra).
URL only (neither) Fetches the URL and routes by content type (HTML/PDF).

body_extracted and body_html are mutually exclusive. URL-only ingestion enqueues a FETCH_AND_INDEX job instead of INDEX_PAGE.

The ingest → index → enrich flow

POST /v1/pages ──▶ canonicalize URL ──▶ blacklist check ──▶ upsert page ──▶ enqueue job
       │                                                                        │
    202 Accepted (returned immediately)                                         ▼
                                                              INDEX_PAGE / FETCH_AND_INDEX
                                                                   │
                                       extract (if needed) → chunk → embed → upsert vectors
                                                                   │
                                                          status ← indexed
                                                                   │
                                                          enqueue EXTRACT_ENTITIES
  1. Ingest (IngestService) — canonicalize the URL (tracking params stripped, per-domain keep rules), reject blacklisted patterns with 403, upsert the page (one row per canonical URL, never versioned), and enqueue a durable job. The API returns 202 immediately.
  2. Index (IndexingService, via the queue consumer) — extract content if needed, chunk it (sentence-aware, model independent), embed each chunk with the active model, and upsert chunk vectors plus a pooled page vector into the vector store. The page becomes indexed, and an EXTRACT_ENTITIES job is enqueued.
  3. Enrich (entities) — run the extractor chain over the page and canonicalize mentions into entities. This is an enrichment job: its failure is visible under features.entities but does not change the page's retrieval status.

YouTube transcripts

With the youtube extra installed (uv add 'refindery[youtube]'), a URL-only ingest of a YouTube video URL fetches the video's transcript via yt-dlp instead of the JS-heavy watch page: manual captions are preferred, then auto-generated captions, then — when the transcribe (or transcribe-mlx on Apple Silicon) extra is installed — the audio is downloaded and transcribed locally with Whisper. The page's title becomes the video title and its body the transcript. A video with no captions and no transcriber installed fails fetch (the page eventually goes dead); HTML is never indexed for videos. The audio path needs ffmpeg on the host. Configure with REFINDERY_FETCH__YOUTUBE_* (languages, auto-caption opt-out, Whisper model); to watch whole playlists/channels see Watches.

Audio transcription

With a local Whisper transcriber installed (the transcribe extra, or transcribe-mlx on Apple Silicon, plus ffmpeg), a URL-only ingest of an audio file URL (.mp3, .m4a, .ogg, .opus, .wav, .flac, …) is transcribed instead of fetched as a page: the audio streams to a temp file — SSRF-guarded, capped by REFINDERY_FETCH__AUDIO_MAX_BYTES (default 250 MB, far above the 10 MB in-memory page cap) — and the transcript becomes the page body. A response that is not audio (an HTML error page at an .mp3 URL) fails the fetch before any transcription. The Whisper model is shared with REFINDERY_FETCH__YOUTUBE_WHISPER_MODEL; disable the whole path with REFINDERY_FETCH__AUDIO_TRANSCRIPTS=false, after which audio URLs fetch as plain pages again. To follow podcasts see Watches. One dedup caveat: hosts that re-publish enclosures with changed cache-buster params (e.g. ?updated=) create a new page per variant — add such params to REFINDERY_CANONICALIZE__TRACKING_PARAMS to collapse them.

URL canonicalization

Two requests for the "same" page must collapse to one row. Canonicalization lowercases the scheme and host, strips the default port, www., the fragment, and tracking parameters (utm_*, fbclid, gclid, ref, si, …), sorts the remaining query params, and strips the trailing slash. Per-domain overrides apply where the query matters (e.g. YouTube keeps only v), and YouTube video URL forms (youtu.be/<id>, /shorts/<id>, /live/<id>) fold into youtube.com/watch?v=<id> so every form of one video dedups to one page. Configure overrides with REFINDERY_CANONICALIZE__*.

Revisit semantics

When a POST arrives for a canonical_url that already exists:

  • last_seen_at is set to now and visit_count is incremented;
  • the supplied body_extracted, body_html, or absent body is discarded, even if its content hash differs;
  • the response reports revisit: true plus a differing-hash flag for observability.

Two distinct URLs with identical content remain two pages in v1 — the clusterer co-locates them. (Content hashes are stored so that content-mediated collapse can be added in v2.)

Page lifecycle

queued ──▶ indexing ──▶ indexed
                   │
                   ├──▶ failed ──▶ dead   (after exhausted attempts)

Only indexed pages are used by search, similarity, and suggestions. Core indexing failures best-effort delete partial chunks and vectors before marking the page failed; dead jobs are queryable and manually re-enqueueable. See Operations for the job lease and recovery model, and the status reference.