Services¶
The application services orchestrate the use cases over the ports. They are
framework-free — no FastAPI, no SQL — so the same objects drive the HTTP routes,
the MCP tools, and the eval CLI. The composition root
constructs them in build_container.
The module inventory below is discovered from
refindery/application/services at build time. The conceptual guides explain
the main workflows: search,
ingest, models,
clustering, deletion,
and evaluation.
Model backfill: embed the existing corpus for a newly registered model.
Resumable via a page-granular cursor (a crash re-embeds at most one page; vector upserts are idempotent). Rate-limited by a Clock-driven token bucket. Cost estimates are exact — chunk token counts are already stored — and USD appears only when the user configured a price map (prices drift; never hardcode them).
Corpus-internal entity canonicalization (no external knowledge base).
Incremental (per page at ingest): exact normalized-alias match, then blocking on (type, first token) with edit-distance-first matching (free) and cosine fallback over surface-form embeddings.
Periodic (with each cluster run): within each block holding several entities, group by threshold-graph connected components (single-linkage — simpler than agglomerative average-linkage and dependency-free; flagged heuristic) and merge each group into its highest-mention member. Merges are snapshot-logged first and undoable (LIFO).
CanonicalizationService(
*,
store,
surface_embedder,
clock,
cosine_threshold=0.85,
edit_threshold=0.15,
)
Links mentions to canonical entities; merges near-duplicates.
Chapter-aware chunking: split within section boundaries, never across them.
Wraps any :class:Chunker so podcast chapters (or any titled section spans)
become hard chunk boundaries: each section is chunked independently, chunks
carry their section title/timestamp, and the chapter title is prepended into
the chunk text so it is searchable without a schema change. The underlying
Chunker port is deliberately untouched — this is pure orchestration.
Chunk text within each section span, preserving a global ordinal.
With no sections, delegates to the flat chunker.chunk. Otherwise every
section is chunked on its own slice (so no chunk crosses a boundary), char
offsets are shifted back onto text, and each chunk is labelled with its
section title/timestamp. When prepend_titles is set and the section has
a title, the title is prepended to the chunk text (so chunk.text is no
longer byte-equal to text[char_start:char_end] — by design; no consumer
relies on that equality).
Cluster run triggers: idle detection (cron and manual live elsewhere).
Idle rule (spec §8.3, run-duration-derived): re-cluster only when ingest has
been quiet for clamp(median(last 5 run durations) * 3, 5min, 60min) —
the system never re-clusters more often than clustering costs — AND at least
min_new_pages were indexed since the last run.
Cluster run orchestration: fit -> stable-id match -> persist -> label.
Always a full refit. Noise (-1) never becomes a cluster. Stable ids come from the Jaccard/Hungarian matching layer; tombstoned clusters remain resolvable by id so stale agent references degrade gracefully.
A/B model comparison: the delta isolates the embedder.
The full pipeline runs once per model. The sparse arm is fetched once
(chunk ids are canonical across model spaces) and the reranker instance is
identical across arms. Each arm is logged as a compare_arm query-log
row sharing a compare_id.
dataclass
¶
dataclass
¶
dataclass
¶
Entity extraction job: extract entities from an indexed page and link them.
Offline retrieval eval: score the query log, replay regression diffs.
Two modes share the golden-set substrate:
score_logreads logged runs read-only and computes nDCG@k / MRR / recall@k against feedback labels — no retrieval re-run, no container.replayre-runs golden queries live under two arm configurations (model and/or rerank toggled) viaCompareService.replay_armand diffs the aggregate metrics. Replays are never logged.
Reranker lift in score_log is exact only for rollup=max rows
without exact-match pins or recency decay: first-occurrence page order of
the fused candidate set is then precisely the rerank-off ranking. Rows
with other rollups are excluded from lift.
dataclass
¶
dataclass
¶
QueryScore(
query_id,
query_text,
model,
ndcg,
reciprocal_rank,
recall,
recall_candidates,
rerank_lift,
)
Metrics for one logged run.
dataclass
¶
Mean metrics over one model's scored runs.
dataclass
¶
dataclass
¶
dataclass
¶
dataclass
¶
dataclass
¶
Scores and replays the query log.
Assemble golden queries: dedupe by text, union positives.
Runs without feedback are dropped; so are queries whose labels are all negative (no positives means every metric is undefined).
Routes fetched content to the extractor registered for its content type.
Content-type -> ContentExtractor dispatch; extensible by registration.
Relevance feedback: append-only, joined to the query log at eval time.
Unknown query_ids are accepted by design — validating them would route reads through the single DuckDB writer; orphans are dropped by the eval-time join.
Forget: purge a URL or domain and blacklist it, atomically.
Metadata deletion is authoritative and immediate (purged pages can never surface in results — hydration drops them); vector deletion is asynchronous via tombstones with a verification sweep.
dataclass
¶
Purge + blacklist + vector tombstone lifecycle.
Retrieval surfaces hydrate INDEXED pages only; one filter enforces it.
Search, suggestions, similar, compare, and clustering all hydrate page ids that may point at queued/failed/purged pages. Metadata is authoritative: anything not currently INDEXED is dropped here, in one place.
Indexing pipeline.
Chunk -> embed (every indexable model) -> upsert -> page-vector rollup -> status transitions.
Chunk ids are deterministic — uuid5(page_id : content_hash : ordinal) —
so retries and re-index runs upsert the same vector-store points instead of
orphaning old ones.
Entity extraction is a separate durable job, so retrieval artifacts can be visible even when entity enrichment needs a retry.
IndexingService(
*,
store,
vector_store,
chunker,
registry,
clock,
fetcher,
router,
queue=None,
pooling=MEAN,
podcast_producer=None,
)
Executes index_page and fetch_and_index jobs.
async
¶
index_page job: run the pipeline for an already-resolved body.
async
¶
fetch_and_index job: resolve the body by fetching, then index.
async
¶
Dead-job callback: the page is excluded from search.
async
¶
Manual-retry callback: undo mark_page_dead; the page re-enters the queue.
async
¶
Enqueue entity jobs lost in the indexed->enqueue crash window.
Stable chunk id so retries upsert the same vector-store points.
Ingest service: the single entry point for new pages.
Flow: canonicalize -> blacklist check -> revisit detection -> body resolution -> insert + enqueue. POST returns immediately; when no body was supplied, a fetch_and_index job resolves it asynchronously.
Embedding model registry: registration rules and embedder lookup.
The invariant this service owns: no registered model may have a token budget below the canonical chunk hard max — accepting one would force a re-chunk, invalidating every other model's index and destroying A/B comparability.
module-attribute
¶
INDEXABLE_STATUSES = frozenset(
{ModelStatus.READY, ModelStatus.BACKFILLING}
)
Hybrid search: embed -> dense+sparse -> RRF -> rerank -> rollup -> pages.
Includes the exact-match pre-pass: a query that parses as a URL or bare domain pins the matching page(s) at rank 1 — the cheap, high-value "I'm pasting the URL back" refind case.
dataclass
¶
First-class filters, pushed into the vector store.
dataclass
¶
SearchQuery(
query,
k=10,
offset=0,
candidates=100,
rerank=True,
chunks_per_page=2,
rollup=MAX,
rollup_m=3,
rrf_k=60,
suggest=3,
mediation=VECTOR,
recency_half_life_days=None,
filters=None,
)
Validated search parameters (API defaults mirror the spec).
dataclass
¶
dataclass
¶
Watch service: scheduled pull sources that fan out into ingest.
Scheduling invariants:
tickadvancesnext_run_atat enqueue time, never the poll handler, so a permanently failing poll job cannot freeze a watch's schedule.- The poll job's idempotency key embeds the scheduled
next_run_at, so a racing duplicate tick dedups to a no-op while later ticks still enqueue. - Discovered URLs go through
IngestService.ingest, which dedups by canonical URL globally (across watches and manual adds).
dataclass
¶
Partial watch update with explicit clearing for nullable fields.
dataclass
¶
CRUD, scheduling tick, and poll handler for watches.
property
¶
Watch kinds with a wired source (extras may disable some).
async
¶
Create a watch due immediately; None when (kind, url) exists.
async
¶
Enqueue an immediate poll and advance the schedule.
Returns the poll job id (None only on a same-instant duplicate); raises WatchNotFoundError for an unknown watch.
async
¶
Discover a watch's items and fan them out into ingest.