Domain models¶
refindery.domain holds the pure domain types and algorithms — dataclasses,
enums, identifiers, URL normalization, ranking metrics, clustering helpers, and
retrieval functions with no I/O. For the SQL shape these map onto, see the
Data model.
Pure audio URL and MIME predicates for transcribable audio.
Used by the fetch router (audio URLs get the transcript fetcher instead of the plain HTTP fetcher), podcast discovery, download validation, and watch creation (a podcast watch takes a feed URL, not a direct audio file).
module-attribute
¶
AUDIO_EXTENSIONS = frozenset(
{
".mp3",
".m4a",
".m4b",
".aac",
".ogg",
".oga",
".opus",
".wav",
".flac",
}
)
module-attribute
¶
GENERIC_AUDIO_CONTENT_TYPES = frozenset(
{"application/octet-stream", "application/ogg"}
)
Accept normalized audio MIME types and generic podcast CDN types.
Pure URL canonicalization.
Rules (per spec): lowercase scheme + host; strip default port; strip www.;
strip fragment; strip tracking params (configurable glob patterns); sort the
remaining query params; strip trailing slash. Per-domain overrides can
restrict which query params survive (e.g. YouTube keeps only v).
module-attribute
¶
module-attribute
¶
DEFAULT_DOMAIN_RULES = {
"youtube.com": DomainRule(keep_params=frozenset({"v"}))
}
dataclass
¶
Per-domain override: only keep_params survive canonicalization.
dataclass
¶
CanonicalizationRules(
tracking_params=DEFAULT_TRACKING_PARAMS,
domain_rules=(lambda: dict(DEFAULT_DOMAIN_RULES))(),
)
Configurable canonicalization behavior.
class-attribute
instance-attribute
¶
domain_rules = field(
default_factory=lambda: dict(DEFAULT_DOMAIN_RULES)
)
dataclass
¶
Canonicalize url and derive its domain.
Raises ValueError for URLs without a scheme or host.
Clustering domain: dynamic parameters and the stable-ID matching layer.
HDBSCAN gives no stable ids across runs; match_clusters does, via
Jaccard cost + Hungarian assignment. Noise (label -1) is a legitimate
outcome and never becomes a cluster.
dataclass
¶
dataclass
¶
Corpus-size-scaled (min_cluster_size, min_samples).
Heuristic (config-overridable): sqrt(n)/2 clamped to [5, 25];
min_samples is half that, clamped to [3, 15]. Reproduces the spec's
static defaults (5, 3) around n=100. TODO: tune against a real corpus.
Assign stable ids to new labels and emit lineage.
Hungarian assignment over the Jaccard cost matrix; a match holds only at Jaccard >= 0.5. Unmatched new clusters are created (or splits when they overlap an old cluster > 0.3); unmatched old clusters are tombstoned as dissolved, or merged when most of their members landed in one new cluster.
Content hashing for revisit detection and v2 near-duplicate collapsing.
Class-based TF-IDF keywords for cluster labeling (pure, in-house).
The formula from BERTopic's c-TF-IDF, over one pseudo-document per cluster, without the BERTopic dependency. Keywords are always populated regardless of whether an LLM label lands later.
Top c-TF-IDF terms per cluster from one pseudo-document each.
Entity domain: taxonomy, surface-form normalization, blocking.
Canonicalization is corpus-internal (no Wikidata): normalization + blocking
here, matching/merging in the canonicalization service. inflect (pure
Python) is the one allowed non-stdlib dependency for singularization.
dataclass
¶
A canonical entity aggregated over the corpus.
Casefold, strip diacritics and punctuation, singularize the last word.
Levenshtein distance normalized to [0, 1] (0 = identical).
Small pure implementation; the canonicalization service uses rapidfuzz when available and falls back to this.
Domain error hierarchy.
Messages are built inside each exception class (keeps call sites clean and satisfies structured error handling at the API layer).
Bases: RefinderyError
The application cannot start with the supplied configuration.
Bases: RefinderyError
A model's token budget is below the canonical chunk hard max.
Bases: RefinderyError
None of a watch poll's discovered items could be ingested.
Bases: RefinderyError
A requested capability lands in a later milestone.
Bases: RefinderyError
Fetching the URL failed (network error or non-2xx status).
Bases: RefinderyError
An external provider's circuit breaker is open; the call was not attempted.
Bases: RefinderyError
The extractor for this content type needs an optional extra installed.
Typed identifiers and uuid7 generation.
uuid7 identifiers are time-ordered, which keeps SQLite b-tree inserts
append-mostly and makes ORDER BY id a creation-time ordering.
Idempotency-key builders, one per durable job kind.
These strings are persisted dedupe keys: jobs.idempotency_key is UNIQUE
and enqueueing a duplicate is a silent no-op, so the format for an existing
kind must never change — a new format would stop deduplicating against keys
already in the ledger. Golden tests in tests/unit/test_job_keys.py lock
every format and the one-prefix-per-kind invariant.
Builders are pure; impure inputs (clock readings, fresh uuids) stay at the call sites and are passed in.
Deferred-fetch key; keyed by page only — content is unknown until fetched.
Index key; a new content hash is new work, same hash dedupes.
Entity-extraction key over one resolved body.
Canonicalization key; one pass per cluster run.
Backfill key; re-requesting the same model later is distinct work.
Purge key; order-insensitive digest of the page-id set.
Eval-replay key; callers pass a fresh uuid so replays never dedupe.
Poll key; one scheduled run dedupes, while manual runs are namespaced.
Core domain entities and their status enums.
These are plain dataclasses, not pydantic models: rows come from our own migrations, so re-validating them per row buys nothing. Pydantic lives at trust boundaries (HTTP requests/responses, fetched content, settings).
module-attribute
¶
IngestOutcome = (
IngestQueued | IngestRevisit | IngestBlacklisted
)
dataclass
¶
Page(
id,
canonical_url,
original_url,
domain,
title,
body_text,
content_hash,
source,
metadata,
first_seen_at,
last_seen_at,
visit_count,
indexed_at,
status,
)
One canonical web page. One row per canonical_url; never versioned.
body_text and content_hash are None only while a
fetch_and_index job is still resolving the body.
dataclass
¶
A titled, contiguous span of a page's body text (e.g. a podcast chapter).
char_start/char_end index into the page body text; start_time_s
is the source timestamp (podcast chapter start), when the section derives
from timed media.
dataclass
¶
Chunk(
id,
page_id,
ordinal,
text,
token_count,
char_start,
char_end,
section_title=None,
section_start_s=None,
)
A canonical, model-independent span of a page's body text.
section_title/section_start_s label the chunk with the chapter it
belongs to when the page was chunked along section boundaries; both are
None for ordinary flat chunking.
dataclass
¶
A registered embedding model and its vector space.
dataclass
¶
Job(
id,
kind,
payload,
status,
idempotency_key,
created_at,
updated_at,
attempts=0,
max_attempts=5,
lease_until=None,
last_error=None,
)
Durable job ledger row; huey executes, this row is the source of truth.
dataclass
¶
dataclass
¶
Watch(
id,
kind,
url,
title,
enabled,
interval_hours,
config,
next_run_at,
last_run_at,
last_status,
last_error,
last_item_count,
created_at,
updated_at,
)
A pull source polled on its own schedule; discovered URLs are ingested.
next_run_at is advanced by the scheduling tick at enqueue time, never
by the poll handler, so a permanently failing poll cannot freeze the
schedule. config holds per-kind options (e.g. max_entries).
dataclass
¶
Cluster(
id,
label,
keywords,
size,
model_id,
created_at,
updated_at,
tombstoned_at=None,
centroid=None,
)
A stable-id cluster of pages.
dataclass
¶
ClusterRun(
id,
trigger,
algorithm,
params,
started_at,
finished_at=None,
duration_ms=None,
n_pages=None,
n_clusters=None,
n_noise=None,
)
One clustering run's record.
dataclass
¶
dataclass
¶
dataclass
¶
dataclass
¶
ModelBackfill(
model_id,
total_chunks,
total_tokens,
started_at,
updated_at,
cursor_page_id=None,
embedded_chunks=0,
finished_at=None,
last_error=None,
)
Resumable backfill state for one model.
dataclass
¶
dataclass
¶
Output of a content extractor: markdown body plus optional title.
sections carries structural boundaries (e.g. podcast chapters) discovered
during extraction; None means "no structure", and chunking falls back to
the flat sentence-based path.
dataclass
¶
A single entity mention found in page text.
dataclass
¶
Outcome: a new page was accepted and queued for indexing.
dataclass
¶
Ranking metrics: agreement stats for /compare, relevance metrics for eval.
Hand-implemented (small, pure, well-defined): no maintained PyPI RBO exists, and scipy's tau would be a heavy import for fifteen lines. RBO follows the extrapolated form of Webber, Moffat & Zobel (2010), eq. 32. Relevance metrics (nDCG, MRR, recall) use binary gains, matching the boolean feedback labels they are scored against.
Tau-a over the ordered intersection; None when it has < 2 items.
Restricted to the intersection, both rankings are tie-free permutations, so plain pair counting suffices (O(n^2), trivial at n <= 100).
Binary-gain nDCG@k; None when there are no relevant items.
DCG uses the 1/log2(rank + 1) discount; the ideal DCG places
min(len(relevant), k) relevant items at the top. absolute_ranks
preserves stored positions for a paginated ranking slice.
1/rank of the first relevant item; optionally use stored positions.
Relevant fraction in top k, optionally using stored absolute positions.
Pure retrieval primitives shared by all vector-store adapters.
Fusion is deliberately client-side in every adapter (Qdrant's server-side RRF
exposes no k parameter, and the query log needs both arms anyway), so this
one function defines hybrid fusion for the whole system. The conformance suite
asserts every adapter's fused output equals rrf_fuse of its arms.
dataclass
¶
Bases: StrEnum
How chunk scores pool into a page score.
No magic-number bonuses in v1: rerankers are calibrated, and arbitrary boosts would destroy their ranking. A bonus variant may be added later only if eval proves lift.
dataclass
¶
A candidate chunk carrying both fusion and (optional) rerank scores.
dataclass
¶
rollup_pages(*, chunks, strategy=MAX, top_m=3)
Roll chunk scores up to ranked pages.
sum_rrf orders pages by summed fusion scores (the reranker then only
affects chunk display order); the other strategies use effective scores.
Decay page scores by age: score * 0.5 ** (age_days / half_life).
Reciprocal-rank-fuse two ranked arms into one ranked list.
Score per chunk is sum(1 / (k + rank)) over the arms it appears in
(rank is 1-based). Ties break on chunk_id for determinism.
Pure YouTube URL logic: detection, classification, and canonical rewrites.
Used by canonicalization (folding youtu.be/shorts/live forms into the
watch?v= page), by the fetch router (only video URLs get the caption
fetcher), and by watch creation (only playlists/channels are watchable).
Match youtu.be, youtube.com, youtube-nocookie.com, and subdomains.
Extract the 11-char video id from watch/youtu.be/shorts/live URLs.
Report whether the URL identifies a single video (routing predicate).
Classify a YouTube URL as video, playlist, or channel; None otherwise.
Fetch-time form for yt-dlp flat extraction; never persisted.
watch?list= becomes the explicit playlist page; a bare channel URL
gets its /videos tab appended so extraction deterministically yields
the uploads playlist.
Fold youtu.be/
Called by canonicalize() with the www-stripped lowercase host. The
original query is discarded on rewrite (t=/si= would be dropped
by the youtube.com keep-params rule anyway); non-matching URLs pass
through unchanged — /watch paths in particular, so existing stored
canonical URLs are unaffected.