Match listener system context¶

Read this document before analyzing match-listener behavior, logs, or failures. It supplies system context shared by the task-specific documents under docs/agents/.

What the listener does¶

The match listener is a long-running async process that subscribes to one Sportradar match event stream. Its entry point is src/listener/listen.py.

For test clones, the stream URL usually points at srsim. By default (Wait for listener off) the clone injects session=<internal match id> and srsim runs a continuous in-process producer from Match.scheduled so cancel/replace rejoins the live tip. With Wait for listener on (Speed through), session= is omitted and srsim uses one-shot archive replay. The listener process itself is unchanged (subscribe → 302 → NDJSON). See simulator query parameters — continuous sessions.

For each stream record, the listener:

Parses it into a MatchListenerEvent.
Identifies heartbeats, synthetic markers, and repeated event IDs (seen-ID set in process memory). Synthetic events never run issue/resolve.
May append the event to the on-disk recording file.
Processes question resolution before attempting a new question issuance.
In production, persists question state and enqueues answer scoring.

The listener handles one configured match. The value inside square brackets in its logger name is the match shortcode ({sr_id[:8]}-{internal_match_id[:8]}, from listener_sr_request_match_uuid and listener_match_id / Match.id), not the full Sportradar or internal UUID. Under data/listener/{league}/ (live listener write path only):

Event recording: {sr_id[:8]}-{internal_match_id[:8]}.json (one continuum per match; restarts append after repair — no epoch rotation). Rows are enriched MatchListenerEvent dumps (gid / ts / eid / data / corrective). Live non-test runs may also append synthetic markers when the stream or Beat refresh moves Match.status to delayed / resumed / postponed (see) simulator query parameters). While Match.status is delayed, the listener stays up: periodic status checks log status=delayed and reset the consecutive-heartbeat counter so rain-delay silence does not reach the idle max (keep listener_max_consecutive_heartbeats above the status-check interval). That avoids tearing down and restarting mid-delay (which would double-append game_delay_start on a new process). srsim unwraps archive rows to Sportradar wire shape when streaming.
Text log: {sr_id[:8]}-{internal_match_id[:8]}.{listener_record_id[:8]}.log when a MatchListenerRecord exists (one file per run).

When the match is terminal and the events JSON is a closed array (trailing ]), Celery archives those files to S3 (AWS_S3_MATCH_ARCHIVE_BUCKET) as {shortcode}.json / {shortcode}.{id8}.log and sets Match.listener_events_archived (on listener exit enqueue, and via the 5-minute sweep_pending_listener_archives which enqueues per-match archive tasks for terminal matches scheduled within a lookback window (default 48 hours; override with lookback_hours when enqueueing). The sweep skips matches with no closed listener run that had event recording enabled (listener_write_events_to_file true or unset). Only staging/main archive — develop and test keep recordings on local disk. Admin download, clone, and srsim load from the S3 archive only (staging/main) — they do not fall back to the local filesystem.

When MATCH_EVENT_METRICS_ENABLED is on (default off), the listener also best-effort inserts slim type/time/period/situation/clock rows into match_event_metrics for Metabase (match timeline). Off: no batch list and no flush loop. Heartbeats and is_test matches are skipped. Catch-up (ingest_match_event_metrics) delete+replaces from the closed recording or S3 archive; listener_events_metrics_ingested is set only after catch-up. Rows older than 14 days are pruned. Correctives are stored with is_corrective and charted as corrective:{type}.

Important domain objects¶

QuestionDefinition¶

A reusable question template for a league. It defines:

when the question may be issued;
state extracted from the issuance event;
rendered question and outcome text;
answer choices and their correctness conditions;
resolution conditions;
an optional resolution correction window.

Definitions live in the PostgreSQL question_definitions table and are edited through /admin/question-definitions. Editing the database is the only way to change listener behavior for templates. For inventing and authoring definitions, see ../authoring_question_definitions.md.

QuestionIssuance¶

One concrete instance of a question for one match. It contains:

issuance_group: concurrency lane snapshot from the definition at issue time (default default);
issuance_min_interval: min-interval snapshot from the definition at issue time (default 300); after resolve, the next same-group wait is max(match floor, live group interval, this snapshot);
issuance_event: event that created the question;
store_state: values extracted at issuance and reused during resolution;
answer_choices: instantiated answer choices;
resolution_event: final event used to calculate correctness;
resolved_at: set after resolution.

Live match_state is not a column on the issuance. The listener merges the current per-match scoreboard into issue/resolve JSONPath context (read-only keys per league; see the authoring guide).

The issuance UUID is the main correlation key for following one question through logs. group= on issue/resolve log lines is the concurrency lane.

QuestionAnswer¶

A user's selected answer to an issuance. Question resolution first determines which answer-choice identifiers are correct. A separate answer-evaluation stage then scores individual user answers.

Event envelope and IDs¶

A normalized listener event has:

gid: Sportradar match id (same as Match.sr_id);
eid: event ID;
ts: stream timestamp;
data: source payload and metadata;
corrective: whether this eid was already observed by this listener process.

“Corrective” currently means a repeated, non-null eid. It is not inferred from payload differences or a dedicated Sportradar correction flag. The seen-ID set is in memory and resets when the listener restarts.

The same event may therefore appear first as:

Event received: pitch event_id=<eid>

(with a Sportradar description appended when present)

and later as:

Skipping corrective event: pitch event_id=<eid>

or, when a pending exists or an open issuance has resolution_start_on_corrective:

Processing corrective event: pitch event_id=<eid>

“Skipping” means the corrective does not run question logic. When processed, a corrective may update a pending same-eid resolution candidate, and—if the definition’s resolution_start_on_corrective is true—may also start defer/resolve when the corrected payload newly matches resolution conditions. For issuance, correctives normally do not create questions; exceptions are an already-deferred issue candidate (update/drop) or a definition with issuance_start_on_corrective (may start defer/create when triggers newly match).

Question pipeline¶

The listener runs these phases for each accepted non-corrective match event:

Resolve existing unresolved issuances (all open groups).
Commit resolved state and enqueue user-answer evaluation in production.
Check match-wide issuance limits (max count, period distribution).
Select an active definition from the match’s issuance pool whose issuance_group is free (no unresolved in that group; wait elapsed since the last resolution in that group using max(match.question_min_interval, live group.issuance_min_interval, last.issuance_min_interval); plus any live issuance_min_interval_after_groups waits measured from those other groups' last issuance) and issue at most one new question if its trigger matches. The pool is the effective allowlist: match override_question_definition_ids when set, else league default_question_definition_ids, else all active league definitions. If the next claimable sponsorship slot pins a question_definition_id, the pool is restricted to that definition.

Resolution runs before issuance. One event can therefore resolve an old question and issue a new one (including into a different concurrency group).

QuestionDefinition.issuance_group is the group's immutable slug (snapshot). The live QuestionGroup row (is_active, issuance_min_interval, issuance_min_interval_after_groups, period_distribution, max_per_match) also gates selection. QuestionDefinition.issuance_min_interval (snapshotted on the issuance) can only lengthen the next wait: max(match floor, live group interval, last snapshot). Match-wide max/period caps still apply to non-pivotal definitions. is_pivotal_moment skips those regular match gates (not the group period map). Different groups can be open at once; the listener still issues only one new question per event. Match questions do not use question groups.

Question definitions use JSONPath-like expressions against:

the current source event;
store_state captured when the question was issued;
live match_state (listener-published per-league keys; authors read only).

Resolution correction window¶

QuestionDefinition.resolution_correction_window_seconds controls whether a matching resolution event resolves immediately.

0: resolve immediately — unless the play is under review (see below).
greater than 0: retain the event as an in-memory pending candidate.
a same-eid corrective replaces the candidate payload.
after the window, correctness is calculated from the latest candidate.
if the corrected candidate no longer satisfies resolution conditions, the pending candidate is dropped and the issuance remains unresolved.
while a play is under review, the window does not expire (including 0). After the feed marks the play final, or after a short hold cap (MAX_PLAY_REVIEW_HOLD_SECONDS, 180s wall-clock or stream-ts per LISTENER_TIMING_SOURCE), the configured seconds run as call-on-the-field from the last non-review payload. Issuance is not held.

QuestionDefinition.resolution_start_on_corrective (default true for new definitions, pinned on the issuance snapshot) controls whether a corrective may open a pending candidate when none existed yet for that eid. Existing rows keep their stored value until operators save. Introduced after TOR @ WSH (2026-07-29), where a half-inning run question stayed pending while scoring detail arrived only on correctives, then resolved No at half end.

false: correctives only update an already-deferred same-eid pending.
true: if the corrective newly satisfies resolution conditions, the listener may defer/resolve from that corrective (log: Starting resolution from corrective for issuance … (resolution_start_on_corrective)).

Pending resolutions are process-local and do not survive listener restart.

Issuance correction window¶

QuestionDefinition.issuance_correction_window_seconds controls whether a selected definition creates an issuance immediately.

0: create (publish) immediately when selected.
greater than 0: retain an in-memory pending issue candidate (no DB row yet).
a same-eid corrective replaces the candidate payload; if triggers no longer match, the pending is dropped and nothing is published.
after the window, triggers and match gates are re-checked; only then is the issuance created. The pending holds issuance_group so another def in that group cannot issue meanwhile (at most one deferred candidate per listener).

QuestionDefinition.issuance_start_on_corrective (default false) controls whether a corrective may open a pending (or immediate) issue when none existed yet for that eid:

false: correctives only update/drop an already-deferred same-eid pending issue.
true: if the corrective newly satisfies issuance triggers, the listener may defer/create from that corrective (log: Starting issuance from corrective for definition … (issuance_start_on_corrective)).

QuestionDefinition.issuance_retract_on_corrective (default false) controls post-publish retract:

when true, a same-eid corrective that no longer matches issuance triggers closes the open issuance as resolution_reason=retracted (0 points; excluded from period/max caps; question.closed with reason=retracted).
log: Retracting issuance ….

Pending issuances are process-local and do not survive listener restart.

Timing source¶

LISTENER_TIMING_SOURCE controls both correction-window timing and question minimum-interval timing:

realtime (default): real elapsed seconds. Correction windows use async timers and can finish without another stream event. Issuance pacing compares real-time with prior resolved_at in the same issuance_group, waiting max(match floor, live group interval, last issuance snapshot).
ts: stream event-time deltas. Intended for simulator and compressed replay. Correction windows expire when a later event advances ts; issuance pacing compares stream timestamps against the prior resolution event ts in that group with the same max(...) wait.

The original event ts is retained in logs and persisted event payloads in both modes.

Post-issuance pause¶

LISTENER_POST_ISSUANCE_PAUSE_SECONDS (default 0) sleeps that many real-time seconds after a successful question issuance before the listener consumes the next stream line. Always real-time — independent of LISTENER_TIMING_SOURCE. Intended for fast simulator playback so humans can still answer; leave unset/0 for live.

Dry-run versus production¶

LISTENER_DRY_RUN=true evaluates issuance and resolution in memory:

definitions and match configuration are still read from the database;
no issuance, resolution, answer, or listener-lifecycle rows are written;
no answer-evaluation task is enqueued;
relevant question logs begin with DRY_RUN.

Dry-run logs are behavioral evidence, not evidence that database state changed.

Production behavior:

issuance and resolution are committed;
answer evaluation is dispatched after resolution;
listener startup and shutdown may be recorded in the database.

How listeners are started¶

In staging/main, Celery Beat schedules dispatch_upcoming_match_listeners every minute (live + test). On develop, Beat schedules dispatch_upcoming_test_match_listeners (is_test=True) instead. That task finds scheduled/created/delayed/inprogress matches with Match.auto_dispatch_listener (default True) on an enabled league that still need a listener (with schedule lookback/lookahead), then enqueues start_match_listener on the match_listeners queue. Redis celerylock: keys and open MatchListenerRecord rows prevent duplicate listeners. Settings merge call overrides, Match.listener_start_settings, then env files / defaults.

Hard stop (compose kill, crash) can leave Redis start locks until the open row is Abandoned (exit 14, ~3m stale heartbeat). Abandon clears those barriers so Beat / dispatch_upcoming_match_listeners can re-enqueue while auto_dispatch_listener stays on. Operator Stop (exit 15) clears barriers and sets auto_dispatch_listener=False so dispatch will not try again — use Start listener or re-enable dispatch.

Case catalog (badges, failures, skips, recovery): docs/guides/match_listener_startup.md#lifecycle-cases-status-failures-recovery.

Full flow and hierarchy: docs/guides/match_listener_startup.md.

Configuration and source locations¶

Listener orchestration: src/listener/listen.py
Listener start / settings merge: src/listener/lib/start.py
Listener settings: src/listener/listener_environment.py
Event parsing: src/listener/listener_event.py
Duplicate-ID tracking: src/listener/listener_status.py
Issuance stage: src/listener/lib/stages/issuance.py
Resolution stage: src/listener/lib/stages/issuance_resolution.py
Question log formatting: src/listener/lib/stages/question_logging.py
Timing policy: src/listener/lib/timing.py
Definition model: src/lib/db/models/question_definition.py
Issuance model: src/lib/db/models/question_issuance.py
Definition helpers: src/lib/question_definitions/

For deeper human-oriented documentation, see:

docs/guides/match_listener_startup.md
docs/guides/question_definitions.md
docs/guides/question_issuance_resolution_flow.md
docs/guides/question_definitions_league_event_state.md
docs/agents/authoring_question_definitions.md