Sentry¶

How this repository initializes and reports to Sentry. There is no hardcoded DSN: each process module owns an optional sentry_dsn on its own settings class. Reporting is off for that process unless its module DSN is set. There is no shared SENTRY_DSN and no fallback between modules.

When Sentry is enabled¶

lib.sentry_setup.init_sentry initializes the SDK only when all of:

app_env is staging or main
A DSN is passed by the caller (from that module’s settings)
Sentry is not already active in this process

Otherwise it returns False and does nothing. Develop/test never init via this helper.

Per-module DSNs¶

Env var	Settings class	Process
`SENTRY_DSN_API`	`ApiEnvironment` (`api/api_environment.py`)	FastAPI
`SENTRY_DSN_SRSIM`	`SrsimEnvironment` (`srsim/srsim_environment.py`)	SRSim
`SENTRY_DSN_CELERY`	`CeleryEnvironment` (`tasks/celery_environment.py`)	General Celery worker
`SENTRY_DSN_LISTENER`	`MatchListenerEnvironment`	Listener Celery worker + CLI

Attribute name on each class is sentry_dsn (bound via validation_alias to the env var above). Sibling SENTRY_DSN_* keys are ignored when loading another module’s settings (extra="ignore").

See Environment configuration.

Shared init (`lib.sentry_setup`)¶

Piece	Role
`init_sentry(app_env=…, dsn=…, service=…, traces_sample_rate=0.0)`	Single entry for SDK init
`service`	`"api"` / `"srsim"` / `"celery"` / `"listener"` — selects integrations and sets tag `service`
`release` from `SENTRY_RELEASE`	When set (staging deploy git SHA), events attach to that release
`send_default_pii=True`	SDK attaches request IP (and similar defaults) on events
`profiles_sample_rate` from `SENTRY_PROFILES_SAMPLE_RATE`	Continuous profiling fraction (default `0.0`; read from env inside init)
`in_app_include`	`api`, `lib`, `listener`, `tasks`, `srsim` — first-party stack frames
`ignore_errors`	`KeyboardInterrupt`, `asyncio.CancelledError` (not `SystemExit`)
`before_send`	Redacts `Authorization`, `Cookie`, `X-Api-Key` headers on events/breadcrumbs
`LoggingIntegration` / `RedisIntegration` / `HttpxIntegration`	Always registered
`CeleryIntegration`	When `service` is `celery` or `listener`
`FastApiIntegration` (+ Starlette + Sqlalchemy)	When `service` is `api` or `srsim`
`traces_sampler`	Always installed; skips noisy HTTP paths; non-HTTP uses the rate

Pair send_default_pii with Sentry project Security & Privacy scrubbing (default scrubbers for Authorization / API keys; Scrub IP Addresses if IPs must not be stored). This flag does not replace explicit set_sentry_user(...) for authenticated app users.

Settings dumps attached as Sentry contexts (e.g. listener startup_config) must use model_dump_for_logs() so fields marked json_schema_extra={"sensitive": True} are obfuscated — see Environment configuration.

Code map: src/lib/sentry_setup.py.

Authenticated users¶

After a successful API-key resolve, set_sentry_user attaches the app User:

Field / tag	Value
`user.id`	`User.id` (UUID string)
`user.email` / `user.username`	Profile identity (`username` falls back to email)
tag `is_admin`	`"true"` / `"false"`
tag `route_surface`	`"api"` (Bearer), `"admin"` (session cookie), `"play"` (WebSocket)
tag `api_key_prefix`	Public key prefix only (never the plaintext secret)

Call sites: api.dependencies.auth.get_current_user, api.admin.auth.require_admin_session, api.routes.play_stream.play_stream. Unauthenticated routes leave Sentry user unset. Celery / listener processes do not set a human user.

Performance tracing (P90/P95 + DB spans)¶

Sentry Performance records a fraction of HTTP requests as transactions. The Performance UI shows duration percentiles (p50/p75/p90/p95/p99) per endpoint (route template, e.g. GET /api/matches/{match_id}). Filter by environment (staging / main) and release (deploy SHA).

Sampling¶

Setting	Process	Default	Notes
`SENTRY_TRACES_SAMPLE_RATE`	All modules (`BaseEnvironment.sentry_traces_sample_rate`)	`0.0`	Passed into every `init_sentry` call. Staging example uses `0.1`.
`SENTRY_PROFILES_SAMPLE_RATE`	All modules (`BaseEnvironment.sentry_profiles_sample_rate`)	`0.0`	Read from env inside `init_sentry`. Only useful when traces > 0. Staging example `0.01`.

Develop / test: Sentry init stays off (APP_ENV not staging/main), so the rate is irrelevant.
Staging: traces around 0.05–0.10 (.env.staging.example sets 0.1); profiles start at 0.01.
Main: start traces lower (0.01–0.05) until Performance quota is clear.
Percentiles are estimated from the sampled set, not from 100% of traffic.
Error events use the separate error sample_rate (SDK default = all errors).

Paths never sampled (HTTP)¶

init_sentry always installs a traces_sampler that returns 0 for:

/metrics, /health, /favicon.ico
anything under /admin/static/

Other HTTP paths use the configured rate. Non-HTTP sampling contexts (Celery) have no path and use the rate. Distributed parent_sampled is honored when present.

DB query spans¶

There is no separate DB sample rate. SQLAlchemy query spans are children of the sampled HTTP transaction: if the request is traced, its DB work appears in the waterfall; if not, those queries are not sent as Performance spans.

init_sentry(..., service="api"|"srsim") registers SqlalchemyIntegration so DB query spans attach under sampled HTTP transactions. This repo uses async create_async_engine / SQLModel AsyncSession (lib/db/utils.py).

Staging smoke check (after deploy with SENTRY_DSN_API + SENTRY_TRACES_SAMPLE_RATE>0):

Hit a few DB-backed API routes (e.g. GET /api/matches, GET /api/leagues).
In Sentry → Performance → open a transaction for that route.
Confirm child db spans for SQL executes (statement shape + duration).

Chatty endpoints inflate span count per transaction; prefer fixing N+1 over raising the sample rate. Postgres-side tools (pg_stat_statements, RDS Performance Insights) remain complementary for unsampled host-level query stats.

Celery workers¶

On worker_init (src/tasks/celery_app.py), each worker process selects a DSN from the entry module role (tasks.celery_app.SENTRY_WORKER_ROLE):

tasks.celery → "celery" → get_celery_environment().sentry_dsn (SENTRY_DSN_CELERY)
tasks.celery_listener → "listener" → get_listener_environment().sentry_dsn (SENTRY_DSN_LISTENER)

init_sentry(
    app_env=env.app_env,
    dsn=env.sentry_dsn,
    service="listener" if SENTRY_WORKER_ROLE == "listener" else "celery",
    traces_sample_rate=env.sentry_traces_sample_rate,
)

That is the one SDK init for that process when a DSN is set. Later listener code does not call sentry_sdk.init again if the client is already active; see Match listeners. Because the first init wins, the worker path must pass traces_sample_rate (it does).

Related: Celery (acks, queues, Beat).

API and SRSim¶

FastAPI lifespan on api.app and srsim.app loads the module environment and calls init_sentry(..., service="api"|"srsim", traces_sample_rate=env.sentry_traces_sample_rate) when the module DSN is set (SENTRY_TRACES_SAMPLE_RATE from BaseEnvironment).

Match listeners¶

Init vs per-run scope¶

When a listener starts (_execute_listener_run → listener.listen._setup_sentry):

Calls init_sentry(..., dsn=config.sentry_dsn, service="listener", traces_sample_rate=config.sentry_traces_sample_rate) — no-op if the listener Celery worker already initialized Sentry; still the init path for CLI / non-Celery runs (SENTRY_DSN_LISTENER).
If init succeeded (or was already active), sets for this run:
context startup_config — MatchListenerEnvironment.model_dump_for_logs() (sensitive values obfuscated)
tag sr_match_id — Sportradar match id (listener_sr_request_match_uuid) when set (not Sentry user)

So: one init per process; each listener start refreshes scope context/tags.

In-process captures (`listener.lib.sentry`)¶

During stream/question handling, MatchListener._capture_exception uses capture_listener_exception, which opens a new scope and attaches:

Field / context	Source
`startup_config`	Listener env dump with sensitive values obfuscated
`match_listener_id`	`MatchListenerRecord.id` when lifecycle was recorded
`celery_task_id`	Celery task id when under a worker
`match_id`	Internal match id (`listener_match_id`)
`sr_match_id`	Sportradar match id (`listener_sr_request_match_uuid`)
`match_display_title` / `match_display_label`	Cached from `Match` at start
context `match`	Same identity fields together
tag `listener_phase`	Call-site phase (`parse_event`, `write_event`, `finalize_event_file`, `question_logic`, `match_event`, `pending_resolution`)
context `listener_event`	Compact event only: `eid`, `gid`, `ts`, `corrective`, `sport_event_type`, `sport_event_summary` (never full `data`)
tags `event_eid` / `sport_event_type`	From compact event when present
context `listener_issuance`	`issuance_id` and optional definition id/name
tag `issuance_id`	When finalizing a deferred resolution (or otherwise known)
context `listener_extra`	Small scalars: truncated raw line snippet, `write_path` basename, `events_written`, `pending_count`, `generation`, `window_seconds`

Question-path phases (question_logic, pending_resolution, match_event) also set fingerprint ["listener-question-logic", "{internal_match_id}", "{phase}"] so definition/eval bugs group per match and phase instead of every unique message.

Code map: src/listener/lib/sentry.py, call sites in src/listener/listen.py.

Start-task failures (`start_match_listener`)¶

Permanent start failures raise (Celery FAILURE). Before raise, apply_start_match_listener_sentry_scope sets:

Fingerprint ["start-match-listener", "{internal_match_id}"] so all attempts for one match group as one issue
Tags/extras: match_id, attempt (1…15), exit_code when applicable, error type/message

CeleryIntegration reports the uncaught exception. Failed starts are capped at 15 attempts; see Match listener startup.

Do not rely on soft logger.error + Celery SUCCESS for these paths — that only produces LoggingIntegration Message events and used to mask real failures.

Ops testing¶

listener_force_exit_code forces SystemExit before stream work so staging/clone can exercise SUCCESS vs FAILURE mapping and Sentry without a live feed fault. Documented on the clone admin page and env field help.

Configuration¶

Setting	Where
`SENTRY_DSN_API`	`ApiEnvironment.sentry_dsn` (optional)
`SENTRY_DSN_SRSIM`	`SrsimEnvironment.sentry_dsn` (optional)
`SENTRY_DSN_CELERY`	`CeleryEnvironment.sentry_dsn` (optional)
`SENTRY_DSN_LISTENER`	`MatchListenerEnvironment.sentry_dsn` (optional)
`SENTRY_TRACES_SAMPLE_RATE`	`BaseEnvironment.sentry_traces_sample_rate` (default `0.0`; staging example `0.1`; all modules)
`SENTRY_PROFILES_SAMPLE_RATE`	`BaseEnvironment.sentry_profiles_sample_rate` (default `0.0`; staging example `0.01`; read by init from env)
`SENTRY_RELEASE`	Git SHA (or other release id); set by staging deploy into containers
`APP_ENV`	Must be `staging` or `main` for init

See Environment configuration.

Releases and deployments¶

Staging deploys notify Sentry so Releases and Deploys stay in sync with what is running.

Push to staging runs .github/workflows/deploy-staging.yml.
Compose starts with SENTRY_RELEASE=${{ github.sha }} so every Python service tags events with that release (init_sentry reads SENTRY_RELEASE).
After a successful deploy, getsentry/action-release@v3 creates the same release id, associates commits, and records a deploy to environment staging for each configured Sentry project.

One-time GitHub + Sentry setup¶

In Sentry: create an Organization Auth Token (Settings → Developer Settings → Auth Tokens) with at least project:releases and org:read. Optionally connect the GitHub integration for suspect commits.

In this repo → Settings → Secrets and variables → Actions:

Name	Type	Purpose
`SENTRY_AUTH_TOKEN`	Secret	Org auth token
`SENTRY_ORG`	Secret	Sentry organization slug
`SENTRY_PROJECT` or `SENTRY_PROJECTS`	Secret	Space-separated project slugs matching the DSNs (API / srsim / celery / listener)

These must be repository secrets (Settings → Secrets and variables → Actions → Secrets). A name under Secrets is not the same as an Actions Variable (vars.*).

Without these, the deploy still succeeds but the “Create Sentry release” step fails. The release id in CI and SENTRY_RELEASE in containers must match (both use the full git SHA today).

Code map¶

Path	Role
`src/lib/sentry_setup.py`	Shared `init_sentry`, `set_sentry_user`, `before_send`
`src/tasks/celery_app.py`	`worker_init` → role-based DSN + `service`
`src/tasks/celery.py` / `celery_listener.py`	Set `SENTRY_WORKER_ROLE`
`src/api/app.py` / `src/srsim/app.py`	Lifespan → module DSN + `service`
`src/api/dependencies/auth.py` / `api/admin/auth.py`	`set_sentry_user` after resolve
`src/api/routes/play_stream.py`	`set_sentry_user` after WS key resolve
`src/listener/listen.py`	`_setup_sentry`, `_capture_exception`
`src/listener/lib/sentry.py`	Listener capture + start-failure fingerprint helpers
`src/tasks/jobs/match_listeners.py`	Raises on permanent start failure; applies start fingerprint
`.github/workflows/deploy-staging.yml`	Staging deploy + Sentry release/deploy