Staging on Scarlett¶

Parlay Rewards staging runs as its own Docker Compose project on Scarlett, joined to the shared services-net network from the services stack (Caddy, Postgres, Redis).

Endpoints¶

Service	URL
API	https://pr-api-staging.jvsassoc.com
Live play WS	`wss://pr-api-staging.jvsassoc.com/api/play/stream` (same API upstream)
srsim	https://pr-srsim-staging.jvsassoc.com
Docs	https://pr-docs-staging.jvsassoc.com
Flower	https://pr-flower-staging.jvsassoc.com (Celery monitoring; basic auth)

Caddy (services repo) should expose a dedicated matcher for /api/play/stream* with WebSocket upgrade to the same pr-fastapi upstream as /api/*, so prod ALB can mirror path-based routing later. See Environment configuration — Live play realtime.

One-time host setup (Scarlett)¶

Shared stack (from the services repo): ensure network is named services-net, reload Caddy with the Parlay routes, create the DB. Read the Postgres user from the container env so you don't hit role "root" does not exist:

docker compose up -d caddy
# If the network still has a project-prefixed name, recreate so it is literally `services-net`.
docker exec -it services-postgres sh -c 'psql -U "$POSTGRES_USER" -d postgres -c "CREATE DATABASE parlay_staging;"'

Self-hosted GitHub Actions runner for this repo (Scarlett is a Mac):
Follow GitHub’s runner docs and pick the macOS / ARM64 runner (Apple Silicon).
The workflow uses runs-on: [self-hosted, macOS, ARM64], which matches the default macOS runner labels — no custom label needed. (If you add one, make sure runs-on still matches.)
Install as a service: ./svc.sh install && ./svc.sh start (no sudo needed on macOS for a per-user service).
Docker permissions on macOS: there is no docker group step. The runner runs as your user; just confirm docker ps works in that user's shell and set Docker Desktop to start at login.
Prefer a private repo; the workflow only triggers on push to staging (not pull_request).
Sentry release tracking (optional but recommended): add repo Actions secrets SENTRY_AUTH_TOKEN, SENTRY_ORG, and SENTRY_PROJECT or SENTRY_PROJECTS (space-separated Sentry project slugs for API / srsim / celery / listener). After each successful deploy the workflow creates a Sentry release (git SHA) and a staging deploy, and passes SENTRY_RELEASE into Compose so events attach to that release. See Sentry.
Sentry Performance (API latency): .env.staging.example sets SENTRY_TRACES_SAMPLE_RATE=0.1. Copy that into server .env.staging to sample ~10% of API requests for P90/P95 (and DB spans on those requests). See Sentry Performance.
Staging release notes → Slack (optional): a separate workflow (.github/workflows/staging-release-notes.yml) runs after Deploy staging completes. It does not affect deploy success/failure. Add:
- Actions secret CURSOR_API_KEY (Cursor user or service-account API key)
- Actions secret SLACK_BOT_TOKEN (bot with chat:write in the target channel)
- Actions variable or secret SLACK_CHANNEL_ID (e.g. C0123…)
- Optional Actions variable CURSOR_MODEL (default grok-4.6, not Fast) The runner user on Scarlett must be able to launch the Cursor local agent bridge used by cursor-sdk (uv run --with cursor-sdk …). Install/login Cursor tooling for that user if the notes job logs bridge/auth errors. See Slack release notes below.
Staging daily pytest → Slack (optional): .github/workflows/staging-daily-pytest.yml runs uv run pytest nightly at 2am EDT against staging. Failure/timeout posts a Slack root (Actions + Fix with AI) and a thread of failed tests. Reuses SLACK_BOT_TOKEN and SLACK_CHANNEL_ID. Cron is registered from the default-branch workflow file (develop). See Staging daily pytest below.
Slack agents (optional): shared Signing Secret verifies slash commands, Events API, and Interactivity. Add to staging .env.staging (API process):
- SLACK_SIGNING_SECRET (Slack app Signing Secret)
- SLACK_BOT_TOKEN (bot OAuth token — thread acks from the API)
- GITHUB_DISPATCH_TOKEN (PAT / fine-grained with actions:write)
- Optional GITHUB_REPOSITORY (default jordanvs/parlaysports-backend)
- For QD editor: SLACK_QD_EDITOR_USER_IDS, SLACK_QD_EDITOR_API_KEY (session state uses the normal REDIS_URL)
- For daily pytest Fix with AI: SLACK_PYTEST_FIX_USER_IDS (click allowlist; empty denies all). Does not use SLACK_QD_EDITOR_API_KEY. Reuse Actions secrets CURSOR_API_KEY + SLACK_BOT_TOKEN. Optional Actions variable CURSOR_MODEL (default grok-4.6, not Fast). Jobs source /opt/parlay-rewards/.env.staging for S3/DB credentials. See Slack agents, investigate, and QD editor below.

Deploy directory and filesystem data (run as the runner user):

sudo mkdir -p /opt/parlay-rewards
sudo chown -R "$(whoami)" /opt/parlay-rewards
mkdir -p "$HOME/parlay-rewards-data"/{listener,localstack,prometheus}
cp .env.staging.example /opt/parlay-rewards/.env.staging
# Edit .env.staging: set DATABASE_URL with shared Postgres user/password, API keys, etc.
# App default filesystem_container_data_dir=/app/data (no env needed).
# Compose hardcodes host $HOME/parlay-rewards-data → /app/data.
# Listener recordings: ~/parlay-rewards-data/listener/{mlb,nhl,...}/

If you previously used /opt/parlay-rewards-data, copy listener/ and localstack/ into $HOME/parlay-rewards-data once before the next deploy (Docker Desktop on Mac often cannot share /opt).

First deploy: push to staging, or from /opt/parlay-rewards after a manual sync:

docker compose -f docker-compose.staging.yml --env-file .env.staging up -d --build
make alembic_upgrade ENV=staging

Ongoing¶

Pushes to staging rsync into /opt/parlay-rewards (excluding .env.staging, .git, .venv, data/, and terraform/) and rebuild. Runtime data is not under the deploy tree:

Side	Path
Host (compose bind)	`$HOME/parlay-rewards-data`
Container / app	`/app/data` (`filesystem_container_data_dir` default)

Update /opt/parlay-rewards/.env.staging when secrets change — Actions will not overwrite it.
Keep $HOME/parlay-rewards-data/localstack if you need LocalStack S3 persistence across deploys.
Keep $HOME/parlay-rewards-data/listener for match event recordings from celery-listener-worker.
Manual migrations (do not run host uv run alembic — that misses .env.staging and container /app/data): from /opt/parlay-rewards, make alembic_upgrade ENV=staging.
After each deploy, the workflow abandons open listener rows immediately (sweep_stale_match_listeners(heartbeat_stale_seconds=0), exit 14) and runs dispatch_upcoming_match_listeners so in-progress matches restart without waiting for the normal ~3m heartbeat stale window. Do not use Admin Stop listener for deploy recovery (that turns auto-dispatch off).
After a successful deploy, the workflow also creates a Sentry release for the push SHA and records a deploy to environment staging (requires the Sentry Actions secrets/variable above). Containers receive the same SHA as SENTRY_RELEASE.
After Deploy staging completes, the separate Staging release notes workflow may post to Slack (see below). That job is best-effort and never gates the deploy run.
To copy those recordings into local develop (./data/listener/), run make listener_sync_staging_to_develop (rsync over SSH; does not delete local-only files).

Slack release notes¶

Workflow: .github/workflows/staging-release-notes.yml
Trigger: workflow_run of Deploy staging (completed), or manual workflow_dispatch.

Path	Slack shape
Deploy succeeded	Channel root: Block Kit; header `✅ Deployment Notification`; Environment (🟡 `staging` / 🔵 `main`) + Commit; module context; Compare. Sidebar color: amber `#ECB22E` (staging) vs blue `#1D9BD1` (main). Thread: one Block Kit reply per module (`api` / `listener` / `terraform` / `srsim` / `other`) via `.cursor/skills/release-notes/SKILL.md`.
Deploy failed	Channel root only; header `🔴 Deployment Notification`; Deploy run button. Sidebar: staging `#E01E5A`, main `#9B111E`. No agent.

Implementation notes:

Uses Slack Web API chat.postMessage (bot token) for thread_ts and attachment color bars. Root wraps blocks in attachments[{color, blocks}]; thread replies use top-level blocks. Fallback plain mrkdwn if module headings cannot be parsed.
Script: scripts/ci/post_release_notes.py with APP_ENV=staging (--soft-fail always in CI). Same script accepts APP_ENV=main for a future main-deploy notes workflow (blue #1D9BD1 success bar; same ✅ Deployment Notification header — env still in the Environment field).
Agent reads .cursor/skills/release-notes/SKILL.md with setting_sources=["project"].
Commit range is previous successful Deploy staging tip → this deploy’s SHA (via curl on the GitHub Actions API). If that cannot be resolved, the script falls back to the deploy commit’s parent (after~1), not a multi-commit window.
Job has continue-on-error: true. Missing secrets → log and exit 0.
Manual re-post: Actions → Staging release notes → Run workflow (optional after_sha / before_sha, or failure for a failure-style root).

Staging daily pytest¶

Workflow: .github/workflows/staging-daily-pytest.yml
Trigger: cron 0 6 * * * (06:00 UTC = 2am EDT / 1am EST), or manual workflow_dispatch.

Always checks out staging and runs the default suite (uv run pytest --junitxml=pytest-results.xml, -m 'not external') on Scarlett. Success is silent. Failure or the 90-minute job timeout posts a Slack root (header Staging daily pytest failed; staging + short SHA; Open Actions run + Fix with AI) and a thread reply listing failing tests and short reasons from junit (capped; remainder points at Actions).

Fix with AI is visible to everyone in the channel. Clicks are gated by SLACK_PYTEST_FIX_USER_IDS on the staging API. Allowlisted users dispatch .github/workflows/fix-daily-pytest.yml (ref=develop), which launches a Cursor Cloud Agent (autoCreatePR, startingRef: develop) that follows .cursor/skills/fix-daily-pytest/SKILL.md and posts the PR URL in the thread.

Implementation notes:

Reuses Actions secret SLACK_BOT_TOKEN and variable or secret SLACK_CHANNEL_ID (same as Slack release notes).
Notify is if: failure() with continue-on-error: true so a Slack miss does not hide the pytest failure. Missing token/channel → log and skip.
Script: scripts/ci/notify_daily_pytest_failure.py (uv run; no gh / jq on the runner PATH). Junit is also uploaded as artifact pytest-results.
Slack SHA is git rev-parse HEAD after checkout (staging tip), not github.sha (schedule events point at the default-branch commit).
Pytest step creates $GITHUB_WORKSPACE/data/listener and sets placeholder DATABASE_URL / REDIS_URL / keys / FILESYSTEM_CONTAINER_DATA_DIR. Import-time BaseEnvironment (via db_sessionmanager) requires those fields; a clean Actions checkout has no .env, and /app/data does not exist on the host. Testcontainers still replace DATABASE_URL and REDIS_URL (do not point tests at Scarlett's staging Redis). pytest_env supplies LISTENER_SR_URL_LEAGUE. Do not source .env.staging into the pytest process (staging DB/Redis). The job sets LISTENER_SR_URL_BASE to the staging srsim; requires_srsim tests skip when that host is unreachable.
GitHub registers cron from the default-branch workflow file (develop); the job then checks out staging. Merge the YAML to develop before expecting the nightly run.
Manual: Actions → Staging daily pytest → Run workflow (still tests staging).

Slack agents¶

Shared ingress on the staging API (signature-verified):

Endpoint	Purpose
`POST /api/webhooks/slack/commands`	Slash commands (routed by `command` name)
`POST /api/webhooks/slack/events`	Events API (`url_verification`, `app_mention`)
`POST /api/webhooks/slack/interactions`	Block Kit Apply / Cancel and daily-pytest Fix with AI

Mention traffic goes through a skill registry (src/lib/slack_agents/): new thread → classify intent (prefix, then keywords) → bind skill_id on a Redis session keyed by channel_id:thread_ts → skill dispatch. Thread replies stay on that skill. Adding a future mention skill = register an entry + workflow/handler; Events/Interactions routes stay shared.

Registered skills:

qd_editor (prefixes qd, question-def, …) — mention → propose → Apply/Cancel
pytest_fix — interaction-only (pytest_fix button on daily pytest failure alerts). Not a mention skill.

Issuance forensics remains slash-only until registered as a mention skill.

Slack app setup (shared)¶

Create (or reuse) a Slack app.
Slash commands Request URL: https://pr-api-staging.jvsassoc.com/api/webhooks/slack/commands
Event Subscriptions Request URL: https://pr-api-staging.jvsassoc.com/api/webhooks/slack/events Subscribe to bot event app_mention only (skill turns require an @mention; bare thread chatter must not re-propose).
Interactivity Request URL: https://pr-api-staging.jvsassoc.com/api/webhooks/slack/interactions
Bot scopes: chat:write, app_mentions:read. Invite the bot to target channels.
Copy Signing Secret → .env.staging SLACK_SIGNING_SECRET. Put the bot token in .env.staging SLACK_BOT_TOKEN and Actions secret SLACK_BOT_TOKEN. Redeploy / restart the API.

Slack investigate issuance¶

Workflow: .github/workflows/investigate-issuance.yml
Trigger: Slack slash command /investigate-staging → staging FastAPI POST /api/webhooks/slack/commands (command-name routed) → GitHub workflow_dispatch.

Step	Slack shape
Slash command ACK	Ephemeral “Queued…” to the invoking user
Job start	Channel root (Block Kit): header, Asked, context “investigating…”, View Actions run
Job done	Root updated via `chat.update` to “details in thread”; thread has Match + Timing field grids then story (What happened / Why / Bottom line / Recommendations / optional IDs). Cursor investigates via `investigate-listener-resolution`
Job failed	Root marked failed; thread warning Block Kit (+ Actions run button)

Usage¶

/investigate-staging <issuance-uuid>
/investigate-staging shortcode=abcd1234-efgh5678 why did "Will it be a hit?" resolve no for Player X

Default environment for the agent is staging unless the query says otherwise.

Implementation notes:

Script: scripts/ci/investigate_issuance.py.
Uses the same Cursor local-agent bridge as release notes (cursor-sdk on the self-hosted Scarlett runner). Default model is grok-4.6 (not Fast).
Credentials: job sources /opt/parlay-rewards/.env.staging (AWS keys, AWS_S3_MATCH_ARCHIVE_BUCKET, etc.) into the step env so the agent can aws s3 against the staging match-archive bucket. Unsets AWS_PROFILE so those keys win.
Staging DB: DATABASE_URL host postgres is Docker-network-only — the agent is instructed to query via docker compose … exec from /opt/parlay-rewards, not host-local Postgres.
Manual re-run: Actions → Investigate issuance → Run workflow (pass query + channel_id).

Slack QD editor¶

Workflow: .github/workflows/edit-question-definition.yml
Trigger: @bot qd <request> (or @bot … again in the same thread) → Events API → Redis session → workflow_dispatch propose turn → Slack Apply/Cancel → Interactions → staging PUT/POST/clone/archive on /api/question-definitions with SLACK_QD_EDITOR_API_KEY.

Bare thread replies without @mentioning the bot are ignored (so asking a colleague to approve does not start another propose).

Step	Behavior
Mention	Allowlist check (`SLACK_QD_EDITOR_USER_IDS`); staging-only gate
Propose	Cursor local agent (`grok-4.6`, not Fast) + `author-question-definitions` skill; propose only
Confirm	Block Kit Apply / Cancel (`qd_apply` / `qd_cancel`)
Apply	Full writable payload on update; archive only (no hard delete); snapshot reminder

Extra setup¶

Create a dedicated staging admin User + API key; put the plaintext key in .env.staging as SLACK_QD_EDITOR_API_KEY.
Set SLACK_QD_EDITOR_USER_IDS to comma-separated Slack user IDs.
Propose script uses the same REDIS_URL as the API (sourced from .env.staging). When the Redis hostname is Docker-only (redis, services-redis, or otherwise unresolvable on the runner), session get/set runs via docker compose … run fastapi on services-net.

Usage¶

@bot qd fix MLB "Will it be a hit?" No path for fielder's choice
@bot qd archive definition named "Old prop"

Thread replies after the first qd mention stay on the QD skill when you @bot again (no need to repeat qd). Messages that do not @mention the bot are ignored. Start a new thread to switch skills later.

Slack Fix with AI (daily pytest)¶

Workflow: .github/workflows/fix-daily-pytest.yml
Trigger: Fix with AI on the daily pytest failure root → Interactions → allowlist (SLACK_PYTEST_FIX_USER_IDS) → workflow_dispatch on develop → Scarlett bridge → Cursor Cloud Agents API (POST /v1/agents, autoCreatePR, base develop).

The Cloud Agent uses the same project skill .cursor/skills/fix-daily-pytest/SKILL.md with OPEN_PR=1. Invoking that skill in Cursor (IDE) stays local-only (no commit/PR).

Extra setup	Where
`SLACK_PYTEST_FIX_USER_IDS`	`/opt/parlay-rewards/.env.staging` (API). Redeploy/restart API.
`CURSOR_API_KEY`	Actions secret (same as investigate / QD).
Optional `CURSOR_CLOUD_ENV`	Actions variable: named Cloud Agents Environment if the default VM cannot `uv run pytest`.
GitHub + Cloud Agents	Cursor Dashboard → Integrations (GitHub) and Cloud Agents → Environments.
Workflow YAML	Merge to `develop` before the button can dispatch.

Does not use SLACK_QD_EDITOR_API_KEY. Interactivity Request URL is already the staging API (same as QD Apply/Cancel).

Adding another mention skill later¶

Register a SlackSkill in src/lib/slack_agents/ (prefixes, allowlist env, action_id_prefix, workflow, apply handler). Shared Events/Interactions URLs do not change. Button-only skills (like pytest_fix) omit prefixes and handle a start action in interactions.py.

Gzip listener archives (one-time)¶

New listener archives are uploaded gzip-compressed (Content-Encoding: gzip) under the same keys, and reads decode transparently, so no ongoing action is needed. To compress recordings that were archived before this change, run the one-time script inside staging (it reads AWS_S3_MATCH_ARCHIVE_BUCKET and credentials from .env.staging):

# Dry run — lists what would change, writes nothing:
docker compose -f docker-compose.staging.yml --env-file .env.staging \
  run --rm fastapi uv run python scripts/s3/gzip_listener_archives.py

# Apply — rewrites objects as gzip and purges old versions:
docker compose -f docker-compose.staging.yml --env-file .env.staging \
  run --rm fastapi uv run python scripts/s3/gzip_listener_archives.py --apply

Each object is rewritten as gzip and read back to confirm it decodes to the original bytes before the old versions are removed, so an interrupted run is safe to rerun.

Warning: --apply PERMANENTLY deletes the prior uncompressed object versions (this cannot be undone) and requires s3:ListBucketVersions and s3:DeleteObjectVersion on the match-archive bucket. Grant those to the IAM user whose keys are in .env.staging before running --apply. Without them the gzip rewrite still succeeds and verifies; only the old-version cleanup fails (reported per key, non-fatal), and those versions age out via the bucket's Glacier lifecycle instead.

Compose notes¶

No host port publishes; Caddy reverse-proxies by hostname.
Shared Postgres DB: parlay_staging. Shared Redis: DB 0 (app + celerylock task locks), DB 1 (Celery redbeat), DB 2 (Celery broker). Broker visibility_timeout is 12h so long-running match_listeners tasks are not redelivered mid-game.
LocalStack stays in this project (PERSISTENCE=1, bind mount under $HOME/parlay-rewards-data/localstack).
MkDocs is built into an image (not bind-mounted). Docker Desktop on Scarlett often cannot see /opt/parlay-rewards, which made mkdocs.yml look missing. The same /opt limitation is why filesystem data lives under $HOME.
Flower runs by default (alias pr-flower, public at pr-flower-staging.jvsassoc.com) with basic auth via --basic-auth; update the credentials in docker-compose.staging.yml.
Staging compose bind-mounts ${HOME}/parlay-rewards-data:/app/data. App defaults filesystem_container_data_dir to /app/data (omit from .env.staging).
Optional profiles: monitoring (Prometheus), listener.
celery-listener-worker consumes the dedicated match_listeners queue (long-running match listeners). Beat runs dispatch_upcoming_match_listeners every minute (APP_ENV != develop). Develop instead schedules test-clone dispatch only. See Match listener startup.