Staging on Scarlett¶
Parlay Rewards staging runs as its own Docker Compose project on Scarlett, joined to the shared services-net network from the services stack (Caddy, Postgres, Redis).
Endpoints¶
| Service | URL |
|---|---|
| API | https://pr-api-staging.jvsassoc.com |
| Live play WS | wss://pr-api-staging.jvsassoc.com/api/play/stream (same API upstream) |
| srsim | https://pr-srsim-staging.jvsassoc.com |
| Docs | https://pr-docs-staging.jvsassoc.com |
| Flower | https://pr-flower-staging.jvsassoc.com (Celery monitoring; basic auth) |
Caddy (services repo) should expose a dedicated matcher for /api/play/stream*
with WebSocket upgrade to the same pr-fastapi upstream as /api/*, so prod
ALB can mirror path-based routing later. See
Environment configuration — Live play realtime.
One-time host setup (Scarlett)¶
-
Shared stack (from the
servicesrepo): ensure network is namedservices-net, reload Caddy with the Parlay routes, create the DB. Read the Postgres user from the container env so you don't hitrole "root" does not exist: -
Self-hosted GitHub Actions runner for this repo (Scarlett is a Mac):
- Follow GitHub’s runner docs and pick the macOS / ARM64 runner (Apple Silicon).
- The workflow uses
runs-on: [self-hosted, macOS, ARM64], which matches the default macOS runner labels — no custom label needed. (If you add one, make sureruns-onstill matches.) - Install as a service:
./svc.sh install && ./svc.sh start(nosudoneeded on macOS for a per-user service). - Docker permissions on macOS: there is no
dockergroup step. The runner runs as your user; just confirmdocker psworks in that user's shell and set Docker Desktop to start at login. - Prefer a private repo; the workflow only triggers on push to
staging(notpull_request). - Sentry release tracking (optional but recommended): add repo Actions secrets
SENTRY_AUTH_TOKEN,SENTRY_ORG, andSENTRY_PROJECTorSENTRY_PROJECTS(space-separated Sentry project slugs for API / srsim / celery / listener). After each successful deploy the workflow creates a Sentry release (git SHA) and astagingdeploy, and passesSENTRY_RELEASEinto Compose so events attach to that release. See Sentry. - Sentry Performance (API latency):
.env.staging.examplesetsSENTRY_TRACES_SAMPLE_RATE=0.1. Copy that into server.env.stagingto sample ~10% of API requests for P90/P95 (and DB spans on those requests). See Sentry Performance. - Staging release notes → Slack (optional): a separate workflow
(
.github/workflows/staging-release-notes.yml) runs after Deploy staging completes. It does not affect deploy success/failure. Add:- Actions secret
CURSOR_API_KEY(Cursor user or service-account API key) - Actions secret
SLACK_BOT_TOKEN(bot withchat:writein the target channel) - Actions variable or secret
SLACK_CHANNEL_ID(e.g.C0123…) - Optional Actions variable
CURSOR_MODEL(defaultgrok-4.6, not Fast) The runner user on Scarlett must be able to launch the Cursor local agent bridge used bycursor-sdk(uv run --with cursor-sdk …). Install/login Cursor tooling for that user if the notes job logs bridge/auth errors. See Slack release notes below.
- Actions secret
- Staging daily pytest → Slack (optional):
.github/workflows/staging-daily-pytest.ymlrunsuv run pytestnightly at 2am EDT againststaging. Failure/timeout posts a Slack root with a link to the Actions run and a thread of failed tests. ReusesSLACK_BOT_TOKENandSLACK_CHANNEL_ID. Cron is registered from the default-branch workflow file (develop). See Staging daily pytest below. -
Parlay Ops (optional): a Slack app whose events and button clicks hit the staging API. See Parlay Ops.
-
Deploy directory and filesystem data (run as the runner user):
sudo mkdir -p /opt/parlay-rewards sudo chown -R "$(whoami)" /opt/parlay-rewards mkdir -p "$HOME/parlay-rewards-data"/{listener,localstack,prometheus} cp .env.staging.example /opt/parlay-rewards/.env.staging # Edit .env.staging: set DATABASE_URL with shared Postgres user/password, API keys, etc. # App default filesystem_container_data_dir=/app/data (no env needed). # Compose hardcodes host $HOME/parlay-rewards-data → /app/data. # Listener recordings: ~/parlay-rewards-data/listener/{mlb,nhl,...}/
If you previously used /opt/parlay-rewards-data, copy listener/ and
localstack/ into $HOME/parlay-rewards-data once before the next deploy
(Docker Desktop on Mac often cannot share /opt).
- First deploy: push to
staging, or from/opt/parlay-rewardsafter a manual sync:
Ongoing¶
- Pushes to
stagingrsync into/opt/parlay-rewards(excluding.env.staging,.git,.venv,data/, andterraform/) and rebuild. Runtime data is not under the deploy tree:
| Side | Path |
|---|---|
| Host (compose bind) | $HOME/parlay-rewards-data |
| Container / app | /app/data (filesystem_container_data_dir default) |
- Update
/opt/parlay-rewards/.env.stagingwhen secrets change — Actions will not overwrite it. - Keep
$HOME/parlay-rewards-data/localstackif you need LocalStack S3 persistence across deploys. - Keep
$HOME/parlay-rewards-data/listenerfor match event recordings fromcelery-listener-worker. - Manual migrations (do not run host
uv run alembic— that misses.env.stagingand container/app/data): from/opt/parlay-rewards,make alembic_upgrade ENV=staging. That target runsalembic upgrade head, then applies any newscripts/sql/question-definition-migrations/*.sqlonce and records the file checksum. That directory is for in-game question definitions and Match question definitions. The deploy workflow runs the same two commands. Ops keeps editing those rows in admin, including definitions no script has touched. A later script must not overwrite those edits unless there is a good reason, stated in the script and its changelog. Do not edit a script whose checksum is already recorded; add a new dated pair.scripts/sql/nhl_question_definitions.staging.sqlandscripts/sql/nfl_ops_test_fixes.staging.sqlstay hand-run. - After each deploy, the workflow abandons open listener rows immediately
(
sweep_stale_match_listeners(heartbeat_stale_seconds=0), exit 14) and runsdispatch_upcoming_match_listenersso in-progress matches restart without waiting for the normal ~3m heartbeat stale window. Do not use Admin Stop listener for deploy recovery (that turns auto-dispatch off). - After a successful deploy, the workflow also creates a Sentry release for the
push SHA and records a deploy to environment
staging(requires the Sentry Actions secrets/variable above). Containers receive the same SHA asSENTRY_RELEASE. - After Deploy staging completes, the separate Staging release notes workflow may post to Slack (see below). That job is best-effort and never gates the deploy run.
- To copy those recordings into local develop (
./data/listener/), runmake listener_sync_staging_to_develop(rsync over SSH; does not delete local-only files).
Slack release notes¶
Workflow: .github/workflows/staging-release-notes.yml
Trigger: workflow_run of Deploy staging (completed), or manual
workflow_dispatch.
| Path | Slack shape |
|---|---|
| Deploy succeeded | Channel root: Block Kit; header ✅ Deployment Notification; Environment (🟡 staging / 🔵 main) + Commit; module context; Compare. When the range includes a question-definition script, the root also has a Question definitions line with those script names. Sidebar color: amber #ECB22E (staging) vs blue #1D9BD1 (main). Thread: one Block Kit reply per module (api / listener / terraform / srsim / other) via .cursor/skills/release-notes/SKILL.md. If the same git range contains scripts/sql/question-definition-migrations/YYYYMMDD_##_*.md, one more thread reply headed Question Definition Changelog [script name]. Slack folds that body behind Show more / Show less. No reply when the range has none. |
| Deploy failed | Channel root only; header 🔴 Deployment Notification; Deploy run button. Sidebar: staging #E01E5A, main #9B111E. No agent. |
Implementation notes:
- Uses Slack Web API
chat.postMessage(bot token) forthread_tsand attachment color bars. Root wraps blocks inattachments[{color, blocks}]; thread replies use top-levelblocks. Fallback plain mrkdwn if module headings cannot be parsed. - Script:
scripts/ci/post_release_notes.pywithAPP_ENV=staging(--soft-failalways in CI). Same script acceptsAPP_ENV=mainfor a future main-deploy notes workflow (blue#1D9BD1success bar; same✅ Deployment Notificationheader — env still in the Environment field). - Agent reads
.cursor/skills/release-notes/SKILL.mdwithsetting_sources=["project"]. - Commit range is previous successful Deploy staging tip → this deploy’s
SHA (via
curlon the GitHub Actions API). If that cannot be resolved, the script falls back to the deploy commit’s parent (after~1), not a multi-commit window. - Job has
continue-on-error: true. Missing secrets → log and exit 0. - Manual re-post: Actions → Staging release notes → Run workflow (optional
after_sha/before_sha, orfailurefor a failure-style root).
Staging daily pytest¶
Workflow: .github/workflows/staging-daily-pytest.yml
Trigger: cron 0 6 * * * (06:00 UTC = 2am EDT / 1am EST), or manual
workflow_dispatch.
Always checks out staging and runs the default suite
(uv run pytest --junitxml=pytest-results.xml, -m 'not external') on
Scarlett. Success is silent. Failure or the 90-minute job timeout posts a Slack
root (header Staging daily pytest failed; staging + short SHA;
Open Actions run) and a thread reply listing failing
tests and short reasons from junit (capped; remainder points at Actions).
Implementation notes:
- Reuses Actions secret
SLACK_BOT_TOKENand variable or secretSLACK_CHANNEL_ID(same as Slack release notes). - Notify is
if: failure()withcontinue-on-error: trueso a Slack miss does not hide the pytest failure. Missing token/channel → log and skip. - Script:
scripts/ci/notify_daily_pytest_failure.py(uv run; nogh/jqon the runner PATH). Junit is also uploaded as artifactpytest-results. - Slack SHA is
git rev-parse HEADafter checkout (staging tip), notgithub.sha(schedule events point at the default-branch commit). - Pytest step creates
$GITHUB_WORKSPACE/data/listenerand sets placeholderDATABASE_URL/REDIS_URL/ keys /FILESYSTEM_CONTAINER_DATA_DIR. Import-timeBaseEnvironment(viadb_sessionmanager) requires those fields; a clean Actions checkout has no.env, and/app/datadoes not exist on the host. Testcontainers still replaceDATABASE_URLandREDIS_URL(do not point tests at Scarlett's staging Redis).pytest_envsuppliesLISTENER_SR_URL_LEAGUE. Do not source.env.staginginto the pytest process (staging DB/Redis). The job setsLISTENER_SR_URL_BASEto the staging srsim;requires_srsimtests skip when that host is unreachable. - GitHub registers cron from the default-branch workflow file (
develop); the job then checks outstaging. Merge the YAML todevelopbefore expecting the nightly run. - Manual: Actions → Staging daily pytest → Run workflow (still tests
staging).
Parlay Ops¶
Mention the bot in the allowlisted ops channel. It reads the whole thread, calls the admin API, and asks for Approve before any write.
Compose services, each capped at 1 GB and 0.5 CPU:
ops-agent— Slack worker. OpenRouter model calls. Reads.env.stagingthrough Composeenv_file. The app database URL and AWS keys are cleared on this container.ops-mcp— private FastMCP server onservices-netonly. Same.env.stagingfile. Read-only SQL roleops_agent_roviaOPS_AGENT_DATABASE_URL. The appDATABASE_URLis cleared. Recording search. Docs mounted from./docs.
Edit those keys in /opt/parlay-rewards/.env.staging on Scarlett. A deploy
does not copy that file. Recreate the two containers after a change.
docker restart keeps the old environment.
Create the app from slack/parlay-ops.manifest.json (api.slack.com → Create New App → From a manifest). The staging API must already answer those URLs so Slack can verify them. Then install the app and copy the signing secret and bot token into .env.staging.
Slack Request URLs:
- Events:
https://pr-api-staging.jvsassoc.com/api/webhooks/slack-ops/events - Interactivity:
https://pr-api-staging.jvsassoc.com/api/webhooks/slack-ops/interactions
Bot scopes: app_mentions:read, chat:write, channels:history,
groups:history.
.env.staging keys: SLACK_OPS_AGENT_SIGNING_SECRET,
SLACK_OPS_AGENT_BOT_TOKEN, SLACK_OPS_AGENT_USER_IDS,
SLACK_OPS_AGENT_CHANNEL_IDS, OPS_AGENT_ADMIN_API_KEY,
OPS_AGENT_MCP_TOKEN, OPS_AGENT_DATABASE_URL, OPENROUTER_API_KEY.
Which model runs is src/ops_agent/models.json, not an environment variable.
A lookup starts on the cheap pool. An investigation or a write starts on
x-ai/grok-4.6:floor. A cheap pass that ends with nothing to post runs
again on that model.
Create the Postgres role ops_agent_ro on Scarlett with SELECT only, then
uv run alembic upgrade head so the migration can grant that role. Empty
allowlists deny everyone.
Guides: docs/agents/ops_agent.md and
docs/agents/ops_question_definitions.md.
Gzip listener archives (one-time)¶
New listener archives are uploaded gzip-compressed (Content-Encoding: gzip)
under the same keys, and reads decode transparently, so no ongoing action is
needed. To compress recordings that were archived before this change, run the
one-time script inside staging (it reads AWS_S3_MATCH_ARCHIVE_BUCKET and
credentials from .env.staging):
# Dry run — lists what would change, writes nothing:
docker compose -f docker-compose.staging.yml --env-file .env.staging \
run --rm fastapi uv run python scripts/s3/gzip_listener_archives.py
# Apply — rewrites objects as gzip and purges old versions:
docker compose -f docker-compose.staging.yml --env-file .env.staging \
run --rm fastapi uv run python scripts/s3/gzip_listener_archives.py --apply
Each object is rewritten as gzip and read back to confirm it decodes to the original bytes before the old versions are removed, so an interrupted run is safe to rerun.
Warning:
--applyPERMANENTLY deletes the prior uncompressed object versions (this cannot be undone) and requiress3:ListBucketVersionsands3:DeleteObjectVersionon the match-archive bucket. Grant those to the IAM user whose keys are in.env.stagingbefore running--apply. Without them the gzip rewrite still succeeds and verifies; only the old-version cleanup fails (reported per key, non-fatal), and those versions age out via the bucket's Glacier lifecycle instead.
Compose notes¶
- No host port publishes; Caddy reverse-proxies by hostname.
- Shared Postgres DB:
parlay_staging. Shared Redis: DB 0 (app +celerylocktask locks), DB 1 (Celery redbeat), DB 2 (Celery broker). Brokervisibility_timeoutis 12h so long-runningmatch_listenerstasks are not redelivered mid-game. - LocalStack stays in this project (
PERSISTENCE=1, bind mount under$HOME/parlay-rewards-data/localstack). - MkDocs is built into an image (not bind-mounted). Docker Desktop on Scarlett
often cannot see
/opt/parlay-rewards, which mademkdocs.ymllook missing. The same/optlimitation is why filesystem data lives under$HOME. - Flower runs by default (alias
pr-flower, public atpr-flower-staging.jvsassoc.com) with basic auth via--basic-auth; update the credentials indocker-compose.staging.yml. - Staging compose bind-mounts
${HOME}/parlay-rewards-data:/app/data. App defaultsfilesystem_container_data_dirto/app/data(omit from.env.staging). - Optional profiles:
monitoring(Prometheus),listener. celery-listener-workerconsumes the dedicatedmatch_listenersqueue (long-running match listeners). Beat runsdispatch_upcoming_match_listenersevery minute (APP_ENV != develop). Develop instead schedules test-clone dispatch only. See Match listener startup.