Skip to content

Staging on Scarlett

Parlay Rewards staging runs as its own Docker Compose project on Scarlett, joined to the shared services-net network from the services stack (Caddy, Postgres, Redis).

Endpoints

Service URL
API https://pr-api-staging.jvsassoc.com
Live play WS wss://pr-api-staging.jvsassoc.com/api/play/stream (same API upstream)
srsim https://pr-srsim-staging.jvsassoc.com
Docs https://pr-docs-staging.jvsassoc.com
Flower https://pr-flower-staging.jvsassoc.com (Celery monitoring; basic auth)

Caddy (services repo) should expose a dedicated matcher for /api/play/stream* with WebSocket upgrade to the same pr-fastapi upstream as /api/*, so prod ALB can mirror path-based routing later. See Environment configuration — Live play realtime.

One-time host setup (Scarlett)

  1. Shared stack (from the services repo): ensure network is named services-net, reload Caddy with the Parlay routes, create the DB. Read the Postgres user from the container env so you don't hit role "root" does not exist:

    docker compose up -d caddy
    # If the network still has a project-prefixed name, recreate so it is literally `services-net`.
    docker exec -it services-postgres sh -c 'psql -U "$POSTGRES_USER" -d postgres -c "CREATE DATABASE parlay_staging;"'
    

  2. Self-hosted GitHub Actions runner for this repo (Scarlett is a Mac):

  3. Follow GitHub’s runner docs and pick the macOS / ARM64 runner (Apple Silicon).
  4. The workflow uses runs-on: [self-hosted, macOS, ARM64], which matches the default macOS runner labels — no custom label needed. (If you add one, make sure runs-on still matches.)
  5. Install as a service: ./svc.sh install && ./svc.sh start (no sudo needed on macOS for a per-user service).
  6. Docker permissions on macOS: there is no docker group step. The runner runs as your user; just confirm docker ps works in that user's shell and set Docker Desktop to start at login.
  7. Prefer a private repo; the workflow only triggers on push to staging (not pull_request).
  8. Sentry release tracking (optional but recommended): add repo Actions secrets SENTRY_AUTH_TOKEN, SENTRY_ORG, and SENTRY_PROJECT or SENTRY_PROJECTS (space-separated Sentry project slugs for API / srsim / celery / listener). After each successful deploy the workflow creates a Sentry release (git SHA) and a staging deploy, and passes SENTRY_RELEASE into Compose so events attach to that release. See Sentry.
  9. Sentry Performance (API latency): .env.staging.example sets SENTRY_TRACES_SAMPLE_RATE=0.1. Copy that into server .env.staging to sample ~10% of API requests for P90/P95 (and DB spans on those requests). See Sentry Performance.
  10. Staging release notes → Slack (optional): a separate workflow (.github/workflows/staging-release-notes.yml) runs after Deploy staging completes. It does not affect deploy success/failure. Add:
    • Actions secret CURSOR_API_KEY (Cursor user or service-account API key)
    • Actions secret SLACK_BOT_TOKEN (bot with chat:write in the target channel)
    • Actions variable or secret SLACK_CHANNEL_ID (e.g. C0123…)
    • Optional Actions variable CURSOR_MODEL (default grok-4.6, not Fast) The runner user on Scarlett must be able to launch the Cursor local agent bridge used by cursor-sdk (uv run --with cursor-sdk …). Install/login Cursor tooling for that user if the notes job logs bridge/auth errors. See Slack release notes below.
  11. Staging daily pytest → Slack (optional): .github/workflows/staging-daily-pytest.yml runs uv run pytest nightly at 2am EDT against staging. Failure/timeout posts a Slack root with a link to the Actions run and a thread of failed tests. Reuses SLACK_BOT_TOKEN and SLACK_CHANNEL_ID. Cron is registered from the default-branch workflow file (develop). See Staging daily pytest below.
  12. Parlay Ops (optional): a Slack app whose events and button clicks hit the staging API. See Parlay Ops.

  13. Deploy directory and filesystem data (run as the runner user):

    sudo mkdir -p /opt/parlay-rewards
    sudo chown -R "$(whoami)" /opt/parlay-rewards
    mkdir -p "$HOME/parlay-rewards-data"/{listener,localstack,prometheus}
    cp .env.staging.example /opt/parlay-rewards/.env.staging
    # Edit .env.staging: set DATABASE_URL with shared Postgres user/password, API keys, etc.
    # App default filesystem_container_data_dir=/app/data (no env needed).
    # Compose hardcodes host $HOME/parlay-rewards-data → /app/data.
    # Listener recordings: ~/parlay-rewards-data/listener/{mlb,nhl,...}/
    

If you previously used /opt/parlay-rewards-data, copy listener/ and localstack/ into $HOME/parlay-rewards-data once before the next deploy (Docker Desktop on Mac often cannot share /opt).

  1. First deploy: push to staging, or from /opt/parlay-rewards after a manual sync:
    docker compose -f docker-compose.staging.yml --env-file .env.staging up -d --build
    make alembic_upgrade ENV=staging
    

Ongoing

  • Pushes to staging rsync into /opt/parlay-rewards (excluding .env.staging, .git, .venv, data/, and terraform/) and rebuild. Runtime data is not under the deploy tree:
Side Path
Host (compose bind) $HOME/parlay-rewards-data
Container / app /app/data (filesystem_container_data_dir default)
  • Update /opt/parlay-rewards/.env.staging when secrets change — Actions will not overwrite it.
  • Keep $HOME/parlay-rewards-data/localstack if you need LocalStack S3 persistence across deploys.
  • Keep $HOME/parlay-rewards-data/listener for match event recordings from celery-listener-worker.
  • Manual migrations (do not run host uv run alembic — that misses .env.staging and container /app/data): from /opt/parlay-rewards, make alembic_upgrade ENV=staging. That target runs alembic upgrade head, then applies any new scripts/sql/question-definition-migrations/*.sql once and records the file checksum. That directory is for in-game question definitions and Match question definitions. The deploy workflow runs the same two commands. Ops keeps editing those rows in admin, including definitions no script has touched. A later script must not overwrite those edits unless there is a good reason, stated in the script and its changelog. Do not edit a script whose checksum is already recorded; add a new dated pair. scripts/sql/nhl_question_definitions.staging.sql and scripts/sql/nfl_ops_test_fixes.staging.sql stay hand-run.
  • After each deploy, the workflow abandons open listener rows immediately (sweep_stale_match_listeners(heartbeat_stale_seconds=0), exit 14) and runs dispatch_upcoming_match_listeners so in-progress matches restart without waiting for the normal ~3m heartbeat stale window. Do not use Admin Stop listener for deploy recovery (that turns auto-dispatch off).
  • After a successful deploy, the workflow also creates a Sentry release for the push SHA and records a deploy to environment staging (requires the Sentry Actions secrets/variable above). Containers receive the same SHA as SENTRY_RELEASE.
  • After Deploy staging completes, the separate Staging release notes workflow may post to Slack (see below). That job is best-effort and never gates the deploy run.
  • To copy those recordings into local develop (./data/listener/), run make listener_sync_staging_to_develop (rsync over SSH; does not delete local-only files).

Slack release notes

Workflow: .github/workflows/staging-release-notes.yml
Trigger: workflow_run of Deploy staging (completed), or manual workflow_dispatch.

Path Slack shape
Deploy succeeded Channel root: Block Kit; header ✅ Deployment Notification; Environment (🟡 staging / 🔵 main) + Commit; module context; Compare. When the range includes a question-definition script, the root also has a Question definitions line with those script names. Sidebar color: amber #ECB22E (staging) vs blue #1D9BD1 (main). Thread: one Block Kit reply per module (api / listener / terraform / srsim / other) via .cursor/skills/release-notes/SKILL.md. If the same git range contains scripts/sql/question-definition-migrations/YYYYMMDD_##_*.md, one more thread reply headed Question Definition Changelog [script name]. Slack folds that body behind Show more / Show less. No reply when the range has none.
Deploy failed Channel root only; header 🔴 Deployment Notification; Deploy run button. Sidebar: staging #E01E5A, main #9B111E. No agent.

Implementation notes:

  • Uses Slack Web API chat.postMessage (bot token) for thread_ts and attachment color bars. Root wraps blocks in attachments[{color, blocks}]; thread replies use top-level blocks. Fallback plain mrkdwn if module headings cannot be parsed.
  • Script: scripts/ci/post_release_notes.py with APP_ENV=staging (--soft-fail always in CI). Same script accepts APP_ENV=main for a future main-deploy notes workflow (blue #1D9BD1 success bar; same ✅ Deployment Notification header — env still in the Environment field).
  • Agent reads .cursor/skills/release-notes/SKILL.md with setting_sources=["project"].
  • Commit range is previous successful Deploy staging tip → this deploy’s SHA (via curl on the GitHub Actions API). If that cannot be resolved, the script falls back to the deploy commit’s parent (after~1), not a multi-commit window.
  • Job has continue-on-error: true. Missing secrets → log and exit 0.
  • Manual re-post: Actions → Staging release notes → Run workflow (optional after_sha / before_sha, or failure for a failure-style root).

Staging daily pytest

Workflow: .github/workflows/staging-daily-pytest.yml
Trigger: cron 0 6 * * * (06:00 UTC = 2am EDT / 1am EST), or manual workflow_dispatch.

Always checks out staging and runs the default suite (uv run pytest --junitxml=pytest-results.xml, -m 'not external') on Scarlett. Success is silent. Failure or the 90-minute job timeout posts a Slack root (header Staging daily pytest failed; staging + short SHA; Open Actions run) and a thread reply listing failing tests and short reasons from junit (capped; remainder points at Actions).

Implementation notes:

  • Reuses Actions secret SLACK_BOT_TOKEN and variable or secret SLACK_CHANNEL_ID (same as Slack release notes).
  • Notify is if: failure() with continue-on-error: true so a Slack miss does not hide the pytest failure. Missing token/channel → log and skip.
  • Script: scripts/ci/notify_daily_pytest_failure.py (uv run; no gh / jq on the runner PATH). Junit is also uploaded as artifact pytest-results.
  • Slack SHA is git rev-parse HEAD after checkout (staging tip), not github.sha (schedule events point at the default-branch commit).
  • Pytest step creates $GITHUB_WORKSPACE/data/listener and sets placeholder DATABASE_URL / REDIS_URL / keys / FILESYSTEM_CONTAINER_DATA_DIR. Import-time BaseEnvironment (via db_sessionmanager) requires those fields; a clean Actions checkout has no .env, and /app/data does not exist on the host. Testcontainers still replace DATABASE_URL and REDIS_URL (do not point tests at Scarlett's staging Redis). pytest_env supplies LISTENER_SR_URL_LEAGUE. Do not source .env.staging into the pytest process (staging DB/Redis). The job sets LISTENER_SR_URL_BASE to the staging srsim; requires_srsim tests skip when that host is unreachable.
  • GitHub registers cron from the default-branch workflow file (develop); the job then checks out staging. Merge the YAML to develop before expecting the nightly run.
  • Manual: Actions → Staging daily pytest → Run workflow (still tests staging).

Parlay Ops

Mention the bot in the allowlisted ops channel. It reads the whole thread, calls the admin API, and asks for Approve before any write.

Compose services, each capped at 1 GB and 0.5 CPU:

  • ops-agent — Slack worker. OpenRouter model calls. Reads .env.staging through Compose env_file. The app database URL and AWS keys are cleared on this container.
  • ops-mcp — private FastMCP server on services-net only. Same .env.staging file. Read-only SQL role ops_agent_ro via OPS_AGENT_DATABASE_URL. The app DATABASE_URL is cleared. Recording search. Docs mounted from ./docs.

Edit those keys in /opt/parlay-rewards/.env.staging on Scarlett. A deploy does not copy that file. Recreate the two containers after a change. docker restart keeps the old environment.

Create the app from slack/parlay-ops.manifest.json (api.slack.com → Create New App → From a manifest). The staging API must already answer those URLs so Slack can verify them. Then install the app and copy the signing secret and bot token into .env.staging.

Slack Request URLs:

  • Events: https://pr-api-staging.jvsassoc.com/api/webhooks/slack-ops/events
  • Interactivity: https://pr-api-staging.jvsassoc.com/api/webhooks/slack-ops/interactions

Bot scopes: app_mentions:read, chat:write, channels:history, groups:history.

.env.staging keys: SLACK_OPS_AGENT_SIGNING_SECRET, SLACK_OPS_AGENT_BOT_TOKEN, SLACK_OPS_AGENT_USER_IDS, SLACK_OPS_AGENT_CHANNEL_IDS, OPS_AGENT_ADMIN_API_KEY, OPS_AGENT_MCP_TOKEN, OPS_AGENT_DATABASE_URL, OPENROUTER_API_KEY. Which model runs is src/ops_agent/models.json, not an environment variable. A lookup starts on the cheap pool. An investigation or a write starts on x-ai/grok-4.6:floor. A cheap pass that ends with nothing to post runs again on that model.

Create the Postgres role ops_agent_ro on Scarlett with SELECT only, then uv run alembic upgrade head so the migration can grant that role. Empty allowlists deny everyone.

Guides: docs/agents/ops_agent.md and docs/agents/ops_question_definitions.md.

Gzip listener archives (one-time)

New listener archives are uploaded gzip-compressed (Content-Encoding: gzip) under the same keys, and reads decode transparently, so no ongoing action is needed. To compress recordings that were archived before this change, run the one-time script inside staging (it reads AWS_S3_MATCH_ARCHIVE_BUCKET and credentials from .env.staging):

# Dry run — lists what would change, writes nothing:
docker compose -f docker-compose.staging.yml --env-file .env.staging \
  run --rm fastapi uv run python scripts/s3/gzip_listener_archives.py

# Apply — rewrites objects as gzip and purges old versions:
docker compose -f docker-compose.staging.yml --env-file .env.staging \
  run --rm fastapi uv run python scripts/s3/gzip_listener_archives.py --apply

Each object is rewritten as gzip and read back to confirm it decodes to the original bytes before the old versions are removed, so an interrupted run is safe to rerun.

Warning: --apply PERMANENTLY deletes the prior uncompressed object versions (this cannot be undone) and requires s3:ListBucketVersions and s3:DeleteObjectVersion on the match-archive bucket. Grant those to the IAM user whose keys are in .env.staging before running --apply. Without them the gzip rewrite still succeeds and verifies; only the old-version cleanup fails (reported per key, non-fatal), and those versions age out via the bucket's Glacier lifecycle instead.

Compose notes

  • No host port publishes; Caddy reverse-proxies by hostname.
  • Shared Postgres DB: parlay_staging. Shared Redis: DB 0 (app + celerylock task locks), DB 1 (Celery redbeat), DB 2 (Celery broker). Broker visibility_timeout is 12h so long-running match_listeners tasks are not redelivered mid-game.
  • LocalStack stays in this project (PERSISTENCE=1, bind mount under $HOME/parlay-rewards-data/localstack).
  • MkDocs is built into an image (not bind-mounted). Docker Desktop on Scarlett often cannot see /opt/parlay-rewards, which made mkdocs.yml look missing. The same /opt limitation is why filesystem data lives under $HOME.
  • Flower runs by default (alias pr-flower, public at pr-flower-staging.jvsassoc.com) with basic auth via --basic-auth; update the credentials in docker-compose.staging.yml.
  • Staging compose bind-mounts ${HOME}/parlay-rewards-data:/app/data. App defaults filesystem_container_data_dir to /app/data (omit from .env.staging).
  • Optional profiles: monitoring (Prometheus), listener.
  • celery-listener-worker consumes the dedicated match_listeners queue (long-running match listeners). Beat runs dispatch_upcoming_match_listeners every minute (APP_ENV != develop). Develop instead schedules test-clone dispatch only. See Match listener startup.