CI/CD & Environments

This guide covers the deployment pipeline, environment types, and automation workflows.

Environments

The monorepo supports three environment types:

Environment Differences

CI/CD Workflows

Deployment Workflows

Verification Workflow (verify.yml)

Runs on every PR (ready for review / synchronize):

1. Lint — pnpm check --affected (lint + type-check across affected workspaces)
2. Backend Tests — pnpm run test in apps/backend (with coverage)
3. Check PR conventions — Validates conventional commit subjects and the pull request title

Deployment Pipeline (base-deploy.yml)

The reusable deployment workflow runs these jobs:

frontend-build ──┐
                  ├──► frontend-upload
infrastructure ──┘
     │
     └──► publish-docs (production + integration only)

1. frontend-build — Installs deps, builds Vite app, uploads artifact
2. infrastructure — Runs the deploy orchestration script (see Deploy Orchestration below)
3. frontend-upload — Downloads build artifact, syncs to S3, invalidates CloudFront
4. publish-docs — Publishes Scalar documentation (production and integration only)

Docs Publishing: Documentation is published for production and integration only. Each environment has its own Scalar project and custom domain: production at docs.gamecraft.learnwith.ai, integration at docs-integration.gamecraft.rp.devfactory.com. The API reference automatically points to the environment's own API, so reviewers can test endpoints directly from the docs. Ephemeral (PR) environments do not publish docs.

Deploy Orchestration

The infrastructure job runs pnpm run deploy:<type> from apps/infra/, which invokes deploy.orchestrator.ts. This orchestrator runs the full deploy lifecycle in a single process:

1. Seed reservations — provisions shared account-level resources (CloudFront origin request policies) and writes SSM reservations to protect them from concurrent cleanup
2. Synth — runs cdk synth to produce the CloudFormation template (separate step because pre-deploy reads the synthesized template)
3. Pre-deploy (pre-deploy.hook.ts) — imports orphaned CloudWatch log groups so CDK can adopt them
4. CDK deploy — applies the CloudFormation changeset (Lambda, API Gateway, DynamoDB, Cognito, CloudFront)
5. Post-deploy (post-deploy.hook.ts):
   • Publishes environment config JSON to S3 (used by frontend for environment switching)
   • Runs optional external integrations via the integration registry (e.g., Slack notifications, Datadog dashboard upserts)
   • Seeds ephemeral environments (copies Cognito users and the DynamoDB table from integration, excluding environment-bound studio content — games, chats, shares — and re-applies admin designations)
   • Seeds the API and E2E test users (ephemeral and integration; never production) and stores their credentials in SSM, where pnpm script update-env and the post-deploy test jobs pick them up

Persistence

All environments run on DynamoDB. There is no runtime persistence selector and no alternate store path in the codebase.

Destroy Orchestration

When a PR is closed, destroy-ephemeral.yml runs pnpm run destroy:ephemeral from apps/infra/, which invokes destroy.orchestrator.ts. The same workflow also accepts manual workflow_dispatch for orphaned ephemeral stacks whose automatic destroy failed or never ran (for example, a closed PR whose stack still exists).

Manual dispatch must check out a fresh fixed ref (typically main at the current tip), not re-run an old failed workflow attempt. Destroy hooks evolve with the repo; replaying the code snapshot from the original PR-close run can miss pre-destroy teardown or grant assumptions that later commits fixed.

1. Pre-destroy (pre-destroy.hook.ts): tears down hook-managed Studio AgentCore resources before CloudFormation deletes the stack:
   • Deletes the studio_<env> AgentCore runtime and its workload identity
   • Deletes the per-env studio execution role
   • Fail-closes: any failure aborts the orchestrator before cdk destroy (while the shared CI deploy role still holds the studio lifecycle grants)
2. CDK destroy: tears down the CloudFormation stack
3. Post-destroy (post-destroy.hook.ts): cleans up environment artifacts that outlive the stack:
   • Deletes S3 environment configs
   • Removes per-env SSM parameters (including the studio runtime ARN)
   • Cleans up orphaned CloudWatch log groups from CDK custom resources
   • Runs optional integration cleanup (e.g., revokes external resources registered by integrations)
   • Releases and sweeps SSM reservations for shared resources

Post-destroy is intentionally skipped when CDK destroy fails: the stack still exists, so cleaning up reservations or shared policies would be incorrect. On failure, a separate report-failure job (no uses: steps) opens a GitHub issue for manual intervention so a setup-stage Actions outage still surfaces.

Ephemeral reconciliation (reconcile-ephemeral.yml)

A single PR-close destroy shot can miss (dropped webhook, Actions infrastructure failure). Every six hours, reconcile-ephemeral.yml lists live gamecraft-pr* CloudFormation stacks in the integration account, resolves each stack's pull request, and dispatches destroy-ephemeral.yml for orphans whose PR closed outside a grace window (default 60 minutes). Refusal guards match prune-ai-games-branches: preview is the default for manual dispatch, --max-orphans bounds the verdict, the caller must be the integration account, and AWS clients are region-pinned. Scheduled runs always execute. Branch pruning stays a separate script (prune-ai-games-branches); this workflow does not schedule it — see ai-games branch pruning for the dedicated sweep that does.

# Preview the verdict (no destroys dispatched)
gh workflow run reconcile-ephemeral.yml --ref main

# Execute after reviewing a preview
gh workflow run reconcile-ephemeral.yml --ref main -f execute=true

The CI OIDC role needs cloudformation:ListStacks via ReconcileEphemeralRoleGrantConstruct (non-ephemeral only); that grant attaches when the integration stack redeploys after merge.

ai-games branch pruning (prune-ai-games-branches.yml)

The pre-destroy hook's ai-games branch deletion is non-fatal, so a failed deletion leaks games/pr*/* branches with nothing to retry it. Every six hours at 03/09/15/21 UTC (three hours after each reconcile-ephemeral.yml slot), prune-ai-games-branches.yml lists games/<env>/* on the ai-games remote, retains durable environments and live ephemeral stacks, and deletes branches of pr<digits> environments whose gamecraft-<env>-stack is absent. The three-hour offset means branches of an environment the stack sweep just destroyed are pruned in the same window rather than the next one. An environment whose stack is mid-delete (DELETE_IN_PROGRESS, or stuck DELETE_FAILED) is logged as Retaining live environment and swept at the following slot, so a leak's lifetime is one window and occasionally two. Scheduled runs always execute; manual dispatch previews by default. --max-dead (default 5) bounds the destructive verdict: a larger backlog refuses before deleting anything, the report-failure job opens an issue titled ai-games branch prune sweep is failing, and recovery is a preview followed by -f execute=true -f max-dead=<n>. No new IAM grant is involved: liveness uses DescribeStacks (already held by the CI OIDC role), and ref deletion uses the integration backend secret's bot token.

# Preview the verdict (no branches deleted)
gh workflow run prune-ai-games-branches.yml --ref main

# Execute after reviewing a preview
gh workflow run prune-ai-games-branches.yml --ref main -f execute=true

Shared CI deploy-role grants (non-ephemeral only)

The shared GithubActionsOidcCdkDeployRole (one per account; see CI_DEPLOY_ROLE_NAME in apps/infra/constants/app.constant.ts) is what CI deploys and workflow-dispatched scripts run under. Grant constructs attach inline policies to it, and the wiring rule is: only the integration and production stacks may seed account-wide grants, because multiple CloudFormation stacks must not claim policies on the same shared role, and an account-family wildcard (e.g. gamecraft-*-backend-table) already covers every environment's resources.

• StudioDeployRoleGrantConstruct — ECR push, studio execution-role lifecycle, and AgentCore runtime lifecycle for the studio deploy hooks.
• RefreshServedGamesRoleGrantConstruct — dynamodb:GetItem on the backend-table family for the refresh-served-games workflow's per-game busy check (gamecraft#342).
• ReconcileEphemeralRoleGrantConstruct — cloudformation:ListStacks for the ephemeral reconciliation sweep (gamecraft#515). Policy name is deterministic (gamecraft-<env>-reconcile-ephemeral-grant).
• TestCleanupRoleGrantConstruct — Cognito admin actions on userpool/* for post-deploy test cleanup. Formerly wired per stack (ephemeral included); gamecraft#323 made it non-ephemeral-only because the account-wide wildcard copy already authorised every environment's pools. Caveat: when expanding its actions, integration must redeploy before an ephemeral post-deploy can rely on the new action.

Hand-provisioned grants (development account only): the shared-resource seeds run before cdk deploy (see .claude/skills/infra/account-wide-resources/SKILL.md), so their grants cannot be CDK-managed — a grant construct could not authorize its own first run. They are applied once by hand in the development account (tapdev, 730335631880); the production role in tap (010526244253) carries AdministratorAccess, so shared-resource grants are a development-account concern only. Three exist today: ManageSharedOriginRequestPolicies (origin-request-policy pool), ManageSharedCloudFrontFunctions (shared CloudFront function pool, gamecraft#620) and ManageSharedCachePolicies (shared cache-policy pool, owned by trilogy-group/trashcat, trashcat#1125).

ManageSharedCloudFrontFunctions was applied 2026-08-19 with:

aws iam put-role-policy --profile tapdev \
  --role-name GithubActionsOidcCdkDeployRole \
  --policy-name ManageSharedCloudFrontFunctions \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      { "Effect": "Allow",
        "Action": ["cloudfront:ListFunctions","cloudfront:DescribeFunction","cloudfront:GetFunction",
                   "cloudfront:CreateFunction","cloudfront:PublishFunction","cloudfront:DeleteFunction"],
        "Resource": "*" },
      { "Effect": "Allow",
        "Action": ["ssm:PutParameter","ssm:GetParameter","ssm:GetParametersByPath","ssm:DeleteParameter"],
        "Resource": "arn:aws:ssm:*:730335631880:parameter/wseng/auto-function-refs*" }
    ]
  }'

ManageSharedCachePolicies was applied for trashcat's cache-policy pool (pooled names wseng-auto-cache-policy-<hash12>, reservations under /wseng/auto-cache-policy-refs/<repo>/<env>) with:

aws iam put-role-policy --profile tapdev \
  --role-name GithubActionsOidcCdkDeployRole \
  --policy-name ManageSharedCachePolicies \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      { "Effect": "Allow",
        "Action": ["cloudfront:ListCachePolicies","cloudfront:GetCachePolicy",
                   "cloudfront:CreateCachePolicy","cloudfront:DeleteCachePolicy"],
        "Resource": "*" },
      { "Effect": "Allow",
        "Action": ["ssm:PutParameter","ssm:GetParameter","ssm:GetParametersByPath","ssm:DeleteParameter"],
        "Resource": "arn:aws:ssm:*:730335631880:parameter/wseng/auto-cache-policy-refs*" }
    ]
  }'

The role is shared across repos, so a grant seeded for another repo's pool still counts against this account's budget below; verify any of them live with aws iam get-role-policy --role-name GithubActionsOidcCdkDeployRole --policy-name <name> --profile tapdev.

Byte budget: inline policies on a role share a hard 10,240 non-whitespace-character cap, aggregated across every stack in the account — other repos included (trashcat's per-PR grants land on this same role at ~555 characters each). Measured 2026-08-22 in tapdev: 6,258 characters across 16 policies (including the 400-character ManageSharedCachePolicies), leaving ~3,900 of headroom; a CDK grant of this class costs roughly 230–250. Before adding a grant, check the headroom: aws iam list-role-policies + get-role-policy, counting the policy JSON without whitespace. Exhausting the cap fails PutRolePolicy during cdk deploy for every new environment.

Note: Manual API calls can be made via the Scalar interactive docs (/docs/viewer) or by running the auto-generated tests in packages/api-tests/.

Automation Workflows

Deployment Flow (End-to-End)

A typical integration deployment follows this sequence:

1. Developer merges a PR into main
2. deploy-integration.yml triggers automatically
3. It calls base-deploy.yml with environment: integration
4. frontend-build compiles the React app with Vite
5. infrastructure runs the deploy orchestration script (deploy:integration), which seeds account-level resources, synthesizes, runs pre-deploy hooks, deploys the CDK stack, and runs post-deploy hooks (environment config publish, integration sync)
6. frontend-upload syncs the built assets to S3 and invalidates the CloudFront cache
7. publish-docs generates and publishes Scalar documentation with the latest API spec
8. post-deploy-tests (post-deploy-tests.yml) seeds credentials then runs API and E2E suites. Before seeding, two cleanup passes keep durable environments from accumulating test fixtures (gamecraft#535; ITD docs/itds/2026-08-12-integration-fixture-game-cleanup.md):
   • Purge leftover studio fixture games — authenticates as the studio-admin test identity and deletes games whose ai-games folder slug comes from a checked-in fixture (pnpm script cleanup-fixture-games --execute). Human-owned games, and human games under the admin identity (e.g. Trashcat), are retained: a game built while signed in as the studio-admin test identity survives only if its folder slug is not a fixture slug.
   • Prune orphaned ai-games game branches — deletes games/<env>/* refs on trilogy-group/ai-games that no live StudioGame record references (pnpm script prune-orphan-game-branches --execute, targeting the pipeline's ambient ENVIRONMENT).

For PR environments, the same pipeline runs with environment: prNNNN (the environment type is derived automatically from the name), but the publish-docs job is skipped (docs are only published for production and integration).

Managing Environments

Syncing Local .env from a Deployed Environment

# From the integration environment
pnpm script update-env integration

# From a specific PR environment
pnpm script update-env 20

# Wait for a stack that is still deploying
pnpm script update-env integration --wait

The script auto-selects the AWS profile matching the target environment's account (--env/--pr flags remain supported).

Rotating STUDIO_LEDGER_SIGNING_SECRET

The Studio ledger signing secret lives at SSM /gamecraft/<env>/studio/ledger-signing-secret. Post-deploy creates it once via ensureStudioLedgerSigningSecret and deliberately never regenerates it per revision — unlike STUDIO_GATE_MEMO_SECRET, which is fresh random bytes on every deploy and needs no operator runbook.

Rotate only when the secret may have leaked or when deliberately invalidating every signed playtest ledger and judge-dispute store in that environment:

# Ephemeral example — never run against integration or production without an explicit incident decision
aws ssm delete-parameter \
  --name /gamecraft/prNNNN/studio/ledger-signing-secret \
  --profile <account-profile> \
  --region us-east-1

# Redeploy so the runtime upsert re-mints the parameter
pnpm run deploy:ephemeral   # or the matching deploy:<type> for the environment

After redeploy:

• Every existing playtest/playtest-ledger.json and playtest/judge-disputes.json in that environment verifies as empty (readVerifiedPlaytestLedger / readVerifiedDisputes fail safe toward a red gate). Retirements and disputes are re-earned on the next live runs; history is not silently trusted under the new key.
• The same fail-safe applies when a game folder is moved or renamed: Phase 1 binds the HMAC key to the relative game path, so that one game's ledger and dispute store stop verifying until re-earned under the new path.

Confirm rotation on an ephemeral stack only: record a non-zero carried count on a game whose ledger already holds a failing row, rotate, redeploy, and expect a subsequent playtest technical line of 0 carried, 0 retired.

Accessing Deployed Environments

Ephemeral environment URLs are output by the deploy step and posted as a PR comment.