Fix/security reliability workflow hardening

[james2177]

Description

This PR addresses four medium-tier security and reliability issues found in .github/workflows/ and infrastructure documentation. Each fix is an isolated commit on fix/security-reliability-workflow-hardening.

---

1. Pin third-party GitHub Actions to commit SHAs (Security)

Problem: Only aquasecurity/trivy-action and trufflesecurity/trufflehog were pinned to full commit SHAs. Every other third-party action across all 14 workflows was referenced by a mutable major-version tag (@v4, @v12, etc.) — a compromised maintainer account could silently repoint the tag to malicious code with zero diff visible in this repo.

What changed: Resolved the current commit SHA for each tag/branch via git ls-remote and re-pinned every third-party (non-actions/, non-github/) action, each with a trailing # vX comment for readability:

dtolnay/rust-toolchain, hashicorp/setup-terraform, aws-actions/configure-aws-credentials, bridgecrewio/checkov-action, docker/setup-buildx-action, docker/login-action, docker/metadata-action, docker/build-push-action, slackapi/slack-github-action, actions-rs/toolchain, softprops/action-gh-release, orhun/git-cliff-action, peter-evans/create-pull-request, codecov/codecov-action, returntocorp/semgrep-action, gitleaks/gitleaks-action.

actions/* and github/* actions were intentionally left on tags, per the issue's stated scope.

Verification: grep -rEn "uses: [a-zA-Z0-9.-]+/[a-zA-Z0-9.-]+@v[0-9]" .github/workflows now returns zero third-party matches.

---

2. Route CHANGELOG updates through a PR instead of pushing to main (Security)

Problem: release.yml's changelog job ran git commit + git push origin HEAD:main using the default GITHUB_TOKEN (contents: write), letting an automated bot push straight to main with no PR or review step — either requiring an undocumented branch-protection exception for github-actions[bot], or silently failing if protection is enforced.

What changed:

• Replaced the direct push with peter-evans/create-pull-request, which opens a changelog-update/ branch against main instead of committing directly.
• Added pull-requests: write to the workflow's top-level permissions so the action can open the PR.
• protect-changelog.yml unconditionally fails any PR touching CHANGELOG.md, which would have blocked this new automated PR. Added a narrow, documented exception: only a PR from github-actions[bot] on a changelog-update/* branch is allowed through. Every ot is still blocked exactly as before — no blanket bot bypass.

---

3. Fix ROLLBACK.md S3 bucket/key drift (Reliability)

Problem: infrastructure/ROLLBACK.md's "Rollback via Terraform State" section referenced --bucket predictiq-terraform-state and --prefix prod/terraform.tfstate — neither of which exist. The
real backend config uses per-environment names.

What changed: Corrected every command in that section to match the actual backe

┌─────────────┬──────────────────────────────────────┬─────────────────────────
│ Environment │ Bucket │ Key │
├─────────────┼──────────────────────────────────────┼─────────────────────────
│ Staging │ predictiq-terraform-state-staging │ staging/terraform.tfstate │
├─────────────┼──────────────────────────────────────┼─────────────────────────
│ Production │ predictiq-terraform-state-production │ production/terraform.tfstate │
└─────────────┴──────────────────────────────────────┴─────────────────────────

Also fixed:

• terraform init -backend-config=... now points at the correct backend file.
• -var-file="environments/prod.tfvars" → environments/production/terraform.tfvaxists).
• Verification-section resource filters/IDs (Environment=prod → production, predictiq-prod → predictiq-production).
• Added a bucket/key reference table so future backend changes are easy to cros

---

4. Add timeout-minutes to every CI job (Reliability)

Problem: None of the 14 workflows set a job-level timeout. A hung step (cargo install cargo-audit/soroban-cli, or a stuck Terraform apply) would run until GitHub's 360-minute default — costly in runner minutes, and for deploy.yml's Terraform apply, it also holds a DynamoDB state lock, blocking every other environment behind it in the max-parallel: 1 matrix.

What changed:

• Added timeout-minutes: 20 to every job across all 14 workflow files.
• Tightened to timeout-minutes: 15 for the jobs most likely to hang and most costly to block on: deploy.yml's terraform-apply job, and the cargo-audit install jobs in dependency-scan.ymland test.yml.

Verification: grep -Lc timeout-minutes .github/workflows/*.yml returns no files.

---

Documentation

Added WORKFLOW_HARDENING.md at the repo root summarizing all four changes in one place, since each individual commit's diff is small (under 150 lines) and benefits from consolidatedcontext.

Known pre-existing issue (not fixed, out of scope)

While validating YAML syntax, found that test.yml's clippy and oracle-quality-gate jobs are missing a with: key before components: clippy on the dtolnay/rust-toolchain step — this is invalid YAML that predates this branch (present on main before any of these changes) and isn't part of the four issues being fixed here. Flagging it for a separate fix rather than foldingit into this PR's scope.

Testing

Per instructions, this PR was not built or run locally. Before merging, please:

• Run actionlint across .github/workflows/ and confirm zero warnings (aside from the pre-existing test.yml issue noted above).
• Confirm grep -rEn "uses: [a-zA-Z0-9.-]+/[a-zA-Z0-9.-]+@v[0-9]" .github/workflows returns no third-party matches.
• Trigger release.yml against a test tag and confirm the CHANGELOG update arrives as a PR, not a direct push.
• Run the corrected aws s3 ls s3://predictiq-terraform-state-production/production/ command and confirm it lists the real state file.
• Confirm every workflow file declares timeout-minutes per job.

Type of Change

• Bug fix
• New feature
• Refactor / code cleanup
• Documentation update
• CI / tooling change
• Breaking change

Testing Done

Bundle Size

Chunk	Before	After
vendor.js
main*.js
pages/_app*.js

Checklist

• Tests pass locally
• Documentation updated (if applicable)
• No breaking changes, or breaking changes are documented above
• If you added or changed an API endpoint, regenerated the OpenAPI spec and committed the result:

  cd services/api && cargo run --bin generate-openapi > openapi.yaml
  git add openapi.yaml && git commit -m "chore: regenerate openapi.yaml"

• If you changed system architecture (new service, database, external dependency, or network boundary), updated docs/architecture.md
• Bundle size checked (if frontend changes)

Related Issues

Closes #1212
Closes #1213
Closes #1214
Closes #1215

[drips-wave]

@james2177 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits