Repository navigation
feat(staging): wire protected M5 service releases - #86
Conversation
|
The exact-head Compliance image and kind readiness check (run 39057023208) still fails at startup. The CI, budget regression, Terraform, M5 offline, and Helm checks pass, but this failure is the reusable-workflow caller-permissions issue being fixed in #77. Please rerun this check after #77 merges; keep #86 unmerged until it passes. |
rcncai
left a comment
There was a problem hiding this comment.
M2 review at exact head 72ce31b requests changes. This release deploys the four business charts, but their workload NetworkPolicy values still select Keycloak with app.kubernetes.io/name: keycloak (for example deploy/helm/values/label-workflow.yaml; the same peer appears in specification.yaml, formulation.yaml, and compliance.yaml). The Operator-managed Keycloak selector used in this PR is app: keycloak (render-server.py and the updated platform-policy assertions). With default-deny egress, the business pods cannot reach Keycloak JWKS, so Label JWT validation and authenticated business calls will fail. Please change the workload peers (and the optional staging-shared-label-workflow overlay) to the Operator label and add a Helm-rendered assertion covering Keycloak egress for all affected workloads. Also, PR #86 exact-head run 39057023208 previously failed at reusable-workflow startup; #77 now has the permissions fix but its exact-head checks are still running, so rerun #86 after that change is on main before merge. Terraform plan is skipped and the PR body says live Secrets/config and M1 staging approval are still missing; no live acceptance is evidenced.
rcncai
left a comment
There was a problem hiding this comment.
M2 review at exact head 72ce31b requests changes. This release deploys the four business charts, but their workload NetworkPolicy values still select Keycloak with app.kubernetes.io/name: keycloak (for example deploy/helm/values/label-workflow.yaml; the same peer appears in specification.yaml, formulation.yaml, and compliance.yaml). The Operator-managed Keycloak selector used in this PR is app: keycloak (render-server.py and the updated platform-policy assertions). With default-deny egress, the business pods cannot reach Keycloak JWKS, so Label JWT validation and authenticated business calls will fail. Please change the workload peers (and the optional staging-shared-label-workflow overlay) to the Operator label and add a Helm-rendered assertion covering Keycloak egress for all affected workloads. Also, PR #86 exact-head run 39057023208 previously failed at reusable-workflow startup; #77 now has the permissions fix but its exact-head checks are still running, so rerun #86 after that change is on main before merge. Terraform plan is skipped and the PR body says live Secrets/config and M1 staging approval are still missing; no live acceptance is evidenced.
|
M2: I addressed the requested Keycloak NetworkPolicy selector changes on exact head |
|
M2: I fixed the requested Keycloak selectors and Helm-render assertions on |
|
M2 re-review request for exact head
Please review the NetworkPolicy changes at this head. Exact-head CI is still running, so this is not a merge request; any real Dependency-Check finding remains a blocker. |
|
M2 update for exact head
Please independently re-review the NetworkPolicy fix at this exact head. The review request remains open; this comment does not request merging or approval of the security findings. |
rcncai
left a comment
There was a problem hiding this comment.
M2 re-review on exact head 288bc7bcb1c94b2efd9a3f3f5651ad81bd418571 (CI run 39065731400): the app: keycloak NetworkPolicy selector fix and render assertions are present. The unresolved blocker is the enabled OWASP Dependency-Check gate: it fails at CVSS >= 7 for HttpClient 5.5.1 (CVE-2026-71290, CVSS 9.1) and HttpCore 5.3.6 (CVE-2026-54399, CVE-2026-54428, CVSS 7.5), bundled under docker-java-transport-zerodep:3.7.1. Please upgrade/replace the affected dependency path so the scan passes without lowering its threshold or skipping/suppressing these findings, then rerun the exact-head security workflow. compliance / kind was skipped as a consequence and must pass after security is green. Terraform Plan is also skipped; no state-backed plan or live staging validation is evidenced. I cannot approve this head yet.
Security gate evidence (exact head 82bee4e)I merged the latest The current Maven dependency tree confirms: Replacing zerodep with docker-java's separate HttpClient5 artifact is not a drop-in fix: Testcontainers 2.0.5 bytecode constructs The exact-head OWASP job is still running: compliance / security. No pass or CVE-clearance claim is being made while it is in progress. No threshold reduction, new suppression, or Dependency-Check skip was added. M2/M1: please advise whether to hold this PR pending an upstream zerodep/Testcontainers release that rebuilds the shaded transport, or authorize a separately reviewed compatibility implementation/fork with Docker integration and Dependency-Check evidence. Until that exists, the security gate is unresolved and PR #86 should not be merged. |
|
Thanks for the exact-head evidence. M2 recommendation: hold #86; the exact-head Dependency-Check run failed, so the security gate is unresolved. I cannot authorize a local compatibility fork in this review. Please keep the PR blocked until a patched/rebuilt ZeroDep artifact is available, or submit a separately reviewed compatibility change with Docker integration tests and a passing exact-head Dependency-Check report. Do not merge or suppress these findings. |
rcncai
left a comment
There was a problem hiding this comment.
M2 re-review on exact head 82bee4e7771278b94c2ab2bbe060b48fd6f6c2ef (security run 39097297653): the dependency scan is still failing. It reports shaded HttpClient 5.5.1 (CVE-2026-71290, CVE-2026-64607) and HttpCore 5.3.6 (CVE-2026-54399, CVE-2026-54428) inside docker-java-transport-zerodep:3.7.1; the gate is configured to fail at CVSS >= 7. compliance / kind and the protected Terraform Plan are skipped as a result/configuration. I agree this should stay on hold until a patched/rebuilt ZeroDep/Testcontainers artifact is available or a separate compatibility implementation is independently reviewed with Docker integration and a passing exact-head scan. I do not authorize a local compatibility fork in this PR. Do not suppress the findings, skip the security module, or merge this head.
rcncai
left a comment
There was a problem hiding this comment.
M2 review of exact head 82bee4e7771278b94c2ab2bbe060b48fd6f6c2ef (run 39097297653): still blocked. The current OWASP Dependency-Check job fails on the shaded Testcontainers docker-java ZeroDep dependency: HttpClient 5.5.1 reports CVE-2026-71290 and CVE-2026-64607; HttpCore 5.3.6 reports CVE-2026-54399 and CVE-2026-54428. The gate remains at CVSS >= 7 with no new suppression/skip, and compliance / kind plus Terraform plan are skipped. Please hold until a patched/rebuilt ZeroDep artifact is available, or submit the separately reviewed compatibility change with Docker integration and a passing exact-head scan that you proposed. This PR is not approved for merge or deployment.
|
M2 synced this branch with latest main after #88 merged (ordinary merge, no history rewrite). The merge preserved #88's tested starter-test transport/POM and restored NVD-key/cache behavior in the OWASP workflow; the now-obsolete testcontainers-mysql false-positive suppression was removed. New exact-head CI is running at 5708d6c. Local Maven starter-test validation could not run here because the installed JDK rejects Java release 21; I will use the exact-head CI result for the review gate. No AWS Plan/Apply or staging action was performed. |
rcncai
left a comment
There was a problem hiding this comment.
Re-review at exact head 5708d6c: the dependency scan and main CI now pass, but run 39128853263 fails in compliance / kind. Helm waited 5 minutes while Deployment/compliance/compliance-compliance stayed Available: 0/1, then timed out. Please identify and fix the pod-readiness cause, preserve safe pod-event/log evidence on failure, and rerun this exact-head check successfully. Terraform Plan and deployment remain skipped, so there is no staging acceptance yet.
|
Fixed the failed compliance kind readiness check at 571b4d0. The pod was ImagePullBackOff because the service values pinned a GHCR digest, which overrode the local image that the kind job had built and loaded; the Helm invocation now clears image.digest so it uses the loaded local tag. This is CI-only verification; the exact new head is awaiting rerun. |
rcncai
left a comment
There was a problem hiding this comment.
Re-review at exact head 571b4d0 (Compliance run 39142036306): the new --set-string image.digest= override is present, but compliance/kind still times out after five minutes with Deployment/compliance/compliance-compliance at Available: 0/1. The Capture safe kind readiness diagnostics step reports 0s and exposes no pod events, container logs, or artifact, so the readiness cause cannot be verified. Please capture sanitized pod status/events/logs on failure and rerun this exact head successfully. Main CI and Compliance security pass, but publish/deploy and Terraform Plan remain skipped; there is no deployment or staging acceptance.
|
The rerun on 571b4d0 confirmed the local image fix: the pod now resolves local/compliance:, but readiness then exposed the next missing kind fixture, compliance-rabbitmq-user-credentials. Added a namespace-local dummy credential Secret to deploy/kind/compliance-dependencies.yaml at d905a7a; YAML parses and git diff --check passes. The new exact-head CI is running. |
rcncai
left a comment
There was a problem hiding this comment.
M2 re-review at exact head d905a7a: the CI-only, namespace-scoped RabbitMQ Secret fixture uses dummy kind-only credentials and resolves the Compliance kind startup failure. The exact-head main CI, Compliance checks/security/kind, Terraform validation, budget regression, M5 offline validation, Compliance contracts, Helm template, staging-window estimate, and Sonar quality gate all pass. Terraform Plan and publish/deploy remain skipped by their gates; this approval covers the code only and is not staging deployment or acceptance. M1/code-owner approval is still required.
|
At exact head |
|
@hxj04121-lab M1 Code Owner review requested for the exact current head M2 has approved this head. Current exact-head CI is green: main CI (including backend, frontend, containers, security, DB privileges, deployment evidence), Compliance checks/security/Kind, Terraform validation, budget regression, Helm, offline validation, and Sonar. Please review the protected M5 service-release workflow and its runtime/auth wiring, then record the Code Owner decision. The protected Terraform Plan and Compliance publish/deploy jobs are skipped, so this is a code-review request only; no staging deployment or live acceptance is claimed. |
hxj04121-lab
left a comment
There was a problem hiding this comment.
M1 Code Owner review @ d905a7a.
Overall the release path looks right and I'm happy with the design: main-only + staging environment + self-hosted VPC runner, the USD 150 cap/window/budget gates, digest verification against GHCR (the four digests match the images published by run 39050128320), helm-ownership adoption checks, server-side dry-run, --atomic, and post-deploy checks on digest/NetworkPolicy/RDS CIDR/CORS/OIDC. The IAM additions are tightly scoped (DescribeNodegroup on compliance/* only, the secret read limited to spectrace-staging/mysql/master-*), and the node-reader ClusterRole is read-only for nodes. Both have tftest assertions.
Blocking (1): port=18101 is set in the Validate step (L88), but it's only used in the Verify Pods… step. Each run: is a separate shell, so with set -u that step fails with port: unbound variable on the first kubectl port-forward. That happens after helm upgrade --atomic has already succeeded, so the releases would be live, the run red, and the verification evidence missing. I left a suggestion inline: move the assignment into the Verify step.
Non-blocking / for the release order:
- The CORS assertion requires the Gateway to be re-released with #72 first. The live staging preflight with
Origin: https://staging.52zzx.topstill returns 403, so the order has to be Gateway deploy → this workflow, otherwise it fails at the final step. - The
netty.version4.2.19 bump in the poms is not in the pinned images, because those digests were built from 6de8d2b. That's fine if it's only for the fs/Testcontainers scan. If the runtime needs it, re-run publish-service-images after merge and update the digests. - The
restartCount==0check is strict. A single restart while RDS/RabbitMQ warms up will fail the run even when the pod later becomes healthy. Acceptable, but worth knowing when you read a failure.
I'll approve once the port fix lands, and CI stays green.
| --approved-spend-usd "$APPROVED_STAGING_SPEND_USD" --require-gate \ | ||
| > "$RUNNER_TEMP/business-services-budget-gate.log" | ||
| python3 scripts/staging/verify_service_digests.py > "$RUNNER_TEMP/service-images.json" | ||
| port=18101 |
There was a problem hiding this comment.
This assignment doesn't survive into the Verify step (separate shell). Remove it here:
| port=18101 |
| env: | ||
| GATEWAY_ISSUER: ${{ vars.GATEWAY_ISSUER }} | ||
| run: | | ||
| set -euo pipefail |
There was a problem hiding this comment.
…and set it where it's used (otherwise set -u aborts at "$port:8080"):
| set -euo pipefail | |
| set -euo pipefail | |
| port=18101 |
|
M1 upstream release readiness (Specification / Formulation / Label Workflow) for this release Images: ready. The digests pinned in
Business data for a real Impact event: ready in the images. Accounts are the gap.
|
|
M1's blocking shell-scope finding is fixed on exact head |
|
Correction to my readiness note above. CVE-2026-77214 (libexpat 2.8.5-r0, HIGH) was published after run 39050128320. Publish run 39403988220 failed the Trivy image gate for specification, formulation and label-workflow on it, so the three digests pinned here ( The plan:
|
|
@SHJ-SHJ0128 The Please merge main into this branch and pin:
Merging main conflicts in After the exact-head CI is green, I'll approve. Deployment still waits on #89 → Plan → Apply → Gateway. |
|
|
@hxj04121-lab @rcncai Re-review requested for exact head Merged current main Final image digests:
Local YAML parsing, all 54 Helm checks, service-digest/RDS/release-inventory/window/budget/Keycloak tests, runtime-secret and MySQL workflow checks passed. Exact-head Actions passed: CI, Terraform validation, Helm, M5 offline, and Compliance checks/security/kind. Terraform Plan and publish/deploy remain skipped. No cloud or staging write was performed. M1/M2, please review this exact head; no automatic merge was performed. |
rcncai
left a comment
There was a problem hiding this comment.
M2 re-review at c12b0b5. No blocking source findings. The main merge and three refreshed service digests match the successful release receipts and unchanged image-security gates from e9af8e2. Compliance correctly retains the accepted 73aac3f0 digest and its original provenance. The Verify-step port initialization, kind-only digest override, and rendering regressions remain intact. All eight current-head workflows pass. This is source approval only; protected Plan and deployment remain separate gated steps, and no staging acceptance is established.
hxj04121-lab
left a comment
There was a problem hiding this comment.
M1 Code Owner approval @ c12b0b5. The port is initialised in the Verify step. The four digests match publish run 39409889122 (spec 83b705f0…, formulation 658757b2…, label-workflow 768c1f7e…) and #92 (compliance 73aac3f0…). The main merge resolved cleanly and the exact-head CI is green. Release order is unchanged: #89 → Plan → Apply → Gateway → this workflow.
Summary
Adds the protected, main-only Staging release workflow for specification, formulation, compliance, and label-workflow, pinned to the previously verified immutable GHCR digests.
app: keycloakon TCP/8080. Render assertions cover all affected workloads; default-deny and existing isolation remain.Exact-head validation
Current head:
d905a7a42f525106ec1169aab52c85dce4d6f6a1.288bc7bare historical; the exact-head security job now passes without lowering the threshold or suppressing those findings.Operational gates
No deployment, Terraform Plan/Apply, or Kubernetes write was run. No current S3-backed Plan or live cluster acceptance is claimed. M2 approved the code at the current head; independent M1, M3, and M4 reviews are still requested. Before any protected deployment, verify current M1 Environment approval, staging configuration/Secrets, and owner-approved cost/window. The protected Plan and Compliance publish/deploy jobs remain skipped, so this PR does not evidence staging acceptance. Do not merge automatically.