Bootstrapping an Isolated PCI-DSS Payments Environment on GCP

Table of Contents§

• The Problem
• The Solution
  • Isolation at the project level
  • API surface
  • Terraform state, locked down
  • Three identities, three different jobs
  • The network layer, resource by resource
  • Encryption keys
• The Results
• Bootstrapping the Projects
  • Project IDs are a global namespace, not an org-scoped one
  • The bootstrap sequence
  • A freshly created service account is not immediately usable
• Checking Whether Each Layer Was Actually Necessary
  • Customer-managed keys: dropped
  • VPC and Serverless VPC Access connector: kept
• CI/CD Authentication Without a Stored Key
  • The pool and provider
  • The service account and the narrow binding
  • Requesting the token in the pipeline
  • The exchange script
• A Private Artifact Registry
  • Private by default, not private by configuration
  • Who can push, who can pull
  • Vulnerability scanning happens automatically, verify it actually ran
• Removing Default Identity Grants
• Deploy Jobs Added to the Pipeline
• The Load Balancer
• Web Application Firewall
  • How the sidecar works today
  • Cloud Armor
• Static Egress IP
• Database Access Design (Planned, Not Yet Applied)
  • Identity categories
  • Enabling IAM database authentication
  • The application's own connection
  • Running migrations from CI
  • Deploying and switching revisions from CI
  • Rolling back a bad deployment
  • Everyday human access: rw/ro per schema
  • Connecting From Outside the VPC
  • Three Ways to Load a Dump Into a Private-IP Instance
  • Adding a new schema
  • Temporary elevated access (break-glass)
  • What this design gets, once built
  • Application Compatibility Gap
• Application Secrets
  • Classification
  • Secret Manager, shells only
  • Populating values
  • Per-environment secret sets
• Restoring a Dump: Grants and Ownership
  • Why a plain GRANT is not enough
  • Reassigning ownership
  • One sequence, run after every restore
• Connecting the Core Application
  • A callback URL with a stale hardcoded default
  • The reverse direction
  • WAF status: logging, not blocking
  • Closing the direct Cloud Run URL
  • Public-facing Cloud Run services need an explicit invoker grant
  • minScale has two separate annotation namespaces
  • CPU throttling applies even to minScale-kept-warm instances
  • Match the container memory limit to the JVM flags it's running, not the platform default
  • Spring profile files named for a combined environment-platform pair need that exact combined name active
  • Set allow_failure explicitly on jobs that must block the pipeline
• WAF Enforcement Prep
  • Backend service access logging is off by default
  • Exempt trusted internal callers from content-inspection rules
  • Reading preview verdicts
  • Daily digest to Slack
  • First-use IAM, one grant per resource type
• Audit Log Retention

The Problem§

A payments application, referred to here as payment-processor-app, needed a PCI-DSS 4.0 compliant environment on GCP. The company already ran a general-purpose GCP environment for its main product, and payment-processor-app itself already had a working, previously certified PCI-DSS environment on a different cloud provider. Neither of those could be reused directly.

The general-purpose GCP environment could not host the new payments infrastructure. It had no network segmentation controls (no WAF, default firewall rules open to the internet, no encryption-key management enabled), and mixing a PCI-scoped workload into a project that also runs unrelated internal services expands audit scope to everything in that project, not just the payments path.

The existing PCI environment on the other cloud was the reference architecture (network layout, per-environment resource split, encryption key handling), but it could not simply be copied. The task was to reproduce its intent on GCP's own primitives, then migrate the payments workload onto it, not to lift and shift.

The starting constraint: two new, fully isolated GCP projects, one per environment, with no shared network, no shared identity, and no accidental inheritance of the general-purpose environment's existing gaps.

The Solution§

Isolation at the project level§

Two GCP projects, one per environment, both directly under the organization, no intermediate folder:

gcloud projects create <env>-project-id \
  --organization=<org-id> \
  --name="<display name>"

gcloud billing projects link <env>-project-id \
  --billing-account=<billing-account-id>

A folder was considered, for centralizing IAM and org-policy constraints across both projects, and rejected. The isolation properties that matter (separate IAM, separate network, separate Terraform state, a VPC Service Controls perimeter later) do not depend on folder membership, a folder adds an organizational layer for no functional gain here.

One Terraform repository per environment, not one repository with per-environment subdirectories. A mistake in one environment's .tf files cannot apply to the other because there is no shared root module for it to apply through.

API surface§

Every GCP API used by the eventual infrastructure was enabled explicitly, per project, from a fixed list:

APIS=(
  run.googleapis.com                  # serverless compute
  sqladmin.googleapis.com             # managed database
  compute.googleapis.com              # VPC, firewall, load balancing
  vpcaccess.googleapis.com            # serverless-to-VPC connector
  servicenetworking.googleapis.com    # private service networking
  cloudkms.googleapis.com             # encryption keys
  secretmanager.googleapis.com        # application secrets
  artifactregistry.googleapis.com     # container images
  containeranalysis.googleapis.com    # image vulnerability scanning
  logging.googleapis.com
  cloudasset.googleapis.com
  accesscontextmanager.googleapis.com # VPC Service Controls
  iam.googleapis.com
  storage.googleapis.com
)

No API was enabled speculatively. Each line maps to a specific resource type used later. This list itself became a form of documentation: reviewing it answers "what categories of infrastructure exist here" without reading any .tf file.

Terraform state, locked down§

One GCS bucket per environment, created with public access prevention enforced at creation and re-enforced on every update, plus uniform bucket-level access so nothing can be granted through legacy per-object ACLs:

gcloud storage buckets create "gs://<env>-tfstate" \
  --location="<region>" \
  --uniform-bucket-level-access \
  --public-access-prevention

gcloud storage buckets update "gs://<env>-tfstate" \
  --public-access-prevention \
  --versioning

Versioning is on, so a bad terraform apply that corrupts state is recoverable. Verification after creation was not just "does the bucket exist", but an explicit dump of its IAM policy, checked for the absence of allUsers or allAuthenticatedUsers:

gcloud storage buckets get-iam-policy "gs://<env>-tfstate" --format="json(bindings)"

Three identities, three different jobs§

Not one broadly-privileged account. Three service accounts, each scoped to one job:

# read-only discovery: verifies every step below without being able to change anything
gcloud iam service-accounts create readonly-sa --project="${PROJECT_ID}"
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
  --member="serviceAccount:readonly-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/viewer"

# Terraform runtime: starts with nothing but its own state bucket
gcloud iam service-accounts create terraform-sa --project="${PROJECT_ID}"
gcloud storage buckets add-iam-policy-binding "gs://${PROJECT_ID}-tfstate" \
  --member="serviceAccount:terraform-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/storage.objectAdmin"

# application runtime: created later, in the Terraform itself, once the compute
# layer exists (see the Database Access Design section below)

terraform-sa's bucket-only scoping produced an expected failure on the first apply:

Error: Error creating Network: googleapi: Error 403:
Required 'compute.networks.create' permission for
'projects/<project>/global/networks/<vpc-name>', forbidden

The fix was not a blanket roles/editor grant. It was four roles, each tied to a specific resource type the network layer actually creates:

ROLES=(
  roles/compute.networkAdmin            # VPC, subnets, routes, router, NAT
  roles/compute.securityAdmin           # firewall rules
  roles/servicenetworking.networksAdmin # private service networking peering
  roles/vpcaccess.admin                 # serverless VPC connector
)

Every later infrastructure layer (database, secrets, encryption keys, compute) will need its own additional role granted the same way, incrementally, only once that layer is actually being built. The Terraform account's permission set is meant to be a running log of exactly what infrastructure exists, not an upfront guess at everything it might eventually need.

The network layer, resource by resource§

No VM subnet, no cluster subnet, the compute layer is fully serverless. Six resource types, one file:

resource "google_compute_network" "vpc" {
  name                    = "<app>-vpc"
  project                 = var.project_id
  auto_create_subnetworks = false # GCP's default network is auto-mode, not used here
}

# dedicated subnet for the Serverless VPC Access connector, cannot share with anything else
resource "google_compute_subnetwork" "connector" {
  name          = "connector"
  project       = var.project_id
  region        = var.region
  network       = google_compute_network.vpc.id
  ip_cidr_range = cidrsubnet(var.vpc_cidr, 12, 0) # a /28

  log_config {
    aggregation_interval = "INTERVAL_5_SEC"
    flow_sampling        = 1.0
    metadata              = "INCLUDE_ALL_METADATA"
  }
}

# reserved range + peering: lets Cloud SQL hand out a private IP in this VPC
resource "google_compute_global_address" "private_services" {
  name          = "private-services"
  project       = var.project_id
  purpose       = "VPC_PEERING"
  address_type  = "INTERNAL"
  prefix_length = 20
  network       = google_compute_network.vpc.id
  address       = cidrhost(var.vpc_cidr, 4096)
}

resource "google_service_networking_connection" "private_vpc" {
  network                 = google_compute_network.vpc.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_services.name]
}

resource "google_vpc_access_connector" "connector" {
  name    = "run-connector"
  project = var.project_id
  region  = var.region
  subnet {
    name = google_compute_subnetwork.connector.name
  }
  machine_type  = "e2-micro"
  min_instances = 2
  max_instances = 3
}

# deny everything by default, then one explicit allow for internal traffic only
resource "google_compute_firewall" "deny_all_ingress" {
  name      = "deny-all-ingress"
  project   = var.project_id
  network   = google_compute_network.vpc.id
  priority  = 65534
  direction = "INGRESS"
  deny { protocol = "all" }
  source_ranges = ["0.0.0.0/0"]
}

resource "google_compute_firewall" "allow_internal" {
  name      = "allow-internal"
  project   = var.project_id
  network   = google_compute_network.vpc.id
  priority  = 1000
  direction = "INGRESS"
  allow { protocol = "tcp" }
  allow { protocol = "udp" }
  allow { protocol = "icmp" }
  source_ranges = [var.vpc_cidr]
}

resource "google_compute_router" "router" {
  name    = "router"
  project = var.project_id
  region  = var.region
  network = google_compute_network.vpc.id
}

resource "google_compute_router_nat" "nat" {
  name                               = "nat"
  project                            = var.project_id
  router                             = google_compute_router.router.name
  region                             = var.region
  nat_ip_allocate_option             = "AUTO_ONLY"
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"

  log_config {
    enable = true
    filter = "ERRORS_ONLY"
  }
}

Verify no 0.0.0.0/0 allow rule exists (the general-purpose environment's default firewall had default-allow-ssh and default-allow-rdp open to the internet, this configuration has neither):

gcloud compute firewall-rules list --project="${PROJECT_ID}" --format="table(name,sourceRanges.list(),allowed[].map().firewall_rule().list())"

Encryption keys§

Two HSM-backed Cloud KMS keys per environment, not one shared key for everything: one for the managed database's disk encryption, one for the secrets manager's customer-managed encryption. Each on a 90-day rotation period.

resource "google_kms_key_ring" "app" {
  name     = "<app-name>"
  project  = var.project_id
  location = var.region
}

resource "google_kms_crypto_key" "sql" {
  name            = "sql-cmek"
  key_ring        = google_kms_key_ring.app.id
  rotation_period = "7776000s" # 90 days
  version_template {
    algorithm        = "GOOGLE_SYMMETRIC_ENCRYPTION"
    protection_level = "HSM"
  }

  lifecycle {
    prevent_destroy = true
  }
}

A key ring is a location-scoped grouping of keys, it has no encryption behavior of its own and, unlike a key, was never a candidate for prevent_destroy, GCP has no delete operation for key rings at all, they exist for as long as the project does. The crypto key resource inside it is what actually holds key material.

protection_level = "HSM" means the key material is generated and used inside a FIPS 140-2 Level 3 hardware module and never leaves it in plaintext form, as opposed to the default SOFTWARE protection level, where Google still manages the key but it is not backed by dedicated hardware. algorithm = "GOOGLE_SYMMETRIC_ENCRYPTION" is AES-256-GCM under the hood, the only symmetric option Cloud KMS currently offers for this purpose.

rotation_period does not re-encrypt anything that already exists. On the schedule, GCP generates a new primary key version and every new encrypt operation uses it, but every version that ever existed stays enabled and able to decrypt data encrypted under it, until a version is explicitly disabled or scheduled for destruction. A downstream resource like a managed database is not automatically re-encrypted when the key rotates; it just keeps using whichever version encrypted it originally.

Attaching this key to a downstream resource is a two-part requirement, not just a field. A managed database's encryption_key_name (or equivalent field on whatever service consumes the key) takes the crypto key's full resource ID, but the database's own Google-managed service agent also needs roles/cloudkms.cryptoKeyEncrypterDecrypter granted on that specific key, or the database creation fails at the point it tries to use the key, not at the point the key itself is created. The key existing and the key being usable by a given resource are two separate grants.

lifecycle.prevent_destroy is enforced entirely on the Terraform side, before any API call is made. It stops terraform destroy and any plan that would replace the resource, but it has no effect on someone deleting or disabling key versions directly through the console or gcloud, outside of Terraform. It protects against this tool doing it, not against every possible path to doing it.

A third key was planned for application-level envelope encryption of card numbers, used directly by the application code through the KMS encrypt/decrypt API rather than wired into any infrastructure resource. It was written, then commented out rather than deleted: the payments application does not currently store card numbers in its database, so the key has nothing to protect yet. The commented block stays in place with a note explaining why, so enabling it later is a one-line uncomment instead of rediscovering the design from scratch. Deleting unused-but-planned configuration in favor of "add it back when needed" throws away the reasoning along with the code.

Granting the Terraform account permission to create these keys followed the same incremental pattern as the network layer: nothing was pre-granted, the apply failed on a missing cloudkms.admin permission, and that single role was added at that point, not earlier.

CMEK key resources need lifecycle.prevent_destroy = true. Cloud KMS does not support deleting a key once created, but Terraform can still schedule every version of a key for destruction, through an explicit destroy or an unrelated change that forces the resource to be replaced, making anything encrypted with that key permanently unrecoverable. Applied to both keys, and to the commented-out third key as well:

lifecycle {
  prevent_destroy = true
}

The Results§

• Two GCP projects exist, fully separated from the general-purpose environment: no shared VPC, no shared IAM bindings, no shared Terraform state.
• Both Terraform state buckets are confirmed private: public access prevention enforced, versioning on, IAM policy verified to contain no public bindings.
• The network layer applies cleanly under a Terraform service account holding only the four roles it actually needs, not a broad administrative role.
• Access is split across three identities by purpose (read-only audit, infrastructure provisioning, eventual application runtime), rather than one identity reused everywhere out of convenience.
• Two HSM-backed CMEK keys exist per environment (database, secrets), both with prevent_destroy set, confirmed present in both projects after apply.
• What is deliberately not yet built: the load balancer and web application firewall, an explicit VPC Service Controls perimeter enforcing the isolation from the general-purpose environment at the network-policy level (not just by not having wired anything up), and alerting on top of the log retention below. The project list of remaining work was derived directly from a real PCI-DSS 4.0 audit question set collected during the prior certification on the other cloud, not from a generic compliance checklist, so each remaining item maps to a specific question that will need a specific piece of evidence, not just "compliant" as an adjective.

Build order matters here: google_service_networking_connection (the private services peering) has to exist before Cloud SQL is created with a private IP, not after. Cloud SQL's private-IP option has no effect if the peering isn't there yet, the dependency runs network before database, and depends_on in the database resource should say so explicitly rather than relying on implicit resource-reference ordering.

Bootstrapping the Projects§

What follows is the bootstrap sequence itself: not Terraform, the steps that have to exist before any Terraform can run at all.

Project IDs are a global namespace, not an org-scoped one§

The first attempt at a new project ID failed:

ERROR: (gcloud.projects.create) Project creation failed. The project ID
you specified is already in use by another project. Please try an
alternative ID.

A GCP project ID is unique across every GCP customer that has ever existed, not scoped to an organization. A short, generic ID can already belong to an unrelated account on the other side of the world, with no dispute process and no way to reclaim it. Checking whether a candidate ID is actually available before deciding on a naming pattern:

gcloud projects describe <candidate-project-id> --account=<account>

A "permission denied" response means it exists and belongs to someone else, not that it is available. A project ID, once created, is also permanent. The display name can be changed freely (gcloud projects update <project-id> --name="New Name"), the ID itself cannot, the only way to change it is creating a new project and migrating.

The bootstrap sequence§

Three scripts, run in order, each safe to re-run if interrupted partway:

APIs and service identities, per project:

APIS=(
  cloudresourcemanager.googleapis.com
  run.googleapis.com
  sqladmin.googleapis.com
  compute.googleapis.com
  vpcaccess.googleapis.com
  servicenetworking.googleapis.com
  cloudkms.googleapis.com
  secretmanager.googleapis.com
  artifactregistry.googleapis.com
  containeranalysis.googleapis.com
  logging.googleapis.com
  cloudasset.googleapis.com
  accesscontextmanager.googleapis.com
  iam.googleapis.com
  storage.googleapis.com
  cloudidentity.googleapis.com
)

gcloud services enable "${APIS[@]}" --project="${PROJECT_ID}"

gcloud beta services identity create \
  --service=sqladmin.googleapis.com \
  --project="${PROJECT_ID}"

The second command matters on its own. Cloud SQL's per-project service identity is normally created automatically the first time the Cloud SQL Admin API is actually used, but that did not happen reliably here, and its absence surfaces later as an opaque failure during instance creation, "Per-Product Per-Project Service Account is not found," with nothing in the error pointing back to a missing identity. Creating it explicitly during bootstrap avoids debugging that message a second time.

The Terraform state bucket, private from the moment it exists:

gcloud storage buckets create "gs://${PROJECT_ID}-tfstate" \
  --location="${REGION}" \
  --uniform-bucket-level-access \
  --public-access-prevention

gcloud storage buckets update "gs://${PROJECT_ID}-tfstate" \
  --public-access-prevention \
  --versioning

Two service accounts, not one, before any Terraform runs:

• A read-only account (roles/viewer), used to verify every later step from outside the write path, without any ability to change anything.
• A Terraform runtime account, granted nothing but roles/storage.objectAdmin scoped to its own state bucket, no project-level permissions yet. Every permission it needs beyond that gets added later, one role at a time, only once the layer of infrastructure that needs it is actually being written.

gcloud iam service-accounts create readonly-sa --project="${PROJECT_ID}"
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
  --member="serviceAccount:readonly-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/viewer"

gcloud iam service-accounts create terraform-sa --project="${PROJECT_ID}"
gcloud storage buckets add-iam-policy-binding "gs://${PROJECT_ID}-tfstate" \
  --member="serviceAccount:terraform-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/storage.objectAdmin"

A freshly created service account is not immediately usable§

Granting a role to a service account in the same script, right after creating it, failed intermittently:

ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT:
Service account <email> does not exist.

The account exists, gcloud iam service-accounts describe confirms it, but the IAM policy service has not caught up yet. The fix is a short poll loop before attempting the grant, rather than a fixed sleep:

for i in $(seq 1 12); do
  if gcloud iam service-accounts describe "${SA_EMAIL}" --project="${PROJECT_ID}" >/dev/null 2>&1; then
    break
  fi
  sleep 5
done

Twelve attempts at five seconds covers the delay in every case seen so far, without hardcoding a single long sleep that is either too short under load or wastefully long the rest of the time.

Checking Whether Each Layer Was Actually Necessary§

Customer-managed keys: dropped§

Cloud SQL and Secret Manager are encrypted at rest either way, default platform-managed key or customer-managed key (CMEK), the data is never unencrypted. PCI-DSS's requirement for protecting stored data asks for strong cryptography, which the default managed key already provides. CMEK does not change whether that requirement is met, it adds: a self-defined rotation schedule, the ability to revoke the key independent of the platform, a separate audit trail of key usage.

CMEK's cost: the KMS API, a key ring and its keys, a service-agent IAM grant, and a lifecycle.prevent_destroy guard that exists only because CMEK introduces a failure mode (destroying the key permanently loses the data) the default key doesn't have.

Removed all of it. The resources deleted from the Terraform:

google_kms_key_ring.app
google_kms_crypto_key.sql
google_kms_crypto_key.secrets
google_kms_crypto_key_iam_member.sql_service_agent

And the encryption_key_name argument on google_sql_database_instance.pg was dropped, reverting the instance to the default managed key. Reintroducing CMEK later, if a specific contract or assessor requirement calls for it, is additive: the same four resource blocks and one argument, nothing else in the config has to change.

VPC and Serverless VPC Access connector: kept§

The private network exists for one reason: it lets the compute layer reach the database over a private IP instead of a public one. Without it, the compute layer needs no network presence at all, it can reach the public internet on its own.

The alternative, a public-IP database restricted by an authorized-networks allowlist plus enforced TLS, is also defensible under PCI-DSS Requirement 1's segmentation intent, but weaker: a reachable public endpoint exists, gated by an IP list and a TLS handshake instead of not being reachable at all. It's also not fully achievable as described, the compute layer's default outbound traffic has no fixed IP, so the allowlist would either stay effectively open or need its own static-egress-IP infrastructure to function at all, which removes most of the simplicity that was the point of skipping the private network in the first place.

Kept the private network. Two Graphviz diagrams trace the actual request path for both options, with-vpc-connector.dot and without-vpc-connector.dot, plus a combined side-by-side version, vpc-connector-comparison.dot.

CI/CD Authentication Without a Stored Key§

No downloaded service account key anywhere in the deploy pipeline. GitLab authenticates to GCP through Workload Identity Federation (WIF): a pool and provider that trust GitLab.com's OIDC tokens, plus a service account that pipeline jobs are allowed to impersonate, never a static credential.

The pool and provider§

resource "google_iam_workload_identity_pool" "gitlab" {
  workload_identity_pool_id = "gitlab-pool"
  project                   = var.project_id
}

resource "google_iam_workload_identity_pool_provider" "gitlab" {
  workload_identity_pool_id          = google_iam_workload_identity_pool.gitlab.workload_identity_pool_id
  workload_identity_pool_provider_id = "gitlab-provider"
  project                            = var.project_id

  attribute_mapping = {
    "google.subject"       = "assertion.project_id"
    "attribute.path"       = "assertion.project_path"
    "attribute.project_id" = "assertion.project_id"
  }
  attribute_condition = "attribute.path.startsWith('<gitlab-group>/')"

  oidc {
    issuer_uri = "https://gitlab.com"
  }
}

attribute_condition here is intentionally broad, any repository under the group can present a token to this provider at all. It is not the access-control boundary, it just filters which tokens the provider will bother parsing. The actual restriction happens next.

The service account and the narrow binding§

resource "google_service_account" "deploy" {
  account_id   = "deploy-sa"
  project      = var.project_id
  display_name = "GitLab CI deploy"
}

resource "google_project_iam_member" "deploy_run_admin" {
  project = var.project_id
  role    = "roles/run.admin"
  member  = "serviceAccount:${google_service_account.deploy.email}"
}

resource "google_project_iam_member" "deploy_artifact_writer" {
  project = var.project_id
  role    = "roles/artifactregistry.writer"
  member  = "serviceAccount:${google_service_account.deploy.email}"
}

resource "google_project_iam_member" "deploy_sql_client" {
  project = var.project_id
  role    = "roles/cloudsql.client"
  member  = "serviceAccount:${google_service_account.deploy.email}"
}

# the actual access control: only these specific repos can become deploy-sa
resource "google_service_account_iam_member" "deploy_wif_binding" {
  for_each = toset([
    "<gitlab-group>/app-repo",
    "<gitlab-group>/deployment-scripts",
  ])
  service_account_id = google_service_account.deploy.name
  role                = "roles/iam.workloadIdentityUser"
  member              = "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.gitlab.name}/attribute.path/${each.value}"
}

deploy-sa gets run.admin, artifactregistry.writer, cloudsql.client, nothing else, no owner, no editor. Widening the pool's trust and narrowing the binding, instead of putting all restriction in one place, means adding a new repository later is a one-line addition to the for_each set, not a redesign of the provider.

Requesting the token in the pipeline§

deploy:
  id_tokens:
    GCP_OIDC_TOKEN:
      aud: "//iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/gitlab-pool/providers/gitlab-provider"
  script:
    - ./gcloud-auth.sh
    - gcloud run services replace values-stage.yaml --project "$GCP_PROJECT_ID" --region "$GCP_REGION"

The aud field has to exactly match the provider's resource path, GitLab embeds it as the token's audience claim, and GCP's STS rejects a token whose audience does not match the provider it is presented to.

The exchange script§

This is the actual mechanism, not a black box. Two HTTP calls, no library required:

#!/bin/bash
set -e

WIF_AUDIENCE="//iam.googleapis.com/projects/${GCP_WIP_PROJECT_NUMBER}/locations/global/workloadIdentityPools/gitlab-pool/providers/gitlab-provider"
GCP_SERVICE_ACCOUNT="deploy-sa@${GCP_PROJECT_ID}.iam.gserviceaccount.com"

# step 1: exchange GitLab's signed JWT for a Google federated token
STS_RESPONSE=$(curl -s -X POST https://sts.googleapis.com/v1/token \
  --header "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "audience=${WIF_AUDIENCE}" \
  --data-urlencode "grant_type=urn:ietf:params:oauth:grant-type:token-exchange" \
  --data-urlencode "requested_token_type=urn:ietf:params:oauth:token-type:access_token" \
  --data-urlencode "scope=https://www.googleapis.com/auth/cloud-platform" \
  --data-urlencode "subject_token=${GCP_OIDC_TOKEN}" \
  --data-urlencode "subject_token_type=urn:ietf:params:oauth:token-type:jwt")

FEDERATED_TOKEN=$(echo "$STS_RESPONSE" | jq -r .access_token)

# step 2: use the federated token to impersonate the actual service account
IAM_RESPONSE=$(curl -s -X POST \
  "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${GCP_SERVICE_ACCOUNT}:generateAccessToken" \
  -H "Authorization: Bearer ${FEDERATED_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"scope": ["https://www.googleapis.com/auth/cloud-platform"]}')

ACCESS_TOKEN=$(echo "$IAM_RESPONSE" | jq -r .accessToken)

export CLOUDSDK_AUTH_ACCESS_TOKEN="${ACCESS_TOKEN}"

$GCP_OIDC_TOKEN is the ID token GitLab injected as an environment variable from the id_tokens block above, it never touches disk unless the script writes it there deliberately. $CLOUDSDK_AUTH_ACCESS_TOKEN, once exported, is picked up automatically by every subsequent gcloud call in the job. For docker push, pipe the same access token into docker login -u oauth2accesstoken --password-stdin.

That access token is valid for about an hour. Nothing generated in this entire exchange outlives the job that requested it.

A Private Artifact Registry§

One Docker repository per environment, holding every application's images, distinguished by image name rather than by a separate repository each:

resource "google_artifact_registry_repository" "app_repo" {
  repository_id = "app-repo"
  project       = var.project_id
  location      = var.region
  format        = "DOCKER"
  mode          = "STANDARD_REPOSITORY"
}

Private by default, not private by configuration§

Unlike a GCS bucket, an Artifact Registry repository has no public-read concept until something explicitly grants one. There is no equivalent of a bucket's legacy ACLs to accidentally leave open, access is IAM-only from creation, and the only way to make a repository public is to bind allUsers or allAuthenticatedUsers to a role on it, which this configuration never does. Confirm that directly rather than assuming it:

gcloud artifacts repositories get-iam-policy app-repo \
  --location="${REGION}" \
  --project="${PROJECT_ID}"

An empty or fully internal binding list is the expected result. Any allUsers entry here would be a finding, not a default.

Who can push, who can pull§

deploy-sa (from the CI/CD section above) has roles/artifactregistry.writer at the project level, that's what lets the pipeline push new images. Pulling, at deploy time, is handled by Cloud Run's own per-project service agent, which has implicit read access to Artifact Registry repositories in the same project without a separate grant. No IAM binding was added for pulling specifically, if gcloud run deploy or services replace ever fails on an image pull permission error, check the Cloud Run service agent's own bindings before adding a new one, the existing default is normally sufficient.

Vulnerability scanning happens automatically, verify it actually ran§

containeranalysis.googleapis.com, enabled during the original API bootstrap, turns on automatic vulnerability scanning for every image pushed to Artifact Registry. Nothing in the repository resource itself configures this, it's a project-level API flag, not a repository setting. Confirm scan results exist for a pushed image, don't just assume the flag did something:

gcloud artifacts docker images list \
  "${REGION}-docker.pkg.dev/${PROJECT_ID}/app-repo" \
  --include-vulnerabilities

An image with no vulnerability data listed either hasn't finished scanning yet (scans run asynchronously after push, not before it completes) or was pushed before the API was enabled. Re-push or wait a few minutes before treating an empty result as a clean bill of health.

Removing Default Identity Grants§

Checked whether the default compute service account itself was still carrying privilege left over from GCP's older automatic-grant behavior, where every project used to receive a default compute identity with roles/editor attached the moment the Compute Engine API was enabled, whether or not anything ever used that identity:

gcloud projects get-iam-policy "${PROJECT_ID}" \
  --flatten="bindings[].members" \
  --filter="bindings.members:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com"

It was, in both projects. Before removing it, checked what actually depended on that identity rather than assuming the answer: zero Compute Engine instances, zero Cloud Run services running as it, the Cloud Functions API never enabled in either project. Nothing used it.

gcloud projects remove-iam-policy-binding "${PROJECT_ID}" \
  --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
  --role="roles/editor"

Also considered enforcing an org policy constraint (constraints/iam.automaticIamGrantsForDefaultServiceAccounts) so this could not silently recur for some future default service account. Setting it failed. The account applying Terraform held project Owner, and Owner does not include the permission needed to set an org policy, orgpolicy.policy.set, that permission lives only in a role Google restricts to org- or folder-level binding, not project-level, so unblocking it would have meant either a broader org-wide grant than these two projects needed, or standing up a custom IAM role just to flip one boolean. Dropped it. The actual requirement, an over-privileged identity existing right now, was already satisfied by removing the existing grant. Guarding against a hypothetical future recurrence was a nice-to-have, not a requirement, and not worth the privilege escalation it would have taken to wire up.

Deploy Jobs Added to the Pipeline§

The existing deploy pipeline (the one from the prior PCI-DSS environment on the other cloud) was left in place and untouched. New jobs were added alongside it, running the same build in parallel, deploying to the new environment through the CI/CD authentication mechanism described above. Every one of the new jobs is marked allow_failure: true:

deploy_new_env_stage:
  stage: deploy
  id_tokens:
    GCP_OIDC_TOKEN:
      aud: "//iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/gitlab-pool/providers/gitlab-provider"
  script:
    - git clone <deployment-scripts-repo-url>
    - bash deployment-scripts/gcloud-auth.sh
    - bash deployment-scripts/app/deploy-stage-gcp.sh
  allow_failure: true

This is a new, still-untested path. A failure in it must not block a release through the path already running in production, hence allow_failure: true on every job in it.

What follows are the failures hit getting this working end to end, in order.

The builder image had no gcloud CLI. The container image used for the new jobs was already used successfully elsewhere in the same pipeline, for jobs that only ever ran Docker and Maven. It failed immediately on the first gcloud call:

deploy-stage-gcp.sh: line 17: gcloud: command not found

An image being proven elsewhere in the pipeline says nothing about which tools it contains. Fixed by pointing the new jobs at a different image already in the same registry, one that already had both Docker and the Cloud SDK installed, instead of maintaining a second custom image or patching a Dockerfile that would need a separate manual rebuild step outside the pipeline.

A push job talked to Docker with no DOCKER_HOST set. The job that builds and saves an image as a .tar artifact declares a Docker-in-Docker service and sets DOCKER_HOST/TLS variables to point the client at it. The next job, which loads that .tar and pushes it onward, also declares the same DinD service, but never set those variables:

variables:
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_TLS_VERIFY: 1
  DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"

variables: blocks do not carry across jobs implicitly, needs: only orders execution and passes artifacts, it does not merge configuration. extends: does merge variables: blocks between a hidden template job and the jobs that extend it, which is why the fix lives in the shared template, not duplicated per job.

The migration job's Cloud Run Job spec failed to parse:

ERROR: (gcloud.run.jobs.replace) Failed to parse value(s) in protobuf [Job]:
  Job.spec.template.spec.template.metadata

The YAML had metadata.annotations (the VPC connector and Cloud SQL instance annotations) nested one level too deep, under spec.template.spec.template, when the Cloud Run Jobs API expects them at spec.template.metadata. A Cloud Run Job (Execution wrapping a Task template) nests one level deeper than a Cloud Run Service (Revision template directly), and the annotation placement from a working Service spec does not carry over unchanged. Fixed by moving the metadata block up one level and adding the taskCount field the API also expects at that same level.

First-ever deploy of a new service failed on its traffic block:

ERROR: (gcloud.run.services.replace) spec.traffic.percent: traffic percentage adds to 0, should be 100

The deploy spec used a percent: 0-then-promote traffic pattern, deploy the new revision with no traffic, then a second command shifts 100% onto it once the deploy itself succeeds. That pattern only works when a prior revision already exists to hold the other 100% of traffic. On a from-scratch service creation, with no prior revision, there is nothing to hold the remaining 100%, and the total must already equal 100 in the initial spec. Fixed by setting percent: 100 directly, since the deploy script already runs an unconditional traffic-promotion step right after regardless, that pattern provided no actual canary gate as written, only a bug on first deploy.

The migration job denied iam.serviceaccounts.actAs on its own deploying account:

Permission 'iam.serviceaccounts.actAs' denied on service account deploy-sa@<project>.iam.gserviceaccount.com

The migration job runs as the deploying account itself, not a separate runtime identity. GCP does not imply that a principal can act as itself, that grant has to exist explicitly like any other:

resource "google_service_account_iam_member" "deploy_can_act_as_self" {
  service_account_id = google_service_account.deploy.name
  role                = "roles/iam.serviceAccountUser"
  member              = "serviceAccount:${google_service_account.deploy.email}"
}

Every one of these surfaced only by running the pipeline end to end against a real project, none of them from reading the Terraform or the YAML.

The Load Balancer§

A second Cloud Run service shares this project alongside payment-processor-app. A bare Cloud Run domain mapping is one domain to one service, it cannot route by host or path to two different backends, and a web application firewall attaches to a real load balancer, not to a domain mapping. Both of those, independently, ruled out skipping a real load balancer.

Each Cloud Run service gets its own regional serverless NEG:

resource "google_compute_region_network_endpoint_group" "service_a" {
  name                  = "service-a-neg"
  project               = var.project_id
  region                = var.region
  network_endpoint_type = "SERVERLESS"
  cloud_run {
    service = "service-a"
  }
}

resource "google_compute_region_network_endpoint_group" "service_b" {
  name                  = "service-b-neg"
  project               = var.project_id
  region                = var.region
  network_endpoint_type = "SERVERLESS"
  cloud_run {
    service = "service-b"
  }
}

The cloud_run.service field is a name, a string, not a reference to a Terraform-managed resource. The Cloud Run service itself does not have to be declared anywhere in this Terraform for the NEG to apply cleanly, it only has to already exist under that name in the project, which it does here from the deploy pipeline above, Terraform never created it.

One global backend service per NEG, one URL map routing by hostname, not by path, matching how the two services are actually addressed:

resource "google_compute_backend_service" "service_a" {
  name                  = "service-a-backend"
  project               = var.project_id
  protocol              = "HTTPS"
  load_balancing_scheme = "EXTERNAL_MANAGED"
  backend {
    group = google_compute_region_network_endpoint_group.service_a.id
  }
}

resource "google_compute_url_map" "lb" {
  name            = "payments-lb"
  project         = var.project_id
  default_service = google_compute_backend_service.service_b.id

  host_rule {
    hosts        = [var.service_a_domain]
    path_matcher = "service-a"
  }
  path_matcher {
    name            = "service-a"
    default_service = google_compute_backend_service.service_a.id
  }
}

One Google-managed certificate covers both hostnames, one target HTTPS proxy, one global static IP, one forwarding rule on port 443. A second forwarding rule on port 80, pointed at a URL map whose only job is an unconditional HTTPS redirect, so nothing is ever served in plaintext:

resource "google_compute_url_map" "redirect" {
  name    = "payments-http-redirect"
  project = var.project_id
  default_url_redirect {
    https_redirect = true
    strip_query    = false
  }
}

The managed certificate stays in a provisioning state until DNS for both hostnames actually resolves to the load balancer's static IP. That state is expected, not a failure, and it can sit there for a while if DNS is managed somewhere outside this project's control.

What is not yet done: both Cloud Run services still accept traffic directly on their own default run.app URLs, bypassing the load balancer entirely. Locking that down means setting ingress to internal-and-load-balancer-only on each service, a setting on the deploy pipeline's own service spec, not on anything in this Terraform, and it has to wait until the load balancer and its DNS are confirmed working end to end, or it locks out the only path in.

Web Application Firewall§

The second service (the customer-facing frontend behind the load balancer above) runs its WAF as a sidecar container in the same pod on the existing environment, not as a platform feature. Replaced here with a load-balancer-level policy instead, worth documenting the sidecar mechanism first, since the replacement only makes sense in contrast to it.

How the sidecar works today§

Each pod runs two containers, not one: the application container serving static files on its own port, and a second container running nginx built on ModSecurity v3 with the OWASP Core Rule Set, listening on a separate port. The pod's Service exposes both ports, but ingress is configured to route all external traffic to the WAF container's port, never to the application container's port directly:

paths:
  - path: /
    pathType: Prefix
    backend:
      service:
        name: frontend-service
        port:
          number: 8080 # the WAF sidecar, not the app

The WAF container's own nginx config proxies whatever ModSecurity lets through back to the application container on localhost, both containers share a pod network namespace, so no service discovery is needed for that hop:

server {
  listen 8080;
  location / {
    proxy_pass http://localhost:80;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

The WAF image itself is built from an upstream owasp/modsecurity-crs base image (nginx plus ModSecurity plus the Core Rule Set already compiled in), not compiled from source in this project's own build. An earlier version of the same image did compile ModSecurity from source against a plain nginx base, that Dockerfile is still in the repository but unused, superseded by the simpler upstream-base version once one became available with the same rule set already built in.

Rule enforcement mode (SecRuleEngine On, blocking rather than log-only) is set once, in the image, not per-deploy. Response security headers (CSP, HSTS, X-Content-Type-Options) are set separately, by the ingress controller's own annotations, not by the WAF container, ModSecurity inspects and blocks requests, it does not add response headers.

Cloud Armor§

Cloud Armor replaces the sidecar container entirely, not just the rule engine inside it. One policy, one resource, referenced from both backend services behind the load balancer built above:

resource "google_compute_security_policy" "payments" {
  name    = "payments-waf"
  project = var.project_id

  rule {
    action      = "allow"
    priority    = 2147483647
    description = "default allow, overridden by rules below"
    match {
      versioned_expr = "SRC_IPS_V1"
      config {
        src_ip_ranges = ["*"]
      }
    }
  }
}

Every google_compute_security_policy needs exactly one rule at the lowest possible priority (2147483647, the maximum int32) that matches everything, the policy's implicit default. Every rule added after this one sits at a lower priority number, evaluated first, the numerically lowest priority wins.

The preconfigured OWASP-category rules, one google_compute_security_policy_rule per category, expressed as a for_each over a local map rather than nine near-identical resource blocks:

locals {
  waf_preconfigured_rules = {
    sqli               = { priority = 1000, expr = "sqli-v33-stable" }
    xss                = { priority = 1001, expr = "xss-v33-stable" }
    lfi                = { priority = 1002, expr = "lfi-v33-stable" }
    rfi                = { priority = 1003, expr = "rfi-v33-stable" }
    rce                = { priority = 1004, expr = "rce-v33-stable" }
    protocol_attack    = { priority = 1005, expr = "protocolattack-v33-stable" }
    scanner_detection  = { priority = 1006, expr = "scannerdetection-v33-stable" }
    session_fixation   = { priority = 1007, expr = "sessionfixation-v33-stable" }
    method_enforcement = { priority = 1008, expr = "methodenforcement-v33-stable" }
  }
}

resource "google_compute_security_policy_rule" "preconfigured" {
  for_each        = local.waf_preconfigured_rules
  security_policy = google_compute_security_policy.payments.name
  priority        = each.value.priority
  preview         = true
  action          = "deny(403)"
  description     = each.key
  match {
    expr {
      expression = "evaluatePreconfiguredExpr('${each.value.expr}')"
    }
  }
}

Each evaluatePreconfiguredExpr string names one rule group Google maintains, adapted from the same OWASP Core Rule Set the sidecar runs. preview = true on every one of them means the rule evaluates and logs a match, it does not block, that flag is what makes this a log-only rollout rather than a live cutover.

Rate limiting, the one capability the sidecar never had at all:

resource "google_compute_security_policy_rule" "rate_limit" {
  security_policy = google_compute_security_policy.payments.name
  priority        = 2000
  preview         = true
  action          = "throttle"
  description     = "rate limit"
  match {
    versioned_expr = "SRC_IPS_V1"
    config {
      src_ip_ranges = ["*"]
    }
  }
  rate_limit_options {
    conform_action = "allow"
    exceed_action  = "deny(429)"
    enforce_on_key = "IP"
    rate_limit_threshold {
      count        = 100
      interval_sec = 60
    }
  }
}

Attached to both backend services with one field each, nothing else about them changes:

resource "google_compute_backend_service" "service_a" {
  # ...
  security_policy = google_compute_security_policy.payments.id
}

Applying it the first time failed, not on the policy itself, on attaching it:

Error: Error setting Backend Service security policy: googleapi: Error 400:
Security policy GLOBAL:0/PROJECT:<project-number>/SECURITY_POLICY:<policy-id>
is not in a ready state., invalid

The policy reports created successfully to Terraform, the API call returns, but a policy carrying this many rules takes a short window to actually finish provisioning across Google's edge network before anything can reference it. The same propagation-lag shape as a freshly created service account not yet visible to the IAM policy service, earlier in this project, a different resource type, the identical cause: the control plane accepting a create request is not the same moment as the resource being usable everywhere that resource type gets used. Re-running terraform apply a minute later applied cleanly, no code change, nothing to fix, just a wait.

No custom nginx image to build, compile, or keep patched against new ModSecurity CVEs, no second container per pod, no sidecar-specific proxy config to maintain. One policy, attached twice, covering every service behind the load balancer rather than one WAF build per application.

What does not carry over automatically: the response security headers currently set by the ingress controller's annotations have no Cloud Armor equivalent, Cloud Armor only inspects and blocks requests, it does not modify responses. Those headers move into the application's own web server config instead, where a subset of them (CSP, X-Content-Type-Options) already exist as a baseline and need the frame-ancestors allowlist updated for whatever new hostnames the load balancer serves.

What is not yet done: every rule above is preview = true, logging matches, not blocking anything. Switching to enforcing means reviewing those logs against real traffic first, and it still does nothing against direct traffic to a Cloud Run service's own default URL, which bypasses the load balancer, and this policy with it, entirely, until ingress on both services is locked to internal-and-load-balancer-only.

Static Egress IP§

Cloud NAT can allocate its outbound IP two ways: automatically, from Google's own pool, or from a reserved static address supplied explicitly. This project used automatic allocation. A second, older environment in the same organization, built earlier, used a reserved static address instead.

Automatic allocation is not guaranteed to stay static. Google can add, remove, or replace the IPs backing an automatically-allocated NAT gateway as capacity needs change, with no action taken on this project's side and no guarantee any specific address stays assigned. For a payments workload, where an outbound integration may need a fixed IP for allowlisting on the other end, that is not a theoretical problem, it is a name of a hostname a partner will not have if the IP behind it moves.

Fixed by reserving an address and pointing the NAT at it explicitly:

resource "google_compute_address" "nat_gateway_ip" {
  name    = "nat-gateway-ip"
  project = var.project_id
  region  = var.region
}

resource "google_compute_router_nat" "nat" {
  # ...
  nat_ip_allocate_option = "MANUAL_ONLY"
  nat_ips                = [google_compute_address.nat_gateway_ip.self_link]
}

Neither nat_ip_allocate_option nor nat_ips forces replacement of the NAT resource in this provider, the change applies in place. No router or NAT gateway recreation, no gap in outbound connectivity while it takes effect.

Database Access Design (Planned, Not Yet Applied)§

The database itself is multi-tenant at the schema level: one Postgres schema per client organization, referred to here generically as schema_a, schema_b, and so on, currently at least five, growing by several a year as new clients onboard. This section covers the access model designed for it before any of it was built, kept separate from the Results above because none of it is applied yet.

Identity categories§

Four non-human identities, plus human developers through group membership, each with a distinct scope:

Human developers never get an individual database credential. They get added to one of a fixed set of Cloud Identity groups, and the group is what has database privileges.

None of these five identities can do another's job. The CI account that deploys the application cannot touch the database. The CI account that migrates the database cannot deploy anything. This is deliberate: a compromised deploy pipeline should not be able to read data, and a compromised migration job should not be able to change what code is running.

Enabling IAM database authentication§

One instance flag, set once per Cloud SQL instance:

resource "google_sql_database_instance" "pg" {
  # ...
  settings {
    database_flags {
      name  = "cloudsql.iam_authentication"
      value = "on"
    }
  }
}

With this on, Postgres roles can be created with type = "CLOUD_IAM_USER", "CLOUD_IAM_SERVICE_ACCOUNT", or "CLOUD_IAM_GROUP" instead of a password, and Cloud SQL validates a short-lived OAuth token from the connecting identity instead of a stored secret.

The application's own connection§

The application connects through the Cloud SQL connector's built-in IAM support, using its own Cloud Run runtime service account as the database identity:

resource "google_sql_user" "app_runtime" {
  name     = trimsuffix(google_service_account.cloud_run_runtime.email, ".gserviceaccount.com")
  instance = google_sql_database_instance.pg.name
  type     = "CLOUD_IAM_SERVICE_ACCOUNT"
}

No password field exists on this resource, GCP forbids setting one for CLOUD_IAM_SERVICE_ACCOUNT and CLOUD_IAM_USER types outright, it is not just unused. The .gserviceaccount.com suffix has to be trimmed off the service account's email for Postgres specifically, a Postgres identifier length limit, so the registered database user is the shortened form, not the full address used everywhere else in IAM. The Cloud Run service connects with the IAM-auth mode built into the Cloud SQL connector libraries, which mint a short-lived token from the runtime service account's own identity on every connection. There is nothing to rotate and nothing to leak into a Cloud Run environment variable, which is exactly the failure mode found earlier in this project's reference environment, a plaintext database password sitting directly in a running service's configuration.

Running migrations from CI§

The migrations service account authenticates the same way, but through the Cloud SQL Auth Proxy running as a CI job step rather than the application connector library:

migrate:
  stage: migrate
  script:
    - cloud-sql-proxy --auto-iam-authn "$INSTANCE_CONNECTION_NAME" &
    - sleep 2
    - psql "host=127.0.0.1 dbname=$TARGET_DB user=$MIGRATIONS_SA_USER sslmode=disable" \
        -f migrations/schema_a/0042_add_index.sql
  environment:
    name: $CI_ENVIRONMENT_NAME
  rules:
    - if: '$CI_ENVIRONMENT_NAME == "prod"'
      when: manual
    - when: on_success

The proxy's --auto-iam-authn flag generates the IAM token, the CI job never handles a database credential directly, and the proxy must run as the same IAM principal as the database user being connected as, the migrations service account's own identity, not a separate credential passed in. $TARGET_DB selects which of the databases on the instance to run against (dev and stage happen to share one Cloud SQL instance, prod has its own). Anything targeting the prod environment requires a manual approval click in the pipeline before it runs, non-prod runs automatically on every merge to the relevant branch.

$MIGRATIONS_SA_USER is not the service account's full email. For Postgres specifically, Cloud SQL requires the .gserviceaccount.com suffix omitted from the service account email when it is registered as a CLOUD_IAM_SERVICE_ACCOUNT database user, due to a Postgres identifier length limit, so the google_sql_user resource and the connection string both use the shortened form, not the address that shows up anywhere else in IAM.

Deploying and switching revisions from CI§

The deployment service account has no database role at all, only Cloud Run and Artifact Registry permissions:

deploy:
  stage: deploy
  script:
    - gcloud run deploy payment-processor-app \
        --image "$REGION-docker.pkg.dev/$PROJECT_ID/payment-processor-app/api:$CI_COMMIT_SHA" \
        --region "$REGION" \
        --project "$PROJECT_ID" \
        --no-traffic \
        --tag "$CI_COMMIT_SHORT_SHA"
    - gcloud run services update-traffic payment-processor-app \
        --region "$REGION" \
        --project "$PROJECT_ID" \
        --to-latest

The first command deploys a new revision but sends it no traffic (--no-traffic), tagged with the commit SHA so it is individually addressable. The second command is the actual cutover, moving live traffic to the newest revision only after the deploy step succeeded, giving a deliberate point between "new code exists" and "new code is serving requests" rather than one combined step.

Rolling back a bad deployment§

Cloud Run keeps prior revisions available by default, a rollback does not redeploy anything, it repoints traffic:

gcloud run revisions list \
  --service payment-processor-app \
  --region "$REGION" \
  --project "$PROJECT_ID"

gcloud run services update-traffic payment-processor-app \
  --region "$REGION" \
  --project "$PROJECT_ID" \
  --to-revisions "payment-processor-app-00041-abc=100"

This is faster than a redeploy because no build or image pull happens, and it is the same command whether the rollback is triggered by a human or by a CI job reacting to a failed post-deploy health check. If a rollback needs to happen because of a bad migration rather than bad application code, the migration itself is not automatically reverted, migrations in this design are forward-only, a broken migration gets fixed with a new migration, not an automatic down-migration.

Everyday human access: rw/ro per schema§

Two Cloud Identity groups per schema, plus one admin group shared across all schemas on an instance:

resource "google_cloud_identity_group" "db_rw_schema_a" {
  display_name         = "db-rw-schema_a"
  parent               = "customers/${var.customer_id}"
  group_key { id = "db-rw-schema_a@example.com" }
  labels = { "cloudidentity.googleapis.com/groups.security" = "" }
}

resource "google_sql_user" "db_rw_schema_a" {
  name     = "db-rw-schema_a@example.com"
  instance = google_sql_database_instance.pg.name
  type     = "CLOUD_IAM_GROUP"
}

The groups.security label is what makes this a security group rather than a discussion-forum-style Google Group, it is what makes the group eligible to be referenced in an IAM policy or a Cloud SQL user resource at all. It is also immutable once set, a group created without it cannot be converted later, it has to be recreated.

The admin group's google_sql_user is declared the same way, its actual admin-level privilege is a Postgres grant, not a property of the group or the Cloud SQL user resource: cloudsqlsuperuser is Cloud SQL's built-in role for elevated-but-not-true-superuser access, and unlike a BUILT_IN password user, an IAM-authenticated role does not receive it automatically, it has to be granted explicitly with GRANT cloudsqlsuperuser TO "db-admin@example.com"; once, the same way any other schema privilege is granted.

The group existing and being registered as a Cloud SQL user is not the same as it having any privileges. That part is plain SQL, run once when the schema and its groups are created:

GRANT USAGE ON SCHEMA schema_a TO "db-rw-schema_a@example.com";
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA schema_a TO "db-rw-schema_a@example.com";
ALTER DEFAULT PRIVILEGES IN SCHEMA schema_a
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO "db-rw-schema_a@example.com";

GRANT USAGE ON SCHEMA schema_a TO "db-ro-schema_a@example.com";
GRANT SELECT ON ALL TABLES IN SCHEMA schema_a TO "db-ro-schema_a@example.com";
ALTER DEFAULT PRIVILEGES IN SCHEMA schema_a
  GRANT SELECT ON TABLES TO "db-ro-schema_a@example.com";

The ALTER DEFAULT PRIVILEGES statements matter as much as the GRANT statements. Without them, a table created next month by a migration is not covered by last year's grant, and someone has to remember to re-run the grant by hand every time the schema changes shape.

Adding a developer to a schema's read/write group, or removing them, is a single Cloud Identity group membership change, not a database operation:

gcloud identity groups memberships add \
  --group-email "db-rw-schema_a@example.com" \
  --member-email "developer@example.com"

No connection to the database happens for this, no credential is issued or revoked, membership itself is the access control. Group membership changes take up to about 15 minutes to propagate to Cloud SQL, this is not instant, and is worth stating plainly in whatever runbook a developer reads while waiting to get in during an incident. The schema-level GRANT itself, once a group already has a Cloud SQL user, applies immediately, the delay is specifically in Cloud SQL recognizing new or removed group membership, not in privilege changes for a group that is already recognized.

Connecting From Outside the VPC§

The Cloud SQL instance has no public IP. Running the Cloud SQL Auth Proxy from a local machine with default flags fails:

failed to connect to instance: config error: instance does not have IP of type "PUBLIC"

The proxy defaults to the instance's public IP. Fix: --private-ip.

--private-ip alone is not sufficient. --auto-iam-authn controls authentication, not network reachability. A private IP is only routable from inside its own VPC, or a network peered or VPN'd into it. This project has neither. A client outside the VPC with --private-ip set will pass the config check, then time out on the connection itself, no route to the address. Reaching the instance from outside the VPC requires a client that is itself inside it, e.g. a Compute Engine VM in the same network, or a Cloud Shell session attached via a VPC connector.

Three Ways to Load a Dump Into a Private-IP Instance§

Option 1: gcloud sql import from Cloud Storage. Cloud SQL's import mechanism reads the source file from a GCS bucket over Google's internal network path, not over a client TCP connection. VPC reachability from the operator's machine is not a factor. Requires a plain-SQL dump (not the custom pg_dump format), a bucket, and read access on that bucket for the instance's own service account:

resource "google_storage_bucket" "db_dump" {
  name                        = "${var.project_id}-db-dump"
  project                     = var.project_id
  location                    = var.region
  uniform_bucket_level_access = true
  force_destroy               = false
  public_access_prevention    = "enforced"
}

resource "google_storage_bucket_iam_member" "sql_import_reader" {
  bucket = google_storage_bucket.db_dump.name
  role   = "roles/storage.objectViewer"
  member = "serviceAccount:${google_sql_database_instance.pg.service_account_email_address}"
}

Operation itself is one command, no proxy, no tunnel:

gcloud storage cp payments-stage.sql "gs://${BUCKET}/${OBJECT}"
gcloud sql import sql payments-pg "gs://${BUCKET}/${OBJECT}" --database=payments-stage-db

Option 2: a short-lived Compute Engine VM inside the VPC, reached over IAP. The VM is not Terraform-managed, it is created and deleted per use, ephemeral by design. What is Terraform-managed is the persistent prerequisite, a firewall rule scoped to Google's own IAP TCP-forwarding range, not to any real IP an operator connects from:

resource "google_compute_firewall" "allow_iap_ssh" {
  name      = "allow-iap-ssh"
  project   = var.project_id
  network   = google_compute_network.vpc.id
  priority  = 1000
  direction = "INGRESS"
  allow {
    protocol = "tcp"
    ports    = ["22"]
  }
  source_ranges = ["35.235.240.0/20"] # IAP TCP forwarding, not a public range
}

resource "google_project_iam_member" "operator_iap_tunnel" {
  project = var.project_id
  role    = "roles/iap.tunnelResourceAccessor"
  member  = "user:developer@example.com"
}

No public IP on the VM, no bastion host standing by between uses:

gcloud compute ssh restore-vm --zone=europe-west3-a --tunnel-through-iap

psql on the VM connects to the instance's private IP directly, no auth proxy needed, the VM is already inside the network the private IP belongs to.

Option 3: a one-off Cloud Run Job using the same VPC connector the application and the migration job already use. Not yet built, no Cloud Run resource in this project is Terraform-managed. Shape of it, if it were:

resource "google_cloud_run_v2_job" "db_restore" {
  name     = "payments-db-restore"
  project  = var.project_id
  location = var.region

  template {
    template {
      service_account = google_service_account.gitlab_pusher.email
      vpc_access {
        connector = google_vpc_access_connector.connector.id
        egress    = "PRIVATE_RANGES_ONLY"
      }
      containers {
        image = "${var.region}-docker.pkg.dev/${var.project_id}/app-repo/db-restore-tool:latest"
        env {
          name  = "POSTGRES_HOST"
          value = "/cloudsql/${google_sql_database_instance.pg.connection_name}"
        }
      }
      max_retries = 0
    }
  }
}

Most consistent with how the rest of this project already reaches the private instance, the network path is identical to what the application uses. Also the most setup for a task run infrequently: a container image to build and keep current, a job spec to maintain, for something Option 1 does in two commands.

Used: Option 1. No new compute to provision or tear down, no image to maintain, and the private-IP restriction that blocks a direct client connection has no bearing on how Cloud Storage imports work at all.

Adding a new schema§

This happens several times a year, so it is written as a single repeatable procedure, not re-derived each time:

1. Create the schema itself, as part of that client's first migration.
2. Create the two Cloud Identity groups (db-rw-<schema>, db-ro-<schema>) and their matching google_sql_user resources, in Terraform.
3. Run the four GRANT/ALTER DEFAULT PRIVILEGES statements for the new schema, from the bootstrap SQL script, parameterized by schema name.
4. Add whoever needs access to the new groups.

Nothing about the migrations service account, the deployment service account, or the application's own runtime account needs to change when a schema is added: migrations already has DDL across all schemas, the application's runtime account is already granted access at the database level rather than being re-granted per schema (its own access is provisioned once, broadly, precisely because it is a single trusted identity rather than a set of individually revocable human accounts). Only the human-facing rw/ro groups are schema-specific, because those are the ones where "which humans can see this specific schema's data" is the actual question being answered.

Temporary elevated access (break-glass)§

Standing membership in the admin group is not the mechanism for "a developer needs to fix something in prod right now." That is handled through Privileged Access Manager: a time-bound grant into the admin entitlement, requested, approved, automatically expired, and logged, rather than an addition to a group that someone has to remember to later remove.

gcloud pam grants create \
  --entitlement "db-admin-emergency-access" \
  --requested-duration "2h" \
  --justification "investigating incident INC-1234" \
  --location "$REGION" \
  --project "$PROJECT_ID"

Verify the exact command surface against the current gcloud pam documentation before relying on it, Privileged Access Manager is a newer GCP product and its CLI has changed shape across releases. The properties that matter for the design, not the exact flags, are: the grant is time-bound by default rather than requiring a separate revocation step, it requires a justification string that becomes part of the audit record, and it can require an approver distinct from the requester for a production-scoped entitlement.

What this design gets, once built§

• Five identity categories, application, migrations, deployment, human developers, break-glass admin, none of which can perform another's job.
• No stored database password anywhere in the design, not for the application, not for CI, not for a single human developer.
• Adding or removing a developer's access to a specific schema is a Cloud Identity group membership change, individually attributable, requiring no database-side operation and no credential handling.
• Onboarding a new schema is a fixed four-step procedure that does not require touching the application, migrations, or deployment identities.
• Emergency production access is time-bound and requires a justification and an audit trail by construction, rather than relying on someone remembering to revoke a manually-added group membership afterward.

Application Compatibility Gap§

The deploy scripts build the IAM-auth JDBC connection string this design assumes (enableIamAuth=true, no password field, socket-factory transport). Running an actual migration job against a freshly built instance failed before it reached the database at all:

Required key 'POSTGRES_PASSWORD' not found

The application's own database configuration code builds a plain host:port connection string unconditionally and requires a password field to be set, regardless of which profile is active. It has no code path for the IAM-auth, socket-factory connection this design assumes, it was never written to expect one.

This is a gap between the infrastructure and the application, not a Terraform or CI bug. Closing it means changing application code, not infrastructure, so it stays out of scope here and is tracked as separate application work. Everything above remains the target design, not the applied one, for that specific reason now, not only because the layer hadn't been reached yet.

Application Secrets§

The application reads on the order of dozens of environment variables. Most name a third-party integration credential, the rest are identifiers or plain config. The two categories are not interchangeable and are handled differently.

Classification§

• OAuth-style client IDs, account IDs, merchant IDs: identifiers, not credentials, by the providers' own convention. Knowing one does not let anyone authenticate as anything.
• Keys explicitly named "public" or "publishable": non-secret by definition, meant to ship in client-side code.
• Everything else with "secret", "password", "token", "key" in a non-publishable sense, plus one internal JWT signing key: real credentials, go in Secret Manager.
• Identifiers and plain operational config (URLs, log level, pool size) stay as literal values in the deploy template, no Secret Manager entry.

Secret Manager, shells only§

locals {
  app_secrets = [
    "jwt-symmetric-key",
    "provider-a-api-key",
    "provider-a-webhook-secret",
    # ...
  ]
}

resource "google_secret_manager_secret" "app" {
  for_each  = toset(local.app_secrets)
  secret_id = each.value
  project   = var.project_id
  replication {
    auto {}
  }
}

resource "google_secret_manager_secret_iam_member" "runtime_accessor" {
  for_each  = toset(local.app_secrets)
  secret_id = google_secret_manager_secret.app[each.value].id
  role      = "roles/secretmanager.secretAccessor"
  member    = "serviceAccount:${google_service_account.cloud_run_runtime.email}"
}

No secret values in this Terraform, no secret values in state, no secret values in git. The resource creates an empty container and a narrow read grant, one secret at a time, not a project-wide accessor role. Populating an actual version is a separate step, run manually, outside Terraform and outside CI.

The deploy template references real credentials by secretKeyRef, plain config values stay literal:

- name: PROVIDER_A_API_KEY
  valueFrom:
    secretKeyRef:
      name: provider-a-api-key
      key: "latest"
- name: LOG_LEVEL
  value: "info"

Populating values§

Secret values are written with a script, run manually, outside Terraform and outside CI, never checked into either. It prints every secret ID about to receive a new version and the target project, then requires explicit confirmation before writing anything:

echo "About to create a new secret version in project ${GCP_PROJECT_ID} for:"
for SECRET_ID in "${!SECRET_VALUES[@]}"; do
  echo "  ${SECRET_ID}"
done
read -r -p "Proceed? [y/N] " CONFIRM
[[ "${CONFIRM}" == "y" ]] || exit 1

Per-environment secret sets§

The two environments do not share one secret list. Different tenants and providers are active per environment, so the Secret Manager shells and the deploy template's secretKeyRef entries are each built from that environment's own inventory, not copied from the other environment with values swapped.

Restoring a Dump: Grants and Ownership§

Loading a dump into a fresh Cloud SQL instance and then running the application's own migrations against it hit a permission model gap that only shows up once real data is in place.

Why a plain GRANT is not enough§

The dump is restored through gcloud sql import, connected as the operator's own individually-attributable IAM database user (Requirement 8.6, no shared or service credential for a manual operation), not as the migrations service account. Every table and sequence the restore creates is therefore owned by the operator's identity, not the migrations identity, regardless of what schema-level privileges are granted afterward:

GRANT USAGE, CREATE ON SCHEMA schema_a TO "migrations-sa@example.iam";
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA schema_a TO "migrations-sa@example.iam";

GRANT ALL PRIVILEGES covers SELECT/INSERT/UPDATE/DELETE/TRUNCATE/REFERENCES/TRIGGER. It does not cover ownership, and several DDL operations, CREATE INDEX among them, require ownership specifically, not just a broad grant:

ERROR: must be owner of table payment_events

This surfaces the first time a migration tries to add an index to a table the restore created, not at restore time itself, and not on every table, only ones an in-flight migration happens to touch.

Reassigning ownership§

Fixed with an ownership sweep over every non-public schema, scoped to objects not already owned by the migrations identity:

DO $$
DECLARE
  target_schema text;
  target_table text;
  target_sequence text;
BEGIN
  FOR target_schema IN
    SELECT nspname FROM pg_namespace
    WHERE nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast', 'public')
      AND nspname NOT LIKE 'pg_temp%'
      AND nspname NOT LIKE 'pg_toast_temp%'
  LOOP
    FOR target_table IN
      SELECT tablename FROM pg_tables
      WHERE schemaname = target_schema
        AND tableowner <> 'migrations-sa@example.iam'
    LOOP
      EXECUTE format('ALTER TABLE %I.%I OWNER TO %I', target_schema, target_table, 'migrations-sa@example.iam');
    END LOOP;

    FOR target_sequence IN
      SELECT sequencename FROM pg_sequences
      WHERE schemaname = target_schema
        AND sequenceowner <> 'migrations-sa@example.iam'
    LOOP
      EXECUTE format('ALTER SEQUENCE %I.%I OWNER TO %I', target_schema, target_sequence, 'migrations-sa@example.iam');
    END LOOP;
  END LOOP;
END $$;

Run the same way the schema grants are, through gcloud sql import, which executes with Cloud SQL's own elevated import mechanism rather than as a normal client connection subject to the connecting identity's actual privileges. That elevation is also why public stays excluded from the loop: public's owner in Postgres 15+ is the special pg_database_owner role, and reassigning ownership against that specific role hits must be member of role even through the import mechanism. Every other schema is a regular one with no such restriction, and reassigns cleanly.

One sequence, run after every restore§

A fresh restore, not an incremental one, wipes out any grants and ownership from a previous pass along with the data. Three steps, in order, every time a dump is loaded into an environment, chained into one script rather than three manual ones:

#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

"${SCRIPT_DIR}/grant-migrations-create.sh"   # schema usage/create + table/sequence grants, migrations identity
"${SCRIPT_DIR}/grant-runtime-create.sh"      # schema usage + DML only, runtime identity, no CREATE
"${SCRIPT_DIR}/fix-ownership-create.sh"      # ALTER ... OWNER TO for pre-existing restored objects

ALTER DEFAULT PRIVILEGES, present in the grant scripts for objects a role creates in the future, does not help here: it only applies to objects subsequently created by whichever identity ran the ALTER DEFAULT PRIVILEGES statement itself, the import mechanism's own identity, not to objects a later restore creates while connected as the operator. It has no effect on this specific gap and is not a substitute for the ownership sweep.

Connecting the Core Application§

payment-processor-app's own isolated environment, load balancer, and secrets are one side of the migration. The other side is the company's main application (referred to here as core-app), calling into payment-processor-app across the project boundary.

A callback URL with a stale hardcoded default§

payment-processor-app builds two outbound callback URLs from a single environment variable, PAYMENT_DOMAIN_URL: a 3DS challenge return URL and a legacy provider callback base URL, both public URLs a customer's browser gets redirected to mid-payment.

challenge-callback-base-url: "${PAYMENT_DOMAIN_URL:https://payments.old-cloud.example.com/api/v1}"

If the variable isn't set, the application does not fail to start, it silently falls back to a hardcoded default, and that default is still the old cloud provider's hostname, baked into the GCP-profile config file itself. A deploy that never set this variable would keep generating callback URLs pointing at the retired environment, not the new one, with no error to signal it.

The variable is meant to be supplied as a CI/CD pipeline variable, substituted into the deploy template at deploy time, not a value with any equivalent in the running pod, so there's no live value to copy from.

The reverse direction§

core-app calling payment-processor-app is a second, separate config point, on the caller's side, in core-app's own pipeline or app config, not in payment-processor-app's Terraform or deploy scripts. It targets the retired environment's hostname. The load balancer and DNS on the payments side are reachable independent of this variable, one side existing does not imply the other side points at it.

WAF status: logging, not blocking§

The web application firewall in front of payment-processor-app runs every rule in preview mode, meaning it logs what it would have blocked without blocking anything, default action is allow for all source IPs. No firewall or WAF change is needed for core-app to reach payment-processor-app today. The one thing to do before switching preview off: check the preview logs for core-app's actual traffic first, or add a higher-priority allow rule scoped to core-app's known static egress IP, since generic SQLi/XSS/method-enforcement rules can false-positive on legitimate JSON API payloads, and losing legitimate service-to-service traffic to a newly-enforcing WAF rule is a worse failure mode than a slightly later cutover.

Closing the direct Cloud Run URL§

Every Cloud Run service also gets a default run.app URL that works on its own, independent of the load balancer or the WAF in front of it. Right now both services still accept traffic on that URL directly, a live bypass of the WAF entirely.

Ingress is a service-level setting, not a per-revision one, so it belongs in the deploy template's top-level metadata.annotations, not the revision template's:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: payment-processor-app
  annotations:
    run.googleapis.com/ingress: internal-and-cloud-load-balancing
spec:
  template:
    metadata:
      annotations:
        run.googleapis.com/vpc-access-connector: run-connector

internal-and-cloud-load-balancing allows traffic through the load balancer plus anything already inside the VPC, and nothing else. It doesn't change what the load balancer already sends through, it only removes the direct run.app path. Takes effect on the next normal deploy, no separate command or pipeline change, the deploy scripts already apply the whole template as one unit.

Public-facing Cloud Run services need an explicit invoker grant§

A load balancer and WAF in front of Cloud Run are not the access control by default. Cloud Run's own IAM invoker check runs first. Check:

gcloud run services get-iam-policy SERVICE_NAME --region REGION --project PROJECT_ID

An empty policy ({"etag": "ACAB"}, no bindings) means nothing can invoke the service, not the load balancer, not anyone. 403 Forbidden with the body text "does not have permission to get URL ... from this server" is this, not a WAF block. Grant public invoke and let the WAF and ingress lock be the actual access control:

gcloud run services add-iam-policy-binding SERVICE_NAME \
  --region REGION \
  --project PROJECT_ID \
  --member="allUsers" \
  --role="roles/run.invoker"

Idempotent. Put it in the deploy script, not a one-time manual step, Cloud Run isn't otherwise state-managed here.

minScale has two separate annotation namespaces§

Setting autoscaling.knative.dev/minScale on the revision template alone is not sufficient. Cloud Run also has a service-level run.googleapis.com/minScale, on the Service's own top-level metadata, not the revision template's. Set both, same value, two different places in the same file:

metadata:
  annotations:
    run.googleapis.com/minScale: "1"
    run.googleapis.com/maxScale: "2"
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1"
        autoscaling.knative.dev/maxScale: "2"

CPU throttling applies even to minScale-kept-warm instances§

Cloud Run throttles a container's CPU to near zero outside of active request handling by default. A minScale instance stays running but still gets throttled between requests, so it can still behave like a cold start once real work has to happen. Disable it, and enable the separate startup boost while at it, no added standing cost beyond what minScale already keeps running:

run.googleapis.com/cpu-throttling: "false"
run.googleapis.com/startup-cpu-boost: "true"

Match the container memory limit to the JVM flags it's running, not the platform default§

JAVA_OPTS carried over from the previous platform unchanged does not carry that platform's memory limit with it. Check the actual numbers: -Xmx615m -XX:MaxDirectMemorySize=300m is 915Mi before metaspace, thread stacks, or JIT overhead. Cloud Run's own default container memory limit is 512Mi if nothing sets it explicitly. That gap OOM-kills the container on a loop, visible directly in the log stream:

Memory limit of 512 MiB exceeded with 516 MiB used.

Set the limit explicitly, sized to the JVM flags, not the platform default:

resources:
  limits:
    cpu: "1"
    memory: 1536Mi

Spring profile files named for a combined environment-platform pair need that exact combined name active§

A file named application-{environment}-{platform}.yml only loads if a profile literally named {environment}-{platform} is active. Two separate profiles, environment and platform, comma-separated, does not activate it, Spring loads one file per individual active profile name, not one per combination. List-valued config (arrays, not scalars) does not merge cleanly enough across profile files to rely on for provider routing config even when both the base and the combined-name profile are active together. Activate the combined name directly, and keep the platform name active too for anything else gated on it alone:

SPRING_PROFILE=environment-platform,platform

Set allow_failure explicitly on jobs that must block the pipeline§

A job with allow_failure: true can fail without failing the pipeline. Audit every migrate/deploy job for the newer platform against the equivalent job for the platform already in production, same repo, same pipeline file. Match allow_failure and the manual/automatic trigger rule to the established platform's jobs exactly, field by field, don't assume they already match.

allow_failure: false

WAF Enforcement Prep§

The WAF has run in preview mode since it was built, logging what it would block without blocking anything. Moving it to enforcing needs three things in place first: a way to exempt trusted internal callers, a way to actually see what preview would have done, and a way to get that information in front of someone without manually running a query.

Backend service access logging is off by default§

Cloud Armor's per-request verdict, preview or enforced, is recorded as part of the load balancer's own request log, not the backend's application log. google_compute_backend_service does not log by default:

resource "google_compute_backend_service" "payments" {
  name                  = "payments-backend"
  security_policy       = google_compute_security_policy.payments.id

  log_config {
    enable      = true
    sample_rate = 1.0
  }

  backend {
    group = google_compute_region_network_endpoint_group.payments.id
  }
}

sample_rate is a fraction, 1.0 logs every request, 0.1 would log roughly one in ten. At low request volume (low tens of thousands per month) 100% logging stays inside Cloud Logging's free ingestion tier by a wide margin, a few hundred MB against a 50 GiB/project/month allowance. Reconsider only once real volume is high enough for that math to change.

Exempt trusted internal callers from content-inspection rules§

A caller with a known, stable source IP (a backend service reaching this one through a static NAT egress IP) does not need to go through SQLi/XSS/method-enforcement inspection. Add an allow rule with a priority lower than every other rule in the policy, Cloud Armor evaluates in ascending priority order and stops at the first match:

resource "google_compute_security_policy_rule" "allow_core_app" {
  security_policy = google_compute_security_policy.payments.name
  priority         = 500
  preview          = false
  action           = "allow"
  description      = "core-app static egress IP, bypass WAF content inspection"
  match {
    versioned_expr = "SRC_IPS_V1"
    config {
      src_ip_ranges = ["<core-app-nat-ip>/32"]
    }
  }
}

Confirm the caller's egress is actually static before relying on this. Cloud Run's default egress uses ephemeral IPs, only run.googleapis.com/vpc-access-egress: all-traffic through a VPC connector with a Cloud NAT gateway attached produces a stable, predictable source IP.

Reading preview verdicts§

Once backend logging is on, preview and enforced verdicts show up under resource.type="http_load_balancer", in jsonPayload.previewSecurityPolicy and jsonPayload.enforcedSecurityPolicy:

# what would preview have denied, last 24h
gcloud logging read '
resource.type="http_load_balancer"
jsonPayload.previewSecurityPolicy.configuredAction="DENY"
' --project=PROJECT_ID --freshness=24h --format=json

# did the rate limit rule ever actually exceed threshold
gcloud logging read '
resource.type="http_load_balancer"
jsonPayload.previewSecurityPolicy.rateLimitAction.outcome="RATE_LIMIT_THRESHOLD_EXCEED"
' --project=PROJECT_ID --freshness=24h

A DENY hit is not automatically a false positive. Check httpRequest.requestUrl, remoteIp, and userAgent on the matched entry. Requests against the bare load balancer IP instead of the real hostname, paths like /.env or known exploit probe paths, and scanner user agents (zgrab, similar) are internet background noise, not application traffic, and are exactly what the rule should catch once enforcing.

Daily digest to Slack§

A scheduled summary beats a real-time alert on every match, background scanning traffic fires preview rules constantly and a per-event alert would be mostly noise. One Cloud Scheduler job triggers one Cloud Run Job daily, no custom container build needed, google/cloud-sdk:slim plus an inline script covers it:

resource "google_cloud_run_v2_job" "waf_digest" {
  name     = "waf-daily-digest"
  location = var.region

  template {
    template {
      service_account = google_service_account.waf_digest_run.email
      containers {
        image   = "google/cloud-sdk:slim"
        command = ["bash", "-c"]
        args    = [file("${path.module}/scripts/waf-daily-digest.sh")]

        env {
          name  = "GCP_PROJECT_ID"
          value = var.project_id
        }
        env {
          name = "SLACK_WEBHOOK_URL"
          value_source {
            secret_key_ref {
              secret  = google_secret_manager_secret.slack_webhook.secret_id
              version = "latest"
            }
          }
        }
      }
    }
  }
}

resource "google_cloud_scheduler_job" "waf_digest" {
  name      = "waf-daily-digest-trigger"
  region    = var.region
  schedule  = "0 9 * * *"
  time_zone = "UTC"

  http_target {
    http_method = "POST"
    uri         = "https://${var.region}-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/${var.project_id}/jobs/${google_cloud_run_v2_job.waf_digest.name}:run"
    oauth_token {
      service_account_email = google_service_account.waf_digest_scheduler.email
    }
  }
}

file() must reference a path inside the same repo the Terraform lives in. A relative path pointing outside the repo resolves fine locally and breaks the moment someone else clones just this repo, or CI does.

A Slack incoming webhook needs no bearer token and no channel field, the URL itself is already bound to a workspace and channel:

curl -s -X POST "${SLACK_WEBHOOK_URL}" \
  -H "Content-Type: application/json" \
  -d "$(python3 -c "import json,sys; print(json.dumps({'text': sys.argv[1]}))" "${MESSAGE}")"

Store the webhook URL as a plain Secret Manager value, same shells-only-in-Terraform pattern as every other secret in this project, populated by a script, not committed anywhere.

First-use IAM, one grant per resource type§

None of google_cloud_run_v2_job, google_cloud_scheduler_job, or attaching a service account to either had ever been touched by this project's Terraform service account before this job. Each surfaces its own 403 on first apply, fix them in this order, they only appear one at a time:

# create the Cloud Run Job
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:terraform-sa@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/run.admin"

# attach a service account to a Cloud Run resource
gcloud iam service-accounts add-iam-policy-binding \
  RUNTIME_SA@PROJECT_ID.iam.gserviceaccount.com \
  --member="serviceAccount:terraform-sa@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountUser"

# first use of Cloud Scheduler in this project at all
gcloud services enable cloudscheduler.googleapis.com --project=PROJECT_ID

# create a Cloud Scheduler job
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:terraform-sa@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/cloudscheduler.admin"

# attach a service account to the Cloud Scheduler job specifically
gcloud iam service-accounts add-iam-policy-binding \
  SCHEDULER_SA@PROJECT_ID.iam.gserviceaccount.com \
  --member="serviceAccount:terraform-sa@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountUser"

run.admin, not run.developer, because Terraform also needs to set IAM policy on the job itself (the scheduler's own invoker binding), which run.developer does not cover. The two serviceAccountUser grants are scoped to one service account each, not project-wide, matching every other IAM grant in this project.

A grant that shows as applied via get-iam-policy and still fails on retry is propagation lag, not a wrong role or a missing binding. Wait roughly a minute, retry with no changes.

Audit Log Retention§

VPC flow logs were already enabled on the connector subnet from the network layer above. What was missing: Cloud Audit Log data access logging, and somewhere for logs to live for the retention period PCI-DSS Requirement 10.5.1 asks for, 12 months, with at least 3 immediately available online. Cloud Logging's own default log bucket only keeps 30 days.

resource "google_project_iam_audit_config" "all_services" {
  project = var.project_id
  service = "allServices"

  audit_log_config {
    log_type = "ADMIN_READ"
  }
  audit_log_config {
    log_type = "DATA_READ"
  }
  audit_log_config {
    log_type = "DATA_WRITE"
  }
}

resource "google_logging_project_bucket_config" "default_90d" {
  project        = var.project_id
  location       = "global"
  bucket_id      = "_Default"
  retention_days = 90
}

Admin Activity and System Event audit logs are always on, free of charge, and cannot be disabled. Data Access logs are off by default for most services and billed the same as any other ingested log volume once enabled. Enabling them for allServices rather than a specific list is the simplest way to satisfy "log individual access to sensitive data" without auditing which services touch sensitive data one at a time, at the cost of a real increase in log volume and its associated cost.

90 days online covers the 3-month minimum. The 12-month figure is a separate archive, not the online bucket:

resource "google_storage_bucket" "log_archive" {
  name                        = "${var.project_id}-log-archive"
  project                     = var.project_id
  location                    = var.region
  uniform_bucket_level_access = true
  force_destroy               = false
  public_access_prevention    = "enforced"

  retention_policy {
    retention_period = 31536000 # 365 days
  }
}

resource "google_logging_project_sink" "audit_archive" {
  name        = "audit-log-archive"
  project     = var.project_id
  destination = "storage.googleapis.com/${google_storage_bucket.log_archive.name}"
  filter      = "logName:\"logs/cloudaudit.googleapis.com\" OR logName:\"logs/compute.googleapis.com%2Fvpc_flows\""

  unique_writer_identity = true
}

resource "google_storage_bucket_iam_member" "sink_writer" {
  bucket = google_storage_bucket.log_archive.name
  role   = "roles/storage.objectCreator"
  member = google_logging_project_sink.audit_archive.writer_identity
}

The sink's own identity needs write access to the destination bucket, not the Terraform account. unique_writer_identity creates a Google-managed service account specific to this one sink, and that identity gets exactly roles/storage.objectCreator on exactly this one bucket, nothing broader.

retention_policy here is not locked. GCS bucket lock makes a retention policy permanently unremovable, including by the project owner, the stronger tamper-protection PCI-DSS Requirement 10.3 is really asking for, but it is irreversible: once locked, the retention period can only be increased, never shortened or removed, for the life of the bucket. Left unlocked for now, revisited once the retention period is confirmed settled rather than locking in a number that might still need to change.