Developer Access to a Private Cloud SQL Instance Without a VPN or a Login-able VM

draft

Giving developers a path to a Postgres instance with no public IP, without a self-managed bastion anyone logs into and without standing up a VPN. Uses GCP Identity-Aware Proxy TCP forwarding to a headless relay VM that runs one socat process and nothing else, no SSH keys, no OS Login, IAM gates the tunnel instead of a login. Covers the network layer and the separate, non-optional step after it: Postgres-level group authentication and privilege grants, since reaching the instance is not the same as being able to query it.

August 31, 2026infra2916 words · 15 min read
#gcp#terraform#pci-dss#iam#cloud-sql#postgres#iap#infra

Table of Contents

The Problem

payment-processor-app's Postgres instance on GCP Cloud SQL has no public IP. This is a requirement, not a preference: the instance is in scope for PCI-DSS, and direct internet access to a database holding payment records is disallowed regardless of firewall rules layered on top of it. The instance is reachable only over its private IP, inside its own VPC.

Developers still need to connect to it, for local debugging, one-off queries, running migrations by hand when the pipeline is not the right tool for the job. A private-IP-only instance means the client needs network reachability into the VPC. That reachability does not exist by default from a laptop on the open internet, and the options for creating it each had a cost:

  • A traditional bastion host: a VM developers SSH into, then connect to the database from inside it. This works, but the VM becomes a standing target: SSH key management, OS patching, vulnerability scanning, and a documented hardening baseline, all now in scope for the same compliance program that required no public IP on the database in the first place. Under audit, a machine people log into is a machine that gets questions asked about it.
  • A client VPN: the cloud provider in use has no native point-to-site VPN product, only site-to-site (two networks peered together, each side needs its own gateway and config). Getting individual laptops onto the network natively would mean running a VPN server, which is the bastion problem again with extra steps.
  • A third-party mesh VPN: solves the individual-client problem well, but adds an external SaaS dependency and still requires a relay node with an actual presence inside the VPC. Worth it for durable, broad access. Overkill for "let a few developers run psql."

The Solution

Two mechanisms combined, both provided by the cloud platform, neither requiring a VPN or a login-able machine.

IAP TCP forwarding, not IAP's web flow

Identity-Aware Proxy has two distinct products under one name. One fronts HTTPS applications and requires an OAuth consent screen (an "IAP brand"). The other, TCP forwarding, tunnels a local port to a port listening on a specific Compute Engine instance, gated purely by IAM: roles/iap.tunnelResourceAccessor on that instance. No OAuth brand, no web application, no SSH involved in the authorization step. A developer with the role runs:

gcloud compute start-iap-tunnel <instance> <port> \
  --local-host-port=localhost:<port> \
  --zone=<zone> \
  --project=<project>

and gets a local port that tunnels through the cloud provider's own edge to that port on that instance, encrypted, IAM-checked, no VPC peering or VPN required on the client side.

The instance forwards bytes and does nothing else

TCP forwarding reaches a port on the instance itself, not arbitrary other hosts the instance can reach. Something has to listen on that instance and forward to the database's private IP. The database already serves plain Postgres protocol on its private IP, no proxy needed for that leg, so the forwarder does not need to understand Postgres, TLS, or Cloud SQL's IAM authentication scheme. It just needs to move bytes:

socat TCP-LISTEN:5432,fork,reuseaddr TCP:<db-private-ip>:5432

That is the entire job of the instance. Running it as the single container on Container-Optimized OS (Google-managed image, minimal package set, automatic patching) keeps the instance's own attack surface small and keeps "what does this machine do" answerable in one sentence.

The instance has no SSH keys, block-project-ssh-keys is set, and no OS Login role is granted to anyone. There is no interactive login path to it under normal operation. If it ever needs to change, the fix is terraform apply, not a shell session.

A firewall rule restricts ingress on the relay's port to Identity-Aware Proxy's fixed source range (35.235.240.0/20) only, nothing else, and the instance's own service account carries no Cloud SQL permissions at all: it never calls a Cloud SQL API, it only forwards TCP.

Authentication stays at the database

The relay does not participate in authentication. Each developer connects through the tunnel with their own Postgres credentials, native username and password, or Cloud SQL IAM database authentication using their own identity token as the password. Two different developers tunneling through the same relay at the same time are indistinguishable to the relay and fully distinguishable to Postgres, because the relay never terminates the protocol, it only moves bytes between two TCP sockets.

A platform change caught mid-build

The instance was first built using a declarative container specification in instance metadata (gce-container-declaration), the standard mechanism for years. Applying it failed:

Error 400: You are creating a container VM. The option to deploy a
container during VM instance creation that relies on a container
startup agent is discontinued.

The cloud provider had discontinued that startup agent for new instance creation. The documented replacement is a boot-time docker run in a plain startup script, which still runs on Container-Optimized OS, still needs no package installation beyond what the image ships with. One difference from the declarative form: a startup script runs on every boot, not once, and a container started with --restart=always survives a reboot on its own. Rerunning docker run with the same container name on the next boot then fails on a name collision. The fix is a docker rm -f <name> || true before the docker run, making the script idempotent regardless of how many times it runs.

A Working Tunnel Is Not Database Access

Everything above solves network reachability. It does not grant a single privilege inside Postgres. A developer with a working tunnel and a valid Cloud SQL IAM login can authenticate and immediately hit permission denied for schema on every query, because logging in and being able to read data are two separate, independently configured layers.

Group login, not per-user Terraform resources

Cloud SQL supports IAM group authentication for PostgreSQL directly: a google_sql_user of type CLOUD_IAM_GROUP, named after the group's own email, rather than one resource per person.

resource "google_sql_user" "developers_group" {
  name     = "<developers-group>@<domain>"
  project  = var.project_id
  instance = google_sql_database_instance.db.name
  type     = "CLOUD_IAM_GROUP"
}

resource "google_project_iam_member" "developers_group_sql_client" {
  project = var.project_id
  role    = "roles/cloudsql.client"
  member  = "group:<developers-group>@<domain>"
}

resource "google_project_iam_member" "developers_group_sql_instance_user" {
  project = var.project_id
  role    = "roles/cloudsql.instanceUser"
  member  = "group:<developers-group>@<domain>"
}

When an individual member of that group connects, Cloud SQL creates a database account for them automatically on first login, using their own identity, not a shared group credential. No individual google_sql_user resource is required for this to work. Membership changes in the identity provider (someone added to or removed from the group) take effect without a Terraform change on this side, roughly a 15-minute propagation delay applies.

A Terraform argument that does not exist yet

An early attempt tried to attach an elevated database role at the same time as the user resource:

resource "google_sql_user" "admin" {
  name           = "<admin-user>@<domain>"
  project        = var.project_id
  instance       = google_sql_database_instance.db.name
  type           = "CLOUD_IAM_USER"
  database_roles = ["cloudsqlsuperuser"]
}

terraform apply rejected it: Error: Unsupported argument. The database_roles field exists in the provider, confirmed by reading the provider's own source, but not in the major version pinned in this repository (~> 5.0). A sibling repository in the same organization, upgraded further, resolved to 7.32.0 and has the field available. Bumping a provider's major version to gain one argument is its own decision, with its own blast radius across every other resource in the repository, not something to do as a side effect of an unrelated task. The practical fix on a repository pinned to an older major version: grant the role with SQL instead of Terraform.

General lesson: when a documented provider argument is rejected, check it against the resolved version actually in use (the lock file, not ~> 5.0 as written, and not the provider's latest source), before assuming the argument name is wrong.

Bootstrapping requires an already-privileged connection

Granting any privilege requires connecting as a role that already holds grant authority. An individual developer's freshly created IAM login holds none by default, it cannot grant privileges to itself or to anyone else. The built-in postgres account is the one account on a Cloud SQL Postgres instance with that authority out of the box:

gcloud sql users set-password postgres \
  --instance=<instance> \
  --project=<project> \
  --prompt-for-password

Every statement in this section runs through a session authenticated as postgres, connected the same way as any other client, through the relay tunnel:

# terminal 1, leave running
gcloud compute start-iap-tunnel <instance> 5432 \
  --local-host-port=localhost:5432 \
  --zone=<zone> \
  --project=<project>

# terminal 2
psql "host=localhost port=5432 dbname=<db> user=postgres sslmode=require"

Verify object ownership before writing default-privilege statements

A schema's nominal owner and the role that actually creates new tables inside it, day to day, through a migration pipeline, are not guaranteed to be the same role, and a migration tool's own source (a changelog file, an old schema-bootstrap script) is not a reliable source for the current answer either. Check the live instance:

SELECT nspname, nspowner::regrole
FROM pg_namespace
WHERE nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast')
  AND nspname NOT LIKE 'pg\_temp\_%'
  AND nspname NOT LIKE 'pg\_toast\_temp\_%';

SELECT schemaname, tablename, tableowner
FROM pg_tables
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
ORDER BY schemaname
LIMIT 20;

The role that shows up as tableowner on existing tables, not the CI/CD identity that opens the migration connection, not a name found in an old migration file, is the value that belongs in FOR ROLE below. On the instance this was built against, the migration pipeline authenticates as one IAM service account, but the tables it creates end up owned by a separate, broader database role that account inherits, confirmed only by querying pg_tables directly.

A read-only role, granted once to the group

CREATE ROLE developers_ro NOLOGIN;

DO $$
DECLARE
  schema_name text;
BEGIN
  FOR schema_name IN
    SELECT nspname FROM pg_namespace
    WHERE nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast')
      AND nspname NOT LIKE 'pg\_temp\_%'
      AND nspname NOT LIKE 'pg\_toast\_temp\_%'
  LOOP
    EXECUTE format('GRANT USAGE ON SCHEMA %I TO developers_ro', schema_name);
    EXECUTE format('GRANT SELECT ON ALL TABLES IN SCHEMA %I TO developers_ro', schema_name);
    EXECUTE format('ALTER DEFAULT PRIVILEGES FOR ROLE <table-owner-role> IN SCHEMA %I GRANT SELECT ON TABLES TO developers_ro', schema_name);
  END LOOP;
END $$;

GRANT developers_ro TO "<developers-group>@<domain>";

GRANT SELECT ON ALL TABLES covers only tables that exist at the moment the statement runs. ALTER DEFAULT PRIVILEGES FOR ROLE <table-owner-role> is a standing rule, not a one-time grant: any table created afterward, by that same role, in that same schema, automatically carries the grant, no follow-up statement required per new table. It does not cover a schema that does not exist yet. Onboarding a new schema still means re-running the grant for it, or folding the grant into whatever process creates the schema in the first place.

A read-write variant follows the identical shape, wider privilege list, same default-privilege mechanism:

CREATE ROLE developers_rw NOLOGIN;

DO $$
DECLARE
  schema_name text;
BEGIN
  FOR schema_name IN
    SELECT nspname FROM pg_namespace
    WHERE nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast')
      AND nspname NOT LIKE 'pg\_temp\_%'
      AND nspname NOT LIKE 'pg\_toast\_temp\_%'
  LOOP
    EXECUTE format('GRANT USAGE ON SCHEMA %I TO developers_rw', schema_name);
    EXECUTE format('GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA %I TO developers_rw', schema_name);
    EXECUTE format('GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA %I TO developers_rw', schema_name);
    EXECUTE format('ALTER DEFAULT PRIVILEGES FOR ROLE <table-owner-role> IN SCHEMA %I GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO developers_rw', schema_name);
    EXECUTE format('ALTER DEFAULT PRIVILEGES FOR ROLE <table-owner-role> IN SCHEMA %I GRANT USAGE, SELECT ON SEQUENCES TO developers_rw', schema_name);
  END LOOP;
END $$;

GRANT developers_rw TO "<developers-group>@<domain>";

Neither role includes CREATE, DROP, or ALTER TABLE. Schema and table structure changes stay on the migration pipeline's own identity and on postgres. Granting developers_rw to the same group that already holds developers_ro makes the read-only grant redundant for that population, since read-write is a strict superset, an intentional simplification only if every member of the group is meant to have write access, not a default to reach for without checking that assumption first.

Renaming a resource address without destroying the object

An individually managed google_sql_user for one specific developer predated the group-based design above and needed to be folded into a shared, for_each-driven collection alongside other named users. Removing the old resource block and adding an equivalent one under a new address changes nothing about the real object's identity from Terraform's point of view: the old address disappears, a new one appears with nothing tracked at it, and the default plan is destroy the old, create the new.

For a Cloud SQL user this means the underlying Postgres role gets dropped and recreated. Role drop and recreate does not touch table data, roles and data are unrelated concepts in Postgres, but it does drop anything granted directly to that specific role: membership in developers_ro or developers_rw, or in cloudsqlsuperuser. If nothing has been granted to the role yet, the operation is a brief, harmless gap in that one person's login. If a privilege grant is already attached, do the move without a gap:

terraform state mv 'google_sql_user.old_address' 'google_sql_user.new_collection["user@domain"]'
terraform plan

terraform plan should report no changes for that resource afterward, confirming the rename landed without a destroy. If it still proposes a destroy and create, stop before applying and check the address strings match exactly what terraform plan shows, including the map key's quoting.

A transient provider error under parallel IAM grants

Applying several google_project_iam_member resources that grant the same role to different members, run in parallel by default since they are otherwise independent resources, occasionally produced:

Error: Provider produced inconsistent result after apply

When applying changes to google_project_iam_member.developers_sql_client["user@domain"], provider
"provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value: Root object was present, but now absent.

Every one of these resources reads, modifies, and writes the same underlying project IAM policy document. Several doing that concurrently race against Cloud Resource Manager's eventually-consistent API and can surface exactly this error mid-race. It is not a configuration mistake. Re-running terraform apply resolves it in the overwhelming majority of cases, the grant that appeared to fail has usually already landed and the retry is a no-op. If it recurs, force serial application:

terraform apply -parallelism=1

Reproduce This

Assumes an existing VPC with a Cloud SQL instance already on a private IP, and Identity-Aware Proxy's API enabled on the project.

  1. A dedicated subnet for the relay, separate from any subnet reserved for a serverless VPC connector, those are not meant to host other resources:
resource "google_compute_subnetwork" "relay" {
  name          = "cloud-sql-relay"
  region        = var.region
  network       = google_compute_network.vpc.id
  ip_cidr_range = "<a small unused range inside the VPC's CIDR>"
}
  1. A firewall rule admitting only IAP's fixed range, on the relay's port, targeted by tag:
resource "google_compute_firewall" "allow_iap_to_relay" {
  name          = "allow-iap-to-cloud-sql-relay"
  network       = google_compute_network.vpc.id
  direction     = "INGRESS"
  allow {
    protocol = "tcp"
    ports    = ["5432"]
  }
  source_ranges = ["35.235.240.0/20"]
  target_tags   = ["cloud-sql-relay"]
}
  1. A service account with no elevated roles, logging and monitoring write only:
resource "google_service_account" "relay" {
  account_id   = "cloud-sql-relay"
  display_name = "Cloud SQL relay VM"
}
  1. The instance, Container-Optimized OS, no public IP, no SSH keys, shielded VM options on:
resource "google_compute_instance" "relay" {
  name         = "cloud-sql-relay"
  zone         = "<zone>"
  machine_type = "e2-micro"
  tags         = ["cloud-sql-relay"]

  boot_disk {
    initialize_params {
      image = "cos-cloud/cos-stable"
    }
  }

  network_interface {
    network    = google_compute_network.vpc.id
    subnetwork = google_compute_subnetwork.relay.id
    # no access_config block: no public IP
  }

  service_account {
    email  = google_service_account.relay.email
    scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring.write",
    ]
  }

  shielded_instance_config {
    enable_secure_boot          = true
    enable_vtpm                 = true
    enable_integrity_monitoring = true
  }

  metadata = {
    block-project-ssh-keys = "true"
    startup-script = <<-EOF
      #!/bin/bash
      docker rm -f cloud-sql-relay 2>/dev/null || true
      docker run --restart=always --detach --name cloud-sql-relay \
        -p 5432:5432 \
        alpine/socat \
        TCP-LISTEN:5432,fork,reuseaddr TCP:${google_sql_database_instance.db.private_ip_address}:5432
    EOF
  }

  deletion_protection = true
}
  1. The IAM binding that actually gates access, scoped to this one instance, not the project:
resource "google_iap_tunnel_instance_iam_member" "relay_access" {
  zone     = google_compute_instance.relay.zone
  instance = google_compute_instance.relay.name
  role     = "roles/iap.tunnelResourceAccessor"
  member   = "group:<developers-group>"
}
  1. Connect from a client machine, two terminals:
# terminal 1, leave running
gcloud compute start-iap-tunnel cloud-sql-relay 5432 \
  --local-host-port=localhost:5432 \
  --zone=<zone> \
  --project=<project>

# terminal 2
psql "host=localhost port=5432 dbname=<db> user=<user> sslmode=require"

A connection-refused or timeout on the psql side points at the relay or the firewall rule. A Postgres-level authentication error means the tunnel and relay both worked, and the credentials themselves need fixing, a separate, expected step: this design gives network reachability, not database identity.

The Results

  • Developers reach the database from an unmodified laptop, no VPN client, no relay-side credentials to manage.
  • The database keeps zero public IP exposure.
  • The relay VM has no interactive login path under normal operation: no SSH keys, no OS Login grant. Under audit, the answer to "who can log into this machine" is "nobody, by design," not a list of names and a key-rotation policy.
  • The relay's own service account has no Cloud SQL permissions and no elevated project roles, its entire footprint is "can write logs and forward TCP."
  • Access control is one IAM binding on one instance, not a shared secret, not a certificate to distribute, not a VPN client config to hand out.
  • The mechanism is deliberately interim. It solves the immediate access problem with the smallest infrastructure footprint; a broader, identity-based mesh VPN remains the eventual answer for reasons unrelated to this specific database, and this relay is expected to be retired once that lands, not extended.
  • Group membership in the identity provider is the single source of truth for who can log in, adding or removing a developer requires no Terraform change and no per-user Cloud SQL resource.
  • Read and read-write access are two separate roles granted once to the group, not per-user grants repeated for every new hire, and future tables in existing schemas inherit the grant automatically through ALTER DEFAULT PRIVILEGES.
  • No individual developer's own login carries any privilege by default. Every grant traces back to a single bootstrap credential (postgres), used once per privilege change, not for routine access.