Moving a Live Key Vault Between Terraform Repos

Table of Contents§

• The Problem
• The Solution
  • Backups
  • The move
  • Resource group mismatch
  • Distinguishing used from unused
• The Results

The Problem§

A backend service had its own infrastructure repository: AKS clusters, databases, networking. Its two Azure Key Vaults, dev and prod, were declared in a different repository, the one belonging to the main platform. This was documented in a code comment:

# NOTE: All these resources must be moved to the service's own tf repo!

The move had not been done. The task requires relocating two live Key Vaults, one of them backing production traffic, from one Terraform state to another, with no downtime and no destroy-and-recreate.

The move became a prerequisite for an unrelated task: archiving decommissioned repositories in the platform group. Archival could not proceed while those repositories still owned live resources belonging to another service.

The two repositories used separate Terraform backends: different storage accounts, different containers, no shared state. terraform state mv operates on a single state file. Moving a resource between two independently backed remote states requires pulling both down locally, performing the move on local copies, and pushing each back in order.

The Solution§

Backups§

Before any state operation, all four remote states (dev and prod, source and destination) were pulled and saved as plain files:

az storage blob download --account-name <source-account> --container-name <dev-container>  --name terraform.tfstate --file source-dev.tfstate.bak
az storage blob download --account-name <source-account> --container-name <prod-container> --name terraform.tfstate --file source-prod.tfstate.bak
az storage blob download --account-name <dest-account>   --container-name <dev-container>  --name terraform.tfstate --file dest-dev.tfstate.bak
az storage blob download --account-name <dest-account>   --container-name <prod-container> --name terraform.tfstate --file dest-prod.tfstate.bak

Restoration, if needed, is a single terraform state push against a backup file.

The move§

The destination repository's HCL already contained the new resource blocks, copied from the source. The source repository's HCL already had those blocks removed. What remained was state bookkeeping, three commands per environment:

# pull both sides down locally
cd source-repo/envs/dev && terraform init && terraform state pull > /tmp/source-dev.tfstate
cd dest-repo/envs/dev   && terraform init && terraform state pull > /tmp/dest-dev.tfstate

# move the resource between the two local files
terraform state mv \
  -state=/tmp/source-dev.tfstate \
  -state-out=/tmp/dest-dev.tfstate \
  azurerm_key_vault.service azurerm_key_vault.service

# push destination first, then source
cd dest-repo/envs/dev   && terraform state push /tmp/dest-dev.tfstate
cd source-repo/envs/dev && terraform state push /tmp/source-dev.tfstate

Destination is pushed first. If the process is interrupted between the two pushes, the resource exists in both states rather than neither. That is recoverable by completing the second push. The reverse ordering is not: an interruption would leave the resource in neither state.

The HCL and state changes were shipped as two paired merge requests, one per repository, each marked not to be merged before the state migration completed. Merging either alone, with state and config out of sync, would have caused Terraform to either destroy the vault (source repository: resource removed from config, still in state) or attempt to create a duplicate with a name Azure already had in use (destination repository: resource in config, absent from state).

Resource group mismatch§

Before merging, terraform plan was run in the destination repository. It proposed to destroy and recreate the Key Vault:

# azurerm_key_vault.service must be replaced
-/+ resource "azurerm_key_vault" "service" {
      ~ resource_group_name = "platform-rg" -> "service-rg" # forces replacement
      ...

The draft HCL had copied resource_group_name = data.azurerm_resource_group.main_rg.name from the source repository. Both repositories define a data source named main_rg. In the source repository it resolves to the platform's resource group. In the destination repository, the same name resolves to a different group belonging to the destination service. The vault's actual resource group, confirmed with az keyvault show, was the platform's group, not the destination's own. Key Vault's resource_group_name is immutable. A mismatch between config and reality leaves Terraform one option: destroy and recreate.

Internal documentation was checked first and was wrong about which Azure region the resources were in. The fix used the cloud provider's own CLI output, not the documentation.

The fix: a dedicated data source pointing at the correct resource group, independent of the destination repository's own main_rg:

data "azurerm_resource_group" "vault_rg" {
  name = "platform-rg" # the vault's real resource group, not this repo's own
}

resource "azurerm_key_vault" "service" {
  resource_group_name = data.azurerm_resource_group.vault_rg.name
  # ...
}

terraform plan returned no changes on both sides, in both environments, after the fix.

Distinguishing used from unused§

A second, unrelated Key Vault existed in the same source file, used for an internal secrets-management auto-unseal key. Most other Terraform files in that environment had been renamed with a .decommissioned suffix, a convention that makes Terraform ignore a file without deleting it. This vault's file carried the same suffix. terraform plan proposed to destroy it as well, a resource outside the scope of the migration.

The filename convention was not treated as sufficient evidence on its own. Verification:

• The application source and deployment configuration were searched for the vault's name, its service-principal ID, and any Key Vault SDK dependency. No matches.
• The vault under migration was searched the same way. Its URL and service-principal credentials were present as environment variables in the running production deployment, and a code comment from another engineer confirmed a component resolved secrets from it at startup.

One vault was referenced by running code. The other was referenced by nothing outside its own Terraform declaration. That distinction, not the filename convention, was the basis for the decision. The unreferenced vault was destroyed, with soft-delete and purge protection enabled, giving a recovery window if the decision proved wrong. The referenced vault was moved, not destroyed.

The Results§

• Both Key Vaults now reside in the correct repository, correct state, correct resource group, matching their actual location in Azure.
• No downtime, no resource recreation, for the vault in production use.
• The resource-group mismatch was caught by terraform plan before merge, not after, as an incident. The paired merge requests made the ordering constraint explicit rather than relying on memory.
• One unreferenced resource was removed, based on a code search across all potential consumers rather than a filename convention, with a soft-delete recovery window as a safeguard.
• The archival task blocked on this migration is unblocked.

General note: when resource_group_name, or any attribute marked "forces replacement" in a terraform plan, differs between two configurations, do not resolve it by preferring whichever value is closer at hand. Check the actual resource with the cloud provider's CLI before writing HCL. In this migration, both internal documentation and a copy-pasted data source name were wrong. az keyvault show was not.