Codify Vault Internal PKI using Terraform

Automate your Certificates Lifecycle with Vault — Part 5

To make sure that you can automate things correctly using Vault’s Internal PKI, you need to know how to do some steps manually. You can learn about these important requirements by reading the detailed article Build an Internal PKI with Vault.

This article is a part of a series about how to
Automate your Certificates Lifecycle with Vault.

1. Build an Internal PKI with Vault
2. Issue, Deploy and Renew your Private Certificates with Vault and Consul-Template
3. Rotate your CA seamlessly using a Vault PKI
4. Securely store Public Certificates in Vault, generated by acme.sh
5. Codify Vault Internal PKI using Terraform (this article)

Introduction

This article contains a walkthrough to codifying a branch of a three-tier PKI CA with Vault using an offline Root CA generated with certstrap.

Three-Tier PKI CA Hierarchy

In order to respect the IT security principle of Least Privilege And Separation Of Duties and to obtain flexibility of use, the Vault PKI built in this article is a branch of a three-tier PKI Certification Authority hierarchy.

Three-Tier PKI Certification Authority Hierarchy

Usually, the root of trust is anchored within an existing company Root CA outside of Vault. So, the Root CA is stored offline, namely outside of Vault.

Then, the Intermediate CA and Issuing CA are stored inside of Vault using a dedicated PKI secret engine for each. Moreover, a dedicated Role is required for issuing Leaf Certificates.

PKI CA Hierarchy with Vault

Pre-Requisites

• Install vault 1.14.0 or higher
• Install terraform 1.15.3 or higher
• Install certstrap 1.3.0 or higher
• Install jq 1.6 or higher

Vault Server in Dev mode

First, launch Vault Server in Dev mode using root as a root token id. The root token is used on purpose to focus only on the PKI secret engine features.

Disclaimer: On production
- Don’t use the root token
- Create auth methods, entities, groups and policies to grant or forbid access to PKI features

$ vault server -dev -dev-root-token-id=root

In a second terminal, export Vault variables and check its status.

$ export VAULT_ADDR=http://127.0.0.1:8200
$ export VAULT_TOKEN=root
$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.11.3
Build Date      2022-08-26T10:27:10Z
Storage Type    inmem
Cluster Name    vault-cluster-80f46c53
Cluster ID      075e1954-dd54-6614-f8a0-09351c840504
HA Enabled      false

So, the Vault server is running now.

Root CA

Let’s start with the Root CA.

The offline Root CA is simulated using certstrap open source tool from Square.

$ certstrap --depot-path root init \
    --organization "Example" \
    --common-name "Example Labs Root CA v1" \
    --expires "10 years" \
    --curve P-256 \
    --path-length 2 \
    --passphrase "secret"
Created root/Example_Labs_Root_CA_v1.key (encrypted by passphrase)
Created root/Example_Labs_Root_CA_v1.crt
Created root/Example_Labs_Root_CA_v1.crl

The private key and certificate are stored in root folder.

Codify the PKI

Execute Terraform on the running Vault server to codify the Intermediate CA, the Issuing CA and the role.

First, you have to configure the Vault provider in providers.tf file. The provider uses VAULT_ADDR and VAULT_TOKEN environment variables to connect to the Vault server.

# providers.tf
terraform {
  required_version = ">= 1.5.3"
  required_providers {
    vault = "~> 3.18.0"
  }
}

provider "vault" {}

Then, create both PKI secret engines using vault_mount resource in main.tf file.

# main.tf
resource "vault_mount" "pki_int" {
  path                  = "pki_int"
  type                  = "pki"
  description           = "PKI engine hosting intermediate CA"
  max_lease_ttl_seconds = 157680000
}

resource "vault_mount" "pki_iss" {
  path                  = "pki_iss"
  type                  = "pki"
  description           = "PKI engine hosting issuing CA"
  max_lease_ttl_seconds = 31536000
}

Initialise and download the Vault provider by initiating Terraform, and create both PKI secret engines.

$ terraform init
$ terraform plan -out plan.out
$ terraform apply plan.out

Issuer modules

Since multiple issuers will be created to build the entire PKI CA hierarchy and during its lifetime when rotating a CA, using a module is appropriate to simplify the configuration.

Since the generation and signing of the certificate can be either external for the Intermediate CA or internal for the Issuing CA, it is preferable to have two separate modules: isser_external_ca and issuer_internal_ca.

Issuer using external CA

First, you have to define the issuervariable to collect the required issuer configuration. The attributes are self-explaining and certificate attribute will be used to pass the certificate generated and signed by the Root CA

# modules/issuer_external_ca/variables.tf
variable "issuer" {
  type = object({
    name             = string
    backend          = string
    organization     = string
    certificate_name = string
    key_type         = string
    key_bits         = number
    certificate      = string
  })
  description = "Issuer configuration"
}

Then, the required resources are the following:

• Create a Private key using the vault_pki_secret_backend_key resource
• Generate a CSR, used by Root CA to generate and sign the certificate, using the vault_pki_secret_backend_intermediate_cert_request resource
• Store the resulting Certificate using the vault_pki_secret_backend_intermediate_set_signed resource
• Name the new issuer using the vault_pki_secret_backend_issuer resource

As the last two steps must be managed after the certificate generation, the related resources are conditional.

# modules/issuer_external_ca/main.tf
# Generate a key
resource "vault_pki_secret_backend_key" "this" {
  backend  = var.issuer.backend
  type     = "internal"
  key_type = var.issuer.key_type
  key_bits = var.issuer.key_bits
  key_name = var.issuer.name
}

# Generate a CSR
resource "vault_pki_secret_backend_intermediate_cert_request" "this" {
  backend      = var.issuer.backend
  type         = "existing"
  organization = var.issuer.organization
  common_name  = var.issuer.certificate_name
  key_ref      = vault_pki_secret_backend_key.this.key_id
}

# Store the signed certificate
resource "vault_pki_secret_backend_intermediate_set_signed" "this" {
  count       = var.issuer.certificate != null ? 1 : 0
  backend     = var.issuer.backend
  certificate = var.issuer.certificate
}

# Name the issuer
resource "vault_pki_secret_backend_issuer" "this" {
  count       = var.issuer.certificate != null ? 1 : 0
  backend     = var.issuer.backend
  issuer_ref  = vault_pki_secret_backend_intermediate_set_signed.this[0].imported_issuers[0]
  issuer_name = var.issuer.name
}

Finally, the outputs are the following:

• the generated CSR
• the certificate and the issuer ID

# modules/issuer_external_ca/outputs.tf
output "csr" {
  description = "PEM-encoded CSR"
  value       = vault_pki_secret_backend_intermediate_cert_request.this.csr
}

output "certificate" {
  description = "PEM-encoded certificate"
  value       = length(vault_pki_secret_backend_intermediate_set_signed.this) > 0 ? vault_pki_secret_backend_intermediate_set_signed.this[0].certificate : null
}

output "issuer_id" {
  description = "ID of the issuer"
  value       = length(vault_pki_secret_backend_intermediate_set_signed.this) > 0 ? vault_pki_secret_backend_intermediate_set_signed.this[0].imported_issuers[0] : null
}

Issuer using internal CA

This module is similar to the previous, except that the CSR can be used directly inside the Vault server that simplify the module.

First, you have to define the issuervariable to collect the required issuer configuration. The attributes are self-explaining and parent_backend attribute will be used to pass the Intermediate CA backend mount path.

# modules/issuer_external_ca/variables.tf
variable "issuer" {
  type = object({
    name             = string
    backend          = string
    organization     = string
    certificate_name = string
    key_type         = string
    key_bits         = number
    parent_backend   = string
  })
  description = "Issuer configuration"
}

Then, the required resources are the following:

• Create a Private key using the vault_pki_secret_backend_key resource
• Generate a CSR using the vault_pki_secret_backend_intermediate_cert_request resource
• Generate and sign a certificate by Intermediate CA using the vault_pki_secret_backend_root_sign_intermediate resource
• Store the resulting Certificate using the vault_pki_secret_backend_intermediate_set_signed resource
• Name the new issuer using the vault_pki_secret_backend_issuer resource

va# Generate a key
resource "vault_pki_secret_backend_key" "this" {
  backend  = var.issuer.backend
  type     = "internal"
  key_type = var.issuer.key_type
  key_bits = var.issuer.key_bits
  key_name = var.issuer.name
}

# Generate a CSR
resource "vault_pki_secret_backend_intermediate_cert_request" "this" {
  backend      = var.issuer.backend
  type         = "existing"
  organization = var.issuer.organization
  common_name  = var.issuer.certificate_name
  key_ref      = vault_pki_secret_backend_key.this.key_id
}

# Sign the CSR using the parent CA
resource "vault_pki_secret_backend_root_sign_intermediate" "this" {
  backend      = var.issuer.parent_backend
  organization = var.issuer.organization
  common_name  = var.issuer.certificate_name
  csr          = vault_pki_secret_backend_intermediate_cert_request.this.csr
}

# Store the signed certificate
resource "vault_pki_secret_backend_intermediate_set_signed" "this" {
  backend     = var.issuer.backend
  certificate = vault_pki_secret_backend_root_sign_intermediate.this.certificate_bundle
}

# Name the issuer
resource "vault_pki_secret_backend_issuer" "this" {
  backend     = var.issuer.backend
  issuer_ref  = vault_pki_secret_backend_intermediate_set_signed.this.imported_issuers[0]
  issuer_name = var.issuer.name
}

Finally, the outputs are the following:

• the generated CSR
• the certificate and the issuer ID

output "csr" {
  description = "PEM-encoded CSR"
  value       = vault_pki_secret_backend_intermediate_cert_request.this.csr
}

output "certificate" {
  description = "PEM-encoded certificate"
  value       = vault_pki_secret_backend_intermediate_set_signed.this.certificate
}

output "issuer_id" {
  description = "ID of the issuer"
  value       = vault_pki_secret_backend_intermediate_set_signed.this.imported_issuers[0]
}

Intermediate CA

Let’s use the issuer_external_ca module to build the Intermediate CA.

First, the certificate attribute must be nullas you need the CSR first to generate and sign the certificate using the Root CA.

# pki_int_v1.1.tf
module "issuer_v1_1" {
  source = "./modules/issuer_external_ca"
  issuer = {
    name             = "v1.1"
    backend          = "pki_int"
    organization     = "Example"
    certificate_name = "Example Labs Intermediate CA v1.1"
    key_type         = "ec"
    key_bits         = 256
    certificate      = null
  }
  depends_on = [vault_mount.pki_int]
}

output "csr_v1_1" {
  description = "CSR for v1.1"
  value       = module.issuer_v1_1.csr
  sensitive   = true
}

output "certificate_v1_1" {
  description = "CRT for v1.1"
  value       = module.issuer_v1_1.certificate
  sensitive   = true
}

output "issuer_v1_1" {
  description = "v1.1 issuer ID"
  value       = module.issuer_v1_1.issuer_id
}

Initialise the new module, plan and apply the plan to generate the CSR.

$ terraform init
$ terraform plan -out plan.out
$ terraform apply plan.out

Extract the CSR from Terraform outputs, then generate and sign the certificate using the Root CA into pki_int_v1.1.crt file.

$ terraform output -json > pki_int_v1.1.json
$ jq -r .csr_v1_1.value pki_int_v1.1.json > pki_int_v1.1.csr
$ certstrap --depot-path root sign \
  --CA "Example Labs Root CA v1" \
  --passphrase "secret" \
  --intermediate \
  --csr pki_int_v1.1.csr \
  --expires "5 years" \
  --path-length 1 \
  --cert pki_int_v1.1.crt \
  "Example Labs Intermediate CA v1.1"

Use the generated certificate to update the pki_int_v1.1.tfconfiguration file by adding certificate = file("${path.root}/pki_int_v1.1.tf").

# pki_int_v1.1.tf
module "issuer_v1_1" {
    ...
    certificate      = file("${path.root}/pki_int_v1.1.tf")
    ...
}

Then, plan and apply the plan to generate to store the resulting Certificate into the Intermediate CA backend.

$ terraform plan -out plan.out
$ terraform apply plan.out

Finally, verify the presence of the new issuer in the Intermedia CA.

$ vault list -detailed pki_int/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
0e31e03b-2955-63b8-810c-1871b347ace7    true          v1.1           d15dfcee-bd40-335b-06fb-6487225aaacf    40:6e:7b:11:e2:02:7e:16:54:d4:31:89:3d:36:4f:dd

The Intermediate CA is now configured, let’s move to the configuration of the Issuing CA.

Issuing CA

Let’s use the issuer_internal_ca module to build the Issuing CA.

First, the parent_backend attribute must be equal to Intermediate CA backend mount path.

# pki_iss_v1.1.1.tf
module "issuer_v1_1_1" {
  source = "./modules/issuer_internal_ca"
  issuer = {
    name             = "v1.1.1"
    backend          = "pki_iss"
    parent_backend   = "pki_int"
    organization     = "Example"
    certificate_name = "Example Labs Issuing CA v1.1.1"
    key_type         = "ec"
    key_bits         = 256
  }
  depends_on = [module.issuer_v1_1]
}

output "certificate_v1_1_1" {
  description = "CRT for v1.1.1"
  value       = module.issuer_v1_1_1.certificate
  sensitive   = true
}

output "issuer_v1_1_1" {
  description = "Issuer ID for v1.1.1"
  value       = module.issuer_v1_1_1.issuer_id
}

Initialise the new module, plan and apply the plan to configure the Issuing CA backend.

$ terraform init
$ terraform plan -out plan.out
$ terraform apply plan.out

Finally, verify the presence of the new issuer in the Issuing CA.

$ vault list -detailed pki_iss/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
36dd2a84-7aeb-89d1-b241-bbc3bfa9e25e    false         n/a            n/a                                     e2:dc:ad:0f:3d:ba:28:e8:f3:62:e9:cb:87:fd:4e:ab
b5d00900-dda8-94fa-0e1f-7a36dc5ff7b1    true          v1.1.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    34:9b:68:13:93:e8:f7:36:07:7e:e0:f5:2d:56:cc:19:ff:33:9a:4a

Role

Finally, let’s create a role for Issuing CA secret engine to issue leaf certificates on the example.com domain.

Create an issuing role using thevault_pki_secret_backend_role resource. Note that, the issuer_ref parameter is set to default.

resource "vault_pki_secret_backend_role" "example_com" {
  backend                     = "pki_iss"
  name                        = "example_com"
  organization                = ["Example"]
  key_type                    = "ec"
  key_bits                    = 256
  max_ttl                     = 3600
  allowed_domains             = ["example.com"]
  allow_subdomains            = true
  allow_ip_sans               = true
  allow_wildcard_certificates = false
  issuer_ref                  = "default"
}

Plan and apply the plan to configure the Role.

$ terraform plan -out plan.out
$ terraform apply plan.out

Finally, verify the Role creation by issuing a certificate for sample.example.com domain.

$ vault write -format=json \
    pki_iss/issue/example_com \
    common_name="sample.example.com" \
    ttl="5m"

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            16:05:ec:8e:cb:c1:8c:63:3d:09:71:d5:a1:1d:ed:c4:47:1b:91:54
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: O = Example, CN = Example Labs Issuing CA v1.1.1
        Validity
            Not Before: Jul 16 12:08:58 2023 GMT
            Not After : Jul 16 12:14:28 2023 GMT
        Subject: O = Example, CN = sample.example.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:b6:14:e9:c3:0a:79:fa:20:57:5a:d6:9e:5b:9f:
                    73:a3:61:27:57:5d:56:b3:3f:bc:8e:13:16:70:78:
                    b9:25:50:ff:c2:af:61:35:81:27:d8:9e:20:11:5a:
                    86:1d:db:6e:ed:8b:f7:9d:24:a4:28:1b:cd:c8:4d:
                    35:dd:14:7b:cd
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                4D:26:FB:D5:11:3C:25:45:51:08:01:00:0A:F0:A8:27:DC:6C:C3:67
            X509v3 Authority Key Identifier:
                6B:EB:EB:F7:20:E8:E2:C3:DE:84:C9:DE:81:80:89:D5:E0:21:37:F4
            X509v3 Subject Alternative Name:
                DNS:sample.example.com
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:46:02:21:00:ae:03:7f:2d:b8:2d:0d:a1:82:b3:4e:3a:c0:
        49:b0:3b:0f:e0:81:ea:df:7e:2b:8e:dc:c5:4c:da:2b:96:c3:
        6a:02:21:00:ae:51:f8:b0:a8:6a:07:5f:4b:6b:43:22:a6:5f:
        73:a6:02:c1:81:f1:d0:87:f7:95:a9:47:8c:6c:21:05:ee:9f

Having successfully codified the PKI using Terraform, it is now time to proceed with the rotations of each CA.

Rotate the Issuing CA

First, you need to create a new issuer for the Issuing CA.

So, copy the file pki_iss_v1.1.1.tf configuration file into pki_iss_v1.1.2.tf, and replace v1.1.1 by v1.1.2.

# pki_iss_v1.1.2.tf
module "issuer_v1_1_2" {
  source = "./modules/issuer_internal_ca"
  issuer = {
    name             = "v1.1.2"
    backend          = "pki_iss"
    parent_backend   = "pki_int"
    organization     = "Example"
    certificate_name = "Example Labs Issuing CA v1.1.2"
    key_type         = "ec"
    key_bits         = 256
  }
  depends_on = [module.issuer_v1_1]
}

output "certificate_v1_1_2" {
  description = "CRT for v1.1.2"
  value       = module.issuer_v1_1_2.certificate
  sensitive   = true
}

output "issuer_v1_1_2" {
  description = "Issuer ID for v1.1.2"
  value       = module.issuer_v1_1_2.issuer_id
}

Initialise the new module, plan and apply the plan to configure the Issuing CA backend.

$ terraform init
$ terraform plan -out plan.out
$ terraform apply plan.out

Then, verify the presence of the new issuer in the Issuing CA.

$ vault list -detailed pki_iss/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
36dd2a84-7aeb-89d1-b241-bbc3bfa9e25e    false         n/a            n/a                                     e2:dc:ad:0f:3d:ba:28:e8:f3:62:e9:cb:87:fd:4e:ab
b5d00900-dda8-94fa-0e1f-7a36dc5ff7b1    true          v1.1.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    34:9b:68:13:93:e8:f7:36:07:7e:e0:f5:2d:56:cc:19:ff:33:9a:4a
e16fae6c-2b43-034f-e912-9a9fd859d031    false         v1.1.2         f1674fc0-3cc4-38a7-68d5-991582d23c56    3b:de:74:75:6b:48:f4:f8:bc:c7:47:91:ee:21:91:5b:70:bb:07:40

Default issuer is still v1.1.1

The new issuer hasn’t been set as the default one. So, the role example_com will continue to issue certificates with v1.1.1 issuer.

To update the issuer configuration, use the vault_pki_secret_backend_config_issuers resource

# main.tf
resource "vault_pki_secret_backend_config_issuers" "iss" {
  backend = "pki_iss"
  default = "e16fae6c-2b43-034f-e912-9a9fd859d031"
}

Plan and apply the plan to configure the default issuer.

$ terraform plan -out plan.out
$ terraform apply plan.out

Default issuer is v1.1.2

Finally, verify the default issuer configuration in the Issuing CA, and issue a certificate for sample.example.com domain.

$ vault list -detailed pki_iss/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
36dd2a84-7aeb-89d1-b241-bbc3bfa9e25e    false         n/a            n/a                                     e2:dc:ad:0f:3d:ba:28:e8:f3:62:e9:cb:87:fd:4e:ab
b5d00900-dda8-94fa-0e1f-7a36dc5ff7b1    false         v1.1.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    34:9b:68:13:93:e8:f7:36:07:7e:e0:f5:2d:56:cc:19:ff:33:9a:4a
e16fae6c-2b43-034f-e912-9a9fd859d031    true          v1.1.2         f1674fc0-3cc4-38a7-68d5-991582d23c56    3b:de:74:75:6b:48:f4:f8:bc:c7:47:91:ee:21:91:5b:70:bb:07:40

$ vault write -format=json \
    pki_iss/issue/example_com \
    common_name="sample.example.com" \
    ttl="5m"

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            2c:e6:f1:3f:19:c3:3e:8c:a1:c4:45:e2:6b:98:27:77:3f:a3:0c:ec
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: O = Example, CN = Example Labs Issuing CA v1.1.2
        Validity
            Not Before: Jul 16 12:25:02 2023 GMT
            Not After : Jul 16 12:30:32 2023 GMT
        Subject: O = Example, CN = sample.example.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:78:72:51:34:60:a7:8e:ff:d4:7b:26:d0:7d:54:
                    bc:9d:d5:8b:dd:f0:89:28:eb:5c:66:80:28:59:0a:
                    28:49:4f:17:52:d6:5b:25:86:ac:e2:3e:11:86:df:
                    e1:42:b6:51:73:55:7c:43:c5:58:28:3f:20:70:61:
                    f7:50:59:b8:80
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                C8:4C:FF:03:B4:EC:76:F7:85:42:0C:DB:8C:16:AD:8F:3D:4A:68:E1
            X509v3 Authority Key Identifier:
                6B:EB:EB:F7:20:E8:E2:C3:DE:84:C9:DE:81:80:89:D5:E0:21:37:F4
            X509v3 Subject Alternative Name:
                DNS:sample.example.com
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:44:02:20:43:ba:c4:4c:49:c9:b7:d9:72:1b:e3:f6:a6:e8:
        64:77:b2:01:fd:d2:7f:4c:8f:80:e0:04:4d:67:28:b8:85:46:
        02:20:69:5e:81:32:9c:b4:12:79:bc:86:27:69:eb:ad:24:71:
        61:a1:f6:68:78:39:d2:50:96:f5:2e:96:cd:0c:9b:d6

In the next section, the Intermediate CA is rotated.

Rotate the Intermediate CA

Now, let’s update the Intermediate CA.

Generate v1.2 issuer for Intermediate CA

First, you need to create a new issuer for the Intermediate CA.

So, copy the file pki_int_v1.1.tf configuration file into pki_int_v1.2.tf, and replace v1.1 by v1.2.

# pki_int_v1.2.tf
module "issuer_v1_2" {
  source = "./modules/issuer_external_ca"
  issuer = {
    name             = "v1.2"
    backend          = "pki_int"
    organization     = "Example"
    certificate_name = "Example Labs Intermediate CA v1.2"
    key_type         = "ec"
    key_bits         = 256
    certificate      = null
  }
  depends_on = [vault_mount.pki_int]
}

output "csr_v1_2" {
  description = "CSR for v1.2"
  value       = module.issuer_v1_2.csr
  sensitive   = true
}

output "certificate_v1_2" {
  description = "CRT for v1.2"
  value       = module.issuer_v1_2.certificate
  sensitive   = true
}

output "issuer_v1_2" {
  description = "v1.2 issuer ID"
  value       = module.issuer_v1_2.issuer_id
}

Initialise the new module, plan and apply the plan to generate the CSR.

$ terraform init
$ terraform plan -out plan.out
$ terraform apply plan.out

Extract the CSR from Terraform outputs, then generate and sign the certificate using the Root CA into pki_int_v1.2.crt file.

$ terraform output -json > pki_int_v1.2.json
$ jq -r .csr_v1_2.value pki_int_v1.2.json > pki_int_v1.2.csr
$ certstrap --depot-path root sign \
  --CA "Example Labs Root CA v1" \
  --passphrase "secret" \
  --intermediate \
  --csr pki_int_v1.2.csr \
  --expires "5 years" \
  --path-length 1 \
  --cert pki_int_v1.2.crt \
  "Example Labs Intermediate CA v1.2"

Use the generated certificate to update the pki_int_v1.2.tfconfiguration file by adding certificate = file("${path.root}/pki_int_v1.2.tf").

# pki_int_v1.2.tf
module "issuer_v1_2" {
    ...
    certificate      = file("${path.root}/pki_int_v1.2.tf")
    ...
}

Then, plan and apply the plan to generate to store the resulting Certificate into the Intermediate CA backend.

$ terraform plan -out plan.out
$ terraform apply plan.out

Finally, verify the presence of the new issuer in the Intermedia CA.

$ vault list -detailed pki_int/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
0e31e03b-2955-63b8-810c-1871b347ace7    true          v1.1           d15dfcee-bd40-335b-06fb-6487225aaacf    40:6e:7b:11:e2:02:7e:16:54:d4:31:89:3d:36:4f:dd
c17e8d1d-66b2-990d-61da-4c710ffcd3a6    false         v1.2           59344008-f547-1329-b9e5-12a35f220d79    60:fb:1a:ab:59:2e:22:53:9f:2b:88:af:18:b1:05:71

The new issuer hasn’t been set as the default one.

To update the issuer configuration, use the vault_pki_secret_backend_config_issuers resource

# main.tf
resource "vault_pki_secret_backend_config_issuers" "int" {
  backend = "pki_int"
  default = "c17e8d1d-66b2-990d-61da-4c710ffcd3a6"
}

Plan and apply the plan to configure the default issuer.

$ terraform plan -out plan.out
$ terraform apply plan.out

Finally, verify the default issuer configuration in the Intermediate CA.

$ vault list -detailed pki_int/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
0e31e03b-2955-63b8-810c-1871b347ace7    false         v1.1           d15dfcee-bd40-335b-06fb-6487225aaacf    40:6e:7b:11:e2:02:7e:16:54:d4:31:89:3d:36:4f:dd
c17e8d1d-66b2-990d-61da-4c710ffcd3a6    true          v1.2           59344008-f547-1329-b9e5-12a35f220d79    60:fb:1a:ab:59:2e:22:53:9f:2b:88:af:18:b1:05:71

The Intermediate CA is now updated, let’s move to the update of the Issuing CA to use this new issuer.

Update the Issuing CA

First, you need to create a new issuer for the Issuing CA.

Generate v1.2.1 issuer for Issuing CA

So, copy the file pki_iss_v1.1.1.tf configuration file into pki_iss_v1.2.1.tf, and replace v1.1.1 by v1.2.1.

# pki_iss_v1.2.1.tf
module "issuer_v1_2_1" {
  source = "./modules/issuer_internal_ca"
  issuer = {
    name             = "v1.2.1"
    backend          = "pki_iss"
    parent_backend   = "pki_int"
    organization     = "Example"
    certificate_name = "Example Labs Issuing CA v1.2.1"
    key_type         = "ec"
    key_bits         = 256
  }
  depends_on = [module.issuer_v1_2]
}

output "certificate_v1_2_1" {
  description = "CRT for v1.2.1"
  value       = module.issuer_v1_2_1.certificate
  sensitive   = true
}

output "issuer_v1_2_1" {
  description = "Issuer ID for v1.2.1"
  value       = module.issuer_v1_2_1.issuer_id
}

Initialise the new module, plan and apply the plan to configure the Issuing CA backend.

$ terraform init
$ terraform plan -out plan.out
$ terraform apply plan.out

Then, verify the presence of the new issuer in the Issuing CA.

$ vault list -detailed pki_iss/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
187fe1c5-ddee-aa2d-0b26-65295c4147a1    false         n/a            n/a                                     60:fb:1a:ab:59:2e:22:53:9f:2b:88:af:18:b1:05:71
36dd2a84-7aeb-89d1-b241-bbc3bfa9e25e    false         n/a            n/a                                     e2:dc:ad:0f:3d:ba:28:e8:f3:62:e9:cb:87:fd:4e:ab
b5d00900-dda8-94fa-0e1f-7a36dc5ff7b1    false         v1.1.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    34:9b:68:13:93:e8:f7:36:07:7e:e0:f5:2d:56:cc:19:ff:33:9a:4a
e16fae6c-2b43-034f-e912-9a9fd859d031    true          v1.1.2         f1674fc0-3cc4-38a7-68d5-991582d23c56    3b:de:74:75:6b:48:f4:f8:bc:c7:47:91:ee:21:91:5b:70:bb:07:40
f360913a-dc2d-ea2c-b2f1-653110f505df    false         v1.2.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    3f:73:a1:54:34:56:11:15:24:74:ad:36:56:6c:dd:cc:29:54:ee:5c

The new issuer hasn’t been set as the default one. So, the role example_com will continue to issue certificates with v1.1.2 issuer.

To update the issuer configuration, use the vault_pki_secret_backend_config_issuers resource

# main.tf
resource "vault_pki_secret_backend_config_issuers" "iss" {
  backend = "pki_iss"
  default = "f360913a-dc2d-ea2c-b2f1-653110f505df"
}

Plan and apply the plan to configure the default issuer.

$ terraform plan -out plan.out
$ terraform apply plan.out

Default issuer is v1.2.1

Finally, verify the default issuer configuration in the Issuing CA, and issue a certificate for sample.example.com domain.

$ vault list -detailed pki_iss/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
187fe1c5-ddee-aa2d-0b26-65295c4147a1    false         n/a            n/a                                     60:fb:1a:ab:59:2e:22:53:9f:2b:88:af:18:b1:05:71
36dd2a84-7aeb-89d1-b241-bbc3bfa9e25e    false         n/a            n/a                                     e2:dc:ad:0f:3d:ba:28:e8:f3:62:e9:cb:87:fd:4e:ab
b5d00900-dda8-94fa-0e1f-7a36dc5ff7b1    false         v1.1.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    34:9b:68:13:93:e8:f7:36:07:7e:e0:f5:2d:56:cc:19:ff:33:9a:4a
e16fae6c-2b43-034f-e912-9a9fd859d031    false         v1.1.2         f1674fc0-3cc4-38a7-68d5-991582d23c56    3b:de:74:75:6b:48:f4:f8:bc:c7:47:91:ee:21:91:5b:70:bb:07:40
f360913a-dc2d-ea2c-b2f1-653110f505df    true          v1.2.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    3f:73:a1:54:34:56:11:15:24:74:ad:36:56:6c:dd:cc:29:54:ee:5c

$ vault write -format=json \
    pki_iss/issue/example_com \
    common_name="sample.example.com" \
    ttl="5m"

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            63:47:5d:72:1b:19:6e:13:4d:27:c0:85:7a:6a:ed:af:c6:33:4e:28
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: O = Example, CN = Example Labs Issuing CA v1.2.1
        Validity
            Not Before: Jul 16 12:57:35 2023 GMT
            Not After : Jul 16 13:03:05 2023 GMT
        Subject: O = Example, CN = sample.example.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:40:cf:87:c9:aa:d3:1a:7b:d6:21:09:10:33:8d:
                    f2:8d:67:e2:87:b1:47:7c:3c:a4:eb:2c:eb:62:d3:
                    eb:71:ba:7d:28:f8:6c:e5:08:1b:fa:59:a7:2e:0b:
                    0a:c7:7e:82:fb:7e:da:93:1e:24:86:a7:bc:de:e7:
                    59:7a:47:69:1b
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                B5:9F:10:FB:65:ED:FA:2C:87:85:A0:86:F1:4C:26:E4:27:D0:0B:34
            X509v3 Authority Key Identifier:
                6B:EB:EB:F7:20:E8:E2:C3:DE:84:C9:DE:81:80:89:D5:E0:21:37:F4
            X509v3 Subject Alternative Name:
                DNS:sample.example.com
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:44:02:20:35:2d:c2:d3:1f:1f:5c:b9:b8:0b:e7:6b:0b:7e:
        93:3d:71:b5:a1:15:b2:e0:d3:c9:49:ce:04:78:06:f0:1c:d7:
        02:20:2b:1b:21:c3:de:2c:77:66:ba:ed:b1:58:1f:dc:62:ba:
        3e:64:ac:71:04:4f:c2:69:42:47:2e:4e:b4:f9:91:a0

In the next section, the Root CA is rotated.

Rotate the Root CA

Now, let’s update the Root CA.

Generate Root CA v2

First, generate a self-signed Root CA v2 using certstrap

$ certstrap --depot-path root \
    init \
    --organization "Example" \
    --common-name "Example Labs Root CA v2" \
    --expires "10 years" \
    --curve P-256 \
    --path-length 2 \
    --passphrase "secret"
Created root/Example_Labs_Root_CA_v2.key (encrypted by passphrase)
Created root/Example_Labs_Root_CA_v2.crt
Created root/Example_Labs_Root_CA_v2.crl

The Root CA is now updated, let’s move to the update of the Intermediate CA to use this new Root CA.

Update the Intermediate CA

Now, let’s update the Intermediate CA.

Generate v2.1 issuer for Intermediate CA

First, you need to create a new issuer for the Intermediate CA.

So, copy the file pki_int_v1.1.tf configuration file into pki_int_v2.1.tf, and replace v1.1 by v2.1.

# pki_int_v2.1.tf
module "issuer_v2_1" {
  source = "./modules/issuer_external_ca"
  issuer = {
    name             = "v2.1"
    backend          = "pki_int"
    organization     = "Example"
    certificate_name = "Example Labs Intermediate CA v2.1"
    key_type         = "ec"
    key_bits         = 256
    certificate      = null
  }
  depends_on = [vault_mount.pki_int]
}

output "csr_v2_1" {
  description = "CSR for v2.1"
  value       = module.issuer_v2_1.csr
  sensitive   = true
}

output "certificate_v2_1" {
  description = "CRT for v2.1"
  value       = module.issuer_v2_1.certificate
  sensitive   = true
}

output "issuer_v2_1" {
  description = "v2.1 issuer ID"
  value       = module.issuer_v2_1.issuer_id
}

Initialise the new module, plan and apply the plan to generate the CSR.

$ terraform init
$ terraform plan -out plan.out
$ terraform apply plan.out

Extract the CSR from Terraform outputs, then generate and sign the certificate using the new Root CA into pki_int_v2.1.crt file.

$ terraform output -json > pki_int_v2.1.json
$ jq -r .csr_v2_1.value pki_int_v2.1.json > pki_int_v2.1.csr
$ certstrap --depot-path root sign \
  --CA "Example Labs Root CA v2" \
  --passphrase "secret" \
  --intermediate \
  --csr pki_int_v2.1.csr \
  --expires "5 years" \
  --path-length 1 \
  --cert pki_int_v2.1.crt \
  "Example Labs Intermediate CA v2.1"

Use the generated certificate to update the pki_int_v2.1.tfconfiguration file by adding certificate = file("${path.root}/pki_int_v2.1.tf").

# pki_int_v2.1.tf
module "issuer_v2_1" {
    ...
    certificate      = file("${path.root}/pki_int_v2.1.tf")
    ...
}

Then, plan and apply the plan to generate to store the resulting Certificate into the Intermediate CA backend.

$ terraform plan -out plan.out
$ terraform apply plan.out

Finally, verify the presence of the new issuer in the Intermedia CA.

$ vault list -detailed pki_int/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
0e31e03b-2955-63b8-810c-1871b347ace7    false         v1.1           d15dfcee-bd40-335b-06fb-6487225aaacf    40:6e:7b:11:e2:02:7e:16:54:d4:31:89:3d:36:4f:dd
c17e8d1d-66b2-990d-61da-4c710ffcd3a6    true          v1.2           59344008-f547-1329-b9e5-12a35f220d79    60:fb:1a:ab:59:2e:22:53:9f:2b:88:af:18:b1:05:71
02ed0617-823f-00b5-56e7-e71764c1576b    false         v2.1           59344008-f547-1329-b9e5-12a35f220d79    b2:a4:29:13:a9:87:be:00:d2:4c:a0:41:6a:f9:2e:0c

The new issuer hasn’t been set as the default one.

To update the issuer configuration, use the vault_pki_secret_backend_config_issuers resource

# main.tf
resource "vault_pki_secret_backend_config_issuers" "int" {
  backend = "pki_int"
  default = "02ed0617-823f-00b5-56e7-e71764c1576b"
}

Plan and apply the plan to configure the default issuer.

$ terraform plan -out plan.out
$ terraform apply plan.out

Finally, verify the default issuer configuration in the Intermediate CA.

$ vault list -detailed pki_int/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
0e31e03b-2955-63b8-810c-1871b347ace7    false         v1.1           d15dfcee-bd40-335b-06fb-6487225aaacf    40:6e:7b:11:e2:02:7e:16:54:d4:31:89:3d:36:4f:dd
c17e8d1d-66b2-990d-61da-4c710ffcd3a6    false         v1.2           59344008-f547-1329-b9e5-12a35f220d79    60:fb:1a:ab:59:2e:22:53:9f:2b:88:af:18:b1:05:71
02ed0617-823f-00b5-56e7-e71764c1576b    true          v2.1           59344008-f547-1329-b9e5-12a35f220d79    b2:a4:29:13:a9:87:be:00:d2:4c:a0:41:6a:f9:2e:0c

The Intermediate CA is now updated, let’s move to the update of the Issuing CA to use this new issuer.

Update the Issuing CA

First, you need to create a new issuer for the Issuing CA.

Generate v2.1.1 issuer for Issuing CA

So, copy the file pki_iss_v1.1.1.tf configuration file into pki_iss_v2.1.1.tf, and replace v1.1.1 by v2.1.1.

# pki_iss_v2.1.1.tf
module "issuer_v2_1_1" {
  source = "./modules/issuer_internal_ca"
  issuer = {
    name             = "v2.1.1"
    backend          = "pki_iss"
    parent_backend   = "pki_int"
    organization     = "Example"
    certificate_name = "Example Labs Issuing CA v2.1.1"
    key_type         = "ec"
    key_bits         = 256
  }
  depends_on = [module.issuer_v2_1]
}

output "certificate_v2_1_1" {
  description = "CRT for v2.1.1"
  value       = module.issuer_v2_1_1.certificate
  sensitive   = true
}

output "issuer_v2_1_1" {
  description = "Issuer ID for v2.1.1"
  value       = module.issuer_v2_1_1.issuer_id
}

Initialise the new module, plan and apply the plan to configure the Issuing CA backend.

$ terraform init
$ terraform plan -out plan.out
$ terraform apply plan.out

Then, verify the presence of the new issuer in the Issuing CA.

$ vault list -detailed pki_iss/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
187fe1c5-ddee-aa2d-0b26-65295c4147a1    false         n/a            n/a                                     60:fb:1a:ab:59:2e:22:53:9f:2b:88:af:18:b1:05:71
36dd2a84-7aeb-89d1-b241-bbc3bfa9e25e    false         n/a            n/a                                     e2:dc:ad:0f:3d:ba:28:e8:f3:62:e9:cb:87:fd:4e:ab
c53cb6f8-f9b6-525f-3503-457b0a82c349    false         n/a            n/a                                     b2:a4:29:13:a9:87:be:00:d2:4c:a0:41:6a:f9:2e:0c
b5d00900-dda8-94fa-0e1f-7a36dc5ff7b1    false         v1.1.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    34:9b:68:13:93:e8:f7:36:07:7e:e0:f5:2d:56:cc:19:ff:33:9a:4a
e16fae6c-2b43-034f-e912-9a9fd859d031    false         v1.1.2         f1674fc0-3cc4-38a7-68d5-991582d23c56    3b:de:74:75:6b:48:f4:f8:bc:c7:47:91:ee:21:91:5b:70:bb:07:40
f360913a-dc2d-ea2c-b2f1-653110f505df    true          v1.2.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    3f:73:a1:54:34:56:11:15:24:74:ad:36:56:6c:dd:cc:29:54:ee:5c
5a1fe736-4042-427a-a855-4597559a54d3    false         v2.1.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    4a:c6:71:32:ec:c4:6d:e7:cc:e7:ff:26:88:11:d2:f3:cc:5c:cb:4a

The new issuer hasn’t been set as the default one. So, the role example_com will continue to issue certificates with v1.2.1 issuer.

To update the issuer configuration, use the vault_pki_secret_backend_config_issuers resource

# main.tf
resource "vault_pki_secret_backend_config_issuers" "iss" {
  backend = "pki_iss"
  default = "5a1fe736-4042-427a-a855-4597559a54d3"
}

Plan and apply the plan to configure the default issuer.

$ terraform plan -out plan.out
$ terraform apply plan.out

Default issuer is v2.1.1

Finally, verify the default issuer configuration in the Issuing CA, and issue a certificate for sample.example.com domain.

$ vault list -detailed pki_iss/issuers
Keys                                    is_default    issuer_name    key_id                                  serial_number
----                                    ----------    -----------    ------                                  -------------
187fe1c5-ddee-aa2d-0b26-65295c4147a1    false         n/a            n/a                                     60:fb:1a:ab:59:2e:22:53:9f:2b:88:af:18:b1:05:71
36dd2a84-7aeb-89d1-b241-bbc3bfa9e25e    false         n/a            n/a                                     e2:dc:ad:0f:3d:ba:28:e8:f3:62:e9:cb:87:fd:4e:ab
c53cb6f8-f9b6-525f-3503-457b0a82c349    false         n/a            n/a                                     b2:a4:29:13:a9:87:be:00:d2:4c:a0:41:6a:f9:2e:0c
b5d00900-dda8-94fa-0e1f-7a36dc5ff7b1    false         v1.1.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    34:9b:68:13:93:e8:f7:36:07:7e:e0:f5:2d:56:cc:19:ff:33:9a:4a
e16fae6c-2b43-034f-e912-9a9fd859d031    false         v1.1.2         f1674fc0-3cc4-38a7-68d5-991582d23c56    3b:de:74:75:6b:48:f4:f8:bc:c7:47:91:ee:21:91:5b:70:bb:07:40
f360913a-dc2d-ea2c-b2f1-653110f505df    false         v1.2.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    3f:73:a1:54:34:56:11:15:24:74:ad:36:56:6c:dd:cc:29:54:ee:5c
5a1fe736-4042-427a-a855-4597559a54d3    true          v2.1.1         f1674fc0-3cc4-38a7-68d5-991582d23c56    4a:c6:71:32:ec:c4:6d:e7:cc:e7:ff:26:88:11:d2:f3:cc:5c:cb:4a

$ vault write -format=json \
    pki_iss/issue/example_com \
    common_name="sample.example.com" \
    ttl="5m"

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            52:9b:7f:2c:b5:27:09:f3:69:74:4b:02:e2:83:ce:6e:50:d9:62:cc
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: O = Example, CN = Example Labs Issuing CA v2.1.1
        Validity
            Not Before: Jul 16 13:18:45 2023 GMT
            Not After : Jul 16 13:24:15 2023 GMT
        Subject: O = Example, CN = sample.example.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:e7:a7:58:a8:10:92:50:83:50:0d:df:e2:f8:c2:
                    92:fe:22:5c:32:88:0b:7d:4d:ae:08:f0:5b:85:81:
                    53:90:c0:72:64:25:9f:1f:f3:b2:3f:73:35:9d:83:
                    4d:ec:c6:ff:b3:82:9d:27:f2:ec:60:8f:95:dc:02:
                    fb:d7:dd:c9:8a
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                9F:01:C8:21:F8:56:9E:D2:4D:97:9C:30:75:A3:ED:05:9F:CD:51:60
            X509v3 Authority Key Identifier:
                6B:EB:EB:F7:20:E8:E2:C3:DE:84:C9:DE:81:80:89:D5:E0:21:37:F4
            X509v3 Subject Alternative Name:
                DNS:sample.example.com
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:46:02:21:00:fd:78:9a:3a:fa:be:06:f5:b5:a2:cf:90:e8:
        8b:9e:b5:ba:84:27:93:0c:63:fa:ba:4f:3d:9e:19:a8:63:e8:
        9d:02:21:00:9e:86:7a:81:f4:70:69:1b:2c:ae:5f:7e:0d:c9:
        c8:8e:06:0d:90:b3:2f:4f:bd:6e:1e:d2:41:dd:0c:b0:36:b8

As a result, the Root CA, the Intermediate CA and the Issuing CA have been rotated, and the generation of leaf certificates is passed on seamlessly to the new issuer.

The following repository is the companion of this post:
https://github.com/sestegra/vault_pki_terraform

Conclusion

In summary, using Terraform to turn Vault’s Internal PKI into code has many advantages. It helps organizations ensure they always use the same secure practices by automating how they get and handle certificates and certificate authorities.

Keeping track of infrastructure changes and controlling different versions makes it easier to maintain and work together as a team. Terraform’s flexibility and Vault’s strength combine to make a powerful method that simplifies things and improves overall security.

Nevertheless, the PKI codification requires multiple states, especially when the Root CA is offline and when managing the CA rotations. So, multiple “Write, Plan, Apply” cycles are required when using only Terraform.

Is the complete automation of the PKI lifecycle through only Terraform optimal? For instance, can Consul-Terraform-Sync be another approach to manage dynamicity better? I will check this hypothesis and will write a future article on this subject.

References

GitHub - sestegra/vault_pki_terraform

PKI - Secrets Engines - HTTP API | Vault | HashiCorp Developer

Terraform Registry