HashiCorp Vault

自建金鑰管理最佳實踐

Che Chia Chang | Vault 鐵人賽 workshop

about me

About Me

Outline

from dev to prod-ready

  • terraform deployment IaC
    • on aws / azure / gcp / k8s
    • 如何開始
  • terraform configuration IaC
    • IaC everything
      • secret backends / auth method / role / policy / audit …
    • 工作流程自動化 gitflow / tested / automation

Vault 基礎的學習資源

production deploy checklist

  • End-to-End TLS
  • Single Tenancy
  • Enable Auditing
  • Immutable Upgrades
  • Upgrade Frequently
  • Restrict Storage Access

Infrastructure as Code

導入 IaC ，做到頻繁且安全的 vault 更新，但同時又要有效率甚至全自動化

  • multiple env: dev, stag, prod
  • programable / reusable: 標準化，可重複使用的 code
  • tested infrastructure

如何開始 IaC for Vault

  • immutable Vault server
  • storage backend
  • load balancer
  • security group / firewall rule

https://www.terraform.io/

Deploy on public cloud

Deploy terraform-aws-vault

terraform-aws-vault

  • ELB -> AWS Autoscaling Group -> EC2
  • Backend: consul cluster
  • Vault AMI
  • security group
  • tested by hashicorp with terratest

terraform-aws-vault

terraform-aws-vault-starter

Integrated storage

On Kubernetes

argocd + vault helm chart

server 與 injector 建議分開兩個 argocd applicatoin / helm release獨立deploy

Deploy On Kubernetes

warning

回到 Outline: prod-ready

  • terraform configuration IaC
    • auth method / config / role
    • secret mounts
    • policy
      • team / people
      • application / service account
    • audit log

Vault Configuration IaC

  • VCS / PR reviewed
  • well-formed / linted / no-typo
  • multiple env: dev, stag, prod
  • programable / reusable: 標準化，可重複使用的 code
  • tested
    • 可以為 terraform code 寫測試
    • 可以寫整合測試腳本測試 vault dev server
  • automation
  • auditable

hashicorp official tutorial

tee admin-policy.hcl <<EOF
# Read system health check
path "sys/health"
{
  capabilities = ["read", "sudo"]
}

# List existing policies
path "sys/policies/acl"
{
  capabilities = ["list"]
}
EOF

hashicorp official tutorial

vault policy write admin admin-policy.hcl

vault policy list

vault policy read admin

How to use

prerequisites: vault, terraform, terragrunt

git clone git@github.com:chechiachang/vault-playground.git

cd vault-playground/usage/03-terraform-lives/

terragrunt init
terragrunt plan

VCS & PR review

  • local lint with git pre-commit hook
  • PR
    • lint check
    • pipeline module test terraform test on github action
    • integration test against vault dev server
    • review
  • merge / automation
    • apply to dev environment automatically
    • release candidate tag will apply to stag automatically
    • release tag will push to pre-production and production

Multiple environment

  • dev -> stag -> prod 環境很接近
    • IaC + config as code
  • 搭設新環境只需1分鐘(VM)，甚至數秒(container)，可以進行大量的測試

Test

  • IaC code 可以使用 terraform test
  • config as code 可以使用 terraform test
  • live infrastructure 可以使用 terratest

Test Example

resource "test_assertions" "main" {
  component = "main"
  equal "mount_path" {
    description = "default mount_path is ${local.mount_path}"
    got         = local.mount_path
    want        = local.mount_path
  }

  equal "max_versions" {
    description = "default max_versions is 10"
    got         = local.max_versions
    want        = 10
  }

  equal "delete_version_after" {
    description = "default delete_version_after is 10"
    got         = local.delete_version_after
    want        = 12600
  }

Policy as code for vault

Gitflow & automation

  • PR
  • PR merged
  • PR apply
    • apply to dev environment automatically with gitflow
    • release candidate tag will apply to stag automatically
    • release tag will push to pre-production and production

Summary

  • 使用 IaC deploy vault，並依照團隊需求逐步調整 infra
  • 使用 IaC 管理 vault 內部一切 config

Questions?