Resource as Code for K8s Object

如何管理 k8s object

Che Chia Chang

Outline

Manage Kubernetes Objects in Gitflow

  • kubectl
  • helm chart
  • argocd (gitflow)
    • applicationset
  • test

Kubectl

# Imperative commands 
kubectl create deployment nginx --image nginx

# Imperative object configuration 
kubectl create -f nginx.yaml

# Declarative object configuration
kubectl apply -R -f configs/

https://kubernetes.io/docs/concepts/overview/working-with-objects/object-management/#management-techniques

Issue

# Imperative commands 
kubectl create deployment nginx --image nginx

# Imperative object configuration 
kubectl create -f nginx.yaml

# Declarative object configuration
kubectl apply -R -f configs/

Issue

  • change review / diff before apply
  • source of live record
  • template / repetitive apply
  • sync local to live

Declarative object configuration

nginx
├── deployment.yaml
├── ingress.yaml
└── service.yaml
redis
├── deployment.yaml
├── ingress.yaml
└── service.yaml
microservice-a b c ...

Helm chart

k8s object 的開發，打包，測試，release

  • k8s 十分強大，享受 orchestration
  • k8s object 變得太複雜
  • 標準化 template，release + upgrade

Helm Chart Library

helm repo list

NAME       URL
bitnami    https://charts.bitnami.com/bitnami
argocd     https://argoproj.github.io/argo-helm
chaos-mesh https://charts.chaos-mesh.org

helm 生態系

ex. helmfile

repositories:
- name: argocd
  url: https://argoproj.github.io/argo-helm

helmDefaults:
  kubeContext: general
  #verify: true
  wait: true
  timeout: 300

context: general

releases:
- name: argocd
  namespace: argocd
  chart: argocd/argo-cd
  version: 5.31.0
  values:
  - values/argocd.yaml
- name: redis

- name: mysql

https://github.com/cdwv/awesome-helm

更高層級的封裝

api-services
├── nginx
├── mysql
└── ingress -> nlb
daemon-services
├── redis
├── mysql
└── kafka

微服務

微服務不是問題，微服務底下的 k8s object 才是問題

  • 可以快速，標準化的產生經過測試，微服務單元

Issues

  • V change review / diff before apply
  • source of live record
  • V template / repetitive apply
  • sync local to live

Argo CD

Declarative GitOps CD for Kubernetes

Why Argo CD?

Application definitions, configurations, and environments should be declarative and version controlled. Application deployment and lifecycle management should be automated, auditable, and easy to understand.

Argo CD

  • gitflow / git repository
  • argocd application sync file from repository
  • argocd controller 自動化的確保 sync
    • un-sync 自動化處理
    • 無法 sync 時通知

applicationset

git generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: general-argocd
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: https://github.com/chechiachang/azure-argo
      revision: master
      directories:
      - path: clusters/dev/dev-general/argocd/*
  template:
    metadata:
      name: 'dev-general-argocd-{{path.basename}}'
      labels:
        environment: dev
        type: infra
        function: argocd
        cluster: dev-general
    spec:
      project: default
      syncPolicy:
        automated:
          prune: true
      source:
        repoURL: https://github.com/chechiachang/azure-argo
        targetRevision: master
        path: 'clusters/dev/dev-general/argocd/{{path.basename}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd

applicationset

dev-general
├── default/redis
├── default/mysql
├── default/my-app
├── nginx-ingress/nginx-ingress
└── argocd/argocd
stag-general
prod-general

cluster-wide 的 k8s object

cluster-wide 的 k8s object 也很適合

  • 使用 helm template helper 來管理 value.yaml label / annotation / env / …
  • namespace
  • rbac

Issues

  • V change review / diff before apply
  • V source of live record
  • V template / repetitive apply
  • V sync local to live

More Issues: multi-hybrid cluster

  • multiple k8s
    • dev / stag / prod
  • hybrid k8s
    • bare metal / public cloud

More Issues: multi-hybrid cluster

kind: ApplicationSet
metadata:
  name: aws-cni
  namespace: kube-system
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          eks: true
          #bare-metal: true
          staging: true

More Issues: multi-hybrid cluster

aws-eks-1
├── aws cni
├── aws ingress controller
└── nginx-ingress controller
bare-metal-1
├── cilium
├── bare-metal ingress
└── nginx-ingress controller

More Issues: multi-hybrid cluster

  • applicationset

More Issues: Test

k8s object 需不需要測試

  • infra-test (bare-metal / self-hosted)
  • release (helm chart)
  • live status
  • stress / load test / chaos engineering

Test: ansible playbook

  • 測試 apply 後 k8s object 的 status
    • ingress 是否 ready
    • endpoint 是否有產生
  • 可以先測
    • networking (ingress / svc)
    • storage (csi / pvc)
- name: Test nginx deployment
  hosts: {{ host }}
  gather_facts: no
  vars:
    deployment_name: "nginx"
  tasks:
  - name: Get deployment status
    shell: kubectl get deployment {{ deployment_name }} -o=jsonpath='{.status.readyReplicas}'
    register: deployment_status
    failed_when: deployment_status.rc != 0
  - name: Verify deployment is running
    assert:
      that:
        - deployment_status.stdout != 'null'
        - deployment_status.stdout != '0'
      fail_msg: 'Deployment {{ deployment_name }} is not running.'

More Issues: Test

Test Monitoring

  • prometheus rule with helm chart
  • ServiceMonitor

More Issues: Test

Summary

  • kubectl + gitflow
  • helm chart
  • argocd: applicationset / generator
  • test

Q & A