一篇帶你kubebuilder 進階: webhook
作者：lailinxyz
準入控制存在兩種 WebHook，變更準入控制 MutatingAdmissionWebhook，和驗證準入控制 ValidatingAdmissionWebhook，執行的順序是先執行 MutatingAdmissionWebhook 再執行 ValidatingAdmissionWebhook。
在前面的文章當中我們已經完成了 NodePool Operator 的基本功能開發與測試，但是有時候我們會有這種需求，例如創建或者刪除資源的時候需要對資源進行一些檢查的操作，如果校驗不成功就不通過。或者是需要在完成實際的創建之前做一些其他操作，例如我創建一個 pod 之前對 pod 的資源做一些調整等。這些都可以通過準入控制的WebHook來實現。
準入控制存在兩種 WebHook，變更準入控制 MutatingAdmissionWebhook，和驗證準入控制 ValidatingAdmissionWebhook，執行的順序是先執行 MutatingAdmissionWebhook 再執行 ValidatingAdmissionWebhook。
創建 webhook
我們通過命令創建相關的腳手架代碼和 api
- kubebuilder create webhook --group nodes --version v1 --kind NodePool --defaulting --programmatic-validation
執行之后可以看到多了一些 webhook 相關的文件和配置
- ├── api
- │ └── v1
- │ ├── groupversion_info.go
- │ ├── nodepool_types.go
- + │ ├── nodepool_webhook.go # 在這里實現 webhook 的相關接口
- + │ ├── webhook_suite_test.go # webhook 測試
- │ └── zz_generated.deepcopy.go
- ├── bin
- ├── config
- + │ ├── certmanager # 用于部署
- │ ├── crd
- │ │ ├── bases
- │ │ │ └── nodes.lailin.xyz_nodepools.yaml
- │ │ ├── kustomization.yaml
- │ │ ├── kustomizeconfig.yaml
- │ │ └── patches
- │ │ ├── cainjection_in_nodepools.yaml
- + │ │ └── webhook_in_nodepools.yaml
- │ ├── default
- │ │ ├── kustomization.yaml
- │ │ ├── manager_auth_proxy_patch.yaml
- │ │ ├── manager_config_patch.yaml
- + │ │ ├── manager_webhook_patch.yaml
- + │ │ └── webhookcainjection_patch.yaml
- │ ├── manager
- │ ├── prometheus
- │ ├── rbac
- │ ├── samples
- │ │ └── nodes_v1_nodepool.yaml
- + │ └── webhook # webhook 部署配置
- ├── controllers
- ├── main.go
實現邏輯
實現 MutatingAdmissionWebhook 接口
這個只需要實現 Default 方法就行
- // Default implements webhook.Defaulter so a webhook will be registered for the type
- func (r *NodePool) Default() {
- nodepoollog.Info("default", "name", r.Name)
- // 如果 labels 為空，我們就給 labels 加一個默認值
- if len(r.Labels) == 0 {
- r.Labels["node-pool.lailin.xyz"] = r.Name
- }
- }
實現 ValidatingAdmissionWebhook 接口
實現 ValidatingAdmissionWebhook也是一樣只需要實現對應的方法就行了，默認是注冊了 Create 和 Update 事件的校驗，我們這里主要是限制 Labels 和 Taints 的 key 只能是滿足正則 ^node-pool.lailin.xyz/*[a-zA-z0-9]*$ 的固定格式
- // TODO(user): change verbs to "verbs=create;update;delete" if you want to enable deletion validation.
- //+kubebuilder:webhook:path=/validate-nodes-lailin-xyz-v1-nodepool,mutating=false,failurePolicy=fail,sideEffects=None,groups=nodes.lailin.xyz,resources=nodepools,verbs=create;update,versions=v1,name=vnodepool.kb.io,admissionReviewVersions={v1,v1beta1}
- var _ webhook.Validator = &NodePool{}
- // ValidateCreate implements webhook.Validator so a webhook will be registered for the type
- func (r *NodePool) ValidateCreate() error {
- nodePoolLog.Info("validate create", "name", r.Name)
- return r.validate()
- }
- // ValidateUpdate implements webhook.Validator so a webhook will be registered for the type
- func (r *NodePool) ValidateUpdate(old runtime.Object) error {
- nodePoolLog.Info("validate update", "name", r.Name)
- return r.validate()
- }
- // ValidateDelete implements webhook.Validator so a webhook will be registered for the type
- func (r *NodePool) ValidateDelete() error {
- nodePoolLog.Info("validate delete", "name", r.Name)
- // TODO(user): fill in your validation logic upon object deletion.
- return nil
- }
- // validate 驗證
- func (r *NodePool) validate() error {
- err := errors.Errorf("taint or label key must validatedy by %s", keyReg.String())
- for k := range r.Spec.Labels {
- if !keyReg.MatchString(k) {
- return errors.WithMessagef(err, "label key: %s", k)
- }
- }
- for _, taint := range r.Spec.Taints {
- if !keyReg.MatchString(taint.Key) {
- return errors.WithMessagef(err, "taint key: %s", taint.Key)
- }
- }
- return nil
- }
部署
實現了之后直接在 make run 是跑不起來的，因為 webhook 注冊的地址不對，我們這里先看一下如何進行部署運行，然后再來看如何對 WebHook 進行本地調試。
WebHook 的運行需要校驗證書，kubebuilder 官方建議我們使用 cert-manager 簡化對證書的管理，所以我們先部署一下 cert-manager 的服務
- kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml
然后我們 build 鏡像并且將鏡像 load 到集群中
- make docker-build
- kind load docker-image --name kind --nodes kind-worker controller:latest
然后查看一下 config/default/kustomization.yaml文件，確認 webhook 相關的配置沒有被注釋掉
- # Adds namespace to all resources.
- namespace: node-pool-operator-system
- # Value of this field is prepended to the
- # names of all resources, e.g. a deployment named
- # "wordpress" becomes "alices-wordpress".
- # Note that it should also match with the prefix (text before '-') of the namespace
- # field above.
- namePrefix: node-pool-operator-
- # Labels to add to all resources and selectors.
- #commonLabels:
- # someName: someValue
- bases:
- - ../crd
- - ../rbac
- - ../manager
- # [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
- # crd/kustomization.yaml
- - ../webhook
- # [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
- - ../certmanager
- # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
- #- ../prometheus
- patchesStrategicMerge:
- # Protect the /metrics endpoint by putting it behind auth.
- # If you want your controller-manager to expose the /metrics
- # endpoint w/o any authn/z, please comment the following line.
- - manager_auth_proxy_patch.yaml
- # Mount the controller config file for loading manager configurations
- # through a ComponentConfig type
- #- manager_config_patch.yaml
- # [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
- # crd/kustomization.yaml
- - manager_webhook_patch.yaml
- # [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
- # Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
- # 'CERTMANAGER' needs to be enabled to use ca injection
- - webhookcainjection_patch.yaml
- # the following config is for teaching kustomize how to do var substitution
- vars:
- # [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
- - name: CERTIFICATE_NAMESPACE # namespace of the certificate CR
- objref:
- kind: Certificate
- group: cert-manager.io
- version: v1
- name: serving-cert # this name should match the one in certificate.yaml
- fieldref:
- fieldpath: metadata.namespace
- - name: CERTIFICATE_NAME
- objref:
- kind: Certificate
- group: cert-manager.io
- version: v1
- name: serving-cert # this name should match the one in certificate.yaml
- - name: SERVICE_NAMESPACE # namespace of the service
- objref:
- kind: Service
- version: v1
- name: webhook-service
- fieldref:
- fieldpath: metadata.namespace
- - name: SERVICE_NAME
- objref:
- kind: Service
- version: v1
- name: webhook-service
檢查一下 manager/manager.yaml 是否存在 imagePullPolicy: IfNotPresent不存在要加上
然后執行部署命令即可
- make deploy
- # 檢查 pod 是否正常啟動
- ▶ kubectl -n node-pool-operator-system get pods
- NAME READY STATUS RESTARTS AGE
- node-pool-operator-controller-manager-66bd747899-lf7xb 0/2 ContainerCreating 0 7s
使用 yaml 文件測試一下
- apiVersion: nodes.lailin.xyz/v1
- kind: NodePool
- metadata:
- name: worker
- spec:
- labels:
- "xxx": "10"
- handler: runc
提交之后可以發現報錯，因為 label key 不滿足我們的要求
- ▶ kubectl apply -f config/samples/
- Error from server (label key: xxx: taint or label key must validatedy by ^node-pool.lailin.xyz/*[a-zA-z0-9]*$): error when applying patch:
- {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"nodes.lailin.xyz/v1\",\"kind\":\"NodePool\",\"metadata\":{\"annotations\":{},\"name\":\"worker\"},\"spec\":{\"handler\":\"runc\",\"labels\":{\"xxx\":\"10\"}}}\n"}},"spec":{"labels":{"node-pool.lailin.xyz/worker":null,"xxx":"10"},"taints":null}}
- to:
- Resource: "nodes.lailin.xyz/v1, Resource=nodepools", GroupVersionKind: "nodes.lailin.xyz/v1, Kind=NodePool"
- Name: "worker", Namespace: ""
- for: "config/samples/nodes_v1_nodepool.yaml": admission webhook "vnodepool.kb.io" denied the request: label key: xxx: taint or label key must validatedy by ^node-pool.lailin.xyz/*[a-zA-z0-9]*$
再用一個正常的 yaml 測試
- apiVersion: nodes.lailin.xyz/v1
- kind: NodePool
- metadata:
- name: worker
- spec:
- labels:
- "node-pool.lailin.xyz/xxx": "10"
- handler: runc
可以正常提交
- ▶ kubectl apply -f config/samples/
- nodepool.nodes.lailin.xyz/worker configured
本地調試
雖然 kubebuilder 已經為我們做了很多事情將服務部署運行基本傻瓜化了，但是每次做一點點修改就需要重新編譯部署還是非常的麻煩，所以我們來看看如何在本地進行聯調。
- PS: 這里會用到之前 4. kustomize 簡明教程 講到的 kustomize 的特性構建開發環境，如果忘記了可以先看看之前的文章哦
我們先看看 config/webhook/manifests.yaml這里面包含了兩個準入控制的信息，不過他們的配置類似，我們看一個就行了，這里以 MutatingWebhookConfiguration 為例
- apiVersion: admissionregistration.k8s.io/v1
- kind: MutatingWebhookConfiguration
- metadata:
- creationTimestamp: null
- name: mutating-webhook-configuration
- webhooks:
- - admissionReviewVersions:
- - v1
- - v1beta1
- clientConfig:
- service:
- name: webhook-service
- namespace: system
- path: /mutate-nodes-lailin-xyz-v1-nodepool
- failurePolicy: Fail
- name: mnodepool.kb.io
- rules:
- - apiGroups:
- - nodes.lailin.xyz
- apiVersions:
- - v1
- operations:
- - CREATE
- - UPDATE
- resources:
- - nodepools
- sideEffects: None
主要是 clientConfig 的配置，如果想要本地聯調，我們需要將 clientConfig.service 刪掉，替換成
- clientConfig:
- url: https://host.docker.internal:9443/mutate-nodes-lailin-xyz-v1-nodepool
注意: host.docker.internal是 docker desktop 的默認域名，通過這個可以調用到宿主機上的服務，url path mutate-nodes-lailin-xyz-v1-nodepool需要和 service 中的 path 保持一致
然后再加上 caBundle
- clientConfig:
- caBundle: CA證書 base64 后的字符串
證書
想要本地聯調需要先生成證書，我們使用 openssl 來生成，先創建一個 config/cert 文件夾，我們把證書都放到這里
首先創建一個 csr.conf文件
- [ req ]
- default_bits = 2048
- prompt = no
- default_md = sha256
- req_extensions = req_ext
- distinguished_name = dn
- [ dn ]
- C = CN
- ST = Guangzhou
- L = Shenzhen
- CN = host.docker.internal
- [ req_ext ]
- subjectAltName = @alt_names
- [ alt_names ]
- DNS.1 = host.docker.internal # 這里由于我們直接訪問的是域名所以用 DNS
- [ v3_ext ]
- authorityKeyIdentifier=keyid,issuer:always
- basicConstraints=CA:FALSE
- keyUsage=keyEncipherment,dataEncipherment
- extendedKeyUsage=serverAuth,clientAuth
- subjectAltName=@alt_names
然后生成 CA 證書并且簽發本地證書
- # 生成 CA 證書
- openssl genrsa -out ca.key 2048
- openssl req -x509 -new -nodes -key ca.key -subj "/CN=host.docker.internal" -days 10000 -out ca.crt
- # 簽發本地證書
- openssl genrsa -out tls.key 2048
- openssl req -new -SHA256 -newkey rsa:2048 -nodes -keyout tls.key -out tls.csr -subj "/C=CN/ST=Shanghai/L=Shanghai/O=/OU=/CN=host.docker.internal"
- openssl req -new -key tls.key -out tls.csr -config csr.conf
- openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt -days 10000 -extensions v3_ext -extfile csr.conf
配置變更
我們為了和原本的開發體驗保持一致，所以利用 kustomize 的特性新建一個 config/dev 文件夾，包含兩個文件修改我們想要的配置
- ▶ tree config/dev
- config/dev
- ├── kustomization.yaml
- └── webhook_patch.yaml
先看一下 kustomization.yaml，從 default 文件夾中繼承配置，然后使用 patches 修改一些配置，主要是分別給兩種準入控制 WebHook 添加 url 字段，然后使用 webhook_patch.yaml對兩個文件做些統一的配置
- resources:
- - ../default
- patches:
- - patch: |
- - op: "add"
- path: "/webhooks/0/clientConfig/url"
- value: "https://host.docker.internal:9443/mutate-nodes-lailin-xyz-v1-nodepool"
- target:
- kind: MutatingWebhookConfiguration
- - patch: |
- - op: "add"
- path: "/webhooks/0/clientConfig/url"
- value: "https://host.docker.internal:9443/validate-nodes-lailin-xyz-v1-nodepool"
- target:
- kind: ValidatingWebhookConfiguration
- - path: webhook_patch.yaml
- target:
- group: admissionregistration.k8s.io
webhook_patch.yaml 這個主要是移除 cert-manager.io 的 annotation，本地調試不需要使用它進行證書注入，然后移除掉 service 并且添加 CA 證書
- - op: "remove"
- path: "/metadata/annotations/cert-manager.io~1inject-ca-from"
- - op: "remove"
- path: "/webhooks/0/clientConfig/service"
- - op: "add"
- path: "/webhooks/0/clientConfig/caBundle"
- value: CA 證書 base64 后的值
CA 證書的值可以通過以下命令獲取
- cat config/cert/ca.crt | base64 | tr -d '\n'
然后修改一下 main.go將證書文件夾指定到我們剛剛生成好的文件目錄
- mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
- Scheme: scheme,
- MetricsBindAddress: metricsAddr,
- Port: 9443,
- HealthProbeBindAddress: probeAddr,
- LeaderElection: enableLeaderElection,
- LeaderElectionID: "97acaccf.lailin.xyz",
- + CertDir: "config/cert/", // 手動指定證書位置用于測試
- })
為了方便調試，在 makefile 中添加
- dev: manifests kustomize
- cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
- $(KUSTOMIZE) build config/dev | kubectl apply -f -
最后執行一下 make dev 然后再執行 make run 就行了
總結
今天完成了準入控制 WebHook 的實現，雖然這個例子可能不太好，如果只需要校驗正則，直接配置一下//+kubebuilder:validation:Pattern=string就行了，但是學習了這個之后其實可以做很多事情，例如給 pod 增加 sidecar 根據應用類型的不同注入不同的一些 agent 等等.
本文轉載自微信公眾號「mohuishou」，可以通過以下二維碼關注。轉載本文請聯系mohuishou公眾號。
責任編輯：姜華
來源：
mohuishou
