K8s使用新版NFS Provisioner配置Subdir
作者： 小姜
NFS在k8s中作為volume存儲已經沒有什么新奇的了，這個是最簡單也是最容易上手的一種文件存儲。最近有一個需求需要在k8s中使用NFS存儲，于是記錄如下，并且還存在一些騷操作和過程中遇到的坑點，同時記錄如下。
本文轉載自微信公眾號「運維開發故事」，作者小姜。轉載本文請聯系運維開發故事公眾號。
背景
NFS在k8s中作為volume存儲已經沒有什么新奇的了，這個是最簡單也是最容易上手的一種文件存儲。最近有一個需求需要在k8s中使用NFS存儲，于是記錄如下，并且還存在一些騷操作和過程中遇到的坑點，同時記錄如下。
訪問nfs provisioner的GitHub倉庫會發現他提示你該倉庫已經被個人歸檔并且狀態已經是只讀了。
老的NFS倉庫地址：https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client
下面Deprecated表示倉庫已經移動到了另外一個GitHub地址了，也就是說他這個倉庫已經不在更新了，持續更新的倉庫是Moved to 后面指定的倉庫了。
新倉庫地址：https://github.com/lorenzofaresin/nfs-subdir-external-provisioner
發現更新的倉庫中相比老倉庫多了一個功能：添加了一個參數pathPattern,實際上也就是通過設置這個參數可以配置PV的子目錄。
nfs-client-provisioner部署
帶著好奇心我們來部署一下新的NFS，以下yaml配置文件可以在項目中的deploy目錄中找到。我這里的配置根據我的環境稍微做了更改，比如NFS的服務的IP地址。你們根據實際情況修改成自己的nfs服務器地址和path路徑。
「本次實踐在k8s 1.19.0上」
class.yaml
- $ cat class.yaml
- apiVersion: storage.k8s.io/v1
- kind: StorageClass
- metadata:
- name: managed-nfs-storage
- provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
- parameters:
- archiveOnDelete: "false"
deployment.yaml
- apiVersion: apps/v1
- kind: Deployment
- metadata:
- name: nfs-client-provisioner
- labels:
- app: nfs-client-provisioner
- # replace with namespace where provisioner is deployed
- namespace: kube-system
- spec:
- replicas: 1
- strategy:
- type: Recreate
- selector:
- matchLabels:
- app: nfs-client-provisioner
- template:
- metadata:
- labels:
- app: nfs-client-provisioner
- spec:
- serviceAccountName: nfs-client-provisioner
- containers:
- - name: nfs-client-provisioner
- image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
- volumeMounts:
- - name: nfs-client-root
- mountPath: /persistentvolumes
- env:
- - name: PROVISIONER_NAME
- value: k8s-sigs.io/nfs-subdir-external-provisioner
- - name: NFS_SERVER
- value: 172.16.33.4
- - name: NFS_PATH
- value: /
- volumes:
- - name: nfs-client-root
- nfs:
- server: 172.16.33.4
- path: /
rbac.yaml
- apiVersion: v1
- kind: ServiceAccount
- metadata:
- name: nfs-client-provisioner
- # replace with namespace where provisioner is deployed
- namespace: kube-system
- ---
- kind: ClusterRole
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: nfs-client-provisioner-runner
- rules:
- - apiGroups: [""]
- resources: ["nodes"]
- verbs: ["get", "list", "watch"]
- - apiGroups: [""]
- resources: ["persistentvolumes"]
- verbs: ["get", "list", "watch", "create", "delete"]
- - apiGroups: [""]
- resources: ["persistentvolumeclaims"]
- verbs: ["get", "list", "watch", "update"]
- - apiGroups: ["storage.k8s.io"]
- resources: ["storageclasses"]
- verbs: ["get", "list", "watch"]
- - apiGroups: [""]
- resources: ["events"]
- verbs: ["create", "update", "patch"]
- ---
- kind: ClusterRoleBinding
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: run-nfs-client-provisioner
- subjects:
- - kind: ServiceAccount
- name: nfs-client-provisioner
- # replace with namespace where provisioner is deployed
- namespace: kube-system
- roleRef:
- kind: ClusterRole
- name: nfs-client-provisioner-runner
- apiGroup: rbac.authorization.k8s.io
- ---
- kind: Role
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: leader-locking-nfs-client-provisioner
- # replace with namespace where provisioner is deployed
- namespace: kube-system
- rules:
- - apiGroups: [""]
- resources: ["endpoints"]
- verbs: ["get", "list", "watch", "create", "update", "patch"]
- ---
- kind: RoleBinding
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: leader-locking-nfs-client-provisioner
- # replace with namespace where provisioner is deployed
- namespace: kube-system
- subjects:
- - kind: ServiceAccount
- name: nfs-client-provisioner
- # replace with namespace where provisioner is deployed
- namespace: kube-system
- roleRef:
- kind: Role
- name: leader-locking-nfs-client-provisioner
- apiGroup: rbac.authorization.k8s.io
注意：
- 鏡像無法拉取的話可以從國外機器拉取鏡像然后再導入
- rbac基本無需更改，配置子目錄的時候需要更改class.yaml文件，后面會說
創建所有資源文件
- kubectl apply -f class.yaml -f deployment.yaml -f rbac.yaml
通過一個簡單的例子來創建pvc
- $ cat test-pvc-2.yaml
- kind: PersistentVolumeClaim
- apiVersion: v1
- metadata:
- name: test-pvc-2
- namespace: nacos
- spec:
- storageClassName: "managed-nfs-storage"
- accessModes:
- - ReadWriteMany
- resources:
- requests:
- storage: 10Gi
- $ cat test-nacos-pod-2.yaml
- apiVersion: apps/v1
- kind: StatefulSet
- metadata:
- name: nacos-c1-sit-tmp-1
- labels:
- appEnv: sit
- appName: nacos-c1-sit-tmp-1
- namespace: nacos
- spec:
- serviceName: nacos-c1-sit-tmp-1
- replicas: 3
- selector:
- matchLabels:
- appEnv: sit
- appName: nacos-c1-sit-tmp-1
- template:
- metadata:
- labels:
- appEnv: sit
- appName: nacos-c1-sit-tmp-1
- spec:
- dnsPolicy: ClusterFirst
- containers:
- - name: nacos
- image: www.ayunw.cn/library/nacos/nacos-server:1.4.1
- ports:
- - containerPort: 8848
- env:
- - name: NACOS_REPLICAS
- value: "1"
- - name: MYSQL_SERVICE_HOST
- value: mysql.ayunw.cn
- - name: MYSQL_SERVICE_DB_NAME
- value: nacos_c1_sit
- - name: MYSQL_SERVICE_PORT
- value: "3306"
- - name: MYSQL_SERVICE_USER
- value: nacos
- - name: MYSQL_SERVICE_PASSWORD
- value: xxxxxxxxx
- - name: MODE
- value: cluster
- - name: NACOS_SERVER_PORT
- value: "8848"
- - name: PREFER_HOST_MODE
- value: hostname
- - name: SPRING_DATASOURCE_PLATFORM
- value: mysql
- - name: TOMCAT_ACCESSLOG_ENABLED
- value: "true"
- - name: NACOS_AUTH_ENABLE
- value: "true"
- - name: NACOS_SERVERS
- value: nacos-c1-sit-0.nacos-c1-sit-tmp-1.nacos.svc.cluster.local:8848 nacos-c1-sit-1.nacos-c1-sit-tmp-1.nacos.svc.cluster.local:8848 nacos-c1-sit-2.nacos-c1-sit-tmp-1.nacos.svc.cluster.local:8848
- imagePullPolicy: IfNotPresent
- resources:
- limits:
- cpu: 500m
- memory: 5Gi
- requests:
- cpu: 100m
- memory: 512Mi
- volumeMounts:
- - name: data
- mountPath: /home/nacos/plugins/peer-finder
- subPath: peer-finder
- - name: data
- mountPath: /home/nacos/data
- subPath: data
- volumeClaimTemplates:
- - metadata:
- name: data
- spec:
- storageClassName: "managed-nfs-storage"
- accessModes:
- - "ReadWriteMany"
- resources:
- requests:
- storage: 10Gi
查看pvc以及nfs存儲中的數據
- # ll
- total 12
- drwxr-xr-x 4 root root 4096 Aug 3 13:30 nacos-data-nacos-c1-sit-tmp-1-0-pvc-90d74547-0c71-4799-9b1c-58d80da51973
- drwxr-xr-x 4 root root 4096 Aug 3 13:30 nacos-data-nacos-c1-sit-tmp-1-1-pvc-18b3e220-d7e5-4129-89c4-159d9d9f243b
- drwxr-xr-x 4 root root 4096 Aug 3 13:31 nacos-data-nacos-c1-sit-tmp-1-2-pvc-26737f88-35cd-42dc-87b6-3b3c78d823da
- # ll nacos-data-nacos-c1-sit-tmp-1-0-pvc-90d74547-0c71-4799-9b1c-58d80da51973
- total 8
- drwxr-xr-x 2 root root 4096 Aug 3 13:30 data
- drwxr-xr-x 2 root root 4096 Aug 3 13:30 peer-finder
可以發現手動創建了一個PVC，并且創建一個nacos的deployment使用這個PVC后已經自動創建出相應的PV并且與之綁定，且掛載了數據。
配置子目錄
刪除之前創建的class.yaml，添加pathPattern參數，然后重新生成sc
- $ kubectl delete -f class.yaml
- $ vi class.yaml
- apiVersion: storage.k8s.io/v1
- kind: StorageClass
- metadata:
- name: managed-nfs-storage
- provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
- parameters:
- archiveOnDelete: "false"
- # 添加以下參數
- pathPattern: "${.PVC.namespace}/${.PVC.annotations.nfs.io/storage-path}"
創建pvc來測試生成的PV目錄是否生成了子目錄
- $ cat test-pvc.yaml
- kind: PersistentVolumeClaim
- apiVersion: v1
- metadata:
- name: test-pvc-2
- namespace: nacos
- annotations:
- nfs.io/storage-path: "test-path-two" # not required, depending on whether this annotation was shown in the storage class description
- spec:
- storageClassName: "managed-nfs-storage"
- accessModes:
- - ReadWriteMany
- resources:
- requests:
- storage: 100Mi
創建資源
- kubectl apply -f class.yaml -f test-pvc.yaml
查看結果
- # pwd
- /data/nfs# ll nacos/
- total 4
- drwxr-xr-x 2 root root 4096 Aug 3 10:21 nacos-pvc-c1-pro
- # tree -L 2 .
- .
- └── nacos
- └── nacos-pvc-c1-pro
- 2 directories, 0 files
在mount了nfs的機器上查看生成的目錄，發現子目錄的確已經生成，并且子目錄的層級是以"命名空間/注解名稱"為規則的。剛好符合了上面StorageClass中定義的pathPattern規則。
provisioner高可用
生產環境中應該盡可能的避免單點故障，因此此處考慮provisioner的高可用架構 更新后的provisioner配置如下：
- $ cat nfs-deployment.yaml
- apiVersion: apps/v1
- kind: Deployment
- metadata:
- name: nfs-client-provisioner
- labels:
- app: nfs-client-provisioner
- # replace with namespace where provisioner is deployed
- namespace: kube-system
- spec:
- # 因為要實現高可用，所以配置3個pod副本
- replicas: 3
- strategy:
- type: Recreate
- selector:
- matchLabels:
- app: nfs-client-provisioner
- template:
- metadata:
- labels:
- app: nfs-client-provisioner
- spec:
- serviceAccountName: nfs-client-provisioner
- imagePullSecrets:
- - name: registry-auth-paas
- containers:
- - name: nfs-client-provisioner
- image: www.ayunw.cn/nfs-subdir-external-provisioner:v4.0.2-31-gcb203b4
- imagePullPolicy: IfNotPresent
- volumeMounts:
- - name: nfs-client-root
- mountPath: /persistentvolumes
- env:
- - name: PROVISIONER_NAME
- value: k8s-sigs.io/nfs-subdir-external-provisioner
- # 設置高可用允許選舉
- - name: ENABLE_LEADER_ELECTION
- value: "True"
- - name: NFS_SERVER
- value: 172.16.33.4
- - name: NFS_PATH
- value: /
- volumes:
- - name: nfs-client-root
- nfs:
- server: 172.16.33.4
- path: /
重建資源
- kubectl delete -f nfs-class.yaml -f nfs-deployment.yaml
- kubectl apply -f nfs-class.yaml -f nfs-deployment.yaml
查看provisioner高可用是否生效
- # kubectl get po -n kube-system | grep nfs
- nfs-client-provisioner-666df4d979-fdl8l 1/1 Running 0 20s
- nfs-client-provisioner-666df4d979-n54ps 1/1 Running 0 20s
- nfs-client-provisioner-666df4d979-s4cql 1/1 Running 0 20s
- # kubectl logs -f --tail=20 nfs-client-provisioner-666df4d979-fdl8l -n kube-system
- I0803 06:04:41.406441 1 leaderelection.go:242] attempting to acquire leader lease kube-system/nfs-provisioner-baiducfs...
- ^C
- # kubectl logs -f --tail=20 -n kube-system nfs-client-provisioner-666df4d979-n54ps
- I0803 06:04:41.961617 1 leaderelection.go:242] attempting to acquire leader lease kube-system/nfs-provisioner-baiducfs...
- ^C
- [root@qing-core-kube-master-srv1 nfs-storage]# kubectl logs -f --tail=20 -n kube-system nfs-client-provisioner-666df4d979-s4cql
- I0803 06:04:39.574258 1 leaderelection.go:242] attempting to acquire leader lease kube-system/nfs-provisioner-baiducfs...
- I0803 06:04:39.593388 1 leaderelection.go:252] successfully acquired lease kube-system/nfs-provisioner-baiducfs
- I0803 06:04:39.593519 1 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"nfs-provisioner-baiducfs", UID:"3d5cdef6-57da-445e-bcd4-b82d0181fee4", APIVersion:"v1", ResourceVersion:"1471379708", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' nfs-client-provisioner-666df4d979-s4cql_590ac6eb-ccfd-4653-9de5-57015f820b84 became leader
- I0803 06:04:39.593559 1 controller.go:820] Starting provisioner controller nfs-provisioner-baiducfs_nfs-client-provisioner-666df4d979-s4cql_590ac6eb-ccfd-4653-9de5-57015f820b84!
- I0803 06:04:39.694505 1 controller.go:869] Started provisioner controller nfs-provisioner-baiducfs_nfs-client-provisioner-666df4d979-s4cql_590ac6eb-ccfd-4653-9de5-57015f820b84!
通過successfully acquired lease kube-system/nfs-provisioner-baiducfs可以看到第三個pod成功被選舉為leader節點了，高可用生效。
報錯
在操作過程中遇到describe pod發現報錯如下：
- Mounting arguments: --description=Kubernetes transient mount for /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs 172.16.41.7:/data/nfs_storage /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root
- Output: Running scope as unit: run-rdcc7cfa6560845969628fc551606e69d.scope
- mount: /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.
- Warning FailedMount 10s kubelet, node1.ayunw.cn MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
- Mounting command: systemd-run
解決方式：經排查原因是pod被調度到的節點上沒有安裝nfs客戶端，只需要安裝一下nfs客戶端nfs-utils即可。
責任編輯：武曉燕
來源：
運維開發故事
