Kubernetes 集群備份工具 velero 使用指南
作者：whale_life
Velero 對存儲的支持較好，可以支持很多種存儲資源，比如 AWS S3、Azure Blob、Google Cloud Storage、Alibaba Cloud OSS、Swift、MinIO 等等。
簡介
Velero 是一款可以安全的備份、恢復和遷移 Kubernetes 集群資源和持久卷等資源的備份恢復軟件。
Velero 實現的 kubernetes 資源備份能力，可以輕松實現 Kubernetes 集群的數據備份和恢復、復制 kubernetes 集群資源到其他 kubernetes 集群或者快速復制生產環境到測試環境等功能，這種備份就類似于把資源的 yaml 文件進行整體備份，從而保障資源的完整性。
Velero 對存儲的支持較好，可以支持很多種存儲資源，比如 AWS S3、Azure Blob、Google Cloud Storage、Alibaba Cloud OSS、Swift、MinIO 等等.
流程圖
注意：本文檔采用 MinIO 作為后端存儲。
Docker 部署 MinIO
因為我們要保證存儲和集群隔離，這樣能保證在集群出問題的時候，不影響備份，所以就將 MinIO 部署在其他服務器上，與 Kubernetes 環境隔離。避免因意外情況宕機的時候影響更多。
mkdir -pv /data/{minio,config}
docker pull minio/minio
docker pull minio/mc:latest
docker run -d -p 9000:9000 -p 9090:9090 --name=minio --restart=always -e "MINIO_ROOT_USER=admin" -e "MINIO_ROOT_PASSWORD=admin123456" -v /data/minio:/data -v /data/config:/root/.minio minio/minio server /data --console-address ":9090" --address ":9000"
# 對minio 授權
docker run -it --entrypoint=/bin/sh -v /data/config:/root/.minio minio/mc
mc config host add velero http://127.0.0.1:9000 admin admin123456 --api S3v4
mc mb -p velero/velero
注意，mc 使用的是 api 端口，并不是 console 界面端口，所以 mc 在授權的時候需要使用 API 端口。
然后瀏覽器通過配置的賬號和 password 即可登錄：
安裝 velero 在 k8s master 上
下載 velero：
https://github.com/vmware-tanzu/velero/releases/download/v1.9.0/velero-v1.9.0-linux-amd64.tar.gz。
1、創建密鑰并安裝velero
cat > credentials-velero <<EOF
[default]
aws_access_key_id = admin
aws_secret_access_key = admin123456
EOF
# 安裝velero，配置自動補全
cp velero /usr/local/bin/
echo 'source <(velero completion bash)' >> /etc/profile && source /etc/profile
2、K8s集群安裝velero
# 需要提前下載鏡像 velero/velero-plugin-for-aws:v1.2.1 和velero/velero:v1.9.0
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.2.1 \
--bucket velero \
--secret-file ./credentials-velero \
--use-volume-snapshots=false \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.0.2.196:9000
安裝時產生的日志。
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource client
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
Velero is installed! ? Use 'kubectl logs deployment/velero -n velero' to view the status.
驗證是否安裝完畢。
[root@master ~]# kubectl get pod -n velero
NAME READY STATUS RESTARTS AGE
velero-cf79784ff-zdfp9 1/1 Running 0 47s
[root@master ~]# kubectl api-versions | grep velero
velero.io/v1
3、velero 命令
velero create backup NAME [flags]
# 剔除 namespace
--exclude-namespaces stringArray namespaces to exclude from the backup
# 剔除資源類型
--exclude-resources stringArray resources to exclude from the backup, formatted as resource.group, such as storageclasses.storage.k8s.io
# 包含集群資源類型
--include-cluster-resources optionalBool[=true] include cluster-scoped resources in the backup
# 包含 namespace
--include-namespaces stringArray namespaces to include in the backup (use '*' for all namespaces) (default *)
# 包含 namespace 資源類型
--include-resources stringArray resources to include in the backup, formatted as resource.group, such as storageclasses.storage.k8s.io (use '*' for all resources)
# 給這個備份加上標簽
--labels mapStringString labels to apply to the backup
-o, --output string Output display format. For create commands, display the object but do not send it to the server. Valid formats are 'table', 'json', and 'yaml'. 'table' is not valid for the install command.
# 對指定標簽的資源進行備份
-l, --selector labelSelector only back up resources matching this label selector (default <none>)
# 對 PV 創建快照
--snapshot-volumes optionalBool[=true] take snapshots of PersistentVolumes as part of the backup
# 指定備份的位置
--storage-location string location in which to store the backup
# 備份數據多久刪掉
--ttl duration how long before the backup can be garbage collected (default 720h0m0s)
# 指定快照的位置，也就是哪一個公有云驅動
--volume-snapshot-locations strings list of locations (at most one per provider) where volume snapshots should be stored
4、測試
velero 在安裝包中有一個測試demo，我們可以利用測試demo來進行測試驗證。
[root@master ~]# cd velero-v1.9.0-linux-amd64/examples/nginx-app/
[root@master nginx-app]# ls
base.yaml README.md with-pv.yaml
# 注意修改 with-pv 中的 storageclass
storageClassName: longhorn
# 修改 svc 類型為 ClusterIP
type: ClusterIP
[root@master nginx-app]# kubectl apply -f with-pv.yaml
namespace/nginx-example created
persistentvolumeclaim/nginx-logs created
deployment.apps/nginx-deployment created
service/my-nginx created
[root@master nginx-app]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-nginx ClusterIP 10.108.189.178 <none> 80/TCP 79s
[root@master nginx-app]# kubectl get all -n nginx-example
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-6779884c68-4nv2s 2/2 Running 0 2m16s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-nginx ClusterIP 10.108.189.178 <none> 80/TCP 2m16s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 1/1 1 1 2m16s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-6779884c68 1 1 1 2m16s
5、備份
[root@master nginx-app]# velero backup create nginx-backup --include-namespaces nginx-example
Backup request "nginx-backup" submitted successfully.
Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details.
[root@master nginx-app]# velero backup describe nginx-backup
Name: nginx-backup
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.23.5
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=23
Phase: Failed (run `velero backup logs nginx-backup` for more information)
Errors: 0
Warnings: 0
Namespaces:
Included: nginx-example
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: auto
TTL: 720h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2022-06-30 17:44:49 +0800 CST
Completed: 2022-06-30 17:44:49 +0800 CST
Expiration: 2022-07-30 17:44:49 +0800 CST
Velero-Native Snapshots: <none included>
[root@master nginx-app]# velero backup logs nginx-backup
An error occurred: gzip: invalid header
登錄 minio 查看，備份記錄。
6、恢復測試
刪除 nginx-app：
[root@master velero-v1.9.0-linux-amd64]# kubectl delete -f examples/nginx-app/base.yaml恢復 nginx：
[root@master velero-v1.9.0-linux-amd64]# velero restore create --from-backup nginx-backup --wait
Restore request "nginx-backup-20220630192521" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe nginx-backup-20220630192521` and `velero restore logs nginx-backup-20220630192521`.
[root@master velero-v1.9.0-linux-amd64]# kubectl get pods -n nginx-example
NAME READY STATUS RESTARTS AGE
nginx-deployment-6779884c68-4nv2s 2/2 Running 0 27s
恢復會新建一個 restores。
目錄注意：??velero restore??? 恢復不會覆蓋??已有的資源???，只恢復當前集群中??不存在的資源??。已有的資源不會回滾到之前的版本，如需要回滾，需在restore之前提前刪除現有的資源。
備份所有命名空間：
velero backup logs k8s-all
7、定時備份
比較推薦做定時全量備份，這個能保證我們是一直全量最新備份的狀態。
# 每天12點備份一次
velero create schedule k8s-all --schedule="0 0 * * *"
查看備份：
velero get backup #備份查看
velero get schedule #查看定時備份
velero get restore #查看已有的恢復
velero get plugins #查看插件
責任編輯：jianghua
來源：
51CTO開源基礎軟件社區
