[cka] kodekloud-Backup and Restore

1.We have a working kubernetes cluster with a set of applications running. Let us first explore the setup.

How many deployments exist in the cluster?

k get deployments

A) 2 

2.What is the version of ETCD running on the cluster? Check the ETCD Pod or Process

k get pods etcd-controlplane -n kube-system
k get pods -n kube-system
k get pods etcd-controlplane -n kube-system
k describe pods etcd-controlplane -n kube-system

 : describe 명령어의 image 버전 확인 가능 

A) 3.5.1

 

3.At what address can you reach the ETCD cluster from the controlplane node?

Check the ETCD Service configuration in the ETCD POD

k describe pods etcd-controlplane -n kube-system

A) https://127.0.0.1:2379

 

4.Where is the ETCD server certificate file located? Note this path down as you will need to use it later

k describe pods etcd-controlplane -n kube-system

A) /etc/kubernetes/pki/etcd/server.crt

 

5.Where is the ETCD CA Certificate file located? Note this path down as you will need to use it later.

A)/etc/kubernetes/pki/etcd/ca.crt

 

6.The master node in our cluster is planned for a regular maintenance reboot tonight. While we do not anticipate anything to go wrong, we are required to take the necessary backups. Take a snapshot of the ETCD database using the built-in snapshot functionality.

Store the backup file at location /opt/snapshot-pre-boot.db

• Backup ETCD to /opt/snapshot-pre-boot.db

 : 백업하는 방법 

# export ETCDCTL_API=3


#ls /opt/snapshot-pre-boot.db

1) ETCD환경변수 등록하기

 export ETCDCTL_API=3

2) ETCD 백업 실행 

etcdctl  snapshot save 
--endpoints=[listen-client-urls]
--cacert=[trusted-ca-file]
--cert=[cert-file]
--key=[key-file]
[ETCD 백업경로]
# k describe pods etcd-controlplane -n kube-system

# etcdctl snapshot save --endpoints=127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
/opt/snapshot-pre-boot.db

이렇게 작성하면 된다. 

 

3) 백업 경로에 해당 파일이 백업되었는지 확인

#ls /opt/snapshot-pre-boot.db [ 백업경로 ] 

 

7. 문제 자체가 그냥 기다리라고 함. 

A) ok

 

8.Wake up! We have a conference call! After the reboot the master nodes came back online, but none of our applications are accessible. Check the status of the applications on the cluster. What's wrong? 

# k get deploy
# k get svc
# k get pods 

A) All of the above 

 

9. Luckily we took a backup. Restore the original state of the cluster using the backup file.

 : 복구

 

1) ETCD 복구방법 

ETCDCTL_API=3 etcdctl --data-dir /var/lib/etcd-from-backup \ snapshot restore /opt/snapshot-pre-boot.db

2) 

vi /etc/kubernetes/manifests/etcd.yaml

 : volumes 의 hostPath 를 /opt/snapshot-pre-boot.db 백업 했을때 경로로 변경

 : 상위의 VolumeMounts 의 경로와 동일하지 않아도 된다.

 : 백업 받았던 파일의 경로로 읽을 수 있도록 위치를 변경한다. 

 

3) 급 에러 발생함. 

k get -n kube-system 

k delete pod etcd-controlplane -n kube-system 

 : 확인 시 pending 되어 있는 모습을 확인 함.  

 : 해당 파드를 지우고 다시 확인

4) 다시 확인 

k get pods -n kube-system