This article was last updated on: February 7, 2024 pm
overview
Sometimes we operate incorrectly, or there is a problem with the order of deletion, or a critical service is not started, resulting in Kubernetes often being unable to delete NameSpace. What should we do in this case?
Standardize the deletion process
In fact, many times this situation occurs, mainly because our deletion operation is not standardized, typical of the following situations:
There are problems with the order of deletion, such as:
Remove the key components of Traefik before attempting to delete CRDs that contain Traefik Ingress or EdgeIngress
A critical service is not started, such as:
For Kubernetes clusters with Prometheus Operator + custom adapter installed, remove NameSpaces that contain these monitoring CRDs or HPA custom metrics in the case of scaling down some of Prometheus’ key components
…
To sum up, in most cases, NameSpace cannot be deleted, because we are wrong in the first place.
In order to avoid the recurrence of such mistakes, it is recommended to set up the deletion according to the following process:
Ensure that all basic service components are in a healthy state (e.g., ingress component, monitoring component, servicemesh component…)
Check all resources under NameSpace to delete,CRD in particular , recommended here Krew - CLI plugin manager for Kubernetes Installation get-allgenuine to get all the resources under the NameSpace, as shown in the code block that follows:
For some of these CRDs or special resources, it’s a good idea to explicitly specify the deletion first and ensure that it can be deleted successfully
Finally, delete the NameSpace
Code block for step 2: (there are so many CRDs)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 ❯ kubectl get-all -n cert-manager
Try force deletion
If NameSpace is already in terminating status, and can not be deleted for a long time, you can try to add these 2 parameters to force the deletion:
1 kubectl delete ns ${NAMESPACE} --force --grace-period=0
Force delete failed? Let’s try this again
Force delete failed? Try this again:Call the Kubernetes API to delete
Hard Way steps
First, get the JSON file where you want to delete NameSpace:
1 2 NAMESPACE=cert-manager${NAMESPACE} -o json > namespace.json
Then, edit namespace.jsonfrom finalizers field delete kubernetes and save, an example is as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 { "apiVersion" : "v1" , "kind" : "Namespace" , "metadata" : { : ...} , "spec" : { "finalizers" : [ ] } , "status" : { "phase" : "Terminating" } }
After that, it can be passed kubectl proxy Set the ephemeral IP and port for APIServer
1 kubectl proxy --port=6880 &
Finally, make an API call to force the deletion:
1 curl -k -H "Content-Type: application/json" -X PUT --data-binary @namespace.json http://127.0.0.1:6880/api/v1/namespaces/${NAMESPACE} /finalize
Verify that the deletion was successful:
1 kubectl get ns ${NAMESPACE}
Scripted
📝Notes:
Dependent components:
force-delete-ns.sh
1 2 3 4 5 6 7 8 #!/bin/bash set -ex$PATH :.$1 kill -9 $(ps -ef|grep proxy|grep -v grep |awk '{print $2}' )${NAMESPACE} -o json |jq '.spec = {"finalizers":[]}' > namespace.json"Content-Type: application/json" -X PUT --data-binary @namespace.json 127.0.0.1:6880/api/v1/namespaces/${NAMESPACE} /finalize
Examples of how to use it:
1 bash force-delete-ns.sh cert-manager
🎉🎉🎉
summary
It is often encountered that the NameSpace of Kubernetes cannot be deleted, how should I solve it? Here are 3 scenarios:
Try not to have the above situation (😑 eh… nonsense)
plus --force flag Force deletion
Call the namespace’s finalize API to force the deletion
However, when it comes to the stage where it needs to be forcefully deleted, 2/3 of the parts cannot be guaranteed to be 100% successful.
EOF