Why database on Kubernetes ?
TLDR
Want a quick introduction check this excellent video from my colleague Michael Cade.
Having your database in Kubernetes has several advantages :
- Performance : you colocalize the data and the application
- Security : All stay within the kubernetes network
- Ease of deployment :
kubectl apply -f ...and you’re good to go. - Automation : You define your desired state, the operator make it happen.
- Self Healing : Benefits for all workloads in Kubernetes.
However there are challenges :
- Enterprise scale : HA and big volumes are required.
- Efficient Protection : Incremental backup, transaction consistency.
- Ease of use : GUI and easy to use API.
- Skill shortage : Database and kubernetes double skills are hard to find.
- Migration and replication : we are more than ever in a hybrid momentum.
Especially the last one Migration and replication : no serious player would want to get trapped with a single cloud provider. How can you negotiate your bill without the flexibility to move? How can you adapt to legal compliance changes? How do you move and replicate accross the cloud and the datacenters ?
EDB and Kasten together can overcome all these challenges let’s see how.
How EDB and Kasten work together ?
EDB on Kubernetes
EDB offer a full management of postgres clusters on Kubernetes. The only thing you have to do is create a custom resource like this one :
apiVersion: postgresql.k8s.enterprisedb.io/v1
kind: Cluster
metadata:
name: cluster-example
annotations:
"k8s.enterprisedb.io/addons": '["kasten"]'
spec:
instances: 3
storage:
size: 1Gi
“Et Voila”, you immediately get a postgres cluster with one primary and 2 replicas. If the primary becomes unavailable, the operator immediately handles the failover to one of the replicas and promotes it to primary. In addition to the failover, the operator automatically adds a new replica to bring the database to a desired state. All that automatically.
EDB is the company behind the very well known Open Source Kubernetes project CloudNativePG, and you can get support and professional services from EDB through their commercial offering. You can see an EDB database as a managed database but without being tied in a cloud provider or a data center.
Solution
In it’s regular run EDB manages the postgres cluster ensuring there is always one primary (read-write) and 2 replicas (read-only) to handle failover scenarios.
When Kasten launch the backup it calls the command to “Fence” one of the replica. It means that the primary instance is not affected and all the transactions can continue on the primary. Customer won’t notice any downtime or performance drop because of the backup !
When an instance is “fenced” , EDB guarantees consistency by committing all transactions to the database. This ensures that the backups taken by Kasten is a full consistent snapshot of the database.
When the snapshot of the “fenced” instance is complete, Kasten will “unfence” and the replica catches up with the last transaction that was missed during the fencing.
During restore, Kasten restores the PVC that was fenced, and when complete the operator promotes it as a primary instance. Two read replicas are then created from the primary.
Let’s try it now
Install the operator
If you already have EDB operator installed on kubernetes you can skip this part
kubectl apply -f https://get.enterprisedb.io/cnp/postgresql-operator-1.20.2.yaml
This will create the operator namespace where the controller will be running.
Create an EDB cluster, a client and some data
kubectl create ns edb
kubectl apply -f https://github.com/michaelcourcy/edb-kasten/raw/main/cluster-example-2.yaml -n edb
Wait for the cluster to be fully ready.
kubectl get clusters.postgresql.k8s.enterprisedb.io -n edb
NAME AGE INSTANCES READY STATUS PRIMARY
cluster-example 19m 3 3 Cluster in healthy state cluster-example-1
Install the cnp plugin if you haven’t it yet
curl -sSfL \
https://github.com/EnterpriseDB/kubectl-cnp/raw/main/install.sh | \
sudo sh -s -- -b /usr/local/bin
Create a client certificate to the database
kubectl cnp certificate cluster-app \
--cnp-cluster cluster-example \
--cnp-user app \
-n edb
Now you can create the client
kubectl create -f https://github.com/michaelcourcy/edb-kasten/raw/main/client.yaml -n edb
Create some data
kubectl exec -it deploy/cert-test -- bash
psql " $DATABASE_URL "
\c app
DROP TABLE IF EXISTS links;
CREATE TABLE links (
id SERIAL PRIMARY KEY,
url VARCHAR(255) NOT NULL,
name VARCHAR(255) NOT NULL,
description VARCHAR (255),
last_update DATE
);
INSERT INTO links (url, name, description, last_update) VALUES('https://kasten.io','Kasten','Backup on kubernetes',NOW());
select * from links;
\q
exit
Add the backup decorator annotations to the cluster
Skip this step if you have created the cluster using the instructions from the previous section. With cluster-example-2 the yaml includes the kasten addon.
If you haven’t used the instructions from the previous section, add this annotation to your cluster CR
"k8s.enterprisedb.io/addons": '["kasten"]'
If your version of EDB is old and does not support the kasten addons you can create all the annotations and labels manually using the example in cluster-example.yaml .
Install the edb blueprint
kubectl create -f https://github.com/michaelcourcy/edb-kasten/raw/main/edb-hooks.yaml
Create a backup policy with the exclude filters and the hooks
Create a Kasten policy for the edb namespace: set up a location profile for export and kanister actions.
Add the exlude filters :
kasten-enterprisedb.io/excluded:true
Add the hooks :
Launch a backup
Launch a backup, this creates 2 restorepoints a local and a remote.
When checking the kasten restore point you can notice that only one PVC has been backed up (the one that was “fenced”).
Let’s test a restore
Delete the namespace edb
kubectl delete ns edb
Restore
Because you deleted the namespace all the volumesnaphot are gone hence you need to restore from the external location profile.
Just click restore and wait for the EDB cluster to restart.
You should see pod cluster-example-2 immediatly starting (without initialization of the database) and the cluster-example-3 and cluster-example-4 joining.
kubectl get po -n edb -w
NAME READY STATUS RESTARTS AGE
cert-test-5dcf5cb6b8-fhf4m 1/1 Running 0 3s
cluster-example-2 0/1 Init:0/1 0 1s
cluster-example-2 0/1 PodInitializing 0 3s
cluster-example-2 0/1 Running 0 4s
cluster-example-2 1/1 Running 0 5s
cluster-example-2 1/1 Running 0 5s
cluster-example-3-join-vm6d9 0/1 Pending 0 0s
cluster-example-3-join-vm6d9 0/1 Pending 0 5s
cluster-example-3-join-vm6d9 0/1 Init:0/1 0 5s
cluster-example-3-join-vm6d9 0/1 PodInitializing 0 10s
cluster-example-3-join-vm6d9 1/1 Running 0 11s
cluster-example-3-join-vm6d9 0/1 Completed 0 13s
cluster-example-3-join-vm6d9 0/1 Completed 0 15s
cluster-example-3-join-vm6d9 0/1 Completed 0 16s
cluster-example-3 0/1 Pending 0 0s
cluster-example-3 0/1 Pending 0 0s
cluster-example-3 0/1 Init:0/1 0 0s
cluster-example-3 0/1 PodInitializing 0 4s
cluster-example-3 0/1 Running 0 5s
cluster-example-3 0/1 Running 0 5s
cluster-example-3 1/1 Running 0 6s
cluster-example-4-join-rtpxf 0/1 Pending 0 0s
cluster-example-4-join-rtpxf 0/1 Pending 0 5s
cluster-example-4-join-rtpxf 0/1 Init:0/1 0 5s
cluster-example-4-join-rtpxf 0/1 PodInitializing 0 9s
cluster-example-4-join-rtpxf 0/1 Completed 0 10s
cluster-example-4-join-rtpxf 0/1 Completed 0 12s
cluster-example-4-join-rtpxf 0/1 Completed 0 13s
cluster-example-4 0/1 Pending 0 0s
cluster-example-4 0/1 Pending 0 0s
cluster-example-4 0/1 Init:0/1 0 0s
cluster-example-4 0/1 PodInitializing 0 6s
cluster-example-4 0/1 Running 0 7s
cluster-example-4 0/1 Running 0 7s
cluster-example-4 1/1 Running 0 8s
cluster-example-4-join-rtpxf 0/1 Terminating 0 21s
cluster-example-3-join-vm6d9 0/1 Terminating 0 43s
cluster-example-4-join-rtpxf 0/1 Terminating 0 21s
cluster-example-3-join-vm6d9 0/1 Terminating 0 43s
cluster-example-3 1/1 Running 0 28s
cluster-example-2 1/1 Running 0 50s
cluster-example-4 1/1 Running 0 9s
Check your data are back.
As Kasten restores everything that was backed up by default it has also restored the client. Connect to the client and check the data.
kubectl exec -it deploy/cert-test -- bash
psql " $DATABASE_URL "
\c app
select * from links;
\q
exit
You should see your data back
app=> select * from links;
id | url | name | description | last_update
----+-------------------+--------+----------------------+-------------
1 | https://kasten.io | Kasten | Backup on kubernetes | 2024-03-25
Conclusion
This partnership between Kasten and EDB is really a great opportunity for devops that were looking for a postgres managed database without being tied to a specific cloud provider. Now your database is fully managed and you can migrate it seamlessly in your different Kubernetes clusters.
