hero-FG-blog

Kasten K10 Blog

All Things Kubernetes and Data Management

  Latest Posts

Use Kasten K10 to Protect Cloud Native Applications and their Data on HPE Ezmeral Container Platform

Co-author: Jaiganesh Karthikeyan, Senior Software Engineer, InfraCloud Technologies. 

This article offers a step-by-step guide on how to configure and use Kasten K10 for data protection, for cloud native applications running on HPE Ezmeral Container Platform.

Companies are looking for ways to deploy and manage Kubernetes clusters in production and at scale. HPE and Kasten by Veeam are working together to make this possible. Used together, HPE Ezmeral Container Platform and Kasten K10 by Veeam simplify and streamline data protection in Kubernetes with an integrated and easy-to-use solution.

HPE Ezmeral and Kasten K10 Overview

image1

An integrated component of HPE, the Ezmeral Container platform enables IT operators to deliver and manage end-to-end, production-ready Kubernetes environments with push-button simplicity, all while preserving a native user experience. Every Ezmeral cluster is deployed with HPE Datafabric full-featured CSI driver, which natively integrates with HPE storage solutions, volumes and file, to deliver persistent storage for stateful containerized applications. S3-compatible storage is also easy to set up using HPE Datafabric.

HPE Datafabric Object is a software-defined object storage solution that non-disruptively scales out while lowering overall storage costs. It’s designed with an S3-compatible REST API interface to handle large amounts of unstructured data. 

The HPE Datafabric Objects is an ideal target for Kasten K10 backup export, as it provides long-term retention and archiving, as well as cross-region replication. As such, Kasten K10 is a perfect solution for managing the protection and the mobility of cloud-native applications on the HPE Ezmeral Container Platform.

image5

Purpose-built for Kubernetes, Kasten K10 is a data protection software platform that runs on a Kubernetes cluster in its own namespace and can protect one or multiple  clusters. Kasten K10 provides secure multi-tenancy with fine-grained, role-based access control. 

Kasten K10 offers: 

  • Pre-qualified integrations with leading data sources, including Relational and NoSQL data services.
  • Support for all major cloud-based managed Kubernetes offerings and all leading on-prem distributions.
  • Support for storage via Container-Storage Interface (CSI), as well as direct storage integrations for efficiency.

Using Kasten K10 with the HPE Ezmeral Container Platform

Here’s a summary of the steps we’ll walk you through as we demonstrate how to integrate HPE Ezmeral with Kasten K10:

  1. Add snapshot capacity to the Exmeral cluster
  2. Create a volume snapshot class
  3. Install Kasten K10
  4. Test the snapshot and recovery of an sample MySQL application
  5. Access the dashboard  
  6. Screenshot the application
  7. Back up the application
  8. Back up and restore applications with cluster-scoped resources 
  9. Restore the application

We will also demonstrate how to export the snapshot to this object store and test the following three scenarios: 

  1. Restoration of data to same namespace 
  2. Restoration of a deleted namespace 
  3. K10 Disaster Recovery

Creating a Kubernetes Cluster with the HPE Ezmeral Container Platform

Install and deploy a Kubernetes cluster using HPE Ezmeral Container Platform by following these instructions.

1. Add snapshot capacity to the Ezmeral cluster with the HPE Ezmeral Data Fabric CSI Driver


Using the ​ CSI snapshot API​ with Kasten K10 offers numerous advantages:

  • Kasten K10 relies on the storage layer that, in turn, makes snapshots in the most efficient manner.
  • A snapshot is crash-consistent (all files’ states are taken at the same time).
  • Snapshots are local, which makes the restoration of an application faster. We can also export the snapshot in a portable way to an object storage such as the HPE Datafabric Object storage, and we can manage different retention policies between local and exported snapshots (because the cost of the storage is much less on object storage).

By default, the HPE Ezmeral cluster does not come with a volumeSnapshotClass deployed in it. You can create the volumeSnapshotClass with the steps below.

(Note that you’ll need  admin access to your cluster and a working kubeconfig/kubectl environment.)

2. Create a Volume Snapshot Class


Get the CSI secret name of the storageClass to create your volumeSnapshotClass by following the steps below:

#Get the name of the secret that contains credentials for HPE Datafabric cluster
SECRETNAME=$(kubectl get sc -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io\/is-default-class=="true")].parameters.csi\.storage\.k8s\.io\/provisioner-secret-name}')

#Get the namespace in which the secret HPE Datafabric cluster is deployed
SECRETNAMESPACE=$(kubectl get sc -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io\/is-default-class=="true")].parameters.csi\.storage\.k8s\.io\/provisioner-secret-namespace}')

#Get the HPE datafabric cluster’s rest server ip addresses
RESTSERVER=$(kubectl get sc -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io\/is-default-class=="true")].parameters.restServers}')

#Get the HPE datafabric cluster’s name
CLUSTER=$(kubectl get sc -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io\/is-default-class=="true")].parameters.cluster}')

cat <<EOF | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
name: mapr-snapshotclass
namespace: $SECRETNAMESPACE
driver: com.mapr.csi-kdf
deletionPolicy: Delete
parameters:
restServers: $RESTSERVER
cluster: $CLUSTER
csi.storage.k8s.io/snapshotter-secret-name: $SECRETNAME
csi.storage.k8s.io/snapshotter-secret-namespace: $SECRETNAMESPACE
EOF

3. Install Kasten K10


Installing Kasten K10 requires several steps:

  • Run the preflight script

To check Kasten K10 is working fine we propose a preflight script. First add the helm chart repository:

helm repo add kasten ​ https://charts.kasten.io​ --force-update && helm repo update

Next, create a namespace “kasten-io” where we will be installing Kasten K10 services:

kubectl create ns kasten-io

To test the snapshot functionalities using the preflight script, annotate the VolumeSnapshotClass as mentioned below. When Kasten K10 detects volumes that were provisioned via a CSI driver, it will look for a VolumeSnapshotClass with the Kasten K10 annotation for the identified CSI driver and use it to create snapshots:

kubectl annotate volumesnapshotclass mapr-snapshotclass \
k10.kasten.io/is-snapshot-class=true

Now we can run the preflight script and validate the output:

curl -s https://docs.kasten.io/tools/k10_primer.sh  | bash

Namespace option not provided, using default namespace
Checking for tools
 --> Found kubectl
 --> Found helm
Checking if the Kasten Helm repo is present
 --> The Kasten Helm repo was found
Checking for required Helm Tiller version (>= v2.16.0)
 --> No Tiller needed with Helm v3.3.4
K10Primer image
 --> Using Image (gcr.io/kasten-images/k10tools:3.0.8) to run test
Checking access to the Kubernetes context kubernetes-admin@k8s-46
 --> Able to access the default Kubernetes namespace
   
Running K10Primer Job in cluster with command-
  ./k10tools primer
serviceaccount/k10-primer created
clusterrolebinding.rbac.authorization.k8s.io/k10-primer created
job.batch/k10primer created
Waiting for pod k10primer-r446q to be ready - ContainerCreating
Waiting for pod k10primer-r446q to be ready - ContainerCreating
Waiting for pod k10primer-r446q to be ready -
Pod Ready!
   
Kubernetes Version Check:
 Valid kubernetes version (v1.18.6)  -  OK
   
RBAC Check:
 Kubernetes RBAC is enabled  -  OK
   
Aggregated Layer Check:
 The Kubernetes Aggregated Layer is enabled  -  OK
   
CSI Capabilities Check:
 Using CSI GroupVersion snapshot.storage.k8s.io/v1beta1  -  OK
   
Validating Provisioners:
com.mapr.csi-kdf:
 Is a CSI Provisioner  -  OK
 Missing/Failed to Fetch CSIDriver Object
 Storage Classes:
hcp-mapr-cluster
  Valid Storage Class  -  OK
 Volume Snapshot Classes:
mapr-snapshotclass
  Has k10.kasten.io/is-snapshot-class annotation set to true  -  OK
  Has deletionPolicy 'Delete'  -  OK
   
Validate Generic Volume Snapshot:
 Pod Created successfully  -  OK
 GVS Backup command executed successfully  -  OK
 Pod deleted successfully  -  OK
   
serviceaccount "k10-primer" deleted
clusterrolebinding.rbac.authorization.k8s.io "k10-primer" deleted
job.batch "k10primer" deleted

(Note: The Missing/Failed to Fetch CSIDriver Object error can be ignored, as not all the CSI implementations have CSIDriver Object.)

  • Preflight check for CSI snapshot validation

We  strongly recommend that you use the Primer tool to perform a more complete CSI validation using the following command:

curl -s https://docs.kasten.io/tools/k10_primer.sh  | bash /dev/stdin -c "storage csi-checker -s hcp-mapr-cluster --runAsUser=1000"

Namespace option not provided, using default namespace
Checking for tools
--> Found kubectl
--> Found helm
Checking if the Kasten Helm repo is present
--> The Kasten Helm repo was found
Checking for required Helm Tiller version (>= v2.16.0)
--> No Tiller needed with Helm v3.3.4
K10Primer image
--> Using Image (gcr.io/kasten-images/k10tools:3.0.8) to run test
Checking access to the Kubernetes context kubernetes-admin@k8s-46
--> Able to access the default Kubernetes namespace

Running K10Primer Job in cluster with command-
    ./k10tools primer storage csi-checker -s hcp-mapr-cluster

serviceaccount/k10-primer created
clusterrolebinding.rbac.authorization.k8s.io/k10-primer created
job.batch/k10primer created

Waiting for pod k10primer-hgvgw to be ready - ContainerCreating
Waiting for pod k10primer-hgvgw to be ready - ContainerCreating
Waiting for pod k10primer-hgvgw to be ready -
Pod Ready!

Starting CSI Checker. Could take up to 5 minutes

Failed to discover pod configuration for Pod (k8master1.test1.com): (pods "k8master1.test1.com" not found)

I0304 00:29:22.508757    1874 request.go:655] Throttling request took 1.046455236s, request: GET:https://gateway1.test1.com:10001/apis/autoscaling/v1?timeout=32s

Creating application
-> Created pod (kubestr-csi-original-podj8k7s) and pvc (kubestr-csi-original-pvcq52q8)

Taking a snapshot
-> Created snapshot (kubestr-snapshot-20210304002923)

Restoring application
-> Restored pod (kubestr-csi-cloned-podhxmsj) and pvc (kubestr-csi-cloned-pvc6rhzk)

Cleaning up resources
CSI Snapshot Walkthrough:
Using annotated VolumeSnapshotClass (mapr-snapshotclass)
Successfully tested snapshot restore functionality.  -  OK
  • Use the Helm chart to install Kasten K10

For the purpose of this demo, we’ll install Kasten with nearly no options. And focus on creating policy for protecting the namespace. We will omit authentication​ and ​ authorization​ as well as how to ​expose the Kasten K10 dashboard​.

helm install k10 kasten/k10 --namespace=kasten-io
  NAME: k10
  LAST DEPLOYED: Thu Feb 18 02:06:30 2021
  NAMESPACE: kasten-io
  STATUS: deployed
  REVISION: 1
  TEST SUITE: None
  NOTES:
  Thank you for installing Kasten's K10 Data Management Platform!
  Documentation can be found at https://docs.kasten.io/.
  How to access the K10 Dashboard:
  The K10 dashboard is not exposed externally. To establish a connection to it
  use the following `kubectl` command:
  `kubectl --namespace kasten-io port-forward service/gateway 8080:8000`
  The Kasten dashboard will be available at: `http://127.0.0.1:8080/k10/#/`

Verify if everything works by checking that all the pods are up and running in the kasten-io namespace:

kubectl get pods -n kasten-io
NAME                              READY STATUS RESTARTS  AGE
aggregatedapis-svc-854548cd9b-vdz28 1/1 Running   0      3d14h
auth-svc-7bb46fdb95-p6xqw         1/1 Running   0      3d14h
catalog-svc-766898555f-r2jvs      2/2 Running   0      3d14h
config-svc-6cb68d6745-dgrg7       1/1 Running   0      3d14h
crypto-svc-7fc8cb8f57-m9289       1/1 Running   0      3d14h
dashboardbff-svc-85c47c85c9-k4zdv 1/1 Running   0      3d14h
executor-svc-78d6cfbf88-rlz4z     2/2 Running   0      3d14h
executor-svc-78d6cfbf88-sl2x6     2/2 Running   0      3d14h
executor-svc-78d6cfbf88-sr7cn     2/2 Running   0      3d14h
frontend-svc-7cbc66648f-lr7pb     1/1 Running   0      3d14h
gateway-69997d4768-xtzh9          1/1 Running   0      3d14h
jobs-svc-cf48db89-plbrz           1/1 Running   0      3d14h
kanister-svc-66f7d457c6-jf4dt     1/1 Running   0      3d14h
logging-svc-b5dc947cd-q7595       1/1 Running   0      3d14h
metering-svc-68797f4978-mlnqb     1/1 Running   0      3d14h
prometheus-server-78b94b85fb-w745z 2/2 Running   0      3d14h
state-svc-78d847595f-9wfmq        1/1 Running   0      3d14h

4. Test the snapshot by backing up and restoring an application

  • Install the test application

For this demo, we’ll install a postgresql application for testing out snapshot backups and restore on the cluster:

helm repo add bitnami https://charts.bitnami.com/bitnami --force-update && helm repo update

kubectl create ns postgresql

helm install postgresql bitnami/postgresql --namespace=postgresql --set volumePermissions.enabled=true

Next, we’ll insert some data into the postgresql pod once it is ready:

#To get the password for "postgres" run:

export POSTGRES_PASSWORD=$(kubectl get secret --namespace postgresql
postgresql -o jsonpath=
"{.data.postgresql-password}" | base64 --decode)

#To connect to your database run the following command:

kubectl run postgresql-client --rm --tty -i --restart='Never' --namespace postgresql --image docker.io/bitnami/postgresql:11.11.0-debian-10-r50 --env="PGPASSWORD=$POSTGRES_PASSWORD" --command -- psql --host postgresql -U postgres -d postgres -p 5432

#Insert data into the test database

CREATE DATABASE test;
\c test;
CREATE TABLE pets (name VARCHAR(20), owner VARCHAR(20), species VARCHAR(20),
sex CHAR(1), birth DATE, death DATE);
INSERT INTO pets VALUES ('Puffball','Diane','hamster','f','2010-03-30',NULL);
INSERT INTO pets VALUES ('Spike','Mike','pitbull','m','2011-04-28',NULL);
INSERT INTO pets VALUES ('Ashton','Varoon','German Sheperd','m','2014-06-15',NULL);
INSERT INTO pets VALUES ('Bear','Chris','Rottweiler','m','2013-10-10',NULL);
INSERT INTO pets VALUES ('Toby','Jenny','Golden Retriever','m','2019-03-19',NULL);

#Validate the data in the table PETS
test=# select * from pets;
  name   | owner  | species  | sex |   birth | death
----------+--------+------------------+-----+------------+-------
Puffball | Diane  | hamster      | f   | 2010-03-30 |
Spike | Mike   | pitbull      | m   | 2011-04-28 |
Ashton   | Varoon | German Shepherd | m   | 2014-06-15 |
Bear | Chris  | Rottweiler    | m   | 2013-10-10 |
Toby | Jenny  | Golden Retriever| m   | 2019-03-19 |
(5 rows)

5. Access the Dashboard


All the coming operations will be done on the dashboard. Without the LoadBalancer/Ingress, we have two options to access the dashboard: 

kubectl --namespace kasten-io port-forward service/gateway 8080:8000
  • The Dashboard is made available using NodePort service (The Kasten dashboard will be available at http://<worker-node-ip>:<NodePort> where the nodePort in this example is 32161):
kubectl expose service gateway -n kasten-io --type=NodePort --name=gateway-nodeport


# Validate the service and get the nodeport details
kubectl get svc -n  kasten-io gateway-nodeport   
NAME           TYPE    CLUSTER-IP   EXTERNAL-IP   PORT(S)     
gateway-nodeport   NodePort   10.100.127.231   <none>   
8000:32161/TCP

Once you have access to the dashboard, fill out and accept the end user license. Below is the Dashboard page of K10:

image16

6. Snapshot the Application

Next, take the complete state of the application without exporting it to an external storage:

Go to Applications --> PostgreSQL --> Create a Policy:

image8

Provide the policy a name and select the action as Snapshot. You will then be able to select the backup frequency in the Advanced options:

image2

Select the retention of the snapshot, and select/filter resources by Name/Labels to snapshot based on the requirement. Then, click the Create Policy button:

image12

Click run once from the policy menu to run an ad hoc snapshot manually:

image9

Go to the main dashboard and scroll down under Actions to view progress.

image15

When the circle turns solid green, click on the job to open a details pane and view all artifacts that were captured.

image14

7. Back Up the Application

Backing up an application involves exporting the snapshot to an external backup target, which is recommended to be an S3 API-compatible object storage. For this use case, we will use HPE Data Fabric as the object storage backup target.

  • Create an external object storage with HPE Data Fabric

Follow the instructions here to configure MapR Object Store, then the instructions here to create an S3 bucket on HPE DataFabric.

  • Set up the location profile In Kasten K10 Dashboard

Go to settings -> location profile -> New Profile -> S3 compatible:

image18

Name the profile and fill in the details of the endpoint, access key, secret key and name of the bucket created using the previous steps. Then, click on Save Profile.

image19

  • Change the policy to add an export profile

You can now change the postgresql-backup policy to include an export to the object store.

Find the appropriate policy under Dashboard --> Policies, then click Edit:

image3Select Enable Backups via Snapshot Exports, and select the appropriate profile. Then,provide separate retention for exported backups, as well:

image20Click Edit Policy and click Run once again. Go to the main Dashboard and scroll down under Actions to view your progress:

image11

Export will start as soon as the snapshot action finishes. When the circle turns solid green, click on the job to open a details pane and view all artifacts that were captured, as in the previous step.

8. Back up and restore applications with cluster-scoped resources

Kasten K10 protects cluster-scoped resources in the same way that it protects applications, with snapshot policies, backups and manual snapshots.

Some applications have cluster-scoped resources such as StorageClasses,  CustomResourceDefinitions or Cluster Role, as well as namespaced components such as StatefulSets. To create a policy that protects the entire application, create a policy that protects both the application and its associated cluster-scoped resources.

Use filters to include and exclude cluster-scoped resources while creating the policy.

When this policy runs, it will create both a restore point for the application and a cluster restore point with artifacts that capture the application's cluster-scoped resources:

image20

9. Restore the Application

Select the postgresql in the applications menu and click the restore button:

image7

You now have two restore points: a local restore point and an exported restore point:

image22

Select the local snapshot to restore the application and all the objects that are backed-up as a part of the namespace. It can be restored to overwrite the objects in the same namespace or cloned to a new namespace.

image4

Restoration and Disaster Recovery

1. Restoring a Deleted namespace


The exported restore point can be used, even if the namespace is deleted or even if the cluster is deleted. It can also be used to restore a namespace to another cluster for which Kasten K10 is installed.

Delete the postgresql namespace and restore it from the exported snapshot:

kubectl delete ns postgresql

Once the namespace is deleted, you will not be able to see it in the application tab by default. You will have to select the filter to list removed applications:

image7

Select the exported restore point and restore as in the last step.

You can also apply Transforms to modify the manifest file if there are any requirements. For example, If you are going to restore the application to a different cluster with a different storageClass, you will be able to replace the storageClass spec using Transforms before the restoration.

You can also test out the Transforms applied to see the rendered manifests and validate before the restoration:

image13

There can be other use cases such as moving an application to a faster storage or adding/removing/replacing specific annotations to the specs for which you can use the Transforms feature.

Finally, verify if the restored pod has all the data:

# To get the password for "postgres" run:
export POSTGRES_PASSWORD=$(kubectl get secret --namespace postgresql postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
# To connect to your database run the following command:
kubectl run postgresql-client --rm --tty -i --restart='Never' --namespace postgresql --image docker.io/bitnami/postgresql:11.11.0-debian-10-r50 --env="PGPASSWORD=$POSTGRES_PASSWORD" --command -- psql --host postgresql -U postgres -d postgres -p 5432

postgres=# \c test
You are now connected to database "test" as user "postgres".
test=# select * from pets
test-# ;
  name   | owner  | species  | sex |   birth | death
----------+--------+------------------+-----+------------+-------
Puffball | Diane  | hamster      | f   | 2010-03-30 |
Spike | Mike   | pitbull      | m   | 2011-04-28 |
Ashton   | Varoon | German Shepherd | m   | 2014-06-15 |
Bear | Chris  | Rottweiler    | m   | 2013-10-10 |
Toby | Jenny  | Golden Retriever| m   | 2019-03-19 |
(5 rows)

2. Kasten K10 Disaster Recovery 

Kasten K10 Disaster Recovery (DR) aims to protect K10 from underlying infrastructure failures. In particular, this feature provides the ability to recover the Kasten K10 platform in case of a variety of disasters, such as the accidental deletion of Kasten K10, failure of underlying storage that Kasten K10 uses for its catalog, or even the accidental destruction of the Kubernetes cluster on which Kasten K10 is deployed.

Kasten K10 enables disaster recovery (DR) with the help of an internal policy to back up its own data stores and store them in an object storage bucket, or an NFS file storage location configured using a Location Profile.

Kasten K10 DR settings can be accessed from the Settings icon at the top-right corner of the Dashboard.

On the Settings page, select K10 Disaster Recovery, then click the Enable K10 DR button to enable DR:

image17

The Passphrase provided will be saved as a secret k10-dr-secret in the kasten-io namespace.

This creates a policy for Kasten K10 DR, and you can modify its schedule/retention per the requirements.

image10

After enabling Kasten K10 DR, it is essential that you copy and save the following to successfully recover K10 from a disaster:

  1. The cluster ID displayed on the disaster recovery page
  2. The DR passphrase entered above
  3. The credentials and object storage bucket or the NFS file storage information (used in the location profile configuration above)

Without this information, Kasten K10 DR will not be possible.

3. Restoring Kasten K10 to a New Cluster

Recovering from a Kasten K10 backup involves the following sequence of actions:

  • Create a Kubernetes Secret, k10-dr-secret, using the passphrase provided while enabling DR:
kubectl create secret generic k10-dr-secret \
  --namespace kasten-io \
  --from-literal key=<passphrase>
#Install the helm chart that creates the K10 restore job and wait for 
completion of the `k10-restore` job


#Assumes that K10 is installed in 'kasten-io' namespace.

helm install k10-restore kasten/k10restore --namespace=kasten-io \
    --set sourceClusterID=<source-clusterID> \
    --set profile.name=<location-profile-name>

After the Kasten K10 restore job is complete, you can restore cluster-scoped resources and application resources. 

(Note: Prior to recovering applications, you might want to restore cluster-scoped resources. Cluster-scoped resources may be needed for cluster configuration or as part of application recovery.)

Conclusion 


The HPE and Kasten by Veeam partnership helps companies confidently run and protect cloud‑native applications on Kubernetes, across private, hybrid and multi‑cloud environments. Kasten K10 can restore applications on the HPE Ezmeral Container platform to a known state with granular control of backup and restore capabilities for the entire application stack. These container applications can also be seamlessly recovered and redeployed across clusters and sites to facilitate effective DR strategies.

For more information and to explore additional relevant topics, check out the following resources:

 

Michael Courcy

I started my career as a solutions architect, focused mainly on JAVA/JEE for government projects. Now I work as a DevOps architect, building cloud native solutions based on Kubernetes and the main cloud providers like AWS, Azure, and many more.


Share

Recent Blog Posts