Deploying Dynamic NFS Provisioning in Kubernetes

The following documentation is intended to explain the procedure for deploying Dynamic NFS Provisioning in Kubernetes.

Dynamic NFS Provisioning: is allows storage volumes to be created on-demand. The dynamic provisioning feature eliminates the need for cluster administrators to code-provision storage. Instead, it automatically provisions storage when it is requested by users.

Code-Requisites for Dynamic NFS Provisioning in Kubernetes

• Linux Work Station
• K8 Cluster with no other load balancer installed
• Kubernetes-cli or kubectl program
• Kubernetes version v1.15.1 (any version should work)
• Routable IP network with DHCP configured
• Helm Package Manager installed
• Tiller Service Running

For a tutorial on Building a K8s Cluster using Vagrant visit: Building a Kubernetes Cluster using Vagrant
For a tutorial on Installing Helm Package Manager (Tiller) visit: Deploying Helm (Package Manager) in Kubernetes.

Step 1) Installing the NFS Server

In this particular example, we'll allocate a local filesystem from which PersistenceVolume Claims can be made. We'll first create "/srv/nfs/kubedata"

[vagrant@kmaster ~]$ sudo mkdir /srv/nfs/kubedata -p

Change the ownership to "nfsnobody"

[vagrant@kmaster ~]$ sudo chown nfsnobody: /srv/nfs/kubedata/

Next, install the NFS-utils. This example is for centos 7

vagrant@kmaster ~]$ sudo yum install -y nfs-utils

Next, enable and start the userspace NFS server using systemctl.

[vagrant@kmaster ~]$ sudo systemctl enable nfs-server
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.

[vagrant@kmaster ~]$ sudo systemctl start nfs-server

[vagrant@kmaster ~]$ sudo systemctl status nfs-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor codeset: disabled)
   Active: active (exited) since Sat 2019-11-02 22:06:49 UTC; 12s ago

Next, we need to edit the exports file to add the file system we created to be exported to remote hosts.

[vagrant@kmaster ~]$ sudo vi /etc/exports

/srv/nfs/kubedata    *(rw,sync,no_subtree_check,no_root_squash,no_all_squash,insecure)

Next, run the exportfs command to make the local directory we configured available to remote hosts.

[vagrant@kmaster ~]$ sudo exportfs -rav
exporting *:/srv/nfs/kubedata

If you want to see more details about our export file system, you can run "exportfs -v".

vagrant@kmaster ~]$ sudo exportfs -v
/srv/nfs/kubedata
        <world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash)

Next, let's test the nfs configurations. Log onto one of the worker nodes and mount the NFS filesystem andverify.

[vagrant@kworker1 ~]$ sudo mount -t nfs 172.42.42.100:/srv/nfs/kubedata /mnt

[vagrant@kworker1 ~]$ mount | grep kubedata
172.42.42.100:/srv/nfs/kubedata on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.42.42.101,local_lock=none,addr=172.42.42.100)

After verifying that NFS is configured correctly and working we can unmount the filesystem.

[vagrant@kworker1 ~]$ sudo umount /mnt)

Step 2) Deploying Service Account and Role Bindings

Next, we'll configure a service account and role bindings. We'll use role-based access control to do the configuration. The first step is to download the NFS-provisioning repo and change into the NFS-provisioning directory.

git clone https://exxsyseng@bitbucket.org/exxsyseng/nfs-provisioning.git

cd nfs-provisioning

In this directory, we have 4 files. (class.yaml default-sc.yaml deployment.yaml rbac.yaml) We will use the rbac.yaml file to create the service account for NFS and cluster roles and bindings.

[vagrant@kmaster nfs-provisioning]$ kubectl create -f rbac.yaml

We can verify that the service account, clusterrole, and binding was created.

[vagrant@kmaster nfs-provisioning]$ kubectl get clusterrole,clusterrolebinding,role,rolebinding | grep nfs

clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner 20m

clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner 20m
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner 20m
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner 20m

Step 3) Deploying Storage Class

Next, let's run the "class.yaml" to set up the storageclass. A storageclass provides a way for administrators to describe the “classes” of storage they offer.

Let's edit the "class.yaml" file and set both the storageclass name and the provisioner name.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
provisioner: example.com/nfs
parameters:
archiveOnDelete: "false"

Once we've updated the class.yaml file we can execute the file using kubectl create

[vagrant@kmaster nfs-provisioning]$ kubectl create -f class.yaml
storageclass.storage.k8s.io/managed-nfs-storage created

Next, check that the storage class was created.

[vagrant@kmaster nfs-provisioning]$ kubectl get storageclass
NAME                  PROVISIONER       AGE
managed-nfs-storage   example.com/nfs   48s

Step 4) Deploying NFS Provisioner

Now let's deploy the NFS provisioner. But first we'll need to edit the deployment.yaml file. In this file, we'll need to specify the IP Address of our NFS Server (kmaster) 172.42.42.100.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
spec:
  selector:
    matchLabels:
      app: nfs-client-provisioner
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: example.com/nfs
            - name: NFS_SERVER
              value: 172.42.42.100
            - name: NFS_PATH
              value: /srv/nfs/kubedata
      volumes:
        - name: nfs-client-root
          nfs:
            server: 172.42.42.100
            path: /srv/nfs/kubedata

Once we've made the changes, save the file and apply the changes by running "kubectl create".

[vagrant@kmaster nfs-provisioning]$ kubectl create -f deployment.yaml
deployment.apps/nfs-client-provisioner created

After applying the changes, we should see a pod was created for nfs-client-provisioner.

[vagrant@kmaster nfs-provisioning]$ kubectl get all
NAME                                          READY   STATUS    RESTARTS   AGE
pod/nfs-client-provisioner-5b4f5775c7-9j2dw   1/1     Running   0          4m2s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d22h

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nfs-client-provisioner   1/1     1            1           4m2s

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/nfs-client-provisioner-5b4f5775c7   1         1         1       4m2s

We can run "kubectl describe" to see more details about the pod.

[vagrant@kmaster ~]$ kubectl describe pod nfs-client-provisioner-5b4f5775c7-9j2dw
Name:         nfs-client-provisioner-5b4f5775c7-9j2dw
Namespace:    default
Priority:     0
Node:         kworker2.example.com/172.42.42.102
Start Time:   Sun, 03 Nov 2019 20:11:51 +0000
Labels:       app=nfs-client-provisioner
              pod-template-hash=5b4f5775c7
Annotations:  cni.projectcalico.org/podIP: 192.168.136.65/32
Status:       Running
IP:           192.168.136.65
IPs:
  IP:           192.168.136.65
Controlled By:  ReplicaSet/nfs-client-provisioner-5b4f5775c7
Containers:
  nfs-client-provisioner:
    Container ID:   docker://95432ef4c256b48746b61f44a0292557b73abaced78342acafeae3c36681343b
    Image:          quay.io/external_storage/nfs-client-provisioner:latest
    Image ID:       docker-pullable://quay.io/external_storage/nfs-client-provisioner@sha256:022ea0b0d69834b652a4c53655d78642ae23f0324309097be874fb58d09d2919
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sun, 03 Nov 2019 20:11:56 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      PROVISIONER_NAME:  example.com/nfs
      NFS_SERVER:        172.42.42.100
      NFS_PATH:          /srv/nfs/kubedata
    Mounts:
      /persistentvolumes from nfs-client-root (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from nfs-client-provisioner-token-wgwct (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  nfs-client-root:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    172.42.42.100
    Path:      /srv/nfs/kubedata
    ReadOnly:  false
  nfs-client-provisioner-token-wgwct:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nfs-client-provisioner-token-wgwct
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Step 5) Creating Persistent Volume and Persistent Volume Claims

Persistent Volume Claims are objects that request storage resources from your cluster. They’re similar to a voucher that your deployment can redeem for storage access.
Persistent Volume is a resource that can be used by a pod to store data that will persist beyond the lifetime of the pod. It is a storage volume that in this case is a NFS volume.

If we check our cluster we'll see that there are currently no Persistent Volumes or Persistent Volume Claims.

[vagrant@kmaster ~]$ kubectl get pv,pvc
No resources found in default namespace.

Also, we can look in the directory we allocated for Persistent Volumes and see there nothing there.

[vagrant@kmaster ~]$ ls /srv/nfs/kubedata/

Let's create a PVC. Inside the nfs-provisioning repo, there is a file "4-pvc-nfs.yaml". In this example, we will allocate 500 MegaBytes.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1
spec:
  storageClassName: managed-nfs-storage
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Mi

We can create the PVC by running "kubectl create" against the 4-pvc-nfs.yaml" file.

[vagrant@kmaster nfs-provisioning]$ kubectl create -f 4-pvc-nfs.yaml
persistentvolumeclaim/pvc1 created

We can now view the PVC and PV that was allocated. As we can see below a PCV was created "persistentvolumeclaim/pvc1" and its bound to a PV "pvc-eca295aa-bc2c-420c-b60e-9a6894fc9daf". The PV was created automatically by the nfs-provisoner.

[vagrant@kmaster nfs-provisioning]$ kubectl get pvc,pv
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
persistentvolumeclaim/pvc1   Bound    pvc-eca295aa-bc2c-420c-b60e-9a6894fc9daf   500Mi      RWX            managed-nfs-storage   2m30s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM          STORAGECLASS          REASON   AGE
persistentvolume/pvc-eca295aa-bc2c-420c-b60e-9a6894fc9daf   500Mi      RWX            Delete           Bound    default/pvc1   managed-nfs-storage            2m30s
[vagrant@kmaster nfs-provisioning]$

Step 6) Creating a Pod to use Persistent Volume Claims

Now that we have our nfs-provisoner working and we have both a PVC and OV that it is bound to. Let's create a pod to use our PVC. If we take a quick look at the existing pods we'll see that only the "nfs-client-provisioner" pod is running.

[vagrant@kmaster nfs-provisioning]$ kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-5b4f5775c7-9j2dw   1/1     Running   0          4h36m

Next, we'll create a pod using the "4-busybox-pv-nfs.yaml" file. But first let's take a look at the file's contents.

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  volumes:
  - name: host-volume
    persistentVolumeClaim:
      claimName: pvc1
  containers:
  - image: busybox
    name: busybox
    command: ["/bin/sh"]
    args: ["-c", "sleep 600"]
    volumeMounts:
    - name: host-volume
      mountPath: /mydata

We'll execute test-pod-pvc1.yaml using "kubectl create".

[vagrant@kmaster nfs-provisioning]$ kubectl create -f 4-busybox-pv-nfs.yaml
pod/busybox created

We can now see that the pod is up and running.

[vagrant@kmaster nfs-provisioning]$ kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
busybox                                   1/1     Running   0          69s
nfs-client-provisioner-5b4f5775c7-9j2dw   1/1     Running   0          7h33m

We can describe the pod to see more details.

[vagrant@kmaster nfs-provisioning]$ kubectl describe pod busybox
Name:         busybox
Namespace:    default
Priority:     0
Node:         kworker1.example.com/172.42.42.101
Start Time:   Mon, 04 Nov 2019 03:44:30 +0000
Labels:       <none>
Annotations:  cni.projectcalico.org/podIP: 192.168.33.194/32
Status:       Running
IP:           192.168.33.194
IPs:
  IP:  192.168.33.194
Containers:
  busybox:
    Container ID:  docker://f27b38404abbfd3ab77fe81b23e148e0a15f4779420ddfcb17eebcbe699767f3
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      sleep 600
    State:          Running
      Started:      Mon, 04 Nov 2019 03:44:34 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mydata from host-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p2ctq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  host-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvc1
    ReadOnly:   false
  default-token-p2ctq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p2ctq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s

We can log into the container to view the mount point and create a file for testing

[vagrant@kmaster nfs-provisioning]$ kubectl exec -it busybox -- ./bin/sh
/ # 
/ # ls /mydata/

/ # > /mydata/myfile

/ # ls /mydata/
myfile

Now that we've created a file called myfile, we can log into the mastrer node and verify the file by looking in the PV directory that was allocated for this pod.

[vagrant@kmaster nfs-provisioning]$ ls /srv/nfs/kubedata/default-pvc1-pvc-eca295aa-bc2c-420c-b60e-9a6894fc9daf/
myfile

Step 7) Deleting Pods with Persistent Volume Claims

• To delete the pod just use "kubectl delete pod [pod name]"

[vagrant@kmaster nfs-provisioning]$ kubectl delete pod busybox
pod "busybox" deleted

• Deleting the pod will delete the pod but not the PV and PVC. This will have to be done separately.

[vagrant@kmaster nfs-provisioning]$ kubectl get pvc,pv
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
persistentvolumeclaim/pvc1   Bound    pvc-eca295aa-bc2c-420c-b60e-9a6894fc9daf   500Mi      RWX            managed-nfs-storage   3h26m

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM          STORAGECLASS          REASON   AGE
persistentvolume/pvc-eca295aa-bc2c-420c-b60e-9a6894fc9daf   500Mi      RWX            Delete           Bound    default/pvc1   managed-nfs-storage            3h26m

• To delete the PV and PVC use "kubectl delete"

[vagrant@kmaster nfs-provisioning]$ kubectl delete pvc --all
persistentvolumeclaim "pvc1" deleted

• PV and PVC resources are gone

[vagrant@kmaster nfs-provisioning]$ kubectl get pvc,pv
No resources found in default namespace.

Final Thoughts on Dynamic NFS Provisioning in Kubernetes.

Dynamic NFS provisioning allows storage volumes to be created on-demand. Without dynamic NFS provisioning, cluster administrators have to manually create new storage volumes, and then create

PersistentVolume

objects to recodesent them in Kubernetes. The dynamic NFS provisioning feature eliminates the need for cluster administrators to code-provision storage. Instead, it automatically provisions storage when it is requested by users.

Related Blogs

• Building a Kubernetes Cluster Using Vagrant