Configure Persistent Image Registry in Openshift using NFS

In this article, we will see how to configure persistent image registry in Openshift by using NFS with Persistent Volume (PV) and Persistent Volume claim (PVC) resources.

By default Openshift installer configures default registry. Installer setup the volume for registry by exporting NFS volume from the master node. But this is not ideal for production setup. So we usually need to configure persistent storage for registry.

Verify that the OCP internal registry is running and includes a default PersistentVolumeClaim (PVC) named registry-claim.

Step 1: Login to master node with system user and select default project.

[root@master ~]# oc login -u system:admin

Logged into “https://master.lab.example.com:8443” as “system:admin” using existing credentials.

You have access to the following projects and can switch between them with ‘oc project <projectname>’:

* default

kube-system

logging

management-infra

openshift

openshift-infra

Using project “default”.

 

Step 2: Verify that the docker-registry pod is running and find the pod name

[root@master ~]# oc get pods

docker-registry-6-d21wk    1/1       Running   1          21h

registry-console-1-ph7zv   1/1       Running   1          21h

router-1-vi46b             1/1       Running   1          21h

Step 3: Verify the default persistent volume and persistent volume claim created by the installer

[root@master ~]# oc get pv; oc get pvc

NAME              CAPACITY   ACCESSMODES   ..   STATUS    CLAIM

registry-volume   5Gi        RWX           ..   Bound     default/registry-claim

 

NAME             STATUS    VOLUME            CAPACITY   ACCESSMODES   AGE

registry-claim   Bound     registry-volume   5Gi        RWX           13h

 

Step 4: Use the oc volume pod command to determine if the docker-registry pod identified in above step has a PVC defined as registry-claim

[root@master ~]# oc volume pod docker-registry-6-d21wk

pods/docker-registry-6-d21wk

pvc/registry-claim (allocated 5GiB) as registry-storage

mounted at /registry

secret/registry-certificates as volume-a579i

mounted at /etc/secrets

secret/registry-token-fnw7y as registry-token-fnw7y

mounted at /var/run/secrets/kubernetes.io/serviceaccount

 

Step 5: Find the registry DeploymentConfig name

[root@master ~]# oc status

In project default on server https://master.lab.example.com:8443

https://docker-registry-default.cloudapps.lab.example.com (passthrough) to pod port 5000-tcp (svc/docker-registry)

dc/docker-registry deploys docker.io/openshift3/ose-docker-registry:v3.4.0.39

deployment #6 deployed 13 hours ago – 1 pod

 

Step 6: Verify that the pod mounts the default PVC to /registry from the default registry DeploymentConfig

[root@master ~]# oc volume dc docker-registry

deploymentconfigs/docker-registry

pvc/registry-claim (allocated 5GiB) as registry-storage

mounted at /registry

secret/registry-certificates as volume-dad50

mounted at /etc/secrets

 

Step 7: Verify that the current registry DeploymentConfig shows volumes and volumeMounts attributes

[root@master ~]# oc get dc docker-registry -o json | less

“spec”: {

“volumes”: [

{

“name”: “registry-storage”,

“persistentVolumeClaim”: {

“claimName”: “registry-claim”

}

},

“volumeMounts”: [

{

“name”: “registry-storage”,

“mountPath”: “/registry”

},

Step 8: Create NFS share from master host and export it with nfsnobody user. The reason behind this is each container has random UID, in that case NFS share will not accessible inside pod.

[root@master ~]# mkdir -p /var/export/registryvol

[root@master ~]# chown nfsnobody:nfsnobody /var/export/registryvol

[root@master ~]# chmod 700 /var/export/registryvol

Export the folder

[root@master ~]# vi /etc/exports.d/training-registry.exports

/var/export/registryvol *(rw,async,all_squash)

Save and exit file.

[root@master ~]# exportfs –a

[root@master ~]# showmount –e

Export list for master.lab.example.com:

/var/export/registryvol *

 

Step 9: On master host create new Persistent Volume (PV) resource which will use NFS share from master host, following are resource definition of PV in json format.

[root@master ~]# vi training-registry-volume.json

{

“apiVersion”: “v1”,

“kind”: “PersistentVolume”,

“metadata”: {

“name”: “training-registry-volume”,

“labels”: {

“deploymentconfig”: “docker-registry”

}

},

“spec”: {

“capacity”: {

“storage”: “10Gi”

},

“accessModes”: [ “ReadWriteMany” ],

“nfs”: {

“path”: “/var/export/registryvol”,

“server”: “master.lab.example.com”

}

}

}

Step 10: Create PV using oc create command and check PV status.

[root@master ~]# oc create –f training-registry-volume.json

persistentvolume “training-registry-volume” created

[root@master ~]# oc get pv

NAME                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM

registry-volume            5Gi        RWX           Retain          Bound       default/registry-claim

training-registry-volume   10Gi       RWX           Retain          Available

Step 11: On master host create Persistent Volume Claim (PVC) Definition.

[root@master ~]#  vi /root/DO280/labs/deploy-registry/training-registry-pvclaim.json

{

“apiVersion”: “v1”,

“kind”: “PersistentVolumeClaim”,

“metadata”: {

“name”: “training-registry-pvclaim”,

“labels”: {

“deploymentconfig”: “docker-registry”

}

},

“spec”: {

“accessModes”: [ “ReadWriteMany” ],

“resources”: {

“requests”: {

“storage”: “10Gi”

}

}

}

}

Step 12: Create PVC using oc create command and check PVC status.

[root@master ~]# oc create -f training-registry-pvclaim.json

persistentvolumeclaim “training-registry-pvclaim” created

[root@master ~]# oc get pvc

NAME                        STATUS    VOLUME                     CAPACITY   ACCESSMODES   AGE

registry-claim              Bound     registry-volume            5Gi        RWX           17h

training-registry-pvclaim   Bound     training-registry-volume   10Gi       RWX           55s

 

Step 13: Attach PV to deployment config of docker registry with oc volume command as below.

[root@master ~]# oc volume dc docker-registry \

–add –overwrite -t pvc \

–claim-name=training-registry-pvclaim –name=registry-storage

deploymentconfig “docker-registry” updated

 

Note: where, –claim-name specifies the PVC name and –name specifies the pod volume name.

 

Step 14: Verify that the DeploymentConfig of docker-registry was changed to use the new PVC

[root@master ~]# oc get dc docker-registry -o json  | less

“spec”: {

“volumes”: [

{

“name”: “registry-storage”,

“persistentVolumeClaim”: {

“claimName”: “training-registry-pvclaim”

}

},

Step 15: Verify that the DeploymentConfig docker-registry started a new registry pod after detecting that the deployment configuration had been changed

[root@master ~]# watch oc status -v

In project default on server https://master.lab.example.com:8443

https://docker-registry-default.cloudapps.lab.example.com (passthrough) to pod port 5000-tcp (svc/docker-registry)

dc/docker-registry deploys docker.io/openshift3/ose-docker-registry:v3.4.0.39

deployment #7 deployed about a minute ago – 1 pod

deployment #6 deployed 17 hours ago

 

Step 16: Verify docker registry pod is running.

[root@master ~]# oc get pods

NAME                       READY     STATUS    RESTARTS   AGE

docker-registry-7-1gwd4    1/1       Running   0          9m

registry-console-1-zlrry   1/1       Running   2          17h

router-1-32toa             1/1       Running   2          17h

 

Finally we completed configuration of openshift image registry with persistent storage i.e using NFS.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s