This blogpost paraphrases my Keptn slack thread around the same issue so that:

  1. It’s easier to understand
  2. It’s available in public (searchable through Google)

Problem

How do you fix an error like this in the mongodb pod(s) when installing Keptn

1
2
3
4

mongodb 05:51:53.11 
mongodb 05:51:53.11 INFO  ==> ** Starting MongoDB setup **
mongodb 05:51:53.12 INFO  ==> Validating settings in MONGODB_* env vars...
mkdir: cannot create directory '/bitnami/mongodb/data': Permission denied

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

suraj@suraj:~$ kubectl get po 
NAME                                     READY   STATUS             RESTARTS        AGE
api-gateway-nginx-b47f6d57c-v885m        1/1     Running            0               6m1s
api-service-6457b97dd-gpc6w              2/2     Running            3 (3m50s ago)   6m1s
approval-service-6f9cfc9f99-r66zv        2/2     Running            0               6m1s
bridge-db68fc84c-pjcvf                   1/1     Running            0               6m1s
configuration-service-5869576875-gtbx4   1/1     Running            0               6m
keptn-mongo-74bf4c476f-klfdh             0/1     CrashLoopBackOff   5 (109s ago)    6m1s
keptn-nats-0                             2/2     Running            0               6m1s
lighthouse-service-845f699b8c-5dt54      2/2     Running            0               6m1s
mongodb-datastore-5bd7f977d9-82955       2/2     Running            3 (3m53s ago)   6m1s
remediation-service-8577f49d8c-zpbd4     2/2     Running            0               6m1s
secret-service-85f68fd54-gjrpn           1/1     Running            0               6m
shipyard-controller-76f494c5c8-r5q5j     0/1     Running            5 (46s ago)     6m1s
statistics-service-78d9848787-qb95m      2/2     Running            3 (4m12s ago)   6m1s
webhook-service-7d8fc7cc74-dfgmv         2/2     Running            0               6m1s

And you’d see the following events for keptn-mongo-xxx pod:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  9m20s                 default-scheduler  Successfully assigned keptn/keptn-mongo-74bf4c476f-klfdh to datadog-demo-m02
  Normal   Pulling    9m19s                 kubelet            Pulling image "docker.io/bitnami/mongodb:4.4.13-debian-10-r52"
  Normal   Pulled     8m5s                  kubelet            Successfully pulled image "docker.io/bitnami/mongodb:4.4.13-debian-10-r52" in 1m14.356743512s
  Normal   Created    7m24s (x4 over 8m5s)  kubelet            Created container mongodb
  Normal   Pulled     7m24s (x3 over 8m4s)  kubelet            Container image "docker.io/bitnami/mongodb:4.4.13-debian-10-r52" already present on machine
  Normal   Started    7m23s (x4 over 8m5s)  kubelet            Started container mongodb
  Warning  BackOff    4m8s (x29 over 8m3s)  kubelet            Back-off restarting failed container

Keptn uses mongodb for datastore. If the keptn-mongo-xxx pod goes into CrashLoopBackOff it prevents other dependent pods (like shipyard-controller above) from coming up. The problem happened with me for Keptn version 0.15.0 but it can happen for any other Keptn version (example) as well.

Why does this happen?

We use bitnami image (check the pod logs above) for mongodb. Here’s a longer version of the logs above to put things in context:

1
2
3
4
5
6
7
8
9

suraj@suraj:~$ kubectl logs  keptn-mongo-74bf4c476f-klfdh
mongodb 05:51:53.10 
mongodb 05:51:53.10 Welcome to the Bitnami mongodb container
mongodb 05:51:53.10 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
mongodb 05:51:53.10 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
mongodb 05:51:53.11 
mongodb 05:51:53.11 INFO  ==> ** Starting MongoDB setup **
mongodb 05:51:53.12 INFO  ==> Validating settings in MONGODB_* env vars...
mkdir: cannot create directory '/bitnami/mongodb/data': Permission denied

At the time of writing this blogpost, Keptn helm chart is dependent on bitnami’s mongodb helm chart:

1
2
3
4
5

  - name: mongodb
    version: 12.1.31
    repository: https://charts.bitnami.com/bitnami
    condition: mongo.enabled
    alias: mongo

https://github.com/keptn/keptn/blob/22b396734a3f5a48fccedcd26e78471d8de72b48/installer/manifests/keptn/Chart.yaml#L42-L46

Bitnami charts are configured to use, by default, a Kubernetes SecurityContext to automatically modify the ownership of the attached volumes. However, this feature does not work if:

  • Your Kubernetes distribution has no support for SecurityContexts.
  • The Storage Class used to provision the Persistent Volumes has no support to modify the volumes' filesystem.

https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/

Bitnami also provides a solution to this problem:

Upgrade the chart release enabling the initContainer that adapts the permissions:
helm upgrade MY-RELEASE bitnami/mongodb --set volumePermissions.enabled=true

Solution

Set volumePermissions.enabled=true

In context of Keptn, there are two ways in which you can install Keptn at the time of writing this blogpost:

  1. Using Keptn CLI
  2. Using Keptn Helm chart

Note: Support for installing Keptn via CLI has been removed starting 0.18.0 version of Keptn

Irrespective of what you use both do the same thing i.e., Keptn CLI also installs Keptn helm chart under the hood with the right Helm values. You can check what values were used for installing your Keptn helm chart:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

suraj@suraj:~$ helm get values keptn
USER-SUPPLIED VALUES:
continuous-delivery:
  enabled: true
control-plane:
  apiGatewayNginx:
    type: ClusterIP
  bridge:
    installationType: QUALITY_GATES,CONTINUOUS_OPERATIONS,CONTINUOUS_DELIVERY
  enabled: true

Aside: Keptn has supported installing Keptn via Helm since as far back as 0.7.0 version

Copy everything below USER-SUPPLIED VALUES: and paste it in a file new-values.yaml. Add volumePermissions.enabled=true:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# new-values.yaml

# existing values
continuous-delivery:
  enabled: true
control-plane:
  apiGatewayNginx:
    type: ClusterIP
  bridge:
    installationType: QUALITY_GATES,CONTINUOUS_OPERATIONS,CONTINUOUS_DELIVERY
  enabled: true
  mongo: # new value (if your Keptn version is < 0.17.0 and >= 11.0)
    volumePermissions:
      enabled: true 
  

# new value (if your Keptn version is >= 0.17.0)
mongo:
  volumePermissions:
    enabled: true 
#

Please note that bitnami/mongodb is aliased to mongo

Upgrade Keptn helm chart

You can find your current Keptn version using

1
2
3
4

suraj@suraj:~/sandbox$ helm ls -nkeptn
NAME           	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART                 	APP VERSION
...   
keptn          	keptn    	1       	2022-09-08 23:41:37.503597202 +0530 IST	deployed	keptn-0.18.1          	0.18.1

0.18.1 in keptn-0.18.1 is our current chart version.

1

helm upgrade keptn keptn/keptn -f installer/manifests/keptn/current-values.yaml --version <your-current-keptn-version> -nkeptn

This should fix the issue. If you still face this issue, reach out on Keptn slack in #help or #general channel or at #keptn channel in CNCF Slack workspace.