On Kubernetes service account token

In one project, we encountered the following issue, which demanded us to dig deep into Kubernetes service account: Jupyterhub deployed on Kubernetes needs to have certain permission to launch Jupyterhub notebook (e.g. create, delete, watch Pod, etc.). However, the service account we get does not have above permission, which prevents Jupyterhub to spawn notebook instance.

Why service account does not above permission?

First, in order for a Kubernetes Pod to manage Pod (e.g. create, delete Pod), it will use the service account that the Pod runs under to talk to Kubernete API. By default, Kubernetes will put JSON Web token under /var/run/secrets/kubernetes.io/serviceaccount/token (along with other certificate). For more information, refer to the following article

Managing Service Accounts

https://github.com/kubernetes-client/python-base/blob/master/config/incluster_config.py

However, by default, this token is generated at Pod creation time and does not expire, which poses some security concern when this token is compromised. There is a proposal about “bound service account token”. However, it is just proposal for now.

Thought process

1. Since we cannot get a service account with appropriate permission, the only workaround is to use a user account with above permission. We are using openshift, and you can login with user account. Actually the login command contains a user account session token (which expires periodically). If we can replace the Pod service account token with this token, the Pod will be able to manage Pod
2. By default, Kubernetes will inject service account secret which contains the token. This secret is mounted as readonly volume. Can we make this readwrite to write our user account token? No.
3. Since we cannot change secret or configmap-mounted volume, we need to leverage emptyDir, to which we can write contents. We will mount secret volume to a temporary location, emptyDir mount at /var/run/secrets/kubernetes.io/serviceaccount, then copy the content from temporary location to /var/run/secrets/kubernetes.io/serviceaccount.

volumeMounts:
   # mount username, password, ca.crt, namespace, service-ca.crt
   - mountPath: /tmp1
     name: volume-svca-temp
     readOnly: false
   # can generate time-scoped token and overwrite token 
   - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
     name: volume-svca
     readOnly: false
...
volumes:
   - name: volume-svca
     emptyDir: {}
   - name: volume-svca-temp
     secret:
       secretName: svc-token
       defaultMode: 0777

1. Command to get session token
   curl -u <username>:<password> -skv -H "X-CSRF-Token: xxx" 'https://master.cluster.local:8443/oauth/authorize?client_id=openshift-challenging-client&response_type=token' | grep -oP "access_token=\K[^&]*"
2. Since we override entire /var/run/secrets/kubernetes.io/serviceaccount, we need original ca.crt, namespace, service-ca.crt (stored in secret svc-token), for this folder to function