Allow mounting Secrets on Workspaces

[thesuperzapper]

There has been a lot of discussion on how we can enable users to store credentials in Kubernetes secrets, and then make them available for usage in their Workspace Pods.

Goal

Avoid users storing sensitive things within their notebooks (e.g. db/git credentials), and let them use Kubernetes secrets instead.

Exclusions

• Not allow users to set arbitrary env-vars, only allow mounting secrets as files with projected volume mounts.

Features

• We will let users mount keys from Secrets as files on the Workspace container.
• We will let users create new secrets (or mount existing secrets) when spawning/updating a Workspace.
• We will let users mount existing secrets when spawning/updating a Workspace.
• We will provide a "Secrets" page, similar to the existing "Volumes" page (which also needs to be re-implemented for NB2.0)

Implementation

STEP 1: Update Controller + Workspace CRD

We will need to add spec.podTemplate.volumes.secrets[] to the Workspace CRD:

spec:
  podTemplate:
    volumes:
      secrets:
        - secretName: "workspace-secret"
          mountPath: "/secrets/my-secret"
          defaultMode: 420 # same as 0644 in octal

When we implement the reconciliation in the controller, we MUST use remember that Kubernetes does not update secrets mounted with subPath (except on container restarts), so its best to mount the whole secret as a folder, or possibly extend this to specific items in the future (but still in a folder for the secret).

STEP 2: Extend Workspace Backend

We then need to:

• extend the Workspace GET/LIST/CREATE/UPDATE API to support the new secrets field
• implement API paths for listing Secrets in the namespace
  • WARNING: we need to be careful when using controller-runtime with Secrets, because controller-runtime will try and cache all secrets in the cluster (to do its list-watch approach), this is a problem in very large clusters because they might have gigabytes of secrets and all would need to be in memory.
    • In other projects, a couple of approaches been taken.
    • OPTION 1: use a metadata-only cache, which does not cache the data part of the secret
    • OPTION 2: disable the list-watch cache for all secrets (fixes memory usage at the cost of Kube API calls, can be done in combination with option 1)
    • OPTION 3: use some kind of partial cache which only caches secrets with a specific label (probably will not work in our case)

The create workspace api might look like this after the updates:

# POST /api/v1/workspaces/{namespace}
curl -X POST http://localhost:4000/api/v1/workspaces/default \
    -H "Content-Type: application/json" \
    -d '{
    "data": {
        "name": "dora",
        "kind": "jupyterlab",
        "paused": false,
        "deferUpdates": false,
        "podTemplate": {
            "podMetadata": {
                "labels": {
                    "app": "dora"
                },
                "annotations": {
                    "app": "dora"
                }
            },
            "volumes": {
                "home": "workspace-home-bella",
                "data": [
                    {
                        "pvcName": "workspace-data-bella",
                        "mountPath": "/data/my-data",
                        "readOnly": false
                    }
                ],
                "secrets": [
                    {
                        "secretName": "workspace-secret",
                        "mountPath": "/secrets/my-secret",
                        "defaultMode": 420
                    }
                ]
            },
            "options": {
                "imageConfig": "jupyterlab_scipy_190",
                "podConfig": "tiny_cpu"
            }
        }
    }
}'

STEP 3: Update Workspace Spawner UI

We need to update the Workspace spawner/editor UI to prompt the user to mount known secrets from the namespace of the workspace.

Possibly also prompt them to create a new secret, so they can safely store credentials for things like databases.

STEP 4: Create a "Secrets" tab in the UI

Similar to the volumes tab, we can create one for secrets, which lets users with the required Kubernetes RBAC (based on their userid-header value) do things like get/list/create new secrets.

We can also provide integration with the workspaces by showing which workspaces are mounting each secret.

(Similarly, we should move the volumes to a tab within the Workspace frontend, rather than the old CRUD webapps).

[ederign]

/retitle Allow mounting Secrets on Workspaces
/area controller
/area v2

[thaorell]

@thesuperzapper @ederign For the UI task, I think we should allow users to input octal values for defaultMode for better UX (644 as default), and then convert it back to decimal before submitting the data to the API. What are your thoughts on this?