Private clusters

​

How it works

• The agent connects to Lumovi’s api/agent with a WebSocket, naming its cluster and sending its token. Lumovi checks them against the agents it knows.
• It tells Lumovi how to reach its cluster: the API server’s certificate authority, and its service account’s token. That service account may only impersonate users and groups.
• Lumovi’s connections to the API server run through it. The agent connects each one to the API server, at the address Kubernetes gives its pod. TLS runs inside, from Lumovi to the API server: Lumovi checks the API server’s certificate against the cluster’s certificate authority, as kubernetes.default.svc, and the agent only passes on bytes it can’t read.
• Lumovi acts as whoever signed in, impersonating them with the service account’s token, so the cluster applies their RBAC, as with any member.
• Kubernetes rotates the token, and the agent reads it again every minute. When it changed, the agent sends Lumovi the new one.

​

Set it up

TOKEN=$(openssl rand -hex 32)
printf %s "$TOKEN" | sha256sum   # shasum -a 256 on macOS

- name: private-eu
  tokenSha256: <the token's SHA-256, from step 1>
  labels: { env: production, region: eu-west }

kubectl create secret generic lumovi-agents --namespace lumovi --from-file agents.yaml

fleet:
  agentsSecret: lumovi-agents

kubectl create namespace lumovi
kubectl create secret generic lumovi-agent --namespace lumovi \
  --from-literal token="$TOKEN"
helm install lumovi oci://ghcr.io/lumovi/charts/lumovi --namespace lumovi \
  --set mode=agent \
  --set clusterName=private-eu \
  --set agent.hubUrl=https://lumovi.example.com \
  --set agent.tokenSecret=lumovi-agent

kubectl logs --namespace lumovi deployment/lumovi-agent

Lumovi 1.1.0's agent for private-eu, relaying to 10.96.0.1:443
Connected to https://lumovi.example.com as private-eu

​

The agents Lumovi knows

​

What the chart makes

• A Deployment, lumovi-agent for a release called lumovi, that runs the agent from Lumovi’s image. It has one replica, whatever replicaCount says, and is replaced with the Recreate strategy: a second agent of the same cluster would take over from the first.
• A service account, lumovi, with a ClusterRole and a ClusterRoleBinding, lumovi-impersonate, that let it impersonate users and groups, and nothing else (unless rbac.create is false). Its token is mounted in the agent’s pod.
• Health checks on port 8081. /healthz answers 200 connected while the agent is connected to Lumovi, and 503 not connected otherwise, so the pod is ready only while connected. It’s alive as long as the port answers.

​

Run it another way

containers:
  - name: agent
    image: ghcr.io/lumovi/lumovi:1.1.0
    args: [/app/out/agent/agent.js]
    env:
      - name: LUMOVI_HUB_URL
        value: https://lumovi.example.com
      - name: LUMOVI_AGENT_NAME
        value: private-eu
      - name: LUMOVI_AGENT_TOKEN
        valueFrom:
          secretKeyRef: { name: lumovi-agent, key: token }

LUMOVI_HUB_URL must be set: see https://docs.lumovi.dev/server/fleet/agents
LUMOVI_HUB_URL must be the hub's address, like https://lumovi.example.com, not "lumovi.example.com".
No service account token in /var/run/secrets/kubernetes.io/serviceaccount: the agent needs its pod's (automountServiceAccountToken).

​

Connecting, and staying connected

• When it can’t reach Lumovi, or its connection closes, the agent tries again after 1 second, then 2, 4, 8 and 15, and every 30 seconds after that. Once connected, it starts again from 1.
• When Lumovi restarts, for an upgrade say, it closes its agents’ connections, and they connect to the new one.
• Lumovi pings its agents as often as it checks pages’ connections (LUMOVI_HEARTBEAT_SECONDS, every 30 seconds), and closes a connection that doesn’t answer.
• The agent listens for those pings, as Lumovi tells it how often they come. A connection dropped somewhere on the way, without a word, goes quiet: after two heartbeats and a second without one (61 seconds), the agent says Nothing from the hub in 61 s, closes it, and connects again.
• When Lumovi refuses it, the agent stops, as trying again wouldn’t help, and Kubernetes restarts it until it’s fixed:

​

In front of Lumovi

• Behind an authenticating proxy, let api/agent through to Lumovi without signing in (with oauth2-proxy, --skip-auth-route). Agents can’t sign in to a proxy, and Lumovi checks their names and tokens itself. A proxy that answers 401 makes an agent stop, saying Lumovi doesn’t know its name and token.
• With the chart’s networkPolicy, its from must let in whatever agents come through, like your ingress controller or proxy.
• A server of one cluster refuses agents, with 403: only a fleet takes them.

​

Security

• Nothing in the cluster listens. The agent only dials out, to Lumovi, and has no Service.
• The agent can’t read what it relays. Lumovi checks the API server’s certificate itself, end to end, against the certificate authority the agent sent.
• Its service account may only impersonate, like a member’s, and Lumovi uses its token only while the agent is connected.
• Whoever has an agent’s token can connect as its cluster. As the agent says which certificate authority to trust, they could stand in for the cluster with a server of their own, and see what people send it: their own tokens too, with forwardToken. Keep the token in a Secret in that cluster that only its administrators can read, and give Lumovi only its SHA-256 (tokenSha256), so Lumovi’s settings don’t hold it.
• A takeover shows. When another agent connects with the same name and token, the first one stops, and its log says another agent connected as it. Unless two clusters were given the same name and token by mistake, that’s a sign the token is somewhere it shouldn’t be: make a new one.