> ## Documentation Index > Fetch the complete documentation index at: https://docs.lumovi.dev/llms.txt > Use this file to discover all available pages before exploring further. # Private clusters > Clusters a fleet's Lumovi can't reach: an agent in each one dials Lumovi, and relays its connections to the API server. When a cluster's API server is in a network Lumovi can't reach (a private network, behind NAT, in a data center of your own), run Lumovi's agent in it. The agent dials Lumovi over HTTPS, and Lumovi reaches the API server through that connection. Nothing in the cluster listens for Lumovi, and no firewall has to let anything in. ## How it works ```mermaid theme={"theme":{"light":"github-light","dark":"github-dark-default"}} flowchart LR L["Lumovi"] <-->|"A WebSocket the agent opened"| G["The agent"] G -->|"Each of Lumovi's connections"| A["The API server"] L -.->|"TLS, end to end"| A ``` * **The agent connects** to Lumovi's `api/agent` with a WebSocket, naming its cluster and sending its token. Lumovi checks them against the agents it knows. * **It tells Lumovi how to reach its cluster**: the API server's certificate authority, and its service account's token. That service account may only impersonate users and groups. * **Lumovi's connections to the API server run through it.** The agent connects each one to the API server, at the address Kubernetes gives its pod. TLS runs inside, from Lumovi to the API server: Lumovi checks the API server's certificate against the cluster's certificate authority, as `kubernetes.default.svc`, and the agent only passes on bytes it can't read. * **Lumovi acts as whoever signed in**, impersonating them with the service account's token, so the cluster applies their RBAC, as with any [member](/server/fleet/members). * **Kubernetes rotates the token**, and the agent reads it again every minute. When it changed, the agent sends Lumovi the new one. ## Set it up The agent proves which cluster it is with a token. Lumovi only needs its SHA-256: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-default"}} TOKEN=$(openssl rand -hex 32) printf %s "$TOKEN" | sha256sum # shasum -a 256 on macOS ``` Hash the token itself, without a line break after it, as `printf %s` does. List the agents that may connect in `LUMOVI_FLEET_AGENTS`, as YAML, or that encoded in base64 on one line: ```yaml agents.yaml theme={"theme":{"light":"github-light","dark":"github-dark-default"}} - name: private-eu tokenSha256: labels: { env: production, region: eu-west } ``` With the Helm chart, put the list in a Secret, under `agents.yaml`, in Lumovi's namespace, and name it in `fleet.agentsSecret`: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-default"}} kubectl create secret generic lumovi-agents --namespace lumovi --from-file agents.yaml ``` ```yaml values.yaml theme={"theme":{"light":"github-light","dark":"github-dark-default"}} fleet: agentsSecret: lumovi-agents ``` Lumovi reads the list when it starts: restart it after changing the list (`kubectl rollout restart deployment/lumovi --namespace lumovi`). Its pod doesn't start until the Secret is there. Until the agent connects, the cluster's card says **Unreachable**: `Its agent isn't connected.` With your kubeconfig pointing at the private cluster, store the token, and install the chart as an agent, with the name Lumovi knows it by: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-default"}} kubectl create namespace lumovi kubectl create secret generic lumovi-agent --namespace lumovi \ --from-literal token="$TOKEN" helm install lumovi oci://ghcr.io/lumovi/charts/lumovi --namespace lumovi \ --set mode=agent \ --set clusterName=private-eu \ --set agent.hubUrl=https://lumovi.example.com \ --set agent.tokenSecret=lumovi-agent ``` `agent.hubUrl` is the address people open Lumovi at, with its base path if it has one. `clusterName` must be the agent's name in Lumovi's list: unless set, it's `in-cluster`. The chart's notes, after installing, print a command that gives the SHA-256 of the token in `agent.tokenSecret`, and the entry for Lumovi's list, if you'd rather make the token here first. The agent's log says when it's connected: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-default"}} kubectl logs --namespace lumovi deployment/lumovi-agent ``` ```text theme={"theme":{"light":"github-light","dark":"github-dark-default"}} Lumovi 1.1.0's agent for private-eu, relaying to 10.96.0.1:443 Connected to https://lumovi.example.com as private-eu ``` Lumovi's log says `The agent of private-eu connected (Lumovi 1.1.0)`, and the cluster is on the fleet page. ## The agents Lumovi knows `LUMOVI_FLEET_AGENTS` (or the chart's `fleet.agentsSecret`) is a list, each with: The cluster's name in the fleet, and the agent's `LUMOVI_AGENT_NAME` (the chart's `clusterName`). Two agents can't have the same name. The SHA-256 of the agent's token, in hex: 64 characters. Lumovi's settings then don't hold the token itself. The agent's token itself, at least 32 characters, instead of `tokenSha256`. The cluster's labels, like `{ env: production }`. Only people in one of these groups see the cluster: groups as your provider or proxy names them, without prefixes. Everyone signed in does, without it, and an empty list hides it from everyone. `true`: requests carry each person's own token, for an API server that trusts your provider, instead of impersonating them. Lumovi must have their tokens: set `LUMOVI_OIDC_FORWARD_TOKEN`. An agent's cluster takes the server's prefixes for the names it impersonates. A list that isn't like this stops Lumovi, saying what's wrong, like `LUMOVI_FLEET_AGENTS[0] (private-eu) needs a token of at least 32 characters, or its tokenSha256.` ## What the chart makes With `mode: agent`, the chart makes no Lumovi in the cluster, only its agent: * **A Deployment**, `lumovi-agent` for a release called `lumovi`, that runs the agent from Lumovi's image. It has one replica, whatever `replicaCount` says, and is replaced with the `Recreate` strategy: a second agent of the same cluster would take over from the first. * **A service account**, `lumovi`, with a ClusterRole and a ClusterRoleBinding, `lumovi-impersonate`, that let it impersonate users and groups, and nothing else (unless `rbac.create` is `false`). Its token is mounted in the agent's pod. * **Health checks** on port 8081. `/healthz` answers `200 connected` while the agent is connected to Lumovi, and `503 not connected` otherwise, so the pod is ready only while connected. It's alive as long as the port answers. There's no Service: nothing connects to the agent. It requests `10m` of CPU and `32Mi` of memory, and is limited to `128Mi` (`agent.resources`). The chart's `image`, `extraEnv`, `podSecurityContext`, `securityContext`, `nodeSelector`, `tolerations`, `affinity` and `priorityClassName` apply to it too. See [Helm values](/server/helm-values#an-agent). ## Run it another way The agent is in Lumovi's image, as `/app/out/agent/agent.js`. It runs in the cluster it relays to: it connects to the API server at `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` (443 unless set), which Kubernetes sets in every pod, and needs its pod's service account token and CA certificate. Give its service account permission to impersonate users and groups, and nothing more, as a [member's](/server/fleet/members#without-the-chart) has. ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-default"}} containers: - name: agent image: ghcr.io/lumovi/lumovi:1.1.0 args: [/app/out/agent/agent.js] env: - name: LUMOVI_HUB_URL value: https://lumovi.example.com - name: LUMOVI_AGENT_NAME value: private-eu - name: LUMOVI_AGENT_TOKEN valueFrom: secretKeyRef: { name: lumovi-agent, key: token } ``` Lumovi's address, with its base path if it has one: an `http` or `https` URL. The agent connects to `api/agent` below it, with a `wss` or `ws` WebSocket. Use `https`: the token travels in the connection's headers. The cluster's name, as Lumovi's list of agents has it. The agent's token. Spaces and line breaks around it are left out, so hash it without them. Where `GET /healthz` says whether the agent is connected. `0`: nowhere. How often the agent reads its service account's token again, in seconds. Where the pod's service account token and CA certificate are. Without one of the required settings (`KUBERNETES_SERVICE_HOST` among them), or without the service account's token, the agent stops at once and says why: ```text theme={"theme":{"light":"github-light","dark":"github-dark-default"}} LUMOVI_HUB_URL must be set: see https://docs.lumovi.dev/server/fleet/agents LUMOVI_HUB_URL must be the hub's address, like https://lumovi.example.com, not "lumovi.example.com". No service account token in /var/run/secrets/kubernetes.io/serviceaccount: the agent needs its pod's (automountServiceAccountToken). ``` When Kubernetes stops it, it closes its connection, and Lumovi's log says `The agent of private-eu disconnected`. ## Connecting, and staying connected * **When it can't reach Lumovi**, or its connection closes, the agent tries again after 1 second, then 2, 4, 8 and 15, and every 30 seconds after that. Once connected, it starts again from 1. * **When Lumovi restarts**, for an upgrade say, it closes its agents' connections, and they connect to the new one. * **Lumovi pings its agents** as often as it checks pages' connections (`LUMOVI_HEARTBEAT_SECONDS`, every 30 seconds), and closes a connection that doesn't answer. * **The agent listens for those pings**, as Lumovi tells it how often they come. A connection dropped somewhere on the way, without a word, goes quiet: after two heartbeats and a second without one (61 seconds), the agent says `Nothing from the hub in 61 s`, closes it, and connects again. * **When Lumovi refuses it, the agent stops**, as trying again wouldn't help, and Kubernetes restarts it until it's fixed: | The agent's log | Why | | - | - | | The hub at [https://lumovi.example.com](https://lumovi.example.com) refused this agent: it doesn't know this name and token. | Lumovi has no agent of that name, or its token's SHA-256 isn't that one. Lumovi's log says `An agent was refused: "private-eu", with a token that doesn't match`. | | The hub at [https://lumovi.example.com](https://lumovi.example.com) refused this agent: another agent connected as it (do two clusters use the same name and token?). | A second agent connected with the same name and token, and took over. | While it's away, Lumovi's log says `The agent of private-eu disconnected`, and the cluster's card says `Its agent isn't connected.` An agent that's connected but can't reach its own API server stays connected, and the cluster's card says **Unreachable**, as for any cluster that doesn't answer. Other problems reaching Lumovi are in the agent's log, like `Can't reach the hub at https://lumovi.example.com: …`, then `Connecting again in 4 s: it couldn't connect`. ## In front of Lumovi Agents connect to `api/agent` below Lumovi's base path, with a WebSocket, and keep it open. Whatever is in front of Lumovi (an ingress, a load balancer, a CDN) must pass WebSockets there, as it does for pages at `api/socket`, and keep idle connections open longer than the heartbeat. Run one replica of Lumovi: an agent connects to one of them. * **Behind an [authenticating proxy](/server/auth/proxy)**, let `api/agent` through to Lumovi without signing in (with oauth2-proxy, `--skip-auth-route`). Agents can't sign in to a proxy, and Lumovi checks their names and tokens itself. A proxy that answers `401` makes an agent stop, saying Lumovi doesn't know its name and token. * **With the chart's `networkPolicy`**, its `from` must let in whatever agents come through, like your ingress controller or proxy. * **A server of one cluster** refuses agents, with `403`: only a fleet takes them. ## Security * **Nothing in the cluster listens.** The agent only dials out, to Lumovi, and has no Service. * **The agent can't read what it relays.** Lumovi checks the API server's certificate itself, end to end, against the certificate authority the agent sent. * **Its service account may only impersonate**, like a member's, and Lumovi uses its token only while the agent is connected. * **Whoever has an agent's token can connect as its cluster.** As the agent says which certificate authority to trust, they could stand in for the cluster with a server of their own, and see what people send it: their own tokens too, with `forwardToken`. Keep the token in a Secret in that cluster that only its administrators can read, and give Lumovi only its SHA-256 (`tokenSha256`), so Lumovi's settings don't hold it. * **A takeover shows.** When another agent connects with the same name and token, the first one stops, and its log says another agent connected as it. Unless two clusters were given the same name and token by mistake, that's a sign the token is somewhere it shouldn't be: make a new one. The fleet page, and the clusters Lumovi reaches itself. Every fleet setting. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.