How it works
- The agent connects to Lumovi’s
api/agentwith a WebSocket, naming its cluster and sending its token. Lumovi checks them against the agents it knows. - It tells Lumovi how to reach its cluster: the API server’s certificate authority, and its service account’s token. That service account may only impersonate users and groups.
- Lumovi’s connections to the API server run through it. The agent connects each one to the API server, at the address Kubernetes gives its pod. TLS runs inside, from Lumovi to the API server: Lumovi checks the API server’s certificate against the cluster’s certificate authority, as
kubernetes.default.svc, and the agent only passes on bytes it can’t read. - Lumovi acts as whoever signed in, impersonating them with the service account’s token, so the cluster applies their RBAC, as with any member.
- Kubernetes rotates the token, and the agent reads it again every minute. When it changed, the agent sends Lumovi the new one.
Connect one from the Fleet page
Lumovi’s admins connect a private cluster from the Fleet page: Add cluster, then Connect with an agent…. Nobody pastes the cluster’s credentials into Lumovi, and nothing changes in Lumovi’s settings. It needs Lumovi to have admins, and to keep its state across restarts (auth.keepSessions, or LUMOVI_DATA_DIR): otherwise the page says why it can’t.
1
Name it
Its Name, which the fleet calls it, and its agent joins as; its Labels; and its Groups, who sees it besides admins (with none, everyone signed in). A name the fleet has already is refused.
2
Run the command, in the cluster
The page shows the cluster’s join token, once, and a command to run where you reach the cluster, with your own access to it:It’s for bash or zsh. It waits for the token: paste it, and press ↵. The token isn’t shown, nor kept in your shell’s history or after the command, and Helm reads it from its input, not its command line. It’s one command, so the shell reads it all before it asks: pasting it can’t answer its own question.The token works once, for an hour. Lost it, or past its hour? Create a new command: the one before stops working. Until its agent connects, the cluster waits on the Fleet page as a card, with Cancel it.
3
Wait for it
The page says Waiting for edge-ap-south to connect…, with the time left. The agent dials Lumovi with its join token, and Lumovi gives it a token of its own for it, which the agent keeps in a Secret the chart makes for it, in its cluster (
lumovi-agent, with this command). The agent may change that Secret alone. From then on, it connects with its own token: the join token, as the page showed it, and as Helm keeps it in its history, no longer works for anything.4
Check its certificate authority
Once it’s connected, the page asks you to check the certificate authority it sent: run the command it shows, with your own access to the cluster, not through Lumovi, and paste what it prints. See Trusting it, by its SHA-256.
Remove it
A cluster connected from the page has Remove from the fleet… in its settings: type its name to confirm. People stop seeing it, and its agent’s token stops working: its connection closes at once, and on Lumovi’s other replicas within 10 seconds. What was trusted of its certificate authority goes too. The agent keeps running in its cluster until you uninstall it there:Set it up
To list an agent in Lumovi’s settings yourself:1
Make a token for the agent
The agent proves which cluster it is with a token. Lumovi only needs its SHA-256:Hash the token itself, without a line break after it, as
printf %s does.2
Tell Lumovi about it
List the agents that may connect in Give its line as it is (Lumovi reads the list when it starts: restart it after changing the list (
LUMOVI_FLEET_AGENTS, as YAML, or that encoded in base64 on one line:agents.yaml
caSha256 is the SHA-256 of the private cluster’s certificate authority, the one its service accounts’ ca.crt holds. With your kubeconfig pointing at that cluster:sha256 Fingerprint=AB:CD:…), or just the hash. With it, Lumovi trusts no other certificate authority from that agent: see Its cluster’s certificate authority.With the Helm chart, put the list in a Secret, under agents.yaml, in Lumovi’s namespace, and name it in fleet.agentsSecret:values.yaml
kubectl rollout restart deployment/lumovi --namespace lumovi). Its pod doesn’t start until the Secret is there. Until the agent connects, the cluster’s card says Unreachable: Its agent isn't connected.3
Install the agent in the cluster
With your kubeconfig pointing at the private cluster, store the token, and install the chart as an agent, with the name Lumovi knows it by:
agent.hubUrl is the address people open Lumovi at, with its base path if it has one, over https: the agent’s token, and its cluster’s, travel there. An http one fails the install, but on the same machine (localhost, 127.0.0.1 or [::1]): agent.hubUrl must be https, not http://lumovi.example.com: the agent's token and its cluster's would cross the network as they are (http only reaches a hub on the same machine). clusterName must be the agent’s name in Lumovi’s list: unless set, it’s in-cluster.The chart’s notes, after installing, print a command that gives the SHA-256 of the token in agent.tokenSecret, and the entry for Lumovi’s list, if you’d rather make the token here first.4
See it connect
The agent’s log says when it’s connected:Lumovi’s log says
The agent of private-eu connected (Lumovi 1.23.2), and the cluster is on the fleet page. Without caSha256, it also says it trusts the certificate authority the agent first sent: see below.The agents Lumovi knows
LUMOVI_FLEET_AGENTS (or the chart’s fleet.agentsSecret) is a list, each with:
The cluster’s name in the fleet, and the agent’s
LUMOVI_AGENT_NAME (the chart’s clusterName). Two agents can’t have the same name.The SHA-256 of the agent’s token, in hex: 64 characters. Lumovi’s settings then don’t hold the token itself.
The cluster’s labels, like
{ env: production }. Unset, Lumovi’s admins may label it on the Fleet page.Only people in one of these groups see the cluster, besides Lumovi’s admins, who see every cluster: groups as your provider or proxy names them, without prefixes. Everyone signed in does, without it, and an empty list hides it from everyone but admins. Unset, admins may set who sees it on the Fleet page. Lumovi is the only way into a private cluster, so every admin can reach it: see Admins see every cluster.
The SHA-256 of the cluster’s certificate authority, as
openssl x509 -noout -fingerprint -sha256 prints it (sha256 Fingerprint=AB:CD:…), or just the hash, with or without its colons, in either case. While its certificate authority is rotated, a list of the old and the new: [<old>, <new>]. Lumovi then trusts no other certificate authority the agent sends. One that isn’t a SHA-256 stops Lumovi: LUMOVI_FLEET_AGENTS[0].caSha256 (private-eu) must be the SHA-256 of its cluster’s certificate authority, as openssl x509 -noout -fingerprint -sha256 prints it, or a list of them. Without it, Lumovi trusts the one the agent first sends: see Its cluster’s certificate authority.true: requests carry each person’s own token, for an API server that trusts your provider, instead of impersonating them. Lumovi must have their tokens: set LUMOVI_OIDC_FORWARD_TOKEN.LUMOVI_FLEET_AGENTS[0] (private-eu) needs a token of at least 32 characters, or its tokenSha256.
Its cluster’s certificate authority
The agent says which certificate authority its cluster’s API server is signed by, and Lumovi checks the API server’s certificate against it, end to end. So Lumovi trusts only one, for each agent:- The one
caSha256names. Of what the agent sends, only certificates with that SHA-256 become its cluster’s certificate authority; anything else it sends is ignored. - Without
caSha256, the one the agent first sends. Lumovi keeps it with what it keeps across restarts (the chart’sauth.keepSessions, orLUMOVI_DATA_DIR; otherwise only until it restarts), and records it in the audit log, asagent.pinned: “Trusted the certificate authority the agent of private-eu first sent”. Its log saysTrusting the certificate authority the agent of private-eu first sent (SHA-256 4a2b9c0d1e2f3a4b…).
caSha256 anyway: without it, whoever connects first with the agent’s token is the one trusted. Lumovi says so as it starts, naming each agent without one:
LUMOVI_DATA_DIR nor the chart’s auth.keepSessions), it adds , and, as nothing keeps what it trusts (no LUMOVI_DATA_DIR, nor the chart’s auth.keepSessions), again after every restart. The chart’s notes, after installing a fleet with fleet.agentsSecret, say the same; an agent’s notes give the command for its caSha256.
The agent’s health says whether Lumovi uses it: /healthz answers 200 connected only once Lumovi trusts its cluster. Before then it answers 503 connected, not yet trusted, and when refused, 503 connected, but refused: and why, so its pod isn’t ready. Trusted, its log says The hub trusts this cluster.
When it sends another
An agent that sends another certificate authority stays connected, but its cluster isn’t used, and its pod isn’t ready. Lumovi refuses it, once for each one it sends, in the audit log asagent.refused (refused), in its log (The agent of private-eu:, and why), and in the agent’s (The hub doesn’t trust this cluster:, and why). The cluster’s card says:
Trusting it, by its SHA-256
Lumovi’s admins, or anyone signed in where it has none, see a banner on the fleet page for each connected agent withoutcaSha256 that needs them:
- One refused: “private-eu’s agent sends a certificate authority other than the one it’s trusted with.”, with Trust it….
- One trusted as it first sent it: “private-eu’s agent is trusted with the certificate authority it first sent, which nobody has checked.”, with Check it….
sha256 Fingerprint=AB:CD:…), or just the hash, with or without colons. Trust it trusts the agent with that certificate authority only if it’s one the agent sends now, and Lumovi refuses anything else, so whoever has the agent’s token can’t be trusted on its own say:
- A hash the agent doesn’t send: “private-eu’s agent doesn’t send a certificate authority with that SHA-256: it sends 76cd6766….”, the start of each it sends.
- Not a SHA-256: “Give the SHA-256 of its cluster’s certificate authority, as openssl x509 -noout -fingerprint -sha256 prints it.”
- An agent that isn’t connected: “private-eu’s agent isn’t connected: it can be trusted once it is, and says what it sends.”
agent.trusted: “Trusted the agent of private-eu with the certificate authority whose SHA-256 was given”, with what it was trusted with before and now. Lumovi’s log says The agent of private-eu is trusted with the certificate authority …, as an admin said. Where Lumovi has admins, others don’t see the banner, and Lumovi refuses them: “Only Lumovi’s admins trust an agent.”
Where Lumovi has no admins (access.admins), anyone signed in sees the banner, and can check or trust an agent, the same way: with the SHA-256 from the cluster itself, recorded under their name. Giving each agent its caSha256 in LUMOVI_FLEET_AGENTS is still the surest: then Lumovi never trusts what an agent first sends.
To rotate a cluster’s certificate authority without a gap, list both in caSha256, the old and the new, until the old one is gone.
What the chart makes
Withmode: agent, the chart makes no Lumovi in the cluster, only its agent:
- A Deployment,
lumovi-agentfor a release calledlumovi, that runs the agent from Lumovi’s image. It has one replica, whateverreplicaCountsays, and is replaced with theRecreatestrategy: a second agent of the same cluster would take over from the first. - A service account,
lumovi, with a ClusterRole and a ClusterRoleBinding,lumovi-impersonate, that let it impersonate users and groups, and nothing else (unlessrbac.createisfalse). Its token is mounted in the agent’s pod. - Health checks on port 8081.
/healthzanswers200 connectedwhile the agent is connected to Lumovi, and Lumovi trusts its cluster; otherwise503, withnot connected,connected, not yet trusted, orconnected, but refused:and why. So the pod is ready only while Lumovi uses it. It’s alive as long as the port answers.
10m of CPU and 32Mi of memory, and is limited to 128Mi (agent.resources). The chart’s image, extraEnv, podSecurityContext, securityContext, openshift, proxy (and proxy.secret), extraCA, nodeSelector, tolerations, affinity and priorityClassName apply to it too. See Helm values.
Run it another way
The agent is in Lumovi’s image, as/app/out/agent/agent.js. It runs in the cluster it relays to: it connects to the API server at KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT (443 unless set), which Kubernetes sets in every pod, and needs its pod’s service account token and CA certificate. Give its service account permission to impersonate users and groups, and nothing more, as a member’s has.
Lumovi’s address, with its base path if it has one: an
https URL. The agent connects to api/agent below it, with a wss WebSocket: its token, and its cluster’s, travel in the connection. http is taken only on this computer (localhost, 127.0.0.1 or [::1]), with ws.The agent’s token. Spaces and line breaks around it are left out, so hash it without them.
Where
GET /healthz says whether the agent is connected, and its cluster trusted. 0: nowhere.Where the pod’s service account token and CA certificate are.
The proxy the agent reaches Lumovi through, if it needs one, as Lumovi reads them: see Proxies and certificate authorities. Its own API server is always reached directly. The chart sets them from
proxy.Files of certificate authorities to trust besides Node’s and the system’s, separated by commas: the one a proxy that inspects HTTPS signs with, say. One that can’t be read stops the agent, saying which, as does a proxy that isn’t a URL. The chart sets it from
extraCA.KUBERNETES_SERVICE_HOST among them), or without the service account’s token, the agent stops at once and says why:
The agent of private-eu disconnected.
Connecting, and staying connected
- When it can’t reach Lumovi, or its connection closes, the agent tries again after 1 second, then 2, 4, 8 and 15, and every 30 seconds after that. Once connected, it starts again from 1.
- When Lumovi restarts, for an upgrade say, it closes its agents’ connections, and they connect to the new one.
- Lumovi pings its agents as often as it checks pages’ connections (
LUMOVI_HEARTBEAT_SECONDS, every 30 seconds), and closes a connection that doesn’t answer. - The agent listens for those pings, as Lumovi tells it how often they come. A connection dropped somewhere on the way, without a word, goes quiet: after two heartbeats and a second without one (61 seconds), the agent says
Nothing from the hub in 61 s, closes it, and connects again. - When Lumovi refuses it, the agent stops, as trying again wouldn’t help, and Kubernetes restarts it until it’s fixed:
While it’s away, Lumovi’s log says
The agent of private-eu disconnected, and the cluster’s card says Its agent isn't connected. An agent that’s connected but can’t reach its own API server stays connected, and the cluster’s card says Unreachable, as for any cluster that doesn’t answer.
Other problems reaching Lumovi are in the agent’s log, like Can't reach the hub at https://lumovi.example.com: …, then Connecting again in 4 s: it couldn't connect. Through a proxy, the agent’s log says which it uses as it starts, without its credentials. A proxy that refuses the connection is named, with where it was asked to go: The proxy http://proxy.corp.example.com:3128 needs credentials to lumovi.example.com:443 (407): give them in its URL, as http://user:password@host:port., or didn't open a tunnel with what it answered.
In front of Lumovi
Agents connect toapi/agent below Lumovi’s base path, with a WebSocket, and keep it open. Whatever is in front of Lumovi (an ingress, a load balancer, a CDN) must pass WebSockets there, as it does for pages at api/socket, and keep idle connections open longer than the heartbeat. Run one replica of Lumovi: an agent connects to one of them.
- Behind an authenticating proxy, let
api/agentthrough to Lumovi without signing in (with oauth2-proxy,--skip-auth-route). Agents can’t sign in to a proxy, and Lumovi checks their names and tokens itself. A proxy that answers401makes an agent stop, saying Lumovi doesn’t know its name and token. - With the chart’s
networkPolicy, itsfrommust let in whatever agents come through, like your ingress controller. Behind an authenticating proxy, only the proxy’s pods may reach Lumovi, so agents come through the proxy. - A server of one cluster refuses agents, with
403: only a fleet takes them.
Security
- Nothing in the cluster listens. The agent only dials out, to Lumovi, and has no Service.
- The agent can’t read what it relays. Lumovi checks the API server’s certificate itself, end to end, against the certificate authority it trusts for that cluster: the one
caSha256names, or the one the agent first sent. - Its service account may only impersonate, like a member’s, and Lumovi uses its token only while the agent is connected.
- Whoever has an agent’s token can connect as its agent. With
caSha256, or once Lumovi trusts the certificate authority the agent first sent, that’s not enough to stand in for its cluster: a server of their own needs a certificate from that authority. Before then, at first contact, whoever connects first with the token decides which authority Lumovi trusts, and could stand in for the cluster and see what people send it: their own tokens too, withforwardToken. GivecaSha256, so first contact decides nothing, or check what was first trusted, with Check it…. An admin trusts a new one only with the SHA-256 they give, from the cluster itself. Keep the token in a Secret in that cluster that only its administrators can read, and give Lumovi only its SHA-256 (tokenSha256), so Lumovi’s settings don’t hold it. - A takeover shows. When another agent connects with the same name and token, the first one stops, and its log says another agent connected as it. Unless two clusters were given the same name and token by mistake, that’s a sign the token is somewhere it shouldn’t be: make a new one.
A fleet of clusters
The fleet page, and the clusters Lumovi reaches itself.
Server configuration
Every fleet setting.