Skip to main content
When a cluster’s API server is in a network Lumovi can’t reach (a private network, behind NAT, in a data center of your own), run Lumovi’s agent in it. The agent dials Lumovi over HTTPS, and Lumovi reaches the API server through that connection. Nothing in the cluster listens for Lumovi, and no firewall has to let anything in. Lumovi’s admins connect one from the Fleet page, with one command. Or set it up yourself, in Lumovi’s settings.

How it works

  • The agent connects to Lumovi’s api/agent with a WebSocket, naming its cluster and sending its token. Lumovi checks them against the agents it knows.
  • It tells Lumovi how to reach its cluster: the API server’s certificate authority, and its service account’s token. That service account may only impersonate users and groups.
  • Lumovi’s connections to the API server run through it. The agent connects each one to the API server, at the address Kubernetes gives its pod. TLS runs inside, from Lumovi to the API server: Lumovi checks the API server’s certificate against the cluster’s certificate authority, as kubernetes.default.svc, and the agent only passes on bytes it can’t read.
  • Lumovi acts as whoever signed in, impersonating them with the service account’s token, so the cluster applies their RBAC, as with any member.
  • Kubernetes rotates the token, and the agent reads it again every minute. When it changed, the agent sends Lumovi the new one.

Connect one from the Fleet page

Lumovi’s admins connect a private cluster from the Fleet page: Add cluster, then Connect with an agent…. Nobody pastes the cluster’s credentials into Lumovi, and nothing changes in Lumovi’s settings. It needs Lumovi to have admins, and to keep its state across restarts (auth.keepSessions, or LUMOVI_DATA_DIR): otherwise the page says why it can’t.
1

Name it

Its Name, which the fleet calls it, and its agent joins as; its Labels; and its Groups, who sees it besides admins (with none, everyone signed in). A name the fleet has already is refused.
2

Run the command, in the cluster

The page shows the cluster’s join token, once, and a command to run where you reach the cluster, with your own access to it:
It’s for bash or zsh. It waits for the token: paste it, and press ↵. The token isn’t shown, nor kept in your shell’s history or after the command, and Helm reads it from its input, not its command line. It’s one command, so the shell reads it all before it asks: pasting it can’t answer its own question.The token works once, for an hour. Lost it, or past its hour? Create a new command: the one before stops working. Until its agent connects, the cluster waits on the Fleet page as a card, with Cancel it.
3

Wait for it

The page says Waiting for edge-ap-south to connect…, with the time left. The agent dials Lumovi with its join token, and Lumovi gives it a token of its own for it, which the agent keeps in a Secret the chart makes for it, in its cluster (lumovi-agent, with this command). The agent may change that Secret alone. From then on, it connects with its own token: the join token, as the page showed it, and as Helm keeps it in its history, no longer works for anything.
4

Check its certificate authority

Once it’s connected, the page asks you to check the certificate authority it sent: run the command it shows, with your own access to the cluster, not through Lumovi, and paste what it prints. See Trusting it, by its SHA-256.
The first agent to use the join token is the cluster Lumovi trusts, with the certificate authority it sends. So only Lumovi’s admins see the cluster until one of them checks its certificate authority: an agent that had the token before yours could stand in for the cluster, and nobody else would use it. Keep the token to whoever installs the agent.
Making a command, cancelling one, the agent joining, and a token tried after it stopped working, are each in the audit log.

Remove it

A cluster connected from the page has Remove from the fleet… in its settings: type its name to confirm. People stop seeing it, and its agent’s token stops working: its connection closes at once, and on Lumovi’s other replicas within 10 seconds. What was trusted of its certificate authority goes too. The agent keeps running in its cluster until you uninstall it there:
An agent’s token doesn’t expire. To give it a new one, remove the cluster, and connect it again.

Set it up

To list an agent in Lumovi’s settings yourself:
1

Make a token for the agent

The agent proves which cluster it is with a token. Lumovi only needs its SHA-256:
Hash the token itself, without a line break after it, as printf %s does.
2

Tell Lumovi about it

List the agents that may connect in LUMOVI_FLEET_AGENTS, as YAML, or that encoded in base64 on one line:
agents.yaml
caSha256 is the SHA-256 of the private cluster’s certificate authority, the one its service accounts’ ca.crt holds. With your kubeconfig pointing at that cluster:
Give its line as it is (sha256 Fingerprint=AB:CD:…), or just the hash. With it, Lumovi trusts no other certificate authority from that agent: see Its cluster’s certificate authority.With the Helm chart, put the list in a Secret, under agents.yaml, in Lumovi’s namespace, and name it in fleet.agentsSecret:
values.yaml
Lumovi reads the list when it starts: restart it after changing the list (kubectl rollout restart deployment/lumovi --namespace lumovi). Its pod doesn’t start until the Secret is there. Until the agent connects, the cluster’s card says Unreachable: Its agent isn't connected.
3

Install the agent in the cluster

With your kubeconfig pointing at the private cluster, store the token, and install the chart as an agent, with the name Lumovi knows it by:
agent.hubUrl is the address people open Lumovi at, with its base path if it has one, over https: the agent’s token, and its cluster’s, travel there. An http one fails the install, but on the same machine (localhost, 127.0.0.1 or [::1]): agent.hubUrl must be https, not http://lumovi.example.com: the agent's token and its cluster's would cross the network as they are (http only reaches a hub on the same machine). clusterName must be the agent’s name in Lumovi’s list: unless set, it’s in-cluster.The chart’s notes, after installing, print a command that gives the SHA-256 of the token in agent.tokenSecret, and the entry for Lumovi’s list, if you’d rather make the token here first.
4

See it connect

The agent’s log says when it’s connected:
Lumovi’s log says The agent of private-eu connected (Lumovi 1.23.2), and the cluster is on the fleet page. Without caSha256, it also says it trusts the certificate authority the agent first sent: see below.

The agents Lumovi knows

LUMOVI_FLEET_AGENTS (or the chart’s fleet.agentsSecret) is a list, each with:
string
required
The cluster’s name in the fleet, and the agent’s LUMOVI_AGENT_NAME (the chart’s clusterName). Two agents can’t have the same name.
string
The SHA-256 of the agent’s token, in hex: 64 characters. Lumovi’s settings then don’t hold the token itself.
string
The agent’s token itself, at least 32 characters, instead of tokenSha256.
object
The cluster’s labels, like { env: production }. Unset, Lumovi’s admins may label it on the Fleet page.
string[]
Only people in one of these groups see the cluster, besides Lumovi’s admins, who see every cluster: groups as your provider or proxy names them, without prefixes. Everyone signed in does, without it, and an empty list hides it from everyone but admins. Unset, admins may set who sees it on the Fleet page. Lumovi is the only way into a private cluster, so every admin can reach it: see Admins see every cluster.
string | string[]
The SHA-256 of the cluster’s certificate authority, as openssl x509 -noout -fingerprint -sha256 prints it (sha256 Fingerprint=AB:CD:…), or just the hash, with or without its colons, in either case. While its certificate authority is rotated, a list of the old and the new: [<old>, <new>]. Lumovi then trusts no other certificate authority the agent sends. One that isn’t a SHA-256 stops Lumovi: LUMOVI_FLEET_AGENTS[0].caSha256 (private-eu) must be the SHA-256 of its cluster’s certificate authority, as openssl x509 -noout -fingerprint -sha256 prints it, or a list of them. Without it, Lumovi trusts the one the agent first sends: see Its cluster’s certificate authority.
boolean
default:"false"
true: requests carry each person’s own token, for an API server that trusts your provider, instead of impersonating them. Lumovi must have their tokens: set LUMOVI_OIDC_FORWARD_TOKEN.
An agent’s cluster takes the server’s prefixes for the names it impersonates. A list that isn’t like this stops Lumovi, saying what’s wrong, like LUMOVI_FLEET_AGENTS[0] (private-eu) needs a token of at least 32 characters, or its tokenSha256.

Its cluster’s certificate authority

The agent says which certificate authority its cluster’s API server is signed by, and Lumovi checks the API server’s certificate against it, end to end. So Lumovi trusts only one, for each agent:
  • The one caSha256 names. Of what the agent sends, only certificates with that SHA-256 become its cluster’s certificate authority; anything else it sends is ignored.
  • Without caSha256, the one the agent first sends. Lumovi keeps it with what it keeps across restarts (the chart’s auth.keepSessions, or LUMOVI_DATA_DIR; otherwise only until it restarts), and records it in the audit log, as agent.pinned: “Trusted the certificate authority the agent of private-eu first sent”. Its log says Trusting the certificate authority the agent of private-eu first sent (SHA-256 4a2b9c0d1e2f3a4b…).
Give caSha256 anyway: without it, whoever connects first with the agent’s token is the one trusted. Lumovi says so as it starts, naming each agent without one:
Where nothing keeps what it trusts (neither LUMOVI_DATA_DIR nor the chart’s auth.keepSessions), it adds , and, as nothing keeps what it trusts (no LUMOVI_DATA_DIR, nor the chart’s auth.keepSessions), again after every restart. The chart’s notes, after installing a fleet with fleet.agentsSecret, say the same; an agent’s notes give the command for its caSha256. The agent’s health says whether Lumovi uses it: /healthz answers 200 connected only once Lumovi trusts its cluster. Before then it answers 503 connected, not yet trusted, and when refused, 503 connected, but refused: and why, so its pod isn’t ready. Trusted, its log says The hub trusts this cluster.

When it sends another

An agent that sends another certificate authority stays connected, but its cluster isn’t used, and its pod isn’t ready. Lumovi refuses it, once for each one it sends, in the audit log as agent.refused (refused), in its log (The agent of private-eu:, and why), and in the agent’s (The hub doesn’t trust this cluster:, and why). The cluster’s card says:

Trusting it, by its SHA-256

Lumovi’s admins, or anyone signed in where it has none, see a banner on the fleet page for each connected agent without caSha256 that needs them:
  • One refused: “private-eu’s agent sends a certificate authority other than the one it’s trusted with.”, with Trust it….
  • One trusted as it first sent it: “private-eu’s agent is trusted with the certificate authority it first sent, which nobody has checked.”, with Check it….
Either opens the same form. It shows the start of what the agent sends, and of what it’s trusted with (“It sends 76cd6766…”), never the whole hash, so it can’t be copied from the page: the whole ones are in Lumovi’s log and the audit log. It asks for the SHA-256 taken from the cluster itself: “Its SHA-256, from the cluster itself, not from what its agent says. Run this with your own access to that cluster, not through Lumovi (which reaches it through this agent):”
Paste its line as it is (sha256 Fingerprint=AB:CD:…), or just the hash, with or without colons. Trust it trusts the agent with that certificate authority only if it’s one the agent sends now, and Lumovi refuses anything else, so whoever has the agent’s token can’t be trusted on its own say:
  • A hash the agent doesn’t send: “private-eu’s agent doesn’t send a certificate authority with that SHA-256: it sends 76cd6766….”, the start of each it sends.
  • Not a SHA-256: “Give the SHA-256 of its cluster’s certificate authority, as openssl x509 -noout -fingerprint -sha256 prints it.”
  • An agent that isn’t connected: “private-eu’s agent isn’t connected: it can be trusted once it is, and says what it sends.”
Trusted, it says “Trusted private-eu’s agent”, and only that one certificate is kept, as checked, recorded as agent.trusted: “Trusted the agent of private-eu with the certificate authority whose SHA-256 was given”, with what it was trusted with before and now. Lumovi’s log says The agent of private-eu is trusted with the certificate authority …, as an admin said. Where Lumovi has admins, others don’t see the banner, and Lumovi refuses them: “Only Lumovi’s admins trust an agent.” Where Lumovi has no admins (access.admins), anyone signed in sees the banner, and can check or trust an agent, the same way: with the SHA-256 from the cluster itself, recorded under their name. Giving each agent its caSha256 in LUMOVI_FLEET_AGENTS is still the surest: then Lumovi never trusts what an agent first sends. To rotate a cluster’s certificate authority without a gap, list both in caSha256, the old and the new, until the old one is gone.

What the chart makes

With mode: agent, the chart makes no Lumovi in the cluster, only its agent:
  • A Deployment, lumovi-agent for a release called lumovi, that runs the agent from Lumovi’s image. It has one replica, whatever replicaCount says, and is replaced with the Recreate strategy: a second agent of the same cluster would take over from the first.
  • A service account, lumovi, with a ClusterRole and a ClusterRoleBinding, lumovi-impersonate, that let it impersonate users and groups, and nothing else (unless rbac.create is false). Its token is mounted in the agent’s pod.
  • Health checks on port 8081. /healthz answers 200 connected while the agent is connected to Lumovi, and Lumovi trusts its cluster; otherwise 503, with not connected, connected, not yet trusted, or connected, but refused: and why. So the pod is ready only while Lumovi uses it. It’s alive as long as the port answers.
There’s no Service: nothing connects to the agent. It requests 10m of CPU and 32Mi of memory, and is limited to 128Mi (agent.resources). The chart’s image, extraEnv, podSecurityContext, securityContext, openshift, proxy (and proxy.secret), extraCA, nodeSelector, tolerations, affinity and priorityClassName apply to it too. See Helm values.

Run it another way

The agent is in Lumovi’s image, as /app/out/agent/agent.js. It runs in the cluster it relays to: it connects to the API server at KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT (443 unless set), which Kubernetes sets in every pod, and needs its pod’s service account token and CA certificate. Give its service account permission to impersonate users and groups, and nothing more, as a member’s has.
string
required
Lumovi’s address, with its base path if it has one: an https URL. The agent connects to api/agent below it, with a wss WebSocket: its token, and its cluster’s, travel in the connection. http is taken only on this computer (localhost, 127.0.0.1 or [::1]), with ws.
string
required
The cluster’s name, as Lumovi’s list of agents has it.
string
required
The agent’s token. Spaces and line breaks around it are left out, so hash it without them.
number
default:"8081"
Where GET /healthz says whether the agent is connected, and its cluster trusted. 0: nowhere.
number
default:"60"
How often the agent reads its service account’s token again, in seconds.
string
default:"/var/run/secrets/kubernetes.io/serviceaccount"
Where the pod’s service account token and CA certificate are.
string
The proxy the agent reaches Lumovi through, if it needs one, as Lumovi reads them: see Proxies and certificate authorities. Its own API server is always reached directly. The chart sets them from proxy.
string
Files of certificate authorities to trust besides Node’s and the system’s, separated by commas: the one a proxy that inspects HTTPS signs with, say. One that can’t be read stops the agent, saying which, as does a proxy that isn’t a URL. The chart sets it from extraCA.
Without one of the required settings (KUBERNETES_SERVICE_HOST among them), or without the service account’s token, the agent stops at once and says why:
When Kubernetes stops it, it closes its connection, and Lumovi’s log says The agent of private-eu disconnected.

Connecting, and staying connected

  • When it can’t reach Lumovi, or its connection closes, the agent tries again after 1 second, then 2, 4, 8 and 15, and every 30 seconds after that. Once connected, it starts again from 1.
  • When Lumovi restarts, for an upgrade say, it closes its agents’ connections, and they connect to the new one.
  • Lumovi pings its agents as often as it checks pages’ connections (LUMOVI_HEARTBEAT_SECONDS, every 30 seconds), and closes a connection that doesn’t answer.
  • The agent listens for those pings, as Lumovi tells it how often they come. A connection dropped somewhere on the way, without a word, goes quiet: after two heartbeats and a second without one (61 seconds), the agent says Nothing from the hub in 61 s, closes it, and connects again.
  • When Lumovi refuses it, the agent stops, as trying again wouldn’t help, and Kubernetes restarts it until it’s fixed:
While it’s away, Lumovi’s log says The agent of private-eu disconnected, and the cluster’s card says Its agent isn't connected. An agent that’s connected but can’t reach its own API server stays connected, and the cluster’s card says Unreachable, as for any cluster that doesn’t answer. Other problems reaching Lumovi are in the agent’s log, like Can't reach the hub at https://lumovi.example.com: …, then Connecting again in 4 s: it couldn't connect. Through a proxy, the agent’s log says which it uses as it starts, without its credentials. A proxy that refuses the connection is named, with where it was asked to go: The proxy http://proxy.corp.example.com:3128 needs credentials to lumovi.example.com:443 (407): give them in its URL, as http://user:password@host:port., or didn't open a tunnel with what it answered.

In front of Lumovi

Agents connect to api/agent below Lumovi’s base path, with a WebSocket, and keep it open. Whatever is in front of Lumovi (an ingress, a load balancer, a CDN) must pass WebSockets there, as it does for pages at api/socket, and keep idle connections open longer than the heartbeat. Run one replica of Lumovi: an agent connects to one of them.
  • Behind an authenticating proxy, let api/agent through to Lumovi without signing in (with oauth2-proxy, --skip-auth-route). Agents can’t sign in to a proxy, and Lumovi checks their names and tokens itself. A proxy that answers 401 makes an agent stop, saying Lumovi doesn’t know its name and token.
  • With the chart’s networkPolicy, its from must let in whatever agents come through, like your ingress controller. Behind an authenticating proxy, only the proxy’s pods may reach Lumovi, so agents come through the proxy.
  • A server of one cluster refuses agents, with 403: only a fleet takes them.

Security

  • Nothing in the cluster listens. The agent only dials out, to Lumovi, and has no Service.
  • The agent can’t read what it relays. Lumovi checks the API server’s certificate itself, end to end, against the certificate authority it trusts for that cluster: the one caSha256 names, or the one the agent first sent.
  • Its service account may only impersonate, like a member’s, and Lumovi uses its token only while the agent is connected.
  • Whoever has an agent’s token can connect as its agent. With caSha256, or once Lumovi trusts the certificate authority the agent first sent, that’s not enough to stand in for its cluster: a server of their own needs a certificate from that authority. Before then, at first contact, whoever connects first with the token decides which authority Lumovi trusts, and could stand in for the cluster and see what people send it: their own tokens too, with forwardToken. Give caSha256, so first contact decides nothing, or check what was first trusted, with Check it…. An admin trusts a new one only with the SHA-256 they give, from the cluster itself. Keep the token in a Secret in that cluster that only its administrators can read, and give Lumovi only its SHA-256 (tokenSha256), so Lumovi’s settings don’t hold it.
  • A takeover shows. When another agent connects with the same name and token, the first one stops, and its log says another agent connected as it. Unless two clusters were given the same name and token by mistake, that’s a sign the token is somewhere it shouldn’t be: make a new one.

A fleet of clusters

The fleet page, and the clusters Lumovi reaches itself.

Server configuration

Every fleet setting.