XDmon Manual

XDmon is a self-hosted platform for fleets of Raspberry Pi CM5 devices. One server, the hub, gives every device a private network connection, an OS image with safe A/B updates, app deployment per fleet, monitoring, an event log and remote support from the browser.

This manual covers the whole product: installing the hub, bringing devices online, working in the dashboard, rolling out apps and updates, managing access, and the reference material administrators need on a bad day.

Who this manual is for

Key terms

How XDmon fits together

Devices dial out to the hub. Nothing on site needs an open port, a VPN client or a public address.

The hub stack and the three kinds of device it serves.

What runs where

• Outer Caddy (port 443) is the web server you already run on the VPS. It handles TLS, the OIDC login and the role policies, then forwards everything to XDmon on localhost:8080.
• xdmon-caddy routes /management/ to the dashboard and /<route key>/ to each device through the tunnel. Every device route asks the dashboard first whether the signed-in user may reach that device. A second site on 10.20.0.1:9080 serves update files, reachable over WireGuard only.
• xdmon-dashboard is the web app and API. It keeps fleet.json (devices, keys, fleets, assignments), hub statistics and the audit log, and it writes and reloads the WireGuard and Caddy configuration.
• xdmon-hub is the WireGuard endpoint. Devices cannot reach each other; only the hub.
• The XDmon Agent on each device keeps 28 days of telemetry locally. The hub stores only heartbeats and its own counters, and fetches history from the device when you open a chart.

The three update channels

Each layer on a device updates on its own, so a new app never needs a new OS and an OS update never touches your data.

Device data in /data (WireGuard keys, app data, settings, logs) survives every update and a reflash of the OS slots.

Quick start

From an empty server to the first device on the dashboard. Each step links to the full instructions.

1. Prepare the server. A Debian or Ubuntu VPS with Docker, UDP 51820 and TCP 443 open, and a DNS name such as fleet.example.com. Installing the hub
2. Deploy and initialise the hub. Push the images, run init, start the stack. Steps 2 to 5
3. Connect your login. Paste the generated snippet into your outer Caddy and set up the OIDC roles. Outer Caddy and login
4. Publish the agent and an OS bundle. ./deploy-agent.sh v2.73 and ./deploy-os.sh from the build Mac. Agent, image
5. Flash and add a device. Flash a CM5 with the XDmon image, add it in the dashboard and pick its fleet. Adding devices
6. Pair it. Paste the config on the device's pairing page, or plug in a USB stick. Within a minute the device shows a green dot. Provisioning methods
7. Give the fleet its apps. Publish app packages and assign them to the fleet; every device in it installs them. Applications

Installing the hub

The hub is one Docker Compose stack. Images are built on a workstation and pushed to the server, so the server needs nothing but Docker.

Requirements

Step by step

1. Prepare the server

   ssh root@<server> mkdir -p ~/xdmon
   scp hub/setup-hub-docker.sh root@<server>:~/xdmon/
   ssh root@<server> bash ~/xdmon/setup-hub-docker.sh

   The script installs Docker if needed, enables IP forwarding permanently and creates an empty .env next to itself.

2. Build and push the hub images from the workstation. Set VPS_HOST and VPS_DIR at the top of the deploy scripts first.

   ./deploy-hub.sh                 # both images; or: ./deploy-hub.sh dashboard | wireguard

   This builds xdmon:latest and xdmon-wireguard:latest for linux/amd64, loads them on the server, copies docker-compose.yml and xdmon.sh, and starts the stack.

3. Initialise the fleet on the server. The first two commands use --no-deps because the WireGuard container waits for a config that does not exist yet.

   cd ~/xdmon
   docker compose run --rm --no-deps dashboard init \
     --hub-endpoint <server public IP> \
     --proxy-domain fleet.example.com \
     --caddy-port 8080
   docker compose run --rm --no-deps dashboard gen-hub-conf --out /data/wg/wg0.conf

   init creates the hub key pair, the subnet and the ports in data/fleet/fleet.json. Use --caddy-port 8080 when your own Caddy owns port 443, which is the standard setup.

4. Balena token (only with Balena devices). Put BALENA_API_TOKEN=… in ~/xdmon/.env. A fleet of XDmon devices only skips this.
5. Start the stack

   cd ~/xdmon && docker compose up -d

   The dashboard waits for a healthy WireGuard container, then listens on 127.0.0.1:5100 and 10.20.0.1:5100 only.

6. Add the CLI alias

   echo "alias xdmon='~/xdmon/xdmon.sh'" >> ~/.bashrc && source ~/.bashrc

7. Connect the outer Caddy as described in the next section, then open https://fleet.example.com/. It redirects to /management/.

Outer Caddy and login

XDmon does not edit your outer Caddyfile. It generates a snippet you paste once: ☰ Menu → Hub config & apply → Snippet (or xdmon gen-caddy-snippet). The snippet holds the site block for your domain with the forward to localhost:8080, a styled 403 page, a logout handler and an error page for when the stack is down.

Your caddy-security devices policy must pass the user's roles on to XDmon. Add the two inject header lines once:

authorization policy devices {
    set auth url /caddy-security/oauth2/<provider>
    set forbidden url /forbidden
    allow roles admin devices vnc-fullcontrol vnc-readonly
    inject headers with claims
    inject header "X-Auth-Groups" from "roles"
    inject header "X-Auth-User" from "sub"
}

The roles themselves and the VNC policies are described in Users and access. Afterwards open Hub config & apply → Caddy and click Apply Caddy to write the inner Caddyfile.

Files on the server

~/xdmon/
├── docker-compose.yml     pushed by deploy-hub.sh
├── .env                   secrets (Balena token), created by you
├── xdmon.sh               CLI wrapper
├── data/
│   ├── fleet/             fleet.json (device keys!), hub-stats.db, audit/, sessions/, ssh/
│   ├── wg/                wg0.conf
│   └── caddy/             Caddyfile
└── updates/               served on 10.20.0.1:9080 to devices only
    ├── agent/<version>/
    ├── os/xdmon-os-<version>.raucb
    └── apps/<name>/

Updating the hub

./deploy-hub.sh dashboard      # dashboard only, about 30 seconds
./deploy-hub.sh wireguard      # WireGuard container only
./deploy-hub.sh                # both

Devices keep their tunnels during a dashboard restart. The dashboard restores the last known device state from a snapshot on start, so the page is filled before the first new heartbeats arrive.

The dashboard

One page for everyone. What you see and may change depends on your roles; nothing changes until you switch to Configure.

Page layout

• Header: the hub pill (devices online), navigation to Hub health, Updates, Audit log and Tools, the View | Configure switch, your user chip and the ☰ Menu. Below 1280 px the navigation moves into the menu.
• Toolbar: search by name, device id, address or fleet (press / to focus), the filters All, Online, Offline and Attention with counts, collapse or expand all cards, and the age of the data.
• Fleet cards: one per fleet, sorted by name. Devices without a fleet are under Other devices. On wide screens each row also shows CPU, the agent version and the WireGuard address.
• Side panel: the Hub card with live tiles and a one-hour sparkline, Needs attention, and in Configure mode the Manage card.

Reading a device row

View and Configure

The page always opens in View. Everything there is read-only, so support staff can browse without risk. Administrators switch to Configure in the header or in a device window footer; the header turns orange and a bar with Done appears. The choice is remembered per browser tab.

The Manage card (and the same items in the menu) holds: Add device, Add router, Fleets, Balena fleets, Access groups, Analyzer files, Deploy all, Clean up old versions, Apply WireGuard, Apply Caddy and the hub config files. Chosen from View, these items switch to Configure first.

Needs attention

This list collects devices that are offline, miss heartbeats, have a WireGuard, internet or gateway link down, cannot reach a ping target, or run hot or busy. Click a name to open the device. The Attention filter shows the same devices in the main list.

On a phone

The dashboard works on a phone. The hub tiles become a scrolling row above the list, the navigation moves into the menu and the device window uses the full screen.

Your session is kept alive while the page is open. When it expires, the page sends you to the login and restores the window you had open.

Adding and provisioning devices

A device needs two things: an entry on the hub, and its WireGuard config on the device. After that the device fetches the XDmon Agent and its fleet's apps by itself.

1. Add the device on the hub

In Configure mode choose Manage → Add device. Enter a name, leave Balena UUID empty for an XDmon device, and pick the Fleet: the device then gets that fleet's apps and settings as soon as it connects.

The hub assigns the next free address in the overlay, for example 10.20.0.15. The device becomes reachable at https://fleet.example.com/1020015/.

From the CLI: xdmon add-device --name kiosk-utr-03

Adding a device from the dashboard applies the WireGuard and Caddy configuration for you when the hub has Docker access. After a CLI change or a direct edit of fleet.json, click Apply WireGuard and Apply Caddy.

2. Give the device its config

The XDmon image looks for a WireGuard config in this order: a USB stick, then /data/wireguard/wg0.conf on the persistent partition, then a config baked into the image. Pick the method that suits the site.

Pairing page (recommended)

1. On the device, start the pairing server: sudo systemctl start xdmon-pair. It only runs while the device has no config.
2. In the dashboard, open the device and click Provision (Settings tab, Configure mode).
3. Copy the config blob.
4. On a computer in the same network, open http://<device-ip>:8080, paste the config, check the device name and click Apply Configuration.

The device stores the config and its name in /data, brings the tunnel up and stops the pairing server. The Provision window also shows an SSH one-liner as a fallback.

USB stick

xdmon gen-usb --name kiosk-utr-03 --mount /Volumes/STICK
xdmon gen-usb --name kiosk-utr-03 --mount /Volumes/STICK --wifi-ssid "Shop WiFi"   # also WiFi; asks for the password

This writes xdmon/wg0.conf (and xdmon/wifi.conf) on the stick. Plug it into the device, running or at boot. The device imports the files, renames them to .imported so they do not trigger again, removes the WiFi password from the stick and writes the result to xdmon/wifi-result.txt. No reboot is needed.

SSH

tools/provision-ssh.sh kiosk-utr-03 192.168.1.50

Writes /data/wireguard/wg0.conf, sets the hostname and starts the tunnel. On the XDmon image /etc is read-only; always write the config to /data/wireguard/.

3. Check that it is online

The device appears with a green dot within a minute. On first connect it downloads the XDmon Agent from the hub; the Overview tab fills once the first heartbeat arrives. On the device itself:

sudo wg show wg0          # handshake time and transfer counters
ping -c 3 10.20.0.1       # the hub
systemctl status xdmon-agent

New keys, removing a device

• Reinstall (device window → Settings → Danger zone) creates a new key pair with the same address and opens the Provision window. On the device, move the old config aside and pair again: sudo mv /data/wireguard/wg0.conf /data/wireguard/wg0.conf.old && sudo systemctl start xdmon-pair.
• Revoke removes the device from the hub and applies WireGuard and Caddy. Its tunnel stops working immediately.
• Rename changes the display name. XDmon devices pick up the new hostname with their next heartbeat.

Routers

Manage → Add router adds a site router as a WireGuard peer with one or more LAN subnets behind it, so the hub can reach equipment on that LAN. Download its config from the Routers list in Configure mode.

The device window

Click any device to open its window. It has the same size on every tab; only the content scrolls.

Action buttons

Buttons that need the tunnel are dimmed while the device is offline and come back by themselves when it reconnects.

Overview

• Alerts at the top: a link or ping target down, throttling, a stale heartbeat, a recent restart.
• Now: CPU, memory, root and data disk, temperature and fan from the latest heartbeat. History opens the charts.
• Connection: WireGuard state, address, last heartbeat, uptime, and the last restart with how long the device was down before it. History ▾ lists the last ten reboots.
• Traffic & reachability: data and uptime as seen by the hub for 1 hour, 24 hours and 7 days, plus a 24-hour traffic bar chart. Available while the device is offline.
• Software: XDmon Agent, OS version and slot, last OS update, fleet.
• Technical details (collapsed): device id, public key, endpoint, WireGuard counters, date added.

Charts

One view switch and one period selector for all charts: live 10 minutes, live 1 hour, 6 hours, 24 hours, 2, 3 or 7 days. The history comes from the device itself, which keeps 28 days of samples; Availability comes from the hub and also works while the device is offline.

Gaps in the line mean the device was off. The x-axis always covers the full period and is anchored to the device's latest sample, so a device clock that is slightly off does not distort the chart.

Apps

For XDmon devices: every app with its version, state (running, failed, stopped), restarts and the stateful and kiosk tags, plus whether the device matches what the hub assigned. In Configure mode each row also shows whether the app comes from the fleet or from this device only, with editors for version, run on or off, settings and secrets. See Applications.

For Balena devices: the containers of the running release, read-only.

Settings

Everything that can be changed for one device, with a strip at the top that says whether the device has applied the latest change. In View mode the values are read-only and each row has a Change button that switches to Configure and opens the editor.

Remote support

See the screen, read the logs, open a shell and check the history, all from the browser and all through the hub. Nobody needs a VPN or a Balena account.

Remote display (VNC)

Display starts a VNC session on the device on demand and opens it in the browser. Choose view only or full control when starting; a session started view-only is enforced on the device for every viewer.

VNC is not running when nobody needs it. It stops by itself after 5 minutes without a viewer, and anyone with a VNC role can stop it earlier. Starting and every viewer joining or leaving are written to the audit log.

Who may do what is set by roles: vnc-fullcontrol may start either mode, vnc-readonly only view-only; admins can always. The fleet's VNC enabled setting decides whether the button appears for that fleet.

Console

The Console streams the device's system journal live. All filtering happens in the browser, so changing a filter never interrupts the stream.

• Apps, System or All; the text filter hides lines that do not match.
• Filters: minimum level (emergency to debug) and source chips. Tap a chip to hide it; double-tap, or tap a name in the log, to show only that source.
• Pause, clear the screen, hide timestamps. The stream reconnects by itself and keeps the last 5000 lines.

Terminal

Terminal opens an SSH session to an XDmon device in a new page. The hub makes the connection with its own key and logs in as the restricted user support; the browser never sees a key.

• Who: roles admin or terminal, plus access to that device.
• What support may do: read the journal, check status, and run a short list of commands without typing sudo: xdmon-app start|stop|restart|rollback, xdmon-os-update status, xdmon-update.sh, wg show, restart the kiosk, agent or tunnel, and reboot. Type xdmon-help for the list.
• Recorded: every session is saved as an asciicast and can be replayed by admins from the audit log. Recordings are kept 180 days.
• Limits: closed after 15 minutes without activity and after 4 hours.

Events

The Events window is the device's history, kept on the device for 90 days:

• Network: each outage per target (wireguard, internet, gateway, your ping targets) with its duration.
• System: reboots with downtime, time syncs, high temperature, undervoltage, disk and memory warnings, display lost.
• Updates: agent and OS version changes and the result of each OS update.
• Apps: installed, updated, removed, failed, rolled back, restarted.
• Access (admins): who reached the device, terminal sessions with replay, VNC viewers and changes.

Active problems are shown at the top. The window refreshes every 15 seconds.

Tools and log analysers

A log analyser is a standalone HTML page that reads a log file. Upload it under Tools → Analyzer files, then attach it to a fleet in Fleet services → Log Server with a name, a URL slug and a file pattern such as *.log.

XDmon adds a Files button to the analyser: pick a device and a log file and it is fetched from the device through the tunnel. From a device window, Analyzer opens the newest matching log of that device directly.

Device web pages and proxy endpoints

Each device is reachable at https://fleet.example.com/<route key>/, behind your login and the per-device access check. By default this shows port 80 on the device. With proxy endpoints you route paths to different local services, for example / to the app UI on 127.0.0.1:5000 and /api to 127.0.0.1:3000. Each endpoint gets its own button in the device window.

Endpoints are set per fleet in Fleet services → Proxy, or per device. The hub sends the configuration to the device with the heartbeat; if the device's Caddy refuses it, the previous configuration stays active and the error is shown in the device's Caddy preview and in Updates.

Fleets

A fleet is a group of XDmon devices that run the same apps with the same settings. Configure the fleet once; every device in it follows with its next heartbeat.

A fleet carries:

• Apps with their versions, settings and secrets.
• Device settings: ping targets and screen rotation.
• Services: proxy endpoints, log server and analysers, VNC.
• A scope for OS rollouts and a filter in Updates.

Managing fleets

• Create fleet: type a name, then tick its devices.
• Devices…: tick the devices that belong to the fleet. A device from another fleet shows moves from ….
• ⚙ Settings: opens Fleet services for that fleet.
• ✎ Rename: apps, settings, services and a running OS rollout move along. Renaming to an existing name is refused, so fleets are never merged by accident.
• ✕ Delete: only for an empty fleet. Its apps, settings and services are removed too.

A single device moves with the fleet picker in its Settings tab. Fleet names may contain letters, digits, spaces, dots, dashes and underscores, up to 48 characters, and cannot be only digits.

Fleet services

The ⚙ button on a fleet card opens the fleet's configuration with five tabs.

Applications

Your own programs (.NET, Go, Swift) and services such as MongoDB run as signed app packages next to the OS. The hub decides which apps a device runs; the agent makes the device match.

How an app runs on a device

• Installed in /data/apps/<name>/, running as xdmon-app@<name>.service under its own user.
• Sandboxed: a read-only OS, no view of WireGuard keys or other apps, a private /tmp. Only its own package (read-only) and its data/ folder (writable).
• Hardware access through groups such as dialout, gpio, i2c, video, optionally narrowed to specific devices.
• Restarted when it stops. After an install or upgrade it must pass a health check; if not, the previous version comes back automatically.
• data/ and the settings survive updates. Stateful apps such as databases get a backup of data/ before every upgrade.

The package

A folder with an app.json and the program files. Programs must be self-contained, because the OS cannot install packages.

{
  "name": "payment-bridge",
  "version": "1.9.0",
  "exec": "bin/PaymentBridge",
  "args": ["--port", "5080"],
  "env": { "ASPNETCORE_URLS": "http://127.0.0.1:5080" },
  "groups": ["dialout"],
  "devices": ["char-ttyUSB"],
  "requires": ["mongodb"],
  "memoryMax": "512M",
  "health": { "http": "http://127.0.0.1:5080/health", "timeoutSec": 60 },
  "kioskUrl": "http://127.0.0.1:5080/"
}

How to build self-contained programs:

Apps get the device identity in their environment: XDMON_DEVICE_NAME, XDMON_DEVICE_ADDRESS and XDMON_DEVICE_ROUTE_KEY. Settings may contain ${XDMON_DEVICE_*} placeholders, so one fleet value fits every device, for example BASE_URL=/${XDMON_DEVICE_ROUTE_KEY}/app/.

Package and publish

tools/package-app.sh ./out/payment-bridge
tools/publish-app.sh ~/xdmon-images/apps/xdmon-app-payment-bridge-1.9.0.tar.gz
tools/publish-app.sh --list

The package is reproducible and signed with the OS signing key. Devices refuse a package whose signature does not match. Published packages are served to devices over WireGuard only.

Assign apps

1. Fleet: fleet card ⚙ → Apps → Add app. Pick the package, version, run on or off, settings and secrets. Every device in the fleet installs it.
2. Per device, if needed: device window → Apps in Configure mode. Pin another version to try it on one device first, override a setting, switch an app off, skip a fleet app on this device, or add an app for this device only.
3. Watch it arrive: within about 30 seconds the device reports the new state. The Apps tab and the Updates overview show syncing, then in sync or the error per app.

The kiosk screen

The screen shows one app at a time. An app with kioskUrl or display: "native" claims the screen when it starts; the last one to claim it wins. The setting XDMON_KIOSK_URL overrides this per fleet or device (a local URL, or off). With no app on screen the device shows the XDmon splash.

On the device

xdmon-app status                         # all apps, state, last hub sync
sudo xdmon-app restart payment-bridge
sudo xdmon-app rollback payment-bridge   # previous version
sudo xdmon-app rollback mongodb --restore-data
journalctl -u xdmon-app@payment-bridge -f

MongoDB

A recipe packages the official MongoDB 8.0 build: tools/apps/mongodb/make-mongodb-app.sh 8.0.32. It listens on 127.0.0.1:27017, is stateful, and upgrades one major version at a time. Data from MongoDB 4.4 on Balena devices moves with mongodump and mongorestore. A mongo-express app is available for debugging through the hub.

Device settings

Ping targets, screen rotation and WiFi are kept on the hub. They survive a reflash, apply to devices that are offline as soon as they return, and can be set for a whole fleet.

Ping targets

Every device always pings the hub, the internet (8.8.8.8) and its default gateway. Add your own targets, such as a payment terminal or printer, to see their latency and outages in the Ping chart and the Events. Fleet targets and device targets add up; when both name the same host, the device's label wins. Up to 20 targets; a renamed target keeps its history.

If a gateway blocks ping while the internet answers, XDmon stops counting the gateway as down and shows a note instead of false alarms.

Screen rotation

0, 90, 180 or 270°, set on the fleet and optionally overridden per device. The agent turns the screen at once, without a restart, and checks it again every minute.

WiFi

For a device that is only reachable over WiFi there are two ways, and they share the same setting on the device:

• USB stick, on site: xdmon/wifi.conf with SSID, PASSWORD, optional HIDDEN and COUNTRY. The WiFi dialog can generate this file in your browser. The device removes the password from the stick after import.
• Dashboard, remotely for a device that is online: Settings → WiFi → Save & test.

A change from the dashboard is tested for up to 2 minutes. It stays only when WiFi connects, gets an address and the hub answers through it; otherwise the device goes back to the previous setting and the dialog tells you why (wrong password, network not found, no DHCP address, hub not reachable). Ethernet is preferred when both are connected.

Updates

The Updates window shows, per device, what the hub wants against what the device reports, for the OS, the XDmon Agent, apps, settings and proxy configuration.

Overview

Hover a cell for the target, the reported state and the reason. Click an OS or Agent cell to change that device; click Apps or Settings to open the matching device tab. Only devices that need attention hides everything that is fine. The ⇅ button on a fleet card opens this view for that fleet.

XDmon Agent

The agent performs every other update and is the only way to reach a device remotely, so it updates with extra care.

1. Publish from the build Mac: ./deploy-agent.sh v2.74. The release is signed. Once a stable version is set, publishing alone changes nothing on the devices.
2. Try it: Updates → XDmon Agent, set the candidate, choose one or a few canary devices, Save rollout.
3. Watch: a canary goes pending, downloading, starting, then up to date, or rolled back with the reason.
4. Promote when the canaries are fine.

A new agent is on probation until its first heartbeat reaches the hub. If it crashes repeatedly or sends no heartbeat within 10 minutes, the device restores the previous agent by itself and reports why. That version is not tried again until you click ↻ Retry.

OS updates

An OS update writes the new image into the inactive slot, boots it once as a trial and keeps it only after a health check: the data partition is mounted, the system is up, and the tunnel has a fresh handshake. A failed or hung trial boot returns to the previous slot by itself. Devices stream the bundle and fetch only the blocks that changed, typically a few percent of the image.

1. Publish from the build Mac: ./deploy-os.sh 2026.10.2 builds the signed bundle and puts it on the hub. Nothing is installed yet.
2. Roll out: Updates → OS rollout. Pick the version, all devices or one fleet, optional pilot devices, how many at a time, and after how many failures to stop.
3. Watch the progress bar and the state per device. Pause, continue after a halt, or cancel at any time; a device already installing always finishes.

A device that reports no result within 75 minutes counts as timed out. Failed devices are not retried by the rollout. For a single device, click its OS cell in the Overview, or Settings → Software → OS Change….

On the device, sudo xdmon-os-update status shows the version, booted and committed slot and the last update.

Stored versions

Every published OS bundle, agent release and app package stays on the hub until removed. Updates → Stored versions lists them with size, date, why a version is in use and which devices still run it, and preselects what can go.

A version that is in use (the stable or candidate agent, a pinned version, the bundle of a running rollout, an app version assigned to a fleet or device) cannot be deleted. Deleting needs Configure mode and is written to the audit log. Devices keep their own rollback copy, so deleting a version a device runs only means it cannot be installed again from the hub.

Hub health

The hub monitors itself every 15 seconds and keeps 7 days of history. Available to the roles admin and hubhealth.

Since agent 2.64 a heartbeat sends only what changed, about 150 bytes, with a full report every 30 minutes. An idle device uses roughly 4 MB of traffic per day.

Users and access

XDmon has no user database of its own. People sign in with your OIDC provider; their groups decide what they may open, per role and per device.

Two layers

1. Outer Caddy checks the coarse roles: may this person use XDmon at all, and may they use VNC.
2. XDmon checks per device on every device request (web pages, console, VNC, charts) and refuses admin actions to non-admins.

Roles

Which devices a user sees

Groups add up: a user in fleet:signage and group:Amsterdam sees both sets. Someone opening a device link they may not use gets a styled Access Denied page; every refusal is logged.

Access groups

Manage → Access groups defines groups of devices across fleets, for example all devices a service partner maintains. Create a group, tick its devices, then create the matching group:<name> group in your OIDC provider and add the people.

VNC policies in the outer Caddy

Full-control VNC needs its own policy so that view-only users cannot start it. Caddy sorts handle blocks by how specific they are, so the full-control path matches first:

authorization policy vnc_full {
    set auth url /caddy-security/login
    set forbidden url /forbidden
    allow roles admin vnc-fullcontrol
}
authorization policy vnc_view {
    set auth url /caddy-security/login
    set forbidden url /forbidden
    allow roles admin vnc-fullcontrol vnc-readonly
}

fleet.example.com {
    handle /api/vnc/*/start/full-control {
        authorize with vnc_full
        reverse_proxy localhost:8080
    }
    handle /api/vnc/* {
        authorize with vnc_view
        reverse_proxy localhost:8080
    }
    # … the rest of the generated snippet
}

Audit log and recordings

Who did what in the portal, and who had access to which device. Always an answer, kept for 400 days.

Filter by period, kind, user and device. Per device, the same entries appear in its Events window under Access. Request bodies are never logged, because they can contain secrets.

Terminal recordings are stored on the hub for 180 days and can be played back at 1× to 8×, or downloaded as an asciicast file. Treat recordings like the device itself: they contain whatever was on screen.

Balena devices and migration

XDmon needs no Balena. It can run next to Balena, so an existing fleet moves over device by device without losing sight of any device in between.

Phase 1: the XDmon Agent on your Balena devices

1. Set BALENA_API_TOKEN in ~/xdmon/.env.
2. Add the wireguard service (the XDmon Agent) to your Balena docker-compose.yml, using balena/docker-compose.wireguard.yml as reference, and push a release.
3. ☰ → Balena fleets: tick the fleets to watch.
4. Open an unprovisioned device and click Provision. The hub creates the peer and sets WG_DEPLOY on the device; the tunnel comes up and WG_DEPLOY clears itself.

Balena devices get monitoring, charts, events, console, VNC, web access through the hub, analysers and per-device access. Releases, balenaOS and the supervisor stay with Balena; the Updates view marks those cells managed by Balena. The terminal is for XDmon devices only.

Phase 2: new devices on the XDmon image

Build and flash the XDmon image, package your applications as app packages and assign them to a fleet, then add and pair devices as in chapter 5.

Phase 3: move existing devices

1. Run the device's apps as app packages on a test device first.
2. Back up what the device must keep, for example with mongodump. Reflashing replaces balenaOS and its data.
3. Clear the Balena UUID on the hub entry and set its fleet, so the device keeps its address, name and history; then Reinstall for new keys. The device's URL changes from the Balena UUID to its route key.
4. Flash the CM5 with the XDmon image, pair it, restore the data into the app's data/.
5. Remove the device from its Balena fleet.

Phase 4: Balena off

When no Balena devices are left, remove BALENA_API_TOKEN and deselect the Balena fleets. The Balena cards and actions disappear with the last Balena device.

The OS image

One golden image for every device. Device identity (WireGuard keys, name) is added at provisioning, never baked in.

What is in the image

• Debian trixie for the CM5, with a read-only root filesystem.
• Two OS slots (A/B) with Raspberry Pi tryboot, and a persistent partition that grows to fill the eMMC on first boot.
• WireGuard with the tunnel watchdog, USB import, pairing page and WiFi support.
• Sway kiosk with Chromium for web apps, GPU rendering, on-demand VNC.
• The app manager xdmon-app. The XDmon Agent is not in the image: the device downloads it from the hub on first connect.
• SSH with keys only. The admin key and the hub's terminal key come from the image, so a key change reaches every device with an OS update.

One-time setup on the build Mac

1. Docker Desktop with at least 8 GB memory and 60 GB disk.
2. Create the signing PKI: OPENSSL=$(brew --prefix openssl@3)/bin/openssl ./tools/rauc-ca.sh. Copy ~/xdmon-pki/ca.cert.pem to rootfs-overlay/etc/rauc/keyring.pem in the repo. Keep ca.key.pem offline.
3. Put at least one admin SSH public key in device/cm5-xdmon/ssh-authorized-keys, and the hub's terminal key in device/cm5-xdmon/hub-ssh-keys.

Build, publish, flash

./deploy-os.sh 2026.10.2              # build the signed OS bundle and publish it on the hub
./deploy-os.sh 2026.10.2 --image      # also the flash image for new devices
./deploy-os.sh --build-only           # build only

# flash a CM5 in rpiboot mass-storage mode
sudo tools/flash-image.py ~/xdmon-images/xdmon-kiosk.img.gz /dev/diskN

A build takes about 4 minutes on Apple Silicon. flash-image.py writes only what a new device needs (about 2.2 GB of the 7.3 GB image) and verifies it. A published version is never overwritten without --replace.

A bring-up variant (--bringup) adds password SSH, a rescue SSH on port 2222, a console login and boot diagnostics for test devices. It moves to and from production with a normal OS update.

Command reference

The hub CLI runs inside the dashboard container through ~/xdmon/xdmon.sh. Add --help to any command for its options.

CLI changes are not applied to the running hub. Click Apply WireGuard and Apply Caddy afterwards.

Build workstation

On a device

Troubleshooting

A device stays offline after pairing

• On the device: systemctl status wg-fleet and journalctl -u wg-fleet -n 30. Without a config the log names the pairing URL.
• Check that UDP 51820 to the hub is not blocked on site: nc -u -z <hub-ip> 51820.
• Make sure the hub has the device as a peer: Apply WireGuard, then docker exec xdmon-hub wg show wg0 on the server.

Online, but no data in the Overview

The agent has not started yet. It is downloaded on first connect; check systemctl status xdmon-agent and journalctl -u xdmon-update. On the device, curl -si http://10.20.0.1:9080/manifest.json | head -1 answers 200 or 404 when the update site is reachable, 403 when the address is not recognised.

Charts say Device not reachable

Charts, console and VNC go through the tunnel and the device's agent. If the dot is green but charts fail, restart the agent (sudo systemctl restart xdmon-agent, also allowed from the terminal) and check that Apply Caddy was run after the device was added.

An update shows failed

• Hover the cell in Updates for the reason. The device's Events window lists the result under Updates or Apps.
• Agent rolled back: the new agent did not send a heartbeat in time. Fix, publish a new version, or ↻ Retry.
• OS rolled back: the trial boot failed its health check. The device runs the previous OS unchanged.
• App failed: the health check did not pass and the previous version runs again. Look at journalctl -u xdmon-app@<name> in the console or terminal.

Missed heartbeats on every device at once

That is the hub, not the devices: a restart, an overloaded server or a network problem at the hosting provider. Check Hub health and docker logs xdmon-dashboard.

Terminal refuses to connect after a reflash

The device has a new SSH host key. Use Forget host key on the terminal page (admin), then reconnect.

WiFi change reverted

The dialog shows why: wrong password, network not found (tick hidden if it does not broadcast), no DHCP address, or the hub not reachable over WiFi (a firewall or captive portal blocking UDP 51820).

The page shows Stack Unavailable

The outer Caddy cannot reach XDmon on port 8080. On the server: cd ~/xdmon && docker compose ps and docker compose up -d.

Reference

Ports

Files on a device

Telemetry collected

API

Everything in the dashboard is also available as a JSON API under /management/api/, with the same roles. Administrators on the server can reach it through an SSH tunnel to port 5100. The main endpoints:

The full list is in the README of the release.