Monitoring

The broker's job that matters most is getting save data back to RomM. When that fails, the save archive waits on the container and nobody is told unless something is watching. This page sets that watch up.

There are two separate questions, and they get two separate answers:

• Is the broker up? Answered by GET /api/health. Safe to use as a container healthcheck.
• Did anything go wrong with a game or its saves? Answered by scripts/check_broker.py. Run it on a schedule and send its output to a person. Do not use it as a container healthcheck (see below).

The check script

scripts/check_broker.py reads three routes (/api/health, /api/session/status and /api/session/exports), prints one line and exits with a Nagios-style code:

It needs Python 3 and nothing else, so it runs anywhere that can reach the broker. It reads the broker secret from the BROKER_SECRET environment variable.

Where to run it

The script is not installed in the container image, so it is fed in from a checkout of this repo, or from a copy of the single file.

From the Docker host (recommended)

Pipe the script into the container's own Python. It then talks to the broker directly (on 127.0.0.1:8000 unless BROKER_HOST or BROKER_PORT move it), with no TLS and no reverse proxy in the way, and docker exec hands it the container's BROKER_SECRET, BROKER_PORT and SUBFOLDER, so there is no second copy of any of them to keep in sync:

docker exec -i webstation python3 - < scripts/check_broker.py

Pass options after the -:

docker exec -i webstation python3 - --max-session-hours 6 < scripts/check_broker.py

Replace webstation with your container_name.

From another machine

Point it at the container's HTTP port, 3000. If you'd rather go through your reverse proxy, use RomM's own address instead:

BROKER_SECRET=change-me python3 check_broker.py \
  --url http://yourhost:3000/streaming

On a schedule

Every 5 minutes is plenty: nothing it watches changes faster than a game session does. A cron entry that only speaks up when something is wrong:

*/5 * * * * out=$(docker exec -i webstation python3 - < /opt/romm-broker/scripts/check_broker.py) || echo "$out"

Cron mails any output to the crontab's owner, so this sends one message per failing run and stays silent while things are OK. To send it somewhere else, replace echo "$out" with your notifier (curl to an ntfy, Gotify, Discord or Slack webhook, for example).

Any monitoring system that runs a command and reads its exit code (Nagios, Icinga, Checkmk, Zabbix, Uptime Kuma's push monitors driven by cron) can run the same command directly and use its own alert routing.

The container healthcheck

For a healthcheck, ask only whether the broker answers:

services:
  webstation:
    healthcheck:
      test: ["CMD", "curl", "-fs", "http://127.0.0.1:8000/streaming/api/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 240s

start_period is long on purpose: the broker does not start until selkies is up, and a cold first boot can take a few minutes.

Why the check is not a container healthcheck

Docker marks a container unhealthy whenever its healthcheck fails, and tools such as autoheal (common on Unraid) restart unhealthy containers. Most of what check_broker.py reports is fixed by a person, not by a restart. A failed upload is not fixed by one, and an emulator that crashed mid-game still has save data on disk that a restart would strand. A restart also ends every game in progress and clears last_exit, so the alert that explained the problem goes too.

What normal looks like

Idle, the check prints:

OK: broker up, idle, 0 kept archive(s)

During a game it names the session instead of idle.

In the broker log (docker logs webstation), every game is bracketed by two lines:

activate: session 3f2a... launched pcsx2 in 4.2s (restored 3 save file(s), tokens pushed: True)
exit: session 3f2a... done in 18.6s: state_saved=True upload=uploaded kept=-

A launch usually takes a few seconds; shadPS4 and RPCS3 can take minutes while they unpack a .pkg. An exit takes from a few seconds up to about a minute, depending on how long the emulator takes to save its state and stop.

On the exit: line, upload is one of:

When the emulator goes mid-session

Between those two lines, the broker checks every 15 seconds (BROKER_EMULATOR_WATCH_INTERVAL) that the session's emulator is still running. If it is gone while the session is still open, one line is logged for that session:

emulator watch: session 3f2a...: pcsx2 exited while the session was open (exit code -11, rom Game); exit the session from RomM to save and upload what it wrote

It is an ERROR for a crash (a signal or an error code) and a WARNING for exit code 0, which is usually a player quitting from the emulator's own menu. To be told about crashes as they happen, have your log shipper alert on emulator watch: at ERROR. What each exit code means, and what to do, is in The game froze mid-session.

What each alert means

Worst first. When one run finds several problems it prints them all on one line, separated by ;, under the worst state.

Alerts about the last exit clear on their own once they are dealt with. A failed upload with a kept archive stays raised until that archive is deleted from BROKER_EXPORT_DIR. A failed dump, an upload that kept nothing and a slow exit leave nothing on disk to track, so they stay raised for --max-last-exit-hours (24 by default) and then drop; the next exit also replaces them. Deal with them inside that window, or raise it if nobody reads the alerts that often.

Recovering a kept archive

List what is waiting, download each archive, then delete it once it is safe elsewhere (in RomM, or anywhere you keep backups):

curl -H "X-Broker-Secret: $BROKER_SECRET" http://yourhost:3000/streaming/api/session/exports
curl -H "X-Broker-Secret: $BROKER_SECRET" -o save.zip \
  http://yourhost:3000/streaming/api/session/exports/<name>
curl -X DELETE -H "X-Broker-Secret: $BROKER_SECRET" \
  http://yourhost:3000/streaming/api/session/exports/<name>

See Save archives for the archive layout.

What the check cannot see

• A restart forgets the last exit. last_exit lives in memory. A failed upload is still caught after a restart, because its archive stays on disk, but a slow exit or a failed dump from before the restart is not.
• Old alerts expire. A failed dump or slow exit older than --max-last-exit-hours stops being reported even if nobody acted on it. The exit: line in the log keeps the record.
• It samples; it does not watch. An emulator that crashes and is exited between two runs never shows as not running. The broker's own emulator watch still logs the crash, and the exit's own line records how its saves went.
• It only knows what the broker knows. An archive RomM accepted but filed wrongly shows as uploaded.