Production operations
Production operations turn the safety model into daily practice: durable supervision, restart verification, task lifecycle, observability, and troubleshooting.
Durable process supervision
Run the bridge under a durable supervisor such as systemd, launchd, a container supervisor, a process manager, or CI wrapper. Keep secrets outside the repository and capture logs.
<your-service-manager> start hermes-satellite-bridge <your-service-manager> status hermes-satellite-bridge <your-service-manager> logs hermes-satellite-bridge --since 10m
Restart and post-change verification
| Step | Evidence |
|---|---|
| Stop or reload | Supervisor reports old listener gone. |
| Start privately | Listener is on intended private address only. |
| Check liveness | /healthz returns OK, but health endpoint is liveness only. |
| Check authorization | Unauthenticated MCP initialize must fail; authenticated MCP initialize must succeed. |
| Submit tiny task | Terminal result, transcript evidence, and cost reporting are known or explicitly unknown. |
Task lifecycle
Tasks move through pending, running, completed, failed, cancelled, and corrective follow-up. A failed task is terminal unless an earlier completed run produced a resumable session for hermes_respond.
Observability concepts
Watch task state, MCP events, terminal result text, transcript evidence, and cost snapshots together. Events explain the bridge; transcripts explain Hermes; costs explain known, unknown, or unreconciled spend.
The hst CLI
On the Hermes host, hst reads the bridge's SQLite state directly and renders it for an operator. It is a single-file Bun CLI over ~/.hermes/async_bridge.db. The database opens read-write for WAL compatibility, then PRAGMA query_only=ON preserves read-only semantics. A database that cannot be opened produces a one-line hst: cannot open bridge db error and a clean exit.
hst health service, port, and recent activity hst tasks [-n N] [-s STATUS] recent queue with status, caller, and gist hst task <id-prefix> one task in full: summary, runs, cost, timeline, result hst watch [id-prefix] stream events; with an id, exits on terminal status hst events [-n N] [id-prefix] bridge MCP event log hst costs [-n N] cost snapshots per task, with caller hst transcript <id-prefix> export session transcript (JSONL), print path hst logs [-f] bridge process logs
--json is available on tasks, task, events, and costs. JSON output is for scripts and never calls a gist provider.
| Command | Use it when | What to read |
|---|---|---|
hst health | Starting a host check or proving the bridge is alive. | Service state, port, database path, and recent activity. |
hst tasks | Scanning the queue before choosing a task to inspect. | Status, caller, age, duration, and the gist column. |
hst task <id-prefix> | Investigating one run. | Prompt, result, run rows, per-loop costs, timeline, and terminal error text. |
hst watch [id-prefix] | Keeping a terminal on live bridge activity. | All events, or one task until it reaches a terminal status. |
hst events | Debugging MCP calls and bridge-side state changes. | Initialize, tools/call, status, result, and error events. |
hst costs | Checking spend after one or many tasks. | Caller, model/provider, token counts, billing mode, and honest cost rendering. |
hst transcript | Needing T2 evidence for verification. | The exported session JSONL path. |
hst logs [-f] | Checking the bridge process log without leaving the host. | Recent process output, optionally followed live. |
The gist column in hst tasks replaces the truncated raw prompt with a short generated summary of at most eight words. Gists come from Groq (llama-3.1-8b-instant) when GROQ_API_KEY is set in the environment or ~/.hermes/.env, otherwise from OpenRouter (meta-llama/llama-3.3-70b-instruct:free) when OPENROUTER_API_KEY is set; HST_GIST_MODEL overrides the model. New tasks are summarized once in a single batched call and cached at ~/.hermes/cache/hst_summaries.json, so later runs work offline. With no key, timeout, network failure, or provider error, the column silently falls back to the raw prompt. OpenRouter :free endpoints require account privacy settings that permit them and can be upstream-rate-limited with 429s at peak; hst self-heals on a later run.
Cost rendering follows the same rule as the troubleshooting matrix below: never report zero unless proven. hst costs shows the caller for each snapshot, and a $0 snapshot renders as $0 (subscription) only when billing is subscription-included; any other $0 renders as unknown, never free. hst task applies the same rendering to its per-loop cost lines.
| Variable | Effect |
|---|---|
HERMES_ASYNC_BRIDGE_DB | Override the bridge DB path; default is ~/.hermes/async_bridge.db. |
HERMES_HOME | Override the Hermes home used for the default DB path, .env, and gist cache. |
GROQ_API_KEY | Preferred gist provider key, read from the environment or ~/.hermes/.env. |
OPENROUTER_API_KEY | Fallback gist provider key, read from the environment or ~/.hermes/.env. |
HST_GIST_MODEL | Override the selected gist model. |
NO_COLOR | Disable ANSI color for pasteable output. |
Troubleshooting is intentionally small. hst: cannot open bridge db means the path is wrong or unreadable; check HERMES_ASYNC_BRIDGE_DB, HERMES_HOME, and that the bridge has created the SQLite file. If gists show raw prompts, there is no usable key, the provider timed out, or OpenRouter rate-limited the free endpoint; the cache fills on a later successful run.
Troubleshooting matrix
| Symptom | Likely cause | Recovery |
|---|---|---|
| Auth failure | Wrong token, missing header, rotated secret, or cached client config. | Reload secrets, restart client, repeat auth smoke test. |
| Client discovery failure | Client expects stdio or did not restart. | Configure URL Streamable HTTP and restart. |
| Polling timeout | Task exceeded 600 seconds or executor hung. | Stop polling, report stuck state, inspect logs/events. |
| Missing result | Terminal state not reached or result fetch skipped. | Poll status, then call hermes_result once. |
| Missing transcript | No session id or transcript export unavailable. | Report T0/T1 cap or rerun with transcript support. |
| Missing cost | Provider counters unavailable or unreconciled. | Report unknown/unreconciled; never report zero unless proven. |
