Operations/Production operations

Production operations

Production operations turn the safety model into daily practice: durable supervision, restart verification, task lifecycle, observability, and troubleshooting.

Durable process supervision

Run the bridge under a durable supervisor such as systemd, launchd, a container supervisor, a process manager, or CI wrapper. Keep secrets outside the repository and capture logs.

text
<your-service-manager> start hermes-satellite-bridge
<your-service-manager> status hermes-satellite-bridge
<your-service-manager> logs hermes-satellite-bridge --since 10m

Restart and post-change verification

StepEvidence
Stop or reloadSupervisor reports old listener gone.
Start privatelyListener is on intended private address only.
Check liveness/healthz returns OK, but health endpoint is liveness only.
Check authorizationUnauthenticated MCP initialize must fail; authenticated MCP initialize must succeed.
Submit tiny taskTerminal result, transcript evidence, and cost reporting are known or explicitly unknown.

Task lifecycle

Tasks move through pending, running, completed, failed, cancelled, and corrective follow-up. A failed task is terminal unless an earlier completed run produced a resumable session for hermes_respond.

Observability concepts

Watch task state, MCP events, terminal result text, transcript evidence, and cost snapshots together. Events explain the bridge; transcripts explain Hermes; costs explain known, unknown, or unreconciled spend.

The hst CLI

On the Hermes host, hst reads the bridge's SQLite state directly and renders it for an operator. It is a single-file Bun CLI over ~/.hermes/async_bridge.db. The database opens read-write for WAL compatibility, then PRAGMA query_only=ON preserves read-only semantics. A database that cannot be opened produces a one-line hst: cannot open bridge db error and a clean exit.

text
hst health                    service, port, and recent activity
hst tasks [-n N] [-s STATUS]  recent queue with status, caller, and gist
hst task <id-prefix>          one task in full: summary, runs, cost, timeline, result
hst watch [id-prefix]         stream events; with an id, exits on terminal status
hst events [-n N] [id-prefix] bridge MCP event log
hst costs [-n N]              cost snapshots per task, with caller
hst transcript <id-prefix>    export session transcript (JSONL), print path
hst logs [-f]                 bridge process logs

--json is available on tasks, task, events, and costs. JSON output is for scripts and never calls a gist provider.

CommandUse it whenWhat to read
hst healthStarting a host check or proving the bridge is alive.Service state, port, database path, and recent activity.
hst tasksScanning the queue before choosing a task to inspect.Status, caller, age, duration, and the gist column.
hst task <id-prefix>Investigating one run.Prompt, result, run rows, per-loop costs, timeline, and terminal error text.
hst watch [id-prefix]Keeping a terminal on live bridge activity.All events, or one task until it reaches a terminal status.
hst eventsDebugging MCP calls and bridge-side state changes.Initialize, tools/call, status, result, and error events.
hst costsChecking spend after one or many tasks.Caller, model/provider, token counts, billing mode, and honest cost rendering.
hst transcriptNeeding T2 evidence for verification.The exported session JSONL path.
hst logs [-f]Checking the bridge process log without leaving the host.Recent process output, optionally followed live.

The gist column in hst tasks replaces the truncated raw prompt with a short generated summary of at most eight words. Gists come from Groq (llama-3.1-8b-instant) when GROQ_API_KEY is set in the environment or ~/.hermes/.env, otherwise from OpenRouter (meta-llama/llama-3.3-70b-instruct:free) when OPENROUTER_API_KEY is set; HST_GIST_MODEL overrides the model. New tasks are summarized once in a single batched call and cached at ~/.hermes/cache/hst_summaries.json, so later runs work offline. With no key, timeout, network failure, or provider error, the column silently falls back to the raw prompt. OpenRouter :free endpoints require account privacy settings that permit them and can be upstream-rate-limited with 429s at peak; hst self-heals on a later run.

Cost rendering follows the same rule as the troubleshooting matrix below: never report zero unless proven. hst costs shows the caller for each snapshot, and a $0 snapshot renders as $0 (subscription) only when billing is subscription-included; any other $0 renders as unknown, never free. hst task applies the same rendering to its per-loop cost lines.

VariableEffect
HERMES_ASYNC_BRIDGE_DBOverride the bridge DB path; default is ~/.hermes/async_bridge.db.
HERMES_HOMEOverride the Hermes home used for the default DB path, .env, and gist cache.
GROQ_API_KEYPreferred gist provider key, read from the environment or ~/.hermes/.env.
OPENROUTER_API_KEYFallback gist provider key, read from the environment or ~/.hermes/.env.
HST_GIST_MODELOverride the selected gist model.
NO_COLORDisable ANSI color for pasteable output.

Troubleshooting is intentionally small. hst: cannot open bridge db means the path is wrong or unreadable; check HERMES_ASYNC_BRIDGE_DB, HERMES_HOME, and that the bridge has created the SQLite file. If gists show raw prompts, there is no usable key, the provider timed out, or OpenRouter rate-limited the free endpoint; the cache fills on a later successful run.

Troubleshooting matrix

SymptomLikely causeRecovery
Auth failureWrong token, missing header, rotated secret, or cached client config.Reload secrets, restart client, repeat auth smoke test.
Client discovery failureClient expects stdio or did not restart.Configure URL Streamable HTTP and restart.
Polling timeoutTask exceeded 600 seconds or executor hung.Stop polling, report stuck state, inspect logs/events.
Missing resultTerminal state not reached or result fetch skipped.Poll status, then call hermes_result once.
Missing transcriptNo session id or transcript export unavailable.Report T0/T1 cap or rerun with transcript support.
Missing costProvider counters unavailable or unreconciled.Report unknown/unreconciled; never report zero unless proven.