Operations
Once your config works interactively, you usually want the bridge and poller running unattended. This page covers macOS launchd, Linux systemd, what to restart after a config change, and how to inspect runtime state.
Long-lived services
ctrlrelay generates platform-appropriate service unit files for you with
ctrlrelay install. The two daemons it manages:
| Service | Command | Why you want it running |
|---|---|---|
| Bridge | ctrlrelay bridge start |
Delivers BLOCKED questions to your chat (Telegram or Mattermost) and routes answers back. |
| Poller | ctrlrelay poller start --interval 300 |
Watches GitHub for newly assigned issues and runs the dev pipeline. |
| Status page (optional) | ctrlrelay web start |
Read-only page and JSON API of what ctrlrelay is doing. See Status page. |
Both should restart on failure and on login.
ctrlrelay bridge start and ctrlrelay poller start daemonize by default
(fork, write a PID file, return to the shell). Under a process supervisor
(launchd, systemd) pass --foreground so the supervisor can track the PID
and restart on failure — the templates below already do this.
macOS — launchd
Recommended: let ctrlrelay install launchd render the plists for you.
It substitutes ${USER}, ${HOME}, the absolute ctrlrelay path,
your working directory, and (when exported) your chat token into the
in-package templates and writes them to ~/Library/LaunchAgents/.
The token goes into the plist under the variable your transport names
in transport.<type>.bot_token_env — CTRLRELAY_TELEGRAM_TOKEN for
Telegram, CTRLRELAY_MATTERMOST_TOKEN for Mattermost, or whatever you
renamed that setting to. The command reads your orchestrator.yaml to
find out which; pass --config if it is somewhere the usual search
does not look. Export that variable before running, or the plist keeps
a placeholder and the command says which variable to export.
export CTRLRELAY_MATTERMOST_TOKEN=your-bot-token # or _TELEGRAM_, per your config
ctrlrelay install launchd \
--workdir ~/Projects/ctrlrelay \
--label-prefix com.yourname \
--poller-interval 300
# Then load the agents:
mkdir -p ~/.ctrlrelay/logs
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.yourname.ctrlrelay-bridge.plist
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.yourname.ctrlrelay-poller.plist
Pick a reverse-DNS --label-prefix you own (e.g. com.yourname); it
flows into both the filename and the <Label> value so launchctl
list output is unambiguous. Pass --dry-run first to see what would
be written; pass --force to overwrite a previously installed plist.
If you’d rather hand-write the plists, here are the full templates the
installer uses. They show CTRLRELAY_TELEGRAM_TOKEN; on a Mattermost config
use the variable your transport.mattermost.bot_token_env names
(CTRLRELAY_MATTERMOST_TOKEN by default).
~/Library/LaunchAgents/com.example.ctrlrelay-bridge.plist:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.ctrlrelay-bridge</string>
<key>ProgramArguments</key>
<array>
<string>/opt/homebrew/bin/ctrlrelay</string>
<string>bridge</string>
<string>start</string>
<string>--foreground</string>
</array>
<key>EnvironmentVariables</key>
<dict>
<key>CTRLRELAY_TELEGRAM_TOKEN</key>
<string>your-bot-token-here</string>
<key>PATH</key>
<string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>
</dict>
<key>WorkingDirectory</key>
<string>/Users/YOU/Projects/ctrlrelay</string>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
<key>StandardOutPath</key>
<string>/Users/YOU/.ctrlrelay/logs/bridge.log</string>
<key>StandardErrorPath</key>
<string>/Users/YOU/.ctrlrelay/logs/bridge.error.log</string>
</dict>
</plist>
~/Library/LaunchAgents/com.example.ctrlrelay-poller.plist:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.ctrlrelay-poller</string>
<key>ProgramArguments</key>
<array>
<string>/opt/homebrew/bin/ctrlrelay</string>
<string>poller</string>
<string>start</string>
<string>--foreground</string>
<string>--interval</string>
<string>300</string>
</array>
<key>EnvironmentVariables</key>
<dict>
<key>CTRLRELAY_TELEGRAM_TOKEN</key>
<string>your-bot-token-here</string>
<key>PATH</key>
<string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>
</dict>
<key>WorkingDirectory</key>
<string>/Users/YOU/Projects/ctrlrelay</string>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
<!-- Give the in-process scheduler time to drain a running secops
sweep on stop (worktree cleanup can take ~120s). launchd's
default ExitTimeOut is 20s, which would SIGKILL mid-cleanup. -->
<key>ExitTimeOut</key>
<integer>180</integer>
<key>StandardOutPath</key>
<string>/Users/YOU/.ctrlrelay/logs/poller.log</string>
<key>StandardErrorPath</key>
<string>/Users/YOU/.ctrlrelay/logs/poller.error.log</string>
</dict>
</plist>
Whether you used ctrlrelay install launchd or hand-wrote the plists,
manage the agents with:
launchctl list | grep ctrlrelay # check loaded
launchctl bootout gui/$(id -u)/com.example.ctrlrelay-poller # stop
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.example.ctrlrelay-poller.plist # start
Linux — systemd
Recommended: ctrlrelay install systemd renders the user unit files
for you (no root needed) and writes them to ~/.config/systemd/user/.
As on macOS, the bridge and poller units carry the token under the
variable transport.<type>.bot_token_env names, read from your
orchestrator.yaml (--config to point at a different one). Export
that variable before running.
export CTRLRELAY_MATTERMOST_TOKEN=your-bot-token # or _TELEGRAM_, per your config
ctrlrelay install systemd \
--workdir ~/Projects/ctrlrelay \
--poller-interval 300
# Then enable and start:
mkdir -p ~/.ctrlrelay/logs
systemctl --user daemon-reload
systemctl --user enable --now ctrlrelay-bridge.service
systemctl --user enable --now ctrlrelay-poller.service
To survive logout (start at boot, stop at shutdown):
sudo loginctl enable-linger "$USER"
If you’d rather hand-write the units, here are the full templates the
installer uses. Save under ~/.config/systemd/user/.
~/.config/systemd/user/ctrlrelay-bridge.service:
[Unit]
Description=ctrlrelay chat bridge (Telegram or Mattermost)
After=network-online.target
[Service]
Type=simple
WorkingDirectory=%h/Projects/ctrlrelay
# The variable your transport names in bot_token_env — use
# CTRLRELAY_MATTERMOST_TOKEN on a Mattermost config.
Environment=CTRLRELAY_TELEGRAM_TOKEN=your-bot-token-here
ExecStart=%h/.local/bin/ctrlrelay bridge start --foreground
Restart=always
RestartSec=5
StandardOutput=append:%h/.ctrlrelay/logs/bridge.log
StandardError=append:%h/.ctrlrelay/logs/bridge.error.log
[Install]
WantedBy=default.target
~/.config/systemd/user/ctrlrelay-poller.service:
[Unit]
Description=ctrlrelay issue poller
After=network-online.target ctrlrelay-bridge.service
[Service]
Type=simple
WorkingDirectory=%h/Projects/ctrlrelay
# The variable your transport names in bot_token_env — use
# CTRLRELAY_MATTERMOST_TOKEN on a Mattermost config.
Environment=CTRLRELAY_TELEGRAM_TOKEN=your-bot-token-here
ExecStart=%h/.local/bin/ctrlrelay poller start --foreground --interval 300
Restart=always
RestartSec=5
# Give the in-process scheduler time to drain a running secops sweep on
# stop (worktree cleanup can take ~120s). systemd's default
# TimeoutStopSec is 90s, which would SIGKILL the poller mid-cleanup and
# leak git worktree admin state.
TimeoutStopSec=180
StandardOutput=append:%h/.ctrlrelay/logs/poller.log
StandardError=append:%h/.ctrlrelay/logs/poller.error.log
[Install]
WantedBy=default.target
Once written, enable and start with the same systemctl --user enable
--now commands shown earlier in the section. Check status / logs:
systemctl --user status ctrlrelay-poller
journalctl --user -u ctrlrelay-poller -f
Status page
ctrlrelay web start serves a read-only page and JSON API showing what each
agent is doing (phase, current activity, last command), what is waiting and
why, the questions that need an answer, and a feed of what ctrlrelay did and
said. It is a third process next to the bridge and the poller, so a slow or
broken page can never stall them.
On a running card, phase (the bar) is the furthest step the session has reached, from the agent’s transcript plus ctrlrelay’s own CI, review and PR events, so it never moves back. Current activity is the most recent step: an open CI wait or review if it started after the agent’s latest step, else that step. It can be earlier than the bar, for example a read after the PR or a fix round after review. The card links the PR once ctrlrelay has recorded its number.
It is read-only: it opens state.db read-only, reads the tail of each
agent’s transcript, and asks GitHub (through the gh login ctrlrelay already
uses) for the waiting line once a minute. It writes nothing but its PID file
(web.pid, next to state.db), and serves GET only; any other method is
405. It has no login and shows repository names, issue titles and the
questions agents asked, so never expose it to the internet. Keep it on
loopback or on the home network.
| Path | What |
|---|---|
/ |
The page. Refreshes itself every web.refresh_seconds. |
/api/status |
Everything on the page, as JSON (the contract is in src/ctrlrelay/web/status.py). A section that could not be read is null and errors says why; [] means “read, and empty”. The feed holds at most the newest feed_limit (200) events: feed_total is how many the history window holds (null if unreadable) and feed_truncated is true when some were left out, in which case the page says “showing newest N of M”. |
/api/events?since=<epoch seconds> |
The feed, newest first, at most limit (200) events. Without since, the last web.history_days days. Carries the same feed_total and feed_truncated, plus the limit used. |
/api/events?kind=<kind>&day=today |
One Today counter’s events, for the server’s local day, newest first, at most 200: {kind, events, total, truncated, error}. kind is dispatched, pr_opened, merged, failed, blocked or message_sent. A failed or blocked event also carries reason: exit_code, the run’s summary (up to 500 characters) and output_tail (up to 2,000), both scrubbed of credential shapes again before they are sent, the question asked, and session: "not found" when the run’s row is gone. events is null with error set when they could not be read. A query with kind and anything other than exactly kind and day=today is refused with 400, except that one with since is the feed, as before. A summary or question longer than 500 characters ends in “…”. On the page, clicking a Today counter shows this list under it, read again with every refresh. |
The page header shows how long ago it last heard from the server (“updated 12s ago”, ticking every second) and a dot that is green while the last good answer is at most two refresh intervals old and red otherwise. Elapsed times on cards tick between polls too.
Fields added in #232 (all in /api/status):
| Field | What |
|---|---|
url |
On every running, needs_you, waiting.items, skipped and feed item (and on /api/events events): a link to the GitHub issue, or for a feed event that names a PR by number, the PR. Built only as https://github.com/<owner>/<repo>/issues/<n> or /pull/<n>, from a repo matching owner/name and an integer; null otherwise (a secops run with no issue, a restart). A pr_url an agent reported is never used. |
refs |
On every needs_you item (#241): one {"number", "url"} per distinct #N in the question, in order, at most 20, with url built the same way as url above from the item’s repo and N. A #N right after a letter, digit, _, / or & is not counted. The page turns each listed #N in the question into a link, so a secops question’s PRs are clickable. [] when there are none. |
skipped |
Every open issue assigned to you that the poller will not pick up and is not running, from the same search as the waiting line (no extra GitHub calls): repo, issue_number, title, url, updated/updated_ts and reason, one of repo not in config, excluded label <label>, already handled once (won't retry), missing required label <label> (several joined with “or”: any one admits it), assigned by someone else (not you and not an account in allowed_accounts), blocked by <#N, owner/repo#N> or could not read its blockers; asked again every poll (#310, see Issues blocked by other issues; these two are listed for a label-triggered issue too). null until GitHub has answered once; waiting.status and waiting.last_good_ts say how current it is. The page shows it collapsed, with a count. |
server |
This machine’s health, read on each request: load_1m, load_5m, load_15m, cpu_count, memory (total_bytes, available_bytes, available_pct, low), disks (one each for home and worktrees: name, path, total_bytes, free_bytes, free_pct, low), uptime_s, and unavailable (why any number is null; memory and uptime are null off Linux). low is true below 10% free and the page marks it. The whole section is null, with a line in errors, only if reading it failed outright. |
Added in #247:
| Field | What |
|---|---|
node |
This machine’s own block: name (node_id); usage, the agent subscription’s usage as the last run or probe read it: status, read_at, and for five_hour and seven_day each {used (0 to 1), resets_at}, or "unknown" for a window it could not read; the whole usage is "unknown" before any reading. pause_at is usage.pause_at, and paused is {reason, until} while the machine takes no new issues, else null. The page shows it in the Usage box. node is null, with a line in errors, only if reading it failed outright. |
Added in #223, on every running card (see Run budgets):
| Field | What |
|---|---|
tokens_used |
Tokens the session has used so far, over all its spawns: input + cache writes + output, what the budget counts. null before anything was measured, on a backend that is not measured, or when a transcript row could not be read (then reason gives the number of rows and the floor). Never 0 for unknown. |
cache_read_tokens |
Cache reads, recorded but not counted. null as for tokens_used. |
cost_usd |
The sum of total_cost_usd the agent CLI reported at the end of each spawn. Information only. null until a spawn reported one. |
token_soft_limit, token_hard_limit |
The limits applied to this run. null when no limit applies or nothing was measured yet. |
The page shows them in a tokens: row on each running card; the count is in
the warning colour once it passes the soft limit.
Added in #237: node.github_api is the account’s GitHub API budget as this
machine last read it: read_at, and for core, search and graphql each
{remaining, limit, resets_at, state} with state one of ok, low, out,
or unknown once its reset time has passed (the numbers are then old), or
"unknown" in place of the object for a bucket it could not read; the whole value is "unknown"
before any reading. The page shows it in the Usage box; a bucket whose
reset time has passed since the last read shows as “refilled at
Added in #249:
| Field | What |
|---|---|
node.version, node.account_label, node.chat |
This machine’s version, its fleet.account_label, and its chat: {transport, connected} (connected is whether the bridge socket exists), or null with no chat configured. |
node.state |
working, idle, paused-limit (paused for usage), silent (the daemon’s heartbeat is missing or older than fleet.silent_after_seconds), or unknown (the state database could not be read). |
node.slots |
{total, busy}. The daemon works one issue at a time, so total is 1. |
node.claims |
What this machine holds: every running issue and every issue waiting for an answer, each {repo, issue_number, kind, phase, since, url}. null when the state database could not be read. |
node.heartbeat_at, node.last_poll_at |
When the daemon last wrote its heartbeat (every 30 seconds, from its own task, so it moves while a run is going) and last polled GitHub. From heartbeat.json beside the state database. |
node.open_questions |
How many questions are waiting for an answer. |
fleet |
{peers: [...]} with one item per fleet.peers URL: url, name, reachable, checked_at, error, state, node (that peer’s node block), last_known (its last block while it does not answer) and needs_you. null when no peers are configured. |
Added in #307:
| Field | What |
|---|---|
node.config_fingerprint |
{v, repos_hash, profiles, repos, schedules}, v is 2. profiles maps a profile hash to {automation, steps}, each a map of field name to a hash of its value; repos maps each repo name to its profile hash, so repos with the same configuration share one profile; repos_hash is the hash of the sorted, lowercased repo names; schedules maps each schedules field to a hash. Hashes and repo names only, never a value. null if it could not be built. A peer’s fingerprint is unknown unless every hash is 16 lowercase hex characters, repos_hash is the hash of its own repo names, each profile key is the hash of that profile, and each repo names a profile it carries. A field’s hash cannot be checked against its value, which is never sent. |
fleet.peers[].drift |
{status, reason, only_here, only_there, fields, lines}. status is same, differs or unknown (with reason: not read yet, not reachable, the peer sends no fingerprint in this format, or the comparison failed); fields maps a field path such as automation.exclude_labels to the repos it differs on; lines is what the page shows. Leaves out fleet.expected_drift. The peer’s own fingerprint is not repeated in fleet.peers[].node. |
Added in #305:
| Field | What |
|---|---|
node.sandbox |
This machine’s agent.sandbox: "none" (the agent runs with the user’s full access) or "bwrap". In fleet.peers[].node it is null when the peer is on an older version or sent anything else, which means unknown, never off. The page shows sandbox off in the warning colour for "none". |
Added in #230: numbers is how this machine’s runs went, for the last 7 and
the last 30 days (numbers.windows, each with days, since/since_ts,
until/until_ts). The page shows it in the Numbers box, with a table
per repo under it. Numbers across machines are not here; they belong to the
control plane. Every field is defined in src/ctrlrelay/web/numbers.py.
Field (in overall and in each by_repo item) |
What |
|---|---|
started |
Runs (rows in the sessions table) started in the window, both ends included. A blocked run resumed after an answer is still one run. |
done, failed, blocked, running, other |
How many of those runs have that status now. running includes a run left behind by a daemon that stopped mid-run. The five add up to started. |
success_rate |
done / (done + failed + blocked), 0 to 1; null when none has finished. |
time_to_pr_s, time_to_merge_s |
{median, n}: seconds from the run’s start to its first pr_opened (or merged) event, over the n runs that have one. Events stamped after now are not read. null when events cannot be read, or when a run in the window started before the oldest event still kept: its PR event may have been deleted. The daemon deletes events older than web.history_days (7 by default), so with the default the 30-day times are null. To get them, set web.history_days to 30; they appear once the events kept reach back past every run in the window (up to 30 days after the change, since deleted events do not come back). |
came_back |
Distinct issues dispatched again in the window after an earlier run on the same issue had ended done. The window’s came_back_issues lists them (repo, issue_number, url; at most 50). |
reopened |
Issues the relay finished (it saw their PR merge and closed them) that were reopened on GitHub in the window, dated by GitHub’s reopened event. Counted whether or not the relay picks the issue up again; one per finish. The window’s reopened_issues lists them. The poller notices a reopen only for an issue its search returns (open and assigned to the operator, or wearing a trigger label), with no extra GitHub call unless a finished issue shows up open. Finishes are recorded in the state database from this version on, plus those still in the kept merged events when the table is created. null on a database the daemon has not migrated yet. A reopened event goes to the feed when one is recorded. |
cost_per_run |
What the runs that ended (done or failed) in the window used, from the token usage the run budget records on each run (#223, see Run budgets): {tokens, cost_usd, runs_without_usage}. tokens is {median, total, n} over the n ended runs whose usage was recorded and read in full, counted as the budget counts (input + cache writes + output; cache reads left out). cost_usd is {median, total, n} in US dollars over those of them whose agent reported a cost (total_cost_usd, information only), or null when none did. runs_without_usage is how many ended runs are left out: no usage recorded (runs from before #223, or a backend that is not measured, such as opencode) or usage that could not be read in full. They are never counted as 0. Blocked and running runs are left out, since their count still grows. null when no ended run in the window has its usage measured, or on a database the daemon has not migrated yet; the window’s unavailable says which, and why any other field is null. |
stuck |
Runs the relay stopped as stuck (#224), also counted in failed. null only on a database the daemon has not migrated yet. |
numbers is null, with a line in errors, when the sessions cannot be read.
Each machine also sends these 7-day numbers to the chat once a week; see configuration: schedules.weekly_summary.
The page with peers. A Machines strip shows one card per machine,
this one first: state, slots, 5-hour and 7-day usage, heartbeat age. A card is
red when the machine is paused, silent, unknown, not reachable, or not read
yet. A peer that answers with a different version from this machine’s is red
too, with version X, this machine runs Y on its card, so a node a release
missed shows up; a peer whose version is unknown is not marked. A peer that
does not
answer is shown as not reachable from here, with the reason (timeout,
refused, name not found, bad host header, an HTTP status) and its last
known numbers greyed out, never as idle. Who has what lists every claimed
issue across the machines, and Needs you shows every machine’s questions,
each tagged with its machine. The rest of the page is this machine’s own.
Pausing for usage. Every agent run reports the subscription’s usage, and
an idle machine refreshes it with one tiny call every usage.probe_minutes.
When the 5-hour or 7-day window reaches usage.pause_at, or the status is not
allowed, the machine stops taking new issues (runs already going finish),
sends one chat line, and takes issues again after the window’s reset, with one
more chat line. A usage it could not read is shown as unknown and never pauses
it. See configuration: usage.
Try it on loopback first (the default web.host is 127.0.0.1, port 8787):
ctrlrelay web start
ctrlrelay web status
curl -s http://127.0.0.1:8787/api/status | python3 -m json.tool | head -40
curl -s "http://127.0.0.1:8787/api/events?since=$(( $(date +%s) - 3600 ))"
ctrlrelay web stop
To open it from another machine on the home network, set web.host in
orchestrator.yaml to this machine’s LAN address (see
web), then run it under
systemd. ctrlrelay install systemd renders ctrlrelay-web.service with the
other two units. It refuses to overwrite units that already exist, so on a
host where the bridge and poller are installed, render into a scratch
directory and copy only the web unit:
ctrlrelay install systemd --workdir ~/Projects/ctrlrelay --target-dir ~/ctrlrelay-units
cp ~/ctrlrelay-units/ctrlrelay-web.service ~/.config/systemd/user/
rm -r ~/ctrlrelay-units # the other two carry the chat token
systemctl --user daemon-reload
systemctl --user enable --now ctrlrelay-web.service
journalctl --user -u ctrlrelay-web -n 20
tail -n 20 ~/.ctrlrelay/logs/web.log
The first line in web.log is a web.started event naming the address it
bound. If web.host is 0.0.0.0 it also logs a warning: that listens on
every interface, which is rarely what you want.
With web.host set to a LAN address the page also answers on
127.0.0.1:8787, so a browser or health check on the machine itself is not
refused, and web.log has a second web.started line for it. The LAN side
still listens on web.host and nothing else. If that loopback port is taken
by something else, web.loopback_unavailable is logged and the LAN page
starts anyway.
Every 30 seconds the server checks that web.host is still one of this
machine’s addresses. If it is not (a new DHCP lease, a router reboot,
another network), it logs web.address_gone to web.log, prints why to
web.error.log and exits 1. systemd (or launchd’s KeepAlive) restarts it;
on the old address that start fails with Cannot listen on ...: Cannot
assign requested address in web.error.log, again every few seconds, until
the address comes back or you set web.host to the new one. A check that
cannot be made (a web.host name that does not resolve) is logged once as
web.address_unchecked and the page keeps running.
The page answers only to names it knows: localhost, 127.0.0.1, [::1],
web.host, and this machine’s own hostname (plus .local). A request whose
Host header carries any other name gets 421 misdirected request, and the
refused name is logged once as web.misdirected_host. So if you open it from
another machine by a name that is none of those, for example a DNS name your
router hands out, add that name to web.allowed_hosts and restart the unit:
web:
host: "192.168.1.20"
allowed_hosts: ["mybox.lan"]
grep misdirected_host ~/.ctrlrelay/logs/web.log shows any name that was
refused, which is the quickest way to find the one to add.
Several machines on one account
Two or more machines can run ctrlrelay against the same repositories under the same GitHub account. Each machine’s seen-list and run lock are its own, so the claim comment on the issue is what keeps two machines off one issue.
The claim. Before a dev or task run starts, the machine posts a comment naming itself:
Taken by ctrlrelay on **node-a**. It is working on this issue; ...
<!-- ctrlrelay:claimed node=node-a id=3f9c0a1b2c4d state=working run=5e1d02aa seen=... -->
node is the config’s node_id (the hostname by default). Give every
machine a different one. run names the process that last wrote the claim
and seen when, by its clock: a machine warns when a claim under its own
name is refreshed by another process (two machines sharing a node_id). Only comments by the account the relay runs as
count; a marker pasted by anyone else is ignored.
The race check. Two machines can see a new issue in the same poll. Each
posts its claim, waits claims.race_wait_seconds, and reads the comments
again. The live claim with the lowest comment id wins: GitHub numbers the
comments, so no machine’s clock decides. The loser posts “Backing off:
ctrlrelay on node-a has this issue” and drops it.
Skipping. A machine that finds another machine’s live claim does not
start the run, and logs claims.skipped with the owner. It looks before
announcing the issue in chat, so it stays quiet about issues already taken;
only a machine that loses the race check itself has announced the issue
first.
The stale rule. While a run works, its machine edits the claim every
claims.refresh_seconds, which moves the comment’s updated_at. A
working claim not refreshed for claims.stale_after_seconds is stale:
that machine is treated as stopped, and another machine may take the issue
(its claim names the claim it replaced, and a replaced claim does not count
while the claim that replaced it stands, however it is edited). A skipping machine looks again at
exactly the time the claim could go stale, not on every poll. If a machine
whose claim was taken over is in fact still running, it stops refreshing
its claim and logs claims.taken_over; its run is not stopped.
When the run ends:
- finished, or waiting for an answer: the claim is marked
held. A held claim never goes stale, so a long wait for an answer never hands the issue to another machine. If marking it held fails three times the machine logsclaims.hold_failedand keeps trying everyclaims.refresh_secondsin the background, until the stale time counted from its last good write. If that also fails, or the machine stops, the claim goes stale like a working one. - failed: the failure comment releases that one claim
(
<!-- ctrlrelay:released id=... -->), and the issue is free again. One exception: a run that was waiting for an answer and fails after it is resumed posts no release (the resume path never posted a failure comment), so its claim stays held. Delete the claim comment to free that issue.
A claim from a version before this (<!-- ctrlrelay:claimed --> with no
machine name) counts as taken by unknown and never goes stale.
If the race check cannot see the claim it just posted, the machine does not assume it won: it releases its claim and the run fails before it starts, to be retried on a later poll.
To free an issue by hand, delete the claim comment. Other machines look at a
held issue again every claims.stale_after_seconds, and take it then.
| setting | default | |
|---|---|---|
claims.race_wait_seconds |
5 |
wait between posting a claim and reading it back |
claims.refresh_seconds |
300 |
how often a working machine refreshes its claim |
claims.stale_after_seconds |
1800 |
a working claim older than this is stale; must be at least three refreshes |
Limits to know:
- The stale time compares GitHub’s
updated_atwith this machine’s clock. Keep clocks in sync (NTP); half an hour is far above normal drift. - A machine that loses GitHub for longer than the stale time while its run
keeps working can have the issue taken over. Raise
claims.stale_after_secondsif that is likely. - Mixed versions: a machine on a version before this one reads every failure comment as ending every claim, including another machine’s, and posts claims without a machine name. Upgrade all machines together.
Chat with several machines
Every message a machine posts starts with its name: [node-a] ❓ Blocked on
o/r#7: .... Each machine acts only on replies to its own posts. A reply to
a post another machine made is left to that machine, with nothing said in the
channel; the machine reads the replied-to post to tell (on Telegram, from the
reply itself). On a shared chat, a reply to a post that cannot be read is left
alone, and so is a reply to a post with no machine tag (a person’s post, or a
question posted before this version): answer those by naming the machine,
node-a: ....
Mattermost: machines can share one bot and channel. Set
transport.mattermost.shared: true on each. A message that is not a reply
then reaches a machine only when it names it, node-a: pin or @node-a pin;
a plain message is left alone by every machine, since none could tell it was
meant for it. With one machine, leave shared off and plain answers work as
before.
Telegram: each machine needs its own bot. Telegram lets one process read a bot’s messages at a time, and when a second one starts reading, it is the one already running that Telegram cuts off. There is no way to see the other reader before that happens. So the machine that loses the bot says so once, in its log and in the chat (“Stopped reading replies on this machine: another process started reading this Telegram bot … each machine needs its own bot”), and stops reading, rather than fighting for it every few seconds.
Only the bot’s own posts count as a machine’s: a person typing [node-b] ...
in the channel does not speak for node-b.
Scheduled jobs
The poller daemon also hosts an in-process cron (APScheduler). The jobs run
inside the same asyncio loop as the issue poll, so they inherit the poller’s
supervision (launchd KeepAlive on macOS, systemd Restart=always on Linux)
without needing a separate .timer unit.
| Job | Default schedule | What it does |
|---|---|---|
secops |
0 6 * * * (6am daily) |
Runs the secops pipeline across every configured repo — equivalent to ctrlrelay run secops. |
Schedules are configured under schedules: in orchestrator.yaml, using
standard 5-field cron expressions. The top-level timezone: controls how
they’re interpreted:
timezone: "America/Santiago"
schedules:
secops_cron: "0 6 * * *" # daily 6am; use "0 6 * * 1" for weekly (Mondays)
With several machines on one account
Every machine runs the same schedule. Each job is either split, so it runs once across the machines, or belongs to the machine by design:
| Job | Runs | Where its claim lives |
|---|---|---|
secops (security sweep) |
once per run window across machines, per repo | a git ref in that repo, refs/ctrlrelay/claims/secops/<window> |
| merge watcher and post-merge steps | on the machine that ran the issue | none needed: only that machine watches its own pull requests |
pending_resume_sweeper (every minute) |
on each machine | none: it drains this machine’s own unanswered questions |
question_expiry_sweeper (hourly) |
on each machine | none: it retires this machine’s own questions |
usage_probe (every 5 minutes) |
on each machine | none: each machine reads its own subscription |
github_budget |
on each machine | none: a free read of the shared budget |
personalization_auto_pull |
on each machine | none: each machine syncs its own copy |
How the sweep is split. Before sweeping a repo, a machine creates the
ref refs/ctrlrelay/claims/secops/<window> in it, pointing at a small blob
that holds its node_id (in the same safe form the issue claims use). Creating a ref on GitHub is all or nothing: the
first machine creates it and sweeps; every other machine gets “already
exists”, reads the owner’s name from the blob, logs jobs.skipped, and skips
that repo. The ref is not a branch and GitHub shows it nowhere in its
interface.
<window> is the minute the sweep was due, for example
20261101T0600Z, as the scheduler planned that run, not when it started:
machines a few seconds apart, or one started late, agree on it. A machine
that was asleep runs a missed sweep once, up to an hour late, for the latest
time it was due, and names that window. A machine that dies holding a window loses only
that window: the next one is a new ref. Refs older than 30 days are deleted by
the next machine that wins a claim.
A machine claims a repo only once it holds that repo’s local lock, so it
never wins a claim and then finds the repo busy on its own side. (A machine
that dies right after claiming still leaves that window unswept; the next
window runs.) If a machine cannot create the ref
(GitHub down, no write access), it skips
the repo for that window rather than risk a second sweep, and says so in the
sweep results; the next window tries again. A sweep started by hand
(ctrlrelay run secops) is not claimed and always runs.
An invalid cron expression fails at config load time rather than silently disabling the job. If the machine is asleep at the fire time, the job runs when it wakes — up to one hour late; older misfires are coalesced into a single run.
On poller stop, the scheduler waits up to 150 seconds for an in-flight
job (e.g. git worktree prune cleanup at the end of a secops sweep) to
finish before letting the asyncio loop close. This requires your
supervisor’s stop timeout to be at least that generous — the example
unit files above set ExitTimeOut=180 (launchd) and
TimeoutStopSec=180 (systemd). Without those, the platform default
(launchd 20s, systemd 90s) SIGKILLs the daemon before cleanup finishes
and leaves stale git worktree admin state behind. If you’re upgrading
from an older plist/unit that didn’t set these, add them and reload.
Issues blocked by other issues
When work has an order (B changes the same files as A, so B waits for A),
say so on GitHub with an issue dependency: on B, Relationships → Mark as
blocked by → A. The blocker may be in another repository. Before the
poller claims an issue it reads that issue’s blockers
(GET /repos/{owner}/{repo}/issues/{n}/dependencies/blocked_by), and:
- None is open: the run starts as usual.
- One or more is open: the issue is not claimed and not started. The
log has
poll.issue.blockedwithblocked_by(the open ones,#Nin the same repository,owner/repo#Nin another), the status page lists it under Skipped asblocked by …, and the chat is told once per issue, not every poll. Every later poll asks again, so it starts on the first poll after the last blocker closes, with nothing to do by hand. - The blockers could not be read (an API error, the GitHub API budget
used up, a 404 from a GitHub Enterprise Server without the dependencies
API): treated like an open blocker, never as “no blockers”. The log has
poll.issue.blockers_unknownwith the error on every failed poll, and the next poll tries again. A single failure is usually a blip, so the status page keeps showing the blockers last read and the chat is not told; after 3 failed polls in a row the page sayscould not read its blockersand the chat is told once per issue. If the blockers can then be read and are the same as before, nothing more is said.
Only the dependency link counts. A line such as “Starts after: #A” in the issue body is not read, so add the link as well.
A held issue is kept out of seen_issues (that is what brings it back)
and recorded in blocked in the same poller_state.json, so a restart
neither loses it nor repeats the chat line. Its self-assignment is
checked again every poll (it may have been unassigned and assigned again
by someone else), but a check that fails leaves a held issue for the next
poll instead of marking it handled for good.
Each poll costs a held issue one extra REST call (core budget) for its
blockers, plus the re-read every issue gets just before it starts. A blocked issue that is closed or loses its trigger is dropped
from blocked when the poller next sees it.
Stuck runs
A run that goes in circles is stopped early instead of running until
agent.default_timeout_seconds.
- While the agent works, ctrlrelay reads its live transcript every 30
seconds. If the agent runs the same command with the same input 5 times in
a row and gets the same result back each time, or gets the same error
back from the same command 5 times in a row with no success of that
command in between (edits in between do not count), the agent is stopped.
A command polled while its output moves on is waiting, not a loop; a poll
that gets the identical output back 5 times in a row is still treated as
one. Only the
claudebackend writes a transcript ctrlrelay knows how to read; other backends are not watched. - Between CI fix rounds, each failed job’s log is sorted on its own into
one of three kinds. The round is out of budget if any job is,
infrastructure only if every job is, and a code failure otherwise; a
job whose log could not be fetched counts as a code failure, and a check
that is not an Actions job, or one past the first 3 jobs, is not sorted
at all:
- flaky infrastructure (the runner was lost, the network failed, a registry answered 502/503, and no failing test is named): the failed GitHub Actions jobs are re-run once, without asking the agent. If they fail on infrastructure again, the run stops. A fix round in between resets this, so a new flake later gets its own re-run.
- out of budget (the job hit its maximum execution time or a spending limit): the run stops at once. A re-run would spend the same budget.
- code failure (anything else, including every log that names failing tests): sent to the agent as a fix round, as before. If the same tests fail in 3 CI rounds in a row (the first failure and two fixes that changed nothing), the run stops instead of spending the next round.
A stopped run is recorded as failed with a stuck_reason in the
sessions table, its summary starts with Stuck:, the issue comment gives
the reason, and chat gets 🔁 Stuck on #<n>, stopped: <reason>. The status
page’s numbers count these runs in stuck.
Run budgets
Each run can have a token budget (#223), set in
agent.budget. There is no
limit until one is set.
- What is counted. Every 30 seconds ctrlrelay adds up the usage in the agent’s live transcript, sub-agents included: input tokens, cache writes and output tokens. Cache reads are recorded but not counted. One API message written as several rows is counted once.
- Per session, not per spawn. A dev or task run spawns the agent more than once (the first run, resumes after a question, CI fix rounds). They all count against the same budget, because each spawn starts from what the session row already holds.
- Soft limit. The first time the total passes it, chat gets one line
(
💰 #<n> (<repo>) has used <N> tokens, past its soft limit ...) and the log getsbudget.soft_limit. Once per session. - Hard limit. The agent is stopped through the same path as a stuck run.
The run is recorded as
failedwithstuck_reasonset tobudget: used <N> tokens, hard limit <M>, its summary starts withStopped:, the issue comment gives the reason, and chat gets💸 Over budget on #<n>, stopped: <reason>. A checkpoint the agent wrote just before the stop is not used: the work is not handed over as ready. When no pull request exists yet, the unfinished work is not saved to a branch (only a timeout does that). The status page’s numbers count the run infailed, not instuck. - A stop after the PR was opened (over budget in a CI fix round, or
stuck there): the PR is put back to draft and kept with the work so far,
no merge watcher is started for it, and chat and the issue comment say
PR #<n> was kept as a draft with the work so far.If GitHub refuses the draft, a “Do not merge” comment goes on the PR instead, and the messages say so. - A floor, not an exact bill. Only API calls that write an assistant row to the transcript are counted; calls the CLI makes on its own without one (for example, short helper calls) are not. The count can be a little under what the subscription was charged.
- Unknown is not zero. A transcript row whose usage cannot be read adds
nothing and is counted as unreadable; the log gets
budget.usage_unreadableand the page shows the total as unknown. What could be read is still a true floor, so a run is stopped only when that floor reaches the hard limit. An unreadable row on its own never stops a run. - What is stored.
sessions.usage_tokensis the counted total (a floor when rows were unreadable) andsessions.usage_detailholds the breakdown, the unreadable row count, the cost and the limits applied, as JSON.cost_usdis what the agent CLI’s finalresultevent reported (total_cost_usd), summed over the session’s spawns; it is information only. - Backends. Only the
claudebackend writes a transcript ctrlrelay can read. A run onopencodeis not measured; with a limit set, the log getsdispatcher.budget.unmeasuredand nothing is enforced.
A run whose process died
A dev run saves each step it finishes: plan, code, publish and
verify (#231). Each is a row in the run_steps table of the state DB,
with the run branch’s commit at that moment and what the later steps need
(the plan, the agent’s result, the pull request). No commit is added to the
branch for it.
When the relay process dies in the middle of a run (out of memory, a restart, a power cut), the poller settles that run when it starts again and hands the issue back. The next run for the issue then continues after the last saved step:
- it checks that the branch is still at the saved commit, and starts from the beginning if not (or stops with the reason, when the dead run had already opened its pull request);
- it does not run those steps again: no new agent session for a saved code step, no second push or pull request for a saved publish step;
- it comments on the issue which run it continues and from where;
proofandreviewalways run again, because they read the pull request as it is now.
A dead run that had opened its pull request is handed back too, and only that pull request is reused. A run that failed for its own reasons is not continued; the next run starts from the beginning, as before.
To start again from a past step by hand, for example to redo the code with the same plan:
ctrlrelay run dev --issue 42 --repo your-org/your-repo --from-step code
--from-step takes plan, code or publish, uses the newest run of the
issue that saved steps, and moves the branch back to where it was before
that step. The old head is kept at refs/ctrlrelay/before-retry/<run id>
in the relay’s clone. It refuses a branch behind an open pull request, and a
branch that is on GitHub is synced back to GitHub’s head; delete the remote
branch first in that case.
The log events are dev.step.saved, dev.resume.restored and
dev.resume.head_moved.
Running several agents at once
Set agent.max_parallel to let one machine run more than one agent (#210):
agent:
max_parallel: 3
- One cap over everything. New issues from the poll, runs resumed after you answer a question, and the scheduled security sweep all take a slot from the same pool. A run takes its slot before the repo lock, and gives it back when it ends.
- New issues side by side. Polled issues wait in a backlog, in priority order, and start as slots free up. Two issues of the same repo never run together: the second waits, and an issue of another repo behind it goes first. The loop keeps polling on its interval whatever is running, so one slow issue never holds up new ones.
- A run holds its slot for its whole life, including the wait for CI
and the fix rounds, and while it waits in the run for your answer. Once
it gives up waiting and saves the question, the slot is free again, and
the resume takes one when your answer comes. Two pull requests waiting on
slow CI can fill
max_parallel: 2. - A hold placed while a run waited for its slot is respected. The resume
sweeper and the security sweep check again once they have a slot, so a
self-updatethat started meanwhile is not raced. - On shutdown no new run starts in a slot a stopping run frees, and issues polled but not started are offered again after the restart.
- Unset keeps the old behaviour: new issues one at a time, and nothing limits the total.
Each run uses its own agent session, worktree and memory. Size the number to the machine and to your plan’s usage limits. Start with 2.
The status page shows slots busy/max, and slots N (no cap) when it is
unset. The log event poll.batch.parallel names each batch, the limit and
how many slots were busy.
When to restart
| You changed… | Restart… |
|---|---|
transport.* (type, token env var, chat or channel ID, socket path) |
bridge and poller |
repos[] (added/removed/renamed) |
poller |
agent.* or repos[].agent_type (backend, binary path, timeout) |
poller |
paths.* |
both |
Anything in dashboard.* |
poller |
schedules.* (cron expressions) |
poller |
| Bot token env var | bridge and poller |
After a ctrlrelay package upgrade (uv pip install -e .), restart both.
For a node installed with uv tool, ctrlrelay self-update does the
install and the restarts for you; see below.
Updating a node
ctrlrelay self-update # the latest version on PyPI
ctrlrelay self-update --version 0.23.0 # a given version
It only installs a version already published on PyPI; publishing stays a human decision. In order:
- Reads
https://pypi.org/pypi/ctrlrelay/<version>/jsonand checks that this node’s status page answers on127.0.0.1:<web.port>; without it the update could not be checked, so it refuses. It also refuses when there is no poller or web unit installed, or ctrlrelay is not auv toolinstall. Nothing has changed yet. - Downloads the wheel into
wheels/beside the state database and checks its sha256 against that JSON. A mismatch deletes the file and installs nothing. The wheel is installed by path rather than by name because a node’s package index has served a stale list for more than ten minutes after a release. The file is kept, since uv’s receipt records the path. - Holds new work: it writes
intake-hold.jsonbeside the state database. While it is there the poller leaves new issues for the next poll and skips the scheduled sweep and resumed questions, the same way the usage pause does; runs already going finish. The poller answers inintake-hold.ack.jsonwith whether anything is still in flight. The command waits, up to--wait-minutes(default 60), until the poller says nothing is in flight and the state database has norunningsession. Checking the database alone is not enough: an issue can pass the poller’s checks a few seconds before its session row is written, and a restart in that gap cancels it. uv tool install --force <wheel>, then restarts every installed unit:systemctl --user restart ctrlrelay-{bridge,poller,web}.serviceon Linux,launchctl kickstart -k gui/<uid>/<label>for each*.ctrlrelay-{bridge,poller,web}.plistin~/Library/LaunchAgentson macOS (the label is the file’s name, whatever--label-prefixit was installed with).- Waits up to two minutes for
/api/statusto report the new version, and up to two and a half for a new poller process to report it in its heartbeat (heartbeat.jsonnow carries the poller’spid,versionandstarted_at) and still be the one reporting at least one heartbeat interval (30 s) after it started. The status page’s version is the web server’s own, and a restart command succeeds even when the unit then crash-loops, so neither alone shows the poller is up.
If any step after the install fails, or the command is interrupted (Ctrl-C, SIGTERM) once the install has begun, the old version is installed and restarted the same way, and checked; a poller from before this command writes no version in its heartbeat, so for the old version a fresh heartbeat from a new process is enough. Chat (when a transport is configured) gets one line either way: the update, or the step that failed and whether the old version is running again. The command exits non-zero on any failure. The hold is removed on every way out, including Ctrl-C and SIGTERM; one left by a process that was killed outright is ignored once that process is gone or the hold’s expiry (the wait plus 30 minutes) has passed.
A poller from before this command does not know the hold and never
acknowledges it, so the command stops after a minute without changing
anything. For that one update, --no-ack goes on the state database alone,
with the gap described in step 3. A poller that is not running (its
heartbeat is older than fleet.silent_after_seconds) is not waited on.
A scheduled security sweep whose time comes while the hold is in place is skipped until its next time, not delayed.
After merging a PR that the dev pipeline opened, no manual restart is required — the PR is referenced from the open branch in your bare repo, so cleanup is handled by GitHub on merge. The worktree was already removed when the session finished.
Logs
By default, both daemons log to the paths configured in your launchd /
systemd unit files. The recommended location is ~/.ctrlrelay/logs/.
Tail them:
tail -f ~/.ctrlrelay/logs/poller.log
tail -f ~/.ctrlrelay/logs/bridge.log
tail -f ~/.ctrlrelay/logs/poller.error.log
tail -f ~/.ctrlrelay/logs/web.log # if the status page is installed
The poller prints one line per detected issue and one line per pipeline outcome. The bridge prints connection events and chat API errors (Telegram Bot API or Mattermost REST/WebSocket).
Tracing one agent session
Log lines are newline-delimited JSON, and the dispatcher — the module that spawns and supervises every agent subprocess — tags each of its events with the session id, so one grep reconstructs a whole run:
grep dev-owner-repo-42- ~/.ctrlrelay/logs/poller.log | jq -c '{ts, level, event}'
The events it emits:
| Event | Level | When |
|---|---|---|
dispatcher.session.start |
INFO | Subprocess about to launch. Carries binary, timeout, resume, prompt_hash, prompt_len. |
dispatcher.session.finished |
INFO | Subprocess exited. Carries exit_code, checkpoint_status, elapsed_ms. |
dispatcher.session.subprocess_failed |
ERROR | Non-zero exit, with stderr_tail. |
dispatcher.session.timeout |
ERROR | The timeout seconds elapsed and the child was killed. |
dispatcher.session.spawn_failed |
ERROR | The binary could not be exec’d at all. |
dispatcher.session.cancelled |
INFO | Daemon shutdown killed the child mid-run. |
dispatcher.checkpoint.missing |
WARNING | The session ended without writing a checkpoint. |
dispatcher.checkpoint.read_failed |
ERROR | A checkpoint file existed but would not parse. |
dispatcher.binary.fallback |
WARNING | The agent binary (claude or opencode) was not on PATH; a well-known path was used instead. Expect this when the unit’s PATH is minimal. |
dispatcher.binary.unresolved |
WARNING | Not on PATH and not at any known path — sessions are about to start failing to spawn. |
Prompts and agent output never appear in plaintext: only a short SHA-256 prefix and a length are logged.
Blocked-question events
When a session blocks on a question, the transport and the bridge each emit their own record:
| Event | When |
|---|---|
dev.question.posted |
The question reached the operator. The transport emits it on the bridge’s ACK; the bridge emits it once the chat server (Telegram or Mattermost) has accepted the message, and adds its post_id. |
dev.question.post_failed |
Delivery failed. Carries reason (send_failed, bridge_error, or the exception class on the bridge side) and a truncated error. |
dev.answer.received |
The operator’s reply came back. |
Neither the question nor the answer is written to the log. Each event carries
question_hash / question_length (or answer_hash / answer_length)
instead — the hash is stable, so it still joins the poller’s record to the
bridge’s record for the same question:
grep 44509098fa6eeaf3 ~/.ctrlrelay/logs/*.log | jq -c '{ts, logger, event}'
This matters because these files and the systemd journal have no retention policy tuned for human-entered content, and operator replies are free text.
Inspecting state
ctrlrelay status
The fastest way to see what’s happening:
ctrlrelay status
Shows held repo locks and the 5 most recent sessions with their statuses
(running, done, blocked, failed).
State DB schema
The state DB is plain SQLite at paths.state_db. Open it with sqlite3 for
ad-hoc queries:
sqlite3 ~/.ctrlrelay/state.db
sqlite> .tables
sqlite> .schema sessions
sqlite> SELECT id, pipeline, repo, status, started_at FROM sessions ORDER BY started_at DESC LIMIT 20;
Tables (defined in
src/ctrlrelay/core/state.py):
sessions— every pipeline run. Columns:id,pipeline,repo,issue_number,worktree_path,status,blocked_question,started_at,ended_at,claude_exit_code(the worker’s exit code, whichever backend),summary,agent_session_id,output_tail.repo_locks— currently held per-repo locks. PK onrepo.github_cursor— last-seen issue/PR timestamps per repo (used by the poller to bound API calls).pending_resumes— one row per BLOCKED session’s question, whichever chat it went to: open until answered or expired, then kept forquestion_retention_days. (telegram_pendingstill exists in the schema but is unused.)pr_watches— PRs the poller is watching for merge; survives restarts.automation_decisions— operator decisions onask-policy automation prompts.
Worktrees on disk
Per-session worktrees live under paths.worktrees, named
<owner>-<repo>-<session-id>. ctrlrelay removes them on DONE or terminal
FAILED; BLOCKED sessions keep their worktree so a manual operator can
inspect / resume.
If something goes wrong and you need to reclaim a stuck worktree:
git -C ~/.ctrlrelay/repos/your-org-your-app.git worktree list
git -C ~/.ctrlrelay/repos/your-org-your-app.git worktree remove --force /path/to/orphan
Poller seen-set
The poller’s seen-issue state is a JSON file at
<state_db_dir>/poller_state.json. If you want to force the poller to
re-process a specific issue, delete its number from seen_issues[<repo>] (or
delete the file entirely to reseed from scratch on next start).
A run that fails before it is handed to the agent pipeline (reading
the issue, posting the claim comment, cloning, cutting the worktree) does
not use the issue up. The issue is un-marked and tried again on the next poll, up to 3 runs
in all, with a chat message each time when a chat channel is connected. After the third it stays in
seen_issues and the message says so; fix the cause, then delete the
number as above. The count is kept in pre_dispatch_failures in the same
file, so a restart does not reset it. Each retry posts a fresh claim
comment and, when it fails, a fresh “nothing is working on this” comment
on the issue.
A run whose agent finished but whose branch the relay could not
publish is retried the same way, for the same reason: the issue is
already in seen_issues, so without this nothing would ever bring it
back. Up to 3 runs in all, counted in publish_failures in the same
file, and the retraction comment on the issue says the relay will
return rather than asking you to re-assign it. After the third run the
issue stays in seen_issues and the relay says so on the issue —
the commits are still on the run branch in the local checkout (on
GitHub too, if only opening the pull request failed), and the fix is to
clear the cause and delete the number as above. If that comment cannot
be posted, the chat line says so.
A push the relay refuses (a protected or default branch, a base it
did not expect, somebody else’s commits on the branch) is not retried
at all: the relay reaches the same refusal every time, so another run
would spend a whole agent to arrive at it. The issue stays in
seen_issues and the comment names what to do.
Re-assigning an issue, or re-applying its automation label, does not
make the poller look at it again. The skip decision reads the issue
number only — no assignee and no labels — so the number has to leave
seen_issues (and the relay has to be restarted, since the running
poller holds the set in memory and rewrites the file from it).
An issue held back by an open GitHub blocker (see Issues blocked by
other issues) is never in seen_issues
while it waits; it is listed in blocked[<repo>] as {"open": [...],
"failures": n}: the open blockers as last read (null if never read)
and how many lookups have failed in a row since. Nothing needs deleting: it starts on
its own once the blockers close.
A repo whose history was replaced
If the local copy’s default branch can no longer fast-forward to GitHub’s
(the repo was replaced by a new one with the same name, or its default
branch was force-pushed), ctrlrelay moves the copy to
~/.ctrlrelay/repos/<owner>-<repo>.git.replaced-<UTC time>, clones again,
and says so in chat. The old copy is kept as it was after the failed
fetch: remove it once you do not need it. This is refused while any worktree still uses the copy (a
blocked session keeps its worktree for the resume); the run then fails
with default branch diverged ... not re-cloning while N worktree(s) still
use it, and you can finish or clean up that session first.