Operations

Once your config works interactively, you usually want the bridge and poller running unattended. This page covers macOS launchd, Linux systemd, what to restart after a config change, and how to inspect runtime state.

Long-lived services

ctrlrelay generates platform-appropriate service unit files for you with ctrlrelay install. The two daemons it manages:

Service Command Why you want it running
Bridge ctrlrelay bridge start Delivers BLOCKED questions to your chat (Telegram or Mattermost) and routes answers back.
Poller ctrlrelay poller start --interval 300 Watches GitHub for newly assigned issues and runs the dev pipeline.
Status page (optional) ctrlrelay web start Read-only page and JSON API of what ctrlrelay is doing. See Status page.

Both should restart on failure and on login.

ctrlrelay bridge start and ctrlrelay poller start daemonize by default (fork, write a PID file, return to the shell). Under a process supervisor (launchd, systemd) pass --foreground so the supervisor can track the PID and restart on failure — the templates below already do this.

macOS — launchd

Recommended: let ctrlrelay install launchd render the plists for you. It substitutes ${USER}, ${HOME}, the absolute ctrlrelay path, your working directory, and (when exported) your chat token into the in-package templates and writes them to ~/Library/LaunchAgents/.

The token goes into the plist under the variable your transport names in transport.<type>.bot_token_env — CTRLRELAY_TELEGRAM_TOKEN for Telegram, CTRLRELAY_MATTERMOST_TOKEN for Mattermost, or whatever you renamed that setting to. The command reads your orchestrator.yaml to find out which; pass --config if it is somewhere the usual search does not look. Export that variable before running, or the plist keeps a placeholder and the command says which variable to export.

export CTRLRELAY_MATTERMOST_TOKEN=your-bot-token   # or _TELEGRAM_, per your config
ctrlrelay install launchd \
  --workdir ~/Projects/ctrlrelay \
  --label-prefix com.yourname \
  --poller-interval 300

# Then load the agents:
mkdir -p ~/.ctrlrelay/logs
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.yourname.ctrlrelay-bridge.plist
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.yourname.ctrlrelay-poller.plist

Pick a reverse-DNS --label-prefix you own (e.g. com.yourname); it flows into both the filename and the <Label> value so launchctl list output is unambiguous. Pass --dry-run first to see what would be written; pass --force to overwrite a previously installed plist.

If you’d rather hand-write the plists, here are the full templates the installer uses. They show CTRLRELAY_TELEGRAM_TOKEN; on a Mattermost config use the variable your transport.mattermost.bot_token_env names (CTRLRELAY_MATTERMOST_TOKEN by default).

~/Library/LaunchAgents/com.example.ctrlrelay-bridge.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.example.ctrlrelay-bridge</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/homebrew/bin/ctrlrelay</string>
        <string>bridge</string>
        <string>start</string>
        <string>--foreground</string>
    </array>
    <key>EnvironmentVariables</key>
    <dict>
        <key>CTRLRELAY_TELEGRAM_TOKEN</key>
        <string>your-bot-token-here</string>
        <key>PATH</key>
        <string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>
    </dict>
    <key>WorkingDirectory</key>
    <string>/Users/YOU/Projects/ctrlrelay</string>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/Users/YOU/.ctrlrelay/logs/bridge.log</string>
    <key>StandardErrorPath</key>
    <string>/Users/YOU/.ctrlrelay/logs/bridge.error.log</string>
</dict>
</plist>

~/Library/LaunchAgents/com.example.ctrlrelay-poller.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.example.ctrlrelay-poller</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/homebrew/bin/ctrlrelay</string>
        <string>poller</string>
        <string>start</string>
        <string>--foreground</string>
        <string>--interval</string>
        <string>300</string>
    </array>
    <key>EnvironmentVariables</key>
    <dict>
        <key>CTRLRELAY_TELEGRAM_TOKEN</key>
        <string>your-bot-token-here</string>
        <key>PATH</key>
        <string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>
    </dict>
    <key>WorkingDirectory</key>
    <string>/Users/YOU/Projects/ctrlrelay</string>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <!-- Give the in-process scheduler time to drain a running secops
         sweep on stop (worktree cleanup can take ~120s). launchd's
         default ExitTimeOut is 20s, which would SIGKILL mid-cleanup. -->
    <key>ExitTimeOut</key>
    <integer>180</integer>
    <key>StandardOutPath</key>
    <string>/Users/YOU/.ctrlrelay/logs/poller.log</string>
    <key>StandardErrorPath</key>
    <string>/Users/YOU/.ctrlrelay/logs/poller.error.log</string>
</dict>
</plist>

Whether you used ctrlrelay install launchd or hand-wrote the plists, manage the agents with:

launchctl list | grep ctrlrelay          # check loaded
launchctl bootout gui/$(id -u)/com.example.ctrlrelay-poller    # stop
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.example.ctrlrelay-poller.plist  # start

Linux — systemd

Recommended: ctrlrelay install systemd renders the user unit files for you (no root needed) and writes them to ~/.config/systemd/user/.

As on macOS, the bridge and poller units carry the token under the variable transport.<type>.bot_token_env names, read from your orchestrator.yaml (--config to point at a different one). Export that variable before running.

export CTRLRELAY_MATTERMOST_TOKEN=your-bot-token   # or _TELEGRAM_, per your config
ctrlrelay install systemd \
  --workdir ~/Projects/ctrlrelay \
  --poller-interval 300

# Then enable and start:
mkdir -p ~/.ctrlrelay/logs
systemctl --user daemon-reload
systemctl --user enable --now ctrlrelay-bridge.service
systemctl --user enable --now ctrlrelay-poller.service

To survive logout (start at boot, stop at shutdown):

sudo loginctl enable-linger "$USER"

If you’d rather hand-write the units, here are the full templates the installer uses. Save under ~/.config/systemd/user/.

~/.config/systemd/user/ctrlrelay-bridge.service:

[Unit]
Description=ctrlrelay chat bridge (Telegram or Mattermost)
After=network-online.target

[Service]
Type=simple
WorkingDirectory=%h/Projects/ctrlrelay
# The variable your transport names in bot_token_env — use
# CTRLRELAY_MATTERMOST_TOKEN on a Mattermost config.
Environment=CTRLRELAY_TELEGRAM_TOKEN=your-bot-token-here
ExecStart=%h/.local/bin/ctrlrelay bridge start --foreground
Restart=always
RestartSec=5
StandardOutput=append:%h/.ctrlrelay/logs/bridge.log
StandardError=append:%h/.ctrlrelay/logs/bridge.error.log

[Install]
WantedBy=default.target

~/.config/systemd/user/ctrlrelay-poller.service:

[Unit]
Description=ctrlrelay issue poller
After=network-online.target ctrlrelay-bridge.service

[Service]
Type=simple
WorkingDirectory=%h/Projects/ctrlrelay
# The variable your transport names in bot_token_env — use
# CTRLRELAY_MATTERMOST_TOKEN on a Mattermost config.
Environment=CTRLRELAY_TELEGRAM_TOKEN=your-bot-token-here
ExecStart=%h/.local/bin/ctrlrelay poller start --foreground --interval 300
Restart=always
RestartSec=5
# Give the in-process scheduler time to drain a running secops sweep on
# stop (worktree cleanup can take ~120s). systemd's default
# TimeoutStopSec is 90s, which would SIGKILL the poller mid-cleanup and
# leak git worktree admin state.
TimeoutStopSec=180
StandardOutput=append:%h/.ctrlrelay/logs/poller.log
StandardError=append:%h/.ctrlrelay/logs/poller.error.log

[Install]
WantedBy=default.target

Once written, enable and start with the same systemctl --user enable --now commands shown earlier in the section. Check status / logs:

systemctl --user status ctrlrelay-poller
journalctl --user -u ctrlrelay-poller -f

Status page

ctrlrelay web start serves a read-only page and JSON API showing what each agent is doing (phase, current activity, last command), what is waiting and why, the questions that need an answer, and a feed of what ctrlrelay did and said. It is a third process next to the bridge and the poller, so a slow or broken page can never stall them.

On a running card, phase (the bar) is the furthest step the session has reached, from the agent’s transcript plus ctrlrelay’s own CI, review and PR events, so it never moves back. Current activity is the most recent step: an open CI wait or review if it started after the agent’s latest step, else that step. It can be earlier than the bar, for example a read after the PR or a fix round after review. The card links the PR once ctrlrelay has recorded its number.

It is read-only: it opens state.db read-only, reads the tail of each agent’s transcript, and asks GitHub (through the gh login ctrlrelay already uses) for the waiting line once a minute. It writes nothing but its PID file (web.pid, next to state.db), and serves GET only; any other method is 405. It has no login and shows repository names, issue titles and the questions agents asked, so never expose it to the internet. Keep it on loopback or on the home network.

Path What
/ The page. Refreshes itself every web.refresh_seconds.
/api/status Everything on the page, as JSON (the contract is in src/ctrlrelay/web/status.py). A section that could not be read is null and errors says why; [] means “read, and empty”. The feed holds at most the newest feed_limit (200) events: feed_total is how many the history window holds (null if unreadable) and feed_truncated is true when some were left out, in which case the page says “showing newest N of M”.
/api/events?since=<epoch seconds> The feed, newest first, at most limit (200) events. Without since, the last web.history_days days. Carries the same feed_total and feed_truncated, plus the limit used.
/api/events?kind=<kind>&day=today One Today counter’s events, for the server’s local day, newest first, at most 200: {kind, events, total, truncated, error}. kind is dispatched, pr_opened, merged, failed, blocked or message_sent. A failed or blocked event also carries reason: exit_code, the run’s summary (up to 500 characters) and output_tail (up to 2,000), both scrubbed of credential shapes again before they are sent, the question asked, and session: "not found" when the run’s row is gone. events is null with error set when they could not be read. A query with kind and anything other than exactly kind and day=today is refused with 400, except that one with since is the feed, as before. A summary or question longer than 500 characters ends in “…”. On the page, clicking a Today counter shows this list under it, read again with every refresh.

The page header shows how long ago it last heard from the server (“updated 12s ago”, ticking every second) and a dot that is green while the last good answer is at most two refresh intervals old and red otherwise. Elapsed times on cards tick between polls too.

Fields added in #232 (all in /api/status):

Field What
url On every running, needs_you, waiting.items, skipped and feed item (and on /api/events events): a link to the GitHub issue, or for a feed event that names a PR by number, the PR. Built only as https://github.com/<owner>/<repo>/issues/<n> or /pull/<n>, from a repo matching owner/name and an integer; null otherwise (a secops run with no issue, a restart). A pr_url an agent reported is never used.
refs On every needs_you item (#241): one {"number", "url"} per distinct #N in the question, in order, at most 20, with url built the same way as url above from the item’s repo and N. A #N right after a letter, digit, _, / or & is not counted. The page turns each listed #N in the question into a link, so a secops question’s PRs are clickable. [] when there are none.
skipped Every open issue assigned to you that the poller will not pick up and is not running, from the same search as the waiting line (no extra GitHub calls): repo, issue_number, title, url, updated/updated_ts and reason, one of repo not in config, excluded label <label>, already handled once (won't retry), missing required label <label> (several joined with “or”: any one admits it), assigned by someone else (not you and not an account in allowed_accounts), blocked by <#N, owner/repo#N> or could not read its blockers; asked again every poll (#310, see Issues blocked by other issues; these two are listed for a label-triggered issue too). null until GitHub has answered once; waiting.status and waiting.last_good_ts say how current it is. The page shows it collapsed, with a count.
server This machine’s health, read on each request: load_1m, load_5m, load_15m, cpu_count, memory (total_bytes, available_bytes, available_pct, low), disks (one each for home and worktrees: name, path, total_bytes, free_bytes, free_pct, low), uptime_s, and unavailable (why any number is null; memory and uptime are null off Linux). low is true below 10% free and the page marks it. The whole section is null, with a line in errors, only if reading it failed outright.

Added in #247:

Field What
node This machine’s own block: name (node_id); usage, the agent subscription’s usage as the last run or probe read it: status, read_at, and for five_hour and seven_day each {used (0 to 1), resets_at}, or "unknown" for a window it could not read; the whole usage is "unknown" before any reading. pause_at is usage.pause_at, and paused is {reason, until} while the machine takes no new issues, else null. The page shows it in the Usage box. node is null, with a line in errors, only if reading it failed outright.

Added in #223, on every running card (see Run budgets):

Field What
tokens_used Tokens the session has used so far, over all its spawns: input + cache writes + output, what the budget counts. null before anything was measured, on a backend that is not measured, or when a transcript row could not be read (then reason gives the number of rows and the floor). Never 0 for unknown.
cache_read_tokens Cache reads, recorded but not counted. null as for tokens_used.
cost_usd The sum of total_cost_usd the agent CLI reported at the end of each spawn. Information only. null until a spawn reported one.
token_soft_limit, token_hard_limit The limits applied to this run. null when no limit applies or nothing was measured yet.

The page shows them in a tokens: row on each running card; the count is in the warning colour once it passes the soft limit.

Added in #237: node.github_api is the account’s GitHub API budget as this machine last read it: read_at, and for core, search and graphql each {remaining, limit, resets_at, state} with state one of ok, low, out, or unknown once its reset time has passed (the numbers are then old), or "unknown" in place of the object for a bucket it could not read; the whole value is "unknown" before any reading. The page shows it in the Usage box; a bucket whose reset time has passed since the last read shows as “refilled at

Added in #249:

Field What
node.version, node.account_label, node.chat This machine’s version, its fleet.account_label, and its chat: {transport, connected} (connected is whether the bridge socket exists), or null with no chat configured.
node.state working, idle, paused-limit (paused for usage), silent (the daemon’s heartbeat is missing or older than fleet.silent_after_seconds), or unknown (the state database could not be read).
node.slots {total, busy}. The daemon works one issue at a time, so total is 1.
node.claims What this machine holds: every running issue and every issue waiting for an answer, each {repo, issue_number, kind, phase, since, url}. null when the state database could not be read.
node.heartbeat_at, node.last_poll_at When the daemon last wrote its heartbeat (every 30 seconds, from its own task, so it moves while a run is going) and last polled GitHub. From heartbeat.json beside the state database.
node.open_questions How many questions are waiting for an answer.
fleet {peers: [...]} with one item per fleet.peers URL: url, name, reachable, checked_at, error, state, node (that peer’s node block), last_known (its last block while it does not answer) and needs_you. null when no peers are configured.

Added in #307:

Field What
node.config_fingerprint {v, repos_hash, profiles, repos, schedules}, v is 2. profiles maps a profile hash to {automation, steps}, each a map of field name to a hash of its value; repos maps each repo name to its profile hash, so repos with the same configuration share one profile; repos_hash is the hash of the sorted, lowercased repo names; schedules maps each schedules field to a hash. Hashes and repo names only, never a value. null if it could not be built. A peer’s fingerprint is unknown unless every hash is 16 lowercase hex characters, repos_hash is the hash of its own repo names, each profile key is the hash of that profile, and each repo names a profile it carries. A field’s hash cannot be checked against its value, which is never sent.
fleet.peers[].drift {status, reason, only_here, only_there, fields, lines}. status is same, differs or unknown (with reason: not read yet, not reachable, the peer sends no fingerprint in this format, or the comparison failed); fields maps a field path such as automation.exclude_labels to the repos it differs on; lines is what the page shows. Leaves out fleet.expected_drift. The peer’s own fingerprint is not repeated in fleet.peers[].node.

Added in #305:

Field What
node.sandbox This machine’s agent.sandbox: "none" (the agent runs with the user’s full access) or "bwrap". In fleet.peers[].node it is null when the peer is on an older version or sent anything else, which means unknown, never off. The page shows sandbox off in the warning colour for "none".

Added in #230: numbers is how this machine’s runs went, for the last 7 and the last 30 days (numbers.windows, each with days, since/since_ts, until/until_ts). The page shows it in the Numbers box, with a table per repo under it. Numbers across machines are not here; they belong to the control plane. Every field is defined in src/ctrlrelay/web/numbers.py.

Field (in overall and in each by_repo item) What
started Runs (rows in the sessions table) started in the window, both ends included. A blocked run resumed after an answer is still one run.
done, failed, blocked, running, other How many of those runs have that status now. running includes a run left behind by a daemon that stopped mid-run. The five add up to started.
success_rate done / (done + failed + blocked), 0 to 1; null when none has finished.
time_to_pr_s, time_to_merge_s {median, n}: seconds from the run’s start to its first pr_opened (or merged) event, over the n runs that have one. Events stamped after now are not read. null when events cannot be read, or when a run in the window started before the oldest event still kept: its PR event may have been deleted. The daemon deletes events older than web.history_days (7 by default), so with the default the 30-day times are null. To get them, set web.history_days to 30; they appear once the events kept reach back past every run in the window (up to 30 days after the change, since deleted events do not come back).
came_back Distinct issues dispatched again in the window after an earlier run on the same issue had ended done. The window’s came_back_issues lists them (repo, issue_number, url; at most 50).
reopened Issues the relay finished (it saw their PR merge and closed them) that were reopened on GitHub in the window, dated by GitHub’s reopened event. Counted whether or not the relay picks the issue up again; one per finish. The window’s reopened_issues lists them. The poller notices a reopen only for an issue its search returns (open and assigned to the operator, or wearing a trigger label), with no extra GitHub call unless a finished issue shows up open. Finishes are recorded in the state database from this version on, plus those still in the kept merged events when the table is created. null on a database the daemon has not migrated yet. A reopened event goes to the feed when one is recorded.
cost_per_run What the runs that ended (done or failed) in the window used, from the token usage the run budget records on each run (#223, see Run budgets): {tokens, cost_usd, runs_without_usage}. tokens is {median, total, n} over the n ended runs whose usage was recorded and read in full, counted as the budget counts (input + cache writes + output; cache reads left out). cost_usd is {median, total, n} in US dollars over those of them whose agent reported a cost (total_cost_usd, information only), or null when none did. runs_without_usage is how many ended runs are left out: no usage recorded (runs from before #223, or a backend that is not measured, such as opencode) or usage that could not be read in full. They are never counted as 0. Blocked and running runs are left out, since their count still grows. null when no ended run in the window has its usage measured, or on a database the daemon has not migrated yet; the window’s unavailable says which, and why any other field is null.
stuck Runs the relay stopped as stuck (#224), also counted in failed. null only on a database the daemon has not migrated yet.

numbers is null, with a line in errors, when the sessions cannot be read.

Each machine also sends these 7-day numbers to the chat once a week; see configuration: schedules.weekly_summary.

The page with peers. A Machines strip shows one card per machine, this one first: state, slots, 5-hour and 7-day usage, heartbeat age. A card is red when the machine is paused, silent, unknown, not reachable, or not read yet. A peer that answers with a different version from this machine’s is red too, with version X, this machine runs Y on its card, so a node a release missed shows up; a peer whose version is unknown is not marked. A peer that does not answer is shown as not reachable from here, with the reason (timeout, refused, name not found, bad host header, an HTTP status) and its last known numbers greyed out, never as idle. Who has what lists every claimed issue across the machines, and Needs you shows every machine’s questions, each tagged with its machine. The rest of the page is this machine’s own.

Pausing for usage. Every agent run reports the subscription’s usage, and an idle machine refreshes it with one tiny call every usage.probe_minutes. When the 5-hour or 7-day window reaches usage.pause_at, or the status is not allowed, the machine stops taking new issues (runs already going finish), sends one chat line, and takes issues again after the window’s reset, with one more chat line. A usage it could not read is shown as unknown and never pauses it. See configuration: usage.

Try it on loopback first (the default web.host is 127.0.0.1, port 8787):

ctrlrelay web start
ctrlrelay web status
curl -s http://127.0.0.1:8787/api/status | python3 -m json.tool | head -40
curl -s "http://127.0.0.1:8787/api/events?since=$(( $(date +%s) - 3600 ))"
ctrlrelay web stop

To open it from another machine on the home network, set web.host in orchestrator.yaml to this machine’s LAN address (see web), then run it under systemd. ctrlrelay install systemd renders ctrlrelay-web.service with the other two units. It refuses to overwrite units that already exist, so on a host where the bridge and poller are installed, render into a scratch directory and copy only the web unit:

ctrlrelay install systemd --workdir ~/Projects/ctrlrelay --target-dir ~/ctrlrelay-units
cp ~/ctrlrelay-units/ctrlrelay-web.service ~/.config/systemd/user/
rm -r ~/ctrlrelay-units   # the other two carry the chat token
systemctl --user daemon-reload
systemctl --user enable --now ctrlrelay-web.service
journalctl --user -u ctrlrelay-web -n 20
tail -n 20 ~/.ctrlrelay/logs/web.log

The first line in web.log is a web.started event naming the address it bound. If web.host is 0.0.0.0 it also logs a warning: that listens on every interface, which is rarely what you want.

With web.host set to a LAN address the page also answers on 127.0.0.1:8787, so a browser or health check on the machine itself is not refused, and web.log has a second web.started line for it. The LAN side still listens on web.host and nothing else. If that loopback port is taken by something else, web.loopback_unavailable is logged and the LAN page starts anyway.

Every 30 seconds the server checks that web.host is still one of this machine’s addresses. If it is not (a new DHCP lease, a router reboot, another network), it logs web.address_gone to web.log, prints why to web.error.log and exits 1. systemd (or launchd’s KeepAlive) restarts it; on the old address that start fails with Cannot listen on ...: Cannot assign requested address in web.error.log, again every few seconds, until the address comes back or you set web.host to the new one. A check that cannot be made (a web.host name that does not resolve) is logged once as web.address_unchecked and the page keeps running.

The page answers only to names it knows: localhost, 127.0.0.1, [::1], web.host, and this machine’s own hostname (plus .local). A request whose Host header carries any other name gets 421 misdirected request, and the refused name is logged once as web.misdirected_host. So if you open it from another machine by a name that is none of those, for example a DNS name your router hands out, add that name to web.allowed_hosts and restart the unit:

web:
  host: "192.168.1.20"
  allowed_hosts: ["mybox.lan"]

grep misdirected_host ~/.ctrlrelay/logs/web.log shows any name that was refused, which is the quickest way to find the one to add.

Several machines on one account

Two or more machines can run ctrlrelay against the same repositories under the same GitHub account. Each machine’s seen-list and run lock are its own, so the claim comment on the issue is what keeps two machines off one issue.

The claim. Before a dev or task run starts, the machine posts a comment naming itself:

Taken by ctrlrelay on **node-a**. It is working on this issue; ...

<!-- ctrlrelay:claimed node=node-a id=3f9c0a1b2c4d state=working run=5e1d02aa seen=... -->

node is the config’s node_id (the hostname by default). Give every machine a different one. run names the process that last wrote the claim and seen when, by its clock: a machine warns when a claim under its own name is refreshed by another process (two machines sharing a node_id). Only comments by the account the relay runs as count; a marker pasted by anyone else is ignored.

The race check. Two machines can see a new issue in the same poll. Each posts its claim, waits claims.race_wait_seconds, and reads the comments again. The live claim with the lowest comment id wins: GitHub numbers the comments, so no machine’s clock decides. The loser posts “Backing off: ctrlrelay on node-a has this issue” and drops it.

Skipping. A machine that finds another machine’s live claim does not start the run, and logs claims.skipped with the owner. It looks before announcing the issue in chat, so it stays quiet about issues already taken; only a machine that loses the race check itself has announced the issue first.

The stale rule. While a run works, its machine edits the claim every claims.refresh_seconds, which moves the comment’s updated_at. A working claim not refreshed for claims.stale_after_seconds is stale: that machine is treated as stopped, and another machine may take the issue (its claim names the claim it replaced, and a replaced claim does not count while the claim that replaced it stands, however it is edited). A skipping machine looks again at exactly the time the claim could go stale, not on every poll. If a machine whose claim was taken over is in fact still running, it stops refreshing its claim and logs claims.taken_over; its run is not stopped.

When the run ends:

  • finished, or waiting for an answer: the claim is marked held. A held claim never goes stale, so a long wait for an answer never hands the issue to another machine. If marking it held fails three times the machine logs claims.hold_failed and keeps trying every claims.refresh_seconds in the background, until the stale time counted from its last good write. If that also fails, or the machine stops, the claim goes stale like a working one.
  • failed: the failure comment releases that one claim (<!-- ctrlrelay:released id=... -->), and the issue is free again. One exception: a run that was waiting for an answer and fails after it is resumed posts no release (the resume path never posted a failure comment), so its claim stays held. Delete the claim comment to free that issue.

A claim from a version before this (<!-- ctrlrelay:claimed --> with no machine name) counts as taken by unknown and never goes stale.

If the race check cannot see the claim it just posted, the machine does not assume it won: it releases its claim and the run fails before it starts, to be retried on a later poll.

To free an issue by hand, delete the claim comment. Other machines look at a held issue again every claims.stale_after_seconds, and take it then.

setting default  
claims.race_wait_seconds 5 wait between posting a claim and reading it back
claims.refresh_seconds 300 how often a working machine refreshes its claim
claims.stale_after_seconds 1800 a working claim older than this is stale; must be at least three refreshes

Limits to know:

  • The stale time compares GitHub’s updated_at with this machine’s clock. Keep clocks in sync (NTP); half an hour is far above normal drift.
  • A machine that loses GitHub for longer than the stale time while its run keeps working can have the issue taken over. Raise claims.stale_after_seconds if that is likely.
  • Mixed versions: a machine on a version before this one reads every failure comment as ending every claim, including another machine’s, and posts claims without a machine name. Upgrade all machines together.

Chat with several machines

Every message a machine posts starts with its name: [node-a] ❓ Blocked on o/r#7: .... Each machine acts only on replies to its own posts. A reply to a post another machine made is left to that machine, with nothing said in the channel; the machine reads the replied-to post to tell (on Telegram, from the reply itself). On a shared chat, a reply to a post that cannot be read is left alone, and so is a reply to a post with no machine tag (a person’s post, or a question posted before this version): answer those by naming the machine, node-a: ....

Mattermost: machines can share one bot and channel. Set transport.mattermost.shared: true on each. A message that is not a reply then reaches a machine only when it names it, node-a: pin or @node-a pin; a plain message is left alone by every machine, since none could tell it was meant for it. With one machine, leave shared off and plain answers work as before.

Telegram: each machine needs its own bot. Telegram lets one process read a bot’s messages at a time, and when a second one starts reading, it is the one already running that Telegram cuts off. There is no way to see the other reader before that happens. So the machine that loses the bot says so once, in its log and in the chat (“Stopped reading replies on this machine: another process started reading this Telegram bot … each machine needs its own bot”), and stops reading, rather than fighting for it every few seconds.

Only the bot’s own posts count as a machine’s: a person typing [node-b] ... in the channel does not speak for node-b.

Scheduled jobs

The poller daemon also hosts an in-process cron (APScheduler). The jobs run inside the same asyncio loop as the issue poll, so they inherit the poller’s supervision (launchd KeepAlive on macOS, systemd Restart=always on Linux) without needing a separate .timer unit.

Job Default schedule What it does
secops 0 6 * * * (6am daily) Runs the secops pipeline across every configured repo — equivalent to ctrlrelay run secops.

Schedules are configured under schedules: in orchestrator.yaml, using standard 5-field cron expressions. The top-level timezone: controls how they’re interpreted:

timezone: "America/Santiago"
schedules:
  secops_cron: "0 6 * * *"   # daily 6am; use "0 6 * * 1" for weekly (Mondays)

With several machines on one account

Every machine runs the same schedule. Each job is either split, so it runs once across the machines, or belongs to the machine by design:

Job Runs Where its claim lives
secops (security sweep) once per run window across machines, per repo a git ref in that repo, refs/ctrlrelay/claims/secops/<window>
merge watcher and post-merge steps on the machine that ran the issue none needed: only that machine watches its own pull requests
pending_resume_sweeper (every minute) on each machine none: it drains this machine’s own unanswered questions
question_expiry_sweeper (hourly) on each machine none: it retires this machine’s own questions
usage_probe (every 5 minutes) on each machine none: each machine reads its own subscription
github_budget on each machine none: a free read of the shared budget
personalization_auto_pull on each machine none: each machine syncs its own copy

How the sweep is split. Before sweeping a repo, a machine creates the ref refs/ctrlrelay/claims/secops/<window> in it, pointing at a small blob that holds its node_id (in the same safe form the issue claims use). Creating a ref on GitHub is all or nothing: the first machine creates it and sweeps; every other machine gets “already exists”, reads the owner’s name from the blob, logs jobs.skipped, and skips that repo. The ref is not a branch and GitHub shows it nowhere in its interface.

<window> is the minute the sweep was due, for example 20261101T0600Z, as the scheduler planned that run, not when it started: machines a few seconds apart, or one started late, agree on it. A machine that was asleep runs a missed sweep once, up to an hour late, for the latest time it was due, and names that window. A machine that dies holding a window loses only that window: the next one is a new ref. Refs older than 30 days are deleted by the next machine that wins a claim.

A machine claims a repo only once it holds that repo’s local lock, so it never wins a claim and then finds the repo busy on its own side. (A machine that dies right after claiming still leaves that window unswept; the next window runs.) If a machine cannot create the ref (GitHub down, no write access), it skips the repo for that window rather than risk a second sweep, and says so in the sweep results; the next window tries again. A sweep started by hand (ctrlrelay run secops) is not claimed and always runs.

An invalid cron expression fails at config load time rather than silently disabling the job. If the machine is asleep at the fire time, the job runs when it wakes — up to one hour late; older misfires are coalesced into a single run.

On poller stop, the scheduler waits up to 150 seconds for an in-flight job (e.g. git worktree prune cleanup at the end of a secops sweep) to finish before letting the asyncio loop close. This requires your supervisor’s stop timeout to be at least that generous — the example unit files above set ExitTimeOut=180 (launchd) and TimeoutStopSec=180 (systemd). Without those, the platform default (launchd 20s, systemd 90s) SIGKILLs the daemon before cleanup finishes and leaves stale git worktree admin state behind. If you’re upgrading from an older plist/unit that didn’t set these, add them and reload.

Issues blocked by other issues

When work has an order (B changes the same files as A, so B waits for A), say so on GitHub with an issue dependency: on B, Relationships → Mark as blocked by → A. The blocker may be in another repository. Before the poller claims an issue it reads that issue’s blockers (GET /repos/{owner}/{repo}/issues/{n}/dependencies/blocked_by), and:

  • None is open: the run starts as usual.
  • One or more is open: the issue is not claimed and not started. The log has poll.issue.blocked with blocked_by (the open ones, #N in the same repository, owner/repo#N in another), the status page lists it under Skipped as blocked by …, and the chat is told once per issue, not every poll. Every later poll asks again, so it starts on the first poll after the last blocker closes, with nothing to do by hand.
  • The blockers could not be read (an API error, the GitHub API budget used up, a 404 from a GitHub Enterprise Server without the dependencies API): treated like an open blocker, never as “no blockers”. The log has poll.issue.blockers_unknown with the error on every failed poll, and the next poll tries again. A single failure is usually a blip, so the status page keeps showing the blockers last read and the chat is not told; after 3 failed polls in a row the page says could not read its blockers and the chat is told once per issue. If the blockers can then be read and are the same as before, nothing more is said.

Only the dependency link counts. A line such as “Starts after: #A” in the issue body is not read, so add the link as well.

A held issue is kept out of seen_issues (that is what brings it back) and recorded in blocked in the same poller_state.json, so a restart neither loses it nor repeats the chat line. Its self-assignment is checked again every poll (it may have been unassigned and assigned again by someone else), but a check that fails leaves a held issue for the next poll instead of marking it handled for good. Each poll costs a held issue one extra REST call (core budget) for its blockers, plus the re-read every issue gets just before it starts. A blocked issue that is closed or loses its trigger is dropped from blocked when the poller next sees it.

Stuck runs

A run that goes in circles is stopped early instead of running until agent.default_timeout_seconds.

  • While the agent works, ctrlrelay reads its live transcript every 30 seconds. If the agent runs the same command with the same input 5 times in a row and gets the same result back each time, or gets the same error back from the same command 5 times in a row with no success of that command in between (edits in between do not count), the agent is stopped. A command polled while its output moves on is waiting, not a loop; a poll that gets the identical output back 5 times in a row is still treated as one. Only the claude backend writes a transcript ctrlrelay knows how to read; other backends are not watched.
  • Between CI fix rounds, each failed job’s log is sorted on its own into one of three kinds. The round is out of budget if any job is, infrastructure only if every job is, and a code failure otherwise; a job whose log could not be fetched counts as a code failure, and a check that is not an Actions job, or one past the first 3 jobs, is not sorted at all:
    • flaky infrastructure (the runner was lost, the network failed, a registry answered 502/503, and no failing test is named): the failed GitHub Actions jobs are re-run once, without asking the agent. If they fail on infrastructure again, the run stops. A fix round in between resets this, so a new flake later gets its own re-run.
    • out of budget (the job hit its maximum execution time or a spending limit): the run stops at once. A re-run would spend the same budget.
    • code failure (anything else, including every log that names failing tests): sent to the agent as a fix round, as before. If the same tests fail in 3 CI rounds in a row (the first failure and two fixes that changed nothing), the run stops instead of spending the next round.

A stopped run is recorded as failed with a stuck_reason in the sessions table, its summary starts with Stuck:, the issue comment gives the reason, and chat gets 🔁 Stuck on #<n>, stopped: <reason>. The status page’s numbers count these runs in stuck.

Run budgets

Each run can have a token budget (#223), set in agent.budget. There is no limit until one is set.

  • What is counted. Every 30 seconds ctrlrelay adds up the usage in the agent’s live transcript, sub-agents included: input tokens, cache writes and output tokens. Cache reads are recorded but not counted. One API message written as several rows is counted once.
  • Per session, not per spawn. A dev or task run spawns the agent more than once (the first run, resumes after a question, CI fix rounds). They all count against the same budget, because each spawn starts from what the session row already holds.
  • Soft limit. The first time the total passes it, chat gets one line (💰 #<n> (<repo>) has used <N> tokens, past its soft limit ...) and the log gets budget.soft_limit. Once per session.
  • Hard limit. The agent is stopped through the same path as a stuck run. The run is recorded as failed with stuck_reason set to budget: used <N> tokens, hard limit <M>, its summary starts with Stopped:, the issue comment gives the reason, and chat gets 💸 Over budget on #<n>, stopped: <reason>. A checkpoint the agent wrote just before the stop is not used: the work is not handed over as ready. When no pull request exists yet, the unfinished work is not saved to a branch (only a timeout does that). The status page’s numbers count the run in failed, not in stuck.
  • A stop after the PR was opened (over budget in a CI fix round, or stuck there): the PR is put back to draft and kept with the work so far, no merge watcher is started for it, and chat and the issue comment say PR #<n> was kept as a draft with the work so far. If GitHub refuses the draft, a “Do not merge” comment goes on the PR instead, and the messages say so.
  • A floor, not an exact bill. Only API calls that write an assistant row to the transcript are counted; calls the CLI makes on its own without one (for example, short helper calls) are not. The count can be a little under what the subscription was charged.
  • Unknown is not zero. A transcript row whose usage cannot be read adds nothing and is counted as unreadable; the log gets budget.usage_unreadable and the page shows the total as unknown. What could be read is still a true floor, so a run is stopped only when that floor reaches the hard limit. An unreadable row on its own never stops a run.
  • What is stored. sessions.usage_tokens is the counted total (a floor when rows were unreadable) and sessions.usage_detail holds the breakdown, the unreadable row count, the cost and the limits applied, as JSON. cost_usd is what the agent CLI’s final result event reported (total_cost_usd), summed over the session’s spawns; it is information only.
  • Backends. Only the claude backend writes a transcript ctrlrelay can read. A run on opencode is not measured; with a limit set, the log gets dispatcher.budget.unmeasured and nothing is enforced.

A run whose process died

A dev run saves each step it finishes: plan, code, publish and verify (#231). Each is a row in the run_steps table of the state DB, with the run branch’s commit at that moment and what the later steps need (the plan, the agent’s result, the pull request). No commit is added to the branch for it.

When the relay process dies in the middle of a run (out of memory, a restart, a power cut), the poller settles that run when it starts again and hands the issue back. The next run for the issue then continues after the last saved step:

  • it checks that the branch is still at the saved commit, and starts from the beginning if not (or stops with the reason, when the dead run had already opened its pull request);
  • it does not run those steps again: no new agent session for a saved code step, no second push or pull request for a saved publish step;
  • it comments on the issue which run it continues and from where;
  • proof and review always run again, because they read the pull request as it is now.

A dead run that had opened its pull request is handed back too, and only that pull request is reused. A run that failed for its own reasons is not continued; the next run starts from the beginning, as before.

To start again from a past step by hand, for example to redo the code with the same plan:

ctrlrelay run dev --issue 42 --repo your-org/your-repo --from-step code

--from-step takes plan, code or publish, uses the newest run of the issue that saved steps, and moves the branch back to where it was before that step. The old head is kept at refs/ctrlrelay/before-retry/<run id> in the relay’s clone. It refuses a branch behind an open pull request, and a branch that is on GitHub is synced back to GitHub’s head; delete the remote branch first in that case.

The log events are dev.step.saved, dev.resume.restored and dev.resume.head_moved.

Running several agents at once

Set agent.max_parallel to let one machine run more than one agent (#210):

agent:
  max_parallel: 3
  • One cap over everything. New issues from the poll, runs resumed after you answer a question, and the scheduled security sweep all take a slot from the same pool. A run takes its slot before the repo lock, and gives it back when it ends.
  • New issues side by side. Polled issues wait in a backlog, in priority order, and start as slots free up. Two issues of the same repo never run together: the second waits, and an issue of another repo behind it goes first. The loop keeps polling on its interval whatever is running, so one slow issue never holds up new ones.
  • A run holds its slot for its whole life, including the wait for CI and the fix rounds, and while it waits in the run for your answer. Once it gives up waiting and saves the question, the slot is free again, and the resume takes one when your answer comes. Two pull requests waiting on slow CI can fill max_parallel: 2.
  • A hold placed while a run waited for its slot is respected. The resume sweeper and the security sweep check again once they have a slot, so a self-update that started meanwhile is not raced.
  • On shutdown no new run starts in a slot a stopping run frees, and issues polled but not started are offered again after the restart.
  • Unset keeps the old behaviour: new issues one at a time, and nothing limits the total.

Each run uses its own agent session, worktree and memory. Size the number to the machine and to your plan’s usage limits. Start with 2.

The status page shows slots busy/max, and slots N (no cap) when it is unset. The log event poll.batch.parallel names each batch, the limit and how many slots were busy.

When to restart

You changed… Restart…
transport.* (type, token env var, chat or channel ID, socket path) bridge and poller
repos[] (added/removed/renamed) poller
agent.* or repos[].agent_type (backend, binary path, timeout) poller
paths.* both
Anything in dashboard.* poller
schedules.* (cron expressions) poller
Bot token env var bridge and poller

After a ctrlrelay package upgrade (uv pip install -e .), restart both. For a node installed with uv tool, ctrlrelay self-update does the install and the restarts for you; see below.

Updating a node

ctrlrelay self-update                    # the latest version on PyPI
ctrlrelay self-update --version 0.23.0   # a given version

It only installs a version already published on PyPI; publishing stays a human decision. In order:

  1. Reads https://pypi.org/pypi/ctrlrelay/<version>/json and checks that this node’s status page answers on 127.0.0.1:<web.port>; without it the update could not be checked, so it refuses. It also refuses when there is no poller or web unit installed, or ctrlrelay is not a uv tool install. Nothing has changed yet.
  2. Downloads the wheel into wheels/ beside the state database and checks its sha256 against that JSON. A mismatch deletes the file and installs nothing. The wheel is installed by path rather than by name because a node’s package index has served a stale list for more than ten minutes after a release. The file is kept, since uv’s receipt records the path.
  3. Holds new work: it writes intake-hold.json beside the state database. While it is there the poller leaves new issues for the next poll and skips the scheduled sweep and resumed questions, the same way the usage pause does; runs already going finish. The poller answers in intake-hold.ack.json with whether anything is still in flight. The command waits, up to --wait-minutes (default 60), until the poller says nothing is in flight and the state database has no running session. Checking the database alone is not enough: an issue can pass the poller’s checks a few seconds before its session row is written, and a restart in that gap cancels it.
  4. uv tool install --force <wheel>, then restarts every installed unit: systemctl --user restart ctrlrelay-{bridge,poller,web}.service on Linux, launchctl kickstart -k gui/<uid>/<label> for each *.ctrlrelay-{bridge,poller,web}.plist in ~/Library/LaunchAgents on macOS (the label is the file’s name, whatever --label-prefix it was installed with).
  5. Waits up to two minutes for /api/status to report the new version, and up to two and a half for a new poller process to report it in its heartbeat (heartbeat.json now carries the poller’s pid, version and started_at) and still be the one reporting at least one heartbeat interval (30 s) after it started. The status page’s version is the web server’s own, and a restart command succeeds even when the unit then crash-loops, so neither alone shows the poller is up.

If any step after the install fails, or the command is interrupted (Ctrl-C, SIGTERM) once the install has begun, the old version is installed and restarted the same way, and checked; a poller from before this command writes no version in its heartbeat, so for the old version a fresh heartbeat from a new process is enough. Chat (when a transport is configured) gets one line either way: the update, or the step that failed and whether the old version is running again. The command exits non-zero on any failure. The hold is removed on every way out, including Ctrl-C and SIGTERM; one left by a process that was killed outright is ignored once that process is gone or the hold’s expiry (the wait plus 30 minutes) has passed.

A poller from before this command does not know the hold and never acknowledges it, so the command stops after a minute without changing anything. For that one update, --no-ack goes on the state database alone, with the gap described in step 3. A poller that is not running (its heartbeat is older than fleet.silent_after_seconds) is not waited on.

A scheduled security sweep whose time comes while the hold is in place is skipped until its next time, not delayed.

After merging a PR that the dev pipeline opened, no manual restart is required — the PR is referenced from the open branch in your bare repo, so cleanup is handled by GitHub on merge. The worktree was already removed when the session finished.

Logs

By default, both daemons log to the paths configured in your launchd / systemd unit files. The recommended location is ~/.ctrlrelay/logs/.

Tail them:

tail -f ~/.ctrlrelay/logs/poller.log
tail -f ~/.ctrlrelay/logs/bridge.log
tail -f ~/.ctrlrelay/logs/poller.error.log
tail -f ~/.ctrlrelay/logs/web.log      # if the status page is installed

The poller prints one line per detected issue and one line per pipeline outcome. The bridge prints connection events and chat API errors (Telegram Bot API or Mattermost REST/WebSocket).

Tracing one agent session

Log lines are newline-delimited JSON, and the dispatcher — the module that spawns and supervises every agent subprocess — tags each of its events with the session id, so one grep reconstructs a whole run:

grep dev-owner-repo-42- ~/.ctrlrelay/logs/poller.log | jq -c '{ts, level, event}'

The events it emits:

Event Level When
dispatcher.session.start INFO Subprocess about to launch. Carries binary, timeout, resume, prompt_hash, prompt_len.
dispatcher.session.finished INFO Subprocess exited. Carries exit_code, checkpoint_status, elapsed_ms.
dispatcher.session.subprocess_failed ERROR Non-zero exit, with stderr_tail.
dispatcher.session.timeout ERROR The timeout seconds elapsed and the child was killed.
dispatcher.session.spawn_failed ERROR The binary could not be exec’d at all.
dispatcher.session.cancelled INFO Daemon shutdown killed the child mid-run.
dispatcher.checkpoint.missing WARNING The session ended without writing a checkpoint.
dispatcher.checkpoint.read_failed ERROR A checkpoint file existed but would not parse.
dispatcher.binary.fallback WARNING The agent binary (claude or opencode) was not on PATH; a well-known path was used instead. Expect this when the unit’s PATH is minimal.
dispatcher.binary.unresolved WARNING Not on PATH and not at any known path — sessions are about to start failing to spawn.

Prompts and agent output never appear in plaintext: only a short SHA-256 prefix and a length are logged.

Blocked-question events

When a session blocks on a question, the transport and the bridge each emit their own record:

Event When
dev.question.posted The question reached the operator. The transport emits it on the bridge’s ACK; the bridge emits it once the chat server (Telegram or Mattermost) has accepted the message, and adds its post_id.
dev.question.post_failed Delivery failed. Carries reason (send_failed, bridge_error, or the exception class on the bridge side) and a truncated error.
dev.answer.received The operator’s reply came back.

Neither the question nor the answer is written to the log. Each event carries question_hash / question_length (or answer_hash / answer_length) instead — the hash is stable, so it still joins the poller’s record to the bridge’s record for the same question:

grep 44509098fa6eeaf3 ~/.ctrlrelay/logs/*.log | jq -c '{ts, logger, event}'

This matters because these files and the systemd journal have no retention policy tuned for human-entered content, and operator replies are free text.

Inspecting state

ctrlrelay status

The fastest way to see what’s happening:

ctrlrelay status

Shows held repo locks and the 5 most recent sessions with their statuses (running, done, blocked, failed).

State DB schema

The state DB is plain SQLite at paths.state_db. Open it with sqlite3 for ad-hoc queries:

sqlite3 ~/.ctrlrelay/state.db
sqlite> .tables
sqlite> .schema sessions
sqlite> SELECT id, pipeline, repo, status, started_at FROM sessions ORDER BY started_at DESC LIMIT 20;

Tables (defined in src/ctrlrelay/core/state.py):

  • sessions — every pipeline run. Columns: id, pipeline, repo, issue_number, worktree_path, status, blocked_question, started_at, ended_at, claude_exit_code (the worker’s exit code, whichever backend), summary, agent_session_id, output_tail.
  • repo_locks — currently held per-repo locks. PK on repo.
  • github_cursor — last-seen issue/PR timestamps per repo (used by the poller to bound API calls).
  • pending_resumes — one row per BLOCKED session’s question, whichever chat it went to: open until answered or expired, then kept for question_retention_days. (telegram_pending still exists in the schema but is unused.)
  • pr_watches — PRs the poller is watching for merge; survives restarts.
  • automation_decisions — operator decisions on ask-policy automation prompts.

Worktrees on disk

Per-session worktrees live under paths.worktrees, named <owner>-<repo>-<session-id>. ctrlrelay removes them on DONE or terminal FAILED; BLOCKED sessions keep their worktree so a manual operator can inspect / resume.

If something goes wrong and you need to reclaim a stuck worktree:

git -C ~/.ctrlrelay/repos/your-org-your-app.git worktree list
git -C ~/.ctrlrelay/repos/your-org-your-app.git worktree remove --force /path/to/orphan

Poller seen-set

The poller’s seen-issue state is a JSON file at <state_db_dir>/poller_state.json. If you want to force the poller to re-process a specific issue, delete its number from seen_issues[<repo>] (or delete the file entirely to reseed from scratch on next start).

A run that fails before it is handed to the agent pipeline (reading the issue, posting the claim comment, cloning, cutting the worktree) does not use the issue up. The issue is un-marked and tried again on the next poll, up to 3 runs in all, with a chat message each time when a chat channel is connected. After the third it stays in seen_issues and the message says so; fix the cause, then delete the number as above. The count is kept in pre_dispatch_failures in the same file, so a restart does not reset it. Each retry posts a fresh claim comment and, when it fails, a fresh “nothing is working on this” comment on the issue.

A run whose agent finished but whose branch the relay could not publish is retried the same way, for the same reason: the issue is already in seen_issues, so without this nothing would ever bring it back. Up to 3 runs in all, counted in publish_failures in the same file, and the retraction comment on the issue says the relay will return rather than asking you to re-assign it. After the third run the issue stays in seen_issues and the relay says so on the issue — the commits are still on the run branch in the local checkout (on GitHub too, if only opening the pull request failed), and the fix is to clear the cause and delete the number as above. If that comment cannot be posted, the chat line says so.

A push the relay refuses (a protected or default branch, a base it did not expect, somebody else’s commits on the branch) is not retried at all: the relay reaches the same refusal every time, so another run would spend a whole agent to arrive at it. The issue stays in seen_issues and the comment names what to do.

Re-assigning an issue, or re-applying its automation label, does not make the poller look at it again. The skip decision reads the issue number only — no assignee and no labels — so the number has to leave seen_issues (and the relay has to be restarted, since the running poller holds the set in memory and rewrites the file from it).

An issue held back by an open GitHub blocker (see Issues blocked by other issues) is never in seen_issues while it waits; it is listed in blocked[<repo>] as {"open": [...], "failures": n}: the open blockers as last read (null if never read) and how many lookups have failed in a row since. Nothing needs deleting: it starts on its own once the blockers close.

A repo whose history was replaced

If the local copy’s default branch can no longer fast-forward to GitHub’s (the repo was replaced by a new one with the same name, or its default branch was force-pushed), ctrlrelay moves the copy to ~/.ctrlrelay/repos/<owner>-<repo>.git.replaced-<UTC time>, clones again, and says so in chat. The old copy is kept as it was after the failed fetch: remove it once you do not need it. This is refused while any worktree still uses the copy (a blocked session keeps its worktree for the resume); the run then fails with default branch diverged ... not re-cloning while N worktree(s) still use it, and you can finish or clean up that session first.