Troubleshooting
Find the problem fleetd doctor reported, or why work is stuck, and fix it. fleetd doctor checks the
hub and every worker.
Run the doctor
Section titled “Run the doctor”On the hub:
fleetd doctorExample output, from a new hub with one worker, test-1, where nothing is logged in yet:
✓ Where the hub listens The hub answers at http://127.0.0.1:4790/healthz✓ Machines that run agents test-1 is online✗ Claude test-1: claude is not logged in, so its sessions can't start. Fix: claude auth login! GitHub gh is not logged in on the hub; the merge train and PR checks need it. Fix: gh auth login– Invite people optional · fleetd init --step people– Google optional · fleetd init --step google– Notion optional · fleetd init --step notion✗ doctor test-1 No git commit identity is set on test-1. Fix: git config --global user.email '<id>+<user>@users.noreply.github.com'✓ doctor 4 of 6 other checks pass[exit 1]The hub’s setup steps come first, one line each; fleetd init --step <id> opens
one. Then each other check that doesn’t pass, named by where it is: hub, or a machine. Checks that pass aren’t
listed, only counted on the last line.
| Mark | Means |
|---|---|
| ✗ | A failure. Fix it. |
| ! | A warning, or a check that couldn’t tell. |
| – | Not checked yet, or a setup step not needed yet. |
| ✓ | A setup step that’s done, and on the last line, how many other checks passed. |
Most findings end with Fix: and what to do, on the machine the finding names. The app’s Machines screen shows the
same checks. The tables below give each check’s id in brackets, such as [claude_login].
The hub
Section titled “The hub”| Symptom | Cause | Fix |
|---|---|---|
“the hub does not answer at http://…” [exit 4] |
The hub isn’t running. | fleetd init. If you run the hub yourself (--no-service): fleetd hub start. Log: ~/.fleetd/hub/hub.log. |
“The hub does not answer at its own address …/healthz” [reachable] |
The hub service is stuck. | Restart the hub service. |
“gh is not logged in on the hub” [gh_auth] |
gh is missing or logged out. |
gh auth login. If gh is missing, install it from cli.github.com first. Repositories the hub’s GitHub App is installed on don’t need it. |
“The hub’s address http://… is not HTTPS” [https] |
The hub listens on plain HTTP outside loopback or a tailnet. | Serve it over HTTPS, then point fleetd at it (below). |
“No CI on owner/repo: fleetd merges only on green CI, so their ready PRs wait.” [repo_ci] |
The repo has no GitHub Actions workflow and requires no check, so its reviewed PRs can’t go green. | Add a workflow that runs on pull requests. The waiting PRs go on by themselves once it exists. |
“The hub has no Claude API key.” [claude_key] |
Nothing is wrong: leads and unrequested work run on the hub owner’s subscription, else each machine’s own login. | To give the hub a key: Settings › Hub accounts › Claude. |
“The hub’s Claude API key failed its Test on devbox: …” [claude_key] |
The key was revoked, is out of credit, or the endpoint is wrong. | Settings › Hub accounts › Claude: Change, paste the key and endpoint again, then Save and test. |
“… isn’t on buildbox, so nothing runs on it there.” [claude_key] [claude_owner] ! |
That machine was offline when the key or token was saved. | Save it again while the machine is online. |
“… is saved but not tested yet.” [claude_key] [claude_owner] |
No Test since it was saved. | Press Test beside it. |
“Sam Rivera’s Claude subscription failed its Test on devbox: …” [claude_owner] |
The owner’s token expired or was revoked. | The owner runs claude setup-token again and pastes the token in Settings › Your accounts › Claude. |
“The hub has no address set up”, or setup.json is unreadable [https] |
~/.fleetd/hub/setup.json is missing or broken. |
Same as above: fleetd init --url <https address>. |
“… has no hub set up” [exit 4] |
The hub was started by hand without fleetd init first, so its home has no setup.json. |
Run the fix it prints. For a hub you run yourself, that is fleetd init --no-service --home <home> --listen <host:port>, which leaves the running hub alone. |
This is what doctor shows when the hub is down:
$ fleetd doctor✗ doctor the hub does not answer at http://127.0.0.1:4778: error sending request for url (http://127.0.0.1:4778/mcp): client error (Connect): tcp connect error: Connection refused (os error 111). Fix: fleetd init (it starts the hub and repairs its setup)[exit 4]To serve the hub over HTTPS with Tailscale, stop the hub, then:
tailscale serve --bg 4790fleetd init --url https://<that address>Plain HTTP is accepted on loopback and on a tailnet address (100.64.0.0/10 or a *.ts.net name). If the HTTPS
address stops answering, check where it points with tailscale serve status.
Claude
Section titled “Claude”These apply to every worker.
| Symptom | Cause | Fix |
|---|---|---|
“claude 2.1.302, not logged in, so its sessions can’t start” [claude_login] |
claude on that machine is logged out, or its login expired. |
claude auth login on that machine, or log in through your proxy. |
“~/.claude/settings.json on devbox sets ANTHROPIC_BASE_URL in its “env”, which claude applies over the Claude account fleetd gives each session there.” [claude_account_override] ! |
The hub runs work on Claude accounts, and that machine’s claude settings name a credential or gateway of their own, which would win over the account. Read only: fleetd never changes the file. | Move those variables out of the settings’ "env" (into claude.env for the machine’s own login), then restart the node. |
“claude is not installed on laptop” [claude_version] |
No claude on the machine’s PATH. |
curl -fsSL https://claude.ai/install.sh | bash |
“claude 2.1.296 is older than fleetd is tested with” [claude_version] ! |
An older claude. It may work: fleetd checks its protocol at every start. |
claude update |
“claude 2.2.0 is newer than fleetd is tested with” [claude_version] ! |
A claude past fleetd’s tested range. Sessions run. |
If they misbehave, claude install 2.1.302 |
“claude --version printed …, which fleetd can’t read” [claude_version] |
A broken install. | curl -fsSL https://claude.ai/install.sh | bash -s 2.1.302 |
“did not answer fleetd’s protocol probe as expected” [runner_conformance] |
This claude speaks a different protocol. |
claude install 2.1.302 |
“claude --version and the running claude disagree” [runner_conformance] |
Two claude installs on PATH. |
Reinstall with curl -fsSL https://claude.ai/install.sh | bash, and remove the other. |
“has not probed its claude yet” [runner_conformance] |
The node just started. | Nothing: it clears by itself. |
| An agent’s activity says “claude offered its own agent tools (…)” | A newer claude has an agent tool fleetd’s deny list didn’t name. fleetd stopped that session, and the next start denies the tool. |
Nothing. If it happens again, claude install 2.1.302. |
fleetd is tested with claude >=2.1.300, <2.2.0; other versions run, with a warning. A machine whose claude is
missing or logged out runs no sessions until it is fixed. Updating fleetd moves claude to the version it was tested
with, and the node turns claude’s own background update off (see Claude Code’s version).
| Symptom | Cause | Fix |
|---|---|---|
“claude on … updates itself in the background” [claude_self_update] ! |
The node couldn’t write DISABLE_AUTOUPDATER into claude’s settings.json (it isn’t valid JSON, say). |
Set "env": {"DISABLE_AUTOUPDATER": "1"} in that file, then restart the node. |
| An update’s Claude Code line names Homebrew or npm, or says “not installed” | claude came from Homebrew, npm or a package, or isn’t there; fleetd leaves it as it is. | Run the command the line names. |
Machines
Section titled “Machines”| Symptom | Cause | Fix |
|---|---|---|
“laptop is offline (last seen …)” [online] |
The node is down, or the machine is asleep or off the network. | Wake it, then restart the node. Log: ~/.fleetd/logs/node.log. |
“laptop is degraded: …” [online] |
Usually a claude problem on that machine. | The fix the finding names: see Claude. |
“No git commit identity is set on laptop” [commit_identity] |
The node has no commit author. | Set up over SSH: fleetd machine add laptop. Joined with a code: fleetd pair --worker laptop, then run what it prints on laptop. The hub machine: fleetd machine add --local. |
“Commits on laptop use … ; the hub’s identity is …” [commit_identity] |
The machine’s git config differs from the hub’s. | Same as above. |
“Paused new agents: 14 GB free on laptop” [disk] |
Low disk where the worktrees live. Each writer gets its own git worktree and build output. | Free space on that disk. New agents start again by themselves once it has room. See Disk and load. |
“Paused new agents: load 70.2 on 32 CPUs on laptop” [disk] |
The machine is overloaded. | Nothing: new agents start once the load falls. |
“Paused new agents: memory 93% on laptop” [disk] |
The machine is short of memory. | Close what you don’t need. New agents start once less than 81% is in use. |
“Paused new agents: memory 84% with swap 100% on laptop” or “memory pressure 23% on laptop” [disk] |
Swap is full and memory nearly so, or the kernel is stalling tasks to find memory. Files in a tmpfs /tmp count as memory too. |
Close what you don’t need, or clear large files from /tmp. Running agents keep going; new ones start once it eases. |
“Paused new agents: 5-hour limit 91% on laptop as of Tue 13:42 UTC, resets … (pauses at 90%, lifts below 85%)” [disk] |
Its Claude login has nearly used a usage window, your own Claude Code sessions included. | Nothing: new agents start below 85% or at the reset; a reading with no reset time counts for 15 minutes. Running agents keep going. |
“New turns on laptop are paused by the usage breaker”, “The usage limit was reached on laptop’s login; resets …” [usage_breaker] |
Its Claude login hit a plan or rate limit, and claude’s message said when it resets. | Nothing: agents go on at that reset. To go on sooner, call usage_resume from an MCP client once it has capacity. |
“Rate-limited by your proxy on laptop; checking again shortly” [usage_breaker] |
claude goes through a proxy (such as CLIProxyAPI) that rests its credentials for a model after a 429, and the message names no reset. fleetd never makes one up. | Nothing: fleetd sends one parked agent on after 1, 2, 5 and 10 minutes, then every 15, and lifts the pause once that agent’s turn completes (a turn that has only started proves nothing: claude may still be retrying the 429), or as soon as any turn on that login completes. A check with no parked agent left to try lifts the pause and says so, and the next turn there tries the login. To go on sooner, call usage_resume. |
“Automatic recovery is paused on laptop: 3 agents died there within 10 min (…)” [recovery] |
claude kept exiting or stalling there, often out of memory or a broken toolchain, so fleetd stopped resuming its dead agents there. Their leads were told once. | Find out what kills them. Recovery resumes by itself 30 min after the last death, or as soon as an agent there completes. To go on sooner, a lead continues one with message_agent. |
“laptop runs 1 session no agent or lead of this hub owns: …” [sessions] |
Its node runs a claude session that no task or lead of this hub started or still tracks. Nothing watches it or counts it toward the cap. | Read it with thread_tail. If nothing should be running it, stop its claude on that machine. |
Keep a machine awake
Section titled “Keep a machine awake”While agents run, the node keeps its machine awake. Between jobs nothing does. Sleep on battery only is fine.
| Desktop | Fix |
|---|---|
| macOS | sudo pmset -c sleep 0 |
| GNOME | gsettings set org.gnome.settings-daemon.plugins.power sleep-inactive-ac-type 'nothing' |
| KDE Plasma | System Settings > Power Management > On AC Power > When inactive: Do nothing |
logind IdleAction |
IdleAction=ignore in /etc/systemd/logind.conf.d/fleetd.conf, then reboot |
When a machine wakes, its node reconnects at once, resumes any run that went quiet, and the machine strip shows “slept 10m46s” for a while.
Restart a service
Section titled “Restart a service”launchctl kickstart -k gui/$(id -u)/sh.fleetd.hub # the hublaunchctl kickstart -k gui/$(id -u)/sh.fleetd.node # a node, on that machinesystemctl --user restart fleetd-hub.service # the hubsystemctl --user restart fleetd-node.service # a node, on that machine| Log | Where |
|---|---|
| Hub | ~/.fleetd/hub/hub.log. Run by hand: the terminal. |
| Node | ~/.fleetd/logs/node.log. Run by hand: the terminal. |
| The desktop app | ~/Library/Logs/fleetd/app.log on macOS, ~/.local/state/fleetd/app.log on Linux. Help → Reveal Logs opens it; Help → Copy Diagnostics copies the app’s version, the OS, the hub connection and the log’s last lines, for a bug report. |
Work is stuck
Section titled “Work is stuck”Ask the lead in the chat what it is waiting on. fleetd status with the project shows its state, why it is blocked
and why its bar last fell:
fleetd status storefront| Symptom | Cause | Fix |
|---|---|---|
| A pull request stays queued on the merge train | CI or a review is pending, or the train can’t read GitHub. | The train_get tool shows each PR on the train with its state and last error. |
| The lead hears “verdict not recorded” for a PR | Its reviewer ended without recording a verdict with report, usually because its fleetd tools didn’t start. Whatever its last words say, that is no approval. |
Nothing at first: the train starts one more reviewer. If that one ends the same way, the train starts no more until a new head; check the reviewer’s transcript for the fleetd tools server. |
| An agent’s activity says “node retry 1/3 in 57 s: overloaded” | Claude’s API was overloaded or the network failed, after claude’s own retries. | Nothing. The node continues the same run after about 1, 4 and 15 minutes, then fails it with claude’s words. |
| Your message waits and the chat says why | The lead can’t start: claude logged out, out of range, or the worker offline. |
See Claude and Machines. |
Then check the logs (above) for the hub and the project’s worker.