Stop the SSH suite's server refusing connections at random #4

2026-08-10T09:46:40Z

jaap-jan commented

2026-08-10 09:46:40 +00:00

The SSH suite fails intermittently with SshConnectionException: The connection was closed by the remote host, within tens of milliseconds, on whichever test happens to connect first. Seen in CI, reproduces locally. This raises sshd's MaxStartups in SshServerFixture.

Unrelated to the macOS PR, structurally: DodoSSH.Client.Ssh.Tests references only DodoSSH.Client.Ssh and DodoSSH.Client.Terminal, neither of which #3 touched.

The mechanism

sshd's compiled-in default is 10:30:100 — past ten unauthenticated connections in flight it refuses new ones at random, 30% of the time, rising to always at 100. The image ships the line commented out, so that default was what ran. xUnit runs test classes in parallel and most classes here connect, so ten in flight is reachable in the opening seconds. A refusal looks exactly like the observed failure, because a dropped connection and a server that never answered are indistinguishable from the client.

◆ It is a mitigation, not a demonstrated cure — please read this before approving

I could not measure the flake rate. On the Windows dev machine the identical unmodified suite ran 85/85 clean and, an hour later, failed 13 runs out of 15. A Linux container gave 30/30 clean and then failed on the first run of the next batch. Docker throughput on that host swings far enough to swamp the effect being measured, so every before/after comparison I took there was noise — including two I briefly believed and reported as results.

It is proposed anyway on the narrower argument that it is right either way: a connection throttle is hardening this suite has no interest in reproducing. It exists to test an SSH client, not to survive a rate limit.

The other candidate is the reload window — pkill returns when SIGHUP is delivered, not when sshd has finished closing its listeners and re-execing. A wait requiring three consecutive banner reads was written and removed: it could not be shown to change anything either, and a fixture carrying two unproven fixes for one symptom is worse than one, because the next person has to disprove both. Both candidates and how to tell them apart with sshd's own log are recorded in the fixture comment and docs/platform-flags.md.

If you want this held until it can be measured somewhere with stable Docker throughput, that is a reasonable call and I would not argue against it. The honest state is: plausible mechanism, correct-anyway change, unverified effect.

The SSH suite fails intermittently with `SshConnectionException: The connection was closed by the remote host`, within tens of milliseconds, on whichever test happens to connect first. Seen in CI, reproduces locally. This raises sshd's `MaxStartups` in `SshServerFixture`. **Unrelated to the macOS PR**, structurally: `DodoSSH.Client.Ssh.Tests` references only `DodoSSH.Client.Ssh` and `DodoSSH.Client.Terminal`, neither of which #3 touched. ### The mechanism sshd's compiled-in default is `10:30:100` — past ten *unauthenticated* connections in flight it refuses new ones at random, 30% of the time, rising to always at 100. The image ships the line commented out, so that default was what ran. xUnit runs test classes in parallel and most classes here connect, so ten in flight is reachable in the opening seconds. A refusal looks exactly like the observed failure, because a dropped connection and a server that never answered are indistinguishable from the client. ### ◆ It is a mitigation, not a demonstrated cure — please read this before approving I could not measure the flake rate. On the Windows dev machine the **identical unmodified suite** ran 85/85 clean and, an hour later, failed **13 runs out of 15**. A Linux container gave 30/30 clean and then failed on the first run of the next batch. Docker throughput on that host swings far enough to swamp the effect being measured, so every before/after comparison I took there was noise — including two I briefly believed and reported as results. It is proposed anyway on the narrower argument that it is right either way: a connection throttle is hardening this suite has no interest in reproducing. It exists to test an SSH client, not to survive a rate limit. The other candidate is the reload window — `pkill` returns when `SIGHUP` is delivered, not when sshd has finished closing its listeners and re-execing. A wait requiring three consecutive banner reads was written and removed: it could not be shown to change anything either, and a fixture carrying two unproven fixes for one symptom is worse than one, because the next person has to disprove both. Both candidates and how to tell them apart with sshd's own log are recorded in the fixture comment and `docs/platform-flags.md`. **If you want this held until it can be measured somewhere with stable Docker throughput, that is a reasonable call and I would not argue against it.** The honest state is: plausible mechanism, correct-anyway change, unverified effect.

jaap-jan added 1 commit 2026-08-10 09:46:41 +00:00

Stop the SSH suite's server refusing connections at random

ci / build and test (pull_request) Successful in 2m21s

Details

ci / desktop nightly (pull_request) Skipped

Details

ci / android head (pull_request) Successful in 3m20s

Details

ci / api image (pull_request) Successful in 4s

Details

e41eca01a8

The suite fails intermittently with SshConnectionException "The connection
was closed by the remote host", within tens of milliseconds, on whichever
test happens to connect first. It has been seen in CI and reproduces
locally. This raises sshd's MaxStartups in the fixture, which is the most
likely cause and is worth doing regardless.

sshd's compiled-in default is 10:30:100: past ten unauthenticated
connections in flight it refuses new ones at random, thirty percent of the
time, rising to always at a hundred. The image ships the line commented
out, so that default was what ran. xUnit runs test classes in parallel and
most of the classes here open a connection, so ten in flight is reachable
during the opening seconds — and a refusal presents to the client exactly
as observed, because a dropped connection and a server that never answered
are indistinguishable from that end.

◆ IT IS A MITIGATION AND NOT A DEMONSTRATED CURE, AND THE COMMENT SAYS SO.

The flake rate could not be measured. On the Windows development machine
the identical unmodified suite ran 85/85 clean and, an hour later, failed
13 runs out of 15; a Linux container gave 30/30 clean and then failed on
the first run of the next batch. Docker throughput on that host swings far
enough to swamp the effect, so every before/after comparison taken there
was noise — including two that were briefly believed.

It is committed on the narrower argument that it is right either way. A
connection throttle is hardening this suite has no interest in
reproducing: it exists to test an SSH client, not to survive a rate limit,
and a test server that drops connections at random is a bad test server
whether or not it is the cause of this particular flake.

The other candidate was the reload window — pkill returns when SIGHUP is
delivered, not when sshd has finished closing its listeners and re-execing,
so a connection immediately afterwards can be refused the same way. A wait
that required three consecutive banner reads before returning was written
and then removed: it could not be shown to change anything either, and a
fixture carrying two unproven fixes for one symptom is worse than one,
because the next person has to disprove both. Both candidates, and how to
tell them apart with sshd's own log, are recorded in the fixture and in
docs/platform-flags.md.

jaap-jan merged commit 9755b5f6ae into main

2026-08-10 09:56:29 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: DodoTech-Public/DodoSSH#4