Public Access
jaap-jan
766fe6aebe
The SSH suite has failed intermittently for months with SshConnectionException "The connection was closed by the remote host", within milliseconds, on whichever class happened to be running. Two previous attempts guessed at the cause and said so honestly; this one has a mechanism and a before/after. ◆ THE CAUSE IS PerSourcePenalties, WHICH THIS SUITE PROVOKES BY DESIGN. OpenSSH 9.8 added per-source penalties and 10.x enables them by default; the image runs 10.3 and its config never mentions the keyword, so the compiled-in default was what ran. A source address that repeatedly disconnects without attempting authentication gets penalised, and while the penalty holds every connection from it is answered with the clear-text line "Not allowed at this time" and then closed. That is exactly the traffic this suite generates. This client's first contact with an unknown host is a connection deliberately refused at the host key — a disconnect with no authentication attempt — so every helper that learns a host key by being turned away first, plus RefusingTheHostKey_AbortsTheConnection and AnUntrustedHost_IsRefusedExactlyAsAShellWouldBe, feeds the penalty counter. Enough of them close together and sshd stops talking to the test host for a while, then starts again. Measured on a fresh container, probing 200 times with connections of that shape: with the image default, the first refusal came back at probe 18 and 183 of the 200 were refused. With PerSourcePenalties no, none of 200 were. That is the before/after the earlier attempts could not produce. It also explains the shape of the failure, which never fitted a throttle. The class that failed lost EVERY connection it made rather than a random few — including the one test that expects a refusal, which passed throughout for the wrong reason — while the classes around it were untouched. That is a window in which the server refuses one source, not a probabilistic drop. Both earlier diagnoses are recorded in the fixture so they are not tried again. MaxStartups was blamed on the reasoning that xUnit runs test classes in parallel, so ten unauthenticated connections would be in flight at once; but every class touching this server shares one collection and xUnit parallelises collections, not classes, so they run one after another and never have more than a connection or two open. The reload window was blamed next, and a wait for the banner was written and removed as unproven — it was unproven because the banner answers perfectly right up until the penalty lands, so a check that stopped at the first "SSH-" ran entirely inside the good part. MaxStartups is kept, on the narrower argument that it is right regardless: a connection throttle is hardening a test server has no business reproducing. Removing it would be a second change riding along with this one. The readiness gate that replaces the reconfigure's silence is a guard rather than a wait. It requires 25 connections answered back to back, which is the specific provocation rather than a soak test: 25 is above the measured threshold of 18 on purpose, and it costs under a second when the setting is off. Ten was tried first and was worse than useless — it sits below the threshold, so it passed against a server that was still penalising. With the fix removed the gate now fails in a minute naming PerSourcePenalties and quoting the server's own "Not allowed at this time", instead of the suite failing later somewhere unrelated. The gate also closes a hole the container's own readiness cannot: a log line and netstat showing :2222 both pass on a container whose sshd has gone, because Docker publishes the port with a host-side proxy that accepts before it has anything to forward to. It is probed from the host rather than with docker exec for the same reason it matters — that is the path the tests take, and penalties are counted per source address. Rejected: patching sshd_config from /custom-cont-init.d to avoid the reload entirely. It looks like the right hook and is not — the container's log puts "sshd is listening on port 2222" before "[custom-init] Files found, executing", so a script there edits a file the running server has already read. It leaves a config that greps correctly and a server behaving as though it were never touched, which is the same trap as patching the wrong one of the image's two config files. Twenty-eight tests failed before that was noticed; the finding is in the fixture. Four consecutive full-solution runs clean, and the SSH suite green on every run since. 1,861 tests, none failing.