Stop a flooding remote growing the process, and let an ended session go
ci / build and test (push) Successful in 3m12s
ci / desktop nightly (push) Successful in 1m16s
ci / api image (push) Successful in 45s
ci / android head (push) Successful in 7m11s

The terminal's memory was bounded by an argument that is not true of the SSH
library underneath it, and the whole of CreditWindow's remark was that argument:
the pump stops reading when the renderer falls behind, SSH's own receive window
closes, the remote sshd blocks on write, and the backpressure reaches the process
producing the output. The middle step does not happen. Renci.SshNet's
Channels.Channel.OnData calls its window adjustment before it raises DataReceived
and re-opens the channel window as soon as it drops below one packet, and
ShellStream's own handler copies the bytes into a System.Net.ArrayBuffer without
blocking and without a ceiling — a buffer that doubles on demand and is not given
back until the stream is disposed. So a reader that stopped reading slowed the
remote down by nothing at all. It only chose whose buffer the flood landed in,
and it chose the one nothing bounds.

Measured against the container this repository already runs sshd in: a reader
that reads nothing while `yes` runs leaves ShellStream holding 8.9 MB after one
second, 26.8 MB after three and 44.3 MB after five, its capacity stepping 16 → 32
→ 64 MB. That is the report of "a massive memory leak on Windows"; it is not a
Windows fault, it is simply where long sessions with heavy output live.

The fix is to read continuously, which is what keeps SSH.NET's buffer empty, and
to put the bound where it can actually be enforced. OutputBacklog is that bound:
a byte-capped queue of unsent output, 4 MiB by default, which drops its oldest
bytes when full and counts what it discarded. Oldest rather than newest, because
the tail of a flood is the prompt somebody is waiting for.

The two bounds are now separate and each is enforced where it means something.
The backlog bounds what is waiting to be sent. The credit window goes on bounding
what has been sent and not yet drawn — the queue inside the WebView that nothing
on this side can measure — and is reserved against in the flush loop rather than
the read loop. Reading is part of neither, because reading was never what the
remote was waiting on.

Dropping is announced rather than silent. A terminal quietly showing a different
transcript from the one the remote sent is worse than one that says it could not
keep up, because the second can be believed. The line goes out under its own
opcode, TerminalServerOpcode.OutputDropped, rather than spliced into the output
stream: what this application says about the transcript is not part of the
transcript, and keeping them apart is also what makes the arithmetic hold — every
byte read is either in an Output payload or counted in BytesDropped. It opens
with an SGR reset, because the cut lands at an arbitrary offset and whatever
escape sequence the remote was in the middle of has been left half-said.

Frames are now capped at 64 KiB and taken straight into the frame buffer. The cap
is the large-object-heap threshold rather than a guess — a quarter-megabyte array
allocated sixty times a second is collected only by a generation 2 pass — and the
direct take removes an allocation and a copy of every byte the terminal shows.

The second half is the one that made the first half's cost persist. Nothing gave
a session back when the remote hung up: TerminalWorkspace kept the entry, and
with it an undisposed ISshConnection — an authenticated socket, its cipher state,
and a keep-alive putting a packet on the wire every thirty seconds — plus the
pump and SSH.NET's read buffer at whatever size the session's busiest moment had
grown it to, for as long as the tab stayed open. A tab somebody typed `exit` into
an hour ago was indistinguishable from a live one in everything except what it
drew. ReleaseAsync disposes both, from AnnounceEndedAsync and after the
announcement: the tab is told first, because tearing down an SSH connection in
front of that would be latency on the one thing the user is watching for. The
dictionary entry stays, because LiveSessionCount, IsSessionLive and the tab's own
ENDED and RECONNECT all read it. SendInputAsync therefore asks about liveness
rather than presence, since the shell channel is gone the moment the remote is.

Two faults introduced on the way here and fixed, both worth naming because
neither would have been visible from the outside:

The read loop left through a condition on its cancellation token, and RunAsync
reads a read loop that returns as the remote having closed the channel. A tab the
user closed would have reported itself as a shell that exited on its own. It
throws instead.

With the reader finished and a renderer that had stopped acknowledging, the flush
loop span: output to send, no credit to send it with, and a wait for more output
that returned at once because there was some. RunAsync never completed, so the tab
was never told and the connection was never released — a session invisible and
permanent. The loop now leaves for the drain as soon as the backlog is complete,
and RunAsync resets the credit window alongside completing it so a flusher already
parked on an acknowledgement that will never come is released.

Four tests asserted the old contract and now assert its reverse, which is the
point rather than a relaxation — WhenCreditRunsOut_ThePumpStopsReading is now
WhenCreditRunsOut_ThePumpKeepsReadingAndTheBacklogHoldsTheLine, and has to prove
both halves at once. The throughput harness gained a counting transport, because
a consumer that copies every frame into a 32 MB list was the slowest thing in the
pipeline and a harness whose consumer cannot keep up measures itself. The fake
shell gained a pace, so "a renderer that keeps up loses nothing" can be asked of a
producer a network could actually deliver rather than of one running at memory
bandwidth.

The test that would have caught this in the first place is
ARunawayRemote_KeepsBeingReadAndStaysInsideTheBacklog, and it needs the real
SSH.NET and the real sshd to say anything: a fake shell hands over bytes when it
is asked, and a real channel hands them over when they arrive. Under the old code
its BytesRead would have stopped at exactly WindowBytes.

121 terminal tests pass over three consecutive runs with no flakes, 105 against
the container, and the app, session, layout and remaining suites are unchanged and
green. Manual checks 1.10a and 1.10b cover the two symptoms a person can see: a
working set that plateaus under `yes`, and a connection that goes when the shell
does rather than when the tab does.
This commit is contained in:
jaap-jan committed 2026-09-04 17:16:23 +02:00
1 parent 597e5ae016
commit 9ead42171c
11 files changed
+1150 -144

No files matched your search

@@ -162,6 +162,84 @@ public sealed class PumpOverRealSshTests(SshServerFixture fixture)
"the strip under the terminal prints this sentence, so an empty one is a blank status line");
}
/// <summary>
/// A remote flooding a renderer that never draws anything is bounded, and the pump keeps reading.
/// </summary>
/// <remarks>
/// <para>
/// ◆ THIS IS THE TEST THE OLD DESIGN WOULD FAIL, AND IT NEEDS A REAL SSH.NET UNDERNEATH IT TO SAY SO.
/// The pump used to stop reading once a renderer had a credit window's worth of unrendered output, on
/// the reasoning that SSH's own receive window would then close and block the remote. It does not:
/// <c>Renci.SshNet.Channels.Channel.OnData</c> adjusts the window as bytes <em>arrive</em>, before it
/// hands them on, so the remote goes on sending at full speed and <c>ShellStream</c> copies every byte
/// into a buffer that doubles and is never given back. Against this test, <c>yes</c> would have been
/// read at <c>BytesRead == WindowBytes</c> exactly and then stopped for ever while the process grew.
/// A fake shell cannot show that, because a fake hands over bytes when it is asked and a real channel
/// hands them over when they arrive.
/// </para>
/// <para>
/// So the assertion is in two halves and needs both. Reading keeps going, which is what keeps SSH.NET's
/// buffer drained; and what this side holds stays inside the backlog, which is what keeps the drained
/// bytes from simply piling up here instead. Either alone is satisfied by a design that leaks.
/// </para>
/// </remarks>
[Fact]
public async Task ARunawayRemote_KeepsBeingReadAndStaysInsideTheBacklog()
{
const int Window = 64 * 1024;
const int Backlog = 512 * 1024;
await using var connection = await ConnectTrustedAsync();
var shell = await connection.OpenShellAsync(
new TerminalSize(100, 30, 1000, 750), TestContext.Current.CancellationToken);
// Deliberately never acknowledges: this is a renderer that has stopped drawing, which is the whole
// condition under examination. CountingTransport only acknowledges when it is given a pump.
var transport = new CountingTransport();
await using var pump = new TerminalSessionPump(
1,
shell,
transport,
TimeProvider.System,
new TerminalPumpOptions
{
FlushInterval = TimeSpan.FromMilliseconds(20),
WindowBytes = Window,
BacklogBytes = Backlog,
});
var run = pump.RunAsync(TestContext.Current.CancellationToken);
await pump.WriteInputAsync("yes dodossh\n"u8.ToArray(), TestContext.Current.CancellationToken);
// Long enough that a pump which had stopped at the window would plainly have stopped, and that one
// which had not would have read many times the backlog's capacity.
var deadline = TimeProvider.System.GetUtcNow() + TimeSpan.FromSeconds(10);
while (TimeProvider.System.GetUtcNow() < deadline && pump.BytesRead < Backlog * 4L)
{
await Task.Delay(100, TestContext.Current.CancellationToken);
pump.BytesPending.ShouldBeLessThanOrEqualTo(
Backlog,
"output is accumulating on this side of the socket, which is the growth the backlog bounds");
}
pump.BytesRead.ShouldBeGreaterThan(
Backlog * 4L,
"the pump stopped reading, so the flood is piling up inside SSH.NET where nothing bounds it");
pump.BytesDropped.ShouldBeGreaterThan(
0,
"nothing was dropped, so something between here and the screen is still holding all of it");
await pump.DisposeAsync();
await Should.NotThrowAsync(() => run.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken));
}
/// <summary>Trusts the relay's identity, then connects through it.</summary>
/// <remarks>
/// Pinned under the relay's own address, which is the honest thing here: what is under examination is a