Stop a flooding remote growing the process, and let an ended session go
ci / build and test (push) Successful in 3m12s
ci / desktop nightly (push) Successful in 1m16s
ci / api image (push) Successful in 45s
ci / android head (push) Successful in 7m11s

The terminal's memory was bounded by an argument that is not true of the SSH
library underneath it, and the whole of CreditWindow's remark was that argument:
the pump stops reading when the renderer falls behind, SSH's own receive window
closes, the remote sshd blocks on write, and the backpressure reaches the process
producing the output. The middle step does not happen. Renci.SshNet's
Channels.Channel.OnData calls its window adjustment before it raises DataReceived
and re-opens the channel window as soon as it drops below one packet, and
ShellStream's own handler copies the bytes into a System.Net.ArrayBuffer without
blocking and without a ceiling — a buffer that doubles on demand and is not given
back until the stream is disposed. So a reader that stopped reading slowed the
remote down by nothing at all. It only chose whose buffer the flood landed in,
and it chose the one nothing bounds.

Measured against the container this repository already runs sshd in: a reader
that reads nothing while `yes` runs leaves ShellStream holding 8.9 MB after one
second, 26.8 MB after three and 44.3 MB after five, its capacity stepping 16 → 32
→ 64 MB. That is the report of "a massive memory leak on Windows"; it is not a
Windows fault, it is simply where long sessions with heavy output live.

The fix is to read continuously, which is what keeps SSH.NET's buffer empty, and
to put the bound where it can actually be enforced. OutputBacklog is that bound:
a byte-capped queue of unsent output, 4 MiB by default, which drops its oldest
bytes when full and counts what it discarded. Oldest rather than newest, because
the tail of a flood is the prompt somebody is waiting for.

The two bounds are now separate and each is enforced where it means something.
The backlog bounds what is waiting to be sent. The credit window goes on bounding
what has been sent and not yet drawn — the queue inside the WebView that nothing
on this side can measure — and is reserved against in the flush loop rather than
the read loop. Reading is part of neither, because reading was never what the
remote was waiting on.

Dropping is announced rather than silent. A terminal quietly showing a different
transcript from the one the remote sent is worse than one that says it could not
keep up, because the second can be believed. The line goes out under its own
opcode, TerminalServerOpcode.OutputDropped, rather than spliced into the output
stream: what this application says about the transcript is not part of the
transcript, and keeping them apart is also what makes the arithmetic hold — every
byte read is either in an Output payload or counted in BytesDropped. It opens
with an SGR reset, because the cut lands at an arbitrary offset and whatever
escape sequence the remote was in the middle of has been left half-said.

Frames are now capped at 64 KiB and taken straight into the frame buffer. The cap
is the large-object-heap threshold rather than a guess — a quarter-megabyte array
allocated sixty times a second is collected only by a generation 2 pass — and the
direct take removes an allocation and a copy of every byte the terminal shows.

The second half is the one that made the first half's cost persist. Nothing gave
a session back when the remote hung up: TerminalWorkspace kept the entry, and
with it an undisposed ISshConnection — an authenticated socket, its cipher state,
and a keep-alive putting a packet on the wire every thirty seconds — plus the
pump and SSH.NET's read buffer at whatever size the session's busiest moment had
grown it to, for as long as the tab stayed open. A tab somebody typed `exit` into
an hour ago was indistinguishable from a live one in everything except what it
drew. ReleaseAsync disposes both, from AnnounceEndedAsync and after the
announcement: the tab is told first, because tearing down an SSH connection in
front of that would be latency on the one thing the user is watching for. The
dictionary entry stays, because LiveSessionCount, IsSessionLive and the tab's own
ENDED and RECONNECT all read it. SendInputAsync therefore asks about liveness
rather than presence, since the shell channel is gone the moment the remote is.

Two faults introduced on the way here and fixed, both worth naming because
neither would have been visible from the outside:

The read loop left through a condition on its cancellation token, and RunAsync
reads a read loop that returns as the remote having closed the channel. A tab the
user closed would have reported itself as a shell that exited on its own. It
throws instead.

With the reader finished and a renderer that had stopped acknowledging, the flush
loop span: output to send, no credit to send it with, and a wait for more output
that returned at once because there was some. RunAsync never completed, so the tab
was never told and the connection was never released — a session invisible and
permanent. The loop now leaves for the drain as soon as the backlog is complete,
and RunAsync resets the credit window alongside completing it so a flusher already
parked on an acknowledgement that will never come is released.

Four tests asserted the old contract and now assert its reverse, which is the
point rather than a relaxation — WhenCreditRunsOut_ThePumpStopsReading is now
WhenCreditRunsOut_ThePumpKeepsReadingAndTheBacklogHoldsTheLine, and has to prove
both halves at once. The throughput harness gained a counting transport, because
a consumer that copies every frame into a 32 MB list was the slowest thing in the
pipeline and a harness whose consumer cannot keep up measures itself. The fake
shell gained a pace, so "a renderer that keeps up loses nothing" can be asked of a
producer a network could actually deliver rather than of one running at memory
bandwidth.

The test that would have caught this in the first place is
ARunawayRemote_KeepsBeingReadAndStaysInsideTheBacklog, and it needs the real
SSH.NET and the real sshd to say anything: a fake shell hands over bytes when it
is asked, and a real channel hands them over when they arrive. Under the old code
its BytesRead would have stopped at exactly WindowBytes.

121 terminal tests pass over three consecutive runs with no flakes, 105 against
the container, and the app, session, layout and remaining suites are unchanged and
green. Manual checks 1.10a and 1.10b cover the two symptoms a person can see: a
working set that plateaus under `yes`, and a connection that goes when the shell
does rather than when the tab does.
This commit is contained in:
jaap-jan committed 2026-09-04 17:16:23 +02:00
1 parent 597e5ae016
commit 9ead42171c
11 files changed
+1150 -144

No files matched your search

@@ -83,9 +83,9 @@ public sealed class TerminalWorkspaceTests
}
/// <remarks>
/// The case a naive <c>sessions.Count</c> gets wrong. Nothing removes the entry when the remote
/// closes the channel by itself — the session is still in the dictionary, holding a connection that
/// is finished — so counting entries would report a shell that exited as still running. A user
/// The case a naive <c>sessions.Count</c> gets wrong. Nothing removes the entry when the remote closes
/// the channel by itself — what it held is given back, but the entry stays so the tab can go on saying
/// the session ended — so counting entries would report a shell that exited as still running. A user
/// deciding whether it is safe to walk away is the person that lie is told to.
/// </remarks>
[Fact]
@@ -645,6 +645,86 @@ public sealed class TerminalWorkspaceTests
await workspace.SendInputAsync(sessionId, "x"u8.ToArray(), TestContext.Current.CancellationToken));
}
/// <summary>
/// A shell that ended on its own gives its connection back at once, without waiting for the tab.
/// </summary>
/// <remarks>
/// ◆ THE HALF THAT WAS MISSING. Closing a tab disposed the connection and always had; a remote hanging
/// up did not, so a session that ended went on holding an authenticated socket, a keep-alive putting a
/// packet on the wire every thirty seconds, and SSH.NET's read buffer at whatever size the session's
/// busiest moment had grown it to — for as long as the user left the tab open, which for a tab showing
/// the last thing a job said is indefinitely.
/// </remarks>
[Fact]
public async Task ASessionWhoseRemoteHasExited_GivesBackItsConnectionWithoutWaitingForTheTab()
{
// A shell with no output to give: its first read returns 0, which is the remote closing the
// channel, so the pump finishes with nobody having asked it to.
var connections = new FakeConnectionFactory(bytesPerShell: 0);
await using var workspace = CreateWorkspace(connections);
var sessionId = await workspace.OpenSessionAsync(
Request(), TerminalSize.Default, progress: null, TestContext.Current.CancellationToken);
await WaitUntilAsync(() => connections.Connections.TrueForAll(connection => connection.IsDisposed));
var connection = connections.Connections.ShouldHaveSingleItem();
connection.IsConnected.ShouldBeFalse();
connection.Shell.ShouldNotBeNull().IsOpen.ShouldBeFalse();
// The entry itself stays, and has to: the tab is still on screen, still says it ended, and still
// offers a reconnect — all of which read this workspace rather than the tab strip's own memory.
workspace.CreditsFor(sessionId).ShouldNotBeNull();
workspace.IsSessionLive(sessionId).ShouldBeFalse();
}
/// <remarks>
/// The consequence of the release above, and the reason <c>SendInputAsync</c> asks about liveness
/// rather than presence: the shell channel is disposed the moment the remote hangs up, so an accessory
/// key pressed at a tab that ended reaches a disposed stream. It has to be as ordinary as pressing one
/// at a tab that has closed, which is to say nothing happens.
/// </remarks>
[Fact]
public async Task SendingInputToASessionWhoseRemoteHasExited_IsIgnored()
{
var connections = new FakeConnectionFactory(bytesPerShell: 0);
await using var workspace = CreateWorkspace(connections);
var sessionId = await workspace.OpenSessionAsync(
Request(), TerminalSize.Default, progress: null, TestContext.Current.CancellationToken);
await WaitUntilAsync(() => !workspace.IsSessionLive(sessionId));
await Should.NotThrowAsync(async () =>
await workspace.SendInputAsync(sessionId, "x"u8.ToArray(), TestContext.Current.CancellationToken));
}
/// <remarks>
/// Closing a tab over a session that already released itself must still work, because that is the
/// ordinary way a dead tab goes away. Everything the close does is idempotent by construction — the
/// entry is removed under the lock, and both disposals are guarded — and this is the test that says so
/// rather than leaving it to be inferred from three separate classes.
/// </remarks>
[Fact]
public async Task ClosingATabOverASessionThatAlreadyEnded_IsStillSafe()
{
var connections = new FakeConnectionFactory(bytesPerShell: 0);
await using var workspace = CreateWorkspace(connections);
var sessionId = await workspace.OpenSessionAsync(
Request(), TerminalSize.Default, progress: null, TestContext.Current.CancellationToken);
await WaitUntilAsync(() => !workspace.IsSessionLive(sessionId));
await Should.NotThrowAsync(() => workspace.CloseSessionAsync(sessionId));
workspace.CreditsFor(sessionId).ShouldBeNull();
}
/// <param name="connections">
/// How connections are made. The renderer-gate tests never reach it — they stop at the gate — but they
/// take a fake anyway, because reaching a real host from here would make this a network test.