Notice when a shell ends, say which way it ended, and offer the way back
ci / build and test (push) Successful in 3m28s
ci / android head (push) Successful in 4m48s
ci / desktop nightly (push) Successful in 1m17s
ci / api image (push) Successful in 28s

A terminal whose machine rebooted stayed green for ever. Not "went stale and
recovered late" — for ever: the tab claimed CONNECTED, the elapsed timer went
on counting, and the only thing that ever corrected it was closing the
application. That is the defect this fixes, and it turned out to be two
separate failures wearing one symptom.

◆ THE HANG IS IN SSH.NET, AND KEEP-ALIVE ALONE WOULD NOT HAVE FIXED IT.

ShellStream ends a blocked read from exactly two places: its channel being
closed, and the session raising Disconnected. A shell that exits gives the
first. But Session raises Disconnected from one method only — OnDisconnectReceived,
for a server that politely says goodbye — and a socket that simply fails goes
down a different path entirely: RaiseError fires the error event, sets the
disconnecting flag and disposes the socket, without ever raising Disconnected.
So the stream was never disposed, its Monitor.Wait was never pulsed, and the
read blocked until the process ended. The transport was gone, the client knew
the transport was gone, and nothing woke the one thread that was waiting.

That is why the first attempt at a test for this sat at thirty seconds and
failed. Both halves are needed and neither is sufficient:

  - Keep-alive makes the failure HAPPEN. An idle SSH session is a socket
    nobody writes to, and TCP has no way to discover that the far end has gone
    without writing to it — nothing was asked, so nothing could fail. Thirty
    seconds, matching OpenSSH's own ServerAliveInterval examples. It is the
    write that matters and not a reply: SSH_MSG_IGNORE need not be answered,
    so this is traffic rather than a heartbeat with a deadline, and there is
    no timeout invented here to be wrong about a slow link.

  - Disposing the stream from the error handler makes it ARRIVE. Queued to the
    thread pool and not done inline, which is not tidiness: this runs on
    SSH.NET's message-listener thread, and closing a channel waits for the
    server's close — processed by that same thread — so an inline dispose is
    the listener thread waiting on itself.

SFTP gets the keep-alive too. Its operations are request/response and bounded
by ConnectionInfo.Timeout already, so it needs no equivalent wake-up.

◆ exit AND reboot ARE TOLD APART BY A LATCH, BECAUSE THE OBVIOUS WAY IS A RACE.

Wait for the read to end and then ask whether the transport is still up, and
against a real server the answer is wrong about half the time: this was the
only channel on the connection, so sshd hangs up a millisecond after closing
it. By the time anything could ask, the transport is gone either way — and a
poll would report every ordinary `exit` as a dropped connection.

So ISshShellSession.TransportFailure is set at the moment SSH.NET raises the
failure, which its own ordering puts strictly before the disconnect that wakes
the read. A read that has just returned zero consults a settled value: set if
the transport failed, still null if the remote merely closed the channel. The
pump turns that into a TerminalSessionOutcome — RemoteClosed, ConnectionLost,
ClosedLocally — carried on SessionEnded, which used to hand over a session id
and nothing else. A subscriber cannot work this out afterwards; by the time it
hears, there is no transport left to interrogate.

Deliberately no fourth value for "the remote rebooted". Nothing on the wire
distinguishes a reboot from a cable being pulled, and a name this client
cannot check would be a guess dressed as a fact.

◆ WHAT IS DRAWN, AND WHY IT IS NOT DRAWN OVER THE PANE.

The status bar says DISCONNECTED or SHELL EXITED where it used to say NOT
CONNECTED for both — a word that is honest about a tab still dialling and
useless about one whose shell has stopped. Red for the first and grey for the
second: one is a machine that stopped answering and the other is work
finished, and crying wolf on the ordinary end of every session is its own bug.
The tab strip's dot gains the same fourth state, on both heads.

Under the terminal, a strip carrying the reason and two buttons. It takes its
own Auto grid row rather than overlaying the renderer, and that is forced
rather than chosen: the WebView is a native child view composited above
everything Avalonia draws, so the alternative is collapsing the terminal — and
the scrollback is precisely what the user wants to read, because the last
thing the remote said is usually why it went. terminal.js already writes the
same sentence into the pane; this is the copy that does not scroll away.

RECONNECT reuses the tab in place, which is what the word means to somebody
who pressed it: the terminal they were working in comes back, where it was.
It dials through the vault's ordinary connect path, so every refusal — a key
that has left the keychain, an unknown host key, a group default that has
changed since — is decided in the one place that already decides them; what
this adds is which tab the attempt belongs to. The old session is closed as
part of it, because a dead shell still holds an authenticated connection this
process has stopped reading, and its pane still holds an xterm, a scrollback
and a WebGL context, of which a browser hands out about sixteen.

Two refusals it makes rather than pretending: a host reached by typing an
address has no keychain entry to redial, and one that wants a password typed
goes to the keychain with the sheet raised rather than being dialled with an
empty box. Both say so on the strip where the button was pressed, not only in
the vault's status on a screen that surface is not.

TRY AGAIN appears on a refused connection for the same reason, and it is worth
as much: the commonest refusal is a host that was not up yet.

◆ WHAT IS ACTUALLY VERIFIED.

Two tests against a real sshd in a container. `exit` is reported as the remote
closing the session — the one a polling implementation fails, and a fake shell
cannot show it because a fake has no server behind it deciding to hang up. And
a live shell losing its transport is reported as a lost connection, produced
by dialling through a plain TCP relay the test owns and then cutting it; the
container is shared by the collection, so killing sshd was never an option.

The first draft of that second test used LoopbackProxyTests' SOCKS forward and
timed out: disconnecting the carrier of a ForwardedPortDynamic does not close
the loopback socket it accepted. Recorded in the test, because it is a
property of SSH.NET's forwarder rather than of anything here, and it is not
what a dying machine looks like.

What no test reproduces is the waiting. A relay that closes sends a FIN, so
the socket reports itself at once; a machine switched off sends nothing, and
finding that out is the keep-alive's business and the OS's retransmit budget's
— neither of which a test can hurry along without a clock this code does not
own. A reboot is prompt in practice because the machine comes back and RSTs.

Beside those: three pump outcomes, three workspace announcements, and seven
shell-flow tests covering the drop, the clean exit, reuse of the tab, retry
after a refusal, and both ways a reconnect can honestly come to nothing.

Also fixed in passing: SshNetShellSession.IsOpen read ShellStream.CanRead,
which is documented as always true — the stream stays readable after disposal
so buffered output can be drained — so it had answered "yes" for every shell
that ever existed. It reads CanWrite now. And MainWindowViewModel's session-
ended handler moved from Dispatcher.UIThread.Post to the injected post
delegate, which is what makes any of the shell half testable at all; the
remark listing the three unreachable posts is one shorter.
This commit is contained in:
jaap-jan committed 2026-08-24 10:36:15 +02:00
1 parent c2c860b4f2
commit e2733e5cbc
21 files changed
+1755 -43

No files matched your search

+53 -9
View File
@@ -29,6 +29,14 @@ internal sealed class FakeSshConnectionFactory : ISshConnectionFactory, ISftpSes
/// <summary>Requests this factory was asked for, in order.</summary>
internal List<SshConnectionRequest> Requests { get; } = [];
/// <summary>Every shell this factory has opened, in order.</summary>
/// <remarks>
/// Exists so a test can reach one and drop it — see <see cref="FakeSshShellSession.Drop"/>. There was no
/// reason to hold them while the only ending a session had was the tab being closed, which the shell can
/// ask for directly.
/// </remarks>
internal List<FakeSshShellSession> Shells { get; } = [];
/// <summary>Requests for a file-transfer session, in order.</summary>
/// <remarks>
/// Kept apart from <see cref="Requests"/> deliberately: file transfer is a separate connection, and a
@@ -65,7 +73,7 @@ internal sealed class FakeSshConnectionFactory : ISshConnectionFactory, ISftpSes
progress?.Report(SshConnectionPhase.CheckingHostKey);
progress?.Report(SshConnectionPhase.Authenticating);
return new FakeSshConnection(request);
return new FakeSshConnection(request, Shells);
}
/// <summary>The file-transfer session this factory last handed out, or null when it never has.</summary>
@@ -188,7 +196,8 @@ internal sealed class FakeSftpSession(SshConnectionRequest request) : ISftpSessi
}
}
internal sealed class FakeSshConnection(SshConnectionRequest request) : ISshConnection
internal sealed class FakeSshConnection(SshConnectionRequest request, List<FakeSshShellSession> shells)
: ISshConnection
{
/// <inheritdoc />
public bool IsConnected { get; private set; } = true;
@@ -203,8 +212,13 @@ internal sealed class FakeSshConnection(SshConnectionRequest request) : ISshConn
/// <inheritdoc />
public Task<ISshShellSession> OpenShellAsync(
TerminalSize size,
CancellationToken cancellationToken) =>
Task.FromResult<ISshShellSession>(new FakeSshShellSession());
CancellationToken cancellationToken)
{
var shell = new FakeSshShellSession();
shells.Add(shell);
return Task.FromResult<ISshShellSession>(shell);
}
/// <inheritdoc />
public ValueTask DisposeAsync()
@@ -214,11 +228,11 @@ internal sealed class FakeSshConnection(SshConnectionRequest request) : ISshConn
}
}
/// <summary>A shell that is open, silent and never closes on its own.</summary>
/// <summary>A shell that is open, silent and closes only when it is told to.</summary>
/// <remarks>
/// <see cref="ReadAsync"/> blocks rather than returning 0. Returning 0 means the remote closed the
/// channel, which would end the session the moment it was opened and make the test assert against a
/// connection that had already gone.
/// connection that had already gone. <see cref="Drop"/> and <see cref="Exit"/> are the two ways out.
/// </remarks>
internal sealed class FakeSshShellSession : ISshShellSession
{
@@ -227,13 +241,43 @@ internal sealed class FakeSshShellSession : ISshShellSession
/// <inheritdoc />
public bool IsOpen { get; private set; } = true;
/// <inheritdoc />
public string? TransportFailure { get; private set; }
/// <summary>
/// Ends this shell the way a rebooted machine does: the read stops, with a transport failure latched.
/// </summary>
/// <remarks>
/// Latched before the read is released, which is the ordering the real session promises and the whole
/// reason the distinction is race-free — see <see cref="ISshShellSession.TransportFailure"/>. A fake
/// that set it afterwards would be testing a guarantee the product does not make.
/// </remarks>
internal void Drop(string reason)
{
TransportFailure = reason;
closed.Cancel();
}
/// <summary>Ends this shell the way <c>exit</c> does: the read stops, with nothing wrong underneath.</summary>
internal void Exit() => closed.Cancel();
/// <inheritdoc />
public async ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken)
{
using var linked = CancellationTokenSource.CreateLinkedTokenSource(
cancellationToken, closed.Token);
try
{
await Task.Delay(Timeout.InfiniteTimeSpan, closed.Token).ConfigureAwait(false);
}
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
{
// This shell ending, not the pump being asked to stop. Reported as end of stream rather than
// rethrown, because that is what a real read does for both of the endings above: the channel
// stops producing bytes, and what happened to the transport is told by the latch, not by an
// exception coming out of here.
return 0;
}
await Task.Delay(Timeout.InfiniteTimeSpan, linked.Token).ConfigureAwait(false);
cancellationToken.ThrowIfCancellationRequested();
return 0;
}
@@ -1126,6 +1126,270 @@ public sealed class ShellFlowTests : IAsyncLifetime
// directly beside the pre-existing `RefreshConnectedHosts();` — is covered indirectly: it is the same
// VaultViewModel.RefreshLastConnectedAsync the activation test above already exercises.
// ---- A session that stops: the status it gets, and the way back ----
/// <remarks>
/// <para>
/// ◆ THE REBOOT CASE, as far as this shell can see it. A machine going away is a transport failure
/// latched under a read that then reports the end of the shell — see <c>ISshShellSession.TransportFailure</c>
/// — and what has to come of it here is that the tab stops claiming to be connected and says which of the
/// two endings it was.
/// </para>
/// <para>
/// The words are asserted rather than only the flags. DISCONNECTED against SHELL EXITED is the whole
/// difference a user reads, and a status bar that had a bool but printed the same thing for both would
/// pass a test written against <c>IsLive</c> alone.
/// </para>
/// </remarks>
[Fact]
public async Task ASessionWhoseConnectionBreaks_SaysSoAndOffersTheWayBack()
{
var vault = await ReadyToConnectAsync();
await using var renderer = await FakeRenderer.AttachAsync(workspace, Token);
await vault.ConnectCommand.ExecuteAsync(null);
var tab = shell.Tabs.ShouldHaveSingleItem();
tab.IsLive.ShouldBeTrue();
shell.IsSessionConnected.ShouldBeTrue();
ssh.Shells.ShouldHaveSingleItem().Drop("An established connection was aborted by the server.");
await EventuallyAsync(() => !tab.IsLive, "the dropped shell should have ended the session");
tab.EndReason.ShouldBe(SshSessionEnd.ConnectionLost);
tab.IsDropped.ShouldBeTrue("the strip draws a dropped connection differently from a clean exit");
shell.IsSessionConnected.ShouldBeFalse();
shell.SessionEndedText.ShouldBe("DISCONNECTED");
shell.SessionEndedDetail.ShouldBe("An established connection was aborted by the server.");
shell.IsSessionDropped.ShouldBeTrue();
shell.IsSessionIdle.ShouldBeFalse("NOT CONNECTED must not be drawn beside a named ending");
shell.CanReconnectSelectedTab.ShouldBeTrue("the host is still in the keychain to dial again");
// The pane stays, which is the whole reason a dead tab is kept: the last thing the remote said is
// usually why it went.
tab.HasSession.ShouldBeTrue();
shell.IsTerminalShowing.ShouldBeTrue();
}
/// <remarks>
/// ◆ THE <c>exit</c> CASE, and the reason it is a separate test from the one above rather than an extra
/// assertion inside it: the two differ by a latched transport failure and nothing else, all the way down
/// to the read that ends them, and the point is that they do not report the same thing. A shell somebody
/// exited is work finished — grey, no alarm — and calling it a dropped connection would be crying wolf on
/// the ordinary end of every session.
/// </remarks>
[Fact]
public async Task AShellThatExits_SaysThatRatherThanClaimingTheConnectionBroke()
{
var vault = await ReadyToConnectAsync();
await using var renderer = await FakeRenderer.AttachAsync(workspace, Token);
await vault.ConnectCommand.ExecuteAsync(null);
var tab = shell.Tabs.ShouldHaveSingleItem();
ssh.Shells.ShouldHaveSingleItem().Exit();
await EventuallyAsync(() => !tab.IsLive, "the exiting shell should have ended the session");
tab.EndReason.ShouldBe(SshSessionEnd.RemoteClosed);
tab.IsDropped.ShouldBeFalse();
shell.SessionEndedText.ShouldBe("SHELL EXITED");
shell.IsSessionDropped.ShouldBeFalse();
// Still worth offering, because the tab and the machine are both still there — what it is not is an
// alarm.
shell.CanReconnectSelectedTab.ShouldBeTrue();
}
/// <remarks>
/// <para>
/// The tab is reused rather than a second one appearing beside it. That is what reconnecting means to
/// somebody who pressed the button: the terminal they were working in comes back, in the place it was.
/// </para>
/// <para>
/// The old session is released as part of it, which the assertion on the connection's disposal stands
/// for. A dead shell still holds an authenticated connection this process has stopped reading, and its
/// pane still holds an xterm, a scrollback and a WebGL context — of which a browser hands out about
/// sixteen. Leaking one of each per reconnect would be invisible until a working day's worth of them ran
/// the renderer out.
/// </para>
/// </remarks>
[Fact]
public async Task ReconnectingADroppedTab_DialsAgainInTheSameTab()
{
var vault = await ReadyToConnectAsync();
// Bound to a key, which is what makes this host redialable without anybody present: a host that
// wants a password typed is diverted to the keychain instead, and has a test of its own below.
await AddKeyAsync(vault, "deploy");
await BindKeyAsync(vault, vault.Hosts[0], vault.Keys[0].EntityId);
await using var renderer = await FakeRenderer.AttachAsync(workspace, Token);
await vault.ConnectCommand.ExecuteAsync(null);
var tab = shell.Tabs.ShouldHaveSingleItem();
var firstSession = tab.SessionId;
ssh.Shells[0].Drop("An established connection was aborted by the server.");
await EventuallyAsync(() => !tab.IsLive, "the dropped shell should have ended the session");
await shell.ReconnectTabCommand.ExecuteAsync(tab);
shell.Tabs.ShouldHaveSingleItem().ShouldBe(tab, "a reconnect reuses the tab it was asked for");
tab.IsLive.ShouldBeTrue();
tab.SessionId.ShouldNotBe(firstSession, "a reconnect is a new session, not the old one resumed");
tab.EndReason.ShouldBeNull("the ending belonged to the session that has been replaced");
shell.SessionEndedText.ShouldBeNull();
shell.IsSessionConnected.ShouldBeTrue();
shell.SelectedTab.ShouldBe(tab);
ssh.Requests.Count.ShouldBe(2, "the vault's own connect path was used, not a second one");
await EventuallyAsync(
() => !workspace.IsSessionLive(firstSession),
"the session that was replaced should have been released with its pane");
}
/// <remarks>
/// A connection that was refused is worth retrying for the same reason a dropped one is, and the
/// commonest refusal — a machine that was not up yet — is the one most likely to succeed a minute later.
/// The card's button is the same command, which is what makes this worth asserting here rather than
/// trusting to two code paths that happen to agree today.
/// </remarks>
[Fact]
public async Task ARefusedConnection_CanBeRetriedInItsOwnTab()
{
var vault = await ReadyToConnectAsync();
await AddKeyAsync(vault, "deploy");
await BindKeyAsync(vault, vault.Hosts[0], vault.Keys[0].EntityId);
await using var renderer = await FakeRenderer.AttachAsync(workspace, Token);
ssh.Failure = new IOException("The host did not answer.");
await vault.ConnectCommand.ExecuteAsync(null);
var tab = shell.Tabs.ShouldHaveSingleItem();
tab.IsFailed.ShouldBeTrue(tab.Status);
shell.CanReconnectSelectedTab.ShouldBeTrue();
ssh.Failure = null;
await shell.ReconnectTabCommand.ExecuteAsync(tab);
shell.Tabs.ShouldHaveSingleItem().ShouldBe(tab);
tab.IsLive.ShouldBeTrue();
tab.HasSession.ShouldBeTrue();
}
/// <remarks>
/// <para>
/// A host that wants a password typed cannot be redialled on its own: this client does not keep the
/// password, so dialling with an empty box would be an authentication failure dressed up as a reconnect.
/// It goes to the keychain with the password sheet raised instead, which is the one screen that sheet is
/// drawn on.
/// </para>
/// <para>
/// The tab is deliberately left as it was rather than put back to connecting. Nothing has been dialled
/// yet — it waits on somebody typing — and a tab that had thrown away its ending and its pane to sit on a
/// step list nothing was advancing would be the worse of the two wrongs.
/// </para>
/// </remarks>
[Fact]
public async Task ReconnectingAHostThatWantsAPassword_AsksForOneRatherThanDiallingWithout()
{
var vault = await ReadyToConnectAsync();
await using var renderer = await FakeRenderer.AttachAsync(workspace, Token);
await vault.ConnectCommand.ExecuteAsync(null);
var tab = shell.Tabs.ShouldHaveSingleItem();
ssh.Shells[0].Drop("An established connection was aborted by the server.");
await EventuallyAsync(() => !tab.IsLive, "the dropped shell should have ended the session");
await shell.ReconnectTabCommand.ExecuteAsync(tab);
vault.IsAskingForConnectPassword.ShouldBeTrue();
shell.Screen.ShouldBe(ShellScreen.Hosts, "the password sheet is drawn there and nowhere else");
ssh.Requests.ShouldHaveSingleItem("nothing should have been dialled without the password");
tab.HasSession.ShouldBeTrue("the tab keeps its pane while it waits on somebody typing");
}
/// <remarks>
/// A machine reached by typing an address has no keychain entry, so there is nothing to dial again with —
/// this client would need the password, which it does not keep. The button is drawn disabled rather than
/// hidden, so the refusal has to be a fact the shell states rather than an absence.
/// </remarks>
[Fact]
public async Task ATabWhoseHostIsNotInTheKeychain_IsNotOfferedAReconnect()
{
var vault = await ReadyToConnectAsync();
await using var renderer = await FakeRenderer.AttachAsync(workspace, Token);
await vault.ConnectCommand.ExecuteAsync(null);
var tab = shell.Tabs.ShouldHaveSingleItem();
ssh.Shells[0].Drop("An established connection was aborted by the server.");
await EventuallyAsync(() => !tab.IsLive, "the dropped shell should have ended the session");
// The host leaves, as deleting it or locking the vault would take it. The tab outlives the vault
// that opened it, which is why this state is reachable at all.
vault.Hosts.Clear();
shell.CanReconnectSelectedTab.ShouldBeFalse();
await shell.ReconnectTabCommand.ExecuteAsync(tab);
ssh.Requests.ShouldHaveSingleItem("nothing should have been dialled");
// And it says so where the button that was pressed is, rather than only in the vault's own status on
// a screen this surface is not.
shell.SessionEndedDetail.ShouldNotBeNull().ShouldContain("not in this keychain");
shell.SessionEndedText.ShouldBe("DISCONNECTED", "the ending it is still in is unchanged");
}
/// <remarks>
/// The other way a reconnect can come to nothing: the host is still there, but what it authenticates
/// with is not. The vault refuses before it dials — which is the whole reason a reconnect goes through
/// the vault's own connect path rather than reaching past it — and the refusal has to arrive where the
/// button was pressed, because the vault draws its own status on the keychain screen.
/// </remarks>
[Fact]
public async Task AReconnectTheVaultRefusesBeforeDialling_SaysWhyOnTheTab()
{
var vault = await ReadyToConnectAsync();
await AddKeyAsync(vault, "deploy");
await BindKeyAsync(vault, vault.Hosts[0], vault.Keys[0].EntityId);
await using var renderer = await FakeRenderer.AttachAsync(workspace, Token);
await vault.ConnectCommand.ExecuteAsync(null);
var tab = shell.Tabs.ShouldHaveSingleItem();
ssh.Shells[0].Drop("An established connection was aborted by the server.");
await EventuallyAsync(() => !tab.IsLive, "the dropped shell should have ended the session");
// The key the host is bound to leaves, which is a binding that no longer resolves.
vault.Keys.Clear();
await shell.ReconnectTabCommand.ExecuteAsync(tab);
ssh.Requests.ShouldHaveSingleItem("a binding that cannot be resolved must not be dialled with");
shell.SessionEndedDetail.ShouldNotBeNull().ShouldContain("SSH key");
tab.HasSession.ShouldBeTrue("the tab keeps the pane it had; there was nothing to replace it with");
}
/// <remarks>
/// The phone's connect menu is drawn over the terminal's own rectangle, so it obeys the rule the palette
/// does: whatever covers the renderer collapses it instead. The surface stays, because the bar the menu
@@ -1,3 +1,5 @@
using System.Net;
using System.Net.Sockets;
using System.Text;
using DodoSSH.Client.Terminal;
@@ -34,6 +36,253 @@ public sealed class PumpOverRealSshTests(SshServerFixture fixture)
return await factory.ConnectAsync(request, progress: null, TestContext.Current.CancellationToken);
}
/// <summary>
/// <c>exit</c> against a real sshd is reported as the remote closing the session, not as a lost
/// connection.
/// </summary>
/// <remarks>
/// <para>
/// ◆ THIS IS THE TEST THE RACE WOULD FAIL. The obvious way to tell an ordinary exit from a dropped
/// connection is to wait for the read to end and then ask whether the transport is still up — and against
/// a real server it is wrong most of the time, because this was the only channel on the connection and
/// sshd hangs up within a millisecond of closing it. By the time anything here could ask, the transport
/// is gone. A fake shell cannot show that: it has no server behind it deciding to disconnect.
/// </para>
/// <para>
/// What makes the answer settled instead is the failure being latched when it happens rather than
/// inferred afterwards — see <see cref="ISshShellSession.TransportFailure"/>. Nothing latches one here,
/// because nothing failed.
/// </para>
/// </remarks>
[Fact]
public async Task AShellThatExits_IsReportedAsTheRemoteClosingItRatherThanAsALostConnection()
{
await using var connection = await ConnectTrustedAsync();
var shell = await connection.OpenShellAsync(
new TerminalSize(100, 30, 1000, 750), TestContext.Current.CancellationToken);
var transport = new CountingTransport();
await using var pump = new TerminalSessionPump(
1,
shell,
transport,
TimeProvider.System,
new TerminalPumpOptions { FlushInterval = TimeSpan.FromMilliseconds(20) });
transport.Pump = pump;
var run = pump.RunAsync(TestContext.Current.CancellationToken);
await pump.WriteInputAsync("exit\n"u8.ToArray(), TestContext.Current.CancellationToken);
// No disposal and no cancellation: the run has to end because the shell did, which is the whole
// point. A timeout here would mean the session hangs on `exit`.
await run.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken);
pump.Outcome.End.ShouldBe(
SshSessionEnd.RemoteClosed,
$"exit is not a dropped connection; got <{pump.Outcome.Description}>");
}
/// <summary>
/// A transport that goes away under a live shell is reported as a lost connection, and the read does not
/// hang.
/// </summary>
/// <remarks>
/// <para>
/// ◆ THE REBOOT CASE, produced without rebooting anything. The connection is dialled through a plain TCP
/// relay this test owns and then the relay is torn down, which takes the socket out from under a live
/// shell exactly as a machine going down does. The container is shared by every test in this collection,
/// so killing <c>sshd</c> itself was never an option; killing something this test owns is.
/// </para>
/// <para>
/// <b>What is not reproduced here is the waiting.</b> A relay that is closed sends a FIN, so the socket
/// reports itself at once; a machine that has been switched off sends nothing at all, and finding that
/// out requires writing to it — which is what the keep-alive in <see cref="SshNetConnectionFactory"/>
/// exists to do, and which no test can hurry along without a clock this code does not own. So this covers
/// the half that can be checked, and it is the half everything above the socket depends on: a transport
/// that has failed is <em>reported as one</em> rather than being called an ordinary exit or leaving the
/// read blocked for ever.
/// </para>
/// <para>
/// A relay rather than the SOCKS forward <c>LoopbackProxyTests</c> uses, which was the first attempt:
/// disconnecting the carrier of a <c>ForwardedPortDynamic</c> does not close the loopback socket it
/// accepted, so the inner connection was left holding a socket with nothing behind it and the read did
/// not return within thirty seconds. That is a property of SSH.NET's forwarder rather than of anything
/// this repository ships, and it is not what a dying machine looks like.
/// </para>
/// </remarks>
[Fact]
public async Task AShellWhoseTransportDies_IsReportedAsALostConnection()
{
using var relay = new TcpRelay(fixture.Host, fixture.Port);
await using var connection = await ConnectThroughAsync(relay);
var shell = await connection.OpenShellAsync(
new TerminalSize(100, 30, 1000, 750), TestContext.Current.CancellationToken);
var transport = new CountingTransport();
await using var pump = new TerminalSessionPump(
1,
shell,
transport,
TimeProvider.System,
new TerminalPumpOptions { FlushInterval = TimeSpan.FromMilliseconds(20) });
transport.Pump = pump;
var run = pump.RunAsync(TestContext.Current.CancellationToken);
// The shell is up and talking before anything is broken, so what follows is a live session losing its
// transport rather than one that never started.
var deadline = TimeProvider.System.GetUtcNow() + TimeSpan.FromSeconds(20);
while (TimeProvider.System.GetUtcNow() < deadline && pump.BytesRead == 0)
{
await Task.Delay(50, TestContext.Current.CancellationToken);
}
pump.BytesRead.ShouldBeGreaterThan(0, "the shell should have produced a prompt before the relay goes");
// And the ground goes out from under it.
relay.Kill();
// No disposal and no cancellation: the run has to end because the transport did.
await run.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken);
pump.Outcome.End.ShouldBe(
SshSessionEnd.ConnectionLost,
$"a transport that died is not a shell that exited; got <{pump.Outcome.Description}>");
pump.Outcome.Description.ShouldNotBeNullOrWhiteSpace(
"the strip under the terminal prints this sentence, so an empty one is a blank status line");
}
/// <summary>Trusts the relay's identity, then connects through it.</summary>
/// <remarks>
/// Pinned under the relay's own address, which is the honest thing here: what is under examination is a
/// transport dying, and this store has one entry in it either way.
/// </remarks>
private static async Task<ISshConnection> ConnectThroughAsync(TcpRelay relay)
{
var knownHosts = new InMemoryKnownHostStore();
var factory = new SshNetConnectionFactory(knownHosts);
var request = new SshConnectionRequest(
"127.0.0.1",
relay.Port,
SshServerFixture.Username,
new SshPasswordCredential(SshServerFixture.Password));
var unknown = await Should.ThrowAsync<SshHostKeyUnknownException>(async () =>
await factory.ConnectAsync(request, progress: null, TestContext.Current.CancellationToken));
await knownHosts.TrustAsync(unknown.Presentation, TestContext.Current.CancellationToken);
return await factory.ConnectAsync(request, progress: null, TestContext.Current.CancellationToken);
}
/// <summary>A loopback TCP forwarder that can be cut, standing in for a machine that goes away.</summary>
/// <remarks>
/// Deliberately dumb: it accepts one connection, opens one to the far end and copies bytes both ways
/// until something stops. SSH is opaque to it, which is the point — what is being simulated is the wire
/// disappearing, and a relay that understood the protocol could not simulate that without also deciding
/// how to say goodbye.
/// </remarks>
private sealed class TcpRelay : IDisposable
{
private readonly TcpListener listener;
private readonly CancellationTokenSource lifetime = new();
private readonly List<TcpClient> sockets = [];
private readonly Lock gate = new();
internal TcpRelay(string host, int port)
{
listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
Port = ((IPEndPoint)listener.LocalEndpoint).Port;
_ = AcceptAsync(host, port);
}
/// <summary>The loopback port to dial instead of the far end's.</summary>
internal int Port { get; }
/// <summary>
/// Drops every socket, in both directions, without warning either end first.
/// </summary>
/// <remarks>
/// Disposing the sockets is what closes them, and closing is what puts a FIN on the wire. Nothing
/// here speaks to the SSH session running over it, which is the whole idea: this is the network
/// going away, not a disconnect being negotiated.
/// </remarks>
internal void Kill()
{
lifetime.Cancel();
listener.Stop();
lock (gate)
{
foreach (var socket in sockets)
{
socket.Dispose();
}
sockets.Clear();
}
}
public void Dispose()
{
Kill();
lifetime.Dispose();
}
private async Task AcceptAsync(string host, int port)
{
try
{
while (!lifetime.Token.IsCancellationRequested)
{
var inbound = await listener.AcceptTcpClientAsync(lifetime.Token).ConfigureAwait(false);
var outbound = new TcpClient();
await outbound.ConnectAsync(host, port, lifetime.Token).ConfigureAwait(false);
lock (gate)
{
sockets.Add(inbound);
sockets.Add(outbound);
}
_ = CopyAsync(inbound, outbound);
_ = CopyAsync(outbound, inbound);
}
}
catch (Exception exception) when (exception is not OutOfMemoryException)
{
// Killed, or the listener stopped. There is no caller on this thread to hand anything to.
}
}
private async Task CopyAsync(TcpClient from, TcpClient to)
{
try
{
await from.GetStream().CopyToAsync(to.GetStream(), lifetime.Token).ConfigureAwait(false);
}
catch (Exception exception) when (exception is not OutOfMemoryException)
{
// Either end going is the ordinary way this ends, and is what the test is arranging.
}
}
}
[Fact]
public async Task ThePump_ForwardsRealShellOutput()
{
@@ -35,6 +35,14 @@ internal sealed class FakeShellSession : ISshShellSession
/// <inheritdoc />
public bool IsOpen { get; private set; } = true;
/// <inheritdoc />
/// <remarks>
/// Settable, because it is the whole of what tells a shell that exited apart from one whose connection
/// broke — a test sets it for the second case and leaves it null for the first, which is exactly what the
/// real session's latch does off SSH.NET's error event.
/// </remarks>
public string? TransportFailure { get; set; }
/// <summary>Reads issued against this session, to detect a pump that kept reading.</summary>
public int ReadCount { get; private set; }
@@ -133,6 +141,9 @@ internal sealed class FakeShellSession : ISshShellSession
internal sealed class FakeConnectionFactory(long bytesPerShell = long.MaxValue, bool blockShellReads = false)
: ISshConnectionFactory
{
/// <summary>Armed on every shell handed out, so a workspace-level test can produce a dropped connection.</summary>
internal string? TransportFailure { get; set; }
/// <summary>Connections handed out, in order.</summary>
internal List<FakeConnection> Connections { get; } = [];
@@ -148,7 +159,11 @@ internal sealed class FakeConnectionFactory(long bytesPerShell = long.MaxValue,
progress?.Report(SshConnectionPhase.Reaching);
progress?.Report(SshConnectionPhase.Authenticating);
var connection = new FakeConnection(request, bytesPerShell, blockShellReads);
var connection = new FakeConnection(request, bytesPerShell, blockShellReads)
{
TransportFailure = TransportFailure,
};
Connections.Add(connection);
return Task.FromResult<ISshConnection>(connection);
@@ -172,13 +187,17 @@ internal sealed class FakeConnection(SshConnectionRequest request, long bytesPer
/// <summary>The shell this connection opened, if it opened one.</summary>
internal FakeShellSession? Shell { get; private set; }
/// <inheritdoc cref="ISshShellSession.TransportFailure" />
/// <remarks>Handed to whatever shell this connection opens, so the arming happens before the pump runs.</remarks>
internal string? TransportFailure { get; init; }
/// <summary>Whether the connection was disposed, which is what closing a session must do.</summary>
internal bool IsDisposed { get; private set; }
/// <inheritdoc />
public Task<ISshShellSession> OpenShellAsync(TerminalSize size, CancellationToken cancellationToken)
{
Shell = new FakeShellSession(bytesPerShell, blockShellReads);
Shell = new FakeShellSession(bytesPerShell, blockShellReads) { TransportFailure = TransportFailure };
return Task.FromResult<ISshShellSession>(Shell);
}
@@ -217,6 +217,80 @@ public sealed class TerminalSessionPumpTests
System.Text.Encoding.UTF8.GetString(payload).ShouldNotBeNullOrWhiteSpace();
}
/// <remarks>
/// <c>exit</c>, as the pump sees it: the read loop runs out and nothing latched a transport failure on
/// the way. This is the ending that must not be reported as a dropped connection, because it is the
/// ordinary end of a session and there is nothing wrong with the machine.
/// </remarks>
[Fact]
public async Task WhenTheShellExits_TheOutcomeIsTheRemoteClosingIt()
{
await using var session = new FakeShellSession(bytesToProduce: 32);
var transport = new RecordingTransport();
await using var pump = CreatePump(session, transport);
transport.AutoAcknowledge = pump;
await pump.RunAsync(TestContext.Current.CancellationToken);
pump.Outcome.End.ShouldBe(SshSessionEnd.RemoteClosed);
pump.Outcome.Description.ShouldBe("The remote closed the session.");
}
/// <remarks>
/// ◆ A reboot, as the pump sees it. The shell's read ends exactly as it does for <c>exit</c> — there is
/// no separate signal on that path — so the only thing telling the two apart is the failure the session
/// latched off the transport before the read was woken. Without it, a machine that went away is reported
/// as somebody having typed <c>exit</c>, which is the state this whole wave exists to stop drawing.
/// </remarks>
[Fact]
public async Task WhenTheTransportFailed_TheOutcomeIsALostConnectionCarryingWhatFailed()
{
await using var session = new FakeShellSession(bytesToProduce: 32)
{
TransportFailure = "An established connection was aborted by the server.",
};
var transport = new RecordingTransport();
await using var pump = CreatePump(session, transport);
transport.AutoAcknowledge = pump;
await pump.RunAsync(TestContext.Current.CancellationToken);
pump.Outcome.End.ShouldBe(SshSessionEnd.ConnectionLost);
pump.Outcome.Description.ShouldBe("An established connection was aborted by the server.");
// And the renderer is told the same sentence, so the line in the scrollback and whatever the shell
// draws around the tab cannot disagree.
TerminalFrame.TryRead(transport.Frames[^1], out var opcode, out _, out var payload).ShouldBeTrue();
opcode.ShouldBe((byte)TerminalServerOpcode.SessionClosed);
System.Text.Encoding.UTF8.GetString(payload)
.ShouldBe("An established connection was aborted by the server.");
}
/// <remarks>
/// The third ending, and the one nothing draws a status for: this client asked. A tab being closed and a
/// reconnect starting both arrive here, and neither is news to whoever caused it.
/// </remarks>
[Fact]
public async Task WhenThisClientEndsIt_TheOutcomeSaysSoRatherThanBlamingTheRemote()
{
await using var session = new FakeShellSession(bytesToProduce: long.MaxValue);
var transport = new RecordingTransport();
var pump = CreatePump(session, transport);
transport.AutoAcknowledge = pump;
var run = pump.RunAsync(TestContext.Current.CancellationToken);
await pump.DisposeAsync();
await run;
pump.Outcome.End.ShouldBe(SshSessionEnd.ClosedLocally);
}
[Fact]
public async Task Input_ReachesTheRemote()
{
@@ -344,6 +344,84 @@ public sealed class TerminalWorkspaceTests
liveAtAnnouncement.ShouldBe(0, "announced after the drain, so the count already excludes it");
}
/// <remarks>
/// ◆ The announcement carries how it ended, not just that it did. A subscriber cannot work that out for
/// itself afterwards — by the time it hears, there is no transport left to interrogate — and the
/// difference between a shell that exited and a connection that broke is the difference between a tab
/// that is finished and one worth offering to dial again.
/// </remarks>
[Fact]
public async Task ASessionWhoseConnectionBroke_IsAnnouncedAsALostConnection()
{
var connections = new FakeConnectionFactory(bytesPerShell: 0)
{
TransportFailure = "An established connection was aborted by the server.",
};
await using var workspace = CreateWorkspace(connections);
TerminalSessionOutcome? outcome = null;
workspace.SessionEnded += (_, e) => Volatile.Write(ref outcome, e.Outcome);
await workspace.OpenSessionAsync(
Request(), TerminalSize.Default, progress: null, TestContext.Current.CancellationToken);
await WaitUntilAsync(() => Volatile.Read(ref outcome) is not null);
var announced = Volatile.Read(ref outcome)!;
announced.End.ShouldBe(SshSessionEnd.ConnectionLost);
announced.Description.ShouldBe("An established connection was aborted by the server.");
}
/// <remarks>
/// The same shell ending with nothing wrong underneath it, which is what <c>exit</c> looks like from
/// here. Its own test rather than an assertion tacked onto the one above, because the pair is the point:
/// one fake differs from the other by a latched failure and nothing else, and the two must not report
/// the same thing.
/// </remarks>
[Fact]
public async Task ASessionWhoseShellExited_IsAnnouncedAsTheRemoteClosingIt()
{
var connections = new FakeConnectionFactory(bytesPerShell: 0);
await using var workspace = CreateWorkspace(connections);
TerminalSessionOutcome? outcome = null;
workspace.SessionEnded += (_, e) => Volatile.Write(ref outcome, e.Outcome);
await workspace.OpenSessionAsync(
Request(), TerminalSize.Default, progress: null, TestContext.Current.CancellationToken);
await WaitUntilAsync(() => Volatile.Read(ref outcome) is not null);
Volatile.Read(ref outcome)!.End.ShouldBe(SshSessionEnd.RemoteClosed);
}
/// <remarks>
/// A tab being closed, announced as this client's own doing however the pump unwound. It matters for the
/// shell reusing a tab: a reconnect closes the session it is replacing, and an announcement blaming the
/// remote for that would draw a dropped connection over an attempt that is already under way.
/// </remarks>
[Fact]
public async Task ClosingASession_IsAnnouncedAsThisClientsDoing()
{
var connections = new FakeConnectionFactory();
await using var workspace = CreateWorkspace(connections);
TerminalSessionOutcome? outcome = null;
workspace.SessionEnded += (_, e) => outcome = e.Outcome;
var sessionId = await workspace.OpenSessionAsync(
Request(), TerminalSize.Default, progress: null, TestContext.Current.CancellationToken);
await workspace.CloseSessionAsync(sessionId);
outcome.ShouldNotBeNull();
outcome.End.ShouldBe(SshSessionEnd.ClosedLocally);
}
/// <remarks>
/// Disposal is the other path that closes sessions, because it is process shutdown. Asserted so
/// that the SSH connections are known to be released rather than assumed to be — and that these closes,