Merge pull request 'Reconnect the terminal view when somebody comes back to it' (#14) from claude/terminal-reconnection-stuck-4e57ff into main
ci / build and test (push) Successful in 2m47s
ci / desktop nightly (push) Successful in 46s
ci / android head (push) Successful in 3m47s
ci / api image (push) Successful in 33s

Reviewed-on: #14
This commit was merged in pull request #14.
This commit is contained in:
2026-08-14 14:13:28 +00:00
10 changed files with 837 additions and 7 deletions
+15
View File
@@ -221,5 +221,20 @@
mistakes that cause real authorization holes. mistakes that cause real authorization holes.
--> -->
<PackageVersion Include="WireMock.Net" Version="2.13.0" /> <PackageVersion Include="WireMock.Net" Version="2.13.0" />
<!--
A JavaScript interpreter, in managed code, so that terminal.js can be tested as the file that
ships rather than as a transcription of it into C#. Test-only and referenced by exactly one
project; nothing in src depends on it.
Chosen over shelling out to node, which is the obvious alternative and needs node present
wherever the suite runs — CI installs one already, but a test that only runs when a separate
command is remembered is a test that stops being run. Chosen over a headless browser for the
same reason several times over.
What it does not buy: Jint is not Chromium, so this proves the page's own logic and nothing
about how WebView2 or Android's WebView behave. That line is drawn in RendererPage's remark and
picked up by docs/manual-checks.md 1.10 and 11.12a.
-->
<PackageVersion Include="Jint" Version="4.16.0" />
</ItemGroup> </ItemGroup>
</Project> </Project>
+49
View File
@@ -171,6 +171,32 @@ bound to the same side of `IsImportOpen`. SettingsNav lighting a different row w
over the importer would be a guard added to `OnKeyDown` for `IsSettingsMode` that the design never asked for over the importer would be a guard added to `OnKeyDown` for `IsSettingsMode` that the design never asked for
and this application's own quick-connect card was built to reach past. and this application's own quick-connect card was built to reach past.
### 1.10 A terminal left alone for a long time is still a terminal · **needs an hour, or a debugger**
Open a shell, leave the terminal for another screen — HOSTS, FILES, anything — and leave the application
alone for long enough that the window has been in the background for the better part of an hour. Locking the
machine or letting it sleep counts and is the easier way to get there. Come back and click the session's
tab.
**Pass:** the pane is exactly where it was and takes input straight away. If anything is shown at all it is
`Reconnecting the terminal view…` for a moment, in the second or so before the socket is back — never a
status that is still there after that.
**Both of the page's own rules here are covered by `RendererReconnectionTests`**, which runs `terminal.js`
itself in a fake browser — so a failure of this check is more likely to be the WebView behaving unlike that
fake than the page's logic being wrong. That is exactly the division: the test owns the logic, this owns the
platform.
**Failure means:** a banner that stays up is the page's retry not running. It is a `setTimeout` chain, and a
chain is what a WebView is entitled to throttle or freeze while nobody is looking at the page; `terminal.js`
answers that with wake-ups on `visibilitychange`, `focus` and `online`, none of which can be throttled,
plus a watchdog for a handshake that never finishes. A banner that flickers on and off every second or two
instead is the opposite fault — a reconnect loop, in which each attempt displaces the socket before it
through `TerminalDataPlane.UpgradeAsync`'s takeover and the displaced socket's close schedules the next.
That is what the "is this still the page's socket" guard in `connect()` exists to stop. A pane that takes no
input while the banner is *clear* is neither: the socket is open and dead, which nothing on this page can
currently see — see the note at the end of 11.12a.
--- ---
## Phase 2 — Known Hosts as its own page ## Phase 2 — Known Hosts as its own page
@@ -1797,6 +1823,29 @@ worse than saying nothing: the banner exists so this is never silently wrong. If
banner is there, the session's credit window was not reset on reattach and the shell is frozen behind it — banner is there, the session's credit window was not reset on reattach and the shell is frozen behind it —
see `TerminalWorkspace.ReplayAfterAttachAsync`. see `TerminalWorkspace.ReplayAfterAttachAsync`.
### 11.12a Coming back to a terminal screen left alone for a long while
The same shape as 11.12 and a different trigger: rather than backgrounding the app, stay in it. With a shell
open, leave the terminal for HOSTS, FILES or MORE — which collapses the renderer to GONE, so the page is
hidden by Chromium's reckoning — and leave the phone alone for at least ten minutes with the screen off.
Then come back to the app and to the terminal.
**Pass:** the pane is there and takes input at once, or reconnects visibly within about a second of the
screen appearing. `Reconnecting the terminal view…` on the way in is fine; still being there once the
terminal has been on screen for a couple of seconds is not.
**Failure means:** the page's retry did not survive being hidden. A hidden WebView has its timers throttled
— once a minute after five minutes hidden — and a renderer that was frozen or reclaimed runs none of them,
which is why `terminal.js` does not rely on the timer alone: `visibilitychange` is the event that says the
screen is back, and it reconnects immediately rather than waiting to be asked twice. Ten minutes is chosen
to clear the five-minute threshold with room to spare.
**Not covered by either check, and worth knowing:** a socket that is *open and dead* — the connection gone
without either end noticing, which a suspended renderer can leave behind — shows no banner at all, because
`readyState` still reads OPEN and nothing on this page probes further. The symptom is a terminal that looks
connected and swallows what is typed. If that is ever seen, it is a different bug from this one and needs a
liveness probe rather than a faster retry.
--- ---
## Phase 12 — Shared vaults: the operations that span two accounts ## Phase 12 — Shared vaults: the operations that span two accounts
+170 -7
View File
@@ -16,7 +16,8 @@
happened to straddle a frame boundary, which shows up as occasional mojibake in exactly the happened to straddle a frame boundary, which shows up as occasional mojibake in exactly the
conditions that are hardest to reproduce. conditions that are hardest to reproduce.
3. The socket reconnects itself, forever, with backoff. This page's WebView is routinely killed 3. The socket reconnects itself, forever, with backoff and on being looked at again, which is not
the same thing and is the half a timer cannot cover. This page's WebView is routinely killed
and reloaded by Android under memory pressure or simply for being backgrounded, so "the and reloaded by Android under memory pressure or simply for being backgrounded, so "the
socket closed" is an ordinary event here, not the end of the terminal's life see connect(). socket closed" is an ordinary event here, not the end of the terminal's life see connect().
A reloaded page starts with an empty session map, so createSession is idempotent (a session A reloaded page starts with an empty session map, so createSession is idempotent (a session
@@ -52,6 +53,22 @@ const SCROLLBACK_LINES = 5000;
const RECONNECT_INITIAL_DELAY_MS = 1000; const RECONNECT_INITIAL_DELAY_MS = 1000;
const RECONNECT_MAX_DELAY_MS = 5000; const RECONNECT_MAX_DELAY_MS = 5000;
/*
How long a socket is given to finish its handshake before it is treated as a failure.
A socket that cannot connect normally says so and says it quickly 'error' then 'close', within a
millisecond or two of a loopback refusal. The case this covers is the one that says nothing: an attempt
parked in CONNECTING with no event ever coming, which is what a WebSocket opened by a renderer that is
then suspended, or one whose handshake the host never answers, leaves this page holding.
Without the watchdog that state is terminal, and quietly so. Every retry in this file is scheduled by a
close or an error, so an attempt that produces neither schedules nothing: the banner says the view is
reconnecting for the rest of the page's life while nothing whatever is reconnecting. Five seconds is
generous against a handshake that ordinarily takes about a millisecond, and short against somebody
waiting on their terminal to come back.
*/
const HANDSHAKE_TIMEOUT_MS = 5000;
/* /*
Styled like the SESSION_CLOSED banner (matching \x1b[38;5;244, the same dim grey), but written by Styled like the SESSION_CLOSED banner (matching \x1b[38;5;244, the same dim grey), but written by
createSession's caller rather than by createSession itself: only a *replay* landing on a pane that createSession's caller rather than by createSession itself: only a *replay* landing on a pane that
@@ -583,8 +600,47 @@ function handleFrame(buffer) {
} }
} }
/*
THE SOCKET COMES BACK BY ITSELF, INCLUDING FOR A PAGE NOBODY HAS LOOKED AT IN AN HOUR
Retrying on a timer is the easy half and was the whole of this. The three rules below are what make the
retry actually reach a page that has been sitting collapsed each of them a way this page was found
showing "Reconnecting the terminal view…" over a socket that nothing was reconnecting.
1. ONLY THE CURRENT ATTEMPT'S EVENTS COUNT. connect() abandons whatever socket was attached before it,
and an abandoned socket still reports its end: the host aborts it the moment the newcomer upgrades,
which is TerminalDataPlane.UpgradeAsync's takeover doing exactly what it is meant to. Counting that
close as a fresh failure schedules a retry against a socket that has just succeeded, and the
takeover then aborts *that* one, whose close schedules the next a loop with no fixed point, in
which the terminal reconnects every second or so forever and the banner is up for most of it. Every
handler below asks whether it is still the page's socket before it does anything.
2. AN ATTEMPT THAT NEVER FINISHES IS A FAILURE TOO. See HANDSHAKE_TIMEOUT_MS.
3. COMING BACK IS A REASON TO TRY, NOT ONLY THE CLOCK. The retry is a setTimeout chain, and a chain is
precisely what a WebView is entitled to stop running. Chromium throttles timers in a page nobody is
looking at down to once a minute once it has been hidden five minutes and a renderer that is
frozen, or reclaimed and not yet reloaded, runs none of them at all. So the socket drops while nobody
is watching, the banner goes up, the retry is scheduled, and the retry is then the one thing not
running: coming back shows a terminal that says it is reconnecting and, for as long as that lasts,
is not.
The phone is where this is easiest to reach, because leaving the terminal for another screen
collapses its WebView to GONE see createSession's note on the GPU context and a WebView with no
surface is a page the platform may treat as hidden. Which of the two mechanisms actually bit is not
established here, and does not need to be: both end with a timer that will not run, and the fix is
not to guess at either but to stop depending on the timer alone.
Hence the wake-ups at the bottom of this file. What they add is not a faster retry; it is a retry
driven by the one thing that cannot be throttled, which is the user arriving.
*/
/** @type {number | null} */ /** @type {number | null} */
let reconnectTimer = null; let reconnectTimer = null;
/** @type {number | null} */
let handshakeTimer = null;
let reconnectDelay = RECONNECT_INITIAL_DELAY_MS; let reconnectDelay = RECONNECT_INITIAL_DELAY_MS;
/** /**
@@ -608,16 +664,86 @@ function scheduleReconnect() {
reconnectDelay = Math.min(reconnectDelay * 2, RECONNECT_MAX_DELAY_MS); reconnectDelay = Math.min(reconnectDelay * 2, RECONNECT_MAX_DELAY_MS);
} }
/**
* Tries the socket again now, if it is down.
*
* The entry point for "somebody is looking at this page again", and it has to be safe to call as often
* as that happens which on the desktop is every time the window is clicked. A socket that is up makes
* this nothing at all.
*
* An attempt still in CONNECTING is left alone rather than restarted: it may be about to succeed, and
* the one that is not is already the handshake watchdog's to give up on.
*/
function reconnectIfDown() {
if (socket !== null
&& (socket.readyState === WebSocket.OPEN || socket.readyState === WebSocket.CONNECTING)) {
return;
}
// Whatever the timer was going to do, this is doing now. Left pending it would land on top of the
// socket this call is about to open, and the takeover that followed is the loop rule 1 describes.
if (reconnectTimer !== null) {
clearTimeout(reconnectTimer);
reconnectTimer = null;
}
// Back to the quick attempt. The wait grew to space out retries during an outage nobody was watching,
// and being called at all means somebody is watching now.
reconnectDelay = RECONNECT_INITIAL_DELAY_MS;
connect();
}
function connect() { function connect() {
const token = root.dataset.token; const token = root.dataset.token;
const url = root.dataset.socket; const url = root.dataset.socket;
/*
Whatever was attached is abandoned here, and closed rather than dropped: the host takes the socket
over regardless, but one left open is a connection it has to abort and an event this page then has to
ignore. Cleared out of `socket` before the close, so that every handler including that close,
whenever it lands can already tell the attempt is no longer the page's.
*/
const abandoned = socket;
socket = null;
abandoned?.close();
// The token travels as a subprotocol rather than a query parameter, which keeps it out of // The token travels as a subprotocol rather than a query parameter, which keeps it out of
// anything that logs URLs. // anything that logs URLs.
socket = new WebSocket(url, ['dodossh.terminal.v1', `token.${token}`]); const pending = new WebSocket(url, ['dodossh.terminal.v1', `token.${token}`]);
socket.binaryType = 'arraybuffer'; pending.binaryType = 'arraybuffer';
socket = pending;
socket.addEventListener('open', () => { /** Whether this attempt is still the page's, rather than one a later connect() has replaced. */
const isCurrent = () => socket === pending;
const forgetHandshakeTimer = () => {
if (handshakeTimer !== null) {
clearTimeout(handshakeTimer);
handshakeTimer = null;
}
};
forgetHandshakeTimer();
handshakeTimer = setTimeout(() => {
handshakeTimer = null;
// Still CONNECTING with nothing on its way. Only the retry is arranged here; abandoning the attempt
// is left to the connect() that retry runs, which is the one place a socket is replaced.
if (isCurrent() && pending.readyState === WebSocket.CONNECTING) {
scheduleReconnect();
}
}, HANDSHAKE_TIMEOUT_MS);
pending.addEventListener('open', () => {
// A replaced attempt cannot reach here — connect() closes what it abandons, and a socket closed
// while connecting never opens — so this is a guard against the ordering rather than a live case.
if (!isCurrent()) {
return;
}
forgetHandshakeTimer();
setTransportStatus(''); setTransportStatus('');
// Back to the quick attempt for whatever the next failure turns out to be. Kept slow between // Back to the quick attempt for whatever the next failure turns out to be. Kept slow between
@@ -625,14 +751,25 @@ function connect() {
reconnectDelay = RECONNECT_INITIAL_DELAY_MS; reconnectDelay = RECONNECT_INITIAL_DELAY_MS;
}); });
socket.addEventListener('message', (event) => handleFrame(event.data)); pending.addEventListener('message', (event) => {
if (isCurrent()) {
handleFrame(event.data);
}
});
// Both close and error retry. They are not the same event on every failure — a socket that never // Both close and error retry. They are not the same event on every failure — a socket that never
// opens can fire only 'error', one that opens and later drops fires only 'close' — and the host // opens can fire only 'error', one that opens and later drops fires only 'close' — and the host
// side of this same problem (TerminalDataPlane.UpgradeAsync's takeover) is exactly why retrying is // side of this same problem (TerminalDataPlane.UpgradeAsync's takeover) is exactly why retrying is
// safe: whichever attempt eventually reaches the host, a fresh valid upgrade always wins the socket. // safe: whichever attempt eventually reaches the host, a fresh valid upgrade always wins the socket.
socket.addEventListener('close', scheduleReconnect); const retry = () => {
socket.addEventListener('error', scheduleReconnect); if (isCurrent()) {
forgetHandshakeTimer();
scheduleReconnect();
}
};
pending.addEventListener('close', retry);
pending.addEventListener('error', retry);
} }
// One observer for the whole root rather than one per pane: resizes arrive in bursts while a // One observer for the whole root rather than one per pane: resizes arrive in bursts while a
@@ -645,4 +782,30 @@ new ResizeObserver(() => {
window.addEventListener('beforeunload', () => socket?.close()); window.addEventListener('beforeunload', () => socket?.close());
/*
The ways this page finds out somebody is looking at it again rule 3 at the top of the transport
section. All three end in the same check, and that check is what makes them safe to be as noisy as they
are: with a healthy socket every one of them does nothing.
'visibilitychange' is the phone's. A WebView collapsed to GONE is a hidden page, so coming back to the
terminal screen is the event that says so, and it is the same event Chromium lifts its own throttling
on this page simply does not wait to be asked twice.
'focus' is the desktop's, where the page is *not* marked hidden while the WebView is collapsed
(measured; see docs/platform-flags.md) but a window left in the background for hours has had its timers
throttled all the same. It fires when the terminal takes the keyboard back, which is the same gesture.
'online' is neither head's ordinary case, because the socket is loopback and has nothing to do with the
network but the event does follow a machine coming back from sleep, and a page with a dead socket has
no reason to ignore any hint that the world has moved.
*/
document.addEventListener('visibilitychange', () => {
if (!document.hidden) {
reconnectIfDown();
}
});
window.addEventListener('focus', reconnectIfDown);
window.addEventListener('online', reconnectIfDown);
connect(); connect();
@@ -4,10 +4,44 @@
The throughput and backpressure harness the plan requires before any UI exists. Nothing The throughput and backpressure harness the plan requires before any UI exists. Nothing
here needs a WebView: the flow control is what is most likely to be wrong, and it is pure here needs a WebView: the flow control is what is most likely to be wrong, and it is pure
logic once ITerminalTransport is a seam. logic once ITerminalTransport is a seam.
And, since RendererReconnectionTests, the renderer's own half of the same protocol — the page
that reads these frames, run as a script rather than described in C#. It lives here rather than
in a test project of the shell's own because this is where the other end of the socket is
tested, and the two halves of "the renderer stays attached" are one mechanism: the host takes
a second upgrade over the first, and the page is what makes that second upgrade happen. A
project for the shell would still need a reference to this one's subject to say anything.
--> -->
<ItemGroup> <ItemGroup>
<ProjectReference Include="../../src/DodoSSH.Client.Terminal/DodoSSH.Client.Terminal.csproj" /> <ProjectReference Include="../../src/DodoSSH.Client.Terminal/DodoSSH.Client.Terminal.csproj" />
</ItemGroup> </ItemGroup>
<ItemGroup>
<!--
A JavaScript engine, so terminal.js can be run as it ships instead of transcribed into C#.
See RendererPage for the trade against a node script, and for what an engine that is not
Chromium can and cannot prove.
-->
<PackageReference Include="Jint" />
</ItemGroup>
<ItemGroup>
<!--
◆ THE PAGE IS COPIED OUT OF THE SHELL PROJECT, WHICH IS THE ONE ODD THING IN THIS FILE.
Reaching across for a source file is not something else here does, and the alternatives are
worse: referencing DodoSSH.Client.Shell would drag Avalonia into a suite that draws nothing,
and a copy of the page checked in beside the tests would be a copy — the thing that quietly
stops matching what ships, which is precisely the failure this test exists to catch.
PreserveNewest rather than Always so that editing the page is what rebuilds it.
-->
<None Include="../../src/DodoSSH.Client.Shell/WebAssets/terminal.js"
Link="Renderer/terminal.js"
CopyToOutputDirectory="PreserveNewest" />
<None Include="Renderer/renderer-harness.js" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>
</Project> </Project>
@@ -0,0 +1,222 @@
'use strict';
/*
The world terminal.js is loaded into by RendererReconnectionTests.
It fakes exactly what the page's transport touches and nothing else: a WebSocket whose every step is
driven from the test, a clock that only moves when a test moves it, and the two DOM objects the page
reads at load. There is no Terminal, no fit addon and no WebGL here, because nothing on the reconnection
path builds a pane a test that opens a session will have to add them, and should, rather than this
file guessing now at what such a test would want.
NOTHING HERE MAY DECLARE A NAME terminal.js ALSO DECLARES.
Both files are evaluated as scripts into the same global scope, and the page's own `const root` against
a `var root` here is a SyntaxError before a line of either runs. That is why the page's two elements are
`pageRoot` and `pageBanner` below and reached through getElementById, which is how the page reaches them
anyway.
AND THE FAKE SOCKET'S BEHAVIOUR IS THE PART TO GET RIGHT.
close() is the one method with a rule that is not obvious and that the tests lean on: a socket closed
while it is still CONNECTING never fires 'close' at all, per the WebSocket specification, while one
closed after it opened does. The page relies on exactly that connect() closes the attempt it abandons
and expects to hear nothing back from it so a fake that fired 'close' either way would report the loop
the page's guards exist to prevent, and one that fired it neither way would hide it.
*/
var clock = { now: 0, next: 1, timers: {} };
/** Every socket the page has opened, in order, live or dead. */
var sockets = [];
function listeners(target) {
target.handlers = {};
target.addEventListener = function (name, handler) {
if (!target.handlers[name]) {
target.handlers[name] = [];
}
target.handlers[name].push(handler);
};
target.fire = function (name) {
var handlers = target.handlers[name] || [];
for (var i = 0; i < handlers.length; i++) {
handlers[i]({});
}
};
return target;
}
function FakeSocket(url, protocols) {
this.url = url;
this.protocols = protocols;
this.readyState = FakeSocket.CONNECTING;
this.binaryType = '';
listeners(this);
sockets.push(this);
}
FakeSocket.CONNECTING = 0;
FakeSocket.OPEN = 1;
FakeSocket.CLOSING = 2;
FakeSocket.CLOSED = 3;
FakeSocket.prototype.send = function () {};
/** What the page calls. See the note above on why a connecting socket goes quietly. */
FakeSocket.prototype.close = function () {
if (this.readyState === FakeSocket.CLOSED) {
return;
}
var wasConnecting = this.readyState === FakeSocket.CONNECTING;
this.readyState = FakeSocket.CLOSED;
if (!wasConnecting) {
this.fire('close');
}
};
var pageBanner = { textContent: 'Connecting…' };
var pageRoot = listeners({
dataset: { token: 'test-token', socket: 'ws://127.0.0.1:1/socket' },
appendChild: function () {},
});
var document = listeners({
hidden: false,
getElementById: function (id) { return id === 'root' ? pageRoot : pageBanner; },
createElement: function () { return { dataset: {}, style: {}, remove: function () {} }; },
});
var window = listeners({});
// The page warns through this on paths no reconnection test reaches. Present so that a test which does
// reach one fails on its own assertion rather than on a missing global.
var console = { warn: function () {}, log: function () {} };
function setTimeout(callback, delay) {
var id = clock.next++;
clock.timers[id] = { at: clock.now + (delay || 0), callback: callback };
return id;
}
function clearTimeout(id) {
delete clock.timers[id];
}
function ResizeObserver() {
this.observe = function () {};
}
// ── What the tests drive the page with ──────────────────────────────────────────────────────────────
/**
* Runs every timer due within the next `ms`, in the order they fall due.
*
* One at a time and re-scanned each round rather than collected up front, because a timer's callback
* routinely schedules the next one which is the whole shape of the page's retry and a snapshot taken
* before the first callback ran would miss it.
*/
function advance(ms) {
var target = clock.now + ms;
for (;;) {
var dueId = null;
var due = null;
for (var id in clock.timers) {
var timer = clock.timers[id];
if (timer.at <= target && (due === null || timer.at < due.at)) {
due = timer;
dueId = id;
}
}
if (due === null) {
break;
}
delete clock.timers[dueId];
clock.now = due.at;
due.callback();
}
clock.now = target;
}
/** How many sockets the page has opened since it loaded. */
function attempts() {
return sockets.length;
}
/** What the one status element says — the banner the user sees. */
function banner() {
return pageBanner.textContent;
}
/** Whether that attempt has been closed, by the page or by the far end. */
function isClosed(index) {
return sockets[index].readyState === FakeSocket.CLOSED;
}
/** The host accepted the upgrade. */
function accept(index) {
sockets[index].readyState = FakeSocket.OPEN;
sockets[index].fire('open');
}
/** An established socket goes away and the page is told. */
function drop(index) {
sockets[index].readyState = FakeSocket.CLOSED;
sockets[index].fire('close');
}
/** An attempt that never connects: 'error' then 'close', as a refused connection reports itself. */
function fail(index) {
sockets[index].readyState = FakeSocket.CLOSED;
sockets[index].fire('error');
sockets[index].fire('close');
}
/**
* A close arriving for a socket that died earlier the host's takeover abort, landing late.
*
* Separate from drop() because the point of it is the delay: the socket is already dead by the time the
* event is delivered, which is what a suspended renderer's queued events look like on the way back.
*/
function deliverLateClose(index) {
sockets[index].readyState = FakeSocket.CLOSED;
sockets[index].fire('close');
}
/** The screen the page is on goes away, and comes back. */
function becomeHidden() {
document.hidden = true;
document.fire('visibilitychange');
}
function becomeVisible() {
document.hidden = false;
document.fire('visibilitychange');
}
function takeFocus() {
window.fire('focus');
}
function comeOnline() {
window.fire('online');
}
var WebSocket = FakeSocket;
@@ -0,0 +1,79 @@
using System.Globalization;
using Jint;
namespace DodoSSH.Client.Terminal.Tests;
/// <summary>
/// One load of <c>terminal.js</c>, in a fake browser, for one test.
/// </summary>
/// <remarks>
/// <para>
/// <b>The page's own source is what runs.</b> Not a transcription of its logic into C# — that would test a
/// copy, and the copy would be the thing that stayed correct. The file is read from the shell project and
/// evaluated as it ships, so a change to the page that breaks the reconnection rules fails here.
/// </para>
/// <para>
/// <b>Jint rather than node, and that is a deliberate trade.</b> A node script would be the obvious way to
/// run JavaScript and would need node on every machine and in every CI job that runs the suite — so it
/// would be a second test command, run separately, and the first thing to be forgotten. This runs inside
/// <c>dotnet test</c> with everything else. What it costs is that the engine is not the engine the page
/// actually runs in: Jint is not Chromium, so this can prove the page's own logic and can prove nothing
/// about how WebView2 or Android's WebView behave. That boundary is exactly where
/// <c>docs/manual-checks.md</c> picks up — see 1.10 and 11.12a.
/// </para>
/// <para>
/// A fresh engine per test, because the page is a script with module-level state and there is no unloading
/// it: two tests sharing one engine would share a socket list, a clock and a backoff.
/// </para>
/// </remarks>
internal sealed class RendererPage
{
private readonly Engine engine;
private RendererPage(Engine engine) => this.engine = engine;
/// <summary>How many sockets the page has opened since it loaded.</summary>
/// <remarks>
/// The measurement nearly every test here turns on, and it is deliberately a count of *attempts*
/// rather than of live sockets: the failures being guarded against are a page that stops trying and a
/// page that never stops, and both are counted rather than observed.
/// </remarks>
public int Attempts => (int)engine.Evaluate("attempts()").AsNumber();
/// <summary>What the status element says — the banner a user would be looking at.</summary>
public string Banner => engine.Evaluate("banner()").AsString();
/// <summary>
/// Loads the harness and then the page, leaving the page exactly as it is a moment after the WebView
/// navigated to it: one socket opened and still connecting.
/// </summary>
public static RendererPage Load()
{
var browser = new Engine();
// Order matters and is not incidental: the page connects on its last line, so every global it
// touches — the socket constructor above all — has to be in place before it is evaluated.
browser.Execute(Read("Renderer/renderer-harness.js"));
browser.Execute(Read("Renderer/terminal.js"));
return new RendererPage(browser);
}
/// <summary>Runs a line of the harness's own vocabulary — <c>accept(0)</c>, <c>advance(1000)</c>.</summary>
public void Do(string script) => engine.Execute(script);
/// <summary>Whether that attempt has been closed, by the page or by the far end.</summary>
public bool IsClosed(int attempt) =>
engine.Evaluate(
string.Create(CultureInfo.InvariantCulture, $"isClosed({attempt})"))
.AsBoolean();
/// <remarks>
/// Both files are copied beside the test assembly by the project file, the page out of the shell
/// project it belongs to. Read from disk rather than embedded so that the copy which runs here is the
/// same bytes the host serves, with nothing in between that could go stale.
/// </remarks>
private static string Read(string relativePath) =>
File.ReadAllText(Path.Combine(AppContext.BaseDirectory, relativePath));
}
@@ -0,0 +1,192 @@
namespace DodoSSH.Client.Terminal.Tests;
/// <summary>
/// The renderer page's half of staying attached — <c>terminal.js</c>'s <c>connect()</c> and what drives it.
/// </summary>
/// <remarks>
/// <para>
/// The host's half is <see cref="TerminalDataPlaneTests"/>, and the two are one mechanism: a socket that
/// drops is ordinary here, and the page coming back for another is what makes it ordinary. What these
/// tests protect is the property that failure of this mechanism has no other symptom — a terminal whose
/// page has given up looks exactly like a terminal whose remote has gone quiet, except for a line of text
/// nobody reads twice.
/// </para>
/// <para>
/// Four of these were written against a page that failed them — the stale close, the handshake that never
/// finishes, and the two wake-ups — and the rest describe behaviour that was already right and is easy to
/// break while fixing those. The two that assert a wake-up does *nothing* pass against either version,
/// which is the point of them: they are what stops the cure being worse, and they can only ever fail
/// against a future change. See <see cref="RendererPage"/> for how the real file is loaded and for what
/// this cannot reach.
/// </para>
/// </remarks>
public sealed class RendererReconnectionTests
{
[Fact]
public void ThePage_ConnectsWhenItLoads()
{
var page = RendererPage.Load();
page.Attempts.ShouldBe(1);
}
[Fact]
public void ADroppedSocket_IsRetriedAndTheBannerClears()
{
var page = RendererPage.Load();
page.Do("accept(0)");
page.Banner.ShouldBe("");
page.Do("drop(0)");
page.Banner.ShouldStartWith("Reconnecting");
page.Do("advance(1000)");
page.Attempts.ShouldBe(2);
page.Do("accept(1)");
page.Banner.ShouldBe("");
}
/// <remarks>
/// The wait grows within one outage and goes back to a second once a socket has actually opened, so
/// that the next outage is not paid for at the previous one's rate.
/// </remarks>
[Fact]
public void TheWait_GrowsWithinAnOutageAndResetsAfterIt()
{
var page = RendererPage.Load();
page.Do("fail(0); advance(1000)");
page.Attempts.ShouldBe(2);
page.Do("fail(1); advance(1999)");
page.Attempts.ShouldBe(2);
page.Do("advance(1)");
page.Attempts.ShouldBe(3);
page.Do("accept(2); drop(2); advance(1000)");
page.Attempts.ShouldBe(4);
}
/// <summary>
/// A close for a socket the page has already replaced must not start a reconnect.
/// </summary>
/// <remarks>
/// The loop this forbids costs nothing to enter and never leaves: the host aborts the displaced socket
/// on every takeover — see <c>TerminalDataPlane.UpgradeAsync</c> — so a stale close that schedules a
/// retry displaces the socket that has just succeeded, whose own close schedules the next. The visible
/// end of it is a terminal that reconnects every second forever with the banner up for most of it.
/// </remarks>
[Fact]
public void AStaleClose_DoesNotDisplaceTheSocketThatSucceeded()
{
var page = RendererPage.Load();
page.Do("accept(0); drop(0); advance(1000)");
page.Do("accept(1)");
page.Attempts.ShouldBe(2);
// The first socket's end, arriving after the page has moved on.
page.Do("deliverLateClose(0)");
page.Do("advance(60000)");
page.Attempts.ShouldBe(2);
page.Banner.ShouldBe("");
}
/// <summary>
/// An attempt that never finishes its handshake is given up on rather than waited on forever.
/// </summary>
/// <remarks>
/// Every other retry in the page is scheduled by a close or an error, so a socket that reports neither
/// — which is what a renderer suspended mid-handshake leaves behind — used to schedule nothing at all.
/// The page then held a banner saying it was reconnecting with no timer pending and no socket coming,
/// for the rest of its life.
/// </remarks>
[Fact]
public void AHandshakeThatNeverFinishes_IsAbandonedAndRetried()
{
var page = RendererPage.Load();
// Nothing whatever from the first attempt: no open, no error, no close.
page.Do("advance(5000)");
page.Attempts.ShouldBe(1);
page.Do("advance(1000)");
page.Attempts.ShouldBe(2);
page.IsClosed(0).ShouldBeTrue();
page.Do("accept(1)");
page.Banner.ShouldBe("");
}
/// <summary>
/// Coming back to the page reconnects it, without waiting for a timer that may not be running.
/// </summary>
/// <remarks>
/// The case the whole wake-up path exists for, and the one a test can only approximate: the harness's
/// clock stands still here because a real hidden page's clock is throttled rather than stopped, and
/// standing still is the honest worst case of that. What is being asserted is that the page does not
/// need the clock at all to notice it is back.
/// </remarks>
[Fact]
public void BecomingVisibleAgain_ReconnectsWithoutTheTimer()
{
var page = RendererPage.Load();
page.Do("accept(0); becomeHidden(); drop(0)");
page.Attempts.ShouldBe(1);
page.Do("becomeVisible()");
page.Attempts.ShouldBe(2);
// And the timer that was pending when the page woke must not open a third socket on top of the
// one that just succeeded — which would be the takeover loop, entered from the other side.
page.Do("accept(1); advance(60000)");
page.Attempts.ShouldBe(2);
page.Banner.ShouldBe("");
}
[Fact]
public void TakingTheKeyboardBack_AlsoReconnects()
{
var page = RendererPage.Load();
page.Do("accept(0); drop(0); takeFocus()");
page.Attempts.ShouldBe(2);
}
/// <remarks>
/// The wake-ups fire on gestures as ordinary as clicking the window, so the check they make has to be
/// the thing that keeps them cheap rather than the frequency. A page whose socket is up must treat all
/// of them as nothing at all — anything else would be the takeover loop with a person's mouse driving it.
/// </remarks>
[Fact]
public void WakingUpOverAHealthySocket_DoesNothing()
{
var page = RendererPage.Load();
page.Do("accept(0)");
page.Do("takeFocus(); becomeVisible(); comeOnline(); becomeHidden(); becomeVisible()");
page.Attempts.ShouldBe(1);
page.Banner.ShouldBe("");
}
/// <remarks>
/// An attempt already in flight is left to finish or to time out. Restarting it on every wake-up would
/// mean a page being clicked during a slow handshake never completing one.
/// </remarks>
[Fact]
public void WakingUpWhileConnecting_LeavesTheAttemptAlone()
{
var page = RendererPage.Load();
page.Do("takeFocus(); becomeVisible(); comeOnline()");
page.Attempts.ShouldBe(1);
}
}
@@ -416,6 +416,36 @@ public sealed class TerminalDataPlaneTests : IAsyncDisposable
TimeProvider.System, TimeProvider.System,
new TerminalPumpOptions { FlushInterval = TimeSpan.FromMilliseconds(10) }); new TerminalPumpOptions { FlushInterval = TimeSpan.FromMilliseconds(10) });
/// <summary>
/// Opens a renderer's socket and returns once the plane is actually holding it.
/// </summary>
/// <remarks>
/// <para>
/// <b>Connected and attached are two different moments, and the gap between them is where this used to
/// flake.</b> <c>ClientWebSocket.ConnectAsync</c> completes on the 101, which
/// <see cref="TerminalDataPlane.UpgradeAsync"/> writes before it has a <see cref="WebSocket"/> to
/// attach — it builds one from the stream and swaps it in a few instructions later, on the accept
/// thread. A frame sent in between is dropped, by design rather than by accident: the transport has
/// nowhere to put a frame for a renderer that is not there, and queueing it is the unbounded growth the
/// credit window exists to prevent.
/// </para>
/// <para>
/// So a test that connected and immediately expected a frame was racing that window on every run. It
/// lost one on CI — <c>Output_ReachesTheRenderer</c> read the output frame first and asked why it was
/// not the session's opening one, the opening one having been dropped a moment earlier — which is a
/// scheduling accident on a loaded machine and says nothing whatever about the transport.
/// </para>
/// <para>
/// Production does not race it and needs no change: everything that opens a session waits on
/// <c>TerminalWorkspace.WaitForRendererAsync</c> first, and that resolves from the same few lines this
/// event is raised from.
/// </para>
/// <para>
/// Subscribed before the connection rather than after it, because the event is raised on the accept
/// thread and can be over before <c>ConnectAsync</c> has returned here. Bounded, so that a socket that
/// never attaches fails this helper rather than hanging the suite in a later receive.
/// </para>
/// </remarks>
private async Task<ClientWebSocket> ConnectAsync( private async Task<ClientWebSocket> ConnectAsync(
string? token = "", string? token = "",
string? origin = null) string? origin = null)
@@ -433,17 +463,31 @@ public sealed class TerminalDataPlaneTests : IAsyncDisposable
"Origin", "Origin",
origin ?? string.Create(CultureInfo.InvariantCulture, $"http://127.0.0.1:{plane.Port}")); origin ?? string.Create(CultureInfo.InvariantCulture, $"http://127.0.0.1:{plane.Port}"));
var attached = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
void OnAttached(object? sender, EventArgs e) => attached.TrySetResult();
plane.SocketAttached += OnAttached;
try try
{ {
await socket.ConnectAsync( await socket.ConnectAsync(
new Uri($"ws://127.0.0.1:{plane.Port}{TerminalDataPlane.SocketPath}"), new Uri($"ws://127.0.0.1:{plane.Port}{TerminalDataPlane.SocketPath}"),
TestContext.Current.CancellationToken); TestContext.Current.CancellationToken);
await attached.Task.WaitAsync(
TimeSpan.FromSeconds(5),
TestContext.Current.CancellationToken);
} }
catch catch
{ {
socket.Dispose(); socket.Dispose();
throw; throw;
} }
finally
{
plane.SocketAttached -= OnAttached;
}
return socket; return socket;
} }
@@ -538,10 +538,26 @@ public sealed class TerminalWorkspaceTests
new("host.invalid", 22, "dodo", new SshPasswordCredential("irrelevant")); new("host.invalid", 22, "dodo", new SshPasswordCredential("irrelevant"));
/// <remarks> /// <remarks>
/// <para>
/// Attaches the way the real page does: by fetching the served page, reading the token and socket URL /// Attaches the way the real page does: by fetching the served page, reading the token and socket URL
/// back out of it, and presenting them on the upgrade — rather than reaching into the workspace for a /// back out of it, and presenting them on the upgrade — rather than reaching into the workspace for a
/// token it does not expose. A shortcut here would prove only that a socket can be opened, not that the /// token it does not expose. A shortcut here would prove only that a socket can be opened, not that the
/// workspace serves a page a renderer could actually attach with. /// workspace serves a page a renderer could actually attach with.
/// </para>
/// <para>
/// And it waits for the attach rather than only for the handshake, for the reason
/// <c>TerminalDataPlaneTests.ConnectAsync</c> sets out at length: the 101 is written before the socket
/// is attachable, and a frame sent in between is dropped. A caller that opens a session on the socket
/// this returns and then reads its opening frame is exactly the shape that loses that race.
/// </para>
/// <para>
/// <c>WaitForRendererAsync</c> answers only for the first renderer ever to attach, so a second call
/// returns immediately without proving anything about the second socket. That is enough here and is
/// not luck: the only frames a second socket is given are the replay, and the replay is *caused* by the
/// attach — <see cref="TerminalWorkspace.RendererReattached"/> and the frames before it cannot be sent
/// early. The day a test sends something else down a reattached socket, this needs the data plane's own
/// <c>SocketAttached</c>, which the workspace does not forward today.
/// </para>
/// </remarks> /// </remarks>
private static async Task<ClientWebSocket> ConnectRendererAsync(TerminalWorkspace workspace) private static async Task<ClientWebSocket> ConnectRendererAsync(TerminalWorkspace workspace)
{ {
@@ -561,6 +577,8 @@ public sealed class TerminalWorkspaceTests
try try
{ {
await client.ConnectAsync(new Uri(socketUrl), TestContext.Current.CancellationToken); await client.ConnectAsync(new Uri(socketUrl), TestContext.Current.CancellationToken);
await workspace.WaitForRendererAsync(TestContext.Current.CancellationToken);
} }
catch catch
{ {
@@ -2,6 +2,15 @@
"version": 2, "version": 2,
"dependencies": { "dependencies": {
"net10.0": { "net10.0": {
"Jint": {
"type": "Direct",
"requested": "[4.16.0, )",
"resolved": "4.16.0",
"contentHash": "YHofgoVtjWzqmG2GsGsp6eYMmcBfGgJOcH+Ki2UdZXNsYgKGnWrK2hSqRSPG/6HqeiipT5YPhzEH+bwxFO+YAQ==",
"dependencies": {
"Acornima": "1.7.0"
}
},
"Meziantou.Analyzer": { "Meziantou.Analyzer": {
"type": "Direct", "type": "Direct",
"requested": "[3.0.137, )", "requested": "[3.0.137, )",
@@ -48,6 +57,11 @@
"xunit.v3.mtp-v1": "[3.2.2]" "xunit.v3.mtp-v1": "[3.2.2]"
} }
}, },
"Acornima": {
"type": "Transitive",
"resolved": "1.7.0",
"contentHash": "a2I4O4qkuAdB0oSaGz6/k0n/bXxMbGcLBra5dNTTVSLmTwM7uORA8ebpOrNMmDfFWOMwvuaRL6gPprRV4HTV0w=="
},
"Castle.Core": { "Castle.Core": {
"type": "Transitive", "type": "Transitive",
"resolved": "5.1.1", "resolved": "5.1.1",