Public Access
Keep the doorbell listening between callers, and stop a hung test taking CI with it
The test step ran for twenty minutes and had to be stopped by hand. Eighteen suites
had passed; DodoSSH.Client.App.Tests never reported.
◆ ON UNIX, A FRESH SERVER PER CONNECTION LOSES THE NEXT ONE.
The doorbell's accept loop built a NamedPipeServerStream inside the loop, served one
caller and disposed it. That is fine on Windows and quietly wrong on Linux, where a
named pipe is a Unix domain socket: disposing the server closes the *listening*
socket and unlinks its path, so a second launch that connected while the first was
being served has its connection dropped. Its Connect returned, its write succeeded
into a buffer, and nobody ever read it — the launch was told it had rung the
doorbell and the window never came forward.
Reproduced in a container rather than reasoned about, because the two platforms
disagree and this machine is only one of them. Three rings back to back:
one instance, Disconnect between callers -> 3 of 3 answered
a fresh instance per connection -> 1 of 3 answered
So there is one server for the life of the doorbell now, Disconnect()ed between
conversations, and the socket is never not listening. Which makes leaveOpen on the
reader load-bearing too: a StreamReader disposes the stream it was given, and that
stream now has to survive into the next iteration.
Two things I had wrong about this, both worth recording. Cancelling
WaitForConnectionAsync *does* unblock a pending accept on Linux — that was my first
theory and the container disproved it, so DisposeAsync awaiting the loop was never
the problem. And the app itself never hung: the lost ring is a launch that is
ignored, not a process that stops.
◆ AND THE REASON IT COST TWENTY MINUTES RATHER THAN TWENTY SECONDS.
await twice.Task.WaitAsync(TestContext.Current.CancellationToken);
That token carries no timeout. A ring that never arrives is then not a failing test,
it is a job that sits there — which is exactly what happened, and it is the more
expensive of the two mistakes because it turns any future bug of this shape into a
runner somebody has to go and stop. Both awaits are bounded now.
TheDoorbellAnswersMoreThanOnce keeps its shape deliberately: two rings back to back
rather than one and a pause, because the gap between accepts is the whole defect. A
test that waited for the first to be answered would pass against the broken version.
ci.yml had no timeout-minutes anywhere, on any job. Every job has a ceiling now, and
the test step has a tighter one of its own — ten minutes against a suite that takes
two — because a hang there is the likeliest of the lot and a step-level timeout names
the step instead of leaving a reader to work out which of fifteen was still running.
None of them is a performance budget; a build that legitimately grows past one should
have the number raised.
Verified both ways on Linux, in a container, against the committed tree and this one:
31c836f SingleInstanceTests -> hung in TheDoorbellAnswersMoreThanOnce,
still going when killed at 3 minutes
this SingleInstanceTests -> 7 of 7 in 1.3s
this the whole suite -> 570 tests, 0 failed, 88s
And on Windows, where none of this was ever broken: 570 tests, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
31c836f6ca
commit
dabc691c35
3 files changed
+109
-20
No files matched your search
@@ -29,6 +29,18 @@ jobs:
|
|||||||
build:
|
build:
|
||||||
name: build and test
|
name: build and test
|
||||||
runs-on: [linux]
|
runs-on: [linux]
|
||||||
|
# ◆ A CEILING, BECAUSE THIS FILE HAD NONE AND A HUNG TEST RAN UNTIL SOMEBODY NOTICED.
|
||||||
|
#
|
||||||
|
# One test in DodoSSH.Client.App.Tests waited on a named pipe that, on Linux, nothing was ever going
|
||||||
|
# to answer — and it waited on a token with no timeout, so the job did not fail, it sat there. It was
|
||||||
|
# killed by hand after twenty minutes, having reported nothing but the eighteen suites that had
|
||||||
|
# already passed.
|
||||||
|
#
|
||||||
|
# Twenty-five minutes against a job that ordinarily takes three. It is not a performance budget and
|
||||||
|
# must not be read as one: a build that legitimately grows past this should have the number raised.
|
||||||
|
# What it buys is that the failure mode is a red job with a named step rather than a runner somebody
|
||||||
|
# has to go and stop.
|
||||||
|
timeout-minutes: 25
|
||||||
steps:
|
steps:
|
||||||
# act_runner runs every `uses:` action with node inside the job container, and this
|
# act_runner runs every `uses:` action with node inside the job container, and this
|
||||||
# runner's image has none — the run died on the first line of actions/checkout with
|
# runner's image has none — the run died on the first line of actions/checkout with
|
||||||
@@ -220,7 +232,12 @@ jobs:
|
|||||||
- name: build
|
- name: build
|
||||||
run: dotnet build DodoSSH.slnx --no-restore --configuration Release
|
run: dotnet build DodoSSH.slnx --no-restore --configuration Release
|
||||||
|
|
||||||
|
# Tighter than the job's own ceiling, and deliberately so: a hang here is the likeliest of the
|
||||||
|
# lot — it is the only step running code written for this repository rather than a tool — and a
|
||||||
|
# step-level timeout names the step in the failure instead of leaving a reader to work out which
|
||||||
|
# of fifteen was still running. Ten minutes against a suite that takes two.
|
||||||
- name: test
|
- name: test
|
||||||
|
timeout-minutes: 10
|
||||||
run: dotnet test DodoSSH.slnx --no-build --configuration Release
|
run: dotnet test DodoSSH.slnx --no-build --configuration Release
|
||||||
|
|
||||||
# The one build shape nothing else here exercises: a self-contained RID-specific publish. Its
|
# The one build shape nothing else here exercises: a self-contained RID-specific publish. Its
|
||||||
@@ -556,6 +573,8 @@ jobs:
|
|||||||
exit 0
|
exit 0
|
||||||
android:
|
android:
|
||||||
name: android head
|
name: android head
|
||||||
|
# See the build job. The docker pull and the toolchain image make this one legitimately slower.
|
||||||
|
timeout-minutes: 40
|
||||||
runs-on: [linux]
|
runs-on: [linux]
|
||||||
# Writes the nightly release at the end of the job; see that step for what the capability is and why
|
# Writes the nightly release at the end of the job; see that step for what the capability is and why
|
||||||
# it is acceptable here and nowhere else. Job-scoped, so nothing else in this file gains it.
|
# it is acceptable here and nowhere else. Job-scoped, so nothing else in this file gains it.
|
||||||
@@ -929,6 +948,8 @@ jobs:
|
|||||||
|
|
||||||
desktop:
|
desktop:
|
||||||
name: desktop nightly
|
name: desktop nightly
|
||||||
|
# See the build job. Two publishes, a pack and an upload of about 120 MB.
|
||||||
|
timeout-minutes: 40
|
||||||
# Gated on the tests, like the image job and for a stronger version of its reason: this one is
|
# Gated on the tests, like the image job and for a stronger version of its reason: this one is
|
||||||
# installed by people and replaces itself afterwards. Nothing anybody runs should come out of a
|
# installed by people and replaces itself afterwards. Nothing anybody runs should come out of a
|
||||||
# commit whose suite was red.
|
# commit whose suite was red.
|
||||||
@@ -1297,6 +1318,8 @@ jobs:
|
|||||||
|
|
||||||
image:
|
image:
|
||||||
name: api image
|
name: api image
|
||||||
|
# See the build job. A cold layer cache is the slow case here.
|
||||||
|
timeout-minutes: 40
|
||||||
# Gated on the tests rather than parallel with them, which costs a few minutes of wall
|
# Gated on the tests rather than parallel with them, which costs a few minutes of wall
|
||||||
# clock on every main commit and buys the thing worth having: no image reaches the
|
# clock on every main commit and buys the thing worth having: no image reaches the
|
||||||
# registry from a commit whose tests were red. An image is not a build artefact anyone
|
# registry from a commit whose tests were red. An image is not a build artefact anyone
|
||||||
|
|||||||
@@ -249,9 +249,30 @@ internal static class SingleInstance
|
|||||||
|
|
||||||
/// <summary>The accept loop that answers refused launches.</summary>
|
/// <summary>The accept loop that answers refused launches.</summary>
|
||||||
/// <remarks>
|
/// <remarks>
|
||||||
/// One server instance at a time and a fresh stream per connection, which is the shape
|
/// <para>
|
||||||
/// <see cref="NamedPipeServerStream"/> wants: a stream that has served a client cannot serve another.
|
/// One server instance for the life of the doorbell, disconnected between callers rather than disposed
|
||||||
/// The loop is what makes a third and fourth launch work as well as the second.
|
/// and rebuilt. The loop is what makes a third and fourth launch work as well as the second.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// ◆ <b>THE ONE INSTANCE IS THE WHOLE CORRECTNESS OF THIS CLASS ON LINUX, AND A FRESH ONE PER CALLER
|
||||||
|
/// LOSES LAUNCHES.</b> This was written the obvious way — a <c>using</c> inside the loop, a new stream
|
||||||
|
/// per connection — which is fine on Windows and quietly wrong on Unix, where a named pipe is a Unix
|
||||||
|
/// domain socket. Disposing the server there closes the <em>listening</em> socket and unlinks its path,
|
||||||
|
/// so a second launch that connected while the first was being served has its connection dropped on the
|
||||||
|
/// floor: its <c>Connect</c> returned, its write succeeded into a buffer, and nobody ever read it. The
|
||||||
|
/// launch was told it had rung the doorbell and the window never came forward.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Measured rather than reasoned about, in a container, because the two platforms disagree and this
|
||||||
|
/// machine is only one of them: three rings back to back are answered three times with one instance and
|
||||||
|
/// once with a fresh one per connection. <c>Disconnect</c> ends the conversation and keeps the socket, so
|
||||||
|
/// there is never a moment with nothing listening.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Which makes <c>leaveOpen</c> on the reader load-bearing as well. A <see cref="StreamReader"/> disposes
|
||||||
|
/// the stream it was given, and the stream is now the server that has to survive into the next
|
||||||
|
/// iteration — so the reader that was harmless per-connection would close the doorbell after one ring.
|
||||||
|
/// </para>
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
private sealed class Doorbell : IAsyncDisposable
|
private sealed class Doorbell : IAsyncDisposable
|
||||||
{
|
{
|
||||||
@@ -286,10 +307,6 @@ internal static class SingleInstance
|
|||||||
}
|
}
|
||||||
|
|
||||||
private async Task RunAsync(CancellationToken cancellationToken)
|
private async Task RunAsync(CancellationToken cancellationToken)
|
||||||
{
|
|
||||||
while (!cancellationToken.IsCancellationRequested)
|
|
||||||
{
|
|
||||||
try
|
|
||||||
{
|
{
|
||||||
using var server = new NamedPipeServerStream(
|
using var server = new NamedPipeServerStream(
|
||||||
name,
|
name,
|
||||||
@@ -298,10 +315,15 @@ internal static class SingleInstance
|
|||||||
PipeTransmissionMode.Byte,
|
PipeTransmissionMode.Byte,
|
||||||
PipeOptions.Asynchronous);
|
PipeOptions.Asynchronous);
|
||||||
|
|
||||||
|
while (!cancellationToken.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
await server.WaitForConnectionAsync(cancellationToken).ConfigureAwait(false);
|
await server.WaitForConnectionAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
using var reader = new StreamReader(server, Encoding.UTF8);
|
// leaveOpen, because this server outlives the conversation. See the remark above.
|
||||||
|
using (var reader = new StreamReader(server, Encoding.UTF8, false, 1024, leaveOpen: true))
|
||||||
|
{
|
||||||
var said = await reader.ReadLineAsync(cancellationToken).ConfigureAwait(false);
|
var said = await reader.ReadLineAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (string.Equals(said, ComeForward, StringComparison.Ordinal))
|
if (string.Equals(said, ComeForward, StringComparison.Ordinal))
|
||||||
@@ -309,17 +331,42 @@ internal static class SingleInstance
|
|||||||
comeForward();
|
comeForward();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
HangUp(server);
|
||||||
|
}
|
||||||
catch (OperationCanceledException)
|
catch (OperationCanceledException)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
catch (IOException)
|
catch (IOException)
|
||||||
{
|
{
|
||||||
// A client that connected and hung up, or a name the platform took away. Round again:
|
// A client that connected and hung up mid-sentence. Round again: giving up here would
|
||||||
// giving up here would leave every later launch with nothing to talk to, which is the
|
// leave every later launch with nothing to talk to, which is the silent-exit behaviour
|
||||||
// silent-exit behaviour this class exists to avoid.
|
// this class exists to avoid.
|
||||||
|
HangUp(server);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>Ends one conversation without ending the doorbell.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Guarded, because <c>Disconnect</c> throws on a server that is not connected — which is the state
|
||||||
|
/// an <see cref="IOException"/> above may already have left it in, and the state a second call would
|
||||||
|
/// find. Throwing out of the error path would take the loop with it.
|
||||||
|
/// </remarks>
|
||||||
|
private static void HangUp(NamedPipeServerStream server)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
if (server.IsConnected)
|
||||||
|
{
|
||||||
|
server.Disconnect();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
catch (InvalidOperationException)
|
||||||
|
{
|
||||||
|
// Already disconnected. Nothing to end.
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -42,6 +42,19 @@ public sealed class SingleInstanceTests : IDisposable
|
|||||||
/// </remarks>
|
/// </remarks>
|
||||||
private static readonly TimeSpan Impatient = TimeSpan.Zero;
|
private static readonly TimeSpan Impatient = TimeSpan.Zero;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// How long a doorbell is given to ring before the test says it never will.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// ◆ <b>A bound at all, which is the lesson rather than the number.</b> These two awaits were written
|
||||||
|
/// against <c>TestContext.Current.CancellationToken</c> alone, which carries no timeout — so the run
|
||||||
|
/// where a ring was genuinely lost did not fail, it <em>hung</em>, and took the whole CI job with it for
|
||||||
|
/// as long as anybody let it. Twenty seconds is a hundred times what a local pipe needs and a fifth of
|
||||||
|
/// what it costs to notice; a bound that is never reached costs nothing and a missing one costs the
|
||||||
|
/// build.
|
||||||
|
/// </remarks>
|
||||||
|
private static readonly TimeSpan LongEnoughForAPipe = TimeSpan.FromSeconds(20);
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public void Dispose()
|
public void Dispose()
|
||||||
{
|
{
|
||||||
@@ -106,16 +119,22 @@ public sealed class SingleInstanceTests : IDisposable
|
|||||||
|
|
||||||
SingleInstance.Claim(paths, Impatient).ShouldBeNull("the profile is taken");
|
SingleInstance.Claim(paths, Impatient).ShouldBeNull("the profile is taken");
|
||||||
|
|
||||||
// Awaited rather than polled: the callback comes off a thread pool thread, and the test's own
|
// Awaited rather than polled: the callback comes off a thread pool thread. Bounded explicitly,
|
||||||
// cancellation is what bounds a doorbell that never rings.
|
// because the test's own token is not — see LongEnoughForAPipe.
|
||||||
await asked.Task.WaitAsync(TestContext.Current.CancellationToken);
|
await asked.Task.WaitAsync(LongEnoughForAPipe, TestContext.Current.CancellationToken);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <remarks>
|
/// <remarks>
|
||||||
/// A third and a fourth launch work as well as the second, which is what the accept loop is for: a
|
/// <para>
|
||||||
/// <c>NamedPipeServerStream</c> that has served one client cannot serve another, so a doorbell that
|
/// A third and a fourth launch work as well as the second, and this is the test that caught the doorbell
|
||||||
/// forgot to go round again would answer exactly one refused launch and then leave every later one to
|
/// getting that wrong on Linux and only on Linux. A fresh <c>NamedPipeServerStream</c> per connection
|
||||||
/// exit in silence.
|
/// closes the listening socket between callers there, so the second ring was accepted by nobody while
|
||||||
|
/// the launch that made it was told it had succeeded. See <c>SingleInstance.Doorbell</c>.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Two rings back to back rather than one and a pause, because the gap is the whole defect: a test that
|
||||||
|
/// waited for the first to be answered before making the second would pass against the broken version.
|
||||||
|
/// </para>
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task TheDoorbellAnswersMoreThanOnce()
|
public async Task TheDoorbellAnswersMoreThanOnce()
|
||||||
@@ -139,7 +158,7 @@ public sealed class SingleInstanceTests : IDisposable
|
|||||||
SingleInstance.AskTheRunningOneToComeForward(paths).ShouldBeTrue();
|
SingleInstance.AskTheRunningOneToComeForward(paths).ShouldBeTrue();
|
||||||
SingleInstance.AskTheRunningOneToComeForward(paths).ShouldBeTrue();
|
SingleInstance.AskTheRunningOneToComeForward(paths).ShouldBeTrue();
|
||||||
|
|
||||||
await twice.Task.WaitAsync(TestContext.Current.CancellationToken);
|
await twice.Task.WaitAsync(LongEnoughForAPipe, TestContext.Current.CancellationToken);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <remarks>
|
/// <remarks>
|
||||||
|
|||||||
Reference in new issue
Block a user