Public Access
Merge branch 'claude/vault-realtime-push-d64c61'
This commit is contained in:
@@ -356,6 +356,42 @@ Both default to your personal vault and neither moves on its own, because an ite
|
||||
visible to everybody holding that vault's key. Choosing a vault in the host editor also decides which groups
|
||||
it can be filed under: a group is an item like any other and lives in exactly one vault.
|
||||
|
||||
### Changes that do not wait
|
||||
|
||||
A client holds a WebSocket open to the server — `GET /api/v1/events`, subprotocol
|
||||
`dodossh.events.v1` — and the server sends a line down it whenever something you can read has moved. The
|
||||
client's answer is the same delta pull it would have run on its timer, only now rather than in up to a
|
||||
minute. Two things you can see: an edit somebody else makes appears while you are looking at the list, and
|
||||
a vault shared with you turns up as soon as they share it.
|
||||
|
||||
**What is on that socket is a notice, not your data.** A frame says which vault changed and how far its
|
||||
change log has got, and nothing else: no item, no ciphertext, not even which item it was. That is the
|
||||
decision the rest of this rests on, and it is deliberate twice over — the server has nothing else it
|
||||
*could* send, and keeping it that way means there is still exactly one path that applies a change to your
|
||||
keychain, so the socket can be wrong or absent without anything being applied incorrectly.
|
||||
|
||||
**Polling is still there and is still what guarantees a pass.** The minute timer is unchanged. A network
|
||||
that eats WebSockets, a server with `Events:Enabled` off, an older server, a proxy that will not upgrade, a
|
||||
notice dropped because your machine was too slow to read it — every one of those leaves you with exactly
|
||||
what this product did before the socket existed. Nothing is reachable only this way, and nothing is
|
||||
supposed to become so.
|
||||
|
||||
Three limits are worth knowing rather than discovering:
|
||||
|
||||
- **One node.** Fan-out is in-process, so a deployment running more than one API replica only pushes for
|
||||
writes that its own replica handled. The rest arrive on the timer. The seam for a PostgreSQL
|
||||
`LISTEN`/`NOTIFY` backplane is in place and is not implemented, because an untested backplane would be
|
||||
worse than a documented gap.
|
||||
- **The socket does not outlive your access token.** It is closed at the token's expiry and the client
|
||||
reconnects with a fresh one, which is a gap you will not see. That, plus re-reading your vault list every
|
||||
few minutes, is what bounds how long a withdrawn grant can keep producing notices — and what it bounds is
|
||||
*metadata*, because reading a vault needs a key the server has never held.
|
||||
- **You are told about your own writes.** Your client pushed, so it has already pulled; the extra pass finds
|
||||
nothing. Notices are coalesced over a quarter of a second so that a burst is one pass rather than a dozen.
|
||||
|
||||
The reasoning, including why this is a WebSocket rather than server-sent events and where a shared terminal
|
||||
session will attach to it, is in [ADR 0012](docs/adr/0012-realtime-push.md).
|
||||
|
||||
### The Android head
|
||||
|
||||
`src/DodoSSH.Client.Android` is a phone-first head that shares every view model with the desktop one — the
|
||||
@@ -624,6 +660,20 @@ keychain plus a terminal — and the spike that gates all of it.
|
||||
directories, an interrupted **upload** starts again rather than resuming (an object cannot be written from
|
||||
the middle), and a rename is a copy and a delete rather than one atomic operation. Downloads do resume — a
|
||||
ranged GET is part of the protocol, which is the one place a bucket beats SFTP.
|
||||
|
||||
*Realtime done:* a WebSocket the client holds open, over which the server says which vault has moved so a
|
||||
pull happens now rather than within the minute. What crosses it is a notice and never an item, which is
|
||||
what keeps one code path applying changes and makes a dropped socket cost latency and nothing else — the
|
||||
timer is unchanged and is still the guarantee. Two limits are stated rather than implied: fan-out is
|
||||
in-process, so a multi-replica deployment falls back to the timer for writes another replica handled, and
|
||||
a socket is closed at its access token's expiry rather than outliving the credential that authorised it.
|
||||
See [Changes that do not wait](#changes-that-do-not-wait) and
|
||||
[ADR 0012](docs/adr/0012-realtime-push.md).
|
||||
|
||||
It is also the transport a **shared terminal session** will use — one person's shell, watched or driven by
|
||||
somebody else. Nothing of that exists yet, and ADR 0012 records the one decision made early so it need not
|
||||
be renegotiated: session data will be binary frames on this same socket, because base64 in a JSON envelope
|
||||
is the wrong shape for the one payload here that is continuous rather than occasional.
|
||||
- **M3 — shared vaults**, sharing, ACLs. *Done.* Membership with roles, a public-key directory, the
|
||||
append-only key log served for clients to verify against, shared vaults, and vault key grants
|
||||
wrapped by a client and stored opaquely by the server. `VaultAccessService` now resolves team
|
||||
@@ -647,10 +697,12 @@ keychain plus a terminal — and the spike that gates all of it.
|
||||
the rotation is re-sealed as it is pushed, so nothing reaches the server under a superseded key at all.
|
||||
See [ADR 0010](docs/adr/0010-vault-key-rotation.md).
|
||||
|
||||
**A vault shared with you arrives on the next synchronisation pass**, within the minute, with no sign-in
|
||||
and nothing to press. There is no push channel, so each pass asks the server which vaults this account can
|
||||
reach before syncing the ones it already knows — which is also how a vault that has been deleted, or one
|
||||
whose grant was withdrawn, stops being listed.
|
||||
**A vault shared with you arrives at once**, with no sign-in and nothing to press. Each pass asks the
|
||||
server which vaults this account can reach before syncing the ones it already knows — which is also how a
|
||||
vault that has been deleted, or one whose grant was withdrawn, stops being listed — and the server now
|
||||
says so the moment somebody wraps a key to you rather than leaving it for the next pass. Without a
|
||||
reachable socket that becomes "within the minute", which is what it always was; see
|
||||
[Changes that do not wait](#changes-that-do-not-wait).
|
||||
|
||||
**Ownership transfer is here, and it is one write rather than two.** The member you name becomes owner
|
||||
and you become an admin, in a single transaction — because ownership is sole, so promoting first leaves
|
||||
|
||||
@@ -58,7 +58,11 @@ the Npgsql connection string** — the default; do not enable multiplexing.
|
||||
- One place enforces revision, change-log and ACL invariants. That halves both the endpoint
|
||||
count and the authorization surface, which is the main reason for the single write path.
|
||||
- Delta pull makes frequent polling cheap, so multi-device feels live; push notification over
|
||||
SSE or the existing WebSocket can layer on with polling as the fallback.
|
||||
SSE or the existing WebSocket can layer on with polling as the fallback. **That has since been
|
||||
built — see [ADR 0012](0012-realtime-push.md)** — and nothing in this ADR changed to accommodate
|
||||
it. The socket carries a notice naming a vault and a sequence, whose answer is the delta pull
|
||||
above, so there is still exactly one path that applies a change; and polling is still what
|
||||
guarantees a pass rather than a legacy route kept for old clients.
|
||||
- Conflict resolution is entirely client-side. The client retains a `BaseCiphertext` common
|
||||
ancestor and performs a field-level three-way merge for structured items, or creates a
|
||||
visible conflicted copy for opaque ones. **It must never silently drop a key or a host.**
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
# ADR 0012 — A WebSocket that carries notices, not data
|
||||
|
||||
- Status: accepted
|
||||
- Date: 2026-08-04
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0003](0003-sync-protocol.md) built a delta pull that is cheap enough to run on a timer, and
|
||||
the client does: one pass a minute. That is the difference between a colleague's change appearing
|
||||
"soon" and appearing *now*, and it shows up in three places that are not equally forgivable.
|
||||
|
||||
- **A vault shared with you** arrives on the next pass. `AdmitNewVaultsAsync` says so in its own
|
||||
remarks — "the recipient is handed nothing — there is no push channel" — and the README repeats
|
||||
it. Sharing works and looks broken.
|
||||
- **Two people editing one keychain** see each other up to a minute late, which is long enough to
|
||||
make the same edit twice and produce a conflict that nobody needed to have.
|
||||
- **A revoked grant** keeps serving a client that has not noticed yet, for up to a pass.
|
||||
|
||||
Shortening the interval is the obvious answer and the wrong one: it costs a request per client per
|
||||
interval whether or not anything happened, and it does not converge on *immediate* — it converges
|
||||
on a busier server that is still late.
|
||||
|
||||
There is also a second thing coming that this decision has to not preclude. The intended feature is
|
||||
a **shared terminal session** — one person's shell, watched or driven by another, TeamViewer-shaped.
|
||||
That is bidirectional, continuous, and latency-sensitive in a way a keychain notice is not.
|
||||
|
||||
## Decision
|
||||
|
||||
### One WebSocket per signed-in client, at `GET /api/v1/events`
|
||||
|
||||
Subprotocol `dodossh.events.v1`. The client opens it after unlock and keeps it open; the server
|
||||
sends a notice whenever something the client can read has changed.
|
||||
|
||||
**Not SSE.** Server-sent events would carry today's notices perfectly well and would be less code.
|
||||
It is one-directional, so the shared-session feature would need a second mechanism next to it, and
|
||||
then two transports would need reconnection, authorization and lifetime rules that agree. The cost
|
||||
of a WebSocket over SSE is small; the cost of two transports is not.
|
||||
|
||||
**Not SignalR.** It brings hub protocol negotiation, its own serialisation and transport fallbacks,
|
||||
none of which are wanted here: `DodoSSH.Contracts` and its source-generated serialiser are "the
|
||||
actual contract between the two sides", and a second wire format alongside it is exactly the silent
|
||||
drift `Setup/Json.cs` records having already cost this project once.
|
||||
|
||||
### The notice carries no ciphertext
|
||||
|
||||
A `vault.changed` frame is `{ kind, vaultId, sequence }` and nothing else. The client's answer to it
|
||||
is the pull it would have done on the timer anyway.
|
||||
|
||||
This is the load-bearing decision, and it is worth being explicit about why the tempting alternative
|
||||
is refused. Pushing the changed items themselves would save a round trip and would fork the code
|
||||
path that applies a change into two — one that arrives by pull and one that arrives by socket — with
|
||||
the cursor, the merge and the tombstone rules duplicated across both. ADR 0003 put every mutation
|
||||
through one write path for exactly that reason; this keeps every *read* on one path for the same
|
||||
one. The socket decides *when* to sync. It never decides *what* a vault contains.
|
||||
|
||||
It also means a dropped notice is harmless, which is what lets everything below be simple.
|
||||
|
||||
### Polling stays, and is the fallback rather than a legacy path
|
||||
|
||||
The one-minute pass is unchanged. The socket makes it *early*; it does not make it *necessary*. A
|
||||
client on a network that eats WebSockets, an older client, a server that has the feature off, a
|
||||
notice dropped under backpressure, a second API replica that did not see the write — every one of
|
||||
those degrades to what the product does today, which is correct and up to a minute late.
|
||||
|
||||
Nothing may be reachable only by socket. That is a rule about future features, not an observation
|
||||
about this one.
|
||||
|
||||
### Authorization: the bearer token on the upgrade, not a ticket
|
||||
|
||||
[ADR 0004](0004-relay-authorization.md) gives the relay a two-step ticket so its WebSocket carries
|
||||
no API authority. This one goes the other way and takes the ordinary bearer JWT on the upgrade
|
||||
request, which is an ordinary authenticated HTTP request. The difference is not inconsistency:
|
||||
|
||||
- The relay's socket is a **byte pipe to a third party**, and its whole authorization decision —
|
||||
which host, which IPs, which port — is made *before* the socket opens and never revisited. It is
|
||||
also the extraction seam for a standalone relay process that must not hold ACL code.
|
||||
- This socket is a **view of the caller's own vault list**, and it has to keep answering "what may
|
||||
this account read" for as long as it is open. It needs the full ACL context, in-process, for the
|
||||
life of the connection. A ticket would carry that context in a token instead, and it would be
|
||||
wrong the moment the account's access changed.
|
||||
|
||||
A long-lived connection authorised by a short-lived token is the problem this creates, and it is met
|
||||
head-on rather than ignored:
|
||||
|
||||
1. **The socket does not outlive the token.** The `exp` claim is read at accept, and the connection
|
||||
is closed with `4401` when it passes. The client reconnects with a fresh token; that is a
|
||||
sub-second gap in a channel whose failure mode is already "poll instead".
|
||||
2. **The vault set is re-resolved periodically** (`Events:AccessRefreshInterval`, default five
|
||||
minutes) as well as on the changes that are known to affect it. A withdrawn grant therefore stops
|
||||
producing notices within that window at the latest, and immediately in the ordinary case.
|
||||
|
||||
Both are bounds on **metadata** — the fact that a vault changed and roughly when — because that is
|
||||
all a notice contains. Nobody's ciphertext is behind this socket, and a client that stayed subscribed
|
||||
one interval too long could still not read a byte of it: reading requires a vault key grant, which
|
||||
this server has never held.
|
||||
|
||||
### The frames
|
||||
|
||||
Text frames, JSON, `DodoSshJsonContext`. Server to client:
|
||||
|
||||
| kind | meaning |
|
||||
| --- | --- |
|
||||
| `hello` | accepted; carries the heartbeat interval and the vault count subscribed |
|
||||
| `vault.changed` | `vaultId` moved to `sequence`; pull it |
|
||||
| `vaults.changed` | the set of vaults this account can reach is different; re-read it |
|
||||
| `ping` | heartbeat; the client answers `pong` |
|
||||
|
||||
Client to server: `ping`, answered with `pong`. Nothing else — subscription is decided by the server
|
||||
from the caller's access, not asked for by the client, because a client that could ask to subscribe
|
||||
to a vault id is a client that can probe for vault ids.
|
||||
|
||||
`kind` is a **string**, not an enum, and that is deliberate. `UseStringEnumConverter` throws on a
|
||||
value it does not know, so a newer server sending a kind an older client has never heard of would
|
||||
not add an unknown frame — it would break that client's socket entirely. A string is ignored
|
||||
instead, which is what makes the table above extensible. `ProblemCodes` is the same shape for the
|
||||
same reason.
|
||||
|
||||
### Where the shared session will attach
|
||||
|
||||
The socket is the seam, and one thing about it is chosen now so that it need not be renegotiated
|
||||
later: **session data will be binary frames on this same connection, not JSON on the table above.**
|
||||
Terminal output base64'd into a JSON envelope would cost a third of the bandwidth for nothing, on
|
||||
the one payload here that is continuous rather than occasional. Control — offer, accept, resize,
|
||||
end — is JSON like everything else.
|
||||
|
||||
That is as far as this ADR goes. Two questions are open and are not being answered by implication:
|
||||
whether a shared session's bytes go through the API at all or peer-to-peer past it, and what
|
||||
end-to-end encryption means when the second party is watching a stream rather than holding a key.
|
||||
Both are ADR 0001 questions and deserve their own decision. What this one buys is that they will not
|
||||
also be transport questions.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Fan-out is in-process, and the deployment is therefore single-node for this feature.** Every
|
||||
connection is held by the node that accepted it; a write handled by another node produces no
|
||||
notice on this one. `IVaultEventPublisher` is the seam a backplane implements — PostgreSQL
|
||||
`LISTEN`/`NOTIFY` needs no infrastructure this stack does not already run — and it is deliberately
|
||||
**not implemented**, because an untested backplane is worse than a documented gap. Multiple API
|
||||
replicas do not break: they degrade to polling, which is the state before this ADR. `/api/v1/meta`
|
||||
advertises `events` so a client knows which it is getting.
|
||||
- **Per-connection queues are bounded and drop the oldest.** A notice is "pull vault X, which is at
|
||||
least at sequence N", so the newest is strictly more useful than the one it displaces and the
|
||||
client's answer is identical either way. A slow reader costs itself latency, never the publisher's
|
||||
progress — the publish path never blocks and never awaits a socket.
|
||||
- **The publish happens after the transaction commits**, outside the advisory lock ADR 0003 takes.
|
||||
A notice sent from inside it would name a sequence a reader cannot yet see, and would hold the
|
||||
per-vault write lock across a socket write.
|
||||
- **A client is notified of its own writes.** It pushed, so it already pulled; the extra pass finds
|
||||
nothing. The client coalesces notices over a short window rather than the server suppressing an
|
||||
echo, because suppressing it correctly needs a per-*device* identity on the socket and the same
|
||||
user's other machines must still be told.
|
||||
- **Connections are capped** per user and per node (`Events:MaxConnectionsPerUser`,
|
||||
`Events:MaxConnectionsTotal`). A socket is cheap but not free, and an unbounded count of them is a
|
||||
denial of service that authenticates first.
|
||||
- The feature can be turned off entirely (`Events:Enabled`). A deployment behind a proxy that will
|
||||
not upgrade should say so rather than have every client discover it by failing.
|
||||
|
||||
### Rejected
|
||||
|
||||
- **Shorter polling.** Cheaper to build, converges on a busier server that is still late.
|
||||
- **Long polling.** No new transport and genuinely immediate, but it holds a request thread and a
|
||||
connection per client for the same money as a WebSocket while offering none of the bidirectionality
|
||||
the shared session needs.
|
||||
- **Pushing the changed items down the socket.** Saves a round trip; forks the apply path in two. See
|
||||
above.
|
||||
- **Client-chosen subscriptions.** A `subscribe(vaultId)` frame is an existence oracle for vault ids,
|
||||
which is the disclosure `SyncPullEndpoint` answers 404 rather than 403 to avoid.
|
||||
@@ -1500,3 +1500,83 @@ delivery survives the failure because it is held against the transfer rather tha
|
||||
**Failure means:** a retry that succeeds but leaves the destination empty is the delivery having been
|
||||
dropped on the failure. An error saying the staged file is missing is the copy having been deleted at the
|
||||
stop, which is what `QueueDeliveredDownload` documents it does not do.
|
||||
|
||||
## Phase 15 — Changes that arrive without a timer
|
||||
|
||||
The socket is covered by tests on both sides: the endpoint suite opens a real one against a real
|
||||
`TestServer` and proves a push produces a notice, that another account's push does not, and that a frame
|
||||
carries no ciphertext; the shell suite proves a notice wakes the synchronisation loop long before the
|
||||
minute. What none of that can reach is **the network in between**, and that is where this feature is most
|
||||
likely to fail: a reverse proxy that will not upgrade, one that drops an idle socket without telling either
|
||||
end, a corporate middlebox, a phone moving between Wi-Fi and mobile data. Every one of those looks the same
|
||||
from inside a test host, which has no proxy and no radio.
|
||||
|
||||
The pass condition throughout is *two* things, and the second matters as much as the first: it arrives
|
||||
quickly, **and** it still arrives when the socket is gone. A build where the timer had stopped working would
|
||||
pass every "it was fast" check here and fail nobody until somebody's proxy changed.
|
||||
|
||||
### 15.1 A colleague's edit appears while you are looking at it
|
||||
|
||||
Two accounts sharing a vault, both unlocked, both on the Hosts screen. On the first machine, rename a host
|
||||
in the shared vault and save.
|
||||
|
||||
**Pass:** the second machine's list shows the new name within a second or two, with nothing pressed and no
|
||||
screen flicker — the row updates, the selection does not move, and the status line is not repainted with a
|
||||
sync report.
|
||||
|
||||
**Failure means:** nothing within a minute, then the new name, is the socket not being established at all —
|
||||
that is the timer doing its job, which is the correct fallback and not the feature. Check `/api/v1/meta`
|
||||
lists `events`, then whether the proxy in front of the API forwards `Upgrade` and `Connection`. A list that
|
||||
never updates at all is a synchronisation failure and has nothing to do with this phase.
|
||||
|
||||
### 15.2 A vault shared with you turns up as it is shared
|
||||
|
||||
The second account signed in and unlocked, sitting on the VAULTS screen. From the first, add them to a team
|
||||
and press SHARE KEY.
|
||||
|
||||
**Pass:** the vault appears in their list within a second or two of the key being wrapped, and reads as
|
||||
waiting for a key until the share, then as readable.
|
||||
|
||||
**Failure means:** the vault appearing only on the minute is the `vaults.changed` notice not being published
|
||||
or not being followed. Both the membership add and the grant publish one; if the membership arrives promptly
|
||||
and the key does not, the grant path is the one to look at.
|
||||
|
||||
### 15.3 It still works with the socket taken away
|
||||
|
||||
On the second machine, block the WebSocket — the simplest way is a proxy rule rejecting the upgrade, or
|
||||
setting `Events:Enabled` to `false` on the server and restarting it.
|
||||
|
||||
**Pass:** everything above still happens, within the minute rather than within seconds. Nothing on the
|
||||
screen says anything is wrong, because nothing is: no error, no OFFLINE badge, no repeated status message.
|
||||
The Sync button still works and still reports.
|
||||
|
||||
**Failure means:** an error message, a titlebar claiming to be offline, or a status line that repaints with
|
||||
a socket failure is the client treating an absent push channel as a fault. It is not one — the timer is the
|
||||
guarantee and the socket is the optimisation, and a user with a strict proxy must never be told their
|
||||
keychain is broken.
|
||||
|
||||
### 15.4 A laptop that slept comes back on its own
|
||||
|
||||
With the second machine idle and connected, close the lid for a few minutes — or disable Wi-Fi for two
|
||||
minutes and re-enable it. Then make a change on the first machine.
|
||||
|
||||
**Pass:** the change arrives quickly again, without the vault having been locked or the application
|
||||
restarted. The reconnection is invisible.
|
||||
|
||||
**Failure means:** changes that arrive only on the timer from then on are the stream having given up after
|
||||
its socket died — the reconnection loop is what should make that impossible, and a client that reconnects
|
||||
once and not twice is the specific defect its tests exist to catch. Changes that never arrive again, timer
|
||||
included, are a different and worse bug in the synchronisation loop rather than in the socket.
|
||||
|
||||
### 15.5 An expiring token does not end the push
|
||||
|
||||
This one needs a short access-token lifetime in the identity provider — the dev realm's Keycloak client can
|
||||
be set to a couple of minutes. Leave a machine unlocked and idle for longer than that, then make a change
|
||||
elsewhere.
|
||||
|
||||
**Pass:** the change still arrives quickly. The socket is closed by the server at the token's expiry and the
|
||||
client reconnects with a fresh one, which should be invisible.
|
||||
|
||||
**Failure means:** notices stopping at roughly the token's lifetime is the reconnection not asking for a new
|
||||
token — it would be dialling with the spent one and being closed again immediately. A burst of reconnection
|
||||
attempts in the server log is the same defect seen from the other end.
|
||||
|
||||
@@ -0,0 +1,533 @@
|
||||
using System.Collections.Frozen;
|
||||
using System.Globalization;
|
||||
using System.Net.WebSockets;
|
||||
using System.Security.Claims;
|
||||
using System.Text.Json;
|
||||
using DodoSSH.Api.Authorization;
|
||||
using DodoSSH.Api.Setup;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain.Authorization;
|
||||
using FastEndpoints;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace DodoSSH.Api.Features.Events;
|
||||
|
||||
/// <summary>
|
||||
/// The socket that says "pull now" so a client does not have to wait for its timer.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Everything this endpoint sends is a <em>notice</em>. It never carries an item, a payload or a
|
||||
/// cursor: the client's answer to a notice is the delta pull it would have run on its own anyway, so
|
||||
/// there is exactly one code path that applies a change and this is not it. See ADR 0012 for why
|
||||
/// pushing the items themselves is refused.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The bearer token authorises the upgrade, unlike the relay's ticket in ADR 0004. The relay's socket
|
||||
/// is a byte pipe whose whole authorization decision is made before it opens; this one is a view of
|
||||
/// the caller's own vault list and has to keep answering "what may this account read" for as long as
|
||||
/// it is held. Its two bounds on that — the token's own expiry, and a periodic re-resolve — are in
|
||||
/// <see cref="MindAsync"/>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal sealed class VaultEventsEndpoint(
|
||||
IServiceScopeFactory scopes,
|
||||
VaultEventHub hub,
|
||||
IOptions<EventsOptions> options,
|
||||
TimeProvider clock,
|
||||
IHostApplicationLifetime lifetime,
|
||||
ILogger<VaultEventsEndpoint> logger)
|
||||
: EndpointWithoutRequest
|
||||
{
|
||||
/// <summary>
|
||||
/// The largest message this endpoint will read from a client.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A client sends nothing but <c>ping</c>, so the cap is three orders of magnitude of headroom and
|
||||
/// still small enough that a hostile client cannot make the server buffer anything worth having.
|
||||
/// </remarks>
|
||||
private const int MaxInboundFrameBytes = 4 * 1024;
|
||||
|
||||
/// <summary>How long to wait for the close handshake before dropping the socket.</summary>
|
||||
private static readonly TimeSpan CloseTimeout = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <inheritdoc />
|
||||
public override void Configure()
|
||||
{
|
||||
Get(VaultEvents.Path);
|
||||
|
||||
// Enrolled, matching sync. A caller with no identity key holds no vault key either, so every
|
||||
// notice this socket could send is about ciphertext they cannot read.
|
||||
Policies(Auth.EnrolledPolicy);
|
||||
|
||||
Description(b => b
|
||||
.WithName("VaultEvents")
|
||||
.WithSummary("Pushes a notice when a vault the caller can read has changed.")
|
||||
.WithTags("Events"));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task HandleAsync(CancellationToken ct)
|
||||
{
|
||||
if (await RefusedAsync().ConfigureAwait(false))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var (userId, vaults) = await ResolveAccessAsync(ct).ConfigureAwait(false);
|
||||
|
||||
// Admitted before the upgrade so a refusal costs nothing, but answered *through* the socket
|
||||
// rather than as an HTTP status: a constrained WebSocket client cannot read the status of a
|
||||
// failed upgrade, and "you have too many open" is precisely the case where the client needs
|
||||
// to know to back off rather than retry. Same reasoning as ADR 0004 on request headers.
|
||||
var connection = hub.TryAdmit(userId, vaults);
|
||||
|
||||
try
|
||||
{
|
||||
using var socket = await HttpContext.WebSockets
|
||||
.AcceptWebSocketAsync(new WebSocketAcceptContext { SubProtocol = VaultEvents.SubProtocol })
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (connection is null)
|
||||
{
|
||||
await CloseAsync(
|
||||
socket,
|
||||
new Closure(
|
||||
VaultEvents.TooManyConnectionsCloseCode, "Too many open event sockets."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
await PumpAsync(socket, connection, ct).ConfigureAwait(false);
|
||||
}
|
||||
finally
|
||||
{
|
||||
// Inside a try that starts *before* the upgrade, because an accept that throws — a client
|
||||
// that abandoned the handshake — would otherwise leave an admitted connection in the hub
|
||||
// for the life of the process, counting against this account's cap and taking a slot from
|
||||
// the sockets that did open.
|
||||
if (connection is not null)
|
||||
{
|
||||
hub.Remove(connection);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Answers the requests that are not an event socket at all, as ordinary HTTP.
|
||||
/// </summary>
|
||||
/// <returns>Whether a response was sent and the handler should stop.</returns>
|
||||
/// <remarks>
|
||||
/// All three answers are problem documents rather than bare statuses, because each one has a
|
||||
/// different remedy and a client that cannot tell them apart would retry the two that will never
|
||||
/// succeed. Answered before the upgrade, so a caller that got the handshake wrong reads why in a
|
||||
/// body rather than inferring it from a socket that closed.
|
||||
/// </remarks>
|
||||
private async Task<bool> RefusedAsync()
|
||||
{
|
||||
if (!options.Value.Enabled)
|
||||
{
|
||||
// 404 rather than 501: the feature is absent from this deployment, and /api/v1/meta does
|
||||
// not advertise it. A client that dialled anyway keeps polling, which is correct.
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status404NotFound,
|
||||
ProblemCodes.EventsUnavailable,
|
||||
"This server does not push vault changes. Synchronise on a timer instead; "
|
||||
+ "GET /api/v1/meta lists the features it does offer."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!HttpContext.WebSockets.IsWebSocketRequest)
|
||||
{
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status400BadRequest,
|
||||
ProblemCodes.MalformedRequest,
|
||||
"This endpoint is a WebSocket. Send an upgrade request offering the "
|
||||
+ $"'{VaultEvents.SubProtocol}' subprotocol."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
// The subprotocol is this API's version negotiation for the socket, so an upgrade that does
|
||||
// not offer it is refused rather than accepted and answered in a dialect the caller may not
|
||||
// read. See VaultEvents.SubProtocol.
|
||||
if (!HttpContext.WebSockets.WebSocketRequestedProtocols
|
||||
.Contains(VaultEvents.SubProtocol, StringComparer.Ordinal))
|
||||
{
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status400BadRequest,
|
||||
ProblemCodes.MalformedRequest,
|
||||
$"This server speaks '{VaultEvents.SubProtocol}', which the upgrade request did "
|
||||
+ "not offer."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads who the caller is and which vaults they may follow, in a scope of its own.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A fresh scope, disposed at once, rather than services injected into this endpoint — which is
|
||||
/// the lesson ADR 0004 records paying for on the relay. This handler runs for as long as the
|
||||
/// socket is open, so anything scoped it held would be a <c>DbContext</c> alive for hours, and a
|
||||
/// few hundred of those exhaust the connection pool. The database is touched here and in
|
||||
/// <see cref="RefreshAsync"/>, briefly, and nowhere else.
|
||||
/// </remarks>
|
||||
private async Task<(Guid UserId, FrozenSet<Guid> Vaults)> ResolveAccessAsync(
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var scope = scopes.CreateAsyncScope();
|
||||
await using var _ = scope.ConfigureAwait(false);
|
||||
|
||||
var currentUser = scope.ServiceProvider.GetRequiredService<ICurrentUserContext>();
|
||||
var vaultAccess = scope.ServiceProvider.GetRequiredService<IVaultAccessService>();
|
||||
|
||||
var user = await currentUser.GetOrProvisionAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return (user.Id, await ReachableAsync(vaultAccess, user.Id, cancellationToken).ConfigureAwait(false));
|
||||
}
|
||||
|
||||
private static async Task<FrozenSet<Guid>> ReachableAsync(
|
||||
IVaultAccessService vaultAccess,
|
||||
Guid userId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var accessible = await vaultAccess.ListAsync(userId, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return accessible
|
||||
.Where(access => access.Vault is not null
|
||||
&& access.Permissions.HasFlag(PermissionFlags.Read))
|
||||
.Select(access => access.Vault!.Id)
|
||||
.ToFrozenSet();
|
||||
}
|
||||
|
||||
/// <summary>Runs the socket until something ends it, then closes it politely.</summary>
|
||||
/// <remarks>
|
||||
/// Three loops rather than one: reading a socket and writing to it are independent waits, and the
|
||||
/// clock is a third. They are joined by <see cref="Task.WhenAny(Task[])"/> and then <em>all</em>
|
||||
/// awaited before the close is written, because a close frame racing a notice frame is a protocol
|
||||
/// violation that presents as a client dropping its connection for no visible reason.
|
||||
/// </remarks>
|
||||
private async Task PumpAsync(
|
||||
WebSocket socket,
|
||||
VaultEventConnection connection,
|
||||
CancellationToken requestAborted)
|
||||
{
|
||||
var settings = options.Value;
|
||||
|
||||
using var pump = CancellationTokenSource.CreateLinkedTokenSource(
|
||||
requestAborted, lifetime.ApplicationStopping);
|
||||
|
||||
var closure = new Closure(
|
||||
(int)WebSocketCloseStatus.NormalClosure, string.Empty);
|
||||
|
||||
connection.TryEnqueue(new VaultEvent(
|
||||
VaultEventKinds.Hello,
|
||||
ServerTime: clock.GetUtcNow(),
|
||||
HeartbeatSeconds: (int)settings.HeartbeatInterval.TotalSeconds,
|
||||
VaultCount: connection.VaultCount));
|
||||
|
||||
var sending = SendAsync(socket, connection, pump.Token);
|
||||
var receiving = ReceiveAsync(socket, connection, pump.Token);
|
||||
var minding = MindAsync(connection, closure, pump.Token);
|
||||
|
||||
await Task.WhenAny(sending, receiving, minding).ConfigureAwait(false);
|
||||
|
||||
await pump.CancelAsync().ConfigureAwait(false);
|
||||
|
||||
// Nothing may still be mid-send when the close frame goes out.
|
||||
await Task.WhenAll(Settled(sending), Settled(receiving), Settled(minding)).ConfigureAwait(false);
|
||||
|
||||
if (lifetime.ApplicationStopping.IsCancellationRequested)
|
||||
{
|
||||
// 1001 "going away", so a client knows to reconnect immediately rather than treating a
|
||||
// rolling deployment as a server that has broken.
|
||||
closure.Set((int)WebSocketCloseStatus.EndpointUnavailable, "The server is shutting down.");
|
||||
}
|
||||
|
||||
EventsLog.ClosingConnection(logger, connection.UserId, closure.Reason);
|
||||
|
||||
await CloseAsync(socket, closure).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes queued frames to the socket, one at a time.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The <em>only</em> writer, which is what makes concurrent sends impossible without a lock: the
|
||||
/// heartbeat and the pong both go into the same queue rather than to the socket. A WebSocket
|
||||
/// permits one send at a time and faults permanently on a second, so this is not a tidiness
|
||||
/// preference.
|
||||
/// </remarks>
|
||||
private async Task SendAsync(
|
||||
WebSocket socket,
|
||||
VaultEventConnection connection,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
await foreach (var frame in connection.Outbound
|
||||
.ReadAllAsync(cancellationToken)
|
||||
.ConfigureAwait(false))
|
||||
{
|
||||
// Re-resolved before the notice is forwarded, not after: the client's answer to this frame
|
||||
// is to re-read its vault list, and the point of a newly shared vault is that the *next*
|
||||
// change to it produces a notice too. A socket that forwarded first would not follow the
|
||||
// new vault until its next periodic refresh.
|
||||
if (string.Equals(frame.Kind, VaultEventKinds.VaultsChanged, StringComparison.Ordinal))
|
||||
{
|
||||
await RefreshAsync(connection, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
var bytes = JsonSerializer.SerializeToUtf8Bytes(frame, DodoSshJsonContext.Default.VaultEvent);
|
||||
|
||||
await socket
|
||||
.SendAsync(bytes, WebSocketMessageType.Text, endOfMessage: true, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads what the client sends, which in this version is heartbeats and a close.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A client cannot ask to follow a vault, and that is deliberate rather than unfinished: a
|
||||
/// <c>subscribe(vaultId)</c> frame is an existence oracle for vault ids, which is the disclosure
|
||||
/// <c>SyncPullEndpoint</c> answers 404 rather than 403 to avoid. Subscription is decided from the
|
||||
/// caller's access and nothing else.
|
||||
/// </remarks>
|
||||
private static async Task ReceiveAsync(
|
||||
WebSocket socket,
|
||||
VaultEventConnection connection,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var buffer = new byte[MaxInboundFrameBytes];
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var received = await socket.ReceiveAsync(buffer, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (received.MessageType == WebSocketMessageType.Close)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Oversized, or split across frames. Nothing this protocol sends is either, so the client
|
||||
// is broken or probing; ending the socket is cheaper than reassembling for it.
|
||||
if (!received.EndOfMessage)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Binary is unused in v1 and skipped rather than refused, because ADR 0012 reserves it for
|
||||
// shared-session data — an older server meeting a newer client must ignore those, not
|
||||
// close on them.
|
||||
if (received.MessageType != WebSocketMessageType.Text)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (Kind(buffer.AsSpan(0, received.Count)) is VaultEventKinds.Ping)
|
||||
{
|
||||
connection.TryEnqueue(new VaultEvent(VaultEventKinds.Pong));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads a frame's kind, or null if it is not one this server understands.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A frame that will not parse is skipped rather than closing the socket. This is a control
|
||||
/// channel whose failure mode is "the client polls instead", so tolerating a frame from a newer
|
||||
/// client costs nothing and refusing one costs that client its push for the whole session.
|
||||
/// </remarks>
|
||||
private static string? Kind(ReadOnlySpan<byte> utf8)
|
||||
{
|
||||
try
|
||||
{
|
||||
return JsonSerializer.Deserialize(utf8, DodoSshJsonContext.Default.VaultEvent)?.Kind;
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Keeps the heartbeat going, the vault set current, and the socket inside its token's lifetime.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The deadline is the earlier of the access token's <c>exp</c> and a hard cap on how long any one
|
||||
/// socket may live. Closing on expiry is what keeps a long-lived connection from outliving the
|
||||
/// short-lived credential that authorised it; the client answers by reconnecting with a fresh
|
||||
/// token, which is a sub-second gap in a channel that degrades to polling anyway.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The wait is the shorter of the heartbeat and the time left, so the deadline is met to within a
|
||||
/// tick rather than to within a heartbeat.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private async Task MindAsync(
|
||||
VaultEventConnection connection,
|
||||
Closure closure,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var settings = options.Value;
|
||||
var started = clock.GetUtcNow();
|
||||
|
||||
var deadline = TokenExpiry() is { } expiry && expiry < started + settings.MaxConnectionDuration
|
||||
? (Expiry: expiry, ForToken: true)
|
||||
: (Expiry: started + settings.MaxConnectionDuration, ForToken: false);
|
||||
|
||||
var refreshed = started;
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var now = clock.GetUtcNow();
|
||||
var remaining = deadline.Expiry - now;
|
||||
|
||||
if (remaining <= TimeSpan.Zero)
|
||||
{
|
||||
closure.Set(
|
||||
deadline.ForToken
|
||||
? VaultEvents.TokenExpiredCloseCode
|
||||
: (int)WebSocketCloseStatus.NormalClosure,
|
||||
deadline.ForToken
|
||||
? "The access token has expired. Reconnect with a fresh one."
|
||||
: "This connection reached its maximum lifetime.");
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
var wait = settings.HeartbeatInterval < remaining ? settings.HeartbeatInterval : remaining;
|
||||
|
||||
await Task.Delay(wait, clock, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
now = clock.GetUtcNow();
|
||||
|
||||
if (now - refreshed >= settings.AccessRefreshInterval)
|
||||
{
|
||||
// The backstop for a grant withdrawn while this socket was open. What it bounds is
|
||||
// metadata — that a vault changed — because that is all a notice carries and reading
|
||||
// the vault still needs a key this server has never held. See ADR 0012.
|
||||
await RefreshAsync(connection, cancellationToken).ConfigureAwait(false);
|
||||
refreshed = now;
|
||||
}
|
||||
|
||||
connection.TryEnqueue(new VaultEvent(VaultEventKinds.Ping, ServerTime: now));
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Re-reads which vaults this socket may follow.</summary>
|
||||
/// <remarks>
|
||||
/// A failure is logged and swallowed. The alternative is dropping a working socket because one
|
||||
/// database call timed out, which would trade an occasionally stale vault set for an outage.
|
||||
/// </remarks>
|
||||
private async Task RefreshAsync(VaultEventConnection connection, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
var scope = scopes.CreateAsyncScope();
|
||||
await using var _ = scope.ConfigureAwait(false);
|
||||
|
||||
var vaultAccess = scope.ServiceProvider.GetRequiredService<IVaultAccessService>();
|
||||
|
||||
connection.Resubscribe(
|
||||
await ReachableAsync(vaultAccess, connection.UserId, cancellationToken)
|
||||
.ConfigureAwait(false));
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// The socket is closing.
|
||||
}
|
||||
catch (Exception exception) when (exception is not OutOfMemoryException)
|
||||
{
|
||||
EventsLog.AccessRefreshFailed(logger, connection.UserId, exception);
|
||||
}
|
||||
}
|
||||
|
||||
/// <remarks>
|
||||
/// <c>MapInboundClaims</c> is off — see <see cref="Auth"/> — so the claim is spelled as the
|
||||
/// provider issued it rather than as a WS-Federation URI. Null is treated as "no bound from the
|
||||
/// token", which the bearer handler's <c>RequireExpirationTime</c> should make unreachable; the
|
||||
/// lifetime cap covers it either way.
|
||||
/// </remarks>
|
||||
private DateTimeOffset? TokenExpiry() =>
|
||||
long.TryParse(
|
||||
HttpContext.User.FindFirstValue("exp"),
|
||||
NumberStyles.Integer,
|
||||
CultureInfo.InvariantCulture,
|
||||
out var seconds)
|
||||
? DateTimeOffset.FromUnixTimeSeconds(seconds)
|
||||
: null;
|
||||
|
||||
private static async Task CloseAsync(WebSocket socket, Closure closure)
|
||||
{
|
||||
if (socket.State is not (WebSocketState.Open or WebSocketState.CloseReceived))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
using var timeout = new CancellationTokenSource(CloseTimeout);
|
||||
|
||||
try
|
||||
{
|
||||
await socket
|
||||
.CloseOutputAsync((WebSocketCloseStatus)closure.Code, closure.Reason, timeout.Token)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception)
|
||||
when (exception is OperationCanceledException or WebSocketException or ObjectDisposedException)
|
||||
{
|
||||
// The peer is already gone. There is nothing to tell it and nothing to recover.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Awaits a pump loop, treating its cancellation and its socket faults as the ordinary end.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Every one of these loops ends by being cancelled or by the socket going away, so an exception
|
||||
/// here is the expected shape of "this connection is over" rather than a fault to propagate — and
|
||||
/// propagating it would skip the close frame the other side is waiting for.
|
||||
/// </remarks>
|
||||
private static async Task Settled(Task loop)
|
||||
{
|
||||
try
|
||||
{
|
||||
await loop.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception)
|
||||
when (exception is OperationCanceledException or WebSocketException or ObjectDisposedException)
|
||||
{
|
||||
// Expected.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Why the socket is being closed, decided by whichever loop ended first.</summary>
|
||||
/// <remarks>
|
||||
/// Mutable and shared, and safe without a lock for one specific reason: it is written by the pump
|
||||
/// loops and read only after <see cref="Task.WhenAll(Task[])"/> over all of them, which is a
|
||||
/// memory barrier. Writes race only with each other, and any of them is a true answer.
|
||||
/// </remarks>
|
||||
private sealed class Closure(int code, string reason)
|
||||
{
|
||||
internal int Code { get; private set; } = code;
|
||||
|
||||
internal string Reason { get; private set; } = reason;
|
||||
|
||||
internal void Set(int code, string reason)
|
||||
{
|
||||
Code = code;
|
||||
Reason = reason;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
namespace DodoSSH.Api.Features.Events;
|
||||
|
||||
/// <summary>Source-generated log messages for the event socket.</summary>
|
||||
/// <remarks>
|
||||
/// Ids and counts only, as everywhere else. A notice carries no ciphertext to leak, but which vault
|
||||
/// changed and when is still the metadata ADR 0001 asks be kept to what is diagnostically useful.
|
||||
/// </remarks>
|
||||
internal static partial class EventsLog
|
||||
{
|
||||
[LoggerMessage(
|
||||
EventId = 2201,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Event socket opened for user {UserId} following {VaultCount} vault(s); "
|
||||
+ "{ConnectionCount} open on this node.")]
|
||||
internal static partial void ConnectionOpened(
|
||||
ILogger logger,
|
||||
Guid userId,
|
||||
int vaultCount,
|
||||
int connectionCount);
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 2202,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Event socket closed for user {UserId}; {ConnectionCount} open on this node.")]
|
||||
internal static partial void ConnectionClosed(ILogger logger, Guid userId, int connectionCount);
|
||||
|
||||
/// <remarks>
|
||||
/// Information rather than Debug: a refused socket is a client that will poll for the rest of its
|
||||
/// session, and an operator seeing these has a cap to raise.
|
||||
/// </remarks>
|
||||
[LoggerMessage(
|
||||
EventId = 2203,
|
||||
Level = LogLevel.Information,
|
||||
Message = "Refused an event socket for user {UserId}: {Limit} is already reached.")]
|
||||
internal static partial void ConnectionRefused(ILogger logger, Guid userId, string limit);
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 2204,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Announced vault {VaultId} at sequence {Sequence} to {ConnectionCount} socket(s).")]
|
||||
internal static partial void VaultChangePublished(
|
||||
ILogger logger,
|
||||
Guid vaultId,
|
||||
long sequence,
|
||||
int connectionCount);
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 2205,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Announced a vault access change to {ConnectionCount} socket(s) of user {UserId}.")]
|
||||
internal static partial void AccessChangePublished(ILogger logger, Guid userId, int connectionCount);
|
||||
|
||||
/// <remarks>
|
||||
/// Warning, and it is worth being loud: the socket is still open and still delivering, but it is
|
||||
/// delivering about a vault set that may be stale. Everything else here is routine.
|
||||
/// </remarks>
|
||||
[LoggerMessage(
|
||||
EventId = 2206,
|
||||
Level = LogLevel.Warning,
|
||||
Message = "Could not re-resolve which vaults user {UserId}'s event socket may follow.")]
|
||||
internal static partial void AccessRefreshFailed(ILogger logger, Guid userId, Exception exception);
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 2207,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Closing user {UserId}'s event socket: {Reason}.")]
|
||||
internal static partial void ClosingConnection(ILogger logger, Guid userId, string reason);
|
||||
}
|
||||
@@ -0,0 +1,242 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Collections.Frozen;
|
||||
using System.Threading.Channels;
|
||||
using DodoSSH.Api.Setup;
|
||||
using DodoSSH.Contracts;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace DodoSSH.Api.Features.Events;
|
||||
|
||||
/// <summary>
|
||||
/// Tells connected clients that something they can read has moved.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Every method is <c>void</c> and returns having queued, never having sent. That is the contract, not
|
||||
/// an implementation detail: the callers are write paths that have just committed a transaction, and a
|
||||
/// publish that could block on a slow socket would make one client's bad network everybody else's
|
||||
/// latency. A notice that cannot be queued is dropped, which is safe because the client polls anyway.
|
||||
/// See ADR 0012.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// An interface because this is the seam a multi-node backplane implements — PostgreSQL
|
||||
/// <c>LISTEN</c>/<c>NOTIFY</c> is the obvious one and needs no infrastructure this stack does not
|
||||
/// already run. It is deliberately not implemented: fan-out today is in-process, so a deployment with
|
||||
/// more than one API replica notices writes handled by other replicas on the polling interval rather
|
||||
/// than at once. That is the behaviour before this feature existed, which is why it degrades rather
|
||||
/// than breaks.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public interface IVaultEventPublisher
|
||||
{
|
||||
/// <summary>Announces that a vault's change log has reached <paramref name="sequence"/>.</summary>
|
||||
/// <remarks>
|
||||
/// Call <em>after</em> the transaction commits, and outside the per-vault advisory lock ADR 0003
|
||||
/// takes. A notice sent from inside names a sequence no reader can see yet, and holds the vault's
|
||||
/// write lock across a socket write.
|
||||
/// </remarks>
|
||||
void VaultChanged(Guid vaultId, long sequence);
|
||||
|
||||
/// <summary>Announces that the set of vaults an account can reach is no longer what it was.</summary>
|
||||
/// <remarks>
|
||||
/// Takes the <em>recipient</em>, not the actor. Sharing is something one account does to another's
|
||||
/// list, and it is the other account that has to re-read.
|
||||
/// </remarks>
|
||||
void VaultAccessChanged(Guid userId);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Every event socket this node is holding.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Publishing walks the whole connection list and asks each one whether it cares, rather than keeping
|
||||
/// an index from vault to subscribers. With a per-node connection cap in the hundreds and an event rate
|
||||
/// bounded by how often people edit keychains, the walk is not measurable — and the index is not free:
|
||||
/// a connection's vault set is re-resolved while it is live, so every re-subscription would have to
|
||||
/// move it between buckets under a lock that publishing also takes. The simpler shape is the one whose
|
||||
/// races are obvious.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A singleton, holding no scoped service and no database context. Connections outlive requests by
|
||||
/// design and anything request-scoped they captured would outlive its scope with them.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal sealed class VaultEventHub(
|
||||
IOptions<EventsOptions> options,
|
||||
TimeProvider clock,
|
||||
ILogger<VaultEventHub> logger) : IVaultEventPublisher
|
||||
{
|
||||
private readonly ConcurrentDictionary<Guid, VaultEventConnection> connections = new();
|
||||
|
||||
/// <summary>
|
||||
/// Serialises admission so the caps are caps rather than approximations.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Counting and inserting under one lock, because the two done separately let N simultaneous
|
||||
/// connects all read the same under-cap count and all insert. Contended only by connects, which
|
||||
/// happen once per client per session; publishing never takes it.
|
||||
/// </remarks>
|
||||
private readonly Lock admission = new();
|
||||
|
||||
/// <summary>How many sockets this node is holding. Diagnostics and tests.</summary>
|
||||
internal int Count => connections.Count;
|
||||
|
||||
/// <summary>
|
||||
/// Admits a socket, or refuses it because a cap is already met.
|
||||
/// </summary>
|
||||
/// <returns>The connection, or null when a limit refused it.</returns>
|
||||
internal VaultEventConnection? TryAdmit(Guid userId, FrozenSet<Guid> vaults)
|
||||
{
|
||||
var limits = options.Value;
|
||||
|
||||
lock (admission)
|
||||
{
|
||||
if (connections.Count >= limits.MaxConnectionsTotal)
|
||||
{
|
||||
EventsLog.ConnectionRefused(logger, userId, "the node limit");
|
||||
return null;
|
||||
}
|
||||
|
||||
var held = 0;
|
||||
foreach (var existing in connections.Values)
|
||||
{
|
||||
if (existing.UserId == userId && ++held >= limits.MaxConnectionsPerUser)
|
||||
{
|
||||
EventsLog.ConnectionRefused(logger, userId, "the per-account limit");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
var connection = new VaultEventConnection(userId, vaults, limits.OutboundQueueDepth);
|
||||
|
||||
// Cannot collide: the id is fresh and this is the only insert.
|
||||
connections[connection.Id] = connection;
|
||||
|
||||
EventsLog.ConnectionOpened(logger, userId, vaults.Count, connections.Count);
|
||||
|
||||
return connection;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Forgets a socket that has closed.</summary>
|
||||
internal void Remove(VaultEventConnection connection)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(connection);
|
||||
|
||||
connections.TryRemove(connection.Id, out _);
|
||||
connection.Complete();
|
||||
|
||||
EventsLog.ConnectionClosed(logger, connection.UserId, connections.Count);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void VaultChanged(Guid vaultId, long sequence)
|
||||
{
|
||||
var notice = new VaultEvent(
|
||||
VaultEventKinds.VaultChanged,
|
||||
VaultId: vaultId,
|
||||
Sequence: sequence,
|
||||
ServerTime: clock.GetUtcNow());
|
||||
|
||||
var delivered = 0;
|
||||
|
||||
foreach (var connection in connections.Values)
|
||||
{
|
||||
if (connection.IsSubscribedTo(vaultId) && connection.TryEnqueue(notice))
|
||||
{
|
||||
delivered++;
|
||||
}
|
||||
}
|
||||
|
||||
if (delivered > 0)
|
||||
{
|
||||
EventsLog.VaultChangePublished(logger, vaultId, sequence, delivered);
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void VaultAccessChanged(Guid userId)
|
||||
{
|
||||
var notice = new VaultEvent(VaultEventKinds.VaultsChanged, ServerTime: clock.GetUtcNow());
|
||||
|
||||
var delivered = 0;
|
||||
|
||||
foreach (var connection in connections.Values)
|
||||
{
|
||||
if (connection.UserId == userId && connection.TryEnqueue(notice))
|
||||
{
|
||||
delivered++;
|
||||
}
|
||||
}
|
||||
|
||||
if (delivered > 0)
|
||||
{
|
||||
EventsLog.AccessChangePublished(logger, userId, delivered);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One open socket, as the hub sees it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Deliberately knows nothing about WebSockets. The hub queues frames here and the endpoint's pump
|
||||
/// takes them away, which is what keeps a publish from ever touching a socket — and what lets the
|
||||
/// whole fan-out be tested without one.
|
||||
/// </remarks>
|
||||
internal sealed class VaultEventConnection
|
||||
{
|
||||
private readonly Channel<VaultEvent> outbound;
|
||||
|
||||
private FrozenSet<Guid> vaults;
|
||||
|
||||
internal VaultEventConnection(Guid userId, FrozenSet<Guid> vaults, int queueDepth)
|
||||
{
|
||||
Id = Guid.CreateVersion7();
|
||||
UserId = userId;
|
||||
this.vaults = vaults;
|
||||
|
||||
// DropOldest, and the choice is what makes a slow reader harmless. A notice says "vault X has
|
||||
// moved to at least sequence N", so a newer one subsumes the one it displaces and the client's
|
||||
// answer — pull that vault — is identical either way. The writer therefore never waits and
|
||||
// TryWrite never fails, which is what lets the publish path be non-blocking and void.
|
||||
outbound = Channel.CreateBounded<VaultEvent>(new BoundedChannelOptions(queueDepth)
|
||||
{
|
||||
FullMode = BoundedChannelFullMode.DropOldest,
|
||||
SingleReader = true,
|
||||
SingleWriter = false,
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>Identifies this connection within the hub. Never sent to a client.</summary>
|
||||
internal Guid Id { get; }
|
||||
|
||||
/// <summary>The account that opened it.</summary>
|
||||
internal Guid UserId { get; }
|
||||
|
||||
/// <summary>Frames waiting to be written to the socket.</summary>
|
||||
internal ChannelReader<VaultEvent> Outbound => outbound.Reader;
|
||||
|
||||
/// <summary>How many vaults this socket currently follows.</summary>
|
||||
internal int VaultCount => Volatile.Read(ref vaults).Count;
|
||||
|
||||
/// <summary>Whether a change to this vault concerns this socket.</summary>
|
||||
internal bool IsSubscribedTo(Guid vaultId) => Volatile.Read(ref vaults).Contains(vaultId);
|
||||
|
||||
/// <summary>
|
||||
/// Replaces what this socket follows, after its account's access was re-resolved.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A whole-set swap of an immutable set rather than a mutation, so a publish walking the list
|
||||
/// concurrently reads either the old set or the new one and never a half-built one. No lock: the
|
||||
/// only writer is this connection's own pump.
|
||||
/// </remarks>
|
||||
internal void Resubscribe(FrozenSet<Guid> replacement) => Volatile.Write(ref vaults, replacement);
|
||||
|
||||
/// <summary>Queues a frame. Never blocks, and never fails — see the channel's full mode.</summary>
|
||||
internal bool TryEnqueue(VaultEvent frame) => outbound.Writer.TryWrite(frame);
|
||||
|
||||
/// <summary>Signals that nothing more will be queued, which ends the pump's drain loop.</summary>
|
||||
internal void Complete() => outbound.Writer.TryComplete();
|
||||
}
|
||||
@@ -19,6 +19,7 @@ namespace DodoSSH.Api.Features.Meta;
|
||||
internal sealed class GetMetaEndpoint(
|
||||
IOptions<SyncOptions> sync,
|
||||
IOptions<RelayOptions> relay,
|
||||
IOptions<EventsOptions> events,
|
||||
IOptions<ServerOptions> server)
|
||||
: EndpointWithoutRequest<Ok<MetaResponse>>
|
||||
{
|
||||
@@ -52,6 +53,14 @@ internal sealed class GetMetaEndpoint(
|
||||
features.Add(RelayFeature);
|
||||
}
|
||||
|
||||
// Advertised so a client knows whether to hold a socket open or rely on its timer. Absence is
|
||||
// not an error — synchronising on a timer is the supported behaviour and the socket only makes
|
||||
// it early — which is why this is a feature flag rather than a version bump. See ADR 0012.
|
||||
if (events.Value.Enabled)
|
||||
{
|
||||
features.Add(VaultEvents.Feature);
|
||||
}
|
||||
|
||||
return Task.FromResult(TypedResults.Ok(new MetaResponse(
|
||||
ServerVersion: ServerVersion,
|
||||
ApiVersions: [1],
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using DodoSSH.Api.Authorization;
|
||||
using DodoSSH.Api.Features.Events;
|
||||
using DodoSSH.Api.Setup;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain.Authorization;
|
||||
@@ -81,6 +82,7 @@ internal sealed class SyncPullEndpoint(
|
||||
internal sealed class SyncPushEndpoint(
|
||||
ICurrentUserContext currentUser,
|
||||
IVaultAccessService vaultAccess,
|
||||
IVaultEventPublisher events,
|
||||
SyncService sync)
|
||||
: Endpoint<SyncPushRequest, Results<Ok<SyncPushResponse>, NotFound, ProblemHttpResult>>
|
||||
{
|
||||
@@ -127,6 +129,8 @@ internal sealed class SyncPushEndpoint(
|
||||
// single stale item cannot block everything else a client queued while offline.
|
||||
var response = await sync.PushAsync(access.Vault!, user.Id, req, ct).ConfigureAwait(false);
|
||||
|
||||
Announce(access.Vault!.Id, response);
|
||||
|
||||
return TypedResults.Ok(response);
|
||||
}
|
||||
catch (PushBatchTooLargeException exception)
|
||||
@@ -142,4 +146,41 @@ internal sealed class SyncPushEndpoint(
|
||||
StatusCodes.Status400BadRequest, ProblemCodes.PushBatchTooLarge, exception.Message);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Tells every socket following this vault that it has moved.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Here rather than inside <see cref="SyncService.PushAsync"/>, and that placement is the point:
|
||||
/// the push has committed and released the per-vault advisory lock by the time this runs. Announced
|
||||
/// from inside, it would name a sequence no reader could see yet and would hold the lock that
|
||||
/// serialises writers across a fan-out. See ADR 0003 and ADR 0012.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The highest <em>applied</em> sequence, ignoring duplicates: a duplicate means an earlier push of
|
||||
/// that operation already landed, and it was announced then. Nothing applied means nothing to say —
|
||||
/// a batch of pure conflicts moved no vault, and announcing one anyway would have every client pull
|
||||
/// for a change that is not there.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private void Announce(Guid vaultId, SyncPushResponse response)
|
||||
{
|
||||
var highest = 0L;
|
||||
|
||||
foreach (var result in response.Results)
|
||||
{
|
||||
if (result.Status == SyncOperationStatus.Applied
|
||||
&& result.ChangeSequence is { } sequence
|
||||
&& sequence > highest)
|
||||
{
|
||||
highest = sequence;
|
||||
}
|
||||
}
|
||||
|
||||
if (highest > 0)
|
||||
{
|
||||
events.VaultChanged(vaultId, highest);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System.Globalization;
|
||||
using DodoSSH.Api.Features.Events;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain;
|
||||
using DodoSSH.Infrastructure;
|
||||
@@ -62,6 +63,7 @@ internal readonly record struct TeamAccess(Team? Team, TeamRole Role)
|
||||
internal sealed class TeamService(
|
||||
DodoDbContext database,
|
||||
TimeProvider clock,
|
||||
IVaultEventPublisher events,
|
||||
ILogger<TeamService> logger)
|
||||
{
|
||||
/// <summary>Longest acceptable slug. Matches the column.</summary>
|
||||
@@ -548,6 +550,11 @@ internal sealed class TeamService(
|
||||
|
||||
TeamLog.MemberAdded(logger, teamId, target.Id, role, actor.Id);
|
||||
|
||||
// Membership is what the server will serve, so every vault this team owns has just appeared in
|
||||
// the new member's list — before anybody wraps a key to them, which is a separate act and its
|
||||
// own notice. Told at once rather than on their next pass. See ADR 0012.
|
||||
events.VaultAccessChanged(target.Id);
|
||||
|
||||
return await DescribeAsync(target, membership, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
@@ -744,6 +751,11 @@ internal sealed class TeamService(
|
||||
}).ConfigureAwait(false);
|
||||
|
||||
TeamLog.MemberRemoved(logger, teamId, memberId, actor.Id, revoked);
|
||||
|
||||
// After the commit, so their client re-reads a list the server has already stopped serving
|
||||
// those vaults from. Their open socket re-resolves as it forwards this, which is what stops it
|
||||
// announcing changes to vaults they have just lost.
|
||||
events.VaultAccessChanged(memberId);
|
||||
}
|
||||
|
||||
/// <summary>Revokes one user's grants on every vault a team owns, and flags each for rekey.</summary>
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System.Security.Cryptography;
|
||||
using DodoSSH.Api.Features.Events;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain;
|
||||
using DodoSSH.Infrastructure;
|
||||
@@ -27,6 +28,7 @@ namespace DodoSSH.Api.Features.Teams;
|
||||
internal sealed class VaultGrantService(
|
||||
DodoDbContext database,
|
||||
TimeProvider clock,
|
||||
IVaultEventPublisher events,
|
||||
ILogger<VaultGrantService> logger)
|
||||
{
|
||||
/// <summary>
|
||||
@@ -414,6 +416,11 @@ internal sealed class VaultGrantService(
|
||||
|
||||
TeamLog.GrantIssued(
|
||||
logger, vault.Id, generation, request.RecipientUserId, actor.Id);
|
||||
|
||||
// The recipient, never the actor. This is the whole of what makes a shared vault arrive at
|
||||
// once rather than on the recipient's next pass — and it is the case the README has had to
|
||||
// apologise for since sharing shipped. See ADR 0012.
|
||||
events.VaultAccessChanged(request.RecipientUserId);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -701,6 +708,11 @@ internal sealed class VaultGrantService(
|
||||
|
||||
TeamLog.GrantRevoked(logger, vault.Id, recipientUserId, actor.Id);
|
||||
|
||||
// Told so their client stops showing a vault it can no longer open, rather than leaving it
|
||||
// listed until the next pass. It does not reach what they already pulled — nothing can, see
|
||||
// ADR 0001 — and the server-side effect is immediate regardless of whether this arrives.
|
||||
events.VaultAccessChanged(recipientUserId);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using DodoSSH.Api.Authorization;
|
||||
using DodoSSH.Api.Features.Events;
|
||||
using DodoSSH.Api.Features.Identity;
|
||||
using DodoSSH.Api.Features.Sync;
|
||||
using DodoSSH.Api.Features.Teams;
|
||||
@@ -47,6 +48,14 @@ builder.Services.AddScoped<VaultGrantService>();
|
||||
builder.Services.AddScoped<IIdentityBindingVerifier, IdentityBindingVerifier>();
|
||||
builder.Services.AddSingleton<ICursorKeyProvider, CursorKeyProvider>();
|
||||
|
||||
// A singleton, because the sockets it holds outlive the requests that opened them. Registered twice
|
||||
// resolving to the same instance, for the reason the invitation claim above is: the endpoint needs the
|
||||
// whole hub — admit, remove, count — while the write paths that announce a change need only the two
|
||||
// methods that announce one, and should not gain a reference to connection management to get them.
|
||||
builder.Services.AddSingleton<VaultEventHub>();
|
||||
builder.Services.AddSingleton<IVaultEventPublisher>(
|
||||
provider => provider.GetRequiredService<VaultEventHub>());
|
||||
|
||||
// Scoped rather than the AddAuthorization default of singleton: the handler reads the request's
|
||||
// DbContext, and a singleton would capture one for the lifetime of the process.
|
||||
builder.Services.AddScoped<IAuthorizationHandler, EnrolledHandler>();
|
||||
@@ -65,6 +74,13 @@ var app = builder.Build();
|
||||
|
||||
app.BlockFastEndpointsRouteTable();
|
||||
|
||||
// Before the authentication middleware, because the upgrade handshake has to survive it: the events
|
||||
// endpoint answers an ordinary authenticated request that happens to become a socket, and without
|
||||
// this the upgrade is never offered and the handler sees a plain GET. No allow-list of origins is
|
||||
// configured, deliberately — every client here is a native application sending a bearer token, so
|
||||
// there is no browser origin to trust and nothing a cross-site request could reach without one.
|
||||
app.UseWebSockets();
|
||||
|
||||
app.UseAuthentication();
|
||||
app.UseAuthorization();
|
||||
|
||||
|
||||
@@ -38,6 +38,22 @@ internal static class Configuration
|
||||
"Sync:DefaultPullLimit must not exceed Sync:MaxPullLimit.")
|
||||
.ValidateOnStart();
|
||||
|
||||
services.AddOptions<EventsOptions>()
|
||||
.BindConfiguration(EventsOptions.SectionName)
|
||||
.ValidateDataAnnotations()
|
||||
.Validate(
|
||||
options => options.HeartbeatInterval > TimeSpan.Zero,
|
||||
"Events:HeartbeatInterval must be greater than zero.")
|
||||
.Validate(
|
||||
options => options.AccessRefreshInterval > TimeSpan.Zero,
|
||||
"Events:AccessRefreshInterval must be greater than zero.")
|
||||
.Validate(
|
||||
options => options.MaxConnectionDuration > options.AccessRefreshInterval,
|
||||
"Events:MaxConnectionDuration must exceed Events:AccessRefreshInterval; a connection "
|
||||
+ "that never lives long enough to re-read its own access has none of the bound that "
|
||||
+ "setting exists to provide.")
|
||||
.ValidateOnStart();
|
||||
|
||||
services.AddOptions<RelayOptions>()
|
||||
.BindConfiguration(RelayOptions.SectionName)
|
||||
.ValidateDataAnnotations()
|
||||
|
||||
@@ -159,6 +159,82 @@ public sealed class RelayOptions
|
||||
public TimeSpan DrainTimeout { get; set; } = TimeSpan.FromSeconds(30);
|
||||
}
|
||||
|
||||
/// <summary>Realtime push settings. See ADR 0012.</summary>
|
||||
/// <remarks>
|
||||
/// Every one of these bounds a socket rather than a feature: with the whole thing off, or every cap
|
||||
/// met, clients synchronise on their timer exactly as they did before this existed. That is what
|
||||
/// makes it safe for an operator to turn any of them down.
|
||||
/// </remarks>
|
||||
public sealed class EventsOptions
|
||||
{
|
||||
/// <summary>Configuration section name.</summary>
|
||||
public const string SectionName = "Events";
|
||||
|
||||
/// <summary>
|
||||
/// Whether this deployment pushes vault changes at all.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// On by default, unlike the relay: this needs no outbound network, no target resolution and no
|
||||
/// new trust, and a deployment behind a proxy that will not upgrade should say so here rather than
|
||||
/// have every client discover it by failing.
|
||||
/// </remarks>
|
||||
public bool Enabled { get; set; } = true;
|
||||
|
||||
/// <summary>Maximum concurrent sockets per node.</summary>
|
||||
[Range(1, 100_000)]
|
||||
public int MaxConnectionsTotal { get; set; } = 500;
|
||||
|
||||
/// <summary>
|
||||
/// Maximum concurrent sockets per account.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Per account rather than per device, because the server cannot see a device here. Eight is a
|
||||
/// laptop, a desktop, a phone and room to reconnect before the old socket has been reaped.
|
||||
/// </remarks>
|
||||
[Range(1, 1000)]
|
||||
public int MaxConnectionsPerUser { get; set; } = 8;
|
||||
|
||||
/// <summary>
|
||||
/// How many notices may be queued for one socket before the oldest are dropped.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A notice names a vault and a position, so a newer one subsumes the one it replaces. The depth
|
||||
/// therefore buys smoothness over a brief stall and nothing else — losing the tail of a burst
|
||||
/// costs a client nothing, because the newest notice still says to pull.
|
||||
/// </remarks>
|
||||
[Range(1, 10_000)]
|
||||
public int OutboundQueueDepth { get; set; } = 64;
|
||||
|
||||
/// <summary>
|
||||
/// How often the server pings an idle socket.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Below the sixty seconds most reverse proxies idle out at, because a silent socket that a proxy
|
||||
/// has quietly dropped is indistinguishable from a quiet one until something is sent down it.
|
||||
/// </remarks>
|
||||
public TimeSpan HeartbeatInterval { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// How often an open socket re-reads which vaults its account may follow.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The backstop for a grant withdrawn mid-connection. Grants and membership changes publish
|
||||
/// immediately, so this is what covers the paths that do not — and what bounds the window if one
|
||||
/// is ever added without remembering to.
|
||||
/// </remarks>
|
||||
public TimeSpan AccessRefreshInterval { get; set; } = TimeSpan.FromMinutes(5);
|
||||
|
||||
/// <summary>
|
||||
/// The longest any one socket may live, regardless of its token.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A socket normally ends at its access token's expiry, which is far shorter. This is the bound
|
||||
/// for a provider that issues long-lived tokens, and it is what makes "no connection is older than
|
||||
/// this" a property of the server rather than of the identity provider's configuration.
|
||||
/// </remarks>
|
||||
public TimeSpan MaxConnectionDuration { get; set; } = TimeSpan.FromHours(12);
|
||||
}
|
||||
|
||||
/// <summary>Sync protocol limits.</summary>
|
||||
public sealed class SyncOptions
|
||||
{
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using DodoSSH.Api.Features.Events;
|
||||
using DodoSSH.Api.Features.Identity;
|
||||
using DodoSSH.Api.Features.Meta;
|
||||
using DodoSSH.Api.Features.Sync;
|
||||
@@ -43,6 +44,7 @@ internal static class EndpointRegistration
|
||||
typeof(ReadKeyLogEndpoint),
|
||||
typeof(SyncPullEndpoint),
|
||||
typeof(SyncPushEndpoint),
|
||||
typeof(VaultEventsEndpoint),
|
||||
typeof(CreateTeamEndpoint),
|
||||
typeof(ListTeamsEndpoint),
|
||||
typeof(UpdateTeamEndpoint),
|
||||
|
||||
@@ -26,6 +26,11 @@
|
||||
"MaxConcurrentSessionsPerUser": 10,
|
||||
"MaxConcurrentSessionsTotal": 200
|
||||
},
|
||||
"Events": {
|
||||
"Enabled": true,
|
||||
"MaxConnectionsTotal": 500,
|
||||
"MaxConnectionsPerUser": 8
|
||||
},
|
||||
"Sync": {
|
||||
"MaxOperationsPerPush": 500,
|
||||
"MaxPayloadBytes": 8388608,
|
||||
|
||||
@@ -0,0 +1,552 @@
|
||||
using System.Net.WebSockets;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Channels;
|
||||
using DodoSSH.Contracts;
|
||||
|
||||
namespace DodoSSH.Client.Api;
|
||||
|
||||
/// <summary>
|
||||
/// A server's "pull now" notices, as everything above the transport needs them.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// A queue to read from rather than an event to subscribe to, and that shape is the point: the one
|
||||
/// consumer is a synchronisation loop that already waits on a timer, so it can wait on this the same
|
||||
/// way and keep every continuation on the thread it started from. An event would deliver on whichever
|
||||
/// thread the socket happened to complete on, which in a user interface is the difference between
|
||||
/// working and an intermittent rendering fault nobody can reproduce.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Reading this is never how a change is applied.</b> A notice says which vault moved and nothing
|
||||
/// else; the answer to it is the ordinary delta pull. See ADR 0012.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public interface IVaultEventStream : IDisposable
|
||||
{
|
||||
/// <summary>Whether a socket is currently established.</summary>
|
||||
/// <remarks>
|
||||
/// For the interface to say whether it is live, not for a caller to branch on before reading:
|
||||
/// synchronising is correct whether or not this is true, because the timer is the fallback.
|
||||
/// </remarks>
|
||||
bool IsConnected { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Waits for the next notice.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Connects on the first call and reconnects for as long as it is read, so a caller neither starts
|
||||
/// nor restarts anything. A server that cannot be reached is not an error here — it is a wait that
|
||||
/// has not finished — because the caller's alternative is the timer it is already running.
|
||||
/// </remarks>
|
||||
ValueTask<VaultEvent> ReadAsync(CancellationToken cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Takes a notice if one is already waiting, without blocking.
|
||||
/// </summary>
|
||||
/// <returns>Whether there was one.</returns>
|
||||
/// <remarks>
|
||||
/// How a caller coalesces a burst. Five people saving at once produces five notices whose answer
|
||||
/// is a single synchronisation pass, so the loop reads one, waits a moment, and swallows the rest
|
||||
/// rather than running the same pull five times.
|
||||
/// </remarks>
|
||||
bool TryRead(out VaultEvent notice);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A stream that never delivers anything.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// For a server that does not advertise the <c>events</c> feature, and for tests. Deliberately waits
|
||||
/// for ever rather than completing: a caller selecting between this and a timer must fall through to
|
||||
/// the timer, and a read that returned immediately would spin that loop as fast as the machine allows.
|
||||
/// </remarks>
|
||||
public sealed class IdleVaultEventStream : IVaultEventStream
|
||||
{
|
||||
/// <summary>The one instance. It holds nothing.</summary>
|
||||
public static IdleVaultEventStream Instance { get; } = new();
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool IsConnected => false;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask<VaultEvent> ReadAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
await Task.Delay(System.Threading.Timeout.Infinite, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// Unreachable: the delay above only ever ends by throwing.
|
||||
return new VaultEvent(VaultEventKinds.Ping);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool TryRead(out VaultEvent notice)
|
||||
{
|
||||
notice = new VaultEvent(VaultEventKinds.Ping);
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose()
|
||||
{
|
||||
// Nothing is held.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Tuning for <see cref="VaultEventStream"/>.</summary>
|
||||
/// <remarks>
|
||||
/// Every value here bounds a reconnection rather than a feature. With the socket permanently
|
||||
/// unavailable the client synchronises on its timer, so the cost of getting these wrong is latency,
|
||||
/// never correctness.
|
||||
/// </remarks>
|
||||
public sealed record VaultEventStreamOptions
|
||||
{
|
||||
/// <summary>The defaults.</summary>
|
||||
public static VaultEventStreamOptions Default { get; } = new();
|
||||
|
||||
/// <summary>How long to wait before the first reconnection attempt.</summary>
|
||||
public TimeSpan InitialBackoff { get; init; } = TimeSpan.FromSeconds(1);
|
||||
|
||||
/// <summary>The longest the backoff may grow to.</summary>
|
||||
/// <remarks>
|
||||
/// A minute, which is the polling interval: past that point reconnecting sooner buys nothing,
|
||||
/// because the timer has already done the work the socket would have prompted.
|
||||
/// </remarks>
|
||||
public TimeSpan MaxBackoff { get; init; } = TimeSpan.FromMinutes(1);
|
||||
|
||||
/// <summary>
|
||||
/// How long a socket may be silent before it is presumed dead.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The server pings on an interval it states in its <c>hello</c>, so silence past a multiple of
|
||||
/// that means the connection is gone rather than idle — which is otherwise indistinguishable, and
|
||||
/// is exactly what a reverse proxy that quietly drops idle sockets produces. Used only until a
|
||||
/// <c>hello</c> arrives; after that the server's own figure is trusted.
|
||||
/// </remarks>
|
||||
public TimeSpan InitialSilenceTimeout { get; init; } = TimeSpan.FromSeconds(90);
|
||||
|
||||
/// <summary>How many notices may be waiting before the oldest are dropped.</summary>
|
||||
/// <remarks>
|
||||
/// Small on purpose. A notice means "pull that vault", so a newer one subsumes the one it
|
||||
/// displaces; a backlog would only make the loop pull repeatedly for work it has already done.
|
||||
/// </remarks>
|
||||
public int QueueDepth { get; init; } = 32;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Holds a socket to one server open, and hands over what it says.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The whole class is a reconnection policy. A dropped socket is the ordinary case — laptops sleep,
|
||||
/// proxies time out, tokens expire, servers are redeployed — so nothing here treats a failure as
|
||||
/// exceptional: it backs off and dials again, for as long as somebody is reading.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// It is safe to have no server at all. Every failure path ends in "wait, then try again", and the
|
||||
/// caller's synchronisation timer runs regardless, which is what makes it correct for this class to
|
||||
/// stay silent about problems rather than surface them.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class VaultEventStream : IVaultEventStream, IAsyncDisposable
|
||||
{
|
||||
private readonly Uri endpoint;
|
||||
private readonly IAccessTokenProvider tokens;
|
||||
private readonly TimeProvider clock;
|
||||
private readonly VaultEventStreamOptions options;
|
||||
private readonly Func<Uri, string, CancellationToken, Task<WebSocket>> connect;
|
||||
private readonly Channel<VaultEvent> notices;
|
||||
private readonly CancellationTokenSource closing = new();
|
||||
private readonly Lock starting = new();
|
||||
|
||||
private Task? pump;
|
||||
private bool disposed;
|
||||
|
||||
/// <summary>Creates a stream against one server.</summary>
|
||||
/// <param name="serverUrl">The server's base URL, as an ordinary <c>http</c> or <c>https</c> address.</param>
|
||||
/// <param name="tokens">Supplies a bearer token, refreshing it when it is due.</param>
|
||||
/// <param name="clock">Time source, for the backoff and the silence timeout.</param>
|
||||
/// <param name="options">Tuning, or null for the defaults.</param>
|
||||
public VaultEventStream(
|
||||
Uri serverUrl,
|
||||
IAccessTokenProvider tokens,
|
||||
TimeProvider clock,
|
||||
VaultEventStreamOptions? options = null)
|
||||
: this(serverUrl, tokens, clock, DialAsync, options)
|
||||
{
|
||||
}
|
||||
|
||||
/// <remarks>
|
||||
/// The connector is injected so the suite can drive this against a test host's in-memory socket.
|
||||
/// Reconnection is the entire behaviour of this class, and testing it against a real network would
|
||||
/// mean testing it against the one thing that cannot be made to fail on demand.
|
||||
/// </remarks>
|
||||
internal VaultEventStream(
|
||||
Uri serverUrl,
|
||||
IAccessTokenProvider tokens,
|
||||
TimeProvider clock,
|
||||
Func<Uri, string, CancellationToken, Task<WebSocket>> connect,
|
||||
VaultEventStreamOptions? options = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(serverUrl);
|
||||
ArgumentNullException.ThrowIfNull(tokens);
|
||||
ArgumentNullException.ThrowIfNull(clock);
|
||||
ArgumentNullException.ThrowIfNull(connect);
|
||||
|
||||
endpoint = EventsUrl(serverUrl);
|
||||
this.tokens = tokens;
|
||||
this.clock = clock;
|
||||
this.connect = connect;
|
||||
this.options = options ?? VaultEventStreamOptions.Default;
|
||||
|
||||
notices = Channel.CreateBounded<VaultEvent>(new BoundedChannelOptions(this.options.QueueDepth)
|
||||
{
|
||||
FullMode = BoundedChannelFullMode.DropOldest,
|
||||
SingleReader = true,
|
||||
SingleWriter = true,
|
||||
});
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool IsConnected { get; private set; }
|
||||
|
||||
/// <inheritdoc />
|
||||
public ValueTask<VaultEvent> ReadAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(disposed, this);
|
||||
|
||||
Start();
|
||||
|
||||
return notices.Reader.ReadAsync(cancellationToken);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// Does not start the connection, unlike <see cref="ReadAsync"/>: a caller draining a burst has
|
||||
/// already read one notice, and "is there another right now" is not a reason to dial a server.
|
||||
/// </remarks>
|
||||
public bool TryRead(out VaultEvent notice) => notices.Reader.TryRead(out notice!);
|
||||
|
||||
/// <summary>
|
||||
/// Ends the connection, without waiting for it to unwind.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// What a shell calls when a connection is dropped, from a synchronous path that must not block —
|
||||
/// <c>IVaultServer</c> is <see cref="IDisposable"/>, and blocking on a socket teardown from the
|
||||
/// user-interface thread is exactly the sync-over-async this repository bans. Cancelling is enough:
|
||||
/// every loop reads the token, and the pump has nothing to flush.
|
||||
/// </remarks>
|
||||
public void Dispose()
|
||||
{
|
||||
if (disposed)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
disposed = true;
|
||||
closing.Cancel();
|
||||
|
||||
// The source is deliberately left undisposed. The pump may still be inside a linked token
|
||||
// source derived from this one, and disposing a parent out from under a live child is how a
|
||||
// clean shutdown becomes an ObjectDisposedException on a background thread. It holds no timer
|
||||
// and no handle once cancelled; DisposeAsync is the path that cleans it up properly.
|
||||
}
|
||||
|
||||
/// <summary>Ends the connection and waits for it to unwind.</summary>
|
||||
/// <remarks>The deterministic form, for a caller that can await one — tests, mostly.</remarks>
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (disposed)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
disposed = true;
|
||||
|
||||
await closing.CancelAsync().ConfigureAwait(false);
|
||||
|
||||
if (pump is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
await pump.ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// The point of the cancel above.
|
||||
}
|
||||
}
|
||||
|
||||
closing.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Turns a server's base URL into its event socket's.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The path is replaced rather than appended, matching every other call in this client: request
|
||||
/// paths here are absolute — <c>/api/v1/…</c> — so a deployment behind a path prefix is already
|
||||
/// unsupported, and pretending otherwise in this one place would be a difference nobody could act
|
||||
/// on.
|
||||
/// </remarks>
|
||||
private static Uri EventsUrl(Uri serverUrl) =>
|
||||
new UriBuilder(serverUrl)
|
||||
{
|
||||
Scheme = string.Equals(serverUrl.Scheme, Uri.UriSchemeHttps, StringComparison.OrdinalIgnoreCase)
|
||||
? "wss"
|
||||
: "ws",
|
||||
Path = VaultEvents.Path,
|
||||
Query = string.Empty,
|
||||
Fragment = string.Empty,
|
||||
}.Uri;
|
||||
|
||||
private static async Task<WebSocket> DialAsync(Uri url, string token, CancellationToken cancellationToken)
|
||||
{
|
||||
var socket = new ClientWebSocket();
|
||||
|
||||
try
|
||||
{
|
||||
socket.Options.AddSubProtocol(VaultEvents.SubProtocol);
|
||||
|
||||
// A header rather than the Sec-WebSocket-Protocol smuggling ADR 0004 needs for the relay:
|
||||
// this client is a native application and can set one, and the token here is the ordinary
|
||||
// bearer credential rather than a ticket.
|
||||
socket.Options.SetRequestHeader("Authorization", $"Bearer {token}");
|
||||
|
||||
await socket.ConnectAsync(url, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return socket;
|
||||
}
|
||||
catch
|
||||
{
|
||||
socket.Dispose();
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
private void Start()
|
||||
{
|
||||
if (pump is not null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
lock (starting)
|
||||
{
|
||||
pump ??= Task.Run(() => RunAsync(closing.Token), closing.Token);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Connects, reads until it cannot, waits, and does it again.</summary>
|
||||
private async Task RunAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var backoff = options.InitialBackoff;
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var outcome = await AttemptAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// A socket that lived long enough to say hello proves the server is there and willing, so
|
||||
// the next failure starts from the bottom again rather than inheriting the backoff that
|
||||
// got us here. Without this a laptop that woke, connected, and then lost its network an
|
||||
// hour later would wait a full minute before trying, having already proved it need not.
|
||||
if (outcome == Outcome.Established)
|
||||
{
|
||||
backoff = options.InitialBackoff;
|
||||
}
|
||||
|
||||
// The server said this token is spent, which the token provider can fix without waiting.
|
||||
// Reconnecting at once is the whole reason that close code is distinct.
|
||||
var wait = outcome == Outcome.TokenExpired ? TimeSpan.Zero : Jitter(backoff);
|
||||
|
||||
if (wait > TimeSpan.Zero)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(wait, clock, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
backoff = backoff < options.MaxBackoff
|
||||
? Shorter(backoff * 2, options.MaxBackoff)
|
||||
: options.MaxBackoff;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>One connection, from dial to close.</summary>
|
||||
private async Task<Outcome> AttemptAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
WebSocket? socket = null;
|
||||
|
||||
try
|
||||
{
|
||||
var token = await tokens.GetAccessTokenAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
socket = await connect(endpoint, token, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
IsConnected = true;
|
||||
|
||||
return await PumpAsync(socket, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return Outcome.Cancelled;
|
||||
}
|
||||
catch (Exception exception) when (exception is not OutOfMemoryException)
|
||||
{
|
||||
// Every failure this can meet — no network, a refused upgrade, a server that has not been
|
||||
// deployed with this feature, a token that cannot be refreshed — has the same remedy, and
|
||||
// none of them is worth telling a user about. The synchronisation timer is still running.
|
||||
return Outcome.Failed;
|
||||
}
|
||||
finally
|
||||
{
|
||||
IsConnected = false;
|
||||
socket?.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Reads frames until the socket ends or goes quiet.</summary>
|
||||
private async Task<Outcome> PumpAsync(WebSocket socket, CancellationToken cancellationToken)
|
||||
{
|
||||
var buffer = new byte[8 * 1024];
|
||||
var silence = options.InitialSilenceTimeout;
|
||||
var established = false;
|
||||
|
||||
while (socket.State == WebSocketState.Open && !cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
// Rebuilt per frame rather than reset, because a linked source cannot be un-cancelled and
|
||||
// the deadline is what detects a socket that has silently gone away.
|
||||
using var deadline = new CancellationTokenSource(silence, clock);
|
||||
using var quiet = CancellationTokenSource.CreateLinkedTokenSource(
|
||||
cancellationToken, deadline.Token);
|
||||
|
||||
WebSocketReceiveResult received;
|
||||
|
||||
try
|
||||
{
|
||||
received = await socket.ReceiveAsync(buffer, quiet.Token).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
// Silent for longer than the server said it would be. The socket is gone in a way that
|
||||
// only reconnecting can discover, which is what a proxy dropping an idle connection
|
||||
// looks like from this end.
|
||||
return Ended(established);
|
||||
}
|
||||
|
||||
if (received.MessageType == WebSocketMessageType.Close)
|
||||
{
|
||||
return (int?)received.CloseStatus == VaultEvents.TokenExpiredCloseCode
|
||||
? Outcome.TokenExpired
|
||||
: Ended(established);
|
||||
}
|
||||
|
||||
// Binary is reserved by ADR 0012 for shared-session data, and text that arrived in pieces
|
||||
// is longer than anything this protocol defines. Skipped rather than fatal, so a newer
|
||||
// server does not cost this client its push for the whole session.
|
||||
if (received.MessageType != WebSocketMessageType.Text || !received.EndOfMessage)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (Parse(buffer.AsSpan(0, received.Count)) is not { } frame)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
established = true;
|
||||
silence = await AbsorbAsync(socket, frame, silence, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
return Ended(established);
|
||||
}
|
||||
|
||||
/// <summary>Deals with one frame, and says how long the socket may now stay quiet.</summary>
|
||||
private async Task<TimeSpan> AbsorbAsync(
|
||||
WebSocket socket,
|
||||
VaultEvent frame,
|
||||
TimeSpan silence,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (frame.HeartbeatSeconds is > 0 and var seconds)
|
||||
{
|
||||
// Three missed heartbeats. Two is within one paused thread of a false positive, and a
|
||||
// false positive here costs a reconnection rather than anything a user sees.
|
||||
silence = TimeSpan.FromSeconds(seconds * 3);
|
||||
}
|
||||
|
||||
if (string.Equals(frame.Kind, VaultEventKinds.Ping, StringComparison.Ordinal))
|
||||
{
|
||||
await socket.SendAsync(
|
||||
JsonSerializer.SerializeToUtf8Bytes(
|
||||
new VaultEvent(VaultEventKinds.Pong), DodoSshJsonContext.Default.VaultEvent),
|
||||
WebSocketMessageType.Text,
|
||||
endOfMessage: true,
|
||||
cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return silence;
|
||||
}
|
||||
|
||||
// Everything else, including a kind this build has never heard of, goes to the reader — which
|
||||
// is what makes the frame table extensible. An unrecognised kind is one the caller ignores;
|
||||
// refusing it here would be this class deciding what a newer server may say.
|
||||
notices.Writer.TryWrite(frame);
|
||||
|
||||
return silence;
|
||||
}
|
||||
|
||||
private static Outcome Ended(bool established) =>
|
||||
established ? Outcome.Established : Outcome.Failed;
|
||||
|
||||
private static VaultEvent? Parse(ReadOnlySpan<byte> utf8)
|
||||
{
|
||||
try
|
||||
{
|
||||
return JsonSerializer.Deserialize(utf8, DodoSshJsonContext.Default.VaultEvent);
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Spreads reconnections out, so a server that restarts is not met by every client at once.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <c>RandomNumberGenerator</c> because <c>System.Random</c> is banned repo-wide. Nothing here is
|
||||
/// security-relevant — the ban exists so that nothing key-, token- or nonce-adjacent can reach for
|
||||
/// the weak one by habit, and paying a few microseconds to keep that rule absolute is the cheaper
|
||||
/// side of the trade.
|
||||
/// </remarks>
|
||||
private static TimeSpan Jitter(TimeSpan delay)
|
||||
{
|
||||
var milliseconds = (int)Math.Clamp(delay.TotalMilliseconds, 1, int.MaxValue / 2);
|
||||
|
||||
return TimeSpan.FromMilliseconds(
|
||||
milliseconds + RandomNumberGenerator.GetInt32(0, Math.Max(1, milliseconds / 2)));
|
||||
}
|
||||
|
||||
private static TimeSpan Shorter(TimeSpan left, TimeSpan right) => left < right ? left : right;
|
||||
|
||||
/// <summary>How one connection attempt ended.</summary>
|
||||
private enum Outcome
|
||||
{
|
||||
/// <summary>Never got as far as a frame. Back off.</summary>
|
||||
Failed,
|
||||
|
||||
/// <summary>Ran, and then ended. Back off, but from the bottom.</summary>
|
||||
Established,
|
||||
|
||||
/// <summary>The server closed it because the token expired. Reconnect at once with a new one.</summary>
|
||||
TokenExpired,
|
||||
|
||||
/// <summary>The stream is being disposed.</summary>
|
||||
Cancelled,
|
||||
}
|
||||
}
|
||||
@@ -150,6 +150,17 @@ public interface IVaultServer : IDisposable
|
||||
/// <summary>Vault key grants: who can open a vault, and who let them.</summary>
|
||||
IVaultGrantApi Grants { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Notices that something changed, so a synchronisation need not wait for the timer.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Always present, never null: a server that does not offer the feature — or a test standing in
|
||||
/// for one — supplies <see cref="IdleVaultEventStream"/>, which simply never delivers. That keeps
|
||||
/// every caller on one shape, because the correct behaviour without a socket is the behaviour
|
||||
/// with a silent one: synchronise on the timer. See ADR 0012.
|
||||
/// </remarks>
|
||||
IVaultEventStream Events { get; }
|
||||
|
||||
/// <summary>Obtains the identity provider's signature over a key statement.</summary>
|
||||
IKeyBindingAuthorizer KeyBinding { get; }
|
||||
|
||||
@@ -197,7 +208,8 @@ public sealed class ServerConnection : IVaultServer
|
||||
MetaResponse meta,
|
||||
OidcClient oidc,
|
||||
RefreshingAccessTokenProvider tokens,
|
||||
DodoSshApiClient api)
|
||||
DodoSshApiClient api,
|
||||
TimeProvider clock)
|
||||
{
|
||||
ServerUrl = serverUrl;
|
||||
this.http = http;
|
||||
@@ -206,6 +218,14 @@ public sealed class ServerConnection : IVaultServer
|
||||
Oidc = oidc;
|
||||
this.tokens = tokens;
|
||||
Api = api;
|
||||
|
||||
// Decided from what this server said it supports rather than attempted and allowed to fail,
|
||||
// which is the same capability negotiation SyncOptions below does — see ADR 0002. A client
|
||||
// that dialled anyway would reconnect against a 404 for the whole session, and would look
|
||||
// from the outside exactly like one whose network was eating WebSockets.
|
||||
Events = meta.Features.Contains(VaultEvents.Feature, StringComparer.Ordinal)
|
||||
? new VaultEventStream(serverUrl, tokens, clock)
|
||||
: IdleVaultEventStream.Instance;
|
||||
}
|
||||
|
||||
/// <summary>The server this is connected to.</summary>
|
||||
@@ -238,6 +258,9 @@ public sealed class ServerConnection : IVaultServer
|
||||
/// <inheritdoc />
|
||||
public IVaultGrantApi Grants => Api;
|
||||
|
||||
/// <inheritdoc />
|
||||
public IVaultEventStream Events { get; }
|
||||
|
||||
/// <inheritdoc />
|
||||
public IKeyBindingAuthorizer KeyBinding => Oidc;
|
||||
|
||||
@@ -311,7 +334,8 @@ public sealed class ServerConnection : IVaultServer
|
||||
meta,
|
||||
oidc,
|
||||
refreshing,
|
||||
new DodoSshApiClient(transport, refreshing));
|
||||
new DodoSshApiClient(transport, refreshing),
|
||||
clock);
|
||||
}
|
||||
catch
|
||||
{
|
||||
@@ -382,7 +406,8 @@ public sealed class ServerConnection : IVaultServer
|
||||
meta,
|
||||
oidc,
|
||||
refreshing,
|
||||
new DodoSshApiClient(transport, refreshing));
|
||||
new DodoSshApiClient(transport, refreshing),
|
||||
clock);
|
||||
}
|
||||
catch
|
||||
{
|
||||
@@ -400,6 +425,12 @@ public sealed class ServerConnection : IVaultServer
|
||||
}
|
||||
|
||||
disposed = true;
|
||||
|
||||
// First, and without waiting: the socket's own loops read the token provider and the transport
|
||||
// below, so tearing either down while it is still dialling would surface as a fault on a
|
||||
// background thread at the moment a user signed out.
|
||||
Events.Dispose();
|
||||
|
||||
tokens.Dispose();
|
||||
http.Dispose();
|
||||
}
|
||||
|
||||
@@ -10,6 +10,7 @@ using DodoSSH.Client.Session;
|
||||
using DodoSSH.Client.Ssh;
|
||||
using DodoSSH.Client.Sync;
|
||||
using DodoSSH.Client.Terminal;
|
||||
using DodoSSH.Contracts;
|
||||
|
||||
namespace DodoSSH.Client.Shell.ViewModels;
|
||||
|
||||
@@ -1066,13 +1067,29 @@ internal sealed partial class VaultViewModel(
|
||||
VaultVisibility? visibility = null) : ObservableObject, IAsyncDisposable
|
||||
{
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// A minute. The pull is a delta keyed on a cursor, so an idle pass is one small request and costs the
|
||||
/// server almost nothing; the number that matters is how stale a teammate's change may look, and a
|
||||
/// minute is short enough not to be noticed. Anything much shorter would be polling for its own sake,
|
||||
/// and a change made on this machine does not wait for the timer anyway — saving pushes immediately.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Unchanged by the push channel, and deliberately so. The socket makes a pass <em>early</em>; this is
|
||||
/// what makes one happen at all, for a client whose network eats WebSockets, whose server has the
|
||||
/// feature off, or whose notice was dropped. See <see cref="WaitForWorkAsync"/> and ADR 0012.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private static readonly TimeSpan AutoSyncInterval = TimeSpan.FromMinutes(1);
|
||||
|
||||
/// <summary>How long a pushed notice waits, in case more are on their way.</summary>
|
||||
/// <remarks>
|
||||
/// A quarter of a second, which is below what anybody perceives and above the gap between the
|
||||
/// notices one person's save produces — a host and its activity log entry are two items in one
|
||||
/// push, and a colleague clearing a folder is a burst. Without it each notice would run its own
|
||||
/// full pass, and the pass a burst deserves is one.
|
||||
/// </remarks>
|
||||
private static readonly TimeSpan NoticeDebounce = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
/// <summary>How often the logs are pruned, at most.</summary>
|
||||
/// <remarks>
|
||||
/// Hours rather than minutes, because pruning writes tombstones that sync. Retention is measured in days
|
||||
@@ -4289,10 +4306,16 @@ internal sealed partial class VaultViewModel(
|
||||
/// <para>
|
||||
/// <b>This is the whole of how a shared vault arrives.</b> Sharing is two acts on two machines: the
|
||||
/// person sharing wraps the vault key to the recipient, and the recipient's own client has to notice.
|
||||
/// The recipient is handed nothing — there is no push channel — so without this the vault list stayed
|
||||
/// exactly as it was cached at sign-in, and a vault shared with somebody appeared on their machine only
|
||||
/// if they happened to sign in through the browser again. Everything else was already right, which is
|
||||
/// why it looked like sharing was broken rather than like a list that was never re-read.
|
||||
/// Without this the vault list stayed exactly as it was cached at sign-in, and a vault shared with
|
||||
/// somebody appeared on their machine only if they happened to sign in through the browser again.
|
||||
/// Everything else was already right, which is why it looked like sharing was broken rather than like a
|
||||
/// list that was never re-read.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The server now says when this is worth doing — a <c>vaults.changed</c> notice wakes the pass, so the
|
||||
/// vault turns up as it is shared rather than within the minute — but that only decides <em>when</em>.
|
||||
/// This call is still what discovers the vault, on the notice and on every timed pass alike, because a
|
||||
/// client with no socket has to arrive at the same place. See ADR 0012.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A failure is left to the caller, which treats it as the pass failing: the call is to the same server
|
||||
@@ -4339,6 +4362,7 @@ internal sealed partial class VaultViewModel(
|
||||
private async Task RunAutoSyncLoopAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
using var timer = new PeriodicTimer(AutoSyncInterval);
|
||||
var waits = new AutoSyncWaits();
|
||||
|
||||
try
|
||||
{
|
||||
@@ -4353,7 +4377,7 @@ internal sealed partial class VaultViewModel(
|
||||
// user is doing something.
|
||||
await SyncOnOpenAsync(cancellationToken).ConfigureAwait(true);
|
||||
|
||||
while (await timer.WaitForNextTickAsync(cancellationToken).ConfigureAwait(true))
|
||||
while (await WaitForWorkAsync(timer, waits, cancellationToken).ConfigureAwait(true))
|
||||
{
|
||||
await AutoSyncAsync(cancellationToken).ConfigureAwait(true);
|
||||
}
|
||||
@@ -4364,6 +4388,98 @@ internal sealed partial class VaultViewModel(
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Waits for the timer to come round, or for the server to say there is something to fetch.
|
||||
/// </summary>
|
||||
/// <returns>Whether to run a pass. False means the loop is over.</returns>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The timer is unchanged and is still what guarantees a pass. The socket only makes one
|
||||
/// <em>early</em>, which is why nothing here treats its absence as a problem: no connection, a
|
||||
/// server without the feature, a network that eats WebSockets, or a notice dropped under
|
||||
/// backpressure all leave a loop that behaves exactly as it did before this existed. See ADR 0012.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Both waits are held across iterations, and that is load-bearing rather than an
|
||||
/// optimisation.</b> <see cref="PeriodicTimer"/> permits only one outstanding
|
||||
/// <c>WaitForNextTickAsync</c> and throws on a second, and an abandoned channel read stays
|
||||
/// registered and consumes the next notice written — which would silently lose exactly the wake-up
|
||||
/// this is for. Whichever wait did not win is kept and awaited again.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private async Task<bool> WaitForWorkAsync(
|
||||
PeriodicTimer timer,
|
||||
AutoSyncWaits waits,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
// Re-read every time, because signing out and back in replaces the connection — and with it
|
||||
// the stream. A read still pending against the old one is left to be cancelled with it.
|
||||
var stream = connection()?.Events;
|
||||
|
||||
if (!ReferenceEquals(stream, waits.Watching))
|
||||
{
|
||||
waits.Watching = stream;
|
||||
waits.Notice = null;
|
||||
}
|
||||
|
||||
waits.Tick ??= timer.WaitForNextTickAsync(cancellationToken).AsTask();
|
||||
waits.Notice ??= stream?.ReadAsync(cancellationToken).AsTask();
|
||||
|
||||
if (waits.Notice is null)
|
||||
{
|
||||
var only = waits.Tick;
|
||||
waits.Tick = null;
|
||||
|
||||
return await only.ConfigureAwait(true);
|
||||
}
|
||||
|
||||
var first = await Task.WhenAny(waits.Tick, waits.Notice).ConfigureAwait(true);
|
||||
|
||||
if (ReferenceEquals(first, waits.Tick))
|
||||
{
|
||||
var ticked = waits.Tick;
|
||||
waits.Tick = null;
|
||||
|
||||
return await ticked.ConfigureAwait(true);
|
||||
}
|
||||
|
||||
// Observed so a faulted read does not go unhandled, and so a stream that has been disposed
|
||||
// ends this wait rather than being asked again.
|
||||
await waits.Notice.ConfigureAwait(true);
|
||||
waits.Notice = null;
|
||||
|
||||
// A burst — one person's save is two items, and a colleague tidying a folder is a dozen —
|
||||
// deserves one pass rather than one each.
|
||||
await Task.Delay(NoticeDebounce, cancellationToken).ConfigureAwait(true);
|
||||
|
||||
while (stream!.TryRead(out _))
|
||||
{
|
||||
// Swallowed on purpose. Every notice means the same thing, which is what the pass about to
|
||||
// run already does; what they say about *which* vault is not read, because a pass syncs
|
||||
// every vault this session can reach anyway.
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>The two waits the background loop keeps alive between passes.</summary>
|
||||
/// <remarks>
|
||||
/// A class rather than three locals because <see cref="WaitForWorkAsync"/> has to hand them back
|
||||
/// changed, and a method that took three <c>ref</c> parameters could not be <c>async</c>. See that
|
||||
/// method for why abandoning either of them is a defect rather than a tidiness question.
|
||||
/// </remarks>
|
||||
private sealed class AutoSyncWaits
|
||||
{
|
||||
/// <summary>The pending timer tick, or null when the last one has been consumed.</summary>
|
||||
internal Task<bool>? Tick { get; set; }
|
||||
|
||||
/// <summary>The pending read from the server's push channel.</summary>
|
||||
internal Task<VaultEvent>? Notice { get; set; }
|
||||
|
||||
/// <summary>The stream <see cref="Notice"/> was taken from, to notice a reconnection.</summary>
|
||||
internal IVaultEventStream? Watching { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>Shows one kind of item, if nothing is being edited.</summary>
|
||||
/// <remarks>
|
||||
/// Takes the section rather than there being one command per kind, so a third kind is an enum member and
|
||||
|
||||
@@ -66,6 +66,11 @@ namespace DodoSSH.Contracts;
|
||||
// Registered in its own right, not only as a member of the sync DTOs: the client's local
|
||||
// cache seals this record under the LocalCacheKey and needs its type info directly.
|
||||
[JsonSerializable(typeof(SyncPlaintextFields))]
|
||||
|
||||
// The event socket's only frame type. Registered although nothing else references it: frames are
|
||||
// written straight onto a WebSocket rather than through a response body, so the resolver never
|
||||
// infers it from an endpoint's signature the way it does for every DTO above.
|
||||
[JsonSerializable(typeof(VaultEvent))]
|
||||
[JsonSerializable(typeof(RelayTicketRequest))]
|
||||
[JsonSerializable(typeof(RelayTicketResponse))]
|
||||
[JsonSerializable(typeof(RelaySessionSummary))]
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
namespace DodoSSH.Contracts;
|
||||
|
||||
/// <summary>
|
||||
/// The event socket's protocol constants.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The version lives in the subprotocol name rather than in the URL, for the reason ADR 0002 gives
|
||||
/// about the rest of this API: a client and a server that upgrade independently have to agree by
|
||||
/// negotiating rather than by assuming, and a WebSocket handshake already has a field for exactly
|
||||
/// that. A server that does not offer <see cref="SubProtocol"/> fails the handshake, which a client
|
||||
/// can act on — rather than opening a socket that then speaks a dialect it cannot read.
|
||||
/// </remarks>
|
||||
public static class VaultEvents
|
||||
{
|
||||
/// <summary>The path the event socket is served from.</summary>
|
||||
public const string Path = "/api/v1/events";
|
||||
|
||||
/// <summary>The only subprotocol this version speaks.</summary>
|
||||
public const string SubProtocol = "dodossh.events.v1";
|
||||
|
||||
/// <summary>The <c>/api/v1/meta</c> feature flag advertising that this server pushes at all.</summary>
|
||||
public const string Feature = "events";
|
||||
|
||||
/// <summary>
|
||||
/// Close code for a socket whose access token has expired.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// In the 4000–4999 range, which the WebSocket specification reserves for applications. Its own
|
||||
/// code because it is the one close a client should answer by reconnecting immediately with a
|
||||
/// fresh token, rather than by backing off as it would for a server that went away.
|
||||
/// </remarks>
|
||||
public const int TokenExpiredCloseCode = 4401;
|
||||
|
||||
/// <summary>
|
||||
/// Close code for a caller already holding as many sockets as it may.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Distinguished from <see cref="TokenExpiredCloseCode"/> because the remedy is the opposite:
|
||||
/// reconnecting at once is what caused it. A client that meets this backs off and keeps polling.
|
||||
/// </remarks>
|
||||
public const int TooManyConnectionsCloseCode = 4429;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The kinds of event this socket carries.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Constants rather than an <c>enum</c>, and that is a compatibility decision rather than a style
|
||||
/// one. <c>DodoSshJsonContext</c> sets <c>UseStringEnumConverter</c>, which <em>throws</em> on a
|
||||
/// value it does not know — so a newer server sending a kind an older client has never heard of
|
||||
/// would not merely add an unreadable frame, it would break that client's socket. A string is
|
||||
/// ignored instead, which is what makes this list extensible. <see cref="ProblemCodes"/> is the same
|
||||
/// shape for the same reason.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Anything a client cannot parse must be skipped, not treated as an error.</b> That rule is what
|
||||
/// the shared-session frames of ADR 0012 will rely on when they arrive.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class VaultEventKinds
|
||||
{
|
||||
/// <summary>The server accepted the socket. Always the first frame.</summary>
|
||||
public const string Hello = "hello";
|
||||
|
||||
/// <summary>
|
||||
/// A vault has changes at or before <see cref="VaultEvent.Sequence"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Carries no ciphertext and no item identity — the client's answer is the delta pull it would
|
||||
/// have run on its timer anyway. See ADR 0012 for why pushing the items themselves is refused.
|
||||
/// </remarks>
|
||||
public const string VaultChanged = "vault.changed";
|
||||
|
||||
/// <summary>
|
||||
/// The set of vaults this account can reach is no longer what it was.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A vault shared with the caller, or a grant withdrawn. Deliberately says nothing about
|
||||
/// <em>which</em>: the client re-reads the list, which is the same call it already makes at the
|
||||
/// start of every synchronisation pass.
|
||||
/// </remarks>
|
||||
public const string VaultsChanged = "vaults.changed";
|
||||
|
||||
/// <summary>Heartbeat. Whichever side receives one answers <see cref="Pong"/>.</summary>
|
||||
public const string Ping = "ping";
|
||||
|
||||
/// <summary>The answer to a <see cref="Ping"/>.</summary>
|
||||
public const string Pong = "pong";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One frame on the event socket.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// One flat record for every kind, with the fields a given kind does not use left null, rather than
|
||||
/// a polymorphic hierarchy. The set is small, the frames are tiny, and <c>System.Text.Json</c>
|
||||
/// polymorphism would put a second discriminator mechanism next to the <see cref="Kind"/> string
|
||||
/// that is already the discriminator. Nothing else in <c>DodoSSH.Contracts</c> is polymorphic.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Nothing here is secret, by construction.</b> The server cannot read a vault's contents, so a
|
||||
/// notice cannot describe them; what it does disclose — that a vault changed, and when — is the same
|
||||
/// metadata ADR 0001 already accepts the server holding.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="Kind">One of <see cref="VaultEventKinds"/>. An unrecognised kind must be ignored.</param>
|
||||
/// <param name="VaultId">The vault a <see cref="VaultEventKinds.VaultChanged"/> is about.</param>
|
||||
/// <param name="Sequence">
|
||||
/// The change-log position that vault has reached. A hint for logging and for coalescing, not a
|
||||
/// cursor: cursors are opaque and HMAC-tagged, and this is neither.
|
||||
/// </param>
|
||||
/// <param name="ServerTime">
|
||||
/// The server's clock when the frame was written. The client already measures skew against
|
||||
/// <c>SyncPullResponse.ServerTime</c>; this lets a socket that is quiet for other reasons keep that
|
||||
/// measurement current.
|
||||
/// </param>
|
||||
/// <param name="HeartbeatSeconds">
|
||||
/// How often the server will ping, sent with <see cref="VaultEventKinds.Hello"/>. The client uses it
|
||||
/// to decide when silence means the connection is dead rather than idle.
|
||||
/// </param>
|
||||
/// <param name="VaultCount">
|
||||
/// How many vaults this socket is subscribed to, sent with <see cref="VaultEventKinds.Hello"/>.
|
||||
/// Diagnostic: a socket subscribed to nothing is a real state — an account with no vaults yet — and
|
||||
/// is otherwise indistinguishable from one that is quietly broken.
|
||||
/// </param>
|
||||
public sealed record VaultEvent(
|
||||
string Kind,
|
||||
Guid? VaultId = null,
|
||||
long? Sequence = null,
|
||||
DateTimeOffset? ServerTime = null,
|
||||
int? HeartbeatSeconds = null,
|
||||
int? VaultCount = null);
|
||||
@@ -66,6 +66,16 @@ public static class ProblemCodes
|
||||
/// </remarks>
|
||||
public const string MalformedRequest = "malformed-request";
|
||||
|
||||
/// <summary>
|
||||
/// This deployment does not push vault changes over a socket.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Not a failure to recover from: synchronising on a timer is the supported behaviour and the
|
||||
/// socket only ever made it early. A client that meets this stops dialling and keeps polling. See
|
||||
/// ADR 0012.
|
||||
/// </remarks>
|
||||
public const string EventsUnavailable = "events-unavailable";
|
||||
|
||||
/// <summary>The relay refused the requested target. Never states why, to avoid a probe oracle.</summary>
|
||||
public const string RelayTargetRejected = "relay-target-rejected";
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
const DodoSSH.Contracts.ProblemCodes.AlreadyEnrolled = "already-enrolled" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.ClientTooOld = "client-too-old" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.EnrollmentRequired = "enrollment-required" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.EventsUnavailable = "events-unavailable" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.Forbidden = "forbidden" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.IdempotencyKeyReuse = "idempotency-key-reuse" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.IdentityBindingInvalid = "identity-binding-invalid" -> string!
|
||||
@@ -22,6 +23,16 @@ const DodoSSH.Contracts.ProblemCodes.TeamNotEmpty = "team-not-empty" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.TeamSlugTaken = "team-slug-taken" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.TypeBaseUri = "https://dodossh.dev/problems/" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.VaultConflict = "vault-conflict" -> string!
|
||||
const DodoSSH.Contracts.VaultEventKinds.Hello = "hello" -> string!
|
||||
const DodoSSH.Contracts.VaultEventKinds.Ping = "ping" -> string!
|
||||
const DodoSSH.Contracts.VaultEventKinds.Pong = "pong" -> string!
|
||||
const DodoSSH.Contracts.VaultEventKinds.VaultChanged = "vault.changed" -> string!
|
||||
const DodoSSH.Contracts.VaultEventKinds.VaultsChanged = "vaults.changed" -> string!
|
||||
const DodoSSH.Contracts.VaultEvents.Feature = "events" -> string!
|
||||
const DodoSSH.Contracts.VaultEvents.Path = "/api/v1/events" -> string!
|
||||
const DodoSSH.Contracts.VaultEvents.SubProtocol = "dodossh.events.v1" -> string!
|
||||
const DodoSSH.Contracts.VaultEvents.TokenExpiredCloseCode = 4401 -> int
|
||||
const DodoSSH.Contracts.VaultEvents.TooManyConnectionsCloseCode = 4429 -> int
|
||||
DodoSSH.Contracts.AddTeamMemberRequest
|
||||
DodoSSH.Contracts.AddTeamMemberRequest.<Clone>$() -> DodoSSH.Contracts.AddTeamMemberRequest!
|
||||
DodoSSH.Contracts.AddTeamMemberRequest.AddTeamMemberRequest(System.Guid UserId, DodoSSH.Contracts.TeamMemberRole Role, string? Email = null) -> void
|
||||
@@ -686,6 +697,25 @@ DodoSSH.Contracts.UpdateVaultRequest.Equals(DodoSSH.Contracts.UpdateVaultRequest
|
||||
DodoSSH.Contracts.UpdateVaultRequest.Name.get -> string!
|
||||
DodoSSH.Contracts.UpdateVaultRequest.Name.init -> void
|
||||
DodoSSH.Contracts.UpdateVaultRequest.UpdateVaultRequest(string! Name) -> void
|
||||
DodoSSH.Contracts.VaultEvent
|
||||
DodoSSH.Contracts.VaultEvent.<Clone>$() -> DodoSSH.Contracts.VaultEvent!
|
||||
DodoSSH.Contracts.VaultEvent.Deconstruct(out string! Kind, out System.Guid? VaultId, out long? Sequence, out System.DateTimeOffset? ServerTime, out int? HeartbeatSeconds, out int? VaultCount) -> void
|
||||
DodoSSH.Contracts.VaultEvent.Equals(DodoSSH.Contracts.VaultEvent? other) -> bool
|
||||
DodoSSH.Contracts.VaultEvent.HeartbeatSeconds.get -> int?
|
||||
DodoSSH.Contracts.VaultEvent.HeartbeatSeconds.init -> void
|
||||
DodoSSH.Contracts.VaultEvent.Kind.get -> string!
|
||||
DodoSSH.Contracts.VaultEvent.Kind.init -> void
|
||||
DodoSSH.Contracts.VaultEvent.Sequence.get -> long?
|
||||
DodoSSH.Contracts.VaultEvent.Sequence.init -> void
|
||||
DodoSSH.Contracts.VaultEvent.ServerTime.get -> System.DateTimeOffset?
|
||||
DodoSSH.Contracts.VaultEvent.ServerTime.init -> void
|
||||
DodoSSH.Contracts.VaultEvent.VaultCount.get -> int?
|
||||
DodoSSH.Contracts.VaultEvent.VaultCount.init -> void
|
||||
DodoSSH.Contracts.VaultEvent.VaultEvent(string! Kind, System.Guid? VaultId = null, long? Sequence = null, System.DateTimeOffset? ServerTime = null, int? HeartbeatSeconds = null, int? VaultCount = null) -> void
|
||||
DodoSSH.Contracts.VaultEvent.VaultId.get -> System.Guid?
|
||||
DodoSSH.Contracts.VaultEvent.VaultId.init -> void
|
||||
DodoSSH.Contracts.VaultEventKinds
|
||||
DodoSSH.Contracts.VaultEvents
|
||||
DodoSSH.Contracts.VaultGrantsResponse
|
||||
DodoSSH.Contracts.VaultGrantsResponse.<Clone>$() -> DodoSSH.Contracts.VaultGrantsResponse!
|
||||
DodoSSH.Contracts.VaultGrantsResponse.Deconstruct(out System.Guid VaultId, out uint KeyGeneration, out bool RekeyRequired, out System.Collections.Generic.IReadOnlyList<DodoSSH.Contracts.VaultGrantSummary!>! Grants) -> void
|
||||
@@ -877,6 +907,9 @@ override DodoSSH.Contracts.UpdateTeamRequest.ToString() -> string!
|
||||
override DodoSSH.Contracts.UpdateVaultRequest.Equals(object? obj) -> bool
|
||||
override DodoSSH.Contracts.UpdateVaultRequest.GetHashCode() -> int
|
||||
override DodoSSH.Contracts.UpdateVaultRequest.ToString() -> string!
|
||||
override DodoSSH.Contracts.VaultEvent.Equals(object? obj) -> bool
|
||||
override DodoSSH.Contracts.VaultEvent.GetHashCode() -> int
|
||||
override DodoSSH.Contracts.VaultEvent.ToString() -> string!
|
||||
override DodoSSH.Contracts.VaultGrantsResponse.Equals(object? obj) -> bool
|
||||
override DodoSSH.Contracts.VaultGrantsResponse.GetHashCode() -> int
|
||||
override DodoSSH.Contracts.VaultGrantsResponse.ToString() -> string!
|
||||
@@ -972,6 +1005,8 @@ static DodoSSH.Contracts.UpdateTeamRequest.operator !=(DodoSSH.Contracts.UpdateT
|
||||
static DodoSSH.Contracts.UpdateTeamRequest.operator ==(DodoSSH.Contracts.UpdateTeamRequest? left, DodoSSH.Contracts.UpdateTeamRequest? right) -> bool
|
||||
static DodoSSH.Contracts.UpdateVaultRequest.operator !=(DodoSSH.Contracts.UpdateVaultRequest? left, DodoSSH.Contracts.UpdateVaultRequest? right) -> bool
|
||||
static DodoSSH.Contracts.UpdateVaultRequest.operator ==(DodoSSH.Contracts.UpdateVaultRequest? left, DodoSSH.Contracts.UpdateVaultRequest? right) -> bool
|
||||
static DodoSSH.Contracts.VaultEvent.operator !=(DodoSSH.Contracts.VaultEvent? left, DodoSSH.Contracts.VaultEvent? right) -> bool
|
||||
static DodoSSH.Contracts.VaultEvent.operator ==(DodoSSH.Contracts.VaultEvent? left, DodoSSH.Contracts.VaultEvent? right) -> bool
|
||||
static DodoSSH.Contracts.VaultGrantsResponse.operator !=(DodoSSH.Contracts.VaultGrantsResponse? left, DodoSSH.Contracts.VaultGrantsResponse? right) -> bool
|
||||
static DodoSSH.Contracts.VaultGrantsResponse.operator ==(DodoSSH.Contracts.VaultGrantsResponse? left, DodoSSH.Contracts.VaultGrantsResponse? right) -> bool
|
||||
static DodoSSH.Contracts.VaultGrantSummary.operator !=(DodoSSH.Contracts.VaultGrantSummary? left, DodoSSH.Contracts.VaultGrantSummary? right) -> bool
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
using System.Net.Http.Headers;
|
||||
using System.Net.WebSockets;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Infrastructure;
|
||||
using Microsoft.AspNetCore.Hosting;
|
||||
using Microsoft.AspNetCore.Mvc.Testing;
|
||||
@@ -107,6 +109,34 @@ public sealed class ApiFixture : WebApplicationFactory<Program>, IAsyncLifetime
|
||||
|
||||
/// <summary>Opens a database scope for arranging state and asserting on it.</summary>
|
||||
public AsyncServiceScope CreateScope() => Services.CreateAsyncScope();
|
||||
|
||||
/// <summary>
|
||||
/// Opens the event socket as the given subject, through the real pipeline.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The bearer token goes on the upgrade request, which is the whole of the socket's authorization
|
||||
/// — see ADR 0012 — so a test that stubbed it would be testing nothing. The subprotocol is offered
|
||||
/// because the server refuses an upgrade that does not, and that refusal is itself under test.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <c>TestServer</c> speaks WebSockets in-memory with no port and no network, so these run
|
||||
/// wherever the rest of the suite does.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public Task<WebSocket> ConnectEventsAsync(string subject, CancellationToken cancellationToken)
|
||||
{
|
||||
var token = IdentityProvider.MintToken(subject);
|
||||
var client = Server.CreateWebSocketClient();
|
||||
|
||||
client.SubProtocols.Add(VaultEvents.SubProtocol);
|
||||
// The server-side request, so the header is a raw string rather than a typed value.
|
||||
client.ConfigureRequest = request => request.Headers.Authorization = $"Bearer {token}";
|
||||
|
||||
return client.ConnectAsync(
|
||||
new Uri(Server.BaseAddress, VaultEvents.Path.TrimStart('/')),
|
||||
cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Shares one host and container across every test class in the assembly.</summary>
|
||||
|
||||
@@ -56,6 +56,12 @@ public sealed class EndpointInventoryTests(ApiFixture fixture)
|
||||
"POST /api/v1/vaults/{vaultId:guid}/sync/pull name=SyncPull tags=Sync policies=Enrolled anon=False",
|
||||
"POST /api/v1/vaults/{vaultId:guid}/sync/push name=SyncPush tags=Sync policies=Enrolled anon=False",
|
||||
|
||||
// The WebSocket, gated exactly as sync is and for the same reason — it announces changes to
|
||||
// vaults, and a caller who could not read one has nothing to be told about. It appears here as
|
||||
// an ordinary route because that is what it is until the upgrade: the bearer token authorises
|
||||
// the handshake, unlike the relay's ticket. See ADR 0012.
|
||||
"GET /api/v1/events name=VaultEvents tags=Events policies=Enrolled anon=False",
|
||||
|
||||
// Enrolled, because the answer exists to be wrapped to and a caller with no key of their own has
|
||||
// nothing to wrap and no signature to attribute it with. There is no search here — see
|
||||
// DirectoryService for why an exact-match-only directory is a decision rather than a shortcut.
|
||||
|
||||
@@ -0,0 +1,461 @@
|
||||
using System.Net;
|
||||
using System.Net.WebSockets;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain;
|
||||
using DodoSSH.Infrastructure;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
|
||||
namespace DodoSSH.Api.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The push channel, over a real socket through the real authentication pipeline.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The tests that matter most here are the two negatives: an unauthenticated upgrade is refused, and a
|
||||
/// change to somebody else's vault does not reach this socket. A push channel that leaked <em>which
|
||||
/// vault ids exist and when they change</em> would be a disclosure the pull path takes deliberate
|
||||
/// trouble to avoid — <c>SyncPullEndpoint</c> answers 404 rather than 403 for exactly that reason —
|
||||
/// and it would be invisible in a test that only checked that notices arrive.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Ordering is asserted rather than absence-within-a-timeout wherever possible. "Nothing arrived in
|
||||
/// two seconds" is a test that passes on a slow machine for the wrong reason; "the first notice this
|
||||
/// socket saw was about its own vault, although another vault was written to first" is not.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
[Collection(ApiCollection.Name)]
|
||||
public sealed class EventsEndpointTests(ApiFixture fixture)
|
||||
{
|
||||
private static readonly DateTimeOffset Now = new(2026, 8, 4, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
/// <summary>
|
||||
/// How long a test will wait for a frame before calling it a failure.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Generous, because it is not a measurement: every wait here is for something already committed,
|
||||
/// so the only thing this bounds is how long a genuinely broken build hangs before it reports.
|
||||
/// </remarks>
|
||||
private static readonly TimeSpan FrameTimeout = TimeSpan.FromSeconds(30);
|
||||
|
||||
// ---- The handshake ----
|
||||
|
||||
[Fact]
|
||||
public async Task WithoutAToken_TheUpgradeIsRefused()
|
||||
{
|
||||
var client = fixture.Server.CreateWebSocketClient();
|
||||
client.SubProtocols.Add(VaultEvents.SubProtocol);
|
||||
|
||||
var connecting = client.ConnectAsync(
|
||||
new Uri(fixture.Server.BaseAddress, VaultEvents.Path.TrimStart('/')),
|
||||
TestContext.Current.CancellationToken);
|
||||
|
||||
await Should.ThrowAsync<InvalidOperationException>(connecting);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task BeforeEnrolling_Is403WithAnActionableCode()
|
||||
{
|
||||
// The same bar as sync: a caller with no identity key holds no vault key either, so every
|
||||
// notice this socket could carry is about ciphertext they cannot read.
|
||||
var client = fixture.CreateClientFor(NewSubject());
|
||||
|
||||
var response = await client.GetAsync(
|
||||
new Uri(VaultEvents.Path, UriKind.Relative),
|
||||
TestContext.Current.CancellationToken);
|
||||
|
||||
response.StatusCode.ShouldBe(HttpStatusCode.Forbidden);
|
||||
|
||||
var problem = await response.Content.ReadProblemAsync();
|
||||
problem.ShouldNotBeNull();
|
||||
problem.Code.ShouldBe(ProblemCodes.EnrollmentRequired);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task APlainGet_SaysItIsAWebSocket()
|
||||
{
|
||||
// A person, or a client with the wrong URL. Answering with a problem document rather than a
|
||||
// socket that closes is the difference between a diagnosable mistake and a mysterious one.
|
||||
var client = fixture.CreateClientFor(await SeedEnrolledUserAsync());
|
||||
|
||||
var response = await client.GetAsync(
|
||||
new Uri(VaultEvents.Path, UriKind.Relative),
|
||||
TestContext.Current.CancellationToken);
|
||||
|
||||
response.StatusCode.ShouldBe(HttpStatusCode.BadRequest);
|
||||
|
||||
var problem = await response.Content.ReadProblemAsync();
|
||||
problem.ShouldNotBeNull();
|
||||
problem.Code.ShouldBe(ProblemCodes.MalformedRequest);
|
||||
problem.Detail.ShouldNotBeNull().ShouldContain(VaultEvents.SubProtocol);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnUpgradeWithoutTheSubprotocol_IsRefused()
|
||||
{
|
||||
// The subprotocol is this socket's version negotiation, so accepting an upgrade that did not
|
||||
// offer it would mean answering a client in a dialect it never agreed to read.
|
||||
var (subject, _) = await SeedUserWithVaultAsync();
|
||||
|
||||
var client = fixture.Server.CreateWebSocketClient();
|
||||
client.ConfigureRequest = request => request.Headers.Authorization =
|
||||
$"Bearer {fixture.IdentityProvider.MintToken(subject)}";
|
||||
|
||||
var connecting = client.ConnectAsync(
|
||||
new Uri(fixture.Server.BaseAddress, VaultEvents.Path.TrimStart('/')),
|
||||
TestContext.Current.CancellationToken);
|
||||
|
||||
await Should.ThrowAsync<InvalidOperationException>(connecting);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TheFirstFrameIsHello()
|
||||
{
|
||||
var (subject, _) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
var hello = await ReadAsync(socket, timeout.Token);
|
||||
|
||||
hello.Kind.ShouldBe(VaultEventKinds.Hello);
|
||||
|
||||
// Sent so a client knows when silence means the socket is dead rather than quiet, and so a
|
||||
// socket following nothing — a real state, for an account with no vaults — is distinguishable
|
||||
// from one that is broken.
|
||||
hello.HeartbeatSeconds.ShouldNotBeNull().ShouldBeGreaterThan(0);
|
||||
hello.VaultCount.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MetaAdvertisesTheFeature()
|
||||
{
|
||||
// How a client decides whether to hold a socket open at all. Absence is not an error — it
|
||||
// means synchronise on the timer, which is what every client did before this existed.
|
||||
var client = fixture.CreateClient();
|
||||
|
||||
var meta = await (await client.GetAsync(
|
||||
new Uri("/api/v1/meta", UriKind.Relative),
|
||||
TestContext.Current.CancellationToken))
|
||||
.Content.ReadContractAsync<MetaResponse>();
|
||||
|
||||
meta.ShouldNotBeNull();
|
||||
meta.Features.ShouldContain(
|
||||
feature => string.Equals(feature, VaultEvents.Feature, StringComparison.Ordinal));
|
||||
}
|
||||
|
||||
// ---- Notices ----
|
||||
|
||||
[Fact]
|
||||
public async Task APush_AnnouncesTheVaultToAFollowingSocket()
|
||||
{
|
||||
var (subject, vaultId) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
var client = fixture.CreateClientFor(subject);
|
||||
var push = await client.PostContractAsync(PushUrl(vaultId), NewCreateBatch());
|
||||
push.EnsureSuccessStatusCode();
|
||||
|
||||
var notice = await ReadUntilAsync(socket, VaultEventKinds.VaultChanged, timeout.Token);
|
||||
|
||||
notice.VaultId.ShouldBe(vaultId);
|
||||
|
||||
// A hint for logging and coalescing, never a cursor: cursors are opaque and integrity-tagged,
|
||||
// and a client that tried to resume from this would be resuming from a number it invented.
|
||||
notice.Sequence.ShouldNotBeNull().ShouldBeGreaterThan(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ANoticeCarriesNoCiphertext()
|
||||
{
|
||||
// The load-bearing property of the whole design. A notice says only that a vault moved; the
|
||||
// client's answer is the delta pull it would have run on its timer anyway, which keeps exactly
|
||||
// one code path applying changes. See ADR 0012.
|
||||
var (subject, vaultId) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
var client = fixture.CreateClientFor(subject);
|
||||
var batch = NewCreateBatch();
|
||||
await client.PostContractAsync(PushUrl(vaultId), batch);
|
||||
|
||||
var raw = await ReadRawUntilAsync(socket, VaultEventKinds.VaultChanged, timeout.Token);
|
||||
|
||||
// The envelope this test pushed, as it would appear if a payload had been forwarded.
|
||||
raw.ShouldNotContain(Convert.ToBase64String(batch.Operations[0].Payload!.Envelope));
|
||||
raw.ShouldNotContain("payload", Case.Insensitive);
|
||||
raw.ShouldNotContain(batch.Operations[0].EntityId.ToString());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task APushToAnotherAccountsVault_IsNotAnnouncedHere()
|
||||
{
|
||||
// The disclosure that would matter: a socket learning that vault ids it cannot read exist,
|
||||
// and when somebody works on them.
|
||||
var (subject, vaultId) = await SeedUserWithVaultAsync();
|
||||
var (stranger, strangersVaultId) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
// The stranger's write goes first, so a socket that leaked would have announced it before the
|
||||
// one this test then waits for. Ordering, not a timeout: "nothing arrived in two seconds"
|
||||
// passes on a slow machine for the wrong reason.
|
||||
var strangersClient = fixture.CreateClientFor(stranger);
|
||||
(await strangersClient.PostContractAsync(PushUrl(strangersVaultId), NewCreateBatch()))
|
||||
.EnsureSuccessStatusCode();
|
||||
|
||||
var ownClient = fixture.CreateClientFor(subject);
|
||||
(await ownClient.PostContractAsync(PushUrl(vaultId), NewCreateBatch()))
|
||||
.EnsureSuccessStatusCode();
|
||||
|
||||
var notice = await ReadUntilAsync(socket, VaultEventKinds.VaultChanged, timeout.Token);
|
||||
|
||||
notice.VaultId.ShouldBe(vaultId);
|
||||
notice.VaultId.ShouldNotBe(strangersVaultId);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task APushThatAppliedNothing_AnnouncesNothing()
|
||||
{
|
||||
// A batch of pure conflicts moved no vault. Announcing one anyway would have every client on
|
||||
// it pull for a change that is not there.
|
||||
var (subject, vaultId) = await SeedUserWithVaultAsync();
|
||||
var client = fixture.CreateClientFor(subject);
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
// An update to an item that does not exist: rejected as a conflict, nothing written.
|
||||
var stale = new SyncPushRequest(
|
||||
[
|
||||
NewOperation(Guid.CreateVersion7(), expectedVersion: 7, envelope: [9, 9]),
|
||||
]);
|
||||
|
||||
var conflicted = await client.PostContractAsync(PushUrl(vaultId), stale);
|
||||
conflicted.EnsureSuccessStatusCode();
|
||||
|
||||
var results = await conflicted.Content.ReadContractAsync<SyncPushResponse>();
|
||||
results.ShouldNotBeNull();
|
||||
results.Results[0].Status.ShouldBe(SyncOperationStatus.Conflict);
|
||||
|
||||
// Then a write that did land. The first notice must be that one.
|
||||
(await client.PostContractAsync(PushUrl(vaultId), NewCreateBatch())).EnsureSuccessStatusCode();
|
||||
|
||||
var notice = await ReadUntilAsync(socket, VaultEventKinds.VaultChanged, timeout.Token);
|
||||
|
||||
notice.Sequence.ShouldNotBeNull().ShouldBeGreaterThan(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task APing_IsAnswered()
|
||||
{
|
||||
var (subject, _) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
await SendAsync(socket, new VaultEvent(VaultEventKinds.Ping), timeout.Token);
|
||||
|
||||
var pong = await ReadUntilAsync(socket, VaultEventKinds.Pong, timeout.Token);
|
||||
|
||||
pong.Kind.ShouldBe(VaultEventKinds.Pong);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AFrameThisServerCannotRead_DoesNotEndTheSocket()
|
||||
{
|
||||
// A control channel whose failure mode is "the client polls instead" should tolerate a frame
|
||||
// from a newer client rather than cost that client its push for the whole session.
|
||||
var (subject, vaultId) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
await socket.SendAsync(
|
||||
Encoding.UTF8.GetBytes("{ not json at all"),
|
||||
WebSocketMessageType.Text,
|
||||
endOfMessage: true,
|
||||
timeout.Token);
|
||||
|
||||
var client = fixture.CreateClientFor(subject);
|
||||
(await client.PostContractAsync(PushUrl(vaultId), NewCreateBatch())).EnsureSuccessStatusCode();
|
||||
|
||||
var notice = await ReadUntilAsync(socket, VaultEventKinds.VaultChanged, timeout.Token);
|
||||
|
||||
notice.VaultId.ShouldBe(vaultId);
|
||||
socket.State.ShouldBe(WebSocketState.Open);
|
||||
}
|
||||
|
||||
// ---- Helpers ----
|
||||
|
||||
/// <summary>A token that gives up rather than letting a broken build hang the suite.</summary>
|
||||
private static CancellationTokenSource Timeout()
|
||||
{
|
||||
var source = CancellationTokenSource.CreateLinkedTokenSource(
|
||||
TestContext.Current.CancellationToken);
|
||||
|
||||
source.CancelAfter(FrameTimeout);
|
||||
|
||||
return source;
|
||||
}
|
||||
|
||||
private static async Task<VaultEvent> ReadAsync(WebSocket socket, CancellationToken cancellationToken)
|
||||
{
|
||||
var json = await ReadRawAsync(socket, cancellationToken);
|
||||
|
||||
return JsonSerializer.Deserialize(json, DodoSshJsonContext.Default.VaultEvent)
|
||||
?? throw new InvalidOperationException($"The server sent a null frame: {json}");
|
||||
}
|
||||
|
||||
private static async Task<string> ReadRawAsync(WebSocket socket, CancellationToken cancellationToken)
|
||||
{
|
||||
var buffer = new byte[8 * 1024];
|
||||
|
||||
var received = await socket.ReceiveAsync(buffer, cancellationToken);
|
||||
|
||||
if (received.MessageType == WebSocketMessageType.Close)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"The server closed the socket: {received.CloseStatus} {received.CloseStatusDescription}");
|
||||
}
|
||||
|
||||
return Encoding.UTF8.GetString(buffer, 0, received.Count);
|
||||
}
|
||||
|
||||
/// <summary>Reads past the frames a test does not care about — hello, and heartbeats.</summary>
|
||||
private static async Task<VaultEvent> ReadUntilAsync(
|
||||
WebSocket socket,
|
||||
string kind,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
var frame = await ReadAsync(socket, cancellationToken);
|
||||
|
||||
if (string.Equals(frame.Kind, kind, StringComparison.Ordinal))
|
||||
{
|
||||
return frame;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The same, but keeping the bytes.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Deserialising and asserting on the fields would prove only that this <em>record</em> has no
|
||||
/// payload member, which is a tautology. Asserting on what actually crossed the socket is what
|
||||
/// would catch a field added to the frame later without anybody thinking about disclosure.
|
||||
/// </remarks>
|
||||
private static async Task<string> ReadRawUntilAsync(
|
||||
WebSocket socket,
|
||||
string kind,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
var json = await ReadRawAsync(socket, cancellationToken);
|
||||
var frame = JsonSerializer.Deserialize(json, DodoSshJsonContext.Default.VaultEvent);
|
||||
|
||||
if (string.Equals(frame?.Kind, kind, StringComparison.Ordinal))
|
||||
{
|
||||
return json;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static Task SendAsync(WebSocket socket, VaultEvent frame, CancellationToken cancellationToken) =>
|
||||
socket.SendAsync(
|
||||
JsonSerializer.SerializeToUtf8Bytes(frame, DodoSshJsonContext.Default.VaultEvent),
|
||||
WebSocketMessageType.Text,
|
||||
endOfMessage: true,
|
||||
cancellationToken);
|
||||
|
||||
private static string PushUrl(Guid vaultId) => $"/api/v1/vaults/{vaultId}/sync/push";
|
||||
|
||||
private static string NewSubject() => $"events-{Guid.CreateVersion7():N}";
|
||||
|
||||
private static SyncPushOperation NewOperation(Guid entityId, int? expectedVersion, byte[] envelope) =>
|
||||
new(
|
||||
Guid.CreateVersion7(),
|
||||
SyncEntityType.Host,
|
||||
entityId,
|
||||
SyncOperation.Upsert,
|
||||
expectedVersion,
|
||||
new EncryptedPayload(envelope, [0xD, 0xE], Guid.CreateVersion7(), 1, 1),
|
||||
new SyncPlaintextFields());
|
||||
|
||||
private static SyncPushRequest NewCreateBatch() =>
|
||||
new([NewOperation(Guid.CreateVersion7(), expectedVersion: null, envelope: [1, 2, 3, 4])]);
|
||||
|
||||
private async Task<string> SeedEnrolledUserAsync()
|
||||
{
|
||||
var subject = NewSubject();
|
||||
|
||||
await using var scope = fixture.CreateScope();
|
||||
var database = scope.ServiceProvider.GetRequiredService<DodoDbContext>();
|
||||
|
||||
var user = NewUser(subject);
|
||||
database.Users.Add(user);
|
||||
database.UserKeys.Add(Seed.CurrentKey(user.Id, Now));
|
||||
await database.SaveChangesAsync();
|
||||
|
||||
return subject;
|
||||
}
|
||||
|
||||
private async Task<(string Subject, Guid VaultId)> SeedUserWithVaultAsync()
|
||||
{
|
||||
var subject = NewSubject();
|
||||
|
||||
await using var scope = fixture.CreateScope();
|
||||
var database = scope.ServiceProvider.GetRequiredService<DodoDbContext>();
|
||||
|
||||
var user = NewUser(subject);
|
||||
|
||||
var vault = new Vault
|
||||
{
|
||||
Id = Guid.CreateVersion7(),
|
||||
Name = "Personal",
|
||||
OwnerKind = VaultOwnerKind.Personal,
|
||||
OwnerUserId = user.Id,
|
||||
KeyGeneration = 1,
|
||||
CreatedAtUtc = Now,
|
||||
UpdatedAtUtc = Now,
|
||||
};
|
||||
|
||||
database.Users.Add(user);
|
||||
database.UserKeys.Add(Seed.CurrentKey(user.Id, Now));
|
||||
database.Vaults.Add(vault);
|
||||
await database.SaveChangesAsync();
|
||||
|
||||
return (subject, vault.Id);
|
||||
}
|
||||
|
||||
private UserAccount NewUser(string subject) => new()
|
||||
{
|
||||
Id = Guid.CreateVersion7(),
|
||||
Issuer = fixture.IdentityProvider.Authority,
|
||||
Subject = subject,
|
||||
Status = UserStatus.Active,
|
||||
CreatedAtUtc = Now,
|
||||
UpdatedAtUtc = Now,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,388 @@
|
||||
using System.Net.WebSockets;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Channels;
|
||||
using DodoSSH.Client.Api;
|
||||
using DodoSSH.Contracts;
|
||||
|
||||
namespace DodoSSH.Client.Api.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The reconnection policy, which is what this class actually is.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// A dropped socket is the ordinary case here rather than the exception — laptops sleep, proxies time
|
||||
/// out, tokens expire, servers are redeployed — so the behaviour worth covering is what happens
|
||||
/// <em>after</em> a failure, not the happy path. Driven through the injected connector, because the one
|
||||
/// thing a test cannot do to a real network is make it fail on cue.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The backoff is configured down to milliseconds throughout. What is under test is the shape of the
|
||||
/// policy — does it try again, does it wait, does it stop waiting when told the token was the problem —
|
||||
/// and none of that depends on the intervals a shipped client uses.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class VaultEventStreamTests
|
||||
{
|
||||
private static readonly Uri Server = new("https://dodossh.example");
|
||||
|
||||
private static readonly VaultEventStreamOptions Impatient = new()
|
||||
{
|
||||
InitialBackoff = TimeSpan.FromMilliseconds(1),
|
||||
MaxBackoff = TimeSpan.FromMilliseconds(5),
|
||||
InitialSilenceTimeout = TimeSpan.FromSeconds(30),
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public async Task ItDialsTheWebSocketFormOfTheServersUrl()
|
||||
{
|
||||
// https becomes wss, and the path is the one in the contract. Getting either wrong is a client
|
||||
// that reconnects against a 404 for the whole session, which from outside is indistinguishable
|
||||
// from a network that eats WebSockets.
|
||||
var dialled = new List<Uri>();
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream(
|
||||
(url, _, _) =>
|
||||
{
|
||||
dialled.Add(url);
|
||||
return Task.FromResult<WebSocket>(socket);
|
||||
});
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
|
||||
await stream.ReadAsync(Token);
|
||||
|
||||
dialled[0].ShouldBe(new Uri("wss://dodossh.example/api/v1/events"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ItSendsTheBearerTokenOnTheUpgrade()
|
||||
{
|
||||
// The whole of this socket's authorization, unlike the relay's ticket. See ADR 0012.
|
||||
var presented = new List<string>();
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream(
|
||||
(_, token, _) =>
|
||||
{
|
||||
presented.Add(token);
|
||||
return Task.FromResult<WebSocket>(socket);
|
||||
});
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
|
||||
await stream.ReadAsync(Token);
|
||||
|
||||
presented[0].ShouldBe(StubTokens.Token);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ANoticeReachesTheReader()
|
||||
{
|
||||
var vaultId = Guid.CreateVersion7();
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream((_, _, _) => Task.FromResult<WebSocket>(socket));
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, vaultId, 42));
|
||||
|
||||
var notice = await stream.ReadAsync(Token);
|
||||
|
||||
notice.Kind.ShouldBe(VaultEventKinds.VaultChanged);
|
||||
notice.VaultId.ShouldBe(vaultId);
|
||||
notice.Sequence.ShouldBe(42);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AHeartbeatIsAnsweredAndNotHandedToTheReader()
|
||||
{
|
||||
// A ping is housekeeping between the two ends. Passing it up would wake a synchronisation loop
|
||||
// every thirty seconds for a frame that says nothing happened.
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream((_, _, _) => Task.FromResult<WebSocket>(socket));
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.Ping, HeartbeatSeconds: 30));
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
|
||||
var first = await stream.ReadAsync(Token);
|
||||
|
||||
first.Kind.ShouldBe(VaultEventKinds.VaultChanged, "the ping should not have been forwarded");
|
||||
|
||||
var answered = await socket.SentAsync(Token);
|
||||
answered.Kind.ShouldBe(VaultEventKinds.Pong);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AKindThisBuildDoesNotKnow_IsStillHandedOver()
|
||||
{
|
||||
// What makes the frame table extensible: this class must not decide what a newer server may
|
||||
// say. Deciding to ignore it is the caller's, and costs that caller one redundant pass.
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream((_, _, _) => Task.FromResult<WebSocket>(socket));
|
||||
|
||||
socket.Deliver(new VaultEvent("session.offered"));
|
||||
|
||||
var notice = await stream.ReadAsync(Token);
|
||||
|
||||
notice.Kind.ShouldBe("session.offered");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AFailedDial_IsRetried()
|
||||
{
|
||||
// No server yet, or no network. Neither is an error to report: the caller's synchronisation
|
||||
// timer is running regardless, which is what lets this stay silent and keep trying.
|
||||
var attempts = 0;
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream((_, _, _) =>
|
||||
{
|
||||
if (++attempts < 3)
|
||||
{
|
||||
throw new WebSocketException("no route to host");
|
||||
}
|
||||
|
||||
return Task.FromResult<WebSocket>(socket);
|
||||
});
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
|
||||
var notice = await stream.ReadAsync(Token);
|
||||
|
||||
notice.Kind.ShouldBe(VaultEventKinds.VaultChanged);
|
||||
attempts.ShouldBe(3);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ADroppedSocket_IsReplaced()
|
||||
{
|
||||
// The case that decides whether this feature survives a laptop lid. A stream that gave up on
|
||||
// the first close would work all morning and be silently dead after lunch.
|
||||
var sockets = new List<FakeWebSocket>();
|
||||
|
||||
await using var stream = Stream((_, _, _) =>
|
||||
{
|
||||
var socket = new FakeWebSocket();
|
||||
sockets.Add(socket);
|
||||
|
||||
if (sockets.Count == 1)
|
||||
{
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
socket.Close(WebSocketCloseStatus.EndpointUnavailable);
|
||||
}
|
||||
else
|
||||
{
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 2));
|
||||
}
|
||||
|
||||
return Task.FromResult<WebSocket>(socket);
|
||||
});
|
||||
|
||||
(await stream.ReadAsync(Token)).Sequence.ShouldBe(1);
|
||||
(await stream.ReadAsync(Token)).Sequence.ShouldBe(2);
|
||||
|
||||
sockets.Count.ShouldBeGreaterThanOrEqualTo(2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnExpiredTokenClose_ReconnectsAndAsksForAFreshToken()
|
||||
{
|
||||
// The bound that lets a long-lived socket be authorised by a short-lived credential: the server
|
||||
// closes at the token's expiry and the client comes straight back with a new one. The token
|
||||
// provider being asked again is the half that matters — reconnecting with the spent token would
|
||||
// be an unbroken loop of closes.
|
||||
var tokens = new StubTokens();
|
||||
var sockets = 0;
|
||||
|
||||
await using var stream = new VaultEventStream(
|
||||
Server,
|
||||
tokens,
|
||||
TimeProvider.System,
|
||||
(_, _, _) =>
|
||||
{
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
if (++sockets == 1)
|
||||
{
|
||||
socket.Close((WebSocketCloseStatus)VaultEvents.TokenExpiredCloseCode);
|
||||
}
|
||||
else
|
||||
{
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 7));
|
||||
}
|
||||
|
||||
return Task.FromResult<WebSocket>(socket);
|
||||
},
|
||||
Impatient);
|
||||
|
||||
(await stream.ReadAsync(Token)).Sequence.ShouldBe(7);
|
||||
|
||||
tokens.Requests.ShouldBeGreaterThanOrEqualTo(2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TryRead_TakesWhatIsWaitingAndSaysWhenNothingIs()
|
||||
{
|
||||
// How a caller coalesces a burst: read one, wait a moment, swallow the rest. Without this a
|
||||
// colleague tidying a folder would produce a synchronisation pass per item.
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream((_, _, _) => Task.FromResult<WebSocket>(socket));
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 2));
|
||||
|
||||
(await stream.ReadAsync(Token)).Sequence.ShouldBe(1);
|
||||
|
||||
// Delivery is asynchronous, so the second may not have landed yet; this is the same
|
||||
// wait-then-drain the caller performs.
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(200), Token);
|
||||
|
||||
stream.TryRead(out var queued).ShouldBeTrue();
|
||||
queued.Sequence.ShouldBe(2);
|
||||
|
||||
stream.TryRead(out _).ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnIdleStream_NeverDelivers()
|
||||
{
|
||||
// What a server without the feature supplies. Waiting for ever rather than completing is the
|
||||
// point: a caller selecting between this and a timer has to fall through to the timer, and a
|
||||
// read that returned at once would spin that loop as fast as the machine allows.
|
||||
using var stream = IdleVaultEventStream.Instance;
|
||||
using var giveUp = CancellationTokenSource.CreateLinkedTokenSource(Token);
|
||||
|
||||
giveUp.CancelAfter(TimeSpan.FromMilliseconds(100));
|
||||
|
||||
await Should.ThrowAsync<OperationCanceledException>(
|
||||
async () => await stream.ReadAsync(giveUp.Token));
|
||||
|
||||
stream.TryRead(out _).ShouldBeFalse();
|
||||
stream.IsConnected.ShouldBeFalse();
|
||||
}
|
||||
|
||||
private static CancellationToken Token => TestContext.Current.CancellationToken;
|
||||
|
||||
private static VaultEventStream Stream(
|
||||
Func<Uri, string, CancellationToken, Task<WebSocket>> connect) =>
|
||||
new(Server, new StubTokens(), TimeProvider.System, connect, Impatient);
|
||||
|
||||
/// <summary>A token provider that hands out one value and counts who asked.</summary>
|
||||
private sealed class StubTokens : IAccessTokenProvider
|
||||
{
|
||||
internal const string Token = "access-token";
|
||||
|
||||
internal int Requests { get; private set; }
|
||||
|
||||
public ValueTask<string> GetAccessTokenAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
Requests++;
|
||||
|
||||
return ValueTask.FromResult(Token);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A socket a test writes the server's half of.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Frames queued with <see cref="Deliver"/> are handed out by <see cref="ReceiveAsync"/> in order;
|
||||
/// once the queue is empty the receive waits, which is what an idle connection does. <see
|
||||
/// cref="Close"/> queues the close instead, so a test can script "two notices and then the server
|
||||
/// went away" as a value rather than as a race.
|
||||
/// </remarks>
|
||||
private sealed class FakeWebSocket : WebSocket
|
||||
{
|
||||
private readonly Channel<byte[]> inbound = Channel.CreateUnbounded<byte[]>();
|
||||
private readonly Channel<VaultEvent> outbound = Channel.CreateUnbounded<VaultEvent>();
|
||||
|
||||
private WebSocketCloseStatus? closing;
|
||||
private WebSocketState state = WebSocketState.Open;
|
||||
|
||||
public override WebSocketCloseStatus? CloseStatus => closing;
|
||||
|
||||
public override string? CloseStatusDescription => null;
|
||||
|
||||
public override WebSocketState State => state;
|
||||
|
||||
public override string? SubProtocol => VaultEvents.SubProtocol;
|
||||
|
||||
/// <summary>Queues a frame for the client to read.</summary>
|
||||
internal void Deliver(VaultEvent frame) =>
|
||||
inbound.Writer.TryWrite(
|
||||
JsonSerializer.SerializeToUtf8Bytes(frame, DodoSshJsonContext.Default.VaultEvent));
|
||||
|
||||
/// <summary>Ends the socket, after everything already queued has been read.</summary>
|
||||
internal void Close(WebSocketCloseStatus status)
|
||||
{
|
||||
closing = status;
|
||||
inbound.Writer.TryWrite([]);
|
||||
}
|
||||
|
||||
/// <summary>The next frame the client sent.</summary>
|
||||
internal ValueTask<VaultEvent> SentAsync(CancellationToken cancellationToken) =>
|
||||
outbound.Reader.ReadAsync(cancellationToken);
|
||||
|
||||
public override async Task<WebSocketReceiveResult> ReceiveAsync(
|
||||
ArraySegment<byte> buffer,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var frame = await inbound.Reader.ReadAsync(cancellationToken);
|
||||
|
||||
// The empty frame Close queues. Reported as a close, exactly as a real socket does once the
|
||||
// peer's close frame arrives.
|
||||
if (frame.Length == 0)
|
||||
{
|
||||
state = WebSocketState.Closed;
|
||||
|
||||
return new WebSocketReceiveResult(
|
||||
0, WebSocketMessageType.Close, endOfMessage: true, closing, null);
|
||||
}
|
||||
|
||||
frame.CopyTo(buffer.Array!, buffer.Offset);
|
||||
|
||||
return new WebSocketReceiveResult(frame.Length, WebSocketMessageType.Text, endOfMessage: true);
|
||||
}
|
||||
|
||||
public override Task SendAsync(
|
||||
ArraySegment<byte> buffer,
|
||||
WebSocketMessageType messageType,
|
||||
bool endOfMessage,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var json = Encoding.UTF8.GetString(buffer.Array!, buffer.Offset, buffer.Count);
|
||||
|
||||
if (JsonSerializer.Deserialize(json, DodoSshJsonContext.Default.VaultEvent) is { } frame)
|
||||
{
|
||||
outbound.Writer.TryWrite(frame);
|
||||
}
|
||||
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public override void Abort() => state = WebSocketState.Aborted;
|
||||
|
||||
public override Task CloseAsync(
|
||||
WebSocketCloseStatus closeStatus,
|
||||
string? statusDescription,
|
||||
CancellationToken cancellationToken) => CloseOutputAsync(
|
||||
closeStatus, statusDescription, cancellationToken);
|
||||
|
||||
public override Task CloseOutputAsync(
|
||||
WebSocketCloseStatus closeStatus,
|
||||
string? statusDescription,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
state = WebSocketState.Closed;
|
||||
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public override void Dispose() => state = WebSocketState.Closed;
|
||||
}
|
||||
}
|
||||
@@ -57,6 +57,17 @@ internal sealed class StubTeamServer : IVaultServer, ITeamApi, IVaultGrantApi
|
||||
/// <inheritdoc />
|
||||
public IKeyBindingAuthorizer KeyBinding => throw new NotSupportedException();
|
||||
|
||||
/// <summary>
|
||||
/// A push channel that never pushes.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Not <c>NotSupportedException</c> like its neighbours: the background synchronisation loop reads
|
||||
/// this on every wait, so a layout test that opened a screen would throw from a timer thread rather
|
||||
/// than draw anything. Waiting for ever is the honest stand-in — an offline layout test has no
|
||||
/// server to be pushed from.
|
||||
/// </remarks>
|
||||
public IVaultEventStream Events => IdleVaultEventStream.Instance;
|
||||
|
||||
/// <inheritdoc />
|
||||
public SyncOptions SyncOptions => new();
|
||||
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
using System.Threading.Channels;
|
||||
using DodoSSH.Client.Api;
|
||||
using DodoSSH.Contracts;
|
||||
|
||||
namespace DodoSSH.Client.App.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// A server's push channel, driven by a test rather than by a socket.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The real <c>VaultEventStream</c> is a reconnection policy wrapped round a WebSocket, and none of
|
||||
/// that is what the shell's behaviour depends on: what the shell does with a notice is the same
|
||||
/// whether it arrived over a healthy socket, after four reconnections, or from this. Driving it by
|
||||
/// hand is what makes "the loop synchronised because it was told to, not because a minute passed" a
|
||||
/// test that finishes in milliseconds and cannot flake.
|
||||
/// </remarks>
|
||||
internal sealed class FakeVaultEventStream : IVaultEventStream
|
||||
{
|
||||
private readonly Channel<VaultEvent> notices = Channel.CreateUnbounded<VaultEvent>();
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool IsConnected => true;
|
||||
|
||||
/// <summary>How many times the shell has waited on this. Proves the loop is watching at all.</summary>
|
||||
internal int Reads { get; private set; }
|
||||
|
||||
/// <summary>Delivers a notice, as a server would.</summary>
|
||||
internal void Push(Guid vaultId, long sequence = 1) =>
|
||||
notices.Writer.TryWrite(
|
||||
new VaultEvent(VaultEventKinds.VaultChanged, vaultId, sequence));
|
||||
|
||||
/// <summary>Delivers the notice that says the caller's vault list has changed.</summary>
|
||||
internal void PushAccessChanged() =>
|
||||
notices.Writer.TryWrite(new VaultEvent(VaultEventKinds.VaultsChanged));
|
||||
|
||||
/// <inheritdoc />
|
||||
public ValueTask<VaultEvent> ReadAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
Reads++;
|
||||
|
||||
return notices.Reader.ReadAsync(cancellationToken);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool TryRead(out VaultEvent notice) => notices.Reader.TryRead(out notice!);
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose() => notices.Writer.TryComplete();
|
||||
}
|
||||
@@ -39,6 +39,16 @@ internal sealed partial class FakeVaultServer : IVaultServer, IAccountApi, ISync
|
||||
|
||||
internal int PushCount { get; private set; }
|
||||
|
||||
/// <summary>
|
||||
/// How many delta reads this server has served.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The one observable a synchronisation pass always produces. <see cref="PushCount"/> only moves when
|
||||
/// there is something queued, so a test asking "did a pass run" — which is what the push channel's
|
||||
/// whole purpose comes down to — has to count pulls.
|
||||
/// </remarks>
|
||||
internal int PullCount { get; private set; }
|
||||
|
||||
internal bool IsEnrolled => statement is not null;
|
||||
|
||||
/// <summary>
|
||||
@@ -99,6 +109,19 @@ internal sealed partial class FakeVaultServer : IVaultServer, IAccountApi, ISync
|
||||
/// <inheritdoc />
|
||||
public IKeyBindingAuthorizer KeyBinding => this;
|
||||
|
||||
/// <summary>
|
||||
/// The push channel, which a test drives by hand.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A real queue rather than an idle stand-in, because the behaviour worth covering here is the one
|
||||
/// the socket exists for: a notice arriving makes the background loop synchronise without waiting
|
||||
/// out its minute. See <see cref="FakeVaultEventStream.Push"/>.
|
||||
/// </remarks>
|
||||
internal FakeVaultEventStream Notices { get; } = new();
|
||||
|
||||
/// <inheritdoc />
|
||||
public IVaultEventStream Events => Notices;
|
||||
|
||||
/// <inheritdoc />
|
||||
public SyncOptions SyncOptions => SyncOptions.Default;
|
||||
|
||||
@@ -238,6 +261,8 @@ internal sealed partial class FakeVaultServer : IVaultServer, IAccountApi, ISync
|
||||
SyncPullRequest request,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
PullCount++;
|
||||
|
||||
if (SyncFailure is { } failure)
|
||||
{
|
||||
return Task.FromException<SyncPullResponse>(failure);
|
||||
|
||||
@@ -523,6 +523,90 @@ public sealed class ShellFlowTests : IAsyncLifetime
|
||||
vault.Status.ShouldContain("bad day");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The whole point of the push channel: a pass that did not wait for the minute.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The timing is what makes this an assertion rather than a hope. The background timer is a full
|
||||
/// minute and the wait below gives up in ten seconds, so a pull that arrives can only have been
|
||||
/// caused by the notice — there is no interval at which the timer could have produced it.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The vault id in the notice is arbitrary, and deliberately so: a pass synchronises every vault
|
||||
/// this session can reach, so the loop reads the notice as "there is something to fetch" and never
|
||||
/// as "fetch this one". A test that seeded a real id would imply a targeting this does not do.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
[Fact]
|
||||
public async Task APushedNotice_SynchronisesWithoutWaitingForTheTimer()
|
||||
{
|
||||
await UnlockedAsync();
|
||||
|
||||
// The unlock starts the loop, whose first act is a pass; waited out so the count below is a
|
||||
// baseline rather than a race with it.
|
||||
await EventuallyAsync(
|
||||
() => server.PullCount > 0,
|
||||
"the pass on open should have run");
|
||||
|
||||
var before = server.PullCount;
|
||||
|
||||
server.Notices.Push(Guid.CreateVersion7());
|
||||
|
||||
await EventuallyAsync(
|
||||
() => server.PullCount > before,
|
||||
"a notice should have woken the loop long before the one-minute timer");
|
||||
}
|
||||
|
||||
/// <remarks>
|
||||
/// The half that is easy to get wrong. The loop selects between two waits, and both have to survive
|
||||
/// losing: <c>PeriodicTimer</c> throws if a second wait is started while one is outstanding, and an
|
||||
/// abandoned channel read stays registered and swallows the next notice written. Either defect
|
||||
/// leaves the first notice working and every one after it silently lost, which is why one notice is
|
||||
/// not enough to prove this.
|
||||
/// </remarks>
|
||||
[Fact]
|
||||
public async Task NoticesKeepWakingTheLoop_NotJustTheFirst()
|
||||
{
|
||||
await UnlockedAsync();
|
||||
|
||||
await EventuallyAsync(() => server.PullCount > 0, "the pass on open should have run");
|
||||
|
||||
for (var round = 1; round <= 3; round++)
|
||||
{
|
||||
var before = server.PullCount;
|
||||
|
||||
server.Notices.Push(Guid.CreateVersion7());
|
||||
|
||||
await EventuallyAsync(
|
||||
() => server.PullCount > before,
|
||||
$"notice {round} should have woken the loop as the first one did");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Waits for something a background loop is expected to do, or fails saying what.</summary>
|
||||
/// <remarks>
|
||||
/// Polled rather than signalled because the thing under test is a loop nobody hands a completion
|
||||
/// source to. The bound is generous — this is not measuring latency, only proving that the timer
|
||||
/// cannot be what caused the result.
|
||||
/// </remarks>
|
||||
private static async Task EventuallyAsync(Func<bool> condition, string because)
|
||||
{
|
||||
var deadline = TimeProvider.System.GetUtcNow().AddSeconds(10);
|
||||
|
||||
while (TimeProvider.System.GetUtcNow() < deadline)
|
||||
{
|
||||
if (condition())
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(20), Token);
|
||||
}
|
||||
|
||||
throw new ShouldAssertException(because);
|
||||
}
|
||||
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The page's own <c>term.focus()</c> focuses the textarea inside the document, which does nothing
|
||||
|
||||
Reference in New Issue
Block a user