Public Access
Say when a vault has moved, so nobody waits out the minute
The delta pull was cheap enough to run on a timer and the client did, once a minute. That is fine for a machine and wrong for two people: an edit a colleague makes is up to a minute stale, which is long enough for both of them to make it and produce a conflict neither needed to have. Shortening the interval is the obvious answer and the wrong one — it costs a request per client per interval whether or not anything happened, and it converges on a busier server that is still late. So the server now says so. A client holds a WebSocket open at GET /api/v1/events, subprotocol dodossh.events.v1, and gets a line down it when something it can read has changed. ADR 0012 has the reasoning; three parts of it are worth repeating here, because they are what everything else rests on. **What crosses the socket is a notice, never data.** A frame names a vault and how far its change log has got. No item, no ciphertext, not even which item it was. The client's answer is the delta pull it would have run anyway, so there is still exactly one code path that applies a change to a keychain, and it is not this one. Pushing the items themselves would save a round trip and fork that path in two, with the cursor, the merge and the tombstone rules duplicated across both — ADR 0003 put every mutation through one write path for that reason, and this keeps every read on one for the same one. It also makes a dropped notice harmless, which is what lets the fan-out below be as simple as it is. **Polling stays, and is what guarantees a pass.** The minute timer is unchanged. A network that eats WebSockets, a server with Events:Enabled off, an older server, a proxy that will not upgrade, a notice dropped under backpressure — every one of those leaves a client behaving exactly as it did before this commit. Nothing is reachable only over the socket and nothing is meant to become so; VaultViewModel's AutoSyncInterval remark now says that where somebody changing it will read it. **The bearer token authorises the upgrade, unlike the relay's ticket.** Not an inconsistency with ADR 0004: the relay's socket is a byte pipe whose whole authorization decision — which host, which IPs, which port — is made before it opens and never revisited, and it is the extraction seam for a process that must hold no ACL code. This one is a view of the caller's own vault list and has to keep answering "what may this account read" for as long as it is held. A ticket would carry that answer in a token and be wrong the moment the account's access changed. The two bounds that arrangement needs are met rather than waved at: the socket is closed at the token's exp with close code 4401 and the client comes straight back with a fresh one, and the vault set is re-resolved every few minutes as well as on the changes known to affect it. Both bound *metadata*, because a notice contains nothing else and reading a vault still needs a key this server has never held. **The fan-out.** VaultEventHub is a singleton holding the sockets this node accepted; publishing walks them and asks each whether it cares, rather than keeping a vault-to-subscriber index that every re-subscription would have to move entries between under a lock publishing also takes. At a few hundred sockets per node and an event rate bounded by how often people edit keychains, the walk is not measurable and its races are obvious. Per-connection queues are bounded and drop the *oldest*: a notice means "pull vault X, which is at least at sequence N", so the newest subsumes what it displaces and the client's answer is identical either way — which is what lets the publish path be void, never block, and never fail. Announced from the endpoint rather than from SyncService, and that placement is the point: by then the push has committed and released the per-vault advisory lock. From inside it would name a sequence no reader can see yet and would hold the lock that serialises writers across a socket write. Only the highest *applied* sequence, so a batch of pure conflicts announces nothing, and a duplicate — already announced when it first landed — announces nothing either. Grants and membership publish too, and those take the *recipient* rather than the actor. This is what AdmitNewVaultsAsync has been apologising for since sharing shipped — "the recipient is handed nothing, there is no push channel" — and the README with it. A vault shared with somebody now turns up as it is shared. The comment and the README paragraph both say what is true now, and both keep saying that the pass is what *discovers* the vault, because a client with no socket has to arrive at the same place. **On the client**, VaultEventStream is really a reconnection policy wrapped round a ClientWebSocket: a dropped socket is the ordinary case here — laptops sleep, proxies time out, tokens expire, servers are redeployed — so nothing in it treats a failure as exceptional, and every path ends in "wait, then dial again". A connection that lived long enough to say hello resets the backoff, so a laptop that woke, worked, and lost its network an hour later does not inherit a minute-long wait it has already proved it need not take. A 4401 close skips the backoff entirely and asks the token provider again, which is the whole reason that close code is distinct. A server that does not advertise the events feature gets IdleVaultEventStream, which never delivers — so IVaultServer.Events is never null and every caller stays on one shape, because the correct behaviour without a socket is the behaviour with a silent one. The shell's background loop now selects between the timer and a notice, and both waits are held across iterations. That is load-bearing rather than tidy: PeriodicTimer permits one outstanding WaitForNextTickAsync and throws on a second, and an abandoned channel read stays registered and consumes the next notice written. Either defect leaves the first notice working and every one after it silently lost, which is why NoticesKeepWakingTheLoop_NotJustTheFirst pushes three and not one. Notices are coalesced over a quarter of a second, so one person's save — a host and its log entry are two items — and a colleague clearing a folder each cost one pass rather than a dozen. **The kind is a string, not an enum**, and that is a compatibility decision. UseStringEnumConverter throws on a value it does not know, so a newer server sending a kind an older client had never heard of would not add an unreadable frame — it would break that client's socket outright. A string is ignored instead. ProblemCodes is the same shape for the same reason. **Tested on both sides, through the real pipeline.** The endpoint suite opens a genuine socket against TestServer and proves a push produces a notice, that another account's push does not reach it, that a ping is answered, and that a frame this server cannot parse does not end the connection. Two of those assert on *ordering* rather than on absence within a timeout — the stranger's write goes first, so a socket that leaked would have announced it before the one the test waits for — because "nothing arrived in two seconds" is a test that passes on a slow machine for the wrong reason. And ANoticeCarriesNoCiphertext asserts on the bytes that crossed the wire rather than on the record's fields, since the latter would only prove that this type has no payload member, which is a tautology; the former is what catches a field added later without anybody thinking about disclosure. The client suite drives VaultEventStream through an injected connector, because the one thing a test cannot do to a real network is make it fail on cue — and failure is the entire subject. The shell suite proves a notice produces a pull inside ten seconds against a sixty-second timer, so the timer cannot be what caused it. **Two limits, stated rather than left to be discovered.** Fan-out is in-process, so a deployment running more than one API replica only pushes for writes its own replica handled and the rest arrive on the timer. IVaultEventPublisher is the seam a PostgreSQL LISTEN/NOTIFY backplane implements and it is deliberately not implemented: an untested backplane is worse than a documented gap, and multiple replicas degrade to the behaviour before this commit rather than breaking. And a client is notified of its own writes; it pushed, so it already pulled, and the extra pass finds nothing. Suppressing that echo correctly needs a per-device identity on the socket, and the same user's other machines must still be told. Manual checks phase 15 covers what no test here can reach, which is the network in between: a proxy that will not upgrade, one that drops an idle socket without telling either end, a laptop lid, a token expiring. Every one of those is invisible inside a test host, and every check there passes only if the change arrives quickly *and* still arrives with the socket taken away. ADR 0012 also fixes one thing about the shared terminal session this is the transport for, so it need not be renegotiated later: session data will be binary frames on this same socket, because base64 in a JSON envelope is the wrong shape for the one payload here that is continuous rather than occasional. Two questions it explicitly does not answer by implication — whether those bytes go through the API at all, and what end-to-end encryption means when the second party watches a stream rather than holding a key — are ADR 0001 questions and get their own decision. 1512 tests pass. DodoSSH.SystemTests was not run — it needs the whole compose stack — so the end-to-end path is unverified for this change beyond what the manual checks describe.
This commit is contained in:
@@ -350,6 +350,42 @@ Both default to your personal vault and neither moves on its own, because an ite
|
||||
visible to everybody holding that vault's key. Choosing a vault in the host editor also decides which groups
|
||||
it can be filed under: a group is an item like any other and lives in exactly one vault.
|
||||
|
||||
### Changes that do not wait
|
||||
|
||||
A client holds a WebSocket open to the server — `GET /api/v1/events`, subprotocol
|
||||
`dodossh.events.v1` — and the server sends a line down it whenever something you can read has moved. The
|
||||
client's answer is the same delta pull it would have run on its timer, only now rather than in up to a
|
||||
minute. Two things you can see: an edit somebody else makes appears while you are looking at the list, and
|
||||
a vault shared with you turns up as soon as they share it.
|
||||
|
||||
**What is on that socket is a notice, not your data.** A frame says which vault changed and how far its
|
||||
change log has got, and nothing else: no item, no ciphertext, not even which item it was. That is the
|
||||
decision the rest of this rests on, and it is deliberate twice over — the server has nothing else it
|
||||
*could* send, and keeping it that way means there is still exactly one path that applies a change to your
|
||||
keychain, so the socket can be wrong or absent without anything being applied incorrectly.
|
||||
|
||||
**Polling is still there and is still what guarantees a pass.** The minute timer is unchanged. A network
|
||||
that eats WebSockets, a server with `Events:Enabled` off, an older server, a proxy that will not upgrade, a
|
||||
notice dropped because your machine was too slow to read it — every one of those leaves you with exactly
|
||||
what this product did before the socket existed. Nothing is reachable only this way, and nothing is
|
||||
supposed to become so.
|
||||
|
||||
Three limits are worth knowing rather than discovering:
|
||||
|
||||
- **One node.** Fan-out is in-process, so a deployment running more than one API replica only pushes for
|
||||
writes that its own replica handled. The rest arrive on the timer. The seam for a PostgreSQL
|
||||
`LISTEN`/`NOTIFY` backplane is in place and is not implemented, because an untested backplane would be
|
||||
worse than a documented gap.
|
||||
- **The socket does not outlive your access token.** It is closed at the token's expiry and the client
|
||||
reconnects with a fresh one, which is a gap you will not see. That, plus re-reading your vault list every
|
||||
few minutes, is what bounds how long a withdrawn grant can keep producing notices — and what it bounds is
|
||||
*metadata*, because reading a vault needs a key the server has never held.
|
||||
- **You are told about your own writes.** Your client pushed, so it has already pulled; the extra pass finds
|
||||
nothing. Notices are coalesced over a quarter of a second so that a burst is one pass rather than a dozen.
|
||||
|
||||
The reasoning, including why this is a WebSocket rather than server-sent events and where a shared terminal
|
||||
session will attach to it, is in [ADR 0012](docs/adr/0012-realtime-push.md).
|
||||
|
||||
### The Android head
|
||||
|
||||
`src/DodoSSH.Client.Android` is a phone-first head that shares every view model with the desktop one — the
|
||||
@@ -618,6 +654,20 @@ keychain plus a terminal — and the spike that gates all of it.
|
||||
directories, an interrupted **upload** starts again rather than resuming (an object cannot be written from
|
||||
the middle), and a rename is a copy and a delete rather than one atomic operation. Downloads do resume — a
|
||||
ranged GET is part of the protocol, which is the one place a bucket beats SFTP.
|
||||
|
||||
*Realtime done:* a WebSocket the client holds open, over which the server says which vault has moved so a
|
||||
pull happens now rather than within the minute. What crosses it is a notice and never an item, which is
|
||||
what keeps one code path applying changes and makes a dropped socket cost latency and nothing else — the
|
||||
timer is unchanged and is still the guarantee. Two limits are stated rather than implied: fan-out is
|
||||
in-process, so a multi-replica deployment falls back to the timer for writes another replica handled, and
|
||||
a socket is closed at its access token's expiry rather than outliving the credential that authorised it.
|
||||
See [Changes that do not wait](#changes-that-do-not-wait) and
|
||||
[ADR 0012](docs/adr/0012-realtime-push.md).
|
||||
|
||||
It is also the transport a **shared terminal session** will use — one person's shell, watched or driven by
|
||||
somebody else. Nothing of that exists yet, and ADR 0012 records the one decision made early so it need not
|
||||
be renegotiated: session data will be binary frames on this same socket, because base64 in a JSON envelope
|
||||
is the wrong shape for the one payload here that is continuous rather than occasional.
|
||||
- **M3 — shared vaults**, sharing, ACLs. *Done.* Membership with roles, a public-key directory, the
|
||||
append-only key log served for clients to verify against, shared vaults, and vault key grants
|
||||
wrapped by a client and stored opaquely by the server. `VaultAccessService` now resolves team
|
||||
@@ -641,10 +691,12 @@ keychain plus a terminal — and the spike that gates all of it.
|
||||
the rotation is re-sealed as it is pushed, so nothing reaches the server under a superseded key at all.
|
||||
See [ADR 0010](docs/adr/0010-vault-key-rotation.md).
|
||||
|
||||
**A vault shared with you arrives on the next synchronisation pass**, within the minute, with no sign-in
|
||||
and nothing to press. There is no push channel, so each pass asks the server which vaults this account can
|
||||
reach before syncing the ones it already knows — which is also how a vault that has been deleted, or one
|
||||
whose grant was withdrawn, stops being listed.
|
||||
**A vault shared with you arrives at once**, with no sign-in and nothing to press. Each pass asks the
|
||||
server which vaults this account can reach before syncing the ones it already knows — which is also how a
|
||||
vault that has been deleted, or one whose grant was withdrawn, stops being listed — and the server now
|
||||
says so the moment somebody wraps a key to you rather than leaving it for the next pass. Without a
|
||||
reachable socket that becomes "within the minute", which is what it always was; see
|
||||
[Changes that do not wait](#changes-that-do-not-wait).
|
||||
|
||||
**Ownership transfer is here, and it is one write rather than two.** The member you name becomes owner
|
||||
and you become an admin, in a single transaction — because ownership is sole, so promoting first leaves
|
||||
|
||||
@@ -58,7 +58,11 @@ the Npgsql connection string** — the default; do not enable multiplexing.
|
||||
- One place enforces revision, change-log and ACL invariants. That halves both the endpoint
|
||||
count and the authorization surface, which is the main reason for the single write path.
|
||||
- Delta pull makes frequent polling cheap, so multi-device feels live; push notification over
|
||||
SSE or the existing WebSocket can layer on with polling as the fallback.
|
||||
SSE or the existing WebSocket can layer on with polling as the fallback. **That has since been
|
||||
built — see [ADR 0012](0012-realtime-push.md)** — and nothing in this ADR changed to accommodate
|
||||
it. The socket carries a notice naming a vault and a sequence, whose answer is the delta pull
|
||||
above, so there is still exactly one path that applies a change; and polling is still what
|
||||
guarantees a pass rather than a legacy route kept for old clients.
|
||||
- Conflict resolution is entirely client-side. The client retains a `BaseCiphertext` common
|
||||
ancestor and performs a field-level three-way merge for structured items, or creates a
|
||||
visible conflicted copy for opaque ones. **It must never silently drop a key or a host.**
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
# ADR 0012 — A WebSocket that carries notices, not data
|
||||
|
||||
- Status: accepted
|
||||
- Date: 2026-08-04
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0003](0003-sync-protocol.md) built a delta pull that is cheap enough to run on a timer, and
|
||||
the client does: one pass a minute. That is the difference between a colleague's change appearing
|
||||
"soon" and appearing *now*, and it shows up in three places that are not equally forgivable.
|
||||
|
||||
- **A vault shared with you** arrives on the next pass. `AdmitNewVaultsAsync` says so in its own
|
||||
remarks — "the recipient is handed nothing — there is no push channel" — and the README repeats
|
||||
it. Sharing works and looks broken.
|
||||
- **Two people editing one keychain** see each other up to a minute late, which is long enough to
|
||||
make the same edit twice and produce a conflict that nobody needed to have.
|
||||
- **A revoked grant** keeps serving a client that has not noticed yet, for up to a pass.
|
||||
|
||||
Shortening the interval is the obvious answer and the wrong one: it costs a request per client per
|
||||
interval whether or not anything happened, and it does not converge on *immediate* — it converges
|
||||
on a busier server that is still late.
|
||||
|
||||
There is also a second thing coming that this decision has to not preclude. The intended feature is
|
||||
a **shared terminal session** — one person's shell, watched or driven by another, TeamViewer-shaped.
|
||||
That is bidirectional, continuous, and latency-sensitive in a way a keychain notice is not.
|
||||
|
||||
## Decision
|
||||
|
||||
### One WebSocket per signed-in client, at `GET /api/v1/events`
|
||||
|
||||
Subprotocol `dodossh.events.v1`. The client opens it after unlock and keeps it open; the server
|
||||
sends a notice whenever something the client can read has changed.
|
||||
|
||||
**Not SSE.** Server-sent events would carry today's notices perfectly well and would be less code.
|
||||
It is one-directional, so the shared-session feature would need a second mechanism next to it, and
|
||||
then two transports would need reconnection, authorization and lifetime rules that agree. The cost
|
||||
of a WebSocket over SSE is small; the cost of two transports is not.
|
||||
|
||||
**Not SignalR.** It brings hub protocol negotiation, its own serialisation and transport fallbacks,
|
||||
none of which are wanted here: `DodoSSH.Contracts` and its source-generated serialiser are "the
|
||||
actual contract between the two sides", and a second wire format alongside it is exactly the silent
|
||||
drift `Setup/Json.cs` records having already cost this project once.
|
||||
|
||||
### The notice carries no ciphertext
|
||||
|
||||
A `vault.changed` frame is `{ kind, vaultId, sequence }` and nothing else. The client's answer to it
|
||||
is the pull it would have done on the timer anyway.
|
||||
|
||||
This is the load-bearing decision, and it is worth being explicit about why the tempting alternative
|
||||
is refused. Pushing the changed items themselves would save a round trip and would fork the code
|
||||
path that applies a change into two — one that arrives by pull and one that arrives by socket — with
|
||||
the cursor, the merge and the tombstone rules duplicated across both. ADR 0003 put every mutation
|
||||
through one write path for exactly that reason; this keeps every *read* on one path for the same
|
||||
one. The socket decides *when* to sync. It never decides *what* a vault contains.
|
||||
|
||||
It also means a dropped notice is harmless, which is what lets everything below be simple.
|
||||
|
||||
### Polling stays, and is the fallback rather than a legacy path
|
||||
|
||||
The one-minute pass is unchanged. The socket makes it *early*; it does not make it *necessary*. A
|
||||
client on a network that eats WebSockets, an older client, a server that has the feature off, a
|
||||
notice dropped under backpressure, a second API replica that did not see the write — every one of
|
||||
those degrades to what the product does today, which is correct and up to a minute late.
|
||||
|
||||
Nothing may be reachable only by socket. That is a rule about future features, not an observation
|
||||
about this one.
|
||||
|
||||
### Authorization: the bearer token on the upgrade, not a ticket
|
||||
|
||||
[ADR 0004](0004-relay-authorization.md) gives the relay a two-step ticket so its WebSocket carries
|
||||
no API authority. This one goes the other way and takes the ordinary bearer JWT on the upgrade
|
||||
request, which is an ordinary authenticated HTTP request. The difference is not inconsistency:
|
||||
|
||||
- The relay's socket is a **byte pipe to a third party**, and its whole authorization decision —
|
||||
which host, which IPs, which port — is made *before* the socket opens and never revisited. It is
|
||||
also the extraction seam for a standalone relay process that must not hold ACL code.
|
||||
- This socket is a **view of the caller's own vault list**, and it has to keep answering "what may
|
||||
this account read" for as long as it is open. It needs the full ACL context, in-process, for the
|
||||
life of the connection. A ticket would carry that context in a token instead, and it would be
|
||||
wrong the moment the account's access changed.
|
||||
|
||||
A long-lived connection authorised by a short-lived token is the problem this creates, and it is met
|
||||
head-on rather than ignored:
|
||||
|
||||
1. **The socket does not outlive the token.** The `exp` claim is read at accept, and the connection
|
||||
is closed with `4401` when it passes. The client reconnects with a fresh token; that is a
|
||||
sub-second gap in a channel whose failure mode is already "poll instead".
|
||||
2. **The vault set is re-resolved periodically** (`Events:AccessRefreshInterval`, default five
|
||||
minutes) as well as on the changes that are known to affect it. A withdrawn grant therefore stops
|
||||
producing notices within that window at the latest, and immediately in the ordinary case.
|
||||
|
||||
Both are bounds on **metadata** — the fact that a vault changed and roughly when — because that is
|
||||
all a notice contains. Nobody's ciphertext is behind this socket, and a client that stayed subscribed
|
||||
one interval too long could still not read a byte of it: reading requires a vault key grant, which
|
||||
this server has never held.
|
||||
|
||||
### The frames
|
||||
|
||||
Text frames, JSON, `DodoSshJsonContext`. Server to client:
|
||||
|
||||
| kind | meaning |
|
||||
| --- | --- |
|
||||
| `hello` | accepted; carries the heartbeat interval and the vault count subscribed |
|
||||
| `vault.changed` | `vaultId` moved to `sequence`; pull it |
|
||||
| `vaults.changed` | the set of vaults this account can reach is different; re-read it |
|
||||
| `ping` | heartbeat; the client answers `pong` |
|
||||
|
||||
Client to server: `ping`, answered with `pong`. Nothing else — subscription is decided by the server
|
||||
from the caller's access, not asked for by the client, because a client that could ask to subscribe
|
||||
to a vault id is a client that can probe for vault ids.
|
||||
|
||||
`kind` is a **string**, not an enum, and that is deliberate. `UseStringEnumConverter` throws on a
|
||||
value it does not know, so a newer server sending a kind an older client has never heard of would
|
||||
not add an unknown frame — it would break that client's socket entirely. A string is ignored
|
||||
instead, which is what makes the table above extensible. `ProblemCodes` is the same shape for the
|
||||
same reason.
|
||||
|
||||
### Where the shared session will attach
|
||||
|
||||
The socket is the seam, and one thing about it is chosen now so that it need not be renegotiated
|
||||
later: **session data will be binary frames on this same connection, not JSON on the table above.**
|
||||
Terminal output base64'd into a JSON envelope would cost a third of the bandwidth for nothing, on
|
||||
the one payload here that is continuous rather than occasional. Control — offer, accept, resize,
|
||||
end — is JSON like everything else.
|
||||
|
||||
That is as far as this ADR goes. Two questions are open and are not being answered by implication:
|
||||
whether a shared session's bytes go through the API at all or peer-to-peer past it, and what
|
||||
end-to-end encryption means when the second party is watching a stream rather than holding a key.
|
||||
Both are ADR 0001 questions and deserve their own decision. What this one buys is that they will not
|
||||
also be transport questions.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Fan-out is in-process, and the deployment is therefore single-node for this feature.** Every
|
||||
connection is held by the node that accepted it; a write handled by another node produces no
|
||||
notice on this one. `IVaultEventPublisher` is the seam a backplane implements — PostgreSQL
|
||||
`LISTEN`/`NOTIFY` needs no infrastructure this stack does not already run — and it is deliberately
|
||||
**not implemented**, because an untested backplane is worse than a documented gap. Multiple API
|
||||
replicas do not break: they degrade to polling, which is the state before this ADR. `/api/v1/meta`
|
||||
advertises `events` so a client knows which it is getting.
|
||||
- **Per-connection queues are bounded and drop the oldest.** A notice is "pull vault X, which is at
|
||||
least at sequence N", so the newest is strictly more useful than the one it displaces and the
|
||||
client's answer is identical either way. A slow reader costs itself latency, never the publisher's
|
||||
progress — the publish path never blocks and never awaits a socket.
|
||||
- **The publish happens after the transaction commits**, outside the advisory lock ADR 0003 takes.
|
||||
A notice sent from inside it would name a sequence a reader cannot yet see, and would hold the
|
||||
per-vault write lock across a socket write.
|
||||
- **A client is notified of its own writes.** It pushed, so it already pulled; the extra pass finds
|
||||
nothing. The client coalesces notices over a short window rather than the server suppressing an
|
||||
echo, because suppressing it correctly needs a per-*device* identity on the socket and the same
|
||||
user's other machines must still be told.
|
||||
- **Connections are capped** per user and per node (`Events:MaxConnectionsPerUser`,
|
||||
`Events:MaxConnectionsTotal`). A socket is cheap but not free, and an unbounded count of them is a
|
||||
denial of service that authenticates first.
|
||||
- The feature can be turned off entirely (`Events:Enabled`). A deployment behind a proxy that will
|
||||
not upgrade should say so rather than have every client discover it by failing.
|
||||
|
||||
### Rejected
|
||||
|
||||
- **Shorter polling.** Cheaper to build, converges on a busier server that is still late.
|
||||
- **Long polling.** No new transport and genuinely immediate, but it holds a request thread and a
|
||||
connection per client for the same money as a WebSocket while offering none of the bidirectionality
|
||||
the shared session needs.
|
||||
- **Pushing the changed items down the socket.** Saves a round trip; forks the apply path in two. See
|
||||
above.
|
||||
- **Client-chosen subscriptions.** A `subscribe(vaultId)` frame is an existence oracle for vault ids,
|
||||
which is the disclosure `SyncPullEndpoint` answers 404 rather than 403 to avoid.
|
||||
@@ -1465,3 +1465,83 @@ delivery survives the failure because it is held against the transfer rather tha
|
||||
**Failure means:** a retry that succeeds but leaves the destination empty is the delivery having been
|
||||
dropped on the failure. An error saying the staged file is missing is the copy having been deleted at the
|
||||
stop, which is what `QueueDeliveredDownload` documents it does not do.
|
||||
|
||||
## Phase 15 — Changes that arrive without a timer
|
||||
|
||||
The socket is covered by tests on both sides: the endpoint suite opens a real one against a real
|
||||
`TestServer` and proves a push produces a notice, that another account's push does not, and that a frame
|
||||
carries no ciphertext; the shell suite proves a notice wakes the synchronisation loop long before the
|
||||
minute. What none of that can reach is **the network in between**, and that is where this feature is most
|
||||
likely to fail: a reverse proxy that will not upgrade, one that drops an idle socket without telling either
|
||||
end, a corporate middlebox, a phone moving between Wi-Fi and mobile data. Every one of those looks the same
|
||||
from inside a test host, which has no proxy and no radio.
|
||||
|
||||
The pass condition throughout is *two* things, and the second matters as much as the first: it arrives
|
||||
quickly, **and** it still arrives when the socket is gone. A build where the timer had stopped working would
|
||||
pass every "it was fast" check here and fail nobody until somebody's proxy changed.
|
||||
|
||||
### 15.1 A colleague's edit appears while you are looking at it
|
||||
|
||||
Two accounts sharing a vault, both unlocked, both on the Hosts screen. On the first machine, rename a host
|
||||
in the shared vault and save.
|
||||
|
||||
**Pass:** the second machine's list shows the new name within a second or two, with nothing pressed and no
|
||||
screen flicker — the row updates, the selection does not move, and the status line is not repainted with a
|
||||
sync report.
|
||||
|
||||
**Failure means:** nothing within a minute, then the new name, is the socket not being established at all —
|
||||
that is the timer doing its job, which is the correct fallback and not the feature. Check `/api/v1/meta`
|
||||
lists `events`, then whether the proxy in front of the API forwards `Upgrade` and `Connection`. A list that
|
||||
never updates at all is a synchronisation failure and has nothing to do with this phase.
|
||||
|
||||
### 15.2 A vault shared with you turns up as it is shared
|
||||
|
||||
The second account signed in and unlocked, sitting on the VAULTS screen. From the first, add them to a team
|
||||
and press SHARE KEY.
|
||||
|
||||
**Pass:** the vault appears in their list within a second or two of the key being wrapped, and reads as
|
||||
waiting for a key until the share, then as readable.
|
||||
|
||||
**Failure means:** the vault appearing only on the minute is the `vaults.changed` notice not being published
|
||||
or not being followed. Both the membership add and the grant publish one; if the membership arrives promptly
|
||||
and the key does not, the grant path is the one to look at.
|
||||
|
||||
### 15.3 It still works with the socket taken away
|
||||
|
||||
On the second machine, block the WebSocket — the simplest way is a proxy rule rejecting the upgrade, or
|
||||
setting `Events:Enabled` to `false` on the server and restarting it.
|
||||
|
||||
**Pass:** everything above still happens, within the minute rather than within seconds. Nothing on the
|
||||
screen says anything is wrong, because nothing is: no error, no OFFLINE badge, no repeated status message.
|
||||
The Sync button still works and still reports.
|
||||
|
||||
**Failure means:** an error message, a titlebar claiming to be offline, or a status line that repaints with
|
||||
a socket failure is the client treating an absent push channel as a fault. It is not one — the timer is the
|
||||
guarantee and the socket is the optimisation, and a user with a strict proxy must never be told their
|
||||
keychain is broken.
|
||||
|
||||
### 15.4 A laptop that slept comes back on its own
|
||||
|
||||
With the second machine idle and connected, close the lid for a few minutes — or disable Wi-Fi for two
|
||||
minutes and re-enable it. Then make a change on the first machine.
|
||||
|
||||
**Pass:** the change arrives quickly again, without the vault having been locked or the application
|
||||
restarted. The reconnection is invisible.
|
||||
|
||||
**Failure means:** changes that arrive only on the timer from then on are the stream having given up after
|
||||
its socket died — the reconnection loop is what should make that impossible, and a client that reconnects
|
||||
once and not twice is the specific defect its tests exist to catch. Changes that never arrive again, timer
|
||||
included, are a different and worse bug in the synchronisation loop rather than in the socket.
|
||||
|
||||
### 15.5 An expiring token does not end the push
|
||||
|
||||
This one needs a short access-token lifetime in the identity provider — the dev realm's Keycloak client can
|
||||
be set to a couple of minutes. Leave a machine unlocked and idle for longer than that, then make a change
|
||||
elsewhere.
|
||||
|
||||
**Pass:** the change still arrives quickly. The socket is closed by the server at the token's expiry and the
|
||||
client reconnects with a fresh one, which should be invisible.
|
||||
|
||||
**Failure means:** notices stopping at roughly the token's lifetime is the reconnection not asking for a new
|
||||
token — it would be dialling with the spent one and being closed again immediately. A burst of reconnection
|
||||
attempts in the server log is the same defect seen from the other end.
|
||||
|
||||
@@ -0,0 +1,533 @@
|
||||
using System.Collections.Frozen;
|
||||
using System.Globalization;
|
||||
using System.Net.WebSockets;
|
||||
using System.Security.Claims;
|
||||
using System.Text.Json;
|
||||
using DodoSSH.Api.Authorization;
|
||||
using DodoSSH.Api.Setup;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain.Authorization;
|
||||
using FastEndpoints;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace DodoSSH.Api.Features.Events;
|
||||
|
||||
/// <summary>
|
||||
/// The socket that says "pull now" so a client does not have to wait for its timer.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Everything this endpoint sends is a <em>notice</em>. It never carries an item, a payload or a
|
||||
/// cursor: the client's answer to a notice is the delta pull it would have run on its own anyway, so
|
||||
/// there is exactly one code path that applies a change and this is not it. See ADR 0012 for why
|
||||
/// pushing the items themselves is refused.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The bearer token authorises the upgrade, unlike the relay's ticket in ADR 0004. The relay's socket
|
||||
/// is a byte pipe whose whole authorization decision is made before it opens; this one is a view of
|
||||
/// the caller's own vault list and has to keep answering "what may this account read" for as long as
|
||||
/// it is held. Its two bounds on that — the token's own expiry, and a periodic re-resolve — are in
|
||||
/// <see cref="MindAsync"/>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal sealed class VaultEventsEndpoint(
|
||||
IServiceScopeFactory scopes,
|
||||
VaultEventHub hub,
|
||||
IOptions<EventsOptions> options,
|
||||
TimeProvider clock,
|
||||
IHostApplicationLifetime lifetime,
|
||||
ILogger<VaultEventsEndpoint> logger)
|
||||
: EndpointWithoutRequest
|
||||
{
|
||||
/// <summary>
|
||||
/// The largest message this endpoint will read from a client.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A client sends nothing but <c>ping</c>, so the cap is three orders of magnitude of headroom and
|
||||
/// still small enough that a hostile client cannot make the server buffer anything worth having.
|
||||
/// </remarks>
|
||||
private const int MaxInboundFrameBytes = 4 * 1024;
|
||||
|
||||
/// <summary>How long to wait for the close handshake before dropping the socket.</summary>
|
||||
private static readonly TimeSpan CloseTimeout = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <inheritdoc />
|
||||
public override void Configure()
|
||||
{
|
||||
Get(VaultEvents.Path);
|
||||
|
||||
// Enrolled, matching sync. A caller with no identity key holds no vault key either, so every
|
||||
// notice this socket could send is about ciphertext they cannot read.
|
||||
Policies(Auth.EnrolledPolicy);
|
||||
|
||||
Description(b => b
|
||||
.WithName("VaultEvents")
|
||||
.WithSummary("Pushes a notice when a vault the caller can read has changed.")
|
||||
.WithTags("Events"));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task HandleAsync(CancellationToken ct)
|
||||
{
|
||||
if (await RefusedAsync().ConfigureAwait(false))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var (userId, vaults) = await ResolveAccessAsync(ct).ConfigureAwait(false);
|
||||
|
||||
// Admitted before the upgrade so a refusal costs nothing, but answered *through* the socket
|
||||
// rather than as an HTTP status: a constrained WebSocket client cannot read the status of a
|
||||
// failed upgrade, and "you have too many open" is precisely the case where the client needs
|
||||
// to know to back off rather than retry. Same reasoning as ADR 0004 on request headers.
|
||||
var connection = hub.TryAdmit(userId, vaults);
|
||||
|
||||
try
|
||||
{
|
||||
using var socket = await HttpContext.WebSockets
|
||||
.AcceptWebSocketAsync(new WebSocketAcceptContext { SubProtocol = VaultEvents.SubProtocol })
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (connection is null)
|
||||
{
|
||||
await CloseAsync(
|
||||
socket,
|
||||
new Closure(
|
||||
VaultEvents.TooManyConnectionsCloseCode, "Too many open event sockets."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
await PumpAsync(socket, connection, ct).ConfigureAwait(false);
|
||||
}
|
||||
finally
|
||||
{
|
||||
// Inside a try that starts *before* the upgrade, because an accept that throws — a client
|
||||
// that abandoned the handshake — would otherwise leave an admitted connection in the hub
|
||||
// for the life of the process, counting against this account's cap and taking a slot from
|
||||
// the sockets that did open.
|
||||
if (connection is not null)
|
||||
{
|
||||
hub.Remove(connection);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Answers the requests that are not an event socket at all, as ordinary HTTP.
|
||||
/// </summary>
|
||||
/// <returns>Whether a response was sent and the handler should stop.</returns>
|
||||
/// <remarks>
|
||||
/// All three answers are problem documents rather than bare statuses, because each one has a
|
||||
/// different remedy and a client that cannot tell them apart would retry the two that will never
|
||||
/// succeed. Answered before the upgrade, so a caller that got the handshake wrong reads why in a
|
||||
/// body rather than inferring it from a socket that closed.
|
||||
/// </remarks>
|
||||
private async Task<bool> RefusedAsync()
|
||||
{
|
||||
if (!options.Value.Enabled)
|
||||
{
|
||||
// 404 rather than 501: the feature is absent from this deployment, and /api/v1/meta does
|
||||
// not advertise it. A client that dialled anyway keeps polling, which is correct.
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status404NotFound,
|
||||
ProblemCodes.EventsUnavailable,
|
||||
"This server does not push vault changes. Synchronise on a timer instead; "
|
||||
+ "GET /api/v1/meta lists the features it does offer."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!HttpContext.WebSockets.IsWebSocketRequest)
|
||||
{
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status400BadRequest,
|
||||
ProblemCodes.MalformedRequest,
|
||||
"This endpoint is a WebSocket. Send an upgrade request offering the "
|
||||
+ $"'{VaultEvents.SubProtocol}' subprotocol."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
// The subprotocol is this API's version negotiation for the socket, so an upgrade that does
|
||||
// not offer it is refused rather than accepted and answered in a dialect the caller may not
|
||||
// read. See VaultEvents.SubProtocol.
|
||||
if (!HttpContext.WebSockets.WebSocketRequestedProtocols
|
||||
.Contains(VaultEvents.SubProtocol, StringComparer.Ordinal))
|
||||
{
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status400BadRequest,
|
||||
ProblemCodes.MalformedRequest,
|
||||
$"This server speaks '{VaultEvents.SubProtocol}', which the upgrade request did "
|
||||
+ "not offer."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads who the caller is and which vaults they may follow, in a scope of its own.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A fresh scope, disposed at once, rather than services injected into this endpoint — which is
|
||||
/// the lesson ADR 0004 records paying for on the relay. This handler runs for as long as the
|
||||
/// socket is open, so anything scoped it held would be a <c>DbContext</c> alive for hours, and a
|
||||
/// few hundred of those exhaust the connection pool. The database is touched here and in
|
||||
/// <see cref="RefreshAsync"/>, briefly, and nowhere else.
|
||||
/// </remarks>
|
||||
private async Task<(Guid UserId, FrozenSet<Guid> Vaults)> ResolveAccessAsync(
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var scope = scopes.CreateAsyncScope();
|
||||
await using var _ = scope.ConfigureAwait(false);
|
||||
|
||||
var currentUser = scope.ServiceProvider.GetRequiredService<ICurrentUserContext>();
|
||||
var vaultAccess = scope.ServiceProvider.GetRequiredService<IVaultAccessService>();
|
||||
|
||||
var user = await currentUser.GetOrProvisionAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return (user.Id, await ReachableAsync(vaultAccess, user.Id, cancellationToken).ConfigureAwait(false));
|
||||
}
|
||||
|
||||
private static async Task<FrozenSet<Guid>> ReachableAsync(
|
||||
IVaultAccessService vaultAccess,
|
||||
Guid userId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var accessible = await vaultAccess.ListAsync(userId, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return accessible
|
||||
.Where(access => access.Vault is not null
|
||||
&& access.Permissions.HasFlag(PermissionFlags.Read))
|
||||
.Select(access => access.Vault!.Id)
|
||||
.ToFrozenSet();
|
||||
}
|
||||
|
||||
/// <summary>Runs the socket until something ends it, then closes it politely.</summary>
|
||||
/// <remarks>
|
||||
/// Three loops rather than one: reading a socket and writing to it are independent waits, and the
|
||||
/// clock is a third. They are joined by <see cref="Task.WhenAny(Task[])"/> and then <em>all</em>
|
||||
/// awaited before the close is written, because a close frame racing a notice frame is a protocol
|
||||
/// violation that presents as a client dropping its connection for no visible reason.
|
||||
/// </remarks>
|
||||
private async Task PumpAsync(
|
||||
WebSocket socket,
|
||||
VaultEventConnection connection,
|
||||
CancellationToken requestAborted)
|
||||
{
|
||||
var settings = options.Value;
|
||||
|
||||
using var pump = CancellationTokenSource.CreateLinkedTokenSource(
|
||||
requestAborted, lifetime.ApplicationStopping);
|
||||
|
||||
var closure = new Closure(
|
||||
(int)WebSocketCloseStatus.NormalClosure, string.Empty);
|
||||
|
||||
connection.TryEnqueue(new VaultEvent(
|
||||
VaultEventKinds.Hello,
|
||||
ServerTime: clock.GetUtcNow(),
|
||||
HeartbeatSeconds: (int)settings.HeartbeatInterval.TotalSeconds,
|
||||
VaultCount: connection.VaultCount));
|
||||
|
||||
var sending = SendAsync(socket, connection, pump.Token);
|
||||
var receiving = ReceiveAsync(socket, connection, pump.Token);
|
||||
var minding = MindAsync(connection, closure, pump.Token);
|
||||
|
||||
await Task.WhenAny(sending, receiving, minding).ConfigureAwait(false);
|
||||
|
||||
await pump.CancelAsync().ConfigureAwait(false);
|
||||
|
||||
// Nothing may still be mid-send when the close frame goes out.
|
||||
await Task.WhenAll(Settled(sending), Settled(receiving), Settled(minding)).ConfigureAwait(false);
|
||||
|
||||
if (lifetime.ApplicationStopping.IsCancellationRequested)
|
||||
{
|
||||
// 1001 "going away", so a client knows to reconnect immediately rather than treating a
|
||||
// rolling deployment as a server that has broken.
|
||||
closure.Set((int)WebSocketCloseStatus.EndpointUnavailable, "The server is shutting down.");
|
||||
}
|
||||
|
||||
EventsLog.ClosingConnection(logger, connection.UserId, closure.Reason);
|
||||
|
||||
await CloseAsync(socket, closure).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes queued frames to the socket, one at a time.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The <em>only</em> writer, which is what makes concurrent sends impossible without a lock: the
|
||||
/// heartbeat and the pong both go into the same queue rather than to the socket. A WebSocket
|
||||
/// permits one send at a time and faults permanently on a second, so this is not a tidiness
|
||||
/// preference.
|
||||
/// </remarks>
|
||||
private async Task SendAsync(
|
||||
WebSocket socket,
|
||||
VaultEventConnection connection,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
await foreach (var frame in connection.Outbound
|
||||
.ReadAllAsync(cancellationToken)
|
||||
.ConfigureAwait(false))
|
||||
{
|
||||
// Re-resolved before the notice is forwarded, not after: the client's answer to this frame
|
||||
// is to re-read its vault list, and the point of a newly shared vault is that the *next*
|
||||
// change to it produces a notice too. A socket that forwarded first would not follow the
|
||||
// new vault until its next periodic refresh.
|
||||
if (string.Equals(frame.Kind, VaultEventKinds.VaultsChanged, StringComparison.Ordinal))
|
||||
{
|
||||
await RefreshAsync(connection, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
var bytes = JsonSerializer.SerializeToUtf8Bytes(frame, DodoSshJsonContext.Default.VaultEvent);
|
||||
|
||||
await socket
|
||||
.SendAsync(bytes, WebSocketMessageType.Text, endOfMessage: true, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads what the client sends, which in this version is heartbeats and a close.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A client cannot ask to follow a vault, and that is deliberate rather than unfinished: a
|
||||
/// <c>subscribe(vaultId)</c> frame is an existence oracle for vault ids, which is the disclosure
|
||||
/// <c>SyncPullEndpoint</c> answers 404 rather than 403 to avoid. Subscription is decided from the
|
||||
/// caller's access and nothing else.
|
||||
/// </remarks>
|
||||
private static async Task ReceiveAsync(
|
||||
WebSocket socket,
|
||||
VaultEventConnection connection,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var buffer = new byte[MaxInboundFrameBytes];
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var received = await socket.ReceiveAsync(buffer, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (received.MessageType == WebSocketMessageType.Close)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Oversized, or split across frames. Nothing this protocol sends is either, so the client
|
||||
// is broken or probing; ending the socket is cheaper than reassembling for it.
|
||||
if (!received.EndOfMessage)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Binary is unused in v1 and skipped rather than refused, because ADR 0012 reserves it for
|
||||
// shared-session data — an older server meeting a newer client must ignore those, not
|
||||
// close on them.
|
||||
if (received.MessageType != WebSocketMessageType.Text)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (Kind(buffer.AsSpan(0, received.Count)) is VaultEventKinds.Ping)
|
||||
{
|
||||
connection.TryEnqueue(new VaultEvent(VaultEventKinds.Pong));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads a frame's kind, or null if it is not one this server understands.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A frame that will not parse is skipped rather than closing the socket. This is a control
|
||||
/// channel whose failure mode is "the client polls instead", so tolerating a frame from a newer
|
||||
/// client costs nothing and refusing one costs that client its push for the whole session.
|
||||
/// </remarks>
|
||||
private static string? Kind(ReadOnlySpan<byte> utf8)
|
||||
{
|
||||
try
|
||||
{
|
||||
return JsonSerializer.Deserialize(utf8, DodoSshJsonContext.Default.VaultEvent)?.Kind;
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Keeps the heartbeat going, the vault set current, and the socket inside its token's lifetime.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The deadline is the earlier of the access token's <c>exp</c> and a hard cap on how long any one
|
||||
/// socket may live. Closing on expiry is what keeps a long-lived connection from outliving the
|
||||
/// short-lived credential that authorised it; the client answers by reconnecting with a fresh
|
||||
/// token, which is a sub-second gap in a channel that degrades to polling anyway.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The wait is the shorter of the heartbeat and the time left, so the deadline is met to within a
|
||||
/// tick rather than to within a heartbeat.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private async Task MindAsync(
|
||||
VaultEventConnection connection,
|
||||
Closure closure,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var settings = options.Value;
|
||||
var started = clock.GetUtcNow();
|
||||
|
||||
var deadline = TokenExpiry() is { } expiry && expiry < started + settings.MaxConnectionDuration
|
||||
? (Expiry: expiry, ForToken: true)
|
||||
: (Expiry: started + settings.MaxConnectionDuration, ForToken: false);
|
||||
|
||||
var refreshed = started;
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var now = clock.GetUtcNow();
|
||||
var remaining = deadline.Expiry - now;
|
||||
|
||||
if (remaining <= TimeSpan.Zero)
|
||||
{
|
||||
closure.Set(
|
||||
deadline.ForToken
|
||||
? VaultEvents.TokenExpiredCloseCode
|
||||
: (int)WebSocketCloseStatus.NormalClosure,
|
||||
deadline.ForToken
|
||||
? "The access token has expired. Reconnect with a fresh one."
|
||||
: "This connection reached its maximum lifetime.");
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
var wait = settings.HeartbeatInterval < remaining ? settings.HeartbeatInterval : remaining;
|
||||
|
||||
await Task.Delay(wait, clock, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
now = clock.GetUtcNow();
|
||||
|
||||
if (now - refreshed >= settings.AccessRefreshInterval)
|
||||
{
|
||||
// The backstop for a grant withdrawn while this socket was open. What it bounds is
|
||||
// metadata — that a vault changed — because that is all a notice carries and reading
|
||||
// the vault still needs a key this server has never held. See ADR 0012.
|
||||
await RefreshAsync(connection, cancellationToken).ConfigureAwait(false);
|
||||
refreshed = now;
|
||||
}
|
||||
|
||||
connection.TryEnqueue(new VaultEvent(VaultEventKinds.Ping, ServerTime: now));
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Re-reads which vaults this socket may follow.</summary>
|
||||
/// <remarks>
|
||||
/// A failure is logged and swallowed. The alternative is dropping a working socket because one
|
||||
/// database call timed out, which would trade an occasionally stale vault set for an outage.
|
||||
/// </remarks>
|
||||
private async Task RefreshAsync(VaultEventConnection connection, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
var scope = scopes.CreateAsyncScope();
|
||||
await using var _ = scope.ConfigureAwait(false);
|
||||
|
||||
var vaultAccess = scope.ServiceProvider.GetRequiredService<IVaultAccessService>();
|
||||
|
||||
connection.Resubscribe(
|
||||
await ReachableAsync(vaultAccess, connection.UserId, cancellationToken)
|
||||
.ConfigureAwait(false));
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// The socket is closing.
|
||||
}
|
||||
catch (Exception exception) when (exception is not OutOfMemoryException)
|
||||
{
|
||||
EventsLog.AccessRefreshFailed(logger, connection.UserId, exception);
|
||||
}
|
||||
}
|
||||
|
||||
/// <remarks>
|
||||
/// <c>MapInboundClaims</c> is off — see <see cref="Auth"/> — so the claim is spelled as the
|
||||
/// provider issued it rather than as a WS-Federation URI. Null is treated as "no bound from the
|
||||
/// token", which the bearer handler's <c>RequireExpirationTime</c> should make unreachable; the
|
||||
/// lifetime cap covers it either way.
|
||||
/// </remarks>
|
||||
private DateTimeOffset? TokenExpiry() =>
|
||||
long.TryParse(
|
||||
HttpContext.User.FindFirstValue("exp"),
|
||||
NumberStyles.Integer,
|
||||
CultureInfo.InvariantCulture,
|
||||
out var seconds)
|
||||
? DateTimeOffset.FromUnixTimeSeconds(seconds)
|
||||
: null;
|
||||
|
||||
private static async Task CloseAsync(WebSocket socket, Closure closure)
|
||||
{
|
||||
if (socket.State is not (WebSocketState.Open or WebSocketState.CloseReceived))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
using var timeout = new CancellationTokenSource(CloseTimeout);
|
||||
|
||||
try
|
||||
{
|
||||
await socket
|
||||
.CloseOutputAsync((WebSocketCloseStatus)closure.Code, closure.Reason, timeout.Token)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception)
|
||||
when (exception is OperationCanceledException or WebSocketException or ObjectDisposedException)
|
||||
{
|
||||
// The peer is already gone. There is nothing to tell it and nothing to recover.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Awaits a pump loop, treating its cancellation and its socket faults as the ordinary end.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Every one of these loops ends by being cancelled or by the socket going away, so an exception
|
||||
/// here is the expected shape of "this connection is over" rather than a fault to propagate — and
|
||||
/// propagating it would skip the close frame the other side is waiting for.
|
||||
/// </remarks>
|
||||
private static async Task Settled(Task loop)
|
||||
{
|
||||
try
|
||||
{
|
||||
await loop.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception)
|
||||
when (exception is OperationCanceledException or WebSocketException or ObjectDisposedException)
|
||||
{
|
||||
// Expected.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Why the socket is being closed, decided by whichever loop ended first.</summary>
|
||||
/// <remarks>
|
||||
/// Mutable and shared, and safe without a lock for one specific reason: it is written by the pump
|
||||
/// loops and read only after <see cref="Task.WhenAll(Task[])"/> over all of them, which is a
|
||||
/// memory barrier. Writes race only with each other, and any of them is a true answer.
|
||||
/// </remarks>
|
||||
private sealed class Closure(int code, string reason)
|
||||
{
|
||||
internal int Code { get; private set; } = code;
|
||||
|
||||
internal string Reason { get; private set; } = reason;
|
||||
|
||||
internal void Set(int code, string reason)
|
||||
{
|
||||
Code = code;
|
||||
Reason = reason;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
namespace DodoSSH.Api.Features.Events;
|
||||
|
||||
/// <summary>Source-generated log messages for the event socket.</summary>
|
||||
/// <remarks>
|
||||
/// Ids and counts only, as everywhere else. A notice carries no ciphertext to leak, but which vault
|
||||
/// changed and when is still the metadata ADR 0001 asks be kept to what is diagnostically useful.
|
||||
/// </remarks>
|
||||
internal static partial class EventsLog
|
||||
{
|
||||
[LoggerMessage(
|
||||
EventId = 2201,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Event socket opened for user {UserId} following {VaultCount} vault(s); "
|
||||
+ "{ConnectionCount} open on this node.")]
|
||||
internal static partial void ConnectionOpened(
|
||||
ILogger logger,
|
||||
Guid userId,
|
||||
int vaultCount,
|
||||
int connectionCount);
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 2202,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Event socket closed for user {UserId}; {ConnectionCount} open on this node.")]
|
||||
internal static partial void ConnectionClosed(ILogger logger, Guid userId, int connectionCount);
|
||||
|
||||
/// <remarks>
|
||||
/// Information rather than Debug: a refused socket is a client that will poll for the rest of its
|
||||
/// session, and an operator seeing these has a cap to raise.
|
||||
/// </remarks>
|
||||
[LoggerMessage(
|
||||
EventId = 2203,
|
||||
Level = LogLevel.Information,
|
||||
Message = "Refused an event socket for user {UserId}: {Limit} is already reached.")]
|
||||
internal static partial void ConnectionRefused(ILogger logger, Guid userId, string limit);
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 2204,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Announced vault {VaultId} at sequence {Sequence} to {ConnectionCount} socket(s).")]
|
||||
internal static partial void VaultChangePublished(
|
||||
ILogger logger,
|
||||
Guid vaultId,
|
||||
long sequence,
|
||||
int connectionCount);
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 2205,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Announced a vault access change to {ConnectionCount} socket(s) of user {UserId}.")]
|
||||
internal static partial void AccessChangePublished(ILogger logger, Guid userId, int connectionCount);
|
||||
|
||||
/// <remarks>
|
||||
/// Warning, and it is worth being loud: the socket is still open and still delivering, but it is
|
||||
/// delivering about a vault set that may be stale. Everything else here is routine.
|
||||
/// </remarks>
|
||||
[LoggerMessage(
|
||||
EventId = 2206,
|
||||
Level = LogLevel.Warning,
|
||||
Message = "Could not re-resolve which vaults user {UserId}'s event socket may follow.")]
|
||||
internal static partial void AccessRefreshFailed(ILogger logger, Guid userId, Exception exception);
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 2207,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Closing user {UserId}'s event socket: {Reason}.")]
|
||||
internal static partial void ClosingConnection(ILogger logger, Guid userId, string reason);
|
||||
}
|
||||
@@ -0,0 +1,242 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Collections.Frozen;
|
||||
using System.Threading.Channels;
|
||||
using DodoSSH.Api.Setup;
|
||||
using DodoSSH.Contracts;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace DodoSSH.Api.Features.Events;
|
||||
|
||||
/// <summary>
|
||||
/// Tells connected clients that something they can read has moved.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Every method is <c>void</c> and returns having queued, never having sent. That is the contract, not
|
||||
/// an implementation detail: the callers are write paths that have just committed a transaction, and a
|
||||
/// publish that could block on a slow socket would make one client's bad network everybody else's
|
||||
/// latency. A notice that cannot be queued is dropped, which is safe because the client polls anyway.
|
||||
/// See ADR 0012.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// An interface because this is the seam a multi-node backplane implements — PostgreSQL
|
||||
/// <c>LISTEN</c>/<c>NOTIFY</c> is the obvious one and needs no infrastructure this stack does not
|
||||
/// already run. It is deliberately not implemented: fan-out today is in-process, so a deployment with
|
||||
/// more than one API replica notices writes handled by other replicas on the polling interval rather
|
||||
/// than at once. That is the behaviour before this feature existed, which is why it degrades rather
|
||||
/// than breaks.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public interface IVaultEventPublisher
|
||||
{
|
||||
/// <summary>Announces that a vault's change log has reached <paramref name="sequence"/>.</summary>
|
||||
/// <remarks>
|
||||
/// Call <em>after</em> the transaction commits, and outside the per-vault advisory lock ADR 0003
|
||||
/// takes. A notice sent from inside names a sequence no reader can see yet, and holds the vault's
|
||||
/// write lock across a socket write.
|
||||
/// </remarks>
|
||||
void VaultChanged(Guid vaultId, long sequence);
|
||||
|
||||
/// <summary>Announces that the set of vaults an account can reach is no longer what it was.</summary>
|
||||
/// <remarks>
|
||||
/// Takes the <em>recipient</em>, not the actor. Sharing is something one account does to another's
|
||||
/// list, and it is the other account that has to re-read.
|
||||
/// </remarks>
|
||||
void VaultAccessChanged(Guid userId);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Every event socket this node is holding.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Publishing walks the whole connection list and asks each one whether it cares, rather than keeping
|
||||
/// an index from vault to subscribers. With a per-node connection cap in the hundreds and an event rate
|
||||
/// bounded by how often people edit keychains, the walk is not measurable — and the index is not free:
|
||||
/// a connection's vault set is re-resolved while it is live, so every re-subscription would have to
|
||||
/// move it between buckets under a lock that publishing also takes. The simpler shape is the one whose
|
||||
/// races are obvious.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A singleton, holding no scoped service and no database context. Connections outlive requests by
|
||||
/// design and anything request-scoped they captured would outlive its scope with them.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal sealed class VaultEventHub(
|
||||
IOptions<EventsOptions> options,
|
||||
TimeProvider clock,
|
||||
ILogger<VaultEventHub> logger) : IVaultEventPublisher
|
||||
{
|
||||
private readonly ConcurrentDictionary<Guid, VaultEventConnection> connections = new();
|
||||
|
||||
/// <summary>
|
||||
/// Serialises admission so the caps are caps rather than approximations.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Counting and inserting under one lock, because the two done separately let N simultaneous
|
||||
/// connects all read the same under-cap count and all insert. Contended only by connects, which
|
||||
/// happen once per client per session; publishing never takes it.
|
||||
/// </remarks>
|
||||
private readonly Lock admission = new();
|
||||
|
||||
/// <summary>How many sockets this node is holding. Diagnostics and tests.</summary>
|
||||
internal int Count => connections.Count;
|
||||
|
||||
/// <summary>
|
||||
/// Admits a socket, or refuses it because a cap is already met.
|
||||
/// </summary>
|
||||
/// <returns>The connection, or null when a limit refused it.</returns>
|
||||
internal VaultEventConnection? TryAdmit(Guid userId, FrozenSet<Guid> vaults)
|
||||
{
|
||||
var limits = options.Value;
|
||||
|
||||
lock (admission)
|
||||
{
|
||||
if (connections.Count >= limits.MaxConnectionsTotal)
|
||||
{
|
||||
EventsLog.ConnectionRefused(logger, userId, "the node limit");
|
||||
return null;
|
||||
}
|
||||
|
||||
var held = 0;
|
||||
foreach (var existing in connections.Values)
|
||||
{
|
||||
if (existing.UserId == userId && ++held >= limits.MaxConnectionsPerUser)
|
||||
{
|
||||
EventsLog.ConnectionRefused(logger, userId, "the per-account limit");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
var connection = new VaultEventConnection(userId, vaults, limits.OutboundQueueDepth);
|
||||
|
||||
// Cannot collide: the id is fresh and this is the only insert.
|
||||
connections[connection.Id] = connection;
|
||||
|
||||
EventsLog.ConnectionOpened(logger, userId, vaults.Count, connections.Count);
|
||||
|
||||
return connection;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Forgets a socket that has closed.</summary>
|
||||
internal void Remove(VaultEventConnection connection)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(connection);
|
||||
|
||||
connections.TryRemove(connection.Id, out _);
|
||||
connection.Complete();
|
||||
|
||||
EventsLog.ConnectionClosed(logger, connection.UserId, connections.Count);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void VaultChanged(Guid vaultId, long sequence)
|
||||
{
|
||||
var notice = new VaultEvent(
|
||||
VaultEventKinds.VaultChanged,
|
||||
VaultId: vaultId,
|
||||
Sequence: sequence,
|
||||
ServerTime: clock.GetUtcNow());
|
||||
|
||||
var delivered = 0;
|
||||
|
||||
foreach (var connection in connections.Values)
|
||||
{
|
||||
if (connection.IsSubscribedTo(vaultId) && connection.TryEnqueue(notice))
|
||||
{
|
||||
delivered++;
|
||||
}
|
||||
}
|
||||
|
||||
if (delivered > 0)
|
||||
{
|
||||
EventsLog.VaultChangePublished(logger, vaultId, sequence, delivered);
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void VaultAccessChanged(Guid userId)
|
||||
{
|
||||
var notice = new VaultEvent(VaultEventKinds.VaultsChanged, ServerTime: clock.GetUtcNow());
|
||||
|
||||
var delivered = 0;
|
||||
|
||||
foreach (var connection in connections.Values)
|
||||
{
|
||||
if (connection.UserId == userId && connection.TryEnqueue(notice))
|
||||
{
|
||||
delivered++;
|
||||
}
|
||||
}
|
||||
|
||||
if (delivered > 0)
|
||||
{
|
||||
EventsLog.AccessChangePublished(logger, userId, delivered);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One open socket, as the hub sees it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Deliberately knows nothing about WebSockets. The hub queues frames here and the endpoint's pump
|
||||
/// takes them away, which is what keeps a publish from ever touching a socket — and what lets the
|
||||
/// whole fan-out be tested without one.
|
||||
/// </remarks>
|
||||
internal sealed class VaultEventConnection
|
||||
{
|
||||
private readonly Channel<VaultEvent> outbound;
|
||||
|
||||
private FrozenSet<Guid> vaults;
|
||||
|
||||
internal VaultEventConnection(Guid userId, FrozenSet<Guid> vaults, int queueDepth)
|
||||
{
|
||||
Id = Guid.CreateVersion7();
|
||||
UserId = userId;
|
||||
this.vaults = vaults;
|
||||
|
||||
// DropOldest, and the choice is what makes a slow reader harmless. A notice says "vault X has
|
||||
// moved to at least sequence N", so a newer one subsumes the one it displaces and the client's
|
||||
// answer — pull that vault — is identical either way. The writer therefore never waits and
|
||||
// TryWrite never fails, which is what lets the publish path be non-blocking and void.
|
||||
outbound = Channel.CreateBounded<VaultEvent>(new BoundedChannelOptions(queueDepth)
|
||||
{
|
||||
FullMode = BoundedChannelFullMode.DropOldest,
|
||||
SingleReader = true,
|
||||
SingleWriter = false,
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>Identifies this connection within the hub. Never sent to a client.</summary>
|
||||
internal Guid Id { get; }
|
||||
|
||||
/// <summary>The account that opened it.</summary>
|
||||
internal Guid UserId { get; }
|
||||
|
||||
/// <summary>Frames waiting to be written to the socket.</summary>
|
||||
internal ChannelReader<VaultEvent> Outbound => outbound.Reader;
|
||||
|
||||
/// <summary>How many vaults this socket currently follows.</summary>
|
||||
internal int VaultCount => Volatile.Read(ref vaults).Count;
|
||||
|
||||
/// <summary>Whether a change to this vault concerns this socket.</summary>
|
||||
internal bool IsSubscribedTo(Guid vaultId) => Volatile.Read(ref vaults).Contains(vaultId);
|
||||
|
||||
/// <summary>
|
||||
/// Replaces what this socket follows, after its account's access was re-resolved.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A whole-set swap of an immutable set rather than a mutation, so a publish walking the list
|
||||
/// concurrently reads either the old set or the new one and never a half-built one. No lock: the
|
||||
/// only writer is this connection's own pump.
|
||||
/// </remarks>
|
||||
internal void Resubscribe(FrozenSet<Guid> replacement) => Volatile.Write(ref vaults, replacement);
|
||||
|
||||
/// <summary>Queues a frame. Never blocks, and never fails — see the channel's full mode.</summary>
|
||||
internal bool TryEnqueue(VaultEvent frame) => outbound.Writer.TryWrite(frame);
|
||||
|
||||
/// <summary>Signals that nothing more will be queued, which ends the pump's drain loop.</summary>
|
||||
internal void Complete() => outbound.Writer.TryComplete();
|
||||
}
|
||||
@@ -19,6 +19,7 @@ namespace DodoSSH.Api.Features.Meta;
|
||||
internal sealed class GetMetaEndpoint(
|
||||
IOptions<SyncOptions> sync,
|
||||
IOptions<RelayOptions> relay,
|
||||
IOptions<EventsOptions> events,
|
||||
IOptions<ServerOptions> server)
|
||||
: EndpointWithoutRequest<Ok<MetaResponse>>
|
||||
{
|
||||
@@ -52,6 +53,14 @@ internal sealed class GetMetaEndpoint(
|
||||
features.Add(RelayFeature);
|
||||
}
|
||||
|
||||
// Advertised so a client knows whether to hold a socket open or rely on its timer. Absence is
|
||||
// not an error — synchronising on a timer is the supported behaviour and the socket only makes
|
||||
// it early — which is why this is a feature flag rather than a version bump. See ADR 0012.
|
||||
if (events.Value.Enabled)
|
||||
{
|
||||
features.Add(VaultEvents.Feature);
|
||||
}
|
||||
|
||||
return Task.FromResult(TypedResults.Ok(new MetaResponse(
|
||||
ServerVersion: ServerVersion,
|
||||
ApiVersions: [1],
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using DodoSSH.Api.Authorization;
|
||||
using DodoSSH.Api.Features.Events;
|
||||
using DodoSSH.Api.Setup;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain.Authorization;
|
||||
@@ -81,6 +82,7 @@ internal sealed class SyncPullEndpoint(
|
||||
internal sealed class SyncPushEndpoint(
|
||||
ICurrentUserContext currentUser,
|
||||
IVaultAccessService vaultAccess,
|
||||
IVaultEventPublisher events,
|
||||
SyncService sync)
|
||||
: Endpoint<SyncPushRequest, Results<Ok<SyncPushResponse>, NotFound, ProblemHttpResult>>
|
||||
{
|
||||
@@ -127,6 +129,8 @@ internal sealed class SyncPushEndpoint(
|
||||
// single stale item cannot block everything else a client queued while offline.
|
||||
var response = await sync.PushAsync(access.Vault!, user.Id, req, ct).ConfigureAwait(false);
|
||||
|
||||
Announce(access.Vault!.Id, response);
|
||||
|
||||
return TypedResults.Ok(response);
|
||||
}
|
||||
catch (PushBatchTooLargeException exception)
|
||||
@@ -142,4 +146,41 @@ internal sealed class SyncPushEndpoint(
|
||||
StatusCodes.Status400BadRequest, ProblemCodes.PushBatchTooLarge, exception.Message);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Tells every socket following this vault that it has moved.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Here rather than inside <see cref="SyncService.PushAsync"/>, and that placement is the point:
|
||||
/// the push has committed and released the per-vault advisory lock by the time this runs. Announced
|
||||
/// from inside, it would name a sequence no reader could see yet and would hold the lock that
|
||||
/// serialises writers across a fan-out. See ADR 0003 and ADR 0012.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The highest <em>applied</em> sequence, ignoring duplicates: a duplicate means an earlier push of
|
||||
/// that operation already landed, and it was announced then. Nothing applied means nothing to say —
|
||||
/// a batch of pure conflicts moved no vault, and announcing one anyway would have every client pull
|
||||
/// for a change that is not there.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private void Announce(Guid vaultId, SyncPushResponse response)
|
||||
{
|
||||
var highest = 0L;
|
||||
|
||||
foreach (var result in response.Results)
|
||||
{
|
||||
if (result.Status == SyncOperationStatus.Applied
|
||||
&& result.ChangeSequence is { } sequence
|
||||
&& sequence > highest)
|
||||
{
|
||||
highest = sequence;
|
||||
}
|
||||
}
|
||||
|
||||
if (highest > 0)
|
||||
{
|
||||
events.VaultChanged(vaultId, highest);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System.Globalization;
|
||||
using DodoSSH.Api.Features.Events;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain;
|
||||
using DodoSSH.Infrastructure;
|
||||
@@ -62,6 +63,7 @@ internal readonly record struct TeamAccess(Team? Team, TeamRole Role)
|
||||
internal sealed class TeamService(
|
||||
DodoDbContext database,
|
||||
TimeProvider clock,
|
||||
IVaultEventPublisher events,
|
||||
ILogger<TeamService> logger)
|
||||
{
|
||||
/// <summary>Longest acceptable slug. Matches the column.</summary>
|
||||
@@ -548,6 +550,11 @@ internal sealed class TeamService(
|
||||
|
||||
TeamLog.MemberAdded(logger, teamId, target.Id, role, actor.Id);
|
||||
|
||||
// Membership is what the server will serve, so every vault this team owns has just appeared in
|
||||
// the new member's list — before anybody wraps a key to them, which is a separate act and its
|
||||
// own notice. Told at once rather than on their next pass. See ADR 0012.
|
||||
events.VaultAccessChanged(target.Id);
|
||||
|
||||
return await DescribeAsync(target, membership, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
@@ -744,6 +751,11 @@ internal sealed class TeamService(
|
||||
}).ConfigureAwait(false);
|
||||
|
||||
TeamLog.MemberRemoved(logger, teamId, memberId, actor.Id, revoked);
|
||||
|
||||
// After the commit, so their client re-reads a list the server has already stopped serving
|
||||
// those vaults from. Their open socket re-resolves as it forwards this, which is what stops it
|
||||
// announcing changes to vaults they have just lost.
|
||||
events.VaultAccessChanged(memberId);
|
||||
}
|
||||
|
||||
/// <summary>Revokes one user's grants on every vault a team owns, and flags each for rekey.</summary>
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System.Security.Cryptography;
|
||||
using DodoSSH.Api.Features.Events;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain;
|
||||
using DodoSSH.Infrastructure;
|
||||
@@ -27,6 +28,7 @@ namespace DodoSSH.Api.Features.Teams;
|
||||
internal sealed class VaultGrantService(
|
||||
DodoDbContext database,
|
||||
TimeProvider clock,
|
||||
IVaultEventPublisher events,
|
||||
ILogger<VaultGrantService> logger)
|
||||
{
|
||||
/// <summary>
|
||||
@@ -414,6 +416,11 @@ internal sealed class VaultGrantService(
|
||||
|
||||
TeamLog.GrantIssued(
|
||||
logger, vault.Id, generation, request.RecipientUserId, actor.Id);
|
||||
|
||||
// The recipient, never the actor. This is the whole of what makes a shared vault arrive at
|
||||
// once rather than on the recipient's next pass — and it is the case the README has had to
|
||||
// apologise for since sharing shipped. See ADR 0012.
|
||||
events.VaultAccessChanged(request.RecipientUserId);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -701,6 +708,11 @@ internal sealed class VaultGrantService(
|
||||
|
||||
TeamLog.GrantRevoked(logger, vault.Id, recipientUserId, actor.Id);
|
||||
|
||||
// Told so their client stops showing a vault it can no longer open, rather than leaving it
|
||||
// listed until the next pass. It does not reach what they already pulled — nothing can, see
|
||||
// ADR 0001 — and the server-side effect is immediate regardless of whether this arrives.
|
||||
events.VaultAccessChanged(recipientUserId);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using DodoSSH.Api.Authorization;
|
||||
using DodoSSH.Api.Features.Events;
|
||||
using DodoSSH.Api.Features.Identity;
|
||||
using DodoSSH.Api.Features.Sync;
|
||||
using DodoSSH.Api.Features.Teams;
|
||||
@@ -47,6 +48,14 @@ builder.Services.AddScoped<VaultGrantService>();
|
||||
builder.Services.AddScoped<IIdentityBindingVerifier, IdentityBindingVerifier>();
|
||||
builder.Services.AddSingleton<ICursorKeyProvider, CursorKeyProvider>();
|
||||
|
||||
// A singleton, because the sockets it holds outlive the requests that opened them. Registered twice
|
||||
// resolving to the same instance, for the reason the invitation claim above is: the endpoint needs the
|
||||
// whole hub — admit, remove, count — while the write paths that announce a change need only the two
|
||||
// methods that announce one, and should not gain a reference to connection management to get them.
|
||||
builder.Services.AddSingleton<VaultEventHub>();
|
||||
builder.Services.AddSingleton<IVaultEventPublisher>(
|
||||
provider => provider.GetRequiredService<VaultEventHub>());
|
||||
|
||||
// Scoped rather than the AddAuthorization default of singleton: the handler reads the request's
|
||||
// DbContext, and a singleton would capture one for the lifetime of the process.
|
||||
builder.Services.AddScoped<IAuthorizationHandler, EnrolledHandler>();
|
||||
@@ -65,6 +74,13 @@ var app = builder.Build();
|
||||
|
||||
app.BlockFastEndpointsRouteTable();
|
||||
|
||||
// Before the authentication middleware, because the upgrade handshake has to survive it: the events
|
||||
// endpoint answers an ordinary authenticated request that happens to become a socket, and without
|
||||
// this the upgrade is never offered and the handler sees a plain GET. No allow-list of origins is
|
||||
// configured, deliberately — every client here is a native application sending a bearer token, so
|
||||
// there is no browser origin to trust and nothing a cross-site request could reach without one.
|
||||
app.UseWebSockets();
|
||||
|
||||
app.UseAuthentication();
|
||||
app.UseAuthorization();
|
||||
|
||||
|
||||
@@ -38,6 +38,22 @@ internal static class Configuration
|
||||
"Sync:DefaultPullLimit must not exceed Sync:MaxPullLimit.")
|
||||
.ValidateOnStart();
|
||||
|
||||
services.AddOptions<EventsOptions>()
|
||||
.BindConfiguration(EventsOptions.SectionName)
|
||||
.ValidateDataAnnotations()
|
||||
.Validate(
|
||||
options => options.HeartbeatInterval > TimeSpan.Zero,
|
||||
"Events:HeartbeatInterval must be greater than zero.")
|
||||
.Validate(
|
||||
options => options.AccessRefreshInterval > TimeSpan.Zero,
|
||||
"Events:AccessRefreshInterval must be greater than zero.")
|
||||
.Validate(
|
||||
options => options.MaxConnectionDuration > options.AccessRefreshInterval,
|
||||
"Events:MaxConnectionDuration must exceed Events:AccessRefreshInterval; a connection "
|
||||
+ "that never lives long enough to re-read its own access has none of the bound that "
|
||||
+ "setting exists to provide.")
|
||||
.ValidateOnStart();
|
||||
|
||||
services.AddOptions<RelayOptions>()
|
||||
.BindConfiguration(RelayOptions.SectionName)
|
||||
.ValidateDataAnnotations()
|
||||
|
||||
@@ -159,6 +159,82 @@ public sealed class RelayOptions
|
||||
public TimeSpan DrainTimeout { get; set; } = TimeSpan.FromSeconds(30);
|
||||
}
|
||||
|
||||
/// <summary>Realtime push settings. See ADR 0012.</summary>
|
||||
/// <remarks>
|
||||
/// Every one of these bounds a socket rather than a feature: with the whole thing off, or every cap
|
||||
/// met, clients synchronise on their timer exactly as they did before this existed. That is what
|
||||
/// makes it safe for an operator to turn any of them down.
|
||||
/// </remarks>
|
||||
public sealed class EventsOptions
|
||||
{
|
||||
/// <summary>Configuration section name.</summary>
|
||||
public const string SectionName = "Events";
|
||||
|
||||
/// <summary>
|
||||
/// Whether this deployment pushes vault changes at all.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// On by default, unlike the relay: this needs no outbound network, no target resolution and no
|
||||
/// new trust, and a deployment behind a proxy that will not upgrade should say so here rather than
|
||||
/// have every client discover it by failing.
|
||||
/// </remarks>
|
||||
public bool Enabled { get; set; } = true;
|
||||
|
||||
/// <summary>Maximum concurrent sockets per node.</summary>
|
||||
[Range(1, 100_000)]
|
||||
public int MaxConnectionsTotal { get; set; } = 500;
|
||||
|
||||
/// <summary>
|
||||
/// Maximum concurrent sockets per account.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Per account rather than per device, because the server cannot see a device here. Eight is a
|
||||
/// laptop, a desktop, a phone and room to reconnect before the old socket has been reaped.
|
||||
/// </remarks>
|
||||
[Range(1, 1000)]
|
||||
public int MaxConnectionsPerUser { get; set; } = 8;
|
||||
|
||||
/// <summary>
|
||||
/// How many notices may be queued for one socket before the oldest are dropped.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A notice names a vault and a position, so a newer one subsumes the one it replaces. The depth
|
||||
/// therefore buys smoothness over a brief stall and nothing else — losing the tail of a burst
|
||||
/// costs a client nothing, because the newest notice still says to pull.
|
||||
/// </remarks>
|
||||
[Range(1, 10_000)]
|
||||
public int OutboundQueueDepth { get; set; } = 64;
|
||||
|
||||
/// <summary>
|
||||
/// How often the server pings an idle socket.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Below the sixty seconds most reverse proxies idle out at, because a silent socket that a proxy
|
||||
/// has quietly dropped is indistinguishable from a quiet one until something is sent down it.
|
||||
/// </remarks>
|
||||
public TimeSpan HeartbeatInterval { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// How often an open socket re-reads which vaults its account may follow.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The backstop for a grant withdrawn mid-connection. Grants and membership changes publish
|
||||
/// immediately, so this is what covers the paths that do not — and what bounds the window if one
|
||||
/// is ever added without remembering to.
|
||||
/// </remarks>
|
||||
public TimeSpan AccessRefreshInterval { get; set; } = TimeSpan.FromMinutes(5);
|
||||
|
||||
/// <summary>
|
||||
/// The longest any one socket may live, regardless of its token.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A socket normally ends at its access token's expiry, which is far shorter. This is the bound
|
||||
/// for a provider that issues long-lived tokens, and it is what makes "no connection is older than
|
||||
/// this" a property of the server rather than of the identity provider's configuration.
|
||||
/// </remarks>
|
||||
public TimeSpan MaxConnectionDuration { get; set; } = TimeSpan.FromHours(12);
|
||||
}
|
||||
|
||||
/// <summary>Sync protocol limits.</summary>
|
||||
public sealed class SyncOptions
|
||||
{
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using DodoSSH.Api.Features.Events;
|
||||
using DodoSSH.Api.Features.Identity;
|
||||
using DodoSSH.Api.Features.Meta;
|
||||
using DodoSSH.Api.Features.Sync;
|
||||
@@ -43,6 +44,7 @@ internal static class EndpointRegistration
|
||||
typeof(ReadKeyLogEndpoint),
|
||||
typeof(SyncPullEndpoint),
|
||||
typeof(SyncPushEndpoint),
|
||||
typeof(VaultEventsEndpoint),
|
||||
typeof(CreateTeamEndpoint),
|
||||
typeof(ListTeamsEndpoint),
|
||||
typeof(UpdateTeamEndpoint),
|
||||
|
||||
@@ -26,6 +26,11 @@
|
||||
"MaxConcurrentSessionsPerUser": 10,
|
||||
"MaxConcurrentSessionsTotal": 200
|
||||
},
|
||||
"Events": {
|
||||
"Enabled": true,
|
||||
"MaxConnectionsTotal": 500,
|
||||
"MaxConnectionsPerUser": 8
|
||||
},
|
||||
"Sync": {
|
||||
"MaxOperationsPerPush": 500,
|
||||
"MaxPayloadBytes": 8388608,
|
||||
|
||||
@@ -0,0 +1,552 @@
|
||||
using System.Net.WebSockets;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Channels;
|
||||
using DodoSSH.Contracts;
|
||||
|
||||
namespace DodoSSH.Client.Api;
|
||||
|
||||
/// <summary>
|
||||
/// A server's "pull now" notices, as everything above the transport needs them.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// A queue to read from rather than an event to subscribe to, and that shape is the point: the one
|
||||
/// consumer is a synchronisation loop that already waits on a timer, so it can wait on this the same
|
||||
/// way and keep every continuation on the thread it started from. An event would deliver on whichever
|
||||
/// thread the socket happened to complete on, which in a user interface is the difference between
|
||||
/// working and an intermittent rendering fault nobody can reproduce.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Reading this is never how a change is applied.</b> A notice says which vault moved and nothing
|
||||
/// else; the answer to it is the ordinary delta pull. See ADR 0012.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public interface IVaultEventStream : IDisposable
|
||||
{
|
||||
/// <summary>Whether a socket is currently established.</summary>
|
||||
/// <remarks>
|
||||
/// For the interface to say whether it is live, not for a caller to branch on before reading:
|
||||
/// synchronising is correct whether or not this is true, because the timer is the fallback.
|
||||
/// </remarks>
|
||||
bool IsConnected { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Waits for the next notice.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Connects on the first call and reconnects for as long as it is read, so a caller neither starts
|
||||
/// nor restarts anything. A server that cannot be reached is not an error here — it is a wait that
|
||||
/// has not finished — because the caller's alternative is the timer it is already running.
|
||||
/// </remarks>
|
||||
ValueTask<VaultEvent> ReadAsync(CancellationToken cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Takes a notice if one is already waiting, without blocking.
|
||||
/// </summary>
|
||||
/// <returns>Whether there was one.</returns>
|
||||
/// <remarks>
|
||||
/// How a caller coalesces a burst. Five people saving at once produces five notices whose answer
|
||||
/// is a single synchronisation pass, so the loop reads one, waits a moment, and swallows the rest
|
||||
/// rather than running the same pull five times.
|
||||
/// </remarks>
|
||||
bool TryRead(out VaultEvent notice);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A stream that never delivers anything.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// For a server that does not advertise the <c>events</c> feature, and for tests. Deliberately waits
|
||||
/// for ever rather than completing: a caller selecting between this and a timer must fall through to
|
||||
/// the timer, and a read that returned immediately would spin that loop as fast as the machine allows.
|
||||
/// </remarks>
|
||||
public sealed class IdleVaultEventStream : IVaultEventStream
|
||||
{
|
||||
/// <summary>The one instance. It holds nothing.</summary>
|
||||
public static IdleVaultEventStream Instance { get; } = new();
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool IsConnected => false;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask<VaultEvent> ReadAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
await Task.Delay(System.Threading.Timeout.Infinite, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// Unreachable: the delay above only ever ends by throwing.
|
||||
return new VaultEvent(VaultEventKinds.Ping);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool TryRead(out VaultEvent notice)
|
||||
{
|
||||
notice = new VaultEvent(VaultEventKinds.Ping);
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose()
|
||||
{
|
||||
// Nothing is held.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Tuning for <see cref="VaultEventStream"/>.</summary>
|
||||
/// <remarks>
|
||||
/// Every value here bounds a reconnection rather than a feature. With the socket permanently
|
||||
/// unavailable the client synchronises on its timer, so the cost of getting these wrong is latency,
|
||||
/// never correctness.
|
||||
/// </remarks>
|
||||
public sealed record VaultEventStreamOptions
|
||||
{
|
||||
/// <summary>The defaults.</summary>
|
||||
public static VaultEventStreamOptions Default { get; } = new();
|
||||
|
||||
/// <summary>How long to wait before the first reconnection attempt.</summary>
|
||||
public TimeSpan InitialBackoff { get; init; } = TimeSpan.FromSeconds(1);
|
||||
|
||||
/// <summary>The longest the backoff may grow to.</summary>
|
||||
/// <remarks>
|
||||
/// A minute, which is the polling interval: past that point reconnecting sooner buys nothing,
|
||||
/// because the timer has already done the work the socket would have prompted.
|
||||
/// </remarks>
|
||||
public TimeSpan MaxBackoff { get; init; } = TimeSpan.FromMinutes(1);
|
||||
|
||||
/// <summary>
|
||||
/// How long a socket may be silent before it is presumed dead.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The server pings on an interval it states in its <c>hello</c>, so silence past a multiple of
|
||||
/// that means the connection is gone rather than idle — which is otherwise indistinguishable, and
|
||||
/// is exactly what a reverse proxy that quietly drops idle sockets produces. Used only until a
|
||||
/// <c>hello</c> arrives; after that the server's own figure is trusted.
|
||||
/// </remarks>
|
||||
public TimeSpan InitialSilenceTimeout { get; init; } = TimeSpan.FromSeconds(90);
|
||||
|
||||
/// <summary>How many notices may be waiting before the oldest are dropped.</summary>
|
||||
/// <remarks>
|
||||
/// Small on purpose. A notice means "pull that vault", so a newer one subsumes the one it
|
||||
/// displaces; a backlog would only make the loop pull repeatedly for work it has already done.
|
||||
/// </remarks>
|
||||
public int QueueDepth { get; init; } = 32;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Holds a socket to one server open, and hands over what it says.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The whole class is a reconnection policy. A dropped socket is the ordinary case — laptops sleep,
|
||||
/// proxies time out, tokens expire, servers are redeployed — so nothing here treats a failure as
|
||||
/// exceptional: it backs off and dials again, for as long as somebody is reading.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// It is safe to have no server at all. Every failure path ends in "wait, then try again", and the
|
||||
/// caller's synchronisation timer runs regardless, which is what makes it correct for this class to
|
||||
/// stay silent about problems rather than surface them.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class VaultEventStream : IVaultEventStream, IAsyncDisposable
|
||||
{
|
||||
private readonly Uri endpoint;
|
||||
private readonly IAccessTokenProvider tokens;
|
||||
private readonly TimeProvider clock;
|
||||
private readonly VaultEventStreamOptions options;
|
||||
private readonly Func<Uri, string, CancellationToken, Task<WebSocket>> connect;
|
||||
private readonly Channel<VaultEvent> notices;
|
||||
private readonly CancellationTokenSource closing = new();
|
||||
private readonly Lock starting = new();
|
||||
|
||||
private Task? pump;
|
||||
private bool disposed;
|
||||
|
||||
/// <summary>Creates a stream against one server.</summary>
|
||||
/// <param name="serverUrl">The server's base URL, as an ordinary <c>http</c> or <c>https</c> address.</param>
|
||||
/// <param name="tokens">Supplies a bearer token, refreshing it when it is due.</param>
|
||||
/// <param name="clock">Time source, for the backoff and the silence timeout.</param>
|
||||
/// <param name="options">Tuning, or null for the defaults.</param>
|
||||
public VaultEventStream(
|
||||
Uri serverUrl,
|
||||
IAccessTokenProvider tokens,
|
||||
TimeProvider clock,
|
||||
VaultEventStreamOptions? options = null)
|
||||
: this(serverUrl, tokens, clock, DialAsync, options)
|
||||
{
|
||||
}
|
||||
|
||||
/// <remarks>
|
||||
/// The connector is injected so the suite can drive this against a test host's in-memory socket.
|
||||
/// Reconnection is the entire behaviour of this class, and testing it against a real network would
|
||||
/// mean testing it against the one thing that cannot be made to fail on demand.
|
||||
/// </remarks>
|
||||
internal VaultEventStream(
|
||||
Uri serverUrl,
|
||||
IAccessTokenProvider tokens,
|
||||
TimeProvider clock,
|
||||
Func<Uri, string, CancellationToken, Task<WebSocket>> connect,
|
||||
VaultEventStreamOptions? options = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(serverUrl);
|
||||
ArgumentNullException.ThrowIfNull(tokens);
|
||||
ArgumentNullException.ThrowIfNull(clock);
|
||||
ArgumentNullException.ThrowIfNull(connect);
|
||||
|
||||
endpoint = EventsUrl(serverUrl);
|
||||
this.tokens = tokens;
|
||||
this.clock = clock;
|
||||
this.connect = connect;
|
||||
this.options = options ?? VaultEventStreamOptions.Default;
|
||||
|
||||
notices = Channel.CreateBounded<VaultEvent>(new BoundedChannelOptions(this.options.QueueDepth)
|
||||
{
|
||||
FullMode = BoundedChannelFullMode.DropOldest,
|
||||
SingleReader = true,
|
||||
SingleWriter = true,
|
||||
});
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool IsConnected { get; private set; }
|
||||
|
||||
/// <inheritdoc />
|
||||
public ValueTask<VaultEvent> ReadAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(disposed, this);
|
||||
|
||||
Start();
|
||||
|
||||
return notices.Reader.ReadAsync(cancellationToken);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// Does not start the connection, unlike <see cref="ReadAsync"/>: a caller draining a burst has
|
||||
/// already read one notice, and "is there another right now" is not a reason to dial a server.
|
||||
/// </remarks>
|
||||
public bool TryRead(out VaultEvent notice) => notices.Reader.TryRead(out notice!);
|
||||
|
||||
/// <summary>
|
||||
/// Ends the connection, without waiting for it to unwind.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// What a shell calls when a connection is dropped, from a synchronous path that must not block —
|
||||
/// <c>IVaultServer</c> is <see cref="IDisposable"/>, and blocking on a socket teardown from the
|
||||
/// user-interface thread is exactly the sync-over-async this repository bans. Cancelling is enough:
|
||||
/// every loop reads the token, and the pump has nothing to flush.
|
||||
/// </remarks>
|
||||
public void Dispose()
|
||||
{
|
||||
if (disposed)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
disposed = true;
|
||||
closing.Cancel();
|
||||
|
||||
// The source is deliberately left undisposed. The pump may still be inside a linked token
|
||||
// source derived from this one, and disposing a parent out from under a live child is how a
|
||||
// clean shutdown becomes an ObjectDisposedException on a background thread. It holds no timer
|
||||
// and no handle once cancelled; DisposeAsync is the path that cleans it up properly.
|
||||
}
|
||||
|
||||
/// <summary>Ends the connection and waits for it to unwind.</summary>
|
||||
/// <remarks>The deterministic form, for a caller that can await one — tests, mostly.</remarks>
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (disposed)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
disposed = true;
|
||||
|
||||
await closing.CancelAsync().ConfigureAwait(false);
|
||||
|
||||
if (pump is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
await pump.ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// The point of the cancel above.
|
||||
}
|
||||
}
|
||||
|
||||
closing.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Turns a server's base URL into its event socket's.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The path is replaced rather than appended, matching every other call in this client: request
|
||||
/// paths here are absolute — <c>/api/v1/…</c> — so a deployment behind a path prefix is already
|
||||
/// unsupported, and pretending otherwise in this one place would be a difference nobody could act
|
||||
/// on.
|
||||
/// </remarks>
|
||||
private static Uri EventsUrl(Uri serverUrl) =>
|
||||
new UriBuilder(serverUrl)
|
||||
{
|
||||
Scheme = string.Equals(serverUrl.Scheme, Uri.UriSchemeHttps, StringComparison.OrdinalIgnoreCase)
|
||||
? "wss"
|
||||
: "ws",
|
||||
Path = VaultEvents.Path,
|
||||
Query = string.Empty,
|
||||
Fragment = string.Empty,
|
||||
}.Uri;
|
||||
|
||||
private static async Task<WebSocket> DialAsync(Uri url, string token, CancellationToken cancellationToken)
|
||||
{
|
||||
var socket = new ClientWebSocket();
|
||||
|
||||
try
|
||||
{
|
||||
socket.Options.AddSubProtocol(VaultEvents.SubProtocol);
|
||||
|
||||
// A header rather than the Sec-WebSocket-Protocol smuggling ADR 0004 needs for the relay:
|
||||
// this client is a native application and can set one, and the token here is the ordinary
|
||||
// bearer credential rather than a ticket.
|
||||
socket.Options.SetRequestHeader("Authorization", $"Bearer {token}");
|
||||
|
||||
await socket.ConnectAsync(url, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return socket;
|
||||
}
|
||||
catch
|
||||
{
|
||||
socket.Dispose();
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
private void Start()
|
||||
{
|
||||
if (pump is not null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
lock (starting)
|
||||
{
|
||||
pump ??= Task.Run(() => RunAsync(closing.Token), closing.Token);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Connects, reads until it cannot, waits, and does it again.</summary>
|
||||
private async Task RunAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var backoff = options.InitialBackoff;
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var outcome = await AttemptAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// A socket that lived long enough to say hello proves the server is there and willing, so
|
||||
// the next failure starts from the bottom again rather than inheriting the backoff that
|
||||
// got us here. Without this a laptop that woke, connected, and then lost its network an
|
||||
// hour later would wait a full minute before trying, having already proved it need not.
|
||||
if (outcome == Outcome.Established)
|
||||
{
|
||||
backoff = options.InitialBackoff;
|
||||
}
|
||||
|
||||
// The server said this token is spent, which the token provider can fix without waiting.
|
||||
// Reconnecting at once is the whole reason that close code is distinct.
|
||||
var wait = outcome == Outcome.TokenExpired ? TimeSpan.Zero : Jitter(backoff);
|
||||
|
||||
if (wait > TimeSpan.Zero)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(wait, clock, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
backoff = backoff < options.MaxBackoff
|
||||
? Shorter(backoff * 2, options.MaxBackoff)
|
||||
: options.MaxBackoff;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>One connection, from dial to close.</summary>
|
||||
private async Task<Outcome> AttemptAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
WebSocket? socket = null;
|
||||
|
||||
try
|
||||
{
|
||||
var token = await tokens.GetAccessTokenAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
socket = await connect(endpoint, token, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
IsConnected = true;
|
||||
|
||||
return await PumpAsync(socket, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return Outcome.Cancelled;
|
||||
}
|
||||
catch (Exception exception) when (exception is not OutOfMemoryException)
|
||||
{
|
||||
// Every failure this can meet — no network, a refused upgrade, a server that has not been
|
||||
// deployed with this feature, a token that cannot be refreshed — has the same remedy, and
|
||||
// none of them is worth telling a user about. The synchronisation timer is still running.
|
||||
return Outcome.Failed;
|
||||
}
|
||||
finally
|
||||
{
|
||||
IsConnected = false;
|
||||
socket?.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Reads frames until the socket ends or goes quiet.</summary>
|
||||
private async Task<Outcome> PumpAsync(WebSocket socket, CancellationToken cancellationToken)
|
||||
{
|
||||
var buffer = new byte[8 * 1024];
|
||||
var silence = options.InitialSilenceTimeout;
|
||||
var established = false;
|
||||
|
||||
while (socket.State == WebSocketState.Open && !cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
// Rebuilt per frame rather than reset, because a linked source cannot be un-cancelled and
|
||||
// the deadline is what detects a socket that has silently gone away.
|
||||
using var deadline = new CancellationTokenSource(silence, clock);
|
||||
using var quiet = CancellationTokenSource.CreateLinkedTokenSource(
|
||||
cancellationToken, deadline.Token);
|
||||
|
||||
WebSocketReceiveResult received;
|
||||
|
||||
try
|
||||
{
|
||||
received = await socket.ReceiveAsync(buffer, quiet.Token).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
// Silent for longer than the server said it would be. The socket is gone in a way that
|
||||
// only reconnecting can discover, which is what a proxy dropping an idle connection
|
||||
// looks like from this end.
|
||||
return Ended(established);
|
||||
}
|
||||
|
||||
if (received.MessageType == WebSocketMessageType.Close)
|
||||
{
|
||||
return (int?)received.CloseStatus == VaultEvents.TokenExpiredCloseCode
|
||||
? Outcome.TokenExpired
|
||||
: Ended(established);
|
||||
}
|
||||
|
||||
// Binary is reserved by ADR 0012 for shared-session data, and text that arrived in pieces
|
||||
// is longer than anything this protocol defines. Skipped rather than fatal, so a newer
|
||||
// server does not cost this client its push for the whole session.
|
||||
if (received.MessageType != WebSocketMessageType.Text || !received.EndOfMessage)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (Parse(buffer.AsSpan(0, received.Count)) is not { } frame)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
established = true;
|
||||
silence = await AbsorbAsync(socket, frame, silence, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
return Ended(established);
|
||||
}
|
||||
|
||||
/// <summary>Deals with one frame, and says how long the socket may now stay quiet.</summary>
|
||||
private async Task<TimeSpan> AbsorbAsync(
|
||||
WebSocket socket,
|
||||
VaultEvent frame,
|
||||
TimeSpan silence,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (frame.HeartbeatSeconds is > 0 and var seconds)
|
||||
{
|
||||
// Three missed heartbeats. Two is within one paused thread of a false positive, and a
|
||||
// false positive here costs a reconnection rather than anything a user sees.
|
||||
silence = TimeSpan.FromSeconds(seconds * 3);
|
||||
}
|
||||
|
||||
if (string.Equals(frame.Kind, VaultEventKinds.Ping, StringComparison.Ordinal))
|
||||
{
|
||||
await socket.SendAsync(
|
||||
JsonSerializer.SerializeToUtf8Bytes(
|
||||
new VaultEvent(VaultEventKinds.Pong), DodoSshJsonContext.Default.VaultEvent),
|
||||
WebSocketMessageType.Text,
|
||||
endOfMessage: true,
|
||||
cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return silence;
|
||||
}
|
||||
|
||||
// Everything else, including a kind this build has never heard of, goes to the reader — which
|
||||
// is what makes the frame table extensible. An unrecognised kind is one the caller ignores;
|
||||
// refusing it here would be this class deciding what a newer server may say.
|
||||
notices.Writer.TryWrite(frame);
|
||||
|
||||
return silence;
|
||||
}
|
||||
|
||||
private static Outcome Ended(bool established) =>
|
||||
established ? Outcome.Established : Outcome.Failed;
|
||||
|
||||
private static VaultEvent? Parse(ReadOnlySpan<byte> utf8)
|
||||
{
|
||||
try
|
||||
{
|
||||
return JsonSerializer.Deserialize(utf8, DodoSshJsonContext.Default.VaultEvent);
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Spreads reconnections out, so a server that restarts is not met by every client at once.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <c>RandomNumberGenerator</c> because <c>System.Random</c> is banned repo-wide. Nothing here is
|
||||
/// security-relevant — the ban exists so that nothing key-, token- or nonce-adjacent can reach for
|
||||
/// the weak one by habit, and paying a few microseconds to keep that rule absolute is the cheaper
|
||||
/// side of the trade.
|
||||
/// </remarks>
|
||||
private static TimeSpan Jitter(TimeSpan delay)
|
||||
{
|
||||
var milliseconds = (int)Math.Clamp(delay.TotalMilliseconds, 1, int.MaxValue / 2);
|
||||
|
||||
return TimeSpan.FromMilliseconds(
|
||||
milliseconds + RandomNumberGenerator.GetInt32(0, Math.Max(1, milliseconds / 2)));
|
||||
}
|
||||
|
||||
private static TimeSpan Shorter(TimeSpan left, TimeSpan right) => left < right ? left : right;
|
||||
|
||||
/// <summary>How one connection attempt ended.</summary>
|
||||
private enum Outcome
|
||||
{
|
||||
/// <summary>Never got as far as a frame. Back off.</summary>
|
||||
Failed,
|
||||
|
||||
/// <summary>Ran, and then ended. Back off, but from the bottom.</summary>
|
||||
Established,
|
||||
|
||||
/// <summary>The server closed it because the token expired. Reconnect at once with a new one.</summary>
|
||||
TokenExpired,
|
||||
|
||||
/// <summary>The stream is being disposed.</summary>
|
||||
Cancelled,
|
||||
}
|
||||
}
|
||||
@@ -150,6 +150,17 @@ public interface IVaultServer : IDisposable
|
||||
/// <summary>Vault key grants: who can open a vault, and who let them.</summary>
|
||||
IVaultGrantApi Grants { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Notices that something changed, so a synchronisation need not wait for the timer.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Always present, never null: a server that does not offer the feature — or a test standing in
|
||||
/// for one — supplies <see cref="IdleVaultEventStream"/>, which simply never delivers. That keeps
|
||||
/// every caller on one shape, because the correct behaviour without a socket is the behaviour
|
||||
/// with a silent one: synchronise on the timer. See ADR 0012.
|
||||
/// </remarks>
|
||||
IVaultEventStream Events { get; }
|
||||
|
||||
/// <summary>Obtains the identity provider's signature over a key statement.</summary>
|
||||
IKeyBindingAuthorizer KeyBinding { get; }
|
||||
|
||||
@@ -197,7 +208,8 @@ public sealed class ServerConnection : IVaultServer
|
||||
MetaResponse meta,
|
||||
OidcClient oidc,
|
||||
RefreshingAccessTokenProvider tokens,
|
||||
DodoSshApiClient api)
|
||||
DodoSshApiClient api,
|
||||
TimeProvider clock)
|
||||
{
|
||||
ServerUrl = serverUrl;
|
||||
this.http = http;
|
||||
@@ -206,6 +218,14 @@ public sealed class ServerConnection : IVaultServer
|
||||
Oidc = oidc;
|
||||
this.tokens = tokens;
|
||||
Api = api;
|
||||
|
||||
// Decided from what this server said it supports rather than attempted and allowed to fail,
|
||||
// which is the same capability negotiation SyncOptions below does — see ADR 0002. A client
|
||||
// that dialled anyway would reconnect against a 404 for the whole session, and would look
|
||||
// from the outside exactly like one whose network was eating WebSockets.
|
||||
Events = meta.Features.Contains(VaultEvents.Feature, StringComparer.Ordinal)
|
||||
? new VaultEventStream(serverUrl, tokens, clock)
|
||||
: IdleVaultEventStream.Instance;
|
||||
}
|
||||
|
||||
/// <summary>The server this is connected to.</summary>
|
||||
@@ -238,6 +258,9 @@ public sealed class ServerConnection : IVaultServer
|
||||
/// <inheritdoc />
|
||||
public IVaultGrantApi Grants => Api;
|
||||
|
||||
/// <inheritdoc />
|
||||
public IVaultEventStream Events { get; }
|
||||
|
||||
/// <inheritdoc />
|
||||
public IKeyBindingAuthorizer KeyBinding => Oidc;
|
||||
|
||||
@@ -311,7 +334,8 @@ public sealed class ServerConnection : IVaultServer
|
||||
meta,
|
||||
oidc,
|
||||
refreshing,
|
||||
new DodoSshApiClient(transport, refreshing));
|
||||
new DodoSshApiClient(transport, refreshing),
|
||||
clock);
|
||||
}
|
||||
catch
|
||||
{
|
||||
@@ -382,7 +406,8 @@ public sealed class ServerConnection : IVaultServer
|
||||
meta,
|
||||
oidc,
|
||||
refreshing,
|
||||
new DodoSshApiClient(transport, refreshing));
|
||||
new DodoSshApiClient(transport, refreshing),
|
||||
clock);
|
||||
}
|
||||
catch
|
||||
{
|
||||
@@ -400,6 +425,12 @@ public sealed class ServerConnection : IVaultServer
|
||||
}
|
||||
|
||||
disposed = true;
|
||||
|
||||
// First, and without waiting: the socket's own loops read the token provider and the transport
|
||||
// below, so tearing either down while it is still dialling would surface as a fault on a
|
||||
// background thread at the moment a user signed out.
|
||||
Events.Dispose();
|
||||
|
||||
tokens.Dispose();
|
||||
http.Dispose();
|
||||
}
|
||||
|
||||
@@ -10,6 +10,7 @@ using DodoSSH.Client.Session;
|
||||
using DodoSSH.Client.Ssh;
|
||||
using DodoSSH.Client.Sync;
|
||||
using DodoSSH.Client.Terminal;
|
||||
using DodoSSH.Contracts;
|
||||
|
||||
namespace DodoSSH.Client.Shell.ViewModels;
|
||||
|
||||
@@ -1025,13 +1026,29 @@ internal sealed partial class VaultViewModel(
|
||||
VaultVisibility? visibility = null) : ObservableObject, IAsyncDisposable
|
||||
{
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// A minute. The pull is a delta keyed on a cursor, so an idle pass is one small request and costs the
|
||||
/// server almost nothing; the number that matters is how stale a teammate's change may look, and a
|
||||
/// minute is short enough not to be noticed. Anything much shorter would be polling for its own sake,
|
||||
/// and a change made on this machine does not wait for the timer anyway — saving pushes immediately.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Unchanged by the push channel, and deliberately so. The socket makes a pass <em>early</em>; this is
|
||||
/// what makes one happen at all, for a client whose network eats WebSockets, whose server has the
|
||||
/// feature off, or whose notice was dropped. See <see cref="WaitForWorkAsync"/> and ADR 0012.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private static readonly TimeSpan AutoSyncInterval = TimeSpan.FromMinutes(1);
|
||||
|
||||
/// <summary>How long a pushed notice waits, in case more are on their way.</summary>
|
||||
/// <remarks>
|
||||
/// A quarter of a second, which is below what anybody perceives and above the gap between the
|
||||
/// notices one person's save produces — a host and its activity log entry are two items in one
|
||||
/// push, and a colleague clearing a folder is a burst. Without it each notice would run its own
|
||||
/// full pass, and the pass a burst deserves is one.
|
||||
/// </remarks>
|
||||
private static readonly TimeSpan NoticeDebounce = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
/// <summary>How often the logs are pruned, at most.</summary>
|
||||
/// <remarks>
|
||||
/// Hours rather than minutes, because pruning writes tombstones that sync. Retention is measured in days
|
||||
@@ -4030,10 +4047,16 @@ internal sealed partial class VaultViewModel(
|
||||
/// <para>
|
||||
/// <b>This is the whole of how a shared vault arrives.</b> Sharing is two acts on two machines: the
|
||||
/// person sharing wraps the vault key to the recipient, and the recipient's own client has to notice.
|
||||
/// The recipient is handed nothing — there is no push channel — so without this the vault list stayed
|
||||
/// exactly as it was cached at sign-in, and a vault shared with somebody appeared on their machine only
|
||||
/// if they happened to sign in through the browser again. Everything else was already right, which is
|
||||
/// why it looked like sharing was broken rather than like a list that was never re-read.
|
||||
/// Without this the vault list stayed exactly as it was cached at sign-in, and a vault shared with
|
||||
/// somebody appeared on their machine only if they happened to sign in through the browser again.
|
||||
/// Everything else was already right, which is why it looked like sharing was broken rather than like a
|
||||
/// list that was never re-read.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The server now says when this is worth doing — a <c>vaults.changed</c> notice wakes the pass, so the
|
||||
/// vault turns up as it is shared rather than within the minute — but that only decides <em>when</em>.
|
||||
/// This call is still what discovers the vault, on the notice and on every timed pass alike, because a
|
||||
/// client with no socket has to arrive at the same place. See ADR 0012.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A failure is left to the caller, which treats it as the pass failing: the call is to the same server
|
||||
@@ -4080,6 +4103,7 @@ internal sealed partial class VaultViewModel(
|
||||
private async Task RunAutoSyncLoopAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
using var timer = new PeriodicTimer(AutoSyncInterval);
|
||||
var waits = new AutoSyncWaits();
|
||||
|
||||
try
|
||||
{
|
||||
@@ -4094,7 +4118,7 @@ internal sealed partial class VaultViewModel(
|
||||
// user is doing something.
|
||||
await SyncOnOpenAsync(cancellationToken).ConfigureAwait(true);
|
||||
|
||||
while (await timer.WaitForNextTickAsync(cancellationToken).ConfigureAwait(true))
|
||||
while (await WaitForWorkAsync(timer, waits, cancellationToken).ConfigureAwait(true))
|
||||
{
|
||||
await AutoSyncAsync(cancellationToken).ConfigureAwait(true);
|
||||
}
|
||||
@@ -4105,6 +4129,98 @@ internal sealed partial class VaultViewModel(
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Waits for the timer to come round, or for the server to say there is something to fetch.
|
||||
/// </summary>
|
||||
/// <returns>Whether to run a pass. False means the loop is over.</returns>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The timer is unchanged and is still what guarantees a pass. The socket only makes one
|
||||
/// <em>early</em>, which is why nothing here treats its absence as a problem: no connection, a
|
||||
/// server without the feature, a network that eats WebSockets, or a notice dropped under
|
||||
/// backpressure all leave a loop that behaves exactly as it did before this existed. See ADR 0012.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Both waits are held across iterations, and that is load-bearing rather than an
|
||||
/// optimisation.</b> <see cref="PeriodicTimer"/> permits only one outstanding
|
||||
/// <c>WaitForNextTickAsync</c> and throws on a second, and an abandoned channel read stays
|
||||
/// registered and consumes the next notice written — which would silently lose exactly the wake-up
|
||||
/// this is for. Whichever wait did not win is kept and awaited again.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private async Task<bool> WaitForWorkAsync(
|
||||
PeriodicTimer timer,
|
||||
AutoSyncWaits waits,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
// Re-read every time, because signing out and back in replaces the connection — and with it
|
||||
// the stream. A read still pending against the old one is left to be cancelled with it.
|
||||
var stream = connection()?.Events;
|
||||
|
||||
if (!ReferenceEquals(stream, waits.Watching))
|
||||
{
|
||||
waits.Watching = stream;
|
||||
waits.Notice = null;
|
||||
}
|
||||
|
||||
waits.Tick ??= timer.WaitForNextTickAsync(cancellationToken).AsTask();
|
||||
waits.Notice ??= stream?.ReadAsync(cancellationToken).AsTask();
|
||||
|
||||
if (waits.Notice is null)
|
||||
{
|
||||
var only = waits.Tick;
|
||||
waits.Tick = null;
|
||||
|
||||
return await only.ConfigureAwait(true);
|
||||
}
|
||||
|
||||
var first = await Task.WhenAny(waits.Tick, waits.Notice).ConfigureAwait(true);
|
||||
|
||||
if (ReferenceEquals(first, waits.Tick))
|
||||
{
|
||||
var ticked = waits.Tick;
|
||||
waits.Tick = null;
|
||||
|
||||
return await ticked.ConfigureAwait(true);
|
||||
}
|
||||
|
||||
// Observed so a faulted read does not go unhandled, and so a stream that has been disposed
|
||||
// ends this wait rather than being asked again.
|
||||
await waits.Notice.ConfigureAwait(true);
|
||||
waits.Notice = null;
|
||||
|
||||
// A burst — one person's save is two items, and a colleague tidying a folder is a dozen —
|
||||
// deserves one pass rather than one each.
|
||||
await Task.Delay(NoticeDebounce, cancellationToken).ConfigureAwait(true);
|
||||
|
||||
while (stream!.TryRead(out _))
|
||||
{
|
||||
// Swallowed on purpose. Every notice means the same thing, which is what the pass about to
|
||||
// run already does; what they say about *which* vault is not read, because a pass syncs
|
||||
// every vault this session can reach anyway.
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>The two waits the background loop keeps alive between passes.</summary>
|
||||
/// <remarks>
|
||||
/// A class rather than three locals because <see cref="WaitForWorkAsync"/> has to hand them back
|
||||
/// changed, and a method that took three <c>ref</c> parameters could not be <c>async</c>. See that
|
||||
/// method for why abandoning either of them is a defect rather than a tidiness question.
|
||||
/// </remarks>
|
||||
private sealed class AutoSyncWaits
|
||||
{
|
||||
/// <summary>The pending timer tick, or null when the last one has been consumed.</summary>
|
||||
internal Task<bool>? Tick { get; set; }
|
||||
|
||||
/// <summary>The pending read from the server's push channel.</summary>
|
||||
internal Task<VaultEvent>? Notice { get; set; }
|
||||
|
||||
/// <summary>The stream <see cref="Notice"/> was taken from, to notice a reconnection.</summary>
|
||||
internal IVaultEventStream? Watching { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>Shows one kind of item, if nothing is being edited.</summary>
|
||||
/// <remarks>
|
||||
/// Takes the section rather than there being one command per kind, so a third kind is an enum member and
|
||||
|
||||
@@ -66,6 +66,11 @@ namespace DodoSSH.Contracts;
|
||||
// Registered in its own right, not only as a member of the sync DTOs: the client's local
|
||||
// cache seals this record under the LocalCacheKey and needs its type info directly.
|
||||
[JsonSerializable(typeof(SyncPlaintextFields))]
|
||||
|
||||
// The event socket's only frame type. Registered although nothing else references it: frames are
|
||||
// written straight onto a WebSocket rather than through a response body, so the resolver never
|
||||
// infers it from an endpoint's signature the way it does for every DTO above.
|
||||
[JsonSerializable(typeof(VaultEvent))]
|
||||
[JsonSerializable(typeof(RelayTicketRequest))]
|
||||
[JsonSerializable(typeof(RelayTicketResponse))]
|
||||
[JsonSerializable(typeof(RelaySessionSummary))]
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
namespace DodoSSH.Contracts;
|
||||
|
||||
/// <summary>
|
||||
/// The event socket's protocol constants.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The version lives in the subprotocol name rather than in the URL, for the reason ADR 0002 gives
|
||||
/// about the rest of this API: a client and a server that upgrade independently have to agree by
|
||||
/// negotiating rather than by assuming, and a WebSocket handshake already has a field for exactly
|
||||
/// that. A server that does not offer <see cref="SubProtocol"/> fails the handshake, which a client
|
||||
/// can act on — rather than opening a socket that then speaks a dialect it cannot read.
|
||||
/// </remarks>
|
||||
public static class VaultEvents
|
||||
{
|
||||
/// <summary>The path the event socket is served from.</summary>
|
||||
public const string Path = "/api/v1/events";
|
||||
|
||||
/// <summary>The only subprotocol this version speaks.</summary>
|
||||
public const string SubProtocol = "dodossh.events.v1";
|
||||
|
||||
/// <summary>The <c>/api/v1/meta</c> feature flag advertising that this server pushes at all.</summary>
|
||||
public const string Feature = "events";
|
||||
|
||||
/// <summary>
|
||||
/// Close code for a socket whose access token has expired.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// In the 4000–4999 range, which the WebSocket specification reserves for applications. Its own
|
||||
/// code because it is the one close a client should answer by reconnecting immediately with a
|
||||
/// fresh token, rather than by backing off as it would for a server that went away.
|
||||
/// </remarks>
|
||||
public const int TokenExpiredCloseCode = 4401;
|
||||
|
||||
/// <summary>
|
||||
/// Close code for a caller already holding as many sockets as it may.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Distinguished from <see cref="TokenExpiredCloseCode"/> because the remedy is the opposite:
|
||||
/// reconnecting at once is what caused it. A client that meets this backs off and keeps polling.
|
||||
/// </remarks>
|
||||
public const int TooManyConnectionsCloseCode = 4429;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The kinds of event this socket carries.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Constants rather than an <c>enum</c>, and that is a compatibility decision rather than a style
|
||||
/// one. <c>DodoSshJsonContext</c> sets <c>UseStringEnumConverter</c>, which <em>throws</em> on a
|
||||
/// value it does not know — so a newer server sending a kind an older client has never heard of
|
||||
/// would not merely add an unreadable frame, it would break that client's socket. A string is
|
||||
/// ignored instead, which is what makes this list extensible. <see cref="ProblemCodes"/> is the same
|
||||
/// shape for the same reason.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Anything a client cannot parse must be skipped, not treated as an error.</b> That rule is what
|
||||
/// the shared-session frames of ADR 0012 will rely on when they arrive.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class VaultEventKinds
|
||||
{
|
||||
/// <summary>The server accepted the socket. Always the first frame.</summary>
|
||||
public const string Hello = "hello";
|
||||
|
||||
/// <summary>
|
||||
/// A vault has changes at or before <see cref="VaultEvent.Sequence"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Carries no ciphertext and no item identity — the client's answer is the delta pull it would
|
||||
/// have run on its timer anyway. See ADR 0012 for why pushing the items themselves is refused.
|
||||
/// </remarks>
|
||||
public const string VaultChanged = "vault.changed";
|
||||
|
||||
/// <summary>
|
||||
/// The set of vaults this account can reach is no longer what it was.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A vault shared with the caller, or a grant withdrawn. Deliberately says nothing about
|
||||
/// <em>which</em>: the client re-reads the list, which is the same call it already makes at the
|
||||
/// start of every synchronisation pass.
|
||||
/// </remarks>
|
||||
public const string VaultsChanged = "vaults.changed";
|
||||
|
||||
/// <summary>Heartbeat. Whichever side receives one answers <see cref="Pong"/>.</summary>
|
||||
public const string Ping = "ping";
|
||||
|
||||
/// <summary>The answer to a <see cref="Ping"/>.</summary>
|
||||
public const string Pong = "pong";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One frame on the event socket.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// One flat record for every kind, with the fields a given kind does not use left null, rather than
|
||||
/// a polymorphic hierarchy. The set is small, the frames are tiny, and <c>System.Text.Json</c>
|
||||
/// polymorphism would put a second discriminator mechanism next to the <see cref="Kind"/> string
|
||||
/// that is already the discriminator. Nothing else in <c>DodoSSH.Contracts</c> is polymorphic.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Nothing here is secret, by construction.</b> The server cannot read a vault's contents, so a
|
||||
/// notice cannot describe them; what it does disclose — that a vault changed, and when — is the same
|
||||
/// metadata ADR 0001 already accepts the server holding.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="Kind">One of <see cref="VaultEventKinds"/>. An unrecognised kind must be ignored.</param>
|
||||
/// <param name="VaultId">The vault a <see cref="VaultEventKinds.VaultChanged"/> is about.</param>
|
||||
/// <param name="Sequence">
|
||||
/// The change-log position that vault has reached. A hint for logging and for coalescing, not a
|
||||
/// cursor: cursors are opaque and HMAC-tagged, and this is neither.
|
||||
/// </param>
|
||||
/// <param name="ServerTime">
|
||||
/// The server's clock when the frame was written. The client already measures skew against
|
||||
/// <c>SyncPullResponse.ServerTime</c>; this lets a socket that is quiet for other reasons keep that
|
||||
/// measurement current.
|
||||
/// </param>
|
||||
/// <param name="HeartbeatSeconds">
|
||||
/// How often the server will ping, sent with <see cref="VaultEventKinds.Hello"/>. The client uses it
|
||||
/// to decide when silence means the connection is dead rather than idle.
|
||||
/// </param>
|
||||
/// <param name="VaultCount">
|
||||
/// How many vaults this socket is subscribed to, sent with <see cref="VaultEventKinds.Hello"/>.
|
||||
/// Diagnostic: a socket subscribed to nothing is a real state — an account with no vaults yet — and
|
||||
/// is otherwise indistinguishable from one that is quietly broken.
|
||||
/// </param>
|
||||
public sealed record VaultEvent(
|
||||
string Kind,
|
||||
Guid? VaultId = null,
|
||||
long? Sequence = null,
|
||||
DateTimeOffset? ServerTime = null,
|
||||
int? HeartbeatSeconds = null,
|
||||
int? VaultCount = null);
|
||||
@@ -66,6 +66,16 @@ public static class ProblemCodes
|
||||
/// </remarks>
|
||||
public const string MalformedRequest = "malformed-request";
|
||||
|
||||
/// <summary>
|
||||
/// This deployment does not push vault changes over a socket.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Not a failure to recover from: synchronising on a timer is the supported behaviour and the
|
||||
/// socket only ever made it early. A client that meets this stops dialling and keeps polling. See
|
||||
/// ADR 0012.
|
||||
/// </remarks>
|
||||
public const string EventsUnavailable = "events-unavailable";
|
||||
|
||||
/// <summary>The relay refused the requested target. Never states why, to avoid a probe oracle.</summary>
|
||||
public const string RelayTargetRejected = "relay-target-rejected";
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
const DodoSSH.Contracts.ProblemCodes.AlreadyEnrolled = "already-enrolled" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.ClientTooOld = "client-too-old" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.EnrollmentRequired = "enrollment-required" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.EventsUnavailable = "events-unavailable" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.Forbidden = "forbidden" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.IdempotencyKeyReuse = "idempotency-key-reuse" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.IdentityBindingInvalid = "identity-binding-invalid" -> string!
|
||||
@@ -22,6 +23,16 @@ const DodoSSH.Contracts.ProblemCodes.TeamNotEmpty = "team-not-empty" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.TeamSlugTaken = "team-slug-taken" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.TypeBaseUri = "https://dodossh.dev/problems/" -> string!
|
||||
const DodoSSH.Contracts.ProblemCodes.VaultConflict = "vault-conflict" -> string!
|
||||
const DodoSSH.Contracts.VaultEventKinds.Hello = "hello" -> string!
|
||||
const DodoSSH.Contracts.VaultEventKinds.Ping = "ping" -> string!
|
||||
const DodoSSH.Contracts.VaultEventKinds.Pong = "pong" -> string!
|
||||
const DodoSSH.Contracts.VaultEventKinds.VaultChanged = "vault.changed" -> string!
|
||||
const DodoSSH.Contracts.VaultEventKinds.VaultsChanged = "vaults.changed" -> string!
|
||||
const DodoSSH.Contracts.VaultEvents.Feature = "events" -> string!
|
||||
const DodoSSH.Contracts.VaultEvents.Path = "/api/v1/events" -> string!
|
||||
const DodoSSH.Contracts.VaultEvents.SubProtocol = "dodossh.events.v1" -> string!
|
||||
const DodoSSH.Contracts.VaultEvents.TokenExpiredCloseCode = 4401 -> int
|
||||
const DodoSSH.Contracts.VaultEvents.TooManyConnectionsCloseCode = 4429 -> int
|
||||
DodoSSH.Contracts.AddTeamMemberRequest
|
||||
DodoSSH.Contracts.AddTeamMemberRequest.<Clone>$() -> DodoSSH.Contracts.AddTeamMemberRequest!
|
||||
DodoSSH.Contracts.AddTeamMemberRequest.AddTeamMemberRequest(System.Guid UserId, DodoSSH.Contracts.TeamMemberRole Role, string? Email = null) -> void
|
||||
@@ -686,6 +697,25 @@ DodoSSH.Contracts.UpdateVaultRequest.Equals(DodoSSH.Contracts.UpdateVaultRequest
|
||||
DodoSSH.Contracts.UpdateVaultRequest.Name.get -> string!
|
||||
DodoSSH.Contracts.UpdateVaultRequest.Name.init -> void
|
||||
DodoSSH.Contracts.UpdateVaultRequest.UpdateVaultRequest(string! Name) -> void
|
||||
DodoSSH.Contracts.VaultEvent
|
||||
DodoSSH.Contracts.VaultEvent.<Clone>$() -> DodoSSH.Contracts.VaultEvent!
|
||||
DodoSSH.Contracts.VaultEvent.Deconstruct(out string! Kind, out System.Guid? VaultId, out long? Sequence, out System.DateTimeOffset? ServerTime, out int? HeartbeatSeconds, out int? VaultCount) -> void
|
||||
DodoSSH.Contracts.VaultEvent.Equals(DodoSSH.Contracts.VaultEvent? other) -> bool
|
||||
DodoSSH.Contracts.VaultEvent.HeartbeatSeconds.get -> int?
|
||||
DodoSSH.Contracts.VaultEvent.HeartbeatSeconds.init -> void
|
||||
DodoSSH.Contracts.VaultEvent.Kind.get -> string!
|
||||
DodoSSH.Contracts.VaultEvent.Kind.init -> void
|
||||
DodoSSH.Contracts.VaultEvent.Sequence.get -> long?
|
||||
DodoSSH.Contracts.VaultEvent.Sequence.init -> void
|
||||
DodoSSH.Contracts.VaultEvent.ServerTime.get -> System.DateTimeOffset?
|
||||
DodoSSH.Contracts.VaultEvent.ServerTime.init -> void
|
||||
DodoSSH.Contracts.VaultEvent.VaultCount.get -> int?
|
||||
DodoSSH.Contracts.VaultEvent.VaultCount.init -> void
|
||||
DodoSSH.Contracts.VaultEvent.VaultEvent(string! Kind, System.Guid? VaultId = null, long? Sequence = null, System.DateTimeOffset? ServerTime = null, int? HeartbeatSeconds = null, int? VaultCount = null) -> void
|
||||
DodoSSH.Contracts.VaultEvent.VaultId.get -> System.Guid?
|
||||
DodoSSH.Contracts.VaultEvent.VaultId.init -> void
|
||||
DodoSSH.Contracts.VaultEventKinds
|
||||
DodoSSH.Contracts.VaultEvents
|
||||
DodoSSH.Contracts.VaultGrantsResponse
|
||||
DodoSSH.Contracts.VaultGrantsResponse.<Clone>$() -> DodoSSH.Contracts.VaultGrantsResponse!
|
||||
DodoSSH.Contracts.VaultGrantsResponse.Deconstruct(out System.Guid VaultId, out uint KeyGeneration, out bool RekeyRequired, out System.Collections.Generic.IReadOnlyList<DodoSSH.Contracts.VaultGrantSummary!>! Grants) -> void
|
||||
@@ -877,6 +907,9 @@ override DodoSSH.Contracts.UpdateTeamRequest.ToString() -> string!
|
||||
override DodoSSH.Contracts.UpdateVaultRequest.Equals(object? obj) -> bool
|
||||
override DodoSSH.Contracts.UpdateVaultRequest.GetHashCode() -> int
|
||||
override DodoSSH.Contracts.UpdateVaultRequest.ToString() -> string!
|
||||
override DodoSSH.Contracts.VaultEvent.Equals(object? obj) -> bool
|
||||
override DodoSSH.Contracts.VaultEvent.GetHashCode() -> int
|
||||
override DodoSSH.Contracts.VaultEvent.ToString() -> string!
|
||||
override DodoSSH.Contracts.VaultGrantsResponse.Equals(object? obj) -> bool
|
||||
override DodoSSH.Contracts.VaultGrantsResponse.GetHashCode() -> int
|
||||
override DodoSSH.Contracts.VaultGrantsResponse.ToString() -> string!
|
||||
@@ -972,6 +1005,8 @@ static DodoSSH.Contracts.UpdateTeamRequest.operator !=(DodoSSH.Contracts.UpdateT
|
||||
static DodoSSH.Contracts.UpdateTeamRequest.operator ==(DodoSSH.Contracts.UpdateTeamRequest? left, DodoSSH.Contracts.UpdateTeamRequest? right) -> bool
|
||||
static DodoSSH.Contracts.UpdateVaultRequest.operator !=(DodoSSH.Contracts.UpdateVaultRequest? left, DodoSSH.Contracts.UpdateVaultRequest? right) -> bool
|
||||
static DodoSSH.Contracts.UpdateVaultRequest.operator ==(DodoSSH.Contracts.UpdateVaultRequest? left, DodoSSH.Contracts.UpdateVaultRequest? right) -> bool
|
||||
static DodoSSH.Contracts.VaultEvent.operator !=(DodoSSH.Contracts.VaultEvent? left, DodoSSH.Contracts.VaultEvent? right) -> bool
|
||||
static DodoSSH.Contracts.VaultEvent.operator ==(DodoSSH.Contracts.VaultEvent? left, DodoSSH.Contracts.VaultEvent? right) -> bool
|
||||
static DodoSSH.Contracts.VaultGrantsResponse.operator !=(DodoSSH.Contracts.VaultGrantsResponse? left, DodoSSH.Contracts.VaultGrantsResponse? right) -> bool
|
||||
static DodoSSH.Contracts.VaultGrantsResponse.operator ==(DodoSSH.Contracts.VaultGrantsResponse? left, DodoSSH.Contracts.VaultGrantsResponse? right) -> bool
|
||||
static DodoSSH.Contracts.VaultGrantSummary.operator !=(DodoSSH.Contracts.VaultGrantSummary? left, DodoSSH.Contracts.VaultGrantSummary? right) -> bool
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
using System.Net.Http.Headers;
|
||||
using System.Net.WebSockets;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Infrastructure;
|
||||
using Microsoft.AspNetCore.Hosting;
|
||||
using Microsoft.AspNetCore.Mvc.Testing;
|
||||
@@ -107,6 +109,34 @@ public sealed class ApiFixture : WebApplicationFactory<Program>, IAsyncLifetime
|
||||
|
||||
/// <summary>Opens a database scope for arranging state and asserting on it.</summary>
|
||||
public AsyncServiceScope CreateScope() => Services.CreateAsyncScope();
|
||||
|
||||
/// <summary>
|
||||
/// Opens the event socket as the given subject, through the real pipeline.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The bearer token goes on the upgrade request, which is the whole of the socket's authorization
|
||||
/// — see ADR 0012 — so a test that stubbed it would be testing nothing. The subprotocol is offered
|
||||
/// because the server refuses an upgrade that does not, and that refusal is itself under test.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <c>TestServer</c> speaks WebSockets in-memory with no port and no network, so these run
|
||||
/// wherever the rest of the suite does.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public Task<WebSocket> ConnectEventsAsync(string subject, CancellationToken cancellationToken)
|
||||
{
|
||||
var token = IdentityProvider.MintToken(subject);
|
||||
var client = Server.CreateWebSocketClient();
|
||||
|
||||
client.SubProtocols.Add(VaultEvents.SubProtocol);
|
||||
// The server-side request, so the header is a raw string rather than a typed value.
|
||||
client.ConfigureRequest = request => request.Headers.Authorization = $"Bearer {token}";
|
||||
|
||||
return client.ConnectAsync(
|
||||
new Uri(Server.BaseAddress, VaultEvents.Path.TrimStart('/')),
|
||||
cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Shares one host and container across every test class in the assembly.</summary>
|
||||
|
||||
@@ -56,6 +56,12 @@ public sealed class EndpointInventoryTests(ApiFixture fixture)
|
||||
"POST /api/v1/vaults/{vaultId:guid}/sync/pull name=SyncPull tags=Sync policies=Enrolled anon=False",
|
||||
"POST /api/v1/vaults/{vaultId:guid}/sync/push name=SyncPush tags=Sync policies=Enrolled anon=False",
|
||||
|
||||
// The WebSocket, gated exactly as sync is and for the same reason — it announces changes to
|
||||
// vaults, and a caller who could not read one has nothing to be told about. It appears here as
|
||||
// an ordinary route because that is what it is until the upgrade: the bearer token authorises
|
||||
// the handshake, unlike the relay's ticket. See ADR 0012.
|
||||
"GET /api/v1/events name=VaultEvents tags=Events policies=Enrolled anon=False",
|
||||
|
||||
// Enrolled, because the answer exists to be wrapped to and a caller with no key of their own has
|
||||
// nothing to wrap and no signature to attribute it with. There is no search here — see
|
||||
// DirectoryService for why an exact-match-only directory is a decision rather than a shortcut.
|
||||
|
||||
@@ -0,0 +1,461 @@
|
||||
using System.Net;
|
||||
using System.Net.WebSockets;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain;
|
||||
using DodoSSH.Infrastructure;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
|
||||
namespace DodoSSH.Api.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The push channel, over a real socket through the real authentication pipeline.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The tests that matter most here are the two negatives: an unauthenticated upgrade is refused, and a
|
||||
/// change to somebody else's vault does not reach this socket. A push channel that leaked <em>which
|
||||
/// vault ids exist and when they change</em> would be a disclosure the pull path takes deliberate
|
||||
/// trouble to avoid — <c>SyncPullEndpoint</c> answers 404 rather than 403 for exactly that reason —
|
||||
/// and it would be invisible in a test that only checked that notices arrive.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Ordering is asserted rather than absence-within-a-timeout wherever possible. "Nothing arrived in
|
||||
/// two seconds" is a test that passes on a slow machine for the wrong reason; "the first notice this
|
||||
/// socket saw was about its own vault, although another vault was written to first" is not.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
[Collection(ApiCollection.Name)]
|
||||
public sealed class EventsEndpointTests(ApiFixture fixture)
|
||||
{
|
||||
private static readonly DateTimeOffset Now = new(2026, 8, 4, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
/// <summary>
|
||||
/// How long a test will wait for a frame before calling it a failure.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Generous, because it is not a measurement: every wait here is for something already committed,
|
||||
/// so the only thing this bounds is how long a genuinely broken build hangs before it reports.
|
||||
/// </remarks>
|
||||
private static readonly TimeSpan FrameTimeout = TimeSpan.FromSeconds(30);
|
||||
|
||||
// ---- The handshake ----
|
||||
|
||||
[Fact]
|
||||
public async Task WithoutAToken_TheUpgradeIsRefused()
|
||||
{
|
||||
var client = fixture.Server.CreateWebSocketClient();
|
||||
client.SubProtocols.Add(VaultEvents.SubProtocol);
|
||||
|
||||
var connecting = client.ConnectAsync(
|
||||
new Uri(fixture.Server.BaseAddress, VaultEvents.Path.TrimStart('/')),
|
||||
TestContext.Current.CancellationToken);
|
||||
|
||||
await Should.ThrowAsync<InvalidOperationException>(connecting);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task BeforeEnrolling_Is403WithAnActionableCode()
|
||||
{
|
||||
// The same bar as sync: a caller with no identity key holds no vault key either, so every
|
||||
// notice this socket could carry is about ciphertext they cannot read.
|
||||
var client = fixture.CreateClientFor(NewSubject());
|
||||
|
||||
var response = await client.GetAsync(
|
||||
new Uri(VaultEvents.Path, UriKind.Relative),
|
||||
TestContext.Current.CancellationToken);
|
||||
|
||||
response.StatusCode.ShouldBe(HttpStatusCode.Forbidden);
|
||||
|
||||
var problem = await response.Content.ReadProblemAsync();
|
||||
problem.ShouldNotBeNull();
|
||||
problem.Code.ShouldBe(ProblemCodes.EnrollmentRequired);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task APlainGet_SaysItIsAWebSocket()
|
||||
{
|
||||
// A person, or a client with the wrong URL. Answering with a problem document rather than a
|
||||
// socket that closes is the difference between a diagnosable mistake and a mysterious one.
|
||||
var client = fixture.CreateClientFor(await SeedEnrolledUserAsync());
|
||||
|
||||
var response = await client.GetAsync(
|
||||
new Uri(VaultEvents.Path, UriKind.Relative),
|
||||
TestContext.Current.CancellationToken);
|
||||
|
||||
response.StatusCode.ShouldBe(HttpStatusCode.BadRequest);
|
||||
|
||||
var problem = await response.Content.ReadProblemAsync();
|
||||
problem.ShouldNotBeNull();
|
||||
problem.Code.ShouldBe(ProblemCodes.MalformedRequest);
|
||||
problem.Detail.ShouldNotBeNull().ShouldContain(VaultEvents.SubProtocol);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnUpgradeWithoutTheSubprotocol_IsRefused()
|
||||
{
|
||||
// The subprotocol is this socket's version negotiation, so accepting an upgrade that did not
|
||||
// offer it would mean answering a client in a dialect it never agreed to read.
|
||||
var (subject, _) = await SeedUserWithVaultAsync();
|
||||
|
||||
var client = fixture.Server.CreateWebSocketClient();
|
||||
client.ConfigureRequest = request => request.Headers.Authorization =
|
||||
$"Bearer {fixture.IdentityProvider.MintToken(subject)}";
|
||||
|
||||
var connecting = client.ConnectAsync(
|
||||
new Uri(fixture.Server.BaseAddress, VaultEvents.Path.TrimStart('/')),
|
||||
TestContext.Current.CancellationToken);
|
||||
|
||||
await Should.ThrowAsync<InvalidOperationException>(connecting);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TheFirstFrameIsHello()
|
||||
{
|
||||
var (subject, _) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
var hello = await ReadAsync(socket, timeout.Token);
|
||||
|
||||
hello.Kind.ShouldBe(VaultEventKinds.Hello);
|
||||
|
||||
// Sent so a client knows when silence means the socket is dead rather than quiet, and so a
|
||||
// socket following nothing — a real state, for an account with no vaults — is distinguishable
|
||||
// from one that is broken.
|
||||
hello.HeartbeatSeconds.ShouldNotBeNull().ShouldBeGreaterThan(0);
|
||||
hello.VaultCount.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MetaAdvertisesTheFeature()
|
||||
{
|
||||
// How a client decides whether to hold a socket open at all. Absence is not an error — it
|
||||
// means synchronise on the timer, which is what every client did before this existed.
|
||||
var client = fixture.CreateClient();
|
||||
|
||||
var meta = await (await client.GetAsync(
|
||||
new Uri("/api/v1/meta", UriKind.Relative),
|
||||
TestContext.Current.CancellationToken))
|
||||
.Content.ReadContractAsync<MetaResponse>();
|
||||
|
||||
meta.ShouldNotBeNull();
|
||||
meta.Features.ShouldContain(
|
||||
feature => string.Equals(feature, VaultEvents.Feature, StringComparison.Ordinal));
|
||||
}
|
||||
|
||||
// ---- Notices ----
|
||||
|
||||
[Fact]
|
||||
public async Task APush_AnnouncesTheVaultToAFollowingSocket()
|
||||
{
|
||||
var (subject, vaultId) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
var client = fixture.CreateClientFor(subject);
|
||||
var push = await client.PostContractAsync(PushUrl(vaultId), NewCreateBatch());
|
||||
push.EnsureSuccessStatusCode();
|
||||
|
||||
var notice = await ReadUntilAsync(socket, VaultEventKinds.VaultChanged, timeout.Token);
|
||||
|
||||
notice.VaultId.ShouldBe(vaultId);
|
||||
|
||||
// A hint for logging and coalescing, never a cursor: cursors are opaque and integrity-tagged,
|
||||
// and a client that tried to resume from this would be resuming from a number it invented.
|
||||
notice.Sequence.ShouldNotBeNull().ShouldBeGreaterThan(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ANoticeCarriesNoCiphertext()
|
||||
{
|
||||
// The load-bearing property of the whole design. A notice says only that a vault moved; the
|
||||
// client's answer is the delta pull it would have run on its timer anyway, which keeps exactly
|
||||
// one code path applying changes. See ADR 0012.
|
||||
var (subject, vaultId) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
var client = fixture.CreateClientFor(subject);
|
||||
var batch = NewCreateBatch();
|
||||
await client.PostContractAsync(PushUrl(vaultId), batch);
|
||||
|
||||
var raw = await ReadRawUntilAsync(socket, VaultEventKinds.VaultChanged, timeout.Token);
|
||||
|
||||
// The envelope this test pushed, as it would appear if a payload had been forwarded.
|
||||
raw.ShouldNotContain(Convert.ToBase64String(batch.Operations[0].Payload!.Envelope));
|
||||
raw.ShouldNotContain("payload", Case.Insensitive);
|
||||
raw.ShouldNotContain(batch.Operations[0].EntityId.ToString());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task APushToAnotherAccountsVault_IsNotAnnouncedHere()
|
||||
{
|
||||
// The disclosure that would matter: a socket learning that vault ids it cannot read exist,
|
||||
// and when somebody works on them.
|
||||
var (subject, vaultId) = await SeedUserWithVaultAsync();
|
||||
var (stranger, strangersVaultId) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
// The stranger's write goes first, so a socket that leaked would have announced it before the
|
||||
// one this test then waits for. Ordering, not a timeout: "nothing arrived in two seconds"
|
||||
// passes on a slow machine for the wrong reason.
|
||||
var strangersClient = fixture.CreateClientFor(stranger);
|
||||
(await strangersClient.PostContractAsync(PushUrl(strangersVaultId), NewCreateBatch()))
|
||||
.EnsureSuccessStatusCode();
|
||||
|
||||
var ownClient = fixture.CreateClientFor(subject);
|
||||
(await ownClient.PostContractAsync(PushUrl(vaultId), NewCreateBatch()))
|
||||
.EnsureSuccessStatusCode();
|
||||
|
||||
var notice = await ReadUntilAsync(socket, VaultEventKinds.VaultChanged, timeout.Token);
|
||||
|
||||
notice.VaultId.ShouldBe(vaultId);
|
||||
notice.VaultId.ShouldNotBe(strangersVaultId);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task APushThatAppliedNothing_AnnouncesNothing()
|
||||
{
|
||||
// A batch of pure conflicts moved no vault. Announcing one anyway would have every client on
|
||||
// it pull for a change that is not there.
|
||||
var (subject, vaultId) = await SeedUserWithVaultAsync();
|
||||
var client = fixture.CreateClientFor(subject);
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
// An update to an item that does not exist: rejected as a conflict, nothing written.
|
||||
var stale = new SyncPushRequest(
|
||||
[
|
||||
NewOperation(Guid.CreateVersion7(), expectedVersion: 7, envelope: [9, 9]),
|
||||
]);
|
||||
|
||||
var conflicted = await client.PostContractAsync(PushUrl(vaultId), stale);
|
||||
conflicted.EnsureSuccessStatusCode();
|
||||
|
||||
var results = await conflicted.Content.ReadContractAsync<SyncPushResponse>();
|
||||
results.ShouldNotBeNull();
|
||||
results.Results[0].Status.ShouldBe(SyncOperationStatus.Conflict);
|
||||
|
||||
// Then a write that did land. The first notice must be that one.
|
||||
(await client.PostContractAsync(PushUrl(vaultId), NewCreateBatch())).EnsureSuccessStatusCode();
|
||||
|
||||
var notice = await ReadUntilAsync(socket, VaultEventKinds.VaultChanged, timeout.Token);
|
||||
|
||||
notice.Sequence.ShouldNotBeNull().ShouldBeGreaterThan(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task APing_IsAnswered()
|
||||
{
|
||||
var (subject, _) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
await SendAsync(socket, new VaultEvent(VaultEventKinds.Ping), timeout.Token);
|
||||
|
||||
var pong = await ReadUntilAsync(socket, VaultEventKinds.Pong, timeout.Token);
|
||||
|
||||
pong.Kind.ShouldBe(VaultEventKinds.Pong);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AFrameThisServerCannotRead_DoesNotEndTheSocket()
|
||||
{
|
||||
// A control channel whose failure mode is "the client polls instead" should tolerate a frame
|
||||
// from a newer client rather than cost that client its push for the whole session.
|
||||
var (subject, vaultId) = await SeedUserWithVaultAsync();
|
||||
|
||||
using var timeout = Timeout();
|
||||
using var socket = await fixture.ConnectEventsAsync(subject, timeout.Token);
|
||||
|
||||
await ReadAsync(socket, timeout.Token);
|
||||
|
||||
await socket.SendAsync(
|
||||
Encoding.UTF8.GetBytes("{ not json at all"),
|
||||
WebSocketMessageType.Text,
|
||||
endOfMessage: true,
|
||||
timeout.Token);
|
||||
|
||||
var client = fixture.CreateClientFor(subject);
|
||||
(await client.PostContractAsync(PushUrl(vaultId), NewCreateBatch())).EnsureSuccessStatusCode();
|
||||
|
||||
var notice = await ReadUntilAsync(socket, VaultEventKinds.VaultChanged, timeout.Token);
|
||||
|
||||
notice.VaultId.ShouldBe(vaultId);
|
||||
socket.State.ShouldBe(WebSocketState.Open);
|
||||
}
|
||||
|
||||
// ---- Helpers ----
|
||||
|
||||
/// <summary>A token that gives up rather than letting a broken build hang the suite.</summary>
|
||||
private static CancellationTokenSource Timeout()
|
||||
{
|
||||
var source = CancellationTokenSource.CreateLinkedTokenSource(
|
||||
TestContext.Current.CancellationToken);
|
||||
|
||||
source.CancelAfter(FrameTimeout);
|
||||
|
||||
return source;
|
||||
}
|
||||
|
||||
private static async Task<VaultEvent> ReadAsync(WebSocket socket, CancellationToken cancellationToken)
|
||||
{
|
||||
var json = await ReadRawAsync(socket, cancellationToken);
|
||||
|
||||
return JsonSerializer.Deserialize(json, DodoSshJsonContext.Default.VaultEvent)
|
||||
?? throw new InvalidOperationException($"The server sent a null frame: {json}");
|
||||
}
|
||||
|
||||
private static async Task<string> ReadRawAsync(WebSocket socket, CancellationToken cancellationToken)
|
||||
{
|
||||
var buffer = new byte[8 * 1024];
|
||||
|
||||
var received = await socket.ReceiveAsync(buffer, cancellationToken);
|
||||
|
||||
if (received.MessageType == WebSocketMessageType.Close)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"The server closed the socket: {received.CloseStatus} {received.CloseStatusDescription}");
|
||||
}
|
||||
|
||||
return Encoding.UTF8.GetString(buffer, 0, received.Count);
|
||||
}
|
||||
|
||||
/// <summary>Reads past the frames a test does not care about — hello, and heartbeats.</summary>
|
||||
private static async Task<VaultEvent> ReadUntilAsync(
|
||||
WebSocket socket,
|
||||
string kind,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
var frame = await ReadAsync(socket, cancellationToken);
|
||||
|
||||
if (string.Equals(frame.Kind, kind, StringComparison.Ordinal))
|
||||
{
|
||||
return frame;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The same, but keeping the bytes.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Deserialising and asserting on the fields would prove only that this <em>record</em> has no
|
||||
/// payload member, which is a tautology. Asserting on what actually crossed the socket is what
|
||||
/// would catch a field added to the frame later without anybody thinking about disclosure.
|
||||
/// </remarks>
|
||||
private static async Task<string> ReadRawUntilAsync(
|
||||
WebSocket socket,
|
||||
string kind,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
var json = await ReadRawAsync(socket, cancellationToken);
|
||||
var frame = JsonSerializer.Deserialize(json, DodoSshJsonContext.Default.VaultEvent);
|
||||
|
||||
if (string.Equals(frame?.Kind, kind, StringComparison.Ordinal))
|
||||
{
|
||||
return json;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static Task SendAsync(WebSocket socket, VaultEvent frame, CancellationToken cancellationToken) =>
|
||||
socket.SendAsync(
|
||||
JsonSerializer.SerializeToUtf8Bytes(frame, DodoSshJsonContext.Default.VaultEvent),
|
||||
WebSocketMessageType.Text,
|
||||
endOfMessage: true,
|
||||
cancellationToken);
|
||||
|
||||
private static string PushUrl(Guid vaultId) => $"/api/v1/vaults/{vaultId}/sync/push";
|
||||
|
||||
private static string NewSubject() => $"events-{Guid.CreateVersion7():N}";
|
||||
|
||||
private static SyncPushOperation NewOperation(Guid entityId, int? expectedVersion, byte[] envelope) =>
|
||||
new(
|
||||
Guid.CreateVersion7(),
|
||||
SyncEntityType.Host,
|
||||
entityId,
|
||||
SyncOperation.Upsert,
|
||||
expectedVersion,
|
||||
new EncryptedPayload(envelope, [0xD, 0xE], Guid.CreateVersion7(), 1, 1),
|
||||
new SyncPlaintextFields());
|
||||
|
||||
private static SyncPushRequest NewCreateBatch() =>
|
||||
new([NewOperation(Guid.CreateVersion7(), expectedVersion: null, envelope: [1, 2, 3, 4])]);
|
||||
|
||||
private async Task<string> SeedEnrolledUserAsync()
|
||||
{
|
||||
var subject = NewSubject();
|
||||
|
||||
await using var scope = fixture.CreateScope();
|
||||
var database = scope.ServiceProvider.GetRequiredService<DodoDbContext>();
|
||||
|
||||
var user = NewUser(subject);
|
||||
database.Users.Add(user);
|
||||
database.UserKeys.Add(Seed.CurrentKey(user.Id, Now));
|
||||
await database.SaveChangesAsync();
|
||||
|
||||
return subject;
|
||||
}
|
||||
|
||||
private async Task<(string Subject, Guid VaultId)> SeedUserWithVaultAsync()
|
||||
{
|
||||
var subject = NewSubject();
|
||||
|
||||
await using var scope = fixture.CreateScope();
|
||||
var database = scope.ServiceProvider.GetRequiredService<DodoDbContext>();
|
||||
|
||||
var user = NewUser(subject);
|
||||
|
||||
var vault = new Vault
|
||||
{
|
||||
Id = Guid.CreateVersion7(),
|
||||
Name = "Personal",
|
||||
OwnerKind = VaultOwnerKind.Personal,
|
||||
OwnerUserId = user.Id,
|
||||
KeyGeneration = 1,
|
||||
CreatedAtUtc = Now,
|
||||
UpdatedAtUtc = Now,
|
||||
};
|
||||
|
||||
database.Users.Add(user);
|
||||
database.UserKeys.Add(Seed.CurrentKey(user.Id, Now));
|
||||
database.Vaults.Add(vault);
|
||||
await database.SaveChangesAsync();
|
||||
|
||||
return (subject, vault.Id);
|
||||
}
|
||||
|
||||
private UserAccount NewUser(string subject) => new()
|
||||
{
|
||||
Id = Guid.CreateVersion7(),
|
||||
Issuer = fixture.IdentityProvider.Authority,
|
||||
Subject = subject,
|
||||
Status = UserStatus.Active,
|
||||
CreatedAtUtc = Now,
|
||||
UpdatedAtUtc = Now,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,388 @@
|
||||
using System.Net.WebSockets;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Channels;
|
||||
using DodoSSH.Client.Api;
|
||||
using DodoSSH.Contracts;
|
||||
|
||||
namespace DodoSSH.Client.Api.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The reconnection policy, which is what this class actually is.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// A dropped socket is the ordinary case here rather than the exception — laptops sleep, proxies time
|
||||
/// out, tokens expire, servers are redeployed — so the behaviour worth covering is what happens
|
||||
/// <em>after</em> a failure, not the happy path. Driven through the injected connector, because the one
|
||||
/// thing a test cannot do to a real network is make it fail on cue.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The backoff is configured down to milliseconds throughout. What is under test is the shape of the
|
||||
/// policy — does it try again, does it wait, does it stop waiting when told the token was the problem —
|
||||
/// and none of that depends on the intervals a shipped client uses.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class VaultEventStreamTests
|
||||
{
|
||||
private static readonly Uri Server = new("https://dodossh.example");
|
||||
|
||||
private static readonly VaultEventStreamOptions Impatient = new()
|
||||
{
|
||||
InitialBackoff = TimeSpan.FromMilliseconds(1),
|
||||
MaxBackoff = TimeSpan.FromMilliseconds(5),
|
||||
InitialSilenceTimeout = TimeSpan.FromSeconds(30),
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public async Task ItDialsTheWebSocketFormOfTheServersUrl()
|
||||
{
|
||||
// https becomes wss, and the path is the one in the contract. Getting either wrong is a client
|
||||
// that reconnects against a 404 for the whole session, which from outside is indistinguishable
|
||||
// from a network that eats WebSockets.
|
||||
var dialled = new List<Uri>();
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream(
|
||||
(url, _, _) =>
|
||||
{
|
||||
dialled.Add(url);
|
||||
return Task.FromResult<WebSocket>(socket);
|
||||
});
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
|
||||
await stream.ReadAsync(Token);
|
||||
|
||||
dialled[0].ShouldBe(new Uri("wss://dodossh.example/api/v1/events"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ItSendsTheBearerTokenOnTheUpgrade()
|
||||
{
|
||||
// The whole of this socket's authorization, unlike the relay's ticket. See ADR 0012.
|
||||
var presented = new List<string>();
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream(
|
||||
(_, token, _) =>
|
||||
{
|
||||
presented.Add(token);
|
||||
return Task.FromResult<WebSocket>(socket);
|
||||
});
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
|
||||
await stream.ReadAsync(Token);
|
||||
|
||||
presented[0].ShouldBe(StubTokens.Token);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ANoticeReachesTheReader()
|
||||
{
|
||||
var vaultId = Guid.CreateVersion7();
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream((_, _, _) => Task.FromResult<WebSocket>(socket));
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, vaultId, 42));
|
||||
|
||||
var notice = await stream.ReadAsync(Token);
|
||||
|
||||
notice.Kind.ShouldBe(VaultEventKinds.VaultChanged);
|
||||
notice.VaultId.ShouldBe(vaultId);
|
||||
notice.Sequence.ShouldBe(42);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AHeartbeatIsAnsweredAndNotHandedToTheReader()
|
||||
{
|
||||
// A ping is housekeeping between the two ends. Passing it up would wake a synchronisation loop
|
||||
// every thirty seconds for a frame that says nothing happened.
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream((_, _, _) => Task.FromResult<WebSocket>(socket));
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.Ping, HeartbeatSeconds: 30));
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
|
||||
var first = await stream.ReadAsync(Token);
|
||||
|
||||
first.Kind.ShouldBe(VaultEventKinds.VaultChanged, "the ping should not have been forwarded");
|
||||
|
||||
var answered = await socket.SentAsync(Token);
|
||||
answered.Kind.ShouldBe(VaultEventKinds.Pong);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AKindThisBuildDoesNotKnow_IsStillHandedOver()
|
||||
{
|
||||
// What makes the frame table extensible: this class must not decide what a newer server may
|
||||
// say. Deciding to ignore it is the caller's, and costs that caller one redundant pass.
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream((_, _, _) => Task.FromResult<WebSocket>(socket));
|
||||
|
||||
socket.Deliver(new VaultEvent("session.offered"));
|
||||
|
||||
var notice = await stream.ReadAsync(Token);
|
||||
|
||||
notice.Kind.ShouldBe("session.offered");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AFailedDial_IsRetried()
|
||||
{
|
||||
// No server yet, or no network. Neither is an error to report: the caller's synchronisation
|
||||
// timer is running regardless, which is what lets this stay silent and keep trying.
|
||||
var attempts = 0;
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream((_, _, _) =>
|
||||
{
|
||||
if (++attempts < 3)
|
||||
{
|
||||
throw new WebSocketException("no route to host");
|
||||
}
|
||||
|
||||
return Task.FromResult<WebSocket>(socket);
|
||||
});
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
|
||||
var notice = await stream.ReadAsync(Token);
|
||||
|
||||
notice.Kind.ShouldBe(VaultEventKinds.VaultChanged);
|
||||
attempts.ShouldBe(3);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ADroppedSocket_IsReplaced()
|
||||
{
|
||||
// The case that decides whether this feature survives a laptop lid. A stream that gave up on
|
||||
// the first close would work all morning and be silently dead after lunch.
|
||||
var sockets = new List<FakeWebSocket>();
|
||||
|
||||
await using var stream = Stream((_, _, _) =>
|
||||
{
|
||||
var socket = new FakeWebSocket();
|
||||
sockets.Add(socket);
|
||||
|
||||
if (sockets.Count == 1)
|
||||
{
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
socket.Close(WebSocketCloseStatus.EndpointUnavailable);
|
||||
}
|
||||
else
|
||||
{
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 2));
|
||||
}
|
||||
|
||||
return Task.FromResult<WebSocket>(socket);
|
||||
});
|
||||
|
||||
(await stream.ReadAsync(Token)).Sequence.ShouldBe(1);
|
||||
(await stream.ReadAsync(Token)).Sequence.ShouldBe(2);
|
||||
|
||||
sockets.Count.ShouldBeGreaterThanOrEqualTo(2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnExpiredTokenClose_ReconnectsAndAsksForAFreshToken()
|
||||
{
|
||||
// The bound that lets a long-lived socket be authorised by a short-lived credential: the server
|
||||
// closes at the token's expiry and the client comes straight back with a new one. The token
|
||||
// provider being asked again is the half that matters — reconnecting with the spent token would
|
||||
// be an unbroken loop of closes.
|
||||
var tokens = new StubTokens();
|
||||
var sockets = 0;
|
||||
|
||||
await using var stream = new VaultEventStream(
|
||||
Server,
|
||||
tokens,
|
||||
TimeProvider.System,
|
||||
(_, _, _) =>
|
||||
{
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
if (++sockets == 1)
|
||||
{
|
||||
socket.Close((WebSocketCloseStatus)VaultEvents.TokenExpiredCloseCode);
|
||||
}
|
||||
else
|
||||
{
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 7));
|
||||
}
|
||||
|
||||
return Task.FromResult<WebSocket>(socket);
|
||||
},
|
||||
Impatient);
|
||||
|
||||
(await stream.ReadAsync(Token)).Sequence.ShouldBe(7);
|
||||
|
||||
tokens.Requests.ShouldBeGreaterThanOrEqualTo(2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TryRead_TakesWhatIsWaitingAndSaysWhenNothingIs()
|
||||
{
|
||||
// How a caller coalesces a burst: read one, wait a moment, swallow the rest. Without this a
|
||||
// colleague tidying a folder would produce a synchronisation pass per item.
|
||||
var socket = new FakeWebSocket();
|
||||
|
||||
await using var stream = Stream((_, _, _) => Task.FromResult<WebSocket>(socket));
|
||||
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 1));
|
||||
socket.Deliver(new VaultEvent(VaultEventKinds.VaultChanged, Guid.CreateVersion7(), 2));
|
||||
|
||||
(await stream.ReadAsync(Token)).Sequence.ShouldBe(1);
|
||||
|
||||
// Delivery is asynchronous, so the second may not have landed yet; this is the same
|
||||
// wait-then-drain the caller performs.
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(200), Token);
|
||||
|
||||
stream.TryRead(out var queued).ShouldBeTrue();
|
||||
queued.Sequence.ShouldBe(2);
|
||||
|
||||
stream.TryRead(out _).ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnIdleStream_NeverDelivers()
|
||||
{
|
||||
// What a server without the feature supplies. Waiting for ever rather than completing is the
|
||||
// point: a caller selecting between this and a timer has to fall through to the timer, and a
|
||||
// read that returned at once would spin that loop as fast as the machine allows.
|
||||
using var stream = IdleVaultEventStream.Instance;
|
||||
using var giveUp = CancellationTokenSource.CreateLinkedTokenSource(Token);
|
||||
|
||||
giveUp.CancelAfter(TimeSpan.FromMilliseconds(100));
|
||||
|
||||
await Should.ThrowAsync<OperationCanceledException>(
|
||||
async () => await stream.ReadAsync(giveUp.Token));
|
||||
|
||||
stream.TryRead(out _).ShouldBeFalse();
|
||||
stream.IsConnected.ShouldBeFalse();
|
||||
}
|
||||
|
||||
private static CancellationToken Token => TestContext.Current.CancellationToken;
|
||||
|
||||
private static VaultEventStream Stream(
|
||||
Func<Uri, string, CancellationToken, Task<WebSocket>> connect) =>
|
||||
new(Server, new StubTokens(), TimeProvider.System, connect, Impatient);
|
||||
|
||||
/// <summary>A token provider that hands out one value and counts who asked.</summary>
|
||||
private sealed class StubTokens : IAccessTokenProvider
|
||||
{
|
||||
internal const string Token = "access-token";
|
||||
|
||||
internal int Requests { get; private set; }
|
||||
|
||||
public ValueTask<string> GetAccessTokenAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
Requests++;
|
||||
|
||||
return ValueTask.FromResult(Token);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A socket a test writes the server's half of.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Frames queued with <see cref="Deliver"/> are handed out by <see cref="ReceiveAsync"/> in order;
|
||||
/// once the queue is empty the receive waits, which is what an idle connection does. <see
|
||||
/// cref="Close"/> queues the close instead, so a test can script "two notices and then the server
|
||||
/// went away" as a value rather than as a race.
|
||||
/// </remarks>
|
||||
private sealed class FakeWebSocket : WebSocket
|
||||
{
|
||||
private readonly Channel<byte[]> inbound = Channel.CreateUnbounded<byte[]>();
|
||||
private readonly Channel<VaultEvent> outbound = Channel.CreateUnbounded<VaultEvent>();
|
||||
|
||||
private WebSocketCloseStatus? closing;
|
||||
private WebSocketState state = WebSocketState.Open;
|
||||
|
||||
public override WebSocketCloseStatus? CloseStatus => closing;
|
||||
|
||||
public override string? CloseStatusDescription => null;
|
||||
|
||||
public override WebSocketState State => state;
|
||||
|
||||
public override string? SubProtocol => VaultEvents.SubProtocol;
|
||||
|
||||
/// <summary>Queues a frame for the client to read.</summary>
|
||||
internal void Deliver(VaultEvent frame) =>
|
||||
inbound.Writer.TryWrite(
|
||||
JsonSerializer.SerializeToUtf8Bytes(frame, DodoSshJsonContext.Default.VaultEvent));
|
||||
|
||||
/// <summary>Ends the socket, after everything already queued has been read.</summary>
|
||||
internal void Close(WebSocketCloseStatus status)
|
||||
{
|
||||
closing = status;
|
||||
inbound.Writer.TryWrite([]);
|
||||
}
|
||||
|
||||
/// <summary>The next frame the client sent.</summary>
|
||||
internal ValueTask<VaultEvent> SentAsync(CancellationToken cancellationToken) =>
|
||||
outbound.Reader.ReadAsync(cancellationToken);
|
||||
|
||||
public override async Task<WebSocketReceiveResult> ReceiveAsync(
|
||||
ArraySegment<byte> buffer,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var frame = await inbound.Reader.ReadAsync(cancellationToken);
|
||||
|
||||
// The empty frame Close queues. Reported as a close, exactly as a real socket does once the
|
||||
// peer's close frame arrives.
|
||||
if (frame.Length == 0)
|
||||
{
|
||||
state = WebSocketState.Closed;
|
||||
|
||||
return new WebSocketReceiveResult(
|
||||
0, WebSocketMessageType.Close, endOfMessage: true, closing, null);
|
||||
}
|
||||
|
||||
frame.CopyTo(buffer.Array!, buffer.Offset);
|
||||
|
||||
return new WebSocketReceiveResult(frame.Length, WebSocketMessageType.Text, endOfMessage: true);
|
||||
}
|
||||
|
||||
public override Task SendAsync(
|
||||
ArraySegment<byte> buffer,
|
||||
WebSocketMessageType messageType,
|
||||
bool endOfMessage,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var json = Encoding.UTF8.GetString(buffer.Array!, buffer.Offset, buffer.Count);
|
||||
|
||||
if (JsonSerializer.Deserialize(json, DodoSshJsonContext.Default.VaultEvent) is { } frame)
|
||||
{
|
||||
outbound.Writer.TryWrite(frame);
|
||||
}
|
||||
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public override void Abort() => state = WebSocketState.Aborted;
|
||||
|
||||
public override Task CloseAsync(
|
||||
WebSocketCloseStatus closeStatus,
|
||||
string? statusDescription,
|
||||
CancellationToken cancellationToken) => CloseOutputAsync(
|
||||
closeStatus, statusDescription, cancellationToken);
|
||||
|
||||
public override Task CloseOutputAsync(
|
||||
WebSocketCloseStatus closeStatus,
|
||||
string? statusDescription,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
state = WebSocketState.Closed;
|
||||
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public override void Dispose() => state = WebSocketState.Closed;
|
||||
}
|
||||
}
|
||||
@@ -57,6 +57,17 @@ internal sealed class StubTeamServer : IVaultServer, ITeamApi, IVaultGrantApi
|
||||
/// <inheritdoc />
|
||||
public IKeyBindingAuthorizer KeyBinding => throw new NotSupportedException();
|
||||
|
||||
/// <summary>
|
||||
/// A push channel that never pushes.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Not <c>NotSupportedException</c> like its neighbours: the background synchronisation loop reads
|
||||
/// this on every wait, so a layout test that opened a screen would throw from a timer thread rather
|
||||
/// than draw anything. Waiting for ever is the honest stand-in — an offline layout test has no
|
||||
/// server to be pushed from.
|
||||
/// </remarks>
|
||||
public IVaultEventStream Events => IdleVaultEventStream.Instance;
|
||||
|
||||
/// <inheritdoc />
|
||||
public SyncOptions SyncOptions => new();
|
||||
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
using System.Threading.Channels;
|
||||
using DodoSSH.Client.Api;
|
||||
using DodoSSH.Contracts;
|
||||
|
||||
namespace DodoSSH.Client.App.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// A server's push channel, driven by a test rather than by a socket.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The real <c>VaultEventStream</c> is a reconnection policy wrapped round a WebSocket, and none of
|
||||
/// that is what the shell's behaviour depends on: what the shell does with a notice is the same
|
||||
/// whether it arrived over a healthy socket, after four reconnections, or from this. Driving it by
|
||||
/// hand is what makes "the loop synchronised because it was told to, not because a minute passed" a
|
||||
/// test that finishes in milliseconds and cannot flake.
|
||||
/// </remarks>
|
||||
internal sealed class FakeVaultEventStream : IVaultEventStream
|
||||
{
|
||||
private readonly Channel<VaultEvent> notices = Channel.CreateUnbounded<VaultEvent>();
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool IsConnected => true;
|
||||
|
||||
/// <summary>How many times the shell has waited on this. Proves the loop is watching at all.</summary>
|
||||
internal int Reads { get; private set; }
|
||||
|
||||
/// <summary>Delivers a notice, as a server would.</summary>
|
||||
internal void Push(Guid vaultId, long sequence = 1) =>
|
||||
notices.Writer.TryWrite(
|
||||
new VaultEvent(VaultEventKinds.VaultChanged, vaultId, sequence));
|
||||
|
||||
/// <summary>Delivers the notice that says the caller's vault list has changed.</summary>
|
||||
internal void PushAccessChanged() =>
|
||||
notices.Writer.TryWrite(new VaultEvent(VaultEventKinds.VaultsChanged));
|
||||
|
||||
/// <inheritdoc />
|
||||
public ValueTask<VaultEvent> ReadAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
Reads++;
|
||||
|
||||
return notices.Reader.ReadAsync(cancellationToken);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool TryRead(out VaultEvent notice) => notices.Reader.TryRead(out notice!);
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose() => notices.Writer.TryComplete();
|
||||
}
|
||||
@@ -39,6 +39,16 @@ internal sealed partial class FakeVaultServer : IVaultServer, IAccountApi, ISync
|
||||
|
||||
internal int PushCount { get; private set; }
|
||||
|
||||
/// <summary>
|
||||
/// How many delta reads this server has served.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The one observable a synchronisation pass always produces. <see cref="PushCount"/> only moves when
|
||||
/// there is something queued, so a test asking "did a pass run" — which is what the push channel's
|
||||
/// whole purpose comes down to — has to count pulls.
|
||||
/// </remarks>
|
||||
internal int PullCount { get; private set; }
|
||||
|
||||
internal bool IsEnrolled => statement is not null;
|
||||
|
||||
/// <summary>
|
||||
@@ -99,6 +109,19 @@ internal sealed partial class FakeVaultServer : IVaultServer, IAccountApi, ISync
|
||||
/// <inheritdoc />
|
||||
public IKeyBindingAuthorizer KeyBinding => this;
|
||||
|
||||
/// <summary>
|
||||
/// The push channel, which a test drives by hand.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A real queue rather than an idle stand-in, because the behaviour worth covering here is the one
|
||||
/// the socket exists for: a notice arriving makes the background loop synchronise without waiting
|
||||
/// out its minute. See <see cref="FakeVaultEventStream.Push"/>.
|
||||
/// </remarks>
|
||||
internal FakeVaultEventStream Notices { get; } = new();
|
||||
|
||||
/// <inheritdoc />
|
||||
public IVaultEventStream Events => Notices;
|
||||
|
||||
/// <inheritdoc />
|
||||
public SyncOptions SyncOptions => SyncOptions.Default;
|
||||
|
||||
@@ -238,6 +261,8 @@ internal sealed partial class FakeVaultServer : IVaultServer, IAccountApi, ISync
|
||||
SyncPullRequest request,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
PullCount++;
|
||||
|
||||
if (SyncFailure is { } failure)
|
||||
{
|
||||
return Task.FromException<SyncPullResponse>(failure);
|
||||
|
||||
@@ -523,6 +523,90 @@ public sealed class ShellFlowTests : IAsyncLifetime
|
||||
vault.Status.ShouldContain("bad day");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The whole point of the push channel: a pass that did not wait for the minute.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The timing is what makes this an assertion rather than a hope. The background timer is a full
|
||||
/// minute and the wait below gives up in ten seconds, so a pull that arrives can only have been
|
||||
/// caused by the notice — there is no interval at which the timer could have produced it.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The vault id in the notice is arbitrary, and deliberately so: a pass synchronises every vault
|
||||
/// this session can reach, so the loop reads the notice as "there is something to fetch" and never
|
||||
/// as "fetch this one". A test that seeded a real id would imply a targeting this does not do.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
[Fact]
|
||||
public async Task APushedNotice_SynchronisesWithoutWaitingForTheTimer()
|
||||
{
|
||||
await UnlockedAsync();
|
||||
|
||||
// The unlock starts the loop, whose first act is a pass; waited out so the count below is a
|
||||
// baseline rather than a race with it.
|
||||
await EventuallyAsync(
|
||||
() => server.PullCount > 0,
|
||||
"the pass on open should have run");
|
||||
|
||||
var before = server.PullCount;
|
||||
|
||||
server.Notices.Push(Guid.CreateVersion7());
|
||||
|
||||
await EventuallyAsync(
|
||||
() => server.PullCount > before,
|
||||
"a notice should have woken the loop long before the one-minute timer");
|
||||
}
|
||||
|
||||
/// <remarks>
|
||||
/// The half that is easy to get wrong. The loop selects between two waits, and both have to survive
|
||||
/// losing: <c>PeriodicTimer</c> throws if a second wait is started while one is outstanding, and an
|
||||
/// abandoned channel read stays registered and swallows the next notice written. Either defect
|
||||
/// leaves the first notice working and every one after it silently lost, which is why one notice is
|
||||
/// not enough to prove this.
|
||||
/// </remarks>
|
||||
[Fact]
|
||||
public async Task NoticesKeepWakingTheLoop_NotJustTheFirst()
|
||||
{
|
||||
await UnlockedAsync();
|
||||
|
||||
await EventuallyAsync(() => server.PullCount > 0, "the pass on open should have run");
|
||||
|
||||
for (var round = 1; round <= 3; round++)
|
||||
{
|
||||
var before = server.PullCount;
|
||||
|
||||
server.Notices.Push(Guid.CreateVersion7());
|
||||
|
||||
await EventuallyAsync(
|
||||
() => server.PullCount > before,
|
||||
$"notice {round} should have woken the loop as the first one did");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Waits for something a background loop is expected to do, or fails saying what.</summary>
|
||||
/// <remarks>
|
||||
/// Polled rather than signalled because the thing under test is a loop nobody hands a completion
|
||||
/// source to. The bound is generous — this is not measuring latency, only proving that the timer
|
||||
/// cannot be what caused the result.
|
||||
/// </remarks>
|
||||
private static async Task EventuallyAsync(Func<bool> condition, string because)
|
||||
{
|
||||
var deadline = TimeProvider.System.GetUtcNow().AddSeconds(10);
|
||||
|
||||
while (TimeProvider.System.GetUtcNow() < deadline)
|
||||
{
|
||||
if (condition())
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(20), Token);
|
||||
}
|
||||
|
||||
throw new ShouldAssertException(because);
|
||||
}
|
||||
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The page's own <c>term.focus()</c> focuses the textarea inside the document, which does nothing
|
||||
|
||||
Reference in New Issue
Block a user