Public Access
Say when a vault has moved, so nobody waits out the minute
The delta pull was cheap enough to run on a timer and the client did, once a minute. That is fine for a machine and wrong for two people: an edit a colleague makes is up to a minute stale, which is long enough for both of them to make it and produce a conflict neither needed to have. Shortening the interval is the obvious answer and the wrong one — it costs a request per client per interval whether or not anything happened, and it converges on a busier server that is still late. So the server now says so. A client holds a WebSocket open at GET /api/v1/events, subprotocol dodossh.events.v1, and gets a line down it when something it can read has changed. ADR 0012 has the reasoning; three parts of it are worth repeating here, because they are what everything else rests on. **What crosses the socket is a notice, never data.** A frame names a vault and how far its change log has got. No item, no ciphertext, not even which item it was. The client's answer is the delta pull it would have run anyway, so there is still exactly one code path that applies a change to a keychain, and it is not this one. Pushing the items themselves would save a round trip and fork that path in two, with the cursor, the merge and the tombstone rules duplicated across both — ADR 0003 put every mutation through one write path for that reason, and this keeps every read on one for the same one. It also makes a dropped notice harmless, which is what lets the fan-out below be as simple as it is. **Polling stays, and is what guarantees a pass.** The minute timer is unchanged. A network that eats WebSockets, a server with Events:Enabled off, an older server, a proxy that will not upgrade, a notice dropped under backpressure — every one of those leaves a client behaving exactly as it did before this commit. Nothing is reachable only over the socket and nothing is meant to become so; VaultViewModel's AutoSyncInterval remark now says that where somebody changing it will read it. **The bearer token authorises the upgrade, unlike the relay's ticket.** Not an inconsistency with ADR 0004: the relay's socket is a byte pipe whose whole authorization decision — which host, which IPs, which port — is made before it opens and never revisited, and it is the extraction seam for a process that must hold no ACL code. This one is a view of the caller's own vault list and has to keep answering "what may this account read" for as long as it is held. A ticket would carry that answer in a token and be wrong the moment the account's access changed. The two bounds that arrangement needs are met rather than waved at: the socket is closed at the token's exp with close code 4401 and the client comes straight back with a fresh one, and the vault set is re-resolved every few minutes as well as on the changes known to affect it. Both bound *metadata*, because a notice contains nothing else and reading a vault still needs a key this server has never held. **The fan-out.** VaultEventHub is a singleton holding the sockets this node accepted; publishing walks them and asks each whether it cares, rather than keeping a vault-to-subscriber index that every re-subscription would have to move entries between under a lock publishing also takes. At a few hundred sockets per node and an event rate bounded by how often people edit keychains, the walk is not measurable and its races are obvious. Per-connection queues are bounded and drop the *oldest*: a notice means "pull vault X, which is at least at sequence N", so the newest subsumes what it displaces and the client's answer is identical either way — which is what lets the publish path be void, never block, and never fail. Announced from the endpoint rather than from SyncService, and that placement is the point: by then the push has committed and released the per-vault advisory lock. From inside it would name a sequence no reader can see yet and would hold the lock that serialises writers across a socket write. Only the highest *applied* sequence, so a batch of pure conflicts announces nothing, and a duplicate — already announced when it first landed — announces nothing either. Grants and membership publish too, and those take the *recipient* rather than the actor. This is what AdmitNewVaultsAsync has been apologising for since sharing shipped — "the recipient is handed nothing, there is no push channel" — and the README with it. A vault shared with somebody now turns up as it is shared. The comment and the README paragraph both say what is true now, and both keep saying that the pass is what *discovers* the vault, because a client with no socket has to arrive at the same place. **On the client**, VaultEventStream is really a reconnection policy wrapped round a ClientWebSocket: a dropped socket is the ordinary case here — laptops sleep, proxies time out, tokens expire, servers are redeployed — so nothing in it treats a failure as exceptional, and every path ends in "wait, then dial again". A connection that lived long enough to say hello resets the backoff, so a laptop that woke, worked, and lost its network an hour later does not inherit a minute-long wait it has already proved it need not take. A 4401 close skips the backoff entirely and asks the token provider again, which is the whole reason that close code is distinct. A server that does not advertise the events feature gets IdleVaultEventStream, which never delivers — so IVaultServer.Events is never null and every caller stays on one shape, because the correct behaviour without a socket is the behaviour with a silent one. The shell's background loop now selects between the timer and a notice, and both waits are held across iterations. That is load-bearing rather than tidy: PeriodicTimer permits one outstanding WaitForNextTickAsync and throws on a second, and an abandoned channel read stays registered and consumes the next notice written. Either defect leaves the first notice working and every one after it silently lost, which is why NoticesKeepWakingTheLoop_NotJustTheFirst pushes three and not one. Notices are coalesced over a quarter of a second, so one person's save — a host and its log entry are two items — and a colleague clearing a folder each cost one pass rather than a dozen. **The kind is a string, not an enum**, and that is a compatibility decision. UseStringEnumConverter throws on a value it does not know, so a newer server sending a kind an older client had never heard of would not add an unreadable frame — it would break that client's socket outright. A string is ignored instead. ProblemCodes is the same shape for the same reason. **Tested on both sides, through the real pipeline.** The endpoint suite opens a genuine socket against TestServer and proves a push produces a notice, that another account's push does not reach it, that a ping is answered, and that a frame this server cannot parse does not end the connection. Two of those assert on *ordering* rather than on absence within a timeout — the stranger's write goes first, so a socket that leaked would have announced it before the one the test waits for — because "nothing arrived in two seconds" is a test that passes on a slow machine for the wrong reason. And ANoticeCarriesNoCiphertext asserts on the bytes that crossed the wire rather than on the record's fields, since the latter would only prove that this type has no payload member, which is a tautology; the former is what catches a field added later without anybody thinking about disclosure. The client suite drives VaultEventStream through an injected connector, because the one thing a test cannot do to a real network is make it fail on cue — and failure is the entire subject. The shell suite proves a notice produces a pull inside ten seconds against a sixty-second timer, so the timer cannot be what caused it. **Two limits, stated rather than left to be discovered.** Fan-out is in-process, so a deployment running more than one API replica only pushes for writes its own replica handled and the rest arrive on the timer. IVaultEventPublisher is the seam a PostgreSQL LISTEN/NOTIFY backplane implements and it is deliberately not implemented: an untested backplane is worse than a documented gap, and multiple replicas degrade to the behaviour before this commit rather than breaking. And a client is notified of its own writes; it pushed, so it already pulled, and the extra pass finds nothing. Suppressing that echo correctly needs a per-device identity on the socket, and the same user's other machines must still be told. Manual checks phase 15 covers what no test here can reach, which is the network in between: a proxy that will not upgrade, one that drops an idle socket without telling either end, a laptop lid, a token expiring. Every one of those is invisible inside a test host, and every check there passes only if the change arrives quickly *and* still arrives with the socket taken away. ADR 0012 also fixes one thing about the shared terminal session this is the transport for, so it need not be renegotiated later: session data will be binary frames on this same socket, because base64 in a JSON envelope is the wrong shape for the one payload here that is continuous rather than occasional. Two questions it explicitly does not answer by implication — whether those bytes go through the API at all, and what end-to-end encryption means when the second party watches a stream rather than holding a key — are ADR 0001 questions and get their own decision. 1512 tests pass. DodoSSH.SystemTests was not run — it needs the whole compose stack — so the end-to-end path is unverified for this change beyond what the manual checks describe.
This commit is contained in:
jaap-jan committed
1 parent
176df67861
commit
4b706bc3c3
31 files changed
+3285
-13
No files matched your search
@@ -0,0 +1,533 @@
|
||||
using System.Collections.Frozen;
|
||||
using System.Globalization;
|
||||
using System.Net.WebSockets;
|
||||
using System.Security.Claims;
|
||||
using System.Text.Json;
|
||||
using DodoSSH.Api.Authorization;
|
||||
using DodoSSH.Api.Setup;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain.Authorization;
|
||||
using FastEndpoints;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace DodoSSH.Api.Features.Events;
|
||||
|
||||
/// <summary>
|
||||
/// The socket that says "pull now" so a client does not have to wait for its timer.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Everything this endpoint sends is a <em>notice</em>. It never carries an item, a payload or a
|
||||
/// cursor: the client's answer to a notice is the delta pull it would have run on its own anyway, so
|
||||
/// there is exactly one code path that applies a change and this is not it. See ADR 0012 for why
|
||||
/// pushing the items themselves is refused.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The bearer token authorises the upgrade, unlike the relay's ticket in ADR 0004. The relay's socket
|
||||
/// is a byte pipe whose whole authorization decision is made before it opens; this one is a view of
|
||||
/// the caller's own vault list and has to keep answering "what may this account read" for as long as
|
||||
/// it is held. Its two bounds on that — the token's own expiry, and a periodic re-resolve — are in
|
||||
/// <see cref="MindAsync"/>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal sealed class VaultEventsEndpoint(
|
||||
IServiceScopeFactory scopes,
|
||||
VaultEventHub hub,
|
||||
IOptions<EventsOptions> options,
|
||||
TimeProvider clock,
|
||||
IHostApplicationLifetime lifetime,
|
||||
ILogger<VaultEventsEndpoint> logger)
|
||||
: EndpointWithoutRequest
|
||||
{
|
||||
/// <summary>
|
||||
/// The largest message this endpoint will read from a client.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A client sends nothing but <c>ping</c>, so the cap is three orders of magnitude of headroom and
|
||||
/// still small enough that a hostile client cannot make the server buffer anything worth having.
|
||||
/// </remarks>
|
||||
private const int MaxInboundFrameBytes = 4 * 1024;
|
||||
|
||||
/// <summary>How long to wait for the close handshake before dropping the socket.</summary>
|
||||
private static readonly TimeSpan CloseTimeout = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <inheritdoc />
|
||||
public override void Configure()
|
||||
{
|
||||
Get(VaultEvents.Path);
|
||||
|
||||
// Enrolled, matching sync. A caller with no identity key holds no vault key either, so every
|
||||
// notice this socket could send is about ciphertext they cannot read.
|
||||
Policies(Auth.EnrolledPolicy);
|
||||
|
||||
Description(b => b
|
||||
.WithName("VaultEvents")
|
||||
.WithSummary("Pushes a notice when a vault the caller can read has changed.")
|
||||
.WithTags("Events"));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task HandleAsync(CancellationToken ct)
|
||||
{
|
||||
if (await RefusedAsync().ConfigureAwait(false))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var (userId, vaults) = await ResolveAccessAsync(ct).ConfigureAwait(false);
|
||||
|
||||
// Admitted before the upgrade so a refusal costs nothing, but answered *through* the socket
|
||||
// rather than as an HTTP status: a constrained WebSocket client cannot read the status of a
|
||||
// failed upgrade, and "you have too many open" is precisely the case where the client needs
|
||||
// to know to back off rather than retry. Same reasoning as ADR 0004 on request headers.
|
||||
var connection = hub.TryAdmit(userId, vaults);
|
||||
|
||||
try
|
||||
{
|
||||
using var socket = await HttpContext.WebSockets
|
||||
.AcceptWebSocketAsync(new WebSocketAcceptContext { SubProtocol = VaultEvents.SubProtocol })
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (connection is null)
|
||||
{
|
||||
await CloseAsync(
|
||||
socket,
|
||||
new Closure(
|
||||
VaultEvents.TooManyConnectionsCloseCode, "Too many open event sockets."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
await PumpAsync(socket, connection, ct).ConfigureAwait(false);
|
||||
}
|
||||
finally
|
||||
{
|
||||
// Inside a try that starts *before* the upgrade, because an accept that throws — a client
|
||||
// that abandoned the handshake — would otherwise leave an admitted connection in the hub
|
||||
// for the life of the process, counting against this account's cap and taking a slot from
|
||||
// the sockets that did open.
|
||||
if (connection is not null)
|
||||
{
|
||||
hub.Remove(connection);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Answers the requests that are not an event socket at all, as ordinary HTTP.
|
||||
/// </summary>
|
||||
/// <returns>Whether a response was sent and the handler should stop.</returns>
|
||||
/// <remarks>
|
||||
/// All three answers are problem documents rather than bare statuses, because each one has a
|
||||
/// different remedy and a client that cannot tell them apart would retry the two that will never
|
||||
/// succeed. Answered before the upgrade, so a caller that got the handshake wrong reads why in a
|
||||
/// body rather than inferring it from a socket that closed.
|
||||
/// </remarks>
|
||||
private async Task<bool> RefusedAsync()
|
||||
{
|
||||
if (!options.Value.Enabled)
|
||||
{
|
||||
// 404 rather than 501: the feature is absent from this deployment, and /api/v1/meta does
|
||||
// not advertise it. A client that dialled anyway keeps polling, which is correct.
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status404NotFound,
|
||||
ProblemCodes.EventsUnavailable,
|
||||
"This server does not push vault changes. Synchronise on a timer instead; "
|
||||
+ "GET /api/v1/meta lists the features it does offer."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!HttpContext.WebSockets.IsWebSocketRequest)
|
||||
{
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status400BadRequest,
|
||||
ProblemCodes.MalformedRequest,
|
||||
"This endpoint is a WebSocket. Send an upgrade request offering the "
|
||||
+ $"'{VaultEvents.SubProtocol}' subprotocol."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
// The subprotocol is this API's version negotiation for the socket, so an upgrade that does
|
||||
// not offer it is refused rather than accepted and answered in a dialect the caller may not
|
||||
// read. See VaultEvents.SubProtocol.
|
||||
if (!HttpContext.WebSockets.WebSocketRequestedProtocols
|
||||
.Contains(VaultEvents.SubProtocol, StringComparer.Ordinal))
|
||||
{
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status400BadRequest,
|
||||
ProblemCodes.MalformedRequest,
|
||||
$"This server speaks '{VaultEvents.SubProtocol}', which the upgrade request did "
|
||||
+ "not offer."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads who the caller is and which vaults they may follow, in a scope of its own.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A fresh scope, disposed at once, rather than services injected into this endpoint — which is
|
||||
/// the lesson ADR 0004 records paying for on the relay. This handler runs for as long as the
|
||||
/// socket is open, so anything scoped it held would be a <c>DbContext</c> alive for hours, and a
|
||||
/// few hundred of those exhaust the connection pool. The database is touched here and in
|
||||
/// <see cref="RefreshAsync"/>, briefly, and nowhere else.
|
||||
/// </remarks>
|
||||
private async Task<(Guid UserId, FrozenSet<Guid> Vaults)> ResolveAccessAsync(
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var scope = scopes.CreateAsyncScope();
|
||||
await using var _ = scope.ConfigureAwait(false);
|
||||
|
||||
var currentUser = scope.ServiceProvider.GetRequiredService<ICurrentUserContext>();
|
||||
var vaultAccess = scope.ServiceProvider.GetRequiredService<IVaultAccessService>();
|
||||
|
||||
var user = await currentUser.GetOrProvisionAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return (user.Id, await ReachableAsync(vaultAccess, user.Id, cancellationToken).ConfigureAwait(false));
|
||||
}
|
||||
|
||||
private static async Task<FrozenSet<Guid>> ReachableAsync(
|
||||
IVaultAccessService vaultAccess,
|
||||
Guid userId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var accessible = await vaultAccess.ListAsync(userId, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return accessible
|
||||
.Where(access => access.Vault is not null
|
||||
&& access.Permissions.HasFlag(PermissionFlags.Read))
|
||||
.Select(access => access.Vault!.Id)
|
||||
.ToFrozenSet();
|
||||
}
|
||||
|
||||
/// <summary>Runs the socket until something ends it, then closes it politely.</summary>
|
||||
/// <remarks>
|
||||
/// Three loops rather than one: reading a socket and writing to it are independent waits, and the
|
||||
/// clock is a third. They are joined by <see cref="Task.WhenAny(Task[])"/> and then <em>all</em>
|
||||
/// awaited before the close is written, because a close frame racing a notice frame is a protocol
|
||||
/// violation that presents as a client dropping its connection for no visible reason.
|
||||
/// </remarks>
|
||||
private async Task PumpAsync(
|
||||
WebSocket socket,
|
||||
VaultEventConnection connection,
|
||||
CancellationToken requestAborted)
|
||||
{
|
||||
var settings = options.Value;
|
||||
|
||||
using var pump = CancellationTokenSource.CreateLinkedTokenSource(
|
||||
requestAborted, lifetime.ApplicationStopping);
|
||||
|
||||
var closure = new Closure(
|
||||
(int)WebSocketCloseStatus.NormalClosure, string.Empty);
|
||||
|
||||
connection.TryEnqueue(new VaultEvent(
|
||||
VaultEventKinds.Hello,
|
||||
ServerTime: clock.GetUtcNow(),
|
||||
HeartbeatSeconds: (int)settings.HeartbeatInterval.TotalSeconds,
|
||||
VaultCount: connection.VaultCount));
|
||||
|
||||
var sending = SendAsync(socket, connection, pump.Token);
|
||||
var receiving = ReceiveAsync(socket, connection, pump.Token);
|
||||
var minding = MindAsync(connection, closure, pump.Token);
|
||||
|
||||
await Task.WhenAny(sending, receiving, minding).ConfigureAwait(false);
|
||||
|
||||
await pump.CancelAsync().ConfigureAwait(false);
|
||||
|
||||
// Nothing may still be mid-send when the close frame goes out.
|
||||
await Task.WhenAll(Settled(sending), Settled(receiving), Settled(minding)).ConfigureAwait(false);
|
||||
|
||||
if (lifetime.ApplicationStopping.IsCancellationRequested)
|
||||
{
|
||||
// 1001 "going away", so a client knows to reconnect immediately rather than treating a
|
||||
// rolling deployment as a server that has broken.
|
||||
closure.Set((int)WebSocketCloseStatus.EndpointUnavailable, "The server is shutting down.");
|
||||
}
|
||||
|
||||
EventsLog.ClosingConnection(logger, connection.UserId, closure.Reason);
|
||||
|
||||
await CloseAsync(socket, closure).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes queued frames to the socket, one at a time.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The <em>only</em> writer, which is what makes concurrent sends impossible without a lock: the
|
||||
/// heartbeat and the pong both go into the same queue rather than to the socket. A WebSocket
|
||||
/// permits one send at a time and faults permanently on a second, so this is not a tidiness
|
||||
/// preference.
|
||||
/// </remarks>
|
||||
private async Task SendAsync(
|
||||
WebSocket socket,
|
||||
VaultEventConnection connection,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
await foreach (var frame in connection.Outbound
|
||||
.ReadAllAsync(cancellationToken)
|
||||
.ConfigureAwait(false))
|
||||
{
|
||||
// Re-resolved before the notice is forwarded, not after: the client's answer to this frame
|
||||
// is to re-read its vault list, and the point of a newly shared vault is that the *next*
|
||||
// change to it produces a notice too. A socket that forwarded first would not follow the
|
||||
// new vault until its next periodic refresh.
|
||||
if (string.Equals(frame.Kind, VaultEventKinds.VaultsChanged, StringComparison.Ordinal))
|
||||
{
|
||||
await RefreshAsync(connection, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
var bytes = JsonSerializer.SerializeToUtf8Bytes(frame, DodoSshJsonContext.Default.VaultEvent);
|
||||
|
||||
await socket
|
||||
.SendAsync(bytes, WebSocketMessageType.Text, endOfMessage: true, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads what the client sends, which in this version is heartbeats and a close.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A client cannot ask to follow a vault, and that is deliberate rather than unfinished: a
|
||||
/// <c>subscribe(vaultId)</c> frame is an existence oracle for vault ids, which is the disclosure
|
||||
/// <c>SyncPullEndpoint</c> answers 404 rather than 403 to avoid. Subscription is decided from the
|
||||
/// caller's access and nothing else.
|
||||
/// </remarks>
|
||||
private static async Task ReceiveAsync(
|
||||
WebSocket socket,
|
||||
VaultEventConnection connection,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var buffer = new byte[MaxInboundFrameBytes];
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var received = await socket.ReceiveAsync(buffer, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (received.MessageType == WebSocketMessageType.Close)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Oversized, or split across frames. Nothing this protocol sends is either, so the client
|
||||
// is broken or probing; ending the socket is cheaper than reassembling for it.
|
||||
if (!received.EndOfMessage)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Binary is unused in v1 and skipped rather than refused, because ADR 0012 reserves it for
|
||||
// shared-session data — an older server meeting a newer client must ignore those, not
|
||||
// close on them.
|
||||
if (received.MessageType != WebSocketMessageType.Text)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (Kind(buffer.AsSpan(0, received.Count)) is VaultEventKinds.Ping)
|
||||
{
|
||||
connection.TryEnqueue(new VaultEvent(VaultEventKinds.Pong));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads a frame's kind, or null if it is not one this server understands.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A frame that will not parse is skipped rather than closing the socket. This is a control
|
||||
/// channel whose failure mode is "the client polls instead", so tolerating a frame from a newer
|
||||
/// client costs nothing and refusing one costs that client its push for the whole session.
|
||||
/// </remarks>
|
||||
private static string? Kind(ReadOnlySpan<byte> utf8)
|
||||
{
|
||||
try
|
||||
{
|
||||
return JsonSerializer.Deserialize(utf8, DodoSshJsonContext.Default.VaultEvent)?.Kind;
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Keeps the heartbeat going, the vault set current, and the socket inside its token's lifetime.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The deadline is the earlier of the access token's <c>exp</c> and a hard cap on how long any one
|
||||
/// socket may live. Closing on expiry is what keeps a long-lived connection from outliving the
|
||||
/// short-lived credential that authorised it; the client answers by reconnecting with a fresh
|
||||
/// token, which is a sub-second gap in a channel that degrades to polling anyway.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The wait is the shorter of the heartbeat and the time left, so the deadline is met to within a
|
||||
/// tick rather than to within a heartbeat.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private async Task MindAsync(
|
||||
VaultEventConnection connection,
|
||||
Closure closure,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var settings = options.Value;
|
||||
var started = clock.GetUtcNow();
|
||||
|
||||
var deadline = TokenExpiry() is { } expiry && expiry < started + settings.MaxConnectionDuration
|
||||
? (Expiry: expiry, ForToken: true)
|
||||
: (Expiry: started + settings.MaxConnectionDuration, ForToken: false);
|
||||
|
||||
var refreshed = started;
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var now = clock.GetUtcNow();
|
||||
var remaining = deadline.Expiry - now;
|
||||
|
||||
if (remaining <= TimeSpan.Zero)
|
||||
{
|
||||
closure.Set(
|
||||
deadline.ForToken
|
||||
? VaultEvents.TokenExpiredCloseCode
|
||||
: (int)WebSocketCloseStatus.NormalClosure,
|
||||
deadline.ForToken
|
||||
? "The access token has expired. Reconnect with a fresh one."
|
||||
: "This connection reached its maximum lifetime.");
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
var wait = settings.HeartbeatInterval < remaining ? settings.HeartbeatInterval : remaining;
|
||||
|
||||
await Task.Delay(wait, clock, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
now = clock.GetUtcNow();
|
||||
|
||||
if (now - refreshed >= settings.AccessRefreshInterval)
|
||||
{
|
||||
// The backstop for a grant withdrawn while this socket was open. What it bounds is
|
||||
// metadata — that a vault changed — because that is all a notice carries and reading
|
||||
// the vault still needs a key this server has never held. See ADR 0012.
|
||||
await RefreshAsync(connection, cancellationToken).ConfigureAwait(false);
|
||||
refreshed = now;
|
||||
}
|
||||
|
||||
connection.TryEnqueue(new VaultEvent(VaultEventKinds.Ping, ServerTime: now));
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Re-reads which vaults this socket may follow.</summary>
|
||||
/// <remarks>
|
||||
/// A failure is logged and swallowed. The alternative is dropping a working socket because one
|
||||
/// database call timed out, which would trade an occasionally stale vault set for an outage.
|
||||
/// </remarks>
|
||||
private async Task RefreshAsync(VaultEventConnection connection, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
var scope = scopes.CreateAsyncScope();
|
||||
await using var _ = scope.ConfigureAwait(false);
|
||||
|
||||
var vaultAccess = scope.ServiceProvider.GetRequiredService<IVaultAccessService>();
|
||||
|
||||
connection.Resubscribe(
|
||||
await ReachableAsync(vaultAccess, connection.UserId, cancellationToken)
|
||||
.ConfigureAwait(false));
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// The socket is closing.
|
||||
}
|
||||
catch (Exception exception) when (exception is not OutOfMemoryException)
|
||||
{
|
||||
EventsLog.AccessRefreshFailed(logger, connection.UserId, exception);
|
||||
}
|
||||
}
|
||||
|
||||
/// <remarks>
|
||||
/// <c>MapInboundClaims</c> is off — see <see cref="Auth"/> — so the claim is spelled as the
|
||||
/// provider issued it rather than as a WS-Federation URI. Null is treated as "no bound from the
|
||||
/// token", which the bearer handler's <c>RequireExpirationTime</c> should make unreachable; the
|
||||
/// lifetime cap covers it either way.
|
||||
/// </remarks>
|
||||
private DateTimeOffset? TokenExpiry() =>
|
||||
long.TryParse(
|
||||
HttpContext.User.FindFirstValue("exp"),
|
||||
NumberStyles.Integer,
|
||||
CultureInfo.InvariantCulture,
|
||||
out var seconds)
|
||||
? DateTimeOffset.FromUnixTimeSeconds(seconds)
|
||||
: null;
|
||||
|
||||
private static async Task CloseAsync(WebSocket socket, Closure closure)
|
||||
{
|
||||
if (socket.State is not (WebSocketState.Open or WebSocketState.CloseReceived))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
using var timeout = new CancellationTokenSource(CloseTimeout);
|
||||
|
||||
try
|
||||
{
|
||||
await socket
|
||||
.CloseOutputAsync((WebSocketCloseStatus)closure.Code, closure.Reason, timeout.Token)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception)
|
||||
when (exception is OperationCanceledException or WebSocketException or ObjectDisposedException)
|
||||
{
|
||||
// The peer is already gone. There is nothing to tell it and nothing to recover.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Awaits a pump loop, treating its cancellation and its socket faults as the ordinary end.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Every one of these loops ends by being cancelled or by the socket going away, so an exception
|
||||
/// here is the expected shape of "this connection is over" rather than a fault to propagate — and
|
||||
/// propagating it would skip the close frame the other side is waiting for.
|
||||
/// </remarks>
|
||||
private static async Task Settled(Task loop)
|
||||
{
|
||||
try
|
||||
{
|
||||
await loop.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception)
|
||||
when (exception is OperationCanceledException or WebSocketException or ObjectDisposedException)
|
||||
{
|
||||
// Expected.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Why the socket is being closed, decided by whichever loop ended first.</summary>
|
||||
/// <remarks>
|
||||
/// Mutable and shared, and safe without a lock for one specific reason: it is written by the pump
|
||||
/// loops and read only after <see cref="Task.WhenAll(Task[])"/> over all of them, which is a
|
||||
/// memory barrier. Writes race only with each other, and any of them is a true answer.
|
||||
/// </remarks>
|
||||
private sealed class Closure(int code, string reason)
|
||||
{
|
||||
internal int Code { get; private set; } = code;
|
||||
|
||||
internal string Reason { get; private set; } = reason;
|
||||
|
||||
internal void Set(int code, string reason)
|
||||
{
|
||||
Code = code;
|
||||
Reason = reason;
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user