Public Access
Let a host marked for the relay actually be reached through it
The relay checkbox said it was not wired up, and the plan said the server half was built and shipped. It was not: there was ADR 0004, the contract records, RelayOptions and a CHECK constraint, and RelayOptions said in its own remarks that there was no endpoint and no dialler. So both halves are built here, as step 3 of docs/reaching-a-host-you-cannot-dial.md. Server, in Features/Relay: - POST /api/v1/relay/tickets makes every decision: the host, Connect on its vault, the relay switched on, and the stored name resolved once and judged by RelayAddressPolicy. Loopback, link-local, 0.0.0.0/8, multicast and the metadata endpoints are refused with no configuration escape, with IPv4-mapped and NAT64 spellings unwrapped first; then the operator's denials, allow-list and private-network switch. What passes goes into a Data Protection ticket, and nothing resolves the name again. Every target refusal is the same 403, so the endpoint cannot be used to map the operator's DNS; the real reason is logged. - GET /api/v1/relay/connect takes the ticket as a second subprotocol, checks its expiry against TimeProvider and spends it with a unique insert into relay_ticket_use before accepting the upgrade. After the upgrade a refusal is a close code: 4502 for a host the server could not reach, 4429 for a limit, 4408 idle, 4409 lifetime, 1001 after a drain. - relay_session is the audit row, written from short-lived scopes. Client: - RelayBridge is a loopback SOCKS5 listener that accepts once and pipes to the relay socket. It rides on the new SshLoopbackProxy.Lifetime, so the connection disposes it on every way a session ends and a failed connect disposes it before throwing. - ServerRelay, exposed as IVaultServer.Relay from /api/v1/meta, asks for the ticket, opens the socket, starts the bridge and turns every refusal into a sentence. - VaultViewModel and TransfersViewModel reach a relayed host through it, never fall back to dialling it directly, and replace SSH.NET's transport error with the relay's own reason when it gave one. Neither side ever cancels a WebSocket operation. Cancelling one aborts the socket, and an aborted socket sends no close frame, which is the only way the client learns the server could not reach the host. AnIdleSession_IsClosedWithTheIdleCode asserts that the code arrives. Three departures from ADR 0004, recorded in it: awaited copy loops rather than Pipelines (same backpressure, two 32 KiB buffers), no ?ticket= fallback, and no device id in the ticket. Tests: API 245, shell 582 (5 skipped), layout 170, SSH 110 including a real login through the bridge, API client 30, contracts 25. Not run: the Android build (its change is copy only) and the Alpine layout oracle.
This commit is contained in:
1 parent
54bc142dce
commit
b20f7cae04
51 files changed
+6455
-88
No files matched your search
@@ -0,0 +1,722 @@
|
||||
using System.Buffers;
|
||||
using System.Net;
|
||||
using System.Net.WebSockets;
|
||||
using DodoSSH.Api.Setup;
|
||||
using DodoSSH.Contracts;
|
||||
using DodoSSH.Domain;
|
||||
using DodoSSH.Infrastructure;
|
||||
using FastEndpoints;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Npgsql;
|
||||
|
||||
namespace DodoSSH.Api.Features.Relay;
|
||||
|
||||
/// <summary>
|
||||
/// The relay socket: a ticket in, raw bytes to one host and back.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Anonymous at the HTTP layer, because the ticket is the authorization and it is the whole of it. A
|
||||
/// ticket carries no API authority — it names one host, one set of addresses and one session — so this
|
||||
/// endpoint can accept it from a client that cannot set an <c>Authorization</c> header on an upgrade,
|
||||
/// which is the constraint ADR 0004 was written around. It is verified, checked for freshness and
|
||||
/// redeemed before the upgrade is accepted, so a refused ticket costs a problem document and no socket.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Nothing here is scoped to the request.</b> A session lasts as long as a shell does, which is hours,
|
||||
/// and anything scoped held across it is a pooled database connection held across it. The database is
|
||||
/// touched three times — redeem, open the audit row, close it — each from a scope of its own that is
|
||||
/// disposed before the next byte moves. That is ADR 0004's consequence about <c>DbContext</c> lifetime.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Backpressure is the await. Each direction reads one buffer, writes it, and does not read again until
|
||||
/// the write has completed, so a slow reader on either side stops this side reading, which fills the
|
||||
/// kernel's receive window, which throttles the sender at the far end. Nothing is queued in between, and
|
||||
/// a session costs its two buffers whatever the peers do.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal sealed class RelayConnectEndpoint(
|
||||
IServiceScopeFactory scopes,
|
||||
IRelayNetwork network,
|
||||
RelayTicketCodec codec,
|
||||
RelaySessionRegistry sessions,
|
||||
IOptions<RelayOptions> options,
|
||||
TimeProvider clock,
|
||||
IHostApplicationLifetime lifetime,
|
||||
ILogger<RelayConnectEndpoint> logger)
|
||||
: EndpointWithoutRequest
|
||||
{
|
||||
/// <summary>
|
||||
/// One direction's buffer.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Room for an SSH packet at the size OpenSSH sends them, so a bulk transfer moves in one read and
|
||||
/// one write per packet rather than several.
|
||||
/// </remarks>
|
||||
private const int BufferBytes = 32 * 1024;
|
||||
|
||||
/// <summary>How long a close handshake, or a peer's last write, may take before the socket is dropped.</summary>
|
||||
private static readonly TimeSpan CloseTimeout = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <inheritdoc />
|
||||
public override void Configure()
|
||||
{
|
||||
Get(RelayProtocol.ConnectPath);
|
||||
|
||||
// The ticket is the authorization. See the remarks on this class, and ADR 0004.
|
||||
AllowAnonymous();
|
||||
|
||||
Description(b => b
|
||||
.WithName("RelayConnect")
|
||||
.WithSummary("Pipes raw bytes between a WebSocket and the host a relay ticket names.")
|
||||
.WithTags("Relay"));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task HandleAsync(CancellationToken ct)
|
||||
{
|
||||
var ticket = await AdmitTicketAsync(ct).ConfigureAwait(false);
|
||||
|
||||
if (ticket is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var lease = sessions.TryAdmit(ticket.UserId);
|
||||
|
||||
try
|
||||
{
|
||||
using var socket = await HttpContext.WebSockets
|
||||
.AcceptWebSocketAsync(new WebSocketAcceptContext { SubProtocol = RelayProtocol.SubProtocol })
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (lease is null)
|
||||
{
|
||||
RelayLog.LimitReached(logger, ticket.UserId, sessions.Count);
|
||||
|
||||
await CloseAsync(
|
||||
socket,
|
||||
new Ending(
|
||||
RelayCloseReason.Unspecified,
|
||||
RelayProtocol.TooManySessionsCloseCode,
|
||||
"Too many relay sessions are open."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
if (lifetime.ApplicationStopping.IsCancellationRequested)
|
||||
{
|
||||
await CloseAsync(socket, ShuttingDown).ConfigureAwait(false);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
await RelayAsync(socket, ticket, ct).ConfigureAwait(false);
|
||||
}
|
||||
finally
|
||||
{
|
||||
// Inside a try that starts before the upgrade, for the reason VaultEventsEndpoint gives: an
|
||||
// accept that throws would otherwise hold a place against this user's cap forever.
|
||||
lease?.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
private static Ending ShuttingDown { get; } = new(
|
||||
RelayCloseReason.ServerShutdown,
|
||||
(int)WebSocketCloseStatus.EndpointUnavailable,
|
||||
"The server is shutting down.");
|
||||
|
||||
/// <summary>
|
||||
/// Answers everything that is not a redeemable ticket, as ordinary HTTP, before any socket exists.
|
||||
/// </summary>
|
||||
/// <returns>The redeemed ticket, or null when a response has been sent.</returns>
|
||||
/// <remarks>
|
||||
/// A ticket that is forged, expired, spent, or simply absent all answer 401
|
||||
/// <c>relay-ticket-invalid</c>, with one remedy — ask for another — and no hint of which it was.
|
||||
/// </remarks>
|
||||
private async Task<RelayTicket?> AdmitTicketAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (!options.Value.Enabled)
|
||||
{
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status404NotFound,
|
||||
ProblemCodes.RelayUnavailable,
|
||||
"This server does not relay. GET /api/v1/meta lists the features it does offer."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
var offered = HttpContext.WebSockets.WebSocketRequestedProtocols;
|
||||
|
||||
if (!HttpContext.WebSockets.IsWebSocketRequest
|
||||
|| !offered.Contains(RelayProtocol.SubProtocol, StringComparer.Ordinal))
|
||||
{
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status400BadRequest,
|
||||
ProblemCodes.MalformedRequest,
|
||||
"This endpoint is a WebSocket. Send an upgrade request offering the "
|
||||
+ $"'{RelayProtocol.SubProtocol}' subprotocol and the ticket as a second one, "
|
||||
+ $"'{RelayProtocol.TicketProtocolPrefix}<ticket>'."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
var sealedTicket = offered
|
||||
.FirstOrDefault(protocol => protocol.StartsWith(RelayProtocol.TicketProtocolPrefix, StringComparison.Ordinal))
|
||||
?[RelayProtocol.TicketProtocolPrefix.Length..];
|
||||
|
||||
var ticket = sealedTicket is { Length: > 0 } ? codec.TryOpen(sealedTicket) : null;
|
||||
|
||||
if (ticket is null || !await RedeemAsync(ticket, cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
RelayLog.TicketRefused(logger, ticket?.SessionId);
|
||||
|
||||
await Send.ResultAsync(Problems.Coded(
|
||||
StatusCodes.Status401Unauthorized,
|
||||
ProblemCodes.RelayTicketInvalid,
|
||||
"This relay ticket is not valid: it is expired, already used, or was not issued here. "
|
||||
+ $"Ask for another from POST {RelayProtocol.TicketsPath}."))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
return ticket;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Spends the ticket, or answers false when it has been spent already.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// A unique insert, which is the only form of "once" that holds across nodes: two redemptions racing
|
||||
/// on two nodes both attempt the insert and the database lets exactly one of them win.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Rows for tickets that can no longer verify are swept on the way. They guard nothing, and sweeping
|
||||
/// here — on a path that runs once per session rather than once per request — keeps the table the
|
||||
/// size of the last thirty seconds without a background job to forget about.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private async Task<bool> RedeemAsync(RelayTicket ticket, CancellationToken cancellationToken)
|
||||
{
|
||||
var scope = scopes.CreateAsyncScope();
|
||||
await using var _ = scope.ConfigureAwait(false);
|
||||
|
||||
var database = scope.ServiceProvider.GetRequiredService<DodoDbContext>();
|
||||
var now = clock.GetUtcNow();
|
||||
|
||||
await database.RelayTicketUses
|
||||
.Where(use => use.ExpiresAtUtc < now)
|
||||
.ExecuteDeleteAsync(cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
database.RelayTicketUses.Add(new RelayTicketUse
|
||||
{
|
||||
TicketId = ticket.TicketId,
|
||||
ExpiresAtUtc = ticket.ExpiresAt,
|
||||
UsedAtUtc = now,
|
||||
});
|
||||
|
||||
try
|
||||
{
|
||||
await database.SaveChangesAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
catch (DbUpdateException exception)
|
||||
when (string.Equals(
|
||||
(exception.InnerException as PostgresException)?.SqlState,
|
||||
PostgresErrorCodes.UniqueViolation,
|
||||
StringComparison.Ordinal))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Dials the target, pipes until something ends it, and writes down what happened.</summary>
|
||||
private async Task RelayAsync(WebSocket socket, RelayTicket ticket, CancellationToken requestAborted)
|
||||
{
|
||||
var started = clock.GetUtcNow();
|
||||
|
||||
await OpenAuditAsync(ticket, started).ConfigureAwait(false);
|
||||
|
||||
var dialled = await DialAsync(ticket, requestAborted).ConfigureAwait(false);
|
||||
|
||||
if (dialled is not var (target, address))
|
||||
{
|
||||
RelayLog.TargetUnreachable(logger, ticket.SessionId, ticket.HostId);
|
||||
|
||||
var unreachable = new Ending(
|
||||
RelayCloseReason.TargetUnreachable,
|
||||
RelayProtocol.TargetUnreachableCloseCode,
|
||||
"The relay could not reach the host.");
|
||||
|
||||
await CloseAsync(socket, unreachable).ConfigureAwait(false);
|
||||
await CloseAuditAsync(ticket.SessionId, null, new Counters(clock, started), unreachable.Reason).ConfigureAwait(false);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
var counters = new Counters(clock, started);
|
||||
Ending ending;
|
||||
|
||||
await using (target.ConfigureAwait(false))
|
||||
{
|
||||
RelayLog.SessionOpened(logger, ticket.SessionId, ticket.UserId, ticket.HostId, sessions.Count);
|
||||
|
||||
ending = await PumpAsync(socket, target, counters, started, requestAborted).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
RelayLog.SessionClosed(
|
||||
logger,
|
||||
ticket.SessionId,
|
||||
ending.Reason,
|
||||
counters.ClientToTarget,
|
||||
counters.TargetToClient);
|
||||
|
||||
await CloseAuditAsync(ticket.SessionId, address, counters, ending.Reason).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Connects to the first of the ticket's addresses that answers, within the connect timeout overall.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Overall rather than per address, so a name with sixteen dead addresses costs five seconds rather
|
||||
/// than eighty. In the order resolved, which is the order the operating system's resolver prefers.
|
||||
/// </remarks>
|
||||
private async Task<(Stream Target, IPAddress Address)?> DialAsync(
|
||||
RelayTicket ticket,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
|
||||
timeout.CancelAfter(options.Value.ConnectTimeout);
|
||||
|
||||
foreach (var address in ticket.TargetAddresses)
|
||||
{
|
||||
try
|
||||
{
|
||||
var target = await network
|
||||
.ConnectAsync(address, ticket.TargetPort, timeout.Token)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return (target, address);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or System.Net.Sockets.SocketException)
|
||||
{
|
||||
// The next address, if there is one.
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs both directions and the clock until one of them ends the session, then ends it politely.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>No operation on the WebSocket is ever cancelled.</b> Cancelling a pending receive or send on a
|
||||
/// .NET WebSocket aborts it, and an aborted socket cannot send the close frame that is the only way
|
||||
/// a client learns <em>why</em> its session ended — which, for an idle timeout or a drain, is the
|
||||
/// difference between "reconnect" and "something is broken". So the token stops the target side and
|
||||
/// the clock; the sender is let finish its last write; the close goes out; and the receiver ends on
|
||||
/// the client's answering close. Each wait is bounded, and the socket is aborted only when a peer
|
||||
/// has stopped answering altogether.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private async Task<Ending> PumpAsync(
|
||||
WebSocket socket,
|
||||
Stream target,
|
||||
Counters counters,
|
||||
DateTimeOffset started,
|
||||
CancellationToken requestAborted)
|
||||
{
|
||||
using var pump = CancellationTokenSource.CreateLinkedTokenSource(requestAborted);
|
||||
|
||||
var upstream = ClientToTargetAsync(socket, target, counters, pump.Token);
|
||||
var downstream = TargetToClientAsync(target, socket, counters, pump.Token);
|
||||
var minding = MindAsync(counters, started, pump.Token);
|
||||
|
||||
var first = await Task.WhenAny(upstream, downstream, minding).ConfigureAwait(false);
|
||||
var ending = await first.ConfigureAwait(false) ?? new Ending(
|
||||
RelayCloseReason.ClientClosed, (int)WebSocketCloseStatus.NormalClosure, string.Empty);
|
||||
|
||||
await pump.CancelAsync().ConfigureAwait(false);
|
||||
|
||||
// The only sender must be done before the close frame is written: a WebSocket permits one send
|
||||
// at a time and faults permanently on a second.
|
||||
if (!await SettlesAsync(downstream).ConfigureAwait(false))
|
||||
{
|
||||
socket.Abort();
|
||||
}
|
||||
|
||||
await CloseAsync(socket, ending).ConfigureAwait(false);
|
||||
|
||||
if (!await SettlesAsync(upstream).ConfigureAwait(false))
|
||||
{
|
||||
socket.Abort();
|
||||
await SettlesAsync(upstream).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
await SettlesAsync(minding).ConfigureAwait(false);
|
||||
|
||||
return ending;
|
||||
}
|
||||
|
||||
/// <summary>Client to target: every binary message is the next bytes of the stream.</summary>
|
||||
private static async Task<Ending?> ClientToTargetAsync(
|
||||
WebSocket socket,
|
||||
Stream target,
|
||||
Counters counters,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var buffer = ArrayPool<byte>.Shared.Rent(BufferBytes);
|
||||
|
||||
try
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
ValueWebSocketReceiveResult received;
|
||||
|
||||
try
|
||||
{
|
||||
// Not cancellable, deliberately. See PumpAsync.
|
||||
received = await socket.ReceiveAsync(buffer.AsMemory(), CancellationToken.None)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception) when (exception is WebSocketException or OperationCanceledException)
|
||||
{
|
||||
return ClientGone;
|
||||
}
|
||||
|
||||
if (received.MessageType == WebSocketMessageType.Close)
|
||||
{
|
||||
return ClientGone;
|
||||
}
|
||||
|
||||
// A byte pipe has no text. A client sending some has the wrong idea about this endpoint,
|
||||
// and forwarding it into an SSH stream would corrupt that stream at best.
|
||||
if (received.MessageType != WebSocketMessageType.Binary)
|
||||
{
|
||||
return new Ending(
|
||||
RelayCloseReason.Error,
|
||||
(int)WebSocketCloseStatus.InvalidMessageType,
|
||||
"The relay carries binary messages only.");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await target.WriteAsync(buffer.AsMemory(0, received.Count), cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
return TargetGone;
|
||||
}
|
||||
|
||||
counters.AddClientToTarget(received.Count);
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
ArrayPool<byte>.Shared.Return(buffer);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Target to client: every read becomes one binary message.</summary>
|
||||
private static async Task<Ending?> TargetToClientAsync(
|
||||
Stream target,
|
||||
WebSocket socket,
|
||||
Counters counters,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var buffer = ArrayPool<byte>.Shared.Rent(BufferBytes);
|
||||
|
||||
try
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
int read;
|
||||
|
||||
try
|
||||
{
|
||||
read = await target.ReadAsync(buffer.AsMemory(0, BufferBytes), cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
return TargetGone;
|
||||
}
|
||||
|
||||
if (read == 0)
|
||||
{
|
||||
return TargetGone;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
// Not cancellable, deliberately. See PumpAsync.
|
||||
await socket
|
||||
.SendAsync(
|
||||
buffer.AsMemory(0, read),
|
||||
WebSocketMessageType.Binary,
|
||||
endOfMessage: true,
|
||||
CancellationToken.None)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception) when (exception is WebSocketException or ObjectDisposedException)
|
||||
{
|
||||
return ClientGone;
|
||||
}
|
||||
|
||||
counters.AddTargetToClient(read);
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
ArrayPool<byte>.Shared.Return(buffer);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Ends the session at the idle timeout, the lifetime cap, or the end of a shutdown's drain.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// A shutdown does not end a session; it starts a countdown. ADR 0004 asks that a
|
||||
/// <c>docker compose up -d</c> not guillotine live shells, so a session in progress when the server is
|
||||
/// told to stop is given the drain timeout to finish — most will, because the user was already
|
||||
/// closing it or the deployment waits — and is closed with 1001 only when that runs out, which tells
|
||||
/// the client to reconnect rather than to report a fault. The host's own shutdown timeout is set
|
||||
/// longer than the drain for this to be possible at all.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Idle means no bytes in either direction. WebSocket pings are not bytes of the stream and do not
|
||||
/// count; SSH's own keepalives do, which is how a client that wants an idle shell kept open says so.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private async Task<Ending?> MindAsync(Counters counters, DateTimeOffset started, CancellationToken cancellationToken)
|
||||
{
|
||||
var settings = options.Value;
|
||||
var hardDeadline = started + settings.MaxSessionDuration;
|
||||
DateTimeOffset? drainDeadline = null;
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var now = clock.GetUtcNow();
|
||||
|
||||
if (drainDeadline is null && lifetime.ApplicationStopping.IsCancellationRequested)
|
||||
{
|
||||
drainDeadline = now + settings.DrainTimeout;
|
||||
}
|
||||
|
||||
if (now >= hardDeadline)
|
||||
{
|
||||
return new Ending(
|
||||
RelayCloseReason.DurationLimit,
|
||||
RelayProtocol.DurationLimitCloseCode,
|
||||
"This relay session reached its maximum lifetime.");
|
||||
}
|
||||
|
||||
if (drainDeadline is { } drain && now >= drain)
|
||||
{
|
||||
return ShuttingDown;
|
||||
}
|
||||
|
||||
var idleDeadline = counters.LastActivity + settings.IdleTimeout;
|
||||
|
||||
if (now >= idleDeadline)
|
||||
{
|
||||
return new Ending(
|
||||
RelayCloseReason.IdleTimeout,
|
||||
RelayProtocol.IdleTimeoutCloseCode,
|
||||
"This relay session carried nothing for the idle timeout.");
|
||||
}
|
||||
|
||||
var next = new[] { hardDeadline, idleDeadline, drainDeadline ?? DateTimeOffset.MaxValue }.Min();
|
||||
|
||||
// Woken early by the start of a shutdown, so the drain is measured from when it began rather
|
||||
// than from whenever the idle clock next came round.
|
||||
using var wake = drainDeadline is null
|
||||
? CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, lifetime.ApplicationStopping)
|
||||
: CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
|
||||
|
||||
try
|
||||
{
|
||||
await Task.Delay(next - now, clock, wake.Token).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
// The shutdown began. Round again to start the drain.
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static Ending ClientGone { get; } =
|
||||
new(RelayCloseReason.ClientClosed, (int)WebSocketCloseStatus.NormalClosure, string.Empty);
|
||||
|
||||
private static Ending TargetGone { get; } =
|
||||
new(RelayCloseReason.TargetClosed, (int)WebSocketCloseStatus.NormalClosure, "The host closed the connection.");
|
||||
|
||||
/// <summary>
|
||||
/// Opens the audit row, before the target is dialled, so a session that never reached its host is
|
||||
/// still one the operator can see.
|
||||
/// </summary>
|
||||
private async Task OpenAuditAsync(RelayTicket ticket, DateTimeOffset started)
|
||||
{
|
||||
var scope = scopes.CreateAsyncScope();
|
||||
await using var _ = scope.ConfigureAwait(false);
|
||||
|
||||
var database = scope.ServiceProvider.GetRequiredService<DodoDbContext>();
|
||||
|
||||
database.RelaySessions.Add(new RelaySession
|
||||
{
|
||||
Id = ticket.SessionId,
|
||||
UserId = ticket.UserId,
|
||||
HostId = ticket.HostId,
|
||||
VaultId = ticket.VaultId,
|
||||
TargetHost = ticket.TargetHost,
|
||||
TargetPort = ticket.TargetPort,
|
||||
ClientAddress = HttpContext.Connection.RemoteIpAddress?.ToString(),
|
||||
StartedAtUtc = started,
|
||||
});
|
||||
|
||||
// Not the request's token: the row has to be written even for a client that has already gone,
|
||||
// or a session that was cut off would be the one with no record.
|
||||
await database.SaveChangesAsync(CancellationToken.None).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Closes the audit row. A failure is logged and swallowed: the bytes have already moved, and
|
||||
/// throwing here would only turn a finished session into an unhandled exception.
|
||||
/// </summary>
|
||||
private async Task CloseAuditAsync(
|
||||
Guid sessionId,
|
||||
IPAddress? address,
|
||||
Counters counters,
|
||||
RelayCloseReason reason)
|
||||
{
|
||||
try
|
||||
{
|
||||
var scope = scopes.CreateAsyncScope();
|
||||
await using var _ = scope.ConfigureAwait(false);
|
||||
|
||||
var database = scope.ServiceProvider.GetRequiredService<DodoDbContext>();
|
||||
var ended = clock.GetUtcNow();
|
||||
var dialled = address?.ToString();
|
||||
|
||||
await database.RelaySessions
|
||||
.Where(session => session.Id == sessionId)
|
||||
.ExecuteUpdateAsync(
|
||||
set => set
|
||||
.SetProperty(session => session.EndedAtUtc, ended)
|
||||
.SetProperty(session => session.TargetAddress, dialled)
|
||||
.SetProperty(session => session.BytesClientToTarget, counters.ClientToTarget)
|
||||
.SetProperty(session => session.BytesTargetToClient, counters.TargetToClient)
|
||||
.SetProperty(session => session.CloseReason, reason),
|
||||
CancellationToken.None)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception) when (exception is DbUpdateException or NpgsqlException or InvalidOperationException)
|
||||
{
|
||||
RelayLog.AuditCloseFailed(logger, sessionId, exception);
|
||||
}
|
||||
}
|
||||
|
||||
private static async Task CloseAsync(WebSocket socket, Ending ending)
|
||||
{
|
||||
if (socket.State is not (WebSocketState.Open or WebSocketState.CloseReceived))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
using var timeout = new CancellationTokenSource(CloseTimeout);
|
||||
|
||||
try
|
||||
{
|
||||
await socket
|
||||
.CloseOutputAsync((WebSocketCloseStatus)ending.Code, ending.Description, timeout.Token)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception)
|
||||
when (exception is OperationCanceledException or WebSocketException or ObjectDisposedException)
|
||||
{
|
||||
// The peer is already gone. There is nothing to tell it.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Waits for a loop to finish, for at most the close timeout.</summary>
|
||||
/// <returns>Whether it finished.</returns>
|
||||
private static async Task<bool> SettlesAsync(Task loop)
|
||||
{
|
||||
try
|
||||
{
|
||||
await loop.WaitAsync(CloseTimeout).ConfigureAwait(false);
|
||||
|
||||
return true;
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
catch (Exception exception)
|
||||
when (exception is OperationCanceledException or WebSocketException or IOException or ObjectDisposedException)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>How a session ended, and what the client is told.</summary>
|
||||
private sealed record Ending(RelayCloseReason Reason, int Code, string Description);
|
||||
|
||||
/// <summary>
|
||||
/// Byte counts and the time of the last byte, written by the two directions and read by the clock.
|
||||
/// </summary>
|
||||
private sealed class Counters(TimeProvider clock, DateTimeOffset started)
|
||||
{
|
||||
private long clientToTarget;
|
||||
private long targetToClient;
|
||||
private long lastActivityTicks = started.UtcTicks;
|
||||
|
||||
internal long ClientToTarget => Interlocked.Read(ref clientToTarget);
|
||||
|
||||
internal long TargetToClient => Interlocked.Read(ref targetToClient);
|
||||
|
||||
internal DateTimeOffset LastActivity => new(Interlocked.Read(ref lastActivityTicks), TimeSpan.Zero);
|
||||
|
||||
internal void AddClientToTarget(int count)
|
||||
{
|
||||
Interlocked.Add(ref clientToTarget, count);
|
||||
Interlocked.Exchange(ref lastActivityTicks, clock.GetUtcNow().UtcTicks);
|
||||
}
|
||||
|
||||
internal void AddTargetToClient(int count)
|
||||
{
|
||||
Interlocked.Add(ref targetToClient, count);
|
||||
Interlocked.Exchange(ref lastActivityTicks, clock.GetUtcNow().UtcTicks);
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user