Let a host marked for the relay actually be reached through it
ci / build and test (pull_request) Successful in 6m27s
ci / desktop nightly (pull_request) Skipped
ci / android head (pull_request) Successful in 15m47s
ci / api image (pull_request) Failing after 28s

The relay checkbox said it was not wired up, and the plan said the server
half was built and shipped. It was not: there was ADR 0004, the contract
records, RelayOptions and a CHECK constraint, and RelayOptions said in its
own remarks that there was no endpoint and no dialler. So both halves are
built here, as step 3 of docs/reaching-a-host-you-cannot-dial.md.

Server, in Features/Relay:

- POST /api/v1/relay/tickets makes every decision: the host, Connect on
  its vault, the relay switched on, and the stored name resolved once and
  judged by RelayAddressPolicy. Loopback, link-local, 0.0.0.0/8,
  multicast and the metadata endpoints are refused with no configuration
  escape, with IPv4-mapped and NAT64 spellings unwrapped first; then the
  operator's denials, allow-list and private-network switch. What passes
  goes into a Data Protection ticket, and nothing resolves the name again.
  Every target refusal is the same 403, so the endpoint cannot be used to
  map the operator's DNS; the real reason is logged.
- GET /api/v1/relay/connect takes the ticket as a second subprotocol,
  checks its expiry against TimeProvider and spends it with a unique
  insert into relay_ticket_use before accepting the upgrade. After the
  upgrade a refusal is a close code: 4502 for a host the server could not
  reach, 4429 for a limit, 4408 idle, 4409 lifetime, 1001 after a drain.
- relay_session is the audit row, written from short-lived scopes.

Client:

- RelayBridge is a loopback SOCKS5 listener that accepts once and pipes
  to the relay socket. It rides on the new SshLoopbackProxy.Lifetime, so
  the connection disposes it on every way a session ends and a failed
  connect disposes it before throwing.
- ServerRelay, exposed as IVaultServer.Relay from /api/v1/meta, asks for
  the ticket, opens the socket, starts the bridge and turns every refusal
  into a sentence.
- VaultViewModel and TransfersViewModel reach a relayed host through it,
  never fall back to dialling it directly, and replace SSH.NET's transport
  error with the relay's own reason when it gave one.

Neither side ever cancels a WebSocket operation. Cancelling one aborts
the socket, and an aborted socket sends no close frame, which is the only
way the client learns the server could not reach the host.
AnIdleSession_IsClosedWithTheIdleCode asserts that the code arrives.

Three departures from ADR 0004, recorded in it: awaited copy loops rather
than Pipelines (same backpressure, two 32 KiB buffers), no ?ticket=
fallback, and no device id in the ticket.

Tests: API 245, shell 582 (5 skipped), layout 170, SSH 110 including a
real login through the bridge, API client 30, contracts 25. Not run: the
Android build (its change is copy only) and the Alpine layout oracle.
This commit is contained in:
jaap-jan committed 2026-09-28 14:37:20 +02:00
1 parent 54bc142dce
commit b20f7cae04
51 files changed
+6455 -88

No files matched your search

@@ -26,9 +26,6 @@ internal sealed class GetMetaEndpoint(
/// <summary>Sync semantics version. Bumped when push or pull behaviour changes.</summary>
internal const int SyncProtocolVersion = 1;
/// <summary>Feature flag for the relay.</summary>
internal const string RelayFeature = "relay";
/// <inheritdoc />
public override void Configure()
{
@@ -50,7 +47,7 @@ internal sealed class GetMetaEndpoint(
List<string> features = ["teams"];
if (relay.Value.Enabled)
{
features.Add(RelayFeature);
features.Add(RelayProtocol.Feature);
}
// Advertised so a client knows whether to hold a socket open or rely on its timer. Absence is
@@ -0,0 +1,192 @@
using System.Net;
using System.Net.Sockets;
using DodoSSH.Api.Setup;
namespace DodoSSH.Api.Features.Relay;
/// <summary>
/// Decides which addresses and ports the relay may dial. ADR 0004 §3, in the order it is written.
/// </summary>
/// <remarks>
/// <para>
/// This runs on <em>addresses</em>, never on names. The name is resolved once, at ticket issue, and what
/// this permits goes into the ticket; the socket dials those addresses and nothing else. A check on the
/// name would be defeated by DNS rebinding — answer with a public address for the check and a loopback
/// one for the dial — which is the attack ADR 0004 is built around.
/// </para>
/// <para>
/// Three layers, and the order is the security property. The <see cref="Refused"/> list comes first and
/// has no configuration at all: nothing an operator sets, and nothing a host row says, reaches a
/// loopback, link-local, unspecified, multicast or metadata address. Then the operator's own denials,
/// then their allow-list, then the private-network switch.
/// </para>
/// <para>
/// <b>An address that embeds another is judged as the one it embeds.</b> <c>::ffff:127.0.0.1</c> is
/// loopback however it is spelled, and a check that compared it against IPv6 loopback alone would pass
/// it — the classic bypass the ADR names. The NAT64 well-known prefix is unwrapped the same way, because
/// on a network with a NAT64 gateway <c>64:ff9b::a9fe:a9fe</c> <em>is</em> the metadata endpoint.
/// </para>
/// </remarks>
internal sealed class RelayAddressPolicy
{
/// <summary>
/// Addresses the relay never dials, whatever it is configured to do.
/// </summary>
/// <remarks>
/// Broader than the ADR's list in two places, both deliberately. <c>240.0.0.0/4</c> — reserved, and
/// containing the limited broadcast address — has no host worth reaching. And IPv6 <c>::/96</c>, the
/// deprecated IPv4-compatible block, covers the unspecified address and loopback while also refusing
/// the spellings of IPv4 that a stack may or may not map; nothing legitimate lives there.
/// </remarks>
private static readonly IPNetwork[] Refused =
[
// IPv4: "this network", loopback, link-local (which holds 169.254.169.254), multicast, reserved.
IPNetwork.Parse("0.0.0.0/8"),
IPNetwork.Parse("127.0.0.0/8"),
IPNetwork.Parse("169.254.0.0/16"),
IPNetwork.Parse("224.0.0.0/4"),
IPNetwork.Parse("240.0.0.0/4"),
// Alibaba Cloud's metadata endpoint, which sits inside carrier-grade NAT space and so would
// otherwise be permitted as private — the range a Tailnet also uses, which is why the whole
// block cannot simply be refused.
IPNetwork.Parse("100.100.100.200/32"),
// IPv6: IPv4-compatible (with unspecified and loopback), link-local, multicast, and the AWS
// metadata endpoint's IPv6 address.
IPNetwork.Parse("::/96"),
IPNetwork.Parse("fe80::/10"),
IPNetwork.Parse("ff00::/8"),
IPNetwork.Parse("fd00:ec2::254/128"),
];
/// <summary>
/// Ranges <see cref="RelayOptions.AllowPrivateNetworks"/> governs.
/// </summary>
/// <remarks>
/// Carrier-grade NAT is here rather than treated as public, because it is where a Tailnet lives and
/// "the server is on the same Tailnet" is one of the shapes the relay exists for. Site-local
/// <c>fec0::/10</c> is deprecated and was private while it lived.
/// </remarks>
private static readonly IPNetwork[] Private =
[
IPNetwork.Parse("10.0.0.0/8"),
IPNetwork.Parse("172.16.0.0/12"),
IPNetwork.Parse("192.168.0.0/16"),
IPNetwork.Parse("100.64.0.0/10"),
IPNetwork.Parse("fc00::/7"),
IPNetwork.Parse("fec0::/10"),
];
/// <summary>The NAT64 well-known prefix, RFC 6052, whose low 32 bits are an IPv4 address.</summary>
private static readonly IPNetwork Nat64 = IPNetwork.Parse("64:ff9b::/96");
/// <summary>
/// Ports the relay never dials.
/// </summary>
/// <remarks>
/// SMTP and nothing else, per ADR 0004: a relay that reached port 25 is a spam cannon with the
/// operator's address on it. Database ports stay open, because forwarding to one is a headline use,
/// and the control that matters there is who may dial the host at all.
/// </remarks>
private static readonly int[] RefusedPorts = [25, 465, 587];
private readonly bool allowPrivate;
private readonly IPNetwork[] allowed;
private readonly IPNetwork[] denied;
public RelayAddressPolicy(RelayOptions options)
{
ArgumentNullException.ThrowIfNull(options);
allowPrivate = options.AllowPrivateNetworks;
allowed = ParseAll(options.AllowedNetworks);
denied = ParseAll(options.DeniedNetworks);
}
/// <summary>Whether a port may be dialled at all.</summary>
internal static bool PermitsPort(int port) =>
port is >= 1 and <= 65535 && !RefusedPorts.Contains(port);
/// <summary>
/// Whether every entry is a network this policy can read.
/// </summary>
/// <remarks>
/// For startup validation. A typo in a deny list that was silently skipped would be a hole the
/// operator believes is closed, so an unreadable entry stops the server instead.
/// <para>
/// Host bits are refused as well, although <see cref="IPNetwork.TryParse(string?, out IPNetwork)"/>
/// would mask them off quietly. <c>10.20.5.7/16</c> could be a /16 written from memory or a /32 with
/// the wrong suffix, and in a deny list those differ by sixty-five thousand addresses.
/// </para>
/// </remarks>
internal static bool AreNetworks(IEnumerable<string> entries) =>
entries.All(entry =>
IPNetwork.TryParse(entry, out var network)
&& IPAddress.TryParse(entry.AsSpan(0, entry.IndexOf('/', StringComparison.Ordinal)), out var written)
&& written.Equals(network.BaseAddress));
/// <summary>
/// The address the relay would actually be talking to, with any embedding unwrapped.
/// </summary>
internal static IPAddress Normalise(IPAddress address)
{
ArgumentNullException.ThrowIfNull(address);
if (address.IsIPv4MappedToIPv6)
{
return address.MapToIPv4();
}
if (address.AddressFamily == AddressFamily.InterNetworkV6 && Nat64.Contains(address))
{
var bytes = address.GetAddressBytes();
return new IPAddress(bytes.AsSpan(12, 4));
}
// A scope id is how an address names an interface on this machine, which is not something a
// host row gets to choose. Dropped, so what is judged is what is dialled.
return address.AddressFamily == AddressFamily.InterNetworkV6 && address.ScopeId != 0
? new IPAddress(address.GetAddressBytes())
: address;
}
/// <summary>Whether the relay may dial this address.</summary>
/// <remarks>
/// Judges the normalised address. Callers should dial <see cref="Normalise"/>'s answer too, so that
/// the address checked and the address connected to are the same object rather than two spellings
/// the operating system is trusted to agree about.
/// </remarks>
internal bool Permits(IPAddress address)
{
var target = Normalise(address);
if (Refused.Any(network => Within(network, target)))
{
return false;
}
if (denied.Any(network => Within(network, target)))
{
return false;
}
if (allowed.Length > 0)
{
return allowed.Any(network => Within(network, target));
}
return allowPrivate || !Private.Any(network => Within(network, target));
}
/// <remarks>
/// <see cref="IPNetwork.Contains"/> answers false across address families rather than throwing, so
/// a list mixing IPv4 and IPv6 needs no sorting — but it is spelled out here because the one-line
/// alternative reads as though it compared across families.
/// </remarks>
private static bool Within(IPNetwork network, IPAddress address) =>
network.BaseAddress.AddressFamily == address.AddressFamily && network.Contains(address);
private static IPNetwork[] ParseAll(IEnumerable<string> entries) =>
[.. entries.Select(entry => IPNetwork.Parse(entry))];
}
@@ -0,0 +1,722 @@
using System.Buffers;
using System.Net;
using System.Net.WebSockets;
using DodoSSH.Api.Setup;
using DodoSSH.Contracts;
using DodoSSH.Domain;
using DodoSSH.Infrastructure;
using FastEndpoints;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Options;
using Npgsql;
namespace DodoSSH.Api.Features.Relay;
/// <summary>
/// The relay socket: a ticket in, raw bytes to one host and back.
/// </summary>
/// <remarks>
/// <para>
/// Anonymous at the HTTP layer, because the ticket is the authorization and it is the whole of it. A
/// ticket carries no API authority — it names one host, one set of addresses and one session — so this
/// endpoint can accept it from a client that cannot set an <c>Authorization</c> header on an upgrade,
/// which is the constraint ADR 0004 was written around. It is verified, checked for freshness and
/// redeemed before the upgrade is accepted, so a refused ticket costs a problem document and no socket.
/// </para>
/// <para>
/// <b>Nothing here is scoped to the request.</b> A session lasts as long as a shell does, which is hours,
/// and anything scoped held across it is a pooled database connection held across it. The database is
/// touched three times — redeem, open the audit row, close it — each from a scope of its own that is
/// disposed before the next byte moves. That is ADR 0004's consequence about <c>DbContext</c> lifetime.
/// </para>
/// <para>
/// Backpressure is the await. Each direction reads one buffer, writes it, and does not read again until
/// the write has completed, so a slow reader on either side stops this side reading, which fills the
/// kernel's receive window, which throttles the sender at the far end. Nothing is queued in between, and
/// a session costs its two buffers whatever the peers do.
/// </para>
/// </remarks>
internal sealed class RelayConnectEndpoint(
IServiceScopeFactory scopes,
IRelayNetwork network,
RelayTicketCodec codec,
RelaySessionRegistry sessions,
IOptions<RelayOptions> options,
TimeProvider clock,
IHostApplicationLifetime lifetime,
ILogger<RelayConnectEndpoint> logger)
: EndpointWithoutRequest
{
/// <summary>
/// One direction's buffer.
/// </summary>
/// <remarks>
/// Room for an SSH packet at the size OpenSSH sends them, so a bulk transfer moves in one read and
/// one write per packet rather than several.
/// </remarks>
private const int BufferBytes = 32 * 1024;
/// <summary>How long a close handshake, or a peer's last write, may take before the socket is dropped.</summary>
private static readonly TimeSpan CloseTimeout = TimeSpan.FromSeconds(5);
/// <inheritdoc />
public override void Configure()
{
Get(RelayProtocol.ConnectPath);
// The ticket is the authorization. See the remarks on this class, and ADR 0004.
AllowAnonymous();
Description(b => b
.WithName("RelayConnect")
.WithSummary("Pipes raw bytes between a WebSocket and the host a relay ticket names.")
.WithTags("Relay"));
}
/// <inheritdoc />
public override async Task HandleAsync(CancellationToken ct)
{
var ticket = await AdmitTicketAsync(ct).ConfigureAwait(false);
if (ticket is null)
{
return;
}
var lease = sessions.TryAdmit(ticket.UserId);
try
{
using var socket = await HttpContext.WebSockets
.AcceptWebSocketAsync(new WebSocketAcceptContext { SubProtocol = RelayProtocol.SubProtocol })
.ConfigureAwait(false);
if (lease is null)
{
RelayLog.LimitReached(logger, ticket.UserId, sessions.Count);
await CloseAsync(
socket,
new Ending(
RelayCloseReason.Unspecified,
RelayProtocol.TooManySessionsCloseCode,
"Too many relay sessions are open."))
.ConfigureAwait(false);
return;
}
if (lifetime.ApplicationStopping.IsCancellationRequested)
{
await CloseAsync(socket, ShuttingDown).ConfigureAwait(false);
return;
}
await RelayAsync(socket, ticket, ct).ConfigureAwait(false);
}
finally
{
// Inside a try that starts before the upgrade, for the reason VaultEventsEndpoint gives: an
// accept that throws would otherwise hold a place against this user's cap forever.
lease?.Dispose();
}
}
private static Ending ShuttingDown { get; } = new(
RelayCloseReason.ServerShutdown,
(int)WebSocketCloseStatus.EndpointUnavailable,
"The server is shutting down.");
/// <summary>
/// Answers everything that is not a redeemable ticket, as ordinary HTTP, before any socket exists.
/// </summary>
/// <returns>The redeemed ticket, or null when a response has been sent.</returns>
/// <remarks>
/// A ticket that is forged, expired, spent, or simply absent all answer 401
/// <c>relay-ticket-invalid</c>, with one remedy — ask for another — and no hint of which it was.
/// </remarks>
private async Task<RelayTicket?> AdmitTicketAsync(CancellationToken cancellationToken)
{
if (!options.Value.Enabled)
{
await Send.ResultAsync(Problems.Coded(
StatusCodes.Status404NotFound,
ProblemCodes.RelayUnavailable,
"This server does not relay. GET /api/v1/meta lists the features it does offer."))
.ConfigureAwait(false);
return null;
}
var offered = HttpContext.WebSockets.WebSocketRequestedProtocols;
if (!HttpContext.WebSockets.IsWebSocketRequest
|| !offered.Contains(RelayProtocol.SubProtocol, StringComparer.Ordinal))
{
await Send.ResultAsync(Problems.Coded(
StatusCodes.Status400BadRequest,
ProblemCodes.MalformedRequest,
"This endpoint is a WebSocket. Send an upgrade request offering the "
+ $"'{RelayProtocol.SubProtocol}' subprotocol and the ticket as a second one, "
+ $"'{RelayProtocol.TicketProtocolPrefix}<ticket>'."))
.ConfigureAwait(false);
return null;
}
var sealedTicket = offered
.FirstOrDefault(protocol => protocol.StartsWith(RelayProtocol.TicketProtocolPrefix, StringComparison.Ordinal))
?[RelayProtocol.TicketProtocolPrefix.Length..];
var ticket = sealedTicket is { Length: > 0 } ? codec.TryOpen(sealedTicket) : null;
if (ticket is null || !await RedeemAsync(ticket, cancellationToken).ConfigureAwait(false))
{
RelayLog.TicketRefused(logger, ticket?.SessionId);
await Send.ResultAsync(Problems.Coded(
StatusCodes.Status401Unauthorized,
ProblemCodes.RelayTicketInvalid,
"This relay ticket is not valid: it is expired, already used, or was not issued here. "
+ $"Ask for another from POST {RelayProtocol.TicketsPath}."))
.ConfigureAwait(false);
return null;
}
return ticket;
}
/// <summary>
/// Spends the ticket, or answers false when it has been spent already.
/// </summary>
/// <remarks>
/// <para>
/// A unique insert, which is the only form of "once" that holds across nodes: two redemptions racing
/// on two nodes both attempt the insert and the database lets exactly one of them win.
/// </para>
/// <para>
/// Rows for tickets that can no longer verify are swept on the way. They guard nothing, and sweeping
/// here — on a path that runs once per session rather than once per request — keeps the table the
/// size of the last thirty seconds without a background job to forget about.
/// </para>
/// </remarks>
private async Task<bool> RedeemAsync(RelayTicket ticket, CancellationToken cancellationToken)
{
var scope = scopes.CreateAsyncScope();
await using var _ = scope.ConfigureAwait(false);
var database = scope.ServiceProvider.GetRequiredService<DodoDbContext>();
var now = clock.GetUtcNow();
await database.RelayTicketUses
.Where(use => use.ExpiresAtUtc < now)
.ExecuteDeleteAsync(cancellationToken)
.ConfigureAwait(false);
database.RelayTicketUses.Add(new RelayTicketUse
{
TicketId = ticket.TicketId,
ExpiresAtUtc = ticket.ExpiresAt,
UsedAtUtc = now,
});
try
{
await database.SaveChangesAsync(cancellationToken).ConfigureAwait(false);
return true;
}
catch (DbUpdateException exception)
when (string.Equals(
(exception.InnerException as PostgresException)?.SqlState,
PostgresErrorCodes.UniqueViolation,
StringComparison.Ordinal))
{
return false;
}
}
/// <summary>Dials the target, pipes until something ends it, and writes down what happened.</summary>
private async Task RelayAsync(WebSocket socket, RelayTicket ticket, CancellationToken requestAborted)
{
var started = clock.GetUtcNow();
await OpenAuditAsync(ticket, started).ConfigureAwait(false);
var dialled = await DialAsync(ticket, requestAborted).ConfigureAwait(false);
if (dialled is not var (target, address))
{
RelayLog.TargetUnreachable(logger, ticket.SessionId, ticket.HostId);
var unreachable = new Ending(
RelayCloseReason.TargetUnreachable,
RelayProtocol.TargetUnreachableCloseCode,
"The relay could not reach the host.");
await CloseAsync(socket, unreachable).ConfigureAwait(false);
await CloseAuditAsync(ticket.SessionId, null, new Counters(clock, started), unreachable.Reason).ConfigureAwait(false);
return;
}
var counters = new Counters(clock, started);
Ending ending;
await using (target.ConfigureAwait(false))
{
RelayLog.SessionOpened(logger, ticket.SessionId, ticket.UserId, ticket.HostId, sessions.Count);
ending = await PumpAsync(socket, target, counters, started, requestAborted).ConfigureAwait(false);
}
RelayLog.SessionClosed(
logger,
ticket.SessionId,
ending.Reason,
counters.ClientToTarget,
counters.TargetToClient);
await CloseAuditAsync(ticket.SessionId, address, counters, ending.Reason).ConfigureAwait(false);
}
/// <summary>
/// Connects to the first of the ticket's addresses that answers, within the connect timeout overall.
/// </summary>
/// <remarks>
/// Overall rather than per address, so a name with sixteen dead addresses costs five seconds rather
/// than eighty. In the order resolved, which is the order the operating system's resolver prefers.
/// </remarks>
private async Task<(Stream Target, IPAddress Address)?> DialAsync(
RelayTicket ticket,
CancellationToken cancellationToken)
{
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
timeout.CancelAfter(options.Value.ConnectTimeout);
foreach (var address in ticket.TargetAddresses)
{
try
{
var target = await network
.ConnectAsync(address, ticket.TargetPort, timeout.Token)
.ConfigureAwait(false);
return (target, address);
}
catch (OperationCanceledException)
{
return null;
}
catch (Exception exception) when (exception is IOException or System.Net.Sockets.SocketException)
{
// The next address, if there is one.
}
}
return null;
}
/// <summary>
/// Runs both directions and the clock until one of them ends the session, then ends it politely.
/// </summary>
/// <remarks>
/// <para>
/// <b>No operation on the WebSocket is ever cancelled.</b> Cancelling a pending receive or send on a
/// .NET WebSocket aborts it, and an aborted socket cannot send the close frame that is the only way
/// a client learns <em>why</em> its session ended — which, for an idle timeout or a drain, is the
/// difference between "reconnect" and "something is broken". So the token stops the target side and
/// the clock; the sender is let finish its last write; the close goes out; and the receiver ends on
/// the client's answering close. Each wait is bounded, and the socket is aborted only when a peer
/// has stopped answering altogether.
/// </para>
/// </remarks>
private async Task<Ending> PumpAsync(
WebSocket socket,
Stream target,
Counters counters,
DateTimeOffset started,
CancellationToken requestAborted)
{
using var pump = CancellationTokenSource.CreateLinkedTokenSource(requestAborted);
var upstream = ClientToTargetAsync(socket, target, counters, pump.Token);
var downstream = TargetToClientAsync(target, socket, counters, pump.Token);
var minding = MindAsync(counters, started, pump.Token);
var first = await Task.WhenAny(upstream, downstream, minding).ConfigureAwait(false);
var ending = await first.ConfigureAwait(false) ?? new Ending(
RelayCloseReason.ClientClosed, (int)WebSocketCloseStatus.NormalClosure, string.Empty);
await pump.CancelAsync().ConfigureAwait(false);
// The only sender must be done before the close frame is written: a WebSocket permits one send
// at a time and faults permanently on a second.
if (!await SettlesAsync(downstream).ConfigureAwait(false))
{
socket.Abort();
}
await CloseAsync(socket, ending).ConfigureAwait(false);
if (!await SettlesAsync(upstream).ConfigureAwait(false))
{
socket.Abort();
await SettlesAsync(upstream).ConfigureAwait(false);
}
await SettlesAsync(minding).ConfigureAwait(false);
return ending;
}
/// <summary>Client to target: every binary message is the next bytes of the stream.</summary>
private static async Task<Ending?> ClientToTargetAsync(
WebSocket socket,
Stream target,
Counters counters,
CancellationToken cancellationToken)
{
var buffer = ArrayPool<byte>.Shared.Rent(BufferBytes);
try
{
while (true)
{
ValueWebSocketReceiveResult received;
try
{
// Not cancellable, deliberately. See PumpAsync.
received = await socket.ReceiveAsync(buffer.AsMemory(), CancellationToken.None)
.ConfigureAwait(false);
}
catch (Exception exception) when (exception is WebSocketException or OperationCanceledException)
{
return ClientGone;
}
if (received.MessageType == WebSocketMessageType.Close)
{
return ClientGone;
}
// A byte pipe has no text. A client sending some has the wrong idea about this endpoint,
// and forwarding it into an SSH stream would corrupt that stream at best.
if (received.MessageType != WebSocketMessageType.Binary)
{
return new Ending(
RelayCloseReason.Error,
(int)WebSocketCloseStatus.InvalidMessageType,
"The relay carries binary messages only.");
}
try
{
await target.WriteAsync(buffer.AsMemory(0, received.Count), cancellationToken)
.ConfigureAwait(false);
}
catch (OperationCanceledException)
{
return null;
}
catch (IOException)
{
return TargetGone;
}
counters.AddClientToTarget(received.Count);
}
}
finally
{
ArrayPool<byte>.Shared.Return(buffer);
}
}
/// <summary>Target to client: every read becomes one binary message.</summary>
private static async Task<Ending?> TargetToClientAsync(
Stream target,
WebSocket socket,
Counters counters,
CancellationToken cancellationToken)
{
var buffer = ArrayPool<byte>.Shared.Rent(BufferBytes);
try
{
while (true)
{
int read;
try
{
read = await target.ReadAsync(buffer.AsMemory(0, BufferBytes), cancellationToken)
.ConfigureAwait(false);
}
catch (OperationCanceledException)
{
return null;
}
catch (IOException)
{
return TargetGone;
}
if (read == 0)
{
return TargetGone;
}
try
{
// Not cancellable, deliberately. See PumpAsync.
await socket
.SendAsync(
buffer.AsMemory(0, read),
WebSocketMessageType.Binary,
endOfMessage: true,
CancellationToken.None)
.ConfigureAwait(false);
}
catch (Exception exception) when (exception is WebSocketException or ObjectDisposedException)
{
return ClientGone;
}
counters.AddTargetToClient(read);
}
}
finally
{
ArrayPool<byte>.Shared.Return(buffer);
}
}
/// <summary>
/// Ends the session at the idle timeout, the lifetime cap, or the end of a shutdown's drain.
/// </summary>
/// <remarks>
/// <para>
/// A shutdown does not end a session; it starts a countdown. ADR 0004 asks that a
/// <c>docker compose up -d</c> not guillotine live shells, so a session in progress when the server is
/// told to stop is given the drain timeout to finish — most will, because the user was already
/// closing it or the deployment waits — and is closed with 1001 only when that runs out, which tells
/// the client to reconnect rather than to report a fault. The host's own shutdown timeout is set
/// longer than the drain for this to be possible at all.
/// </para>
/// <para>
/// Idle means no bytes in either direction. WebSocket pings are not bytes of the stream and do not
/// count; SSH's own keepalives do, which is how a client that wants an idle shell kept open says so.
/// </para>
/// </remarks>
private async Task<Ending?> MindAsync(Counters counters, DateTimeOffset started, CancellationToken cancellationToken)
{
var settings = options.Value;
var hardDeadline = started + settings.MaxSessionDuration;
DateTimeOffset? drainDeadline = null;
while (!cancellationToken.IsCancellationRequested)
{
var now = clock.GetUtcNow();
if (drainDeadline is null && lifetime.ApplicationStopping.IsCancellationRequested)
{
drainDeadline = now + settings.DrainTimeout;
}
if (now >= hardDeadline)
{
return new Ending(
RelayCloseReason.DurationLimit,
RelayProtocol.DurationLimitCloseCode,
"This relay session reached its maximum lifetime.");
}
if (drainDeadline is { } drain && now >= drain)
{
return ShuttingDown;
}
var idleDeadline = counters.LastActivity + settings.IdleTimeout;
if (now >= idleDeadline)
{
return new Ending(
RelayCloseReason.IdleTimeout,
RelayProtocol.IdleTimeoutCloseCode,
"This relay session carried nothing for the idle timeout.");
}
var next = new[] { hardDeadline, idleDeadline, drainDeadline ?? DateTimeOffset.MaxValue }.Min();
// Woken early by the start of a shutdown, so the drain is measured from when it began rather
// than from whenever the idle clock next came round.
using var wake = drainDeadline is null
? CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, lifetime.ApplicationStopping)
: CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
try
{
await Task.Delay(next - now, clock, wake.Token).ConfigureAwait(false);
}
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
{
// The shutdown began. Round again to start the drain.
}
catch (OperationCanceledException)
{
return null;
}
}
return null;
}
private static Ending ClientGone { get; } =
new(RelayCloseReason.ClientClosed, (int)WebSocketCloseStatus.NormalClosure, string.Empty);
private static Ending TargetGone { get; } =
new(RelayCloseReason.TargetClosed, (int)WebSocketCloseStatus.NormalClosure, "The host closed the connection.");
/// <summary>
/// Opens the audit row, before the target is dialled, so a session that never reached its host is
/// still one the operator can see.
/// </summary>
private async Task OpenAuditAsync(RelayTicket ticket, DateTimeOffset started)
{
var scope = scopes.CreateAsyncScope();
await using var _ = scope.ConfigureAwait(false);
var database = scope.ServiceProvider.GetRequiredService<DodoDbContext>();
database.RelaySessions.Add(new RelaySession
{
Id = ticket.SessionId,
UserId = ticket.UserId,
HostId = ticket.HostId,
VaultId = ticket.VaultId,
TargetHost = ticket.TargetHost,
TargetPort = ticket.TargetPort,
ClientAddress = HttpContext.Connection.RemoteIpAddress?.ToString(),
StartedAtUtc = started,
});
// Not the request's token: the row has to be written even for a client that has already gone,
// or a session that was cut off would be the one with no record.
await database.SaveChangesAsync(CancellationToken.None).ConfigureAwait(false);
}
/// <summary>
/// Closes the audit row. A failure is logged and swallowed: the bytes have already moved, and
/// throwing here would only turn a finished session into an unhandled exception.
/// </summary>
private async Task CloseAuditAsync(
Guid sessionId,
IPAddress? address,
Counters counters,
RelayCloseReason reason)
{
try
{
var scope = scopes.CreateAsyncScope();
await using var _ = scope.ConfigureAwait(false);
var database = scope.ServiceProvider.GetRequiredService<DodoDbContext>();
var ended = clock.GetUtcNow();
var dialled = address?.ToString();
await database.RelaySessions
.Where(session => session.Id == sessionId)
.ExecuteUpdateAsync(
set => set
.SetProperty(session => session.EndedAtUtc, ended)
.SetProperty(session => session.TargetAddress, dialled)
.SetProperty(session => session.BytesClientToTarget, counters.ClientToTarget)
.SetProperty(session => session.BytesTargetToClient, counters.TargetToClient)
.SetProperty(session => session.CloseReason, reason),
CancellationToken.None)
.ConfigureAwait(false);
}
catch (Exception exception) when (exception is DbUpdateException or NpgsqlException or InvalidOperationException)
{
RelayLog.AuditCloseFailed(logger, sessionId, exception);
}
}
private static async Task CloseAsync(WebSocket socket, Ending ending)
{
if (socket.State is not (WebSocketState.Open or WebSocketState.CloseReceived))
{
return;
}
using var timeout = new CancellationTokenSource(CloseTimeout);
try
{
await socket
.CloseOutputAsync((WebSocketCloseStatus)ending.Code, ending.Description, timeout.Token)
.ConfigureAwait(false);
}
catch (Exception exception)
when (exception is OperationCanceledException or WebSocketException or ObjectDisposedException)
{
// The peer is already gone. There is nothing to tell it.
}
}
/// <summary>Waits for a loop to finish, for at most the close timeout.</summary>
/// <returns>Whether it finished.</returns>
private static async Task<bool> SettlesAsync(Task loop)
{
try
{
await loop.WaitAsync(CloseTimeout).ConfigureAwait(false);
return true;
}
catch (TimeoutException)
{
return false;
}
catch (Exception exception)
when (exception is OperationCanceledException or WebSocketException or IOException or ObjectDisposedException)
{
return true;
}
}
/// <summary>How a session ended, and what the client is told.</summary>
private sealed record Ending(RelayCloseReason Reason, int Code, string Description);
/// <summary>
/// Byte counts and the time of the last byte, written by the two directions and read by the clock.
/// </summary>
private sealed class Counters(TimeProvider clock, DateTimeOffset started)
{
private long clientToTarget;
private long targetToClient;
private long lastActivityTicks = started.UtcTicks;
internal long ClientToTarget => Interlocked.Read(ref clientToTarget);
internal long TargetToClient => Interlocked.Read(ref targetToClient);
internal DateTimeOffset LastActivity => new(Interlocked.Read(ref lastActivityTicks), TimeSpan.Zero);
internal void AddClientToTarget(int count)
{
Interlocked.Add(ref clientToTarget, count);
Interlocked.Exchange(ref lastActivityTicks, clock.GetUtcNow().UtcTicks);
}
internal void AddTargetToClient(int count)
{
Interlocked.Add(ref targetToClient, count);
Interlocked.Exchange(ref lastActivityTicks, clock.GetUtcNow().UtcTicks);
}
}
}
@@ -0,0 +1,94 @@
using DodoSSH.Domain;
namespace DodoSSH.Api.Features.Relay;
/// <summary>Source-generated log messages for the relay.</summary>
/// <remarks>
/// <para>
/// Ids, counts and reasons. Never a ticket, which is a bearer credential for thirty seconds, and never
/// the target's address: that is in the audit table, where retention is the operator's decision, and
/// copying it into a log would make it the log pipeline's decision too.
/// </para>
/// <para>
/// <see cref="TargetRejected"/> is the one place the reason for a refused target is written down. The
/// client is told nothing specific, deliberately — see <c>RelayTicketEndpoint</c> — so this line is how
/// an operator answers "why will the relay not reach this host".
/// </para>
/// </remarks>
internal static partial class RelayLog
{
[LoggerMessage(
EventId = 2301,
Level = LogLevel.Information,
Message = "Relay ticket refused for user {UserId} and host {HostId}: {Reason}.")]
internal static partial void TargetRejected(ILogger logger, Guid userId, Guid hostId, string reason);
[LoggerMessage(
EventId = 2302,
Level = LogLevel.Debug,
Message = "Relay ticket issued to user {UserId} for host {HostId}, session {SessionId}, with "
+ "{AddressCount} permitted address(es).")]
internal static partial void TicketIssued(
ILogger logger,
Guid userId,
Guid hostId,
Guid sessionId,
int addressCount);
/// <remarks>
/// The session id is known only for a ticket that verified and had already been spent, which is the
/// case worth an operator's attention: a replay, or a client retrying a redemption it thinks failed.
/// </remarks>
[LoggerMessage(
EventId = 2303,
Level = LogLevel.Information,
Message = "Relay ticket refused at the socket (session {SessionId}): forged, expired or already used.")]
internal static partial void TicketRefused(ILogger logger, Guid? sessionId);
[LoggerMessage(
EventId = 2304,
Level = LogLevel.Information,
Message = "Relay session refused for user {UserId}: a session limit is met, with {SessionCount} "
+ "open on this node.")]
internal static partial void LimitReached(ILogger logger, Guid userId, int sessionCount);
[LoggerMessage(
EventId = 2305,
Level = LogLevel.Information,
Message = "Relay session {SessionId} could not reach host {HostId} on any permitted address.")]
internal static partial void TargetUnreachable(ILogger logger, Guid sessionId, Guid hostId);
[LoggerMessage(
EventId = 2306,
Level = LogLevel.Debug,
Message = "Relay session {SessionId} opened for user {UserId} to host {HostId}; {SessionCount} "
+ "open on this node.")]
internal static partial void SessionOpened(
ILogger logger,
Guid sessionId,
Guid userId,
Guid hostId,
int sessionCount);
[LoggerMessage(
EventId = 2307,
Level = LogLevel.Debug,
Message = "Relay session {SessionId} closed ({Reason}) after {ClientToTarget} byte(s) up and "
+ "{TargetToClient} down.")]
internal static partial void SessionClosed(
ILogger logger,
Guid sessionId,
RelayCloseReason reason,
long clientToTarget,
long targetToClient);
/// <remarks>
/// A warning rather than an error: the session itself is over and was fine. What is lost is the
/// audit row's second half, which leaves it looking like a session still open.
/// </remarks>
[LoggerMessage(
EventId = 2308,
Level = LogLevel.Warning,
Message = "Relay session {SessionId} ended but its audit record could not be closed.")]
internal static partial void AuditCloseFailed(ILogger logger, Guid sessionId, Exception exception);
}
@@ -0,0 +1,61 @@
using System.Net;
using System.Net.Sockets;
namespace DodoSSH.Api.Features.Relay;
/// <summary>
/// The two things the relay asks of the network: what a name resolves to, and a connection to an address.
/// </summary>
/// <remarks>
/// <para>
/// A seam for tests, and a narrow one on purpose. <see cref="RelayAddressPolicy"/> is not behind it:
/// whatever this answers, the policy judges, and a test that replaces this still runs every address
/// through the real rules. What a test cannot do without the seam is dial anything, because every
/// address a test machine can reach on its own — loopback — is one the policy refuses outright.
/// </para>
/// </remarks>
internal interface IRelayNetwork
{
/// <summary>Resolves a stored hostname, which may be an address literal.</summary>
Task<IPAddress[]> ResolveAsync(string host, CancellationToken cancellationToken);
/// <summary>Opens a TCP connection to one address.</summary>
/// <returns>A stream that owns the connection.</returns>
Task<Stream> ConnectAsync(IPAddress address, int port, CancellationToken cancellationToken);
}
/// <summary>The operating system's resolver and sockets.</summary>
internal sealed class SystemRelayNetwork : IRelayNetwork
{
/// <inheritdoc />
public Task<IPAddress[]> ResolveAsync(string host, CancellationToken cancellationToken) =>
// A literal parses to itself without a lookup, which is what an address typed into a host
// editor should do — and what makes "10.0.0.5" mean the same thing on every node.
IPAddress.TryParse(host, out var literal)
? Task.FromResult<IPAddress[]>([literal])
: Dns.GetHostAddressesAsync(host, cancellationToken);
/// <inheritdoc />
public async Task<Stream> ConnectAsync(IPAddress address, int port, CancellationToken cancellationToken)
{
var socket = new Socket(address.AddressFamily, SocketType.Stream, ProtocolType.Tcp)
{
// An interactive shell is a stream of keystrokes; Nagle would hold each one back for the
// acknowledgement of the last, which is the latency the user feels as a sticky terminal.
NoDelay = true,
};
try
{
await socket.ConnectAsync(new IPEndPoint(address, port), cancellationToken).ConfigureAwait(false);
return new NetworkStream(socket, ownsSocket: true);
}
catch
{
socket.Dispose();
throw;
}
}
}
@@ -0,0 +1,111 @@
using DodoSSH.Api.Setup;
using Microsoft.Extensions.Options;
namespace DodoSSH.Api.Features.Relay;
/// <summary>
/// Counts the relay sessions open on this node, and refuses the ones over ADR 0004's limits.
/// </summary>
/// <remarks>
/// <para>
/// Per node, and so per user <em>per node</em>: a user behind a round-robin balancer with three nodes
/// could hold three times the per-user cap. That is accepted rather than fixed, because fixing it means
/// shared state on the hot path of every connect, and the cap exists to stop one account exhausting a
/// node — which a per-node count does exactly.
/// </para>
/// <para>
/// A singleton, because a session outlives the request that opened it. It holds counts and nothing else;
/// the sockets belong to the handlers that opened them.
/// </para>
/// </remarks>
internal sealed class RelaySessionRegistry(IOptions<RelayOptions> options)
{
private readonly Lock gate = new();
private readonly Dictionary<Guid, int> perUser = [];
private int total;
/// <summary>How many sessions are open on this node.</summary>
internal int Count
{
get
{
lock (gate)
{
return total;
}
}
}
/// <summary>Whether a session for this user would be admitted now.</summary>
/// <remarks>
/// Advisory, for the ticket endpoint: it lets a caller already at the cap hear so as an HTTP status
/// before it has spent a ticket, but only <see cref="TryAdmit"/> decides, since the answer here can
/// be stale by the time the socket arrives — possibly at another node.
/// </remarks>
internal bool HasRoomFor(Guid userId)
{
var settings = options.Value;
lock (gate)
{
return total < settings.MaxConcurrentSessionsTotal
&& perUser.GetValueOrDefault(userId) < settings.MaxConcurrentSessionsPerUser;
}
}
/// <summary>Admits a session, or answers null when a limit is met.</summary>
/// <returns>A lease that gives the place back when disposed.</returns>
internal Lease? TryAdmit(Guid userId)
{
var settings = options.Value;
lock (gate)
{
var held = perUser.GetValueOrDefault(userId);
if (total >= settings.MaxConcurrentSessionsTotal || held >= settings.MaxConcurrentSessionsPerUser)
{
return null;
}
perUser[userId] = held + 1;
total++;
return new Lease(this, userId);
}
}
private void Release(Guid userId)
{
lock (gate)
{
total--;
// Removed at zero rather than left behind, so the dictionary is the size of the users with a
// session open and not of every user who ever had one.
if (perUser[userId] <= 1)
{
perUser.Remove(userId);
}
else
{
perUser[userId]--;
}
}
}
/// <summary>One admitted session's place. Idempotent to dispose.</summary>
internal sealed class Lease(RelaySessionRegistry registry, Guid userId) : IDisposable
{
private int released;
/// <inheritdoc />
public void Dispose()
{
if (Interlocked.Exchange(ref released, 1) == 0)
{
registry.Release(userId);
}
}
}
}
@@ -0,0 +1,171 @@
using System.Net;
using System.Security.Cryptography;
using Microsoft.AspNetCore.DataProtection;
using Microsoft.AspNetCore.WebUtilities;
namespace DodoSSH.Api.Features.Relay;
/// <summary>
/// Everything the relay socket needs to know, decided by the ticket endpoint.
/// </summary>
/// <remarks>
/// The addresses are the resolved ones and the socket dials nothing else. There is no hostname for it
/// to resolve again — <see cref="TargetHost"/> is carried for the audit row only — which is what keeps a
/// DNS answer that changes between issue and connect from changing where the bytes go.
/// </remarks>
/// <param name="TicketId">The replay guard's key.</param>
/// <param name="SessionId">The audit row's id, already given to the client.</param>
/// <param name="UserId">Who asked.</param>
/// <param name="HostId">The host it is for.</param>
/// <param name="VaultId">The host's vault at issue.</param>
/// <param name="TargetHost">The hostname as stored, for the audit row.</param>
/// <param name="TargetAddresses">The permitted addresses the name resolved to, in the order to try.</param>
/// <param name="TargetPort">The port.</param>
/// <param name="ExpiresAt">When the ticket stops verifying.</param>
internal sealed record RelayTicket(
Guid TicketId,
Guid SessionId,
Guid UserId,
Guid HostId,
Guid VaultId,
string TargetHost,
IReadOnlyList<IPAddress> TargetAddresses,
int TargetPort,
DateTimeOffset ExpiresAt);
/// <summary>
/// Seals and opens relay tickets.
/// </summary>
/// <remarks>
/// <para>
/// A Data Protection payload, per ADR 0004: authenticated and encrypted, with key rotation handled by
/// the framework, and verifiable on any node that shares the key ring. Encrypted rather than merely
/// signed because the ticket travels in a header every proxy on the way can read, and what it holds is
/// the resolved address of a machine on the operator's network.
/// </para>
/// <para>
/// The expiry is inside the payload and checked against <see cref="TimeProvider"/>, rather than left to
/// the framework's time-limited protector, which reads the system clock directly — the thing this
/// repository bans so that time can be faked in tests.
/// </para>
/// <para>
/// The layout is a version byte and fixed fields, written by hand. Small enough that a serialiser would
/// be the larger half of the code, and a hand layout has no reflection for trimming to break.
/// </para>
/// </remarks>
internal sealed class RelayTicketCodec(IDataProtectionProvider protection, TimeProvider clock)
{
/// <summary>The purpose string, which isolates these payloads from every other protector's.</summary>
private const string Purpose = "DodoSSH.Relay.Ticket";
private const byte LayoutVersion = 1;
/// <summary>More addresses than any real name resolves to, and a bound on what a ticket can carry.</summary>
internal const int MaxAddresses = 16;
private readonly IDataProtector protector = protection.CreateProtector(Purpose);
internal string Seal(RelayTicket ticket)
{
ArgumentNullException.ThrowIfNull(ticket);
using var buffer = new MemoryStream();
using (var writer = new BinaryWriter(buffer))
{
writer.Write(LayoutVersion);
writer.Write(ticket.TicketId.ToByteArray());
writer.Write(ticket.SessionId.ToByteArray());
writer.Write(ticket.UserId.ToByteArray());
writer.Write(ticket.HostId.ToByteArray());
writer.Write(ticket.VaultId.ToByteArray());
writer.Write(ticket.ExpiresAt.ToUnixTimeMilliseconds());
writer.Write(ticket.TargetPort);
writer.Write(ticket.TargetHost);
writer.Write((byte)ticket.TargetAddresses.Count);
foreach (var address in ticket.TargetAddresses)
{
var bytes = address.GetAddressBytes();
writer.Write((byte)bytes.Length);
writer.Write(bytes);
}
}
// Base64url, whose alphabet is all token characters, so it can stand in a subprotocol name.
return WebEncoders.Base64UrlEncode(protector.Protect(buffer.ToArray()));
}
/// <summary>Opens a ticket, or answers null for one that is forged, damaged or expired.</summary>
/// <remarks>
/// One answer for all three, because the caller's remedy is the same — ask for another — and a
/// distinct "expired" would tell a forger which of their attempts had verified.
/// </remarks>
internal RelayTicket? TryOpen(string sealedTicket)
{
byte[] payload;
try
{
payload = protector.Unprotect(WebEncoders.Base64UrlDecode(sealedTicket));
}
catch (Exception exception) when (exception is CryptographicException or FormatException)
{
return null;
}
RelayTicket ticket;
try
{
ticket = Read(payload);
}
catch (Exception exception)
when (exception is EndOfStreamException or ArgumentException or FormatException)
{
return null;
}
return ticket.ExpiresAt > clock.GetUtcNow() ? ticket : null;
}
private static RelayTicket Read(byte[] payload)
{
using var reader = new BinaryReader(new MemoryStream(payload));
if (reader.ReadByte() != LayoutVersion)
{
throw new FormatException("Unknown ticket layout.");
}
var ticketId = new Guid(reader.ReadBytes(16));
var sessionId = new Guid(reader.ReadBytes(16));
var userId = new Guid(reader.ReadBytes(16));
var hostId = new Guid(reader.ReadBytes(16));
var vaultId = new Guid(reader.ReadBytes(16));
var expiresAt = DateTimeOffset.FromUnixTimeMilliseconds(reader.ReadInt64());
var port = reader.ReadInt32();
var host = reader.ReadString();
var count = reader.ReadByte();
if (count is 0 or > MaxAddresses)
{
throw new FormatException("A ticket names between one and sixteen addresses.");
}
var addresses = new IPAddress[count];
for (var index = 0; index < count; index++)
{
var length = reader.ReadByte();
if (length is not (4 or 16))
{
throw new FormatException("An address is four or sixteen bytes.");
}
addresses[index] = new IPAddress(reader.ReadBytes(length));
}
return new RelayTicket(ticketId, sessionId, userId, hostId, vaultId, host, addresses, port, expiresAt);
}
}
@@ -0,0 +1,231 @@
using System.Net;
using System.Net.Sockets;
using DodoSSH.Api.Authorization;
using DodoSSH.Api.Setup;
using DodoSSH.Contracts;
using DodoSSH.Domain.Authorization;
using DodoSSH.Infrastructure;
using FastEndpoints;
using Microsoft.AspNetCore.Http.HttpResults;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Options;
namespace DodoSSH.Api.Features.Relay;
/// <summary>
/// Issues the ticket that lets exactly one relay socket reach exactly one host.
/// </summary>
/// <remarks>
/// <para>
/// Every decision the relay makes is made here, with the full bearer token and the full access model in
/// hand: which host, whether the caller may connect to it, which addresses its name resolves to and
/// which of those the policy permits. The socket that follows decides nothing. It checks that the ticket
/// is genuine, fresh and unused, and dials what the ticket says. See ADR 0004.
/// </para>
/// <para>
/// <b>Every refusal about the target is the same refusal.</b> A host that does not exist, one in a vault
/// the caller cannot reach, one without the relay switched on, one whose name does not resolve, and one
/// whose every address the policy refuses all answer 403 <c>relay-target-rejected</c> with the same
/// sentence. Anyone who can write a host can set its address to anything, so a distinct answer for
/// "resolves, but refused" would turn this endpoint into a scanner for the operator's internal DNS. The
/// real reason goes to the server log, where the operator can read it.
/// </para>
/// </remarks>
internal sealed class RelayTicketEndpoint(
ICurrentUserContext currentUser,
IVaultAccessService vaultAccess,
DodoDbContext database,
IRelayNetwork network,
RelayTicketCodec codec,
RelaySessionRegistry sessions,
IOptions<RelayOptions> options,
TimeProvider clock,
ILogger<RelayTicketEndpoint> logger)
: Endpoint<RelayTicketRequest, Results<Ok<RelayTicketResponse>, ProblemHttpResult>>
{
/// <summary>What every target refusal says, whatever the reason was.</summary>
private const string RejectedDetail =
"The relay will not connect to this host. Check that the relay is switched on for it and that its "
+ "address is one this server may reach; the server's log records which.";
/// <inheritdoc />
public override void Configure()
{
Post(RelayProtocol.TicketsPath);
// Enrolled, matching sync. A relayed host is still a host whose name, username and key are
// ciphertext, so a caller with no identity key has no way to have learned which host to ask for.
Policies(Auth.EnrolledPolicy);
Description(b => b
.WithName("IssueRelayTicket")
.WithSummary("Issues a single-use ticket for one relay connection to one host.")
.WithTags("Relay"));
}
/// <inheritdoc />
public override async Task<Results<Ok<RelayTicketResponse>, ProblemHttpResult>> ExecuteAsync(
RelayTicketRequest req,
CancellationToken ct)
{
var settings = options.Value;
if (!settings.Enabled)
{
return Problems.Coded(
StatusCodes.Status404NotFound,
ProblemCodes.RelayUnavailable,
"This server does not relay. Connect to the host directly; GET /api/v1/meta lists the "
+ "features this server does offer.");
}
var user = await currentUser.GetOrProvisionAsync(ct).ConfigureAwait(false);
var (target, refusal) = await FindTargetAsync(user.Id, req, ct).ConfigureAwait(false);
if (target is not var (hostname, port, vaultId))
{
return Rejected(user.Id, req.HostId, refusal!);
}
// Before resolving rather than after: a caller already at the cap is told so without this server
// spending a DNS query on a ticket that could not be redeemed.
if (!sessions.HasRoomFor(user.Id))
{
RelayLog.LimitReached(logger, user.Id, sessions.Count);
return Problems.Coded(
StatusCodes.Status429TooManyRequests,
ProblemCodes.RelayLimitReached,
"You have as many relay sessions open as this server allows. Close one and try again.");
}
var addresses = await ResolvePermittedAsync(hostname, ct).ConfigureAwait(false);
if (addresses is not { Length: > 0 })
{
return Rejected(user.Id, req.HostId, addresses is null
? "its name did not resolve"
: "every address its name resolved to is refused by policy");
}
var ticket = new RelayTicket(
TicketId: Guid.CreateVersion7(),
SessionId: Guid.CreateVersion7(),
UserId: user.Id,
HostId: req.HostId,
VaultId: vaultId,
TargetHost: hostname,
TargetAddresses: addresses,
TargetPort: port,
ExpiresAt: clock.GetUtcNow() + settings.TicketLifetime);
RelayLog.TicketIssued(logger, user.Id, req.HostId, ticket.SessionId, addresses.Length);
return TypedResults.Ok(new RelayTicketResponse(
Ticket: codec.Seal(ticket),
WebSocketUrl: new Uri(settings.WebSocketUrl!, UriKind.Absolute),
SubProtocol: RelayProtocol.SubProtocol,
SessionId: ticket.SessionId,
ExpiresAt: ticket.ExpiresAt));
}
/// <summary>
/// Finds the host, and decides whether this caller may be relayed to it at all.
/// </summary>
/// <returns>The stored target, or the reason there is none — for the log, never for the caller.</returns>
private async Task<((string Hostname, int Port, Guid VaultId)? Target, string? Refusal)> FindTargetAsync(
Guid userId,
RelayTicketRequest req,
CancellationToken cancellationToken)
{
if (req.PortForwardId is not null)
{
return (null, "a port forward was requested, which v1 does not relay");
}
var host = await database.Hosts
.AsNoTracking()
.SingleOrDefaultAsync(h => h.Id == req.HostId && h.DeletedAtUtc == null, cancellationToken)
.ConfigureAwait(false);
if (host is null)
{
return (null, "no such host");
}
var access = await vaultAccess.ResolveAsync(userId, host.VaultId, cancellationToken).ConfigureAwait(false);
// Connect rather than Read, because Connect is the permission that says what this is. Every role
// that reads today also connects — see VaultAccessService.ForRole — but a role that ever split
// them should be refused a relay here without anybody remembering to look.
if (!access.Granted || !access.Permissions.HasFlag(PermissionFlags.Connect))
{
return (null, "the caller may not connect to this host's vault");
}
// The CHECK constraint makes the second half unreachable while the first is true. Tested anyway,
// because the cost of trusting it is dialling a null.
if (!host.RelayEnabled || host.Hostname is not { } hostname || host.Port is not { } port)
{
return (null, "the relay is not switched on for this host");
}
return RelayAddressPolicy.PermitsPort(port)
? ((hostname, port, host.VaultId), null)
: (null, $"port {port} is refused by policy");
}
/// <summary>
/// Resolves the stored name, and keeps the addresses the policy permits.
/// </summary>
/// <returns>The permitted addresses, possibly none; or null when the name did not resolve.</returns>
/// <remarks>
/// <para>
/// Filtered rather than refused as a whole when one answer is bad. A name that resolves to a public
/// address and a link-local one is dialled on the public address, and never on the other — the
/// ticket carries only what passed, so there is nothing else for the socket to try.
/// </para>
/// <para>
/// The addresses kept are the normalised ones, so what is dialled is exactly what was judged rather
/// than a second spelling of it.
/// </para>
/// </remarks>
private async Task<IPAddress[]?> ResolvePermittedAsync(string hostname, CancellationToken cancellationToken)
{
var policy = new RelayAddressPolicy(options.Value);
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
timeout.CancelAfter(options.Value.ConnectTimeout);
IPAddress[] resolved;
try
{
resolved = await network.ResolveAsync(hostname, timeout.Token).ConfigureAwait(false);
}
catch (SocketException)
{
return null;
}
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
{
return null;
}
return
[
.. resolved
.Where(policy.Permits)
.Select(RelayAddressPolicy.Normalise)
.Distinct()
.Take(RelayTicketCodec.MaxAddresses),
];
}
private ProblemHttpResult Rejected(Guid userId, Guid hostId, string reason)
{
RelayLog.TargetRejected(logger, userId, hostId, reason);
return Problems.Coded(StatusCodes.Status403Forbidden, ProblemCodes.RelayTargetRejected, RejectedDetail);
}
}