CI: deploy ssh.dodotech.cloud after the image is pushed, and fail unless it comes back ready
ci / build and test (push) Successful in 3m38s
ci / desktop nightly (push) Successful in 1m42s
ci / api image (push) Successful in 20s
ci / deploy ssh.dodotech.cloud (push) Failing after 0s
ci / android head (push) Successful in 6m3s

The image job pushed :main and stopped there, so nothing told the Spelunk
stack a new image existed and a commit carrying a migration never reached
the server's schema.

A deploy job now calls the stack's deploy hook with SPELUNK_DEPLOY_TOKEN and
then waits up to ten minutes for /healthz/ready. The new container applies
pending migrations as it starts, and readiness stays unhealthy while any is
pending, so a migration that did not apply fails the run instead of passing
quietly.

The Dockerfile no longer says nothing migrates from inside the container,
which had stopped being true and pointed at the setting that causes this.
This commit is contained in:
jaap-jan committed 2026-10-07 11:15:20 +02:00
1 parent 509937c4b8
commit 902761d37b
2 files changed
+110 -3

No files matched your search

+106
View File
@@ -1632,6 +1632,112 @@ jobs:
if: always() && github.event_name != 'pull_request'
run: docker logout "$REGISTRY"
# Rolls ssh.dodotech.cloud onto the image the job above just pushed, by calling the deploy hook of the
# Spelunk stack that runs it. Without this nothing told the server a new image existed, so a commit with
# a migration in it reached the registry and stopped there, and the schema never moved.
#
# The migration itself is applied by the new container as it starts — Database:AutoMigrate, see
# Persistence.MigrateDodoDatabaseAsync — under an advisory lock, before its port opens. So triggering
# the deploy is what applies it, and the wait below is what proves it: /healthz/ready is unhealthy for
# as long as any migration the running image knows about is pending, and an image whose migration
# failed does not start at all.
#
# main only, like the :main tag the stack follows. A failure here fails the run, unlike Spelunk's own
# trigger: a server left behind its schema is the outage this job exists to prevent, and a green build
# over one is the quiet version of it.
#
# SPELUNK_DEPLOY_TOKEN is a repository secret (Settings -> Actions -> Secrets), sent as a header so it
# stays out of every proxy and runner log a URL would be written to.
deploy:
name: deploy ssh.dodotech.cloud
needs: [image]
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: [linux]
timeout-minutes: 15
# A newer push to main cancels this run like any other (see concurrency at the top). That is safe:
# once the hook has answered, the rollout and its migration happen on the server, and the newer
# run's own deploy job does the waiting instead.
env:
DEPLOY_HOOK: https://api.spelunk.dodotech.cloud/api/hooks/deploy
SERVER: https://ssh.dodotech.cloud
steps:
# See the desktop job's step of the same name.
- name: ensure curl
run: |
set -eu
SUDO=""
[ "$(id -u)" -eq 0 ] || SUDO="sudo"
if ! command -v curl >/dev/null 2>&1; then
echo "Installing curl"
if command -v apk >/dev/null 2>&1; then
$SUDO apk add --no-cache curl
elif command -v apt-get >/dev/null 2>&1; then
$SUDO apt-get update -qq
$SUDO apt-get install -y --no-install-recommends curl
elif command -v dnf >/dev/null 2>&1; then
$SUDO dnf install -y curl
else
echo "No apk, apt-get or dnf here, so curl cannot be installed from inside the job." >&2
exit 1
fi
fi
curl --version | head -1
- name: trigger the deploy
env:
SPELUNK_DEPLOY_TOKEN: ${{ secrets.SPELUNK_DEPLOY_TOKEN }}
run: |
set -eu
if [ -z "${SPELUNK_DEPLOY_TOKEN:-}" ]; then
echo "SPELUNK_DEPLOY_TOKEN is empty or unset, so the server was not told about this image." >&2
echo "Add it under Settings -> Actions -> Secrets (not Variables)." >&2
exit 1
fi
code="$(curl -sS -o "$RUNNER_TEMP/deploy-response" -w '%{http_code}' -X POST \
-H "Authorization: Bearer $SPELUNK_DEPLOY_TOKEN" \
"$DEPLOY_HOOK" || echo 000)"
case "$code" in
200|202|204) echo "Deploy hook accepted (HTTP $code)." ;;
401|404)
echo "Deploy hook rejected the token (HTTP $code): revoked or regenerated, or the stack is gone." >&2
exit 1 ;;
409)
echo "Deploy hook refused the token (HTTP 409): it is limited to a service the stack no longer has." >&2
exit 1 ;;
*)
echo "Deploy hook answered HTTP $code:" >&2
cat "$RUNNER_TEMP/deploy-response" >&2 || true
exit 1 ;;
esac
# Ten minutes, polled every ten seconds. A rollout restarts the container, so a few failed probes
# in the middle are expected; what fails the job is never becoming ready again — a migration that
# is still pending, one that crashed the start, or a stack holding the deploy for its schedule.
- name: wait for the server to be ready
run: |
set -eu
deadline=$(( $(date +%s) + 600 ))
last=""
while [ "$(date +%s)" -lt "$deadline" ]; do
sleep 10
last="$(curl -sS -m 10 -w ' (HTTP %{http_code})' "$SERVER/healthz/ready" 2>&1 || true)"
case "$last" in
"Healthy (HTTP 200)")
echo "$SERVER is ready, with no migration pending."
curl -sS -m 10 "$SERVER/api/v1/meta" || true
echo
exit 0 ;;
esac
echo "not ready yet: $last"
done
echo "$SERVER did not become ready within ten minutes. Last answer: $last" >&2
echo "A pending migration is named in the container's log (PendingMigrations)." >&2
exit 1
# No job here publishes the desktop *release* channel, and there is not going to be one. The desktop
# nightly job above publishes a different application — its own pack id, its own Velopack channel, its own
# install directory and profile — and the distance between those two sentences is the whole design. What