Redline experiment: operator setup (Tailscale-only)

Use this guide to deploy Redline’s Stage 1 experiment on the GitLab Docker host: the dashboard and API on one origin, production at https://yqa.fio.sh and persistent QA at https://qa-yqa.fio.sh, reachable only on the tailnet. It covers experiment plan tasks T4.3 and T6.7. Follow the sections in order.

QA can instead run on any computer on the tailnet, published through a Cloudflare Tunnel: see Redline local QA. In that case skip the qa-yqa.fio.sh DNS record, certificate name and nginx server block here, and start only production in section 13.

This replaces the Cloudflare Access and Tunnel parts of the container runbook for the experiment. Sections 1, 4 and 7 of that runbook still apply where this guide points to them.

Result

Reader on the tailnet
  ├─ https://yqa.fio.sh/      → host nginx → 127.0.0.1:18787 → fio-redline     (released)
  └─ https://qa-yqa.fio.sh/   → host nginx → 127.0.0.1:18788 → fio-redline-qa  (verified)
        both containers → glab.fio.sh-bridge → http://glab.fio.sh-server
Off the tailnet: both names resolve to a 100.x address that does not answer.

Each environment serves the dashboard at / and the API at /v1/ from the same container. No Cloudflare Access, Tunnel or cloudflared container.

Before you begin

You need:

• shell access to the GitLab Docker host, with Docker, Docker Compose, nginx and certbot;
• GitLab administrator access (OAuth application, deploy token, group token);
• DNS edit access to the fio.sh zone;
• a device on the tailnet with this repository checked out and Node 22, for the checks in sections 11 to 13;
• a password manager for values GitLab displays once.

Never paste a token or secret into a terminal command, an issue, chat, or this repository. Paste them only into the files named below, with an editor.

Values you will create

1. Verify the host

Run runbook section 1 unchanged. Then confirm the two loopback ports this deployment uses are free:

ss -ltn | grep -E ':(18787|18788)\b' || echo "ports free"

Expected: ports free. If either port is taken, pick free ones and change them in docker-compose.tailscale.yml and in the nginx file (section 9) together.

2. DNS

glab.fio.sh already resolves to the host’s Tailscale address. The two new names point at the same address.

1. In the DNS provider for fio.sh, open the record for glab.fio.sh and note its value, a 100.x.y.z address.

2. If a record for yqa.fio.sh already exists (for example a Cloudflare Tunnel CNAME from the earlier runbook), delete it.

3. Add two A records with that same address. If the provider is Cloudflare, set Proxy status to DNS only:

   Name	Type	Value
   yqa	A	the glab.fio.sh address
   qa-yqa	A	the glab.fio.sh address

4. From a tailnet device, check both names:

   dig +short yqa.fio.sh
   dig +short qa-yqa.fio.sh

   Expected: the same 100. address for both. Keep this output; it is evidence for T4.3.

3. TLS certificate

Issue the certificate the same way the glab.fio.sh certificate is issued. The infrastructure how-to shows certbot --standalone, but an HTTP-01 challenge cannot reach a name that resolves to a tailnet address, so first check which method the existing certificate really uses:

sudo grep -E '^(authenticator|dns_|server)' /etc/letsencrypt/renewal/glab.fio.sh.conf

Then issue one certificate covering both names with the same authenticator and its options. For example, if the output shows authenticator = dns-cloudflare with a credentials file:

sudo certbot certonly --cert-name yqa.fio.sh -d yqa.fio.sh -d qa-yqa.fio.sh \
  --dns-cloudflare --dns-cloudflare-credentials <the same credentials path>

Expected: certificates under /etc/letsencrypt/live/yqa.fio.sh/. Renewal follows the existing certbot timer.

4. GitLab OAuth application

One application serves both environments.

1. Sign in to https://glab.fio.sh as an administrator and open Admin > Applications.

2. If an application named Fridai Redline exists, select Edit. Otherwise select New application and name it Fridai Redline.

3. Set:

   Field	Value
   Redirect URI	two lines, exactly: https://yqa.fio.sh/v1/auth/gitlab/callback and https://qa-yqa.fio.sh/v1/auth/gitlab/callback
   Confidential	selected
   Trusted	not selected
   Scopes	api only

4. Save. Copy the Application ID (GITLAB_OAUTH_CLIENT_ID). For a new application, copy the Secret immediately into your password manager; for an existing one, use its stored secret.

5. Keep a screenshot of the redirect URIs; it is evidence for T4.3.

5. GitLab read token

The service reads history and merge requests with one token before a reader connects their own GitLab account. Because the production service also keeps the yyz site registered, a token for one project is not enough.

1. Open the group fridai, then Settings > Access tokens > Add new token.
2. Set: name fio-redline-read, role Reporter, scope read_api only, an approved expiry date.
3. Copy the token into your password manager as GITLAB_TOKEN, and add the expiry to the operational calendar.

If you prefer a project token, create it on fridai/fio-dep/fio-redline (project 16) instead, and delete the yyz entry from SITES_JSON in service.env (section 8).

6. Registry pull credential

The images live in fridai/fio-registry.

1. Open fridai/fio-registry, then Settings > Repository > Deploy tokens.

2. Create a token named fio-redline-host with scope read_registry only. Store the username and token as REGISTRY_DEPLOY_USER and REGISTRY_DEPLOY_TOKEN.

3. On the host:

   docker login glab.fio.sh:5050 --username <REGISTRY_DEPLOY_USER>

   Paste the token at the password prompt.

7. Host directory and secrets

sudo install -d -m 0700 -o "$USER" -g "$(id -gn)" /srv/fio-redline
install -d -m 0700 /srv/fio-redline/secrets
cd /srv/fio-redline/secrets
for name in session-secret session-secret-qa draft-signing-secret gitlab-webhook-secret; do
  openssl rand -hex 32 -out "$name" && chmod 0600 "$name"
done
install -m 0600 /dev/null gitlab-token
install -m 0600 /dev/null gitlab-oauth-client-secret
# Declared by the compose file for the Tunnel profile, unused here: an empty placeholder.
install -m 0600 /dev/null cloudflared-tunnel-token
${EDITOR:-vi} gitlab-token                  # paste GITLAB_TOKEN only
${EDITOR:-vi} gitlab-oauth-client-secret    # paste the OAuth application secret only

Production and QA share every secret except the session secret, so a session cookie from one environment is never accepted by the other.

8. Deployment files

From a trusted checkout of this repository at the merged main:

install -m 0644 ops/docker-compose.yml /srv/fio-redline/docker-compose.yml
install -m 0644 ops/docker-compose.tailscale.yml /srv/fio-redline/docker-compose.tailscale.yml
install -m 0600 ops/.env.example /srv/fio-redline/.env
install -m 0600 ops/service.env.example /srv/fio-redline/service.env
install -m 0600 ops/service-qa.env.example /srv/fio-redline/service-qa.env

Edit /srv/fio-redline/.env and uncomment the override line, so it reads:

FIO_REDLINE_IMAGE=glab.fio.sh:5050/fridai/fio-registry/fio-redline:released
FIO_REDLINE_QA_IMAGE=glab.fio.sh:5050/fridai/fio-registry/fio-redline:verified
GLAB_DOCKER_NETWORK=glab.fio.sh-bridge
COMPOSE_FILE=docker-compose.yml:docker-compose.tailscale.yml

Edit /srv/fio-redline/service-qa.env: replace GITLAB_OAUTH_CLIENT_ID=replace-with-gitlab-oauth-application-id with the Application ID. Everything else is already the experiment configuration.

Edit /srv/fio-redline/service.env: set the Application ID the same way, then change these lines and delete the two ACCESS_* lines:

IDENTITY_MODE=gitlab-oauth
ALLOW_ANONYMOUS_READ=true
REQUIRE_ACCESS=false
WRITE_ENABLED=true

SITES_JSON in both files already registers fio-redline on its own origin.

Check the files without printing them:

cd /srv/fio-redline
for f in gitlab-token gitlab-oauth-client-secret session-secret session-secret-qa draft-signing-secret gitlab-webhook-secret; do
  test -s "secrets/$f" || echo "missing secrets/$f"
done
docker compose --profile qa config --quiet && echo "compose ok"

Expected: only compose ok.

9. Host nginx

Create /etc/nginx/sites-available/yqa.fio.sh:

server {
    listen 80;
    server_name yqa.fio.sh qa-yqa.fio.sh;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name yqa.fio.sh;

    ssl_certificate     /etc/letsencrypt/live/yqa.fio.sh/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yqa.fio.sh/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    add_header Strict-Transport-Security "max-age=31536000" always;
    client_max_body_size 4m;

    location / {
        proxy_pass http://127.0.0.1:18787;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;
        proxy_read_timeout 60;
    }
}

server {
    listen 443 ssl http2;
    server_name qa-yqa.fio.sh;

    ssl_certificate     /etc/letsencrypt/live/yqa.fio.sh/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yqa.fio.sh/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    add_header Strict-Transport-Security "max-age=31536000" always;
    client_max_body_size 4m;

    location / {
        proxy_pass http://127.0.0.1:18788;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;
        proxy_read_timeout 60;
    }
}

If the host has a public interface, bind each listen to the Tailscale address, as the glab.fio.sh how-to describes. Then:

sudo ln -s /etc/nginx/sites-available/yqa.fio.sh /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Keep the nginx -t output; it is evidence for T4.3. Until the containers run, both hosts answer 502; that is expected.

10. Start QA

QA tracks the verified channel, which carries the dashboard once the experiment branch is merged to main and that pipeline promotes it. Start only QA for now: production stays on released until section 13 promotes a build that understands the experiment settings.

cd /srv/fio-redline
docker compose --profile qa pull service-qa
docker compose --profile qa up --detach service-qa
docker compose --profile qa ps

Expected: fio-redline-qa is healthy. If not:

docker compose --profile qa logs --tail 100 service-qa

11. Webhooks

GitLab tells each environment about changes so its cache refreshes. On the project fridai/fio-dep/fio-redline, Settings > Webhooks, add two hooks with the triggers Push events, Tag push events, Comments and Merge request events, and the contents of secrets/gitlab-webhook-secret as Secret token:

Use Test > Push events on the QA hook; Recent events must show 202. Test the production hook after section 13. If GitLab refuses the internal URL, see runbook section 12.

12. Verify QA and run the Stage 1 demonstration

From a tailnet device, in this repository:

npm ci
npx playwright install chromium
curl -s https://qa-yqa.fio.sh/v1/health
npm run smoke -- https://qa-yqa.fio.sh fio-redline kb/index.md
REDLINE_LIVE_URL=https://qa-yqa.fio.sh npm run test:live

Expected: health has "ok":true and "reachable":true; the smoke prints one JSON line with "state":"connect-required" (or "live") and exits 0; the live spec prints 4 passed.

Then disconnect Tailscale on that device and run the smoke again:

npm run smoke -- https://qa-yqa.fio.sh fio-redline kb/index.md; echo "exit $?"

Expected: offline-network and exit 2. Reconnect Tailscale.

The demonstration, in a browser on the tailnet:

1. Open a merge request on fridai/fio-dep/fio-redline that changes one paragraph of a page under kb/ (for example a sentence in kb/setup/index.md). Leave it open.
2. Open that page on https://qa-yqa.fio.sh/kb/…/. Press the merge request’s toggle above the article: the change shows inline as struck and inserted words. Take a screenshot.
3. Select a few words in the article, press Connect GitLab to comment, authorize, and post a comment.
4. In GitLab, open the issue [Document Review] kb/… on the project: your comment is there under your own account.

13. Promote and start production

1. In the main pipeline that produced the QA image, run the manual job promote-container-released.

2. On the host:

   cd /srv/fio-redline
   docker compose pull service
   docker compose up --detach service
   docker compose ps

   Expected: fio-redline is healthy; fio-redline-qa is still running.

3. Test the production webhook (section 11).

4. Repeat section 12’s checks with https://yqa.fio.sh.

14. Evidence to record

Send these to the executor, or paste them under T4.3 and T6.7 in kb/plans/redline-experiment/STATUS.md. No secrets, no cookies.

• dig +short for both names, the nginx -t output, a screenshot of the OAuth redirect URIs;
• docker compose --profile qa ps showing both image tags;
• per environment: the health JSON, the smoke output on and off the tailnet, the test:live output;
• the merge request URL, the issue URL and the screenshot path from the demonstration;
• the dates.

Updating and rolling back

QA follows verified and production follows released, each on its next docker compose pull <service> && docker compose up --detach <service>. To roll back, set the image in .env to an exact earlier version (for example FIO_REDLINE_IMAGE=glab.fio.sh:5050/fridai/fio-registry/fio-redline:26.10.10-84.5ce4d2ec) and run the same two commands. Never rebuild an image to release it.