System administration & support

This is the handbook for whoever deploys, runs, and supports a HeadHunter instance — Waves Technology's platform operator. Unlike every other manual in this set, it isn't scoped to an NGO role: you sit outside any organization, on your own host, and your job is to keep the platform healthy for everyone on it without ever reaching into a tenant's data unless you're explicitly, temporarily invited.

It comes in two halves:

• Part A — the platform operator console: the ops realm you sign in to, and its three layers of access (aggregate → redacted → break-glass).
• Part B — deploy, run, operate: standing up the stack, provisioning, database, configuration, the candidate realm, health, and the support runbook.

The four principals — know which realm you're in

HeadHunter has four kinds of user, and their sessions are deliberately kept apart so a weakness in one can't cross into another:

One monolith serves all three hosts and routes by Host header. The ops session is separate by design — a shorter 2-hour life and its own secret isolate the operator console's blast radius from the org and candidate realms. Your account lives in platform_admins, not in any org's users table.

Part A — the platform operator console

Signing in

Go to /ops/login on the ops host (ops.localhost in dev, ops.<domain> in production). You authenticate against the platform-admin realm, which is entirely separate from every org's login — a wrong email and a wrong password return the same generic error, so the page never reveals whether an account exists. On success you get a 2-hour hh_ops_session; each request re-reads your record and re-confirms your account is still active. There is no self-serve signup — operators are provisioned from the command line (see Provisioning in Part B).

Note: the ops login does not yet have MFA or brute-force lockout — that's a planned fast-follow. Keep the ops host access tight at the network layer in the meantime.

The four operator roles

Your capabilities come from your platform role (platform_admins.role), a map kept completely separate from org RBAC so an operator can never inherit a tenant-scoped grant:

The rule to internalize: only superadmin and support can ever reach real candidate data, and only under an active grant. readonly and ai_agent are blocked from break-glass not just by the permission map but by a second structural check — loosening the map wouldn't let them in.

The three layers of access

The console is built as three concentric layers. You spend almost all your time in the outer two, which never expose personal data; the innermost is a rare, consent-gated exception.

Layer 1 — the aggregate dashboard (/ops/dashboard)

Your daily heartbeat view, PII-free by construction — its only data source computes counts and groupings and selects no personal columns (a test enforces that no PII column name even appears in it). It shows:

• Orgs — total and active (active = at least one application in the last 30 days).
• Users and postings (draft / open / closed).
• Applications — total and broken down by status.
• New-org signups per month — a six-month sparkline.
• Parse-queue health — queued / running / error, plus recent errors in the last 24h.
• Billing summary — pool-entitled orgs, active subscriptions by tier, recent manual payments.
• System health — database reachability + applied-migration count, and parser liveness.

A Layer-1 read is monitoring, not an action, so it writes no audit row.

What Layer 1 does not show: there are no candidate metrics here — no candidate-account count, no verification counts. The only "signups" figure is new orgs per month. Candidate accounts are self-serve on the marketing site and are not surfaced in the operator dashboard.

Layer 2 — per-org diagnostics (/ops/orgs, /ops/orgs/[id])

When an org needs support, start here. The directory (/ops/orgs) lists one row per tenant — status, subscription tier, whether suspended, and counts of users, open postings, and applications — with an optional ?q= search on the org name. It's PII-free and the directory read isn't audited.

Open an org to drill in. Every drill-in is audited (org.viewed), and what you see is deliberately redacted:

• Feature flags as booleans only — e.g. "email intake configured: yes/no," "WhatsApp configured: yes/no." The raw ingest slug or phone number is never surfaced.
• Staff accounts — org users' emails are shown; they are your client contacts, not candidates.
• Parser health with recent failures — where a failing CV appears, the candidate is masked to initials only. The raw name is used solely to derive those initials and is never returned; no CV file path or filename is exposed.
• The org's own audit trail, with the staff actor joined in.

From here a superadmin can suspend or reactivate a tenant (gated on platform.org.suspend); both actions are audited (org.suspended / org.reactivated).

Layer 3 — break-glass (/ops/orgs/[id]/breakglass)

This is the only path to real, un-redacted candidate data (name, email, phone, location, whether a CV exists), and it is fenced on every side. Treat it as the rare exception it is.

• Two gates, not one. You need the platform.breakglass.request permission and you must pass a structural eligibility re-check. Only superadmin and support are eligible; ai_agent and readonly can never enter, regardless of the permission map.
• A live, org-granted, time-boxed consent must exist. Access is computed at read time: the moment the grant expires or the org revokes it, the page falls back to a "no access — request access from the org" state and reads no data at all.
• Every real-data read is audited, on both sides, and stamped with the grant's id. No grant → no data → no audit row.
• TTL policy: a grant lasts at minimum 1 hour, defaults to 24 hours, and caps at 72 hours (3 days).

How a grant is created (the org side). The organization opens Settings → Support access (/settings/support-access on their app. host — an org_admin action). They pick a named, eligible operator, give a reason and an optional support reference, and set a TTL within the 1–72h window. That writes a grant with an expiry and audits it on both sides — the org's own trail (support_access.granted) and the operator mirror (breakglass.granted). The org can revoke instantly at any time, which is likewise mirrored. A grant is "live" only while it is unexpired and un-revoked.

The shape to remember: the org holds the key. You can ask, but you only ever see a tenant's real data while that tenant is actively, knowingly, and temporarily letting you — and every second of it is on the record for both of you.

Part B — deploy, run, operate

Stack & topology

HeadHunter is a boring monolith run with Docker Compose — four services, one command to bring up:

• nginx (nginx:alpine) — the TLS edge on ports 80 and 443. It terminates HTTPS and forwards the real Host as X-Forwarded-Host and the scheme as X-Forwarded-Proto, so the one app can serve all three hosts.
• app — the SvelteKit 2 / adapter-node application (built from ./app).
• parser — the one Python 3.12 FastAPI sidecar: stateless, CPU-only, and not publicly exposed. Only the app reaches it, over the internal compose network at http://parser:8000.
• db — postgres:15-alpine.

It's designed to fit a modest Hostinger EU VPS (the EU hosting is also the GDPR trust signal in the product's privacy promise) — a KVM 1 (1 vCPU / 4 GB) suffices for a pilot; KVM 2 (2 vCPU / 8 GB) gives headroom. Persistent state lives in two named volumes: postgres_data (the database) and cv_storage mounted at /app/uploads (raw CV files — PII, kept off the image and out of git). The app and parser run hardened (no-new-privileges, and the app drops all Linux capabilities but the few it needs).

Lifecycle — ./headhunter.sh

Everything routine goes through the lifecycle script at the repo root:

./headhunter.sh setup      # first run: create .env (with strong random secrets) + a self-signed cert
./headhunter.sh up         # build + start the stack (docker compose up -d --build)
./headhunter.sh status     # container state (docker compose ps)
./headhunter.sh logs       # follow the app logs (Ctrl-C to stop)
./headhunter.sh down       # stop everything — the postgres_data volume is preserved
./headhunter.sh restart    # down + up

setup copies .env.example → .env and fills DB_PASSWORD, SESSION_SECRET, and OPS_SESSION_SECRET with openssl rand values. In production, set CANDIDATE_SESSION_SECRET explicitly — see Configuration & secrets. Pass -y / --yes to skip confirmation prompts in scripted use.

Provisioning

There is no self-serve signup for orgs or operators — both are created deliberately from the command line. Run these from /app with DATABASE_URL set (or via the headhunter.sh wrappers, which run them against the live db container).

A new organization and its first org_admin:

npm run provision -- \
  --org-name "Mercy NGO" --org-name-ar "منظمة الرحمة" \
  --email admin@mercy.org --password 'StrongPass123' --full-name "Aisha Ali" \
  [--role org_admin] [--locale ar] [--tier pilot] [--pool-entitlement]
# or: ./headhunter.sh provision --org-name "Mercy NGO" --email admin@mercy.org ...

A platform operator (the first superadmin is created this way; further operators are then managed in-console):

npm run provision-admin -- \
  --email ops@waves.tech --password 'StrongPass123' --full-name "Operator Name" \
  [--role superadmin|support|readonly|ai_agent]
# or: ./headhunter.sh provision-admin --email ops@waves.tech ... [--role support]

Role defaults to superadmin if omitted. Re-running with an existing email is rejected (emails are globally unique). Passwords are hashed with Argon2id; a minimum length of 8 is enforced.

Database & migrations

Schema changes are numbered SQL migrations — 0NNN_name.sql with a matching 0NNN_name_down.sql — under postgres/migrations/. init.sql is the baseline only and is never edited for live schema.

npm run db:migrate         # apply pending migrations (tsx scripts/migrate.ts up) — needs DATABASE_URL
npm run db:rollback        # roll back the most recent migration (down)
# via the stack: ./headhunter.sh migrate | rollback | db-reset

On a fresh database, the compose db container runs apply-migrations.sh after init.sql, applying every migration in order automatically. The /health endpoint reports the applied-migration count, so you can confirm a deploy is fully migrated at a glance. ./headhunter.sh db-reset wipes the DB volume and rebuilds a clean baseline — destructive; it asks for confirmation.

Configuration & secrets

Configuration is environment variables in .env (never committed). The ones that matter most:

Two footguns worth calling out:

• Never set ORIGIN. adapter-node derives the per-request origin from the forwarded proxy headers so the one app can serve marketing, app, and ops hosts. A fixed ORIGIN would flatten the split and break host routing.
• CANDIDATE_SESSION_SECRET and the empty-string trap. docker-compose injects CANDIDATE_SESSION_SECRET= (an empty string) when the variable is absent from .env; the app treats empty as unset and falls back to SESSION_SECRET so a minimal deploy still works. That's fine for dev, but in production set it to its own value — the whole point is to isolate the candidate realm's blast radius.

Outbound email / WhatsApp and the LLM are provider-agnostic. For dev and pilot without a live provider, outbound messages are written as JSON lines to a file outbox (COMMS_OUTBOX_FILE) instead of actually being sent — useful, but see the candidate-realm caveat below.

The candidate realm — operating the 4th principal

Phase 5 added the candidate account: an internet-facing, self-serve realm at scale, so it earns its own operational attention.

• What it is. The candidate_accounts, candidate_profiles, and candidate_email_tokens tables (migrations 0041+, 0043), behind the public /jobs board and the signed-in /me/* console, with the separate hh_candidate_session cookie. It lives on the marketing apex — not on app. or ops.. The operator console does not manage candidate accounts; they are self-serve end to end.

• Mandatory verified email. A new account is pending until the candidate clicks the emailed verification link, and cannot apply or receive org comms until verified. This makes a live email provider a hard dependency for the candidate realm — the dev/pilot file-outbox will not deliver a real verification link, so plan a real sender before opening the board to real job-seekers. Deliverability into Sudan is your concern to monitor.

• Retention. Enforced by an idempotent job on the VPS (manual or cron), not an in-app worker:

  npm run candidate-retention -- [--days 30] [--dry-run]     # from /app, needs DATABASE_URL
  

  It purges never-verified (pending) accounts older than the window (default 30 days) plus their CV files, writing a candidate.deleted audit row (reason retention_unverified) before deletion so the trail survives anonymized. --dry-run lists what would go without deleting. Dormant active accounts are deliberately out of scope — there is no automatic purge of a verified account; that would be a separate, deliberate policy decision, not a silent default.

• Export & delete (self-serve). A candidate downloads their own data at /me/settings → Export (GET /me/settings/export → my-data-export.json): account, profile, CV metadata (the binaries stay downloadable separately, not inlined), applications with candidate-safe status, and saved jobs. They erase their account from Settings → Delete account (re-auth + a typed DELETE confirmation): it audits candidate.deleted first, then removes the self-owned rows, detaches the org intake record (applications.candidate_account_id set to null, so the org's record survives without the personal link), and best-effort deletes the CV files from disk. These fulfil the EU-hosting / no-selling privacy promise. You only act here if a candidate genuinely can't self-serve — the flows are theirs, not yours.

• Abuse posture. The internet-facing signup is rate-limited (5 sign-ups per 10 minutes per IP) and carries a honeypot field that, when tripped by a bot, is swallowed as a neutral success (the bot is never told it failed). Verification-email resends and email changes are throttled too (3 per 10 minutes per account). Watch for spikes in 429s or unusual signup volume.

Health & monitoring

• /health (on the app) is the liveness probe. It reports database connectivity with the applied-migration count, and the parser sidecar's own /health, and degrades gracefully — a missing DB or parser reports its own error status instead of 500-ing the probe. The parser container also has its own compose healthcheck against /health.
• The parse queue is surfaced on the Layer-1 dashboard (queued / running / error + recent 24h errors) — your first signal that CV parsing is backing up or failing.
• Remember: candidate signups and verification do not appear in Layer 1. If you need to reason about candidate-account volume, that's the database, not the dashboard.

Support runbook

Route every request through the least-privileged path that answers it. Redacted Layer 2 first; break-glass only as a consent-gated last resort.

Backups. The durable state is the postgres_data volume plus the cv_storage (/app/uploads) CV files. Back up both together on the VPS — the database alone, without the CV files it references, is an incomplete restore. (./headhunter.sh down preserves the volumes; only db-reset destroys the database.)

Quick reference

You run the platform that other people's careers and hiring decisions rest on. The whole design here points one way: keep it healthy from the aggregate view, help tenants from a redacted distance, and reach into real data only when a tenant is actively, temporarily holding the door open for you.