Skip to main content

Watching the identity provider

Never paste a real secret, key, token or password into this site.

The Suite is its own identity provider, so its logs are the only place where a failing login shows up. Unless something here is watched, the first report of a problem is a phone call from whoever cannot get in.

What is emitted​

Every authentication event is one JSON line on the Node service journal, written by lib/oidc/auth-telemetry.ts. Each line carries kind: "auth" and an event name:

EventMeaning
login.succeededA password login completed
login.password_change_requiredCorrect temporary password; the session can only reach the change-password page
login.failedWrong credentials, or a request malformed enough to refuse before reading one
login.rate_limitedThe blocking rules of #30 refused the attempt
login.failure_spikeTen or more failures from one address or against one account inside ten minutes
token.issued / token.refreshedAn authorization code or a refresh token was exchanged
token.rejectedThe token endpoint refused, with the OAuth error code in oauthError

The fields are fixed: accountId, clientId, ipAddress, reason, oauthError, failures, windowSeconds, retryAfterSeconds, durationMs. There is no free-form field, and that is the point.

Why there is nothing to redact​

The acceptance criterion on this work says no credentials, tokens or codes in the logs — not even truncated. A rule like that cannot be kept by remembering it, because the moment you most want to log a token is the moment you are debugging why one was rejected.

So the event type has no field a token would fit in, and every string that does get written passes a check first: anything longer than an identifier, or shaped like a JWT, is replaced with [redacted] before the line is built. A truncated token is still the first half of a token, so nothing is shortened — it is dropped.

The identifier someone typed at the login form is never logged either. A failed login records the address it came from and the fact that it failed; the account id appears only once an account has actually been authenticated. That way the log cannot be read backwards to learn which email addresses exist.

Where to look​

On each Oracle VM, inspect journalctl -u suite.service. Authentication events are JSON lines in the journal's MESSAGE field. Useful filters:

kind = "auth" # everything from the provider
event = "login.failed" # failures only
event = "login.failure_spike" # the ones worth waking up for
event = "token.rejected" # a broken client, or a replay
oauthError = "invalid_grant" # specifically a code or refresh token that did not check out

durationMs is on the success events too. It is the cheapest early warning there is for the password hashing: if it starts climbing, inspect CPU pressure and database latency on the Oracle VM.

Checks from outside​

/api/health answers ok when the Node service reaches SQLite. It does not prove that Harlock can sign in, so the identity provider is checked on its own by scripts/ci/oidc-smoke.mjs, without a credential and without producing a login attempt:

  • discovery names https://suite.sabrobotix.com as issuer, lists endpoints on that origin and advertises PKCE S256, both grant types and RS256;
  • the JWKS publishes at least one RS256 signing key and no private key material (d, p, q, dp, dq, qi, k);
  • authorize for the harlock client with prompt=none and no session redirects to Harlock's registered callback with error=login_required and the same state, which exercises the client lookup in SQLite and PKCE validation.

With the SMOKE_OIDC_IDENTIFIER and SMOKE_OIDC_PASSWORD repository secrets it also signs in the way Harlock does: password login, authorization code with PKCE, an id_token whose RS256 signature is checked against the JWKS together with issuer, audience and nonce, userinfo, and finally logout with id_token_hint, which revokes the refresh token the run was given. Nothing it handles is printed.

The account is login-monitor@sabrobotix.com, role Client, linked to no customer: it can sign in and sees nothing. Its failed attempts count towards the same lockout as anyone else's, so a wrong secret shows up as a failing check, not as a locked real user. To rotate its password, hash a new one with hashPassword from lib/oidc/password-hashing.ts, update its user_credentials row on the Oracle SQLite database, and replace the secret.

It runs after every production deploy (production.yml) and every two hours in uptime.yml, as a separate step after the health check. A failing run opens the usual Guasto automatico issue.

Health checkIdentity checkWhere to look first
failsnot runThe Oracle service or SQLite: systemd journal, the last deploy, docs/database-restore.md
passesfails on discovery or keysThe OIDC routes or the signing keys in SQLite (oidc_signing_keys, OIDC_KEY_WRAP_SECRET)
passesfails on silent authorizationThe harlock row in oauth_clients, or the authorize route
passesfails on sign-inLogin defenses (a locked monitoring account), the token endpoint, the signing keys, or the secret itself

Run it by hand with node scripts/ci/oidc-smoke.mjs https://suite.sabrobotix.com.

The two alarms​

The Authentication alarms job in uptime.yml reads the last two hours of the Oracle suite journal over a dedicated restricted SSH connection. It uses SUITE_PROD_LOGS_SSH_KEY plus the existing SUITE_PROD_BACKUP_HOST and SUITE_PROD_BACKUP_KNOWN_HOSTS variables. The server must install suite-auth-logs and authorize that dedicated key as described in the internal docs/oracle-production.md runbook. Missing access fails the check explicitly; the backup/deploy keys cannot read logs.

It opens or updates Scheduled failure: Auth alarms when either condition is true:

  • at least one login.failure_spike in the last two hours. That event already means ten failures from one address or against one account in ten minutes;
  • at least five token.rejected events in the last two hours. The normal value is zero; this threshold tolerates isolated expired or already-used grants without hiding a broken client or repeated replay.

The query and thresholds live in scripts/ci/check-auth-alerts.mjs. The old Workers Observability endpoint is no longer used, because production no longer writes there.

Test the alarm and recovery path​

Run the Uptime workflow manually and enable Force a controlled authentication alarm. The journal export still has to succeed, then the drill deliberately fails the job and opens Scheduled failure: Auth alarms. Run the workflow again with the option disabled; both telemetry queries must pass. Check the run link in the issue and close it only after that recovery run succeeds.

The drill does not submit a bad password, create a rejected token, lock an account or change production data.

What is deliberately not here​

There is no metrics pipeline and no Analytics Engine binding. At this number of users a queryable log line answers the same questions, and a pipeline would be one more thing to keep alive — which is the opposite of the reason the identity provider was moved here at all.