Watching the identity provider
Never paste a real secret, key, token or password into this site.
The Suite is its own identity provider, so its logs are the only place where a failing login shows up. Unless something here is watched, the first report of a problem is a phone call from whoever cannot get in.
What is emitted
Every authentication event is one JSON line on the Node service journal, written by lib/oidc/auth-telemetry.ts. Each line carries kind: "auth" and an event name:
| Event | Meaning |
|---|---|
login.succeeded | A password login completed |
login.password_change_required | Correct temporary password; the session can only reach the change-password page |
login.failed | Wrong credentials, or a request malformed enough to refuse before reading one |
login.rate_limited | The blocking rules of #30 refused the attempt |
login.failure_spike | Ten or more failures from one address or against one account inside ten minutes |
token.issued / token.refreshed | An authorization code or a refresh token was exchanged |
token.rejected | The token endpoint refused, with the OAuth error code in oauthError |
The fields are fixed: accountId, clientId, ipAddress, reason, oauthError, failures, windowSeconds, retryAfterSeconds, durationMs. There is no free-form field, and that is the point.
Why there is nothing to redact
The acceptance criterion on this work says no credentials, tokens or codes in the logs — not even truncated. A rule like that cannot be kept by remembering it, because the moment you most want to log a token is the moment you are debugging why one was rejected.
So the event type has no field a token would fit in, and every string that does get written passes a check first: anything longer than an identifier, or shaped like a JWT, is replaced with [redacted] before the line is built. A truncated token is still the first half of a token, so nothing is shortened — it is dropped.
The identifier someone typed at the login form is never logged either. A failed login records the address it came from and the fact that it failed; the account id appears only once an account has actually been authenticated. That way the log cannot be read backwards to learn which email addresses exist.
Where to look
On each Oracle VM, inspect journalctl -u suite.service. Authentication events are JSON lines in the journal's MESSAGE field. Useful filters:
kind = "auth" # everything from the provider
event = "login.failed" # failures only
event = "login.failure_spike" # the ones worth waking up for
event = "token.rejected" # a broken client, or a replay
oauthError = "invalid_grant" # specifically a code or refresh token that did not check out
durationMs is on the success events too. It is the cheapest early warning there is for the password hashing: if it starts climbing, inspect CPU pressure and database latency on the Oracle VM.
Checks from outside
/api/health answers ok when the Node service reaches SQLite. It does not prove that Harlock can sign in, so the identity provider is checked on its own by scripts/ci/oidc-smoke.mjs, without a credential and without producing a login attempt:
- discovery names
https://suite.sabrobotix.comas issuer, lists endpoints on that origin and advertises PKCES256, both grant types andRS256; - the JWKS publishes at least one RS256 signing key and no private key material (
d,p,q,dp,dq,qi,k); authorizefor theharlockclient withprompt=noneand no session redirects to Harlock's registered callback witherror=login_requiredand the samestate, which exercises the client lookup in SQLite and PKCE validation.
With the SMOKE_OIDC_IDENTIFIER and SMOKE_OIDC_PASSWORD repository secrets it also signs in the way Harlock does: password login, authorization code with PKCE, an id_token whose RS256 signature is checked against the JWKS together with issuer, audience and nonce, userinfo, and finally logout with id_token_hint, which revokes the refresh token the run was given. Nothing it handles is printed.
The account is login-monitor@sabrobotix.com, role Client, linked to no customer: it can sign in and sees nothing. Its failed attempts count towards the same lockout as anyone else's, so a wrong secret shows up as a failing check, not as a locked real user. To rotate its password, hash a new one with hashPassword from lib/oidc/password-hashing.ts, update its user_credentials row on the Oracle SQLite database, and replace the secret.
It runs after every production deploy (production.yml) and every two hours in uptime.yml, as a separate step after the health check. A failing run opens the usual Guasto automatico issue.
| Health check | Identity check | Where to look first |
|---|---|---|
| fails | not run | The Oracle service or SQLite: systemd journal, the last deploy, docs/database-restore.md |
| passes | fails on discovery or keys | The OIDC routes or the signing keys in SQLite (oidc_signing_keys, OIDC_KEY_WRAP_SECRET) |
| passes | fails on silent authorization | The harlock row in oauth_clients, or the authorize route |
| passes | fails on sign-in | Login defenses (a locked monitoring account), the token endpoint, the signing keys, or the secret itself |
Run it by hand with node scripts/ci/oidc-smoke.mjs https://suite.sabrobotix.com.
The two alarms
The Authentication alarms job in uptime.yml reads the last two hours of the Oracle suite journal over a dedicated restricted SSH connection. It uses SUITE_PROD_LOGS_SSH_KEY plus the existing SUITE_PROD_BACKUP_HOST and SUITE_PROD_BACKUP_KNOWN_HOSTS variables. The server must install suite-auth-logs and authorize that dedicated key as described in the internal docs/oracle-production.md runbook. Missing access fails the check explicitly; the backup/deploy keys cannot read logs.
It opens or updates Scheduled failure: Auth alarms when either condition is true:
- at least one
login.failure_spikein the last two hours. That event already means ten failures from one address or against one account in ten minutes; - at least five
token.rejectedevents in the last two hours. The normal value is zero; this threshold tolerates isolated expired or already-used grants without hiding a broken client or repeated replay.
The query and thresholds live in scripts/ci/check-auth-alerts.mjs. The old Workers Observability endpoint is no longer used, because production no longer writes there.
Test the alarm and recovery path
Run the Uptime workflow manually and enable Force a controlled authentication alarm. The journal export still has to succeed, then the drill deliberately fails the job and opens Scheduled failure: Auth alarms. Run the workflow again with the option disabled; both telemetry queries must pass. Check the run link in the issue and close it only after that recovery run succeeds.
The drill does not submit a bad password, create a rejected token, lock an account or change production data.
What is deliberately not here
There is no metrics pipeline and no Analytics Engine binding. At this number of users a queryable log line answers the same questions, and a pipeline would be one more thing to keep alive — which is the opposite of the reason the identity provider was moved here at all.