# Hosting municipalities (operator runbook)

Run one HelixIAM installation for several Dutch municipalities (gemeenten), one realm each: onboarding, day-2
operations, upgrades and offboarding.

## What it is

This runbook is for the operator who hosts HelixIAM for municipalities. It uses only standard Kubernetes and Helm, so
it applies to any conformant cluster, including VNG Haven clusters. Every municipality gets its own
[realm](../manage/realms.md): its own users, applications, signing keys, sign-in flows, theme, DigiD and eHerkenning
connections, and administrators.

The Helm chart ships the files this runbook uses. After `helm registry login ghcr.io` with the registry credentials
from your licence, unpack it with
`helm pull oci://ghcr.io/melmarzgioui/helixiam/charts/helixiam --version 1.1.0-rc.14 --untar`:

| Piece | Where |
|---|---|
| Hosting profile (values file) | `helixiam/examples/values-multi-tenant-municipalities.yaml` in the chart |
| Chart documentation: HA, monitoring and alerts, backups, Haven | `helixiam/README.md` in the chart |
| Generic `pg_dump` backup CronJob | `helixiam/examples/postgres-backup-cronjob.yaml` in the chart |
| Readiness pack in Dutch: BIO2 mapping, DPIA input, DigiD/eHerkenning onboarding, processor-agreement input | available to licensed customers on request |

## Install the platform once

1. **Create the Secret.** The chart never inlines secrets. The Secret holds the database credentials, the
   `db-encryption` key (16, 24 or 32 characters; it encrypts the realm signing keys and every other secret in the
   database and must **never change**), the bootstrap admin password and the secret of the provisioning service
   account:

    ```bash
    kubectl create namespace identity
    kubectl -n identity create secret generic helixiam-secrets \
      --from-literal=db-username=helixiam \
      --from-literal=db-password="$DB_PASSWORD" \
      --from-literal=db-encryption="$(openssl rand -hex 16)" \
      --from-literal=admin-password="$(openssl rand -base64 24)" \
      --from-literal=bootstrap-client-secret="$(openssl rand -hex 32)"
    ```

    Copy the `db-encryption` value into your secrets vault **now**, separately from any database backup. A database
    backup is useless without it.

2. **Adapt the profile.** Copy `helixiam/examples/values-multi-tenant-municipalities.yaml` from the unpacked chart and change every value marked `CHANGE`: the
   host names (`config.idpBaseUrl`, `config.spBaseUrl`, `ingress.hosts`, `ingress.tls`, the console's beheer host in
   `console.ingress` and `config.adminAllowedHosts`), the ingress controller's pod
   CIDR in `config.trustedProxies`, `database.host`, the namespace selectors of the NetworkPolicy, and the labels your
   Prometheus selects on (`serviceMonitor.labels`, `prometheusRule.labels`). Set `imagePullSecrets` to a
   `docker-registry` Secret in the namespace with the registry credentials from your licence. The profile connects to PostgreSQL with
   `database.sslMode: verify-full`: create the Secret with the CA that signed the database server certificate first
   (`kubectl -n identity create secret generic postgres-ca --from-file=ca.crt=./postgres-ca.pem`, or set
   `database.caSecret.name` to yours). Decide the audit-log retention with the municipalities
   (`retention.auditLogMaxAge`, empty keeps it).

3. **Install:**

    ```bash
    helm upgrade --install helixiam oci://ghcr.io/melmarzgioui/helixiam/charts/helixiam --version <chart version> \
      -n identity -f values-municipalities.yaml
    kubectl -n identity rollout status deploy/helixiam
    kubectl -n identity rollout status deploy/helixiam-console
    ```

What the profile sets, and why:

| Concern | Values |
|---|---|
| High availability | `replicaCount: 3` (two at least), `strategy` RollingUpdate with `maxUnavailable: 0`, `topologySpread.enabled` (zones and nodes), `podDisruptionBudget.minAvailable: 2` |
| Shared state | `config.sessionStore: queue`, `config.tokenStore: queue`, `config.clusterStateStore: jdbc`: sessions, tokens, flow state and rate limits in PostgreSQL, counted once for the cluster. The chart passes `replicaCount` to the server as `HELIX_CLUSTER_EXPECTED_REPLICAS`. With Redis instead, set `redis.host` and the stores to `redis` |
| Proxies and host names | `config.trustedProxies` (`HELIX_TRUSTED_PROXIES`), `config.allowedHosts` (`HELIX_ALLOWED_HOSTS`): see [Security hardening](security.md#reverse-proxies-and-host-names) |
| Pod security | non-root, `readOnlyRootFilesystem`, all capabilities dropped, RuntimeDefault seccomp, no service-account token, resource requests and limits |
| Network | `networkPolicy.allowSameNamespaceIngress: false`, ingress only from the ingress controller's and the monitoring namespace, egress restricted to DNS, PostgreSQL, 443 and SMTP |
| Monitoring | `config.prometheusAnonymous: true` with `serviceMonitor.enabled` and `prometheusRule.enabled` |
| Database TLS | `database.sslMode: verify-full` with `database.caSecret` (mounted read-only, passed as `DB_SSLMODE` / `DB_SSL_ROOT_CERT`): the connection is encrypted and the server certificate and host name are verified. Without it the JDBC driver default (`prefer`) does not verify the server |
| Data retention | Expired sessions, tokens, one-time codes and links are deleted on a schedule (`retention.enabled`, batches, safe with several replicas); the audit log only with `retention.auditLogMaxAge`, a policy decision |
| Admin console | `console.enabled` with 2 replicas, its own Deployment on an internal **beheer** host (`console.ingress`, removed from the server Ingress), an ingress-only `console.networkPolicy`. The server still scales to 3+ on its own. See [Architecture](../getting-started/architecture.md#the-admin-console) |
| Administration only on the beheer host | `config.adminAllowedHosts: [beheer host]`, `config.adminRestrictMasterRealm: true`: `/admin/**`, the API docs and the master realm answer `404` on the public sign-in host; the municipalities' realms are unaffected. See [Install](../getting-started/install.md#admin-api-only-on-the-admin-host) |
| Provisioning | `secrets.bootstrapClientId: provisioner`: a `client_credentials` client with the master `admin` role, so automation never signs in as the bootstrap admin and `config.mfaEnabled` stays on |

!!! danger "Keep `/actuator/prometheus` off the public ingress"
    The ServiceMonitor needs anonymous scraping. The NetworkPolicy admits only the monitoring namespace besides the
    ingress controller, but the chart's Ingress routes every path of its hosts: deny `/actuator/` in your ingress
    controller or gateway.

### A token for the admin API

The examples below call the admin API with a bearer token of the provisioning service account, so they need no CSRF
header (see [Authenticating to the API](../getting-started/api-authentication.md)). The profile serves the admin API and
the master realm **only on the beheer host** (`config.adminAllowedHosts`), so `HELIX_URL` is the beheer host here, not
the public sign-in host (which answers `404` for both). Run them from a network that reaches the internal beheer
ingress, or call an in-cluster name you also listed in `config.adminAllowedHosts`:

```bash
export HELIX_URL=https://beheer.login.example.nl
PROVISIONER_SECRET=$(kubectl -n identity get secret helixiam-secrets \
  -o jsonpath='{.data.bootstrap-client-secret}' | base64 -d)
TOKEN=$(curl -s -u "provisioner:$PROVISIONER_SECRET" -d grant_type=client_credentials \
  "$HELIX_URL/realms/master/oauth2/token" | jq -r .access_token)
AUTH="Authorization: Bearer $TOKEN"
```

The token is valid for the master realm's access-token lifetime (one hour by default); fetch a new one when it
expires.

## Onboard a municipality

The HelixIAM release tests run steps 1 to 3, 5 and the export of the offboarding section against a two-replica
install, with every request of a sign-in alternating between the pods.

### 1. Create the realm

Pick the realm id first: it is part of every URL, every token's `iss` and the SAML entity ids
(`https://login.example.nl/realms/<realm>/…`), and DigiD and eHerkenning are registered against those URLs. Use a
stable, URL-safe name such as `gemeente-voorbeeld`.

=== "Admin API"

    Saving the realm's settings creates it:

    ```bash
    REALM=gemeente-voorbeeld
    curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" \
      -d '{"displayName":"Gemeente Voorbeeld","accessTokenTtlSeconds":300,"refreshTokenTtlSeconds":86400,
           "enabled":true,"passwordMinLength":12,"ssoSessionIdleTimeoutSeconds":1800,
           "ssoSessionMaxLifetimeSeconds":36000}' \
      "$HELIX_URL/admin/realms/$REALM/settings"
    ```

=== "Realm config folder"

    Keep one realm document per municipality in git (the format of `GET /admin/realms/{realm}/export`, see
    [Import / export](../integration/import-export.md)); the file name is the realm id. Mount them with the chart's
    `realmConfig` values; every replica applies them at start-up:

    ```bash
    kubectl -n identity create configmap helixiam-realms \
      --from-file=gemeente-voorbeeld.json --dry-run=client -o yaml | kubectl apply -f -
    helm upgrade helixiam oci://ghcr.io/melmarzgioui/helixiam/charts/helixiam --version <chart version> \
      -n identity --reset-then-reuse-values \
      --set realmConfig.enabled=true --set realmConfig.configMap.name=helixiam-realms
    ```

    With `realmConfig.onConflict=skip` (the default) a restart never changes what already exists; `overwrite`
    reconciles the realm to the files. Secrets in the files are `${HELIX_<REALM>_…}` placeholders: provide them with
    `config.extraEnv` from a Secret, and set `realmConfig.requireEnv=true` so a file whose secret is missing is skipped
    instead of half-applied. A minimal file:

    ```json
    {"formatVersion": 2, "realm": {"displayName": "Gemeente Voorbeeld", "accessTokenTtlSeconds": 300,
     "refreshTokenTtlSeconds": 86400, "enabled": true, "passwordMinLength": 12}}
    ```

    Up to and including 1.0.0-rc.12, a skip-mode import does not create a realm that does not exist yet (fixed in
    1.0.0-rc.13). On those older releases create the realm with the admin API first, or use
    `realmConfig.onConflict=overwrite`.

Set the realm's sign-in policy (MFA, password policy, lockout) in the same settings, see
[Realm settings](realm-settings.md) and [Password policy & lockout](../authentication/password-policy.md).

### 2. Administrators

Creating a realm also creates its **bootstrap admin**, `admin-<realm>`, with the bootstrap admin password of the
installation (the same for every realm). Replace that password and disable the account before anyone else can use it,
then give the municipality named administrators:

```bash
BOOT_ID=$(curl -s -H "$AUTH" "$HELIX_URL/admin/realms/$REALM/users" \
  | jq -r --arg u "admin-$REALM" '.[] | select(.username == $u) | .userId')
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" \
  -d "{\"newPassword\":\"$(openssl rand -base64 32)\"}" \
  "$HELIX_URL/admin/realms/$REALM/users/$BOOT_ID/password"
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" \
  -d "{\"username\":\"admin-$REALM\",\"enabled\":false}" \
  "$HELIX_URL/admin/realms/$REALM/users/$BOOT_ID"

# A named administrator with a one-time password they must change at first sign-in (TOTP enrolment follows
# with config.mfaEnabled=true)
ADMIN_ID=$(curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"username":"j.jansen","email":"j.jansen@voorbeeld.nl","enabled":true}' \
  "$HELIX_URL/admin/realms/$REALM/users" | jq -r .userId)
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" \
  -d "{\"newPassword\":\"$ONE_TIME_PASSWORD\"}" "$HELIX_URL/admin/realms/$REALM/users/$ADMIN_ID/password"
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" -d '{"requiredActions":"UPDATE_PASSWORD"}' \
  "$HELIX_URL/admin/realms/$REALM/users/$ADMIN_ID/required-actions"
ROLE_ID=$(curl -s -H "$AUTH" "$HELIX_URL/admin/realms/$REALM/roles" | jq -r '.[] | select(.name == "admin") | .roleId')
curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" -d "{\"roleId\":\"$ROLE_ID\"}" \
  "$HELIX_URL/admin/realms/$REALM/users/$ADMIN_ID/roles"
```

Hand the one-time password over out of band.

The bootstrap admin is recreated only when it is missing, so keep it disabled rather than deleting it. The realm role
`admin` holds `realm-admin`; give help-desk and audit staff narrower roles ([Admin roles](admin-roles.md)). A
municipality's administrators manage only their own realm; keep the `master` realm to the hosting team.

### 3. Branding

A realm theme (colours, logo, texts, fonts) is set with `PUT /admin/realms/{realm}/theme`; [Theming](theming.md)
describes the model. Shared, file-based themes can be mounted with the chart's `themes` values and
selected per realm with `PUT /admin/realms/{realm}/theme/base`.

```bash
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"colors":{"primary":{"light":"#1d4ed8"}},"assets":{"logoUrl":"https://www.voorbeeld.nl/logo.svg"}}' \
  "$HELIX_URL/admin/realms/$REALM/theme"
```

Also set the realm's email provider and templates, so codes and notices come from the municipality
([Notifications](notifications.md)).

### 4. DigiD and eHerkenning

Each municipality brings its own DigiD connection (Logius) and eHerkenning contract (a broker), and its own
PKIoverheid certificates. Create the connections in the municipality's realm as described in
[DigiD](../federation/digid.md) and [eHerkenning](../federation/eherkenning.md):

- Use separate connections for the scheme's test and production environments, for example aliases `digid-preprod`
  and `digid`, each with its own metadata and certificates.
- The Assertion Consumer Service URL to register is `https://login.example.nl/realms/<realm>/broker/<alias>/callback`.
- Set `minimumLoa` to the level each service needs; it is requested and enforced.
- The artifact binding resolves over mutual TLS on port 443 (allowed by the profile's egress policy), with the SP
  signing certificate as the client certificate. The Logius server certificate must chain to the image's default Java
  trust store; there is no per-connection trust store.
- Decide which applications may receive the BSN. A DigiD user's username is the BSN unless the connection has a
  username template ([Mappers](../federation/mappers.md)), and `preferred_username` carries the username.

The Dutch onboarding guide in the readiness pack (available to licensed customers) maps every Logius and
eHerkenning requirement to these settings.

### 5. The municipality's own signing certificate

Each realm signs tokens and SAML assertions with its own key. To use a certificate from the municipality's CA, let
Helix generate the key and a CSR (the private key never leaves the server), have it signed, attach the chain, and
activate the key once relying parties have picked it up ([Realm keys](realm-keys.md)):

```bash
curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"algorithm":"RS256","subject":"CN=login.example.nl, O=Gemeente Voorbeeld, C=NL","dnsNames":["login.example.nl"]}' \
  "$HELIX_URL/admin/realms/$REALM/keys/csr" | tee csr-response.json | jq -r .csrPem > realm.csr
KEY_ID=$(jq -r .key.keyId csr-response.json)

# After the CA returns the certificate (leaf first, then the issuing CA certificates):
jq -n --rawfile chain signed-chain.pem '{certificateChainPem: $chain}' |
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" --data-binary @- \
  "$HELIX_URL/admin/realms/$REALM/keys/$KEY_ID/certificate"

# The key is PENDING and already in the JWKS; activate it after the staging time (5 minutes by default)
curl -s -X POST -H "$AUTH" "$HELIX_URL/admin/realms/$REALM/keys/$KEY_ID/activate"
```

SAML service providers that pin the signing certificate must get the new certificate (from the realm's SAML IdP
metadata) before you activate.

### 6. Applications

Register each application of the municipality as an [Application](../manage/applications.md) with an
[OIDC client](../manage/oidc-clients.md) and/or a [SAML relying party](../manage/saml-clients.md), and release only the
claims it needs ([Client scopes & claims](../manage/scopes-and-claims.md)).

### Onboarding checklist

- [ ] Realm created, settings and MFA policy set
- [ ] `admin-<realm>` has a random password and is disabled; named administrators hold the least role they need
- [ ] Theme and email provider set
- [ ] DigiD / eHerkenning test connections working, then production connections after the scheme's acceptance
- [ ] Realm signing key with the municipality's certificate active (if required)
- [ ] Applications registered with minimal claims
- [ ] Realm document committed to git (`GET /admin/realms/{realm}/export`)

## Day-2 operations

### Backups and restore drills

The database holds all durable state ([Backup & disaster recovery](backup.md)). Use your platform's managed
PostgreSQL backups with point-in-time recovery, or run the generic CronJob from the chart (unpacked with `helm pull … --untar`, above):

```bash
kubectl -n identity apply -f helixiam/examples/postgres-backup-cronjob.yaml
kubectl -n identity create job helixiam-pg-backup-manual --from=cronjob/helixiam-pg-backup
```

Copy the dumps off the cluster. Back up the `db-encryption` value separately and securely: the realm signing keys,
identity-provider keys and other secrets in the dump are encrypted with it.

Restore drill (quarterly, and after every major upgrade), into a separate namespace so production is untouched:

1. Create an empty database and restore the latest dump into it:
   `pg_restore --no-owner --dbname helixiam_restore helixiam-<timestamp>.dump`.
2. Create the Secret in the drill namespace with the **same** `db-encryption` value from your vault.
3. Install the chart there with `database.name=helixiam_restore` and drill host names.
4. Check readiness (`/actuator/health/readiness`), a sign-in in one municipality's realm, and that its
   `/realms/<realm>/oauth2/jwks` lists the same key ids as production.
5. Record the time it took (your RTO) and the age of the dump (your RPO), then delete the drill namespace.

After restoring production itself, restart every replica: a replica reloads identity providers only when their
version goes up, and a restore can lower it.

### Key and certificate rotation

| What | How |
|---|---|
| Realm signing key / certificate | Before expiry, attach a renewed certificate to the active key (`PUT /admin/realms/{realm}/keys/{keyId}/certificate`), or stage a new key (CSR or import) and activate it after relying parties picked it up. See [Realm keys](realm-keys.md) |
| DigiD / eHerkenning SP certificates (PKIoverheid) | Update `spSigningCertificate` / `spSigningPrivateKey` / `spDecryptionPrivateKey` of the connection, coordinated with Logius or the broker; the keys are write-only |
| Provisioning service account secret | `POST /admin/realms/master/clients/{id}/secret`, then update the Secret |
| Email/SMS provider credentials | Update the Secret (global email credentials are read at every send) or the realm's provider |
| `DB_ENCRYPTION` | **Never.** It cannot be rotated |

Certificate expiry raises `REALM_KEY_CERTIFICATE_EXPIRING` / `REALM_KEY_CERTIFICATE_EXPIRED` audit events 30 days ahead,
a daily WARN, and shows in the `realmSigningKeys` details of `/actuator/health/readiness`. An expired active
certificate keeps signing, so alert on these events.

### Monitoring

- Probes: `/actuator/health/liveness` and `/actuator/health/readiness` (anonymous); see
  [Observability & health](observability.md).
- Metrics: `/actuator/prometheus`, scraped by the chart's ServiceMonitor.
- Alerts: the chart's PrometheusRule covers scrape failures, unavailable replicas (needs kube-state-metrics), 5xx rate,
  failed-sign-in ratio per realm (`helix_login_total`), email provider authentication failures, the email retry backlog
  and unreadable encrypted columns. The chart README lists them.
- Log-based alerts (the pods log to stdout; audit events are JSON lines of the `helix.audit` logger): realm certificate
  expiry (above); the WARN `Rate limit '<name>': the shared <store> store failed` (a replica counts rate limits on its
  own while PostgreSQL or Redis is unreachable); `PRIVILEGED_LEGACY_LINK` and `PRIVILEGED_AMBIGUOUS_LINK` after the
  1.0.0-rc.12 upgrade; `DEFAULT_ROLE_HAS_ADMIN_PERMISSIONS`; and the start-up WARN `Production readiness — N insecure
  setting(s) in effect`.

### Incident response pointers

- **Who did what:** the realm's audit log (`GET /admin/realms/{realm}/events`, filterable; see
  [Events & audit](../manage/events.md)), and the `helix.audit` lines in your log pipeline.
- **Cut off access:** revoke sessions (`DELETE /admin/realms/{realm}/sessions/{id}`, see [Sessions](../manage/sessions.md)),
  disable the user, rotate a leaked client secret, or disable the affected identity provider.
- **Suspected key compromise:** stage and activate a new realm key, then retire the old one
  (`DELETE /admin/realms/{realm}/keys/{keyId}`, after it is no longer ACTIVE); relying parties that cache the JWKS
  keep trusting the old key until their cache expires.
- **Scheme obligations:** a DigiD incident is also reported to Logius under the connection agreement; personal data
  breaches to the Dutch DPA (Autoriteit Persoonsgegevens) by the municipality as controller.
- **Upgrades with security fixes:** read the security advisories in each release's upgrade notes (1.0.0-rc.12 fixed two
  High privilege escalations).

### Upgrades

1. Read the release's upgrade notes ([Upgrades & migrations](upgrades.md) and the per-release pages). Some upgrades
   cannot mix versions and need `strategy.type=Recreate` for that one upgrade.
2. Back up the database (and check you have the `db-encryption` value).
3. `helm upgrade helixiam oci://ghcr.io/melmarzgioui/helixiam/charts/helixiam --version <new chart version> -n identity --reset-then-reuse-values`. With the profile
   (`maxSurge: 1`, `maxUnavailable: 0`) one new pod starts at a time and old pods keep serving.
4. **Flyway:** the first new pod applies pending migrations at start-up (`HELIX_MIGRATIONS_ENABLED=true`); Flyway's
   lock in the database keeps the other pods from migrating at the same time. Migrations are additive, so old pods
   keep working against the new schema during the roll. `flyway_schema_history` records every applied version.
5. Check readiness, one sign-in per realm with an eID connection, and the start-up reports in the log.

For **1.0.0-rc.11 → 1.0.0-rc.12** ([the guide](upgrade-rc11-rc12.md)):

- Flyway V86 rewrites `realm_key` and places federated links in realms; read the `PRIVILEGED_LEGACY_LINK` report
  afterwards and revoke trust in suspect links.
- **Activate an ES256 realm key only after every replica runs 1.0.0-rc.12 or later**: a 1.0.0-rc.11 replica cannot load an EC key. Importing
  and staging keys during the roll is fine (1.0.0-rc.11 replicas ignore PENDING keys).
- Wallet providers (`trusted_authorities`, from 1.0.0-rc.11): an older replica refuses a DCQL query that carries
  `trusted_authorities`, so a wallet sign-in started on an upgraded pod and answered on an old one fails. Upgrade all
  replicas in one step, drain wallet transactions first (`transactionTtlSeconds`, 300 s by default), or set HAIP-profile
  providers' `dcql.trustedAuthorities` to Off for the roll.
- Admin automation that used `manage-realm` for key writes and other sensitive settings now needs `realm-admin`.

For **1.1.0-rc.1 → 1.1.0-rc.2** ([the guide](upgrade-1.1.0-rc1-rc2.md)):

- The profile now runs the admin console as its own Deployment on the beheer host and sets `config.adminAllowedHosts`:
  remove the beheer host from the server's `ingress.hosts` / `ingress.tls`, and move every script, Terraform
  configuration and CI job that calls `/admin` or gets master-realm tokens to the beheer host.
- Custom scripts that read the CSRF cookie must read `__Host-XSRF-TOKEN`.

## Offboard a municipality

Helix has **no realm-delete operation** in the console or the admin API. Offboarding is therefore an export followed
by removing the realm's data object by object:

1. **Agree the scope** with the municipality: what to hand over, what to delete, which retention periods apply to the
   audit log and backups.
2. **Export** the configuration and hand over personal data:

    ```bash
    curl -s -H "$AUTH" "$HELIX_URL/admin/realms/$REALM/export" > "$REALM-export.json"
    ```

    The export carries the realm's configuration and its users' profiles and role assignments, never passwords, MFA
    secrets, signing keys, sessions or the audit log ([Import / export](../integration/import-export.md)). Per user,
    `GET /admin/realms/{realm}/users/{userId}/gdpr/export` produces the data-subject export
    ([GDPR & privacy](gdpr.md)). Export the audit log from your log pipeline or with
    `GET /admin/realms/{realm}/events`.
3. **Stop sign-ins:** set the realm's `enabled` to `false` (`PUT /admin/realms/{realm}/settings`, or **Realm settings**
   in the console). A disabled realm refuses every sign-in, token issuance and refresh, introspects tokens as inactive
   and serves only discovery, JWKS and metadata; the admin API stays reachable ([Realms](../manage/realms.md#disabling-a-realm)).
   Then delete the realm's applications (cascades to their OIDC clients and SAML relying parties) and its identity
   providers (`DELETE /admin/realms/{realm}/identity-providers/{alias}`), and revoke the remaining sessions.
4. **Erase users:** `DELETE /admin/realms/{realm}/users/{userId}/gdpr` for every user. This also removes their
   DigiD, eHerkenning and wallet links and their (encrypted) BSN.
5. **Retire keys** that are not ACTIVE (`DELETE /admin/realms/{realm}/keys/{keyId}`); the ACTIVE RS256 key cannot be
   retired and stays, encrypted, with the realm's settings row.
6. **What remains** in the database: the realm's settings row, its active key, roles, flows and other configuration
   that has no delete endpoint, and its audit-log rows (removed only by `retention.auditLogMaxAge`, when set). Removing those is
   database work outside the product's supported operations. If an operator deletes the realm's `realm_config` row,
   the Flyway **V85** trigger deletes that realm's wallet pepper, verifier access keys, in-flight wallet transactions and
   trusted list sources in the same transaction, so a realm created later under the same id starts without them
   ([EUDI Wallet: when a realm is deleted](../federation/eudi-wallet.md#when-a-realm-is-deleted)); it removes nothing
   else.
7. **Backups:** the municipality's data stays in older backups until they expire under your backup retention; record
   that date in the offboarding report.
8. **Schemes:** end the DigiD connection with Logius and the eHerkenning contract with the broker.

## See also

- [Realms](../manage/realms.md) · [Realm keys](realm-keys.md) · [Admin roles (RBAC)](admin-roles.md)
- [DigiD](../federation/digid.md) · [eHerkenning](../federation/eherkenning.md)
- [Backup & disaster recovery](backup.md) · [Upgrades & migrations](upgrades.md) · [Observability & health](observability.md)
- [Security hardening](security.md) · [GDPR & privacy](gdpr.md)