Hosting municipalities (operator runbook)

Run one HelixIAM installation for several Dutch municipalities (gemeenten), one realm each: onboarding, day-2 operations, upgrades and offboarding.

What it is

This runbook is for the operator who hosts HelixIAM for municipalities. It uses only standard Kubernetes and Helm, so it applies to any conformant cluster, including VNG Haven clusters. Every municipality gets its own realm: its own users, applications, signing keys, sign-in flows, theme, DigiD and eHerkenning connections, and administrators.

The Helm chart ships the files this runbook uses. After helm registry login ghcr.io with the registry credentials from your licence, unpack it with helm pull oci://ghcr.io/melmarzgioui/helixiam/charts/helixiam --version 1.1.0-rc.14 --untar:

Piece	Where
Hosting profile (values file)	helixiam/examples/values-multi-tenant-municipalities.yaml in the chart
Chart documentation: HA, monitoring and alerts, backups, Haven	helixiam/README.md in the chart
Generic pg_dump backup CronJob	helixiam/examples/postgres-backup-cronjob.yaml in the chart
Readiness pack in Dutch: BIO2 mapping, DPIA input, DigiD/eHerkenning onboarding, processor-agreement input	available to licensed customers on request

Install the platform once

1. Create the Secret. The chart never inlines secrets. The Secret holds the database credentials, the db-encryption key (16, 24 or 32 characters; it encrypts the realm signing keys and every other secret in the database and must never change), the bootstrap admin password and the secret of the provisioning service account:

   kubectl create namespace identity
   kubectl -n identity create secret generic helixiam-secrets \
     --from-literal=db-username=helixiam \
     --from-literal=db-password="$DB_PASSWORD" \
     --from-literal=db-encryption="$(openssl rand -hex 16)" \
     --from-literal=admin-password="$(openssl rand -base64 24)" \
     --from-literal=bootstrap-client-secret="$(openssl rand -hex 32)"
   

   Copy the db-encryption value into your secrets vault now, separately from any database backup. A database backup is useless without it.

2. Adapt the profile. Copy helixiam/examples/values-multi-tenant-municipalities.yaml from the unpacked chart and change every value marked CHANGE: the host names (config.idpBaseUrl, config.spBaseUrl, ingress.hosts, ingress.tls, the console's beheer host in console.ingress and config.adminAllowedHosts), the ingress controller's pod CIDR in config.trustedProxies, database.host, the namespace selectors of the NetworkPolicy, and the labels your Prometheus selects on (serviceMonitor.labels, prometheusRule.labels). Set imagePullSecrets to a docker-registry Secret in the namespace with the registry credentials from your licence. The profile connects to PostgreSQL with database.sslMode: verify-full: create the Secret with the CA that signed the database server certificate first (kubectl -n identity create secret generic postgres-ca --from-file=ca.crt=./postgres-ca.pem, or set database.caSecret.name to yours). Decide the audit-log retention with the municipalities (retention.auditLogMaxAge, empty keeps it).

3. Install:

   helm upgrade --install helixiam oci://ghcr.io/melmarzgioui/helixiam/charts/helixiam --version <chart version> \
     -n identity -f values-municipalities.yaml
   kubectl -n identity rollout status deploy/helixiam
   kubectl -n identity rollout status deploy/helixiam-console
   

What the profile sets, and why:

Concern	Values
High availability	replicaCount: 3 (two at least), strategy RollingUpdate with maxUnavailable: 0, topologySpread.enabled (zones and nodes), podDisruptionBudget.minAvailable: 2
Shared state	config.sessionStore: queue, config.tokenStore: queue, config.clusterStateStore: jdbc: sessions, tokens, flow state and rate limits in PostgreSQL, counted once for the cluster. The chart passes replicaCount to the server as HELIX_CLUSTER_EXPECTED_REPLICAS. With Redis instead, set redis.host and the stores to redis
Proxies and host names	config.trustedProxies (HELIX_TRUSTED_PROXIES), config.allowedHosts (HELIX_ALLOWED_HOSTS): see Security hardening
Pod security	non-root, readOnlyRootFilesystem, all capabilities dropped, RuntimeDefault seccomp, no service-account token, resource requests and limits
Network	networkPolicy.allowSameNamespaceIngress: false, ingress only from the ingress controller's and the monitoring namespace, egress restricted to DNS, PostgreSQL, 443 and SMTP
Monitoring	config.prometheusAnonymous: true with serviceMonitor.enabled and prometheusRule.enabled
Database TLS	database.sslMode: verify-full with database.caSecret (mounted read-only, passed as DB_SSLMODE / DB_SSL_ROOT_CERT): the connection is encrypted and the server certificate and host name are verified. Without it the JDBC driver default (prefer) does not verify the server
Data retention	Expired sessions, tokens, one-time codes and links are deleted on a schedule (retention.enabled, batches, safe with several replicas); the audit log only with retention.auditLogMaxAge, a policy decision
Admin console	console.enabled with 2 replicas, its own Deployment on an internal beheer host (console.ingress, removed from the server Ingress), an ingress-only console.networkPolicy. The server still scales to 3+ on its own. See Architecture
Administration only on the beheer host	config.adminAllowedHosts: [beheer host], config.adminRestrictMasterRealm: true: /admin/**, the API docs and the master realm answer 404 on the public sign-in host; the municipalities' realms are unaffected. See Install
Provisioning	secrets.bootstrapClientId: provisioner: a client_credentials client with the master admin role, so automation never signs in as the bootstrap admin and config.mfaEnabled stays on

Keep /actuator/prometheus off the public ingress

The ServiceMonitor needs anonymous scraping. The NetworkPolicy admits only the monitoring namespace besides the ingress controller, but the chart's Ingress routes every path of its hosts: deny /actuator/ in your ingress controller or gateway.

A token for the admin API

The examples below call the admin API with a bearer token of the provisioning service account, so they need no CSRF header (see Authenticating to the API). The profile serves the admin API and the master realm only on the beheer host (config.adminAllowedHosts), so HELIX_URL is the beheer host here, not the public sign-in host (which answers 404 for both). Run them from a network that reaches the internal beheer ingress, or call an in-cluster name you also listed in config.adminAllowedHosts:

export HELIX_URL=https://beheer.login.example.nl
PROVISIONER_SECRET=$(kubectl -n identity get secret helixiam-secrets \
  -o jsonpath='{.data.bootstrap-client-secret}' | base64 -d)
TOKEN=$(curl -s -u "provisioner:$PROVISIONER_SECRET" -d grant_type=client_credentials \
  "$HELIX_URL/realms/master/oauth2/token" | jq -r .access_token)
AUTH="Authorization: Bearer $TOKEN"

The token is valid for the master realm's access-token lifetime (one hour by default); fetch a new one when it expires.

Onboard a municipality

The HelixIAM release tests run steps 1 to 3, 5 and the export of the offboarding section against a two-replica install, with every request of a sign-in alternating between the pods.

1. Create the realm

Pick the realm id first: it is part of every URL, every token's iss and the SAML entity ids (https://login.example.nl/realms/<realm>/…), and DigiD and eHerkenning are registered against those URLs. Use a stable, URL-safe name such as gemeente-voorbeeld.

Admin APIRealm config folder

Saving the realm's settings creates it:

REALM=gemeente-voorbeeld
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"displayName":"Gemeente Voorbeeld","accessTokenTtlSeconds":300,"refreshTokenTtlSeconds":86400,
       "enabled":true,"passwordMinLength":12,"ssoSessionIdleTimeoutSeconds":1800,
       "ssoSessionMaxLifetimeSeconds":36000}' \
  "$HELIX_URL/admin/realms/$REALM/settings"

Keep one realm document per municipality in git (the format of GET /admin/realms/{realm}/export, see Import / export); the file name is the realm id. Mount them with the chart's realmConfig values; every replica applies them at start-up:

kubectl -n identity create configmap helixiam-realms \
  --from-file=gemeente-voorbeeld.json --dry-run=client -o yaml | kubectl apply -f -
helm upgrade helixiam oci://ghcr.io/melmarzgioui/helixiam/charts/helixiam --version <chart version> \
  -n identity --reset-then-reuse-values \
  --set realmConfig.enabled=true --set realmConfig.configMap.name=helixiam-realms

With realmConfig.onConflict=skip (the default) a restart never changes what already exists; overwrite reconciles the realm to the files. Secrets in the files are ${HELIX_<REALM>_…} placeholders: provide them with config.extraEnv from a Secret, and set realmConfig.requireEnv=true so a file whose secret is missing is skipped instead of half-applied. A minimal file:

{"formatVersion": 2, "realm": {"displayName": "Gemeente Voorbeeld", "accessTokenTtlSeconds": 300,
 "refreshTokenTtlSeconds": 86400, "enabled": true, "passwordMinLength": 12}}

Up to and including 1.0.0-rc.12, a skip-mode import does not create a realm that does not exist yet (fixed in 1.0.0-rc.13). On those older releases create the realm with the admin API first, or use realmConfig.onConflict=overwrite.

Set the realm's sign-in policy (MFA, password policy, lockout) in the same settings, see Realm settings and Password policy & lockout.

2. Administrators

Creating a realm also creates its bootstrap admin, admin-<realm>, with the bootstrap admin password of the installation (the same for every realm). Replace that password and disable the account before anyone else can use it, then give the municipality named administrators:

BOOT_ID=$(curl -s -H "$AUTH" "$HELIX_URL/admin/realms/$REALM/users" \
  | jq -r --arg u "admin-$REALM" '.[] | select(.username == $u) | .userId')
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" \
  -d "{\"newPassword\":\"$(openssl rand -base64 32)\"}" \
  "$HELIX_URL/admin/realms/$REALM/users/$BOOT_ID/password"
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" \
  -d "{\"username\":\"admin-$REALM\",\"enabled\":false}" \
  "$HELIX_URL/admin/realms/$REALM/users/$BOOT_ID"

# A named administrator with a one-time password they must change at first sign-in (TOTP enrolment follows
# with config.mfaEnabled=true)
ADMIN_ID=$(curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"username":"j.jansen","email":"[email protected]","enabled":true}' \
  "$HELIX_URL/admin/realms/$REALM/users" | jq -r .userId)
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" \
  -d "{\"newPassword\":\"$ONE_TIME_PASSWORD\"}" "$HELIX_URL/admin/realms/$REALM/users/$ADMIN_ID/password"
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" -d '{"requiredActions":"UPDATE_PASSWORD"}' \
  "$HELIX_URL/admin/realms/$REALM/users/$ADMIN_ID/required-actions"
ROLE_ID=$(curl -s -H "$AUTH" "$HELIX_URL/admin/realms/$REALM/roles" | jq -r '.[] | select(.name == "admin") | .roleId')
curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" -d "{\"roleId\":\"$ROLE_ID\"}" \
  "$HELIX_URL/admin/realms/$REALM/users/$ADMIN_ID/roles"

Hand the one-time password over out of band.

The bootstrap admin is recreated only when it is missing, so keep it disabled rather than deleting it. The realm role admin holds realm-admin; give help-desk and audit staff narrower roles (Admin roles). A municipality's administrators manage only their own realm; keep the master realm to the hosting team.

3. Branding

A realm theme (colours, logo, texts, fonts) is set with PUT /admin/realms/{realm}/theme; Theming describes the model. Shared, file-based themes can be mounted with the chart's themes values and selected per realm with PUT /admin/realms/{realm}/theme/base.

curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"colors":{"primary":{"light":"#1d4ed8"}},"assets":{"logoUrl":"https://www.voorbeeld.nl/logo.svg"}}' \
  "$HELIX_URL/admin/realms/$REALM/theme"

Also set the realm's email provider and templates, so codes and notices come from the municipality (Notifications).

4. DigiD and eHerkenning

Each municipality brings its own DigiD connection (Logius) and eHerkenning contract (a broker), and its own PKIoverheid certificates. Create the connections in the municipality's realm as described in DigiD and eHerkenning:

• Use separate connections for the scheme's test and production environments, for example aliases digid-preprod and digid, each with its own metadata and certificates.
• The Assertion Consumer Service URL to register is https://login.example.nl/realms/<realm>/broker/<alias>/callback.
• Set minimumLoa to the level each service needs; it is requested and enforced.
• The artifact binding resolves over mutual TLS on port 443 (allowed by the profile's egress policy), with the SP signing certificate as the client certificate. The Logius server certificate must chain to the image's default Java trust store; there is no per-connection trust store.
• Decide which applications may receive the BSN. A DigiD user's username is the BSN unless the connection has a username template (Mappers), and preferred_username carries the username.

The Dutch onboarding guide in the readiness pack (available to licensed customers) maps every Logius and eHerkenning requirement to these settings.

5. The municipality's own signing certificate

Each realm signs tokens and SAML assertions with its own key. To use a certificate from the municipality's CA, let Helix generate the key and a CSR (the private key never leaves the server), have it signed, attach the chain, and activate the key once relying parties have picked it up (Realm keys):

curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"algorithm":"RS256","subject":"CN=login.example.nl, O=Gemeente Voorbeeld, C=NL","dnsNames":["login.example.nl"]}' \
  "$HELIX_URL/admin/realms/$REALM/keys/csr" | tee csr-response.json | jq -r .csrPem > realm.csr
KEY_ID=$(jq -r .key.keyId csr-response.json)

# After the CA returns the certificate (leaf first, then the issuing CA certificates):
jq -n --rawfile chain signed-chain.pem '{certificateChainPem: $chain}' |
curl -s -X PUT -H "$AUTH" -H "Content-Type: application/json" --data-binary @- \
  "$HELIX_URL/admin/realms/$REALM/keys/$KEY_ID/certificate"

# The key is PENDING and already in the JWKS; activate it after the staging time (5 minutes by default)
curl -s -X POST -H "$AUTH" "$HELIX_URL/admin/realms/$REALM/keys/$KEY_ID/activate"

SAML service providers that pin the signing certificate must get the new certificate (from the realm's SAML IdP metadata) before you activate.

6. Applications

Register each application of the municipality as an Application with an OIDC client and/or a SAML relying party, and release only the claims it needs (Client scopes & claims).

Onboarding checklist

• [ ] Realm created, settings and MFA policy set
• [ ] admin-<realm> has a random password and is disabled; named administrators hold the least role they need
• [ ] Theme and email provider set
• [ ] DigiD / eHerkenning test connections working, then production connections after the scheme's acceptance
• [ ] Realm signing key with the municipality's certificate active (if required)
• [ ] Applications registered with minimal claims
• [ ] Realm document committed to git (GET /admin/realms/{realm}/export)

Day-2 operations

Backups and restore drills

The database holds all durable state (Backup & disaster recovery). Use your platform's managed PostgreSQL backups with point-in-time recovery, or run the generic CronJob from the chart (unpacked with helm pull … --untar, above):

kubectl -n identity apply -f helixiam/examples/postgres-backup-cronjob.yaml
kubectl -n identity create job helixiam-pg-backup-manual --from=cronjob/helixiam-pg-backup

Copy the dumps off the cluster. Back up the db-encryption value separately and securely: the realm signing keys, identity-provider keys and other secrets in the dump are encrypted with it.

Restore drill (quarterly, and after every major upgrade), into a separate namespace so production is untouched:

1. Create an empty database and restore the latest dump into it: pg_restore --no-owner --dbname helixiam_restore helixiam-<timestamp>.dump.
2. Create the Secret in the drill namespace with the same db-encryption value from your vault.
3. Install the chart there with database.name=helixiam_restore and drill host names.
4. Check readiness (/actuator/health/readiness), a sign-in in one municipality's realm, and that its /realms/<realm>/oauth2/jwks lists the same key ids as production.
5. Record the time it took (your RTO) and the age of the dump (your RPO), then delete the drill namespace.

After restoring production itself, restart every replica: a replica reloads identity providers only when their version goes up, and a restore can lower it.

Key and certificate rotation

What	How
Realm signing key / certificate	Before expiry, attach a renewed certificate to the active key (PUT /admin/realms/{realm}/keys/{keyId}/certificate), or stage a new key (CSR or import) and activate it after relying parties picked it up. See Realm keys
DigiD / eHerkenning SP certificates (PKIoverheid)	Update spSigningCertificate / spSigningPrivateKey / spDecryptionPrivateKey of the connection, coordinated with Logius or the broker; the keys are write-only
Provisioning service account secret	POST /admin/realms/master/clients/{id}/secret, then update the Secret
Email/SMS provider credentials	Update the Secret (global email credentials are read at every send) or the realm's provider
DB_ENCRYPTION	Never. It cannot be rotated

Certificate expiry raises REALM_KEY_CERTIFICATE_EXPIRING / REALM_KEY_CERTIFICATE_EXPIRED audit events 30 days ahead, a daily WARN, and shows in the realmSigningKeys details of /actuator/health/readiness. An expired active certificate keeps signing, so alert on these events.

Monitoring

• Probes: /actuator/health/liveness and /actuator/health/readiness (anonymous); see Observability & health.
• Metrics: /actuator/prometheus, scraped by the chart's ServiceMonitor.
• Alerts: the chart's PrometheusRule covers scrape failures, unavailable replicas (needs kube-state-metrics), 5xx rate, failed-sign-in ratio per realm (helix_login_total), email provider authentication failures, the email retry backlog and unreadable encrypted columns. The chart README lists them.
• Log-based alerts (the pods log to stdout; audit events are JSON lines of the helix.audit logger): realm certificate expiry (above); the WARN Rate limit '<name>': the shared <store> store failed (a replica counts rate limits on its own while PostgreSQL or Redis is unreachable); PRIVILEGED_LEGACY_LINK and PRIVILEGED_AMBIGUOUS_LINK after the 1.0.0-rc.12 upgrade; DEFAULT_ROLE_HAS_ADMIN_PERMISSIONS; and the start-up WARN Production readiness — N insecure setting(s) in effect.

Incident response pointers

• Who did what: the realm's audit log (GET /admin/realms/{realm}/events, filterable; see Events & audit), and the helix.audit lines in your log pipeline.
• Cut off access: revoke sessions (DELETE /admin/realms/{realm}/sessions/{id}, see Sessions), disable the user, rotate a leaked client secret, or disable the affected identity provider.
• Suspected key compromise: stage and activate a new realm key, then retire the old one (DELETE /admin/realms/{realm}/keys/{keyId}, after it is no longer ACTIVE); relying parties that cache the JWKS keep trusting the old key until their cache expires.
• Scheme obligations: a DigiD incident is also reported to Logius under the connection agreement; personal data breaches to the Dutch DPA (Autoriteit Persoonsgegevens) by the municipality as controller.
• Upgrades with security fixes: read the security advisories in each release's upgrade notes (1.0.0-rc.12 fixed two High privilege escalations).

Upgrades

1. Read the release's upgrade notes (Upgrades & migrations and the per-release pages). Some upgrades cannot mix versions and need strategy.type=Recreate for that one upgrade.
2. Back up the database (and check you have the db-encryption value).
3. helm upgrade helixiam oci://ghcr.io/melmarzgioui/helixiam/charts/helixiam --version <new chart version> -n identity --reset-then-reuse-values. With the profile (maxSurge: 1, maxUnavailable: 0) one new pod starts at a time and old pods keep serving.
4. Flyway: the first new pod applies pending migrations at start-up (HELIX_MIGRATIONS_ENABLED=true); Flyway's lock in the database keeps the other pods from migrating at the same time. Migrations are additive, so old pods keep working against the new schema during the roll. flyway_schema_history records every applied version.
5. Check readiness, one sign-in per realm with an eID connection, and the start-up reports in the log.

For 1.0.0-rc.11 → 1.0.0-rc.12 (the guide):

• Flyway V86 rewrites realm_key and places federated links in realms; read the PRIVILEGED_LEGACY_LINK report afterwards and revoke trust in suspect links.
• Activate an ES256 realm key only after every replica runs 1.0.0-rc.12 or later: a 1.0.0-rc.11 replica cannot load an EC key. Importing and staging keys during the roll is fine (1.0.0-rc.11 replicas ignore PENDING keys).
• Wallet providers (trusted_authorities, from 1.0.0-rc.11): an older replica refuses a DCQL query that carries trusted_authorities, so a wallet sign-in started on an upgraded pod and answered on an old one fails. Upgrade all replicas in one step, drain wallet transactions first (transactionTtlSeconds, 300 s by default), or set HAIP-profile providers' dcql.trustedAuthorities to Off for the roll.
• Admin automation that used manage-realm for key writes and other sensitive settings now needs realm-admin.

For 1.1.0-rc.1 → 1.1.0-rc.2 (the guide):

• The profile now runs the admin console as its own Deployment on the beheer host and sets config.adminAllowedHosts: remove the beheer host from the server's ingress.hosts / ingress.tls, and move every script, Terraform configuration and CI job that calls /admin or gets master-realm tokens to the beheer host.
• Custom scripts that read the CSRF cookie must read __Host-XSRF-TOKEN.

Offboard a municipality

Helix has no realm-delete operation in the console or the admin API. Offboarding is therefore an export followed by removing the realm's data object by object:

1. Agree the scope with the municipality: what to hand over, what to delete, which retention periods apply to the audit log and backups.

2. Export the configuration and hand over personal data:

   curl -s -H "$AUTH" "$HELIX_URL/admin/realms/$REALM/export" > "$REALM-export.json"
   

   The export carries the realm's configuration and its users' profiles and role assignments, never passwords, MFA secrets, signing keys, sessions or the audit log (Import / export). Per user, GET /admin/realms/{realm}/users/{userId}/gdpr/export produces the data-subject export (GDPR & privacy). Export the audit log from your log pipeline or with GET /admin/realms/{realm}/events. 3. Stop sign-ins: set the realm's enabled to false (PUT /admin/realms/{realm}/settings, or Realm settings in the console). A disabled realm refuses every sign-in, token issuance and refresh, introspects tokens as inactive and serves only discovery, JWKS and metadata; the admin API stays reachable (Realms). Then delete the realm's applications (cascades to their OIDC clients and SAML relying parties) and its identity providers (DELETE /admin/realms/{realm}/identity-providers/{alias}), and revoke the remaining sessions. 4. Erase users: DELETE /admin/realms/{realm}/users/{userId}/gdpr for every user. This also removes their DigiD, eHerkenning and wallet links and their (encrypted) BSN. 5. Retire keys that are not ACTIVE (DELETE /admin/realms/{realm}/keys/{keyId}); the ACTIVE RS256 key cannot be retired and stays, encrypted, with the realm's settings row. 6. What remains in the database: the realm's settings row, its active key, roles, flows and other configuration that has no delete endpoint, and its audit-log rows (removed only by retention.auditLogMaxAge, when set). Removing those is database work outside the product's supported operations. If an operator deletes the realm's realm_config row, the Flyway V85 trigger deletes that realm's wallet pepper, verifier access keys, in-flight wallet transactions and trusted list sources in the same transaction, so a realm created later under the same id starts without them (EUDI Wallet: when a realm is deleted); it removes nothing else. 7. Backups: the municipality's data stays in older backups until they expire under your backup retention; record that date in the offboarding report. 8. Schemes: end the DigiD connection with Logius and the eHerkenning contract with the broker.

See also

• Realms · Realm keys · Admin roles (RBAC)
• DigiD · eHerkenning
• Backup & disaster recovery · Upgrades & migrations · Observability & health
• Security hardening · GDPR & privacy