# Key custody (HSM and KMS) Keep a realm's private signing keys out of the database: in a hardware security module (PKCS#11) or in a Vault / OpenBao Transit engine you run yourself. You choose per realm how far the keys are kept away from HelixIAM, and every request that needs an unavailable backend fails closed. The same applies to the credential issuer keys, the development CA, the eID SP keys (signing, mutual TLS and decryption) and signed SAML IdP metadata. !!! info "Added in 1.1.0-rc.11, extended in 1.1.0-rc.12 and 1.1.0-rc.13" Realm token, SAML and wallet verifier keys came with 1.1.0-rc.11; the credential issuer keys, the development CA, the eID SP keys and signed SAML metadata are added in 1.1.0-rc.12. 1.1.0-rc.13 adds a KMIP backend, a plugin interface with plugins for AWS KMS, Azure Key Vault / Managed HSM and Google Cloud KMS (Preview), a [sovereign key-store policy](#sovereign-key-stores-only) per realm and [online rotation of the database encryption key](#rotating-the-database-encryption-key). Key custody is opt-in. Nothing changes on upgrade: every realm, new or upgraded, starts in the **Standard** profile, which is how HelixIAM has always worked (software keys in the database, encrypted with `DB_ENCRYPTION`). ## Security levels (profiles) Each realm has one custody profile. A master-realm administrator chooses it per realm. | Profile | Where the private keys are | Signing | What an attacker needs | | --- | --- | --- | --- | | **Standard** (default) | Generated in the server, stored in the database encrypted with `DB_ENCRYPTION` | In the server | The database and the `DB_ENCRYPTION` key | | **Protected** | Generated in the server, stored in the database **only wrapped** (AES-256-GCM) by the realm's key-encryption key (KEK) in the backend. Each pod unwraps them into memory. | In the server, as fast as Standard | The database, plus the right to call the backend's decrypt with the realm's KEK | | **High assurance** | **Generated inside the backend; they never leave it.** The database holds only a reference. | Every signature is a backend call | The right to call the backend's sign with the realm's keys | High assurance covers every realm-key purpose (`access`, `id`, `saml`): none of these keys exists outside the backend, and importing a software realm key is refused. The same holds for wallet verifier keys: they are generated in the backend with **Generate in HSM/KMS**, importing a software verifier key into a High-assurance realm is refused (400 on `privateKeyPem`), and a software verifier key stored before the switch no longer signs (see [Wallet verifier keys](#wallet-verifier-keys)). A High-assurance realm's token throughput is the backend's signing capacity (see [Sizing](#sizing)). The issuer, eID and metadata keys follow the same three profiles. See [High-assurance rules for the issuer, eID and metadata keys](#high-assurance-rules-for-the-issuer-eid-and-metadata-keys) for what each profile allows. ### Keys per purpose A realm in Protected or High assurance gets its own keys per purpose, one ACTIVE key per algorithm family (RSA, EC, Ed25519) and purpose, all in the realm's custody: | Purpose | Signs | | --- | --- | | `access` | Access tokens, RFC 8693 / delegation tokens, workload-identity tokens | | `id` | ID tokens, logout tokens, JARM responses | | `saml` | SAML assertions | | `verifier` | Wallet (OpenID4VP) request objects; separate verifier keys, see [Wallet verifier keys](#wallet-verifier-keys) | | `issuer` | Credentials and status lists of the OpenID4VCI issuer (ES256 only); see [Credential issuer keys and the development CA](#credential-issuer-keys-and-the-development-ca) | | `issuerca` | The realm's development CA, which certifies generated test issuer keys (ES256 only) | | `federation` | The SP keys of eID connections: AuthnRequest and ArtifactResolve signatures, mutual TLS and assertion decryption; see [eID SP keys](#eid-sp-keys) (RSA only) | `issuer`, `issuerca` and `federation` are not realm-key purposes: they belong to their own screens and endpoints, the realm-key CSR (`POST /admin/realms/{realmId}/keys/csr`) refuses them, and they are not in the realm's JWKS. The JWKS publishes every purpose's keys; the token type picks the key. There is no fallback to another purpose, to a software key or to RS256: a key that cannot sign fails the request (see [Failure behaviour](#failure-behaviour)). In High assurance a client's `id_token_signed_response_alg` must be an algorithm the backend signs; saving a client with another one is refused (`fieldErrors.idTokenSignatureAlg`). Standard realms keep their single key set (purpose `all`) and their current behaviour, including the RS256 fallback with its warning. See [Realm keys](realm-keys.md). ## Backends A backend is configured by the operator, never in the console: it involves secrets and infrastructure. The types are `software` (always present), `pkcs11`, `vault`, `kmip` and the types of the [plugins](#plugins) you install (`aws-kms`, `azure-kv`, `gcp-kms` from HelixIAM). The server reads the backends from `helix.custody.backends[n]` (`id`, `type`, `settings.*`, and optionally `realms`: the realms it serves, see [Dedicated key stores](#dedicated-key-stores)). The `software` backend always exists and needs no entry. Settings with secrets (PIN, token, secret id, CA bundle) are **file paths**: the secret itself is read from the file at each login and never stored, logged, audited or shown in the console. ### With the Helm chart List the backends under `custody.backends`. The chart renders them into a ConfigMap (`custody.yaml`, mounted at `/etc/helix/custody/` and loaded through `SPRING_CONFIG_ADDITIONAL_LOCATION`), mounts each backend's Secrets as files under `/var/run/helix/custody//`, and rolls the pods when a custody setting changes (the `checksum/custody` annotation). Allow the backend's port in `networkPolicy.extraEgressPorts` (8200 for Vault, 5696 for KMIP, 443 for a cloud key service, your HSM's port for a network HSM). A vendor PKCS#11 module is not in the image: mount it with `extraVolumes` / `extraVolumeMounts`. See [Install: key custody values](../getting-started/install.md#key-custody-hsm-and-kms). ### Without the chart Put the same structure in a YAML file and point `SPRING_CONFIG_ADDITIONAL_LOCATION` at it. Use a file, not environment variables: Spring lower-cases environment-variable names, and the setting keys are camelCase (`tokenLabel`, `pinFile`, `tokenFile`). ```yaml helix: custody: backends: - id: hsm-1 # appears in key references; letters, digits, . _ - type: pkcs11 settings: { ... } # below - id: vault-1 type: vault settings: { ... } realms: [tenant-a] # optional: only these realms; absent, every realm (shared) ``` Unknown settings are refused at start-up (the message names the setting, never its value). An algorithm a backend cannot sign is refused when a realm is moved to it, never at signing time. ### PKCS#11 (HSM) | Setting | Meaning | | --- | --- | | `library` | Path of the vendor's PKCS#11 module inside the container. With the chart, mount it with `extraVolumes` / `extraVolumeMounts`. | | `tokenLabel` | The token (partition) label. Refused when no token or more than one token has it. | | `pinFile` or `pinEnv` | The file (chart: `pinSecret`, mounted at `/var/run/helix/custody//pin`), or the name of an environment variable, holding the user PIN. Exactly one. | | `sessions` | Session pool size, default 8 (1–256). Also the number of signing threads per pod for this backend. | | `opTimeout` | ISO-8601 time bound per call, default `PT2S`, at most `PT1M`. | | `connectTimeout` | ISO-8601 time that start-up, or a call, waits for a connect (module, token, login), default `PT5S`, at most `PT1M`. A slower connect goes on in the background, the backend disconnected meanwhile. | | `algorithms` | Optional: restrict to these JWS algorithms. | | `keyTransport` | Optional: the RSA-OAEP key transports an eID decryption key may use, comma-separated `METHOD;DIGEST;MGF` values such as `RSA_OAEP_MGF1P;SHA1;SHA1,RSA_OAEP;SHA256;SHA256` (chart: a list). Unset, it is every transport whose hash mechanisms the token lists and that passes the backend's decryption self-test on each connect (a known-good ciphertext must decrypt); a configured transport that fails it is refused at start-up. An unknown value is refused at start-up. See [OAEP key transport on each backend](#oaep-key-transport-on-each-backend). | What HelixIAM does on the token: - Signing keys are created with `CKA_TOKEN`, `CKA_PRIVATE`, `CKA_SENSITIVE` = true and `CKA_EXTRACTABLE` = false (RSA also `CKA_DECRYPT` / `CKA_UNWRAP` = false, EC `CKA_DERIVE` = false), labelled `helix:::`. The one exception is an eID SP key made for decryption (purpose `FEDERATION`, usage signing and decryption): it has `CKA_DECRYPT` = true, is still sensitive and not extractable, and the read-back checks `CKA_DECRYPT` for exactly the usage that was asked. Elliptic-curve keys get `CKA_DECRYPT` = false explicitly. The attributes are read back after generation, and checked again at first use and by the health check for the keys in use (up to 16 per check, round robin): a token object, sensitive, not extractable, always sensitive, never extractable, `CKA_LOCAL`. A key that fails (it can leave the token, or it was imported, for example an object replaced out of band under the same label) is **quarantined** like a Vault key made exportable: never used, `503` for every request that needs it, health DOWN naming its label. See [A key that can leave the backend](#a-key-that-can-leave-the-backend). - The realm KEK is an AES-256 token key (sensitive, not extractable) used with `CKM_AES_GCM`. - Algorithms come from `C_GetMechanismList`: RS256, PS256, ES256, ES384, and EdDSA **only when the token lists `CKM_EDDSA`**. Realm keys are RSA 2048. eID SP keys are RSA 2048, 3072 or 4096, limited to the sizes the token lists for `CKM_RSA_PKCS_KEY_PAIR_GEN`. A token that lists no range makes no RSA size available for them (realm keys are not affected). - One login per process; on `CKR_USER_NOT_LOGGED_IN` or a lost session it logs in again once and retries once, then fails closed. `C_Finalize` is never called. - If the HSM is unreachable at start-up, the backend starts disconnected and reports DOWN; it connects on use. The JVM side: - The JVM needs `--enable-native-access=ALL-UNNAMED`. The image's entry point sets it; add it yourself when you run the jar directly. - **The binding's JNI library (`libpkcs11wrapper.so`) ships in the image** under `/opt/helix/native/`, read-only, for the image's own architecture (linux/amd64 in 1.1.0-rc.12; arm64 images are planned to return), and the entry point puts that directory on `-Djava.library.path`. Nothing is unpacked into or run from `/tmp`, so a `noexec` `/tmp` (a Docker or Compose `tmpfs`, a pod with `readOnlyRootFilesystem` and an `emptyDir`) works. The JVM reads `-Djava.library.path` only at start, so it can be changed only through `JAVA_OPTS` (a later `-Djava.library.path` there replaces the image's: keep `/opt/helix/native` on it), not through a Spring setting or environment variable of the server. - **Running the jar yourself:** the binding finds the library on `-Djava.library.path` first. Without it there, the binding unpacks its own copy into `java.io.tmpdir` and loads it from there, which needs a temporary directory that allows execution. If neither works, the backend starts disconnected with "The PKCS#11 wrapper library (libpkcs11wrapper.so) could not be loaded", the path it searched and the cause, and every call is unavailable. #### Configuring the module (SoftHSM and other HSMs) A PKCS#11 module is not in the image, and most modules read a configuration file of their own: HelixIAM only loads `library` and logs in to `tokenLabel`. Mount the module, its configuration and anything else it needs with `extraVolumes` / `extraVolumeMounts`, and set the environment variable that names its configuration through `config.extraEnv`. The server runs as uid/gid 10001 with a read-only root file system, so whatever the module writes must be a mounted volume that uid 10001 can write. **SoftHSM (tests and demos).** SoftHSM finds its configuration through `SOFTHSM2_CONF` and keeps its tokens as files in `directories.tokendir`. A recipe with a node directory `/srv/softhsm` (a `hostPath`; a PersistentVolumeClaim works the same way): ``` /srv/softhsm/lib/libsofthsm2.so # built for the image's OS (Ubuntu 24.04) and the node's architecture /srv/softhsm/softhsm2.conf # directories.tokendir = /opt/softhsm/tokens # objectstore.backend = file /srv/softhsm/tokens/ # owned by 10001:10001 (SoftHSM writes its objects here) ``` Initialise the token once with the same configuration, for example `SOFTHSM2_CONF=/srv/softhsm/softhsm2.conf softhsm2-util --init-token --free --label helix --so-pin --pin ` (with `tokendir` pointing at `/srv/softhsm/tokens` for that run, or run it in a container that mounts the directory at `/opt/softhsm`), then `chown -R 10001:10001 /srv/softhsm/tokens`. A `hostPath` volume is not changed by the pod's `fsGroup`, so the ownership must be right on the node. Then: ```yaml replicaCount: 1 # SoftHSM's file token is one node's state: one replica is recommended config: extraEnv: - name: SOFTHSM2_CONF value: /opt/softhsm/softhsm2.conf custody: backends: - id: hsm-1 type: pkcs11 library: /opt/softhsm/lib/libsofthsm2.so tokenLabel: helix pinSecret: {name: helix-hsm-pin, key: pin} # kubectl create secret generic helix-hsm-pin --from-literal=pin= extraVolumes: - name: softhsm hostPath: {path: /srv/softhsm, type: Directory} extraVolumeMounts: - name: softhsm mountPath: /opt/softhsm ``` Run **one replica** with SoftHSM. Its file store is not designed for several processes writing at once, and two pods with their own copies of the token directory (on different nodes) hold different keys. Several replicas on one node sharing the same `hostPath` token directory worked in testing for signing and rotation, but two pods creating keys at the same time is unsupported. For more than one replica, or more than one node, use a real network HSM. `GET /admin/key-custody/backends` should then list `hsm-1` as healthy. **A production HSM** (network or PCIe) usually needs its client module, a client configuration file (server address, slot or partition) and client credentials (a certificate, a key, an enrolment file), each named by a path or an environment variable the vendor documents. Mount the credentials from a Secret, set the variables through `config.extraEnv`, and allow the HSM's port in `networkPolicy.extraEgressPorts`. A network HSM is shared by every pod, so the replica count is not limited, but check the vendor's session limit against `sessions` × replicas. !!! warning "Tested with SoftHSM only" PKCS#11 is tested with SoftHSM 2.6.1 (Ubuntu 24.04) and 2.7.0 only; both decrypt RSA-OAEP with SHA-1/MGF1-SHA-1 only (2.6.1 answers `CKR_GENERAL_ERROR` for a ciphertext that does not decrypt, which HelixIAM treats as a refusal while the key can still be read). No network HSM (Luna, nShield, Utimaco, YubiHSM) has been run against HelixIAM yet: vendor template rules, FIPS-mode GCM IV rules, fail-over and session limits are untested. Run the checks in [Compliance](#compliance) on your own HSM before you rely on it. ### Vault and OpenBao Transit `type: vault` works with HashiCorp Vault and with OpenBao (same API). Tested against Vault 2.2.1 and OpenBao 2.7.2 dev servers. | Setting | Meaning | | --- | --- | | `address` | `https://host[:port]`. Plain `http://` only to a loopback host, or with `insecureHttp: "true"` (for a mesh-encrypted link; a warning is logged at start-up). | | `mount` | The Transit mount, default `transit`. | | `namespace` | Optional (Vault Enterprise namespaces). | | `auth.kind` | `approle`, `token` or `kubernetes`. | | `auth.role` | AppRole: the role id. Kubernetes: the role name. | | `auth.secretIdFile` | AppRole: the file with the secret id (chart: `auth.secretIdSecret`, mounted at `.../secret-id`). | | `auth.tokenFile` | Token: the file with the token (chart: `auth.tokenSecret`, `.../token`). Kubernetes: the service-account JWT, default the pod's projected token. | | `auth.path` | The auth mount, default `approle` / `kubernetes`. | | `caFile` | PEM bundle of the CA certificates to trust (chart: `caSecret`, `.../ca.crt`). | | `opTimeout` | Connect and read timeout per call, default `PT2S`, at most `PT60S`. | - Every login reads its file again, so a rotated token, a fresh **single-use AppRole secret id** or a refreshed projected service-account token is picked up without a restart. The token is renewed in the background; login tokens are revoked on shutdown (a configured `token` is never revoked). - Keys are named `helix---` and, for key-encryption keys, `helixkek--` (earlier releases: `helix--kek-`, still accepted), created with `exportable=false` and `allow_plaintext_backup=false`, and read back. Every signature names its `key_version`. An eID SP key that decrypts is named `helixdec--federation-` (prefix `helixdec-`, purpose `federation` only), so a policy can let the backend decrypt with these keys and with no signing key. - Algorithms: RS256, PS256, ES256, ES384, EdDSA. Realm keys are `rsa-2048`; eID SP keys are `rsa-2048`, `rsa-3072` or `rsa-4096`. - The KEK is a non-derived `aes256-gcm96` key; the realm and key id are bound as GCM associated data (`associated_data`). - AppRole and token login are tested against Vault and OpenBao. Kubernetes login is parsed and configured but has not been run against a real cluster yet. It needs the service-account token in the pod (`automountServiceAccountToken: true` or a projected token volume) and a role bound to the pod's service account. #### Minimal policy HelixIAM needs exactly these paths. A trailing `*` is a prefix match; `+` matches a whole path segment only, so a pattern such as `helix-+-kek-+` matches no key name and is not usable. ```hcl # HelixIAM key custody: signing keys helix-*, decryption keys helixdec-*, key-encryption keys helixkek-*; # never export or backup. A trailing * is a prefix match: it also covers keys//rotate and keys//config. # denied_parameters: this token can never make a key exportable or allow a plaintext backup (on keys//config, # where neither can be undone, nor at creation; HelixIAM never sends them, the defaults are false). path "transit/keys/helix-*" { capabilities = ["create", "read", "update", "delete"] denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] } } path "transit/keys/helixdec-*" { capabilities = ["create", "read", "update", "delete"] denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] } } path "transit/keys/helixkek-*" { capabilities = ["create", "read", "update", "delete"] denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] } } path "transit/sign/helix-*" { capabilities = ["update"] } path "transit/sign/helixdec-*" { capabilities = ["update"] } # Encrypt and decrypt: the realms' KEKs and the eID decryption keys, never a signing key. path "transit/encrypt/helixkek-*" { capabilities = ["update"] } path "transit/decrypt/helixkek-*" { capabilities = ["update"] } path "transit/decrypt/helixdec-*" { capabilities = ["update"] } # DB_ENCRYPTION_KEK (made by the operator's DbKeyWrapTool, policy below): the server only reads it (its flags) and # unwraps the data key at start-up; it never wraps with it, disables or destroys it path "transit/keys/helixdbkek-*" { capabilities = ["read"] } path "transit/decrypt/helixdbkek-*" { capabilities = ["update"] } ``` The operator's `DbKeyWrapTool` (for `DB_ENCRYPTION_KEK`, see [`DB_ENCRYPTION` under a KEK](#db_encryption-under-a-kek-envelope)) runs with credentials of its own and this policy: it creates, reads, rotates and disables `helixdbkek-` keys and wraps and unwraps with them, and can never make one exportable or allow a plaintext backup: ```hcl path "transit/keys/helixdbkek-*" { capabilities = ["create", "read", "update"] denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] } } path "transit/encrypt/helixdbkek-*" { capabilities = ["update"] } path "transit/decrypt/helixdbkek-*" { capabilities = ["update"] } ``` The contract tests run the server with the minimal policy above (it reads and unwraps a `helixdbkek-` key, and is refused `403` when it would wrap with it or reconfigure it) and the tool with this one (create, wrap, disable; `403` for `exportable` and `allow_plaintext_backup`), on Vault 2.2.1 and OpenBao 2.7.2. - **Do not grant `transit/decrypt/helix-*` or `transit/encrypt/helix-*`.** They would let the backend itself decrypt (or encrypt) with the RSA signing keys. HelixIAM never asks it to, and refuses before any call to decrypt with a key that is not a `helixdec-` key, but the policy should not allow what the server never needs. The contract tests ran against Vault 2.2.1 and OpenBao 2.7.2 with the policy above and show that a decrypt with an RSA signing key is `403` under it (also for a realm whose id ends in `-kek`) and `200` with the root token. - **KEKs with the earlier name.** A KEK made before this release (`helix--kek-`: the realm KEKs of Protected and High-assurance realms, and a `DB_ENCRYPTION_KEK` created as the phase-1 guide said) keeps its name and keeps working. Its realm needs these two lines, one pair per such realm: ```hcl path "transit/encrypt/helix--kek-*" { capabilities = ["update"] } path "transit/decrypt/helix--kek-*" { capabilities = ["update"] } ``` Such a line is a prefix match, so it also matches the signing keys of a realm whose id starts with `-kek` (for `master`: `master-kek`, `master-kek-eu`, ...). HelixIAM therefore refuses to create a realm with such an id: an id that starts with the id of an existing realm followed by `-kek`, and any id that ends in `-kek`, answers `400` with `fieldErrors.realmId` (the realm settings API and a realm import). Realms created before this release are not checked, so make sure none has such an id before you add the lines. A new KEK (a realm moved to Protected or High assurance from now on) is a `helixkek-` key and needs no per-realm line. - `create`/`update` on `keys/helix-*`, `keys/helixdec-*` and `keys/helixkek-*`: create a realm's keys and KEK, rotate (`/rotate`), disable and allow deletion (`/config`). `read`: public keys and key versions. `delete`: the retention job's `destroy`. - **`exportable` and `allow_plaintext_backup` are denied** on those paths. `keys//config` accepts both, and once `true` they can never be set back: a stolen HelixIAM token could otherwise make every key exportable for another identity with export or backup rights. Tested on Vault 2.2.1 and OpenBao 2.7.2: such a request is `403`. - **HelixIAM checks both flags itself** when it first reads a key on a server (its public key, before signing), before a KEK's or decryption key's first use, and again for the keys in use on every health check (up to 16 keys per check, round robin). A key with either flag not `false` is **quarantined**: it is never used again on that server, and it is logged as an error naming the key. See [A key that can leave the backend](#a-key-that-can-leave-the-backend) for what clients see and how to replace it. - Nothing grants `export`, `backup`, `restore` or `datakey`. Lookup and renewal of its own token come from Vault's `default` policy. - A backend entry with this policy serves every realm on the installation: realm isolation inside the mount is enforced by HelixIAM (a key reference resolves only when its key name embeds its own realm), not by Vault. For isolation by Vault, give each tenant a backend entry of its own with a credential limited to its realm, and bind the entry to the realm: see [Dedicated key stores](#dedicated-key-stores). Set it up (Vault CLI; with OpenBao use `bao`): ```bash vault secrets enable transit vault policy write helixiam-custody helixiam-custody.hcl # AppRole (recommended outside Kubernetes); a single-use secret id works too (-secret-id-num-uses=1 on the role) vault auth enable approle vault write auth/approle/role/helixiam token_policies=helixiam-custody token_ttl=1h token_max_ttl=24h vault read -field=role_id auth/approle/role/helixiam/role-id # -> auth.role vault write -f -field=secret_id auth/approle/role/helixiam/secret-id # -> the secret-id file ``` ### OAEP key transport on each backend An eID decryption key decrypts only the `EncryptedKey` of an assertion: the identity provider encrypts a random content key to the SP certificate with RSA-OAEP, and the backend decrypts that. HelixIAM maps the XML Encryption identifiers exactly and never substitutes a parameter: - `http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p`: OAEP digest SHA-1, MGF1 with SHA-1, nothing else. - `http://www.w3.org/2009/xmlenc11#rsa-oaep`: the digest from `ds:DigestMethod` (default SHA-1) and the MGF from `xenc11:MGF` (default MGF1-SHA-1), each one of SHA-1, SHA-256, SHA-384 or SHA-512, set independently. - `http://www.w3.org/2001/04/xmlenc#rsa-1_5` (PKCS#1 v1.5 key transport) is **refused**. XML Encryption 1.1 warns against it (chosen-ciphertext attacks), the eIDAS cryptographic requirements do not list it, and none of the three eID schemes requires it. A non-empty OAEP `PSourceAlgorithm` (a label) is refused too. Each decryption key is registered for exactly one `{method, digest, mgf}`. When the key is made, HelixIAM encrypts 32 random bytes to its public key with exactly those parameters and has the backend decrypt them, and it also checks that a ciphertext made with other parameters is not decrypted. A backend that cannot honour the combination, or silently uses other parameters, is refused then (400 on `keyTransport`) and the new backend key is disabled. A login never discovers an unsupported combination. What each backend really decrypted when measured (a 3072-bit key, `target/*-oaep-matrix.txt` of the test runs): | Key transport | SoftHSM 2.6.1 / 2.7.0 (PKCS#11) | Vault 2.2.1 | OpenBao 2.7.2 | |---|---|---|---| | `rsa-oaep-mgf1p` (SHA-1, MGF1-SHA-1) | decrypts | decrypts | refused at key creation | | `rsa-oaep`, SHA-1 / MGF1-SHA-1 | decrypts | decrypts | refused at key creation | | `rsa-oaep`, SHA-256 / MGF1-SHA-256 | refused (2.7.0: `CKR_ARGUMENTS_BAD`, 2.6.1: `CKR_GENERAL_ERROR`) | decrypts | decrypts | | `rsa-oaep`, SHA-384 / MGF1-SHA-384 | refused | decrypts | refused at key creation | | `rsa-oaep`, SHA-512 / MGF1-SHA-512 | refused | decrypts | refused at key creation | | `rsa-oaep`, digest and MGF1 hash different (the other 12 of the 16 `rsa-oaep` combinations) | refused | refused before any call | refused before any call | - **PKCS#11 sets the digest and the MGF separately** (`CK_RSA_PKCS_OAEP_PARAMS`), so a token can in principle serve every combination. SoftHSM 2.6.1 and 2.7.0 hard-code SHA-1 for both, and so decrypt only the first two rows; their advertised mechanisms list SHA-2, so the backend's decryption self-test when it connects drops the others (they are not offered, and creating a key for one is a 400 on `keyTransport`). A backend whose self-test passes no transport refuses every eID decryption key at configuration time and says so; it still signs. **SHA-2 OAEP has not been tested with a decryption through a PKCS#11 backend of HelixIAM.** It was checked at the mechanism level only: the parameters HelixIAM sends were run on a raw session of the NSS soft token for all 17 combinations (all decrypt), and a recording test shows the exact parameters sent. NSS is not a supported backend: it reports `CKA_LOCAL` = false for keys it made itself, and HelixIAM refuses such a key. A real HSM must be tested with the self-test at key creation before it is relied on for SHA-2 OAEP. - **Vault and OpenBao Transit use one hash for the OAEP digest and MGF1** (`padding_scheme=oaep`, `hash_algorithm`). A combination with different hashes is refused on them at configuration time, with a message saying so; DigiD and eHerkenning documents that allow a SHA-2 digest with MGF1-SHA-1 therefore cannot use those backends for that combination (use `rsa-oaep-mgf1p`). - **OpenBao 2.7.2 ignores `padding_scheme` and `hash_algorithm` on decrypt:** it always decrypts OAEP with SHA-256 and MGF1-SHA-256 and answers `200` with a warning that it ignored them. HelixIAM refuses such an answer for every transport but exactly SHA-256 / MGF1-SHA-256, so a ciphertext made with other parameters never yields bytes. On OpenBao the only usable transport is `rsa-oaep` SHA-256 / SHA-256 (the eIDAS default). DigiD and eHerkenning connections whose identity provider cannot be set to that transport cannot decrypt on OpenBao. - **The decrypted content key exists in server memory.** The backend returns the unwrapped content key (a symmetric AES key) to the server, and the server uses it to decrypt the assertion. The SP's private key never leaves the backend, but the content key of an assertion is in the memory of the pod that handles the login, for the duration of the decryption (the buffers are zeroed afterwards). Treat a pod's memory as able to read the assertions it decrypts. - **KMIP offers a key transport only when the `keyTransport` setting lists it**, proven for each key when it is made; it has not been run against a live KMIP server (PyKMIP decrypts with symmetric keys only). **AWS KMS, Azure Key Vault / Managed HSM and Google Cloud KMS offer none** in this release, so an eID connection's decryption key cannot be kept on them. - The `keyTransport` setting of a PKCS#11 or KMIP backend and `GET .../sp-keys/algorithms` (below) show which transports a configured backend offers. The offer is the backend's mechanism list, not a promise: the self-test decides per key. ### KMIP (key managers) `type: kmip` talks to a KMIP key manager (an HSM-backed key server with a KMIP interface, for example) over mutual TLS. It is a built-in backend, counted as a sovereign key store (see [Sovereign key stores only](#sovereign-key-stores-only)). **What was tested.** The KMIP client is HelixIAM's own (TTLV over TLS, no third-party KMIP library). It is tested against **PyKMIP 0.10.0, patched for the KMIP 2.x gaps it has** (`e2e/kmip/patch_pykmip.py`: key attributes, links, X.509 public keys, error codes and the version a request is decoded in), and against a protocol fake in the unit tests. **No commercial KMIP server has been tested**, so interoperability with a specific product is not verified: run the HelixIAM test matrix (generate, sign per algorithm, KEK round trip, restart, certificate rotation) on your key manager before you rely on it. PyKMIP makes RSA keys only (no EC), cannot sign RSA-PSS with a 32-byte salt and decrypts only with symmetric keys (no RSA-OAEP), so **ECDSA (ES256, ES384) and RSA-OAEP key transport over KMIP have not been run against a live server** (they are built and unit-tested against the fake). Treat KMIP as **Preview** until it has run against an independent key manager. **Requirements.** KMIP 2.0 or later (Discover Versions picks the highest common version; a server that offers none is refused at start-up), mutual TLS 1.3 or 1.2 with host-name verification, and the operations Query lists: Create Key Pair, Activate, Get Attributes, Get, Sign, Revoke and Destroy for signing keys; Create, Activate, Get Attributes, Encrypt, Decrypt, Revoke and Destroy for key-encryption keys. No setting turns TLS, the client certificate or host-name verification off, and the server's certificate is trusted only through your `ca.crt` (never the JDK's defaults). | Setting | Default | Meaning | |---|---|---| | `address` | required | `host[:port]` of the KMIP server (`[v6-address]:port` for IPv6); port 5696. | | `serverName` | the address's host | The name the server's certificate must carry. | | `algorithms` | **required** | Comma-separated (chart: a list) out of `RS256`, `PS256`, `ES256`, `ES384`: what your key manager signs. EdDSA is never offered over KMIP. | | `kek` | **required** | `true` or `false`: whether the key manager does AES-256-GCM with the caller's associated data (key-encryption keys, Protected and High assurance, `DB_ENCRYPTION_KEK`). | | `keyTransport` | none | The RSA-OAEP key transports an eID decryption key may use, as `METHOD;DIGEST;MGF` (an RSA algorithm must be listed in `algorithms`). | | `opTimeout`, `connectTimeout` | `helix.custody.op-timeout`, `PT2S` | ISO-8601, above zero, at most `PT60S`. One `opTimeout` covers the whole of a call, including every round trip of a key generation. | | `connections` | 8 | Persistent TLS connections per pod, 1 to 64. | | `objectGroupPrefix` | `helix-` | A realm's keys are in the Object Group `` (1 to 32 letters, digits, `.`, `_`, `:`, `-`). | `algorithms`, `kek` and `keyTransport` are **the operator's statement, not probed**: when a pod starts it only reads (Discover Versions, Query) and makes nothing on the key manager, so every pod offers the same and no extra permission is needed. Each new key is then proven when it is generated (a signature that verifies, a KEK round trip that refuses another associated data); a key that does not do what you stated is disabled and the request refused. State only what your key manager does. The consequence: a wrong `algorithms` shows at the first key generation, not at start-up. The TLS material is three files in `/var/run/helix/custody//`: `tls.crt` (the client certificate chain, leaf first), `tls.key` (its PKCS#8 PEM private key; RSA or EC) and `ca.crt` (the CA certificates that sign the server's certificate). The server refuses to start when `tls.key` is **readable by others or writable by anyone but the owner**; owner and group read (mode 0440, with the pod's `fsGroup`) is accepted, which is what the chart mounts. The files are read again after a handshake fails on a certificate, so a rotated Secret is used without a restart. ```yaml custody: backends: - id: kmip-1 type: kmip address: kms.example.com:5696 serverName: kms.example.com algorithms: [RS256, ES256] # what your key manager signs kek: true tlsSecret: {name: helix-kmip-client} # a kubernetes.io/tls Secret: tls.crt and tls.key caSecret: {name: helix-kmip-ca, key: ca.crt} ``` Allow the KMIP port in `networkPolicy.extraEgressPorts` (5696). Keys are created Sensitive and not Extractable, named `helix:::` in the Object Group of the realm, and are read back after creation; a key that reads back extractable is destroyed (a new one) or quarantined (an existing one, like the other backends). The first use of a reference checks that the key's name and Object Group are the reference's realm, so a key of another realm is never used. **A revoked KMIP key cannot be activated again**, so a disabled key-encryption key cannot be enabled again. A permission or authentication denial (KMIP Permission Denied) is a `503` with `lastError: policyDenied` in the backend's health until a call works; an explicit refusal of an operation or its parameters is a refusal (`400`/`409`). ### Plugins A backend type that is not built in (`software`, `pkcs11`, `vault`, `kmip`) comes from a **plugin**: one jar behind the public key custody SPI, pinned by its SHA-256 and delivered in a signed plugin image. HelixIAM ships the cloud key store plugins below in its own plugin image; you can write another against the SPI. The server loads nothing unless you pin it (`custody.plugins.jars`; the chart leaves it empty), refuses to start with an unpinned jar, a mismatching pin, a writable plugin directory or a type that clashes, and lists what it loaded at `GET /admin/key-custody/plugins` (master-realm administrators; type, plugin and SPI version, jar file name and the first 12 hex characters of the pin) and on the master realm's **Realm keys** page. Every plugin backend gets the same treatment as a built-in one: the time bound, the circuit breaker, the in-flight limit, the realm binding, metrics, audit, and a self-test of every key it generates. The plugin guide, [custody-plugins.md](custody-plugins.md), has the contract, packaging, installing and the trust model. A plugin runs in the server process with the server's permissions: its class loader isolates its libraries, it is **not a sandbox**. ### Cloud provider key stores (AWS KMS, Azure Key Vault / Managed HSM, Google Cloud KMS) Three plugins, built from `helix-custody-plugins/` into the plugin image, let a realm keep its keys in a hyperscaler's key service. They are a **customer choice, never a default**: a realm on one of them is not a sovereign-key-store realm (see [Sovereign key stores only](#sovereign-key-stores-only)), and the key service is operated by the provider. They are **Preview**: they are tested against emulators and recordings built from the providers' published API shapes (`aws-kms` against LocalStack; `azure-kv` and `gcp-kms` against recordings and stand-ins), **not yet against a real AWS, Azure or Google Cloud account**. Verify them in your own account (create a throwaway realm, switch it to High assurance, sign, rotate, destroy) before production use, and check the permissions below against the provider's current documentation. What all three share: - **Workload identity only.** The pod proves who it is with the platform's workload identity (AWS IRSA or EKS Pod Identity, Azure Workload Identity, GKE Workload Identity Federation or an `external_account` file). No static key, client secret, key file or default credential chain is read; a setting that names one is refused. A chart `settings` key that looks like a credential is refused at render. - **Pinned endpoints, no environment redirection, no egress proxy.** Endpoints are fixed or set explicitly; the providers' environment variables that redirect an endpoint are not read. Every HTTP client is pinned to **no proxy** and the JVM's proxy properties are ignored: **egress through a proxy is not supported yet**. The pod must reach the provider's hosts directly (`networkPolicy.extraEgressPorts` opens 443); where a proxy is forced, exempt those hosts from it. The settings `insecureEndpointHosts` and (Azure, GCP) `identityEndpoint` are **for tests only**: `identityEndpoint` sends the pod's identity token or credential to whatever it names, so never set it in a deployment. - **Single-purpose keys, no eID decryption.** A signing key signs, a key-encryption key encrypts. No cloud backend offers RSA-OAEP key transport or sign-and-decrypt keys in this release, so an eID connection's decryption key cannot live on one (use PKCS#11, Vault/OpenBao or KMIP for that), and no backend offers EdDSA. The wallet's per-transaction ECDH key and the pepper stay software secrets. - **Keys are tagged for their realm** (AWS tags `helix:realm`, `helix:purpose`, `helix:kind`; Azure tags and GCP labels `helix-realm`, `helix-purpose`, `helix-kind`), and a key tagged for another realm, one that can leave the service, or one with imported key material is never used. - **Authentication or permission denial is a `503`** with `lastError: policyDenied` in the backend's health, never a fallback. Key reference, chart keys and every setting are in `values.yaml` (the `custody` block's comments) and in the plugin guide. Set-up per provider follows. #### AWS KMS (`aws-kms`) ```yaml serviceAccount: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/helix-custody # IRSA custody: plugins: {...} # the aws-kms jar (a release's packaged chart already carries it) backends: - id: aws-prod type: aws-kms region: eu-central-1 # required auth: web-identity # or pod-identity (EKS Pod Identity association for the service account) # algorithms: [RS256, PS256, ES256, ES384] # default all four ``` The role's **trust policy** (IRSA) names the cluster's OIDC provider and exactly the HelixIAM service account: ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"Federated": "arn:aws:iam::123456789012:oidc-provider/"}, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": {"StringEquals": { ":sub": "system:serviceaccount::", ":aud": "sts.amazonaws.com"}} }] } ``` **Not yet tested against a real account; adapt and verify before use.** A **least-privilege permissions policy** (an example to adapt: scope the region and account, and for one realm per role replace the `Null` conditions with `StringEquals` on `aws:ResourceTag/helix:realm`; the keys' default key policy, made by `CreateKey` without a policy, delegates to IAM policies of the account): ```json { "Version": "2012-10-17", "Statement": [ {"Sid": "CreateTaggedKeys", "Effect": "Allow", "Action": ["kms:CreateKey", "kms:TagResource"], "Resource": "*", "Condition": {"Null": {"aws:RequestTag/helix:realm": "false"}, "ForAllValues:StringEquals": {"aws:TagKeys": ["helix:realm", "helix:purpose", "helix:kind"]}}}, {"Sid": "UseHelixKeysOnly", "Effect": "Allow", "Action": ["kms:DescribeKey", "kms:ListResourceTags", "kms:GetPublicKey", "kms:Sign", "kms:Encrypt", "kms:Decrypt", "kms:DisableKey", "kms:EnableKey", "kms:ScheduleKeyDeletion"], "Resource": "arn:aws:kms:eu-central-1:123456789012:key/*", "Condition": {"Null": {"aws:ResourceTag/helix:realm": "false"}}}, {"Sid": "HealthProbe", "Effect": "Allow", "Action": "kms:ListKeys", "Resource": "*"} ] } ``` Limits: AWS KMS **does not offer sign-and-decrypt keys** (an RSA key has one usage), no EdDSA, and with `origin: AWS_CLOUDHSM` (with `customKeyStoreId`) only key-encryption keys. The plugin **pins the KMS and STS endpoints** to the region's partition default, so the AWS SDK's `AWS_USE_FIPS_ENDPOINT` and dual-stack switches have no effect: **operators who need the FIPS endpoints (for example in AWS GovCloud, `us-gov-*`) set `endpoint` and `stsEndpoint` explicitly** (for example `https://kms-fips.us-gov-west-1.amazonaws.com`). `endpoint` and `stsEndpoint` also take interface VPC endpoints. A key-encryption key's encryption context carries the caller's associated data as `v1.`, so it is never empty. A destroyed key goes into AWS's deletion window (`deletionWindowDays`, 7 to 30, default 30). #### Azure Key Vault and Managed HSM (`azure-kv`) ```yaml podLabels: {azure.workload.identity/use: "true"} serviceAccount: annotations: {azure.workload.identity/client-id: ""} custody: backends: - id: azure-prod type: azure-kv vaultUrl: https://helix-prod.managedhsm.azure.net # Managed HSM; kind: managed-hsm follows from the URL ``` **Not yet tested against a real account; adapt and verify before use.** Federate the identity with the service account, then give it the roles (the example uses the Azure CLI; check the current syntax in the Azure documentation): ```bash az identity federated-credential create --name helixiam --identity-name helix-custody --resource-group \ --issuer --subject system:serviceaccount:: \ --audiences api://AzureADTokenExchange # Managed HSM (local RBAC): key lifecycle and the cryptographic operations, on the key scope only az keyvault role assignment create --hsm-name helix-prod --assignee --scope /keys \ --role "Managed HSM Crypto Officer" az keyvault role assignment create --hsm-name helix-prod --assignee --scope /keys \ --role "Managed HSM Crypto User" ``` The identity needs on the keys: create, get, list, update, sign, encrypt, decrypt, delete and purge (the purge permission is only for a clean-up that is best effort: without it a deleted key stays soft-deleted until its retention ends). On a plain Key Vault with Azure RBAC, assign **Key Vault Crypto Officer** on the vault's resource ID. Scope to the one vault or HSM, never a subscription. Limits: **a realm needs a Managed HSM.** A plain Key Vault has no AES-GCM, so it holds no key-encryption keys, and the server refuses to switch a realm to Protected or High assurance on it (both need the realm's key-encryption key); only a Managed HSM holds `oct-HSM` AES-256 keys. Azure Key Vault has no Ed25519 (no EdDSA) and no sign-and-decrypt keys; keys are `exportable=false` with no release policy. The caller's associated data is sent to Managed HSM exactly as given. The three workload-identity values `AZURE_CLIENT_ID`, `AZURE_TENANT_ID` and `AZURE_FEDERATED_TOKEN_FILE` are read from a **Java system property of that name first, then from the environment**; no other `AZURE_*` variable is read, and the Entra ID authority is fixed by the vault's cloud (`login.microsoftonline.com`, `login.chinacloudapi.cn`, `login.microsoftonline.us`), never `AZURE_AUTHORITY_HOST`. Preview: tested against published REST API 7.5 shapes only. #### Google Cloud KMS (`gcp-kms`) ```yaml custody: backends: - id: gcp-prod type: gcp-kms project: helix-prod location: europe-west4 keyRing: helix-ring # an existing key ring: HelixIAM creates keys in it, never the ring auth: gke-workload-identity # or external-account (workload identity federation, any Kubernetes) # protectionLevel: HSM # default (Cloud HSM); SOFTWARE ``` **Not yet tested against a real account; adapt and verify before use.** A custom role with exactly the permissions the plugin uses, bound on the key ring (not the project) to the Kubernetes service account's principal (GKE Workload Identity) or to the workload identity pool principal (`external-account`): ```bash gcloud iam roles create helixCustody --project=helix-prod --title="HelixIAM key custody" --permissions=\ cloudkms.keyRings.get,cloudkms.cryptoKeys.create,cloudkms.cryptoKeys.get,cloudkms.cryptoKeyVersions.get,\ cloudkms.cryptoKeyVersions.update,cloudkms.cryptoKeyVersions.destroy,cloudkms.cryptoKeyVersions.viewPublicKey,\ cloudkms.cryptoKeyVersions.useToSign,cloudkms.cryptoKeyVersions.useToEncrypt,cloudkms.cryptoKeyVersions.useToDecrypt gcloud kms keyrings add-iam-policy-binding helix-ring --location=europe-west4 --project=helix-prod \ --role=projects/helix-prod/roles/helixCustody \ --member="principal://iam.googleapis.com/projects//locations/global/workloadIdentityPools/helix-prod.svc.id.goog/subject/ns//sa/" ``` With `external-account`, `credentialConfigFile` names the `external_account` file (a ConfigMap or Secret; it holds **no secret**: `gcloud iam workload-identity-pools create-cred-config` writes it) and `tokenAudience` makes the chart project the pod's service-account token for the pool provider. A `service_account` key file, Application Default Credentials and executable credential sources are refused. Limits: **Cloud EKM (`EXTERNAL`, `EXTERNAL_VPC`) is not supported** (each EKM key version needs an external key URI, so HelixIAM cannot generate keys there; use KMIP for an external key manager). No EdDSA (Cloud KMS signs Ed25519 over raw messages only) and no sign-and-decrypt keys. A signing key has one algorithm (the one it was made for). Cloud KMS never deletes a key's id: destroying a key schedules its version's destruction (`destroyScheduledDuration`, default `P30D`). Every request and answer is checked against Cloud KMS's CRC32C fields. Preview: tested against published REST API v1 shapes only. For Azure and Google Cloud the FIPS level of the service is the provider's (see [Compliance](#compliance)). ## Dedicated key stores A backend entry is one HSM partition (a token with its own PIN) or one Vault/OpenBao credential. The operator decides how many entries there are and which realms each one serves; the console only assigns a realm to an entry. Three tiers: | Tier | Set-up | What separates the tenants | |---|---|---| | **Shared** (default) | One entry, no `realms`. Every realm can be assigned to it. | HelixIAM: a key reference resolves only in its own realm, and key names embed the realm. One HSM PIN or Vault credential reaches every realm's keys. | | **Dedicated partition** | One entry per tenant: its own HSM partition (slot, token, PIN) or its own Vault credential, limited to that realm's key names, and `realms: []`. | The HSM or Vault itself: the tenant's credential cannot reach another tenant's keys (on Vault/OpenBao: only while the `keyCustody` health shows no `prefixOverlap` for it, see below; a mount or namespace per tenant needs no such check). HelixIAM refuses every other realm on the entry before it calls the backend. | | **Customer-owned** | The tenant runs the HSM or Vault and gives the operator a credential for it; configured like a dedicated partition. | As dedicated, plus the tenant controls the key store and can revoke the credential (which stops its realm signing). | ### Binding a backend to realms `helix.custody.backends[n].realms` (chart: `custody.backends[n].realms`) lists the realm ids an entry serves. Absent, the entry is **shared** (every realm, the behaviour before rc.12). With ids, it is **dedicated** to them. An empty list also means shared in the server's configuration; the chart refuses an explicit `realms: []` (someone writing it for "no realm yet" would get the opposite), so leave the key out for a shared entry. Name entries neutrally (`hsm-a`, `vault-tenant-2`): an entry's id is shown to the admins of the realms it serves, and to a realm it no longer serves. - **When a realm is assigned** (`PUT /admin/realms/{realmId}/key-custody`), an entry that does not list the realm is refused: `400` with `fieldErrors.backendId` and `fieldErrorCodes.backendId = backendRealmNotAllowed`. Nothing is made in the backend. The console lists only the entries that serve the realm (and says how many it left out). - **Every custody call** checks the binding before the backend is called: a new key or key-encryption key of another realm is refused (`backendRealmNotAllowed`; the admin endpoints that make keys answer `400`: realm key rotation and CSR, wallet verifier keys `generate`, credential issuer keys, eID SP keys), and any use of a key of another realm (sign, decrypt, unwrap, the KEK rotation's unwrap, public key, disable, destroy) is refused as unavailable: `503 temporarily_unavailable` with `Retry-After` on protocol endpoints, `409 {"error":"backendRealmNotAllowed","code": "backendRealmNotAllowed"}` on the custody admin endpoints (the console shows its own copy). Never a `500`, never another key. The first refusal per entry and realm is logged as an error naming both. - **A key-encryption key rotation** of a realm its entry no longer serves is refused before the backend is called (start, resume and abandon: `400` with `code` `backendRealmNotAllowed`; no KEK is made; the console does not offer them). A rotation running when the binding changes stops on the next pass of the rotator (`FAILED`, `lastErrorCode` `backendRealmNotAllowed`): nothing is unwrapped, disabled or destroyed for a realm its entry no longer serves, and the retention job does not destroy that realm's old KEK either. Put the realm back in `realms`, then start the rotation again to resume it, or lower the realm to Standard (below): that stays allowed, and ends the stopped rotation. - **Realm ids** (the Vault/OpenBao prefix caveat below): while any entry lists `realms`, a new realm (admin API or import) whose id is another realm's id, or an id in some entry's `realms`, followed by `-`, or the start of one (`tenant-a-eu` next to `tenant-a`, or `tenant` next to `tenant-a`), or that differs from one only in case, is refused: `400` with `fieldErrors.realmId` and `fieldErrorCodes.realmId = realmIdPrefix` (an import reports the realm entry as failed with that code; the console shows its own text, which names no other realm). Two realms created at the same moment cannot both pass: the rules are checked again under a cluster-wide realm-creation lock. Compared ignoring case. - **Realm ids that overlapped before an entry was bound** (or that a later `realms` change makes overlap) are reported by the binding check (at start-up and every `helix.custody.binding-check-interval`, default `PT1M`), in the `keyCustody` health (`prefixOverlaps: {"": ["", ...]}`) and in the master-only `GET /admin/key-custody/backends` (`prefixOverlaps: [{"realm", "overlapsWith", "isolationEnforced"}]`): - when the other realm's keys are in the **same key space** (a Vault/OpenBao entry with the same address, mount and namespace), the bound entry is `DOWN` with `realmBinding: {"": "prefixOverlap"}` and `isolationEnforced: false`: the tenant policy `helix--*` also reaches the other realm's keys, so the dedicated tier's guarantee does not hold for it; - otherwise it is a warning (`warnings: {"prefixOverlap:": ""}`), the entry stays `UP`. Remediation: give the tenant a Transit mount or namespace of its own (entry setting `mount` or `namespace`; then no prefix matters), or move the other realm's keys out of that key space, or replace the overlapping realm with one whose id does not overlap. An HSM is never affected (its labels are `helix::...`, and `:` is not in a realm id). - **At start-up and then every `helix.custody.binding-check-interval`**, every Protected or High-assurance realm assigned to an entry that no longer lists it (an operator changed `realms`) is named in the entry's `realmBinding`: `GET /admin/key-custody/backends` lists the entry with `healthy: true` (the backend itself answers) and `realmBinding: {"": "notAllowed"}`, and the aggregate `/actuator/health` is `DOWN` (the per-backend detail of the `keyCustody` component is shown only where health details are enabled). Its custody keys are refused as above, and its wrapped keys never keep the pod from becoming ready; every other realm keeps working. A Standard realm on such an entry is named only while a key of it that can still sign or verify lives there (see below). A realm id in `realms` that no realm has yet is allowed (realms can be created later) and logged as a warning. - A realm id that is not a realm id (1-100 letters, digits, `.`, `_` or `-`, starting with a letter or digit), a realm listed twice, or `realms` on the `software` backend is refused at start-up; the chart refuses the first two, and an explicit empty list, at render time. - **The binding is read once per pod.** During a rollout that changes `realms`, old and new pods disagree: a pod that does not serve the realm answers `503` for it, and its rotator stops a KEK rotation that a new pod started (`FAILED`, `backendRealmNotAllowed`, nothing disabled). Both fail closed; assign realms and start KEK rotations only once every pod runs the new `realms`. - **Do not reuse a bound realm id for another customer.** The id is in the entry's `realms` and in the tenant policy (`helix--*`), so a new realm with the same id gets the old realm's entry and Vault policy, and with them any of the old realm's keys still in the backend. `GET /admin/key-custody/backends` (master-realm admins) returns each entry's `isolation` (`shared` or `dedicated`), `realms` (ids only, never a setting), and, when there is something to say, `realmBinding`, `prefixOverlaps` and `warnings`. A realm's view, `GET /admin/realms/{realmId}/key-custody`, which the realm's own admins read too, returns `backendIsolation` (`level`, `realmCount`, `dedicatedToRealm`, `allowsRealm`) and never the other realms' ids. The console shows it as **Shared key store**, **Dedicated to this realm** or **Dedicated to N realms** (in Dutch the section is "Sleutelbewaring", custody, and an entry a "sleutelopslag", key store). **Taking a realm off an entry that no longer serves it.** Either put the realm back into `realms` and restart, or lower the realm to Standard. Lowering asks nothing of the entry, and is allowed also while a KEK rotation stopped because the entry no longer serves the realm (it ends that rotation, `ABANDONED`). After the lowering the realm is **released** on the entry: its keys there may be ended (disabled, destroyed), never used or made. Then: 1. The realm's new software keys sign once their publish window has passed (or at once when activated). Its old custody keys stay published, refused, until their tokens have expired; retire them (Realm keys; an emergency retire for keys whose tokens you do not need). 2. Once no key of the realm that can sign or verify lives on the entry, the binding check no longer names it: the `keyCustody` health is `UP` again, without a restart. Each pod notices at its own next binding check (`helix.custody.binding-check-interval`, default `PT1M`), not at once: for up to a minute after the last custody key is retired one pod can still be `DOWN` while another is `UP`. 3. The retention job disables and destroys the retired keys there. The realm's key-encryption key(s) there (its KEK, one a rotation was moving off, and the new and superseded KEKs of its abandoned rotations) are released in stages. Each stage is re-checked under the realm key lock: still Standard with the same KEK, still not served by the entry, no key there that can sign or verify (a wrapped key that is not RETIRED, with or without a recorded KEK), and no unfinished KEK rotation: 1. the first retention run that sees the realm released records it (UTC; audit `CUSTODY_KEK_RELEASE_START`); 2. `helix.custody.destroy-after` later the KEK(s) are **disabled** (`CUSTODY_KEY_DISABLE`); 3. `helix.custody.destroy-after` after the disable they are **destroyed** (`CUSTODY_KEY_DESTROY`; one another pod destroyed already counts as destroyed) and the entry is forgotten (`backendId` becomes null). The realm can then be raised again on another entry. From the disable until the destroy is done, a profile change of the realm is refused (`409`, `code` `kekReleasePending`); the console shows the stage in **Realm keys → Key custody**. A pod that serves the realm again (a `realms` typo fixed) **cancels** a release that has not started to destroy: the KEKs this release disabled (the realm's KEK, and the one a rotation was moving off) are enabled again (`CUSTODY_KEY_ENABLE`), never a KEK a rotation disabled on purpose (an abandoned or quarantined one), and the release is forgotten (`CUSTODY_KEK_RELEASE_CANCEL`); a later release starts the clock again. A disable that a cancel overtook is undone by the pod that made it. Once the destroy has started (`DESTROYING`), any pod that configures the entry finishes it, also one that serves the realm again. While pods disagree (a rollout), each pod that serves the realm cancels, so nothing is disabled before every pod has stopped serving it for a full `destroy-after`. A KEK that is the configured `DB_ENCRYPTION_KEK` (any version), or is named like one (`helixdbkek-`, `:DBKEK:`), is never released, disabled or destroyed: an error is logged instead. Re-enabling works on Vault, OpenBao and PKCS#11. **When the enable is refused for good** (an HSM policy that does not allow `CKA_ENCRYPT` / `CKA_DECRYPT` back, or a credential that may no longer write the key's config), the release stays and the profile change stays refused; the pod logs an error after three refused attempts. The way out: enable the key in the backend by hand, or remove the realm from the entry's `realms` again (a restart), so that the release goes on to destroy the KEK as above; then the realm can be raised on another entry. **When no pod can finish a destroy** (the entry was removed from the configuration while the release was at `DESTROYING`), a master-realm administrator finishes it: **Realm keys → Key custody → Finish key release** (typing the realm id), or `POST /admin/realms/{realmId}/key-custody/kek-release/finish`. Where the server configures the entry the KEKs are destroyed; otherwise they are only forgotten, and the answer (and the console) names them to remove in the backend by hand. Audited `CUSTODY_KEK_RELEASE_FINISH`; `409` `kekReleaseNotDestroying` when there is nothing to finish. If the entry's credential can no longer reach the realm's keys (the operator also narrowed its policy), each disable or destroy is an unavailability retried on every run; remove those keys in the backend by hand, or give the credential back its access until the retention job is done. The `DB_ENCRYPTION_KEK` is the operator's key, not a realm's, and belongs on an entry **shared by every realm** (no `realms`): a tenant's credential must never open the database encryption key. `DbKeyWrapTool --create-kek` on an entry with `realms`, and a new wrap (`--kek`, or the new KEK of `--rewrap-from`) by a KEK on such an entry, are refused (`createDbEncryptionKek` refuses it too, code `dbEncryptionKekNeedsSharedBackend`). Moving the data key off such a KEK (`--rewrap-from` it) stays possible. A server whose `DB_ENCRYPTION_KEK` is on an entry with `realms` still starts (it must open its own database), logs a warning, and the `keyCustody` health names it (`warnings: {"dbEncryptionKek": "onBoundEntry"}`, the entry stays `UP`): move the data key to a KEK on a shared entry. ### High assurance on a dedicated key store `helix.custody.require-dedicated-backend-for-high` (chart: `custody.requireDedicatedBackendForHigh`), default `false`. When `true`, a High-assurance realm must be on an entry whose `realms` is exactly that realm: - a switch to High assurance on any other entry (shared, or dedicated to several realms) is refused: `400` with `fieldErrorCodes.backendId = backendNotDedicated`; the console disables those entries for High assurance and says why; - a High-assurance realm already on such an entry is flagged at start-up: the aggregate `/actuator/health` is `DOWN`, and the backends list shows that entry with `realmBinding: {"": "notDedicated"}`. The realm **keeps signing** (the setting is a policy check, not a key fault). A High-assurance realm on a shared entry gets a calm advisory in the console either way. **Upgrading from rc.11.** The setting is off by default, so High-assurance realms on a shared entry keep working. To turn it on: give each High-assurance realm an entry of its own first. Because a realm stays on its entry, the move is: lower the realm to Protected or Standard, wait for its new keys to sign, raise it again on its new entry, then set the setting. The health shows any realm still on a shared entry. ### Per-tenant recipe: Vault and OpenBao One AppRole (or token) per tenant, whose policy covers only that realm's key names. With realm `tenant-a`: ```hcl # helixiam-tenant-a.hcl: only tenant-a's signing keys, decryption keys and key-encryption keys path "transit/keys/helix-tenant-a-*" { capabilities = ["create", "read", "update", "delete"] denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] } } path "transit/keys/helixdec-tenant-a-*" { capabilities = ["create", "read", "update", "delete"] denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] } } path "transit/keys/helixkek-tenant-a-*" { capabilities = ["create", "read", "update", "delete"] denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] } } path "transit/sign/helix-tenant-a-*" { capabilities = ["update"] } path "transit/sign/helixdec-tenant-a-*" { capabilities = ["update"] } path "transit/encrypt/helixkek-tenant-a-*" { capabilities = ["update"] } path "transit/decrypt/helixkek-tenant-a-*" { capabilities = ["update"] } path "transit/decrypt/helixdec-tenant-a-*" { capabilities = ["update"] } # Only while tenant-a's key-encryption key has the name of before rc.12 (helix-tenant-a-kek-): its wrapped keys # are unwrapped with it. Rotate that KEK (it becomes helixkek-tenant-a-...), then remove these two lines. path "transit/encrypt/helix-tenant-a-kek-*" { capabilities = ["update"] } path "transit/decrypt/helix-tenant-a-kek-*" { capabilities = ["update"] } ``` ```bash vault policy write helixiam-tenant-a helixiam-tenant-a.hcl vault write auth/approle/role/helixiam-tenant-a token_policies=helixiam-tenant-a token_ttl=1h token_max_ttl=24h vault read -field=role_id auth/approle/role/helixiam-tenant-a/role-id # -> auth.role of entry vault-tenant-a vault write -f -field=secret_id auth/approle/role/helixiam-tenant-a/secret-id # -> its own Kubernetes Secret ``` - **The realm id in the policy must be the real realm id.** The example uses `tenant-a` as the realm id; a policy that names another id than the realm's (a copy of the example for realm `acme`, say) denies every call for that realm's keys. HelixIAM answers `503` (the backend is unusable from this server), never a `500`, and says so in two places: one `WARN` per backend and key name in the server log (rate limited, no secrets) that names the backend id and the key name and says the call was denied by the backend's policy, and `lastError: policyDenied` in that backend's entry of the `keyCustody` health, until a call for that key works again. Fix the policy's key-name paths; no restart is needed. - **Realm ids that are prefixes of each other.** `helix-tenant-a-*` also matches the keys of a realm `tenant-a-eu` (`helix-tenant-a-eu-...`). While an entry lists `realms`, HelixIAM refuses to create such a realm (`realmIdPrefix`, see [Binding a backend to realms](#binding-a-backend-to-realms)); realms that already overlapped are named in the `keyCustody` health (`DOWN` when they share the key space) and the backends list. Key names were not changed instead: a realm id may contain `-`, `.` and `_`, so no character of a Vault key name can end a realm id unambiguously, and existing keys and policies keep working. The stronger form below needs no rule at all. - **Stronger: a mount or namespace per tenant.** A Transit mount per tenant (`vault secrets enable -path=transit-tenant-a transit`, entry setting `mount: transit-tenant-a`, policy paths under `transit-tenant-a/`), or a Vault Enterprise namespace per tenant (entry setting `namespace`). Then no prefix matters, and the tenant's keys are not even listed to another credential. - Tested (rc.12, Vault 2.2.1 and OpenBao 2.7.2): two AppRoles on one mount, each with the policy above for its realm (on Vault each entry bound to its realm). Tenant A's credential creating or using tenant B's key names (signing keys and KEKs, also the KEK rotation's unwrap) gets the backend's `403`, which HelixIAM answers as an unavailability (`503` on request paths), never a `500` and never a key. ### Per-tenant recipe: HSM (PKCS#11) One partition (slot, token) per tenant, each with its own user PIN in its own Kubernetes Secret, one entry each. On SoftHSM (for a test; a network HSM has its own partition tooling): ```bash softhsm2-util --init-token --free --label tenant-a --so-pin "$SO_PIN_A" --pin "$PIN_A" softhsm2-util --init-token --free --label tenant-b --so-pin "$SO_PIN_B" --pin "$PIN_B" kubectl create secret generic hsm-tenant-a-pin --from-literal=pin="$PIN_A" kubectl create secret generic hsm-tenant-b-pin --from-literal=pin="$PIN_B" ``` Each tenant's keys and key-encryption key are made on its own token only; another entry's PIN cannot reach them. Tested (rc.12, SoftHSM 2.6): two tokens in two slots with different PINs, each bound to its realm. ### Helm example: two tenants ```yaml custody: requireDedicatedBackendForHigh: true backends: - id: vault-tenant-a type: vault address: https://vault.internal:8200 auth: {kind: approle, role: "", secretIdSecret: {name: vault-tenant-a-secret-id, key: secret-id}} caSecret: {name: vault-ca, key: ca.crt} realms: [tenant-a] - id: hsm-tenant-b type: pkcs11 library: /opt/hsm/lib/libsofthsm2.so tokenLabel: tenant-b pinSecret: {name: hsm-tenant-b-pin, key: pin} realms: [tenant-b] - id: vault-shared # Standard and Protected realms of other tenants type: vault address: https://vault.internal:8200 auth: {kind: approle, role: "", secretIdSecret: {name: vault-secret-id, key: secret-id}} caSecret: {name: vault-ca, key: ca.crt} ``` Then, as a master-realm administrator, move each tenant onto its entry (console: **Realm keys → Key custody → Change profile**, or `PUT /admin/realms/tenant-a/key-custody {"profile": "HIGH", "backendId": "vault-tenant-a"}`). ## Sovereign key stores only A realm can be restricted to **sovereign key stores**: key stores the customer or the operator controls, never a hyperscaler's key service. It is a per-realm policy with two values, `any` (the default, the behaviour before rc.13) and `sovereign`. **What counts as sovereign.** Every configured backend has a class: | Class | Backend types | |---|---| | Sovereign | `pkcs11`, `vault` (Vault and OpenBao), `kmip`, and any plugin type the operator lists in `helix.custody.sovereign-types` (chart: `custody.sovereignTypes`) | | Cloud provider (`hyperscaler`) | `aws-kms`, `azure-kv`, `gcp-kms`, always, even if listed | | Software | `software` | | Partner (`plugin`) | any other plugin type | **Software keys do not count as sovereign.** A `sovereign` realm cannot be in the Standard profile (its keys would be software keys in the database, and the server refuses to lower such a realm to Standard), and it cannot import a software key. It must be Protected or High assurance on a sovereign backend. A type in `sovereign-types` is **your own assertion** about that plugin's key store: HelixIAM does not verify it. The server refuses to start when the list names a cloud provider type, `software`, or a malformed name. **Setting it.** Only **administrators of the master realm** can set or lower the policy (`403` for anyone else); the console shows it as a choice on **Realm keys → Key custody**, where the key-store picker then offers sovereign stores only. API: ```bash # set the policy on its own (200 {"realmId", "keyStorePolicy", "changed"}) curl -X PUT "$BASE/admin/realms/acme/key-custody/key-store-policy" -H 'Content-Type: application/json' \ -d '{"policy": "sovereign"}' # or with the profile switch: {"profile": "HIGH", "backendId": "hsm-1", "keyStorePolicy": "sovereign"} curl -X PUT "$BASE/admin/realms/acme/key-custody" -H 'Content-Type: application/json' \ -d '{"profile": "HIGH", "backendId": "hsm-1", "keyStorePolicy": "sovereign"}' # the realm's policy and the class of its backend (any realm admin may read it) curl "$BASE/admin/realms/acme/key-custody" # keyStorePolicy, backendStoreClass # the backends this installation offers, with their class (master-realm admins) curl "$BASE/admin/key-custody/backends" # storeClass: software | sovereign | hyperscaler | plugin ``` **What is refused or enforced.** - Setting `sovereign` on a realm that is Standard, or whose backend is not sovereign, or that has **any live key on another key store, answers `409` (`keyStorePolicyConflict`)**. This includes a key that is only still in its grace period (rotated, not yet retired) on the old, non-sovereign backend: **it blocks the policy until that key is retired.** Software keys left in their grace period by the switch from Standard (verify-only, never used to sign again) do not block it. Move the realm first, then let the old keys retire (or retire them), then set the policy. This check covers every key family of the realm: realm keys, wallet verifier keys, issuer keys and eID SP keys. - With the policy set, a switch to a non-sovereign backend, a lowering to Standard, an import of a software key and a new key on a non-sovereign backend are refused with the code `keyStorePolicy`. Retiring (disabling, destroying) a key stays allowed. - At run time a key of a `sovereign` realm on a non-sovereign backend is never used (`503`, the backend is not called), and the `keyCustody` health goes DOWN naming the realm (also for a sovereign realm listed in the `realms` of a non-sovereign entry). This can only arise from a direct database edit or a configuration change; a correction clears it at the next check. - Realm export carries the policy (`keyCustody.keyStorePolicy`) and never the backend; an import is checked against it before anything is written. - Lowering the policy to `any` is also master-realm only. Every change is audited as `CUSTODY_KEY_STORE_POLICY` (`from`, `to`). The policy is a guard on which stores the server will use, not a statement about the stores: whether a store is sovereign in the legal sense (who operates it, under which jurisdiction) is the operator's decision when it lists a plugin type, and the customer's when it picks Vault, an HSM or a KMIP key manager. ## Moving a realm Console: **Realm keys → Key custody → Change profile** (master-realm administrators only; others see the profile read-only). API: sign in as a master-realm administrator on the admin host (see [Authenticating to the API](../getting-started/api-authentication.md)): ```bash # The backends this installation offers: id, type, algorithms, healthy, latencyMs (never their settings) curl -s -b cookies.txt "$HELIX_URL/admin/key-custody/backends" # A realm's profile, backend health and keys (purpose, algorithm, custody, keyRef, status) curl -s -b cookies.txt "$HELIX_URL/admin/realms/$REALM/key-custody" # Move it: 202 with the planned keys and when each starts signing curl -s -b cookies.txt -X PUT -H "X-XSRF-TOKEN: $CSRF" -H "Content-Type: application/json" \ -d '{"profile":"HIGH","backendId":"hsm-1"}' \ "$HELIX_URL/admin/realms/$REALM/key-custody" ``` `profile` is `STANDARD`, `PROTECTED` or `HIGH`. What a move does, in order: 1. Checks the backend: known, not `software`, serves the realm (its `realms`, `backendRealmNotAllowed`; for High assurance with `helix.custody.require-dedicated-backend-for-high`, dedicated to the realm alone, `backendNotDedicated`), reachable (otherwise 503), holds key-encryption keys, non-extractable keys for High assurance, and signs every algorithm the realm's purposes use (High assurance also needs PS256 for RSA). A refusal is `400` with `fieldErrors.backendId` naming the algorithm and purposes, e.g. `cannot sign ES256 (access, id, saml keys)`. Then checks what in the realm the target profile must serve, each refusal a `400` with `fieldErrors.profile` naming what conflicts: an enabled SAML service provider that asks for **RSA-SHA1** (any target other than Standard) or **RSA-SHA512** (High assurance, whose backend does not sign RS512), and, for High assurance, an enabled wallet connection that still signs with a software verifier key. The issuer, eID and metadata keys add more (see [High-assurance rules for the issuer, eID and metadata keys](#high-assurance-rules-for-the-issuer-eid-and-metadata-keys)): an enabled credential issuer when the target backend cannot sign ES256 (Protected and High assurance), and, for High assurance, an ACTIVE or PENDING software issuer key, a live development CA, an enabled eID connection that still stores a pasted SP key, and an enabled eID connection the backend cannot serve (no signature algorithm of its scheme, no RSA size at the scheme's minimum, no key transport it decrypts, or a live custody key whose transport it cannot decrypt). 2. **Creates the realm's own KEK** in the backend (one per realm; its reference is stored on the realm, it is a public reference, not a secret). 3. Stores the profile and schedules a rotation of every purpose and family into the new custody. The new keys are published at once and **start signing after the JWKS cache window** (the same publish-before-use windows as any rotation), so relying parties have the new keys before they see them. Activate earlier only when nobody can have cached the realm's JWKS (`POST /admin/realms/{realm}/keys/{keyId}/activate?force=true`). 4. The old keys stop signing when the new ones start and stay published until their tokens have expired. 5. Audits `CUSTODY_PROFILE_CHANGE` (and `CUSTODY_KEY_GENERATE` per new key). Lowering a profile works the same way (the console asks for confirmation); lowering to Standard needs `DB_ENCRYPTION`. The realm keeps its KEK, so its wrapped keys still decrypt while they are published. A realm already in custody cannot be moved to **another** backend in this release. ### Day-to-day key work Day-to-day key work stays with the realm's own administrators (`realm-admin` on that realm): - **Rotate** (`POST /admin/realms/{realm}/keys/rotate`), with the usual pending window. - **Emergency rotate** (`{"emergency": true}`) or emergency retire (`DELETE /admin/realms/{realm}/keys/{keyId}?emergency=true`): the new key signs at once and the old key is **disabled in the backend at once** (`CUSTODY_KEY_DISABLE`), without waiting for the platform operator. See [Emergency rotation and caches](#emergency-rotation-and-caches). - **Rotation is per algorithm.** A rotate request rotates one algorithm family, in every purpose (`access`, `id`, `saml`) that holds a key of it. Without `algorithm` it is RS256: `{"emergency": true}` replaces only the RSA keys. The ES256 keys need `{"emergency": true, "algorithm": "ES256"}` (EdDSA: `"algorithm": "EdDSA"`). To replace every key of a realm, send one request per algorithm its keys use (the `algorithm` column of `GET /admin/realms/{realm}/keys`). - **CSR and certificate** for keys that need a CA-issued certificate (DigiD, eHerkenning, PKIoverheid SAML signing): `POST /admin/realms/{realm}/keys/csr` with a `purpose` (`all`, `access`, `id` or `saml`; empty means `all`) generates the key in the realm's custody and returns the CSR, signed inside the backend; upload the issued chain with `PUT /admin/realms/{realm}/keys/{keyId}/certificate`. In Protected and High assurance such a key cannot be activated before its certificate is attached. ```bash curl -s -b cookies.txt -X POST -H "X-XSRF-TOKEN: $CSRF" -H "Content-Type: application/json" \ -d '{"purpose":"saml","algorithm":"RS256","subject":"CN=idp.example.com, O=Example, C=NL","dnsNames":["idp.example.com"]}' \ "$HELIX_URL/admin/realms/$REALM/keys/csr" | jq -r .csrPem > saml.csr ``` Retired backend keys are disabled at retirement, and **destroyed** by the retention job `helix.custody.destroy-after` (default `P30D`) after they were disabled; the job runs every `helix.custody.retention-check-interval` (default `PT1H`) and is audited (`CUSTODY_KEY_DISABLE`, `CUSTODY_KEY_DESTROY`, actor `system:master`). There is no destroy endpoint. **Requests nobody certifies are bounded.** A realm holds at most `helix.custody.max-pending-keys-per-realm` (default 5) CSR keys waiting for their certificate, and at most as many generated verifier keys waiting for theirs; one more is refused with `400` (attach a certificate to a waiting key, or retire one, first) before any backend is called. A CSR key still without a certificate `helix.custody.pending-key-ttl` (default `P30D`, because a CA can take weeks to issue) after it was made is retired by the retention job (`CUSTODY_KEY_EXPIRE`), then disabled in its backend and destroyed after `helix.custody.destroy-after` like any retired key. A generated verifier key still without a certificate after the same time is expired the same way (`CUSTODY_KEY_EXPIRE`, resource `verifier-key`): it no longer counts toward the limit, is no longer listed and takes no CSR or certificate; it is disabled in its backend at once (`CUSTODY_KEY_DISABLE`) and destroyed there after `helix.custody.destroy-after` (`CUSTODY_KEY_DESTROY`), when its name becomes free again. So abandoned requests never lock a realm out of new keys. If your CA needs longer, raise `helix.custody.pending-key-ttl`. ### A key that can leave the backend A Vault or OpenBao key whose `exportable` or `allow_plaintext_backup` flag was set out of band (the documented policy denies HelixIAM itself both), or an HSM key whose PKCS#11 attributes say it can leave the token or was not generated on it (`CKA_EXTRACTABLE` true, `CKA_SENSITIVE` false, `CKA_NEVER_EXTRACTABLE` or `CKA_ALWAYS_SENSITIVE` false, `CKA_LOCAL` false, or not a token object), is quarantined on the server that finds it: - Every request that needs that key answers like any custody outage: **`503 temporarily_unavailable`** with `Retry-After`, never a `500` and never another key (token endpoint, SAML SSO, credential issuance, eID logins). - `keyCustody` health shows the backend **DOWN**, with a detail that names the key, for example `quarantined key(s) that can leave the backend (exportable or plaintext backup allowed) still in use: helix-acme-access- (realm acme); …`, and `/actuator/health` is DOWN (503). Readiness is not affected. - **Only that key is affected.** The backend is still reachable: its circuit breaker does not open, it still signs with its other keys, and it still makes new keys and disables the quarantined one. A backend that cannot be reached is a different condition (every request that needs it answers 503, see [Failure behaviour](#failure-behaviour)). Recover in this order, on any pod and without a restart: 1. **Close the hole first.** Vault/OpenBao: make sure the policy of every token that can write `keys//config` denies both parameters (the policy above), and find out from Vault's audit device who set the flag. A flag set to `true` can never be set back, so the key itself cannot be repaired. HSM: find out who could create or replace objects on the partition (its user PIN), and change that PIN. 2. **Replace the key.** For a realm signing key, an emergency rotation of the key's algorithm: `POST /admin/realms/{realm}/keys/rotate` with `{"emergency": true, "algorithm": ""}` (the detail names the key; `GET /admin/realms/{realm}/keys` shows its algorithm). Without `algorithm` only RS256 is rotated. For an issuer key or an eID SP key, make a new key and retire the quarantined one. Either way the quarantined key is disabled in the backend at once. 3. **Health is UP again** within the probe window (about 6 s) once no quarantined key can still sign, encrypt or decrypt (HSM: `CKA_SIGN` false, and `CKA_DECRYPT` false for an RSA key, as the rotation's disable sets them; or the object destroyed). Each pod reads the quarantined key's state from the backend, so a rotation on one pod clears every pod. Tokens are signed by the new key at once. Detection is **per pod**: each pod finds the flag on its own next use of the key or its own health check, so a few seconds of difference between pods are expected, and until a pod has found it that pod still signs with the key. Emergency rotation is the remedy. 4. The quarantined key stays refused on every pod that saw it, and the retention job destroys it after `helix.custody.destroy-after`, like any retired key. A quarantined **key-encryption key** (KEK) of a Protected realm is replaced by a KEK rotation: see [Rotating a key-encryption key](#rotating-a-key-encryption-key), "A KEK that may have left the backend". Until the rotation is done, a pod that already holds the realm's keys keeps signing with them; a pod that must unwrap a key with the quarantined KEK answers 503 for it, and health stays DOWN. ### Rotating a key-encryption key A Protected realm's signing keys are stored wrapped by the realm's KEK. A KEK rotation creates a new KEK in the realm's backend, wraps every key again under it, and then disables the old KEK. Rotate it on a schedule your policy sets, after a backend operator with access to the KEK leaves, or at once when the KEK may have left the backend (it is quarantined). Only a **Protected** realm's KEK is rotated: a High-assurance realm keeps its keys inside the backend, and a Standard realm has none (both answer `409`). Console: **Realm keys → Key custody → Rotate key-encryption key** (master-realm administrators only; the dialog asks you to type the realm id). The section then shows the progress, a pause while the backend cannot be reached, and the outcome. API, as a master-realm administrator on a master host: ```bash # start (202 with the rotation); a second call while it runs answers 200 with the same rotation curl -s -X POST https://iam.example.com/admin/realms/tenant-a/key-custody/kek-rotation -H "Authorization: Bearer $TOKEN" # progress: {"rotation": {"state": "REWRAPPING", "rowsDone": 5, "rowsTotal": 12, ...}}; any admin of the realm may read it curl -s https://iam.example.com/admin/realms/tenant-a/key-custody/kek-rotation -H "Authorization: Bearer $TOKEN" # abandon a stopped rotation that moved no key (200; 409 with a code otherwise) curl -s -X DELETE https://iam.example.com/admin/realms/tenant-a/key-custody/kek-rotation -H "Authorization: Bearer $TOKEN" ``` `GET /admin/realms/{realmId}/key-custody` also carries `backendReachable` (false only when the backend cannot be reached; a backend DOWN because of a quarantined key is reachable) and `kekQuarantined` (true when the quarantined key is this realm's KEK). The console uses them to tell an outage from a quarantined KEK, and offers the rotation for the latter. What a rotation does, in order: 1. Checks that the realm is Protected and its backend reachable (`409` with `code` `kekRotationNotProtected` or `kekRotationNoKek`, also for a stored reference that is malformed or of another realm; `409` `kekRotationDbEncryptionKek` when the realm's KEK is also `DB_ENCRYPTION_KEK`, see [`DB_ENCRYPTION` under a KEK](#db_encryption-under-a-kek-envelope); `503` when the backend does not answer; a backend that is DOWN only because the KEK is quarantined is reachable and passes). Creates the new KEK in the realm's backend (`helixkek--` on Vault and OpenBao, an AES-256 token key `helix::KEK:` on an HSM), then records the rotation and makes the new KEK the realm's KEK in one transaction: **every key made from then on is wrapped by the new KEK**. Audited `CUSTODY_KEK_ROTATION_START` and `CUSTODY_KEY_GENERATE` (resource `realm-kek`). 2. Wraps the realm's keys again, in the background, in batches of `helix.custody.kek-rotation.batch-size` (default 16): every wrapped key that is ACTIVE, PENDING or ROTATED (purposes `access`, `id`, `saml`, every algorithm). Each key is unwrapped with the KEK its row names, wrapped by the new KEK, and the new value is decrypted again before it replaces the old one, only if the row is unchanged. Tokens keep being signed throughout: every pod unwraps each key with the KEK its row names, old or new. A RETIRED key is not wrapped again; it never signs again, and destroying the old KEK makes it unreadable for good. 3. When no key needs the old KEK any more (counted again under the realm lock right before; a key that still names it sends the rotation back to step 2), **disables it** in the backend: it neither encrypts nor decrypts (Vault and OpenBao: `min_encryption_version` and `min_decryption_version` above its version; HSM: `CKA_ENCRYPT` and `CKA_DECRYPT` false). The rotation is `DONE`; audited `CUSTODY_KEK_ROTATION_COMPLETE` and `CUSTODY_KEY_DISABLE`. 4. The retention job **destroys** the old KEK `helix.custody.destroy-after` (default `P30D`) after it was disabled (`CUSTODY_KEY_DESTROY`, resource `realm-kek`), unless a key that can sign or verify names it again, or a wrapped key of the realm records no KEK (written by a server of an earlier release; it may be under it): an old KEK is never destroyed while the realm has such a key. A start whose commit reports a failure never disables the new KEK unless a fresh read shows nothing uses it (the commit may have gone through); the same holds for the KEK a failed profile switch made. While a rotation is unfinished (running, paused or stopped), the realm's custody profile cannot be changed (`409`, `code` `kekRotationRunning`; the console disables **Change profile** with the reason). **Several pods.** Every pod runs the rotation worker (every `helix.custody.kek-rotation.interval`, default `PT5S`); one pod at a time holds a rotation's lease (`helix.custody.kek-rotation.lease`, default `PT60S`, renewed before every key, so a slow backend never lets another pod take over a batch that is moving) and does the work, the others keep serving. A pod that stops mid-rotation (a crash, a rollout) leaves every key unwrappable: its lease runs out and another pod continues where it stopped. A pod that does not configure the realm's backend leaves the rotation to one that does. A pod whose entry no longer serves the realm (`helix.custody.backends[n].realms`) stops it (`FAILED`, `backendRealmNotAllowed`) without disabling anything; see [Binding a backend to realms](#binding-a-backend-to-realms). **Backend outage.** A backend that does not answer pauses the rotation: the status endpoint answers `503 temporarily_unavailable` with `Retry-After`, `code` `kekRotationPaused` and the rotation with its progress (any other 503, such as overload shedding, has no such code; the console shows "Paused" with the progress), and the rotation continues on its own once the backend answers. Keys already wrapped again and keys not yet wrapped again both keep signing, also on a pod that starts while the realm is half rotated. **A rotation that stops** (`state` `FAILED`, with `lastErrorCode` and `lastError`): `unwrapRefused` names keys whose ciphertext the old KEK does not open (they cannot sign either): retire them, then start the rotation again; `wrapRefused` or `disableRefused`: the backend refused the new KEK's encrypt (or the new value did not decrypt back to the same key) or the old KEK's disable, so check its policy, then start again; `dbEncryptionKek`: the old KEK is `DB_ENCRYPTION_KEK` and is never disabled, so move that first; `backendRealmNotAllowed`: the realm's entry no longer serves it (nothing was disabled), so put the realm back in `realms`, then start again; `newKekQuarantined`: the new KEK became quarantined during its own rotation. Starting again (console: **Resume rotation**) continues the same rotation where it stopped; after `newKekQuarantined` it creates a fresh KEK, the keys already on the quarantined one are moved once more, and the quarantined one is disabled with the old KEK at the end and destroyed with it. A stopped rotation that moved no key can be **abandoned** (`DELETE`, console **Abandon rotation**): the realm's KEK is the earlier one again, the rotation is `ABANDONED`, the new KEK is disabled; once a key is on the new KEK, abandoning is refused (`409` `kekRotationNewKekInUse`) and the rotation has to continue. Abandoning is refused too (`409` `kekRotationUnlabelledKeys`) while the realm has a wrapped key without a recorded KEK: a server of an earlier release wrote it during the rotation, possibly with the new KEK; resuming lets the rotator find its KEK and move it. Audited `CUSTODY_KEK_ROTATION_ABANDON`. **A KEK that may have left the backend** (quarantined: made exportable or plaintext-backup-able out of band, or an HSM key that can leave the token). The rotation is allowed to decrypt with it, only to move the keys off it, and never wraps with it. In this order: 1. Close the hole (policy, PIN), as for any [key that can leave the backend](#a-key-that-can-leave-the-backend). 2. Rotate the KEK and wait for `DONE`. Health is UP again within the probe window once the old KEK is disabled. 3. The wrapped keys may be known to whoever could export the KEK (with a copy of the database). **Emergency-rotate the realm's signing keys**, one request per algorithm the realm uses (see "Rotation is per algorithm" above); their new keys are wrapped by the new KEK. Then purge caches as in [Emergency rotation and caches](#emergency-rotation-and-caches). **Vault and OpenBao policy.** No new lines: the `helixkek-*` lines of the [minimal policy](#vault-and-openbao-transit) already allow creating the new KEK, encrypting and decrypting with both KEKs, disabling the old one (`keys//rotate` and `keys//config`) and destroying it. The contract tests run a full rotation with exactly that policy on Vault 2.2.1 and OpenBao 2.7.2. A KEK with the earlier name (`helix--kek-`) is rotated to a `helixkek-` KEK; once its rotation is `DONE` (and it is destroyed), that realm's `transit/encrypt|decrypt/helix--kek-*` lines are no longer needed. **HSM:** the user PIN's session creates the new AES key and sets the old one's `CKA_ENCRYPT` and `CKA_DECRYPT` to false, as an emergency rotation already does for signing keys. Metrics: `helix_custody_kek_rewrap_total{result}` (`ok`, `unavailable`, `refused`) and `helix_custody_kek_rotations_active` (running rotations, as this pod last saw them). Alert on a rotation that stays paused, and on `result="refused"`. Tested end to end on Vault 2.2.1, OpenBao 2.7.2 and SoftHSM 2.7.0: a Protected realm with keys of every purpose and family (RSA, P-256, Ed25519; ACTIVE, PENDING, ROTATED) rotated through the admin API, every key signing afterwards also on a cold pod and the old KEK disabled in the backend; on Vault also a quarantined KEK, an outage mid-rotation, two pods, a pod that stops mid-rotation, keys made while the rotation runs, and a pod whose session held a row before it was re-wrapped (on OpenBao also the quarantined KEK). A quarantined new KEK, abandoning, racing starts and an ambiguous commit are tested on PostgreSQL with an in-memory backend. Not run against a network HSM. ### Emergency rotation and caches An emergency rotation or retirement removes the compromised key from the realm's JWKS on every replica at once. Copies that relying parties, gateways and CDNs keep are a different matter: a cache that honours the JWKS headers may keep accepting tokens signed with the removed key until its copy expires. That is `max-age + stale-while-revalidate` (65 minutes by default), plus `stale-if-error` (default 5 minutes) while the server answers with errors. An attacker who holds the key gains from making the server fail, which is why `stale-if-error` is short. What HelixIAM does: for that whole window (`max-age + stale-while-revalidate + stale-if-error` after the removal) the realm's JWKS and discovery documents are sent `Cache-Control: no-cache`. A cache that fetches them in that time revalidates on every use (a cheap `304` while nothing changed) and never serves them stale. A copy cached **before** the emergency is not reached by that header. What the operator does, right after the emergency rotation: 1. **Purge the CDN or edge cache** for `/realms//oauth2/jwks` and the realm's discovery documents (`/realms//.well-known/openid-configuration`, `/realms//.well-known/oauth-authorization-server`, `/.well-known/oauth-authorization-server/realms/`, `/realms//.well-known/oauth-protected-resource`), on every host the realm is served from. 2. Ask relying parties that cache the JWKS themselves (API gateways, resource servers with a long JWKS cache) to refresh it, or restart them. 3. Check that the removed `kid` is gone: ```bash curl -s "$HELIX_URL/realms/$REALM/oauth2/jwks" | jq -r '.keys[].kid' ``` A JWKS that cannot be built on a pod (its keys cannot be read, a cold start against a failed-over database) is answered `503` with `Cache-Control: no-store` and `Retry-After`, never an empty key set that a cache would keep. ### Wallet verifier keys In the wallet provider form, **Generate in HSM/KMS** creates an ES256 or ES384 verifier key (only EC, as HAIP requires) in the realm's assigned backend: the backend the master administrator gave the realm with a profile other than Standard. Any other backend id, and a Standard realm, is refused with `400`; an unknown and an unassigned backend id get the same answer, so backend ids cannot be probed, and the console offers only the assigned backend. The key then waits for a certificate (**Waiting for a certificate → Download CSR → Upload certificate**). The certificate must be for the custody key's public key. A renewed certificate reaches the realm's store-backed wallet providers at once; a wallet provider configured in YAML (`helix.federation.wallet[*]`) needs a restart. | Method & path (under `/admin/realms/{realm}`) | Body | Purpose | | --- | --- | --- | | `POST /verifier-keys/generate` | `{name, algorithm, backendId}` | Generate an ES256 / ES384 verifier key in the realm's assigned backend (201) | | `POST /verifier-keys/{name}/csr` | `{subject, dnsNames}` | The key's CSR, signed in the backend (`{"csrPem"}`) | | `PUT /verifier-keys/{name}/certificate` | `{certificateChainPem}` | Attach the CA-issued chain | | `GET /verifier-keys` | | Keys that can sign; `?pending=true` adds keys still waiting for a certificate (expired ones are not listed) | **High assurance has no software verifier keys.** In a High-assurance realm `POST /admin/realms/{realm}/verifier-keys` (import) is refused with 400 on `privateKeyPem`, and a software key (`software:/`) no longer resolves, so a wallet connection that names one is refused (400 on `accessKeyRef`) and never signs with it; so is a dev-profile private key in a connection's settings (400 on `accessCertificatePrivateKey`). A switch rotates the realm keys into the backend itself, but it cannot replace a verifier key, whose replacement needs a CA-issued certificate. So **switching a realm to High assurance is refused** (400 on `profile`, naming each connection) while an enabled wallet connection of the realm still signs with a software key (an `accessKeyRef` of the software store, or a dev-profile `accessCertificatePrivateKey`; for the master realm also the connections in `helix.federation.wallet`). Before the switch: generate a verifier key in the backend, attach its certificate, select it in each connection, then switch. Protected keeps software verifier keys, as it keeps software signing. ### Credential issuer keys and the development CA The OpenID4VCI issuer of a realm signs credentials (SD-JWT VC, mdoc) and status lists with an ES256 issuer key (**Credential issuance → Issuer key**). Issuer keys are ES256 only, as the HAIP profile requires. | Profile | Test key (**Generate test key**) | Development CA | CSR key (**Create CSR**) | Import (**Import key**) | |---|---|---|---|---| | Standard | Software key, certified by a software development CA | Software, encrypted with `DB_ENCRYPTION` | Software key | Allowed | | Protected | Generated in the realm's HSM/KMS, certified by a development CA that also lives there | Generated in the HSM/KMS on first use | Generated in the HSM/KMS | Allowed (software) | | High assurance | **Refused** (400, code `devCaRefused`) | **Refused**; a live one blocks the switch | Generated in the HSM/KMS | **Refused** (400 on `privateKeyPem`, code `softwareKeyRefused`) | In Protected and High assurance the key is created in the realm's assigned backend, never in one a request names, and the backend signs the CSR. The server checks that the public key in the CSR, and in a certificate you attach later, is the backend key's. Such a key never exists outside the backend, so there is nothing to export. - **Development CA.** A realm's test CA certifies generated test keys; wallets and verifiers trust it only if they add its certificate, so it is not for production. In Protected, a development CA made while the realm was Standard stays a software key, and **Generate test key** is refused (409, `code: devCaSoftware`) until you retire it, because a software CA would certify a backend key. `GET /admin/realms/{realmId}/credential-issuance/keys/ca` returns `{certificatePem, custody}` (`software` or `direct`). - **Retiring the CA** (`DELETE /admin/realms/{realmId}/credential-issuance/keys/ca`) answers 204, 404 when there is no live one, and 409 while an ACTIVE or PENDING generated key still chains to it. A software CA loses its private key; the certificate stays, so credentials it certified remain recognised as the realm's own (they are never trusted as identity evidence). The next test key makes a new CA. - **Certificates and the pending cap.** A CSR key waits for its certificate: at most `helix.custody.max-pending-keys-per-realm` per realm (one more is refused with `400` on `fieldErrors.subject`), and one still without a certificate after `helix.custody.pending-key-ttl` is retired by the retention job (`CUSTODY_KEY_EXPIRE`). - **Retiring a key** (`POST .../keys/{keyId}/retire`) disables a backend key at once. The retention job destroys it after `helix.custody.destroy-after`; the row stays with an empty custody reference. The ACTIVE key cannot be retired (409): activate another key first. - **Outage.** A credential request that needs the backend while it is unreachable answers `503 temporarily_unavailable` with `Retry-After: 5`, counts `helix.issuer.credential.unavailable` and issues nothing. A status list that cannot be re-signed keeps its last token and is re-signed on the next cycle. A backend refusal is `credential_request_denied`. A wallet that retries after a 503 needs a fresh `c_nonce`. - **Wrong key.** An issuer key whose stored public key or certificate is not the backend key's is not used (the request is denied, logged), never signed with another key. API, under `/admin/realms/{realmId}/credential-issuance/keys` (a realm administrator writes; another admin with only the credential-issuance permission can read): - `GET` lists the keys (`id, kid, status, source, subject, issuer, notBefore, notAfter, fingerprint, chainPem, csrPem, createdAt, activatedAt, rotatedAt, custody, backendId`; `custody` is `software` or `direct`; never a reference or key material). The realm profile is in the `X-Helix-Custody-Profile` response header; - `POST /generate` (test key), `POST /import` `{privateKeyPem, certificateChainPem}`, `POST /csr` `{subject, dnsNames}`, `PUT /admin/realms/{realmId}/credential-issuance/keys/{keyId}/certificate` `{certificateChainPem}`, and `POST` to the same path with `/activate` or `/retire` instead of `/certificate`; - `GET /ca`, `DELETE /ca` as above. Refusals are `400` with `fieldErrors` (and `fieldErrorCodes`) for configuration, `409` for state, and `503` for an unreachable backend. Audit events: `ISSUER_KEY_CSR`, `ISSUER_KEY_GENERATE`, `ISSUER_CA_RETIRE` (written after the key's transaction committed), plus `CUSTODY_KEY_*` for the retention steps (resource `issuer-key`; the CA is `development-ca:`). **Switching a production issuer to High assurance, in order:** 1. A master-realm administrator moves the realm to **Protected** (so new issuer keys are made in the HSM/KMS). 2. **Create CSR**, have your CA sign it, and attach the chain (**Attach certificate**). 3. **Activate** the key. It rotates the previous ACTIVE key. 4. **Retire the development CA** (it must certify no ACTIVE or PENDING generated key; retire those first), and retire any software issuer key that is still PENDING. Activating the new key already rotated the previous ACTIVE one. 5. A master-realm administrator moves the realm to **High assurance**. The switch is refused (400 on `profile`) while a software issuer key or a development CA is left, and it names them. ### eID SP keys The SAML SP keys of an eID connection (`eid`, `digid`, `eherkenning`, `eidas`) can be generated in the realm's HSM/KMS backend (Protected or High assurance; never the software backend). One key signs, decrypts, or does both. DigiD/eID and eHerkenning connections commonly publish one SP certificate for both, and their documents allow it; eIDAS requires different keys and certificates for signing and encryption (eIDAS Cryptographic Requirements v1.4.1 §3.5.2), so a `SIGNING_DECRYPTION` key is refused for an eIDAS connection (400 on `usage`). A decrypting key is registered for exactly one key transport `{method, digest, mgf}` and passes a self-test at generation (encrypt 32 random bytes with the public key and exactly those parameters, decrypt them in the backend, and check that a ciphertext made with other parameters does not decrypt). A combination the backend cannot honour is refused then (400 on `keyTransport`) and the backend key is disabled, so a login never discovers it. Vault and OpenBao Transit use one hash for the OAEP digest and MGF1, so a combination with different hashes is refused on them. API (`manage-identity-providers`; for a BSN-authoritative connection such as DigiD or eIDAS, every write also needs a realm administrator, as editing its pasted keys does: 403 otherwise), under `/admin/realms/{realmId}/identity-providers/{alias}/sp-keys`: - `GET` → `[{id, usage, status, rsaBits, keyTransport, custody, backendId, subject, issuer, notAfter, fingerprint, chainPem, csrPem, createdAt, activatedAt}]` (public facts only); - `GET /algorithms` → `{rsaBits, keyTransport: [{method, digest, mgf}], signatureAlgorithms, defaultKeyTransport, separateKeys}`: the scheme table below intersected with the backend's capabilities, the scheme's recommended transport (`rsa-oaep-mgf1p` for DigiD/eID and eHerkenning, `rsa-oaep` SHA-256/SHA-256 for eIDAS) and whether it needs separate signing and decryption keys; - both carry the realm custody profile in the `X-Helix-Custody-Profile` response header; - `POST /csr` `{usage: SIGNING|DECRYPTION|SIGNING_DECRYPTION, rsaBits, keyTransport, subject, dnsNames}` → 201 with the key and its CSR (keyUsage `digitalSignature`, plus `keyEncipherment` for a decrypting key); - `PUT /admin/realms/{realmId}/identity-providers/{alias}/sp-keys/{keyId}/certificate` `{certificateChainPem}`, and `POST` to the same path with `/activate` or `/retire` instead of `/certificate`. Activating a key turns the provider's ACTIVE keys with an overlapping usage ROTATED (a rotated decrypting key keeps decrypting until it is retired) and removes the connection's pasted `spSigningPrivateKey` when the new key signs (audited `IDP_SP_KEY_REPLACED_PEM`). A pasted `spDecryptionPrivateKey` stays as the last decryption fallback during the rollover in Standard and Protected realms, because the IdP may keep encrypting to the old certificate until it refreshes the SP metadata; in a High-assurance realm it is removed too (never a software decryption key there). The ACTIVE key is retired only after its successor is activated (409); a retired key is disabled at once and destroyed by the retention job, and so is a PENDING key not activated within `helix.custody.pending-key-ttl` (with or without its certificate). More than `helix.custody.max-pending-keys-per-realm` keys waiting for a certificate are refused with 409 (`code: tooManyPending`). A key belongs to the provider instance and eID scheme it was made for. Deleting the connection, or saving it with another scheme, retires its keys (all statuses) and disables them in the backend; a connection created again under the same alias starts without keys. **Where the console shows it:** the connection's **Certificates** tab, section **Keys in HSM/KMS** (Standard realms see an explanation instead; High assurance realms do not show the pasted-key fields). What a custody key does at run time: - It signs the AuthnRequest (Redirect and POST binding) and the signed ArtifactResolve, with the scheme's algorithm: `xmldsig-more#rsa-sha256` (RS256) for DigiD, eID and eHerkenning, `xmldsig-more#sha256-rsa-MGF1` (PS256) for eIDAS. A pasted RS256 signature is byte-identical to what earlier versions produced; a pasted eIDAS key now signs PS256 as well. - It is the TLS client key of the artifact-resolution back channel (mutual TLS), in TLS 1.3 and 1.2, signing with `rsa_pss_rsae_sha256` or `rsa_pkcs1_sha256` as the backend allows. The private key never leaves the backend. - It decrypts the `EncryptedKey` of an assertion (see [OAEP key transport on each backend](#oaep-key-transport-on-each-backend)), with exactly its registered transport. A pasted decryption key accepts only the key transports of its scheme's table (below) and the data encryption algorithms AES-CBC and AES-GCM; `rsa-1_5` is refused for it as well. - **Fail closed.** When the backend cannot be reached, the login start and the callback answer `503 temporarily_unavailable` with `Retry-After`. The pasted key is never tried as a fallback during an outage. A pasted signing key is used only while no custody signing key is ACTIVE, and never in a High-assurance realm. **Pasted keys.** Standard and Protected realms can still paste PEM keys (`spSigningPrivateKey`, `spDecryptionPrivateKey`). A pasted key must be an unencrypted PKCS#8 RSA key (`BEGIN PRIVATE KEY`; code `notPkcs8Key`; an EC key is `notRsaKey`), at least the scheme's minimum size (2048, eIDAS 3072; `rsaKeyTooSmall`), and for eIDAS not the same key for signing and decryption (`sameKeyForSigningAndDecryption`). A PEM for a usage that an ACTIVE custody key of the same provider already covers is refused (`coveredByCustodyKey`). A High-assurance realm accepts none (`pastedKeyInHighAssurance`), and a realm import runs the same rules. Clear a pasted key with `clearSpSigningPrivateKey` or `clearSpDecryptionPrivateKey` set to `true`. **Switching an eID connection to High assurance, in order:** 1. A master-realm administrator moves the realm to **Protected**. 2. On the connection's **Certificates** tab, make the signing key and the decryption key (**New key in HSM/KMS**), have the CA sign each CSR, **Upload certificate** and **Activate**. The scheme's recommended key transport is preselected only when the backend can decrypt it; an eIDAS connection needs two keys. 3. Activating the signing key removes the pasted signing key. The pasted decryption key stays as the last fallback in Standard and Protected while the identity provider may still encrypt to the old certificate, so clear it (`clearSpDecryptionPrivateKey`) once the identity provider has the new SP metadata. 4. A master-realm administrator moves the realm to **High assurance**. The switch is refused (400 on `profile`) while an enabled connection still stores a pasted key, or the backend cannot serve the connection (no RSA size at the scheme's minimum, none of the scheme's signature algorithms, no key transport it decrypts). #### eID scheme algorithms What each scheme's own documents require of an SP key (read 2026-10-29; the same table, with each source as a field, is `EidSchemeAlgorithms` in the server). No scheme requires `xmlenc#rsa-1_5` (PKCS#1 v1.5 key transport), which is never accepted. HelixIAM signs `xmldsig-more#rsa-sha256` (RS256) and `xmldsig-more#sha256-rsa-MGF1` (PS256) with custody keys; other signature algorithms a scheme lists are not offered. | Scheme | Key transport (method; `ds:DigestMethod`; MGF) | XML signature | Data encryption | Min. RSA | Separate keys | Source | |---|---|---|---|---|---|---| | DigiD / eID (`digid`, `eid`) | `xmlenc#rsa-oaep-mgf1p` (SHA-1, MGF1-SHA-1); `xmlenc11#rsa-oaep` with MGF1-SHA-1 (`xmlenc11#mgf1sha1`) and a SHA-1/256/384/512 digest; default `rsa-oaep-mgf1p` | `rsa-sha256` (also `rsa-sha384`, `rsa-sha512`) | `xmlenc#aes256-cbc` | 2048 (PKIoverheid) | no | [Logius Stelsel Toegang SAML 1.0.0](https://gitdocumentatie.logius.nl/publicatie/st/saml/1.0.0/) §9.1, §9.3, example EncryptedID (derived from eID SAML 4.4/4.5) | | eHerkenning | `xmlenc#rsa-oaep-mgf1p`, or `xmlenc11#rsa-oaep` with SHA-1 digest and MGF1-SHA-1 ("OAEP padding and a SHA-1 digest"); default `rsa-oaep-mgf1p` | `rsa-sha256` | XML Encryption 1.1 (no URI named) | 2048 (PKIoverheid) | no | Afsprakenstelsel eToegang AS1.24e: [Encryption](https://afsprakenstelsel.etoegang.nl/Startpagina/as/encryption), [Message Signing](https://afsprakenstelsel.etoegang.nl/Startpagina/as/digital-signature) | | eIDAS | `xmlenc11#rsa-oaep` only; digest and MGF each SHA-256, SHA-384 or SHA-512 (SHA-1 and `rsa-oaep-mgf1p` refused); default SHA-256/SHA-256 | `sha256/384/512-rsa-MGF1` (RSASSA-PSS) or ECDSA; no `rsa-sha256` | `xmlenc11#aes128-gcm`, `aes256-gcm` (`aes192-gcm` MAY) | 3072 | yes (§3.5.2) | [eIDAS Cryptographic Requirements v1.4.1](https://ec.europa.eu/digital-building-blocks/sites/download/attachments/467109280/eIDAS%20Cryptographic%20Requirement%20v.1.4.1_final.pdf?version=2&modificationDate=1729176493701&api=v2) §3.1.1, §3.2.1, §3.2.2.1, §3.3.1, §3.5.2 | The legacy DigiD SAML 3.5 interface (Logius, 6 May 2020, §6.1 and §6.3) signs RSA-SHA256 and encrypts nothing; it is a subset of the eID row. eID SAML 4.4 (§9.3) says the SHA-1 MGF "SHOULD NOT be used", while its successor Stelsel Toegang SAML 1.0.0 says it "MUST be used" and its example uses `rsa-oaep-mgf1p`; the table follows the current document. ## SAML metadata signing A realm's SAML IdP metadata (`/realms//saml/idp/metadata`, anonymous) can be signed with the realm's SAML signing key. **It is off by default**, and while it is off the document is exactly the one earlier versions served. - **Setting.** Console: **SAML clients**, block **Realm-wide: sign IdP metadata** at the top of the page (switch **Sign metadata**, field **Validity (hours)**, button **Save for the whole realm**). API, with the `manage-clients` permission (`view-clients` reads): `GET` and `PUT /admin/realms/{realmId}/saml-idp/metadata-settings` with `{"signed": true, "validityHours": 168}`. Both fields are required; the validity is 1 to 720 hours (default shown for a realm that never set it: 168). The answer also carries `lastError` (`{code, at}` or null): the last signing failure **on the server that answered**, other servers may differ. Every write is audited as `SAML_METADATA_SETTINGS`. - **What is signed.** An enveloped XML signature over the `EntityDescriptor` (exclusive canonicalisation, SHA-256): `rsa-sha256` for an RSA SAML key, `ecdsa-sha256` for an EC key. The document gets a fresh `ID`, `validUntil` = signing time + the validity, and `cacheDuration` = `PT1H`. A signed copy is cached per realm, made by one request at a time, and re-signed after half its validity, when the document changes (a rotation, new endpoints) and when the realm's keys or this setting change. The anonymous endpoint never makes a backend call per request. - **Turning it on is checked.** `PUT` with `signed: true` makes one probe signature first. If the key cannot sign (a refusal, or no key or certificate) the answer is `400` on `fieldErrors.signed` and nothing is stored; an unreachable backend is `503`. - **Fail closed.** If a new signature cannot be made, the cached copy is served while it is within its `validUntil` **and** its signing certificate is still published by the realm; otherwise the endpoint answers `503 temporarily_unavailable` with `Retry-After` (the time until the backend is tried again). Unsigned XML is never served while signing is on. After an emergency rotation or retirement that removed the signing key, no copy signed by that key is served. - **Metrics:** `helix_saml_metadata_stale_served_total` (a cached copy served because re-signing failed) and `helix_saml_metadata_sign_failures_total{reason}` (`refused`, `unavailable`, `error`). - **Service providers must pin the signing certificate.** A signature only proves anything against a certificate the service provider already trusts; the one inside the document proves nothing. Give each service provider the realm's SAML signing certificate out of band and have it verify the metadata signature and `validUntil` against that pin. The metadata lists the next key's certificate before it signs (state PENDING), so a provider that verified a copy can add it to its pins before a rotation takes effect. Every service provider must re-read the metadata after the setting changes. - **The `encryption` KeyDescriptor is gone for a realm whose SAML key is in the HSM/KMS (High assurance).** A key in a backend signs only, so the metadata no longer offers it for encryption. Protected realms (software keys, unwrapped in memory) keep it. ## High-assurance rules for the issuer, eID and metadata keys | Rule | Standard | Protected | High assurance | |---|---|---|---| | Issuer key (CSR, Generate test key) | Software | Generated in the realm's backend | Generated in the realm's backend (CSR only) | | Development CA | Software | Generated in the realm's backend | **Refused**; a live one blocks the switch | | Importing a software issuer key | Allowed | Allowed | **Refused** | | eID SP keys in the backend | Refused (no custody) | Allowed | Allowed | | Pasted eID SP PEM keys | Allowed | Allowed | **Refused**; an enabled connection that still stores one blocks the switch | | Software decryption key | n/a | A pasted decryption key stays as the last fallback | **Never** | - **No request names a backend.** New keys are always made in the backend the master assigned to the realm (`realm_custody_settings`), so a tenant cannot place a key in another tenant's partition. - **Realm isolation.** A stored key reference resolves only in its own realm, and an eID SP key only for its own provider alias, instance and scheme. A reference that fails the check is treated as no key, never as a fallback. - **Fail closed.** A custody failure never falls back to a software key, a pasted PEM key, another purpose or another algorithm. A request that needs an unavailable backend is `503 temporarily_unavailable` with `Retry-After`; a refusal by the backend is `400` (configuration) or `409` (state). Readiness does not include backend health. - **Switching to High assurance** is refused (400 on `fieldErrors.profile`, naming each item) while any software issuer key, development CA or pasted eID key exists, and when the backend cannot serve the issuer or an enabled eID connection. The steps are in [Credential issuer keys](#credential-issuer-keys-and-the-development-ca) and [eID SP keys](#eid-sp-keys). ## `DB_ENCRYPTION` under a KEK (envelope) `DB_ENCRYPTION` encrypts the database columns (and the software keys of Standard realms). Instead of the plain key you can give the server the key **wrapped** by a KEK in a backend: | Variable | Meaning | |---|---| | `DB_ENCRYPTION_WRAPPED` | The data key, wrapped (Base64), as `DbKeyWrapTool` prints it. | | `DB_ENCRYPTION_KEK` | The KEK's key reference, e.g. `vault:vault-1/master/helixdbkek-master-#1` or `pkcs11:hsm-1/master/helix:master:DBKEK:`. | **`DB_ENCRYPTION_KEK` is a key of its own, never a realm's KEK.** A realm's KEK is rotated through the admin API, and a rotation disables the old KEK and later destroys it; a data key wrapped by it would be lost, and with it every encrypted column. So its names differ from a realm KEK's (`helixdbkek--` on Vault and OpenBao, label `helix::DBKEK:` on an HSM), the tool creates it that way and refuses to wrap a data key with a key named like a realm KEK, and a realm KEK rotation refuses (`409`, `code` `kekRotationDbEncryptionKek`) a realm whose KEK is the configured `DB_ENCRYPTION_KEK` (any version of it) or has a `DB_ENCRYPTION_KEK` name, and never disables or destroys it. A data key still wrapped by a key with a realm-KEK name (an earlier guide used `helix-master-kek-`, then `helixkek-master-`) keeps working; the server warns at start-up: move it with `--rewrap-from` (below). At start-up the server builds only the referenced backend, calls its decrypt once (associated data `db-key`) and keeps the data key in memory only. Start-up **fails** when the unwrap fails, when only one of the two is set, or when `DB_ENCRYPTION`, or the same setting from a properties file or a `DATABASE_ENCRYPTION` variable, is set as well (they are mutually exclusive). A `software:` KEK is refused by name: the software backend keeps no key-encryption keys. Messages name the KEK reference and the backend, never the key or the wrapped value. Chart: `dbEncryption.wrapped.secret` (`{name, key}`) and `dbEncryption.wrapped.kekRef`, with `secrets.dbEncryptionKey: ""`. The tool in the server image (`io.helixiam.persistence.security.DbKeyWrapTool`) takes the backend settings as `-D` properties and never prints a data key (exit 0 ok, 1 failure, 2 usage). Run it with credentials that may create keys (the tool's policy is in [Vault and OpenBao Transit](#vault-and-openbao-transit); the server's own token only reads and decrypts). Create the KEK, then wrap an existing key read from stdin (no echo on a terminal), or generate one: ```bash TOOL="java -Dhelix.custody.backends[0].id=vault-1 -Dhelix.custody.backends[0].type=vault \ -Dhelix.custody.backends[0].settings.address=https://vault.example:8200 \ -Dhelix.custody.backends[0].settings.auth.kind=approle \ -Dhelix.custody.backends[0].settings.auth.role=$ROLE_ID \ -Dhelix.custody.backends[0].settings.auth.secretIdFile=/run/secrets/secret-id \ -cp /app/app.jar io.helixiam.persistence.security.DbKeyWrapTool" KEK_REF=$($TOOL --create-kek vault-1) # prints e.g. vault:vault-1/master/helixdbkek-master-#1 printf %s "$CURRENT_DB_ENCRYPTION" | $TOOL --kek "$KEK_REF" # --generate instead of stdin: a new random 32-character key (only for a new installation: existing data needs the old key) ``` On an HSM the backend is `type=pkcs11` with its settings (`settings.library`, `settings.tokenLabel`, `settings.pinFile`, as for the server); `--create-kek hsm-1` makes a sensitive, non-extractable AES-256 token key labelled `helix:master:DBKEK:` and prints `pkcs11:hsm-1/master/helix:master:DBKEK:`. Checked end to end against a Vault 2.2.1 dev server with token auth and the minimal policy: the tool wrapped a key without printing it, a server started with `DB_ENCRYPTION_WRAPPED` + `DB_ENCRYPTION_KEK` became ready (the data key appears nowhere in its log), and a server with both the plain and the wrapped key refused to start. The tool's create, wrap, rewrap and disable are also tested against a SoftHSM token. **Rotating `DB_ENCRYPTION_KEK`.** This KEK is the operator's (it is not in the database), so it is rotated with the tool, not the admin API. The data key and the data stay as they are; only the wrapped value changes: 1. Create the new KEK: `NEW_KEK_REF=$($TOOL --create-kek vault-1)` (or `hsm-1`). 2. Move the wrapped value to it. The tool reads the current `DB_ENCRYPTION_WRAPPED` on stdin, unwraps it with the old KEK (also a quarantined one, or one with a realm-KEK name), wraps the data key under the new KEK, checks that the new value unwraps to the same key, and prints it: ```bash printf %s "$DB_ENCRYPTION_WRAPPED" | $TOOL --rewrap-from "$OLD_KEK_REF" --kek "$NEW_KEK_REF" ``` 3. Put the printed value in the `DB_ENCRYPTION_WRAPPED` secret and the new reference in `DB_ENCRYPTION_KEK` (`dbEncryption.wrapped.kekRef`), and roll the pods; each becomes ready once it unwrapped the data key. 4. When every pod runs with the new value, disable the old KEK: `$TOOL --disable "$OLD_KEK_REF"`. It then neither encrypts nor decrypts (Vault and OpenBao: a new version is made when the referenced one is the latest, and both minimum versions are raised above the referenced version; HSM: `CKA_ENCRYPT` and `CKA_DECRYPT` false). The tool disables only a key with a `DB_ENCRYPTION_KEK` name (`helixdbkek-`, `:DBKEK:`; any other key may be a realm's), and never the one the environment's `DB_ENCRYPTION_KEK` names (the next pod would not start). An old KEK with a realm-KEK name (`helix-master-kek-`, `helixkek-master-`, HSM `helix:master:KEK:`) is disabled by hand, after checking that no realm uses it: **Realm keys → Key custody** of each realm in Protected or High assurance (the master realm too) must not show it as the realm's key-encryption key, and no KEK rotation may list it. Then: - Vault and OpenBao: `vault write -f transit/keys//rotate` if its version is the latest, then `vault write transit/keys//config min_decryption_version= min_encryption_version=`. - HSM: with the HSM's own administration tool, as the user that owns the partition, set `CKA_ENCRYPT` and `CKA_DECRYPT` to false on the secret key object labelled `helix:master:KEK:` (`C_SetAttributeValue`, both `CK_FALSE`). This is what HelixIAM's own disable of a KEK does, and a KEK with `CKA_DECRYPT` false no longer opens a wrapped value (tested on SoftHSM). Do not destroy it yet. Delete it after your retention period (Vault: `deletion_allowed=true`, then `vault delete transit/keys/`; HSM: destroy the object). Checked against a Vault 2.2.1 dev server and a SoftHSM token: the moved value unwraps with the new KEK through the server's start-up path, and the disabled old KEK no longer opens the old value. Rotating the data key itself (re-encrypting the data) is a separate, online procedure: see [Rotating the database encryption key](#rotating-the-database-encryption-key). A plain `DB_ENCRYPTION` that looks weak (fewer than 16 distinct characters, a known demo value) is reported as an insecure setting at start-up and in `helix_db_encryption_weak_key` (1); it is a warning, not a refusal. A new `DB_ENCRYPTION_NEXT` has no such excuse: a weak or short one (under 32 characters) refuses start-up. ## Rotating the database encryption key `DB_ENCRYPTION` is the data key: it encrypts sensitive database columns, the software signing keys of Standard realms and the values derived from it (the wallet evidence seal, the profile provenance and the trusted-list integrity check). From rc.13 you can replace it **online**, with no downtime: a second key (`DB_ENCRYPTION_NEXT`) is added beside the current one, a master-realm administrator starts the rotation, and every replica rewrites the stored values under the new key in the background while all replicas keep serving. This is the data key; the realm key-encryption keys have their own rotation ([Rotating a key-encryption key](#rotating-a-key-encryption-key)) and `DB_ENCRYPTION_KEK` is rotated with the tool ([`DB_ENCRYPTION` under a KEK](#db_encryption-under-a-kek-envelope)). **Before you start.** - **Every pod must run rc.13 or later.** An rc.12 pod writes no heartbeat, so the start cannot see it, and it cannot read the key-tagged values a rotation writes. Finish the upgrade first. **Once a rotation has started, rolling back to a release before rc.13 is not supported** (its values are key-tagged `hx2..`). Until the first rotation, values keep the format of rc.12, so a mixed rollout and a rollback still work. - Take a database backup, and keep the old key until the console says it can go (below). - Only administrators of the **master realm** can see or change it (`403` otherwise). **The steps.** 1. **Make the new key.** A plain key is at least 32 characters and passes the same weak-key check as `DB_ENCRYPTION` (a weak or short one **refuses start-up**; this is a new key, so there is no legacy excuse): `openssl rand -base64 32`. Or a wrapped key, as for `DB_ENCRYPTION_WRAPPED`: wrap the new key with the tool and the same or another KEK (`printf %s "$NEW_KEY" | $TOOL --kek "$KEK_REF"`); the key must be 16, 24 or 32 bytes. 2. **Configure it as `DB_ENCRYPTION_NEXT` on every replica** and roll the pods. Either `DB_ENCRYPTION_NEXT` (plain) or `DB_ENCRYPTION_NEXT_WRAPPED` together with `DB_ENCRYPTION_NEXT_KEK` (default: `DB_ENCRYPTION_KEK`); both at once, or a NEXT KEK without a wrapped key, refuses start-up. Chart: `dbEncryption.next.secret` (`{name, key}`, plain) or `dbEncryption.next.wrapped` (`secret` and `kekRef`; an empty `kekRef` uses `dbEncryption.wrapped.kekRef`), not both. `DB_ENCRYPTION` stays as it is: every replica now reads with either key and still writes under the current one. 3. **Start the rotation.** Console: the master realm's **Realm keys** page, section **Database encryption**, which shows the current key, the new key (by id: 16 characters derived from the key, never the key) and how many replicas hold it; **Start rotation** is offered when every live replica has it. API: `POST /admin/database-encryption/rotation` (optional body `{"toKeyId": ""}`; `202` with the rotation, `200` with the running one if called again). The start is refused with `409` and a `code`: `noNextKey`, `replicaMissingNextKey` (it gives the counts, never a host name), `nextKeyChanged`, `rotationFailed` (retry the stopped one instead). From the start, every replica **writes under the new key**; a leased background resealer (one replica at a time, a crash is picked up by another) rewrites the existing values in bounded batches, resumable from a cursor, and never holds a database connection while it opens and seals a value. A row that a concurrent writer changed is looked at again. 4. **Wait for `DRAINING`, then `DONE`.** When no row is left, the rotation waits for the values it does not rewrite to expire: short-lived flow state and wallet transactions (`helix.db-encryption.rotation.drain`, default `PT30M`: it must be at least the longest lifetime of those values, so raise it if you extended flow-state or wallet-transaction lifetimes beyond 30 minutes) and **sessions that were created before the start**. Then it counts every table once more (back to rewriting if a value appeared), waits until every live replica writes under the new key, and records `DONE`. **With the default Redis session store (`HELIX_SESSION_STORE=redis`) a rotation therefore stays `DRAINING` for at least the longest absolute session lifetime of any realm (`sso_session_max_lifetime_seconds`, 36000 s = about 10 hours by default), and both keys are required for all of it.** With JDBC sessions it waits for the last live session created before the start. With the **Redis token store** (`HELIX_TOKEN_STORE=redis`, default `queue`) authorizations are not rewritten: keep the old key until `oldKeyNeededUntil` (the start plus the longest refresh-token lifetime), which `GET /admin/database-encryption` and the console show. 5. **Swap the keys, in this order.** When the status is `DONE` and `oldKeyNeededUntil` is empty or past: (a) set `DB_ENCRYPTION` to the new key and remove `DB_ENCRYPTION_NEXT` on every replica, (b) restart them, (c) **only then** remove the old key from wherever you keep it. Do not remove the old key before (b): until then the replicas still hold it as `DB_ENCRYPTION`. A server that lacks a key the database still needs **refuses to start**, naming key ids, states and counts (never a key): a rotation that is not `DONE` needs both keys, and so does a value tagged with a key the server does not hold. **Pause, resume, retry.** `POST .../rotation/pause` stops the rewrite (and the waiting) until `POST .../rotation/resume`; nothing is lost, and the old key stays needed. A rotation that **stopped** (`FAILED`) is continued from its cursor with `POST .../rotation/retry`; a stopped rotation blocks a new start (`409 rotationFailed`) until then. It stops, naming the table, column and row id and never a value, when a stored value no configured key reads (`unreadableRow`: make sure the old key is still configured on every replica), when it makes no progress (`resealStuck`) or when the database refuses a write (`writeRefused`). A state the call cannot change is `409` with `noRotation`, `rotationNotRunning`, `rotationNotPaused` or `rotationNotFailed`. All of these are master-realm only and audited. **Reading the state.** `GET /admin/database-encryption` answers `writeKeyId`, `keyIds`, `nextKeyPresent`, `nextKeyId`, `replicas` (`live`, `withNextKey`, `onWriteKey`) and `rotation` (`state` `RESEALING`, `DRAINING`, `DONE` or `FAILED`, `paused`, `targetsTotal`/`targetsDone`, `rowsDone`, `drainUntil`, `lastError` and `lastErrorCode`, `oldKeyNeededUntil`; `null` when there has been none). While `DRAINING`, `lastErrorCode` is `null` until `drainUntil` has passed; after that it says what is still awaited: `ephemeralValuesLive`, `sessionsLive` or `replicasNotOnWriteKey`. Key ids and counts only. **Readiness.** The `dataKey` health contributor is part of the readiness group. A replica is **not ready** when it does not hold the key the database writes under (it could not read new values), and, **while a rotation is open**, when its heartbeat (`helix.db-encryption.heartbeat`, default `PT10S`) has not been written for three intervals (the rotation can no longer see it, so it must not keep writing). With no rotation open, or when that read fails or takes longer than 2 s (the database is down: a shared dependency), a replica stays ready. This is a condition of the pod's own data keys, not backend health, so the rule that readiness never includes backend health does not apply to it. Metrics: `helix_db_data_key_reseal_total{result}` (`ok`, `changed`, `unreadable`) and `helix_db_data_key_rotation_active`. Audit: `DATA_KEY_ROTATION_START`, `_COMPLETE`, `_FAILED`, `_PAUSE`, `_RESUME` and `_RETRY` (key ids and counts only). Settings: `helix.db-encryption.rotation.batch-size` (default 100, 1 to 1000), `.lease` (`PT60S`, at least `PT1S`), `.interval` (`PT5S`), `.drain` (`PT30M`), `.enabled` (`true`: whether this replica runs the resealer; the API still starts rotations) and `helix.db-encryption.heartbeat` (`PT10S`). ## Failure behaviour **Fail closed, always.** A request that needs a backend that does not answer is never signed with a software key, another purpose's key or RS256. - **Token, SAML and admin requests** that need an unavailable backend answer **`503`** with `Retry-After` and the OAuth error `temporarily_unavailable` (token endpoint and other protocol endpoints: `HELIX_OVERLOAD_RETRY_AFTER_SECONDS`, default 2 s; custody admin endpoints: 5 s). So does a key that is quarantined because it can leave the backend ([A key that can leave the backend](#a-key-that-can-leave-the-backend)), and a token (any signed JWT) that custody refuses to sign, such as a custody realm with no ACTIVE key of the requested algorithm: a server-side condition, never a `500`. A refusal on an admin endpoint (a disabled key, an algorithm the backend cannot sign) is a `400` or `409`, never a 503. - **Time bound:** each call is given up after the backend's `opTimeout` (default 2 s; `helix.custody.op-timeout` for a backend without one). - **Saturation shedding:** each pod runs at most `helix.custody.max-concurrent-calls` (default 64) calls in flight per remote backend, with no queue. A call beyond that is shed at once as a 503. Shedding does not open the circuit breaker. - **Circuit breaker** per remote backend and pod: 5 consecutive unavailable results within 10 s open it; while open, calls fail at once (503) without reaching the backend; after 5 s one trial call goes through, and its success closes it. Refusals and quarantined keys never count. State: `helix_custody_breaker_open{backend}`. The built-in `software` backend has no breaker (its gauge stays 0): it serves every realm, so a Protected realm whose KEK backend is down fails on its own and never sheds the Standard realms. - **Issuer, eID and metadata keys during an outage.** A credential request answers 503 and issues nothing (status lists keep their last token and are re-signed later); an eID login start or callback answers 503, with no fallback to a pasted key; signed SAML metadata answers 503 once no still-valid copy can be served. Details are in each key's section above. - **What keeps working during an outage:** discovery, JWKS and unsigned SAML metadata (certificates only), every Standard realm, and every Protected realm on a pod that already holds its unwrapped keys (they stay in memory with no time-based eviction; they leave only on rotation or retirement). High-assurance realms answer 503 for anything they must sign. No pod restarts. - **Readiness does not include backend health** (one backend serves every pod; including it would turn a backend blip into a total outage). One exception, per pod: a pod that **starts** and cannot unwrap the wrapped keys of its realms because the backend does not answer stays **not ready**, retries every `helix.custody.unwrap-retry` (default `PT10S`) and becomes ready when the unwrap succeeds; liveness stays up, and discovery and JWKS already answer. A definite refusal for one key (no KEK on this pod, a ciphertext that does not verify) does not hold readiness: that key fails closed, is counted in `helix_custody_unusable_keys{realm}` and audited once as `CUSTODY_KEY_UNAVAILABLE`. An ACTIVE key whose custody reference cannot be used on this pod (its backend is not configured here) is published but cannot sign, and is audited the same way. - **Aggregate health:** the `keyCustody` component of `/actuator/health` (management port) lists each backend (UP or DOWN, detail, latency, breaker state). Any backend DOWN makes the aggregate `/actuator/health` DOWN (503), while `/actuator/health/readiness` and `/actuator/health/liveness` stay UP. Probes are bounded to 1 s, one in flight per backend, and cached for 5 s, so the aggregate shows an outage up to about 6 s late. A backend whose realm binding is broken is DOWN too, with `realmBinding` naming each realm and why (`notAllowed`, `notDedicated`); see [Dedicated key stores](#dedicated-key-stores). - **A key of a realm its backend does not serve** (`helix.custody.backends[n].realms`) is never used: `503` on protocol endpoints, `409 backendRealmNotAllowed` on custody admin endpoints, `400` where a new key would be made. ## Metrics, audit, console | Metric | Meaning | | --- | --- | | `helix_custody_ops_total{backend,op,result}` | Calls per backend; `op` = sign, encrypt, decrypt, decrypt_key (an RSA-OAEP key-transport decryption of an eID assertion), generate, disable, destroy; `result` = ok, unavailable, refused, or `contract` (a plugin threw something other than the three exceptions of the SPI: counted as unavailable and against the circuit breaker). | | `helix_custody_op_seconds{backend,op}` | Latency of the calls that reached the backend (histogram, 5 ms to 5 s buckets). | | `helix_custody_breaker_open{backend}` | 1 while the breaker sheds calls. | | `helix_custody_unusable_keys{realm}` | Wrapped signing keys this pod cannot unwrap (they fail closed). | | `helix_db_encryption_weak_key` | 1 when the plain `DB_ENCRYPTION` looks weak. | | `helix_custody_kek_rewrap_total{result}` | Keys wrapped again by a KEK rotation (`ok`), paused by an unreachable backend (`unavailable`) or refused by their KEK (`refused`). | | `helix_custody_kek_rotations_active` | KEK rotations running (wrapping keys again or disabling the old KEK), as this pod last saw them. | | `helix_db_data_key_reseal_total{result}` | Values rewritten under the new data key by a data-key rotation (`ok`), changed by another writer while they were rewritten and looked at again (`changed`), or read by no configured key (`unreadable`: the rotation stops). | | `helix_db_data_key_rotation_active` | 1 while a data-key rotation is rewriting or draining, as this pod last saw it. | | `helix_issuer_credential_unavailable_total` | Credential requests answered 503 because the issuer key's backend was unreachable (Micrometer name `helix.issuer.credential.unavailable`). | | `helix_saml_metadata_stale_served_total` | Signed SAML metadata served from the cached copy because re-signing failed. | | `helix_saml_metadata_sign_failures_total{reason}` | Failed metadata signatures: `refused`, `unavailable` or `error`. | Alert on `helix_custody_breaker_open == 1`, on a rising `result="unavailable"` rate, on `helix_custody_unusable_keys > 0`, and on the `keyCustody` health component. See also [Observability & health](observability.md). Audit event types (details carry realm, purpose, algorithm, custody kind, backend and key reference; never key material, PINs or tokens): `CUSTODY_PROFILE_CHANGE`, `CUSTODY_KEY_GENERATE`, `CUSTODY_KEY_DISABLE` (reason `retired` or `orphan`, or by the retention job), `CUSTODY_KEY_DESTROY`, `CUSTODY_KEY_EXPIRE`, `CUSTODY_KEY_UNAVAILABLE`, `VERIFIER_KEY_GENERATE`, `VERIFIER_KEY_CSR`, `VERIFIER_KEY_CERTIFICATE`. This release adds `ISSUER_KEY_CSR`, `ISSUER_KEY_GENERATE`, `ISSUER_CA_RETIRE`, `IDP_SP_KEY_CSR`, `IDP_SP_KEY_CERTIFICATE`, `IDP_SP_KEY_ACTIVATE`, `IDP_SP_KEY_RETIRE`, `IDP_SP_KEY_REPLACED_PEM` (a pasted eID key removed by an activation) and `SAML_METADATA_SETTINGS`. A KEK rotation writes `CUSTODY_KEK_ROTATION_START` (also a refused start, and a resume), `CUSTODY_KEK_ROTATION_COMPLETE`, `CUSTODY_KEK_ROTATION_FAILED` and `CUSTODY_KEK_ROTATION_ABANDON` (also a refused one), and `CUSTODY_KEY_GENERATE`, `CUSTODY_KEY_DISABLE` and `CUSTODY_KEY_DESTROY` with the resource `realm-kek`; the retention steps of the new keys use `CUSTODY_KEY_*` with the resources `issuer-key` and `federation-sp-key`. Phase 3 adds `CUSTODY_KEY_STORE_POLICY` (`from`, `to`) and, for the data key, `DATA_KEY_ROTATION_START`, `DATA_KEY_ROTATION_COMPLETE`, `DATA_KEY_ROTATION_FAILED`, `DATA_KEY_ROTATION_PAUSE`, `DATA_KEY_ROTATION_RESUME` and `DATA_KEY_ROTATION_RETRY` (key ids and counts, never a key or a value). Signatures are counted in the metrics, not audited one by one. Console (EN and NL): **Realm keys → Key custody** shows the profile, the backend, its health and latency, and the keys per purpose with their custody (Software, Wrapped by HSM/KMS, Direct (HSM/KMS)); key references are shown as technical details only. For a Protected realm it also shows the key-encryption key's rotation and **Rotate key-encryption key**. The CSR form has a purpose choice. The wallet provider form generates verifier keys in a backend. Phase 3 adds the key-store picker (grouped as sovereign key stores, cloud providers and partner plugins, with the reason a store is not offered) and the **sovereign key stores only** setting on that page, and, on the **master** realm's page only, the installed plugins and the **Database encryption** section (the rotation above). Only master-realm administrators see or change these. This release adds **Credential issuance → Issuer key** (custody column, **Create CSR**, **Retire development CA**), the **Keys in HSM/KMS** section of an eID connection's **Certificates** tab, and **Realm-wide: sign IdP metadata** on the **SAML clients** page. They all say "HSM/KMS", never a product name, and show key references nowhere. ## Sizing Measured with HelixIAM's fixed token benchmark on one machine, one mode after the other: - **Machine:** Apple M5 Max, Docker Desktop VM (aarch64, 18 CPUs), server image built from the key-custody branch. - **Server:** one container, `--cpus=2` pinned to two CPUs, 1536 MiB; PostgreSQL, Redis and the load generator on their own CPUs; the Vault dev container (2.2.1, in-memory storage, no TLS, no audit device) on two further CPUs with no other limit. SoftHSM 2.6.1 runs inside the server container (a benchmark-only image layer), so its CPU counts as server CPU. - **Load:** `client_credentials` on one confidential client, one run per mode: fixed rates 500, 1,000 and 1,500 per second for 20 s, then a ramp to the highest rate with p95 under 50 ms and errors (503s included) under 1 %. CPU per token is server-container CPU time per successful token at 1,000 per second. - Single runs on a shared developer machine: treat differences under about 20 % as noise. Measured 2026-10-28: | Profile | Backend | Realm key | Server CPU per token at 1,000/s | Vault CPU per token | p95 at 1,000/s | Highest sustainable rate | | --- | --- | --- | --- | --- | --- | --- | | Standard | software | ES256 | 0.434 ms (first run 0.456 ms) | n/a | 0.8 ms | 8,519/s (first run 2,070/s: one ramp step at 2,160/s had 1.03 % 503s; the rerun is the figure) | | Standard | software | EdDSA | 0.370 ms | n/a | 0.9 ms | 8,906/s | | Standard | software | RS256 | 0.787 ms | n/a | 1.6 ms | 2,851/s | | Protected | Vault | ES256 | 0.461 ms | 0.002 ms | 1.3 ms | 8,132/s | | High assurance | Vault | ES256 | 0.534 ms | 0.360 ms | 1.9 ms | at least 3,222/s (the ramp's step limit; not saturated) | | High assurance | Vault | RS256 | 0.475 ms | 0.952 ms | 2.8 ms | 2,050/s (Vault's RSA signing is the limit) | | High assurance | SoftHSM (PKCS#11) | ES256 | 0.433 ms | n/a (SoftHSM is in the server) | 0.8 ms | at least 3,222/s (the ramp's step limit; not saturated) | | High assurance | SoftHSM (PKCS#11) | RS256 | 0.816 ms | n/a (SoftHSM is in the server) | 1.6 ms | 2,573/s | For comparison, a server image from just before key custody was added, on the same machine right after: 0.515 ms and 5,153/s (ES256, Standard, one run). Key custody does not make Standard realms slower. The High-assurance ramps started at 300/s and stopped at their last step (3,222/s), so for ES256 the backend's real ceiling is higher than measured. What this means: - **Protected costs nothing at signing time:** the keys sign in memory, as in Standard. The backend is used only to create the KEK, to wrap a new key and to unwrap keys when a pod starts. - **High assurance moves every signature to the backend.** Its ceiling is the backend's signing rate divided by the signatures per request (one per access token; an authorization-code flow with an ID token signs twice), plus one network round trip per signature (about 0.3–2 ms inside a cluster). The Vault numbers above are a single dev-mode node with no TLS and no audit device on the same machine: an upper bound, not a production figure. SoftHSM is a software token in the server process, not an HSM. - **Size a High-assurance realm against the HSM you will actually use**, with a load test of your own, and keep headroom: when the backend is the limit, the pods shed the excess as 503 instead of queueing. ### KMIP and the cloud key stores **Nothing above was measured on KMIP or on a cloud key service, and none of it should be read as a sizing figure for them.** - **KMIP.** The only KMIP server HelixIAM has run against is PyKMIP 0.10.0 (patched for KMIP 2.x gaps), a **functional test server** for protocol tests. The run (`helix-iam-server/target/pykmip-capabilities.txt`, written by `KmipCustodyOnPyKmipTest`) records what that server does with the backend configured `algorithms=RS256`, `kek=true`: signing RS256, non-extractable keys, AES-256-GCM with the caller's associated data (a wrong one is refused), RSA 2048, 3072 and 4096, and no key transport; it records no latency or rate. A High-assurance realm on KMIP signs through the key manager once per signature, so its ceiling is your key manager's signing rate plus a network round trip, over at most `connections` (default 8) persistent TLS connections per pod and `helix.custody.max-concurrent-calls` calls in flight. Size it by measuring your key manager with the benchmark, as for an HSM. - **AWS KMS, Azure Key Vault / Managed HSM, Google Cloud KMS.** The capacity is the provider's **documented request quota**, shared by everything else in the same account, subscription or project; HelixIAM measured none of it. The quotas and how to raise them: [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/requests-per-second.html), [Azure Key Vault](https://learn.microsoft.com/en-us/azure/key-vault/general/service-limits) and [Managed HSM](https://learn.microsoft.com/en-us/azure/key-vault/managed-hsm/service-limits), [Google Cloud KMS](https://cloud.google.com/kms/quotas). Over its quota a provider throttles; HelixIAM then answers `503` with `Retry-After` like any unavailable backend. Latency to a regional cloud service can be much higher than to an in-cluster HSM: for a High-assurance realm measure it from your cluster, and prefer Protected for a realm whose signing rate is high (the cloud service is then used only for the key-encryption key). - **Data-key rotation** has not been benchmarked either: it rewrites values in batches of `helix.db-encryption.rotation.batch-size` (default 100) by one replica at a time, and each batch is one short transaction. Run the first one off-peak. ### Credential issuance, eID logins and metadata The numbers above are for token signing. The issuer, eID and metadata keys add these backend calls: - **Credential issuance:** one ES256 signature per issued credential (SD-JWT VC or mdoc), plus one per status-list re-sign (JWT and CWT) in the background. - **eID login:** one signature for the AuthnRequest, one for the ArtifactResolve, one for each new mutual-TLS handshake of the back channel, and one OAEP decryption per `EncryptedKey` of an encrypted assertion. These calls count against the same in-flight limit and circuit breaker as token signing (`op` = `decrypt_key` for the decryption). - **Signed SAML metadata:** one signature per realm each time the cached copy is re-signed (after half its validity, or on a change), not one per request. Measured credential issuance (OpenID4VCI SD-JWT VC; each credential is one token, one nonce and one credential request). Machine: Apple M5 Max, Docker Desktop (aarch64, 18 CPUs). The server container had a 1536 MiB memory limit (the chart's minimum; the chart default is 2 GiB) and was pinned to 2 CPUs; the issuer key algorithm was ES256. PostgreSQL, Redis, the load generator (k6 1.3.0) and the key backend ran on separate CPUs. CPU per credential and latency are measured at 100 credentials/s and are the median of 3 runs. "Max/s" is the highest rate that kept the p95 of the credential request under 100 ms with less than 1 % errors; the cell shows the median of 3 runs, then the three runs. | Custody mode | Key backend | Server CPU per credential | Backend CPU per credential | p50 / p99 at 100/s | Max/s (median; runs) | | --- | --- | --- | --- | --- | --- | | software | none (key in the database) | 3.1 ms | - | 3.1 / 5.2 ms | 630 (690, 398, 630) | | direct, Vault Transit | Vault Transit (test server) | 3.8 ms | 0.5 ms | 3.8 / 10.1 ms | 416 (416, 416, 600) | | direct, SoftHSM (PKCS#11, in-process) | SoftHSM 2.6.1-3 | 3.4 ms | in the server | 3.3 / 8.2 ms | 331 (331, 331, 575) | How to read the table: - **The maximum rate varies a lot between runs** (up to 1.7 times in one mode), because the machine was shared with other work during the measurements (load average 4 to 11 on 18 CPUs, with the benchmark pinned to its own CPUs). Plan on the lower runs, not the best one: about 400 credentials/s per pod with 2 CPUs for software and Vault Transit, and about 330 for SoftHSM. Three runs per mode are too few for a tight confidence interval. - **What limits each mode at its ceiling:** the server's 2 CPUs. At the first rate above the ceiling the server used 1.5 to 1.9 of its 2 CPUs (SoftHSM: 1.1 to 1.8). The excess is then shed as `503` with `Retry-After` by the in-flight request limit (`helix.overload.max-in-flight`, default 40); 503s were 2 to 32 % of requests at the first step above the ceiling (up to 39 % at higher rates). The key backend itself is not the limit: Vault used about 0.5 ms of CPU per credential, and SoftHSM runs inside the server's CPU budget. A key backend with a high signing latency would hold requests in flight longer and reach that limit at a lower rate. - **Database pool:** no pool timeouts ("Connection is not available") in any run whose server log was kept (the three third runs), and the 503s are not pool-related. Earlier versions stalled the 10-connection database pool under issuance load; that is fixed. - **Memory:** peak memory held by the container reached the 1536 MiB limit in every run, as the kernel fills the rest with reclaimable file cache (1498 to 1536 MiB). The non-reclaimable (anonymous) memory peaked at 1407 to 1412 MiB (software, 2 runs measured), 1437 to 1456 MiB (Vault, 2 runs) and 1414 to 1421 MiB (SoftHSM, 3 runs), that is 92 to 95 % of the limit, while the load was above the ceiling. No container was OOM-killed. The headroom below the limit is small under sustained overload; the chart default is 2 GiB for this reason (minimum 1.5 GiB); keep it and do not go below the minimum if you run issuance close to its ceiling. - The Vault figures come from a single Vault test server with no TLS and no audit device on the same machine, and the SoftHSM figures from a software token: neither is a production HSM. Vendor figures (vendor-stated maximums, not measured with HelixIAM; performance tiers are licence-locked on Luna and nShield): | HSM | RSA-2048 signatures/s | P-256 signatures/s | Source | | --- | --- | --- | --- | | Thales Luna Network HSM 7 A700 / A750 / A790 | 1,000 / 5,000 / 10,000 | "over 20,000 ECC" (top model) | [Thales product brief](https://cpl.thalesgroup.com/sites/default/files/content/product_briefs/luna-network-attached-hsm-7-pb.pdf) | | Entrust nShield 5c Base / Mid / High | 670 / 3,949 / 13,614 | not published | [Entrust nShield 5c](https://www.entrust.com/products/hsm/nshield-5c) | | Yubico YubiHSM 2 | about 7 (139 ms each) | about 14 (73 ms each) | [YubiHSM 2 specifications](https://docs.yubico.com/hardware/yubihsm-2/hsm-2-user-guide/hsm2-device-specs.html) | A YubiHSM 2 is a serial USB device: fit for development and very small High-assurance realms only. Prefer ES256 (or EdDSA where the HSM supports it) for High-assurance realms: HSMs sign P-256 several times faster than RSA-2048. ## Compliance What HelixIAM claims: - In High assurance, the realm's private keys are generated inside the configured backend with non-extractable attributes, checked after generation and before use, and every signature is made by the backend. With a FIPS 140-3 or Common Criteria certified HSM in its certified mode, **the keys are protected by that module**: the HSM is the cryptographic boundary and HelixIAM orchestrates it. - In Protected, the private keys exist in plaintext only in pod memory; at rest they are encrypted by a KEK that stays in the backend. - In High assurance the same holds for the credential issuer keys and every eID SP key: they are generated in the backend, non-extractable, and sign (and, for eID decryption keys, unwrap the assertion's content key) there. The content key itself is in server memory while an assertion is decrypted. - These behaviours are tested against SoftHSM 2 (PKCS#11), Vault 2.2.1 and OpenBao 2.7.2. SHA-2 OAEP decryption through PKCS#11 has not been tested with a real token (see [OAEP key transport on each backend](#oaep-key-transport-on-each-backend)). !!! danger "What HelixIAM does not claim" - **HelixIAM itself holds no FIPS 140-3 validation and no Common Criteria certification.** Correct wording for a deployment: "HelixIAM uses a FIPS 140-3 Level 3 validated module in its approved mode (certificate #…)", never "HelixIAM is FIPS validated". - **No HSM model has been verified with HelixIAM yet** (no Luna, nShield, Utimaco or YubiHSM in the test matrix): the certified models below are what you can buy, not what HelixIAM has been tested on. Run the checks (generate, sign per algorithm, GCM KEK, re-login, fail-over) on your HSM in its certified mode before relying on it. - **Vault Community and OpenBao are not FIPS 140 validated.** Vault Transit gives sovereignty (keys in a service you run) but not FIPS. A FIPS or CC claim on that path needs Vault Enterprise "FIPS 140-3 Inside" builds, or Vault / OpenBao with HSM-backed keys; neither is tested with HelixIAM. - SoftHSM is a software token for tests: no assurance claim. - **KMIP** is tested against PyKMIP 0.10.0, a functional test server, patched for KMIP 2.x gaps (not a validated module); **no commercial KMIP server has been verified**, and ECDSA and RSA-OAEP over KMIP have not been run against a live server. What a KMIP deployment achieves depends on the key manager behind it. - **The cloud backends** (AWS KMS, Azure Key Vault / Managed HSM, Google Cloud KMS) are Preview and have not been run against a real account. **The FIPS 140 level and any other certification of those services are the provider's, for the service and the key protection level you configure** (an HSM-backed key, a Managed HSM, a software key): HelixIAM claims none for them, and it does not check which level a key has beyond the protection level you set (Azure `protection`, GCP `protectionLevel`). Ask the provider for the certificate and its scope. **FIPS 140-3** (ISO/IEC 19790): Level 3 adds tamper detection and response, identity-based operator authentication and separated interfaces for critical security parameters. FIPS 140-2 certificates are moving to the historical list; ask for 140-3 in new procurements. **Common Criteria EAL4+ (AVA_VAN.5, ALC_FLR.2) against EN 419 221-5** (protection profile for cryptographic modules of trust service providers) is the eIDAS reference for the key modules of qualified trust service providers. An IAM is usually not a QTSP; DigiD, eHerkenning and wallet relying-party schemes set their own requirements, which this page does not cover. Vendor-stated certifications (confirm the certificate numbers on the NIST CMVP and CC portals before quoting them): | Model | FIPS | CC / EN 419 221-5 | | --- | --- | --- | | Thales Luna Network HSM 7 | 140-3 Level 3 | EAL4+ AVA_VAN.5, ALC_FLR.2, EN 419 221-5 | | Entrust nShield 5c / 5s | 140-3 Level 3 (#4745 listed for 5s) | EAL4+ AVA_VAN.5, ALC_FLR.2, EN 419 221-5 | | Yubico YubiHSM 2 FIPS | 140-2 Level 3 (140-3 announced by the vendor) | none found | | Vault Enterprise | "FIPS 140-3 Inside" builds | n/a | Algorithms inside the boundary: RS256 (RSASSA-PKCS1-v1_5, SHA-256), PS256 (RSASSA-PSS, SHA-256), ES256 / ES384 (ECDSA P-256 / P-384), EdDSA (Ed25519; whether it is approved in an HSM's FIPS mode depends on its firmware), and AES-256-GCM for the KEK. For PKCS#11, HelixIAM uses the token's hashing mechanism when listed, otherwise it hashes in the JVM and signs the digest in the token (the private-key operation always stays in the HSM). ## Limits of this release - **KEK rotation is for Protected realms.** A High-assurance realm's KEK (kept for keys from an earlier Protected period) is not rotated: retire those keys once their tokens have expired. `DB_ENCRYPTION_KEK` is rotated with `DbKeyWrapTool --rewrap-from`, not the API (see [`DB_ENCRYPTION` under a KEK](#db_encryption-under-a-kek-envelope)). - **Keys written by a server of an earlier release record no KEK.** The upgrade labels them with the realm's KEK only in a realm that never had a KEK rotation (the label is certain there); afterwards they stay unlabelled, the rotator and the key store try each KEK of the realm, no KEK is disabled as unused while one exists, and a rotation that stopped cannot be abandoned until a resume has moved them. - **Finish a rollout before you rotate a KEK, and do not roll back during one.** A pod of an earlier release unwraps every key with the realm's current KEK only, so while a rotation runs it cannot unwrap the keys not yet wrapped again (they fail closed on that pod). - **RETIRED wrapped keys are not wrapped again**: once the old KEK is destroyed they can never be read (they never sign again). - **A stopped rotation can be abandoned only while no key is on its new KEK**; after that it can only continue. - **Quarantine is per pod.** Each pod quarantines a key when it reads the key's flags: at first use, or when its health check reads them again (up to 16 keys in use per check, round robin, at most once per 5 s probe window and only while something polls `/actuator/health` or the console). Until then a pod that already holds the key's public key can still sign with it. Monitor `/actuator/health` on every pod; the recovery above clears every pod. - **No backend-to-backend moves:** a realm in custody stays on its backend, and so does a realm lowered to Standard that keeps its key-encryption key there. Only a realm lowered to Standard on a backend that no longer serves it is forgotten there, once the retention job has released its KEK (at least twice `helix.custody.destroy-after`) (see [Binding a backend to realms](#binding-a-backend-to-realms)); it can then be raised on another backend. The issuer, eID and metadata keys live in the realm's backend too. - **A dedicated key store is configuration, read at start-up.** Changing `realms` needs a restart (the chart rolls the pods; assign realms or rotate KEKs only once every pod runs the new `realms`). A realm left out of its entry's `realms` is refused until it is put back or lowered to Standard. A Vault `403` for a key name the entry's policy does not allow is answered like any credential failure of that entry: `503` and it counts toward that entry's circuit breaker (with `realms` set, HelixIAM never sends such a call). - **SAML RSA-SHA1 only in Standard realms; RSA-SHA512 only with software keys (Standard and Protected):** a move out of Standard (RSA-SHA1) or to High assurance (RSA-SHA512) is refused while an enabled service provider asks for it, and saving a service provider with that algorithm in such a realm is refused (`400` with `fieldErrors.signatureAlgorithm`). RSA-SHA1 is deprecated: the server warns about each service provider that asks for it at start-up. - **RSA key sizes.** Realm keys (access, ID, SAML) are RSA 2048 in every backend (Vault `rsa-2048`, PKCS#11 2048). eID SP keys can be 2048, 3072 (default) or 4096, limited to the sizes the backend lists and to the scheme's minimum (eIDAS 3072). A realm key above 2048 bits is not offered. - **Issuer and development CA keys are ES256 only** (HAIP). A backend that cannot sign ES256 cannot serve an enabled issuer in Protected or High assurance, and the move is refused. - **eID SP keys are RSA only,** and a decrypting key serves exactly one key transport. SHA-2 OAEP decryption is proven with Vault and OpenBao (SHA-256 on OpenBao); on PKCS#11 it is proven only at the mechanism level, not with a real token (SoftHSM decrypts SHA-1 only). Vault and OpenBao refuse mixed digest and MGF hashes. See [OAEP key transport on each backend](#oaep-key-transport-on-each-backend). `rsa-1_5` key transport is never accepted. - **A decrypted content key is in server memory** for the duration of an assertion's decryption (see above). - **Signed SAML metadata is only as strong as the pin:** service providers must pin the realm's signing certificate. Metadata signing is off by default, and a realm whose SAML key is in the HSM/KMS no longer advertises an `encryption` key. - **EdDSA in PKCS#11 only where the token lists `CKM_EDDSA`**; generated wallet verifier keys are ES256 / ES384 only. - **Vault AppRole secret ids may be single-use:** the file is read at each login, so put a fresh secret id there before the token's max TTL runs out (or use a periodic token, or Kubernetes auth). - **Kubernetes auth is not tested** against a real cluster; AppRole and token are. - **Disabling a Vault or OpenBao eID decryption key** raises `min_decryption_version`, so the backend then no longer returns that version's public key; HelixIAM keeps the key's public key and certificate in its own row. - **Not tested:** a PKCS#11 token that decrypts SHA-2 OAEP, a network HSM for the issuer, eID and metadata keys, and Vault policies other than the one above. - **Cloud key stores are Preview.** AWS KMS, Azure Key Vault / Managed HSM and Google Cloud KMS are tested against emulators and recordings built from the providers' published API shapes, **not against a real account**: AWS against LocalStack; **Azure and GCP against published API examples only** (until a test against real accounts; no date committed). The wallet's per-transaction ECDH key and the wallet pepper stay software secrets, protected by the database key. - **No EdDSA on any cloud key store** (the `aws-kms` plugin does not offer it in this release, Azure Key Vault has no Ed25519, Cloud KMS signs Ed25519 over raw messages only), and none over KMIP. A realm that needs EdDSA tokens uses PKCS#11 (where the token lists `CKM_EDDSA`), Vault/OpenBao, or the Standard and Protected profiles. - **AWS KMS has no sign-and-decrypt keys** (an RSA key has one usage), so an eID SP key that must sign and decrypt cannot be one key there; and no cloud backend offers RSA-OAEP key transport in this release, so eID decryption keys need PKCS#11, Vault/OpenBao or KMIP. - **Azure: the key-encryption key needs a Managed HSM.** A plain Key Vault has no AES-GCM, so a realm cannot be Protected or High on it, and an `azure-kv` backend on a plain Key Vault cannot host a realm in this release. - **AWS FIPS and GovCloud endpoints:** the plugin pins its endpoints, so the SDK's FIPS switch has no effect; set `endpoint` and `stsEndpoint` to the FIPS endpoints yourself. **Cloud plugins do not use an egress proxy** (every HTTP client is pinned to no proxy). Google **Cloud EKM** is not supported. - **ECDSA over KMIP is not CI-tested against a live server** (PyKMIP makes RSA keys only); it is built and unit-tested against a protocol fake, and every key is proven by the generate-time self-test. **No commercial KMIP server has been tested**; verification against an independent KMIP implementation is planned (no date committed). RSA-OAEP key transport over KMIP is likewise not run live. A revoked KMIP key cannot be enabled again. - **Plugins are trusted code.** A plugin runs in the server process with the server's permissions; its class loader isolates dependencies and is not a sandbox. Trust rests on the SHA-256 pin and the signature of the plugin image. - **After a data-key rotation has started, rolling back to a release before rc.13 is not supported** (its values are key-tagged). Start a rotation only when every pod runs rc.13. A rotation with Redis sessions stays `DRAINING` for at least the longest absolute session lifetime (about 10 hours by default), with both keys configured; with the Redis token store the old key is needed until `oldKeyNeededUntil`. The data key is not rotated automatically, and the rotation does not re-encrypt backups or database dumps: they stay readable only with the key they were made under. - **A realm restricted to sovereign key stores** cannot be Standard, cannot import a software key, and cannot be set while any live key held on a backend (a key still in its grace period included) is on a non-sovereign backend. Software keys still in their grace period (verify-only) do not block it. - **Not yet measured:** a network HSM, Vault with TLS, an audit device and Raft storage, and a multi-pod installation sharing one backend. - A native PKCS#11 call that hangs cannot be interrupted: the caller gets a 503 at `opTimeout`, but that session and thread stay busy until the HSM returns. ## Settings reference | Setting | Default | Meaning | | --- | --- | --- | | `helix.custody.backends[n].id` / `.type` / `.settings.*` | none | The backends (above). | | `helix.custody.backends[n].realms` | none (shared) | The realm ids the backend serves; any other realm is refused. See [Dedicated key stores](#dedicated-key-stores). | | `helix.custody.require-dedicated-backend-for-high` | `false` | A High-assurance realm needs a backend whose `realms` is exactly that realm. | | `helix.custody.binding-check-interval` | `PT1M` | How often each pod checks the realms assigned to bound backends, the prefix overlaps and the released realms (the `keyCustody` health); minimum 5 seconds. | | `helix.custody.op-timeout` | `PT2S` | Time bound of a backend without its own `opTimeout`. | | `helix.custody.max-concurrent-calls` | 64 | Calls in flight per remote backend per pod (1–4096); beyond that, shed. | | `helix.custody.unwrap-retry` | `PT10S` | Retry interval of a starting pod that cannot unwrap its keys yet. | | `helix.custody.retention-check-interval` | `PT1H` | How often retired backend keys are disabled and destroyed. | | `helix.custody.destroy-after` | `P30D` | Time from disable to destroy. | | `helix.custody.max-pending-keys-per-realm` | 5 | CSR keys (and, separately, generated verifier keys) a realm may have waiting for a certificate. It also bounds issuer CSR keys and eID SP keys waiting for a certificate. | | `helix.custody.pending-key-ttl` | `P30D` | A CSR key, generated verifier key, issuer CSR key or eID SP key without a certificate (an eID SP key: not activated) after this is expired, then disabled and destroyed. | | `helix.custody.backends[n].settings.keyTransport` | every transport the token lists (KMIP: none) | PKCS#11 and KMIP: the RSA-OAEP key transports an eID decryption key may use, as `METHOD;DIGEST;MGF` values, comma-separated (chart: a list). | | `DB_ENCRYPTION_WRAPPED`, `DB_ENCRYPTION_KEK` | unset | The envelope (above). | | `helix.custody.kek-rotation.batch-size` | 16 | Keys a KEK rotation wraps again per batch (1–256). | | `helix.custody.kek-rotation.interval` | `PT5S` | How often each pod's rotation worker looks for work. | | `helix.custody.kek-rotation.lease` | `PT60S` | How long one pod holds a rotation without renewing it; after that another pod continues it. | | `helix.custody.kek-rotation.enabled` | `true` | Whether this pod runs the rotation worker (the API still starts rotations). | | `helix.custody.plugin-dir` | `/opt/helix/plugins` | Where the plugin jars are (read-only to the server user). See [custody-plugins.md](custody-plugins.md). | | `helix.custody.plugins[n].file` / `.sha256` / `.type` | none | The pins: one per jar in the plugin directory (chart: `custody.plugins.jars`, with `custody.plugins.image`). | | `helix.custody.sovereign-types` | none | Plugin types you assert are sovereign key stores (chart: `custody.sovereignTypes`). `pkcs11`, `vault` and `kmip` always are; a cloud provider type or `software` here refuses start-up. | | `helix.custody.backends[n].settings.*` (`kmip`, `aws-kms`, `azure-kv`, `gcp-kms`, other plugins) | none | The backend's own settings; see [KMIP](#kmip-key-managers), [Cloud provider key stores](#cloud-provider-key-stores-aws-kms-azure-key-vault-managed-hsm-google-cloud-kms) and `values.yaml`. | | `DB_ENCRYPTION_NEXT` | unset | The next data key of a rotation, plain: at least 32 characters, strength-checked (chart: `dbEncryption.next.secret`). | | `DB_ENCRYPTION_NEXT_WRAPPED`, `DB_ENCRYPTION_NEXT_KEK` | unset | The next data key, wrapped (16, 24 or 32 bytes unwrapped); the KEK defaults to `DB_ENCRYPTION_KEK` (chart: `dbEncryption.next.wrapped`). Not together with `DB_ENCRYPTION_NEXT`. | | `helix.db-encryption.rotation.batch-size` | 100 | Values a data-key rotation rewrites per batch (1–1000). | | `helix.db-encryption.rotation.lease` | `PT60S` | How long one replica holds the rotation without renewing it (at least `PT1S`). | | `helix.db-encryption.rotation.interval` | `PT5S` | How often each replica's resealer looks for work. | | `helix.db-encryption.rotation.drain` | `PT30M` | The longest lifetime of the values a rotation does not rewrite (flow state, wallet transactions); waited for before it finishes. Sessions and the Redis token store add their own waits. | | `helix.db-encryption.rotation.enabled` | `true` | Whether this replica runs the resealer (the API still starts rotations). | | `helix.db-encryption.heartbeat` | `PT10S` | How often a replica records which data keys it holds and writes under; a rotation sees a replica for three intervals. | Per-realm settings for these keys are API and console settings, not server settings: the realm's custody profile (above) and the SAML metadata signing setting (`PUT /admin/realms/{realmId}/saml-idp/metadata-settings`). ## See also - [Realm keys](realm-keys.md) - [Security hardening](security.md) - [Install: key custody values](../getting-started/install.md#key-custody-hsm-and-kms) - [API reference: key custody](../integration/api-reference.md#key-custody-hsm-kms)