Key custody (HSM and KMS)

Keep a realm's private signing keys out of the database: in a hardware security module (PKCS#11) or in a Vault / OpenBao Transit engine you run yourself. You choose per realm how far the keys are kept away from HelixIAM, and every request that needs an unavailable backend fails closed. The same applies to the credential issuer keys, the development CA, the eID SP keys (signing, mutual TLS and decryption) and signed SAML IdP metadata.

Added in 1.1.0-rc.11, extended in 1.1.0-rc.12 and 1.1.0-rc.13

Realm token, SAML and wallet verifier keys came with 1.1.0-rc.11; the credential issuer keys, the development CA, the eID SP keys and signed SAML metadata are added in 1.1.0-rc.12. 1.1.0-rc.13 adds a KMIP backend, a plugin interface with plugins for AWS KMS, Azure Key Vault / Managed HSM and Google Cloud KMS (Preview), a sovereign key-store policy per realm and online rotation of the database encryption key. Key custody is opt-in. Nothing changes on upgrade: every realm, new or upgraded, starts in the Standard profile, which is how HelixIAM has always worked (software keys in the database, encrypted with DB_ENCRYPTION).

Security levels (profiles)

Each realm has one custody profile. A master-realm administrator chooses it per realm.

Profile	Where the private keys are	Signing	What an attacker needs
Standard (default)	Generated in the server, stored in the database encrypted with DB_ENCRYPTION	In the server	The database and the DB_ENCRYPTION key
Protected	Generated in the server, stored in the database only wrapped (AES-256-GCM) by the realm's key-encryption key (KEK) in the backend. Each pod unwraps them into memory.	In the server, as fast as Standard	The database, plus the right to call the backend's decrypt with the realm's KEK
High assurance	Generated inside the backend; they never leave it. The database holds only a reference.	Every signature is a backend call	The right to call the backend's sign with the realm's keys

High assurance covers every realm-key purpose (access, id, saml): none of these keys exists outside the backend, and importing a software realm key is refused. The same holds for wallet verifier keys: they are generated in the backend with Generate in HSM/KMS, importing a software verifier key into a High-assurance realm is refused (400 on privateKeyPem), and a software verifier key stored before the switch no longer signs (see Wallet verifier keys). A High-assurance realm's token throughput is the backend's signing capacity (see Sizing).

The issuer, eID and metadata keys follow the same three profiles. See High-assurance rules for the issuer, eID and metadata keys for what each profile allows.

Keys per purpose

A realm in Protected or High assurance gets its own keys per purpose, one ACTIVE key per algorithm family (RSA, EC, Ed25519) and purpose, all in the realm's custody:

Purpose	Signs
access	Access tokens, RFC 8693 / delegation tokens, workload-identity tokens
id	ID tokens, logout tokens, JARM responses
saml	SAML assertions
verifier	Wallet (OpenID4VP) request objects; separate verifier keys, see Wallet verifier keys
issuer	Credentials and status lists of the OpenID4VCI issuer (ES256 only); see Credential issuer keys and the development CA
issuerca	The realm's development CA, which certifies generated test issuer keys (ES256 only)
federation	The SP keys of eID connections: AuthnRequest and ArtifactResolve signatures, mutual TLS and assertion decryption; see eID SP keys (RSA only)

issuer, issuerca and federation are not realm-key purposes: they belong to their own screens and endpoints, the realm-key CSR (POST /admin/realms/{realmId}/keys/csr) refuses them, and they are not in the realm's JWKS.

The JWKS publishes every purpose's keys; the token type picks the key. There is no fallback to another purpose, to a software key or to RS256: a key that cannot sign fails the request (see Failure behaviour). In High assurance a client's id_token_signed_response_alg must be an algorithm the backend signs; saving a client with another one is refused (fieldErrors.idTokenSignatureAlg).

Standard realms keep their single key set (purpose all) and their current behaviour, including the RS256 fallback with its warning. See Realm keys.

Backends

A backend is configured by the operator, never in the console: it involves secrets and infrastructure. The types are software (always present), pkcs11, vault, kmip and the types of the plugins you install (aws-kms, azure-kv, gcp-kms from HelixIAM). The server reads the backends from helix.custody.backends[n] (id, type, settings.*, and optionally realms: the realms it serves, see Dedicated key stores). The software backend always exists and needs no entry. Settings with secrets (PIN, token, secret id, CA bundle) are file paths: the secret itself is read from the file at each login and never stored, logged, audited or shown in the console.

With the Helm chart

List the backends under custody.backends. The chart renders them into a ConfigMap (custody.yaml, mounted at /etc/helix/custody/ and loaded through SPRING_CONFIG_ADDITIONAL_LOCATION), mounts each backend's Secrets as files under /var/run/helix/custody/<id>/, and rolls the pods when a custody setting changes (the checksum/custody annotation). Allow the backend's port in networkPolicy.extraEgressPorts (8200 for Vault, 5696 for KMIP, 443 for a cloud key service, your HSM's port for a network HSM). A vendor PKCS#11 module is not in the image: mount it with extraVolumes / extraVolumeMounts. See Install: key custody values.

Without the chart

Put the same structure in a YAML file and point SPRING_CONFIG_ADDITIONAL_LOCATION at it. Use a file, not environment variables: Spring lower-cases environment-variable names, and the setting keys are camelCase (tokenLabel, pinFile, tokenFile).

helix:
  custody:
    backends:
      - id: hsm-1                  # appears in key references; letters, digits, . _ -
        type: pkcs11
        settings: { ... }          # below
      - id: vault-1
        type: vault
        settings: { ... }
        realms: [tenant-a]         # optional: only these realms; absent, every realm (shared)

Unknown settings are refused at start-up (the message names the setting, never its value). An algorithm a backend cannot sign is refused when a realm is moved to it, never at signing time.

PKCS#11 (HSM)

Setting	Meaning
library	Path of the vendor's PKCS#11 module inside the container. With the chart, mount it with extraVolumes / extraVolumeMounts.
tokenLabel	The token (partition) label. Refused when no token or more than one token has it.
pinFile or pinEnv	The file (chart: pinSecret, mounted at /var/run/helix/custody/<id>/pin), or the name of an environment variable, holding the user PIN. Exactly one.
sessions	Session pool size, default 8 (1–256). Also the number of signing threads per pod for this backend.
opTimeout	ISO-8601 time bound per call, default PT2S, at most PT1M.
connectTimeout	ISO-8601 time that start-up, or a call, waits for a connect (module, token, login), default PT5S, at most PT1M. A slower connect goes on in the background, the backend disconnected meanwhile.
algorithms	Optional: restrict to these JWS algorithms.
keyTransport	Optional: the RSA-OAEP key transports an eID decryption key may use, comma-separated METHOD;DIGEST;MGF values such as RSA_OAEP_MGF1P;SHA1;SHA1,RSA_OAEP;SHA256;SHA256 (chart: a list). Unset, it is every transport whose hash mechanisms the token lists and that passes the backend's decryption self-test on each connect (a known-good ciphertext must decrypt); a configured transport that fails it is refused at start-up. An unknown value is refused at start-up. See OAEP key transport on each backend.

What HelixIAM does on the token:

• Signing keys are created with CKA_TOKEN, CKA_PRIVATE, CKA_SENSITIVE = true and CKA_EXTRACTABLE = false (RSA also CKA_DECRYPT / CKA_UNWRAP = false, EC CKA_DERIVE = false), labelled helix:<realm>:<PURPOSE>:<uuid>. The one exception is an eID SP key made for decryption (purpose FEDERATION, usage signing and decryption): it has CKA_DECRYPT = true, is still sensitive and not extractable, and the read-back checks CKA_DECRYPT for exactly the usage that was asked. Elliptic-curve keys get CKA_DECRYPT = false explicitly. The attributes are read back after generation, and checked again at first use and by the health check for the keys in use (up to 16 per check, round robin): a token object, sensitive, not extractable, always sensitive, never extractable, CKA_LOCAL. A key that fails (it can leave the token, or it was imported, for example an object replaced out of band under the same label) is quarantined like a Vault key made exportable: never used, 503 for every request that needs it, health DOWN naming its label. See A key that can leave the backend.
• The realm KEK is an AES-256 token key (sensitive, not extractable) used with CKM_AES_GCM.
• Algorithms come from C_GetMechanismList: RS256, PS256, ES256, ES384, and EdDSA only when the token lists CKM_EDDSA. Realm keys are RSA 2048. eID SP keys are RSA 2048, 3072 or 4096, limited to the sizes the token lists for CKM_RSA_PKCS_KEY_PAIR_GEN. A token that lists no range makes no RSA size available for them (realm keys are not affected).
• One login per process; on CKR_USER_NOT_LOGGED_IN or a lost session it logs in again once and retries once, then fails closed. C_Finalize is never called.
• If the HSM is unreachable at start-up, the backend starts disconnected and reports DOWN; it connects on use.

The JVM side:

• The JVM needs --enable-native-access=ALL-UNNAMED. The image's entry point sets it; add it yourself when you run the jar directly.
• The binding's JNI library (libpkcs11wrapper.so) ships in the image under /opt/helix/native/, read-only, for the image's own architecture (linux/amd64 in 1.1.0-rc.12; arm64 images are planned to return), and the entry point puts that directory on -Djava.library.path. Nothing is unpacked into or run from /tmp, so a noexec /tmp (a Docker or Compose tmpfs, a pod with readOnlyRootFilesystem and an emptyDir) works. The JVM reads -Djava.library.path only at start, so it can be changed only through JAVA_OPTS (a later -Djava.library.path there replaces the image's: keep /opt/helix/native on it), not through a Spring setting or environment variable of the server.
• Running the jar yourself: the binding finds the library on -Djava.library.path first. Without it there, the binding unpacks its own copy into java.io.tmpdir and loads it from there, which needs a temporary directory that allows execution. If neither works, the backend starts disconnected with "The PKCS#11 wrapper library (libpkcs11wrapper.so) could not be loaded", the path it searched and the cause, and every call is unavailable.

Configuring the module (SoftHSM and other HSMs)

A PKCS#11 module is not in the image, and most modules read a configuration file of their own: HelixIAM only loads library and logs in to tokenLabel. Mount the module, its configuration and anything else it needs with extraVolumes / extraVolumeMounts, and set the environment variable that names its configuration through config.extraEnv. The server runs as uid/gid 10001 with a read-only root file system, so whatever the module writes must be a mounted volume that uid 10001 can write.

SoftHSM (tests and demos). SoftHSM finds its configuration through SOFTHSM2_CONF and keeps its tokens as files in directories.tokendir. A recipe with a node directory /srv/softhsm (a hostPath; a PersistentVolumeClaim works the same way):

/srv/softhsm/lib/libsofthsm2.so   # built for the image's OS (Ubuntu 24.04) and the node's architecture
/srv/softhsm/softhsm2.conf        # directories.tokendir = /opt/softhsm/tokens
                                  # objectstore.backend = file
/srv/softhsm/tokens/              # owned by 10001:10001 (SoftHSM writes its objects here)

Initialise the token once with the same configuration, for example SOFTHSM2_CONF=/srv/softhsm/softhsm2.conf softhsm2-util --init-token --free --label helix --so-pin <so-pin> --pin <pin> (with tokendir pointing at /srv/softhsm/tokens for that run, or run it in a container that mounts the directory at /opt/softhsm), then chown -R 10001:10001 /srv/softhsm/tokens. A hostPath volume is not changed by the pod's fsGroup, so the ownership must be right on the node. Then:

replicaCount: 1                      # SoftHSM's file token is one node's state: one replica is recommended
config:
  extraEnv:
    - name: SOFTHSM2_CONF
      value: /opt/softhsm/softhsm2.conf
custody:
  backends:
    - id: hsm-1
      type: pkcs11
      library: /opt/softhsm/lib/libsofthsm2.so
      tokenLabel: helix
      pinSecret: {name: helix-hsm-pin, key: pin}   # kubectl create secret generic helix-hsm-pin --from-literal=pin=<pin>
extraVolumes:
  - name: softhsm
    hostPath: {path: /srv/softhsm, type: Directory}
extraVolumeMounts:
  - name: softhsm
    mountPath: /opt/softhsm

Run one replica with SoftHSM. Its file store is not designed for several processes writing at once, and two pods with their own copies of the token directory (on different nodes) hold different keys. Several replicas on one node sharing the same hostPath token directory worked in testing for signing and rotation, but two pods creating keys at the same time is unsupported. For more than one replica, or more than one node, use a real network HSM. GET /admin/key-custody/backends should then list hsm-1 as healthy.

A production HSM (network or PCIe) usually needs its client module, a client configuration file (server address, slot or partition) and client credentials (a certificate, a key, an enrolment file), each named by a path or an environment variable the vendor documents. Mount the credentials from a Secret, set the variables through config.extraEnv, and allow the HSM's port in networkPolicy.extraEgressPorts. A network HSM is shared by every pod, so the replica count is not limited, but check the vendor's session limit against sessions × replicas.

Tested with SoftHSM only

PKCS#11 is tested with SoftHSM 2.6.1 (Ubuntu 24.04) and 2.7.0 only; both decrypt RSA-OAEP with SHA-1/MGF1-SHA-1 only (2.6.1 answers CKR_GENERAL_ERROR for a ciphertext that does not decrypt, which HelixIAM treats as a refusal while the key can still be read). No network HSM (Luna, nShield, Utimaco, YubiHSM) has been run against HelixIAM yet: vendor template rules, FIPS-mode GCM IV rules, fail-over and session limits are untested. Run the checks in Compliance on your own HSM before you rely on it.

Vault and OpenBao Transit

type: vault works with HashiCorp Vault and with OpenBao (same API). Tested against Vault 2.2.1 and OpenBao 2.7.2 dev servers.

Setting	Meaning
address	https://host[:port]. Plain http:// only to a loopback host, or with insecureHttp: "true" (for a mesh-encrypted link; a warning is logged at start-up).
mount	The Transit mount, default transit.
namespace	Optional (Vault Enterprise namespaces).
auth.kind	approle, token or kubernetes.
auth.role	AppRole: the role id. Kubernetes: the role name.
auth.secretIdFile	AppRole: the file with the secret id (chart: auth.secretIdSecret, mounted at .../secret-id).
auth.tokenFile	Token: the file with the token (chart: auth.tokenSecret, .../token). Kubernetes: the service-account JWT, default the pod's projected token.
auth.path	The auth mount, default approle / kubernetes.
caFile	PEM bundle of the CA certificates to trust (chart: caSecret, .../ca.crt).
opTimeout	Connect and read timeout per call, default PT2S, at most PT60S.

• Every login reads its file again, so a rotated token, a fresh single-use AppRole secret id or a refreshed projected service-account token is picked up without a restart. The token is renewed in the background; login tokens are revoked on shutdown (a configured token is never revoked).
• Keys are named helix-<realm>-<purpose>-<uuid> and, for key-encryption keys, helixkek-<realm>-<uuid> (earlier releases: helix-<realm>-kek-<uuid>, still accepted), created with exportable=false and allow_plaintext_backup=false, and read back. Every signature names its key_version. An eID SP key that decrypts is named helixdec-<realm>-federation-<uuid> (prefix helixdec-, purpose federation only), so a policy can let the backend decrypt with these keys and with no signing key.
• Algorithms: RS256, PS256, ES256, ES384, EdDSA. Realm keys are rsa-2048; eID SP keys are rsa-2048, rsa-3072 or rsa-4096.
• The KEK is a non-derived aes256-gcm96 key; the realm and key id are bound as GCM associated data (associated_data).
• AppRole and token login are tested against Vault and OpenBao. Kubernetes login is parsed and configured but has not been run against a real cluster yet. It needs the service-account token in the pod (automountServiceAccountToken: true or a projected token volume) and a role bound to the pod's service account.

Minimal policy

HelixIAM needs exactly these paths. A trailing * is a prefix match; + matches a whole path segment only, so a pattern such as helix-+-kek-+ matches no key name and is not usable.

# HelixIAM key custody: signing keys helix-*, decryption keys helixdec-*, key-encryption keys helixkek-*;
# never export or backup. A trailing * is a prefix match: it also covers keys/<name>/rotate and keys/<name>/config.
# denied_parameters: this token can never make a key exportable or allow a plaintext backup (on keys/<name>/config,
# where neither can be undone, nor at creation; HelixIAM never sends them, the defaults are false).
path "transit/keys/helix-*" {
  capabilities      = ["create", "read", "update", "delete"]
  denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] }
}
path "transit/keys/helixdec-*" {
  capabilities      = ["create", "read", "update", "delete"]
  denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] }
}
path "transit/keys/helixkek-*" {
  capabilities      = ["create", "read", "update", "delete"]
  denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] }
}
path "transit/sign/helix-*"       { capabilities = ["update"] }
path "transit/sign/helixdec-*"    { capabilities = ["update"] }
# Encrypt and decrypt: the realms' KEKs and the eID decryption keys, never a signing key.
path "transit/encrypt/helixkek-*" { capabilities = ["update"] }
path "transit/decrypt/helixkek-*" { capabilities = ["update"] }
path "transit/decrypt/helixdec-*" { capabilities = ["update"] }
# DB_ENCRYPTION_KEK (made by the operator's DbKeyWrapTool, policy below): the server only reads it (its flags) and
# unwraps the data key at start-up; it never wraps with it, disables or destroys it
path "transit/keys/helixdbkek-*"    { capabilities = ["read"] }
path "transit/decrypt/helixdbkek-*" { capabilities = ["update"] }

The operator's DbKeyWrapTool (for DB_ENCRYPTION_KEK, see DB_ENCRYPTION under a KEK) runs with credentials of its own and this policy: it creates, reads, rotates and disables helixdbkek- keys and wraps and unwraps with them, and can never make one exportable or allow a plaintext backup:

path "transit/keys/helixdbkek-*" {
  capabilities      = ["create", "read", "update"]
  denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] }
}
path "transit/encrypt/helixdbkek-*" { capabilities = ["update"] }
path "transit/decrypt/helixdbkek-*" { capabilities = ["update"] }

The contract tests run the server with the minimal policy above (it reads and unwraps a helixdbkek- key, and is refused 403 when it would wrap with it or reconfigure it) and the tool with this one (create, wrap, disable; 403 for exportable and allow_plaintext_backup), on Vault 2.2.1 and OpenBao 2.7.2.

• Do not grant transit/decrypt/helix-* or transit/encrypt/helix-*. They would let the backend itself decrypt (or encrypt) with the RSA signing keys. HelixIAM never asks it to, and refuses before any call to decrypt with a key that is not a helixdec- key, but the policy should not allow what the server never needs. The contract tests ran against Vault 2.2.1 and OpenBao 2.7.2 with the policy above and show that a decrypt with an RSA signing key is 403 under it (also for a realm whose id ends in -kek) and 200 with the root token.
• KEKs with the earlier name. A KEK made before this release (helix-<realm>-kek-<uuid>: the realm KEKs of Protected and High-assurance realms, and a DB_ENCRYPTION_KEK created as the phase-1 guide said) keeps its name and keeps working. Its realm needs these two lines, one pair per such realm:

path "transit/encrypt/helix-<realm>-kek-*" { capabilities = ["update"] }
path "transit/decrypt/helix-<realm>-kek-*" { capabilities = ["update"] }

Such a line is a prefix match, so it also matches the signing keys of a realm whose id starts with <realm>-kek (for master: master-kek, master-kek-eu, ...). HelixIAM therefore refuses to create a realm with such an id: an id that starts with the id of an existing realm followed by -kek, and any id that ends in -kek, answers 400 with fieldErrors.realmId (the realm settings API and a realm import). Realms created before this release are not checked, so make sure none has such an id before you add the lines. A new KEK (a realm moved to Protected or High assurance from now on) is a helixkek- key and needs no per-realm line. - create/update on keys/helix-*, keys/helixdec-* and keys/helixkek-*: create a realm's keys and KEK, rotate (/rotate), disable and allow deletion (/config). read: public keys and key versions. delete: the retention job's destroy. - exportable and allow_plaintext_backup are denied on those paths. keys/<name>/config accepts both, and once true they can never be set back: a stolen HelixIAM token could otherwise make every key exportable for another identity with export or backup rights. Tested on Vault 2.2.1 and OpenBao 2.7.2: such a request is 403. - HelixIAM checks both flags itself when it first reads a key on a server (its public key, before signing), before a KEK's or decryption key's first use, and again for the keys in use on every health check (up to 16 keys per check, round robin). A key with either flag not false is quarantined: it is never used again on that server, and it is logged as an error naming the key. See A key that can leave the backend for what clients see and how to replace it. - Nothing grants export, backup, restore or datakey. Lookup and renewal of its own token come from Vault's default policy. - A backend entry with this policy serves every realm on the installation: realm isolation inside the mount is enforced by HelixIAM (a key reference resolves only when its key name embeds its own realm), not by Vault. For isolation by Vault, give each tenant a backend entry of its own with a credential limited to its realm, and bind the entry to the realm: see Dedicated key stores.

Set it up (Vault CLI; with OpenBao use bao):

vault secrets enable transit
vault policy write helixiam-custody helixiam-custody.hcl
# AppRole (recommended outside Kubernetes); a single-use secret id works too (-secret-id-num-uses=1 on the role)
vault auth enable approle
vault write auth/approle/role/helixiam token_policies=helixiam-custody token_ttl=1h token_max_ttl=24h
vault read -field=role_id auth/approle/role/helixiam/role-id        # -> auth.role
vault write -f -field=secret_id auth/approle/role/helixiam/secret-id # -> the secret-id file

OAEP key transport on each backend

An eID decryption key decrypts only the EncryptedKey of an assertion: the identity provider encrypts a random content key to the SP certificate with RSA-OAEP, and the backend decrypts that. HelixIAM maps the XML Encryption identifiers exactly and never substitutes a parameter:

• http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p: OAEP digest SHA-1, MGF1 with SHA-1, nothing else.
• http://www.w3.org/2009/xmlenc11#rsa-oaep: the digest from ds:DigestMethod (default SHA-1) and the MGF from xenc11:MGF (default MGF1-SHA-1), each one of SHA-1, SHA-256, SHA-384 or SHA-512, set independently.
• http://www.w3.org/2001/04/xmlenc#rsa-1_5 (PKCS#1 v1.5 key transport) is refused. XML Encryption 1.1 warns against it (chosen-ciphertext attacks), the eIDAS cryptographic requirements do not list it, and none of the three eID schemes requires it. A non-empty OAEP PSourceAlgorithm (a label) is refused too.

Each decryption key is registered for exactly one {method, digest, mgf}. When the key is made, HelixIAM encrypts 32 random bytes to its public key with exactly those parameters and has the backend decrypt them, and it also checks that a ciphertext made with other parameters is not decrypted. A backend that cannot honour the combination, or silently uses other parameters, is refused then (400 on keyTransport) and the new backend key is disabled. A login never discovers an unsupported combination.

What each backend really decrypted when measured (a 3072-bit key, target/*-oaep-matrix.txt of the test runs):

Key transport	SoftHSM 2.6.1 / 2.7.0 (PKCS#11)	Vault 2.2.1	OpenBao 2.7.2
rsa-oaep-mgf1p (SHA-1, MGF1-SHA-1)	decrypts	decrypts	refused at key creation
rsa-oaep, SHA-1 / MGF1-SHA-1	decrypts	decrypts	refused at key creation
rsa-oaep, SHA-256 / MGF1-SHA-256	refused (2.7.0: CKR_ARGUMENTS_BAD, 2.6.1: CKR_GENERAL_ERROR)	decrypts	decrypts
rsa-oaep, SHA-384 / MGF1-SHA-384	refused	decrypts	refused at key creation
rsa-oaep, SHA-512 / MGF1-SHA-512	refused	decrypts	refused at key creation
rsa-oaep, digest and MGF1 hash different (the other 12 of the 16 rsa-oaep combinations)	refused	refused before any call	refused before any call

• PKCS#11 sets the digest and the MGF separately (CK_RSA_PKCS_OAEP_PARAMS), so a token can in principle serve every combination. SoftHSM 2.6.1 and 2.7.0 hard-code SHA-1 for both, and so decrypt only the first two rows; their advertised mechanisms list SHA-2, so the backend's decryption self-test when it connects drops the others (they are not offered, and creating a key for one is a 400 on keyTransport). A backend whose self-test passes no transport refuses every eID decryption key at configuration time and says so; it still signs. SHA-2 OAEP has not been tested with a decryption through a PKCS#11 backend of HelixIAM. It was checked at the mechanism level only: the parameters HelixIAM sends were run on a raw session of the NSS soft token for all 17 combinations (all decrypt), and a recording test shows the exact parameters sent. NSS is not a supported backend: it reports CKA_LOCAL = false for keys it made itself, and HelixIAM refuses such a key. A real HSM must be tested with the self-test at key creation before it is relied on for SHA-2 OAEP.
• Vault and OpenBao Transit use one hash for the OAEP digest and MGF1 (padding_scheme=oaep, hash_algorithm). A combination with different hashes is refused on them at configuration time, with a message saying so; DigiD and eHerkenning documents that allow a SHA-2 digest with MGF1-SHA-1 therefore cannot use those backends for that combination (use rsa-oaep-mgf1p).
• OpenBao 2.7.2 ignores padding_scheme and hash_algorithm on decrypt: it always decrypts OAEP with SHA-256 and MGF1-SHA-256 and answers 200 with a warning that it ignored them. HelixIAM refuses such an answer for every transport but exactly SHA-256 / MGF1-SHA-256, so a ciphertext made with other parameters never yields bytes. On OpenBao the only usable transport is rsa-oaep SHA-256 / SHA-256 (the eIDAS default). DigiD and eHerkenning connections whose identity provider cannot be set to that transport cannot decrypt on OpenBao.
• The decrypted content key exists in server memory. The backend returns the unwrapped content key (a symmetric AES key) to the server, and the server uses it to decrypt the assertion. The SP's private key never leaves the backend, but the content key of an assertion is in the memory of the pod that handles the login, for the duration of the decryption (the buffers are zeroed afterwards). Treat a pod's memory as able to read the assertions it decrypts.
• KMIP offers a key transport only when the keyTransport setting lists it, proven for each key when it is made; it has not been run against a live KMIP server (PyKMIP decrypts with symmetric keys only). AWS KMS, Azure Key Vault / Managed HSM and Google Cloud KMS offer none in this release, so an eID connection's decryption key cannot be kept on them.
• The keyTransport setting of a PKCS#11 or KMIP backend and GET .../sp-keys/algorithms (below) show which transports a configured backend offers. The offer is the backend's mechanism list, not a promise: the self-test decides per key.

KMIP (key managers)

type: kmip talks to a KMIP key manager (an HSM-backed key server with a KMIP interface, for example) over mutual TLS. It is a built-in backend, counted as a sovereign key store (see Sovereign key stores only).

What was tested. The KMIP client is HelixIAM's own (TTLV over TLS, no third-party KMIP library). It is tested against PyKMIP 0.10.0, patched for the KMIP 2.x gaps it has (e2e/kmip/patch_pykmip.py: key attributes, links, X.509 public keys, error codes and the version a request is decoded in), and against a protocol fake in the unit tests. No commercial KMIP server has been tested, so interoperability with a specific product is not verified: run the HelixIAM test matrix (generate, sign per algorithm, KEK round trip, restart, certificate rotation) on your key manager before you rely on it. PyKMIP makes RSA keys only (no EC), cannot sign RSA-PSS with a 32-byte salt and decrypts only with symmetric keys (no RSA-OAEP), so ECDSA (ES256, ES384) and RSA-OAEP key transport over KMIP have not been run against a live server (they are built and unit-tested against the fake). Treat KMIP as Preview until it has run against an independent key manager.

Requirements. KMIP 2.0 or later (Discover Versions picks the highest common version; a server that offers none is refused at start-up), mutual TLS 1.3 or 1.2 with host-name verification, and the operations Query lists: Create Key Pair, Activate, Get Attributes, Get, Sign, Revoke and Destroy for signing keys; Create, Activate, Get Attributes, Encrypt, Decrypt, Revoke and Destroy for key-encryption keys. No setting turns TLS, the client certificate or host-name verification off, and the server's certificate is trusted only through your ca.crt (never the JDK's defaults).

Setting	Default	Meaning
address	required	host[:port] of the KMIP server ([v6-address]:port for IPv6); port 5696.
serverName	the address's host	The name the server's certificate must carry.
algorithms	required	Comma-separated (chart: a list) out of RS256, PS256, ES256, ES384: what your key manager signs. EdDSA is never offered over KMIP.
kek	required	true or false: whether the key manager does AES-256-GCM with the caller's associated data (key-encryption keys, Protected and High assurance, DB_ENCRYPTION_KEK).
keyTransport	none	The RSA-OAEP key transports an eID decryption key may use, as METHOD;DIGEST;MGF (an RSA algorithm must be listed in algorithms).
opTimeout, connectTimeout	helix.custody.op-timeout, PT2S	ISO-8601, above zero, at most PT60S. One opTimeout covers the whole of a call, including every round trip of a key generation.
connections	8	Persistent TLS connections per pod, 1 to 64.
objectGroupPrefix	helix-	A realm's keys are in the Object Group <prefix><realm> (1 to 32 letters, digits, ., _, :, -).

algorithms, kek and keyTransport are the operator's statement, not probed: when a pod starts it only reads (Discover Versions, Query) and makes nothing on the key manager, so every pod offers the same and no extra permission is needed. Each new key is then proven when it is generated (a signature that verifies, a KEK round trip that refuses another associated data); a key that does not do what you stated is disabled and the request refused. State only what your key manager does. The consequence: a wrong algorithms shows at the first key generation, not at start-up.

The TLS material is three files in /var/run/helix/custody/<id>/: tls.crt (the client certificate chain, leaf first), tls.key (its PKCS#8 PEM private key; RSA or EC) and ca.crt (the CA certificates that sign the server's certificate). The server refuses to start when tls.key is readable by others or writable by anyone but the owner; owner and group read (mode 0440, with the pod's fsGroup) is accepted, which is what the chart mounts. The files are read again after a handshake fails on a certificate, so a rotated Secret is used without a restart.

custody:
  backends:
    - id: kmip-1
      type: kmip
      address: kms.example.com:5696
      serverName: kms.example.com
      algorithms: [RS256, ES256]       # what your key manager signs
      kek: true
      tlsSecret: {name: helix-kmip-client}   # a kubernetes.io/tls Secret: tls.crt and tls.key
      caSecret: {name: helix-kmip-ca, key: ca.crt}

Allow the KMIP port in networkPolicy.extraEgressPorts (5696). Keys are created Sensitive and not Extractable, named helix:<realm>:<purpose token>:<id> in the Object Group of the realm, and are read back after creation; a key that reads back extractable is destroyed (a new one) or quarantined (an existing one, like the other backends). The first use of a reference checks that the key's name and Object Group are the reference's realm, so a key of another realm is never used. A revoked KMIP key cannot be activated again, so a disabled key-encryption key cannot be enabled again. A permission or authentication denial (KMIP Permission Denied) is a 503 with lastError: policyDenied in the backend's health until a call works; an explicit refusal of an operation or its parameters is a refusal (400/409).

Plugins

A backend type that is not built in (software, pkcs11, vault, kmip) comes from a plugin: one jar behind the public key custody SPI, pinned by its SHA-256 and delivered in a signed plugin image. HelixIAM ships the cloud key store plugins below in its own plugin image; you can write another against the SPI. The server loads nothing unless you pin it (custody.plugins.jars; the chart leaves it empty), refuses to start with an unpinned jar, a mismatching pin, a writable plugin directory or a type that clashes, and lists what it loaded at GET /admin/key-custody/plugins (master-realm administrators; type, plugin and SPI version, jar file name and the first 12 hex characters of the pin) and on the master realm's Realm keys page. Every plugin backend gets the same treatment as a built-in one: the time bound, the circuit breaker, the in-flight limit, the realm binding, metrics, audit, and a self-test of every key it generates. The plugin guide, custody-plugins.md, has the contract, packaging, installing and the trust model. A plugin runs in the server process with the server's permissions: its class loader isolates its libraries, it is not a sandbox.

Cloud provider key stores (AWS KMS, Azure Key Vault / Managed HSM, Google Cloud KMS)

Three plugins, built from helix-custody-plugins/ into the plugin image, let a realm keep its keys in a hyperscaler's key service. They are a customer choice, never a default: a realm on one of them is not a sovereign-key-store realm (see Sovereign key stores only), and the key service is operated by the provider. They are Preview: they are tested against emulators and recordings built from the providers' published API shapes (aws-kms against LocalStack; azure-kv and gcp-kms against recordings and stand-ins), not yet against a real AWS, Azure or Google Cloud account. Verify them in your own account (create a throwaway realm, switch it to High assurance, sign, rotate, destroy) before production use, and check the permissions below against the provider's current documentation.

What all three share:

• Workload identity only. The pod proves who it is with the platform's workload identity (AWS IRSA or EKS Pod Identity, Azure Workload Identity, GKE Workload Identity Federation or an external_account file). No static key, client secret, key file or default credential chain is read; a setting that names one is refused. A chart settings key that looks like a credential is refused at render.
• Pinned endpoints, no environment redirection, no egress proxy. Endpoints are fixed or set explicitly; the providers' environment variables that redirect an endpoint are not read. Every HTTP client is pinned to no proxy and the JVM's proxy properties are ignored: egress through a proxy is not supported yet. The pod must reach the provider's hosts directly (networkPolicy.extraEgressPorts opens 443); where a proxy is forced, exempt those hosts from it. The settings insecureEndpointHosts and (Azure, GCP) identityEndpoint are for tests only: identityEndpoint sends the pod's identity token or credential to whatever it names, so never set it in a deployment.
• Single-purpose keys, no eID decryption. A signing key signs, a key-encryption key encrypts. No cloud backend offers RSA-OAEP key transport or sign-and-decrypt keys in this release, so an eID connection's decryption key cannot live on one (use PKCS#11, Vault/OpenBao or KMIP for that), and no backend offers EdDSA. The wallet's per-transaction ECDH key and the pepper stay software secrets.
• Keys are tagged for their realm (AWS tags helix:realm, helix:purpose, helix:kind; Azure tags and GCP labels helix-realm, helix-purpose, helix-kind), and a key tagged for another realm, one that can leave the service, or one with imported key material is never used.
• Authentication or permission denial is a 503 with lastError: policyDenied in the backend's health, never a fallback.

Key reference, chart keys and every setting are in values.yaml (the custody block's comments) and in the plugin guide. Set-up per provider follows.

AWS KMS (aws-kms)

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/helix-custody   # IRSA
custody:
  plugins: {...}                    # the aws-kms jar (a release's packaged chart already carries it)
  backends:
    - id: aws-prod
      type: aws-kms
      region: eu-central-1          # required
      auth: web-identity            # or pod-identity (EKS Pod Identity association for the service account)
      # algorithms: [RS256, PS256, ES256, ES384]   # default all four

The role's trust policy (IRSA) names the cluster's OIDC provider and exactly the HelixIAM service account:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Federated": "arn:aws:iam::123456789012:oidc-provider/<oidc issuer host and path>"},
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {"StringEquals": {
      "<oidc issuer host and path>:sub": "system:serviceaccount:<namespace>:<service account>",
      "<oidc issuer host and path>:aud": "sts.amazonaws.com"}}
  }]
}

Not yet tested against a real account; adapt and verify before use.

A least-privilege permissions policy (an example to adapt: scope the region and account, and for one realm per role replace the Null conditions with StringEquals on aws:ResourceTag/helix:realm; the keys' default key policy, made by CreateKey without a policy, delegates to IAM policies of the account):

{
  "Version": "2012-10-17",
  "Statement": [
    {"Sid": "CreateTaggedKeys", "Effect": "Allow",
     "Action": ["kms:CreateKey", "kms:TagResource"], "Resource": "*",
     "Condition": {"Null": {"aws:RequestTag/helix:realm": "false"},
                   "ForAllValues:StringEquals": {"aws:TagKeys": ["helix:realm", "helix:purpose", "helix:kind"]}}},
    {"Sid": "UseHelixKeysOnly", "Effect": "Allow",
     "Action": ["kms:DescribeKey", "kms:ListResourceTags", "kms:GetPublicKey", "kms:Sign", "kms:Encrypt",
                "kms:Decrypt", "kms:DisableKey", "kms:EnableKey", "kms:ScheduleKeyDeletion"],
     "Resource": "arn:aws:kms:eu-central-1:123456789012:key/*",
     "Condition": {"Null": {"aws:ResourceTag/helix:realm": "false"}}},
    {"Sid": "HealthProbe", "Effect": "Allow", "Action": "kms:ListKeys", "Resource": "*"}
  ]
}

Limits: AWS KMS does not offer sign-and-decrypt keys (an RSA key has one usage), no EdDSA, and with origin: AWS_CLOUDHSM (with customKeyStoreId) only key-encryption keys. The plugin pins the KMS and STS endpoints to the region's partition default, so the AWS SDK's AWS_USE_FIPS_ENDPOINT and dual-stack switches have no effect: operators who need the FIPS endpoints (for example in AWS GovCloud, us-gov-*) set endpoint and stsEndpoint explicitly (for example https://kms-fips.us-gov-west-1.amazonaws.com). endpoint and stsEndpoint also take interface VPC endpoints. A key-encryption key's encryption context carries the caller's associated data as v1.<base64url>, so it is never empty. A destroyed key goes into AWS's deletion window (deletionWindowDays, 7 to 30, default 30).

Azure Key Vault and Managed HSM (azure-kv)

podLabels: {azure.workload.identity/use: "true"}
serviceAccount:
  annotations: {azure.workload.identity/client-id: "<client id of the managed identity>"}
custody:
  backends:
    - id: azure-prod
      type: azure-kv
      vaultUrl: https://helix-prod.managedhsm.azure.net      # Managed HSM; kind: managed-hsm follows from the URL

Not yet tested against a real account; adapt and verify before use.

Federate the identity with the service account, then give it the roles (the example uses the Azure CLI; check the current syntax in the Azure documentation):

az identity federated-credential create --name helixiam --identity-name helix-custody --resource-group <rg> \
  --issuer <the cluster's OIDC issuer URL> --subject system:serviceaccount:<namespace>:<service account> \
  --audiences api://AzureADTokenExchange
# Managed HSM (local RBAC): key lifecycle and the cryptographic operations, on the key scope only
az keyvault role assignment create --hsm-name helix-prod --assignee <identity principal id> --scope /keys \
  --role "Managed HSM Crypto Officer"
az keyvault role assignment create --hsm-name helix-prod --assignee <identity principal id> --scope /keys \
  --role "Managed HSM Crypto User"

The identity needs on the keys: create, get, list, update, sign, encrypt, decrypt, delete and purge (the purge permission is only for a clean-up that is best effort: without it a deleted key stays soft-deleted until its retention ends). On a plain Key Vault with Azure RBAC, assign Key Vault Crypto Officer on the vault's resource ID. Scope to the one vault or HSM, never a subscription.

Limits: a realm needs a Managed HSM. A plain Key Vault has no AES-GCM, so it holds no key-encryption keys, and the server refuses to switch a realm to Protected or High assurance on it (both need the realm's key-encryption key); only a Managed HSM holds oct-HSM AES-256 keys. Azure Key Vault has no Ed25519 (no EdDSA) and no sign-and-decrypt keys; keys are exportable=false with no release policy. The caller's associated data is sent to Managed HSM exactly as given. The three workload-identity values AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_FEDERATED_TOKEN_FILE are read from a Java system property of that name first, then from the environment; no other AZURE_* variable is read, and the Entra ID authority is fixed by the vault's cloud (login.microsoftonline.com, login.chinacloudapi.cn, login.microsoftonline.us), never AZURE_AUTHORITY_HOST. Preview: tested against published REST API 7.5 shapes only.

Google Cloud KMS (gcp-kms)

custody:
  backends:
    - id: gcp-prod
      type: gcp-kms
      project: helix-prod
      location: europe-west4
      keyRing: helix-ring              # an existing key ring: HelixIAM creates keys in it, never the ring
      auth: gke-workload-identity      # or external-account (workload identity federation, any Kubernetes)
      # protectionLevel: HSM           # default (Cloud HSM); SOFTWARE

Not yet tested against a real account; adapt and verify before use.

A custom role with exactly the permissions the plugin uses, bound on the key ring (not the project) to the Kubernetes service account's principal (GKE Workload Identity) or to the workload identity pool principal (external-account):

gcloud iam roles create helixCustody --project=helix-prod --title="HelixIAM key custody" --permissions=\
cloudkms.keyRings.get,cloudkms.cryptoKeys.create,cloudkms.cryptoKeys.get,cloudkms.cryptoKeyVersions.get,\
cloudkms.cryptoKeyVersions.update,cloudkms.cryptoKeyVersions.destroy,cloudkms.cryptoKeyVersions.viewPublicKey,\
cloudkms.cryptoKeyVersions.useToSign,cloudkms.cryptoKeyVersions.useToEncrypt,cloudkms.cryptoKeyVersions.useToDecrypt
gcloud kms keyrings add-iam-policy-binding helix-ring --location=europe-west4 --project=helix-prod \
  --role=projects/helix-prod/roles/helixCustody \
  --member="principal://iam.googleapis.com/projects/<project number>/locations/global/workloadIdentityPools/helix-prod.svc.id.goog/subject/ns/<namespace>/sa/<service account>"

With external-account, credentialConfigFile names the external_account file (a ConfigMap or Secret; it holds no secret: gcloud iam workload-identity-pools create-cred-config writes it) and tokenAudience makes the chart project the pod's service-account token for the pool provider. A service_account key file, Application Default Credentials and executable credential sources are refused.

Limits: Cloud EKM (EXTERNAL, EXTERNAL_VPC) is not supported (each EKM key version needs an external key URI, so HelixIAM cannot generate keys there; use KMIP for an external key manager). No EdDSA (Cloud KMS signs Ed25519 over raw messages only) and no sign-and-decrypt keys. A signing key has one algorithm (the one it was made for). Cloud KMS never deletes a key's id: destroying a key schedules its version's destruction (destroyScheduledDuration, default P30D). Every request and answer is checked against Cloud KMS's CRC32C fields. Preview: tested against published REST API v1 shapes only.

For Azure and Google Cloud the FIPS level of the service is the provider's (see Compliance).

Dedicated key stores

A backend entry is one HSM partition (a token with its own PIN) or one Vault/OpenBao credential. The operator decides how many entries there are and which realms each one serves; the console only assigns a realm to an entry. Three tiers:

Tier	Set-up	What separates the tenants
Shared (default)	One entry, no realms. Every realm can be assigned to it.	HelixIAM: a key reference resolves only in its own realm, and key names embed the realm. One HSM PIN or Vault credential reaches every realm's keys.
Dedicated partition	One entry per tenant: its own HSM partition (slot, token, PIN) or its own Vault credential, limited to that realm's key names, and realms: [<realm>].	The HSM or Vault itself: the tenant's credential cannot reach another tenant's keys (on Vault/OpenBao: only while the keyCustody health shows no prefixOverlap for it, see below; a mount or namespace per tenant needs no such check). HelixIAM refuses every other realm on the entry before it calls the backend.
Customer-owned	The tenant runs the HSM or Vault and gives the operator a credential for it; configured like a dedicated partition.	As dedicated, plus the tenant controls the key store and can revoke the credential (which stops its realm signing).

Binding a backend to realms

helix.custody.backends[n].realms (chart: custody.backends[n].realms) lists the realm ids an entry serves. Absent, the entry is shared (every realm, the behaviour before rc.12). With ids, it is dedicated to them. An empty list also means shared in the server's configuration; the chart refuses an explicit realms: [] (someone writing it for "no realm yet" would get the opposite), so leave the key out for a shared entry.

Name entries neutrally (hsm-a, vault-tenant-2): an entry's id is shown to the admins of the realms it serves, and to a realm it no longer serves.

• When a realm is assigned (PUT /admin/realms/{realmId}/key-custody), an entry that does not list the realm is refused: 400 with fieldErrors.backendId and fieldErrorCodes.backendId = backendRealmNotAllowed. Nothing is made in the backend. The console lists only the entries that serve the realm (and says how many it left out).
• Every custody call checks the binding before the backend is called: a new key or key-encryption key of another realm is refused (backendRealmNotAllowed; the admin endpoints that make keys answer 400: realm key rotation and CSR, wallet verifier keys generate, credential issuer keys, eID SP keys), and any use of a key of another realm (sign, decrypt, unwrap, the KEK rotation's unwrap, public key, disable, destroy) is refused as unavailable: 503 temporarily_unavailable with Retry-After on protocol endpoints, 409 {"error":"backendRealmNotAllowed","code": "backendRealmNotAllowed"} on the custody admin endpoints (the console shows its own copy). Never a 500, never another key. The first refusal per entry and realm is logged as an error naming both.
• A key-encryption key rotation of a realm its entry no longer serves is refused before the backend is called (start, resume and abandon: 400 with code backendRealmNotAllowed; no KEK is made; the console does not offer them). A rotation running when the binding changes stops on the next pass of the rotator (FAILED, lastErrorCode backendRealmNotAllowed): nothing is unwrapped, disabled or destroyed for a realm its entry no longer serves, and the retention job does not destroy that realm's old KEK either. Put the realm back in realms, then start the rotation again to resume it, or lower the realm to Standard (below): that stays allowed, and ends the stopped rotation.
• Realm ids (the Vault/OpenBao prefix caveat below): while any entry lists realms, a new realm (admin API or import) whose id is another realm's id, or an id in some entry's realms, followed by -, or the start of one (tenant-a-eu next to tenant-a, or tenant next to tenant-a), or that differs from one only in case, is refused: 400 with fieldErrors.realmId and fieldErrorCodes.realmId = realmIdPrefix (an import reports the realm entry as failed with that code; the console shows its own text, which names no other realm). Two realms created at the same moment cannot both pass: the rules are checked again under a cluster-wide realm-creation lock. Compared ignoring case.
• Realm ids that overlapped before an entry was bound (or that a later realms change makes overlap) are reported by the binding check (at start-up and every helix.custody.binding-check-interval, default PT1M), in the keyCustody health (prefixOverlaps: {"<bound realm>": ["<other realm>", ...]}) and in the master-only GET /admin/key-custody/backends (prefixOverlaps: [{"realm", "overlapsWith", "isolationEnforced"}]):
• when the other realm's keys are in the same key space (a Vault/OpenBao entry with the same address, mount and namespace), the bound entry is DOWN with realmBinding: {"<realm>": "prefixOverlap"} and isolationEnforced: false: the tenant policy helix-<realm>-* also reaches the other realm's keys, so the dedicated tier's guarantee does not hold for it;
• otherwise it is a warning (warnings: {"prefixOverlap:<realm>": "<other realms>"}), the entry stays UP.

Remediation: give the tenant a Transit mount or namespace of its own (entry setting mount or namespace; then no prefix matters), or move the other realm's keys out of that key space, or replace the overlapping realm with one whose id does not overlap. An HSM is never affected (its labels are helix:<realm>:..., and : is not in a realm id). - At start-up and then every helix.custody.binding-check-interval, every Protected or High-assurance realm assigned to an entry that no longer lists it (an operator changed realms) is named in the entry's realmBinding: GET /admin/key-custody/backends lists the entry with healthy: true (the backend itself answers) and realmBinding: {"<realm>": "notAllowed"}, and the aggregate /actuator/health is DOWN (the per-backend detail of the keyCustody component is shown only where health details are enabled). Its custody keys are refused as above, and its wrapped keys never keep the pod from becoming ready; every other realm keeps working. A Standard realm on such an entry is named only while a key of it that can still sign or verify lives there (see below). A realm id in realms that no realm has yet is allowed (realms can be created later) and logged as a warning. - A realm id that is not a realm id (1-100 letters, digits, ., _ or -, starting with a letter or digit), a realm listed twice, or realms on the software backend is refused at start-up; the chart refuses the first two, and an explicit empty list, at render time. - The binding is read once per pod. During a rollout that changes realms, old and new pods disagree: a pod that does not serve the realm answers 503 for it, and its rotator stops a KEK rotation that a new pod started (FAILED, backendRealmNotAllowed, nothing disabled). Both fail closed; assign realms and start KEK rotations only once every pod runs the new realms. - Do not reuse a bound realm id for another customer. The id is in the entry's realms and in the tenant policy (helix-<realm>-*), so a new realm with the same id gets the old realm's entry and Vault policy, and with them any of the old realm's keys still in the backend.

GET /admin/key-custody/backends (master-realm admins) returns each entry's isolation (shared or dedicated), realms (ids only, never a setting), and, when there is something to say, realmBinding, prefixOverlaps and warnings. A realm's view, GET /admin/realms/{realmId}/key-custody, which the realm's own admins read too, returns backendIsolation (level, realmCount, dedicatedToRealm, allowsRealm) and never the other realms' ids. The console shows it as Shared key store, Dedicated to this realm or Dedicated to N realms (in Dutch the section is "Sleutelbewaring", custody, and an entry a "sleutelopslag", key store).

Taking a realm off an entry that no longer serves it. Either put the realm back into realms and restart, or lower the realm to Standard. Lowering asks nothing of the entry, and is allowed also while a KEK rotation stopped because the entry no longer serves the realm (it ends that rotation, ABANDONED). After the lowering the realm is released on the entry: its keys there may be ended (disabled, destroyed), never used or made. Then:

1. The realm's new software keys sign once their publish window has passed (or at once when activated). Its old custody keys stay published, refused, until their tokens have expired; retire them (Realm keys; an emergency retire for keys whose tokens you do not need).
2. Once no key of the realm that can sign or verify lives on the entry, the binding check no longer names it: the keyCustody health is UP again, without a restart. Each pod notices at its own next binding check (helix.custody.binding-check-interval, default PT1M), not at once: for up to a minute after the last custody key is retired one pod can still be DOWN while another is UP.
3. The retention job disables and destroys the retired keys there. The realm's key-encryption key(s) there (its KEK, one a rotation was moving off, and the new and superseded KEKs of its abandoned rotations) are released in stages. Each stage is re-checked under the realm key lock: still Standard with the same KEK, still not served by the entry, no key there that can sign or verify (a wrapped key that is not RETIRED, with or without a recorded KEK), and no unfinished KEK rotation:
4. the first retention run that sees the realm released records it (UTC; audit CUSTODY_KEK_RELEASE_START);
5. helix.custody.destroy-after later the KEK(s) are disabled (CUSTODY_KEY_DISABLE);
6. helix.custody.destroy-after after the disable they are destroyed (CUSTODY_KEY_DESTROY; one another pod destroyed already counts as destroyed) and the entry is forgotten (backendId becomes null). The realm can then be raised again on another entry.

From the disable until the destroy is done, a profile change of the realm is refused (409, code kekReleasePending); the console shows the stage in Realm keys → Key custody. A pod that serves the realm again (a realms typo fixed) cancels a release that has not started to destroy: the KEKs this release disabled (the realm's KEK, and the one a rotation was moving off) are enabled again (CUSTODY_KEY_ENABLE), never a KEK a rotation disabled on purpose (an abandoned or quarantined one), and the release is forgotten (CUSTODY_KEK_RELEASE_CANCEL); a later release starts the clock again. A disable that a cancel overtook is undone by the pod that made it. Once the destroy has started (DESTROYING), any pod that configures the entry finishes it, also one that serves the realm again. While pods disagree (a rollout), each pod that serves the realm cancels, so nothing is disabled before every pod has stopped serving it for a full destroy-after. A KEK that is the configured DB_ENCRYPTION_KEK (any version), or is named like one (helixdbkek-, :DBKEK:), is never released, disabled or destroyed: an error is logged instead. Re-enabling works on Vault, OpenBao and PKCS#11.

When the enable is refused for good (an HSM policy that does not allow CKA_ENCRYPT / CKA_DECRYPT back, or a credential that may no longer write the key's config), the release stays and the profile change stays refused; the pod logs an error after three refused attempts. The way out: enable the key in the backend by hand, or remove the realm from the entry's realms again (a restart), so that the release goes on to destroy the KEK as above; then the realm can be raised on another entry.

When no pod can finish a destroy (the entry was removed from the configuration while the release was at DESTROYING), a master-realm administrator finishes it: Realm keys → Key custody → Finish key release (typing the realm id), or POST /admin/realms/{realmId}/key-custody/kek-release/finish. Where the server configures the entry the KEKs are destroyed; otherwise they are only forgotten, and the answer (and the console) names them to remove in the backend by hand. Audited CUSTODY_KEK_RELEASE_FINISH; 409 kekReleaseNotDestroying when there is nothing to finish.

If the entry's credential can no longer reach the realm's keys (the operator also narrowed its policy), each disable or destroy is an unavailability retried on every run; remove those keys in the backend by hand, or give the credential back its access until the retention job is done.

The DB_ENCRYPTION_KEK is the operator's key, not a realm's, and belongs on an entry shared by every realm (no realms): a tenant's credential must never open the database encryption key. DbKeyWrapTool --create-kek on an entry with realms, and a new wrap (--kek, or the new KEK of --rewrap-from) by a KEK on such an entry, are refused (createDbEncryptionKek refuses it too, code dbEncryptionKekNeedsSharedBackend). Moving the data key off such a KEK (--rewrap-from it) stays possible. A server whose DB_ENCRYPTION_KEK is on an entry with realms still starts (it must open its own database), logs a warning, and the keyCustody health names it (warnings: {"dbEncryptionKek": "onBoundEntry"}, the entry stays UP): move the data key to a KEK on a shared entry.

High assurance on a dedicated key store

helix.custody.require-dedicated-backend-for-high (chart: custody.requireDedicatedBackendForHigh), default false. When true, a High-assurance realm must be on an entry whose realms is exactly that realm:

• a switch to High assurance on any other entry (shared, or dedicated to several realms) is refused: 400 with fieldErrorCodes.backendId = backendNotDedicated; the console disables those entries for High assurance and says why;
• a High-assurance realm already on such an entry is flagged at start-up: the aggregate /actuator/health is DOWN, and the backends list shows that entry with realmBinding: {"<realm>": "notDedicated"}. The realm keeps signing (the setting is a policy check, not a key fault).

A High-assurance realm on a shared entry gets a calm advisory in the console either way.

Upgrading from rc.11. The setting is off by default, so High-assurance realms on a shared entry keep working. To turn it on: give each High-assurance realm an entry of its own first. Because a realm stays on its entry, the move is: lower the realm to Protected or Standard, wait for its new keys to sign, raise it again on its new entry, then set the setting. The health shows any realm still on a shared entry.

Per-tenant recipe: Vault and OpenBao

One AppRole (or token) per tenant, whose policy covers only that realm's key names. With realm tenant-a:

# helixiam-tenant-a.hcl: only tenant-a's signing keys, decryption keys and key-encryption keys
path "transit/keys/helix-tenant-a-*" {
  capabilities      = ["create", "read", "update", "delete"]
  denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] }
}
path "transit/keys/helixdec-tenant-a-*" {
  capabilities      = ["create", "read", "update", "delete"]
  denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] }
}
path "transit/keys/helixkek-tenant-a-*" {
  capabilities      = ["create", "read", "update", "delete"]
  denied_parameters = { "exportable" = [], "allow_plaintext_backup" = [] }
}
path "transit/sign/helix-tenant-a-*"       { capabilities = ["update"] }
path "transit/sign/helixdec-tenant-a-*"    { capabilities = ["update"] }
path "transit/encrypt/helixkek-tenant-a-*" { capabilities = ["update"] }
path "transit/decrypt/helixkek-tenant-a-*" { capabilities = ["update"] }
path "transit/decrypt/helixdec-tenant-a-*" { capabilities = ["update"] }
# Only while tenant-a's key-encryption key has the name of before rc.12 (helix-tenant-a-kek-<uuid>): its wrapped keys
# are unwrapped with it. Rotate that KEK (it becomes helixkek-tenant-a-...), then remove these two lines.
path "transit/encrypt/helix-tenant-a-kek-*" { capabilities = ["update"] }
path "transit/decrypt/helix-tenant-a-kek-*" { capabilities = ["update"] }

vault policy write helixiam-tenant-a helixiam-tenant-a.hcl
vault write auth/approle/role/helixiam-tenant-a token_policies=helixiam-tenant-a token_ttl=1h token_max_ttl=24h
vault read -field=role_id auth/approle/role/helixiam-tenant-a/role-id          # -> auth.role of entry vault-tenant-a
vault write -f -field=secret_id auth/approle/role/helixiam-tenant-a/secret-id  # -> its own Kubernetes Secret

• The realm id in the policy must be the real realm id. The example uses tenant-a as the realm id; a policy that names another id than the realm's (a copy of the example for realm acme, say) denies every call for that realm's keys. HelixIAM answers 503 (the backend is unusable from this server), never a 500, and says so in two places: one WARN per backend and key name in the server log (rate limited, no secrets) that names the backend id and the key name and says the call was denied by the backend's policy, and lastError: policyDenied in that backend's entry of the keyCustody health, until a call for that key works again. Fix the policy's key-name paths; no restart is needed.
• Realm ids that are prefixes of each other. helix-tenant-a-* also matches the keys of a realm tenant-a-eu (helix-tenant-a-eu-...). While an entry lists realms, HelixIAM refuses to create such a realm (realmIdPrefix, see Binding a backend to realms); realms that already overlapped are named in the keyCustody health (DOWN when they share the key space) and the backends list. Key names were not changed instead: a realm id may contain -, . and _, so no character of a Vault key name can end a realm id unambiguously, and existing keys and policies keep working. The stronger form below needs no rule at all.
• Stronger: a mount or namespace per tenant. A Transit mount per tenant (vault secrets enable -path=transit-tenant-a transit, entry setting mount: transit-tenant-a, policy paths under transit-tenant-a/), or a Vault Enterprise namespace per tenant (entry setting namespace). Then no prefix matters, and the tenant's keys are not even listed to another credential.
• Tested (rc.12, Vault 2.2.1 and OpenBao 2.7.2): two AppRoles on one mount, each with the policy above for its realm (on Vault each entry bound to its realm). Tenant A's credential creating or using tenant B's key names (signing keys and KEKs, also the KEK rotation's unwrap) gets the backend's 403, which HelixIAM answers as an unavailability (503 on request paths), never a 500 and never a key.

Per-tenant recipe: HSM (PKCS#11)

One partition (slot, token) per tenant, each with its own user PIN in its own Kubernetes Secret, one entry each. On SoftHSM (for a test; a network HSM has its own partition tooling):

softhsm2-util --init-token --free --label tenant-a --so-pin "$SO_PIN_A" --pin "$PIN_A"
softhsm2-util --init-token --free --label tenant-b --so-pin "$SO_PIN_B" --pin "$PIN_B"
kubectl create secret generic hsm-tenant-a-pin --from-literal=pin="$PIN_A"
kubectl create secret generic hsm-tenant-b-pin --from-literal=pin="$PIN_B"

Each tenant's keys and key-encryption key are made on its own token only; another entry's PIN cannot reach them. Tested (rc.12, SoftHSM 2.6): two tokens in two slots with different PINs, each bound to its realm.

Helm example: two tenants

custody:
  requireDedicatedBackendForHigh: true
  backends:
    - id: vault-tenant-a
      type: vault
      address: https://vault.internal:8200
      auth: {kind: approle, role: "<role id of helixiam-tenant-a>",
             secretIdSecret: {name: vault-tenant-a-secret-id, key: secret-id}}
      caSecret: {name: vault-ca, key: ca.crt}
      realms: [tenant-a]
    - id: hsm-tenant-b
      type: pkcs11
      library: /opt/hsm/lib/libsofthsm2.so
      tokenLabel: tenant-b
      pinSecret: {name: hsm-tenant-b-pin, key: pin}
      realms: [tenant-b]
    - id: vault-shared            # Standard and Protected realms of other tenants
      type: vault
      address: https://vault.internal:8200
      auth: {kind: approle, role: "<role id of helixiam>", secretIdSecret: {name: vault-secret-id, key: secret-id}}
      caSecret: {name: vault-ca, key: ca.crt}

Then, as a master-realm administrator, move each tenant onto its entry (console: Realm keys → Key custody → Change profile, or PUT /admin/realms/tenant-a/key-custody {"profile": "HIGH", "backendId": "vault-tenant-a"}).

Sovereign key stores only

A realm can be restricted to sovereign key stores: key stores the customer or the operator controls, never a hyperscaler's key service. It is a per-realm policy with two values, any (the default, the behaviour before rc.13) and sovereign.

What counts as sovereign. Every configured backend has a class:

Class	Backend types
Sovereign	pkcs11, vault (Vault and OpenBao), kmip, and any plugin type the operator lists in helix.custody.sovereign-types (chart: custody.sovereignTypes)
Cloud provider (hyperscaler)	aws-kms, azure-kv, gcp-kms, always, even if listed
Software	software
Partner (plugin)	any other plugin type

Software keys do not count as sovereign. A sovereign realm cannot be in the Standard profile (its keys would be software keys in the database, and the server refuses to lower such a realm to Standard), and it cannot import a software key. It must be Protected or High assurance on a sovereign backend. A type in sovereign-types is your own assertion about that plugin's key store: HelixIAM does not verify it. The server refuses to start when the list names a cloud provider type, software, or a malformed name.

Setting it. Only administrators of the master realm can set or lower the policy (403 for anyone else); the console shows it as a choice on Realm keys → Key custody, where the key-store picker then offers sovereign stores only. API:

# set the policy on its own (200 {"realmId", "keyStorePolicy", "changed"})
curl -X PUT "$BASE/admin/realms/acme/key-custody/key-store-policy" -H 'Content-Type: application/json' \
  -d '{"policy": "sovereign"}'
# or with the profile switch: {"profile": "HIGH", "backendId": "hsm-1", "keyStorePolicy": "sovereign"}
curl -X PUT "$BASE/admin/realms/acme/key-custody" -H 'Content-Type: application/json' \
  -d '{"profile": "HIGH", "backendId": "hsm-1", "keyStorePolicy": "sovereign"}'
# the realm's policy and the class of its backend (any realm admin may read it)
curl "$BASE/admin/realms/acme/key-custody"      # keyStorePolicy, backendStoreClass
# the backends this installation offers, with their class (master-realm admins)
curl "$BASE/admin/key-custody/backends"         # storeClass: software | sovereign | hyperscaler | plugin

What is refused or enforced.

• Setting sovereign on a realm that is Standard, or whose backend is not sovereign, or that has any live key on another key store, answers 409 (keyStorePolicyConflict). This includes a key that is only still in its grace period (rotated, not yet retired) on the old, non-sovereign backend: it blocks the policy until that key is retired. Software keys left in their grace period by the switch from Standard (verify-only, never used to sign again) do not block it. Move the realm first, then let the old keys retire (or retire them), then set the policy. This check covers every key family of the realm: realm keys, wallet verifier keys, issuer keys and eID SP keys.
• With the policy set, a switch to a non-sovereign backend, a lowering to Standard, an import of a software key and a new key on a non-sovereign backend are refused with the code keyStorePolicy. Retiring (disabling, destroying) a key stays allowed.
• At run time a key of a sovereign realm on a non-sovereign backend is never used (503, the backend is not called), and the keyCustody health goes DOWN naming the realm (also for a sovereign realm listed in the realms of a non-sovereign entry). This can only arise from a direct database edit or a configuration change; a correction clears it at the next check.
• Realm export carries the policy (keyCustody.keyStorePolicy) and never the backend; an import is checked against it before anything is written.
• Lowering the policy to any is also master-realm only. Every change is audited as CUSTODY_KEY_STORE_POLICY (from, to).

The policy is a guard on which stores the server will use, not a statement about the stores: whether a store is sovereign in the legal sense (who operates it, under which jurisdiction) is the operator's decision when it lists a plugin type, and the customer's when it picks Vault, an HSM or a KMIP key manager.

Moving a realm

Console: Realm keys → Key custody → Change profile (master-realm administrators only; others see the profile read-only). API: sign in as a master-realm administrator on the admin host (see Authenticating to the API):

# The backends this installation offers: id, type, algorithms, healthy, latencyMs (never their settings)
curl -s -b cookies.txt "$HELIX_URL/admin/key-custody/backends"
# A realm's profile, backend health and keys (purpose, algorithm, custody, keyRef, status)
curl -s -b cookies.txt "$HELIX_URL/admin/realms/$REALM/key-custody"
# Move it: 202 with the planned keys and when each starts signing
curl -s -b cookies.txt -X PUT -H "X-XSRF-TOKEN: $CSRF" -H "Content-Type: application/json" \
  -d '{"profile":"HIGH","backendId":"hsm-1"}' \
  "$HELIX_URL/admin/realms/$REALM/key-custody"

profile is STANDARD, PROTECTED or HIGH. What a move does, in order:

1. Checks the backend: known, not software, serves the realm (its realms, backendRealmNotAllowed; for High assurance with helix.custody.require-dedicated-backend-for-high, dedicated to the realm alone, backendNotDedicated), reachable (otherwise 503), holds key-encryption keys, non-extractable keys for High assurance, and signs every algorithm the realm's purposes use (High assurance also needs PS256 for RSA). A refusal is 400 with fieldErrors.backendId naming the algorithm and purposes, e.g. cannot sign ES256 (access, id, saml keys). Then checks what in the realm the target profile must serve, each refusal a 400 with fieldErrors.profile naming what conflicts: an enabled SAML service provider that asks for RSA-SHA1 (any target other than Standard) or RSA-SHA512 (High assurance, whose backend does not sign RS512), and, for High assurance, an enabled wallet connection that still signs with a software verifier key. The issuer, eID and metadata keys add more (see High-assurance rules for the issuer, eID and metadata keys): an enabled credential issuer when the target backend cannot sign ES256 (Protected and High assurance), and, for High assurance, an ACTIVE or PENDING software issuer key, a live development CA, an enabled eID connection that still stores a pasted SP key, and an enabled eID connection the backend cannot serve (no signature algorithm of its scheme, no RSA size at the scheme's minimum, no key transport it decrypts, or a live custody key whose transport it cannot decrypt).
2. Creates the realm's own KEK in the backend (one per realm; its reference is stored on the realm, it is a public reference, not a secret).
3. Stores the profile and schedules a rotation of every purpose and family into the new custody. The new keys are published at once and start signing after the JWKS cache window (the same publish-before-use windows as any rotation), so relying parties have the new keys before they see them. Activate earlier only when nobody can have cached the realm's JWKS (POST /admin/realms/{realm}/keys/{keyId}/activate?force=true).
4. The old keys stop signing when the new ones start and stay published until their tokens have expired.
5. Audits CUSTODY_PROFILE_CHANGE (and CUSTODY_KEY_GENERATE per new key).

Lowering a profile works the same way (the console asks for confirmation); lowering to Standard needs DB_ENCRYPTION. The realm keeps its KEK, so its wrapped keys still decrypt while they are published. A realm already in custody cannot be moved to another backend in this release.

Day-to-day key work

Day-to-day key work stays with the realm's own administrators (realm-admin on that realm):

• Rotate (POST /admin/realms/{realm}/keys/rotate), with the usual pending window.
• Emergency rotate ({"emergency": true}) or emergency retire (DELETE /admin/realms/{realm}/keys/{keyId}?emergency=true): the new key signs at once and the old key is disabled in the backend at once (CUSTODY_KEY_DISABLE), without waiting for the platform operator. See Emergency rotation and caches.
• Rotation is per algorithm. A rotate request rotates one algorithm family, in every purpose (access, id, saml) that holds a key of it. Without algorithm it is RS256: {"emergency": true} replaces only the RSA keys. The ES256 keys need {"emergency": true, "algorithm": "ES256"} (EdDSA: "algorithm": "EdDSA"). To replace every key of a realm, send one request per algorithm its keys use (the algorithm column of GET /admin/realms/{realm}/keys).
• CSR and certificate for keys that need a CA-issued certificate (DigiD, eHerkenning, PKIoverheid SAML signing): POST /admin/realms/{realm}/keys/csr with a purpose (all, access, id or saml; empty means all) generates the key in the realm's custody and returns the CSR, signed inside the backend; upload the issued chain with PUT /admin/realms/{realm}/keys/{keyId}/certificate. In Protected and High assurance such a key cannot be activated before its certificate is attached.

curl -s -b cookies.txt -X POST -H "X-XSRF-TOKEN: $CSRF" -H "Content-Type: application/json" \
  -d '{"purpose":"saml","algorithm":"RS256","subject":"CN=idp.example.com, O=Example, C=NL","dnsNames":["idp.example.com"]}' \
  "$HELIX_URL/admin/realms/$REALM/keys/csr" | jq -r .csrPem > saml.csr

Retired backend keys are disabled at retirement, and destroyed by the retention job helix.custody.destroy-after (default P30D) after they were disabled; the job runs every helix.custody.retention-check-interval (default PT1H) and is audited (CUSTODY_KEY_DISABLE, CUSTODY_KEY_DESTROY, actor system:master). There is no destroy endpoint.

Requests nobody certifies are bounded. A realm holds at most helix.custody.max-pending-keys-per-realm (default 5) CSR keys waiting for their certificate, and at most as many generated verifier keys waiting for theirs; one more is refused with 400 (attach a certificate to a waiting key, or retire one, first) before any backend is called. A CSR key still without a certificate helix.custody.pending-key-ttl (default P30D, because a CA can take weeks to issue) after it was made is retired by the retention job (CUSTODY_KEY_EXPIRE), then disabled in its backend and destroyed after helix.custody.destroy-after like any retired key. A generated verifier key still without a certificate after the same time is expired the same way (CUSTODY_KEY_EXPIRE, resource verifier-key): it no longer counts toward the limit, is no longer listed and takes no CSR or certificate; it is disabled in its backend at once (CUSTODY_KEY_DISABLE) and destroyed there after helix.custody.destroy-after (CUSTODY_KEY_DESTROY), when its name becomes free again. So abandoned requests never lock a realm out of new keys. If your CA needs longer, raise helix.custody.pending-key-ttl.

A key that can leave the backend

A Vault or OpenBao key whose exportable or allow_plaintext_backup flag was set out of band (the documented policy denies HelixIAM itself both), or an HSM key whose PKCS#11 attributes say it can leave the token or was not generated on it (CKA_EXTRACTABLE true, CKA_SENSITIVE false, CKA_NEVER_EXTRACTABLE or CKA_ALWAYS_SENSITIVE false, CKA_LOCAL false, or not a token object), is quarantined on the server that finds it:

• Every request that needs that key answers like any custody outage: 503 temporarily_unavailable with Retry-After, never a 500 and never another key (token endpoint, SAML SSO, credential issuance, eID logins).
• keyCustody health shows the backend DOWN, with a detail that names the key, for example quarantined key(s) that can leave the backend (exportable or plaintext backup allowed) still in use: helix-acme-access-<uuid> (realm acme); …, and /actuator/health is DOWN (503). Readiness is not affected.
• Only that key is affected. The backend is still reachable: its circuit breaker does not open, it still signs with its other keys, and it still makes new keys and disables the quarantined one. A backend that cannot be reached is a different condition (every request that needs it answers 503, see Failure behaviour).

Recover in this order, on any pod and without a restart:

1. Close the hole first. Vault/OpenBao: make sure the policy of every token that can write keys/<name>/config denies both parameters (the policy above), and find out from Vault's audit device who set the flag. A flag set to true can never be set back, so the key itself cannot be repaired. HSM: find out who could create or replace objects on the partition (its user PIN), and change that PIN.
2. Replace the key. For a realm signing key, an emergency rotation of the key's algorithm: POST /admin/realms/{realm}/keys/rotate with {"emergency": true, "algorithm": "<the key's algorithm>"} (the detail names the key; GET /admin/realms/{realm}/keys shows its algorithm). Without algorithm only RS256 is rotated. For an issuer key or an eID SP key, make a new key and retire the quarantined one. Either way the quarantined key is disabled in the backend at once.
3. Health is UP again within the probe window (about 6 s) once no quarantined key can still sign, encrypt or decrypt (HSM: CKA_SIGN false, and CKA_DECRYPT false for an RSA key, as the rotation's disable sets them; or the object destroyed). Each pod reads the quarantined key's state from the backend, so a rotation on one pod clears every pod. Tokens are signed by the new key at once. Detection is per pod: each pod finds the flag on its own next use of the key or its own health check, so a few seconds of difference between pods are expected, and until a pod has found it that pod still signs with the key. Emergency rotation is the remedy.
4. The quarantined key stays refused on every pod that saw it, and the retention job destroys it after helix.custody.destroy-after, like any retired key.

A quarantined key-encryption key (KEK) of a Protected realm is replaced by a KEK rotation: see Rotating a key-encryption key, "A KEK that may have left the backend". Until the rotation is done, a pod that already holds the realm's keys keeps signing with them; a pod that must unwrap a key with the quarantined KEK answers 503 for it, and health stays DOWN.

Rotating a key-encryption key

A Protected realm's signing keys are stored wrapped by the realm's KEK. A KEK rotation creates a new KEK in the realm's backend, wraps every key again under it, and then disables the old KEK. Rotate it on a schedule your policy sets, after a backend operator with access to the KEK leaves, or at once when the KEK may have left the backend (it is quarantined). Only a Protected realm's KEK is rotated: a High-assurance realm keeps its keys inside the backend, and a Standard realm has none (both answer 409).

Console: Realm keys → Key custody → Rotate key-encryption key (master-realm administrators only; the dialog asks you to type the realm id). The section then shows the progress, a pause while the backend cannot be reached, and the outcome. API, as a master-realm administrator on a master host:

# start (202 with the rotation); a second call while it runs answers 200 with the same rotation
curl -s -X POST https://iam.example.com/admin/realms/tenant-a/key-custody/kek-rotation -H "Authorization: Bearer $TOKEN"
# progress: {"rotation": {"state": "REWRAPPING", "rowsDone": 5, "rowsTotal": 12, ...}}; any admin of the realm may read it
curl -s https://iam.example.com/admin/realms/tenant-a/key-custody/kek-rotation -H "Authorization: Bearer $TOKEN"
# abandon a stopped rotation that moved no key (200; 409 with a code otherwise)
curl -s -X DELETE https://iam.example.com/admin/realms/tenant-a/key-custody/kek-rotation -H "Authorization: Bearer $TOKEN"

GET /admin/realms/{realmId}/key-custody also carries backendReachable (false only when the backend cannot be reached; a backend DOWN because of a quarantined key is reachable) and kekQuarantined (true when the quarantined key is this realm's KEK). The console uses them to tell an outage from a quarantined KEK, and offers the rotation for the latter.

What a rotation does, in order:

1. Checks that the realm is Protected and its backend reachable (409 with code kekRotationNotProtected or kekRotationNoKek, also for a stored reference that is malformed or of another realm; 409 kekRotationDbEncryptionKek when the realm's KEK is also DB_ENCRYPTION_KEK, see DB_ENCRYPTION under a KEK; 503 when the backend does not answer; a backend that is DOWN only because the KEK is quarantined is reachable and passes). Creates the new KEK in the realm's backend (helixkek-<realm>-<uuid> on Vault and OpenBao, an AES-256 token key helix:<realm>:KEK:<uuid> on an HSM), then records the rotation and makes the new KEK the realm's KEK in one transaction: every key made from then on is wrapped by the new KEK. Audited CUSTODY_KEK_ROTATION_START and CUSTODY_KEY_GENERATE (resource realm-kek).
2. Wraps the realm's keys again, in the background, in batches of helix.custody.kek-rotation.batch-size (default 16): every wrapped key that is ACTIVE, PENDING or ROTATED (purposes access, id, saml, every algorithm). Each key is unwrapped with the KEK its row names, wrapped by the new KEK, and the new value is decrypted again before it replaces the old one, only if the row is unchanged. Tokens keep being signed throughout: every pod unwraps each key with the KEK its row names, old or new. A RETIRED key is not wrapped again; it never signs again, and destroying the old KEK makes it unreadable for good.
3. When no key needs the old KEK any more (counted again under the realm lock right before; a key that still names it sends the rotation back to step 2), disables it in the backend: it neither encrypts nor decrypts (Vault and OpenBao: min_encryption_version and min_decryption_version above its version; HSM: CKA_ENCRYPT and CKA_DECRYPT false). The rotation is DONE; audited CUSTODY_KEK_ROTATION_COMPLETE and CUSTODY_KEY_DISABLE.
4. The retention job destroys the old KEK helix.custody.destroy-after (default P30D) after it was disabled (CUSTODY_KEY_DESTROY, resource realm-kek), unless a key that can sign or verify names it again, or a wrapped key of the realm records no KEK (written by a server of an earlier release; it may be under it): an old KEK is never destroyed while the realm has such a key. A start whose commit reports a failure never disables the new KEK unless a fresh read shows nothing uses it (the commit may have gone through); the same holds for the KEK a failed profile switch made.

While a rotation is unfinished (running, paused or stopped), the realm's custody profile cannot be changed (409, code kekRotationRunning; the console disables Change profile with the reason).

Several pods. Every pod runs the rotation worker (every helix.custody.kek-rotation.interval, default PT5S); one pod at a time holds a rotation's lease (helix.custody.kek-rotation.lease, default PT60S, renewed before every key, so a slow backend never lets another pod take over a batch that is moving) and does the work, the others keep serving. A pod that stops mid-rotation (a crash, a rollout) leaves every key unwrappable: its lease runs out and another pod continues where it stopped. A pod that does not configure the realm's backend leaves the rotation to one that does. A pod whose entry no longer serves the realm (helix.custody.backends[n].realms) stops it (FAILED, backendRealmNotAllowed) without disabling anything; see Binding a backend to realms.

Backend outage. A backend that does not answer pauses the rotation: the status endpoint answers 503 temporarily_unavailable with Retry-After, code kekRotationPaused and the rotation with its progress (any other 503, such as overload shedding, has no such code; the console shows "Paused" with the progress), and the rotation continues on its own once the backend answers. Keys already wrapped again and keys not yet wrapped again both keep signing, also on a pod that starts while the realm is half rotated.

A rotation that stops (state FAILED, with lastErrorCode and lastError): unwrapRefused names keys whose ciphertext the old KEK does not open (they cannot sign either): retire them, then start the rotation again; wrapRefused or disableRefused: the backend refused the new KEK's encrypt (or the new value did not decrypt back to the same key) or the old KEK's disable, so check its policy, then start again; dbEncryptionKek: the old KEK is DB_ENCRYPTION_KEK and is never disabled, so move that first; backendRealmNotAllowed: the realm's entry no longer serves it (nothing was disabled), so put the realm back in realms, then start again; newKekQuarantined: the new KEK became quarantined during its own rotation. Starting again (console: Resume rotation) continues the same rotation where it stopped; after newKekQuarantined it creates a fresh KEK, the keys already on the quarantined one are moved once more, and the quarantined one is disabled with the old KEK at the end and destroyed with it. A stopped rotation that moved no key can be abandoned (DELETE, console Abandon rotation): the realm's KEK is the earlier one again, the rotation is ABANDONED, the new KEK is disabled; once a key is on the new KEK, abandoning is refused (409 kekRotationNewKekInUse) and the rotation has to continue. Abandoning is refused too (409 kekRotationUnlabelledKeys) while the realm has a wrapped key without a recorded KEK: a server of an earlier release wrote it during the rotation, possibly with the new KEK; resuming lets the rotator find its KEK and move it. Audited CUSTODY_KEK_ROTATION_ABANDON.

A KEK that may have left the backend (quarantined: made exportable or plaintext-backup-able out of band, or an HSM key that can leave the token). The rotation is allowed to decrypt with it, only to move the keys off it, and never wraps with it. In this order:

1. Close the hole (policy, PIN), as for any key that can leave the backend.
2. Rotate the KEK and wait for DONE. Health is UP again within the probe window once the old KEK is disabled.
3. The wrapped keys may be known to whoever could export the KEK (with a copy of the database). Emergency-rotate the realm's signing keys, one request per algorithm the realm uses (see "Rotation is per algorithm" above); their new keys are wrapped by the new KEK. Then purge caches as in Emergency rotation and caches.

Vault and OpenBao policy. No new lines: the helixkek-* lines of the minimal policy already allow creating the new KEK, encrypting and decrypting with both KEKs, disabling the old one (keys/<name>/rotate and keys/<name>/config) and destroying it. The contract tests run a full rotation with exactly that policy on Vault 2.2.1 and OpenBao 2.7.2. A KEK with the earlier name (helix-<realm>-kek-<uuid>) is rotated to a helixkek- KEK; once its rotation is DONE (and it is destroyed), that realm's transit/encrypt|decrypt/helix-<realm>-kek-* lines are no longer needed. HSM: the user PIN's session creates the new AES key and sets the old one's CKA_ENCRYPT and CKA_DECRYPT to false, as an emergency rotation already does for signing keys.

Metrics: helix_custody_kek_rewrap_total{result} (ok, unavailable, refused) and helix_custody_kek_rotations_active (running rotations, as this pod last saw them). Alert on a rotation that stays paused, and on result="refused".

Tested end to end on Vault 2.2.1, OpenBao 2.7.2 and SoftHSM 2.7.0: a Protected realm with keys of every purpose and family (RSA, P-256, Ed25519; ACTIVE, PENDING, ROTATED) rotated through the admin API, every key signing afterwards also on a cold pod and the old KEK disabled in the backend; on Vault also a quarantined KEK, an outage mid-rotation, two pods, a pod that stops mid-rotation, keys made while the rotation runs, and a pod whose session held a row before it was re-wrapped (on OpenBao also the quarantined KEK). A quarantined new KEK, abandoning, racing starts and an ambiguous commit are tested on PostgreSQL with an in-memory backend. Not run against a network HSM.

Emergency rotation and caches

An emergency rotation or retirement removes the compromised key from the realm's JWKS on every replica at once. Copies that relying parties, gateways and CDNs keep are a different matter: a cache that honours the JWKS headers may keep accepting tokens signed with the removed key until its copy expires. That is max-age + stale-while-revalidate (65 minutes by default), plus stale-if-error (default 5 minutes) while the server answers with errors. An attacker who holds the key gains from making the server fail, which is why stale-if-error is short.

What HelixIAM does: for that whole window (max-age + stale-while-revalidate + stale-if-error after the removal) the realm's JWKS and discovery documents are sent Cache-Control: no-cache. A cache that fetches them in that time revalidates on every use (a cheap 304 while nothing changed) and never serves them stale. A copy cached before the emergency is not reached by that header.

What the operator does, right after the emergency rotation:

1. Purge the CDN or edge cache for /realms/<realm>/oauth2/jwks and the realm's discovery documents (/realms/<realm>/.well-known/openid-configuration, /realms/<realm>/.well-known/oauth-authorization-server, /.well-known/oauth-authorization-server/realms/<realm>, /realms/<realm>/.well-known/oauth-protected-resource), on every host the realm is served from.
2. Ask relying parties that cache the JWKS themselves (API gateways, resource servers with a long JWKS cache) to refresh it, or restart them.
3. Check that the removed kid is gone:

curl -s "$HELIX_URL/realms/$REALM/oauth2/jwks" | jq -r '.keys[].kid'

A JWKS that cannot be built on a pod (its keys cannot be read, a cold start against a failed-over database) is answered 503 with Cache-Control: no-store and Retry-After, never an empty key set that a cache would keep.

Wallet verifier keys

In the wallet provider form, Generate in HSM/KMS creates an ES256 or ES384 verifier key (only EC, as HAIP requires) in the realm's assigned backend: the backend the master administrator gave the realm with a profile other than Standard. Any other backend id, and a Standard realm, is refused with 400; an unknown and an unassigned backend id get the same answer, so backend ids cannot be probed, and the console offers only the assigned backend. The key then waits for a certificate (Waiting for a certificate → Download CSR → Upload certificate). The certificate must be for the custody key's public key. A renewed certificate reaches the realm's store-backed wallet providers at once; a wallet provider configured in YAML (helix.federation.wallet[*]) needs a restart.

Method & path (under /admin/realms/{realm})	Body	Purpose
POST /verifier-keys/generate	{name, algorithm, backendId}	Generate an ES256 / ES384 verifier key in the realm's assigned backend (201)
POST /verifier-keys/{name}/csr	{subject, dnsNames}	The key's CSR, signed in the backend ({"csrPem"})
PUT /verifier-keys/{name}/certificate	{certificateChainPem}	Attach the CA-issued chain
GET /verifier-keys		Keys that can sign; ?pending=true adds keys still waiting for a certificate (expired ones are not listed)

High assurance has no software verifier keys. In a High-assurance realm POST /admin/realms/{realm}/verifier-keys (import) is refused with 400 on privateKeyPem, and a software key (software:<realm>/<name>) no longer resolves, so a wallet connection that names one is refused (400 on accessKeyRef) and never signs with it; so is a dev-profile private key in a connection's settings (400 on accessCertificatePrivateKey). A switch rotates the realm keys into the backend itself, but it cannot replace a verifier key, whose replacement needs a CA-issued certificate. So switching a realm to High assurance is refused (400 on profile, naming each connection) while an enabled wallet connection of the realm still signs with a software key (an accessKeyRef of the software store, or a dev-profile accessCertificatePrivateKey; for the master realm also the connections in helix.federation.wallet). Before the switch: generate a verifier key in the backend, attach its certificate, select it in each connection, then switch. Protected keeps software verifier keys, as it keeps software signing.

Credential issuer keys and the development CA

The OpenID4VCI issuer of a realm signs credentials (SD-JWT VC, mdoc) and status lists with an ES256 issuer key (Credential issuance → Issuer key). Issuer keys are ES256 only, as the HAIP profile requires.

Profile	Test key (Generate test key)	Development CA	CSR key (Create CSR)	Import (Import key)
Standard	Software key, certified by a software development CA	Software, encrypted with DB_ENCRYPTION	Software key	Allowed
Protected	Generated in the realm's HSM/KMS, certified by a development CA that also lives there	Generated in the HSM/KMS on first use	Generated in the HSM/KMS	Allowed (software)
High assurance	Refused (400, code devCaRefused)	Refused; a live one blocks the switch	Generated in the HSM/KMS	Refused (400 on privateKeyPem, code softwareKeyRefused)

In Protected and High assurance the key is created in the realm's assigned backend, never in one a request names, and the backend signs the CSR. The server checks that the public key in the CSR, and in a certificate you attach later, is the backend key's. Such a key never exists outside the backend, so there is nothing to export.

• Development CA. A realm's test CA certifies generated test keys; wallets and verifiers trust it only if they add its certificate, so it is not for production. In Protected, a development CA made while the realm was Standard stays a software key, and Generate test key is refused (409, code: devCaSoftware) until you retire it, because a software CA would certify a backend key. GET /admin/realms/{realmId}/credential-issuance/keys/ca returns {certificatePem, custody} (software or direct).
• Retiring the CA (DELETE /admin/realms/{realmId}/credential-issuance/keys/ca) answers 204, 404 when there is no live one, and 409 while an ACTIVE or PENDING generated key still chains to it. A software CA loses its private key; the certificate stays, so credentials it certified remain recognised as the realm's own (they are never trusted as identity evidence). The next test key makes a new CA.
• Certificates and the pending cap. A CSR key waits for its certificate: at most helix.custody.max-pending-keys-per-realm per realm (one more is refused with 400 on fieldErrors.subject), and one still without a certificate after helix.custody.pending-key-ttl is retired by the retention job (CUSTODY_KEY_EXPIRE).
• Retiring a key (POST .../keys/{keyId}/retire) disables a backend key at once. The retention job destroys it after helix.custody.destroy-after; the row stays with an empty custody reference. The ACTIVE key cannot be retired (409): activate another key first.
• Outage. A credential request that needs the backend while it is unreachable answers 503 temporarily_unavailable with Retry-After: 5, counts helix.issuer.credential.unavailable and issues nothing. A status list that cannot be re-signed keeps its last token and is re-signed on the next cycle. A backend refusal is credential_request_denied. A wallet that retries after a 503 needs a fresh c_nonce.
• Wrong key. An issuer key whose stored public key or certificate is not the backend key's is not used (the request is denied, logged), never signed with another key.

API, under /admin/realms/{realmId}/credential-issuance/keys (a realm administrator writes; another admin with only the credential-issuance permission can read):

• GET lists the keys (id, kid, status, source, subject, issuer, notBefore, notAfter, fingerprint, chainPem, csrPem, createdAt, activatedAt, rotatedAt, custody, backendId; custody is software or direct; never a reference or key material). The realm profile is in the X-Helix-Custody-Profile response header;
• POST /generate (test key), POST /import {privateKeyPem, certificateChainPem}, POST /csr {subject, dnsNames}, PUT /admin/realms/{realmId}/credential-issuance/keys/{keyId}/certificate {certificateChainPem}, and POST to the same path with /activate or /retire instead of /certificate;
• GET /ca, DELETE /ca as above.

Refusals are 400 with fieldErrors (and fieldErrorCodes) for configuration, 409 for state, and 503 for an unreachable backend. Audit events: ISSUER_KEY_CSR, ISSUER_KEY_GENERATE, ISSUER_CA_RETIRE (written after the key's transaction committed), plus CUSTODY_KEY_* for the retention steps (resource issuer-key; the CA is development-ca:<id>).

Switching a production issuer to High assurance, in order:

1. A master-realm administrator moves the realm to Protected (so new issuer keys are made in the HSM/KMS).
2. Create CSR, have your CA sign it, and attach the chain (Attach certificate).
3. Activate the key. It rotates the previous ACTIVE key.
4. Retire the development CA (it must certify no ACTIVE or PENDING generated key; retire those first), and retire any software issuer key that is still PENDING. Activating the new key already rotated the previous ACTIVE one.
5. A master-realm administrator moves the realm to High assurance. The switch is refused (400 on profile) while a software issuer key or a development CA is left, and it names them.

eID SP keys

The SAML SP keys of an eID connection (eid, digid, eherkenning, eidas) can be generated in the realm's HSM/KMS backend (Protected or High assurance; never the software backend). One key signs, decrypts, or does both. DigiD/eID and eHerkenning connections commonly publish one SP certificate for both, and their documents allow it; eIDAS requires different keys and certificates for signing and encryption (eIDAS Cryptographic Requirements v1.4.1 §3.5.2), so a SIGNING_DECRYPTION key is refused for an eIDAS connection (400 on usage). A decrypting key is registered for exactly one key transport {method, digest, mgf} and passes a self-test at generation (encrypt 32 random bytes with the public key and exactly those parameters, decrypt them in the backend, and check that a ciphertext made with other parameters does not decrypt). A combination the backend cannot honour is refused then (400 on keyTransport) and the backend key is disabled, so a login never discovers it. Vault and OpenBao Transit use one hash for the OAEP digest and MGF1, so a combination with different hashes is refused on them.

API (manage-identity-providers; for a BSN-authoritative connection such as DigiD or eIDAS, every write also needs a realm administrator, as editing its pasted keys does: 403 otherwise), under /admin/realms/{realmId}/identity-providers/{alias}/sp-keys:

• GET → [{id, usage, status, rsaBits, keyTransport, custody, backendId, subject, issuer, notAfter, fingerprint, chainPem, csrPem, createdAt, activatedAt}] (public facts only);
• GET /algorithms → {rsaBits, keyTransport: [{method, digest, mgf}], signatureAlgorithms, defaultKeyTransport, separateKeys}: the scheme table below intersected with the backend's capabilities, the scheme's recommended transport (rsa-oaep-mgf1p for DigiD/eID and eHerkenning, rsa-oaep SHA-256/SHA-256 for eIDAS) and whether it needs separate signing and decryption keys;
• both carry the realm custody profile in the X-Helix-Custody-Profile response header;
• POST /csr {usage: SIGNING|DECRYPTION|SIGNING_DECRYPTION, rsaBits, keyTransport, subject, dnsNames} → 201 with the key and its CSR (keyUsage digitalSignature, plus keyEncipherment for a decrypting key);
• PUT /admin/realms/{realmId}/identity-providers/{alias}/sp-keys/{keyId}/certificate {certificateChainPem}, and POST to the same path with /activate or /retire instead of /certificate.

Activating a key turns the provider's ACTIVE keys with an overlapping usage ROTATED (a rotated decrypting key keeps decrypting until it is retired) and removes the connection's pasted spSigningPrivateKey when the new key signs (audited IDP_SP_KEY_REPLACED_PEM). A pasted spDecryptionPrivateKey stays as the last decryption fallback during the rollover in Standard and Protected realms, because the IdP may keep encrypting to the old certificate until it refreshes the SP metadata; in a High-assurance realm it is removed too (never a software decryption key there). The ACTIVE key is retired only after its successor is activated (409); a retired key is disabled at once and destroyed by the retention job, and so is a PENDING key not activated within helix.custody.pending-key-ttl (with or without its certificate). More than helix.custody.max-pending-keys-per-realm keys waiting for a certificate are refused with 409 (code: tooManyPending).

A key belongs to the provider instance and eID scheme it was made for. Deleting the connection, or saving it with another scheme, retires its keys (all statuses) and disables them in the backend; a connection created again under the same alias starts without keys.

Where the console shows it: the connection's Certificates tab, section Keys in HSM/KMS (Standard realms see an explanation instead; High assurance realms do not show the pasted-key fields).

What a custody key does at run time:

• It signs the AuthnRequest (Redirect and POST binding) and the signed ArtifactResolve, with the scheme's algorithm: xmldsig-more#rsa-sha256 (RS256) for DigiD, eID and eHerkenning, xmldsig-more#sha256-rsa-MGF1 (PS256) for eIDAS. A pasted RS256 signature is byte-identical to what earlier versions produced; a pasted eIDAS key now signs PS256 as well.
• It is the TLS client key of the artifact-resolution back channel (mutual TLS), in TLS 1.3 and 1.2, signing with rsa_pss_rsae_sha256 or rsa_pkcs1_sha256 as the backend allows. The private key never leaves the backend.
• It decrypts the EncryptedKey of an assertion (see OAEP key transport on each backend), with exactly its registered transport. A pasted decryption key accepts only the key transports of its scheme's table (below) and the data encryption algorithms AES-CBC and AES-GCM; rsa-1_5 is refused for it as well.
• Fail closed. When the backend cannot be reached, the login start and the callback answer 503 temporarily_unavailable with Retry-After. The pasted key is never tried as a fallback during an outage. A pasted signing key is used only while no custody signing key is ACTIVE, and never in a High-assurance realm.

Pasted keys. Standard and Protected realms can still paste PEM keys (spSigningPrivateKey, spDecryptionPrivateKey). A pasted key must be an unencrypted PKCS#8 RSA key (BEGIN PRIVATE KEY; code notPkcs8Key; an EC key is notRsaKey), at least the scheme's minimum size (2048, eIDAS 3072; rsaKeyTooSmall), and for eIDAS not the same key for signing and decryption (sameKeyForSigningAndDecryption). A PEM for a usage that an ACTIVE custody key of the same provider already covers is refused (coveredByCustodyKey). A High-assurance realm accepts none (pastedKeyInHighAssurance), and a realm import runs the same rules. Clear a pasted key with clearSpSigningPrivateKey or clearSpDecryptionPrivateKey set to true.

Switching an eID connection to High assurance, in order:

1. A master-realm administrator moves the realm to Protected.
2. On the connection's Certificates tab, make the signing key and the decryption key (New key in HSM/KMS), have the CA sign each CSR, Upload certificate and Activate. The scheme's recommended key transport is preselected only when the backend can decrypt it; an eIDAS connection needs two keys.
3. Activating the signing key removes the pasted signing key. The pasted decryption key stays as the last fallback in Standard and Protected while the identity provider may still encrypt to the old certificate, so clear it (clearSpDecryptionPrivateKey) once the identity provider has the new SP metadata.
4. A master-realm administrator moves the realm to High assurance. The switch is refused (400 on profile) while an enabled connection still stores a pasted key, or the backend cannot serve the connection (no RSA size at the scheme's minimum, none of the scheme's signature algorithms, no key transport it decrypts).

eID scheme algorithms

What each scheme's own documents require of an SP key (read 2026-10-29; the same table, with each source as a field, is EidSchemeAlgorithms in the server). No scheme requires xmlenc#rsa-1_5 (PKCS#1 v1.5 key transport), which is never accepted. HelixIAM signs xmldsig-more#rsa-sha256 (RS256) and xmldsig-more#sha256-rsa-MGF1 (PS256) with custody keys; other signature algorithms a scheme lists are not offered.

Scheme	Key transport (method; ds:DigestMethod; MGF)	XML signature	Data encryption	Min. RSA	Separate keys	Source
DigiD / eID (digid, eid)	xmlenc#rsa-oaep-mgf1p (SHA-1, MGF1-SHA-1); xmlenc11#rsa-oaep with MGF1-SHA-1 (xmlenc11#mgf1sha1) and a SHA-1/256/384/512 digest; default rsa-oaep-mgf1p	rsa-sha256 (also rsa-sha384, rsa-sha512)	xmlenc#aes256-cbc	2048 (PKIoverheid)	no	Logius Stelsel Toegang SAML 1.0.0 §9.1, §9.3, example EncryptedID (derived from eID SAML 4.4/4.5)
eHerkenning	xmlenc#rsa-oaep-mgf1p, or xmlenc11#rsa-oaep with SHA-1 digest and MGF1-SHA-1 ("OAEP padding and a SHA-1 digest"); default rsa-oaep-mgf1p	rsa-sha256	XML Encryption 1.1 (no URI named)	2048 (PKIoverheid)	no	Afsprakenstelsel eToegang AS1.24e: Encryption, Message Signing
eIDAS	xmlenc11#rsa-oaep only; digest and MGF each SHA-256, SHA-384 or SHA-512 (SHA-1 and rsa-oaep-mgf1p refused); default SHA-256/SHA-256	sha256/384/512-rsa-MGF1 (RSASSA-PSS) or ECDSA; no rsa-sha256	xmlenc11#aes128-gcm, aes256-gcm (aes192-gcm MAY)	3072	yes (§3.5.2)	eIDAS Cryptographic Requirements v1.4.1 §3.1.1, §3.2.1, §3.2.2.1, §3.3.1, §3.5.2

The legacy DigiD SAML 3.5 interface (Logius, 6 May 2020, §6.1 and §6.3) signs RSA-SHA256 and encrypts nothing; it is a subset of the eID row. eID SAML 4.4 (§9.3) says the SHA-1 MGF "SHOULD NOT be used", while its successor Stelsel Toegang SAML 1.0.0 says it "MUST be used" and its example uses rsa-oaep-mgf1p; the table follows the current document.

SAML metadata signing

A realm's SAML IdP metadata (/realms/<realm>/saml/idp/metadata, anonymous) can be signed with the realm's SAML signing key. It is off by default, and while it is off the document is exactly the one earlier versions served.

• Setting. Console: SAML clients, block Realm-wide: sign IdP metadata at the top of the page (switch Sign metadata, field Validity (hours), button Save for the whole realm). API, with the manage-clients permission (view-clients reads): GET and PUT /admin/realms/{realmId}/saml-idp/metadata-settings with {"signed": true, "validityHours": 168}. Both fields are required; the validity is 1 to 720 hours (default shown for a realm that never set it: 168). The answer also carries lastError ({code, at} or null): the last signing failure on the server that answered, other servers may differ. Every write is audited as SAML_METADATA_SETTINGS.
• What is signed. An enveloped XML signature over the EntityDescriptor (exclusive canonicalisation, SHA-256): rsa-sha256 for an RSA SAML key, ecdsa-sha256 for an EC key. The document gets a fresh ID, validUntil = signing time + the validity, and cacheDuration = PT1H. A signed copy is cached per realm, made by one request at a time, and re-signed after half its validity, when the document changes (a rotation, new endpoints) and when the realm's keys or this setting change. The anonymous endpoint never makes a backend call per request.
• Turning it on is checked. PUT with signed: true makes one probe signature first. If the key cannot sign (a refusal, or no key or certificate) the answer is 400 on fieldErrors.signed and nothing is stored; an unreachable backend is 503.
• Fail closed. If a new signature cannot be made, the cached copy is served while it is within its validUntil and its signing certificate is still published by the realm; otherwise the endpoint answers 503 temporarily_unavailable with Retry-After (the time until the backend is tried again). Unsigned XML is never served while signing is on. After an emergency rotation or retirement that removed the signing key, no copy signed by that key is served.
• Metrics: helix_saml_metadata_stale_served_total (a cached copy served because re-signing failed) and helix_saml_metadata_sign_failures_total{reason} (refused, unavailable, error).
• Service providers must pin the signing certificate. A signature only proves anything against a certificate the service provider already trusts; the one inside the document proves nothing. Give each service provider the realm's SAML signing certificate out of band and have it verify the metadata signature and validUntil against that pin. The metadata lists the next key's certificate before it signs (state PENDING), so a provider that verified a copy can add it to its pins before a rotation takes effect. Every service provider must re-read the metadata after the setting changes.
• The encryption KeyDescriptor is gone for a realm whose SAML key is in the HSM/KMS (High assurance). A key in a backend signs only, so the metadata no longer offers it for encryption. Protected realms (software keys, unwrapped in memory) keep it.

High-assurance rules for the issuer, eID and metadata keys

Rule	Standard	Protected	High assurance
Issuer key (CSR, Generate test key)	Software	Generated in the realm's backend	Generated in the realm's backend (CSR only)
Development CA	Software	Generated in the realm's backend	Refused; a live one blocks the switch
Importing a software issuer key	Allowed	Allowed	Refused
eID SP keys in the backend	Refused (no custody)	Allowed	Allowed
Pasted eID SP PEM keys	Allowed	Allowed	Refused; an enabled connection that still stores one blocks the switch
Software decryption key	n/a	A pasted decryption key stays as the last fallback	Never

• No request names a backend. New keys are always made in the backend the master assigned to the realm (realm_custody_settings), so a tenant cannot place a key in another tenant's partition.
• Realm isolation. A stored key reference resolves only in its own realm, and an eID SP key only for its own provider alias, instance and scheme. A reference that fails the check is treated as no key, never as a fallback.
• Fail closed. A custody failure never falls back to a software key, a pasted PEM key, another purpose or another algorithm. A request that needs an unavailable backend is 503 temporarily_unavailable with Retry-After; a refusal by the backend is 400 (configuration) or 409 (state). Readiness does not include backend health.
• Switching to High assurance is refused (400 on fieldErrors.profile, naming each item) while any software issuer key, development CA or pasted eID key exists, and when the backend cannot serve the issuer or an enabled eID connection. The steps are in Credential issuer keys and eID SP keys.

DB_ENCRYPTION under a KEK (envelope)

DB_ENCRYPTION encrypts the database columns (and the software keys of Standard realms). Instead of the plain key you can give the server the key wrapped by a KEK in a backend:

Variable	Meaning
DB_ENCRYPTION_WRAPPED	The data key, wrapped (Base64), as DbKeyWrapTool prints it.
DB_ENCRYPTION_KEK	The KEK's key reference, e.g. vault:vault-1/master/helixdbkek-master-<uuid>#1 or pkcs11:hsm-1/master/helix:master:DBKEK:<uuid>.

DB_ENCRYPTION_KEK is a key of its own, never a realm's KEK. A realm's KEK is rotated through the admin API, and a rotation disables the old KEK and later destroys it; a data key wrapped by it would be lost, and with it every encrypted column. So its names differ from a realm KEK's (helixdbkek-<realm>-<uuid> on Vault and OpenBao, label helix:<realm>:DBKEK:<uuid> on an HSM), the tool creates it that way and refuses to wrap a data key with a key named like a realm KEK, and a realm KEK rotation refuses (409, code kekRotationDbEncryptionKek) a realm whose KEK is the configured DB_ENCRYPTION_KEK (any version of it) or has a DB_ENCRYPTION_KEK name, and never disables or destroys it. A data key still wrapped by a key with a realm-KEK name (an earlier guide used helix-master-kek-<uuid>, then helixkek-master-<uuid>) keeps working; the server warns at start-up: move it with --rewrap-from (below).

At start-up the server builds only the referenced backend, calls its decrypt once (associated data db-key) and keeps the data key in memory only. Start-up fails when the unwrap fails, when only one of the two is set, or when DB_ENCRYPTION, or the same setting from a properties file or a DATABASE_ENCRYPTION variable, is set as well (they are mutually exclusive). A software: KEK is refused by name: the software backend keeps no key-encryption keys. Messages name the KEK reference and the backend, never the key or the wrapped value. Chart: dbEncryption.wrapped.secret ({name, key}) and dbEncryption.wrapped.kekRef, with secrets.dbEncryptionKey: "".

The tool in the server image (io.helixiam.persistence.security.DbKeyWrapTool) takes the backend settings as -D properties and never prints a data key (exit 0 ok, 1 failure, 2 usage). Run it with credentials that may create keys (the tool's policy is in Vault and OpenBao Transit; the server's own token only reads and decrypts). Create the KEK, then wrap an existing key read from stdin (no echo on a terminal), or generate one:

TOOL="java -Dhelix.custody.backends[0].id=vault-1 -Dhelix.custody.backends[0].type=vault \
  -Dhelix.custody.backends[0].settings.address=https://vault.example:8200 \
  -Dhelix.custody.backends[0].settings.auth.kind=approle \
  -Dhelix.custody.backends[0].settings.auth.role=$ROLE_ID \
  -Dhelix.custody.backends[0].settings.auth.secretIdFile=/run/secrets/secret-id \
  -cp /app/app.jar io.helixiam.persistence.security.DbKeyWrapTool"
KEK_REF=$($TOOL --create-kek vault-1)     # prints e.g. vault:vault-1/master/helixdbkek-master-<uuid>#1
printf %s "$CURRENT_DB_ENCRYPTION" | $TOOL --kek "$KEK_REF"
# --generate instead of stdin: a new random 32-character key (only for a new installation: existing data needs the old key)

On an HSM the backend is type=pkcs11 with its settings (settings.library, settings.tokenLabel, settings.pinFile, as for the server); --create-kek hsm-1 makes a sensitive, non-extractable AES-256 token key labelled helix:master:DBKEK:<uuid> and prints pkcs11:hsm-1/master/helix:master:DBKEK:<uuid>.

Checked end to end against a Vault 2.2.1 dev server with token auth and the minimal policy: the tool wrapped a key without printing it, a server started with DB_ENCRYPTION_WRAPPED + DB_ENCRYPTION_KEK became ready (the data key appears nowhere in its log), and a server with both the plain and the wrapped key refused to start. The tool's create, wrap, rewrap and disable are also tested against a SoftHSM token.

Rotating DB_ENCRYPTION_KEK. This KEK is the operator's (it is not in the database), so it is rotated with the tool, not the admin API. The data key and the data stay as they are; only the wrapped value changes:

1. Create the new KEK: NEW_KEK_REF=$($TOOL --create-kek vault-1) (or hsm-1).
2. Move the wrapped value to it. The tool reads the current DB_ENCRYPTION_WRAPPED on stdin, unwraps it with the old KEK (also a quarantined one, or one with a realm-KEK name), wraps the data key under the new KEK, checks that the new value unwraps to the same key, and prints it:

printf %s "$DB_ENCRYPTION_WRAPPED" | $TOOL --rewrap-from "$OLD_KEK_REF" --kek "$NEW_KEK_REF"

3. Put the printed value in the DB_ENCRYPTION_WRAPPED secret and the new reference in DB_ENCRYPTION_KEK (dbEncryption.wrapped.kekRef), and roll the pods; each becomes ready once it unwrapped the data key. 4. When every pod runs with the new value, disable the old KEK: $TOOL --disable "$OLD_KEK_REF". It then neither encrypts nor decrypts (Vault and OpenBao: a new version is made when the referenced one is the latest, and both minimum versions are raised above the referenced version; HSM: CKA_ENCRYPT and CKA_DECRYPT false). The tool disables only a key with a DB_ENCRYPTION_KEK name (helixdbkek-, :DBKEK:; any other key may be a realm's), and never the one the environment's DB_ENCRYPTION_KEK names (the next pod would not start).

An old KEK with a realm-KEK name (helix-master-kek-<uuid>, helixkek-master-<uuid>, HSM helix:master:KEK:<uuid>) is disabled by hand, after checking that no realm uses it: Realm keys → Key custody of each realm in Protected or High assurance (the master realm too) must not show it as the realm's key-encryption key, and no KEK rotation may list it. Then: - Vault and OpenBao: vault write -f transit/keys/<name>/rotate if its version is the latest, then vault write transit/keys/<name>/config min_decryption_version=<its version + 1> min_encryption_version=<its version + 1>. - HSM: with the HSM's own administration tool, as the user that owns the partition, set CKA_ENCRYPT and CKA_DECRYPT to false on the secret key object labelled helix:master:KEK:<uuid> (C_SetAttributeValue, both CK_FALSE). This is what HelixIAM's own disable of a KEK does, and a KEK with CKA_DECRYPT false no longer opens a wrapped value (tested on SoftHSM). Do not destroy it yet.

Delete it after your retention period (Vault: deletion_allowed=true, then vault delete transit/keys/<name>; HSM: destroy the object).

Checked against a Vault 2.2.1 dev server and a SoftHSM token: the moved value unwraps with the new KEK through the server's start-up path, and the disabled old KEK no longer opens the old value. Rotating the data key itself (re-encrypting the data) is a separate, online procedure: see Rotating the database encryption key.

A plain DB_ENCRYPTION that looks weak (fewer than 16 distinct characters, a known demo value) is reported as an insecure setting at start-up and in helix_db_encryption_weak_key (1); it is a warning, not a refusal. A new DB_ENCRYPTION_NEXT has no such excuse: a weak or short one (under 32 characters) refuses start-up.

Rotating the database encryption key

DB_ENCRYPTION is the data key: it encrypts sensitive database columns, the software signing keys of Standard realms and the values derived from it (the wallet evidence seal, the profile provenance and the trusted-list integrity check). From rc.13 you can replace it online, with no downtime: a second key (DB_ENCRYPTION_NEXT) is added beside the current one, a master-realm administrator starts the rotation, and every replica rewrites the stored values under the new key in the background while all replicas keep serving. This is the data key; the realm key-encryption keys have their own rotation (Rotating a key-encryption key) and DB_ENCRYPTION_KEK is rotated with the tool (DB_ENCRYPTION under a KEK).

Before you start.

• Every pod must run rc.13 or later. An rc.12 pod writes no heartbeat, so the start cannot see it, and it cannot read the key-tagged values a rotation writes. Finish the upgrade first. Once a rotation has started, rolling back to a release before rc.13 is not supported (its values are key-tagged hx2.<key id>.). Until the first rotation, values keep the format of rc.12, so a mixed rollout and a rollback still work.
• Take a database backup, and keep the old key until the console says it can go (below).
• Only administrators of the master realm can see or change it (403 otherwise).

The steps.

1. Make the new key. A plain key is at least 32 characters and passes the same weak-key check as DB_ENCRYPTION (a weak or short one refuses start-up; this is a new key, so there is no legacy excuse): openssl rand -base64 32. Or a wrapped key, as for DB_ENCRYPTION_WRAPPED: wrap the new key with the tool and the same or another KEK (printf %s "$NEW_KEY" | $TOOL --kek "$KEK_REF"); the key must be 16, 24 or 32 bytes.
2. Configure it as DB_ENCRYPTION_NEXT on every replica and roll the pods. Either DB_ENCRYPTION_NEXT (plain) or DB_ENCRYPTION_NEXT_WRAPPED together with DB_ENCRYPTION_NEXT_KEK (default: DB_ENCRYPTION_KEK); both at once, or a NEXT KEK without a wrapped key, refuses start-up. Chart: dbEncryption.next.secret ({name, key}, plain) or dbEncryption.next.wrapped (secret and kekRef; an empty kekRef uses dbEncryption.wrapped.kekRef), not both. DB_ENCRYPTION stays as it is: every replica now reads with either key and still writes under the current one.
3. Start the rotation. Console: the master realm's Realm keys page, section Database encryption, which shows the current key, the new key (by id: 16 characters derived from the key, never the key) and how many replicas hold it; Start rotation is offered when every live replica has it. API: POST /admin/database-encryption/rotation (optional body {"toKeyId": "<id the administrator saw>"}; 202 with the rotation, 200 with the running one if called again). The start is refused with 409 and a code: noNextKey, replicaMissingNextKey (it gives the counts, never a host name), nextKeyChanged, rotationFailed (retry the stopped one instead). From the start, every replica writes under the new key; a leased background resealer (one replica at a time, a crash is picked up by another) rewrites the existing values in bounded batches, resumable from a cursor, and never holds a database connection while it opens and seals a value. A row that a concurrent writer changed is looked at again.
4. Wait for DRAINING, then DONE. When no row is left, the rotation waits for the values it does not rewrite to expire: short-lived flow state and wallet transactions (helix.db-encryption.rotation.drain, default PT30M: it must be at least the longest lifetime of those values, so raise it if you extended flow-state or wallet-transaction lifetimes beyond 30 minutes) and sessions that were created before the start. Then it counts every table once more (back to rewriting if a value appeared), waits until every live replica writes under the new key, and records DONE. With the default Redis session store (HELIX_SESSION_STORE=redis) a rotation therefore stays DRAINING for at least the longest absolute session lifetime of any realm (sso_session_max_lifetime_seconds, 36000 s = about 10 hours by default), and both keys are required for all of it. With JDBC sessions it waits for the last live session created before the start. With the Redis token store (HELIX_TOKEN_STORE=redis, default queue) authorizations are not rewritten: keep the old key until oldKeyNeededUntil (the start plus the longest refresh-token lifetime), which GET /admin/database-encryption and the console show.
5. Swap the keys, in this order. When the status is DONE and oldKeyNeededUntil is empty or past: (a) set DB_ENCRYPTION to the new key and remove DB_ENCRYPTION_NEXT on every replica, (b) restart them, (c) only then remove the old key from wherever you keep it. Do not remove the old key before (b): until then the replicas still hold it as DB_ENCRYPTION. A server that lacks a key the database still needs refuses to start, naming key ids, states and counts (never a key): a rotation that is not DONE needs both keys, and so does a value tagged with a key the server does not hold.

Pause, resume, retry. POST .../rotation/pause stops the rewrite (and the waiting) until POST .../rotation/resume; nothing is lost, and the old key stays needed. A rotation that stopped (FAILED) is continued from its cursor with POST .../rotation/retry; a stopped rotation blocks a new start (409 rotationFailed) until then. It stops, naming the table, column and row id and never a value, when a stored value no configured key reads (unreadableRow: make sure the old key is still configured on every replica), when it makes no progress (resealStuck) or when the database refuses a write (writeRefused). A state the call cannot change is 409 with noRotation, rotationNotRunning, rotationNotPaused or rotationNotFailed. All of these are master-realm only and audited.

Reading the state. GET /admin/database-encryption answers writeKeyId, keyIds, nextKeyPresent, nextKeyId, replicas (live, withNextKey, onWriteKey) and rotation (state RESEALING, DRAINING, DONE or FAILED, paused, targetsTotal/targetsDone, rowsDone, drainUntil, lastError and lastErrorCode, oldKeyNeededUntil; null when there has been none). While DRAINING, lastErrorCode is null until drainUntil has passed; after that it says what is still awaited: ephemeralValuesLive, sessionsLive or replicasNotOnWriteKey. Key ids and counts only.

Readiness. The dataKey health contributor is part of the readiness group. A replica is not ready when it does not hold the key the database writes under (it could not read new values), and, while a rotation is open, when its heartbeat (helix.db-encryption.heartbeat, default PT10S) has not been written for three intervals (the rotation can no longer see it, so it must not keep writing). With no rotation open, or when that read fails or takes longer than 2 s (the database is down: a shared dependency), a replica stays ready. This is a condition of the pod's own data keys, not backend health, so the rule that readiness never includes backend health does not apply to it.

Metrics: helix_db_data_key_reseal_total{result} (ok, changed, unreadable) and helix_db_data_key_rotation_active. Audit: DATA_KEY_ROTATION_START, _COMPLETE, _FAILED, _PAUSE, _RESUME and _RETRY (key ids and counts only).

Settings: helix.db-encryption.rotation.batch-size (default 100, 1 to 1000), .lease (PT60S, at least PT1S), .interval (PT5S), .drain (PT30M), .enabled (true: whether this replica runs the resealer; the API still starts rotations) and helix.db-encryption.heartbeat (PT10S).

Failure behaviour

Fail closed, always. A request that needs a backend that does not answer is never signed with a software key, another purpose's key or RS256.

• Token, SAML and admin requests that need an unavailable backend answer 503 with Retry-After and the OAuth error temporarily_unavailable (token endpoint and other protocol endpoints: HELIX_OVERLOAD_RETRY_AFTER_SECONDS, default 2 s; custody admin endpoints: 5 s). So does a key that is quarantined because it can leave the backend (A key that can leave the backend), and a token (any signed JWT) that custody refuses to sign, such as a custody realm with no ACTIVE key of the requested algorithm: a server-side condition, never a 500. A refusal on an admin endpoint (a disabled key, an algorithm the backend cannot sign) is a 400 or 409, never a 503.
• Time bound: each call is given up after the backend's opTimeout (default 2 s; helix.custody.op-timeout for a backend without one).
• Saturation shedding: each pod runs at most helix.custody.max-concurrent-calls (default 64) calls in flight per remote backend, with no queue. A call beyond that is shed at once as a 503. Shedding does not open the circuit breaker.
• Circuit breaker per remote backend and pod: 5 consecutive unavailable results within 10 s open it; while open, calls fail at once (503) without reaching the backend; after 5 s one trial call goes through, and its success closes it. Refusals and quarantined keys never count. State: helix_custody_breaker_open{backend}. The built-in software backend has no breaker (its gauge stays 0): it serves every realm, so a Protected realm whose KEK backend is down fails on its own and never sheds the Standard realms.
• Issuer, eID and metadata keys during an outage. A credential request answers 503 and issues nothing (status lists keep their last token and are re-signed later); an eID login start or callback answers 503, with no fallback to a pasted key; signed SAML metadata answers 503 once no still-valid copy can be served. Details are in each key's section above.
• What keeps working during an outage: discovery, JWKS and unsigned SAML metadata (certificates only), every Standard realm, and every Protected realm on a pod that already holds its unwrapped keys (they stay in memory with no time-based eviction; they leave only on rotation or retirement). High-assurance realms answer 503 for anything they must sign. No pod restarts.
• Readiness does not include backend health (one backend serves every pod; including it would turn a backend blip into a total outage). One exception, per pod: a pod that starts and cannot unwrap the wrapped keys of its realms because the backend does not answer stays not ready, retries every helix.custody.unwrap-retry (default PT10S) and becomes ready when the unwrap succeeds; liveness stays up, and discovery and JWKS already answer. A definite refusal for one key (no KEK on this pod, a ciphertext that does not verify) does not hold readiness: that key fails closed, is counted in helix_custody_unusable_keys{realm} and audited once as CUSTODY_KEY_UNAVAILABLE. An ACTIVE key whose custody reference cannot be used on this pod (its backend is not configured here) is published but cannot sign, and is audited the same way.
• Aggregate health: the keyCustody component of /actuator/health (management port) lists each backend (UP or DOWN, detail, latency, breaker state). Any backend DOWN makes the aggregate /actuator/health DOWN (503), while /actuator/health/readiness and /actuator/health/liveness stay UP. Probes are bounded to 1 s, one in flight per backend, and cached for 5 s, so the aggregate shows an outage up to about 6 s late. A backend whose realm binding is broken is DOWN too, with realmBinding naming each realm and why (notAllowed, notDedicated); see Dedicated key stores.
• A key of a realm its backend does not serve (helix.custody.backends[n].realms) is never used: 503 on protocol endpoints, 409 backendRealmNotAllowed on custody admin endpoints, 400 where a new key would be made.

Metrics, audit, console

Metric	Meaning
helix_custody_ops_total{backend,op,result}	Calls per backend; op = sign, encrypt, decrypt, decrypt_key (an RSA-OAEP key-transport decryption of an eID assertion), generate, disable, destroy; result = ok, unavailable, refused, or contract (a plugin threw something other than the three exceptions of the SPI: counted as unavailable and against the circuit breaker).
helix_custody_op_seconds{backend,op}	Latency of the calls that reached the backend (histogram, 5 ms to 5 s buckets).
helix_custody_breaker_open{backend}	1 while the breaker sheds calls.
helix_custody_unusable_keys{realm}	Wrapped signing keys this pod cannot unwrap (they fail closed).
helix_db_encryption_weak_key	1 when the plain DB_ENCRYPTION looks weak.
helix_custody_kek_rewrap_total{result}	Keys wrapped again by a KEK rotation (ok), paused by an unreachable backend (unavailable) or refused by their KEK (refused).
helix_custody_kek_rotations_active	KEK rotations running (wrapping keys again or disabling the old KEK), as this pod last saw them.
helix_db_data_key_reseal_total{result}	Values rewritten under the new data key by a data-key rotation (ok), changed by another writer while they were rewritten and looked at again (changed), or read by no configured key (unreadable: the rotation stops).
helix_db_data_key_rotation_active	1 while a data-key rotation is rewriting or draining, as this pod last saw it.
helix_issuer_credential_unavailable_total	Credential requests answered 503 because the issuer key's backend was unreachable (Micrometer name helix.issuer.credential.unavailable).
helix_saml_metadata_stale_served_total	Signed SAML metadata served from the cached copy because re-signing failed.
helix_saml_metadata_sign_failures_total{reason}	Failed metadata signatures: refused, unavailable or error.

Alert on helix_custody_breaker_open == 1, on a rising result="unavailable" rate, on helix_custody_unusable_keys > 0, and on the keyCustody health component. See also Observability & health.

Audit event types (details carry realm, purpose, algorithm, custody kind, backend and key reference; never key material, PINs or tokens): CUSTODY_PROFILE_CHANGE, CUSTODY_KEY_GENERATE, CUSTODY_KEY_DISABLE (reason retired or orphan, or by the retention job), CUSTODY_KEY_DESTROY, CUSTODY_KEY_EXPIRE, CUSTODY_KEY_UNAVAILABLE, VERIFIER_KEY_GENERATE, VERIFIER_KEY_CSR, VERIFIER_KEY_CERTIFICATE. This release adds ISSUER_KEY_CSR, ISSUER_KEY_GENERATE, ISSUER_CA_RETIRE, IDP_SP_KEY_CSR, IDP_SP_KEY_CERTIFICATE, IDP_SP_KEY_ACTIVATE, IDP_SP_KEY_RETIRE, IDP_SP_KEY_REPLACED_PEM (a pasted eID key removed by an activation) and SAML_METADATA_SETTINGS. A KEK rotation writes CUSTODY_KEK_ROTATION_START (also a refused start, and a resume), CUSTODY_KEK_ROTATION_COMPLETE, CUSTODY_KEK_ROTATION_FAILED and CUSTODY_KEK_ROTATION_ABANDON (also a refused one), and CUSTODY_KEY_GENERATE, CUSTODY_KEY_DISABLE and CUSTODY_KEY_DESTROY with the resource realm-kek; the retention steps of the new keys use CUSTODY_KEY_* with the resources issuer-key and federation-sp-key. Phase 3 adds CUSTODY_KEY_STORE_POLICY (from, to) and, for the data key, DATA_KEY_ROTATION_START, DATA_KEY_ROTATION_COMPLETE, DATA_KEY_ROTATION_FAILED, DATA_KEY_ROTATION_PAUSE, DATA_KEY_ROTATION_RESUME and DATA_KEY_ROTATION_RETRY (key ids and counts, never a key or a value). Signatures are counted in the metrics, not audited one by one.

Console (EN and NL): Realm keys → Key custody shows the profile, the backend, its health and latency, and the keys per purpose with their custody (Software, Wrapped by HSM/KMS, Direct (HSM/KMS)); key references are shown as technical details only. For a Protected realm it also shows the key-encryption key's rotation and Rotate key-encryption key. The CSR form has a purpose choice. The wallet provider form generates verifier keys in a backend. Phase 3 adds the key-store picker (grouped as sovereign key stores, cloud providers and partner plugins, with the reason a store is not offered) and the sovereign key stores only setting on that page, and, on the master realm's page only, the installed plugins and the Database encryption section (the rotation above). Only master-realm administrators see or change these. This release adds Credential issuance → Issuer key (custody column, Create CSR, Retire development CA), the Keys in HSM/KMS section of an eID connection's Certificates tab, and Realm-wide: sign IdP metadata on the SAML clients page. They all say "HSM/KMS", never a product name, and show key references nowhere.

Sizing

Measured with HelixIAM's fixed token benchmark on one machine, one mode after the other:

• Machine: Apple M5 Max, Docker Desktop VM (aarch64, 18 CPUs), server image built from the key-custody branch.
• Server: one container, --cpus=2 pinned to two CPUs, 1536 MiB; PostgreSQL, Redis and the load generator on their own CPUs; the Vault dev container (2.2.1, in-memory storage, no TLS, no audit device) on two further CPUs with no other limit. SoftHSM 2.6.1 runs inside the server container (a benchmark-only image layer), so its CPU counts as server CPU.
• Load: client_credentials on one confidential client, one run per mode: fixed rates 500, 1,000 and 1,500 per second for 20 s, then a ramp to the highest rate with p95 under 50 ms and errors (503s included) under 1 %. CPU per token is server-container CPU time per successful token at 1,000 per second.
• Single runs on a shared developer machine: treat differences under about 20 % as noise.

Measured 2026-10-28:

Profile	Backend	Realm key	Server CPU per token at 1,000/s	Vault CPU per token	p95 at 1,000/s	Highest sustainable rate
Standard	software	ES256	0.434 ms (first run 0.456 ms)	n/a	0.8 ms	8,519/s (first run 2,070/s: one ramp step at 2,160/s had 1.03 % 503s; the rerun is the figure)
Standard	software	EdDSA	0.370 ms	n/a	0.9 ms	8,906/s
Standard	software	RS256	0.787 ms	n/a	1.6 ms	2,851/s
Protected	Vault	ES256	0.461 ms	0.002 ms	1.3 ms	8,132/s
High assurance	Vault	ES256	0.534 ms	0.360 ms	1.9 ms	at least 3,222/s (the ramp's step limit; not saturated)
High assurance	Vault	RS256	0.475 ms	0.952 ms	2.8 ms	2,050/s (Vault's RSA signing is the limit)
High assurance	SoftHSM (PKCS#11)	ES256	0.433 ms	n/a (SoftHSM is in the server)	0.8 ms	at least 3,222/s (the ramp's step limit; not saturated)
High assurance	SoftHSM (PKCS#11)	RS256	0.816 ms	n/a (SoftHSM is in the server)	1.6 ms	2,573/s

For comparison, a server image from just before key custody was added, on the same machine right after: 0.515 ms and 5,153/s (ES256, Standard, one run). Key custody does not make Standard realms slower. The High-assurance ramps started at 300/s and stopped at their last step (3,222/s), so for ES256 the backend's real ceiling is higher than measured.

What this means:

• Protected costs nothing at signing time: the keys sign in memory, as in Standard. The backend is used only to create the KEK, to wrap a new key and to unwrap keys when a pod starts.
• High assurance moves every signature to the backend. Its ceiling is the backend's signing rate divided by the signatures per request (one per access token; an authorization-code flow with an ID token signs twice), plus one network round trip per signature (about 0.3–2 ms inside a cluster). The Vault numbers above are a single dev-mode node with no TLS and no audit device on the same machine: an upper bound, not a production figure. SoftHSM is a software token in the server process, not an HSM.
• Size a High-assurance realm against the HSM you will actually use, with a load test of your own, and keep headroom: when the backend is the limit, the pods shed the excess as 503 instead of queueing.

KMIP and the cloud key stores

Nothing above was measured on KMIP or on a cloud key service, and none of it should be read as a sizing figure for them.

• KMIP. The only KMIP server HelixIAM has run against is PyKMIP 0.10.0 (patched for KMIP 2.x gaps), a functional test server for protocol tests. The run (helix-iam-server/target/pykmip-capabilities.txt, written by KmipCustodyOnPyKmipTest) records what that server does with the backend configured algorithms=RS256, kek=true: signing RS256, non-extractable keys, AES-256-GCM with the caller's associated data (a wrong one is refused), RSA 2048, 3072 and 4096, and no key transport; it records no latency or rate. A High-assurance realm on KMIP signs through the key manager once per signature, so its ceiling is your key manager's signing rate plus a network round trip, over at most connections (default 8) persistent TLS connections per pod and helix.custody.max-concurrent-calls calls in flight. Size it by measuring your key manager with the benchmark, as for an HSM.
• AWS KMS, Azure Key Vault / Managed HSM, Google Cloud KMS. The capacity is the provider's documented request quota, shared by everything else in the same account, subscription or project; HelixIAM measured none of it. The quotas and how to raise them: AWS KMS, Azure Key Vault and Managed HSM, Google Cloud KMS. Over its quota a provider throttles; HelixIAM then answers 503 with Retry-After like any unavailable backend. Latency to a regional cloud service can be much higher than to an in-cluster HSM: for a High-assurance realm measure it from your cluster, and prefer Protected for a realm whose signing rate is high (the cloud service is then used only for the key-encryption key).
• Data-key rotation has not been benchmarked either: it rewrites values in batches of helix.db-encryption.rotation.batch-size (default 100) by one replica at a time, and each batch is one short transaction. Run the first one off-peak.

Credential issuance, eID logins and metadata

The numbers above are for token signing. The issuer, eID and metadata keys add these backend calls:

• Credential issuance: one ES256 signature per issued credential (SD-JWT VC or mdoc), plus one per status-list re-sign (JWT and CWT) in the background.
• eID login: one signature for the AuthnRequest, one for the ArtifactResolve, one for each new mutual-TLS handshake of the back channel, and one OAEP decryption per EncryptedKey of an encrypted assertion. These calls count against the same in-flight limit and circuit breaker as token signing (op = decrypt_key for the decryption).
• Signed SAML metadata: one signature per realm each time the cached copy is re-signed (after half its validity, or on a change), not one per request.

Measured credential issuance (OpenID4VCI SD-JWT VC; each credential is one token, one nonce and one credential request). Machine: Apple M5 Max, Docker Desktop (aarch64, 18 CPUs). The server container had a 1536 MiB memory limit (the chart's minimum; the chart default is 2 GiB) and was pinned to 2 CPUs; the issuer key algorithm was ES256. PostgreSQL, Redis, the load generator (k6 1.3.0) and the key backend ran on separate CPUs. CPU per credential and latency are measured at 100 credentials/s and are the median of 3 runs. "Max/s" is the highest rate that kept the p95 of the credential request under 100 ms with less than 1 % errors; the cell shows the median of 3 runs, then the three runs.

Custody mode	Key backend	Server CPU per credential	Backend CPU per credential	p50 / p99 at 100/s	Max/s (median; runs)
software	none (key in the database)	3.1 ms	-	3.1 / 5.2 ms	630 (690, 398, 630)
direct, Vault Transit	Vault Transit (test server)	3.8 ms	0.5 ms	3.8 / 10.1 ms	416 (416, 416, 600)
direct, SoftHSM (PKCS#11, in-process)	SoftHSM 2.6.1-3	3.4 ms	in the server	3.3 / 8.2 ms	331 (331, 331, 575)

How to read the table:

• The maximum rate varies a lot between runs (up to 1.7 times in one mode), because the machine was shared with other work during the measurements (load average 4 to 11 on 18 CPUs, with the benchmark pinned to its own CPUs). Plan on the lower runs, not the best one: about 400 credentials/s per pod with 2 CPUs for software and Vault Transit, and about 330 for SoftHSM. Three runs per mode are too few for a tight confidence interval.
• What limits each mode at its ceiling: the server's 2 CPUs. At the first rate above the ceiling the server used 1.5 to 1.9 of its 2 CPUs (SoftHSM: 1.1 to 1.8). The excess is then shed as 503 with Retry-After by the in-flight request limit (helix.overload.max-in-flight, default 40); 503s were 2 to 32 % of requests at the first step above the ceiling (up to 39 % at higher rates). The key backend itself is not the limit: Vault used about 0.5 ms of CPU per credential, and SoftHSM runs inside the server's CPU budget. A key backend with a high signing latency would hold requests in flight longer and reach that limit at a lower rate.
• Database pool: no pool timeouts ("Connection is not available") in any run whose server log was kept (the three third runs), and the 503s are not pool-related. Earlier versions stalled the 10-connection database pool under issuance load; that is fixed.
• Memory: peak memory held by the container reached the 1536 MiB limit in every run, as the kernel fills the rest with reclaimable file cache (1498 to 1536 MiB). The non-reclaimable (anonymous) memory peaked at 1407 to 1412 MiB (software, 2 runs measured), 1437 to 1456 MiB (Vault, 2 runs) and 1414 to 1421 MiB (SoftHSM, 3 runs), that is 92 to 95 % of the limit, while the load was above the ceiling. No container was OOM-killed. The headroom below the limit is small under sustained overload; the chart default is 2 GiB for this reason (minimum 1.5 GiB); keep it and do not go below the minimum if you run issuance close to its ceiling.
• The Vault figures come from a single Vault test server with no TLS and no audit device on the same machine, and the SoftHSM figures from a software token: neither is a production HSM.

Vendor figures (vendor-stated maximums, not measured with HelixIAM; performance tiers are licence-locked on Luna and nShield):

HSM	RSA-2048 signatures/s	P-256 signatures/s	Source
Thales Luna Network HSM 7 A700 / A750 / A790	1,000 / 5,000 / 10,000	"over 20,000 ECC" (top model)	Thales product brief
Entrust nShield 5c Base / Mid / High	670 / 3,949 / 13,614	not published	Entrust nShield 5c
Yubico YubiHSM 2	about 7 (139 ms each)	about 14 (73 ms each)	YubiHSM 2 specifications

A YubiHSM 2 is a serial USB device: fit for development and very small High-assurance realms only. Prefer ES256 (or EdDSA where the HSM supports it) for High-assurance realms: HSMs sign P-256 several times faster than RSA-2048.

Compliance

What HelixIAM claims:

• In High assurance, the realm's private keys are generated inside the configured backend with non-extractable attributes, checked after generation and before use, and every signature is made by the backend. With a FIPS 140-3 or Common Criteria certified HSM in its certified mode, the keys are protected by that module: the HSM is the cryptographic boundary and HelixIAM orchestrates it.
• In Protected, the private keys exist in plaintext only in pod memory; at rest they are encrypted by a KEK that stays in the backend.
• In High assurance the same holds for the credential issuer keys and every eID SP key: they are generated in the backend, non-extractable, and sign (and, for eID decryption keys, unwrap the assertion's content key) there. The content key itself is in server memory while an assertion is decrypted.
• These behaviours are tested against SoftHSM 2 (PKCS#11), Vault 2.2.1 and OpenBao 2.7.2. SHA-2 OAEP decryption through PKCS#11 has not been tested with a real token (see OAEP key transport on each backend).

What HelixIAM does not claim

• HelixIAM itself holds no FIPS 140-3 validation and no Common Criteria certification. Correct wording for a deployment: "HelixIAM uses a FIPS 140-3 Level 3 validated module in its approved mode (certificate #…)", never "HelixIAM is FIPS validated".
• No HSM model has been verified with HelixIAM yet (no Luna, nShield, Utimaco or YubiHSM in the test matrix): the certified models below are what you can buy, not what HelixIAM has been tested on. Run the checks (generate, sign per algorithm, GCM KEK, re-login, fail-over) on your HSM in its certified mode before relying on it.
• Vault Community and OpenBao are not FIPS 140 validated. Vault Transit gives sovereignty (keys in a service you run) but not FIPS. A FIPS or CC claim on that path needs Vault Enterprise "FIPS 140-3 Inside" builds, or Vault / OpenBao with HSM-backed keys; neither is tested with HelixIAM.
• SoftHSM is a software token for tests: no assurance claim.
• KMIP is tested against PyKMIP 0.10.0, a functional test server, patched for KMIP 2.x gaps (not a validated module); no commercial KMIP server has been verified, and ECDSA and RSA-OAEP over KMIP have not been run against a live server. What a KMIP deployment achieves depends on the key manager behind it.
• The cloud backends (AWS KMS, Azure Key Vault / Managed HSM, Google Cloud KMS) are Preview and have not been run against a real account. The FIPS 140 level and any other certification of those services are the provider's, for the service and the key protection level you configure (an HSM-backed key, a Managed HSM, a software key): HelixIAM claims none for them, and it does not check which level a key has beyond the protection level you set (Azure protection, GCP protectionLevel). Ask the provider for the certificate and its scope.

FIPS 140-3 (ISO/IEC 19790): Level 3 adds tamper detection and response, identity-based operator authentication and separated interfaces for critical security parameters. FIPS 140-2 certificates are moving to the historical list; ask for 140-3 in new procurements.

Common Criteria EAL4+ (AVA_VAN.5, ALC_FLR.2) against EN 419 221-5 (protection profile for cryptographic modules of trust service providers) is the eIDAS reference for the key modules of qualified trust service providers. An IAM is usually not a QTSP; DigiD, eHerkenning and wallet relying-party schemes set their own requirements, which this page does not cover.

Vendor-stated certifications (confirm the certificate numbers on the NIST CMVP and CC portals before quoting them):

Model	FIPS	CC / EN 419 221-5
Thales Luna Network HSM 7	140-3 Level 3	EAL4+ AVA_VAN.5, ALC_FLR.2, EN 419 221-5
Entrust nShield 5c / 5s	140-3 Level 3 (#4745 listed for 5s)	EAL4+ AVA_VAN.5, ALC_FLR.2, EN 419 221-5
Yubico YubiHSM 2 FIPS	140-2 Level 3 (140-3 announced by the vendor)	none found
Vault Enterprise	"FIPS 140-3 Inside" builds	n/a

Algorithms inside the boundary: RS256 (RSASSA-PKCS1-v1_5, SHA-256), PS256 (RSASSA-PSS, SHA-256), ES256 / ES384 (ECDSA P-256 / P-384), EdDSA (Ed25519; whether it is approved in an HSM's FIPS mode depends on its firmware), and AES-256-GCM for the KEK. For PKCS#11, HelixIAM uses the token's hashing mechanism when listed, otherwise it hashes in the JVM and signs the digest in the token (the private-key operation always stays in the HSM).

Limits of this release

• KEK rotation is for Protected realms. A High-assurance realm's KEK (kept for keys from an earlier Protected period) is not rotated: retire those keys once their tokens have expired. DB_ENCRYPTION_KEK is rotated with DbKeyWrapTool --rewrap-from, not the API (see DB_ENCRYPTION under a KEK).
• Keys written by a server of an earlier release record no KEK. The upgrade labels them with the realm's KEK only in a realm that never had a KEK rotation (the label is certain there); afterwards they stay unlabelled, the rotator and the key store try each KEK of the realm, no KEK is disabled as unused while one exists, and a rotation that stopped cannot be abandoned until a resume has moved them.
• Finish a rollout before you rotate a KEK, and do not roll back during one. A pod of an earlier release unwraps every key with the realm's current KEK only, so while a rotation runs it cannot unwrap the keys not yet wrapped again (they fail closed on that pod).
• RETIRED wrapped keys are not wrapped again: once the old KEK is destroyed they can never be read (they never sign again).
• A stopped rotation can be abandoned only while no key is on its new KEK; after that it can only continue.
• Quarantine is per pod. Each pod quarantines a key when it reads the key's flags: at first use, or when its health check reads them again (up to 16 keys in use per check, round robin, at most once per 5 s probe window and only while something polls /actuator/health or the console). Until then a pod that already holds the key's public key can still sign with it. Monitor /actuator/health on every pod; the recovery above clears every pod.
• No backend-to-backend moves: a realm in custody stays on its backend, and so does a realm lowered to Standard that keeps its key-encryption key there. Only a realm lowered to Standard on a backend that no longer serves it is forgotten there, once the retention job has released its KEK (at least twice helix.custody.destroy-after) (see Binding a backend to realms); it can then be raised on another backend. The issuer, eID and metadata keys live in the realm's backend too.
• A dedicated key store is configuration, read at start-up. Changing realms needs a restart (the chart rolls the pods; assign realms or rotate KEKs only once every pod runs the new realms). A realm left out of its entry's realms is refused until it is put back or lowered to Standard. A Vault 403 for a key name the entry's policy does not allow is answered like any credential failure of that entry: 503 and it counts toward that entry's circuit breaker (with realms set, HelixIAM never sends such a call).
• SAML RSA-SHA1 only in Standard realms; RSA-SHA512 only with software keys (Standard and Protected): a move out of Standard (RSA-SHA1) or to High assurance (RSA-SHA512) is refused while an enabled service provider asks for it, and saving a service provider with that algorithm in such a realm is refused (400 with fieldErrors.signatureAlgorithm). RSA-SHA1 is deprecated: the server warns about each service provider that asks for it at start-up.
• RSA key sizes. Realm keys (access, ID, SAML) are RSA 2048 in every backend (Vault rsa-2048, PKCS#11 2048). eID SP keys can be 2048, 3072 (default) or 4096, limited to the sizes the backend lists and to the scheme's minimum (eIDAS 3072). A realm key above 2048 bits is not offered.
• Issuer and development CA keys are ES256 only (HAIP). A backend that cannot sign ES256 cannot serve an enabled issuer in Protected or High assurance, and the move is refused.
• eID SP keys are RSA only, and a decrypting key serves exactly one key transport. SHA-2 OAEP decryption is proven with Vault and OpenBao (SHA-256 on OpenBao); on PKCS#11 it is proven only at the mechanism level, not with a real token (SoftHSM decrypts SHA-1 only). Vault and OpenBao refuse mixed digest and MGF hashes. See OAEP key transport on each backend. rsa-1_5 key transport is never accepted.
• A decrypted content key is in server memory for the duration of an assertion's decryption (see above).
• Signed SAML metadata is only as strong as the pin: service providers must pin the realm's signing certificate. Metadata signing is off by default, and a realm whose SAML key is in the HSM/KMS no longer advertises an encryption key.
• EdDSA in PKCS#11 only where the token lists CKM_EDDSA; generated wallet verifier keys are ES256 / ES384 only.
• Vault AppRole secret ids may be single-use: the file is read at each login, so put a fresh secret id there before the token's max TTL runs out (or use a periodic token, or Kubernetes auth).
• Kubernetes auth is not tested against a real cluster; AppRole and token are.
• Disabling a Vault or OpenBao eID decryption key raises min_decryption_version, so the backend then no longer returns that version's public key; HelixIAM keeps the key's public key and certificate in its own row.
• Not tested: a PKCS#11 token that decrypts SHA-2 OAEP, a network HSM for the issuer, eID and metadata keys, and Vault policies other than the one above.
• Cloud key stores are Preview. AWS KMS, Azure Key Vault / Managed HSM and Google Cloud KMS are tested against emulators and recordings built from the providers' published API shapes, not against a real account: AWS against LocalStack; Azure and GCP against published API examples only (until a test against real accounts; no date committed). The wallet's per-transaction ECDH key and the wallet pepper stay software secrets, protected by the database key.
• No EdDSA on any cloud key store (the aws-kms plugin does not offer it in this release, Azure Key Vault has no Ed25519, Cloud KMS signs Ed25519 over raw messages only), and none over KMIP. A realm that needs EdDSA tokens uses PKCS#11 (where the token lists CKM_EDDSA), Vault/OpenBao, or the Standard and Protected profiles.
• AWS KMS has no sign-and-decrypt keys (an RSA key has one usage), so an eID SP key that must sign and decrypt cannot be one key there; and no cloud backend offers RSA-OAEP key transport in this release, so eID decryption keys need PKCS#11, Vault/OpenBao or KMIP.
• Azure: the key-encryption key needs a Managed HSM. A plain Key Vault has no AES-GCM, so a realm cannot be Protected or High on it, and an azure-kv backend on a plain Key Vault cannot host a realm in this release.
• AWS FIPS and GovCloud endpoints: the plugin pins its endpoints, so the SDK's FIPS switch has no effect; set endpoint and stsEndpoint to the FIPS endpoints yourself. Cloud plugins do not use an egress proxy (every HTTP client is pinned to no proxy). Google Cloud EKM is not supported.
• ECDSA over KMIP is not CI-tested against a live server (PyKMIP makes RSA keys only); it is built and unit-tested against a protocol fake, and every key is proven by the generate-time self-test. No commercial KMIP server has been tested; verification against an independent KMIP implementation is planned (no date committed). RSA-OAEP key transport over KMIP is likewise not run live. A revoked KMIP key cannot be enabled again.
• Plugins are trusted code. A plugin runs in the server process with the server's permissions; its class loader isolates dependencies and is not a sandbox. Trust rests on the SHA-256 pin and the signature of the plugin image.
• After a data-key rotation has started, rolling back to a release before rc.13 is not supported (its values are key-tagged). Start a rotation only when every pod runs rc.13. A rotation with Redis sessions stays DRAINING for at least the longest absolute session lifetime (about 10 hours by default), with both keys configured; with the Redis token store the old key is needed until oldKeyNeededUntil. The data key is not rotated automatically, and the rotation does not re-encrypt backups or database dumps: they stay readable only with the key they were made under.
• A realm restricted to sovereign key stores cannot be Standard, cannot import a software key, and cannot be set while any live key held on a backend (a key still in its grace period included) is on a non-sovereign backend. Software keys still in their grace period (verify-only) do not block it.
• Not yet measured: a network HSM, Vault with TLS, an audit device and Raft storage, and a multi-pod installation sharing one backend.
• A native PKCS#11 call that hangs cannot be interrupted: the caller gets a 503 at opTimeout, but that session and thread stay busy until the HSM returns.

Settings reference

Setting	Default	Meaning
helix.custody.backends[n].id / .type / .settings.*	none	The backends (above).
helix.custody.backends[n].realms	none (shared)	The realm ids the backend serves; any other realm is refused. See Dedicated key stores.
helix.custody.require-dedicated-backend-for-high	false	A High-assurance realm needs a backend whose realms is exactly that realm.
helix.custody.binding-check-interval	PT1M	How often each pod checks the realms assigned to bound backends, the prefix overlaps and the released realms (the keyCustody health); minimum 5 seconds.
helix.custody.op-timeout	PT2S	Time bound of a backend without its own opTimeout.
helix.custody.max-concurrent-calls	64	Calls in flight per remote backend per pod (1–4096); beyond that, shed.
helix.custody.unwrap-retry	PT10S	Retry interval of a starting pod that cannot unwrap its keys yet.
helix.custody.retention-check-interval	PT1H	How often retired backend keys are disabled and destroyed.
helix.custody.destroy-after	P30D	Time from disable to destroy.
helix.custody.max-pending-keys-per-realm	5	CSR keys (and, separately, generated verifier keys) a realm may have waiting for a certificate. It also bounds issuer CSR keys and eID SP keys waiting for a certificate.
helix.custody.pending-key-ttl	P30D	A CSR key, generated verifier key, issuer CSR key or eID SP key without a certificate (an eID SP key: not activated) after this is expired, then disabled and destroyed.
helix.custody.backends[n].settings.keyTransport	every transport the token lists (KMIP: none)	PKCS#11 and KMIP: the RSA-OAEP key transports an eID decryption key may use, as METHOD;DIGEST;MGF values, comma-separated (chart: a list).
DB_ENCRYPTION_WRAPPED, DB_ENCRYPTION_KEK	unset	The envelope (above).
helix.custody.kek-rotation.batch-size	16	Keys a KEK rotation wraps again per batch (1–256).
helix.custody.kek-rotation.interval	PT5S	How often each pod's rotation worker looks for work.
helix.custody.kek-rotation.lease	PT60S	How long one pod holds a rotation without renewing it; after that another pod continues it.
helix.custody.kek-rotation.enabled	true	Whether this pod runs the rotation worker (the API still starts rotations).
helix.custody.plugin-dir	/opt/helix/plugins	Where the plugin jars are (read-only to the server user). See custody-plugins.md.
helix.custody.plugins[n].file / .sha256 / .type	none	The pins: one per jar in the plugin directory (chart: custody.plugins.jars, with custody.plugins.image).
helix.custody.sovereign-types	none	Plugin types you assert are sovereign key stores (chart: custody.sovereignTypes). pkcs11, vault and kmip always are; a cloud provider type or software here refuses start-up.
helix.custody.backends[n].settings.* (kmip, aws-kms, azure-kv, gcp-kms, other plugins)	none	The backend's own settings; see KMIP, Cloud provider key stores and values.yaml.
DB_ENCRYPTION_NEXT	unset	The next data key of a rotation, plain: at least 32 characters, strength-checked (chart: dbEncryption.next.secret).
DB_ENCRYPTION_NEXT_WRAPPED, DB_ENCRYPTION_NEXT_KEK	unset	The next data key, wrapped (16, 24 or 32 bytes unwrapped); the KEK defaults to DB_ENCRYPTION_KEK (chart: dbEncryption.next.wrapped). Not together with DB_ENCRYPTION_NEXT.
helix.db-encryption.rotation.batch-size	100	Values a data-key rotation rewrites per batch (1–1000).
helix.db-encryption.rotation.lease	PT60S	How long one replica holds the rotation without renewing it (at least PT1S).
helix.db-encryption.rotation.interval	PT5S	How often each replica's resealer looks for work.
helix.db-encryption.rotation.drain	PT30M	The longest lifetime of the values a rotation does not rewrite (flow state, wallet transactions); waited for before it finishes. Sessions and the Redis token store add their own waits.
helix.db-encryption.rotation.enabled	true	Whether this replica runs the resealer (the API still starts rotations).
helix.db-encryption.heartbeat	PT10S	How often a replica records which data keys it holds and writes under; a rotation sees a replica for three intervals.

Per-realm settings for these keys are API and console settings, not server settings: the realm's custody profile (above) and the SAML metadata signing setting (PUT /admin/realms/{realmId}/saml-idp/metadata-settings).

See also

• Realm keys
• Security hardening
• Install: key custody values
• API reference: key custody