Skip to content

Upgrading from 1.1.0-rc.12 to 1.1.0-rc.13

Key custody phase 3 (KMIP, plugins, cloud key stores, sovereign key-store policy) and online data-key rotation.

Nothing changes for a realm until you configure something: no backend, no plugin and no policy is enabled by default, and a realm stays in the profile it has. What every installation meets: the three migrations below, the new dataKey readiness contributor, the SPI install for local builds, and the value format of a data-key rotation once you start one. The operator guides are Key custody and Key custody plugins.

Before you upgrade (summary)

  • Migration V189 adds realm_custody_settings.key_store_policy (varchar(16) NOT NULL DEFAULT 'ANY') and a check (ANY or SOVEREIGN, added NOT VALID). Constant default: catalog-only, no table rewrite. Every realm is ANY.
  • Migration V190 adds the table data_key_state (at most one row: the id of the data key new values are written under). It stays empty until the first rotation. Catalog-only.
  • Migration V191 adds the tables data_key_replica (each replica's heartbeat: key ids only) and data_key_rotation, a change notification on data_key_state, and widens five columns to text because a key-tagged value is 21 characters longer: service_provider_oauth.client_secret, user_credentials.mfa_secret, user_bsn.bsn, webhook_subscription.secret and scim_target.token. varchar to text is binary-coercible in PostgreSQL (no rewrite, no index rebuild), but each ALTER takes a short exclusive lock on its table: run the upgrade outside a burst of sign-ins. None of these changes touches a value.
  • Rolling upgrade. rc.12 and rc.13 pods can run side by side: until the first data-key rotation starts, values keep the format of rc.12, so an rc.12 pod reads everything an rc.13 pod writes. Nothing needs a pause.
  • A new dataKey readiness contributor. A pod is unready when it does not hold the data key the database writes under, and, only while a rotation is open, when its heartbeat has stopped. With no rotation (every upgrade), it is always ready.
  • Local builds need the SPI installed first. The server compiles against io.helixiam:helix-custody-spi (helix-iam-server/custody-spi): run mvn -f helix-iam-server/custody-spi/pom.xml install once, and after every SPI change, before mvn package or mvn test in helix-iam-server. CI, the release build and the container image build do this for you.
  • If you built against the internal custody factory (a KeyCustodyFactory of your own compiled against rc.12's io.helixiam.authorization.service.key.custody package, which was never a supported interface): the types moved to the public package io.helixiam.custody.spi, create now takes a BackendDefinition and a KeyCustodyContext, and the services file is renamed to META-INF/services/io.helixiam.custody.spi.KeyCustodyFactory (a factory listed in the old file is not found). Rebuild it as a plugin: see Key custody plugins. The built-in pkcs11 and vault backends are unchanged for operators.
  • The plugin image. The release publishes a second, signed image holding the plugin jars (aws-kms, azure-kv, gcp-kms). The packaged chart already carries its digest and the SHA-256 pin of each jar in custody.plugins; nothing loads until a backend of such a type is configured, and the chart refuses a backend whose type no pinned plugin provides. If you use your own plugin image, pin it by digest and list each jar with its SHA-256 and type; a server with an unpinned or mismatching jar, or a writable plugin directory, refuses to start. Without the chart, put the jars in a read-only directory and set helix.custody.plugin-dir and helix.custody.plugins.
  • New chart keys, all optional: custody.plugins.image/.jars, custody.sovereignTypes, the kmip, aws-kms, azure-kv and gcp-kms backend types (the chart accepted no kmip backend before), and dbEncryption.next. Allow the backend's port in networkPolicy.extraEgressPorts (5696 for KMIP, 443 for the cloud services).
  • The KMIP client key must not be readable by others. The server refuses a tls.key that is world-readable or writable by anyone but its owner; owner and group read (mode 0440 with the pod's fsGroup) is accepted, which is what the chart mounts.
  • KMIP and the cloud plugins are Preview. Do not move a production realm onto one before you have tried it in your own account or key manager (see the Limits of the operator guide).

Rotating the database encryption key (new)

You can now rotate DB_ENCRYPTION online; the full procedure is in "Rotating the database encryption key" of Key custody. What matters for the upgrade:

  • Finish the rollout to rc.13 on every pod before you start a rotation. An rc.12 pod writes no heartbeat, so the start cannot see it, and it cannot read key-tagged values.
  • Once a rotation has started, rolling back to a release before rc.13 is not supported. From the start, values are written in a new key-tagged format (hx2.<key id>.) that rc.12 cannot read. Back up the database first.
  • A rotation needs DB_ENCRYPTION_NEXT (at least 32 characters, strength-checked; a weak or short one refuses start-up) or DB_ENCRYPTION_NEXT_WRAPPED with DB_ENCRYPTION_NEXT_KEK (the unwrapped key is 16, 24 or 32 bytes) on every pod, chart dbEncryption.next. Both keys must stay configured until the console says the old one can go; then set DB_ENCRYPTION to the new key and remove NEXT, restart, and only then remove the old key.
  • With Redis sessions (the chart default) the rotation stays DRAINING for at least the longest absolute session lifetime (sso_session_max_lifetime_seconds, about 10 hours by default) with both keys required. With the Redis token store keep the old key until oldKeyNeededUntil. If you extended flow-state or wallet-transaction lifetimes beyond 30 minutes (including their Redis TTLs), raise helix.db-encryption.rotation.drain to at least the longest.
  • A server that lacks a data key the database still needs refuses to start, with the key ids and the count of values affected in the message. Never start a pod with only the new key before the rotation is DONE.