# Upgrading from 1.1.0-rc.12 to 1.1.0-rc.13

Key custody phase 3 (KMIP, plugins, cloud key stores, sovereign key-store policy) and online data-key rotation.

Nothing changes for a realm until you configure something: no backend, no plugin and no policy is enabled by default,
and a realm stays in the profile it has. What every installation meets: the three migrations below, the new `dataKey`
readiness contributor, the SPI install for local builds, and the value format of a data-key rotation once you start one.
The operator guides are [Key custody](key-custody.md) and [Key custody plugins](custody-plugins.md).

## Before you upgrade (summary)

- **Migration V189** adds `realm_custody_settings.key_store_policy` (`varchar(16) NOT NULL DEFAULT 'ANY'`) and a check
  (`ANY` or `SOVEREIGN`, added `NOT VALID`). Constant default: catalog-only, no table rewrite. Every realm is `ANY`.
- **Migration V190** adds the table `data_key_state` (at most one row: the id of the data key new values are written
  under). It stays empty until the first rotation. Catalog-only.
- **Migration V191** adds the tables `data_key_replica` (each replica's heartbeat: key ids only) and `data_key_rotation`,
  a change notification on `data_key_state`, and widens five columns to `text` because a key-tagged value is 21
  characters longer: `service_provider_oauth.client_secret`, `user_credentials.mfa_secret`, `user_bsn.bsn`,
  `webhook_subscription.secret` and `scim_target.token`. `varchar` to `text` is binary-coercible in PostgreSQL (no rewrite,
  no index rebuild), but each `ALTER` takes a short exclusive lock on its table: run the upgrade outside a burst of
  sign-ins. None of these changes touches a value.
- **Rolling upgrade.** rc.12 and rc.13 pods can run side by side: until the first data-key rotation starts, values keep
  the format of rc.12, so an rc.12 pod reads everything an rc.13 pod writes. Nothing needs a pause.
- **A new `dataKey` readiness contributor.** A pod is unready when it does not hold the data key the database writes
  under, and, only while a rotation is open, when its heartbeat has stopped. With no rotation (every upgrade), it is
  always ready.
- **Local builds need the SPI installed first.** The server compiles against `io.helixiam:helix-custody-spi`
  (`helix-iam-server/custody-spi`): run `mvn -f helix-iam-server/custody-spi/pom.xml install` once, and after every SPI
  change, before `mvn package` or `mvn test` in `helix-iam-server`. CI, the release build and the container image build
  do this for you.
- **If you built against the internal custody factory** (a `KeyCustodyFactory` of your own compiled against rc.12's
  `io.helixiam.authorization.service.key.custody` package, which was never a supported interface): the types moved to
  the public package `io.helixiam.custody.spi`, `create` now takes a `BackendDefinition` and a `KeyCustodyContext`, and the
  services file is renamed to `META-INF/services/io.helixiam.custody.spi.KeyCustodyFactory` (a factory listed in the old
  file is not found). Rebuild it as a plugin: see [Key custody plugins](custody-plugins.md). The built-in `pkcs11` and `vault`
  backends are unchanged for operators.
- **The plugin image.** The release publishes a second, signed image holding the plugin jars (`aws-kms`, `azure-kv`,
  `gcp-kms`). The packaged chart already carries its digest and the SHA-256 pin of each jar in `custody.plugins`; nothing
  loads until a backend of such a type is configured, and the chart refuses a backend whose type no pinned plugin
  provides. If you use your own plugin image, pin it by digest and list each jar with its SHA-256 and type; a server with
  an unpinned or mismatching jar, or a writable plugin directory, refuses to start. Without the chart, put the jars in a
  read-only directory and set `helix.custody.plugin-dir` and `helix.custody.plugins`.
- **New chart keys, all optional:** `custody.plugins.image`/`.jars`, `custody.sovereignTypes`, the `kmip`, `aws-kms`,
  `azure-kv` and `gcp-kms` backend types (the chart accepted no `kmip` backend before), and `dbEncryption.next`. Allow
  the backend's port in `networkPolicy.extraEgressPorts` (5696 for KMIP, 443 for the cloud services).
- **The KMIP client key must not be readable by others.** The server refuses a `tls.key` that is world-readable or
  writable by anyone but its owner; owner and group read (mode 0440 with the pod's `fsGroup`) is accepted, which is what
  the chart mounts.
- **KMIP and the cloud plugins are Preview.** Do not move a production realm onto one before you have tried it in your own
  account or key manager (see the Limits of the operator guide).

## Rotating the database encryption key (new)

You can now rotate `DB_ENCRYPTION` online; the full procedure is in "Rotating the database encryption key" of
[Key custody](key-custody.md). What matters for the upgrade:

- **Finish the rollout to rc.13 on every pod before you start a rotation.** An rc.12 pod writes no heartbeat, so the start
  cannot see it, and it cannot read key-tagged values.
- **Once a rotation has started, rolling back to a release before rc.13 is not supported.** From the start, values are
  written in a new key-tagged format (`hx2.<key id>.`) that rc.12 cannot read. Back up the database first.
- A rotation needs `DB_ENCRYPTION_NEXT` (at least 32 characters, strength-checked; a weak or short one refuses start-up) or
  `DB_ENCRYPTION_NEXT_WRAPPED` with `DB_ENCRYPTION_NEXT_KEK` (the unwrapped key is 16, 24 or 32 bytes) on every pod, chart
  `dbEncryption.next`. Both keys must stay configured until the console says the old one can go; then set
  `DB_ENCRYPTION` to the new key and remove NEXT, restart, and only then remove the old key.
- **With Redis sessions (the chart default) the rotation stays `DRAINING` for at least the longest absolute session
  lifetime (`sso_session_max_lifetime_seconds`, about 10 hours by default)** with both keys required. With the Redis
  token store keep the old key until `oldKeyNeededUntil`. If you extended flow-state or wallet-transaction lifetimes
  beyond 30 minutes (including their Redis TTLs), raise `helix.db-encryption.rotation.drain` to at least the longest.
- A server that lacks a data key the database still needs refuses to start, with the key ids and the count of values
  affected in the message. Never start a pod with only the new key before the rotation is `DONE`.