# Key custody plugins: developer and operator guide A key custody plugin adds a backend type (a cloud KMS, a vendor HSM, a key manager of your own) to HelixIAM without changing the server. The plugin interface is public, documented and versioned; the built-in `pkcs11`, `vault` and `kmip` backends use the same interface. This guide is for the people who write a plugin and the operators who install one. HelixIAM ships three plugins of its own (`aws-kms`, `azure-kv`, `gcp-kms`, all **Preview**: see below). The operator guide for the backends, the sovereign key-store policy and the set-up of each cloud provider is [key-custody.md](key-custody.md); the SPI is **Beta**: the contract below is versioned, but expect additions in minor releases. To build the server or a plugin locally, install the SPI first (it is a module of its own that both compile against): `mvn -f helix-iam-server/custody-spi/pom.xml install`, once and after every SPI change. ## What a plugin is One jar with one `KeyCustodyFactory`, built against the SPI artifact `io.helixiam:helix-custody-spi` (package `io.helixiam.custody.spi`, no third-party dependencies). The server finds the factory with `ServiceLoader`, asks it for a `KeyCustody` per configured backend entry, and calls that `KeyCustody` for everything a key needs: generate, sign, decrypt, encrypt, disable, destroy. The type names `software`, `pkcs11`, `vault` and `kmip` are reserved. A plugin cannot replace a built-in. ## The contract ### The factory ```java public interface KeyCustodyFactory { String type(); // [a-z][a-z0-9-]{1,31}; also the key reference scheme KeyCustody create(BackendDefinition definition, KeyCustodyContext context); default Optional keySpace(Map settings) { return Optional.empty(); } } ``` - A public no-argument constructor, and no state but the backends it built. - `create` refuses (with `CustodyRefusedException`) a setting it does not know and an algorithm the backend cannot sign with. It never logs `definition.settings()`: they can name credential locations. - `keySpace` returns the key namespace two entries share (for example the endpoint and key ring). The server then refuses two entries with different realm bindings that reach the same keys. Leave the default when entries never share keys. - `KeyCustodyContext` is everything the server hands in: `secretsDir()` (`/var/run/helix/custody/`, read-only, may not exist), `clock()`, `logger(name)` and `opTimeout()`. Read files, time and logs only through it. ### The backend, method by method Every method resolves a `KeyRef` only in the reference's own realm. A reference of another realm answers `NoSuchElementException`, never the key. | Method | What it does | Notes | | --- | --- | --- | | `backendId()` | `definition.id()` | | | `capabilities()` | Signing algorithms, `nonExtractable`, `encrypt`, RSA sizes, key transports | What you advertise is what an administrator may configure. An algorithm outside `signing()` is refused when the configuration is saved, never at signing time. | | `generate(realm, purpose, alg)` | A new signing key | Non-extractable where the backend supports it. | | `generate(realm, purpose, alg, rsaBits, usage)` | Larger RSA keys, `SIGN_AND_DECRYPT` | Optional. The default only does 2048-bit signing keys and refuses the rest. | | `publicKey(ref)` | The public key (JCA `PublicKey`) | | | `sign(ref, alg, data, encoding)` | The signature of `data` (you hash it) | `Encoding.JOSE` is raw `r\|\|s` for ECDSA (RFC 7518 section 3.4), `Encoding.DER` the ASN.1 form. Convert your provider's native form with `Signatures.derToJose` / `joseToDer`. | | `decryptKeyTransport(ref, transport, encryptedKey)` | RSA-OAEP key transport for eID decryption keys | Optional. Exactly the parameters in `transport`, never others. Advertise them in `capabilities().keyTransport()`. | | `createKek(realm)` | A new AES-256 key-encryption key | | | `createDbEncryptionKek(realm)` | The operator's `DB_ENCRYPTION_KEK` | Optional; named as a key of its own (kind `DB_KEK`). | | `encrypt(kek, plaintext, aad)` / `decrypt(kek, ciphertext, aad)` | AES-256-GCM with the caller's AAD, exactly | A backend without authenticated encryption under caller-supplied AAD reports `encrypt=false` and serves `direct` keys only. A wrong AAD is refused. | | `decryptForRewrap(kek, ciphertext, aad)` | `decrypt` for a KEK rotation, also for a quarantined KEK | The default calls `decrypt`. A disabled KEK is refused. | | `disable(ref)` / `enable(ref)` | Stop a key from signing (a KEK then neither encrypts nor decrypts) / turn a KEK on again | `enable` is optional. | | `destroy(ref)` | Remove the key for good | | | `health()` | `Health.up(latency)`, `Health.down(detail)`, `Health.policyViolation(...)` | Must answer within the op timeout. Never a secret in `detail`. | | `lastError()` | `policyDenied` or null | Set it when the provider's policy refused a call that a fresh login did not cure. | | `kind(ref)` | `SIGNING`, `DECRYPTION`, `KEK` or `DB_KEK`, answered locally | Never a remote call (the server asks while it prepares its configuration). Read `ref.locatorKind()`. | | `keysChanged(realm)` | Drop cached keys or handles of that realm | `null` or `*` means every realm. | ### Failure semantics A method throws exactly three exception types, and nothing else: - `CustodyUnavailableException`: the backend cannot answer now (no answer in time, unreachable, an answer that cannot be used). The caller gets 503 `temporarily_unavailable` with `Retry-After`. Counts against the circuit breaker. - `CustodyRefusedException`: the backend or the standard says no (a size, usage or algorithm that is not allowed, a ciphertext that does not decrypt, a wrong AAD). Never counts against the breaker. - `NoSuchElementException`: the key does not exist, or belongs to another realm. Authentication or permission denial by the provider (HTTP 401/403, `AccessDenied`) is `CustodyUnavailableException` with `lastError()` = `policyDenied`: the operator can fix it. Only the provider's explicit refusal of the operation or its parameters is `CustodyRefusedException`. Nothing falls back to a software key, another backend, another purpose or another algorithm. Never return a different key or algorithm than the one asked for. Any other exception you throw is mapped by the host to an unavailable result (metered `result="contract"`, counted against the breaker, never a 500) and the conformance kit fails the check it came from. Messages carry no key material, PIN, token, credential or plaintext. ### Key references A key reference is `://[#]`: the scheme is your factory's `type()`, the version (optional) `[A-Za-z0-9._-]{1,64}`, and the locator one segment that **begins with the kind token** you set at generation: | Kind | Token | Example | | --- | --- | --- | | `SIGNING` | `sig-` | `aws-kms:prod/acme/sig-4f9c...` | | `DECRYPTION` | `dec-` | `aws-kms:prod/acme/dec-...` | | `KEK` | `kek-` | `aws-kms:prod/acme/kek-...` | | `DB_KEK` | `dbkek-` | `aws-kms:prod/acme/dbkek-...` | Give the provider's own key name, alias or label the same token; `kind(ref)` then reads it back without a remote call. Provider details (region, key ring, vault) are backend settings, not part of the reference. ### Manifest The jar manifest carries three attributes. The server reads them before any class of the plugin loads. ``` Helix-Custody-SPI-Version: 1.0 Helix-Custody-Plugin-Type: aws-kms Helix-Custody-Plugin-Version: 1.2.3 ``` - The SPI version is `.`: the server accepts the same major and a minor no newer than its own. - The type equals `KeyCustodyFactory.type()` and the `type` the operator pinned. - The plugin version is semver. - `META-INF/services/io.helixiam.custody.spi.KeyCustodyFactory` lists exactly one factory. ### Packaging Build **one shaded jar** that contains the plugin and everything it needs (provider SDK, HTTP client, JSON library), except the SPI classes themselves: depend on `helix-custody-spi` with `provided` scope and relocate nothing from `io.helixiam.custody.spi`. A jar that carries its own copy of an SPI class cannot load that class. Keep the licence notices of the libraries you shade in the jar (`META-INF/`). Each plugin gets its own class loader. Its parent shows the plugin the SPI package and the JDK, nothing else, so your shaded libraries cannot clash with the server's or another plugin's. This is dependency isolation, not a sandbox and not a security boundary: plugin code runs in the server process with the server's permissions. What makes a plugin trusted is its SHA-256 pin (below) and the signature of the plugin image it comes from. ### Versioning (SPI policy) The SPI has its own version (currently `1.0`), independent of the product. - A minor release only adds `default` methods or new types. A plugin built against `1.0` runs on a `1.3` server. - Anything else (a removed or changed method, a changed contract) is a major release. A plugin of another major is refused at start-up. - A plugin built against a newer minor than the server is refused: it could call what the server lacks. ## Test your plugin: the conformance kit `io.helixiam.custody.spi.conformance.CustodyConformance` ships in the SPI artifact. It has no test framework dependency; call it from a test of your own against a real or emulated key service, and assert the report is clean. ```java final KeyCustody custody = new MyFactory().create(definition, context); final ConformanceReport report = CustodyConformance.run( custody, "conformance-realm", CustodyConformance.Options.from(custody.capabilities())); assertTrue(report.ok(), () -> String.join("\n", report.failed())); ``` It makes keys in the realm you give it and destroys them again: use a key store meant for tests. The checks, for everything `Options` selects: - per signing algorithm: a new key, its public key, a signature of 32 random bytes in each `Encoding` that verifies with the JDK against that public key, a disabled key that refuses to sign, the key destroyed; - a KEK that round-trips under its AAD, refuses another AAD, answers `NoSuchElementException` for the same key named in another realm, and refuses to encrypt once disabled; - for each key transport you list: a decryption key that decrypts a JDK ciphertext of exactly those parameters; - `health()` answers within the op timeout; - every exception is one of the three allowed ones. Passing it is required, not sufficient: run it against the real service before you ship. ## What the host guarantees The server wraps every backend the same way, plugin or built-in: - **Time bound:** a call that finds no answer within `opTimeout` (the backend's own setting, else `helix.custody.op-timeout`, 2 s) is given up and is unavailable. Calls in flight are bounded per backend (`helix.custody.max-concurrent-calls`, 64); an excess call is shed at once, never queued. - **Circuit breaker:** 5 consecutive unavailable results within 10 s open it; it half-opens after 5 s with one trial call. Refusals and saturation sheds do not count. - **Self-tests on every new key:** a signing key signs a 32-byte challenge that must verify with the JDK against `publicKey(ref)` in the requested encoding; a KEK must round-trip and refuse a wrong AAD. A key that fails is disabled and the call refused. This catches a plugin that returns DER for JOSE, or the wrong key. - **Realm binding:** a backend bound to realms is never called for a key of another realm. - **Fail closed:** an outage is a 503 with `Retry-After`, never a fallback to a software key. Readiness never includes backend health. - **Metrics and audit:** `helix_custody_*` metrics per backend and result, audit events after commit. Secrets never reach logs, audit or the console. ## Installing a plugin ### Kubernetes (Helm) A plugin ships in a signed plugin image that holds the jars under `/plugins`. The chart copies them with an init container into an `emptyDir`, which the server mounts **read-only** at `/opt/helix/plugins`. The image needs only a static `cp` (the command is `cp -r /plugins/. /target/`: no shell, no glob). ```yaml custody: plugins: image: repository: registry.example.com/helix-custody-plugins digest: sha256:<64 hex> # by digest; a tag alone fails the render jars: - file: example-kms.jar sha256: "<64 hex of the jar's bytes>" type: example-kms sovereignTypes: [] # plugin types you assert are sovereign key stores backends: - id: prod type: example-kms # must be pkcs11, vault, kmip, or the type of a plugin above realms: [acme] settings: # reach the plugin as BackendDefinition.settings (strings) region: eu-west-1 secretFiles: # optional; mounted as /var/run/helix/custody/prod/ - {secret: kms-trust, key: ca.crt, file: ca.crt} # e.g. a CA bundle or TLS trust file ``` The plugins HelixIAM ships have typed settings instead of a `settings` map (the chart checks each key). `aws-kms` (AWS KMS, **Preview**: tested against LocalStack and recordings, not yet against a real AWS account; a customer choice next to the sovereign key stores, never a default): ```yaml backends: - id: aws-prod type: aws-kms region: eu-central-1 # required auth: web-identity # required: web-identity (IRSA) or pod-identity (EKS Pod Identity) roleArn: arn:aws:iam::123456789012:role/helix-custody # web-identity only; default AWS_ROLE_ARN # origin: AWS_CLOUDHSM, customKeyStoreId: cks-... # key-encryption keys in a CloudHSM key store # deletionWindowDays: 30, opTimeout: PT2S, algorithms: [ES256, RS256] # endpoint: https://vpce-....kms.eu-central-1.vpce.amazonaws.com # KMS only # stsEndpoint: https://vpce-....sts.eu-central-1.vpce.amazonaws.com # STS (default: regional) ``` It authenticates with the pod's workload identity only: the default AWS credential chain is never used, static keys in the environment are never read, and a static-key setting is refused at start-up. Both endpoints are always set explicitly (by default the regional ones), so `AWS_ENDPOINT_URL*` variables never redirect the token exchange or a signature. The same pinning switches off the SDK's FIPS and dual-stack variables, so **operators in AWS GovCloud (`us-gov-*`) or anywhere the FIPS endpoints are required set `endpoint` and `stsEndpoint` to those endpoints explicitly**. The HTTP client uses no proxy (the JVM's proxy properties are ignored): **egress through a proxy is not supported yet**, the pod must reach KMS and STS directly. `insecureEndpointHosts` lets `endpoint` and `stsEndpoint` be `http://` to the hosts it lists; it is **for tests only** (LocalStack) and must never be set in a deployment. Keys are single-purpose (no sign-and-decrypt keys, no EdDSA), tagged `helix:realm`, `helix:purpose` and `helix:kind`, and a key tagged for another realm, an AWS managed key or imported key material (`Origin=EXTERNAL`) is never used. The chart's `values.yaml` lists the IAM actions the role needs. `azure-kv` (Azure Key Vault or Azure Key Vault Managed HSM, **Preview**: tested against recordings built from the published REST API 7.5 shapes and a stand-in, not yet against a real Azure account; a customer choice, never a default): ```yaml backends: - id: azure-prod type: azure-kv vaultUrl: https://helix-prod.managedhsm.azure.net # or https://.vault.azure.net (no key-encryption keys) # kind: managed-hsm # default: the URL's; must match it # auth: workload-identity # default; managed-identity only when you opt in # clientId / tenantId # default AZURE_CLIENT_ID / AZURE_TENANT_ID from the workload identity webhook # protection: hsm # default (RSA-HSM / EC-HSM); software on a vault only # opTimeout: PT2S, algorithms: [ES256, RS256] ``` It authenticates with Azure Workload Identity: the projected service-account token (`AZURE_FEDERATED_TOKEN_FILE`) is exchanged with Entra ID as a federated client assertion, at the authority of the vault's cloud (`login.microsoftonline.com`, `login.chinacloudapi.cn` or `login.microsoftonline.us`), never at `AZURE_AUTHORITY_HOST`. A managed identity (the instance metadata service) is used only with `auth: managed-identity`. No client secret, client certificate or default credential chain is ever used: a setting that names one is refused at start-up. The plugin bundles no Azure SDK: it speaks the Key Vault REST API and the token endpoint itself over the JDK's HTTP client, so no `AZURE_*` variable other than the three of workload identity is read and every endpoint is pinned. `AZURE_CLIENT_ID`, `AZURE_TENANT_ID` and `AZURE_FEDERATED_TOKEN_FILE` are each read from a **Java system property of that name first, then from the environment**. `endpoint` and `identityEndpoint` send the calls elsewhere (the vault URL still names the keys and the token's scope); **`identityEndpoint` receives the pod's federated token or credential, so it is for tests and private stand-ins only, never a deployment**; `insecureEndpointHosts` lets them be `http://` and is **for tests only**. Like every cloud plugin, the HTTP client uses no proxy: egress through a proxy is not supported yet. A Key Vault has no AES-GCM, so it holds no key-encryption keys, and the server refuses to switch a realm to Protected or High (both need the realm's KEK) on it: **a realm needs a Managed HSM**, which holds `oct-HSM` AES-256 keys and serves both. Keys are made `exportable=false` with no release policy, named `helix--` and tagged `helix-realm`, `helix-purpose` and `helix-kind`; a key that is exportable, has a release policy, backs a certificate or is tagged for another realm is never used. Key Vault returns raw `r||s` ECDSA signatures, converted to the encoding asked for. No EdDSA (Key Vault has no Ed25519) and no sign-and-decrypt keys. A destroyed key is deleted and purged when its vault allows it; with purge protection on it stays soft-deleted until its retention period ends, which is not an error. The chart's `values.yaml` lists the pod label, service-account annotation and permissions it needs. `gcp-kms` (Google Cloud KMS, **Preview**: tested against recordings built from the published Cloud KMS REST API v1 shapes and a stand-in, not yet against a real Google Cloud account; a customer choice, never a default): ```yaml backends: - id: gcp-prod type: gcp-kms project: helix-prod # the existing key ring projects//locations//keyRings/ location: europe-west4 keyRing: helix-ring auth: gke-workload-identity # or external-account (workload identity federation, any Kubernetes) # credentialConfigFile: # external-account only: the external_account file (no secret in it) # configMap: gcp-wif # or secret: # key: credential-config.json # tokenAudience: //iam.googleapis.com/projects//locations/global/workloadIdentityPools//providers/

# protectionLevel: HSM # default (Cloud HSM); SOFTWARE; Cloud EKM (EXTERNAL) is refused # destroyScheduledDuration: P30D # default; P1D to P120D # opTimeout: PT2S, algorithms: [ES256, RS256] ``` It authenticates with GKE Workload Identity (the metadata server's token for the pod's Kubernetes service account, at `169.254.169.254`, never at `GCE_METADATA_HOST`) or with an `external_account` file: the subject token named by its `credential_source` (a `file`, such as the service-account token the chart projects with `tokenAudience`, or a `url`) is exchanged at `sts.googleapis.com`, then, when the file names one, a service account is impersonated at `iamcredentials.googleapis.com`. The file is checked at start-up: its type must be `external_account`, its `token_url` and impersonation URL the pinned Google ones, its universe `googleapis.com`; a `service_account` key file, a file with a private key, client secret or token, an `executable` credential source (no command is ever run) and the AWS source are refused, and so is any setting that names a key file or Application Default Credentials. The `external_account` file holds no secret. The plugin bundles no Google SDK and reads no environment variable (not `GOOGLE_APPLICATION_CREDENTIALS` or `GOOGLE_CLOUD_UNIVERSE_DOMAIN`); every endpoint is pinned, and `endpoint` (a regional or Private Service Connect Cloud KMS endpoint) is the only override for a deployment (`identityEndpoint` receives the pod's credential and, like `insecureEndpointHosts`, is **for tests only**). **Cloud EKM (`EXTERNAL`, `EXTERNAL_VPC`) is not supported**, and no egress proxy is used. A Cloud KMS key has one purpose: signing keys are `ASYMMETRIC_SIGN` keys of exactly one algorithm (`EC_SIGN_P256_SHA256`, `EC_SIGN_P384_SHA384`, `RSA_SIGN_PKCS1__SHA256`, `RSA_SIGN_PSS__SHA256`), so a key signs only with the algorithm it was made for, and key-encryption keys are `ENCRYPT_DECRYPT` keys (`GOOGLE_SYMMETRIC_ENCRYPTION`, AES-256-GCM) used with the caller's associated data as `additionalAuthenticatedData`. Every request carries the CRC32C of its inputs and every answer's CRC32C fields are checked, as Google's data-integrity guidance asks; a mismatch is reported as unavailable, never used. Cloud KMS returns DER ECDSA signatures, converted to the encoding asked for. No EdDSA (Cloud KMS signs Ed25519 over raw messages only) and no sign-and-decrypt keys. Keys are named `hx--` and labelled `helix-realm`, `helix-purpose` and `helix-kind`; a key labelled for another realm, of another protection level than the configured one, or with imported key material (an `importJob`) is never used. Destroying a key schedules its version's destruction (`destroyScheduledDuration`); Cloud KMS keeps the key's id. The chart's `values.yaml` lists the Workload Identity set-up and the permissions the identity needs on the key ring. The render fails when a jar has a malformed pin, an image has no digest, a type clashes with a built-in or with another jar, or a backend has a type that no plugin provides. `settings` end up in a ConfigMap, so a key that names a credential (secret, password, token, credential, private, apikey, accesskey, in any case) is refused: mount files with `secretFiles`. Cloud backends authenticate through the pod's workload identity (IRSA or Pod Identity, Azure Workload Identity, GKE Workload Identity), never static keys, so a cloud plugin takes no credential at all. A release's packaged chart already carries the digest and pins of the plugins built with it (the release refuses a plugin image without the `aws-kms`, `azure-kv` and `gcp-kms` jars, and never pins the `fixture` plugin); the values are empty in git, so nothing loads by default. Do not point the plugin directory at a ConfigMap, Secret or PersistentVolume: they add `..data` symlinks, `..` directories or `lost+found`, which the loader refuses. ### Elsewhere Put the jars in a directory, make that directory and the jars read-only for the user the server runs as (owned by another user, mode 0555/0444, or a read-only mount), and configure: ```yaml helix: custody: plugin-dir: /opt/helix/plugins # the default plugins: - {file: aws-kms.jar, sha256: "<64 hex>", type: aws-kms} sovereign-types: [] ``` ### What the server checks, and refuses at start-up Every check runs on every jar before any plugin class loads. The server refuses to start with a clear, secret-free message when: - a file in the plugin directory is not pinned, or a pin has no jar; - an entry is a symlink or not a `.jar`; - the directory or a jar is writable by the server process (a writable `/opt/helix/plugins` in a custom image counts); - the jar's SHA-256 differs from the pin (the server hashes the bytes it then loads, so nothing changes in between); - the manifest is missing an attribute, names an SPI version the server does not run, a type other than the pinned one, a reserved type or a type that is already loaded, or the services file does not list exactly one factory; - a configured backend has a type no loaded factory provides. Compute the pin with `sha256sum aws-kms.jar` (lower-case hex).