Key custody plugins: developer and operator guide¶

A key custody plugin adds a backend type (a cloud KMS, a vendor HSM, a key manager of your own) to HelixIAM without changing the server. The plugin interface is public, documented and versioned; the built-in pkcs11, vault and kmip backends use the same interface. This guide is for the people who write a plugin and the operators who install one. HelixIAM ships three plugins of its own (aws-kms, azure-kv, gcp-kms, all Preview: see below). The operator guide for the backends, the sovereign key-store policy and the set-up of each cloud provider is key-custody.md; the SPI is Beta: the contract below is versioned, but expect additions in minor releases.

To build the server or a plugin locally, install the SPI first (it is a module of its own that both compile against): mvn -f helix-iam-server/custody-spi/pom.xml install, once and after every SPI change.

What a plugin is¶

One jar with one KeyCustodyFactory, built against the SPI artifact io.helixiam:helix-custody-spi (package io.helixiam.custody.spi, no third-party dependencies). The server finds the factory with ServiceLoader, asks it for a KeyCustody per configured backend entry, and calls that KeyCustody for everything a key needs: generate, sign, decrypt, encrypt, disable, destroy.

The type names software, pkcs11, vault and kmip are reserved. A plugin cannot replace a built-in.

The contract¶

The factory¶

public interface KeyCustodyFactory {
    String type();                                              // [a-z][a-z0-9-]{1,31}; also the key reference scheme
    KeyCustody create(BackendDefinition definition, KeyCustodyContext context);
    default Optional<String> keySpace(Map<String, String> settings) { return Optional.empty(); }
}

A public no-argument constructor, and no state but the backends it built.
create refuses (with CustodyRefusedException) a setting it does not know and an algorithm the backend cannot sign with. It never logs definition.settings(): they can name credential locations.
keySpace returns the key namespace two entries share (for example the endpoint and key ring). The server then refuses two entries with different realm bindings that reach the same keys. Leave the default when entries never share keys.
KeyCustodyContext is everything the server hands in: secretsDir() (/var/run/helix/custody/<id>, read-only, may not exist), clock(), logger(name) and opTimeout(). Read files, time and logs only through it.

The backend, method by method¶

Every method resolves a KeyRef only in the reference's own realm. A reference of another realm answers NoSuchElementException, never the key.

Method	What it does	Notes
`backendId()`	`definition.id()`
`capabilities()`	Signing algorithms, `nonExtractable`, `encrypt`, RSA sizes, key transports	What you advertise is what an administrator may configure. An algorithm outside `signing()` is refused when the configuration is saved, never at signing time.
`generate(realm, purpose, alg)`	A new signing key	Non-extractable where the backend supports it.
`generate(realm, purpose, alg, rsaBits, usage)`	Larger RSA keys, `SIGN_AND_DECRYPT`	Optional. The default only does 2048-bit signing keys and refuses the rest.
`publicKey(ref)`	The public key (JCA `PublicKey`)
`sign(ref, alg, data, encoding)`	The signature of `data` (you hash it)	`Encoding.JOSE` is raw `r\\|\\|s` for ECDSA (RFC 7518 section 3.4), `Encoding.DER` the ASN.1 form. Convert your provider's native form with `Signatures.derToJose` / `joseToDer`.
`decryptKeyTransport(ref, transport, encryptedKey)`	RSA-OAEP key transport for eID decryption keys	Optional. Exactly the parameters in `transport`, never others. Advertise them in `capabilities().keyTransport()`.
`createKek(realm)`	A new AES-256 key-encryption key
`createDbEncryptionKek(realm)`	The operator's `DB_ENCRYPTION_KEK`	Optional; named as a key of its own (kind `DB_KEK`).
`encrypt(kek, plaintext, aad)` / `decrypt(kek, ciphertext, aad)`	AES-256-GCM with the caller's AAD, exactly	A backend without authenticated encryption under caller-supplied AAD reports `encrypt=false` and serves `direct` keys only. A wrong AAD is refused.
`decryptForRewrap(kek, ciphertext, aad)`	`decrypt` for a KEK rotation, also for a quarantined KEK	The default calls `decrypt`. A disabled KEK is refused.
`disable(ref)` / `enable(ref)`	Stop a key from signing (a KEK then neither encrypts nor decrypts) / turn a KEK on again	`enable` is optional.
`destroy(ref)`	Remove the key for good
`health()`	`Health.up(latency)`, `Health.down(detail)`, `Health.policyViolation(...)`	Must answer within the op timeout. Never a secret in `detail`.
`lastError()`	`policyDenied` or null	Set it when the provider's policy refused a call that a fresh login did not cure.
`kind(ref)`	`SIGNING`, `DECRYPTION`, `KEK` or `DB_KEK`, answered locally	Never a remote call (the server asks while it prepares its configuration). Read `ref.locatorKind()`.
`keysChanged(realm)`	Drop cached keys or handles of that realm	`null` or `*` means every realm.

Failure semantics¶

A method throws exactly three exception types, and nothing else:

CustodyUnavailableException: the backend cannot answer now (no answer in time, unreachable, an answer that cannot be used). The caller gets 503 temporarily_unavailable with Retry-After. Counts against the circuit breaker.
CustodyRefusedException: the backend or the standard says no (a size, usage or algorithm that is not allowed, a ciphertext that does not decrypt, a wrong AAD). Never counts against the breaker.
NoSuchElementException: the key does not exist, or belongs to another realm.

Authentication or permission denial by the provider (HTTP 401/403, AccessDenied) is CustodyUnavailableException with lastError() = policyDenied: the operator can fix it. Only the provider's explicit refusal of the operation or its parameters is CustodyRefusedException.

Nothing falls back to a software key, another backend, another purpose or another algorithm. Never return a different key or algorithm than the one asked for. Any other exception you throw is mapped by the host to an unavailable result (metered result="contract", counted against the breaker, never a 500) and the conformance kit fails the check it came from. Messages carry no key material, PIN, token, credential or plaintext.

Key references¶

A key reference is <scheme>:<backendId>/<realm>/<locator>[#<version>]: the scheme is your factory's type(), the version (optional) [A-Za-z0-9._-]{1,64}, and the locator one segment that begins with the kind token you set at generation:

Kind	Token	Example
`SIGNING`	`sig-`	`aws-kms:prod/acme/sig-4f9c...`
`DECRYPTION`	`dec-`	`aws-kms:prod/acme/dec-...`
`KEK`	`kek-`	`aws-kms:prod/acme/kek-...`
`DB_KEK`	`dbkek-`	`aws-kms:prod/acme/dbkek-...`

Give the provider's own key name, alias or label the same token; kind(ref) then reads it back without a remote call. Provider details (region, key ring, vault) are backend settings, not part of the reference.

Manifest¶

The jar manifest carries three attributes. The server reads them before any class of the plugin loads.

Helix-Custody-SPI-Version: 1.0
Helix-Custody-Plugin-Type: aws-kms
Helix-Custody-Plugin-Version: 1.2.3

The SPI version is <major>.<minor>: the server accepts the same major and a minor no newer than its own.
The type equals KeyCustodyFactory.type() and the type the operator pinned.
The plugin version is semver.
META-INF/services/io.helixiam.custody.spi.KeyCustodyFactory lists exactly one factory.

Packaging¶

Build one shaded jar that contains the plugin and everything it needs (provider SDK, HTTP client, JSON library), except the SPI classes themselves: depend on helix-custody-spi with provided scope and relocate nothing from io.helixiam.custody.spi. A jar that carries its own copy of an SPI class cannot load that class. Keep the licence notices of the libraries you shade in the jar (META-INF/).

Each plugin gets its own class loader. Its parent shows the plugin the SPI package and the JDK, nothing else, so your shaded libraries cannot clash with the server's or another plugin's. This is dependency isolation, not a sandbox and not a security boundary: plugin code runs in the server process with the server's permissions. What makes a plugin trusted is its SHA-256 pin (below) and the signature of the plugin image it comes from.

Versioning (SPI policy)¶

The SPI has its own version (currently 1.0), independent of the product.

A minor release only adds default methods or new types. A plugin built against 1.0 runs on a 1.3 server.
Anything else (a removed or changed method, a changed contract) is a major release. A plugin of another major is refused at start-up.
A plugin built against a newer minor than the server is refused: it could call what the server lacks.

Test your plugin: the conformance kit¶

io.helixiam.custody.spi.conformance.CustodyConformance ships in the SPI artifact. It has no test framework dependency; call it from a test of your own against a real or emulated key service, and assert the report is clean.

final KeyCustody custody = new MyFactory().create(definition, context);
final ConformanceReport report = CustodyConformance.run(
    custody, "conformance-realm", CustodyConformance.Options.from(custody.capabilities()));
assertTrue(report.ok(), () -> String.join("\n", report.failed()));

It makes keys in the realm you give it and destroys them again: use a key store meant for tests. The checks, for everything Options selects:

per signing algorithm: a new key, its public key, a signature of 32 random bytes in each Encoding that verifies with the JDK against that public key, a disabled key that refuses to sign, the key destroyed;
a KEK that round-trips under its AAD, refuses another AAD, answers NoSuchElementException for the same key named in another realm, and refuses to encrypt once disabled;
for each key transport you list: a decryption key that decrypts a JDK ciphertext of exactly those parameters;
health() answers within the op timeout;
every exception is one of the three allowed ones.

Passing it is required, not sufficient: run it against the real service before you ship.

What the host guarantees¶

The server wraps every backend the same way, plugin or built-in:

Time bound: a call that finds no answer within opTimeout (the backend's own setting, else helix.custody.op-timeout, 2 s) is given up and is unavailable. Calls in flight are bounded per backend (helix.custody.max-concurrent-calls, 64); an excess call is shed at once, never queued.
Circuit breaker: 5 consecutive unavailable results within 10 s open it; it half-opens after 5 s with one trial call. Refusals and saturation sheds do not count.
Self-tests on every new key: a signing key signs a 32-byte challenge that must verify with the JDK against publicKey(ref) in the requested encoding; a KEK must round-trip and refuse a wrong AAD. A key that fails is disabled and the call refused. This catches a plugin that returns DER for JOSE, or the wrong key.
Realm binding: a backend bound to realms is never called for a key of another realm.
Fail closed: an outage is a 503 with Retry-After, never a fallback to a software key. Readiness never includes backend health.
Metrics and audit: helix_custody_* metrics per backend and result, audit events after commit. Secrets never reach logs, audit or the console.

Installing a plugin¶

Kubernetes (Helm)¶

A plugin ships in a signed plugin image that holds the jars under /plugins. The chart copies them with an init container into an emptyDir, which the server mounts read-only at /opt/helix/plugins. The image needs only a static cp (the command is cp -r /plugins/. /target/: no shell, no glob).

custody:
  plugins:
    image:
      repository: registry.example.com/helix-custody-plugins
      digest: sha256:<64 hex>            # by digest; a tag alone fails the render
    jars:
      - file: example-kms.jar
        sha256: "<64 hex of the jar's bytes>"
        type: example-kms
  sovereignTypes: []                     # plugin types you assert are sovereign key stores
  backends:
    - id: prod
      type: example-kms                  # must be pkcs11, vault, kmip, or the type of a plugin above
      realms: [acme]
      settings:                          # reach the plugin as BackendDefinition.settings (strings)
        region: eu-west-1
      secretFiles:                       # optional; mounted as /var/run/helix/custody/prod/<file>
        - {secret: kms-trust, key: ca.crt, file: ca.crt}   # e.g. a CA bundle or TLS trust file

The plugins HelixIAM ships have typed settings instead of a settings map (the chart checks each key). aws-kms (AWS KMS, Preview: tested against LocalStack and recordings, not yet against a real AWS account; a customer choice next to the sovereign key stores, never a default):

  backends:
    - id: aws-prod
      type: aws-kms
      region: eu-central-1               # required
      auth: web-identity                 # required: web-identity (IRSA) or pod-identity (EKS Pod Identity)
      roleArn: arn:aws:iam::123456789012:role/helix-custody   # web-identity only; default AWS_ROLE_ARN
      # origin: AWS_CLOUDHSM, customKeyStoreId: cks-...      # key-encryption keys in a CloudHSM key store
      # deletionWindowDays: 30, opTimeout: PT2S, algorithms: [ES256, RS256]
      # endpoint: https://vpce-....kms.eu-central-1.vpce.amazonaws.com     # KMS only
      # stsEndpoint: https://vpce-....sts.eu-central-1.vpce.amazonaws.com  # STS (default: regional)

It authenticates with the pod's workload identity only: the default AWS credential chain is never used, static keys in the environment are never read, and a static-key setting is refused at start-up. Both endpoints are always set explicitly (by default the regional ones), so AWS_ENDPOINT_URL* variables never redirect the token exchange or a signature. The same pinning switches off the SDK's FIPS and dual-stack variables, so operators in AWS GovCloud (us-gov-*) or anywhere the FIPS endpoints are required set endpoint and stsEndpoint to those endpoints explicitly. The HTTP client uses no proxy (the JVM's proxy properties are ignored): egress through a proxy is not supported yet, the pod must reach KMS and STS directly. insecureEndpointHosts lets endpoint and stsEndpoint be http:// to the hosts it lists; it is for tests only (LocalStack) and must never be set in a deployment. Keys are single-purpose (no sign-and-decrypt keys, no EdDSA), tagged helix:realm, helix:purpose and helix:kind, and a key tagged for another realm, an AWS managed key or imported key material (Origin=EXTERNAL) is never used. The chart's values.yaml lists the IAM actions the role needs.

azure-kv (Azure Key Vault or Azure Key Vault Managed HSM, Preview: tested against recordings built from the published REST API 7.5 shapes and a stand-in, not yet against a real Azure account; a customer choice, never a default):

  backends:
    - id: azure-prod
      type: azure-kv
      vaultUrl: https://helix-prod.managedhsm.azure.net   # or https://<name>.vault.azure.net (no key-encryption keys)
      # kind: managed-hsm                # default: the URL's; must match it
      # auth: workload-identity          # default; managed-identity only when you opt in
      # clientId / tenantId              # default AZURE_CLIENT_ID / AZURE_TENANT_ID from the workload identity webhook
      # protection: hsm                  # default (RSA-HSM / EC-HSM); software on a vault only
      # opTimeout: PT2S, algorithms: [ES256, RS256]

It authenticates with Azure Workload Identity: the projected service-account token (AZURE_FEDERATED_TOKEN_FILE) is exchanged with Entra ID as a federated client assertion, at the authority of the vault's cloud (login.microsoftonline.com, login.chinacloudapi.cn or login.microsoftonline.us), never at AZURE_AUTHORITY_HOST. A managed identity (the instance metadata service) is used only with auth: managed-identity. No client secret, client certificate or default credential chain is ever used: a setting that names one is refused at start-up. The plugin bundles no Azure SDK: it speaks the Key Vault REST API and the token endpoint itself over the JDK's HTTP client, so no AZURE_* variable other than the three of workload identity is read and every endpoint is pinned. AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_FEDERATED_TOKEN_FILE are each read from a Java system property of that name first, then from the environment. endpoint and identityEndpoint send the calls elsewhere (the vault URL still names the keys and the token's scope); identityEndpoint receives the pod's federated token or credential, so it is for tests and private stand-ins only, never a deployment; insecureEndpointHosts lets them be http:// and is for tests only. Like every cloud plugin, the HTTP client uses no proxy: egress through a proxy is not supported yet.

A Key Vault has no AES-GCM, so it holds no key-encryption keys, and the server refuses to switch a realm to Protected or High (both need the realm's KEK) on it: a realm needs a Managed HSM, which holds oct-HSM AES-256 keys and serves both. Keys are made exportable=false with no release policy, named helix-<kind>-<uuid> and tagged helix-realm, helix-purpose and helix-kind; a key that is exportable, has a release policy, backs a certificate or is tagged for another realm is never used. Key Vault returns raw r||s ECDSA signatures, converted to the encoding asked for. No EdDSA (Key Vault has no Ed25519) and no sign-and-decrypt keys. A destroyed key is deleted and purged when its vault allows it; with purge protection on it stays soft-deleted until its retention period ends, which is not an error. The chart's values.yaml lists the pod label, service-account annotation and permissions it needs.

gcp-kms (Google Cloud KMS, Preview: tested against recordings built from the published Cloud KMS REST API v1 shapes and a stand-in, not yet against a real Google Cloud account; a customer choice, never a default):

  backends:
    - id: gcp-prod
      type: gcp-kms
      project: helix-prod                # the existing key ring projects/<project>/locations/<location>/keyRings/<keyRing>
      location: europe-west4
      keyRing: helix-ring
      auth: gke-workload-identity        # or external-account (workload identity federation, any Kubernetes)
      # credentialConfigFile:            # external-account only: the external_account file (no secret in it)
      #   configMap: gcp-wif             # or secret: <name>
      #   key: credential-config.json
      #   tokenAudience: //iam.googleapis.com/projects/<n>/locations/global/workloadIdentityPools/<pool>/providers/<p>
      # protectionLevel: HSM             # default (Cloud HSM); SOFTWARE; Cloud EKM (EXTERNAL) is refused
      # destroyScheduledDuration: P30D   # default; P1D to P120D
      # opTimeout: PT2S, algorithms: [ES256, RS256]

It authenticates with GKE Workload Identity (the metadata server's token for the pod's Kubernetes service account, at 169.254.169.254, never at GCE_METADATA_HOST) or with an external_account file: the subject token named by its credential_source (a file, such as the service-account token the chart projects with tokenAudience, or a url) is exchanged at sts.googleapis.com, then, when the file names one, a service account is impersonated at iamcredentials.googleapis.com. The file is checked at start-up: its type must be external_account, its token_url and impersonation URL the pinned Google ones, its universe googleapis.com; a service_account key file, a file with a private key, client secret or token, an executable credential source (no command is ever run) and the AWS source are refused, and so is any setting that names a key file or Application Default Credentials. The external_account file holds no secret. The plugin bundles no Google SDK and reads no environment variable (not GOOGLE_APPLICATION_CREDENTIALS or GOOGLE_CLOUD_UNIVERSE_DOMAIN); every endpoint is pinned, and endpoint (a regional or Private Service Connect Cloud KMS endpoint) is the only override for a deployment (identityEndpoint receives the pod's credential and, like insecureEndpointHosts, is for tests only). Cloud EKM (EXTERNAL, EXTERNAL_VPC) is not supported, and no egress proxy is used.

A Cloud KMS key has one purpose: signing keys are ASYMMETRIC_SIGN keys of exactly one algorithm (EC_SIGN_P256_SHA256, EC_SIGN_P384_SHA384, RSA_SIGN_PKCS1_<bits>_SHA256, RSA_SIGN_PSS_<bits>_SHA256), so a key signs only with the algorithm it was made for, and key-encryption keys are ENCRYPT_DECRYPT keys (GOOGLE_SYMMETRIC_ENCRYPTION, AES-256-GCM) used with the caller's associated data as additionalAuthenticatedData. Every request carries the CRC32C of its inputs and every answer's CRC32C fields are checked, as Google's data-integrity guidance asks; a mismatch is reported as unavailable, never used. Cloud KMS returns DER ECDSA signatures, converted to the encoding asked for. No EdDSA (Cloud KMS signs Ed25519 over raw messages only) and no sign-and-decrypt keys. Keys are named hx-<kind>-<uuid> and labelled helix-realm, helix-purpose and helix-kind; a key labelled for another realm, of another protection level than the configured one, or with imported key material (an importJob) is never used. Destroying a key schedules its version's destruction (destroyScheduledDuration); Cloud KMS keeps the key's id. The chart's values.yaml lists the Workload Identity set-up and the permissions the identity needs on the key ring.

The render fails when a jar has a malformed pin, an image has no digest, a type clashes with a built-in or with another jar, or a backend has a type that no plugin provides. settings end up in a ConfigMap, so a key that names a credential (secret, password, token, credential, private, apikey, accesskey, in any case) is refused: mount files with secretFiles. Cloud backends authenticate through the pod's workload identity (IRSA or Pod Identity, Azure Workload Identity, GKE Workload Identity), never static keys, so a cloud plugin takes no credential at all. A release's packaged chart already carries the digest and pins of the plugins built with it (the release refuses a plugin image without the aws-kms, azure-kv and gcp-kms jars, and never pins the fixture plugin); the values are empty in git, so nothing loads by default.

Do not point the plugin directory at a ConfigMap, Secret or PersistentVolume: they add ..data symlinks, ..<timestamp> directories or lost+found, which the loader refuses.

Elsewhere¶

Put the jars in a directory, make that directory and the jars read-only for the user the server runs as (owned by another user, mode 0555/0444, or a read-only mount), and configure:

helix:
  custody:
    plugin-dir: /opt/helix/plugins       # the default
    plugins:
      - {file: aws-kms.jar, sha256: "<64 hex>", type: aws-kms}
    sovereign-types: []

What the server checks, and refuses at start-up¶

Every check runs on every jar before any plugin class loads. The server refuses to start with a clear, secret-free message when:

a file in the plugin directory is not pinned, or a pin has no jar;
an entry is a symlink or not a .jar;
the directory or a jar is writable by the server process (a writable /opt/helix/plugins in a custom image counts);
the jar's SHA-256 differs from the pin (the server hashes the bytes it then loads, so nothing changes in between);
the manifest is missing an attribute, names an SPI version the server does not run, a type other than the pinned one, a reserved type or a type that is already loaded, or the services file does not list exactly one factory;
a configured backend has a type no loaded factory provides.

Compute the pin with sha256sum aws-kms.jar (lower-case hex).