Kryptkeeper
Lab records

Tested environments and proof

For each area: where it ran, what ran, how it ended, and a link to the record. The gaps are listed too.

Commit hashes on this page are from the development repository. Where a version has a public release, the public tag and commit are shown beside it. Every entry comes from a record kept in the repository. A pass means the test exited cleanly on the named build. "Stand-ins" means the other side was a local copy or emulator, not the real service. Each record names the build it ran on. Some are from earlier releases and say so.

  • Pass 12 real
  • Partial 2 real
  • Mock pass 3 mocks only

Real systems: 12 pass, 2 partial. Mocks only: 3. 1 record includes failed checks.

Lab records

  1. Release 1.1.0 lab regression sweep

    PartialReal systems and stand-ins

    Full lab regression · 2026-10-26 · 1.1.0 (66b2f9c0) · public v1.1.0 6ca9f70 · smoke 446/446

    Environment
    Linux runner (Ubuntu 24.04 amd64, VM, 8 cores) with podman 4.9.3, kind 0.30, OpenSSL 3.5.4 and cosign 3.1.3. Compose, SSE and HA legs on an Apple silicon Mac with podman 6.1.3.
    What ran
    The 1.0.0 tests/live harness sweep, repeated on the 1.1.0 tag, plus the smoke suite.
    Note
    Three harnesses did not pass. pqc-wave failed on the runner because Go 1.26 cannot complete a TLS handshake with an ML-DSA certificate; it passed on the Mac and the build moved to Go 1.27. inventory-sources failed from test rot (the Entrust fixture). The storm profile finalizes 2000 of 2000 in both legs with no duplicates but breaks its strict limit gate. Two areas failed: inventory sources (test fixture out of date) and the renewal storm limit gate (8 and 18 rate-limit violations, 375 s against a 360 s goal). Neither was a regression from 1.0.0.

    Output from the record, verbatim

    Run header: the code under test from docs/release/LIVE_RESULTS_v1.1.0_REGRESSION.md

    - Code under test: `66b2f9c002fab8c30f125a9cd2232acee0347cd6` (tag v1.1.0), synced to the runner by git bundle

    Kubernetes, compose and smoke rows of the harness table from docs/release/LIVE_RESULTS_v1.1.0_REGRESSION.md

    | Kubernetes | k8s, certmanager, helm-kind, helm-ha, mesh, istio-mtls, promote-rollback, airgap (`KK_AIRGAP_LOCAL_IMAGES=1`, 60 s capture) | PASS |
    | Compose | compose, sse (600 s stream), ha (chaos, Mac podman) | PASS |
    | Smoke | `tests/smoke/run.sh` | 446 / 446 passed (1.0.0: 445) |
    16 areas in this record: 13 pass, 1 partial, 2 fail
    • Pass Protocols (ACME, EST, SCEP, CMP, OCSP, CRL, TLS, SSH CA, DNS providers, upstream renew) Two harnesses passed on rerun after environment failures.
    • Pass Let's Encrypt staging with Cloudflare DNS-01 and SSH install 152 s, no challenge record left.
    • Pass CAs and keys (hierarchy, offline root, ceremony, PKCS#11, escrow, KMS emulators, Vault) Two harnesses passed on rerun.
    • Pass Code signing (codesign, cosign OIDC)
    • Pass PQC with OpenSSL 3.5 (ML-DSA interop)
    • Partial PQC wave delivery Failed on the runner (Go 1.26), passed on the Mac (Go 1.27). Fixed by moving the build to Go 1.27.
    • Pass Discovery (network, SSH, agentless sudo, fingerprints, cloud inventory)
    • Fail Inventory sources Test rot: the Entrust fixture uses the old credential shape. Not fixed in this pass.
    • Pass Agents and install (agent, autopilot 29/29, SSH install, rollout, plugins, packages)
    • Pass Revocation bench
    • Fail Renewal storm, full profile 2000/2000 in both legs, 0 duplicates, but 8 and 18 CA limit violations and leg A took 375 s against a 360 s goal.
    • Pass Identity and governance (SSO, Keycloak, SPIFFE, approvals, MCP, LDAP, migration)
    • Pass Notifications and ops (mail, SIEM, ITSM, SLO, cloud delivery, egress short profile, Terraform) Egress ran the short profile (7/7), not the full 10 minutes.
    • Pass Kubernetes (cert-manager, Helm HA, mesh, Istio, promote and rollback, air-gap)
    • Pass Compose, SSE stream, HA chaos
    • Pass Smoke suite 446 of 446.

    Not proven

    • Windows domain harnesses (GPO autoenroll, WSTEP matrix, WinRM discovery, IIS, RDP and SQL install, agent MSI) were not run in this pass.
    • macOS keychain, vendor SCEP appliances, F5 and NetScaler, and real cloud key stores were not run.
    • make lint, go test and the UI suites were not part of this pass.
    • The storm profile misses its own zero-violation gate; the successor run on the key-usage branch had one acme failure and one pass.
  2. Release 1.1.1 signature check

    PassReal systems

    Release signatures · 2026-10-27 · 1.1.1 · public v1.1.1 f5e8011 · signature Verified OK; archive checksum OK

    Environment
    The maintainer's Apple silicon Mac on the public internet with cosign 3.1.3. The files came from the public GitHub release, into an empty directory. Nothing was built locally.
    What ran
    cosign verify-blob on checksums.txt with the signature and certificate from the release, pinned to this repository's release workflow and GitHub's Actions issuer, then a SHA-256 check of the macOS arm64 archive against checksums.txt.

    Output from the record, verbatim

    Commands and output, as run from docs/release/VERIFY_RELEASE_SIGNATURES_v1.1.1.md 11 lines
    $ cosign verify-blob \
        --certificate-identity-regexp '^https://github\.com/bpmforge/kryptkeeper/\.github/workflows/release\.yml@refs/tags/v' \
        --certificate-oidc-issuer https://token.actions.githubusercontent.com \
        --signature checksums.txt.sig \
        --certificate checksums.txt.pem \
        checksums.txt
    Flag --signature has been deprecated, please use --bundle to provide a signature
    Flag --certificate has been deprecated, please use --bundle with --trusted-root to provide the public certificate
    Verified OK
    $ shasum -a 256 -c checksums.txt --ignore-missing
    kryptkeeper_1.1.1_darwin_arm64.tar.gz: OK

    Not proven

    • The other archives, the deb and rpm packages, the SBOM files and the container image signatures were not checked in this run.
    • The release workflow's plugin catalogue job failed for lack of signing secrets; every other job, including its own signature check, passed.
    • The binary inside the archive was not unpacked or run.
  3. Release 1.1.0 signature check

    PassReal systems

    Release signatures · 2026-10-27 · 1.1.0 (6ca9f70) · public v1.1.0 6ca9f70 · signature Verified OK; archive checksum OK

    Environment
    The maintainer's Apple silicon Mac on the public internet with cosign 3.1.3. The files came from the public GitHub release, into an empty directory. Nothing was built locally.
    What ran
    cosign verify-blob on checksums.txt with the signature and certificate from the release, pinned to this repository's release workflow and GitHub's Actions issuer, then a SHA-256 check of the macOS arm64 archive against checksums.txt.

    Output from the record, verbatim

    Commands and output, as run from docs/release/VERIFY_RELEASE_SIGNATURES_v1.1.0.md 11 lines
    $ cosign verify-blob \
        --certificate-identity-regexp '^https://github\.com/bpmforge/kryptkeeper/\.github/workflows/release\.yml@refs/tags/v' \
        --certificate-oidc-issuer https://token.actions.githubusercontent.com \
        --signature checksums.txt.sig \
        --certificate checksums.txt.pem \
        checksums.txt
    Flag --signature has been deprecated, please use --bundle to provide a signature
    Flag --certificate has been deprecated, please use --bundle with --trusted-root to provide the public certificate
    Verified OK
    $ shasum -a 256 -c checksums.txt --ignore-missing
    kryptkeeper_1.1.0_darwin_arm64.tar.gz: OK

    Screenshots

    Terminal output, rendered from the verbatim text of the record. The commands and output are the same as the evidence panel above.
    Terminal output, rendered from the verbatim text of the record. The commands and output are the same as the evidence panel above. Captured 2026-10-27.
    The public GitHub release page for v1.1.0, tag v1.1.0, commit 6ca9f70.
    The public GitHub release page for v1.1.0, tag v1.1.0, commit 6ca9f70. Captured 2026-10-27.
    The release's asset list as GitHub serves it, unstyled, showing checksums.txt with its .pem certificate and .sig signature.
    The release's asset list as GitHub serves it, unstyled, showing checksums.txt with its .pem certificate and .sig signature. Captured 2026-10-27.
    The public GitHub Actions run of the Reproducible build workflow on the v1.1.0 tag: double rebuild and SBOM completeness both green.
    The public GitHub Actions run of the Reproducible build workflow on the v1.1.0 tag: double rebuild and SBOM completeness both green. Captured 2026-10-27.

    Not proven

    • The other archives, the deb and rpm packages, the SBOM files and the container image signatures were not checked in this run.
    • The Release workflow run on this tag has two failed jobs (Plugin catalogue and Helm chart OCI). The job that built and signed these files passed; its own signature-verify job was skipped.
    • The binary inside the archive was not unpacked or run.
  4. Release 1.0.0 final lab pass

    PassReal systems and stand-ins

    Full lab pass · 2026-10-18 · 1.0.0 (78494f48, 2878b6e5) · public v1.0.0 e0d82a4 · smoke 445/445; 86 Go packages ok

    Environment
    Linux runner (Ubuntu 24.04 amd64, podman 4.9, kind 0.30) in a VM. Windows test domain: Server 2025 DC with an Enterprise CA, Windows 11 25H2 and Server 2022 members.
    What ran
    Lint, unit, UI and smoke gates on main, the tests/live harness sweep, and the Windows autoenrollment, WinRM, IIS and agent checks.
    Note
    The pass found and fixed product bugs, among them Postgres pool exhaustion, a SQLite deadlock, and two WinRM argument-encoding bugs that broke agentless Windows discovery.

    Output from the record, verbatim

    Gate on main from docs/release/LIVE_RESULTS_v1.0.0.md

    | Check | Result |
    |---|---|
    | `make lint` | 0 issues |
    | `go test ./...` | 86 packages ok (93 incl. no-test packages), 0 failures |
    | Live-Postgres suite (`internal/bootstrap -run Postgres`) | ok |
    | UI `tsc -b`, `vite build`, vitest | pass; 1018 tests |
    | OpenAPI drift | none (238 routes / 293 paths) |
    | Smoke (`tests/smoke/run.sh`) | 445 / 445 |
    5 areas in this record: 5 pass
    • Pass GPO machine autoenrollment, Windows 11 and Server 2022 (RSA-3072)
    • Pass Certificate renewal with predecessor, template filter, clock skew refusal Both Windows versions.
    • Pass WinRM discovery, Server 2022 24 certificates found.
    • Pass IIS push, verify and rollback, Server 2022
    • Pass Windows agent MSI (enroll, discovery, no key egress, uninstall) Server 2022.

    Not proven

    • F5 BIG-IP and Citrix NetScaler (mock device only).
    • Real AWS KMS, Azure Key Vault and Google Cloud KMS (emulators only).
    • SCEP from a vendor appliance.
    • Windows autoenrollment on Windows 10 and Server 2019; Server 2025 as a member.
    • macOS Keychain was not rerun in this pass.
  5. Windows accepts the new key usages; Server 2025 member

    PassReal systems

    Windows · 2026-10-26 · main 99fe06db · 3 of 3 hosts autoenrolled (7 to 9 s)

    Environment
    Test Active Directory domain: Windows 11 25H2 (build 26200), Server 2022 (20348), and a new Server 2025 Standard member (26100). Server 2025 DC with an Enterprise CA. Hub built from main, run as a second process on a copy of the lab hub data.
    What ran
    GPO autoenrollment of an RSA-3072 machine certificate on all three hosts, then ECDSA TLS, time-stamping and OCSP-signer certificates by CSR, checked with certutil, IIS and Schannel.

    Output from the record, verbatim

    ECDSA, time-stamping and OCSP-signer leaves on the Windows client from docs/release/LIVE_RESULTS_WINDOWS_KU_WS2025_v1.1.md

    | Profile | Hub response | On the client (both OSes) |
    |---|---|---|
    | ECDSA TLS server (`KKWebEC`) | KU `[digitalSignature]`, EKU serverAuth | accepted into `LocalMachine\My` with the key; KU critical `Digital Signature (80)`, **no Key Encipherment**; `certutil -verify -urlfetch` success, CRL and OCSP verified |
    | Time stamping (`KKTimeStamp`) | KU digitalSignature | EKU `Time Stamping` **critical=True**; verify success |
    | OCSP signer (`KKOcspSigner`) | KU digitalSignature | EKU OCSP Signing plus `1.3.6.1.5.5.7.48.1.5` (`05 00`, ocsp-nocheck); verify success |

    TLS probe of the ECDSA leaf served by Schannel from docs/release/LIVE_RESULTS_WINDOWS_KU_WS2025_v1.1.md

    `Verification: OK`, TLS 1.3 `TLS_AES_256_GCM_SHA384`, peer signature `ecdsa_secp256r1_sha256`;

    Autoenrollment audit line from docs/release/LIVE_RESULTS_WINDOWS_KU_WS2025_v1.1.md

    Kerberos (SPNEGO) succeeded for each machine account; audit `xcep.policy success offered=[KKMachine(min_key=3072)]` then `wstep.enroll success`.

    Not proven

    • ECDSA or ML-DSA through GPO autoenrollment: the policy service is RSA-only today.
    • The Kerberos clock-skew refusal on Server 2025, and time-based renewal.
    • Windows 10 and Server 2019.
    • The lab hub itself still runs the older build; the new code ran beside it on a copy of its data.
    • Renewals were audited as enroll, not renew, on all three bumps. Open observation, cause not settled.
  6. Windows install modes: RDP, SQL Server, NLA discovery

    PassReal systems

    Windows · 2026-10-25 · feat/v1.1-windows (d5df926b, da51d63e) · 3 scenarios pass, 2 not run

    Environment
    Server 2022 and Windows 11 in a test Active Directory domain. Hub on loopback with a fresh data directory and a local test CA. SQL Server 2025 Express installed on the Server 2022 host.
    What ran
    Certificate install and rollback for RDP on both hosts and for SQL Server, hub discovery of an RDP listener that requires NLA, and a domain-account install over sealed NTLM. Each install was checked by an independent probe from a Mac.

    Output from the record, verbatim

    Verdict line from docs/release/LIVE_RESULTS_W39_WINDOWS_v1.1.md

    ## Verdict: PASS for W35-04 rdp (both hosts), W24-48 (hub discovery, NLA), W35-02 sqlserver. W24-50 and W35-03 not run.

    Scenarios that were not run from docs/release/LIVE_RESULTS_W39_WINDOWS_v1.1.md

    | 4. W24-50 Windows agent | NOT RUN: no agent installed (agent is discovery-only; see v1 doc) |
    | 5. W35-03 Exchange | NOT RUN: no Exchange server in the lab |

    Rollback result from docs/release/LIVE_RESULTS_W39_WINDOWS_v1.1.md

    Rollback (second renewal with `install.verify_host_port` pointed at `<ip>:5986`, which does not speak RDP): both hosts `install_outcome=verify_failed`,
    "no certificate served at the verify address", `verified=false`. The WMI hash stayed at the first installed leaf (DDD0F87A / 57503C13) and the Mac probe still served the first leaf's fingerprint. Audit: three

    Not proven

    • The Windows agent scenario was not run.
    • Exchange install mode was not run: there is no Exchange server in the lab.
    • The follow-up cleanup also removed the certificate SQL Server used; it was repaired by hand afterwards.
  7. Trust store push, hub to agent

    PassReal systems

    Trust stores · 2026-10-26 · 1.1.0 (66b2f9c0) · public v1.1.0 6ca9f70 · 49/49 on Postgres, 49/49 on SQLite

    Environment
    Apple silicon Mac, podman 6.1.3. Hub in a container with Postgres 16 or embedded SQLite. The real agent in Debian 12 and Rocky Linux 9 containers, unprivileged, with only the opt-in sudoers rule.
    What ran
    A trust bundle add and a removal, each proposed by one person and approved by another, applied by both agents to the OS trust store and reported back to the hub. Run once on Postgres and once on SQLite.

    Output from the record, verbatim

    Verdict and step table from docs/release/LIVE_RESULTS_TRUST_STORE_E2E_v1.1.md 19 lines
    ## Verdict: PASS, Postgres (49 of 49) and SQLite (49 of 49)
    
    | Step | Debian 12 | Rocky 9 | Hub |
    |---|---|---|---|
    | Hub boots, two agents enrol via `/agent/register`, hub lists both active | | | PASS |
    | Leaf untrusted before the push (`openssl verify`) | PASS | PASS | |
    | alice POST `/trust-bundles` (add) returns 202 and snapshots 2 targets | | | PASS |
    | alice approving her own proposal: 403 | | | PASS |
    | No bundle exists and no host changes while pending | PASS | PASS | PASS |
    | bob approves: 200, bundle created | | | PASS |
    | Agents apply; `openssl verify` against the system store accepts the leaf | PASS | PASS | |
    | `GET /trust-bundles/{id}`: both hosts `applied` | | | PASS |
    | Anchor `kryptkeeper-<sha256>.crt` in the distribution directory | PASS | PASS | |
    | bob proposes the removal: 202 | | | PASS |
    | Removal warning: none for the throwaway root; for the hub root "removing the root of hub CA ... distrusts 2 live managed certificate(s)" | | | PASS |
    | bob approving his own removal: 403; root still trusted while pending | PASS | PASS | PASS |
    | alice approves the removal: 200 | | | PASS |
    | Leaf no longer trusted after removal | PASS | PASS | |
    | Removal status `applied` on both hosts; anchor files removed | PASS | PASS | PASS |

    Screenshots

    The Trust stores panel on the Satellite Agents page, which lists each bundle with a result per host. This is the guide capture of the same screen from a test hub, not a frame recorded during the lab run.
    The Trust stores panel on the Satellite Agents page, which lists each bundle with a result per host. This is the guide capture of the same screen from a test hub, not a frame recorded during the lab run. Captured 2026-10-27.

    Not proven

    • Windows (certutil) and macOS (security) hosts.
    • Ubuntu, Alma, Fedora and SUSE.
    • Quorum rules, a third approver, multi-tenancy, and agents registered after the proposal.
    • Production polling timing: agents polled every 2 seconds here.
  8. ML-DSA delivery to Apache, and Kubernetes Secret rollback

    PassReal systems

    Post-quantum · 2026-10-26 · 1.1.0 (66b2f9c0) · public v1.1.0 6ca9f70

    Environment
    Apple silicon Mac, podman 6.1.3, kind 0.33. Apache httpd 2.4.69 with mod_ssl on OpenSSL 3.5.9 (Alpine; systemd and apachectl replaced by shims). A kind workload on OpenSSL 3.5.9 reading a TLS Secret.
    What ran
    A three-member wave. One Apache host took an ML-DSA-65 certificate over SSH. A second Apache host and the Kubernetes Secret were forced to fail verification and had to roll back to the old certificate.
    Note
    During a Kubernetes rollback the workload can serve the new certificate for up to one kubelet sync period before the old one returns.

    Output from the record, verbatim

    Two-principal execute and the hub's result from docs/release/LIVE_RESULTS_PQC_APACHE_K8S_ROLLBACK_v1.1.md

    === 6. two-principal execute ===
    PASS: alice proposes execute -> 201
    PASS: alice cannot approve her own proposal -> 403
    PASS: bob approves and the wave runs -> 200
    execute result: {'migrated': 1, 'failed': 2, 'skipped': 0, 'unsupported': 0, 'regressed': True, 'wave_status': 'regressed'}

    What the endpoints served after the wave, including the serial sequence and rollback from docs/release/LIVE_RESULTS_PQC_APACHE_K8S_ROLLBACK_v1.1.md

    === 8. what the endpoints actually serve, observed by openssl s_client in a container ===
    PASS: [ep-a] apache serial A768086FC1474ADA491688852C8D6DC -> 64392159802DFD6FC21814DBA2F307BA (the successor)
    PASS: [ep-a] served certificate is ML-DSA-65
    PASS: [ep-a] httpd -t accepts the delivered ML-DSA cert and key
    PASS: [ep-af] rolled back: old serial 8E8F69E34597360EB9B9A968A7B40CC2 still served
    PASS: [ep-af] cert and key files restored byte-for-byte
    [k8s] serials seen by the probe after the wave (in order): 49EF02A7DB0EE7625305F5523FDF6F28 155050286FBBD0F351C38C4EAD30A6CE 49EF02A7DB0EE7625305F5523FDF6F28
    PASS: [k8s] rolled back: workload stably serves the previous serial 49EF02A7DB0EE7625305F5523FDF6F28 again
    PASS: [k8s] workload serves a non-ML-DSA certificate after rollback ( Public Key Algorithm: id-ecPublicKey)
    PASS: [k8s] Secret tls.crt and tls.key restored byte-for-byte

    Summary from docs/release/LIVE_RESULTS_PQC_APACHE_K8S_ROLLBACK_v1.1.md

    === 10. summary ===
    ALL PASS

    Screenshots

    The migration wave screen after a wave is created and its execution proposed: it needs approval by a different administrator. This is the guide capture of the same screen from a test hub, not a frame recorded during the lab run.
    The migration wave screen after a wave is created and its execution proposed: it needs approval by a different administrator. This is the guide capture of the same screen from a test hub, not a frame recorded during the lab run. Captured 2026-10-27.

    Not proven

    • A verify failure caused by a real product fault, such as a workload that cannot parse an ML-DSA key.
    • Apache from Debian or RHEL packages, mod_md, or a proxy in front of httpd.
    • ingress-nginx; the workload stands in for an ingress.
    • Tomcat, Windows, F5 and Citrix are still refused for ML-DSA.
  9. ML-DSA delivery to HAProxy and Kubernetes

    PassReal systems

    Post-quantum · 2026-10-25 · a91c4f29 (working tree of the W40-01 branch)

    Environment
    Apple silicon Mac, podman 6.1.3, kind. HAProxy 3.2.26 on OpenSSL 3.5.9 (systemctl reload replaced by a signal shim). A kind workload on OpenSSL 3.5.9 reading a TLS Secret.
    What ran
    A three-member wave: HAProxy and Kubernetes took ML-DSA-65 certificates; a second HAProxy host was forced to fail verification and rolled back byte for byte.
    Note
    The first two attempts failed. The second showed a product defect: the wave approve response was cut by the server write timeout. It was fixed before the passing third run.

    Output from the record, verbatim

    What the endpoints served after the wave from docs/release/LIVE_RESULTS_PQC_W40_v1.1.md

    === 8. what the endpoints actually serve, observed by openssl s_client in a container ===
    PASS: [ep-h] haproxy serial BD6755ED3BB528EAA3D346DE794BA5D4 -> 31FC212FCA074286149265901AD0C80 (the successor)
    PASS: [ep-h] served certificate is ML-DSA-65
    PASS: [ep-h] haproxy -c accepts the delivered combined PEM
    PASS: [ep-hf] rolled back: old serial 3E006CE8824A9575A39D44A38EC6757E still served
    PASS: [ep-hf] combined PEM restored byte-for-byte
    PASS: [k8s] workload serial B6014D4BC0B411F01337545CA7A23AC4 -> CED104EFBBA1ACE13D0709C296F69880 (the successor)
    PASS: [k8s] served certificate is ML-DSA-65
    PASS: [k8s] Secret tls.key is a PKCS#8 PRIVATE KEY PEM

    Summary from docs/release/LIVE_RESULTS_PQC_W40_v1.1.md

    === 10. summary ===
    ALL PASS

    Screenshots

    The migration wave screen after a wave is created and its execution proposed: it needs approval by a different administrator. This is the guide capture of the same screen from a test hub, not a frame recorded during the lab run.
    The migration wave screen after a wave is created and its execution proposed: it needs approval by a different administrator. This is the guide capture of the same screen from a test hub, not a frame recorded during the lab run. Captured 2026-10-27.

    Not proven

    • Kubernetes Secret rollback after a failed verify was not run live in this record (see the Apache record).
    • ingress-nginx; the workload stands in for an ingress.
    • Tomcat, Windows, F5 and Citrix are refused for ML-DSA.
  10. CT monitoring via crt.sh and Cert Spotter

    PassReal systems

    Certificate Transparency · 2026-10-26 · 1.1.0 (66b2f9c0) · public v1.1.0 6ca9f70 · 6 findings, 6 notifications, first finding in 17 s

    Environment
    Maintainer's Apple silicon Mac on the public internet, no Cert Spotter key. Real crt.sh and Cert Spotter feeds for kryptkeeper.io.
    What ran
    The real CT monitor with a real SQLite store and a recording notification channel, scanning for certificates this hub did not issue.

    Output from the record, verbatim

    Live result block from the harness from docs/release/LIVE_RESULTS_CT_v1.1.md 12 lines
    === CT LIVE RESULT ===
    date: 2026-10-26T14:18:38Z
    domain: kryptkeeper.io
    sources: crtsh,certspotter
    elapsed_to_first_finding: 17s
    finding: id=crtsh:29986675827 issuer="C=US, O=Google Trust Services, CN=WE1" serial=0095ee5897fc803c8b13737ce2868d9eac
    new_findings: 6
    notifications: 6
    source_errors_seen: 2
    rescan_silent: true
    acknowledged_silent: true
    === END ===

    Screenshots

    crt.sh listing for kryptkeeper.io, the public log search the monitor reads. The first row, crt.sh ID 29986675827, is the finding id in the result block above.
    crt.sh listing for kryptkeeper.io, the public log search the monitor reads. The first row, crt.sh ID 29986675827, is the finding id in the result block above. Captured 2026-10-27.

    Not proven

    • The exclusion of certificates already in the inventory was not exercised.
    • The monitor reads crt.sh and Cert Spotter, not the RFC 6962 logs directly.
    • Delivery to a real channel such as Slack or email.
    • Behaviour on a large domain portfolio.
  11. Issue and revoke through Vault PKI (live), EJBCA REST (live) and AWS Private CA (emulator)

    PassReal systems and stand-ins

    Upstream CAs · 2026-10-25 · 132dc8e0 · 47 pass, 0 fail

    Environment
    Apple silicon Mac, podman 6.1.3. HashiCorp Vault 1.18.5, EJBCA Community 9.3.7 and the moto 5.0.28 AWS emulator, with a tamper proxy in front of Vault.
    What ran
    Issue by REST, ACME and EST, revoke through the hub, tampered upstream responses, and an upstream outage, against each CA.
    Note
    The first run found two product bugs in the external CA wiring. They were fixed without changing any verification check.

    Output from the record, verbatim

    Run result from docs/release/LIVE_RESULTS_UPSTREAM_CAS_v1.1.md

    Result: **47 PASS, 0 FAIL, 0 skipped** on the final run.
    Scenario A: HashiCorp Vault PKI steps from docs/release/LIVE_RESULTS_UPSTREAM_CAS_v1.1.md 14 lines
    | Step | Result | Evidence |
    |---|---|---|
    | Vault up, pki mount, internal root, role `uat` for `uat.example` and subdomains | PASS | root `CN=UAT Vault PKI Root` |
    | outbound credential `{token}`, `vault_pki` connector, `POST /connectors/{id}/test` | PASS | `200 {"status":"healthy"}` |
    | `POST /cas/external` with the Vault root as `trust_pem` | PASS | 202 (dual control), bob approves, alice redeems, 201 |
    | Issue by REST `POST /api/v1/certificates` | PASS | 201, serial `1F214A6D3F33D846B6A5AA315217CCF5CC52B131`, issuer `CN=UAT Vault PKI Root`, `openssl verify -CAfile vaultroot` OK |
    | Issue by ACME (lego, http-01, provisioner `vaultca` bound to the external CA) | PASS | serial `719ECBFB19522AEDECFB7037A31EAF328EDA4340`, issuer Vault root, verify OK |
    | Issue by EST simpleenroll (`KRYPTKEEPER_DEFAULT_CA_ID` = external CA) | PASS | 200, serial `524291F5056601F41072CE9C318B50A6164CC4D9`, issuer Vault root, verify OK |
    | Revoke through the hub | PASS | 200 `revocation_scope=upstream`; Vault `pki/cert/1f:21:...:b1:31` `revocation_time=1792893555` |
    | Tamper (a): proxy swaps the CSR for one with another key | PASS | refused, no certificate stored |
    | Tamper (b): proxy adds an extra SAN | PASS | refused; audit reason `DNS name "extra-san.uat.example" was not requested` |
    | Audit for both tampers | PASS | two `certificate.issue.upstream_rejected` rows with ca, connector, fingerprint and reason |
    | Vault stopped, revoke another certificate | PASS | 502 `upstream_revoke_failed`; certificate still has `revoked_at=None`, no scope |
    | Audit for the failed revoke | PASS | `certificate.revoke.upstream_failed` present |

    Screenshots

    The Save and test step of the Add issuer wizard for a Vault issuer. The healthy result of the real run is in the output above. This is the guide capture of the same screen from a test hub, not a frame recorded during the lab run.
    The Save and test step of the Add issuer wizard for a Vault issuer. The healthy result of the real run is in the output above. This is the guide capture of the same screen from a test hub, not a frame recorded during the lab run. Captured 2026-10-27.

    Not proven

    • AWS Private CA revocation: moto's revoke does nothing, so only the hub side is shown.
    • GCP CAS, Sectigo, GlobalSign, Entrust and DigiCert run on recorded vendor responses only.
    • EJBCA was run for REST issue and revoke only, not ACME, EST or the tamper proxy.
  12. Keycloak OIDC, SAML and SCIM

    PassReal systems

    Single sign-on · 2026-10-17 · 0.21 (2aba78fe) · 39/39

    Environment
    Apple silicon Mac, podman 6.1.3, Keycloak 26.0.8 in a container, hub on loopback.
    What ran
    OIDC and SAML logins with role mapping, replay and tamper refusals, an unmapped group, and SCIM create and deactivate.

    Output from the record, verbatim

    Result and check table from docs/release/LIVE_RESULTS_KEYCLOAK_v0.21.md 15 lines
    Exit code 0, wall time 2m39s, `=== RESULT: 39 passed, 0 failed ===`, wrapper counts OIDC role PASS lines 7, SAML role PASS lines 2.
    
    | Leg | Check | Result |
    |---|---|---|
    | OIDC | alice whoami role = pki-admin | PASS |
    | OIDC | bob whoami role = pki-operator | PASS |
    | OIDC | carol whoami role = auditor | PASS |
    | OIDC | dave (unmapped group) callback 403, no session | PASS |
    | OIDC | dave with `APP_AUTH_UNMAPPED_ROLE=read-only`: role = viewer | PASS |
    | OIDC | replayed callback URL 401; mutated state 401 | PASS |
    | SAML | alice via SAML role = pki-admin | PASS |
    | SAML | bob via SAML role = pki-operator | PASS |
    | SAML | replayed SAMLResponse 401; expired assertion 401; wrong audience 401; IdP-initiated control 302 | PASS |
    | SAML | dave (unmapped group) ACS 403, no session | PASS |
    | SCIM | create erin 201, OIDC login role = pki-operator, deactivate, old session 401 after 0s, fresh login 401 | PASS |

    Not proven

    • The full-harness negative test (breaking a role mapping) was not run.
    • Identity providers other than Keycloak.
  13. Notification plugins: Teams, PagerDuty, Opsgenie

    Mock passStand-ins only

    Plugins · 2026-10-25 · e882ae74 plus the W38-02 change

    Environment
    Go test run in one process on SQLite. The real wasm of each plugin spoke https to a local receiver that records requests and answers with a chosen status.
    What ran
    Delivery, request shape, a 500 and a 429 response, audit rows, and a check that no credential value leaks, for all three plugins.

    Output from the record, verbatim

    Verdict line from docs/release/LIVE_RESULTS_NOTIFY_PLUGINS_v1.1.md

    ## Verdict: PASS against local stand-in receivers; no real Teams, PagerDuty or Opsgenie tenant was used

    Results table from docs/release/LIVE_RESULTS_NOTIFY_PLUGINS_v1.1.md

    | Check | Teams | PagerDuty | Opsgenie |
    |---|---|---|---|
    | Delivery worker result `sent`, 1 attempt, stand-in saw exactly one request | PASS | PASS | PASS |
    | Request shape: Teams `type:message` + one adaptive-card attachment at the webhook path; PagerDuty `POST /v2/enqueue` `trigger` with `dedup_key` = event id and severity critical; Opsgenie `POST /v2/alerts` with `GenieKey` auth, alias = event id, priority P1 | PASS | PASS | PASS |
    | Stand-in answers 500: result `failed`, 2 attempts, error names the plugin | PASS | PASS | PASS |
    | Stand-in answers 429: result `failed` | PASS | PASS | PASS |
    | Audit: `delivery_plugin.notify` rows with `kind=notify`, version, sha256, both outcomes | PASS | PASS | PASS |
    | No credential value (routing key, API key, webhook URL signature) in the failure error or any audit row | PASS | PASS | PASS |

    Not proven

    • A real Teams flow, PagerDuty service or Opsgenie team.
    • Each vendor's live rate-limit behaviour.
  14. Ticket plugins: Zendesk, Freshservice

    Mock passStand-ins only

    Plugins · 2026-10-25 · 04cd67d7 plus the W38-09 change

    Environment
    Go test run in one process on SQLite. The real wasm of each plugin spoke https to a local service that keeps tickets in memory and can delete one behind the hub's back.
    What ran
    Test connection, create, reconcile, recreate after an external delete, resolve, an outage, a refused credential, and audit rows, for both plugins.

    Output from the record, verbatim

    Verdict line from docs/release/LIVE_RESULTS_TICKET_PLUGINS_v1.1.md

    ## Verdict: PASS against local stand-in services; no real Zendesk account or Freshservice domain was used
    Results table from docs/release/LIVE_RESULTS_TICKET_PLUGINS_v1.1.md 11 lines
    | Step | Zendesk | Freshservice |
    |---|---|---|
    | Test connection: read-only `ticket.exists` for key `0`, "no such ticket" is OK | PASS | PASS |
    | Create: event submitted, drained, row `created` with a key and an https link; the stand-in holds one open ticket | PASS | PASS |
    | Exists: reconcile of a live ticket changes nothing (not recreated) | PASS | PASS |
    | Externally deleted: reconcile recreates the ticket once, row flagged `recreated`, new key at the service | PASS | PASS |
    | Resolve: `ResolveByFingerprint` closes the recreated ticket (Zendesk `solved`, Freshservice status 4), row `resolved` | PASS | PASS |
    | Outage (stand-in answers 500): row stays `pending`, attempts 1, error classed temporary | PASS | PASS |
    | Refused credential (401): row `failed`, error classed permanent; test connection returns a permanent error | PASS | PASS |
    | Audit: native `itsm.ticket.create` success x2 and `itsm.ticket.resolve` x1; `delivery_plugin.ticket` rows carry kind, version and sha256 | PASS | PASS |
    | No credential value (API token, API key) in any row error, alert or audit summary | PASS | PASS |

    Not proven

    • A real Zendesk account or Freshservice domain: plan limits, required fields, soft-delete responses, rate limits.
    • A ticket created just before a hub crash is not adopted on retry. This is a design limit.
  15. DNS-01 plugins against local stand-ins

    Mock passStand-ins only

    Plugins · 2026-10-25 · feat/v1.1-integrations

    Environment
    Go test run in one process on SQLite, with the repo's own ACME server. Local TLS stand-ins shaped like each vendor's API.
    What ran
    A full ACME order through the DigitalOcean plugin, with failure and tenant-isolation cases, and the shared DNS-01 conformance suite against all four plugins.

    Output from the record, verbatim

    Verdict line from docs/release/LIVE_RESULTS_DNS_PLUGINS_v1.1.md

    ## Verdict: PASS against local stand-ins; no real DigitalOcean, GoDaddy, NS1 or Akamai account and no public CA was used

    Results of the full ACME order from docs/release/LIVE_RESULTS_DNS_PLUGINS_v1.1.md

    | Check | Result |
    |---|---|
    | The order issues a certificate through the plugin (create TXT, hub propagation check, CA validates, finalize) | PASS |
    | The stand-in saw a `POST` (create) and a `DELETE` (cleanup), every request carried `Authorization: Bearer <token>` and `Host: api.digitalocean.com` | PASS |
    | No challenge record is left at the vendor after the order | PASS |
    | Audit rows `delivery_plugin.dns01_present` and `delivery_plugin.dns01_cleanup`, outcome success, with version and module hash | PASS |
    | No audit row contains the vendor token | PASS |
    | Vendor answers 500: the order fails with an error naming the plugin, no token in it, no record left, a failure audit row | PASS |

    Not proven

    • A real account at any of the four vendors, including token scopes and record visibility delay.
    • A run through a public CA such as Let's Encrypt staging.
    • Only the DigitalOcean plugin ran a full order; the other three ran the conformance suite.
  16. Target credentials from Vault and OpenBao

    PassReal systems and stand-ins

    Credentials · 2026-10-25 · feat/v1.1-integrations · 58 pass lines, both flavors

    Environment
    Podman network with a real HashiCorp Vault 1.20 or OpenBao 2 server (TLS), a real sshd host and a hub container. CyberArk and Delinea plugins run against mocks.
    What ran
    A least-privilege token and an AppRole login drove a real SSH discovery scan. Version pinning, refusals, audit rows and a key-leak check were run on both flavors.

    Output from the record, verbatim

    Verdict line from docs/release/LIVE_RESULTS_CREDENTIAL_SOURCES_v1.1.md

    ## Verdict: PASS against real HashiCorp Vault 1.20 and real OpenBao 2 with a real sshd. CyberArk and Delinea were not available: those two plugins are proven against mocks only

    Command and pass count from docs/release/LIVE_RESULTS_CREDENTIAL_SOURCES_v1.1.md

    `bash tests/live/credential-sources/run.sh` (58 PASS lines, `ALL PASS`, both flavors).

    Not proven

    • CyberArk CCP and Delinea Secret Server: mocks only, no real service.
    • Windows (WinRM) credentials fetched this way.
    • PostgreSQL; everything ran on SQLite.
  17. 100,000-certificate benchmark

    PartialReal systems

    Performance · 2026-10-26 · 1.1 (base 66b2f9c0) · search p99 101 to 130 ms (bar under 500); peak RSS about 1.8 GiB (bar under 2 GiB)

    Environment
    Apple M4 Max, 64 GiB, macOS, Go 1.27.1, PostgreSQL 16 on the same machine. Hosted nightly runs on GitHub ubuntu-latest, 4 vCPU.
    What ran
    Search, list and issue load against a hub holding 100,000 certificates, before and after a refresh fix and a memory-limit change.
    Note
    The first run missed the search and memory bars. A refresh fix brought search p99 from about 970 ms to about 110 ms.

    Output from the record, verbatim

    Local check after the memory-limit and warm-up changes from docs/perf/rc4/03-measurements.md

    | Metric | Before (no GOMEMLIMIT, one pass) | GOMEMLIMIT only | After (both changes, run A) | After (run B) | Bar |
    |---|---:|---:|---:|---:|---|
    | peak RSS (MiB) | 2180.9 | 1646.2 | 1827.8 (+15k issued vs 5k) | 1810.0 | < 2048 |
    | search p99 (ms) | 126.0 | 104.6 | 101.0 | 130.0 | < 500 |
    | list p95 (ms) | 6.1 | 7.2 | 5.4 | 40.9 | < 300 |
    | issue rps c=16 | 1386.5 | 1532.8 | 1610.3 (median of 1610, 1690, 1433) | 730.3 (median of 730, 815, 508) | >= 500 |

    Not proven

    • The 1,000,000-certificate run is still pending.
    • The last hosted nightly (before the harness fix) failed the memory bar; the next nightly is the check.
    • Issue rate varies 508 to 1690 requests a second on a shared machine, so that bar rests on a median of three passes.
    • RSS margin is about 200 MiB; larger estates need a redesign that is not done.

Not proven yet

These features work against emulators, mock devices or stand-in services in our lab. Each is marked proven only after a run against the real system. The list comes from the roadmap.

Needs an account (11)

  • F5 BIG-IP and Citrix NetScaler certificate deliveryProven against a mock appliance only.
  • SCEP enrollment from F5 BIG-IPNot run against a real appliance.
  • Intune SCEP enrollmentRun against a stand-in for Entra and Intune; not run against a real tenant.
  • AWS KMS, Azure Key Vault and Google Cloud KMS as CA key storesProven against emulators; real permissions and HSM tiers are not proven.
  • Disaster-recovery drill with CA keys in a cloud KMSPasses with software and SoftHSM keys; the KMS half needs a cloud account.
  • Delivery to AWS, Azure and Google Cloud certificate servicesProven for HashiCorp Vault; the cloud legs need accounts.
  • ServiceNow, Jira Cloud, Zendesk and Freshservice connectorsProven against stand-in services only.
  • Slack, PagerDuty and Microsoft Teams alertsProven against local receivers only.
  • DNS-01 in Route 53, Azure DNS and Google Cloud DNSProven with local DNS and with Cloudflare against Let's Encrypt staging.
  • DNS-01 through the DigitalOcean, GoDaddy, NS1 and Akamai pluginsProven against local stand-ins; no real account or public CA.
  • Terraform deployment into a real AWS accountThe module validates; apply and destroy are not run.

Needs partner or hardware (8)

  • Disaster-recovery drill with CA keys in a vendor HSMPasses with software and SoftHSM keys; no vendor HSM in the lab.
  • YubiKey attestation for code-signing requestsThe refusal path is proven; the success path needs a real YubiKey.
  • ACME device attestation on real devicesProven with generated attestation chains only.
  • SCEP enrollment from a Palo Alto applianceNot run against a real appliance.
  • DigiCert inventory sync and upstream renewalProven with fixtures; the sandbox run needs a contract.
  • Target credentials from CyberArk and DelineaPass the conformance suite against mocks only.
  • Service fingerprinting on Active Directory, F5, Citrix and other appliancesTested against recorded fixtures only.
  • Issuance and revocation through Sectigo, GlobalSign, Entrust, DigiCert and Google CASBuilt against recorded API responses; Vault PKI and EJBCA REST have run live.

Later (1)

  • Reference deploymentsThese come from real customers, not from us.