148 lines
15 KiB
Markdown
148 lines
15 KiB
Markdown
# Experimental concealed protocol: implementation and continuation
|
|
|
|
The design is recorded in [CONCEALED-PROTOCOL-DESIGN.md](CONCEALED-PROTOCOL-DESIGN.md); exact byte layouts and cryptographic inputs are in [CONCEALED-PROTOCOL-WIRE.md](CONCEALED-PROTOCOL-WIRE.md). This implementation is experimental. The custom handshake/stream composition has not received an external cryptographic audit. Elligator uses variable-time bignum arithmetic on public coordinates in the current call graph; the [timing assessment](CONCEALED-PROTOCOL-TIMING.md) narrows the original warning and records measured variation without claiming a demonstrated private-key leak.
|
|
|
|
## Implemented components
|
|
|
|
- Common Kotlin Elligator 2 mapping and full-curve key generation, using the existing bignum/libsodium dependencies. Includes published reference vectors, subgroup and representation tests.
|
|
- Four-flight handshake: concealed HELO, protected signed server reply, session-encrypted client identity proof, session-encrypted acceptance. Exact transmitted representatives, both DH keys, options and identities are bound through the transcript. A fresh random 24-byte IV protects each bootstrap/finished envelope.
|
|
- Independently decryptable UDP records with random 24-byte IVs, encrypted sequence counters, authenticated padding and the existing replay bitmap.
|
|
- Continuous TCP ChaCha20 stream with preserved partial-block state, hidden bounded lengths and keyed BLAKE2b-128 record authentication. The existing binding does not expose portable XChaCha20 stream operations, so a derived 8-byte nonce is used with the original ChaCha20 64-bit block-counter API. This does not change UDP's 24-byte IVs.
|
|
- Complete-block integration with the existing Transport RPC engine. Ciphertext never passes through the legacy L0Call wrapper. One priority scheduler owns admission and sending. TCP authenticates/encrypts after queue selection; UDP protects before admission. Both reuse OutQueue's bounded-overtaking algorithm.
|
|
- Deferred readiness, application credential policy through the existing non-suspending KiloInterface hooks, client/server scope identities and tokens, duplex RPC, pushes, exception mapping, protected closure, finite client establishment retries and no automatic retry of application operations.
|
|
- Raw TCP, UDP and binary WebSocket carriers, plus bounded listener admission on sockets. The old TCP/UDP envelopes and keepalive bytes are absent from the new socket adapters.
|
|
|
|
The exact footprint and completed validation are recorded below. No publication or deployment is implied by these source changes.
|
|
|
|
## Using the experimental API
|
|
|
|
Initialize crypto as usual. The server's encryption key and identity key are separate and supplied by the application. Share one `ConcealedServerIdentity` between accepted connections to share its recent-DH guard and cryptographic concurrency budget. Key provisioning and rotation remain application policy.
|
|
|
|
```kotlin
|
|
@OptIn(ExperimentalConcealedProtocol::class)
|
|
suspend fun serve(
|
|
api: KiloInterface<Unit>,
|
|
encryptionKey: DecryptingSecretKey,
|
|
signingKey: SigningKey,
|
|
) = coroutineScope {
|
|
val identity = ConcealedServerIdentity(encryptionKey, signingKey)
|
|
val listener = ConcealedTcpListener(port = 7443)
|
|
try {
|
|
listener.carriers.collect { carrier ->
|
|
launch {
|
|
try {
|
|
ConcealedConnection.server(api, carrier, Unit, identity).run()
|
|
} catch (_: CancellationException) {
|
|
// Parent shutdown.
|
|
} catch (_: Exception) {
|
|
// Local logging/metrics policy; do not reply in plaintext.
|
|
} finally { carrier.close() }
|
|
}
|
|
}
|
|
} finally { listener.close() }
|
|
}
|
|
```
|
|
|
|
Client establishment retries use a new carrier and fresh ephemeral material on each attempt:
|
|
|
|
```kotlin
|
|
@OptIn(ExperimentalConcealedProtocol::class)
|
|
suspend fun connectExample(
|
|
api: KiloInterface<Unit>,
|
|
serverEncryption: EncryptingPublicKey,
|
|
trustedServerIdentity: VerifyingPublicKey,
|
|
clientIdentity: SigningKey,
|
|
) = coroutineScope {
|
|
val options = ConcealedOptions(
|
|
padding = ConcealedOptions.Padding(maxBytes = 255),
|
|
handshake = ConcealedOptions.Handshake(maxAttempts = 3),
|
|
)
|
|
val client = ConcealedClient(api, Unit, serverEncryption,
|
|
trustedServerIdentity, clientIdentity, options) {
|
|
connectConcealedTcpCarrier(NetworkAddress("server.example", 7443))
|
|
}
|
|
val runner = launch { client.run() }
|
|
try {
|
|
client.awaitReady()
|
|
// client.call(command, args), client.push(command, args)
|
|
} finally {
|
|
client.close()
|
|
runner.cancelAndJoin()
|
|
}
|
|
}
|
|
```
|
|
|
|
`ConcealedConnection.client(...)` is the lower-level single-attempt form. A `ConcealedConnection`/`ConcealedClient` instance is run only once. `ConcealedClient` retries establishment failures within its finite policy, stops on authenticated signature/policy failures, and does not reconnect/reissue an RPC after an established connection fails. The caller decides how business operations recover from an uncertain result.
|
|
|
|
Raw TCP/UDP APIs are available on the source sets configured for Ktor sockets (including JVM, Android, Linux and Apple targets). Common JS/Wasm code can use supported WebSocket or custom carriers; compilation alone does not establish browser runtime qualification.
|
|
|
|
For UDP use `ConcealedUdpListener` / `connectConcealedUdpCarrier`. One protected message fits one datagram. The default complete-datagram budget is 1200 bytes; oversize objects fail locally. This implementation does not introduce fragmentation or reliable delivery for ordinary RPCs.
|
|
|
|
For WebSocket use `connectConcealedWebSocketCarrier(url)` or a caller-owned Ktor WebSocket session's `asConcealedCarrier()`. The server selects the explicit route and installs Ktor WebSockets. Protocol bytes use binary frames; frame boundaries are just stream chunk boundaries. Caller-owned HTTP clients remain caller-owned. Padding obscures message-size relationships, while the HTTP/WebSocket handshake remains visible.
|
|
|
|
Existing `onConnected` hooks are ordinary, non-suspending functions. They may reject credentials or launch work that awaits RPC readiness. They must not block the event loop waiting for an RPC on the same not-yet-ready connection. No new suspending authorization callback or special pre-establishment RPC capability is introduced.
|
|
|
|
## Configuration and fixed properties
|
|
|
|
`ConcealedOptions` groups handshake, framing/padding, queue, UDP and closure limits. Socket listeners expose their own carrier-count, source-table and new-connection rate budgets; `ConcealedServerIdentity` exposes recent-DH capacity and crypto concurrency. Keep them associated with a listener, not rebuilt per packet.
|
|
|
|
Receive body size, datagram budget and padding ceilings are advertised inside authentication and reconciled with local bounds. Queue limits count admitted encoded data; stream protection adds bounded padding and authentication overhead when the selected record is written. At most one record is encrypted/written outside the queue at a time. A large record already writing is not preemptible.
|
|
|
|
Server HELO authentication failures now perform one discarded decryption with a mutated copy of a separate random throwaway key. This is enabled by default through `Handshake.decoyOnHelloAuthenticationFailure`; it remains inside the existing crypto admission limit, cannot authenticate a peer or send a response, and does no extra work for successful HELOs. It roughly doubled failed-HELO CPU time in the measured JVM run; see the [timing assessment](CONCEALED-PROTOCOL-TIMING.md#7-requested-server-side-decoy-decryption) for exact behavior, tests and limits.
|
|
|
|
The 16-byte authentication tag, identity checks, DH rejection, role separation, verified-before-decoding rule and fail-before-counter-wrap behavior are not disable switches. Per-direction limits additionally cap records at 2^32 and TCP encrypted bytes at 1 TiB; usage exhaustion requires a new session, not counter reset under an old key.
|
|
|
|
Low-level record state is externally serialized by the connection. Do not share a cipher/handshake instance between connections or restart it with reused keys/initialization material. Close/cancel the old run before abandoning its carrier.
|
|
|
|
## Build and dependency details
|
|
|
|
- The prototype consumes `crypto2:0.9.6-SNAPSHOT`, which contains the peer-DH length guard. This snapshot was compiled with Kotlin 2.4, so the project and its included interoperability fixtures now use Kotlin 2.4.0 as well.
|
|
- The existing `multiplatform-crypto-libsodium-bindings:0.9.6` backend is resolved from the YoungBlood Maven repository; keep that repository alongside SergeychWorks, Maven Central and Universa as shown in the main README. A standalone private-Maven-only consumer exposed the missing repository in its initial fixture; adding the existing backend repository resolved it.
|
|
- Direct use of the existing libsodium backend is explicit in the build. No additional crypto runtime library such as Monocypher was added.
|
|
- The existing libsodium binding eagerly captures `_sodium_init`. In JS runtime 0.7.16 that export appears only after readiness, so the build pins transitive `libsodium-sumo` to compatible 0.7.15 for Kotlin JS/Wasm. This is isolated from the separate lightweight TypeScript package. A proper initialization fix exists in the separate bindings repository and can be qualified later.
|
|
- A stable release must use a qualified stable crypto2 dependency and complete the qualification tasks. The first concealed distribution is 0.9.0-SNAPSHOT; see the [qualification record](CONCEALED-PROTOCOL-QUALIFICATION.md) for publication status and exact checks.
|
|
|
|
## Remaining qualification
|
|
|
|
Complete the independent protocol review, review the documented Elligator timing/dataflow assessment across supported targets, extend malformed-input fuzzing and sustained-load tests, and qualify platform runtimes according to actual evidence. Deterministic tests and randomness checks are valuable regression evidence, not proofs of cryptographic security.
|
|
|
|
The first implementation has no automatic key rotation, shared-port flavor detection, UDP fragmentation, address migration, mandatory heartbeat, cover traffic or timing shaping. These decisions are recorded in the design and are not silently supplied by the encryption layer.
|
|
|
|
## Verified run: 2026-10-22
|
|
|
|
- Full JVM suite: **109 passed**, including existing protocol regressions, raw TCP/UDP loopback, and real Netty WebSocket tests.
|
|
- Full Kotlin/JS Node suite: **82 passed**, with the compatible libsodium runtime resolution.
|
|
- Full Linux x64 Native suite: **85 passed**.
|
|
- Compilation passed for Android release, Wasm, Linux arm64, Windows x64, macOS arm64/x64, and iOS arm64/x64/simulator arm64. These compile checks do not qualify device runtimes or final Apple application linking.
|
|
- TypeScript unit suite: **38 passed**, one source-server interoperability test intentionally skipped by its ordinary unit command; the separate live source-JVM interoperability command passed pinned identity, duplex RPC, pushes, errors, parallel calls and reconnection.
|
|
- Tests cover independent Elligator/KDF/MAC vectors, all eight torsion components, alias encodings, signatures/transcripts, bad DH inputs, duplicate/stale records, lost UDP handshake flights, unauthenticated noise, negotiated limits, deadline trickling, deferred initialization, cancellation, finite retries, uncertain-operation behavior and explicit key cleanup with rejection of destroyed record state.
|
|
- Native testing exposed an existing empty-response hazard: the generic `utools.unpack` helper maps empty bytes to generic null. The transport's encrypted-response and complete-block network boundaries now use strict BiPack decoding, avoiding null-backed unsigned arrays. The legacy forged-ACK regression passes on all three tested runtimes.
|
|
- The unshrunk JVM JAR is 828,075 bytes versus the published 0.8.0 baseline's 537,671 bytes: **+290,404 bytes (about 284 KiB)**. This includes the prototype, raw carriers, compiler/dependency metadata changes and coroutine classes. It is not an APK, native executable or tree-shaken browser size measurement. No additional crypto runtime library was added.
|
|
|
|
Validation command (using JDK 21 and configured SDK/toolchains):
|
|
|
|
```sh
|
|
./gradlew '-Dorg.gradle.jvmargs=-Xmx2g -XX:MaxMetaspaceSize=1g' \
|
|
jvmTest jsNodeTest linuxX64Test compileKotlinWasmJs compileReleaseKotlinAndroid \
|
|
compileKotlinLinuxArm64 compileKotlinMingwX64 compileKotlinMacosArm64 \
|
|
compileKotlinMacosX64 compileKotlinIosArm64 compileKotlinIosX64 \
|
|
compileKotlinIosSimulatorArm64 jvmJar
|
|
```
|
|
|
|
The combined command completed successfully after the final lifecycle cleanup. A legacy WebSocket test was changed to await the actual disconnected state with a real-time deadline instead of using a virtual delay to wait for socket IO. New WebSocket test fixtures own their server coroutine scopes, observe peer closure, and have bounded coroutine/JUnit deadlines. An earlier failed run exposed that fixture cleanup issue; it was corrected before the successful run. Existing JS/Wasm lockfile policy may require `kotlinUpgradeYarnLock` when changing the Kotlin toolchain; the libsodium runtime is explicitly pinned by the build.
|
|
|
|
Remaining work before a production release: independent full-protocol review, target-specific Elligator timing/dataflow qualification and any hardening justified by those findings, extended fuzz/soak workloads and actual browser/mobile/Apple runtime tests. This paragraph records the initial implementation checkpoint; current source/distribution status is in the [snapshot qualification record](CONCEALED-PROTOCOL-QUALIFICATION.md#7-distribution-verification).
|
|
|
|
|
|
## Subsequent timing and failure-path validation: 2026-10-22
|
|
|
|
The [timing report](CONCEALED-PROTOCOL-TIMING.md) records opt-in JVM/Linux Native sampling, actual handshake-stage measurements, and the exact 50 ms grid plus whole-millisecond jitter experiment. The original timing warning was narrowed: the current bignum call graph operates on public coordinates, not the server's private scalar.
|
|
|
|
After adding the requested bounded decoy behavior, focused `HelloFailureNoiseTest` and `ConcealedConnectionTest` runs passed **11 tests each on JVM, Kotlin/JS and Linux Native**. Android release and Wasm compilation passed. A separate actual-path JVM timing probe passed with 64 samples per class. These are subsequent focused checks, separate from the earlier full-suite counts and JAR size recorded above. No artificial delay or random division loop was added.
|
|
|
|
|
|
The subsequent 0.9.0-SNAPSHOT release qualification passed 127 JVM tests (including extended campaigns), 90 Kotlin/JS tests and 94 Linux Native tests. [The qualification record](CONCEALED-PROTOCOL-QUALIFICATION.md) includes independent composition vectors, real socket payload captures, optimized Native performance, allocation data and publication verification.
|
|
|
|
|
|
Published 0.9.0-SNAPSHOT to Maven Local and private Maven; all 13 publications / 58 downloaded files passed SHA-256 verification. A standalone application consuming only private-Maven dependencies passed actual TCP duplex RPC. API documentation was deployed and verified. See [distribution verification](CONCEALED-PROTOCOL-QUALIFICATION.md#7-distribution-verification) for precise evidence and repository requirements.
|