lyng/proposals/lyngio_network_module.md

Proposal: Ktor-Backed Networking Modules For lyngio

Status: Draft
Date: 2026-04-02
Owner: lyngio

Context

lyngio currently provides optional modules for filesystem, process execution, and console access. Networking is still missing.

The first networking release should optimize for:

• broad multiplatform reach from day one
• a stable Lyng-native API surface
• small implementation cost by building on existing, well-tested KMP libraries
• explicit capability discovery, because not every transport exists on every platform

The proposed implementation base is:

• io.ktor:ktor-client-* for HTTP/HTTPS
• Ktor WebSockets client on top of the HTTP client for ws and wss
• io.ktor:ktor-network for TCP client/server sockets and UDP datagrams
• no ktor-server

The critical design constraint is that Lyng must not expose Ktor types directly. Ktor is an implementation dependency, not the public scripting model.

Goals

• Add first-release networking support with a practical breadth of features:
  • HTTP/HTTPS client
  • WebSocket client (ws / wss)
  • TCP client socket
  • TCP server socket where supported
  • UDP socket with a minimal datagram API where supported
• Keep the Lyng-facing API uniform across platforms.
• Provide capability discovery functions so scripts can branch safely.
• Keep declarations in .lyng files as the source of truth.
• Preserve room for later lower-level or non-universal socket APIs.

Non-goals (phase 1)

• exposing Ktor classes, channels, frames, engines, or plugins directly to Lyng
• HTTP server support
• TLS configuration beyond default secure client behavior
• fully general stream/channel APIs for every transport
• advanced socket tuning beyond a minimal useful set
• promising identical runtime support on every platform

Proposed module split

Use three Lyng modules instead of one monolith:

• lyng.io.http
• lyng.io.ws
• lyng.io.net

Rationale:

• HTTP and WebSocket are much more portable than raw sockets.
• TCP/UDP belong in a transport module, not in the HTTP client API.
• Each module can have its own capability checks while still sharing common address and security types underneath.
• The user-facing import structure stays clear and unsurprising.

Uniform capability model

Every module should expose explicit capability discovery.

Recommended shape:

• Http.isSupported(): Bool
• Ws.isSupported(): Bool
• Net.isSupported(): Bool
• Net.isTcpAvailable(): Bool
• Net.isTcpServerAvailable(): Bool
• Net.isUdpAvailable(): Bool

Optional richer form for later:

• Http.details(): HttpSupport
• Ws.details(): WsSupport
• Net.details(): NetSupport

For v1, booleans are enough.

Scripts should be able to write:

if (Net.isTcpServerAvailable()) {
    val server = Net.tcpListen(4040)
}

This is preferable to hard-failing during import or construction.

Design principles

1. Lyng-native API, Ktor-backed implementation

Public Lyng objects should be small and script-oriented.

Do not expose:

• HttpClient
• HttpResponse
• ByteReadChannel
• ByteWriteChannel
• Frame
• Ktor engine names
• Ktor pipeline/plugin concepts

Instead, expose a narrow stable surface implemented internally via Ktor.

2. Minimal but broad v1

The first release should support more transport kinds, but only with the smallest coherent API for each kind.

That means:

• HTTP: request/response, headers, text/body bytes, status, method
• WebSocket: connect, send text/binary, receive text/binary, close
• TCP: connect, accept, read, write, close
• UDP: bind, send datagram, receive datagram, close

Notably absent from v1:

• interceptors
• cookies/session stores beyond defaults
• multipart builders
• HTTP/2 tuning knobs
• ping/pong tuning for WebSockets
• socket option explosion

3. Uniform shapes where practical

Across transports, prefer the same ideas:

• isOpen()
• close()
• localAddress()
• remoteAddress() when meaningful
• read(...), write(...)
• receive(), send(...) for datagrams/messages
• isSupported() / capability methods on module singletons

This keeps discovery predictable without forcing false sameness where semantics differ.

Proposed Lyng modules

lyng.io.http

Purpose:

• high-level HTTP/HTTPS client backed by Ktor client

Recommended declaration sketch:

package lyng.io.http

extern class HttpHeaders : Map<String, String> {
    fun get(name: String): String?
    fun getAll(name: String): List<String>
    fun names(): List<String>
}

extern class HttpRequest {
    var method: String
    var url: String
    var headers: Map<String, String>
    var bodyText: String?
    var bodyBytes: Buffer?
    var timeoutMillis: Int?
}

extern class HttpResponse {
    val status: Int
    val statusText: String
    val headers: HttpHeaders
    fun text(): String
    fun bytes(): Buffer
}

extern object Http {
    fun isSupported(): Bool
    fun request(req: HttpRequest): HttpResponse
    fun get(url: String, headers...): HttpResponse
    fun post(url: String, bodyText: String = "", contentType: String? = null, headers...): HttpResponse
    fun postBytes(url: String, body: Buffer, contentType: String? = null, headers...): HttpResponse
}

Notes:

• HttpRequest is mutable because it is ergonomic in scripts.
• headers is intentionally simple in v1: Map<String, String> for programmatic request construction.
• HttpHeaders is primarily for response headers, where repeated names matter.
• text() and bytes() should cache decoded content inside the response object.
• For convenience methods, headers... should accept MapEntry values such as "X-Token" => "abc" and [key, value] pairs, then normalize them internally into request headers.

lyng.io.ws

Purpose:

• WebSocket client built on Ktor client WebSockets

Recommended declaration sketch:

package lyng.io.ws

extern class WsMessage {
    val isText: Bool
    val text: String?
    val data: Buffer?
}

extern class WsSession {
    fun isOpen(): Bool
    fun url(): String
    fun sendText(text: String): void
    fun sendBytes(data: Buffer): void
    fun receive(): WsMessage?
    fun close(code: Int = 1000, reason: String = ""): void
}

extern object Ws {
    fun isSupported(): Bool
    fun connect(url: String, headers...): WsSession
}

Notes:

• Keep frames hidden. Lyng sees messages.
• receive() returns null on clean close.
• v1 supports text and binary messages only.
• headers... should follow the same rules as Http.get(...): MapEntry values or [key, value] pairs.

lyng.io.net

Purpose:

• transport sockets backed by ktor-network

Recommended declaration sketch:

package lyng.io.net

enum IpVersion {
    IPV4,
    IPV6
}

extern class SocketAddress {
    val host: String
    val port: Int
    val ipVersion: IpVersion
    val resolved: Bool
    fun toString(): String
}

extern class Datagram {
    val data: Buffer
    val address: SocketAddress
}

extern class TcpSocket {
    fun isOpen(): Bool
    fun localAddress(): SocketAddress
    fun remoteAddress(): SocketAddress
    fun read(maxBytes: Int = 65536): Buffer?
    fun readLine(): String?
    fun write(data: Buffer): void
    fun writeUtf8(text: String): void
    fun flush(): void
    fun close(): void
}

extern class TcpServer {
    fun isOpen(): Bool
    fun localAddress(): SocketAddress
    fun accept(): TcpSocket
    fun close(): void
}

extern class UdpSocket {
    fun isOpen(): Bool
    fun localAddress(): SocketAddress
    fun receive(maxBytes: Int = 65536): Datagram?
    fun send(data: Buffer, host: String, port: Int): void
    fun close(): void
}

extern object Net {
    fun isSupported(): Bool
    fun isTcpAvailable(): Bool
    fun isTcpServerAvailable(): Bool
    fun isUdpAvailable(): Bool

    fun resolve(host: String, port: Int): List<SocketAddress>

    fun tcpConnect(
        host: String,
        port: Int,
        timeoutMillis: Int? = null,
        noDelay: Bool = true
    ): TcpSocket

    fun tcpListen(
        port: Int,
        host: String? = null,
        backlog: Int = 128,
        reuseAddress: Bool = true
    ): TcpServer

    fun udpBind(
        port: Int = 0,
        host: String? = null,
        reuseAddress: Bool = true
    ): UdpSocket
}

Notes:

• UDP is included in v1, but only with a minimal datagram API.
• udpBind should bind an ephemeral port by default.
• TCP server methods exist only where the backend supports them, but the declaration stays uniform.

Example usage

HTTP:

import lyng.io.http

if (!Http.isSupported())
    throw IllegalStateException("http is not supported here")

val r = Http.get(
    "https://example.com",
    "Accept" => "text/plain",
    "X-Trace" => "demo"
)
println(r.status)
println(r.text())

WebSocket:

import lyng.io.ws

if (Ws.isSupported()) {
    val ws = Ws.connect(
        "wss://echo.example/ws",
        "Authorization" => "Bearer <token>"
    )
    try {
        ws.sendText("hello")
        val msg = ws.receive()
        if (msg != null && msg.isText)
            println(msg.text)
    } finally {
        ws.close()
    }
}

TCP:

import lyng.io.net

if (Net.isTcpAvailable()) {
    val s = Net.tcpConnect("example.com", 80)
    try {
        s.writeUtf8("GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n")
        s.flush()
        while (true) {
            val line = s.readLine()
            if (line == null) break
            println(line)
        }
    } finally {
        s.close()
    }
}

TCP server:

import lyng.io.net

if (Net.isTcpServerAvailable()) {
    val server = Net.tcpListen(4040, host="127.0.0.1")
    println("listening on " + server.localAddress())

    try {
        while (server.isOpen()) {
            val client = server.accept()
            launch {
                try {
                    println("accepted " + client.remoteAddress())

                    val requestLine = client.readLine()
                    if (requestLine != null)
                        println("request: " + requestLine)

                    while (true) {
                        val line = client.readLine()
                        if (line == null || line == "") break
                    }

                    client.writeUtf8("hello from lyng server\n")
                    client.flush()
                } finally {
                    client.close()
                }
            }
        }
    } finally {
        server.close()
    }
}

UDP:

import lyng.io.net

if (Net.isUdpAvailable()) {
    val udp = Net.udpBind()
    try {
        udp.send(Buffer("ping"), "127.0.0.1", 4041)
        val d = udp.receive()
        if (d != null)
            println("from " + d.address + ": " + d.data)
    } finally {
        udp.close()
    }
}

Platform support model

The interface is uniform. Support is platform-dependent.

Initial intent:

Platform	HTTP	WS client	TCP client	TCP server	UDP
JVM	yes	yes	yes	yes	yes
Android	yes	yes	likely yes	likely yes	likely yes
Native Linux/macOS	yes	yes	target	target	target
Native Windows	yes	yes	target	target	target
NodeJS	yes	engine-dependent	target	target	target
Browser / Wasm	yes	engine-dependent	no or target-specific	no	no

Interpretation rules:

• import succeeds when the module is installed
• construction/operations may still fail if the specific feature is unavailable
• discovery methods must let scripts avoid that path cleanly
• isSupported() means the module has at least one useful implementation on the current runtime

Security model

Use separate policies per module to avoid muddy authorization.

Recommended policies:

• HttpAccessPolicy
• WsAccessPolicy
• NetAccessPolicy

Suggested operations:

sealed interface HttpAccessOp {
    data class Request(val method: String, val url: String) : HttpAccessOp
}

sealed interface WsAccessOp {
    data class Connect(val url: String) : WsAccessOp
    data class Send(val url: String, val bytes: Int, val isText: Boolean) : WsAccessOp
    data class Receive(val url: String) : WsAccessOp
}

sealed interface NetAccessOp {
    data class Resolve(val host: String, val port: Int) : NetAccessOp
    data class TcpConnect(val host: String, val port: Int) : NetAccessOp
    data class TcpListen(val host: String?, val port: Int, val backlog: Int) : NetAccessOp
    data class TcpAccept(val localHost: String?, val localPort: Int, val remoteHost: String, val remotePort: Int) : NetAccessOp
    data class TcpRead(val remoteHost: String, val remotePort: Int, val requestedBytes: Int) : NetAccessOp
    data class TcpWrite(val remoteHost: String, val remotePort: Int, val bytes: Int) : NetAccessOp
    data class UdpBind(val host: String?, val port: Int) : NetAccessOp
    data class UdpSend(val host: String, val port: Int, val bytes: Int) : NetAccessOp
    data class UdpReceive(val localHost: String?, val localPort: Int, val requestedBytes: Int) : NetAccessOp
}

This is intentionally strict. It lets embedders enforce:

• allow HTTP but forbid raw sockets
• allow outbound TCP but forbid listening sockets
• allow UDP only on loopback
• allow WebSockets only to whitelisted hosts

Error mapping

Like other lyngio modules, host exceptions should be mapped into Lyng runtime errors.

Recommended v1 mapping:

• policy denial -> illegal operation
• malformed URL / invalid host / invalid port -> illegal argument
• connect failure / bind failure / timeout / protocol failure -> illegal state

Later, if needed, add typed Lyng exceptions such as:

• HttpException
• WebSocketException
• NetworkException
• TimeoutException

Implementation outline

Recommended layout:

• lyngio/stdlib/lyng/io/http.lyng
• lyngio/stdlib/lyng/io/ws.lyng
• lyngio/stdlib/lyng/io/net.lyng
• lyngio/src/commonMain/kotlin/net/sergeych/lyng/io/http/LyngHttpModule.kt
• lyngio/src/commonMain/kotlin/net/sergeych/lyng/io/ws/LyngWsModule.kt
• lyngio/src/commonMain/kotlin/net/sergeych/lyng/io/net/LyngNetModule.kt
• lyngio/src/commonMain/kotlin/net/sergeych/lyngio/http/...
• lyngio/src/commonMain/kotlin/net/sergeych/lyngio/ws/...
• lyngio/src/commonMain/kotlin/net/sergeych/lyngio/net/...

Module installers should mirror existing lyngio patterns:

fun createHttpModule(policy: HttpAccessPolicy, scope: Scope): Boolean
fun createWsModule(policy: WsAccessPolicy, scope: Scope): Boolean
fun createNetModule(policy: NetAccessPolicy, scope: Scope): Boolean

As with lyng.io.console, the registrar should evaluate embedded .lyng declarations into module scope.

Why this is better than the earlier raw-sockets-first plan

• it delivers useful networking sooner
• it covers HTTP and WebSocket immediately, not as future work
• it reuses mature KMP networking code instead of creating a transport stack from scratch
• it still leaves room for a later low-level socket API if Ktor abstractions prove insufficient

Open questions

1. Should Http use static convenience methods only in v1, or also expose a reusable HttpClient-like Lyng object later?
2. Should Ws.connect(...) accept only URL plus headers, or also subprotocols in v1?
3. Should TcpSocket.readExact(...) be included from the start, or wait until there is a real use case?
4. Should UdpSocket.send(...) accept SocketAddress overloads in v1, or only host and port?
5. Should isSupported() return true when only a subset of the module works, or should module support be stricter?

Recommended implementation order

1. lyng.io.http
2. lyng.io.ws
3. lyng.io.net TCP client
4. lyng.io.net TCP server
5. lyng.io.net UDP

This order matches both practical usefulness and implementation risk.