Compare commits

..
8 Commits
70 changed files with 7573 additions and 232 deletions

No files matched your search

+1
View File
@@ -13,6 +13,7 @@
- Prefer defining Lyng entities (enums/classes/type shapes) in `.lyng` files; only define them in Kotlin when there is Kotlin/platform-specific implementation detail that cannot be expressed in Lyng.
- Avoid hardcoding Lyng API documentation in Kotlin registrars when it can be declared in `.lyng`; Kotlin-side docs should be fallback/bridge only.
- For mixed pluggable modules (Lyng + Kotlin), embed module `.lyng` sources as generated Kotlin string literals, evaluate them into module scope during registration, then attach Kotlin implementations/bindings.
- When a change adds or changes Lyng-visible runtime/module behavior, update the corresponding `.lyng` declaration in the same change, including declaration-level docs/comments for new API surface.
## Kotlin/Wasm generation guardrails
- Avoid creating suspend lambdas for compiler runtime statements. Prefer explicit `object : Statement()` with `override suspend fun execute(...)`.
+4
View File
@@ -92,6 +92,10 @@ Requires installing `lyngio` into the import manager from host code.
- `import lyng.io.http` (HTTP/HTTPS client API)
- `import lyng.io.ws` (WebSocket client API; currently supported on JVM, capability-gated elsewhere)
- `import lyng.io.net` (TCP/UDP transport API; currently supported on JVM, capability-gated elsewhere)
- Shared network value-type packages are also available when installed by host code:
- `import lyng.io.http.types` (`HttpHeaders`)
- `import lyng.io.ws.types` (`WsMessage`)
- `import lyng.io.net.types` (`IpVersion`, `SocketAddress`, `Datagram`)
## 7. AI Generation Tips
- Assume `lyng.stdlib` APIs exist in regular script contexts.
+167 -23
View File
@@ -1,9 +1,32 @@
# Json support
Since 1.0.5 we start adding JSON support. Versions 1,0,6* support serialization of the basic types, including lists and
maps, and simple classes. Multiple inheritance may produce incorrect results, it is work in progress.
Lyng now has two distinct JSON-facing layers:
## Serialization in Lyng
- plain JSON projection:
- `Obj.toJson()`
- `Obj.toJsonString()`
- canonical JSON round-trip format:
- `Json.encode(value)`
- `Json.decode(text)`
- typed canonical JSON round-trip format:
- `Json.encodeAs(Type, value)`
- `Json.decodeAs(Type, text)`
Use the first when you need ordinary JSON for interop.
Use the second when you need Lyng value round-trip semantics through JSON text with no schema.
Use the third when both sides already know the Lyng type and you want the same round-trip semantics with fewer type
tags in the JSON.
This distinction is intentional:
- plain JSON projection is optimized for compatibility with ordinary JSON tooling
- canonical `Json.encode()` is optimized for semantic fidelity to Lyng and Lynon and stays self-describing
- typed canonical `Json.encodeAs()` is optimized for the same fidelity when the schema is provided externally
- these goals conflict for values such as sets, exceptions, singleton objects, buffers, and maps with non-string keys
## Plain JSON projection in Lyng
// in lyng
assertEquals("{\"a\":1}", {a: 1}.toJsonString())
@@ -20,7 +43,8 @@ Simple classes serialization is supported:
assertEquals( "{\"foo\":1,\"bar\":2}", Point(1,2).toJsonString() )
>>> void
Note that mutable members are serialized by default. You can exclude any member (including constructor parameters) from JSON serialization using the `@Transient` attribute:
Note that mutable members are serialized by default. You can exclude any member (including constructor parameters) from
JSON serialization using the `@Transient` attribute:
import lyng.serialization
@@ -31,7 +55,7 @@ Note that mutable members are serialized by default. You can exclude any member
assertEquals( "{\"bar\":2,\"visible\":100}", Point2(1,2).toJsonString() )
>>> void
Note that if you override json serialization:
Note that if you override plain JSON serialization:
import lyng.serialization
@@ -46,8 +70,8 @@ Note that if you override json serialization:
assertEquals( "{\"custom\":true}", Point2(1,2).toJsonString() )
>>> void
Custom serialization of user classes is possible by overriding `toJsonObject` method. It must return an object which is
serializable to Json. Most often it is a map, but any object is accepted, that makes it very flexible:
Custom serialization of user classes is possible by overriding `toJsonObject`. It must return an object which is
serializable to JSON. Most often it is a map, but any object is accepted:
import lyng.serialization
@@ -70,12 +94,87 @@ serializable to Json. Most often it is a map, but any object is accepted, that m
Please note that `toJsonString` should be used to get serialized string representation of the object. Don't call
`toJsonObject` directly, it is not intended to be used outside the serialization library.
## Canonical Json round-trip format
`Json.encode()` and `Json.decode()` are now the JSON equivalents of `Lynon.encode()` and `Lynon.decode()`.
They still use JSON text, but they add Lyng-specific type tags where plain JSON would otherwise lose information.
When a map already fits ordinary JSON object rules, canonical JSON keeps that traditional object shape. In particular,
maps with string keys are still serialized as JSON objects, not as tagged entry lists.
Example:
```lyng
import lyng.serialization
import lyng.time
enum Color { Red, Green }
class Point(x,y) { var z = 42 }
val p = Point(1,2)
p.z = 99
val value = List(
p,
Map([1, "one"], ["two", 2]),
Set(1,2,3),
"hello".encodeUtf8(),
Date(2026,4,15),
Color.Green
)
assertEquals(value, Json.decode(Json.encode(value)))
```
The canonical `Json` format is intended for Lyng-to-Lyng transfer through JSON text.
The plain `toJson()` projection is intended for ordinary JSON interop.
Canonical `Json.encode()` should be read as the JSON analogue of `Lynon.encode()`: when Lynon already preserves a
Lyng distinction, canonical JSON tries to preserve it too, using tags only where ordinary JSON is insufficient.
## Typed canonical Json round-trip format
`Json.encodeAs(Type, value)` and `Json.decodeAs(Type, text)` use the same canonical rules, but with a declared target
type available during the whole traversal.
This changes one thing only: type tags may be omitted when the declared type is already exact enough to restore the
value unambiguously.
The same map rule still applies here: `Map<String, T>` stays a normal JSON object, while non-string-key maps fall back
to canonical entry encoding.
Example:
```lyng
import lyng.serialization
closed class Point(x: Int, y: Int)
closed class Segment(a: Point, b: Point)
val value = Segment(Point(0, 1), Point(2, 3))
val encoded = Json.encodeAs(Segment, value)
assertEquals("{\"a\":{\"x\":0,\"y\":1},\"b\":{\"x\":2,\"y\":3}}", encoded)
assertEquals(value, Json.decodeAs(Segment, encoded))
```
Subtype information is still preserved when the declared type is wider than the runtime one. For example, if a field is
declared as `Base` but contains `Derived`, canonical subtype tags remain in that field.
This is why the APIs are split:
- `toJson()` stays plain and interop-friendly
- `Json.encode()` stays fully self-describing and safe to decode without a schema
- `Json.encodeAs()` uses the supplied schema to reduce noise, but only where that schema is sufficient
## Kotlin side interfaces
The "Batteries included" principle is also applied to serialization.
- `Obj.toJson()` provides Kotlin `JsonElement`
- `Obj.toJsonString()` provides Json string representation
- `Obj.toJson()` provides Kotlin `JsonElement` for the plain JSON projection
- `Obj.toJsonString()` provides plain JSON string representation
- `Obj.decodeSerializableWith()` and `Obj.decodeSerializable()` allows to decode Lyng classes as Kotlin objects using
`kotlinx.serialization`:
@@ -104,10 +203,9 @@ suspend inline fun <reified T> Obj.decodeSerializable(scope: Scope = Scope()) =
decodeSerializableWith<T>(serializer<T>(), scope)
```
Note that lyng-2-kotlin deserialization with `kotlinx.serialization` uses JsonElement as information carrier without
formatting and parsing actual Json strings. This is why we use `Json.decodeFromJsonElement` instead of
`Json.decodeFromString`. Such an approach gives satisfactory performance without writing and supporting custom
`kotlinx.serialization` codecs.
Note that Lyng-to-Kotlin deserialization with `kotlinx.serialization` is based on the plain JSON projection,
not the canonical `Json.encode()` format. It uses `JsonElement` as the information carrier without formatting and
parsing actual JSON strings. This is why we use `Json.decodeFromJsonElement` instead of `Json.decodeFromString`.
### Pitfall: JSON objects and Map<String, Any?>
@@ -134,7 +232,8 @@ fun deserializeMapWithJsonTest() = runTest {
But what if your map has objects of different types? The approach of using polymorphism is partially applicable, but what to do with `{ one: 1, two: "two" }`?
The answer is pretty simple: use `JsonObject` in your deserializable object. This class is capable of holding any JSON types and structures and is sort of a silver bullet for such cases:
The answer is simple: use `JsonObject` in your deserializable object. This class is capable of holding any JSON types
and structures:
~~~kotlin
@Serializable
@@ -154,26 +253,71 @@ fun deserializeAnyMapWithJsonTest() = runTest {
~~~
# List of supported types
## Supported shapes
### Plain JSON projection
| Lyng type | JSON type | notes |
|-----------|-----------|-------------|
| `Int` | number | |
| `Real` | number | |
| `Real` | number | finite values only as plain numbers |
| `String` | string | |
| `Bool` | boolean | |
| `null` | null | |
| `Instant` | string | ISO8601 (1) |
| `List` | array | (2) |
| `Map` | object | (2) |
| `Map` | object | string keys only |
| simple class instance | object | constructor fields + mutable vars |
| enum | string | entry name |
### Canonical `Json.encode`
This format can also round-trip:
- maps with non-string keys
- sets
- immutable collections
- buffers and bit buffers
- class instances
- singleton objects
- enums
- exceptions
- `Date`, `Instant`, `DateTime`
- non-finite reals
- `void`
### Typed canonical `Json.encodeAs`
This format round-trips the same value space as canonical `Json.encode`, but it can emit simpler JSON for:
- closed classes and other exactly-known class fields
- enums when the enum type is known
- typed collections whose element types are known
- nested object graphs where declared field types are precise
It still falls back to canonical tagged encoding when exact runtime type information would otherwise be lost.
It does so by adding Lyng-specific type tags only when necessary.
## Kotlin-side extension point for more formats
Additional formats can be exported from Kotlin modules by subclassing `ObjSerializationFormatClass` and registering the
format in module scope with `bindSerializationFormat(...)`.
```kotlin
module.bindSerializationFormat(
object : ObjSerializationFormatClass("MyFormat") {
override suspend fun encodeValue(scope: Scope, value: Obj): Obj = ...
override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj = ...
}
)
```
This makes `MyFormat.encode(...)` and `MyFormat.decode(...)` available from Lyng after importing the module.
(1)
: ISO8601 flavor 1970-05-06T06:00:00.000Z in used; number of fractional digits depends on the truncation
on [Instant](time.md), see `Instant.truncateTo...` functions.
: ISO8601 flavor `1970-05-06T06:00:00.000Z` is used; number of fractional digits depends on truncation on
`Instant`, see `Instant.truncateTo...` functions.
(2)
: List may contain any objects serializable to Json.
(3)
: Map keys must be strings, map values may be any objects serializable to Json.
: Lists may contain any values serializable by the selected JSON layer.
+178 -14
View File
@@ -1,4 +1,4 @@
### lyng.io.db — SQL database access for Lyng scripts
# lyng.io.db — SQL database access for Lyng scripts
This module provides the portable SQL database contract for Lyng. The current shipped providers are SQLite via `lyng.io.db.sqlite` and a JVM-only JDBC bridge via `lyng.io.db.jdbc`.
@@ -6,7 +6,7 @@ This module provides the portable SQL database contract for Lyng. The current sh
---
#### Install the module into a Lyng session
## Install the module into a Lyng session
For SQLite-backed database access, install both the generic DB module and the SQLite provider:
@@ -54,7 +54,7 @@ suspend fun bootstrapJdbc() {
---
#### Using from Lyng scripts
## Using from Lyng scripts
Typed SQLite open helper:
@@ -188,34 +188,98 @@ assertThrows(RollbackException) {
---
#### Portable API
## Runnable serialization sample
##### `Database`
A complete runnable example is in [examples/sqlite_serialization.lyng](../examples/sqlite_serialization.lyng).
It uses:
- `@DbJson`
- `@DbLynon`
- `@DbExcept`
- `@cols(...)`, `@vals(...)`, `@set(...)`
- `decodeAs<T>()`
The current direct read form that works under `jlyng` is:
```lyng
tx.select("select * from item where id = ?", 1).decodeAs<Item>().first
```
If we want a shorter form such as:
```lyng
tx.selectAllAs<Item>("item where id = ?", 1).first
```
it should be added as a built-in `SqlTransaction` API. A pure Lyng generic wrapper around `decodeAs<T>()` does not currently preserve `T` reliably enough under `jlyng`.
---
## Portable API
### `Database`
- `transaction(block)` — opens a transaction, commits on normal exit, rolls back on uncaught failure.
##### `SqlTransaction`
### `SqlTransaction`
- `select(clause, params...)` — execute a statement whose primary result is a row set.
- `execute(clause, params...)` — execute a side-effect statement and return `ExecutionResult`.
- `transaction(block)` — nested transaction with real savepoint semantics.
##### `ResultSet`
`select(...)` and `execute(...)` also support SQL object-expansion macros for declaration-driven writes:
- `@cols(?1)` — expand object argument `?1` to a comma-separated column list
- `@vals(?1)` — expand object argument `?1` to matching placeholders and bind values
- `@set(?1)` — expand object argument `?1` to `column = ?` pairs and bind values
Each macro also supports an optional clause-local exclusion list:
```lyng
tx.execute("update item set @set(?1 except: \"id\", \"createdAt\") where id = ?2", item, item.id)
```
Example:
```lyng
tx.execute("insert into item(@cols(?1)) values(@vals(?1))", item)
tx.execute("update item set @set(?1) where id = ?2", item, item.id)
```
When a clause uses any of these macros, non-expanded scalar parameters in the same SQL string must use explicit indexed placeholders such as `?2`, `?3`, and so on.
### `ResultSet`
- `columns` — positional `SqlColumn` metadata, available before iteration.
- `size()` — result row count.
- `isEmpty()` — fast emptiness check where possible.
- `iterator()` — normal row iteration while the transaction is active.
- `toList()` — materialize detached `SqlRow` snapshots that may be used after the transaction ends.
- `decodeAs<T>()` — transaction-scoped iterable view that decodes each row into `T`.
##### `SqlRow`
### `SqlRow`
- `row[index]` — zero-based positional access.
- `row["columnName"]` — case-insensitive lookup by output column label.
- `row.decodeAs<T>()` — decode one row into a typed Lyng value.
Name-based access fails with `SqlUsageException` if the name is missing or ambiguous.
##### `ExecutionResult`
### `DbFieldAdapter`
Custom DB field projection hook used by `@DbDecodeWith(...)` and `@DbSerializeWith(...)`.
- `decode(rawValue, column, row, targetType)` — adapt one raw DB field value to a Lyng value for the requested target type.
- `encode(value, targetType)` — adapt one Lyng value to a direct DB-bindable value for SQL object expansion.
Use `@DbDecodeWith(adapter)` on class constructor parameters and class-body fields/properties that participate in `decodeAs<T>()`.
Use `@DbSerializeWith(adapter)` on constructor parameters and class-body fields/properties that participate in `@cols(...)`, `@vals(...)`, and `@set(...)` object expansion.
Annotation arguments are evaluated once when the declaration is created, and the resulting adapter instance is retained in declaration metadata.
### `ExecutionResult`
- `affectedRowsCount`
- `getGeneratedKeys()`
@@ -224,7 +288,7 @@ Statements that return rows directly, such as `... returning ...`, should use `s
---
#### Value mapping
## Value mapping
Portable bind values:
@@ -237,6 +301,88 @@ Portable bind values:
Unsupported parameter values fail with `SqlUsageException`.
SQL object-expansion write rules:
- constructor parameters participate in projection by declaration order
- matching serializable class-body fields/properties also participate
- `@Transient` fields are excluded automatically
- `@DbExcept` fields are excluded automatically
- `except:` excludes additional fields for one specific macro use
- direct DB-bindable values are written as-is
- `@DbJson` fields are encoded as canonical JSON text
- `@DbLynon` fields are encoded as Lynon binary
- `@DbSerializeWith(adapter)` fields are encoded through the adapter
- unannotated non-bindable object fields fail with `SqlUsageException`
Write-side encoding is intentionally explicit. The runtime does not try to infer target DB column types from SQL text or backend metadata during statement preparation.
Example:
```lyng
import lyng.io.db
import lyng.io.db.sqlite
class Payload(name: String, count: Int)
object TrimAdapter: DbFieldAdapter {
override fun encode(value, targetType) =
when(value) {
null -> null
else -> value.toString().trim()
}
}
class Item(
id: Int,
@DbSerializeWith(TrimAdapter) title: String,
@DbJson meta: Payload,
@DbLynon state: Payload
) {
var note: String = ""
@DbExcept var cache: String = ""
}
val db = openSqlite(":memory:")
val restored = db.transaction { tx ->
tx.execute(
"create table item(id integer not null, title text not null, meta text not null, state blob not null, note text not null)"
)
val item = Item(1, " first ", Payload("json", 10), Payload("bin", 20))
item.note = "created"
item.cache = "not stored"
tx.execute("insert into item(@cols(?1)) values(@vals(?1))", item)
item.title = " second "
item.meta = Payload("json2", 11)
item.state = Payload("bin2", 21)
item.note = "updated"
tx.execute(
"update item set @set(?1 except: \"id\") where id = ?2",
item,
item.id
)
tx.select("select id, title, meta, state, note from item").decodeAs<Item>().first
}
assertEquals("second", restored.title)
assertEquals("json2", restored.meta.name)
assertEquals(21, restored.state.count)
assertEquals("updated", restored.note)
```
This example shows:
- `@DbSerializeWith(...)` trimming a string before write
- `@DbJson` storing structured data in a text column
- `@DbLynon` storing structured data in a binary column
- `@DbExcept` excluding a field from automatic projection
- `@set(... except: "id")` skipping one field for an update clause
- `decodeAs<Item>()` reconstructing the object on read
Portable result metadata categories:
- `Binary`
@@ -249,11 +395,27 @@ Portable result metadata categories:
- `DateTime`
- `Instant`
Typed row decode rules:
- object/class targets map constructor parameters by column label, case-insensitively
- remaining matching serializable mutable fields are assigned after constructor call
- `@DbDecodeWith(adapter)` on a constructor parameter or class-body field/property takes precedence over built-in JSON/Lynon decoding
- `@DbDecodeWith(adapter)` must receive exactly one adapter instance implementing `DbFieldAdapter`
- adapter output must match the target member type or decoding fails with `SqlUsageException`
- missing required non-null constructor fields fail
- defaulted or nullable constructor fields may be omitted from the result
- extra result columns currently fail in strict mode
- if a row has exactly one column, that value may be decoded directly as the requested target type
- JSON-like native column types (`json`, `jsonb`) are decoded through typed canonical `Json` when the target type is not `String`
- binary columns are decoded through `Lynon` when the target type is not `Buffer`
- `Buffer` targets keep the raw binary payload without Lynon decoding
- plain text columns are not implicitly treated as JSON
For temporal types, see [time functions](time.md).
---
#### SQLite provider
## SQLite provider
`lyng.io.db.sqlite` currently provides the first concrete backend.
@@ -297,7 +459,7 @@ Open-time validation failures:
- malformed URL or bad option shape -> `IllegalArgumentException`
- runtime open failure -> `DatabaseException`
#### JDBC provider
## JDBC provider
`lyng.io.db.jdbc` is currently implemented on the JVM target only. The `lyngio-jvm` artifact bundles and explicitly loads these JDBC drivers:
@@ -369,7 +531,7 @@ PostgreSQL-specific notes:
---
#### Lifetime rules
## Lifetime rules
`ResultSet` is valid only while its owning transaction is active.
@@ -387,12 +549,14 @@ This means:
- do not keep `ResultSet` objects after the transaction block returns
- materialize rows with `toList()` inside the transaction when they must outlive it
- the iterable returned by `decodeAs<T>()` is also transaction-scoped
- decoded objects produced while iterating `decodeAs<T>()` are detached ordinary Lyng values
The same rule applies to generated keys from `ExecutionResult.getGeneratedKeys()`: the `ResultSet` is transaction-scoped, but rows returned by `toList()` are detached.
---
#### Platform support
## Platform support
- `lyng.io.db` — generic contract, available when host code installs it
- `lyng.io.db.sqlite` — implemented on JVM and Linux Native in the current release tree
+2
View File
@@ -3,6 +3,8 @@
This module provides a compact HTTP client API for Lyng scripts. It is implemented in `lyngio` and backed by Ktor on supported runtimes.
> **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes.
>
> **Shared type note:** `HttpHeaders` is also available from `lyng.io.http.types` when host code wants the reusable value type without relying on the HTTP client module itself.
---
+2
View File
@@ -4,6 +4,8 @@ This module provides minimal raw transport networking for Lyng scripts. It is im
> **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes.
>
> **Shared type note:** `IpVersion`, `SocketAddress`, and `Datagram` are also available from `lyng.io.net.types` when host code wants reusable transport value types without depending on the `Net` capability object itself.
>
> **Important native platform limit:** current native TCP/UDP support is backed by a selector with a per-process file descriptor ceiling. On Linux/macOS native targets this makes high-connection-count servers and same-process load tests unsuitable once the process approaches that limit.
>
> **Recommendation:** for serious HTTP/TCP servers, prefer the JVM target today. On native targets, keep concurrency bounded, batch local load tests in waves, and use multiple worker processes behind a reverse proxy if you need more throughput before the backend is reworked.
+2
View File
@@ -3,6 +3,8 @@
This module provides a compact WebSocket client API for Lyng scripts. It is implemented in `lyngio` and currently backed by Ktor WebSockets on the JVM.
> **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes.
>
> **Shared type note:** `WsMessage` is also available from `lyng.io.ws.types` when host code wants the reusable message type without depending on the WebSocket client module itself.
---
+1
View File
@@ -19,6 +19,7 @@
- **[lyng.io.http](lyng.io.http.md):** HTTP/HTTPS client access. Provides `Http`, `HttpRequest`, `HttpResponse`, and `HttpHeaders`.
- **[lyng.io.ws](lyng.io.ws.md):** WebSocket client access. Provides `Ws`, `WsSession`, and `WsMessage`.
- **[lyng.io.net](lyng.io.net.md):** Transport networking. Provides `Net`, `TcpSocket`, `TcpServer`, `UdpSocket`, and `SocketAddress`.
- **Shared networking type packages:** `lyng.io.http.types`, `lyng.io.ws.types`, and `lyng.io.net.types` expose reusable value types such as `HttpHeaders`, `WsMessage`, `IpVersion`, `SocketAddress`, and `Datagram` when host code wants type-only imports without installing the corresponding capability object module.
---
+1 -1
View File
@@ -4,7 +4,7 @@ Saved on April 4, 2026 before the `List<Int>` indexed-access follow-up fix.
Benchmark target:
- [examples/pi-bench.py](/home/sergeych/dev/lyng/examples/pi-bench.py)
- [examples/pi-bench.lyng](/home/sergeych/dev/lyng/examples/pi-bench.lyng)
- [examples/pi-bench.lyng](../examples/pi-bench.lyng)
Execution path:
- Python: `python3 examples/pi-bench.py`
+162 -6
View File
@@ -1,6 +1,40 @@
# Lyng serialization
Lyng has builting binary bit-effective serialization format, called Lynon for LYng Object Notation. It is typed, binary, implements caching, automatic compression, variable-length ints, one-bit Booleans an many nice features.
Lyng has a built-in serialization module, `lyng.serialization`.
There are now two built-in formats with different goals:
- `Lynon`: the canonical binary format for Lyng values.
- `Json`: the canonical JSON-based round-trip format for Lyng values.
In addition, `Obj.toJson()` / `toJsonString()` remain available as a plain JSON projection for interoperability with
regular JSON tools and Kotlin `kotlinx.serialization`.
## Canonical formats
`Lynon` and `Json` are both exposed as format objects with the same surface:
- `Format.encode(value)`
- `Format.decode(encodedValue)`
For the built-in formats:
- `Lynon.encode(x)` returns `BitBuffer`
- `Lynon.decode(bitBuffer)` returns the original Lyng value
- `Json.encode(x)` returns `String`
- `Json.decode(jsonString)` returns the original Lyng value
`Json` also provides a typed canonical mode:
- `Json.encodeAs(Type, value)` returns `String`
- `Json.decodeAs(Type, jsonString)` returns the original Lyng value of the specified type
This is still canonical JSON, but it is schema-driven instead of fully self-describing.
## Lynon
Lynon is LYng Object Notation. It is typed, binary, bit-effective, implements caching, automatic compression,
variable-length integers, one-bit booleans, and preserves Lyng runtime structure well.
It is as simple as:
@@ -20,7 +54,8 @@ It is as simple as:
assert( text.length > (encodedBits.toBuffer() as Buffer).size )
>>> void
Any class you create is serializable by default; lynon serializes first constructor fields, then any `var` member fields.
Any class you create is serializable by default; Lynon serializes first constructor fields, then any `var` member
fields.
## Transient Fields
@@ -40,7 +75,7 @@ class MyData(@Transient val tempSecret, val publicData) {
Transient fields:
- Are **omitted** from Lynon binary streams.
- Are **omitted** from JSON output (via `toJson`).
- Are **omitted** from JSON output (`toJson`) and canonical `Json.encode(...)`.
- Are **ignored** during structural equality checks (`==`).
- If a transient constructor parameter has a **default value**, it will be restored to that default value during deserialization. Otherwise, it will be `null`.
- Class body fields marked as `@Transient` will keep their initial values (or values assigned in `init`) after deserialization.
@@ -49,8 +84,131 @@ Transient fields:
- **Singleton Objects**: `object` declarations are serializable by name. Their state (mutable fields) is also serialized and restored, respecting `@Transient`.
- **Classes**: Class objects themselves can be serialized. They are serialized by their full qualified name. When converted to JSON, a class object includes its public static fields (excluding those marked `@Transient`).
- **Exceptions**: canonical formats preserve exception class, message, extra data, and captured stack trace.
## Custom Serialization
## Plain JSON projection vs canonical Json format
There are two JSON-related APIs and they serve different purposes:
- `Obj.toJson()` / `toJsonString()`
- produce ordinary JSON values
- best for interop with external JSON systems
- best for `Obj.decodeSerializable()` / `decodeSerializableWith()`
- may be lossy for Lyng-specific structures
- `Json.encode()` / `Json.decode()`
- produce JSON text too
- use Lyng-specific type tags so the payload is self-describing
- intended for round-tripping Lyng values
- intended to match Lynon semantics where JSON can carry them
- still keep ordinary string-key maps in traditional JSON object form
- can preserve values that plain JSON cannot represent directly, such as:
- maps with non-string keys
- sets
- buffers and bit buffers
- class instances
- singleton objects
- enums
- exceptions
- date/time objects
- non-finite reals
- `void`
- `Json.encodeAs(Type, value)` / `Json.decodeAs(Type, text)`
- also round-trip Lyng values through JSON text
- use the declared or requested type as decoding schema
- recursively omit type tags when the declared type is already exact enough
- keep canonical tags when the runtime value is more specific than the declared type
- produce less noisy JSON for closed and otherwise precisely-typed object graphs
- still keep ordinary `Map<String, ...>` values in traditional JSON object form
Why this split exists:
- plain `toJson()` must remain ordinary JSON so it stays convenient for external JSON systems and Kotlin
`kotlinx.serialization`
- canonical `Json.encode()` is for Lyng-to-Lyng transport through JSON text without any external schema, so it must
remain self-describing and preserve Lyng runtime
distinctions whenever possible
- `Json.encodeAs()` exists for the cases where a schema is known on both sides and we want canonical round-trip
behavior with fewer tags
- one API cannot optimize for both goals at once: either you get too many Lyng tags for ordinary JSON interop, or you
get lossy round-trips
Example:
import lyng.serialization
import lyng.time
enum Color { Red, Green }
class Point(x,y) { var z = 42 }
val p = Point(1,2)
p.z = 99
val x = List(
p,
Map([1, "one"], ["two", 2]),
Set(1,2,3),
"hello".encodeUtf8(),
Date(2026,4,15),
Color.Green
)
assertEquals(x, Json.decode(Json.encode(x)))
>>> void
Typed canonical example:
import lyng.serialization
closed class Point(x: Int, y: Int)
closed class Segment(a: Point, b: Point)
val value = Segment(Point(0,1), Point(2,3))
val json = Json.encodeAs(Segment, value)
assertEquals("{\"a\":{\"x\":0,\"y\":1},\"b\":{\"x\":2,\"y\":3}}", json)
assertEquals(value, Json.decodeAs(Segment, json))
>>> void
## Adding more formats from Kotlin modules
External modules can add new formats on the Kotlin side.
The common base class is:
```kotlin
abstract class ObjSerializationFormatClass(className: String) : ObjClass(className) {
abstract suspend fun encodeValue(scope: Scope, value: Obj): Obj
abstract suspend fun decodeValue(scope: Scope, encoded: Obj): Obj
}
```
To export a new format from a module:
```kotlin
im.addPackage("test.formats") { module ->
module.bindSerializationFormat(
object : ObjSerializationFormatClass("Reverse") {
override suspend fun encodeValue(scope: Scope, value: Obj): Obj =
ObjString(value.toString(scope).value.reversed())
override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj =
ObjString((encoded as ObjString).value.reversed())
}
)
}
```
Then from Lyng, after importing the Kotlin module above, usage looks like:
```lyng
import test.formats
assertEquals("cba", Reverse.encode("abc"))
assertEquals("abc", Reverse.decode("cba"))
```
## Notes
Important is to understand that normally `Lynon.decode` wants [BitBuffer], as `Lynon.encode` produces. If you have the regular [Buffer], be sure to convert it:
@@ -59,5 +217,3 @@ Important is to understand that normally `Lynon.decode` wants [BitBuffer], as `L
this possibly creates extra zero bits at the end, as bit content could be shorter than byte-grained but for the Lynon format it does not make sense. Note that when you serialize [BitBuffer], exact number of bits is written. To convert bit buffer to bytes:
Lynon.encode("hello").toBuffer()
(topic is incomplete and under construction)
+1 -1
View File
@@ -327,7 +327,7 @@ Singleton objects are declared using the `object` keyword. They provide a conven
```lyng
object Config {
val version = "1.5.5"
val version = "1.5.6-SNAPSHOT"
fun show() = println("Config version: " + version)
}
View File
Whitespace-only changes.
+56
View File
@@ -0,0 +1,56 @@
import lyng.io.db.sqlite
println("SQLite serialization demo: write-side projection and decodeAs<T>()")
class Payload(name: String, count: Int)
class Item(
id: Int,
title: String,
@DbJson meta: Payload,
@DbLynon state: Payload
) {
var note: String = ""
@DbExcept var cache: String = ""
}
val restored = openSqlite(":memory:").transaction { tx ->
tx.execute(
"create table item(id integer not null, title text not null, meta text not null, state blob not null, note text not null)"
)
val item = Item(1, "first", Payload("json", 10), Payload("bin", 20))
item.note = "created"
item.cache = "not stored"
tx.execute("insert into item(@cols(?1)) values(@vals(?1))", item)
item.title = "second"
item.meta = Payload("json2", 11)
item.state = Payload("bin2", 21)
item.note = "updated"
tx.execute(
"update item set @set(?1 except: \"id\") where id = ?2",
item,
item.id
)
val restored = tx.select("select * from item where id = ?", 1).decodeAs<Item>().first
assertEquals("second", restored.title)
assertEquals("json2", restored.meta.name)
assertEquals(11, restored.meta.count)
assertEquals("bin2", restored.state.name)
assertEquals(21, restored.state.count)
assertEquals("updated", restored.note)
restored
}
println("Restored item:")
println(" id=" + restored.id)
println(" title=" + restored.title)
println(" meta=" + restored.meta.name + "/" + restored.meta.count)
println(" state=" + restored.state.name + "/" + restored.state.count)
println(" note=" + restored.note)
println("OK")
+12 -2
View File
@@ -32,6 +32,7 @@ import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import kotlinx.coroutines.withContext
import net.sergeych.lyng.EvalSession
import net.sergeych.lyng.ExecutionError
import net.sergeych.lyng.LyngVersion
import net.sergeych.lyng.Pos
import net.sergeych.lyng.Scope
@@ -153,8 +154,8 @@ val baseScopeDefer = globalDefer {
baseCliImportManagerDefer.await().copy().apply {
invalidateCliModuleCaches()
}.newStdScope().apply {
installCliDeclarations()
installCliBuiltins()
installCliDeclarations()
addConst("ARGV", ObjList(mutableListOf()))
}
}
@@ -364,8 +365,8 @@ private fun registerLocalCliModules(manager: ImportManager, modules: List<LocalC
private suspend fun ImportManager.newCliScope(argv: List<String>): Scope =
newStdScope().apply {
installCliDeclarations()
installCliBuiltins()
installCliDeclarations()
addConst("ARGV", ObjList(argv.map { ObjString(it) }.toMutableList()))
}
@@ -547,6 +548,15 @@ suspend fun executeSource(source: Source, initialScope: Scope? = null) {
evalOnCliDispatcher(session, source)
} catch (e: CliExitRequested) {
requestedExitCode = e.code
} catch (e: ExecutionError) {
val cliExit = generateSequence<Throwable>(e) { it.cause }
.filterIsInstance<CliExitRequested>()
.firstOrNull()
if (cliExit != null) {
requestedExitCode = cliExit.code
} else {
throw e
}
}
} finally {
shutdownHooks.uninstall()
File diff suppressed because it is too large. Load diff
@@ -46,8 +46,10 @@ import net.sergeych.lyngio.http.security.HttpAccessDeniedException
import net.sergeych.lyngio.http.security.HttpAccessOp
import net.sergeych.lyngio.http.security.HttpAccessPolicy
import net.sergeych.lyngio.stdlib_included.httpLyng
import net.sergeych.lyngio.stdlib_included.http_typesLyng
private const val HTTP_MODULE_NAME = "lyng.io.http"
internal const val HTTP_TYPES_MODULE_NAME = "lyng.io.http.types"
fun createHttpModule(policy: HttpAccessPolicy, scope: Scope): Boolean =
createHttpModule(policy, scope.importManager)
@@ -55,6 +57,7 @@ fun createHttpModule(policy: HttpAccessPolicy, scope: Scope): Boolean =
fun createHttp(policy: HttpAccessPolicy, scope: Scope): Boolean = createHttpModule(policy, scope)
fun createHttpModule(policy: HttpAccessPolicy, manager: ImportManager): Boolean {
createHttpTypesModule(manager)
if (manager.packageNames.contains(HTTP_MODULE_NAME)) return false
manager.addPackage(HTTP_MODULE_NAME) { module ->
buildHttpModule(module, policy)
@@ -64,6 +67,19 @@ fun createHttpModule(policy: HttpAccessPolicy, manager: ImportManager): Boolean
fun createHttp(policy: HttpAccessPolicy, manager: ImportManager): Boolean = createHttpModule(policy, manager)
internal fun createHttpTypesModule(manager: ImportManager): Boolean {
if (manager.packageNames.contains(HTTP_TYPES_MODULE_NAME)) return false
manager.addPackage(HTTP_TYPES_MODULE_NAME) { module ->
buildHttpTypesModule(module)
}
return true
}
private suspend fun buildHttpTypesModule(module: ModuleScope) {
module.eval(Source(HTTP_TYPES_MODULE_NAME, http_typesLyng))
module.addConst("HttpHeaders", ObjHttpHeaders.type)
}
private suspend fun buildHttpModule(module: ModuleScope, policy: HttpAccessPolicy) {
module.eval(Source(HTTP_MODULE_NAME, httpLyng))
val engine = getSystemHttpEngine()
@@ -139,7 +155,7 @@ private suspend inline fun ScopeFacade.httpGuard(crossinline block: suspend () -
}
}
private class ObjHttpHeaders(
internal class ObjHttpHeaders(
singleValueHeaders: Map<String, String> = emptyMap(),
private val allHeaders: Map<String, List<String>> = emptyMap(),
) : Obj() {
@@ -201,6 +217,11 @@ private class ObjHttpHeaders(
).invokeInstanceMethod(requireScope(), "iterator")
}
}
internal fun fromHeaders(
singleValueHeaders: Map<String, String>,
allHeaders: Map<String, List<String>>,
): ObjHttpHeaders = ObjHttpHeaders(singleValueHeaders, allHeaders)
}
private fun valuesOf(name: String): List<String> = allHeaders[lookupKey(name)] ?: emptyList()
@@ -0,0 +1,514 @@
package net.sergeych.lyng.io.http.server
import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Scope
import net.sergeych.lyng.ScopeFacade
import net.sergeych.lyng.Source
import net.sergeych.lyng.Arguments
import net.sergeych.lyng.TypeDecl
import net.sergeych.lyng.asFacade
import net.sergeych.lyng.obj.Obj
import net.sergeych.lyng.obj.ObjBool
import net.sergeych.lyng.obj.ObjBuffer
import net.sergeych.lyng.obj.ObjClass
import net.sergeych.lyng.obj.ObjExternCallable
import net.sergeych.lyng.obj.ObjInt
import net.sergeych.lyng.obj.ObjList
import net.sergeych.lyng.obj.ObjNull
import net.sergeych.lyng.obj.ObjProperty
import net.sergeych.lyng.obj.ObjString
import net.sergeych.lyng.obj.ObjVoid
import net.sergeych.lyng.obj.requiredArg
import net.sergeych.lyng.obj.thisAs
import net.sergeych.lyng.io.http.ObjHttpHeaders
import net.sergeych.lyng.io.http.createHttpTypesModule
import net.sergeych.lyng.io.ws.ObjWsMessage
import net.sergeych.lyng.io.ws.createWsTypesModule
import net.sergeych.lyng.pacman.ImportManager
import net.sergeych.lyng.raiseIllegalOperation
import net.sergeych.lyng.requireNoArgs
import net.sergeych.lyng.requireScope
import net.sergeych.lyngio.http.server.HttpHandlerResult
import net.sergeych.lyngio.http.server.HttpHeader
import net.sergeych.lyngio.http.server.HttpRequest
import net.sergeych.lyngio.http.server.HttpResponse
import net.sergeych.lyngio.http.server.HttpServerConfig
import net.sergeych.lyngio.http.server.HttpWebSocketSession
import net.sergeych.lyngio.http.server.defaultReason
import net.sergeych.lyngio.http.server.startHttpServer
import net.sergeych.lyngio.net.security.NetAccessDeniedException
import net.sergeych.lyngio.net.security.NetAccessOp
import net.sergeych.lyngio.net.security.NetAccessPolicy
import net.sergeych.lyngio.stdlib_included.http_serverLyng
import net.sergeych.lyng.bytecode.BytecodeLambdaCallable
private const val HTTP_SERVER_MODULE_NAME = "lyng.io.http.server"
fun createHttpServerModule(policy: NetAccessPolicy, scope: Scope): Boolean =
createHttpServerModule(policy, scope.importManager)
fun createHttpServer(policy: NetAccessPolicy, scope: Scope): Boolean = createHttpServerModule(policy, scope)
fun createHttpServerModule(policy: NetAccessPolicy, manager: ImportManager): Boolean {
createHttpTypesModule(manager)
createWsTypesModule(manager)
if (manager.packageNames.contains(HTTP_SERVER_MODULE_NAME)) return false
manager.addPackage(HTTP_SERVER_MODULE_NAME) { module ->
buildHttpServerModule(module, policy)
}
return true
}
fun createHttpServer(policy: NetAccessPolicy, manager: ImportManager): Boolean = createHttpServerModule(policy, manager)
private suspend fun buildHttpServerModule(module: ModuleScope, policy: NetAccessPolicy) {
module.eval(Source(HTTP_SERVER_MODULE_NAME, http_serverLyng))
module.addConst("HttpHeaders", ObjHttpHeaders.type)
module.addConst("WsMessage", ObjWsMessage.type)
module.addConst("ServerRequest", ObjServerRequest.type)
module.addConst("ServerExchange", ObjServerExchange.type)
module.addConst("ServerWebSocket", ObjServerWebSocket.type)
module.addConst("HttpServerHandle", ObjHttpServerHandle.type)
module.addConst("HttpServer", ObjLyngHttpServer.type(policy))
}
private suspend inline fun ScopeFacade.httpServerGuard(crossinline block: suspend () -> Obj): Obj {
return try {
block()
} catch (e: NetAccessDeniedException) {
raiseIllegalOperation(e.reasonDetail ?: "http server access denied")
} catch (e: Exception) {
raiseIllegalOperation(e.message ?: "http server error")
}
}
private data class RegisteredCallable(
val callable: Obj,
val scope: Scope,
)
private fun captureCallable(scope: Scope, rawCallable: Obj): RegisteredCallable {
val captured = if (scope is ModuleScope) scope else scope.snapshotForClosure()
val callable = (rawCallable as? BytecodeLambdaCallable)?.freezeForLaunch(captured) ?: rawCallable
return RegisteredCallable(callable, captured)
}
private suspend fun RegisteredCallable.call(vararg args: Obj): Obj =
scope.asFacade().call(callable, Arguments(args.toList()))
private val stringType = TypeDecl.Simple("String", false)
private val nullableStringType = TypeDecl.Simple("String", true)
private val boolType = TypeDecl.Simple("Bool", false)
private val intType = TypeDecl.Simple("Int", false)
private val bufferType = TypeDecl.Simple("Buffer", false)
private val nullableBufferType = TypeDecl.Simple("Buffer", true)
private val voidType = TypeDecl.Simple("Void", false)
private val httpHeadersType = TypeDecl.Simple("HttpHeaders", false)
private val serverRequestType = TypeDecl.Simple("ServerRequest", false)
private val serverExchangeType = TypeDecl.Simple("ServerExchange", false)
private val serverWebSocketType = TypeDecl.Simple("ServerWebSocket", false)
private val nullableServerWsMessageType = TypeDecl.Simple("WsMessage", true)
private val httpServerHandleType = TypeDecl.Simple("HttpServerHandle", false)
private val httpServerType = TypeDecl.Simple("HttpServer", false)
private val nullableAnyType = TypeDecl.TypeNullableAny
private fun listType(item: TypeDecl) = TypeDecl.Generic("List", listOf(item), false)
private fun fnType(returnType: TypeDecl, vararg params: TypeDecl) =
TypeDecl.Function(receiver = null, params = params.toList(), returnType = returnType)
private fun bridgeFn(
owner: ObjClass,
name: String,
typeDecl: TypeDecl.Function,
code: suspend ScopeFacade.() -> Obj,
) {
owner.createField(
name = name,
initialValue = ObjExternCallable.fromBridge { code() },
type = net.sergeych.lyng.obj.ObjRecord.Type.Fun,
typeDecl = typeDecl,
)
}
private fun bridgeProperty(
owner: ObjClass,
name: String,
typeDecl: TypeDecl,
getter: suspend ScopeFacade.() -> Obj,
) {
owner.createField(
name = name,
initialValue = ObjProperty(name, ObjExternCallable.fromBridge { getter() }, null),
type = net.sergeych.lyng.obj.ObjRecord.Type.Property,
typeDecl = typeDecl,
)
}
private class ObjLyngHttpServer(
private val netPolicy: NetAccessPolicy,
) : Obj() {
private val methodRoutes = linkedMapOf<String, LinkedHashMap<String, RegisteredCallable>>()
private val anyRoutes = linkedMapOf<String, RegisteredCallable>()
private val wsRoutes = linkedMapOf<String, RegisteredCallable>()
private var fallback: RegisteredCallable? = null
private var handle: net.sergeych.lyngio.http.server.HttpServer? = null
override val objClass: ObjClass
get() = type(netPolicy)
companion object {
private val types = mutableMapOf<NetAccessPolicy, ObjClass>()
fun type(netPolicy: NetAccessPolicy): ObjClass =
types.getOrPut(netPolicy) {
object : ObjClass("HttpServer") {
override suspend fun callOn(scope: Scope): Obj {
if (scope.args.list.isNotEmpty()) scope.raiseError("HttpServer() does not accept arguments")
return ObjLyngHttpServer(netPolicy)
}
}.apply {
val exchangeHandlerType = fnType(nullableAnyType, serverExchangeType)
val webSocketHandlerType = fnType(nullableAnyType, serverWebSocketType, serverExchangeType)
bridgeFn(this, "get", fnType(httpServerType, stringType, exchangeHandlerType)) {
thisAs<ObjLyngHttpServer>().registerRoute("GET", this)
}
bridgeFn(this, "post", fnType(httpServerType, stringType, exchangeHandlerType)) {
thisAs<ObjLyngHttpServer>().registerRoute("POST", this)
}
bridgeFn(this, "put", fnType(httpServerType, stringType, exchangeHandlerType)) {
thisAs<ObjLyngHttpServer>().registerRoute("PUT", this)
}
bridgeFn(this, "delete", fnType(httpServerType, stringType, exchangeHandlerType)) {
thisAs<ObjLyngHttpServer>().registerRoute("DELETE", this)
}
bridgeFn(this, "any", fnType(httpServerType, stringType, exchangeHandlerType)) {
thisAs<ObjLyngHttpServer>().registerAny(this)
}
bridgeFn(this, "ws", fnType(httpServerType, stringType, webSocketHandlerType)) {
thisAs<ObjLyngHttpServer>().registerWs(this)
}
bridgeFn(this, "fallback", fnType(httpServerType, exchangeHandlerType)) {
thisAs<ObjLyngHttpServer>().registerFallback(this)
}
bridgeFn(this, "listen", fnType(httpServerHandleType, intType, nullableStringType, intType)) {
thisAs<ObjLyngHttpServer>().listen(this)
}
}
}
}
private fun ensureMutable(scope: ScopeFacade) {
if (handle != null) scope.raiseIllegalState("HttpServer routes cannot be modified after listen()")
}
private fun requirePath(scope: ScopeFacade, index: Int): String {
val path = scope.requiredArg<ObjString>(index).value
if (!path.startsWith('/')) scope.raiseIllegalArgument("path must start with '/'")
return path
}
private suspend fun registerRoute(method: String, scope: ScopeFacade): Obj = scope.httpServerGuard {
ensureMutable(scope)
val path = requirePath(scope, 0)
val handler = captureCallable(scope.requireScope(), scope.args.list[1])
val routes = methodRoutes.getOrPut(method) { linkedMapOf() }
if (routes.containsKey(path)) scope.raiseIllegalArgument("duplicate route for $method $path")
routes[path] = handler
scope.thisObj
}
private suspend fun registerAny(scope: ScopeFacade): Obj = scope.httpServerGuard {
ensureMutable(scope)
val path = requirePath(scope, 0)
val handler = captureCallable(scope.requireScope(), scope.args.list[1])
if (anyRoutes.containsKey(path)) scope.raiseIllegalArgument("duplicate route for ANY $path")
anyRoutes[path] = handler
scope.thisObj
}
private suspend fun registerWs(scope: ScopeFacade): Obj = scope.httpServerGuard {
ensureMutable(scope)
val path = requirePath(scope, 0)
val handler = captureCallable(scope.requireScope(), scope.args.list[1])
if (wsRoutes.containsKey(path)) scope.raiseIllegalArgument("duplicate websocket route for $path")
wsRoutes[path] = handler
scope.thisObj
}
private suspend fun registerFallback(scope: ScopeFacade): Obj = scope.httpServerGuard {
ensureMutable(scope)
fallback = captureCallable(scope.requireScope(), scope.args.list[0])
scope.thisObj
}
private suspend fun listen(scope: ScopeFacade): Obj = scope.httpServerGuard {
ensureMutable(scope)
val port = scope.requiredArg<ObjInt>(0).value.toInt()
val host = scope.args.list.getOrNull(1)?.let { objOrNullToString(scope, it, "host") }
val backlog = scope.args.list.getOrNull(2)?.let { objToInt(scope, it, "backlog") } ?: 128
if (port !in 0..65535) scope.raiseIllegalArgument("port must be in 0..65535")
if (backlog <= 0) scope.raiseIllegalArgument("backlog must be positive")
netPolicy.require(NetAccessOp.TcpListen(host, port, backlog))
val started = startHttpServer(
config = HttpServerConfig(host = host ?: "127.0.0.1", port = port, backlog = backlog),
) { request ->
dispatchRequest(request)
}
handle = started
ObjHttpServerHandle(started)
}
private suspend fun dispatchRequest(request: HttpRequest): HttpHandlerResult {
val path = request.head.path
if (request.head.wantsWebSocketUpgrade) {
wsRoutes[path]?.let { route ->
return HttpHandlerResult.WebSocket { session ->
val exchange = ObjServerExchange(request)
route.call(ObjServerWebSocket(session), exchange)
}
}
}
val route = methodRoutes[request.head.method.uppercase()]?.get(path)
?: anyRoutes[path]
?: fallback
if (route == null) {
return HttpHandlerResult.Response(HttpResponse(status = 404, body = "not found".encodeToByteArray()))
}
val exchange = ObjServerExchange(request)
route.call(exchange)
return when (val result = exchange.result) {
is ExchangeResult.Http -> result.value
is ExchangeResult.WebSocket -> result.value
ExchangeResult.Unhandled -> {
if (route === fallback) {
HttpHandlerResult.Response(HttpResponse(status = 404, body = "not found".encodeToByteArray()))
} else {
HttpHandlerResult.Response(HttpResponse(status = 500, body = "route handler did not handle exchange".encodeToByteArray(), close = true))
}
}
}
}
}
private class ObjHttpServerHandle(
private val handle: net.sergeych.lyngio.http.server.HttpServer,
) : Obj() {
override val objClass: ObjClass
get() = type
companion object {
val type = object : ObjClass("HttpServerHandle") {
override suspend fun callOn(scope: Scope): Obj {
scope.raiseError("HttpServerHandle cannot be created directly")
}
}.apply {
addFn("localPort") {
ObjInt(thisAs<ObjHttpServerHandle>().handle.localAddress().port.toLong())
}
addFn("close") {
requireNoArgs()
thisAs<ObjHttpServerHandle>().handle.close()
ObjVoid
}
}
}
}
private class ObjServerRequest(
private val request: HttpRequest,
) : Obj() {
override val objClass: ObjClass
get() = type
companion object {
val type = object : ObjClass("ServerRequest") {
override suspend fun callOn(scope: Scope): Obj {
scope.raiseError("ServerRequest cannot be created directly")
}
}.apply {
bridgeProperty(this, "method", stringType) {
ObjString(thisAs<ObjServerRequest>().request.head.method)
}
bridgeProperty(this, "target", stringType) {
ObjString(thisAs<ObjServerRequest>().request.head.target)
}
bridgeProperty(this, "path", stringType) {
ObjString(thisAs<ObjServerRequest>().request.head.path)
}
bridgeProperty(this, "query", nullableStringType) {
thisAs<ObjServerRequest>().request.head.query?.let(::ObjString) ?: ObjNull
}
bridgeProperty(this, "headers", httpHeadersType) {
requestHeadersObj(thisAs<ObjServerRequest>().request.head.headers)
}
bridgeProperty(this, "body", bufferType) {
ObjBuffer(thisAs<ObjServerRequest>().request.body.toUByteArray())
}
bridgeFn(this, "text", fnType(stringType)) {
ObjString(thisAs<ObjServerRequest>().request.body.decodeToString())
}
bridgeFn(this, "isWebSocketUpgrade", fnType(boolType)) {
ObjBool(thisAs<ObjServerRequest>().request.head.wantsWebSocketUpgrade)
}
}
}
}
private sealed interface ExchangeResult {
data object Unhandled : ExchangeResult
data class Http(val value: HttpHandlerResult.Response) : ExchangeResult
data class WebSocket(val value: HttpHandlerResult.WebSocket) : ExchangeResult
}
private class ObjServerExchange(
private val request: HttpRequest,
) : Obj() {
private val responseHeaders = linkedMapOf<String, MutableList<String>>()
var result: ExchangeResult = ExchangeResult.Unhandled
private set
override val objClass: ObjClass
get() = type
companion object {
val type = object : ObjClass("ServerExchange") {
override suspend fun callOn(scope: Scope): Obj {
scope.raiseError("ServerExchange cannot be created directly")
}
}.apply {
bridgeProperty(this, "request", serverRequestType) {
ObjServerRequest(thisAs<ObjServerExchange>().request)
}
bridgeFn(this, "respond", fnType(voidType, intType, nullableBufferType)) {
val self = thisAs<ObjServerExchange>()
val status = args.list.getOrNull(0)?.let { objToInt(this, it, "status") } ?: 200
val body = args.list.getOrNull(1)?.let { objBufferOrNull(this, it, "body") }
self.setHttpResponse(status, body?.byteArray?.toByteArray() ?: ByteArray(0))
ObjVoid
}
bridgeFn(this, "respondText", fnType(voidType, intType, stringType)) {
val self = thisAs<ObjServerExchange>()
val status = args.list.getOrNull(0)?.let { objToInt(this, it, "status") } ?: 200
val bodyText = args.list.getOrNull(1)?.let { objOrNullToString(this, it, "bodyText") } ?: ""
self.setHttpResponse(status, bodyText.encodeToByteArray())
ObjVoid
}
bridgeFn(this, "setHeader", fnType(voidType, stringType, stringType)) {
val self = thisAs<ObjServerExchange>()
val name = requiredArg<ObjString>(0).value
val value = requiredArg<ObjString>(1).value
self.ensureMutable(this)
self.responseHeaders[name] = mutableListOf(value)
ObjVoid
}
bridgeFn(this, "addHeader", fnType(voidType, stringType, stringType)) {
val self = thisAs<ObjServerExchange>()
val name = requiredArg<ObjString>(0).value
val value = requiredArg<ObjString>(1).value
self.ensureMutable(this)
self.responseHeaders.getOrPut(name) { mutableListOf() }.add(value)
ObjVoid
}
bridgeFn(
this,
"acceptWebSocket",
fnType(voidType, fnType(nullableAnyType, serverWebSocketType, serverExchangeType))
) {
val self = thisAs<ObjServerExchange>()
val registered = captureCallable(requireScope(), args.list[0])
self.ensureMutable(this)
self.result = ExchangeResult.WebSocket(
HttpHandlerResult.WebSocket { session ->
registered.call(ObjServerWebSocket(session), self)
}
)
ObjVoid
}
bridgeFn(this, "isHandled", fnType(boolType)) {
ObjBool(thisAs<ObjServerExchange>().result !== ExchangeResult.Unhandled)
}
}
}
private fun ensureMutable(scope: ScopeFacade) {
if (result !== ExchangeResult.Unhandled) {
scope.raiseIllegalState("exchange has already been handled")
}
}
private fun setHttpResponse(status: Int, body: ByteArray) {
result = ExchangeResult.Http(
HttpHandlerResult.Response(
HttpResponse(
status = status,
reason = defaultReason(status),
headers = responseHeaders.entries.flatMap { (name, values) -> values.map { HttpHeader(name, it) } },
body = body,
)
)
)
}
}
private class ObjServerWebSocket(
private val session: HttpWebSocketSession,
) : Obj() {
override val objClass: ObjClass
get() = type
companion object {
val type = object : ObjClass("ServerWebSocket") {
override suspend fun callOn(scope: Scope): Obj {
scope.raiseError("ServerWebSocket cannot be created directly")
}
}.apply {
bridgeFn(this, "isOpen", fnType(boolType)) {
ObjBool(thisAs<ObjServerWebSocket>().session.isOpen())
}
bridgeFn(this, "sendText", fnType(voidType, stringType)) {
thisAs<ObjServerWebSocket>().session.sendText(requiredArg<ObjString>(0).value)
ObjVoid
}
bridgeFn(this, "sendBytes", fnType(voidType, bufferType)) {
thisAs<ObjServerWebSocket>().session.sendBytes(requiredArg<ObjBuffer>(0).byteArray.toByteArray())
ObjVoid
}
bridgeFn(this, "receive", fnType(nullableServerWsMessageType)) {
thisAs<ObjServerWebSocket>().session.receive()?.let(ObjWsMessage::from) ?: ObjNull
}
bridgeFn(this, "close", fnType(voidType, intType, stringType)) {
val code = args.list.getOrNull(0)?.let { objToInt(this, it, "code") } ?: 1000
val reason = args.list.getOrNull(1)?.let { objOrNullToString(this, it, "reason") } ?: ""
thisAs<ObjServerWebSocket>().session.close(code, reason)
ObjVoid
}
}
}
}
private fun requestHeadersObj(headers: net.sergeych.lyngio.http.server.HttpHeaders): ObjHttpHeaders {
val all = headers.entries().groupBy(HttpHeader::name, HttpHeader::value)
val single = all.mapValues { (_, values) -> values.first() }
return ObjHttpHeaders.fromHeaders(single, all)
}
private suspend fun objOrNullToString(scope: ScopeFacade, value: Obj, name: String): String? = when (value) {
ObjNull -> null
else -> scope.toStringOf(value).value
}
private fun objToInt(scope: ScopeFacade, value: Obj, name: String): Int = when (value) {
is ObjInt -> value.value.toInt()
else -> scope.raiseClassCastError("$name must be Int")
}
private fun objBufferOrNull(scope: ScopeFacade, value: Obj, name: String): ObjBuffer? = when (value) {
ObjNull -> null
is ObjBuffer -> value
else -> scope.raiseClassCastError("$name must be Buffer or null")
}
@@ -47,8 +47,10 @@ import net.sergeych.lyngio.net.security.NetAccessDeniedException
import net.sergeych.lyngio.net.security.NetAccessOp
import net.sergeych.lyngio.net.security.NetAccessPolicy
import net.sergeych.lyngio.stdlib_included.netLyng
import net.sergeych.lyngio.stdlib_included.net_typesLyng
private const val NET_MODULE_NAME = "lyng.io.net"
internal const val NET_TYPES_MODULE_NAME = "lyng.io.net.types"
fun createNetModule(policy: NetAccessPolicy, scope: Scope): Boolean =
createNetModule(policy, scope.importManager)
@@ -56,6 +58,7 @@ fun createNetModule(policy: NetAccessPolicy, scope: Scope): Boolean =
fun createNet(policy: NetAccessPolicy, scope: Scope): Boolean = createNetModule(policy, scope)
fun createNetModule(policy: NetAccessPolicy, manager: ImportManager): Boolean {
createNetTypesModule(manager)
if (manager.packageNames.contains(NET_MODULE_NAME)) return false
manager.addPackage(NET_MODULE_NAME) { module ->
buildNetModule(module, policy)
@@ -65,6 +68,21 @@ fun createNetModule(policy: NetAccessPolicy, manager: ImportManager): Boolean {
fun createNet(policy: NetAccessPolicy, manager: ImportManager): Boolean = createNetModule(policy, manager)
internal fun createNetTypesModule(manager: ImportManager): Boolean {
if (manager.packageNames.contains(NET_TYPES_MODULE_NAME)) return false
manager.addPackage(NET_TYPES_MODULE_NAME) { module ->
buildNetTypesModule(module)
}
return true
}
private suspend fun buildNetTypesModule(module: ModuleScope) {
module.eval(Source(NET_TYPES_MODULE_NAME, net_typesLyng))
val enumValues = NetEnumValues.load(module)
module.addConst("SocketAddress", ObjSocketAddress.type(enumValues))
module.addConst("Datagram", ObjDatagram.type(enumValues))
}
private suspend fun buildNetModule(module: ModuleScope, policy: NetAccessPolicy) {
module.eval(Source(NET_MODULE_NAME, netLyng))
val engine = getSystemNetEngine()
@@ -164,10 +182,12 @@ private class ObjSocketAddress(
override suspend fun defaultToString(scope: Scope): ObjString = ObjString(renderAddress(address))
companion object {
private val types = mutableMapOf<NetEnumValues, ObjClass>()
private data class EnumKey(val ipv4: Obj, val ipv6: Obj)
private val types = mutableMapOf<EnumKey, ObjClass>()
fun type(enumValues: NetEnumValues): ObjClass =
types.getOrPut(enumValues) {
types.getOrPut(EnumKey(enumValues.ipv4, enumValues.ipv6)) {
object : ObjClass("SocketAddress") {
override suspend fun callOn(scope: Scope): Obj {
scope.raiseError("SocketAddress cannot be created directly")
@@ -191,10 +211,12 @@ private class ObjDatagram(
get() = type(enumValues)
companion object {
private val types = mutableMapOf<NetEnumValues, ObjClass>()
private data class EnumKey(val ipv4: Obj, val ipv6: Obj)
private val types = mutableMapOf<EnumKey, ObjClass>()
fun type(enumValues: NetEnumValues): ObjClass =
types.getOrPut(enumValues) {
types.getOrPut(EnumKey(enumValues.ipv4, enumValues.ipv6)) {
object : ObjClass("Datagram") {
override suspend fun callOn(scope: Scope): Obj {
scope.raiseError("Datagram cannot be created directly")
@@ -37,6 +37,7 @@ import net.sergeych.lyng.raiseIllegalOperation
import net.sergeych.lyng.requireNoArgs
import net.sergeych.lyng.requireScope
import net.sergeych.lyngio.stdlib_included.wsLyng
import net.sergeych.lyngio.stdlib_included.ws_typesLyng
import net.sergeych.lyngio.ws.LyngWsEngine
import net.sergeych.lyngio.ws.LyngWsMessage
import net.sergeych.lyngio.ws.LyngWsSession
@@ -46,6 +47,7 @@ import net.sergeych.lyngio.ws.security.WsAccessOp
import net.sergeych.lyngio.ws.security.WsAccessPolicy
private const val WS_MODULE_NAME = "lyng.io.ws"
internal const val WS_TYPES_MODULE_NAME = "lyng.io.ws.types"
fun createWsModule(policy: WsAccessPolicy, scope: Scope): Boolean =
createWsModule(policy, scope.importManager)
@@ -53,6 +55,7 @@ fun createWsModule(policy: WsAccessPolicy, scope: Scope): Boolean =
fun createWs(policy: WsAccessPolicy, scope: Scope): Boolean = createWsModule(policy, scope)
fun createWsModule(policy: WsAccessPolicy, manager: ImportManager): Boolean {
createWsTypesModule(manager)
if (manager.packageNames.contains(WS_MODULE_NAME)) return false
manager.addPackage(WS_MODULE_NAME) { module ->
buildWsModule(module, policy)
@@ -62,6 +65,19 @@ fun createWsModule(policy: WsAccessPolicy, manager: ImportManager): Boolean {
fun createWs(policy: WsAccessPolicy, manager: ImportManager): Boolean = createWsModule(policy, manager)
internal fun createWsTypesModule(manager: ImportManager): Boolean {
if (manager.packageNames.contains(WS_TYPES_MODULE_NAME)) return false
manager.addPackage(WS_TYPES_MODULE_NAME) { module ->
buildWsTypesModule(module)
}
return true
}
private suspend fun buildWsTypesModule(module: ModuleScope) {
module.eval(Source(WS_TYPES_MODULE_NAME, ws_typesLyng))
module.addConst("WsMessage", ObjWsMessage.type)
}
private suspend fun buildWsModule(module: ModuleScope, policy: WsAccessPolicy) {
module.eval(Source(WS_MODULE_NAME, wsLyng))
val engine = getSystemWsEngine()
@@ -92,7 +108,7 @@ private suspend inline fun ScopeFacade.wsGuard(crossinline block: suspend () ->
}
}
private class ObjWsMessage(
internal class ObjWsMessage(
private val message: LyngWsMessage,
) : Obj() {
override val objClass: ObjClass
@@ -112,6 +128,8 @@ private class ObjWsMessage(
thisAs<ObjWsMessage>().message.data?.let { ObjBuffer(it.toUByteArray()) } ?: ObjNull
})
}
internal fun from(message: LyngWsMessage): ObjWsMessage = ObjWsMessage(message)
}
}
@@ -152,7 +170,7 @@ private class ObjWsSession(
addFn("receive") {
val self = thisAs<ObjWsSession>()
self.policy.require(WsAccessOp.Receive(self.targetUrl))
self.session.receive()?.let(::ObjWsMessage) ?: ObjNull
self.session.receive()?.let(ObjWsMessage::from) ?: ObjNull
}
addFn("close") {
val self = thisAs<ObjWsSession>()
@@ -0,0 +1,50 @@
package net.sergeych.lyngio.http.server
import net.sergeych.lyngio.net.LyngTcpSocket
internal class BufferedSocketReader(
private val socket: LyngTcpSocket,
) {
private var pending = ByteArray(0)
suspend fun readLine(
maxBytes: Int,
overflowStatus: Int,
overflowMessage: String,
): String? {
require(maxBytes > 0) { "maxBytes must be positive" }
val out = ByteArray(maxBytes)
var count = 0
while (true) {
val next = readByte() ?: return if (count == 0) null else out.copyOf(count).decodeToString()
if (next == '\n'.code.toByte()) {
return if (count > 0 && out[count - 1] == '\r'.code.toByte()) {
out.copyOf(count - 1).decodeToString()
} else {
out.copyOf(count).decodeToString()
}
}
if (count >= maxBytes) throw HttpProtocolException(overflowStatus, overflowMessage)
out[count++] = next
}
}
suspend fun readExact(byteCount: Int): ByteArray? {
require(byteCount >= 0) { "byteCount must be non-negative" }
if (byteCount == 0) return ByteArray(0)
while (pending.size < byteCount) {
val chunk = socket.read(maxOf(4096, byteCount - pending.size)) ?: break
if (chunk.isEmpty()) break
pending += chunk
}
if (pending.size < byteCount) return null
val result = pending.copyOfRange(0, byteCount)
pending = pending.copyOfRange(byteCount, pending.size)
return result
}
private suspend fun readByte(): Byte? {
val bytes = readExact(1) ?: return null
return bytes[0]
}
}
@@ -0,0 +1,181 @@
package net.sergeych.lyngio.http.server
internal class HttpProtocolException(
val status: Int,
message: String,
) : IllegalStateException(message)
internal suspend fun parseHttpRequest(
reader: BufferedSocketReader,
config: HttpServerConfig,
): HttpRequest? {
val requestLine = reader.readLine(
maxBytes = config.maxRequestLineBytes,
overflowStatus = 414,
overflowMessage = "request line is too long",
) ?: return null
val requestHead = parseRequestLine(requestLine, config)
val headerEntries = parseHeaders(reader, config)
val headers = HttpHeaders(headerEntries)
validateHost(headers)
val contentLength = parseContentLength(headers, config)
validateUnsupportedRequestFeatures(headers)
val wantsWebSocketUpgrade = isWebSocketUpgrade(requestHead.method, headers)
validateWebSocketUpgradeRequest(headers, requestHead.method, contentLength, wantsWebSocketUpgrade)
val body = if (contentLength != null) {
reader.readExact(contentLength)
?: throw HttpProtocolException(400, "unexpected EOF while reading request body")
} else {
ByteArray(0)
}
return HttpRequest(
head = HttpRequestHead(
method = requestHead.method,
target = requestHead.target,
path = requestHead.path,
query = requestHead.query,
version = requestHead.version,
headers = headers,
contentLength = contentLength,
wantsClose = headers.containsToken("Connection", "close"),
wantsWebSocketUpgrade = wantsWebSocketUpgrade,
),
body = body,
)
}
private data class ParsedRequestLine(
val method: String,
val target: String,
val path: String,
val query: String?,
val version: String,
)
private fun parseRequestLine(line: String, config: HttpServerConfig): ParsedRequestLine {
val firstSpace = line.indexOf(' ')
val lastSpace = line.lastIndexOf(' ')
if (firstSpace <= 0 || lastSpace <= firstSpace || lastSpace == line.lastIndex) {
throw HttpProtocolException(400, "malformed request line")
}
val method = line.substring(0, firstSpace)
val target = line.substring(firstSpace + 1, lastSpace)
val version = line.substring(lastSpace + 1)
if (!method.all(::isHttpTokenChar)) {
throw HttpProtocolException(400, "invalid HTTP method")
}
if (version != "HTTP/1.1") {
throw HttpProtocolException(505, "unsupported HTTP version: $version")
}
if (target.length > config.maxRequestLineBytes) {
throw HttpProtocolException(414, "request target is too long")
}
if (!target.startsWith('/')) {
throw HttpProtocolException(400, "only origin-form request targets are supported")
}
val queryAt = target.indexOf('?')
val path = if (queryAt >= 0) target.substring(0, queryAt) else target
val query = if (queryAt >= 0) target.substring(queryAt + 1) else null
return ParsedRequestLine(method = method, target = target, path = path, query = query, version = version)
}
private suspend fun parseHeaders(
reader: BufferedSocketReader,
config: HttpServerConfig,
): List<HttpHeader> {
val headers = ArrayList<HttpHeader>()
var totalBytes = 0
while (true) {
val line = reader.readLine(
maxBytes = config.maxHeaderBytes,
overflowStatus = 431,
overflowMessage = "request headers are too large",
)
?: throw HttpProtocolException(400, "unexpected EOF while reading headers")
totalBytes += line.length + 2
if (totalBytes > config.maxHeaderBytes) {
throw HttpProtocolException(431, "request headers are too large")
}
if (line.isEmpty()) return headers
if (line.firstOrNull() == ' ' || line.firstOrNull() == '\t') {
throw HttpProtocolException(400, "obsolete folded headers are not supported")
}
val colonAt = line.indexOf(':')
if (colonAt <= 0) throw HttpProtocolException(400, "invalid header syntax")
val name = line.substring(0, colonAt)
if (!name.all(::isHttpTokenChar)) {
throw HttpProtocolException(400, "invalid header name: $name")
}
val value = line.substring(colonAt + 1).trim(' ', '\t')
if (value.any { it == '\r' || it == '\n' || it.code < 0x20 && it != '\t' }) {
throw HttpProtocolException(400, "invalid header value")
}
headers += HttpHeader(name, value)
if (headers.size > config.maxHeaderCount) {
throw HttpProtocolException(431, "too many headers")
}
}
}
private fun validateHost(headers: HttpHeaders) {
val values = headers.all("Host").map(String::trim)
if (values.isEmpty()) throw HttpProtocolException(400, "Host header is required")
if (values.distinct().size > 1) throw HttpProtocolException(400, "conflicting Host header values")
}
private fun parseContentLength(headers: HttpHeaders, config: HttpServerConfig): Int? {
val values = headers.all("Content-Length")
if (values.isEmpty()) return null
val normalized = values.flatMap { raw -> raw.split(',').map(String::trim) }
if (normalized.any { it.isEmpty() }) throw HttpProtocolException(400, "invalid Content-Length")
val distinct = normalized.distinct()
if (distinct.size > 1) throw HttpProtocolException(400, "conflicting Content-Length values")
val parsed = distinct.single().toLongOrNull() ?: throw HttpProtocolException(400, "invalid Content-Length")
if (parsed < 0L || parsed > Int.MAX_VALUE.toLong()) throw HttpProtocolException(400, "invalid Content-Length")
if (parsed > config.maxBodyBytes.toLong()) throw HttpProtocolException(413, "request body is too large")
return parsed.toInt()
}
private fun validateUnsupportedRequestFeatures(headers: HttpHeaders) {
if (headers.all("Transfer-Encoding").isNotEmpty()) {
throw HttpProtocolException(501, "Transfer-Encoding is not supported")
}
if (headers.first("Expect")?.equals("100-continue", ignoreCase = true) == true) {
throw HttpProtocolException(501, "Expect: 100-continue is not supported")
}
val upgrade = headers.first("Upgrade")
if (upgrade != null && !upgrade.equals("websocket", ignoreCase = true)) {
throw HttpProtocolException(501, "unsupported Upgrade value")
}
}
private fun isWebSocketUpgrade(method: String, headers: HttpHeaders): Boolean =
method.equals("GET", ignoreCase = true) &&
headers.first("Upgrade")?.equals("websocket", ignoreCase = true) == true &&
headers.containsToken("Connection", "upgrade")
private fun validateWebSocketUpgradeRequest(
headers: HttpHeaders,
method: String,
contentLength: Int?,
wantsWebSocketUpgrade: Boolean,
) {
if (!wantsWebSocketUpgrade) return
if (!method.equals("GET", ignoreCase = true)) {
throw HttpProtocolException(400, "websocket upgrade requires GET")
}
if (contentLength != null && contentLength != 0) {
throw HttpProtocolException(400, "websocket upgrade request must not include a body")
}
if (headers.first("Sec-WebSocket-Key").isNullOrBlank()) {
throw HttpProtocolException(400, "missing Sec-WebSocket-Key")
}
if (headers.first("Sec-WebSocket-Version") != "13") {
throw HttpProtocolException(400, "unsupported Sec-WebSocket-Version")
}
}
private fun isHttpTokenChar(ch: Char): Boolean =
ch in '0'..'9' || ch in 'A'..'Z' || ch in 'a'..'z' || ch in setOf(
'!', '#', '$', '%', '&', '\'', '*', '+', '-', '.', '^', '_', '`', '|', '~'
)
@@ -0,0 +1,101 @@
package net.sergeych.lyngio.http.server
import net.sergeych.lyngio.net.LyngSocketAddress
import net.sergeych.lyngio.ws.LyngWsMessage
internal data class HttpServerConfig(
val host: String? = "127.0.0.1",
val port: Int = 0,
val backlog: Int = 128,
val reuseAddress: Boolean = true,
val maxRequestLineBytes: Int = 8 * 1024,
val maxHeaderBytes: Int = 32 * 1024,
val maxHeaderCount: Int = 100,
val maxBodyBytes: Int = 1024 * 1024,
val keepAliveTimeoutMillis: Long = 15_000,
)
internal data class HttpHeader(
val name: String,
val value: String,
)
internal class HttpHeaders(
private val headerEntries: List<HttpHeader>,
) {
fun first(name: String): String? =
headerEntries.firstOrNull { it.name.equals(name, ignoreCase = true) }?.value
fun all(name: String): List<String> =
headerEntries.filter { it.name.equals(name, ignoreCase = true) }.map(HttpHeader::value)
fun containsToken(name: String, token: String): Boolean =
all(name).flatMap { value -> value.split(',') }
.any { it.trim().equals(token, ignoreCase = true) }
fun entries(): List<HttpHeader> = headerEntries
}
internal data class HttpRequestHead(
val method: String,
val target: String,
val path: String,
val query: String?,
val version: String,
val headers: HttpHeaders,
val contentLength: Int?,
val wantsClose: Boolean,
val wantsWebSocketUpgrade: Boolean,
)
internal data class HttpRequest(
val head: HttpRequestHead,
val body: ByteArray,
)
internal data class HttpResponse(
val status: Int,
val reason: String = defaultReason(status),
val headers: List<HttpHeader> = emptyList(),
val body: ByteArray = ByteArray(0),
val close: Boolean = false,
)
internal interface HttpWebSocketSession {
fun isOpen(): Boolean
suspend fun sendText(text: String)
suspend fun sendBytes(data: ByteArray)
suspend fun receive(): LyngWsMessage?
suspend fun close(code: Int = 1000, reason: String = "")
}
internal sealed interface HttpHandlerResult {
data class Response(val response: HttpResponse) : HttpHandlerResult
data class WebSocket(val handler: suspend (HttpWebSocketSession) -> Unit) : HttpHandlerResult
}
internal fun interface HttpHandler {
suspend fun handle(request: HttpRequest): HttpHandlerResult
}
internal interface HttpServer {
fun isOpen(): Boolean
fun localAddress(): LyngSocketAddress
fun close()
}
internal fun defaultReason(status: Int): String = when (status) {
101 -> "Switching Protocols"
200 -> "OK"
204 -> "No Content"
400 -> "Bad Request"
404 -> "Not Found"
413 -> "Payload Too Large"
414 -> "URI Too Long"
426 -> "Upgrade Required"
431 -> "Request Header Fields Too Large"
500 -> "Internal Server Error"
501 -> "Not Implemented"
505 -> "HTTP Version Not Supported"
else -> "HTTP $status"
}
@@ -0,0 +1,149 @@
package net.sergeych.lyngio.http.server
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.cancel
import kotlinx.coroutines.launch
import kotlinx.coroutines.withTimeout
import net.sergeych.lyngio.net.LyngNetEngine
import net.sergeych.lyngio.net.LyngSocketAddress
import net.sergeych.lyngio.net.LyngTcpServer
import net.sergeych.lyngio.net.LyngTcpSocket
import net.sergeych.lyngio.net.getSystemNetEngine
internal fun startHttpServer(
config: HttpServerConfig = HttpServerConfig(),
netEngine: LyngNetEngine = getSystemNetEngine(),
handler: HttpHandler,
): HttpServer {
if (!netEngine.isSupported || !netEngine.isTcpServerAvailable) {
throw UnsupportedOperationException("HTTP server is not supported on this runtime")
}
return StartedHttpServer(config, netEngine, handler)
}
private class StartedHttpServer(
private val config: HttpServerConfig,
private val netEngine: LyngNetEngine,
private val handler: HttpHandler,
) : HttpServer {
private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
private var serverRef: LyngTcpServer? = null
private var open = true
init {
scope.launch {
val server = netEngine.tcpListen(
host = config.host,
port = config.port,
backlog = config.backlog,
reuseAddress = config.reuseAddress,
)
serverRef = server
acceptLoop(server)
}
}
override fun isOpen(): Boolean = open && (serverRef?.isOpen() ?: true)
override fun localAddress(): LyngSocketAddress =
serverRef?.localAddress() ?: throw IllegalStateException("server is not bound yet")
override fun close() {
if (!open) return
open = false
serverRef?.close()
scope.cancel()
}
private suspend fun acceptLoop(server: LyngTcpServer) {
try {
while (open && server.isOpen()) {
val socket = try {
server.accept()
} catch (e: CancellationException) {
throw e
} catch (_: Throwable) {
if (!open || !server.isOpen()) break
continue
}
scope.launch {
handleConnection(socket)
}
}
} finally {
open = false
server.close()
}
}
private suspend fun handleConnection(socket: LyngTcpSocket) {
val reader = BufferedSocketReader(socket)
try {
while (socket.isOpen()) {
val request = try {
withTimeout(config.keepAliveTimeoutMillis) {
parseHttpRequest(reader, config)
}
} catch (_: CancellationException) {
throw CancellationException()
} catch (e: HttpProtocolException) {
safeWriteError(socket, e.status, e.message ?: defaultReason(e.status))
break
} catch (_: Throwable) {
safeWriteError(socket, 400, defaultReason(400))
break
} ?: break
val result = try {
handler.handle(request)
} catch (_: CancellationException) {
throw CancellationException()
} catch (_: Throwable) {
HttpHandlerResult.Response(HttpResponse(status = 500, close = true))
}
when (result) {
is HttpHandlerResult.Response -> {
val close = request.head.wantsClose || result.response.close
writeHttpResponse(socket, result.response, closeConnection = close)
if (close) break
}
is HttpHandlerResult.WebSocket -> {
if (!request.head.wantsWebSocketUpgrade) {
writeHttpResponse(
socket,
HttpResponse(status = 400, close = true, body = "WebSocket upgrade required".encodeToByteArray()),
closeConnection = true,
)
break
}
val session = upgradeToWebSocket(socket, request)
try {
result.handler(session)
} finally {
session.close()
}
break
}
}
}
} catch (_: CancellationException) {
} finally {
socket.close()
}
}
private suspend fun safeWriteError(socket: LyngTcpSocket, status: Int, message: String) {
try {
writeHttpResponse(
socket,
HttpResponse(status = status, body = message.encodeToByteArray(), close = true),
closeConnection = true,
)
} catch (_: Throwable) {
}
}
}
@@ -0,0 +1,44 @@
package net.sergeych.lyngio.http.server
import net.sergeych.lyngio.net.LyngTcpSocket
internal suspend fun writeHttpResponse(
socket: LyngTcpSocket,
response: HttpResponse,
closeConnection: Boolean,
) {
val body = response.body
val headerLines = LinkedHashMap<String, MutableList<String>>()
response.headers.forEach { header ->
headerLines.getOrPut(header.name) { mutableListOf() }.add(header.value)
}
if (headerLines.keys.none { it.equals("Content-Length", ignoreCase = true) }) {
headerLines["Content-Length"] = mutableListOf(body.size.toString())
}
if (closeConnection) {
val connectionKey = headerLines.keys.firstOrNull { it.equals("Connection", ignoreCase = true) }
if (connectionKey != null) {
headerLines.remove(connectionKey)
}
headerLines["Connection"] = mutableListOf("close")
}
val head = buildString {
append("HTTP/1.1 ")
append(response.status)
append(' ')
append(response.reason)
append("\r\n")
headerLines.forEach { (name, values) ->
values.forEach { value ->
append(name)
append(": ")
append(value)
append("\r\n")
}
}
append("\r\n")
}
socket.writeUtf8(head)
if (body.isNotEmpty()) socket.write(body)
socket.flush()
}
@@ -0,0 +1,295 @@
package net.sergeych.lyngio.http.server
import net.sergeych.lyngio.net.LyngTcpSocket
import net.sergeych.lyngio.ws.LyngWsMessage
import net.sergeych.mp_tools.encodeToBase64
internal suspend fun upgradeToWebSocket(
socket: LyngTcpSocket,
request: HttpRequest,
): HttpWebSocketSession {
val key = request.head.headers.first("Sec-WebSocket-Key")
?: throw HttpProtocolException(400, "missing Sec-WebSocket-Key")
val response = buildString {
append("HTTP/1.1 101 Switching Protocols\r\n")
append("Upgrade: websocket\r\n")
append("Connection: Upgrade\r\n")
append("Sec-WebSocket-Accept: ")
append(websocketAcceptKey(key))
append("\r\n\r\n")
}
socket.writeUtf8(response)
socket.flush()
return SocketHttpWebSocketSession(socket)
}
private class SocketHttpWebSocketSession(
private val socket: LyngTcpSocket,
) : HttpWebSocketSession {
private val reader = BufferedSocketReader(socket)
private var closed = false
private var fragmentedOpcode: Int? = null
private var fragmentedPayload = ByteArray(0)
private var closeSent = false
override fun isOpen(): Boolean = !closed && socket.isOpen()
override suspend fun sendText(text: String) {
ensureOpen()
sendFrame(OPCODE_TEXT, text.encodeToByteArray())
}
override suspend fun sendBytes(data: ByteArray) {
ensureOpen()
sendFrame(OPCODE_BINARY, data)
}
override suspend fun receive(): LyngWsMessage? {
while (!closed) {
val frame = readFrame() ?: run {
release()
return null
}
when (frame.opcode) {
OPCODE_CONTINUATION -> {
val opcode = fragmentedOpcode ?: throw IllegalStateException("unexpected websocket continuation frame")
fragmentedPayload += frame.payload
if (frame.fin) {
val payload = fragmentedPayload
fragmentedOpcode = null
fragmentedPayload = ByteArray(0)
return payload.toMessage(opcode)
}
}
OPCODE_TEXT, OPCODE_BINARY -> {
if (frame.fin) return frame.payload.toMessage(frame.opcode)
fragmentedOpcode = frame.opcode
fragmentedPayload = frame.payload
}
OPCODE_CLOSE -> {
if (!closeSent) {
sendFrame(OPCODE_CLOSE, frame.payload)
closeSent = true
}
release()
return null
}
OPCODE_PING -> sendFrame(OPCODE_PONG, frame.payload)
OPCODE_PONG -> Unit
else -> Unit
}
}
return null
}
override suspend fun close(code: Int, reason: String) {
if (closed) return
val reasonBytes = reason.encodeToByteArray()
val payload = ByteArray(reasonBytes.size + 2)
payload[0] = (code shr 8).toByte()
payload[1] = code.toByte()
reasonBytes.copyInto(payload, destinationOffset = 2)
try {
if (!closeSent) {
sendFrame(OPCODE_CLOSE, payload)
closeSent = true
}
} finally {
release()
}
}
private suspend fun sendFrame(opcode: Int, payload: ByteArray) {
socket.write(buildFrameHeader(opcode, payload.size, masked = false) + payload)
socket.flush()
}
private suspend fun readFrame(): WsFrame? {
val head = reader.readExact(2) ?: return null
val fin = (head[0].toInt() and 0x80) != 0
val opcode = head[0].toInt() and 0x0f
val masked = (head[1].toInt() and 0x80) != 0
val payloadLength = when (val lengthCode = head[1].toInt() and 0x7f) {
126 -> {
val extended = reader.readExact(2) ?: return null
((extended[0].toInt() and 0xff) shl 8) or (extended[1].toInt() and 0xff)
}
127 -> {
val extended = reader.readExact(8) ?: return null
var acc = 0L
extended.forEach { byte ->
acc = (acc shl 8) or (byte.toInt() and 0xff).toLong()
}
require(acc <= Int.MAX_VALUE.toLong()) { "websocket frame is too large" }
acc.toInt()
}
else -> lengthCode
}
if (!masked) throw IllegalStateException("client websocket frames must be masked")
val mask = reader.readExact(4) ?: return null
val payload = if (payloadLength > 0) reader.readExact(payloadLength) ?: return null else ByteArray(0)
payload.indices.forEach { index ->
payload[index] = (payload[index].toInt() xor mask[index % mask.size].toInt()).toByte()
}
return WsFrame(fin = fin, opcode = opcode, payload = payload)
}
private fun ensureOpen() {
if (!isOpen()) throw IllegalStateException("websocket session is closed")
}
private fun release() {
if (closed) return
closed = true
socket.close()
}
}
private data class WsFrame(
val fin: Boolean,
val opcode: Int,
val payload: ByteArray,
)
private fun ByteArray.toMessage(opcode: Int): LyngWsMessage = when (opcode) {
OPCODE_TEXT -> LyngWsMessage(isText = true, text = decodeToString())
OPCODE_BINARY -> LyngWsMessage(isText = false, data = copyOf())
else -> throw IllegalStateException("unsupported websocket opcode: $opcode")
}
private fun websocketAcceptKey(key: String): String =
sha1((key + WS_GUID).encodeToByteArray()).encodeToBase64()
private fun buildFrameHeader(opcode: Int, payloadSize: Int, masked: Boolean): ByteArray {
require(payloadSize >= 0) { "payload size must be non-negative" }
val firstByte = (0x80 or (opcode and 0x0f)).toByte()
val maskBit = if (masked) 0x80 else 0
return when {
payloadSize <= 125 -> byteArrayOf(firstByte, (maskBit or payloadSize).toByte())
payloadSize <= 0xffff -> byteArrayOf(
firstByte,
(maskBit or 126).toByte(),
((payloadSize ushr 8) and 0xff).toByte(),
(payloadSize and 0xff).toByte(),
)
else -> byteArrayOf(
firstByte,
(maskBit or 127).toByte(),
0,
0,
0,
0,
((payloadSize ushr 24) and 0xff).toByte(),
((payloadSize ushr 16) and 0xff).toByte(),
((payloadSize ushr 8) and 0xff).toByte(),
(payloadSize and 0xff).toByte(),
)
}
}
private fun sha1(input: ByteArray): ByteArray {
var h0 = 0x67452301
var h1 = 0xEFCDAB89.toInt()
var h2 = 0x98BADCFE.toInt()
var h3 = 0x10325476
var h4 = 0xC3D2E1F0.toInt()
val msgLen = input.size
val bitLen = msgLen.toLong() * 8L
val totalLen = ((msgLen + 1 + 8 + 63) / 64) * 64
val padded = ByteArray(totalLen).also { buf ->
input.copyInto(buf)
buf[msgLen] = 0x80.toByte()
for (i in 0..7) {
buf[totalLen - 8 + i] = ((bitLen ushr (56 - i * 8)) and 0xff).toByte()
}
}
val words = IntArray(80)
var blockStart = 0
while (blockStart < padded.size) {
for (i in 0..15) {
val off = blockStart + i * 4
words[i] = ((padded[off].toInt() and 0xff) shl 24) or
((padded[off + 1].toInt() and 0xff) shl 16) or
((padded[off + 2].toInt() and 0xff) shl 8) or
(padded[off + 3].toInt() and 0xff)
}
for (i in 16..79) {
val mixed = words[i - 3] xor words[i - 8] xor words[i - 14] xor words[i - 16]
words[i] = (mixed shl 1) or (mixed ushr 31)
}
var a = h0
var b = h1
var c = h2
var d = h3
var e = h4
for (i in 0..19) {
val f = (b and c) or (b.inv() and d)
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0x5A827999 + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
for (i in 20..39) {
val f = b xor c xor d
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0x6ED9EBA1 + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
for (i in 40..59) {
val f = (b and c) or (b and d) or (c and d)
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0x8F1BBCDC.toInt() + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
for (i in 60..79) {
val f = b xor c xor d
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0xCA62C1D6.toInt() + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
h0 += a
h1 += b
h2 += c
h3 += d
h4 += e
blockStart += 64
}
return ByteArray(20).also { out ->
fun putInt(offset: Int, value: Int) {
out[offset] = (value ushr 24).toByte()
out[offset + 1] = (value ushr 16).toByte()
out[offset + 2] = (value ushr 8).toByte()
out[offset + 3] = value.toByte()
}
putInt(0, h0)
putInt(4, h1)
putInt(8, h2)
putInt(12, h3)
putInt(16, h4)
}
}
private const val WS_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
private const val OPCODE_CONTINUATION = 0x0
private const val OPCODE_TEXT = 0x1
private const val OPCODE_BINARY = 0x2
private const val OPCODE_CLOSE = 0x8
private const val OPCODE_PING = 0x9
private const val OPCODE_PONG = 0xA
@@ -0,0 +1,122 @@
package net.sergeych.lyngio.http.server
import net.sergeych.lyngio.net.LyngIpVersion
import net.sergeych.lyngio.net.LyngSocketAddress
import net.sergeych.lyngio.net.LyngTcpSocket
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFailsWith
class HttpParserTest {
@Test
fun tooLargeHeadersMapTo431() = kotlinx.coroutines.test.runTest {
val request = buildString {
append("GET / HTTP/1.1\r\n")
append("Host: localhost\r\n")
append("X-Big: ")
append("a".repeat(64))
append("\r\n\r\n")
}
val error = assertFailsWith<HttpProtocolException> {
parse(request, HttpServerConfig(maxHeaderBytes = 32))
}
assertEquals(431, error.status)
}
@Test
fun conflictingDuplicateHostIsRejected() = kotlinx.coroutines.test.runTest {
val error = assertFailsWith<HttpProtocolException> {
parse(
"GET / HTTP/1.1\r\n" +
"Host: one.example\r\n" +
"Host: two.example\r\n\r\n"
)
}
assertEquals(400, error.status)
}
@Test
fun conflictingDuplicateContentLengthIsRejected() = kotlinx.coroutines.test.runTest {
val error = assertFailsWith<HttpProtocolException> {
parse(
"POST /echo HTTP/1.1\r\n" +
"Host: localhost\r\n" +
"Content-Length: 4\r\n" +
"Content-Length: 5\r\n\r\nping!"
)
}
assertEquals(400, error.status)
}
@Test
fun malformedRequestLineIsRejected() = kotlinx.coroutines.test.runTest {
val error = assertFailsWith<HttpProtocolException> {
parse("GET /only-two-parts\r\nHost: localhost\r\n\r\n")
}
assertEquals(400, error.status)
}
@Test
fun identicalDuplicateContentLengthIsAccepted() = kotlinx.coroutines.test.runTest {
val request = parse(
"POST /echo HTTP/1.1\r\n" +
"Host: localhost\r\n" +
"Content-Length: 4\r\n" +
"Content-Length: 4\r\n\r\nping"
)
assertEquals("POST", request.head.method)
assertEquals("/echo", request.head.path)
assertEquals(4, request.head.contentLength)
assertEquals("ping", request.body.decodeToString())
}
private suspend fun parse(
rawRequest: String,
config: HttpServerConfig = HttpServerConfig(),
): HttpRequest {
val socket = FakeTcpSocket(rawRequest.encodeToByteArray())
val reader = BufferedSocketReader(socket)
return parseHttpRequest(reader, config) ?: error("expected parsed request")
}
}
private class FakeTcpSocket(
source: ByteArray,
) : LyngTcpSocket {
private var input = source
private var output = ByteArray(0)
private var open = true
override fun isOpen(): Boolean = open
override fun localAddress(): LyngSocketAddress =
LyngSocketAddress("127.0.0.1", 8080, LyngIpVersion.IPV4, resolved = true)
override fun remoteAddress(): LyngSocketAddress =
LyngSocketAddress("127.0.0.1", 12345, LyngIpVersion.IPV4, resolved = true)
override suspend fun read(maxBytes: Int): ByteArray? {
if (!open || input.isEmpty()) return null
val count = minOf(maxBytes, input.size)
val chunk = input.copyOfRange(0, count)
input = input.copyOfRange(count, input.size)
return chunk
}
override suspend fun readLine(): String? = error("BufferedSocketReader should not call LyngTcpSocket.readLine()")
override suspend fun write(data: ByteArray) {
output += data
}
override suspend fun writeUtf8(text: String) {
output += text.encodeToByteArray()
}
override suspend fun flush() = Unit
override fun close() {
open = false
}
}
@@ -0,0 +1,242 @@
package net.sergeych.lyngio.http.server
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeout
import net.sergeych.lyngio.net.LyngTcpSocket
import net.sergeych.lyngio.net.getSystemNetEngine
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
class HttpServerLoopbackTest {
@Test
fun simpleGetReturnsResponse() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable || !engine.isTcpServerAvailable) return@runBlocking
withTimeout(10_000) {
val server = startHttpServer { request ->
HttpHandlerResult.Response(
HttpResponse(
status = 200,
headers = listOf(HttpHeader("Content-Type", "text/plain")),
body = "hello:${request.head.path}".encodeToByteArray(),
)
)
}
try {
val port = waitForPort(server)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
client.writeUtf8("GET /demo HTTP/1.1\r\nHost: localhost\r\n\r\n")
client.flush()
val text = readHttpResponse(client)
assertTrue(text.startsWith("HTTP/1.1 200 OK\r\n"), text)
assertTrue(text.contains("Content-Type: text/plain\r\n"), text)
assertTrue(text.endsWith("hello:/demo"), text)
} finally {
client.close()
}
} finally {
server.close()
}
}
}
@Test
fun keepAliveServesTwoRequestsOnOneSocket() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable || !engine.isTcpServerAvailable) return@runBlocking
withTimeout(10_000) {
val server = startHttpServer { request ->
HttpHandlerResult.Response(
HttpResponse(status = 200, body = request.head.path.encodeToByteArray())
)
}
try {
val port = waitForPort(server)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
client.writeUtf8(
"GET /one HTTP/1.1\r\nHost: localhost\r\n\r\n" +
"GET /two HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n"
)
client.flush()
val first = readHttpResponse(client)
val second = readHttpResponse(client)
assertTrue(first.endsWith("/one"), first)
assertTrue(second.contains("Connection: close\r\n"), second)
assertTrue(second.endsWith("/two"), second)
} finally {
client.close()
}
} finally {
server.close()
}
}
}
@Test
fun postWithContentLengthReadsBody() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable || !engine.isTcpServerAvailable) return@runBlocking
withTimeout(10_000) {
val server = startHttpServer { request ->
HttpHandlerResult.Response(
HttpResponse(status = 200, body = (request.head.method + ":" + request.body.decodeToString()).encodeToByteArray())
)
}
try {
val port = waitForPort(server)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
client.writeUtf8(
"POST /echo HTTP/1.1\r\nHost: localhost\r\nContent-Length: 4\r\nConnection: close\r\n\r\nping"
)
client.flush()
val text = readHttpResponse(client)
assertTrue(text.endsWith("POST:ping"), text)
} finally {
client.close()
}
} finally {
server.close()
}
}
}
@Test
fun transferEncodingIsRejected() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable || !engine.isTcpServerAvailable) return@runBlocking
withTimeout(10_000) {
val server = startHttpServer { _ ->
HttpHandlerResult.Response(HttpResponse(status = 200, body = "ok".encodeToByteArray()))
}
try {
val port = waitForPort(server)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
client.writeUtf8(
"POST /x HTTP/1.1\r\nHost: localhost\r\nTransfer-Encoding: chunked\r\n\r\n"
)
client.flush()
val text = readHttpResponse(client)
assertTrue(text.startsWith("HTTP/1.1 501 Not Implemented\r\n"), text)
} finally {
client.close()
}
} finally {
server.close()
}
}
}
@Test
fun websocketUpgradeEchoesText() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable || !engine.isTcpServerAvailable) return@runBlocking
withTimeout(10_000) {
val server = startHttpServer { request ->
if (request.head.path != "/ws") {
HttpHandlerResult.Response(HttpResponse(status = 404, close = true))
} else {
HttpHandlerResult.WebSocket { session ->
val message = session.receive() ?: return@WebSocket
session.sendText("echo:${message.text}")
}
}
}
try {
val port = waitForPort(server)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
val key = "dGhlIHNhbXBsZSBub25jZQ=="
client.writeUtf8(
"GET /ws HTTP/1.1\r\n" +
"Host: localhost\r\n" +
"Upgrade: websocket\r\n" +
"Connection: Upgrade\r\n" +
"Sec-WebSocket-Key: $key\r\n" +
"Sec-WebSocket-Version: 13\r\n\r\n"
)
client.flush()
val headers = ArrayList<String>()
while (true) {
val line = client.readLine() ?: break
if (line.isEmpty()) break
headers += line
}
assertEquals("HTTP/1.1 101 Switching Protocols", headers.first())
sendMaskedTextFrame(client, "ping")
val reply = readServerTextFrame(client)
assertEquals("echo:ping", reply)
} finally {
client.close()
}
} finally {
server.close()
}
}
}
private suspend fun waitForPort(server: HttpServer): Int {
repeat(100) {
runCatching { return server.localAddress().port }
kotlinx.coroutines.delay(10)
}
error("server did not bind in time")
}
private suspend fun readHttpResponse(client: LyngTcpSocket): String {
val statusLine = client.readLine() ?: error("missing status line")
val headers = linkedMapOf<String, String>()
while (true) {
val line = client.readLine() ?: error("unexpected EOF in response headers")
if (line.isEmpty()) break
val colonAt = line.indexOf(':')
if (colonAt > 0) headers[line.substring(0, colonAt)] = line.substring(colonAt + 1).trim()
}
val bodyLength = headers["Content-Length"]?.toIntOrNull() ?: 0
val body = if (bodyLength > 0) readExact(client, bodyLength).decodeToString() else ""
return buildString {
append(statusLine).append("\r\n")
headers.forEach { (name, value) -> append(name).append(": ").append(value).append("\r\n") }
append("\r\n")
append(body)
}
}
private suspend fun sendMaskedTextFrame(client: LyngTcpSocket, text: String) {
val payload = text.encodeToByteArray()
val mask = byteArrayOf(1, 2, 3, 4)
val masked = payload.copyOf()
masked.indices.forEach { index ->
masked[index] = (masked[index].toInt() xor mask[index % mask.size].toInt()).toByte()
}
val frame = byteArrayOf(0x81.toByte(), (0x80 or payload.size).toByte()) + mask + masked
client.write(frame)
client.flush()
}
private suspend fun readServerTextFrame(client: LyngTcpSocket): String {
val head = readExact(client, 2)
val len = head[1].toInt() and 0x7f
val payload = if (len > 0) readExact(client, len) else ByteArray(0)
return payload.decodeToString()
}
private suspend fun readExact(client: LyngTcpSocket, size: Int): ByteArray {
var pending = ByteArray(0)
while (pending.size < size) {
val chunk = client.read(size - pending.size) ?: error("unexpected EOF")
pending += chunk
}
return pending
}
}
@@ -137,6 +137,433 @@ class LyngSqliteModuleTest {
assertEquals(2L, result.value)
}
@Test
fun testTransactionGenericReturnTypeFlowsToOuterVal() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db
import lyng.io.db.sqlite
class Payload(name: String, count: Int)
class Item(id: Int, title: String, @DbJson meta: Payload, @DbLynon state: Payload) {
var note: String = ""
}
val restored = openSqlite(":memory:").transaction { tx ->
tx.execute("create table item(id integer not null, title text not null, meta text not null, state blob not null, note text not null)")
val item = Item(1, "first", Payload("json", 10), Payload("bin", 20))
item.note = "created"
tx.execute("insert into item(@cols(?1)) values(@vals(?1))", item)
item.title = "second"
item.meta = Payload("json2", 11)
item.state = Payload("bin2", 21)
item.note = "updated"
tx.execute("update item set @set(?1 except: \"id\") where id = ?2", item, item.id)
val restored = tx.select("select * from item where id = ?", 1).decodeAs<Item>().first
assertEquals("second", restored.title)
assertEquals("json2", restored.meta.name)
assertEquals(11, restored.meta.count)
assertEquals("bin2", restored.state.name)
assertEquals(21, restored.state.count)
assertEquals("updated", restored.note)
restored
}
restored.id
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-transaction-return-inference>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(1L, result.value)
}
@Test
fun testDecodeAsProjectsJsonColumnIntoObjectField() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Point(x: Int, y: Int)
class Row(id: Int, payload: Point)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(id integer not null, payload json not null)")
tx.execute("insert into data(id, payload) values(?, ?)", 7, "{\"x\":4,\"y\":5}")
val row = tx.select("select id, payload from data").decodeAs<Row>().first
assertEquals(7, row.id)
assertEquals(4, row.payload.x)
assertEquals(5, row.payload.y)
row.payload.y
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-decode-json-field>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(5L, result.value)
}
@Test
fun testDecodeAsSupportsSingleJsonColumnProjection() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Point(x: Int, y: Int)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(payload json not null)")
tx.execute("insert into data(payload) values(?)", "{\"x\":9,\"y\":11}")
val point = tx.select("select payload from data").decodeAs<Point>().first
assertEquals(9, point.x)
assertEquals(11, point.y)
point.x + point.y
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-decode-json-single>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(20L, result.value)
}
@Test
fun testDecodeAsDoesNotAutoDecodePlainTextAsJson() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Point(x: Int, y: Int)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(payload text not null)")
tx.execute("insert into data(payload) values(?)", "{\"x\":1,\"y\":2}")
tx.select("select payload from data").decodeAs<Point>().first
}
""".trimIndent()
val error = assertFailsWith<ExecutionError> {
Compiler.compile(Source("<sqlite-decode-json-text-guard>", code), scope.importManager).execute(scope)
}
assertEquals("SqlUsageException", error.errorObject.objClass.className)
}
@Test
fun testDecodeAsSupportsSingleLynonBinaryProjection() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
import lyng.serialization
class Point(x: Int, y: Int)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(payload blob not null)")
tx.execute("insert into data(payload) values(?)", Lynon.encode(Point(6, 8)).toBuffer())
val point = tx.select("select payload from data").decodeAs<Point>().first
assertEquals(6, point.x)
assertEquals(8, point.y)
point.x + point.y
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-decode-lynon-single>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(14L, result.value)
}
@Test
fun testDecodeAsSupportsDbDecodeWithOnConstructorParamsAndFields() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db
import lyng.io.db.sqlite
object TrimmedStringAdapter: DbFieldAdapter {
override fun decode(rawValue, column, row, targetType) =
when(rawValue) {
null -> null
else -> rawValue.toString().trim()
}
}
class User(
id: Int,
@DbDecodeWith(TrimmedStringAdapter) name: String
) {
@DbDecodeWith(TrimmedStringAdapter)
var note: String = ""
}
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(id integer not null, name text not null, note text not null)")
tx.execute("insert into data(id, name, note) values(?, ?, ?)", 10, " Alice ", " hello ")
val user = tx.select("select id, name, note from data").decodeAs<User>().first
assertEquals(10, user.id)
assertEquals("Alice", user.name)
assertEquals("hello", user.note)
user.note.size
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-decode-dbdecodewith>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(5L, result.value)
}
@Test
fun testDecodeAsFailsWhenDbDecodeWithReturnsWrongType() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db
import lyng.io.db.sqlite
object BadAdapter: DbFieldAdapter {
override fun decode(rawValue, column, row, targetType) = 42
}
class User(@DbDecodeWith(BadAdapter) name: String)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(name text not null)")
tx.execute("insert into data(name) values(?)", "Alice")
tx.select("select name from data").decodeAs<User>().first
}
""".trimIndent()
val error = assertFailsWith<ExecutionError> {
Compiler.compile(Source("<sqlite-decode-dbdecodewith-bad-type>", code), scope.importManager).execute(scope)
}
assertEquals("SqlUsageException", error.errorObject.objClass.className)
}
@Test
fun testDecodeAsKeepsRawBufferForBufferTarget() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
import lyng.buffer
import lyng.serialization
class Point(x: Int, y: Int)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(payload blob not null)")
val encoded = Lynon.encode(Point(1, 2)).toBuffer()
tx.execute("insert into data(payload) values(?)", encoded)
val payload = tx.select("select payload from data").decodeAs<Buffer>().first
assertEquals(encoded.size, payload.size)
payload.size
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-decode-buffer-raw>", code), scope.importManager).execute(scope) as ObjInt
assertTrue(result.value > 0)
}
@Test
fun testDecodeAsFailsForNonLynonBinaryTypedProjection() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
import lyng.buffer
class Point(x: Int, y: Int)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(payload blob not null)")
tx.execute("insert into data(payload) values(?)", "hello".encodeUtf8())
tx.select("select payload from data").decodeAs<Point>().first
}
""".trimIndent()
val error = assertFailsWith<ExecutionError> {
Compiler.compile(Source("<sqlite-decode-lynon-binary-guard>", code), scope.importManager).execute(scope)
}
assertEquals("SqlUsageException", error.errorObject.objClass.className)
}
@Test
fun testSqlObjectExpansionInsertAndUpdateUseDbAnnotations() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Payload(name: String, count: Int)
class Item(
id: Int,
title: String,
@DbJson meta: Payload,
@DbLynon state: Payload
) {
var note: String = ""
@DbExcept var cache: String = "skip"
@Transient var transientNote: String = "temp"
}
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table item(id integer not null, title text not null, meta text not null, state blob not null, note text not null)")
val item = Item(1, "first", Payload("json", 10), Payload("bin", 20))
item.note = "created"
tx.execute("insert into item(@cols(?1)) values(@vals(?1))", item)
item.title = "second"
item.meta = Payload("json2", 11)
item.state = Payload("bin2", 21)
item.note = "updated"
item.cache = "must-not-be-written"
item.transientNote = "must-not-be-written"
tx.execute("update item set @set(?1) where id = ?2", item, 1)
val restored = tx.select("select id, title, meta, state, note from item").decodeAs<Item>().first
assertEquals(1, restored.id)
assertEquals("second", restored.title)
assertEquals("json2", restored.meta.name)
assertEquals(11, restored.meta.count)
assertEquals("bin2", restored.state.name)
assertEquals(21, restored.state.count)
assertEquals("updated", restored.note)
restored.state.count
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-object-expansion>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(21L, result.value)
}
@Test
fun testSqlObjectExpansionSupportsDbSerializeWithAdapter() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db
import lyng.io.db.sqlite
object TrimAdapter: DbFieldAdapter {
override fun encode(value, targetType) =
when(value) {
null -> null
else -> value.toString().trim()
}
}
class User(
id: Int,
@DbSerializeWith(TrimAdapter) name: String
)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table user(id integer not null, name text not null)")
tx.execute("insert into user(@cols(?1)) values(@vals(?1))", User(7, " Alice "))
val row = tx.select("select id, name from user").first
val storedName: String = row["name"] as String
assertEquals(7, row["id"])
assertEquals("Alice", storedName)
storedName.size
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-object-expansion-adapter>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(5L, result.value)
}
@Test
fun testSqlObjectExpansionSupportsClauseLevelExceptFilter() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Item(id: Int, title: String, note: String) {
var stamp: String = ""
@DbExcept var cache: String = "skip"
}
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table item(id integer not null, title text not null, note text not null, stamp text not null default '')")
val item = Item(1, "first", "keep")
item.stamp = "created"
tx.execute(
"insert into item(@cols(?1 except: \"stamp\")) values(@vals(?1 except: \"stamp\"))",
item
)
item.title = "second"
item.note = "changed-but-excluded"
item.stamp = "updated"
item.cache = "still-skip"
tx.execute(
"update item set @set(?1 except: \"id\", \"note\") where id = ?2",
item,
1
)
val restored = tx.select("select id, title, note, stamp from item").decodeAs<Item>().first
assertEquals(1, restored.id)
assertEquals("second", restored.title)
assertEquals("keep", restored.note)
assertEquals("updated", restored.stamp)
restored.stamp.size
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-object-expansion-except>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(7L, result.value)
}
@Test
fun testSqlObjectExpansionRejectsUnannotatedComplexFields() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Payload(name: String)
class BadRecord(id: Int, payload: Payload)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table bad_record(id integer not null, payload text not null)")
tx.execute("insert into bad_record(@cols(?1)) values(@vals(?1))", BadRecord(1, Payload("x")))
}
""".trimIndent()
val error = assertFailsWith<ExecutionError> {
Compiler.compile(Source("<sqlite-object-expansion-bad-field>", code), scope.importManager).execute(scope)
}
assertEquals("SqlUsageException", error.errorObject.objClass.className)
}
@Test
fun testNestedTransactionRollbackUsesSavepoint() = runTest {
val scope = Script.newScope()
@@ -0,0 +1,131 @@
package net.sergeych.lyng.io.http.server
import kotlinx.coroutines.delay
import kotlinx.coroutines.runBlocking
import net.sergeych.lyng.Compiler
import net.sergeych.lyng.Pos
import net.sergeych.lyng.Script
import net.sergeych.lyng.io.http.server.createHttpServerModule
import net.sergeych.lyng.io.ws.createWsModule
import net.sergeych.lyng.obj.Obj
import net.sergeych.lyng.obj.ObjInt
import net.sergeych.lyng.io.http.createHttpModule
import net.sergeych.lyngio.net.getSystemNetEngine
import net.sergeych.lyngio.http.security.PermitAllHttpAccessPolicy
import net.sergeych.lyngio.net.security.PermitAllNetAccessPolicy
import net.sergeych.lyngio.ws.security.PermitAllWsAccessPolicy
import kotlin.test.Test
import kotlin.test.assertSame
import kotlin.test.assertTrue
class LyngHttpServerModuleTest {
@Test
fun serverModuleReusesSharedHttpHeadersRuntimeType() = runBlocking {
val scope = Script.newScope()
createHttpModule(PermitAllHttpAccessPolicy, scope)
createWsModule(PermitAllWsAccessPolicy, scope)
createHttpServerModule(PermitAllNetAccessPolicy, scope)
val httpModule = scope.importManager.createModuleScope(Pos.builtIn, "lyng.io.http")
val wsModule = scope.importManager.createModuleScope(Pos.builtIn, "lyng.io.ws")
val serverModule = scope.importManager.createModuleScope(Pos.builtIn, "lyng.io.http.server")
val sharedTypesModule = scope.importManager.createModuleScope(Pos.builtIn, "lyng.io.http.types")
val sharedWsTypesModule = scope.importManager.createModuleScope(Pos.builtIn, "lyng.io.ws.types")
assertSame(sharedTypesModule.get("HttpHeaders")?.value, httpModule.get("HttpHeaders")?.value)
assertSame(sharedTypesModule.get("HttpHeaders")?.value, serverModule.get("HttpHeaders")?.value)
assertSame(sharedWsTypesModule.get("WsMessage")?.value, wsModule.get("WsMessage")?.value)
assertSame(sharedWsTypesModule.get("WsMessage")?.value, serverModule.get("WsMessage")?.value)
}
@Test
fun exactRouteAndFallbackWork() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable) return@runBlocking
val scope = Script.newScope()
createHttpServerModule(PermitAllNetAccessPolicy, scope)
val code = """
import lyng.io.http.server
val server = HttpServer()
server.get("/hello") { ex ->
ex.setHeader("Content-Type", "text/plain")
ex.respondText(200, "hello from lyng")
}
server.fallback { ex ->
ex.respondText(404, "miss:" + ex.request.path)
}
server.listen(0, "127.0.0.1")
""".trimIndent()
val handle = Compiler.compile(code).execute(scope)
val port = waitForPort(handle, scope)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
client.writeUtf8("GET /hello HTTP/1.1\r\nHost: localhost\r\n\r\n")
client.flush()
val hello = readHttpResponse(client)
assertTrue(hello.contains("200 OK"), hello)
assertTrue(hello.endsWith("hello from lyng"), hello)
} finally {
client.close()
}
val client2 = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
client2.writeUtf8("GET /other HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n")
client2.flush()
val miss = readHttpResponse(client2)
assertTrue(miss.contains("404"), miss)
assertTrue(miss.endsWith("miss:/other"), miss)
} finally {
client2.close()
}
handle.invokeInstanceMethod(scope, "close")
}
private suspend fun waitForPort(handle: Obj, scope: net.sergeych.lyng.Scope): Int {
repeat(100) {
val port = runCatching {
val value = handle.invokeInstanceMethod(scope, "localPort")
(value as ObjInt).value.toInt()
}.getOrNull()
if (port != null && port > 0) return port
delay(10)
}
error("server did not bind in time")
}
private suspend fun readHttpResponse(client: net.sergeych.lyngio.net.LyngTcpSocket): String {
val statusLine = client.readLine() ?: error("missing status line")
val headers = linkedMapOf<String, String>()
while (true) {
val line = client.readLine() ?: error("unexpected EOF in response headers")
if (line.isEmpty()) break
val colonAt = line.indexOf(':')
if (colonAt > 0) headers[line.substring(0, colonAt)] = line.substring(colonAt + 1).trim()
}
val bodyLength = headers["Content-Length"]?.toIntOrNull() ?: 0
val body = if (bodyLength > 0) readExact(client, bodyLength).decodeToString() else ""
return buildString {
append(statusLine).append("\r\n")
headers.forEach { (name, value) -> append(name).append(": ").append(value).append("\r\n") }
append("\r\n")
append(body)
}
}
private suspend fun readExact(client: net.sergeych.lyngio.net.LyngTcpSocket, size: Int): ByteArray {
var pending = ByteArray(0)
while (pending.size < size) {
val chunk = client.read(size - pending.size) ?: error("unexpected EOF")
pending += chunk
}
return pending
}
}
@@ -22,6 +22,7 @@ import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withContext
import net.sergeych.lyng.Compiler
import net.sergeych.lyng.ExecutionError
import net.sergeych.lyng.Pos
import net.sergeych.lyng.Script
import net.sergeych.lyngio.fs.security.AccessContext
import net.sergeych.lyngio.fs.security.AccessDecision
@@ -35,11 +36,25 @@ import java.net.ServerSocket
import java.net.Socket
import kotlin.concurrent.thread
import kotlin.test.Test
import kotlin.test.assertSame
import kotlin.test.assertFailsWith
import kotlin.test.assertTrue
class LyngNetModuleTest {
@Test
fun testSharedNetTypesModuleExportsCanonicalTypes() = runBlocking {
val scope = Script.newScope()
createNetModule(PermitAllNetAccessPolicy, scope)
val netModule = scope.importManager.createModuleScope(Pos.builtIn, "lyng.io.net")
val typesModule = scope.importManager.createModuleScope(Pos.builtIn, "lyng.io.net.types")
assertSame(typesModule.get("IpVersion")?.value, netModule.get("IpVersion")?.value)
assertSame(typesModule.get("SocketAddress")?.value, netModule.get("SocketAddress")?.value)
assertSame(typesModule.get("Datagram")?.value, netModule.get("Datagram")?.value)
}
@Test
fun testResolveAndCapabilities() = runBlocking {
val scope = Script.newScope()
@@ -19,11 +19,13 @@ package net.sergeych.lyng.io.db.sqlite
import kotlinx.coroutines.test.runTest
import kotlinx.datetime.TimeZone
import net.sergeych.lyng.Compiler
import net.sergeych.lyng.ExecutionError
import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Pos
import net.sergeych.lyng.Scope
import net.sergeych.lyng.Script
import net.sergeych.lyng.Source
import net.sergeych.lyng.obj.Obj
import net.sergeych.lyng.obj.ObjBool
import net.sergeych.lyng.obj.ObjBuffer
@@ -231,6 +233,392 @@ class LyngSqliteModuleNativeTest {
assertEquals("beta", stringValue(scope, rows.getAt(scope, ObjInt.of(1)).getAt(scope, ObjString("name"))))
}
@Test
fun testDecodeAsProjectsJsonColumnIntoObjectField() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Point(x: Int, y: Int)
class Row(id: Int, payload: Point)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(id integer not null, payload json not null)")
tx.execute("insert into data(id, payload) values(?, ?)", 7, "{\"x\":4,\"y\":5}")
val row = tx.select("select id, payload from data").decodeAs<Row>().first
assertEquals(7, row.id)
assertEquals(4, row.payload.x)
assertEquals(5, row.payload.y)
row.payload.y
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-native-decode-json-field>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(5L, result.value)
}
@Test
fun testDecodeAsSupportsSingleJsonColumnProjection() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Point(x: Int, y: Int)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(payload json not null)")
tx.execute("insert into data(payload) values(?)", "{\"x\":9,\"y\":11}")
val point = tx.select("select payload from data").decodeAs<Point>().first
assertEquals(9, point.x)
assertEquals(11, point.y)
point.x + point.y
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-native-decode-json-single>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(20L, result.value)
}
@Test
fun testDecodeAsDoesNotAutoDecodePlainTextAsJson() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Point(x: Int, y: Int)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(payload text not null)")
tx.execute("insert into data(payload) values(?)", "{\"x\":1,\"y\":2}")
tx.select("select payload from data").decodeAs<Point>().first
}
""".trimIndent()
val error = assertFailsWith<ExecutionError> {
Compiler.compile(Source("<sqlite-native-decode-json-text-guard>", code), scope.importManager).execute(scope)
}
assertEquals("SqlUsageException", error.errorObject.objClass.className)
}
@Test
fun testDecodeAsSupportsSingleLynonBinaryProjection() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
import lyng.serialization
class Point(x: Int, y: Int)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(payload blob not null)")
tx.execute("insert into data(payload) values(?)", Lynon.encode(Point(6, 8)).toBuffer())
val point = tx.select("select payload from data").decodeAs<Point>().first
assertEquals(6, point.x)
assertEquals(8, point.y)
point.x + point.y
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-native-decode-lynon-single>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(14L, result.value)
}
@Test
fun testDecodeAsSupportsDbDecodeWithOnConstructorParamsAndFields() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db
import lyng.io.db.sqlite
object TrimmedStringAdapter: DbFieldAdapter {
override fun decode(rawValue, column, row, targetType) =
when(rawValue) {
null -> null
else -> rawValue.toString().trim()
}
}
class User(
id: Int,
@DbDecodeWith(TrimmedStringAdapter) name: String
) {
@DbDecodeWith(TrimmedStringAdapter)
var note: String = ""
}
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(id integer not null, name text not null, note text not null)")
tx.execute("insert into data(id, name, note) values(?, ?, ?)", 10, " Alice ", " hello ")
val user = tx.select("select id, name, note from data").decodeAs<User>().first
assertEquals(10, user.id)
assertEquals("Alice", user.name)
assertEquals("hello", user.note)
user.note.size
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-native-decode-dbdecodewith>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(5L, result.value)
}
@Test
fun testDecodeAsFailsWhenDbDecodeWithReturnsWrongType() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db
import lyng.io.db.sqlite
object BadAdapter: DbFieldAdapter {
override fun decode(rawValue, column, row, targetType) = 42
}
class User(@DbDecodeWith(BadAdapter) name: String)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(name text not null)")
tx.execute("insert into data(name) values(?)", "Alice")
tx.select("select name from data").decodeAs<User>().first
}
""".trimIndent()
val error = assertFailsWith<ExecutionError> {
Compiler.compile(Source("<sqlite-native-decode-dbdecodewith-bad-type>", code), scope.importManager).execute(scope)
}
assertEquals("SqlUsageException", error.errorObject.objClass.className)
}
@Test
fun testDecodeAsKeepsRawBufferForBufferTarget() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
import lyng.buffer
import lyng.serialization
class Point(x: Int, y: Int)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(payload blob not null)")
val encoded = Lynon.encode(Point(1, 2)).toBuffer()
tx.execute("insert into data(payload) values(?)", encoded)
val payload = tx.select("select payload from data").decodeAs<Buffer>().first
assertEquals(encoded.size, payload.size)
payload.size
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-native-decode-buffer-raw>", code), scope.importManager).execute(scope) as ObjInt
assertTrue(result.value > 0)
}
@Test
fun testDecodeAsFailsForNonLynonBinaryTypedProjection() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
import lyng.buffer
class Point(x: Int, y: Int)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table data(payload blob not null)")
tx.execute("insert into data(payload) values(?)", "hello".encodeUtf8())
tx.select("select payload from data").decodeAs<Point>().first
}
""".trimIndent()
val error = assertFailsWith<ExecutionError> {
Compiler.compile(Source("<sqlite-native-decode-lynon-binary-guard>", code), scope.importManager).execute(scope)
}
assertEquals("SqlUsageException", error.errorObject.objClass.className)
}
@Test
fun testSqlObjectExpansionInsertAndUpdateUseDbAnnotations() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Payload(name: String, count: Int)
class Item(
id: Int,
title: String,
@DbJson meta: Payload,
@DbLynon state: Payload
) {
var note: String = ""
@DbExcept var cache: String = "skip"
@Transient var transientNote: String = "temp"
}
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table item(id integer not null, title text not null, meta text not null, state blob not null, note text not null)")
val item = Item(1, "first", Payload("json", 10), Payload("bin", 20))
item.note = "created"
tx.execute("insert into item(@cols(?1)) values(@vals(?1))", item)
item.title = "second"
item.meta = Payload("json2", 11)
item.state = Payload("bin2", 21)
item.note = "updated"
item.cache = "must-not-be-written"
item.transientNote = "must-not-be-written"
tx.execute("update item set @set(?1) where id = ?2", item, 1)
val restored = tx.select("select id, title, meta, state, note from item").decodeAs<Item>().first
assertEquals(1, restored.id)
assertEquals("second", restored.title)
assertEquals("json2", restored.meta.name)
assertEquals(11, restored.meta.count)
assertEquals("bin2", restored.state.name)
assertEquals(21, restored.state.count)
assertEquals("updated", restored.note)
restored.state.count
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-native-object-expansion>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(21L, result.value)
}
@Test
fun testSqlObjectExpansionSupportsDbSerializeWithAdapter() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db
import lyng.io.db.sqlite
object TrimAdapter: DbFieldAdapter {
override fun encode(value, targetType) =
when(value) {
null -> null
else -> value.toString().trim()
}
}
class User(
id: Int,
@DbSerializeWith(TrimAdapter) name: String
)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table user(id integer not null, name text not null)")
tx.execute("insert into user(@cols(?1)) values(@vals(?1))", User(7, " Alice "))
val row = tx.select("select id, name from user").first
val storedName: String = row["name"] as String
assertEquals(7, row["id"])
assertEquals("Alice", storedName)
storedName.size
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-native-object-expansion-adapter>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(5L, result.value)
}
@Test
fun testSqlObjectExpansionSupportsClauseLevelExceptFilter() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Item(id: Int, title: String, note: String) {
var stamp: String = ""
@DbExcept var cache: String = "skip"
}
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table item(id integer not null, title text not null, note text not null, stamp text not null default '')")
val item = Item(1, "first", "keep")
item.stamp = "created"
tx.execute(
"insert into item(@cols(?1 except: \"stamp\")) values(@vals(?1 except: \"stamp\"))",
item
)
item.title = "second"
item.note = "changed-but-excluded"
item.stamp = "updated"
item.cache = "still-skip"
tx.execute(
"update item set @set(?1 except: \"id\", \"note\") where id = ?2",
item,
1
)
val restored = tx.select("select id, title, note, stamp from item").decodeAs<Item>().first
assertEquals(1, restored.id)
assertEquals("second", restored.title)
assertEquals("keep", restored.note)
assertEquals("updated", restored.stamp)
restored.stamp.size
}
""".trimIndent()
val result = Compiler.compile(Source("<sqlite-native-object-expansion-except>", code), scope.importManager).execute(scope) as ObjInt
assertEquals(7L, result.value)
}
@Test
fun testSqlObjectExpansionRejectsUnannotatedComplexFields() = runTest {
val scope = Script.newScope()
createSqliteModule(scope.importManager)
val code = """
import lyng.io.db.sqlite
class Payload(name: String)
class BadRecord(id: Int, payload: Payload)
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table bad_record(id integer not null, payload text not null)")
tx.execute("insert into bad_record(@cols(?1)) values(@vals(?1))", BadRecord(1, Payload("x")))
}
""".trimIndent()
val error = assertFailsWith<ExecutionError> {
Compiler.compile(Source("<sqlite-native-object-expansion-bad-field>", code), scope.importManager).execute(scope)
}
assertEquals("SqlUsageException", error.errorObject.objClass.className)
}
@Test
fun testExecuteRejectsReturningButSelectSupportsIt() = runTest {
val scope = Script.newScope()
+81
View File
@@ -20,6 +20,38 @@ extern class SqlColumn {
val nativeType: String
}
/*
Adapter interface for custom DB field projection.
Use it with `@DbDecodeWith(adapter)` on class constructor parameters or
class-body fields/properties participating in `decodeAs<T>()` projection,
and with `@DbSerializeWith(adapter)` on fields participating in SQL object
expansion macros such as `@vals(?1)` and `@set(?1)`.
`targetType` is the requested Lyng target type represented as a runtime
type object, such as a class or a type expression.
Default methods throw `NotImplementedException`.
*/
interface DbFieldAdapter {
/*
Decode one raw database field value into a Lyng value suitable for the
requested target type.
*/
fun decode(rawValue: Object?, column: SqlColumn, row: SqlRow, targetType: Object): Object? =
throw NotImplementedException("DB field adapter decode is not implemented")
/*
Encode one Lyng value into a database field representation.
The result must be a direct DB-bindable value:
null, Bool, Int, Double/Real, Decimal, String, Buffer, Date, DateTime,
or Instant.
*/
fun encode(value: Object?, targetType: Object): Object? =
throw NotImplementedException("DB field adapter encode is not implemented")
}
extern class SqlRow {
/* Number of columns in the row */
val size: Int
@@ -34,6 +66,22 @@ extern class SqlRow {
names and invalid indexes should also fail.
*/
override fun getAt(indexOrName: String | Int): Object?
/*
Decode this row into a typed Lyng value.
For object/class targets, constructor parameters are matched by column
label first and then remaining matching serializable mutable fields are
assigned.
If a constructor parameter or class-body field/property has
`@DbDecodeWith(adapter)`, the adapter is applied first and its result
must match the target member type.
For single-column rows, the column value may also be decoded directly to
the requested target type.
*/
fun decodeAs<T>(): T
}
/*
@@ -69,6 +117,16 @@ extern class ResultSet : Iterable<SqlRow> {
internally, but this must not change visible later iteration behavior.
*/
override fun isEmpty(): Bool
/*
Return a transaction-scoped iterable view that decodes each row with
`SqlRow.decodeAs<T>()`.
The returned iterable itself must not be used after the owning
transaction ends. Materialized decoded objects may outlive the
transaction.
*/
fun decodeAs<T>(): Iterable<T>
}
extern class ExecutionResult {
@@ -133,6 +191,29 @@ extern class SqlTransaction {
- Date, DateTime, Instant
Unsupported parameter values should fail with `SqlUsageException`.
SQL object expansion macros are also supported:
- `@cols(?1)` expands one object argument to a comma-separated column list
- `@vals(?1)` expands the same object to matching `?` placeholders and
generated bind values
- `@set(?1)` expands to `col = ?` pairs and generated bind values
- each macro also accepts an optional `except:` filter, for example
`@set(?1 except: "id", "updatedAt")`
When a clause uses `@cols`, `@vals`, or `@set`, any non-expanded scalar
parameters in the same clause must use explicit indexed placeholders such
as `?2`, `?3`, and so on.
Projection is declaration-driven:
- `@Transient` fields are excluded
- `@DbExcept` fields are excluded
- `except:` excludes additional fields for that specific macro use
- `@DbJson` fields are encoded as JSON text
- `@DbLynon` fields are encoded as Lynon binary
- `@DbSerializeWith(adapter)` fields are encoded through the adapter
- unannotated non-primitive fields fail with `SqlUsageException`
*/
fun select(clause: String, params...): ResultSet
+1 -12
View File
@@ -1,17 +1,6 @@
package lyng.io.http
/*
Response/header view that behaves like a map for the first value of each header name.
Multi-valued headers are exposed through `getAll`.
*/
extern class HttpHeaders : Map<String, String> {
/* Return the first value for the given header name, or null when absent. */
fun get(name: String): String?
/* Return all values for the given header name, preserving wire order when available. */
fun getAll(name: String): List<String>
/* Return distinct header names present in this response. */
fun names(): List<String>
}
import lyng.io.http.types
/* Mutable request descriptor for programmatic HTTP calls. */
extern class HttpRequest {
+54
View File
@@ -0,0 +1,54 @@
package lyng.io.http.server
import lyng.io.http.types
import lyng.io.ws.types
/* Immutable parsed incoming server request. */
extern class ServerRequest {
val method: String
val target: String
val path: String
val query: String?
val headers: HttpHeaders
val body: Buffer
fun text(): String
fun isWebSocketUpgrade(): Bool
}
/* Active server-side WebSocket session. */
extern class ServerWebSocket {
fun isOpen(): Bool
fun sendText(text: String): void
fun sendBytes(data: Buffer): void
fun receive(): WsMessage?
fun close(code: Int = 1000, reason: String = ""): void
}
/* Mutable exchange object for one incoming request. */
extern class ServerExchange {
val request: ServerRequest
fun respond(status: Int = 200, body: Buffer? = null): void
fun respondText(status: Int = 200, bodyText: String = ""): void
fun setHeader(name: String, value: String): void
fun addHeader(name: String, value: String): void
fun acceptWebSocket(handler: (ServerWebSocket, ServerExchange) -> Object?): void
fun isHandled(): Bool
}
/* Running listener handle. */
extern class HttpServerHandle {
fun localPort(): Int
fun close(): void
}
/* Exact-path HTTP/WebSocket server with built-in router. */
extern class HttpServer {
fun get(path: String, handler: (ServerExchange) -> Object?): HttpServer
fun post(path: String, handler: (ServerExchange) -> Object?): HttpServer
fun put(path: String, handler: (ServerExchange) -> Object?): HttpServer
fun delete(path: String, handler: (ServerExchange) -> Object?): HttpServer
fun any(path: String, handler: (ServerExchange) -> Object?): HttpServer
fun ws(path: String, handler: (ServerWebSocket, ServerExchange) -> Object?): HttpServer
fun fallback(handler: (ServerExchange) -> Object?): HttpServer
fun listen(port: Int, host: String? = null, backlog: Int = 128): HttpServerHandle
}
+14
View File
@@ -0,0 +1,14 @@
package lyng.io.http.types
/*
Response/header view that behaves like a map for the first value of each header name.
Multi-valued headers are exposed through `getAll`.
*/
extern class HttpHeaders : Map<String, String> {
/* Return the first value for the given header name, or null when absent. */
fun get(name: String): String?
/* Return all values for the given header name, preserving wire order when available. */
fun getAll(name: String): List<String>
/* Return distinct header names present in this response. */
fun names(): List<String>
}
+1 -25
View File
@@ -1,30 +1,6 @@
package lyng.io.net
/* Address family for resolved or bound endpoints. */
enum IpVersion {
IPV4,
IPV6
}
/* Concrete socket endpoint. */
extern class SocketAddress {
/* Numeric or host-form address string. */
val host: String
/* Transport port number. */
val port: Int
/* Address family. */
val ipVersion: IpVersion
/* True when obtained from DNS resolution rather than raw bind input. */
val resolved: Bool
/* Stable printable form such as `127.0.0.1:4040` or `[::1]:4040`. */
override fun toString(): String
}
/* Datagram payload paired with sender/peer address. */
extern class Datagram {
val data: Buffer
val address: SocketAddress
}
import lyng.io.net.types
/* Connected TCP socket. */
extern class TcpSocket {
+27
View File
@@ -0,0 +1,27 @@
package lyng.io.net.types
/* Address family for resolved or bound endpoints. */
enum IpVersion {
IPV4,
IPV6
}
/* Concrete socket endpoint. */
extern class SocketAddress {
/* Numeric or host-form address string. */
val host: String
/* Transport port number. */
val port: Int
/* Address family. */
val ipVersion: IpVersion
/* True when obtained from DNS resolution rather than raw bind input. */
val resolved: Bool
/* Stable printable form such as `127.0.0.1:4040` or `[::1]:4040`. */
override fun toString(): String
}
/* Datagram payload paired with sender/peer address. */
extern class Datagram {
val data: Buffer
val address: SocketAddress
}
+1 -9
View File
@@ -1,14 +1,6 @@
package lyng.io.ws
/* Received WebSocket message. */
extern class WsMessage {
/* True when this message carries text payload. */
val isText: Bool
/* Text payload for text messages, otherwise null. */
val text: String?
/* Binary payload for binary messages, otherwise null. */
val data: Buffer?
}
import lyng.io.ws.types
/* Active WebSocket client session. */
extern class WsSession {
+11
View File
@@ -0,0 +1,11 @@
package lyng.io.ws.types
/* Received WebSocket message. */
extern class WsMessage {
/* True when this message carries text payload. */
val isText: Bool
/* Text payload for text messages, otherwise null. */
val text: String?
/* Binary payload for binary messages, otherwise null. */
val data: Buffer?
}
+1 -1
View File
@@ -21,7 +21,7 @@ import org.jetbrains.kotlin.gradle.ExperimentalWasmDsl
import org.jetbrains.kotlin.gradle.dsl.JvmTarget
group = "net.sergeych"
version = "1.5.5"
version = "1.5.6-SNAPSHOT"
// Removed legacy buildscript classpath declarations; plugins are applied via the plugins DSL below
@@ -88,7 +88,8 @@ data class ArgsDeclaration(val params: List<Item>, val endTokenType: Token.Type)
a.visibility ?: defaultVisibility,
recordType = recordType,
declaringClass = declaringClass,
isTransient = a.isTransient
isTransient = a.isTransient,
annotations = a.annotations
)
}
return
@@ -108,7 +109,8 @@ data class ArgsDeclaration(val params: List<Item>, val endTokenType: Token.Type)
a.visibility ?: defaultVisibility,
recordType = recordType,
declaringClass = declaringClass,
isTransient = a.isTransient
isTransient = a.isTransient,
annotations = a.annotations
)
}
@@ -505,5 +507,7 @@ data class ArgsDeclaration(val params: List<Item>, val endTokenType: Token.Type)
val accessType: AccessType? = null,
val visibility: Visibility? = null,
val isTransient: Boolean = false,
val annotationSpecs: List<ParsedDeclAnnotation> = emptyList(),
val annotations: List<DeclAnnotation> = emptyList(),
)
}
@@ -41,6 +41,17 @@ data class ClassDeclSpec(
val initScope: List<Statement>,
)
private suspend fun evaluateConstructorAnnotations(scope: Scope, args: ArgsDeclaration?): ArgsDeclaration? {
if (args == null) return null
if (args.params.none { it.annotationSpecs.isNotEmpty() }) return args
return args.copy(
params = args.params.map { item ->
if (item.annotationSpecs.isEmpty()) item
else item.copy(annotations = item.annotationSpecs.evaluateDeclAnnotations(scope))
}
)
}
internal suspend fun executeClassDecl(
scope: Scope,
spec: ClassDeclSpec,
@@ -60,7 +71,8 @@ internal suspend fun executeClassDecl(
val newClass = ObjInstanceClass(spec.className, *parentClasses.toTypedArray())
newClass.isAnonymous = spec.isAnonymous
newClass.constructorMeta = ArgsDeclaration(emptyList(), Token.Type.RPAREN)
newClass.isSingletonObject = true
newClass.constructorMeta = evaluateConstructorAnnotations(scope, ArgsDeclaration(emptyList(), Token.Type.RPAREN))
for (i in parentClasses.indices) {
val argsList = spec.baseSpecs[i].args
if (argsList != null) newClass.directParentArgs[parentClasses[i]] = argsList
@@ -85,6 +97,7 @@ internal suspend fun executeClassDecl(
}
if (spec.isExtern) {
val evaluatedConstructorArgs = evaluateConstructorAnnotations(scope, spec.constructorArgs)
val parentClasses = spec.baseSpecs.mapNotNull { baseSpec ->
val rec = scope[baseSpec.name]
val cls = rec?.value as? ObjClass
@@ -105,8 +118,8 @@ internal suspend fun executeClassDecl(
}
val stub = resolved ?: ObjInstanceClass(spec.className, *parentClasses.toTypedArray()).apply {
this.isAbstract = true
constructorMeta = spec.constructorArgs
spec.constructorArgs?.params?.forEach { p ->
constructorMeta = evaluatedConstructorArgs
evaluatedConstructorArgs?.params?.forEach { p ->
if (p.accessType != null) {
createField(
p.name,
@@ -117,6 +130,7 @@ internal suspend fun executeClassDecl(
pos = Pos.builtIn,
isTransient = p.isTransient,
type = ObjRecord.Type.ConstructorField,
annotations = p.annotations,
fieldId = spec.constructorFieldIds?.get(p.name)
)
}
@@ -160,16 +174,17 @@ internal suspend fun executeClassDecl(
}
}
val evaluatedConstructorArgs = evaluateConstructorAnnotations(scope, spec.constructorArgs)
val newClass = ObjInstanceClass(spec.className, *parentClasses.toTypedArray()).also {
it.isAbstract = spec.isAbstract
it.isClosed = spec.isClosed
it.instanceConstructor = constructorCode
it.constructorMeta = spec.constructorArgs
it.constructorMeta = evaluatedConstructorArgs
for (i in parentClasses.indices) {
val argsList = spec.baseSpecs[i].args
if (argsList != null) it.directParentArgs[parentClasses[i]] = argsList
}
spec.constructorArgs?.params?.forEach { p ->
evaluatedConstructorArgs?.params?.forEach { p ->
if (p.accessType != null) {
it.createField(
p.name,
@@ -180,6 +195,7 @@ internal suspend fun executeClassDecl(
pos = Pos.builtIn,
isTransient = p.isTransient,
type = ObjRecord.Type.ConstructorField,
annotations = p.annotations,
fieldId = spec.constructorFieldIds?.get(p.name)
)
}
@@ -38,6 +38,7 @@ class ClassInstanceFieldDeclStatement(
val isClosed: Boolean,
val isOverride: Boolean,
val isTransient: Boolean,
val annotationSpecs: List<ParsedDeclAnnotation> = emptyList(),
val fieldId: Int?,
val initStatement: Statement?,
override val pos: Pos,
@@ -56,6 +57,7 @@ class ClassInstancePropertyDeclStatement(
val isClosed: Boolean,
val isOverride: Boolean,
val isTransient: Boolean,
val annotationSpecs: List<ParsedDeclAnnotation> = emptyList(),
val prop: ObjProperty,
val methodId: Int?,
val initStatement: Statement?,
@@ -75,6 +77,7 @@ class ClassInstanceDelegatedDeclStatement(
val isClosed: Boolean,
val isOverride: Boolean,
val isTransient: Boolean,
val annotationSpecs: List<ParsedDeclAnnotation> = emptyList(),
val methodId: Int?,
val initStatement: Statement?,
override val pos: Pos,
@@ -32,6 +32,7 @@ class ClassStaticFieldInitStatement(
val initializer: Statement?,
val isDelegated: Boolean,
val isTransient: Boolean,
val annotationSpecs: List<ParsedDeclAnnotation> = emptyList(),
private val startPos: Pos,
) : Statement() {
override val pos: Pos = startPos
@@ -39,6 +40,7 @@ class ClassStaticFieldInitStatement(
override suspend fun execute(scope: Scope): Obj {
val initValue = initializer?.let { execBytecodeOnly(scope, it, "class static field init") }?.byValueCopy()
?: ObjNull
val annotations = annotationSpecs.evaluateDeclAnnotations(scope)
val cls = scope.thisObj as? ObjClass
?: scope.raiseIllegalState("static field init requires class scope")
return if (isDelegated) {
@@ -61,7 +63,8 @@ class ClassStaticFieldInitStatement(
writeVisibility,
startPos,
isTransient = isTransient,
type = ObjRecord.Type.Delegated
type = ObjRecord.Type.Delegated,
annotations = annotations
).apply {
delegate = finalDelegate
}
@@ -72,7 +75,8 @@ class ClassStaticFieldInitStatement(
visibility,
writeVisibility,
recordType = ObjRecord.Type.Delegated,
isTransient = isTransient
isTransient = isTransient,
annotations = annotations
).apply {
delegate = finalDelegate
}
@@ -85,7 +89,8 @@ class ClassStaticFieldInitStatement(
visibility,
writeVisibility,
startPos,
isTransient = isTransient
isTransient = isTransient,
annotations = annotations
)
scope.addItem(
name,
@@ -94,7 +99,8 @@ class ClassStaticFieldInitStatement(
visibility,
writeVisibility,
recordType = ObjRecord.Type.Field,
isTransient = isTransient
isTransient = isTransient,
annotations = annotations
)
initValue
}
@@ -189,6 +189,7 @@ class Compiler(
private val callableReturnTypeDeclByName: MutableMap<String, TypeDecl> = mutableMapOf()
private val callSignatureByName: MutableMap<String, CallSignature> = mutableMapOf()
private val lambdaReturnTypeByRef: MutableMap<ObjRef, ObjClass> = mutableMapOf()
private val lambdaTypeDeclByRef: MutableMap<ObjRef, TypeDecl.Function> = mutableMapOf()
private val exactLambdaRefByScopeId: MutableMap<Int, MutableMap<Int, LambdaFnRef>> = mutableMapOf()
private val lambdaCaptureEntriesByRef: MutableMap<ValueFnRef, List<net.sergeych.lyng.bytecode.LambdaCaptureEntry>> =
mutableMapOf()
@@ -682,6 +683,7 @@ class Compiler(
return CompileClassInfo(
name,
cls.logicalPackageName,
emptyList(),
fieldIds,
methodIds,
nextFieldId,
@@ -1750,6 +1752,7 @@ class Compiler(
private data class CompileClassInfo(
val name: String,
val packageName: String?,
val typeParams: List<String>,
val fieldIds: Map<String, Int>,
val methodIds: Map<String, Int>,
val nextFieldId: Int,
@@ -2014,6 +2017,7 @@ class Compiler(
private var lastAnnotation: (suspend (Scope, ObjString, Statement) -> Statement)? = null
private var isTransientFlag: Boolean = false
private val pendingDeclAnnotations: MutableList<ParsedDeclAnnotation> = mutableListOf()
private var lastLabel: String? = null
private val strictSlotRefs: Boolean = settings.strictSlotRefs
private val allowUnresolvedRefs: Boolean = settings.allowUnresolvedRefs
@@ -2689,6 +2693,7 @@ class Compiler(
lastAnnotation = null
lastLabel = null
isTransientFlag = false
pendingDeclAnnotations.clear()
while (true) {
val t = cc.next()
return when (t.type) {
@@ -2706,15 +2711,17 @@ class Compiler(
}
Token.Type.ATLABEL -> {
val label = t.value
if (label == "Transient") {
val parsedAnnotation = parseDeclAnnotation(t)
if (parsedAnnotation.name == "Transient") {
isTransientFlag = true
continue
}
if (cc.peekNextNonWhitespace().type == Token.Type.LBRACE) {
lastLabel = label
lastLabel = parsedAnnotation.name
}
pendingDeclAnnotations += parsedAnnotation
if (parsedAnnotation.name != "Transient") {
lastAnnotation = parsedAnnotation.toStatementAnnotation()
}
lastAnnotation = parseAnnotation(t)
continue
}
@@ -2952,7 +2959,11 @@ class Compiler(
Token.Type.LPAREN -> {
cc.next()
if (shouldTreatAsClassScopeCall(left, next.value)) {
val parsed = parseArgs(null, implicitItTypeForMemberLambda(left, next.value))
val parsed = parseArgs(
null,
implicitItTypeForMemberLambda(left, next.value),
FieldRef(left, next.value, isOptional)
)
val args = parsed.first
val tailBlock = parsed.second
isCall = true
@@ -2969,7 +2980,11 @@ class Compiler(
val receiverType = if (next.value == "apply" || next.value == "run") {
inferReceiverTypeFromRef(left)
} else null
val parsed = parseArgs(receiverType, implicitItTypeForMemberLambda(left, next.value))
val parsed = parseArgs(
receiverType,
implicitItTypeForMemberLambda(left, next.value),
FieldRef(left, next.value, isOptional)
)
val args = parsed.first
val tailBlock = parsed.second
if (left is LocalVarRef && left.name == "scope") {
@@ -3492,10 +3507,12 @@ class Compiler(
paramTypeDeclMap[slot] = typeDecl
}
val lambdaParamTypeDecls = mutableListOf<TypeDecl>()
if (argsDeclaration != null) {
val expectedParams = expectedCallableType?.params.orEmpty()
argsDeclaration.params.forEachIndexed { index, param ->
val effectiveType = if ((param.type == TypeDecl.TypeAny || param.type == TypeDecl.TypeNullableAny) &&
val rawType = if ((param.type == TypeDecl.TypeAny || param.type == TypeDecl.TypeNullableAny) &&
index < expectedParams.size
) {
expectedParams[index]
@@ -3506,15 +3523,20 @@ class Compiler(
} else {
param.type
}
val effectiveType = if (param.isEllipsis) TypeDecl.Ellipsis(rawType) else rawType
lambdaParamTypeDecls += effectiveType
if (effectiveType != TypeDecl.TypeAny && effectiveType != TypeDecl.TypeNullableAny) {
seedLambdaParamType(param.name, effectiveType)
seedLambdaParamType(param.name, rawType)
}
}
} else {
val effectiveImplicitItType = implicitItType
?: expectedCallableType?.params?.singleOrNull()
if (effectiveImplicitItType != null) {
lambdaParamTypeDecls += effectiveImplicitItType
seedLambdaParamType("it", effectiveImplicitItType)
} else {
lambdaParamTypeDecls += TypeDecl.Ellipsis(TypeDecl.TypeAny)
}
}
@@ -3576,6 +3598,7 @@ class Compiler(
} else {
emptyList()
}
val inferredReturnDecl = inferReturnTypeDeclFromStatement(body)
val returnClass = inferReturnClassFromStatement(body)
val paramKnownClasses = mutableMapOf<String, ObjClass>()
argsDeclaration?.params?.forEach { param ->
@@ -3780,6 +3803,13 @@ class Compiler(
returnLabels = returnLabels,
pos = startPos
)
val lambdaTypeDecl = TypeDecl.Function(
receiver = null,
params = lambdaParamTypeDecls.toList(),
returnType = inferredReturnDecl ?: returnClass?.let { TypeDecl.Simple(it.className, false) } ?: TypeDecl.TypeAny,
nullable = false
)
lambdaTypeDeclByRef[ref] = lambdaTypeDecl
if (returnClass != null) {
lambdaReturnTypeByRef[ref] = returnClass
}
@@ -4033,11 +4063,17 @@ class Compiler(
Token.Type.ID, Token.Type.ATLABEL -> {
var isTransient = false
if (t.type == Token.Type.ATLABEL) {
if (t.value == "Transient") {
val annotationSpecs = mutableListOf<ParsedDeclAnnotation>()
while (t.type == Token.Type.ATLABEL) {
val spec = parseDeclAnnotation(t)
annotationSpecs += spec
if (spec.name == "Transient") {
isTransient = true
t = cc.next()
} else throw ScriptError(t.pos, "Unexpected label in argument list")
}
t = cc.next()
}
if (annotationSpecs.isNotEmpty() && !isClassDeclaration) {
throw ScriptError(t.pos, "parameter annotations are currently supported only on class constructor parameters")
}
// visibility
@@ -4114,7 +4150,8 @@ class Compiler(
defaultValue,
effectiveAccess,
visibility,
isTransient
isTransient,
annotationSpecs = annotationSpecs
)
// important: valid argument list continues with ',' and ends with '->' or ')'
@@ -4736,6 +4773,7 @@ class Compiler(
private fun inferTypeDeclFromRef(ref: ObjRef): TypeDecl? {
resolveReceiverTypeDecl(ref)?.let { return it }
return when (ref) {
is ValueFnRef -> lambdaTypeDeclByRef[ref]
is ListLiteralRef -> inferListLiteralTypeDecl(ref)
is MapLiteralRef -> inferMapLiteralTypeDecl(ref)
is ConstRef -> inferTypeDeclFromConst(ref.constValue)
@@ -5046,6 +5084,37 @@ class Compiler(
return null
}
private fun substituteReceiverTypeParams(receiverType: TypeDecl?, ownerClassName: String?, memberType: TypeDecl?): TypeDecl? {
if (receiverType !is TypeDecl.Generic || ownerClassName == null || memberType == null) return memberType
val info = resolveCompileClassInfo(ownerClassName) ?: return memberType
if (info.typeParams.isEmpty()) return memberType
val bindings = LinkedHashMap<String, TypeDecl>(info.typeParams.size)
for ((index, typeParamName) in info.typeParams.withIndex()) {
val argType = receiverType.args.getOrNull(index) ?: continue
bindings[typeParamName] = argType
}
if (bindings.isEmpty()) return memberType
return substituteTypeAliasTypeVars(memberType, bindings)
}
private fun inferExtensionPropertyTypeDecl(receiverDecl: TypeDecl?, receiverClass: ObjClass?, memberName: String): TypeDecl? {
if (receiverClass == null) return null
for (cls in receiverClass.mro) {
val wrapperName = extensionPropertyGetterName(cls.className, memberName)
val resolved = resolveImportBinding(wrapperName, Pos.builtIn) ?: continue
registerImportBinding(wrapperName, resolved.binding, Pos.builtIn)
val wrapperType = resolved.record.typeDecl as? TypeDecl.Function ?: continue
val bindings = mutableMapOf<String, TypeDecl>()
val receiverParam = wrapperType.params.firstOrNull() ?: wrapperType.receiver
if (receiverParam != null && receiverDecl != null) {
collectTypeVarBindings(receiverParam, receiverDecl, bindings)
}
return if (bindings.isEmpty()) wrapperType.returnType
else substituteTypeAliasTypeVars(wrapperType.returnType, bindings)
}
return null
}
private fun classMemberTypeDecl(targetClass: ObjClass?, name: String): TypeDecl? {
if (targetClass == null) return null
if (targetClass == ObjDynamic.type) return TypeDecl.TypeAny
@@ -5181,7 +5250,12 @@ class Compiler(
is FieldRef -> {
val targetDecl = resolveReceiverTypeDecl(ref.target) ?: return null
val targetClass = resolveTypeDeclObjClass(targetDecl) ?: resolveReceiverClassForMember(ref.target)
classMemberTypeDecl(targetClass, ref.name)?.let { return it }
classMemberTypeDecl(targetClass, ref.name)?.let { declared ->
val ownerClassName = targetClass?.getInstanceMemberOrNull(ref.name, includeAbstract = true)
?.declaringClass?.className ?: targetClass?.className
return substituteReceiverTypeParams(targetDecl, ownerClassName, declared)
}
inferExtensionPropertyTypeDecl(targetDecl, targetClass, ref.name)?.let { return it }
classFieldTypesByName[targetClass?.className]?.get(ref.name)
?.let { return TypeDecl.Simple(it.className, false) }
when (targetDecl) {
@@ -5518,8 +5592,22 @@ class Compiler(
private fun inferMethodCallReturnTypeDecl(ref: MethodCallRef): TypeDecl? {
methodReturnTypeDeclByRef[ref]?.let { return it }
val inferred = inferMethodCallReturnTypeDecl(ref.name, resolveReceiverTypeDecl(ref.receiver), ref.args)
?: classMethodReturnTypeDecl(resolveReceiverClassForMember(ref.receiver), ref.name)
val receiverDecl = resolveReceiverTypeDecl(ref.receiver)
val inferred = inferMethodCallReturnTypeDecl(ref.name, receiverDecl, ref.args)
?: inferDeclaredMethodCallReturnTypeDecl(
ref.name,
receiverDecl,
resolveReceiverClassForMember(ref.receiver),
ref.args,
ref.explicitTypeArgs
)
?: run {
val receiverClass = resolveReceiverClassForMember(ref.receiver)
val declared = classMethodReturnTypeDecl(receiverClass, ref.name)
val ownerClassName = receiverClass?.getInstanceMemberOrNull(ref.name, includeAbstract = true)
?.declaringClass?.className ?: receiverClass?.className
substituteReceiverTypeParams(receiverDecl, ownerClassName, declared)
}
if (inferred != null) {
methodReturnTypeDeclByRef[ref] = inferred
}
@@ -5624,6 +5712,108 @@ class Compiler(
}
}
private fun inferDeclaredMethodCallReturnTypeDecl(
name: String,
receiverDecl: TypeDecl?,
receiverClass: ObjClass?,
args: List<ParsedArgument>,
explicitTypeArgs: List<TypeDecl>? = null
): TypeDecl? {
if (receiverClass == null) return null
val ownerClassName = receiverClass.getInstanceMemberOrNull(name, includeAbstract = true)
?.declaringClass?.className ?: receiverClass.className
val memberType = substituteReceiverTypeParams(
receiverDecl,
ownerClassName,
classMemberTypeDecl(receiverClass, name)
) as? TypeDecl.Function ?: return null
fun argTypeDecl(arg: ParsedArgument): TypeDecl? {
val stmt = arg.value as? ExpressionStatement ?: return null
val directRef = stmt.ref
return inferTypeDeclFromRef(directRef)
?: inferObjClassFromRef(directRef)?.let { TypeDecl.Simple(it.className, false) }
}
val bindings = mutableMapOf<String, TypeDecl>()
collectExplicitMethodTypeBindings(memberType, explicitTypeArgs, bindings)
memberType.receiver?.let { declaredReceiver ->
receiverDecl?.let { collectTypeVarBindings(declaredReceiver, it, bindings) }
}
val paramList = memberType.params
val ellipsisIndex = paramList.indexOfFirst { it is TypeDecl.Ellipsis }
if (ellipsisIndex < 0) {
val limit = minOf(paramList.size, args.size)
for (i in 0 until limit) {
val argType = argTypeDecl(args[i]) ?: continue
collectTypeVarBindings(paramList[i], argType, bindings)
}
} else {
val headCount = ellipsisIndex
val tailCount = paramList.size - ellipsisIndex - 1
val argCount = args.size
val headLimit = minOf(headCount, argCount)
for (i in 0 until headLimit) {
val argType = argTypeDecl(args[i]) ?: continue
collectTypeVarBindings(paramList[i], argType, bindings)
}
val tailStartArg = maxOf(headCount, argCount - tailCount)
for (i in tailStartArg until argCount) {
val paramIndex = paramList.size - (argCount - i)
val argType = argTypeDecl(args[i]) ?: continue
collectTypeVarBindings(paramList[paramIndex], argType, bindings)
}
val ellipsisArgEnd = argCount - tailCount
val ellipsisType = paramList[ellipsisIndex] as TypeDecl.Ellipsis
for (i in headCount until ellipsisArgEnd) {
val argType = if (args[i].isSplat) {
val stmt = args[i].value as? ExpressionStatement
stmt?.ref?.let { inferElementTypeFromSpread(it) }
} else {
argTypeDecl(args[i])
} ?: continue
collectTypeVarBindings(ellipsisType.elementType, argType, bindings)
}
}
return if (bindings.isEmpty()) memberType.returnType
else substituteTypeAliasTypeVars(memberType.returnType, bindings)
}
private fun collectExplicitMethodTypeBindings(
memberType: TypeDecl.Function,
explicitTypeArgs: List<TypeDecl>?,
out: MutableMap<String, TypeDecl>
) {
if (explicitTypeArgs.isNullOrEmpty()) return
val typeVars = LinkedHashSet<String>()
memberType.receiver?.let { collectTypeVarNamesInOrder(it, typeVars) }
memberType.params.forEach { collectTypeVarNamesInOrder(it, typeVars) }
collectTypeVarNamesInOrder(memberType.returnType, typeVars)
val names = typeVars.toList()
val limit = minOf(names.size, explicitTypeArgs.size)
for (i in 0 until limit) {
out[names[i]] = explicitTypeArgs[i]
}
}
private fun collectTypeVarNamesInOrder(type: TypeDecl, out: MutableSet<String>) {
when (type) {
is TypeDecl.TypeVar -> out += type.name
is TypeDecl.Generic -> type.args.forEach { collectTypeVarNamesInOrder(it, out) }
is TypeDecl.Function -> {
type.receiver?.let { collectTypeVarNamesInOrder(it, out) }
type.params.forEach { collectTypeVarNamesInOrder(it, out) }
collectTypeVarNamesInOrder(type.returnType, out)
}
is TypeDecl.Ellipsis -> collectTypeVarNamesInOrder(type.elementType, out)
is TypeDecl.Union -> type.options.forEach { collectTypeVarNamesInOrder(it, out) }
is TypeDecl.Intersection -> type.options.forEach { collectTypeVarNamesInOrder(it, out) }
else -> {}
}
}
private fun inferCallableReturnTypeDeclFromArgument(arg: ParsedArgument): TypeDecl? {
val stmt = arg.value as? ExpressionStatement ?: return null
val ref = stmt.ref
@@ -6093,6 +6283,7 @@ class Compiler(
args: List<ParsedArgument>,
pos: Pos
) {
if (shouldSkipStaticCallableChecks(target)) return
lookupNamedFunctionDecl(target)?.let { decl ->
val hasComplexArgs = args.any { it.name != null } ||
decl.typeParams.isNotEmpty() ||
@@ -6190,6 +6381,22 @@ class Compiler(
return null
}
private fun shouldSkipStaticCallableChecks(target: ObjRef): Boolean {
resolveExactLambdaRef(target)?.let { lambda ->
if (lambda.argsDeclaration == null) return true
}
val name = when (target) {
is LocalVarRef -> target.name
is LocalSlotRef -> target.name
is FastLocalVarRef -> target.name
else -> null
}
if (name != null && callSignatureForName(name) != null) {
return true
}
return lookupNamedCallableRecord(target)?.callSignature != null
}
private fun inferCallReturnTypeDecl(ref: CallRef): TypeDecl? {
callReturnTypeDeclByRef[ref]?.let { return it }
val targetDecl = (resolveReceiverTypeDecl(ref.target) ?: seedTypeDeclFromRef(ref.target)) as? TypeDecl.Function
@@ -6252,6 +6459,7 @@ class Compiler(
args: List<ParsedArgument>,
pos: Pos
) {
if (shouldSkipStaticCallableChecks(target)) return
lookupNamedFunctionDecl(target)?.let { decl ->
val hasComplexArgs = args.any { it.name != null } ||
decl.typeParams.isNotEmpty() ||
@@ -6345,7 +6553,7 @@ class Compiler(
return
}
val seededCallable = lookupNamedCallableRecord(target)
if (seededCallable != null && seededCallable.type == ObjRecord.Type.Fun && seededCallable.value !is ObjExternCallable) {
if (seededCallable != null && seededCallable.type == ObjRecord.Type.Fun) {
return
}
val decl = (resolveReceiverTypeDecl(target) as? TypeDecl.Function)
@@ -6459,6 +6667,17 @@ class Compiler(
}
}
}
is TypeDecl.Function -> {
if (argType is TypeDecl.Function && paramType.params.size == argType.params.size) {
if (paramType.receiver != null && argType.receiver != null) {
collectTypeVarBindings(paramType.receiver, argType.receiver, out)
}
for (i in paramType.params.indices) {
collectTypeVarBindings(paramType.params[i], argType.params[i], out)
}
collectTypeVarBindings(paramType.returnType, argType.returnType, out)
}
}
is TypeDecl.Union -> {
if (argType is TypeDecl.Union) {
val limit = minOf(paramType.options.size, argType.options.size)
@@ -7120,19 +7339,25 @@ class Compiler(
return parseNumberOrNull(isPlus) ?: throw ScriptError(cc.currentPos(), "Expecting number")
}
suspend fun parseAnnotation(t: Token): (suspend (Scope, ObjString, Statement) -> Statement) {
private suspend fun parseDeclAnnotation(t: Token): ParsedDeclAnnotation {
val extraArgs = parseArgsOrNull()
resolutionSink?.reference(t.value, t.pos)
// println("annotation ${t.value}: args: $extraArgs")
return { scope, name, body ->
val extras = extraArgs?.first?.toArguments(scope, extraArgs.second)?.list
val required = listOf(name, body)
val args = extras?.let { required + it } ?: required
val fn = scope.get(t.value)?.value ?: scope.raiseSymbolNotFound("annotation not found: ${t.value}")
if (fn !is Statement) scope.raiseIllegalArgument("annotation must be callable, got ${fn.objClass}")
(fn.execute(scope.createChildScope(Arguments(args))) as? Statement)
?: scope.raiseClassCastError("function annotation must return callable")
}
val compiledArgs = extraArgs?.first?.map { arg ->
val value = arg.value
arg.copy(
value = if (value is Statement) wrapBytecode(value) else value
)
} ?: emptyList()
return ParsedDeclAnnotation(
name = t.value,
args = compiledArgs,
tailBlockMode = extraArgs?.second ?: false,
pos = t.pos
)
}
suspend fun parseAnnotation(t: Token): (suspend (Scope, ObjString, Statement) -> Statement) {
return parseDeclAnnotation(t).toStatementAnnotation()
}
suspend fun parseArgsOrNull(): Pair<List<ParsedArgument>, Boolean>? =
@@ -7803,6 +8028,7 @@ class Compiler(
compileClassInfos[qualifiedName] = CompileClassInfo(
name = qualifiedName,
packageName = packageName,
typeParams = emptyList(),
fieldIds = fieldIds,
methodIds = methodIds,
nextFieldId = fieldIds.size,
@@ -7926,6 +8152,7 @@ class Compiler(
compileClassInfos[className] = CompileClassInfo(
name = className,
packageName = packageName,
typeParams = emptyList(),
fieldIds = ctx.memberFieldIds.toMap(),
methodIds = ctx.memberMethodIds.toMap(),
nextFieldId = ctx.nextFieldId,
@@ -7968,6 +8195,7 @@ class Compiler(
compileClassInfos[className] = CompileClassInfo(
name = className,
packageName = packageName,
typeParams = emptyList(),
fieldIds = baseIds.fieldIds,
methodIds = baseIds.methodIds,
nextFieldId = baseIds.nextFieldId,
@@ -8233,6 +8461,7 @@ class Compiler(
compileClassInfos[qualifiedName] = CompileClassInfo(
name = qualifiedName,
packageName = packageName,
typeParams = typeParamDecls.map { it.name },
fieldIds = ctx.memberFieldIds.toMap(),
methodIds = ctx.memberMethodIds.toMap(),
nextFieldId = ctx.nextFieldId,
@@ -8250,6 +8479,7 @@ class Compiler(
compileClassInfos[qualifiedName] = CompileClassInfo(
name = qualifiedName,
packageName = packageName,
typeParams = typeParamDecls.map { it.name },
fieldIds = ctx.memberFieldIds.toMap(),
methodIds = ctx.memberMethodIds.toMap(),
nextFieldId = ctx.nextFieldId,
@@ -8338,6 +8568,7 @@ class Compiler(
compileClassInfos[qualifiedName] = CompileClassInfo(
name = qualifiedName,
packageName = packageName,
typeParams = typeParamDecls.map { it.name },
fieldIds = ctx.memberFieldIds.toMap(),
methodIds = ctx.memberMethodIds.toMap(),
nextFieldId = ctx.nextFieldId,
@@ -9232,6 +9463,8 @@ class Compiler(
isTransient: Boolean = isTransientFlag
): Statement {
isTransientFlag = false
val declarationAnnotationSpecs = pendingDeclAnnotations.toList()
pendingDeclAnnotations.clear()
val actualExtern = isExtern || (codeContexts.lastOrNull() as? CodeContext.ClassBody)?.isExtern == true
var start = cc.currentPos()
var extTypeName: String? = null
@@ -9520,7 +9753,8 @@ class Compiler(
resolutionSink?.exitScope(cc.currentPos())
}
val rawFnStatements = parsedFnStatements?.let { unwrapBytecodeDeep(it) }
val inferredReturnClass = returnTypeDecl?.let { resolveTypeDeclObjClass(it) }
val inferredReturnDecl = returnTypeDecl ?: inferReturnTypeDeclFromStatement(rawFnStatements)
val inferredReturnClass = inferredReturnDecl?.let { resolveTypeDeclObjClass(it) }
?: inferReturnClassFromStatement(rawFnStatements)
if (parentContext is CodeContext.ClassBody && !isStatic && extTypeName == null) {
val ownerClassName = parentContext.name
@@ -9528,15 +9762,12 @@ class Compiler(
val memberTypeDecl = TypeDecl.Function(
receiver = receiverTypeDecl,
params = argsDeclaration.params.map { it.type },
returnType = returnTypeDecl
?: inferredReturnClass?.let { TypeDecl.Simple(it.className, false) }
?: TypeDecl.TypeAny,
returnType = inferredReturnDecl ?: TypeDecl.TypeAny,
nullable = false
)
classMemberTypeDeclByName
.getOrPut(ownerClassName) { mutableMapOf() }[name] = memberTypeDecl
val returnDecl = returnTypeDecl
?: inferredReturnClass?.let { TypeDecl.Simple(it.className, false) }
val returnDecl = inferredReturnDecl
if (returnDecl != null) {
classMethodReturnTypeDeclByName
.getOrPut(ownerClassName) { mutableMapOf() }[name] = returnDecl
@@ -9557,7 +9788,6 @@ class Compiler(
inferredReturnClass
}
}
val inferredReturnDecl = returnTypeDecl ?: inferredReturnClass?.let { TypeDecl.Simple(it.className, false) }
if (declKind != SymbolKind.MEMBER && inferredReturnDecl != null) {
callableReturnTypeDeclByName[name] = inferredReturnDecl
}
@@ -9700,6 +9930,7 @@ class Compiler(
isClosed = isClosed,
isOverride = isOverride,
isTransient = isTransient,
annotations = emptyList(),
accessTypeLabel = "Callable",
initializer = initExpr,
pos = start
@@ -9861,6 +10092,40 @@ class Compiler(
}
}
private fun inferReturnTypeDeclFromStatement(stmt: Statement?): TypeDecl? {
if (stmt == null) return null
val unwrapped = unwrapBytecodeDeep(stmt)
return when (unwrapped) {
is ExpressionStatement -> inferTypeDeclFromInitializer(unwrapped)
is ReturnStatement -> unwrapped.resultExpr?.let { inferTypeDeclFromInitializer(it) }
is VarDeclStatement -> unwrapped.typeDecl ?: unwrapped.initializer?.let { inferTypeDeclFromInitializer(it) }
is BlockStatement -> {
val stmts = unwrapped.statements()
val returnTypes = stmts.mapNotNull { s ->
(s as? ReturnStatement)?.resultExpr?.let { inferTypeDeclFromInitializer(it) }
}
if (returnTypes.isNotEmpty()) {
val first = returnTypes.first()
if (returnTypes.all { typeDeclKey(it) == typeDeclKey(first) }) first else null
} else {
inferReturnTypeDeclFromStatement(stmts.lastOrNull())
}
}
is InlineBlockStatement -> inferReturnTypeDeclFromStatement(unwrapped.statements().lastOrNull())
is IfStatement -> {
val ifType = inferReturnTypeDeclFromStatement(unwrapped.ifBody)
val elseType = unwrapped.elseBody?.let { inferReturnTypeDeclFromStatement(it) }
when {
ifType == null -> elseType
elseType == null -> ifType
typeDeclKey(ifType) == typeDeclKey(elseType) -> ifType
else -> null
}
}
else -> null
}
}
private fun unwrapDirectRef(initializer: Statement?): ObjRef? {
var initStmt = initializer
while (initStmt is BytecodeStatement) {
@@ -9941,8 +10206,11 @@ class Compiler(
inferMethodCallReturnClass(directRef)
}
is FieldRef -> {
val targetClass = resolveReceiverClassForMember(directRef.target)
inferFieldReturnClass(targetClass, directRef.name)
resolveReceiverTypeDecl(directRef)?.let { resolveTypeDeclObjClass(it) }
?: run {
val targetClass = resolveReceiverClassForMember(directRef.target)
inferFieldReturnClass(targetClass, directRef.name)
}
}
is ImplicitThisMemberRef -> resolveReceiverClassForMember(directRef)
is CallRef -> {
@@ -10251,6 +10519,8 @@ class Compiler(
isTransient: Boolean = isTransientFlag
): Statement {
isTransientFlag = false
val declarationAnnotationSpecs = pendingDeclAnnotations.toList()
pendingDeclAnnotations.clear()
val actualExtern = isExtern || (codeContexts.lastOrNull() as? CodeContext.ClassBody)?.isExtern == true
val markStart = cc.savePos()
val nextToken = cc.next()
@@ -10345,6 +10615,7 @@ class Compiler(
nameStartPos = nameToken.pos
}
val receiverNormalization = normalizeReceiverTypeDecl(receiverTypeDecl, emptySet())
receiverTypeDecl = receiverNormalization.first
val implicitTypeParams = receiverNormalization.second
if (implicitTypeParams.isNotEmpty()) pendingTypeParamStack.add(implicitTypeParams)
try {
@@ -10577,6 +10848,7 @@ class Compiler(
val directRef = unwrapDirectRef(initialExpression)
val declClass = resolveTypeDeclObjClass(varTypeDecl)
val initFromExpr = resolveInitializerObjClass(initialExpression)
val inferredInitTypeDecl = directRef?.let { inferTypeDeclFromRef(it) }
val isNullLiteral = (directRef as? ConstRef)?.constValue == ObjNull
val initObjClass = if (declClass != null && isNullLiteral) declClass else initFromExpr ?: declClass
if (varTypeDecl !is TypeDecl.TypeAny && varTypeDecl !is TypeDecl.TypeNullableAny) {
@@ -10584,16 +10856,29 @@ class Compiler(
slotTypeDeclByScopeId.getOrPut(scopeId) { mutableMapOf() }[slotIndex] = varTypeDecl
}
nameTypeDecl[name] = varTypeDecl
}
if (directRef is ValueFnRef) {
val returnClass = lambdaReturnTypeByRef[directRef]
if (returnClass != null) {
} else {
val inferredFunctionType = inferredInitTypeDecl as? TypeDecl.Function
if (inferredFunctionType != null) {
if (slotIndex != null && scopeId != null) {
callableReturnTypeByScopeId.getOrPut(scopeId) { mutableMapOf() }[slotIndex] = returnClass
slotTypeDeclByScopeId.getOrPut(scopeId) { mutableMapOf() }[slotIndex] = inferredFunctionType
}
callableReturnTypeByName[name] = returnClass
nameTypeDecl[name] = inferredFunctionType
}
}
val inferredCallableReturnClass = when {
directRef is ValueFnRef -> lambdaReturnTypeByRef[directRef]
directRef != null -> (inferredInitTypeDecl as? TypeDecl.Function)
?.returnType
?.let { resolveTypeDeclObjClass(it) }
else -> null
}
if (inferredCallableReturnClass != null) {
if (slotIndex != null && scopeId != null) {
callableReturnTypeByScopeId.getOrPut(scopeId) { mutableMapOf() }[slotIndex] =
inferredCallableReturnClass
}
callableReturnTypeByName[name] = inferredCallableReturnClass
}
if (directRef is MethodCallRef && directRef.name == "encode") {
val payloadClass = inferEncodedPayloadClass(directRef.args)
if (payloadClass != null) {
@@ -10643,6 +10928,9 @@ class Compiler(
!isStatic &&
!isProperty
) {
if (declarationAnnotationSpecs.isNotEmpty()) {
throw ScriptError(start, "declaration annotations are currently supported only on class members")
}
if (isDelegate) {
val initExpr = initialExpression ?: throw ScriptError(start, "Delegate must be initialized")
val slotPlan = slotPlanStack.lastOrNull()
@@ -10663,6 +10951,9 @@ class Compiler(
}
if (isStatic) {
if (extTypeName != null && declarationAnnotationSpecs.isNotEmpty()) {
throw ScriptError(start, "declaration annotations are not supported on extension properties")
}
if (declaringClassNameCaptured != null) {
val directRef = unwrapDirectRef(initialExpression)
val declClass = resolveTypeDeclObjClass(varTypeDecl)
@@ -10685,6 +10976,7 @@ class Compiler(
initializer = initialExpression,
isDelegated = isDelegate,
isTransient = isTransient,
annotationSpecs = declarationAnnotationSpecs,
startPos = start
)
return NopStatement
@@ -10847,6 +11139,27 @@ class Compiler(
}
if (extTypeName != null) {
if (declarationAnnotationSpecs.isNotEmpty()) {
throw ScriptError(start, "declaration annotations are not supported on extension properties")
}
val getterTypeDecl = receiverTypeDecl?.let { recv ->
TypeDecl.Function(
receiver = null,
params = listOf(recv),
returnType = varTypeDecl,
nullable = false
)
}
val setterTypeDecl = if (setter != null && receiverTypeDecl != null) {
TypeDecl.Function(
receiver = null,
params = listOf(receiverTypeDecl, varTypeDecl),
returnType = TypeDecl.Simple("void", false),
nullable = false
)
} else {
null
}
declareLocalName(extensionPropertyGetterName(extTypeName, name), isMutable = false)
if (setter != null) {
declareLocalName(extensionPropertySetterName(extTypeName, name), isMutable = false)
@@ -10863,6 +11176,8 @@ class Compiler(
property = prop,
visibility = visibility,
setterVisibility = setterVisibility,
getterTypeDecl = getterTypeDecl,
setterTypeDecl = setterTypeDecl,
startPos = start
)
}
@@ -10883,6 +11198,7 @@ class Compiler(
isClosed = isClosed,
isOverride = isOverride,
isTransient = isTransient,
annotations = emptyList(),
accessTypeLabel = accessType,
initializer = initExpr,
pos = start
@@ -10898,6 +11214,7 @@ class Compiler(
isClosed = isClosed,
isOverride = isOverride,
isTransient = isTransient,
annotationSpecs = declarationAnnotationSpecs,
methodId = memberMethodId,
initStatement = initStmt,
pos = start
@@ -10919,6 +11236,7 @@ class Compiler(
isClosed = isClosed,
isOverride = isOverride,
isTransient = isTransient,
annotations = emptyList(),
prop = prop,
pos = start
)
@@ -10933,6 +11251,7 @@ class Compiler(
isClosed = isClosed,
isOverride = isOverride,
isTransient = isTransient,
annotationSpecs = declarationAnnotationSpecs,
prop = prop,
methodId = memberMethodId,
initStatement = initStmt,
@@ -10950,6 +11269,7 @@ class Compiler(
isClosed = isClosed,
isOverride = isOverride,
isTransient = isTransient,
annotations = emptyList(),
isLateInitVal = isLateInitVal,
initializer = initialExpression,
pos = start
@@ -10966,6 +11286,7 @@ class Compiler(
isClosed = isClosed,
isOverride = isOverride,
isTransient = isTransient,
annotationSpecs = declarationAnnotationSpecs,
fieldId = memberFieldId,
initStatement = initStmt,
pos = start
@@ -0,0 +1,129 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng
import net.sergeych.lyng.bytecode.BytecodeStatement
import net.sergeych.lyng.obj.Obj
import net.sergeych.lyng.obj.ObjIterable
import net.sergeych.lyng.obj.ObjList
import net.sergeych.lyng.obj.ObjMap
import net.sergeych.lyng.obj.ObjString
/**
* Preserved declaration annotation evaluated at declaration-creation time.
*/
data class DeclAnnotation(
val name: String,
val positional: List<Obj> = emptyList(),
val named: Map<String, Obj> = emptyMap(),
)
/**
* Parsed declaration annotation awaiting declaration-time evaluation.
*/
data class ParsedDeclAnnotation(
val name: String,
val args: List<ParsedArgument> = emptyList(),
val tailBlockMode: Boolean = false,
val pos: Pos = Pos.builtIn,
) {
suspend fun evaluate(scope: Scope): DeclAnnotation {
val resolved = evaluateDeclAnnotationArguments(scope, args, tailBlockMode)
return DeclAnnotation(name, resolved.list, resolved.named)
}
fun toStatementAnnotation(): suspend (Scope, ObjString, Statement) -> Statement = { scope, declName, body ->
val extras = args.toArguments(scope, tailBlockMode).list
val required = listOf(declName, body)
val callArgs = if (extras.isEmpty()) required else required + extras
val fn = scope.get(name)?.value ?: scope.raiseSymbolNotFound("annotation not found: $name")
if (fn !is Statement) scope.raiseIllegalArgument("annotation must be callable, got ${fn.objClass}")
(fn.execute(scope.createChildScope(Arguments(callArgs))) as? Statement)
?: scope.raiseClassCastError("function annotation must return callable")
}
}
suspend fun Iterable<ParsedDeclAnnotation>.evaluateDeclAnnotations(scope: Scope): List<DeclAnnotation> {
val result = mutableListOf<DeclAnnotation>()
for (spec in this) {
result += spec.evaluate(scope)
}
return result
}
private suspend fun evaluateDeclAnnotationArguments(
scope: Scope,
args: List<ParsedArgument>,
tailBlockMode: Boolean,
): Arguments {
suspend fun eval(value: Obj): Obj = when (value) {
is BytecodeBodyProvider -> (value.bytecodeBody() ?: scope.raiseIllegalState("annotation argument requires bytecode body")).execute(scope)
is Statement -> BytecodeStatement.wrap(value, "@annotation", allowLocalSlots = true).execute(scope)
else -> value.callOn(scope)
}
val resolved = ArrayList<ParsedArgument>(args.size)
for (arg in args) {
resolved += arg.copy(value = eval(arg.value))
}
val positional: MutableList<Obj> = mutableListOf()
var named: MutableMap<String, Obj>? = null
var namedSeen = false
for ((idx, x) in resolved.withIndex()) {
if (x.name != null) {
if (named == null) named = linkedMapOf()
if (named.containsKey(x.name)) scope.raiseIllegalArgument("argument '${x.name}' is already set")
named[x.name] = x.value
namedSeen = true
continue
}
val value = x.value
if (x.isSplat) {
when {
value is ObjMap -> {
if (named == null) named = linkedMapOf()
for ((k, v) in value.map) {
if (k !is ObjString) scope.raiseIllegalArgument("named splat expects a Map with string keys")
val key = k.value
if (named.containsKey(key)) scope.raiseIllegalArgument("argument '$key' is already set")
named[key] = v
}
namedSeen = true
}
value is ObjList -> {
if (namedSeen) scope.raiseIllegalArgument("positional splat cannot follow named arguments")
positional.addAll(value.list)
}
value.isInstanceOf(ObjIterable) -> {
if (namedSeen) scope.raiseIllegalArgument("positional splat cannot follow named arguments")
val iterable = value.invokeInstanceMethod(scope, "toList") as ObjList
positional.addAll(iterable.list)
}
else -> scope.raiseClassCastError("expected list of objects for splat argument")
}
} else {
val isLast = idx == resolved.size - 1
if (namedSeen && !(isLast && tailBlockMode)) {
scope.raiseIllegalArgument("positional argument cannot follow named arguments")
}
positional.add(value)
}
}
return Arguments(positional, tailBlockMode, named ?: emptyMap())
}
@@ -24,6 +24,8 @@ class ExtensionPropertyDeclStatement(
val property: ObjProperty,
val visibility: Visibility,
val setterVisibility: Visibility?,
val getterTypeDecl: TypeDecl?,
val setterTypeDecl: TypeDecl?,
private val startPos: Pos,
) : Statement() {
override val pos: Pos = startPos
@@ -33,6 +33,7 @@ class InstanceFieldInitStatement(
val isClosed: Boolean,
val isOverride: Boolean,
val isTransient: Boolean,
val annotations: List<DeclAnnotation> = emptyList(),
val isLateInitVal: Boolean,
val initializer: Statement?,
override val pos: Pos,
@@ -50,7 +51,8 @@ class InstanceFieldInitStatement(
isAbstract = isAbstract,
isClosed = isClosed,
isOverride = isOverride,
isTransient = isTransient
isTransient = isTransient,
annotations = annotations
)
return ObjVoid
}
@@ -74,6 +76,7 @@ class InstancePropertyInitStatement(
val isClosed: Boolean,
val isOverride: Boolean,
val isTransient: Boolean,
val annotations: List<DeclAnnotation> = emptyList(),
val prop: ObjProperty,
override val pos: Pos,
) : Statement() {
@@ -88,7 +91,8 @@ class InstancePropertyInitStatement(
isAbstract = isAbstract,
isClosed = isClosed,
isOverride = isOverride,
isTransient = isTransient
isTransient = isTransient,
annotations = annotations
)
return ObjVoid
}
@@ -104,6 +108,7 @@ class InstanceDelegatedInitStatement(
val isClosed: Boolean,
val isOverride: Boolean,
val isTransient: Boolean,
val annotations: List<DeclAnnotation> = emptyList(),
val accessTypeLabel: String,
val initializer: Statement,
override val pos: Pos,
@@ -130,7 +135,8 @@ class InstanceDelegatedInitStatement(
isAbstract = isAbstract,
isClosed = isClosed,
isOverride = isOverride,
isTransient = isTransient
isTransient = isTransient,
annotations = annotations
).apply {
delegate = finalDelegate
}
@@ -718,6 +718,7 @@ open class Scope(
isTransient: Boolean = false,
callSignature: CallSignature? = null,
typeDecl: TypeDecl? = null,
annotations: List<DeclAnnotation> = emptyList(),
fieldId: Int? = null,
methodId: Int? = null
): ObjRecord {
@@ -731,6 +732,7 @@ open class Scope(
isTransient = isTransient,
callSignature = callSignature,
typeDecl = typeDecl,
annotations = annotations,
memberName = name,
fieldId = fieldId,
methodId = methodId
@@ -27,6 +27,8 @@ import net.sergeych.lyng.bytecode.CmdVm
import net.sergeych.lyng.bytecode.BytecodeLambdaCallable
import net.sergeych.lyng.miniast.*
import net.sergeych.lyng.obj.*
import net.sergeych.lyng.serialization.ObjJsonClass
import net.sergeych.lyng.serialization.bindSerializationFormat
import net.sergeych.lyng.pacman.ImportManager
import net.sergeych.lyng.stdlib_included.complexLyng
import net.sergeych.lyng.stdlib_included.decimalLyng
@@ -949,11 +951,13 @@ class Script(
)
}
addPackage("lyng.serialization") {
it.addConstDoc(
name = "Lynon",
value = ObjLynonClass,
doc = "Lynon serialization utilities: encode/decode data structures to a portable binary/text form.",
type = type("lyng.Class")
it.bindSerializationFormat(
ObjLynonClass,
doc = "Lynon serialization utilities: encode/decode data structures to a portable binary format."
)
it.bindSerializationFormat(
ObjJsonClass,
doc = "Universal JSON serialization utilities with bidirectional Lyng object support."
)
}
addPackage("lyng.time") {
@@ -6360,7 +6360,8 @@ class BytecodeCompiler(
isMutable = stmt.isMutable,
visibility = stmt.visibility,
writeVisibility = stmt.writeVisibility,
isTransient = stmt.isTransient
isTransient = stmt.isTransient,
annotationSpecs = stmt.annotationSpecs
)
)
} else {
@@ -6370,7 +6371,8 @@ class BytecodeCompiler(
isMutable = stmt.isMutable,
visibility = stmt.visibility,
writeVisibility = stmt.writeVisibility,
isTransient = stmt.isTransient
isTransient = stmt.isTransient,
annotationSpecs = stmt.annotationSpecs
)
)
}
@@ -6397,6 +6399,7 @@ class BytecodeCompiler(
writeVisibility = stmt.writeVisibility,
typeDecl = stmt.typeDecl,
isTransient = stmt.isTransient,
annotationSpecs = stmt.annotationSpecs,
isAbstract = stmt.isAbstract,
isClosed = stmt.isClosed,
isOverride = stmt.isOverride,
@@ -6419,6 +6422,7 @@ class BytecodeCompiler(
visibility = stmt.visibility,
writeVisibility = stmt.writeVisibility,
isTransient = stmt.isTransient,
annotationSpecs = stmt.annotationSpecs,
isAbstract = stmt.isAbstract,
isClosed = stmt.isClosed,
isOverride = stmt.isOverride,
@@ -6442,6 +6446,7 @@ class BytecodeCompiler(
visibility = stmt.visibility,
writeVisibility = stmt.writeVisibility,
isTransient = stmt.isTransient,
annotationSpecs = stmt.annotationSpecs,
isAbstract = stmt.isAbstract,
isClosed = stmt.isClosed,
isOverride = stmt.isOverride,
@@ -6475,6 +6480,7 @@ class BytecodeCompiler(
visibility = stmt.visibility,
writeVisibility = stmt.writeVisibility,
isTransient = stmt.isTransient,
annotations = stmt.annotations,
isAbstract = stmt.isAbstract,
isClosed = stmt.isClosed,
isOverride = stmt.isOverride
@@ -6497,6 +6503,7 @@ class BytecodeCompiler(
visibility = stmt.visibility,
writeVisibility = stmt.writeVisibility,
isTransient = stmt.isTransient,
annotations = stmt.annotations,
isAbstract = stmt.isAbstract,
isClosed = stmt.isClosed,
isOverride = stmt.isOverride
@@ -6517,6 +6524,7 @@ class BytecodeCompiler(
visibility = stmt.visibility,
writeVisibility = stmt.writeVisibility,
isTransient = stmt.isTransient,
annotations = stmt.annotations,
isAbstract = stmt.isAbstract,
isClosed = stmt.isClosed,
isOverride = stmt.isOverride,
@@ -7995,7 +8003,9 @@ class BytecodeCompiler(
stmt.extTypeName,
stmt.property,
stmt.visibility,
stmt.setterVisibility
stmt.setterVisibility,
stmt.getterTypeDecl,
stmt.setterTypeDecl
)
)
val slot = allocSlot()
@@ -8636,29 +8646,73 @@ class BytecodeCompiler(
}
private fun inferCallReturnClass(ref: CallRef): ObjClass? {
fun exactLambdaReturnClass(slot: Int): ObjClass? =
exactLambdaRefBySlot[slot]?.inferredReturnClass
fun callableReturnClassFromSlot(slot: Int): ObjClass? {
exactLambdaReturnClass(slot)?.let { return it }
typeDeclForSlot(slot)?.let { decl ->
val functionDecl = decl as? TypeDecl.Function
if (functionDecl != null) {
resolveClassFromTypeDecl(functionDecl.returnType)?.let { return it }
}
}
return null
}
fun callableResultClassOrNull(
directReturnClass: ObjClass?,
directTypeDecl: TypeDecl?,
nameClass: ObjClass?,
typeNameFallback: String?
): ObjClass? {
if (directReturnClass != null) return directReturnClass
if (directTypeDecl is TypeDecl.Function) {
return null
}
if (nameClass == ObjClassType) {
return typeNameFallback?.let { resolveTypeNameClass(it) } ?: ObjDynamic.type
}
if (nameClass == Statement.type) {
return null
}
return nameClass ?: typeNameFallback?.let { resolveTypeNameClass(it) }
}
return when (val target = ref.target) {
is LocalSlotRef -> {
callableReturnTypeByScopeId[target.scopeId]?.get(target.slot)
?: run {
val nameClass = nameObjClass[target.name]
if (nameClass == ObjClassType) {
resolveTypeNameClass(target.name) ?: ObjDynamic.type
} else {
nameClass ?: resolveTypeNameClass(target.name)
}
}
val mappedSlot = resolveLocalSlotByRefOrName(target)
callableResultClassOrNull(
directReturnClass = mappedSlot?.let { callableReturnClassFromSlot(it) }
?: exactLambdaRefByScopeId[target.scopeId]?.get(target.slot)?.inferredReturnClass
?: callableReturnTypeByScopeId[target.scopeId]?.get(target.slot),
directTypeDecl = mappedSlot?.let { typeDeclForSlot(it) }
?: slotTypeDeclByScopeId[target.scopeId]?.get(target.slot),
nameClass = nameObjClass[target.name],
typeNameFallback = target.name
)
}
is LocalVarRef -> {
callableReturnTypeByName[target.name]
?: run {
val nameClass = nameObjClass[target.name]
if (nameClass == ObjClassType) {
resolveTypeNameClass(target.name) ?: ObjDynamic.type
} else {
nameClass ?: resolveTypeNameClass(target.name)
}
}
val directSlot = resolveDirectNameSlot(target.name)?.slot
callableResultClassOrNull(
directReturnClass = directSlot?.let { callableReturnClassFromSlot(it) }
?: callableReturnTypeByName[target.name],
directTypeDecl = directSlot?.let { typeDeclForSlot(it) },
nameClass = nameObjClass[target.name],
typeNameFallback = target.name
)
}
is FastLocalVarRef -> {
val directSlot = resolveDirectNameSlot(target.name)?.slot
callableResultClassOrNull(
directReturnClass = directSlot?.let { callableReturnClassFromSlot(it) }
?: callableReturnTypeByName[target.name],
directTypeDecl = directSlot?.let { typeDeclForSlot(it) },
nameClass = nameObjClass[target.name],
typeNameFallback = target.name
)
}
is BoundLocalVarRef -> callableReturnClassFromSlot(target.slotIndex())
is ConstRef -> target.constValue as? ObjClass
else -> null
}
@@ -18,6 +18,7 @@
package net.sergeych.lyng.bytecode
import net.sergeych.lyng.ArgsDeclaration
import net.sergeych.lyng.ParsedDeclAnnotation
import net.sergeych.lyng.Pos
import net.sergeych.lyng.TypeDecl
import net.sergeych.lyng.Visibility
@@ -65,6 +66,8 @@ sealed class BytecodeConst {
val property: ObjProperty,
val visibility: Visibility,
val setterVisibility: Visibility?,
val getterTypeDecl: TypeDecl?,
val setterTypeDecl: TypeDecl?,
) : BytecodeConst()
data class LocalDecl(
val name: String,
@@ -85,6 +88,7 @@ sealed class BytecodeConst {
val visibility: Visibility,
val writeVisibility: Visibility?,
val isTransient: Boolean,
val annotationSpecs: List<ParsedDeclAnnotation>,
) : BytecodeConst()
data class ClassDelegatedDecl(
val name: String,
@@ -92,6 +96,7 @@ sealed class BytecodeConst {
val visibility: Visibility,
val writeVisibility: Visibility?,
val isTransient: Boolean,
val annotationSpecs: List<ParsedDeclAnnotation>,
) : BytecodeConst()
data class ClassInstanceInitDecl(
val initStatement: Obj,
@@ -103,6 +108,7 @@ sealed class BytecodeConst {
val writeVisibility: Visibility?,
val typeDecl: TypeDecl?,
val isTransient: Boolean,
val annotationSpecs: List<ParsedDeclAnnotation>,
val isAbstract: Boolean,
val isClosed: Boolean,
val isOverride: Boolean,
@@ -116,6 +122,7 @@ sealed class BytecodeConst {
val visibility: Visibility,
val writeVisibility: Visibility?,
val isTransient: Boolean,
val annotationSpecs: List<ParsedDeclAnnotation>,
val isAbstract: Boolean,
val isClosed: Boolean,
val isOverride: Boolean,
@@ -130,6 +137,7 @@ sealed class BytecodeConst {
val visibility: Visibility,
val writeVisibility: Visibility?,
val isTransient: Boolean,
val annotationSpecs: List<ParsedDeclAnnotation>,
val isAbstract: Boolean,
val isClosed: Boolean,
val isOverride: Boolean,
@@ -143,6 +151,7 @@ sealed class BytecodeConst {
val visibility: Visibility,
val writeVisibility: Visibility?,
val isTransient: Boolean,
val annotations: List<net.sergeych.lyng.DeclAnnotation>,
val isAbstract: Boolean,
val isClosed: Boolean,
val isOverride: Boolean,
@@ -153,6 +162,7 @@ sealed class BytecodeConst {
val visibility: Visibility,
val writeVisibility: Visibility?,
val isTransient: Boolean,
val annotations: List<net.sergeych.lyng.DeclAnnotation>,
val isAbstract: Boolean,
val isClosed: Boolean,
val isOverride: Boolean,
@@ -164,6 +174,7 @@ sealed class BytecodeConst {
val visibility: Visibility,
val writeVisibility: Visibility?,
val isTransient: Boolean,
val annotations: List<net.sergeych.lyng.DeclAnnotation>,
val isAbstract: Boolean,
val isClosed: Boolean,
val isOverride: Boolean,
@@ -365,6 +365,7 @@ class BytecodeStatement private constructor(
stmt.initializer?.let { unwrapDeep(it) },
stmt.isDelegated,
stmt.isTransient,
stmt.annotationSpecs,
stmt.pos
)
}
@@ -385,6 +386,7 @@ class BytecodeStatement private constructor(
stmt.isClosed,
stmt.isOverride,
stmt.isTransient,
stmt.annotationSpecs,
stmt.fieldId,
stmt.initStatement?.let { unwrapDeep(it) },
stmt.pos
@@ -400,6 +402,7 @@ class BytecodeStatement private constructor(
stmt.isClosed,
stmt.isOverride,
stmt.isTransient,
stmt.annotationSpecs,
stmt.prop,
stmt.methodId,
stmt.initStatement?.let { unwrapDeep(it) },
@@ -416,6 +419,7 @@ class BytecodeStatement private constructor(
stmt.isClosed,
stmt.isOverride,
stmt.isTransient,
stmt.annotationSpecs,
stmt.methodId,
stmt.initStatement?.let { unwrapDeep(it) },
stmt.pos
@@ -431,6 +435,7 @@ class BytecodeStatement private constructor(
stmt.isClosed,
stmt.isOverride,
stmt.isTransient,
stmt.annotations,
stmt.isLateInitVal,
stmt.initializer?.let { unwrapDeep(it) },
stmt.pos
@@ -446,6 +451,7 @@ class BytecodeStatement private constructor(
stmt.isClosed,
stmt.isOverride,
stmt.isTransient,
stmt.annotations,
stmt.prop,
stmt.pos
)
@@ -461,6 +467,7 @@ class BytecodeStatement private constructor(
stmt.isClosed,
stmt.isOverride,
stmt.isTransient,
stmt.annotations,
stmt.accessTypeLabel,
unwrapDeep(stmt.initializer),
stmt.pos
@@ -2805,6 +2805,7 @@ class CmdDeclClassField(internal val constId: Int, internal val slot: Int) : Cmd
val decl = frame.fn.constants[constId] as? BytecodeConst.ClassFieldDecl
?: error("DECL_CLASS_FIELD expects ClassFieldDecl at $constId")
val scope = frame.ensureScope()
val annotations = decl.annotationSpecs.evaluateDeclAnnotations(scope)
val cls = scope.thisObj as? ObjClass
?: scope.raiseIllegalState("class field init requires class scope")
val value = frame.slotToObj(slot).byValueCopy()
@@ -2815,7 +2816,8 @@ class CmdDeclClassField(internal val constId: Int, internal val slot: Int) : Cmd
decl.visibility,
decl.writeVisibility,
Pos.builtIn,
isTransient = decl.isTransient
isTransient = decl.isTransient,
annotations = annotations
)
scope.addItem(
decl.name,
@@ -2824,7 +2826,8 @@ class CmdDeclClassField(internal val constId: Int, internal val slot: Int) : Cmd
decl.visibility,
decl.writeVisibility,
recordType = ObjRecord.Type.Field,
isTransient = decl.isTransient
isTransient = decl.isTransient,
annotations = annotations
)
return
}
@@ -2835,6 +2838,7 @@ class CmdDeclClassDelegated(internal val constId: Int, internal val slot: Int) :
val decl = frame.fn.constants[constId] as? BytecodeConst.ClassDelegatedDecl
?: error("DECL_CLASS_DELEGATED expects ClassDelegatedDecl at $constId")
val scope = frame.ensureScope()
val annotations = decl.annotationSpecs.evaluateDeclAnnotations(scope)
val cls = scope.thisObj as? ObjClass
?: scope.raiseIllegalState("class delegated init requires class scope")
val initValue = frame.slotToObj(slot)
@@ -2857,7 +2861,8 @@ class CmdDeclClassDelegated(internal val constId: Int, internal val slot: Int) :
decl.writeVisibility,
Pos.builtIn,
isTransient = decl.isTransient,
type = ObjRecord.Type.Delegated
type = ObjRecord.Type.Delegated,
annotations = annotations
).apply {
delegate = finalDelegate
}
@@ -2868,7 +2873,8 @@ class CmdDeclClassDelegated(internal val constId: Int, internal val slot: Int) :
decl.visibility,
decl.writeVisibility,
recordType = ObjRecord.Type.Delegated,
isTransient = decl.isTransient
isTransient = decl.isTransient,
annotations = annotations
).apply {
delegate = finalDelegate
}
@@ -2895,6 +2901,7 @@ class CmdDeclClassInstanceField(internal val constId: Int, internal val slot: In
val decl = frame.fn.constants[constId] as? BytecodeConst.ClassInstanceFieldDecl
?: error("DECL_CLASS_INSTANCE_FIELD expects ClassInstanceFieldDecl at $constId")
val scope = frame.ensureScope()
val annotations = decl.annotationSpecs.evaluateDeclAnnotations(scope)
val cls = scope.thisObj as? ObjClass
?: scope.raiseIllegalState("class instance field requires class scope")
cls.createField(
@@ -2911,7 +2918,8 @@ class CmdDeclClassInstanceField(internal val constId: Int, internal val slot: In
isTransient = decl.isTransient,
typeDecl = decl.typeDecl,
type = ObjRecord.Type.Field,
fieldId = decl.fieldId
fieldId = decl.fieldId,
annotations = annotations
)
if (!decl.isAbstract) {
decl.initStatement?.let { cls.instanceInitializers += it }
@@ -2926,6 +2934,7 @@ class CmdDeclClassInstanceProperty(internal val constId: Int, internal val slot:
val decl = frame.fn.constants[constId] as? BytecodeConst.ClassInstancePropertyDecl
?: error("DECL_CLASS_INSTANCE_PROPERTY expects ClassInstancePropertyDecl at $constId")
val scope = frame.ensureScope()
val annotations = decl.annotationSpecs.evaluateDeclAnnotations(scope)
val cls = scope.thisObj as? ObjClass
?: scope.raiseIllegalState("class instance property requires class scope")
cls.addProperty(
@@ -2938,7 +2947,8 @@ class CmdDeclClassInstanceProperty(internal val constId: Int, internal val slot:
isOverride = decl.isOverride,
pos = decl.pos,
prop = decl.prop,
methodId = decl.methodId
methodId = decl.methodId,
annotations = annotations
)
if (!decl.isAbstract) {
decl.initStatement?.let { cls.instanceInitializers += it }
@@ -2953,6 +2963,7 @@ class CmdDeclClassInstanceDelegated(internal val constId: Int, internal val slot
val decl = frame.fn.constants[constId] as? BytecodeConst.ClassInstanceDelegatedDecl
?: error("DECL_CLASS_INSTANCE_DELEGATED expects ClassInstanceDelegatedDecl at $constId")
val scope = frame.ensureScope()
val annotations = decl.annotationSpecs.evaluateDeclAnnotations(scope)
val cls = scope.thisObj as? ObjClass
?: scope.raiseIllegalState("class instance delegated requires class scope")
cls.createField(
@@ -2968,7 +2979,8 @@ class CmdDeclClassInstanceDelegated(internal val constId: Int, internal val slot
isOverride = decl.isOverride,
isTransient = decl.isTransient,
type = ObjRecord.Type.Delegated,
methodId = decl.methodId
methodId = decl.methodId,
annotations = annotations
)
if (!decl.isAbstract) {
decl.initStatement?.let { cls.instanceInitializers += it }
@@ -2994,7 +3006,8 @@ class CmdDeclInstanceField(internal val constId: Int, internal val slot: Int) :
isAbstract = decl.isAbstract,
isClosed = decl.isClosed,
isOverride = decl.isOverride,
isTransient = decl.isTransient
isTransient = decl.isTransient,
annotations = decl.annotations
)
if (slot >= frame.fn.scopeSlotCount) {
val localIndex = slot - frame.fn.scopeSlotCount
@@ -3023,7 +3036,8 @@ class CmdDeclInstanceProperty(internal val constId: Int, internal val slot: Int)
isAbstract = decl.isAbstract,
isClosed = decl.isClosed,
isOverride = decl.isOverride,
isTransient = decl.isTransient
isTransient = decl.isTransient,
annotations = decl.annotations
)
if (slot >= frame.fn.scopeSlotCount) {
val localIndex = slot - frame.fn.scopeSlotCount
@@ -3076,7 +3090,8 @@ class CmdDeclInstanceDelegated(internal val constId: Int, internal val slot: Int
isAbstract = decl.isAbstract,
isClosed = decl.isClosed,
isOverride = decl.isOverride,
isTransient = decl.isTransient
isTransient = decl.isTransient,
annotations = decl.annotations
).apply {
delegate = finalDelegate
}
@@ -3253,7 +3268,14 @@ class CmdDeclExtProperty(internal val constId: Int, internal val slot: Int) : Cm
)
val getterName = extensionPropertyGetterName(decl.extTypeName, decl.property.name)
val getterWrapper = ObjExtensionPropertyGetterCallable(decl.property.name, decl.property)
frame.ensureScope().addItem(getterName, false, getterWrapper, decl.visibility, recordType = ObjRecord.Type.Fun)
frame.ensureScope().addItem(
getterName,
false,
getterWrapper,
decl.visibility,
recordType = ObjRecord.Type.Fun,
typeDecl = decl.getterTypeDecl
)
val getterLocal = resolveLocalSlotIndex(frame.fn, getterName, preferCapture = false)
if (getterLocal != null) {
frame.setObjUnchecked(frame.fn.scopeSlotCount + getterLocal, getterWrapper)
@@ -3262,7 +3284,14 @@ class CmdDeclExtProperty(internal val constId: Int, internal val slot: Int) : Cm
val setterName = extensionPropertySetterName(decl.extTypeName, decl.property.name)
val setterWrapper = ObjExtensionPropertySetterCallable(decl.property.name, decl.property)
frame.ensureScope()
.addItem(setterName, false, setterWrapper, decl.visibility, recordType = ObjRecord.Type.Fun)
.addItem(
setterName,
false,
setterWrapper,
decl.visibility,
recordType = ObjRecord.Type.Fun,
typeDecl = decl.setterTypeDecl
)
val setterLocal = resolveLocalSlotIndex(frame.fn, setterName, preferCapture = false)
if (setterLocal != null) {
frame.setObjUnchecked(frame.fn.scopeSlotCount + setterLocal, setterWrapper)
@@ -3705,9 +3734,26 @@ class CmdGetClassScope(
decl = declared
break
}
val resolved = rec ?: scope.raiseSymbolNotFound(name)
val declClass = decl ?: cls
val resolvedRec = cls.resolveRecord(scope, resolved, name, declClass)
val resolvedRec = if (rec != null) {
val declClass = decl ?: cls
cls.resolveRecord(scope, rec, name, declClass)
} else {
val metaRec = cls.objClass.getInstanceMemberOrNull(name)
if (metaRec == null || metaRec.isAbstract) {
scope.raiseSymbolNotFound(name)
}
val declClass = metaRec.declaringClass ?: cls.objClass
val resolved = cls.resolveRecord(scope, metaRec, name, declClass)
if (resolved.type == ObjRecord.Type.Fun) {
resolved.copy(
value = ObjExternCallable.fromBridge {
resolved.value.invoke(scope, cls, args, declClass)
}
)
} else {
resolved
}
}
val value = resolvedRec.value
frame.storeObjResult(dst, value)
return
@@ -24,6 +24,9 @@ import net.sergeych.lyng.parseLyng
/** Extension that converts a [Pos] (line/column) into absolute character offset in the [Source] text. */
fun Source.offsetOf(pos: Pos): Int {
if (lines.isEmpty()) return 0
if (pos.line < 0) return 0
if (pos.line >= lines.size) return text.length
var off = 0
// Sum full preceding lines + one '\n' per line (lines[] were created by String.lines())
var i = 0
@@ -31,8 +34,8 @@ fun Source.offsetOf(pos: Pos): Int {
off += lines[i].length + 1 // assume \n as separator
i++
}
off += pos.column
return off
off += pos.column.coerceIn(0, lines[pos.line].length)
return off.coerceAtMost(text.length)
}
private val reservedIdKeywords = setOf("constructor", "property")
@@ -28,6 +28,23 @@ import net.sergeych.lynon.LynonType
// Simple id generator for class identities (not thread-safe; fine for scripts)
private object ClassIdGen { var c: Long = 1L; fun nextId(): Long = c++ }
private fun DeclAnnotation.toObj(): Obj {
val namedArgs = linkedMapOf<Obj, Obj>()
for ((k, v) in named) {
namedArgs[ObjString(k)] = v
}
return ObjMap(
linkedMapOf(
ObjString("name") to ObjString(name),
ObjString("positional") to ObjImmutableList(positional),
ObjString("named") to ObjMap(namedArgs)
)
)
}
private fun annotationListObj(annotations: List<DeclAnnotation>): Obj =
ObjImmutableList(annotations.map { it.toObj() })
val ObjClassType by lazy {
object : ObjClass("Class") {
override suspend fun deserialize(scope: Scope, decoder: LynonDecoder, lynonType: LynonType?): Obj {
@@ -98,6 +115,30 @@ val ObjClassType by lazy {
val rec = cls.getInstanceMemberOrNull(name)
rec?.value ?: ObjNull
}
addFnDoc(
name = "getConstructorAnnotations",
doc = "Return preserved annotations for a constructor parameter by name as descriptor maps with fields `name`, `positional`, and `named`.",
params = listOf(ParamDoc("name", type("lyng.String"))),
returns = TypeGenericDoc(type("lyng.List"), listOf(type("lyng.Map"))),
moduleName = "lyng.stdlib"
) {
val cls = thisAs<ObjClass>()
val name = requiredArg<ObjString>(0).value
val param = cls.constructorMeta?.params?.firstOrNull { it.name == name }
annotationListObj(param?.annotations ?: emptyList())
}
addFnDoc(
name = "getMemberAnnotations",
doc = "Return preserved annotations for a member by name as descriptor maps with fields `name`, `positional`, and `named`.",
params = listOf(ParamDoc("name", type("lyng.String"))),
returns = TypeGenericDoc(type("lyng.List"), listOf(type("lyng.Map"))),
moduleName = "lyng.stdlib"
) {
val cls = thisAs<ObjClass>()
val name = requiredArg<ObjString>(0).value
val rec = cls.getInstanceMemberOrNull(name) ?: cls.classScope?.objects?.get(name)
annotationListObj(rec?.annotations ?: emptyList())
}
}
}
@@ -119,6 +160,7 @@ open class ObjClass(
var isAbstract: Boolean = false
var isClosed: Boolean = false
var isSingletonObject: Boolean = false
var logicalPackageNameOverride: String? = null
// Stable identity and simple structural version for PICs
@@ -850,6 +892,7 @@ open class ObjClass(
methodId: Int? = null,
typeDecl: net.sergeych.lyng.TypeDecl? = null,
callSignature: net.sergeych.lyng.CallSignature? = null,
annotations: List<DeclAnnotation> = emptyList(),
): ObjRecord {
// Validation of override rules: only for non-system declarations
var existing: ObjRecord? = null
@@ -952,6 +995,7 @@ open class ObjClass(
type = type,
callSignature = callSignature,
typeDecl = typeDecl,
annotations = annotations,
memberName = name,
fieldId = effectiveFieldId,
methodId = effectiveMethodId
@@ -978,7 +1022,8 @@ open class ObjClass(
type: ObjRecord.Type = ObjRecord.Type.Field,
fieldId: Int? = null,
methodId: Int? = null,
callSignature: net.sergeych.lyng.CallSignature? = null
callSignature: net.sergeych.lyng.CallSignature? = null,
annotations: List<DeclAnnotation> = emptyList()
): ObjRecord {
initClassScope()
val existing = classScope!!.objects[name]
@@ -1020,6 +1065,7 @@ open class ObjClass(
recordType = type,
isTransient = isTransient,
callSignature = callSignature,
annotations = annotations,
fieldId = effectiveFieldId,
methodId = effectiveMethodId
)
@@ -1066,7 +1112,8 @@ open class ObjClass(
isOverride: Boolean = false,
pos: Pos = Pos.builtIn,
prop: ObjProperty? = null,
methodId: Int? = null
methodId: Int? = null,
annotations: List<DeclAnnotation> = emptyList()
) {
val g = getter?.let { ObjExternCallable.fromBridge { it() } }
val s = setter?.let { ObjExternCallable.fromBridge { it(requiredArg(0)); ObjVoid } }
@@ -1075,7 +1122,8 @@ open class ObjClass(
name, finalProp, false, visibility, writeVisibility, pos, declaringClass,
isAbstract = isAbstract, isClosed = isClosed, isOverride = isOverride,
type = ObjRecord.Type.Property,
methodId = methodId
methodId = methodId,
annotations = annotations
)
}
@@ -16,6 +16,7 @@
*/
package net.sergeych.lyng.obj
import net.sergeych.lyng.DeclAnnotation
import net.sergeych.lyng.Scope
import net.sergeych.lyng.Visibility
@@ -40,6 +41,7 @@ data class ObjRecord(
var receiver: Obj? = null,
val callSignature: net.sergeych.lyng.CallSignature? = null,
val typeDecl: net.sergeych.lyng.TypeDecl? = null,
val annotations: List<DeclAnnotation> = emptyList(),
val memberName: String? = null,
val fieldId: Int? = null,
val methodId: Int? = null,
@@ -0,0 +1,742 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng.serialization
import kotlinx.datetime.LocalDate
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonNull
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.booleanOrNull
import kotlinx.serialization.json.doubleOrNull
import kotlinx.serialization.json.jsonPrimitive
import kotlinx.serialization.json.longOrNull
import net.sergeych.lyng.Arguments
import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Pos
import net.sergeych.lyng.Scope
import net.sergeych.lyng.TypeDecl
import net.sergeych.lyng.obj.Obj
import net.sergeych.lyng.obj.ObjBitBuffer
import net.sergeych.lyng.obj.ObjBool
import net.sergeych.lyng.obj.ObjBuffer
import net.sergeych.lyng.obj.ObjClass
import net.sergeych.lyng.obj.ObjDate
import net.sergeych.lyng.obj.ObjDateTime
import net.sergeych.lyng.obj.ObjEnumClass
import net.sergeych.lyng.obj.ObjEnumEntry
import net.sergeych.lyng.obj.ObjException
import net.sergeych.lyng.obj.ObjImmutableList
import net.sergeych.lyng.obj.ObjImmutableMap
import net.sergeych.lyng.obj.ObjImmutableSet
import net.sergeych.lyng.obj.ObjInstance
import net.sergeych.lyng.obj.ObjInt
import net.sergeych.lyng.obj.ObjInstant
import net.sergeych.lyng.obj.ObjList
import net.sergeych.lyng.obj.ObjMap
import net.sergeych.lyng.obj.ObjNull
import net.sergeych.lyng.obj.ObjReal
import net.sergeych.lyng.obj.ObjRecord
import net.sergeych.lyng.obj.ObjSet
import net.sergeych.lyng.obj.ObjString
import net.sergeych.lyng.obj.ObjTypeExpr
import net.sergeych.lyng.obj.ObjVoid
import net.sergeych.lyng.obj.matchesTypeDecl
import net.sergeych.lyng.requireExactCount
import net.sergeych.lyng.requireScope
import net.sergeych.lynon.BitArray
import net.sergeych.mp_tools.decodeBase64Url
import net.sergeych.mp_tools.encodeToBase64Url
import kotlin.time.Instant
private const val TYPE_KEY = "@lyng"
private const val VALUE_KEY = "value"
private const val ITEMS_KEY = "items"
private const val ENTRIES_KEY = "entries"
private const val CLASS_KEY = "class"
private const val NAME_KEY = "name"
private const val ARGS_KEY = "args"
private const val VARS_KEY = "vars"
private const val BASE64_KEY = "base64"
private const val LAST_BYTE_BITS_KEY = "lastByteBits"
private const val MESSAGE_KEY = "message"
private const val EXTRA_DATA_KEY = "extraData"
private const val STACK_TRACE_KEY = "stackTrace"
object ObjJsonClass : ObjSerializationFormatClass("Json") {
init {
addClassFn("encodeAs") {
requireExactCount(2)
val targetType = typeDeclFromJsonTarget(requireScope(), args[0])
ObjString(encodeToJsonElement(requireScope(), args[1], targetType).toString())
}
addClassFn("decodeAs") {
requireExactCount(2)
val scope = requireScope()
val targetType = typeDeclFromJsonTarget(scope, args[0])
val text = when (val encoded = args[1]) {
is ObjString -> encoded.value
else -> encoded.toString(scope).value
}
decodeFromJsonElement(scope, Json.parseToJsonElement(text), targetType)
}
}
override suspend fun encodeValue(scope: Scope, value: Obj): Obj =
ObjString(encodeToJsonElement(scope, value).toString())
override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj {
val text = when (encoded) {
is ObjString -> encoded.value
else -> encoded.toString(scope).value
}
return decodeFromJsonElement(scope, Json.parseToJsonElement(text))
}
suspend fun encodeToJsonElement(scope: Scope, value: Obj, expectedType: TypeDecl? = null): JsonElement =
UniversalJsonCodec.encode(scope, value, expectedType)
suspend fun decodeFromJsonElement(scope: Scope, element: JsonElement, expectedType: TypeDecl? = null): Obj =
UniversalJsonCodec.decode(scope, element, expectedType)
}
suspend fun Obj.toUniversalJsonElement(scope: Scope = Scope()): JsonElement =
ObjJsonClass.encodeToJsonElement(scope, this)
suspend fun decodeUniversalJsonElement(element: JsonElement, scope: Scope = Scope()): Obj =
ObjJsonClass.decodeFromJsonElement(scope, element)
private fun typeDeclFromJsonTarget(scope: Scope, target: Obj): TypeDecl = when (target) {
is ObjTypeExpr -> target.typeDecl
is ObjClass -> TypeDecl.Simple(target.className, false)
is ObjInstance -> TypeDecl.Simple(target.objClass.className, false)
is ObjString -> TypeDecl.Simple(target.value, false)
else -> scope.raiseClassCastError("Json.encodeAs/decodeAs expects a class or type expression")
}
private object UniversalJsonCodec {
suspend fun encode(scope: Scope, value: Obj, expectedType: TypeDecl? = null): JsonElement {
if (expectedType != null) {
encodeWithExpectedType(scope, value, expectedType)?.let { return it }
}
return encodeCanonical(scope, value)
}
suspend fun decode(scope: Scope, element: JsonElement, expectedType: TypeDecl? = null): Obj {
if (expectedType != null) {
if (element is JsonObject && TYPE_KEY in element) {
return ensureMatchesExpectedType(scope, decodeCanonical(scope, element), expectedType)
}
decodeWithExpectedType(scope, element, expectedType)?.let {
return ensureMatchesExpectedType(scope, it, expectedType)
}
}
return decodeCanonical(scope, element)
}
private suspend fun encodeCanonical(scope: Scope, value: Obj): JsonElement = when (value) {
ObjVoid -> tagged("void")
ObjNull -> JsonNull
is ObjBool -> JsonPrimitive(value.value)
is ObjInt -> JsonPrimitive(value.value)
is ObjReal -> if (value.value.isFinite()) {
JsonPrimitive(value.value)
} else {
tagged("real", VALUE_KEY to JsonPrimitive(value.value.toString()))
}
is ObjString -> JsonPrimitive(value.value)
is ObjDate -> tagged("date", VALUE_KEY to JsonPrimitive(value.date.toString()))
is ObjInstant -> tagged("instant", VALUE_KEY to JsonPrimitive(value.instant.toString()))
is ObjDateTime -> tagged("dateTime", VALUE_KEY to JsonPrimitive(value.toRFC3339()))
is ObjBuffer -> tagged("buffer", BASE64_KEY to JsonPrimitive(value.base64))
is ObjBitBuffer -> tagged(
"bitBuffer",
BASE64_KEY to JsonPrimitive(value.bitArray.asUByteArray().asByteArray().encodeToBase64Url()),
LAST_BYTE_BITS_KEY to JsonPrimitive(value.bitArray.lastByteBits)
)
is ObjImmutableList -> tagged("immutableList", ITEMS_KEY to JsonArray(value.toMutableList().map { encodeCanonical(scope, it) }))
is ObjList -> JsonArray(value.list.map { encodeCanonical(scope, it) })
is ObjImmutableSet -> tagged("immutableSet", ITEMS_KEY to JsonArray(value.toMutableSet().map { encodeCanonical(scope, it) }))
is ObjSet -> tagged("set", ITEMS_KEY to JsonArray(value.set.map { encodeCanonical(scope, it) }))
is ObjImmutableMap -> tagged("immutableMap", ENTRIES_KEY to encodeEntries(scope, value.map.entries.map { it.toPair() }))
is ObjMap -> encodeCanonicalMap(scope, value)
is ObjEnumEntry -> tagged(
"enum",
CLASS_KEY to JsonPrimitive(value.objClass.className),
NAME_KEY to JsonPrimitive(value.name.value)
)
is ObjException -> tagged(
"exception",
CLASS_KEY to JsonPrimitive(value.exceptionClass.className),
MESSAGE_KEY to encodeCanonical(scope, value.message),
EXTRA_DATA_KEY to encodeCanonical(scope, value.extraData),
STACK_TRACE_KEY to encodeCanonical(scope, value.getStackTrace())
)
is ObjClass -> tagged("class", NAME_KEY to JsonPrimitive(value.className))
is ObjInstance -> if (value.objClass.isSingletonObject) {
encodeCanonicalSingletonObject(scope, value)
} else {
encodeCanonicalInstance(scope, value)
}
else -> scope.raiseNotImplemented("Json.encode can't serialize ${value.objClass.className}")
}
private suspend fun decodeCanonical(scope: Scope, element: JsonElement): Obj = when (element) {
JsonNull -> ObjNull
is JsonPrimitive -> decodePrimitive(element)
is JsonArray -> ObjList(element.map { decodeCanonical(scope, it) }.toMutableList())
is JsonObject -> decodeCanonicalObject(scope, element)
}
private suspend fun encodeWithExpectedType(scope: Scope, value: Obj, expectedType: TypeDecl): JsonElement? {
if (value === ObjNull) return JsonNull
when (value) {
is ObjBool -> return JsonPrimitive(value.value)
is ObjInt -> return JsonPrimitive(value.value)
is ObjReal -> if (value.value.isFinite()) return JsonPrimitive(value.value)
is ObjString -> return JsonPrimitive(value.value)
is ObjDate -> if (isExpectedExactClass(scope, expectedType, value.objClass)) return JsonPrimitive(value.date.toString())
is ObjInstant -> if (isExpectedExactClass(scope, expectedType, value.objClass)) return JsonPrimitive(value.instant.toString())
is ObjDateTime -> if (isExpectedExactClass(scope, expectedType, value.objClass)) return JsonPrimitive(value.toRFC3339())
is ObjBuffer -> if (isExpectedExactClass(scope, expectedType, value.objClass)) return JsonPrimitive(value.base64)
is ObjBitBuffer -> if (isExpectedExactClass(scope, expectedType, value.objClass)) {
return JsonObject(
linkedMapOf(
BASE64_KEY to JsonPrimitive(value.bitArray.asUByteArray().asByteArray().encodeToBase64Url()),
LAST_BYTE_BITS_KEY to JsonPrimitive(value.bitArray.lastByteBits)
)
)
}
is ObjEnumEntry -> if (isExpectedExactClass(scope, expectedType, value.objClass)) {
return JsonPrimitive(value.name.value)
}
is ObjList -> if (expectedBaseName(expectedType) == "List") {
return JsonArray(value.list.map { encode(scope, it, expectedElementType(expectedType)) })
}
is ObjImmutableList -> if (expectedBaseName(expectedType) == "ImmutableList") {
return JsonArray(value.toMutableList().map { encode(scope, it, expectedElementType(expectedType)) })
}
is ObjSet -> if (expectedBaseName(expectedType) == "Set") {
return JsonArray(value.set.map { encode(scope, it, expectedElementType(expectedType)) })
}
is ObjImmutableSet -> if (expectedBaseName(expectedType) == "ImmutableSet") {
return JsonArray(value.toMutableSet().map { encode(scope, it, expectedElementType(expectedType)) })
}
is ObjMap -> if (expectedBaseName(expectedType) == "Map") {
return encodeTypedMap(scope, value.map, expectedKeyType(expectedType), expectedValueType(expectedType))
}
is ObjImmutableMap -> if (expectedBaseName(expectedType) == "ImmutableMap") {
return encodeTypedMap(scope, value.map, expectedKeyType(expectedType), expectedValueType(expectedType))
}
is ObjInstance -> if (isExpectedExactClass(scope, expectedType, value.objClass)) {
return if (value.objClass.isSingletonObject) encodeTypedSingletonObject(scope, value) else encodeTypedInstance(scope, value)
}
else -> Unit
}
return null
}
private suspend fun decodeWithExpectedType(scope: Scope, element: JsonElement, expectedType: TypeDecl): Obj? = when (element) {
JsonNull -> ObjNull
is JsonPrimitive -> decodePrimitiveWithExpectedType(scope, element, expectedType)
is JsonArray -> decodeArrayWithExpectedType(scope, element, expectedType)
is JsonObject -> decodeObjectWithExpectedType(scope, element, expectedType)
}
private suspend fun encodeCanonicalMap(scope: Scope, value: ObjMap): JsonElement {
if (value.map.keys.all { it is ObjString } && TYPE_KEY !in value.map.keys.map { (it as ObjString).value }) {
return JsonObject(
value.map.entries.associate { (k, v) ->
(k as ObjString).value to encodeCanonical(scope, v)
}
)
}
return tagged("map", ENTRIES_KEY to encodeEntries(scope, value.map.entries.map { it.toPair() }))
}
private suspend fun encodeTypedMap(
scope: Scope,
map: Map<Obj, Obj>,
keyType: TypeDecl?,
valueType: TypeDecl?
): JsonElement {
val stringKeys = keyType != null && expectedBaseName(keyType) == "String"
if (stringKeys && map.keys.all { it is ObjString } && TYPE_KEY !in map.keys.map { (it as ObjString).value }) {
return JsonObject(
map.entries.associate { (k, v) ->
(k as ObjString).value to encode(scope, v, valueType)
}
)
}
return JsonArray(
map.entries.map { (k, v) ->
JsonArray(listOf(encode(scope, k, keyType), encode(scope, v, valueType)))
}
)
}
private suspend fun encodeEntries(scope: Scope, entries: List<Pair<Obj, Obj>>): JsonArray =
JsonArray(entries.map { (k, v) -> JsonArray(listOf(encodeCanonical(scope, k), encodeCanonical(scope, v))) })
private suspend fun encodeCanonicalInstance(scope: Scope, value: ObjInstance): JsonElement {
val meta = value.objClass.constructorMeta
?: scope.raiseError("can't serialize non-serializable object (no constructor meta)")
val args = linkedMapOf<String, JsonElement>()
for (param in meta.params) {
if (!param.isTransient) {
args[param.name] = encodeCanonical(scope, value.readField(scope, param.name).value)
}
}
val vars = linkedMapOf<String, JsonElement>()
for ((key, record) in value.serializingVars) {
vars[key.substringAfterLast("::")] = encodeCanonical(scope, record.value)
}
return tagged(
"instance",
CLASS_KEY to JsonPrimitive(value.objClass.className),
ARGS_KEY to JsonObject(args),
VARS_KEY to JsonObject(vars)
)
}
private suspend fun encodeTypedInstance(scope: Scope, value: ObjInstance): JsonObject {
val meta = value.objClass.constructorMeta
?: scope.raiseError("can't serialize non-serializable object (no constructor meta)")
val fields = linkedMapOf<String, JsonElement>()
for (param in meta.params) {
if (!param.isTransient) {
fields[param.name] = encode(scope, value.readField(scope, param.name).value, param.type)
}
}
for ((key, record) in value.serializingVars) {
fields[key.substringAfterLast("::")] = encode(scope, record.value, record.typeDecl)
}
return JsonObject(fields)
}
private suspend fun encodeCanonicalSingletonObject(scope: Scope, value: ObjInstance): JsonElement {
val vars = linkedMapOf<String, JsonElement>()
for ((key, record) in value.serializingVars) {
vars[key.substringAfterLast("::")] = encodeCanonical(scope, record.value)
}
return tagged(
"object",
NAME_KEY to JsonPrimitive(value.objClass.className),
VARS_KEY to JsonObject(vars)
)
}
private suspend fun encodeTypedSingletonObject(scope: Scope, value: ObjInstance): JsonObject {
val vars = linkedMapOf<String, JsonElement>()
for ((key, record) in value.serializingVars) {
vars[key.substringAfterLast("::")] = encode(scope, record.value, record.typeDecl)
}
return JsonObject(vars)
}
private suspend fun decodeCanonicalObject(scope: Scope, element: JsonObject): Obj {
val tag = element[TYPE_KEY]?.jsonPrimitive?.content
if (tag == null) {
val map = linkedMapOf<Obj, Obj>()
for ((k, v) in element) {
map[ObjString(k)] = decodeCanonical(scope, v)
}
return ObjMap(map.toMutableMap())
}
return when (tag) {
"void" -> ObjVoid
"real" -> decodeTaggedReal(element)
"date" -> ObjDate(LocalDate.parse(requiredString(element, VALUE_KEY)))
"instant" -> ObjInstant(Instant.parse(requiredString(element, VALUE_KEY)))
"dateTime" -> ObjDateTime.type.invokeInstanceMethod(scope, "parseRFC3339", ObjString(requiredString(element, VALUE_KEY)))
"buffer" -> ObjBuffer(requiredString(element, BASE64_KEY).decodeBase64Url().asUByteArray())
"bitBuffer" -> ObjBitBuffer(
BitArray(
requiredString(element, BASE64_KEY).decodeBase64Url().asUByteArray(),
requiredInt(element, LAST_BYTE_BITS_KEY)
)
)
"immutableList" -> ObjImmutableList(requiredArray(element, ITEMS_KEY).map { decodeCanonical(scope, it) })
"set" -> ObjSet(requiredArray(element, ITEMS_KEY).map { decodeCanonical(scope, it) }.toMutableSet())
"immutableSet" -> ObjImmutableSet(requiredArray(element, ITEMS_KEY).map { decodeCanonical(scope, it) })
"map" -> decodeCanonicalMap(scope, requiredArray(element, ENTRIES_KEY), mutable = true)
"immutableMap" -> decodeCanonicalMap(scope, requiredArray(element, ENTRIES_KEY), mutable = false)
"class" -> resolveClass(scope, requiredString(element, NAME_KEY))
"enum" -> decodeCanonicalEnum(scope, element)
"instance" -> decodeCanonicalInstance(scope, element)
"object" -> decodeCanonicalSingletonObject(scope, element)
"exception" -> decodeCanonicalException(scope, element)
else -> scope.raiseIllegalArgument("unknown Json type tag '$tag'")
}
}
private fun decodePrimitive(element: JsonPrimitive): Obj {
element.booleanOrNull?.let { return ObjBool(it) }
if (element.isString) return ObjString(element.content)
val raw = element.content
return if (!raw.contains('.') && !raw.contains('e', ignoreCase = true)) {
element.longOrNull?.let { ObjInt.of(it) } ?: ObjReal(raw.toDouble())
} else {
ObjReal(element.doubleOrNull ?: raw.toDouble())
}
}
private suspend fun decodePrimitiveWithExpectedType(scope: Scope, element: JsonPrimitive, expectedType: TypeDecl): Obj? {
val expectedClass = expectedExactClass(scope, expectedType)
val baseName = expectedBaseName(expectedType)
return when {
expectedClass is ObjEnumClass && element.isString ->
expectedClass.invokeInstanceMethod(scope, "valueOf", ObjString(element.content))
baseName == "Bool" -> element.booleanOrNull?.let { ObjBool(it) }
baseName == "Int" -> if (!element.isString) element.longOrNull?.let { ObjInt.of(it) } else null
baseName == "Real" -> decodeExpectedReal(element)
baseName == "String" -> if (element.isString) ObjString(element.content) else null
baseName == "Date" && element.isString -> ObjDate(LocalDate.parse(element.content))
baseName == "Instant" && element.isString -> ObjInstant(Instant.parse(element.content))
baseName == "DateTime" && element.isString ->
ObjDateTime.type.invokeInstanceMethod(scope, "parseRFC3339", ObjString(element.content))
baseName == "Buffer" && element.isString -> ObjBuffer(element.content.decodeBase64Url().asUByteArray())
else -> decodePrimitive(element)
}
}
private fun decodeExpectedReal(element: JsonPrimitive): ObjReal? {
if (element.isString) {
return when (element.content) {
"NaN" -> ObjReal(Double.NaN)
"Infinity" -> ObjReal(Double.POSITIVE_INFINITY)
"-Infinity" -> ObjReal(Double.NEGATIVE_INFINITY)
else -> null
}
}
val raw = element.content
return ObjReal((element.doubleOrNull ?: raw.toDouble()))
}
private fun decodeTaggedReal(element: JsonObject): ObjReal {
val raw = requiredString(element, VALUE_KEY)
val value = when (raw) {
"NaN" -> Double.NaN
"Infinity" -> Double.POSITIVE_INFINITY
"-Infinity" -> Double.NEGATIVE_INFINITY
else -> raw.toDouble()
}
return ObjReal(value)
}
private suspend fun decodeArrayWithExpectedType(scope: Scope, element: JsonArray, expectedType: TypeDecl): Obj? {
val itemType = expectedElementType(expectedType)
return when (expectedBaseName(expectedType)) {
"List" -> ObjList(element.map { decode(scope, it, itemType) }.toMutableList())
"ImmutableList" -> ObjImmutableList(element.map { decode(scope, it, itemType) })
"Set" -> ObjSet(element.map { decode(scope, it, itemType) }.toMutableSet())
"ImmutableSet" -> ObjImmutableSet(element.map { decode(scope, it, itemType) })
"Map" -> decodeTypedMap(scope, element, mutable = true, keyType = expectedKeyType(expectedType), valueType = expectedValueType(expectedType))
"ImmutableMap" -> decodeTypedMap(scope, element, mutable = false, keyType = expectedKeyType(expectedType), valueType = expectedValueType(expectedType))
else -> null
}
}
private suspend fun decodeObjectWithExpectedType(scope: Scope, element: JsonObject, expectedType: TypeDecl): Obj? {
return when (expectedBaseName(expectedType)) {
"Map" -> decodeTypedMapObject(scope, element, mutable = true, valueType = expectedValueType(expectedType))
"ImmutableMap" -> decodeTypedMapObject(scope, element, mutable = false, valueType = expectedValueType(expectedType))
"BitBuffer" -> {
val base64 = element[BASE64_KEY]?.jsonPrimitive?.content ?: return null
val bits = element[LAST_BYTE_BITS_KEY]?.jsonPrimitive?.content?.toInt() ?: return null
ObjBitBuffer(BitArray(base64.decodeBase64Url().asUByteArray(), bits))
}
else -> {
val klass = expectedExactClass(scope, expectedType) ?: return null
when {
klass.isSingletonObject -> decodeTypedSingletonObject(scope, klass, element)
klass is ObjEnumClass -> null
else -> decodeTypedInstance(scope, klass, element)
}
}
}
}
private suspend fun decodeCanonicalMap(scope: Scope, entries: JsonArray, mutable: Boolean): Obj {
val pairs = entries.map { item ->
val pair = item as? JsonArray ?: scope.raiseIllegalArgument("map entry must be a JSON array")
if (pair.size != 2) scope.raiseIllegalArgument("map entry must contain exactly 2 items")
decodeCanonical(scope, pair[0]) to decodeCanonical(scope, pair[1])
}
return if (mutable) ObjMap(pairs.toMap().toMutableMap()) else ObjImmutableMap(pairs.toMap())
}
private suspend fun decodeTypedMap(
scope: Scope,
entries: JsonArray,
mutable: Boolean,
keyType: TypeDecl?,
valueType: TypeDecl?
): Obj {
val pairs = entries.map { item ->
val pair = item as? JsonArray ?: scope.raiseIllegalArgument("map entry must be a JSON array")
if (pair.size != 2) scope.raiseIllegalArgument("map entry must contain exactly 2 items")
decode(scope, pair[0], keyType) to decode(scope, pair[1], valueType)
}
return if (mutable) ObjMap(pairs.toMap().toMutableMap()) else ObjImmutableMap(pairs.toMap())
}
private suspend fun decodeTypedMapObject(
scope: Scope,
element: JsonObject,
mutable: Boolean,
valueType: TypeDecl?
): Obj {
val map = linkedMapOf<Obj, Obj>()
for ((k, v) in element) {
map[ObjString(k)] = decode(scope, v, valueType)
}
return if (mutable) ObjMap(map.toMutableMap()) else ObjImmutableMap(map)
}
private suspend fun decodeCanonicalEnum(scope: Scope, element: JsonObject): Obj {
val klass = resolveClass(scope, requiredString(element, CLASS_KEY))
if (klass !is ObjEnumClass) scope.raiseClassCastError("${klass.className} is not an enum")
return klass.invokeInstanceMethod(scope, "valueOf", ObjString(requiredString(element, NAME_KEY)))
}
private suspend fun decodeCanonicalInstance(scope: Scope, element: JsonObject): Obj {
val klass = resolveClass(scope, requiredString(element, CLASS_KEY))
return decodeCanonicalInstanceWithClass(scope, klass, requiredObject(element, ARGS_KEY), requiredObject(element, VARS_KEY))
}
private suspend fun decodeTypedInstance(scope: Scope, klass: ObjClass, element: JsonObject): Obj {
val meta = klass.constructorMeta
?: scope.raiseError("can't deserialize ${klass.className} from Json: no constructor meta")
val namedArgs = linkedMapOf<String, Obj>()
for (param in meta.params) {
if (param.isTransient) continue
val encoded = element[param.name]
if (encoded == null) {
if (param.defaultValue == null && !param.type.isNullable) {
scope.raiseIllegalArgument("missing constructor field '${param.name}' for ${klass.className}")
}
} else {
namedArgs[param.name] = decode(scope, encoded, param.type)
}
}
val callScope = scope.createChildScope(args = Arguments(list = emptyList(), named = namedArgs))
val instance = klass.callOn(callScope)
if (instance is ObjInstance) {
val ctorNames = meta.params.map { it.name }.toSet()
for ((name, encoded) in element) {
if (name in ctorNames) continue
val target = resolveSerializableVar(instance, name)
?: scope.raiseIllegalArgument("unknown serializable field '${klass.className}.$name'")
target.value = decode(scope, encoded, target.typeDecl)
}
}
return instance
}
private suspend fun decodeCanonicalInstanceWithClass(
scope: Scope,
klass: ObjClass,
argsObject: JsonObject,
varsObject: JsonObject
): Obj {
val meta = klass.constructorMeta
?: scope.raiseError("can't deserialize ${klass.className} from Json: no constructor meta")
val namedArgs = linkedMapOf<String, Obj>()
for (param in meta.params) {
if (param.isTransient) continue
val encoded = argsObject[param.name]
if (encoded == null) {
if (param.defaultValue == null && !param.type.isNullable) {
scope.raiseIllegalArgument("missing constructor field '${param.name}' for ${klass.className}")
}
} else {
namedArgs[param.name] = decodeCanonical(scope, encoded)
}
}
val callScope = scope.createChildScope(args = Arguments(list = emptyList(), named = namedArgs))
val instance = klass.callOn(callScope)
if (instance is ObjInstance) {
for ((name, encoded) in varsObject) {
val target = resolveSerializableVar(instance, name)
?: scope.raiseIllegalArgument("unknown serializable field '${klass.className}.$name'")
target.value = decodeCanonical(scope, encoded)
}
}
return instance
}
private suspend fun decodeCanonicalSingletonObject(scope: Scope, element: JsonObject): Obj {
val instance = resolveObject(scope, requiredString(element, NAME_KEY))
val varsObject = requiredObject(element, VARS_KEY)
for ((name, encoded) in varsObject) {
val target = resolveSerializableVar(instance, name)
?: scope.raiseIllegalArgument("unknown serializable field '${instance.objClass.className}.$name'")
target.value = decodeCanonical(scope, encoded)
}
return instance
}
private suspend fun decodeTypedSingletonObject(scope: Scope, klass: ObjClass, element: JsonObject): Obj {
val instance = resolveObject(scope, klass.className)
for ((name, encoded) in element) {
val target = resolveSerializableVar(instance, name)
?: scope.raiseIllegalArgument("unknown serializable field '${instance.objClass.className}.$name'")
target.value = decode(scope, encoded, target.typeDecl)
}
return instance
}
private suspend fun decodeCanonicalException(scope: Scope, element: JsonObject): Obj {
val klass = resolveClass(scope, requiredString(element, CLASS_KEY))
if (klass !is ObjException.Companion.ExceptionClass) {
scope.raiseClassCastError("${klass.className} is not an exception class")
}
val message = decodeCanonical(scope, requireElement(element, MESSAGE_KEY)) as? ObjString
?: scope.raiseClassCastError("exception message must be a string")
val extraData = decodeCanonical(scope, requireElement(element, EXTRA_DATA_KEY))
val stackTrace = decodeCanonical(scope, requireElement(element, STACK_TRACE_KEY)) as? ObjList
?: scope.raiseClassCastError("exception stackTrace must be a list")
return ObjException(klass, scope, message, extraData, stackTrace)
}
private suspend fun ensureMatchesExpectedType(scope: Scope, value: Obj, expectedType: TypeDecl): Obj {
if (!matchesTypeDecl(scope, value, expectedType)) {
scope.raiseClassCastError("decoded Json value of type ${value.objClass.className} does not match expected type ${typeName(expectedType)}")
}
return value
}
private fun resolveSerializableVar(instance: ObjInstance, name: String): ObjRecord? =
instance.serializingVars[name]
?: instance.serializingVars.entries.singleOrNull { it.key.substringAfterLast("::") == name }?.value
private suspend fun resolveClass(scope: Scope, className: String): ObjClass {
scope.get(className)?.value?.let {
if (it is ObjClass) return it
if (it is ObjInstance && it.objClass.className == className) return it.objClass
scope.raiseClassCastError("Expected class $className, got ${it.objClass.className}")
}
val resolved = scope.resolveQualifiedIdentifier(className)
if (resolved is ObjClass) return resolved
if (resolved is ObjInstance && resolved.objClass.className == className) return resolved.objClass
scope.raiseClassCastError("Expected class $className, got ${resolved.objClass.className}")
return resolved as ObjClass
}
private suspend fun resolveObject(scope: Scope, objectName: String): ObjInstance {
scope.get(objectName)?.value?.let {
if (it is ObjInstance) return it
scope.raiseClassCastError("Expected object $objectName, got ${it.objClass.className}")
}
if (objectName.contains('.')) {
val resolved = scope.resolveQualifiedIdentifier(objectName)
val inst = resolved as? ObjInstance
if (inst != null) return inst
scope.raiseClassCastError("Expected object $objectName, got ${resolved.objClass.className}")
}
scope.raiseSymbolNotFound(objectName)
}
private suspend fun expectedExactClass(scope: Scope, expectedType: TypeDecl): ObjClass? {
val nonNullable = nonNullableType(expectedType)
val className = when (nonNullable) {
is TypeDecl.Simple -> nonNullable.name
is TypeDecl.Generic -> nonNullable.name
else -> return null
}
return resolveClass(scope, className)
}
private suspend fun isExpectedExactClass(scope: Scope, expectedType: TypeDecl, actualClass: ObjClass): Boolean =
expectedExactClass(scope, expectedType) == actualClass
private fun expectedBaseName(expectedType: TypeDecl?): String? {
val nonNullable = expectedType?.let { nonNullableType(it) } ?: return null
return when (nonNullable) {
is TypeDecl.Simple -> nonNullable.name.substringAfterLast('.')
is TypeDecl.Generic -> nonNullable.name.substringAfterLast('.')
else -> null
}
}
private fun expectedTypeArgs(expectedType: TypeDecl?): List<TypeDecl> = when (val nonNullable = expectedType?.let { nonNullableType(it) }) {
is TypeDecl.Generic -> nonNullable.args
else -> emptyList()
}
private fun expectedElementType(expectedType: TypeDecl?): TypeDecl? = expectedTypeArgs(expectedType).getOrNull(0)
private fun expectedKeyType(expectedType: TypeDecl?): TypeDecl? = expectedTypeArgs(expectedType).getOrNull(0)
private fun expectedValueType(expectedType: TypeDecl?): TypeDecl? = expectedTypeArgs(expectedType).getOrNull(1)
private fun nonNullableType(type: TypeDecl): TypeDecl = when (type) {
is TypeDecl.Function -> type.copy(nullable = false)
is TypeDecl.Ellipsis -> type.copy(nullable = false)
is TypeDecl.TypeVar -> type.copy(nullable = false)
is TypeDecl.Union -> type.copy(nullable = false)
is TypeDecl.Intersection -> type.copy(nullable = false)
is TypeDecl.Simple -> TypeDecl.Simple(type.name, false)
is TypeDecl.Generic -> TypeDecl.Generic(type.name, type.args, false)
else -> type
}
private fun typeName(type: TypeDecl): String = when (type) {
TypeDecl.TypeAny -> "Any"
TypeDecl.TypeNullableAny -> "Any?"
is TypeDecl.Simple -> type.name + if (type.isNullable) "?" else ""
is TypeDecl.Generic -> buildString {
append(type.name)
append('<')
append(type.args.joinToString(",") { typeName(it) })
append('>')
if (type.isNullable) append('?')
}
is TypeDecl.Function -> "Callable"
is TypeDecl.Ellipsis -> typeName(type.elementType) + "..."
is TypeDecl.TypeVar -> type.name + if (type.isNullable) "?" else ""
is TypeDecl.Union -> type.options.joinToString(" | ") { typeName(it) }
is TypeDecl.Intersection -> type.options.joinToString(" & ") { typeName(it) }
}
private fun tagged(type: String, vararg fields: Pair<String, JsonElement>): JsonObject =
JsonObject(linkedMapOf(TYPE_KEY to JsonPrimitive(type), *fields))
private fun requiredString(element: JsonObject, key: String): String =
requireElement(element, key).jsonPrimitive.content
private fun requiredInt(element: JsonObject, key: String): Int =
requireElement(element, key).jsonPrimitive.content.toInt()
private fun requiredArray(element: JsonObject, key: String): JsonArray =
requireElement(element, key) as? JsonArray
?: throw IllegalArgumentException("field '$key' must be a JSON array")
private fun requiredObject(element: JsonObject, key: String): JsonObject =
requireElement(element, key) as? JsonObject
?: throw IllegalArgumentException("field '$key' must be a JSON object")
private fun requireElement(element: JsonObject, key: String): JsonElement =
element[key] ?: throw IllegalArgumentException("missing field '$key'")
}
@@ -0,0 +1,59 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng.serialization
import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Scope
import net.sergeych.lyng.miniast.addConstDoc
import net.sergeych.lyng.miniast.type
import net.sergeych.lyng.obj.Obj
import net.sergeych.lyng.obj.ObjClass
import net.sergeych.lyng.requireOnlyArg
import net.sergeych.lyng.requireScope
abstract class ObjSerializationFormatClass(
className: String
) : ObjClass(className) {
abstract suspend fun encodeValue(scope: Scope, value: Obj): Obj
abstract suspend fun decodeValue(scope: Scope, encoded: Obj): Obj
init {
addClassFn("encode") {
encodeValue(requireScope(), requireOnlyArg())
}
addClassFn("decode") {
decodeValue(requireScope(), requireOnlyArg())
}
}
}
suspend fun ModuleScope.bindSerializationFormat(
format: ObjSerializationFormatClass,
exportName: String = format.className,
doc: String = "${format.className} serialization format."
): ObjSerializationFormatClass {
addConstDoc(
name = exportName,
value = format,
doc = doc,
type = type("lyng.Class")
)
return format
}
@@ -18,14 +18,13 @@
package net.sergeych.lynon
import net.sergeych.lyng.Scope
import net.sergeych.lyng.requireOnlyArg
import net.sergeych.lyng.requireScope
import net.sergeych.lyng.serialization.ObjSerializationFormatClass
import net.sergeych.lyng.obj.*
// Most often used types:
object ObjLynonClass : ObjClass("Lynon") {
object ObjLynonClass : ObjSerializationFormatClass("Lynon") {
suspend fun encodeAny(scope: Scope, obj: Obj): ObjBitBuffer {
val bout = MemoryBitOutput()
@@ -41,15 +40,9 @@ object ObjLynonClass : ObjClass("Lynon") {
return deserializer.decodeAny(scope)
}
init {
addClassConst("test", ObjString("test_const"))
addClassFn("encode") {
encodeAny(requireScope(), requireOnlyArg<Obj>())
}
addClassFn("decode") {
decodeAny(requireScope(), requireOnlyArg<Obj>())
}
}
override suspend fun encodeValue(scope: Scope, value: Obj): Obj = encodeAny(scope, value)
override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj = decodeAny(scope, encoded)
}
/**
+267
View File
@@ -32,6 +32,8 @@ import kotlinx.serialization.json.encodeToJsonElement
import net.sergeych.lyng.*
import net.sergeych.lyng.obj.*
import net.sergeych.lyng.pacman.InlineSourcesImportProvider
import net.sergeych.lyng.serialization.ObjSerializationFormatClass
import net.sergeych.lyng.serialization.bindSerializationFormat
import net.sergeych.lyng.thisAs
import net.sergeych.mp_tools.globalDefer
import net.sergeych.tools.bm
@@ -4663,6 +4665,243 @@ class ScriptTest {
)
}
@Test
fun testUniversalJsonRoundTrip() = runTest {
eval(
"""
import lyng.serialization
import lyng.time
enum Color { Red, Green }
class Point(foo,bar) {
var z = 42
}
val point = Point(1,2)
val color = Color.Green
point.z = 99
val value = List(
point,
Map([1, "one"], ["two", 2]),
Set(1,2,3),
"hello".encodeUtf8(),
Date(2026,4,15),
color
)
val restored = Json.decode(Json.encode(value))
assertEquals(value, restored)
assertEquals(99, restored[0].z)
assertEquals("one", restored[1][1])
assertEquals(true, restored[2].contains(3))
assertEquals("hello", restored[3].decodeUtf8())
assertEquals("2026-04-15", restored[4].toString())
assertEquals(Color.Green, restored[5])
""".trimIndent()
)
}
@Test
fun testJsonDecodePlainJsonString() = runTest {
val decoded = eval(
"""
import lyng.serialization
Json.decode("{\"foo\":[1,true,\"bar\"]}")
""".trimIndent()
)
assertEquals("""{"foo":[1,true,"bar"]}""", decoded.toJson().toString())
}
@Test
fun testUniversalJsonExceptionRoundTrip() = runTest {
eval(
"""
import lyng.serialization
class MyException : Exception("boom")
val ex = MyException()
val restored = Json.decode(Json.encode(ex))
assert(restored is MyException)
assert(restored is Exception)
assertEquals(ex.message, restored.message)
assert(restored.stackTrace.size > 0)
""".trimIndent()
)
}
@Test
fun testUniversalJsonSingletonObjectRoundTrip() = runTest {
eval(
"""
import lyng.serialization
object Counter {
var value = 1
}
Counter.value = 77
val restored = Json.decode(Json.encode(Counter))
assertEquals(Counter, restored)
assertEquals(77, Counter.value)
assertEquals(77, restored.value)
""".trimIndent()
)
}
@Test
fun testCanonicalJsonUsesTraditionalObjectsForStringKeyMaps() = runTest {
eval(
"""
import lyng.serialization
val value = Map(["foo", 1], ["bar", 2])
val encoded = Json.encode(value)
assertEquals("{\"foo\":1,\"bar\":2}", encoded)
val restored = Json.decode(encoded)
assertEquals(value, restored)
assertEquals(1, restored["foo"])
""".trimIndent()
)
}
@Test
fun testTypedJsonRoundTripOmitsExactTypeInformation() = runTest {
eval(
"""
import lyng.serialization
closed class Point(x: Int, y: Int)
val point = Point(0, 1)
val encoded = Json.encodeAs(Point, point)
assertEquals("{\"x\":0,\"y\":1}", encoded)
val restored = Json.decodeAs(Point, encoded)
assertEquals(point, restored)
""".trimIndent()
)
}
@Test
fun testTypedJsonUsesTraditionalObjectsForStringKeyMaps() = runTest {
eval(
"""
import lyng.serialization
closed class Payload(values: Map<String, Int>)
val value = Payload(Map(["foo", 1], ["bar", 2]))
val encoded = Json.encodeAs(Payload, value)
assertEquals("{\"values\":{\"foo\":1,\"bar\":2}}", encoded)
val restored = Json.decodeAs(Payload, encoded)
assertEquals(value, restored)
assertEquals(2, restored.values["bar"])
""".trimIndent()
)
}
@Test
fun testTypedJsonRecursesUsingDeclaredFieldTypes() = runTest {
eval(
"""
import lyng.serialization
closed class Point(x: Int, y: Int)
closed class Segment(a: Point, b: Point)
val value = Segment(Point(0, 1), Point(2, 3))
val encoded = Json.encodeAs(Segment, value)
assertEquals("{\"a\":{\"x\":0,\"y\":1},\"b\":{\"x\":2,\"y\":3}}", encoded)
val restored = Json.decodeAs(Segment, encoded)
assertEquals(value, restored)
""".trimIndent()
)
}
@Test
fun testTypedJsonHandlesNullableFields() = runTest {
eval(
"""
import lyng.serialization
closed class Point(x: Int, y: Int)
closed class MaybePoint(point: Point?, label: String?)
val value = MaybePoint(null, "origin")
val encoded = Json.encodeAs(MaybePoint, value)
assertEquals("{\"point\":null,\"label\":\"origin\"}", encoded)
assertEquals(value, Json.decodeAs(MaybePoint, encoded))
val value2 = MaybePoint(Point(3, 4), null)
val encoded2 = Json.encodeAs(MaybePoint, value2)
assertEquals("{\"point\":{\"x\":3,\"y\":4},\"label\":null}", encoded2)
assertEquals(value2, Json.decodeAs(MaybePoint, encoded2))
""".trimIndent()
)
}
@Test
fun testTypedJsonKeepsSubtypeTagsWhenDeclaredTypeIsWider() = runTest {
eval(
"""
import lyng.serialization
class Base(baseX: Int)
class Derived(derivedX: Int, z: Int): Base(derivedX)
closed class Holder(item: Base)
val value = Holder(Derived(1, 2))
val encoded = Json.encodeAs(Holder, value)
val restored = Json.decodeAs(Holder, encoded)
assert(restored.item is Derived)
assertEquals(2, restored.item.z)
""".trimIndent()
)
}
@Test
fun testTypedJsonUsesEntriesForNonStringKeyMaps() = runTest {
eval(
"""
import lyng.serialization
closed class Numbered(values: Map<Int, String>)
val value = Numbered(Map([1, "one"], [2, "two"]))
val encoded = Json.encodeAs(Numbered, value)
assertEquals("{\"values\":[[1,\"one\"],[2,\"two\"]]}", encoded)
val restored = Json.decodeAs(Numbered, encoded)
assertEquals(value, restored)
assertEquals("one", restored.values[1])
""".trimIndent()
)
}
@Test
fun testTypedJsonOmitsEnumTagsWhenEnumTypeIsKnown() = runTest {
eval(
"""
import lyng.serialization
enum Color { Red, Green }
closed class Paint(color: Color)
val value = Paint(Color.Green)
val encoded = Json.encodeAs(Paint, value)
assertEquals("{\"color\":\"Green\"}", encoded)
val restored = Json.decodeAs(Paint, encoded)
assertEquals(value, restored)
assertEquals(Color.Green, restored.color)
""".trimIndent()
)
}
@Serializable
data class TestJson2(
val value: Int,
@@ -4722,6 +4961,34 @@ class ScriptTest {
assertEquals(TestJson4(TestEnum.One), x)
}
@Test
fun testExternalSerializationFormatRegistration() = runTest {
val im = Script.defaultImportManager.copy()
im.addPackage("test.formats") { module ->
module.bindSerializationFormat(
object : ObjSerializationFormatClass("Reverse") {
override suspend fun encodeValue(scope: Scope, value: Obj): Obj =
ObjString(value.toString(scope).value.reversed())
override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj {
val text = (encoded as? ObjString)?.value
?: scope.raiseClassCastError("Reverse.decode expects String")
return ObjString(text.reversed())
}
},
doc = "Simple test format that reverses strings."
)
}
val scope = im.newStdScope()
scope.eval(
"""
import test.formats
assertEquals("cba", Reverse.encode("abc"))
assertEquals("abc", Reverse.decode("cba"))
""".trimIndent()
)
}
@Test
fun testStringLast() = runTest {
eval(
@@ -104,4 +104,39 @@ class TypeInferenceTest {
Pool(2).closeAll()
""".trimIndent())
}
@Test
fun testIterableFirstPreservesElementTypeForBlockReturnInference() = runBlocking<Unit> {
eval("""
class Item(title: String)
fun restored() {
val values = [Item("ok")]
values.first
}
val item = restored()
assertEquals("ok", item.title)
""".trimIndent())
}
@Test
fun testCallableLocalInitializedFromFunctionCallPreservesReturnType() = runBlocking<Unit> {
eval("""
fun makeAdder(base) {
return { x -> x + base + 0.5 }
}
fun run() {
val add = makeAdder(2)
val value = add(3) + 4
assert(value is Real)
value
}
val result = run()
assert(result is Real)
assertEquals(9.5, result)
""".trimIndent())
}
}
@@ -0,0 +1,68 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng
import kotlinx.coroutines.test.runTest
import net.sergeych.lyng.obj.ObjInt
import kotlin.test.Test
import kotlin.test.assertEquals
class DeclAnnotationIntrospectionTest {
@Test
fun classAnnotationQueriesExposeConstructorAndMemberAnnotations() = runTest {
val scope = Scope()
val result = scope.eval(
"""
val suffix = "!"
object Marker
class Sample(
@Transient @Tag(1, label: "ctor", extra: suffix) val x: Int
) {
@Transient @DbDecodeWith(Marker)
var y: Int = 10
}
val ctorAnnotations: ImmutableList<Map<String, Object>> = Sample.getConstructorAnnotations("x")
val ctorTag: Map<String, Object> = ctorAnnotations[1]
val ctorPositional: ImmutableList<Object> = ctorTag["positional"] as ImmutableList<Object>
val ctorNamed: Map<String, Object> = ctorTag["named"] as Map<String, Object>
assertEquals(2, ctorAnnotations.size)
assertEquals("Transient", ctorAnnotations[0]["name"])
assertEquals("Tag", ctorTag["name"])
assertEquals(1, ctorPositional[0])
assertEquals("ctor", ctorNamed["label"])
assertEquals("!", ctorNamed["extra"])
val memberAnnotations: ImmutableList<Map<String, Object>> = Sample.getMemberAnnotations("y")
val memberDecodeWith: Map<String, Object> = memberAnnotations[1]
val memberPositional: ImmutableList<Object> = memberDecodeWith["positional"] as ImmutableList<Object>
assertEquals(2, memberAnnotations.size)
assertEquals("Transient", memberAnnotations[0]["name"])
assertEquals("DbDecodeWith", memberDecodeWith["name"])
assertEquals(Marker, memberPositional[0])
memberAnnotations.size + ctorAnnotations.size
""".trimIndent()
) as ObjInt
assertEquals(4L, result.value)
}
}
+19 -1
View File
@@ -14,6 +14,24 @@ extern class NotImplementedException
/* Raised when an awaited asynchronous task was cancelled before producing a value. */
extern class CancellationException : Exception
/* Runtime metaobject describing a class. */
extern class Class {
/* Full name of this class including package if available. */
val className: String
/* Simple name of this class (without package). */
val name: String
/* Declared instance fields of this class and its ancestors (C3 order), without duplicates. */
val fields: List<String>
/* Declared instance methods of this class and its ancestors (C3 order), without duplicates. */
val methods: List<String>
/* Lookup a member by name in this class (including ancestors) and return it, or null if absent. */
fun get(name: String): Object?
/* Preserved annotations for a constructor parameter as descriptor maps with keys `name`, `positional`, and `named`. */
fun getConstructorAnnotations(name: String): ImmutableList<Map<String, Object>>
/* Preserved annotations for a member as descriptor maps with keys `name`, `positional`, and `named`. */
fun getMemberAnnotations(name: String): ImmutableList<Map<String, Object>>
}
/* A handle to a running asynchronous task. */
extern class Deferred {
/* Cancel the task if it is still active. Safe to call multiple times. */
@@ -648,4 +666,4 @@ class LaunchPool(maxWorkers, maxQueueSize = Channel.UNLIMITED) {
w.await()
}
}
}
}
+351
View File
@@ -0,0 +1,351 @@
# ResultSet typed decode API
Status: draft design note
## Goal
Extend `lyng.io.db` with row deserialization into ordinary Lyng objects using the new typed serialization-style API naming.
Primary use case:
```lyng
class Point(x: Real, y: Real)
val point = db.transaction { tx ->
tx.select(
"select row as x, col as y from data where not is_deleted"
).decodeAs<Point>().first
}
```
## Agreed API
Use `decodeAs<T>()` as the only public API form in v1.
Rationale:
- matches the new typed serialization naming (`Json.decodeAs(...)`)
- communicates decoding/materialization, not casting
- keeps the common case strongly typed and chain-friendly
- avoids adding a second runtime-type overload before it is needed
Planned Lyng-facing declarations:
```lyng
extern class SqlRow {
fun decodeAs<T>(): T
}
extern class ResultSet : Iterable<SqlRow> {
fun decodeAs<T>(): Iterable<T>
}
```
## Lifetime semantics
`ResultSet.decodeAs<T>()` returns a transaction-scoped iterable view over the underlying result set.
Rules:
- the returned iterable must not be used after the owning transaction ends
- decoded objects created during iteration are detached ordinary Lyng objects
- to keep decoded values after the transaction, materialize them inside the transaction
- normal materialization forms are `toList()`, `first`, `findFirst`, or manual iteration
Valid:
```lyng
val points = db.transaction { tx ->
tx.select("select x, y from point")
.decodeAs<Point>()
.toList()
}
```
Invalid:
```lyng
val decoded = db.transaction { tx ->
tx.select("select x, y from point").decodeAs<Point>()
}
decoded.first
```
## ResultSet shape
`ResultSet.decodeAs<T>()` should preserve the current `ResultSet` paradigm:
- `ResultSet` stays the row-producing source
- `decodeAs<T>()` is a projection from `Iterable<SqlRow>` to `Iterable<T>`
- no new DB-specific collection type is introduced in v1
Implementation-wise, `ResultSet.decodeAs<T>()` can be defined as a lazy iterable that decodes each row via `SqlRow.decodeAs<T>()`.
## Mapping discussion to finalize
The following mapping behavior still needs explicit design decisions:
- how constructor parameters are matched from columns
- whether matching is case-insensitive
- whether mutable serializable fields are populated after constructor call
- treatment of default constructor values
- treatment of nullable vs non-nullable targets
- behavior for missing columns
- behavior for extra columns
- behavior for duplicate/ambiguous column labels
- whether `onDeserialized()` is called after row decode
- whether v1 supports only flat object decode or also nested shapes
## Current direction for mapping
Current likely direction, not finalized yet:
- constructor parameters map by column label
- matching is case-insensitive, consistent with `SqlRow["name"]`
- after constructor call, remaining matching serializable mutable fields may be assigned
- missing required non-null constructor values fail
- missing nullable constructor parameters become `null`
- defaulted constructor parameters use their defaults when the column is absent
- ambiguous duplicate column labels fail
- extra columns likely fail in strict mode for v1
- `onDeserialized()` likely should run after the object is fully populated
- v1 should likely stay flat and avoid nested/prefix-based mapping
## Projection/conversion rules
### General principle
Row decoding should be strict and predictable.
It should not globally treat every SQL string column as serialized JSON or every binary column as Lynon.
That would be too implicit:
- ordinary text columns are common and must stay ordinary text by default
- ordinary binary/blob columns are common and must stay raw binary by default
- automatic format decoding should happen only when there is a clear signal
### Proposed conversion precedence
For each constructor parameter or serializable mutable field:
1. resolve the source column by name
2. if the source value already matches the target type, use it directly
3. if an explicit DB decoding attribute is present on the target member, apply that decoding rule
4. otherwise, if the column metadata clearly indicates a special encoded DB type and the target is not the raw DB carrier type, apply the built-in format rule
5. otherwise fail with a decode/type mismatch error
### Direct match
Direct match means the row value is already assignable to the target type after the normal SQL backend conversion.
Examples:
- SQL numeric column already surfaced as `Int`/`Real`/`Decimal`
- SQL bool column surfaced as `Bool`
- SQL date/time column surfaced as `Date`, `DateTime`, `Instant`
- SQL text column surfaced as `String`
- SQL binary column surfaced as `Buffer`
These should not trigger any extra JSON/Lynon decoding.
### Built-in encoded-column rules
Current likely direction:
- JSON/JSONB-like columns should decode through typed canonical `Json` when the target is not `String`
- binary columns should decode through `Lynon` when the target is not `Buffer`
This implies the current default:
- string -> non-string is eligible for automatic typed `Json` decode only when the column metadata says the DB column is JSON-like
- binary -> non-binary is decoded through `Lynon`
- binary -> `Buffer` stays raw `Buffer`
Examples:
- PostgreSQL `json` / `jsonb` column into `Point` -> use typed `Json` decode
- PostgreSQL `jsonb` column into `Map<String, Object?>` -> use typed `Json` decode
- plain `text` / `varchar` column into `Point` -> fail unless explicitly annotated
- `bytea` / `blob` column into `Buffer` -> direct match, no Lynon decode
- `bytea` / `blob` column into `Point` -> decode with `Lynon`
### Attribute-based explicit decoding
Common explicit attributes look useful:
- `@DbJson`
- `@DbLynon`
Applied to constructor parameters and serializable mutable fields.
Meaning:
- `@DbJson` means decode the column value as typed canonical JSON into the target member type
- `@DbLynon` means decode the column value as Lynon into the target member type
Example:
```lyng
class Record(
id: Int,
@DbJson payload: Payload,
@DbLynon cachedState: CacheEntry
)
```
This keeps the common DB formats easy to use without making plain `String` or `Buffer` columns magical.
Implementation note:
- declaration metadata now preserves evaluated constructor-parameter and class-member annotation arguments
- annotation arguments are evaluated once at declaration creation time and retained for the lifetime of the declaration
- `@DbDecodeWith(...)` now uses that preserved metadata path
### Generic custom decoder hook
A generic hook is useful too, but it should be adapter-based, not lambda-based.
Planned shape:
- `@DbDecodeWith(adapter)`
- `adapter` should be an instance of a dedicated interface such as `DbFieldAdapter`
Reason:
- a named adapter interface is easier to document and evolve than arbitrary callables
- it gives us room for richer decoding context without baking ad-hoc callable signatures into annotations
- it keeps the DB mapping API explicit and self-describing
Current design direction:
```lyng
interface DbFieldAdapter {
fun decode(rawValue: Object?, column: SqlColumn, row: SqlRow, targetType: Object): Object? =
throw NotImplementedException("DB field adapter decode is not implemented")
fun encode(value: Object?, targetType: Object): Object? =
throw NotImplementedException("DB field adapter encode is not implemented")
}
```
Decided:
- `decode(...)` should receive the target type
- adapters may be any ordinary instance, not only singleton objects
- the same abstraction should later support symmetric `encode(...)`
- adapter result must be checked against the target member type after decoding
Still open before full implementation:
- exact annotation shape for `@DbDecodeWith(...)`
- whether target member name should also be passed
- whether `targetType` should later get a more specific declaration type than plain `Object`
Implemented in the current design:
- `@DbDecodeWith(adapter)` on constructor parameters
- `@DbDecodeWith(adapter)` on class-body fields/properties participating in `decodeAs<T>()`
Future improvement:
- compiler warning when preserved annotation metadata captures runtime state/closures
- extend preserved annotation metadata beyond constructor parameters and class members to functions and top-level declarations
### Arrays and maps
Arrays and maps should not get DB-specific bespoke mapping in v1 unless they are coming through a recognized encoded format.
Reason:
- portable SQL array/map support is backend-specific and inconsistent
- JSON columns already give us a portable representation for `List` and `Map`
- adding DB-native array semantics now would complicate the contract too early
So in v1:
- if the backend already surfaces a value that directly matches the target type, use it
- otherwise `List` / `Map` reconstruction should happen via `@DbJson` or recognized JSON-like column metadata
### Recommended v1 policy
Current recommended projection policy:
- direct type match first
- then explicit member attribute (`@DbJson`, `@DbLynon`)
- then metadata-driven JSON decode for recognized JSON-like DB columns
- then Lynon decode for binary columns when the target is not `Buffer`
- no implicit JSON decode for arbitrary text columns
- fail on anything else
## Write-side SQL object expansion
The symmetric write-side convenience should be explicit and declaration-driven, but it should not attempt semantic SQL analysis.
Agreed v1 surface:
- `@cols(?1)` expands one object argument to projected column names
- `@vals(?1)` expands the same object argument to matching placeholders and encoded bind values
- `@set(?1)` expands the same object argument to `column = ?` pairs and encoded bind values
- each macro accepts an optional `except:` filter, for example `@set(?1 except: "id", "updatedAt")`
Examples:
```lyng
tx.execute(
"insert into item(@cols(?1)) values(@vals(?1))",
item
)
tx.execute(
"update item set @set(?1) where id = ?2",
item,
item.id
)
```
Rules:
- once a clause uses `@cols`, `@vals`, or `@set`, plain sequential `?` placeholders are not allowed in the same clause
- non-expanded parameters in macro clauses must use explicit indexed placeholders such as `?2`
- the same object argument may be referenced multiple times
- object expansion is based on declaration metadata, not SQL metadata
- v1 excludes `@Transient` and `@DbExcept` fields automatically
- `except:` excludes additional fields for one specific macro use
### Write-side field encoding policy
Write-side encoding cannot rely on DB column type inference, so non-trivial field serialization must be explicit.
For each projected field:
1. if the value is already directly DB-bindable, bind it as-is
2. else if `@DbJson` is present, encode to canonical JSON text
3. else if `@DbLynon` is present, encode to Lynon binary
4. else if `@DbSerializeWith(adapter)` is present, call `adapter.encode(value, targetType)`
5. else fail with `SqlUsageException`
Direct DB-bindable values in v1:
- `null`
- `Bool`
- `Int`, `Real`, `Decimal`
- `String`
- `Buffer`
- `Date`, `DateTime`, `Instant`
This is intentionally stricter than decode-side behavior. On writes, there is no portable, reliable way to infer the intended target DB representation from SQL text alone.
### Adapter role
`DbFieldAdapter` is now symmetric by design:
- `decode(rawValue, column, row, targetType)` is used by `decodeAs<T>()`
- `encode(value, targetType)` is used by SQL object expansion
The adapter instance is captured in preserved declaration annotation metadata, not passed ad hoc at the call site.
Future task:
- consider warnings or lints for risky annotation captures such as stateful adapters or closure-capturing instances
+600
View File
@@ -0,0 +1,600 @@
# Proposal: Minimal HTTP/1.1 + WebSocket Server For `lyngio`
Status: Draft
Date: 2026-04-26
Owner: `lyngio`
## Context
`lyngio` already provides:
- HTTP client support via `lyng.io.http`
- WebSocket client support via `lyng.io.ws`
- raw TCP/UDP transport via `lyng.io.net`
The current transport layer is already multiplatform and exposes a small common Kotlin interface:
- `LyngTcpSocket`
- `LyngTcpServer`
- `LyngNetEngine`
This makes it practical to add a minimal server implementation in pure Kotlin without introducing a second public networking model.
The intended deployment model for this server is:
- behind a frontend proxy such as nginx
- no TLS termination in `lyngio`
- no HTTP/2 in `lyngio` v1
- minimal, strict HTTP/1.1 subset
- classic HTTP/1.1 WebSocket upgrade support
This proposal deliberately does **not** attempt to implement HTTP/2. That work is substantially larger because it requires binary framing, stream multiplexing, HPACK, and flow control. For the intended deployment model, a frontend proxy can provide TLS and public HTTP/2 while `lyngio` speaks HTTP/1.1 on the backend.
## Goals
- Add a minimal HTTP server implementation in pure Kotlin.
- Keep the implementation compatible with Kotlin Multiplatform common code constraints.
- Reuse the existing `lyngio.net` TCP transport layer.
- Support a strict, useful HTTP/1.1 subset.
- Support classic WebSocket upgrade from HTTP/1.1.
- Keep the API and implementation small enough to be auditable and testable.
- Preserve room for later richer server APIs or JVM-specific backends.
## Non-goals
- HTTP/2
- TLS
- ALPN
- proxy protocol support
- request pipelining
- chunked request bodies
- HTTP trailers
- content compression
- multipart/form-data parsing
- range requests
- streaming request bodies in v1
- streaming response bodies in v1
- WebSocket extensions
- WebSocket subprotocol negotiation in v1
- exposing Ktor server APIs or types
## Design principles
### 1. Common-code first
The implementation should live primarily in `commonMain` and depend only on existing common abstractions built on top of `LyngTcpSocket` and `LyngTcpServer`.
### 2. Strict subset over broad tolerance
The server should reject unsupported or ambiguous protocol constructs instead of trying to be maximally permissive.
This reduces complexity, avoids parser edge cases, and makes connection reuse easier to reason about.
### 3. Small surface area
The first version should only implement what is needed for:
- ordinary backend HTTP request/response handling behind a proxy
- WebSocket upgrade and session handling
- persistent HTTP/1.1 connections when message framing is unambiguous
### 4. Frontend proxy assumption
The server is expected to run behind nginx or a similar reverse proxy that can provide:
- TLS termination
- public HTTP/2 if needed
- request filtering and size limiting
- buffering and slow-client protection
- optional compression and edge-specific behavior
## Proposed package
Add a new internal package:
- `net.sergeych.lyngio.http.server`
This proposal defines an internal Kotlin API first. Lyng-facing scripting bindings are explicitly out of scope for the first phase.
## Supported HTTP request subset
### Request line
Accepted format:
- `METHOD SP request-target SP HTTP/1.1`
Rules:
- request line must split into exactly 3 parts
- `METHOD` must be a non-empty HTTP token
- version must be exactly `HTTP/1.1`
- request target must be origin-form only
Accepted request-target examples:
- `/`
- `/hello`
- `/hello/world?x=1&y=2`
Rejected request-target forms:
- absolute-form: `http://example.com/x`
- authority-form
- asterisk-form: `*`
### Methods
The parser should accept any syntactically valid token as a method and expose it as a string.
The handler layer may then decide what to do with it.
This keeps the parser generic and avoids hardcoding a small method list.
### Headers
Rules:
- header section ends at the first empty line
- each header line must have `name:value` form
- header names are case-insensitive for lookup
- original header values are preserved
- repeated headers are preserved as repeated values
- obsolete line folding is rejected
- embedded CR or LF in header values is rejected
### Host header
Rules:
- `Host` is required on every request
- there must be exactly one effective host value after normalization
- duplicate `Host` values are allowed only if they are identical after trimming
- conflicting `Host` values are rejected
### Request bodies
v1 accepted request body framing:
- no body
- body with a valid `Content-Length`
v1 rejected request body framing:
- any `Transfer-Encoding`
- chunked request bodies
- ambiguous or conflicting body framing
### Keep-alive
HTTP/1.1 persistent connections are supported.
Rules:
- keep-alive is the default
- the server closes the connection if the client sends `Connection: close`
- the server may close the connection after any response if it chooses
- the server closes the connection on parse errors or framing errors
- after a successful WebSocket upgrade, the HTTP request loop ends for that socket
### WebSocket upgrade
v1 supports classic HTTP/1.1 upgrade to WebSocket.
Required request properties:
- method is `GET`
- `Upgrade: websocket`
- `Connection` contains token `upgrade`
- `Sec-WebSocket-Key` is present
- `Sec-WebSocket-Version: 13`
v1 behavior:
- no subprotocol negotiation
- no extension negotiation
- no HTTP/2 WebSocket support
- no fallback upgrade modes beyond the standard HTTP/1.1 handshake
## Rejection and error rules
### `400 Bad Request`
Return `400` for:
- malformed request line
- invalid HTTP token in method or header name
- unsupported request-target form
- missing `Host`
- conflicting `Host` values
- invalid header syntax
- obsolete folded headers
- invalid `Content-Length`
- conflicting duplicate `Content-Length`
- invalid WebSocket upgrade request
### `413 Payload Too Large`
Return `413` when request body exceeds configured maximum size.
### `414 URI Too Long`
Return `414` when the request-target exceeds configured limits.
### `431 Request Header Fields Too Large`
Return `431` when:
- total header bytes exceed the configured limit
- header count exceeds the configured limit
- an individual header line exceeds the configured limit if such a per-line limit is introduced
### `501 Not Implemented`
Return `501` for:
- `Transfer-Encoding` in requests
- chunked request bodies
- `Expect: 100-continue`
- unsupported `Upgrade` values
- request features intentionally excluded from v1
### `505 HTTP Version Not Supported`
Return `505` for any HTTP version other than `HTTP/1.1`.
### `500 Internal Server Error`
Return `500` when the request was parsed successfully but the application handler throws or otherwise fails unexpectedly.
## Response model
v1 responses should be fully materialized before writing.
Rules:
- always send a status line
- always send response headers
- prefer sending `Content-Length` on all normal responses
- do not emit chunked responses in v1
- if response framing is ambiguous, close the connection instead of attempting reuse
Connection closing rules:
- include `Connection: close` when the server intends to close after the response
- close after the response if the request asked for `Connection: close`
- close after protocol errors
- after `101 Switching Protocols`, the HTTP server loop yields ownership of the socket to the WebSocket session
## Suggested defaults and limits
Default operational limits:
- maximum request line bytes: `8 KiB`
- maximum total header bytes: `32 KiB`
- maximum header count: `100`
- maximum request body bytes: `1 MiB`
- keep-alive idle timeout: `15_000 ms`
These should be configurable per server instance.
## Internal Kotlin API
The following shape is recommended as the initial internal API.
```kotlin
data class HttpServerConfig(
val host: String? = "127.0.0.1",
val port: Int = 0,
val backlog: Int = 128,
val reuseAddress: Boolean = true,
val maxRequestLineBytes: Int = 8 * 1024,
val maxHeaderBytes: Int = 32 * 1024,
val maxHeaderCount: Int = 100,
val maxBodyBytes: Int = 1 * 1024 * 1024,
val keepAliveTimeoutMillis: Long = 15_000,
)
data class HttpHeader(
val name: String,
val value: String,
)
class HttpHeaders(
private val entries: List<HttpHeader>,
) {
fun first(name: String): String?
fun all(name: String): List<String>
fun containsToken(name: String, token: String): Boolean
fun entries(): List<HttpHeader>
}
data class HttpRequestHead(
val method: String,
val target: String,
val path: String,
val query: String?,
val version: String,
val headers: HttpHeaders,
val contentLength: Int?,
val wantsClose: Boolean,
val wantsWebSocketUpgrade: Boolean,
)
data class HttpRequest(
val head: HttpRequestHead,
val body: ByteArray,
)
data class HttpResponse(
val status: Int,
val reason: String = defaultReason(status),
val headers: List<HttpHeader> = emptyList(),
val body: ByteArray = ByteArray(0),
val close: Boolean = false,
)
interface HttpWebSocketSession {
fun isOpen(): Boolean
suspend fun sendText(text: String)
suspend fun sendBytes(data: ByteArray)
suspend fun receive(): net.sergeych.lyngio.ws.LyngWsMessage?
suspend fun close(code: Int = 1000, reason: String = "")
}
sealed interface HttpHandlerResult {
data class Response(val response: HttpResponse) : HttpHandlerResult
data class WebSocket(val handler: suspend (HttpWebSocketSession) -> Unit) : HttpHandlerResult
}
fun interface HttpHandler {
suspend fun handle(request: HttpRequest): HttpHandlerResult
}
interface HttpServer {
fun isOpen(): Boolean
fun localAddress(): net.sergeych.lyngio.net.LyngSocketAddress
fun close()
}
```
## Implementation architecture
The implementation should be split into a small number of focused components.
### 1. `HttpServer.kt`
Contains:
- public internal interfaces and data classes
- config and response models
- default reason phrase mapping
### 2. `BufferedSocketReader.kt`
A small internal reader built on top of `LyngTcpSocket`.
Responsibilities:
- buffered reads
- line reads with explicit limits
- exact byte reads for request bodies and WebSocket frames
- avoiding fragile mixing of raw `read()` and `readLine()` semantics
This reader should be internal and should not require changes to `LyngTcpSocket` in v1.
### 3. `HttpParser.kt`
Responsibilities:
- request line parsing
- target parsing into `path` and optional query
- header parsing and normalization
- validation of `Host`, `Content-Length`, and connection semantics
- mapping parse failures into typed HTTP errors
### 4. `HttpWriter.kt`
Responsibilities:
- writing status line and headers
- adding `Content-Length` where needed
- setting `Connection: close` when the server intends to close
- writing the response body
- flushing output
### 5. `HttpServerLoop.kt`
Responsibilities:
- accept loop over `LyngTcpServer`
- per-connection request loop
- keep-alive timeout handling
- error-to-response mapping
- handing off upgraded sockets to WebSocket session implementation
### 6. `ServerWebSocket.kt`
Responsibilities:
- validating upgrade request
- computing `Sec-WebSocket-Accept`
- writing `101 Switching Protocols`
- reading and writing WebSocket frames
- close handling
This should reuse the client-side frame and handshake logic already present in spirit, but server-side behavior should stay separate and explicit.
## Connection processing model
Per accepted TCP connection:
1. read request line
2. read headers
3. validate request
4. read request body if `Content-Length` is present
5. call the application handler
6. if handler returns HTTP response, write it and decide whether to continue
7. if handler returns WebSocket upgrade, send `101`, create a WebSocket session, and transfer ownership of the socket
8. continue until close, error, timeout, or upgrade
The server should process one request at a time per connection.
Pipelining is out of scope.
## Detailed parser rules
### Method parsing
- method must be a valid HTTP token
- parser does not enforce a fixed method allowlist
### Target parsing
- target must begin with `/`
- split on the first `?`
- `path` is the portion before `?`
- `query` is the portion after `?`, or `null`
- no URL decoding is required in v1; raw target text may be exposed
### Header parsing
- split each header line on the first `:`
- trim outer spaces and tabs from the value
- reject control characters other than horizontal tab if any are allowed at all
- do case-insensitive matching by normalized header name
- preserve the original values as supplied
### Content-Length rules
- absent means no request body
- one valid decimal value is accepted
- multiple values are accepted only if all normalized values are identical
- negative values are rejected
- values above configured maximum body size are rejected with `413`
### Connection token parsing
- `Connection` is tokenized case-insensitively on commas
- surrounding spaces are ignored
- helper methods should support `containsToken("Connection", "close")`
- helper methods should support `containsToken("Connection", "upgrade")`
## WebSocket v1 rules
### Upgrade acceptance
Accept only if all of the following are true:
- request method is `GET`
- request version is `HTTP/1.1`
- request body is empty
- `Upgrade` contains `websocket`
- `Connection` contains `upgrade`
- `Sec-WebSocket-Key` is present and syntactically valid
- `Sec-WebSocket-Version` equals `13`
Otherwise return a regular HTTP error response.
### WebSocket features in v1
Supported:
- text messages
- binary messages
- ping/pong handling
- close handshake
Not supported in v1:
- permessage-deflate
- subprotocol negotiation
- fragmented-message streaming to the application
- very large frame optimizations beyond a reasonable implementation limit
## Testing plan
A server like this should be tested at three levels.
### 1. Parser unit tests
Cases:
- valid request line parsing
- invalid request line parsing
- target parsing with and without query
- header case-insensitive lookup
- duplicate `Host` handling
- duplicate `Content-Length` handling
- oversized request line rejection
- oversized headers rejection
- `Transfer-Encoding` rejection
### 2. Engine-level loopback tests
Using existing TCP backends:
- simple `GET` request and response
- `POST` with `Content-Length`
- keep-alive with two sequential requests on one socket
- `Connection: close`
- malformed request closes connection
- handler exception becomes `500`
- body too large becomes `413`
### 3. WebSocket upgrade tests
Cases:
- successful upgrade handshake
- text echo
- binary echo
- ping/pong behavior
- clean close handshake
- invalid upgrade headers rejected as HTTP errors
## Implementation phases
### Phase 1: internal HTTP server core
Implement:
- config
- buffered reader
- parser
- writer
- request loop
- fixed-body responses
- keep-alive
### Phase 2: server-side WebSocket upgrade
Implement:
- upgrade validation
- `101 Switching Protocols`
- WebSocket frame IO
- session object
- close and ping/pong handling
### Phase 3: host integration and optional Lyng exposure
Possible future work:
- host-facing convenience factory APIs
- Lyng module exposure if there is a clear scripting use case
- route helpers or lightweight dispatching
- JVM-specific richer backends if requirements grow
## Open questions
1. Should the first version expose only a Kotlin host API, or should it also be surfaced to Lyng scripts immediately?
2. Should response headers be represented as repeated `HttpHeader` entries only, or should a convenience builder API be added from the start?
3. Should the first version include a small path router helper, or should routing stay entirely in host code?
4. Should very small chunked response support be added later if keep-alive plus unknown response length becomes a real need, or should v1 require fully materialized responses only?
## Recommendation
Proceed with this strict HTTP/1.1 + WebSocket subset.
It is small enough to finish in common Kotlin, fits the current `lyngio` transport architecture, and avoids turning the project into a full protocol-stack implementation effort.
@@ -0,0 +1,25 @@
# Serialization Format Registry
Current status:
- no global serialization-format registry
- formats are exported explicitly from modules and used explicitly, e.g. `Lynon.encode(...)`, `Json.decode(...)`, or `MyFormat.encode(...)`
Why no registry now:
- explicit module exports already solve the current use case
- a registry adds global mutable state and naming semantics we do not currently need
- there is no current runtime feature that needs format discovery by string name
When a registry may become worth adding:
- config-driven format selection, e.g. `"format": "json"`
- host-side introspection such as "list installed serialization formats"
- collision detection across independently loaded modules
- admin or tooling APIs that need to resolve a format without importing its module explicitly
If added later, the registry should be:
- optional, not required for normal explicit usage
- based on stable fully qualified ids, not just short export names
- designed as a host/tooling facility first, not as part of ordinary script-level serialization
+73 -4
View File
@@ -78,18 +78,73 @@ kotlin {
}
// Generate an index of markdown documents under project /docs as a JSON array
val generateSampleDocPages by tasks.registering {
group = "documentation"
description = "Generates Markdown wrapper pages for Lyng sample files"
val examplesDir = rootProject.projectDir.resolve("examples")
val docsSamplesDir = rootProject.projectDir.resolve("docs/samples")
val outDir = layout.buildDirectory.dir("generated-sample-docs/docs")
inputs.dir(examplesDir)
inputs.dir(docsSamplesDir)
outputs.dir(outDir)
doLast {
val outRoot = outDir.get().asFile
outRoot.mkdirs()
fun generateFrom(sourceRoot: java.io.File, targetSubdir: String) {
if (!sourceRoot.exists()) return
sourceRoot.walkTopDown()
.filter { it.isFile && it.extension.equals("lyng", ignoreCase = true) }
.forEach { source ->
val rel = sourceRoot.toPath().relativize(source.toPath()).toString().replace('\\', '/')
val target = outRoot.resolve("$targetSubdir/$rel.md")
target.parentFile.mkdirs()
val title = source.name
val sourceText = source.readText()
val body = buildString {
append("# ").append(title).append("\n\n")
append("Generated from `")
append(
when (targetSubdir) {
"examples" -> "examples/$rel"
"samples" -> "docs/samples/$rel"
else -> "$targetSubdir/$rel"
}
)
append("` during site build.\n\n")
append("```lyng\n")
append(sourceText)
if (!sourceText.endsWith("\n")) append('\n')
append("```\n")
}
target.writeText(body)
}
}
generateFrom(examplesDir, "examples")
generateFrom(docsSamplesDir, "samples")
}
}
val generateDocsIndex by tasks.registering {
group = "documentation"
description = "Generates docs-index.json listing all Markdown files under /docs"
val docsDir = rootProject.projectDir.resolve("docs")
val generatedDocsDir = layout.buildDirectory.dir("generated-sample-docs/docs")
val outDir = layout.buildDirectory.dir("generated-resources")
inputs.dir(docsDir)
inputs.dir(generatedDocsDir)
outputs.dir(outDir)
dependsOn(generateSampleDocPages)
doLast {
val docs = mutableListOf<String>()
val docs = linkedSetOf<String>()
if (docsDir.exists()) {
docsDir.walkTopDown()
.filter { it.isFile && it.extension.equals("md", ignoreCase = true) }
@@ -100,6 +155,16 @@ val generateDocsIndex by tasks.registering {
docs += "docs/$rel"
}
}
val generatedRoot = generatedDocsDir.get().asFile
if (generatedRoot.exists()) {
generatedRoot.walkTopDown()
.filter { it.isFile && it.extension.equals("md", ignoreCase = true) }
.forEach { f ->
val rel = generatedRoot.toPath().relativize(f.toPath()).toString()
.replace('\\', '/')
docs += "docs/$rel"
}
}
val out = outDir.get().asFile
out.mkdirs()
val file = out.resolve("docs-index.json")
@@ -113,7 +178,7 @@ val generateDocsIndex by tasks.registering {
append(']')
}
file.writeText(json)
println("Generated ${'$'}{file.absolutePath} with ${'$'}{docs.size} entries")
println("Generated ${file.absolutePath} with ${docs.size} entries")
}
}
@@ -137,7 +202,7 @@ val generateSiteVersion by tasks.registering(Copy::class) {
// Ensure any ProcessResources task depends on docs index generation so the JSON is packaged
tasks.configureEach {
if (name.endsWith("ProcessResources")) {
dependsOn(generateDocsIndex, generateSiteVersion)
dependsOn(generateSampleDocPages, generateDocsIndex, generateSiteVersion)
}
}
@@ -148,16 +213,20 @@ listOf(
"jsProcessResources"
).forEach { taskName ->
tasks.matching { it.name == taskName }.configureEach {
dependsOn(generateDocsIndex)
dependsOn(generateSampleDocPages, generateDocsIndex)
}
}
// Copy Markdown docs into the "docs/" folder in the final resources, so paths in docs-index.json match files
tasks.named<Copy>("jsProcessResources").configure {
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
// Ensure we don't end up with two copies at root; we no longer add docs as a plain resources srcDir
from(rootProject.projectDir.resolve("docs")) {
into("docs")
}
from(layout.buildDirectory.dir("generated-sample-docs/docs")) {
into("docs")
}
}
// Optional: configure toolchain if needed by the project; uses root Kotlin version from version catalog
+6 -2
View File
@@ -986,10 +986,14 @@ fun rewriteAnchors(
}
continue
}
if (href.contains(".md")) {
if (href.contains(".md") || href.contains(".lyng")) {
val parts = href.split('#', limit = 2)
val mdPath = parts[0]
val rawPath = parts[0]
val frag = if (parts.size > 1) parts[1] else null
val mdPath = when {
rawPath.endsWith(".lyng") -> "$rawPath.md"
else -> rawPath
}
val target = if (mdPath.startsWith("docs/")) {
normalizePath(mdPath)
} else {
+27 -3
View File
@@ -21,8 +21,32 @@
<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>Lyng language</title>
<script>
(function () {
var path = window.location.pathname || "/";
var hash = window.location.hash || "";
if (hash) return;
var normalized = path.replace(/^\/+/, "");
var route = null;
if (normalized.startsWith("docs/") && normalized.endsWith(".md")) {
route = normalized;
} else if (normalized.startsWith("examples/") && normalized.endsWith(".lyng")) {
route = "docs/" + normalized + ".md";
} else if (normalized.startsWith("samples/") && normalized.endsWith(".lyng")) {
route = "docs/" + normalized + ".md";
}
if (route) {
var target = window.location.origin + "/#/" + route;
if (window.location.search) target += window.location.search;
window.location.replace(target);
}
})();
</script>
<!-- Site favicon (SVG, adapts to light/dark) -->
<link rel="icon" type="image/svg+xml" href="icons/lyng-favicon.svg"/>
<link rel="icon" type="image/svg+xml" href="/icons/lyng-favicon.svg"/>
<!-- GitHub Markdown CSS (light and dark). We toggle these from the app. -->
<link
id="md-light"
@@ -332,7 +356,7 @@
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.11.1/highlight.min.js"></script>
<!-- and it's easy to individually load additional languages -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.11.1/languages/go.min.js"></script>
<script src="lyng-version.js"></script>
<script src="/lyng-version.js"></script>
</head>
@@ -455,7 +479,7 @@
<div id="root" class="pt-4" tabindex="-1" aria-live="polite" aria-atomic="false"></div>
<!-- App bundle (produced by Kotlin/JS). The Gradle config forces this name. -->
<script src="site.js"></script>
<script src="/site.js"></script>
<!-- Bootstrap 5.3 JS bundle (includes Popper) -->
<script