diff --git a/docs/json_and_kotlin_serialization.md b/docs/json_and_kotlin_serialization.md index 99df608..8be5619 100644 --- a/docs/json_and_kotlin_serialization.md +++ b/docs/json_and_kotlin_serialization.md @@ -1,9 +1,25 @@ # Json support -Since 1.0.5 we start adding JSON support. Versions 1,0,6* support serialization of the basic types, including lists and -maps, and simple classes. Multiple inheritance may produce incorrect results, it is work in progress. +Lyng now has two distinct JSON-facing layers: -## Serialization in Lyng +- plain JSON projection: + - `Obj.toJson()` + - `Obj.toJsonString()` +- canonical JSON round-trip format: + - `Json.encode(value)` + - `Json.decode(text)` + +Use the first when you need ordinary JSON for interop. + +Use the second when you need Lyng value round-trip semantics through JSON text. + +This distinction is intentional: + +- plain JSON projection is optimized for compatibility with ordinary JSON tooling +- canonical `Json.encode()` is optimized for semantic fidelity to Lyng and Lynon +- these goals conflict for values such as sets, exceptions, singleton objects, buffers, and maps with non-string keys + +## Plain JSON projection in Lyng // in lyng assertEquals("{\"a\":1}", {a: 1}.toJsonString()) @@ -20,7 +36,8 @@ Simple classes serialization is supported: assertEquals( "{\"foo\":1,\"bar\":2}", Point(1,2).toJsonString() ) >>> void -Note that mutable members are serialized by default. You can exclude any member (including constructor parameters) from JSON serialization using the `@Transient` attribute: +Note that mutable members are serialized by default. You can exclude any member (including constructor parameters) from +JSON serialization using the `@Transient` attribute: import lyng.serialization @@ -31,7 +48,7 @@ Note that mutable members are serialized by default. You can exclude any member assertEquals( "{\"bar\":2,\"visible\":100}", Point2(1,2).toJsonString() ) >>> void -Note that if you override json serialization: +Note that if you override plain JSON serialization: import lyng.serialization @@ -46,8 +63,8 @@ Note that if you override json serialization: assertEquals( "{\"custom\":true}", Point2(1,2).toJsonString() ) >>> void -Custom serialization of user classes is possible by overriding `toJsonObject` method. It must return an object which is -serializable to Json. Most often it is a map, but any object is accepted, that makes it very flexible: +Custom serialization of user classes is possible by overriding `toJsonObject`. It must return an object which is +serializable to JSON. Most often it is a map, but any object is accepted: import lyng.serialization @@ -70,12 +87,49 @@ serializable to Json. Most often it is a map, but any object is accepted, that m Please note that `toJsonString` should be used to get serialized string representation of the object. Don't call `toJsonObject` directly, it is not intended to be used outside the serialization library. +## Canonical Json round-trip format + +`Json.encode()` and `Json.decode()` are now the JSON equivalents of `Lynon.encode()` and `Lynon.decode()`. + +They still use JSON text, but they add Lyng-specific type tags where plain JSON would otherwise lose information. + +Example: + +```lyng +import lyng.serialization +import lyng.time + +enum Color { Red, Green } +class Point(x,y) { var z = 42 } + +val p = Point(1,2) +p.z = 99 + +val value = List( + p, + Map([1, "one"], ["two", 2]), + Set(1,2,3), + "hello".encodeUtf8(), + Date(2026,4,15), + Color.Green +) + +assertEquals(value, Json.decode(Json.encode(value))) +``` + +The canonical `Json` format is intended for Lyng-to-Lyng transfer through JSON text. + +The plain `toJson()` projection is intended for ordinary JSON interop. + +Canonical `Json.encode()` should be read as the JSON analogue of `Lynon.encode()`: when Lynon already preserves a +Lyng distinction, canonical JSON tries to preserve it too, using tags only where ordinary JSON is insufficient. + ## Kotlin side interfaces The "Batteries included" principle is also applied to serialization. -- `Obj.toJson()` provides Kotlin `JsonElement` -- `Obj.toJsonString()` provides Json string representation +- `Obj.toJson()` provides Kotlin `JsonElement` for the plain JSON projection +- `Obj.toJsonString()` provides plain JSON string representation - `Obj.decodeSerializableWith()` and `Obj.decodeSerializable()` allows to decode Lyng classes as Kotlin objects using `kotlinx.serialization`: @@ -104,10 +158,9 @@ suspend inline fun Obj.decodeSerializable(scope: Scope = Scope()) = decodeSerializableWith(serializer(), scope) ``` -Note that lyng-2-kotlin deserialization with `kotlinx.serialization` uses JsonElement as information carrier without -formatting and parsing actual Json strings. This is why we use `Json.decodeFromJsonElement` instead of -`Json.decodeFromString`. Such an approach gives satisfactory performance without writing and supporting custom -`kotlinx.serialization` codecs. +Note that Lyng-to-Kotlin deserialization with `kotlinx.serialization` is based on the plain JSON projection, +not the canonical `Json.encode()` format. It uses `JsonElement` as the information carrier without formatting and +parsing actual JSON strings. This is why we use `Json.decodeFromJsonElement` instead of `Json.decodeFromString`. ### Pitfall: JSON objects and Map @@ -134,7 +187,8 @@ fun deserializeMapWithJsonTest() = runTest { But what if your map has objects of different types? The approach of using polymorphism is partially applicable, but what to do with `{ one: 1, two: "two" }`? -The answer is pretty simple: use `JsonObject` in your deserializable object. This class is capable of holding any JSON types and structures and is sort of a silver bullet for such cases: +The answer is simple: use `JsonObject` in your deserializable object. This class is capable of holding any JSON types +and structures: ~~~kotlin @Serializable @@ -154,26 +208,60 @@ fun deserializeAnyMapWithJsonTest() = runTest { ~~~ -# List of supported types +## Supported shapes + +### Plain JSON projection | Lyng type | JSON type | notes | |-----------|-----------|-------------| | `Int` | number | | -| `Real` | number | | +| `Real` | number | finite values only as plain numbers | | `String` | string | | | `Bool` | boolean | | | `null` | null | | | `Instant` | string | ISO8601 (1) | | `List` | array | (2) | -| `Map` | object | (2) | +| `Map` | object | string keys only | +| simple class instance | object | constructor fields + mutable vars | +| enum | string | entry name | +### Canonical `Json.encode` + +This format can also round-trip: + +- maps with non-string keys +- sets +- immutable collections +- buffers and bit buffers +- class instances +- singleton objects +- enums +- exceptions +- `Date`, `Instant`, `DateTime` +- non-finite reals +- `void` + +It does so by adding Lyng-specific type tags only when necessary. + +## Kotlin-side extension point for more formats + +Additional formats can be exported from Kotlin modules by subclassing `ObjSerializationFormatClass` and registering the +format in module scope with `bindSerializationFormat(...)`. + +```kotlin +module.bindSerializationFormat( + object : ObjSerializationFormatClass("MyFormat") { + override suspend fun encodeValue(scope: Scope, value: Obj): Obj = ... + override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj = ... + } +) +``` + +This makes `MyFormat.encode(...)` and `MyFormat.decode(...)` available from Lyng after importing the module. (1) -: ISO8601 flavor 1970-05-06T06:00:00.000Z in used; number of fractional digits depends on the truncation -on [Instant](time.md), see `Instant.truncateTo...` functions. +: ISO8601 flavor `1970-05-06T06:00:00.000Z` is used; number of fractional digits depends on truncation on +`Instant`, see `Instant.truncateTo...` functions. (2) -: List may contain any objects serializable to Json. - -(3) -: Map keys must be strings, map values may be any objects serializable to Json. +: Lists may contain any values serializable by the selected JSON layer. diff --git a/docs/serialization.md b/docs/serialization.md index 49a2193..b42258c 100644 --- a/docs/serialization.md +++ b/docs/serialization.md @@ -1,6 +1,33 @@ # Lyng serialization -Lyng has builting binary bit-effective serialization format, called Lynon for LYng Object Notation. It is typed, binary, implements caching, automatic compression, variable-length ints, one-bit Booleans an many nice features. +Lyng has a built-in serialization module, `lyng.serialization`. + +There are now two built-in formats with different goals: + +- `Lynon`: the canonical binary format for Lyng values. +- `Json`: the canonical JSON-based round-trip format for Lyng values. + +In addition, `Obj.toJson()` / `toJsonString()` remain available as a plain JSON projection for interoperability with +regular JSON tools and Kotlin `kotlinx.serialization`. + +## Canonical formats + +`Lynon` and `Json` are both exposed as format objects with the same surface: + +- `Format.encode(value)` +- `Format.decode(encodedValue)` + +For the built-in formats: + +- `Lynon.encode(x)` returns `BitBuffer` +- `Lynon.decode(bitBuffer)` returns the original Lyng value +- `Json.encode(x)` returns `String` +- `Json.decode(jsonString)` returns the original Lyng value + +## Lynon + +Lynon is LYng Object Notation. It is typed, binary, bit-effective, implements caching, automatic compression, +variable-length integers, one-bit booleans, and preserves Lyng runtime structure well. It is as simple as: @@ -20,7 +47,8 @@ It is as simple as: assert( text.length > (encodedBits.toBuffer() as Buffer).size ) >>> void -Any class you create is serializable by default; lynon serializes first constructor fields, then any `var` member fields. +Any class you create is serializable by default; Lynon serializes first constructor fields, then any `var` member +fields. ## Transient Fields @@ -40,7 +68,7 @@ class MyData(@Transient val tempSecret, val publicData) { Transient fields: - Are **omitted** from Lynon binary streams. -- Are **omitted** from JSON output (via `toJson`). +- Are **omitted** from JSON output (`toJson`) and canonical `Json.encode(...)`. - Are **ignored** during structural equality checks (`==`). - If a transient constructor parameter has a **default value**, it will be restored to that default value during deserialization. Otherwise, it will be `null`. - Class body fields marked as `@Transient` will keep their initial values (or values assigned in `init`) after deserialization. @@ -49,8 +77,105 @@ Transient fields: - **Singleton Objects**: `object` declarations are serializable by name. Their state (mutable fields) is also serialized and restored, respecting `@Transient`. - **Classes**: Class objects themselves can be serialized. They are serialized by their full qualified name. When converted to JSON, a class object includes its public static fields (excluding those marked `@Transient`). +- **Exceptions**: canonical formats preserve exception class, message, extra data, and captured stack trace. -## Custom Serialization +## Plain JSON projection vs canonical Json format + +There are two JSON-related APIs and they serve different purposes: + +- `Obj.toJson()` / `toJsonString()` + - produce ordinary JSON values + - best for interop with external JSON systems + - best for `Obj.decodeSerializable()` / `decodeSerializableWith()` + - may be lossy for Lyng-specific structures + +- `Json.encode()` / `Json.decode()` + - produce JSON text too + - but use Lyng-specific type tags where needed + - intended for round-tripping Lyng values + - intended to match Lynon semantics where JSON can carry them + - can preserve values that plain JSON cannot represent directly, such as: + - maps with non-string keys + - sets + - buffers and bit buffers + - class instances + - singleton objects + - enums + - exceptions + - date/time objects + - non-finite reals + - `void` + +Why this split exists: + +- plain `toJson()` must remain ordinary JSON so it stays convenient for external JSON systems and Kotlin + `kotlinx.serialization` +- canonical `Json.encode()` is for Lyng-to-Lyng transport through JSON text, so it must preserve Lyng runtime + distinctions whenever possible +- one API cannot optimize for both goals at once: either you get too many Lyng tags for ordinary JSON interop, or you + get lossy round-trips + +Example: + + import lyng.serialization + import lyng.time + + enum Color { Red, Green } + class Point(x,y) { var z = 42 } + + val p = Point(1,2) + p.z = 99 + val x = List( + p, + Map([1, "one"], ["two", 2]), + Set(1,2,3), + "hello".encodeUtf8(), + Date(2026,4,15), + Color.Green + ) + + assertEquals(x, Json.decode(Json.encode(x))) + >>> void + +## Adding more formats from Kotlin modules + +External modules can add new formats on the Kotlin side. + +The common base class is: + +```kotlin +abstract class ObjSerializationFormatClass(className: String) : ObjClass(className) { + abstract suspend fun encodeValue(scope: Scope, value: Obj): Obj + abstract suspend fun decodeValue(scope: Scope, encoded: Obj): Obj +} +``` + +To export a new format from a module: + +```kotlin +im.addPackage("test.formats") { module -> + module.bindSerializationFormat( + object : ObjSerializationFormatClass("Reverse") { + override suspend fun encodeValue(scope: Scope, value: Obj): Obj = + ObjString(value.toString(scope).value.reversed()) + + override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj = + ObjString((encoded as ObjString).value.reversed()) + } + ) +} +``` + +Then from Lyng, after importing the Kotlin module above, usage looks like: + +```lyng +import test.formats + +assertEquals("cba", Reverse.encode("abc")) +assertEquals("abc", Reverse.decode("cba")) +``` + +## Notes Important is to understand that normally `Lynon.decode` wants [BitBuffer], as `Lynon.encode` produces. If you have the regular [Buffer], be sure to convert it: @@ -59,5 +184,3 @@ Important is to understand that normally `Lynon.decode` wants [BitBuffer], as `L this possibly creates extra zero bits at the end, as bit content could be shorter than byte-grained but for the Lynon format it does not make sense. Note that when you serialize [BitBuffer], exact number of bits is written. To convert bit buffer to bytes: Lynon.encode("hello").toBuffer() - -(topic is incomplete and under construction) diff --git a/docs/whats_new.md b/docs/whats_new.md index 00ac065..640693c 100644 --- a/docs/whats_new.md +++ b/docs/whats_new.md @@ -327,7 +327,7 @@ Singleton objects are declared using the `object` keyword. They provide a conven ```lyng object Config { - val version = "1.5.5" + val version = "1.5.6-SNAPSHOT" fun show() = println("Config version: " + version) } diff --git a/lynglib/build.gradle.kts b/lynglib/build.gradle.kts index 20a9c09..8416ca1 100644 --- a/lynglib/build.gradle.kts +++ b/lynglib/build.gradle.kts @@ -21,7 +21,7 @@ import org.jetbrains.kotlin.gradle.ExperimentalWasmDsl import org.jetbrains.kotlin.gradle.dsl.JvmTarget group = "net.sergeych" -version = "1.5.5" +version = "1.5.6-SNAPSHOT" // Removed legacy buildscript classpath declarations; plugins are applied via the plugins DSL below diff --git a/lynglib/src/commonMain/kotlin/net/sergeych/lyng/ClassDeclStatement.kt b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/ClassDeclStatement.kt index 49e7a8e..d0f48c1 100644 --- a/lynglib/src/commonMain/kotlin/net/sergeych/lyng/ClassDeclStatement.kt +++ b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/ClassDeclStatement.kt @@ -60,6 +60,7 @@ internal suspend fun executeClassDecl( val newClass = ObjInstanceClass(spec.className, *parentClasses.toTypedArray()) newClass.isAnonymous = spec.isAnonymous + newClass.isSingletonObject = true newClass.constructorMeta = ArgsDeclaration(emptyList(), Token.Type.RPAREN) for (i in parentClasses.indices) { val argsList = spec.baseSpecs[i].args diff --git a/lynglib/src/commonMain/kotlin/net/sergeych/lyng/Script.kt b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/Script.kt index 5ef1c0e..80f2b27 100644 --- a/lynglib/src/commonMain/kotlin/net/sergeych/lyng/Script.kt +++ b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/Script.kt @@ -27,6 +27,8 @@ import net.sergeych.lyng.bytecode.CmdVm import net.sergeych.lyng.bytecode.BytecodeLambdaCallable import net.sergeych.lyng.miniast.* import net.sergeych.lyng.obj.* +import net.sergeych.lyng.serialization.ObjJsonClass +import net.sergeych.lyng.serialization.bindSerializationFormat import net.sergeych.lyng.pacman.ImportManager import net.sergeych.lyng.stdlib_included.complexLyng import net.sergeych.lyng.stdlib_included.decimalLyng @@ -949,11 +951,13 @@ class Script( ) } addPackage("lyng.serialization") { - it.addConstDoc( - name = "Lynon", - value = ObjLynonClass, - doc = "Lynon serialization utilities: encode/decode data structures to a portable binary/text form.", - type = type("lyng.Class") + it.bindSerializationFormat( + ObjLynonClass, + doc = "Lynon serialization utilities: encode/decode data structures to a portable binary format." + ) + it.bindSerializationFormat( + ObjJsonClass, + doc = "Universal JSON serialization utilities with bidirectional Lyng object support." ) } addPackage("lyng.time") { diff --git a/lynglib/src/commonMain/kotlin/net/sergeych/lyng/highlight/SimpleLyngHighlighter.kt b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/highlight/SimpleLyngHighlighter.kt index d09f0b7..9feb887 100644 --- a/lynglib/src/commonMain/kotlin/net/sergeych/lyng/highlight/SimpleLyngHighlighter.kt +++ b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/highlight/SimpleLyngHighlighter.kt @@ -24,6 +24,9 @@ import net.sergeych.lyng.parseLyng /** Extension that converts a [Pos] (line/column) into absolute character offset in the [Source] text. */ fun Source.offsetOf(pos: Pos): Int { + if (lines.isEmpty()) return 0 + if (pos.line < 0) return 0 + if (pos.line >= lines.size) return text.length var off = 0 // Sum full preceding lines + one '\n' per line (lines[] were created by String.lines()) var i = 0 @@ -31,8 +34,8 @@ fun Source.offsetOf(pos: Pos): Int { off += lines[i].length + 1 // assume \n as separator i++ } - off += pos.column - return off + off += pos.column.coerceIn(0, lines[pos.line].length) + return off.coerceAtMost(text.length) } private val reservedIdKeywords = setOf("constructor", "property") diff --git a/lynglib/src/commonMain/kotlin/net/sergeych/lyng/obj/ObjClass.kt b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/obj/ObjClass.kt index ffba564..97b043e 100644 --- a/lynglib/src/commonMain/kotlin/net/sergeych/lyng/obj/ObjClass.kt +++ b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/obj/ObjClass.kt @@ -119,6 +119,7 @@ open class ObjClass( var isAbstract: Boolean = false var isClosed: Boolean = false + var isSingletonObject: Boolean = false var logicalPackageNameOverride: String? = null // Stable identity and simple structural version for PICs diff --git a/lynglib/src/commonMain/kotlin/net/sergeych/lyng/serialization/JsonFormat.kt b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/serialization/JsonFormat.kt new file mode 100644 index 0000000..c052df1 --- /dev/null +++ b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/serialization/JsonFormat.kt @@ -0,0 +1,382 @@ +/* + * Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package net.sergeych.lyng.serialization + +import kotlinx.datetime.LocalDate +import kotlinx.serialization.json.Json +import kotlinx.serialization.json.JsonArray +import kotlinx.serialization.json.JsonElement +import kotlinx.serialization.json.JsonNull +import kotlinx.serialization.json.JsonObject +import kotlinx.serialization.json.JsonPrimitive +import kotlinx.serialization.json.booleanOrNull +import kotlinx.serialization.json.doubleOrNull +import kotlinx.serialization.json.jsonPrimitive +import kotlinx.serialization.json.longOrNull +import net.sergeych.lyng.Arguments +import net.sergeych.lyng.ModuleScope +import net.sergeych.lyng.Pos +import net.sergeych.lyng.Scope +import net.sergeych.lyng.obj.Obj +import net.sergeych.lyng.obj.ObjBitBuffer +import net.sergeych.lyng.obj.ObjBool +import net.sergeych.lyng.obj.ObjBuffer +import net.sergeych.lyng.obj.ObjClass +import net.sergeych.lyng.obj.ObjDate +import net.sergeych.lyng.obj.ObjDateTime +import net.sergeych.lyng.obj.ObjEnumClass +import net.sergeych.lyng.obj.ObjEnumEntry +import net.sergeych.lyng.obj.ObjException +import net.sergeych.lyng.obj.ObjImmutableList +import net.sergeych.lyng.obj.ObjImmutableMap +import net.sergeych.lyng.obj.ObjImmutableSet +import net.sergeych.lyng.obj.ObjInstance +import net.sergeych.lyng.obj.ObjInt +import net.sergeych.lyng.obj.ObjInstant +import net.sergeych.lyng.obj.ObjList +import net.sergeych.lyng.obj.ObjMap +import net.sergeych.lyng.obj.ObjNull +import net.sergeych.lyng.obj.ObjReal +import net.sergeych.lyng.obj.ObjSet +import net.sergeych.lyng.obj.ObjString +import net.sergeych.lyng.obj.ObjVoid +import net.sergeych.lynon.BitArray +import net.sergeych.mp_tools.decodeBase64Url +import net.sergeych.mp_tools.encodeToBase64Url +import kotlin.time.Instant + +private const val TYPE_KEY = "@lyng" +private const val VALUE_KEY = "value" +private const val ITEMS_KEY = "items" +private const val ENTRIES_KEY = "entries" +private const val CLASS_KEY = "class" +private const val NAME_KEY = "name" +private const val ARGS_KEY = "args" +private const val VARS_KEY = "vars" +private const val BASE64_KEY = "base64" +private const val LAST_BYTE_BITS_KEY = "lastByteBits" +private const val MESSAGE_KEY = "message" +private const val EXTRA_DATA_KEY = "extraData" +private const val STACK_TRACE_KEY = "stackTrace" + +object ObjJsonClass : ObjSerializationFormatClass("Json") { + + override suspend fun encodeValue(scope: Scope, value: Obj): Obj = + ObjString(encodeToJsonElement(scope, value).toString()) + + override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj { + val text = when (encoded) { + is ObjString -> encoded.value + else -> encoded.toString(scope).value + } + return decodeFromJsonElement(scope, Json.parseToJsonElement(text)) + } + + suspend fun encodeToJsonElement(scope: Scope, value: Obj): JsonElement = + UniversalJsonCodec.encode(scope, value) + + suspend fun decodeFromJsonElement(scope: Scope, element: JsonElement): Obj = + UniversalJsonCodec.decode(scope, element) +} + +suspend fun Obj.toUniversalJsonElement(scope: Scope = Scope()): JsonElement = + ObjJsonClass.encodeToJsonElement(scope, this) + +suspend fun decodeUniversalJsonElement(element: JsonElement, scope: Scope = Scope()): Obj = + ObjJsonClass.decodeFromJsonElement(scope, element) + +private object UniversalJsonCodec { + suspend fun encode(scope: Scope, value: Obj): JsonElement = when (value) { + ObjVoid -> tagged("void") + ObjNull -> JsonNull + is ObjBool -> JsonPrimitive(value.value) + is ObjInt -> JsonPrimitive(value.value) + is ObjReal -> if (value.value.isFinite()) { + JsonPrimitive(value.value) + } else { + tagged("real", VALUE_KEY to JsonPrimitive(value.value.toString())) + } + is ObjString -> JsonPrimitive(value.value) + is ObjDate -> tagged("date", VALUE_KEY to JsonPrimitive(value.date.toString())) + is ObjInstant -> tagged("instant", VALUE_KEY to JsonPrimitive(value.instant.toString())) + is ObjDateTime -> tagged("dateTime", VALUE_KEY to JsonPrimitive(value.toRFC3339())) + is ObjBuffer -> tagged("buffer", BASE64_KEY to JsonPrimitive(value.base64)) + is ObjBitBuffer -> tagged( + "bitBuffer", + BASE64_KEY to JsonPrimitive(value.bitArray.asUByteArray().asByteArray().encodeToBase64Url()), + LAST_BYTE_BITS_KEY to JsonPrimitive(value.bitArray.lastByteBits) + ) + is ObjImmutableList -> tagged("immutableList", ITEMS_KEY to JsonArray(value.toMutableList().map { encode(scope, it) })) + is ObjList -> JsonArray(value.list.map { encode(scope, it) }) + is ObjImmutableSet -> tagged("immutableSet", ITEMS_KEY to JsonArray(value.toMutableSet().map { encode(scope, it) })) + is ObjSet -> tagged("set", ITEMS_KEY to JsonArray(value.set.map { encode(scope, it) })) + is ObjImmutableMap -> tagged("immutableMap", ENTRIES_KEY to encodeEntries(scope, value.map.entries.map { it.toPair() })) + is ObjMap -> encodeMap(scope, value) + is ObjEnumEntry -> tagged( + "enum", + CLASS_KEY to JsonPrimitive(value.objClass.className), + NAME_KEY to JsonPrimitive(value.name.value) + ) + is ObjException -> tagged( + "exception", + CLASS_KEY to JsonPrimitive(value.exceptionClass.className), + MESSAGE_KEY to encode(scope, value.message), + EXTRA_DATA_KEY to encode(scope, value.extraData), + STACK_TRACE_KEY to encode(scope, value.getStackTrace()) + ) + is ObjClass -> tagged("class", NAME_KEY to JsonPrimitive(value.className)) + is ObjInstance -> if (value.objClass.isSingletonObject) { + encodeSingletonObject(scope, value) + } else { + encodeInstance(scope, value) + } + else -> scope.raiseNotImplemented("Json.encode can't serialize ${value.objClass.className}") + } + + suspend fun decode(scope: Scope, element: JsonElement): Obj = when (element) { + JsonNull -> ObjNull + is JsonPrimitive -> decodePrimitive(element) + is JsonArray -> ObjList(element.map { decode(scope, it) }.toMutableList()) + is JsonObject -> decodeObject(scope, element) + } + + private suspend fun encodeMap(scope: Scope, value: ObjMap): JsonElement { + if (value.map.keys.all { it is ObjString } && TYPE_KEY !in value.map.keys.map { (it as ObjString).value }) { + return JsonObject( + value.map.entries.associate { (k, v) -> + (k as ObjString).value to encode(scope, v) + } + ) + } + return tagged("map", ENTRIES_KEY to encodeEntries(scope, value.map.entries.map { it.toPair() })) + } + + private suspend fun encodeEntries(scope: Scope, entries: List>): JsonArray = + JsonArray(entries.map { (k, v) -> JsonArray(listOf(encode(scope, k), encode(scope, v))) }) + + private suspend fun encodeInstance(scope: Scope, value: ObjInstance): JsonElement { + val meta = value.objClass.constructorMeta + ?: scope.raiseError("can't serialize non-serializable object (no constructor meta)") + val args = linkedMapOf() + for (param in meta.params) { + if (!param.isTransient) { + args[param.name] = encode(scope, value.readField(scope, param.name).value) + } + } + val vars = linkedMapOf() + for ((key, record) in value.serializingVars) { + vars[key.substringAfterLast("::")] = encode(scope, record.value) + } + return tagged( + "instance", + CLASS_KEY to JsonPrimitive(value.objClass.className), + ARGS_KEY to JsonObject(args), + VARS_KEY to JsonObject(vars) + ) + } + + private suspend fun encodeSingletonObject(scope: Scope, value: ObjInstance): JsonElement { + val vars = linkedMapOf() + for ((key, record) in value.serializingVars) { + vars[key.substringAfterLast("::")] = encode(scope, record.value) + } + return tagged( + "object", + NAME_KEY to JsonPrimitive(value.objClass.className), + VARS_KEY to JsonObject(vars) + ) + } + + private suspend fun decodeObject(scope: Scope, element: JsonObject): Obj { + val tag = element[TYPE_KEY]?.jsonPrimitive?.content + if (tag == null) { + val map = linkedMapOf() + for ((k, v) in element) { + map[ObjString(k)] = decode(scope, v) + } + return ObjMap(map.toMutableMap()) + } + return when (tag) { + "void" -> ObjVoid + "real" -> decodeTaggedReal(element) + "date" -> ObjDate(LocalDate.parse(requiredString(element, VALUE_KEY))) + "instant" -> ObjInstant(Instant.parse(requiredString(element, VALUE_KEY))) + "dateTime" -> ObjDateTime.type.invokeInstanceMethod(scope, "parseRFC3339", ObjString(requiredString(element, VALUE_KEY))) + "buffer" -> ObjBuffer(requiredString(element, BASE64_KEY).decodeBase64Url().asUByteArray()) + "bitBuffer" -> ObjBitBuffer( + BitArray( + requiredString(element, BASE64_KEY).decodeBase64Url().asUByteArray(), + requiredInt(element, LAST_BYTE_BITS_KEY) + ) + ) + "immutableList" -> ObjImmutableList(requiredArray(element, ITEMS_KEY).map { decode(scope, it) }) + "set" -> ObjSet(requiredArray(element, ITEMS_KEY).map { decode(scope, it) }.toMutableSet()) + "immutableSet" -> ObjImmutableSet(requiredArray(element, ITEMS_KEY).map { decode(scope, it) }) + "map" -> decodeMap(scope, requiredArray(element, ENTRIES_KEY), mutable = true) + "immutableMap" -> decodeMap(scope, requiredArray(element, ENTRIES_KEY), mutable = false) + "class" -> resolveClass(scope, requiredString(element, NAME_KEY)) + "enum" -> decodeEnum(scope, element) + "instance" -> decodeInstance(scope, element) + "object" -> decodeSingletonObject(scope, element) + "exception" -> decodeException(scope, element) + else -> scope.raiseIllegalArgument("unknown Json type tag '$tag'") + } + } + + private fun decodePrimitive(element: JsonPrimitive): Obj { + element.booleanOrNull?.let { return ObjBool(it) } + if (element.isString) return ObjString(element.content) + val raw = element.content + return if (!raw.contains('.') && !raw.contains('e', ignoreCase = true)) { + element.longOrNull?.let { ObjInt.of(it) } ?: ObjReal(raw.toDouble()) + } else { + ObjReal(element.doubleOrNull ?: raw.toDouble()) + } + } + + private fun decodeTaggedReal(element: JsonObject): ObjReal { + val raw = requiredString(element, VALUE_KEY) + val value = when (raw) { + "NaN" -> Double.NaN + "Infinity" -> Double.POSITIVE_INFINITY + "-Infinity" -> Double.NEGATIVE_INFINITY + else -> raw.toDouble() + } + return ObjReal(value) + } + + private suspend fun decodeMap(scope: Scope, entries: JsonArray, mutable: Boolean): Obj { + val pairs = entries.map { item -> + val pair = item as? JsonArray ?: scope.raiseIllegalArgument("map entry must be a JSON array") + if (pair.size != 2) scope.raiseIllegalArgument("map entry must contain exactly 2 items") + decode(scope, pair[0]) to decode(scope, pair[1]) + } + return if (mutable) ObjMap(pairs.toMap().toMutableMap()) else ObjImmutableMap(pairs.toMap()) + } + + private suspend fun decodeEnum(scope: Scope, element: JsonObject): Obj { + val klass = resolveClass(scope, requiredString(element, CLASS_KEY)) + if (klass !is ObjEnumClass) scope.raiseClassCastError("${klass.className} is not an enum") + return klass.invokeInstanceMethod(scope, "valueOf", ObjString(requiredString(element, NAME_KEY))) + } + + private suspend fun decodeInstance(scope: Scope, element: JsonObject): Obj { + val klass = resolveClass(scope, requiredString(element, CLASS_KEY)) + val meta = klass.constructorMeta + ?: scope.raiseError("can't deserialize ${klass.className} from Json: no constructor meta") + val argsObject = requiredObject(element, ARGS_KEY) + val namedArgs = linkedMapOf() + for (param in meta.params) { + if (param.isTransient) continue + val encoded = argsObject[param.name] + if (encoded == null) { + if (param.defaultValue == null && !param.type.isNullable) { + scope.raiseIllegalArgument("missing constructor field '${param.name}' for ${klass.className}") + } + } else { + namedArgs[param.name] = decode(scope, encoded) + } + } + val callScope = scope.createChildScope(args = Arguments(list = emptyList(), named = namedArgs)) + val instance = klass.callOn(callScope) + if (instance is ObjInstance) { + val varsObject = requiredObject(element, VARS_KEY) + for ((name, encoded) in varsObject) { + val target = resolveSerializableVar(instance, name) + ?: scope.raiseIllegalArgument("unknown serializable field '${klass.className}.$name'") + target.value = decode(scope, encoded) + } + } + return instance + } + + private suspend fun decodeSingletonObject(scope: Scope, element: JsonObject): Obj { + val instance = resolveObject(scope, requiredString(element, NAME_KEY)) + val varsObject = requiredObject(element, VARS_KEY) + for ((name, encoded) in varsObject) { + val target = resolveSerializableVar(instance, name) + ?: scope.raiseIllegalArgument("unknown serializable field '${instance.objClass.className}.$name'") + target.value = decode(scope, encoded) + } + return instance + } + + private suspend fun decodeException(scope: Scope, element: JsonObject): Obj { + val klass = resolveClass(scope, requiredString(element, CLASS_KEY)) + if (klass !is ObjException.Companion.ExceptionClass) { + scope.raiseClassCastError("${klass.className} is not an exception class") + } + val message = decode(scope, requireElement(element, MESSAGE_KEY)) as? ObjString + ?: scope.raiseClassCastError("exception message must be a string") + val extraData = decode(scope, requireElement(element, EXTRA_DATA_KEY)) + val stackTrace = decode(scope, requireElement(element, STACK_TRACE_KEY)) as? ObjList + ?: scope.raiseClassCastError("exception stackTrace must be a list") + return ObjException(klass, scope, message, extraData, stackTrace) + } + + private fun resolveSerializableVar(instance: ObjInstance, name: String) = + instance.serializingVars[name] + ?: instance.serializingVars.entries.singleOrNull { it.key.substringAfterLast("::") == name }?.value + + private suspend fun resolveClass(scope: Scope, className: String): ObjClass { + scope.get(className)?.value?.let { + if (it is ObjClass) return it + if (it is ObjInstance && it.objClass.className == className) return it.objClass + scope.raiseClassCastError("Expected class $className, got ${it.objClass.className}") + } + val resolved = scope.resolveQualifiedIdentifier(className) + if (resolved is ObjClass) return resolved + if (resolved is ObjInstance && resolved.objClass.className == className) return resolved.objClass + scope.raiseClassCastError("Expected class $className, got ${resolved.objClass.className}") + return resolved as ObjClass + } + + private suspend fun resolveObject(scope: Scope, objectName: String): ObjInstance { + scope.get(objectName)?.value?.let { + if (it is ObjInstance) return it + scope.raiseClassCastError("Expected object $objectName, got ${it.objClass.className}") + } + if (objectName.contains('.')) { + val resolved = scope.resolveQualifiedIdentifier(objectName) + val inst = resolved as? ObjInstance + if (inst != null) return inst + scope.raiseClassCastError("Expected object $objectName, got ${resolved.objClass.className}") + } + scope.raiseSymbolNotFound(objectName) + } + + private fun tagged(type: String, vararg fields: Pair): JsonObject = + JsonObject(linkedMapOf(TYPE_KEY to JsonPrimitive(type), *fields)) + + private fun requiredString(element: JsonObject, key: String): String = + requireElement(element, key).jsonPrimitive.content + + private fun requiredInt(element: JsonObject, key: String): Int = + requireElement(element, key).jsonPrimitive.content.toInt() + + private fun requiredArray(element: JsonObject, key: String): JsonArray = + requireElement(element, key) as? JsonArray + ?: throw IllegalArgumentException("field '$key' must be a JSON array") + + private fun requiredObject(element: JsonObject, key: String): JsonObject = + requireElement(element, key) as? JsonObject + ?: throw IllegalArgumentException("field '$key' must be a JSON object") + + private fun requireElement(element: JsonObject, key: String): JsonElement = + element[key] ?: throw IllegalArgumentException("missing field '$key'") +} diff --git a/lynglib/src/commonMain/kotlin/net/sergeych/lyng/serialization/SerializationFormats.kt b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/serialization/SerializationFormats.kt new file mode 100644 index 0000000..c52c802 --- /dev/null +++ b/lynglib/src/commonMain/kotlin/net/sergeych/lyng/serialization/SerializationFormats.kt @@ -0,0 +1,59 @@ +/* + * Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package net.sergeych.lyng.serialization + +import net.sergeych.lyng.ModuleScope +import net.sergeych.lyng.Scope +import net.sergeych.lyng.miniast.addConstDoc +import net.sergeych.lyng.miniast.type +import net.sergeych.lyng.obj.Obj +import net.sergeych.lyng.obj.ObjClass +import net.sergeych.lyng.requireOnlyArg +import net.sergeych.lyng.requireScope + +abstract class ObjSerializationFormatClass( + className: String +) : ObjClass(className) { + + abstract suspend fun encodeValue(scope: Scope, value: Obj): Obj + + abstract suspend fun decodeValue(scope: Scope, encoded: Obj): Obj + + init { + addClassFn("encode") { + encodeValue(requireScope(), requireOnlyArg()) + } + addClassFn("decode") { + decodeValue(requireScope(), requireOnlyArg()) + } + } +} + +suspend fun ModuleScope.bindSerializationFormat( + format: ObjSerializationFormatClass, + exportName: String = format.className, + doc: String = "${format.className} serialization format." +): ObjSerializationFormatClass { + addConstDoc( + name = exportName, + value = format, + doc = doc, + type = type("lyng.Class") + ) + return format +} diff --git a/lynglib/src/commonMain/kotlin/net/sergeych/lynon/packer.kt b/lynglib/src/commonMain/kotlin/net/sergeych/lynon/packer.kt index 3364b70..9c986e8 100644 --- a/lynglib/src/commonMain/kotlin/net/sergeych/lynon/packer.kt +++ b/lynglib/src/commonMain/kotlin/net/sergeych/lynon/packer.kt @@ -18,14 +18,13 @@ package net.sergeych.lynon import net.sergeych.lyng.Scope -import net.sergeych.lyng.requireOnlyArg -import net.sergeych.lyng.requireScope +import net.sergeych.lyng.serialization.ObjSerializationFormatClass import net.sergeych.lyng.obj.* // Most often used types: -object ObjLynonClass : ObjClass("Lynon") { +object ObjLynonClass : ObjSerializationFormatClass("Lynon") { suspend fun encodeAny(scope: Scope, obj: Obj): ObjBitBuffer { val bout = MemoryBitOutput() @@ -41,15 +40,9 @@ object ObjLynonClass : ObjClass("Lynon") { return deserializer.decodeAny(scope) } - init { - addClassConst("test", ObjString("test_const")) - addClassFn("encode") { - encodeAny(requireScope(), requireOnlyArg()) - } - addClassFn("decode") { - decodeAny(requireScope(), requireOnlyArg()) - } - } + override suspend fun encodeValue(scope: Scope, value: Obj): Obj = encodeAny(scope, value) + + override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj = decodeAny(scope, encoded) } /** diff --git a/lynglib/src/commonTest/kotlin/ScriptTest.kt b/lynglib/src/commonTest/kotlin/ScriptTest.kt index 45f95ec..2e1f437 100644 --- a/lynglib/src/commonTest/kotlin/ScriptTest.kt +++ b/lynglib/src/commonTest/kotlin/ScriptTest.kt @@ -32,6 +32,8 @@ import kotlinx.serialization.json.encodeToJsonElement import net.sergeych.lyng.* import net.sergeych.lyng.obj.* import net.sergeych.lyng.pacman.InlineSourcesImportProvider +import net.sergeych.lyng.serialization.ObjSerializationFormatClass +import net.sergeych.lyng.serialization.bindSerializationFormat import net.sergeych.lyng.thisAs import net.sergeych.mp_tools.globalDefer import net.sergeych.tools.bm @@ -4663,6 +4665,89 @@ class ScriptTest { ) } + @Test + fun testUniversalJsonRoundTrip() = runTest { + eval( + """ + import lyng.serialization + import lyng.time + + enum Color { Red, Green } + class Point(foo,bar) { + var z = 42 + } + + val point = Point(1,2) + val color = Color.Green + point.z = 99 + val value = List( + point, + Map([1, "one"], ["two", 2]), + Set(1,2,3), + "hello".encodeUtf8(), + Date(2026,4,15), + color + ) + val restored = Json.decode(Json.encode(value)) + assertEquals(value, restored) + assertEquals(99, restored[0].z) + assertEquals("one", restored[1][1]) + assertEquals(true, restored[2].contains(3)) + assertEquals("hello", restored[3].decodeUtf8()) + assertEquals("2026-04-15", restored[4].toString()) + assertEquals(Color.Green, restored[5]) + """.trimIndent() + ) + } + + @Test + fun testJsonDecodePlainJsonString() = runTest { + val decoded = eval( + """ + import lyng.serialization + Json.decode("{\"foo\":[1,true,\"bar\"]}") + """.trimIndent() + ) + assertEquals("""{"foo":[1,true,"bar"]}""", decoded.toJson().toString()) + } + + @Test + fun testUniversalJsonExceptionRoundTrip() = runTest { + eval( + """ + import lyng.serialization + + class MyException : Exception("boom") + + val ex = MyException() + val restored = Json.decode(Json.encode(ex)) + assert(restored is MyException) + assert(restored is Exception) + assertEquals(ex.message, restored.message) + assert(restored.stackTrace.size > 0) + """.trimIndent() + ) + } + + @Test + fun testUniversalJsonSingletonObjectRoundTrip() = runTest { + eval( + """ + import lyng.serialization + + object Counter { + var value = 1 + } + + Counter.value = 77 + val restored = Json.decode(Json.encode(Counter)) + assertEquals(Counter, restored) + assertEquals(77, Counter.value) + assertEquals(77, restored.value) + """.trimIndent() + ) + } + @Serializable data class TestJson2( val value: Int, @@ -4722,6 +4807,34 @@ class ScriptTest { assertEquals(TestJson4(TestEnum.One), x) } + @Test + fun testExternalSerializationFormatRegistration() = runTest { + val im = Script.defaultImportManager.copy() + im.addPackage("test.formats") { module -> + module.bindSerializationFormat( + object : ObjSerializationFormatClass("Reverse") { + override suspend fun encodeValue(scope: Scope, value: Obj): Obj = + ObjString(value.toString(scope).value.reversed()) + + override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj { + val text = (encoded as? ObjString)?.value + ?: scope.raiseClassCastError("Reverse.decode expects String") + return ObjString(text.reversed()) + } + }, + doc = "Simple test format that reverses strings." + ) + } + val scope = im.newStdScope() + scope.eval( + """ + import test.formats + assertEquals("cba", Reverse.encode("abc")) + assertEquals("abc", Reverse.decode("cba")) + """.trimIndent() + ) + } + @Test fun testStringLast() = runTest { eval( diff --git a/proposals/serialization_format_registry.md b/proposals/serialization_format_registry.md new file mode 100644 index 0000000..978025e --- /dev/null +++ b/proposals/serialization_format_registry.md @@ -0,0 +1,25 @@ +# Serialization Format Registry + +Current status: + +- no global serialization-format registry +- formats are exported explicitly from modules and used explicitly, e.g. `Lynon.encode(...)`, `Json.decode(...)`, or `MyFormat.encode(...)` + +Why no registry now: + +- explicit module exports already solve the current use case +- a registry adds global mutable state and naming semantics we do not currently need +- there is no current runtime feature that needs format discovery by string name + +When a registry may become worth adding: + +- config-driven format selection, e.g. `"format": "json"` +- host-side introspection such as "list installed serialization formats" +- collision detection across independently loaded modules +- admin or tooling APIs that need to resolve a format without importing its module explicitly + +If added later, the registry should be: + +- optional, not required for normal explicit usage +- based on stable fully qualified ids, not just short export names +- designed as a host/tooling facility first, not as part of ordinary script-level serialization