Compare commits

..
100 Commits
Author SHA1 Message Date
sergeych 7fa491c0bb Publish AI reference discovery and embedding instructions 2026-10-21 03:57:27 +04:00
sergeych 0d727d64a5 Forward Gradle version properties through site deployment 2026-10-21 00:08:56 +04:00
sergeych c76b10e9fd Document compiler fixes and repair standalone documentation images 2026-10-21 00:04:58 +04:00
sergeych c34371b784 Preserve static return types and initialize singleton static methods 2026-10-21 00:01:11 +04:00
sergeych 04c9343a9d Reject class-named members and unsupported constructor delegation 2026-10-20 23:57:41 +04:00
sergeych 26a1ca9710 Add offline HTML parsing module to LyngIO 2026-10-16 02:01:49 +04:00
sergeych 66ab7db3d8 Fix multiline list parsing and expose actionable HTTP failures 2026-10-13 02:19:56 +04:00
sergeych 8b0d847776 Limit lyng.io HTTP redirect chains 2026-08-18 13:39:25 +04:00
Junie AI 4fa9b80592 Update build configurations for Java 17 compatibility and remove unused Modelfile 2026-08-15 17:08:49 +04:00
sergeych 9451eb574f Fix compatibility issues with Kotlin/Native commonization tasks during build and configuration cache 2026-07-28 21:18:27 +04:00
sergeych 7635664dbc version bump to 1.5.10 2026-07-28 17:22:21 +04:00
sergeych e66f646b25 Improve type inference for default initializers and add support for assign operator methods 2026-07-28 17:20:02 +04:00
sergeych 2f118a1fef docs on new function metadata API 2026-07-28 00:36:12 +04:00
sergeych 898ebfa536 Add resolved export metadata and fix imported accessors 2026-07-27 22:35:49 +04:00
sergeych 1a22b60b7d 1.5.9 deployed 2026-06-28 00:41:00 +03:00
sergeych 14080c3875 upgrade kotlin version to 2.4.0 2026-06-27 04:54:44 +03:00
sergeych 90e46a3fff fix another type inference problem with external structures 2026-06-27 04:35:51 +03:00
sergeych 7af6d3925e release 1.5.6 2026-05-16 11:18:48 +03:00
sergeych 234d1ef02b Add lyng.io.process module for process execution utilities and shell scripting sugar (e.g., sh, exec, CommandRun). 2026-05-10 15:04:32 +03:00
sergeych 3e74019d9d fixed CLI input problem and tetris sample 2026-05-10 13:00:08 +03:00
sergeych 31fac1a73c site: fixed .md/lyng code display 2026-04-30 09:50:10 +03:00
sergeych 53a9d21a19 docs improved 2026-04-29 22:39:43 +03:00
sergeych 739fdfc94b Add HTTP respondHtml sugar 2026-04-29 22:28:37 +03:00
sergeych c8e03d69ad Add Lyng HTML DSL helpers 2026-04-29 22:07:30 +03:00
sergeych b2200e71ff Add context receiver extensions for DSLs 2026-04-29 20:49:50 +03:00
sergeych e107296bca Add receiver-stack function types 2026-04-29 12:59:24 +03:00
sergeych 1bababa058 Normalize site TOC heading hierarchy 2026-04-27 15:21:44 +03:00
sergeych 35f4c968a4 T? ?: break inference fixed 2026-04-26 21:42:36 +03:00
sergeych 79429d5f2d added nullable ?: break support with proper inference ;) 2026-04-26 20:57:46 +03:00
sergeych fae9965bdf fix CLI boostrap of http server 2026-04-26 17:23:21 +03:00
sergeych 2dc4fb8230 more http server docs 2026-04-26 16:16:59 +03:00
sergeych f74ed9afe4 Switch HTTP server API to RequestContext receivers 2026-04-26 15:57:57 +03:00
sergeych ca4a0d4b12 Extend HTTP server routing and JSON exchange helpers 2026-04-26 13:42:14 +03:00
sergeych b969edd30a Fix commonMain cancellation handling in HTTP server loop 2026-04-26 10:21:33 +03:00
sergeych 01ceecd7df Add minimal HTTP server and shared network type packages 2026-04-26 10:09:25 +03:00
sergeych 79b015ee56 Fix callable return inference regressions 2026-04-25 19:07:10 +03:00
sergeych eba7158330 Fix SPA sample links and sample doc publishing 2026-04-25 16:46:51 +03:00
sergeych 66b8806b11 Add runnable DB serialization example 2026-04-25 16:23:08 +03:00
sergeych 92e9325f40 Add SQL object expansion serialization support 2026-04-25 16:09:30 +03:00
sergeych 50e34e520e Add DB decode annotations and preserved declaration metadata 2026-04-25 13:36:33 +03:00
sergeych 2abe7e2f96 Add typed canonical JSON encoding 2026-04-25 00:36:18 +03:00
sergeych 2bedaa0969 Add canonical Json format and external format binding 2026-04-25 00:03:00 +03:00
sergeych 9735774efd v1.5.5 release 2026-04-23 15:18:47 +03:00
sergeych 14214e91e1 Fix escaping immutable closure captures 2026-04-22 18:50:49 +03:00
sergeychandClaude Sonnet 4.6 0f5343fa17 Add VPS deployment target and Ansible provisioning for lynglang.com
- deploy_site now defaults to new VPS (94.130.36.94, /var/www/lynglang);
  use --old to deploy to d.lynglang.com as before
- infra/setup_vps.yml: Ansible playbook installs nginx + certbot, obtains
  SSL cert for lynglang.com, fixes Debian buster EOL apt sources
- infra/inventory.ini and infra/templates/nginx_lynglang.conf.j2 included

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 12:31:01 +03:00
sergeych d0f51928de fixed wrong error message in val ++/-- 2026-04-22 08:56:02 +03:00
sergeych 0973a6afeb Fix JVM import caching and class/object bytecode dispatch 2026-04-21 23:25:07 +03:00
sergeych 953f237ca3 Optimize primitive list fill capacity and append 2026-04-21 21:37:04 +03:00
sergeych fbb5688696 Drive higher-order lambda inlining from call metadata 2026-04-21 20:51:16 +03:00
sergeych f4ab2ebab4 Extend lambda inlining to getOrPut and implicit it calls 2026-04-21 19:20:25 +03:00
sergeych 0c3242cbd8 Generalize higher-order lambda inlining 2026-04-21 19:12:16 +03:00
sergeych 1d5caaa836 Broaden lambda method inlining with captures 2026-04-21 19:07:05 +03:00
sergeych 3be2892025 Use fast compiled callbacks in dynamic and flow helpers 2026-04-21 18:06:31 +03:00
sergeych c80900c503 Prevent cross-script module slot aliasing 2026-04-21 17:34:00 +03:00
sergeych ee634c8dff Extend generic fast callable entry points 2026-04-21 17:26:53 +03:00
sergeych 6c91b77a85 Use fast lambda calls in scope facade 2026-04-21 16:43:02 +03:00
sergeych 3721ee8332 Fast-path compiled lambda calls with args 2026-04-21 16:04:04 +03:00
sergeych 97990f00ce Cover when-based exact callable fast paths 2026-04-21 14:37:27 +03:00
sergeych 80693e7690 Inline exact lambdas through wrapper expressions 2026-04-21 14:36:18 +03:00
sergeych 029fe874fa Propagate exact callable refs across expressions 2026-04-21 14:33:23 +03:00
sergeych ffa64d691b Use direct calls for constant constructors 2026-04-21 13:30:09 +03:00
sergeych 30c9a5a565 Propagate exact lambda refs across bytecode slots 2026-04-21 13:20:27 +03:00
sergeych db3a780645 Generalize bytecode fast-call dispatch 2026-04-21 13:09:50 +03:00
sergeych fc7d26ee4b Fast-path bytecode statement execution 2026-04-21 11:39:47 +03:00
sergeych a61b5a31be Add non-suspending fast calls for bytecode callables 2026-04-21 11:35:20 +03:00
sergeych 33d170f525 Optimize fast-path bytecode lambda invocation 2026-04-21 11:29:14 +03:00
sergeych f72cdfdf83 Inline simple lambdas in bytecode fast paths 2026-04-21 09:05:18 +03:00
sergeych 10fa4de4fa regression fix 2026-04-20 18:48:28 +03:00
sergeych 4acee56de8 small optimization of List.fill 2026-04-20 14:09:09 +03:00
sergeych 87060c7eb7 +inference bugs
+List.ensureCapacity
optimized List.fill
2026-04-20 13:56:47 +03:00
sergeych 6ba128f7ba fixed some more inference bugs 2026-04-20 12:23:17 +03:00
sergeych a8f73dc8bd added LaunchPool 2026-04-18 23:37:21 +03:00
sergeychandClaude Sonnet 4.6 145c3ae34a Add completeExceptionally, cancelAndJoin, and refine LaunchPool spec
- CompletableDeferred.completeExceptionally(exception): fails the deferred
  with a Lyng exception; the original exception object is preserved as
  errorObject so assertThrows/catch see the correct class (works for both
  built-in ObjException and user-declared class instances)
- LaunchPool: add maxQueueSize parameter (suspends launch when full),
  add cancelAndJoin() for graceful shutdown that waits for running tasks,
  clarify all semantics in the extern declaration comments
- Two tests for completeExceptionally (built-in and user-defined exception)
- Fix parallelism.md channel doc-test to be illustration-only (avoid
  flakiness from coroutine scheduling in the doc-test runner)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 03:00:40 +03:00
sergeychandClaude Sonnet 4.6 f9a07f176a Add Channel primitive for coroutine inter-task communication
Exposes Kotlin's Channel<Obj> to Lyng scripts as a first-class type
with rendezvous, buffered, and unlimited capacity modes. Supports
suspending send/receive, non-suspending tryReceive, close/drain
semantics, and isClosedForSend/isClosedForReceive properties.

Also fixes a pre-existing typo in BookTest.kt that blocked JVM test
compilation, and adds Channel reference docs and a parallelism.md section.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 02:28:55 +03:00
sergeych 30e56946a0 improved JDBC provider for lyng.io.db 2026-04-18 02:06:29 +03:00
sergeych 49fc700233 Detach materialized SQL rows from transaction lifetime 2026-04-17 20:38:40 +03:00
sergeych 07cb5a519c Add JDBC and websocket/native follow-up changes 2026-04-16 23:35:15 +03:00
sergeych 8f66cd7680 Fix lynglib binding regressions and green test suite 2026-04-16 23:34:45 +03:00
sergeych bb2119b1d1 ignores 2026-04-16 22:12:19 +03:00
sergeych 64273ac60a Add JDBC database provider for JVM 2026-04-16 22:10:36 +03:00
sergeych ee392daa13 another sample 2026-04-16 13:35:12 +03:00
sergeych 0affa92674 better SQL example 2026-04-16 01:08:31 +03:00
sergeych a2e3f80ab6 better sql exaple, some inference bugs fixed 2026-04-16 00:54:00 +03:00
sergeych f8d2533b48 Make SQLite native linker lookup portable 2026-04-15 23:47:38 +03:00
sergeych bb9af2258b Fix SQLite release deployment 2026-04-15 23:35:59 +03:00
sergeych b3be908242 Add Date type and database release docs 2026-04-15 23:20:23 +03:00
sergeych 14dc73db3e Harden SQLite provider release behavior 2026-04-15 23:11:50 +03:00
sergeych f9bbdd56bf Test DB rollback precedence in SQLite 2026-04-15 22:38:11 +03:00
sergeych 04e80c384e Stabilize DB provider registration test 2026-04-15 22:35:19 +03:00
sergeych 6d340824e4 Tighten SQLite transaction failures and tests 2026-04-15 22:20:29 +03:00
sergeych 5f6f6b9ae4 Align SQLite value conversion across JVM and native 2026-04-15 22:03:42 +03:00
sergeych 55ba6113e7 Add Lyng DB contract and SQLite provider skeleton 2026-04-15 20:39:38 +03:00
sergeych b42ceec686 Preserve VM throw sites in stack traces 2026-04-14 01:17:04 +03:00
sergeych fada848907 Add backtick string literals and formatter support 2026-04-13 23:08:18 +03:00
sergeych e925195495 yet another import fix 2026-04-13 22:49:02 +03:00
sergeych 3b6bdda0a4 another import fix 2026-04-13 20:43:11 +03:00
sergeych ab39110834 Add repeated local import regression coverage 2026-04-12 11:11:32 +03:00
sergeych 47eb2d7d61 Fix transitive import re-exports and stabilize captures 2026-04-12 09:16:58 +03:00
sergeych eefa4c8f1a Fix stdlib import aliasing and launch captures 2026-04-11 20:24:44 +03:00
sergeych 6171b0d95f Fix duplicate stdlib import redefinition 2026-04-11 19:24:08 +03:00
258 changed files with 29176 additions and 1261 deletions

No files matched your search

+5
View File
@@ -28,3 +28,8 @@ debug.log
/compile_metadata_output.txt /compile_metadata_output.txt
test_output*.txt test_output*.txt
/site/src/version-template/lyng-version.js /site/src/version-template/lyng-version.js
/bugcontents.db
/bugs/
contents.db
/docs/idea_plugin.md.deploy_backup
/.claude/settings.local.json
+2
View File
@@ -1,6 +1,7 @@
# AI Agent Notes # AI Agent Notes
## Canonical AI References ## Canonical AI References
- Public discovery: `https://lynglang.com/llms.txt`; workflow: `docs/ai_agents.md`. Keep these links current when moving canonical references.
- Use `docs/ai_language_reference.md` as the primary, compiler-verified Lyng language reference for code generation. - Use `docs/ai_language_reference.md` as the primary, compiler-verified Lyng language reference for code generation.
- For generics-heavy code generation, follow `docs/ai_language_reference.md` section `7.1 Generics Runtime Model and Bounds` and `7.2 Differences vs Java / Kotlin / Scala`. - For generics-heavy code generation, follow `docs/ai_language_reference.md` section `7.1 Generics Runtime Model and Bounds` and `7.2 Differences vs Java / Kotlin / Scala`.
- Use `docs/ai_stdlib_reference.md` for default runtime/module APIs and stdlib surface. - Use `docs/ai_stdlib_reference.md` for default runtime/module APIs and stdlib surface.
@@ -13,6 +14,7 @@
- Prefer defining Lyng entities (enums/classes/type shapes) in `.lyng` files; only define them in Kotlin when there is Kotlin/platform-specific implementation detail that cannot be expressed in Lyng. - Prefer defining Lyng entities (enums/classes/type shapes) in `.lyng` files; only define them in Kotlin when there is Kotlin/platform-specific implementation detail that cannot be expressed in Lyng.
- Avoid hardcoding Lyng API documentation in Kotlin registrars when it can be declared in `.lyng`; Kotlin-side docs should be fallback/bridge only. - Avoid hardcoding Lyng API documentation in Kotlin registrars when it can be declared in `.lyng`; Kotlin-side docs should be fallback/bridge only.
- For mixed pluggable modules (Lyng + Kotlin), embed module `.lyng` sources as generated Kotlin string literals, evaluate them into module scope during registration, then attach Kotlin implementations/bindings. - For mixed pluggable modules (Lyng + Kotlin), embed module `.lyng` sources as generated Kotlin string literals, evaluate them into module scope during registration, then attach Kotlin implementations/bindings.
- When a change adds or changes Lyng-visible runtime/module behavior, update the corresponding `.lyng` declaration in the same change, including declaration-level docs/comments for new API surface.
## Kotlin/Wasm generation guardrails ## Kotlin/Wasm generation guardrails
- Avoid creating suspend lambdas for compiler runtime statements. Prefer explicit `object : Statement()` with `override suspend fun execute(...)`. - Avoid creating suspend lambdas for compiler runtime statements. Prefer explicit `object : Statement()` with `override suspend fun execute(...)`.
+71
View File
@@ -7,6 +7,77 @@ History note:
- Entries below are synchronized and curated for `1.5.x`. - Entries below are synchronized and curated for `1.5.x`.
- Earlier history may be incomplete and should be cross-checked with git tags/commits when needed. - Earlier history may be incomplete and should be cross-checked with git tags/commits when needed.
## Unreleased
- Reject member functions named after their enclosing class and diagnose unsupported `this(...)` constructor delegation at compile time.
- Preserve declared, inferred, and generic return types from static method calls.
- Initialize singleton static members and dispatch their calls through class scope, including trailing-lambda calls.
- Update the language reference and OOP guide with constructor naming rules and typed static factory examples.
## 1.5.6 (2026-05-16)
### Process execution and CLI scripting
- Added `lyng.io.process` for coroutine-friendly external process execution.
- Added `sh(...)`, `exec(...)`, and `CommandRun` with captured output, streaming output, `wait()`, `check()`, and process status helpers.
- Documented safe shell-vs-argv usage and common script patterns for CLI automation.
### HTTP server and HTML rendering
- Added the minimal `lyng.io.http.server` API for embedded HTTP/1.1 services, local tools, test fixtures, and lightweight app backends.
- Added exact-path, regex, and path-template routing with named route parameters.
- Added `RequestContext` receiver sugar, JSON helpers (`jsonBody<T>()`, `respondJson(...)`), reusable routers, and WebSocket routes.
- Added `respondHtml { ... }` and the pure Lyng `lyng.io.html` builder DSL with escaped text, attributes, common tag helpers, and generic tag escape hatches.
### Language and serialization
- Added receiver-stack function types and context receiver extensions for DSL-style APIs.
- Added canonical JSON round-trip APIs with `Json.encode(...)` / `Json.decode(...)`.
- Added typed canonical JSON APIs with `Json.encodeAs(Type, value)` / `Json.decodeAs(Type, text)`.
- Extended database serialization support with typed canonical JSON encoding, SQL object expansion, decode annotations, and preserved declaration metadata.
### Runtime, docs, and samples
- Fixed nullable Elvis-with-`break` inference and callable return inference regressions.
- Fixed CLI bootstrap behavior for HTTP server scripts and CLI input handling.
- Expanded docs for HTTP, WebSocket, serialization, process execution, DSL receivers, and the generated site.
- Release metadata and README now point to `1.5.6`.
## 1.5.5 (2026-04-23)
### Concurrency and collections
- Added coroutine coordination primitives and helpers for everyday parallel code:
- `Channel` for coroutine-to-coroutine communication
- `LaunchPool` for bounded-concurrency task execution
- `Iterable<Deferred>.joinAll()` to await a whole collection of deferreds in input order
- `CompletableDeferred.completeExceptionally(...)` and `Deferred.cancelAndJoin()`
- Added docs and examples for the new concurrency APIs, including `joinAll()` coverage in iterable and parallelism references.
### Database and time APIs
- Added the portable `lyng.io.db` SQL contract and the first concrete providers:
- `lyng.io.db.sqlite` on JVM and Linux Native
- `lyng.io.db.jdbc` on JVM
- Added SQLite/JDBC release hardening:
- nested transactions via savepoints
- detached materialized rows
- generated-key support through `ExecutionResult.getGeneratedKeys()`
- schema-driven value conversion for `Bool`, `Decimal`, `Date`, `DateTime`, and `Instant`
- portable SQLite linker/deployment fixes and documented runtime options
- Added `Date` to `lyng.time` and the core runtime as a first-class calendar-date type, plus conversions and arithmetic across `Instant`, `DateTime`, and `Date`.
### Language, stdlib, and tooling
- Added extensions on singleton `object` declarations, including object-scoped indexer overrides for bracket syntax.
- Added backtick string literals and formatter support.
- Added `lyng.legacy_digest` for SHA-1 compatibility work, `String.replace`, and `buffer.base64std`.
- Improved CLI/runtime behavior with `atExit` shutdown handlers, native release-binary work, and follow-up CLI packaging/import fixes.
- Expanded docs across the tutorial, stdlib references, database docs, networking docs, and release notes.
### Runtime/compiler stability and performance
- Extended exact-call and higher-order lambda inlining through the bytecode compiler, including compiled fast paths for simple lambdas, wrappers, captures, and common higher-order helpers.
- Fixed import caching and class/object bytecode dispatch on JVM.
- Fixed immutable `val` compound assignments so true mutating `*Assign` operations continue to work while fallback reassignments report the correct read-only error.
- Fixed closure/capture and import regressions across launched loops, singleton/object extensions, aliasing, transitive re-exports, and immutable capture escaping.
- Improved list-fill/list-append fast paths, nullable-let inference, Decimal/Complex interop, and related regression coverage.
### Release notes
- Release metadata, homepage samples, docs, and README now point to `1.5.5`.
## 1.5.4 (2026-04-03) ## 1.5.4 (2026-04-03)
### Runtime and compiler stability ### Runtime and compiler stability
+11 -3
View File
@@ -47,11 +47,19 @@ assertEquals(A.E.One, A.One)
## Resources: ## Resources:
- [Language home](https://lynglang.com) - [Language home](https://lynglang.com)
- [AI agent instructions](docs/ai_agents.md) and [machine-readable entry point](https://lynglang.com/llms.txt)
- [introduction and tutorial](docs/tutorial.md) - start here please - [introduction and tutorial](docs/tutorial.md) - start here please
- [Latest release notes (1.5.4)](docs/whats_new.md) - [Latest release notes (1.5.6)](docs/whats_new.md)
- [What's New in 1.5](docs/whats_new_1_5.md) - [What's New in 1.5](docs/whats_new_1_5.md)
- [Testing and Assertions](docs/Testing.md) - [Testing and Assertions](docs/Testing.md)
- [Filesystem and Processes (lyngio)](docs/lyngio.md) - [Filesystem and Processes (lyngio)](docs/lyngio.md)
- [SQL Databases (lyng.io.db)](docs/lyng.io.db.md)
- [HTTP Server (lyng.io.http.server)](docs/lyng.io.http.server.md)
- [HTML Builder (lyng.io.html)](docs/lyng.io.html.md)
- [HTML Parsing (lyng.io.html.parse)](docs/lyng.io.html.parse.md)
- [Process Execution (lyng.io.process)](docs/lyng.io.process.md)
- [Time and Calendar Types](docs/time.md)
- [Json support](docs/json_and_kotlin_serialization.md)
- [Return Statement](docs/return_statement.md) - [Return Statement](docs/return_statement.md)
- [Efficient Iterables in Kotlin Interop](docs/EfficientIterables.md) - [Efficient Iterables in Kotlin Interop](docs/EfficientIterables.md)
- [Samples directory](docs/samples) - [Samples directory](docs/samples)
@@ -64,7 +72,7 @@ assertEquals(A.E.One, A.One)
### Add dependency to your project ### Add dependency to your project
```kotlin ```kotlin
val lyngVersion = "1.5.4" val lyngVersion = "1.5.6"
repositories { repositories {
// ... // ...
@@ -184,7 +192,7 @@ Designed to add scripting to kotlin multiplatform application in easy and effici
# Language Roadmap # Language Roadmap
The current stable release is **v1.5.4**: the 1.5 cycle is feature-complete, compiler/runtime stabilization work is in, and the language, tooling, and site are aligned around the current release. The current stable release is **v1.5.6**: the 1.5 cycle now includes the database/date/concurrency additions, process execution, lightweight HTTP server and HTML DSL work, canonical Json serialization, and the latest compiler/runtime stabilization work.
Ready features: Ready features:
+48 -8
View File
@@ -17,15 +17,55 @@
# #
# #
set -e set -euo pipefail
echo "publishing all artifacts"
echo
./gradlew publishToMavenLocal site:jsBrowserDistribution publish buildInstallablePlugin :lyng:linkReleaseExecutableLinuxX64 :lyng:installJvmDist --parallel
#echo GRADLE_ARGS=(--no-configuration-cache --console=plain)
#echo "Creating plugin"
#echo echo "actualizing Kotlin JS/Wasm lock files"
#./gradlew buildInstallablePlugin echo
./gradlew kotlinUpgradeYarnLock kotlinWasmUpgradeYarnLock "${GRADLE_ARGS[@]}"
echo "building all artifacts"
echo
./gradlew publishToMavenLocal site:jsBrowserDistribution buildInstallablePlugin :lyng:linkReleaseExecutableLinuxX64 :lyng:installJvmDist --parallel "${GRADLE_ARGS[@]}"
echo
echo "publishing remote Maven artifacts"
echo
tasks_log="$(mktemp)"
./gradlew tasks --all "${GRADLE_ARGS[@]}" >"$tasks_log"
mapfile -t publish_tasks < <(
sed -n 's/^\([^[:space:]]*publish.*PublicationToMavenRepository\) - .*$/:\1/p' "$tasks_log" |
sort -u
)
rm -f "$tasks_log"
if [ "${#publish_tasks[@]}" -eq 0 ]; then
echo "No remote Maven publication tasks found."
exit 1
fi
for task in "${publish_tasks[@]}"; do
echo
echo "Publishing $task"
publish_log="$(mktemp)"
if ./gradlew "$task" "${GRADLE_ARGS[@]}" >"$publish_log" 2>&1; then
cat "$publish_log"
rm -f "$publish_log"
continue
fi
if grep -Eiq '(^|[^0-9])409([^0-9]|$)|already exists|already been published|cannot be updated|Package file already exists|Repository does not allow updating assets' "$publish_log"; then
cat "$publish_log"
echo "Skipping $task: artifact already exists in the Maven repository."
rm -f "$publish_log"
continue
fi
cat "$publish_log"
rm -f "$publish_log"
exit 1
done
echo echo
echo "building CLI tools" echo "building CLI tools"
+19 -12
View File
@@ -17,11 +17,17 @@
# #
# #
# Pass snapshot overrides through explicitly, e.g. -Plyng.version=1.5.11-SNAPSHOT.
GRADLE_ARGS=()
upload_only=false upload_only=false
target=vps # default: new server; use --old for d.lynglang.com
for arg in "$@"; do for arg in "$@"; do
if [[ "$arg" == "-u" || "$arg" == "--upload-only" ]]; then if [[ "$arg" == "-u" || "$arg" == "--upload-only" ]]; then
upload_only=true upload_only=true
break elif [[ "$arg" == -P* ]]; then
GRADLE_ARGS+=("$arg")
elif [[ "$arg" == "--old" ]]; then
target=com
fi fi
done done
@@ -88,19 +94,20 @@ function updateIdeaPluginDownloadLink() {
fi fi
} }
# default target settings # target settings (-t com | -t vps)
case "com" in case "$target" in
com) com)
SSH_HOST=sergeych@d.lynglang.com # host to deploy to SSH_HOST=sergeych@d.lynglang.com
SSH_PORT=22 # ssh port on it SSH_PORT=22
ROOT=/bigstore/sergeych_pub/lyng # directory to rsync to ROOT=/bigstore/sergeych_pub/lyng
;;
vps)
SSH_HOST=sergeych@94.130.36.94
SSH_PORT=22
ROOT=/var/www/lynglang
;; ;;
# com)
# SSH_HOST=vvk@front-01.neurodatalab.com
# ROOT=/home/vvk
# ;;
*) *)
echo "*** ERROR: target not specified (use deploy com | dev)" echo "*** ERROR: unknown target '$target' (use -t com | -t vps)"
echo "*** stop" echo "*** stop"
exit 101 exit 101
esac esac
@@ -122,7 +129,7 @@ if [[ "$upload_only" == false ]]; then
refreshTextmateZip refreshTextmateZip
updateIdeaPluginDownloadLink || echo "WARN: proceeding without updating IDEA plugin download link" updateIdeaPluginDownloadLink || echo "WARN: proceeding without updating IDEA plugin download link"
./gradlew site:jsBrowserDistribution ./gradlew site:jsBrowserDistribution "${GRADLE_ARGS[@]}"
BUILD_RC=$? BUILD_RC=$?
# Always restore original doc if backup exists # Always restore original doc if backup exists
+2 -1
View File
@@ -113,9 +113,10 @@ process_file() {
# - [text](file.md) -> [text](#file.md) # - [text](file.md) -> [text](#file.md)
# - [text](dir/file.md) -> [text](#dir_file.md) # - [text](dir/file.md) -> [text](#dir_file.md)
# - [text](file.md#anchor) -> [text](#anchor) # - [text](file.md#anchor) -> [text](#anchor)
# - Fix image links: [alt](../images/...) -> [alt](images/...) if needed, but none found yet. # - Resolve site-root image URLs against the local site resources for embedding.
cat "$full_path" | \ cat "$full_path" | \
perl -pe 's{(!\[[^\]]*\]\()/}{$1site/src/jsMain/resources/}g' | \
perl -pe 's/\[([^\]]+)\]\(([^)]+)\.md\)/"[$1](#" . ($2 =~ s|\/|_|gr) . ".md)"/ge' | \ perl -pe 's/\[([^\]]+)\]\(([^)]+)\.md\)/"[$1](#" . ($2 =~ s|\/|_|gr) . ".md)"/ge' | \
perl -pe 's/\[([^\]]+)\]\(([^)]+)\.md#([^)]+)\)/[$1](#$3)/g' >> "$MERGED_MD" perl -pe 's/\[([^\]]+)\]\(([^)]+)\.md#([^)]+)\)/[$1](#$3)/g' >> "$MERGED_MD"
+17
View File
@@ -21,6 +21,23 @@ plugins {
alias(libs.plugins.vanniktech.mavenPublish) apply false alias(libs.plugins.vanniktech.mavenPublish) apply false
} }
// Some JVM dependencies are published as Java 21 bytecode. Keep the library's
// Java 17 compilation target, but use a compatible runtime for its JVM tests.
project(":lynglib") {
afterEvaluate {
val toolchains = extensions.getByType<org.gradle.jvm.toolchain.JavaToolchainService>()
tasks.withType<Test>().configureEach {
javaLauncher.set(toolchains.launcherFor {
languageVersion.set(JavaLanguageVersion.of(21))
})
// An explicit jvmTest invocation should execute the suite rather than
// silently reusing the previous test task outputs or build-cache entry.
outputs.upToDateWhen { false }
outputs.cacheIf { false }
}
}
}
// Convenience alias to run the IntelliJ IDE with the Lyng plugin from the project root. // Convenience alias to run the IntelliJ IDE with the Lyng plugin from the project root.
// Usage: ./gradlew runIde // Usage: ./gradlew runIde
// It simply delegates to :lyng-idea:runIde provided by the Gradle IntelliJ Plugin. // It simply delegates to :lyng-idea:runIde provided by the Gradle IntelliJ Plugin.
+211
View File
@@ -0,0 +1,211 @@
# Channel
A `Channel` is a **hot, bidirectional pipe** for passing values between concurrently running coroutines.
Unlike a [Flow], which is cold and replayed on every collection, a `Channel` is stateful: each value
sent is consumed by exactly one receiver.
Channels model the classic _producer / consumer_ pattern and are the right tool when:
- two or more coroutines need to exchange individual values at their own pace;
- you want back-pressure (rendezvous) or explicit buffering control;
- you need a push-based, hot data source (opposite of the pull-based, cold [Flow]).
## Constructors
```
Channel() // rendezvous — sender and receiver must meet
Channel(n: Int) // buffered — sender may run n items ahead of the receiver
Channel(Channel.UNLIMITED) // no limit on buffered items
```
**Rendezvous** (`Channel()`, capacity 0): `send` suspends until a matching `receive` is ready,
and vice-versa. This gives the tightest synchronisation and the smallest memory footprint.
**Buffered** (`Channel(n)`): `send` only suspends when the internal buffer is full. Allows the
producer to get up to _n_ items ahead of the consumer.
**Unlimited** (`Channel(Channel.UNLIMITED)`): `send` never suspends. Useful when the producer is
bursty and you do not want it blocked, but be careful not to grow the buffer without bound.
## Sending and receiving
```lyng
val ch = Channel() // rendezvous channel
val producer = launch {
ch.send("hello") // suspends until the receiver is ready
ch.send("world")
ch.close() // signal: no more values
}
val a = ch.receive() // suspends until "hello" arrives
val b = ch.receive() // suspends until "world" arrives
val c = ch.receive() // channel is closed and drained → null
assertEquals("hello", a)
assertEquals("world", b)
assertEquals(null, c)
```
`receive()` returns `null` when the channel is both **closed** _and_ **fully drained** — that is
the idiomatic loop termination condition:
```lyng
val ch = Channel(4)
launch {
for (i in 1..5) ch.send(i)
ch.close()
}
var item = ch.receive()
while (item != null) {
println(item)
item = ch.receive()
}
```
## Non-suspending poll
`tryReceive()` never suspends. It returns the next buffered value, or `null` if the buffer is
empty or the channel is closed.
```lyng
val ch = Channel(8)
ch.send(42)
println(ch.tryReceive()) // 42
println(ch.tryReceive()) // null — nothing buffered right now
```
Use `tryReceive` for _polling_ patterns where blocking would be unacceptable, for example when
combining channel checks with other work inside a coroutine loop.
## Closing a channel
`close()` marks the channel so that no further `send` calls are accepted. Any items already in the
buffer can still be received. Once the buffer is drained, `receive()` returns `null` and
`isClosedForReceive` becomes `true`.
```lyng
val ch = Channel(2)
ch.send(1)
ch.send(2)
ch.close()
assert(ch.isClosedForSend)
assert(!ch.isClosedForReceive) // still has 2 buffered items
ch.receive() // 1
ch.receive() // 2
assert(ch.isClosedForReceive) // drained
```
Calling `send` after `close()` throws `IllegalStateException`.
## Properties
| property | type | description |
|---------------------|--------|----------------------------------------------------------|
| `isClosedForSend` | `Bool` | `true` after `close()` is called |
| `isClosedForReceive`| `Bool` | `true` when closed _and_ every buffered item is consumed |
## Methods
| method | suspends | description |
|-----------------|----------|----------------------------------------------------------------------------------|
| `send(value)` | yes | send a value; suspends when buffer full (rendezvous: always until partner ready) |
| `receive()` | yes | receive next value; suspends when empty; returns `null` when closed + drained |
| `tryReceive()` | no | return next buffered value or `null`; never suspends |
| `close()` | no | signal end of production; existing buffer items are still receivable |
## Static constants
| constant | value | description |
|---------------------|------------------|-------------------------------------|
| `Channel.UNLIMITED` | `Int.MAX_VALUE` | capacity for an unlimited-buffer channel |
## Common patterns
### Producer / consumer
```lyng
val ch = Channel()
val results = []
val mu = Mutex()
val consumer = launch {
var item = ch.receive()
while (item != null) {
mu.withLock { results += item }
item = ch.receive()
}
}
launch {
for (i in 1..5) ch.send("msg:$i")
ch.close()
}.await()
consumer.await()
println(results)
```
### Fan-out: one channel, many consumers
```lyng
val ch = Channel(16)
// multiple consumers
val workers = (1..4).map { id ->
launch {
var task = ch.receive()
while (task != null) {
println("worker $id handles $task")
task = ch.receive()
}
}
}
// single producer
for (i in 1..20) ch.send(i)
ch.close()
workers.forEach { it.await() }
```
### Ping-pong between two coroutines
```lyng
val ping = Channel()
val pong = Channel()
launch {
repeat(3) {
val msg = ping.receive()
println("got: $msg → sending pong")
pong.send("pong")
}
}
repeat(3) {
ping.send("ping")
println(pong.receive())
}
```
## Channel vs Flow
| | [Flow] | Channel |
|---|---|---|
| **temperature** | cold (lazy) | hot (eager) |
| **replay** | every collector gets a fresh run | each item is consumed once |
| **consumers** | any number; each gets all items | one receiver per item |
| **back-pressure** | built-in via rendezvous | configurable (rendezvous / buffered / unlimited) |
| **typical use** | transform pipelines, sequences | producer–consumer, fan-out |
## See also
- [parallelism] — `launch`, `Deferred`, `Mutex`, `Flow`, and the full concurrency picture
- [Flow] — cold async sequences
[Flow]: parallelism.md#flow
[parallelism]: parallelism.md
+18
View File
@@ -55,6 +55,23 @@ Here is the sample:
assertEquals( (1..3).joinToString { it * 10 }, "10 20 30") assertEquals( (1..3).joinToString { it * 10 }, "10 20 30")
>>> void >>> void
## joinAll
`joinAll()` is an `Iterable<Deferred>` helper that awaits every deferred in iteration order and returns a `List`
with the collected results.
val jobs = (1..4).map { n ->
launch { n * n }
}
assertEquals([1, 4, 9, 16], jobs.joinAll())
>>> void
Notes:
- it does not start any task by itself; it only awaits the deferreds already present in the iterable.
- awaiting happens in iteration order, so the result list keeps the same order as the input iterable.
- if any deferred fails or was cancelled, that `await()` error is propagated from `joinAll()`.
## `sum` and `sumOf` ## `sum` and `sumOf`
These, again, does the thing: These, again, does the thing:
@@ -184,6 +201,7 @@ Search for the first element that satisfies the given predicate:
| sortedWith(comparator) | sort using a comparator that compares elements (1) | | sortedWith(comparator) | sort using a comparator that compares elements (1) |
| sortedBy(predicate) | sort by comparing results of the predicate function | | sortedBy(predicate) | sort by comparing results of the predicate function |
| joinToString(s,t) | convert iterable to string, see (2) | | joinToString(s,t) | convert iterable to string, see (2) |
| joinAll() | for `Iterable<Deferred>`, await all items in order and collect results to [List] |
| reversed() | create a list containing items from this in reverse order | | reversed() | create a list containing items from this in reverse order |
| shuffled() | create a list of shuffled elements | | shuffled() | create a list of shuffled elements |
+110
View File
@@ -0,0 +1,110 @@
# LaunchPool
`LaunchPool` is a bounded-concurrency task pool: you submit lambdas with `launch`, and the pool runs them using a fixed number of worker coroutines.
## Constructor
```
LaunchPool(maxWorkers, maxQueueSize = Channel.UNLIMITED)
```
| Parameter | Description |
|-----------|-------------|
| `maxWorkers` | Maximum number of tasks that run in parallel. |
| `maxQueueSize` | Maximum number of tasks that may wait in the queue. When the queue is full, `launch` suspends the caller until space becomes available. Defaults to `Channel.UNLIMITED` (no bound). |
## Methods
### `launch(lambda): Deferred`
Schedules `lambda` for execution and returns a `Deferred` for its result.
- Suspends if the queue is full (`maxQueueSize` reached).
- Throws `IllegalStateException` if the pool is already closed or cancelled.
- Any exception thrown by `lambda` is captured in the returned `Deferred` and **does not escape the pool**.
```lyng
val pool = LaunchPool(4)
val d1 = pool.launch { computeSomething() }
val d2 = pool.launch { computeOther() }
pool.closeAndJoin()
println(d1.await())
println(d2.await())
```
### `closeAndJoin()`
Stops accepting new tasks and suspends until all queued and running tasks complete normally. After this call, any further `launch` throws `IllegalStateException`. Idempotent — safe to call multiple times.
### `cancel()`
Immediately closes the queue and cancels all worker coroutines. Queued but unstarted tasks are discarded. After this call, `launch` throws `IllegalStateException`. Idempotent.
### `cancelAndJoin()`
Like `cancel()`, but also suspends until all worker coroutines have stopped. Useful when you need to be sure no worker code is still running before proceeding. Idempotent.
## Exception handling
Exceptions from submitted lambdas are captured per-task in the returned `Deferred`. The pool itself continues running after a task failure:
```lyng
val pool = LaunchPool(2)
val good = pool.launch { 42 }
val bad = pool.launch { throw IllegalArgumentException("boom") }
pool.closeAndJoin()
assertEquals(42, good.await())
assertThrows(IllegalArgumentException) { bad.await() }
```
## Bounded queue / back-pressure
When `maxQueueSize` is set, the producer suspends if the queue fills up, providing automatic back-pressure:
```lyng
// 1 worker, queue of 2 — producer can be at most 2 tasks ahead of what's running
val pool = LaunchPool(1, 2)
val d1 = pool.launch { delay(10); "a" }
val d2 = pool.launch { delay(10); "b" }
val d3 = pool.launch { delay(10); "c" } // suspends until d1 is picked up by the worker
pool.closeAndJoin()
```
## Collecting all results
`launch` returns a `Deferred`, so you can collect results with `joinAll()`:
```lyng
val pool = LaunchPool(4)
val jobs = (1..10).map { n -> pool.launch { n * n } }
pool.closeAndJoin()
val results = jobs.joinAll()
// results == [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
```
## Concurrency limit in practice
With `maxWorkers = 2`, at most 2 tasks run simultaneously regardless of how many are queued:
```lyng
val mu = Mutex()
var active = 0
var maxSeen = 0
val pool = LaunchPool(2)
(1..8).map {
pool.launch {
mu.withLock { active++; if (active > maxSeen) maxSeen = active }
delay(5)
mu.withLock { active-- }
}
}
pool.closeAndJoin()
assert(maxSeen <= 2)
```
## See also
- [parallelism.md](parallelism.md) — `launch`, `Deferred`, `Mutex`, `Channel`, and coroutine basics
- [Channel.md](Channel.md) — the underlying channel primitive used by `LaunchPool`
+7
View File
@@ -175,6 +175,8 @@ List could be sorted in place, just like [Collection] provide sorted copies, in
| `[Range]` | get slice of the array (copy) | Range | | `[Range]` | get slice of the array (copy) | Range |
| `+=` | append element(s) (2) | List or Obj | | `+=` | append element(s) (2) | List or Obj |
| `List.fill(size, block)` | build a new list from indices `0..<size` | Int, Callable | | `List.fill(size, block)` | build a new list from indices `0..<size` | Int, Callable |
| `List.fill(size,capacity,block)` | same, pre-allocating capacity slots | Int, Int, Callable |
| `ensureCapacity(count)` | pre-allocate storage for at least `count` elements without reallocation (5) | Int |
| `sort()` | in-place sort, natural order | void | | `sort()` | in-place sort, natural order | void |
| `sortBy(predicate)` | in-place sort bu `predicate` call result (3) | void | | `sortBy(predicate)` | in-place sort bu `predicate` call result (3) | void |
| `sortWith(comparator)` | in-place sort using `comarator` function (4) | void | | `sortWith(comparator)` | in-place sort using `comarator` function (4) | void |
@@ -197,6 +199,11 @@ order, e.g. is same as `list.sortWith { a,b -> predicate(a) <=> predicate(b) }`
positive if first is greater, and zero if they are equal. For example, the equvalent comparator positive if first is greater, and zero if they are equal. For example, the equvalent comparator
for `sort()` will be `sort { a, b -> a <=> b } for `sort()` will be `sort { a, b -> a <=> b }
(5)
: if the current capacity is already ≥ `count`, this is a no-op. Otherwise the internal storage
is reallocated to hold at least `count` elements. Use this before a bulk `+=` loop to avoid
repeated reallocations. `List.fill(size, capacity, block)` calls this automatically.
It inherits from [Iterable] too and thus all iterable methods are applicable to any list. It inherits from [Iterable] too and thus all iterable methods are applicable to any list.
## Observable list hooks ## Observable list hooks
+67 -1
View File
@@ -42,6 +42,12 @@ a _constructor_ that requires two parameters for fields. So when creating it wit
Form now on `Point` is a class, it's type is `Class`, and we can create instances with it as in the Form now on `Point` is a class, it's type is `Class`, and we can create instances with it as in the
example above. example above.
A member function cannot have the same name as its enclosing class, including a
static member function. Constructors are declared in the class header; use the
class name to create another instance. `this(...)` and `this@Type(...)` do not
declare or delegate to secondary constructors. Local functions may shadow class
names according to the normal lexical scope rules.
## Singleton Objects ## Singleton Objects
Singleton objects are declared using the `object` keyword. An `object` declaration defines both a class and a single instance of that class at the same time. This is perfect for stateless utilities, global configuration, or shared delegates. Singleton objects are declared using the `object` keyword. An `object` declaration defines both a class and a single instance of that class at the same time. This is perfect for stateless utilities, global configuration, or shared delegates.
@@ -454,6 +460,43 @@ Key rules and features:
- For arbitrary receivers, use casts: `(expr as Type).member(...)` or `(expr as? Type)?.member(...)`. - For arbitrary receivers, use casts: `(expr as Type).member(...)` or `(expr as? Type)?.member(...)`.
- Qualified access does not relax visibility. - Qualified access does not relax visibility.
### Receiver-stack lambdas
Qualified `this@Type` is also used outside inheritance when a lambda has multiple visible receivers.
This is common in DSL-style builders.
- `A & B` means one receiver value that implements both types.
- `context(A, B) C.()->R` means a receiver stack:
- primary `this` is `C`
- outer/context receivers are `A`, then `B`
- Unqualified lookup checks the primary receiver first.
- If the primary receiver does not define a member and several outer/context receivers do, Lyng reports a compile-time ambiguity. Use `this@Type` to select one explicitly.
Example:
```lyng
class Html { fun title() = "html" }
class Head { fun title() = "head" }
class Body
val block: context(Html, Head) Body.()->String = {
// title() // compile-time ambiguity: Html vs Head
this@Html.title()
}
```
Context receivers can also constrain extension functions. The extension is visible only when the required receiver is
already in the implicit receiver stack:
```lyng
class Tag { fun addText(text: String) { /* ... */ } }
context(Tag)
fun String.unaryPlus() {
this@Tag.addText(this)
}
```
- Field inheritance (`val`/`var`) and collisions - Field inheritance (`val`/`var`) and collisions
- Instance storage is kept per declaring class, internally disambiguated; unqualified read/write resolves to the first match in the resolution order (leftmost base). - Instance storage is kept per declaring class, internally disambiguated; unqualified read/write resolves to the first match in the resolution order (leftmost base).
- Qualified read/write (via `this@Type` or casts) targets the chosen ancestor’s storage. - Qualified read/write (via `this@Type` or casts) targets the chosen ancestor’s storage.
@@ -625,10 +668,14 @@ Unary operators are overloaded by defining methods with no arguments:
| Operator | Method Name | | Operator | Method Name |
| :--- | :--- | | :--- | :--- |
| `+a` | `unaryPlus()` |
| `-a` | `negate()` | | `-a` | `negate()` |
| `!a` | `logicalNot()` | | `!a` | `logicalNot()` |
| `~a` | `bitNot()` | | `~a` | `bitNot()` |
`unaryPlus()` is useful in DSL-style builders where `+"text"` should append text to
the current receiver. See [samples/html_builder_dsl.lyng](samples/html_builder_dsl.lyng).
### Assignment Operators ### Assignment Operators
Assignment operators like `+=` first attempt to call a specific assignment method. If that method is not defined, they fall back to a combination of the binary operator and a regular assignment (e.g., `a = a + b`). Assignment operators like `+=` first attempt to call a specific assignment method. If that method is not defined, they fall back to a combination of the binary operator and a regular assignment (e.g., `a = a + b`).
@@ -1001,6 +1048,25 @@ Static fields can be accessed from static methods via the class qualifier:
assertEquals("bar", Test.getData() ) assertEquals("bar", Test.getData() )
>>> void >>> void
Static calls preserve their declared or inferred return types, so their results
can be used directly in chained expressions or assigned without a type annotation.
Generic static factories also preserve the type inferred from their arguments:
class StaticBox<T>(val value: T) {
static fun of<T>(value: T): StaticBox<T> = StaticBox(value)
}
val box = StaticBox.of("hello")
assertEquals(5, box.value.length)
>>> void
Named singleton objects also support static methods:
object StaticGreeting {
static fun text(): String = "hello"
}
assertEquals(5, StaticGreeting.text().length)
>>> void
# Extension members # Extension members
Sometimes an existing type or named singleton object is missing some particular functionality that can be _added to it_ without rewriting its inner logic and without using its private state. In this case, _extension members_ can be used. Sometimes an existing type or named singleton object is missing some particular functionality that can be _added to it_ without rewriting its inner logic and without using its private state. In this case, _extension members_ can be used.
@@ -1207,7 +1273,7 @@ scopeB()
assertThrows { 42.description } assertThrows { 42.description }
``` ```
This isolation ensures that libraries can use extensions internally without worrying about name collisions with other libraries or the user's code. When a module is imported using `use`, its top-level extensions become available in the importing scope. This isolation ensures that libraries can use extensions internally without worrying about name collisions with other libraries or the user's code. When a package is imported using `import`, its top-level extensions become available in the importing scope. An imported extension property's getter and setter retain the lexical scope of the package that declared them, so they can safely call that package's private helpers and bound `extern` functions and properties.
## dynamic symbols ## dynamic symbols
+24
View File
@@ -0,0 +1,24 @@
# Lyng instructions for AI agents
Before writing Lyng, read the [current language reference](ai_language_reference.md)
and [compact standard-library reference](ai_stdlib_reference.md). These are the
canonical implementation-oriented references; older specifications and proposals
must not override them. Discover them from <https://lynglang.com/llms.txt>.
Use the language reference for syntax, types, generic bounds, imports, and current
limitations. Use the stdlib reference to distinguish default globals from explicit
imports and optional host modules. Follow links to individual API pages when you
need more detail. Do not guess Kotlin/Java method names or assume optional modules
are installed. Check the host runtime version and validate uncertain constructs
with a small read-only script before changing user data.
For offline embedding, copy both compact references from the same committed Lyng
revision, retain their paths and commit ID, and split long results at section
boundaries to fit the host's tool output limit. Keep host-specific capability and
permission documentation alongside them. In Tetradion, call `agent.docsList()` or
`agent.docsSearch("query")`, then `agent.docs("reference.syntax")`,
`agent.docs("reference.flow")`, `agent.docs("reference.types")`,
`agent.docs("reference.modules")`, or `agent.docs("reference.stdlib")`.
The site publishes these Markdown files directly under `/docs/`, without requiring
JavaScript or browser automation.
+23 -6
View File
@@ -15,15 +15,17 @@ Primary sources used: `lynglib/src/commonMain/kotlin/net/sergeych/lyng/{Parser,T
## 2. Lexical Syntax ## 2. Lexical Syntax
- Comments: `// line`, `/* block */`. - Comments: `// line`, `/* block */`.
- Strings: `"..."` (supports escapes). Multiline string content is normalized by indentation logic. - Strings: `"..."` or `` `...` `` (supports escapes). Multiline string content is normalized by indentation logic.
- Supported escapes: `\n`, `\r`, `\t`, `\"`, `\\`, `\uXXXX` (4 hex digits). - AI generation preference: use `"..."` by default, including multiline strings; `"` strings are also multiline-capable and should be preferred for ordinary code/doc/SQL text. Use backtick strings mainly when the content contains many double quotes and backticks would make the source clearer.
- Shared escapes: `\n`, `\r`, `\t`, `\\`, `\uXXXX` (4 hex digits).
- Delimiter escapes: `\"` inside `"..."`, ``\` `` inside `` `...` ``.
- Unicode escapes use exactly 4 hex digits (for example: `"\u0416"` -> `Ж`). - Unicode escapes use exactly 4 hex digits (for example: `"\u0416"` -> `Ж`).
- Unknown `\x` escapes in strings are preserved literally as two characters (`\` and `x`). - Unknown `\x` escapes in strings are preserved literally as two characters (`\` and `x`).
- String interpolation is supported: - String interpolation is supported:
- identifier form: `"$name"` - identifier form: `"$name"` or `` `$name` ``
- expression form: `"${expr}"` - expression form: `"${expr}"` or `` `${expr}` ``
- escaped dollar: `"\$"` and `"$$"` both produce literal `$`. - escaped dollar: `"\$"`, `"$$"`, `` `\$` ``, and `` `$$` `` all produce literal `$`.
- `\\$x` means backslash + interpolated `x`. - `\\$x` means backslash + interpolated `x` in either delimiter form.
- Per-file opt-out is supported via leading comment directive: - Per-file opt-out is supported via leading comment directive:
- `// feature: interpolation: off` - `// feature: interpolation: off`
- with this directive, `$...` stays literal text. - with this directive, `$...` stays literal text.
@@ -41,6 +43,8 @@ Primary sources used: `lynglib/src/commonMain/kotlin/net/sergeych/lyng/{Parser,T
## 3. Literals and Core Expressions ## 3. Literals and Core Expressions
- Scalars: `null`, `true`, `false`, `void`. - Scalars: `null`, `true`, `false`, `void`.
- List literals allow multiline layout, comments between entries and trailing commas.
Newlines before the closing `]` are whitespace, including in nested and empty lists.
- List literal: `[a, b, c]`, spreads with `...`. - List literal: `[a, b, c]`, spreads with `...`.
- Spread positions: beginning, middle, end are all valid: `[...a]`, `[0, ...a, 4]`, `[head, ...mid, tail]`. - Spread positions: beginning, middle, end are all valid: `[...a]`, `[0, ...a, 4]`, `[head, ...mid, tail]`.
- Spread source must be a `List` at runtime (non-list spread raises an error). - Spread source must be a `List` at runtime (non-list spread raises an error).
@@ -63,6 +67,8 @@ Primary sources used: `lynglib/src/commonMain/kotlin/net/sergeych/lyng/{Parser,T
- Declaration-side variadic parameters use ellipsis suffix: - Declaration-side variadic parameters use ellipsis suffix:
- functions: `fun f(head, tail...) { ... }` - functions: `fun f(head, tail...) { ... }`
- lambdas: `{ x, rest... -> ... }` - lambdas: `{ x, rest... -> ... }`
- a typed parameter `values: T...` has signature type `T...`, but inside the callable `values`
is a `List<T>` containing the collected arguments.
- Call-side splats use `...expr` and are expanded by argument kind: - Call-side splats use `...expr` and are expanded by argument kind:
- positional splat: `f(...[1,2,3])` - positional splat: `f(...[1,2,3])`
- named splat: `f(...{ a: 1, b: 2 })` (map-style) - named splat: `f(...{ a: 1, b: 2 })` (map-style)
@@ -81,6 +87,7 @@ Primary sources used: `lynglib/src/commonMain/kotlin/net/sergeych/lyng/{Parser,T
## 4. Operators (implemented) ## 4. Operators (implemented)
- Assignment: `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `?=`. - Assignment: `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `?=`.
- Logical: `||`, `&&`, unary `!`. - Logical: `||`, `&&`, unary `!`.
- Unary arithmetic/bitwise: unary `+`, unary `-`, `~`.
- Bitwise: `|`, `^`, `&`, `~`, shifts `<<`, `>>`. - Bitwise: `|`, `^`, `&`, `~`, shifts `<<`, `>>`.
- Equality/comparison: `==`, `!=`, `===`, `!==`, `<`, `<=`, `>`, `>=`, `<=>`, `=~`, `!~`. - Equality/comparison: `==`, `!=`, `===`, `!==`, `<`, `<=`, `>`, `>=`, `<=>`, `=~`, `!~`.
- Type/containment: `is`, `!is`, `in`, `!in`, `as`, `as?`. - Type/containment: `is`, `!is`, `in`, `!in`, `as`, `as?`.
@@ -117,7 +124,9 @@ Primary sources used: `lynglib/src/commonMain/kotlin/net/sergeych/lyng/{Parser,T
- shorthand: `fun f(x) = expr`. - shorthand: `fun f(x) = expr`.
- generics: `fun f<T>(x: T): T`. - generics: `fun f<T>(x: T): T`.
- extension functions: `fun Type.name(...) { ... }`. - extension functions: `fun Type.name(...) { ... }`.
- context-aware extension functions: `context(Tag) fun String.unaryPlus() { this@Tag.addText(this) }`.
- named singleton `object` declarations can be extension receivers too: `fun Config.describe(...) { ... }`, `val Config.tag get() = ...`. - named singleton `object` declarations can be extension receivers too: `fun Config.describe(...) { ... }`, `val Config.tag get() = ...`.
- Static class/object methods preserve declared or inferable return types at call sites, including chained member access and generic return types inferred from arguments. For example, `static fun of<T>(v: T): Box<T> = Box(v)` lets `Box.of("abc").value.length` retain the `String` member type.
- static extension functions are callable on the type object: `static fun List<T>.fill(...)` -> `List.fill(...)`. - static extension functions are callable on the type object: `static fun List<T>.fill(...)` -> `List.fill(...)`.
- delegated callable: `fun f(...) by delegate`. - delegated callable: `fun f(...) by delegate`.
- Type aliases: - Type aliases:
@@ -125,6 +134,8 @@ Primary sources used: `lynglib/src/commonMain/kotlin/net/sergeych/lyng/{Parser,T
- generic: `type Box<T> = List<T>` - generic: `type Box<T> = List<T>`
- aliases are expanded structurally. - aliases are expanded structurally.
- Classes/objects/enums/interfaces: - Classes/objects/enums/interfaces:
- Constructors use the class name. A direct member function (including `static`) cannot have the same name as its enclosing class; local functions may still shadow names normally.
- `this(...)` and `this@Type(...)` are not secondary-constructor delegation syntax; call the class by name to construct another instance.
- `interface` is parsed as abstract class synonym. - `interface` is parsed as abstract class synonym.
- `object` supports named singleton and anonymous object expression forms. - `object` supports named singleton and anonymous object expression forms.
- enums support lifted entries: `enum E* { A, B }`. - enums support lifted entries: `enum E* { A, B }`.
@@ -162,7 +173,12 @@ Primary sources used: `lynglib/src/commonMain/kotlin/net/sergeych/lyng/{Parser,T
- unions `A | B` - unions `A | B`
- intersections `A & B` - intersections `A & B`
- function types `(A, B)->R` and receiver form `Receiver.(A)->R` - function types `(A, B)->R` and receiver form `Receiver.(A)->R`
- receiver-stack function types via `context(A, B) Receiver.(P)->R`
- variadics in function type via ellipsis (`T...`) - variadics in function type via ellipsis (`T...`)
- `A & B` means one value implementing both types.
- `context(A, B) Receiver.(P)->R` is different: it declares an ordered implicit-receiver stack where `Receiver` is primary `this`, then `A`, then `B`.
- Nested receiver lambdas keep outer receivers in scope; unqualified lookup prefers the innermost receiver, and `this@Type` can select an outer/context receiver explicitly.
- If the primary receiver does not provide a member and multiple outer/context receivers do, the lookup is a compile-time ambiguity and must be disambiguated with `this@Type`.
- Generics: - Generics:
- type params on classes/functions/type aliases - type params on classes/functions/type aliases
- bounds via `:` with union/intersection expressions - bounds via `:` with union/intersection expressions
@@ -215,6 +231,7 @@ Primary sources used: `lynglib/src/commonMain/kotlin/net/sergeych/lyng/{Parser,T
- Disambiguation helpers are supported: - Disambiguation helpers are supported:
- qualified this: `this@Base.member()` - qualified this: `this@Base.member()`
- cast view: `(obj as Base).member()` - cast view: `(obj as Base).member()`
- In nested receiver lambdas, `this@Type` can target any receiver visible through the receiver stack, not just inheritance ancestors.
- On unknown receiver types, compiler allows only Object-safe members: - On unknown receiver types, compiler allows only Object-safe members:
- `toString`, `toInspectString`, `let`, `also`, `apply`, `run` - `toString`, `toInspectString`, `let`, `also`, `apply`, `run`
- Other members require known receiver type or explicit cast. - Other members require known receiver type or explicit cast.
+14
View File
@@ -0,0 +1,14 @@
# AI notes: publish JVM CLI updates with `bin/local_jrelease`
[//]: # (excludeFromIndex)
When a change affects the JVM CLI launcher used as `jlyng`, refresh the installed local distribution with:
```bash
bin/local_jrelease
```
Why:
- `jlyng` in this repo is installed from `~/bin/jlyng-jvm/lyng-jvm`, not directly from `lyng/build/install`.
- Manual copying from Gradle build output can leave the actual launcher on `PATH` stale.
- `bin/local_jrelease` rebuilds `lyng/build/distributions/lyng-jvm.zip`, reinstalls it under `~/bin/jlyng-jvm`, and recreates the `~/bin/jlyng` symlink.
+10
View File
@@ -0,0 +1,10 @@
# AI notes: heading levels must be consecutive
[//]: # (excludeFromIndex)
When editing repository documentation:
- Use heading levels in order: `#`, then `##`, then `###`, and so on.
- Do not skip levels, for example `#` directly to `###`.
- Keep the heading tree balanced inside each document; sibling sections should use the same level.
- If you add a subsection and the parent is `##`, the child must be `###`.
+13 -1
View File
@@ -18,6 +18,7 @@ Sources: `lynglib/src/commonMain/kotlin/net/sergeych/lyng/Script.kt`, `lynglib/s
- Async/concurrency: `launch`, `yield`, `flow`, `delay`. - Async/concurrency: `launch`, `yield`, `flow`, `delay`.
- `Deferred.cancel()` cancels an active task. - `Deferred.cancel()` cancels an active task.
- `Deferred.await()` throws `CancellationException` if that task was cancelled. - `Deferred.await()` throws `CancellationException` if that task was cancelled.
- `Iterable<Deferred>.joinAll()` awaits every deferred in iteration order and returns a `List` of results.
- Math: `floor`, `ceil`, `round`, `sin`, `cos`, `tan`, `asin`, `acos`, `atan`, `sinh`, `cosh`, `tanh`, `asinh`, `acosh`, `atanh`, `exp`, `ln`, `log10`, `log2`, `pow`, `sqrt`, `abs`, `clamp`. - Math: `floor`, `ceil`, `round`, `sin`, `cos`, `tan`, `asin`, `acos`, `atan`, `sinh`, `cosh`, `tanh`, `asinh`, `acosh`, `atanh`, `exp`, `ln`, `log10`, `log2`, `pow`, `sqrt`, `abs`, `clamp`.
- These helpers also accept `lyng.decimal.Decimal`. - These helpers also accept `lyng.decimal.Decimal`.
- Exact Decimal path today: `abs`, `floor`, `ceil`, `round`, and `pow` with integral exponent. - Exact Decimal path today: `abs`, `floor`, `ceil`, `round`, and `pow` with integral exponent.
@@ -81,16 +82,27 @@ Sources: `lynglib/src/commonMain/kotlin/net/sergeych/lyng/Script.kt`, `lynglib/s
- `import lyng.serialization` - `import lyng.serialization`
- `Lynon` serialization utilities. - `Lynon` serialization utilities.
- `import lyng.time` - `import lyng.time`
- `Instant`, `DateTime`, `Duration`, and module `delay`. - `Instant`, `Date`, `DateTime`, `Duration`, and module `delay`.
## 6. Optional (lyngio) Modules ## 6. Optional (lyngio) Modules
Requires installing `lyngio` into the import manager from host code. Requires installing `lyngio` into the import manager from host code.
- `import lyng.io.fs` (filesystem `Path` API) - `import lyng.io.fs` (filesystem `Path` API)
- `import lyng.io.process` (process execution API) - `import lyng.io.process` (process execution API)
- Shell-script sugar: `sh(command): CommandRun` and `exec(executable, args=[]): CommandRun`.
- Prefer `sh("git status --short").out` for small shell output, `sh("...").lines` for large stdout streams, and `.check()` for commands that must exit with code 0.
- Prefer `exec("git", ["add", file])` when arguments come from data, filenames, or user input; it bypasses shell parsing.
- `CommandRun` is active and owns process pipes; choose one consumption path per stream (`out` or `lines`, `err` or `errorLines`).
- `import lyng.io.console` (console capabilities, geometry, ANSI/output, events) - `import lyng.io.console` (console capabilities, geometry, ANSI/output, events)
- `import lyng.io.http` (HTTP/HTTPS client API) - `import lyng.io.http` (HTTP/HTTPS client API)
- `import lyng.io.http.server` (minimal HTTP/1.1 and WebSocket server API)
- `import lyng.io.ws` (WebSocket client API; currently supported on JVM, capability-gated elsewhere) - `import lyng.io.ws` (WebSocket client API; currently supported on JVM, capability-gated elsewhere)
- `import lyng.io.net` (TCP/UDP transport API; currently supported on JVM, capability-gated elsewhere) - `import lyng.io.net` (TCP/UDP transport API; currently supported on JVM, capability-gated elsewhere)
- `import lyng.io.html` (pure Lyng HTML builder DSL: `html { body { h3 { +"text" } } }`)
- `import lyng.io.html.parse` (offline HTML5 parsing: `Html.parse(source, baseUrl = "")`, document title/text, CSS selectors, element text/attributes/absolute URLs; see [HTML parsing](lyng.io.html.parse.md))
- Shared network value-type packages are also available when installed by host code:
- `import lyng.io.http.types` (`HttpHeaders`)
- `import lyng.io.ws.types` (`WsMessage`)
- `import lyng.io.net.types` (`IpVersion`, `SocketAddress`, `Datagram`)
## 7. AI Generation Tips ## 7. AI Generation Tips
- Assume `lyng.stdlib` APIs exist in regular script contexts. - Assume `lyng.stdlib` APIs exist in regular script contexts.
+12
View File
@@ -41,6 +41,18 @@ Ellipsis can also appear in **function types** to denote a variadic position:
Ellipsis argument receives what is left from arguments after processing regular one that could be before or after. Ellipsis argument receives what is left from arguments after processing regular one that could be before or after.
For a typed declaration, the ellipsis type describes each argument while the name inside the
function is a list of those arguments. Thus `values: Int...` is `Int...` in the function signature
and `List<Int>` in the function body:
```lyng
fun sum(values: Int...): Int {
var result = 0
for (value in values) result += value
return result
}
```
Ellipsis could be a first argument: Ellipsis could be a first argument:
fun testCountArgs(data...,size) { fun testCountArgs(data...,size) {
+73
View File
@@ -107,6 +107,79 @@ val run2 = script.execute(scope)
`Scope.eval("...")` is the low-level shortcut that compiles and executes on the given scope. `Scope.eval("...")` is the low-level shortcut that compiles and executes on the given scope.
For most embedding use cases, prefer `session.eval("...")`. For most embedding use cases, prefer `session.eval("...")`.
### 2.2) Inspect function signatures without executing source
Declaration generators and other host tools can ask the compiler for semantic metadata about
top-level functions without running the script:
```kotlin
val source = Source(
"service.lyng",
"""
@Export
fun greet<T: Object = String>(
first: T,
rest: T...,
punctuation: String = "!",
): String = first.toString() + punctuation
""".trimIndent()
)
val functions = Compiler.resolveFunctionMetadata(
source,
Script.defaultImportManager,
)
val greet = functions.single()
println(greet.name) // greet
println(greet.annotations) // [Export]
println(greet.parameters[1].isEllipsis) // true
println(greet.parameters[2].defaultSource) // "!"
println(greet.returnType) // String
```
`resolveFunctionMetadata(...)` parses the source and resolves its imports and types, but it does
not execute the script, annotation functions, or default expressions. It returns
`ResolvedFunctionMetadata` entries in source order. Each entry provides:
- the function name, visibility, annotations, and source positions;
- generic parameters with their variance, bounds, and default types;
- parameters, variadic markers, and the original text of default expressions when available;
- the complete function type, including receivers, context receivers, and the declared or inferred
return type.
For a variadic declaration such as `values: Int...`, `ResolvedFunctionParameter.type` is the
element type `Int`, `isEllipsis` is `true`, and the parameter in `functionType` is `Int...`.
Inside Lyng function code, the collected `values` variable has type `List<Int>`.
If you already compiled the source, obtain the same information without resolving it again:
```kotlin
val script = Compiler.compile(source, Script.defaultImportManager)
val functions = script.resolvedFunctionMetadata()
```
Only top-level function declarations are returned. Use the Mini-AST APIs when tooling also needs
nested declarations or a syntax-oriented representation.
### 2.3) Prepare imports separately from execution
Low-level hosts that own a scope can install a compiled script's imports without executing its
statements:
```kotlin
val script = Compiler.compile(source, importManager)
val scope = importManager.newModule()
script.importInto(scope)
val result = script.execute(scope)
```
`importInto(...)` is idempotent for a scope and is useful when import preparation and execution
belong to different host lifecycle phases. To create a fresh raw module with the script's imports
already installed, use `script.instantiateModule(seedScope)`. When supplied, `seedScope` provides
the import provider and any seed-bound imports used by the new module.
### 3) Preferred: bind extern globals from Kotlin ### 3) Preferred: bind extern globals from Kotlin
For module-level APIs, the default workflow is: For module-level APIs, the default workflow is:
+1 -1
View File
@@ -27,6 +27,6 @@ See `docs/lyng_d_files.md` for `.lyng.d` syntax and examples.
- Alternatively, if/when the plugin is published to a marketplace, you will be able to install it - Alternatively, if/when the plugin is published to a marketplace, you will be able to install it
directly from the “Marketplace” tab (not yet available). directly from the “Marketplace” tab (not yet available).
### [Download plugin v0.0.2-SNAPSHOT](https://lynglang.com/distributables/lyng-idea-0.0.2-SNAPSHOT.zip) ### [Download plugin v0.0.5-SNAPSHOT](https://lynglang.com/distributables/lyng-idea-0.0.5-SNAPSHOT.zip)
Your ideas and bugreports are welcome on the [project gitea page](https://gitea.sergeych.net/SergeychWorks/lyng/issues) Your ideas and bugreports are welcome on the [project gitea page](https://gitea.sergeych.net/SergeychWorks/lyng/issues)
+167 -23
View File
@@ -1,9 +1,32 @@
# Json support # Json support
Since 1.0.5 we start adding JSON support. Versions 1,0,6* support serialization of the basic types, including lists and Lyng now has two distinct JSON-facing layers:
maps, and simple classes. Multiple inheritance may produce incorrect results, it is work in progress.
## Serialization in Lyng - plain JSON projection:
- `Obj.toJson()`
- `Obj.toJsonString()`
- canonical JSON round-trip format:
- `Json.encode(value)`
- `Json.decode(text)`
- typed canonical JSON round-trip format:
- `Json.encodeAs(Type, value)`
- `Json.decodeAs(Type, text)`
Use the first when you need ordinary JSON for interop.
Use the second when you need Lyng value round-trip semantics through JSON text with no schema.
Use the third when both sides already know the Lyng type and you want the same round-trip semantics with fewer type
tags in the JSON.
This distinction is intentional:
- plain JSON projection is optimized for compatibility with ordinary JSON tooling
- canonical `Json.encode()` is optimized for semantic fidelity to Lyng and Lynon and stays self-describing
- typed canonical `Json.encodeAs()` is optimized for the same fidelity when the schema is provided externally
- these goals conflict for values such as sets, exceptions, singleton objects, buffers, and maps with non-string keys
## Plain JSON projection in Lyng
// in lyng // in lyng
assertEquals("{\"a\":1}", {a: 1}.toJsonString()) assertEquals("{\"a\":1}", {a: 1}.toJsonString())
@@ -20,7 +43,8 @@ Simple classes serialization is supported:
assertEquals( "{\"foo\":1,\"bar\":2}", Point(1,2).toJsonString() ) assertEquals( "{\"foo\":1,\"bar\":2}", Point(1,2).toJsonString() )
>>> void >>> void
Note that mutable members are serialized by default. You can exclude any member (including constructor parameters) from JSON serialization using the `@Transient` attribute: Note that mutable members are serialized by default. You can exclude any member (including constructor parameters) from
JSON serialization using the `@Transient` attribute:
import lyng.serialization import lyng.serialization
@@ -31,7 +55,7 @@ Note that mutable members are serialized by default. You can exclude any member
assertEquals( "{\"bar\":2,\"visible\":100}", Point2(1,2).toJsonString() ) assertEquals( "{\"bar\":2,\"visible\":100}", Point2(1,2).toJsonString() )
>>> void >>> void
Note that if you override json serialization: Note that if you override plain JSON serialization:
import lyng.serialization import lyng.serialization
@@ -46,8 +70,8 @@ Note that if you override json serialization:
assertEquals( "{\"custom\":true}", Point2(1,2).toJsonString() ) assertEquals( "{\"custom\":true}", Point2(1,2).toJsonString() )
>>> void >>> void
Custom serialization of user classes is possible by overriding `toJsonObject` method. It must return an object which is Custom serialization of user classes is possible by overriding `toJsonObject`. It must return an object which is
serializable to Json. Most often it is a map, but any object is accepted, that makes it very flexible: serializable to JSON. Most often it is a map, but any object is accepted:
import lyng.serialization import lyng.serialization
@@ -70,12 +94,87 @@ serializable to Json. Most often it is a map, but any object is accepted, that m
Please note that `toJsonString` should be used to get serialized string representation of the object. Don't call Please note that `toJsonString` should be used to get serialized string representation of the object. Don't call
`toJsonObject` directly, it is not intended to be used outside the serialization library. `toJsonObject` directly, it is not intended to be used outside the serialization library.
## Canonical Json round-trip format
`Json.encode()` and `Json.decode()` are now the JSON equivalents of `Lynon.encode()` and `Lynon.decode()`.
They still use JSON text, but they add Lyng-specific type tags where plain JSON would otherwise lose information.
When a map already fits ordinary JSON object rules, canonical JSON keeps that traditional object shape. In particular,
maps with string keys are still serialized as JSON objects, not as tagged entry lists.
Example:
```lyng
import lyng.serialization
import lyng.time
enum Color { Red, Green }
class Point(x,y) { var z = 42 }
val p = Point(1,2)
p.z = 99
val value = List(
p,
Map([1, "one"], ["two", 2]),
Set(1,2,3),
"hello".encodeUtf8(),
Date(2026,4,15),
Color.Green
)
assertEquals(value, Json.decode(Json.encode(value)))
```
The canonical `Json` format is intended for Lyng-to-Lyng transfer through JSON text.
The plain `toJson()` projection is intended for ordinary JSON interop.
Canonical `Json.encode()` should be read as the JSON analogue of `Lynon.encode()`: when Lynon already preserves a
Lyng distinction, canonical JSON tries to preserve it too, using tags only where ordinary JSON is insufficient.
## Typed canonical Json round-trip format
`Json.encodeAs(Type, value)` and `Json.decodeAs(Type, text)` use the same canonical rules, but with a declared target
type available during the whole traversal.
This changes one thing only: type tags may be omitted when the declared type is already exact enough to restore the
value unambiguously.
The same map rule still applies here: `Map<String, T>` stays a normal JSON object, while non-string-key maps fall back
to canonical entry encoding.
Example:
```lyng
import lyng.serialization
closed class Point(x: Int, y: Int)
closed class Segment(a: Point, b: Point)
val value = Segment(Point(0, 1), Point(2, 3))
val encoded = Json.encodeAs(Segment, value)
assertEquals("{\"a\":{\"x\":0,\"y\":1},\"b\":{\"x\":2,\"y\":3}}", encoded)
assertEquals(value, Json.decodeAs(Segment, encoded))
```
Subtype information is still preserved when the declared type is wider than the runtime one. For example, if a field is
declared as `Base` but contains `Derived`, canonical subtype tags remain in that field.
This is why the APIs are split:
- `toJson()` stays plain and interop-friendly
- `Json.encode()` stays fully self-describing and safe to decode without a schema
- `Json.encodeAs()` uses the supplied schema to reduce noise, but only where that schema is sufficient
## Kotlin side interfaces ## Kotlin side interfaces
The "Batteries included" principle is also applied to serialization. The "Batteries included" principle is also applied to serialization.
- `Obj.toJson()` provides Kotlin `JsonElement` - `Obj.toJson()` provides Kotlin `JsonElement` for the plain JSON projection
- `Obj.toJsonString()` provides Json string representation - `Obj.toJsonString()` provides plain JSON string representation
- `Obj.decodeSerializableWith()` and `Obj.decodeSerializable()` allows to decode Lyng classes as Kotlin objects using - `Obj.decodeSerializableWith()` and `Obj.decodeSerializable()` allows to decode Lyng classes as Kotlin objects using
`kotlinx.serialization`: `kotlinx.serialization`:
@@ -104,10 +203,9 @@ suspend inline fun <reified T> Obj.decodeSerializable(scope: Scope = Scope()) =
decodeSerializableWith<T>(serializer<T>(), scope) decodeSerializableWith<T>(serializer<T>(), scope)
``` ```
Note that lyng-2-kotlin deserialization with `kotlinx.serialization` uses JsonElement as information carrier without Note that Lyng-to-Kotlin deserialization with `kotlinx.serialization` is based on the plain JSON projection,
formatting and parsing actual Json strings. This is why we use `Json.decodeFromJsonElement` instead of not the canonical `Json.encode()` format. It uses `JsonElement` as the information carrier without formatting and
`Json.decodeFromString`. Such an approach gives satisfactory performance without writing and supporting custom parsing actual JSON strings. This is why we use `Json.decodeFromJsonElement` instead of `Json.decodeFromString`.
`kotlinx.serialization` codecs.
### Pitfall: JSON objects and Map<String, Any?> ### Pitfall: JSON objects and Map<String, Any?>
@@ -134,7 +232,8 @@ fun deserializeMapWithJsonTest() = runTest {
But what if your map has objects of different types? The approach of using polymorphism is partially applicable, but what to do with `{ one: 1, two: "two" }`? But what if your map has objects of different types? The approach of using polymorphism is partially applicable, but what to do with `{ one: 1, two: "two" }`?
The answer is pretty simple: use `JsonObject` in your deserializable object. This class is capable of holding any JSON types and structures and is sort of a silver bullet for such cases: The answer is simple: use `JsonObject` in your deserializable object. This class is capable of holding any JSON types
and structures:
~~~kotlin ~~~kotlin
@Serializable @Serializable
@@ -154,26 +253,71 @@ fun deserializeAnyMapWithJsonTest() = runTest {
~~~ ~~~
# List of supported types ## Supported shapes
### Plain JSON projection
| Lyng type | JSON type | notes | | Lyng type | JSON type | notes |
|-----------|-----------|-------------| |-----------|-----------|-------------|
| `Int` | number | | | `Int` | number | |
| `Real` | number | | | `Real` | number | finite values only as plain numbers |
| `String` | string | | | `String` | string | |
| `Bool` | boolean | | | `Bool` | boolean | |
| `null` | null | | | `null` | null | |
| `Instant` | string | ISO8601 (1) | | `Instant` | string | ISO8601 (1) |
| `List` | array | (2) | | `List` | array | (2) |
| `Map` | object | (2) | | `Map` | object | string keys only |
| simple class instance | object | constructor fields + mutable vars |
| enum | string | entry name |
### Canonical `Json.encode`
This format can also round-trip:
- maps with non-string keys
- sets
- immutable collections
- buffers and bit buffers
- class instances
- singleton objects
- enums
- exceptions
- `Date`, `Instant`, `DateTime`
- non-finite reals
- `void`
### Typed canonical `Json.encodeAs`
This format round-trips the same value space as canonical `Json.encode`, but it can emit simpler JSON for:
- closed classes and other exactly-known class fields
- enums when the enum type is known
- typed collections whose element types are known
- nested object graphs where declared field types are precise
It still falls back to canonical tagged encoding when exact runtime type information would otherwise be lost.
It does so by adding Lyng-specific type tags only when necessary.
## Kotlin-side extension point for more formats
Additional formats can be exported from Kotlin modules by subclassing `ObjSerializationFormatClass` and registering the
format in module scope with `bindSerializationFormat(...)`.
```kotlin
module.bindSerializationFormat(
object : ObjSerializationFormatClass("MyFormat") {
override suspend fun encodeValue(scope: Scope, value: Obj): Obj = ...
override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj = ...
}
)
```
This makes `MyFormat.encode(...)` and `MyFormat.decode(...)` available from Lyng after importing the module.
(1) (1)
: ISO8601 flavor 1970-05-06T06:00:00.000Z in used; number of fractional digits depends on the truncation : ISO8601 flavor `1970-05-06T06:00:00.000Z` is used; number of fractional digits depends on truncation on
on [Instant](time.md), see `Instant.truncateTo...` functions. `Instant`, see `Instant.truncateTo...` functions.
(2) (2)
: List may contain any objects serializable to Json. : Lists may contain any values serializable by the selected JSON layer.
(3)
: Map keys must be strings, map values may be any objects serializable to Json.
+565
View File
@@ -0,0 +1,565 @@
# lyng.io.db — SQL database access for Lyng scripts
This module provides the portable SQL database contract for Lyng. The current shipped providers are SQLite via `lyng.io.db.sqlite` and a JVM-only JDBC bridge via `lyng.io.db.jdbc`.
> **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes.
---
## Install the module into a Lyng session
For SQLite-backed database access, install both the generic DB module and the SQLite provider:
```kotlin
import net.sergeych.lyng.EvalSession
import net.sergeych.lyng.Scope
import net.sergeych.lyng.io.db.createDbModule
import net.sergeych.lyng.io.db.sqlite.createSqliteModule
suspend fun bootstrapDb() {
val session = EvalSession()
val scope: Scope = session.getScope()
createDbModule(scope)
createSqliteModule(scope)
session.eval("""
import lyng.io.db
import lyng.io.db.sqlite
""".trimIndent())
}
```
`createSqliteModule(...)` also registers the `sqlite:` scheme for generic `openDatabase(...)`.
For JVM JDBC-backed access, install the JDBC provider as well:
```kotlin
import net.sergeych.lyng.EvalSession
import net.sergeych.lyng.Scope
import net.sergeych.lyng.io.db.createDbModule
import net.sergeych.lyng.io.db.jdbc.createJdbcModule
suspend fun bootstrapJdbc() {
val session = EvalSession()
val scope: Scope = session.getScope()
createDbModule(scope)
createJdbcModule(scope)
session.eval("""
import lyng.io.db
import lyng.io.db.jdbc
""".trimIndent())
}
```
`createJdbcModule(...)` registers `jdbc:`, `h2:`, `postgres:`, and `postgresql:` for `openDatabase(...)`.
---
## Using from Lyng scripts
Typed SQLite open helper:
```lyng
import lyng.io.db.sqlite
val db = openSqlite(":memory:")
val userCount = db.transaction { tx ->
tx.execute("create table user(id integer primary key autoincrement, name text not null)")
tx.execute("insert into user(name) values(?)", "Ada")
tx.execute("insert into user(name) values(?)", "Linus")
tx.select("select count(*) as count from user").toList()[0]["count"]
}
assertEquals(2, userCount)
```
Generic provider-based open:
```lyng
import lyng.io.db
import lyng.io.db.sqlite
val db = openDatabase(
"sqlite:./app.db",
Map(
"foreignKeys" => true,
"busyTimeoutMillis" => 5000
)
)
```
JVM JDBC open with H2:
```lyng
import lyng.io.db.jdbc
val db = openH2("mem:demo;DB_CLOSE_DELAY=-1")
val names = db.transaction { tx ->
tx.execute("create table person(id bigint auto_increment primary key, name varchar(120) not null)")
tx.execute("insert into person(name) values(?)", "Ada")
tx.execute("insert into person(name) values(?)", "Linus")
tx.select("select name from person order by id").toList()
}
assertEquals("Ada", names[0]["name"])
assertEquals("Linus", names[1]["name"])
```
Generic JDBC open through `openDatabase(...)`:
```lyng
import lyng.io.db
import lyng.io.db.jdbc
val db = openDatabase(
"jdbc:h2:mem:demo2;DB_CLOSE_DELAY=-1",
Map()
)
val answer = db.transaction { tx ->
tx.select("select 42 as answer").toList()[0]["answer"]
}
assertEquals(42, answer)
```
PostgreSQL typed open:
```lyng
import lyng.io.db.jdbc
val db = openPostgres(
"jdbc:postgresql://127.0.0.1/appdb",
"appuser",
"secret"
)
val titles = db.transaction { tx ->
tx.execute("create table if not exists task(id bigserial primary key, title text not null)")
tx.execute("insert into task(title) values(?)", "Ship JDBC provider")
tx.execute("insert into task(title) values(?)", "Test PostgreSQL path")
tx.select("select title from task order by id").toList()
}
assertEquals("Ship JDBC provider", titles[0]["title"])
```
Nested transactions use real savepoint semantics:
```lyng
import lyng.io.db
import lyng.io.db.sqlite
val db = openSqlite(":memory:")
db.transaction { tx ->
tx.execute("create table item(id integer primary key autoincrement, name text not null)")
tx.execute("insert into item(name) values(?)", "outer")
try {
tx.transaction { inner ->
inner.execute("insert into item(name) values(?)", "inner")
throw IllegalStateException("rollback nested")
}
} catch (_: IllegalStateException) {
}
assertEquals(1, tx.select("select count(*) as count from item").toList()[0]["count"])
}
```
Intentional rollback without treating it as a backend failure:
```lyng
import lyng.io.db
import lyng.io.db.sqlite
val db = openSqlite(":memory:")
assertThrows(RollbackException) {
db.transaction { tx ->
tx.execute("create table item(id integer primary key autoincrement, name text not null)")
tx.execute("insert into item(name) values(?)", "temporary")
throw RollbackException("stop here")
}
}
```
---
## Runnable serialization sample
A complete runnable example is in [examples/sqlite_serialization.lyng](../examples/sqlite_serialization.lyng).
It uses:
- `@DbJson`
- `@DbLynon`
- `@DbExcept`
- `@cols(...)`, `@vals(...)`, `@set(...)`
- `decodeAs<T>()`
The current direct read form that works under `jlyng` is:
```lyng
tx.select("select * from item where id = ?", 1).decodeAs<Item>().first
```
If we want a shorter form such as:
```lyng
tx.selectAllAs<Item>("item where id = ?", 1).first
```
it should be added as a built-in `SqlTransaction` API. A pure Lyng generic wrapper around `decodeAs<T>()` does not currently preserve `T` reliably enough under `jlyng`.
---
## Portable API
### `Database`
- `transaction(block)` — opens a transaction, commits on normal exit, rolls back on uncaught failure.
### `SqlTransaction`
- `select(clause, params...)` — execute a statement whose primary result is a row set.
- `execute(clause, params...)` — execute a side-effect statement and return `ExecutionResult`.
- `transaction(block)` — nested transaction with real savepoint semantics.
`select(...)` and `execute(...)` also support SQL object-expansion macros for declaration-driven writes:
- `@cols(?1)` — expand object argument `?1` to a comma-separated column list
- `@vals(?1)` — expand object argument `?1` to matching placeholders and bind values
- `@set(?1)` — expand object argument `?1` to `column = ?` pairs and bind values
Each macro also supports an optional clause-local exclusion list:
```lyng
tx.execute("update item set @set(?1 except: \"id\", \"createdAt\") where id = ?2", item, item.id)
```
Example:
```lyng
tx.execute("insert into item(@cols(?1)) values(@vals(?1))", item)
tx.execute("update item set @set(?1) where id = ?2", item, item.id)
```
When a clause uses any of these macros, non-expanded scalar parameters in the same SQL string must use explicit indexed placeholders such as `?2`, `?3`, and so on.
### `ResultSet`
- `columns` — positional `SqlColumn` metadata, available before iteration.
- `size()` — result row count.
- `isEmpty()` — fast emptiness check where possible.
- `iterator()` — normal row iteration while the transaction is active.
- `toList()` — materialize detached `SqlRow` snapshots that may be used after the transaction ends.
- `decodeAs<T>()` — transaction-scoped iterable view that decodes each row into `T`.
### `SqlRow`
- `row[index]` — zero-based positional access.
- `row["columnName"]` — case-insensitive lookup by output column label.
- `row.decodeAs<T>()` — decode one row into a typed Lyng value.
Name-based access fails with `SqlUsageException` if the name is missing or ambiguous.
### `DbFieldAdapter`
Custom DB field projection hook used by `@DbDecodeWith(...)` and `@DbSerializeWith(...)`.
- `decode(rawValue, column, row, targetType)` — adapt one raw DB field value to a Lyng value for the requested target type.
- `encode(value, targetType)` — adapt one Lyng value to a direct DB-bindable value for SQL object expansion.
Use `@DbDecodeWith(adapter)` on class constructor parameters and class-body fields/properties that participate in `decodeAs<T>()`.
Use `@DbSerializeWith(adapter)` on constructor parameters and class-body fields/properties that participate in `@cols(...)`, `@vals(...)`, and `@set(...)` object expansion.
Annotation arguments are evaluated once when the declaration is created, and the resulting adapter instance is retained in declaration metadata.
### `ExecutionResult`
- `affectedRowsCount`
- `getGeneratedKeys()`
Statements that return rows directly, such as `... returning ...`, should use `select(...)`, not `execute(...)`.
---
## Value mapping
Portable bind values:
- `null`
- `Bool`
- `Int`, `Double`, `Decimal`
- `String`
- `Buffer`
- `Date`, `DateTime`, `Instant`
Unsupported parameter values fail with `SqlUsageException`.
SQL object-expansion write rules:
- constructor parameters participate in projection by declaration order
- matching serializable class-body fields/properties also participate
- `@Transient` fields are excluded automatically
- `@DbExcept` fields are excluded automatically
- `except:` excludes additional fields for one specific macro use
- direct DB-bindable values are written as-is
- `@DbJson` fields are encoded as canonical JSON text
- `@DbLynon` fields are encoded as Lynon binary
- `@DbSerializeWith(adapter)` fields are encoded through the adapter
- unannotated non-bindable object fields fail with `SqlUsageException`
Write-side encoding is intentionally explicit. The runtime does not try to infer target DB column types from SQL text or backend metadata during statement preparation.
Example:
```lyng
import lyng.io.db
import lyng.io.db.sqlite
class Payload(name: String, count: Int)
object TrimAdapter: DbFieldAdapter {
override fun encode(value, targetType) =
when(value) {
null -> null
else -> value.toString().trim()
}
}
class Item(
id: Int,
@DbSerializeWith(TrimAdapter) title: String,
@DbJson meta: Payload,
@DbLynon state: Payload
) {
var note: String = ""
@DbExcept var cache: String = ""
}
val db = openSqlite(":memory:")
val restored = db.transaction { tx ->
tx.execute(
"create table item(id integer not null, title text not null, meta text not null, state blob not null, note text not null)"
)
val item = Item(1, " first ", Payload("json", 10), Payload("bin", 20))
item.note = "created"
item.cache = "not stored"
tx.execute("insert into item(@cols(?1)) values(@vals(?1))", item)
item.title = " second "
item.meta = Payload("json2", 11)
item.state = Payload("bin2", 21)
item.note = "updated"
tx.execute(
"update item set @set(?1 except: \"id\") where id = ?2",
item,
item.id
)
tx.select("select id, title, meta, state, note from item").decodeAs<Item>().first
}
assertEquals("second", restored.title)
assertEquals("json2", restored.meta.name)
assertEquals(21, restored.state.count)
assertEquals("updated", restored.note)
```
This example shows:
- `@DbSerializeWith(...)` trimming a string before write
- `@DbJson` storing structured data in a text column
- `@DbLynon` storing structured data in a binary column
- `@DbExcept` excluding a field from automatic projection
- `@set(... except: "id")` skipping one field for an update clause
- `decodeAs<Item>()` reconstructing the object on read
Portable result metadata categories:
- `Binary`
- `String`
- `Int`
- `Double`
- `Decimal`
- `Bool`
- `Date`
- `DateTime`
- `Instant`
Typed row decode rules:
- object/class targets map constructor parameters by column label, case-insensitively
- remaining matching serializable mutable fields are assigned after constructor call
- `@DbDecodeWith(adapter)` on a constructor parameter or class-body field/property takes precedence over built-in JSON/Lynon decoding
- `@DbDecodeWith(adapter)` must receive exactly one adapter instance implementing `DbFieldAdapter`
- adapter output must match the target member type or decoding fails with `SqlUsageException`
- missing required non-null constructor fields fail
- defaulted or nullable constructor fields may be omitted from the result
- extra result columns currently fail in strict mode
- if a row has exactly one column, that value may be decoded directly as the requested target type
- JSON-like native column types (`json`, `jsonb`) are decoded through typed canonical `Json` when the target type is not `String`
- binary columns are decoded through `Lynon` when the target type is not `Buffer`
- `Buffer` targets keep the raw binary payload without Lynon decoding
- plain text columns are not implicitly treated as JSON
For temporal types, see [time functions](time.md).
---
## SQLite provider
`lyng.io.db.sqlite` currently provides the first concrete backend.
Typed helper:
```lyng
openSqlite(
path: String,
readOnly: Bool = false,
createIfMissing: Bool = true,
foreignKeys: Bool = true,
busyTimeoutMillis: Int = 5000
): Database
```
Accepted generic URL forms:
- `sqlite::memory:`
- `sqlite:relative/path.db`
- `sqlite:/absolute/path.db`
Supported `openDatabase(..., extraParams)` keys for SQLite:
- `readOnly: Bool`
- `createIfMissing: Bool`
- `foreignKeys: Bool`
- `busyTimeoutMillis: Int`
SQLite write/read policy in v1:
- `Bool` writes as `0` / `1`
- `Decimal` writes as canonical text
- `Date` writes as `YYYY-MM-DD`
- `DateTime` writes as ISO local timestamp text without timezone
- `Instant` writes as ISO UTC timestamp text with explicit timezone marker
- `TIME*` values stay `String`
- `TIMESTAMP` / `DATETIME` reject timezone-bearing stored text
Open-time validation failures:
- malformed URL or bad option shape -> `IllegalArgumentException`
- runtime open failure -> `DatabaseException`
## JDBC provider
`lyng.io.db.jdbc` is currently implemented on the JVM target only. The `lyngio-jvm` artifact bundles and explicitly loads these JDBC drivers:
- SQLite
- H2
- PostgreSQL
Typed helpers:
```lyng
openJdbc(
connectionUrl: String,
user: String? = null,
password: String? = null,
driverClass: String? = null,
properties: Map<String, Object?>? = null
): Database
openH2(
connectionUrl: String,
user: String? = null,
password: String? = null,
properties: Map<String, Object?>? = null
): Database
openPostgres(
connectionUrl: String,
user: String? = null,
password: String? = null,
properties: Map<String, Object?>? = null
): Database
```
Accepted generic URL forms:
- `jdbc:h2:mem:test;DB_CLOSE_DELAY=-1`
- `h2:mem:test;DB_CLOSE_DELAY=-1`
- `jdbc:postgresql://localhost/app`
- `postgres://localhost/app`
- `postgresql://localhost/app`
Supported `openDatabase(..., extraParams)` keys for JDBC:
- `driverClass: String`
- `user: String`
- `password: String`
- `properties: Map<String, Object?>`
Behavior notes for the JDBC bridge:
- the portable `Database` / `SqlTransaction` API stays the same as for SQLite
- nested transactions use JDBC savepoints
- JDBC connection properties are built from `user`, `password`, and `properties`
- `properties` values are stringified before being passed to JDBC
- statements with row-returning clauses still must use `select(...)`, not `execute(...)`
Platform support for this provider:
- `lyng.io.db.jdbc` — JVM only
- `openH2(...)` — works out of the box with `lyngio-jvm`
- `openPostgres(...)` — driver included, but an actual PostgreSQL server is still required
PostgreSQL-specific notes:
- `openPostgres(...)` accepts either a full JDBC URL or shorthand forms such as `//localhost/app`
- local peer/trust setups may use an empty password string
- generated keys work with PostgreSQL `bigserial` / identity columns through `ExecutionResult.getGeneratedKeys()`
- for reproducible automated tests, prefer a disposable PostgreSQL instance such as Docker/Testcontainers instead of a long-lived shared server
---
## Lifetime rules
`ResultSet` is valid only while its owning transaction is active.
`SqlRow` values are detached snapshots once materialized, so this pattern is valid:
```lyng
val rows = db.transaction { tx ->
tx.select("select name from person order by id").toList()
}
assertEquals("Ada", rows[0]["name"])
```
This means:
- do not keep `ResultSet` objects after the transaction block returns
- materialize rows with `toList()` inside the transaction when they must outlive it
- the iterable returned by `decodeAs<T>()` is also transaction-scoped
- decoded objects produced while iterating `decodeAs<T>()` are detached ordinary Lyng values
The same rule applies to generated keys from `ExecutionResult.getGeneratedKeys()`: the `ResultSet` is transaction-scoped, but rows returned by `toList()` are detached.
---
## Platform support
- `lyng.io.db` — generic contract, available when host code installs it
- `lyng.io.db.sqlite` — implemented on JVM and Linux Native in the current release tree
- `lyng.io.db.jdbc` — implemented on JVM in the current release tree
For the broader I/O overview, see [lyngio overview](lyngio.md).
+166
View File
@@ -0,0 +1,166 @@
# lyng.io.html
For reading existing HTML, see [lyng.io.html.parse](lyng.io.html.parse.md).
`lyng.io.html` provides a pure Lyng HTML builder DSL. It uses Lyng context
receiver extensions, so text can be appended with `+"text"` inside tag blocks
without global builder state.
Host code installs the package from `lyngio` with `createHtmlModule(...)`:
```kotlin
val scope = Script.newScope()
createHtmlModule(scope.importManager)
```
Lyng code can then import it:
```lyng
import lyng.io.html
val page = html {
head {
title { +"Demo" }
}
body {
nav {
a(href: "/") { +"Home" }
}
h3 { +"Heading 3" }
p {
attr("data-id", 123)
+"Text is escaped: <safe>"
}
img(src: "/logo.png", alt: "Logo")
}
}
```
`html { ... }` returns a `String` beginning with `<!doctype html>`.
## Escaping
Text appended with unary `+` is HTML-escaped:
```lyng
html {
body {
p { +"Text & <more>" }
}
}
```
produces:
```html
<!doctype html><html><body><p>Text &amp; &lt;more&gt;</p></body></html>
```
Attribute values are escaped with HTML attribute rules:
```lyng
p {
attr("data-x", "\"quoted\" & <tag>")
+"content"
}
```
Use `raw(...)` only for trusted markup:
```lyng
div {
raw("<span>already escaped or trusted</span>")
}
```
## Tag Helpers
Current tag helpers cover common structural tags (`head`, `body`, `main`,
`section`, `article`, `header`, `footer`, `nav`, `div`, `span`, `p`), headings
(`h1` through `h6`), lists (`ul`, `ol`, `li`), and text/code tags (`strong`,
`em`, `code`, `pre`, `script`, `style`).
```lyng
body {
main {
section {
h2 { +"News" }
p { +"First item" }
}
}
}
```
Common void tags are also available: `meta`, `link`, `img`, `br`, and `input`.
```lyng
head {
meta { attr("charset", "utf-8") }
link {
attr("rel", "stylesheet")
attr("href", "/site.css")
}
}
```
## Attributes
Use `attr(name, value)` inside a tag block to set an escaped attribute value.
`id(...)` and `classes(...)` are small aliases:
```lyng
div {
id("root")
classes("app shell")
}
```
Use `flag(name)` for boolean attributes:
```lyng
input {
attr("type", "checkbox")
flag("checked")
}
```
## Convenience Helpers
Convenience helpers include `metaCharset()`, `stylesheet(href)`,
`a(href) { ... }`, `img(src, alt)`, and `input(type, name, value)`.
```lyng
head {
metaCharset()
stylesheet("/site.css")
}
body {
nav {
a(href: "/home") { +"Home" }
}
img(src: "/logo.png", alt: "Logo & mark")
input(type: "hidden", name: "token", value: "abc")
}
```
## Generic Elements
Use `tag(name) { ... }` and `voidTag(name) { ... }` for elements that do not
have dedicated helpers yet:
```lyng
body {
tag("custom-element") {
flag("hidden")
+"Secret"
}
voidTag("source") {
attr("srcset", "/image.webp")
attr("type", "image/webp")
}
}
```
These helpers are intentionally simple escape hatches. Prefer a dedicated helper
when one exists because it can encode safer defaults and clearer parameter names.
+41
View File
@@ -0,0 +1,41 @@
# lyng.io.html.parse
Offline HTML5 parsing backed by the core [Ksoup](https://github.com/fleeksoft/ksoup)
parser (no network extensions). Installed from `lyngio`:
```kotlin
createHtmlParseModule(scope.importManager)
```
```lyng
import lyng.io.html.parse
val doc = Html.parse("<title>News</title><main><p>Hello &amp; welcome</p><a href='/next'>Next</a></main>", "https://example.com/")
println(doc.title)
println(doc.text())
for (raw in doc.select("a[href]")) {
val link = raw as HtmlElement
println(link.text() + " => " + link.absUrl("href"))
}
```
`Html.parse(source, baseUrl = "")` accepts documents, fragments and malformed HTML.
It returns `HtmlDocument`, with `title: String`, `text(): String` and
`select(css): List<HtmlElement>`. Elements have `text()`, `attr(name)`,
`absUrl(name)` and `select(css)`. Missing selections return an empty list;
missing attributes or unresolvable absolute URLs return an empty string.
Invalid CSS selectors raise an error. `<base href>` participates in URL resolution.
Use the final fetched URL as the base when available.
Text extraction decodes entities, normalizes whitespace, preserves block and
`br` boundaries as newlines, and excludes `script`, `style`, `template`, `noscript`
and `[hidden]` subtrees. Selection retains the original DOM. This is readable
text extraction, not CSS rendering or article detection; navigation and cookie
banners may remain. It does not preserve preformatted whitespace.
Parsing never fetches resources or executes JavaScript. `absUrl` only resolves
URLs; it does not filter schemes. Fetch through the host's authorized HTTP API,
check status and content type, and choose bounded excerpts to return to an AI.
This API parses HTML only; XML parsing is separate future work.
Canonical declarations: `lyngio/stdlib/lyng/io/html_parse.lyng`.
+42 -11
View File
@@ -1,12 +1,14 @@
### lyng.io.http — HTTP/HTTPS client for Lyng scripts # lyng.io.http — HTTP/HTTPS client for Lyng scripts
This module provides a compact HTTP client API for Lyng scripts. It is implemented in `lyngio` and backed by Ktor on supported runtimes. This module provides a compact HTTP client API for Lyng scripts. It is implemented in `lyngio` and backed by Ktor on supported runtimes.
> **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes. > **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes.
>
> **Shared type note:** `HttpHeaders` is also available from `lyng.io.http.types` when host code wants the reusable value type without relying on the HTTP client module itself.
--- ---
#### Add the library to your project (Gradle) ## Add the library to your project (Gradle)
If you use this repository as a multi-module project, add a dependency on `:lyngio`: If you use this repository as a multi-module project, add a dependency on `:lyngio`:
@@ -20,7 +22,7 @@ For external projects, ensure you also use the Lyng Maven repository described i
--- ---
#### Install the module into a Lyng session ## Install the module into a Lyng session
The HTTP module is not installed automatically. Install it into the session scope and provide a policy. The HTTP module is not installed automatically. Install it into the session scope and provide a policy.
@@ -40,9 +42,24 @@ suspend fun bootstrapHttp() {
} }
``` ```
Redirects are followed transparently, up to 10 hops by default. Hosts can set a different
limit when installing the module:
```kotlin
createHttpModule(
PermitAllHttpAccessPolicy,
scope,
LyngHttpConfig(maxRedirects = 5),
)
```
Exceeding the limit raises an `HTTP_REDIRECT_LIMIT_EXCEEDED` error containing the configured
limit, initial URL, and last redirect target. HTTPS-to-HTTP redirects remain disabled, and the
`Authorization` header is removed when a redirect changes authority.
--- ---
#### Using from Lyng scripts ## Using from Lyng scripts
Simple GET: Simple GET:
@@ -84,9 +101,9 @@ HTTPS GET:
--- ---
#### API reference ## API reference
##### `Http` (static methods) ### `Http` (static methods)
- `isSupported(): Bool` — Whether HTTP client support is available on the current runtime. - `isSupported(): Bool` — Whether HTTP client support is available on the current runtime.
- `request(req: HttpRequest): HttpResponse` — Execute a request described by a mutable request object. - `request(req: HttpRequest): HttpResponse` — Execute a request described by a mutable request object.
@@ -99,7 +116,7 @@ For convenience methods, `headers...` accepts:
- `MapEntry`, e.g. `"Accept" => "text/plain"` - `MapEntry`, e.g. `"Accept" => "text/plain"`
- 2-item lists, e.g. `["Accept", "text/plain"]` - 2-item lists, e.g. `["Accept", "text/plain"]`
##### `HttpRequest` ### `HttpRequest`
- `method: String` - `method: String`
- `url: String` - `url: String`
@@ -110,7 +127,7 @@ For convenience methods, `headers...` accepts:
Only one of `bodyText` and `bodyBytes` should be set. Only one of `bodyText` and `bodyBytes` should be set.
##### `HttpResponse` ### `HttpResponse`
- `status: Int` - `status: Int`
- `statusText: String` - `statusText: String`
@@ -120,7 +137,7 @@ Only one of `bodyText` and `bodyBytes` should be set.
Response body decoding is cached inside the response object. Response body decoding is cached inside the response object.
##### `HttpHeaders` ### `HttpHeaders`
`HttpHeaders` behaves like `Map<String, String>` for the first value of each header name and additionally exposes: `HttpHeaders` behaves like `Map<String, String>` for the first value of each header name and additionally exposes:
@@ -132,7 +149,7 @@ Header lookup is case-insensitive.
--- ---
#### Security policy ## Security policy
The module uses `HttpAccessPolicy` to authorize requests before they are sent. The module uses `HttpAccessPolicy` to authorize requests before they are sent.
@@ -168,7 +185,7 @@ val allowLocalOnly = object : HttpAccessPolicy {
--- ---
#### Platform support ## Platform support
- **JVM:** supported - **JVM:** supported
- **Android:** supported via the Ktor CIO client backend - **Android:** supported via the Ktor CIO client backend
@@ -177,3 +194,17 @@ val allowLocalOnly = object : HttpAccessPolicy {
- **Windows native:** supported via the Ktor WinHttp client backend - **Windows native:** supported via the Ktor WinHttp client backend
- **Apple native:** supported via the Ktor Darwin client backend - **Apple native:** supported via the Ktor Darwin client backend
- **Other targets:** may report unsupported; use `Http.isSupported()` before relying on it - **Other targets:** may report unsupported; use `Http.isSupported()` before relying on it
## Request failure diagnostics
Transport exceptions report the request method and target, a category, the
underlying exception/cause chain, and recovery advice. Query strings and userinfo
are redacted from the diagnostic target. Categories are `HTTP_TLS_FAILED`,
`HTTP_TIMEOUT`, `HTTP_DNS_FAILED`, `HTTP_CONNECTION_FAILED` and
`HTTP_REQUEST_FAILED`. Permission denials are `HTTP_ACCESS_DENIED`; redirect
limit errors retain `HTTP_REDIRECT_LIMIT_EXCEEDED`.
`HTTP_TLS_FAILED` means the handshake failed before an HTTP response. Do not
retry the same URL unchanged; use another source. A remote TLS fatal alert is
not a Lyng syntax error. Non-2xx responses remain ordinary response objects, and
request cancellation propagates rather than becoming a generic HTTP error.
+446
View File
@@ -0,0 +1,446 @@
# `lyng.io.http.server` - Minimal HTTP/1.1 And WebSocket Server
This module provides a small server-side HTTP API for Lyng scripts. It is implemented in `lyngio` on top of the existing TCP layer and is intended for embedded tools, local services, test fixtures, and lightweight app backends.
It supports:
- HTTP/1.1 request parsing
- keep-alive
- exact-path routing
- regex routing
- path-template routing with named parameters
- websocket upgrade and server-side sessions
It does not aim to replace a full reverse proxy. Typical deployment is behind nginx, Caddy, or another frontend that handles TLS and public-facing edge concerns.
> **Security note:** this module uses the same `NetAccessPolicy` capability model as raw TCP sockets. If scripts are allowed to listen on TCP, they can host an HTTP server.
## Install The Module Into A Lyng Session
Kotlin bootstrap example:
```kotlin
import net.sergeych.lyng.EvalSession
import net.sergeych.lyng.Scope
import net.sergeych.lyng.io.http.server.createHttpServerModule
import net.sergeych.lyngio.net.security.PermitAllNetAccessPolicy
suspend fun bootstrapHttpServer() {
val session = EvalSession()
val scope: Scope = session.getScope()
createHttpServerModule(PermitAllNetAccessPolicy, scope)
session.eval("import lyng.io.http.server")
}
```
## RequestContext Sugar
Route handlers use `RequestContext` as the receiver, so inside handlers you normally write direct calls such as:
- `jsonBody<T>()`
- `respondJson(...)`
- `respondHtml { ... }`
- `respondText(...)`
- `setHeader(...)`
- `request.path`
- `routeParams["id"]`
This keeps ordinary HTTP endpoints compact and avoids passing an explicit request or exchange parameter through every route lambda.
## HTML Response Sugar
Use `respondHtml { ... }` to render an HTML document with the `lyng.io.html` DSL and send it as `text/html; charset=utf-8`.
```lyng
import lyng.io.http.server
import lyng.io.html
val server = HttpServer()
server.get("/") {
respondHtml {
head {
title { +"Lyng status" }
}
body {
h3 { +"Service is running" }
p { +"Path: ${request.path}" }
}
}
}
server.listen(8080, "127.0.0.1")
```
Pass `code:` when the route should return a non-200 status:
```lyng
server.get("/accepted") {
respondHtml(code: 202) {
body { h3 { +"Accepted" } }
}
}
```
## JSON API Sugar
For ordinary JSON APIs, `RequestContext` includes two primary helpers:
- `jsonBody<T>()` decodes the request body with typed `Json.decodeAs(...)`
- `respondJson(body, status = 200)` sets JSON content type and responds with plain `toJsonString()`
These helpers intentionally use ordinary JSON projection for HTTP interop, not canonical `Json.encode(...)`.
### Typed JSON POST With Route Params
```lyng
import lyng.io.http.server
closed class CreateResultRequest(title: String, score: Int)
closed class CreateResultResponse(id: String, userId: String, title: String, score: Int)
val server = HttpServer()
server.postPath("/api/users/{userId}/results") {
val req = jsonBody<CreateResultRequest>()
if (req.title.isBlank()) {
respondJson({ error: "title must not be empty" }, 400)
return
}
respondJson(
CreateResultResponse("r-101", routeParams["userId"], req.title, req.score),
201
)
}
server.listen(8080, "127.0.0.1")
```
### JSON Response With Route Params
```lyng
import lyng.io.http.server
val server = HttpServer()
server.getPath("/api/users/{id}") {
respondJson({
id: routeParams["id"],
path: request.path,
ok: true
})
}
server.listen(8080, "127.0.0.1")
```
## Request And Route Data
`ServerRequest` exposes parsed HTTP request data:
- `method: String`
- `target: String`
- `path: String`
- `pathParts: List<String>`
- `queryString: String?`
- `query: Map<String, String>`
- `headers: HttpHeaders`
- `body: Buffer`
`RequestContext` exposes routing context and response controls:
- `request: ServerRequest`
- `routeMatch: RegexMatch?`
- `routeParams: Map<String, String>`
- `jsonBody<T>()`
- `respond(...)`
- `respondText(...)`
- `respondJson(body, status = 200)`
- `respondHtml(code: 200) { ... }`
- `setHeader(...)`
- `addHeader(...)`
- `acceptWebSocket(...)`
For exact routes, `routeMatch` is `null` and `routeParams` is empty.
For regex routes, `routeMatch` is set and `routeParams` is empty.
For path-template routes, both `routeMatch` and `routeParams` are set.
## Reusable Routers
`Router` collects the same route kinds as `HttpServer`, but does not listen on sockets by itself.
Mount it into `HttpServer` or another `Router`.
```lyng
import lyng.io.http.server
val api = Router()
api.get("/health") {
respondText(200, "ok")
}
val users = Router()
users.getPath("/users/{id}") {
respondJson({ id: routeParams["id"] })
}
api.mount(users)
val server = HttpServer()
server.mount(api)
server.listen(8080, "127.0.0.1")
```
Mounted routers reuse the built-in server router. They are configuration-time composition, not an extra per-request Lyng dispatch layer.
## WebSocket Routes
You can route websocket upgrades by exact path, regex, or path template.
```lyng
server.ws("/chat") { ws ->
ws.sendText("hello")
ws.close()
}
server.wsPath("/ws/{room}") { ws ->
ws.sendText("room=" + routeParams["room"])
ws.close()
}
```
A websocket handler runs only for requests that actually ask for websocket upgrade. Ordinary HTTP requests to the same path are not treated as websocket sessions.
### Choosing Between `ws(...)` And `acceptWebSocket(...)`
Use `server.ws(...)` or `server.wsPath(...)` when the route is always a websocket endpoint.
Use `acceptWebSocket(...)` inside a normal HTTP handler when the same route may inspect the request first and then decide whether to upgrade.
```lyng
server.get("/maybe-upgrade") {
if (!request.isWebSocketUpgrade()) {
respondText(400, "websocket upgrade required")
return
}
acceptWebSocket { ws ->
ws.sendText("connected")
ws.close()
}
}
```
### Reading Incoming Messages
Inside a websocket handler, call `ws.receive()` to wait for the next application message.
What `receive()` returns:
- `WsMessage` for the next text or binary message.
- `null` after the client sends a close frame.
- `null` after the socket is already closed and no more frames can arrive.
What reaches Lyng code:
- Text frames become `WsMessage(isText = true, text = ...)`.
- Binary frames become `WsMessage(isText = false, data = ...)`.
- Fragmented websocket messages are reassembled before they are returned.
- Ping and pong control frames are handled internally and do not appear in Lyng.
- A client close frame is answered by the server close handshake, then `receive()` returns `null`.
Typical server receive loop:
```lyng
import lyng.buffer
server.ws("/echo") { ws ->
while (true) {
val msg = ws.receive() ?: break
if (msg.isText) {
ws.sendText("echo:" + msg.text)
} else {
ws.sendBytes(msg.data as Buffer)
}
}
}
```
### Sending Outgoing Messages
Use:
- `ws.sendText(text)` for text messages.
- `ws.sendBytes(data)` for binary messages.
Example:
```lyng
import lyng.buffer
server.ws("/push") { ws ->
ws.sendText("ready")
ws.sendBytes(Buffer(1, 2, 3))
ws.close()
}
```
Send behavior:
- Each call sends one websocket message.
- The server API does not expose frame-by-frame streaming.
- Once the session is closed, send calls fail with a websocket error.
### What Happens When The Connection Closes
There are three practical cases:
1. The client closes first.
The runtime replies with a close frame, releases the socket, and `receive()` returns `null`.
2. Your handler closes first with `ws.close(...)`.
The runtime sends a close frame and releases the socket locally.
3. The transport disappears unexpectedly.
The session is released and no more messages can be received; subsequent sends fail.
What Lyng code should do:
- Treat `receive() == null` as end-of-session.
- Exit the handler or break the receive loop at that point.
- Do not keep sending after close has been observed.
The current server-side API does not expose the peer close code or close reason to Lyng.
### Closing The Connection Yourself
Call `ws.close()` when you want to terminate the websocket session.
```lyng
server.ws("/chat") { ws ->
ws.sendText("server shutting down")
ws.close(1000, "done")
}
```
Close semantics:
- `close()` sends a websocket close frame with the given code and reason.
- Defaults are `code = 1000` and `reason = ""`.
- `close()` is idempotent; calling it again after close does nothing.
- After local close, the session should be treated as unusable.
- After close, `isOpen()` becomes false and further sends fail.
### WebSocket Handler Pattern
```lyng
import lyng.io.http.server
val server = HttpServer()
server.wsPath("/rooms/{room}") { ws ->
val room = routeParams["room"] ?: "<unknown>"
ws.sendText("joined:" + room)
while (true) {
val msg = ws.receive() ?: break
if (msg.isText) {
ws.sendText(room + ":" + msg.text)
}
}
ws.close()
}
server.listen(8080, "127.0.0.1")
```
## Path-Template Routes
Path templates are sugar on top of regex routes. Template parameters are exposed as decoded `routeParams`.
```lyng
server.getPath("/users/{userId}/posts/{postId}") {
respondText(
200,
routeParams["userId"] + ":" + routeParams["postId"]
)
}
```
Template rules:
- template must start with `/`
- a segment is either literal text or `{name}`
- parameter names must be valid identifiers
- parameter values match one path segment only
- parameter values use path decoding rules:
- valid percent-encoding is decoded
- `+` stays `+`
- malformed `%` stays literal
## Regex Routes
Regex routes match the whole request path, not a substring.
```lyng
server.get("^/users/([0-9]+)/posts/([0-9]+)$".re) {
val m = routeMatch!!
respondText(200, "user=" + m[1] + ", post=" + m[2])
}
```
## Basic Exact Route
```lyng
import lyng.io.http.server
val server = HttpServer()
server.get("/hello") {
setHeader("Content-Type", "text/plain")
respondText(200, "hello")
}
server.listen(8080, "127.0.0.1")
```
## Route Precedence
Dispatch order is:
1. exact method route
2. exact `any` route
3. regex method route, registration order
4. regex `any` route, registration order
5. fallback
This means exact routes stay fast and always win over template or regex routes for the same path.
## API Surface
### `Router` Route Registration Methods
- `get(path: String|Regex, handler)`
- `getPath(pathTemplate: String, handler)`
- `post(path: String|Regex, handler)`
- `postPath(pathTemplate: String, handler)`
- `put(path: String|Regex, handler)`
- `putPath(pathTemplate: String, handler)`
- `delete(path: String|Regex, handler)`
- `deletePath(pathTemplate: String, handler)`
- `any(path: String|Regex, handler)`
- `anyPath(pathTemplate: String, handler)`
- `ws(path: String|Regex, handler)`
- `wsPath(pathTemplate: String, handler)`
- `fallback(handler)`
- `mount(router)`
### `HttpServer` Route Registration Methods
- `get(path: String|Regex, handler)`
- `getPath(pathTemplate: String, handler)`
- `post(path: String|Regex, handler)`
- `postPath(pathTemplate: String, handler)`
- `put(path: String|Regex, handler)`
- `putPath(pathTemplate: String, handler)`
- `delete(path: String|Regex, handler)`
- `deletePath(pathTemplate: String, handler)`
- `any(path: String|Regex, handler)`
- `anyPath(pathTemplate: String, handler)`
- `ws(path: String|Regex, handler)`
- `wsPath(pathTemplate: String, handler)`
- `fallback(handler)`
- `mount(router)`
- `listen(port, host = null, backlog = 128)`
+2
View File
@@ -4,6 +4,8 @@ This module provides minimal raw transport networking for Lyng scripts. It is im
> **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes. > **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes.
> >
> **Shared type note:** `IpVersion`, `SocketAddress`, and `Datagram` are also available from `lyng.io.net.types` when host code wants reusable transport value types without depending on the `Net` capability object itself.
>
> **Important native platform limit:** current native TCP/UDP support is backed by a selector with a per-process file descriptor ceiling. On Linux/macOS native targets this makes high-connection-count servers and same-process load tests unsuitable once the process approaches that limit. > **Important native platform limit:** current native TCP/UDP support is backed by a selector with a per-process file descriptor ceiling. On Linux/macOS native targets this makes high-connection-count servers and same-process load tests unsuitable once the process approaches that limit.
> >
> **Recommendation:** for serious HTTP/TCP servers, prefer the JVM target today. On native targets, keep concurrency bounded, batch local load tests in waves, and use multiple worker processes behind a reverse proxy if you need more throughput before the backend is reworked. > **Recommendation:** for serious HTTP/TCP servers, prefer the JVM target today. On native targets, keep concurrency bounded, batch local load tests in waves, and use multiple worker processes behind a reverse proxy if you need more throughput before the backend is reworked.
+267 -6
View File
@@ -3,6 +3,7 @@
This module provides a way to run external processes and shell commands from Lyng scripts. It is designed to be multiplatform and uses coroutines for non-blocking execution. This module provides a way to run external processes and shell commands from Lyng scripts. It is designed to be multiplatform and uses coroutines for non-blocking execution.
> **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes. > **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes.
> The reference `lyng`/`jlyng` CLI installs this module in its import manager, so CLI scripts can use `import lyng.io.process` directly.
--- ---
@@ -44,24 +45,241 @@ suspend fun bootstrapProcess() {
--- ---
#### Using from Lyng scripts #### Shell-script shorthand
Most CLI scripts should start with the shorthand API:
```lyng
import lyng.io.process
val branch = sh("git branch --show-current").out.trim()
println("Branch: " + branch)
```
`sh(command)` starts `command` through the platform shell and returns an active `CommandRun`.
The process is already running when the handle is returned. The handle gives you both convenient capture properties and streaming line flows:
```lyng
val run = sh("git status --short")
println(run.out) // capture all stdout as String
println(run.err) // capture all stderr as String
println(run.code) // known after out/err/wait/check, otherwise null
println(run.ok) // true, false, or null
```
Use `.check()` when a non-zero exit code should fail the script:
```lyng
sh("lyng fmt --check examples/*.lyng").check()
sh("git diff --check").check()
```
Use `.wait()` when the exit code is data and should not raise:
```lyng
val code = sh("git diff --quiet").wait()
if (code == 0) {
println("clean")
} else {
println("changed")
}
```
Use `.lines` for output that may be large. It streams stdout instead of buffering the entire output:
```lyng
for (file in sh("git ls-files").lines) {
if (file.endsWith(".lyng")) {
println(file)
}
}
```
You can combine it with normal iterable helpers:
```lyng
val count = sh("git ls-files").lines.count {
it.endsWith(".lyng")
}
println("Lyng files: " + count)
```
Use `exec(executable, args)` when arguments come from data, filenames, or user input:
```lyng
val file = ARGV[0] as String
exec("git", ["add", file]).check()
```
`exec(...)` bypasses the shell, so arguments are passed as argv entries rather than parsed by shell quoting rules.
```lyng
val status = exec("git", ["status", "--short"]).out
println(status)
```
---
#### Capture vs streaming
`CommandRun` is active: it owns a running process. Choose the consumption style intentionally.
- `.out` captures stdout into memory, waits for the process to finish, and also drains stderr concurrently.
- `.err` captures stderr into memory, waits for the process to finish, and also drains stdout concurrently.
- `.check()` captures both streams, waits, and fails if the exit code is non-zero.
- `.lines` streams stdout line by line and does not buffer the whole output.
- `.errorLines` streams stderr line by line and does not buffer the whole error output.
- `.wait()` only waits and returns the exit code.
Treat a `CommandRun` as a one-shot stream owner. Pick one stdout consumption path (`out` or `lines`) and one stderr consumption path (`err` or `errorLines`). Do not collect the same stream twice; it is backed by the process pipe.
For small output, capture is simplest:
```lyng
val version = sh("git describe --tags --always").out.trim()
```
For large output, stream:
```lyng
for (line in sh("find . -type f").lines) {
if (line.endsWith(".lyng")) {
println(line)
}
}
```
For commands that may write a lot to both stdout and stderr, consume both streams. One practical pattern is to collect stderr in a background task while processing stdout:
```lyng
val r = sh("some-command-that-writes-a-lot")
val errors: List<String> = []
val stderrTask = launch {
for (line in r.errorLines) {
errors.add(line)
}
}
for (line in r.lines) {
println("OUT: " + line)
}
val code = r.wait()
stderrTask.await()
if (code != 0) {
println(errors.joinToString("\n"))
exit(code)
}
```
As with ordinary shell programming, avoid using `.out` for unbounded output. It is meant for command results that comfortably fit in memory.
---
#### Common script patterns
Capture one small value:
```lyng
val currentCommit = exec("git", ["rev-parse", "--short", "HEAD"]).out.trim()
```
Run a command for its side effects and fail on error:
```lyng
exec("mkdir", ["-p", "build/reports"]).check()
sh("lyng fmt --check src/**/*.lyng").check()
```
Filter a long output stream:
```lyng
for (path in sh("git ls-files").lines) {
if (path.endsWith(".lyng")) {
println(path)
}
}
```
Branch on a command status:
```lyng
if (exec("git", ["diff", "--quiet"]).wait() != 0) {
println("working tree has changes")
}
```
Keep user-controlled text out of the shell:
```lyng
val file = ARGV[0] as String
exec("git", ["log", "--oneline", "--", file]).check()
```
---
#### `sh` vs `exec`
Use `sh(...)` when you intentionally want shell syntax:
```lyng
sh("git status --short | wc -l").out.trim()
sh("find . -name '*.lyng' | sort").check()
```
Use `exec(...)` when you already have structured arguments:
```lyng
exec("git", ["commit", "-m", message]).check()
exec("cp", [sourcePath, targetPath]).check()
```
This distinction matters for safety. Shell commands are parsed by `/bin/sh -c` on Unix-like systems or the platform shell on supported platforms. If you interpolate untrusted text into a shell string, it can change the command being run. Prefer `exec(...)` for user input and filenames.
---
#### Low-level process API
The shorthand API is built on top of `Process.execute(...)` and `Process.shell(...)`. Use the low-level API when you need direct process control, including explicit signals or forceful termination:
```lyng ```lyng
import lyng.io.process import lyng.io.process
// Execute a process with arguments // Execute a process with arguments
val p = Process.execute("ls", ["-l", "/tmp"]) val p = Process.execute("ls", ["-l", "/tmp"])
for (line in p.stdout) { for (line in p.stdout()) {
println("OUT: " + line) println("OUT: " + line)
} }
val exitCode = p.waitFor() val exitCode = p.waitFor()
println("Process exited with: " + exitCode) println("Process exited with: " + exitCode)
// Run a shell command // Run a shell command
val sh = Process.shell("echo 'Hello from shell' | wc -w") val shellProcess = Process.shell("echo 'Hello from shell' | wc -w")
for (line in sh.stdout) { for (line in shellProcess.stdout()) {
println("Word count: " + line.trim()) println("Word count: " + line.trim())
} }
```
Signals and termination:
```lyng
val server = Process.shell("python3 -m http.server 8080")
delay(1000)
server.signal("SIGTERM")
val code = server.waitFor()
println("server exited: " + code)
```
---
#### Platform information
```lyng
import lyng.io.process
// Platform information // Platform information
val details = Platform.details() val details = Platform.details()
@@ -77,15 +295,57 @@ if (Platform.isSupported()) {
--- ---
#### Executable script example
```lyng
#!/usr/bin/env lyng
import lyng.io.process
if (ARGV.size == 0) {
println("usage: changed-files.lyng <extension>")
exit(2)
}
val ext = ARGV[0] as String
var matches = 0
for (file in sh("git status --short").lines) {
if (file.endsWith(ext)) {
println(file)
matches++
}
}
println("matched: " + matches)
```
---
#### API Reference #### API Reference
##### `Process` (static methods) ##### `Process` (static methods)
- `execute(executable: String, args: List<String>): RunningProcess` — Start an external process. - `execute(executable: String, args: List<String>): RunningProcess` — Start an external process.
- `shell(command: String): RunningProcess` — Run a command through the system shell (e.g., `/bin/sh` or `cmd.exe`). - `shell(command: String): RunningProcess` — Run a command through the system shell (e.g., `/bin/sh` or `cmd.exe`).
##### Shorthand functions
- `sh(command: String): CommandRun` — Run a command through the system shell and return an active handle.
- `exec(executable: String, args: List<String> = []): CommandRun` — Run an executable with argv-style arguments and return an active handle.
##### `CommandRun`
- `command: String` — Original command display text.
- `lines: Flow<String>` — Streaming stdout lines. Use for large output.
- `errorLines: Flow<String>` — Streaming stderr lines. Use for large error output.
- `out: String` — Captured stdout. Captures both stdout and stderr concurrently and waits for completion.
- `err: String` — Captured stderr. Captures both stdout and stderr concurrently and waits for completion.
- `wait(): Int` — Wait for the command to exit and return the exit code.
- `check(): CommandRun` — Capture output, wait, and fail if the exit code is non-zero.
- `code: Int?` — Known exit code after `wait`, `check`, `out`, or `err`; otherwise `null`.
- `ok: Bool?` — `true` for exit code 0, `false` for non-zero, or `null` before the exit code is known.
##### `RunningProcess` (instance methods) ##### `RunningProcess` (instance methods)
- `stdout: Flow` — Standard output stream as a Lyng Flow of lines. - `stdout(): Flow` — Standard output stream as a Lyng Flow of lines.
- `stderr: Flow` — Standard error stream as a Lyng Flow of lines. - `stderr(): Flow` — Standard error stream as a Lyng Flow of lines.
- `waitFor(): Int` — Wait for the process to exit and return the exit code. - `waitFor(): Int` — Wait for the process to exit and return the exit code.
- `signal(name: String)` — Send a signal to the process (e.g., `"SIGINT"`, `"SIGTERM"`, `"SIGKILL"`). - `signal(name: String)` — Send a signal to the process (e.g., `"SIGINT"`, `"SIGTERM"`, `"SIGKILL"`).
- `destroy()` — Forcefully terminate the process. - `destroy()` — Forcefully terminate the process.
@@ -110,6 +370,7 @@ Example of a restricted policy in Kotlin:
```kotlin ```kotlin
import net.sergeych.lyngio.fs.security.AccessDecision import net.sergeych.lyngio.fs.security.AccessDecision
import net.sergeych.lyngio.fs.security.AccessContext
import net.sergeych.lyngio.fs.security.Decision import net.sergeych.lyngio.fs.security.Decision
import net.sergeych.lyngio.process.security.ProcessAccessOp import net.sergeych.lyngio.process.security.ProcessAccessOp
import net.sergeych.lyngio.process.security.ProcessAccessPolicy import net.sergeych.lyngio.process.security.ProcessAccessPolicy
+191 -60
View File
@@ -1,14 +1,14 @@
### lyng.io.ws — WebSocket client for Lyng scripts # `lyng.io.ws` - WebSocket client for Lyng scripts
This module provides a compact WebSocket client API for Lyng scripts. It is implemented in `lyngio` and currently backed by Ktor WebSockets on the JVM. This module provides a compact WebSocket client API for Lyng scripts. It is implemented in `lyngio` and currently backed by Ktor WebSockets on the JVM.
> **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes. > **Note:** `lyngio` is a separate library module. It must be explicitly added as a dependency to your host application and initialized in your Lyng scopes.
>
> **Shared type note:** `WsMessage` is also available from `lyng.io.ws.types` when host code wants the reusable message type without depending on the WebSocket client module itself.
--- ## Install The Module Into A Lyng Session
#### Install the module into a Lyng session Kotlin host bootstrap example:
Kotlin (host) bootstrap example:
```kotlin ```kotlin
import net.sergeych.lyng.EvalSession import net.sergeych.lyng.EvalSession
@@ -24,59 +24,189 @@ suspend fun bootstrapWs() {
} }
``` ```
--- ## Using From Lyng Scripts
#### Using from Lyng scripts ### Text Exchange
Simple text message exchange: ```lyng
import lyng.io.ws
import lyng.io.ws val ws = Ws.connect(WS_TEST_URL)
ws.sendText("ping")
val m: WsMessage = ws.receive()
ws.close()
[ws.url() == WS_TEST_URL, m.isText, m.text]
>>> [true,true,echo:ping]
```
val ws = Ws.connect(WS_TEST_URL) ### Binary Exchange
```lyng
import lyng.buffer
import lyng.io.ws
val ws = Ws.connect(WS_TEST_BINARY_URL)
ws.sendBytes(Buffer(9, 8, 7))
val m: WsMessage = ws.receive()
ws.close()
[m.isText, (m.data as Buffer).hex]
>>> [false,010203090807]
```
### Secure `wss` Exchange
```lyng
import lyng.io.ws
val ws = Ws.connect(WSS_TEST_URL)
ws.sendText("ping")
val m: WsMessage = ws.receive()
ws.close()
[ws.url() == WSS_TEST_URL, m.text]
>>> [true,secure:ping]
```
## Message Flow And Session Lifecycle
### Reading Incoming Messages
Call `ws.receive()` to wait for the next application message.
What `receive()` returns:
- `WsMessage` for the next text or binary message.
- `null` after the peer closes the connection cleanly.
- `null` after the transport has already been closed and no more messages can arrive.
What reaches Lyng code:
- Text frames are exposed as `WsMessage(isText = true, text = ...)`.
- Binary frames are exposed as `WsMessage(isText = false, data = ...)`.
- Fragmented websocket messages are reassembled before they are returned.
- Ping and pong control frames are handled internally and are not returned by `receive()`.
- Incoming close frames are handled internally; after that `receive()` returns `null`.
Typical receive loop:
```lyng
import lyng.buffer
import lyng.io.ws
val ws = Ws.connect(WS_URL)
while (true) {
val msg = ws.receive() ?: break
if (msg.isText) {
println("text=" + msg.text)
} else {
println("bytes=" + ((msg.data as Buffer).size))
}
}
println("peer closed the websocket")
```
### Sending Outgoing Messages
Use:
- `ws.sendText(text)` for UTF-8 text messages.
- `ws.sendBytes(data)` for binary messages.
Example:
```lyng
import lyng.buffer
import lyng.io.ws
val ws = Ws.connect(WS_URL)
ws.sendText("hello")
ws.sendBytes(Buffer(1, 2, 3, 4))
```
Send behavior:
- Each call sends one websocket message.
- The API does not expose partial-frame streaming; send the whole message in one call.
- If the session is already closed, `sendText(...)` and `sendBytes(...)` fail with a websocket error.
- If the transport breaks during send, the session is released and the send call fails.
### Detecting Closed Connections
Use both signals together:
- `ws.isOpen()` tells you whether the session is still considered open right now.
- `ws.receive() == null` tells you the receive side has reached the end of the websocket session.
Practical rule:
- If `receive()` returns `null`, stop reading and treat the session as closed.
- After close has been observed, do not attempt further sends.
The API does not currently expose the peer close code or close reason to Lyng code.
### Closing The Connection Yourself
Call `ws.close()` when you are done.
```lyng
import lyng.io.ws
val ws = Ws.connect(WS_URL)
ws.sendText("bye")
ws.close(1000, "done")
```
Close semantics:
- `close()` sends a websocket close frame with the given code and reason.
- Defaults are `code = 1000` and `reason = ""`.
- After `close()`, the session is released locally and should be treated as closed immediately.
- Calling `close()` on an already closed session is a no-op.
- After local close, `receive()` returns `null` and further sends fail.
### Recommended Usage Pattern
For request-response style exchanges:
```lyng
import lyng.io.ws
val ws = Ws.connect(WS_URL)
try {
ws.sendText("ping") ws.sendText("ping")
val m: WsMessage = ws.receive() val reply = ws.receive() ?: error("socket closed before reply")
println(reply.text)
} finally {
ws.close() ws.close()
[ws.url() == WS_TEST_URL, m.isText, m.text] }
>>> [true,true,echo:ping] ```
Binary message exchange: For long-lived consumers:
import lyng.buffer ```lyng
import lyng.io.ws import lyng.io.ws
val ws = Ws.connect(WS_TEST_BINARY_URL) val ws = Ws.connect(WS_URL)
ws.sendBytes(Buffer(9, 8, 7))
val m: WsMessage = ws.receive() try {
while (true) {
val msg = ws.receive() ?: break
if (msg.isText) {
println(msg.text)
}
}
} finally {
ws.close() ws.close()
[m.isText, (m.data as Buffer).hex] }
>>> [false,010203090807] ```
Secure websocket (`wss`) exchange: ## API Reference
import lyng.io.ws ### `Ws`
val ws = Ws.connect(WSS_TEST_URL) - `isSupported(): Bool` - whether WebSocket client support is available on the current runtime.
ws.sendText("ping") - `connect(url: String, headers...): WsSession` - open a client websocket session.
val m: WsMessage = ws.receive()
ws.close()
[ws.url() == WSS_TEST_URL, m.text]
>>> [true,secure:ping]
---
#### API reference
##### `Ws` (static methods)
- `isSupported(): Bool` — Whether WebSocket client support is available on the current runtime.
- `connect(url: String, headers...): WsSession` — Open a client websocket session.
`headers...` accepts: `headers...` accepts:
- `MapEntry`, for example `"Authorization" => "Bearer x"`
- 2-item lists, for example `["Authorization", "Bearer x"]`
- `MapEntry`, e.g. `"Authorization" => "Bearer x"` ### `WsSession`
- 2-item lists, e.g. `["Authorization", "Bearer x"]`
##### `WsSession`
- `isOpen(): Bool` - `isOpen(): Bool`
- `url(): String` - `url(): String`
@@ -85,24 +215,27 @@ Secure websocket (`wss`) exchange:
- `receive(): WsMessage?` - `receive(): WsMessage?`
- `close(code: Int = 1000, reason: String = ""): void` - `close(code: Int = 1000, reason: String = ""): void`
`receive()` returns `null` after a clean close. Behavior summary:
- `receive()` returns `null` after close.
- `close()` is safe to call more than once.
- send operations require an open session.
##### `WsMessage` ### `WsMessage`
- `isText: Bool` - `isText: Bool`
- `text: String?` - `text: String?`
- `data: Buffer?` - `data: Buffer?`
Text messages populate `text`; binary messages populate `data`. Payload rules:
- Text messages populate `text` and leave `data == null`.
- Binary messages populate `data` and leave `text == null`.
--- ## Security Policy
#### Security policy
The module uses `WsAccessPolicy` to authorize websocket operations. The module uses `WsAccessPolicy` to authorize websocket operations.
- `WsAccessPolicy` — interface for custom policies - `WsAccessPolicy` - interface for custom policies.
- `PermitAllWsAccessPolicy` — allows all websocket operations - `PermitAllWsAccessPolicy` - allows all websocket operations.
- `WsAccessOp.Connect(url)` - `WsAccessOp.Connect(url)`
- `WsAccessOp.Send(url, bytes, isText)` - `WsAccessOp.Send(url, bytes, isText)`
- `WsAccessOp.Receive(url)` - `WsAccessOp.Receive(url)`
@@ -135,14 +268,12 @@ val allowLocalOnly = object : WsAccessPolicy {
} }
``` ```
--- ## Platform Support
#### Platform support - **JVM:** supported.
- **Android:** supported via the Ktor CIO websocket client backend.
- **JVM:** supported - **JS:** supported via the Ktor JS websocket client backend.
- **Android:** supported via the Ktor CIO websocket client backend - **Linux native:** supported via the Ktor Curl websocket client backend.
- **JS:** supported via the Ktor JS websocket client backend - **Windows native:** supported via the Ktor WinHttp websocket client backend.
- **Linux native:** supported via the Ktor Curl websocket client backend - **Apple native:** supported via the Ktor Darwin websocket client backend.
- **Windows native:** supported via the Ktor WinHttp websocket client backend - **Other targets:** may report unsupported; use `Ws.isSupported()` before relying on websocket client access.
- **Apple native:** supported via the Ktor Darwin websocket client backend
- **Other targets:** may report unsupported; use `Ws.isSupported()` before relying on websocket client access
+10
View File
@@ -4,6 +4,7 @@ The Lyng CLI is the reference command-line tool for the Lyng language. It lets y
- Run Lyng scripts from files or inline strings (shebangs accepted) - Run Lyng scripts from files or inline strings (shebangs accepted)
- Use standard argument passing (`ARGV`) to your scripts. - Use standard argument passing (`ARGV`) to your scripts.
- Import `lyng.io.process` for shell-script style process execution (`sh`, `exec`, and `CommandRun`).
- Resolve local file imports from the executed script's directory tree. - Resolve local file imports from the executed script's directory tree.
- Format Lyng source files via the built-in `fmt` subcommand. - Format Lyng source files via the built-in `fmt` subcommand.
- Register synchronous process-exit handlers with `atExit(...)`. - Register synchronous process-exit handlers with `atExit(...)`.
@@ -81,6 +82,15 @@ lyng -- -my-script.lyng arg1 arg2
lyng -x "println(\"Hello\")" more args lyng -x "println(\"Hello\")" more args
``` ```
The CLI installs `lyng.io.process`, so scripts can use the shell/process shorthand after importing it:
```lyng
import lyng.io.process
val branch = sh("git branch --show-current").out.trim()
exec("git", ["status", "--short"]).check()
```
- Print version/help: - Print version/help:
``` ```
+28 -11
View File
@@ -12,12 +12,15 @@
#### Included Modules #### Included Modules
- **[lyng.io.db](lyng.io.db.md):** Portable SQL database access. Provides `Database`, `SqlTransaction`, `ResultSet`, SQLite support through `lyng.io.db.sqlite`, and JVM JDBC support through `lyng.io.db.jdbc`.
- **[lyng.io.fs](lyng.io.fs.md):** Async filesystem access. Provides the `Path` class for file/directory operations, streaming, and globbing. - **[lyng.io.fs](lyng.io.fs.md):** Async filesystem access. Provides the `Path` class for file/directory operations, streaming, and globbing.
- **[lyng.io.process](lyng.io.process.md):** External process execution and shell commands. Provides `Process`, `RunningProcess`, and `Platform` information. - **[lyng.io.process](lyng.io.process.md):** External process execution and shell commands. Provides shell-script sugar (`sh`, `exec`, `CommandRun`), low-level `Process`/`RunningProcess`, and `Platform` information.
- **[lyng.io.console](lyng.io.console.md):** Rich console/TTY access. Provides `Console` capability detection, geometry, output, and iterable events. - **[lyng.io.console](lyng.io.console.md):** Rich console/TTY access. Provides `Console` capability detection, geometry, output, and iterable events.
- **[lyng.io.http](lyng.io.http.md):** HTTP/HTTPS client access. Provides `Http`, `HttpRequest`, `HttpResponse`, and `HttpHeaders`. - **[lyng.io.http](lyng.io.http.md):** HTTP/HTTPS client access. Provides `Http`, `HttpRequest`, `HttpResponse`, and `HttpHeaders`.
- **[lyng.io.http.server](lyng.io.http.server.md):** Minimal HTTP/1.1 and WebSocket server. Provides `HttpServer`, `Router`, `ServerRequest`, `RequestContext`, and `ServerWebSocket`.
- **[lyng.io.ws](lyng.io.ws.md):** WebSocket client access. Provides `Ws`, `WsSession`, and `WsMessage`. - **[lyng.io.ws](lyng.io.ws.md):** WebSocket client access. Provides `Ws`, `WsSession`, and `WsMessage`.
- **[lyng.io.net](lyng.io.net.md):** Transport networking. Provides `Net`, `TcpSocket`, `TcpServer`, `UdpSocket`, and `SocketAddress`. - **[lyng.io.net](lyng.io.net.md):** Transport networking. Provides `Net`, `TcpSocket`, `TcpServer`, `UdpSocket`, and `SocketAddress`.
- **Shared networking type packages:** `lyng.io.http.types`, `lyng.io.ws.types`, and `lyng.io.net.types` expose reusable value types such as `HttpHeaders`, `WsMessage`, `IpVersion`, `SocketAddress`, and `Datagram` when host code wants type-only imports without installing the corresponding capability object module.
--- ---
@@ -43,6 +46,9 @@ To use `lyngio` modules in your scripts, you must install them into your Lyng sc
```kotlin ```kotlin
import net.sergeych.lyng.EvalSession import net.sergeych.lyng.EvalSession
import net.sergeych.lyng.io.db.createDbModule
import net.sergeych.lyng.io.db.jdbc.createJdbcModule
import net.sergeych.lyng.io.db.sqlite.createSqliteModule
import net.sergeych.lyng.io.fs.createFs import net.sergeych.lyng.io.fs.createFs
import net.sergeych.lyng.io.process.createProcessModule import net.sergeych.lyng.io.process.createProcessModule
import net.sergeych.lyng.io.console.createConsoleModule import net.sergeych.lyng.io.console.createConsoleModule
@@ -61,6 +67,9 @@ suspend fun runMyScript() {
val scope = session.getScope() val scope = session.getScope()
// Install modules with policies // Install modules with policies
createDbModule(scope)
createJdbcModule(scope)
createSqliteModule(scope)
createFs(PermitAllAccessPolicy, scope) createFs(PermitAllAccessPolicy, scope)
createProcessModule(PermitAllProcessAccessPolicy, scope) createProcessModule(PermitAllProcessAccessPolicy, scope)
createConsoleModule(PermitAllConsoleAccessPolicy, scope) createConsoleModule(PermitAllConsoleAccessPolicy, scope)
@@ -70,6 +79,9 @@ suspend fun runMyScript() {
// Now scripts can import them // Now scripts can import them
session.eval(""" session.eval("""
import lyng.io.db
import lyng.io.db.jdbc
import lyng.io.db.sqlite
import lyng.io.fs import lyng.io.fs
import lyng.io.process import lyng.io.process
import lyng.io.console import lyng.io.console
@@ -77,6 +89,8 @@ suspend fun runMyScript() {
import lyng.io.net import lyng.io.net
import lyng.io.ws import lyng.io.ws
println("H2 JDBC available: " + (openH2("mem:demo;DB_CLOSE_DELAY=-1") != null))
println("SQLite available: " + (openSqlite(":memory:") != null))
println("Working dir: " + Path(".").readUtf8()) println("Working dir: " + Path(".").readUtf8())
println("OS: " + Platform.details().name) println("OS: " + Platform.details().name)
println("TTY: " + Console.isTty()) println("TTY: " + Console.isTty())
@@ -94,6 +108,7 @@ suspend fun runMyScript() {
`lyngio` is built with a "Secure by Default" philosophy. Every I/O or process operation is checked against a policy. `lyngio` is built with a "Secure by Default" philosophy. Every I/O or process operation is checked against a policy.
- **Filesystem Security:** Implement `FsAccessPolicy` to restrict access to specific paths or operations (e.g., read-only access to a sandbox directory). - **Filesystem Security:** Implement `FsAccessPolicy` to restrict access to specific paths or operations (e.g., read-only access to a sandbox directory).
- **Database Installation:** Database access is still explicit-capability style. The host must install `lyng.io.db` and at least one provider such as `lyng.io.db.sqlite` or `lyng.io.db.jdbc`; otherwise scripts cannot open databases.
- **Process Security:** Implement `ProcessAccessPolicy` to restrict which executables can be run or to disable shell execution entirely. - **Process Security:** Implement `ProcessAccessPolicy` to restrict which executables can be run or to disable shell execution entirely.
- **Console Security:** Implement `ConsoleAccessPolicy` to control output writes, event reads, and raw mode switching. - **Console Security:** Implement `ConsoleAccessPolicy` to control output writes, event reads, and raw mode switching.
- **HTTP Security:** Implement `HttpAccessPolicy` to restrict which requests scripts may send. - **HTTP Security:** Implement `HttpAccessPolicy` to restrict which requests scripts may send.
@@ -101,10 +116,12 @@ suspend fun runMyScript() {
- **WebSocket Security:** Implement `WsAccessPolicy` to restrict websocket connects and message flow. - **WebSocket Security:** Implement `WsAccessPolicy` to restrict websocket connects and message flow.
For more details, see the specific module documentation: For more details, see the specific module documentation:
- [Database Module Details](lyng.io.db.md)
- [Filesystem Security Details](lyng.io.fs.md#access-policy-security) - [Filesystem Security Details](lyng.io.fs.md#access-policy-security)
- [Process Security Details](lyng.io.process.md#security-policy) - [Process Security Details](lyng.io.process.md#security-policy)
- [Console Module Details](lyng.io.console.md) - [Console Module Details](lyng.io.console.md)
- [HTTP Module Details](lyng.io.http.md) - [HTTP Module Details](lyng.io.http.md)
- [HTTP Server Module Details](lyng.io.http.server.md)
- [Transport Networking Details](lyng.io.net.md) - [Transport Networking Details](lyng.io.net.md)
- [WebSocket Module Details](lyng.io.ws.md) - [WebSocket Module Details](lyng.io.ws.md)
@@ -112,16 +129,16 @@ For more details, see the specific module documentation:
#### Platform Support Overview #### Platform Support Overview
| Platform | lyng.io.fs | lyng.io.process | lyng.io.console | lyng.io.http | lyng.io.ws | lyng.io.net | | Platform | lyng.io.db/sqlite | lyng.io.db/jdbc | lyng.io.fs | lyng.io.process | lyng.io.console | lyng.io.http | lyng.io.ws | lyng.io.net |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **JVM** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | **JVM** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **Linux Native** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | **Linux Native** | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **Apple Native** | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ | | **Apple Native** | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ |
| **Windows Native** | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | | **Windows Native** | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ |
| **Android** | ✅ | ❌ | ⚠️ | ✅ | ✅ | ✅ | | **Android** | ⚠️ | ❌ | ✅ | ❌ | ⚠️ | ✅ | ✅ | ✅ |
| **JS / Node** | ✅ | ❌ | ⚠️ | ✅ | ✅ | ✅ | | **JS / Node** | ❌ | ❌ | ✅ | ❌ | ⚠️ | ✅ | ✅ | ✅ |
| **JS / Browser** | ✅ (in-memory) | ❌ | ⚠️ | ✅ | ✅ | ❌ | | **JS / Browser** | ❌ | ❌ | ✅ (in-memory) | ❌ | ⚠️ | ✅ | ✅ | ❌ |
| **Wasm** | ✅ (in-memory) | ❌ | ⚠️ | ❌ | ❌ | ❌ | | **Wasm** | ❌ | ❌ | ✅ (in-memory) | ❌ | ⚠️ | ❌ | ❌ | ❌ |
Legend: Legend:
- `✅` supported - `✅` supported
+78
View File
@@ -36,6 +36,17 @@ This example shows how to launch a coroutine with `launch` which returns [Deferr
Launch has the only argument which should be a callable (lambda usually) that is run in parallel (or cooperatively in parallel), and return anything as the result. Launch has the only argument which should be a callable (lambda usually) that is run in parallel (or cooperatively in parallel), and return anything as the result.
When you have an iterable of deferreds, use `joinAll()` to await all of them and collect results in input order:
val jobs = (1..4).map { n ->
launch {
delay(1)
n * 10
}
}
assertEquals([10, 20, 30, 40], jobs.joinAll())
>>> void
If you no longer need the result, cancel the deferred. Awaiting a cancelled deferred throws `CancellationException`: If you no longer need the result, cancel the deferred. Awaiting a cancelled deferred throws `CancellationException`:
var reached = false var reached = false
@@ -219,6 +230,73 @@ Flows allow easy transforming of any [Iterable]. See how the standard Lyng libra
} }
} }
## Channel
A [Channel] is a **hot pipe** between coroutines: values are pushed in by a producer and pulled out by a consumer, with each value consumed exactly once.
Unlike a `Flow` (which is cold and re-runs its generator on every collection), a `Channel` is stateful — the right tool for classic _producer / consumer_ work.
val ch = Channel() // rendezvous: sender waits for receiver
val producer = launch {
for (i in 1..5) ch.send(i)
ch.close() // signal: no more values
}
var item = ch.receive() // suspends until a value is ready
while (item != null) {
println(item)
item = ch.receive()
}
// prints 1 2 3 4 5
`receive()` returns `null` when the channel is both closed and fully drained — that is the idiomatic loop termination condition.
Channels can also be buffered so the producer can run ahead:
val ch = Channel(4) // buffer up to 4 items without blocking
ch.send(10)
ch.send(20)
ch.send(30)
ch.close()
assertEquals(10, ch.receive())
assertEquals(20, ch.receive())
assertEquals(30, ch.receive())
assertEquals(null, ch.receive()) // drained
For the full API — including `tryReceive`, `Channel.UNLIMITED`, and the fan-out / ping-pong patterns — see the [Channel] reference page.
## LaunchPool
When you need **bounded concurrency** — run at most *N* tasks at the same time without spawning a new coroutine per task — use [LaunchPool]:
```lyng
val pool = LaunchPool(4) // at most 4 tasks run in parallel
val jobs = (1..20).map { n ->
pool.launch { expensiveCompute(n) }
}
pool.closeAndJoin() // wait for all tasks to complete
val results = jobs.joinAll()
```
Exceptions thrown inside a submitted lambda are captured in the returned `Deferred` and do not crash the pool, so other tasks continue running normally.
See [LaunchPool] for the full API including bounded queues and cancellation.
[LaunchPool]: LaunchPool.md
| | Flow | Channel |
|---|---|---|
| **temperature** | cold (lazy) | hot (eager) |
| **replay** | every collector gets a fresh run | each item consumed once |
| **consumers** | any number, each gets all items | one receiver per item |
| **typical use** | transform pipelines, sequences | producer–consumer, fan-out |
[Channel]: Channel.md
[Iterable]: Iterable.md [Iterable]: Iterable.md
+1 -1
View File
@@ -4,7 +4,7 @@ Saved on April 4, 2026 before the `List<Int>` indexed-access follow-up fix.
Benchmark target: Benchmark target:
- [examples/pi-bench.py](/home/sergeych/dev/lyng/examples/pi-bench.py) - [examples/pi-bench.py](/home/sergeych/dev/lyng/examples/pi-bench.py)
- [examples/pi-bench.lyng](/home/sergeych/dev/lyng/examples/pi-bench.lyng) - [examples/pi-bench.lyng](../examples/pi-bench.lyng)
Execution path: Execution path:
- Python: `python3 examples/pi-bench.py` - Python: `python3 examples/pi-bench.py`
+50
View File
@@ -0,0 +1,50 @@
class Tag(name: String) {
val name = name
var inner = ""
fun child(tagName: String, block: Tag.()->void) {
val child = Tag(tagName)
with(child) { block(this) }
inner += child.render()
}
fun head(block: Tag.()->void) { child("head", block) }
fun body(block: Tag.()->void) { child("body", block) }
fun title(block: Tag.()->void) { child("title", block) }
fun h1(block: Tag.()->void) { child("h1", block) }
fun addText(text: String) {
inner += text
}
fun render() {
"<" + name + ">" + inner + "</" + name + ">"
}
}
context(Tag)
fun String.unaryPlus() {
this@Tag.addText(this)
}
fun html(block: Tag.()->void) {
val root = Tag("html")
with(root) { block(this) }
root.render()
}
val page = html {
head {
title {
+"Demo"
}
}
body {
h1 {
+"Heading 1"
}
}
}
println(page)
assertEquals("<html><head><title>Demo</title></head><body><h1>Heading 1</h1></body></html>", page)
+8
View File
@@ -1,6 +1,9 @@
// Sample: Operator Overloading in Lyng // Sample: Operator Overloading in Lyng
class Vector<T>(val x: T, val y: T) { class Vector<T>(val x: T, val y: T) {
// Overload unary +
fun unaryPlus() = this
// Overload + // Overload +
fun plus(other: Vector<U>) = Vector(x + other.x, y + other.y) fun plus(other: Vector<U>) = Vector(x + other.x, y + other.y)
@@ -28,6 +31,11 @@ val v2 = Vector(5, 5)
println("v1: " + v1) println("v1: " + v1)
println("v2: " + v2) println("v2: " + v2)
// Test unary +
val v0 = +v1
println("+v1 = " + v0)
assertEquals(Vector(10, 20), v0)
// Test binary + // Test binary +
val v3 = v1 + v2 val v3 = v1 + v2
println("v1 + v2 = " + v3) println("v1 + v2 = " + v3)
+162 -6
View File
@@ -1,6 +1,40 @@
# Lyng serialization # Lyng serialization
Lyng has builting binary bit-effective serialization format, called Lynon for LYng Object Notation. It is typed, binary, implements caching, automatic compression, variable-length ints, one-bit Booleans an many nice features. Lyng has a built-in serialization module, `lyng.serialization`.
There are now two built-in formats with different goals:
- `Lynon`: the canonical binary format for Lyng values.
- `Json`: the canonical JSON-based round-trip format for Lyng values.
In addition, `Obj.toJson()` / `toJsonString()` remain available as a plain JSON projection for interoperability with
regular JSON tools and Kotlin `kotlinx.serialization`.
## Canonical formats
`Lynon` and `Json` are both exposed as format objects with the same surface:
- `Format.encode(value)`
- `Format.decode(encodedValue)`
For the built-in formats:
- `Lynon.encode(x)` returns `BitBuffer`
- `Lynon.decode(bitBuffer)` returns the original Lyng value
- `Json.encode(x)` returns `String`
- `Json.decode(jsonString)` returns the original Lyng value
`Json` also provides a typed canonical mode:
- `Json.encodeAs(Type, value)` returns `String`
- `Json.decodeAs(Type, jsonString)` returns the original Lyng value of the specified type
This is still canonical JSON, but it is schema-driven instead of fully self-describing.
## Lynon
Lynon is LYng Object Notation. It is typed, binary, bit-effective, implements caching, automatic compression,
variable-length integers, one-bit booleans, and preserves Lyng runtime structure well.
It is as simple as: It is as simple as:
@@ -20,7 +54,8 @@ It is as simple as:
assert( text.length > (encodedBits.toBuffer() as Buffer).size ) assert( text.length > (encodedBits.toBuffer() as Buffer).size )
>>> void >>> void
Any class you create is serializable by default; lynon serializes first constructor fields, then any `var` member fields. Any class you create is serializable by default; Lynon serializes first constructor fields, then any `var` member
fields.
## Transient Fields ## Transient Fields
@@ -40,7 +75,7 @@ class MyData(@Transient val tempSecret, val publicData) {
Transient fields: Transient fields:
- Are **omitted** from Lynon binary streams. - Are **omitted** from Lynon binary streams.
- Are **omitted** from JSON output (via `toJson`). - Are **omitted** from JSON output (`toJson`) and canonical `Json.encode(...)`.
- Are **ignored** during structural equality checks (`==`). - Are **ignored** during structural equality checks (`==`).
- If a transient constructor parameter has a **default value**, it will be restored to that default value during deserialization. Otherwise, it will be `null`. - If a transient constructor parameter has a **default value**, it will be restored to that default value during deserialization. Otherwise, it will be `null`.
- Class body fields marked as `@Transient` will keep their initial values (or values assigned in `init`) after deserialization. - Class body fields marked as `@Transient` will keep their initial values (or values assigned in `init`) after deserialization.
@@ -49,8 +84,131 @@ Transient fields:
- **Singleton Objects**: `object` declarations are serializable by name. Their state (mutable fields) is also serialized and restored, respecting `@Transient`. - **Singleton Objects**: `object` declarations are serializable by name. Their state (mutable fields) is also serialized and restored, respecting `@Transient`.
- **Classes**: Class objects themselves can be serialized. They are serialized by their full qualified name. When converted to JSON, a class object includes its public static fields (excluding those marked `@Transient`). - **Classes**: Class objects themselves can be serialized. They are serialized by their full qualified name. When converted to JSON, a class object includes its public static fields (excluding those marked `@Transient`).
- **Exceptions**: canonical formats preserve exception class, message, extra data, and captured stack trace.
## Custom Serialization ## Plain JSON projection vs canonical Json format
There are two JSON-related APIs and they serve different purposes:
- `Obj.toJson()` / `toJsonString()`
- produce ordinary JSON values
- best for interop with external JSON systems
- best for `Obj.decodeSerializable()` / `decodeSerializableWith()`
- may be lossy for Lyng-specific structures
- `Json.encode()` / `Json.decode()`
- produce JSON text too
- use Lyng-specific type tags so the payload is self-describing
- intended for round-tripping Lyng values
- intended to match Lynon semantics where JSON can carry them
- still keep ordinary string-key maps in traditional JSON object form
- can preserve values that plain JSON cannot represent directly, such as:
- maps with non-string keys
- sets
- buffers and bit buffers
- class instances
- singleton objects
- enums
- exceptions
- date/time objects
- non-finite reals
- `void`
- `Json.encodeAs(Type, value)` / `Json.decodeAs(Type, text)`
- also round-trip Lyng values through JSON text
- use the declared or requested type as decoding schema
- recursively omit type tags when the declared type is already exact enough
- keep canonical tags when the runtime value is more specific than the declared type
- produce less noisy JSON for closed and otherwise precisely-typed object graphs
- still keep ordinary `Map<String, ...>` values in traditional JSON object form
Why this split exists:
- plain `toJson()` must remain ordinary JSON so it stays convenient for external JSON systems and Kotlin
`kotlinx.serialization`
- canonical `Json.encode()` is for Lyng-to-Lyng transport through JSON text without any external schema, so it must
remain self-describing and preserve Lyng runtime
distinctions whenever possible
- `Json.encodeAs()` exists for the cases where a schema is known on both sides and we want canonical round-trip
behavior with fewer tags
- one API cannot optimize for both goals at once: either you get too many Lyng tags for ordinary JSON interop, or you
get lossy round-trips
Example:
import lyng.serialization
import lyng.time
enum Color { Red, Green }
class Point(x,y) { var z = 42 }
val p = Point(1,2)
p.z = 99
val x = List(
p,
Map([1, "one"], ["two", 2]),
Set(1,2,3),
"hello".encodeUtf8(),
Date(2026,4,15),
Color.Green
)
assertEquals(x, Json.decode(Json.encode(x)))
>>> void
Typed canonical example:
import lyng.serialization
closed class Point(x: Int, y: Int)
closed class Segment(a: Point, b: Point)
val value = Segment(Point(0,1), Point(2,3))
val json = Json.encodeAs(Segment, value)
assertEquals("{\"a\":{\"x\":0,\"y\":1},\"b\":{\"x\":2,\"y\":3}}", json)
assertEquals(value, Json.decodeAs(Segment, json))
>>> void
## Adding more formats from Kotlin modules
External modules can add new formats on the Kotlin side.
The common base class is:
```kotlin
abstract class ObjSerializationFormatClass(className: String) : ObjClass(className) {
abstract suspend fun encodeValue(scope: Scope, value: Obj): Obj
abstract suspend fun decodeValue(scope: Scope, encoded: Obj): Obj
}
```
To export a new format from a module:
```kotlin
im.addPackage("test.formats") { module ->
module.bindSerializationFormat(
object : ObjSerializationFormatClass("Reverse") {
override suspend fun encodeValue(scope: Scope, value: Obj): Obj =
ObjString(value.toString(scope).value.reversed())
override suspend fun decodeValue(scope: Scope, encoded: Obj): Obj =
ObjString((encoded as ObjString).value.reversed())
}
)
}
```
Then from Lyng, after importing the Kotlin module above, usage looks like:
```lyng
import test.formats
assertEquals("cba", Reverse.encode("abc"))
assertEquals("abc", Reverse.decode("cba"))
```
## Notes
Important is to understand that normally `Lynon.decode` wants [BitBuffer], as `Lynon.encode` produces. If you have the regular [Buffer], be sure to convert it: Important is to understand that normally `Lynon.decode` wants [BitBuffer], as `Lynon.encode` produces. If you have the regular [Buffer], be sure to convert it:
@@ -59,5 +217,3 @@ Important is to understand that normally `Lynon.decode` wants [BitBuffer], as `L
this possibly creates extra zero bits at the end, as bit content could be shorter than byte-grained but for the Lynon format it does not make sense. Note that when you serialize [BitBuffer], exact number of bits is written. To convert bit buffer to bytes: this possibly creates extra zero bits at the end, as bit content could be shorter than byte-grained but for the Lynon format it does not make sense. Note that when you serialize [BitBuffer], exact number of bits is written. To convert bit buffer to bytes:
Lynon.encode("hello").toBuffer() Lynon.encode("hello").toBuffer()
(topic is incomplete and under construction)
+122 -63
View File
@@ -1,74 +1,135 @@
# Lyng time functions # Lyng time functions
Lyng date and time support requires importing `lyng.time` packages. Lyng uses simple yet modern time object models: Lyng date and time support requires importing `lyng.time`. The module provides four related types:
- `Instant` class for absolute time stamps with platform-dependent resolution. - `Instant` for absolute timestamps.
- `DateTime` class for calendar-aware points in time within a specific time zone. - `Date` for calendar dates without time-of-day or timezone.
- `Duration` to represent amount of time not depending on the calendar (e.g., milliseconds, seconds). - `DateTime` for calendar-aware points in time in a specific timezone.
- `Duration` for absolute elapsed time.
## Time instant: `Instant` ## Time instant: `Instant`
Represent some moment of time not depending on the calendar. It is similar to `TIMESTAMP` in SQL or `Instant` in Kotlin. `Instant` represents some moment of time independently of the calendar. It is similar to SQL `TIMESTAMP`
or Kotlin `Instant`.
### Constructing and converting ### Constructing and converting
import lyng.time import lyng.time
// default constructor returns time now:
val t1 = Instant() val t1 = Instant()
val t2 = Instant(1704110400)
// constructing from a number is treated as seconds since unix epoch:
val t2 = Instant(1704110400) // 2024-01-01T12:00:00Z
// from RFC3339 string:
val t3 = Instant("2024-01-01T12:00:00.123456Z") val t3 = Instant("2024-01-01T12:00:00.123456Z")
// truncation: val t4 = t3.truncateToMinute()
val t4 = t3.truncateToMinute assertEquals("2024-01-01T12:00:00Z", t4.toRFC3339())
assertEquals(t4.toRFC3339(), "2024-01-01T12:00:00Z")
// to localized DateTime (uses system default TZ if not specified):
val dt = t3.toDateTime("+02:00") val dt = t3.toDateTime("+02:00")
assertEquals(dt.hour, 14) assertEquals(14, dt.hour)
val d = t3.toDate("Z")
assertEquals(Date(2024, 1, 1), d)
### Instant members ### Instant members
| member | description | | member | description |
|--------------------------------|---------------------------------------------------------| |--------------------------------|------------------------------------------------------|
| epochSeconds: Real | positive or negative offset in seconds since Unix epoch | | epochSeconds: Real | offset in seconds since Unix epoch |
| epochWholeSeconds: Int | same, but in _whole seconds_. Slightly faster | | epochWholeSeconds: Int | whole seconds since Unix epoch |
| nanosecondsOfSecond: Int | offset from epochWholeSeconds in nanos | | nanosecondsOfSecond: Int | nanoseconds within the current second |
| isDistantFuture: Bool | true if it `Instant.distantFuture` | | isDistantFuture: Bool | true if it is `Instant.distantFuture` |
| isDistantPast: Bool | true if it `Instant.distantPast` | | isDistantPast: Bool | true if it is `Instant.distantPast` |
| truncateToMinute: Instant | create new instance truncated to minute | | truncateToMinute(): Instant | truncate to minute precision |
| truncateToSecond: Instant | create new instance truncated to second | | truncateToSecond(): Instant | truncate to second precision |
| truncateToMillisecond: Instant | truncate new instance to millisecond | | truncateToMillisecond(): Instant | truncate to millisecond precision |
| truncateToMicrosecond: Instant | truncate new instance to microsecond | | truncateToMicrosecond(): Instant | truncate to microsecond precision |
| toRFC3339(): String | format as RFC3339 string (UTC) | | toRFC3339(): String | format as RFC3339 string in UTC |
| toDateTime(tz?): DateTime | localize to a TimeZone (ID string or offset seconds) | | toDateTime(tz?): DateTime | localize to a timezone |
| toDate(tz?): Date | convert to a calendar date in a timezone |
## Calendar time: `DateTime` ## Calendar date: `Date`
`DateTime` represents a point in time in a specific timezone. It provides access to calendar components like year, `Date` represents a pure calendar date. It has no time-of-day and no attached timezone. Use it for values
month, and day. like birthdays, due dates, invoice dates, and SQL `DATE` columns.
### Constructing
import lyng.time
val today = Date()
val d1 = Date(2026, 4, 15)
val d2 = Date("2024-02-29")
val d3 = Date.parseIso("2024-02-29")
val d4 = Date(DateTime(2024, 5, 20, 15, 30, 45, "+02:00"))
val d5 = Date(Instant("2024-01-01T23:30:00Z"), "+02:00")
### Date members
| member | description |
|--------------------------------|------------------------------------------------------------|
| year: Int | year component |
| month: Int | month component (1..12) |
| day: Int | day of month (alias `dayOfMonth`) |
| dayOfWeek: Int | day of week (1=Monday, 7=Sunday) |
| dayOfYear: Int | day of year (1..365/366) |
| isLeapYear: Bool | whether this date is in a leap year |
| lengthOfMonth: Int | number of days in this month |
| lengthOfYear: Int | 365 or 366 |
| toIsoString(): String | ISO `YYYY-MM-DD` string |
| toSortableString(): String | alias to `toIsoString()` |
| toDateTime(tz="Z"): DateTime | start-of-day `DateTime` in the specified timezone |
| atStartOfDay(tz="Z"): DateTime | alias to `toDateTime()` |
| addDays(n): Date | add or subtract calendar days |
| addMonths(n): Date | add or subtract months, normalizing end-of-month |
| addYears(n): Date | add or subtract years |
| daysUntil(other): Int | calendar days until `other` |
| daysSince(other): Int | calendar days since `other` |
| static today(tz?): Date | today in the specified timezone |
| static parseIso(s): Date | parse ISO `YYYY-MM-DD` |
### Date arithmetic
`Date` supports only whole-day arithmetic. This is deliberate: calendar dates should not silently accept
sub-day durations.
import lyng.time
val d1 = Date(2026, 4, 15)
val d2 = d1.addDays(10)
assertEquals(Date(2026, 4, 25), d2)
assertEquals(Date(2026, 4, 18), d1 + 3.days)
assertEquals(Date(2026, 4, 12), d1 - 3.days)
assertEquals(10, d1.daysUntil(d2))
assertEquals(10, d2.daysSince(d1))
assertEquals(10, d2 - d1)
### Date conversions
import lyng.time
val i = Instant("2024-01-01T23:30:00Z")
assertEquals(Date(2024, 1, 1), i.toDate("Z"))
assertEquals(Date(2024, 1, 2), i.toDate("+02:00"))
val dt = DateTime(2024, 5, 20, 15, 30, 45, "+02:00")
assertEquals(Date(2024, 5, 20), dt.date)
assertEquals(Date(2024, 5, 20), dt.toDate())
assertEquals(DateTime(2024, 5, 20, 0, 0, 0, "Z"), Date(2024, 5, 20).toDateTime("Z"))
assertEquals(DateTime(2024, 5, 20, 0, 0, 0, "+02:00"), Date(2024, 5, 20).atStartOfDay("+02:00"))
## Calendar time: `DateTime`
`DateTime` represents a point in time in a specific timezone. It provides access to calendar components
such as year, month, day, and hour.
### Constructing ### Constructing
import lyng.time import lyng.time
// Current time in system default timezone
val now = DateTime.now() val now = DateTime.now()
// Specific timezone
val offsetTime = DateTime.now("+02:00") val offsetTime = DateTime.now("+02:00")
// From Instant
val dt = Instant().toDateTime("Z") val dt = Instant().toDateTime("Z")
// By components (year, month, day, hour=0, minute=0, second=0, timeZone="UTC")
val dt2 = DateTime(2024, 1, 1, 12, 0, 0, "Z") val dt2 = DateTime(2024, 1, 1, 12, 0, 0, "Z")
// From RFC3339 string
val dt3 = DateTime.parseRFC3339("2024-01-01T12:00:00+02:00") val dt3 = DateTime.parseRFC3339("2024-01-01T12:00:00+02:00")
### DateTime members ### DateTime members
@@ -83,7 +144,9 @@ month, and day.
| second: Int | second component (0..59) | | second: Int | second component (0..59) |
| dayOfWeek: Int | day of week (1=Monday, 7=Sunday) | | dayOfWeek: Int | day of week (1=Monday, 7=Sunday) |
| timeZone: String | timezone ID string | | timeZone: String | timezone ID string |
| date: Date | calendar date component |
| toInstant(): Instant | convert back to absolute Instant | | toInstant(): Instant | convert back to absolute Instant |
| toDate(): Date | extract the calendar date in this timezone |
| toUTC(): DateTime | shortcut to convert to UTC | | toUTC(): DateTime | shortcut to convert to UTC |
| toTimeZone(tz): DateTime | convert to another timezone | | toTimeZone(tz): DateTime | convert to another timezone |
| addMonths(n): DateTime | add/subtract months (normalizes end of month) | | addMonths(n): DateTime | add/subtract months (normalizes end of month) |
@@ -96,28 +159,27 @@ month, and day.
`DateTime` handles calendar arithmetic correctly: `DateTime` handles calendar arithmetic correctly:
import lyng.time
val leapDay = Instant("2024-02-29T12:00:00Z").toDateTime("Z") val leapDay = Instant("2024-02-29T12:00:00Z").toDateTime("Z")
val nextYear = leapDay.addYears(1) val nextYear = leapDay.addYears(1)
assertEquals(nextYear.day, 28) // Feb 29, 2024 -> Feb 28, 2025 assertEquals(28, nextYear.day)
# `Duration` class # `Duration` class
Represent absolute time distance between two `Instant`. `Duration` represents absolute elapsed time between two instants.
import lyng.time import lyng.time
val t1 = Instant() val t1 = Instant()
// yes we can delay to period, and it is not blocking. is suspends!
delay(1.millisecond) delay(1.millisecond)
val t2 = Instant() val t2 = Instant()
// be suspend, so actual time may vary:
assert( t2 - t1 >= 1.millisecond) assert(t2 - t1 >= 1.millisecond)
assert( t2 - t1 < 100.millisecond) assert(t2 - t1 < 100.millisecond)
>>> void >>> void
Duration can be converted from numbers, like `5.minutes` and so on. Extensions are created for Duration values can be created from numbers using extensions on `Int` and `Real`:
`Int` and `Real`, so for n as Real or Int it is possible to create durations::
- `n.millisecond`, `n.milliseconds` - `n.millisecond`, `n.milliseconds`
- `n.second`, `n.seconds` - `n.second`, `n.seconds`
@@ -125,10 +187,9 @@ Duration can be converted from numbers, like `5.minutes` and so on. Extensions a
- `n.hour`, `n.hours` - `n.hour`, `n.hours`
- `n.day`, `n.days` - `n.day`, `n.days`
The bigger time units like months or years are calendar-dependent and can't be used with `Duration`. Larger units like months or years are calendar-dependent and are intentionally not part of `Duration`.
Each duration instance can be converted to number of any of these time units, as `Real` number, if `d` is a `Duration` Each duration instance can be converted to numbers in these units:
instance:
- `d.microseconds` - `d.microseconds`
- `d.milliseconds` - `d.milliseconds`
@@ -137,18 +198,16 @@ instance:
- `d.hours` - `d.hours`
- `d.days` - `d.days`
for example Example:
import lyng.time import lyng.time
assertEquals( 60, 1.minute.seconds )
assertEquals( 10.milliseconds, 0.01.seconds )
assertEquals(60, 1.minute.seconds)
assertEquals(10.milliseconds, 0.01.seconds)
>>> void >>> void
# Utility functions # Utility functions
## delay(duration: Duration) ## `delay(duration: Duration)`
Suspends current coroutine for at least the specified duration.
Suspends the current coroutine for at least the specified duration.
+52 -1
View File
@@ -352,6 +352,40 @@ Sets `this` to the first argument and executes the block. Returns the value retu
assertEquals(3, sum) assertEquals(3, sum)
>>> void >>> void
Receiver lambdas can also keep outer receivers in scope. The primary receiver wins for unqualified lookup, and `this@Type`
selects an outer receiver explicitly:
class Html { fun lang() = "en" }
class Body { fun lang() = "body" }
fun html(block: Html.()->String) = with(Html()) { block(this) }
fun body(block: Body.()->String) = with(Body()) { block(this) }
val result = html {
body {
lang() + ":" + this@Html.lang()
}
}
assertEquals("body:en", result)
>>> void
You can declare the same requirement in a function type:
val block: context(Html) Body.()->String = {
lang() + ":" + this@Html.lang()
}
If the primary receiver does not define a member and multiple outer/context receivers do, Lyng reports an ambiguity instead of picking one silently:
class A { fun title() = "a" }
class B { fun title() = "b" }
class C
val block: context(A, B) C.()->String = {
// title() // compile-time ambiguity
this@A.title()
}
## run ## run
Executes a block after it returning the value passed by the block. for example, can be used with elvis operator: Executes a block after it returning the value passed by the block. for example, can be used with elvis operator:
@@ -1654,15 +1688,27 @@ The type for the character objects is `Char`.
### String literal escapes ### String literal escapes
Lyng string literals can use either double quotes or backticks:
val a = "hello"
val b = `hello`
assert(a == b)
| escape | ASCII value | | escape | ASCII value |
|--------|-----------------------| |--------|-----------------------|
| \n | 0x10, newline | | \n | 0x10, newline |
| \r | 0x13, carriage return | | \r | 0x13, carriage return |
| \t | 0x07, tabulation | | \t | 0x07, tabulation |
| \\ | \ slash character | | \\ | \ slash character |
| \" | " double quote |
| \uXXXX | unicode code point | | \uXXXX | unicode code point |
Delimiter-specific escapes:
| form | escape | value |
|--------|--------|------------------|
| `"..."` | \" | " double quote |
| `` `...` `` | \` | ` backtick |
Unicode escape form is exactly 4 hex digits, e.g. `"\u263A"` -> `☺`. Unicode escape form is exactly 4 hex digits, e.g. `"\u263A"` -> `☺`.
Other `\c` combinations, where c is any char except mentioned above, are left intact, e.g.: Other `\c` combinations, where c is any char except mentioned above, are left intact, e.g.:
@@ -1695,10 +1741,15 @@ Example:
val name = "Lyng" val name = "Lyng"
assertEquals("hello, Lyng!", "hello, $name!") assertEquals("hello, Lyng!", "hello, $name!")
assertEquals("hello, Lyng!", `hello, $name!`)
assertEquals("sum=3", "sum=${1+2}") assertEquals("sum=3", "sum=${1+2}")
assertEquals("sum=3", `sum=${1+2}`)
assertEquals("\$name", "\$name") assertEquals("\$name", "\$name")
assertEquals("\$name", "$$name") assertEquals("\$name", "$$name")
assertEquals("\$name", `\$name`)
assertEquals("\$name", `$$name`)
assertEquals("\\Lyng", "\\$name") assertEquals("\\Lyng", "\\$name")
assertEquals("\\Lyng", `\\$name`)
>>> void >>> void
Interpolation and `printf`-style formatting can be combined when needed: Interpolation and `printf`-style formatting can be combined when needed:
+121 -9
View File
@@ -1,28 +1,140 @@
# What's New in Lyng # What's New in Lyng
This document highlights the current Lyng release, **1.5.4**, and the broader additions from the 1.5 cycle. This document highlights the current Lyng release, **1.5.9**, and the broader additions from the 1.5 cycle.
It is intentionally user-facing: new language features, new modules, new tools, and the practical things you can build with them. It is intentionally user-facing: new language features, new modules, new tools, and the practical things you can build with them.
For a programmer-focused migration summary across 1.5.x, see `docs/whats_new_1_5.md`. For a programmer-focused migration summary across 1.5.x, see `docs/whats_new_1_5.md`.
## Release 1.5.4 Highlights ## Release 1.5.9 Highlights
- `1.5.4` is the stabilization release for the 1.5 feature set. - `1.5.9` extends the 1.5 line with process execution helpers, a lightweight HTTP server, HTML rendering helpers, and better DSL support.
- The 1.5 line now brings together richer ranges and loops, interpolation, math modules, immutable and observable collections, richer `lyngio`, and much better CLI/IDE support. - The 1.5 line now brings together richer ranges and loops, interpolation, math modules, immutable and observable collections, richer `lyngio`, and much better CLI/IDE support.
- `1.5.4` specifically fixes user-visible issues around decimal arithmetic, mixed numeric flows, list behavior, and observable list hooks. - `1.5.9` adds `lyng.io.process` with `sh(...)`, `exec(...)`, and `CommandRun` for shell scripting and external process control.
- `1.5.4` also fixes extension-member registration for named singleton `object` declarations, so `fun X.foo()` and `val X.bar` now work as expected. - `1.5.9` adds the `lyng.io.http.server` module for exact, regex, and path-template routing, JSON helpers, WebSocket routes, and `respondHtml { ... }`.
- `1.5.4` also lets named singleton `object` declarations use scoped indexer extensions with bracket syntax, so patterns like `Storage["name"]` can be implemented with `override fun Storage.getAt(...)` / `putAt(...)`. - `1.5.9` adds the pure Lyng `lyng.io.html` builder DSL and language support for receiver-stack function types and context receiver extensions.
- `1.5.9` expands serialization with canonical `Json.encode(...)` / `Json.decode(...)` and typed `Json.encodeAs(...)` / `Json.decodeAs(...)`.
- The docs, homepage samples, and release metadata now point at the current stable version. - The docs, homepage samples, and release metadata now point at the current stable version.
## User Highlights Across 1.5.x ## User Highlights Across 1.5.x
- Descending ranges and loops with `downTo` / `downUntil` - Descending ranges and loops with `downTo` / `downUntil`
- String interpolation with `$name` and `${expr}` - String interpolation with `$name` and `${expr}`
- Backtick string literals for raw-ish string text
- Decimal arithmetic, matrices/vectors, and complex numbers - Decimal arithmetic, matrices/vectors, and complex numbers
- Calendar `Date` support in `lyng.time`
- `Channel`, `LaunchPool`, and `joinAll()` for coroutine workflows
- Immutable collections and opt-in `ObservableList` - Immutable collections and opt-in `ObservableList`
- Rich `lyngio` modules for console, HTTP, WebSocket, TCP, and UDP - Rich `lyngio` modules for SQLite/JDBC databases, process execution, console, HTTP, HTTP servers, WebSocket, TCP, UDP, and HTML rendering
- Legacy SHA-1 compatibility helpers in `lyng.legacy_digest`
- CLI improvements including the built-in formatter `lyng fmt` - CLI improvements including the built-in formatter `lyng fmt`
- Better IDE support and stronger docs around the released feature set - Better IDE support and stronger docs around the released feature set
## New in 1.5.9
### Process Execution (`lyng.io.process`)
Lyng scripts can now run external commands through a coroutine-friendly process API.
Use `sh(...)` for shell commands and `exec(...)` when command arguments come from data and should bypass shell parsing.
```lyng
import lyng.io.process
val branch = sh("git branch --show-current").out.trim()
println("Branch: " + branch)
exec("git", ["status", "--short"]).check()
for (file in sh("git ls-files").lines) {
if (file.endsWith(".lyng")) {
println(file)
}
}
```
`CommandRun` supports captured output (`out`, `err`), streaming line output (`lines`, `errorLines`), `wait()`, `check()`, and status helpers.
See [lyng.io.process](lyng.io.process.md).
### HTTP Server Routes and HTML Responses
`lyng.io.http.server` now provides a compact HTTP/1.1 server API for embedded tools, local services, test fixtures, and lightweight app backends.
Routes can match exact paths, regexes, or path templates with named parameters.
Handlers run with `RequestContext` as the receiver, so request data and response helpers are available directly.
```lyng
import lyng.io.http.server
val server = HttpServer()
server.getPath("/api/users/{id}") {
respondJson({
id: routeParams["id"],
path: request.path,
ok: true
})
}
server.listen(8080, "127.0.0.1")
```
The server also supports reusable routers, JSON body decoding with `jsonBody<T>()`, WebSocket routes, and `respondHtml { ... }`.
See [lyng.io.http.server](lyng.io.http.server.md).
### HTML Builder DSL (`lyng.io.html`)
`lyng.io.html` provides a pure Lyng HTML builder that escapes text and attributes by default.
It is designed for server-side rendering, generated reports, and small embedded UI surfaces.
```lyng
import lyng.io.html
val page = html {
head {
title { +"Demo" }
}
body {
nav {
a(href: "/") { +"Home" }
}
p { +"Text is escaped: <safe>" }
}
}
```
When used with the HTTP server, `respondHtml { ... }` renders the same DSL directly as a `text/html` response.
See [lyng.io.html](lyng.io.html.md).
### Receiver-Stack Function Types and Context Extensions
Lyng now supports receiver-stack function types such as `context(Html) Body.()->String`.
Nested receiver lambdas keep outer receivers in scope, `this@Type` can select a specific receiver, and ambiguous receiver lookup is reported at compile time.
```lyng
class Html { fun htmlOnly() = "html" }
class Body { fun bodyOnly() = "body" }
val block: context(Html) Body.()->String = {
this@Html.htmlOnly() + ":" + bodyOnly()
}
```
Context receiver extensions let APIs expose helpers only when the right implicit receiver is already active.
This is what makes HTML builder code like `+"text"` work inside tag blocks without global builder state.
See [Object Oriented Programming](OOP.md#receiver-stack-lambdas).
### Canonical and Typed JSON Serialization
The serialization layer now has separate APIs for ordinary JSON projection, self-describing canonical JSON, and schema-guided typed canonical JSON.
```lyng
import lyng.serialization
closed class Point(x: Int, y: Int)
closed class Segment(a: Point, b: Point)
val value = Segment(Point(0, 1), Point(2, 3))
val encoded = Json.encodeAs(Segment, value)
assertEquals("{\"a\":{\"x\":0,\"y\":1},\"b\":{\"x\":2,\"y\":3}}", encoded)
assertEquals(value, Json.decodeAs(Segment, encoded))
```
Use `toJson()` / `toJsonString()` for ordinary JSON interop, `Json.encode(...)` / `Json.decode(...)` for Lyng-to-Lyng round trips without a schema, and `Json.encodeAs(...)` / `Json.decodeAs(...)` when both sides already know the type.
See [Json support](json_and_kotlin_serialization.md).
## Language Features ## Language Features
### Descending Ranges and Loops ### Descending Ranges and Loops
@@ -323,7 +435,7 @@ Singleton objects are declared using the `object` keyword. They provide a conven
```lyng ```lyng
object Config { object Config {
val version = "1.5.4" val version = "1.5.9"
fun show() = println("Config version: " + version) fun show() = println("Config version: " + version)
} }
@@ -579,7 +691,7 @@ ws.close()
These modules are capability-gated and host-installed, keeping Lyng safe by default while making networked scripts practical when enabled. These modules are capability-gated and host-installed, keeping Lyng safe by default while making networked scripts practical when enabled.
See [lyngio overview](lyngio.md), [lyng.io.http](lyng.io.http.md), [lyng.io.ws](lyng.io.ws.md), and [lyng.io.net](lyng.io.net.md). See [lyngio overview](lyngio.md), [lyng.io.db](lyng.io.db.md), [lyng.io.http](lyng.io.http.md), [lyng.io.ws](lyng.io.ws.md), and [lyng.io.net](lyng.io.net.md).
### CLI: Formatting Command ### CLI: Formatting Command
A new `fmt` subcommand has been added to the Lyng CLI. A new `fmt` subcommand has been added to the Lyng CLI.
+325
View File
@@ -0,0 +1,325 @@
#!/usr/bin/env lyng
import lyng.io.db
import lyng.io.db.sqlite
import lyng.io.fs
val DB_FILE_NAME = "contents.db"
val ANSI_ESC = "\u001b["
val NEWLINE = "\n"
val WINDOWS_SEPARATOR = "\\"
val SQLITE_JOURNAL_SUFFIXES = ["-wal", "-shm", "-journal"]
val USAGE_TEXT = "
Lyng content index
Scan a directory tree, diff it against a SQLite snapshot, and optionally refresh the snapshot.
usage:
lyng examples/content_index_db.lyng <root> [-u|--update]
options:
-u, --update write the current scan back to $DB_FILE_NAME
notes:
- the database lives inside <root>/$DB_FILE_NAME
- on first run the snapshot is created automatically
- the script ignores its own SQLite sidecar files
"
val CREATE_FILE_INDEX_SQL = "
create table if not exists file_index(
path text primary key not null,
size integer not null,
mtime integer not null
)
"
val CREATE_CURRENT_SCAN_SQL = "
create temp table current_scan(
path text primary key not null,
size integer not null,
mtime integer not null
)
"
val SELECT_ADDED_SQL = "
select
c.path,
c.size,
c.mtime
from current_scan c
left join file_index f on f.path = c.path
where f.path is null
order by c.path
"
val SELECT_REMOVED_SQL = "
select f.path, f.size, f.mtime
from file_index f
left join current_scan c on c.path = f.path
where c.path is null
order by f.path
"
val SELECT_CHANGED_SQL = "
select c.path, f.size as old_size, c.size as new_size, f.mtime as old_mtime,
c.mtime as new_mtime
from current_scan c
join file_index f on f.path = c.path
where c.size != f.size or c.mtime != f.mtime
order by c.path
"
val DELETE_MISSING_SQL = "
delete from file_index
where not exists (
select 1
from current_scan c
where c.path = file_index.path
)
"
val UPSERT_SCAN_SQL = "
insert or replace into file_index(path, size, mtime)
select path, size, mtime
from current_scan
"
val INSERT_SCAN_ROW_SQL = "
insert into current_scan(path, size, mtime)
values(?, ?, ?)
"
val USE_COLOR = true
class CliOptions(val rootText: String, val updateSnapshot: Bool) {}
fun out(text: String? = null): Void {
if (text == null) {
print(NEWLINE)
return
}
print(text + NEWLINE)
}
fun paint(code: String, text: String): String {
if (!USE_COLOR) return text
ANSI_ESC + code + "m" + text + ANSI_ESC + "0m"
}
fun bold(text: String): String = paint("1", text)
fun dim(text: String): String = paint("2", text)
fun cyan(text: String): String = paint("36", text)
fun green(text: String): String = paint("32", text)
fun yellow(text: String): String = paint("33", text)
fun red(text: String): String = paint("31", text)
fun signed(value: Int): String = if (value > 0) "+" + value else value.toString()
fun plural(count: Int, one: String, many: String): String {
if (count == 1) return one
many
}
fun childPath(parent: Path, name: String): Path {
val base = parent.toString()
if (base.endsWith("/") || base.endsWith(WINDOWS_SEPARATOR)) {
return Path(base + name)
}
Path(base + "/" + name)
}
fun relativePath(root: Path, file: Path): String {
val parts: List<String> = []
for (i in root.segments.size..<file.segments.size) {
parts.add(file.segments[i] as String)
}
parts.joinToString("/")
}
fun isDatabaseArtifact(relative: String): Bool {
relative == DB_FILE_NAME || SQLITE_JOURNAL_SUFFIXES.any { relative == DB_FILE_NAME + (it as String) }
}
fun printUsage(message: String? = null): Void {
if (message != null && message.trim().isNotEmpty()) {
out(red("error: ") + message)
out()
}
out(bold(USAGE_TEXT))
}
fun parseArgs(argv: List<String>): CliOptions? {
var rootText: String? = null
var updateSnapshot = false
for (arg in argv) {
when (arg) {
"-u", "--update" -> updateSnapshot = true
"-h", "--help" -> {
printUsage()
return null
}
else -> {
if (arg.startsWith("-")) {
printUsage("unknown option: " + arg)
return null
}
if (rootText != null) {
printUsage("only one root path is allowed")
return null
}
rootText = arg
}
}
}
if (rootText == null) {
printUsage("missing required <root> argument")
return null
}
CliOptions(rootText as String, updateSnapshot)
}
fun printBanner(root: Path, dbFile: Path, dbWasCreated: Bool, updateSnapshot: Bool): Void {
val mode =
if (dbWasCreated) "bootstrap snapshot"
else if (updateSnapshot) "scan + refresh snapshot"
else "scan only"
out(cyan("== Lyng content index =="))
out(dim("root: " + root))
out(dim("db: " + dbFile))
out(dim("mode: " + mode))
out()
}
fun printSection(title: String, accent: (String)->String, rows: List<SqlRow>, render: (SqlRow)->String): Void {
out(accent(title + " (" + rows.size + ")"))
if (rows.isEmpty()) {
out(dim(" none"))
out()
return
}
for (row in rows) {
out(render(row))
}
out()
}
fun renderAdded(row: SqlRow): String {
val path = row["path"] as String
val size = row["size"] as Int
val mtime = row["mtime"] as Int
" " + green("+") + " " + bold(path) + dim(" %12d B mtime %d"(size, mtime))
}
fun renderRemoved(row: SqlRow): String {
val path = row["path"] as String
val size = row["size"] as Int
val mtime = row["mtime"] as Int
" " + red("-") + " " + bold(path) + dim(" %12d B mtime %d"(size, mtime))
}
fun renderChanged(row: SqlRow): String {
val path = row["path"] as String
val oldSize = row["old_size"] as Int
val newSize = row["new_size"] as Int
val oldMtime = row["old_mtime"] as Int
val newMtime = row["new_mtime"] as Int
val sizeDelta = newSize - oldSize
val mtimeDelta = newMtime - oldMtime
" " + yellow("~") + " " + bold(path) +
dim(
" size %d -> %d (%s B), mtime %d -> %d (%s ms)"(
oldSize,
newSize,
signed(sizeDelta),
oldMtime,
newMtime,
signed(mtimeDelta)
)
)
}
fun loadRows(tx: SqlTransaction, query: String): List<SqlRow> = tx.select(query).toList()
fun main() {
val argv: List<String> = []
for (raw in ARGV as List) {
argv.add(raw as String)
}
val options = parseArgs(argv)
if (options == null) {
return
}
val root = Path(options.rootText)
if (!root.exists()) {
printUsage("root does not exist: " + root)
return
}
if (!root.isDirectory()) {
printUsage("root is not a directory: " + root)
return
}
val dbFile = childPath(root, DB_FILE_NAME)
val dbWasCreated = !dbFile.exists()
val shouldUpdateSnapshot = dbWasCreated || options.updateSnapshot
printBanner(root, dbFile, dbWasCreated, shouldUpdateSnapshot)
val db = openSqlite(dbFile.toString())
db.transaction { tx ->
tx.execute(CREATE_FILE_INDEX_SQL)
tx.execute("drop table if exists temp.current_scan")
tx.execute(CREATE_CURRENT_SCAN_SQL)
var scannedFiles = 0
for (rawEntry in root.glob("**")) {
val entry = rawEntry as Path
if (!entry.isFile()) continue
val relative = relativePath(root, entry)
if (isDatabaseArtifact(relative)) continue
val size = entry.size() ?: 0
val mtime = entry.modifiedAtMillis() ?: 0
tx.execute(INSERT_SCAN_ROW_SQL, relative, size, mtime)
scannedFiles++
}
val added = loadRows(tx, SELECT_ADDED_SQL)
val removed = loadRows(tx, SELECT_REMOVED_SQL)
val changed = loadRows(tx, SELECT_CHANGED_SQL)
val totalChanges = added.size + removed.size + changed.size
out(dim("scanned %d %s under %s"(scannedFiles, plural(scannedFiles, "file", "files"), root.toString())))
out(dim("detected %d %s"(totalChanges, plural(totalChanges, "change", "changes"))))
out()
printSection("Added", { green(it) }, added) { renderAdded(it) }
printSection("Removed", { red(it) }, removed) { renderRemoved(it) }
printSection("Changed", { yellow(it) }, changed) { renderChanged(it) }
if (shouldUpdateSnapshot) {
tx.execute(DELETE_MISSING_SQL)
tx.execute(UPSERT_SCAN_SQL)
val action = if (dbWasCreated) "created" else "updated"
out(cyan("snapshot " + action + " in " + dbFile.name))
} else {
out(dim("snapshot unchanged; re-run with -u or --update to persist the scan"))
}
}
}
main()
+15
View File
@@ -0,0 +1,15 @@
import lyng.time
val n = 700_000
fun tm<T>(block: ()->T): T {
val t = Instant()
block().also {
println("tm: ${Instant() - t}")
}
}
val x = tm { List.fill(n) { it * 10 + 1 } }
val y = tm { List.fill(n, n + 10) { it * 10 + 1 } }
tm { x.add(-1) }
tm { y.add(-2) }
+43
View File
@@ -0,0 +1,43 @@
import lyng.io.db
import lyng.io.db.jdbc
println("H2 JDBC demo: typed open, generic open, generated keys")
val db = openH2("mem:lyng_h2_demo;DB_CLOSE_DELAY=-1")
db.transaction { tx ->
tx.execute("create table if not exists person(id bigint auto_increment primary key, name varchar(120) not null, active boolean not null)")
tx.execute("delete from person")
val firstInsert = tx.execute(
"insert into person(name, active) values(?, ?)",
"Ada",
true
)
val firstId = firstInsert.getGeneratedKeys().toList()[0][0]
assertEquals(1, firstId)
tx.execute(
"insert into person(name, active) values(?, ?)",
"Linus",
false
)
val rows = tx.select("select id, name, active from person order by id").toList()
assertEquals(2, rows.size)
println("#" + rows[0]["id"] + " " + rows[0]["name"] + " active=" + rows[0]["active"])
println("#" + rows[1]["id"] + " " + rows[1]["name"] + " active=" + rows[1]["active"])
}
val genericDb = openDatabase(
"jdbc:h2:mem:lyng_h2_generic;DB_CLOSE_DELAY=-1",
Map()
)
val answer = genericDb.transaction { tx ->
tx.select("select 42 as answer").toList()[0]["answer"]
}
assertEquals(42, answer)
println("Generic JDBC openDatabase(...) also works: answer=$answer")
println("OK")
+19
View File
@@ -0,0 +1,19 @@
import lyng.io.http.server
closed class CreateUserRequest(name: String, age: Int)
closed class CreateUserResponse(id: Int, name: String, age: Int)
val server = HttpServer()
server.postPath("/api/users") {
val req = jsonBody<CreateUserRequest>()
if (req.name.isBlank()) {
respondJson({ error: "name must not be empty" }, 400)
return
}
respondJson(CreateUserResponse(101, req.name, req.age), 201)
}
server.listen(8080, "127.0.0.1")
+71
View File
@@ -0,0 +1,71 @@
import lyng.io.db.jdbc
/*
PostgreSQL JDBC demo.
Usage:
lyng examples/postgres_basic.lyng [jdbc-url] [user] [password]
Typical local URL:
jdbc:postgresql://127.0.0.1/postgres
*/
fun cliArgs(): List<String> {
val result: List<String> = []
for (raw in ARGV as List) {
result.add(raw as String)
}
return result
}
val argv = cliArgs()
val URL = if (argv.size > 0) argv[0] else "jdbc:postgresql://127.0.0.1/postgres"
val USER = if (argv.size > 1) argv[1] else ""
val PASSWORD = if (argv.size > 2) argv[2] else ""
println("PostgreSQL JDBC demo: typed open, generated keys, nested transaction")
val db = openPostgres(URL, USER, PASSWORD)
db.transaction { tx ->
tx.execute("create table if not exists lyng_pg_demo(id bigserial primary key, title text not null, done boolean not null)")
tx.execute("delete from lyng_pg_demo")
val firstInsert = tx.execute(
"insert into lyng_pg_demo(title, done) values(?, ?)",
"Verify PostgreSQL JDBC support",
false
)
val firstId = firstInsert.getGeneratedKeys().toList()[0][0]
println("First generated id=" + firstId)
tx.execute(
"insert into lyng_pg_demo(title, done) values(?, ?)",
"Review documentation",
true
)
try {
tx.transaction { inner ->
inner.execute(
"insert into lyng_pg_demo(title, done) values(?, ?)",
"This row is rolled back",
false
)
throw IllegalStateException("rollback nested")
}
} catch (_: IllegalStateException) {
println("Nested transaction rolled back as expected")
}
val rows = tx.select("select id, title, done from lyng_pg_demo order by id").toList()
for (row in rows) {
println("#" + row["id"] + " " + row["title"] + " done=" + row["done"])
}
val count = tx.select("select count(*) as count from lyng_pg_demo").toList()[0]["count"]
assertEquals(2, count)
println("Visible rows after nested rollback: " + count)
}
println("OK")
+89
View File
@@ -0,0 +1,89 @@
import lyng.io.db
import lyng.io.db.sqlite
import lyng.time
println("SQLite demo: typed open, generic open, result sets, generated keys, nested rollback")
// The typed helper is the simplest entry point when you know you want SQLite.
val db = openSqlite(":memory:")
db.transaction { tx ->
// Keep schema creation and data changes inside one transaction block.
tx.execute("create table task(id integer primary key autoincrement, title text not null, done integer not null, due_date date not null)")
// execute(...) is for side-effect statements. Generated keys are read from
// ExecutionResult rather than from a synthetic row-returning INSERT.
val firstInsert = tx.execute(
"insert into task(title, done, due_date) values(?, ?, ?)",
"Write a SQLite example",
false,
Date(2026, 4, 15)
)
val firstGeneratedKeys = firstInsert.getGeneratedKeys()
val firstId = firstGeneratedKeys.toList()[0][0]
assertEquals(1, firstId)
tx.execute(
"insert into task(title, done, due_date) values(?, ?, ?)",
"Review the DB API",
true,
Date(2026, 4, 16)
)
// Nested transactions are real savepoints. If the inner block fails,
// only the nested work is rolled back.
try {
tx.transaction { inner ->
inner.execute(
"insert into task(title, done, due_date) values(?, ?, ?)",
"This row is rolled back",
false,
Date(2026, 4, 17)
)
throw IllegalStateException("demonstrate nested rollback")
}
} catch (_: IllegalStateException) {
println("Nested transaction rolled back as expected")
}
// select(...) is for row-producing statements. ResultSet exposes metadata,
// cheap emptiness checks, iteration, and conversion to a plain list.
val tasks = tx.select("select id, title, done, due_date from task order by id")
assertEquals(false, tasks.isEmpty())
assertEquals(2, tasks.size())
println("Columns:")
for (column in tasks.columns) {
println(" " + column.name + " -> " + column.sqlType + " (native " + column.nativeType + ")")
}
val taskRows = tasks.toList()
println("Rows:")
for (row in taskRows) {
// Name lookups are case-insensitive and values are already converted.
println(" #" + row["ID"] + " " + row["title"] + " done=" + row["done"] + " due=" + row["due_date"])
}
// toList() materializes detached rows that stay usable after transaction close.
val snapshot = tx.select("select title, due_date from task order by id").toList()
assertEquals("Write a SQLite example", snapshot[0]["title"])
assertEquals(Date(2026, 4, 16), snapshot[1]["due_date"])
val count = tx.select("select count(*) as count from task").toList()[0]["count"]
assertEquals(2, count)
println("Visible rows after nested rollback: $count")
}
// The generic entry point stays useful for config-driven code.
val genericDb = openDatabase(
"sqlite::memory:",{ foreignKeys: true, busyTimeoutMillis: 1000 }
)
val answer = genericDb.transaction { tx ->
tx.select("select 42 as answer").toList()[0]["answer"]
}
assertEquals(42, answer)
println("Generic openDatabase(...) also works: answer=$answer")
println("OK")
+56
View File
@@ -0,0 +1,56 @@
import lyng.io.db.sqlite
println("SQLite serialization demo: write-side projection and decodeAs<T>()")
class Payload(name: String, count: Int)
class Item(
id: Int,
title: String,
@DbJson meta: Payload,
@DbLynon state: Payload
) {
var note: String = ""
@DbExcept var cache: String = ""
}
val restored = openSqlite(":memory:").transaction { tx ->
tx.execute(
"create table item(id integer not null, title text not null, meta text not null, state blob not null, note text not null)"
)
val item = Item(1, "first", Payload("json", 10), Payload("bin", 20))
item.note = "created"
item.cache = "not stored"
tx.execute("insert into item(@cols(?1)) values(@vals(?1))", item)
item.title = "second"
item.meta = Payload("json2", 11)
item.state = Payload("bin2", 21)
item.note = "updated"
tx.execute(
"update item set @set(?1 except: \"id\") where id = ?2",
item,
item.id
)
val restored = tx.select("select * from item where id = ?", 1).decodeAs<Item>().first
assertEquals("second", restored.title)
assertEquals("json2", restored.meta.name)
assertEquals(11, restored.meta.count)
assertEquals("bin2", restored.state.name)
assertEquals(21, restored.state.count)
assertEquals("updated", restored.note)
restored
}
println("Restored item:")
println(" id=" + restored.id)
println(" title=" + restored.title)
println(" meta=" + restored.meta.name + "/" + restored.meta.count)
println(" state=" + restored.state.name + "/" + restored.state.count)
println(" note=" + restored.note)
println("OK")
+1 -6
View File
@@ -636,13 +636,10 @@ if (!Console.isSupported()) {
} }
} }
var inputRunning = true
launch { launch {
while (inputRunning) { while (true) {
try { try {
for (ev in Console.events()) { for (ev in Console.events()) {
if (!inputRunning) break
// Isolate per-event failures so one bad event does not unwind the stream. // Isolate per-event failures so one bad event does not unwind the stream.
try { try {
if (ev is ConsoleKeyEvent) { if (ev is ConsoleKeyEvent) {
@@ -675,7 +672,6 @@ if (!Console.isSupported()) {
} }
} catch (err: Object) { } catch (err: Object) {
// Recover stream-level failures by recreating event stream in next loop turn. // Recover stream-level failures by recreating event stream in next loop turn.
if (!inputRunning) break
logError("Input stream recovered after error", err) logError("Input stream recovered after error", err)
Console.setRawMode(true) Console.setRawMode(true)
delay(50) delay(50)
@@ -803,7 +799,6 @@ if (!Console.isSupported()) {
delay(FRAME_DELAY_MS) delay(FRAME_DELAY_MS)
} }
} finally { } finally {
inputRunning = false
Console.setRawMode(false) Console.setRawMode(false)
Console.setCursorVisible(true) Console.setCursorVisible(true)
Console.leaveAltScreen() Console.leaveAltScreen()
+2 -1
View File
@@ -16,7 +16,8 @@
# #
#Gradle #Gradle
org.gradle.jvmargs=-Xmx4096M -Dfile.encoding=UTF-8 -Dkotlin.daemon.jvm.options\="-Xmx2048M" org.gradle.jvmargs=-Xmx4096M -Dfile.encoding=UTF-8 -Dkotlin.daemon.jvm.options\="-Xmx2048M" --add-opens=java.base/java.lang=ALL-UNNAMED
org.gradle.java.home=/usr/lib/jvm/java-17-openjdk-amd64
org.gradle.caching=true org.gradle.caching=true
org.gradle.configuration-cache=true org.gradle.configuration-cache=true
#Kotlin #Kotlin
+14 -1
View File
@@ -2,7 +2,7 @@
agp = "8.5.2" agp = "8.5.2"
clikt = "5.0.3" clikt = "5.0.3"
mordant = "3.0.2" mordant = "3.0.2"
kotlin = "2.3.20" kotlin = "2.4.0"
android-minSdk = "24" android-minSdk = "24"
android-compileSdk = "34" android-compileSdk = "34"
kotlinx-coroutines = "1.10.2" kotlinx-coroutines = "1.10.2"
@@ -15,6 +15,11 @@ okioVersion = "3.10.2"
compiler = "3.2.0-alpha11" compiler = "3.2.0-alpha11"
ktor = "3.3.1" ktor = "3.3.1"
slf4j = "2.0.17" slf4j = "2.0.17"
sqlite-jdbc = "3.50.3.0"
h2 = "2.4.240"
postgresql = "42.7.8"
testcontainers = "1.20.6"
hikaricp = "6.2.1"
[libraries] [libraries]
clikt = { module = "com.github.ajalt.clikt:clikt", version.ref = "clikt" } clikt = { module = "com.github.ajalt.clikt:clikt", version.ref = "clikt" }
@@ -43,8 +48,16 @@ ktor-client-winhttp = { module = "io.ktor:ktor-client-winhttp", version.ref = "k
ktor-client-websockets = { module = "io.ktor:ktor-client-websockets", version.ref = "ktor" } ktor-client-websockets = { module = "io.ktor:ktor-client-websockets", version.ref = "ktor" }
ktor-network = { module = "io.ktor:ktor-network", version.ref = "ktor" } ktor-network = { module = "io.ktor:ktor-network", version.ref = "ktor" }
slf4j-nop = { module = "org.slf4j:slf4j-nop", version.ref = "slf4j" } slf4j-nop = { module = "org.slf4j:slf4j-nop", version.ref = "slf4j" }
sqlite-jdbc = { module = "org.xerial:sqlite-jdbc", version.ref = "sqlite-jdbc" }
h2 = { module = "com.h2database:h2", version.ref = "h2" }
postgresql = { module = "org.postgresql:postgresql", version.ref = "postgresql" }
testcontainers = { module = "org.testcontainers:testcontainers", version.ref = "testcontainers" }
testcontainers-postgresql = { module = "org.testcontainers:postgresql", version.ref = "testcontainers" }
hikaricp = { module = "com.zaxxer:HikariCP", version.ref = "hikaricp" }
[plugins] [plugins]
androidLibrary = { id = "com.android.library", version.ref = "agp" } androidLibrary = { id = "com.android.library", version.ref = "agp" }
kotlinMultiplatform = { id = "org.jetbrains.kotlin.multiplatform", version.ref = "kotlin" } kotlinMultiplatform = { id = "org.jetbrains.kotlin.multiplatform", version.ref = "kotlin" }
kotlinSerialization = { id = "org.jetbrains.kotlin.plugin.serialization", version.ref = "kotlin" }
kotlinCompose = { id = "org.jetbrains.kotlin.plugin.compose", version.ref = "kotlin" }
vanniktech-mavenPublish = { id = "com.vanniktech.maven.publish", version = "0.29.0" } vanniktech-mavenPublish = { id = "com.vanniktech.maven.publish", version = "0.29.0" }
+2
View File
@@ -0,0 +1,2 @@
[vps]
94.130.36.94 ansible_user=sergeych
+100
View File
@@ -0,0 +1,100 @@
---
- name: Setup lynglang.com static site on VPS
hosts: vps
become: yes
vars:
domain: lynglang.com
web_root: /var/www/lynglang
deploy_user: sergeych
certbot_email: real.sergeych@gmail.com
tasks:
# Debian 10 buster is EOL; security/backports repos moved to archive.debian.org
- name: Fix sources.list for Debian buster EOL
copy:
dest: /etc/apt/sources.list
content: |
deb http://archive.debian.org/debian/ buster main contrib non-free
deb http://archive.debian.org/debian-security/ buster/updates main contrib non-free
deb http://archive.debian.org/debian/ buster-backports main contrib non-free
- name: Remove stale third-party sources (broken for buster EOL)
file:
path: "/etc/apt/sources.list.d/{{ item }}"
state: absent
loop:
- cassandra.list
- icinga.list
- postgres.list
- salt-stack.list
- yarn.list
- name: Install nginx, certbot, and python3-certbot-nginx
apt:
name:
- nginx
- certbot
- python3-certbot-nginx
state: present
update_cache: yes
- name: Create web root directory
file:
path: "{{ web_root }}/release/dist"
state: directory
owner: "{{ deploy_user }}"
group: www-data
mode: "0755"
recurse: yes
- name: Create distributables directory
file:
path: "{{ web_root }}/release/dist/distributables"
state: directory
owner: "{{ deploy_user }}"
group: www-data
mode: "0755"
- name: Deploy nginx site config (HTTP, pre-certbot)
template:
src: templates/nginx_lynglang.conf.j2
dest: /etc/nginx/sites-available/{{ domain }}
notify: reload nginx
- name: Enable nginx site
file:
src: /etc/nginx/sites-available/{{ domain }}
dest: /etc/nginx/sites-enabled/{{ domain }}
state: link
notify: reload nginx
- name: Disable default nginx site
file:
path: /etc/nginx/sites-enabled/default
state: absent
notify: reload nginx
- name: Ensure nginx is started
service:
name: nginx
state: started
enabled: yes
- name: Reload nginx before certbot
meta: flush_handlers
- name: Obtain SSL certificate via certbot (--nginx plugin)
command: >
certbot --nginx
-d {{ domain }} -d www.{{ domain }}
--non-interactive --agree-tos
--email {{ certbot_email }}
--redirect
args:
creates: /etc/letsencrypt/live/{{ domain }}/fullchain.pem
handlers:
- name: reload nginx
service:
name: nginx
state: reloaded
+24
View File
@@ -0,0 +1,24 @@
server {
listen 80;
server_name {{ domain }} www.{{ domain }};
root {{ web_root }}/release/dist;
index index.html;
# SPA fallback
location / {
try_files $uri $uri/ /index.html;
}
# Distributables served directly
location /distributables/ {
try_files $uri =404;
autoindex on;
}
# Long-lived cache for hashed assets
location ~* \.(js|css|woff2?|ttf|eot|svg|png|jpg|ico)$ {
expires 1y;
add_header Cache-Control "public, immutable";
}
}
+74
View File
@@ -0,0 +1,74 @@
### Ticket 1 — Name resolution: a member function named like its class silently shadows the primary constructor inside the class body
**Summary.** `class` bodies accept a member function whose name equals the class name. That member then wins name resolution for **all unqualified calls inside the class body**, including `init` blocks, method bodies and nested lambdas. The primary constructor becomes unreachable from inside the class, with no escape hatch. Compiles cleanly, breaks at runtime.
**Repro**
```lyng
class C1(x) {
fun C1(y): String = "member " + y.toString()
init { println(C1(9)) } // -> "member C1(9)", NOT the constructor
}
class C2(x) {
fun C2(y, z): String = y.toString() + z.toString()
fun probe(): String = C2(7, 8) // member chosen although ctor takes 1 arg
}
class Fact(x) {
fun Fact(y): Fact = Fact(y * 2) // compiles; unqualified Fact(...) = itself
}
Fact(1).Fact(3) // java.lang.StackOverflowError
```
Local functions shadow too:
```lyng
class M3(x) {
fun probe(y): String {
fun M3(z): String = "local " + z.toString()
return M3(y) // local member wins, ctor unreachable
}
}
```
No workaround found:
```lyng
class Q1(x) {
fun Q1(y): String = "member"
fun a(): String = Q1.Q1(3) // IllegalArgumentException: expected 1 arguments, got 0
fun b(): String = (Q1(5) as Q1).Q1(6) // ClassCastException: Cannot cast String to Q1
}
class G1(x) { fun g(): G1 = this(x + 1) } // Exception: operation is not implemented
```
Related symptom: adding `Type::class` in a class that also has a same-name member turns the **declaration** itself into a hard error pointing at the wrong place:
```lyng
class D16(x) {
fun D16(y): String = "member " + y.toString() // Error: expected 1 arguments, got 0
fun lit(): String = D16::class.toString()
}
```
Control cases compile fine: same-name member alone ✔, `::class` alone ✔, the combination ✘.
**Expected.** Either (a) reject a member whose name collides with the class name at compile time (preferred — it is never a constructor in Lyng, there are no secondary constructors), or (b) keep constructor calls resolvable for the primary arity and fall back to members only for non-matching arity, plus a working `this(...)`/`this@Type(...)` delegation. In any case `Type::class` must not be affected by a same-name member, and diagnostics must point at the actual construct.
**Suggested tests.** `fun SameName(...)` declared in a class with the same name → expected compile error; self-recursive factory → must not reach runtime stack overflow; `Type::class` inside such a class → must compile; `this(...)` → either implement or emit a clear compile-time error instead of `operation is not implemented`.
## Resolution
Valid: direct instance/static member functions named after their enclosing class
are now rejected at the function name. This also rejects the collision before
any `::class` expression can produce a misleading downstream diagnostic.
The supported control is `Type::class` in a class without the illegal member;
the suggested test asking the illegal declaration to compile contradicts the
report's preferred resolution.
Direct `this(...)` / `this@Type(...)` calls on the enclosing class instance now
report unsupported constructor delegation during compilation. Use `Type(...)`
to construct another instance; no secondary constructors were introduced.
Local functions shadowing class names are ordinary lexical shadowing, not a bug,
and remain supported. Regression coverage: `ClassNameMemberRegressionTest`.
Before the fix, 2 of its 4 tests failed; the constructor/class-literal and local
shadowing controls passed.
Validation: all 4 focused JVM regression tests passed, followed by the full
`:lynglib:jvmTest` suite (benchmarks excluded by the repository default).
+65
View File
@@ -0,0 +1,65 @@
### Ticket 2 — Static member calls: declared return type is not propagated, receiver becomes `Object`
**Summary.** The **invocation** of a `static` member (class or `object` singleton) loses its declared return type at the call site; the result is treated as a general receiver, so every type-specific member access fails to compile. The runtime value is correct — this is purely a static-typing defect. Constructor calls, instance-method calls and top-level function calls all preserve the declared type correctly, so the inconsistency is specific to static members.
**Repro**
```lyng
class S(x) {
static fun make(y): S = S(y) // return type is explicitly declared
fun hi(): String = "hi"
}
val a: S = S.make(1) // OK — annotation rescues it
println(a.hi())
val b = S.make(2) // b is S == true at runtime
println(b.hi()) // Error: member access requires compile-time receiver type: hi
println(S.make(1).hi())// Error: same, chained call
println((S.make(3) as S).hi()) // OK — explicit cast rescues it
fun need(v: S) = v.hi(); need(S.make(4)) // OK — typed argument position rescues it
```
Not limited to self-returning factories — any declared return type is dropped:
```lyng
class K(x) { static fun text(y): String = "t"; static fun num(y): Int = y }
val s = K.text(1); println(s.length) // Error: ... compile-time receiver type: length
val n = K.num(1); println(n + 1) // same class of failure
object O { static fun text(y): String = "o" }
val o = O.text(1); println(o.length) // Error: ... on O.text result
class B(x) { static fun noRet(y) = B(y) } // untyped static: also general
class Box<T>(val value: T) { static fun of<T>(v: T): Box<T> = Box(v) }
val bx = Box.of(42); println(bx.value) // Error; val bx2: Box<Int> = Box.of(43) works
```
Controls that **do** work correctly: `val d = K8(9); d.hi()` (constructor), `val c = S(3).again(4); c.hi()` (instance method), `val t = topLevelMake(1); t.x` (top-level function).
**Expected.** `StaticClass.member(args)` must produce the declared return type as the static compile-time receiver type, exactly like instance and top-level calls. Only genuinely untyped statics (`fun` without a return type and uninferrable bodies) should fall back to `Object`/`Obj`.
**Suggested tests.** Matrix over `{constructor | instance method | top-level fun | static fun} × {declared ret type | inferred | omitted}` asserting that `.length`, arithmetic, `is T`-then-member, chained calls and generic instantiation behave identically for statics as for the other three; plus `object` singletons and generic statics.
## Resolution
Valid: static method signatures were excluded from the compiler's member type
metadata, both before body compilation and when inferred returns were recorded.
They now use the same callable signature and generic substitution path as
instance methods. Explicit annotations, inferred expression returns, primitive
returns, chained calls, and inferred/explicit generic arguments are covered by
`StaticReturnTypeRegressionTest`.
The singleton example additionally exposed runtime bugs once typing was
corrected: object static initializers were discarded, and a static call on an
object was sent to instance-method dispatch. Object declarations now preserve
and execute their static initialization statements before constructing the
singleton instance.
The compiler now retains its class-scope call decision for bytecode generation;
singleton static calls use the object's class and the existing class-scope
access opcode. Instance methods retain their dispatch path. Tests also cover
static calls with a trailing lambda.
Before the fix, all 5 original regression cases failed with missing compile-time
receiver types. The initial type-only correction passed 4, with the singleton
case then failing at runtime; dispatch is corrected in this same report.
Validation: all 6 focused JVM regression tests passed, followed by all 1,040
`:lynglib:jvmTest` tests (benchmarks excluded by the repository default).
`git diff --check` also passed.
+5
View File
@@ -0,0 +1,5 @@
# Invalid reports
Place reports here when reproduction and the language contract show that they
are not bugs. Append the reason and verification evidence to each moved report.
Valid reports with an invalid subclaim stay with their resolution notes.
@@ -310,7 +310,7 @@ class LyngDocumentationProvider : AbstractDocumentationProvider() {
// Try literal and call-based receiver inference around the dot // Try literal and call-based receiver inference around the dot
val i = TextCtx.prevNonWs(text, dotPos - 1) val i = TextCtx.prevNonWs(text, dotPos - 1)
val className: String? = when { val className: String? = when {
i >= 0 && text[i] == '"' -> "String" i >= 0 && (text[i] == '"' || text[i] == '`') -> "String"
i >= 0 && text[i] == ']' -> "List" i >= 0 && text[i] == ']' -> "List"
i >= 0 && text[i] == '}' -> "Dict" i >= 0 && text[i] == '}' -> "Dict"
i >= 0 && text[i] == ')' -> { i >= 0 && text[i] == ')' -> {
@@ -24,6 +24,7 @@ import com.intellij.psi.codeStyle.CodeStyleManager
import com.intellij.psi.impl.source.codeStyle.PreFormatProcessor import com.intellij.psi.impl.source.codeStyle.PreFormatProcessor
import net.sergeych.lyng.format.LyngFormatConfig import net.sergeych.lyng.format.LyngFormatConfig
import net.sergeych.lyng.format.LyngFormatter import net.sergeych.lyng.format.LyngFormatter
import net.sergeych.lyng.format.LyngStringDelimiterPolicy
import net.sergeych.lyng.idea.LyngLanguage import net.sergeych.lyng.idea.LyngLanguage
/** /**
@@ -170,6 +171,7 @@ class LyngPreFormatProcessor : PreFormatProcessor {
continuationIndentSize = options.CONTINUATION_INDENT_SIZE.coerceAtLeast(options.INDENT_SIZE.coerceAtLeast(1)), continuationIndentSize = options.CONTINUATION_INDENT_SIZE.coerceAtLeast(options.INDENT_SIZE.coerceAtLeast(1)),
applySpacing = true, applySpacing = true,
applyWrapping = false, applyWrapping = false,
stringDelimiterPolicy = LyngStringDelimiterPolicy.PreferFewerEscapes,
) )
val r = if (runFullFileIndent) currentLocalRange() else workingRangeLocal.intersection(currentLocalRange()) ?: currentLocalRange() val r = if (runFullFileIndent) currentLocalRange() else workingRangeLocal.intersection(currentLocalRange()) ?: currentLocalRange()
val text = doc.getText(r) val text = doc.getText(r)
@@ -189,6 +191,7 @@ class LyngPreFormatProcessor : PreFormatProcessor {
continuationIndentSize = options.CONTINUATION_INDENT_SIZE.coerceAtLeast(options.INDENT_SIZE.coerceAtLeast(1)), continuationIndentSize = options.CONTINUATION_INDENT_SIZE.coerceAtLeast(options.INDENT_SIZE.coerceAtLeast(1)),
applySpacing = settings.enableSpacing, applySpacing = settings.enableSpacing,
applyWrapping = true, applyWrapping = true,
stringDelimiterPolicy = LyngStringDelimiterPolicy.PreferFewerEscapes,
) )
val r = if (runFullFileIndent) currentLocalRange() else workingRangeLocal.intersection(currentLocalRange()) ?: currentLocalRange() val r = if (runFullFileIndent) currentLocalRange() else workingRangeLocal.intersection(currentLocalRange()) ?: currentLocalRange()
val text = doc.getText(r) val text = doc.getText(r)
@@ -101,8 +101,8 @@ class LyngLexer : LexerBase() {
return return
} }
// String "..." or '...' with simple escape handling // String "...", `...`, or '...' with simple escape handling
if (ch == '"' || ch == '\'') { if (ch == '"' || ch == '\'' || ch == '`') {
val quote = ch val quote = ch
i++ i++
while (i < endOffset) { while (i < endOffset) {
@@ -0,0 +1,58 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng.idea.highlight
import org.junit.Assert.assertArrayEquals
import org.junit.Assert.assertEquals
import org.junit.Test
class LyngLexerBacktickStringTest {
@Test
fun backtickStringGetsStringTokenAndColor() {
val lexer = LyngLexer()
val source = """val json = `{"name":"lyng","doc":"use \`quotes\`"}`"""
lexer.start(source, 0, source.length, 0)
val tokens = mutableListOf<Pair<String, String>>()
while (lexer.tokenType != null) {
val tokenText = source.substring(lexer.tokenStart, lexer.tokenEnd)
tokens += lexer.tokenType.toString() to tokenText
lexer.advance()
}
assertEquals(
listOf(
"KEYWORD" to "val",
"WHITESPACE" to " ",
"IDENTIFIER" to "json",
"WHITESPACE" to " ",
"PUNCT" to "=",
"WHITESPACE" to " ",
"STRING" to "`{\"name\":\"lyng\",\"doc\":\"use \\`quotes\\`\"}`"
),
tokens
)
val highlighter = LyngSyntaxHighlighter()
assertArrayEquals(
arrayOf(LyngHighlighterColors.STRING),
highlighter.getTokenHighlights(LyngTokenTypes.STRING)
)
}
}
+43
View File
@@ -19,11 +19,39 @@ plugins {
alias(libs.plugins.kotlinMultiplatform) alias(libs.plugins.kotlinMultiplatform)
} }
import org.gradle.api.Project
import org.jetbrains.kotlin.gradle.targets.native.tasks.KotlinNativeTest import org.jetbrains.kotlin.gradle.targets.native.tasks.KotlinNativeTest
group = "net.sergeych" group = "net.sergeych"
version = "unspecified" version = "unspecified"
private fun Project.sqliteLinuxLinkerOpts(vararg defaultDirs: String): List<String> {
val overrideDir = providers.gradleProperty("sqlite3.lib.dir").orNull
?: providers.environmentVariable("SQLITE3_LIB_DIR").orNull
val candidateDirs = buildList {
if (!overrideDir.isNullOrBlank()) {
add(file(overrideDir))
}
defaultDirs.forEach { add(file(it)) }
}.distinctBy { it.absolutePath }
val discoveredLib = sequenceOf("libsqlite3.so", "libsqlite3.so.0")
.mapNotNull { libraryName ->
candidateDirs.firstOrNull { it.resolve(libraryName).isFile }?.let { dir ->
listOf("-L${dir.absolutePath}", "-l:$libraryName")
}
}
.firstOrNull()
?: listOf("-lsqlite3")
return discoveredLib + listOf(
"-ldl",
"-lpthread",
"-lm",
"-Wl,--allow-shlib-undefined"
)
}
repositories { repositories {
mavenCentral() mavenCentral()
maven("https://maven.universablockchain.com/") maven("https://maven.universablockchain.com/")
@@ -55,6 +83,16 @@ kotlin {
binaries { binaries {
executable() executable()
all { all {
linkerOpts(
*project.sqliteLinuxLinkerOpts(
"/lib/x86_64-linux-gnu",
"/usr/lib/x86_64-linux-gnu",
"/lib64",
"/usr/lib64",
"/lib",
"/usr/lib"
).toTypedArray()
)
if (buildType == org.jetbrains.kotlin.gradle.plugin.mpp.NativeBuildType.RELEASE) { if (buildType == org.jetbrains.kotlin.gradle.plugin.mpp.NativeBuildType.RELEASE) {
debuggable = false debuggable = false
optimized = true optimized = true
@@ -113,8 +151,13 @@ kotlin {
tasks.named<KotlinNativeTest>("linuxX64Test") { tasks.named<KotlinNativeTest>("linuxX64Test") {
dependsOn(tasks.named("linkDebugExecutableLinuxX64")) dependsOn(tasks.named("linkDebugExecutableLinuxX64"))
dependsOn(tasks.named("linkReleaseExecutableLinuxX64"))
environment( environment(
"LYNG_CLI_NATIVE_BIN", "LYNG_CLI_NATIVE_BIN",
layout.buildDirectory.file("bin/linuxX64/debugExecutable/lyng.kexe").get().asFile.absolutePath layout.buildDirectory.file("bin/linuxX64/debugExecutable/lyng.kexe").get().asFile.absolutePath
) )
environment(
"LYNG_CLI_NATIVE_RELEASE_BIN",
layout.buildDirectory.file("bin/linuxX64/releaseExecutable/lyng.kexe").get().asFile.absolutePath
)
} }
+35 -5
View File
@@ -17,8 +17,8 @@
package net.sergeych package net.sergeych
import com.github.ajalt.clikt.core.CoreCliktCommand
import com.github.ajalt.clikt.core.Context import com.github.ajalt.clikt.core.Context
import com.github.ajalt.clikt.core.CoreCliktCommand
import com.github.ajalt.clikt.core.main import com.github.ajalt.clikt.core.main
import com.github.ajalt.clikt.core.subcommands import com.github.ajalt.clikt.core.subcommands
import com.github.ajalt.clikt.parameters.arguments.argument import com.github.ajalt.clikt.parameters.arguments.argument
@@ -32,6 +32,7 @@ import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock import kotlinx.coroutines.sync.withLock
import kotlinx.coroutines.withContext import kotlinx.coroutines.withContext
import net.sergeych.lyng.EvalSession import net.sergeych.lyng.EvalSession
import net.sergeych.lyng.ExecutionError
import net.sergeych.lyng.LyngVersion import net.sergeych.lyng.LyngVersion
import net.sergeych.lyng.Pos import net.sergeych.lyng.Pos
import net.sergeych.lyng.Scope import net.sergeych.lyng.Scope
@@ -40,17 +41,24 @@ import net.sergeych.lyng.ScriptError
import net.sergeych.lyng.Source import net.sergeych.lyng.Source
import net.sergeych.lyng.asFacade import net.sergeych.lyng.asFacade
import net.sergeych.lyng.io.console.createConsoleModule import net.sergeych.lyng.io.console.createConsoleModule
import net.sergeych.lyng.io.db.createDbModule
import net.sergeych.lyng.io.db.jdbc.createJdbcModule
import net.sergeych.lyng.io.db.sqlite.createSqliteModule
import net.sergeych.lyng.io.fs.createFs import net.sergeych.lyng.io.fs.createFs
import net.sergeych.lyng.io.html.createHtmlModule
import net.sergeych.lyng.io.http.createHttpModule import net.sergeych.lyng.io.http.createHttpModule
import net.sergeych.lyng.io.http.server.createHttpServerModule
import net.sergeych.lyng.io.net.createNetModule import net.sergeych.lyng.io.net.createNetModule
import net.sergeych.lyng.io.process.createProcessModule
import net.sergeych.lyng.io.ws.createWsModule import net.sergeych.lyng.io.ws.createWsModule
import net.sergeych.lyng.obj.* import net.sergeych.lyng.obj.*
import net.sergeych.lyng.pacman.ImportManager import net.sergeych.lyng.pacman.ImportManager
import net.sergeych.lyngio.net.shutdownSystemNetEngine
import net.sergeych.lyngio.console.security.PermitAllConsoleAccessPolicy import net.sergeych.lyngio.console.security.PermitAllConsoleAccessPolicy
import net.sergeych.lyngio.fs.security.PermitAllAccessPolicy import net.sergeych.lyngio.fs.security.PermitAllAccessPolicy
import net.sergeych.lyngio.http.security.PermitAllHttpAccessPolicy import net.sergeych.lyngio.http.security.PermitAllHttpAccessPolicy
import net.sergeych.lyngio.net.security.PermitAllNetAccessPolicy import net.sergeych.lyngio.net.security.PermitAllNetAccessPolicy
import net.sergeych.lyngio.net.shutdownSystemNetEngine
import net.sergeych.lyngio.process.security.PermitAllProcessAccessPolicy
import net.sergeych.lyngio.ws.security.PermitAllWsAccessPolicy import net.sergeych.lyngio.ws.security.PermitAllWsAccessPolicy
import net.sergeych.mp_tools.globalDefer import net.sergeych.mp_tools.globalDefer
import okio.* import okio.*
@@ -116,8 +124,9 @@ internal class CliExecutionRuntime(
facade.call(handler) facade.call(handler)
} }
} }
session.cancelAndJoin() session.cancel()
shutdownSystemNetEngine() shutdownSystemNetEngine()
session.join()
} }
fun shutdownBlocking() { fun shutdownBlocking() {
@@ -138,7 +147,12 @@ private val baseCliImportManagerDefer = globalDefer {
private fun ImportManager.invalidateCliModuleCaches() { private fun ImportManager.invalidateCliModuleCaches() {
invalidatePackageCache("lyng.io.fs") invalidatePackageCache("lyng.io.fs")
invalidatePackageCache("lyng.io.console") invalidatePackageCache("lyng.io.console")
invalidatePackageCache("lyng.io.db.jdbc")
invalidatePackageCache("lyng.io.db.sqlite")
invalidatePackageCache("lyng.io.html")
invalidatePackageCache("lyng.io.http") invalidatePackageCache("lyng.io.http")
invalidatePackageCache("lyng.io.http.server")
invalidatePackageCache("lyng.io.process")
invalidatePackageCache("lyng.io.ws") invalidatePackageCache("lyng.io.ws")
invalidatePackageCache("lyng.io.net") invalidatePackageCache("lyng.io.net")
} }
@@ -147,8 +161,8 @@ val baseScopeDefer = globalDefer {
baseCliImportManagerDefer.await().copy().apply { baseCliImportManagerDefer.await().copy().apply {
invalidateCliModuleCaches() invalidateCliModuleCaches()
}.newStdScope().apply { }.newStdScope().apply {
installCliDeclarations()
installCliBuiltins() installCliBuiltins()
installCliDeclarations()
addConst("ARGV", ObjList(mutableListOf())) addConst("ARGV", ObjList(mutableListOf()))
} }
} }
@@ -225,7 +239,13 @@ private fun installCliModules(manager: ImportManager) {
// Scripts still need to import the modules they use explicitly. // Scripts still need to import the modules they use explicitly.
createFs(PermitAllAccessPolicy, manager) createFs(PermitAllAccessPolicy, manager)
createConsoleModule(PermitAllConsoleAccessPolicy, manager) createConsoleModule(PermitAllConsoleAccessPolicy, manager)
createDbModule(manager)
createJdbcModule(manager)
createSqliteModule(manager)
createHtmlModule(manager)
createHttpModule(PermitAllHttpAccessPolicy, manager) createHttpModule(PermitAllHttpAccessPolicy, manager)
createHttpServerModule(PermitAllNetAccessPolicy, manager)
createProcessModule(PermitAllProcessAccessPolicy, manager)
createWsModule(PermitAllWsAccessPolicy, manager) createWsModule(PermitAllWsAccessPolicy, manager)
createNetModule(PermitAllNetAccessPolicy, manager) createNetModule(PermitAllNetAccessPolicy, manager)
} }
@@ -355,8 +375,8 @@ private fun registerLocalCliModules(manager: ImportManager, modules: List<LocalC
private suspend fun ImportManager.newCliScope(argv: List<String>): Scope = private suspend fun ImportManager.newCliScope(argv: List<String>): Scope =
newStdScope().apply { newStdScope().apply {
installCliDeclarations()
installCliBuiltins() installCliBuiltins()
installCliDeclarations()
addConst("ARGV", ObjList(argv.map { ObjString(it) }.toMutableList())) addConst("ARGV", ObjList(argv.map { ObjString(it) }.toMutableList()))
} }
@@ -420,6 +440,7 @@ private class Fmt : CoreCliktCommand(name = "fmt") {
val cfg = net.sergeych.lyng.format.LyngFormatConfig( val cfg = net.sergeych.lyng.format.LyngFormatConfig(
applySpacing = enableSpacing, applySpacing = enableSpacing,
applyWrapping = enableWrapping, applyWrapping = enableWrapping,
stringDelimiterPolicy = net.sergeych.lyng.format.LyngStringDelimiterPolicy.PreferFewerEscapes,
) )
var anyChanged = false var anyChanged = false
@@ -537,6 +558,15 @@ suspend fun executeSource(source: Source, initialScope: Scope? = null) {
evalOnCliDispatcher(session, source) evalOnCliDispatcher(session, source)
} catch (e: CliExitRequested) { } catch (e: CliExitRequested) {
requestedExitCode = e.code requestedExitCode = e.code
} catch (e: ExecutionError) {
val cliExit = generateSequence<Throwable>(e) { it.cause }
.filterIsInstance<CliExitRequested>()
.firstOrNull()
if (cliExit != null) {
requestedExitCode = cliExit.code
} else {
throw e
}
} }
} finally { } finally {
shutdownHooks.uninstall() shutdownHooks.uninstall()
@@ -0,0 +1,290 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych
import kotlinx.coroutines.runBlocking
import net.sergeych.lyng.EvalSession
import net.sergeych.lyng.Source
import net.sergeych.lyng.obj.ObjString
import org.junit.After
import org.junit.Before
import java.io.ByteArrayOutputStream
import java.io.PrintStream
import java.nio.file.Files
import kotlin.io.path.writeText
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
class CliLocalModuleImportRegressionJvmTest {
private val originalOut: PrintStream = System.out
private val originalErr: PrintStream = System.err
private class TestExit(val code: Int) : RuntimeException()
@Before
fun setUp() {
jvmExitImpl = { code -> throw TestExit(code) }
}
@After
fun tearDown() {
System.setOut(originalOut)
System.setErr(originalErr)
jvmExitImpl = { code -> kotlin.system.exitProcess(code) }
}
private data class CliResult(val out: String, val err: String, val exitCode: Int?)
private fun runCli(vararg args: String): CliResult {
val outBuf = ByteArrayOutputStream()
val errBuf = ByteArrayOutputStream()
System.setOut(PrintStream(outBuf, true, Charsets.UTF_8))
System.setErr(PrintStream(errBuf, true, Charsets.UTF_8))
var exitCode: Int? = null
try {
runMain(arrayOf(*args))
} catch (e: TestExit) {
exitCode = e.code
} finally {
System.out.flush()
System.err.flush()
}
return CliResult(outBuf.toString("UTF-8"), errBuf.toString("UTF-8"), exitCode)
}
private fun writeTransitiveImportTree(root: java.nio.file.Path) {
val packageDir = Files.createDirectories(root.resolve("package1"))
val nestedDir = Files.createDirectories(packageDir.resolve("nested"))
packageDir.resolve("alpha.lyng").writeText(
"""
package package1.alpha
import lyng.stdlib
import lyng.io.net
class Alpha {
val headers = Map<String, String>()
fun makeTask(port: Int, host: String): Deferred = launch {
host + ":" + port
}
fun netModule() = Net
}
fun alphaValue() = "alpha"
""".trimIndent()
)
packageDir.resolve("beta.lyng").writeText(
"""
package package1.beta
import lyng.stdlib
import package1.alpha
fun betaValue() = alphaValue() + "|beta"
""".trimIndent()
)
nestedDir.resolve("gamma.lyng").writeText(
"""
package package1.nested.gamma
import lyng.io.net
import package1.alpha
import package1.beta
val String.gammaTag get() = this + "|gamma"
fun gammaValue() = betaValue().gammaTag
fun netModule() = Net
""".trimIndent()
)
packageDir.resolve("entry.lyng").writeText(
"""
package package1.entry
import lyng.stdlib
import lyng.io.net
import package1.alpha
import package1.beta
import package1.nested.gamma
fun report() = gammaValue() + "|entry"
""".trimIndent()
)
}
private fun writeNestedLaunchImportBugTree(root: java.nio.file.Path) {
val packageDir = Files.createDirectories(root.resolve("package1"))
packageDir.resolve("alpha.lyng").writeText(
"""
import lyng.io.net
import package1.bravo
class Alpha {
val tcpServer: TcpServer
val headers = Map<String, String>()
fn startListen(port, host) {
var eager = Bravo()
eager.doSomething()
tcpServer = Net.tcpListen(port, host)
println("tcpServer.isOpen: " + tcpServer.isOpen())
launch {
try {
while (true) {
println("wait for accept...")
val tcpSocket = tcpServer.accept()
println("var bravo = Bravo()")
var bravo = Bravo()
println("bravo.doSomething()...")
bravo.doSomething()
println("bravo.doSomething()... OK")
tcpSocket.close()
break
}
} catch (e) {
println("ERR [Alpha.startListen]: '", e, "'")
} finally {
println("FIN [Alpha.startListen]")
tcpServer.close()
}
}
}
}
""".trimIndent()
)
packageDir.resolve("bravo.lyng").writeText(
"""
class Bravo {
fn doSomething() {
println("Bravo.doSomething")
}
}
""".trimIndent()
)
}
@Test
fun localModuleUsingLaunchAndNetImportsWithoutStdlibRedefinition() = runBlocking {
val root = Files.createTempDirectory("lyng-cli-import-regression")
try {
val mainFile = root.resolve("main.lyng")
writeTransitiveImportTree(root)
mainFile.writeText(
"""
import package1.entry
import package1.beta
import package1.nested.gamma
println(report())
""".trimIndent()
)
executeFile(mainFile.toString(), emptyList())
} finally {
root.toFile().deleteRecursively()
}
}
@Test
fun localModuleImportsAreNoOpsWhenEvaldRepeatedlyOnSameCliContext() = runBlocking {
val root = Files.createTempDirectory("lyng-cli-import-regression-repeat")
try {
val mainFile = root.resolve("main.lyng")
writeTransitiveImportTree(root)
mainFile.writeText("println(\"bootstrap\")")
val session = EvalSession(newCliScope(emptyList(), mainFile.toString()))
try {
repeat(5) { index ->
val result = evalOnCliDispatcher(
session,
Source(
"<repeat-local-import-$index>",
"""
import package1.entry
import package1.nested.gamma
import package1.beta
import package1.alpha
report()
""".trimIndent()
)
) as ObjString
assertEquals(
"alpha|beta|gamma|entry",
result.value
)
}
} finally {
session.cancelAndJoin()
}
} finally {
root.toFile().deleteRecursively()
}
}
@Test
fun localModuleImportUsedOnlyInsideMethodLaunchClosureRemainsPrepared() = runBlocking {
val root = Files.createTempDirectory("lyng-cli-import-regression-launch")
try {
val mainFile = root.resolve("main.lyng")
val port = java.net.ServerSocket(0).let {
val selected = it.localPort
it.close()
selected
}
writeNestedLaunchImportBugTree(root)
mainFile.writeText(
"""
import lyng.io.net
import package1.alpha
val alpha = Alpha()
alpha.startListen($port, "127.0.0.1")
delay(50)
val socket = Net.tcpConnect("127.0.0.1", $port)
println("send ping...")
socket.writeUtf8("ping")
socket.flush()
socket.close()
delay(50)
""".trimIndent()
)
val result = runCli(mainFile.toString())
assertTrue(result.err.isBlank(), result.err)
assertFalse(result.out.contains("ERR [Alpha.startListen]"), result.out)
assertFalse(result.out.contains("module capture 'Bravo'"), result.out)
assertTrue(result.out.contains("bravo.doSomething()... OK"), result.out)
assertEquals(2, Regex("Bravo\\.doSomething").findAll(result.out).count(), result.out)
} finally {
root.toFile().deleteRecursively()
}
}
}
@@ -64,6 +64,73 @@ class CliLocalImportsJvmTest {
return CliResult(outBuf.toString("UTF-8"), errBuf.toString("UTF-8"), exitCode) return CliResult(outBuf.toString("UTF-8"), errBuf.toString("UTF-8"), exitCode)
} }
private fun writeTransitiveImportTree(root: java.nio.file.Path) {
val packageDir = Files.createDirectories(root.resolve("package1"))
val nestedDir = Files.createDirectories(packageDir.resolve("nested"))
Files.writeString(
packageDir.resolve("alpha.lyng"),
"""
package package1.alpha
import lyng.stdlib
import lyng.io.net
class Alpha {
val headers = Map<String, String>()
fun makeTask(port: Int, host: String): Deferred = launch {
host + ":" + port
}
fun netModule() = Net
}
fun alphaValue() = "alpha"
""".trimIndent()
)
Files.writeString(
packageDir.resolve("beta.lyng"),
"""
package package1.beta
import lyng.stdlib
import package1.alpha
fun betaValue() = alphaValue() + "|beta"
""".trimIndent()
)
Files.writeString(
nestedDir.resolve("gamma.lyng"),
"""
package package1.nested.gamma
import lyng.io.net
import package1.alpha
import package1.beta
val String.gammaTag get() = this + "|gamma"
fun gammaValue() = betaValue().gammaTag
fun netModule() = Net
""".trimIndent()
)
Files.writeString(
packageDir.resolve("entry.lyng"),
"""
package package1.entry
import lyng.stdlib
import lyng.io.net
import package1.alpha
import package1.beta
import package1.nested.gamma
fun report() = gammaValue() + "|entry"
""".trimIndent()
)
}
@Test @Test
fun cliDiscoversSiblingAndNestedLocalImportsFromEntryRoot() { fun cliDiscoversSiblingAndNestedLocalImportsFromEntryRoot() {
val dir = Files.createTempDirectory("lyng_cli_local_imports_") val dir = Files.createTempDirectory("lyng_cli_local_imports_")
@@ -134,4 +201,37 @@ class CliLocalImportsJvmTest {
dir.toFile().deleteRecursively() dir.toFile().deleteRecursively()
} }
} }
@Test
fun cliHandlesOverlappingDirectoryImportsWithTransitiveStdlibAndNetSymbols() {
val dir = Files.createTempDirectory("lyng_cli_local_imports_transitive_")
try {
val mainFile = dir.resolve("main.lyng")
writeTransitiveImportTree(dir)
Files.writeString(
mainFile,
"""
import package1.entry
import package1.beta
import package1.nested.gamma
println(report())
println(gammaValue())
""".trimIndent()
)
val result = runCli(mainFile.toString())
assertTrue(result.err, result.err.isBlank())
assertTrue(
result.out,
result.out.contains("alpha|beta|gamma|entry")
)
assertTrue(
result.out,
result.out.contains("alpha|beta|gamma")
)
} finally {
dir.toFile().deleteRecursively()
}
}
} }
@@ -73,10 +73,12 @@ class CliNetworkJvmTest {
try { try {
val script = """ val script = """
import lyng.io.http import lyng.io.http
import lyng.io.http.server
import lyng.io.ws import lyng.io.ws
import lyng.io.net import lyng.io.net
assert(Http.isSupported()) assert(Http.isSupported())
assert(HttpServer() is HttpServer)
println("ws=" + Ws.isSupported()) println("ws=" + Ws.isSupported())
println("net=" + Net.isSupported()) println("net=" + Net.isSupported())
@@ -0,0 +1,132 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng_cli
import kotlinx.cinterop.*
import okio.FileSystem
import okio.Path
import okio.Path.Companion.toPath
import platform.posix.O_CREAT
import platform.posix.O_TRUNC
import platform.posix.O_WRONLY
import platform.posix.SIGKILL
import platform.posix.SIGSEGV
import platform.posix._exit
import platform.posix.close
import platform.posix.dup2
import platform.posix.execvp
import platform.posix.fork
import platform.posix.getenv
import platform.posix.getpid
import platform.posix.kill
import platform.posix.open
import platform.posix.usleep
import platform.posix.waitpid
import kotlin.test.Test
import kotlin.test.assertFalse
import kotlin.test.assertTrue
@OptIn(ExperimentalForeignApi::class)
class CliWebSocketNativeRegressionTest {
@Test
fun releaseCliDoesNotSegfaultOnConcurrentWebSocketClients() {
val executable = getenv("LYNG_CLI_NATIVE_RELEASE_BIN")?.toKString()
?: error("LYNG_CLI_NATIVE_RELEASE_BIN is not set")
val fs = FileSystem.SYSTEM
val repoRoot = ascend(executable.toPath(), 6)
val scriptPath = repoRoot / "bugs" / "ws-segfault.lyng"
check(fs.exists(scriptPath)) { "bug repro script not found at $scriptPath" }
val tempDir = "/tmp/lyng_ws_native_${getpid()}_${kotlin.random.Random.nextInt()}".toPath()
val stdoutPath = tempDir / "stdout.txt"
val stderrPath = tempDir / "stderr.txt"
fs.createDirectories(tempDir)
try {
val pid = launchCli(executable, scriptPath, stdoutPath, stderrPath)
usleep(5_000_000u)
if (kill(pid, 0) == 0) {
kill(pid, SIGKILL)
}
val status = waitForPid(pid)
val termSignal = status and 0x7f
val stdout = readUtf8IfExists(fs, stdoutPath)
val stderr = readUtf8IfExists(fs, stderrPath)
val allOutput = "$stdout\n$stderr"
assertFalse(termSignal == SIGSEGV, "native CLI crashed with SIGSEGV. Output:\n$allOutput")
assertTrue(
stdout.lineSequence().count { it == "test send to ws://127.0.0.1:9998... OK" } == 2,
"expected both websocket clients to finish. Output:\n$allOutput"
)
assertFalse(allOutput.contains("Segmentation fault"), "process output reported a segmentation fault:\n$allOutput")
} finally {
fs.deleteRecursively(tempDir, mustExist = false)
}
}
private fun ascend(path: Path, levels: Int): Path {
var current = path
repeat(levels) {
current = current.parent ?: error("cannot ascend $levels levels from $path")
}
return current
}
private fun readUtf8IfExists(fs: FileSystem, path: Path): String {
return if (fs.exists(path)) fs.read(path) { readUtf8() } else ""
}
private fun waitForPid(pid: Int): Int = memScoped {
val status = alloc<IntVar>()
val waited = waitpid(pid, status.ptr, 0)
check(waited == pid) { "waitpid failed for $pid" }
status.value
}
private fun launchCli(
executable: String,
scriptPath: Path,
stdoutPath: Path,
stderrPath: Path,
): Int = memScoped {
val pid = fork()
check(pid >= 0) { "fork failed" }
if (pid == 0) {
val stdoutFd = open(stdoutPath.toString(), O_WRONLY or O_CREAT or O_TRUNC, 0x1A4)
val stderrFd = open(stderrPath.toString(), O_WRONLY or O_CREAT or O_TRUNC, 0x1A4)
if (stdoutFd < 0 || stderrFd < 0) {
_exit(2)
}
dup2(stdoutFd, 1)
dup2(stderrFd, 2)
close(stdoutFd)
close(stderrFd)
val argv = allocArray<CPointerVar<ByteVar>>(3)
argv[0] = executable.cstr.ptr
argv[1] = scriptPath.toString().cstr.ptr
argv[2] = null
execvp(executable, argv)
_exit(127)
}
pid
}
}
+87 -1
View File
@@ -29,7 +29,34 @@ plugins {
} }
group = "net.sergeych" group = "net.sergeych"
version = "0.0.1-SNAPSHOT" version = providers.gradleProperty("lyngio.version").getOrElse("0.0.3-SNAPSHOT")
private fun Project.sqliteLinuxLinkerOpts(vararg defaultDirs: String): List<String> {
val overrideDir = providers.gradleProperty("sqlite3.lib.dir").orNull
?: providers.environmentVariable("SQLITE3_LIB_DIR").orNull
val candidateDirs = buildList {
if (!overrideDir.isNullOrBlank()) {
add(file(overrideDir))
}
defaultDirs.forEach { add(file(it)) }
}.distinctBy { it.absolutePath }
val discoveredLib = sequenceOf("libsqlite3.so", "libsqlite3.so.0")
.mapNotNull { libraryName ->
candidateDirs.firstOrNull { it.resolve(libraryName).isFile }?.let { dir ->
listOf("-L${dir.absolutePath}", "-l:$libraryName")
}
}
.firstOrNull()
?: listOf("-lsqlite3")
return discoveredLib + listOf(
"-ldl",
"-lpthread",
"-lm",
"-Wl,--allow-shlib-undefined"
)
}
kotlin { kotlin {
jvmToolchain(17) jvmToolchain(17)
@@ -58,6 +85,39 @@ kotlin {
// nodejs() // nodejs()
// } // }
targets.withType(org.jetbrains.kotlin.gradle.plugin.mpp.KotlinNativeTarget::class.java).configureEach {
compilations.getByName("main").cinterops.create("sqlite3") {
defFile(project.file("src/nativeInterop/cinterop/sqlite/sqlite3.def"))
packageName("net.sergeych.lyng.io.db.sqlite.cinterop")
includeDirs(project.file("src/nativeInterop/cinterop/sqlite"))
}
binaries.all {
when (konanTarget.name) {
"linux_x64" -> linkerOpts(
*project.sqliteLinuxLinkerOpts(
"/lib/x86_64-linux-gnu",
"/usr/lib/x86_64-linux-gnu",
"/lib64",
"/usr/lib64",
"/lib",
"/usr/lib"
).toTypedArray()
)
"linux_arm64" -> linkerOpts(
*project.sqliteLinuxLinkerOpts(
"/lib/aarch64-linux-gnu",
"/usr/lib/aarch64-linux-gnu",
"/lib64",
"/usr/lib64",
"/lib",
"/usr/lib"
).toTypedArray()
)
else -> linkerOpts("-lsqlite3")
}
}
}
// Keep expect/actual warning suppressed consistently with other modules // Keep expect/actual warning suppressed consistently with other modules
targets.configureEach { targets.configureEach {
compilations.configureEach { compilations.configureEach {
@@ -81,6 +141,7 @@ kotlin {
api(libs.kotlinx.coroutines.core) api(libs.kotlinx.coroutines.core)
api(libs.mordant.core) api(libs.mordant.core)
api(libs.ktor.client.core) api(libs.ktor.client.core)
implementation("com.fleeksoft.ksoup:ksoup:0.2.6")
implementation(libs.ktor.client.websockets) implementation(libs.ktor.client.websockets)
} }
} }
@@ -120,6 +181,12 @@ kotlin {
implementation(libs.kotlinx.coroutines.test) implementation(libs.kotlinx.coroutines.test)
} }
} }
val jvmTest by getting {
dependencies {
implementation(libs.testcontainers)
implementation(libs.testcontainers.postgresql)
}
}
val linuxTest by creating { val linuxTest by creating {
dependsOn(commonTest) dependsOn(commonTest)
} }
@@ -155,6 +222,10 @@ kotlin {
implementation("org.jline:jline-terminal:3.29.0") implementation("org.jline:jline-terminal:3.29.0")
implementation(libs.ktor.client.cio) implementation(libs.ktor.client.cio)
implementation(libs.ktor.network) implementation(libs.ktor.network)
implementation(libs.sqlite.jdbc)
implementation(libs.h2)
implementation(libs.postgresql)
implementation(libs.hikaricp)
} }
} }
// // For Wasm we use in-memory VFS for now // // For Wasm we use in-memory VFS for now
@@ -259,6 +330,15 @@ tasks.matching { it.name.startsWith("lint", ignoreCase = true) }.configureEach {
this.enabled = false this.enabled = false
} }
// Kotlin/Native commonization replaces intermediate .knm files while Gradle is
// snapshotting this task's outputs, which can make publishing fail with a
// FileNotFoundException. These files are generated intermediates, so tracking
// their state provides no useful incremental-build benefit.
tasks.matching { it.name == "commonizeCInterop" }.configureEach {
doNotTrackState("Kotlin/Native commonizer mutates its output tree during execution")
notCompatibleWithConfigurationCache("Kotlin/Native cinterop commonization accesses task projects during execution")
}
publishing { publishing {
val mavenToken by lazy { val mavenToken by lazy {
File("${System.getProperty("user.home")}/.gitea_token").readText() File("${System.getProperty("user.home")}/.gitea_token").readText()
@@ -276,3 +356,9 @@ publishing {
} }
} }
} }
// Lyng's JVM dependencies include Java 21 bytecode; keep the Java 17 library target.
val testToolchains = extensions.getByType<org.gradle.jvm.toolchain.JavaToolchainService>()
tasks.withType<Test>().configureEach {
javaLauncher.set(testToolchains.launcherFor { languageVersion.set(JavaLanguageVersion.of(21)) })
}
@@ -0,0 +1,22 @@
package net.sergeych.lyng.io.db.jdbc
import net.sergeych.lyng.ScopeFacade
import net.sergeych.lyng.io.db.SqlCoreModule
import net.sergeych.lyng.io.db.SqlDatabaseBackend
import net.sergeych.lyng.obj.ObjException
import net.sergeych.lyng.obj.ObjString
import net.sergeych.lyng.requireScope
internal actual suspend fun openJdbcBackend(
scope: ScopeFacade,
core: SqlCoreModule,
options: JdbcOpenOptions,
): SqlDatabaseBackend {
scope.raiseError(
ObjException(
core.databaseException,
scope.requireScope(),
ObjString("lyng.io.db.jdbc is available only on the JVM target")
)
)
}
@@ -0,0 +1,39 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng.io.db.sqlite
import net.sergeych.lyng.ScopeFacade
import net.sergeych.lyng.io.db.SqlCoreModule
import net.sergeych.lyng.io.db.SqlDatabaseBackend
import net.sergeych.lyng.requireScope
import net.sergeych.lyng.obj.ObjException
import net.sergeych.lyng.obj.ObjString
internal actual suspend fun openSqliteBackend(
scope: ScopeFacade,
core: SqlCoreModule,
options: SqliteOpenOptions,
): SqlDatabaseBackend {
scope.raiseError(
ObjException(
core.databaseException,
scope.requireScope(),
ObjString("SQLite provider is not implemented on this platform yet")
)
)
}
@@ -2,4 +2,4 @@ package net.sergeych.lyngio.http
import io.ktor.client.engine.cio.CIO import io.ktor.client.engine.cio.CIO
actual fun getSystemHttpEngine(): LyngHttpEngine = createKtorHttpEngine(CIO) actual fun getSystemHttpEngine(config: LyngHttpConfig): LyngHttpEngine = createKtorHttpEngine(CIO, config)
@@ -0,0 +1,142 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng.io.db
import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Pos
import net.sergeych.lyng.Scope
import net.sergeych.lyng.ScopeFacade
import net.sergeych.lyng.Source
import net.sergeych.lyng.Arguments
import net.sergeych.lyng.obj.Obj
import net.sergeych.lyng.obj.ObjException
import net.sergeych.lyng.obj.ObjNull
import net.sergeych.lyng.obj.ObjString
import net.sergeych.lyng.obj.ObjVoid
import net.sergeych.lyng.obj.requiredArg
import net.sergeych.lyng.pacman.ImportManager
import net.sergeych.lyng.requireScope
import net.sergeych.lyngio.stdlib_included.dbLyng
private const val DB_MODULE_NAME = "lyng.io.db"
fun createDbModule(scope: Scope): Boolean = createDbModule(scope.importManager)
fun createDb(scope: Scope): Boolean = createDbModule(scope)
fun createDbModule(manager: ImportManager): Boolean {
if (manager.packageNames.contains(DB_MODULE_NAME)) return false
manager.addPackage(DB_MODULE_NAME) { module ->
buildDbModule(module)
}
return true
}
fun createDb(manager: ImportManager): Boolean = createDbModule(manager)
private suspend fun buildDbModule(module: ModuleScope) {
module.eval(Source(DB_MODULE_NAME, dbLyng))
val exceptions = installDbExceptionClasses(module)
val registry = DbProviderRegistry()
module.addFn("registerDatabaseProvider") {
val scheme = requiredArg<ObjString>(0).value
val opener = args.list.getOrNull(1)
?: raiseError("Expected exactly 2 arguments, got ${args.list.size}")
registry.register(this, scheme, opener)
ObjVoid
}
module.addFn("openDatabase") {
val connectionUrl = requiredArg<ObjString>(0).value
val extraParams = args.list.getOrNull(1)
?: raiseError("Expected exactly 2 arguments, got ${args.list.size}")
if (!extraParams.isInstanceOf("Map")) {
raiseIllegalArgument("extraParams must be Map")
}
val scheme = parseConnectionScheme(connectionUrl)
?: raiseIllegalArgument("Malformed database connection URL: $connectionUrl")
val opener = registry.providers[scheme]
?: raiseDatabaseException(exceptions.database, "No database provider registered for scheme '$scheme'")
call(opener, Arguments(listOf(ObjString(connectionUrl), extraParams)), newThisObj = ObjNull)
}
}
private data class DbExceptionClasses(
val database: ObjException.Companion.ExceptionClass,
val sqlExecution: ObjException.Companion.ExceptionClass,
val sqlConstraint: ObjException.Companion.ExceptionClass,
val sqlUsage: ObjException.Companion.ExceptionClass,
val rollback: ObjException.Companion.ExceptionClass,
)
private fun installDbExceptionClasses(module: ModuleScope): DbExceptionClasses {
val database = ObjException.Companion.ExceptionClass("DatabaseException", ObjException.Root)
val sqlExecution = ObjException.Companion.ExceptionClass("SqlExecutionException", database)
val sqlConstraint = ObjException.Companion.ExceptionClass("SqlConstraintException", sqlExecution)
val sqlUsage = ObjException.Companion.ExceptionClass("SqlUsageException", database)
val rollback = ObjException.Companion.ExceptionClass("RollbackException", ObjException.Root)
module.addConst("DatabaseException", database)
module.addConst("SqlExecutionException", sqlExecution)
module.addConst("SqlConstraintException", sqlConstraint)
module.addConst("SqlUsageException", sqlUsage)
module.addConst("RollbackException", rollback)
return DbExceptionClasses(
database = database,
sqlExecution = sqlExecution,
sqlConstraint = sqlConstraint,
sqlUsage = sqlUsage,
rollback = rollback,
)
}
private class DbProviderRegistry {
val providers: MutableMap<String, Obj> = linkedMapOf()
fun register(scope: ScopeFacade, rawScheme: String, opener: Obj) {
val scheme = normalizeScheme(rawScheme)
?: scope.raiseIllegalArgument("Database provider scheme must not be empty")
if (!opener.isInstanceOf("Callable")) {
scope.raiseIllegalArgument("Database provider opener must be callable")
}
if (providers.containsKey(scheme)) {
scope.raiseIllegalState("Database provider already registered for scheme '$scheme'")
}
providers[scheme] = opener
}
}
private fun normalizeScheme(rawScheme: String): String? {
val trimmed = rawScheme.trim()
if (trimmed.isEmpty()) return null
if (':' in trimmed) return null
return trimmed.lowercase()
}
private fun parseConnectionScheme(connectionUrl: String): String? {
val colonIndex = connectionUrl.indexOf(':')
if (colonIndex <= 0) return null
return normalizeScheme(connectionUrl.substring(0, colonIndex))
}
private fun ScopeFacade.raiseDatabaseException(
exceptionClass: ObjException.Companion.ExceptionClass,
message: String,
): Nothing = raiseError(ObjException(exceptionClass, requireScope(), ObjString(message)))
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,278 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng.io.db.jdbc
import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Pos
import net.sergeych.lyng.Scope
import net.sergeych.lyng.ScopeFacade
import net.sergeych.lyng.Source
import net.sergeych.lyng.io.db.SqlCoreModule
import net.sergeych.lyng.io.db.SqlDatabaseBackend
import net.sergeych.lyng.io.db.SqlDatabaseObj
import net.sergeych.lyng.io.db.SqlRuntimeTypes
import net.sergeych.lyng.io.db.createDbModule
import net.sergeych.lyng.obj.Obj
import net.sergeych.lyng.obj.ObjBool
import net.sergeych.lyng.obj.ObjImmutableMap
import net.sergeych.lyng.obj.ObjInt
import net.sergeych.lyng.obj.ObjMap
import net.sergeych.lyng.obj.ObjNull
import net.sergeych.lyng.obj.ObjString
import net.sergeych.lyng.obj.ObjVoid
import net.sergeych.lyng.obj.requiredArg
import net.sergeych.lyng.pacman.ImportManager
import net.sergeych.lyng.requireScope
import net.sergeych.lyngio.stdlib_included.db_jdbcLyng
private const val JDBC_MODULE_NAME = "lyng.io.db.jdbc"
private const val DB_MODULE_NAME = "lyng.io.db"
private const val JDBC_SCHEME = "jdbc"
private const val H2_DRIVER = "org.h2.Driver"
private const val POSTGRES_DRIVER = "org.postgresql.Driver"
fun createJdbcModule(scope: Scope): Boolean = createJdbcModule(scope.importManager)
fun createJdbc(scope: Scope): Boolean = createJdbcModule(scope)
fun createJdbcModule(manager: ImportManager): Boolean {
createDbModule(manager)
if (manager.packageNames.contains(JDBC_MODULE_NAME)) return false
manager.addPackage(JDBC_MODULE_NAME) { module ->
buildJdbcModule(module)
}
return true
}
fun createJdbc(manager: ImportManager): Boolean = createJdbcModule(manager)
private suspend fun buildJdbcModule(module: ModuleScope) {
module.eval(Source(JDBC_MODULE_NAME, db_jdbcLyng))
val dbModule = module.importProvider.createModuleScope(Pos.builtIn, DB_MODULE_NAME)
val core = SqlCoreModule.resolve(dbModule)
val runtimeTypes = SqlRuntimeTypes.create("Jdbc", core)
module.addFn("openJdbc") {
val options = parseOpenJdbcArgs(this)
SqlDatabaseObj(runtimeTypes, openJdbcBackend(this, core, options))
}
module.addFn("openH2") {
val options = parseOpenShortcutArgs(this, H2_DRIVER, ::normalizeH2Url)
SqlDatabaseObj(runtimeTypes, openJdbcBackend(this, core, options))
}
module.addFn("openPostgres") {
val options = parseOpenShortcutArgs(this, POSTGRES_DRIVER, ::normalizePostgresUrl)
SqlDatabaseObj(runtimeTypes, openJdbcBackend(this, core, options))
}
registerProvider(dbModule, runtimeTypes, core, JDBC_SCHEME, null)
registerProvider(dbModule, runtimeTypes, core, "h2", H2_DRIVER)
registerProvider(dbModule, runtimeTypes, core, "postgres", POSTGRES_DRIVER)
registerProvider(dbModule, runtimeTypes, core, "postgresql", POSTGRES_DRIVER)
}
private suspend fun registerProvider(
dbModule: ModuleScope,
runtimeTypes: SqlRuntimeTypes,
core: SqlCoreModule,
scheme: String,
implicitDriverClass: String?,
) {
dbModule.callFn(
"registerDatabaseProvider",
ObjString(scheme),
net.sergeych.lyng.obj.ObjExternCallable.fromBridge {
val connectionUrl = requiredArg<ObjString>(0).value
val extraParams = args.list.getOrNull(1)
?: raiseError("Expected exactly 2 arguments, got ${args.list.size}")
val options = parseJdbcConnectionUrl(this, scheme, implicitDriverClass, connectionUrl, extraParams)
SqlDatabaseObj(runtimeTypes, openJdbcBackend(this, core, options))
}
)
}
private suspend fun parseOpenJdbcArgs(scope: ScopeFacade): JdbcOpenOptions {
val rawUrl = readArg(scope, "connectionUrl", 0) ?: scope.raiseError("argument 'connectionUrl' is required")
val connectionUrl = (rawUrl as? ObjString)?.value ?: scope.raiseClassCastError("connectionUrl must be String")
val user = readNullableStringArg(scope, "user", 1)
val password = readNullableStringArg(scope, "password", 2)
val driverClass = readNullableStringArg(scope, "driverClass", 3)
val properties = readPropertiesArg(scope, "properties", 4)
return JdbcOpenOptions(
connectionUrl = normalizeJdbcUrl(connectionUrl, scope),
user = user,
password = password,
driverClass = driverClass,
properties = properties,
)
}
private suspend fun parseOpenShortcutArgs(
scope: ScopeFacade,
defaultDriverClass: String,
normalize: (String, ScopeFacade) -> String,
): JdbcOpenOptions {
val rawUrl = readArg(scope, "connectionUrl", 0) ?: scope.raiseError("argument 'connectionUrl' is required")
val connectionUrl = (rawUrl as? ObjString)?.value ?: scope.raiseClassCastError("connectionUrl must be String")
val user = readNullableStringArg(scope, "user", 1)
val password = readNullableStringArg(scope, "password", 2)
val properties = readPropertiesArg(scope, "properties", 3)
return JdbcOpenOptions(
connectionUrl = normalize(connectionUrl, scope),
user = user,
password = password,
driverClass = defaultDriverClass,
properties = properties,
)
}
private suspend fun parseJdbcConnectionUrl(
scope: ScopeFacade,
scheme: String,
implicitDriverClass: String?,
connectionUrl: String,
extraParams: Obj,
): JdbcOpenOptions {
val driverClass = mapNullableString(extraParams, scope, "driverClass") ?: implicitDriverClass
val user = mapNullableString(extraParams, scope, "user")
val password = mapNullableString(extraParams, scope, "password")
val properties = mapProperties(extraParams, scope, "properties")
val normalizedUrl = when (scheme) {
JDBC_SCHEME -> normalizeJdbcUrl(connectionUrl, scope)
"h2" -> normalizeH2Url(connectionUrl, scope)
"postgres", "postgresql" -> normalizePostgresUrl(connectionUrl, scope)
else -> scope.raiseIllegalArgument("Unsupported JDBC provider scheme: $scheme")
}
return JdbcOpenOptions(
connectionUrl = normalizedUrl,
user = user,
password = password,
driverClass = driverClass,
properties = properties,
)
}
private fun normalizeJdbcUrl(rawUrl: String, scope: ScopeFacade): String {
val trimmed = rawUrl.trim()
if (!trimmed.startsWith("jdbc:", ignoreCase = true)) {
scope.raiseIllegalArgument("JDBC connection URL must start with jdbc:")
}
return trimmed
}
private fun normalizeH2Url(rawUrl: String, scope: ScopeFacade): String {
val trimmed = rawUrl.trim()
if (trimmed.isEmpty()) {
scope.raiseIllegalArgument("H2 connection URL must not be empty")
}
return when {
trimmed.startsWith("jdbc:h2:", ignoreCase = true) -> trimmed
trimmed.startsWith("h2:", ignoreCase = true) -> "jdbc:${trimmed}"
else -> "jdbc:h2:$trimmed"
}
}
private fun normalizePostgresUrl(rawUrl: String, scope: ScopeFacade): String {
val trimmed = rawUrl.trim()
if (trimmed.isEmpty()) {
scope.raiseIllegalArgument("PostgreSQL connection URL must not be empty")
}
return when {
trimmed.startsWith("jdbc:postgresql:", ignoreCase = true) -> trimmed
trimmed.startsWith("postgresql:", ignoreCase = true) -> "jdbc:$trimmed"
trimmed.startsWith("postgres:", ignoreCase = true) -> "jdbc:postgresql:${trimmed.substringAfter(':')}"
else -> "jdbc:postgresql:$trimmed"
}
}
private suspend fun readArg(scope: ScopeFacade, name: String, position: Int): Obj? {
val named = scope.args.named[name]
val positional = scope.args.list.getOrNull(position)
if (named != null && positional != null) {
scope.raiseIllegalArgument("argument '$name' is already set")
}
return named ?: positional
}
private suspend fun readNullableStringArg(scope: ScopeFacade, name: String, position: Int): String? {
val value = readArg(scope, name, position) ?: return null
return when (value) {
ObjNull -> null
is ObjString -> value.value
else -> scope.raiseClassCastError("$name must be String?")
}
}
private suspend fun readPropertiesArg(scope: ScopeFacade, name: String, position: Int): Map<String, String> {
val value = readArg(scope, name, position) ?: return emptyMap()
return when (value) {
ObjNull -> emptyMap()
else -> objToStringMap(value, scope, name)
}
}
private suspend fun mapNullableString(map: Obj, scope: ScopeFacade, key: String): String? {
val value = map.getAt(scope.requireScope(), ObjString(key))
return when (value) {
ObjNull -> null
is ObjString -> value.value
else -> scope.raiseClassCastError("extraParams.$key must be String?")
}
}
private suspend fun mapProperties(map: Obj, scope: ScopeFacade, key: String): Map<String, String> {
val value = map.getAt(scope.requireScope(), ObjString(key))
return when (value) {
ObjNull -> emptyMap()
else -> objToStringMap(value, scope, "extraParams.$key")
}
}
private suspend fun objToStringMap(value: Obj, scope: ScopeFacade, label: String): Map<String, String> {
val rawEntries = when (value) {
is ObjMap -> value.map
is ObjImmutableMap -> value.map
else -> scope.raiseClassCastError("$label must be Map<String, Object?>")
}
val properties = linkedMapOf<String, String>()
for ((rawKey, rawValue) in rawEntries) {
val key = (rawKey as? ObjString)?.value ?: scope.raiseClassCastError("$label keys must be String")
if (rawValue == ObjNull) continue
properties[key] = scope.toStringOf(rawValue).value
}
return properties
}
private suspend fun ModuleScope.callFn(name: String, vararg args: Obj): Obj {
val callee = get(name)?.value ?: error("Missing $name in module")
return callee.invoke(this, ObjNull, *args)
}
internal data class JdbcOpenOptions(
val connectionUrl: String,
val user: String?,
val password: String?,
val driverClass: String?,
val properties: Map<String, String>,
)
internal expect suspend fun openJdbcBackend(
scope: ScopeFacade,
core: SqlCoreModule,
options: JdbcOpenOptions,
): SqlDatabaseBackend
@@ -0,0 +1,194 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng.io.db.sqlite
import net.sergeych.lyng.Arguments
import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Pos
import net.sergeych.lyng.Scope
import net.sergeych.lyng.ScopeFacade
import net.sergeych.lyng.Source
import net.sergeych.lyng.asFacade
import net.sergeych.lyng.io.db.SqlCoreModule
import net.sergeych.lyng.io.db.SqlDatabaseBackend
import net.sergeych.lyng.io.db.SqlDatabaseObj
import net.sergeych.lyng.io.db.SqlRuntimeTypes
import net.sergeych.lyng.io.db.SqlTransactionBackend
import net.sergeych.lyng.io.db.createDbModule
import net.sergeych.lyng.requireScope
import net.sergeych.lyng.obj.Obj
import net.sergeych.lyng.obj.ObjBool
import net.sergeych.lyng.obj.ObjInt
import net.sergeych.lyng.obj.ObjNull
import net.sergeych.lyng.obj.ObjString
import net.sergeych.lyng.obj.ObjVoid
import net.sergeych.lyng.obj.requiredArg
import net.sergeych.lyng.pacman.ImportManager
import net.sergeych.lyngio.stdlib_included.db_sqliteLyng
private const val SQLITE_MODULE_NAME = "lyng.io.db.sqlite"
private const val DB_MODULE_NAME = "lyng.io.db"
fun createSqliteModule(scope: Scope): Boolean = createSqliteModule(scope.importManager)
fun createSqlite(scope: Scope): Boolean = createSqliteModule(scope)
fun createSqliteModule(manager: ImportManager): Boolean {
createDbModule(manager)
if (manager.packageNames.contains(SQLITE_MODULE_NAME)) return false
manager.addPackage(SQLITE_MODULE_NAME) { module ->
buildSqliteModule(module)
}
return true
}
fun createSqlite(manager: ImportManager): Boolean = createSqliteModule(manager)
private suspend fun buildSqliteModule(module: ModuleScope) {
module.eval(Source(SQLITE_MODULE_NAME, db_sqliteLyng))
val dbModule = module.importProvider.createModuleScope(Pos.builtIn, DB_MODULE_NAME)
val core = SqlCoreModule.resolve(dbModule)
val runtimeTypes = SqlRuntimeTypes.create("Sqlite", core)
module.addFn("openSqlite") {
val options = parseOpenSqliteArgs(this)
SqlDatabaseObj(runtimeTypes, openSqliteBackend(this, core, options))
}
dbModule.callFn(
"registerDatabaseProvider",
ObjString("sqlite"),
net.sergeych.lyng.obj.ObjExternCallable.fromBridge {
val connectionUrl = requiredArg<ObjString>(0).value
val extraParams = args.list.getOrNull(1)
?: raiseError("Expected exactly 2 arguments, got ${args.list.size}")
val options = parseSqliteConnectionUrl(this, connectionUrl, extraParams)
SqlDatabaseObj(runtimeTypes, openSqliteBackend(this, core, options))
}
)
}
private suspend fun parseOpenSqliteArgs(scope: ScopeFacade): SqliteOpenOptions {
val pathValue = readArg(scope, "path", 0) ?: scope.raiseError("argument 'path' is required")
val path = (pathValue as? ObjString)?.value ?: scope.raiseClassCastError("path must be String")
val readOnly = readBoolArg(scope, "readOnly", 1, false)
val createIfMissing = readBoolArg(scope, "createIfMissing", 2, true)
val foreignKeys = readBoolArg(scope, "foreignKeys", 3, true)
val busyTimeoutMillis = readIntArg(scope, "busyTimeoutMillis", 4, 5000)
return SqliteOpenOptions(
path = normalizeSqlitePath(path, scope),
readOnly = readOnly,
createIfMissing = createIfMissing,
foreignKeys = foreignKeys,
busyTimeoutMillis = busyTimeoutMillis,
)
}
private suspend fun parseSqliteConnectionUrl(
scope: ScopeFacade,
connectionUrl: String,
extraParams: Obj,
): SqliteOpenOptions {
val prefix = "sqlite:"
if (!connectionUrl.startsWith(prefix, ignoreCase = true)) {
scope.raiseIllegalArgument("Malformed SQLite connection URL: $connectionUrl")
}
val rawPath = connectionUrl.substring(prefix.length)
val path = normalizeSqlitePath(rawPath, scope)
val readOnly = mapBool(extraParams, scope, "readOnly") ?: false
val createIfMissing = mapBool(extraParams, scope, "createIfMissing") ?: true
val foreignKeys = mapBool(extraParams, scope, "foreignKeys") ?: true
val busyTimeoutMillis = mapInt(extraParams, scope, "busyTimeoutMillis") ?: 5000
return SqliteOpenOptions(
path = path,
readOnly = readOnly,
createIfMissing = createIfMissing,
foreignKeys = foreignKeys,
busyTimeoutMillis = busyTimeoutMillis,
)
}
private fun normalizeSqlitePath(rawPath: String, scope: ScopeFacade): String {
val path = rawPath.trim()
if (path.isEmpty()) {
scope.raiseIllegalArgument("SQLite path must not be empty")
}
if (path.startsWith("//")) {
scope.raiseIllegalArgument("Unsupported SQLite URL form: sqlite:$path")
}
return path
}
private suspend fun readArg(scope: ScopeFacade, name: String, position: Int): Obj? {
val named = scope.args.named[name]
val positional = scope.args.list.getOrNull(position)
if (named != null && positional != null) {
scope.raiseIllegalArgument("argument '$name' is already set")
}
return named ?: positional
}
private suspend fun readBoolArg(scope: ScopeFacade, name: String, position: Int, default: Boolean): Boolean {
val value = readArg(scope, name, position) ?: return default
return (value as? ObjBool)?.value ?: scope.raiseClassCastError("$name must be Bool")
}
private suspend fun readIntArg(scope: ScopeFacade, name: String, position: Int, default: Int): Int {
val value = readArg(scope, name, position) ?: return default
return when (value) {
is ObjInt -> value.value.toInt()
else -> scope.raiseClassCastError("$name must be Int")
}
}
private suspend fun mapBool(map: Obj, scope: ScopeFacade, key: String): Boolean? {
val value = map.getAt(scope.requireScope(), ObjString(key))
return when (value) {
ObjNull -> null
is ObjBool -> value.value
else -> scope.raiseClassCastError("extraParams.$key must be Bool")
}
}
private suspend fun mapInt(map: Obj, scope: ScopeFacade, key: String): Int? {
val value = map.getAt(scope.requireScope(), ObjString(key))
return when (value) {
ObjNull -> null
is ObjInt -> value.value.toInt()
else -> scope.raiseClassCastError("extraParams.$key must be Int")
}
}
private suspend fun ModuleScope.callFn(name: String, vararg args: Obj): Obj {
val callee = get(name)?.value ?: error("Missing $name in module")
return callee.invoke(this, ObjNull, *args)
}
internal data class SqliteOpenOptions(
val path: String,
val readOnly: Boolean,
val createIfMissing: Boolean,
val foreignKeys: Boolean,
val busyTimeoutMillis: Int,
)
internal expect suspend fun openSqliteBackend(
scope: ScopeFacade,
core: SqlCoreModule,
options: SqliteOpenOptions,
): SqlDatabaseBackend
@@ -0,0 +1,44 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng.io.html
import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Scope
import net.sergeych.lyng.Source
import net.sergeych.lyng.pacman.ImportManager
import net.sergeych.lyngio.stdlib_included.htmlLyng
private const val HTML_MODULE_NAME = "lyng.io.html"
fun createHtmlModule(scope: Scope): Boolean = createHtmlModule(scope.importManager)
fun createHtml(scope: Scope): Boolean = createHtmlModule(scope)
fun createHtmlModule(manager: ImportManager): Boolean {
if (manager.packageNames.contains(HTML_MODULE_NAME)) return false
manager.addPackage(HTML_MODULE_NAME) { module ->
buildHtmlModule(module)
}
return true
}
fun createHtml(manager: ImportManager): Boolean = createHtmlModule(manager)
private suspend fun buildHtmlModule(module: ModuleScope) {
module.eval(Source(HTML_MODULE_NAME, htmlLyng))
}
@@ -0,0 +1,96 @@
package net.sergeych.lyng.io.html
import com.fleeksoft.ksoup.Ksoup
import com.fleeksoft.ksoup.nodes.Document
import com.fleeksoft.ksoup.nodes.Element
import com.fleeksoft.ksoup.nodes.Node
import com.fleeksoft.ksoup.nodes.TextNode
import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Scope
import net.sergeych.lyng.Source
import net.sergeych.lyng.obj.*
import net.sergeych.lyng.pacman.ImportManager
import net.sergeych.lyngio.stdlib_included.html_parseLyng
private const val MODULE = "lyng.io.html.parse"
fun createHtmlParseModule(scope: Scope): Boolean = createHtmlParseModule(scope.importManager)
fun createHtmlParseModule(manager: ImportManager): Boolean {
if (MODULE in manager.packageNames) return false
manager.addPackage(MODULE) { module -> buildHtmlParseModule(module) }
return true
}
private suspend fun buildHtmlParseModule(module: ModuleScope) {
module.eval(Source(MODULE, html_parseLyng))
val html = ObjClass("Html").apply {
addClassFn("parse") {
val source = requiredArg<ObjString>(0).value
val baseUrl = (args.list.getOrNull(1) as? ObjString)?.value.orEmpty()
ObjHtmlDocument(Ksoup.parse(source, baseUrl))
}
}
module.addConst("Html", html)
module.addConst("HtmlDocument", ObjHtmlDocument.type)
module.addConst("HtmlElement", ObjHtmlElement.type)
}
private class ObjHtmlDocument(val document: Document) : Obj() {
override val objClass get() = type
companion object {
val type = object : ObjClass("HtmlDocument") {
override suspend fun callOn(scope: Scope): Obj = scope.raiseError("Use Html.parse()")
}.apply {
addProperty("title", getter = { ObjString(thisAs<ObjHtmlDocument>().document.title()) })
addFn("text") { ObjString(readableText(thisAs<ObjHtmlDocument>().document.body())) }
addFn("select") {
val css = requiredArg<ObjString>(0).value
ObjList(thisAs<ObjHtmlDocument>().document.select(css).map { ObjHtmlElement(it) as Obj }.toMutableList())
}
}
}
}
private class ObjHtmlElement(val element: Element) : Obj() {
override val objClass get() = type
companion object {
val type = object : ObjClass("HtmlElement") {
override suspend fun callOn(scope: Scope): Obj = scope.raiseError("Use document.select()")
}.apply {
addFn("text") { ObjString(readableText(thisAs<ObjHtmlElement>().element)) }
addFn("attr") { ObjString(thisAs<ObjHtmlElement>().element.attr(requiredArg<ObjString>(0).value)) }
addFn("absUrl") { ObjString(thisAs<ObjHtmlElement>().element.absUrl(requiredArg<ObjString>(0).value)) }
addFn("select") {
val css = requiredArg<ObjString>(0).value
ObjList(thisAs<ObjHtmlElement>().element.select(css).map { ObjHtmlElement(it) as Obj }.toMutableList())
}
}
}
}
private val excludedTags = setOf("script", "style", "template", "noscript")
private val inlineWhitespace = Regex("[\\s\\u00a0]+")
/** Iterative traversal avoids overflowing the host stack on deeply nested input. */
private fun readableText(root: Element): String {
val text = StringBuilder()
val pending = ArrayDeque<Pair<Node, Boolean>>()
pending.addLast(root to false)
while (pending.isNotEmpty()) {
val (node, exit) = pending.removeLast()
when (node) {
is TextNode -> text.append(node.getWholeText().replace(inlineWhitespace, " "))
is Element -> {
if (node.normalName() in excludedTags || node.hasAttr("hidden")) continue
val block = node.isBlock() || node.normalName() == "br"
if (block) text.append('\n')
if (!exit) {
if (block) pending.addLast(node to true)
for (child in node.childNodes().asReversed()) pending.addLast(child to false)
}
}
}
}
return text.toString().lineSequence().map { it.replace(inlineWhitespace, " ").trim() }.filter { it.isNotEmpty() }.joinToString("\n")
}
@@ -17,6 +17,8 @@
package net.sergeych.lyng.io.http package net.sergeych.lyng.io.http
import kotlinx.coroutines.CancellationException
import net.sergeych.lyng.ModuleScope import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Scope import net.sergeych.lyng.Scope
import net.sergeych.lyng.ScopeFacade import net.sergeych.lyng.ScopeFacade
@@ -41,32 +43,64 @@ import net.sergeych.lyng.requireNoArgs
import net.sergeych.lyng.requireScope import net.sergeych.lyng.requireScope
import net.sergeych.lyngio.http.LyngHttpRequest import net.sergeych.lyngio.http.LyngHttpRequest
import net.sergeych.lyngio.http.LyngHttpResponse import net.sergeych.lyngio.http.LyngHttpResponse
import net.sergeych.lyngio.http.LyngHttpConfig
import net.sergeych.lyngio.http.getSystemHttpEngine import net.sergeych.lyngio.http.getSystemHttpEngine
import net.sergeych.lyngio.http.security.HttpAccessDeniedException import net.sergeych.lyngio.http.security.HttpAccessDeniedException
import net.sergeych.lyngio.http.security.HttpAccessOp import net.sergeych.lyngio.http.security.HttpAccessOp
import net.sergeych.lyngio.http.security.HttpAccessPolicy import net.sergeych.lyngio.http.security.HttpAccessPolicy
import net.sergeych.lyngio.stdlib_included.httpLyng import net.sergeych.lyngio.stdlib_included.httpLyng
import net.sergeych.lyngio.stdlib_included.http_typesLyng
private const val HTTP_MODULE_NAME = "lyng.io.http" private const val HTTP_MODULE_NAME = "lyng.io.http"
internal const val HTTP_TYPES_MODULE_NAME = "lyng.io.http.types"
fun createHttpModule(policy: HttpAccessPolicy, scope: Scope): Boolean = fun createHttpModule(
createHttpModule(policy, scope.importManager) policy: HttpAccessPolicy,
scope: Scope,
config: LyngHttpConfig = LyngHttpConfig(),
): Boolean = createHttpModule(policy, scope.importManager, config)
fun createHttp(policy: HttpAccessPolicy, scope: Scope): Boolean = createHttpModule(policy, scope) fun createHttp(
policy: HttpAccessPolicy,
scope: Scope,
config: LyngHttpConfig = LyngHttpConfig(),
): Boolean = createHttpModule(policy, scope, config)
fun createHttpModule(policy: HttpAccessPolicy, manager: ImportManager): Boolean { fun createHttpModule(
policy: HttpAccessPolicy,
manager: ImportManager,
config: LyngHttpConfig = LyngHttpConfig(),
): Boolean {
createHttpTypesModule(manager)
if (manager.packageNames.contains(HTTP_MODULE_NAME)) return false if (manager.packageNames.contains(HTTP_MODULE_NAME)) return false
manager.addPackage(HTTP_MODULE_NAME) { module -> manager.addPackage(HTTP_MODULE_NAME) { module ->
buildHttpModule(module, policy) buildHttpModule(module, policy, config)
} }
return true return true
} }
fun createHttp(policy: HttpAccessPolicy, manager: ImportManager): Boolean = createHttpModule(policy, manager) fun createHttp(
policy: HttpAccessPolicy,
manager: ImportManager,
config: LyngHttpConfig = LyngHttpConfig(),
): Boolean = createHttpModule(policy, manager, config)
private suspend fun buildHttpModule(module: ModuleScope, policy: HttpAccessPolicy) { internal fun createHttpTypesModule(manager: ImportManager): Boolean {
if (manager.packageNames.contains(HTTP_TYPES_MODULE_NAME)) return false
manager.addPackage(HTTP_TYPES_MODULE_NAME) { module ->
buildHttpTypesModule(module)
}
return true
}
private suspend fun buildHttpTypesModule(module: ModuleScope) {
module.eval(Source(HTTP_TYPES_MODULE_NAME, http_typesLyng))
module.addConst("HttpHeaders", ObjHttpHeaders.type)
}
private suspend fun buildHttpModule(module: ModuleScope, policy: HttpAccessPolicy, config: LyngHttpConfig) {
module.eval(Source(HTTP_MODULE_NAME, httpLyng)) module.eval(Source(HTTP_MODULE_NAME, httpLyng))
val engine = getSystemHttpEngine() val engine = getSystemHttpEngine(config)
val headersType = ObjHttpHeaders.type val headersType = ObjHttpHeaders.type
val requestType = ObjHttpRequest.type val requestType = ObjHttpRequest.type
@@ -132,14 +166,16 @@ private suspend fun buildHttpModule(module: ModuleScope, policy: HttpAccessPolic
private suspend inline fun ScopeFacade.httpGuard(crossinline block: suspend () -> Obj): Obj { private suspend inline fun ScopeFacade.httpGuard(crossinline block: suspend () -> Obj): Obj {
return try { return try {
block() block()
} catch (e: CancellationException) {
throw e
} catch (e: HttpAccessDeniedException) { } catch (e: HttpAccessDeniedException) {
raiseIllegalOperation(e.reasonDetail ?: "http access denied") raiseIllegalOperation("HTTP_ACCESS_DENIED: " + (e.reasonDetail ?: "http access denied"))
} catch (e: Exception) { } catch (e: Exception) {
raiseIllegalOperation(e.message ?: "http error") raiseIllegalOperation(e.message ?: "http error")
} }
} }
private class ObjHttpHeaders( internal class ObjHttpHeaders(
singleValueHeaders: Map<String, String> = emptyMap(), singleValueHeaders: Map<String, String> = emptyMap(),
private val allHeaders: Map<String, List<String>> = emptyMap(), private val allHeaders: Map<String, List<String>> = emptyMap(),
) : Obj() { ) : Obj() {
@@ -201,6 +237,11 @@ private class ObjHttpHeaders(
).invokeInstanceMethod(requireScope(), "iterator") ).invokeInstanceMethod(requireScope(), "iterator")
} }
} }
internal fun fromHeaders(
singleValueHeaders: Map<String, String>,
allHeaders: Map<String, List<String>>,
): ObjHttpHeaders = ObjHttpHeaders(singleValueHeaders, allHeaders)
} }
private fun valuesOf(name: String): List<String> = allHeaders[lookupKey(name)] ?: emptyList() private fun valuesOf(name: String): List<String> = allHeaders[lookupKey(name)] ?: emptyList()
@@ -47,8 +47,10 @@ import net.sergeych.lyngio.net.security.NetAccessDeniedException
import net.sergeych.lyngio.net.security.NetAccessOp import net.sergeych.lyngio.net.security.NetAccessOp
import net.sergeych.lyngio.net.security.NetAccessPolicy import net.sergeych.lyngio.net.security.NetAccessPolicy
import net.sergeych.lyngio.stdlib_included.netLyng import net.sergeych.lyngio.stdlib_included.netLyng
import net.sergeych.lyngio.stdlib_included.net_typesLyng
private const val NET_MODULE_NAME = "lyng.io.net" private const val NET_MODULE_NAME = "lyng.io.net"
internal const val NET_TYPES_MODULE_NAME = "lyng.io.net.types"
fun createNetModule(policy: NetAccessPolicy, scope: Scope): Boolean = fun createNetModule(policy: NetAccessPolicy, scope: Scope): Boolean =
createNetModule(policy, scope.importManager) createNetModule(policy, scope.importManager)
@@ -56,6 +58,7 @@ fun createNetModule(policy: NetAccessPolicy, scope: Scope): Boolean =
fun createNet(policy: NetAccessPolicy, scope: Scope): Boolean = createNetModule(policy, scope) fun createNet(policy: NetAccessPolicy, scope: Scope): Boolean = createNetModule(policy, scope)
fun createNetModule(policy: NetAccessPolicy, manager: ImportManager): Boolean { fun createNetModule(policy: NetAccessPolicy, manager: ImportManager): Boolean {
createNetTypesModule(manager)
if (manager.packageNames.contains(NET_MODULE_NAME)) return false if (manager.packageNames.contains(NET_MODULE_NAME)) return false
manager.addPackage(NET_MODULE_NAME) { module -> manager.addPackage(NET_MODULE_NAME) { module ->
buildNetModule(module, policy) buildNetModule(module, policy)
@@ -65,6 +68,21 @@ fun createNetModule(policy: NetAccessPolicy, manager: ImportManager): Boolean {
fun createNet(policy: NetAccessPolicy, manager: ImportManager): Boolean = createNetModule(policy, manager) fun createNet(policy: NetAccessPolicy, manager: ImportManager): Boolean = createNetModule(policy, manager)
internal fun createNetTypesModule(manager: ImportManager): Boolean {
if (manager.packageNames.contains(NET_TYPES_MODULE_NAME)) return false
manager.addPackage(NET_TYPES_MODULE_NAME) { module ->
buildNetTypesModule(module)
}
return true
}
private suspend fun buildNetTypesModule(module: ModuleScope) {
module.eval(Source(NET_TYPES_MODULE_NAME, net_typesLyng))
val enumValues = NetEnumValues.load(module)
module.addConst("SocketAddress", ObjSocketAddress.type(enumValues))
module.addConst("Datagram", ObjDatagram.type(enumValues))
}
private suspend fun buildNetModule(module: ModuleScope, policy: NetAccessPolicy) { private suspend fun buildNetModule(module: ModuleScope, policy: NetAccessPolicy) {
module.eval(Source(NET_MODULE_NAME, netLyng)) module.eval(Source(NET_MODULE_NAME, netLyng))
val engine = getSystemNetEngine() val engine = getSystemNetEngine()
@@ -164,10 +182,12 @@ private class ObjSocketAddress(
override suspend fun defaultToString(scope: Scope): ObjString = ObjString(renderAddress(address)) override suspend fun defaultToString(scope: Scope): ObjString = ObjString(renderAddress(address))
companion object { companion object {
private val types = mutableMapOf<NetEnumValues, ObjClass>() private data class EnumKey(val ipv4: Obj, val ipv6: Obj)
private val types = mutableMapOf<EnumKey, ObjClass>()
fun type(enumValues: NetEnumValues): ObjClass = fun type(enumValues: NetEnumValues): ObjClass =
types.getOrPut(enumValues) { types.getOrPut(EnumKey(enumValues.ipv4, enumValues.ipv6)) {
object : ObjClass("SocketAddress") { object : ObjClass("SocketAddress") {
override suspend fun callOn(scope: Scope): Obj { override suspend fun callOn(scope: Scope): Obj {
scope.raiseError("SocketAddress cannot be created directly") scope.raiseError("SocketAddress cannot be created directly")
@@ -191,10 +211,12 @@ private class ObjDatagram(
get() = type(enumValues) get() = type(enumValues)
companion object { companion object {
private val types = mutableMapOf<NetEnumValues, ObjClass>() private data class EnumKey(val ipv4: Obj, val ipv6: Obj)
private val types = mutableMapOf<EnumKey, ObjClass>()
fun type(enumValues: NetEnumValues): ObjClass = fun type(enumValues: NetEnumValues): ObjClass =
types.getOrPut(enumValues) { types.getOrPut(EnumKey(enumValues.ipv4, enumValues.ipv6)) {
object : ObjClass("Datagram") { object : ObjClass("Datagram") {
override suspend fun callOn(scope: Scope): Obj { override suspend fun callOn(scope: Scope): Obj {
scope.raiseError("Datagram cannot be created directly") scope.raiseError("Datagram cannot be created directly")
@@ -17,10 +17,16 @@
package net.sergeych.lyng.io.process package net.sergeych.lyng.io.process
import kotlinx.coroutines.async
import kotlinx.coroutines.coroutineScope
import kotlinx.coroutines.flow.Flow import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.toList
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import net.sergeych.lyng.ModuleScope import net.sergeych.lyng.ModuleScope
import net.sergeych.lyng.Scope import net.sergeych.lyng.Scope
import net.sergeych.lyng.ScopeFacade import net.sergeych.lyng.ScopeFacade
import net.sergeych.lyng.Source
import net.sergeych.lyng.miniast.* import net.sergeych.lyng.miniast.*
import net.sergeych.lyng.obj.* import net.sergeych.lyng.obj.*
import net.sergeych.lyng.pacman.ImportManager import net.sergeych.lyng.pacman.ImportManager
@@ -28,6 +34,24 @@ import net.sergeych.lyng.requireScope
import net.sergeych.lyngio.process.* import net.sergeych.lyngio.process.*
import net.sergeych.lyngio.process.security.ProcessAccessDeniedException import net.sergeych.lyngio.process.security.ProcessAccessDeniedException
import net.sergeych.lyngio.process.security.ProcessAccessPolicy import net.sergeych.lyngio.process.security.ProcessAccessPolicy
import net.sergeych.lyngio.stdlib_included.processLyng
import kotlin.Boolean
import kotlin.Exception
import kotlin.Int
import kotlin.String
import kotlin.also
import kotlin.apply
import kotlin.collections.joinToString
import kotlin.collections.listOf
import kotlin.collections.map
import kotlin.collections.mutableListOf
import kotlin.collections.mutableMapOf
import kotlin.collections.toTypedArray
import kotlin.let
import kotlin.takeIf
import kotlin.text.isNotBlank
import kotlin.text.uppercase
import kotlin.to
/** /**
* Install Lyng module `lyng.io.process` into the given scope's ImportManager. * Install Lyng module `lyng.io.process` into the given scope's ImportManager.
@@ -47,6 +71,8 @@ fun createProcessModule(policy: ProcessAccessPolicy, manager: ImportManager): Bo
} }
private suspend fun buildProcessModule(module: ModuleScope, policy: ProcessAccessPolicy) { private suspend fun buildProcessModule(module: ModuleScope, policy: ProcessAccessPolicy) {
module.eval(Source("lyng.io.process", processLyng))
val runner = try { val runner = try {
SecuredLyngProcessRunner(getSystemProcessRunner(), policy) SecuredLyngProcessRunner(getSystemProcessRunner(), policy)
} catch (e: Exception) { } catch (e: Exception) {
@@ -54,6 +80,7 @@ private suspend fun buildProcessModule(module: ModuleScope, policy: ProcessAcces
} }
val runningProcessType = object : ObjClass("RunningProcess") {} val runningProcessType = object : ObjClass("RunningProcess") {}
val commandRunType = object : ObjClass("CommandRun") {}
runningProcessType.apply { runningProcessType.apply {
addFnDoc( addFnDoc(
@@ -149,6 +176,87 @@ private suspend fun buildProcessModule(module: ModuleScope, policy: ProcessAcces
} }
} }
commandRunType.apply {
addPropertyDoc(
name = "command",
doc = "Original shell command or argv-style command display text.",
type = type("lyng.String"),
moduleName = module.packageName,
getter = { ObjString(thisAs<ObjCommandRun>().command) }
)
addPropertyDoc(
name = "out",
doc = "Captured standard output as a string. Captures both stdout and stderr concurrently.",
type = type("lyng.String"),
moduleName = module.packageName,
getter = { ObjString(thisAs<ObjCommandRun>().captureAll().stdoutText) }
)
addPropertyDoc(
name = "err",
doc = "Captured standard error as a string. Captures both stdout and stderr concurrently.",
type = type("lyng.String"),
moduleName = module.packageName,
getter = { ObjString(thisAs<ObjCommandRun>().captureAll().stderrText) }
)
addPropertyDoc(
name = "lines",
doc = "Streaming standard output lines. Use this for large output instead of `out`.",
type = type("lyng.Flow"),
moduleName = module.packageName,
getter = { thisAs<ObjCommandRun>().process.stdout.toLyngFlow(this) }
)
addPropertyDoc(
name = "errorLines",
doc = "Streaming standard error lines. Use this for large stderr output instead of `err`.",
type = type("lyng.Flow"),
moduleName = module.packageName,
getter = { thisAs<ObjCommandRun>().process.stderr.toLyngFlow(this) }
)
addPropertyDoc(
name = "code",
doc = "Exit code after `wait`, `check`, `out`, or `err`; otherwise null.",
type = type("lyng.Int?"),
moduleName = module.packageName,
getter = {
val code = thisAs<ObjCommandRun>().knownExitCode()
code?.toObj() ?: ObjNull
}
)
addPropertyDoc(
name = "ok",
doc = "True if the known exit code is zero, false if non-zero, or null before the process is known to have exited.",
type = type("lyng.Bool?"),
moduleName = module.packageName,
getter = {
val code = thisAs<ObjCommandRun>().knownExitCode()
code?.let { (it == 0).toObj() } ?: ObjNull
}
)
addFnDoc(
name = "wait",
doc = "Wait for the process to exit and return its exit code.",
returns = type("lyng.Int"),
moduleName = module.packageName
) {
thisAs<ObjCommandRun>().waitFor().toObj()
}
addFnDoc(
name = "check",
doc = "Capture output, wait for completion, and fail if the exit code is non-zero.",
returns = type("CommandRun"),
moduleName = module.packageName
) {
val command = thisAs<ObjCommandRun>()
val captured = command.captureAll()
if (captured.exitCode != 0) {
val detail = captured.stderrText.takeIf { it.isNotBlank() } ?: captured.stdoutText
val suffix = detail.takeIf { it.isNotBlank() }?.let { ": $it" } ?: ""
raiseError("command failed with exit code ${captured.exitCode}: ${command.command}$suffix")
}
command
}
}
val platformType = object : ObjClass("Platform") {} val platformType = object : ObjClass("Platform") {}
platformType.apply { platformType.apply {
@@ -197,6 +305,45 @@ private suspend fun buildProcessModule(module: ModuleScope, policy: ProcessAcces
type = type("RunningProcess"), type = type("RunningProcess"),
moduleName = module.packageName moduleName = module.packageName
) )
module.addConstDoc(
name = "CommandRun",
value = commandRunType,
doc = "Shell-script friendly handle for a running command.",
type = type("CommandRun"),
moduleName = module.packageName
)
module.addFnDoc(
"sh",
doc = "Run a command via the system shell and return an active command handle.",
params = listOf(ParamDoc("command", type("lyng.String"))),
returns = type("CommandRun"),
moduleName = module.packageName
) {
if (runner == null) raiseError("Processes are not supported on this platform")
processGuard {
val command = requireOnlyArg<ObjString>().value
ObjCommandRun(commandRunType, command, runner.shell(command))
}
}
module.addFnDoc(
"exec",
doc = "Run an executable with argv-style arguments and return an active command handle.",
params = listOf(ParamDoc("executable", type("lyng.String")), ParamDoc("args", type("lyng.List"))),
returns = type("CommandRun"),
moduleName = module.packageName
) {
if (runner == null) raiseError("Processes are not supported on this platform")
processGuard {
if (args.list.size > 2) {
raiseError("Expected at most 2 arguments, got ${args.list.size}")
}
val executable = requiredArg<ObjString>(0).value
val rawArgs = if (args.list.size >= 2) requiredArg<ObjList>(1) else ObjList(mutableListOf())
val argv = rawArgs.list.map { (it as? ObjString)?.value ?: it.toString() }
ObjCommandRun(commandRunType, listOf(executable, *argv.toTypedArray()).joinToString(" "), runner.execute(executable, argv))
}
}
} }
class ObjRunningProcess( class ObjRunningProcess(
@@ -206,6 +353,49 @@ class ObjRunningProcess(
override fun toString(): String = "RunningProcess($process)" override fun toString(): String = "RunningProcess($process)"
} }
private data class CapturedCommandOutput(
val exitCode: Int,
val stdoutText: String,
val stderrText: String
)
private class ObjCommandRun(
override val objClass: ObjClass,
val command: String,
val process: LyngProcess
) : Obj() {
private val captureMutex = Mutex()
private var captured: CapturedCommandOutput? = null
private var exitCode: Int? = null
fun knownExitCode(): Int? = captured?.exitCode ?: exitCode
suspend fun waitFor(): Int {
captured?.let { return it.exitCode }
exitCode?.let { return it }
return process.waitFor().also { exitCode = it }
}
suspend fun captureAll(): CapturedCommandOutput = captureMutex.withLock {
captured?.let { return@withLock it }
coroutineScope {
val stdout = async { process.stdout.toList().joinToString("\n") }
val stderr = async { process.stderr.toList().joinToString("\n") }
val code = async { process.waitFor() }
CapturedCommandOutput(
exitCode = code.await(),
stdoutText = stdout.await(),
stderrText = stderr.await()
).also {
captured = it
exitCode = it.exitCode
}
}
}
override fun toString(): String = "CommandRun($command)"
}
private suspend inline fun ScopeFacade.processGuard(crossinline block: suspend () -> Obj): Obj { private suspend inline fun ScopeFacade.processGuard(crossinline block: suspend () -> Obj): Obj {
return try { return try {
block() block()
@@ -37,6 +37,7 @@ import net.sergeych.lyng.raiseIllegalOperation
import net.sergeych.lyng.requireNoArgs import net.sergeych.lyng.requireNoArgs
import net.sergeych.lyng.requireScope import net.sergeych.lyng.requireScope
import net.sergeych.lyngio.stdlib_included.wsLyng import net.sergeych.lyngio.stdlib_included.wsLyng
import net.sergeych.lyngio.stdlib_included.ws_typesLyng
import net.sergeych.lyngio.ws.LyngWsEngine import net.sergeych.lyngio.ws.LyngWsEngine
import net.sergeych.lyngio.ws.LyngWsMessage import net.sergeych.lyngio.ws.LyngWsMessage
import net.sergeych.lyngio.ws.LyngWsSession import net.sergeych.lyngio.ws.LyngWsSession
@@ -46,6 +47,7 @@ import net.sergeych.lyngio.ws.security.WsAccessOp
import net.sergeych.lyngio.ws.security.WsAccessPolicy import net.sergeych.lyngio.ws.security.WsAccessPolicy
private const val WS_MODULE_NAME = "lyng.io.ws" private const val WS_MODULE_NAME = "lyng.io.ws"
internal const val WS_TYPES_MODULE_NAME = "lyng.io.ws.types"
fun createWsModule(policy: WsAccessPolicy, scope: Scope): Boolean = fun createWsModule(policy: WsAccessPolicy, scope: Scope): Boolean =
createWsModule(policy, scope.importManager) createWsModule(policy, scope.importManager)
@@ -53,6 +55,7 @@ fun createWsModule(policy: WsAccessPolicy, scope: Scope): Boolean =
fun createWs(policy: WsAccessPolicy, scope: Scope): Boolean = createWsModule(policy, scope) fun createWs(policy: WsAccessPolicy, scope: Scope): Boolean = createWsModule(policy, scope)
fun createWsModule(policy: WsAccessPolicy, manager: ImportManager): Boolean { fun createWsModule(policy: WsAccessPolicy, manager: ImportManager): Boolean {
createWsTypesModule(manager)
if (manager.packageNames.contains(WS_MODULE_NAME)) return false if (manager.packageNames.contains(WS_MODULE_NAME)) return false
manager.addPackage(WS_MODULE_NAME) { module -> manager.addPackage(WS_MODULE_NAME) { module ->
buildWsModule(module, policy) buildWsModule(module, policy)
@@ -62,6 +65,19 @@ fun createWsModule(policy: WsAccessPolicy, manager: ImportManager): Boolean {
fun createWs(policy: WsAccessPolicy, manager: ImportManager): Boolean = createWsModule(policy, manager) fun createWs(policy: WsAccessPolicy, manager: ImportManager): Boolean = createWsModule(policy, manager)
internal fun createWsTypesModule(manager: ImportManager): Boolean {
if (manager.packageNames.contains(WS_TYPES_MODULE_NAME)) return false
manager.addPackage(WS_TYPES_MODULE_NAME) { module ->
buildWsTypesModule(module)
}
return true
}
private suspend fun buildWsTypesModule(module: ModuleScope) {
module.eval(Source(WS_TYPES_MODULE_NAME, ws_typesLyng))
module.addConst("WsMessage", ObjWsMessage.type)
}
private suspend fun buildWsModule(module: ModuleScope, policy: WsAccessPolicy) { private suspend fun buildWsModule(module: ModuleScope, policy: WsAccessPolicy) {
module.eval(Source(WS_MODULE_NAME, wsLyng)) module.eval(Source(WS_MODULE_NAME, wsLyng))
val engine = getSystemWsEngine() val engine = getSystemWsEngine()
@@ -92,7 +108,7 @@ private suspend inline fun ScopeFacade.wsGuard(crossinline block: suspend () ->
} }
} }
private class ObjWsMessage( internal class ObjWsMessage(
private val message: LyngWsMessage, private val message: LyngWsMessage,
) : Obj() { ) : Obj() {
override val objClass: ObjClass override val objClass: ObjClass
@@ -112,6 +128,8 @@ private class ObjWsMessage(
thisAs<ObjWsMessage>().message.data?.let { ObjBuffer(it.toUByteArray()) } ?: ObjNull thisAs<ObjWsMessage>().message.data?.let { ObjBuffer(it.toUByteArray()) } ?: ObjNull
}) })
} }
internal fun from(message: LyngWsMessage): ObjWsMessage = ObjWsMessage(message)
} }
} }
@@ -152,7 +170,7 @@ private class ObjWsSession(
addFn("receive") { addFn("receive") {
val self = thisAs<ObjWsSession>() val self = thisAs<ObjWsSession>()
self.policy.require(WsAccessOp.Receive(self.targetUrl)) self.policy.require(WsAccessOp.Receive(self.targetUrl))
self.session.receive()?.let(::ObjWsMessage) ?: ObjNull self.session.receive()?.let(ObjWsMessage::from) ?: ObjNull
} }
addFn("close") { addFn("close") {
val self = thisAs<ObjWsSession>() val self = thisAs<ObjWsSession>()
@@ -0,0 +1,100 @@
@file:OptIn(io.ktor.utils.io.InternalAPI::class)
package net.sergeych.lyngio.http
import io.ktor.client.HttpClient
import io.ktor.client.call.HttpClientCall
import io.ktor.client.plugins.HttpResponseRedirectEvent
import io.ktor.client.plugins.api.ClientPlugin
import io.ktor.client.plugins.api.Send
import io.ktor.client.plugins.api.createClientPlugin
import io.ktor.client.request.HttpRequestBuilder
import io.ktor.http.HttpHeaders
import io.ktor.http.HttpMethod
import io.ktor.http.HttpStatusCode
import io.ktor.http.authority
import io.ktor.http.isSecure
import io.ktor.http.takeFrom
private val redirectMethods = setOf(HttpMethod.Get, HttpMethod.Head)
internal class HttpRedirectLimitExceededException(
val limit: Int,
val initialUrl: String,
val lastUrl: String,
) : IllegalStateException(
"HTTP_REDIRECT_LIMIT_EXCEEDED: limit=$limit, initialUrl=$initialUrl, lastUrl=$lastUrl"
)
internal class CappedHttpRedirectConfig {
var maxRedirects: Int = LyngHttpConfig.DEFAULT_MAX_REDIRECTS
var checkHttpMethod: Boolean = true
var allowHttpsDowngrade: Boolean = false
}
internal val CappedHttpRedirect: ClientPlugin<CappedHttpRedirectConfig> = createClientPlugin(
"CappedHttpRedirect",
::CappedHttpRedirectConfig,
) {
val maxRedirects = pluginConfig.maxRedirects.also {
require(it >= 0) { "maxRedirects must not be negative" }
}
val checkHttpMethod = pluginConfig.checkHttpMethod
val allowHttpsDowngrade = pluginConfig.allowHttpsDowngrade
suspend fun Send.Sender.followRedirects(
context: HttpRequestBuilder,
origin: HttpClientCall,
client: HttpClient,
): HttpClientCall {
var call = origin
var requestBuilder = context
val initialUrl = origin.request.url.toString()
var redirectCount = 0
while (call.response.status.isRedirectStatus()) {
val location = call.response.headers[HttpHeaders.Location] ?: return call
val previousProtocol = call.request.url.protocol
val previousAuthority = call.request.url.authority
requestBuilder = HttpRequestBuilder().apply {
takeFrom(requestBuilder)
url.parameters.clear()
url.takeFrom(location)
if (!allowHttpsDowngrade && previousProtocol.isSecure() && !url.protocol.isSecure()) {
return call
}
if (previousAuthority != url.authority) {
headers.remove(HttpHeaders.Authorization)
}
}
if (redirectCount >= maxRedirects) {
throw HttpRedirectLimitExceededException(
maxRedirects,
initialUrl,
requestBuilder.url.buildString(),
)
}
redirectCount += 1
client.monitor.raise(HttpResponseRedirectEvent, call.response)
call = proceed(requestBuilder)
}
return call
}
on(Send) { request ->
val origin = proceed(request)
if (checkHttpMethod && origin.request.method !in redirectMethods) {
return@on origin
}
followRedirects(request, origin, client)
}
}
private fun HttpStatusCode.isRedirectStatus(): Boolean = when (value) {
HttpStatusCode.MovedPermanently.value,
HttpStatusCode.Found.value,
HttpStatusCode.TemporaryRedirect.value,
HttpStatusCode.PermanentRedirect.value,
HttpStatusCode.SeeOther.value -> true
else -> false
}
@@ -0,0 +1,34 @@
package net.sergeych.lyngio.http
/** Transport diagnostics, preserved by the Lyng bridge instead of a generic "http error". */
class LyngHttpRequestException(request: LyngHttpRequest, cause: Throwable) :
IllegalStateException(describeHttpFailure(request, cause), cause)
internal fun describeHttpFailure(request: LyngHttpRequest, cause: Throwable): String {
val causes = mutableListOf<Throwable>()
var current: Throwable? = cause
repeat(8) {
val item = current ?: return@repeat
if (causes.any { it === item }) return@repeat
causes += item
current = item.cause
}
val details = causes.joinToString(" <- ") { "${it::class.simpleName ?: "Exception"}: ${it.message ?: "no detail"}" }
val text = details.lowercase()
val (code, advice) = when {
"http_redirect_limit_exceeded" in text -> "HTTP_REDIRECT_LIMIT_EXCEEDED" to
"Do not retry this URL unchanged; inspect the redirect chain or use another source."
"tls" in text || "ssl" in text || "handshake" in text || "certificate" in text -> "HTTP_TLS_FAILED" to
"The HTTPS handshake failed before an HTTP response. Do not retry this URL unchanged; changing its path/query will not repair TLS. Use another HTTPS host or source."
"timeout" in text || "timed out" in text -> "HTTP_TIMEOUT" to
"The request exceeded a time limit. At most a bounded retry may help; avoid repeating it indefinitely."
"unknownhost" in text || "unresolvedaddress" in text || "name or service not known" in text -> "HTTP_DNS_FAILED" to
"The hostname could not be resolved. Verify the hostname or use another source."
"connectexception" in text || "connection refused" in text || "connection reset" in text -> "HTTP_CONNECTION_FAILED" to
"The connection failed before a usable response. Verify the endpoint; avoid repeated unchanged requests."
else -> "HTTP_REQUEST_FAILED" to "Use the cause details to choose another source or a different approach."
}
// Query strings and userinfo can contain credentials; the script already knows the full URL.
val target = request.url.substringBefore('?').substringBefore('#').replace(Regex("//[^/@]+@"), "//[redacted]@")
return "$code: ${request.method.uppercase()} $target\nCause: $details\n$advice"
}
@@ -5,6 +5,8 @@ import io.ktor.client.call.body
import io.ktor.client.engine.HttpClientEngineConfig import io.ktor.client.engine.HttpClientEngineConfig
import io.ktor.client.engine.HttpClientEngineFactory import io.ktor.client.engine.HttpClientEngineFactory
import io.ktor.client.plugins.timeout import io.ktor.client.plugins.timeout
import io.ktor.client.plugins.HttpTimeout
import kotlinx.coroutines.CancellationException
import io.ktor.client.request.HttpRequestBuilder import io.ktor.client.request.HttpRequestBuilder
import io.ktor.client.request.request import io.ktor.client.request.request
import io.ktor.client.request.setBody import io.ktor.client.request.setBody
@@ -14,14 +16,21 @@ import io.ktor.http.takeFrom
internal fun createKtorHttpEngine( internal fun createKtorHttpEngine(
engineFactory: HttpClientEngineFactory<HttpClientEngineConfig>, engineFactory: HttpClientEngineFactory<HttpClientEngineConfig>,
): LyngHttpEngine = KtorLyngHttpEngine(engineFactory) config: LyngHttpConfig,
): LyngHttpEngine = KtorLyngHttpEngine(engineFactory, config)
private class KtorLyngHttpEngine( private class KtorLyngHttpEngine(
engineFactory: HttpClientEngineFactory<HttpClientEngineConfig>, engineFactory: HttpClientEngineFactory<HttpClientEngineConfig>,
config: LyngHttpConfig,
) : LyngHttpEngine { ) : LyngHttpEngine {
private val clientResult = runCatching { private val clientResult = runCatching {
HttpClient(engineFactory) { HttpClient(engineFactory) {
expectSuccess = false expectSuccess = false
followRedirects = false
install(HttpTimeout)
install(CappedHttpRedirect) {
maxRedirects = config.maxRedirects
}
} }
} }
@@ -33,15 +42,22 @@ private class KtorLyngHttpEngine(
throw UnsupportedOperationException(it.message ?: "HTTP client is not supported") throw UnsupportedOperationException(it.message ?: "HTTP client is not supported")
} }
val response = httpClient.request { try {
applyRequest(request) val response = httpClient.request { applyRequest(request) }
return LyngHttpResponse(
status = response.status.value,
statusText = response.status.description,
headers = response.headers.entries().associate { it.key to it.value.toList() },
bodyBytes = response.body<ByteArray>(),
)
} catch (e: CancellationException) {
throw e
} catch (e: HttpRedirectLimitExceededException) {
// Keep the existing typed redirect-limit contract for host callers.
throw e
} catch (e: Exception) {
throw LyngHttpRequestException(request, e)
} }
return LyngHttpResponse(
status = response.status.value,
statusText = response.status.description,
headers = response.headers.entries().associate { it.key to it.value.toList() },
bodyBytes = response.body<ByteArray>(),
)
} }
private fun HttpRequestBuilder.applyRequest(request: LyngHttpRequest) { private fun HttpRequestBuilder.applyRequest(request: LyngHttpRequest) {
@@ -33,6 +33,18 @@ data class LyngHttpResponse(
val bodyBytes: ByteArray, val bodyBytes: ByteArray,
) )
data class LyngHttpConfig(
val maxRedirects: Int = DEFAULT_MAX_REDIRECTS,
) {
init {
require(maxRedirects >= 0) { "maxRedirects must not be negative" }
}
companion object {
const val DEFAULT_MAX_REDIRECTS: Int = 10
}
}
interface LyngHttpEngine { interface LyngHttpEngine {
val isSupported: Boolean val isSupported: Boolean
suspend fun request(request: LyngHttpRequest): LyngHttpResponse suspend fun request(request: LyngHttpRequest): LyngHttpResponse
@@ -46,4 +58,4 @@ internal object UnsupportedHttpEngine : LyngHttpEngine {
} }
} }
expect fun getSystemHttpEngine(): LyngHttpEngine expect fun getSystemHttpEngine(config: LyngHttpConfig = LyngHttpConfig()): LyngHttpEngine
@@ -0,0 +1,50 @@
package net.sergeych.lyngio.http.server
import net.sergeych.lyngio.net.LyngTcpSocket
internal class BufferedSocketReader(
private val socket: LyngTcpSocket,
) {
private var pending = ByteArray(0)
suspend fun readLine(
maxBytes: Int,
overflowStatus: Int,
overflowMessage: String,
): String? {
require(maxBytes > 0) { "maxBytes must be positive" }
val out = ByteArray(maxBytes)
var count = 0
while (true) {
val next = readByte() ?: return if (count == 0) null else out.copyOf(count).decodeToString()
if (next == '\n'.code.toByte()) {
return if (count > 0 && out[count - 1] == '\r'.code.toByte()) {
out.copyOf(count - 1).decodeToString()
} else {
out.copyOf(count).decodeToString()
}
}
if (count >= maxBytes) throw HttpProtocolException(overflowStatus, overflowMessage)
out[count++] = next
}
}
suspend fun readExact(byteCount: Int): ByteArray? {
require(byteCount >= 0) { "byteCount must be non-negative" }
if (byteCount == 0) return ByteArray(0)
while (pending.size < byteCount) {
val chunk = socket.read(maxOf(4096, byteCount - pending.size)) ?: break
if (chunk.isEmpty()) break
pending += chunk
}
if (pending.size < byteCount) return null
val result = pending.copyOfRange(0, byteCount)
pending = pending.copyOfRange(byteCount, pending.size)
return result
}
private suspend fun readByte(): Byte? {
val bytes = readExact(1) ?: return null
return bytes[0]
}
}
@@ -0,0 +1,181 @@
package net.sergeych.lyngio.http.server
internal class HttpProtocolException(
val status: Int,
message: String,
) : IllegalStateException(message)
internal suspend fun parseHttpRequest(
reader: BufferedSocketReader,
config: HttpServerConfig,
): HttpRequest? {
val requestLine = reader.readLine(
maxBytes = config.maxRequestLineBytes,
overflowStatus = 414,
overflowMessage = "request line is too long",
) ?: return null
val requestHead = parseRequestLine(requestLine, config)
val headerEntries = parseHeaders(reader, config)
val headers = HttpHeaders(headerEntries)
validateHost(headers)
val contentLength = parseContentLength(headers, config)
validateUnsupportedRequestFeatures(headers)
val wantsWebSocketUpgrade = isWebSocketUpgrade(requestHead.method, headers)
validateWebSocketUpgradeRequest(headers, requestHead.method, contentLength, wantsWebSocketUpgrade)
val body = if (contentLength != null) {
reader.readExact(contentLength)
?: throw HttpProtocolException(400, "unexpected EOF while reading request body")
} else {
ByteArray(0)
}
return HttpRequest(
head = HttpRequestHead(
method = requestHead.method,
target = requestHead.target,
path = requestHead.path,
queryString = requestHead.queryString,
version = requestHead.version,
headers = headers,
contentLength = contentLength,
wantsClose = headers.containsToken("Connection", "close"),
wantsWebSocketUpgrade = wantsWebSocketUpgrade,
),
body = body,
)
}
private data class ParsedRequestLine(
val method: String,
val target: String,
val path: String,
val queryString: String?,
val version: String,
)
private fun parseRequestLine(line: String, config: HttpServerConfig): ParsedRequestLine {
val firstSpace = line.indexOf(' ')
val lastSpace = line.lastIndexOf(' ')
if (firstSpace <= 0 || lastSpace <= firstSpace || lastSpace == line.lastIndex) {
throw HttpProtocolException(400, "malformed request line")
}
val method = line.substring(0, firstSpace)
val target = line.substring(firstSpace + 1, lastSpace)
val version = line.substring(lastSpace + 1)
if (!method.all(::isHttpTokenChar)) {
throw HttpProtocolException(400, "invalid HTTP method")
}
if (version != "HTTP/1.1") {
throw HttpProtocolException(505, "unsupported HTTP version: $version")
}
if (target.length > config.maxRequestLineBytes) {
throw HttpProtocolException(414, "request target is too long")
}
if (!target.startsWith('/')) {
throw HttpProtocolException(400, "only origin-form request targets are supported")
}
val queryAt = target.indexOf('?')
val path = if (queryAt >= 0) target.substring(0, queryAt) else target
val queryString = if (queryAt >= 0) target.substring(queryAt + 1) else null
return ParsedRequestLine(method = method, target = target, path = path, queryString = queryString, version = version)
}
private suspend fun parseHeaders(
reader: BufferedSocketReader,
config: HttpServerConfig,
): List<HttpHeader> {
val headers = ArrayList<HttpHeader>()
var totalBytes = 0
while (true) {
val line = reader.readLine(
maxBytes = config.maxHeaderBytes,
overflowStatus = 431,
overflowMessage = "request headers are too large",
)
?: throw HttpProtocolException(400, "unexpected EOF while reading headers")
totalBytes += line.length + 2
if (totalBytes > config.maxHeaderBytes) {
throw HttpProtocolException(431, "request headers are too large")
}
if (line.isEmpty()) return headers
if (line.firstOrNull() == ' ' || line.firstOrNull() == '\t') {
throw HttpProtocolException(400, "obsolete folded headers are not supported")
}
val colonAt = line.indexOf(':')
if (colonAt <= 0) throw HttpProtocolException(400, "invalid header syntax")
val name = line.substring(0, colonAt)
if (!name.all(::isHttpTokenChar)) {
throw HttpProtocolException(400, "invalid header name: $name")
}
val value = line.substring(colonAt + 1).trim(' ', '\t')
if (value.any { it == '\r' || it == '\n' || it.code < 0x20 && it != '\t' }) {
throw HttpProtocolException(400, "invalid header value")
}
headers += HttpHeader(name, value)
if (headers.size > config.maxHeaderCount) {
throw HttpProtocolException(431, "too many headers")
}
}
}
private fun validateHost(headers: HttpHeaders) {
val values = headers.all("Host").map(String::trim)
if (values.isEmpty()) throw HttpProtocolException(400, "Host header is required")
if (values.distinct().size > 1) throw HttpProtocolException(400, "conflicting Host header values")
}
private fun parseContentLength(headers: HttpHeaders, config: HttpServerConfig): Int? {
val values = headers.all("Content-Length")
if (values.isEmpty()) return null
val normalized = values.flatMap { raw -> raw.split(',').map(String::trim) }
if (normalized.any { it.isEmpty() }) throw HttpProtocolException(400, "invalid Content-Length")
val distinct = normalized.distinct()
if (distinct.size > 1) throw HttpProtocolException(400, "conflicting Content-Length values")
val parsed = distinct.single().toLongOrNull() ?: throw HttpProtocolException(400, "invalid Content-Length")
if (parsed < 0L || parsed > Int.MAX_VALUE.toLong()) throw HttpProtocolException(400, "invalid Content-Length")
if (parsed > config.maxBodyBytes.toLong()) throw HttpProtocolException(413, "request body is too large")
return parsed.toInt()
}
private fun validateUnsupportedRequestFeatures(headers: HttpHeaders) {
if (headers.all("Transfer-Encoding").isNotEmpty()) {
throw HttpProtocolException(501, "Transfer-Encoding is not supported")
}
if (headers.first("Expect")?.equals("100-continue", ignoreCase = true) == true) {
throw HttpProtocolException(501, "Expect: 100-continue is not supported")
}
val upgrade = headers.first("Upgrade")
if (upgrade != null && !upgrade.equals("websocket", ignoreCase = true)) {
throw HttpProtocolException(501, "unsupported Upgrade value")
}
}
private fun isWebSocketUpgrade(method: String, headers: HttpHeaders): Boolean =
method.equals("GET", ignoreCase = true) &&
headers.first("Upgrade")?.equals("websocket", ignoreCase = true) == true &&
headers.containsToken("Connection", "upgrade")
private fun validateWebSocketUpgradeRequest(
headers: HttpHeaders,
method: String,
contentLength: Int?,
wantsWebSocketUpgrade: Boolean,
) {
if (!wantsWebSocketUpgrade) return
if (!method.equals("GET", ignoreCase = true)) {
throw HttpProtocolException(400, "websocket upgrade requires GET")
}
if (contentLength != null && contentLength != 0) {
throw HttpProtocolException(400, "websocket upgrade request must not include a body")
}
if (headers.first("Sec-WebSocket-Key").isNullOrBlank()) {
throw HttpProtocolException(400, "missing Sec-WebSocket-Key")
}
if (headers.first("Sec-WebSocket-Version") != "13") {
throw HttpProtocolException(400, "unsupported Sec-WebSocket-Version")
}
}
private fun isHttpTokenChar(ch: Char): Boolean =
ch in '0'..'9' || ch in 'A'..'Z' || ch in 'a'..'z' || ch in setOf(
'!', '#', '$', '%', '&', '\'', '*', '+', '-', '.', '^', '_', '`', '|', '~'
)
@@ -0,0 +1,207 @@
package net.sergeych.lyngio.http.server
import net.sergeych.lyngio.net.LyngSocketAddress
import net.sergeych.lyngio.ws.LyngWsMessage
internal data class HttpServerConfig(
val host: String? = "127.0.0.1",
val port: Int = 0,
val backlog: Int = 128,
val reuseAddress: Boolean = true,
val maxRequestLineBytes: Int = 8 * 1024,
val maxHeaderBytes: Int = 32 * 1024,
val maxHeaderCount: Int = 100,
val maxBodyBytes: Int = 1024 * 1024,
val keepAliveTimeoutMillis: Long = 15_000,
)
internal data class HttpHeader(
val name: String,
val value: String,
)
internal class HttpHeaders(
private val headerEntries: List<HttpHeader>,
) {
fun first(name: String): String? =
headerEntries.firstOrNull { it.name.equals(name, ignoreCase = true) }?.value
fun all(name: String): List<String> =
headerEntries.filter { it.name.equals(name, ignoreCase = true) }.map(HttpHeader::value)
fun containsToken(name: String, token: String): Boolean =
all(name).flatMap { value -> value.split(',') }
.any { it.trim().equals(token, ignoreCase = true) }
fun entries(): List<HttpHeader> = headerEntries
}
internal data class HttpRequestHead(
val method: String,
val target: String,
val path: String,
val queryString: String?,
val version: String,
val headers: HttpHeaders,
val contentLength: Int?,
val wantsClose: Boolean,
val wantsWebSocketUpgrade: Boolean,
) {
private var pathPartsParsed = false
private var pathPartsCache: List<String> = emptyList()
private var queryParsed = false
private var queryCache: Map<String, String> = emptyMap()
val pathParts: List<String>
get() {
if (!pathPartsParsed) {
pathPartsCache = parsePathParts(path)
pathPartsParsed = true
}
return pathPartsCache
}
val query: Map<String, String>
get() {
if (!queryParsed) {
queryCache = parseQueryParameters(queryString)
queryParsed = true
}
return queryCache
}
}
internal data class HttpRequest(
val head: HttpRequestHead,
val body: ByteArray,
)
internal data class HttpResponse(
val status: Int,
val reason: String = defaultReason(status),
val headers: List<HttpHeader> = emptyList(),
val body: ByteArray = ByteArray(0),
val close: Boolean = false,
)
internal interface HttpWebSocketSession {
fun isOpen(): Boolean
suspend fun sendText(text: String)
suspend fun sendBytes(data: ByteArray)
suspend fun receive(): LyngWsMessage?
suspend fun close(code: Int = 1000, reason: String = "")
}
internal sealed interface HttpHandlerResult {
data class Response(val response: HttpResponse) : HttpHandlerResult
data class WebSocket(val handler: suspend (HttpWebSocketSession) -> Unit) : HttpHandlerResult
}
internal fun interface HttpHandler {
suspend fun handle(request: HttpRequest): HttpHandlerResult
}
internal interface HttpServer {
fun isOpen(): Boolean
fun localAddress(): LyngSocketAddress
fun close()
}
internal fun parsePathParts(path: String): List<String> {
if (path.isEmpty() || path == "/") return emptyList()
val raw = if (path.startsWith('/')) path.substring(1) else path
if (raw.isEmpty()) return emptyList()
return raw.split('/').map(::decodePathSegment)
}
internal fun parseQueryParameters(queryString: String?): Map<String, String> {
if (queryString.isNullOrEmpty()) return emptyMap()
val result = linkedMapOf<String, String>()
var start = 0
while (start <= queryString.length) {
val nextAmp = queryString.indexOf('&', start).let { if (it >= 0) it else queryString.length }
if (nextAmp > start) {
val part = queryString.substring(start, nextAmp)
val eqAt = part.indexOf('=')
val rawKey = if (eqAt >= 0) part.substring(0, eqAt) else part
val rawValue = if (eqAt >= 0) part.substring(eqAt + 1) else ""
result[decodeQueryComponent(rawKey, plusAsSpace = true)] = decodeQueryComponent(rawValue, plusAsSpace = true)
}
if (nextAmp == queryString.length) break
start = nextAmp + 1
}
return result
}
internal fun decodePathSegment(value: String): String = decodeQueryComponent(value, plusAsSpace = false)
private fun decodeQueryComponent(value: String, plusAsSpace: Boolean): String {
if (value.isEmpty()) return value
val out = StringBuilder(value.length)
val bytes = ArrayList<Byte>()
fun flushBytes() {
if (bytes.isEmpty()) return
out.append(bytes.toByteArray().decodeToString())
bytes.clear()
}
var i = 0
while (i < value.length) {
when (val ch = value[i]) {
'+' -> {
flushBytes()
out.append(if (plusAsSpace) ' ' else '+')
i += 1
}
'%' -> {
val decoded = decodePercentByte(value, i)
if (decoded != null) {
bytes += decoded.first.toByte()
i = decoded.second
} else {
flushBytes()
out.append('%')
i += 1
}
}
else -> {
flushBytes()
out.append(ch)
i += 1
}
}
}
flushBytes()
return out.toString()
}
private fun decodePercentByte(value: String, offset: Int): Pair<Int, Int>? {
if (offset + 2 >= value.length) return null
val hi = value[offset + 1].hexDigitValueOrNull() ?: return null
val lo = value[offset + 2].hexDigitValueOrNull() ?: return null
return ((hi shl 4) or lo) to (offset + 3)
}
private fun Char.hexDigitValueOrNull(): Int? = when (this) {
in '0'..'9' -> code - '0'.code
in 'a'..'f' -> code - 'a'.code + 10
in 'A'..'F' -> code - 'A'.code + 10
else -> null
}
internal fun defaultReason(status: Int): String = when (status) {
101 -> "Switching Protocols"
200 -> "OK"
204 -> "No Content"
400 -> "Bad Request"
404 -> "Not Found"
413 -> "Payload Too Large"
414 -> "URI Too Long"
426 -> "Upgrade Required"
431 -> "Request Header Fields Too Large"
500 -> "Internal Server Error"
501 -> "Not Implemented"
505 -> "HTTP Version Not Supported"
else -> "HTTP $status"
}
@@ -0,0 +1,149 @@
package net.sergeych.lyngio.http.server
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.cancel
import kotlinx.coroutines.launch
import kotlinx.coroutines.withTimeout
import net.sergeych.lyngio.net.LyngNetEngine
import net.sergeych.lyngio.net.LyngSocketAddress
import net.sergeych.lyngio.net.LyngTcpServer
import net.sergeych.lyngio.net.LyngTcpSocket
import net.sergeych.lyngio.net.getSystemNetEngine
internal fun startHttpServer(
config: HttpServerConfig = HttpServerConfig(),
netEngine: LyngNetEngine = getSystemNetEngine(),
handler: HttpHandler,
): HttpServer {
if (!netEngine.isSupported || !netEngine.isTcpServerAvailable) {
throw UnsupportedOperationException("HTTP server is not supported on this runtime")
}
return StartedHttpServer(config, netEngine, handler)
}
private class StartedHttpServer(
private val config: HttpServerConfig,
private val netEngine: LyngNetEngine,
private val handler: HttpHandler,
) : HttpServer {
private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
private var serverRef: LyngTcpServer? = null
private var open = true
init {
scope.launch {
val server = netEngine.tcpListen(
host = config.host,
port = config.port,
backlog = config.backlog,
reuseAddress = config.reuseAddress,
)
serverRef = server
acceptLoop(server)
}
}
override fun isOpen(): Boolean = open && (serverRef?.isOpen() ?: true)
override fun localAddress(): LyngSocketAddress =
serverRef?.localAddress() ?: throw IllegalStateException("server is not bound yet")
override fun close() {
if (!open) return
open = false
serverRef?.close()
scope.cancel()
}
private suspend fun acceptLoop(server: LyngTcpServer) {
try {
while (open && server.isOpen()) {
val socket = try {
server.accept()
} catch (e: CancellationException) {
throw e
} catch (_: Throwable) {
if (!open || !server.isOpen()) break
continue
}
scope.launch {
handleConnection(socket)
}
}
} finally {
open = false
server.close()
}
}
private suspend fun handleConnection(socket: LyngTcpSocket) {
val reader = BufferedSocketReader(socket)
try {
while (socket.isOpen()) {
val request = try {
withTimeout(config.keepAliveTimeoutMillis) {
parseHttpRequest(reader, config)
}
} catch (e: CancellationException) {
throw e
} catch (e: HttpProtocolException) {
safeWriteError(socket, e.status, e.message ?: defaultReason(e.status))
break
} catch (_: Throwable) {
safeWriteError(socket, 400, defaultReason(400))
break
} ?: break
val result = try {
handler.handle(request)
} catch (e: CancellationException) {
throw e
} catch (_: Throwable) {
HttpHandlerResult.Response(HttpResponse(status = 500, close = true))
}
when (result) {
is HttpHandlerResult.Response -> {
val close = request.head.wantsClose || result.response.close
writeHttpResponse(socket, result.response, closeConnection = close)
if (close) break
}
is HttpHandlerResult.WebSocket -> {
if (!request.head.wantsWebSocketUpgrade) {
writeHttpResponse(
socket,
HttpResponse(status = 400, close = true, body = "WebSocket upgrade required".encodeToByteArray()),
closeConnection = true,
)
break
}
val session = upgradeToWebSocket(socket, request)
try {
result.handler(session)
} finally {
session.close()
}
break
}
}
}
} catch (_: CancellationException) {
} finally {
socket.close()
}
}
private suspend fun safeWriteError(socket: LyngTcpSocket, status: Int, message: String) {
try {
writeHttpResponse(
socket,
HttpResponse(status = status, body = message.encodeToByteArray(), close = true),
closeConnection = true,
)
} catch (_: Throwable) {
}
}
}
@@ -0,0 +1,44 @@
package net.sergeych.lyngio.http.server
import net.sergeych.lyngio.net.LyngTcpSocket
internal suspend fun writeHttpResponse(
socket: LyngTcpSocket,
response: HttpResponse,
closeConnection: Boolean,
) {
val body = response.body
val headerLines = LinkedHashMap<String, MutableList<String>>()
response.headers.forEach { header ->
headerLines.getOrPut(header.name) { mutableListOf() }.add(header.value)
}
if (headerLines.keys.none { it.equals("Content-Length", ignoreCase = true) }) {
headerLines["Content-Length"] = mutableListOf(body.size.toString())
}
if (closeConnection) {
val connectionKey = headerLines.keys.firstOrNull { it.equals("Connection", ignoreCase = true) }
if (connectionKey != null) {
headerLines.remove(connectionKey)
}
headerLines["Connection"] = mutableListOf("close")
}
val head = buildString {
append("HTTP/1.1 ")
append(response.status)
append(' ')
append(response.reason)
append("\r\n")
headerLines.forEach { (name, values) ->
values.forEach { value ->
append(name)
append(": ")
append(value)
append("\r\n")
}
}
append("\r\n")
}
socket.writeUtf8(head)
if (body.isNotEmpty()) socket.write(body)
socket.flush()
}
@@ -0,0 +1,295 @@
package net.sergeych.lyngio.http.server
import net.sergeych.lyngio.net.LyngTcpSocket
import net.sergeych.lyngio.ws.LyngWsMessage
import net.sergeych.mp_tools.encodeToBase64
internal suspend fun upgradeToWebSocket(
socket: LyngTcpSocket,
request: HttpRequest,
): HttpWebSocketSession {
val key = request.head.headers.first("Sec-WebSocket-Key")
?: throw HttpProtocolException(400, "missing Sec-WebSocket-Key")
val response = buildString {
append("HTTP/1.1 101 Switching Protocols\r\n")
append("Upgrade: websocket\r\n")
append("Connection: Upgrade\r\n")
append("Sec-WebSocket-Accept: ")
append(websocketAcceptKey(key))
append("\r\n\r\n")
}
socket.writeUtf8(response)
socket.flush()
return SocketHttpWebSocketSession(socket)
}
private class SocketHttpWebSocketSession(
private val socket: LyngTcpSocket,
) : HttpWebSocketSession {
private val reader = BufferedSocketReader(socket)
private var closed = false
private var fragmentedOpcode: Int? = null
private var fragmentedPayload = ByteArray(0)
private var closeSent = false
override fun isOpen(): Boolean = !closed && socket.isOpen()
override suspend fun sendText(text: String) {
ensureOpen()
sendFrame(OPCODE_TEXT, text.encodeToByteArray())
}
override suspend fun sendBytes(data: ByteArray) {
ensureOpen()
sendFrame(OPCODE_BINARY, data)
}
override suspend fun receive(): LyngWsMessage? {
while (!closed) {
val frame = readFrame() ?: run {
release()
return null
}
when (frame.opcode) {
OPCODE_CONTINUATION -> {
val opcode = fragmentedOpcode ?: throw IllegalStateException("unexpected websocket continuation frame")
fragmentedPayload += frame.payload
if (frame.fin) {
val payload = fragmentedPayload
fragmentedOpcode = null
fragmentedPayload = ByteArray(0)
return payload.toMessage(opcode)
}
}
OPCODE_TEXT, OPCODE_BINARY -> {
if (frame.fin) return frame.payload.toMessage(frame.opcode)
fragmentedOpcode = frame.opcode
fragmentedPayload = frame.payload
}
OPCODE_CLOSE -> {
if (!closeSent) {
sendFrame(OPCODE_CLOSE, frame.payload)
closeSent = true
}
release()
return null
}
OPCODE_PING -> sendFrame(OPCODE_PONG, frame.payload)
OPCODE_PONG -> Unit
else -> Unit
}
}
return null
}
override suspend fun close(code: Int, reason: String) {
if (closed) return
val reasonBytes = reason.encodeToByteArray()
val payload = ByteArray(reasonBytes.size + 2)
payload[0] = (code shr 8).toByte()
payload[1] = code.toByte()
reasonBytes.copyInto(payload, destinationOffset = 2)
try {
if (!closeSent) {
sendFrame(OPCODE_CLOSE, payload)
closeSent = true
}
} finally {
release()
}
}
private suspend fun sendFrame(opcode: Int, payload: ByteArray) {
socket.write(buildFrameHeader(opcode, payload.size, masked = false) + payload)
socket.flush()
}
private suspend fun readFrame(): WsFrame? {
val head = reader.readExact(2) ?: return null
val fin = (head[0].toInt() and 0x80) != 0
val opcode = head[0].toInt() and 0x0f
val masked = (head[1].toInt() and 0x80) != 0
val payloadLength = when (val lengthCode = head[1].toInt() and 0x7f) {
126 -> {
val extended = reader.readExact(2) ?: return null
((extended[0].toInt() and 0xff) shl 8) or (extended[1].toInt() and 0xff)
}
127 -> {
val extended = reader.readExact(8) ?: return null
var acc = 0L
extended.forEach { byte ->
acc = (acc shl 8) or (byte.toInt() and 0xff).toLong()
}
require(acc <= Int.MAX_VALUE.toLong()) { "websocket frame is too large" }
acc.toInt()
}
else -> lengthCode
}
if (!masked) throw IllegalStateException("client websocket frames must be masked")
val mask = reader.readExact(4) ?: return null
val payload = if (payloadLength > 0) reader.readExact(payloadLength) ?: return null else ByteArray(0)
payload.indices.forEach { index ->
payload[index] = (payload[index].toInt() xor mask[index % mask.size].toInt()).toByte()
}
return WsFrame(fin = fin, opcode = opcode, payload = payload)
}
private fun ensureOpen() {
if (!isOpen()) throw IllegalStateException("websocket session is closed")
}
private fun release() {
if (closed) return
closed = true
socket.close()
}
}
private data class WsFrame(
val fin: Boolean,
val opcode: Int,
val payload: ByteArray,
)
private fun ByteArray.toMessage(opcode: Int): LyngWsMessage = when (opcode) {
OPCODE_TEXT -> LyngWsMessage(isText = true, text = decodeToString())
OPCODE_BINARY -> LyngWsMessage(isText = false, data = copyOf())
else -> throw IllegalStateException("unsupported websocket opcode: $opcode")
}
private fun websocketAcceptKey(key: String): String =
sha1((key + WS_GUID).encodeToByteArray()).encodeToBase64()
private fun buildFrameHeader(opcode: Int, payloadSize: Int, masked: Boolean): ByteArray {
require(payloadSize >= 0) { "payload size must be non-negative" }
val firstByte = (0x80 or (opcode and 0x0f)).toByte()
val maskBit = if (masked) 0x80 else 0
return when {
payloadSize <= 125 -> byteArrayOf(firstByte, (maskBit or payloadSize).toByte())
payloadSize <= 0xffff -> byteArrayOf(
firstByte,
(maskBit or 126).toByte(),
((payloadSize ushr 8) and 0xff).toByte(),
(payloadSize and 0xff).toByte(),
)
else -> byteArrayOf(
firstByte,
(maskBit or 127).toByte(),
0,
0,
0,
0,
((payloadSize ushr 24) and 0xff).toByte(),
((payloadSize ushr 16) and 0xff).toByte(),
((payloadSize ushr 8) and 0xff).toByte(),
(payloadSize and 0xff).toByte(),
)
}
}
private fun sha1(input: ByteArray): ByteArray {
var h0 = 0x67452301
var h1 = 0xEFCDAB89.toInt()
var h2 = 0x98BADCFE.toInt()
var h3 = 0x10325476
var h4 = 0xC3D2E1F0.toInt()
val msgLen = input.size
val bitLen = msgLen.toLong() * 8L
val totalLen = ((msgLen + 1 + 8 + 63) / 64) * 64
val padded = ByteArray(totalLen).also { buf ->
input.copyInto(buf)
buf[msgLen] = 0x80.toByte()
for (i in 0..7) {
buf[totalLen - 8 + i] = ((bitLen ushr (56 - i * 8)) and 0xff).toByte()
}
}
val words = IntArray(80)
var blockStart = 0
while (blockStart < padded.size) {
for (i in 0..15) {
val off = blockStart + i * 4
words[i] = ((padded[off].toInt() and 0xff) shl 24) or
((padded[off + 1].toInt() and 0xff) shl 16) or
((padded[off + 2].toInt() and 0xff) shl 8) or
(padded[off + 3].toInt() and 0xff)
}
for (i in 16..79) {
val mixed = words[i - 3] xor words[i - 8] xor words[i - 14] xor words[i - 16]
words[i] = (mixed shl 1) or (mixed ushr 31)
}
var a = h0
var b = h1
var c = h2
var d = h3
var e = h4
for (i in 0..19) {
val f = (b and c) or (b.inv() and d)
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0x5A827999 + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
for (i in 20..39) {
val f = b xor c xor d
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0x6ED9EBA1 + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
for (i in 40..59) {
val f = (b and c) or (b and d) or (c and d)
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0x8F1BBCDC.toInt() + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
for (i in 60..79) {
val f = b xor c xor d
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0xCA62C1D6.toInt() + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
h0 += a
h1 += b
h2 += c
h3 += d
h4 += e
blockStart += 64
}
return ByteArray(20).also { out ->
fun putInt(offset: Int, value: Int) {
out[offset] = (value ushr 24).toByte()
out[offset + 1] = (value ushr 16).toByte()
out[offset + 2] = (value ushr 8).toByte()
out[offset + 3] = value.toByte()
}
putInt(0, h0)
putInt(4, h1)
putInt(8, h2)
putInt(12, h3)
putInt(16, h4)
}
}
private const val WS_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
private const val OPCODE_CONTINUATION = 0x0
private const val OPCODE_TEXT = 0x1
private const val OPCODE_BINARY = 0x2
private const val OPCODE_CLOSE = 0x8
private const val OPCODE_PING = 0x9
private const val OPCODE_PONG = 0xA
@@ -17,35 +17,46 @@ import kotlinx.coroutines.channels.ClosedReceiveChannelException
internal fun createKtorWsEngine( internal fun createKtorWsEngine(
engineFactory: HttpClientEngineFactory<HttpClientEngineConfig>, engineFactory: HttpClientEngineFactory<HttpClientEngineConfig>,
): LyngWsEngine = KtorLyngWsEngine(engineFactory) shareClient: Boolean = true,
): LyngWsEngine = KtorLyngWsEngine(engineFactory, shareClient)
private class KtorLyngWsEngine( private class KtorLyngWsEngine(
engineFactory: HttpClientEngineFactory<HttpClientEngineConfig>, engineFactory: HttpClientEngineFactory<HttpClientEngineConfig>,
private val shareClient: Boolean,
) : LyngWsEngine { ) : LyngWsEngine {
private val clientResult = runCatching { private val clientFactory: () -> HttpClient = {
HttpClient(engineFactory) { HttpClient(engineFactory) {
install(WebSockets) install(WebSockets)
} }
} }
private val sharedClientResult = if (shareClient) runCatching { clientFactory() } else null
private val supportProbe = if (!shareClient) runCatching { clientFactory().close() } else null
override val isSupported: Boolean override val isSupported: Boolean
get() = clientResult.isSuccess get() = sharedClientResult?.isSuccess ?: supportProbe?.isSuccess ?: true
override suspend fun connect(url: String, headers: Map<String, String>): LyngWsSession { override suspend fun connect(url: String, headers: Map<String, String>): LyngWsSession {
val client = clientResult.getOrElse { val client = (sharedClientResult?.getOrElse {
throw UnsupportedOperationException(it.message ?: "WebSocket client is not supported") throw UnsupportedOperationException(it.message ?: "WebSocket client is not supported")
} } ?: runCatching { clientFactory() }.getOrElse {
throw UnsupportedOperationException(it.message ?: "WebSocket client is not supported")
})
val session = client.webSocketSession { val session = client.webSocketSession {
url(url) url(url)
headers.forEach { (name, value) -> header(name, value) } headers.forEach { (name, value) -> header(name, value) }
} }
return KtorLyngWsSession(url, session) return KtorLyngWsSession(
targetUrl = url,
session = session,
ownedClient = client.takeUnless { shareClient },
)
} }
} }
private class KtorLyngWsSession( private class KtorLyngWsSession(
private val targetUrl: String, private val targetUrl: String,
private val session: DefaultWebSocketSession, private val session: DefaultWebSocketSession,
private val ownedClient: HttpClient? = null,
) : LyngWsSession { ) : LyngWsSession {
private var closed = false private var closed = false
@@ -55,12 +66,22 @@ private class KtorLyngWsSession(
override suspend fun sendText(text: String) { override suspend fun sendText(text: String) {
ensureOpen() ensureOpen()
session.send(text) try {
session.send(text)
} catch (e: Throwable) {
release()
throw e
}
} }
override suspend fun sendBytes(data: ByteArray) { override suspend fun sendBytes(data: ByteArray) {
ensureOpen() ensureOpen()
session.send(data) try {
session.send(data)
} catch (e: Throwable) {
release()
throw e
}
} }
override suspend fun receive(): LyngWsMessage? { override suspend fun receive(): LyngWsMessage? {
@@ -68,14 +89,17 @@ private class KtorLyngWsSession(
val frame = try { val frame = try {
session.incoming.receive() session.incoming.receive()
} catch (_: ClosedReceiveChannelException) { } catch (_: ClosedReceiveChannelException) {
closed = true release()
return null return null
} catch (e: Throwable) {
release()
throw e
} }
return when (frame) { return when (frame) {
is Frame.Text -> LyngWsMessage(isText = true, text = frame.readText()) is Frame.Text -> LyngWsMessage(isText = true, text = frame.readText())
is Frame.Binary -> LyngWsMessage(isText = false, data = frame.data.copyOf()) is Frame.Binary -> LyngWsMessage(isText = false, data = frame.data.copyOf())
is Frame.Close -> { is Frame.Close -> {
closed = true release()
null null
} }
else -> receive() else -> receive()
@@ -84,11 +108,20 @@ private class KtorLyngWsSession(
override suspend fun close(code: Int, reason: String) { override suspend fun close(code: Int, reason: String) {
if (closed) return if (closed) return
closed = true try {
session.close(CloseReason(code.toShort(), reason)) session.close(CloseReason(code.toShort(), reason))
} finally {
release()
}
} }
private fun ensureOpen() { private fun ensureOpen() {
if (closed) throw IllegalStateException("websocket session is closed") if (closed) throw IllegalStateException("websocket session is closed")
} }
private fun release() {
if (closed) return
closed = true
ownedClient?.close()
}
} }
@@ -0,0 +1,392 @@
package net.sergeych.lyngio.ws
import io.ktor.http.Url
import io.ktor.http.URLProtocol
import net.sergeych.lyngio.net.LyngNetEngine
import net.sergeych.lyngio.net.LyngTcpSocket
import net.sergeych.lyngio.net.getSystemNetEngine
import net.sergeych.mp_tools.encodeToBase64
import kotlin.random.Random
internal fun createSocketWsEngine(
secureFallback: LyngWsEngine = UnsupportedLyngWsEngine,
): LyngWsEngine = SocketLyngWsEngine(getSystemNetEngine(), secureFallback)
private class SocketLyngWsEngine(
private val netEngine: LyngNetEngine,
private val secureFallback: LyngWsEngine,
) : LyngWsEngine {
override val isSupported: Boolean
get() = (netEngine.isSupported && netEngine.isTcpAvailable) || secureFallback.isSupported
override suspend fun connect(url: String, headers: Map<String, String>): LyngWsSession {
val parsedUrl = Url(url)
return when (parsedUrl.protocol.name.lowercase()) {
URLProtocol.WS.name.lowercase() -> connectPlain(parsedUrl, headers)
URLProtocol.WSS.name.lowercase() -> secureFallback.connect(url, headers)
else -> throw UnsupportedOperationException("Unsupported websocket scheme: ${parsedUrl.protocol.name}")
}
}
private suspend fun connectPlain(url: Url, headers: Map<String, String>): LyngWsSession {
if (!netEngine.isSupported || !netEngine.isTcpAvailable) {
throw UnsupportedOperationException("WebSocket client is not supported on this runtime")
}
val socket = netEngine.tcpConnect(url.host, url.port, timeoutMillis = null, noDelay = true)
try {
val key = randomBytes(16).encodeToBase64()
val requestPath = buildRequestPath(url)
val hostHeader = buildHostHeader(url)
val request = buildString {
append("GET ").append(requestPath).append(" HTTP/1.1\r\n")
append("Host: ").append(hostHeader).append("\r\n")
append("Upgrade: websocket\r\n")
append("Connection: Upgrade\r\n")
append("Sec-WebSocket-Key: ").append(key).append("\r\n")
append("Sec-WebSocket-Version: 13\r\n")
headers.forEach { (name, value) ->
append(name).append(": ").append(value).append("\r\n")
}
append("\r\n")
}
socket.writeUtf8(request)
socket.flush()
validateHandshake(socket, key)
return SocketLyngWsSession(url.toString(), socket)
} catch (e: Throwable) {
socket.close()
throw e
}
}
}
private class SocketLyngWsSession(
private val targetUrl: String,
private val socket: LyngTcpSocket,
) : LyngWsSession {
private var closed = false
private var closeSent = false
private var pending = ByteArray(0)
private var fragmentedOpcode: Int? = null
private var fragmentedPayload = ByteArray(0)
override fun isOpen(): Boolean = !closed && socket.isOpen()
override fun url(): String = targetUrl
override suspend fun sendText(text: String) {
ensureOpen()
sendFrame(OPCODE_TEXT, text.encodeToByteArray())
}
override suspend fun sendBytes(data: ByteArray) {
ensureOpen()
sendFrame(OPCODE_BINARY, data)
}
override suspend fun receive(): LyngWsMessage? {
if (closed) return null
while (true) {
val frame = readFrame() ?: run {
release()
return null
}
when (frame.opcode) {
OPCODE_CONTINUATION -> {
val opcode = fragmentedOpcode ?: throw IllegalStateException("unexpected websocket continuation frame")
fragmentedPayload += frame.payload
if (frame.fin) {
val payload = fragmentedPayload
fragmentedOpcode = null
fragmentedPayload = ByteArray(0)
return payload.toMessage(opcode)
}
}
OPCODE_TEXT, OPCODE_BINARY -> {
if (frame.fin) return frame.payload.toMessage(frame.opcode)
fragmentedOpcode = frame.opcode
fragmentedPayload = frame.payload
}
OPCODE_CLOSE -> {
if (!closeSent) {
sendFrame(OPCODE_CLOSE, frame.payload)
closeSent = true
}
release()
return null
}
OPCODE_PING -> sendFrame(OPCODE_PONG, frame.payload)
OPCODE_PONG -> Unit
else -> Unit
}
}
}
override suspend fun close(code: Int, reason: String) {
if (closed) return
val reasonBytes = reason.encodeToByteArray()
val payload = ByteArray(reasonBytes.size + 2)
payload[0] = (code shr 8).toByte()
payload[1] = code.toByte()
reasonBytes.copyInto(payload, destinationOffset = 2)
try {
if (!closeSent) {
sendFrame(OPCODE_CLOSE, payload)
closeSent = true
}
} finally {
release()
}
}
private suspend fun sendFrame(opcode: Int, payload: ByteArray) {
ensureOpen()
val header = buildFrameHeader(opcode, payload.size, masked = true)
val mask = randomBytes(4)
val maskedPayload = payload.copyOf()
maskedPayload.indices.forEach { index ->
maskedPayload[index] = (maskedPayload[index].toInt() xor mask[index % mask.size].toInt()).toByte()
}
socket.write(header + mask + maskedPayload)
socket.flush()
}
private suspend fun readFrame(): WsFrame? {
val head = readExact(2) ?: return null
val fin = (head[0].toInt() and 0x80) != 0
val opcode = head[0].toInt() and 0x0f
val masked = (head[1].toInt() and 0x80) != 0
val payloadLength = when (val lengthCode = head[1].toInt() and 0x7f) {
126 -> {
val extended = readExact(2) ?: return null
((extended[0].toInt() and 0xff) shl 8) or (extended[1].toInt() and 0xff)
}
127 -> {
val extended = readExact(8) ?: return null
var acc = 0L
extended.forEach { byte ->
acc = (acc shl 8) or (byte.toInt() and 0xff).toLong()
}
require(acc <= Int.MAX_VALUE.toLong()) { "websocket frame is too large" }
acc.toInt()
}
else -> lengthCode
}
val mask = if (masked) readExact(4) ?: return null else null
val payload = if (payloadLength > 0) readExact(payloadLength) ?: return null else ByteArray(0)
if (mask != null) {
payload.indices.forEach { index ->
payload[index] = (payload[index].toInt() xor mask[index % mask.size].toInt()).toByte()
}
}
return WsFrame(fin = fin, opcode = opcode, payload = payload)
}
private suspend fun readExact(byteCount: Int): ByteArray? {
while (pending.size < byteCount) {
val chunk = socket.read(maxOf(4096, byteCount - pending.size)) ?: break
if (chunk.isEmpty()) break
pending += chunk
}
if (pending.size < byteCount) return null
val result = pending.copyOfRange(0, byteCount)
pending = pending.copyOfRange(byteCount, pending.size)
return result
}
private fun release() {
if (closed) return
closed = true
socket.close()
}
private fun ensureOpen() {
if (closed || !socket.isOpen()) throw IllegalStateException("websocket session is closed")
}
}
private data class WsFrame(
val fin: Boolean,
val opcode: Int,
val payload: ByteArray,
)
private suspend fun validateHandshake(socket: LyngTcpSocket, key: String) {
val statusLine = socket.readLine() ?: error("websocket handshake failed: missing response status")
require(statusLine.startsWith("HTTP/1.1 101") || statusLine.startsWith("HTTP/1.0 101")) {
"websocket handshake failed: $statusLine"
}
val headers = linkedMapOf<String, String>()
while (true) {
val line = socket.readLine() ?: error("websocket handshake failed: unexpected EOF")
if (line.isBlank()) break
val colonAt = line.indexOf(':')
require(colonAt > 0) { "invalid websocket header: $line" }
val name = line.substring(0, colonAt).trim().lowercase()
val value = line.substring(colonAt + 1).trim()
headers[name] = value
}
require(headers["upgrade"]?.lowercase() == "websocket") { "websocket handshake failed: missing Upgrade header" }
require(headers["connection"]?.lowercase()?.contains("upgrade") == true) {
"websocket handshake failed: missing Connection header"
}
require(headers["sec-websocket-accept"] == websocketAcceptKey(key)) {
"websocket handshake failed: invalid Sec-WebSocket-Accept"
}
}
private fun websocketAcceptKey(key: String): String =
sha1((key + WS_GUID).encodeToByteArray()).encodeToBase64()
private fun buildRequestPath(url: Url): String {
val path = url.encodedPath.ifEmpty { "/" }
val query = url.encodedQuery
return if (query.isBlank()) path else "$path?$query"
}
private fun buildHostHeader(url: Url): String {
val host = if (':' in url.host && !url.host.startsWith("[")) "[${url.host}]" else url.host
return if (url.port == url.protocol.defaultPort) host else "$host:${url.port}"
}
private fun buildFrameHeader(opcode: Int, payloadSize: Int, masked: Boolean): ByteArray {
require(payloadSize >= 0) { "payload size must be non-negative" }
val firstByte = (0x80 or (opcode and 0x0f)).toByte()
val maskBit = if (masked) 0x80 else 0
return when {
payloadSize <= 125 -> byteArrayOf(firstByte, (maskBit or payloadSize).toByte())
payloadSize <= 0xffff -> byteArrayOf(
firstByte,
(maskBit or 126).toByte(),
((payloadSize ushr 8) and 0xff).toByte(),
(payloadSize and 0xff).toByte(),
)
else -> byteArrayOf(
firstByte,
(maskBit or 127).toByte(),
0,
0,
0,
0,
((payloadSize ushr 24) and 0xff).toByte(),
((payloadSize ushr 16) and 0xff).toByte(),
((payloadSize ushr 8) and 0xff).toByte(),
(payloadSize and 0xff).toByte(),
)
}
}
private fun ByteArray.toMessage(opcode: Int): LyngWsMessage = when (opcode) {
OPCODE_TEXT -> LyngWsMessage(isText = true, text = decodeToString())
OPCODE_BINARY -> LyngWsMessage(isText = false, data = copyOf())
else -> throw IllegalStateException("unsupported websocket opcode: $opcode")
}
private fun randomBytes(size: Int): ByteArray = ByteArray(size).also(Random.Default::nextBytes)
private fun sha1(input: ByteArray): ByteArray {
var h0 = 0x67452301
var h1 = 0xEFCDAB89.toInt()
var h2 = 0x98BADCFE.toInt()
var h3 = 0x10325476
var h4 = 0xC3D2E1F0.toInt()
val msgLen = input.size
val bitLen = msgLen.toLong() * 8L
val totalLen = ((msgLen + 1 + 8 + 63) / 64) * 64
val padded = ByteArray(totalLen).also { buf ->
input.copyInto(buf)
buf[msgLen] = 0x80.toByte()
for (i in 0..7) {
buf[totalLen - 8 + i] = ((bitLen ushr (56 - i * 8)) and 0xff).toByte()
}
}
val words = IntArray(80)
var blockStart = 0
while (blockStart < padded.size) {
for (i in 0..15) {
val off = blockStart + i * 4
words[i] = ((padded[off].toInt() and 0xff) shl 24) or
((padded[off + 1].toInt() and 0xff) shl 16) or
((padded[off + 2].toInt() and 0xff) shl 8) or
(padded[off + 3].toInt() and 0xff)
}
for (i in 16..79) {
val mixed = words[i - 3] xor words[i - 8] xor words[i - 14] xor words[i - 16]
words[i] = (mixed shl 1) or (mixed ushr 31)
}
var a = h0
var b = h1
var c = h2
var d = h3
var e = h4
for (i in 0..19) {
val f = (b and c) or (b.inv() and d)
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0x5A827999 + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
for (i in 20..39) {
val f = b xor c xor d
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0x6ED9EBA1 + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
for (i in 40..59) {
val f = (b and c) or (b and d) or (c and d)
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0x8F1BBCDC.toInt() + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
for (i in 60..79) {
val f = b xor c xor d
val temp = ((a shl 5) or (a ushr 27)) + f + e + 0xCA62C1D6.toInt() + words[i]
e = d
d = c
c = (b shl 30) or (b ushr 2)
b = a
a = temp
}
h0 += a
h1 += b
h2 += c
h3 += d
h4 += e
blockStart += 64
}
return ByteArray(20).also { out ->
fun putInt(offset: Int, value: Int) {
out[offset] = (value ushr 24).toByte()
out[offset + 1] = (value ushr 16).toByte()
out[offset + 2] = (value ushr 8).toByte()
out[offset + 3] = value.toByte()
}
putInt(0, h0)
putInt(4, h1)
putInt(8, h2)
putInt(12, h3)
putInt(16, h4)
}
}
private const val WS_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
private const val OPCODE_CONTINUATION = 0x0
private const val OPCODE_TEXT = 0x1
private const val OPCODE_BINARY = 0x2
private const val OPCODE_CLOSE = 0x8
private const val OPCODE_PING = 0x9
private const val OPCODE_PONG = 0xA
@@ -0,0 +1,135 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng.io.db
import kotlinx.coroutines.test.runTest
import net.sergeych.lyng.Compiler
import net.sergeych.lyng.ExecutionError
import net.sergeych.lyng.pacman.ImportManager
import net.sergeych.lyng.Pos
import net.sergeych.lyng.Script
import net.sergeych.lyng.Source
import net.sergeych.lyng.obj.Obj
import net.sergeych.lyng.obj.ObjExternCallable
import net.sergeych.lyng.obj.ObjInt
import net.sergeych.lyng.obj.ObjNull
import net.sergeych.lyng.obj.ObjString
import net.sergeych.lyng.obj.requiredArg
import net.sergeych.lyng.requireScope
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFailsWith
import kotlin.test.assertFalse
import kotlin.test.assertTrue
class LyngDbModuleTest {
@Test
fun testModuleRegistrationIsIdempotent() = runTest {
val importManager = ImportManager()
assertTrue(createDbModule(importManager))
assertFalse(createDbModule(importManager))
}
@Test
fun testOpenDatabaseDispatchesByNormalizedScheme() = runTest {
val scope = Script.newScope()
createDbModule(scope.importManager)
val module = scope.importManager.createModuleScope(Pos.builtIn, "lyng.io.db")
module.callFn(
"registerDatabaseProvider",
ObjString("TeSt"),
ObjExternCallable.fromBridge {
val url = requiredArg<ObjString>(0).value
val params = requiredArg<Obj>(1)
val size = (params.invokeInstanceMethod(requireScope(), "size") as ObjInt).value
ObjString("$url|$size")
}
)
val code = """
import lyng.io.db
openDatabase("TEST:demo", Map("a" => 1, "b" => 2))
""".trimIndent()
val result = Compiler.compile(Source("<db-test>", code), scope.importManager).execute(scope) as ObjString
assertEquals("TEST:demo|2", result.value)
}
@Test
fun testDuplicateSchemeRegistrationFailsCaseInsensitively() = runTest {
val importManager = ImportManager()
createDbModule(importManager)
val module = importManager.createModuleScope(Pos.builtIn, "lyng.io.db")
val scheme = "provider_test"
module.callFn("registerDatabaseProvider", ObjString(scheme), trivialOpener())
val error = try {
module.callFn("registerDatabaseProvider", ObjString(scheme.uppercase()), trivialOpener())
kotlin.test.fail("expected duplicate registration to fail")
} catch (e: ExecutionError) {
e
}
assertTrue(error.errorMessage.contains("already registered"), error.errorMessage)
}
@Test
fun testMalformedUrlFailsWithIllegalArgument() = runTest {
val scope = Script.newScope()
createDbModule(scope.importManager)
val code = """
import lyng.io.db
openDatabase("not-a-url", Map())
""".trimIndent()
val error = assertFailsWith<ExecutionError> {
Compiler.compile(Source("<db-test>", code), scope.importManager).execute(scope)
}
assertEquals("IllegalArgumentException", error.errorObject.objClass.className)
assertTrue(error.errorMessage.contains("Malformed database connection URL"), error.errorMessage)
}
@Test
fun testUnknownSchemeFailsWithDatabaseException() = runTest {
val scope = Script.newScope()
createDbModule(scope.importManager)
val code = """
import lyng.io.db
openDatabase("unknown:demo", Map())
""".trimIndent()
val error = assertFailsWith<ExecutionError> {
Compiler.compile(Source("<db-test>", code), scope.importManager).execute(scope)
}
assertEquals("DatabaseException", error.errorObject.objClass.className)
assertTrue(error.errorMessage.contains("No database provider registered"), error.errorMessage)
}
private suspend fun net.sergeych.lyng.ModuleScope.callFn(name: String, vararg args: Obj): Obj {
val callee = get(name)?.value ?: error("Missing $name in module")
return callee.invoke(this, ObjNull, *args)
}
private fun trivialOpener(): Obj = ObjExternCallable.fromBridge { ObjInt.One }
}
@@ -0,0 +1,196 @@
/*
* Copyright 2026 Sergey S. Chernov real.sergeych@gmail.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package net.sergeych.lyng.io.html
import kotlinx.coroutines.test.runTest
import net.sergeych.lyng.Compiler
import net.sergeych.lyng.Script
import net.sergeych.lyng.Source
import net.sergeych.lyng.obj.ObjString
import net.sergeych.lyng.pacman.ImportManager
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
class LyngHtmlModuleTest {
@Test
fun testModuleRegistrationIsIdempotent() = runTest {
val importManager = ImportManager()
assertTrue(createHtmlModule(importManager))
assertFalse(createHtmlModule(importManager))
}
@Test
fun testModuleCanBeImported() = runTest {
val scope = Script.newScope()
createHtmlModule(scope.importManager)
val result = Compiler.compile(
Source(
"<html-test>",
"""
import lyng.io.html
42
""".trimIndent()
),
scope.importManager
).execute(scope)
assertEquals("42", result.inspect(scope))
}
@Test
fun testHtmlDslBuildsNestedDocument() = runTest {
val scope = Script.newScope()
createHtmlModule(scope.importManager)
val result = Compiler.compile(
Source(
"<html-dsl-test>",
"""
import lyng.io.html
html {
head {
title { +"Demo" }
}
body {
h3 { +"Heading 3" }
p {
attr("data-x", "\"quoted\" & <tag>")
+"Text & <more>"
}
}
}
""".trimIndent()
),
scope.importManager
).execute(scope)
assertEquals(
"<!doctype html><html><head><title>Demo</title></head><body><h3>Heading 3</h3><p data-x=\"&quot;quoted&quot; &amp; &lt;tag&gt;\">Text &amp; &lt;more&gt;</p></body></html>",
(result as ObjString).value
)
}
@Test
fun testHtmlDslSupportsRawAndVoidTags() = runTest {
val scope = Script.newScope()
createHtmlModule(scope.importManager)
val result = Compiler.compile(
Source(
"<html-void-test>",
"""
import lyng.io.html
html {
head {
meta { attr("charset", "utf-8") }
}
body {
div {
id("root")
classes("app shell")
raw("<span>trusted</span>")
br {}
}
}
}
""".trimIndent()
),
scope.importManager
).execute(scope)
assertEquals(
"<!doctype html><html><head><meta charset=\"utf-8\"></head><body><div id=\"root\" class=\"app shell\"><span>trusted</span><br></div></body></html>",
(result as ObjString).value
)
}
@Test
fun testHtmlDslTypedAttributeHelpers() = runTest {
val scope = Script.newScope()
createHtmlModule(scope.importManager)
val result = Compiler.compile(
Source(
"<html-typed-attrs-test>",
"""
import lyng.io.html
html {
head {
metaCharset()
stylesheet("/site.css")
}
body {
nav {
a(href: "/home") { +"Home" }
}
img(src: "/logo.png", alt: "Logo & mark")
input(type: "hidden", name: "token", value: "\"abc\"")
}
}
""".trimIndent()
),
scope.importManager
).execute(scope)
assertEquals(
"<!doctype html><html><head><meta charset=\"utf-8\"><link rel=\"stylesheet\" href=\"/site.css\"></head><body><nav><a href=\"/home\">Home</a></nav><img src=\"/logo.png\" alt=\"Logo &amp; mark\"><input type=\"hidden\" name=\"token\" value=\"&quot;abc&quot;\"></body></html>",
(result as ObjString).value
)
}
@Test
fun testHtmlDslGenericTagsAndFlagAttributes() = runTest {
val scope = Script.newScope()
createHtmlModule(scope.importManager)
val result = Compiler.compile(
Source(
"<html-generic-tag-test>",
"""
import lyng.io.html
html {
body {
tag("custom-element") {
flag("hidden")
+"Secret"
}
voidTag("source") {
attr("srcset", "/image.webp")
attr("type", "image/webp")
}
}
}
""".trimIndent()
),
scope.importManager
).execute(scope)
assertEquals(
"<!doctype html><html><body><custom-element hidden>Secret</custom-element><source srcset=\"/image.webp\" type=\"image/webp\"></body></html>",
(result as ObjString).value
)
}
}
@@ -0,0 +1,59 @@
package net.sergeych.lyng.io.html
import kotlinx.coroutines.test.runTest
import net.sergeych.lyng.Compiler
import net.sergeych.lyng.Script
import net.sergeych.lyng.Source
import net.sergeych.lyng.pacman.ImportManager
import kotlin.test.*
class LyngHtmlParseModuleTest {
private suspend fun execute(source: String) {
val scope = Script.newScope()
createHtmlParseModule(scope)
Compiler.compile(Source("html-parse-test", source.trimIndent()), scope.importManager).execute(scope)
}
@Test fun registrationIsIdempotent() {
val manager = ImportManager()
assertTrue(createHtmlParseModule(manager))
assertFalse(createHtmlParseModule(manager))
}
@Test fun parsesMalformedHtmlAndExtractsTextWithoutChangingDom() = runTest {
execute("""
import lyng.io.html.parse
val doc = Html.parse("<title>A &amp; B</title><style>bad css</style><body><h1>Heading</h1><p>Hello <b>world</b> &amp; &#65;<br>Next<p>Last<script>bad script</script><div hidden>secret</div><template>bad template</template><noscript>bad fallback</noscript>")
assert(doc.title == "A & B")
assert(doc.text() == "Heading\nHello world & A\nNext\nLast")
assert(doc.select("p").size == 2)
assert(doc.select("script").size == 1)
assert(doc.select("#missing").size == 0)
assert(doc.text() == doc.text())
""")
}
@Test fun selectorsAttributesBaseAndNestedSelection() = runTest {
execute("""
import lyng.io.html.parse
val doc = Html.parse("<base href='/docs/'><article><a class='ref' href='intro?a=1&amp;b=2'>Read &amp; learn</a></article>", "https://example.com/start")
val articles = doc.select("article")
val article = articles[0] as HtmlElement
val links = article.select("a.ref[href]")
val link = links[0] as HtmlElement
assert(link.text() == "Read & learn")
assert(link.attr("href") == "intro?a=1&b=2")
assert(link.absUrl("href") == "https://example.com/docs/intro?a=1&b=2")
assert(link.attr("missing") == "")
assert(link.absUrl("missing") == "")
val fragment = Html.parse("<p>One</p><p>Two</p>")
assert(fragment.text() == "One\nTwo")
assert(fragment.title == "")
assert(Html.parse("").text() == "")
""")
}
@Test fun badSelectorFailsClearly() = runTest {
assertFails { execute("import lyng.io.html.parse\nHtml.parse(\"<p>Hi\").select(\"[\")") }
}
}
@@ -0,0 +1,25 @@
package net.sergeych.lyngio.http
import kotlin.test.*
class HttpRequestFailureTest {
private val request = LyngHttpRequest("GET", "https://user:secret@example.com/page?key=private#fragment")
@Test fun savedFatalTlsAlertGetsSpecificCauseAndNoUnchangedRetryAdvice() {
val saved = IllegalStateException("Received alert during handshake. Level: FATAL, code: InternalError")
val message = LyngHttpRequestException(request, saved).message!!
assertTrue(message.startsWith("HTTP_TLS_FAILED: GET https://[redacted]@example.com/page"))
assertTrue(message.contains("FATAL, code: InternalError"))
assertTrue(message.contains("Do not retry this URL unchanged"))
assertTrue(message.contains("changing its path/query will not repair TLS"))
assertFalse(message.contains("secret")); assertFalse(message.contains("private"))
}
@Test fun nestedCausesAndTimeoutDnsAndConnectionFailuresRemainDistinct() {
val nested = IllegalStateException("request failed", IllegalArgumentException("TLS certificate rejected"))
assertTrue(describeHttpFailure(request, nested).contains("HTTP_TLS_FAILED"))
for ((detail, code) in listOf("request timed out" to "HTTP_TIMEOUT",
"UnknownHostException: example.com" to "HTTP_DNS_FAILED",
"Connection refused" to "HTTP_CONNECTION_FAILED")) {
assertTrue(describeHttpFailure(request, IllegalStateException(detail)).startsWith(code))
}
}
}
@@ -0,0 +1,155 @@
package net.sergeych.lyngio.http.server
import net.sergeych.lyngio.net.LyngIpVersion
import net.sergeych.lyngio.net.LyngSocketAddress
import net.sergeych.lyngio.net.LyngTcpSocket
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFailsWith
class HttpParserTest {
@Test
fun tooLargeHeadersMapTo431() = kotlinx.coroutines.test.runTest {
val request = buildString {
append("GET / HTTP/1.1\r\n")
append("Host: localhost\r\n")
append("X-Big: ")
append("a".repeat(64))
append("\r\n\r\n")
}
val error = assertFailsWith<HttpProtocolException> {
parse(request, HttpServerConfig(maxHeaderBytes = 32))
}
assertEquals(431, error.status)
}
@Test
fun conflictingDuplicateHostIsRejected() = kotlinx.coroutines.test.runTest {
val error = assertFailsWith<HttpProtocolException> {
parse(
"GET / HTTP/1.1\r\n" +
"Host: one.example\r\n" +
"Host: two.example\r\n\r\n"
)
}
assertEquals(400, error.status)
}
@Test
fun conflictingDuplicateContentLengthIsRejected() = kotlinx.coroutines.test.runTest {
val error = assertFailsWith<HttpProtocolException> {
parse(
"POST /echo HTTP/1.1\r\n" +
"Host: localhost\r\n" +
"Content-Length: 4\r\n" +
"Content-Length: 5\r\n\r\nping!"
)
}
assertEquals(400, error.status)
}
@Test
fun malformedRequestLineIsRejected() = kotlinx.coroutines.test.runTest {
val error = assertFailsWith<HttpProtocolException> {
parse("GET /only-two-parts\r\nHost: localhost\r\n\r\n")
}
assertEquals(400, error.status)
}
@Test
fun identicalDuplicateContentLengthIsAccepted() = kotlinx.coroutines.test.runTest {
val request = parse(
"POST /echo HTTP/1.1\r\n" +
"Host: localhost\r\n" +
"Content-Length: 4\r\n" +
"Content-Length: 4\r\n\r\nping"
)
assertEquals("POST", request.head.method)
assertEquals("/echo", request.head.path)
assertEquals(4, request.head.contentLength)
assertEquals("ping", request.body.decodeToString())
}
@Test
fun queryStringAndDecodedQueryMapAreExposed() = kotlinx.coroutines.test.runTest {
val request = parse(
"GET /echo?a=1&b=hello+world&b=last&utf=%D1%82%D0%B5%D1%81%D1%82&bad=%GG%2&flag HTTP/1.1\r\n" +
"Host: localhost\r\n\r\n"
)
assertEquals("a=1&b=hello+world&b=last&utf=%D1%82%D0%B5%D1%81%D1%82&bad=%GG%2&flag", request.head.queryString)
assertEquals("1", request.head.query["a"])
assertEquals("last", request.head.query["b"])
assertEquals("тест", request.head.query["utf"])
assertEquals("%GG%2", request.head.query["bad"])
assertEquals("", request.head.query["flag"])
}
@Test
fun missingQueryProducesEmptyMap() = kotlinx.coroutines.test.runTest {
val request = parse(
"GET /echo HTTP/1.1\r\n" +
"Host: localhost\r\n\r\n"
)
assertEquals(null, request.head.queryString)
assertEquals(emptyMap(), request.head.query)
}
@Test
fun pathPartsAreLazyDecodedWithoutPlusTranslation() = kotlinx.coroutines.test.runTest {
val request = parse(
"GET /one/two%20words/a+b/%GG/%D1%82%D0%B5%D1%81%D1%82 HTTP/1.1\r\n" +
"Host: localhost\r\n\r\n"
)
assertEquals(listOf("one", "two words", "a+b", "%GG", "тест"), request.head.pathParts)
}
private suspend fun parse(
rawRequest: String,
config: HttpServerConfig = HttpServerConfig(),
): HttpRequest {
val socket = FakeTcpSocket(rawRequest.encodeToByteArray())
val reader = BufferedSocketReader(socket)
return parseHttpRequest(reader, config) ?: error("expected parsed request")
}
}
private class FakeTcpSocket(
source: ByteArray,
) : LyngTcpSocket {
private var input = source
private var output = ByteArray(0)
private var open = true
override fun isOpen(): Boolean = open
override fun localAddress(): LyngSocketAddress =
LyngSocketAddress("127.0.0.1", 8080, LyngIpVersion.IPV4, resolved = true)
override fun remoteAddress(): LyngSocketAddress =
LyngSocketAddress("127.0.0.1", 12345, LyngIpVersion.IPV4, resolved = true)
override suspend fun read(maxBytes: Int): ByteArray? {
if (!open || input.isEmpty()) return null
val count = minOf(maxBytes, input.size)
val chunk = input.copyOfRange(0, count)
input = input.copyOfRange(count, input.size)
return chunk
}
override suspend fun readLine(): String? = error("BufferedSocketReader should not call LyngTcpSocket.readLine()")
override suspend fun write(data: ByteArray) {
output += data
}
override suspend fun writeUtf8(text: String) {
output += text.encodeToByteArray()
}
override suspend fun flush() = Unit
override fun close() {
open = false
}
}
@@ -0,0 +1,242 @@
package net.sergeych.lyngio.http.server
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeout
import net.sergeych.lyngio.net.LyngTcpSocket
import net.sergeych.lyngio.net.getSystemNetEngine
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
class HttpServerLoopbackTest {
@Test
fun simpleGetReturnsResponse() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable || !engine.isTcpServerAvailable) return@runBlocking
withTimeout(10_000) {
val server = startHttpServer { request ->
HttpHandlerResult.Response(
HttpResponse(
status = 200,
headers = listOf(HttpHeader("Content-Type", "text/plain")),
body = "hello:${request.head.path}".encodeToByteArray(),
)
)
}
try {
val port = waitForPort(server)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
client.writeUtf8("GET /demo HTTP/1.1\r\nHost: localhost\r\n\r\n")
client.flush()
val text = readHttpResponse(client)
assertTrue(text.startsWith("HTTP/1.1 200 OK\r\n"), text)
assertTrue(text.contains("Content-Type: text/plain\r\n"), text)
assertTrue(text.endsWith("hello:/demo"), text)
} finally {
client.close()
}
} finally {
server.close()
}
}
}
@Test
fun keepAliveServesTwoRequestsOnOneSocket() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable || !engine.isTcpServerAvailable) return@runBlocking
withTimeout(10_000) {
val server = startHttpServer { request ->
HttpHandlerResult.Response(
HttpResponse(status = 200, body = request.head.path.encodeToByteArray())
)
}
try {
val port = waitForPort(server)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
client.writeUtf8(
"GET /one HTTP/1.1\r\nHost: localhost\r\n\r\n" +
"GET /two HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n"
)
client.flush()
val first = readHttpResponse(client)
val second = readHttpResponse(client)
assertTrue(first.endsWith("/one"), first)
assertTrue(second.contains("Connection: close\r\n"), second)
assertTrue(second.endsWith("/two"), second)
} finally {
client.close()
}
} finally {
server.close()
}
}
}
@Test
fun postWithContentLengthReadsBody() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable || !engine.isTcpServerAvailable) return@runBlocking
withTimeout(10_000) {
val server = startHttpServer { request ->
HttpHandlerResult.Response(
HttpResponse(status = 200, body = (request.head.method + ":" + request.body.decodeToString()).encodeToByteArray())
)
}
try {
val port = waitForPort(server)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
client.writeUtf8(
"POST /echo HTTP/1.1\r\nHost: localhost\r\nContent-Length: 4\r\nConnection: close\r\n\r\nping"
)
client.flush()
val text = readHttpResponse(client)
assertTrue(text.endsWith("POST:ping"), text)
} finally {
client.close()
}
} finally {
server.close()
}
}
}
@Test
fun transferEncodingIsRejected() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable || !engine.isTcpServerAvailable) return@runBlocking
withTimeout(10_000) {
val server = startHttpServer { _ ->
HttpHandlerResult.Response(HttpResponse(status = 200, body = "ok".encodeToByteArray()))
}
try {
val port = waitForPort(server)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
client.writeUtf8(
"POST /x HTTP/1.1\r\nHost: localhost\r\nTransfer-Encoding: chunked\r\n\r\n"
)
client.flush()
val text = readHttpResponse(client)
assertTrue(text.startsWith("HTTP/1.1 501 Not Implemented\r\n"), text)
} finally {
client.close()
}
} finally {
server.close()
}
}
}
@Test
fun websocketUpgradeEchoesText() = runBlocking {
val engine = getSystemNetEngine()
if (!engine.isSupported || !engine.isTcpAvailable || !engine.isTcpServerAvailable) return@runBlocking
withTimeout(10_000) {
val server = startHttpServer { request ->
if (request.head.path != "/ws") {
HttpHandlerResult.Response(HttpResponse(status = 404, close = true))
} else {
HttpHandlerResult.WebSocket { session ->
val message = session.receive() ?: return@WebSocket
session.sendText("echo:${message.text}")
}
}
}
try {
val port = waitForPort(server)
val client = engine.tcpConnect("127.0.0.1", port, 2_000, true)
try {
val key = "dGhlIHNhbXBsZSBub25jZQ=="
client.writeUtf8(
"GET /ws HTTP/1.1\r\n" +
"Host: localhost\r\n" +
"Upgrade: websocket\r\n" +
"Connection: Upgrade\r\n" +
"Sec-WebSocket-Key: $key\r\n" +
"Sec-WebSocket-Version: 13\r\n\r\n"
)
client.flush()
val headers = ArrayList<String>()
while (true) {
val line = client.readLine() ?: break
if (line.isEmpty()) break
headers += line
}
assertEquals("HTTP/1.1 101 Switching Protocols", headers.first())
sendMaskedTextFrame(client, "ping")
val reply = readServerTextFrame(client)
assertEquals("echo:ping", reply)
} finally {
client.close()
}
} finally {
server.close()
}
}
}
private suspend fun waitForPort(server: HttpServer): Int {
repeat(100) {
runCatching { return server.localAddress().port }
kotlinx.coroutines.delay(10)
}
error("server did not bind in time")
}
private suspend fun readHttpResponse(client: LyngTcpSocket): String {
val statusLine = client.readLine() ?: error("missing status line")
val headers = linkedMapOf<String, String>()
while (true) {
val line = client.readLine() ?: error("unexpected EOF in response headers")
if (line.isEmpty()) break
val colonAt = line.indexOf(':')
if (colonAt > 0) headers[line.substring(0, colonAt)] = line.substring(colonAt + 1).trim()
}
val bodyLength = headers["Content-Length"]?.toIntOrNull() ?: 0
val body = if (bodyLength > 0) readExact(client, bodyLength).decodeToString() else ""
return buildString {
append(statusLine).append("\r\n")
headers.forEach { (name, value) -> append(name).append(": ").append(value).append("\r\n") }
append("\r\n")
append(body)
}
}
private suspend fun sendMaskedTextFrame(client: LyngTcpSocket, text: String) {
val payload = text.encodeToByteArray()
val mask = byteArrayOf(1, 2, 3, 4)
val masked = payload.copyOf()
masked.indices.forEach { index ->
masked[index] = (masked[index].toInt() xor mask[index % mask.size].toInt()).toByte()
}
val frame = byteArrayOf(0x81.toByte(), (0x80 or payload.size).toByte()) + mask + masked
client.write(frame)
client.flush()
}
private suspend fun readServerTextFrame(client: LyngTcpSocket): String {
val head = readExact(client, 2)
val len = head[1].toInt() and 0x7f
val payload = if (len > 0) readExact(client, len) else ByteArray(0)
return payload.decodeToString()
}
private suspend fun readExact(client: LyngTcpSocket, size: Int): ByteArray {
var pending = ByteArray(0)
while (pending.size < size) {
val chunk = client.read(size - pending.size) ?: error("unexpected EOF")
pending += chunk
}
return pending
}
}
@@ -2,4 +2,4 @@ package net.sergeych.lyngio.http
import io.ktor.client.engine.darwin.Darwin import io.ktor.client.engine.darwin.Darwin
actual fun getSystemHttpEngine(): LyngHttpEngine = createKtorHttpEngine(Darwin) actual fun getSystemHttpEngine(config: LyngHttpConfig): LyngHttpEngine = createKtorHttpEngine(Darwin, config)
Loaded 100 of 258 files, more files were not shown because too many files have changed in this diff. Show more