lyng/notes/db/sqlite_implementation_plan.md

# SQLite provider implementation plan

Implementation checklist for the first concrete DB provider:
- module: `lyng.io.db.sqlite`
- targets: JVM and Native
- role: reference implementation for the core `lyng.io.db` contract

## Scope

In scope for the first implementation:
- provider registration on module import
- `sqlite:` URL dispatch through `openDatabase(...)`
- typed `openSqlite(...)` helper
- `Database` and `SqlTransaction` implementation
- result-set implementation
- row and column metadata conversion
- SQLite savepoint-based nested transactions
- generated keys for `execute(...)`
- JVM and Native test coverage for the core portable contract

Out of scope for the first implementation:
- schema inspection APIs beyond result-set column metadata
- public prepared statement API
- batch API
- JS support
- heuristic temporal parsing
- heuristic decimal parsing

## Proposed module layout

Core/public declarations:
- `.lyng` declaration source for `lyng.io.db`
- `.lyng` declaration source for `lyng.io.db.sqlite`

Backend/runtime implementation:
- common transaction/result-set abstractions in `commonMain` where possible
- JVM SQLite implementation in `jvmMain`
- Native SQLite implementation in `nativeMain`

## Milestone 1: core DB module skeleton

1. Move or copy the current DB declarations from `notes/db/lyngdb.lyng` into the
   real module declaration source location.
2. Add the provider registry runtime for:
   - normalized lowercase scheme lookup
   - duplicate registration failure
   - unknown-scheme failure
3. Implement generic `openDatabase(connectionUrl, extraParams)` dispatch.
4. Add basic tests for:
   - successful provider registration
   - duplicate scheme registration failure
   - unknown scheme failure
   - malformed URL failure

## Milestone 2: SQLite provider skeleton

1. Create `lyng.io.db.sqlite` declaration source with:
   - typed `openSqlite(...)`
   - provider-specific documentation
2. Register `sqlite` scheme at module initialization time.
3. Parse supported URL forms:
   - `sqlite::memory:`
   - `sqlite:relative/path.db`
   - `sqlite:/absolute/path.db`
4. Convert typed helper arguments into the same normalized open options used by
   the generic `openDatabase(...)` path.
5. Add tests for:
   - import-time registration
   - typed helper open
   - generic URL-based open
   - invalid URL handling

## Milestone 3: JVM backend

Implementation strategy:
- use SQLite JDBC under the hood
- keep JDBC fully internal to the provider
- preserve the Lyng-facing transaction/result contracts rather than exposing
  JDBC semantics directly

Steps:
1. Create JVM `Database` implementation that stores normalized SQLite config.
2. For each outer `Database.transaction {}`:
   - open/acquire one JDBC connection
   - configure connection-level options
   - begin transaction
   - commit on normal exit
   - rollback on uncaught exception
   - close/release connection
   - preserve error precedence:
     - original user exception stays primary on rollback failure
     - rollback failure becomes primary for intentional `RollbackException`
     - commit failure is primary on normal-exit commit failure
3. For nested `SqlTransaction.transaction {}`:
   - create savepoint
   - release savepoint on success
   - rollback to savepoint on failure
   - preserve the same primary/secondary exception rules for savepoint rollback
     failures
4. Implement `select(...)`:
   - bind positional `?` parameters
   - expose result rows through `ResultSet`
   - preserve transaction-scoped lifetime
   - invalidate both the result set and any rows obtained from it when the
     owning transaction ends
5. Implement `execute(...)`:
   - bind positional parameters
   - collect affected row count
   - expose generated keys when supported
6. Implement column metadata normalization:
   - output column label
   - nullable flag where available
   - portable `SqlType`
   - backend native type name
7. Add JVM tests for:
   - transaction commit
   - transaction rollback
   - nested transaction savepoint behavior
   - rollback failure precedence
   - commit failure precedence
   - row lookup by index and name
   - ambiguous name failure
   - result-set use-after-transaction failure
   - row use-after-transaction failure
   - generated keys
   - `RETURNING` via `select(...)` if the backend supports it

## Milestone 4: Native backend

Implementation strategy:
- use direct SQLite C bindings
- keep semantics aligned with the JVM backend

Steps:
1. Create Native `Database` implementation that stores normalized SQLite config.
2. For each outer transaction:
   - open one SQLite handle if needed
   - configure pragmas/options
   - begin transaction
   - commit or rollback
3. Implement nested transactions with SQLite savepoints.
4. Implement prepared statement lifecycle internally for `select(...)` and
   `execute(...)`.
5. Implement result-set iteration and statement finalization.
6. Implement the same value-conversion policy as JVM:
   - exact integer mapping
   - `Double`
   - `String`
   - `Buffer`
   - schema-driven `Bool`
   - schema-driven `Decimal`
   - schema-driven temporal parsing
7. Add Native tests matching the JVM behavioral suite as closely as possible.

## Milestone 5: conversion policy

1. Normalize integer reads:
   - return `Int`
2. Normalize floating-point reads to `Double`.
3. Normalize text reads to `String`.
4. Normalize blob reads to `Buffer`.
5. Parse `Decimal` only for declared/native `DECIMAL` / `NUMERIC` columns.
   - bind Lyng `Decimal` as canonical text using existing Decimal formatting
   - read back with the existing Decimal parser
6. Parse `Bool` only for declared/native `BOOLEAN` / `BOOL` columns:
   - prefer integer `0` / `1`
   - also accept legacy text `true`, `false`, `t`, `f` case-insensitively
   - always write `Bool` as integer `0` / `1`
7. Parse temporal values only from an explicit normalized type-name whitelist:
   - `DATE`
   - `DATETIME`
   - `TIMESTAMP`
   - `TIMESTAMP WITH TIME ZONE`
   - `TIMESTAMPTZ`
   - `DATETIME WITH TIME ZONE`
   - `TIME`
   - `TIME WITHOUT TIME ZONE`
   - `TIME WITH TIME ZONE`
8. Never heuristically parse arbitrary string or numeric values into temporal or
   decimal values.
9. If a declared strong type (`Bool`, `Decimal`, `Date`, `DateTime`, `Instant`)
   cannot be converted from the stored value, fail with `SqlExecutionException`.
10. Add tests for each conversion rule.

## Milestone 6: result-set contract

1. Ensure `ResultSet.columns` is available before iteration.
2. Implement name lookup using result-column labels.
3. Throw `SqlUsageException` for:
   - invalid row index
   - missing column name
   - ambiguous column name
   - use after transaction end
4. Implement cheap `isEmpty()` where practical.
5. Implement `size()` separately, allowing buffering/consumption when required.
6. Verify resource cleanup on:
   - full iteration
   - canceled iteration
   - transaction rollback

## Milestone 7: provider options

Support these typed helper options first:
- `readOnly`
- `createIfMissing`
- `foreignKeys`
- `busyTimeoutMillis`

Expected behavior:
- `foreignKeys` defaults to `true`
- `busyTimeoutMillis` has a non-zero sensible default
- `readOnly` is explicit
- `createIfMissing` is explicit

Add tests for option handling where backend behavior is observable.

## Milestone 8: documentation and examples

1. Add user docs for:
   - importing `lyng.io.db.sqlite`
   - generic `openDatabase(...)`
   - typed `openSqlite(...)`
   - transaction usage
   - nested transactions
2. Add small sample snippets for:
   - in-memory DB
   - file-backed DB
   - schema creation
   - insert/select/update
   - rollback on exception

## Testing priorities

Highest priority behavioral tests:
1. provider registration and scheme dispatch
2. outer transaction commit/rollback
3. nested savepoint semantics
4. row metadata and name lookup behavior
5. generated keys for inserts
6. result-set lifetime rules
7. value conversion rules
8. helper vs generic-open parity

## Risks

Main implementation risks:
- keeping JVM and Native value-conversion behavior identical
- correctly enforcing result-set lifetime across both backends
- SQLite temporal conversion ambiguity
- JDBC metadata differences on JVM
- native statement/finalizer lifecycle bugs

## Suggested implementation order

1. core registry runtime
2. SQLite `.lyng` provider declarations
3. JVM SQLite backend
4. JVM test suite
5. Native SQLite backend
6. shared behavioral test suite refinement
7. documentation/examples