0.1.1-SNAPSHOT: storages, sync tools, docs

README.md

@@ -1,12 +1,17 @@
 # Binary tools and BiPack serializer
 
-Multiplatform binary tools collection, including portable serialization of the compact and fast [Bipack] format, and many useful tools to work with binary data, like CRC family checksums, dumps, etc. It works well also in the browser and in native targets.
+Multiplatform binary tools collection, including portable serialization of the compact and fast [Bipack] format, and
+many useful tools to work with binary data, like CRC family checksums, dumps, etc. It works well also in the browser and
+in native targets.
 
 # Recent changes
 
+- 0.1.1: added serialized KVStorage with handy implementation on JVM and JS platforms and some required synchronization
+  tools.
+-
 - 0.1.0: uses modern kotlin 1.9.*, fixes problem with singleton or empty/object serialization
 
-last 1.8 version is 0.0.8, some fixes are not yet backported to it pls leave an issue of needed. 
+The last 1.8-based version is 0.0.8. Some fixes are not yet backported to it pls leave an issue of needed.
 
 # Usage
 
@@ -19,7 +24,7 @@ repositories {
 }
 ```
 
-And add dependecy to the proper place in yuor project like this:
+And add dependency to the proper place in your project like this:
 
 ```kotlin
 dependencies {
@@ -28,6 +33,66 @@ dependencies {
 }
 ```
 
+## Calculating CRCs:
+
+~~~kotlin
+CRC.crc32("Hello".encodeToByteArray())
+CRC.crc16("happy".encodeToByteArray())
+CRC.crc8("world".encodeToByteArray())
+~~~
+
+## Binary effective serialization with Bipack:
+
+~~~kotlin
+@Serializable
+data class Foo(val bar: String,buzz: Int)
+
+val foo = Foo("bar", 42)
+val bytes = BipackEncoder.encode(foo)
+val bar: Foo = BipackDecoder.decode(bytes)
+assertEquals(foo, bar)
+~~~
+
+## Bipack-based auto-serializing storage:
+
+Allows easily storing whatever `@Serializable` data type using delegates
+and more:
+
+~~~kotlin
+val storage = defaultNamedStorage("test_mp_bintools")
+
+var foo by s1("unknown") // default value makes it a String
+foo = "bar"
+
+// nullable:
+var answer: Int? by storage.optStored()
+answer = 42
+
+s1.delete("foo")
+~~~
+
+## MotherPacker
+
+This conception allows switching encoding on the fly. Create some MotherPacker instance
+and pass it to your encoding/decoding code:
+
+~~~kotlin
+    @Serializable
+    data class FB1(val foo: Int,val bar: String)
+
+    // This is JSON implementation of MotherPacker:
+    val mp = JsonPacker()
+    // it packs and unpacks to JSON:
+    println(mp.pack(mapOf("foo" to 42)).decodeToString())
+    assertEquals("""{"foo":42}""", mp.pack(mapOf("foo" to 42)).decodeToString())
+    val x = mp.unpack<FB1>("""{"foo":42, "bar": "foo"}""".encodeToByteArray())
+    assertEquals(42, x.foo)
+    assertEquals("foo", x.bar)
+~~~
+
+There is also [MotherBipack] `MotherPacker` implementation using Bipack. You can add more formats
+easily by implementing `MotherPacker` interface.
+
 # Bipack
 
 ## Why?
@@ -36,56 +101,55 @@ Bipack is a compact and efficient binary serialization library (and format) was
 
 ### Allow easy unpacking existing binary structures
 
-Yuo describe your structure as `@Serializable` classes, and - voila, bipack decodes and encodes it for you! We aim to make it really easy to convert data from other binary formats by adding more format annotations
+Yuo describe your structure as `@Serializable` classes, and - voilà, bipack decodes and encodes it for you! We aim to make it really easy to convert data from other binary formats by adding more format annotations
 
 ### Be as compact as possible
 
-For this reason it is a binary notation, it uses binary form for decimal numbers and can use variery of encoding for
+For this reason it is a binary notation, it uses binary form for decimal numbers and can use a variety of encoding for
 integers:
 
 #### Varint
 
-Variable-length compact encoding is used internally in some cases. It uses a 0x80 bit in every byte to mark coninuation.
+Variable-length compact encoding is used internally in some cases. It uses a 0x80 bit in every byte to mark continuation.
 See `object Varint`.
 
 #### Smartint
 
-Variable-length compact encoding for signed and unsigned integers use as few bytes as possible to encode integers. It is
-used automatically when serializing integers. It is slightly more sophisticated than straight `Varint`.
+Variable-length compact encoding for signed and unsigned integers uses as few bytes as possible to encode integers. It is used automatically when serializing integers. It is slightly more sophisticated than straight `Varint`.
 
 ### Do not reveal information about stored data
 
-Many extendable formats, like JSON, BSON, BOSS and may others are keeping data in key-value pairs. While it is good in
-many aspets, it has a clear disadvantages: it uses more space, and it reveals inner data structure to the world. It is
-possible to unpack such formats with zero information about inner structure.
+Many extendable formats, like JSON, BSON, BOSS and may others are keeping data in key-value pairs. While it is good in many aspects, it has some disadvantages: it uses more space, and it reveals inner data structure to the world. It is possible to unpack such formats with zero information about inner structure.
 
-Bipack does not store field names, so it is not possible to unpack or interpret it without knowledge of the data
-structure. Only probablistic analysis. Let's not make life of attacker easier :)
+Bipack does not store field names, so it is not possible to unpack or interpret it without knowledge of the data structure. Only probabilistic analysis. Let's not make the life of attacker easier :)
 
-### - allow upgrading data structures with backward compatibility
+### -- allows upgrading data structures with backward compatibility
 
-The dark side of serialization formats of this kind is that you can't change the structures without either loosing
-backward compatibility with already serialzied data or using volumous boilerplate code to implement some sort of
-versioning.
+The serialization formats of this kind have a dark side: you can't change the structures without either losing backward compatibility with already serialized data or using voluminous boilerplate code to implement some sort of versioning.
 
-Not to waste space and reveal more information that needed Bipack allows extending classes marked as [@Extendable] to be
-extended with more data _appended to the end of list of fields with required defaul values_. For such classes, Bipack stores the number of actually serialized fields and atuomatically uses default values for non-serialized ones when unpacking
-old data.
+Not to waste space
+and reveal more information that needed Bipack allows
+extending classes marked
+as [@Extendable] to be extended with more data _appended to the end of the field list with required default values_.
+For such classes,
+Bipack stores the number of actually serialized fields
+and automatically uses default values for non-serialized ones when unpacking old data.
 
 ### Protect data with framing and CRC
 
-When needed, serialization lobrary allow to store/check CRC32 tag of the structure name with `@Framed` (can be overriden
-as usual with `@SerialName`), or be followed with CRC32 of the serialized binary data, that will be checked on
-deserialization, using `@CrcProtected`. This allows checking the data consistency out of the box and only where needed.
+When needed,
+a serialization library allow to store/check CRC32 tag of the structure name with
+`@Framed` (can be overridden as usual with `@SerialName`), or be followed with CRC32 of the serialized binary data, that will be checked on deserialization, using `@CrcProtected`. This allows checking the data consistency out of the box and only where needed.
 
 # Usage
 
-Use kotlinx serializatino as usual. There are the following Bipack-specific annotations at your disposal (can be combined):
+Use `kotlinx.serialization` as usual. There are the following Bipack-specific annotations at your disposal (can be
+combined):
 
 ## @Extendable
 
-Classes marked this way store number of fields. It allows to add to the class data more fields, to the end of list, with
-default initializers, keeping backward compatibility. For example if you have serialized:
+Classes marked this way store number of fields. It allows adding to the class data more fields, to the end of the list, with
+default initializers, keeping backward compatibility. For example, if you have serialized:
 
 ```kotlin
 @Serializable
@@ -101,28 +165,27 @@ and then decided to add a field:
 data class foo(val i: Int, val bar: String = "buzz")
 ```
 
-It adds 1 or more bytes to the serialized data (field counts in `Varint` format)
+It adds one or more bytes to the serialized data (field counts in `Varint` format)
 
-Bipack will properly deserialize the data serialzied for an old version.
+Bipack will properly deserialize the data serialized for an old version.
 
 ## @CrcProtected
 
 Bipack will calculate and store CRC32 of serialized data at the end, and automatically check it on deserializing
 throwing `InvalidFrameCRCException` if it does not match.
 
-It adds 4 bytes to the serialized data.
+It adds four bytes to the serialized data.
 
 ## @Framed
 
 Put the CRC32 of the serializing class name (`@SerialName` allows to change it as usual) and checks it on deserializing.
 Throws `InvalidFrameHeaderException` if it does not match.
 
-It adds 4 bytes to the serialized data.
+It adds four bytes to the serialized data.
 
-## @Unisgned
+## @Unsigned
 
-This __field annontation__ allows to store __integer fields__ of any size more compact by not saving the sign. Could be
-applyed to both signed and unsigned integers of any size.
+This __field annotation__ allows to store __integer fields__ of any size more compact by not saving the sign. It could be applied to both signed and unsigned integers of any size.
 
 ## @FixedSize(size)
 
@@ -131,7 +194,7 @@ at least one byte.
 
 ## @Fixed
 
-Can be used with any integer type to store/restor it as is, fixed-size, big-endian:
+Can be used with any integer type to store/restore it as is, fixed-size, big-endian:
 
 - Short, UShort: 2 bytes
 - Int, UInt: 4 bytes

build.gradle.kts

@@ -8,7 +8,7 @@ plugins {
 val serialization_version = "1.3.4"
 
 group = "net.sergeych"
-version = "0.1.0"
+version = "0.1.1-SNAPSHOT"
 
 repositories {
     mavenCentral()
@@ -62,6 +62,7 @@ kotlin {
         all {
             languageSettings.optIn("kotlinx.serialization.ExperimentalSerializationApi")
             languageSettings.optIn("kotlin.ExperimentalUnsignedTypes")
+            languageSettings.optIn("kotlin.contracts.ExperimentalContracts")
         }
         val commonMain by getting {
             dependencies {
@@ -80,7 +81,11 @@ kotlin {
         }
         val jvmMain by getting
         val jvmTest by getting
-        val jsMain by getting
+        val jsMain by getting {
+            dependencies {
+                implementation("net.sergeych:mp_stools:1.4.3")
+            }
+        }
         val jsTest by getting
         val nativeMain by getting
         val nativeTest by getting

src/commonMain/kotlin/net.sergeych.bintools/CRC.kt

@@ -27,9 +27,6 @@ interface CRC<T> {
         fun crc8(data: ByteArray, polynomial: UByte = 0xA7.toUByte()): UByte =
             CRC8(polynomial).also { it.update(data) }.value
 
-        fun crc8(data: UByteArray, polynomial: UByte = 0xA7.toUByte()): UByte =
-            CRC8(polynomial).also { it.update(data) }.value
-
         /**
          * Calculate CRC16 for a data array using a given polynomial (CRC16-CCITT polynomial (0x1021) by default)
          */

src/commonMain/kotlin/net.sergeych.bintools/DataKVStorage.kt

@@ -0,0 +1,133 @@
+package net.sergeych.bintools
+
+import net.sergeych.bipack.BipackDecoder
+import net.sergeych.bipack.BipackEncoder
+import net.sergeych.synctools.WaitHandle
+import net.sergeych.tools.ProtectedOp
+import net.sergeych.tools.withLock
+
+class DataKVStorage(private val provider: DataProvider) : KVStorage {
+
+    data class Lock(val name: String) {
+        private val exclusive = ProtectedOp()
+
+        private var readerCount = 0
+
+        private var pulses = WaitHandle()
+
+        fun <T> lockExclusive(f: () -> T): T {
+            while (true) {
+                exclusive.withLock {
+                    if (readerCount == 0) {
+                        return f()
+                    } else {
+                        println("can't lock $this: count is $readerCount")
+                    }
+                }
+                pulses.await()
+            }
+        }
+
+        fun <T> lockRead(f: () -> T): T {
+            try {
+                exclusive.withLock { readerCount++ }
+                return f()
+            } finally {
+                exclusive.withLock { readerCount-- }
+                if (readerCount == 0) pulses.wakeUp()
+            }
+        }
+    }
+
+
+    private val locks = mutableMapOf<String, Lock>()
+    private val access = ProtectedOp()
+    private val keyIds = mutableMapOf<String, Int>()
+    private var lastId: Int = 0
+
+    init {
+        access.withLock {
+            // TODO: read keys
+            for (fn in provider.list()) {
+                println("Scanning: $fn")
+                if (fn.endsWith(".d")) {
+                    val id = fn.dropLast(2).toInt(16)
+                    println("found data record: $fn -> $id")
+
+                    val name = provider.read(fn) { BipackDecoder.decode<String>(it) }
+                    println("Key=$name")
+                    keyIds[name] = id
+                    if (id > lastId) lastId = id
+                } else println("ignoring record $fn")
+            }
+        }
+        println("initialized, ${keyIds.size} records found, lastId=$lastId")
+    }
+
+
+    /**
+     * Important: __it must be called with locked access op!__
+     */
+    private fun lockFor(name: String): Lock = locks.getOrPut(name) { Lock(name) }
+
+    private fun recordName(id: Int) = "${id.toString(16)}.d"
+    private fun <T> read(name: String, f: (DataSource) -> T): T {
+        val lock: Lock
+        val id: Int
+        // global lock: fast
+        access.withLock {
+            id = keyIds[name] ?: throw DataProvider.NotFoundException()
+            lock = lockFor(name)
+        }
+        // per-name lock: slow
+        return lock.lockRead {
+            provider.read(recordName(id), f)
+        }
+    }
+
+    private fun write(name: String, f: (DataSink) -> Unit) {
+        val lock: Lock
+        val id: Int
+        // global lock: fast
+        access.withLock {
+            id = keyIds[name] ?: (++lastId).also { keyIds[name] = it }
+            lock = lockFor(name)
+        }
+        // per-name lock: slow
+        lock.lockExclusive { provider.write(recordName(id), f) }
+    }
+
+    fun deleteEntry(name: String) {
+        // fast pre-check:
+        if (name !in keyIds) return
+        // global lock: we can't now detect concurrent delete + write ops, so exclusive:
+        access.withLock {
+            val id = keyIds[name] ?: return
+            provider.delete(recordName(id))
+            locks.remove(name)
+            keyIds.remove(name)
+        }
+    }
+
+    override fun get(key: String): ByteArray? = try {
+        read(key) {
+            BipackDecoder.decode<String>(it)
+            // notice: not nullable byte array here!
+            BipackDecoder.decode<ByteArray>(it)
+        }
+    } catch (_: DataProvider.NotFoundException) {
+        null
+    }
+
+    override fun set(key: String, value: ByteArray?) {
+        if (value == null) {
+            deleteEntry(key)
+        } else write(key) {
+            BipackEncoder.encode(key, it)
+            BipackEncoder.encode(value, it)
+        }
+    }
+
+    override val keys: Set<String>
+        get() = access.withLock { keyIds.keys }
+}

src/commonMain/kotlin/net.sergeych.bintools/DataProvider.kt

@@ -0,0 +1,33 @@
+package net.sergeych.bintools
+
+/**
+ * Abstraction of some file- or named_record- storage. It could be a filesystem
+ * on native and JVM targets and indexed DB or session storage in the browser.
+ */
+interface DataProvider {
+    open class Error(msg: String,cause: Throwable?=null): Exception(msg,cause)
+    class NotFoundException(msg: String="record not found",cause: Throwable? = null): Error(msg, cause)
+    class WriteFailedException(msg: String="can't write", cause: Throwable?=null): Error(msg,cause)
+
+    /**
+     * Read named record/file
+     * @throws NotFoundException
+     */
+    fun <T>read(name: String,f: (DataSource)->T): T
+
+    /**
+     * Write to a named record / file.
+     * @throws WriteFailedException
+     */
+    fun write(name: String,f: (DataSink)->Unit)
+
+    /**
+     * Delete if exists, or do nothing.
+     */
+    fun delete(name: String)
+
+    /**
+     * List all record names in this source
+     */
+    fun list(): List<String>
+}

src/commonMain/kotlin/net.sergeych.bintools/KVStorage.kt

@@ -0,0 +1,216 @@
+package net.sergeych.bintools
+
+import kotlinx.serialization.serializer
+import net.sergeych.bipack.BipackDecoder
+import net.sergeych.bipack.BipackEncoder
+import kotlin.reflect.KProperty
+import kotlin.reflect.KType
+import kotlin.reflect.typeOf
+
+
+/**
+ * Generic storage of binary content. PArsec uses boss encoding to store everything in it
+ * in a convenient way. See [KVStorage.stored], [KVStorage.invoke] and
+ * [KVStorage.optStored] delegates. The [MemoryKVStorage] is an implementation that stores
+ * values in memory, allowing to connect some other (e.g. persistent storage) later in a
+ * completely transparent way. It can also be used to cache values on the fly.
+ *
+ * Also, it is possible to use [read] and [write] where delegated properties
+ * do not fit well.
+ */
+@Suppress("unused")
+interface KVStorage {
+    operator fun get(key: String): ByteArray?
+    operator fun set(key: String, value: ByteArray?)
+
+    /**
+     * Check whether key is in storage.
+     * Default implementation uses [keys]. You may override it for performance
+     */
+    operator fun contains(key: String): Boolean = key in keys
+
+    val keys: Set<String>
+
+
+    /**
+     * Get number of object in the storage
+     * Default implementation uses [keys]. You may override it for performance
+     */
+    val size: Int get() = keys.size
+
+    /**
+     * Clears all objects in the storage
+     * Default implementation uses [keys]. You may override it for performance
+     */
+    fun clear() {
+        for (k in keys) this[k] = null
+    }
+
+    /**
+     * Default implementation uses [keys]. You may override it for performance
+     */
+    fun isEmpty() = size == 0
+
+    /**
+     * Default implementation uses [keys]. You may override it for performance
+     */
+    fun isNotEmpty() = size != 0
+
+    /**
+     * Add all elements from another storage, overwriting any existing
+     * keys.
+     */
+    fun addAll(other: KVStorage) {
+        for (k in other.keys) {
+            this[k] = other[k]
+        }
+    }
+
+    /**
+     * Delete element by key
+     */
+    fun delete(key: String) {
+        set(key, null)
+    }
+}
+
+/**
+ * Write
+ */
+inline fun <reified T: Any>KVStorage.write(key: String,value: T) {
+    this[key] = BipackEncoder.encode(value)
+}
+
+inline fun <reified T:Any>KVStorage.read(key: String): T? =
+    this[key]?.let { BipackDecoder.decode(it) }
+
+
+inline operator fun <reified T> KVStorage.invoke(defaultValue: T,overrideName: String? = null) =
+    KVStorageDelegate<T>(this, typeOf<T>(), defaultValue, overrideName)
+
+inline fun <reified T> KVStorage.stored(defaultValue: T, overrideName: String? = null) =
+    KVStorageDelegate<T>(this, typeOf<T>(), defaultValue, overrideName)
+inline fun <reified T> KVStorage.optStored(overrideName: String? = null) =
+    KVStorageDelegate<T?>(this, typeOf<T?>(), null, overrideName)
+
+class KVStorageDelegate<T>(
+    private val storage: KVStorage,
+    type: KType,
+    private val defaultValue: T,
+    private val overrideName: String? = null,
+) {
+
+    private fun name(property: KProperty<*>): String = overrideName ?: property.name
+
+    private var cachedValue: T = defaultValue
+    private var cacheReady = false
+    private val serializer = serializer(type)
+
+    @Suppress("UNCHECKED_CAST")
+    operator fun getValue(thisRef: Any?, property: KProperty<*>): T {
+        if (cacheReady) return cachedValue
+        val data = storage.get(name(property))
+        println("Got data: ${data?.toDump()}")
+        if (data == null)
+            cachedValue = defaultValue
+        else
+            cachedValue = BipackDecoder.decode(data.toDataSource(), serializer) as T
+        cacheReady = true
+        return cachedValue
+    }
+
+    operator fun setValue(thisRef: Any?, property: KProperty<*>, value: T) {
+//        if (!cacheReady || value != cachedValue) {
+            cachedValue = value
+            cacheReady = true
+            println("set ${name(property)} to ${BipackEncoder.encode(serializer, value).toDump()}")
+            storage[name(property)] = BipackEncoder.encode(serializer, value)
+//        }
+    }
+}
+
+/**
+ * Memory storage. Allows connecting to another storage (e.g. persistent one) at come
+ * point later in a transparent way.
+ */
+class MemoryKVStorage(copyFrom: KVStorage? = null) : KVStorage {
+
+    // is used when connected:
+    private var underlying: KVStorage? = null
+
+    // is used while underlying is null:
+    private val data = mutableMapOf<String, ByteArray>()
+
+    /**
+     * Connect some other storage. All existing data will be copied to the [other]
+     * storage. After this call all data access will be routed to [other] storage.
+     */
+    @Suppress("unused")
+    fun connectToStorage(other: KVStorage) {
+        other.addAll(this)
+        underlying = other
+        data.clear()
+    }
+
+
+    /**
+     * Get  data from either memory or a connected storage, see [connectToStorage]
+     */
+    override fun get(key: String): ByteArray? {
+        underlying?.let {
+            return it[key]
+        }
+        return data[key]
+    }
+
+    /**
+     * Put data to memory storage or connected storage if [connectToStorage] was called
+     */
+    override fun set(key: String, value: ByteArray?) {
+        underlying?.let { it[key] = value } ?: run {
+            if (value != null) data[key] = value
+            else data.remove(key)
+        }
+    }
+
+    /**
+     * Checks the item exists in the memory storage or connected one, see[connectToStorage]
+     */
+    override fun contains(key: String): Boolean {
+        underlying?.let { return key in it }
+        return key in data
+    }
+
+    override val keys: Set<String>
+        get() = underlying?.keys ?: data.keys
+
+    override fun clear() {
+        underlying?.clear() ?: data.clear()
+    }
+
+    init {
+        copyFrom?.let { addAll(it) }
+    }
+}
+
+/**
+ * Create per-platform default named storage.
+ *
+ * - In the browser, it uses the `Window.localStorage` prefixing items
+ *   by a string containing the [name]
+ *
+ * - In the JVM environment it uses folder-based storage on the file system. The name
+ *   is considered to be a folder name (the whole path which will be automatically created)
+ *   using the following rules:
+ *    - when the name starts with slash (`/`) it is treated as an absolute path to a folder
+ *    - when the name contains slash, it is considered to be a relative folder to the
+ *      `User.home` directory, like "`~/`" on unix systems.
+ *    - otherwise, the folder will be created in "`~/.local_storage`" parent directory
+ *      (which also will be created if needed).
+ *
+ *  - For the native platorms it is not yet implemented (but will be soon).
+ *
+ *  See [DataKVStorage] and [DataProvider] to implement a KVStorage on filesystems and like,
+ *  and `FileDataProvider` class on JVM target.
+ */
+expect fun defaultNamedStorage(name: String): KVStorage

src/commonMain/kotlin/net/sergeych/synctools/AtomicCounter.kt

@@ -0,0 +1,15 @@
+package net.sergeych.tools
+
+/**
+ * Thread-safe multiplatform counter
+ */
+@Suppress("unused")
+class AtomicCounter(initialValue: Long = 0) : AtomicValue<Long>(initialValue) {
+
+    fun incrementAndGet(): Long = op { ++actualValue }
+    fun getAndIncrement(): Long = op { actualValue++ }
+
+    fun decrementAndGet(): Long = op { --actualValue }
+
+    fun getAndDecrement(): Long = op { actualValue-- }
+}

src/commonMain/kotlin/net/sergeych/synctools/AtomicValue.kt

@@ -0,0 +1,37 @@
+package net.sergeych.tools
+
+/**
+ * Multiplatform (JS and battery included) atomically mutable value.
+ * Actual value can be either changed in a block of [mutate] when
+ * new value _depends on the current value_ or use a same [value]
+ * property that is thread-safe where there are threads and just safe
+ * otherwise ;)
+ */
+open class AtomicValue<T>(initialValue: T) {
+    var actualValue = initialValue
+        protected set
+
+    protected val op = ProtectedOp()
+
+    /**
+     * Change the value: get the current and set to the returned, all in the
+     * atomic operation. All other mutating requests including assigning to [value]
+     * will be blocked and queued.
+     * @return result of the mutation. Note that immediate call to property [value]
+     *      could already return modified bu some other thread value!
+     */
+    fun mutate(mutator: (T) -> T): T = op {
+        actualValue = mutator(actualValue)
+        actualValue
+    }
+
+    /**
+     * Atomic get or set the value. Atomic get means if there is a [mutate] in progress
+     * it will wait until the mutation finishes and then return the correct result.
+     */
+    var value: T
+        get() = op { actualValue }
+        set(value) {
+            mutate { value }
+        }
+}

src/commonMain/kotlin/net/sergeych/synctools/ProtectedOp.kt

@@ -0,0 +1,60 @@
+package net.sergeych.tools
+
+import kotlin.contracts.ExperimentalContracts
+import kotlin.contracts.InvocationKind
+import kotlin.contracts.contract
+
+/**
+ * Multiplatform interface to perform a regular (not suspend) operation
+ * protected by a platform mutex (where necessary). Get real implementation
+ * with [ProtectedOp] and use it with [ProtectedOpImplementation.withLock] and
+ * [ProtectedOpImplementation.invoke]
+ */
+interface ProtectedOpImplementation {
+    /**
+     * Get a lock. Be sure to release it.
+     * The recommended way is using [ProtectedOpImplementation.withLock] and
+     * [ProtectedOpImplementation.invoke]
+     */
+    fun lock()
+
+    /**
+     * Release a lock.
+     * The recommended way is using [ProtectedOpImplementation.withLock] and
+     * [ProtectedOpImplementation.invoke]
+     */
+    fun unlock()
+}
+
+@ExperimentalContracts
+inline fun <T> ProtectedOpImplementation.withLock(f: () -> T): T {
+    contract {
+        callsInPlace(f, InvocationKind.EXACTLY_ONCE)
+    }
+    lock()
+    return try {
+        f()
+    } finally {
+        unlock()
+    }
+}
+
+/**
+ * Run a block mutualy-exclusively, see [ProtectedOp]
+ */
+operator fun <T> ProtectedOpImplementation.invoke(f: () -> T): T = withLock(f)
+
+/**
+ * Get the platform-depended implementation of a mutex. It does nothing in the
+ * browser and use appropriate mechanics on JVM and native targets. See
+ * [ProtectedOpImplementation.invoke], [ProtectedOpImplementation.withLock]
+ * ```kotlin
+ * val op = ProtectedOp()
+ * //...
+ * op {
+ *      // mutually exclusive execution
+ *      println("sequential execution here")
+ * }
+ * ~~~
+ */
+expect fun ProtectedOp(): ProtectedOpImplementation

src/commonMain/kotlin/net/sergeych/synctools/WaitHandle.kt

@@ -0,0 +1,23 @@
+package net.sergeych.synctools
+
+/**
+ * Platform-independent interface to thread wait/notify. Does nothing in JS/browser,
+ * and uses appropriate mechanics on other platforms.
+ */
+@Suppress("EXPECT_ACTUAL_CLASSIFIERS_ARE_IN_BETA_WARNING")
+expect class WaitHandle() {
+
+    /**
+     * Wait for [wakeUp] as long as [milliseconds] milliseconds, or forever.
+     * Notice it returns immediately on the single-theaded platforms like JS.
+     * @param milliseconds to wait, use 0 to wait indefinitely
+     * @return true if [wakeUp] was called before the timeout, false on timeout.
+     */
+    fun await(milliseconds: Long=0): Boolean
+
+    /**
+     * Awake all [await]'ing threads. Does nothing in the single-threaded JS
+     * environment
+     */
+    fun wakeUp()
+}

src/commonTest/kotlin/bipack/BipackEncoderTest.kt

@@ -50,9 +50,10 @@ data class FoobarFP1(val bar: Int, val foo: Int, val other: Int = -1)
 sealed class SC1 {
 
     @Serializable
-    class Nested: SC1()
+    class Nested : SC1()
 }
 
+@Suppress("unused")
 class BipackEncoderTest {
 
     @Serializable
@@ -356,7 +357,7 @@ class BipackEncoderTest {
     @Test
     fun testInstant() {
         val x = Clock.System.now()
-        println( BipackEncoder.encode(x).toDump() )
+        println(BipackEncoder.encode(x).toDump())
         val y = BipackDecoder.decode<Instant>(BipackEncoder.encode(x))
         assertEquals(x.toEpochMilliseconds(), y.toEpochMilliseconds())
     }
@@ -369,7 +370,7 @@ class BipackEncoderTest {
         @Fixed
         val i: UInt,
         @Fixed
-        val li: ULong
+        val li: ULong,
     )
 
     @Serializable
@@ -380,20 +381,20 @@ class BipackEncoderTest {
         @Unsigned
         val i: UInt,
         @Unsigned
-        val li: ULong
+        val li: ULong,
     )
 
     @Test
     fun vectors() {
-        val x = UInts(7u, 64000.toUShort(), 66000u, 931127140399u);
-        val p = BipackEncoder.encode(x);
+        val x = UInts(7u, 64000.toUShort(), 66000u, 931127140399u)
+        val p = BipackEncoder.encode(x)
         println(p.toDump())
         println(p.encodeToBase64Compact())
         val y = BipackDecoder.decode<UInts>(p)
         assertEquals(x, y)
 
-        val xv = VarUInts(7u, 64000.toUShort(), 66000u, 931127140399u);
-        val pv = BipackEncoder.encode(xv);
+        val xv = VarUInts(7u, 64000.toUShort(), 66000u, 931127140399u)
+        val pv = BipackEncoder.encode(xv)
         println(pv.toDump())
         println(pv.encodeToBase64Compact())
         val yv = BipackDecoder.decode<VarUInts>(pv)
@@ -405,7 +406,8 @@ class BipackEncoderTest {
     @Test
     fun testStrangeUnpack() {
         @Serializable
-        data class SFoo(val code: Int,val s1: String?=null,val s2: String?=null)
+        data class SFoo(val code: Int, val s1: String? = null, val s2: String? = null)
+
         val z = BipackEncoder.encode(117)
         println(z.toDump())
         val sf = BipackDecoder.decode<SFoo>(z)

src/commonTest/kotlin/synctools/AtomicCounterTest.kt

@@ -0,0 +1,21 @@
+package net.sergeych.synctools
+
+import net.sergeych.tools.AtomicCounter
+import kotlin.test.Test
+import kotlin.test.assertEquals
+
+class AtomicCounterTest {
+    @Test
+    fun incrementAndDecrement() {
+        val ac = AtomicCounter(7)
+        assertEquals(7, ac.getAndIncrement())
+        assertEquals(8, ac.value)
+        assertEquals(9, ac.incrementAndGet())
+        assertEquals(9, ac.value)
+
+        assertEquals(9, ac.getAndDecrement())
+        assertEquals(8, ac.value)
+        assertEquals(7, ac.decrementAndGet())
+        assertEquals(7, ac.value)
+    }
+}

src/jsMain/kotlin/net.sergeych.bintools/KVStorage.js.kt

@@ -0,0 +1,42 @@
+package net.sergeych.bintools
+
+import kotlinx.browser.localStorage
+import net.sergeych.mp_tools.decodeBase64Compact
+import net.sergeych.mp_tools.encodeToBase64Compact
+import org.w3c.dom.Storage
+import org.w3c.dom.set
+
+actual fun defaultNamedStorage(name: String): KVStorage =  BrowserKVStorage(name, localStorage)
+
+/**
+ * Default KV storage in browser. Use if with `localStorage` or `sessionStorage`. It uses
+ * prefix for storage values not to collide with other data, beware of it.
+ */
+class BrowserKVStorage(keyPrefix: String, private val bst: Storage) : KVStorage {
+
+    private val prefix = "$keyPrefix:"
+    fun k(key: String) = "$prefix$key"
+    override fun get(key: String): ByteArray? {
+        return bst.getItem(k(key))?.decodeBase64Compact()
+    }
+
+    override fun set(key: String, value: ByteArray?) {
+        val corrected = k(key)
+        if (value == null)
+            bst.removeItem(corrected)
+        else
+            bst.set(corrected, value.encodeToBase64Compact())
+    }
+
+    override val keys: Set<String>
+        get() {
+            val kk = mutableListOf<String>()
+            for (i in 0 until bst.length) {
+                val k = bst.key(i) ?: break
+                if( k.startsWith(prefix)) {
+                    kk += k.substring(prefix.length)
+                }
+            }
+            return kk.toSet()
+        }
+}

src/jsMain/kotlin/net/sergeych/synctools/ProtectedOp.js.kt

@@ -0,0 +1,9 @@
+package net.sergeych.tools
+
+/**
+ * JS is single-threaded, so we don't need any additional protection:
+ */
+actual fun ProtectedOp(): ProtectedOpImplementation = object : ProtectedOpImplementation {
+    override fun lock() {}
+    override fun unlock() {}
+}

src/jsMain/kotlin/net/sergeych/synctools/WaitHandle.js.kt

@@ -0,0 +1,13 @@
+package net.sergeych.synctools
+
+@Suppress("EXPECT_ACTUAL_CLASSIFIERS_ARE_IN_BETA_WARNING")
+actual class WaitHandle {
+    actual fun await(milliseconds: Long): Boolean {
+        // in JS we can't wait: no threads
+        return true
+    }
+
+    actual fun wakeUp() {
+        // in JS we can't wait: no threads
+    }
+}

src/jvmMain/kotlin/net.sergeych.bintools/FileDataProvider.kt

@@ -0,0 +1,47 @@
+package net.sergeych.bintools
+
+import java.io.InputStream
+import java.io.OutputStream
+import java.nio.file.Path
+import kotlin.io.path.*
+
+class FileDataProvider(val folder: Path): DataProvider {
+
+    class Source(private val input: InputStream): DataSource {
+        override fun readByte(): Byte {
+            val b = input.read()
+            if( b < 0) throw DataSource.EndOfData()
+            return b.toByte()
+        }
+
+        override fun readBytes(size: Int): ByteArray {
+            return input.readNBytes(size).also {
+                if( it.size < size ) throw DataSource.EndOfData()
+            }
+        }
+    }
+
+    class Sink(private val out: OutputStream): DataSink {
+        override fun writeByte(data: Byte) {
+            out.write(data.toInt())
+        }
+    }
+
+    override fun <T> read(name: String, f: (DataSource) -> T): T = folder.resolve(name).inputStream().buffered().use {
+        f(Source(it))
+    }
+
+    override fun write(name: String, f: (DataSink) -> Unit) {
+        folder.resolve(name).outputStream().use { f(Sink(it)) }
+    }
+
+    override fun delete(name: String) {
+        println("file: $folder -- $name")
+        folder.resolve(name).deleteExisting()
+    }
+
+    override fun list(): List<String> = folder
+        .listDirectoryEntries()
+        .filter { it.isRegularFile() && it.isReadable() }
+        .map { it.name }
+}

src/jvmMain/kotlin/net.sergeych.bintools/KVStorage.jvm.kt

@@ -0,0 +1,21 @@
+package net.sergeych.bintools
+
+import java.nio.file.Paths
+import kotlin.io.path.createDirectories
+
+actual fun defaultNamedStorage(name: String): KVStorage {
+    val rootFolder = Paths.get(when {
+        // absolute path
+        name.startsWith("/") -> name
+        // path - assume the caller knows what to do
+        name.contains("/") -> name
+        // simple name - we will create it in the user home:
+        else -> {
+            val home = System.getProperty("user.home")
+            "$home/.local_storage/$name"
+        }
+    })
+    rootFolder.createDirectories()
+    val provider = FileDataProvider(rootFolder)
+    return DataKVStorage(provider)
+}

src/jvmMain/kotlin/net/sergeych/synctools/ProtectedOp.jvm.kt

@@ -0,0 +1,19 @@
+package net.sergeych.tools
+
+import java.util.concurrent.locks.ReentrantLock
+
+/**
+ * Get the platform-depended implementation of a mutex-protected operation.
+ * JVM version uses a concealed object synchronization pattern (per-object monitor lock)
+ */
+actual fun ProtectedOp(): ProtectedOpImplementation = object : ProtectedOpImplementation {
+    private val access = ReentrantLock()
+    override fun lock() {
+        access.lock()
+    }
+
+    override fun unlock() {
+        access.unlock()
+    }
+
+}

src/jvmMain/kotlin/net/sergeych/synctools/WaitHandle.jvm.kt

@@ -0,0 +1,22 @@
+package net.sergeych.synctools
+
+@Suppress("EXPECT_ACTUAL_CLASSIFIERS_ARE_IN_BETA_WARNING")
+actual class WaitHandle {
+    private val access = Object()
+
+    actual fun await(milliseconds: Long): Boolean {
+        return synchronized(access) {
+            try {
+                access.wait(milliseconds)
+                true
+            }
+            catch(_: InterruptedException) {
+                false
+            }
+        }
+    }
+
+    actual fun wakeUp() {
+        synchronized(access) { access.notifyAll() }
+    }
+}

src/jvmTest/kotlin/net/sergeych/bintools/KVStorage_jvmKtTest.kt

@@ -0,0 +1,39 @@
+package net.sergeych.bintools
+
+import kotlin.test.Test
+import kotlin.test.assertEquals
+import kotlin.test.assertNull
+import kotlin.test.assertTrue
+
+class KVStorage_jvmKtTest {
+    @Test
+    fun testFileStorage() {
+        val s1 = defaultNamedStorage("test_mp_bintools")
+
+        for( n in s1.keys.toList()) s1.delete(n)
+
+        assertTrue(s1.keys.isEmpty())
+        var foo by s1("unknown")
+        assertEquals(foo, "unknown")
+        foo = "bar"
+        assertEquals(foo, "bar")
+        var answer by s1.optStored<Int>()
+        assertNull(answer)
+        answer = 42
+        assertEquals(answer, 42)
+        answer = 43
+
+        println("----------------------------------------------------------------")
+        val s2 = defaultNamedStorage("test_mp_bintools")
+        val foo1 by s2.stored("?", "foo")
+        val answer1: Int? by s2.optStored("answer")
+
+        assertEquals("bar", foo1)
+        assertEquals(43, answer1)
+
+        for( i in 0..< 13 ) {
+            s2.write("test_$i", "payload_$i")
+        }
+
+    }
+}

src/nativeMain/kotlin/net.sergeych.bintools/KVStorage.native.kt

@@ -0,0 +1,5 @@
+package net.sergeych.bintools
+
+actual fun defaultNamedStorage(name: String): KVStorage {
+    TODO("Not yet implemented")
+}

src/nativeMain/kotlin/net/sergeych/synctools/ProtectedOp.native.kt

@@ -0,0 +1,17 @@
+package net.sergeych.tools
+
+import kotlinx.atomicfu.locks.ReentrantLock
+
+/**
+ * Native implementation uses `ReentrantLock`]
+ */
+actual fun ProtectedOp(): ProtectedOpImplementation = object : ProtectedOpImplementation {
+    private val access = ReentrantLock()
+    override fun lock() {
+        access.lock()
+    }
+
+    override fun unlock() {
+        access.unlock()
+    }
+}

src/nativeMain/kotlin/net/sergeych/synctools/WaitHandle.native.kt

@@ -0,0 +1,34 @@
+package net.sergeych.synctools
+
+import kotlinx.coroutines.TimeoutCancellationException
+import kotlinx.coroutines.channels.Channel
+import kotlinx.coroutines.runBlocking
+import kotlinx.coroutines.withTimeout
+
+@Suppress("EXPECT_ACTUAL_CLASSIFIERS_ARE_IN_BETA_WARNING")
+actual class WaitHandle {
+    private val channel = Channel<Unit>()
+    actual fun await(milliseconds: Long): Boolean {
+        return runBlocking {
+            try {
+                if( milliseconds > 0) {
+                    withTimeout(milliseconds) {
+                        channel.receive()
+                        true
+                    }
+                }
+                else {
+                    channel.receive()
+                    true
+                }
+            }
+            catch(_: TimeoutCancellationException) {
+                false
+            }
+        }
+    }
+
+    actual fun wakeUp() {
+        runBlocking { channel.send(Unit) }
+    }
+}