Binary tools and BiPack serializer

Multiplatform binary tools collection, including portable serialization of the compact and fast [Bipack] format, and many useful tools to work with binary data, like CRC family checksums, dumps, etc. It works well also in the browser and in native targets.

Recent changes

• 0.1.7 built with kotlin 2.0.20 which contains important fix in wasmJS

• 0.1.6 add many useful features, added support to wasmJS and all other platforms. Note to wasmJS: it appears to be a bug in wasm compiler so BipackDecoder could cause wasm loading problem.

• 0.1.1: added serialized KVStorage with handy implementation on JVM and JS platforms and some required synchronization tools.

• 0.1.0: uses modern kotlin 1.9.*, fixes problem with singleton or empty/object serialization

The last 1.8-based version is 0.0.8. Some fixes are not yet backported to it pls leave an issue of needed.

Documentation

Aside of the samples in this readme please see library documentation.

Usage

Add our maven:

repositories {
    // ...
    maven("https://gitea.sergeych.net/api/packages/SergeychWorks/maven")
}

And add dependency to the proper place in your project like this:

dependencies {
    // ...
    implementation("net.sergeych:mp_bintools:0.1.7")
}

Calculating CRCs:

CRC.crc32("Hello".encodeToByteArray())
CRC.crc16("happy".encodeToByteArray())
CRC.crc8("world".encodeToByteArray())

Binary effective serialization with Bipack:

@Serializable
data class Foo(val bar: String,buzz: Int)

val foo = Foo("bar", 42)
val bytes = BipackEncoder.encode(foo)
val bar: Foo = BipackDecoder.decode(bytes)
assertEquals(foo, bar)

Bipack-based auto-serializing storage:

Allows easily storing whatever @Serializable data type using delegates and more:

val storage = defaultNamedStorage("test_mp_bintools")

var foo by s1("unknown") // default value makes it a String
foo = "bar"

// nullable:
var answer: Int? by storage.optStored()
answer = 42

s1.delete("foo")

MotherPacker

This conception allows switching encoding on the fly. Create some MotherPacker instance and pass it to your encoding/decoding code:

    @Serializable
    data class FB1(val foo: Int,val bar: String)

    // This is JSON implementation of MotherPacker:
    val mp = JsonPacker()
    // it packs and unpacks to JSON:
    println(mp.pack(mapOf("foo" to 42)).decodeToString())
    assertEquals("""{"foo":42}""", mp.pack(mapOf("foo" to 42)).decodeToString())
    val x = mp.unpack<FB1>("""{"foo":42, "bar": "foo"}""".encodeToByteArray())
    assertEquals(42, x.foo)
    assertEquals("foo", x.bar)

There is also [MotherBipack] MotherPacker implementation using Bipack. You can add more formats easily by implementing MotherPacker interface.

Bipack

Why?

Bipack is a compact and efficient binary serialization library (and format) was designed with the following main goals:

Allow easy unpacking existing binary structures

Yuo describe your structure as @Serializable classes, and - voilà, bipack decodes and encodes it for you! We aim to make it really easy to convert data from other binary formats by adding more format annotations

Be as compact as possible

For this reason it is a binary notation, it uses binary form for decimal numbers and can use a variety of encoding for integers:

Varint

Variable-length compact encoding is used internally in some cases. It uses a 0x80 bit in every byte to mark continuation. See object Varint.

Smartint

Variable-length compact encoding for signed and unsigned integers uses as few bytes as possible to encode integers. It is used automatically when serializing integers. It is slightly more sophisticated than straight Varint.

Do not reveal information about stored data

Many extendable formats, like JSON, BSON, BOSS and may others are keeping data in key-value pairs. While it is good in many aspects, it has some disadvantages: it uses more space, and it reveals inner data structure to the world. It is possible to unpack such formats with zero information about inner structure.

Bipack does not store field names, so it is not possible to unpack or interpret it without knowledge of the data structure. Only probabilistic analysis. Let's not make the life of attacker easier :)

-- allows upgrading data structures with backward compatibility

The serialization formats of this kind have a dark side: you can't change the structures without either losing backward compatibility with already serialized data or using voluminous boilerplate code to implement some sort of versioning.

Not to waste space and reveal more information that needed Bipack allows extending classes marked as [@Extendable] to be extended with more data appended to the end of the field list with required default values. For such classes, Bipack stores the number of actually serialized fields and automatically uses default values for non-serialized ones when unpacking old data.

Protect data with framing and CRC

When needed, a serialization library allow to store/check CRC32 tag of the structure name with @Framed (can be overridden as usual with @SerialName), or be followed with CRC32 of the serialized binary data, that will be checked on deserialization, using @CrcProtected. This allows checking the data consistency out of the box and only where needed.

Usage

Use kotlinx.serialization as usual. There are the following Bipack-specific annotations at your disposal (can be combined):

@Extendable

Classes marked this way store number of fields. It allows adding to the class data more fields, to the end of the list, with default initializers, keeping backward compatibility. For example, if you have serialized:

@Serializable
@Extendable
data class foo(val i: Int)

and then decided to add a field:

@Serializable
@Extendable
data class foo(val i: Int, val bar: String = "buzz")

It adds one or more bytes to the serialized data (field counts in Varint format)

Bipack will properly deserialize the data serialized for an old version.

@CrcProtected

Bipack will calculate and store CRC32 of serialized data at the end, and automatically check it on deserializing throwing InvalidFrameCRCException if it does not match.

It adds four bytes to the serialized data.

@Framed

Put the CRC32 of the serializing class name (@SerialName allows to change it as usual) and checks it on deserializing. Throws InvalidFrameHeaderException if it does not match.

It adds four bytes to the serialized data.

@Unsigned

This field annotation allows to store integer fields of any size more compact by not saving the sign. It could be applied to both signed and unsigned integers of any size.

@FixedSize(size)

Use it with fixed-size collections (like hashes, keys, etc.) to not keep collection size in the packed binary. It saves at least one byte.

@Fixed

Can be used with any integer type to store/restore it as is, fixed-size, big-endian:

• Short, UShort: 2 bytes
• Int, UInt: 4 bytes
• Long, ULong: 8 bytes

Note that without this modifier all integers are serialized into variable-length compressed format, see class [Smartint] from this library.

Example:

@Serializable
class Foo(
    @Fixed
    val eightBytesLongInt: Long
)

// so:
assertEquals("00 00 00 01 00 00 00 02", BipackEncoder.encode(Foo(0x100000002)).encodeToHex())