DiWAN/bipack_ru
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

merge into: DiWAN:master
Branches Tags
DiWAN:master
DiWAN:main
...
pull from: DiWAN:main
Branches Tags
DiWAN:master
DiWAN:main

No commits in common. "master" and "main" have entirely different histories.
master ... main

16 changed files with 19 additions and 2196 deletions
Show Stats Expand all files Collapse all files

3
.gitignore vendored
View File

@ -1,3 +0,0 @@
/target
/Cargo.lock
/.idea

17
Cargo.toml
View File

@ -1,17 +0,0 @@
[package]
name = "bipack_ru"
version = "0.4.4"
edition = "2021"
license = "Apache-2.0"
description = "binary size-effective format used in Divan smart contracts, wasm bindings, network protocols, etc."
homepage = "https://gitea.sergeych.net/DiWAN/bipack_ru"
repository = "https://gitea.sergeych.net/DiWAN/bipack_ru"
readme = "README.md"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
serde = { version = "1.0", features = ["derive"] }
[dev-dependencies]
base64 = "0.21"
hex = "0.4"

17
LICENSE
View File

@ -54,3 +54,20 @@ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
END OF TERMS AND CONDITIONS END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
Copyright 2023 Divan
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

86
README.md
View File

@ -1,85 +1,3 @@
# bipack_ru: Rust Bipack implementation # bipack_ru
Bipack serialization format is a compact binary format that was created as a replacement for a Bipack format implementation, minimalistic by purpose.
BOSS format, more compact, faster, and not revealing inner data structure as much as possible.
It is used in our next generation data protection systems, in DiWAN products as a base format and also in my new communication package like [kiloparsec](https://gitea.sergeych.net/sergeych/kiloparsec)
(and in underlying [crypto2](https://gitea.sergeych.net/sergeych/crypto2)) which are intended
to replace older `Parsec` protocols family to cope with modern crazy world challenges.
>__IMPORTANT__ due to growing tensions in the world, the main development is moved to our [Gitea repository](https://gitea.sergeych.net/DiWAN/bipack_ru). We will try our best to make it available all around the world.
The most recent code will be there, and we encourage to use it also for issues/discussions.
> Beta test
# Use with serde:
Use `bipack_ru:ser:to_bytes()` and `bipack_ru:de:from_bytes()` functions:
```rust
fn test() -> Result<(),Error> {
let src = HashSet::from(["foo", "bar", "buz"].map(|i| i.to_string()));
let packed = to_bytes(&src)?;
println!("{}", to_dump(&packed));
let restored: HashSet<String> = from_bytes(&packed)?;
println!("{:?}", restored);
assert_eq!(src, restored);
}
```
serde module supports all Rust formats except floats/doubles which are not yet used and
standardized by bipack format.
- All integers, signed and unsigned, are encoded with variable-length `smartint`.
Fixed-size encoding is available using custom de/serializers (exactly as in postcard format):
```rust
#[derive(Serialize)]
pub struct DefinitelyBigEndian {
#[serde(with = "bipack_ru::fixint::be")]
// or #[serde(with = "bipack_ru::fixint::le")]
x: u16,
}
```
## Fixed or variable length arrays?
When encoding arrays, the rule is simple:
- dynamic arrays (arrays, vectors, etc.) are encoded with size, as variable-length
- fixed-size arrays, like tuples `(1,2,3,4,5)`, or something like `[u32; 5]` are encoded as fixed-size entities.
# Direct usage (no serde)
The sample code (see `src/lib.rs` for more:)
```rust
fn test() {
let mut data = Vec::<u8>::new();
data.put_str("Hello, rupack!");
println!("size ${}\n{}", data.len(), to_dump(&data));
let mut src = SliceSource::from(&data);
assert_eq!("Hello, rupack!", src.get_str().unwrap());
}
```
## Tools and macros
- `to_dump` to convert binary slice into human-readable dump
- `StringBuilder` super minimalistic string builder (footprint).
The autodoc documentation is good enough already, so we do not repeat it here now.
## How to
- just ad this package to your dependencies, it is on crates.io.
## Big thanks
- to https://github.com/jamesmunns for the brilliant postcard format which was used as a source of inspiration for the Rust bindings.
# License
For compliance with other modules this work is provided under APACHE 2.0 license a copy of which is included in the file `LICENSE`.

79
src/bipack.rs
View File

@ -1,79 +0,0 @@
use crate::bipack_sink::{BipackSink, IntoU64};
use crate::bipack_source::{BipackSource, Result};
/// The trait to unpack to be used in serializer to come. Please don't use it, it is
/// experimental.
pub trait BiPackable {
fn bi_pack(self: &Self, sink: &mut impl BipackSink) -> Result<()>;
}
/// The trait need by [bipack()] macro and in the serializer to come, packs some
/// type into a generic sink.
pub trait BiUnpackable where Self: Sized {
fn bi_unpack(source: &mut dyn BipackSource) -> Result<Self>;
}
/// Pack all arguments according to their type, using variable-length
/// encoding for integers and default encoding for binaries and string,
/// and return `Vec<u8>` with packed result.
///
/// It you need more fine-grained packing, use [BipackSink] directly.
#[macro_export]
macro_rules! bipack {
( $( $e: expr),* ) => {{
let mut result = Vec::new();
$(
$e.bi_pack(&mut result).unwrap();
)*
result
}};
}
impl<T: IntoU64 + Copy> BiPackable for T {
fn bi_pack(self: &Self, sink: &mut impl BipackSink) -> Result<()> {
sink.put_unsigned(self.into_u64())
}
}
impl BiPackable for &str {
fn bi_pack(self: &Self, sink: &mut impl BipackSink) -> Result<()> {
sink.put_str(self)
}
}
macro_rules! declare_unpack_u {
($($type:ident),*) => {
$(impl BiUnpackable for $type {
fn bi_unpack(source: &mut dyn BipackSource) -> Result<$type> {
Ok(source.get_unsigned()? as $type)
}
})*
};
}
declare_unpack_u!(u16, u32, u64);
// impl<String> BiUnpackable<String> for String {
// fn bi_unpack(source: &mut impl BipackSource) -> Result<Self> {
// source.get_str()
// }
// }
// impl dyn BiUnpackable<u32> {
//
// }
impl BiUnpackable for u8 {
fn bi_unpack(source: &mut dyn BipackSource) -> Result<u8> {
source.get_u8()
}
}
impl BiUnpackable for String {
fn bi_unpack(source: &mut dyn BipackSource) -> Result<String> {
source.get_str()
}
}

174
src/bipack_sink.rs
View File

@ -1,174 +0,0 @@
// Copyright 2023 by Sergey S. Chernov.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::iter::Iterator;
use std::usize;
use crate::bipack_source::Result;
const V0LIMIT: u64 = 1u64 << 6;
const V1LIMIT: u64 = 1u64 << 14;
const V2LIMIT: u64 = 1u64 << 22;
/// Numeric value convertible to Unsigned 64 bit to be used
/// with [BipackSink#put_unsigned] compressed format. It is implemented fir usize
/// and u* types already.
pub trait IntoU64 {
fn into_u64(self) -> u64;
}
macro_rules! into_u64 {
($($type:ident),*) => {
$(impl IntoU64 for $type {
fn into_u64(self) -> u64 {
self as u64
}
})*
};
}
into_u64!(u8, u16, u32, usize, u64);
/// Data sink to encode bipack binary format.
///
/// To implement just override [BipackSink::put_u8] and optionally [BipackSink::put_fixed_bytes].
///
/// Note that the sink is not returning errors, unlike [crate::bipack_source::BipackSource].
/// It is supposed that the sink has unlimited
/// size for the context of data it is used for. This is practical. For the case of overflow-aware
/// sink you can create one that ignores extra data when overflow is detected and report it
/// somehow, but for encoding it does not worth effort (data size could be estimated in advance).
pub trait BipackSink {
fn put_u8(self: &mut Self, data: u8) -> Result<()>;
fn put_fixed_bytes(self: &mut Self, data: &[u8]) -> Result<()> {
for b in data { self.put_u8(*b)?; }
Ok(())
}
fn put_var_bytes(self: &mut Self, data: &[u8]) -> Result<()> {
self.put_unsigned(data.len())?;
self.put_fixed_bytes(data)
}
fn put_str(self: &mut Self, str: &str) -> Result<()> {
self.put_var_bytes(str.as_bytes())
}
fn put_u16(self: &mut Self, mut value: u16) -> Result<()>{
let mut result = [0u8; 2];
for i in (0..result.len()).rev() {
result[i] = value as u8;
value = value >> 8;
}
self.put_fixed_bytes(&result)
}
fn put_u32(self: &mut Self, mut value: u32) -> Result<()> {
let mut result = [0u8; 4];
for i in (0..result.len()).rev() {
result[i] = value as u8;
value = value >> 8;
}
self.put_fixed_bytes(&result)
}
fn put_u64(self: &mut Self, mut value: u64) -> Result<()> {
let mut result = [0u8; 8];
for i in (0..result.len()).rev() {
result[i] = value as u8;
value = value >> 8;
}
self.put_fixed_bytes(&result)
}
fn put_i64(self: &mut Self, value: i64) -> Result<()> {
self.put_u64(value as u64)
}
fn put_i32(self: &mut Self, value: i32) -> Result<()> {
self.put_u32(value as u32)
}
fn put_i16(self: &mut Self, value: i16) -> Result<()> {
self.put_u16(value as u16)
}
fn put_i8(self: &mut Self, value: i8) -> Result<()>{
self.put_u8(value as u8)
}
/// Put unsigned value to compressed variable-length format, `Smartint` in the bipack
/// terms. This format is used to store size of variable-length binaries and strings.
/// Use [crate::bipack_source::BipackSource::get_unsigned] to unpack it.
fn put_unsigned<T: IntoU64>(self: &mut Self, number: T) -> Result<()> {
let value = number.into_u64();
let mut encode_seq = |ty: u8, bytes: &[u64]| -> Result<()> {
if bytes.len() == 0 { self.put_u8(0)?; } else {
if bytes[0] > V0LIMIT { panic!("first byte is too big (internal error)"); }
self.put_u8((ty & 0x03) | ((bytes[0] as u8) << 2))?;
for i in 1..bytes.len() {
self.put_u8(bytes[i] as u8)?;
}
}
Ok(())
};
if value < V0LIMIT {
encode_seq(0, &[value])?;
} else if value < V1LIMIT {
encode_seq(1, &[value & 0x3F, value >> 6])?;
} else if value < V2LIMIT {
encode_seq(2, &[value & 0x3f, value >> 6, value >> 14])?;
} else {
encode_seq(3, &[value & 0x3f, value >> 6, value >> 14])?;
self.put_var_unsigned(value >> 22)?;
}
Ok(())
}
/// Put variable-length encoded integer value. it is packed just like variable-length
/// unsigned value except that LSB (bit 0) is used as negative number flag (when set,
/// the encoded number is negative).
///
/// Note that for really big number using [BipackSink::put_i64] could be more effective
/// than the variable-length.
fn put_signed(self: &mut Self, val: i64) -> Result<()> {
let (neg, val) = if val < 0 { (1, -val) } else { (0, val) };
self.put_unsigned( (neg as u64) | ((val as u64) << 1) )
}
fn put_var_unsigned(self: &mut Self, value: u64) -> Result<()>{
let mut rest = value;
loop {
let x = rest & 127;
rest = rest >> 7;
if rest > 0 {
self.put_u8((x | 0x80) as u8)?;
} else {
self.put_u8(x as u8)?
}
if rest == 0 { break; }
}
Ok(())
}
}
impl BipackSink for Vec<u8> {
fn put_u8(self: &mut Self, data: u8) -> Result<()> {
self.push(data);
Ok(())
}
}

203
src/bipack_source.rs
View File

@ -1,203 +0,0 @@
// Copyright 2023 by Sergey S. Chernov.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fmt::{Display, Formatter};
use std::string::FromUtf8Error;
use crate::bipack_source::BipackError::NoDataError;
/// Result of error-aware bipack function
pub type Result<T> = std::result::Result<T, BipackError>;
/// There is not enough data to fulfill the request
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub enum BipackError {
NoDataError,
BadEncoding(FromUtf8Error),
BufferOverflow,
Message(String),
BadFormat(String),
Eof,
ExtraBytes,
NotSupported,
NotImplemented,
NotPossible
}
impl std::error::Error for BipackError {}
impl Display for BipackError {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
write!(f, "{:?}", self)
}
}
/// Data source compatible with mp_bintools serialization. It supports
/// fixed-size integers in right order and varint ans smartint encodings
/// separately. There is out of the box implementation for [`Vec<u8>`], and
/// it is easy to implements your own.
///
/// To implement source for other type, implement just [BipackSource::get_u8],
/// [BipackSource::eof] or maybe also
/// [BipackSource::get_fixed_bytes] for effectiveness.
///
/// Unlike the [crate::bipack_sink::BipackSink] the source is returning errors. This is because
/// it often appears when reading data do not correspond to the packed one, and this is an often
/// case that requires proper reaction, not just a panic attack :)
pub trait BipackSource {
fn get_u8(self: &mut Self) -> Result<u8>;
/// Tribute to the tradition. End Of File, if true, there is no unread data left
/// in the source.
fn eof(self: &Self) -> bool;
fn get_u16(self: &mut Self) -> Result<u16> {
Ok(((self.get_u8()? as u16) << 8) + (self.get_u8()? as u16))
}
fn get_u32(self: &mut Self) -> Result<u32> {
Ok(((self.get_u16()? as u32) << 16) + (self.get_u16()? as u32))
}
fn get_u64(self: &mut Self) -> Result<u64> {
Ok(((self.get_u32()? as u64) << 32) | (self.get_u32()? as u64))
}
fn get_i64(self: &mut Self) -> Result<i64> {
Ok(self.get_u64()? as i64)
}
fn get_i32(self: &mut Self) -> Result<i32> {
Ok(self.get_u32()? as i32)
}
fn get_i16(self: &mut Self) -> Result<i16> {
Ok(self.get_u16()? as i16)
}
fn get_i8(self: &mut Self) -> Result<i8> {
Ok(self.get_u8()? as i8)
}
/// Unpack variable-length packed unsigned value, used also internally to store size
/// of arrays, binary data, strings, etc. To pack use
/// [crate::bipack_sink::BipackSink::put_unsigned()].
fn get_unsigned(self: &mut Self) -> Result<u64> {
let mut get = || -> Result<u64> { Ok(self.get_u8()? as u64) };
let first = get()?;
let mut ty = first & 3;
let mut result = first >> 2;
if ty == 0 { return Ok(result); }
ty -= 1;
result = result + (get()? << 6);
if ty == 0 { return Ok(result); }
ty -= 1;
result = result + (get()? << 14);
if ty == 0 { return Ok(result); }
Ok(result | (self.get_varint_unsigned()? << 22))
}
/// Unpack variable-length signed value, packed with
/// [crate::bipack_sink::BipackSink::put_signed], see it for the packing details.
fn get_signed(self: &mut Self) -> Result<i64> {
let value = self.get_unsigned()?;
let result = (value >> 1) as i64;
Ok(if value & 1 != 0 { -result } else { result } )
}
/// read 8-bytes varint-packed unsigned value from the source. We dont' recommend
/// using it directly; use [BipackSource::get_unsigned] instead.
fn get_varint_unsigned(self: &mut Self) -> Result<u64> {
let mut result = 0u64;
let mut count = 0;
loop {
let x = self.get_u8()? as u64;
result = result | ((x & 0x7F) << count);
if (x & 0x80) == 0 { return Ok(result); }
count += 7
}
}
/// read 2-bytes unsigned value from the source as smartint-encoded, same as
/// [BipackSource::get_unsigned] as u16
fn get_packed_u16(self: &mut Self) -> Result<u16> {
Ok(self.get_unsigned()? as u16)
}
/// read 4-bytes unsigned value from the source
/// read 2-bytes unsigned value from the source as smartint-encoded, same as
/// [BipackSource::get_unsigned] as u32.
fn get_packed_u32(self: &mut Self) -> Result<u32> { Ok(self.get_unsigned()? as u32) }
/// read exact number of bytes from the source as a vec.
fn get_fixed_bytes(self: &mut Self, size: usize) -> Result<Vec<u8>> {
let mut result = Vec::with_capacity(size);
for i in 0..size { result.push(self.get_u8()?); }
Ok(result)
}
/// Read variable-length byte array from the source (with packed size), created
/// by [crate::bipack_sink::BipackSink::put_var_bytes] or
/// [crate::bipack_sink::BipackSink::put_str]. The size is encoded the same way as does
/// [crate::bipack_sink::BipackSink::put_unsigned] and can be manually read by
/// [BipackSource::get_unsigned].
fn get_var_bytes(self: &mut Self) -> Result<Vec<u8>> {
let size = self.get_unsigned()? as usize;
self.get_fixed_bytes(size)
}
/// REad a variable length string from a source packed with
/// [crate::bipack_sink::BipackSink::put_str]. It is a variable sized array fo utf8 encoded
/// characters.
fn get_str(self: &mut Self) -> Result<String> {
String::from_utf8(
self.get_var_bytes()?
).or_else(|e| Err(BipackError::BadEncoding(e)))
}
}
/// The bipack source capable of extracting data from a slice.
/// use [SliceSource::from()] to create one.
pub struct SliceSource<'a> {
data: &'a [u8],
position: usize,
}
impl<'a> SliceSource<'a> {
pub fn from(src: &'a [u8]) -> SliceSource<'a> {
SliceSource { data: src, position: 0 }
}
}
impl<'x> BipackSource for SliceSource<'x> {
fn get_u8(self: &mut Self) -> Result<u8> {
if self.position >= self.data.len() {
Err(NoDataError)
} else {
let result = self.data[self.position];
self.position += 1;
Ok(result)
}
}
fn eof(self: &Self) -> bool {
self.data.len() >= self.position
}
}

26
src/buffer_sink.rs
View File

@ -1,26 +0,0 @@
use crate::bipack_sink::BipackSink;
use crate::bipack_source::BipackError;
use crate::bipack_source::Result;
pub struct BufferSink<'a> {
pub buffer: & 'a mut [u8],
pub pos: usize
}
impl<'a> BufferSink<'a> {
pub fn result_slice(self) -> & 'a [u8] {
&self.buffer[0..self.pos]
}
}
impl<'a> BipackSink for BufferSink<'a> {
fn put_u8(self: &mut Self, data: u8) -> Result<()> {
if self.pos >= self.buffer.len() {
Err(BipackError::BufferOverflow)
} else {
self.buffer[self.pos] = data;
self.pos+=1;
Ok(())
}
}
}

35
src/contrail.rs
View File

@ -1,35 +0,0 @@
//! Contrails are byte arrays protected by a short crc8 checksum which is enough
//! to protect against human typing error in most cases.
use crate::crc::Crc8;
/// Check that the data is a valid contrail, e.g. contains a valid checksum in the
/// first byte.
pub fn is_valid_contrail(data: &[u8]) -> bool {
Crc8::calc(&data[1..]) == data[0]
}
/// Create contrail from a given bytes slice. Return vec containing a valid
/// contrail. It is always 1 byte longer than a given data.
pub fn create_contrail(data: &[u8]) -> Vec<u8> {
let mut result = Vec::with_capacity(data.len()+1);
result.push(Crc8::calc(&data));
result.extend_from_slice(&data);
result
}
#[cfg(test)]
mod tests {
use crate::contrail::{create_contrail, is_valid_contrail};
#[test]
fn test_contrail() {
// let a = [1,2,3,4,5];
let mut c = create_contrail(&[1,2,3,4,5]);
// println!("{}", to_dump(&c));
assert_eq!(c[0], 134);
assert_eq!(true, is_valid_contrail(&c));
c[1] = 2;
assert_eq!(false, is_valid_contrail(&c));
}
}

73
src/crc.rs
View File

@ -1,73 +0,0 @@
pub struct Crc8 {
table : [u8; 256]
}
impl Crc8 {
pub fn create_msb(polynomial : u8) -> Crc8 {
let msbit : u8 = 0x80;
let mut t : u8 = msbit;
let mut tmp : u8;
let mut i : u32 = 1;
let mut idx : u8;
let mut table : [u8; 256] = [0; 256];
while i < 256 {
if t & msbit != 0 { tmp = polynomial; } else { tmp = 0; }
t = (t << 1) ^ tmp;
for j in 0..i {
idx = (i+j) as u8;
table[idx as usize] = table[j as usize] ^ t;
}
i *= 2;
}
return Crc8{ table };
}
pub fn create_lsb(polynomial :u8) -> Crc8 {
let mut i :u32 = 0xff;
let mut j :u32;
let mut t :u8 = 1;
let mut tmp :u8;
let mut idx :u8;
let mut table : [u8; 256] = [0; 256];
while i != 0 {
if t & 1 != 0 { tmp = polynomial; } else { tmp = 0; }
t = (t >> 1) ^ tmp;
j = 0;
while j < 256 {
idx = (i+j) as u8;
table[idx as usize] = table[j as usize] ^ t;
j += 2 * i;
}
i >>= 1;
}
return Crc8{ table };
}
/// Update CRC using previously calculated value and return new one.
pub fn update(&self, buffer : &[u8], crc: u8) -> u8 {
let mut crc_tmp = crc;
for i in 0..buffer.len() {
crc_tmp = self.table[(crc_tmp ^ buffer[i]) as usize];
}
return crc_tmp;
}
/// Calculate crc8 with default presets compatible with defaults of mp_binary
/// package used in kotlin Divan interfaces (msb, polynomial=0xA7, this is
/// bluetooth presets btw).
pub fn calc(data: &[u8]) -> u8 {
let crc = Crc8::create_msb(0xA7);
crc.update(data, 0)
}
}
#[test]
fn test_crc8() {
let a = [1,2,3,4,5];
assert_eq!(134, Crc8::calc(&a));
}

444
src/de.rs
View File

@ -1,444 +0,0 @@
use serde::de::{self, DeserializeSeed, IntoDeserializer, MapAccess, SeqAccess, Visitor};
use serde::de::value::U32Deserializer;
use serde::Deserialize;
use crate::bipack_source::{BipackError, BipackSource, SliceSource};
use crate::error::Result;
pub struct Deserializer<T: BipackSource> {
// This string starts with the input data and characters are truncated off
// the beginning as data is parsed.
input: T,
}
pub fn from_bytes<'b, T: Deserialize<'b>>(source: &[u8]) -> Result<T> {
let mut des = Deserializer { input: SliceSource::from(&source) };
T::deserialize(&mut des)
}
impl<'de, 'a, T: BipackSource> de::Deserializer<'de> for &'a mut Deserializer<T> {
type Error = BipackError;
fn deserialize_any<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
Err(BipackError::NotPossible)
}
fn deserialize_bool<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_bool(if self.input.get_u8()? == 0 { false } else { true })
}
fn deserialize_i8<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_i8(self.input.get_i8()?)
}
fn deserialize_i16<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_i16(self.input.get_signed()? as i16)
}
fn deserialize_i32<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_i32(self.input.get_signed()? as i32)
}
fn deserialize_i64<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_i64(self.input.get_signed()?)
}
fn deserialize_u8<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_u8(self.input.get_u8()?)
}
fn deserialize_u16<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_u16(self.input.get_unsigned()? as u16)
}
fn deserialize_u32<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_u32(self.input.get_unsigned()? as u32)
}
fn deserialize_u64<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_u64(self.input.get_unsigned()?)
}
fn deserialize_f32<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
Err(BipackError::NotImplemented)
}
fn deserialize_f64<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
Err(BipackError::NotImplemented)
}
fn deserialize_char<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
let ch = self.input.get_str()?;
if ch.len() != 1 {
Err(BipackError::BadFormat(format!("Char length is {}, should be 1 {:?}", ch.len(), ch)))
} else {
visitor.visit_char(ch.chars().next().unwrap())
}
}
fn deserialize_str<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_string(self.input.get_str()?)
}
fn deserialize_string<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_string(self.input.get_str()?)
}
fn deserialize_bytes<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_byte_buf(self.input.get_var_bytes()?)
}
fn deserialize_byte_buf<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_byte_buf(self.input.get_var_bytes()?)
}
fn deserialize_option<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
let marker = self.input.get_u8()?;
if marker == 0 {
visitor.visit_none()
} else {
visitor.visit_some(self)
}
}
fn deserialize_unit<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_unit()
}
fn deserialize_unit_struct<V>(self, name: &'static str, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
self.deserialize_unit(visitor)
}
fn deserialize_newtype_struct<V>(self, name: &'static str, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_newtype_struct(self)
}
fn deserialize_seq<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
let size = self.input.get_unsigned()? as usize;
visitor.visit_seq(SimpleSeq::new(self, size))
}
fn deserialize_tuple<V>(self, len: usize, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_seq(SimpleSeq::new(self, len))
}
fn deserialize_tuple_struct<V>(self, name: &'static str, len: usize, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_seq(SimpleSeq::new(self, len))
}
fn deserialize_map<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
let size = self.input.get_unsigned()?;
visitor.visit_map(SimpleMap { de: self, size: size as usize })
}
fn deserialize_struct<V>(self, name: &'static str, fields: &'static [&'static str], visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
visitor.visit_seq(SimpleSeq::new(self, fields.len()))
}
#[inline]
fn deserialize_enum<V>(
self,
_name: &'static str,
_variants: &'static [&'static str],
visitor: V,
) -> Result<V::Value>
where
V: Visitor<'de>,
{
visitor.visit_enum(self)
}
fn deserialize_identifier<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
Err(BipackError::NotPossible)
}
fn deserialize_ignored_any<V>(self, visitor: V) -> std::result::Result<V::Value, Self::Error> where V: Visitor<'de> {
Err(BipackError::NotPossible)
}
fn is_human_readable(&self) -> bool {
false
}
}
impl<'de, 'a, T: BipackSource> de::EnumAccess<'de> for &'a mut Deserializer<T> {
type Error = BipackError;
type Variant = Self;
#[inline]
fn variant_seed<V: DeserializeSeed<'de>>(self, seed: V) -> Result<(V::Value, Self)> {
let varint: u32 = self.input.get_unsigned()? as u32;
let v = DeserializeSeed::deserialize::<U32Deserializer<BipackError>>(
seed,
varint.into_deserializer())?;
Ok((v, self))
}
}
impl<'de, 'a, T: BipackSource> de::VariantAccess<'de> for &'a mut Deserializer<T> {
type Error = BipackError;
#[inline]
fn unit_variant(self) -> Result<()> {
Ok(())
}
#[inline]
fn newtype_variant_seed<V: DeserializeSeed<'de>>(self, seed: V) -> Result<V::Value> {
DeserializeSeed::deserialize(seed, self)
}
#[inline]
fn tuple_variant<V: Visitor<'de>>(self, len: usize, visitor: V) -> Result<V::Value> {
serde::de::Deserializer::deserialize_tuple(self, len, visitor)
}
#[inline]
fn struct_variant<V: Visitor<'de>>(
self,
fields: &'static [&'static str],
visitor: V,
) -> Result<V::Value> {
serde::de::Deserializer::deserialize_tuple(self, fields.len(), visitor)
}
}
struct SimpleSeq<'a, T: BipackSource> {
de: &'a mut Deserializer<T>,
size: usize,
}
impl<'a, T: BipackSource> SimpleSeq<'a, T> {
fn new(de: &'a mut Deserializer<T>, size: usize) -> Self {
SimpleSeq {
de,
size,
}
}
}
// `SeqAccess` is provided to the `Visitor` to give it the ability to iterate
// through elements of the sequence.
impl<'de, 'a, T: BipackSource> SeqAccess<'de> for SimpleSeq<'a, T> {
type Error = BipackError;
fn next_element_seed<S>(&mut self, seed: S) -> Result<Option<S::Value>>
where
S: DeserializeSeed<'de>,
{
if self.size < 1 {
return Ok(None);
}
self.size -= 1;
seed.deserialize(&mut *self.de).map(Some)
}
}
struct SimpleMap<'a, T: BipackSource> {
de: &'a mut Deserializer<T>,
size: usize,
}
impl<'de, 'a, T: BipackSource> MapAccess<'de> for SimpleMap<'a, T> {
type Error = BipackError;
fn next_key_seed<K>(&mut self, seed: K) -> std::result::Result<Option<K::Value>, Self::Error> where K: DeserializeSeed<'de> {
if self.size < 1 {
Ok(None)
} else {
self.size -= 1;
seed.deserialize(&mut *self.de).map(Some)
}
}
fn next_value_seed<V>(&mut self, seed: V) -> std::result::Result<V::Value, Self::Error> where V: DeserializeSeed<'de> {
seed.deserialize(&mut *self.de)
}
}
#[cfg(test)]
mod tests {
use std::collections::{HashMap, HashSet};
use std::fmt::Debug;
use serde::{Deserialize, Serialize};
use crate::bipack_source::BipackError;
use crate::de::from_bytes;
use crate::error::Result;
use crate::ser::{to_buffer, to_bytes};
use crate::tools::{to_dump, ToDump};
#[test]
fn test_ints() -> Result<()> {
// #[derive(Deserialize, PartialEq, Debug)]
// struct Test {
// int: u32,
// seq: Vec<String>,
// }
let b = vec![7];
assert_eq!(7u8, from_bytes(&vec![7u8])?);
// let j = r#"{"int":1,"seq":["a","b"]}"#;
// let expected = Test {
// int: 1,
// seq: vec!["a".to_owned(), "b".to_owned()],
// };
// assert_eq!(expected, from_str(j).unwrap());
Ok(())
}
#[test]
fn test_struct() -> Result<()> {
#[derive(Serialize, Deserialize, PartialEq, Debug)]
struct Test {
int: u32,
seq: Vec<String>,
}
let expected = Test {
int: 1,
seq: vec!["a".to_owned(), "b".to_owned()],
};
let packed = to_bytes(&expected)?;
println!("::{}", to_dump(&packed));
let unpacked: Test = from_bytes(&packed)?;
println!("::{:?}", unpacked);
assert_eq!(&expected, &unpacked);
Ok(())
// let j = r#"{"int":1,"seq":["a","b"]}"#;
// assert_eq!(expected, from_str(j).unwrap());
}
#[test]
fn test_map() -> Result<()> {
let mut src = HashMap::new();
// src.insert("foo".to_string(), 1);
src.insert("foo".to_string(), 42);
src.insert("bar".to_string(), 1);
src.insert("baz".to_string(), 17);
let packed = to_bytes(&src)?;
println!("{}", to_dump(&packed));
let restored: HashMap<String, i32> = from_bytes(&packed)?;
println!("{:?}", restored);
assert_eq!(src, restored);
Ok(())
}
#[test]
fn test_set() -> Result<()> {
let src = HashSet::from(["foo", "bar", "buz"].map(|i| i.to_string()));
let packed = to_bytes(&src)?;
println!("{}", to_dump(&packed));
let restored: HashSet<String> = from_bytes(&packed)?;
println!("{:?}", restored);
assert_eq!(src, restored);
Ok(())
}
fn testeq<T: Serialize + Deserialize<'static> + PartialEq + Debug>(x: &T) {
{
let packed = to_bytes(x).unwrap();
println!("packed {:?}:\n{}", x, to_dump(&packed));
let z: T = from_bytes(&packed).unwrap();
assert_eq!(*x, from_bytes(&packed).unwrap());
if packed.len() > 1 {
let mut small_buffer = [0u8; 1];
let result = to_buffer(x, &mut small_buffer);
assert_eq!(true, result.is_err());
assert_eq!(BipackError::BufferOverflow, result.err().unwrap());
}
}
let mut buffer = [0u8; 128];
let packed2 = to_buffer(x, &mut buffer).unwrap();
assert_eq!(*x, from_bytes(&packed2).unwrap());
}
#[test]
fn test_enum() -> Result<()> {
#[derive(Serialize, Deserialize, Debug, PartialEq)]
enum E {
Unit,
Unit2,
Newtype(u32),
Tuple(u32, u32),
Struct { a: u32 },
}
let packed = to_bytes(&E::Newtype(7))?;
println!("{}", to_dump(&packed));
let r: E = from_bytes(&packed)?;
println!("{:?}", r);
testeq(&E::Unit);
testeq(&E::Unit2);
testeq(&E::Newtype(101));
testeq(&E::Tuple(17, 42));
testeq(&E::Struct { a: 19 });
Ok(())
}
#[test]
fn test_arrays() {
let x = (1, 2, 3, 4, 5);
// println!("{:?}",x);
// println!("{}", to_dump(to_bytes(&x).unwrap().as_slice()));
assert_eq!(5, to_bytes(&x).unwrap().len());
testeq(&x);
let y: [u8; 5] = [1, 2, 3, 4, 5];
println!("{}", to_dump(to_bytes(&y).unwrap().as_slice()));
assert_eq!(5, to_bytes(&y).unwrap().len());
assert_eq!(y, to_bytes(&y).unwrap()[..]);
testeq(&y);
let z = vec![1, 2, 3, 4, 5];
assert_eq!(6, to_bytes(&z).unwrap().len());
assert_eq!(z, to_bytes(&z).unwrap()[1..]);
}
// This is real life sample bug:
#[derive(Debug,Serialize,Deserialize,Eq, PartialEq)]
pub enum LogLevel {
Debug,
Info,
Warning,
Error,
}
#[derive(Serialize,Deserialize,Debug,PartialEq)]
pub struct LogArgs(
pub LogLevel,
pub String
);
#[test]
fn test_logargs() {
let x = LogArgs(LogLevel::Debug,"hello".to_string());
let packed = to_bytes(&x).unwrap();
// println!("{}", packed.to_dump());
let y: LogArgs = from_bytes(&packed).unwrap();
// println!("{:?}", y);
assert_eq!(LogLevel::Debug, y.0);
assert_eq!("hello", y.1);
assert_eq!(7, packed.len());
}
#[test]
fn test_str() {
let src = "hello";
let x = to_bytes(src);
println!("{}", x.unwrap().to_dump());
}
}

20
src/error.rs
View File

@ -1,20 +0,0 @@
use std;
use std::fmt::Display;
use serde::{de, ser};
use crate::bipack_source::BipackError;
pub type Result<T> = std::result::Result<T, BipackError>;
impl ser::Error for BipackError {
fn custom<T: Display>(msg: T) -> Self {
BipackError::Message(msg.to_string())
}
}
impl de::Error for BipackError {
fn custom<T: Display>(msg: T) -> Self {
BipackError::Message(msg.to_string())
}
}

189
src/fixint.rs
View File

@ -1,189 +0,0 @@
//! # Fixed Size Integers
//!
//! In some cases, the use of variably length encoded data may not be
//! preferable. These modules, for use with `#[serde(with = ...)]`
//! "opt out" of variable length encoding.
//!
//! Support explicitly not provided for `usize` or `isize`, as
//! these types would not be portable between systems of different
//! pointer widths.
//!
//! Although all data in Postcard is typically encoded in little-endian
//! order, these modules provide a choice to the user to encode the data
//! in either little or big endian form, which may be useful for zero-copy
//! applications.
use serde::{Deserialize, Serialize, Serializer};
/// Use with the `#[serde(with = "bipack_ru::fixint::le")]` field attribute.
/// Disables smartint serialization/deserialization for the specified integer
/// field. The integer will always be serialized in the same way as a fixed
/// size array, in **Little Endian** order on the wire.
///
/// ```rust
/// # use serde::Serialize;
/// #[derive(Serialize)]
/// pub struct DefinitelyLittleEndian {
/// #[serde(with = "bipack_ru::fixint::le")]
/// x: u16,
/// }
/// ```
pub mod le {
use serde::{Deserialize, Deserializer, Serialize, Serializer};
use super::LE;
/// Serialize the integer value as a little-endian fixed-size array.
pub fn serialize<S, T>(val: &T, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
T: Copy,
LE<T>: Serialize,
{
LE(*val).serialize(serializer)
}
/// Deserialize the integer value from a little-endian fixed-size array.
pub fn deserialize<'de, D, T>(deserializer: D) -> Result<T, D::Error>
where
D: Deserializer<'de>,
LE<T>: Deserialize<'de>,
{
LE::<T>::deserialize(deserializer).map(|x| x.0)
}
}
/// Use with the `#[serde(with = "bipack_ru::fixint::be")]` field attribute.
/// Disables smartint serialization/deserialization for the specified integer
/// field. The integer will always be serialized in the same way as a fixed
/// size array, in **Big Endian** order on the wire.
///
/// ```rust
/// # use serde::Serialize;
/// #[derive(Serialize)]
/// pub struct DefinitelyBigEndian {
/// #[serde(with = "bipack_ru::fixint::be")]
/// x: u16,
/// }
/// ```
pub mod be {
use serde::{Deserialize, Deserializer, Serialize, Serializer};
use super::BE;
/// Serialize the integer value as a big-endian fixed-size array.
pub fn serialize<S, T>(val: &T, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
T: Copy,
BE<T>: Serialize,
{
BE(*val).serialize(serializer)
}
/// Deserialize the integer value from a big-endian fixed-size array.
pub fn deserialize<'de, D, T>(deserializer: D) -> Result<T, D::Error>
where
D: Deserializer<'de>,
BE<T>: Deserialize<'de>,
{
BE::<T>::deserialize(deserializer).map(|x| x.0)
}
}
#[doc(hidden)]
pub struct LE<T>(T);
#[doc(hidden)]
pub struct BE<T>(T);
macro_rules! impl_fixint {
($( $int:ty ),*) => {
$(
impl Serialize for LE<$int> {
#[inline]
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
self.0.to_le_bytes().serialize(serializer)
}
}
impl<'de> Deserialize<'de> for LE<$int> {
#[inline]
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
<_ as Deserialize>::deserialize(deserializer)
.map(<$int>::from_le_bytes)
.map(Self)
}
}
impl Serialize for BE<$int> {
#[inline]
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
self.0.to_be_bytes().serialize(serializer)
}
}
impl<'de> Deserialize<'de> for BE<$int> {
#[inline]
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
<_ as Deserialize>::deserialize(deserializer)
.map(<$int>::from_be_bytes)
.map(Self)
}
}
)*
};
}
impl_fixint![i16, i32, i64, i128, u16, u32, u64, u128];
#[cfg(test)]
mod tests {
use serde::{Deserialize, Serialize};
use crate::de::from_bytes;
use crate::ser::to_bytes;
#[test]
fn test_little_endian() {
#[derive(Serialize, Deserialize, Debug, PartialEq, Eq)]
pub struct DefinitelyLE {
#[serde(with = "crate::fixint::le")]
x: u16,
}
let input = DefinitelyLE { x: 0xABCD };
let serialized = to_bytes(&input).unwrap();
assert_eq!(serialized, &[0xCD, 0xAB]);
let deserialized: DefinitelyLE = from_bytes(&serialized).unwrap();
assert_eq!(deserialized, input);
}
#[test]
fn test_big_endian() {
#[derive(Serialize, Deserialize, Debug, PartialEq, Eq)]
pub struct DefinitelyBE {
#[serde(with = "crate::fixint::be")]
x: u16,
}
let input = DefinitelyBE { x: 0xABCD };
let serialized = to_bytes(&input).unwrap();
assert_eq!(serialized, &[0xAB, 0xCD]);
let deserialized: DefinitelyBE = from_bytes(&serialized).unwrap();
assert_eq!(deserialized, input);
}
}

296
src/lib.rs
View File

@ -1,296 +0,0 @@
// Copyright 2023 by Sergey S. Chernov.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! # Bipack codec
//!
//! The set of tools to effectively encode and decode bipack values. It is internationally
//! minimalistic to be used wit Divan smart-contracts where number of instructions could
//! be important.
//!
//! - [bipack_source::BipackSource] is used to decode values, there is implementation
//! [bipack_source::SliceSource] that parses binary slice. The trait only needs byte-read
//! method for the implementation.
//!
//! - [bipack_sink::BipackSink] trait that is also implemented for [`Vec<u8>`] allows to encode values
//! into the bipack format. It is the same simple to implement it for any else binary data
//! source.
//!
//! ## Utilities
//!
//! - to simplify encoding of unsigned ints the [bipack_sink::IntoU64] trait is used with
//! implementation for usual u* types.
//!
//! - [tools::to_dump] utility function converts binary data into human-readable dump as in old good
//! times (address, bytes, ASCII characters).
//!
//! - [tools::StringBuilder] minimalistic growing strings builder.
//!
//! ## About Bipack format
//!
//! This is a binary format created around the idea of bit-effectiveness and not disclosing
//! inner data structure. Unlike many known binary and text formats, like JSON, BSON, BOSS, and
//! many others, it does not includes field names into packed binaries.
//!
//! It also uses rationally-packed variable length format very effective for unsigned integers of
//! various sizes. This implementation supports sizes for u8, u16, u32 and u64. It is capable of
//! holding longer values too but for big numbers the fixed size encoding is mostly more effective.
//! This rational encoding format is called `smartint` and is internally used everywhere when one
//! need to pack unsigned number, unless the fixed size is important.
//!
//! ### Varint encoding
//!
//! Smart variable-length long encoding tools, async. It gives byte-size gain from 64 bits numbers,
//! so it is very useful when encoding big numbers or at least very bui long values. In other cases
//! [bipack_sink::BipackSink::put_unsigned] works faster, and extra bits it uses does not play
//!
//! | Bytes sz | varint bits | smartint bits |
//! |:-----:|:------:|:---------:|
//! | 1 | 7 | 6 |
//! | 2 | 14 | 14 |
//! | 3 | 21 | 22 |
//! | 4 | 28 | 29 |
//! | 5 | 35 | 36 |
//! | 6+ | 7*N | 7*N+1 |
//! | 9 | 63 | 64 |
//! | 10 | 64 | --- |
//!
//! In other words, except for very small numbers smartint
//! gives 1 data bit gain for the same packed byte size. For example,
//! full size 64 bits number with smartint takes one byte less (9 bytes vs. 10 in Varint).
//!
//! So, except for values in range 32..63 it gives same or better byte size effectiveness
//! than `Varint`. In particular:
//!
//! The effect of it could be interpreted as:
//!
//! | number values | size |
//! |:--------------|:------:|
//! | 0..31 | same |
//! | 32..63 | worse 1 byte |
//! | 64..1048573 | same |
//! | 1048576..2097151 | 1 byte better |
//! | 2097152..134217727 | same |
//! | 134217728..268435456 | 1 byte better |
//!
//! etc.
//!
//! ## Encoding format
//!
//! Enncoded data could be 1 or more bytes in length. Data are
//! packed as follows:
//!
//! | byte offset | bits range | field |
//! |-------------|------------|-------|
//! | 0 | 0..1 | type |
//! | 0 | 2..7 | v0 |
//! | 1 | 0..7 | v1 (when used) |
//! | 2 | 0..7 | v2 (when used) |
//!
//! Then depending on the `type` field:
//!
//! | type | encoded |
//! |------|---------|
//! | 0 | v0 is the result 0..64 (or -32..32) |
//! | 1 | v0 ## v1 are the result, 14 bits |
//! | 2 | v0 ## v1 ## v2 are the result, 22bits
//! | 3 | v0, ## v1 ## v2 ## (varint encoded rest) |
//!
//! Where `##` means bits concatenation. The bits are interpreted as BIG ENDIAN,
//! for example `24573` will be encoded to `EA FF 02`
//!
//!
#![allow(dead_code)]
#![allow(unused_variables)]
pub mod bipack_source;
pub mod bipack_sink;
pub mod tools;
pub mod bipack;
pub mod error;
pub mod ser;
pub mod de;
pub mod crc;
pub mod contrail;
pub mod fixint;
pub mod buffer_sink;
pub use serde::{Deserialize,Serialize};
#[cfg(test)]
mod tests {
use base64::Engine;
use crate::bipack;
use crate::bipack::{BiPackable, BiUnpackable};
use crate::bipack_sink::BipackSink;
use crate::bipack_source::{BipackSource, Result, SliceSource};
use crate::tools::to_dump;
#[test]
fn fixed_unpack() -> Result<()> {
let mut src = Vec::new();
base64::engine::general_purpose::STANDARD_NO_PAD
.decode_vec("B/oAAAEB0AAAANjLgKAv", &mut src)
.expect("decoded vector");
println!(": {}", hex::encode(&src));
let mut ss = SliceSource::from(&src);
assert_eq!(7, ss.get_u8()?);
assert_eq!(64000, ss.get_u16()?);
assert_eq!(66000, ss.get_u32()?);
assert_eq!(931127140399, ss.get_u64()?);
Ok(())
}
#[test]
fn smartint_unpack() -> Result<()> {
let mut src = Vec::new();
base64::engine::general_purpose::STANDARD_NO_PAD
.decode_vec("BwLoA0IHBL+AAq7GDQ", &mut src)
.expect("decoded vector");
// println!("{}", hex::encode(&src));
let mut ss = SliceSource::from(&src);
assert_eq!(7, ss.get_u8()?);
assert_eq!(64000, ss.get_packed_u16()?);
assert_eq!(66000, ss.get_packed_u32()?);
assert_eq!(931127140399, ss.get_unsigned()?);
Ok(())
}
#[test]
fn fixed_pack() {
let mut data: Vec<u8> = Vec::new();
data.put_u8(7).unwrap();
data.put_u16(64000).unwrap();
data.put_u32(66000).unwrap();
data.put_u64(931127140399).unwrap();
assert_eq!("07fa00000101d0000000d8cb80a02f", hex::encode(&data));
}
#[test]
fn smart_pack() {
let mut data: Vec<u8> = Vec::new();
data.put_u8(7).unwrap();
data.put_unsigned(64000u16).unwrap();
data.put_unsigned(66000u32).unwrap();
data.put_unsigned(931127140399u64).unwrap();
// println!("?? {}", hex::encode(&data));
assert_eq!("0702e803420704bf8002aec60d", hex::encode(&data));
}
#[test]
fn pack_varbinaries_and_string() {
let mut data = Vec::<u8>::new();
data.put_str("Hello, rupack!").unwrap();
println!("size ${}\n{}",data.len(), to_dump(&data));
let mut src = SliceSource::from(&data);
assert_eq!("Hello, rupack!", src.get_str().unwrap());
}
#[test]
fn test_signed() -> Result<()> {
fn test64(value: i64) -> Result<()> {
let mut x = Vec::new();
x.put_i64(value).unwrap();
assert_eq!(value, SliceSource::from(&x).get_i64()?);
Ok(())
}
test64(0)?;
test64(1)?;
test64(-1)?;
test64(9223372036854775807)?;
test64(-9223372036854775808)?;
fn test32(value: i32) -> Result<()> {
let mut x = Vec::new();
x.put_i32(value).unwrap();
assert_eq!(value, SliceSource::from(&x).get_i32()?);
Ok(())
}
test32(0)?;
test32(1)?;
test32(-1)?;
test32(2147483647)?;
test32(-2147483648)?;
fn test16(value: i16) -> Result<()> {
let mut x = Vec::new();
x.put_i16(value).unwrap();
assert_eq!(value, SliceSource::from(&x).get_i16()?);
Ok(())
}
test16(0)?;
test16(1)?;
test16(-1)?;
test16(32767)?;
test16(-32768)?;
Ok(())
}
#[test]
fn test_dump() {
for l in 0..64 {
let mut d2 = Vec::new();
for u in 0..l {
d2.push(u as u8);
}
// println!("size {}\n{}", d2.len(), to_dump(&d2));
if d2.len() == 41 {
let x = to_dump(&d2);
assert_eq!(x, "0000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
0010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................|
0020 20 21 22 23 24 25 26 27 28 | !\"#$%&'( |\n");
}
}
}
#[test]
fn test_varsigned() -> Result<()> {
fn test(value: i64) -> Result<()> {
let mut x = Vec::new();
x.put_signed(value).unwrap();
assert_eq!(value, SliceSource::from(&x).get_signed()?);
Ok(())
}
fn test2(value: i64) -> Result<()> {
test(value)?;
test(-value)?;
Ok(())
}
test(0)?;
test2(1)?;
test2(2)?;
test2(64)?;
test2(65)?;
test2(127)?;
test2(128)?;
test2(255)?;
test2(256)?;
test2(2147483647)?;
test2(2222147483647)?;
Ok(())
}
#[test]
fn test_packer() -> Result<()>{
let a = 177u32;
let b = "hello!";
let sink = bipack!(a, b);
println!("{}", to_dump(&sink));
let mut source = SliceSource::from(&sink);
let a1 = u32::bi_unpack(&mut source)?;
let s1 = String::bi_unpack(&mut source)?;
assert_eq!(177u32, a1);
assert_eq!("hello!", s1);
Ok(())
}
}

439
src/ser.rs
View File

@ -1,439 +0,0 @@
use serde::{ser, Serialize};
use crate::bipack_sink::{BipackSink, IntoU64};
use crate::bipack_source::BipackError;
use crate::buffer_sink::BufferSink;
use crate::error::{Result};
pub struct Serializer<S: BipackSink> {
// This string starts empty and JSON is appended as values are serialized.
output: S,
}
pub fn to_bytes<T>(value: &T) -> Result<Vec<u8>>
where T: Serialize + ?Sized,
{
let mut serializer = Serializer { output: Vec::new() };
value.serialize(&mut serializer)?;
Ok(serializer.output)
}
pub fn to_buffer<'a, T: Serialize>(value: &T, buffer: & 'a mut [u8]) -> Result<& 'a [u8]> {
let mut serializer = Serializer { output: BufferSink { buffer, pos: 0}};
value.serialize(&mut serializer)?;
Ok(serializer.output.result_slice())
}
impl<'a, S: BipackSink> ser::Serializer for &'a mut Serializer<S> {
type Ok = ();
type Error = BipackError;
type SerializeSeq = Self;
type SerializeTuple = Self;
type SerializeTupleStruct = Self;
type SerializeTupleVariant = Self;
type SerializeMap = Self;
type SerializeStruct = Self;
type SerializeStructVariant = Self;
fn serialize_bool(self, v: bool) -> Result<()> {
self.output.put_u8(if v { 1 } else { 0 })?;
Ok(())
}
fn serialize_i8(self, v: i8) -> Result<()> {
self.output.put_i8(v)?;
Ok(())
}
fn serialize_i16(self, v: i16) -> Result<()> {
self.serialize_i64(i64::from(v))?;
Ok(())
}
fn serialize_i32(self, v: i32) -> Result<()> {
self.serialize_i64(i64::from(v))
}
fn serialize_i64(self, v: i64) -> Result<()> {
self.output.put_signed(v)?;
Ok(())
}
fn serialize_u8(self, v: u8) -> Result<()> {
self.output.put_u8(v)?;
Ok(())
}
fn serialize_u16(self, v: u16) -> Result<()> {
self.serialize_u64(v.into_u64())?;
Ok(())
}
fn serialize_u32(self, v: u32) -> Result<()> {
self.serialize_u64(v.into())?;
Ok(())
}
fn serialize_u64(self, v: u64) -> Result<()> {
self.output.put_unsigned(v)?;
Ok(())
}
fn serialize_f32(self, v: f32) -> Result<()> {
Err(ser::Error::custom("not implemented"))
}
fn serialize_f64(self, v: f64) -> Result<()> {
Err(ser::Error::custom("not implemented"))
}
/// Serialize a char as a single-character string, because this is a UTF8-encoded
/// char, e.g. variable length:
fn serialize_char(self, v: char) -> Result<()> {
self.serialize_str(&v.to_string())
}
fn serialize_str(self, v: &str) -> Result<()> {
self.output.put_str(v)?;
Ok(())
}
fn serialize_bytes(self, v: &[u8]) -> Result<()> {
self.output.put_var_bytes(v)?;
Ok(())
}
fn serialize_none(self) -> Result<()> {
self.serialize_u8(0)?;
Ok(())
}
fn serialize_some<T>(self, value: &T) -> Result<()>
where
T: ?Sized + Serialize,
{
self.serialize_u8(1)?;
value.serialize(self)
}
fn serialize_unit(self) -> Result<()> {
Ok(())
}
fn serialize_unit_struct(self, _name: &'static str) -> Result<()> {
self.serialize_unit()
}
fn serialize_unit_variant(
self,
_name: &'static str,
_variant_index: u32,
variant: &'static str,
) -> Result<()> {
self.serialize_u32(_variant_index)
}
fn serialize_newtype_struct<T>(
self,
_name: &'static str,
value: &T,
) -> Result<()>
where
T: ?Sized + Serialize,
{
value.serialize(self)
}
fn serialize_newtype_variant<T>(
self,
_name: &'static str,
_variant_index: u32,
variant: &'static str,
value: &T,
) -> Result<()>
where
T: ?Sized + Serialize,
{
self.serialize_u32(_variant_index)?;
value.serialize(self)
}
/// We expect it to use with vairable-length arrays...
fn serialize_seq(self, _len: Option<usize>) -> Result<Self::SerializeSeq> {
self.output.put_unsigned(_len.unwrap_or(0))?;
Ok(self)
}
fn serialize_tuple(self, len: usize) -> Result<Self::SerializeTuple> {
Ok(self)
// self.serialize_seq(Some(len))
}
fn serialize_tuple_struct(
self,
_name: &'static str,
len: usize,
) -> Result<Self::SerializeTupleStruct> {
Ok(self)
}
fn serialize_tuple_variant(
self,
_name: &'static str,
_variant_index: u32,
variant: &'static str,
_len: usize,
) -> Result<Self::SerializeTupleVariant> {
_variant_index.serialize(&mut *self)?;
Ok(self)
}
fn serialize_map(self, _len: Option<usize>) -> Result<Self::SerializeMap> {
self.output.put_unsigned(_len.unwrap_or(0))?;
Ok(self)
}
fn serialize_struct(
self,
_name: &'static str,
len: usize,
) -> Result<Self::SerializeStruct> {
// self.serialize_map(Some(len))
Ok(self)
}
fn serialize_struct_variant(
self,
_name: &'static str,
_variant_index: u32,
variant: &'static str,
_len: usize,
) -> Result<Self::SerializeStructVariant> {
self.output.put_unsigned(_variant_index)?;
Ok(self)
}
}
impl<'a, S: BipackSink> ser::SerializeSeq for &'a mut Serializer<S> {
// Must match the `Ok` type of the serializer.
type Ok = ();
// Must match the `Error` type of the serializer.
type Error = BipackError;
// Serialize a single element of the sequence.
fn serialize_element<T>(&mut self, value: &T) -> Result<()>
where
T: ?Sized + Serialize,
{
value.serialize(&mut **self)
}
// Close the sequence.
fn end(self) -> Result<()> {
Ok(())
}
}
impl<'a, S: BipackSink> ser::SerializeTuple for &'a mut Serializer<S> {
type Ok = ();
type Error = BipackError;
fn serialize_element<T>(&mut self, value: &T) -> Result<()>
where
T: ?Sized + Serialize,
{
value.serialize(&mut **self)
}
fn end(self) -> Result<()> {
Ok(())
}
}
impl<'a, S: BipackSink> ser::SerializeTupleStruct for &'a mut Serializer<S> {
type Ok = ();
type Error = BipackError;
fn serialize_field<T>(&mut self, value: &T) -> Result<()>
where
T: ?Sized + Serialize,
{
value.serialize(&mut **self)
}
fn end(self) -> Result<()> {
Ok(())
}
}
impl<'a, S: BipackSink> ser::SerializeTupleVariant for &'a mut Serializer<S> {
type Ok = ();
type Error = BipackError;
fn serialize_field<T>(&mut self, value: &T) -> Result<()>
where
T: ?Sized + Serialize,
{
value.serialize(&mut **self)
}
fn end(self) -> Result<()> {
Ok(())
}
}
impl<'a, S: BipackSink> ser::SerializeMap for &'a mut Serializer<S> {
type Ok = ();
type Error = BipackError;
// The Serde data model allows map keys to be any serializable type. JSON
// only allows string keys so the implementation below will produce invalid
// JSON if the key serializes as something other than a string.
//
// A real JSON serializer would need to validate that map keys are strings.
// This can be done by using a different Serializer to serialize the key
// (instead of `&mut **self`) and having that other serializer only
// implement `serialize_str` and return an error on any other data type.
fn serialize_key<T>(&mut self, key: &T) -> Result<()>
where
T: ?Sized + Serialize,
{
key.serialize(&mut **self)
}
// It doesn't make a difference whether the colon is printed at the end of
// `serialize_key` or at the beginning of `serialize_value`. In this case
// the code is a bit simpler having it here.
fn serialize_value<T>(&mut self, value: &T) -> Result<()>
where
T: ?Sized + Serialize,
{
value.serialize(&mut **self)
}
fn end(self) -> Result<()> {
Ok(())
}
}
// Structs are like maps in which the keys are constrained to be compile-time
// constant strings.
impl<'a, S: BipackSink> ser::SerializeStruct for &'a mut Serializer<S> {
type Ok = ();
type Error = BipackError;
fn serialize_field<T>(&mut self, key: &'static str, value: &T) -> Result<()>
where
T: ?Sized + Serialize,
{
value.serialize(&mut **self)
}
fn end(self) -> Result<()> {
Ok(())
}
}
impl<'a, S: BipackSink> ser::SerializeStructVariant for &'a mut Serializer<S> {
type Ok = ();
type Error = BipackError;
fn serialize_field<T>(&mut self, key: &'static str, value: &T) -> Result<()>
where
T: ?Sized + Serialize,
{
value.serialize(&mut **self)
}
fn end(self) -> Result<()> {
Ok(())
}
}
#[cfg(test)]
mod test {
use std::collections::HashMap;
use std::string::FromUtf8Error;
use serde::Serialize;
use crate::bipack_source::{BipackError, BipackSource, SliceSource};
use crate::error;
use crate::ser::to_bytes;
use crate::tools::{to_dump, to_hex};
#[test]
fn test_struct() -> Result<(), BipackError> {
#[derive(Serialize)]
struct Test {
int: u32,
seq: Vec<&'static str>,
}
let test = Test {
int: 17,
seq: vec!["a", "b"],
};
let x = to_bytes(&test).unwrap();
println!("!:\n{}", to_dump(&x));
// let y = x.clone();
// let z = x.clone();
let mut src = SliceSource::from(&x);
assert_eq!(test.int, src.get_unsigned()? as u32);
assert_eq!(test.seq.len(), src.get_unsigned()? as usize);
assert_eq!(test.seq[0], src.get_str()?);
assert_eq!(test.seq[1], src.get_str()?);
Ok(())
}
#[test]
fn test_enum() -> Result<(), FromUtf8Error> {
#[derive(Serialize)]
enum E {
Unit,
Unit2,
Newtype(u32),
Tuple(u32, u32),
Struct { a: u32 },
}
let u = E::Unit;
println!("u:{}", to_dump(to_bytes(&u).unwrap().as_slice()));
assert_eq!("00", to_hex(to_bytes(&u).unwrap())?);
let u2 = E::Unit2;
println!("u:{}", to_dump(to_bytes(&u2).unwrap().as_slice()));
let nt = E::Newtype(17);
println!("u:{}", to_dump(to_bytes(&nt).unwrap().as_slice()));
assert_eq!("08 44", to_hex(to_bytes(&nt).unwrap())?);
let t = E::Tuple(7, 17);
println!("u:{}", to_dump(to_bytes(&t).unwrap().as_slice()));
assert_eq!("0c 1c 44", to_hex(to_bytes(&t).unwrap())?);
let t = E::Struct { a: 17 };
println!("u:{}", to_dump(to_bytes(&t).unwrap().as_slice()));
assert_eq!("10 44", to_hex(to_bytes(&t).unwrap())?);
// let expected = r#""Unit""#;
// assert_eq!(to_string(&u).unwrap(), expected);
//
// let n = E::Newtype(1);
// let expected = r#"{"Newtype":1}"#;
// assert_eq!(to_string(&n).unwrap(), expected);
//
// let t = E::Tuple(1, 2);
// let expected = r#"{"Tuple":[1,2]}"#;
// assert_eq!(to_string(&t).unwrap(), expected);
//
// let s = E::Struct { a: 1 };
// let expected = r#"{"Struct":{"a":1}}"#;
// assert_eq!(to_string(&s).unwrap(), expected);
Ok(())
}
#[test]
fn test_map() -> error::Result<()> {
let mut src = HashMap::new();
// src.insert("foo", 1);
src.insert("foo", 42);
src.insert("bar", 1);
src.insert("baz", 17);
let packed = to_bytes(&src)?;
println!("{}", to_dump(&packed));
Ok(())
}
}

114
src/tools.rs
View File

@ -1,114 +0,0 @@
// Copyright 2023 by Sergey S. Chernov.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// use string_builder::Builder;
use std::string::FromUtf8Error;
/// Absolutely minimalistic string builder (growing string implemented minimal and
/// more or less effective). Just to avoid dependencies for better .wasm usage.
pub struct StringBuilder(Vec<u8>);
impl StringBuilder {
/// Append something string-like (you can use &str and String for example) to the buffer.
fn append<T: AsRef<str>>(self: &mut Self, text: T) {
for b in text.as_ref().bytes() { self.0.push(b) }
}
/// Append char as far as it is a valid char in rust limited sense:
fn append_char(self: &mut Self, c: char) {
self.append(String::from(c))
}
/// Finalize the builder and return the result string.
fn string(self: &mut Self) -> Result<String, FromUtf8Error> {
String::from_utf8(self.0.clone())
}
fn new() -> StringBuilder { StringBuilder(Vec::new()) }
fn len(self: &Self) -> usize { self.0.len() }
}
/// Convert binary data into text dump, human readable, like:
/// ```text
/// 0000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
/// 0010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................|
/// 0020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f | !"#$%&'()*+,-./|
/// 0030 30 31 |01 |
///```
pub fn to_dump(data: &[u8]) -> String {
let mut offset = 0usize;
let mut counter = 0;
let mut result = StringBuilder::new();
fn ascii_dump(result: &mut StringBuilder, counter: usize, data: &[u8], offset: usize) {
for i in counter..16 { result.append(" "); }
result.append("|");
for i in 0..counter {
let b = data[offset - counter + i];
if b >= 32 && b <= 127 {
result.append_char(b as char)
} else {
result.append_char('.');
}
}
for i in counter..16 { result.append_char(' '); }
result.append("|\n");
}
while offset < data.len() {
if counter == 0 {
result.append(format!("{:04X} ", offset))
}
counter += 1;
result.append(format!("{:02x} ", data[offset]));
offset += 1;
if counter == 16 {
ascii_dump(&mut result, counter, data, offset);
counter = 0;
}
}
if counter != 0 { ascii_dump(&mut result, counter, data, offset); }
result.string().unwrap()
}
pub fn to_hex<T: AsRef<[u8]>>(source: T) -> Result<String,FromUtf8Error> {
let mut result = StringBuilder::new();
for b in source.as_ref() {
if result.len() != 0 { result.append( " ") }
result.append(format!("{:02x}", b))
}
result.string()
}
pub trait ToDump {
/// Same as [to_dump] function, but specified as a trait for
/// different dump-able formats.
fn to_dump(&self) -> String;
}
impl ToDump for &[u8] {
fn to_dump(&self) -> String{
to_dump(self)
}
}
impl ToDump for Vec<u8> {
fn to_dump(&self) -> String {
to_dump(&self)
}
}