SergeychWorks/kiloparsec
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 6 Wiki Activity

0.2.5-snapshot native support for tcp client/server

Browse Source
This commit is contained in:
Sergey Chernov 2024-08-03 15:36:04 +02:00
parent 26d1f3522f
commit ffcdcf7350
16 changed files with 524 additions and 580 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.idea/misc.xml generated
View File

@ -1,4 +1,3 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4"> <project version="4">
<component name="ExternalStorageConfigurationManager" enabled="true" /> <component name="ExternalStorageConfigurationManager" enabled="true" />
<component name="FrameworkDetectionExcludesConfiguration"> <component name="FrameworkDetectionExcludesConfiguration">

56
README.md
View File

@ -1,31 +1,36 @@
# Kiloparsec # Kiloparsec
The new generation of __PARanoid SECurity__ protocol, advanced, faster, more secure. It also allows connecting any "block device" transport to the same local interface. Out if the box it The new generation of __PARanoid SECurity__ protocol, advanced, faster, more secure. It also allows connecting any "
block device" transport to the same local interface. Out if the box it
provides the following transports: provides the following transports:
| name | JVM | JS | native | | name | JVM | JS | native |
|----------------|-----|----|--------| |----------------|-----|----|-------------------|
| TCP/IP server | * | | | | TCP/IP server | ✓ | | β @0.2.5-SNAPSHOT |
| TCP/IP client | * | | | | TCP/IP client | ✓ | | β @0.2.5-SNAPSHOT |
| Websock server | * | | | | Websock server | ✓ | | |
| Websock client | * | * | * | | Websock client | ✓ | ✓ | ✓ |
At the moment we're working on supporting TCP/IP on most native targets. This feature is planned to rach public beta in August and production in early september 2024.
At the moment we're working on supporting TCP/IP on most native targets. This feature is planned to rach public beta in
August and production in early september 2024.
## TCP/IP transport ## TCP/IP transport
It is the fastest. JVM implementation uses nio2 async sockets and optimizes TCP socket to play It is the fastest. JVM implementation uses nio2 async sockets and optimizes TCP socket to play
well with blocks (smart NO_DELAY mode). It is multiplatform, nut lacks of async TCP/IP support well with blocks (smart NO_DELAY mode). It is multiplatform, nut lacks of async TCP/IP support
on natvic targetm this is where I need help having little time. I'd prefer to use something asyn like UV on native targets. on natvic targetm this is where I need help having little time. I'd prefer to use something asyn like UV on native
targets.
I know no existing way to implement it in KotlinJS for the modern browsers. I know no existing way to implement it in KotlinJS for the modern browsers.
## Websock server ## Websock server
While it is much slower than pure TCP, it is still faster than any http-based transport. It uses binary frames based on the Ktor server framework to easily integrate with web services. We recommend using it instead of a classic HTTP API as it beats it in terms of speed and server load even with HTTP/2. While it is much slower than pure TCP, it is still faster than any http-based transport. It uses binary frames based on
the Ktor server framework to easily integrate with web services. We recommend using it instead of a classic HTTP API as
it beats it in terms of speed and server load even with HTTP/2.
We recommend to create the `KiloInterface<S>` instance and connect it to the websock and tcp servers in real applications to get easy access from anywhere. We recommend to create the `KiloInterface<S>` instance and connect it to the websock and tcp servers in real
applications to get easy access from anywhere.
# Usage # Usage
@ -64,16 +69,16 @@ and functions available, like:
// Api.kt // Api.kt
@Serializable @Serializable
class FooArgs(val text: String,val number: Int = 42) class FooArgs(val text: String, val number: Int = 42)
// Server-side interface // Server-side interface
val cmdSetFoo by command<FooArgs,Unit>() val cmdSetFoo by command<FooArgs, Unit>()
val cmdGetFoo by command<Unit,FooArgs>() val cmdGetFoo by command<Unit, FooArgs>()
val cmdPing by command<String,String>() val cmdPing by command<String, String>()
val cmdCheckConnected by command<Unit,Boolean>() val cmdCheckConnected by command<Unit, Boolean>()
// client-side interface (called from the server) // client-side interface (called from the server)
val cmdPushClient by command<String,Unit>() val cmdPushClient by command<String, Unit>()
``` ```
## Call it from the client: ## Call it from the client:
@ -93,7 +98,7 @@ val client = websocketClient<Unit>("wss://your.host.com/kp") {
// If we want to collect connected state changes (this is optional) // If we want to collect connected state changes (this is optional)
launch { launch {
client.connectedStateFlow.collect { client.connectedStateFlow.collect {
if( it ) if (it)
println("I am connected") println("I am connected")
else else
println("trying to connect...") println("trying to connect...")
@ -113,7 +118,7 @@ the protocol. With KILOPARSEC it is rather basic operation:\
~~~kotlin ~~~kotlin
// Our session just keeps Foo for cmd{Get|Set}Foo: // Our session just keeps Foo for cmd{Get|Set}Foo:
data class Session(var fooState: FooArgs?=null) data class Session(var fooState: FooArgs? = null)
// Let's now provide interface we export, it will be used on each connection automatically: // Let's now provide interface we export, it will be used on each connection automatically:
@ -136,6 +141,7 @@ val ns: NettyApplicationEngine = embeddedServer(Netty, port = 8080, host = "0.0.
~~~ ~~~
# Details # Details
It is not compatible with parsec family and no more based on an Universa crypto library. To better fit It is not compatible with parsec family and no more based on an Universa crypto library. To better fit
@ -144,9 +150,11 @@ and every connection (while parsec caches session keys to avoid time-consuming k
keys cryptography for session is shifted to use ed25519 curves which are supposed to provide agreeable strength with keys cryptography for session is shifted to use ed25519 curves which are supposed to provide agreeable strength with
enough speed to protect every connection with a unique new keys. Also, we completely get rid of SHA2. enough speed to protect every connection with a unique new keys. Also, we completely get rid of SHA2.
Kiloparsec also uses a denser binary format [bipack](https://gitea.sergeych.net/SergeychWorks/mp_bintools), no more key-values, Kiloparsec also uses a denser binary format [bipack](https://gitea.sergeych.net/SergeychWorks/mp_bintools), no more
key-values,
which reveals much less on the inner data structure, providing advanced which reveals much less on the inner data structure, providing advanced
typed RPC interfaces with kotlinx.serialization. There is also Rust implementation [bipack_ru](https://gitea.sergeych.net/DiWAN/bipack_ru). typed RPC interfaces with kotlinx.serialization. There is also Rust
implementation [bipack_ru](https://gitea.sergeych.net/DiWAN/bipack_ru).
The architecture allows connecting same functional interfaces to several various type channels at once. The architecture allows connecting same functional interfaces to several various type channels at once.
Also, the difference from parsecs is that there are no more unencrypted layer commands available to users. Also, the difference from parsecs is that there are no more unencrypted layer commands available to users.
@ -163,7 +171,9 @@ Integrated tools to prevent MITM attacks include also non-transferred independen
independently on the ends and is never transferred with the network. Comparing it somehow (visually, with QR code, etc) independently on the ends and is never transferred with the network. Comparing it somehow (visually, with QR code, etc)
could add a very robust guarantee of the connection safety and ingenuity. could add a very robust guarantee of the connection safety and ingenuity.
Kiloparsec has built-in completely asynchronous (coroutine based top-down) transport layer based on TCP (JVM only as for now) and the same async Websocket-based transport based on KTOR. Websocket client is multiplatform, though the server is JVM only insofar. Kiloparsec has built-in completely asynchronous (coroutine based top-down) transport layer based on TCP (JVM only as for
now) and the same async Websocket-based transport based on KTOR. Websocket client is multiplatform, though the server is
JVM only insofar.
# Licensing # Licensing

46
build.gradle.kts
View File

@ -1,4 +1,3 @@
plugins { plugins {
kotlin("multiplatform") version "2.0.0" kotlin("multiplatform") version "2.0.0"
id("org.jetbrains.kotlin.plugin.serialization") version "2.0.0" id("org.jetbrains.kotlin.plugin.serialization") version "2.0.0"
@ -59,6 +58,9 @@ kotlin {
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.8.1") implementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.8.1")
} }
} }
val ktorSocketTest by creating {
dependsOn(commonTest)
}
val jvmMain by getting { val jvmMain by getting {
dependencies { dependencies {
implementation("io.ktor:ktor-server-core:$ktor_version") implementation("io.ktor:ktor-server-core:$ktor_version")
@ -69,16 +71,52 @@ kotlin {
} }
dependsOn(ktorSocketMain) dependsOn(ktorSocketMain)
} }
val jvmTest by getting val jvmTest by getting {
dependsOn(ktorSocketTest)
}
val jsMain by getting { val jsMain by getting {
dependencies { dependencies {
implementation("io.ktor:ktor-client-js:$ktor_version") implementation("io.ktor:ktor-client-js:$ktor_version")
} }
} }
val jsTest by getting val jsTest by getting
val macosArm64Main by getting {
dependsOn(ktorSocketMain)
}
val macosArm64Test by getting {
dependsOn(ktorSocketTest)
}
val macosX64Main by getting {
dependsOn(ktorSocketMain)
}
val iosX64Main by getting {
dependsOn(ktorSocketMain)
}
val iosX64Test by getting {
dependsOn(ktorSocketTest)
}
val iosArm64Main by getting {
dependsOn(ktorSocketMain)
}
val iosArm64Test by getting {
dependsOn(ktorSocketTest)
}
val linuxArm64Main by getting {
dependsOn(ktorSocketMain)
}
val linuxArm64Test by getting {
dependsOn(ktorSocketTest)
}
val linuxX64Main by getting {
dependsOn(ktorSocketMain)
}
val linuxX64Test by getting {
dependsOn(ktorSocketTest)
}
for (pm in listOf(linuxMain, macosMain, iosMain, mingwMain)) // for (pm: NamedDomainObjectProvider<KotlinSourceSet> in listOf(macosMain,linuxMain, iosMain, mingwMain))
pm { dependsOn(ktorSocketMain) } // pm.get().dependsOn(ktorSocketMain)
} }

1
gradle.properties
View File

@ -1 +1,2 @@
kotlin.code.style=official kotlin.code.style=official
kotlin.mpp.applyDefaultHierarchyTemplate=false

2
src/commonMain/kotlin/net/sergeych/kiloparsec/KiloServer.kt
View File

@ -43,6 +43,8 @@ class KiloServer<S>(
} }
fun close() { fun close() {
println("PRREEEC")
job.cancel() job.cancel()
println("POOOSTC")
} }
} }

98
src/commonMain/kotlin/net/sergeych/kiloparsec/adapter/NetworkProvider.kt
View File

@ -1,56 +1,56 @@
package net.sergeych.kiloparsec.adapter package net.sergeych.kiloparsec.adapter
import kotlinx.coroutines.channels.ReceiveChannel
import kotlinx.coroutines.flow.Flow
/** /**
* Multiplatform implementation of an internet address. * Multiplatform internet address.
* Notice to implementors. It must provide correct and effective [equals] and [hashCode].
*/ */
interface NetworkAddress { data class NetworkAddress(
val host: String val host: String,
val port: Int val port: Int
} ) {
override fun toString(): String {
/** return "$host:$port"
* Multiplatform datagram abstraction
*/
interface Datagram {
/**
* Received message
*/
val message: UByteArray
/**
* Address from where the message was sent
*/
val address: NetworkAddress
}
@OptIn(ExperimentalStdlibApi::class)
interface DatagramConnector: AutoCloseable {
val incoming: ReceiveChannel<Datagram>
suspend fun send(message: UByteArray, networkAddress: NetworkAddress)
@Suppress("unused")
suspend fun send(message: UByteArray, datagramAddress: String) {
send(message, datagramAddress.toNetworkAddress())
} }
suspend fun send(message: UByteArray,host: String,port: Int) =
send(message, NetworkAddress(host,port))
override fun close()
} }
//
expect fun NetworkAddress(host: String,port: Int): NetworkAddress ///**
// * Multiplatform datagram abstraction
fun String.toNetworkAddress() : NetworkAddress { // */
val (host, port) = this.split(":").map { it.trim()} //interface Datagram {
return NetworkAddress(host, port.toInt()) // /**
} // * Received message
// */
expect fun acceptTcpDevice(port: Int): Flow<InetTransportDevice> // val message: UByteArray
//
expect suspend fun connectTcpDevice(address: NetworkAddress): InetTransportDevice // /**
// * Address from where the message was sent
suspend fun connectTcpDevice(address: String) = connectTcpDevice(address.toNetworkAddress()) // */
// val address: NetworkAddress
//}
//
//@OptIn(ExperimentalStdlibApi::class)
//interface DatagramConnector: AutoCloseable {
//
// val incoming: ReceiveChannel<Datagram>
// suspend fun send(message: UByteArray, networkAddress: NetworkAddress)
// @Suppress("unused")
// suspend fun send(message: UByteArray, datagramAddress: String) {
// send(message, datagramAddress.toNetworkAddress())
// }
//
// suspend fun send(message: UByteArray,host: String,port: Int) =
// send(message, NetworkAddress(host,port))
// override fun close()
//}
//
//expect fun NetworkAddress(host: String,port: Int): NetworkAddress
//
//fun String.toNetworkAddress() : NetworkAddress {
// val (host, port) = this.split(":").map { it.trim()}
// return NetworkAddress(host, port.toInt())
//}
//
//expect fun acceptTcpDevice(port: Int): Flow<InetTransportDevice>
//
//expect suspend fun connectTcpDevice(address: NetworkAddress): InetTransportDevice
//
//suspend fun connectTcpDevice(address: String) = connectTcpDevice(address.toNetworkAddress())

2
src/jsMain/kotlin/net/sergeych/kiloparsec/adapter/DatagramProvider.js.kt
View File

@ -1,2 +0,0 @@
package net.sergeych.kiloparsec.adapter

15
src/jsMain/kotlin/net/sergeych/kiloparsec/adapter/NetworkProvider.js.kt
View File

@ -1,15 +0,0 @@
package net.sergeych.kiloparsec.adapter
import kotlinx.coroutines.flow.Flow
actual fun NetworkAddress(host: String, port: Int): NetworkAddress {
TODO("Not yet implemented")
}
actual fun acceptTcpDevice(port: Int): Flow<InetTransportDevice> {
TODO("Not yet implemented")
}
actual suspend fun connectTcpDevice(address: NetworkAddress): InetTransportDevice {
TODO("Not yet implemented")
}

236
src/jvmMain/kotlin/net/sergeych/kiloparsec/adapter/UdpServer.kt
View File

@ -1,118 +1,118 @@
package net.sergeych.kiloparsec.adapter //package net.sergeych.kiloparsec.adapter
//
import kotlinx.coroutines.* //import kotlinx.coroutines.*
import kotlinx.coroutines.channels.BufferOverflow //import kotlinx.coroutines.channels.BufferOverflow
import kotlinx.coroutines.channels.Channel //import kotlinx.coroutines.channels.Channel
import net.sergeych.mp_logger.LogTag //import net.sergeych.mp_logger.LogTag
import net.sergeych.mp_logger.exception //import net.sergeych.mp_logger.exception
import net.sergeych.mp_logger.info //import net.sergeych.mp_logger.info
import net.sergeych.mp_logger.warning //import net.sergeych.mp_logger.warning
import java.net.* //import java.net.*
import java.util.concurrent.atomic.AtomicInteger //import java.util.concurrent.atomic.AtomicInteger
//
private val counter = AtomicInteger(0) //private val counter = AtomicInteger(0)
//
class JvmNetworkAddress(val inetAddress: InetAddress, override val port: Int) : NetworkAddress { //class JvmNetworkAddress(val inetAddress: InetAddress, override val port: Int) : NetworkAddress {
override val host: String by lazy { inetAddress.canonicalHostName } // override val host: String by lazy { inetAddress.canonicalHostName }
override fun equals(other: Any?): Boolean { // override fun equals(other: Any?): Boolean {
if (this === other) return true // if (this === other) return true
if (other !is JvmNetworkAddress) return false // if (other !is JvmNetworkAddress) return false
//
if (inetAddress != other.inetAddress) return false // if (inetAddress != other.inetAddress) return false
if (port != other.port) return false // if (port != other.port) return false
//
return true // return true
} // }
//
val socketAddress: SocketAddress by lazy { InetSocketAddress(inetAddress,port) } // val socketAddress: SocketAddress by lazy { InetSocketAddress(inetAddress,port) }
//
override fun hashCode(): Int { // override fun hashCode(): Int {
var result = inetAddress.hashCode() // var result = inetAddress.hashCode()
result = 31 * result + port // result = 31 * result + port
return result // return result
} // }
//
override fun toString(): String = "$host:$port" // override fun toString(): String = "$host:$port"
} //}
//
class UdpDatagram(override val message: UByteArray, val inetAddress: InetAddress, val port: Int) : Datagram { //class UdpDatagram(override val message: UByteArray, val inetAddress: InetAddress, val port: Int) : Datagram {
//
override val address: NetworkAddress by lazy { // override val address: NetworkAddress by lazy {
JvmNetworkAddress(inetAddress, port) // JvmNetworkAddress(inetAddress, port)
} // }
//
} //}
//
//
@OptIn(DelicateCoroutinesApi::class) //@OptIn(DelicateCoroutinesApi::class)
class UdpServer(val port: Int) : //class UdpServer(val port: Int) :
DatagramConnector, LogTag("UDPS:${counter.incrementAndGet()}") { // DatagramConnector, LogTag("UDPS:${counter.incrementAndGet()}") {
private var isClosed = false // private var isClosed = false
//
//
private val deferredSocket = CompletableDeferred<DatagramSocket>() // private val deferredSocket = CompletableDeferred<DatagramSocket>()
private var job: Job? = null // private var job: Job? = null
//
private suspend fun start() = try { // private suspend fun start() = try {
coroutineScope { // coroutineScope {
val socket = DatagramSocket(port) // val socket = DatagramSocket(port)
val buffer = ByteArray(16384) // val buffer = ByteArray(16384)
val packet = DatagramPacket(buffer, buffer.size) // val packet = DatagramPacket(buffer, buffer.size)
deferredSocket.complete(socket) // deferredSocket.complete(socket)
while (isActive && !isClosed) { // while (isActive && !isClosed) {
try { // try {
socket.receive(packet) // socket.receive(packet)
val data = packet.data.sliceArray(0..<packet.length) // val data = packet.data.sliceArray(0..<packet.length)
val datagram = UdpDatagram(data.toUByteArray(), packet.address, packet.port) // val datagram = UdpDatagram(data.toUByteArray(), packet.address, packet.port)
if (!channel.trySend(datagram).isSuccess) { // if (!channel.trySend(datagram).isSuccess) {
warning { "packet lost!" } // warning { "packet lost!" }
// and we cause overflow that overwrites the oldest // // and we cause overflow that overwrites the oldest
channel.send(datagram) // channel.send(datagram)
} // }
} catch (e: Exception) { // } catch (e: Exception) {
if (!isClosed) // if (!isClosed)
e.printStackTrace() // e.printStackTrace()
throw e // throw e
} // }
} // }
info { "closing socket and reception loop" } // info { "closing socket and reception loop" }
//
} // }
} catch (_: CancellationException) { // } catch (_: CancellationException) {
info { "server is closed" } // info { "server is closed" }
} catch (t: Throwable) { // } catch (t: Throwable) {
exception { "unexpected end of server" to t } // exception { "unexpected end of server" to t }
} // }
//
init { // init {
job = GlobalScope.launch(Dispatchers.IO) { // job = GlobalScope.launch(Dispatchers.IO) {
start() // start()
} // }
} // }
//
override fun close() { // override fun close() {
if (!isClosed) { // if (!isClosed) {
if (deferredSocket.isCompleted) { // if (deferredSocket.isCompleted) {
runCatching { // runCatching {
deferredSocket.getCompleted().close() // deferredSocket.getCompleted().close()
} // }
} // }
isClosed = true // isClosed = true
job?.cancel(); job = null // job?.cancel(); job = null
} // }
} // }
//
private val channel = Channel<Datagram>(2048, BufferOverflow.DROP_OLDEST) // private val channel = Channel<Datagram>(2048, BufferOverflow.DROP_OLDEST)
override val incoming = channel // override val incoming = channel
//
override suspend fun send(message: UByteArray, networkAddress: NetworkAddress) { // override suspend fun send(message: UByteArray, networkAddress: NetworkAddress) {
networkAddress as JvmNetworkAddress // networkAddress as JvmNetworkAddress
withContext(Dispatchers.IO) { // withContext(Dispatchers.IO) {
val packet = DatagramPacket( // val packet = DatagramPacket(
message.toByteArray(), message.size, // message.toByteArray(), message.size,
networkAddress.inetAddress, networkAddress.port // networkAddress.inetAddress, networkAddress.port
) // )
deferredSocket.await().send(packet) // deferredSocket.await().send(packet)
} // }
} // }
} //}

310
src/jvmMain/kotlin/net/sergeych/kiloparsec/adapter/asyncSocketToDevice.kt
View File

@ -1,155 +1,155 @@
package net.sergeych.kiloparsec.adapter //package net.sergeych.kiloparsec.adapter
//
import kotlinx.coroutines.* //import kotlinx.coroutines.*
import kotlinx.coroutines.channels.Channel //import kotlinx.coroutines.channels.Channel
import kotlinx.coroutines.channels.ClosedReceiveChannelException //import kotlinx.coroutines.channels.ClosedReceiveChannelException
import kotlinx.coroutines.flow.MutableStateFlow //import kotlinx.coroutines.flow.MutableStateFlow
import net.sergeych.crypto2.Contrail //import net.sergeych.crypto2.Contrail
import net.sergeych.crypto2.encodeVarUnsigned //import net.sergeych.crypto2.encodeVarUnsigned
import net.sergeych.crypto2.readVarUnsigned //import net.sergeych.crypto2.readVarUnsigned
import net.sergeych.kiloparsec.RemoteInterface //import net.sergeych.kiloparsec.RemoteInterface
import net.sergeych.kiloparsec.Transport //import net.sergeych.kiloparsec.Transport
import net.sergeych.mp_logger.LogTag //import net.sergeych.mp_logger.LogTag
import net.sergeych.mp_logger.warning //import net.sergeych.mp_logger.warning
import net.sergeych.mp_tools.globalLaunch //import net.sergeych.mp_tools.globalLaunch
import net.sergeych.tools.waitFor //import net.sergeych.tools.waitFor
import java.net.InetSocketAddress //import java.net.InetSocketAddress
import java.net.StandardSocketOptions.TCP_NODELAY //import java.net.StandardSocketOptions.TCP_NODELAY
import java.nio.ByteBuffer //import java.nio.ByteBuffer
import java.nio.channels.AsynchronousSocketChannel //import java.nio.channels.AsynchronousSocketChannel
import kotlin.coroutines.cancellation.CancellationException //import kotlin.coroutines.cancellation.CancellationException
import kotlin.coroutines.suspendCoroutine //import kotlin.coroutines.suspendCoroutine
//
private val log = LogTag("ASTD") //private val log = LogTag("ASTD")
//
/** ///**
* Prepend block with its size, varint-encoded // * Prepend block with its size, varint-encoded
*/ // */
private fun encode(block: UByteArray): ByteArray { //private fun encode(block: UByteArray): ByteArray {
val c = Contrail.create(block) // val c = Contrail.create(block)
return (encodeVarUnsigned(c.size.toUInt()) + c).toByteArray() // return (encodeVarUnsigned(c.size.toUInt()) + c).toByteArray()
} //}
//
/** ///**
* Convert asynchronous socket to a [Transport.Device] using non-blocking nio, // * Convert asynchronous socket to a [Transport.Device] using non-blocking nio,
* in a coroutine-effective manner. Note that it runs coroutines to read/write // * in a coroutine-effective manner. Note that it runs coroutines to read/write
* to the socket in a global scope.These are closed when transport is closed // * to the socket in a global scope.These are closed when transport is closed
* or the socket is closed, for example, by network failure. // * or the socket is closed, for example, by network failure.
*/ // */
suspend fun asyncSocketToDevice(socket: AsynchronousSocketChannel): InetTransportDevice { //suspend fun asyncSocketToDevice(socket: AsynchronousSocketChannel): InetTransportDevice {
val deferredDevice = CompletableDeferred<InetTransportDevice>() // val deferredDevice = CompletableDeferred<InetTransportDevice>()
globalLaunch { // globalLaunch {
coroutineScope { // coroutineScope {
val sendQueueEmpty = MutableStateFlow(true) // val sendQueueEmpty = MutableStateFlow(true)
val receiving = MutableStateFlow(false) // val receiving = MutableStateFlow(false)
// We're in block mode, every block we send worth immediate sending, we do not // // We're in block mode, every block we send worth immediate sending, we do not
// send partial blocks, so: // // send partial blocks, so:
socket.setOption(TCP_NODELAY, true) // socket.setOption(TCP_NODELAY, true)
//
// socket input is to be parsed for blocks, so we receive bytes // // socket input is to be parsed for blocks, so we receive bytes
// and decode them to blocks // // and decode them to blocks
val input = Channel<UByte>(1024) // val input = Channel<UByte>(1024)
val inputBlocks = Channel<UByteArray>() // val inputBlocks = Channel<UByteArray>()
// output is blocks, so we sent transformed, framed blocks: // // output is blocks, so we sent transformed, framed blocks:
val outputBlocks = Channel<UByteArray>() // val outputBlocks = Channel<UByteArray>()
//
fun stop() { // fun stop() {
kotlin.runCatching { inputBlocks.close(RemoteInterface.ClosedException()) } // kotlin.runCatching { inputBlocks.close(RemoteInterface.ClosedException()) }
kotlin.runCatching { outputBlocks.close() } // kotlin.runCatching { outputBlocks.close() }
socket.close() // socket.close()
cancel() // cancel()
} // }
//
//
// copy incoming data from the socket to input channel: // // copy incoming data from the socket to input channel:
launch { // launch {
val data = ByteArray(1024) // val data = ByteArray(1024)
val inb = ByteBuffer.wrap(data) // val inb = ByteBuffer.wrap(data)
kotlin.runCatching { // kotlin.runCatching {
while (isActive) { // while (isActive) {
inb.position(0) // inb.position(0)
val size: Int = suspendCoroutine { continuation -> // val size: Int = suspendCoroutine { continuation ->
socket.read(inb, continuation, IntCompletionHandler) // socket.read(inb, continuation, IntCompletionHandler)
} // }
if (size < 0) stop() // if (size < 0) stop()
else { // else {
// println("recvd:\n${data.sliceArray(0..<size).toDump()}\n------------------") //// println("recvd:\n${data.sliceArray(0..<size).toDump()}\n------------------")
for (i in 0..<size) input.send(data[i].toUByte()) // for (i in 0..<size) input.send(data[i].toUByte())
} // }
} // }
} // }
} // }
//
// copy from output to socket: // // copy from output to socket:
launch { // launch {
try { // try {
while (isActive) { // while (isActive) {
// wait for the first block to send // // wait for the first block to send
sendQueueEmpty.value = outputBlocks.isEmpty // sendQueueEmpty.value = outputBlocks.isEmpty
var data = encode(outputBlocks.receive()) // var data = encode(outputBlocks.receive())
//
// now we're sending, so queue state is sending: // // now we're sending, so queue state is sending:
sendQueueEmpty.value = false // sendQueueEmpty.value = false
//
// if there are more, take them all (NO_DELAY optimization) // // if there are more, take them all (NO_DELAY optimization)
while (!outputBlocks.isEmpty) // while (!outputBlocks.isEmpty)
data += encode(outputBlocks.receive()) // data += encode(outputBlocks.receive())
//
// now send it all together: // // now send it all together:
val outBuff = ByteBuffer.wrap(data) // val outBuff = ByteBuffer.wrap(data)
val cnt = suspendCoroutine { continuation -> // val cnt = suspendCoroutine { continuation ->
socket.write(outBuff, continuation, IntCompletionHandler) // socket.write(outBuff, continuation, IntCompletionHandler)
} // }
// be sure it was all sent // // be sure it was all sent
if (outBuff.position() != data.size || cnt != data.size) { // if (outBuff.position() != data.size || cnt != data.size) {
throw RuntimeException("unexpected partial write") // throw RuntimeException("unexpected partial write")
} // }
} // }
// in the case of just breaking out of the loop: // // in the case of just breaking out of the loop:
sendQueueEmpty.value = true // sendQueueEmpty.value = true
} catch (_: ClosedReceiveChannelException) { // } catch (_: ClosedReceiveChannelException) {
stop() // stop()
} // }
} // }
// transport device copes with blocks: // // transport device copes with blocks:
// decode blocks from a byte channel read from the socket: // // decode blocks from a byte channel read from the socket:
launch { // launch {
try { // try {
while (isActive) { // while (isActive) {
receiving.value = !input.isEmpty // receiving.value = !input.isEmpty
val size = readVarUnsigned(input) // val size = readVarUnsigned(input)
receiving.value = true // receiving.value = true
if (size == 0u) log.warning { "zero size block is ignored!" } // if (size == 0u) log.warning { "zero size block is ignored!" }
else { // else {
val block = UByteArray(size.toInt()) // val block = UByteArray(size.toInt())
for (i in 0..<size.toInt()) { // for (i in 0..<size.toInt()) {
block[i] = input.receive() // block[i] = input.receive()
} // }
Contrail.unpack(block)?.let { inputBlocks.send(it) } // Contrail.unpack(block)?.let { inputBlocks.send(it) }
?: log.warning { "skipping bad block ${block.size} bytes" } // ?: log.warning { "skipping bad block ${block.size} bytes" }
} // }
} // }
} catch (_: CancellationException) { // } catch (_: CancellationException) {
} catch (_: ClosedReceiveChannelException) { // } catch (_: ClosedReceiveChannelException) {
stop() // stop()
} // }
receiving.value = false // receiving.value = false
} // }
//
val addr = socket.remoteAddress as InetSocketAddress // val addr = socket.remoteAddress as InetSocketAddress
deferredDevice.complete( // deferredDevice.complete(
InetTransportDevice(inputBlocks, outputBlocks, JvmNetworkAddress(addr.address, addr.port), { // InetTransportDevice(inputBlocks, outputBlocks, JvmNetworkAddress(addr.address, addr.port), {
yield() // yield()
// wait until all received data are parsed, but not too long // // wait until all received data are parsed, but not too long
withTimeoutOrNull(500) { // withTimeoutOrNull(500) {
receiving.waitFor { !it } // receiving.waitFor { !it }
} // }
// then stop it // // then stop it
stop() // stop()
}) // })
) // )
} // }
globalLaunch { socket.close() } // globalLaunch { socket.close() }
} // }
return deferredDevice.await() // return deferredDevice.await()
} //}

63
src/jvmTest/kotlin/net/sergeych/kiloparsec/ClientTest.kt
View File

@ -3,22 +3,20 @@ package net.sergeych.kiloparsec
import assertThrows import assertThrows
import io.ktor.server.engine.* import io.ktor.server.engine.*
import io.ktor.server.netty.* import io.ktor.server.netty.*
import kotlinx.coroutines.delay
import kotlinx.coroutines.launch import kotlinx.coroutines.launch
import kotlinx.coroutines.test.runTest import kotlinx.coroutines.test.runTest
import net.sergeych.crypto2.initCrypto import net.sergeych.crypto2.initCrypto
import net.sergeych.kiloparsec.adapter.acceptTcpDevice
import net.sergeych.kiloparsec.adapter.connectTcpDevice
import net.sergeych.kiloparsec.adapter.setupWebsocketServer import net.sergeych.kiloparsec.adapter.setupWebsocketServer
import net.sergeych.kiloparsec.adapter.websocketClient import net.sergeych.kiloparsec.adapter.websocketClient
import net.sergeych.mp_logger.Log import net.sergeych.mp_logger.Log
import java.net.InetAddress import java.net.InetAddress
import kotlin.test.* import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
class ClientTest { class ClientTest {
class TestException : Exception("test1")
@Test @Test
fun testAddresses() { fun testAddresses() {
println(InetAddress.getLocalHost()) println(InetAddress.getLocalHost())
@ -26,60 +24,7 @@ class ClientTest {
println(InetAddress.getByName("127.0.0.1")) println(InetAddress.getByName("127.0.0.1"))
println(InetAddress.getByName("mail.ru")) println(InetAddress.getByName("mail.ru"))
} }
@Test
fun testClient() = runTest {
initCrypto()
Log.connectConsole(Log.Level.DEBUG)
data class Session(
var data: String
)
val cmdSave by command<String,Unit>()
val cmdLoad by command<Unit,String>()
val cmdDrop by command<Unit,Unit>()
val cmdException by command<Unit,Unit>()
val cli = KiloInterface<Session>().apply {
registerError { TestException() }
onConnected { session.data = "start" }
on(cmdSave) { session.data = it }
on(cmdLoad) {
session.data
}
on(cmdException) {
throw TestException()
}
on(cmdDrop) {
throw LocalInterface.BreakConnectionException()
}
}
val server = KiloServer(cli, acceptTcpDevice(27101)) {
Session("unknown")
}
val client = KiloClient<Unit>() {
addErrors(cli)
connect { connectTcpDevice("localhost:27101") }
}
delay(500)
println(client.call(cmdLoad))
assertEquals("start", client.call(cmdLoad))
client.call(cmdSave, "foobar")
assertEquals("foobar", client.call(cmdLoad))
//
val res = kotlin.runCatching { client.call(cmdException) }
println(res.exceptionOrNull())
assertIs<TestException>(res.exceptionOrNull())
assertEquals("foobar", client.call(cmdLoad))
assertThrows<RemoteInterface.ClosedException> { client.call(cmdDrop) }
// reconnect?
assertEquals("start", client.call(cmdLoad))
server.close()
}
@Test @Test
fun webSocketTest() = runTest { fun webSocketTest() = runTest {

100
src/jvmTest/kotlin/net/sergeych/kiloparsec/adapters/NetworkTest.kt
View File

@ -1,100 +0,0 @@
package net.sergeych.kiloparsec.adapters
import kotlinx.coroutines.*
import kotlinx.coroutines.test.runTest
import net.sergeych.crypto2.initCrypto
import net.sergeych.kiloparsec.adapter.UdpServer
import net.sergeych.kiloparsec.adapter.acceptTcpDevice
import net.sergeych.kiloparsec.adapter.connectTcpDevice
import net.sergeych.kiloparsec.adapter.toNetworkAddress
import net.sergeych.kiloparsec.decodeFromUByteArray
import net.sergeych.kiloparsec.encodeToUByteArray
import net.sergeych.mp_logger.Log
import net.sergeych.mp_logger.LogTag
import net.sergeych.synctools.ProtectedOp
import net.sergeych.synctools.invoke
import kotlin.test.Test
import kotlin.test.assertContains
import kotlin.test.assertEquals
class NetworkTest {
@Test
fun udpProviderTest() = runTest {
Log.connectConsole(Log.Level.DEBUG)
val s1 = UdpServer(17120)
val s2 = UdpServer(17121)
s1.send("Hello".encodeToUByteArray(), "localhost", 17121)
val d1 = s2.incoming.receive()
assertEquals(d1.address.port, 17120)
assertEquals("Hello", d1.message.toByteArray().decodeToString())
s1.send("world".encodeToUByteArray(), d1.address)
assertEquals("world", s1.incoming.receive().message.toByteArray().decodeToString())
// println("s1: ${s1.bindAddress()}")
}
@Test
fun tcpAsyncConnectionTest() = runTest {
initCrypto()
Log.connectConsole(Log.Level.DEBUG)
coroutineScope {
val serverFlow = acceptTcpDevice(17171)
val op = ProtectedOp()
var pills = setOf<String>()
val j = launch {
println("serf")
serverFlow.collect { device ->
println("serf 0")
launch {
println("serf 1")
device.output.send("Hello, world!".encodeToUByteArray())
device.output.send("Great".encodeToUByteArray())
while (true) {
val x = device.input.receive().decodeFromUByteArray()
if (x.startsWith("die")) {
op.invoke {
pills += x
}
cancel()
}
else
println("ignoring unexpected input: $x")
}
}
}
}
yield()
run {
try {
println("pre-con")
val s = connectTcpDevice("127.0.1.1:17171".toNetworkAddress())
println("2")
assertEquals("Hello, world!", s.input.receive().decodeFromUByteArray())
assertEquals("Great", s.input.receive().decodeFromUByteArray())
s.output.send("Goodbye".encodeToUByteArray())
s.output.send("die1".encodeToUByteArray())
s.close()
}
catch(t: Throwable) {
t.printStackTrace()
throw t
}
}
println("pre-con2")
val s1 = connectTcpDevice("127.0.1.1:17171".toNetworkAddress())
assertEquals("Hello, world!", s1.input.receive().decodeFromUByteArray())
assertEquals("Great", s1.input.receive().decodeFromUByteArray())
s1.output.send("die2".encodeToUByteArray())
s1.close()
// check that channels were flushed prior to closed:
assertContains(pills, "die1")
assertContains(pills, "die2")
// Check that server jobs are closed
j.cancelAndJoin()
}
}
}

74
src/ktorSocketMain/kotlin/adapter/socketClient.kt
View File

@ -2,27 +2,16 @@ package net.sergeych.kiloparsec.adapter
import io.ktor.network.selector.* import io.ktor.network.selector.*
import io.ktor.network.sockets.* import io.ktor.network.sockets.*
import io.ktor.util.network.* import kotlinx.coroutines.*
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.channels.Channel import kotlinx.coroutines.channels.Channel
import kotlinx.coroutines.channels.ClosedReceiveChannelException import kotlinx.coroutines.channels.ClosedReceiveChannelException
import kotlinx.coroutines.flow.Flow import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.flow import kotlinx.coroutines.flow.flow
import kotlinx.coroutines.isActive
import kotlinx.coroutines.launch
import net.sergeych.kiloparsec.AsyncVarint import net.sergeych.kiloparsec.AsyncVarint
import net.sergeych.kiloparsec.LocalInterface import net.sergeych.kiloparsec.LocalInterface
import net.sergeych.mp_logger.* import net.sergeych.mp_logger.*
import net.sergeych.tools.AtomicCounter import net.sergeych.tools.AtomicCounter
import net.sergeych.tools.AtomicValue
class SocketNetworkAddress(override val host: String, override val port: Int) : NetworkAddress {
override fun toString(): String {
return "$host:$port"
}
}
actual fun NetworkAddress(host: String, port: Int): NetworkAddress = SocketNetworkAddress(host, port)
private val logCounter = AtomicCounter(0) private val logCounter = AtomicCounter(0)
@ -30,51 +19,66 @@ class ProtocolException(text: String, cause: Throwable? = null) : RuntimeExcepti
const val MAX_TCP_BLOCK_SIZE = 16776216 const val MAX_TCP_BLOCK_SIZE = 16776216
actual fun acceptTcpDevice(port: Int): Flow<InetTransportDevice> { fun acceptTcpDevice(port: Int): Flow<InetTransportDevice> {
val selectorManager = SelectorManager(Dispatchers.IO) val selectorManager = SelectorManager(Dispatchers.IO)
val serverSocket = aSocket(selectorManager).tcp().bind("127.0.0.1", port) val serverSocket = aSocket(selectorManager).tcp().bind("127.0.0.1", port)
val log = LogTag("TCPS${logCounter.incrementAndGet()}")
return flow { return flow {
while(true) { while (true) {
log.info { "Accepting incoming connections on $port" }
serverSocket.accept().let { sock -> serverSocket.accept().let { sock ->
log.info { "Emitting transport device" }
emit(inetTransportDevice(sock, "srv")) emit(inetTransportDevice(sock, "srv"))
} }
} }
} }
} }
actual suspend fun connectTcpDevice(address: NetworkAddress): InetTransportDevice { suspend fun connectTcpDevice(address: String) = connectTcpDevice(address.toNetworkAddress())
suspend fun connectTcpDevice(address: NetworkAddress): InetTransportDevice {
val selectorManager = SelectorManager(Dispatchers.IO) val selectorManager = SelectorManager(Dispatchers.IO)
val socket = aSocket(selectorManager).tcp().connect(address.host, address.port) val socket = aSocket(selectorManager).tcp().connect(address.host, address.port)
println("Connected to ${address.host}:${address.port}")
return inetTransportDevice(socket) return inetTransportDevice(socket)
} }
fun String.toNetworkAddress(): NetworkAddress {
val (host, port) = this.split(":").map { it.trim() }
return NetworkAddress(host, port.toInt())
}
private fun inetTransportDevice( private fun inetTransportDevice(
sock: Socket, sock: Socket,
suffix: String = "cli", suffix: String = "cli",
): InetTransportDevice { ): InetTransportDevice {
val networkAddress = sock.remoteAddress.toJavaAddress().let { NetworkAddress(it.hostname, it.port) } val networkAddress = (sock.remoteAddress as InetSocketAddress).let { NetworkAddress(it.hostname, it.port) }
val inputBlocks = Channel<UByteArray>(4096) val inputBlocks = Channel<UByteArray>(4096)
val outputBlocks = Channel<UByteArray>(4096) val outputBlocks = Channel<UByteArray>(4096)
val log = LogTag("TCPT${logCounter.incrementAndGet()}:$suffix:$networkAddress") val log = LogTag("TCPT${logCounter.incrementAndGet()}:$suffix:$networkAddress")
val stopCalled = AtomicValue(false)
fun stop() { fun stop() {
log.info { "stopping" } stopCalled.mutate {
runCatching { inputBlocks.close()} if (!it) {
runCatching { outputBlocks.close()} log.debug { "stopping" }
if( !sock.isClosed ) runCatching { sock.close()} runCatching { inputBlocks.close() }
runCatching { outputBlocks.close() }
if (!sock.isClosed)
runCatching {
log.debug { "closing socket by stop" }
sock.close()
}
else
log.debug { "socket is already closed when stop is called" }
} else
log.debug { "already stopped" }
true
}
} }
sock.launch { sock.launch {
log.debug { "opening read channel" } log.debug { "opening read channel" }
val sockInput = runCatching { sock.openReadChannel() }.getOrElse { val sockInput = runCatching { sock.openReadChannel() }.getOrElse {
log.warning { "failed to open read channel $it" } log.warning { "failed to open read channel $it" }
sock.close() stop()
throw IllegalStateException("failed to open read channel") throw IllegalStateException("failed to open read channel")
} }
while (isActive && sock.isActive) { while (isActive && sock.isActive) {
@ -82,15 +86,16 @@ private fun inetTransportDevice(
val size = AsyncVarint.decodeUnsigned(sockInput).toInt() val size = AsyncVarint.decodeUnsigned(sockInput).toInt()
if (size > MAX_TCP_BLOCK_SIZE) // 16M is a max command block if (size > MAX_TCP_BLOCK_SIZE) // 16M is a max command block
throw ProtocolException("Illegal block size: $size should be < $MAX_TCP_BLOCK_SIZE") throw ProtocolException("Illegal block size: $size should be < $MAX_TCP_BLOCK_SIZE")
log.info { "read size: $size" }
val data = ByteArray(size) val data = ByteArray(size)
log.info { "data ready" }
sockInput.readFully(data, 0, size) sockInput.readFully(data, 0, size)
inputBlocks.send(data.toUByteArray()) inputBlocks.send(data.toUByteArray())
} catch (e: ClosedReceiveChannelException) { } catch (e: ClosedReceiveChannelException) {
log.error { "closed receive channel " }
stop() stop()
break break
} catch (_: CancellationException) { } catch (_: CancellationException) {
log.error { "cancellation exception " }
break
} catch (e: Exception) { } catch (e: Exception) {
log.exception { "unexpected exception in TCP socket read" to e } log.exception { "unexpected exception in TCP socket read" to e }
stop() stop()
@ -105,16 +110,17 @@ private fun inetTransportDevice(
val block = outputBlocks.receive() val block = outputBlocks.receive()
AsyncVarint.encodeUnsigned(block.size.toULong(), sockOutput) AsyncVarint.encodeUnsigned(block.size.toULong(), sockOutput)
sockOutput.writeFully(block.toByteArray(), 0, block.size) sockOutput.writeFully(block.toByteArray(), 0, block.size)
log.info { "Client sock output: ${block.size}" }
sockOutput.flush() sockOutput.flush()
} catch (_: CancellationException) { } catch (_: CancellationException) {
log.info { "Caught cancellation, closing transport" } log.debug { "cancellation exception on output" }
stop()
break
} catch (_: LocalInterface.BreakConnectionException) { } catch (_: LocalInterface.BreakConnectionException) {
log.info { "requested connection break" } log.debug { "requested connection break" }
stop() stop()
break break
} catch (_: ClosedReceiveChannelException) { } catch (_: ClosedReceiveChannelException) {
log.info { "receive block channel closed, closing the socket" } log.debug { "receive block channel closed, closing the socket" }
stop() stop()
break break
} catch (e: Exception) { } catch (e: Exception) {
@ -124,10 +130,10 @@ private fun inetTransportDevice(
} }
} }
} }
val device = InetTransportDevice(inputBlocks, outputBlocks, networkAddress, { val device = InetTransportDevice(inputBlocks, outputBlocks, networkAddress, {
log.info { "Close has been called" }
stop() stop()
}) })
log.info { "Transport ready" } log.debug { "Transport ready" }
return device return device
} }

77
src/ktorSocketTest/kotlin/TcpTest.kt Normal file
View File

@ -0,0 +1,77 @@
import kotlinx.coroutines.delay
import kotlinx.coroutines.test.runTest
import net.sergeych.crypto2.initCrypto
import net.sergeych.kiloparsec.*
import net.sergeych.kiloparsec.adapter.acceptTcpDevice
import net.sergeych.kiloparsec.adapter.connectTcpDevice
import net.sergeych.mp_logger.Log
import kotlin.random.Random
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertIs
class TcpTest {
class TestException : Exception("test1")
@Test
fun tcpTest() = runTest {
initCrypto()
Log.connectConsole(Log.Level.DEBUG)
data class Session(
var data: String
)
val port = 27170 + Random.nextInt(1, 200)
val cmdSave by command<String, Unit>()
val cmdLoad by command<Unit, String>()
val cmdDrop by command<Unit, Unit>()
val cmdException by command<Unit, Unit>()
val cli = KiloInterface<Session>().apply {
registerError { TestException() }
onConnected { session.data = "start" }
on(cmdSave) { session.data = it }
on(cmdLoad) {
session.data
}
on(cmdException) {
throw TestException()
}
on(cmdDrop) {
throw LocalInterface.BreakConnectionException()
}
}
val server = KiloServer(cli, acceptTcpDevice(port)) {
Session("unknown")
}
val client = KiloClient<Unit>() {
addErrors(cli)
connect { connectTcpDevice("localhost:$port") }
}
delay(500)
assertEquals("start", client.call(cmdLoad))
client.call(cmdSave, "foobar")
assertEquals("foobar", client.call(cmdLoad))
val res = kotlin.runCatching { client.call(cmdException) }
println(res.exceptionOrNull())
assertIs<TestException>(res.exceptionOrNull())
assertEquals("foobar", client.call(cmdLoad))
println("----------------------------------- pre drops")
assertThrows<RemoteInterface.ClosedException> { client.call(cmdDrop) }
println("----------------------------------- DROPPED")
// reconnect?
assertEquals("start", client.call(cmdLoad))
println("------------------------------=---- RECONNECTED")
server.close()
println("****************************************************************")
}
}

2
src/nativeMain/kotlin/net/sergeych/kiloparsec/adapter/DatagramProvider.native.kt
View File

@ -1,2 +0,0 @@
package net.sergeych.kiloparsec.adapter

15
src/nativeMain/kotlin/net/sergeych/kiloparsec/adapter/NetworkProvider.native.kt
View File

@ -1,15 +0,0 @@
package net.sergeych.kiloparsec.adapter
import kotlinx.coroutines.flow.Flow
actual fun NetworkAddress(host: String, port: Int): NetworkAddress {
TODO("Not yet implemented")
}
actual fun acceptTcpDevice(port: Int): Flow<InetTransportDevice> {
TODO("Not yet implemented")
}
actual suspend fun connectTcpDevice(address: NetworkAddress): InetTransportDevice {
TODO("Not yet implemented")
}