docs+doctest, while loop with break and labels for non-local breaks

docs/tutorial.md

@@ -36,15 +36,31 @@ If you don't want block to return anything, use `void`:
 
 Every construction is an expression that returns something (or `void`):
 
+    val x = 111 // or autotest will fail!
     val limited = if( x > 100 ) 100 else x
+    limited
+    >>> 100
 
 You can use blocks in if statement, as expected:
 
+    val x = 200
     val limited = if( x > 100 ) {
         100 + x * 0.1    
     }
     else 
         x
+    limited
+    >>> 120.0
+
+When putting multiple statments in the same line it is convenient and recommended to use `;`:
+
+    var from; var to;
+    from = 0; to = 100
+    >>> void
+
+Notice: returned value is `void` as assignment operator does not return its value. We might decide to change it.
+
+Most often you can omit `;`, but improves readability and prevent some hardly seen bugs.
 
 So the principles are:
 
@@ -55,18 +71,54 @@ So the principles are:
 
 It is rather simple, like everywhere else:
 
+    val x = 2.0
+    //
     sin(x * π/4) / 2.0
+    >>> 0.5
 
-See [math](math.md) for more on it.
+See [math](math.md) for more on it. Notice using Greek as identifier, all languages are allowed.
+
+# Variables
+
+Much like in kotlin, there are _variables_:
+
+    var name = "Sergey"
+
+Variables can be not initialized at declaration, in which case they must be assigned before use, or an exception
+will be thrown:
+
+    var foo
+    // WRONG! Exception will be thrown at next line:
+    foo + "bar"
+
+Correct pattern is:
+
+    foo = "foo"
+    // now is OK:
+    foo + bar
+
+This is though a rare case when you need uninitialized variables, most often you can use conditional operatorss
+and even loops to assign results (see below).
+
+# Constants
+
+Same as in kotlin:
+
+    val HalfPi = π / 2
+
+Note using greek characters in identifiers! All letters allowed, but remember who might try to read your script, most likely will know some English, the rest is the pure uncertainty.
 
 # Defining functions
 
     fun check(amount) {
         if( amount > 100 )
-            "anough"
+            "enough"
         else
             "more"
     }
+    >>> Callable@...
+
+Notice how function definition return a value, instance of `Callable`. 
 
 You can use both `fn` and `fun`. Note that function declaration _is an expression returning callable_.
 
@@ -74,10 +126,11 @@ There are default parameters in Ling:
 
     fn check(amount, prefix = "answer: ") {
         prefix + if( amount > 100 )
-            "anough"
+            "enough"
         else
             "more" 
     }
+    >>> Callable@...
 
 ## Closures
 
@@ -86,12 +139,12 @@ Each __block has an isolated context that can be accessed from closures__. For e
     var counter = 1
 
     // this is ok: coumter is incremented
-    def increment(amount=1) {
+    fun increment(amount=1) {
         // use counter from a closure:
         counter = counter + amount
     }
 
-    val taskAlias = def someTask() {
+    val taskAlias = fun someTask() {
         // this obscures global outer var with a local one
         var counter = 0
         // ...
@@ -99,6 +152,7 @@ Each __block has an isolated context that can be accessed from closures__. For e
         // ...
         counter
     }
+    >>> void
 
 As was told, `def` statement return callable for the function, it could be used as a parameter, or elsewhere
 to call it:
@@ -132,28 +186,94 @@ Concatenation is a `+`: `"hello " + name` works as expected. No confusion.
 
 String literal could be multiline:
 
-    "
+    "Hello
-        Hello,
-        World!
-    "
-    >>> "Hello
     World"
 
-In that case compiler removes left margin and first/last empty lines. Note that it won't remove margin:
+though multiline literals is yet work in progress.
 
-    "Hello,
+# Flow control operators
-     World
-    "
-    >>> "Hello,
-        World
-       "
 
-because the first line has no margin in the literal.
+## if-then-else
+
+As everywhere else, and as expression:
+
+    val count = 11
+    if( count > 10 )
+        println("too much")
+    else {
+        // do something else
+        println("just "+count)
+    }
+    >>> too much
+    >>> void
+
+Notice returned value `void`: it is because of `println` have no return value, e.g., `void`.
+
+
+Or, more neat:
+
+    var count = 3
+    println( if( count > 10 ) "too much" else "just " + count )
+    >>> just 3
+    >>> void
+
+## while 
+
+Regular pre-condition while loop, as expression, loop returns it's last line result:
+
+    var count = 0
+    while( count < 5 ) {
+        count = count + 1
+        count * 10
+    }
+    >>> 50
+
+We can break as usual:
+
+    var count = 0
+    while( count < 5 ) {
+        if( count < 5 ) break
+        count = count + 1
+        count * 10
+    }
+    >>> void
+
+Why `void`? Because `break` drops out without the chute, not providing anything to return. Indeed, we should provide exit value in the case:
+
+    var count = 0
+    while( count < 5 ) {
+        if( count > 3 ) break "too much"
+        count = count + 1
+        count * 10
+    }
+    >>> too much
+
+## Breaking nested loops
+
+If you have several loops and want to exit not the inner one, use labels:
+
+    var count = 0
+    // notice the label:
+    outerLoop@ while( count < 5 ) {
+        var innerCount = 0
+        while( innerCount < 100 ) {
+            innerCount = innerCount + 1
+
+            if( innerCount == 5 && count == 2 )
+                // and here we break the labelled loop:
+                break@outerLoop "5/2 situation"
+        }
+        count = count + 1
+        count * 10
+    }
+    >>> 5/2 situation
+
+The label can be any valid identifier, even a keyword, labels exist in their own, isolated world, so no risk of occasional clash. Labels are also scoped to their context and do not exist outside it.
 
 # Comments
 
     // single line comment
     var result = null // here we will store the result
-
+    >>> void
 
 

library/src/commonMain/kotlin/net/sergeych/ling/Arguments.kt

@@ -1,6 +1,6 @@
 package net.sergeych.ling
 
-data class Arguments(val callerPos: Pos,val list: List<Info>) {
+data class Arguments(val callerPos: Pos,val list: List<Info>): Iterable<Obj> {
 
     data class Info(val value: Obj,val pos: Pos)
 
@@ -16,4 +16,8 @@ data class Arguments(val callerPos: Pos,val list: List<Info>) {
     companion object {
         val EMPTY = Arguments("".toSource().startPos,emptyList())
     }
+
+    override fun iterator(): Iterator<Obj> {
+        return list.map { it.value }.iterator()
+    }
 }

library/src/commonMain/kotlin/net/sergeych/ling/Compiler.kt

@@ -45,13 +45,23 @@ type alias has target type name. So we have to have something that denotes a _ty
 //}
 
 
+class CompilerContext(tokens: List<Token>) : ListIterator<Token> by tokens.listIterator() {
+    val labels = mutableSetOf<String>()
+
+    fun ensureLabelIsValid(pos: Pos, label: String) {
+        if (label !in labels)
+            throw ScriptError(pos, "Undefined label '$label'")
+    }
+}
+
+
 class Compiler {
 
     fun compile(source: Source): Script {
-        return parseScript(source.startPos, parseLing(source).listIterator())
+        return parseScript(source.startPos, CompilerContext(parseLing(source)))
     }
 
-    private fun parseScript(start: Pos, tokens: ListIterator<Token>): Script {
+    private fun parseScript(start: Pos, tokens: CompilerContext): Script {
         val statements = mutableListOf<Statement>()
         while (parseStatement(tokens)?.also {
                 statements += it
@@ -60,7 +70,7 @@ class Compiler {
         return Script(start, statements)
     }
 
-    private fun parseStatement(tokens: ListIterator<Token>): Statement? {
+    private fun parseStatement(tokens: CompilerContext): Statement? {
         while (true) {
             val t = tokens.next()
             return when (t.type) {
@@ -71,7 +81,7 @@ class Compiler {
                         // this _is_ assignment statement
                         return AssignStatement(
                             t.pos, t.value,
-                            parseExpression(tokens) ?: throw ScriptError(
+                            parseStatement(tokens) ?: throw ScriptError(
                                 t.pos,
                                 "Expecting expression for assignment operator"
                             )
@@ -88,8 +98,11 @@ class Compiler {
                         }
                 }
 
+                Token.Type.LABEL -> continue
                 Token.Type.SINLGE_LINE_COMMENT, Token.Type.MULTILINE_COMMENT -> continue
 
+                Token.Type.NEWLINE -> continue
+
                 Token.Type.SEMICOLON -> continue
 
                 Token.Type.LBRACE -> {
@@ -113,7 +126,7 @@ class Compiler {
         }
     }
 
-    private fun parseExpression(tokens: ListIterator<Token>, level: Int = 0): Statement? {
+    private fun parseExpression(tokens: CompilerContext, level: Int = 0): Statement? {
         if (level == lastLevel)
             return parseTerm(tokens)
         var lvalue = parseExpression(tokens, level + 1)
@@ -136,7 +149,7 @@ class Compiler {
         return lvalue
     }
 
-    fun parseTerm(tokens: ListIterator<Token>): Statement? {
+    fun parseTerm(tokens: CompilerContext): Statement? {
         // call op
         // index op
         // unary op
@@ -187,7 +200,7 @@ class Compiler {
 
     }
 
-    fun parseVarAccess(id: Token, tokens: ListIterator<Token>, path: List<String> = emptyList()): Statement {
+    fun parseVarAccess(id: Token, tokens: CompilerContext, path: List<String> = emptyList()): Statement {
         val nt = tokens.next()
 
         fun resolve(context: Context): Context {
@@ -219,7 +232,7 @@ class Compiler {
                         Token.Type.RPAREN, Token.Type.COMMA -> {}
                         else -> {
                             tokens.previous()
-                            parseExpression(tokens)?.let { args += Arguments.Info(it, t.pos) }
+                            parseStatement(tokens)?.let { args += Arguments.Info(it, t.pos) }
                                 ?: throw ScriptError(t.pos, "Expecting arguments list")
                         }
                     }
@@ -247,6 +260,14 @@ class Compiler {
             else -> {
                 // just access the var
                 tokens.previous()
+//                val t = tokens.next()
+//                tokens.previous()
+//                println(t)
+//                if( path.isEmpty() ) {
+//                     statement?
+//                    tokens.previous()
+//                    parseStatement(tokens) ?: throw ScriptError(id.pos, "Expecting expression/statement")
+//                } else
                 statement(id.pos) {
                     val v = resolve(it).get(id.value) ?: throw ScriptError(id.pos, "Undefined variable: ${id.value}")
                     v.value ?: throw ScriptError(id.pos, "Variable $id is not initialized")
@@ -256,7 +277,7 @@ class Compiler {
     }
 
 
-    fun parseNumber(isPlus: Boolean, tokens: ListIterator<Token>): Obj {
+    fun parseNumber(isPlus: Boolean, tokens: CompilerContext): Obj {
         val t = tokens.next()
         return when (t.type) {
             Token.Type.INT, Token.Type.HEX -> {
@@ -279,43 +300,131 @@ class Compiler {
      * Parse keyword-starting statenment.
      * @return parsed statement or null if, for example. [id] is not among keywords
      */
-    private fun parseKeywordStatement(id: Token, tokens: ListIterator<Token>): Statement? = when (id.value) {
+    private fun parseKeywordStatement(id: Token, cc: CompilerContext): Statement? = when (id.value) {
-        "val" -> parseVarDeclaration(id.value, false, tokens)
+        "val" -> parseVarDeclaration(id.value, false, cc)
-        "var" -> parseVarDeclaration(id.value, true, tokens)
+        "var" -> parseVarDeclaration(id.value, true, cc)
-        "fn", "fun" -> parseFunctionDeclaration(tokens)
+        "while" -> parseWhileStatement(cc)
-        "if" -> parseIfStatement(tokens)
+        "break" -> parseBreakStatement(id.pos, cc)
+        "fn", "fun" -> parseFunctionDeclaration(cc)
+        "if" -> parseIfStatement(cc)
         else -> null
     }
 
-    private fun parseIfStatement(tokens: ListIterator<Token>): Statement {
+    fun getLabel(cc: CompilerContext, maxDepth: Int = 2): String? {
-        var t = tokens.next()
+        var cnt = 0
-        val start = t.pos
+        var found: String? = null
-        if( t.type != Token.Type.LPAREN)
+        while (cc.hasPrevious() && cnt < maxDepth) {
-            throw ScriptError(t.pos, "Bad if statement: expected '('")
+            val t = cc.previous()
+            cnt++
+            if (t.type == Token.Type.LABEL) {
+                found = t.value
+                break
+            }
+        }
+        while (cnt-- > 0) cc.next()
+        return found
+    }
+
+    private fun parseWhileStatement(cc: CompilerContext): Statement {
+        val label = getLabel(cc)?.also { cc.labels += it }
+        val start = ensureLparen(cc)
+        val condition = parseExpression(cc) ?: throw ScriptError(start, "Bad while statement: expected expression")
+        ensureRparen(cc)
+
+        val body = parseStatement(cc) ?: throw ScriptError(start, "Bad while statement: expected statement")
+        label?.also { cc.labels -= it }
+
+        return statement(body.pos) {
+            var result: Obj = ObjVoid
+            while (condition.execute(it).toBool()) {
+                try {
+                    result = body.execute(it)
+                } catch (lbe: LoopBreakContinueException) {
+                    if (lbe.label == label || lbe.label == null) {
+                        if (lbe.doContinue) continue
+                        else {
+                            result = lbe.result
+                            break
+                        }
+                    } else
+                        throw lbe
+                }
+            }
+            result
+        }
+    }
+
+    private fun parseBreakStatement(start: Pos, cc: CompilerContext): Statement {
+        var t = cc.next()
+
+        val label = if (t.pos.line != start.line || t.type != Token.Type.ATLABEL) {
+            cc.previous()
+            null
+        } else {
+            t.value
+        }?.also {
+            // check that label is defined
+            cc.ensureLabelIsValid(start, it)
+        }
+
+        // expression?
+        t = cc.next()
+        cc.previous()
+        val resultExpr = if (t.pos.line == start.line && (!t.isComment &&
+                    t.type != Token.Type.SEMICOLON &&
+                    t.type != Token.Type.NEWLINE)
+        ) {
+            // we have something on this line, could be expression
+            parseStatement(cc)
+        } else null
+
+        return statement(start) {
+            val returnValue = resultExpr?.execute(it)// ?: ObjVoid
+            throw LoopBreakContinueException(
+                doContinue = false,
+                label = label,
+                result = returnValue ?: ObjVoid
+            )
+        }
+    }
+
+    private fun ensureRparen(tokens: CompilerContext): Pos {
+        val t = tokens.next()
+        if (t.type != Token.Type.RPAREN)
+            throw ScriptError(t.pos, "expected ')'")
+        return t.pos
+    }
+
+    private fun ensureLparen(tokens: CompilerContext): Pos {
+        val t = tokens.next()
+        if (t.type != Token.Type.LPAREN)
+            throw ScriptError(t.pos, "expected '('")
+        return t.pos
+    }
+
+    private fun parseIfStatement(tokens: CompilerContext): Statement {
+        val start = ensureLparen(tokens)
 
         val condition = parseExpression(tokens)
-            ?: throw ScriptError(t.pos, "Bad if statement: expected expression")
+            ?: throw ScriptError(start, "Bad if statement: expected expression")
 
-        t = tokens.next()
+        val pos = ensureRparen(tokens)
-        if( t.type != Token.Type.RPAREN)
-            throw ScriptError(t.pos, "Bad if statement: expected ')' after condition expression")
 
-        val ifBody = parseStatement(tokens) ?: throw ScriptError(t.pos, "Bad if statement: expected statement")
+        val ifBody = parseStatement(tokens) ?: throw ScriptError(pos, "Bad if statement: expected statement")
 
         // could be else block:
         val t2 = tokens.next()
 
         // we generate different statements: optimization
-        return if( t2.type == Token.Type.ID && t2.value == "else") {
+        return if (t2.type == Token.Type.ID && t2.value == "else") {
-            val elseBody = parseStatement(tokens) ?: throw ScriptError(t.pos, "Bad else statement: expected statement")
+            val elseBody = parseStatement(tokens) ?: throw ScriptError(pos, "Bad else statement: expected statement")
             return statement(start) {
                 if (condition.execute(it).toBool())
                     ifBody.execute(it)
                 else
                     elseBody.execute(it)
             }
-        }
+        } else {
-        else {
             tokens.previous()
             statement(start) {
                 if (condition.execute(it).toBool())
@@ -332,7 +441,7 @@ class Compiler {
         val defaultValue: Statement? = null
     )
 
-    private fun parseFunctionDeclaration(tokens: ListIterator<Token>): Statement {
+    private fun parseFunctionDeclaration(tokens: CompilerContext): Statement {
         var t = tokens.next()
         val start = t.pos
         val name = if (t.type != Token.Type.ID)
@@ -392,23 +501,24 @@ class Compiler {
         }
     }
 
-    private fun parseBlock(tokens: ListIterator<Token>): Statement {
+    private fun parseBlock(tokens: CompilerContext): Statement {
         val t = tokens.next()
         if (t.type != Token.Type.LBRACE)
             throw ScriptError(t.pos, "Expected block body start: {")
         val block = parseScript(t.pos, tokens)
-            return statement(t.pos) {
+        return statement(t.pos) {
-                // block run on inner context:
+            // block run on inner context:
-                block.execute(it.copy())
+            block.execute(it.copy())
-            }.also {
+        }.also {
             val t1 = tokens.next()
             if (t1.type != Token.Type.RBRACE)
                 throw ScriptError(t1.pos, "unbalanced braces: expected block body end: }")
         }
     }
 
-    private fun parseVarDeclaration(kind: String, mutable: Boolean, tokens: ListIterator<Token>): Statement {
+    private fun parseVarDeclaration(kind: String, mutable: Boolean, tokens: CompilerContext): Statement {
         val nameToken = tokens.next()
+        val start = nameToken.pos
         if (nameToken.type != Token.Type.ID)
             throw ScriptError(nameToken.pos, "Expected identifier after '$kind'")
         val name = nameToken.value
@@ -416,13 +526,13 @@ class Compiler {
         var setNull = false
         if (eqToken.type != Token.Type.ASSIGN) {
             if (!mutable)
-                throw ScriptError(eqToken.pos, "Expected initializer: '=' after '$kind ${name}'")
+                throw ScriptError(start, "val must be initialized")
             else {
                 tokens.previous()
                 setNull = true
             }
         }
-        val initialExpression = if (setNull) null else parseExpression(tokens)
+        val initialExpression = if (setNull) null else parseStatement(tokens)
             ?: throw ScriptError(eqToken.pos, "Expected initializer expression")
         return statement(nameToken.pos) { context ->
             if (context.containsLocal(name))

library/src/commonMain/kotlin/net/sergeych/ling/LoopBreakContinueException.kt

@@ -0,0 +1,7 @@
+package net.sergeych.ling
+
+class LoopBreakContinueException(
+    val doContinue: Boolean,
+    val result: Obj = ObjVoid,
+    val label: String? = null
+) : RuntimeException()

library/src/commonMain/kotlin/net/sergeych/ling/Obj.kt

@@ -64,6 +64,8 @@ object ObjVoid : Obj() {
     override fun compareTo(other: Obj): Int {
         return if (other === this) 0 else -1
     }
+
+    override fun toString(): String = "void"
 }
 
 @Serializable
@@ -127,6 +129,7 @@ data class ObjReal(val value: Double) : Obj(), Numeric {
         return value.compareTo(other.doubleValue)
     }
 
+    override fun toString(): String = value.toString()
 }
 
 @Serializable
@@ -142,6 +145,8 @@ data class ObjInt(val value: Long) : Obj(), Numeric {
         if( other !is Numeric) throw IllegalArgumentException("cannot compare $this with $other")
         return value.compareTo(other.doubleValue)
     }
+
+    override fun toString(): String = value.toString()
 }
 
 @Serializable
@@ -153,7 +158,7 @@ data class ObjBool(val value: Boolean) : Obj() {
         if( other !is ObjBool) throw IllegalArgumentException("cannot compare $this with $other")
         return value.compareTo(other.value)
     }
-
+    override fun toString(): String = value.toString()
 }
 
 data class ObjNamespace(val name: String, val context: Context) : Obj() {

library/src/commonMain/kotlin/net/sergeych/ling/Parser.kt

@@ -1,11 +1,12 @@
 package net.sergeych.ling
 
-private val idFirstChars: Set<Char> = (
+val digitsSet = ('0'..'9').toSet()
-        ('a'..'z') + ('A'..'Z') + '_' + ('а'..'я') + ('А'..'Я')
+val digits = { d: Char -> d in digitsSet }
-        ).toSet()
+val hexDigits = digitsSet + ('a'..'f') + ('A'..'F')
-val idNextChars: Set<Char> = idFirstChars + ('0'..'9')
+val idNextChars = { d: Char -> d.isLetter() || d == '_' || d.isDigit()}
-val digits = ('0'..'9').toSet()
+
-val hexDigits = digits + ('a'..'f') + ('A'..'F')
+@Suppress("unused")
+val idFirstChars = { d: Char -> d.isLetter() || d == '_' }
 
 fun parseLing(source: Source): List<Token> {
     val p = Parser(fromPos = source.startPos)
@@ -100,16 +101,38 @@ private class Parser(fromPos: Pos) {
                 } else
                     Token("&", from, Token.Type.BITAND)
             }
+            '@' -> {
+                val label = loadChars(idNextChars)
+                if( label.isNotEmpty()) Token(label, from, Token.Type.ATLABEL)
+                else raise("unexpected @ character")
+            }
+            '\n' -> Token("\n", from, Token.Type.NEWLINE)
 
             '"' -> loadStringToken()
-            in digits -> {
+            in digitsSet -> {
                 pos.back()
                 decodeNumber(loadChars(digits), from)
             }
 
             else -> {
-                if (ch.isLetter() || ch == '_')
+                // Labels processing is complicated!
-                    Token(ch + loadChars(idNextChars), from, Token.Type.ID)
+                // some@ statement: label 'some', ID 'statement'
+                // statement@some: ID 'statement', LABEL 'some'!
+                if (ch.isLetter() || ch == '_') {
+                    val text = ch + loadChars(idNextChars)
+                    if( currentChar == '@') {
+                        advance()
+                        if( currentChar.isLetter()) {
+                            // break@label or like
+                            pos.back()
+                            Token(text, from, Token.Type.ID)
+                        }
+                        else
+                            Token(text, from, Token.Type.LABEL)
+                    }
+                    else
+                        Token(text, from, Token.Type.ID)
+                }
                 else
                     raise("can't parse token")
             }
@@ -122,7 +145,7 @@ private class Parser(fromPos: Pos) {
         else if (currentChar == '.') {
             // could be decimal
             advance()
-            if (currentChar in digits) {
+            if (currentChar in digitsSet) {
                 // decimal part
                 val p2 = loadChars(digits)
                 // with exponent?
@@ -152,7 +175,7 @@ private class Parser(fromPos: Pos) {
             // could be integer, also hex:
             if (currentChar == 'x' && p1 == "0") {
                 advance()
-                Token(loadChars(hexDigits), start, Token.Type.HEX).also {
+                Token(loadChars({ it in hexDigits}), start, Token.Type.HEX).also {
                     if (currentChar.isLetter())
                         raise("invalid hex literal")
                 }
@@ -197,14 +220,19 @@ private class Parser(fromPos: Pos) {
 
     /**
      * Load characters from the set until it reaches EOF or invalid character found.
-     * stop at EOF on character not in [validChars].
+     * stop at EOF on character filtered by [isValidChar].
+     *
+     * Note this function loads only on one string. Multiline texts are not supported by
+     * this method.
+     *
      * @return the string of valid characters, could be empty
      */
-    private fun loadChars(validChars: Set<Char>): String {
+    private fun loadChars(isValidChar: (Char)->Boolean): String {
+        val startLine = pos.line
         val result = StringBuilder()
-        while (!pos.end) {
+        while (!pos.end && pos.line == startLine) {
             val ch = pos.currentChar
-            if (ch in validChars) {
+            if (isValidChar(ch)) {
                 result.append(ch)
                 advance()
             } else

library/src/commonMain/kotlin/net/sergeych/ling/Pos.kt

@@ -36,15 +36,13 @@ class MutablePos(private val from: Pos) {
     fun advance(): Char? {
         if (end) return null
         val current = lines[line]
-        return if (column+1 < current.length) {
+        return if (column < current.length) {
-            current[column++]
+            column++
+            currentChar
         } else {
             column = 0
-            while( ++line < lines.size && lines[line].isEmpty() ) {
+            if(++line >= lines.size) null
-                // skip empty lines
+            else currentChar
-            }
-            if(line >= lines.size) null
-            else lines[line][column]
         }
     }
 
@@ -55,9 +53,12 @@ class MutablePos(private val from: Pos) {
         else throw IllegalStateException("can't go back from line 0, column 0")
     }
     val currentChar: Char
-        get() =
+        get() {
-            if (end) 0.toChar()
+            if (end) return 0.toChar()
-            else lines[line][column]
+            val current = lines[line]
+            return if (column >= current.length) '\n'
+            else current[column]
+        }
 
     override fun toString() = "($line:$column)"
 

library/src/commonMain/kotlin/net/sergeych/ling/Script.kt

@@ -21,8 +21,12 @@ class Script(
     companion object {
         val defaultContext: Context = Context(null).apply {
             addFn("println") {
-                require(args.size == 1)
+                print("yn: ")
-                println(args[0].asStr.value)
+                for( (i,a) in args.withIndex() ) {
+                    if( i > 0 ) print(' ' + a.asStr.value)
+                    else print(a.asStr.value)
+                }
+                println()
                 ObjVoid
             }
             addFn("floor") {

library/src/commonMain/kotlin/net/sergeych/ling/Token.kt

@@ -1,6 +1,8 @@
 package net.sergeych.ling
 
 data class Token(val value: String, val pos: Pos, val type: Type) {
+    val isComment: Boolean by lazy { type == Type.SINLGE_LINE_COMMENT || type == Type.MULTILINE_COMMENT }
+
     @Suppress("unused")
     enum class Type {
         ID, INT, REAL, HEX, STRING, LPAREN, RPAREN, LBRACE, RBRACE, LBRACKET, RBRACKET, COMMA,
@@ -9,6 +11,8 @@ data class Token(val value: String, val pos: Pos, val type: Type) {
         EQ, NEQ, LT, LTE, GT, GTE,
         AND, BITAND, OR, BITOR, NOT, DOT, ARROW, QUESTION, COLONCOLON, PERCENT,
         SINLGE_LINE_COMMENT, MULTILINE_COMMENT,
+        LABEL,ATLABEL, // label@ at@label
+        NEWLINE,
         EOF,
     }
 

library/src/commonMain/kotlin/net/sergeych/ling/statements.kt

@@ -17,6 +17,8 @@ abstract class Statement(
         throw UnsupportedOperationException("not comparable")
     }
 
+    override fun toString(): String = "Callable@${this.hashCode()}"
+
 }
 
 fun Statement.raise(text: String): Nothing {

library/src/commonTest/kotlin/ScriptTest.kt

@@ -47,6 +47,15 @@ class ScriptTest {
 
     }
 
+    @Test
+    fun parserLabelsTest() {
+        val src = "label@ break@label".toSource()
+        val tt = parseLing(src)
+        assertEquals(Token("label", src.posAt(0, 0), Token.Type.LABEL), tt[0])
+        assertEquals(Token("break", src.posAt(0, 7), Token.Type.ID), tt[1])
+        assertEquals(Token("label", src.posAt(0, 12), Token.Type.ATLABEL), tt[2])
+    }
+
     @Test
     fun parse0Test() {
         val src = """
@@ -109,10 +118,14 @@ class ScriptTest {
     @Test
     fun varsAndConstsTest() = runTest {
         val context = Context()
-        assertEquals(ObjVoid,context.eval("""
+        assertEquals(
+            ObjVoid, context.eval(
+                """
             val a = 17
             var b = 3
-        """.trimIndent()))
+        """.trimIndent()
+            )
+        )
         assertEquals(17, context.eval("a").toInt())
         assertEquals(20, context.eval("b + a").toInt())
         assertFailsWith<ScriptError> {
@@ -137,11 +150,13 @@ class ScriptTest {
             assertEquals(17, context.eval("foo(3)").toInt())
         }
 
-        context.eval("""
+        context.eval(
+            """
             fn bar(a, b=10) {
                 a + b + 1
             }
-        """.trimIndent())
+        """.trimIndent()
+        )
         assertEquals(10, context.eval("bar(3, 6)").toInt())
         assertEquals(14, context.eval("bar(3)").toInt())
     }
@@ -166,8 +181,8 @@ class ScriptTest {
     @Test
     fun nullAndVoidTest() = runTest {
         val context = Context()
-        assertEquals(ObjVoid,context.eval("void"))
+        assertEquals(ObjVoid, context.eval("void"))
-        assertEquals(ObjNull,context.eval("null"))
+        assertEquals(ObjNull, context.eval("null"))
     }
 
     @Test
@@ -227,31 +242,34 @@ class ScriptTest {
         assertFalse { eval("3 >= 4").toBool() }
         assertFalse { eval("3 < 2").toBool() }
         assertFalse { eval("3 <= 2").toBool() }
-        assertTrue { eval("3 <= 3").toBool()}
+        assertTrue { eval("3 <= 3").toBool() }
-        assertTrue { eval("3 <= 4").toBool()}
+        assertTrue { eval("3 <= 4").toBool() }
-        assertTrue { eval("3 < 4").toBool()}
+        assertTrue { eval("3 < 4").toBool() }
-        assertFalse { eval("4 < 3").toBool()}
+        assertFalse { eval("4 < 3").toBool() }
-        assertFalse { eval("4 <= 3").toBool()}
+        assertFalse { eval("4 <= 3").toBool() }
     }
 
     @Test
     fun ifTest() = runTest {
         // if - single line
         var context = Context()
-        context.eval("""
+        context.eval(
+            """
             fn test1(n) {
                 var result = "more"
                 if( n >= 10 ) 
                     result = "enough"
                 result
             }
-        """.trimIndent())
+        """.trimIndent()
+        )
         assertEquals("enough", context.eval("test1(11)").toString())
         assertEquals("more", context.eval("test1(1)").toString())
 
         // if - multiline (block)
         context = Context()
-        context.eval("""
+        context.eval(
+            """
             fn test1(n) {
                 var prefix = "answer: "
                 var result = "more"
@@ -262,26 +280,30 @@ class ScriptTest {
                 }
                 prefix + result
             }
-        """.trimIndent())
+        """.trimIndent()
+        )
         assertEquals("answer: enough", context.eval("test1(11)").toString())
         assertEquals("answer: more", context.eval("test1(1)").toString())
 
         // else single line1
         context = Context()
-        context.eval("""
+        context.eval(
+            """
             fn test1(n) {
                 if( n >= 10 ) 
                     "enough"
                 else
                     "more"
             }
-        """.trimIndent())
+        """.trimIndent()
+        )
         assertEquals("enough", context.eval("test1(11)").toString())
         assertEquals("more", context.eval("test1(1)").toString())
 
         // if/else with blocks
         context = Context()
-        context.eval("""
+        context.eval(
+            """
             fn test1(n) {
                 if( n > 20 ) {
                     "too much"
@@ -292,10 +314,150 @@ class ScriptTest {
                     "more"
                 }
             }
-        """.trimIndent())
+        """.trimIndent()
+        )
         assertEquals("enough", context.eval("test1(11)").toString())
         assertEquals("more", context.eval("test1(1)").toString())
         assertEquals("too much", context.eval("test1(100)").toString())
     }
 
+    @Test
+    fun lateInitTest() = runTest {
+        assertEquals(
+            "ok", eval(
+                """
+
+            var late
+            
+            fun init() {
+                late = "ok"
+            }
+            
+            init()
+            late
+        """.trimIndent()
+            ).toString()
+        )
+
+    }
+
+    @Test
+    fun whileTest() = runTest {
+        assertEquals(
+            5.0,
+            eval(
+                """
+                var acc = 0
+                while( acc < 5 ) acc = acc + 0.5
+                acc
+                """
+            ).toDouble()
+        )
+        assertEquals(
+            5.0,
+            eval(
+                """
+                var acc = 0
+                // return from while
+                while( acc < 5 ) {
+                    acc = acc + 0.5
+                    acc
+                }
+                """
+            ).toDouble()
+        )
+        assertEquals(
+            3.0,
+            eval(
+                """
+                var acc = 0
+                while( acc < 5 ) {
+                    acc = acc + 0.5
+                    if( acc >= 3 ) break
+                }
+
+                acc
+
+                """
+            ).toDouble()
+        )
+        assertEquals(
+            17.0,
+            eval(
+                """
+                var acc = 0
+                while( acc < 5 ) {
+                    acc = acc + 0.5
+                    if( acc >= 3 ) break 17
+                }
+                """
+            ).toDouble()
+        )
+    }
+
+    @Test
+    fun whileNonLocalBreakTest() = runTest {
+        assertEquals(
+            "ok2:3:7", eval(
+                """
+            var t1 = 10
+            outer@ while( t1 > 0 ) {
+                var t2 = 10
+                while( t2 > 0 ) {
+                    t2 = t2 - 1
+                    if( t2 == 3 && t1 == 7) {
+                        break@outer "ok2:"+t2+":"+t1
+                    }
+                }
+                t1 = t1 - 1
+                t1
+            }
+        """.trimIndent()
+            ).toString()
+        )
+    }
+
+    @Test
+    fun bookTest0() = runTest {
+        assertEquals(
+            "just 3",
+            eval(
+                """
+                val count = 3
+                val res = if( count > 10 ) "too much" else "just " + count
+                println(res)
+                res
+                """.trimIndent()
+            )
+                .toString()
+        )
+        assertEquals(
+            "just 3",
+            eval(
+                """
+                val count = 3
+                var res = if( count > 10 ) "too much" else "it's " + count
+                res = if( count > 10 ) "too much" else "just " + count
+                println(res)
+                res
+                """.trimIndent()
+            )
+                .toString()
+        )
+    }
+    @Test
+    fun bookTest1() = runTest {
+//        assertEquals(
+//            "just 3",
+            eval(
+                """
+                val count = 3
+                println( 
+                    if( count > 10 ) "too much" else "just " + count 
+                )
+                """.trimIndent()
+            )
+//                .toString()
+//        )
+    }
 }

library/src/jvmTest/kotlin/BookTest.kt

@@ -0,0 +1,177 @@
+import kotlinx.coroutines.Dispatchers
+import kotlinx.coroutines.flow.Flow
+import kotlinx.coroutines.flow.flow
+import kotlinx.coroutines.flow.flowOn
+import kotlinx.coroutines.test.runTest
+import net.sergeych.ling.Context
+import net.sergeych.ling.ObjVoid
+import java.nio.file.Files.readAllLines
+import java.nio.file.Paths
+import kotlin.test.Test
+import kotlin.test.assertEquals
+import kotlin.test.fail
+
+fun leftMargin(s: String): Int {
+    var cnt = 0
+    for (c in s) {
+        when (c) {
+            ' ' -> cnt++
+            '\t' -> cnt = (cnt / 4.0 + 0.9).toInt() * 4
+            else -> break
+        }
+    }
+    return cnt
+}
+
+data class DocTest(
+    val fileName: String,
+    val line: Int,
+    val code: String,
+    val expectedOutput: String,
+    val expectedResult: String,
+    val expectedError: String? = null
+) {
+    val sourceLines by lazy { code.lines() }
+
+    override fun toString(): String {
+        return "DocTest:$fileName:${line+1}..${line + sourceLines.size}"
+    }
+
+    val detailedString by lazy {
+        val codeWithLines = sourceLines.withIndex().map { (i, s) -> "${i + line}: $s" }.joinToString("\n")
+        "$this\n" +
+                codeWithLines + "\n" +
+                "--------expected output--------\n" +
+                expectedOutput +
+                "-----expected return value-----\n" +
+                expectedResult
+    }
+}
+
+fun parseDocTests(name: String): Flow<DocTest> = flow {
+    val book = readAllLines(Paths.get("../docs/tutorial.md"))
+    var startOffset = 0
+    val block = mutableListOf<String>()
+    var startIndex = 0
+    for ((index, l) in book.withIndex()) {
+        val off = leftMargin(l)
+        when {
+            off < startOffset && startOffset != 0 -> {
+                if (l.isBlank()) {
+                    continue
+                }
+                // end of block or just text:
+                if (block.isNotEmpty()) {
+                    // check/create block
+                    // 2 lines min
+                    if (block.size > 1) {
+                        // remove prefix
+                        for ((i, s) in block.withIndex()) {
+                            var x = s
+                            // could be tabs :(
+                            val initial = leftMargin(x)
+                            do {
+                                x = x.drop(1)
+                            } while (initial - leftMargin(x) != startOffset)
+                            block[i] = x
+                        }
+//                        println(block.joinToString("\n") { "${startIndex + ii++}: $it" })
+                        val outStart = block.indexOfFirst { it.startsWith(">>>") }
+                        if (outStart < 0) {
+                            // println("No output at block from line ${startIndex+1}")
+                        } else {
+                            var isValid = true
+                            val result = mutableListOf<String>()
+                            while (block.size > outStart) {
+                                val line = block.removeAt(outStart)
+                                if (!line.startsWith(">>> ")) {
+                                    println("invalid output line, must start with '>>> ', block from ${startIndex + 1}: $line")
+                                    isValid = false
+                                    break
+                                }
+                                result.add(line.drop(4))
+                            }
+                            if (isValid) {
+                                emit(
+                                    DocTest(
+                                        name, startIndex,
+                                        block.joinToString("\n"),
+                                        if (result.size > 1)
+                                            result.dropLast(1).joinToString { it + "\n" }
+                                        else "",
+                                        result.last()
+                                    )
+                                )
+                            }
+                        }
+                        // last line '>>>'
+                    }
+                    block.clear()
+                    startOffset = 0
+                }
+            }
+
+            off != 0 && startOffset == 0 -> {
+                // start
+                block.clear()
+                startIndex = index
+                block.add(l)
+                startOffset = off
+            }
+
+            off != 0 -> {
+                block.add(l)
+            }
+
+            off == 0 && startOffset == 0 -> {
+                // skip
+            }
+
+            else -> {
+                throw RuntimeException("Unexpected line: ($off/$startOffset) $l")
+            }
+        }
+    }
+}
+    .flowOn(Dispatchers.IO)
+
+suspend fun DocTest.test() {
+    val collectedOutput = StringBuilder()
+    val context = Context().apply {
+        addFn("println") {
+            for ((i, a) in args.withIndex()) {
+                if (i > 0) collectedOutput.append(' '); collectedOutput.append(a)
+                collectedOutput.append('\n')
+            }
+            ObjVoid
+        }
+    }
+    var error: Throwable? = null
+    val result = try {
+        context.eval(code)
+    }
+    catch (e: Throwable) {
+        error = e
+        null
+    }?.toString()?.replace(Regex("@\\d+"), "@...")
+
+    if (error != null || expectedOutput != collectedOutput.toString() ||
+        expectedResult != result
+    ) {
+        println("Test failed: ${this.detailedString}")
+    }
+    error?.let { fail(it.toString()) }
+    assertEquals(expectedOutput, collectedOutput.toString(), "script output do not match")
+    assertEquals(expectedResult, result.toString(), "script result does not match")
+    //    println("OK: $this")
+}
+
+class BookTest {
+
+    @Test
+    fun testsFromTutorial() = runTest {
+        parseDocTests("../docs/tutorial.md").collect { dt ->
+            dt.test()
+        }
+    }
+}