lambda syntax added

docs/advanced_topics.md

@@ -2,24 +2,24 @@
 
 ## Closures/scopes isolation
 
-Each block has own scope, in which it can safely uses closures and override
+Each block has own scope, in which it can safely use closures and override
-outer vars:
+outer vars. Lets use some lambdas to create isolated scopes:
-
-> blocks are no-yet-ready lambda declaration so this sample will soon be altered
 
     var param = "global"
     val prefix = "param in "
+  
     val scope1 = {
         var param = prefix + "scope1"
         param
     }
+    
     val scope2 = {
         var param = prefix + "scope2"
         param
     }
-    // note that block returns its last value
+    
-    println(scope1)
+    println(scope1())
-    println(scope2)
+    println(scope2())
     println(param)
     >>> param in scope1
     >>> param in scope2
@@ -38,7 +38,9 @@ One interesting way of using closure isolation is to keep state of the functions
             counter = counter + 1
             was
         }
-    }
+    }()
+    // notice using of () above: it calls the lambda block that returns
+    // a function (callable!) that we will use:
     println(getAndIncrement())
     println(getAndIncrement())
     println(getAndIncrement())
@@ -49,4 +51,27 @@ One interesting way of using closure isolation is to keep state of the functions
 
 Inner `counter` is not accessible from outside, no way; still it is kept 
 between calls in the closure, as inner function `doit`, returned from the
-block, keeps reference to it and keeps it alive.
+block, keeps reference to it and keeps it alive.
+
+The example above could be rewritten using inner lambda, too:
+
+    val getAndIncrement = {
+        // will be updated by doIt()
+        var counter = 0
+
+        // we return callable fn from the block:
+        {
+            val was = counter
+            counter = counter + 1
+            was
+        }
+    }()
+    // notice using of () above: it calls the lambda block that returns
+    // a function (callable!) that we will use:
+    println(getAndIncrement())
+    println(getAndIncrement())
+    println(getAndIncrement())
+    >>> 0
+    >>> 1
+    >>> 2
+    >>> void

docs/tutorial.md

@@ -219,7 +219,17 @@ likely will know some English, the rest is the pure uncertainty.
 
 Notice how function definition return a value, instance of `Callable`.
 
-You can use both `fn` and `fun`. Note that function declaration _is an expression returning callable_.
+You can use both `fn` and `fun`. Note that function declaration _is an expression returning callable_,
+but Lyng syntax requires using the __lambda syntax__ to create such.
+
+    val check = { 
+        it > 0 && it < 100
+    }
+    assert( check(1) )
+    assert( !check(101) )
+    >>> void
+
+See lambdas section below.
 
 There are default parameters in Lyng:
 
@@ -239,13 +249,16 @@ Each __block has an isolated context that can be accessed from closures__. For e
 
     var counter = 1
 
-    // this is ok: coumter is incremented
+    // this is ok: counter is incremented
     fun increment(amount=1) {
         // use counter from a closure:
         counter = counter + amount
     }
 
-    val taskAlias = fun someTask() {
+    increment(10)
+    assert( counter == 11 )
+
+    val callable = {
         // this obscures global outer var with a local one
         var counter = 0
         // ...
@@ -253,24 +266,57 @@ Each __block has an isolated context that can be accessed from closures__. For e
         // ...
         counter
     }
+
+    assert(callable() == 1)
+    // but the global counter is not changed:
+    assert(counter == 11)
     >>> void
 
-As was told, `fun` statement return callable for the function, it could be used as a parameter, or elsewhere
+## Lambda functions
-to call it:
 
-    val taskAlias = fun someTask() {
+Lambda expression is a block with optional argument list ending with `->`. If argument list is omitted,
-        println("Hello")
+the call arguments will be assigned to `it`:
+
+    lambda = {
+        it + "!"
     }
-    // call the callable stored in the var
+    assert( lambda is Callable)
-    taskAlias()
+    assert( lambda("hello") == "hello!" )
-    // or directly:
+    void
-    someTask()
+
-    >>> Hello
+### `it` assignment rules
-    >>> Hello
+
+When lambda is called with:
+
+- no arguments: `it == void`
+- exactly one argument: `it` will be assigned to it
+- more than 1 argument: `it` will be a `List` with these arguments:
+
+Here is an example:
+
+    val lambda = { it }
+    assert( lambda() == void )
+    assert( lambda("one") == "one")
+    assert( lambda("one", "two") == ["one", "two"])
     >>> void
 
 If you need to create _unnamed_ function, use alternative syntax (TBD, like { -> } ?)
 
+### Declaring parameters
+
+Parameter is a list of comma-separated names, with optional default value; last
+one could be with ellipsis that means "the rest pf arguments as List":
+
+    assert( { a -> a }(10) == 10 )
+    assert( { a, b -> [a,b] }(1,2) == [1,2])
+    assert( { a, b=-1 -> [a,b] }(1) == [1,-1])
+    assert( { a, b...-> [a,...b] }(100) == [100]) 
+    // notice that splat syntax in array literal unrills
+    // ellipsis-caught arguments back:
+    assert( { a, b...-> [a,...b] }(100, 1, 2, 3) == [100, 1, 2, 3]) 
+    void
+
+
 # Lists (aka arrays)
 
 Lyng has built-in mutable array class `List` with simple literals:

library/src/commonMain/kotlin/net/sergeych/ling/Arguments.kt

@@ -24,6 +24,8 @@ data class Arguments(val list: List<Info>) : Iterable<Obj> {
 
     operator fun get(index: Int): Obj = list[index].value
 
+    val values: List<Obj>  by lazy { list.map { it.value } }
+
     fun firstAndOnly(): Obj {
         if (list.size != 1) throw IllegalArgumentException("Expected one argument, got ${list.size}")
         return list.first().value

library/src/commonMain/kotlin/net/sergeych/ling/Compiler.kt

@@ -1,5 +1,7 @@
 package net.sergeych.lyng
 
+import net.sergeych.ling.TypeDecl
+
 /**
  * The LYNG compiler.
  */
@@ -28,22 +30,6 @@ class Compiler(
             val t = cc.next()
             return when (t.type) {
                 Token.Type.ID -> {
-                    // could be keyword, assignment or just the expression
-//                    val next = tokens.next()
-//                    if (next.type == Token.Type.ASSIGN) {
-//                         this _is_ assignment statement
-//                        return AssignStatement(
-//                            t.pos, t.value,
-//                            parseStatement(tokens) ?: throw ScriptError(
-//                                t.pos,
-//                                "Expecting expression for assignment operator"
-//                            )
-//                        )
-//                    }
-//                     not assignment, maybe keyword statement:
-//                     get back the token which is not '=':
-//                    tokens.previous()
-                    // try keyword statement
                     parseKeywordStatement(t, cc)
                         ?: run {
                             cc.previous()
@@ -315,11 +301,11 @@ class Compiler(
                     }
                 }
 
-//                Token.Type.LBRACE -> {
+                Token.Type.LBRACE -> {
-//                    if( operand != null ) {
+                    if (operand != null) {
-//                        throw ScriptError(t.pos, "syntax error: lambda expression not allowed here")
+                        throw ScriptError(t.pos, "syntax error: lambda expression not allowed here")
-//                    }
+                    } else operand = parseLambdaExpression(cc)
-//                }
+                }
 
 
                 else -> {
@@ -331,6 +317,56 @@ class Compiler(
         }
     }
 
+    /**
+     * Parse lambda expression, leading '{' is already consumed
+     */
+    private fun parseLambdaExpression(cc: CompilerContext): Accessor {
+        // lambda args are different:
+        val startPos = cc.currentPos()
+        val argsDeclaration = parseArgsDeclaration(cc)
+        if (argsDeclaration != null && argsDeclaration.endTokenType != Token.Type.ARROW)
+            throw ScriptError(startPos, "lambda must have either valid arguments declaration with '->' or no arguments")
+        val pos = cc.currentPos()
+        val body = parseBlock(cc, skipLeadingBrace = true)
+        return Accessor { _ ->
+            statement {
+                val context = this.copy(pos)
+                if (argsDeclaration == null) {
+                    // no args: automatic var 'it'
+                    val l = args.values
+                    val itValue: Obj = when (l.size) {
+                        // no args: it == void
+                        0 -> ObjVoid
+                        // one args: it is this arg
+                        1 -> l[0]
+                        // more args: it is a list of args
+                        else -> ObjList(l.toMutableList())
+                    }
+                    context.addItem("it", false, itValue)
+                } else {
+                    // assign vars as declared
+                    if( args.size != argsDeclaration.args.size && !argsDeclaration.args.last().isEllipsis)
+                        raiseArgumentError("Too many arguments : called with ${args.size}, lambda accepts only ${argsDeclaration.args.size}")
+                    for ((n, a) in argsDeclaration.args.withIndex()) {
+                        if (n >= args.size) {
+                            if (a.initialValue != null)
+                                context.addItem(a.name, false, a.initialValue.execute(context))
+                            else throw ScriptError(a.pos, "argument $n is out of scope")
+                        } else {
+                            val value = if( a.isEllipsis) {
+                                ObjList(args.values.subList(n, args.values.size).toMutableList())
+                            }
+                            else
+                                args[n]
+                            context.addItem(a.name, false, value)
+                        }
+                    }
+                }
+                body.execute(context)
+            }.asReadonly
+        }
+    }
+
     private fun parseArrayLiteral(cc: CompilerContext): List<ListEntry> {
         // it should be called after LBRACKET is consumed
         val entries = mutableListOf<ListEntry>()
@@ -369,6 +405,89 @@ class Compiler(
         }
     }
 
+    data class ArgVar(
+        val name: String,
+        val type: TypeDecl = TypeDecl.Obj,
+        val pos: Pos,
+        val isEllipsis: Boolean,
+        val initialValue: Statement? = null
+    )
+
+    data class ArgsDeclaration(val args: List<ArgVar>, val endTokenType: Token.Type) {
+        init {
+            val i = args.indexOfFirst { it.isEllipsis }
+            if (i >= 0 && i != args.lastIndex) throw ScriptError(args[i].pos, "ellipsis argument must be last")
+        }
+    }
+
+    /**
+     * Parse argument declaration, used in lambda (and later in fn too)
+     * @return declaration or null if there is no valid list of arguments
+     */
+    private fun parseArgsDeclaration(cc: CompilerContext): ArgsDeclaration? {
+        val result = mutableListOf<ArgVar>()
+        var endTokenType: Token.Type? = null
+        val startPos = cc.savePos()
+
+        while (endTokenType == null) {
+            val t = cc.next()
+            when (t.type) {
+                Token.Type.NEWLINE -> {}
+                Token.Type.ID -> {
+                    var defaultValue: Statement? = null
+                    cc.ifNextIs(Token.Type.ASSIGN) {
+                        defaultValue = parseExpression(cc)
+                    }
+                    // type information
+                    val typeInfo = parseTypeDeclaration(cc)
+                    val isEllipsis = cc.skipTokenOfType(Token.Type.ELLIPSIS, isOptional = true)
+                    result += ArgVar(t.value, typeInfo, t.pos, isEllipsis, defaultValue)
+
+                    // important: valid argument list continues with ',' and ends with '->' or ')'
+                    // otherwise it is not an argument list:
+                    when (val tt = cc.next().type) {
+                        Token.Type.RPAREN -> {
+                            // end of arguments
+                            endTokenType = tt
+                        }
+
+                        Token.Type.ARROW -> {
+                            // end of arguments too
+                            endTokenType = tt
+                        }
+
+                        Token.Type.COMMA -> {
+                            // next argument, OK
+                        }
+
+                        else -> {
+                            // this is not a valid list of arguments:
+                            cc.restorePos(startPos) // for the current
+                            return null
+                        }
+                    }
+                }
+
+                else -> {
+                    // if we get here. there os also no valid list of arguments:
+                    cc.restorePos(startPos)
+                    return null
+                }
+            }
+        }
+        // arg list is valid:
+        checkNotNull(endTokenType)
+        return ArgsDeclaration(result, endTokenType)
+    }
+
+    private fun parseTypeDeclaration(cc: CompilerContext): TypeDecl {
+        val result = TypeDecl.Obj
+        cc.ifNextIs(Token.Type.COLON) {
+            TODO("parse type declaration here")
+        }
+        return result
+    }
+
     private fun parseArgs(cc: CompilerContext): List<ParsedArgument> {
         val args = mutableListOf<ParsedArgument>()
         do {
@@ -837,16 +956,19 @@ class Compiler(
         }
     }
 
-    private fun parseBlock(tokens: CompilerContext): Statement {
+    private fun parseBlock(cc: CompilerContext, skipLeadingBrace: Boolean = false): Statement {
-        val t = tokens.next()
+        val startPos = cc.currentPos()
-        if (t.type != Token.Type.LBRACE)
+        if( !skipLeadingBrace ) {
-            throw ScriptError(t.pos, "Expected block body start: {")
+            val t = cc.next()
-        val block = parseScript(t.pos, tokens)
+            if (t.type != Token.Type.LBRACE)
-        return statement(t.pos) {
+                throw ScriptError(t.pos, "Expected block body start: {")
+        }
+        val block = parseScript(startPos, cc)
+        return statement(startPos) {
             // block run on inner context:
-            block.execute(it.copy(t.pos))
+            block.execute(it.copy(startPos))
         }.also {
-            val t1 = tokens.next()
+            val t1 = cc.next()
             if (t1.type != Token.Type.RBRACE)
                 throw ScriptError(t1.pos, "unbalanced braces: expected block body end: }")
         }
@@ -870,7 +992,7 @@ class Compiler(
             }
         }
 
-        val initialExpression = if (setNull) null else parseStatement(tokens)
+        val initialExpression = if (setNull) null else parseExpression(tokens)
             ?: throw ScriptError(eqToken.pos, "Expected initializer expression")
 
         return statement(nameToken.pos) { context ->

library/src/commonMain/kotlin/net/sergeych/ling/CompilerContext.kt

@@ -1,8 +1,18 @@
 package net.sergeych.lyng
 
-internal class CompilerContext(val tokens: List<Token>) : ListIterator<Token> by tokens.listIterator() {
+internal class CompilerContext(val tokens: List<Token>) {
     val labels = mutableSetOf<String>()
 
+    var currentIndex = 0
+
+    fun hasNext() = currentIndex < tokens.size
+    fun hasPrevious() = currentIndex > 0
+    fun next() = tokens.getOrElse(currentIndex) { throw IllegalStateException("No next token") }.also { currentIndex++ }
+    fun previous() = if( !hasPrevious() ) throw IllegalStateException("No previous token") else tokens[--currentIndex]
+
+    fun  savePos() = currentIndex
+    fun restorePos(pos: Int) { currentIndex = pos }
+
     fun ensureLabelIsValid(pos: Pos, label: String) {
         if (label !in labels)
             throw ScriptError(pos, "Undefined label '$label'")

library/src/commonMain/kotlin/net/sergeych/ling/Parser.kt

@@ -83,6 +83,11 @@ private class Parser(fromPos: Pos) {
                         Token("-", from, Token.Type.MINUSASSIGN)
                     }
 
+                    '>' -> {
+                        pos.advance()
+                        Token("->", from, Token.Type.ARROW)
+                    }
+
                     else -> Token("-", from, Token.Type.MINUS)
                 }
             }

library/src/commonMain/kotlin/net/sergeych/ling/Script.kt

@@ -60,7 +60,8 @@ class Script(
             addConst("Char", ObjChar.type)
             addConst("List", ObjList.type)
             addConst("Range", ObjRange.type)
-
+            @Suppress("RemoveRedundantQualifierName")
+            addConst("Callable", Statement.type)
             // interfaces
             addConst("Iterable", ObjIterable)
             addConst("Array", ObjArray)

library/src/commonMain/kotlin/net/sergeych/ling/TypeDecl.kt

@@ -0,0 +1,5 @@
+package net.sergeych.ling
+
+sealed class TypeDecl {
+    object Obj : TypeDecl()
+}

library/src/commonMain/kotlin/net/sergeych/ling/statements.kt

@@ -15,6 +15,8 @@ abstract class Statement(
     val returnType: ObjType = ObjType.Any
 ) : Obj() {
 
+    override val objClass: ObjClass = type
+
     abstract val pos: Pos
     abstract suspend fun execute(context: Context): Obj
 
@@ -28,6 +30,10 @@ abstract class Statement(
 
     override fun toString(): String = "Callable@${this.hashCode()}"
 
+    companion object {
+        val type = ObjClass("Callable")
+    }
+
 }
 
 fun Statement.raise(text: String): Nothing {

library/src/commonTest/kotlin/ScriptTest.kt

@@ -1040,14 +1040,75 @@ class ScriptTest {
     }
 
     @Test
-    fun testLambda1() = runTest {
+    fun testLambdaWithIt1() = runTest {
-        val l = eval("""
+        eval("""
             val x = {
-                122
+                it + "!" 
             }
-            x
+            val y = if( 4 < 3 ) "NG" else "OK"
+            assert( x::class == Callable)
+            assert( x is Callable)
+            assert(y == "OK")
+            assert( x("hello") == "hello!")
         """.trimIndent())
-        println(l)
     }
 
+    @Test
+    fun testLambdaWithIt2() = runTest {
+        eval("""
+            val x = {
+                assert(it == void)
+            }
+            assert( x() == void)
+        """.trimIndent())
+    }
+
+    @Test
+    fun testLambdaWithIt3() = runTest {
+        eval("""
+            val x = {
+                assert( it == [1,2,"end"])
+            }
+            println("0----")
+            assert( x(1, 2, "end") == void)
+        """.trimIndent())
+    }
+
+    @Test
+    fun testLambdaWithArgs() = runTest {
+        eval("""
+            val x = { x, y, z ->
+                assert( [x, y, z] == [1,2,"end"])
+            }
+            assert( x(1, 2, "end") == void)
+        """.trimIndent())
+    }
+
+    @Test
+    fun testLambdaWithArgsEllipsis() = runTest {
+        eval("""
+            val x = { x, y... ->
+                println("-- y=",y)
+                println(":: "+y::class)
+                assert( [x, ...y] == [1,2,"end"])
+            }
+            assert( x(1, 2, "end") == void)
+            assert( x(1, ...[2, "end"]) == void)
+        """.trimIndent())
+    }
+
+    @Test
+    fun testLambdaWithBadArgs() = runTest {
+        assertFails {
+            eval(
+                """
+            val x = { x, y ->
+                void
+            }
+            assert( x(1, 2) == void)
+            assert( x(1, ...[2, "end"]) == void)
+        """.trimIndent()
+            )
+        }
+    }
 }