This project has moved and is read-only. For the latest updates, please go here.

New Antlr Parser

Feb 23, 2014 at 9:41 AM
Due to the growing interest in Expression Evaluator and the increasingly complex situations it may come across, I am considering implementing a new parser or replacement parser for the library using the Antlr runtime. I have the option of either including the Antlr source code or a binary. There will be a lot of changes, I may even have to rewrite the expression tree builder (it's a mess anyway).

The main API should remain the same. The focus on single-line expressions, the ability to compile to a function and compile with a scope parameter are what makes ExpressionEvaluator useful as a programming tool.

Hopefully this improvement will not hinder performance, but the stability and robustness of the Antlr parser should be a great boon to everyone.

If anyone has any objections or observations, (and tips!) on using Antlr, I would be very happy to hear about them.
Feb 25, 2014 at 12:29 PM
Edited Feb 25, 2014 at 12:31 PM
Hello,
Thanks for all these Fish!
I meant it, there is something like this in Java , JEXL [http://commons.apache.org/proper/commons-jexl/]
but till this point, nothing at all like this was here for .NET enthusiasts!
Come to think about it, it forced me to write my own UI framework (and a virtual machine system) using Java.
BUT, thanks to you, I can now write it back in C# - at least have a startup!

I humbly request you to move this into NEXL : .NET Expression Language just like JEXL?
In case you are up for it, I am also game.


However, there are interesting observations if at all you proceed in that direction.
If you look at JEXL, it has its own expression language.
The parser they used is https://javacc.java.net/doc/JJTree.html

That translates into, you can script in JEXL, if, while, for, else, continue, all are there.

I suppose that wont pose a problem for you, in general, because you can in memory create a function and call it using Code Dom.
The next issue is implementing custom literals.
$xxx is a literal, so is @xx .
And, supporting list and hash types.
In the end, we need to have support for a context, that is :-
$x + 2 should have a public context with variables, where $x is the key, and you evaluate on that.
[http://commons.apache.org/proper/commons-jexl/reference/examples.html]
The source will not be hard for .NET at all -- much easier than in Java.

Then, we can have a proper expression language to adapt to any business logic, and almost code-less applications!
Tactically I am looking at a semi procedural - mostly functional and declarative style, which suits test automation.
An expression language is necessity there, NOT a luxury.
Finally, whether or not you pick it up, many many hearty thanks to you for the work you have done anyways.
Mar 11, 2014 at 10:32 AM
Edited Mar 11, 2014 at 10:33 AM
Hi nmondal,

I haven't yet really looked at JEXL. When someone uses the same name as a Java framework, it usually means that they port the existing code over. This project really started as way of learning LINQ Expressions, and I don't want to start from a large existing codebase. I'm not sure what the current differences are between JEXL and ExpressionEvaluator.

How have you used ExpressionEvaluator and what challenges did you have utilizing it?

ANTLR Updates

On topic with ANTLR, I am using the excellent grammar file from http://antlrcsharp.codeplex.com as a starting point for the new version. Right now I have simple expressions and simple external variables working. Member access (i.e. methods and properties are not implemented yet), I might start a new project for this, to keep things clean. The new version should eventually support lambdas, the new operator, anonymous type instantiation, all the C# 3.0 and above goodies that make things easy to use.

I might also be possible to incorporate this into the current project, where it will work seamlessly since in the end both generate an Expression tree. Then it might be possible to conditionally compile, or have some sort of switch to choose between parsers (stable?? vs accurate??).
Mar 11, 2014 at 2:10 PM
My use of expression evaluator was to port my framework ( a tiny virtual machine infra - for test automation) to .NET.
The issues are, at a glance :-
  1. Here, we can not have void expressions -- which are required.
  2. We can not have -- context variables, get and set.
  3. We can not define $variable as identifier, or @xxx as identifiers --> need to support custom identifers .
  4. Namespacing of custom functions, such that xxx:foo() points to some class or function.
Again, your code is the ONLY alternative to JEXL in .NET, I urge you to please go through the JEXL syntax -- and get it checked once!
If you ( even I wont mind participating ) can do it -- that would be of great help writing script based languages over .NET!

As you are already into ANTLR, some of these changes should be trivial in the .g file!
For example - making void expressions returning null.
Mar 11, 2014 at 4:09 PM
would like to keep the c# standard for the language, so I don't think custom identifiers or namespacing with JEXL syntax is in my plans. I think both are already possible in the existing framework through the TypeRegistry.

Not sure what you mean by context variables. void expressions can be called in the latest commits but through a different calling mechanism, not Eval ()
Mar 11, 2014 at 4:21 PM
Actually it is not, with that $foo , it would throw parse exception and all.
All I am saying - that as you are already having a grammar, the support of $ and @ in front of identifiers should be cake walk.
And they have meaning too!
Imagine a script based language where you do not know what variable is local and what is global.
In the one I have created, $xxx is local, while $$xxx is global variable.
@xxx is used in anonymous function call.
Can you please support $, and @ with the identifiers?
Even if you do not I do not see much problem - as long as you make the source open, :D in which case I can modify!

The context is as such:-
http://commons.apache.org/proper/commons-jexl/apidocs/org/apache/commons/jexl2/JexlContext.html
The idea that when you can not find an identifier - like - thing that has no meaning, you pass it to the context - asking if it is defined as a variable already.
If it has , then you use get the value and substitute. The set function however, lets you set the variable using = operator.
"xxx = 'foo'; bar = 'yyy'" would call set with ( "xxx", "foo" ) and ( "bar", "yyy")


Thanks again for all the help, appreciated!
Mar 11, 2014 at 6:07 PM
You are free to create a fork at anytime. :-) That way there could be a version that is jexl specific perhaps. You might want to wait until the antlr version is checked in and more or less stable.
Mar 12, 2014 at 6:51 AM
I have checked in the Antlr-based parser codes into a new branch - AntlrParser. Feel free to take a look at it. It is largely untested, but most basic stuff should work. One problem I did have with Antlr is the Scope feature. With the old parser, it was relatively easy to inject the scope and member accessor tokens into the token stream, but with Antlr I can't seem to do this at all.

My workaround was to have a custom identifier $scope that would bind to the scope parameter expression. I have quite a lot of code in place that used the old method of defaulting to scope if the identifier wasn't found. I guess I shouldn't have done it that way, but now I'm faced with updating all the strings that access scope. It shouldn't be too hard.
Mar 12, 2014 at 8:18 AM
Hi Rupert,
That is why I was suggesting to implement the get/set/has feature using antlr.
Then you can inject scope, member access and all.
Mar 15, 2014 at 5:51 AM
I have found the correct way to simulate the original behavior of scope in version 1.0.X of Expression Evaluator. This makes version 2.0 closer to a release as it can be substituted into 1.0.X code with minimal supporting changes and still have the same behavior.
Mar 16, 2014 at 2:56 PM
Edited Mar 16, 2014 at 3:05 PM
nmondal,

My issue was not with being unable to inject scope, or member access. These have already been in place since the start of the project. I have a use-case in place where all unresolved symbols would be bound to a scope parameter, similar to AngularJS. I have resolved that issue.

As for your issues:

Void expressions are now supported. Although I initially added Call() and ScopeCompileCall() as alternatives to Eval() and ScopeCompile(), I have also added code that wraps the expression into a code block that returns null if the expression returns void. That way Eval() and ScopeCompile() will work with void expressions. You gave me that idea so thanks to you for that.

This support is currently only available through the latest source. The download is outdated.

Context variables are already there through the TypeRegistry. You have to go through an object though i.e. container.x. It is impossible to set the value of an object that is passed by value (the variable outside the expression will not be updated)

Context variables are also supported through ScopeCompile().

Get and set are supported with TypeRegistry this way: register the container, then container.x = value will set and container.x will get. No need for get/set special functions.

If you are using scope, then no need to register a container. x = value will set and x will get. I might consider has.

You can actually create new variables on the scope context in your expression on the fly if you pass a dynamic (ExpandoObject) scope. I will put up an some sample code regarding dynamics.

Custom identifiers ($varname/@@varname) - I can consider this, but since ScopeCompile and TypeRegistry allow you to define identifiers that way, I'm not sure what makes this different.

Namespacing of custom functions, such that xxx:foo() points to some class or function. Again, this is already supported through TypeRegistry.

Again, this is a C# Expression Evaluator, JEXL syntax is not C# and I don't want to support two languages.

I have updated the documentation.

Usage

ScopeCompile
Mar 16, 2014 at 3:07 PM
Thanks Rupert,
I would try my hand in the newer one -- and keep you posted!
Thanks a ton again!
Mar 25, 2014 at 6:46 PM
Hi Rupert,
I tried to build the antler branch -- I am unable to compile it I am afraid.

------ Build started: Project: ExpressionEvaluator, Configuration: Debug Any CPU ------
CSC : warning CS2002: Source file 'E:\RnD\csharpeval\obj\Debug\ExprEvalParser.cs' specified multiple times
CSC : warning CS2002: Source file 'E:\RnD\csharpeval\obj\Debug\ExprEvalLexer.cs' specified multiple times
E:\RnD\csharpeval\obj\Debug\ExprEvalLexer.cs(29,7,29,33): warning CS0105: The using directive for 'System.Collections.Generic' appeared previously in this namespace
E:\RnD\csharpeval\ExpressionHelper.cs(47,17,47,30): warning CS0219: The variable 'isRuntimeType' is assigned but its value is never used
E:\RnD\csharpeval\ExpressionHelper.cs(91,17,91,30): warning CS0219: The variable 'isRuntimeType' is assigned but its value is never used
E:\RnD\csharpeval\ExpressionHelper.cs(212,21,212,22): warning CS0219: The variable 'x' is assigned but its value is never used
E:\RnD\csharpeval\ExpressionHelper.cs(238,17,238,30): warning CS0219: The variable 'isRuntimeType' is assigned but its value is never used
E:\RnD\csharpeval\ExpressionHelper.cs(295,17,295,30): warning CS0219: The variable 'isRuntimeType' is assigned but its value is never used
E:\RnD\csharpeval\obj\Debug\ExprEvalLexer.cs(36,2,36,21): warning CS3021: 'ExpressionEvaluator.ExprEvalLexer' does not need a CLSCompliant attribute because the assembly does not have a CLSCompliant attribute
E:\RnD\csharpeval\obj\Debug\ExprEvalParser.cs(34,2,34,21): warning CS3021: 'ExpressionEvaluator.ExprEvalParser' does not need a CLSCompliant attribute because the assembly does not have a CLSCompliant attribute
E:\RnD\csharpeval\ExprEval.g3.parser.cs(32,13,32,20): warning CS0162: Unreachable code detected
E:\RnD\csharpeval\Operators\MethodResolution.cs(241,13,241,19): warning CS0162: Unreachable code detected
E:\RnD\csharpeval\Operators\MethodResolution.cs(351,28,351,52): error CS0161: 'ExpressionEvaluator.Operators.MethodResolution.IsBetterConversionTarget(System.Type, System.Type)': not all code paths return a value
E:\RnD\csharpeval\Operators\MethodResolution.cs(395,17,395,31): warning CS0219: The variable 'implicitPoints' is assigned but its value is never used
E:\RnD\csharpeval\Operators\MethodResolution.cs(395,37,395,47): warning CS0219: The variable 'typePoints' is assigned but its value is never used
E:\RnD\csharpeval\Operators\MethodResolution.cs(386,34,386,57): error CS0161: 'ExpressionEvaluator.Operators.MethodResolution.GetBetterFunctionMember(System.Reflection.MethodInfo, System.Reflection.MethodInfo, System.Collections.Generic.List<System.Linq.Expressions.Expression>)': not all code paths return a value
E:\RnD\csharpeval\Operators\MethodResolution.cs(430,34,430,52): error CS0161: 'ExpressionEvaluator.Operators.MethodResolution.OverloadResolution(System.Collections.Generic.List<System.Reflection.MemberInfo>, System.Collections.Generic.List<System.Linq.Expressions.Expression>)': not all code paths return a value
E:\RnD\csharpeval\Operators\MethodResolution.cs(528,17,528,18): warning CS0642: Possible mistaken empty statement
E:\RnD\csharpeval\Parser.cs(785,13,785,19): warning CS0162: Unreachable code detected
E:\RnD\csharpeval\Operators\OperatorCustomExpressions.cs(79,31,79,92): error CS1502: The best overloaded method match for 'ExpressionEvaluator.Operators.MethodResolution.GetApplicableMembers(System.Type, ExpressionEvaluator.TypeOrGeneric, System.Collections.Generic.List<System.Linq.Expressions.Expression>)' has some invalid arguments
E:\RnD\csharpeval\Operators\OperatorCustomExpressions.cs(79,75,79,85): error CS1503: Argument 2: cannot convert from 'string' to 'ExpressionEvaluator.TypeOrGeneric'
------ Build started: Project: Tests, Configuration: Debug Any CPU ------
CSC : error CS0006: Metadata file 'E:\RnD\csharpeval\bin\Debug\ExpressionEvaluator.dll' could not be found
========== Build: 0 succeeded, 2 failed, 0 up-to-date, 0 skipped ==========

Error comes for this line.
Do I need to change something somewhere?

__var mis = MethodResolution.GetApplicableMembers(type, membername, args);
__
Mar 25, 2014 at 11:12 PM
Edited Apr 10, 2014 at 5:35 AM
Yes. Unfortunately im using the antlr branch as a repo between home and work and I may have checked in some non working code. Iml working toward c# specification compliance, so expect a lot of things going on with that branch. Sorry about it
Apr 10, 2014 at 5:51 AM
Hi,

I have fixed the compilation issue. I have also rolled back the method handling code to the old (1.0.4) version, so everything in that regard should be working as it was before. I have also added support for if-then-else and switch statements. In order to compile these statements, you need to set the ExpressionType on the CompiledExpression object to CompiledExpressionType.StatementList
var a = new ClassA() { x = 1 };
var t = new TypeRegistry();
t.RegisterSymbol("a", a);
var p = new CompiledExpression { StringToParse = "if(a.x == 1) a.y = 2; else { a.y = 3; } a.z = a.y;", TypeRegistry = t };
p.ExpressionType = CompiledExpressionType.StatementList;
var f = p.Eval();
Assert.AreEqual(a.y, 2);
Assert.AreEqual(a.y, a.z);
The expression is a if then else with a multi-statement, and could be written out like this:
if(a.x == 1) 
   a.y = 2; 
else { 
   a.y = 3; 
} 

a.z = a.y;
There are some caveats with the switch statement, so I'd advise you don't use it unless you know how it works.
Apr 16, 2014 at 8:23 PM
Thank you very much Rupert!
I believe we have a winner here,

var a = new ClassA();
        var t = new TypeRegistry();
        t.RegisterSymbol("a", a);
        t.RegisterSymbol("Console", typeof(Console));

        var p = new CompiledExpression { 
            StringToParse = "Console.WriteLine(\"foo bar\");" , 
            TypeRegistry =t };

        p.ExpressionType = CompiledExpressionType.StatementList;
        p.Eval ();
This code perfectly runs - as expected!
Great Job!
Even voids seems to be got supported!


BUT, the simple assignment like :-

var p = new CompiledExpression {
            StringToParse = "var i = 0;" , 
            TypeRegistry =t };

Throws exception!

Unhandled Exception:
System.ArgumentNullException: Argument cannot be null.
Parameter name: expressions
at System.Linq.Expressions.Expression.RequiresCanRead (System.Linq.Expressions.Expression expression, System.String paramName) [0x0009d] in /private/tmp/source/bockbuild-mono-3.2.6/profiles/mono-mac-xamarin/build-root/mono-3.2.6/mcs/class/dlr/Runtime/Microsoft.Scripting.Core/Ast/Expression.cs:344
at System.Linq.Expressions.Expression.RequiresCanRead (IEnumerable1 items, System.String paramName) [0x0001a] in /private/tmp/source/bockbuild-mono-3.2.6/profiles/mono-mac-xamarin/build-root/mono-3.2.6/mcs/class/dlr/Runtime/Microsoft.Scripting.Core/Ast/Expression.cs:354
at System.Linq.Expressions.Expression.Block (IEnumerable
1 variables, IEnumerable1 expressions) [0x0001d] in /private/tmp/source/bockbuild-mono-3.2.6/profiles/mono-mac-xamarin/build-root/mono-3.2.6/mcs/class/dlr/Runtime/Microsoft.Scripting.Core/Ast/BlockExpression.cs:749
at System.Linq.Expressions.Expression.Block (IEnumerable
1 expressions) [0x00000] in /private/tmp/source/bockbuild-mono-3.2.6/profiles/mono-mac-xamarin/build-root/mono-3.2.6/mcs/class/dlr/Runtime/Microsoft.Scripting.Core/Ast/BlockExpression.cs:694
at ExpressionEvaluator.AntlrParser.Parse (System.Linq.Expressions.Expression scope, Boolean isCall) [0x00000] in <filename unknown>:0

Can you please help me out here?
Ideally what I want to do is :-
so that var i = 0 stuff, i should get into type registry as a variable. Can I do it?
Apr 19, 2014 at 7:42 PM
Hi Rupert,
How can I set a variable -- or create a variable in the expression?
Do we have support for it?
Apr 20, 2014 at 8:08 PM
Hi Rupert,

Thank you for making Expression Evaluator. This is oustanding stuff you provide here. I'm considering using it as part of a project I'm working on.

I faced the same interrogation than nmondal about local and global variable creations but I figured that I could pre process my expressions with regular expressions so that $var becomes context.var where context is an ExpandoObject provided in the TypeRegistry.

The only issue that it raises is that the type of my variables are Object at parsing time and expressions such as context.i++ fails to compile. Do you see a workaround for this issue?

Loops support will make Expression Evaluator truly awesome(r) :)
Apr 22, 2014 at 11:56 PM

Yes. Don't use ++ :-).

You could use context.i = context.i + 1 and it will implicitly cast the right hand context.i to type int before adding.

It's how c# works unfortunately.

You coule implement the ++ operator for object types but I havent done this yet and have no idea if it would work

Apr 23, 2014 at 12:00 AM

Hmmm. Just read up. You can't overload ++ on basic types. Oh well.

Apr 23, 2014 at 12:17 AM
Edited Apr 23, 2014 at 12:22 AM
Hi nmondal, I will look into supporting local variables.

But I'm not sure what you need. You want to declare a local var and assign a value to it, then get that value in the type registry?

Maybe a quick pseudo code of what you are trying to do will help me figure out what you need
Apr 24, 2014 at 6:28 PM
Edited Apr 24, 2014 at 6:32 PM
In addition to avoiding ++ and -- also avoid assignments like += etc.

I would also like the ability to allow local variables within an expression - for now I simply define a bunch of string, bool, double etc. variables and arrays in my scope class and let script writers use those.

If time permits I will be looking at how tough it is to include local variables and attempt to implement for Rupert to comment on
Apr 25, 2014 at 12:23 PM
Edited Apr 25, 2014 at 12:31 PM
Hi everyone.

I have implemented initial support for local variables.

The following works pretty well. It's in the main program if you want to test.
            object obj = new objHolder() { result = false, value = NumEnum.Two };

            registry.RegisterSymbol("obj", obj);
            registry.RegisterType("objHolder", typeof(objHolder));
            registry.RegisterDefaultTypes();

            var cc = new CompiledExpression() { StringToParse = "var x = new objHolder(); x.number = 3; x.number++; var varname = 23; varname++; obj.number = varname -  x.number;", TypeRegistry = registry };
            cc.ExpressionType = CompiledExpressionType.StatementList;
            var result = cc.Eval();
The final value for obj.number is 20 as expected.

I had to change how Statements are processed though but this should not have much impact. I hope that I didn't break anything. I will try to discuss how I implemented it, since I feel it was a bit of a hack to get things done.

It's all experimental as usual and so there may be some missing implementation but for general use this should work.

Please get the latest codes.

I fixed the post ++ -- operators. Apparently Expression.Increment is purely functional and you need to assign the result back to the value. Still won't help with ++ on objects.

However, come to think of it, if it is a DynamicObject then the DLR should handle it and I should check for the type on Increment operator.
Apr 26, 2014 at 1:50 AM
Hi all,

Please be advised I am doing some refactoring, mostly just moving some stuff around into folders. I hope you don't have any changes that will be affected. I am also getting rid of the 1.x legacy code that I was keeping around in the AntlrParser branch for reference.

ExpressionEvaluator - Where the external facing API stuff stays
ExpressionEvaluator.Parser - Where most of the core stuff will go
ExpressionEvaluator.Parser..Expressions - Where container classes that will be used to encapsulate complex expression types will go

I also changed some classes to internal
Apr 29, 2014 at 6:12 AM
Edited Apr 29, 2014 at 12:28 PM
Hi all,

I have some working for/while/do loop code. it supports break and continue and should also support nested loops. breaks and continues are treated as gotos in LINQ Expressions so you need to manage the labels. I am using a Stack implementation to take care of nesting. the stacking isn't fully tested and using break in unexpected places might throw some errors. Error handling is one of the things I will eventually have to work on, but for now please try the loop implementation.

There should be no restrictions on how it's used. You can declare local explicitly and implicitly typed variables or use an external value. You should be able to do multiple assignments and iterators such as
for(int i = 0, j = 0; somecondition; i++, j++)
Please see the unit tests for examples
Apr 29, 2014 at 6:30 AM
Great job Rupert! This is amazing progress!
On the other side of it -- are you using goto: inside as the loop implementation?
http://msdn.microsoft.com/en-us/library/13940fs2.aspx
I suggest it would be much better way of handling any lopp -- I used the same for my stuff.
If there are n levels of depth in the loop - the stack size would be n.
Again, great stuff Rupert!
Apr 29, 2014 at 12:28 PM
Thanks,

Yes I am using goto. it is the only way to do it in LINQ Expressions.

In LINQ there is no FOR, WHILE, DO or FOREACH construct, only a LOOP which basically takes a list of expressions and repeats once you get to the end, so you have to build the FOR/WHILE/DO around that. There is a BREAK, CONTINUE, GOTO contruct, but internally they are all GOTOs with different usages. You end up having to place labels strategically and make sure the loop body references the correct label whenever a break token is parsed, so I opted for a stack to handled nested cases. I don't think we should have problems with stack size.
Apr 29, 2014 at 4:04 PM
I now have ForEach working. It's implemented as a while loop basically with the iterator MoveNext. There's still a lot to test with dynamics and arrays but at least the pattern is there. Phew. I could use a cold beer right now.
Apr 29, 2014 at 4:23 PM
Rupert, if all of these works out I can sponsor you a whole bar. My treat!
Great work - anyway!
Apr 30, 2014 at 7:59 PM
I also owe Rupert a beer or three for all his hard work.

Nested loops, loops with if/else and more all work as expected
            var x = 0;
            for (int i = 0; i < 4; i++)
            {
                for (int j = 0; j < 3; j++)
                {
                    if (j != 1)
                        x++;
                }
            }
            x;
Awesome - you have created a really useful tool here.

Many, many thanks
May 1, 2014 at 7:04 AM
Hmm, that stray x there, is it mean to be used as a return value?

I'm currently trying to get the return keyword to work, so you can exit from within a loop or whenever.
May 1, 2014 at 5:31 PM
Yes - it's a return value.

A 'return' statement would be a nice addition.

Also - while I am asking - what are your thoughts about adding the 'break;' statement to switch/case? The reason for asking is that when writing scripts for EE to execute that are more than a couple of lines I use (and propose to tell others to use) the VS editor in a project that has any registered objects referenced and new'ed up (and of course EE referenced) so Intellisense works.

I understand double quotes are not used for literals (and have to be replaced with single quotes) but currently the other gotcha I run into is VS wants to see break terminating each case. Again, not a biggie but would be nice.

This is a great project - thanks!
May 1, 2014 at 6:42 PM

Double and single quotes are allowed for atring literals. For the switch case, I am working on making it work as expected. The break stuff was easy once I had done the same for loops, but getting return to work is my current challenge.

Also, switch is rather interesting in that it will allow any expression that prevents code from falling through to the next case. So an infinite loop is technically legal. It's probably an edge case though but that does mean a proper implementation would require code analysis.

May 3, 2014 at 2:13 AM
The switch statement should allow breaks now. EE will also check if a case block ends in a break (actually any goto expression, i.e. break, continue, goto, return but only break and continue actually works) and will throw an exception if it doesn't, probably not the most accurate implementation but it should do for most use cases.