This project has moved. For the latest updates, please go here.

Extention Method Support

Jan 17, 2015 at 2:17 AM
How completely are extension methods supported in the current build? In particular, has anyone used this project to compile expressions using linq extension methods? I've been experimenting with some simple examples and not having much luck with them, but i'm not convinced I have the types and symbols in the type registry configured correctly.
Developer
May 16, 2015 at 12:07 AM
Sorry, but extension methods are not currently supported.

Providing support for lambdas and type inference to support such syntax is proving to be an immense challenge. I would like to support extension methods, and simple extension methods that accept non-lambda parameters or have no arguments might be implemented sometime, but LINQ extension methods such as Where will take a while to support, and currently I am rather busy with my daily job.
Sep 5, 2015 at 5:15 PM
I have implemented a very general method call type-inference routine to extend expression tree capabilities. It has been designed to follow the C# type-inference rules specified in the standard. To the best of my knowledge it obeys all the C# method call rules, including overload resolution. I was forced to implement it for my own visual programming language that uses expression trees extensively and I couldn't find any other solution like it on the internet.

The signature is very simple:
static Expression BuildCall(Expression instance, IEnumerable<MethodInfo> methods, params Expression[] arguments)
Basically it takes an instance expression, a set of method candidates (obtained with Type.GetMethods) and a list of arguments, and returns a fully instantiated expression node that represents a call to the method including any necessary type and overload resolution (it also can deal with methods that have variable number of arguments, i.e. params).

I spent quite some time developing this method so I completely understand your pain. I would be more than happy to contribute this code freely to ExpressionEvaluator, as I'm planning to use your library to implement the scripting part of my language soon, so it will be great to have this added in!

If you want I can either provide this as a pull-request adding an independent helper class or I can just send you the code directly.

Cheers,
Developer
Sep 6, 2015 at 6:12 AM

That's amazing, a pull request would be great!

looking forward to it

Sep 6, 2015 at 8:48 AM
Cool, I've forked csharpeval and I'll start adding my helper classes to a MethodResolution branch.

While looking into the existing code base I've realized you had a MethodResolution.cs file with a partial implementation. I'll try to see if I can merge my stuff with yours, but anyway, you'll get the chance to review later. Also, I'll use this as a chance to double-check how thorough my current method resolution is.

P.S.: You should also definitely look into Roslyn as an alternative parser; they will support top-level expressions and REPL in their next release (1.1 - available as pre-release on NuGet) and converting from a syntax tree to an expression tree looks like a very doable project.

Pros: full parser support for C# 6.0 expressions "for free"
Cons: i guess it is not as lightweight as Antlr; anyway, it doesn't look too bad
Sep 6, 2015 at 10:28 AM
Ah, one important limitation of my method resolution implementation is that it doesn't deal with lambdas at all. I guess that is the single feature that I have never used for my own scripting purposes. I might consider adding it this time, although judging by Eric Lippert's discussion this will turn out to be non-trivial...
Sep 6, 2015 at 11:30 AM
Ok, the pull request is up:
https://github.com/RupertAvery/csharpeval/pull/7

Looking forward to improving the type resolution and eventually query expression / extension method evaluation for csharpeval.
For the longest time I've been wanting a complete C# expression tree parser...

Cheers,

P.S.: While looking at the code base I noticed a couple of spelling typos in the main API. Would u like me to submit pull requests for these as well?
Developer
Sep 6, 2015 at 12:01 PM
Thanks a lot for this!

I rarely get contributions to the project, and Extension methods is a huge addition.

I was hoping you had some insight into lambdas. I was holding off implementing Extension methods since the main reason people wanted support for those was to be able to use LINQ extension methods in their expressions.

Yes, I've looked at Eric Lippert's discussion about lambdas. There might be some changes to how Expression Evaluator works internally to support it. But everytime I try I get stuck on how to implement type inference. I could probably allow explicit typing just to get it to work, but it would be very tedious to use.

Please do submit pull requests as you see fit.

At this point I would like to apologize for the sorry state of the code, I tend to try to get things to work and try to clean up later.

Roslyn is Roslyn, I guess. I haven't tried it yet, but having csharpeval allows me to do a lot of things like injecting scope and "fake dynamics". And I have definitely learned a lot about C# the language itself in the process of building it.

Again, thanks a lot for your contribution! and thank you for finding it useful.


Sep 6, 2015 at 2:10 PM
Don't worry, we're on the same page regarding injection of scope, query evaluation and extension methods. I also always start with a rough working draft of functionality and then polish the API into shape, so this is all good.

My hope when I wrote this type inference routine was that it would be general enough that you could add lambda inference on top of it by basically creating a type inference dependency graph (e.g. inferring type A depends on inferring type B, and so on). I also already have code to do "topological sorting" (a.k.a. dependency-resolution) on directed graphs so I kind of have the hope that lambdas will fall-off naturally from these two pieces of algorithm (at least for the kind of non-recursive lambdas we are usually interested in for query expressions).

Extension methods is of course extremely interesting but could also be done incrementally (i.e. first-order solution just concatenate all methods in Enumerable as possible candidates; only after that think about user-defined extension methods).

My hope with Roslyn is that you could use their C# syntax tree to do all of the parser/type-inference heavy lifting and then just convert to expression tree with a visitor. I'll have to read and play with it a bit more to figure out exactly what can be done, especially with regards to the scope injection part that is also very critical for my own work. Last case scenario you can also reuse parts from their reference implementation, now that everything is open-source (yay!).

I'll keep you posted if I make more progress and you may see more pull requests come up.
Thanks for the fast reply!
Cheers,
Sep 6, 2015 at 2:28 PM
By the way, this issue@roslyn.github looks very relevant: apparently we're not the only ones considering expression tree generation from scripts useful (go figure ;-).
Sep 8, 2015 at 11:19 AM
Ok, I started looking a bit more into how to resolve methods with lambdas and it's going to get a bit tricky as expected...

First of all, I don't think we can do the kind of naive resolution of the expression tree where arguments are evaluated before method calls, because now evaluating a lambda argument may depend on partially evaluating its sibling arguments. I could think of two general approaches:

1) Instead of assuming a fixed resolution order based on the call stack, we build a dependency graph where evaluation of each expression node can have arbitrary type resolution dependencies to other nodes. The current resolution order from arguments to method is simply encoded here as links in the graph, but it gives you the ability to introduce additional links between arguments to express more complex relationships such as lambdas.

2) Instead of using directly ExpressionTree nodes to build up the syntax tree, we use an AST type that can keep the syntax representation while delaying evaluation of the semantic model. I noticed that right now you evaluate lambda parameters to System.Object by default, but this does not work because you can't really interpret the syntax at all until you know the exact parameter type (i.e. if you do "x => x * 2" you can't even call Expression.Multiply). You need to delay parsing the body of the lambda until you know the type of the parameters. The limitation of expression trees for this is that they can't represent pure syntax, they always simultaneously encode semantics.

I don't know if you have an opinion on what would be the best approach (maybe we will even need both). I'll give it a try and let you know what happens.
Developer
Sep 8, 2015 at 1:39 PM
We would need to do a major rewrite of the parser to lift the LINQ Expression out of Antlr and just return an AST. I was planning to have some sort of delayed resolution for lambdas where the parameters have placeholder types and are re-evaluated when more information is known but my current implementation got in the way of it.

Sep 8, 2015 at 5:24 PM
A full-blown AST and delayed resolution is definitely the more general solution. An intermediate solution that should work for most LINQ operators is to simply keep a memory of currently resolved types and resolve each argument following the usual calling order.

For example, if you consider the Select method:
public static IEnumerable<TResult> Select<TSource, TResult>(this IEnumerable<TSource, TResult> source, Func<TSource, TResult> selector);
and you call it like this:
Enumerable.Select(new[] { 1, 2, 3 }, x => x  * 2);
Then evaluating the first argument (the array) will resolve the first type argument (TSource), which incidentally is also the type of the lambda input. If this is known, then you can resolve the lambda body as soon as it appears. The assumption here is that the first arguments in the list will resolve types that are required to interpret the lambda inputs that show up later. Given the structure of LINQ, where lambda parameters always come later in the argument list I think this strategy would work for pretty much all methods.

In this way you wouldn't need to have an AST or delayed evaluation of lambdas. You would only need to update the resolved types as you go along and hopefully when you get to the lambda you would already know everything you need. If you don't know, you can throw an exception. I'm pretty confident this could solve 90% of people's use cases.

The only other required change is that type resolution would have to be called at the level of Antlr (as it is now) but we would have to know when a method call is beginning to be resolved, before its arguments are, so we can start evaluating the method types.

Sounds doable, but I will have to make myself more familiar with Antlr before I can say for sure. What do you think?
Developer
Sep 16, 2015 at 1:16 AM
I think that is the same approach I had in mind, but actually implementing it was a bit more difficult than what I had thought out.