jump to navigation

Prototyping With AnonymousTypes, CS-Script and JSON.Net – DataBuilder Part 3 November 14, 2010

Posted by ActiveEngine Sensei in .Net, ASP.Net, C#, CS-Script, DataBuilder, JSON.Net, New Techniques, Open Source.
Tags: , , , ,
4 comments

Sensei hit his head in the shower this morning and instead of the flux-capacitor he saw the following code that could be used for prototyping:

var jsonDataSet = AnonFactory.CreateList(10, new { FirstName = "",
                                 LastName = "",
                                 Age = 0,
                                 Salary = 45000})
                              .WhereAll()
                              .Have("FirstName", "Jim")
                              .BuildList()
                              .ToJSON();

If you have been following the series on DataBuilder, a utility that dynamically creates test data as JSON, you’ll recall one of the highlights was avoiding the need to create an additional Visual Studio project just so you could create test data.  DataBuilder can import your assemblies and with the use of NBuilder it creates test data for you.  You may wish to read those posts for some background.

But consider the scenarios where you do not have an application domain or a .Net assembly to test.  Should you stop?  For prototyping do you really need to compile, change, compile, change, show-to-your-user, repeat ad-nauseum?

Here’s a new concept:  Build data sets that you use in prototyping with Javascript before you break out Visual Studio.  If you can quickly build a tool set to communicate with your business users you’ll be much further ahead by avoiding the back-end client side impedance mismatch.  After all, many times the business user has the concept already hidden upstairs in their heads, why not help get that vision out in the open earlier and save yourself.

So, to be able to write the statement that Sensei saw in his vision we need to achieve to following goals:

  • Create an anonymous type as template for building a list
  • Create a fluent interface for chaining clauses similar to NBuilder
  • Serialize the list of anonymous types to JSON
  • Run this process in DataBuilder – in other words, go dynamic!

You may want to download the source and follow along.

AnonymousType as a Template

It turns out that Sensei has a tool in his bag of tricks, namely the Persistent Anonymous type.  As you are aware, anonymous types in C# have a limited scope, but the Persistent Anonymous type that Sensei discussed allows you to create a structure that mimics a standard anonymous type while being persist that structure beyond the scope of its creation.  The AnonymousType creator, Hugo Benocci created something really great.  Here’s what you can do from the get-go:

var anonType = AnonymousType(new {FirstName = "James",
                                         Middle = "T",
                                         LastName="Kirk"
                                   });

If you recall the examples used in the prior posts on data generation with DataBuilder, NBuilder supplied default values for you when you built an object. You then had to alter those values should you want something different.

With an AnonymousType we use a factory to create a list of AnonymousTypes like so:

var anonTypeList = AnonFactory.CreateListSizeOf(10,
                                  new {FirstName = "James",
                                         Middle = "T",
                                         LastName="Kirk"
                                   });

Nice and simple. If we didn’t have more goals to achieve, we could go home. As it is, on to the next step.

Creating a Fluent Interface for Chaining Clauses

We still need to provide variation in our data set.  NBuilder does a nice job of letting us use clauses like .WhereTheNext(5).Have() or WhereAll.Have() to introduce more structured variation in our data.  We want to achieve the same thing with the AnonymousTypes and can do the following:

var anonTypeList = AnonFactory.CreateListSizeOf(10,
                                  new {FirstName ="",
                                         Middle = "",
                                         LastName=""
                                   })
                                   .WhereAll()
                                       .Have("FirstName", "William")
                                       .Have("LastName", "Riker")
                                   .BuildList();

Like NBuilder we want to specify that different segments get different values. Here’s how that is accomplished:

var anonTypeList = AnonFactory.CreateListSizeOf(10,
                                  new {FirstName = "",
                                         Middle = "",
                                         LastName=""
                                   })
                                   .WhereTheFirst(5)
                                       .Have("FirstName", "William")
                                       .Have("LastName", "Riker")
                                   .AndTheNext(5)
                                       .Have("FirstName", "Jean-Luc")
                                       .Have("LastName","Picard")
                                   .BuildList();

There is also a WhereTheLast() clause for working with data at the end of a list.  All of these extension methods are contained in Extensions.cs.

Now let’s talk about how this works. For our purposes a fluent interface allows us to chain methods together yielding a more readable syntax. This is achieved with methods that return a reference to it’s class as in:

class RetangleCreator
{
    private decimal width;
    private decimal height;
    private string color;

    public RectangleCreator SetWidth(decimal width)
    {
        this.width = width;
        return this;
    }

    public RectangleCreator SetHeight(decimal height)
    {
        this.height = height ;
        return this;
    }

    public RectangleCreator SetColor(string color)
    {
        this.color= color;
        return this;
    }
}

var rectangleCreator = new RectangleCreator()
                                         .SetHeight(4.5)
                                         .SetWidth(5.75)
                                         .SetColor("red");

With a collection or list you might be tempted to create code that simply passes on the list after performing some action. This is fine until you need to perform actions on specific segments of that list. In our case we need to know where an action has left of in order to achieve syntax like WhereTheNext(5).Have(…).AndTheNext(5).

The class AnonTypeRange accomplishes this and allows us to perform actions on a list of AnonymousTypes in the manner we desire. Here is the code:

public void SetRange(int start, int amount)
{
    Enforce.ArgumentGreaterThanZero(amount, "AnonTypeRange.Next - amount must be greater than start");
    Enforce.That((start >= 0), "AnonTypeRange.Next - start must be greater than or equal to 0");
    Enforce.That((amount + start <= this.limit), "AnonTypeRange.Next - amount can not be greater than limit");     this.Start = start;     this.End = (start + amount - 1) > limit ? limit : start + amount - 1;
        }

The ranges themselves are set in the extension methods that comprise our syntax. Examine WhereTheNext() and AndTheNext() methods:

public static AnonTypeRange WhereTheFirst(this List anonTypes, int amount)
{
    Enforce.ArgumentGreaterThanZero(amount, "AnonTypeRange.WhereTheFirst - amount can not be less that 0");

    var anonTypeRange = new AnonTypeRange(anonTypes);
    anonTypeRange.SetRange(0, amount);

    return anonTypeRange;
}

public static AnonTypeRange AndTheNext(this AnonTypeRange anonTypeRange, int amount)
{
    Enforce.ArgumentGreaterThanZero(amount, "AnonTypeRange.AndTheNext - amount can not be less that 0");

    anonTypeRange.SetRange(anonTypeRange.End + 1, amount);

    return anonTypeRange;
}

The only drawback is that the actions are processed serially. In other words you do a group of 5, then another group of 5. If you need to go back you could add a WhereTheFirst() to reset the position of operations.

Before we move on take note of the WhereAll() method. This takes in a List and returns a AnonTypeRange with the range set to 0 spanning to the end:

public static AnonTypeRange WhereAll(this ListanonTypes)
{
    var anonTypeRange = new AnonTypeRange(anonTypes);
    anonTypeRange.SetRange(0, anonTypes.Count);

    return anonTypeRange;
}

Our values are set with the Have clause. Again we try to mimic the great functionality of NBuilder, so you have two options.  You can set a single value on a property over a range:

//  Have
public static AnonTypeRange Have(this AnonTypeRange anonTypeRange, string property, object value)
{
    anonTypeRange.Apply(property, value);
    return anonTypeRange;
}

//  Have calls AnonTypeRange.Apply to save the properties
public void Apply(string property, object value)
{
    Enforce.ArgumentNotNull(property, "AnonTypeRange.Apply - property can not be null");
    Enforce.ArgumentNotNull<object>(value, "AnonTypeRange.Apply - value can not be null");

    int count = (this.End - Start) + 1;
    var range = this.internalAnonList.GetRange(this.Start, count);
    range.ForEach(x => x.Set(property, value));                    }

Or you can create a list of values that will be selected at random and distributed across the range.  This takes advantage of the Pick functionality provided by NBuilder and is quite useful.

public void Apply(string property, List<T>pickList)
{
    Enforce.ArgumentNotNull(property, "AnonTypeRange.Apply - property can not be null");

    int count = (this.End - Start) + 1;
    var range = this.internalAnonList.GetRange(this.Start, count);

    range.ForEach(x => x.Set(property, Pick.RandomItemFrom(pickList)));
}

Serializing the List of AnonymousTypes

AnonymousType has a method that will serialize the properties that it contains.  Since AnonymousType stores the properties and respective values in a Dictionary it’s fairly easy to searlize the Dictionary.  The method uses the JObject from JSON.Net, but you can come up with your own mechanisms if you like:

//  From AnonymousType
public string ToJSON(Func , string, string> function,
                                                string jsonObjectName)
    {
        return function(_Values, jsonObjectName);
    }
//  Delegate method for serializing
public static string SerializeWithJObject(Dictionary values, string name)
{
    var jsonObject = new JObject();

    foreach (KeyValuePair property in values)
    {
        jsonObject.Add(new JProperty(property.Key, property.Value));
    }

    return jsonObject.ToString();
}

Serializing a list of AnonymousTypes is as equally straight forward to accomplish.  You only need to traverse the list and call the ToJSON methods on each AnonymousType object.  So easy it almost makes you feel guilty!

Dynamically Generate Data Sets with DataBuilder

If you’ve made it this far, congratulations, it’s been a bit of a marathon.   What is striking is how very straight forward much of this has been.  And the icing on the cake is that you can use the Snippet section of DataBuilder to run the code.  This required a slight alteration to the ScriptHostBase file, as it was expecting a path to an assembly.  Since the goal is to generate data sets without assemblies it would be pretty silly if you had supply something in the Assembly Path section.  All you need to do is supply something like the code below in the Code Snippet and hit “Build Data”:

var namesPickList = new List();
namesPickList.Add("Geordi");
namesPickList.Add("Que");
namesPickList.Add("Data");
namesPickList.Add("Jean-Luc");

string json = AnonFactory.CreateListOfSize(10,
                     new { LastName = "Kirk",
                              FirstName = "James",
                              MiddleInitial = "T." })
                      .WhereAll()
                            .Have("FirstName", namesPickList)
                      .BuildList()
                      .ToJSON();

parameters["JsonDataSet"] = json;

Faster than a photon torpedo you get back JSON. Now you’re set to start your HTML / Javascript prototypes. Need to alter the data, update the snippet and run it again.  Here’s the new source code, and prototype away!

How Embedded Scripting Makes Dynamically Generated Test Data Possible in ASP.Net – DataBuilder Part 2 November 6, 2010

Posted by ActiveEngine Sensei in .Net Development, ActiveEngine, ASP.Net, C#, CS-Script, DataBuilder, JSON.Net, NBuilder, Problem Solving.
Tags: , , , , , ,
add a comment

Part 1 of a 3 part series.  For the latest DataBuilder capabilities, read this post or download the new source code from here.

Last episode Sensei unveiled a useful little tool called DataBuilder.  DataBuilder helps you to generate test data for you domain objects.  Just point DataBuilder to your assemblies, and with the magic of NBuilder, CS-Script you can create test data as JSON.  How is this possible?  This post will focus on the behind the scenes magic that makes DataBuilder so flexible.

The main problem that DataBuilder solves is that to create test data for your classes you normally need to fire up Visual Studio and a project, create code, compile, etc. to produce anything and this can cause needless context switching and headache.  What if you wish to simply wish to mock up a UI and need some data sets to work with?  DataBuilder helps in that you can create test data for any existing assembly.  You can also create different types of test data based on what ever criteria you need.  This is accomplished by taking the input supplied in the Snippet Editor screen, compiling it to an in-memory assembly and executing it.  No need to fire up Visual Studio and add a TestGeneration project to your .Net solution.

The “dynamic” nature of DataBuilder is implemented with CS-Script.  In short, CS-Script is an embedded scripting system that uses ECMA-compliant C #, with full access to the CLR and OS.  For an in-depth review see  Oleg Shilo’s fantastic article on CodeProject where he describes his product.

As Oleg describes, CS-Script will compile your code into an assembly, load that assembly into a separate app domain, then execute that assembly.  There are two scenarios that can be used to host your script.  They are the Isolated Execution Pattern, where the host and script have no knowledge of each other, and the  Simplified Hosting Model for two way type sharing between the host and the script.  The Simplified Hosting Model allows the script file to access assemblies loaded in the host, as well as pass back data to the host.  DataBuilder uses the Simplified Host Model.

Before we get into the particular DataBuilder code, let’s review some samples that Oleg has provided.  The scenario presented is when you wish to remotely load a script and execute it, and the recommendation is to user interface inheritance to avoid the task of using reflection to invoke the method.

// Host contains this interface:
public interface IWordProcessor
{
void CreateDocument();
void CloseDocument();
void OpenDocument(string file);
void SaveDocument(string file);
}

//  The script file implements the interface
public class WordProcessor: IWordProcessor
{
public void CreateDocument() { ... }
public void CloseDocument() { ... }
public void OpenDocument(string file) { ... }
public void SaveDocument(string file) { ... }
}

//  Host executes the script
AsmHelper helper = new AsmHelper(CSScript.Load("script.cs", null, true));

//the only reflection based call
IWordProcessor proc = (IWordProcessor)helper.CreateObject("WordProcessor");

//no reflection, just direct calls
proc.CreateDocument();
proc.SaveDocument("MyDocument.cs");

There are other methods for invoking methods and scripts. It’s well worth your time reading through the script hosting guidelines as Oleg covers performance, reflection, interface alignment with duck typing and other facets that are important to CS-Script.

Now let’s focus on DataBuilder’s embedded scripting implementation.  DataBuilder uses the interface inheritance approach to execute the script that you supply.  Here’s the interface:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace DataGenerator.ScriptHost
{
    public interface IScriptRunner
    {
        void RunScript();
        void RunScript(Dictionary<string, object> parameters);
    }
}

And here is an implementation of the interface:

//CSScript directives - DO NOT REMOVE THE css_ref SECTION!!!
//css_ref System.Core;
//css_ref System.Data.ComponentModel;
//css_ref System.Data.DataSetExtensions;
//css_ref System.Xml.Linq;

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using DataGenerator.Core;
using DataGenerator.ScriptHost;
using System.Linq.Expressions;
using System.Linq;
using Newtonsoft.Json;
using FizzWare.NBuilder;
//  Add a reference to your assemblies as well!!
using UnRelatedAssembly;

public class CreateTestFile : IScriptRunner
{
    public void  RunScript(Dictionary<string,object> parameters)
    {
        var agents = Builder<SalesAgent>.CreateListOfSize(5)
                    .WhereTheFirst(1)
                         .Have(x => x.FirstName = "James")
                         .And(x => x.LastName = "Kirk")
                    .AndTheNext(1)
                          .Have(x => x.FirstName = "Bruce")
                          .And(x => x.LastName = "Campbell")
                    .Build()
                    .ToList();

        parameters["JsonDataSet"] = JsonConvert.SerializeObject(agents);
    }

    public void  RunScript()
    {
 	    throw new NotImplementedException();
    }
}

The script host is derived from ScriptHostBase.  ScriptHostBase will perform the compilation of your script with the method CompileScript(), as well as fetching any remote assemblies that you want to include.  This is a great point of flexibility as it allows you to point to any assembly that you have access to.  Assemblies can come from multiple locations, and as long as you know the namespaces you can include the classes from those assemblies in your scripts.

        /// <summary>
        /// Compile a script and store in a runner object for later
        /// execution
        /// </summary>
        protected void CompileScript()
        {
            if(string.IsNullOrEmpty(this.Script))
            {
                throw new ArgumentNullException("ScriptHostBase - CompileScript : Script can not be blank");
            }

            if (string.IsNullOrEmpty(this.TypeName))
            {
                throw new ArgumentNullException("ScriptHostBase - CompileScript : TypeName can not be blank");
            }

            //  Has an assembly already been loaded?
            string names = string.Empty;
            AppDomain appDomain = AppDomain.CurrentDomain;

            var assemblyPaths = appDomain.GetAssemblies()
                                    .ToList()
                                    .Select(x => x.FullName)
                                    .ToList();

            var fizzWare = assemblyPaths.Where(x => x.Contains("FizzWare.NBuilder"))
                                            .SingleOrDefault();

            var assemblyLoadList = new List<string>();
            assemblyLoadList = this.AssemblyPaths.ToList();

            //  Load if needed
            if (fizzWare != null)
            {
                string remove = assemblyLoadList
                                     .Where(x => x.Contains("FizzWare.NBuilder"))
                                     .SingleOrDefault();
                assemblyLoadList.Remove(remove);
            }
            else
            {
                string path = ConfigurationManager.AppSettings["FizzWarePath"].ToString();
                assemblyPaths.Add(path);
            }

            Assembly compiler = CSScript.LoadCode(this.Script, assemblyLoadList.ToArray());
            AsmHelper asmHelper = new AsmHelper(compiler);
            this.runner = asmHelper.CreateObject(this.TypeName);
        }

You may be scratching your head at the lines of code that explicitly load FizzWare.NBuilder(26 -43).  When first constructing DataBuilder, Sensei struggled with getting NBuilder to compile with the new script.  CS-Script uses an algorithm to probe directories for assemblies as well as probing scripts to resolve namespaces.  In some cases, this probe will NOT locate a namespace based on the naming conventions of an assembly. CS-Script has provisions for handling those scenarios allowing you to specifically load an assembly.  The issue Sensei had at first was that the first execution of a script would complete successfully as NBuilder would be loaded.  The problem lay with the second run of the script, as an exception would be thrown claiming that NBuilder was already loaded and hence there was no need to explicitly load it again!  The work around is to query the loaded assemblies and if NBuilder is loaded, remove that path to FizzWare.NBuilder assembly from the AssemblyPaths list and prevent the script from reloading NBuilder.

Classes derived from ScriptHostBase are responsible for implementing ExecuteScript method.  In this implementation StringScriptHost derives from ScriptHostBase and has the following ExecuteScript method:

        /// <summary>
        /// Compile a script and invoke
        /// </summary>
        public override void ExecuteScript()
        {
            base.CompileScript();

            IScriptRunner scriptRunner = (IScriptRunner)this.runner;
            scriptRunner.RunScript(Parameters);
        }

Other script hosts can be created and handle scenarios where scripts stored in a document database, text fields in SQL Server or other scenarios.

The process of including your import statements, locating any scripts located on a share and passing parameters to scripts is all controlled by the ScriptController.  There are two constructors with one allowing you to specify the script location:

public ScriptController(string scriptShare){}

With the ScriptController you can execute snippets that you type free form with the method ExecuteSnippet.

public void ExecuteSnippet(string snippet, Dictionary<string, object> parameters)
        {
            Enforce.ArgumentNotNull<string>(snippet, "ScriptController.ExecuteAdHoc - snippet can not be null");

            //  Wrap snippet with class declaration and additional using ;
            snippet = snippetHeader + this.UsingFragment + snippetClassName +
                        snippet + snippetFooter;

            var scriptHost = new StringScriptHost();
            scriptHost.Script = snippet;
            scriptHost.TypeName = "AdHoc";
            scriptHost.Parameters = parameters;
            scriptHost.AssemblyPaths = this.assemblyPaths.ToArray();

            scriptHost.ExecuteScript();
        }

Another method ExecuteScript is used for executing script files that you have save on a share.  As you read through the ExecuteSnippet method, you’ll note that the controller will combine the required import and namespace methods.  It’s really just concatenating strings to build a complete script in the format displayed above in the CreateTestFile.cs code.

You create a Dictionary<string, object> called parameters and pass this to the ScriptController.Execute methods.  This allows you great flexibility as you can allow the scripts to instantiate different objects and return them to the host application for further use.  In the case of DataBuilder we are expecting a JsonDataSet object which is our serialized test data in the form of JSON.

That’s it.  Hopefully you find DataBuilder and these posts useful.  CS-Script is quite powerful and can allow you to execute operations without the need to constantly recompile your projects.  It also allows to execute operations dynamically.  DataBuilder would not be possible without it.  When duty calls and fluent solutions are needed, CS-Script and embedded scripting are pretty incredible. Source code is here.