jump to navigation

More RavenDB Resources January 3, 2012

Posted by ActiveEngine Sensei in .Net Development, C#, New Techniques, Open Source, RavenDB.
Tags: , ,
3 comments

Daniel Lang has a great post regarding how to handle relations in RavenDB.  He emphasizes that a document database is vastly different from a relation database and illustrates various scenarios of do’s and don’ts.  Go read it now.

Advertisements

ApprovaFlow: Using the Pipe and Filter Pattern to Build a Workflow Processor May 17, 2011

Posted by ActiveEngine Sensei in .Net, ActiveEngine, Approvaflow, ASP.Net, C#, Stateless, Workflow.
Tags: , , ,
add a comment

This is the third entry in a series of posts for ApprovaFlow, an alternative to Windows Workflow written in C# and JSON.Net.  Source code for this post is here.

What We’ve Accomplished Thus Far

In the last post we discussed how Stateless makes creating a lean workflow engine possible, and we saw that we were able to achieve two of our overall goals for ApprovaFlow.  Here’s what we accomplished:

Model a workflow in a clear format that is readable by both developer and business user. One set of verbiage for all parties.
•. Allow the state of a workflow to be peristed as an integer, string, etc.  Quickly fetch state of a workflow.

So we have these goals left:

•. Create pre and post processing methods that can enforce enforce rules or carry out actions when completing a workflow task.
•. Introduce new functionality while isolating the impact of the new changes. New components should not break old ones
•.Communicate to the client with a standard set of objects. In other words, your solution domain will not change how the user interface will gather data from the user.
•. Use one. aspx page to processes user input for any type of workflow.
•. Provide ability to roll your own customizations to the front end or backend of your application.

Our next goal will be Create pre and post processing methods that can enforce enforce rules or carry out actions when completing a workflow task.  We’ll use the Pipe and Filter Pattern to simplify the processing, and we’ll see that this approach not only streamlines how you handle variation in tasks, but also provides a clean method for extending our application abilities.


The advantage of breaking down the activities of a process is that you can create a series of inter-changeable actions.  There may be some cases where you want to re-order the order of operations at runtime and you can do so easily when the actions are individual components.

Before we proceed applying the Pipe and Filter pattern to our solution, we need to establish some nomenclature for our workflow processing.  The following chart lays out the vocab we’ll use for the rest of series.

Term Definition
State A stage of a workflow.
Trigger A message that tells the workflow how to change states.  If the state is “Phone Ringing” and the trigger is “Answer Phone” the new state for the phone would be “Off hook”.
StateConfig A StateConfig defines a pathway or transition from one state to another.  It is comprised of a State, the Trigger and the Target State.
Step A Step contains the workflow’s current State.  In the course of your workflow you may have many of the same type of steps differentiated by date and time.  In other words, when you workflow has looping capability, the workflow step for a state may be issued many times.
Answer The Step asks a question, waiting for the user response.  The answer the user provides is the trigger that will start the transition from one state to another.  The Answer becomes the Trigger that will change the State.
Workflow A series of Steps compromised of States, Triggers and their respective transition expressed as a series of State Configs.  Think of this as a definition of a process.
Workflow Instance The Workflow Instance is a running workflow.  The Steps of the Workflow Instance are governed by how the Steps are defined by a Workflow.

Essentially a framework for providing an extensible workflow system boils down to answering the following questions asked in this order:
• Is the user authorized to provide an Answer to trigger a change to the step’s State?
• Is a special data set required for this particular State that is not part of the Step properties?
• Is the data provided from the user sufficient / valid for triggering a transition in the Workflow Step’s State?
• Are there actions to be performed such as saving special data?
• Can the system execute custom actions based on the State’s Trigger?

This looks very similar to the Pipe and Filter pattern.  Every time a workflow processes a trigger, the questions we asked above must be answered.  Each question could be considered a filter in the pipe and filter scenario.

The five questions above become the basis for our workflow processor components.  For this post we’ll assume that all data will be simply fetched then saved with no special processing.  We’ll also assume that a Workflow Step is considered to be valid when the following elements are correctly supplied:

public bool IsValidForWorkflowTransition()
{
  return this.Enforce("Step", true)
                 .When("AnsweredBy", Janga.Validation.Compare.NotEqual, string.Empty)
                 .When("Answer", Janga.Validation.Compare.NotEqual, string.Empty)
                 .When("State", Janga.Validation.Compare.NotEqual, string.Empty)
                 .When("WorkflowInstanceId", Janga.Validation.Compare.NotEqual, string.Empty)
                 .IsValid;
}

public bool IsUserValidParticipant()
{
  return this.Enforce("Step", true)
    .When("Participants", Janga.Validation.Compare.Contains, this.AnsweredBy)
    .IsValid;
}

Our Workflow Processor will function in accordance with the Pipe and Filter pattern where no matter what type of workflow instance we wish to process, the questions that we listed above will be answered.  Later we will return to discuss points of where the workflow can execute actions respective to the workflow’s definition.

Workflow Processor Code In Depth

Well, how do we configure a Workflow Processor?  In other words, we want to process an actual workflow, but how will we know the workflow type and what to do?  Some of configuration steps were previewed in Simple Workflows With ApprovaFlow and Stateless and the same principles apply here with the Configure method.  Collect the States, the Triggers and the StateConfigs, load them into Stateless along with the current state and you are ready to accept or reject the Trigger for the next State.  The Workflow Processor will conduct these steps and here is the code:

public WorkflowProcessor ConfigureStateMachine()
        {
            Enforce.That(string.IsNullOrEmpty(this.step.State) == false,
                            "WorkflowProcessor.Confgiure - step.State can not be empty");

            this.stateMachine = new StateMachine(this.step.State);

            //  Get a distinct list of states with a trigger from state configuration
            //  "State => Trigger => TargetState
            var states = this.workflow.StateConfigs.AsQueryable()
                                    .Select(x => x.State)
                                    .Distinct()
                                    .Select(x => x)
                                    .ToList();

            //  Assing triggers to states
            states.ForEach(state =>
            {
                var triggers = this.workflow.StateConfigs.AsQueryable()
                                   .Where(config => config.State == state)
                                   .Select(config => new { Trigger = config.Trigger, TargeState = config.TargetState })
                                   .ToList();

                triggers.ForEach(trig =>
                {
                    this.stateMachine.Configure(state).Permit(trig.Trigger, trig.TargeState);
                });
            });

            return this;
        }

The Workflow Processor will need to know the current state of a workflow instance, the answer supplied, who supplied the answer, as well as any parameters that the filters will need fetching special data. This will be contained in the class Step.cs:

#region Properties

        public string WorkflowInstanceId { get; set; }
        public string WorkflowId { get; set; }
        public string StepId { get; set; }
        public string State { get; set; }
        public string PreviousState { get; set; }
        public string Answer { get; set; }
        public DateTime Created { get; set; }
        public string AnsweredBy { get; set; }
        public string Participants { get; set; }

        public List ErrorList;
        public bool CanProcess { get; set; }
        public IDictionary Parameters { get; set; }

        #endregion

        #region Constructors

        public Step()
            : this(string.Empty, string.Empty, string.Empty, string.Empty,
                   string.Empty, new DateTime(), string.Empty, string.Empty,
                    new Dictionary())
        { }

        public Step(string workflowInstanceId, string stepId, string state, string previousState,
                        string answer, DateTime created, string answeredBy, string participants,
                        Dictionary parameters)
        {
            this.WorkflowInstanceId = workflowInstanceId;
            this.StepId = stepId;
            this.State = state;
            this.PreviousState = previousState;
            this.Answer = answer;
            this.Created = created;
            this.AnsweredBy = answeredBy;
            this.Participants = participants;

            this.ErrorList = new List();
            this.Parameters = parameters;
        }

        #endregion

Our goal with the Workflow Processor is to accept the users answer, process actions, and create the next Step base on the new State all in one pass.  We will create a pipeline of actions that will always be invoked.  Each action or “filter” will be a component that performs and individual task, such as determining if the step is answered by the correct user.  Each filter will point to the subsequent filter in the pipeline, and the succession of the filters can change easily if we see fit.  All that is needed is to add the filters to the pipeline in the order we want.  Here is the class schema for the Pipe and Filter processing:

We’ll quickly find that the information regarding whether the result of an action or the condition of a Step will need to be accessible to each of the filters.  The class Step is the natural place to store this information, so we will include a property CanProcess to indicate that a filter should be invoked, as well a List<string> to act as an error log.  This log can be passed back to the client to communicate any errors to the user.  Note that the Step class has the Dictionary property named “Parameters” that allows a filter to pass data on to next filter in the sequence.

Setting Up the Pipeline

The sequence of filter execution is controlled by the order that the filters are registered.  The class Pipeline is responsible for registering and executing the chain of filters.  Here is the method Register that accepts a filter and retains it for future processing:

We also record the name of the filter so that we may interrogate the pipeline should we want to know if a filter has already been registered.

Pipeline.Register returns a reference to itself, so we can chain together commands fluently:

pipeline.Register(new ValidParticipantFilter())
                      .Register(new SaveDataFilter());

The class FilterBase is the foundation of our filter components.  As stated earlier, each component will point the subsequent filter in the filter chain.  You’ll note that the class also has a Register method.  This takes on the task of point the current filter to the next, and this method is called by the Pipeline as it registers all of the filters. Here is FilterBase:

public abstract class FilterBase : IFilter
{
  private IFilter next;

  protected abstract T Process(T input);

  public T Execute(T input)
  {
    T val = Process(input);

     if (this.next != null)
    {
      val = this.next.Execute(val);
    }

    return val;
  }

  public void Register(IFilter filter)
  {
    if (this.next == null)
    {
      this.next = filter;
    }
    else
    {
      this.next.Register(filter);
    }
  }
}

The method Execute accepts input of type T, and in the Workflow Processor instance this will Step.  Basically the Execute method is a wrapper, as we call the abstract method Process.  Process will be overridden in each filter, and this will contain the logic specific to the tasks that will be performed.  The code for a filter is quite simple:

public class ValidParticipantFilter : FilterBase
{
  protected override Step Process(Step input)
  {
    if (input.CanProcess)
    {
      input.Parameters["ValidFired"] = true;
      input.Parameters["FilterOrder"] += "ValidParticipantFilter;";

      input.CanProcess = input.IsUserValidParticipant();

    if(input.CanProcess == false)
    {
      input.ErrorList.Add("Invalid Pariticipant - " + input.AnsweredBy);
    }
  }

  return input;
}

Here we check to see if we can process, then perform specific actions if appropriate.  Given that the filters have no knowledge of each other, we can see that they can be executed in any order.  In other words you could have a Pipeline that had filters Step1, Step2, Step3 and you could configure a different pipeline to execute Step3, Step1, and Step2.

FilterRegistry Organizes Your Filters

Because we want to be able to use our filters in different successions we’ll need to keep a registry of what is available to use and provide the ability to look up or query different filters depending on our processing needs.  This registry will be created on application start up and will contain all objects of type FilterBase.  Later we’ll add the ability for the registry to load assemblies from a share, so that you can add other filters as simple plugins.  Information about each filter retained in a class FilterDefinition, and the FilterRegistry is merely a glorified List of the FilterDefintions. When we want to create a pipeline of filters we will want to instantiate new copies. Using Expressions we can create Functions that will be stored with with our definition for each filter type.  Here is FilterDefinition:

public class FilterDefinition
{
  public string Name { get; set; }
  public string FilterCategory { get; set; }
  public Type FilterType { get; set; }
  public Func> Filter{get; set;}

  public FilterDefinition() { }
}

We’ll invoke the compiled delegate at runtime to create our filter.  The method AddCreateFilter handles this:

private Func> AddCreateFilter(FilterDefinition filterDef)
        {
            var body = Expression.MemberInit(Expression.New(filterDef.FilterType));
            return Expression.Lambda>>(body, null).Compile();
        }

FilterRegistry is meant to be run once at start up so that all filters are registered and ready to use. You can imagine how slow it could become if every time you process a Workflow Step that you must interrogate all the assemblies.

Once you FilterRegistry has all assemblies registered you can query and create new combinations with the method GetFilters:

public IEnumerable> GetFilters(string filterNames)
        {
            Enforce.That(string.IsNullOrEmpty(filterNames) == false,
                            "FilterRegistry.GetFilters - filterNames can not be null");

            var returnFilters = new List>();
            var names = filterNames.Split(';').ToList();

            names.ForEach(name =>
            {
                var filter = this.filters.Where(x => x.Name == name)
                                           .SingleOrDefault();

                if (filter != null)
                {
                    returnFilters.Add(filter.Filter.Invoke());
                }
            });

            return returnFilters;
        }

Pipeline can accept a list of filters along with the string that represents the order of execution.  The method RegisterFrom accepts a reference to the FilterRegistry along with the names of the filters you want to use.

In the case of the Workflow Processor, we need to divide our filters into pre-trigger and post-trigger activities. Referring back to our 5 questions that our processor asks, question 1 – 3 must be answered before we attempt to transition the Workflow State, while steps 4-5 must be answered after the transition has succeeded. The method ConfigurePipeline in WorkflowProcessor.cs accomplishes this task:

public WorkflowProcessor ConfigurePipeline(string preProcessFilterNames, string postProcessFilterNames)
        {
            Enforce.That(string.IsNullOrEmpty(preProcessFilterNames) == false,
                            "WorkflowProcessor.Configure - preProcessFilterNames can not be null");

            Enforce.That(string.IsNullOrEmpty(postProcessFilterNames) == false,
                            "WorkflowProcessor.Configure - postProcessFilterNames can not be null");

            var actionWrapper = new ActionWrapperFilter(this.ExecuteTriggerFilter);

            this.pipeline.RegisterFromList(preProcessFilterNames, this.filterRegistry)
                            .Register(actionWrapper)
                            .RegisterFromList(postProcessFilterNames, this.filterRegistry);

            return this;
        }

Putting It all Together

A lot of talk and theory, so how does this all fit together?  The test class WorkflowScenarioTests illustrates how our processor works.  We are creating a workflow that implements the process for a Red Shirt requesting a promotion off a landing party.  You may recall that the dude wearing the red shirt usually got killed with in the first few minutes of Star Trek, so this workflow will help those poor saps get off the death list.  The configuration for the Workflow is contained within the file RedShirtPromotion.json.  There are a few simple rules that we want to enforce with the Workflow.  For one, Spock must review the Red Shirt request, but Kirk will have the final say.

Here is a sample from the class WorkflowScenarioTests.cs:

  string source = @"F:\vs10dev\ApprovaFlowSimpleWorkflowProcessor\TestSuite\TestData\RedShirtPromotion.json";
            string preFilterNames = "FetchDataFilter;ValidParticipantFilter;";
            string postFilterNames = "SaveDataFilter";

            var workflow = DeserializeWorkflow(source);

            var parameters = new Dictionary();
            parameters.Add("FilterOrder", string.Empty);
            parameters.Add("FetchDataFired", false);
            parameters.Add("SaveDataFired", false);
            parameters.Add("ValidFired", false);

            var step = new Step("13", "12", "RequestPromotionForm", "",
                                    "Complete", DateTime.Now, "RedShirtGuy", "Data;RedShirtGuy",
                                    parameters);
            step.CanProcess = true;

            var filterRegistry = new FilterRegistry();

            var processor = new WorkflowProcessor(step, filterRegistry, workflow);
            string newState = processor.ConfigurePipeline(preFilterNames, postFilterNames)
                        .ConfigureStateMachine()
                        .ProcessAnswer()
                        .GetCurrentState();

            Assert.AreEqual("FirstOfficerReview", newState);

Study the tests.  We’ve covered a lot together and admittedly there is a lot swallow in this post.  In our next episode we’ll look at how to the Pipe and Filter pattern can help us with extending our workflow processor’s capability without causing us a lot of pain.  Here’s the source code.  Enjoy and check back soon for our next installment.  Sensei will let you take it on out with this groovy theme (click play).

Simple Workflows With ApprovaFlow and Stateless April 2, 2011

Posted by ActiveEngine Sensei in .Net, ActiveEngine, Approvaflow, ASP.Net, C#, JSON.Net, New Techniques, Stateless.
Tags: , , , , ,
add a comment

This is the second in a series of posts for ApprovaFlow, an alternative to Windows Workflow written in C# and JSON.Net. Source code for this post is here.

Last time we laid out out goals for a simple workflow engine, ApprovaFlow, with the following objectives:
• Model a workflow in a clear format that is readable by both developer and business user. One set of verbiage for all parties.
•. Allow the state of a workflow to be peristed as an integer, string. Quicky fetch state of a workflow.
•. Create pre and post nprocessing methods that can enforce enforce rules or carry out actions when completing a workflow task.
•. Introduce new functionality while isolating the impact of the new changes. New components should not break old ones
•.Communicate to the client with a standard set of objects. In other words, your solution domain will not change how the user interface will gather data from the user.
•. Use one. aspx page to processes user input for any type of workflow.
•. Provide ability to roll your own customizations to the front end or backend of your application.

The fulcrum point of all we have set out to do with ApprovaFlow is a state machine that will present a state and accept answers supplied by the users. One of Sensei’s misgivings about Windows Workflow is that it is such a behemoth when all you want to implement is a state machine.
Stateless, created Nicholas Blumhardt, is a shining example of adhering to the rule of “necessary and sufficient”. By using Generics Stateless allows you to create a state machine where the State and Trigger can be represented by an integer, string double, enum – say this sounds like it fulfills our goal:

•. Allow the state of a workflow to be persisted as an integer, string. Quicky fetch state of a workflow.
Stateless constructs a state machine with the following syntax:

var statemachine =
       new StateMachine(TState currentState);

For our discussion we will create a state machine that will process a request for promotion workflow. We’ll use:

var statemachine =
       new StateMachine(string currentstate);

This could very easily take the form of

<int, int>

and will depend on your preferences. Regardless of your choice, if the current state is represent by a primitive like int or string, you can just fetch that from a database or a repository and now your state machine is loaded with the current state. Contrast that with WF where you have multiple projects and confusing nomenclature to learn. Stateless just stays out of our way.
Let’s lay out our request for promotion workflow. Here is our state machine represented in English:

Step: Request Promotion Form
  Answer => Complete
  Next Step => Manager Review

Step: Manager Review
  Answer => Deny
  Next Step => Promotion Denied
  Answer => Request Info
  Next Step => Request Promotion Form
  Answer => Approve
  Next Step => Vice President Approve

Step: Vice President Approve
  Answer => Deny
  Next Step => Promotion Denied
  Answer => Manager Justify
  Next Step => Manager Review
  Answer => Approve
  Next Step => Promoted

Step: Promotion Denied
Step: Promoted

Remember the goal Model a workflow in a clear format that is readable by both developer and business user. One set of verbiage for all parties? We are very close to achieving that goal. If we substitute “Step” with “State” and “Answer” with “Trigger”, then we have a model that matches how Stateless configures a state machine:

var statemachine = new StateMachine(startState);

//  Request Promo form states
statemachine.Configure("RequestPromotionForm")
               .Permit("Complete", "ManagerReview");

//  Manager Review states
statemachine.Configure("ManagerReview")
               .Permit("RequestInfo", "RequestPromotionForm")
               .Permit("Deny", "PromotionDenied")
               .Permit("Approve", "VicePresidentApprove");

Clearly you will not show the code to your business partners or end users, but a simple chart like this should not make anyone’s eyes glaze over:

State: Request Promotion Form
  Trigger => Complete
  Target State => Manager Review

Before we move on you may want to study the test in the file SimpleStateless.cs. Here configuring the state machine and advancing from state to state is laid out for you:

//  Request Promo form states
statemachine.Configure("RequestPromotionForm")
                    .Permit("Complete", "ManagerReview");

//  Manager Review states
statemachine.Configure("ManagerReview")
                     .Permit("RequestInfo", "RequestPromotionForm")
                     .Permit("Deny", "PromotionDenied")
                     .Permit("Approve", "VicePresidentApprove");

//  Vice President state configuration
statemachine.Configure("VicePresidentApprove")
                      .Permit("ManagerJustify", "ManagerReview")
                      .Permit("Deny", "PromotionDenied")
                      .Permit("Approve", "Promoted");

//  Tests
Assert.AreEqual(startState, statemachine.State);

//  Move to next state
statemachine.Fire("Complete");
Assert.IsTrue(statemachine.IsInState("ManagerReview"));

statemachine.Fire("Deny");
Assert.IsTrue(statemachine.IsInState("PromotionDenied"));

The next question that comes to mind is how to represent the various States, Triggers and State configurations as data. Our mission on this project is to adhere to simplicity. One way to represent a Stateless state machine is with JSON:

{WorkflowType : "RequestPromotion",
  States : [{Name : "RequestPromotionForm" ; DisplayName : "Request Promotion Form"}
    {Name : "ManagerReview", DisplayName : "Manager Review"},
    {Name : "VicePresidentApprove", DisplayName : "Vice President Approve"},
    {Name : "PromotionDenied", DisplayName : "Promotion Denied"},
    {Name : "Promoted", DisplayName : "Promoted"}
    ],
  Triggers : [{Name : "Complete", DisplayName : "Complete"},
     {Name : "Approve", DisplayName : "Approve"},
     {Name : "RequestInfo", DisplayName : "Request Info"},
     {Name : "ManagerJustify", DisplayName : "Manager Justify"},
     {Name : "Deny", DisplayName : "Deny"}
  ],
StateConfigs : [{State : "RequestPromotionForm", Trigger : "Complete", TargetState : "ManagerReview"},
     {State : "ManagerReview", Trigger : "RequestInfo", TargetState : "RequestPromotionForm"},
     {State : "ManagerReview", Trigger : "Deny", TargetState : "PromotionDenied"},
     {State : "ManagerReview", Trigger : "Approve", TargetState : "VicePresidentApprove"},
     {State : "VicePresidentApprove", Trigger : "ManagerJustify", TargetState : "ManagerApprove"},
     {State : "VicePresidentApprove", Trigger : "Deny", TargetState : "PromotionDenied"},
     {State : "VicePresidentApprove", Trigger : "Approve", TargetState : "Promoted"}
  ]
}

As you can see we are storing all States and all Triggers with their display names. This will allow you some flexibility with UI screens and reports. Each rule for transitioning a state to another is stored in the StateConfigs node. Here we are simply representing our chart that we created above as JSON.

Since we have a standard way of representing a workflow with JSON de-serializing this definition to objects is straight forward. Here are the corresponding classes that define a state machine:

public class WorkflowDefinition
{
        public string WorkflowType { get; set; }
        public List States { get; set; }
        public List Triggers { get; set; }
        public List StateConfigs { get; set; }

        public WorkflowDefinition() { }
}

public class State
{
        public string Name { get; set; }
        public string DisplayName { get; set; }
}

public class Trigger
{
        public string Name { get; set; }
        public string DisplayName { get; set; }

        public Trigger() { }
}
public class StateConfig
{
        public string State { get; set; }
        public string Trigger { get; set; }
        public string TargetState { get; set; }

        public StateConfig() { }
}

We’ll close out this post with an example that will de-serialize our state machine definition and allow us to respond to the triggers that we supply. Basically it will be a rudimentary workflow. RequestionPromotion.cs will be the workflow processor. The method Configure is where we will perform the de-serialization, and the process is quite straight forward:

  1. Deserialize the States
  2. Deserialize the Triggers
  3. Deserialize the StateConfigs that contain the transitions from state to state
  4. For every StateConfig, configure the state machine.

Here’s the code:

public void Configure()
{
    Enforce.That((string.IsNullOrEmpty(source) == false),
                            "RequestPromotion.Configure - source is null");

    string json = GetJson(source);

    var workflowDefintion = JsonConvert.DeserializeObject(json);

    Enforce.That((string.IsNullOrEmpty(startState) == false),
                            "RequestPromotion.Configure - startStep is null");

    this.stateMachine = new StateMachine(startState);

    //  Get a distinct list of states with a trigger from state configuration
    //  "State => Trigger => TargetState
    var states = workflowDefintion.StateConfigs.AsQueryable()
                                    .Select(x => x.State)
                                    .Distinct()
                                    .Select(x => x)
                                    .ToList();

    //  Assing triggers to states
    states.ForEach(state =>
    {
        var triggers = workflowDefintion.StateConfigs.AsQueryable()
                                   .Where(config => config.State == state)
                                   .Select(config => new { Trigger = config.Trigger, TargeState = config.TargetState })
                                   .ToList();

        triggers.ForEach(trig =>
        {
            this.stateMachine.Configure(state).Permit(trig.Trigger, trig.TargeState);
        });
    });
}

And we advance the workflow with this method:

public void ProgressToNextState(string trigger)
{
Enforce.That((string.IsNullOrEmpty(trigger) == false),
"RequestPromotion.ProgressToNextState – trigger is null");

this.stateMachine.Fire(trigger);
}

The class RequestPromotionTests.cs illustrates how this works.

We we have seen how we can fulfill the objectives laid out for ApprovaFlow and have covered a significant part of the functionality that Stateless will provide for our workflow engine.   Here is the source code.

ApprovaFlow – A Proof of Concept March 25, 2011

Posted by ActiveEngine Sensei in .Net, .Net Development, ActiveEngine, Approvaflow, ASP.Net, C#, JSON.Net, LINQ, New Techniques, Open Source.
Tags: , ,
add a comment

Like Tolkien, Sensei wants to create the landscapes, cultures and languages before he writes his next epic. You can be the judge whether the work is a series of sketches and notes like the Silmarillion or cohesive, compelling story that you want read again and again. As a bonus Sensei will deliver working software that hopefully will be of use to you.  (Photo credit – utnapistim).

The epic will be called ApprovaFlow. ApprovaFlow is a framework / process / methodology that allows you to create workflow applications that are easy to deploy and are configurable. With ApprovaFlow Sensei hopes to demonstrate how to readily encorporate the inevitable changes that your users will ask of you. Deliver changes effortlessly and without groans. Cast off the chains inconvenient builds and focus on creating solutions that stay out of the users way.

Ok. Managent wants bullet points so here are our goals for ApprovaFlow:

• Model a workflow in a clear format that is readable by both developer and business user. One set of verbiage for all parties.
•. Allow the state of a workflow to be peristed as an integer, string. Quicky fetch state of a workflow.
•. Create pre and post nprocessing methods that can enforce enforce rules or carry out actions when completing a workflow task.
•. Introduce new functionality while isolating the impact of the new changes. New components should not break old ones
•.Communicate to the client with a standard set of objects. In other words, your solution domain will not change how the user interface will gather data from the user.
•. Use one. aspx page to processes user input for any type of workflow.
•. Provide ability to roll your own customizations to the front end or backend of your application.

There it is. These goals will probably take us a good amount of time to review and implement. Is it worth it? Hell yeah. We’ll end up with one simple project instead of a bloated framework where it takes forever to find anything. A nice by product will be that you can spend more time thinking about how to solve your users problems rather than trying to figure out a monsterous framework that requires a huge investment of energy and time learning how to get simple things done.

DataTablePager Now Has Multi-Column Sort Capability For DataTables.Net February 9, 2011

Posted by ActiveEngine Sensei in .Net, ActiveEngine, Ajax, ASP.Net, C#, DataTables.Net, jQuery.
Tags: , , , , , , , , ,
21 comments

Some gifts just keep on giving, and many times things can just take on a momentum that grow beyond your expectation.  Bob Sherwood wrote to Sensei and pointed out that DataTables.net supports multiple column sorting.  All you do is hold down the shift key and click on any second or third column and DataTables will add that column to sort criteria.  “Well, how come it doesn’t work with the server side solution?”  Talk about the sound of one hand clapping.  How about that for a flub!  Sensei didn’t think of that!  Then panic set in – would this introduce new complexity to the DataTablePager solution, making it too difficult to maintain a clean implementation?  After some long thought it seemed that a solution could be neatly added.  Before reading, you should download the latest code to follow along.

How DataTables.Net Communicates Which Columns Are Involved in a Sort

If you recall, DataTables.Net uses a structure called aoData to communicate to the server what columns are needed, the page size, and whether a column is a data element or a client side custom column.  We covered that in the last DataTablePager post.  aoData also has a convention for sorting:

bSortColumn_X=ColumnPosition

In our example we are working with the following columns:

,Name,Agent,Center,,CenterId,DealAmount

where column 0 is a custom client side column, column 1 is Name (a mere data column), column 2 is Center (another data column), column 3 is a custom client side column, and the remaining columns are just data columns.

If we are sorting just by Name, then aoData will contain the following:

bSortColumn_0=1

When we wish to sort by Center, then by Name we get the following in aoData”

bSortColumn_0=2

bSortColumn_1=1

In other words, the first column we want to sort by is in position 2 (Center) and the second column(Name) is in position 1.  We’ll want to record this some where so that we can pass this to our order routine.  aoData passes all column information to us on the server, but we’ll have to parse through the columns and check to see if one or many of the columns is actually involved in a sort request and as we do we’ll need to preserve the order of that column of data in the sort.

SearchAndSortable Class to the Rescue

You’ll recall that we have a class called SearchAndSortable that defines how the column is used by the client.  Since we iterate over all the columns in aoData it makes sense that we should take this opportunity to see if any column is involved in a sort and store that information in SearchAndSortable as well.  The new code for the class looks like this:

public class SearchAndSortable
    {
        public string Name { get; set; }
        public int ColumnIndex { get; set; }
        public bool IsSearchable { get; set; }
        public bool IsSortable { get; set; }
        public PropertyInfo Property{ get; set; }
        public int SortOrder { get; set; }
        public bool IsCurrentlySorted { get; set; }
        public string SortDirection { get; set; }

        public SearchAndSortable(string name, int columnIndex, bool isSearchable,
                                bool isSortable)
        {
            this.Name = name;
            this.ColumnIndex = columnIndex;
            this.IsSearchable = isSearchable;
            this.IsSortable = IsSortable;
        }

        public SearchAndSortable() : this(string.Empty, 0, true, true) { }
    }

There are 3 new additions:

IsCurrentlySorted – is this column included in the sort request.

SortDirection – “asc” or “desc” for ascending and descending.

SortOrder – the order of the column in the sort request.  Is it the first or second column in a multicolumn sort.

As we walk through the column definitions, we’ll look to see if each column is involved in a sort and record what direction – ascending or descending – is required. From our previous post you’ll remember that the method PrepAOData is where we parse our column definitions. Here is the new code:

//  Sort columns
this.sortKeyPrefix = aoDataList.Where(x => x.Name.StartsWith(INDIVIDUAL_SORT_KEY_PREFIX))
                                            .Select(x => x.Value)
                                            .ToList();

//  Column list
var cols = aoDataList.Where(x => x.Name == "sColumns"
                                            & string.IsNullOrEmpty(x.Value) == false)
                                     .SingleOrDefault();

if(cols == null)
{
  this.columns = new List();
}
else
{
  this.columns = cols.Value
                       .Split(',')
                       .ToList();
}

//  What column is searchable and / or sortable
//  What properties from T is identified by the columns
var properties = typeof(T).GetProperties();
int i = 0;

//  Search and store all properties from T
this.columns.ForEach(col =>
{
  if (string.IsNullOrEmpty(col) == false)
  {
    var searchable = new SearchAndSortable(col, i, false, false);
    var searchItem = aoDataList.Where(x => x.Name == BSEARCHABLE + i.ToString())
                                     .ToList();
    searchable.IsSearchable = (searchItem[0].Value == "False") ? false : true;
    searchable.Property = properties.Where(x => x.Name == col)
                                                    .SingleOrDefault();

    searchAndSortables.Add(searchable);
  }

  i++;
});

//  Sort
searchAndSortables.ForEach(sortable => {
  var sort = aoDataList.Where(x => x.Name == BSORTABLE + sortable.ColumnIndex.ToString())
                                            .ToList();
  sortable.IsSortable = (sort[0].Value == "False") ? false : true;
                sortable.SortOrder = -1;

  //  Is this item amongst currently sorted columns?
  int order = 0;
  this.sortKeyPrefix.ForEach(keyPrefix => {
    if (sortable.ColumnIndex == Convert.ToInt32(keyPrefix))
    {
      sortable.IsCurrentlySorted = true;

      //  Is this the primary sort column or secondary?
      sortable.SortOrder = order;

     //  Ascending or Descending?
     var ascDesc = aoDataList.Where(x => x.Name == "sSortDir_" + order)
                                                    .SingleOrDefault();
     if(ascDesc != null)
     {
       sortable.SortDirection = ascDesc.Value;
     }
   }

   order++;
 });
});

To sum up, we’ll traverse all of the columns listed in sColumns. For each column we’ll grab the PorpertyInfo from our underlying object of type T. This gives only those properties that will be displayed in the grid on the client. If the column is marked as searchable, we indicate that by setting the IsSearchable property on the SearchAndSortable class.  This happens starting at line 28 through 43.

Next we need to determine what we can sort, and will traverse the new list of SearchAndSortables we created. DataTables will tell us what if the column can be sorted by with following convention:

bSortable_ColNumber = True

So if the column Center were to be “sortable” aoData would contain:

bSortable_1 = True

We record the sortable state as shown on line 49 in the code listing.

Now that we know whether we can sort on this column, we have to look through the sort request and see if the column is actually involved in a sort.  We do that by looking at what DataTables.Net sent to us from the client.  Again the convention is to send bSortColumn_0=1 to indicate that the first column for the sort in the second item listed in sColumns property.  aoData will contain many bSortColum’s so we’ll walk through each one and record the order that column should take in the sort.  That occurs at line 55 where we match the column index with the bSortColumn_x value.

We’ll also determine what the sort direction – ascending or descending – should be.  At line 63 we get the direction of the sort and record this value in the SearchAndSortable.

When the method PrepAOData is completed, we have a complete map of all columns and what columns are being sorted, as well as their respective sort direction.  All of this was sent to us from the client and we are storing this configuration for later use.

Performing the Sort

(Home stretch so play the song!!)

If you can picture what we have so far we just basically created a collection of column names, their respective PropertyInfo’s and have recorded which of these properties are involved in a sort.  At this stage we should be able to query this collection and get back those properties and the order that the sort applies.

You may already be aware that you can have a compound sort statement in LINQ with the following statement:

var sortedCustomers = customer.OrderBy(x => x.LastName)
                                           .ThenBy(x => x.FirstName);

The trick is to run through all the properties and create that compound statement. Remember when we recorded the position of the sort as an integer? This makes it easy for us to sort out the messy scenarios where the second column is the first column of a sort. SearchAndSortable.SortOrder takes care of this for us. Just get the data order by SortOrder in descending order and you’re good to go. So that code would look like the following:

var sorted = this.searchAndSortables.Where(x => x.IsCurrentlySorted == true)
                                     .OrderBy(x => x.SortOrder)
                                     .ToList();

sorted.ForEach(sort => {
             records = records.OrderBy(sort.Name, sort.SortDirection,
             (sort.SortOrder == 0) ? true : false);
});

On line 6 in the code above we are calling our extension method OrderBy in Extensions.cs. We pass the property name, the sort direction, and whether this is the first column of the sort. This last piece is important as it will create either “OrderBy” or the “ThenBy” for us. When it’s the first column, you guessed it we get “OrderBy”. Sensei found this magic on a StackOverflow post by Marc Gravell and others.

Here is the entire method ApplySort from DataTablePager.cs, and note how we still check for the initial display of the data grid and default to the first column that is sortable.

private IQueryable ApplySort(IQueryable records)
{
  var sorted = this.searchAndSortables.Where(x => x.IsCurrentlySorted == true)
                                                .OrderBy(x => x.SortOrder)
                                                .ToList();

  //  Are we at initialization of grid with no column selected?
  if (sorted.Count == 0)
  {
    string firstSortColumn = this.sortKeyPrefix.First();
    int firstColumn = int.Parse(firstSortColumn);

    string sortDirection = "asc";
    sortDirection = this.aoDataList.Where(x => x.Name == INDIVIDUAL_SORT_DIRECTION_KEY_PREFIX +                                                                    "0")
                                                    .Single()
                                                    .Value
                                                    .ToLower();

    if (string.IsNullOrEmpty(sortDirection))
    {
      sortDirection = "asc";
    }

    //  Initial display will set order to first column - column 0
    //  When column 0 is not sortable, find first column that is
    var sortable = this.searchAndSortables.Where(x => x.ColumnIndex == firstColumn)
                                                        .SingleOrDefault();
    if (sortable == null)
    {
      sortable = this.searchAndSortables.First(x => x.IsSortable);
    }

    return records.OrderBy(sortable.Name, sortDirection, true);
  }
  else
  {
      //  Traverse all columns selected for sort
      sorted.ForEach(sort => {
                             records = records.OrderBy(sort.Name, sort.SortDirection,
                            (sort.SortOrder == 0) ? true : false);
      });

    return records;
  }
}

It’s All in the Setup

Test it out. Hold down the shift key and select a second column and WHAMO – multiple column sorts! Hold down the shift key and click the same column twice and KAH-BLAMO multiple column sort with descending order on the second column!!!

The really cool thing is that our process on the server is being directed by DataTables.net on the client.  And even awseomer is that you have zero configuration on the server.  Most awesome-est is that this will work with all of your domain objects, because we have used generics we can apply this to any class in our domain.  So what are you doing to do with all that time you just got back?

How to Create Server-Side Paging for DataTables.Net with ASP.Net December 19, 2010

Posted by ActiveEngine Sensei in .Net, ActiveEngine, C#, DataTables.Net, Fluent, jQuery, New Techniques, Open Source, Problem Solving, Tutorial.
Tags: , , , , , , ,
22 comments

Source code has been updated!! Read about the changes in Dynamically Select Columns with Server-Side Paging and Datatables.Net If you are new to DataTables.Net and Sensei’s paging solution and want to detailed study of how it works, work through this post first, then get the latest edition.  Note, code links in this post are to the first version.

A central theme for 2010 has been fluency, or the continual practice of certain methods to such a degree that your performance improves and you produce increasingly polished, effective solutions.  For software development this has meant tools to save time and increase quality.  It also means keeping an eye toward making the users of your solutions more efficient as well.  In the spirit of “fluent solutions”, Sensei will end the year with a post that examines how to create a data paging solution for the jQuery data grid plug-in DataTables.Net.

DataTables can turn a HTML table into a fully functional data grid like the one offered by Telerik.  This plug-in offers client side sorting, filtering/ search,  as well as support for server-side processing processing of data.  It is an extremely feature rich tool created by Allan Jardine, and is itself worthy of a series of posts.  For this post on data paging Sensei recommends that you read through these examples to get an idea of what the data paging service needs to achieve.

Let’s get started with the goals we need to achieve when providing server-side data paging support:

  • Send data to client in the multiples or “chunks” that the client requests, and respond when the size of the sets requested is changed by the user.
  • Re-order the data set if the user clicks on a column heading.  Honor the data set size when returning the data.
  • Filter across all columns of data based on user input.  Implement this as partial matches, and again, honor the data set size.

Remember this is about flexibility, so we have the additional goals of:

  • Create a solution that can be reused.
  • Provide a mechanism to accommodate any type of .Net class using generics.

Essentially we want to be able to write code like so:

var tenants = tenantRepository.GetAll();
var dataTablePager = new DataTablePager();
var returnDataSet = dataTablePager.Filter(requestParms, tenants);

Before we proceed, Sensei wants to acknowledge those really smart people whose ideas contributed to this solution:

Zack Owens – jQuery DataTables Plugin Meets C#

Jeff Morris – Using Datatables.net JQuery Plug-in with WCF Services

Dave Ward – ASMX ScriptService mistake – Invalid JSON primitive

You may want to download the source before reading the rest of this post.

Communicating with DataTables

DataTables uses the following parameters when processing server-side data:

Sent to the server:

Type Name Info
int iDisplayStart Display start point
int iDisplayLength Number of records to display
int iColumns Number of columns being displayed (useful for getting individual column search info)
string sSearch Global search field
boolean bEscapeRegex Global search is regex or not
boolean bSortable_(int) Indicator for if a column is flagged as sortable or not on the client-side
boolean bSearchable_(int) Indicator for if a column is flagged as searchable or not on the client-side
string sSearch_(int) Individual column filter
boolean bEscapeRegex_(int) Individual column filter is regex or not
int iSortingCols Number of columns to sort on
int iSortCol_(int) Column being sorted on (you will need to decode this number for your database)
string sSortDir_(int) Direction to be sorted – “desc” or “asc”. Note that the prefix for this variable is wrong in 1.5.x where iSortDir_(int) was used)
string sEcho Information for DataTables to use for rendering

Reply from the server

In reply to each request for information that DataTables makes to the server, it expects to get a well formed JSON object with the following parameters.

Type Name Info
int iTotalRecords Total records, before filtering (i.e. the total number of records in the database)
int iTotalDisplayRecords Total records, after filtering (i.e. the total number of records after filtering has been applied – not just the number of records being returned in this result set)
string sEcho An unaltered copy of sEcho sent from the client side. This parameter will change with each draw (it is basically a draw count) – so it is important that this is implemented. Note that it strongly recommended for security reasons that you ‘cast’ this parameter to an integer in order to prevent Cross Site Scripting (XSS) attacks.
string sColumns Optional – this is a string of column names, comma separated (used in combination with sName) which will allow DataTables to reorder data on the client-side if required for display
array array mixed aaData The data in a 2D array

The data sent back is in the following form depicted below. Note that aaData is merely an array of strings – there is no column information. This will present a challenge in that you will not be able to simply serialize a collection and pass back the results.

{
    "sEcho": 3,
    "iTotalRecords": 57,
    "iTotalDisplayRecords": 57,
    "aaData": [
        [
            "Gecko",
            "Firefox 1.0",
            "Win 98+ / OSX.2+",
            "1.7",
            "A"
        ],
        [
            "Gecko",
            "Firefox 1.5",
            "Win 98+ / OSX.2+",
            "1.8",
            "A"
        ],
        ...
    ]
}

As you may be aware, if you wish to use ASP.Net web services to serialize JSON you must POST to the service and instruct it to interpret your parameters as JSON. DataTables will POST variables as value pairs and this won’t work for us when POSTing to a web service. We’ll have to translate the variables to a usable format. Luckily DataTables allows us to intervene with the following code, where we create a JSON string by serializing a structure called aoData:

"fnServerData": function ( sSource, aoData, fnCallback ) {

		        	var jsonAOData = JSON.stringify(aoData);

			        $.ajax( {
                                        contentType: "application/json; charset=utf-8",
				        type: "POST",
				        url: sSource,
				        data: "{jsonAOData : '" + jsonAOData + "'}",
				        success: function(msg){
				            fnCallback(JSON.parse(msg.d));
				        },
				        error: function(XMLHttpRequest, textStatus, errorThrown) {
                            alert(XMLHttpRequest.status);
                            alert(XMLHttpRequest.responseText);

                        }
			        });

Our web service can now de-serialize aoData and parse the appropriate parameters. This gives us important items such as how many records to display, what columns to sort on, and what search terms should be applied in a filter.

DataTablePager Class

DataTablePager.cs is the work horse of our solution.  It will sort, filter and order our data, and as an extra, serialize the results in format required by aaData.  Here’s the constructor:

public DataTablePager(string jsonAOData, IQueryable queryable)
        {
            this.queryable = queryable;
            this.type = typeof(T);
            this.properties = this.type.GetProperties(BindingFlags.Public | BindingFlags.Instance);
            this.aoDataList = new List>();
            this.sortKeyPrefix = new List();

            PrepAOData(jsonAOData);
        }

The parameter jsonAOData is the JSON string that contains the variables iDisplayStart, iDisplayLength, etc.  These will be parsed by the method PrepAOData.  The parameter queryable is the collection of records that will be filtered and parsed into JSON format required by DataTables.

The method Filter() coordinates all of the work.  It’s pretty simple what we want to do:  filter our data based on each column containing the search term, sort the result, then pull out the number of records we need to include in the page, and finally convert the collection into the format DataTables understands.

public FormattedList Filter()
        {
            var formattedList = new FormattedList();

            //  What are the columns in the data set
            formattedList.Import(this.properties.Select(x => x.Name + ",")
                                                 .ToArray());

            //  Return same sEcho that was posted.  Prevents XSS attacks.
            formattedList.sEcho = this.echo;

            //  Return count of all records
            formattedList.iTotalRecords = this.queryable.Count();

            //  Filtered Data
            var records = this.queryable.Where(GenericSearchFilter());
            records = ApplySort(records);

            //  What is filtered data set count now.  This is NOT the
            //  count of what is returned to client
            formattedList.iTotalDisplayRecords = (records.FirstOrDefault() == null) ? 0 : records.Count();

            //  Take a page
            var pagedRecords = records.Skip(this.displayStart)
                     .Take(this.displayLength);

            //  Convert to List of List
            var aaData = new List>();
            var thisRec = new List();

            pagedRecords.ToList()
                    .ForEach(rec => aaData.Add(rec.PropertiesToList()));

            formattedList.aaData = aaData;

            return formattedList;
        }

That said, there is some trickery that goes on in order to make this happen because we are creating a solution to is going to work with any IQueryable to we supply. This means that the filtering and the sorting will need to be dynamic.

To make the filtering dynamic we will build expression trees that will convert each property to a string, convert the string to lower case, then execute a Contains method against the value of that property.  The method GenericSearchFilter() called on line 16 accomplishes this with the following lines of code:

//  Except from GenericSearchFilter
MethodInfo convertToString = typeof(Convert).GetMethod("ToString", Type.EmptyTypes);

 var propertyQuery = (from property in this.properties
        let toStringMethod = Expression.Call(                                          Expression.Call(Expression.Property(paramExpression, property), convertToString, null),                                                            typeof(string).GetMethod("ToLower", new Type[0]))
         select Expression.Call(toStringMethod, typeof(string).GetMethod("Contains"), searchExpression)).ToArray();

We get an array of Expressions that when executed will tell us if the value matches our search term. What we want is to include the item if ANY of the properties is a match, so this means we have to use and OR for all of the properties. That can be accomplished with:

for (int j = 0; j < propertyQuery.Length; j++)
{
  //  Nothing to "or" to yet
  if (j == 0)
  {
    compoundOrExpression = propertyQuery[0];
  }

  compoundOrExpression = Expression.Or(compoundOrExpression,
                                              propertyQuery[j]);
}

So with what is listed above we would be able to match all properties with against a single search term. Pretty cool. But DataTables raises the bar even higher. If you were to go to the samples page and filter using multiple partial words you would find that you could perform some very effective searches with phrases like “new chic”. This would select all records that had properties containing “new” OR “chic”. Imagine the scenario where your user wants to finds all cities “New York” or “Chicago”. We’ve all been there where we have a grid and can only search for one term, or worse, where we have to add a row to a search filter grid and constantly push a “query” button to perform our searches. DataTables does all of the with one search box – just type and the filtering begins.

GenericSearchFilter() handles that scenario. First the search term is parsed into individual terms if there is a ” ”  supplied in the string. This means we will have to perform the propertyQuery for each term that we have. To return all of the records that correspond to each term we still need to perform the OR in groups, but then we need to AND these predicates together so we can get all of the groups per individual term. Here’s the source edited slightly for readability:

//  Split search expression to handle multiple words
var searchTerms = this.genericSearch.Split(' ');

for (int i = 0; i < searchTerms.Length; i++) {    var searchExpression = Expression.Constant( searchTerms[i].ToLower());                   //  For each property, create a contains expression   //  column => column.ToLower().Contains(searchTerm)
  //  Edited for clarity - create the array propertyQuery logic is here ...
  var propertyQuery = ...

  //  Inner loop for grouping all OR's for this search term
  for (int j = 0; j < propertyQuery.Length; j++)   {     //  Nothing to "or" to yet     if (j == 0)     {       compoundOrExpression = propertyQuery[0];     }     compoundOrExpression = Expression.Or(compoundOrExpression, propertyQuery[j]);   }   //  First time around there is no And, only first set of or's   if (i == 0)   {     compoundAndExpression = compoundOrExpression;   }   else   {     compoundAndExpression = Expression.And(compoundAndExpression, compoundOrExpression);   } } 

So GenericSearchFilter will build a humongous expression tree for all the properties in your class. To make this usable for the Where we convert it using Expression.Lambda and our Where clause just goes about its merry way. Because we have used generics, you can supply any class from your assemblies. One caveat, and Sensei is trying to find a resolution. If you have a string property to that is set to null, the expression tree fails. You’ll note that in the classes supplied in the sample, the properties that are of type string in the Tenant class are defaulted to empty in the constructor.  A small price to pay for some great functionality. To sort our data we use the method ApplySort():

 private IQueryable ApplySort(IQueryable records)         {             string firstSortColumn = this.sortKeyPrefix.First();             int firstColumn = int.Parse(firstSortColumn);             string sortDirection = "asc";             sortDirection = this.aoDataList.Where(x => x.Name == INDIVIDUAL_SORT_DIRECTION_KEY_PREFIX +
                                                                      "0")
                                                .Single()
                                                .Value
                                                .ToLower();

            if (string.IsNullOrEmpty(sortDirection))
            {
                sortDirection = "asc";
            }

            return records.OrderBy(this.properties[firstColumn].Name, sortDirection, true);
        }

An extension method OrderBy will accept the name of column, the sort direction as parameters. The parameter initial will indicate if we are sorting mulitple times, so we can accomplish multi-property sort with syntax like

var sortedRecords = records.OrderBy("State", "desc", true)
                                      .OrderBy("City", "asc", false);

public static IOrderedQueryable OrderBy(this IQueryable source, string property, string sortDirection, bool initial)
        {
            string[] props = property.Split('.');
            Type type = typeof(T);
            ParameterExpression arg = Expression.Parameter(type, "x");
            Expression expr = arg;
            foreach (string prop in props)
            {
                // use reflection (not ComponentModel) to mirror LINQ
                PropertyInfo pi = type.GetProperty(prop);
                expr = Expression.Property(expr, pi);
                type = pi.PropertyType;
            }
            Type delegateType = typeof(Func<,>).MakeGenericType(typeof(T), type);
            LambdaExpression lambda = Expression.Lambda(delegateType, expr, arg);

            string methodName = string.Empty;

            //  Asc or Desc
            if (sortDirection.ToLower() == "asc")
            {
                //  First clause?
                if (initial && source is IOrderedQueryable)
                {
                    methodName = "OrderBy";
                }
                else
                {
                    methodName = "ThenBy";
                }
            }
            else
            {
                if (initial && source is IOrderedQueryable)
                {
                    methodName = "OrderByDescending";
                }
                else
                {
                    methodName = "ThenByDescending";
                }
            }

            object result = typeof(Queryable).GetMethods().Single(
                    method => method.Name == methodName
                            && method.IsGenericMethodDefinition
                            && method.GetGenericArguments().Length == 2
                            && method.GetParameters().Length == 2)
                    .MakeGenericMethod(typeof(T), type)
                    .Invoke(null, new object[] { source, lambda });
            return (IOrderedQueryable)result;
        }

All good things …
It’s been a long ride, this post. A lot of code discussed, a lot of ground covered. The solution is here.  As always, play around and see how this can help you. If anything breaks, tell Sensei. If you have improvements, tell Sensei. DataTables is a great tool for your arsenal, hopefully the DataTablePager can help you integrate paging with large datasets as part of your solution offering.

Right now Sensei wants to sign off by toasting to you for wading through all of this, and for having the desire to build up your skills.  Obtaining fluency in what you do is a hard road to travel, but it’s worth it because you get things done quicker and better with each session.

Be sure to read about the latest version in “Dynamically Select Columns with Server-Side Paging and Datatables.Net

Prototyping With AnonymousTypes, CS-Script and JSON.Net – DataBuilder Part 3 November 14, 2010

Posted by ActiveEngine Sensei in .Net, ASP.Net, C#, CS-Script, DataBuilder, JSON.Net, New Techniques, Open Source.
Tags: , , , ,
4 comments

Sensei hit his head in the shower this morning and instead of the flux-capacitor he saw the following code that could be used for prototyping:

var jsonDataSet = AnonFactory.CreateList(10, new { FirstName = "",
                                 LastName = "",
                                 Age = 0,
                                 Salary = 45000})
                              .WhereAll()
                              .Have("FirstName", "Jim")
                              .BuildList()
                              .ToJSON();

If you have been following the series on DataBuilder, a utility that dynamically creates test data as JSON, you’ll recall one of the highlights was avoiding the need to create an additional Visual Studio project just so you could create test data.  DataBuilder can import your assemblies and with the use of NBuilder it creates test data for you.  You may wish to read those posts for some background.

But consider the scenarios where you do not have an application domain or a .Net assembly to test.  Should you stop?  For prototyping do you really need to compile, change, compile, change, show-to-your-user, repeat ad-nauseum?

Here’s a new concept:  Build data sets that you use in prototyping with Javascript before you break out Visual Studio.  If you can quickly build a tool set to communicate with your business users you’ll be much further ahead by avoiding the back-end client side impedance mismatch.  After all, many times the business user has the concept already hidden upstairs in their heads, why not help get that vision out in the open earlier and save yourself.

So, to be able to write the statement that Sensei saw in his vision we need to achieve to following goals:

  • Create an anonymous type as template for building a list
  • Create a fluent interface for chaining clauses similar to NBuilder
  • Serialize the list of anonymous types to JSON
  • Run this process in DataBuilder – in other words, go dynamic!

You may want to download the source and follow along.

AnonymousType as a Template

It turns out that Sensei has a tool in his bag of tricks, namely the Persistent Anonymous type.  As you are aware, anonymous types in C# have a limited scope, but the Persistent Anonymous type that Sensei discussed allows you to create a structure that mimics a standard anonymous type while being persist that structure beyond the scope of its creation.  The AnonymousType creator, Hugo Benocci created something really great.  Here’s what you can do from the get-go:

var anonType = AnonymousType(new {FirstName = "James",
                                         Middle = "T",
                                         LastName="Kirk"
                                   });

If you recall the examples used in the prior posts on data generation with DataBuilder, NBuilder supplied default values for you when you built an object. You then had to alter those values should you want something different.

With an AnonymousType we use a factory to create a list of AnonymousTypes like so:

var anonTypeList = AnonFactory.CreateListSizeOf(10,
                                  new {FirstName = "James",
                                         Middle = "T",
                                         LastName="Kirk"
                                   });

Nice and simple. If we didn’t have more goals to achieve, we could go home. As it is, on to the next step.

Creating a Fluent Interface for Chaining Clauses

We still need to provide variation in our data set.  NBuilder does a nice job of letting us use clauses like .WhereTheNext(5).Have() or WhereAll.Have() to introduce more structured variation in our data.  We want to achieve the same thing with the AnonymousTypes and can do the following:

var anonTypeList = AnonFactory.CreateListSizeOf(10,
                                  new {FirstName ="",
                                         Middle = "",
                                         LastName=""
                                   })
                                   .WhereAll()
                                       .Have("FirstName", "William")
                                       .Have("LastName", "Riker")
                                   .BuildList();

Like NBuilder we want to specify that different segments get different values. Here’s how that is accomplished:

var anonTypeList = AnonFactory.CreateListSizeOf(10,
                                  new {FirstName = "",
                                         Middle = "",
                                         LastName=""
                                   })
                                   .WhereTheFirst(5)
                                       .Have("FirstName", "William")
                                       .Have("LastName", "Riker")
                                   .AndTheNext(5)
                                       .Have("FirstName", "Jean-Luc")
                                       .Have("LastName","Picard")
                                   .BuildList();

There is also a WhereTheLast() clause for working with data at the end of a list.  All of these extension methods are contained in Extensions.cs.

Now let’s talk about how this works. For our purposes a fluent interface allows us to chain methods together yielding a more readable syntax. This is achieved with methods that return a reference to it’s class as in:

class RetangleCreator
{
    private decimal width;
    private decimal height;
    private string color;

    public RectangleCreator SetWidth(decimal width)
    {
        this.width = width;
        return this;
    }

    public RectangleCreator SetHeight(decimal height)
    {
        this.height = height ;
        return this;
    }

    public RectangleCreator SetColor(string color)
    {
        this.color= color;
        return this;
    }
}

var rectangleCreator = new RectangleCreator()
                                         .SetHeight(4.5)
                                         .SetWidth(5.75)
                                         .SetColor("red");

With a collection or list you might be tempted to create code that simply passes on the list after performing some action. This is fine until you need to perform actions on specific segments of that list. In our case we need to know where an action has left of in order to achieve syntax like WhereTheNext(5).Have(…).AndTheNext(5).

The class AnonTypeRange accomplishes this and allows us to perform actions on a list of AnonymousTypes in the manner we desire. Here is the code:

public void SetRange(int start, int amount)
{
    Enforce.ArgumentGreaterThanZero(amount, "AnonTypeRange.Next - amount must be greater than start");
    Enforce.That((start >= 0), "AnonTypeRange.Next - start must be greater than or equal to 0");
    Enforce.That((amount + start <= this.limit), "AnonTypeRange.Next - amount can not be greater than limit");     this.Start = start;     this.End = (start + amount - 1) > limit ? limit : start + amount - 1;
        }

The ranges themselves are set in the extension methods that comprise our syntax. Examine WhereTheNext() and AndTheNext() methods:

public static AnonTypeRange WhereTheFirst(this List anonTypes, int amount)
{
    Enforce.ArgumentGreaterThanZero(amount, "AnonTypeRange.WhereTheFirst - amount can not be less that 0");

    var anonTypeRange = new AnonTypeRange(anonTypes);
    anonTypeRange.SetRange(0, amount);

    return anonTypeRange;
}

public static AnonTypeRange AndTheNext(this AnonTypeRange anonTypeRange, int amount)
{
    Enforce.ArgumentGreaterThanZero(amount, "AnonTypeRange.AndTheNext - amount can not be less that 0");

    anonTypeRange.SetRange(anonTypeRange.End + 1, amount);

    return anonTypeRange;
}

The only drawback is that the actions are processed serially. In other words you do a group of 5, then another group of 5. If you need to go back you could add a WhereTheFirst() to reset the position of operations.

Before we move on take note of the WhereAll() method. This takes in a List and returns a AnonTypeRange with the range set to 0 spanning to the end:

public static AnonTypeRange WhereAll(this ListanonTypes)
{
    var anonTypeRange = new AnonTypeRange(anonTypes);
    anonTypeRange.SetRange(0, anonTypes.Count);

    return anonTypeRange;
}

Our values are set with the Have clause. Again we try to mimic the great functionality of NBuilder, so you have two options.  You can set a single value on a property over a range:

//  Have
public static AnonTypeRange Have(this AnonTypeRange anonTypeRange, string property, object value)
{
    anonTypeRange.Apply(property, value);
    return anonTypeRange;
}

//  Have calls AnonTypeRange.Apply to save the properties
public void Apply(string property, object value)
{
    Enforce.ArgumentNotNull(property, "AnonTypeRange.Apply - property can not be null");
    Enforce.ArgumentNotNull<object>(value, "AnonTypeRange.Apply - value can not be null");

    int count = (this.End - Start) + 1;
    var range = this.internalAnonList.GetRange(this.Start, count);
    range.ForEach(x => x.Set(property, value));                    }

Or you can create a list of values that will be selected at random and distributed across the range.  This takes advantage of the Pick functionality provided by NBuilder and is quite useful.

public void Apply(string property, List<T>pickList)
{
    Enforce.ArgumentNotNull(property, "AnonTypeRange.Apply - property can not be null");

    int count = (this.End - Start) + 1;
    var range = this.internalAnonList.GetRange(this.Start, count);

    range.ForEach(x => x.Set(property, Pick.RandomItemFrom(pickList)));
}

Serializing the List of AnonymousTypes

AnonymousType has a method that will serialize the properties that it contains.  Since AnonymousType stores the properties and respective values in a Dictionary it’s fairly easy to searlize the Dictionary.  The method uses the JObject from JSON.Net, but you can come up with your own mechanisms if you like:

//  From AnonymousType
public string ToJSON(Func , string, string> function,
                                                string jsonObjectName)
    {
        return function(_Values, jsonObjectName);
    }
//  Delegate method for serializing
public static string SerializeWithJObject(Dictionary values, string name)
{
    var jsonObject = new JObject();

    foreach (KeyValuePair property in values)
    {
        jsonObject.Add(new JProperty(property.Key, property.Value));
    }

    return jsonObject.ToString();
}

Serializing a list of AnonymousTypes is as equally straight forward to accomplish.  You only need to traverse the list and call the ToJSON methods on each AnonymousType object.  So easy it almost makes you feel guilty!

Dynamically Generate Data Sets with DataBuilder

If you’ve made it this far, congratulations, it’s been a bit of a marathon.   What is striking is how very straight forward much of this has been.  And the icing on the cake is that you can use the Snippet section of DataBuilder to run the code.  This required a slight alteration to the ScriptHostBase file, as it was expecting a path to an assembly.  Since the goal is to generate data sets without assemblies it would be pretty silly if you had supply something in the Assembly Path section.  All you need to do is supply something like the code below in the Code Snippet and hit “Build Data”:

var namesPickList = new List();
namesPickList.Add("Geordi");
namesPickList.Add("Que");
namesPickList.Add("Data");
namesPickList.Add("Jean-Luc");

string json = AnonFactory.CreateListOfSize(10,
                     new { LastName = "Kirk",
                              FirstName = "James",
                              MiddleInitial = "T." })
                      .WhereAll()
                            .Have("FirstName", namesPickList)
                      .BuildList()
                      .ToJSON();

parameters["JsonDataSet"] = json;

Faster than a photon torpedo you get back JSON. Now you’re set to start your HTML / Javascript prototypes. Need to alter the data, update the snippet and run it again.  Here’s the new source code, and prototype away!

How Embedded Scripting Makes Dynamically Generated Test Data Possible in ASP.Net – DataBuilder Part 2 November 6, 2010

Posted by ActiveEngine Sensei in .Net Development, ActiveEngine, ASP.Net, C#, CS-Script, DataBuilder, JSON.Net, NBuilder, Problem Solving.
Tags: , , , , , ,
add a comment

Part 1 of a 3 part series.  For the latest DataBuilder capabilities, read this post or download the new source code from here.

Last episode Sensei unveiled a useful little tool called DataBuilder.  DataBuilder helps you to generate test data for you domain objects.  Just point DataBuilder to your assemblies, and with the magic of NBuilder, CS-Script you can create test data as JSON.  How is this possible?  This post will focus on the behind the scenes magic that makes DataBuilder so flexible.

The main problem that DataBuilder solves is that to create test data for your classes you normally need to fire up Visual Studio and a project, create code, compile, etc. to produce anything and this can cause needless context switching and headache.  What if you wish to simply wish to mock up a UI and need some data sets to work with?  DataBuilder helps in that you can create test data for any existing assembly.  You can also create different types of test data based on what ever criteria you need.  This is accomplished by taking the input supplied in the Snippet Editor screen, compiling it to an in-memory assembly and executing it.  No need to fire up Visual Studio and add a TestGeneration project to your .Net solution.

The “dynamic” nature of DataBuilder is implemented with CS-Script.  In short, CS-Script is an embedded scripting system that uses ECMA-compliant C #, with full access to the CLR and OS.  For an in-depth review see  Oleg Shilo’s fantastic article on CodeProject where he describes his product.

As Oleg describes, CS-Script will compile your code into an assembly, load that assembly into a separate app domain, then execute that assembly.  There are two scenarios that can be used to host your script.  They are the Isolated Execution Pattern, where the host and script have no knowledge of each other, and the  Simplified Hosting Model for two way type sharing between the host and the script.  The Simplified Hosting Model allows the script file to access assemblies loaded in the host, as well as pass back data to the host.  DataBuilder uses the Simplified Host Model.

Before we get into the particular DataBuilder code, let’s review some samples that Oleg has provided.  The scenario presented is when you wish to remotely load a script and execute it, and the recommendation is to user interface inheritance to avoid the task of using reflection to invoke the method.

// Host contains this interface:
public interface IWordProcessor
{
void CreateDocument();
void CloseDocument();
void OpenDocument(string file);
void SaveDocument(string file);
}

//  The script file implements the interface
public class WordProcessor: IWordProcessor
{
public void CreateDocument() { ... }
public void CloseDocument() { ... }
public void OpenDocument(string file) { ... }
public void SaveDocument(string file) { ... }
}

//  Host executes the script
AsmHelper helper = new AsmHelper(CSScript.Load("script.cs", null, true));

//the only reflection based call
IWordProcessor proc = (IWordProcessor)helper.CreateObject("WordProcessor");

//no reflection, just direct calls
proc.CreateDocument();
proc.SaveDocument("MyDocument.cs");

There are other methods for invoking methods and scripts. It’s well worth your time reading through the script hosting guidelines as Oleg covers performance, reflection, interface alignment with duck typing and other facets that are important to CS-Script.

Now let’s focus on DataBuilder’s embedded scripting implementation.  DataBuilder uses the interface inheritance approach to execute the script that you supply.  Here’s the interface:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace DataGenerator.ScriptHost
{
    public interface IScriptRunner
    {
        void RunScript();
        void RunScript(Dictionary<string, object> parameters);
    }
}

And here is an implementation of the interface:

//CSScript directives - DO NOT REMOVE THE css_ref SECTION!!!
//css_ref System.Core;
//css_ref System.Data.ComponentModel;
//css_ref System.Data.DataSetExtensions;
//css_ref System.Xml.Linq;

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using DataGenerator.Core;
using DataGenerator.ScriptHost;
using System.Linq.Expressions;
using System.Linq;
using Newtonsoft.Json;
using FizzWare.NBuilder;
//  Add a reference to your assemblies as well!!
using UnRelatedAssembly;

public class CreateTestFile : IScriptRunner
{
    public void  RunScript(Dictionary<string,object> parameters)
    {
        var agents = Builder<SalesAgent>.CreateListOfSize(5)
                    .WhereTheFirst(1)
                         .Have(x => x.FirstName = "James")
                         .And(x => x.LastName = "Kirk")
                    .AndTheNext(1)
                          .Have(x => x.FirstName = "Bruce")
                          .And(x => x.LastName = "Campbell")
                    .Build()
                    .ToList();

        parameters["JsonDataSet"] = JsonConvert.SerializeObject(agents);
    }

    public void  RunScript()
    {
 	    throw new NotImplementedException();
    }
}

The script host is derived from ScriptHostBase.  ScriptHostBase will perform the compilation of your script with the method CompileScript(), as well as fetching any remote assemblies that you want to include.  This is a great point of flexibility as it allows you to point to any assembly that you have access to.  Assemblies can come from multiple locations, and as long as you know the namespaces you can include the classes from those assemblies in your scripts.

        /// <summary>
        /// Compile a script and store in a runner object for later
        /// execution
        /// </summary>
        protected void CompileScript()
        {
            if(string.IsNullOrEmpty(this.Script))
            {
                throw new ArgumentNullException("ScriptHostBase - CompileScript : Script can not be blank");
            }

            if (string.IsNullOrEmpty(this.TypeName))
            {
                throw new ArgumentNullException("ScriptHostBase - CompileScript : TypeName can not be blank");
            }

            //  Has an assembly already been loaded?
            string names = string.Empty;
            AppDomain appDomain = AppDomain.CurrentDomain;

            var assemblyPaths = appDomain.GetAssemblies()
                                    .ToList()
                                    .Select(x => x.FullName)
                                    .ToList();

            var fizzWare = assemblyPaths.Where(x => x.Contains("FizzWare.NBuilder"))
                                            .SingleOrDefault();

            var assemblyLoadList = new List<string>();
            assemblyLoadList = this.AssemblyPaths.ToList();

            //  Load if needed
            if (fizzWare != null)
            {
                string remove = assemblyLoadList
                                     .Where(x => x.Contains("FizzWare.NBuilder"))
                                     .SingleOrDefault();
                assemblyLoadList.Remove(remove);
            }
            else
            {
                string path = ConfigurationManager.AppSettings["FizzWarePath"].ToString();
                assemblyPaths.Add(path);
            }

            Assembly compiler = CSScript.LoadCode(this.Script, assemblyLoadList.ToArray());
            AsmHelper asmHelper = new AsmHelper(compiler);
            this.runner = asmHelper.CreateObject(this.TypeName);
        }

You may be scratching your head at the lines of code that explicitly load FizzWare.NBuilder(26 -43).  When first constructing DataBuilder, Sensei struggled with getting NBuilder to compile with the new script.  CS-Script uses an algorithm to probe directories for assemblies as well as probing scripts to resolve namespaces.  In some cases, this probe will NOT locate a namespace based on the naming conventions of an assembly. CS-Script has provisions for handling those scenarios allowing you to specifically load an assembly.  The issue Sensei had at first was that the first execution of a script would complete successfully as NBuilder would be loaded.  The problem lay with the second run of the script, as an exception would be thrown claiming that NBuilder was already loaded and hence there was no need to explicitly load it again!  The work around is to query the loaded assemblies and if NBuilder is loaded, remove that path to FizzWare.NBuilder assembly from the AssemblyPaths list and prevent the script from reloading NBuilder.

Classes derived from ScriptHostBase are responsible for implementing ExecuteScript method.  In this implementation StringScriptHost derives from ScriptHostBase and has the following ExecuteScript method:

        /// <summary>
        /// Compile a script and invoke
        /// </summary>
        public override void ExecuteScript()
        {
            base.CompileScript();

            IScriptRunner scriptRunner = (IScriptRunner)this.runner;
            scriptRunner.RunScript(Parameters);
        }

Other script hosts can be created and handle scenarios where scripts stored in a document database, text fields in SQL Server or other scenarios.

The process of including your import statements, locating any scripts located on a share and passing parameters to scripts is all controlled by the ScriptController.  There are two constructors with one allowing you to specify the script location:

public ScriptController(string scriptShare){}

With the ScriptController you can execute snippets that you type free form with the method ExecuteSnippet.

public void ExecuteSnippet(string snippet, Dictionary<string, object> parameters)
        {
            Enforce.ArgumentNotNull<string>(snippet, "ScriptController.ExecuteAdHoc - snippet can not be null");

            //  Wrap snippet with class declaration and additional using ;
            snippet = snippetHeader + this.UsingFragment + snippetClassName +
                        snippet + snippetFooter;

            var scriptHost = new StringScriptHost();
            scriptHost.Script = snippet;
            scriptHost.TypeName = "AdHoc";
            scriptHost.Parameters = parameters;
            scriptHost.AssemblyPaths = this.assemblyPaths.ToArray();

            scriptHost.ExecuteScript();
        }

Another method ExecuteScript is used for executing script files that you have save on a share.  As you read through the ExecuteSnippet method, you’ll note that the controller will combine the required import and namespace methods.  It’s really just concatenating strings to build a complete script in the format displayed above in the CreateTestFile.cs code.

You create a Dictionary<string, object> called parameters and pass this to the ScriptController.Execute methods.  This allows you great flexibility as you can allow the scripts to instantiate different objects and return them to the host application for further use.  In the case of DataBuilder we are expecting a JsonDataSet object which is our serialized test data in the form of JSON.

That’s it.  Hopefully you find DataBuilder and these posts useful.  CS-Script is quite powerful and can allow you to execute operations without the need to constantly recompile your projects.  It also allows to execute operations dynamically.  DataBuilder would not be possible without it.  When duty calls and fluent solutions are needed, CS-Script and embedded scripting are pretty incredible. Source code is here.

Dynamically Create Test Data with NBuilder, JSON and .Net October 24, 2010

Posted by ActiveEngine Sensei in .Net, ActiveEngine, Ajax, ASP.Net, C#, Fluent, LINQ, Open Source, Problem Solving.
Tags: , , , , ,
5 comments

Part 1 of a 3 part series.  For the latest DataBuilder capabilities, read this post or download the new source code from here.

Building test data should be as easy:

var agentList = Builder<SalesAgent>.CreateListOfSize(5)
                           .WhereTheFirst(1)
                                  .Have(x => x.FirstName = "James")
                                  .And(x => x.LastName = "Kirk")
                            .AndTheNext(1)
                                  .Have(x => x.FirstName = "Bruce")
                                  .And(x => x.LastName = "Campbell")
                            .Build()
                            .ToList();

Wouldn’t be nice if all the properties of your objects were automatically populated:

Product:
       Id              : 1
       Title           : "Title1"
       Description     : "Description1"
       QuantityInStock : 1

NBuilder by provides you with a great fluent interface to accomplish this with ease.  You can even achieve scenarios where you can create hierarchies of data, set property values on a range objects in a list, and even create a specified range of values that you can use populate other objects.  Peruse through the samples and you will see, NBuilder quite capably maps values  the public properties of your objects.  A real time saver.

Sensei is going to kick it up a notch and provide you with a means to create test data with out having to recompile your projects.  This is ideal for when you want to create UI prototypes.  DataBulider uses CS-Script and NBuilder to create a web based data generation tool that can read assemblies and will allow you to script a process that will generate test data in the form of JSON.

This adventure is split into two parts.  First a quick demo, then instructions on how to configure DataBuilder for you environment.  A deeper discussion of CS-Script and embedded scripting in .Net will be part of the sequel to this action/adventure, as we all know the second movie in the series is always the best!.

Operating DataBuilder

In short you have three things to do:

  • Identify the assemblies that contains the objects you want to generate test data for.  The path to the files can be anywhere on your system.  For convenience there is an folder called Assembly that you can copy the files to.  Multiple assemblies from different locations can be imported.
  • Create the import statements.
  • Create the code snippet with the NBuilder statements that will generate your data.

Here’s a screen shot of DataBuilder with each section that corresponds with the three goals stated above.

And here is an example that we’ll be working with.

var agents = Builder<SalesAgent>.CreateListOfSize(5)
                    .WhereTheFirst(1)
                         .Have(x => x.FirstName = "James")
                         .And(x => x.LastName ="Kirk")
                    .AndTheNext(1)
                          .Have(x => x.FirstName = "Bruce")
                          .And(x => x.LastName = "Campbell"})
                    .Build()
                    .ToList();

parameters["JsonDataSet"] = JsonConvert.SerializeObject(agents);

Note that after the end of the code that creates the objects, you need to include a statement

parameters["JsonDataSet"] = JsonConvert.SerializeObject(List);

Without that statement you will not get your data serialized.  If you’ve entered the data as shown, hit the Build button and the resulting JSON is placed in the output box.  That’s it.  Looking through the output you’ll note that the first two sales dudes are James Kirk and Bruce Campbell, while the remaining records are completed by NBuilder.

[{"FirstName":"James","LastName":"Kirk","Salary":1.0,"RegionId":1,"RegionName":"RegionName1","StartDate":"\/Date(1287892800000-0400)\/"},{"FirstName":"Bruce","LastName":"Campbell","Salary":2.0,"RegionId":2,"RegionName":"RegionName2","StartDate":"\/Date(1287979200000-0400)\/"},{"FirstName":"FirstName3","LastName":"LastName3","Salary":3.0,"RegionId":3,"RegionName":"RegionName3","StartDate":"\/Date(1288065600000-0400)\/"},{"FirstName":"FirstName4","LastName":"LastName4","Salary":4.0,"RegionId":4,"RegionName":"RegionName4","StartDate":"\/Date(1288152000000-0400)\/"},{"FirstName":"FirstName5","LastName":"LastName5","Salary":5.0,"RegionId":5,"RegionName":"RegionName5","StartDate":"\/Date(1288238400000-0400)\/"}]

You also can load a script and execute it as well.  That’s done on the “Script Loader” tab.  The location of the scripts is set in the WebConfig and the key name is ScriptPath.  Here’s the screen shot:

Anatonomy of DataBuilder Script

Here’s the complete C# script file that builds your data.  It’s just a class:

//CSScript directives - DO NOT REMOVE THE css_ref SECTION!!!
//css_ref System.Core;
//css_ref System.Data.ComponentModel;
//css_ref System.Data.DataSetExtensions;
//css_ref System.Xml.Linq;

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using DataGenerator.Core;
using DataGenerator.ObjectTypes;
using DataGenerator.ScriptHost;
using System.Linq.Expressions;
using System.Linq;
using Newtonsoft.Json;
using FizzWare.NBuilder;
//  Add a reference to your assemblies as well!!
using UserDeploymentDomain;

public class CreateTestFile : IScriptRunner
{
    public void  RunScript(Dictionary parameters)
    {
        var agents = Builder.CreateListOfSize(5)
                    .WhereTheFirst(1)
                         .Have(x => x.FirstName = "James")
                         .And(x => x.LastName = "Kirk")
                    .AndTheNext(1)
                          .Have(x => x.FirstName = "Bruce")
                          .And(x => x.LastName = "Campbell")
                    .Build()
                    .ToList();

        parameters["JsonDataSet"] = JsonConvert.SerializeObject(agents);
    }

    public void  RunScript()
    {
 	    throw new NotImplementedException();
    }
}

The very top section “CSScript Directives” is required by CS-Script.  These are directives that instruct the CS-Script engine to include assemblies when it compiles the script.  The imports section is pretty straight forward.

You’ll note that the script inherits from an interface.  This is a convention used by CS-Script to allow the host and script to share their respective assemblies.  Sensei will discuss that in next post.  The RunScript method accepts a Dictionary that contains the parameters.  This will house the JsonDataSet that is expected for the screen to display the output of your data.

Advanced NBuilder Experiments
The beauty of NBuilder is that you can create test data that goes beyond “FirstName1”, and allows you to quickly create data that matches what the business users are used to seeing. If you think about it you should be able to generate test data that will exercise any rules that you have in the business domain, such as “Add 5% tax when shipping to New York”. With the scripting capability of DataBuilder you can create suites test data that can evolve as you test your system. You could also use the JsonDataSet to create mocks of your objects as well, maybe use them for prototyping your front end.

We’ll do a quick sample. Our scenario is to create assign real regions to sales agents. Furthermore, we want to only chose a range of regions and assign them at random.

First we build the Regions:

var regions= Builder<Region>.CreateListOfSize(4)
	.WhereTheFirst(1)
		.Have(x => x.State = "Texas")
	.AndTheNext(1)
		.Have(x => x.State = "California")
	.AndTheNext(1)
		.Have(x => x.State = "Ohio")
	.AndTheNext(1)
		.Have(x => x.State = "New York")
	.Build();

Now we’ll create a SalesAgents and using the Pick method from NBuilder we’ll randomly assign a region to the sales agents:

var agents = Builder<SalesAgent>.CreateListOfSize(5)
                    .WhereAll()
                           .HaveDoneToThem(x => x.RegionName = Pick.RandomItemFrom(regions).State)
                    .WhereTheFirst(1)
                         .Have(x => x.FirstName = "James")
                         .And(x => x.LastName = "Kirk")
                    .AndTheNext(1)
                          .Have(x => x.FirstName = "Bruce")
                          .And(x => x.LastName = "Campbell")
                    .Build()
                    .ToList();

The result set now has the range of states distributed to the Sales Agents. Looks like James Kirk needs to cover Texas. You may need to view the source to see the output.

[{"FirstName":"James","LastName":"Kirk","Salary":1.0,"RegionId":1,"RegionName":"Texas","StartDate":"\/Date(1287892800000-0400)\/"},{"FirstName":"Bruce","LastName":"Campbell","Salary":2.0,"RegionId":2,"RegionName":"Texas","StartDate":"\/Date(1287979200000-0400)\/"},{"FirstName":"FirstName3","LastName":"LastName3","Salary":3.0,"RegionId":3,"RegionName":"California","StartDate":"\/Date(1288065600000-0400)\/"},{"FirstName":"FirstName4","LastName":"LastName4","Salary":4.0,"RegionId":4,"RegionName":"California","StartDate":"\/Date(1288152000000-0400)\/"},{"FirstName":"FirstName5","LastName":"LastName5","Salary":5.0,"RegionId":5,"RegionName":"Ohio","StartDate":"\/Date(1288238400000-0400)\/"}]

Configure DataBuilder For Your Environment
Given that DataBuilder is loading assemblies you will want to run it on either your dev environment or on a test server where your co workers won’t mind if you need to take IIS up and down. Also, you’ll want to work with a copy of your assemblies in case you need to make a quick change. There are times when IIS will not release a file and if you need to make changes to the assemblies themselves it’s more convenient to copy them after you’ve re-compiled.

There are two settings you need to change in the WebConfig to match your environment.

ScriptPath – Point this to the share where you want to save any scripts. DataBuilder will scour the directory and list anything you place in there.

FizzWarePath – This needs to point to the location of the NBuilder dll. Most likely this will be the bin folder of the DataBuilder website. In the follow up post Sensei will explain what this does.

Wrapping Up For Now

We covered a lot on the whirlwind tour of DataBuilder.  There’s a lot more that is of interest, particularly with respects to the embedded scripting aspects provided by CS-Script.  For now, have fun playing building you data sets.  In the next installment we’ll cover the scripting aspect in more detail  For now, download and experiment.  Here’s the source for DataBuilder with unit tests.

Deserializing to Persistent AnonymousTypes with JSON.Net October 9, 2010

Posted by ActiveEngine Sensei in .Net, .Net Development, ActiveEngine, C#, Problem Solving.
Tags: , , ,
1 comment so far

A few weeks back Sensei unleashed a crazy idea regarding a class AnonymousType that could persist values from an anonymous object.  The AnonymousType, created by Hugo Benocci models an individual object.  In a sense this is a hyper-charged Dictionary of properties that represent an object.  It’s meta data.  This is similar to a concept called the Adaptive Object Model, the theory that you create mechanisms to describe what your objects should do.   Instead of having a class for SalesAgent or Car you have classes that represent the classes, attributes, relationships and behavior in your domain.  In other words, you create a meta data modeler and feed it the criteria that would represent SalesAgent, Car, etc.

Having a “sound-of-one-hand-clapping” moment, Sensei realized that while “Persistent AnonymousTypes” was in the title of the post, no mechanism for for serializing the AnonymousType was included!!  “What the …”.  Jeeezz!  “Hell, that should be easy”, Sensei says.  Grab JSON.Net and with elbow grease make it work, right?  Anybody?

One thing that should be immediately clear is that all the meta data is locked up in the AnonymousType object, so you can’t just write:

string json = JsonConvert.SerializeObject(anonymousType);

Instead we need a way represent all the properties of our AnonymousType and preserve each property’s name, it’s type, and the underlying value.  Something like:

public class NameTypeValue
{
  public string Name { get; set; }
  public Type Type{get; set;}
  public object Value { get; set; }
}

And wouldn’t it be nice if we could take a serialized stream of an actual object and convert that into an AnonymousType?  Thinking further ahead, it would rather easy to pass around a list of NameTypeValues as you could easily send and receive this object from a web client or other front end, building yourself a modelling or code generation tool.

Serializing the object depicted above is pretty trivial.  Using a Func<Dictionary<string,object>, string,  string> we can serialize any way wish as with two tiny methods:

public string ToJSON(Func, string, string> function, string jsonObjectName)
{
    return function(_Values, jsonObjectName);
}
///  Method to serialize.  You can come up with your own!!
public string SerializeWithJObject(Dictionary values, string name)
{
  var jsonObject = new JObject();

  foreach (KeyValuePair property in values)
  {
    jsonObject.Add(new JProperty(property.Key, property.Value));
  }

  return jsonObject.ToString();
}

If there is another mechanism for serialization that you wish to use you are free to come up with your own.  For illustration here is the JSON output of an AnonymousType for a sales agent, and followed by the JSON for an actual Agent object:

Agent JSON ==>{“Name”:”Sales Guy Rudy”,”Department”:45}

AnonymousType JSON ==>{  “Name”: “Sales Guy Rudy”,  “Department”: 45}

 

Now that we can simply serialize our AnonymousType with the output matching that of an actual object,  we just need a way to interpret a JSON stream and build an AnonymousType.  Along with discussion, Sensei will talk about the second “sound-of-one-hand-clapping” moment he had when working with JSON.Net.  As you may have already surmised, you need to describe the Type of property in order deserialization to happen.  Sensei didn’t and took a trip to the valley for frustration.

Ok.  We have stream of JSON with the Name, Value and Type of each property for an object.  AnonymousType has a Set method to set a new property:

        /// <summary>
        /// Sets the value of a property on an anonymous type
        /// </summary>
        /// <remarks>Anonymous types are read-only - this saves a value to another location</remarks>
        public void Set(string property, object value) {
            this.Set<object>(property, value);
        }

        /// <summary>
        /// Sets the value of a property on an anonymous type
        /// </summary>
        /// <remarks>Anonymous types are read-only - this saves a value to another location</remarks>
        public void Set<T>(string property, T value) {

            //check for the value
            if (!this.Has(property)) {
                this._Values.Add(property, value);

            }
            else {

                //try and return the value
                try {
                    this._Values[property] = value;
                }
                catch (Exception ex) {
                    throw new Exception(
                        string.Format(
                            AnonymousType.EXCEPTION_COULD_NOT_ACCESS_PROPERTY,
                            property,
                            (value == null ? "null" : value.GetType().Name),
                            ex.Message
                            ),
                            ex);
                }
            }

        }

It’s pretty straight forward to accept a NameTypeValue object and perform:

public void AddProperty(string objectName, NameTypeValue nameTypeValue)
{
 //  Object doesn't exist?  Add.
 if (objects.ContainsKey(objectName) == false)
 {
 objects.Add(objectName, new List());
 }

 var properties = objects[objectName];

 //  All properties are unique
 var existingProperty = properties.Where(x => x.Name == nameTypeValue.Name)
 .SingleOrDefault();

 if(existingProperty == null)
 {
 properties.Add(nameTypeValue);
 }
}

and taking this a step further, a List<NameTypeValue> can supply all properties for an object:

properties.ForEach(x => { anonymousType.Set(x.Name, x.Value); });

Accepting a JSON stream of a List<NameTypeValue> should be easy-cheesey mac-n-peasey.  The first version of this looked like the following:

public AnonymousType DeserializeFromJSONProperties(string objectName, string json)
{
  Enforce.ArgumentNotNull(objectName, "AnonFactory.Deserialize - objectName can not be null");
  Enforce.ArgumentNotNull(json, "AnonFactory.Deserialize - json can not be null");

  List propertyList = JsonConvert.DeserializeObject
>(json);

  //  Add properties.  Make sure int is not deserialized to a long since JSON.Net
  //  makes best guess
  propertyList.ForEach(x => AddProperty(objectName, x));

  return CreateAnonymousType(objectName);
}

But one-moooorrree-thing!  Sensei discovered that JSON.Net, when presented with an integer like 5, will deserialize to the largest possible type when not presented with a target.  In other words, when you have this JSON:

{“Department” : 45}

and deserialize to an object, it must accommodate the largest possible type in order to avoid truncating the data.  That means an int is deserialized as Int64!!  The first round of testing was quite aggravating as AnonymousType would accept the property into it’s schema, but when you went to fetch that value later on you would get an exception  In other words, when you did this:

//  Found in JSONTests.MakeItFail()
var anonFactory = new AnonFactory();
var darrellDept = new NameTypeValue();
darrellDept.Name = "Department";
darrellDept.Value = 45;

var darrellName = new NameTypeValue();
darrellName.Name = "Name";
darrellName.Value = "Darrell";

var propertyList = new List();
propertyList.Add(darrellDept);
propertyList.Add(darrellName);

//  Create JSON stream of properties
string darrellPropertyJSON = JsonConvert.SerializeObject(propertyList);

//  Try to deserialize and create an AnonymousType object
var otherDarrell = anonFactory.DeserializeFromJSONProperties("Agent", darrellPropertyJSON);
Assert.AreEqual(otherDarrell.Get("Department"), 45);

you got an InvalidCastException.

Luckily you have the Type so you can perform a conversion as you deserialize the property and add it to AnonymousType’s Dictionary<string, object>.  Here’s the new version:

propertyList.ForEach(x => AddProperty(objectName, ConvertTypeFromDefinition(x)));

private NameTypeValue ConvertTypeFromDefinition(NameTypeValue nameTypeValue)
{
  if (nameTypeValue.Type != nameTypeValue.Value.GetType())
  {
    nameTypeValue.Value = Convert.ChangeType(nameTypeValue.Value, nameTypeValue.Type);
  }

  return nameTypeValue;
}

When you look at the new version of the AnonymoustType project you’ll note that serializing is handled by the AnonymousType itself, while a factory class is used for building the an AnonymousType from the NameTypeValue’s and for deserializing JSON as well.  Sensei struggled a bit with this, as on the one hand if AnonymousType was responsible for serializing itself should it also be able to deserialize a stream?  On the other hand, a factory seemed logical since you could have a registry of AnonymousType objects, thereby centralizing the creation and management of AnonymousTypes.  Don’t like it – create your own and share!  Regardless, looks like we can fit through the mini-Stargate now.  Here’s version 2.

%d bloggers like this: