jump to navigation

jsRazor – New Way of Rendering Json That Is Pure Genius May 9, 2013

Posted by ActiveEngine Sensei in ActiveEngine, Ajax, jQuery, JSON.Net, KnockoutJS, Open Source, Problem Solving.
Tags: , , , , ,
add a comment

DClick this link immediately, and you’ll see wonderful things! Sensei has said many times that simple is better, and rgubarenko of MakeItSoft has a brand spanking new template engine that is so simple it makes you speechless.  His premise:  every possible rendering task can be accomplished as a random combination of two functional primitives:

repeat – repeats fragment of HTML code for each object in array
toggle – shows or hides fragment of HTML depending on boolean flag

Here is an example taken from the github site.  To get the output display below:

jsRazor Example Output

from a Json data set that looks like this:

var data_Themes = [ { name: "Dreaming Theme", colors: ["#2A1910", "#9E7064", "#B0967C", "#E7435E", "#6D4F3F"] },

{ name: "Moth Theme", colors: ["#30382D", "#565539", "#78765F", "#403F2B"] },

{ name: "5 Dark Theme", colors: ["#000000", "#280705", "#2E0500", "#3B0000", "#3C1100"] },

{ name: "Blue Volcano Theme", colors: ["#5077FF", "#8A84FF", "#81C1FF"] } ];

You will have a template that looks like this:

<pre><div id="example">
      <div class="name">{name} ({CountColors} colors)</div>
      <div class="wrap">
        <div class="color" style="background-color:{item};">
          <span style="color:white">{item}</span>
          <span style="color:black">{item}</span>
        <div class="rgb">({R},{G},{B})</div>

The controller that creates the final result is as follows:

<pre>// get initial template
var tmp = document.getElementById("example").innerHTML; 
// repeat theme objects (pass array of themes to repeat functional)
tmp = $.jsrazor.repeat(tmp, "themes", data_Themes, function (tmp, idx, item)
  // repeat inner color objects (pass array of colors of the current theme item)
  tmp = $.jsrazor.repeat(tmp, "colors", item.colors, function (tmp, idx, item)
    // use toggle to show dark or light color text (to be contrast with background)
    tmp = $.jsrazor.toggle(tmp, "dark", hex2rgb(item).mid <= 128);
    tmp = $.jsrazor.toggle(tmp, "light", hex2rgb(item).mid > 128);
    // output RGB representation of the color as custom value
    tmp = tmp
      .replace("{R}", hex2rgb(item).r)  // red
      .replace("{G}", hex2rgb(item).g)  // green
      .replace("{B}", hex2rgb(item).b); // blue
    // return processed template for current INNER item
    return tmp;
  // color counter is not a part of JSON, so we output it as custom value
  tmp = tmp.replace("{CountColors}", item.colors.length);
  // return processed template for current item
  return tmp;
// put processed output back
document.getElementById("example").innerHTML = tmp;</pre>

This looks very promising.  In fact, this could be a very cool way of writing reports in jQuery!  Think about it – with a tool like jLinq for filtering data sets and jsRazor, you could replace SQL Server Reports.

Simplify Your Thoughts For Uninterrupted Flow April 24, 2013

Posted by ActiveEngine Sensei in ActiveEngine, Agile, Coaching, New Techniques, Open Source, Personal Development, Problem Solving.
Tags: , , , ,
add a comment

Sensei recently gave up FogBugz. This was not because of FogBuz, as it is a great product. But Sensei realized that it was not meeting his needs. It was too much. When on the hunt, you can’t be slowed down, and sometimes you have to jettison the extra weight. To be fair, the context here is a prototyping project, where errors / foibles / new features need to be captured. FogBugz is great a teams, but it does require, well, too many clicks. You should always ask yourself this question: which James Bond do I want to be?

doctor-no1_lg bond_large_verge_medium_landscape



Which Bond gets the babe? Pretty easy choice. The unfettered thinker makes them swoon. The guy with the helmet …not so much.

Keeping It Real By Keeping It Simple

Yep – Sensei sounds like a whiny Apple-simplify-your-life-and-wear-a-black-turtle-neck Zen iPad fan boy. Well, that’s not right either. There’s just the right tools for the the right job. So when in the fight with the development environment, brain firing on all cylinders, Seseni uses Workflowy. You can quickly categorize your lists / sentences / thoughts as you go. Just typing, no modal dialog boxes, no creating an item, waiting for it to save, clicking, scrolling, more dialog boxes.

Before you attack, Sensei is not saying this will work for teams, for bug resolution, and other endeavors that FogBugz does very well. But it’s all about eliminating the tactics that get in the way of you achieving your goals. This is critical. And when prototyping you need as much room in your head as possible so you solve the bugs, but not spend more time tracking the bugs. Below is a sample. Issues and features, pretty easy. Click it to see the details.


So What? Well, How About Taking It a Step Further

Sensei hopes that the enterprising readers out there can take this idea and run with it: Why not create system that parses the format shown above? When you edit, each line gets a Guid. Then, start at the top level. Each item at that level is story or a deliverable, maybe broken down by screen or function. A child of each story will have an Issues or Features item, and the child items of Issues naturally belongs to Issues. All else would be ignored when converting to a database record, yet retained in your notes.

bond_01_slide-9f8039ebe14dba26b56314bd850026a107423d9c-s6-c10 - CopyThis would be your starting pointing. Because each of these items has an identifier, later you could parse them into a database format, assign people, etc. The point is that the starting point is easier, is more productive because you just type. That way your work gets done, and you feel more like him.

NuGet Causes Needless Headaches With RavenDB, RestSharp and Json.Net September 25, 2012

Posted by ActiveEngine Sensei in .Net, ActiveEngine, Mythology, Open Source, Problem Solving.
Tags: , , ,
add a comment

Just because something is shiny and promises riches doesn’t mean you should put it in your pocketses.  NuGet, while good for getting an assembly for quick and dirty research and development, is a tour to the valley of frustration when you need to focus and deploy a solution with many dependencies.

NuGet has a pretty dialog box so you think you can just “wire up” and go, but this is an illusion my precious.  Sometimes it doesn’t update the Hint path.  Sometimes it forces you to install packages in certain order, as with SignalR, RestSharp and RavenDB.

Think about it – NuGet pulls down files, creates directories and then references to all the paths that were just created.  We can’t do that ourselves by pulling a directory, the adding the reference to old fashioned way, meaning the way that leaves you feeling more in control, not requiring you to spend hours hunting references in package.config files in different folders?  The main premise is to eliminate this practice, right?  What is going to happen when you introduce AppHarbor into the mix where you won’t be able to look and the production server to divine what assemblies are indeed deployed!

What Works Versus What Makes You Feel Cool

Sensei wants to direct you to a solution where a developer needed to use two versions of Json.Net with RavenDB.  His solution:  add dependentAssembly tag to the web.config, upload the file to a sub directory of the bin folder, and move on.

Ok, so it’s not cool, but the job is done.  And that should be the whole point.  Why use tools that put us in this mess in the first place?  Unfortunately NuGet is becoming the primary way to get assemblies.  The road may go ever on but projects have a deadline!!  Poor Smeagul.

A Great Post on Managing Branches in Git September 20, 2012

Posted by ActiveEngine Sensei in AppHarbor, Open Source, Problem Solving.
Tags: , , ,
add a comment

Vincent Driessen has written an amazing post on managing branches in Git, and how to effectively avoid the fire drill that we’ve been through when we have to push a hot-fix into production while preserving new development.  It is titled A Successful Git Branching Model.

This can be very critical if you are using AppHarbor, a .Net cloud platform where you push your source code from Github to AppHarbor and the AppHarbor elves build it and push to a production server if the app passes its unit tests.  AppHarbor maintains a history of builds so that you can -ahem – roll back on the very occasion that you have made a mistake.  But this only speaks to Vincent’s point that if you configure Git with the appropriate branches to handle development, production and hot-fixes you should use the rollback feature less.

Sensei has been spending a lot time working with Git and AppHarbor and one of the next milestone on his journey will be to learn how to disable auto push for the development branch of his project while maintaining the synchronization with the master branch.  This should be the subject of up-coming, shouldn’t it?  Or maybe one of you, the dear readers of this blog has answer they’d like to share?

Some Pitfalls To Avoid With KnockoutJS “options” Binding September 8, 2012

Posted by ActiveEngine Sensei in ActiveEngine, Ajax, KnockoutJS, Mythology, New Techniques, Open Source, Tutorial.
Tags: , , , , , , , , , ,
add a comment

As Sensei has written earlier, KnockOutJS is a great framework for creating rich client side solutions for you web applications.  Simply said it cuts your development down considerably by performing CSS binding for you, while also bringing better structure to your Javascript through the use of the MVVM pattern.  But even the greatest of all wizardry, magery, grammary, magik has its stumbling blocks.  Each tool you use constrains you in some way.  This week Sensei uncovered another puzzle that has left him wondering still if he found the best solution.  Maybe you will have some insight you can share with the “options” binding from Knockout.

Here is the scenario:  You have a <select> ( or a drop down list as us old school Windows devs are found of saying) that you wish to populate with values from an array.  There are two ways that this select list will be used.  The first is when you create a record, and naturally you would like the list to display “New …”.  The second goal is set the value of the <select> to match the value of a current record.  Here is the JS-Fiddle with the first attempt. Selecting from the list sets the value, and you’ll the update at the bottom. Clicking “Simulate Editing Amys Record” will set the value of the list to “Amy” as though you were performing an edit operation.  Here is the view model code:

var ViewModel = function() {
var self = this;

// Simulated seed data from server
this.seedData = ko.observableArray([
ID: 1,
firstName: 'John',
value: '333'},
ID: 2,
firstName: 'Bob',
value: '333'},
ID: 3,
firstName: 'Amy',
value: '333'}]),

// Simulated data from server
self.data = {
title: ko.observable('This is a sample'),
selectedValue: ko.observable("")

self.prepForNew = function() {

self.changeIt = function() {


var vm = new ViewModel();

Now bear with Sensei as he describes the behavior that was so confounding.  We are initializing the <select> by setting data.selectedValue(“”).  The first time the page is displayed we get the behavior that we want.  Click the “Set List for New Record List”.  Nothing happens.  If you trace with Firebug you’ll see that the value is indeed being set, but once it leaves the method it reverts to the current value in the <select>.

Speak, friend, and enter …

Oookkay. Scratch your head. Walk away. Come back, fiddle so more. Rinse, then repeat for about 5 hours.  This shouldn’t be.  In his frenzy Sensei did not consult StackOverflow.  At last an idea came to mind.  Why not add “New …” as the first entry in <select>, give a value of -1 and an ID of -1.  This way at least it is identifiable.  Seems silly but when you have things to accomplish sometimes you just have to eat the sausage instead of thinking about how its made.  Check out the new JS Fiddle before Sensei explains.

Three simple changes have occured.  First we cheate by adding a new object to the array that supports our <select>.  We gave it the “New …” as the first element.

this.seedData = ko.observableArray([
//  Here is the default caption object
ID: -1,
firstName: 'New ...',
value: ''},
ID: 1,
firstName: 'John',
value: '333'},
ID: 2,
firstName: 'Bob',
value: '333'},
ID: 3,
firstName: 'Amy',
value: '333'}]),

We yanked out the “optionsCaption: ‘New …” entry in the HTML mark for the view.  Finally the altered the method self.prepForNew to set the value of selectedValue to “New …” with the statement selectedValue(“New …”);  This forces KnockOut to sync to what we want.  Remember that we are working with methods when setting values with Knockout, hence the use of (“New …”);

Sensei is happy to have things working.  Being perplexed over finding out the cause instead of simply creating something simple did fret away the hours.  Like with any new tool, there are nuances that won’t become apparent until you are hit over the head with their pitfalls.

Getting KO’ed with KnockoutJS August 31, 2012

Posted by ActiveEngine Sensei in ActiveEngine, Ajax, Approvaflow, ASP.Net, DataTables.Net, jQuery, New Techniques, Open Source.
Tags: , , , , ,

ImageOn the quest to provide a rich user interface experience on his current project, Sensei has been experimenting with KnockoutJS by Steve Sanderson.  If you haven’t reviewed it’s capabilities yet  it would be well worth your while.  Not only has Steve put together a great series of tutorials, but he has been dog fooding it with Knockout.  The entire documentation and tutorial set is completed used Knockout.  Another fine source is Knockmeout.net by Ryan Niemeyer.  Ryan is extremely active on StackOverflow answering questions regarding Knockout, and also has a fine blog that offers very important insight on developing with this framework.

KnockoutJS is a great way to re-organize your client side code.  The goal of  this post is not to teach you KnocoutJS; rather, Sensei wants to point out other benefits – and a few pitfalls – to adopting its use.  In years past, it’s been difficult to avoid writing spaghetti code in Javascript.  Knockout forces you to adopt a new pattern of thought for organizing your UI implementation.  The result is a more maintainable code base.  In the past you may have written code similar to what Sensei use to write.  Take for example assigning a click event to a button or href in order to remove a record from a table:

      <td><a onclick="deleteRecord(1); return false;" href="#">Customer One</a></td>
      <td>1313 Galaxy Way</td>
      <td><a onclick="deleteRecord(2); return false;" href="#">Customer Two</a></td>
      <td>27 Mockingbird Lane</td>

<script type="text/javascript">
function deleteRecord(id){
  //  Do some delete activities ...

You might even went as far as to assign the onclick event like so:

  $("tr a").on('click', function(){
    //  find the customer id and call the delete record

The proposition offered by Knockout is much different.  Many others much more conversant in design patterns and development than Sensei can offer better technical reasons why you sound use Knockout.  Sensei likes the fact that it makes thinking about your code much simpler.  As in:

<td><a data-bind="click: deleteRecord($data)" href="#">Customer One</a></td>

Yep, you have code mixed in with your mark up, but so what.  You can hunt down what’s going on, switch to your external js file to review what deleteRecord is supposed to do.  It’s as simple as that.  Speaking of js files, Knockout forces you to have a more disciplined approach organizing your javascript.  Here is what the supporting javascript could look like:

var CustomerRecord = function(id, name){
  //  The items you want to appear in UI are wrapped with ko.observable
  this.id = ko.observable(id);
  this.name = ko.observable(name);

var ViewModel = function(){
var self = this;
  //  For our demo let's create two customer records.  Normally you'll get Json from the server
  self.customers = ko.observableArray([
    new CustomerRecord(1, "Vandelay Industries"),
    new CustomerRecord(2, "Wiley Acme Associates")

  self.deleteRecord = function(data){
    //  Simply remove the item that matches data from the array

var vm = new ViewModel();

That’s it.  Include this file with your markup and that’s all you have to do.  The html will change too.   Knockout will allow you to produce our table by employing the following syntax:

<tbody data-bind=”foreach: customers”>
<td><a href=”#” data-bind=”click:  deleteRecord($data)”><span data-bind=”text: id”></span></a></td>
<td><span data-bind=’text: name”></span></td>

These Aren’t the Voids You’re Looking For

So we’re all touchy feely because we have organization to our Javascript and that’s a good thing.  Here’s some distressing news – while Knockout is a great framework, getting the hang of it can be really hard.  Part of the reason is Javascript itself.  Because it’s a scripting language, you end up with strange scenarios where you have a property that appear to have the same name but different values.  You see, one of the first rules of using Knockout is that observables ARE METHODS.  You have to access them with (), as in customer.name(), and not customer.name.  In other words, in order for you to assign values to an observable you must:

customer.name("Vandelay Industries");

//  Don't do this - you create another property!!

customer.name = "Vandelay Industries";

What? Actually, as you probably have surmised, you get .name() and .name, and this causes great confusion when you are debugging your application in Firebug.  Imagine you can see that customer.name has a value when you hit a breakpoint, but its not what you’re looking for.  Sensei developed a tactic to help verify that he’s not insane, and it works simply.  When in doubt, go the console in Firebug and access your observable via the ViewModel; so in our case you could issue:


When name() doesn’t match your expectation you’ve most likely added a property with a typo.  Check with


It sounds silly, but you can easily spend a half hour insisting that you’re doing the right thing, but you really confusing a property with a method.  Furthermore, observable arrays can also be a source of frustration:

// This is not the length of the observable array. It will always be zero!!!
vm.customers.length == 0;

// You get the length with this syntax

Knock ’em inta tamarra, Rocky

Had Sensei known the two tips before starting he would have save a lot of time.  There are many others, and they are best described by Ryan Niemeyer in his post 10 things to know about Knockout from day one.  Read this post slowly.  It will save you a lot of headache.  You may familiar with jQuery and Javascript, but Knockout introduces subtle differences that will catch you off guard.  That’s not a bad thing, it’s just different than what you may be used to.  Ryan also makes great use of JS Fiddle and answers most of his StackOverflow questions by using examples.  Those examples are in many cases easier to learn from than the tutorial since the scope is narrower than the instruction that Steve Sanderson gives.  It really allows you play along as you learn.

More RavenDB Resources January 3, 2012

Posted by ActiveEngine Sensei in .Net Development, C#, New Techniques, Open Source, RavenDB.
Tags: , ,

Daniel Lang has a great post regarding how to handle relations in RavenDB.  He emphasizes that a document database is vastly different from a relation database and illustrates various scenarios of do’s and don’ts.  Go read it now.

ApprovaFlow: A Quick Primer On RavenDB December 15, 2011

Posted by ActiveEngine Sensei in .Net, .Net Development, ActiveEngine, Approvaflow, ASP.Net, Business Processes, JSON.Net, Open Source, RavenDB, Workflow.
Tags: , , , , ,

This post focuses on getting started with RavenDB, so we’ll set aside our focus on workflows for a bit.  It’s included in the ApprovaFlow series as it is an important part of the workflow framework we’re building.  To follow along you might want to get the source code.

RavenDB is a document database that provides a flexible means for storing object graphs.  As you’ll see a document database presents you with a different set of challenges than you are normally presented when using a traditional relational database.

The storage “unit” in RavenDB is a schema-less JSON document.  This takes the form of:  Because you are working with documents you now have the flexibility to define documents differently; that is, you can support variations to your data without have to re-craft your data model each time you want to add a new property to a class.  You can adopt a “star” pattern for SQL as depicted here, but querying can become difficult.  Raven excels in this situation and one such sweet spot is:

Dynamic Entities, such as user-customizable entities, entities with a large number of optional fields, etc. – Raven’s schema free nature means that you don’t have to fight a relational model to implement it.

Installing and Running RavenDB

The compiled binaries are easy to install.  Download the latest build and extract the files to a share.  Note that in order to run the console you are required to install Silverlight.  To start the server, navigate to the folder[] and double click “Start.cmd”.  You will see a screen similar to this one once the server is up and running:

The console will launch it self and will resemble this:

How To Start Developing

In Visual Studio, reference Raven with Raven.Client.Lightweight.  For CRUD operations and querying this will be all that you will need.

First you will need to connect to the document store.  It is recommended that you do this once per application.  That is accomplished with

var documentStore = new DocumentStore {Url = "http://localhost:8080"};

Procedures are carried out using the Unit of Work pattern, and in general you will be using these type of blocks:

using(var session = documentStore.OpenSession())
   //... Do some work

RavenDB will work with Plain Old C# Objects and only requires an Id property of type string.  An identity key is generated for Id during this session.  If were were to create multiple steps we would have identities created in succession.  A full discussion of the alternatives to the Id property is here.

Creating a document from your POCOs’ object graphs is very straight forward:

public class Person
    public string FirstName { get; set; }
public string LastName { get; set; }
public string Id { get; set; }
public int DepartmentId { get; set; }
    // ...

var person = new Person();

using(var session = documentStore.OpenSession())

Fetching a document can be accomplished in two manners:  by Id or with a LINQ query.  Here’s how to get a document by id:

string person = "Person/1";  //  Raven will have auto-generated a value for us.
using(var session = documentStore.OpenSession())
   var fetchedPerson = session.Load<Person>(personId);
   //Do some more work

You’ll note that there is no casting or conversion required as Raven will determine the object type and populate the properties for you.

There are naturally cases where you want to query for documents based on attributes other than the Id. Best practices guides that we should create static indexes on our documents as these will offer the best performance. RavenDB also has a dynamic index feature that learns from queries fired at the server and over time these dynamic indexes are memorialized.

For your first bout with RavenDB you can simply query the documents with LINQ.   The test code takes advantage of the dynamic feature.  Later you will want to create indexes based on how you most likely will retrieve the documents.  This is different that a traditional RDMS solution, where the data is optimized for querying.  A document database is NOT.

Continuing with our example of Person documents we would use:

int departmentId = 139;

using(var session = documentStore.OpenSession())
   var people = session.Query<Person>()
                          .Where(x => x.DepartmentId == departmentId)

In the source code for this post there are more examples of querying.

Debugging, Troubleshooting and Dealing with Frustration

Given that this is something new and an open source project you may find yourself searching for help and more guidelines.  One thing to avail yourself of while troubleshooting is the fact that RavenDB has REST interface and you can validate your assumptions – or worse, confirm your errors – by using curl from the command line.  For example, to create a document via http you issue:

curl -X POST http://localhost:8080/docs -d "{ FirstName: 'Bob', LastName: 'Smith', Address: '5 Elm St' }"

Each action that takes place on the RavenDB server is displayed in a log on the server console app.  Sensei had to resort to this technique when troubleshooting some issues when he first started.  This StackOverflow question details the travails.

Another area that threw Sensei for a loop at first was the nature of the RavenDB writing and maintaining indexes.  In short, indexing is a background process, and Raven is designed to be “eventually consistent”.  That means that there can be a latency between when a change is submitted, saved, and indexed in the repository so that it can be fetched via queries.  When running tests from NUnit this code did not operate as expected, yet the console reported that the document was created:


int posttestCount = session.Query<TeamMember>()

According to the documentation you can overcome this inconsistency by declaring that you are willing to wait until RavenDB has completed its current write operation.   This code will get you the expected results:

int posttestCount = session.Query<TeamMember>()
              .Customize(x => x.WaitForNonStaleResults())

Depending on the number of tests you write you may wish to run RavenDB in Embedded mode for faster results.  This might prove useful for automated testing and builds.  The source code provided in this post does NOT use embedded mode; rather, you have need your server running as this gives you the opportunity to inspect documents and acclimate yourself to the database.

There is much more that you can do with RavenDB, such as creating indexes across documents, assign security to individual documents, and much more.  This primer should be enough to get you started.  Next post we’ll see how RavenDB will fit into the ApprovaFlow framework.  Grab the source, play around and get ready for the next exciting episode.


ApprovaFlow – A Proof of Concept March 25, 2011

Posted by ActiveEngine Sensei in .Net, .Net Development, ActiveEngine, Approvaflow, ASP.Net, C#, JSON.Net, LINQ, New Techniques, Open Source.
Tags: , ,
add a comment

Like Tolkien, Sensei wants to create the landscapes, cultures and languages before he writes his next epic. You can be the judge whether the work is a series of sketches and notes like the Silmarillion or cohesive, compelling story that you want read again and again. As a bonus Sensei will deliver working software that hopefully will be of use to you.  (Photo credit – utnapistim).

The epic will be called ApprovaFlow. ApprovaFlow is a framework / process / methodology that allows you to create workflow applications that are easy to deploy and are configurable. With ApprovaFlow Sensei hopes to demonstrate how to readily encorporate the inevitable changes that your users will ask of you. Deliver changes effortlessly and without groans. Cast off the chains inconvenient builds and focus on creating solutions that stay out of the users way.

Ok. Managent wants bullet points so here are our goals for ApprovaFlow:

• Model a workflow in a clear format that is readable by both developer and business user. One set of verbiage for all parties.
•. Allow the state of a workflow to be peristed as an integer, string. Quicky fetch state of a workflow.
•. Create pre and post nprocessing methods that can enforce enforce rules or carry out actions when completing a workflow task.
•. Introduce new functionality while isolating the impact of the new changes. New components should not break old ones
•.Communicate to the client with a standard set of objects. In other words, your solution domain will not change how the user interface will gather data from the user.
•. Use one. aspx page to processes user input for any type of workflow.
•. Provide ability to roll your own customizations to the front end or backend of your application.

There it is. These goals will probably take us a good amount of time to review and implement. Is it worth it? Hell yeah. We’ll end up with one simple project instead of a bloated framework where it takes forever to find anything. A nice by product will be that you can spend more time thinking about how to solve your users problems rather than trying to figure out a monsterous framework that requires a huge investment of energy and time learning how to get simple things done.

How to Create Server-Side Paging for DataTables.Net with ASP.Net December 19, 2010

Posted by ActiveEngine Sensei in .Net, ActiveEngine, C#, DataTables.Net, Fluent, jQuery, New Techniques, Open Source, Problem Solving, Tutorial.
Tags: , , , , , , ,

Source code has been updated!! Read about the changes in Dynamically Select Columns with Server-Side Paging and Datatables.Net If you are new to DataTables.Net and Sensei’s paging solution and want to detailed study of how it works, work through this post first, then get the latest edition.  Note, code links in this post are to the first version.

A central theme for 2010 has been fluency, or the continual practice of certain methods to such a degree that your performance improves and you produce increasingly polished, effective solutions.  For software development this has meant tools to save time and increase quality.  It also means keeping an eye toward making the users of your solutions more efficient as well.  In the spirit of “fluent solutions”, Sensei will end the year with a post that examines how to create a data paging solution for the jQuery data grid plug-in DataTables.Net.

DataTables can turn a HTML table into a fully functional data grid like the one offered by Telerik.  This plug-in offers client side sorting, filtering/ search,  as well as support for server-side processing processing of data.  It is an extremely feature rich tool created by Allan Jardine, and is itself worthy of a series of posts.  For this post on data paging Sensei recommends that you read through these examples to get an idea of what the data paging service needs to achieve.

Let’s get started with the goals we need to achieve when providing server-side data paging support:

  • Send data to client in the multiples or “chunks” that the client requests, and respond when the size of the sets requested is changed by the user.
  • Re-order the data set if the user clicks on a column heading.  Honor the data set size when returning the data.
  • Filter across all columns of data based on user input.  Implement this as partial matches, and again, honor the data set size.

Remember this is about flexibility, so we have the additional goals of:

  • Create a solution that can be reused.
  • Provide a mechanism to accommodate any type of .Net class using generics.

Essentially we want to be able to write code like so:

var tenants = tenantRepository.GetAll();
var dataTablePager = new DataTablePager();
var returnDataSet = dataTablePager.Filter(requestParms, tenants);

Before we proceed, Sensei wants to acknowledge those really smart people whose ideas contributed to this solution:

Zack Owens – jQuery DataTables Plugin Meets C#

Jeff Morris – Using Datatables.net JQuery Plug-in with WCF Services

Dave Ward – ASMX ScriptService mistake – Invalid JSON primitive

You may want to download the source before reading the rest of this post.

Communicating with DataTables

DataTables uses the following parameters when processing server-side data:

Sent to the server:

Type Name Info
int iDisplayStart Display start point
int iDisplayLength Number of records to display
int iColumns Number of columns being displayed (useful for getting individual column search info)
string sSearch Global search field
boolean bEscapeRegex Global search is regex or not
boolean bSortable_(int) Indicator for if a column is flagged as sortable or not on the client-side
boolean bSearchable_(int) Indicator for if a column is flagged as searchable or not on the client-side
string sSearch_(int) Individual column filter
boolean bEscapeRegex_(int) Individual column filter is regex or not
int iSortingCols Number of columns to sort on
int iSortCol_(int) Column being sorted on (you will need to decode this number for your database)
string sSortDir_(int) Direction to be sorted – “desc” or “asc”. Note that the prefix for this variable is wrong in 1.5.x where iSortDir_(int) was used)
string sEcho Information for DataTables to use for rendering

Reply from the server

In reply to each request for information that DataTables makes to the server, it expects to get a well formed JSON object with the following parameters.

Type Name Info
int iTotalRecords Total records, before filtering (i.e. the total number of records in the database)
int iTotalDisplayRecords Total records, after filtering (i.e. the total number of records after filtering has been applied – not just the number of records being returned in this result set)
string sEcho An unaltered copy of sEcho sent from the client side. This parameter will change with each draw (it is basically a draw count) – so it is important that this is implemented. Note that it strongly recommended for security reasons that you ‘cast’ this parameter to an integer in order to prevent Cross Site Scripting (XSS) attacks.
string sColumns Optional – this is a string of column names, comma separated (used in combination with sName) which will allow DataTables to reorder data on the client-side if required for display
array array mixed aaData The data in a 2D array

The data sent back is in the following form depicted below. Note that aaData is merely an array of strings – there is no column information. This will present a challenge in that you will not be able to simply serialize a collection and pass back the results.

    "sEcho": 3,
    "iTotalRecords": 57,
    "iTotalDisplayRecords": 57,
    "aaData": [
            "Firefox 1.0",
            "Win 98+ / OSX.2+",
            "Firefox 1.5",
            "Win 98+ / OSX.2+",

As you may be aware, if you wish to use ASP.Net web services to serialize JSON you must POST to the service and instruct it to interpret your parameters as JSON. DataTables will POST variables as value pairs and this won’t work for us when POSTing to a web service. We’ll have to translate the variables to a usable format. Luckily DataTables allows us to intervene with the following code, where we create a JSON string by serializing a structure called aoData:

"fnServerData": function ( sSource, aoData, fnCallback ) {

		        	var jsonAOData = JSON.stringify(aoData);

			        $.ajax( {
                                        contentType: "application/json; charset=utf-8",
				        type: "POST",
				        url: sSource,
				        data: "{jsonAOData : '" + jsonAOData + "'}",
				        success: function(msg){
				        error: function(XMLHttpRequest, textStatus, errorThrown) {


Our web service can now de-serialize aoData and parse the appropriate parameters. This gives us important items such as how many records to display, what columns to sort on, and what search terms should be applied in a filter.

DataTablePager Class

DataTablePager.cs is the work horse of our solution.  It will sort, filter and order our data, and as an extra, serialize the results in format required by aaData.  Here’s the constructor:

public DataTablePager(string jsonAOData, IQueryable queryable)
            this.queryable = queryable;
            this.type = typeof(T);
            this.properties = this.type.GetProperties(BindingFlags.Public | BindingFlags.Instance);
            this.aoDataList = new List>();
            this.sortKeyPrefix = new List();


The parameter jsonAOData is the JSON string that contains the variables iDisplayStart, iDisplayLength, etc.  These will be parsed by the method PrepAOData.  The parameter queryable is the collection of records that will be filtered and parsed into JSON format required by DataTables.

The method Filter() coordinates all of the work.  It’s pretty simple what we want to do:  filter our data based on each column containing the search term, sort the result, then pull out the number of records we need to include in the page, and finally convert the collection into the format DataTables understands.

public FormattedList Filter()
            var formattedList = new FormattedList();

            //  What are the columns in the data set
            formattedList.Import(this.properties.Select(x => x.Name + ",")

            //  Return same sEcho that was posted.  Prevents XSS attacks.
            formattedList.sEcho = this.echo;

            //  Return count of all records
            formattedList.iTotalRecords = this.queryable.Count();

            //  Filtered Data
            var records = this.queryable.Where(GenericSearchFilter());
            records = ApplySort(records);

            //  What is filtered data set count now.  This is NOT the
            //  count of what is returned to client
            formattedList.iTotalDisplayRecords = (records.FirstOrDefault() == null) ? 0 : records.Count();

            //  Take a page
            var pagedRecords = records.Skip(this.displayStart)

            //  Convert to List of List
            var aaData = new List>();
            var thisRec = new List();

                    .ForEach(rec => aaData.Add(rec.PropertiesToList()));

            formattedList.aaData = aaData;

            return formattedList;

That said, there is some trickery that goes on in order to make this happen because we are creating a solution to is going to work with any IQueryable to we supply. This means that the filtering and the sorting will need to be dynamic.

To make the filtering dynamic we will build expression trees that will convert each property to a string, convert the string to lower case, then execute a Contains method against the value of that property.  The method GenericSearchFilter() called on line 16 accomplishes this with the following lines of code:

//  Except from GenericSearchFilter
MethodInfo convertToString = typeof(Convert).GetMethod("ToString", Type.EmptyTypes);

 var propertyQuery = (from property in this.properties
        let toStringMethod = Expression.Call(                                          Expression.Call(Expression.Property(paramExpression, property), convertToString, null),                                                            typeof(string).GetMethod("ToLower", new Type[0]))
         select Expression.Call(toStringMethod, typeof(string).GetMethod("Contains"), searchExpression)).ToArray();

We get an array of Expressions that when executed will tell us if the value matches our search term. What we want is to include the item if ANY of the properties is a match, so this means we have to use and OR for all of the properties. That can be accomplished with:

for (int j = 0; j < propertyQuery.Length; j++)
  //  Nothing to "or" to yet
  if (j == 0)
    compoundOrExpression = propertyQuery[0];

  compoundOrExpression = Expression.Or(compoundOrExpression,

So with what is listed above we would be able to match all properties with against a single search term. Pretty cool. But DataTables raises the bar even higher. If you were to go to the samples page and filter using multiple partial words you would find that you could perform some very effective searches with phrases like “new chic”. This would select all records that had properties containing “new” OR “chic”. Imagine the scenario where your user wants to finds all cities “New York” or “Chicago”. We’ve all been there where we have a grid and can only search for one term, or worse, where we have to add a row to a search filter grid and constantly push a “query” button to perform our searches. DataTables does all of the with one search box – just type and the filtering begins.

GenericSearchFilter() handles that scenario. First the search term is parsed into individual terms if there is a ” ”  supplied in the string. This means we will have to perform the propertyQuery for each term that we have. To return all of the records that correspond to each term we still need to perform the OR in groups, but then we need to AND these predicates together so we can get all of the groups per individual term. Here’s the source edited slightly for readability:

//  Split search expression to handle multiple words
var searchTerms = this.genericSearch.Split(' ');

for (int i = 0; i < searchTerms.Length; i++) {    var searchExpression = Expression.Constant( searchTerms[i].ToLower());                   //  For each property, create a contains expression   //  column => column.ToLower().Contains(searchTerm)
  //  Edited for clarity - create the array propertyQuery logic is here ...
  var propertyQuery = ...

  //  Inner loop for grouping all OR's for this search term
  for (int j = 0; j < propertyQuery.Length; j++)   {     //  Nothing to "or" to yet     if (j == 0)     {       compoundOrExpression = propertyQuery[0];     }     compoundOrExpression = Expression.Or(compoundOrExpression, propertyQuery[j]);   }   //  First time around there is no And, only first set of or's   if (i == 0)   {     compoundAndExpression = compoundOrExpression;   }   else   {     compoundAndExpression = Expression.And(compoundAndExpression, compoundOrExpression);   } } 

So GenericSearchFilter will build a humongous expression tree for all the properties in your class. To make this usable for the Where we convert it using Expression.Lambda and our Where clause just goes about its merry way. Because we have used generics, you can supply any class from your assemblies. One caveat, and Sensei is trying to find a resolution. If you have a string property to that is set to null, the expression tree fails. You’ll note that in the classes supplied in the sample, the properties that are of type string in the Tenant class are defaulted to empty in the constructor.  A small price to pay for some great functionality. To sort our data we use the method ApplySort():

 private IQueryable ApplySort(IQueryable records)         {             string firstSortColumn = this.sortKeyPrefix.First();             int firstColumn = int.Parse(firstSortColumn);             string sortDirection = "asc";             sortDirection = this.aoDataList.Where(x => x.Name == INDIVIDUAL_SORT_DIRECTION_KEY_PREFIX +

            if (string.IsNullOrEmpty(sortDirection))
                sortDirection = "asc";

            return records.OrderBy(this.properties[firstColumn].Name, sortDirection, true);

An extension method OrderBy will accept the name of column, the sort direction as parameters. The parameter initial will indicate if we are sorting mulitple times, so we can accomplish multi-property sort with syntax like

var sortedRecords = records.OrderBy("State", "desc", true)
                                      .OrderBy("City", "asc", false);

public static IOrderedQueryable OrderBy(this IQueryable source, string property, string sortDirection, bool initial)
            string[] props = property.Split('.');
            Type type = typeof(T);
            ParameterExpression arg = Expression.Parameter(type, "x");
            Expression expr = arg;
            foreach (string prop in props)
                // use reflection (not ComponentModel) to mirror LINQ
                PropertyInfo pi = type.GetProperty(prop);
                expr = Expression.Property(expr, pi);
                type = pi.PropertyType;
            Type delegateType = typeof(Func<,>).MakeGenericType(typeof(T), type);
            LambdaExpression lambda = Expression.Lambda(delegateType, expr, arg);

            string methodName = string.Empty;

            //  Asc or Desc
            if (sortDirection.ToLower() == "asc")
                //  First clause?
                if (initial && source is IOrderedQueryable)
                    methodName = "OrderBy";
                    methodName = "ThenBy";
                if (initial && source is IOrderedQueryable)
                    methodName = "OrderByDescending";
                    methodName = "ThenByDescending";

            object result = typeof(Queryable).GetMethods().Single(
                    method => method.Name == methodName
                            && method.IsGenericMethodDefinition
                            && method.GetGenericArguments().Length == 2
                            && method.GetParameters().Length == 2)
                    .MakeGenericMethod(typeof(T), type)
                    .Invoke(null, new object[] { source, lambda });
            return (IOrderedQueryable)result;

All good things …
It’s been a long ride, this post. A lot of code discussed, a lot of ground covered. The solution is here.  As always, play around and see how this can help you. If anything breaks, tell Sensei. If you have improvements, tell Sensei. DataTables is a great tool for your arsenal, hopefully the DataTablePager can help you integrate paging with large datasets as part of your solution offering.

Right now Sensei wants to sign off by toasting to you for wading through all of this, and for having the desire to build up your skills.  Obtaining fluency in what you do is a hard road to travel, but it’s worth it because you get things done quicker and better with each session.

Be sure to read about the latest version in “Dynamically Select Columns with Server-Side Paging and Datatables.Net
%d bloggers like this: