jump to navigation

Fluent Thinking – C# Plus SQL Equals CouchDB. Maybe. January 23, 2010

Posted by ActiveEngine Sensei in .Net, ActiveEngine, C#, New Techniques, Open Source, Problem Solving.
Tags: , , , , , , ,

I’ve gone too far to turn around
It’s hard to reach for you
When I’m lying face down
I can’t relieve my soul
I’m lost in a moment
Lying face down

Saving Abel

There are moments when life brings too much possiblilty to your attention and your mind catapults into overload.  Take the great potential that CouchDB has, as an example.  Elegant, simple.  Damien Katz has ignited a real fire storm with his work and may have single handedly given rise to the not-a-RDBMS movement.  Sometimes developing under the gun, Sensei has to admit that the 3rd normal form restrictions can be a little inhibiting.   Really, a collection of value pairs will do quite nicely, and XML, when it doesn’t run on for more that 1 page can be quite descriptive, it still is not quite the solution for quick configuration solutions.  JSON is just about right, easy to manipulate, and this is why Damien chose this as the structure for a document in CouchDB.

“But Sensei, I have SQL Server.  We aren’t allowed to say the L-word!”

“You mean Linux, right?  Just checking.”

There may be constraints imposed by the  environment you have at your disposal.And if you have a current system in production you might have a hard justifying to the powers that be that you should download a LAMP server from JumpBox just so you can introduce doucment managment into your current production system. Or perhaps you have an extensive data model and are concerned that that the meta data would too complex to store with the documents.

Here’s a simple yet effective way to introduce a “CouchDB lite” version to a current database platform that is applicable to SQL Server, Oracle, MySql and provides a flexible way to store meta data for quick retrieval.   It’s not badass like Sam Jackson, but it’s not ridiculous like the beast picture next to him either.

The goal stated succinctly:

  • Provide the abilty to associate a document to a record in an existing DBMS
  • Provide the abilty to store the meta data related to a specific record or a document type.
  • Ensure that future meta data and document types will not a change in the cardinality of the database schema nor require the addition of additional database attributes.
  • Provide a object oriented framework that adheres to the Open-Closed principal; that is, open to extension but closed to modification.

Previously Sensei has opined regarding fluency, and this solution adheres to those prinicipals.  For existing RDBMS a single table can introduce the simplicity and clarity you seek.  All document meta data can be stored in a table aptly named Document:

CREATE TABLE [dbo].[Document](
	[DocumentId] [int] IDENTITY(1,1) NOT NULL,
	[UserId] [int] NOT NULL,
	[Name] [varchar](255) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
	[PostedDate] [datetime] NOT NULL,
	[DocumentHash] [varchar](255) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
	[Attributes] [varchar](8000) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
	[DocumentId] ASC

REFERENCES [dbo].[User] ([UserId])

In brief, this table stores an identity key for each document, the name of the document, a foreign key that represents the user / person who saved the document, and the date when the document was posted.  The attribute DocumentHash is a hashkey comprised of the DocumentId, the UserId, the Name of the document, and the PostedDate.  The DocumentHash value is prepended to the document name to create a unique document.  Consider it a way to associate the document with a specific record in the database.  In other words, you should be able to re-create a haskey from the data attributes that matches the value pre-pended to the file name of the document.  Another nice by-product is that you can version documents quite easily.

Focus next on the column named Attributes, as this is where the magic happens.  The meta data for each document is stored here as JSON.  This takes the following form:


This JSON serilization of an array of Key/Value pairs is a format that allows you store whatever meta data you wish. Notice the first line named “Owners”. This allows you to store the user id’s of all users who have the ability to delete or edit the document.  You can set security based on this attribute, and implement a document centric security framework that can exist in addition to security place on the share where the document is stored.

As stated earlier, any type of data can be stored in the Attributes column.  If you notice line 2 that reads “BonusId” you see that keys to other tables in your database schema that can saved with the document record.  This feature allows you to maintain the relationship of a document back to a particular record or set or records.

You may be asking yourself why the Key Value nomenclature in the JSON string.  The attributes can be deserialized to an object of type List<KVPair> where KVPair  has the following structure:

namespace BusObj
    public class KVPair<TKey, TValue>
        public TKey Key { get; set; }
        public TValue Value { get; set; }

        public KVPair(TKey k, TValue v)
            Key = k;
            Value = v;

        public KVPair() { }

The Key Value pair lends itself to simple querying.  Since the attributes can vary from document to document, the meta data can include identity keys to records in your existing database schema, information regarding folder structure where the documents can be stored, and what ever other piece of important information of you need to retain.  Once the JSON is deserialized to List<KVPair> you can use lambdas to query your documents.  Consider the following code that is used to determine document ownership where the owners are represented as

public bool IsDocumentOwner(int userId)
                                                "Document.IsDocumentOwner - parameter userId can not be null");
            //  No attributes?
            if(this.DocumentAttributes.Count == 0)
                return false;

            return DocumentAttributes.Find(x => x.Key == "Owners")
                                                    .Select(s => int.Parse(s)).ToList()

As you can see, multiple users can be designated as an owner of a document. Any operation that is to be performed on a document can be quickly validated against the owner list, so actions such as deleting or displaying document can be quickly validated.  Since the attributes are stored as JSON, you can easily augment existing documents.

The possiblities here are quite exciting. Perhaps you are looking for a quick an easy way to store configuration of objects but XML would prove to be too complicated to maintain. You could create an entry in the Document table with the attributes you wished to store and retrieve quite easily.  Or, you can easily create a categorization system for documents that sit out on shares and you need to correlate these documents to records in a accounting package.  There are numerous things you could do to make your life easier.

Full source code will be posted soon where an in depth discussion of the implementation will continue.

The Economics of Authority – Windows Workflow and Why You Should Cash In Before Rolling the Dice January 18, 2010

Posted by ActiveEngine Sensei in ActiveEngine.
Tags: , , , ,

Although many measure their success by their salaries, bonuses and bank accounts, this report card can be a limiting factor when starting a new endeavor. If you follow this train of thought too closely your creative muscles will atrophy and very likely you will be unable to capitalize on opportunities as they arise. Or worse yet, you will be unable to recognize the need for you to break out of mold.


Sometimes you can obtain a payment of a different kind from struggling with a new idea: authority, recognition, reputation, knowledge, and maybe skill. For those of you who are feeling limited by your development environments , flexing your creative muscles can be an even greater challenge. There is also a danger for a great many of those who are too comfortable, as they are so accustomed to their current development frameworks they don’t know that they are “out of shape”.

It is too easy to become a subject matter expert, drink the cool aid and parrot every podcast that you hear that technology X is the leading edge solution. Going to the next client offers a new opportunity to build a solution based on the gibberish that was misconstrued as leading edge. If it’s new enough then there is a chance that few others will be able to actually challenge the new silver bullet’s validity as a viable solution. In software we do this all the time. It is a form of lying. While in the short term it may appear that you are enhancing your authority by bringing new solutions to a client, in reality you are diminishing your authority.

2635045177_ab5d39ee34Getting out the big guns for a fight, Sensei is taking a stand as he recently repudiated the recommendation to use Windows Workflow Foundation.


Uno – Dos – Tres Mono! What MonoDevelop Means for your ActiveEngine November 18, 2009

Posted by ActiveEngine Sensei in .Net, ActiveEngine, Linux, Mono, New Techniques, Open Source.
Tags: , ,

Much of Sensei’s philosophy is based on stepping out of your comfort zone.  When practiced frequently you enable yourself to pilot your way through situations where you may have to possess more confidence than facts.  And let’s face it, sometimes it just out right fun to wing it when you test your skills with the unknown.

MonoDevelop is a platform that will do that very thing.  It will change the way you think about your application’s architecture as you can branch out from the Windows stack.  Why not a C# back-end with PHP as your UI / presentation layer?  Why wait for “IronPHP”?  For the curious, MindTouch, an open source Wiki / CMS engine does just that.  It runs on both Windows and Linux with the core engine is written in C#.  This engine includes an embedded script parser written in C#, as well as an implementation of tuples and an excellent XML utility class that will make you wonder why you ever why Micorosoft could not have made something, well, easier.

If you are nodding your head in agreement, but are still skeptical that you have the time to make the leap to Linux, MonoDevelop is deployed with a openSUSE VMWare slice that only requires 512 MB of memory.  Really light weight.  It takes longer to download than it does to spin it up.  It’s just ready to go.  the IDE is based SharpDevelop and comes with Intellisense that admittedly does not play well with some cases Lambda’s and Anonymous Types.  This may become apparent when you import an assembly, and it slowed Sensei down for a bit.  Know that this is a failing of the intellisense, as when you get a clean build and test you’ll find that your code is all right.  You’ll need to have confidence in your code and may have to devise additional tests to validate that everything is working.

Save this failing of the IDE, the remaining features are quite nice.  THE IDE LOADS REALLY FAST!  I mean Microsoft should think this through.  VS Studio, while feature rich, is a dog with respect to speed.  Personally Sensei thinks we’ve been lulled into thinking that 2 GB is a minimum and that’s “progress”.  When compiling, watch your CPU on the host machine.  Remember your Guest OS is using 512 MB and it produces a build quickly.  It’s rather impressive.  For VS on Windows Sensei uses 4 GB of RAM and there are times when he sits waiting.  Hmmm…

MonoDevelop comes integrated with NUnit, and natively supports Boo – yeah, the stuff that Ayende raves about – MoonLight, which is a port of SilverLight, and of course ASP.Net.  MonoDevelop lacks robust refactoring features.  If you have ever used Resharper or DevExpress you’ll have shared the same regrets that Sensei has about having wasted time in the IDE.  The same holds for MonoDevelop.  In fact, there are times when the refactoring add-ins throw exceptions.  Outside your comfort zone you will be.  That was Yoda talk.

So where are we with all the man-ish developer bravado?  Mono can be a lateral jump for you into a new series of platforms.  Using VMWare you can easily setup test environments and fail. Fast.  Yeah, that’s a good thing.  With most Linux builds you have the chance to use Lua, a really fast embedded scripting environment.  Lua is native to a new open source database called Tokyo Cabinet you gives you the choice storing data in Value Pairs, B+Tree, or normal indexed table schema.  Here’s a cool presentation detailing the possibilities.  More comfort zone?  How about NHibernate and Spring.Net?  They have been reportedly ported to Linux with Mono.

Wrapping it up ’cause Sensei’s running long.  Just do it.  There are many possibilities here that you are missing if you stay safe.  There are many views that are beyond Microsoft’s vision that  they provide for you.  And that’s the point.  You’ll be more agile on your next gig if you’ve already solved problems beyond the scope of normal.  Yoda translation:  “Agile you will be.”

TDD and Design Pattern Screencasts by Jean Paul Boodhoo December 21, 2007

Posted by ActiveEngine Sensei in .Net Development, ActiveEngine, Design Patterns, TDD, Test Driven Development.
Tags: , , , , ,

If you have read this blog, you’ll have noted that Jean Paul Boodhoo is a great resource to learn from. Below is a compilation of screencasts that he completed for DNR TV with Carl Franklin. You should take the time to review these, as you will see clearly that JP has dedicated himself to refining the art of programming. He himself freely admits that he has had to struggle to learn and develop his skills to the point where they are today. Watching him apply TDD with Resharper is utterly amazing.

You should take the time to view each of these and share them with your team members, as it will jump start your interest in enhancing your development discipline. Work on your core toolkit and improve your ActiveEngine!


Demystifying Design Patterns Part 1

Demystifying Design Patterns Part 2

Demystifying Design Patterns Part 3

Demystifying Design Patterns Part 4

Demystifying Design Patterns Part 5

Food For Thought … December 8, 2007

Posted by ActiveEngine Sensei in .Net Development, ActiveEngine, C#, Mythology, Problem Solving.
Tags: , , , , ,
add a comment

This post by Stevey discusses Java but could be easily applied to C#. For any of you old-time procedural coders that the Alt.Net bullies like to kick around, here may lie some vindication:

Execution in kingdom of nouns

ActiveEngine Sensei’s Mother Comes to Visit – Lessons in Design Patterns December 5, 2007

Posted by ActiveEngine Sensei in ActiveEngine, C#, Design Patterns, Mythology.
Tags: , ,
add a comment

For Thanksgiving Sensei’s mother came to visit the dojo, but this time she brought a new addition to the family, her dog. This puppy is lovable enough, and like all puppies needs training, which Sensei felt compelled to provide. When Sally got out of line and starting biting, Sensei held her down gently but firmly around the collar bone. This is an old trick that Cesar the Dog Whisperer practices with the smaller dogs – firm, gentle and consistent.

After two times, Sally stopped nipping at Sensei. “Wow, that really works quickly”, Sensei’s mother says.

“It’s easy to do, here’s how, ” and he showed her. The next time Sally started nipping, Sensei’s mother repeated the maneuver many times and lo and behold, Sally learned to not bite.

There are some lessons to be learned here for those who wish to practice Agile, Test Driven Development and implement design patterns: you must learn good technique from good practitioners, then repeat, repeat, repeat while seeking feedback. There’s no short cut to internalizing execution, and certainly no short cut to understanding when and why to apply design patterns. This begs the question for senior team members when do you make design decisions and what justification can you convey to junior team members concerning your choices?

Sometimes this can be very easy. The decorator pattern is a great example, as the justification answers the question of “I have many variations to accommodate in a billing process, and do not want a proliferation of classes or difficult logic to support all the exceptions.” To make this connection between technical implementation with business requirements takes practice, and the articulation of the justifications is a great way to obtain fluency. Repetition and review and those key ingredients that help create a powerful ActiveEngine toolkit.

New Blood in Deadwood December 4, 2007

Posted by ActiveEngine Sensei in .Net, ActiveEngine, C#, Problem Solving.
Tags: , ,
add a comment

Somehow this blog is taking on a real western theme, yet there are moments interspersed with Budo philosophy. Who knows, maybe we’ll even have Kung-fu. No jokes about Neo, so be warned.

Anyway, The Code Slinger has two great posts on some work he has been doing with dymanic object instantiation and MSIL. True to the philosophy of ActiveEngine, he is asking why 5 times and proving that his ideas work. Check out his posts:

Dynamic Object Instantiation Part I
Dynamic Object Instantiation Part II

Surely you must be joking … November 16, 2007

Posted by ActiveEngine Sensei in .Net Development, ActiveEngine, C#, Problem Solving.
Tags: , , , , ,
add a comment

It takes all types of people to run a corporation, but only certain individuals can tap into the ActiveEngine and put things into overdrive. This post is not about a great new design pattern that solved a problem or fit neatly into a methodology. More time for that in another post.

This is about the onerous task of updating 200 desktop computers with a technology that barely passes for worst practices from 1997. Yep, ten years of garbage code and retarded customizations, compounded by bad design and awful practices, 10 servers to update and 200 desktops to uninstall the old version and update with next version, the operative word being next.

For those not of the techie world, this can be a technician’s nightmare. This complexity is increased when the software manufacturer pretends that they themselves have wisdom greater than the rest of software industry,shun standard practice, and create some unique yet “better” software update process. Processes that can not be controlled, that when followed would result in 200 computers updating automatically at once, and sentencing a corporate network to a speedy death.

Can you see the challenge? The task, create an update process that circumvents the software manufacturers, is automated with measurable outcomes, saves time, and executes successfully. The risk, if you fail the company can’t invoice or collect the next month end close. Oh, and by the way, the sales force uses a pseudo workflow engine based on this mess as well. Better not disappoint that bunch of patient delicate artisans.

A team of heroes was assembled, people who had the strength of will to overcome the repeated failures and obstacles that such a nightmare offered. A team that turned on the ActiveEngine and fought like hell. For the most part, automation of software processes could be accomplished easily enough, but for a software push to operate successfully presupposes that all target desktops are in a state to be updated, and this is where the nightmare truly began. Picture this:

  • 43 desktop computers could not be connected to remotely, meaning that files could not be copied from a central source using automation. So updates had to be processed by a technician logging directly into the machine.
  • 15 desktop computers continued to execute update processes that prevent the crucial update from completing successfully.
  • 27 computes had corrupted Windows Registries, meaning that network administrator ids could not install new or remove old software, period. This could be repaired but the duration of this task took 10 -15 minutes. If the repair process failed the computer could be rendered useless.

This last issue was the deal breaker, as it indicated that a computer was no longer under control. Scary thought. You see, you wouldn’t find out that the Windows registry was corrupted until 2/3 of the install was completed. The figure above was ascertained after the entire set of 200 computers were updated. The prospect of failure heading into this mess was down right terrifying as what was not known could not be known.

Immediately diagnostics were performed on the registry. For non technical folks, this means that files containing more than 120,000 lines of code and security settings had to be examined. On a lucky break the starting point of the issue was ascertained, but this starting point was narrowed down to c. 20,000 lines. More comparisons to functioning registries were attempted, but the problem remained that comparisons were limited to eyeballing lines of registry keys. The support folks groaned, but offered their standard answer of “we’ll have to compare this manually”.

Unacceptable. That’s acquiescing, not thinking.

The heroes researched and found SubinACL, a tool that creates text file of all registry keys and ACL’s for those keys. The problem still existed that there was no way to compare text files from machine to machine. Not until the ActiveEngine kicked in and agile minds woke up, and a process to index the files – assign the same value for the same registry entry across all files – was created. Next, the files were imported into Access where the registry keys could be queried based on their ACL characteristics, eliminating any need for reading lines of registry dumps. 683 culprit lines were identified, 13 of these were unique. A text file with the new ACL’s was created and SubInACL was executed with this file to apply to the security settings for every machine. Yes, one day’s worth of programming eliminated weeks of work. The beauty of the brute force method was that if the settings were already correct, SubInACL would not apply the changes.

The code for this process will be discussed in a later post. It was a quick and dirty, actually really dirty, C# console app. Extensible enough to make quick changes and now subject to a re-write in hopes to create a flexible text file parser that will accept user defined fields that can be applied to the parsing process on the fly.

%d bloggers like this: