Uncle Bob on TDD

I rarely parrot blogs here but sometimes I find a post I don’t want to lose track of. Robert Martin (Uncle Bob), one of my favorites, just posted The Domain Discontinuity which asks the question: Which came first, the chicken or the road?

Two quick pull quotes.

The first has to do with the appropriate scope of TDD. Why do I include this? Because I am not a TDD fanatic. By fanatic, I mean those developers who would disagree with Uncle Bob here, even going so far as to make private and protected methods public just so their tests can access them.

“Let me stress this more. I do not create a test for every method or every class. I create tests that define behaviors, and then I create the methods and classes that implement those behaviors.

“At the start, when there are just a few tests, I might have only one simple method. But as more and more tests are added, that one simple method grows into something too large. So I extract functions and classes from it -- without changing the tests. I generally wind up with a few public methods that are called by my tests, and a large number of private methods and private classes that those public methods call; and that the tests are utterly ignorant of.

“By the way, this is an essential part of good test design. We don't want the tests coupled to the code; and so we restrict the tests to operate through a small set of public methods.”

The second pull quote is a small set of architectural principles that I completely agree with:

    • Refactoring across architectural boundaries is costly.
    • Behaviors extracted across architectural boundaries need newly rewritten tests.
    • Architecture is an up-front activity.

“The solution to that problem is to know in advance where you are going to put certain behaviors. You need to know, in advance, that there will be a boundary between the GUI, the business rules, and the database. You have to know, in advance, that the features of your system have to be broken up into those areas. In short, before you write your first test, you have to ‘dream up the [boundaries] that you wish you had’.”

I sometimes come across systems that are woefully inadequate to the task, extremely difficult to refactor, and have few, if any, tests that do not require integration across boundaries that are often, if not almost impossible, to mock or fake. Virtually all of these problems are attributable to the false notion that Agile means you rush to coding and skip up-front architectural analysis and design.

Thanks, Uncle Bob!

My Technical 2013

Technically speaking, I had a fun and productive 2013. Here are some highlights worth mentioning.

StorageClient: Client Side Load Balancing

A technology specific problem solved, bypassing server based solution with client side load balancing and fast fail retry algorithms that took us from horrible to nearly 5 nines in reliability while improving overall performance. (This was at the day job, so that's about as much as I can share about that.)

LocalCache: In Memory Cache with Async Persistence

A library that takes advantage of Concurrent Collections in .NET and SQLite to provide fast in-memory caching that persists asynchronously on local disk for rapid rehydration of in-memory cache when an application pool is recycled. This solved a big problem with service level compliance recovery on a critical service, taking complete recovery time from hours to a few minutes. (Also for the day job.)

DuoVia.Net: TCP and NamedPipes Services Library

An extension and revival of RemotingLite that makes intra-process communication easy and fast. This was my first foray into creating and sharing open source software on GitHub and publishing packages on NuGet. I enjoyed it so much, I added 8 more packages to the set. And while these projects were built on my own time, one or two of them are in regular use by one or two teams at the current day job and they have been downloaded over 1,500 times.

VersionedCollections: A Shared Idea Brought to Life

Recently I shared an idea with Ayende Rahien on his blog with respect to creating a snapshot-in-time, read-only view of a collection that is being written to constantly. I'm happy to report that it turned out to be exactly what he needed. And I am honored and appreciative to Ayende for the kudos. Sharing good ideas with community friends is almost as much fun as bringing them to life yourself.

Here's to an equally fun and productive 2014.

Doors or Rooms?

Do you build rooms with doors or doors with rooms?

Should every room be the same size with doors of different size, shape and placement?

Are some doors in the floor or in the ceiling, at floor level or in the middle of the wall?

Do your doors open in our out? Are they too small? Too big?

Rooms are code that does stuff. Does work. Holds stuff. Creates stuff.

Doors are code that gets stuff into and out of rooms.

Which one do you spend more time working on?

Making Distributed Computing Relevant and Accessible

First, let us assume that distributed computing is generally that area of developing and running software designed to process large numbers of long running tasks on servers that are optimally proximal to the data being processed.

Second, let us agree, if for this discussion only, that distributed computing is NOT your collection of services on back end servers that support your service oriented architecture (SOA) for your web and mobile apps.

Third, let us presume that you are NOT already blessed with a job where you write distributed computing software.

How then can distributed computing be relevant to you? And how can you take advantage of distributed computing without becoming an expert in one of the several well known distributed computing platforms on the market today?

Both are excellent questions. Thank you for asking. Let’s try a practical approach.

Imagine you are at your desk and your boss comes to you and ask how fast your web servers respond to the customer. Of course, your first instinct is to write this program to find out:

private static void DoTenUrlsInParallel()
{
   Console.WriteLine("Do 10 urls in parallel");
   var sw = Stopwatch.StartNew();
   ISpeedTest test = new SpeedTest();
   Parallel.ForEach(TestUrls, (url) =>
   {
      var result = test.GetSpeed(url);
      Console.WriteLine("r:{0}, s:{1}, b:{2}, u:{3}",
         result.ResponseTimeMs, result.ReadStreamTimeMs, 
         result.ResponseLength, result.Url);
   });
   sw.Stop();
   Console.WriteLine("Total elapsed time: {0}", 
      sw.ElapsedMilliseconds);
   Console.WriteLine(string.Empty);
}

You take him the results and he says, “But isn’t this from your desk? I want to know what these numbers look like from all around the world. East and west U.S. North and west Europe. And east and south east Asia. And I want a regular stream of these numbers fed into a spreadsheet for me every day.”

Do you say, “No problem.” You do if you have a Windows Azure account and you know about the distributed task parallel library from DuoVia called DuoVia.Net.Distributed. You go back to your desk and modify the code to look like this:

private static void DoTenUrlsThreeTimesEachAroundTheWorldInParallel(bool runLocal = false)
{
   var serverEndpoints = new IPEndPoint[0];
   if (runLocal)
   {
      serverEndpoints = new IPEndPoint[] { new IPEndPoint(IPAddress.Parse("127.0.0.1"), 9096) };
   }
   else
   {
      //these server names are temporary - to run this test use your own
      var servers = new string[]
      {
         "myaz-westus.cloudapp.net",
         "myaz-eastus.cloudapp.net",
         "myaz-northeu.cloudapp.net",
         "myaz-westeu.cloudapp.net",
         "myaz-soeastasia.cloudapp.net",
         "myaz-eastasia.cloudapp.net"
      };

      serverEndpoints = new IPEndPoint[servers.Length];
      for (int i = 0; i < servers.Length; i++)
      {
         var host = Dns.GetHostAddresses(servers[i]);
         var ip = (from n in host 
                   where n.AddressFamily == AddressFamily.InterNetwork 
                   select n).First();
         serverEndpoints[i] = new IPEndPoint(ip, 9096);
      }
   }

   float subscriptionRate = 2.0f; //oversubscribed 
   int logPollingIntervalSeconds = 2;
   using (DistributedClient<ISpeedTest> client = 
          Distributor.Connect<ISpeedTest>(typeof(SpeedTest),
          subscriptionRate,
          logPollingIntervalSeconds,
          LogLevel.Debug,
          serverEndpoints))
   {
      for (int i = 0; i < 3; i++)
      {
         var sw = Stopwatch.StartNew();
         Console.WriteLine(@"round:{0}", i + 1);
         var loopResult = client.ForEach(TestUrls, (url, proxy) => proxy.GetSpeed(url));
         foreach (var result in loopResult.Results)
         {
            Console.WriteLine(@"r:{0}, s:{1}, b:{2}, on: {3}, u:{4}",
               result.ResponseTimeMs, result.ReadStreamTimeMs, 
			   result.ResponseLength, result.MachineName, result.Url);
         }
         sw.Stop();
         Console.WriteLine("Total elapsed time: {0}", sw.ElapsedMilliseconds);
         Console.WriteLine(string.Empty);
      }
   }
}

And you and your boss are happy.

Sometimes distributed computing is more about location and proximity to data or infrastructure than it is to getting massive amounts of data processed in as little time as possible.

You can find the full demo source code here.

Diversions in the D Programming Language

I am not a systems programmer, meaning I do not write operating system device drivers or file systems or operating system modules, etc, all written in a language that will compile down to raw machine code. I write in C# primarily which is arguably an applications programming language, running in the much loved .NET Common Language Runtime.

The vast majority of systems programming is done in C and C++. And for some reason, C++ has always been a daunting mess of libraries, odd syntax and pointer and memory allocation madness to me. Even setting up an environment to get the right build libraries, the right compiler and linker, etc., have always led me to fits of impatience. And for that reason and many others, I have stuck to C# and applications development.

But every once in a while, I look in on systems programming to see if anyone has really solved the problems I love to hate with respect to C and C++. And for a few years I’ve read a little about the D programming language here and there. A week ago, over the weekend, I decided to give it a try and really see what I could learn.

I have to say, I have been impressed. The D programming language offers a few things that I would dearly love to see in C#.

1. Exception Safety – the scope keyword

void abc() 
{ 
  auto resource = getresource();  // acquire some resource 
  scope(exit) resource.close();   // close the resource 
  doSomeProcessing();             // do processing
}

As C# programmers, we’re used to the try..catch..finally blocks. And we clean up a resource in the finally block. The trouble with that is many lines of code can end up separating your resource acquisition code from your resource cleanup code. Yes, with vigilance and well written tests, this is okay. But wouldn’t it be cool to be able to tell the compiler, “Hey, when I’m done with this thing I just now created, clean it up for me, no matter what code comes after this in this method.” I would love to see the scope keyword added to C#.

2. Concurrency approach

int perThread;
shared int PerProcess;

In C#, when you declare a class level variable, it is automatically shared between threads. You can use the [ThreadStatic] attribute to get a per thread instance of a given value or object. But it then has to be static. With the D programming language, you get thread safety in class variables. To override that safety, you have to explicitly tell the compiler you want the value shared. While I’m not advocating a change to C# in this regard, I would love to have a way to assure that a variable cannot be modified across thread boundaries.

3. Message based threads

import std.concurrency, std.stdio;
void main() {
   auto low = 0, high = 100;
   auto tid = spawn(&writer);
   foreach (i; low .. high) {
      writeln("Main thread: ", i);
      tid.send(thisTid, i);
      enforce(receiveOnly!Tid() == tid);
   }
}

void writer() {
   for (;;) {
      auto msg = receiveOnly!(Tid, int)();
      writeln("Secondary thread: ", msg[1]);
      msg[0].send(thisTid);
   }
}

For me, this is perhaps the coolest part of the D programming language’s base class library which they call Phobos. Note that main spawns a thread calling writer. The loop in main then sends a message to writer and the loop in writer receives the messages and operates on them and then sends a message back to the original thread.

You can learn a lot about D on www.dlang.org and read more about D concurrency on Informit. And if you want to play with D in Visual Studio, hop on over to see VisualD on dsource.org.

Choosing a Content Management System

Growth in the content management system (CMS) market is strong as more and more companies look for software to help them manage their burgeoning coffers of content. This growth is partly fed by the growing temptation to use these systems for purposes far beyond managing content with CMS vendors tacking on every new feature they can dream up to feed this frenzy. The result is often a bloated, unwieldy, unmanageable, complexity induced bug-infested mess.

I have a relatively high level of interest in this space given my brief professional experience in this market. So I read with interest a post today on the subject by one of my favorite software development authors, Martin Fowler. In his post entitled Editing Publishing Separation Fowler discusses the advantages of separating the editing data and experience from the publishing experience and published data. I invite you to read his article as he makes a very cogent argument which should be taken into account as you consider choosing a CMS.

Before he makes this point, he sets up his theme with this preamble:

"...a regular theme has been the growing impact of content management systems (CMS). They aren't usually seen as helpful, indeed there is a clear sign that they are becoming a worryingly invasive tool - used for more than their core purpose in such a manner that they hinder overall development."

It is precisely this thought which accurately represents my own limited experience in the CMS market. So here are my recommendations for consideration when choosing a CMS for your content management purposes.

  • As Martin suggests, look for a separate editing and publishing experience including separation of data and the advantages that can come from that.
  • And pay close attention to Martin’s opening quoted above. When looking at a feature, such as e-Commerce, consider whether you should be looking at a full featured e-Commerce platform rather than a feature that may have been cobbled onto the product as an afterthought. Remember that your primary purpose should be the management of content.
  • Whether you choose open or closed, open source or commercial, be sure that your own development team can understand and use the system with alacrity.
  • If you have a PHP team, don’t buy a .NET based CMS. And vice versa.
  • Take your time in evaluation. A quick, under-informed decision can cost you far more than you can imagine in future headaches.
  • Give your development team time to find the real problems with the CMS being evaluated and determine whether and how they can live with the problems they encounter.
  • Be sure to test the support and services teams of the vendor if you are going commercial over open source. Or if you are going open source, look for a proven and reliable partner who can guide you through the rough patches. Because no matter which CMS you choose, there will be some of those.
  • Consider licensing and support costs but don’t fall into the trap that open source is free. The efficiency of your own development and content management team using the CMS over the long haul will have a far larger impact on your overall cost of ownership.
  • Ask someone from Missouri to join your evaluation team. In other words, NEVER just accept the word of a sales representative that the CMS she is selling has the feature that you want and that it works as advertised. Like all software, there is a HUGE amount of advertising fluff and endless feature lists. Make sure that you have the sales team “show” you and then make sure that your own development team can make it work without their help and without having to contact support before you accept the claim, “Oh yes, our CMS does that.”
  • And last but not least, do your own research of satisfied and dissatisfied customers. Make some calls. Ask to speak to someone on the evaluation team where another product was chosen over the one you are considering. Ask if they evaluated the product you have your sights set on. Chances are that a few phone calls will land you a gold mine in helpful information.

Of course, you can apply these same principles to purchasing any enterprise software. And you should.

ASP.NET MVC 4 Beta Released

I’m excited to watch the rollout of ASP.NET MVC 4 Beta and looking forward to finding the time and a good solid project to take advantage of the new mobile support and Azure SDK.

Here’s a basic list of what’s new.

  • ASP.NET Web API
  • ASP.NET Single Page Application
  • Enhancements to Default Project Templates
  • Mobile Project Template
  • Display Modes
  • jQuery Mobile, the View Switcher, and Browser Overriding
  • Recipes for Code Generation in Visual Studio
  • Task Support for Asynchronous Controllers
  • Azure SDK

I want to hear your ASP.NET MVC 4 stories. Let me know what you think about it. You may get a chance to deep dive before I do, given my current project priorities.

Goal Oriented Meetings in Software Development

I’m fascinated with the subject of meetings. It is a topic of discussion across the wide webbed world. There are nearly as many opinions on this subject as there are stars in the sky. So adding my own here cannot possibly disrupt our little corner of the multiverse, nor will it likely establish a new or even affect an existing trend. But before I bore you with my own opinion, let me share with you some links to some very cogent resources that have helped me, in addition to my own years of experience, to form my opinions here.

  • Firemen, donuts and meetings – Seth Godin: “Don’t bother to have a meeting if you’re not prepared to change or make a decision right now.”
  • Reduce Meeting Time by 30 Percent – Max Isaac: “Prior to the meeting, send an email with the proposed Purpose, Process and Preparation, and attach documents that should be reviewed.”
  • Meetings frustrate task-oriented employees – American Psychological Association: “…people high in accomplishment striving, who tend to be highly task and goal oriented, were most negatively affected by meetings...
    ...people who participate most in meetings, such as those who identify themselves as upper-level management, tend to view them most favorably...solely because they get to talk a lot, they inevitably miss other key pieces of meeting effectiveness, including their ability to successfully lead the meeting.”
  • Why Goal-Oriented Requirements Engineering – Yu & Mylopoulos: “The identification of goals naturally lead to the repeated asking of "why", "how" and "how else" questions... Stakeholders also become more aware of potential alternatives for meeting their substantive goals, and are therefore less likely to over-specify by prematurely committing to certain technological solutions.” (This paper is not about meetings but contributes to the opinions expressed below.)
  • Goal-Oriented Idea Generation – multiple authors, taken from summary of results: “The goals obtained from the ideas have high quality. During the activities for grouping ideas, it was frequently observed that the stakeholders could find their misunderstandings and tried to resolve them. As a result, the obtained goals were based on the consensus and the agreement of all of the stakeholders.” (Again, this is not about meetings but contributes to my ideas on the subject.)

It has been my experience that any form of software development, including my personal favorite: behavior driven design/development first introduced by Dan North, generally requires more than one person in the process. There is at least a user and a programmer, at least for any software that one might be paid to produce. And where there is more than one person in a process, there is the inevitability of a meeting.

The real question is what is the nature of the meeting. And by nature I mean how will it be planned and conducted, and what if any follow up will occur as a result of the meeting.

Bad Meetings: Vague, Purposeless, Meandering Time Wasters
The world of software developers could fill terabytes of hard disk space with examples of meetings that were a complete waste of time, ineffective, resulted in no change to the status quo nor the discovery of anything that was not already immediately discoverable in an issue or project tracking system.

Rather than go through any of these bad examples in detail, I will simply summarize that these meetings most typically include in the email invite only a vague subject line, a time and place, a dial-in number for remote attendees, and by virtue of the most commonly used tool to send the invite, a list of attendees, most often the entire team. Too many of these meetings and your organization has fallen hopelessly into the “Meaningless Meetings Machine.”

Goal Oriented Meetings: Purpose Driven, Limited, Action Packed
The very best meetings I have ever attended or conducted (yes, I’ve conducted my share of bad meetings too) have had the following qualities:

  • Goal oriented – the agenda focuses on stated goals. (See Mark Schiller’s cogent treatment in Fixing IT Productivity Destroyers.)
  • Limited scope – the time and agenda are limited to specific related goals.
  • Parking lot – good but unrelated ideas go into a “parking lot” to be discussed at another time and often in a different forum.
  • More listening (less talking) – participants listen carefully with their full attention and speak only when they have something relevant to contribute or need clarification in order to make a decision or understand an action item being assigned to them. And NEVER, NEVER, NEVER do participants rudely talk all at once.
  • Previous action items related to goals on the agenda are reviewed in summary fashion – discussion is limited to reporting outcomes and/or impediments that require attention by the group. This does NOT mean that the group attempts to resolve these impediments during the meeting. They are noted and may generate an action item for a participant, but under NO CIRCUMSTANCES will the meeting get side tracked to solve the problem in the moment.
  • Goals drive actions – starting with diverging opinions and working toward consensus in these steps:
    • affirming the goals are properly stated
    • identifying actions required to achieve the goals
    • identifying (or getting volunteers) to take those actions
  • Decisions are made – work toward a consensus but have a meeting leader who can make a final decision right there, right now, after having considered the input of meeting participants. Perfect consensus is not the goal. Decision and action from team input and reasonable discussion is sufficient. (And as Seth points out, don’t bother having a meeting unless you are prepared to change your mind. If you have already made your decision, just announce it, assign the tasks and follow through—don’t waste your team’s time with a meeting to maintain the illusion that you care about what others think—they know you don’t.)
  • Minutes, including action items and commitments, are recorded and published post haste. They are also maintained in a team public place such as a wiki or project portal for reference and follow up.

Goal Oriented Meetings Foster Better Software Development
I included some links already that talk about goal oriented requirements engineering and behavior driven development (BDD). In Dan North’s treatment of BDD, he presents two single sentence templates for defining each of the requirements of any software system. With my own explanations, they are:

To define a feature or user story:

As a [X], I want [Y], so that [Z].

Where [X] is the role of a user, [Y] is the feature, and [Z] is the benefit or value (the goals) of the feature.

And to define the acceptance criteria for that story:

Given [some initial context—the givens], when [an event occurs], then [ensure some outcomes].

The outcomes to be assured also define the goals of the user specifically within a certain behavioral or system state context.

When the goals of stakeholders are clearly understood and can be codified in a manner such as the BDD form, the development team can move forward more quickly and with greater confidence. And when meetings are equally focused on goals and achieving them, meetings will be an asset rather than a liability. And your team will have a much greater opportunity to succeed.