.NET Software Architect

Platform or technology stack specific modifiers on the title of Software Architect are common. Most software architects know that the platform, like a framework, is an interchangeable implementation choice and not really part of the software architecture (see my post entitled Practical Agile Software Architecture).

Why the distinction then? Two reasons:

  • Architect as Implementer  - Many software architects are involved in guiding and contributing to platform specific implementation which requires specialization that is not specific to the architecture itself. I prefer this combination.
  • Platform Specific Language - The platform specific software architect may be most comfortable or even required to produce artifacts that adopt the vernacular of that platform in order to make those artifacts more easily consumed by the implementation team.

The most common platform and implementation specific language elements in architecture artifacts replace the more generic terms of module, component, and software. Here’s an overly simple conversion table that I have found helpful for Java and .NET.

Architecture Generic Java .NET
Module package with public classes defining "port" and  sometimes an entire jar namespace with public classes defining "port" and sometimes whole assembly
Component sometimes jar only but usually whole container or daemon sometimes an independent assembly but usually a single process/web app
Software (physical allocation) app servlet engine for container or independent daemon IIS web app for one or more components or an independent Windows service

.
Connectors are generally common across these stacks, with a few exceptions such as WCF and JAX. More often connectors are literally specified as a SOAP or REST or even a custom TCP based protocol. Message based connectors are very often technology specific, indicating the specific message queue but be careful to avoid limiting your architecture by specifying implementation choices. The implementation choices should be, as much as possible, left to the design and implementation team effort.

The majority of my implementation experience has been in the .NET stack. But software architecture should be the same across implementation stacks. Of course, there may be some things easier said than done in a specific platform and technology stack, and that may have an impact on your software architecture choices, but minimally so.

Practical Agile Software Architecture

Ask software professionals what software architecture is and you will get many different answers.

In the past I have worked as a software architect and development lead on a variety of projects and teams. Two years ago I joined a larger software development organization as an individual contributor, and for the last eight months I’ve been the sole engineer assigned to create a very large in-memory graph data service that provides sub-second traversals across tens of thousands of edges amongst several billion. It has been a fun and successful project, but lately I’ve been thinking more and more about software architecture.

I’m sure that millions of words have been written on the subject of software architecture. I have read of them. Most recently I’ve been perusing two tomes on the subject: Software Architecture Foundations, Theory and Practice (Taylor, et al) and Software Architecture in Practice - 3rd Edition (Bass, et al). The latter is quickly becoming my favorite, but both of these are exhaustive academic textbooks, so most software professionals would find it impractical to put them to good use. Even so, I recommend them to you.

So what do I mean by “practical agile software architecture” and how can it be used by an agile software development organization? That is the question I will attempt to answer here. Distilling some of my reading through the filter of my own experience, I offer the following thoughts on the matter.

Principles of Practical Agile Software Architecture

  • Architecture is part of the product or a product dependency – Architecture should come first. Any large and complicated software project should have a architectural artifacts (see below) sufficiently defined to guide product backlog story definition to seed and sustain ongoing implementation with minimal risk of disruptive architectural changes.
  • Architecture stories go on the backlog – Architecture stories are generated from the stakeholder/view matrix (see below). Stories may repeat by adding detail to or modifying existing artifacts. Modifications should therefore be taken carefully considering the implementation changes that may be required.
  • Architecture story implementation produces artifacts (scenarios and views) – The architectural scenarios and views (Bass) include the definition of non-functional quality attributes such as security, reliability and performance. These artifacts document the architecture sufficient for the implementation team as well as stakeholders.
  • Architecture quality attributes should be measurable – Simply declaring that the application must be fast is not enough. Quality attribute definitions, as one type of artifact (see scenarios below), must produce measurable, observable, testable indicators.
  • Architecture artifacts evolve iteratively – Each iteration should produce a useable architecture, sufficient to inform stakeholder decisions and guide implementation. This does not mean that the architecture is never revisited. Architecture should be a regular part of every iteration, even if the story is simply to review some part or the whole of the architecture that is relevant to other stories being worked on during that iteration.
  • Architecture change can affect product backlog – Architecture stories that result in a modification to an artifact will generally produce one or more changes to the product backlog in order to accommodate the modification in product implementation. Care should be taken to communicate each new version of the architecture to relevant stakeholders and the implementation team.
  • Architecture complexity should be decomposed – In multi-team multi-service multi-product development organizations, it may be useful to create a centralized architecture team that defines and develops higher level architectural views upon which implementation teams take a dependency. Changes to the central architecture may produce changes to a service or product’s architecture related stories on their own backlog. As scrum of scrums can help to decompose more complex projects, so can the same approach be used to break down the complexity of the overall architecture to ensure agility.

Architecture Artifacts and Views

Documenting an architecture can be challenging. Some documents are produced as an overview. Some with varying levels of detail. Much of this depends upon the extent to which the architecture is complete and the intended audience for the specific view. Whether you call these documents a “view” or a treatment or a physical or logical or quality requirements document, the result should be the same. These documents should convey as concisely as possible the information needed to by stakeholders and implementation teams to make decisions.

These are summarized from Bass with my own particular spin.

  • Qualities Scenarios – There are seven primary non-functional software qualities: security, availability, interoperability, modifiability, performance, testability and usability. These should be well defined with concrete scenarios specifying stimulus source, stimulus, environment, artifact, response and acceptable response measures. Here’s a summary example for each of the seven qualities.
    • Security – A user (source) attempts to view private information (stimulus) in the patient records systems (artifact) on the beta testing system (environment) and an audit trail is maintained (response) and the attempt is prevented and reported (measure).
    • Availability – A heartbeat monitor (source) receives server unresponsive status (stimulus) on a web service (artifact) in production operations (environment) and informs the failover system to take traffic (response) and no downtime is recorded (measure).
    • Interoperability – Our patient billing system (source) sends account to collections (stimulus) to the collections system (artifact) in the testing system (environment) and a test letter is sent (response) and the test letter queue increases by one (measure).
    • Modifiability – Developer (source) is required to change a data source (stimulus) on the records repository (artifact) in the source code (environment) and makes the change with a unit test (response) and completes the task in three hours (measure). 
    • Performance – A user (source) save changes to his profile (stimulus) in the company intranet (artifact) on the intranet (environment) and the change is displayed (response) and the change takes less than two seconds (measure).
    • Testability – An automated unit test runner (source) executes a unit test (stimulus) on the code (artifact) in the test server (environment) and the test passes (response) with 85% code coverage (measure).
    • Usability – A user (source) adds a widget to his blog (stimulus) on the admin page (artifact) in the production system (environment) and is taken to a page with the widget activated (response) with fewer than 3 modifications (measure).
  • Module View – In the module view, design principles such as separation of concerns and loose coupling help to define boundaries around responsibility sets. Often a module is a separately compiled unit (e.g. a .NET assembly DLL).For example, the data repository module depends on and implements the data repository interfaces module which provides data models and the interface definitions for persisting those models. The data repository module can be replaced when a decision is taken to move from one database server to another.
    • Elements – module = implementation units (sets of responsibilities)
    • Relations – is a part of X, depends on Y, and is a Z 
    • Constraints – visibility and availability of module
    • Usage – implementation blueprint, impact analysis, requirements tracing, information design, users stories and use cases
  • Component & Connector View – Components and connections define logical processing units (processes or sub-processes) or data stores and their ports (sometimes called API) for connections to connect to. This could be a web service and the port (not TCP port) that defines the external boundary of that processing unit. A connector may me a message queue pushing and pulling messages or a browser connecting to a RESTful web service.
    • Elements – component = processing unit or data store, connector = paths of interaction (external API and use of same)
    • Relations – attachment = connector connected to a component port, interface delegation = a connector between compatible ports
    • Constraints – components attach via port only to components and vice-versa, never to each other
    • Usage – illustrate how the system works, define communication boundaries, guide implementation and design structure, and guide establishment of quality attributes for each component
  • Physical Allocation View – Define the software that will run on each part of the physical deployment in each environment.
    • Elements – software = what runs on each part of the environment, environment = where software is hosted and on what
    • Usage – assists reasoning about quality attributes, distribution of work and teams, concurrent access to versions of the software, and the form and mechanics of software deployment
  • Qualities View – A quality view document helps to correlate specific qualities and their scenarios with respect to the following:
    • Security – define components and behaviors that have an impact on security and how security risks will be mitigated and prioritized.
    • Communications – document how components will communicate, what networks channels and internal and external systems will be involved in data transfer, and how quality of service (QoS) will be maintained and how communication interruptions will be managed
    • Exception / error handling – define how errors will be detected, reported, monitored and resolved
    • Reliability – define how timing, integrity and other quality measures will be recorded and monitored
    • Performance – establish traffic load expectation and handling patterns, latency and system and network metrics

For more complete details on how these views may be composed, I recommend picking up a copy of Software Architecture in Practice as previously mentioned.

Stakeholder View Matrix

Because the number of possible views and scenarios with varying levels of required detail can overwhelm the architecture backlog, I endorse the Bass recommendation of the stakeholder/view matrix. I’ll try to summarize it here.

Create a matrix of stakeholders, including implementation and testing roles, with common architecture views and qualities. Let’s assume you have executive, product manager, implementation and testing stakeholders. And you have seven qualities and a minimum of four views. This gives you a four by eleven matrix.

For each cell in the matrix enter a level of detail word (none, overview, moderate, high) and a priority number between 1 and 10. Use this matrix to identify artifacts that need to be produced first and what level of detail is required for the first, second and third iterations. This exercise will help you identify the most important architectural stories for your backlog and prioritize your work on the architecture.

Thoughts on Architecture and the Role of the Architect

Much of what a software architect is often required to do is not, strictly speaking, architecturally related. But if an architect wants a team to be successful at implementing the software, she or he will take special care to incorporate these things into their regular work.

  • Patterns and Frameworksarchitecture is NOT a framework. In fact, a framework should be able to be swapped out without really having to change the architecture. Frameworks are tools and as such should be carefully chosen for their implementation multiplier capabilities as well as their simplicity and adaptability. The same is true for implementation patterns. Developers who are guided to select appropriate implementation patterns by being provided prototypes and samples can be far more productive and will produce much more maintainable code. The effort invested by the hands-on architect in these will yield handsome returns.
  • BDD and TDD – architects often wear the design hat as well and when doing so, they should adopt Behavior Driven and Test Driven Development. Read the linked content. It will really help simplify and clarify your user stories, use cases, and test designs. Of course, most design is driven by functional requirements which can affect the architecture in order to support the behaviors defined by the functional requirements. The software architect is appropriately involved in gathering and understanding these requirements. And breaking them down by behavior driven forms is an ideal way to fully understand and document these requirements.
  • UML and Other Modeling Languages – there are many tools available to create diagrams and other artifacts that may help you communicate your architectures and designs. A word of caution. Unless your audience is familiar with the notation and the meaning of the symbols and shapes you use, you need to provide a key with sufficient detail that a less familiar reader can understand what it is your are trying to communicate. If the diagram uses too many abstractions, the reader, even a seasoned software engineer, may have trouble understanding your meaning. And when stakeholders and implementation team members do not understand it, they will not always ask. Most of the time they will either ignore or or make erroneous assumptions that will bite you in the tender flesh later.
  • DevOps and Infrastructure – one of the things that can kill a great architecture and implementation is the lack of sufficient means to build, test and deploy, and roll back when necessary, in an automated fashion. A great DevOps team can make or break your large software development organization. And if you have the necessary operations monitoring and SLA failover triggers enabled with redundancy, you can maintain a very high level of service and reliability. A software architect’s input into the creation and improvement of these is critical to success.

Conclusion

Software architecture is not necessarily easy in part because we often make it more complicated than it need be. It is also true that we rarely define it well within an organization. Many organizations hire and promote to the position of software architect with the expectation that they will be an experienced software engineer who has picked up on what works and what does not and will help lead a team to success. Many experienced software engineer’s make this transition and learn the craft of architecture well. Some do not.

Some software architects spend very little time defining and refining the actual architecture of a the software. Instead they spend far more time chasing down design flaws, resolving thorny implementation issues and untangling behaviors that are at odds with business requirements. And yet, had the architect spent sufficient time and energy producing concise and precise architecture artifacts discussed in this post, continually improving them, reviewing them and coordinating their impact on implementation backlogs, many of the problems they spend so much time on would have been avoided.

I know you have differing opinions. Please share them here.

Doors or Rooms?

Do you build rooms with doors or doors with rooms?

Should every room be the same size with doors of different size, shape and placement?

Are some doors in the floor or in the ceiling, at floor level or in the middle of the wall?

Do your doors open in our out? Are they too small? Too big?

Rooms are code that does stuff. Does work. Holds stuff. Creates stuff.

Doors are code that gets stuff into and out of rooms.

Which one do you spend more time working on?

Making Distributed Computing Relevant and Accessible

First, let us assume that distributed computing is generally that area of developing and running software designed to process large numbers of long running tasks on servers that are optimally proximal to the data being processed.

Second, let us agree, if for this discussion only, that distributed computing is NOT your collection of services on back end servers that support your service oriented architecture (SOA) for your web and mobile apps.

Third, let us presume that you are NOT already blessed with a job where you write distributed computing software.

How then can distributed computing be relevant to you? And how can you take advantage of distributed computing without becoming an expert in one of the several well known distributed computing platforms on the market today?

Both are excellent questions. Thank you for asking. Let’s try a practical approach.

Imagine you are at your desk and your boss comes to you and ask how fast your web servers respond to the customer. Of course, your first instinct is to write this program to find out:

private static void DoTenUrlsInParallel()
{
   Console.WriteLine("Do 10 urls in parallel");
   var sw = Stopwatch.StartNew();
   ISpeedTest test = new SpeedTest();
   Parallel.ForEach(TestUrls, (url) =>
   {
      var result = test.GetSpeed(url);
      Console.WriteLine("r:{0}, s:{1}, b:{2}, u:{3}",
         result.ResponseTimeMs, result.ReadStreamTimeMs, 
         result.ResponseLength, result.Url);
   });
   sw.Stop();
   Console.WriteLine("Total elapsed time: {0}", 
      sw.ElapsedMilliseconds);
   Console.WriteLine(string.Empty);
}

You take him the results and he says, “But isn’t this from your desk? I want to know what these numbers look like from all around the world. East and west U.S. North and west Europe. And east and south east Asia. And I want a regular stream of these numbers fed into a spreadsheet for me every day.”

Do you say, “No problem.” You do if you have a Windows Azure account and you know about the distributed task parallel library from DuoVia called DuoVia.Net.Distributed. You go back to your desk and modify the code to look like this:

private static void DoTenUrlsThreeTimesEachAroundTheWorldInParallel(bool runLocal = false)
{
   var serverEndpoints = new IPEndPoint[0];
   if (runLocal)
   {
      serverEndpoints = new IPEndPoint[] { new IPEndPoint(IPAddress.Parse("127.0.0.1"), 9096) };
   }
   else
   {
      //these server names are temporary - to run this test use your own
      var servers = new string[]
      {
         "myaz-westus.cloudapp.net",
         "myaz-eastus.cloudapp.net",
         "myaz-northeu.cloudapp.net",
         "myaz-westeu.cloudapp.net",
         "myaz-soeastasia.cloudapp.net",
         "myaz-eastasia.cloudapp.net"
      };

      serverEndpoints = new IPEndPoint[servers.Length];
      for (int i = 0; i < servers.Length; i++)
      {
         var host = Dns.GetHostAddresses(servers[i]);
         var ip = (from n in host 
                   where n.AddressFamily == AddressFamily.InterNetwork 
                   select n).First();
         serverEndpoints[i] = new IPEndPoint(ip, 9096);
      }
   }

   float subscriptionRate = 2.0f; //oversubscribed 
   int logPollingIntervalSeconds = 2;
   using (DistributedClient<ISpeedTest> client = 
          Distributor.Connect<ISpeedTest>(typeof(SpeedTest),
          subscriptionRate,
          logPollingIntervalSeconds,
          LogLevel.Debug,
          serverEndpoints))
   {
      for (int i = 0; i < 3; i++)
      {
         var sw = Stopwatch.StartNew();
         Console.WriteLine(@"round:{0}", i + 1);
         var loopResult = client.ForEach(TestUrls, (url, proxy) => proxy.GetSpeed(url));
         foreach (var result in loopResult.Results)
         {
            Console.WriteLine(@"r:{0}, s:{1}, b:{2}, on: {3}, u:{4}",
               result.ResponseTimeMs, result.ReadStreamTimeMs, 
			   result.ResponseLength, result.MachineName, result.Url);
         }
         sw.Stop();
         Console.WriteLine("Total elapsed time: {0}", sw.ElapsedMilliseconds);
         Console.WriteLine(string.Empty);
      }
   }
}

And you and your boss are happy.

Sometimes distributed computing is more about location and proximity to data or infrastructure than it is to getting massive amounts of data processed in as little time as possible.

You can find the full demo source code here.

Estimating Software Features and Fixed Feature Compensation from 8th Light

I just finishing reading a fascinating blog post by Paul Pagel of 8th Light called Fixed Feature. I recommend that you read the post and download and study the linked PDF entitled From Estimate to Commitment.

Companies willing to pay for software development on a time-and-materials basis are fewer and farther between. They don’t have the resources to monitor such work to assure themselves that the team is progressing as it should. Understandably, they cannot shoulder this risk. So they want their teams, internal and external, to give them a fixed bid for a project: fixed time, fixed cost, fixed features.

Pagel points out, “software is a craft with a close problem finding to problem solving loop.” In other words, we’re not always very good at anticipating problems in a development project until they are just about to happen. Sometimes we don’t even see them coming. We can minimize these events through continuous review and refinement of our processes and practices, study and evaluation of past performance versus estimates and the assumptions that went into those. Too often we do not take the time to do so, dooming ourselves to historical repetition.

Rather than attempting to created a fixed bid for an entire project, 8th Light, according to Pagel, estimates each feature using PERT and creates a price point for each feature. At the end of an iteration, they bill the client for features accepted by the client. This is an interesting approach to combining the desire for fixed bid pricing and Agile development.

Of course, this or any variation on fixed bid pricing requires that we improve our track record on creating estimates. If you visit our friends in Mountain View and search for estimating software development, you will find over four million results. There are tools and whitepapers and blog posts galore. But before you spend more money on tools or consultants, look to your own experience. Be willing to record what you’ve done, evaluate it honestly and boldly, and make adjustments based on what you have learned. Surely this will be less painful than blindly repeating the past.

Read Pagel’s work on this subject and let me know what you think.

Software Architecture: Use Cases, Not Frameworks

I enjoy continually learning from Robert Martin (Uncle Bob). Sometimes I disagree with him but usually only until I’ve thoroughly processed what he is trying to teach me. In his latest missive on 8thlight.com, Uncle Bob writes a stunningly cogent brief called Screaming Architecture in which he makes a critical point: software architecture is all about the use cases of a system and NOT the framework.

“Architectures are not (or should not) be about frameworks. Architectures should not be supplied by frameworks. Frameworks are tools to be used, not architectures to be conformed to. If your architecture is based on frameworks, then it cannot be based on your use cases.”

Focus on the use cases and perhaps even group them into logical hierarchies with tidy “40,000 foot level” epochs that describe the thing you are building. Is it a customer relationship management system? The first thing we should notice in an architecture, whether document or code, is that this is a CRM system. Presentation, delivery, storage and communication frameworks are all secondary, just plumbing, not the architecture.

“A good software architecture allows decisions about frameworks, databases, web-servers, and other environmental issues and tools, to be deferred and delayed.”

I’m not completely at this point in my journey toward architectural perfection but I recognize the need to strive for this idea of producing architectures that are designed to solve the problems of the users first rather than designed fit within the strictures of a framework while addressing the use cases after the fact.

I recommend you read the full article. Take Uncle Bob’s advice, as I intend to do, and focus more strongly on architecture for your use cases on your next project. Use frameworks carefully and work to keep your business code independent of your frameworks and thus imminently more testable. And don’t forget your tests.