How Photography Helps Me Be a Better Software Architect


arizonahighways

I want to share with you how my efforts to perfect my semi-commercial hobby of photography help me be a better software architect.

My first camera was a Kodak Pony 135. I had no idea what I was doing, but I learned a lot by experimenting with that camera. My first computer was a Commodore 64. I was only slightly more informed about computing, having learned BASIC on a Commodore PET over two years prior. Again through experimentation and reading every Creative Computing magazine that I could get my hands on, I learned much more about computing and programming.

So now you know that I've been doing photography and computing for longer than many of my colleagues in this business have been alive. Let's see if my insights are of any use to you.

Patterns in Composition

In photography one must always be aware of the composition of the shot. How do the elements of the shot work together to draw the eye to the subject, to the story you want to tell? Learning the rule of thirds can be quite useful in understanding that, as in photography, so too in software we must look for patterns of composition to focus the user experience on the desired outcome. These patterns are often not visible to the untrained eye, just as in photography our audience may not really know why they like the photo, but they do. Software users will know they love or hate the software, but only you will know they love it because you composed it using patterns that make the magic happen.

Following a Process

We rarely get something right the very first time we try it. We're more likely to get something right when we follow a process that has led to success in the past. In photography I have learned that I must follow a process of preparation, analysis, exploration, decision taking and execution. A well thought out process in software development is equally crucial to success.

Breaking the Rules Sometimes

Once you know the rules of your process and craft, you can begin to take some risks in breaking the rules to achieve a desired effect. Only after understanding the rule of thirds in photography and knowing how and why this rule will help you compose your shot in a way that will present your subject well can you begin to experiment with breaking that rule to tell a slightly different story. The same is true in software architecture. Knowing how the rules of SOLID and design patterns will help you achieve the best outcome is crucial. Only then can you begin to experiment with small deviations to the rules to find new an innovative ways to achieve your goals.

Knowing Your Subject

In photography one must know something about the subject of the photograph. The more you know about the subject, whether this is a person, a landscape, a flower or a bug, the better you will understand how to draw out the character of your subject in your photograph. Being a software architect means knowing your subject, your users, stakeholders, the data and systems and the processes needed to make things work. The more you know about that, the better your architecture and design will be and the happier your customer will be.

Best Tools Don't Always Make the Best Work

Some photographers believe they must have the best, most expensive equipment to achieve the greatest possible result. This just is not true. I once shot with a full frame camera and now shoot with a Micro Four Thirds. Some day I'll move to medium format for other reasons, so don't discount having great tools, but don't assume that without the best you cannot produce great work. The same is true in software. We do not always need the best IDE or the most powerful laptop or the biggest amount of RAM to achieve the best result. Indeed if one focuses only on the tools, one might overlook the fact that one's work is to produce something great with the tools you have. Focus on the finished product and whine less about the tools you have been given.

Always Something Beyond Your Control

In photography you cannot control the weather or an unruly portrait subject. You cannot decide when your SD card will fail. You cannot guarantee that every shot will be perfect. There are always elements of the work that are out of your control. To succeed one must embrace the unknown and unexpected. In software architecture this is also true. There are many aspects of a project which you will have the ability to control. Time and business demands are not among them. Resource availability or team skills are not always within your control. You must learn to embrace the things that are not under your control and work as well as you're able to achieve your goals in spite of these.

Try Again Tomorrow

When I'm shooting landscapes, I often think that I've nailed it only to learn in post processing my RAW images that I failed to capture what it was I thought that I had seen when I was in the moment. These failures fall to the digital cutting room floor and push me to go back out and try again. There is a reason we have iterative processes in software development. We don't always get it right. We must continue to hone our craft and work and rework until we do get it right. Even then there is always more we can do. Sometimes the most important aspect of courage is simply to say to oneself, "I'll try again tomorrow." (Taken from a favorite saying borrowed from a friend of mine who passed away recently.)

People and Love Are The Most Important Ingredients

In photography and software architecture and development, I have learned that the most important aspect and ingredient to success is people. The people we work with will come from many different backgrounds. They will have different needs and desires. We must learn to love them and work with them. Only when we do this can we achieve our greatest success because our success depends on their success. When you love the people with whom you work, you will want to see them succeed. You will be kind but honest. You will be patient but urge them to do their best work. You will put their needs ahead of your own. Miraculously you will produce your very best work and so will they.

Allow Your Models to Cross Only One Boundary

Don't let your models cross more than one boundary. Sure there may be exceptions, but I haven't found one that justifies the loss of flexibility achieved when requirements change.

Some years ago I worked with a thoughtful and intelligent programmer named Nick. He convinced me that models, or data transfer objects (DTOs) should not cross more than one service or domain boundary. I was skeptical at first. It seemed like a lot of duplicated code.

He talked me into it. And the next time the business changed a requirement, which of course happens often in most software development shops, I became absolutely convinced. We added something to one model, changed the code that produced the model and the specific code that consumed the model without breaking all other users of other endpoints that used the underlying models and entity models or services that produced them.

Here's what it looks like in one implementation:

models

Of course, your own architecture may involve more boundaries than shown here. You may have a message queue or services bus. I believe the principle still applies.

The greatest advantage is that you have an abstraction layer at every boundary that allows you to move swiftly when you need to support changing requirements. And I've never met a company or set of software developers that do not deal with regularly changing business requirements.

This is a very opinionated post and if you balk at the notion at first, I urge you to at least give it a try. You will find that you can enforce separation of concerns. You will find that you can avoid having a dependency on a framework, such as Entity Framework, in your application server code or in your hard wired client code, the latter of which would need only a reference to your ViewModels assembly. (Assumes you are working in .NET but I believe the principles apply universally.)

You will find that in preventing your business logic from knowing anything about the implementation of the interface provided by the Models assembly, you are forced to inject the dependency and are thus free to replace the dependency with a different implementation or even a mock for unit testing.

The list of benefits is long and I've surely missed a few important ones, but you get my drift. I hope.

TIP: Use AutoMapper or something like it to help you with the tedium when transformations are an exact 1:1 match.

Microsoft REST API Guidelines

I’m a big believer in establishing standards for your REST APIs. I like this one from Microsoft but there are others. What I like about this particular guideline is the structural organization and its brevity. The introductory paragraph provides the reasoning and justification behind REST even when there are language specific SDKs made available.

Developers access most Microsoft Cloud Platform resources via RESTful HTTP interfaces. Although each service typically provides language-specific frameworks to wrap their APIs, all of their operations eventually boil down to REST operations over HTTP. Microsoft must support a wide range of clients and services and cannot rely on rich frameworks being available for every development environment. Thus a goal of these guidelines is to ensure Microsoft REST APIs can be easily and consistently consumed by any client with basic HTTP support.

One thing that would be nice to have is a standard for filtering, paging, etc., on the URL as query parameters. These get defined differently by every REST service architect out there. Or so it seems. A good jumping off point for some research and thought.

And I want to thank my friend Pablo for sending me the link to this.

Agile and Architecture

One of the most misunderstood and misrepresented documents in the history of software development is the Agile Manifesto. This may be due to many of its readers overlooking the phrase “there is value in the items on the right.” Most seem to focus on the items on the left only. Here’s the text that Cunningham, Fowler, Martin and other giants in the field created:

Manifesto for Agile Software Development

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right,
we value the items on the left more.

Note that I have emphasized the items on the right. These do indeed have value but so many advocates of Agile deliberately ignore and even exclude these from their software development process and organization. Some have advocated the elimination of architecture and design entirely, leaving these open to gradual discovery through the iterative process driven by use cases and user stories and backlog tasks.

Recently I have read a number of discussions, blog posts and articles on the question of Agile and architecture. The comments and discussion around the topic have been interesting. Perhaps this is due to the notion that developing software is only about writing the code. The general theme of these sources is that architecture (and design) are at odds with Agile. This is one of the great fallacies of our time.

Architecture and Design are Software Development Artifacts

Teams and organizations who skip architecture and design will sooner or later find themselves off track and repeating work unnecessarily. Such waste is not entirely preventable, but this does not mean we should not try. Teams that incorporate these activities into their iterations, regularly revisiting architectural questions such as non-functional requirements and component design, will find that they are better able to stay on course.

Organizations that have multiple teams will find greater stability in moving forward when guided by a centralized architecture team, comprised of architects or leads who are dedicated to and work within the organizations development teams. The architecture team works in an Agile fashion, with its own backlog and its own products including working prototypes, cross cutting standardized components, documentation of the architecture and designs, and work items to be placed on the backlogs of development teams.

In this way, development teams have dependencies on the architecture team and can make requests for additional guidance or improvements or extensions to shared, standardized libraries for which the architecture team is responsible. These requests keep the backlog of the architecture team charged with work throughout the software development lifecycle.

In addition to formal activities to improve architecture and design across the organization, the architecture team should regularly interact with development teams to include (but not limited to) the following:

  • practice improvement activities—e.g. SOLID principles
  • technology deep dives—e.g. digging deeper into .NET
  • technical solutions brainstorming sessions—solving the hard problems
  • technical debt evaluation and pay-down planning
  • code reviews and walkthroughs—one on one and as a team
  • presenting and sharing solutions and ideas from other development teams
  • exploring and evaluating new technologies and tools

Like testing and coding, architecture and design are a part of the whole of software development. These activities are perfectly suited to Agile development practices, including SCRUM. And when all of these aspects of delivering quality software are taken into account and incorporated into your Agile process, your chances of success are greatly improved.

----

P.S. And if you add to all this a great dev ops team to support your efforts with automated build and deploy systems, your life will be that much easier and your chances of success are automatically improved.

The 4+1 Architectural View Model v2

You can find the well written brief on Phillippe Kruchten’s 4+1 architectural view model on Wikipedia along with a link to the original white paper published nearly 20 years ago. Kruchten worked for Rational, as I understand it, and so this is often referred to as the RUP 4+1 model.

4p1Architectural_View_Model
Courtesy of Wikipedia

Although I have used this model or some variant of it at times, I had quite forgotten its name and origin until someone reminded me of it the other day. Of course, we often forget that UML also came out of Rational’s famous Grady Booch, Ivar Jacobson and James Rumbaugh..

And yet today I find it difficult to find software developers who know much of anything about the 4+1 model and only a few regularly use UML and even then only for simple class diagrams. So why the disconnect?

Perhaps a Version 2 of these approaches is required. What would you get if you mixed the 4+1 model with more clearly defined non-functional requirement scenarios as I’ve previously discussed and the C4 model presented by Simon Brown along with just a bit more UML that most developers are not currently accustomed to using? I’m thinking this through and will post more thoughts on it soon.

Venn-like View of Software Quality Attributes

Two days ago I posted Non-Functional Requirements for the Software Architect in which I suggested that quality attributes or non-functional requirements could be categorized into a two by two grid, operational and non-operational by internal and external.

Thinking further along those lines led me to picture the illustration below and hop out of bed to put it to virtual paper.

qualattribvenn

Most quality attributes will fit into one or two of the big four: security, performance, usability and modifiability. And each of these largely corresponds to its depicted quadrant.

Security is largely an internal concern. Often external stakeholders will profess their concern with this attribute, and hence the Venn-like spill over into the external quadrant. Generally the concern for security and assuring security falls within the domain of internal stakeholders and the implementation team.

External stakeholders are mostly concerned with performance and usability. There are a number of other quality attributes such as availability that fall within or at least are closely related to one or both of those.

Modifiability is almost exclusively a concern of internal stakeholders and implementation teams. Interoperability and testability are largely related to modifiability while interoperability nearly always shares at least some security concerns.

You may mix these differently to match your circumstances and priorities, but the illustration covers the majority of software development efforts but of course does not spell out the details. For guidance on that, I recommend my previous post. If you are concerned with software architecture, perhaps this will help you to visualize how these quality attributes or non-functional requirements compete and compliment one another.

To paraphrase Bass in Software Architecture in Practice, an architecture will either inhibit or enable the achievement of a system’s desired quality attributes. Understanding your desired quality attributes well will drive the critical decisions in your architecture. And your architecture will then provide the requisite containers into which you will place your functionality. To quote, “Functionality is not so much a driver for the architecture as it is a consequence of it.”

While I am not completely certain that is always the case, it is a principle that is well worth considering when creating and refactoring the architectural elements of your system. And it is my hope that by seeing this illustrated, it will get you thinking about how your own desired quality attributes interact with internal and external teams, which are operational and non-operational, and how they may compete with or compliment on another.

Non-Functional Requirements for the Software Architect

Countless failed software development projects have been kicked off with non-functional requirements delivered to the implementation team with little to no detail. Here are a few of the worst but most common examples:

  • Security – The software must be secure.
  • Performance – The software must be fast.
  • Usability – The software must be easy to use.

Most software professionals have been taught that non-functional requirements are important, but many projects skip over them in order to get to functional use cases and writing code. The result can be profound, leaving the implementation team without sufficient input to make critical design decisions that will be very costly to change when the non-functional requirement is later clarified.

What Every Non-Functional Requirement Needs

For every non-functional requirement, the software architect should assure that the following questions have been adequately answered.

  • To whom is this quality important?
    • Users and integrators
    • Management team
    • Implementation team
    • Operations team
  • Who will assure this quality is met?
    • Implementation team
    • Operations team
    • Management team
  • How will this quality be met?
    • Cross cutting constraints in software
    • System and network constraints
    • Log analysis and oversight
  • How will we know this quality is met?
    • Scenarios with measures
    • Monitoring and review
    • Acceptable tolerance percentiles

The items below each question are not meant to be an exhaustive list but simply to give you an idea of what may be involved in answering those questions.

Classification of Non-Functional Software Quality Requirements

Clarifying and prioritizing non-functional software quality requirements may be easier when you classify them into one of four groups by answering two questions: operational or non-operation, and internal or external. The following table is anything but exhaustive but it will give you the general idea.

Quality Classification Internal External
Operational Latency
Capacity
Fault tolerance
Performance
Security
Availability
Non-Operational Maintainability
Portability
Testability
Correctness
Usability
Accessibility

 
Business stakeholders are generally more interested in and will support efforts to meet external qualities. Implementation and IT teams sometimes have to work a little more to garner support for time and effort and expense for internal qualities.

It is often easier to build into an implementation the cross cutting concerns to measure operational qualities. Collecting performance, reliability and security metrics from executing code is always possible with well planned constraints early on in the development effort. If these qualities are defined later, the refactoring process can be challenging.

For non-operational qualities, other systems such as those used to manage support issues and ongoing development efforts are often helpful in measuring the cost of change to the system or whether usability goals are being met. Sometimes time series log analysis can be utilized to extract measures for non-operational qualities, especially those most important to external parties.

Use an Agile Approach to Non-Functional Requirements

However you choose to collect and document non-functional software quality requirements, you should continue to improve and tweak them throughout the development process just as you would with functional requirements, grooming your backlog and prioritizing based on ongoing feedback from stakeholders, users and developers.

Software Architecture for Developers

I have been enjoying an e-book called Software Architecture for Developers by Simon Brown that was very well worth the price. I also just finished watching the author’s presentation at the 2014 GOTO Conference. A very thought provoking presentation.

Here are a few things that I like very much from the book and presentation.

  • “The code is the single point of truth. It is the embodiment of the architecture.”
  • TDD does not replace architecture. Do TDD inside a set of boundaries and frameworks provided by the architecture. (see Why Most Unit Tests Are a Waste by James O Coplien.)
  • If the diagrams don't reflect the code, the diagrams are basically pointless. We're just deceiving ourselves.
  • Component testing is preferable over unit testing but unit tests and mocks for testing against async systems are still useful.
  • Layered architecture can lead to a big ball of mud because too much functionality is exposed for public use by other layers.
  • Component organization over layered is preferred. Components have limited public interfaces (aka ports) and may have layers within the component that are not improperly publicly accessible.
  • If a system has a very large number of unit tests, we may have an out of control layered architecture.
  • “If your software system is hard to work with, change it. This is entirely within your hands.”

C4: Context, Containers, Components, Classes

I especially like Brown’s use of what he calls C4. In essence, every software system can be broken down into a simple hierarchy:

  • Contexts (systems) made up of
    • Containers (web server, app server, database, browser, file system, etc.) which host
      • Components which expose one or more interfaces (sometimes referred to by others as ports) and contain
        • Classes which implement those interfaces and the layers behind them.

By creating architecture diagrams that follow this hierarchy, it is possible to create code that matches it. It is an architecture that developers can use directly. (Side note: I have often used a Visual Studio solution to lay out a project in similar terms directly in code. I have also gone down the layered road only to regret it later and end up pulling those layers apart and encapsulating them into a component-like approach to ensure that the responsibilities and behaviors exposed to the outside world are used properly and that the system remains well ordered.)

Here is my simplification of what the author already makes rather simple with respect to these constructs. I recommend you buy the book and get full details and example diagrams, but the ideas are what is most important.

  • Context Diagram - A context diagram answers what are we building, who is using it and how does it fit into our IT environment and business. It is important to note at a high level how users will interact with the system.
  • Container Diagram - In the container diagram, you define the shape of the software, high-level technology decisions, how responsibilities are grouped and separated, how each container communicates with other containers, and where the code lives for each container.
  • Component Diagram - The component diagrams answers what course grained building blocks of functionality are required, what are their responsibilities, and how will they interact with other components (interface and communication mechanisms such as sync, async, batched, etc.).
  • Class Diagram – Brown does not explicitly discuss this level, as far as I’ve read. I assume this is because it is rather intuitive to most architects and in part because the design at the class level is often better left to the implementation team. However, from my own perspective, the architect should consider defining the public interfaces and primary classes to name and separate key responsibilities into distinct code structures that will be intuitive for the implementation team to complete—naming conventions that separate what may be thought of as traditional layers and that the implementation team will understand is crucial here.

Some of what Brown proposes breaks with what others may consider traditional software architecture. I am excited about Brown’s challenging of the status quo, questioning our assumptions. I believe his ideas will help us narrow the model-code gap (see Just Enough Software Architecture by George Fairbanks—one I’ve just added to my Kindle collection for some fun future reading).

Measurable Non-Functional Quality Scenarios

Brown covers the importance of quality scenarios, but on this topic I prefer the more pedantic but measurable approach of Len Bass in Software Architecture in Practice. I believe his emphasis on defining metrics driven quality scenarios is well worth pursuing, especially to the point of implementing logging and monitoring systems that allow you to constantly measure non-functional quality and improve against those measures. Here’s the structure that Bass recommends:

  • Source – where does the input come from?
  • Stimulus – what is the input?
  • Artifact – what container and component are affected?
  • Environment – in what context does this occur?
  • Response – how and with what did the software respond?
  • Measure – how much time, how many clicks, how many errors, were proper notifications were sent?

Creating quality scenarios that developers understand and can incorporate into the software they are building is critical to long term success of the software. To paraphrase the quality gurus, if you can measure it, you can understand and improve it.

Most important, if your implementation team and your stake holders understand your architecture diagrams and documents, you are more likely to succeed. And if your code does not mirror your diagrams, no amount of code reviewing will tell you whether your architecture has been implemented. And that is Brown’s greatest point in my opinion.