What are the objectives of intentional architecture?

"While we must acknowledge emergence in design and system development, a little planning can avoid much waste." James Coplien, Lean Architecture.

It isn't my aim here to be comprehensive and embrace all sources. For your interest, I have looked at "Open Agile Architecture" and at SAFe. One blogger has reported counting that SAFe has 31 artifacts, 17 events (meetings), 16 roles, 10 principles, 7 competencies, 4 configurations, 3 levels, and 1,000s of pages of needed documentation. 

My aim here is less ambitious. It is to speak to the right-hand side of the graphic above, and expose the ambiguity of "agile architecture", since

• an agile software or architecture development process
• an agile system
• the agility of people as opposed to software

are three very different things.

An agile architecture development processes is perfectly reasonable, provided the architects who document “intentional architectures” understand the wider enterprise context in which the system of interest sits, and understand the various trade-offs outlined later in this chapter.

On the other hand, frameworks and standards for agile architecture can be overblown. This chapter exposes how the term may mislead people, and thus demystifies the concept for architects and their managers.

The need for architectural oversight

At one extreme, enterprise architects constrain all decisions made by solution architects. At the other extreme, solution architects are free to fight their own battles (which may lead disintegration of systems). Naturally, the optimal path lies between the extremes. The same applies to the relationship between software architects and solution architects.

The graphic above features feedback loops between architects at different levels. The higher feedback loop between enterprise and solution architecture is not explored here. This chapter focuses on the lower feedback loop between solution and software architecture.

The graphic is not intended to imply the demand for work on software systems is entirely top-down, strategic, road map-driven. The more decentralized and agile the business, the more the demand is localized. Wherever the demand comes from, there is a need for architecture-level work and documentation. To repeat from the beginning of this book:

“With or without a business strategy, there will always be a need for architects with a long-term and cross-organizational perspective to have oversight of IT-related projects and product developments (with a view to making the business more efficient and effective as whole) and to identify and mitigate technical risks - including the risks of letting a thousand flowers bloom under the direction of software developers wanting to extend their CV."

Some basic concepts

An architect works to ensure the coherence and essential qualities of systems to be built, first by designing them at a high or abstract level, then by governing the work of system builders.

An architecture includes the essential inputs and outputs, structures and behaviors, of one or more human and/or computer activity systems, and essential dependencies between them, along with essential technology choices.

In system theory terms, the architecture is an abstract system, and the built system is a concrete system, instantiated by some human and/or computer actors. In agile architecture terms, an agile architecture approach relates an intentional architecture (in a high-level design document) to an emergent design (abstracted from working code, reflecting the structures and behaviors of a run-time system).

Of course, there should be a feedback loop. An architect's intentional architecture constrains a builders' emerging design. Conversely, the emerging design can prompt revisions to the intentional architecture.

If a development team is empowered to make decisions, then gaps between what was intended and what emerges are likely. When an architecturally-significant design feature emerges in working code, it should be explained to the architect directly, or at the next governance review.

Agile development may mean the governance feedback loop is shorter and more effective, but it doesn't make the intentional architecture any more agile. It doesn’t change the solution architect's role beyond more frequent engagement with the emerging design.

What is documented?

The agile manifesto promotes four values.

• Value individuals and interactions - over processes and tools
• Value working software - over comprehensive documentation
• Value customer collaboration - over contract negotiation
• Value responding to change - over following a plan

Note that the manifesto does not say there is no plan, no process or no documentation. As Adele Goldberg used to say of code: “If it ain't documented; it don't exist".

The graphic above suggests that the intentional architecture is recorded by a solution architect. It is as comprehensive as it needs to be for its audience, which is usually well short of detailed software architecture. This high-level documentation might be maintained by the development team, under the governance of a solution architect who has the wider enterprise picture in mind.

The working code is the most detailed documentation of a system. The framework called SAFe includes the concept of the “The Architectural Runway“ consisting of “the existing code, components, and technical infrastructure needed to implement near-term features without excessive redesign and delay”. It seems the runway contains just enough (rather than too little or too much) implemented code and technology to deploy and run the next few increments.

The emergent design may be thought of as a mental model of the evolving system/solution, reverse-engineered from latest working code. To help developers, software architects may document some of it, but in an agile world, it may not be maintained.

So, what does agile architecture mean?

The term agile means adaptable, readily changed or maintained in the light of changing needs or conditions. The term agile architecture is ambiguous, since the thing we craft to be readily changeable or maintainable may be:

1. an intentional architecture - in a high-level design document, and/or
2. an emergent design - reflecting the structures and behaviours of a run-time system.
3. the entire structure and operations of a business.

The next three sections discuss these three different ideas separately.

1) Agile intentional architecture

The intentional architecture is in a high-level design document, written with a view to ensuring the coherence and the essential qualities of a system. It stops far short of defining the whole of a system to be built. It is an abstraction from a system that is envisaged.

The architect determines the boundary of the system (large or small), defines its external interface(s), and chooses the granularity of parts or subsystems to be documented at an abstract level, with some required features or qualities in mind.

If agile architecture means you high-level design documentation can readily be changed and maintained, then how to make it so? Do less of it, generalize, or encapsulate?

• Less up front design? The less design you do, the less you'll need to change your architecture. But also, the less you can direct, constrain and join up the builders. Which leads to more diversity both within and between systems that are built. The choice is up to you.
• More generic design? The more generic your architectural design, the less it will need to be changed. But generalisation is the enemy of usefulness. Either the builders must go the extra mile to design what is specific to the users’ requirements. Or, the users must go the extra mile to make use of the system. The choice is up to you.
• Design by encapsulation? What is an interface? In TOGAF, it is a data flow - an inter-application exchange of data. To human users, it is a user interface. In software architecture, it is usually an API that encapsulates a component, and is accessible by any number client components, whether they are on the same device or a remote one.

Architects focus on defining interfaces, which should be more stable than whatever is behind them. TOGAF goes so far as to position business and application service contracts in the requirement specification rather than the architecture definition document. But of course, every interface is potentially subject to change requests arising from emergent design.

Most systems decomposed into layers and modules. Every layer (notably, application and database) and every component in the application (from the whole application to the smallest object) can be encapsulated behind an interface. Which interfaces are solution architects expected to specify? Certainly the interface of the whole system of interest. How far a solution architect should be engaged in the division of what is considered to be a single application into layers and components is an open question.

Does a solution architect need to understand anything behind the interface(s) of an application? Usually, yes. The architect should

• know what persistent data is maintained, and how it relates to data maintained by other applications.
• be engaged in decisions that decouple application components so far that they are developed by different teams and/or deployed as microservices (see below).
• ensure a system meets whatever non-functional requirements are important.
• guide technology choices.

In a typical enterprise application, the business data is processed at every level from user interface to database. Some up-front effort to scope and structure persistent data can help to minimize the time and cost of refactoring later, which makes a contribution to agility. Call me old-fashioned, but where they are appropriate, I like to see a list of use cases/epics and a well-drawn data model.

2) Agile emergent design / run-time systems

If agile architecture means the run-time system can readily be changed, then how to make it so? Make it more general, more flexible, or chop it up into smaller relatively independent parts?

• A more general system? The more generic the system is, the less it will need to be changed. But again, generalisation is the enemy of usefulness. It means users must go the extra mile to make use of the system. It means you get no competitive advantage from business-specific features. The choice is up to you.
• A more flexible system? The more flexible the system is, the less it will need to be changed. But design for future flexibility is an antithesis of agile development - its principles are the opposite of KISS and YAGNI. It makes a system more complex, more costly and slower to build, and usually slower in operation. Again, the choice is up to you.
• Smaller relatively independent subsystems? The smaller the system you design, the easier it will be to change. But system decomposition is the enemy of overall coherence.

Wrt to that last: for several decades now, software engineers have been encouraged to decompose large systems into smaller ones, and decouple them, making it easier to change and maintain the individual subsystems. However, to paraphrase Craig Larman: decoupling components now for a future that is not in sight is not time well spent. See the discussion of microservices below.

Again, the choice is up to you. You have to consider the down sides of decomposing, distributing and decoupling systems. To begin with, it generally means:

• Extra messaging
• Extra work for end users.

And it can create the 4Ds.

• Duplication of data in discrete data stores.
• Disintegrities and the complexity of compensating transactions to restore integrity.
• Delays where processes are slowed by the need for inter-application messaging.
• Data analysis difficulties where reports require data from several data stores.

The important thing is to think about these things before you commit to system decomposition and decoupling of the subsystems.

3) Making an "agile transformation"

The term "agile architecture" is sometimes conflated with the idea some call an "agile transformation". That term is also ambiguous. A transformation that proceeds in an agile, incremental and reversible way is one thing. A "transformation to an agile business" is a completely different thing.

Ironically, transforming the entire structures and operations of a business to be more agile implies big up front design and planning. Note also that agility for the sake of being agile is is potentially damaging to efficiency and effectiveness, since design for flexibility tends to makes a system more complex and slower in operation.

• Enterprise architecture (EA) is about the human and computer activity systems an organization participates in, performs and employs.
• Business change management (BCM) is about the impact of changing those systems and/or other things on the actors employed in roles by the social entity that is the organization. Also, about changes to the management structure, culture, ways of thinking, leadership styles.

Where the need is for BCM then, typically, architects work in partnership with business change specialists - be they in the same or a different team. In short, EA and BCM must and do collaborate when needed, as would certainly be the case in a "transformation to an agile business".

Principles for the architects of software projects

Returning to the world of agile software development, the agile manifesto has 12 principles for a team that delivers working code. How to recast those principles for architects who deliver documentation and guidance to such a team? Here’s a suggestion you can adapt as you see fit.

1. The highest priority is to ensure the quality of the delivered solution,
2. Welcome changing requirements.
3. Deliver an intentional architecture early, and maintain it.
4. Work with stakeholders and developers, connect them where it helps.
5. Support and trust developers to get the job done.
6. Communicate face-to-face if possible, Zoom if not.
7. Encourage continual delivery of working software.
8. Don’t overwork and overstress the developers.
9. Strive for good design and technical excellence.
10. KISS (keep it simple); and YAGNI (don’t specify what is unnecessary).
11. Re-organize the architecture and governance processes need be.
12. Regularly reflect on refine your way of working.

Also, don't be a agilist zealot. Be sensitive to the real-world political, social and technical obstacles to agile development (not listed here). Where the obstacles cannot be overcome, then adapt the agile principles to the circumstances of the project and the nature of the solution to be delivered.

Remarks on practice and architectural options

The practices above called agile architecture (meanings 1 and 2 rather than 3) correspond to what I've observed and taught for 20 years as normal practice. A lesson to be learned from experience of empowering stand-alone agile software development projects is that agility should be constrained by architectural oversight, and in the face of various political, social and technical obstacles (not listed here) it is not always possible to apply the whole of the agile development dogma.

Before making architectural decisions, a solution architect should start with analysis of the motivations, and the trade-offs between various design options, including:

• data store modularisation options (large v small)
• application modularisation options (e.g. procedural v object-oriented)
• system/subsystem coordination design patterns (e.g. orchestration v choreography)
• messaging technology options (e.g. synchronous v asynchronous)
• software deployment options (e.g. in-prem v cloud).

Countless permutations of the options are possible. The beginning of wisdom for a solution architect is to understand that making a solution work doesn't mean it is the best solution, or even close to it. All good designers start by understanding the motivations for and trade-offs between alternative designs.

Are microservices the answer?

An architect complained his CIO speaks of "microservices" as meaninglessly as he used to speak of "SOA”. Be sceptical when vendors propose you divide monoliths into microservices, then use their middleware to coordinate them. Be sceptical when software engineers (for their CVs) propose using every fashionable design pattern.

To divide a monolith into microservices conflates several different kinds of design decision into one sledge hammer of a decision. Before doing it, you should start with analysis of the motivations, and the trade-offs between the various options above. Directing all developers down one path without addressing trade-offs is a root cause of problems in practice.

Some advice posted on the internet under the heading of "microservices" confuses microservices in particular with services in general. Wrapping up applications behind APIs and calling them "services" is one thing. The complexities caused by distributing stored data is another thing. The benefit of enabling local agility can be outweighed by the costs - the 4 Ds above, and other complexities.

Further reading

The article above is an extract from the second of two books nearing completion:

1. Systems thinking renewed
2. EA and systems thinking.

The first is written mainly for college teachers and students. The second is written more for professional enterprise and solution architects. For other extracts from the second book, read some of the other articles here. Notably, the article on microservices, on which comments have included: "Superb." "Very informative, good to read the practical points of consideration around business, architecture and design."