Agility, Enterprise Architecture and software development fit together in three models
- architect an agile enterprise
- agile work practices to develop best-practice enterprise architecture
- agile software development & enterprise architecture
Most architects jump to the last. They try to fit two wildly divergent worlds together, usually without stopping to understand either.
We optimize both methods by aligning them to their strengths. The shorthand we use is that enterprise architecture excels before decisions when you do not understand what to do. Agile software development excels at building something that we have never had before.
Both methods suffer from chronic misapplication. Despite TOGAF’s inherent concept of iteration, too many architects latch onto the ADM crop circle diagram and Saw process. It’s so easy to see a waterfall in the ADM diagram. Wrong, but easy. As well, the disorganized leap on the journey of discovery in agile development to hide their disorganization.
The actual world is messy. Unless we are talking about a simple one-trick-pony greenfield, software products must fit into a complex ecosystem. Existing processes, organization, partners, and infrastructure enable the business to serve customers. The new Product must enhance the ecosystem while fitting in.
It’s in real world complexity that effective agile development and enterprise architecture shine. Play to their strengths. We base strong agile software development practice on resolving an essential tension.
Support agile software development
Practitioners who work at the extremes of agile software development and enterprise architecture will likely never encounter each other. They might not even recognize each other’s work product. The essential tension is in their relationship with each other.
When you stop and consider the alignment of best practice EA and best practice agile software development for basic areas where they interact becomes apparent. Practitioners who work at the extremes of either Enterprise Architecture or agile software development may never know what the other person works for the company they may not see each other’s work product.
An EA practitioner working to support strategy and a tech lead on an agile software team work far apart. The tech lead lives an execution window measured in weeks or months. Release plans or architecture runways are long-term thinking. The strategic EA practitioner could have been working for years beforehand on the roadmap that laid out the development of agile software development capability.
I had an amusing conversation with the newly hired Systems Reliability Engineer recently. The SRE specialist was excited. We finally started modern practices CI/CD practices. She asked me what the EA Team was doing to help?
I had to smile when she asked, ‘what is EA Team doing to help?’ What she really meant was, ‘what are you doing to support me today?’ Today, against her immediate challenges it could be – nothing. She jumped to how the EA team supported her day-to-day job.
She never saw the portfolio roadmap that brought containers, test data management, and frankly a laughably insufficient testing suite into existence. She didn’t realize the same old roadmap identified a transition point that funded her job.
EA Team Design
Successful EA teams deliver on their purpose. They do not deliver on anything else. Even if they could.
When linking to agile software development, the same question is asked, what they should the team assist in answering. Some of the alignment is driven by what the enterprise architecture team is designed to support.
- defining the agile approach
- guiding the backlog in sprint
- constraining the sprints
- solving for cross product dependency
A simple framework for thinking about enterprise architecture is to ask what they support?
- support Strategy
- support Portfolio
- support Project
- support Solution Delivery
Architecture boils down to constraints. Every constraint has the potential to limit the creativity of an agile software development team. If there is no overriding need for constraint, do not. Just Don’t.
There is one basic rule: never remove a degree of freedom if you don’t have to. Freedom to innovate and creativity are the lifeblood of agile software development.
Define the agile approach
Typically, an EA Team that is serving Strategy and Portfolio will define the agile approach.
Products appear on a roadmap as gaps or work packages. The EA team may speak about capability or business service. When they do, expect to see the new, or reworked products.
What will support or operate the product? Knowing whether the product is stand-alone, based on a new platform, using an existing platform, or leverage in third-party platforms, will define key elements of your agile approach, usually well before sprint-zero.
Weaker practitioners will fixate on defining terms like “Platform.” A crisp, overly precise definition is a trap. Take a few seconds consideration,it is perfectly reasonable for someone to talk about SAP, 0365, Facebook, Pega, or even Angular and Containers as platforms. The term needs a context for a narrow definition. All you need is the consumer and operator to agree. I have succumbed to the temptation to specify a product that was a platform consumed by others. The pedants tried to determine whether it was product or platform.
Service delivery strategy
Top-level roadmaps better be clear about how to implement the transformation. Examples include.
- Build a robust in-house capability or use third parties?
- Ensure long-lived sustainable systems or treat some or all products as disposable Kleenex?
Major value resting points
We routinely use Value Resting Points as a synonym for architecture transitions. Always provide your stakeholders with an off-ramp. We use off-ramps for two reasons:
- When the effort to reach the next resting point exceeds the incremental value.
This is an ROI conversation. ROI conversations typically lead to change in priority.
- When the effort to reach one resting point could reach a more exciting or rewarding resting.
Trade-off conversations provide one of the most valuable outcomes of a strategy or portfolio roadmap exercise. Resources are finite. Senior leaders are always on the look-out for the best-path-forward, not the highest potential return. The best path. Considerations include comparative value at rest and potential value as a springboard for other activity.
Implementers rarely understand these conversations. They become emotionally vested in one path or resting point. Especially when the existence of the product, or the next release, is under consideration.
Guide backlog in Sprint
EA practitioners who support portfolio and project will often engage with agile teams by guiding backlog in Sprint.
I have observed that it surprises too many agile evangelists that organizations deeply committed to the benefits of agile software development remain deeply committed to planning and longer-term budgeting. Usually, the conversation confuses a mythical cultural preference for a waterfall.
Instead think about the problem. Longer-term planning and budgeting exist because of ecosystem complexity. Every organization has multiple paths forward and different constraints. Successful organizations perform prioritization and trade-off. Without reaching the preferred future in the time and resource constraints, no work should start.
Roadmap to guide product
Roadmap to guide product is among the most difficult deliverables for a talented EA Team. The fundamental tension of knowing where you are going without knowing how to get there is a hot spot.
Too many architecture teams fall into the trap of artificial precision or imagined omniscience. Both are fancy ways of saying waterfall thinking.
Agile development delivers stunning results when the team has every degree of freedom. Just think about the sweet spot of a greenfield stand-alone product.
Often good architecture comes down to providing the minimum constraints to ensure decisions do not fail a test that is not apparent to the decision-maker. If the decision-maker can see the test, you do not need architecture to constrain them. Architecture exists to guide or constrain choice when the factor is outside the architects field of view.
A classic EA roadmap will speak to transitions, gaps, and work packages. It will be incomprehensible to a product team. Transition the roadmap to the right terms with the product owner. The product owner needs to understand the constraints they are operating within.
Roadmap to guide epic
Roadmap to guide epic is the most difficult deliverable for a talented EA Team. I always think about being stuck in a minefield during a bombing. There is no-where safe to go.
Timeline expectations make the common trap of waterfall thinking worse. Jumping ahead of an agile team’s journey of discovery with artificial precision & imagined omniscience undermines everything in agile development. Just don’t do it!
Without tightly integrated, or tightly constrained products, roadmaps for epic are doomed to fail. Examples, where this is necessary, include multiple products coexisting within an ecosystem, and sharing reference data. Consuming each other’s data exhaust will often need associated epics or complex punitive regulation.
Without external pressure, do not perform this activity. Over-specification and over-design by self-designated omniscient practitioners are one thing we invented agile to dispense with. If you find yourself forced to roadmaps to epics, make sure you understand exactly why it is necessary to constrain the creativity of your software developers. Every constraint that restricts their freedom and creativity had better serve over-ride the value delivery that comes from unbridled creativity
When we have made this succeed, we have actively adopted the language of methods like SaFE and framed the roadmap in terms of strategic themes and architecture runways.
Superior architecture will always address common stakeholder concerns. It should provide an internally consistent set of principles that provide constraint on value definition. Here the architecture is ensuring that sloppiness in assessing value does not create downstream costs.
Sloppy value assessment is the most common problem good architecture addresses. Transitory considerations rule. A classic dichotomy is the distinction between tactical time to market and security, resiliency or stability. When your organization has imperatives product managers and scrum teams need to be told. Otherwise they cannot appropriately assess the backlog against metrics that truly matter.
Constrain ‘bottom up’ product owners
Too many product owners do not have visibility into why their product exists or Too many product owners do not have visibility to why their product exists, or what the ecosystem role is. Constrain the freedom of a bottom up product owner either by prohibiting choice or enforcing scoring and assessment.
EA practitioners developing architecture to support project and solution delivery should expect to constrain sprints. Portfolio-oriented practitioners who are developing an ecosystem or platform should also expect to work with the agile teams here.
Constraining sprints can be emotionally difficult for new architects who used to be active in development. It is hard to leave your old job behind. Often subject matter expertise will lead to creeping over-specification and over-design. Every constraint that restricts their freedom and creativity had better serve over-ride the value delivery that comes from unbridled creativity.
The difficulty of leaving your old job is why we treat everyone who is new to enterprise architecture as a newb. The new graduate and the person with 30-years’ experience doing something other than enterprise architecture are new architects.
I watch new architects with years of other experience closer. They have many bad habits that prevent them from being high-functioning architects.
A very simple linkage into a sprint is externally defined acceptance criteria. If the architecture requires something, frame it as acceptance criteria. Unless the level of detail is very close to interfering in agile creativity the constraint is going to be something consistent. There will be a demand from the business architecture, data architecture application architecture, or infrastructure architecture. The architecture specification will be a principal, pattern, or standard. Provide the constraint as acceptance criteria. Either align to a Sprint, or release. Then get out of the way and watch the creativity
- If yes, their interpretation should be accepted as compliance. This is a key point. Good architecture can have multiple implementation choices. The implementer is not required to adhere to opinion. If the implementation choice is a reasonable interpretation, it is compliant.
Value (measures and resting points)
A recurrent point of conflict is understanding value.
Best practice agile software development is value focused. Unfortunately, most practitioners have a limited understanding of value. The quick shorthand is value is expressed in terms of something was delivered. The value is self-evident in the delivery.
In a complex world the thing delivered could easily be degrading value. An easy example are features aimed away from target customers, or the movement of work between work units. This is common when one group is served by the product. We strongly recommend learning basic Lean and Six-Sigma practices about local optimization and value. As well, the Business Model Canvas concepts of Target Customer and Value Proposition are helpful.
To address these challenges, we expect the business architecture to be definitive about how value is assessed. The architecture needs to provide value assessment criteria to the development teams.
Greenfield, Evolution, or Revolution
In TOGAF’s Phase E there is a cool step. Look at the work package and select an appropriate strategy.
Will the work be Greenfield, Evolutionary or Revolutionary? Do we want to preserve as much as possible, radically refactor or start from scratch? This is critical guidance, and a strong constraint on an agile team. Do they to start from scratch (Greenfield)? Improve the existing systems incrementally (evolution)? Or perform radical change in expected to eliminate the friction and hassles we have been living with (Revolutionary)?
The approach choice may not be theirs. When they do not have the choice, it will always be driven by choices above their paygrade.
When a product must fit within an existing enterprise environment, or support an evolving enterprise environment, interfaces are key. Interfaces will be driven by data and method. Even when the product is emerging, in a complex world, there will not be freedom to allow data structure or interface to also emerge. Master data, reference data and existing systems will all constrain agile development.
Existing systems are not going to be changed. Even the F-22 Raptor had to connect with legacy systems using interfaces developed in the 1970s. Even a plane that expensive could not get legacy systems remodeled. The Raptor fit inside existing interface and data structure.
A common constraint overlooked by fast-moving teams is how multi-jurisdictional legislation,or a business plan will impact the product. In these cases, the EA team has a duty to dig in and develop the minimum necessary constraints to prevent the product from requiring radical re-factoring at an inconvenient moment. Like all constraints on creativity, good EA practitioners understand the importance of just enough.
EA practitioners working in an enterprise can be expected to need to step in and solve dependency and conflict that arise between products solutions and systems. Simply look at TOGAF phase G, it highlights the need for guidance and architecture change. Agile development and the scope of modern integration make this long-lived need service more important than it has ever been.
Unblock the portfolio
Even if you are not constrained by legacy, there will be multiple products moving forward. Creative best practice agile development teams are going to find different ways to address problems. It is foolish to expect that creativity will not create conflict. A good EA team working in Phase G, where all agile software development happens, can be expected to unblock the conflict across the enterprises portfolio.
Identify real stakeholders
Real stakeholders are often hard to find. Many tactical decisions are delegated. When the delegation is formal, the agile team can expect to have reasonable access. Where the delegation is informal, local subject-matter-experts and users can be expected to drive an unaligned agenda.
The enterprise architecture team does not have a special ability to gain engagement. They do have an ability to represent the stakeholders concerns through superior architecture.
Cross the portfolio
Product owners, and other members of a development team, are driven to incrementally deliver one product. Minimal viable product and incremental delivery lay at the core of the agile value proposition.
The whole agile approach comes from the observed failure of trying to imagine every interaction across a complex portfolio, this is the failure path. Detailed top-down enterprise design can work for isolated systems. Inside an ecosystem that is changing, top-down design always fails. That failure path is why agile exists and why agile is preferred.
It is nearly impossible to imagine a complex future, efforts to do so often devolve into farce. Yet we need to do just enough. As multiple products emerge, new conflict and synergy will emerge. A good EA team, with a rich understanding of the changes in priorities of their enterprise, will help address synergy and conflict.
When the EA team discovers intersections, they need to use their insight and engagement to get the best answer. Often this will be a risk-managed approach. Rapidly assessing the conflict to determine whether the resolution can be deferred is the first step. Taking advantage of synergy and avoiding conflict are expensive. Expensive and time- consuming. They directly interfere with time-to-market.
We speak in terms of market forces and creative destruction when we must cross the portfolio.
Just enough architecture means that every contingency, every constraint, every conflict, was not discovered prior to release.
There is one case when an architecture team has an emergency. It is post-release crisis. Those rare cases when implications extend past the product. Product is in the wild. End users inside and outside our organization are using it. They are either dealing with a defect, or worse, creating unanticipated risk and liability for our organization.
Both agile development and enterprise architecture suffer from chronic misapplication. Too often at the same time.
Successful EA teams are deliberately designed. They are designed to deliver on their purpose.
When your EA team needs to support agile software development do a thoughtful, deliberate design.
Success is at hand.