Monica Colangelo
← All articles

Architecture · Architecture governance · Reference architectures · Project estimation

Reference Architecture Fit Gap Analysis and Project Estimate Risk

·8 min read

The choice of a published reference architecture does more than decide how the system gets built. The same choice sets what the project will cost, how long the work takes, and what the delivery team is committed to.

Once a pattern has been documented and benchmarked and presented at a conference, it goes into proposals, often before anyone has checked that it fits the system that has to host it, or looked at the risk that choice puts on the project.

The numbers published with a pattern usually answer two questions:

  1. Does the pattern do what it claims?
  2. Is the added latency, or the added infrastructure spend, acceptable?

The answer to a third question decides whether the project can use the pattern at all:

  1. Is the pattern incompatible with a mechanism the system needs?

The documentation is not always going to answer that question, so whoever prepares the estimate has to ask it. In system selection, comparing what a product provides against what the business needs, and then working through each gap that comes out, is called a fit gap analysis. The name fits what has to happen here as well, with a published pattern in place of the product.

Standard Mechanisms an Architecture Pattern Can Be Incompatible With

Some of the mechanisms worth checking before an estimate:

  • Connection reuse and pooling
  • Read replicas, when the design requires reading back what was just written
  • Caching, at any layer
  • Retries, and repeating a call safely
  • Transactions, in particular designs that require the work to happen inside one transaction
  • Bulk and batch work, such as nightly jobs, exports and reporting
  • Schema migrations while the application is serving traffic
  • Backup and restore, including whether a restored copy still applies the same rules
  • Horizontal scaling, with any instance able to serve any request
  • Staged rollout, with the old and the new version running at the same time
  • Logs and traces, when the design forbids some data from appearing in them

A pattern can be incompatible with one of these and still be fine, as long as the system does not need that mechanism. The problem starts when the system needs the mechanism the pattern is incompatible with, because then the design has to change before the pattern can be used. The work of changing it belongs in the estimate, but it is visible only if someone asks the question early.

Tenant Isolation and Connection Pooling as a Worked Example

The idea for this kind of analysis came to me while I was implementing per-request scoped credentials for AI agents on Amazon Bedrock AgentCore.

A documented pattern for tenant isolation issues a credential scoped to one tenant for every request, so a request can reach only the data of that tenant, and a query for another tenant returns nothing. The datastore changes from one implementation to the next, but the mechanism does not.

Every request reaches the database through a client object, and assembling that object takes time, so an application normally assembles it once and reuses it. With credentials issued per request, the part of the object that carries the credentials belongs to one tenant and cannot serve the next one.

The obvious version rebuilds everything each time, and most of the added latency goes there:

# wrong: the session is rebuilt on every request
session = boto3.Session(**tenant_credentials)
table = session.resource("dynamodb").Table(TABLE)

The correct version keeps the session, which carries no tenant identity, and rebuilds only what is bound to the credentials:

# right: the session is assembled once per container
_SESSION = boto3.Session()
table = _SESSION.resource("dynamodb", **tenant_credentials).Table(TABLE)

Both versions implement the same pattern, and the diagram is identical in both cases. Telling them apart takes knowing that a Lambda container is reused across invocations, so anything assembled once survives into the next request, which may belong to another tenant. Details of this kind are architectural constraints, and the decision cannot be made from a diagram alone.

Another example, on a relational database, follows the same reasoning. The constraint is one level below, in how privileges are scoped. In a database like PostgreSQL, privileges are attached to the role of the session, and a session is a connection, so a different role per call means a different connection per call. An open connection cannot be reused for the next call, because that connection still carries the role of the previous call.

On the other side, connection pooling is the established pattern for how an application connects to a relational database. Opening a connection is expensive, so the application keeps a set of connections open and hands them to requests as they arrive, which is what makes a database serve many requests with few connections. An application built on a pool cannot adopt this design. The published latency measurement never shows the problem, because a latency test runs one call at a time and opens a single connection.

The tension between the two has a known resolution, and it works by moving where the isolation is enforced. Instead of a credential per tenant, the application keeps one shared role, enables row level security on the tables, and sets a tenant identifier inside each transaction. The database still filters the rows, and the connections stay interchangeable, which is how AWS describes the pool model for multi-tenant PostgreSQL.

The two are not equivalent, because with row level security the isolation depends on the application setting the identifier in every transaction. Choosing between them requires knowing both at that level of detail. Session behaviour, warm containers, row level security and connection pooling are not implementation notes to settle later, they are what decides whether a pattern fits the system in front of you.

Reference Architecture Examples With Adoption Constraints

The same problem appears in published patterns from other areas, each of them in wide use, each taking away a mechanism a system may depend on.

  • Serverless function concurrency on a relational database. One execution environment serves one request at a time, so the open connections grow with concurrency, and the proxy that solves it shares a connection only if no caller leaves session state on it.
  • Personalized responses behind a CDN. A page that changes with the user cannot be served from the one cached copy, so the origin has to be sized for most of the traffic on that page.
  • Data that must not appear in logs. The traces have to carry identifiers in place of the forbidden fields, or the application stops being debuggable during an incident.

Requirements of this kind often arrive labelled as non-functional. I do not agree with that definition, and I prefer to call them operational requirements: they decide whether the system can be run, and they cost development time like any other requirement.

The list could go on much longer, but I think I have made the point.

Examples of patterns and the risk each one brings.
Examples of patterns and the risk each one brings.

Testing Architecture Assumptions With Benchmarks

Not all of the information needed to decide whether a pattern fits is available while the architecture is being designed, and what is missing is replaced by an assumption. They are called estimates, after all, and not certainties.

An assumption has to be tested on the system that has to run the pattern, on two levels, and the test can support it or disprove it. The qualitative one is whether the mechanisms the system needs still work. The quantitative one is what the pattern costs under the conditions the system will meet.

The conditions of the measurement have to be decided before the measurement itself. Defining them is architectural work, and this is the minimum set to write down, with room for more depending on the system:

  • the load the test runs at, in requests per second and in requests served at the same time
  • what else uses the same resources during the test, from other services on the database to scheduled jobs
  • the state of the system while the test runs, such as a release in progress or a migration on a table being read
  • the environment the test runs on, and where it differs from production
  • the value each measurement has to stay within for the result to be acceptable

The conditions in this form give the test its shape, so that it produces the information the decision needs. They enter the estimate twice: as the criteria the delivered system will be accepted against, and as the work to test it that way.

Measuring a pattern is part of applying it, so the benchmarks belong to the development process rather than to the time left at the end of a project. Run early enough, they either support the decision taken during the estimate or disprove it while there is still room to change it.

A risk identified during the estimate, with a test and its conditions written next to it, can be checked during the project and closed when the test passes.

Reference Architecture Fit Gap Analysis Before a Project Estimate

In summary, four questions have to be answered before the pattern goes into a proposal:

  1. Is this pattern incompatible with a standard mechanism, and which one?
  2. Does the system need that mechanism?
  3. Is there a version of the pattern without the constraint, and what does it give up in exchange, in guarantees, in latency, or in work to build?
  4. Under which conditions will these answers be verified, and what result is acceptable?

The answers to those questions are not in the description of the pattern. Knowing a pattern in detail, down to how it treats session state, or connection reuse, or container reuse, or transactions, or caching, is what turns an assumption into a constraint that can be estimated. A pattern understood only in outline is applied wrongly, and a pattern nobody knows is not applied at all, so the system is left without the property that pattern exists to provide. The absence of that knowledge is the largest risk in an estimate.

References

The comparison between a published pattern and the system that has to run it is close to what TOGAF calls a gap analysis, and splitting the result in two is close to how the ATAM sorts what an evaluation finds, into risks and non-risks.

← All articles