EFL-ES-2025-001 / ENGINEERING STANDARD
Evaluating whether different implementations preserve the same observable contract.
Abstract
As our systems accumulated multiple implementations behind common interfaces, we reached a point where type compatibility was no longer enough to establish confidence in interchangeability.
Two components can expose the same methods, accept the same argument types and return objects with the same structure while still behaving differently in ways that matter to the operation using them. One implementation may normalize an identifier differently, collapse an unresolved outcome into failure, discard provenance that another implementation preserves or interpret absence according to different assumptions. From the language runtime’s perspective, both satisfy the interface. From the application’s perspective, they may no longer represent the same contract.
We encountered this problem while building adapters with more than one technology binding for the same domain. Remote and local implementations could compile against the same interfaces yet differ in normalization, freshness, error representation or treatment of incomplete information. Ordinary unit tests around each implementation often passed because they verified the implementation against itself rather than comparing the observable behavior promised at the shared boundary.
This document presents the conformance model we developed in response. It distinguishes structural, semantic, behavioral and failure conformance, and treats reproducibility as the practical mechanism through which those claims can be tested across implementations. Different implementations are expected to have different internals. The concern here is the behavior that another component has been told it can rely on.
1. The interface was correct, and the implementations still disagreed
The problem first became obvious when more than one implementation began satisfying the same runtime-facing contract.
Consider a simplified registry capability:
lookup(identifier) → Record
One implementation obtains the record through a remote service. Another reads from a synchronized local dataset. Both accept the same identifier and both return the same nominal Record type.
At the language level, there may be nothing to distinguish them.
Now suppose the identifier 0012345 is submitted.
The remote implementation preserves the leading zeros because the authority considers them part of the canonical identifier. The local implementation converts the value through an integer representation during import and returns 12345.
Both return a valid string afterward. Both satisfy the function signature. Only one has preserved the identity required by the domain.
We encountered less obvious variants of the same issue. One implementation represented a missing optional value as null, another as an empty string. One preserved the source version of a local record while another returned the record without any indication of freshness. A remote failure that one connector classified as unresolved could be treated as definitive absence by another.
None of these differences necessarily causes a type error. They become visible when another part of the system relies on the meaning promised by the contract.
2. We needed a stronger meaning for “implements”
Object-oriented languages already give the word implements a precise technical meaning. A class declares an interface and satisfies the methods and signatures required by that interface.
We continue to rely on that mechanism. It is useful and catches an important class of errors early.
Our problem was that architectural discussions had begun using the same word more broadly. When we said that two adapters “implemented the same contract,” we often meant that the runtime should be able to use either without changing the operational assumptions made at that boundary.
We kept using interface compatibility as the first check, but stopped treating it as sufficient evidence that implementations were interchangeable. The remaining differences usually appeared in behavior the type system could not express: interpretation of values, treatment of repeated operations and what an implementation reported when the underlying result was uncertain.
3. Structural conformance
Structural conformance is the closest level to ordinary interface checking.
An implementation should accept and produce the structures defined by the contract. Required fields should exist, types should be compatible and values should satisfy structural constraints established at the boundary.
For example, assume the shared result is defined conceptually as:
Record ├── identifier : string ├── name : string ├── source : string └── retrieved : timestamp
An implementation returning the identifier as an array is structurally non-conforming even if the array happens to contain the correct value. Likewise, omitting a required source field because one implementation does not currently use it changes the observable structure for consumers that do.
Most of these cases are straightforward to automate. Schema validation, type checks and contract-level assertions cover a large part of this layer, and disagreements here tend to be found early. In our work, the harder cases generally appeared after both implementations were already structurally valid.
4. Semantic conformance
Two structurally valid records can still mean different things.
The leading-zero example is a simple case. Both implementations return a string, but they disagree about the identity represented by that string.
Status fields create another common problem. Imagine that two external systems use the value PENDING. One means that a request has been accepted and is still processing. Another uses it before the request has been accepted at all.
Mapping both directly into a shared PENDING state may satisfy a structural schema while hiding a semantic difference that matters during recovery.
Semantic conformance concerns these interpretations at the boundary. Generic validation cannot settle all of them because domain knowledge is sometimes required before we can say whether two representations are equivalent.
We keep those decisions explicit in the contract wherever the difference can affect downstream behavior. If leading zeros are significant, the contract should say so. If absence differs from an unresolved remote outcome, those conditions should not be represented by the same value merely because one implementation has fewer native states.
5. Behavioral conformance became visible through sequences
Individual return values were not enough for some contracts.
Operational components participate in sequences. A lookup may be followed by validation. A submission can later be queried. An adapter may be invoked more than once for the same logical operation.
Two implementations can return equivalent values for a single successful case while behaving differently once state or repetition enters the picture.
Suppose a contract allows the same logical request to be repeated with a stable operation identifier. One implementation recognizes the repeated identifier and returns the existing outcome. Another performs the external effect again.
Their method signatures remain identical. Even many isolated unit tests may pass.
The disagreement becomes visible when the test includes a sequence:
1. submit OP-42 2. lose local confirmation 3. submit OP-42 again 4. inspect externally observable result
We began treating sequences like this as part of the conformance surface when the contract makes claims about behavior over time. They became particularly useful around idempotency, recovery, synchronization and state transitions, where a single function invocation does not contain enough information to evaluate the behavior.
In one binding we may rely on an idempotency mechanism provided by the remote system. A local implementation of the same capability can enforce the operation identifier in its own persistence layer. We have no reason to make those mechanisms resemble each other; the conformance case exercises the repeated operation and observes what each implementation does.
6. Failure behavior belongs to the contract too
Successful cases tend to receive the clearest specifications. We found that implementations diverged more often when something went wrong.
Consider an unavailable remote record.
One implementation may have received a structured response establishing that the record does not exist. Another may have timed out before receiving any authoritative response. Returning null from both implementations would erase a distinction that can change what the caller is allowed to do next.
The same issue appears with malformed input, unavailable dependencies, partial responses and stale local information.
For one capability the relevant outcomes might be:
FOUND NOT_FOUND UNRESOLVED INVALID_INPUT UNAVAILABLE
Another contract may need fewer states or distinguish other conditions. What mattered in our tests was that the meaning assigned to an outcome remained stable across the implementations claiming the same contract. If NOT_FOUND represents authoritative absence, a failed network request cannot establish it.
We began adding these cases deliberately after discovering that happy-path suites gave us almost no information about such disagreements.
7. Conformance does not mean identical output
Requiring byte-for-byte identical results looks like an easy way to compare implementations. It works only when byte identity is itself part of the contract.
A remote implementation may include a retrieval timestamp different from a local one. A provider can return fields in another order. One implementation may preserve additional diagnostic information that another cannot obtain. Those differences can coexist with equivalent contract behavior.
Suppose two implementations produce:
Implementation A identifier = "0012345" name = "Example Company" source = "registry-api" retrieved = 10:32:15 Implementation B identifier = "0012345" name = "Example Company" source = "registry-dataset" retrieved = 08:00:00
If the contract permits different source and retrieval metadata while requiring canonical identity and name preservation, both outputs satisfy that expectation.
Writing vectors this way forced us to identify which fields actually belonged to the common expectation. Earlier tests that compared complete serialized objects had sometimes encoded incidental details of the first implementation, making a second implementation appear incompatible for reasons that had never been part of the contract.
8. Reproducible cases changed how we reviewed contracts
Our earliest cross-implementation tests were created for specific defects.
A particular identifier had been normalized incorrectly. A missing field behaved differently between bindings. A timeout produced a state that allowed an unsafe retry. We kept the input that exposed the problem and wrote an assertion around the expected result.
As the collection grew, those cases became useful beyond regression testing.
When another implementation claimed the same contract, we could run the established cases against it. The discussion changed from whether the new implementation looked equivalent to what it actually did with cases whose semantics had already been established.
A simple vector might contain:
CASE: IDENTIFIER-001
input:
identifier = "0012345"
expected:
canonical_identifier = "0012345"
outcome = FOUND
A failure-oriented case could describe an external condition and the observable result required at the boundary:
CASE: REMOTE-TIMEOUT-004
condition:
request transmitted
no authoritative response observed
expected:
outcome = UNRESOLVED
retry_safe = false
We have represented cases in code fixtures and machine-readable data depending on the project. JSON or YAML are convenient when the same cases need to move between implementations, but the serialization format has never been the important part. What matters is that the expectation can exist independently of the implementation being evaluated.
9. The test vector should not know the implementation
One of the first mistakes in conformance testing is writing vectors that secretly describe the implementation they were created for.
Suppose the remote binding uses HTTP and its test case says:
expect HTTP 200 expect JSON field "status" expect header X-Request-ID
That can be an entirely valid integration test. A local dataset implementation can never satisfy it because the observations belong specifically to HTTP.
At the shared contract level, the case needs to express what the runtime depends on after those details have been interpreted. The remote implementation can still retain separate tests for status codes, headers and JSON parsing.
The same problem appears with persistence. A contract vector should not require a record to exist in a particular SQL table unless that storage location is genuinely part of the contract.
We eventually kept these checks in separate test families because we needed both kinds of information during development:
implementation tests
→ Is this binding or connector built correctly?
conformance tests
→ Does this implementation preserve the shared contract?
Some cases appear in both suites. We do not try to remove that duplication when the same input is useful for answering both questions.
10. A practical conformance hierarchy
By the time we had enough cases, four levels of conformance were recurring in reviews:
STRUCTURAL
↓
SEMANTIC
↓
BEHAVIORAL
↓
FAILURE
We do not treat this as a formal maturity ladder where every contract must have equally elaborate tests at all four levels.
Some contracts are predominantly structural. A serializer with no state may need little behavioral testing. An integration responsible for externally consequential operations needs much stronger failure semantics.
The hierarchy gives us a way to state what a conformance claim actually covers without requiring every component to accumulate the same testing machinery.
An implementation that passes structural vectors may therefore have a perfectly valid structural conformance claim even if ambiguous external outcomes have not yet been tested. We record that limitation rather than promoting the result to a broader claim.
Semantic cases create a similar problem because some of them require domain fixtures that a generic framework suite cannot provide.
The scope of the claim should match the evidence available.
11. The implementation is allowed to know more
A conforming implementation will often know things the shared contract does not expose.
An HTTP binding may know response headers, TLS details and remote timing. A local dataset implementation knows which dataset version produced the record. A connector may retain diagnostic information useful during support.
We do not require those details to disappear. The contract defines the information another component is allowed to depend on, while implementations can preserve richer evidence internally or expose it through diagnostics and specialized interfaces where appropriate.
This became relevant as we narrowed several common contracts. Trying to expose every useful diagnostic value through the shared interface gradually enlarged that interface around the needs of individual implementations. We now leave implementation-specific information where it originates unless another consumer has a demonstrated reason to depend on it.
12. Conformance needs versioned expectations
Contracts change.
Sometimes a change is compatible: an optional diagnostic field is added or another outcome provides more detail without changing existing semantics. Other changes alter an assumption consumers have already relied on.
Test vectors made those changes unusually visible because an updated implementation could suddenly fail cases that had previously represented correct behavior.
We found it useful to version the contract expectation together with its conformance cases. A vector should be interpretable in the context of the contract version against which it was written.
This does not require elaborate versioning infrastructure for every internal interface. The level of ceremony should follow the number of independent implementations and the cost of disagreement.
Where several systems, adapters or external integrations rely on a common contract, preserving the history of its expected behavior becomes considerably more useful. When a historical vector starts failing, we can then determine whether the implementation regressed or whether the expected behavior was intentionally changed.
13. What we currently require before claiming conformance
We use the word conformance more carefully now than we did when the adapter architecture was first introduced.
For a contract with multiple independent implementations, we expect at least an explicit description of the observable structure, known semantic rules that affect consumers and reproducible cases covering the behavior the runtime actually depends on.
Where failure outcomes affect subsequent action, those outcomes belong in the cases as well.
We also expect the same contract-level vectors to be executable against each implementation claiming conformance. Implementation-specific setup is unavoidable—a remote service and a local dataset are prepared differently—but the case itself should not change its expected meaning according to which implementation is under test.
This remains a practical engineering standard rather than a certification framework. The resulting suites vary considerably in size. An internal formatter may need only a small collection of structural fixtures, while an adapter capable of producing an external consequence justifies failure and recovery cases that would be excessive for the formatter. We still record what each suite has actually established when describing its conformance status.
Closing observation
The first versions of these tests existed because particular integrations had failed in ways their interfaces could not describe. We kept the failing inputs, added expected outcomes and used them to prevent the same defect from returning.
Their broader value appeared only after a second implementation reached the same boundary. At that point, the fixtures were doing something ordinary unit tests had not done: they allowed us to compare two different implementations against an expectation neither implementation owned.
Since then we have kept adding cases for much the same reason. Some began as regressions, others came from disagreements discovered while implementing another binding, and a few were written before an implementation existed because the contract contained a behavior we already knew would matter.
That left us with another engineering problem. The cases were useful, but their value depended increasingly on whether somebody other than their original author could execute them against another implementation and obtain a comparable result. We began treating portability and reproducibility of the vectors themselves as part of the work.
Document record
Document ID: EFL-ES-2025-001
Title: A Practical Standard for Contract Conformance
Document type: Engineering Standard
Category: Engineering Standards
Publication year: 2025
Institution: EventFlow Labs
Language: English
Revision: 1.0