EFL-ES-2025-002 / ENGINEERING STANDARD
Designing portable cases that can be executed independently of the implementation that produced them.
Abstract
Conformance testing became useful to us only after the same expectations had to be exercised against more than one implementation. At that point, ordinary fixtures tied closely to a specific codebase were no longer enough. We needed cases that could survive changes in transport, storage mechanism and implementation language while still expressing the behavior expected at a shared boundary.
Many of our first reusable fixtures were simply regression cases retained after integration defects. Others appeared when two implementations of the same capability disagreed despite satisfying the same interface. The difficulty was not moving test data from one codebase to another. Hidden assumptions moved with it: SQL setup, HTTP clients, internal enums, clocks, dataset versions and implementation-specific assertions.
We began separating the case from its execution harness. A reproducible vector records the relevant input or initial condition, the contract version being evaluated and the observable result expected at the boundary. The harness remains responsible for arranging whatever implementation-specific environment is necessary to reproduce the case.
This standard describes the working rules that emerged from that distinction and the limits we found while applying it across remote services, local datasets and operational components with state.
1. Our first reusable fixtures were not actually portable
The earliest fixtures were written to make defects reproducible inside the codebase where those defects occurred.
A malformed identifier would be stored in a test fixture. A remote response that had been interpreted incorrectly might be copied into the suite. The implementation would run against that input and assert the corrected result. For regression testing, this worked well.
The problem appeared when a second implementation reached the same contract. A fixture that looked reusable often depended on assumptions that had never been written down. One test required a particular SQL seed script. Another expected a response object created by a specific HTTP client. A third relied on an internal enum that did not exist outside the original component.
Even when the input itself was stored in JSON, the surrounding test still described how the first implementation worked.
We began separating the case from its execution harness. The case described the condition and expected contract-level observation; each implementation arranged whatever internal setup it needed to reproduce that condition.
2. A vector begins with a contract
A file containing input and expected output does not tell us enough by itself.
Consider:
input:
identifier = "0012345"
expected:
identifier = "0012345"
Several questions remain open. Is the case testing transport parsing, domain normalization or the runtime-facing result? Is the expected value canonical identity or simply an echo of the submitted input? Does whitespace matter? What happens if the authoritative source returns a different representation?
We began recording the contract and, where relevant, its version alongside the case:
vector_id: IDENTIFIER-001
contract: registry.lookup
contract_version: 1.0
input:
identifier: "0012345"
expected:
canonical_identifier: "0012345"
outcome: FOUND
The serialization format varies between projects. The important part is that another engineer can identify the boundary being evaluated before reproducing any implementation-specific setup.
3. Condition and mechanism need different representations
Failure vectors forced this distinction to become more precise.
An early test for an uncertain remote outcome might have been written as:
mock HTTP client throws TimeoutException
That is useful when testing the HTTP implementation. It is not portable to a local implementation that has no HTTP client and can never throw that exception.
At the shared boundary, the condition we cared about was closer to:
operation transmitted authoritative outcome not observed
One harness may reproduce that with a socket timeout. Another can use a deterministic remote simulator that accepts the request and drops the response. A local test implementation can establish the same contract condition through controlled state.
The expected observation remains:
outcome: UNRESOLVED retry_safe: false
Transport-specific tests still exist below this boundary. They verify that a timeout, reset or protocol response is interpreted correctly by the binding. The portable vector starts after enough of that implementation detail has been interpreted for the shared contract to make a claim.
4. Initial state matters when the contract has history
A single input is sufficient for many stateless components. Operational contracts often depend on what happened earlier.
Idempotency made this obvious.
If a logical operation has already produced an effect, repeating the same request is a different test from submitting it for the first time. Two implementations starting from different histories can produce different results without either being wrong.
We therefore include initial state when the contract depends on it:
vector_id: IDEMPOTENCY-003
contract: operation.submit
contract_version: 1.0
initial_state:
operation_id: OP-42
existing_effect: true
action:
submit:
operation_id: OP-42
payload: example-A
expected:
effect_count: 1
outcome: EXISTING_RESULT
A remote harness may prepare that state in a simulator. A local implementation may seed its persistence layer. The vector does not prescribe how the state is created; it states what must already be true before the action is executed.
5. Some cases need a sequence
A number of contracts could not be tested faithfully with one input/output pair.
A submission may succeed remotely, lose local confirmation and later be verified. A synchronization process may begin from one valid dataset, receive an invalid replacement and be expected to preserve the previous active version.
A compact sequence can express this:
vector_id: SYNC-FAILURE-007
contract: reference_dataset.refresh
contract_version: 1.1
initial_state:
active_dataset: V41
V41_valid: true
steps:
- acquire V42
- validation_of_V42 fails
expected:
active_dataset: V41
synchronization_status: FAILED
We keep these scenarios small. When a vector starts describing most of an application workflow, it is usually crossing more than one contract boundary and becomes difficult to diagnose when it fails.
6. Expected output should contain what the contract actually promises
Snapshot comparison was convenient in some early suites. We could serialize a successful output and compare future executions against it.
That became brittle as soon as another implementation appeared.
A remote binding might return a different retrieval timestamp. A local implementation may expose another source identifier. Field order can differ. One implementation may retain additional diagnostics that another cannot obtain.
For example:
Implementation A identifier = "0012345" name = "Example Company" source = "registry-api" retrieved = 10:32:15 Implementation B identifier = "0012345" name = "Example Company" source = "registry-dataset" retrieved = 08:00:00
A contract requiring canonical identity and name preservation can accept both outputs even when source and retrieval metadata differ.
Writing vectors at this level took more work than snapshot comparison, but it removed failures caused only by timestamps, ordering or diagnostic values that had never been part of the shared expectation.
7. Determinism sometimes requires controlling the environment
A portable vector can still produce different results if the environment is allowed to change underneath it.
Live registries change. Time-sensitive behavior depends on the clock. Random identifiers differ between runs. Network conditions can make a failure appear or disappear. A synchronized dataset can legitimately contain different values one month later.
For deterministic conformance work we control these variables where practical. Time can be fixed or injected, randomness seeded, datasets pinned to a version and external behavior reproduced through a simulator or recorded fixture.
We also have cases where the real external system is part of what we want to validate. We keep those, but classify them separately:
deterministic conformance vectors external verification cases
The second group can provide valuable production-oriented evidence, especially before integration changes are released. Its result carries more environmental context because the system under test is not fully controlled.
8. Results became easier to compare once we recorded the run
When executions began being compared across dates and machines, the vector itself was no longer always enough.
A failure observed months later is difficult to investigate if we do not know which contract version, implementation or dataset produced the earlier result. For the suites where those variables matter, we retain a compact execution record:
suite_version contract_version implementation implementation_version environment dataset_version executed_at result
Not every field belongs in every project. We keep the information that can reasonably explain why two executions of the same vector differ.
This has been particularly useful with synchronized external data. An unchanged implementation can produce a different valid result simply because the authoritative dataset moved from one version to another.
9. A vector needs to be understandable outside its original harness
Several conformance disagreements were resolved during review before a test was executed. That made readability more important than we initially expected.
An engineer may need to understand why a vector exists, and sometimes a domain specialist who never touches the implementation has to confirm whether its expected result is correct.
A case like:
case: IDENTIFIER-NORMALIZATION-004
input:
value: "0012345"
expected:
value: "0012345"
is easier to inspect than a serialized internal object containing opaque flags, factory references and implementation-specific enum values.
Large suites still use shared schemas, defaults and reusable setup. We do not require every vector to restate the whole contract. The contract-level meaning of a case should simply remain recoverable without reading the implementation it is testing.
10. The vector schema became a small contract of its own
Once several harnesses consumed the same vectors, changes to the vector format started affecting independent implementations.
Renaming a field could require modifying every harness. Making an optional section mandatory could invalidate historical cases. A convenience added for one implementation could introduce concepts that others had no reason to understand.
We became conservative about these changes. Additive evolution is usually easier. Optional metadata can be introduced without rewriting existing vectors, while semantic changes can be associated with a schema or contract version when compatibility cannot be preserved.
Implementation-specific setup remains outside the shared vector when possible. If one harness needs an HTTP fixture name or a database seed path, that configuration can live with the harness rather than becoming part of the common case format.
11. Negative vectors exposed more disagreement than successful ones
The first suites naturally contained many valid inputs and successful outcomes. They were easy to author and useful when bringing up another implementation.
The cases that later found the most interesting differences were often negative.
We kept examples of identifiers that were structurally valid but semantically unacceptable, remote outcomes that could not be classified conclusively, datasets missing required fields, repeated operations that must preserve identity and synchronization candidates that should be rejected without replacing the previous active state.
These cases exposed assumptions that implementations had made independently.
They also helped prevent seemingly helpful normalization from changing the contract. One implementation may silently coerce an input that another rejects. If the expected behavior is never represented outside either implementation, both can continue to look correct inside their own test suites.
When a defect reveals a contract-level distinction, we usually keep the case rather than reducing it to a test of the particular line of code that failed.
12. Portability does not require every implementation to execute every vector
Some conditions genuinely belong to one class of implementation.
A local dataset binding cannot reproduce every live-service failure. A capability may be optional under a contract. Transport-specific behavior can be worth testing without becoming a requirement for all other bindings.
We associate vectors with the scope they exercise:
required:
IDENTIFIER-001
ABSENCE-002
NORMALIZATION-003
failure_semantics:
UNKNOWN-004
UNAVAILABLE-005
remote_binding:
REMOTE-TIMEOUT-006
The applicable groups depend on the conformance claim being evaluated. The remote-specific case remains valuable for that binding without redefining the general contract around a condition other implementations cannot meaningfully reproduce.
Reports record which vectors were applicable and which were executed. This has prevented a nominally portable suite from gradually accumulating technology-specific requirements simply because those requirements existed in the first implementation.
13. Reproducibility has a practical threshold
It is possible to preserve almost every variable in a test environment: container versions, dependency hashes, fixtures, simulator state, runtime versions and every artifact consumed by the suite.
For some systems that level of control is justified. For others it would cost more than the disagreement being prevented.
A small internal structural contract may need only stable fixtures and deterministic unit tests. An adapter with several independent implementations and the ability to produce external consequences justifies substantially more evidence.
We calibrate the reproducibility effort according to the claim being made and the cost of being wrong.
The scope of the claim should match the evidence available.
A deterministic vector executed against pinned inputs establishes a different kind of evidence from a successful observation against a changing external service. We retain both where they are useful, but do not report them as equivalent.
14. The minimum vector we currently reuse
For contract-level suites, a reusable vector usually contains at least:
vector_id contract contract_version input or initial condition expected contract-level observation
Stateful behavior can add initial state or a short sequence. Execution records carry environmental information when it can affect the result.
Cases encoding a non-obvious domain rule normally include a short explanation or reference. We learned to do this after seeing unusual expected values “corrected” by maintainers who had no way to know that the unusual value was the reason the regression case existed.
The harness remains free to use whatever setup is appropriate for its implementation.
Closing observation
Many of these practices appeared because a supposedly reusable fixture failed the first time we tried to run it somewhere else. The data file moved successfully; its assumptions did not.
We spent more time than expected finding those assumptions in mock objects, database setup, clocks, environment configuration and assertions copied directly from one implementation’s output.
The useful result was a change in what we considered part of the test case. Old regression cases began surviving implementation changes because the expectation was no longer buried entirely inside the code that originally produced the defect.
Some of those vectors now outlive parts of their original harnesses. When another implementation disagrees with them, the discussion can begin from the recorded condition and expected outcome rather than from reverse-engineering an assertion written for a different component.
Document record
Document ID: EFL-ES-2025-002
Title: Reproducible Test Vectors for Integration Boundaries
Document type: Engineering Standard
Category: Engineering Standards
Publication year: 2025
Institution: EventFlow Labs
Language: English
Revision: 1.0