A technically valid record can still be unsuitable for a decision. An invoice may contain every required field while referring to an unknown customer, carrying an unexpected currency or arriving after the reporting cutoff. Data quality therefore needs both structural checks and business context.
This approach places quality controls at ingestion, transformation and publication boundaries. Its purpose is to make reliability inspectable: what was checked, which records failed, how the issue affects a metric and who owns the next action.
Define a quality contract
Begin with an agreed dataset contract. Document its grain, required identifiers, field types, accepted values, expected arrival pattern and authoritative owner. A transaction table might represent one posted ledger line, while an order table might represent the latest state of each order. Their uniqueness rules will differ.
JSON Schema distinguishes between describing a field and requiring it. Defining a property does not make it mandatory; required fields need an explicit rule. Structural validation also needs a deliberate policy for new fields, because accepting every change and rejecting every change both carry operational costs.
Test the dimensions that affect decisions
Choose checks that reflect the consuming process. A finance report may require a balanced ledger and a complete period. An operational alert may tolerate provisional values but require recent events. A single universal quality score can hide these differences, so show the relevant checks and their scope.
| Dimension | Example check | Decision impact |
|---|---|---|
| Completeness | Every posted line has an account and period | Missing categories distort totals |
| Uniqueness | An event identity appears once per source | Duplicates inflate activity |
| Validity | Currency belongs to the agreed value set | Conversion may be unreliable |
| Consistency | Order lines reconcile to the order total | Measures disagree across views |
| Freshness | Latest expected delivery has arrived | Current results may be incomplete |
Separate detection from disposition
A failed check needs a defined outcome. Rejecting an entire dataset may be appropriate for a broken financial reconciliation, while isolating one malformed event may preserve a useful operational feed. The disposition should reflect impact, rather than whichever behaviour is easiest to implement.
Retain a reason code, source reference, affected field and processing version for rejected records. Limit access to sensitive values in diagnostic output. When a correction arrives, preserve the relationship between the original failure and the repaired record so a reviewer can follow the resolution.
Make analytical readiness visible
A publication gate evaluates whether the required checks have passed for a particular use. The same dataset may be ready for a provisional operations view but still awaiting approval for a management report. Record the evaluated partition or period; a passing check from yesterday does not establish today’s completeness.
Expose the latest successful update, unresolved critical checks and review state near the result. Distinguish missing data from a genuine zero. Monitor changes in rejection rates and arrival patterns, because a pipeline can continue running successfully while the information it produces becomes less useful.
Example quality resolution
- 01
Detect
A new currency value fails an agreed transaction rule.
- 02
Contain
Affected records are held with a reason and source reference.
- 03
Review
The data owner confirms whether the source or the contract should change.
- 04
Recover
Reprocess the affected input and rerun the relevant reconciliation.
Design considerations
- Separate an absent value, an unknown value and a meaningful zero.
- Version checks with transformations so historical results remain explainable.
- Set severity and response ownership before a quality incident occurs.
Sources
The following sources explain the underlying technical patterns.
