Data lineage describes how information moves from its sources through transformations into the results people use. It helps answer where a number came from, which outputs depend on a source and what may be affected when a definition changes.
OpenLineage provides a public object model centred on datasets, jobs and runs. This explainer uses that model to make lineage concepts concrete. It is an educational reference for evaluating traceable analytical workflows, not a claim about Covorg’s internal implementation.
Distinguish datasets, jobs and runs
A dataset is an identifiable collection of data, such as a table or a defined object location. A job describes a repeatable transformation. A run is one execution of that job. Keeping these concepts separate allows a team to compare the intended process with a particular execution that succeeded, failed or used different inputs.
| Object | Example | Question it answers |
|---|---|---|
| Dataset | Approved order records | Which data object was read or written? |
| Job | Calculate daily order measures | Which transformation produced the output? |
| Run | One scheduled execution | What happened during this execution? |
| Input metadata | The period read by a run | Which subset of the source was used? |
| Output metadata | Record count for the result | What did this execution produce? |
Use stable identity and useful metadata
OpenLineage identifies a dataset using a namespace and name. This avoids treating identically named tables in different systems as the same object. The chosen naming convention should remain stable enough for changes and dependencies to be tracked over time.
Facets extend the model with metadata. Some details describe the dataset, such as schema; others describe the input or output of a particular run, such as the partition processed. Recording the latter is essential when an otherwise familiar job produces an incomplete period or a different population.
Follow the evidence behind a result
Start an investigation with the affected report edition or metric observation. Identify its producing run, the definition version and the exact input scope. Compare the recorded quality checks and source state before deciding whether the issue originated in extraction, transformation or interpretation.
Lineage is most useful when connected to operational evidence. A diagram alone may show that a source feeds a report, but not whether yesterday’s source delivery was incomplete. Run status, counts, quality outcomes and observation boundaries help explain the particular result being reviewed.
Use dependencies to review change
When a source field or metric definition changes, traverse its downstream relationships to identify affected models, reports and alerts. Review which owners need to approve the change and which historical comparisons may become inconsistent. Column level lineage can add precision where the instrumentation supports it.
Lineage coverage is not automatically complete. Manual spreadsheets, exported documents and external transformations can create gaps. Document those boundaries and establish a way to register important dependencies. Also protect lineage metadata itself when names, query text or diagnostic details reveal sensitive business information.
Example lineage investigation
- 01
Locate the result
Identify the report edition, metric and observation period.
- 02
Inspect the run
Review the transformation version, status and recorded input scope.
- 03
Trace the source
Compare source completeness and quality evidence with the result.
- 04
Assess impact
Locate related outputs and notify their responsible owners through the agreed process.
Design considerations
- Distinguish intended design relationships from evidence of a particular execution.
- Identify gaps caused by manual work and unmanaged exports.
- Restrict sensitive metadata and query details according to their content.
Sources
The following sources explain the underlying technical patterns.
