API
An application programming interface defines how software requests information or actions from another system. A data connection must account for authentication, pagination, rate limits and changes to the interface.
Explore the related guideKnowledge library A shared language
A shared language
for connected intelligence.
Technical concepts are more useful when their meaning is clear. Explore the terms behind data architecture, integration, quality and business analysis.
28 terms
Batch processing
Processing a bounded collection of records together, usually on a schedule or after a file arrives. A batch needs a clear reporting period, completion criteria and a way to recover from partial failure.
Explore the related guideBusiness metric
A quantitative measure with an agreed business meaning. Its definition should specify the population, calculation, time period, exclusions and responsible owner.
Explore the related guideChange data capture
A pattern that identifies changes to source records and makes those changes available to another system. Log based capture reads database change information rather than repeatedly extracting every row.
Explore the related guideData contract
An agreement about the structure, meaning, quality and delivery expectations of a dataset. It helps producers and consumers understand which changes are compatible and who is responsible when expectations are not met.
Explore the related guideData fabric
An architectural approach to connecting distributed data through shared metadata, integration, governance and access. The design depends on the sources, analytical needs and operating responsibilities of the organisation.
Explore the related guideData freshness
How recently an analytical result reflects activity in its source. Freshness is different from query speed and should be measured against the time when the underlying business event occurred.
Explore the related guideData lineage
A record of where data came from and how it was transformed. Lineage can connect a report or metric to the datasets, jobs and source fields that contributed to it.
Explore the related guideData quality
The suitability of data for its intended use. Useful checks can include completeness, validity, uniqueness, consistency, timeliness and reconciliation against an authoritative source.
Explore the related guideDeduplication
Identifying records that represent the same business event and deciding which should be retained. The correct rule depends on stable identifiers and the meaning of an update in the source system.
Explore the related guideDimension
Descriptive context used to group or filter a measure, such as customer segment, product category or reporting period. Consistent dimensions help teams compare the same business population.
Explore the related guideETL and ELT
Extract, transform and load describes transforming data before loading it into a destination. Extract, load and transform moves the transformation into the destination environment. The choice depends on controls, tooling and processing needs.
Explore the related guideEvent time
The time when an event occurred in the business or source system. It may differ from the time when an analytical pipeline receives or processes that event.
Explore the related guideGovernance
The responsibilities, policies and controls that guide how data is defined, used, protected and changed. Governance connects technical controls with accountable business ownership.
Explore the related guideGrain
The level of detail represented by one record in a dataset. For example, one row per order and one row per order item have different grains and cannot always be combined without changing an aggregate.
Explore the related guideIdempotency
A property that allows the same operation to be repeated without introducing an additional unintended effect. In ingestion, stable event identifiers can help a retry avoid inserting the same business event twice.
Explore the related guideIncremental load
A process that transfers records added or changed since an earlier load. It needs a reliable change boundary and a policy for late changes, deleted records and recovery.
Explore the related guideLeast privilege
Granting only the access required for a defined task. For data systems, this includes the records, fields, actions and service credentials available to each person or process.
Explore the related guideMaterialisation
Persisting the result of a transformation or query so it can be read later. Stored results can improve retrieval performance but require an explicit refresh and consistency policy.
Explore the related guideMetadata
Information that describes data, such as field types, ownership, definitions, update times and lineage. Metadata helps people and software interpret a dataset correctly.
Explore the related guideObservability
The ability to understand the condition of a system through evidence such as logs, metrics and traces. A data workflow also needs signals about freshness, quality, volume and delivery failures.
Explore the related guideReconciliation
Comparing records or totals across systems to explain differences. A useful reconciliation specifies the comparison population, timing rules, tolerances and treatment of exceptions.
Explore the related guideSchema
A description of the structure of data, including field names, types and constraints. A change to a schema may affect downstream transformations even when the source application continues to work.
Explore the related guideSemantic model
A business oriented model that defines entities, relationships, dimensions and measures. It gives dashboards, reports and analytical queries a consistent interpretation of underlying data.
Explore the related guideStreaming
Processing an ongoing sequence of events as they become available. A streaming design must consider ordering, duplicate delivery, late arrivals and the boundaries used for aggregation.
Explore the related guideValidation
Checking data against explicit rules before it is accepted for a purpose. Failed records need an understandable reason and a controlled route for investigation or correction.
Explore the related guideWatermark
A boundary used to track progress in a data process. An incremental extraction may use a saved source position, while event processing may use a time boundary to reason about late arrivals.
Explore the related guideWebhook
An HTTP request sent by a source application when a subscribed event occurs. A receiver should verify the sender, acknowledge appropriately and handle retries or duplicate deliveries.
Explore the related guideNo matching terms.
Try a broader concept such as quality, source, event or metric.