Lineage Tracking¶
Track how your data was transformed — which columns came from where, which rows were filtered, and what operations were applied. Lineage is session-only by design: it lives in memory and is never written to disk.
Enabling Lineage¶
Pass lineage=True to any core function:
import additory as add
import polars as pl
orders = pl.DataFrame({
'order_id': [1, 2, 3],
'customer_id': [101, 102, 101],
'amount': [100, 200, 150],
})
customers = pl.DataFrame({
'customer_id': [101, 102],
'name': ['Alice', 'Bob'],
})
result = add.to(orders, customers, 'name', 'customer_id', lineage=True)
Lineage metadata is attached to the resulting DataFrame and carries forward through subsequent operations.
Viewing Lineage¶
Use add.scan('@lineage', df) to inspect the lineage report:
The report shows:
- Operation history — each transformation in order
- Column sources — where each column originated
- Row mapping — how rows were filtered or aggregated
Chaining Operations¶
Lineage accumulates across multiple operations:
# Step 1: Bring data
df = add.to(orders, customers, 'name', 'customer_id', lineage=True)
# Step 2: Calculate
df = add.transform('@calc', df, expression='amount * 1.1', name='total', lineage=True)
# Step 3: Filter
df = add.transform('@filter', df, where='total > 150', lineage=True)
# View complete history
report = add.scan('@lineage', df)
# Shows all 3 operations, column sources, and row transformations
What Gets Tracked¶
Column Sources¶
For each column in the result, lineage records its origin:
- Original — column existed in the input DataFrame
- Brought — column was added via
add.to()from a reference DataFrame - Calculated — column was created by
add.transform('@calc', ...) - Synthetic — column was generated by
add.synthetic()
Row Mapping¶
Lineage tracks how the row count changes:
- Filtering — which rows were excluded and why
- Aggregation — how rows were grouped
- Synthetic augmentation — how many rows were added
Supported Functions¶
| Function | Lineage Support |
|---|---|
add.to() | ✅ |
add.transform() | ✅ |
add.synthetic() | ✅ |
add.scan() | — (reads lineage, doesn't produce it) |
Session-Only Storage¶
Lineage metadata is stored in the DataFrame's native format (as DataFrame attributes). It is:
- ✅ Preserved when passing DataFrames between additory operations
- ❌ Lost when saving to CSV, Parquet, or other file formats
- ❌ Lost when converting between pandas and polars
This is intentional — additory follows a "no file I/O" philosophy. Lineage is a development and debugging tool, not a persistence layer.
Mutual Exclusion with as_type¶
lineage=True and as_type cannot be used together:
# This raises ValueError
result = add.to(orders, customers, 'name', 'customer_id',
lineage=True, as_type='polars')
The reason: as_type converts the DataFrame to a different type, which would strip the lineage metadata. If you need both, track lineage first, then convert separately:
# Track lineage
result = add.to(orders, customers, 'name', 'customer_id', lineage=True)
# Convert separately (lineage is lost)
result_polars = pl.from_pandas(result)
Lineage is lost on type conversion
Converting a DataFrame with as_type or manually (e.g., pl.from_pandas()) strips lineage metadata. View the lineage report before converting.
Performance¶
Lineage tracking adds minimal overhead:
- Less than 3ms per operation
- No significant impact on large DataFrames (tested with 10,000+ rows)
- Memory overhead is proportional to the number of operations, not the data size
Next Steps¶
- Logging & Timing — measure operation performance
- Type Handling — understand the
as_typeparameter - add.scan() — the
@lineagemode in detail