Skip to content

Lineage Tracking

Track how your data was transformed — which columns came from where, which rows were filtered, and what operations were applied. Lineage is session-only by design: it lives in memory and is never written to disk.


Enabling Lineage

Pass lineage=True to any core function:

import additory as add
import polars as pl

orders = pl.DataFrame({
    'order_id': [1, 2, 3],
    'customer_id': [101, 102, 101],
    'amount': [100, 200, 150],
})

customers = pl.DataFrame({
    'customer_id': [101, 102],
    'name': ['Alice', 'Bob'],
})

result = add.to(orders, customers, 'name', 'customer_id', lineage=True)

Lineage metadata is attached to the resulting DataFrame and carries forward through subsequent operations.


Viewing Lineage

Use add.scan('@lineage', df) to inspect the lineage report:

report = add.scan('@lineage', result)
print(report)

The report shows:

  • Operation history — each transformation in order
  • Column sources — where each column originated
  • Row mapping — how rows were filtered or aggregated

Chaining Operations

Lineage accumulates across multiple operations:

# Step 1: Bring data
df = add.to(orders, customers, 'name', 'customer_id', lineage=True)

# Step 2: Calculate
df = add.transform('@calc', df, expression='amount * 1.1', name='total', lineage=True)

# Step 3: Filter
df = add.transform('@filter', df, where='total > 150', lineage=True)

# View complete history
report = add.scan('@lineage', df)
# Shows all 3 operations, column sources, and row transformations

What Gets Tracked

Column Sources

For each column in the result, lineage records its origin:

  • Original — column existed in the input DataFrame
  • Brought — column was added via add.to() from a reference DataFrame
  • Calculated — column was created by add.transform('@calc', ...)
  • Synthetic — column was generated by add.synthetic()

Row Mapping

Lineage tracks how the row count changes:

  • Filtering — which rows were excluded and why
  • Aggregation — how rows were grouped
  • Synthetic augmentation — how many rows were added

Supported Functions

Function Lineage Support
add.to() ✅
add.transform() ✅
add.synthetic() ✅
add.scan() — (reads lineage, doesn't produce it)

Session-Only Storage

Lineage metadata is stored in the DataFrame's native format (as DataFrame attributes). It is:

  • ✅ Preserved when passing DataFrames between additory operations
  • ❌ Lost when saving to CSV, Parquet, or other file formats
  • ❌ Lost when converting between pandas and polars

This is intentional — additory follows a "no file I/O" philosophy. Lineage is a development and debugging tool, not a persistence layer.


Mutual Exclusion with as_type

lineage=True and as_type cannot be used together:

# This raises ValueError
result = add.to(orders, customers, 'name', 'customer_id',
    lineage=True, as_type='polars')

The reason: as_type converts the DataFrame to a different type, which would strip the lineage metadata. If you need both, track lineage first, then convert separately:

# Track lineage
result = add.to(orders, customers, 'name', 'customer_id', lineage=True)

# Convert separately (lineage is lost)
result_polars = pl.from_pandas(result)

Lineage is lost on type conversion

Converting a DataFrame with as_type or manually (e.g., pl.from_pandas()) strips lineage metadata. View the lineage report before converting.


Performance

Lineage tracking adds minimal overhead:

  • Less than 3ms per operation
  • No significant impact on large DataFrames (tested with 10,000+ rows)
  • Memory overhead is proportional to the number of operations, not the data size

Next Steps