Schema and Data Evolution

Schema evolution enables non-breaking modifications to a database table’s structure — such as adding columns, altering data types, or dropping fields — to adapt to evolving data requirements without service interruptions. LanceDB supports ACID-compliant schema evolution through granular operations (add/alter/drop columns), allowing you to:

Iterate Safely: Modify schemas in production with versioned datasets and backward compatibility
Scale Seamlessly: Handle ML model iterations, regulatory changes, or feature additions
Optimize Continuously: Remove unused fields or enforce new constraints without downtime

Schema evolution operations

LanceDB supports four primary schema evolution operations:

Adding new columns: Extend your table with additional attributes
Altering existing columns: Change column names, data types, or nullability
Updating field metadata: Attach or change per-column Arrow metadata
Dropping columns: Remove unnecessary columns from your schema

Schema evolution operations are applied immediately but do not typically require rewriting all data. However, data type changes may involve more substantial operations.

Each schema evolution operation commits a new table version and returns status metadata such as the committed version. Run these operations from a mutable table handle; if you checked out an older version for reads, call checkout_latest / checkoutLatest before modifying the schema.

Add new columns

You can add new columns to a table with the add_columns method in Python, addColumns in TypeScript/JavaScript, or add_columns in Rust. New columns are populated based on SQL expressions you provide.

Set up the example table

First, let’s create a sample table with product data to demonstrate schema evolution:

Add derived columns

You can add new columns that are derived from existing data using SQL expressions. For feature engineering on large existing tables, group related derived features into one add_columns operation instead of running many separate writes. This creates one new table version for the schema change and computes the new columns from the existing rows, which avoids growing the table’s version history with many small updates. The same call can add multiple derived columns at once. For example, if you are building several lightweight features from existing product fields, pass all of the new column expressions together:

LanceDB add_columns does not currently accept Python callables, batch UDFs, or PyArrow RecordBatch iterators for populating new columns. New column values must be defined with SQL expressions, or added as NULL columns from an Arrow field or schema. If your transformation cannot be expressed in SQL, compute the values outside add_columns before writing them back through another workflow.

Add columns with default values

Add boolean columns with default values for status tracking:

Add nullable columns

Add timestamp columns that can contain NULL values:

When adding columns that should contain NULL values, be sure to cast the NULL to the appropriate type, e.g., cast(NULL as timestamp).

Declare computed columns

You can also declare a column whose values are defined by a SQL expression but not evaluated at commit time. LanceDB stores the expression in the column’s field metadata, commits the column with no values, and fills the rows on a later refresh. The column’s type and its input columns are derived from the expression, so you do not pass a data type. Use this form when you want to add a derived column to a large table without paying the cost of computing every row up front. Declaring a computed column costs the same on an empty table as on a large one, because no values are written at declaration time. Regular add_columns transforms, in contrast, evaluate the SQL expression against every existing row and write the results in the same commit.

# Declare a computed column; values are filled by a later refresh.
table.add_columns(computed={"doubled": "x * 2"})

// Declare a computed column; values are filled by a later refresh.
await table.addColumns({
  computed: [{ name: "doubled", valueSql: "x * 2" }],
});

// Declare a computed column; values are filled by a later refresh.
table
    .add_columns()
    .computed("doubled", "x * 2")
    .execute()
    .await?;

A declaration stays authoritative for the column’s lifetime. While it is in place, LanceDB rejects writes and schema changes that would give the column a value or reshape its output:

add, update, merge_insert, and SQL INSERT are refused for the declared column.
The declared column cannot be renamed, retyped, or dropped.
An input column named in the expression cannot be renamed, retyped, or dropped while the declaration reads it.
Volatile expressions (for example, expressions whose value can change between calls) are refused at declaration time.

A refresh fills every fragment that has no value for the declared column, including fragments appended since the last refresh. A refresh does not revisit a fragment it has already filled, so mutating an input row leaves the previously computed value in place. To recompute values, drop the column and declare it again.

Computed columns work on both local tables and LanceDB Enterprise. On Enterprise the declaration is sent to the server, which plans the expression against the published contract; refresh runs as a server-side backfill job (see the next section).

add_columns cannot mix a regular transform with a computed column in the same call. Declare computed columns in a separate add_columns call from any evaluated transforms.

Refresh a computed column

A declared computed column starts empty. Call refresh_column (Python and Rust) or refreshColumn (TypeScript) to evaluate the expression and fill every row that still has no value:

result = table.refresh_column("doubled")
print(result.rows_filled, result.version)

const { rowsFilled, version } = await table.refreshColumn("doubled");

let result = table.refresh_column("doubled").await?;
println!("filled {} rows at version {}", result.rows_filled, result.version);

The call returns the number of rows it filled and the new table version. Each run picks up rows appended since the previous refresh; rows that already have a value are left alone, so calling refresh_column when nothing new needs filling is a no-op that costs one scan of the unfilled rows. Because refresh never revisits a filled row, mutating an input after the fact does not change the stored value — to recompute, drop the column and declare it again. The blocking form is refused when the table uses an LSM write specification, and is refused on LanceDB Enterprise because a remote refresh runs as a server job that does not report a fill count. On Enterprise, submit the refresh with the async form below instead.

Run the refresh in the background

If you don’t want to block on the refresh, call the async variant to get back a job handle. On local tables the job runs as an in-process task; on LanceDB Enterprise the call submits a server-side backfill job and returns a handle that tracks it. Wait for it or poll its status when convenient.

job = table.refresh_column_async("doubled")
job.wait()
print(job.status())  # "finished"

const job = await table.refreshColumnAsync("doubled");
await job.wait();
console.log(await job.status()); // "finished"

let job = table.refresh_column_async("doubled").await?;
job.wait().await?;

Invalid input — an unknown column, or one that is not a declared computed column — is reported by the submitting call rather than by the job, so you learn about mistakes before you start waiting. The returned job may already be complete; treat the column as filled only after wait returns. On LanceDB Enterprise, a successful wait also advances the submitting table handle’s read-freshness baseline so subsequent reads see the refreshed rows — unless a checkout has pinned the handle to a specific version by the time the job completes.

Alter existing columns

You can alter columns using the alter_columns method in Python, alterColumns in TypeScript/JavaScript, or alter_columns in Rust. This allows you to:

Rename a column
Change a column’s data type
Modify nullability (whether a column can contain NULL values)

Set up the example table

Create a table with a custom schema to demonstrate column alterations:

Rename columns

Change column names to better reflect their purpose:

Change data types

Convert column data types for better performance or compatibility:

Make columns nullable

You can alter columns to contain NULL values: Changing a column to nullable affects future writes and merges too: missing values are accepted only when the target column is nullable.

Multiple changes at once

Apply several alterations in a single operation:

Expression-based type changes

For transformations that are not simple casts (for example, converting "$100" to an integer), use a SQL-expression column add, then drop and rename:

Alter embedding types and dimensions

It’s quite common to need to change an embedding column’s schema, in case a new model becomes available with a different embedding dimension.

In Python, the example shows an in-place type update when the cast is compatible.
In TypeScript and Rust, the example shows a dimension change (384 -> 1024), which cannot be cast in-place.

For dimension changes, use this 3-step pattern: add a new column with the target type, drop the old column, then rename the new column to the original name.

FixedSizeList Dimension Changes in TypeScript and RustalterColumns / alter_columns can cast between compatible types, but changing FixedSizeList dimensions (for example 384 -> 1024) is not a compatible cast. For such cases, use addColumns / add_columns (with arrow_cast), then dropColumns / drop_columns, then rename the replacement column.

Changing data types requires rewriting the column data and may be resource-intensive for large tables. Renaming columns or changing nullability is more efficient as it only updates metadata.

Update field metadata

Each column in a LanceDB table can carry a small key/value map of Arrow field metadata — useful for annotating columns with units, provenance, PII flags, embedding model versions, or any other schema-level context your application needs. Use update_field_metadata in Python, updateFieldMetadata in TypeScript/JavaScript, or update_field_metadata in Rust to add, change, or remove these key/value pairs without rewriting the column data. Each call commits a new table version and returns the new version. Each update targets one field by dot-path: top-level columns are addressed by name (for example "embedding"), and nested fields by their full path (for example "address.zip"). By default, the keys you pass are merged into the field’s existing metadata — keys you do not mention are preserved, and passing None (Python) or null (TypeScript) deletes a key. Set replace: true to swap the field’s entire metadata map instead of merging. To overwrite a field’s metadata entirely instead of merging, set replace to true:

You can pass multiple updates in a single call to change metadata on several fields at once — each call commits a single new table version.

Drop columns

You can remove columns using the drop_columns method in Python, dropColumns in TypeScript/JavaScript, or drop_columns in Rust.

Set Up the example table

Create a table with temporary columns that we’ll remove:

Drop single columns

Remove individual columns that are no longer needed:

Drop multiple columns

Remove several columns at once for efficiency:

Dropping columns cannot be undone. Make sure you have backups or are certain before removing columns.

Get started

Model training

Guides

Feature Engineering (Geneva)

Support

Schema and Data Evolution

Schema evolution operations

Add new columns

Set up the example table

Add derived columns

Add columns with default values

Add nullable columns

Declare computed columns

Refresh a computed column

Run the refresh in the background

Alter existing columns

Set up the example table

Rename columns

Change data types

Make columns nullable

Multiple changes at once

Expression-based type changes

Alter embedding types and dimensions

Update field metadata

Drop columns

Set Up the example table

Drop single columns

Drop multiple columns

​Schema evolution operations

​Add new columns

​Set up the example table

​Add derived columns

​Add columns with default values

​Add nullable columns

​Declare computed columns

​Refresh a computed column

​Run the refresh in the background

​Alter existing columns

​Set up the example table

​Rename columns

​Change data types

​Make columns nullable

​Multiple changes at once

​Expression-based type changes

​Alter embedding types and dimensions

​Update field metadata

​Drop columns

​Set Up the example table

​Drop single columns

​Drop multiple columns

Schema evolution operations

Add new columns

Set up the example table

Add derived columns

Add columns with default values

Add nullable columns

Declare computed columns

Refresh a computed column

Run the refresh in the background

Alter existing columns

Set up the example table

Rename columns

Change data types

Make columns nullable

Multiple changes at once

Expression-based type changes

Alter embedding types and dimensions

Update field metadata

Drop columns

Set Up the example table

Drop single columns

Drop multiple columns