Migrations

When you're prototyping, calling define() and define_rules() directly works fine. But once you're running in production with a team, you need more control - versioned schema changes, rollback capability, and a clear record of what's deployed where.

The migration system solves this. It works like Django migrations: you define your models in Python, run a command to detect changes, and it generates a numbered migration file with the exact operations needed. Apply them to any server, revert if something goes wrong, and track what's been deployed.

Installation

Migrations run through the il CLI as il migration <verb> (generate, apply, revert, status). The underlying tool ships with the Python SDK (as the inputlayer-migrate entry point, which il migration delegates to):

pip install inputlayer-client-dev

Verify it's available:

il migration --help

Quick Start

Here's the full workflow from zero to deployed schema.

1. Define Your Models

Start with your Python models in a module (e.g., myapp/models.py):

from inputlayer import Relation, Derived, Vector, From, HnswIndex

class Employee(Relation):
    id: int
    name: str
    department: str
    salary: float

class Document(Relation):
    id: int
    title: str
    embedding: Vector[384]

class Senior(Derived):
    name: str
    rules = [
        From(Employee)
            .where(lambda e: e.salary > 100000)
            .select(name=Employee.name),
    ]

doc_search = HnswIndex(
    name="doc_search",
    relation=Document,
    column="embedding",
    metric="cosine",
)

2. Generate the Initial Migration

il migration generate --models myapp.models

Output:

Created migration: migrations/0001_initial.json
  - Create relation employee
  - Create relation document
  - Create rule senior (1 clause)
  - Create index doc_search

The autodetector inspects your model classes, compares them against the previous migration's state (or an empty state for the first run), and generates the minimal set of operations.

3. Apply the Migration

il migration apply --url ws://localhost:8080/ws --kg production

Output:

Applied 1 migration(s):
  [X] 0001_initial

4. Make Changes and Iterate

Now say you add a new field and a new rule. Update your models, then generate again:

il migration generate --models myapp.models

Output:

Created migration: migrations/0002_auto.json
  - Drop relation employee
  - Create relation employee
  - Replace rule senior

The autodetector noticed that employee has new columns (which requires a drop and recreate since InputLayer doesn't support ALTER) and that the rule definition changed.

Apply it:

il migration apply --url ws://localhost:8080/ws --kg production

5. Check Status

See which migrations are applied on a given server:

il migration status --url ws://localhost:8080/ws --kg production

Output:

  [X] 0001_initial
  [X] 0002_auto

6. Rollback

If something goes wrong, revert to a previous migration:

il migration revert --url ws://localhost:8080/ws --kg production 0001_initial

This runs the backward operations for every migration after 0001_initial, in reverse order. The server state will match what it looked like right after 0001_initial was applied.

CLI Reference

il migration generate

Generate a migration file by diffing your current models against the last migration's state.

il migration generate --models myapp.models [--name descriptive_suffix] [--migrations-dir ./migrations]

The command discovers all Relation subclasses, Derived subclasses, and HnswIndex instances in the given module.

il migration apply

Apply all pending migrations to a server.

il migration apply --url ws://host:port/ws --kg my_graph [options]

il migration revert

Revert all migrations applied after the specified target.

il migration revert --url ws://host:port/ws --kg my_graph 0001_initial [options]

The target migration itself stays applied - everything after it gets reverted in reverse order.

il migration status

Display the applied/pending status of all migrations.

il migration status --url ws://host:port/ws --kg my_graph [options]

Migration File Anatomy

Each generated migration is a self-contained, language-neutral JSON document - no code, no runtime required to read it. Any SDK language can consume the same files. Here's what one looks like:

{
  "format": 1,
  "dependencies": [],
  "operations": [
    {
      "type": "CreateRelation",
      "name": "employee",
      "columns": [["id", "int"], ["name", "string"], ["department", "string"], ["salary", "float"]]
    },
    {
      "type": "CreateRule",
      "name": "senior",
      "clauses": ["+senior(Name) <- employee(_, Name, _, Salary), Salary > 100000"]
    },
    {
      "type": "CreateIndex",
      "name": "doc_search",
      "relation": "document",
      "column": "embedding",
      "metric": "cosine",
      "m": 16,
      "ef_construction": 100,
      "ef_search": 50
    }
  ],
  "state": {
    "relations": {
      "employee": [["id", "int"], ["name", "string"], ["department", "string"], ["salary", "float"]]
    },
    "rules": {
      "senior": ["+senior(Name) <- employee(_, Name, _, Salary), Salary > 100000"]
    },
    "indexes": {
      "doc_search": {"relation": "document", "column": "embedding", "metric": "cosine", "m": 16, "ef_construction": 100, "ef_search": 50}
    }
  }
}

Three key fields:

• dependencies - list of migration names that must be applied first. The autodetector sets this to the previous migration.
• operations - ordered list of forward operations. Each operation also knows how to run backward for reverts.
• state - full snapshot of the model state after this migration. This is what the autodetector diffs against when generating the next migration.

Operations Reference

Every operation knows how to go forward and backward, making all migrations fully reversible.

Custom Operations

For anything not covered by the built-in operations, use RunIQL to run arbitrary IQL commands:

ops.RunIQL(
    forward=["+edge[(1,2), (2,3), (3,4)]"],
    backward=["-edge(X, Y) <- edge(X, Y)"],
)

This is useful for seeding data, running one-off transformations, or anything the autodetector can't generate automatically.

How the Autodetector Works

When you run il migration generate, here's what happens:

1. Discovery - the tool imports your models module and finds all Relation subclasses, Derived subclasses, and HnswIndex instances
2. Current state - it builds a ModelState snapshot from your Python classes, converting types and compiling rules to IQL
3. Previous state - it loads the state dict from your latest migration file (or starts from empty)
4. Diff - it compares the two states and generates the minimal set of operations
5. Ordering - operations are ordered to respect dependencies: new relations first (so rules can reference them), then rules, then dropped relations, then indexes

The ordering matters because InputLayer needs relations to exist before rules can reference them. The autodetector handles this automatically.

Applied Migration Tracking

The migration system tracks which migrations have been applied using an internal relation called __inputlayer_migrations__ in your knowledge graph. Each row stores the migration name and the timestamp it was applied.

This means:

• Different knowledge graphs can be at different migration versions
• You can see exactly what's deployed on any server with il migration status
• The system never re-applies an already-applied migration

Development Workflow

Local Development

# Start your local InputLayer server
inputlayer-server

# Generate and apply migrations as you develop
il migration generate --models myapp.models
il migration apply --url ws://localhost:8080/ws --kg dev

Team Workflow

1. Commit migration files alongside the model changes that generated them
2. Each developer runs il migration apply against their local server to sync up
3. During code review, verify that migration operations match the model diff
4. Never edit or delete a migration that's already been applied - always create new ones

Naming Migrations

By default, migrations are named 0001_initial, 0002_auto, etc. Use --name for clarity:

il migration generate --models myapp.models --name add_user_roles
# -> creates migrations/0003_add_user_roles.json

CI/CD Integration

GitHub Actions

- name: Apply migrations
  run: |
    il migration apply \
      --url ${{ secrets.INPUTLAYER_WS_URL }} \
      --kg production \
      --api-key ${{ secrets.INPUTLAYER_API_KEY }}

Staging Validation

Apply to staging first, verify, then promote to production:

jobs:
  migrate-staging:
    steps:
      - uses: actions/checkout@v4
      - run: pip install inputlayer-client-dev
      - run: |
          il migration apply \
            --url ${{ secrets.STAGING_WS_URL }} \
            --kg staging \
            --api-key ${{ secrets.STAGING_API_KEY }}

  migrate-production:
    needs: migrate-staging
    steps:
      - uses: actions/checkout@v4
      - run: pip install inputlayer-client-dev
      - run: |
          il migration apply \
            --url ${{ secrets.PROD_WS_URL }} \
            --kg production \
            --api-key ${{ secrets.PROD_API_KEY }}

Programmatic Usage

You can also use the migration internals directly from Python. This is useful for testing, scripting, or building custom tooling:

from inputlayer.migrations.autodetector import detect_changes
from inputlayer.migrations.state import ModelState
from inputlayer.migrations.writer import generate_migration

# Build state from your current models
state = ModelState.from_models(
    relations=[Employee, Department],
    derived=[Senior],
    indexes=[doc_search],
)

# Diff against empty state (for initial migration)
ops = detect_changes(ModelState(), state)

# Generate the migration file content
filename, content = generate_migration(
    number=1,
    operations=ops,
    state=state.to_dict(),
    dependencies=[],
    name_suffix="initial",
)
print(f"Generated: {filename}")

Best Practices

One change per migration. Don't mix unrelated schema changes in a single migration. If you're adding a new relation and modifying a rule, that's fine - but adding users and changing the indexing strategy should probably be separate.

Never edit applied migrations. Once a migration has been applied to any server, treat it as immutable. If you need to fix something, create a new migration. Editing an applied migration will cause state drift between your files and the server.

Test on staging first. Always apply migrations to a staging environment before production. Schema changes like dropping and recreating relations are destructive - the migration system makes them reversible, but it's still better to catch issues early.

Commit migrations with your code. Migration files should live in version control right next to the model changes that generated them. This makes code review straightforward - reviewers can verify that the generated operations match the model diff.

Use descriptive names. 0003_add_user_roles.json is much easier to understand at a glance than 0003_auto.json. Use --name when generating migrations.