Indexing Guide

InputLayer provides HNSW (Hierarchical Navigable Small World) indexes for fast approximate nearest neighbor search on vector data.

Why Use Indexes?

Without an index, a similarity query computes the distance to every row, so its cost grows linearly with the relation.

Measured on 10K random 384-dim vectors, cosine, default settings (cargo bench --bench vector_index_benchmarks, Apple M-series):

Trade-off: Indexes use memory and, above 1024 entries, return approximate results (recall@10 is about 0.99 on 3K random 32-dim vectors).

Creating an Index

Basic Syntax

.index create <name> on <relation>(<column>) [options]

Simple Example

// Create a documents table with embeddings
+documents(id: int, title: string, embedding: vector)

// Insert some documents
+documents(1, "Introduction to ML", [0.1, 0.2, 0.3, 0.4])
+documents(2, "Vector Databases", [0.15, 0.25, 0.28, 0.42])
+documents(3, "Graph Theory", [0.8, 0.1, 0.05, 0.05])

.index create doc_emb_idx on documents(embedding)

Creating the index builds it from the rows already in the relation. The relation needs a schema, the indexed column must be a vector, and the first column must be an integer id: hnsw_nearest returns that id.

With Options

.index create doc_emb_idx on documents(embedding) metric cosine m 16 ef_search 50

Index Options

Distance Metrics

Default: cosine

.index create my_idx on vectors(embedding) metric l2
.index create my_idx on vectors(embedding) metric cosine
.index create my_idx on vectors(embedding) metric dot

HNSW Parameters

.index create my_idx on vectors(embedding) m 32 ef_search 100

Parameter Tuning

For higher recall (more accurate results):

.index create my_idx on vectors(embedding) m 32 ef_construction 400 ef_search 100

For faster search (lower recall):

.index create my_idx on vectors(embedding) m 8 ef_search 20

For large datasets (millions of vectors):

.index create my_idx on vectors(embedding) m 48 ef_construction 500 ef_search 200

Managing Indexes

List All Indexes

.index

or

.index list

Output:

Index 'doc_emb_idx' on documents.embedding (type: hnsw, metric: cosine, vectors: 3)

View Index Statistics

.index stats doc_emb_idx

Output:

Index 'doc_emb_idx': relation=documents, column=embedding, type=hnsw, metric=cosine, vectors=3, tombstones=0, dimension=4

vectors counts live rows; tombstones counts deleted or replaced entries still in the graph (see Tombstones and Compaction).

Rebuild an Index

Rebuilds the index from the relation's current rows, dropping tombstones:

.index rebuild doc_emb_idx

Queries already running keep the index they started with.

Drop an Index

.index drop doc_emb_idx

Using Indexes in Queries

Query an index with the hnsw_nearest predicate. Distance functions such as cosine() never use an index; combined with top_k they scan every row (exact, but linear).

Explicit HNSW Search with hnsw_nearest

The hnsw_nearest builtin queries an HNSW index directly, returning the k nearest neighbors with their distances.

Syntax:

hnsw_nearest("index_name", QueryVec, K, IdVar, DistVar)
hnsw_nearest("index_name", QueryVec, K, IdVar, DistVar, EfSearch)

Parameters:

Example - basic nearest neighbor search:

?hnsw_nearest("doc_emb_idx", [0.1, 0.2, 0.3, 0.4], 10, Id, Distance)

Example - using a query vector from a relation:

+query_embedding[([0.1, 0.2, 0.3, 0.4])]
?query_embedding(QV), hnsw_nearest("doc_emb_idx", QV, 5, Id, Dist)

One search runs per distinct vector bound to QV, so ?documents(7, _, QV), hnsw_nearest("doc_emb_idx", QV, 5, Id, Dist) finds the neighbours of document 7.

Example - joining results with the base relation:

?hnsw_nearest("doc_emb_idx", [0.1, 0.2, 0.3, 0.4], 10, Id, Dist),
  documents(Id, Title, _)

Example - with custom ef_search for higher recall:

?hnsw_nearest("doc_emb_idx", [0.1, 0.2, 0.3, 0.4], 10, Id, Dist, 200)

Notes:

• The index must exist; .index list shows what is available. Unknown indexes, query vectors of the wrong dimension, and unbound query variables are reported as errors.
• IdVar is the integer id (first column) of the matching row, with the same integer type as the column, so it joins with the relation.
• DistVar is the distance in the index metric: cosine distance (1 - cos), euclidean, manhattan, or -cos for dot (vectors are normalized).
• EfSearch overrides the index's ef_search for this query only.
• A variable query vector is not supported inside recursive rules.

Index Lifecycle

Build

.index create builds the index from the relation's existing rows (parallel insert) and registers it. Rows the index cannot hold (non-integer id, missing or zero-norm vector for cosine/dot, wrong dimension) make the create fail with the offending row id.

Maintenance

Indexes stay current; there is no invalid state:

• Insert: new rows are added to the HNSW graph in the same write. A row with an existing id replaces that id's vector.
• Insert validation: rows an index cannot hold are rejected before they are stored.
• Delete: the row is tombstoned and never returned again.
• Drop relation: indexes on it are dropped.

Consistency

Every query runs against a snapshot. The snapshot carries the index state from the same moment as its data, so a query never sees a row in the index that is missing from the relation, or the reverse, even while writes continue.

Restart

Index definitions are saved in the knowledge graph's indexes.json when created or dropped. On restart each index is rebuilt from the recovered relation data, so it always matches what is on disk.

Small Indexes

Indexes with up to 1024 entries are searched exhaustively, so results on small data are exact.

Index Architecture

HNSW Structure

Layer 3:  *-------------*
          |             |
Layer 2:  *---*-----*---*---*
          |   |     |   |   |
Layer 1:  *-*-*-*-*-*-*-*-*-*-*
          | | | | | | | | | | |
Layer 0:  ********************* (all nodes)

Each layer has fewer nodes. Search starts at top layer and descends:

1. Find nearest nodes in current layer
2. Use those as entry points for next layer
3. Repeat until layer 0
4. Return k nearest neighbors

Memory Usage

HNSW indexes use approximately:

memory approx n * (d * 4 + m * 8) bytes

Where:

• n = number of vectors
• d = vector dimension
• m = max connections parameter

Example: 1M vectors * 768 dimensions * m=16:

1M * (768 * 4 + 16 * 8) = ~3.2 GB

Tombstones and Compaction

Deleting or replacing a row leaves a tombstone: the old entry stays in the graph as a navigation node but is filtered from results.

.index stats my_idx
  ... vectors=10000, tombstones=500, ...

Automatic Compaction

When tombstones exceed 30% of the graph (and number at least 64), the delete that crosses the threshold rebuilds the index from the relation.

Manual Compaction

.index rebuild my_idx

Best Practices

1. Choose the Right Metric

2. Tune Parameters for Your Use Case

Discovery/Exploration (higher recall matters):

.index create my_idx on docs(emb) m 32 ef_search 100

Production/Speed (latency matters):

.index create my_idx on docs(emb) m 16 ef_search 30

3. Monitor Index Health

.index stats my_idx

Compaction runs automatically past 30% tombstones; rebuild sooner if search quality drops after heavy churn.

4. Bulk Load, Then Create

Creating an index over existing rows uses a parallel build, which is faster than inserting the same rows one batch at a time into an existing index:

+docs[(1, "...", [0.1, ...]),
      (2, "...", [0.2, ...]),
      ...]

.index create my_idx on docs(emb)

5. Use Appropriate Vector Dimensions

Common dimensions:

• OpenAI text-embedding-3-small: 1536
• Cohere embed-english-v3: 1024
• all-MiniLM-L6-v2: 384
• CLIP ViT-B/32: 512

Higher dimensions = more memory, slower search.

Troubleshooting

Search Returns No Results

Possible causes:

1. No vectors in the relation
2. The query variable is bound to no rows (e.g. query_vec is empty)

Debug:

.index stats my_idx
?documents(Id, _, V)  // Check if data exists

Poor Search Quality

Causes:

1. Wrong distance metric for embedding type
2. ef_search too low
3. Many tombstones

Solutions:

// Check metric matches embedding type
.index stats my_idx

// Increase ef_search for one query
?hnsw_nearest("my_idx", [0.1, 0.2], 10, Id, Dist, 200)

// Rebuild to remove tombstones
.index rebuild my_idx

High Memory Usage

Solutions:

1. Reduce m parameter (trades recall for memory)
2. Use vector quantization (see Vectors Guide)
3. Consider approximate embeddings with lower dimensions

Next Steps

• Vector Search Tutorial - Distance functions and semantic search
• Configuration Guide - Index persistence settings