shadylukin/neumann/neumann-troubleshoot

Neumann Troubleshooting Guide

Parse Errors

Parse errors are the most common issue. The parser is strict and error messages point to the exact token that failed.

Reserved keyword collision

Neumann has 150+ reserved keywords. If a column name, node label, or edge type collides with a keyword, you get a parse error.

Symptoms: expected identifier, found keyword 'NODE' or similar.

Fix: Rename the identifier. Common collisions: node, edge, path, type, status, key, value, name, index, limit, order, group, select, table, set, delete, create, drop, insert, update.

NODE CREATE node { ... }          -- WRONG: node is a keyword
NODE CREATE nd { ... }            -- correct: renamed to nd
 
CREATE TABLE status (...)         -- WRONG: status is a keyword
CREATE TABLE user_status (...)    -- correct: prefixed

Unquoted keys with special characters

The parser splits tokens on :, -, and other punctuation.

EMBED STORE 'doc:1' [0.1, 0.2]   -- correct
EMBED STORE doc:1 [0.1, 0.2]     -- WRONG: 3 tokens (doc, :, 1)
 
NODE GET 'node-abc'               -- correct
NODE GET node-abc                 -- WRONG: parsed as node minus abc

Wrong direction keywords

The parser only accepts OUTGOING, INCOMING, BOTH.

NEIGHBORS 'n1' OUTGOING           -- correct
NEIGHBORS 'n1' OUT                -- WRONG: OUT is not a direction keyword
NEIGHBORS 'n1' IN                 -- WRONG: IN means SQL IN operator

Wrong SIMILAR syntax

SIMILAR [0.1, 0.2] LIMIT 5       -- correct
SIMILAR TO [0.1, 0.2] LIMIT 5    -- WRONG: no TO keyword
SIMILAR [0.1, 0.2] BY COSINE     -- WRONG: use METRIC, not BY

Missing property block braces

NODE CREATE person { name: 'Alice' }    -- correct
NODE CREATE person name: 'Alice'        -- WRONG: missing braces

Wrong property separator

NODE CREATE person { name: 'Alice', age: 30 }    -- correct (colon)
NODE CREATE person { name = 'Alice' }             -- WRONG: use colon, not equals

Edge creation syntax

EDGE CREATE 'a' -> 'b' : knows         -- correct
EDGE CREATE 'a' 'b' knows              -- WRONG: missing -> and :
EDGE CREATE (a)-[:knows]->(b)          -- WRONG: that is Cypher, use EDGE syntax

Connection Issues

Port assignments

Port conflict

Symptom: Address already in use on startup.

Fix: Check what is using the port:

lsof -i :9200
lsof -i :9300

Kill the conflicting process or change the Neumann bind address via NEUMANN_BIND_ADDR environment variable.

Health check

grpcurl -plaintext localhost:9200 grpc.health.v1.Health/Check

Expected response: { "status": "SERVING" }.

TLS issues

If TLS is enabled, use the correct certificate:

grpcurl -cacert ca.pem -cert client.pem -key client-key.pem localhost:9200 grpc.health.v1.Health/Check

Authentication

If API key authentication is enabled, include the key in metadata:

grpcurl -plaintext -H 'authorization: Bearer <api-key>' localhost:9200 neumann.QueryService/Query

Performance Issues

Slow queries

Check for missing indexes:

DESCRIBE users                     -- shows column types but not indexes
SHOW TABLES                        -- verify table exists
SHOW VECTOR INDEX                  -- check if HNSW index is built
GRAPH INDEX SHOW NODE              -- check graph property indexes
GRAPH INDEX SHOW EDGE              -- check edge property indexes

Common fixes:

• Relational: CREATE INDEX idx_name ON table (column)
• Vector: EMBED BUILD INDEX (must be run after inserts)
• Graph: GRAPH INDEX CREATE NODE PROPERTY prop_name

Slow SIMILAR queries

If SIMILAR is slow, the HNSW index likely was not built or is stale:

SHOW VECTOR INDEX
EMBED BUILD INDEX

High memory usage

Check embedding count -- each embedding consumes memory proportional to its dimensionality:

COUNT EMBEDDINGS

Check node/edge counts:

GRAPH AGGREGATE COUNT NODES
GRAPH AGGREGATE COUNT EDGES

Large result sets

Always use LIMIT to bound result sizes:

SIMILAR [0.1, 0.2] LIMIT 10       -- bounded
NODE LIST person LIMIT 100         -- bounded
SELECT * FROM users LIMIT 1000     -- bounded

Cluster Issues

Split brain

Symptom: Two nodes both claim to be leader, or queries return different results depending on which node is queried.

Diagnose:

CLUSTER STATUS
CLUSTER LEADER
CLUSTER NODES

Compare commit indices across nodes. The node with the higher commit index has more up-to-date data. The other node likely lost connectivity and elected itself.

Fix: Restart the minority-side node. It will rejoin the cluster and replicate from the true leader.

Leader election timeout

Symptom: CLUSTER LEADER returns no leader or times out.

Causes:

• Network partition between nodes.
• All nodes started simultaneously (election collision). Wait for randomized

timeout to resolve.

• Fewer than quorum nodes are reachable.

Quorum requirements:

Replication lag

Symptom: Writes on the leader are not immediately visible on followers.

This is normal in an eventually-consistent read path. For strong consistency, read from the leader. For followers, wait briefly or verify via:

CLUSTER STATUS

Check that commit_index on followers is close to the leader's.

Node won't join cluster

Verify:

1. NEUMANN_CLUSTER_PEERS includes all peer addresses in node_id=IP:port format.
2. NEUMANN_CLUSTER_NODE_ID is unique across all nodes.
3. Port 9300 is reachable between all nodes (Raft transport).
4. All nodes are running the same Neumann version.

Data Issues

Missing data

Check you are querying the right engine:

• Relational data: SELECT * FROM table WHERE ...
• Graph data: NODE GET 'id' or NODE LIST label
• Vector data: EMBED GET 'key'
• Unified data: ENTITY GET 'key'

A common mistake is inserting into one engine and querying another.

Data integrity verification

CHAIN VERIFY
BLOB VERIFY 'blob-id'

Recovery from checkpoints

CHECKPOINTS                        -- list available checkpoints
CHECKPOINTS LIMIT 5                -- show last 5
ROLLBACK TO 'checkpoint-id'        -- restore to a checkpoint

WAL corruption

If the write-ahead log is corrupted, start Neumann with a fresh WAL directory. Data persisted in checkpoints is still recoverable via ROLLBACK TO.

Debug Logging

Set the RUST_LOG environment variable to control log verbosity.

Levels: error, warn, info, debug, trace.

# All modules at info level
RUST_LOG=info neumann
 
# Specific module at debug level
RUST_LOG=query_router=debug neumann
 
# Multiple modules at different levels
RUST_LOG=tensor_chain=debug,query_router=trace,tensor_store=info neumann
 
# Everything at trace (very verbose)
RUST_LOG=trace neumann

Useful module targets for debugging:

• query_router -- query parsing and routing
• tensor_chain -- Raft consensus and chain operations
• tensor_store -- storage operations
• vector_engine -- HNSW index and similarity search
• graph_engine -- graph traversal and algorithms
• neumann_server -- gRPC request handling

Getting Help

When filing an issue, include:

1. Neumann version (from startup banner or --version).
2. Operating system and architecture.
3. The exact query that failed and the full error message.
4. Relevant log output (set RUST_LOG=debug to capture details).
5. Cluster configuration if applicable (node count, network topology).
6. Steps to reproduce the issue from a fresh state.