flex-analytics/flexviz/flexviz-explore

Explore large data with FlexViz

FlexViz serves interactive cross-filter dashboards from lazy Polars queries. The data stays server-side; the browser gets bounded aggregates and small samples of real values. Parquet larger than RAM streams, so peak memory stays flat as rows grow (box plots excepted).

Every dashboard view is a URL that carries the complete spec: 1 to 4 KB, and a browser tool echoes the page URL in every snapshot. So you never hold one. Record each URL under a number with flexviz history, open it at /h/N, and call it fv:N.

Privacy: rows stay in the lazy engine unless you collect them. The schema, your samples, and the ranges the human selects do enter your context. A share URL embeds column names and filters, so add .flexviz/ to .gitignore.

The loop

1. Inspect the schema

flexviz schema data.parquet

Not on PATH? Use the project venv (.venv/bin/flexviz or uv run flexviz) and keep that form below. If the package is absent, ask the human before installing.

The output names the file, the source, and every column with its dtype. Pick an x column (usually time) and the columns worth plotting. A .head(5).collect() peek is fine. Never collect the full frame.

2. Serve the file (background process)

flexviz serve data.parquet --cache > serve.log 2>&1 &

Run the server, and every command below, from the project directory: the history file (.flexviz/history.jsonl) and /h/N both resolve against a working directory.

• Without --port the server takes a free port and prints its URL on the first

line of the log. Read the port there and use it in every step below.

• Each file becomes a source named by its stem (data.parquet -> "data").
• --cache enables cross-filter cubes and live brushing. Use it whenever

the file will not change while serving.

• Wait for readiness: poll until the answer names YOUR source, for example

until curl -s http://127.0.0.1:<port>/sources | grep -q '"data"'; do sleep 1; done. A bare "it answered" check is not enough: another server can answer instead.

• Keep the process running while the human explores, tell them the port and

PID, and stop it at the end. An orphan server otherwise holds the port.

• To add files later, restart with the FULL file list (old + new).
• Serve on loopback. Another interface exposes unauthenticated endpoints, so

only do it if the human explicitly accepts that.

flexviz serve scans the file as stored: it cannot cast a column, and it has no column the file does not have. A timestamp held as String gives a string x axis with no warning, and a figure on a derived column (hour of day, weekday, a ratio) opens fine and then fails every update with a 500 and ColumnNotFoundError in the serve log, so its panels stay blank. In both cases register the source yourself and build step 3 on that same LazyFrame:

import polars as pl
from flexviz import register_source, run_server
lf = pl.scan_parquet("data.parquet").with_columns(pl.col("timestamp").str.to_datetime())
register_source("data", lf, cache=True)   # call this before run_server
run_server("127.0.0.1", 8077)

3. Build the dashboard and record it

Build and record in ONE script, and print only the number:

import polars as pl
from flexviz import Dashboard, history
dash = Dashboard(pl.scan_parquet("data.parquet"), cache=True)
dash.add_figure().add_line(x="timestamp", y="value", group_by="sensor_id")
dash.add_figure().add_histogram(x="value", bins=50)
url = dash.share_url(server_url="http://127.0.0.1:<port>", source_name="data")
print(history.add(url, note="data.parquet: line + histogram, initial view", actor="agent"))

• Start small: at most 4 figures in the first dashboard. Pick the few columns

that matter and add more only when the human asks. A wall of panels is harder to read than a focused view.

• This script only builds a spec: cheap, lazy, and it exits at once. The serve

process answers every interaction.

• source_name must match the served stem, and cache=True must match

--cache.

• Two or more figures that show the same time column? Add

dash.link_axes(on="timestamp"), so a zoom on one moves them all. With only one such figure there is nothing to link, and the call raises.

• No categorical column to group_by? Split metrics across figures instead,

and use add_histogram2d to relate numeric columns. Add at most one add_corr_heatmap for the whole dashboard, and only when many numeric columns make one worthwhile. It is an overview, not a per-figure default.

4. Open the entry

Open http://127.0.0.1:<port>/h/N for the number step 3 printed, and give the human the same address. The /h/N address is what they open. In everything you write, the entry is fv:N. Tell them: drag on one chart to cross-filter the others, zoom to re-aggregate at higher detail, double-click to reset.

A page that opens is not a page that drew: check the serve log for a traceback after the first open of every new spec.

5. Read the state back

All interaction state lives in ONE browser tab, so which tab the human uses decides what you can read. Assume separate browsers until they confirm otherwise.

Shared tab. Your browser tool drives a window on this machine that the human can also use: a headed Playwright session (never --headless), or an extension attached to the human's own browser. One tab holds the state, so you read it:

window.flexvizState({compact: true})   // via your browser evaluate tool

It returns {version, state, client_state, revision}. revision increases when the serialized state differs from the previous read; it does not say who changed it. Example:

<!-- compact-state -->

{"version": "0.7", "revision": 3, "state": {
  "viewport": {"fig1/x": {"min": 100, "max": 300}},
  "selections": [{"source_figure_uid": "fig1",
    "predicates": [{"clauses": [{"column": "value", "range": [10.0, 25.0]}]}]}],
  "group_domains": {}, "cross_filter_mode": "update"},
 "client_state": {"hover_mode": "off", "live_brush": "auto",
  "axis_locks": {}, "axis_lock_ranges": {}, "axis_links": []}}

• version/revision: the spec version, and a counter that increases when the serialized state differs from the previous read.
• state: server-visible — viewport (fig_uid/axis), selections, group_domains, cross_filter_mode.
• client_state: client-only display state; the engine ignores it. axis_links

lists groups of viewport keys that always hold one range.

Separate browsers. A headless session, or a human at another machine. Your tab and their tab hold independent state, so polling yours tells you nothing about their zooms and selections. They hand the state over: ask them to click Share in the toolbar (it copies a URL that captures their current view) and to run this in the project directory:

flexviz history add "<paste it here>" --actor human --note "what I was looking at"

They tell you only the number it prints; you read it with flexviz history show N. If they will not run a command, let them paste the URL and run it yourself. That is the one place a URL enters your context, and it enters once. The address bar does NOT track interactions; only Share captures them.

6. Record the human's state before you act

With separate browsers the human's history add already made this entry, so skip the step. In the shared tab the page cannot write the history file, so you record what you read.

Your browser tool returns the flexvizState readback as JSON text, not a Python object, and some tools encode it twice. Save it, then load it tolerantly:

import json
from flexviz import history
raw = json.load(open("state.json"))          # whatever your browser tool wrote
compact = json.loads(raw) if isinstance(raw, str) else raw
print(history.record_state(3, compact["state"], compact["client_state"],
                           note="human brushed 09:00-11:00 on sensor 12"))

record_state appends a NEW numbered entry: it reuses entry 3's dashboard, swaps in the state you read, and returns the new number (host and port come from entry 3, so no URL reaches your context). Do it before you change anything: the new entry is what the report cites later.

7. Change the dashboard

A state change (viewport, selections, hover mode, axis locks) goes through the write half, in the tab you control. Smallest payloads. F below is a figure UID, the one you read as a viewport key in step 5 ("<uid>/x"), never the panel's DOM id or its index. A datetime axis stores the date string Plotly reports, e.g. "2024-01-01 00:00:00", not ISO-8601:

await window.flexvizApply({state: {viewport: {"F/x": {min: a, max: b}}}})
await window.flexvizApply({state: {selections: []}})   // clear all selections

To author a cross-filter, source_figure_uid must be a figure that brushes the predicate's column: filter value from the figure whose axis is value, not from a time-series line. A mismatch is coerced silently to that figure's own axis, not rejected:

await window.flexvizApply({state: {selections: [
  {source_figure_uid: "<uid of the value figure>",
   predicates: [{clauses: [{column: "value", range: [2.5, 6.0]}]}]}
]}})

Only state, client_state and layout apply; every other key is ignored with a warning. state and client_state merge one level deep, so a partial state keeps its sibling keys (viewport, group_domains, cross_filter_mode). It re-renders and resolves with the compact state. It rejects with the server's reason when the re-request fails, and the page keeps its previous state. When client_state.axis_links lists a viewport key, set every key of that group to the same range; a patch that sets only one is rejected. Locks follow the same rule: lock every key of a group or none, with one range in axis_lock_ranges. With separate browsers the human does not see it, so record the state with record_state(..., actor="agent") and give them the new /h/N instead; the default actor is "human", because step 6 is its first use.

A structure change (add or remove a figure, change traces) needs a new spec, because panels are built server-side. Rebuild in Python as in step 3, history.add(...), then open the new /h/N. Note your own entries too, so the history reads as a sequence of who did what.

8. Tell them what you found

End the first pass in chat: the findings in a few lines, each one naming the entry that shows it as fv:3, never as "panel 1". Filter the LazyFrame to the brushed range to put a number next to a figure. End that same message with the offer, after the findings, never before: "Say report and I write findings.md with live views, tell me what to look at next, and say whether you use the window I opened or your own browser."

Write that file only when they ask. A line that holds nothing but fv:5 becomes the live dashboard of entry 5, zoomable for as long as the server runs. --md writes a second copy with bare links, for GitHub or chat:

flexviz report findings.md -o findings.html --md findings.expanded.md

API cheat-sheet

Defaults shown. The # notes mark the arguments whose meaning the signature does not give: those are where a call runs cleanly and charts the wrong thing. For the rest, read flexviz/figure.py or https://docs.flexviz.tech.