hardwood-hq/hardwood/hardwood-cli

Inspecting Parquet files with hardwood

hardwood is a command-line tool for looking _inside_ Apache Parquet files without writing any code. It is purpose-built for the moments when you are debugging Parquet read/write code and start to suspect the file itself — a type that does not match, statistics that are missing, an encoding that defeats your optimizer, a dictionary that is not what you expect.

This skill teaches you which subcommand answers which question, how to read the output, and how to chain a few commands into a diagnosis.

Before you start

1. Confirm the binary is available. The CLI ships as a GraalVM native binary

with instant startup. Check for it before assuming a flag will work:

``shell command -v hardwood && hardwood --version ``

1. Every command takes `-f`/`--file`. This is the only required flag for the

read-only commands. It accepts a local path or an s3:// URI (no extra config beyond the standard AWS environment variables AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc.):

``shell hardwood schema -f s3://my-bucket/path/to/data.parquet ``

1. `-h`/`--help` on any command lists its real options — run it when a flag

below does not match what you see, e.g. hardwood inspect pages --help.

1. `dive` needs a TTY; you usually do not. hardwood dive is an interactive

terminal UI. It exits with an error when stdin/stdout is not a real terminal, which is the normal case for an agent. Use the headless subcommands below for your own investigation, and _suggest_ dive to the human user when they want to poke around a file interactively.

Command quick reference

Key output signals to read for

A lot of the diagnostic value is in small labels the commands print. Learn to spot these:

• `info` → `Created By:` reveals the writer library and version

(parquet-cpp-arrow version 24.0.0, parquet-mr, spark, …). Writers differ in whether they emit statistics, page indexes, and which encodings they use — this line is often the first clue in a "pushdown doesn't work" mystery.

• `info` → `Key/Value Metadata (N):` lists file-level metadata other engines

embed alongside the data — ARROW:schema (Arrow), org.apache.spark.sql.parquet.row.metadata (Spark), pandas (pandas/PyArrow). This is the first thing to check when the same file reads differently in two engines: values are truncated by default, so use --kv-key <name> to print one entry in full.

• `inspect pages` → the `(stats: …)` suffix after the column name tells you

whether page-level statistics exist and where they come from:

• (stats: ColumnIndex) — a Page Index (ColumnIndex + OffsetIndex sidecars)

is present. This is what engines need for page skipping.

• (stats: inline) — only legacy statistics embedded in each data-page header.

No page index, so no page-level skipping.

• (no page-level stats) — nothing at page level. Nothing can be skipped at

page granularity.

• (stats: mixed) — different row groups disagree; some skipping only.
• `inspect pages` → the `Min` / `Max` / `Nulls` columns show the actual

bounds per page. If the predicate value falls _inside_ every page's min/max, skipping nothing is correct behavior, not a bug.

• `inspect columns` → `Share` is the column's percentage of the file's

compressed bytes: the row at the top (Rank 1) is the dominant scan cost. `Compression` is compressed ÷ uncompressed for the named Codec; 100.0% means the codec achieved nothing. A # Pages count (vs —) confirms a page index exists for that column.

• `inspect columns` → `Unencoded` is what the values occupy with no

encoding, which predicts read-side memory — a dictionary-encoded column can be small on disk and large once materialised. The file records it for BYTE_ARRAY; for a fixed-width type it is the present-value count times the width. It is — only where the present-value count is unknown.

• `inspect columns` → `Encoding` names what the data pages use, read from

encoding_stats: DICT, PLAIN, DELTA, and so on. The ranked table shows the union across row groups; --column PATH breaks it out per row group, and dive's column chunk detail shows it for a single chunk.

• `inspect columns` → the percentage after `DICT` is the dictionary's

cardinality: entries ÷ values it could hold an entry for. This is the number to scan for. DICT <1% is a dictionary doing its job. `DICT 100%` means every value is distinct, so the column is stored twice — once verbatim in the dictionary page, once as a stream of distinct indices — and no codec undoes the second copy, because distinct indices are high-entropy by construction. A high-cardinality column stays dictionary-encoded silently: writers dictionary-encode by default and fall back only when the dictionary outgrows its size limit, so one that stays under the limit is never reconsidered. What to use instead is a question the metadata cannot answer — re-encode and compare on disk. DELTA_BYTE_ARRAY for sorted or prefix-sharing strings and DELTA_BINARY_PACKED for near-sequential integers are the usual answers. PLAIN+DICT is the one worth acting on — the writer started with a dictionary and abandoned it mid-chunk when it outgrew its size limit, leaving a dictionary page that earns nothing for the pages after the fallback.

• `inspect columns --column PATH` → `tags null` vs `tags empty` distinguishes

a record whose field was absent from one whose list was empty. Nothing else in the metadata separates them: Nulls and the page index null_counts lump both together.

• `inspect dictionary` → `no dictionary (column is not dictionary-encoded)`

versus a table of entries: tells you whether a column is dict-encoded and what its distinct values are.

• `footer` → magic bytes PAR1 at both ends = a plain Parquet file. An

encrypted footer (Parquet Modular Encryption) does not print a magic value: footer instead exits non-zero and writes Encrypted Parquet files are not supported (Parquet Modular Encryption). to stderr — that error, not a PARE string in the output, is the signal the file needs decryption keys.

Playbook: "the source Parquet file looks wrong"

The user is debugging read/write code and has narrowed it down to a suspect file. Establish ground truth in three commands:

hardwood info    -f FILE        # how big, how many row groups, who wrote it
hardwood schema  -f FILE        # exact fields, physical + logical types
hardwood print   -n 10 -f FILE  # the first 10 actual rows

What to compare against the code:

• Physical vs. logical type mismatches. A field the code treats as a

String may physically be BYTE_ARRAY with a STRING annotation, or just bare BYTE_ARRAY. schema prints both, e.g.: `` message schema { required int64 id; optional binary category (STRING); optional int32 ts (TIMESTAMP_MILLIS); } ``

• Repetition / nullability. required vs optional changes null handling.
• `Created By` — if it names a library version known to emit odd encodings,

that is a lead.

To export a sample for a downstream tool, project just the columns you care about (nested fields use dot notation) and pick head or tail:

hardwood print   -n 5  -c "id,account.organization.name" -f FILE  # ASCII
hardwood convert -n 50 --format csv -c "id,value" -f FILE         # CSV to stdout
hardwood convert -n 10 --format json -o sample.json -f FILE       # JSON to a file

Row-limit semantics for -n/--rows (used by both print and convert): positive = first _N_ rows (head), negative = last _N_ rows (tail), ALL = every row, and 0 is rejected.

--skip <n> starts the read at row _n_ and --row-group <i> reads that row group only, both counted from 0, on print and convert. They cannot be combined with each other, nor with a negative -n (which counts from the end of the file). A positive -n stays within the selected row group. Negative or out-of-range positions and empty row groups are rejected. print --row-index numbers rows by their position in the file, including under a negative -n, so a row number seen in hardwood dive reproduces here.

Playbook: "why isn't predicate pushdown / file skipping working?"

A query is scanning far more data than expected. The engine skips at two granularities — row group (using each column chunk's footer Statistics) and page (using the Page Index). Walk down the file to see what skipping is even possible:

hardwood info              -f FILE                              # 1. row-group count + writer
hardwood inspect rowgroups -f FILE                              # 2. row-group sizes / codecs
hardwood inspect columns   -f FILE                              # 3. which columns dominate
hardwood inspect pages     -f FILE -c <predicateColumn>         # 4. stats source + min/max bounds

Interpretation checklist: