apache/datafusion-python/audit-skill-md

Audit skills/datafusion_python/SKILL.md

You are auditing the user-facing skill at `skills/datafusion_python/SKILL.md` against the current state of the Python API. The skill is the source of truth for how AI coding assistants are taught to write datafusion-python code, so it must match what the project actually ships. This skill identifies gaps caused by upstream syncs, refactors, or renames, and (if asked) applies the edits directly to SKILL.md.

The skill is most usefully run after the check-upstream step of an upstream sync (see dev/release/upstream-sync.md) — once any new APIs are exposed, this skill makes sure they get documented.

What the skill covers

The user-facing SKILL.md documents these public surfaces. This list is not exhaustive — if a new top-level area is added (e.g., a new Catalog API exposed at the package root), include it.

Scope argument

The user may specify a scope via $ARGUMENTS to limit the audit. If no scope is given or all is specified, audit every area.

Inputs to read

Before producing the report:

1. skills/datafusion_python/SKILL.md — the document being audited.
2. The relevant Python module(s) for the chosen scope. Public surface is the

__all__ list (where defined) plus class and def symbols not prefixed with _.

1. Cargo.toml (root) for the current datafusion-python version — read

the version field under [workspace.package] (format NN.0.0). The major version always matches the upstream datafusion crate, so a single datafusion-python version expresses both. python/datafusion/__init__.py's __version__ is the same value exposed at runtime.

1. Recent commits touching the relevant module(s) for context on what

changed since the last sync: ``bash git log --oneline -- python/datafusion/dataframe.py | head -20 ``

What to look for

Walk through each scoped area and flag four kinds of issues.

1. New APIs not mentioned

For each public symbol in the module's __all__ (or each public class method), check whether it appears anywhere in SKILL.md. A symbol is "covered" if it shows up in:

• A code block (the strongest signal — it's demonstrated).
• The "Available Functions (Categorized)" list.
• The SQL-to-DataFrame Reference table.

Decide whether each missing symbol deserves an entry. Not every public symbol belongs in SKILL.md — the skill is curated for the patterns users hit daily, not exhaustive API reference. Use these heuristics:

• Add it if it replaces or supersedes something already in the skill

(e.g., a new operation that is the idiomatic alternative to a documented workaround).

• Add it if it fits a category already present (a new aggregate function

goes in the aggregate list; a new join type goes in the joining section).

• Add it if it changes how a documented pattern should be written.
• Skip it if it is genuinely niche / advanced / experimental.
• Skip it if it is internal plumbing exposed for FFI but not user-facing.

When you flag a missing symbol, include a one-line proposed insertion point (which section / which table row) so a reviewer can decide quickly.

2. Stale mentions

For each function name, method name, or import shown in SKILL.md, verify it still exists in the current API:

• Function names mentioned in prose or in the categorized list should appear

in python/datafusion/functions/__init__.py's __all__.

• Spark function names mentioned in the "Spark-Compatible Functions"

subsection should appear in python/datafusion/functions/spark.py's __all__. Also confirm the divergent-semantics table still matches the current spark vs. main signatures.

• Method calls in code blocks should resolve against the current class.
• Imports (from datafusion import ...) should succeed against the current

__init__.py.

A quick way to check imports without running them:

python -c "from datafusion import SessionContext, col, lit; from datafusion import functions as F; print('ok')"

For each stale mention, propose either:

• a rename to the current name, or
• removal if the API is gone with no replacement.

3. Examples that drifted from idiomatic style

The skill teaches a Pythonic style: prefer plain strings to col(...) when a column reference is all you need; prefer raw Python values to lit(...) where auto-wrapping applies. Recent refactors (see the make-pythonic skill) keep moving more functions toward accepting native types.

For each code example in SKILL.md, check:

• Does it use lit(value) where a raw value would work? Comparison RHS,

arithmetic with a column, etc. all auto-wrap. (Reserve lit() for the cases listed in pitfall #2.)

• Does it use col("name") where a plain string would work? select(...),

aggregate([keys], ...), sort(...), sort_by(...) all accept plain name strings.

• Do functions.py calls match the current pythonic signature for that

function? If make-pythonic recently changed a signature (e.g., repeat(string, n: Expr | int)), the example should pass 3 rather than lit(3).

• Does any example use a deprecated or removed parameter name?

For drift, propose the updated snippet. If the change is purely stylistic and the older form still works, mark the suggestion as non-blocking.

4. Missing or stale version notes

When an API depends on a specific version, the skill should say so — otherwise an agent referencing the skill in an older project will write code that fails at import or at runtime.

datafusion-python shares its major version number with the upstream datafusion crate (e.g., datafusion-python 53.x tracks upstream datafusion 53). Always express version requirements in terms of datafusion-python only — there is no need to call out upstream and package versions separately.

Add a version note when:

• A method or function shown in the skill was added in a specific release

(e.g., a new DataFrame method that didn't exist before 53).

• A breaking change altered behavior in a specific release (signature

change, default-value change, new required argument).

• A pitfall was fixed in a specific release. Either annotate the pitfall

block with "fixed in datafusion-python NN, kept here for users on older versions" or remove it once the supported floor moves past that version.

Format for version notes (inline, italicized):

*Requires datafusion-python 53 or newer.*

For each missing/stale version note, propose the exact line and where it belongs.

How to discover changes since the last audit

If the user supplies a previous version or commit SHA where the audit was last run, diff against it:

# Public-API-relevant changes since SHA <prev>
git log --oneline <prev>..HEAD -- python/datafusion/
 
# Whose signatures actually moved
git diff <prev>..HEAD -- python/datafusion/functions.py | grep '^[+-]def '

If no prior audit point is given, fall back to "since the last upstream sync" by inspecting commits that touch Cargo.toml's datafusion pin:

git log --oneline -- Cargo.toml | grep -i datafusion | head -5

Output Format

Produce a report grouped by scope. Each finding is one bullet with a proposed action, so a maintainer can review the list quickly and apply edits in order.

## SKILL.md Audit (scope: <scope>)
 
Audited against:
- skills/datafusion_python/SKILL.md @ <git SHA / "working tree">
- datafusion-python <version>
 
### New APIs to cover
- `DataFrame.foo()` — added in datafusion-python 53. Insert in "DataFrame Operations Quick Reference" under <subsection>.
  Proposed snippet:
  ```python
  df.foo(...)
  ```
 
### Stale mentions
- "old_function_name" referenced in the categorized list (line N) — renamed to "new_function_name". Replace.
 
### Drifted examples
- "Filtering" section, `df.filter(col("a") > lit(10))` — drop `lit(10)`, auto-wrap applies. (non-blocking)
- "Aggregation" section, `df.aggregate([col("region")], ...)` — pass `"region"` as a plain string per "Projection" guidance.
 
### Version notes
- `DataFrame.foo()` block needs *Requires datafusion-python 53 or newer.*
- "Common Pitfalls" #N — fixed in datafusion-python 53; remove the pitfall and update the SQL-to-DataFrame row to no longer flag the workaround.
 
### No-change confirmed
- `SessionContext` data-loading section — all entries match current API.

If asked to apply the changes, edit skills/datafusion_python/SKILL.md directly with Edit tool calls, one finding at a time, and re-run the relevant doctest sanity check at the end:

pytest --doctest-modules python/datafusion -q

What NOT to flag

• Internal helpers / underscored names. Private symbols are not part of

the user-facing surface.

• Functions intentionally omitted. Niche / advanced APIs (custom

catalogs, raw FFI plumbing, low-level execution plan accessors) live in the API reference, not the skill. If an omission was deliberate and a comment / commit explains why, leave it out.

• Style nits inside explanatory prose. The skill mixes example code and

prose; only enforce the pythonic style on actual code blocks.

• Function-by-function coverage of every `functions.py` symbol. The

"Available Functions (Categorized)" list is curated by category, not exhaustive. Adding a single new aggregate to the aggregate list is enough — the user follows the pointer to the API reference for the rest.

Coordination with other skills

• Run /check-upstream first to expose any missing upstream APIs into the

Python layer. Without that, this skill cannot recommend documenting something that is not yet exposed.

• Run /make-pythonic before this skill if a Pythonic-signature pass is

planned for a release — that way this skill can update examples to the final signature in one shot rather than churning them twice.

• The order during an upstream sync (PR 3 of dev/release/upstream-sync.md)

is therefore: /check-upstream → /make-pythonic (optional) → /audit-skill-md.