apache/datafusion-python/check-upstream

Check Upstream DataFusion Feature Coverage

You are auditing the datafusion-python project to find features from the upstream Apache DataFusion Rust library that are not yet exposed in this Python binding project. Your goal is to identify gaps and, if asked, implement the missing bindings.

IMPORTANT: The Python API is the source of truth for coverage. A function or method is considered "exposed" if it exists in the Python API (e.g., python/datafusion/functions.py), even if there is no corresponding entry in the Rust bindings. Many upstream functions are aliases of other functions — the Python layer can expose these aliases by calling a different underlying Rust binding. Do NOT report a function as missing if it appears in the Python __all__ list and has a working implementation, regardless of whether a matching #[pyfunction] exists in Rust.

IMPORTANT: audit the total upstream surface, not the delta since the last pin. Gaps accumulate across syncs. A patch-release bump with a "bug fixes only" changelog does not mean there is nothing to find — pre-existing gaps from earlier majors still need to be surfaced. Always run the full comparison.

Compile-Signal Triggers

If a recent upstream bump required any of the following while fixing compile errors in crates/core/ or the FFI example, treat that as a hard signal that user-facing surface area grew and run this skill before considering the bump done. Each pattern corresponds to a class of gap that frequently shows up in the audit:

PR 1 of dev/release/upstream-sync.md asks you to log these signals as they appear. When you run this skill, use that log as a checklist: every entry must either show up in the audit output or be explicitly skipped with a reason.

Areas to Check

The user may specify an area via $ARGUMENTS. If no area is specified or "all" is given, check all areas.

1. Scalar Functions

Upstream source of truth:

• Rust docs: https://docs.rs/datafusion/latest/datafusion/functions/index.html
• User docs: https://datafusion.apache.org/user-guide/sql/scalar_functions.html

Where they are exposed in this project:

• Python API: python/datafusion/functions.py — each function wraps a call to datafusion._internal.functions
• Rust bindings: crates/core/src/functions.rs — #[pyfunction] definitions registered via init_module()

Evaluated and not requiring separate Python exposure:

• get_field_path — already covered by get_field(expr, *names), which takes a

variadic field path and dispatches to the same underlying functions::core::get_field UDF as the upstream get_field_path helper.

How to check:

1. Fetch the upstream scalar function documentation page
2. Compare against functions listed in python/datafusion/functions.py (check the __all__ list and function definitions)
3. A function is covered if it exists in the Python API — it does NOT need a dedicated Rust #[pyfunction]. Many functions are aliases that reuse another function's Rust binding.
4. Check against the "evaluated and not requiring exposure" list before flagging as a gap
5. Only report functions that are missing from the Python __all__ list / function definitions

2. Aggregate Functions

Upstream source of truth:

• Rust docs: https://docs.rs/datafusion/latest/datafusion/functions_aggregate/index.html
• User docs: https://datafusion.apache.org/user-guide/sql/aggregate_functions.html

Where they are exposed in this project:

• Python API: python/datafusion/functions.py (aggregate functions are mixed in with scalar functions)
• Rust bindings: crates/core/src/functions.rs

Evaluated and not requiring separate Python exposure:

• count_distinct — covered by count(expr, distinct=True). Both forms call

count_udaf with distinct: bool = true and produce the same logical plan.

• sum_distinct — covered by sum(expr, distinct=True).
• avg_distinct — covered by avg(expr, distinct=True).

How to check:

1. Fetch the upstream aggregate function documentation page
2. Compare against aggregate functions in python/datafusion/functions.py (check __all__ list and function definitions)
3. A function is covered if it exists in the Python API, even if it aliases another function's Rust binding
4. Check against the "evaluated and not requiring exposure" list before flagging as a gap
5. Report only functions missing from the Python API

3. Window Functions

Upstream source of truth:

• Rust docs: https://docs.rs/datafusion/latest/datafusion/functions_window/index.html
• User docs: https://datafusion.apache.org/user-guide/sql/window_functions.html

Where they are exposed in this project:

• Python API: python/datafusion/functions.py (window functions like rank, dense_rank, lag, lead, etc.)
• Rust bindings: crates/core/src/functions.rs

How to check:

1. Fetch the upstream window function documentation page
2. Compare against window functions in python/datafusion/functions.py (check __all__ list and function definitions)
3. A function is covered if it exists in the Python API, even if it aliases another function's Rust binding
4. Report only functions missing from the Python API

4. Table Functions

Upstream source of truth:

• Rust docs: https://docs.rs/datafusion/latest/datafusion/functions_table/index.html
• User docs: https://datafusion.apache.org/user-guide/sql/table_functions.html (if available)

Where they are exposed in this project:

• Python API: python/datafusion/functions.py and python/datafusion/user_defined.py (TableFunction/udtf)
• Rust bindings: crates/core/src/functions.rs and crates/core/src/udtf.rs

How to check:

1. Fetch the upstream table function documentation
2. Compare against what's available in the Python API
3. A function is covered if it exists in the Python API, even if it aliases another function's Rust binding
4. Report only functions missing from the Python API

5. DataFrame Operations

Upstream source of truth:

• Rust docs: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html

Where they are exposed in this project:

• Python API: python/datafusion/dataframe.py — the DataFrame class
• Rust bindings: crates/core/src/dataframe.rs — PyDataFrame with #[pymethods]

Evaluated and not requiring separate Python exposure:

• show_limit — already covered by DataFrame.show(), which provides the same functionality with a simpler API
• with_param_values — already covered by the param_values argument on SessionContext.sql(), which accomplishes the same thing more robustly
• union_by_name_distinct — already covered by DataFrame.union_by_name(distinct=True), which provides a more Pythonic API
• to_string — str(df) is the Pythonic way to get a string and already goes through __repr__ and the configurable formatter. A separate to_string() would either duplicate str(df) or render every row through a different path (session datafusion.format.* options, no formatter), giving a third text rendering alongside repr and show()

How to check:

1. Fetch the upstream DataFrame documentation page listing all methods
2. Compare against methods in python/datafusion/dataframe.py — this is the source of truth for coverage
3. The Rust bindings (crates/core/src/dataframe.rs) may be consulted for context, but a method is covered if it exists in the Python API
4. Check against the "evaluated and not requiring exposure" list before flagging as a gap
5. Report only methods missing from the Python API

6. SessionContext Methods

Upstream source of truth:

• Rust docs: https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html

Where they are exposed in this project:

• Python API: python/datafusion/context.py — the SessionContext class
• Rust bindings: crates/core/src/context.rs — PySessionContext with #[pymethods]

How to check:

1. Fetch the upstream SessionContext documentation page listing all methods
2. Compare against methods in python/datafusion/context.py — this is the source of truth for coverage
3. The Rust bindings (crates/core/src/context.rs) may be consulted for context, but a method is covered if it exists in the Python API
4. Report only methods missing from the Python API

7. FFI Types (datafusion-ffi)

Upstream source of truth:

• Crate source: https://github.com/apache/datafusion/tree/main/datafusion/ffi/src
• Rust docs: https://docs.rs/datafusion-ffi/latest/datafusion_ffi/

Where they are exposed in this project:

• Rust bindings: various files under crates/core/src/ and crates/util/src/
• FFI example: examples/datafusion-ffi-example/src/
• Dependency declared in root Cargo.toml and crates/core/Cargo.toml

Discovering currently supported FFI types: Grep for use datafusion_ffi:: in crates/core/src/ and crates/util/src/ to find all FFI types currently imported and used.

Evaluated and not requiring direct Python exposure: These upstream FFI types have been reviewed and do not need to be independently exposed to end users:

• FFI_ExecutionPlan — already used indirectly through table providers; no need for direct exposure
• FFI_PhysicalExpr / FFI_PhysicalSortExpr — internal physical planning types not expected to be needed by end users
• FFI_RecordBatchStream — one level deeper than FFI_ExecutionPlan, used internally when execution plans stream results
• FFI_SessionRef / ForeignSession — session sharing across FFI; Python manages sessions natively via SessionContext
• FFI_SessionConfig — Python can configure sessions natively without FFI
• FFI_ConfigOptions / FFI_TableOptions — internal configuration plumbing
• FFI_PlanProperties / FFI_Boundedness / FFI_EmissionType — read from existing plans, not user-facing
• FFI_Partitioning — supporting type for physical planning
• Supporting/utility types (FFI_Option, FFI_Result, WrappedSchema, WrappedArray, FFI_ColumnarValue, FFI_Volatility, FFI_InsertOp, FFI_AccumulatorArgs, FFI_Accumulator, FFI_GroupsAccumulator, FFI_EmitTo, FFI_AggregateOrderSensitivity, FFI_PartitionEvaluator, FFI_PartitionEvaluatorArgs, FFI_Range, FFI_SortOptions, FFI_Distribution, FFI_ExprProperties, FFI_SortProperties, FFI_Interval, FFI_TableProviderFilterPushDown, FFI_TableType) — used as building blocks within the types above, not independently exposed