libraries-openly-fused/fusedkernellibrary/fkl-language-bindings

Building language bindings for FKL

Proven architecture: dynamic front-ends (like fkl-python) compose an Abstract Syntax Tree (AST) of operations, serialize their parameter structs into aligned memory, and generate a single fused C++ translation unit at runtime.

Why the obvious approaches fail

1. pybind11/SWIG over a fixed API: impossible — there is no fixed API. The kernel is GENERATED by the C++ type system from the user's op sequence; every distinct chain is a new template instantiation.
2. NVRTC: dead end. NVRTC compiles DEVICE code only, but FKL's fusion machinery (build(), BackFuser, Executor, grid computation) is HOST code. You cannot fuse with NVRTC alone.
3. Pre-compiled "bag of kernels": defeats the whole point — the combinatorial space of chains is unbounded.

The AST/Graph Architecture

Instead of binding C++ functions, the host language represents Operations and DPPs as graph nodes. Each node stores the literal C++ type string (e.g., "fk::Add<float>") and the host language allocates its parameter struct.

The host language (Python, Rust, etc.) is responsible for writing the C++ source strings for steps 1, 2, 3, and 4 below, concatenating them to generate the final .cu file:

// user composes lazily (Python graph nodes)
//      |
//      v   first call with concrete dtype/shape
// generate ONE .cu translation unit:
 
extern "C" void fkl_entry(const void** params_ptrs, void* stream)
{ 
    // THE HOST LANGUAGE GENERATES THE C++ SOURCE STRINGS FOR STEPS 1 THROUGH 4:
 
    // 1. Cast pointers to their exact struct types and dereference (guarantees alignment)
    auto in_ptr  = *reinterpret_cast<const fk::RawPtr<fk::ND::_2D, float>*>(params_ptrs[0]);
    auto p1      = *reinterpret_cast<const float*>(params_ptrs[1]);
    auto p2      = *reinterpret_cast<const fk::Rect*>(params_ptrs[2]);
    auto out_ptr = *reinterpret_cast<const fk::RawPtr<fk::ND::_2D, float>*>(params_ptrs[3]);
    
    // 2. Instantiate Read IOp via ::build()
    auto in_op  = fk::PerThreadRead<fk::ND::_2D, float>::build(in_ptr);
    
    // 3. Instantiate Compute and ReadBack IOps via ::build()
    auto op1    = fk::Add<float>::build(p1);
    auto op2    = fk::Crop<>::build(p2);
    
    // 4. Instantiate Write IOp and Execute the static DPP
    auto out_op = fk::PerThreadWrite<fk::ND::_2D, float>::build(out_ptr);
    fk::Stream s(reinterpret_cast<cudaStream_t>(stream)); // non-owning wrapper
    fk::executeOperations<fk::TransformDPP<...>>(s, in_op, op1, op2, out_op);
 
//      |
//      v
// single-step host+device compile to shared library (.so on Linux, .dll on Windows)
//      |
//      v
// disk-cache keyed by TYPE SIGNATURE; load + one FFI call per launch

Key decisions:

1. Types compile, values don't. The cache key contains: op tokens (names + template args), dtypes, layout (2D vs planes), batch size, OS, compiler, and FKL versions. Runtime values are passed through the pointer array at every call. Changing a value NEVER recompiles; changing the chain shape compiles exactly once.
2. Generate what a C++ user would write. The binding emits the exact ::build(params) calls and static executeOperations sequences a human would write.
3. clang first. clang compiles host+device CUDA in one invocation and is Apache-2.0; fall back to nvcc. Keep any required shims (texture header stubs, fatbinary flags) inside the binding.

The ABI (Pointer Array Architecture)

Keep fkl_entry dumb, stable, and immune to memory alignment undefined behavior:

• const void** params_ptrs: All runtime values, inputs, and outputs passed as an array of pointers.
• The host language allocates device pointers/shapes as fk::RawPtr structs alongside standard structs (like floats or Rects) using its native FFI tools, ensuring OS-level memory alignment.
• It then passes an array of pointers to these structs (void**).
• The generated C++ casts each pointer to its exact type. This prevents C++ strict-aliasing and struct-packing alignment crashes while supporting graphs with any number of inputs and outputs without changing the FFI signature.
• void* stream: external CUDA stream. If null, use a TU-local static Stream; if provided, run async and DON'T sync — the caller owns ordering (torch/cupy semantics).

Zero-copy interop

• Inputs: accept anything exposing a device pointer + shape + dtype (__cuda_array_interface__ in Python). Allocate a RawPtr struct on the host containing its pointer and strides, and pass its address in the params_ptrs array. Require C-contiguous or honor strides via the pitch argument.
• Outputs: allocate with the CUDA driver API and expose BOTH the array interface and DLPack so frameworks adopt the memory without copying. On DLPack export, transfer ownership to the consumer's deleter and disarm your own destructor (double-free guard).

Mapping the fusion techniques

Performance budget (measured reference points)

• steady-state fused launch: ~19-70 us/call (one FFI call, no sync path)
• cold compile: 1-3 s per signature, once, then disk-cached forever
• fused 6-op chain at 1080p: ~5x faster than 6 separate kernel launches

Testing a binding

• Validate against CPU references computed in the host language, not against FKL itself.
• Always test BOTH compilers (clang + nvcc), wiping the cache between.
• Test cross-platform generation (.dll on Windows, .so on Linux).
• Test that parameter value changes hit the cache (same library path) and that chain topology changes miss it.