libraries-openly-fused/fusedkernellibrary/fkl-using-the-library

Using the Fused Kernel Library

Mental model (60 seconds)

FKL fuses a SEQUENCE of operations into ONE kernel at C++ compile time. You write the operations in execution order; intermediates live in registers, never in DRAM:

#include <fused_kernel/fused_kernel.h>
using namespace fk;
 
Stream stream;
Ptr2D<uchar3> input(width, height);          // allocates device memory
Ptr2D<float3> output(width, height);
 
executeOperations<TransformDPP<>>(stream,
    PerThreadRead<ND::_2D, uchar3>::build(input),
    Cast<uchar3, float3>::build(),           // explicit dtype changes
    Mul<float3>::build({2.f, 2.f, 2.f}),
    Add<float3>::build({10.f, 20.f, 30.f}),
    PerThreadWrite<ND::_2D, float3>::build(output));
stream.sync();

Rules:

1. First IOp must be a complete Read (PerThreadRead, TensorRead, ReadSet, ...). Last must be a Write (PerThreadWrite, TensorWrite, TensorSplit, ...).
2. Each op's OutputType must match the next op's InputType. Type errors are compile errors with the offending pair in the message.
3. TYPES define the kernel. VALUES (build() arguments) are runtime parameters: changing a factor or a crop rect does NOT create a new kernel.

The two layers

• Operation structs (e.g. Mul<float3>): static exec() + type aliases. Pure compute, no state.
• InstantiableOperation (IOp) = Operation + its runtime params, created with Op::build(args...). What you pass to executeOperations.

Common pipeline patterns (all verified)

DNN preprocessing in one kernel (crop -> resize -> normalize -> planar)

executeOperations<TransformDPP<>>(stream,
    PerThreadRead<ND::_2D, uchar3>::build(frame),
    Crop<>::build(Rect(40, 30, 240, 180)),                         // ReadBack: fused into read
    Resize<InterpolationType::INTER_LINEAR>::build(Size(32, 32)),  // ReadBack, stacks on Crop
    Sub<float3>::build({123.675f, 116.28f, 103.53f}),
    Div<float3>::build({58.395f, 57.12f, 57.375f}),
    TensorSplit<float3>::build(chwTensor));                        // packed -> planar CHW

Many ROIs from one image (Horizontal Fusion: pass an array)

const std::array<Rect, 5> rois{ ... };             // array => batch => HF
Tensor<float3> out(64, 64, 5);                     // 5 planes
executeOperations<TransformDPP<>>(stream,
    PerThreadRead<ND::_2D, uchar3>::build(image),
    Crop<>::build(rois),                           // one kernel, 5 planes
    Resize<InterpolationType::INTER_LINEAR>::build(Size(64, 64)),
    Mul<float3>::build({1/255.f, 1/255.f, 1/255.f}),
    TensorWrite<float3>::build(out));

Batch of separate same-size images

const std::array<Ptr2D<float>, 4> inputs{ imgA, imgB, imgC, imgD };
executeOperations<TransformDPP<>>(stream,
    PerThreadRead<ND::_2D, float>::build(inputs),  // BatchRead under the hood
    Mul<float>::build(0.5f),
    TensorWrite<float>::build(out4planes));

Color conversion + channel ops

ColorConversion<ColorConversionCodes::COLOR_RGB2GRAY, uchar3, uchar>::build()
VectorReorder<uchar3, 2, 1, 0>::build()      // compile-time channel shuffle
VectorReorderRT<uchar3>::build({2, 1, 0})    // runtime shuffle (params)
Discard<uchar4, uchar3>::build()             // drop alpha
SaturateCast<float3, uchar3>::build()        // clamp + convert

Streams

• Stream stream; creates an owning CUDA stream; stream.sync() waits.
• Wrap an external stream zero-cost: Stream s(existingCudaStream); (FKL will NOT destroy it). Use this to interop with torch/cupy streams.
• All executeOperations overloads are async on the given stream.
• Stream is the stream of the default backend (ParArch::GPU_NVIDIA under nvcc, GPU_AMD under hipcc, CPU

otherwise). The multithreaded CPU stream is always explicit: Stream_<ParArch::CPU_MULTITHREAD> (see below).

Out-of-bounds reads

Wrap the read with a BorderReader policy BEFORE ops that may sample outside (Crop past the edge, warps). The backIOp is the complete read:

BorderReader<BorderType::REPLICATE>::build(
    PerThreadRead<ND::_2D, float>::build(input))
// CONSTANT takes the fill value too:
BorderReader<BorderType::CONSTANT>::build(readIOp, 0.f)

Policies: CONSTANT, REPLICATE, REFLECT, WRAP, REFLECT_101.

Temporal video windows

CircularTensor<T, COLOR_PLANES, BATCH, CircularTensorOrder, ColorPlanes> keeps the last BATCH frames on the GPU; update(stream, readIOp, ops..., writeIOp) preprocesses + inserts + rotates in one Divergent-HF kernel.

CPU backends

The same pipelines run on CPU: executeOperations<TransformDPP<ParArch::CPU>> with Stream_<ParArch::CPU> (single thread, runs before returning). Useful for tests without a GPU.

Multithreaded CPU: ParArch::CPU_MULTITHREAD

Available when FK_CPU_MULTITHREAD_ENABLED is defined (CMake option ENABLE_CPU_MULTITHREAD, ON when the thread-pool submodule is checked out: clone with --recursive). The work runs on fk::cpuThreadPool(), shared by all streams (one worker per hardware thread), with the same results as ParArch::CPU. Every DPP has it except the GPU-only attention DPPs; the filter entry points (executeLinearFilter, executeMorphology, executeBoxFilterQuad, ...) have Stream_<ParArch::CPU_MULTITHREAD>& overloads.

#ifdef FK_CPU_MULTITHREAD_ENABLED
Stream_<ParArch::CPU_MULTITHREAD> stream;
Ptr2D<float> input(width, height, 0, MemType::Host);   // explicit: CUDA/HIP builds default to DeviceAndPinned
Ptr2D<float> output(width, height, 0, MemType::Host);
executeOperations<TransformDPP<ParArch::CPU_MULTITHREAD>>(input, output, stream,   // keeps input/output alive
    Mul<float>::build(2.f), Add<float>::build(1.f));
stream.sync();   // waits, and rethrows the first exception the work threw
#endif

Stream rules (as with a CUDA stream):

• Entry points enqueue and return. The executions of a stream run in order; different streams run concurrently.
• sync() waits for the executions enqueued before it, and rethrows the first exception their work threw since the previous sync(). Called on a pool worker (e.g. inside cpuThreadPool().submit(...)), it runs queued work instead of blocking.
• A stream holds at most 1024 pending executions; past that, enqueueing waits.
• Copies of a stream share its queue; the last copy to be destroyed waits for the executions (errors nobody synced are lost).

Memory rules:

• Everything an execution reads or writes must stay valid until it has finished (sync()). The executeOperations overloads that take Ptr/Tensor/std::array<Ptr2D> arguments, and setTo, keep those buffers alive. IOps only hold a RawPtr, even when build() got a Ptr or an Image, so those buffers (and a CircularTensor being updated) are NOT kept alive: do not release them, or access them from the host, before sync().
• Memory must be Host or HostPinned. On this stream Ptr/Image upload/download are ordered no-ops (there is a single buffer), and they and setTo throw std::runtime_error for Device/DeviceAndPinned memory.

Pitfalls (each cost real debugging time)

1. Mismatched adjacent types: read the static_assert chain bottom-up; the first frame names the two ops that disagree.
2. An array-built op (e.g. Crop<>::build(std::array<Rect,N>)) enables Horizontal Fusion (batch planes). Just pass the IOps to executeOperations as usual; the executor/BackFuser handles the fusion automatically. Ensure your output is a batched write (e.g. TensorWrite/TensorSplit) that matches the batch size N.