infermatrixlab/inferrsi/sglang

SGLang inference engineering

Use this graph for SGLang engine reasoning, bounded tuning, and implementation reviews. Before relying on it, compare the checkout with provenance. The original LLM routes retain that snapshot; the diffusion supplement has its own pinned source notes and validation limits. These are portable candidates, not claims that a GPU configuration has passed.

Choose among source pins

Use the question and the retained source scope to select an entry. These pins are independent; each row's inspection date applies only to its stated scope.

Read the HiCache peer separately through its stable ID, for example inferrsi knowledge show sglang.cuda.llm.hierarchical_prefix_cache. The record link identifies retained provenance, not an implicit dependency of a diffusion bundle.

LLM prefix KV tiers do not define diffusion scheduler-instance, component or job-finalizer lifetimes. Common indexing and an engine entry do not qualify CUDA/NPU execution or cross-pin compatibility. Process transport/IPC alone does not establish Network capability.

Choose one main route; compose a second when its boundary is involved:

Record model/tokenizer revision, hardware and interconnect, SGLang commit, dependency versions, workload shape, and the requested outcome. Separate existing configuration, implementation change, and research extension before proposing work. Existing flags do not establish that their combinations are supported.

For the LLM routes, keep the request path explicit: HTTP/Engine → TokenizerManager → Scheduler/ScheduleBatch → TpModelWorker/ForwardBatch → ModelRunner/backend → results → DetokenizerManager → caller. Overlap scheduling and distributed roles change timing and ownership; the architecture reference locates those boundaries.

Prefer the smallest causal hypothesis and its observable check. Keep cache identities, request lifetime, allocation ownership, and collective participation intact. Verify output correctness before accepting a performance gain; report unmeasured effects as hypotheses. If a check fails, preserve the reproducer and return to a known baseline before adding another optimization.

The native candidate map points to specialized repository skills when code changes need them. It does not import or execute those skills automatically. Source map holds checked, commit-pinned evidence; routing metadata records AREX classification.

Scope excludes SGLang frontend language tutorials, diffusion paths beyond the bounded supplement, full deployment automation, and untested combinations across every accelerator. The supplement concerns this repository's multimodal_gen runtime, not SGLang-Omni. Load the relevant native implementation when extending beyond the evidenced paths.