quic/efficient-transformers/transformers-mainline-rebase

Transformers Mainline Rebase

Purpose

Use this skill to rebase QEff from an existing HF baseline to a newer HF release while keeping:

• wrapper behavior stable
• PyTorch parity stable
• ONNX export/runtime stable
• local divergence minimal

For this validated migration:

• baseline: transformers==4.55.0
• target: transformers==5.3.0

This file is also a template for future rebases.

Rebase Strategy Pattern

Use this repeatable structure for any HF mainline jump:

• do one broad compatibility alignment pass first (deps + wrappers + export/runtime glue)
• follow with focused cleanup passes (remove temporary compatibility hacks)
• isolate cache abstraction hardening into dedicated patches
• finish with deterministic test-hygiene improvements

This sequencing keeps diffs understandable, makes regressions easier to bisect, and prevents local hacks from becoming permanent.

Source Of Truth

• Ultimate validation gate: tests/test_model_quickcheck.py
• Do not claim success unless full quickcheck passes in the user-provided environment.

Environment Rule

• Environment is user-provided.
• Use the user’s activation/exec method and user-provided test command.
• For this repo, a known-good example is:
• pyenv activate qeff.mainline
• If the user does not specify a command, default to:
• python -m pytest -q tests/test_model_quickcheck.py -n auto

Rebase Workflow (Always)

1. Audit dependency stack first.
2. Run full quickcheck once to capture failure baseline.
3. Fix failures in focused groups (small diffs, re-test quickly).
4. Re-run full quickcheck.
5. Report exact results and remaining gaps (if any).

Turnkey Operator Flow (Low-Touch Handoff)

Use this exact sequence for a smooth first-time rebase execution:

1. Environment bootstrap

• activate the user-provided environment
• install dependencies from pyproject.toml (plus optional test extras)

1. Hygiene cleanup

• clear stale temp/export/cache artifacts before any test run
• ensure cleanup is single-run under xdist environments

1. Baseline gate

• run full quickcheck once:
• python -m pytest -q tests/test_model_quickcheck.py -n auto
• record failures by bucket:
• causal helper/mask drift
• cache API/object drift
• ONNX export signature drift
• AWQ/quantizer drift
• Mixtral/GPT-OSS MoE drift
• Gemma3/VLM multimodal drift
• Whisper/audio drift

1. Deterministic fix loop

• pick one bucket only
• implement minimal compatibility patch
• run only the relevant targeted quickcheck subset
• repeat until bucket is green, then move to next bucket

1. Final acceptance

• rerun full quickcheck command
• require full green before claiming successful migration

1. Handoff output

• include:
• exact commands used
• final pass/fail counts
• runtime duration
• any temporary shims left and their removal conditions

Blocker Escalation Rules

If a rebase engineer is blocked, follow this order:

1. verify dependency pins and optional imports first
2. verify cache tuple/object conversion boundaries
3. verify wrapper signature drift (head_mask, rotary args, causal helpers)
4. verify export-path differences (runtime path may still be correct)
5. isolate model-family exceptions (Mixtral, GPT-OSS, Gemma3, Whisper, AWQ)

If still blocked after these checks:

• keep the change scoped
• document failing test IDs and exact traceback
• avoid broad refactors until the failing bucket is isolated

Rebase Workflow (Detailed Execution Contract)

1. Dependency and API surface lock

• audit dependency pins together (transformers, huggingface-hub, diffusers, peft, export/runtime libs)
• ensure no mixed-era cache/quantizer imports remain

1. Baseline failure snapshot

• run full quickcheck once immediately after dependency bump
• capture failing test IDs and map each to one drift bucket (cache, attention/mask, export signature, quantizer, MoE, multimodal)

1. Fix by bucket with strict scope

• keep each patch focused to one drift bucket
• preserve upstream semantics first, then add minimal compatibility shim only where needed
• avoid model-specific hacks until shared path is exhausted

1. Deterministic reruns

• run targeted -k subsets after each bucket
• always clean stale artifacts before full rerun
• run full quickcheck with the exact user-provided command as final gate

1. Report with evidence

• include commands, pass/fail counts, duration, and unresolved risk (if any)
• do not claim parity unless HF PyTorch -> QEff PyTorch -> ORT checks are green in quickcheck

Practical Tricks That Keep Rebases Intact

• Keep compatibility shims at boundaries, not deep inside model math:
• input normalization (tuple cache -> cache objects)
• output normalization (cache objects -> legacy tuples for export/tracing)
• Prefer feature detection over version checks:
• use hasattr(...), optional imports, and safe getattr(..., default) guards
• avoid hardcoding behavior to exact HF versions where possible
• Default to upstream semantics first:
• patch wrappers only when parity/export requires it
• if a temporary local hack is added to unblock, schedule its removal once stable path is validated
• Make cache code resilient:
• preserve both legacy tuple and object cache interoperability
• tolerate optional/extra args in cache methods likely to drift upstream
• keep encoder-decoder and decoder-only paths explicit
• Keep ONNX export deterministic:
• prefer one export-call pattern by default and only branch when architecture truly requires it
• keep past_key_values naming and shape handling stable for both tuple/object forms
• Control flaky failures early:
• clean stale temp artifacts before full/regression reruns
• avoid per-worker destructive cleanup races in xdist
• Preserve API behavior externally:
• wrappers should continue to accept previous public inputs and return expected output structure whenever feasible
• if behavior must change, document exact impact and add targeted tests
• Use micro-triage loops:
• run smallest useful -k scope after each bucket fix
• only run full quickcheck after targeted scopes are green

Robustness Rules (Non-Negotiable)

• Do not rely on private or fragile HF internals when a stable public abstraction exists.
• Do not merge rebase work with unrelated refactors.
• Do not claim parity based only on export success; runtime parity and ORT parity must both be validated.
• Do not keep compatibility hacks that are no longer required after upstream-aligned fixes.
• Do not break existing public wrapper call signatures unless explicitly approved.

Dependency Alignment (4.55.0 -> 5.3.0)

Use this known-good set unless user overrides:

• transformers==5.3.0
• diffusers==0.37.0
• huggingface-hub==1.7.1
• peft==0.18.1
• hf_transfer==0.1.9
• datasets==2.20.0
• fsspec==2023.6.0
• sentencepiece==0.2.0
• onnx==1.18.0
• onnxruntime==1.22
• onnxscript==0.2.5
• numpy==1.26.4
• protobuf==6.31.0

Must-Do Cleanup Before Export/Parity Tests

Clean stale artifacts before focused/full quickcheck:

• /tmp/*qeff*, /tmp/*onnx*, /tmp/*qnn*
• ~/.cache/qeff_models (or active QEFF_HOME)
• ensure cleanup is done once per session when running xdist (avoid parallel worker races)

If shell globbing fails, use find cleanup.

4.55.0 -> 5.3.0 Required Fixes

1) Causal wrapper helper drift

Symptom:

• missing helper methods like get_head_mask, _update_causal_mask

Fix:

• remove dependency on removed HF helpers
• use explicit head-mask fallback: head_mask = [None] * num_layers when needed
• keep causal mask creation centralized and consistent
• if temporary mask alignment was introduced to unblock parity, remove it once upstream-compatible path is restored
• when attention_mask is absent, always build an explicit causal mask in wrapper Model.forward (do not rely on model-internal implicit causality only)
• resolve causal-mask target_length defensively for preallocated tuple caches:
• start from past_seen_tokens
• if <=0, fallback to past_key_values.layers[0].keys.shape[-2] when available
• if still <=0, fallback to int(position_ids.max()) + 1
• final fallback: current input sequence length
• for RoPE wrappers, ensure rotary cache length is at least max requested position:
• kv_seq_len = max(kv_seq_len, int(position_ids.max()) + 1)
• this avoids out-of-bounds rotary indexing and avoids silent clamp-induced parity drift

2) Cache API drift

Symptom:

• failures around get_seq_length(...)
• tuple/cache object incompatibility

Fix:

• keep QEffDynamicCache tuple interop and direct Cache API compatibility:
• from_legacy_cache()
• to_legacy_cache()
• provide __getitem__ / __iter__ compatibility where older call paths index cache tuples
• QEffDynamicCache.get_seq_length accepts extra args safely
• in QEffEncoderDecoderCache.from_legacy_cache, use positional ctor:
• cls(QEffDynamicCache(), QEffDynamicCache())
• implement QEffEncoderDecoderCache.to_legacy_cache() for tuple-returning call sites
• when decoupling from HF internals, prefer inheriting from Cache with local layer class behavior instead of relying on unstable upstream DynamicLayer details

3) Runtime cache conversion (run_utils.py)

Fix:

• tuple -> QEffDynamicCache for decoder-only models
• tuple -> HF EncoderDecoderCache(...) for encoder-decoder models when upstream expects native object behavior
• keep model-specific exceptions where required:
• gpt_oss
• Gemma3 family (prefer HF DynamicCache and preserve text-side generation fallback behavior)

4) Encoder export use_cache collisions (BERT-like)

Symptom:

• ... got multiple values for argument 'use_cache'

Fix:

• do not force cache for encoder-only export path
• keep exporter on keyword path by default:
• torch.onnx.export(..., args=(), kwargs=example_inputs, ...)
• avoid mixing positional dict arguments for decoder-like models unless strictly required

5) Whisper drift

Symptoms:

• invalid/zero mask shape failures
• encoder layer arg drift (layer_head_mask)
• encoder-decoder cache constructor drift

Fix:

• tolerate invalid/zero masks (drop instead of hard fail)
• remove stale layer_head_mask forwarding if upstream encoder layer no longer accepts it
• ensure encoder-decoder cache legacy conversion and legacy return path both work

6) AWQ drift

Symptoms:

• enum/config attr drift
• optional field missing (e.g. do_fuse)

Fix:

• tolerate backend/format enum renames
• provide both update_dtype and update_torch_dtype
• guard optional attrs via getattr(...)
• keep post-load hook no-op
• keep matmul-nbits meta-safe guard for .item() usage

7) Mixtral v5 sparse-MoE drift

Symptoms:

• gate output shape/API drift
• ONNX failure from aten::histc

Fix:

• support both gate styles:
• v5 tuple return (router_logits, routing_weights, selected_experts)
• legacy logits path (softmax + topk)
• for v5 experts runtime path, preserve upstream semantics
• for ONNX export, use:
• transformers.integrations.moe.batched_mm_experts_forward(...)
• keep legacy expert-container fallback

8) GPT-OSS v5 MoE/projection drift

Symptom:

• wrapper expects split gate/up projections while upstream uses fused gate_up_proj

Fix:

• initialize split compatibility tensors from fused tensors in QEffGptOssExperts.__qeff_init__
• keep existing MLP math paths unchanged
• retain runtime cache exception behavior for gpt_oss

MoE Rebase Playbook (High Priority)

Use this section first when quickcheck failures are concentrated in Mixtral/GPT-OSS.