gadaalabs/claude-code-on-steroids/gradient

ML Engineering Patterns

Overview

GRADIENT — In ML, the gradient is the directional signal that tells you exactly how to improve. When invoked: assesses pipeline stage (data / training / serving / MLOps), loads the relevant pattern file, and applies ML-specific validation — schema checks, drift detection, training-serving skew guards, latency budgets.

Core principle: ML systems have unique failure modes — data drift, training-serving skew, silent degradation. Test data and models, not just code.

Announce at start: "Running GRADIENT for ML-specific patterns."

Entry Point — First 5 Minutes

STAGE ASSESSMENT:
 
"What stage are you at?"
A) Data collection / ingestion
B) Feature engineering / preprocessing
C) Model training / experimentation
D) Model evaluation / validation
E) Model serving / inference
F) Production monitoring / MLOps
G) Debugging an ML failure

Stage → Section mapping:

• A/B → Data Pipeline Testing (see patterns/data-pipeline.md)
• C → Model Training (see patterns/model-training.md)
• D → ML Evaluation (correctness metrics, calibration)
• E → Model Serving TDD (see patterns/model-serving.md)
• F → MLOps Deployment (see patterns/mlops.md)
• G → Run hunter first, then return with evidence

After identifying stage, ask: "What's the primary constraint — accuracy, latency, cost, or reliability?"

Data Pipeline Testing

Load patterns: `patterns/data-pipeline.md`

Key tests to implement:

1. Schema validation — column set, feature types, label distribution
2. Distribution shift detection — KS test per feature, covariate shift AUC
3. Null / edge case handling — nulls, empty input, extreme values
4. Data leakage check — correlation with future labels, reproducibility

Rule: Write pipeline tests before writing model code.

Model Training Checklist

Load patterns: `patterns/model-training.md`

Before training complex models:

• [ ] Simple baseline implemented (logistic regression / majority class)
• [ ] Baseline metrics documented (accuracy, F1, AUC-ROC, latency)
• [ ] Hyperparameter search configured (Bayesian preferred; grid only if ≤ 3 params)
• [ ] Early stopping rules defined (patience, min_delta, restore_best_weights)
• [ ] Checkpoint strategy set (save_best_only, monitor_metric, metadata)
• [ ] Ablation study planned (each component justified)

Model Serving TDD

Load patterns: `patterns/model-serving.md`

Tests required before deployment:

1. Input validation — schema, range, distribution z-score check
2. Latency — p99 within budget, throughput at expected QPS
3. Fallback — model unavailable → default response, timeout → 504
4. Output calibration — confidence bins match actual accuracy ± 5%

MLOps Deployment

Load patterns: `patterns/mlops.md`

Required components:

1. Model versioning — semver, git commit + data version + metrics in metadata
2. A/B test config — traffic split, primary metric, guardrail metrics, stopping criteria
3. Drift monitoring — PSI for data drift (threshold 0.1), CUSUM for concept drift
4. Rollback triggers — accuracy drop >5%, error rate >1%, latency >2×, PSI >0.25

ML Evaluation Framework

Cost/latency budgets (set before training, enforce in CI):

• p99 latency: 100ms
• cost per 1k requests: $1.00

Red Flags

Never:

• Deploy without baseline comparison
• Skip drift monitoring in production
• Train without checkpointing
• Use test data for training
• Deploy without rollback plan

Always:

• Test data pipelines before model training
• Validate input distribution matches training
• Track model version with data version
• Monitor prediction distribution in production
• Have fallback for model failures

Integration with Superpowers

Final Checklist

• [ ] Data pipeline tests pass (schema, distribution, edge cases)
• [ ] Baseline model established and documented
• [ ] Complex model beats baseline (with ablation study)
• [ ] Input validation tests pass
• [ ] Output calibration verified
• [ ] Latency budget met (p99)
• [ ] Fallback behavior tested
• [ ] Model versioned with metadata
• [ ] Drift monitoring configured
• [ ] Rollback triggers defined