sickn33/agentic-awesome-skills/llmops-platform-engineering

Claude Code and agent skill by sickn33/agentic-awesome-skills

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Build production LLMOps platforms with CI/CD, model promotion workflows,

Install

npx vskill@latest install sickn33/agentic-awesome-skills/llmops-platform-engineering

SKILL.md

LLMOps Platform Engineering

Design and operate an internal LLM platform that supports rapid experimentation without compromising reliability, cost, or compliance.

When to Use This Skill

  • Building an internal platform for teams to deploy and manage LLM-powered features
  • Designing CI/CD pipelines that include model evaluation gates
  • Setting up A/B testing infrastructure for model versions
  • Creating Kubernetes-based model serving infrastructure
  • Establishing governance workflows for model promotion

Prerequisites

  • Kubernetes cluster with GPU node pools (or cloud inference API access)
  • Container registry (Harbor, ECR, GCR, or ACR)
  • CI/CD system (GitHub Actions, GitLab CI, or Argo Workflows)
  • Observability stack (Prometheus + Grafana + OpenTelemetry)
  • Model registry (MLflow or custom metadata store)

Outcomes

  • Standardized path from experiment to production
  • Safe model rollout with quality and safety gates
  • Repeatable infra modules for inference, vector DB, and observability
  • Clear ownership model across platform, app, and security teams

Reference Architecture

  1. Control Plane: model registry, prompt/version catalog, policy checks, eval pipeline.
  2. Data Plane: inference gateway, vector database, cache, feature store.
  3. Ops Plane: telemetry, alerting, SLO dashboards, cost analytics.
  4. Security Plane: IAM boundaries, secret rotation, content filters, audit logs.

Model Promotion Pipeline

yaml
# .github/workflows/model-promotion.yaml
name: Model Promotion Pipeline
on:
  workflow_dispatch:
    inputs:
      model_name:
        description: "Model identifier"
        required: true
      model_version:
        description: "Model version to promote"
        required: true
      target_env:
        description: "Target environment"
        required: true
        type: choice
        options: [staging, production]
 
env:
  MODEL_NAME: ${{ inputs.model_name }}
  MODEL_VERSION: ${{ inputs.model_version }}
  TARGET_ENV: ${{ inputs.target_env }}
 
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Validate model identifiers
        shell: bash
        run: |
          [[ "$MODEL_NAME" =~ ^[a-z0-9][a-z0-9.-]*$ ]]
          [[ "$MODEL_VERSION" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]*$ ]]
 
      - name: Run quality evaluation suite
        run: |
          python -m evals.run \
            --model "${MODEL_NAME}:${MODEL_VERSION}" \
            --suite quality \
            --output results/quality.json
 
      - name: Run safety evaluation suite
        run: |
          python -m evals.run \
            --model "${MODEL_NAME}:${MODEL_VERSION}" \
            --suite safety \
            --output results/safety.json
 
      - name: Run latency benchmark
        run: |
          python -m evals.benchmark \
            --model "${MODEL_NAME}:${MODEL_VERSION}" \
            --concurrent-users 50 \
            --duration 300 \
            --output results/latency.json
 
      - name: Gate check - quality
        run: |
          python -m evals.gate_check \
            --results results/quality.json \
            --threshold-file thresholds/quality.yaml
 
      - name: Gate check - safety
        run: |
          python -m evals.gate_check \
            --results results/safety.json \
            --threshold-file thresholds/safety.yaml
 
      - name: Gate check - latency
        run: |
          python -m evals.gate_check \
            --results results/latency.json \
            --threshold-file thresholds/latency.yaml
 
      - name: Upload eval evidence
        uses: actions/upload-artifact@v4
        with:
          name: eval-results-${{ inputs.model_version }}
          path: results/
 
  approve:
    needs: evaluate
    runs-on: ubuntu-latest
    environment: ${{ inputs.target_env }}
    steps:
      - name: Record approval
        run: |
          echo "Approved by: ${{ github.actor }}"
          echo "Model: ${MODEL_NAME}:${MODEL_VERSION}"
          echo "Target: ${TARGET_ENV}"
          echo "Time: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
 
  deploy:
    needs: approve
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Deploy canary
        run: |
          kubectl set image deployment/${MODEL_NAME}-canary \
            model=${MODEL_NAME}:${MODEL_VERSION} \
            -n ai-${TARGET_ENV}
 
      - name: Wait for canary validation (15 min)
        run: |
          python -m canary.validate \
            --deployment ${MODEL_NAME}-canary \
            --namespace ai-${TARGET_ENV} \
            --duration 900 \
            --quality-threshold 0.85 \
            --error-rate-threshold 0.02
 
      - name: Promote to full rollout
        run: |
          kubectl set image deployment/${MODEL_NAME} \
            model=${MODEL_NAME}:${MODEL_VERSION} \
            -n ai-${TARGET_ENV}
          kubectl rollout status deployment/${MODEL_NAME} \
            -n ai-${TARGET_ENV} --timeout=300s

The evals, canary, thresholds, registry, cluster authentication and protected environment approval rules are application-owned prerequisites, not bundled implementations. Require configured production reviewers; a job named approve does not itself require human approval. Record the evaluated immutable image digest and deploy that same digest; the tag-based snippets are illustrative.

Evaluation Gate Thresholds

yaml
# thresholds/quality.yaml
gates:
  groundedness:
    metric: groundedness_score
    min: 0.85
    comparison: gte
  task_success:
    metric: task_success_rate
    min: 0.90
    comparison: gte
  hallucination:
    metric: hallucination_rate
    max: 0.08
    comparison: lte
  regression:
    metric: quality_delta_vs_baseline
    min: -0.02
    comparison: gte
    description: "Must not regress more than 2% vs current production"
 
# thresholds/latency.yaml
gates:
  p50_latency:
    metric: latency_p50_ms
    max: 800
    comparison: lte
  p95_latency:
    metric: latency_p95_ms
    max: 2000
    comparison: lte
  p99_latency:
    metric: latency_p99_ms
    max: 5000
    comparison: lte
  throughput:
    metric: requests_per_second
    min: 50
    comparison: gte

A/B Testing Configuration

yaml
# ab-test-config.yaml
apiVersion: gateway.ai/v1
kind: ABTest
metadata:
  name: model-comparison-q1
  namespace: ai-production
spec:
  duration: 7d
  traffic_split:
    control:
      model: example-control-model
      weight: 70
    treatment:
      model: example-treatment-model
      weight: 30
  metrics:
    primary:
      - task_success_rate
      - user_satisfaction_score
    secondary:
      - latency_p95
      - cost_per_request
      - hallucination_rate
  guardrails:
    auto_rollback_if:
      - metric: task_success_rate
        threshold: 0.80
        window: 1h
      - metric: hallucination_rate
        threshold: 0.15
        window: 30m
  assignment:
    strategy: sticky_user
    hash_key: user_id

gateway.ai/v1 is an illustrative custom resource; it requires your own installed CRD/controller. Model IDs are placeholders, not available provider snapshots.

Kubernetes Model Serving Deployment

yaml
# model-serving-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  namespace: ai-production
  labels:
    app: llm-inference
    model: local-open-weights
    version: "2025-01"
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
        model: local-open-weights
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: llm-inference
      containers:
        - name: model
          image: registry.internal/vllm-server:0.4.1
          args:
            - "--model=/models/current"
            - "--tensor-parallel-size=1"
            - "--max-model-len=8192"
            - "--gpu-memory-utilization=0.90"
          ports:
            - containerPort: 8000
              name: inference
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "8"
              memory: "32Gi"
              nvidia.com/gpu: "1"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 30
          volumeMounts:
            - name: model-weights
              mountPath: /models
              readOnly: true
            - name: config
              mountPath: /etc/vllm
      volumes:
        - name: model-weights
          persistentVolumeClaim:
            claimName: model-weights-pvc
        - name: config
          configMap:
            name: vllm-config
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      nodeSelector:
        gpu-type: a100
---
apiVersion: v1
kind: Service
metadata:
  name: llm-inference
  namespace: ai-production
spec:
  selector:
    app: llm-inference
  ports:
    - name: inference
      port: 8000
      targetPort: 8000
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-inference-hpa
  namespace: ai-production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-inference
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metric:
          name: llm_queue_depth
        target:
          type: AverageValue
          averageValue: "5"
    - type: Pods
      pods:
        metric:
          name: gpu_utilization_percent
        target:
          type: AverageValue
          averageValue: "75"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 2
          periodSeconds: 120
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 300

The deployment serves locally available open model weights, not proprietary hosted models. Provision the PVC and ConfigMap, choose a tested serving image, and install a custom-metrics adapter for the HPA metrics. Metrics use the inference port in this example.

CI/CD Design for AI Services

  • Build immutable containers with pinned dependencies and model hashes.
  • Use environment promotion: dev -> stage -> prod.
  • Fail deployment if:
  • regression evals drop below baseline,
  • safety tests exceed risk threshold,
  • p95 latency exceeds SLO budget.
  • Store deployment evidence for audits (commit SHA, eval report, approver).

Operational SLOs

SignalTargetMeasurement Window
Availability99.9%30-day rolling
p95 Latency< 1200ms5-min buckets
Cost per request< $0.051-hour average
Task success rate> 90%24-hour rolling
Groundedness> 85%24-hour rolling

Platform Guardrails

  • Enforce tenant quotas and model allow-lists.
  • Require structured output contracts for automation paths.
  • Default to low-risk model settings for critical workflows.
  • Disable unconstrained tool execution in production.

Tooling Stack (Example)

This is the start of the file. Read the full SKILL.md on GitHub.

Security

  • Security scanPassed, 95 of 100Pattern checks on the skill's files.
  • Model reviewPassed

See the full security report

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

Popularity

Repository stars
47,061
Repository forks
6,855
7-day trend score
54

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/sickn33/agentic-awesome-skills/llmops-platform-engineering/badge)](https://verified-skill.com/skills/sickn33/agentic-awesome-skills/llmops-platform-engineering)

Something wrong with this skill or its page? Tell us.