starrocks/starrocks-debug-skills/balance

Balance & Tablet Scheduler Troubleshooting

Investigation guide for tablet scheduling, cluster balance, replica repair, and decommission issues.

Five root causes account for the vast majority of cases:

• Cause A — balance throughput too low (clone tasks not progressing)
• Cause B — balance too aggressive (impacting writes and queries)
• Cause C — decommission BE stuck (replicas cannot be relocated)
• Cause D — clone task repeated failure (destination path or disk issues)
• Cause E — colocate group unstable (group marked Unstable, balance loops)

Metric Taxonomy — Read This First

Before using any metrics, understand the two-layer structure:

FE-level metrics (aggregated, cross-node)

BE-level metrics (per-node)

SQL observability

Key SHOW PROC "/statistic" fields:

Key SHOW PROC "/cluster_balance" sub-paths:

How to retrieve metrics

Option 1 — curl the FE HTTP port (immediate, no Prometheus needed)

# Pull all balance-related metrics from FE
curl -s "http://<fe_ip>:8030/metrics" | grep -E "scheduled_tablet|clone_task"
 
# Pending/running breakdown by type
curl -s "http://<fe_ip>:8030/metrics" | grep "scheduled_pending_tablet_num\|scheduled_running_tablet_num"

Option 2 — curl the BE HTTP port (per-node clone health)

# Clone failures on a specific BE
curl -s "http://<be_ip>:8040/metrics" | grep -E "clone_task|engine_requests.*clone"
 
# Loop across all BEs
for be in 10.0.0.1 10.0.0.2 10.0.0.3; do
  echo "=== $be ==="
  curl -s "http://$be:8040/metrics" | grep "engine_requests_total.*clone"
done

Option 3 — SQL (primary method for balance diagnosis)

-- Cluster-wide unhealthy tablet summary
SHOW PROC "/statistic";
 
-- Unhealthy tablets in a specific database
SHOW PROC "/statistic/<db_id>";
 
-- Balance overview
SHOW PROC "/cluster_balance";
SHOW PROC "/cluster_balance/sched_stat";
SHOW PROC "/cluster_balance/working_slots";
SHOW PROC "/cluster_balance/pending_tablets";
SHOW PROC "/cluster_balance/running_tablets";
SHOW PROC "/cluster_balance/history_tablets";
 
-- Disk usage per BE (v3.5+)
SHOW PROC "/cluster_balance/balance_stat";
SHOW PROC "/cluster_balance/cluster_load_stat/HDD";
SHOW PROC "/cluster_balance/cluster_load_stat/SSD";
 
-- Clone task history for a specific table
SELECT * FROM information_schema.fe_tablet_schedules
WHERE table_id = <table_id>
ORDER BY create_time DESC LIMIT 50;
 
-- Find partitions with imbalanced tablet distribution
SELECT partition_id, MAX(cnt) AS max_cnt, MIN(cnt) AS min_cnt,
       MAX(cnt) - MIN(cnt) AS diff
FROM (
  SELECT partition_id, be_id, COUNT(*) AS cnt
  FROM information_schema.be_tablets
  GROUP BY partition_id, be_id
) t
GROUP BY partition_id
HAVING diff > 1
ORDER BY diff DESC LIMIT 20;
 
-- Tables with replication_num > 2 (relevant for decommission)
SELECT db_name, table_name, partition_name, replication_num
FROM information_schema.partitions_meta
WHERE replication_num > 2;

Default HTTP ports: BE = 8040, FE = 8030. Verify with SHOW BACKENDS or be.conf / fe.conf.

Key rules for interpretation:

1. UnhealthyTabletNum > 0 is the primary alert signal. Drill into SHOW PROC "/statistic/<db_id>" to identify which databases are affected.
2. CloningTabletNum > 0 with UnhealthyTabletNum decreasing means repair is in progress — expected behavior. Only investigate if CloningTabletNum is stuck.
3. starrocks_fe_clone_task_success should be rising. If clone_task_total rises but clone_task_success does not, clone tasks are failing — check history_tablets.ErrMsg.
4. Disk usage difference > 10% between BEs (SHOW BACKENDS: DataUsedCapacity) triggers disk balance clones automatically.
5. working_slots all zero or all full: zero means no work; all full means scheduler is at capacity — check if tablets are actually moving by watching CloningTabletNum.

Balance Health Reference

Phase 1 — Confirm Balance / Repair Issue

Step 1.1 — Quick cluster health signal

-- Run on Leader FE
SHOW PROC "/statistic";

Key interpretation:

• UnhealthyTabletNum = 0 across all DBs → cluster is healthy, no action needed
• UnhealthyTabletNum > 0 → proceed to Step 1.2
• CloningTabletNum > 0 → repair/balance is in progress; check if it's making progress

Step 1.2 — Classify the issue type

SHOW PROC "/cluster_balance/sched_stat";

Check the Increase column to see if counters are advancing:

-- Drill into pending and running queues
SHOW PROC "/cluster_balance/pending_tablets";
SHOW PROC "/cluster_balance/running_tablets";
 
-- Check slot utilization per path
SHOW PROC "/cluster_balance/working_slots";

Step 1.3 — Identify which tablets are problematic

-- Clone task history with error messages
SHOW PROC "/cluster_balance/history_tablets";
 
-- Or query system table for specific table
SELECT tablet_id, type, state, schedule_reason, priority,
       failed_schedule_count, failed_running_count, msg
FROM information_schema.fe_tablet_schedules
WHERE table_id = <table_id>
ORDER BY create_time DESC LIMIT 20;

Phase 2 — Locate and Characterize

Step 2.1 — Identify issue type from Step 1.2

Cause A — Balance too slow

Confirm all match:

• pending_tablets queue large (many tablets waiting)
• working_slots shows low utilization (slots available but few tasks running)
• UnhealthyTabletNum not decreasing over 10+ minutes
• No error messages in history_tablets

Mechanism: The tablet scheduler submits clone tasks into slots — tablet_sched_slot_num_per_path per BE path. If slots are configured conservatively (default 8) and disks are fast, the scheduler underutilizes available IO, leaving balance tasks in the pending queue far longer than necessary. Each clone copies a full tablet replica from source to destination.

→ Go to Phase 3, Cause A

Cause B — Balance too active (impacts queries/writes)

Confirm all match:

• working_slots fill rate near 100%
• running_tablets at or near tablet_sched_max_scheduling_tablets limit
• Query latency elevated; load job write latency increased
• iostat on affected BEs shows disk IO > 80%

Mechanism: Clone tasks perform sequential read on source and sequential write on destination. When many clones run concurrently, they compete with query scan IO and load write IO for disk bandwidth. The scheduler has no IO-awareness — it will fill all slots regardless of disk pressure.

→ Go to Phase 3, Cause B

Cause C — Decommission BE stuck

Confirm all match:

• FE leader log shows: backend <id> lefts N replicas to decommission
• SHOW BACKENDS shows BE in DECOMMISSIONING state for > 30 minutes
• history_tablets shows REPLICA_RELOCATING tasks failing

Mechanism: Decommission moves all replicas off the BE. It can block when: (1) remaining BEs cannot satisfy replication factor (e.g., 3 BEs remaining with 3-replica tables); (2) tablets exist in FE recycle bin holding replica references; (3) statistics system tables have fixed 3-replica config.

→ Go to Phase 3, Cause C

Cause D — Clone task repeated failure

Confirm all match:

• history_tablets.ErrMsg contains unable to find dest path or similar
• failed_running_count > 3 for specific tablets in fe_tablet_schedules
• starrocks_be_engine_requests_total{status="failed",type="clone"} rising on specific BE

Mechanism: Clone tasks fail when the destination BE has no path with sufficient free space, the destination BE is down or unreachable, or the source tablet is in error state. Repeated failures cause the tablet to cycle in the pending queue indefinitely, keeping UnhealthyTabletNum elevated.

→ Go to Phase 3, Cause D

Cause E — Colocate group unstable

Confirm all match:

• SHOW PROC "/colocation_group" shows group in Unstable status
• num of colocate replica mismatch in sched_stat is non-zero and not clearing
• Group was recently affected by BE failure or BE addition

Mechanism: Colocate groups require tablets to be co-located on specific BEs as defined by the bucket sequence. When a BE fails or is added, the group is marked Unstable and the scheduler starts relocating tablets to restore the required co-location pattern. Until the group is Stable again, colocate joins degrade to regular joins.

→ Go to Phase 3, Cause E

Phase 3 — Take Action by Cause

Cause A — Balance too slow

Step A-1: Increase scheduler throughput (dynamic, no restart)