starrocks/starrocks-debug-skills/data-lake

Name: data-lake
Author: starrocks

Claude Code and agent skill by starrocks/starrocks-debug-skills

Trust tier 2

community

LLM review pending

Source check pending

Checking that the SKILL.md file is still in the repository.

Use when external catalog queries (Hive, Iceberg, S3, HDFS, Paimon) are failing, hanging, or returning stale data. Covers Hive Metastore connectivity errors, Kerberos authentication failures (keytab expiry, GSS errors, clock skew), S3 rate limiting and SSL issues, network saturation from HMS, and metadata cache TTL tuning for data freshness.

Install

npx vskill@latest install starrocks/starrocks-debug-skills/data-lake

SKILL.md

Data Lake / External Table Troubleshooting

Investigation guide for Hive Metastore connectivity, Kerberos authentication, HDFS/S3 access errors, and external catalog query failures.

Five root causes account for the vast majority of cases:

Cause A — HMS connectivity failure or metadata listing too slow
Cause B — Kerberos authentication failure (expired keytab or clock skew)
Cause C — S3 / object storage access or rate-limit error
Cause D — Network saturation from HMS traffic on CN/BE nodes
Cause E — Metadata cache staleness (Hive partitions, Iceberg snapshots)

Metric Taxonomy — Read This First

Before using any diagnostic commands, understand the observable signals for external table issues.

FE log signals (`fe.log`)

Signal	Meaning
`Start to connect to Hive metastore` + no subsequent `Connected`	HMS TCP connection blocking or timing out
`getPartitions` / `listPartitions` with high elapsed ms	HMS partition listing slow (large partition count or HMS overload)
`Finished replaying journal` never appears after HMS connect attempt	HMS blocking FE startup sequence
`AuthenticationException` from catalog connector	Kerberos ticket invalid or HMS auth rejected

BE log signals (`be.log` / `be.WARNING`)

Signal	Meaning
`GSS initiate failed`	Kerberos GSSAPI handshake failed — expired keytab or JCE missing
`javax.security.sasl.SaslException`	SASL negotiation failed — auth or encryption mismatch
`503 reduce request rate`	S3 rate limit hit — partition prefix count too low
`Filesystem closed`	HDFS NameNode cache too small — `hdfs_client_max_cache_size` must exceed NN count
Network I/O bytes/sec approaching NIC capacity	External scan saturating link

Network / thread signals

Signal	Meaning
jstack shows threads blocked on `TFilterTransport.readAll`	HMS RPC call blocking; HMS slow or overloaded
`sar -n DEV 1 5` shows RX/TX near line rate on CN/BE host	Network saturation from remote scan
`netstat -s \	grep retransmit` spike during external scan	Packet loss from bandwidth burst

How to retrieve signals

Option 1 — FE log grep

bash

# Check HMS connectivity on FE startup
grep -E "connect.*Hive metastore|Connected to HMS|AuthenticationException" fe.log | tail -30
 
# Check partition listing latency warnings
grep -E "getPartitions|listPartitions" fe.log | tail -20
 
# Check if FE completed startup
grep "Finished replaying journal" fe.log | tail -5

Option 2 — BE log grep

bash

# GSS / Kerberos errors on BE
grep -E "GSS initiate failed|SaslException|kinit" be.log | tail -20
 
# S3 rate limit errors
grep -E "503|reduce request rate|rate limit" be.log | tail -20
 
# HDFS filesystem errors
grep -E "Filesystem closed|hdfs_client" be.log | tail -20

Option 3 — Kerberos ticket status

bash

# Check ticket cache expiry
klist
 
# List principals in keytab file
klist -kt /path/to/keytab
 
# Test kinit manually
kinit -kt /path/to/keytab principal_name@REALM

Option 4 — Network thread analysis

bash

# jstack to detect HMS-blocked threads (FE)
jstack $(cat fe/bin/fe.pid) > /tmp/fe_jstack.log
grep -A 5 "TFilterTransport\|HiveMetaStore" /tmp/fe_jstack.log | head -40
 
# Network I/O rate on BE host
sar -n DEV 1 5
 
# S3 / HDFS endpoint connectivity
curl -v --connect-timeout 10 http://<s3-endpoint>/<bucket>/<key>
telnet <hms-host> <hms-port>

Key rules for interpretation:

If HMS connection attempt has no response for >60s in fe.log, suspect TCP-level block (firewall, wrong port, HMS down) rather than auth.
GSS initiate failed always means Kerberos — check klist first; if ticket is valid, check JCE installation.
S3 rate-limit errors (503) escalate with partition count — the fix is num_partitioned_prefix, not retry tuning.
Filesystem closed with hdfs_client_max_cache_size too small causes random HMS disconnects that look like auth failures.

External Table Issue Reference

Error keyword / symptom	Root cause	Quick confirm
`GSS initiate failed`	Kerberos — expired keytab or missing JCE	`klist` on BE host; `klist -kt <keytab>`
`Server not found in Kerberos database`	SPN mismatch	Verify service principal in `krb5.conf` matches HMS
`Clock skew too great`	NTP not synced	`timedatectl` / `ntpstat` on FE and BE hosts
`Preauthentication failed`	Wrong keytab/password	Regenerate keytab from KDC
`Filesystem closed`	NN cache too small	Check `hdfs_client_max_cache_size` < NN count
`Failed to get current notification event id`	HMS user missing permissions	Check HMS user; set `HADOOP_USER_NAME`
`set_ugi() not successful`	Missing `HADOOP_USER_NAME` in hadoop_env.sh	Set env var; check Kerberos config
`503 reduce request rate` from S3	S3 partition prefix rate limit	Increase `num_partitioned_prefix`
SSL handshake error to private OSS	Default SSL on; param bug in older versions	Upgrade; set `ssl_enable = false` in catalog
jstack: threads on `TFilterTransport.readAll`	HMS overloaded or slow	Check HMS thread pool; add HMS timeout config
New Hive partitions not visible	Metadata cache TTL	`REFRESH DATABASE` or reduce `metastore_cache_ttl_sec`
Iceberg sees old snapshot	Snapshot cache TTL	Reduce `iceberg_table_cache_ttl_sec`; `REFRESH TABLE`

Phase 1 — Classify the Issue

Step 1.1 — Identify failure category

Check in order:

bash

# 1. Is HMS reachable?
telnet <hms-host> <hms-port>
# Success → go to Step 1.2 (auth / metadata)
# Refused / timeout → Cause A (HMS connectivity)
 
# 2. Is auth failing?
grep -E "GSS initiate|SaslException|AuthenticationException" be.log fe.log | tail -10
# Found → Cause B (Kerberos)
 
# 3. Is S3 rate-limiting?
grep -E "503|reduce request rate" be.log | tail -10
# Found → Cause C (S3)
 
# 4. Is network saturating?
sar -n DEV 1 5
# RX/TX near line rate → Cause D (network)
 
# 5. Is data stale?
# If queries succeed but return old/missing data → Cause E (cache freshness)

Symptom → root cause signal:

Pattern	Points toward
HMS TCP connect fails or hangs >60s	Cause A — HMS connectivity
`GSS initiate failed` or `Clock skew` in logs	Cause B — Kerberos auth
`503` or `rate limit` from S3 in BE log	Cause C — S3 rate limit
jstack threads blocked on `TFilterTransport`; NIC near line rate	Cause D — network saturation
Queries succeed but data is stale or partitions missing	Cause E — cache freshness

Phase 2 — Locate and Characterize

Step 2.1 — Confirm HMS connectivity

bash

# Test TCP reachability
telnet <hms-host> 9083
 
# Check FE log for HMS connection timeline
grep -E "connect.*metastore|Finished replaying" fe.log | tail -20
 
# Check if catalog is blocking FE startup
grep "Finished replaying journal" fe.log
# If this line never appears after catalog connect attempt → Cause A (Startup block)

Step 2.2 — Confirm Kerberos state

bash

# Is the ticket valid?
klist
# Look for: Expires < now → renew immediately
 
# Is the keytab readable?
klist -kt /path/to/keytab
 
# Enable debug output (temporary)
# In fe.conf JAVA_OPTS:
# -Dsun.security.krb5.debug=true
# In be/conf/hadoop_env.sh:
# HADOOP_OPTS="$HADOOP_OPTS -Dsun.security.krb5.debug=true"
# Debug output goes to fe.out / be.out

Step 2.3 — Confirm S3 access

bash

# Check S3 endpoint connectivity and rate errors
grep -E "503|rate limit|connection.maximum" be.log | tail -20
 
# Check catalog config for connection pool
SHOW CREATE CATALOG <catalog_name>;
# Look for: fs.s3a.connection.maximum setting

Step 2.4 — Characterize cache staleness

sql

-- Check current catalog cache config
SHOW CREATE CATALOG <catalog_name>;
 
-- Check if manual refresh fixes the issue
REFRESH TABLE <catalog_name>.<db_name>.<table_name>;
-- If query now returns fresh data → Cause E confirmed

Phase 3 — Take Action by Cause

Cause A — HMS Connectivity / Metadata Slow

Confirm all match:

telnet <hms-host> <port> fails or hangs >30s
fe.log shows HMS connect attempt with no completion
OR listPartitions calls show high latency (>10s)

Mechanism: FE catalog module connects to HMS via Thrift RPC at startup and at query planning time. If HMS is unreachable or its thread pool is saturated, FE blocks waiting for partition metadata. For large catalogs (>10k partitions), each planning call lists partitions serially — HMS response time directly extends query planning time.

Actions:

bash

# Step A-1: Verify HMS is reachable
telnet <hms-host> 9083
nc -zv <hms-host> 9083
 
# Step A-2: Check HMS service health independently
# (SSH to HMS host and check HMS logs / process status)
 
# Step A-3: Temporarily remove catalog to allow FE to start
# (edit fe.conf, remove catalog section, restart FE)

sql

-- Step A-4: Enable metadata caching to reduce HMS call frequency
ALTER CATALOG <catalog_name> SET PROPERTIES (
    "hive.metastore.cache.ttl" = "600",
    "metastore_cache_refresh_interval_sec" = "120"
);
 
-- Step A-5: Add HMS connection timeout (prevents indefinite block)
ALTER CATALOG <catalog_name> SET PROPERTIES (
    "hive.metastore.timeout" = "10"
);

bash

# Step A-6: If HMS slow due to large partition count — apply partition pruning
# Ensure queries use partition-key predicates in WHERE clause
# Check EXPLAIN output for PartitionPruning: true

→ Go to Phase 4: Verify Recovery

Cause B — Kerberos Authentication Failure

Confirm all match:

klist shows expired ticket OR klist -kt shows keytab principal mismatch
be.log shows GSS initiate failed or Preauthentication failed
OR Clock skew too great in logs

Mechanism: Kerberos uses time-limited tickets (TGTs). When the TGT expires and automatic renewal is not configured, every HMS/HDFS RPC from BE fails at the GSSAPI layer. Clock skew >5 minutes also invalidates otherwise-valid tickets because Kerberos has a built-in replay-attack window. JCE (Java Cryptography Extensions) is required for AES-256 encryption used by modern KDCs.

Actions:

bash

# Step B-1: Renew ticket immediately
kinit -kt /path/to/keytab principal_name@REALM
klist  # Confirm new expiry time
 
# Step B-2: Fix clock skew if present
timedatectl status
# If not synced: systemctl restart chronyd (or ntpd)
 
# Step B-3: Install JCE if AES-256 is used
# Download from Oracle: Java Cryptography Extension (JCE) Unlimited Strength
# Place in $JAVA_HOME/jre/lib/security/
 
# Step B-4: Set up automatic ticket renewal (permanent fix)
# Add to crontab on each FE/BE host:
# */30 * * * * kinit -R -kt /path/to/keytab principal@REALM 2>/dev/null || kinit -kt /path/to/keytab principal@REALM
 
# Step B-5: Verify service principal matches HMS
# krb5.conf hive.service.kerberos.principal must match HMS server config

Error	Cause	Fix
`GSS initiate failed`	Expired keytab or missing JCE	Renew keytab; install JCE for AES-256
`Server not found in Kerberos database`	SPN mismatch	Verify service principal matches server
`Clock skew too great`	NTP not synced	Ensure NTP/chrony is running
`Preauthentication failed`	Wrong keytab/password	Regenerate keytab from KDC
`Checksum failed`	Encryption type mismatch	Check `krb5.conf` encryption settings

→ Go to Phase 4: Verify Recovery

Cause C — S3 / Object Storage Issues

Confirm all match:

be.log shows 503 reduce request rate from S3 endpoint
OR S3 connection pool exhaustion errors
OR SSL handshake failure to private OSS

This is the start of the file. Read the full SKILL.md on GitHub.

Security

Security scanPassed, 100 of 100Pattern checks on the skill's files.
Model reviewNot runA model review is scheduled for the next rescan.

See the full security report

Quality evaluation

Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

SKILL.md in the repositoryCheckingdata-lake/SKILL.md

Popularity

7-day trend score: 0

Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

Claude Code
Cursor
GitHub Copilot
Windsurf
Codex
Gemini CLI
Cline
Amp
Roo Code
Goose
Continue
Kilo Code
Junie
Trae
OpenHands
Qwen Code
AI Agent
Tabnine Chat
Sourcegraph Cody
Amazon Q
Firebase Genkit
Sweep
Devin
Phind
Mentat
Claude Desktop
Aider
GPT Pilot
Plandex
bolt.new
v0
Replit Agent
Anse
TypingMind
BetterChatGPT
LibreChat
Warp AI
Pieces
Fig AI

Embed badge

[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/starrocks/starrocks-debug-skills/data-lake/badge)](https://verified-skill.com/skills/starrocks/starrocks-debug-skills/data-lake)

Something wrong with this skill or its page? Tell us.

Install

SKILL.md

Data Lake / External Table Troubleshooting

Metric Taxonomy — Read This First

FE log signals (fe.log)

BE log signals (be.log / be.WARNING)

Network / thread signals

How to retrieve signals

External Table Issue Reference

Phase 1 — Classify the Issue

Step 1.1 — Identify failure category

Phase 2 — Locate and Characterize

Step 2.1 — Confirm HMS connectivity

Step 2.2 — Confirm Kerberos state

Step 2.3 — Confirm S3 access

Step 2.4 — Characterize cache staleness

Phase 3 — Take Action by Cause

Cause A — HMS Connectivity / Metadata Slow

Cause B — Kerberos Authentication Failure

Cause C — S3 / Object Storage Issues

Security

Quality evaluation

Source

Popularity

Works with

FE log signals (`fe.log`)

BE log signals (`be.log` / `be.WARNING`)