skills-il/developer-tools/israeli-chatbot-analytics

Israeli Chatbot Analytics

Analyze and optimize Hebrew chatbot performance. This skill covers conversation flow analytics, Hebrew-specific sentiment analysis, drop-off detection, user satisfaction scoring, A/B testing for Hebrew response variants, intent recognition accuracy tracking, anomaly alerting, and reporting dashboards. Use it to understand whether your Hebrew chatbot is actually helping users and where to focus improvements.

Instructions

Step 1: Collect and Structure Conversation Logs

Before analyzing, ensure conversation data is structured consistently. Each conversation session should include:

# Standard conversation log schema
conversation_log = {
    "session_id": "uuid-string",
    "user_id": "anonymous-or-identified",
    "channel": "whatsapp|telegram|web|app",
    "language": "he",           # Primary language detected
    "started_at": "ISO-8601",
    "ended_at": "ISO-8601",
    "messages": [
        {
            "timestamp": "ISO-8601",
            "sender": "user|bot|agent",     # agent = a human who took over
            "text": "שלום, אני צריך עזרה",
            "intent": "greeting",           # Detected intent
            "intent_confidence": 0.92,       # Model confidence
            "entities": [],                  # Extracted entities
            "response_time_ms": 340,         # Bot response latency
        }
    ],
    "outcome": "resolved|escalated|abandoned|unknown",
    "satisfaction_score": null,   # CSAT score if collected
    "metadata": {
        "bot_version": "2.1.0",
        "ab_variant": "formal_he",
    }
}

Two fields in that schema carry every headline number in this skill, and neither one arrives in an export. Define both explicitly before you compute anything.

Deriving `outcome` (do this first). Completion, escalation, abandonment, drop-off, the satisfaction composite and cost per resolved conversation all key off this label. Platform exports do not contain it: the Dialogflow CX parser writes unknown, and a WhatsApp webhook stream has no outcome concept at all. Run the analyzer without deriving it and you get a dashboard of zeros. Write the rule down and version it:

Never take an outcome the bot writes about itself ("flow completed") as an outcome. That is the bot's own belief, and treating it as ground truth is the same error as reading high_confidence_rate as intent_accuracy (Step 8).

Defining the session boundary. The Dialogflow CX BigQuery export has one row per turn: take the session from conversation_name and its start and end from the first and last request_time. On WhatsApp Cloud API, the dominant Israeli channel, there is no session object at all, only a flat webhook stream. You must cut it yourself with an inactivity timeout, and the timeout you pick mechanically sets your abandonment rate and average handle time. A user who replies the next morning is one long resolved session under a 24-hour cut and one abandoned session plus a new one under a 30-minute cut. Both numbers get quoted to management. Pick one (30 minutes is a reasonable default for support), write it into the log-normalization step, and never change it without restating the history. The WhatsApp 24-hour customer-service window and the free-entry-point window (up to 7 days) are BILLING windows: do not reuse them as conversation boundaries.

If your platform exports a different shape, normalize it first. The export method and format for Conversational Agents (Dialogflow CX), Rasa Pro / CALM, Rasa Open Source, Botpress, WhatsApp Cloud API, ManyChat and custom bots, with the Hebrew language-code caveat, are in references/platform-integrations.md.

Step 2: Conversation Flow Analysis

Analyze session-level metrics to understand overall chatbot health:

compute_flow_metrics(conversations) makes one pass, counting sessions per outcome (resolved / escalated / abandoned) and collecting message counts and (ended_at - started_at).total_seconds(). Rates are count / total_sessions, with unknown sessions left in the denominator; also report the average session length and the median duration.

Industry benchmarks for support chatbots (apply with judgment to Hebrew bots):

These are general support-chatbot targets with no Israeli sample behind them. Use them as a starting bar and replace each row with your own baseline after four weeks of data.

Measure what happens after an escalation. escalation_rate on its own rewards a bot that is hard to escape. For every escalated session also track the queue wait from handoff to the first human reply, whether the agent received the transcript (an agent whose first message re-asks for an ID or order number the user already gave did not), the human resolution rate, and the bounce-back rate (handed back to the bot, or escalated again within 24 hours). A falling escalation rate alongside rising bounce-backs or repeat contacts is a bot that deflects, not one that resolves.

Step 3: Drop-off Point Detection

Identify where users abandon. This reveals UX problems, confusing prompts, or missing capabilities:

detect_drop_off_points(conversations) filters to outcome == "abandoned" and returns three Counter.most_common slices: drop-off by conversation depth (message count), by active intent at drop (walking from the tail to the first message that carries an intent), and by last bot message (first 100 chars, walking from the tail for the last sender == "bot").

Keep fallback in the by-intent bucket. Fallback-then-abandon is the most common real drop-off pattern, so filtering it out empties the report for exactly the sessions you most need to see.

detect_conversation_loops(conversations, threshold=3) flags sessions where the bot repeats the same text ≥ threshold times in a row by scanning the bot-message stream and tracking a consecutive-repeat counter; emit {session_id, repeated_message, repeat_count, total_messages} for each looped session.

Step 4: Hebrew Sentiment Analysis

Hebrew sentiment analysis requires special handling due to morphological complexity, negation patterns, and slang. Use DictaBERT (encoder, classification) for production sentiment scoring, AlephBERT (onlplab/alephbert-base from the ONLP Lab at Bar-Ilan University) as an alternative encoder baseline, or a lexicon-based approach for lightweight analysis. When you need one model to classify sentiment AND summarize the conversation in Hebrew prose for the ops team, use Dicta-LM 3.0 (February 2026), the current Hebrew model family from Dicta, with a 65k native context: 24B (adapted from Mistral-Small-3.1), 12B (from NVIDIA Nemotron Nano V2) and 1.7B (from Qwen3-1.7B). The chat variants are not named alike, so copy the exact repo id: DictaLM-3.0-1.7B-Instruct and -1.7B-Thinking, DictaLM-3.0-Nemotron-12B-Instruct, and for 24B only DictaLM-3.0-24B-Thinking (there is no 24B Instruct). The 1.7B Instruct is the practical choice for per-message classification at volume; the 24B Thinking for offline summarization. DictaLM 2.0 (2024, 7B, based on Mistral-7B) is the previous generation and still works, but new builds should start on 3.0.

Using DictaBERT (recommended for production). The simplest path is pipeline("sentiment-analysis", model="dicta-il/dictabert-sentiment"), which resolves the label names off the model config for you. If you drive the model directly for batching control, wrap AutoTokenizer + AutoModelForSequenceClassification, tokenize with truncation=True, max_length=512, padding=True, softmax the logits, and map each row through id2label.

CRITICAL: read the label names from model.config.id2label, never from a hardcoded list. Label order is model metadata, not a convention, and it is not alphabetical. For dicta-il/dictabert-sentiment it is {0: "Positive", 1: "Negative", 2: "Neutral"}. A version of this skill that hardcoded ["negative","neutral","positive"] was wrong at every index and reported every frustration spike as a satisfaction spike.

Full code, batching wrapper and the lexicon fallback: references/hebrew-sentiment-guide.md.

Hebrew-specific sentiment challenges. "לא רע" reads mildly positive in Israeli usage; sarcasm ("יופי, בדיוק מה שחיכיתי לו") is common and deeply negative; slang moves fast ("וואלה" is context-dependent); and users mix scripts in one message ("ה-support שלכם גרוע"). Negation is left to DictaBERT: the lexicon baseline in references/hebrew-sentiment-guide.md (slang lexicon, whole-token lookup) leaves negated messages unscored.

Step 5: Intent Recognition Accuracy Tracking

Track how well your chatbot understands user requests over time:

Build IntentAccuracyTracker to log (predicted, actual, confidence, timestamp) per prediction and expose:

• confusion_matrix(): 2D {actual: {predicted: count}} over the sorted intent universe.
• misclassification_report(min_count=5): top (actual, predicted) pairs where predicted != actual.
• low_confidence_intents(threshold=0.6): intents whose mean confidence is below threshold, with sample_count and below_threshold_pct.
• accuracy_trend(): daily {date, accuracy, sample_count} series for plotting (bucket by timestamp[:10]).

How to get ground truth labels. Sample 100-200 conversations a week and have Hebrew-speaking annotators label the actual intent; this is the gold standard. Supplement it with escalation signals (a user correcting the bot, "לא, התכוונתי ל...", or asking for a human right after a misunderstanding, flags the prior intent as wrong) and post-chat surveys asking "Did the bot understand what you needed?" correlated with the detected intent.

Step 6: User Satisfaction Measurement

Combine multiple signals to build a satisfaction score:

Build a SatisfactionSignals dataclass carrying direct feedback (csat_score 1-5, thumbs_rating "up"/"down"), behavioural signals (session_resolved, escalated_to_human, abandoned, repeated_fallbacks, loop_detected) and sentiment signals (final_sentiment positive/neutral/negative, sentiment_trend improving/stable/declining).

Its composite_score() -> float returns 0.0-1.0. If csat_score is present, return (csat_score - 1) / 4 directly and stop. Otherwise start at 0.5 (or 0.8 / 0.2 for thumbs up / down), then apply: +0.15 resolved, -0.1 escalated, -0.2 abandoned, -0.15 if repeated_fallbacks > 2, -0.2 loop detected, +/-0.1-0.15 for final_sentiment, +/-0.05-0.1 for sentiment_trend; clamp to [0, 1].

Provide collect_post_chat_survey_he() that returns a Hebrew post-chat survey: title "נשמח לשמוע מה חשבת", a 1-5 rating on "עד כמה הצ'אטבוט עזר לך?", a yes/no on "האם הצ'אטבוט הבין את מה שרצית?", and an optional open "רוצה לשתף עוד משהו?" field. Use "שלח משוב" as the submit label.

Step 7: A/B Testing for Hebrew Response Variants

Test different phrasings, formality levels, and gender handling strategies:

Build HebrewABTestManager with three responsibilities: