twofourlabs/agent-docs/ai-integrated-api-backend

User Request (Chat Message)
    ↓
API Endpoint (validate input, check session)
    ↓
Session Manager (create/resume session, lock pricing)
    ↓
External API Layer (if needed: fetch astrology, weather, etc.)
    ├─→ Cache Check (multi-layer: instance → distributed → database)
    ├─→ External API Call (if cache miss)
    └─→ Data Transformation Pipeline (validate → normalize → enrich → format)
    ↓
Prompt Builder
    ├─→ Load Bot LLM Config (model, provider, temperature, metadata)
    ├─→ Render Prompt Template (replace {{variables}} with actual values)
    ├─→ Assemble Context (user data + external data + history)
    └─→ Build Final Prompt (YAML or JSON structured)
    ↓
LLM Gateway Manager
    ├─→ Get/Generate JWT Token (cached)
    ├─→ Build Fallback Chain (provider-specific + universal)
    └─→ POST to Internal Gateway (/v1/generate/)
    ↓
Internal LLM Gateway (separate service)
    ├─→ Validate JWT
    ├─→ Route to Primary Provider (OpenAI, Anthropic, Gemini, Bedrock, etc.)
    ├─→ On Failure: Try Fallback 1
    ├─→ On Failure: Try Fallback 2
    └─→ On Failure: Universal Fallback (Gemini-Flash)
    ↓
Response Processing
    ├─→ Parse LLM Response
    ├─→ Validate Output
    └─→ Store in Database
    ↓
Billing (if session-based)
    ├─→ Calculate Units (tokens, messages, minutes)
    ├─→ Check Wallet Balance
    ├─→ Deduct Amount
    └─→ Log Transaction
    ↓
Save to Database (conversation history)
    ↓
Return to User (API response)

Request → Load BotLLMConfig → Determine Primary Provider
    ↓
Primary: Gemini 2.5-Pro
    ├─→ Fallback 1: Gemini 2.5-Flash
    ├─→ Fallback 2: Gemini 2.0-Flash
    └─→ Universal Fallback: Gemini 2.0-Flash
 
Primary: AWS Bedrock Deepseek-v3 (region: ap-south-1)
    ├─→ Fallback 1: Byteplus Deepseek-v3
    ├─→ Fallback 2: Gemini 2.0-Flash (region-agnostic)
    └─→ Universal Fallback: Gemini 2.0-Flash
 
Primary: Byteplus
    ├─→ Fallback 1: Gemini 2.0-Flash
    └─→ Universal Fallback: Gemini 2.0-Flash

class Bot(models.Model):
    uuid = models.UUIDField(default=uuid4, editable=False, unique=True)
    user = models.ForeignKey("users.UserProfile", on_delete=models.CASCADE, related_name="bots")
    # Identity
    name = models.CharField(max_length=255)
    slug = models.SlugField(max_length=255, unique=True, blank=True, db_index=True)
    description = models.TextField(blank=True)
    avatar = models.CharField(max_length=255, null=True, blank=True)
    # Status and Type
    status = models.CharField(max_length=20, choices=BotStatus.choices, default=BotStatus.DRAFT, db_index=True)
    bot_type = models.CharField(max_length=30, choices=BotType.choices, default=BotType.COMPANION)
    # Configuration
    rank = models.PositiveIntegerField(default=0, db_index=True)
    metadata = models.JSONField(default=dict, blank=True)  # Store required_fields, pricing, etc.
    # Timestamps
    launched_on = models.DateTimeField(null=True, blank=True, db_index=True)
    created_on = models.DateTimeField(auto_now_add=True)
    updated_on = models.DateTimeField(auto_now=True)
    deleted_at = models.DateTimeField(null=True, blank=True, db_index=True)

class BotLLMConfig(models.Model):
    bot = models.ForeignKey(Bot, on_delete=models.CASCADE, related_name="llm_configs")
    # LLM Configuration
    prompt = models.TextField(blank=True)  # Can be template with {{variables}}
    model_name = models.CharField(max_length=100)  # e.g., "gpt-4", "claude-3-sonnet", "gemini-2.5-pro"
    llm_provider = models.CharField(max_length=50, choices=BotLLMProvider.choices)  # openai, anthropic, gemini, bedrock, etc.
    # Generation Parameters
    temperature = models.FloatField(default=0.7)
    top_p = models.FloatField(default=0.95)
    top_k = models.IntegerField(null=True, blank=True)
    max_output_tokens = models.PositiveIntegerField(default=1024)
    # Activation
    is_active = models.BooleanField(default=False)  # Only one active config per bot
    # Additional Configuration
    metadata = models.JSONField(default=dict, blank=True)  # Store region_name, tools, etc.
    created_on = models.DateTimeField(auto_now_add=True)
    updated_on = models.DateTimeField(auto_now=True)
    class Meta:
        db_table = "llm_configurations"
        ordering = ["-is_active", "created_on"]
    def save(self, *args, **kwargs):
        # Ensure only one active config per bot
        if self.is_active:
            BotLLMConfig.objects.filter(bot=self.bot, is_active=True).exclude(pk=self.pk).update(is_active=False)
        super().save(*args, **kwargs)

class BotLLMProvider(models.TextChoices):
    OpenAI = "openai", _("OpenAI")
    GEMINI = "gemini", _("Gemini")
    ANTHROPIC = "anthropic", _("Anthropic")
    GROQ = "groq", _("Groq")
    AI_SDK = "ai-sdk", _("AI-SDK")
    BEDROCK = "bedrock", _("Bedrock")
    BYTEPLUS = "byteplus", _("Byteplus")
    OPENROUTER = "openrouter", _("OpenRouter")

def get_fallback_routing(provider: str, model_name: str, system_instruction: str, llm_metadata: dict = None):
    """
    Ensures system_instruction and region_name are added properly to fallback chain.
    Args:
        provider: Primary provider (gemini, bedrock, byteplus, etc.)
        model_name: Model identifier (gemini-2.5-pro, deepseek.v3-v1:0, etc.)
        system_instruction: System prompt to pass to all fallbacks
        llm_metadata: Additional metadata including region_name for Bedrock
    Returns:
        List of fallback configurations, each containing:
        - provider: str
        - model: str
        - retry: int (0 = no retry, >0 = retry count)
        - system_instruction: str
        - region_name: str (for Bedrock only)
    """
    provider = (provider or "").lower()
    fallbacks = []