Original Architecture v1

Overview

BILL is a plugin-based AI agent that maintains consistent personality across multiple platforms while keeping platform-specific conversations separate through a dual-layer memory architecture. The system supports multiple LLM providers via OpenRouter and includes image generation capabilities through GPT-4o.

Core Architecture

Enhanced Components

LLM Router

Intelligent routing between multiple LLM providers based on task requirements:

interface LLMTask {
  type: 'text' | 'code' | 'analysis' | 'creative';
  complexity: 'simple' | 'medium' | 'complex';
  platform: string;
  requiresVision?: boolean;
}

class LLMRouter {
  private providers: Map<string, LLMProvider>;
  private fallbackChain: string[];

  async selectProvider(task: LLMTask): Promise<LLMProvider> {
    // Route based on task requirements
    if (task.requiresVision) {
      return this.providers.get('gpt-4o')!;
    }
    
    if (task.type === 'code' && task.complexity === 'complex') {
      return this.providers.get('claude-3.5-sonnet')!;
    }
    
    if (task.complexity === 'simple') {
      return this.providers.get('gpt-4-turbo')!;
    }
    
    return this.providers.get('claude-3.5-sonnet')!; // Default
  }
}

Image Generation System

Handles image creation and processing using GPT-4o:

interface ImageRequest {
  prompt: string;
  style?: 'realistic' | 'artistic' | 'diagram' | 'meme';
  size?: '1024x1024' | '1792x1024' | '1024x1792';
  platform: string;
}

class ImageGenerator {
  async generateImage(request: ImageRequest): Promise<ImageResult> {
    const response = await this.gpt4o.images.generate({
      model: 'dall-e-3',
      prompt: this.enhancePrompt(request.prompt, request.style),
      size: request.size || '1024x1024',
      quality: 'standard',
      n: 1
    });

    const imageUrl = response.data[0].url;
    const storedUrl = await this.storeImage(imageUrl, request.platform);
    
    return {
      url: storedUrl,
      prompt: request.prompt,
      revisedPrompt: response.data[0].revised_prompt
    };
  }

  async analyzeImage(imageUrl: string, question?: string): Promise<string> {
    const response = await this.gpt4o.chat.completions.create({
      model: 'gpt-4o',
      messages: [{
        role: 'user',
        content: [
          { type: 'text', text: question || 'Describe this image' },
          { type: 'image_url', image_url: { url: imageUrl } }
        ]
      }]
    });

    return response.choices[0].message.content || '';
  }
}

Dual-Layer Memory Architecture

Memory Isolation Strategy

The system implements a dual-layer memory approach to prevent context bleeding while enabling knowledge sharing:

Platform-Specific Memory

• Purpose: Store conversation history and context per platform

• Scope: Isolated to individual platforms (Twitter, Telegram)

• Storage: Supabase tables with platform-specific schemas

• Access Pattern: Recent conversation retrieval, thread context

• Image Support: Store image URLs and analysis results

Shared Memory

• Purpose: Cross-platform knowledge base and semantic search

• Scope: Available to all platforms

• Storage: Pinecone vector database with embeddings

• Access Pattern: Semantic similarity search, fact retrieval

• Image Support: Store image descriptions and generated content metadata

Enhanced Memory Flow

Database Configuration

Supabase Schema Design

Platform-Specific Tables with Image Support

-- Twitter conversations with image support
CREATE TABLE twitter_conversations (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  message_id TEXT UNIQUE NOT NULL,
  author_id TEXT NOT NULL,
  thread_id TEXT,
  content TEXT NOT NULL,
  response TEXT,
  image_urls TEXT[] DEFAULT '{}',
  generated_images TEXT[] DEFAULT '{}',
  llm_provider TEXT,
  timestamp TIMESTAMPTZ NOT NULL,
  metadata JSONB DEFAULT '{}',
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Telegram conversations with image support
CREATE TABLE telegram_conversations (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  message_id TEXT UNIQUE NOT NULL,
  chat_id TEXT NOT NULL,
  user_id TEXT NOT NULL,
  content TEXT NOT NULL,
  response TEXT,
  image_urls TEXT[] DEFAULT '{}',
  generated_images TEXT[] DEFAULT '{}',
  llm_provider TEXT,
  timestamp TIMESTAMPTZ NOT NULL,
  metadata JSONB DEFAULT '{}',
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Shared knowledge base
CREATE TABLE shared_knowledge (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  content TEXT NOT NULL,
  category TEXT NOT NULL,
  tags TEXT[] DEFAULT '{}',
  importance INTEGER DEFAULT 5,
  source_platform TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- User profiles (cross-platform)
CREATE TABLE user_profiles (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  platform TEXT NOT NULL,
  platform_user_id TEXT NOT NULL,
  username TEXT,
  interaction_count INTEGER DEFAULT 0,
  first_seen TIMESTAMPTZ DEFAULT NOW(),
  last_seen TIMESTAMPTZ DEFAULT NOW(),
  preferences JSONB DEFAULT '{}',
  UNIQUE(platform, platform_user_id)
);

-- Image generation logs
CREATE TABLE image_generations (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  platform TEXT NOT NULL,
  message_id TEXT NOT NULL,
  prompt TEXT NOT NULL,
  revised_prompt TEXT,
  image_url TEXT NOT NULL,
  style TEXT,
  size TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- LLM usage tracking
CREATE TABLE llm_usage (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  provider TEXT NOT NULL,
  model TEXT NOT NULL,
  platform TEXT NOT NULL,
  task_type TEXT NOT NULL,
  tokens_used INTEGER,
  cost_usd DECIMAL(10,6),
  response_time_ms INTEGER,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

Indexes for Performance

-- Platform conversation indexes
CREATE INDEX idx_twitter_conversations_author_id ON twitter_conversations(author_id);
CREATE INDEX idx_twitter_conversations_thread_id ON twitter_conversations(thread_id);
CREATE INDEX idx_twitter_conversations_timestamp ON twitter_conversations(timestamp DESC);

CREATE INDEX idx_telegram_conversations_chat_id ON telegram_conversations(chat_id);
CREATE INDEX idx_telegram_conversations_user_id ON telegram_conversations(user_id);
CREATE INDEX idx_telegram_conversations_timestamp ON telegram_conversations(timestamp DESC);

-- Shared knowledge indexes
CREATE INDEX idx_shared_knowledge_category ON shared_knowledge(category);
CREATE INDEX idx_shared_knowledge_tags ON shared_knowledge USING GIN(tags);
CREATE INDEX idx_shared_knowledge_created_at ON shared_knowledge(created_at DESC);

CREATE INDEX idx_user_profiles_platform_user ON user_profiles(platform, platform_user_id);

Pinecone Vector Database Configuration

Index Structure

• Index Name: bill-agent

• Dimensions: 1536 (OpenAI text-embedding-ada-002)

• Metric: Cosine similarity

• Pod Type: s1.x1 (starter)

Namespace Organization

enum VectorNamespace {
  TWITTER = 'twitter',
  TELEGRAM = 'telegram', 
  SHARED_KNOWLEDGE = 'shared',
  CHARACTER_EXAMPLES = 'examples'
}

Metadata Schema

interface VectorMetadata {
  platform: string;
  messageId: string;
  authorId: string;
  timestamp: string;
  content: string;
  category?: string;
  importance?: number;
}

LLM Provider Configuration

OpenRouter Integration

interface OpenRouterConfig {
  apiKey: string;
  baseUrl: 'https://openrouter.ai/api/v1';
  models: {
    'claude-3.5-sonnet': {
      id: 'anthropic/claude-3.5-sonnet';
      costPer1kTokens: 0.003;
      maxTokens: 200000;
      strengths: ['reasoning', 'code', 'analysis'];
    };
    'gpt-4-turbo': {
      id: 'openai/gpt-4-turbo';
      costPer1kTokens: 0.01;
      maxTokens: 128000;
      strengths: ['general', 'creative'];
    };
    'llama-3.1-70b': {
      id: 'meta-llama/llama-3.1-70b-instruct';
      costPer1kTokens: 0.0004;
      maxTokens: 131072;
      strengths: ['cost-effective', 'general'];
    };
  };
}

Provider Selection Logic

class LLMRouter {
  selectProvider(task: LLMTask, context: MessageContext): string {
    // Vision tasks always use GPT-4o
    if (task.requiresVision) {
      return 'gpt-4o';
    }

    // Complex coding tasks use Claude
    if (task.type === 'code' && task.complexity === 'complex') {
      return 'claude-3.5-sonnet';
    }

    // High-volume simple tasks use Llama for cost efficiency
    if (task.complexity === 'simple' && this.isHighVolumeUser(context.authorId)) {
      return 'llama-3.1-70b';
    }

    // Creative tasks prefer GPT-4
    if (task.type === 'creative') {
      return 'gpt-4-turbo';
    }

    // Default to Claude for balanced performance
    return 'claude-3.5-sonnet';
  }
}

Component Details

Agent Runtime

The core processing engine that coordinates all system components:

class AgentRuntime {
  private character: Character;
  private memoryManager: MemoryManager;
  private llmRouter: LLMRouter;
  private imageGenerator: ImageGenerator;
  private plugins: Map<string, IPlugin>;

  async processMessage(message: Message): Promise<AgentResponse> {
    // 1. Analyze any images in the message
    let imageAnalysis = '';
    if (message.metadata.imageUrls?.length > 0) {
      imageAnalysis = await this.imageGenerator.analyzeImage(
        message.metadata.imageUrls[0], 
        "Describe this image"
      );
    }

    // 2. Retrieve platform-specific context
    const platformContext = await this.memoryManager
      .getPlatformMemory(message.platform)
      .getRelevantContext(message);

    // 3. Search shared knowledge
    const sharedContext = await this.memoryManager
      .getSharedMemory()
      .searchKnowledge(message.content);

    // 4. Analyze task and select optimal LLM
    const task = this.analyzeTask(message, imageAnalysis);
    const provider = await this.llmRouter.selectProvider(task);

    // 5. Build complete context
    const context = this.buildContext(message, platformContext, sharedContext, imageAnalysis);

    // 6. Generate response using selected LLM
    const response = await provider.complete(context);

    // 7. Check if image generation is needed
    let generatedImage: ImageResult | null = null;
    if (this.shouldGenerateImage(response, message.platform)) {
      const imagePrompt = this.extractImagePrompt(response);
      generatedImage = await this.imageGenerator.generateImage({
        prompt: imagePrompt,
        platform: message.platform
      });
    }

    // 8. Store interaction in both memories
    await Promise.all([
      this.memoryManager.getPlatformMemory(message.platform)
        .storeInteraction(message, response),
      this.memoryManager.getSharedMemory()
        .storeInteraction(message, response)
    ]);

    return { 
      content: response, 
      shouldReply: true,
      imageUrl: generatedImage?.url,
      metadata: {
        llmProvider: provider.name,
        hasGeneratedImage: !!generatedImage,
        hasAnalyzedImage: !!imageAnalysis
      }
    };
  }

  private analyzeTask(message: Message, imageAnalysis: string): LLMTask {
    const content = `${message.content} ${imageAnalysis}`.toLowerCase();
    
    if (imageAnalysis) {
      return { 
        type: 'analysis', 
        complexity: 'medium', 
        platform: message.platform, 
        requiresVision: true 
      };
    }
    
    if (content.includes('code') || content.includes('programming')) {
      return { type: 'code', complexity: 'complex', platform: message.platform };
    }
    
    if (content.includes('create') || content.includes('generate')) {
      return { type: 'creative', complexity: 'medium', platform: message.platform };
    }
    
    return { type: 'text', complexity: 'simple', platform: message.platform };
  }

  private shouldGenerateImage(response: string, platform: string): boolean {
    const imageKeywords = ['create image', 'generate image', 'make picture', 'draw', 'visualize'];
    return imageKeywords.some(keyword => response.toLowerCase().includes(keyword));
  }

  private extractImagePrompt(response: string): string {
    const match = response.match(/(?:create|generate|make).*?image.*?(?:of|showing|with)?\s*([^.!?]+)/i);
    return match ? match[1].trim() : 'A helpful illustration';
  }
}

Memory Manager

Coordinates access to both platform-specific and shared memory systems:

class MemoryManager {
  private platformMemories: Map<string, PlatformMemory>;
  private sharedMemory: SharedMemory;

  constructor(
    private supabase: SupabaseClient,
    private pinecone: PineconeClient
  ) {
    this.platformMemories = new Map([
      ['twitter', new TwitterMemory(supabase, pinecone)],
      ['telegram', new TelegramMemory(supabase, pinecone)]
    ]);
    this.sharedMemory = new SharedMemory(supabase, pinecone);
  }

  getPlatformMemory(platform: string): PlatformMemory {
    return this.platformMemories.get(platform)!;
  }

  getSharedMemory(): SharedMemory {
    return this.sharedMemory;
  }
}

Context Building Strategy

The system builds rich context by combining multiple sources:

1. Character System Prompt: Base personality and expertise

2. Platform-Specific History: Recent conversation in the same thread/chat

3. Platform Memory Search: Relevant past interactions on the platform

4. Shared Knowledge: Cross-platform facts and learned information

5. User Profile: Known preferences and interaction patterns

class ContextBuilder {
  buildContext(
    message: Message,
    platformContext: PlatformContext,
    sharedContext: SharedContext
  ): string {
    const sections = [
      this.character.getSystemPrompt(message.platform),
      this.formatPlatformContext(platformContext),
      this.formatSharedContext(sharedContext),
      this.formatCurrentMessage(message)
    ];

    return sections.filter(Boolean).join('\n\n');
  }
}

Data Flow Patterns

Message Processing Pipeline

1. Reception: Platform plugin receives raw message

2. Image Analysis: Analyze any attached images using GPT-4o vision

3. Transformation: Convert to common Message interface

4. Context Retrieval: Parallel fetch from platform and shared memory

5. Task Analysis: Determine message type, complexity, and requirements

6. LLM Selection: Route to optimal provider (Claude/GPT-4/Llama) based on task

7. Context Assembly: Combine all context sources with image analysis

8. Response Generation: Generate response using selected LLM provider

9. Image Generation: Create images if response indicates need

10. Storage: Store interaction in both memory layers with metadata

11. Response Formatting: Platform-specific formatting with image attachment

12. Delivery: Send response via platform API

Memory Storage Strategy

• Immediate Storage: All interactions stored in platform-specific tables

• Embedding Generation: Async generation of embeddings for vector storage

• Batch Processing: Vector upserts batched for efficiency

• Knowledge Extraction: Important facts extracted to shared knowledge

Technology Stack

• Runtime: Node.js with TypeScript

• Package Manager: Bun

• Primary Database: Supabase (PostgreSQL)

• Vector Database: Pinecone

• Caching: Redis (for session management)

• LLM Providers:

  • OpenRouter (Claude 3.5, GPT-4, Llama 3.1)

  • OpenAI Direct (GPT-4o for vision/images)

• Image Generation: DALL-E 3 via GPT-4o

• Image Storage: Supabase Storage

• Deployment: Railway/Render

MVP Scope

Included:

• Dual-layer memory architecture

• Multiple LLM providers via OpenRouter

• Image generation and analysis via GPT-4o

• Twitter mentions and replies with image support

• Telegram bot with image capabilities

• Character system with platform adaptations

• Vector-based semantic search

• Basic user profiling

• Cost tracking and optimization

Not Included:

• Load balancing

• CDN (images served via Supabase Storage)

• Advanced analytics dashboard

• Auto-scaling infrastructure

• Real-time image processing

• Video generation

Cost Optimization Strategy

LLM Cost Management

• Intelligent Routing: Use cheaper models for simple tasks

• Caching: Cache similar responses to reduce API calls

• Rate Limiting: Prevent abuse and control costs

• Usage Tracking: Monitor costs per platform and user

Image Generation Limits

• Daily Limits: Reasonable limits per user/platform

• Quality Settings: Use 'standard' quality for cost efficiency

• Size Optimization: Default to 1024x1024 for most use cases

• Prompt Enhancement: Improve prompts to reduce regeneration needs