Building Vidya: An Ultra-Fast Bilingual Voice AI Tutor. Over the past 10 days, as part of the #VoiceForBharat 10 Days of Voice Agents Challenge , I built Vidya — a real-time, bilingual (English/Hindi) AI Voice Tutor designed to make learning interactive, accessible, and human-like for students across India. In this post, I’ll share the story of how Vidya came to life, dive into the architecture behind ultra-low latency voice agents, highlight the key features built over the 10 days, discuss the toughest challenges faced, and walk you through building your own production-ready voice agent using Murf Falcon TTS , LiveKit Agents , Deepgram STT , and Google Gemini . 1. Why Voice? Meet Vidya 🇮🇳 In India, text-based educational platforms often face a steep digital literacy and language barrier. Millions of learners feel intimidated by typing long queries or struggling through English-only user interfaces. Voice unlocks immediate, natural, and hands-free learning — allowing students to speak naturally in English, Hindi, or code-mixed Hinglish . Vidya serves as a personal AI learning companion: Problem Solved: Overcomes literacy and language barriers by providing empathetic, spoken literacy and science lessons. Track & Target Audience: Education & Accessibility for students, young learners, and non-native English speakers across India. Why Voice is Essential: Voice removes UI friction. Learners can simply talk to Vidya, attempt spoken exercises, ask questions, and receive instant, warm, spoken feedback in a natural Indian accent. 2. Key Features Built Across the 10-Day Journey Over 10 intensive days, Vidya grew from a basic echo bot into a multi-agent system equipped with tools, memory, telephony, analytics, and safety guardrails: 🎙️ 1. Ultra-Low Latency Indian Voice Powered by Murf Falcon Using Murf Falcon TTS ( livekit-murf ), Vidya speaks with a natural, conversational Indian voice ( Anisha ). Streaming TTS with sentence tokenization ( min_sentence_len=2 ) and text pacing delivers speech chunks with sub-second latency, giving the agent a human-like flow. 🇮🇳 2. Code-Mixed (Hinglish) & Bilingual Support Powered by Deepgram Nova-3 STT ( language="multi" ) and Google Gemini , Vidya fluently handles English, Hindi in Devanagari script, and code-mixed Hinglish phrases (e.g., "Namaste! Aaj hum beginner reading practice karenge." ). 🧠 3. Personalized Memory for Returning Users Vidya remembers returning students! Using a persistent profile store ( user_store.py ), Vidya recalls the user's name, preferred language, current learning level, and last interaction date, greeting them warmly: "Namaste Aarav, welcome back! You were working on beginner exercises. Last seen on August 14." 🛠️ 4. Interactive Learning Tools & Scraper Vidya is equipped with specialized function tools: fetch_next_exercise : Retrieves level-appropriate practice prompts tagged with data freshness timestamps ( last_updated ). score_spoken_answer : Evaluates spoken pronunciations and answers on a 0–100 scale. award_learning_star : Awards virtual gold stars (🌟) to keep learners motivated. scrape_website : Fetches live web pages in real-time ( web_scraper.py ) for live context extraction. 📞 5. Outbound Telephony (SIP Calls) Integrated with LiveKit's SIP Trunking ( telephony/outbound/dial.py ), Vidya can initiate active outbound phone calls to learners' mobile phones for daily study check-ins and practice sessions. 🛡️ 6. Human Escalation Engine with Explicit Consent If a learner is stuck, frustrated, or requests a human teacher, create_escalation logs an escalation ticket ( ESC-12345 ) and alerts support staff. Vidya follows strict privacy guardrails — asking for explicit permission before saving any personal details or submitting tickets. 📊 7. Call Analytics Dashboard Session outcomes are tracked in call_store.py — logging call duration, agent type (browser vs. telephony), completion status ( successful / failed ), and success reasons ( exercise_scored , star_awarded , escalated_to_human ). 🔄 8. Dynamic Multi-Agent Handoffs When a student asks a physics question (e.g., "Why does an apple fall from a tree?" ), Vidya seamlessly hands off the conversation to Dr. Homi (Physics Specialist) using LiveKit's context.session.update_agent() . When physics practice ends, Dr. Homi hands the student back to Vidya for reading practice! 3. The Tough Parts: Lessons Learned Building a real-time voice agent isn't just about linking APIs together. Here are three major hurdles faced and solved: Challenge #1: Eliminating Awkward Conversational Latency The Problem: Combining STT, LLM, and TTS in sequence created a 2–3 second delay before the agent responded, ruining natural turn-taking. The Solution: We enabled streaming at every tier — Deepgram Nova-3 streaming STT, Gemini Flash Lite for fast first-token generation, LiveKit's preemptive generation, and Murf Falcon's low-latency streaming TTS with sentence boundary tokenization ( min_sentence_len=2 ). Challenge #2: Explicit Consent & Privacy in Spoken AI The Problem: Function tools automatically firing to record user profiles or create tickets without the user realizing. The Solution: We designed prompt-level guardrails requiring Vidya to explicitly ask out loud: "I can remember that for next time — may I save this?" or "May I share your details with a real teacher?" before invoking mutation tools. Challenge #3: Preserving Session State During Agent Handoffs The Problem: Switching from Vidya to Dr. Homi mid-call could cause audio glitches or lose participant state. The Solution: Leveraged LiveKit Agents SDK's native session.update_agent() combined with WebRTC data channel events ( agent_handoff ) to update the Next.js frontend UI live without dropping the WebRTC room session. 4. Architecture Overview & Code Highlights +------------------+ WebRTC Audio Stream +---------------------+ | | -------------------------> | Deepgram Nova-3 | | Learner / UI | | Streaming STT | | (Next.js App) | <------------------------- +----------+----------+ +--------+---------+ Real-time Audio Out | ^ v | RTC Data Channel +---------------------+ | (State & Handoffs) | Google Gemini LLM | | | (Flash Lite Model) | +-------------------------------------- +----------+----------+ | v +---------------------+ | Murf Falcon TTS | | (Streaming Indian) | +---------------------+ Code Snippet: LiveKit Pipeline with Murf Falcon TTS ( agent.py ) from livekit.agents import AgentSession , AgentServer , room_io from livekit.plugins import deepgram , google , murf , silero , noise_cancellation session = AgentSession ( stt = deepgram . STT ( model = " nova-3 " , language = " multi " ), llm = google . LLM ( model = " gemini-3.5-flash-lite " ), tts = murf . TTS ( voice = " Anisha " , # Murf Falcon Indian accent voice style = " Conversation " , tokenizer = tokenize . basic . SentenceTokenizer ( min_sentence_len = 2 ), text_pacing = True , ), turn_detection = MultilingualModel (), vad = ctx . proc . userdata [ " vad " ], preemptive_generation = True , ) Code Snippet: Dynamic Multi-Agent Handoff ( agent.py ) @function_tool async def transfer_to_physics_specialist ( self , context : RunContext , reason : str ) -> str : """ Hand off the conversation to Dr. Homi when user asks physics questions. """ logger . info ( " Handing off conversation to PhysicsSpecialist. Reason: %s " , reason ) specialist = PhysicsSpecialist () context . session . update_agent ( specialist ) # Notify Next.js frontend UI via WebRTC data channel payload = json . dumps ({ " type " : " agent_handoff " , " from_agent " : " Vidya (Literacy Tutor) " , " to_agent " : " Dr. Homi (Physics Specialist) " , " message " : " 🔄 Switched conversation to Physics Specialist (Dr. Homi) " }) await context . room . local_participant . publish_data ( payload = payload . encode ( " utf-8 " )) return " I will connect you to our physics specialist. " 5. How to Build Your Own Voice Agent (Quickstart) Want to build your own voice AI agent? You can clone and run our open-source repository in minutes! Step 1: Clone the Repository git clone https://github.com/hotokeAtlast/murf-livekit-starter.git cd murf-livekit-starter Step 2: Set Up Backend Environment Keys Copy backend/.env.example to backend/.env.local and fill in your keys: LIVEKIT_URL=wss://your-livekit-project.livekit.cloud LIVEKIT_API_KEY=your_key LIVEKIT_API_SECRET=your_secret MURF_API_KEY=your_murf_api_key DEEPGRAM_API_KEY=your_deepgram_api_key GOOGLE_API_KEY=your_google_gemini_api_key Step 3: Run the Python Backend cd backend uv sync uv run python src/agent.py dev Step 4: Run the Next.js Frontend In a new terminal: cd frontend pnpm install pnpm dev Open http://localhost:3000 , click Connect , and start talking to your voice agent! 6. Links & Repository 🐙 GitHub Repository: murf-livekit-starter on GitHub 🔊 TTS Engine: Powered by Murf Falcon Streaming API ⚡ Voice Framework: Built with LiveKit Agents SDK