Don't type in English. Speak naturally in your own Indian language — Raay understands, plans, and operates your computer automatically.
Built on Gnani.ai's Sovereign Speech Stack (Gnani Prisma STT&Gnani Timbre TTS) for The Great Indian AI Internship Challenge (#GreatIndianAIInternshipChallenge #GnaniAI).
Most people in India don't think in English prompts. If you put a laptop in front of a kirana shopkeeper, our parents, or anyone who speaks Telugu, Hindi, or Tamil at home, the biggest barrier between them and the computer is the English keyboard.
Even when you try voice assistants or chatbots like ChatGPT and Gemini, two things break right away:
- They are stuck inside a browser tab. Ask a chatbot to open your local Excel grocery sheet and check today's UPI payments, or open MS Paint and draw a diagram, and it tells you it can't touch your desktop.
- They fail on how we actually talk. In real life, nobody speaks pure textbook Telugu or pure English—we mix them in the same sentence: "Rahul, grocery Excel file open cheyyi — indulo UPI payments enni?" or "Circle lopa unna red color ni yellow ki change cheyyi."
We built Raay so you don't have to type at all. It connects Gnani.ai's speech models (Prisma STT & Timbre TTS) and a pluggable LLM planner (Claude / GPT / Gemini / Ollama) directly to Windows 11 (UIA, Win32, and Excel COM). You just talk to your laptop in your own language, and it opens the apps and does the work in front of you.
| Real-World Desktop Challenge | Spoken Command (Telugu / Hinglish / Tamil) | Standard AI Chatbot | Raay (Powered by Gnani.ai) |
|---|---|---|---|
| 1. Live Excel Kirana/Grocery BI & GST | "రాహుల్, గ్రోసరీ ఓపెన్ చెయ్ — అందులో యుపిఐ పేమెంట్స్ ఎన్ని?" | ❌ Cannot open local Excel or compute sheet totals | ✅ Opens Excel + BI Charts & speaks exact UPI (₹2,478), Card (₹1,323), Cash (₹940) & GST (₹249) breakdown |
| 2. Multi-Object MS Paint Studio | "రాహుల్, పెయింట్ ఓపెన్ చెయ్ — డ్రా సర్కిల్ విత్ కలర్ రెడ్, దాని పక్కనే హార్ట్ సింబల్" | ❌ Cannot launch MS Paint or control canvas | ✅ Opens MS Paint via UIA, selects shapes & colour swatches, and draws pixel-verified objects live |
| 3. Sub-200ms Live Barge-In & Recolour | "చేంజ్ ద కలర్ ఆఫ్ సర్కిల్ టు ఎల్లో!" / "ఆపు! Undo చెయ్. Gold colour వాడు" | ❌ Cannot interrupt or undo live GUI actions mid-stroke | ✅ Interrupts in under 200ms, triggers Ctrl+Z undo, and recolours the target object in-place |
| 4. Cross-App Design-to-Code Export | "సేమ్ డిజైన్ కాన్సెప్ట్ ని విఎస్ కోడ్ లో కాపీ చేసి ప్రాజెక్ట్ లో సేవ్ చెయ్యి" | ❌ Generates static text snippet; cannot launch VS Code | ✅ Generates responsive HTML5/SVG + JSON blueprint and opens the project live in VS Code |
| 5. Local File Discovery & Summary | "డౌన్లోడ్స్ లో లేటెస్ట్ రిపోర్ట్ ఫైండ్ చేసి సమ్మరీ క్రియేట్ చేసి సేవ్ చెయ్యి" | ❌ No access to local Downloads directory or Notepad |
✅ Scans ~/Downloads, summarizes key findings in Telugu/English, and opens Notepad |
| INITIAL TEST SCORE | 5 Native-Language Desktop Benchmarks | 0 / 5 | 5 / 5 |
graph TB
classDef audio fill:#0284c7,stroke:#38bdf8,stroke-width:2px,color:#ffffff;
classDef gnani fill:#6d28d9,stroke:#a78bfa,stroke-width:2px,color:#ffffff;
classDef brain fill:#0f766e,stroke:#2dd4bf,stroke-width:2px,color:#ffffff;
classDef sec fill:#b91c1c,stroke:#f87171,stroke-width:2px,color:#ffffff;
classDef os fill:#1e293b,stroke:#94a3b8,stroke-width:2px,color:#ffffff;
subgraph Capture["1. Zero-Click Audio Capture & Presentation"]
Mic["Windows WinMM / WebAudio PCM16<br/>16 kHz Mono • 32ms Frames"]:::audio
Listener["Always-On Background Listener<br/>scripts/rahul_background_listener.py"]:::audio
Cockpit["3D Volumetric Web Cockpit<br/>http://127.0.0.1:8765"]:::audio
Speaker["Echo-Suppressed Speaker Output<br/>24 kHz WAV Playback"]:::audio
end
subgraph GnaniCloud["2. Gnani.ai Sovereign Speech Stack"]
Prisma["Gnani Prisma STT<br/>WSS /stt/v3/stream • Sub-200ms VAD<br/>POST /stt/v3 • Noise Cancellation & ITN"]:::gnani
Timbre["Gnani Timbre v3.0 & v2.5 TTS<br/>POST /api/v1/tts/sse • Streaming SSE<br/>10+ Indic Languages • Lakh/Crore Prosody"]:::gnani
end
subgraph VoiceCore["3. Pluggable LLM Reasoning & Conversational Planner"]
Adapters["Universal LLM Adapter Registry (src/providers/)<br/>Anthropic Claude • OpenAI GPT • Google Gemini • Local Ollama"]:::brain
FastStop["Sub-200ms Stop & Undo Barge-In Gate<br/>CancelToken + Emergency Input Release"]:::brain
Planner["Conversational Turn & DAG Task Planner<br/>Script Ratio Verifier & 3-Turn Memory"]:::brain
Cache["SHA-256 Deterministic TTS Cache<br/>Sub-1ms Instant Disk Replay"]:::brain
end
subgraph SecurityCore["4. Host-Side Zero-Trust Security Core"]
Policy["Deterministic Policy Engine<br/>ALLOW • DENY • REQUIRE_APPROVAL"]:::sec
Nonce["256-Bit CSPRNG Approval Broker<br/>Single-Use Nonce • 300s TTL"]:::sec
DPAPI["Windows DPAPI Hardware Vault<br/>CryptProtectData • Zero Plaintext Keys"]:::sec
Audit["SHA-256 Chained Audit Ledger<br/>Tamper-Evident JSONL"]:::sec
end
subgraph WindowsOS["5. Windows 11 Native Execution Controllers"]
Excel["ExcelGroceryController<br/>25-Row Kirana BI + Live COM Charts"]:::os
Paint["Windows11PaintController<br/>UIA Tree + HWND Focus Guard + SceneEditSpec"]:::os
Workflows["SutraShowpieceController<br/>VS Code Export • Downloads Summary • Browser"]:::os
end
Mic --> Listener
Mic --> Cockpit
Listener --> Prisma
Cockpit --> Prisma
Prisma --> FastStop
Prisma --> Planner
Adapters <--> Planner
Planner --> Policy
Policy -->|REQUIRE_APPROVAL| Nonce
Policy -->|ALLOW| Excel
Policy -->|ALLOW| Paint
Policy -->|ALLOW| Workflows
Excel --> Audit
Paint --> Audit
Workflows --> Audit
Planner --> Cache
Cache -->|Cache Miss| Timbre
Cache -->|Cache Hit| Speaker
Timbre --> Speaker
DPAPI -.->|In-Memory Key Hydration| Prisma
DPAPI -.->|In-Memory Key Hydration| Timbre
flowchart LR
A["Spoken Utterance<br/>Telugu / Hindi / Tamil / Hinglish"] --> B["Gnani Prisma STT<br/>WSS /stt/v3/stream"]
B --> C["Pluggable LLM Planner Adapter<br/>Claude • GPT • Gemini • Local Ollama"]
C --> D{"Wake-Word &<br/>Task DAG Router"}
D -->|Instant Wake| E["Pre-Warmed Greeting Cache<br/>Sub-1ms Response"]
D -->|Excel / Kirana BI| F["ExcelGroceryController<br/>Live COM Sheet + UPI/GST Math"]
D -->|MS Paint Studio| G["Windows11PaintController<br/>UIA Shapes + In-Place Recolour"]
D -->|Cross-App Workflows| H["VS Code Export • Downloads Summary<br/>Smart Browser • OS Timers"]
D -->|Conversational Query| I["ConversationalTurnPlanner<br/>Script Verification"]
E --> J["Gnani Timbre v3.0 TTS<br/>SSE Stream + SHA-256 Cache"]
F --> J
G --> J
H --> J
I --> J
sequenceDiagram
autonumber
actor User as User (Telugu / Hinglish)
participant Listener as Background Mic Listener
participant Prisma as Gnani Prisma STT (WSS)
participant Router as Voice Cockpit Router
participant OS as Windows 11 OS (Excel / Paint)
participant Timbre as Gnani Timbre v3.0 TTS
User->>Listener: "రాహుల్, గ్రోసరీ ఓపెన్ చెయ్"
Listener->>Prisma: Stream 16kHz PCM16 frames (min_silence_ms=280)
Prisma-->>Router: Final Transcript ("రాహుల్ గ్రోసరీ ఓపెన్ చెయ్", te-IN)
Router->>OS: Launch Excel Grocery BI Dashboard + Native Charts
OS-->>Router: COMPLETED (25 items • Grand Total Rs.4741 • GST Rs.249)
Router->>Timbre: Synthesize Telugu Summary (Shivani v3.0)
Timbre-->>User: Plays Spoken Telugu Summary
User->>Listener: "అందులో యుపిఐ పేమెంట్స్ ఎన్ని?"
Listener->>Prisma: Stream PCM16 frames
Prisma-->>Router: Final Transcript ("అందులో యుపిఐ పేమెంట్స్ ఎన్ని", te-IN)
Router->>OS: Compute Exact UPI / Card / Cash Totals
Router->>Timbre: "యుపిఐ ద్వారా 12 వస్తువులకు మొత్తం 2478 రూపాయలు..."
Timbre-->>User: Plays UPI, Card & Cash Breakdown
User->>Listener: "రాహుల్, పెయింట్ ఓపెన్ చెయ్ — డ్రా సర్కిల్ విత్ కలర్ రెడ్"
Listener->>Prisma: Stream PCM16 frames
Prisma-->>Router: Final Transcript ("డ్రా సర్కిల్ విత్ కలర్ రెడ్")
Router->>OS: Switch Focus to MS Paint + Draw Red Circle via UIA
User->>Listener: "చేంజ్ ద కలర్ ఆఫ్ సర్కిల్ టు ఎల్లో!"
Prisma-->>Router: Final Transcript ("చేంజ్ ద కలర్ ఆఫ్ సర్కిల్ టు ఎల్లో")
Router->>OS: SceneEditSpec(recolour, target=ellipse, new_fill=yellow)
OS-->>Router: Ctrl+Z Undo + Redraw Yellow Circle (RGB Verified)
Router->>Timbre: "పెయింట్ లో ellipse గీసి దానికి yellow రంగు వేశాను!"
Timbre-->>User: Plays Confirmation
- Gnani Prisma Speech-to-Text (
src/voice/gnani_stt.py):- Realtime WebSocket (
wss://api.vachana.ai/stt/v3/stream): Streams 16 kHz mono PCM16 frames (1,024bytes /32ms) with server-side Voice Activity Detection (x-vad-threshold,x-min-silence-ms,x-min-speech-ms,x-min-segment-ms), domain keyword biasing (x-bias-list,x-bias-score), and dynamic mid-streamupdate_settingsframes. - REST Decode (
POST https://api.vachana.ai/stt/v3): Supports telephonic noise cancellation (nc=true) and Inverse Text Normalization (format=transcribeforhi-INanden-IN). - 10+ Indian Languages & Code-Mixed Speech: Native support for
te-IN,hi-IN,ta-IN,kn-IN,ml-IN,bn-IN,mr-IN,gu-IN,pa-IN,en-IN, and code-mixed Telugish, Hinglish, and Tanglish.
- Realtime WebSocket (
- Gnani Timbre v3.0 & v2.5 Text-to-Speech (
src/voice/gnani_tts.py):- Streaming SSE (
POST https://api.vachana.ai/api/v1/tts/sse) & REST (POST /api/v1/tts/inference): Synthesizes natural multilingual voices (Shivani,Lakshmi,Avinash,Shubham, etc.) with authentic Indian numeral,₹rupee, lakh, and crore cadence. - Streaming WAV Header Repair & SHA-256 Disk Cache: Repairs
0xFFFFFFFFstreaming RIFF headers on the fly and caches synthesized audio bysha256(text + voice + model)in.winai/tts_cache/for sub-millisecond replay.
- Streaming SSE (
Before decomposing a spoken Indian-language request into an OS automation task graph, Raay routes complex open-ended intents through a provider-agnostic BaseModelProvider adapter layer (src/providers/registry.py). You can plug in any external or local LLM for task planning, report summarization, and conversational reasoning:
| Adapter Module | Supported Models | Primary Role Before OS Automation |
|---|---|---|
AnthropicAdapter (src/providers/anthropic_adapter.py) |
Claude 3.5 / 3.7 Sonnet, Claude Haiku | Multi-step DAG task planning, code generation & structured JSON intent extraction |
OpenAIAdapter (src/providers/openai_adapter.py) |
GPT-4o, GPT-4o-mini, o3-mini | Open-ended goal decomposition, Excel BI question answering & document synthesis |
GeminiAdapter (src/providers/gemini_adapter.py) |
Gemini 2.5 / 2.0 Flash & Pro | Ultra-low-latency multilingual (te-IN, hi-IN, ta-IN) conversational planning & local report summarization |
LocalAdapter (src/providers/local_adapter.py) |
Ollama / vLLM / OpenAI-Compatible Local Endpoints | 100% offline/air-gapped local reasoning prior to desktop execution |
├── src/
│ ├── voice/ # Gnani Prisma STT, Timbre TTS, Conversational Planner & WS Proxy
│ │ ├── gnani_stt.py # WSS /stt/v3/stream & POST /stt/v3 client with circuit breaker
│ │ ├── gnani_tts.py # POST /api/v1/tts/sse & /inference client + SHA-256 WAV cache
│ │ ├── conversational_planner.py # Multilingual turn planner, script verifier & wake-word matcher
│ │ ├── speech_input.py # WinMM 16kHz PCM16 ring-buffered microphone capture manager
│ │ ├── speech_output.py # Cancellable speaker playback manager with sub-200ms stop
│ │ └── api_routes.py # Full-duplex WebSocket stream proxy & OS command dispatcher
│ ├── windows_integration/ # Native Windows 11 OS Execution Controllers
│ │ ├── paint_controller.py # UIA-driven MS Paint controller + HWND focus guard + SceneEditSpec
│ │ ├── excel_grocery_controller.py # 25-item Kirana/Grocery BI analytics + live Excel COM charts
│ │ ├── sutra_showpiece_workflows.py# VS Code export, Downloads report summarizer & OS timers
│ │ └── execution_engine.py # Sandboxed Windows tool execution engine
│ ├── security/ # Host-Side Zero-Trust Security Core
│ │ ├── policy_engine.py # Deterministic ALLOW / DENY / REQUIRE_APPROVAL evaluator
│ │ ├── approval_broker.py # Single-use 256-bit CSPRNG human approval nonces
│ │ └── audit_logger.py # Tamper-evident SHA-256 hash-chained JSONL audit logger
│ ├── storage/ # Hardware Credential Vault & SQLite WAL Persistence
│ │ └── credential_vault.py # Windows DPAPI (CryptProtectData) hardware-encrypted key store
│ └── orchestrator/ # FastAPI Server & 3D Volumetric Web Cockpit (dashboard.html)
├── scripts/
│ ├── rahul_background_listener.py # Always-on zero-click Windows microphone listener
│ └── check_repo_layout.py # Automated repository layout & secret-hygiene verifier
├── docs/
│ ├── ARCHITECTURE.md # Full system & voice architecture spec with Mermaid diagrams
│ └── PRD.md # Product requirements specification
├── tests/ # 412+ hermetic unit & integration tests
├── grocery_bill_sheet.csv # 25-item Indian Kirana/Grocery GST dataset for live Excel BI
├── run_vertical_slice.py # Main orchestrator entrypoint (http://127.0.0.1:8765)
└── requirements.txt # Production dependencies
git clone https://github.com/vakrahul/Raay.git
cd Raay
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txtZero API keys are ever stored in plaintext or .env files. Store your Gnani.ai Vachana API key and any preferred external planning LLM key (Claude, GPT, or Gemini) in the Windows DPAPI Hardware Credential Vault:
# 1. Required: Store Gnani.ai Vachana API Key (for Prisma STT & Timbre TTS)
python -c "from src.storage.credential_vault import CredentialVault; CredentialVault().store_credential('gnani_api_key', 'YOUR_GNANI_VACHANA_API_KEY')"
# 2. Optional: Plug in any external LLM adapter for open-ended task planning (Claude, GPT, or Gemini)
python -c "from src.storage.credential_vault import CredentialVault; v = CredentialVault(); v.store_credential('gemini', 'YOUR_GEMINI_API_KEY')"
# Or for OpenAI GPT / Anthropic Claude:
# python -c "from src.storage.credential_vault import CredentialVault; v = CredentialVault(); v.store_credential('openai', 'YOUR_OPENAI_API_KEY')"
# python -c "from src.storage.credential_vault import CredentialVault; v = CredentialVault(); v.store_credential('anthropic', 'YOUR_CLAUDE_API_KEY')"# Terminal 1: Start the Orchestrator & 3D Web Cockpit on http://127.0.0.1:8765
python run_vertical_slice.py
# Terminal 2: Start the Hands-Free Background Voice Listener
python scripts/rahul_background_listener.py --base-url http://127.0.0.1:8765 --language te-INpython scripts/check_repo_layout.py
pytestBuilt for The Great Indian AI Internship Challenge by Gnani.ai:
- Rahul Vakiti — Systems Architecture, Gnani Prisma STT & Timbre v3.0 Streaming Pipeline, Windows 11 UIA / Excel COM / MS Paint Execution Controllers, Sub-200ms Barge-In Engine, and Windows DPAPI Security Core.
- Samyutha — Multilingual Conversational Planner (
te-IN,hi-IN,ta-IN,hi-en,en-IN), Indic Script Ratio Verification, Kirana/Grocery BI Dataset & Analytics Modelling, and 3D Web Cockpit UX.
Released under the MIT License.