Oct 2024 - Dec 2024 · Smart India Hackathon (Bharat Electronics Limited)
Multimodal RAG Chatbot
Smart India Hackathon Grand Finale, Top 1.11%.
Built for the Smart India Hackathon Grand Finale, this multimodal chatbot answers natural-language questions about user-uploaded images and documents, always grounding its answers in a cited source. It placed in the top 1.11% of entries.
Architecture
The problem. Answering questions over a user's own images and documents needs both grounded retrieval and visual understanding, without the model inventing facts.
Hybrid retrieval, always cited. Retrieval fuses pgvector vector search with Postgres full-text search using reciprocal rank fusion, then a cross-encoder reranks, so every answer points back to a real source passage.
Ingest big docs in bounded memory. Uploads stream through a pipeline that extracts, chunks, and embeds each page as it arrives, so a large document never has to fit in memory at once.
See the image, not just read it. Meta SAM2 on a serverless GPU does zero-shot object cutout from point or box prompts, and the cutout feeds the vision model; the text and image paths converge into one cited answer.
What I built
- Built a multimodal RAG chatbot (text + image) that grounds every answer in a cited source passage.
- Engineered hybrid retrieval over pgvector vector search and Postgres full-text search, RRF-fused and cross-encoder reranked.
- Offers three answer paths: a fully local Ollama model (offline, default), a Groq LangGraph ReAct agent (agentic), and Gemini for image questions.
- Built a streaming ingestion pipeline that extracts, chunks, and embeds large document uploads per page in bounded memory.
- Integrated Meta SAM2 on a serverless GPU for zero-shot object cutout via point or box prompts, fed to the vision model.
Under the hood
The stack, the data structures, and the decisions that matter, for engineers.
Hybrid search fused in one SQL query
hybridRetrieve runs both retrievers in a single Postgres query: a pgvector CTE ranks the top thirty chunks by cosine distance (embedding <=> query) and a full-text CTE ranks the top thirty by ts_rank_cd over a tsvector, and the two are merged with reciprocal rank fusion (a sum of 1/(60 + rank) across whichever list each chunk appears in), taking the fused top twenty. Doing it in one query means no round trips and the fusion happens in the database. A user filter and a 'ready' status filter live inside both CTEs so retrieval never leaks across users or reads half-ingested docs.