Document Q&A platform for pharmaceutical teams — RAG pipeline with citation-backed answers
Pharmaceutical teams needed a way to query large document repositories (PDFs, DOCX, PPTX) with accurate, citation-backed answers, while retaining full control over ingested content for compliance.
Brought in mid-project as the GenAI engineer, I took ownership of the complete RAG pipeline from scratch while also taking over all frontend and backend responsibilities.
Built the entire document ingestion pipeline from the ground up: FastAPI, Azure Blob Storage with presigned URLs for direct client-side uploads, and Celery with Redis as the message broker for parallel, asynchronous processing.
On the frontend, built the document upload interface and a pipeline status UI using short-polling to keep users engaged with live progress updates for each document (uploading, extracting, chunking, embedding). On the backend, ran up to 8 Celery workers in parallel so multiple documents could be processed simultaneously. This enabled high-throughput ingestion with full visibility and no bottlenecks.
Implemented SHA-256 based file deduplication to eliminate redundant processing across uploads, keeping the pipeline efficient at scale.
Supported multi-modal ingestion across four formats (PDF, DOCX, PPTX, and URLs) using Unstructured to extract text, tables, and images, stored in PostgreSQL with PGvector for 1536-dimensional semantic search.
Designed the system architecture for the supervisor-agent retrieval and generation pipeline, and built a secure chunk review interface where users can audit, edit, and delete chunks side-by-side with a time-limited document preview. This was critical for pharmaceutical teams handling sensitive documents.
Delivered all production-ready APIs and frontend interfaces for the RAG pipeline end-to-end, working alongside another developer who owned the remaining product areas.