Guide to Building Python and LLM-Based Multimodal Chatbots Course Syllabus

Overview: This course offers a hands-on, end-to-end journey into building modern multimodal chatbots using Python and large language models (LLMs). You'll progress from foundational concepts to deploying full-stack chatbot applications with speech, vision, and retrieval-augmented capabilities. Through interactive coding environments and practical projects, you’ll gain real-world skills in Gradio, Rasa, Ollama, Whisper v3, Gemini, LlamaIndex, and deployment via Hugging Face and React. Total time commitment: approximately 5 hours.

Module 1: Getting Started with AI Chatbots

Estimated time: 0.5 hours

• Evolution of chatbots: rule-based to GenAI-powered systems
• Introduction to Gradio for building UIs
• Building a simple Python chatbot with Gradio
• Interactive quiz on chatbot fundamentals

Module 2: Foundations of AI Chatbots with Rasa

Estimated time: 0.75 hours

• Overview of Rasa framework and components
• Understanding conversational AI pipelines
• Python integration with Rasa
• Creating a rule-based chatbot in-browser

Module 3: Generative Chatbots with Small LLMs

Estimated time: 1 hour

• Introduction to small LLMs: Ollama and Llama
• Integrating LLMs into Gradio interfaces
• Customizing and running SLM-powered chatbots
• Comparing performance across frameworks

Module 4: Multimodal Capabilities – Speech & Vision

Estimated time: 1 hour

• Adding speech input with Whisper v3
• Image understanding using Gemini API
• Processing audio and image inputs in chatbots
• Generating text responses from multimodal inputs

Module 5: RAG Integration with LlamaIndex

Estimated time: 0.75 hours

• Retrieval-Augmented Generation (RAG) fundamentals
• Document indexing and retrieval pipelines
• Enhancing chatbot responses with external knowledge

Module 6: Deployment & Frontend with Hugging Face & React

Estimated time: 0.75 hours

• Deploying models via Hugging Face
• Building React frontends for chatbots
• Integrating OpenAI APIs and managing API keys

Module 7: Capstone & Challenges

Estimated time: 0.5 hours

• Combining multimodal and RAG components
• Finalizing full-stack chatbot architecture
• Deploying a complete Gradio-based project

Prerequisites

• Basic knowledge of Python programming
• Familiarity with command-line interface
• Understanding of fundamental AI concepts

What You'll Be Able to Do After

• Build and deploy Python-based chatbots using Gradio
• Implement rule-based and generative AI chatbots with Rasa and LLMs
• Add multimodal capabilities including speech and image processing
• Enhance chatbot responses using RAG with LlamaIndex
• Deploy chatbots via Hugging Face and integrate with React frontends