Large language models bootcamp

What you'll learn

• Adoption Reality: Why most LLM projects stall between proof-of-concept and production; the competing pressures of cost, accuracy, latency, and infrastructure constraints that make scaling harder than building.
• Technology Challenges: Context-window limits, dataset quality and PII handling, the real cost of fine-tuning (including PEFT approaches), and how inference costs compound with RAG and token volume.
• Business and Risk Challenges: Compliance exposure, legal risk, misaligned customer expectations, and the organizational culture gaps that prevent meaningful KPI alignment.
• Human Behavior Challenges: Fear of AI, resistance to change, biased prompting, and the subjectivity in feedback that makes evaluation harder than it looks.
• Prompting Challenges: Prompt sensitivity, fatigue, overengineering, jailbreaking and injection risks, hallucinations from lack of grounding, and how ambiguity creates silent failures.
• Best Practices: Guardrails and defensive UX design, LLM caching for cost control, structured feedback collection, and evaluation frameworks built around fairness and explainability.

What you'll learn

• Privacy, Security, and Compliance: PII-safe ingestion through anonymization and differential privacy; authenticated access and user filtering at the retrieval layer; logging and data retention for compliance.
• Multimodal and Multilingual RAG: Vision models (CLIP, BLIP-2, Gemini), audio pipelines (Whisper, audio embeddings, noise-tolerant search), and cross-lingual retrieval with LaBSE, mBERT, and XLM-R.
• Advanced Architectures: DSPy for prompt compilation and retrieval chain optimization; KG-RAG for graph-grounded, traceable answers; self-improving retrieval through feedback loops, ranking callbacks, and adaptive scoring.
• Retrieval Layer and Index Optimization: FAISS, Qdrant, Weaviate, and Pinecone compared; flat, HNSW, IVF, and PQ index types; improving search quality with hybrid BM25 + dense retrieval and re-rankers like BGE and ColBERT.
• Evaluation and Metrics: Retrieval metrics (Precision@K, Recall@K, Hit Rate@K), generation metrics (BLEU, ROUGE, LLM-as-a-Judge), and tooling with TruLens, Phoenix, and EvalGen.
• Latency, Cost, and Scalability: Identifying bottlenecks across retrieval, re-ranking, and generation; caching strategies at the embedding, prompt, and disk layers; cost controls through batch queries, static prompts, and quantized models.

What you'll learn

• Introduction to LLMs: Strengths and weaknesses of large language models; discriminative versus generative AI; how predictive and generative models differ in what they're actually doing.
• Transformer Architecture: Tokenization, embeddings, positional encoding, and the attention mechanism that holds it all together — explained from the ground up.
• Embeddings and Similarity: How words become vectors, what proximity in that space actually means, and why it matters for retrieval and reasoning.
• Attention Mechanism: Keys, queries, and values in self-attention; how the model decides what to focus on when generating each token.
• Softmax and Probabilities: How raw attention scores become a probability distribution, and what that means for next-token prediction.
• Training and Fine-Tuning: Adapting pre-trained models with curated data — what changes, what doesn't, and where overfitting quietly creeps in.
• Search and Retrieval: Building a semantic search engine with embeddings; connecting retrieval to generation for grounded, factual answers.
• Hands-On Exercises: Sentence Transformers, semantic search, attention scoring, and implementing attention mechanisms directly.

What you'll learn

• Overview and Rationale: Why vector databases exist, how they differ from traditional search, and the role they play in grounding LLM applications.
• Search Types: Vector search, text search, and hybrid search — when to use each and what you lose by picking the wrong one.
• Indexing Techniques: Product Quantization (PQ), Locality Sensitive Hashing (LSH), and Hierarchical Navigable Small World (HNSW) — the tradeoffs between speed, memory, and recall.
• Retrieval Techniques: Cosine similarity, nearest neighbor search, and how retrieval quality degrades at scale without the right index design.
• Advanced RAG Techniques: Chunking and filtering strategies, query rewriting, hybrid search, auto-cut, and re-ranking for precision at the top of the result set.
• Embedding and Model Selection: Domain-specific embeddings, fine-tuned retrieval models, and compression approaches including scalar quantization, product quantization, and binary and matryoshka methods.
• Adaptive Retrieval and Multi-Tenancy: Multi-phase rescoring, preserving recall under compression, tenant isolation, and resource allocation patterns for shared environments.
• Production Challenges: Scaling, reliability, and cost optimization — the problems that don't surface until you're running real query volumes.
• Hands-On Exercises: Vector search, similarity search, hybrid search, generative search, Weaviate Query Agent, multi-tenancy, vector compression, and semantic caching.

What you'll learn

• Introduction to LangChain: What LangChain is for, what it abstracts away, and the class of RAG challenges it was built to address.
• Core Components: LLMs and chat models, prompt templates, example selectors, document loaders, and transformers as the building blocks of LLM-powered applications.
• Output Parsers: Structured data extraction, consistent formatting across model responses, and error handling when outputs don't conform.
• Retrieval and Vector Stores: Embedding, vectorization, metadata filtering, parent document retrieval, and efficient similarity search optimized for large datasets.
• Chains: Sequential prompt logic with pre- and post-LLM steps; integrating tools and retrieval into coherent, composable workflows.
• Tool Use and Memory: Connecting APIs and external actions, passing results back into the workflow, managing conversation history, storing state between calls, and maintaining context persistence for agents.
• Callbacks and Observability: Event hooks during runs, monitoring and logging, and custom actions on success or failure.
• LCEL (LangChain Expression Language): Piping components with runnables, parallel branches, and modular workflow composition.
• LangGraph and Agents: Graph-based workflows for complex agent orchestration, dynamic control flows, decision-making steps, and combining tools with context for non-linear reasoning.

What you'll learn

• Core Concepts: Transfer learning and why fine-tuning works; full fine-tuning versus LoRA and QLoRA; parameter-efficient tuning and quantization as practical paths to adapting large models without prohibitive compute.
• Key Considerations: Data quality and relevance as the primary lever for fine-tuning outcomes; overfitting risks and the limitations fine-tuning cannot resolve; when fine-tuning is the right choice versus RAG.
• Hands-On Exercises: Instruction fine-tuning, deploying, and evaluating a LLaMA2-7B 4-bit quantized model in class; fine-tuning and deploying OpenAI and Llama models on Azure AI Studio as a take-home project.

What you'll learn

• Need for Evaluation: Why reliability, accuracy, and safety can't be assumed — and how business alignment, ethical accountability, and user trust depend on structured measurement.
• Challenges in Evaluation: Hallucinations, prompt sensitivity, and weak context handling; the difficulty of evaluating outputs where multiple valid answers exist; navigating tradeoffs between accuracy, fluency, and creativity.
• Benchmarking Approaches: MMLU for multitask accuracy, HELM for holistic metrics across accuracy, robustness, and fairness, BBH and HotpotQA for reasoning and multi-hop question answering.
• Text Quality Metrics: BLEU for n-gram precision, ROUGE for recall-based evaluation, BERTScore for semantic similarity, METEOR for synonym and stem alignment, and perplexity as a measure of prediction confidence.
• RAG-Specific Evaluation (RAGAs): Faithfulness, answer relevance, context precision, and context recall — metrics that score retrieval and generation jointly rather than in isolation.
• Open-Ended Output Evaluation (G-Eval): Fluency, faithfulness, answer relevance, and claim-level scoring for outputs that don't have a single correct answer.
• Additional Benchmarks and Metrics: GLUE for NLU tasks, TriviaQA for multi-hop QA, RealToxicityPrompts for safety evaluation, MRR and MAP for ranking performance, and ROSCOE for reasoning quality across semantic alignment, logical integrity, and commonsense coverage.

What you'll learn

• Origins & Motivation: Addressing fragmented integrations and brittle bespoke adapters; introducing a unified, interoperable interface — the "USB-C for AI."
• Protocol Structure: Client–server handshake model; defining resources, tools, and prompts; JSON-RPC transport with structured, schema-driven messages.
• Context Exposure: How MCP surfaces tools, data, and metadata through a consistent schema to enable discoverability, governance, and controlled access.
• Agentic Integration: Connecting MCP endpoints to reflection, planning, tool-use, and multi-agent coordination patterns for modular and scalable systems.
• Hands-On Labs: Setting up an MCP client in Streamlit; discovering and registering tools; automating workflows through data retrieval and validation; logging traces for monitoring and review.

What you'll learn

• Conversational Workflow Orchestration: Design a multi-turn assistant coordinating tasks across specialized agents.
• Knowledge-Enhanced Agent: Integrate search and APIs for grounding, fact-checking, and real-time data access.
• Document-Aware Action Agent: Retrieve and reason over documents; trigger external tools or services based on insights.
• Orchestrated Collaboration (MCP): Build coordinated multi-agent systems using the Model Context Protocol for seamless tool and enterprise integration.
• Attendees receive comprehensive datasets, step-by-step implementation guides, and ready-to-use code templates within Data Science Dojo's sandbox for accelerated development.
• Learners can choose to implement: Virtual Assistant, Content Generation (Marketing Co-pilot), Conversational Agent (Legal & Compliance Assistant), Content Personalizer, or MCP Chatbot — an AI agent with calendar, CRM, and API integrations.
• Outcome: A production-ready multi-agent application demonstrating mastery of reasoning, retrieval, tool use, and protocol-driven interoperability.