Kafka Fundamentals for Beginners
BeginnerLearn the core concepts of Apache Kafka including topics, producers, consumers, and brokers. This course covers distributed systems, Kafka architecture, installation, and hands-on practice.
1.1 Overview of Distributed Systems
🎥Understanding distributed systems vs monolithic architecture and communication patterns.
1.2 Why Kafka?
🎥Learn what makes Kafka ideal for real-time data processing. Explore use cases including real-time analytics, event sourcing, and log aggregation, with examples from Uber, LinkedIn, and Netflix.
1.3 Brief History of Kafka
🎥Discover Kafka's journey from LinkedIn in 2010 to becoming an Apache top-level project. Learn about key milestones including replication (2014), Kafka Streams (2016), and KSQL (2017).
2.1.0 Installing Kafka With Zookeeper
🎥Traditional Kafka installation with Zookeeper for Windows, Mac, Linux, and Docker. Covers environment setup, Zookeeper configuration, and Kafka server startup.
2.1.1 Installing Kafka Without Zookeeper (KRaft Mode)
🎥Complete guide to installing Kafka 3.8.0 in KRaft mode without Zookeeper. Covers installation steps for Linux, Mac, Windows, and Docker with detailed configuration examples.
2.2 Kafka Configuration Files
🎥Overview of Kafka configuration files and key settings.
2.3 Setting Up Zookeeper
🎥Understanding Zookeeper's role and how to configure it for Kafka.
2.4 Understanding KRaft Mode
🎥Learn Kafka's KRaft consensus protocol that eliminates Zookeeper dependency. Covers installation on Linux, Mac, Windows, and Docker with complete configuration examples.
3.1 Kafka Architecture
🎥Deep dive into Kafka's distributed system architecture.
3.2 Key Components
🎥Understanding brokers, producers, consumers, and topics in Kafka.
3.3 Topics and Partitions
🎥How Kafka distributes and stores data using topics and partitions.
3.4 Real-Time vs Batch Processing
🎥Understanding the differences between real-time and batch processing in Kafka.
3.5 Cluster Scaling
🎥Learn how to scale Kafka clusters for increased data volumes and traffic.
4.1 Creating and Managing Topics
🎥Commands and configuration for managing Kafka topics.
4.2 Producing Messages
🎥Using the command line to produce messages and best practices.
4.3 Understanding Kafka Schema
🎥Introduction to schema management in Kafka.
4.4 Schema Evolution
🎥Backward compatibility and schema evolution examples.
4.5 Consuming Messages
🎥Consumer groups, offsets, and message consumption patterns.
4.6 Performance Testing
🎥NFT and performance testing strategies for Kafka.
4.7 In-Sync Replicas and Acknowledgements
🎥Ensuring data reliability with ISR and acknowledgement modes.
5.1 Kafka API Introduction
🎥Overview of Kafka's Producer and Consumer APIs.
5.2 Producer API
🎥Sending data to Kafka using the Producer API.
5.3 Consumer API
🎥Learn to read data from Kafka using Consumer API. Covers consumer groups, polling, offsets, and graceful shutdown with complete Java implementation examples.
5.4 Consumer Offset Management
🎥Master offset management for reliable Kafka consumers. Covers auto vs manual commits, consumer groups, partition assignment, and offset reset strategies with code examples.
5.5 Message Guarantees
🎥At-least-once, at-most-once, and exactly-once delivery semantics.
6.1 Kafka Connect
🎥Integrating external systems with Kafka Connect.
6.2 Running Kafka Connect Locally
🎥Set up Kafka Connect using Docker to stream data from MySQL to file. Learn JDBC Source and File Sink connectors for real-time data integration.
6.3 Kafka Streams
🎥Real-time stream processing with Kafka Streams library. Covers stateless vs stateful operations, exactly-once processing, and event-time windowing.
6.4 Running Kafka Streams Locally
🎥Build and run Kafka Streams application locally using Java and Docker. Filter and transform transaction data in real-time with hands-on examples.
6.5 Kafka Streams Deep Dive
🎥Master KStream, KTable, and GlobalKTable with advanced joins, aggregations, and state management. Includes concurrency and scaling strategies.