Performance Testing

Overview

Performance testing focuses on measuring Kafka's throughput (TPS - Transactions Per Second) and stress testing the system using built-in CLI tools.

kafka-producer-perf-test.sh

This tool generates test data to:

• Measure throughput (TPS)
• Stress test the system
• Establish baseline metrics
• Identify optimization areas

Note: This tool doesn't simulate actual producers with schema validation or business logic - it purely measures broker performance under load.

Key Metrics

Throughput (TPS)

• Messages per second the brokers can handle
• Primary performance indicator
• Varies with message size and configuration

Broker Resource Usage

• CPU utilization
• Memory consumption
• Disk I/O
• Network utilization
• Identifies potential bottlenecks

Environment Setup

Docker Compose with:

• Zookeeper
• Single Kafka broker

Create test topic:

1docker exec -it kafka kafka-topics.sh \\
2  --create --topic payment \\
3  --bootstrap-server localhost:9092 \\
4  --partitions 3 \\
5  --replication-factor 1

Running Performance Tests

Basic Test Command

1kafka-producer-perf-test.sh \\
2  --topic payment \\
3  --broker-list localhost:9092 \\
4  --throughput 10000 \\
5  --messages 1000000 \\
6  --producer-props acks=1 \\
7  --reporting-interval 1000 \\
8  --send-rate 1000

Parameters Explained

• --topic: Target topic for messages
• --broker-list: Kafka broker address
• --throughput: Target messages per second (10,000)
• --messages: Total messages to send (1,000,000)
• --producer-props acks=1: Wait for leader acknowledgment only
• --reporting-interval: Log metrics every 1,000 messages
• --send-rate: Controlled rate of 1,000 messages/second

Understanding Results

Example Output

1
2Total time: 1000 ms
3Messages produced: 1000000
4Throughput: 1000000 messages/sec
5Latency avg: 2.3 ms
6Latency min: 0.1 ms
7Latency max: 10 ms

Metrics Analysis

Total Time: Duration of the test (1,000 ms = 1 second)

Messages Produced: Total messages sent (1,000,000)

Throughput: Messages per second achieved (1,000,000 msg/sec)

Average Latency: Mean time to send a message (2.3 ms)

Minimum Latency: Fastest message transmission (0.1 ms)

Maximum Latency: Slowest message transmission (10 ms)

Interpreting Results

Good Performance

• Consistent latency
• Throughput meets target
• Low max latency
• Stable resource usage

Performance Issues

• High latency variance
• Below-target throughput
• Resource saturation
• Growing message backlog

Tuning for Better Performance

Producer Configuration

• batch.size: Larger batches improve throughput
• linger.ms: Wait time for batching
• compression.type: Use lz4 or snappy
• buffer.memory: Increase for high throughput

Broker Configuration

• num.network.threads: More threads for handling requests
• num.io.threads: More threads for disk operations
• socket.send.buffer.bytes: Larger network buffers
• socket.receive.buffer.bytes: Larger receive buffers

Topic Configuration

• Increase partitions for parallelism
• Adjust replication factor vs performance trade-off
• Configure appropriate retention

Testing Scenarios

Baseline Test

• Standard configuration
• Moderate load
• Establish benchmark

Stress Test

• Maximum throughput
• Identify breaking points
• Test failover scenarios

Sustained Load

• Long-running test
• Monitor resource trends
• Check for degradation

Best Practices

Test Planning

• Define clear objectives
• Test in isolated environment
• Match production configuration
• Document all parameters

Monitoring

• Track all resource metrics
• Use monitoring tools (Prometheus, Grafana)
• Log results for comparison
• Create performance baselines

Analysis

• Compare against requirements
• Identify bottlenecks
• Test incremental changes
• Validate improvements

Summary

Performance testing is essential for:

• Establishing baseline metrics
• Identifying system limits
• Validating configuration changes
• Capacity planning

Use kafka-producer-perf-test.sh to measure broker performance and optimize your Kafka deployment for production workloads.