Skip to content

The Synergistic Symphony of Kafka and Spark Streaming

apache kafka logo


Apache Kafka and Apache Spark Streaming are two popular open-source frameworks used for building real-time data pipelines and streaming applications.

Kafka provides a distributed pub/sub messaging system that allows you to publish and consume streams of records or messages. It can handle large amounts of data and is highly scalable, fault-tolerant, and fast.

Spark Streaming allows you to process real-time data streams using the Spark processing engine. It ingest data streams from various sources like Kafka, apply transformations and aggregations, and write the processed data out to external systems.

Combining Kafka and Spark Streaming provides great synergies for building scalable, high-throughput streaming data pipelines with exactly-once processing semantics. Kafka acts as the data source for Spark Streaming, providing a buffer and queueing mechanism, while Spark processes the data in micro-batches or streams.

In this tutorial, we will walk through setting up a Kafka-Spark integration from end-to-end. We will cover:

  • Architecture for Kafka-Spark integration
  • Setting up Kafka and creating topics
  • Reading data from Kafka topics into Spark Streaming
  • Performing transformations on the stream
  • Writing processed data output to external sinks
  • Ensuring fault-tolerance and exactly-once semantics

By the end, you will understand how to build real-time data pipelines leveraging the combined power of Kafka and Spark Streaming.

Overview of Apache Kafka

Apache Kafka is an open-source distributed event streaming platform developed by the Apache Software Foundation written in Scala and Java. Kafka provides a high-throughput, low-latency platform for handling real-time data feeds.

Some key aspects of Kafka include:

  • Kafka has a distributed architecture that consists of brokers, producers, and consumers. Kafka brokers form a cluster that is responsible for data persistence and replication. Producers publish data to Kafka topics while consumers subscribe to these topics to receive data.
  • Kafka operates as a messaging system, but unlike traditional messaging systems, Kafka maintains and batches events for a configurable period of time. This allows data to be replayed or computed.
  • Kafka is designed as a distributed commit log. Data streams are partitioned and spread over a cluster of machines. Each partition can handle terabytes of data. This allows Kafka to process massive volumes of real-time data efficiently.
  • Kafka delivers high throughput for both publishing and subscribing to streams of data. Kafka can handle hundreds of megabytes of reads and writes per second from thousands of clients.
  • Kafka streams are fault-tolerant. Data is replicated across brokers to prevent data loss. If a broker fails, another broker can take over.
  • Kafka has strong ordering guarantees. Events within a partition are written in the order they occur and consumed in that order. This allows for stream processing with correctness.
  • Kafka enables real-time pipeline data processing. Kafka Connect integrates Kafka with external systems for data import/export. Kafka Streams provides stream processing capabilities. These allow Kafka to ingest, process, and export data in real-time.

In summary, Kafka provides a distributed, high-throughput backbone for streaming data pipelines and applications requiring fast data movement. Its architecture makes it scalable, durable, and performant for handling real-time big data feeds.

Overview of Spark Streaming

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.

Spark Streaming follows a micro-batch processing model, where the streaming computation is treated as a series of short batch jobs. This provides sub-second latency with the benefits of batch processing. The data stream is received and divided into short batches which are then processed by Spark to generate outputs.

The Spark Streaming architecture consists of discretized streams or DStreams, which represent a continuous stream of data. DStreams can be created from input data streams or by applying high-level operations on other DStreams. Under the hood, a DStream is represented as a sequence of RDDs.

Spark Streaming natively integrates with Apache Kafka to ingest real-time data feeds. Kafka’s publish-subscribe messaging model allows Spark Streaming applications to subscribe to one or more Kafka topics and process the stream of data from them. The Kafka integration allows building scalable, fault-tolerant streaming applications.

Some key benefits of the Spark Streaming and Kafka integration:

  • Enables real-time processing of streams of data from Kafka topics
  • Micro-batch execution provides low latency with fault tolerance guarantees
  • Ability to handle high volume data streams with scalability
  • No data loss during failures due to Kafka maintaining streams
  • Exactly once semantics when receiving data from Kafka

Overall, Spark Streaming provides an elegant API for stream processing on top of Spark’s strong batch processing capabilities. Integrating Spark Streaming with Kafka provides an end-to-end pipeline for building real-time, large-scale data pipelines and streaming applications.

Kafka-Spark Streaming Integration Architecture

Apache Kafka and Spark Streaming can be integrated to build real-time stream processing applications. The key components involved are:

  • Kafka clusters and topics – Kafka provides fault-tolerant storage for stream data. Data streams are stored in topics within a Kafka cluster. Each topic represents a stream of records.
  • Spark Streaming receivers – Spark Streaming can connect to Kafka through a receiver to receive data. The receiver pulls data from Kafka topics and feeds it into Spark’s execution engine for processing.
  • Direct approach – Spark Streaming integrates with Kafka through a direct approach where Kafka data is directly fed into Spark without going through HDFS or any other data store.

The data flow works as follows:

  • Kafka publishes messages to various topics. These topics act as streams of data.
  • Spark Streaming uses a Kafka receiver to subscribe to specific topics. The receiver pulls the stream of data from the topics.
  • The receiver feeds the stream into Spark’s execution engine where the data can be operated on using RDDs or DataFrames/Datasets. These provide distributed in-memory processing on the streaming data.
  • The processed data can be pushed out to file systems, databases or live dashboards and monitoring systems.

So in summary, Kafka provides the stream storage and transport while Spark Streaming consumes the streams for processing. The direct integration allows building fast, scalable stream processing systems.

Setting Up the Integration

To integrate Kafka and Spark Streaming, we first need to install both systems and configure Kafka topics for Spark Streaming to read from.

Installing Kafka and Spark

Kafka and Spark Streaming run on the Java Virtual Machine (JVM), so you’ll need to have Java 8 or higher installed.

Download the latest binary releases of Kafka and Spark. For simplicity, install Kafka and Spark on the same server or virtual machine.

Unzip the downloaded archives into directories of your choice. Take note of the location of the main Kafka and Spark home directories containing the respective bin/ directories.

Starting Zookeeper and Kafka

Kafka relies on Apache Zookeeper for coordination between clients and brokers. Zookeeper acts as a centralized service for maintaining configuration information and naming.

Navigate to Kafka’s bin/ directory and start a Zookeeper server:

bin/ config/

In a new terminal session, start the Kafka broker service:

bin/ config/

This will start a single broker Kafka instance. For production deployments, you would run Zookeeper and multiple broker instances in cluster mode.

Creating Kafka Topics

We need to create a Kafka topic that Spark Streaming will consume from.

Run the topic creation command:

bin/ --create --topic spark-test --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092

This creates a Kafka topic named “spark-test” with a single partition and single replica. We can now start publishing data to this topic for Spark Streaming to process.

The integration setup is complete. We have Kafka and Spark installed, with a Kafka topic ready for message streaming. Next we can configure the Spark Streaming application and connect it to this topic.

Reading from Kafka Topic

To read data from a Kafka topic into Spark Streaming, we first need to create a Spark streaming context and define the input data stream. Here are the steps:

Create a Spark context with SparkConf and JavaStreamingContext:

from pyspark import SparkContext
from pyspark.streaming import StreamingContext
conf = SparkConf().setAppName("KafkaSparkApp")    
sc = SparkContext(conf=conf)
ssc = StreamingContext(sc, batchDuration) 

Next, we create a KafkaParams object to store the Kafka broker and topic details:

kafkaParams = {"": "broker1:9092,broker2:9092",
"auto.offset.reset": "smallest"}

Then, we subscribe the Spark stream to the Kafka topic by calling createDirectStream():

kafkaStream = KafkaUtils.createDirectStream(ssc,["topic1"],kafkaParams)

This creates a DStream that will continuously pull data from the Kafka topic.

Finally, we parse the binary Kafka data into a readable format. The Kafka data is in key-value format, so we extract the value and decode it:

parsed = v: v[1].decode('utf-8'))

The decoded Kafka stream can then be processed and analyzed as required.

Processing the Data

Now that we are reading data from Kafka into our Spark Streaming application, we need to process that data to extract insights and generate output. Spark Streaming provides powerful operators to transform, window, and aggregate streaming data.

Applying Transformations

We can apply various transformations on the DStream read from Kafka to process the data. This includes map, flatMap, filter, and reduceByKey. For example, we can parse JSON data from Kafka, extract fields, and map it to a Spark SQL Row object for analysis.

jsonDF = msg: json.loads(msg))
jsonDF = row: Row(**row))

Windowing and Stateful Operations

Spark Streaming can apply windowing to DStreams to aggregate data over time slices. This enables time-based aggregations like counting events per minute.

We can also use mapWithState and updateStateByKey to maintain state across windows to calculate running counts or averages. The state is stored in the Spark checkpoint directory and fault-tolerant.

Checkpointing and Write Ahead Logs

For fault-tolerance, we need to enable checkpointing in Spark Streaming. This saves state information to storage.

We can also enable write ahead logs (WAL) to record data received but not yet processed to durable storage. If a failure occurs, streaming can restart and process data from the checkpoint and WAL.

This ensures our streaming application can recover from failures and process all data exactly once for correctness. The checkpoint + WAL approach helps ensure fault-tolerance and zero data loss.

Writing Processed Data

Once the streaming data has been processed and transformed by Spark, the next step is to output or sink the results. There are a few common ways to handle the output:

Saving to Files

Spark Streaming includes connectors to write processed data out to distributed file systems like HDFS or Amazon S3. This allows for fault-tolerant storage of the streaming results to be used for batch analytics or reporting.

The processed data from the streaming DataFrames or RDDs can be persisted to these file systems using actions like df.writeStream.format().option().start(). This will save the streaming results in a specified format like parquet, json, csv, etc.

Database Output

Structured databases like MySQL, PostgreSQL, and Oracle can also act as sinks for the processed streaming data. The foreach operation in Spark Streaming can be used to apply a function that saves each row to the database.

This allows the streaming aggregation and transformation results to be saved in an OLTP database for serving applications and dashboards. Care should be taken to use batch inserts and other best practices for efficiently writing to databases from Spark Streaming.

Kafka as Output Sink

In some streaming pipelines, the processed data may need to be published back out to another Kafka topic for additional downstream consumption.

Spark Streaming integrators for writing data back to Kafka topics like foreachBatch can send the streaming DataFrames or RDDs. Care must be taken to properly serialize the data to bytes for sending to Kafka and to ensure no data loss.

The processed streaming data can also be saved to multiple different outputs and sinks based on the specific requirements. For example, aggregations may be saved to a database while detailed records are saved to object storage.

Ensuring Fault Tolerance with Kafka and Spark Streaming

To build a robust streaming pipeline, we need to ensure it can handle failures and automatically recover. There are two key aspects for fault tolerance with Kafka and Spark Streaming:

Managing Kafka Consumer Offsets

Spark Streaming integrates with Kafka’s consumer group functionality to track offsets for each topic/partition. This allows Spark to know where it left off in consuming a topic after a failure.

When creating the Kafka direct stream in Spark, we specify the consumer group id. On restart after failure, Spark will connect to Kafka and resume from the last committed offsets for that consumer group.

It’s important to use a unique consumer group id for each Spark Streaming application. This isolates them from each other for offset tracking.

Checkpointing in Spark Streaming

Spark Streaming checkpoints data to fault-tolerant storage like HDFS. This allows periodic snapshots of the state of the stream to be saved.

On restart after failure, Spark will restore from the latest checkpoint to resume where it left off in processing. Not all data may be recovered if the checkpoint interval is long.

To enable checkpointing, we specify a directory when creating the Spark streaming context. The checkpoint data will be reliably stored there.

The checkpoint interval represents a tradeoff between overhead and data loss on failure. A shorter interval has more overhead, but less potential data loss.

Recovering from Failures

Between Kafka offsets and Spark checkpointing, we can achieve fault tolerance and minimize data loss. On restart, Spark will:

  • Restore from the latest checkpoint
  • Resume reading from Kafka at the committed consumer offsets

This allows transparent recovery after a failure – the pipeline picks up from where it left off.

With a robust fault tolerance architecture, we can build streaming applications that provide 24/7 uptime. Kafka and Spark give us the primitives for failure recovery and exactly-once semantics.


This tutorial covered setting up a unified streaming data pipeline using Apache Kafka and Spark Streaming. By integrating these two powerful open-source frameworks, we can build real-time data processing applications that are scalable, fault-tolerant, and high-throughput.

In this tutorial, we walked through:

  • Setting up a Kafka cluster and creating topics to stream data
  • Launching a Spark Streaming application and reading data from Kafka topics
  • Applying transformations on Kafka data via Spark’s concise APIs
  • Writing out processed data to external datastores
  • Configuring Spark for guaranteed at-least once delivery semantics

The main benefits of the Kafka-Spark integration are:

  • Leveraging Kafka’s publish-subscribe messaging system for real-time data feeds
  • Building highly available and fault-tolerant data pipelines
  • Enabling scalable stream processing on Spark clusters
  • Avoiding complex glue code and wiring of components

With the fundamentals covered in this tutorial, you can now build more complex streaming applications:

  • Develop pipelines across various data sources, Kafka, and Spark
  • Implement machine learning on streaming data using Spark MLlib
  • Visualize and dashboard real-time analytics using Spark SQL and DataFrames
  • Monitor pipelines and debug bottlenecks using Spark monitoring APIs

Stream processing opens up possibilities for gaining live business insights, real-time predictive analytics, and instant decision making. Kafka and Spark Streaming provide the reliable and scalable data infrastructure to make these use cases a reality.

Leave a Reply

Your email address will not be published. Required fields are marked *