Logstash vs. Kafka

1. Overview

Logstash and Kafka are two powerful tools for managing real-time data streams. While Kafka excels as a distributed event streaming platform, Logstash is a data processing pipeline for ingesting, filtering, and forwarding data to various outputs.

In this tutorial, we’ll examine the difference between Kafka and Logstash in more detail and provide examples of their usage.

2. Requirements

Before learning the difference between Logstash and Kafka, let’s ensure we have a few prerequisites installed and basic knowledge of the technologies involved. First, we need to install Java 8 or later.

Logstash is part of the ELK stack (Elasticsearch, Logstash, Kibana) but can be installed and used independently. For Logstash, we can visit the official Logstash download page and download the appropriate package for our operating system (Linux, macOS, or Windows).

We also need to install Kafka and have confidence in our understanding of the publisher-subscriber model.

3. Logstash

Let’s look at the main Logstash components and a command-line example to process a log file.

3.1. Logstash Components

Logstash is an open-source data processing pipeline within the ELK Stack used to collect, process, and forward data from multiple sources. It’s composed of several core components that work together to collect, transform, and output data:

1. Inputs: These bring data into Logstash from various sources such as log files, databases, message queues like Kafka, or cloud services. Inputs define where the raw data comes from.
2. Filters: These components process and transform the data. Common filters include Grok for parsing unstructured data, mutate for modifying fields, and date for timestamp formatting. Filters allow for deep customization and data preparation before sending it to its final destination.
3. Outputs: After processing, outputs send the data to destinations such as Elasticsearch, databases, message queues, or local files. Logstash supports multiple parallel outputs, making it ideal for distributing data to various endpoints.
4. Codecs: Codecs encode and decode data streams, such as converting JSON to structured objects or reading plain text. They act as mini-plugins that process the data as it’s being ingested or sent out.
5. Pipelines: A pipeline is a defined data flow through inputs, filters, and outputs. Pipelines can create complex workflows, enabling data processing in multiple stages.

These components work together to make Logstash a powerful tool for centralizing logs, transforming data, and integrating with various external systems.

3.2. Logstash Example

Let’s give an example of how we process an input file to an output in JSON format. Let’s create an example.log input file in the /tmp directory:

2024-10-12 10:01:15 INFO User login successful
2024-10-12 10:05:32 ERROR Database connection failed
2024-10-12 10:10:45 WARN Disk space running low

We can then run the logstash -e command by providing a configuration:

$ sudo logstash -e '
input { 
  file { 
    path => "/tmp/example.log" 
    start_position => "beginning" 
    sincedb_path => "/dev/null" 
  } 
} 
filter { 
  grok { 
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}" }
  } 
  mutate {
    remove_field => ["log", "timestamp", "event", "@timestamp"]
  }
}
output { 
  file {
    path => "/tmp/processed-logs.json"
    codec => json_lines
  }
}'

Let’s explain the different parts of the configuration:

• The whole chain of commands (input/filter/output) is a pipeline.
• Extract timestamp, log level, and message fields from the logs with the grok filter.
• Remove unnecessary info with a mutate filter.
• Apply JSON format with Codec in the output filter.
• After the input example.log file is processed, the output will be encoded in JSON format in the processed-log.json file.

Let’s see an output example:

{"message":["2024-10-12 10:05:32 ERROR Database connection failed","Database connection failed"],"host":{"name":"baeldung"},"@version":"1"}
{"message":["2024-10-12 10:10:45 WARN Disk space running low","Disk space running low"],"host":{"name":"baeldung"},"@version":"1"}
{"message":["2024-10-12 10:01:15 INFO User login successful","User login successful"],"host":{"name":"baeldung"},"@version":"1"}

As we can see, the output file is JSON with additional info, such as the @version, that we can use, for example, to document the change and ensure that any downstream processes (like querying in Elasticsearch) are aware of it to maintain data consistency.

4. Kafka

Let’s look at the main Kakfa component and a command-line example of publishing and consuming a message.

4.1. Kafka Components

Apache Kafka is an open-source distributed event streaming platform for building real-time data pipelines and applications.

Let’s look at its main components:

1. Topics and Partitions: Kafka organizes messages into categories called topics. Each topic is divided into partitions, which allow data to be processed on multiple servers in parallel. For example, in an e-commerce application, you might have separate topics for order data, payment transactions, and user activity logs.
2. Producers and Consumers: Producers publish data (messages) to Kafka topics, while consumers are applications or services that read and process these messages. Producers push data to Kafka’s distributed brokers, ensuring scalability, while consumers can subscribe to topics and read messages from specific partitions. Kafka guarantees that consumers read each message in order.
3. Brokers: Kafka brokers are servers that store and manage topic partitions. Multiple brokers comprise a Kafka cluster, distributing data and ensuring fault tolerance. If one broker fails, other brokers take over the data, providing high availability.
4. Kafka Streams and Kafka Connect: Kafka Streams is a powerful stream processing library that allows real-time data processing directly from Kafka topics. Thus, it enables applications to process and transform data on the fly, such as calculating real-time analytics or detecting patterns in financial transactions. On the other hand, Kafka Connect simplifies the integration of Kafka with external systems. It provides connectors for integrating databases, cloud services, and other applications.
5. ZooKeeper and KRaft: Traditionally, Kafka used ZooKeeper for distributed configuration management, including managing broker metadata and leader election for partition replication. With the introduction of KRaft (Kafka Raft), Kafka now supports ZooKeeper-less architectures, but ZooKeeper is still commonly used in many setups.

Together, these components enable Kafka to deliver a scalable, fault-tolerant, distributed messaging platform that can handle massive volumes of streaming data.

4.2. Kafka Example

Let’s create a topic, publish a simple “Hello, World” message, and consume it.

First, let’s create a topic. It can belong to multiple partitions and typically represents one subject of our domain:

$ /bin/kafka-topics.sh \
  --create \
  --topic hello-world \
  --bootstrap-server localhost:9092 \
  --partitions 1 \
  --replication-factor 1

We’ll get the message of the topic creation:

$ Created topic hello-world.

Let’s now try to send a message to the topic:

$ /bin/kafka-console-producer.sh \
  --topic hello-world \
  --bootstrap-server localhost:9092 \
  <<< "Hello, World!"

Now, we can consume our messages:

$ /bin/kafka-console-consumer.sh \
  --topic hello-world \
  --from-beginning \
  --bootstrap-server localhost:9092

We’ll get messages from the Kafka log storage for that specific topic by consuming them:

Hello, World!

5. Core Differences Between Logstash and Kafka

Logstash and Kafka are integral components of modern data processing architectures, each fulfilling distinct yet complementary roles.

5.1. Logstash

Logstash is an open-source data processing pipeline specializing in ingesting data, transforming it, and sending the results to various outputs. Its strength lies in its ability to parse and enrich data, making it ideal for processing log and event data.

For instance, a typical use case might involve a web application where Logstash ingests logs from multiple servers. Then, it applies filters to extract relevant fields such as timestamps and error messages. Finally, it forwards this enriched data to Elasticsearch for indexing and visualization in Kibana to monitor application performance and diagnose real-time issues.

5.2. Kafka

In contrast, Kafka is a distributed streaming platform that excels in handling high-throughput, fault-tolerant, and real-time data streaming. It functions as a message broker, facilitating the publishing of and subscribing to streams of records.

For example, in an e-commerce architecture, Kafka can capture user activity events from various services, such as website clicks, purchases, and inventory updates. These events can be produced into Kafka topics, allowing multiple downstream services (like recommendation engines, analytics platforms, and notification systems) to consume the data in real-time.

5.3. Differences

While Logstash focuses on data transformation, enriching raw logs, and sending them to various destinations, Kafka emphasizes reliable message delivery and stream processing, allowing real-time data flows across diverse systems.

Let’s look at the main differences:

Together, they enable organizations to build robust data pipelines that facilitate real-time insights and decision-making, demonstrating their critical roles in the evolving landscape of data architecture.

6. Can Logstash and Kafka Work Together?

Logstash and Kafka can seamlessly collaborate to create a robust data processing pipeline, combining their strengths to enhance data ingestion, processing, and delivery.

6.1. From Logstash

For example, Logstash can act as a data collector and processor that ingests various data sources, such as logs, metrics, and events, and then transforms this data to fit specific formats or schemas. For instance, in a microservices architecture, Logstash can collect logs from various microservices, apply filters to extract pertinent information, and then forward the structured data to Kafka topics for further processing.

6.2. To Kafka

Once the data is in Kafka, it can be consumed by multiple applications and services that require real-time processing and analytics. For example, a financial institution may use Kafka to stream transaction data from its payment processing system, which various applications — including fraud detection systems, analytics platforms, and reporting tools — can consume.

6.3. LogStash With Kafka

Logstash facilitates the initial ingestion and transformation of logs and events. At the same time, Kafka is a scalable, fault-tolerant messaging backbone that ensures reliable data delivery across the architecture.

By integrating Logstash and Kafka, organizations can build robust and flexible data pipelines that efficiently handle high volumes of data, enabling real-time analytics and insights. This collaboration allows data ingestion to be decoupled from processing, fostering scalability and resilience within their data architecture.

7. Conclusion

In this tutorial, we saw how Logstash and Kafka work by providing architectural and command-line examples. We saw their main usage and described for which practical usage each is best by describing their main components. Finally, we saw the main differences between these two systems and how they can work together.