What is Kafka

First of all, What is Kafka? Apache Kafka® is a distributed streaming platform. It is often used in real-time streaming data architectures to provide real-time analytics. Written in Scala and Java, Kafka was named after the author Franz Kafka because it is “a system optimized for writing”. Many developers begin exploring messaging when they have to connect lots of things together. When other integration patterns such as shared databases are too dangerous or simply not feasible, Kafka solves this problem. Kafka aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. 

 

Kafka use cases

In order to stay competitive, businesses today rely increasingly on real-time data analysis allowing them to gain faster insights and quicker response times. Real-time insights allow businesses or organisations to make predictions about what they should stock, promote, etc. based on the most up-to-date information possible.

Due to its distributed nature and the streamlined way it manages incoming data, Kafka is capable of operating very quickly  large clusters can be capable of monitoring and reacting to millions of changes to a dataset every second. This means it becomes possible to start working with  and reacting to  streaming data in real-time. By analysing the clickstream data of every session, a greater understanding of user behaviour is achievable. 

Kafka has become widely used, and it is an integral part of the stack at Spotify, Netflix, Uber, Goldman Sachs, Paypal, etc.  which all use it to process streaming data and understand customer, or system, behaviour. 

Actually, Kafka has gained dominance in the travel industry, where its streaming capability makes it ideal for tracking booking details of millions of flights, package holidays and hotel vacancies worldwide.

Kafka provides three main functions to its users:

1. Publish and subscribe to streams of records
2. Effectively store streams of records in the order in which records were generated
3. Process streams of records in real time

Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data.  

For example, if you want to create a data pipeline that takes in user activity data to track how people use your website in real-time, Kafka would be used to ingest and store streaming data while serving reads for the applications powering the data pipeline. Kafka is also often used as a message broker solution, which is a platform that processes and mediates communication between two applications.

 

Let’ look a main Kafka use cases more in details: 

• Messaging

Kafka works well as a replacement for a more traditional message broker. In comparison to most messaging systems Kafka has better throughput, built-in partitioning, replication, and fault-tolerance which makes it a good solution for large scale message processing applications.

 

• Website Activity Tracking

The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. This means site activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type. Activity tracking is often very high volume as many activity messages are generated for each user page view.

 

• Metrics

Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of operational data. 

 

• Log Aggregation

Many people use Kafka as a replacement for a log aggregation solution. Log aggregation typically collects physical log files off servers and puts them in a central place for processing. Kafka abstracts away the details of files and gives a cleaner abstraction of log or event data as a stream of messages. This allows for lower-latency processing and easier support for multiple data sources and distributed data consumption. I

 

• Stream Processing

Many users of Kafka process data in processing pipelines consisting of multiple stages, where raw input data is consumed from Kafka topics and then aggregated, enriched, or otherwise transformed into new topics for further consumption or follow-up processing.

 

• Event Sourcing

Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Kafka’s support for very large stored log data makes it an excellent backend for an application built in this style.

 

• Commit Log

Kafka can serve as a kind of external commit-log for a distributed system. The log helps replicate data between nodes and acts as a re-syncing mechanism for failed nodes to restore their data. 

 

• Publish + Subscribe

At its heart lies the humble, immutable commit log, and from there you can subscribe to it, and publish data to any number of systems or real-time applications. 

 

• Store

An abstraction of a distributed commit log commonly found in distributed databases, Apache Kafka provides durable storage. Kafka can act as a source of truth, being able to distribute data across multiple nodes for a highly available deployment within a single data center or across multiple availability zones.

 

Kafka Users

A lot of large companies who handle a lot of data use Kafka. LinkedIn, where it originated, uses it to track activity data and operational metrics. Twitter uses it as part of Storm to provide a stream processing infrastructure.  It’s also used by other companies like Spotify, Uber, Tumbler, Goldman Sachs, PayPal, Box, Cisco, CloudFlare, and Netflix.

Kafka Benefits

1.Scalable

Kafka’s partitioned log model allows data to be distributed across multiple servers, making it scalable beyond what would fit on a single server. 

 

2.Fast

Kafka decouples data streams so there is very low latency, making it extremely fast. 

 

3.Durable

It helps protect against server failure, making the data very fault-tolerant and durable. 

 

4.Simple

Kafka is easy to set up and use, and it is easy to figure out how Kafka works.

 

5.Performance 

It is stable, provides reliable durability, has a flexible publish-subscribe/queue that scales well, has robust replication, provides producers with tunable consistency guarantees, and it provides preserved ordering at the shard level. 

 

6.React to customers in real-time

It is a big data technology that enables you to process data in motion and quickly determine what is working, what is not. 

There are many other Kafka benefits that can be the reason to start using it. If you know them, feel free to share them in the comments below! If not, but you would like to know more about it, check our upcoming hands-on courses and workshops and grow with us!