This post was originally published on this site

In the following post, I will show how to install Apache Kafka on a Linux VM.

Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java.

First of all we need a Linux VM (e.g Centos 7) with at least 1 GB of RAM

Connect as root and add a new user for Kafka

[[email protected] ~]# adduser kafka

set the password

[[email protected] ~]# passwd kafka
Changing password for user kafka.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

now add the newly created user to sudoers group

[[email protected] ~]# gpasswd -a kafka wheel


[[email protected] ~]# visudo

find the follwing code

## Allow root to run any commands anywhere
root ALL=(ALL) ALL

add the following code below

kafka ALL=(ALL) ALL

update and reboot the system

[[email protected] ~]# yum update -y

#reboot the system

[[email protected] ~]# reboot

login with the user created above and install java

[email protected] ~]$ sudo yum install java-1.8.0-openjdk -y


check the java version

[email protected] ~]$ java -version

openjdk version "1.8.0_144"
OpenJDK Runtime Environment (build 1.8.0_144-b01)
OpenJDK 64-Bit Server VM (build 25.144-b01, mixed mode)

add java variables to .bash_profile

#java settings
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
export JRE_HOME=/usr/lib/jvm/jre

resource profile and check settings

[email protected] ~]$. .bash_profile

[email protected] ~]$ echo $JAVA_HOME
[[email protected] ~]$ echo $JRE_HOME

Now we are ready to install and confgiure Kafka.We will use the confluent platform for this environment. Confluent open source platform is freely available for download.


So first of all we have to import Confluents public key.

[[email protected] ~]$ sudo rpm --import

After that we need to add the Confluent repository to yum.For that create a new file called confluent.repo in /etc/yum.repos.d

[[email protected] ~]$ sudo vi /etc/yum.repos.d/confluent.repo

and paste the following:

name=Confluent repository (dist)

name=Confluent repository

Now clean up your yum caches with

[[email protected] ~]$ sudo yum clean all
Loaded plugins: fastestmirror, langpacks
Cleaning repos: Confluent Confluent.dist base extras updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Cleaning up list of fastest mirrors

The repo is now ready for use and we could install Confluent Open Source

[[email protected] ~]$ sudo yum install confluent-platform-oss-2.11


Dependencies Resolved

Package Arch Version Repository Size
confluent-platform-oss-2.11 noarch 3.3.0-1 Confluent 6.7 k
Installing for dependencies:
confluent-camus noarch 3.3.0-1 Confluent 22 M
confluent-cli noarch 3.3.0-1 Confluent 15 k
confluent-common noarch 3.3.0-1 Confluent 2.4 M
confluent-kafka-2.11 noarch Confluent 42 M
confluent-kafka-connect-elasticsearch noarch 3.3.0-1 Confluent 4.3 M
confluent-kafka-connect-hdfs noarch 3.3.0-1 Confluent 91 M
confluent-kafka-connect-jdbc noarch 3.3.0-1 Confluent 6.0 M
confluent-kafka-connect-s3 noarch 3.3.0-1 Confluent 4.7 M
confluent-kafka-connect-storage-common noarch 3.3.0-1 Confluent 88 M
confluent-kafka-rest noarch 3.3.0-1 Confluent 19 M
confluent-rest-utils noarch 3.3.0-1 Confluent 7.3 M
confluent-schema-registry noarch 3.3.0-1 Confluent 27 M

Transaction Summary
Install 1 Package (+12 Dependent packages)

Downloading packages:
(1/13): confluent-cli-3.3.0-1.noarch.rpm | 15 kB 00:00:00
(2/13): confluent-common-3.3.0-1.noarch.rpm | 2.4 MB 00:00:00
(3/13): confluent-camus-3.3.0-1.noarch.rpm | 22 MB 00:00:06
(4/13): confluent-kafka-connect-elasticsearch-3.3.0-1.noarch.rpm | 4.3 MB 00:00:01
(5/13): confluent-kafka-2.11- | 42 MB 00:00:16
(6/13): confluent-kafka-connect-jdbc-3.3.0-1.noarch.rpm | 6.0 MB 00:00:03
(7/13): confluent-kafka-connect-s3-3.3.0-1.noarch.rpm | 4.7 MB 00:00:01
(8/13): confluent-kafka-connect-hdfs-3.3.0-1.noarch.rpm | 91 MB 00:00:28
(9/13): confluent-kafka-rest-3.3.0-1.noarch.rpm | 19 MB 00:00:07
(10/13): confluent-platform-oss-2.11-3.3.0-1.noarch.rpm | 6.7 kB 00:00:00
(11/13): confluent-kafka-connect-storage-common-3.3.0-1.noarch.rpm | 88 MB 00:00:23
(12/13): confluent-rest-utils-3.3.0-1.noarch.rpm | 7.3 MB 00:00:02
(13/13): confluent-schema-registry-3.3.0-1.noarch.rpm | 27 MB 00:00:04
Total 6.1 MB/s | 314 MB 00:00:51
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : confluent-common-3.3.0-1.noarch 1/13
Installing : confluent-rest-utils-3.3.0-1.noarch 2/13
Installing : confluent-kafka-connect-storage-common-3.3.0-1.noarch 3/13
Installing : confluent-kafka-connect-s3-3.3.0-1.noarch 4/13
Installing : confluent-schema-registry-3.3.0-1.noarch 5/13
Installing : confluent-kafka-rest-3.3.0-1.noarch 6/13
Installing : confluent-kafka-connect-hdfs-3.3.0-1.noarch 7/13
Installing : confluent-kafka-connect-jdbc-3.3.0-1.noarch 8/13
Installing : confluent-kafka-connect-elasticsearch-3.3.0-1.noarch 9/13
Installing : confluent-kafka-2.11- 10/13
Installing : confluent-cli-3.3.0-1.noarch 11/13
Installing : confluent-camus-3.3.0-1.noarch 12/13
Installing : confluent-platform-oss-2.11-3.3.0-1.noarch 13/13
Verifying : confluent-camus-3.3.0-1.noarch 1/13
Verifying : confluent-cli-3.3.0-1.noarch 2/13
Verifying : confluent-schema-registry-3.3.0-1.noarch 3/13
Verifying : confluent-kafka-connect-hdfs-3.3.0-1.noarch 4/13
Verifying : confluent-common-3.3.0-1.noarch 5/13
Verifying : confluent-rest-utils-3.3.0-1.noarch 6/13
Verifying : confluent-kafka-connect-storage-common-3.3.0-1.noarch 7/13
Verifying : confluent-kafka-connect-s3-3.3.0-1.noarch 8/13
Verifying : confluent-platform-oss-2.11-3.3.0-1.noarch 9/13
Verifying : confluent-kafka-rest-3.3.0-1.noarch 10/13
Verifying : confluent-kafka-connect-jdbc-3.3.0-1.noarch 11/13
Verifying : confluent-kafka-connect-elasticsearch-3.3.0-1.noarch 12/13
Verifying : confluent-kafka-2.11- 13/13

confluent-platform-oss-2.11.noarch 0:3.3.0-1

Dependency Installed:
confluent-camus.noarch 0:3.3.0-1 confluent-cli.noarch 0:3.3.0-1
confluent-common.noarch 0:3.3.0-1 confluent-kafka-2.11.noarch 0:
confluent-kafka-connect-elasticsearch.noarch 0:3.3.0-1 confluent-kafka-connect-hdfs.noarch 0:3.3.0-1
confluent-kafka-connect-jdbc.noarch 0:3.3.0-1 confluent-kafka-connect-s3.noarch 0:3.3.0-1
confluent-kafka-connect-storage-common.noarch 0:3.3.0-1 confluent-kafka-rest.noarch 0:3.3.0-1
confluent-rest-utils.noarch 0:3.3.0-1 confluent-schema-registry.noarch 0:3.3.0-1


As the binaries are installed we’re able to launch Kafka and Zookeper 🙂

Check the Zookeeper configuration:

[[email protected] ~]$ cat /etc/kafka/
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
# the port at which the clients will connect
# disable the per-ip limit on the number of connections since this is a non-production config


For a lab environment the above setting should be fine no need to change anything.

So now we could start zookeeper

[[email protected] ~]$ sudo zookeeper-server-start /etc/kafka/
[2017-10-20 12:56:08,253] INFO binding to port (org.apache.zookeeper.server.NIOServerCnxnFactory)

As Zookeeper is running we could now start Kafka.
Therefore leave the current session running and open a new ssh or terminal session.

[[email protected] ~]$ sudo kafka-server-start /etc/kafka/[[email protected] ~]$ sudo kafka-server-start /etc/kafka/
[2017-10-20 12:59:47,386] INFO KafkaConfig values: = null 
advertised.listeners = null 
advertised.port = null = null = 
auto.create.topics.enable = true

[2017-10-20 12:59:48,745] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2017-10-20 12:59:48,809] INFO Waiting 10092 ms for the monitored broker to finish starting up... (
[2017-10-20 12:59:58,904] INFO Monitored broker is now ready (
[2017-10-20 12:59:58,904] INFO Starting metrics collection from monitored broker... (

(output shortened)

Now Kafka is running and we are able to create a new topic.

To check Kafka you could run

[[email protected] lib]$ kafka-topics --list --zookeeper localhost:2181

The output should be similar to (at least no error should occur 😉 )

To create a new topic leave the current terminal or ssh session running an start another terminal or ssh session.

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic mytopic

creates the topic “mytopic”

Topic creation is also visible in Kafka brokers standard out and log:

[2017-10-20 13:04:00,186] INFO Partition [mytopic,0] on broker 0: No checkpointed highwatermark is found for partition mytopic-0 (kafka.cluster.Partition)
[2017-10-20 13:04:00,189] INFO Partition [mytopic,0] on broker 0: mytopic-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)

list the newly created topic

[[email protected] ~]$ kafka-topics --list --zookeeper localhost:2181

Now we could produce some message in the topic mytopic.

[[email protected] ~]$ kafka-console-producer --broker-list localhost:9092 --topic mytopic

>be curios
>learn kafka
>have fun

Leave the producer session running and open a new terminaL or ssh session.
We could run a consumer session which shows all the message of the topic from the beginning

[[email protected] ~]$ kafka-console-consumer --zookeeper localhost:2181 --topic mytopic --from-beginning
>be curios
>learn kafka
>have fun

Also every new message you type in the producer session is visible immediate to your consumer session.

To stop the the producer, consumer and the cluster itself just press CTRL+C.

Hope you enjoyed this post about how to create a small Kafka cluster.

Next post will follow soon 😉

Further reading


The post Apache Kafka installation on Linux appeared first on