Kafka Security Mechanism (SASL/PLAIN)_kafka mechanism

Starting from Kafka 0.10.x Kafka Broker supports username/password authentication. This Mechanism is called SASL/PLAIN. The steps below describe how to set up this mechanism on an IOP 4.2.5 Kafka Cluster.

1. Configure the Kafka brokers and Kafka Clients

Add a JAAS configuration file for each Kafka broker. Create a kafka_plain_jaas.conf file as specified below:

KafkaServer {  
org.apache.kafka.common.security.plain.PlainLoginModule required
   username="kafka"
   password="kafka-secret"
   user_kafka="kafka-secret"
   user_ibm="ibm-secret";
};
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
  username="ibm"
  password="ibm-secret";
};

Let’s understand the content of kafka_plain_jaas.conf file and how Kafka Brokers and Kafka Clients use it.

KafkaServer Section:
The KafkaServer section defines two users: kafka and ibm. The properties username and password are used by the broker to initiate connections to other brokers. In this example, kafka is the user for inter-broker communication. The set of properties user_{userName} defines the passwords for all users that connect to the broker and the broker validates all client connections including those from other brokers using these properties.

KafkaClient Section:
The Kafka Client section describes how the clients, Producer and Consumer, can connect to Kafka Broker. The properties username and password in the Kafka Client section are used by clients to configure the user for client connections. In this example, clients connect to the broker as user “ibm”.

2. Pass the JAAS configuration location as a JVM Parameter to the Kafka broker

 
Update the Advanced kafka-env section to add the properties below:
export KAFKA_PLAIN_PARAMS="-Djava.security.auth.login.config=/usr/iop/4.2.5.0-0000/kafka/conf/kafka_plain_jaas.conf"
export KAFKA_OPTS="$KAFKA_PLAIN_PARAMS $KAFKA_OPTS"
  		

3. Add the properties below to the custom Kafka broker configuration.

4. Update the listeners property in the Kafka Broker section to SASL_PLAINTEXT://localhost:6667

5. Restart Kafka.

Run the Kafka Client Console Producer / Console Consumer to authenticate to the Kafka broker with the specified username and password. In this example, the username used to connect to brokers is “ibm”. The JVM parameter java.security.auth.login.config for console producer and console consumer is set by the Step 2 above.
All the commands shown below are run from the path /usr/iop/4.2.5.0-0000/kafka

Create a topic:

[root@heel1 kafka]# bin/kafka-topics.sh --create --topic plain-topic --zookeeper heel1.fyre.ibm.com:2181 --partitions 1 --replication-factor 1
[2017-04-10 12:06:22,239] WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/iop/4.2.5.0-0000/kafka/conf/kafka_plain_jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
Created topic "plain-topic".

Run Kafka console producer

Before running the Kafka console Producer configure the producer.properties file as shown:

[root@heel1 kafka]# cat producer.properties 
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

[root@heel1 kafka]# bin/kafka-console-producer.sh --broker-list heel1.fyre.ibm.com:6667 --topic plain-topic --producer.config producer.properties 
Message 1
Message 2
Message 3
^C

Run Kafka console consumer

Before running Kafka console consumer configure the consumer.properties file as shown:

[root@heel1 kafka]# cat consumer.properties 
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

[root@heel1 kafka]# bin/kafka-console-consumer.sh --bootstrap-server heel1.fyre.ibm.com:6667 --topic plain-topic --from-beginning --consumer.config consumer.properties 
Message 1
Message 2
Message 3
^CProcessed a total of 3 messages

SASL/PLAIN with ACLs:
The username is used as the authenticated Principal, which is used in authorization (e.g. ACLs).
For usage of ACLS refer to the blog Kafka ACLs

For more information refer to the kafka documentation