Kafka on OpenShift with External Routes

In this blog post, we will learn how to deploy Kafka on OpenShift and make it accessible externally outside of the OpenShift cluster.

Step: 1 Deploy Strimzi Operator

oc new-project kafka-demo
oc apply -f 'https://strimzi.io/install/latest?namespace=kafka-demo' -n kafka-demo
oc get all -n kafka-demo

More details here

Step:2 Deploy Kafka Cluster

• Before applying the manifest file make sure you have default storage class in your OpenShift environment. If you do not have that, you can remove this section to deploy Kafka on ephemeral storage (not recommended for prod)
• Please note that spec > kafka > listeners > external > type: route is important to access Kafka brokers from outside OpenShfit.

oc apply -f https://gist.githubusercontent.com/ksingh7/61d5a62c9885078719cc16b260d107c9/raw/7b4f90259877f73850f356fd7a7e35b1a08f1e00/01_kafka_cluster.yaml
oc get all -n kafka-demo

• Wait for Kafka cluster to get ready

oc wait kafka/my-cluster --for=condition=Ready --timeout=300s -n kafka-demo

Step:3 Prepare to access Kafka externally

• Check OpenShift routes

oc get route --selector=app=my-cluster -n kafka-demo

• Get the correct route host

oc get -n kafka-demo routes my-cluster-kafka-bootstrap -o=jsonpath='{.status.ingress[0].host}{"\n"}'

• Since it will always use TLS, you will always have to configure TLS in your Kafka clients. This includes getting the TLS certificate from the broker and configuring it in the client

oc extract -n kafka-demo secret/my-cluster-cluster-ca-cert --keys=ca.crt --to=- > ca.crt 
keytool -import -trustcacerts -alias root -file ca.crt -keystore truststore.jks -storepass password -noprompt

• Get kafka console producer / consumer binaries to interact with your kafka cluster
• For console producer, remember to use <OpenShift Route endpoint for kaka>:443 as the broker-list address

kafka-console-producer --broker-list my-cluster-kafka-bootstrap-kafka-demo.apps.data-pipeline.ceph-s3.com:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./truststore.jks --topic my-topic

• For console consumer

kafka-console-consumer --bootstrap-server my-cluster-kafka-bootstrap-kafka-demo.apps.data-pipeline.ceph-s3.com:443 --consumer-property security.protocol=SSL --consumer-property ssl.truststore.password=password --consumer-property ssl.truststore.location=./truststore.jks --topic my-topic

For more details on Kafka OpenShift Routes, check out this blog from Jakub Scholz

That’s all Folks, Happy Messaging