Testing

testing upload/download locally

This project uses dockerenv to run the tests against a ‘real’ running kafka instance.

When testing, either manually or via automated tests, it can be helpful to inspect the data for a particular topic.

For example, in the Kafka4mAppTest, once data is written to some unique topic, you might want to ssh onto your locally running kafka instance and inspect the data:

Assuming the Kafka4mApp created topic ‘kafka4m-example’:

docker exec -it  test-kafka /bin/bash
cd /opt/kafka_2.12-2.2.0/bin/
./kafka-topics.sh --list  --zookeeper localhost:2181 | grep kafka4m-example
./kafka-console-consumer.sh  --topic kafka4m-example  --bootstrap-server localhost:9092 --from-beginning

Or simply:

docker exec test-kafka /opt/kafka_2.12-2.2.0/bin/kafka-topics.sh --list  --zookeeper localhost:2181

Testing using docker-compose

The src/test/resources/docker-compose-kafka holds a clone of the wonderful kafka docker-compose work by wurstmeister/kafka-docker

The docker-compose.yml there has been modified to include the Kafka4m ETL job. Assuming we’ve run:

sbt docker

to build the kafka4m image, we can then run:

cd docker-compose-kafka/
docker-compose  up -d 

Which will start zookeeper/kafka, then run an ETL job which writes the contents of that example directory into kafka under the topic ‘kafka4m-example’.

We can then verify that by executing:

docker exec  docker-compose-kafka_kafka_1 /opt/kafka_2.12-2.3.0/bin/kafka-topics.sh --list  --zookeeper zookeeper:2181

to see the topic, or to see the imported records:

docker exec docker-compose-kafka_kafka_1 \
  /opt/kafka_2.12-2.3.0/bin/kafka-console-consumer.sh \
  --topic kafka4m-example \
  --bootstrap-server \
  localhost:9092 \
  --from-beginning

Where you should see the contents of the imported data

Testing On a Kafka Cluster

In a similar way to our docker-compose example, you could fork kubernetes-kafka as I’ve done here and from a gcloud shell run:

git clone https://github.com/aaronp/kubernetes-kafka .
cd kubernetes-kafka

kubectl apply -f namespace
kubectl apply -f rbac-namespace-default
kubectl apply -f zookeeper
kubectl apply -f kafka

And then add/run a kafka4m ETL pod, either as a side-car to the kafka container or in a separate pod.