Kafka

The Message Bus Probe can be configured to integrate with a Kafka server to consume events.

The Message Bus Probe connects to the Kafka server using the Kafka transport. This enables the probe support the Kafka Client version 2.3.1 and Zookeeper version 3.4.14.

Check the Apache Kafka compatibility matrix for the support of the target system with respect to the dependency.

Configuring the Message Bus Probe to consume new events from Kafka

The following configuration files are supplied with the probe for the integration with Kafka:

message_bus_kafka.props
message_bus_kafka.rules
kafkaTransport.properties
kafkaConnectionProperties.json
kafkaClient.properties

To configure the Message Bus Probe to consume new events from Kafka, use the following steps:

Install/update the Message Bus Probe using IBM Installation Manager, see Installing probes.
Edit the probe configuration in the following probe properties file:
$OMNIHOME/probes/<arch>/message_bus_kafka.props

Where <arch> is the architecture directory, for example linux2x86

Update the following property values with the appropriate path:

Manager           : 'Kafka'
MessageLog        : '$OMNIHOME/log/message_bus_kafka.log'
PropsFile         : '$OMNIHOME/probes/linux2x86/message_bus_kafka.props'
RulesFile         : '$OMNIHOME/probes/linux2x86/message_bus_kafka.rules'
TransportType     : 'KAFKA'
TransportFile     : '$OMNIHOME/java/conf/kafkaTransport.properties'
TransformerFile   : '$OMNIHOME/probes/linux2x86/message_bus_parser_config.json'
MessagePayload    : 'JSON'

Configure the Kafka transport properties.
1. Edit the Kafka transport configuration in the following transport properties file:
  $OMNIHOME/probes/java/conf/kafkaTransport.properties
2. Update the following property value with the appropriate path:
```
kafkaClientMode=CONSUMER
connectionPropertiesFile=$OMNIHOME/java/conf/kafkaConnectionProperties.json
```
For descriptions of the Kafka transport properties, see the Configuring the Kafka transport.

Configure the Kafka connection properties.

Kafka connection properties are defined in the kafkaConnectionProperties.json file. This file contains the following properties:

{
    "zookeeper_client" : 
        {
            "target" : "",
            "properties" : "",
            "java_sys_props" : "",
            "topic_watch": true,
            "broker_watch": true
        },
    "brokers" : "",
    "topics": "",
    "kafka_client" : 
        {
            "properties" : "",
            "java_sys_props" : ""
        }
     }

The ZooKeeper event access is required.

Within the sample configuration file supplied with the probe, update the path to the ZooKeeper client properties file.

    "zookeeper_client" : 
        {
            "target" : "localhost:2181",
            "properties" : "<Path to zookeeper client properties file>",
            "java_sys_props" : "",
            "topic_watch": true,
            "broker_watch": true
        },

Update the path to the Kafka client properties file:

    "brokers" : "PLAINTEXT://localhost:9092",
    "topics": "topicABC,topicXYZ",
    "kafka_client" : 
        {
            "properties" : "<omnihome_path>/java/conf/kafkaClient.properties",
            "java_sys_props" : ""	
        }

Specify additional configuration to use with SASL authentication, if SASL authentication is required.

Kafka brokers supports client authentication using SASL. Additional configuration settings are required to authenticate with SASL.

Enable the following properties in the Kafka connection properties file (kafkaConnectionProperties.json)
"java_sys_props" : "<Path to a text file containing java system properties configuration>"

Example:

"java_sys_props" : "java.security.auth.login.config=C:\\IBM\\Tivoli\\Netcool\\omnibus\\java\\conf\\java_sys_prop.conf"

Create and enable the following properties in the Java system properties file (java_sys_prop.conf):

Example:

KafkaClient {
      org.apache.kafka.common.security.plain.PlainLoginModule required
      serviceName="kafka"
      username="wfE7hGteeNl4i9JJ"
      password="xtPqsWiPXN4cSwi6h2BE4GbaJ3uheal2";
   };

Enable and set the following properties in the Kafka client properties file (kafkaClient.properties)

security.protocol
     ssl.enabled.protocols

     ssl.keystore.location
     ssl.keystore.password
     ssl.keystore.type
     
     Included when trust stores are in use.
     ssl.truststore.location
     ssl.truststore.password
     ssl.truststore.type

Example:

security.protocol=SASL_SSL
     ssl.enabled.protocols=TLSv1.2 

     ssl.keystore.location=C:\\Programs\\ibm-java-sdk-80-win-x86_64\\sdk
\\jre\\lib\\security\\cacerts
     ssl.keystore.password=changeit
     ssl.keystore.type=JKS

     ssl.truststore.location=C:\\Programs\\ibm-java-sdk-80-win-x86_64\\sdk
\\jre\\lib\\security\\cacerts
     ssl.truststore.password=changeit
     ssl.truststore.type=JKS

For descriptions of the Kafka connection properties, see the Configuring the Kafka transport.
Start the probe with the propsfile option to specify the Kafka properties file using the following command:
$OMNIHOME/probes/nco_p_message_bus -propsfile $OMNIHOME/probes/linux2x86/message_bus_kafka.props
Verify that the probe is running.