Skip to main content

Connect Apache Kafka

Codefi Orchestrate uses Apache Kafka as message broker and streaming platform used to listen for transactions. Follow these steps to configure Kafka when starting the following services:

tip

Configure each microservice using microservice-specific environment variables. Command line options are also available and take precedence over environment variables.

Global Configuration

Environment VariableCommand line optionDescriptionDefault
KAFKA_URLkafka-urlList of URLs of Kafka servers to connect tolocalhost:9092
KAFKA_VERSIONkafka-versionVersion of Kafka that Orchestrate will assume it is running against1.0.0

SSL/TLS Configuration

In addition to the options above, follow these steps to configure Kafka over TLS communication.

Environment VariableCommand line optionDescriptionDefault
KAFKA_TLS_ENABLEDkafka-tls-enabledWhether or not to use TLSfalse
KAFKA_TLS_INSECURE_SKIP_VERIFYkafka-tls-insecure-skip-verifyWhether the client verifies the server's certificate chain and host namefalse
KAFKA_TLS_CLIENT_CERT_FILEkafka-tls-client-cert-fileTLS certificate file path
KAFKA_TLS_CLIENT_KEY_FILEkafka-tls-client-key-fileTLS key file path
KAFKA_TLS_CA_CERT_FILEkafka-tls-ca-cert-fileTLS Certificate Authority file path

When using TLS communication in production, it is not recommended to set kafka-tls-insecure-skip-verify to true as that will open a breach to man-in-the-middle attacks.

SASL Configuration

In addition to the options above, follow these steps to configure Kafka SASL authentication.

Environment VariableCommand line optionDescriptionDefault
KAFKA_SASL_ENABLEDkafka-sasl-enabledWhether or not to use SASL authenticationfalse
KAFKA_SASL_MECHANISMkafka-sasl-mechanismSASL mechanism (PLAIN, OAUTHBEARER)PLAIN
KAFKA_SASL_HANDSHAKEkafka-sasl-handshakeWhether or not to send the Kafka SASL handshake first. You should only set this to false if you're using a non-Kafka SASL proxy.true
KAFKA_SASL_USERkafka-sasl-userUsername for SASL/PLAIN or SASL/SCRAM authentication
KAFKA_SASL_PASSWORDkafka-sasl-passwordPassword for SASL/PLAIN or SASL/SCRAM authentication
KAFKA_SASL_SCRAMAUTHZIDkafka-sasl-scramauthzidID used for SASL/SCRAM authentication

Configuration: Orchestrate Gateway API

Specific Kafka Configuration options when starting Orchestrate Gateway API. You can use the default values if you are not using multiple instances of Orchestrate.

Environment VariableCommand line optionDescriptionDefault
TOPIC_TX_SENDERtopic-tx-senderTopic for messages between the API and the Tx-Sendertopic-tx-sender

Configuration: Orchestrate Transaction Sender

Specific Kafka Configuration options when starting Orchestrate Transaction Sender in addition to the global Kafka options.

Apache Kafka is certainly the most delicate of Orchestrate's dependencies, we recommend using the default values unless there is a need to modify them for advance usage.

Environment VariableCommand line optionDescriptionDefault
KAFKA_CONSUMER_GROUP_NAMEconsumer-group-nameKafka consumer group namegroup-sender
TOPIC_TX_RECOVERtopic-tx-recoverTopic for failed transaction messagestopic-tx-recover
TOPIC_TX_SENDERtopic-tx-senderTopic for messages between the API and the Tx-Sendertopic-tx-sender
KAFKA_CONSUMER_MAX_WAIT_TIMEkafka-consumer-max-wait-timeMaximum amount of time the broker will wait in ms250ms
KAFKA_CONSUMER_MAXPROCESSINGTIMEkafka-consumer-max-processing-timeMaximum amount of time in ms the consumer expects a message to be processed100ms
KAFKA_CONSUMER_GROUP_SESSION_TIMEOUTkafka-consumer-group-session-timeoutTimeout in seconds used to detect consumer failures when using Kafka's group management facility10s
KAFKA_CONSUMER_GROUP_HEARTBEAT_INTERVALkafka-consumer-group-heartbeat-intervalExpected time in seconds between heartbeats to the consumer coordinator when using Kafka's group management facilities3s
KAFKA_CONSUMER_GROUP_HEARTBEAT_INTERVALkafka-consumer-group-heartbeat-intervalExpected time between heartbeats to the consumer coordinator when using Kafka's group management facilities3s
KAFKA_CONSUMER_GROUP_REBALANCE_TIMEOUTkafka-consumer-group-rebalance-timeoutMaximum allowed time in seconds for each worker to join the group once a re-balance has started60s
KAFKA_CONSUMER_GROUP_REBALANCE_STRATEGYkafka-consumer-group-rebalance-strategyStrategy for allocating topic partitions to members (Range, RoundRobin, Sticky)Range

Configuration: Orchestrate Transaction Listener

Specific Kafka Configuration options when starting Orchestrate Transaction Listener in addition to the global Kafka options. You can use the default values if you are not using multiple instances of Orchestrate.

Environment VariableCommand line optionDescriptionDefault
TOPIC_TX_DECODEDtopic-tx-decodedTopic for successful transaction messages (receipts)topic-tx-decoded