Connect Apache Kafka
Codefi Orchestrate uses Apache Kafka as message broker and streaming platform used to listen for transactions. Follow these steps to configure Kafka when starting the following services:
Configure each microservice using microservice-specific environment variables. Command line options are also available and take precedence over environment variables.
Global Configuration
| Environment Variable | Command line option | Description | Default |
|---|---|---|---|
KAFKA_URL | kafka-url | List of URLs of Kafka servers to connect to | localhost:9092 |
KAFKA_VERSION | kafka-version | Version of Kafka that Orchestrate will assume it is running against | 1.0.0 |
SSL/TLS Configuration
In addition to the options above, follow these steps to configure Kafka over TLS communication.
| Environment Variable | Command line option | Description | Default |
|---|---|---|---|
KAFKA_TLS_ENABLED | kafka-tls-enabled | Whether or not to use TLS | false |
KAFKA_TLS_INSECURE_SKIP_VERIFY | kafka-tls-insecure-skip-verify | Whether the client verifies the server's certificate chain and host name | false |
KAFKA_TLS_CLIENT_CERT_FILE | kafka-tls-client-cert-file | TLS certificate file path | |
KAFKA_TLS_CLIENT_KEY_FILE | kafka-tls-client-key-file | TLS key file path | |
KAFKA_TLS_CA_CERT_FILE | kafka-tls-ca-cert-file | TLS Certificate Authority file path |
When using TLS communication in production, it is not recommended to set kafka-tls-insecure-skip-verify to true as that will open a breach to man-in-the-middle attacks.
SASL Configuration
In addition to the options above, follow these steps to configure Kafka SASL authentication.
| Environment Variable | Command line option | Description | Default |
|---|---|---|---|
KAFKA_SASL_ENABLED | kafka-sasl-enabled | Whether or not to use SASL authentication | false |
KAFKA_SASL_MECHANISM | kafka-sasl-mechanism | SASL mechanism (PLAIN, OAUTHBEARER) | PLAIN |
KAFKA_SASL_HANDSHAKE | kafka-sasl-handshake | Whether or not to send the Kafka SASL handshake first. You should only set this to false if you're using a non-Kafka SASL proxy. | true |
KAFKA_SASL_USER | kafka-sasl-user | Username for SASL/PLAIN or SASL/SCRAM authentication | |
KAFKA_SASL_PASSWORD | kafka-sasl-password | Password for SASL/PLAIN or SASL/SCRAM authentication | |
KAFKA_SASL_SCRAMAUTHZID | kafka-sasl-scramauthzid | ID used for SASL/SCRAM authentication |
Configuration: Orchestrate Gateway API
Specific Kafka Configuration options when starting Orchestrate Gateway API. You can use the default values if you are not using multiple instances of Orchestrate.
| Environment Variable | Command line option | Description | Default |
|---|---|---|---|
TOPIC_TX_SENDER | topic-tx-sender | Topic for messages between the API and the Tx-Sender | topic-tx-sender |
Configuration: Orchestrate Transaction Sender
Specific Kafka Configuration options when starting Orchestrate Transaction Sender in addition to the global Kafka options.
Apache Kafka is certainly the most delicate of Orchestrate's dependencies, we recommend using the default values unless there is a need to modify them for advance usage.
| Environment Variable | Command line option | Description | Default |
|---|---|---|---|
KAFKA_CONSUMER_GROUP_NAME | consumer-group-name | Kafka consumer group name | group-sender |
TOPIC_TX_RECOVER | topic-tx-recover | Topic for failed transaction messages | topic-tx-recover |
TOPIC_TX_SENDER | topic-tx-sender | Topic for messages between the API and the Tx-Sender | topic-tx-sender |
KAFKA_CONSUMER_MAX_WAIT_TIME | kafka-consumer-max-wait-time | Maximum amount of time the broker will wait in ms | 250ms |
KAFKA_CONSUMER_MAXPROCESSINGTIME | kafka-consumer-max-processing-time | Maximum amount of time in ms the consumer expects a message to be processed | 100ms |
KAFKA_CONSUMER_GROUP_SESSION_TIMEOUT | kafka-consumer-group-session-timeout | Timeout in seconds used to detect consumer failures when using Kafka's group management facility | 10s |
KAFKA_CONSUMER_GROUP_HEARTBEAT_INTERVAL | kafka-consumer-group-heartbeat-interval | Expected time in seconds between heartbeats to the consumer coordinator when using Kafka's group management facilities | 3s |
KAFKA_CONSUMER_GROUP_HEARTBEAT_INTERVAL | kafka-consumer-group-heartbeat-interval | Expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities | 3s |
KAFKA_CONSUMER_GROUP_REBALANCE_TIMEOUT | kafka-consumer-group-rebalance-timeout | Maximum allowed time in seconds for each worker to join the group once a re-balance has started | 60s |
KAFKA_CONSUMER_GROUP_REBALANCE_STRATEGY | kafka-consumer-group-rebalance-strategy | Strategy for allocating topic partitions to members (Range, RoundRobin, Sticky) | Range |
Configuration: Orchestrate Transaction Listener
Specific Kafka Configuration options when starting Orchestrate Transaction Listener in addition to the global Kafka options. You can use the default values if you are not using multiple instances of Orchestrate.
| Environment Variable | Command line option | Description | Default |
|---|---|---|---|
TOPIC_TX_DECODED | topic-tx-decoded | Topic for successful transaction messages (receipts) | topic-tx-decoded |