The Kafka feed type in ArcGIS Velocity subscribes to and consumes messages from an externally accessible Kafka broker. Kafka is an open-source distributed streaming platform that you can use to do the following:
- Publish and subscribe to a stream of messages through topics.
- Reliably store streams of messages.
- Process streams of messages.
For more information about Kafka, see the Apache Kafka documentation.
Examples
The following are example uses of the Kafka feed:
- A data analyst wants to ingest sensor data through a Kafka broker.
- A transportation department configures a Kafka feed to consume vehicle positions and attribute information.
Usage notes
Keep the following in mind when working with the Kafka feed:
- This feed type supports connectivity to Confluent Kafka.
- Don't use a single consumer group for multiple topics. Create a consumer group for each topic.
- If there are multiple consumers (a Velocity Kafka feed is a consumer) using a single consumer group to consume data from a topic, each consumer (or feed) only receives part of the data depending on the number of partitions of the topic.
- If there are more consumers of a topic using the same consumer group than the number of partitions, some consumers will not receive any data.
- Do not use the same consumer group referenced by a Velocity Kafka feed outside of Velocity.
- After configuring feed connection parameters, see Configure input data to learn how to define the schema and the key parameters.
Parameters
The following are the parameters for the Kafka feed:
| Parameter | Description | Data type |
|---|---|---|
Broker | A comma-delimited list of Kafka brokers specified as server:port, for example, machine1.domain.com:9092,machine2.domain.com:9092. | String |
Topic | The Kafka topic from which Velocity will consume messages. | String |
Consumer group ID (optional) | The Kafka consumer group Velocity will join as a member. | String |
Authentication | Specifies the type of authentication that will be used to access the HTTP endpoint. Authentication options are None, SASL/PLAIN, SASL/SCRAM-SHA-256, and SASL/SCRAM-SHA-512. | String |
Use SSL | Species whether SSL (SASL_SSL) will be used when connecting to the Kafka broker. Note:This should match the configuration the broker has on a specified port. If false, Velocity will connect via PLAINTEXT. | Boolean |
Username | The username for accessing the Kafka broker. This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512. | String |
Password | The password for accessing the Kafka broker. This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512. | String |
Considerations and limitations
The Kafka broker must be externally accessible on the internet.