The Kafka feed type in ArcGIS Velocity subscribes to and consumes messages from an externally accessible Kafka broker. Kafka is an open-source distributed streaming platform that you can use to do the following:
- Publish and subscribe to a stream of messages through topics.
- Reliably store streams of messages.
- Process streams of messages.
For more information about Kafka, see the Apache Kafka documentation.
Examples
- A data analyst wants to ingest sensor data through a Kafka broker.
- A transportation department configures a Kafka feed to consume vehicle positions and attribute information.
Usage notes
- The Kafka broker must be externally accessible on the internet.
- This feed type supports connectivity to Confluent Kafka.
- A single consumer group should not be used for multiple topics. Create a consumer group for each topic.
- If there are multiple consumers (a Velocity Kafka feed is a consumer) using a single consumer group to consume data from a topic, each consumer (or feed) only get part of the data depending upon the number of partitions of the topic.
- If there are more consumers of a topic using the same consumer group than the number of partitions, some consumers will not get any data.
- Do not use the same consumer group referenced by an Velocity Kafka feed outside of Velocity.
- After configuring feed connection parameters, see Configure input data to learn how to define the schema and the key parameters.
Parameters
| Parameter | Description | Data type |
|---|---|---|
Broker | A comma-delimited list of Kafka brokers specified as server:port, for example, machine1.domain.com:9092,machine2.domain.com:9092. | String |
Topic | Specifies the Kafka topic from which Velocity will consume messages. | String |
Consumer group ID (optional) | Specifies the Kafka consumer group Velocity will join as a member. | String |
Authentication | Specifies the type of authentication to access the HTTP endpoint. Authentication options are None, SASL/PLAIN, SASL/SCRAM-SHA-256, and SASL/SCRAM-SHA-512. | String |
Use SSL | Species whether Velocity will use SSL (SASL_SSL) in attempting to connect to the Kafka broker. Note:This should match the configuration the broker has on a specified port. If disabled, Velocity will connect via PLAINTEXT. | Boolean |
Username | Specifies the user name for accessing the Kafka broker. This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512. | String |
Password | Specifies the password for accessing the Kafka broker. This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512. | String |
Considerations and limitations
The Kafka broker must be externally accessible on the internet.