Kafka

The Kafka feed type in ArcGIS Velocity subscribes to and consumes messages from an externally accessible Kafka broker. Kafka is an open-source distributed streaming platform that you can use to do the following:

  • Publish and subscribe to a stream of messages through topics.
  • Reliably store streams of messages.
  • Process streams of messages.

For more information about Kafka, see the Apache Kafka documentation.

Examples

  • A data analyst wants to ingest sensor data through a Kafka broker.
  • A transportation department configures a Kafka feed to consume vehicle positions and attribute information.

Usage notes

  • The Kafka broker must be externally accessible on the internet.
  • This feed type supports connectivity to Confluent Kafka.
  • After configuring feed connection parameters, see Configure input data to learn how to define the schema and the key parameters.

Parameters

ParameterDescriptionData type

Broker

A comma-delimited list of Kafka brokers specified as server:port, for example, machine1.domain.com:9092,machine2.domain.com:9092.

String

Topic

Specifies the Kafka topic from which Velocity will consume messages.

String

Consumer group ID

(optional)

Specifies the Kafka consumer group Velocity will join as a member.

String

Authentication

Specifies the type of authentication to access the HTTP endpoint. Authentication options are None, SASL/PLAIN, SASL/SCRAM-SHA-256, and SASL/SCRAM-SHA-512.

String

Use SSL

Species whether Velocity will use SSL (SASL_SSL) in attempting to connect to the Kafka broker.

Note:

This should match the configuration the broker has on a specified port. If disabled, Velocity will connect via PLAINTEXT.

Boolean

Username

Specifies the user name for accessing the Kafka broker.

This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512.

String

Password

Specifies the password for accessing the Kafka broker.

This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512.

String

Considerations and limitations

The Kafka broker must be externally accessible on the internet.