Work with input data

Configure input dataIngest real-time dataIngest historical dataLocation parametersDate and time parametersFeed polling interval schedulingFeed to history feature layer

Feeds

Work with feedsFeed details and statusShare feedsAWS IoTAzure Event HubAzure Service BusBaron WeatherCisco Edge IntelligenceGeotab (DeviceStatusInfo)Geotab (LogRecord)gRPCGTFSHAAS AlertFeature LayerFlightAware (Airborne)FlightAware (Surface)HTTP PollerHTTP ReceiverHTTP SimulatorKafkaMQTTPreCise MRMRabbitMQRSSSamsaraStream LayerVectronic AerospaceVerizon Connect RevealWazeWebSocket

Data sources

Work with data sourcesAmazon S3Azure Blob StorageAzure Cosmos DBFeature LayerFeature Layer (archive)HTTP PollerRSSStandard geographies

Data formats

Supported data formatsDelimitedJSONEsri JSONGeoJSONXML

Troubleshoot

Common ingestion issues and solutions
Back to Top

Kafka

In this topic

  1. Examples
  2. Usage notes
  3. Parameters
  4. Considerations and limitations

The Kafka feed type in ArcGIS Velocity subscribes to and consumes messages from an externally accessible Kafka broker. Kafka is an open-source distributed streaming platform that you can use to do the following:

  • Publish and subscribe to a stream of messages through topics.
  • Reliably store streams of messages.
  • Process streams of messages.

For more information about Kafka, see the Apache Kafka documentation.

Examples

The following are example uses of the Kafka feed:

  • A data analyst wants to ingest sensor data through a Kafka broker.
  • A transportation department configures a Kafka feed to consume vehicle positions and attribute information.

Usage notes

Keep the following in mind when working with the Kafka feed:

  • This feed type supports connectivity to Confluent Kafka.
  • Don't use a single consumer group for multiple topics. Create a consumer group for each topic.
  • If there are multiple consumers (a Velocity Kafka feed is a consumer) using a single consumer group to consume data from a topic, each consumer (or feed) only receives part of the data depending on the number of partitions of the topic.
  • If there are more consumers of a topic using the same consumer group than the number of partitions, some consumers will not receive any data.
  • Do not use the same consumer group referenced by a Velocity Kafka feed outside of Velocity.
  • After configuring feed connection parameters, see Configure input data to learn how to define the schema and the key parameters.

Parameters

The following are the parameters for the Kafka feed:

ParameterDescriptionData type

Broker

A comma-delimited list of Kafka brokers specified as server:port, for example, machine1.domain.com:9092,machine2.domain.com:9092.

String

Topic

The Kafka topic from which Velocity will consume messages.

String

Consumer group ID

(optional)

The Kafka consumer group Velocity will join as a member.

String

Authentication

Specifies the type of authentication that will be used to access the HTTP endpoint. Authentication options are None, SASL/PLAIN, SASL/SCRAM-SHA-256, and SASL/SCRAM-SHA-512.

String

Use SSL

Species whether SSL (SASL_SSL) will be used when connecting to the Kafka broker.

Note:

This should match the configuration the broker has on a specified port. If false, Velocity will connect via PLAINTEXT.

Boolean

Username

The username for accessing the Kafka broker.

This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512.

String

Password

The password for accessing the Kafka broker.

This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512.

String

Considerations and limitations

The Kafka broker must be externally accessible on the internet.

Feedback on this topic?

In this topic
  1. Examples
  2. Usage notes
  3. Parameters
  4. Considerations and limitations