Work with input data

Configure input dataIngest real-time dataIngest historical dataLocation parametersDate and time parametersFeed polling interval schedulingFeed to history feature layer

Feeds

Work with feedsFeed details and statusShare feedsADS-B ExchangeAWS IoTAzure Event HubAzure Service BusBaron WeatherCisco Edge IntelligenceDTN WeatherGeotab (DeviceStatusInfo)Geotab (LogRecord)gRPCGTFSHAAS AlertFeature LayerFlightAware (Airborne)FlightAware (Surface)HTTP PollerHTTP ReceiverHTTP SimulatorKafkaKplerMotive AVLMQTTPreCise MRMRabbitMQRSSSamsaraStream LayerTAK Client (Beta)TracPlusVectronic AerospaceVerizon Connect RevealWazeWebSocket

Data sources

Work with data sourcesAmazon S3Azure Blob StorageAzure Cosmos DBFeature LayerFeature Layer (archive)HTTP PollerRSSStandard geographies

Data formats

Supported data formatsDelimitedJSONEsri JSONGeoJSONXML

Troubleshoot

Common ingestion issues and solutions
Back to Top

Kafka

In this topic

  1. Examples
  2. Usage notes
  3. Parameters
  4. Consideration and limitation

The Kafka feed type in ArcGIS Velocity subscribes to and consumes messages from an externally accessible Kafka broker. Kafka is an open-source distributed streaming platform that you can use to do the following:

  • Publish and subscribe to a stream of messages through topics.
  • Reliably store streams of messages.
  • Process streams of messages.

Learn more about Apache Kafka

Examples

The following are example use cases for the feed:

  • A data analyst wants to ingest sensor data through a Kafka broker.
  • A transportation department configures a Kafka feed to consume vehicle positions and attribute information.

Usage notes

Keep the following in mind when working with the feed:

  • This feed type supports connectivity to Confluent Kafka.
  • Do not use a single consumer group for multiple topics. Create a consumer group for each topic.
  • If there are multiple consumers (a Velocity Kafka feed is a consumer) using a single consumer group to consume data from a topic, each consumer (or feed) only receives part of the data depending on the number of partitions of the topic.
  • If there are more consumers of a topic using the same consumer group than the number of partitions, some consumers do not receive any data.
  • Do not use the same consumer group referenced by a Velocity Kafka feed outside of Velocity.
  • After configuring feed connection parameters, configure input data to define the schema and the key parameters.

Parameters

The following are the parameters for the feed:

ParameterDescriptionData type
Broker

A comma-delimited list of Kafka brokers specified as server:port, for example, machine1.domain.com:9092,machine2.domain.com:9092.

String

Topic

The Kafka topic from which Velocity consumes messages.

String

Consumer group ID (optional)

The Kafka consumer group Velocity joins as a member.

String

Authentication

Specifies the type of authentication that is used to access the HTTP endpoint. Authentication options are None, SASL/PLAIN, SASL/SCRAM-SHA-256, and SASL/SCRAM-SHA-512.

String

Use SSL

Species whether SSL (SASL_SSL) is used when connecting to the Kafka broker.

Note:

This should match the configuration the broker has on a specified port. If false, Velocity connects via PLAINTEXT.

Boolean

Username

The username for accessing the Kafka broker.

This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512.

String

Password

The password for accessing the Kafka broker.

This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512.

String

Consideration and limitation

Consider the following when using the feed:

The Kafka broker must be externally accessible on the internet.

Feedback on this topic?

In this topic
  1. Examples
  2. Usage notes
  3. Parameters
  4. Consideration and limitation