Work with input data

Configure input dataIngest real-time dataIngest historical dataLocation parametersDate and time parametersFeed polling interval schedulingFeed to history feature layer

Feeds

Work with feedsFeed details and statusShare feedsAWS IoTAzure Event HubAzure Service BusCisco Edge IntelligenceGeotab (DeviceStatusInfo)Geotab (LogRecord)gRPCGTFSFeature LayerHTTP PollerHTTP ReceiverHTTP SimulatorKafkaMQTTRabbitMQRSSSamsaraStream LayerVerizon Connect RevealWazeWebSocket

Data sources

Work with data sourcesAmazon S3Azure Blob StorageAzure Cosmos DBFeature LayerFeature Layer (archive)HTTP PollerRSSStandard geographies

Data formats

Supported data formatsDelimitedJSONEsri JSONGeoJSONXML

Troubleshoot

Common ingestion issues and solutions
Back to Top

HTTP Poller

In this topic

  1. Example
  2. Usage notes
  3. Parameters
  4. Considerations and limitations

The HTTP Poller feed type in ArcGIS Velocity sends an HTTP request to the specified URL. This collects data that can be retrieved in a web browser by browsing to the same URL.

Example

Updated sensor values are loaded from an API every minute.

Usage notes

Keep the following in mind when working with the HTTP Poller feed:

  • The feed polls the HTTP endpoint to load data at the rate defined by the Feed Polling Interval step of the configuration wizard.
  • When a polling interval is set, the polling occurs at regular time intervals as opposed to the start time of the feed. For example, if a feed is set to poll for data every five minutes, and the feed is started at 8:03 a.m., the first request for data will occur at 8:05 a.m., the next at 8:10 a.m., and so on.
  • Use caution when specifying the Feed Polling Interval value. Ensure that the HTTP endpoint being polled is capable of returning responses within the polling interval specified. It is recommended that you test the response time of a request to the website using the same request that you apply in the feed configuration. Do not set the polling interval to a shorter interval than the typical response time of the request to the website. For details, see Schedule feed polling interval.
  • The POST body, URL parameters, and Custom headers parameters support feed global variables. By default, the feed global variables are converted to epoch time (milliseconds since January 1, 1970 UTC). Optionally, you can convert the feed global variables to epoch seconds (seconds since January 1, 1970) and a string date time format. The first time the feed runs after starting or restarting, the $feed.FeedLastScheduledStart time will be the current start time minus five minutes. The following are the feed analytic variables and the time they represent:
    • $feed.FeedStartTime—The time the feed was started
    • $feed.FeedScheduledStartTime—The time the next recurrence of the feed is scheduled
    • $feed.FeedLastScheduledStartTime—The time the last recurrence of the feed was scheduled
    Feed analytic variables and the time they represent
  • The ArcGIS Velocity subscription license levels have limits to how much data can be consumed by a feed as summarized below. If feeds exceed the consumption limit of your license level, a message is written to the logs and an email is sent notifying you that the limit has been reached. If consumption of data remains above the limit for more than an hour, the feed is stopped. For more information about ArcGIS Velocity licensing, see Licensing. Work with your Esri representative or Esri Technical Support, if necessary, to explore Velocity license levels that support your workflows. License levels are as follows:
    • Standard—Consume and process data at 25 KB/second (approximately 100 events per second).
    • Advanced—Consume and process data at 250 KB/second (approximately 1,000 events per second).
    • Dedicated—Consume and process data at 2,500 KB/second (approximately 10,000 events per second).
  • After configuring the feed connection parameters, see Configure input data to learn how to define the schema and the key parameters.

OAuth authentication

The HTTP Poller feed type supports authentication using OAuth 2.0 for HTTP endpoints that follow the Client Credentials or Password grant types. These OAuth grant types follow a pattern that includes the configuration of an HTTP request to fetch an access token and, optionally, configure a request to use a refresh token to retrieve new access tokens. These are in addition to the data request that is typically configured as part of any HTTP Poller feed type.

During configuration of the access token request, you are prompted to identify the access token and optional refresh token and expiration JSON key fields from the response from the OAuth authorization server. If an expiration is selected, you must define the expiration type (duration or time stamp) and units.

If a refresh token was identified from the access token request, you must use the $refreshToken global variable to define where the refresh token will be used in the configuration of the refresh token request.

Finally, when returning to the configuration of the data request, you must use the $accessToken global variable to define where the access token will be used. This is often in an Authorization custom header with the Bearer $accessToken value. $accessToken can also be used in the POST body and URL parameters, as well as in the URL (for example, https://<<company.domain>>/$accessToken/<<method>>?<<url params>>).

During processing, a feed with OAuth authentication configured manages the access token and handles any errors in retrieving data accordingly. For example, if a refresh token request and expiration are defined, the feed uses the refresh token it initially retrieved with the access token request to retrieve a new token if the token has expired.

Consider the following when authoring a feed with OAuth:

  • Multiple requests to an authorization server and the associated data endpoint occur during authoring. Some APIs implement rate limiting for certain requests and this may affect authoring the feed.
  • For security reasons, access token and refresh token request POST bodies must be reconfigured when editing the authentication parameters.

Parameters

The following are the parameters for the HTTP Poller feed:

ParameterDescriptionData type

URL

The full URL to the externally accessible HTTP endpoint.

This parameter supports the $accessToken global variable.

String

HTTP method

Specifies the HTTP method used for the HTTP endpoint, either GET or POST.

If POST is chosen, you can specify a message body.

String

POST body

The message body to be included when sending a POST HTTP request.

This provides a selection between x-www-form-urlencoded and other generic text content types. It also sets the Content-type header.

This parameter only applies when the HTTP method parameter is set to POST.

This parameter supports feed and $accessToken global variables.

String

Authentication

Specifies the type of authentication used to access the HTTP endpoint. The options are None, Basic, Certificate, and OAuth.

String

Username

The username for accessing the HTTP endpoint.

This parameter only applies when the Authentication parameter is set to Basic.

String

Password

The password for accessing the HTTP endpoint.

This parameter only applies when the Authentication parameter is set to Basic.

Password

Certificate URL

The URL location of the client certificate for accessing the HTTP endpoint.

This parameter only applies when the Authentication parameter is set to Certificate.

String

Password

The client certificate password if one is set.

This parameter only applies when the Authentication parameter is set to Certificate.

Password

Custom headers

Optional custom headers to be included in the HTTP request sent to the specified HTTP endpoint.

For example, custom headers can be used to specify a third-party API key or authorization header.

This parameter supports feed and $accessToken global variables.

Key-Value Pair

URL parameters

Optional URL parameters to be included in the URL of the HTTP request sent to the specified HTTP endpoint.

While parameters can also be set in the URL parameter, the URL parameters entered here will override any matching parameter provided for the URL parameter.

Parameters defined here will always be URL encoded.

This parameter supports feed and $accessToken global variables.

Key-Value Pair

Long polling

Specifies whether the web server will be polled and an open connection maintained until the server sends new data. When new data is received, a new request is immediately sent.

Note:

This is a unique web server configuration that is not common to most web servers.

Boolean

Global variables

Convert the feed global variables to a custom date time format. This parameter is optional

This parameter supports feed and $accessToken global variables.

For more information about setting a date format (epoch milliseconds, epoch seconds, and string format), see Date and time parameters.

String

Additional logging

Specifies whether logging for raw HTTP requests and responses issued by Velocity will be enabled. This parameter is optional.

Note:

Only enable this parameter for troubleshooting purposes and disabled it when troubleshooting is complete. Once enabled, start the feed, and the debug level logs will be available on the feed logs page. Contact Esri Technical Support for assistance with troubleshooting if necessary.

Boolean

Considerations and limitations

The following are considerations and limitations when using the HTTP Poller feed:

  • The HTTP endpoint must be externally accessible on the internet.
  • The size of the HTTP response or file being polled should not exceed 16 MB. If the data being polled exceeds 16 MB, features may not be consumed.
  • As a best practice, use this feed type for real-time data with velocities of approximately 3,000 point features per second or less. Supported velocities will vary depending on the number of data attributes, incoming data format, and polyline or polygon feature complexity.

Feedback on this topic?

In this topic
  1. Example
  2. Usage notes
  3. Parameters
  4. Considerations and limitations