HTTP Poller

The HTTP Poller feed type in ArcGIS Velocity will send an HTTP request to the specified URL. This will collect any data that could be retrieved in a web browser by simply browsing to the same URL.

Examples

  • Updated sensor values are loaded from an API every minute.

Usage notes

  • The feed will poll the HTTP endpoint to load data at the rate defined on the Feed Polling Interval step of the configuration wizard.
  • The HTTP endpoint must be externally accessible on the Internet.
  • The size of the HTTP response or file being polled should not exceed 8 MB. If the data being polled exceeds 8 MB, features may not be ingested.
  • When a polling interval is set, the polling occurs on regular clock intervals as opposed to the start time of the feed. For example, if a feed is set to poll for data every 5 minutes, and the feed is started at 8:03 AM, the first request for data will occur at 8:05 AM, the next at 8:10 AM, and so on.
  • Use caution when defining the Feed Polling Interval. It is essential to ensure the HTTP endpoint being polled is capable of returning responses within the polling interval specified. A recommended best practice is to test the response time of a request to the website using the same request that you apply in the feed configuration. The polling interval should never be set to a shorter interval than the typical response time of the request to the website. For details, see Schedule feed polling interval.
  • Both the URL and Custom headers parameters can make use of feed global variables. The feed global variables translate into epoch time (milliseconds since January 1, 1970 UTC). The following outlines the feed analytic variables and what time they represent.
    • $feed.FeedStartTime—Time the feed was started.
    • $feed.FeedScheduledStartTime—Time the next recurrence of the feed is scheduled.
    • $feed.FeedLastScheduledStartTime—Time the last recurrence of the feed was scheduled.

After configuring feed connection properties, see Configure input data to learn how to define the schema and the key properties.

OAuth authentication

The HTTP Poller feed type supports authentication using OAuth 2.0 for HTTP endpoints that follow the Client Credentials or Password grant types. These OAuth grant types follow a pattern that includes the configuration of an HTTP request to fetch an access token and optionally configure a request to use a refresh token to retrieve new access tokens. These are in addition to the data request that is normally configured as part of any HTTP Poller feed type.

During configuration of the access token request, you will be prompted to identify the access token and optional refresh token and expiration JSON key fields from the response from the OAuth authorization server. If an expiration is selected, you will be required to define the expiration type (duration or timestamp) and units.

If a refresh token was identified from the access token request, you will need to use the global variable $refreshToken to define where the refresh token will be used in the configuration of the refresh token request.

Finally, when returning to the configuration of the data request, you will need to use the global variable $accessToken to define where the access token will be used. This is often in an Authorization custom header with the value Bearer $accessToken.

While running, a feed with OAuth authentication configured will manage the access token and handle any errors in retrieving data accordingly. For example, if a refresh token request and expiration are defined, the feed will use the refresh token it initially retrieved with the access token request to retrieve a new token if the token has expired.

There are some important considerations and limitations when authoring a feed with OAuth:

  • Multiple requests to an authorization server and the associated data endpoint occur during authoring. Some APIs implement rate limiting for certain requests and this could affect authoring the feed.
  • For security reasons, access token and refresh token request POST bodies must be reconfigured upon editing the authentication settings.

Parameters

ParameterExplanationData Type

URL

The full URL to the externally accessible HTTP endpoint.

String

HTTP method

The HTTP method to utilize for the HTTP endpoint, either GET or POST.

If POST is selected, optionally specify a message body.

String

POST body

A message body to be included when sending a POST HTTP request.

Provides a selection between x-www-form-urlencoded and other generic text content types. Sets the Content-type header.

This parameter only applies when HTTP method is set to POST.

String

Authentication

The type of authentication to access the HTTP endpoint. Authentication options include None, Basic, Certificate, and OAuth.

String

Username

The username for accessing the HTTP endpoint.

This parameter only applies when Authentication is Basic.

String

Password

The password for accessing the HTTP endpoint.

This parameter only applies when Authentication is Basic.

Password

Certificate URL

The URL location of the client certificate for accessing the HTTP endpoint.

This parameter only applies when Authentication is Certificate.

String

Password

The client certificate password, if one is set.

This parameter only applies when Authentication is set to Certificate.

Password

Custom headers

Optional custom headers to be included in the HTTP request sent to the specified HTTP endpoint. For example, custom headers can be used to specify a third party API key or authorization header.

Supports feed global variables.

Key-Value Pair

URL parameters

Optional URL parameters to be included in the URL of the HTTP request sent to the specified HTTP endpoint.

While parameters can also be set in the URL parameter above, the URL parameters entered here as a separate property will override any matching parameter supplied in the URL parameter.

Parameters defined in this property will always be URL encoded.

Supports feed global variables.

Key-Value Pair

Long polling

ArcGIS Velocity will poll the web server and maintain an open connection, waiting for the server to send new data. When Velocity receives data it will immediately send a new request.

Note:

This is a unique web server configuration not common to most web servers.

Boolean

Considerations and limitations

  • The HTTP endpoint must be externally accessible on the Internet.
  • The size of the HTTP response or file being polled cannot exceed 8 MB.
  • The polling interval should never be set to a shorter interval than the typical response time of the request to the HTTP endpoint. For details, see Schedule feed polling interval.
  • As a best practice, use this feed type for real-time data with velocities of ~3000 point features per second or less. Supported velocities will vary depending upon the number of data attributes, incoming data format, and polyline/polygon feature complexity.