HTTP Poller

The HTTP Poller feed type in ArcGIS Velocity sends an HTTP request to the specified URL. This collects data that can be retrieved in a web browser by browsing to the same URL.

Example

Updated sensor values are loaded from an API every minute.

Usage notes

  • The feed will poll the HTTP endpoint to load data at the rate defined on the Feed Polling Interval step of the configuration wizard.
  • The HTTP endpoint must be externally accessible on the internet.
  • The size of the HTTP response or file being polled should not exceed 16 MB. If the data being polled exceeds 16 MB, features may not be ingested.
  • When a polling interval is set, the polling occurs at regular time intervals as opposed to the start time of the feed. For example, if a feed is set to poll for data every five minutes, and the feed is started at 8:03 a.m., the first request for data will occur at 8:05 a.m., the next at 8:10 a.m., and so on.
  • Use caution when specifying the Feed Polling Interval value. Ensure that the HTTP endpoint being polled is capable of returning responses within the polling interval specified. It is recommended that you test the response time of a request to the website using the same request that you apply in the feed configuration. Do not set the polling interval to a shorter interval than the typical response time of the request to the website. For details, see Schedule feed polling interval.
  • Both the URL and Custom headers parameters can use feed global variables. The feed global variables are converted into epoch time (milliseconds since January 1, 1970 UTC). The following outlines the feed analytic variables and the time they represent.
    • $feed.FeedStartTime—The time the feed was started
    • $feed.FeedScheduledStartTime—The time the next recurrence of the feed is scheduled
    • $feed.FeedLastScheduledStartTime—The time the last recurrence of the feed was scheduled

After configuring feed connection parameters, see Configure input data to learn how to define the schema and the key parameters.

OAuth authentication

The HTTP Poller feed type supports authentication using OAuth 2.0 for HTTP endpoints that follow the Client Credentials or Password grant types. These OAuth grant types follow a pattern that includes the configuration of an HTTP request to fetch an access token and, optionally, configure a request to use a refresh token to retrieve new access tokens. These are in addition to the data request that is typically configured as part of any HTTP Poller feed type.

During configuration of the access token request, you are prompted to identify the access token and optional refresh token and expiration JSON key fields from the response from the OAuth authorization server. If an expiration is selected, you must define the expiration type (duration or time stamp) and units.

If a refresh token was identified from the access token request, you must use the $refreshToken global variable to define where the refresh token will be used in the configuration of the refresh token request.

Finally, when returning to the configuration of the data request, you must use the $accessToken global variable to define where the access token will be used. This is often in an Authorization custom header with the Bearer $accessToken value.

While running, a feed with OAuth authentication configured manages the access token and handles any errors in retrieving data accordingly. For example, if a refresh token request and expiration are defined, the feed uses the refresh token it initially retrieved with the access token request to retrieve a new token if the token has expired.

Consider the following when authoring a feed with OAuth:

  • Multiple requests to an authorization server and the associated data endpoint occur during authoring. Some APIs implement rate limiting for certain requests and this may affect authoring the feed.
  • For security reasons, access token and refresh token request POST bodies must be reconfigured when editing the authentication parameters.

Parameters

ParameterDescriptionData type

URL

The full URL to the externally accessible HTTP endpoint.

String

HTTP method

The HTTP method used for the HTTP endpoint, either GET or POST.

If POST is chosen, you can specify a message body.

String

POST body

The message body to be included when sending a POST HTTP request.

This provides a selection between x-www-form-urlencoded and other generic text content types. It also sets the Content-type header.

This parameter only applies when HTTP method is set to POST.

String

Authentication

The type of authentication used to access the HTTP endpoint. Authentication options are None, Basic, Certificate, and OAuth.

String

Username

The user name for accessing the HTTP endpoint.

This parameter only applies when Authentication is set to Basic.

String

Password

The password for accessing the HTTP endpoint.

This parameter only applies when Authentication is set to Basic.

Password

Certificate URL

The URL location of the client certificate for accessing the HTTP endpoint.

This parameter only applies when Authentication is set to Certificate.

String

Password

The client certificate password if one is set.

This parameter only applies when Authentication is set to Certificate.

Password

Custom headers

Optional custom headers to be included in the HTTP request sent to the specified HTTP endpoint.

For example, custom headers can be used to specify a third-party API key or authorization header.

This parameter supports feed global variables.

Key-Value Pair

URL parameters

Optional URL parameters to be included in the URL of the HTTP request sent to the specified HTTP endpoint.

While parameters can also be set in the URL parameter, the URL parameters entered here as a separate property will override any matching parameter supplied for the URL parameter.

Parameters defined here will always be URL encoded.

This parameter supports feed global variables.

Key-Value Pair

Long polling

ArcGIS Velocity will poll the web server and maintain an open connection until the server sends new data. When Velocity receives data, a new request is immediately sent.

Note:

This is a unique web server configuration that is not common to most web servers.

Boolean

Considerations and limitations

  • The HTTP endpoint must be externally accessible on the internet.
  • The size of the HTTP response or file being polled cannot exceed 16 MB.
  • Do not set the polling interval to a shorter interval than the typical response time of the request to the HTTP endpoint. For details, see Schedule feed polling interval.
  • As a best practice, use this feed type for real-time data with velocities of approximately 3,000 point features per second or less. Supported velocities will vary depending on the number of data attributes, incoming data format, and polyline or polygon feature complexity.