The HTTP Poller feed type in ArcGIS Velocity sends an HTTP request to the specified URL. This collects data that can be retrieved in a web browser by browsing to the same URL.
Example
Updated sensor values are loaded from an API every minute.
Usage notes
- The feed will poll the HTTP endpoint to load data at the rate defined on the Feed Polling Interval step of the configuration wizard.
- The HTTP endpoint must be externally accessible on the internet.
- The size of the HTTP response or file being polled should not exceed 16 MB. If the data being polled exceeds 16 MB, features may not be ingested.
- When a polling interval is set, the polling occurs at regular time intervals as opposed to the start time of the feed. For example, if a feed is set to poll for data every five minutes, and the feed is started at 8:03 a.m., the first request for data will occur at 8:05 a.m., the next at 8:10 a.m., and so on.
- Use caution when specifying the Feed Polling Interval value. Ensure that the HTTP endpoint being polled is capable of returning responses within the polling interval specified. It is recommended that you test the response time of a request to the website using the same request that you apply in the feed configuration. Do not set the polling interval to a shorter interval than the typical response time of the request to the website. For details, see Schedule feed polling interval.
- The POST body, URL parameters, and Custom headers parameters can use feed global variables. By default, the feed global variables are converted into epoch time (milliseconds since January 1, 1970 UTC). Optionally, you can convert the feed global variables to epoch seconds (seconds since January 1, 1970) and a desired string date time format. The following outlines the feed analytic variables and the time they represent.
- $feed.FeedStartTime—The time the feed was started
- $feed.FeedScheduledStartTime—The time the next recurrence of the feed is scheduled
- $feed.FeedLastScheduledStartTime—The time the last recurrence of the feed was scheduled
- The ArcGIS Velocity subscription license levels have limits to how much data can be ingested by a feed as summarized below. If feeds exceed the ingestion limit of your license level, a message is written to the logs and an email is sent to the user notifying them the limit has been reached. If ingestion of data remains above the limit for more than an hour, the feed will be stopped. For more information on ArcGIS Velocity licensing, see Licensing. If necessary, work with your Esri representative or Esri Technical Support to explore Velocity license levels that support your workflows.
- Standard—Ingest and process data at 25 KB/second (roughly 100 events per second).
- Advanced—Ingest and process data at 250 KB/second (roughly 1,000 events per second).
- Dedicated—Ingest and process data at 2,500 KB/second (roughly 10,000 events per second).
After configuring feed connection parameters, see Configure input data to learn how to define the schema and the key parameters.
OAuth authentication
The HTTP Poller feed type supports authentication using OAuth 2.0 for HTTP endpoints that follow the Client Credentials or Password grant types. These OAuth grant types follow a pattern that includes the configuration of an HTTP request to fetch an access token and, optionally, configure a request to use a refresh token to retrieve new access tokens. These are in addition to the data request that is typically configured as part of any HTTP Poller feed type.
During configuration of the access token request, you are prompted to identify the access token and optional refresh token and expiration JSON key fields from the response from the OAuth authorization server. If an expiration is selected, you must define the expiration type (duration or time stamp) and units.
If a refresh token was identified from the access token request, you must use the $refreshToken global variable to define where the refresh token will be used in the configuration of the refresh token request.
Finally, when returning to the configuration of the data request, you must use the $accessToken global variable to define where the access token will be used. This is often in an Authorization custom header with the Bearer $accessToken value. $accessToken can also be used in the POST body, URL parameters, and in the URL itself (for example, https://<<company.domain>>/$accessToken/<<method>>?<<url params>>).
While running, a feed with OAuth authentication configured manages the access token and handles any errors in retrieving data accordingly. For example, if a refresh token request and expiration are defined, the feed uses the refresh token it initially retrieved with the access token request to retrieve a new token if the token has expired.
Consider the following when authoring a feed with OAuth:
- Multiple requests to an authorization server and the associated data endpoint occur during authoring. Some APIs implement rate limiting for certain requests and this may affect authoring the feed.
- For security reasons, access token and refresh token request POST bodies must be reconfigured when editing the authentication parameters.
Parameters
| Parameter | Description | Data type |
|---|---|---|
URL | The full URL to the externally accessible HTTP endpoint. This parameter supports the $accessToken global variable. | String |
HTTP method | The HTTP method used for the HTTP endpoint, either GET or POST. If POST is chosen, you can specify a message body. | String |
POST body | The message body to be included when sending a POST HTTP request. This provides a selection between x-www-form-urlencoded and other generic text content types. It also sets the Content-type header. This parameter only applies when the HTTP method parameter is set to POST. This parameter supports feed and $accessToken global variables. | String |
Authentication | The type of authentication used to access the HTTP endpoint. Authentication options are None, Basic, Certificate, and OAuth. | String |
Username | The username for accessing the HTTP endpoint. This parameter only applies when the Authentication parameter is set to Basic. | String |
Password | The password for accessing the HTTP endpoint. This parameter only applies when the Authentication parameter is set to Basic. | Password |
Certificate URL | The URL location of the client certificate for accessing the HTTP endpoint. This parameter only applies when the Authentication parameter is set to Certificate. | String |
Password | The client certificate password if one is set. This parameter only applies when the Authentication parameter is set to Certificate. | Password |
Custom headers | Optional custom headers to be included in the HTTP request sent to the specified HTTP endpoint. For example, custom headers can be used to specify a third-party API key or authorization header. This parameter supports feed and $accessToken global variables. | Key-Value Pair |
URL parameters | Optional URL parameters to be included in the URL of the HTTP request sent to the specified HTTP endpoint. While parameters can also be set in the URL parameter, the URL parameters entered here will override any matching parameter supplied for the URL parameter. Parameters defined here will always be URL encoded. This parameter supports feed and $accessToken global variables. | Key-Value Pair |
Long polling | ArcGIS Velocity will poll the web server and maintain an open connection until the server sends new data. When Velocity receives data, a new request is immediately sent. Note:This is a unique web server configuration that is not common to most web servers. | Boolean |
Global variables | Optional parameter that can be used to convert the feed global variables to a custom date time format. This parameter supports feed and $accessToken global variables. For more information on setting a date format (epoch milliseconds, epoch seconds, and string format), see Date and time parameters. | String |
Additional logging | Optionally enable logging for raw HTTP requests and responses issued by Velocity. Note:This parameter should only be enabled for troubleshooting purposes and disabled when troubleshooting is complete. Once enabled, start the feed and the debug level logs will be available on the feed logs page. If necessary, contact Esri Technical Support for assistance with troubleshooting. | Boolean |
Considerations and limitations
- The HTTP endpoint must be externally accessible on the internet.
- The size of the HTTP response or file being polled cannot exceed 16 MB.
- Do not set the polling interval to a shorter interval than the typical response time of the request to the HTTP endpoint. For details, see Schedule feed polling interval.
- As a best practice, use this feed type for real-time data with velocities of approximately 3,000 point features per second or less. Supported velocities will vary depending on the number of data attributes, incoming data format, and polyline or polygon feature complexity.