The HTTP Poller data source in ArcGIS Velocity sends an HTTP request to the specified URL. This ingests data that can be retrieved in a web browser by browsing to the same URL.
Example
A GIS analyst wants to load district polygons stored in a .csv file on a web server into Velocity.
Usage notes
Keep the following in mind when working with the HTTP Poller data source:
- After configuring source connection parameters, configure the input data to define the schema and the key parameters.
- Both the URL parameters and Custom headers parameters support analytic global variables for scheduling recurring big data analysis. By default, these global variables are converted to epoch time (milliseconds since January 1, 1970 UTC). The following are the available analytic variables and the time they represent.
- $analytic.AnalyticLastScheduledStartTime—The time the last run of the analytic was scheduled to start.
- $analytic.AnalyticScheduledStartTime—The time the current run of the analytic was scheduled to start.
- $analytic.AnalyticLastEndTime—The time the last run of the analytic ended.
- $analytic.AnalyticStartTime—The time the current run of the analytic started. The difference between this and $analytic.AnalyticScheduledStartTime is the startup delay.
- If using OAuth authentication the following apply:
- The HTTP Poller source supports authentication using OAuth 2.0 for HTTP endpoints that follow the Client Credentials or Password grant types. These OAuth grant types follow a pattern that includes the configuration of an HTTP request to fetch an access token. These are in addition to the data request that is typically configured as part of any HTTP Poller source type.
- During configuration of the access token request, you are prompted to identify the access token from the response from the OAuth authorization server.
- When returning to the configuration of the data request, you must use the $accessToken global variable to define where the access token will be used. This is often in an Authorization custom header with the Bearer $accessToken value. $accessToken can also be used in the POST body and URL parameters, as well as in the URL (for example, https://<<company.domain>>/$accessToken/<<method>>?<<url params>>).
Parameters
The following are the parameters for the HTTP Poller source:
| Parameter | Description | Data type |
|---|---|---|
URL | The full URL to the externally accessible HTTP endpoint. This parameter supports the $accessToken global variable. | String |
HTTP method | Specifies the HTTP method used for the HTTP endpoint, either GET or POST. If POST is chosen, you can specify a message body. | String |
POST body | The message body to be included when sending a POST HTTP request. This provides a selection between x-www-form-urlencoded and other generic text content types. It also sets the Content-type header. This parameter only applies when the HTTP method parameter is set to POST. This parameter supports analytic and $accessToken global variables. | String |
Authentication | Specifies the type of authentication used to access the HTTP endpoint. The options are None, Basic, Certificate, and OAuth. | String |
Username | The username for accessing the HTTP endpoint. This parameter only applies when the Authentication parameter is set to Basic. | String |
Password | The password for accessing the HTTP endpoint. This parameter only applies when the Authentication parameter is set to Basic. | Password |
Certificate URL | The URL location of the client certificate for accessing the HTTP endpoint. This parameter only applies when the Authentication parameter is set to Certificate. | String |
Password | The client certificate password if one is set. This parameter only applies when the Authentication parameter is set to Certificate. | Password |
Custom headers | Optional custom headers to be included in the HTTP request that is sent to the specified HTTP endpoint. For example, custom headers can be used to specify a third-party API key or authorization header. This parameter supports analytic and $accessToken global variables. | Key-Value Pair |
URL parameters | Optional URL parameters to be included in the URL of the HTTP request that is sent to the specified HTTP endpoint. While parameters can also be set in the URL parameter, the URL parameters entered here will override any matching parameter provided for the URL parameter. Parameters defined here will always be URL encoded. This parameter supports analytic and $accessToken global variables. | Key-Value Pair |
Considerations and limitations
The following are considerations and limitations when using the HTTP Poller source:
- The HTTP endpoint must be externally accessible on the internet.
- The size of the HTTP response or file being polled cannot exceed 16 MB. If the data or file being polled exceeds 16 MB, features will not be consumed.
- When
using OAuth authentication, the following apply:
- Multiple requests to an authorization server and the associated data endpoint occur during authoring. Some APIs implement rate limiting for certain requests and this may affect authoring the feed.
- For security reasons, the access token POST body must be reconfigured when editing the authentication parameters.