Compute resources provide the processing power for interactively authoring data pipelines and running data pipeline jobs from scheduled tasks or ArcGIS API for Python. Compute resources support capabilities such as running the data pipeline, inferring dataset schemas, generating data previews, caching inputs, and generating error and warning messages.
Status
The Data Pipelines editor connects to a dedicated compute resource to power your processing. The status of the compute resource is displayed at the top of the application on the connection details dialog box. Compute resources have the following status types:
- Connecting—The compute resource for the data pipeline is being provisioned and the connection is initiating.
- Connected—The compute resource has started and is currently active. To stop the compute resource, click the Disconnect all button on the connection details dialog box. This will disconnect all editors.
- Disconnected—The compute resource has stopped and is currently inactive. When the editor is disconnected, the following options are available in the disconnected modal:
- Leave—Leave the editor without saving changes made to the data pipeline. You will be returned to the Data Pipelines gallery page.
- Save and leave—Save any changes made to the data pipeline and leave. You will be returned to the Data Pipelines gallery page.
- Reconnect—Reconnect and continue working in the data pipeline editor.
- Reconnecting—The compute resource has stopped and the application is attempting to reconnect.
For jobs run using scheduled data pipeline tasks, ArcGIS API for Python, or using the run option from the Data Pipelines gallery page, a compute resource is active while the job is running, and inactive when the job completes.
Credit consumption
Credits are consumed when a compute resource is active. Compute resources are active in the following scenarios:
- Interactive editing—While authoring or editing data pipelines in the editor, credits are consumed while the connection status is Connected. The credit rate is 50 credits per hour, calculated per minute, with a 10-minute minimum.
- Jobs—Jobs are run for scheduled data pipeline tasks, when you run a data pipeline using ArcGIS API for Python, or when you use the run option from the Data Pipelines gallery page. Jobs only consume credits while the data pipeline is running. Credits are charged per run for the time it takes to complete at a rate of 70 credits per hour, calculated per minute. There is no minimum charge for jobs.
- After using the disconnect all button on the connection details dialog box. This disconnects all connected editors and credits will not be consumed until at least one editor is reconnected.
- After all browser tabs with connected editors have been closed for at least 10 minutes. Credits are not consumed for those 10 minutes.
- After 30 minutes of inactivity in all editor browser tabs. The status will be Disconnected.
- When a scheduled data pipeline task run is complete.
- When a data pipeline run using ArcGIS API for Python is complete.
Considerations
For interactive editing, consider the following:
- Each user has at most one compute resource that powers all browser tabs with an open data pipeline editor.
- If the status is Reconnecting, the compute resource is active, but credits will not be consumed for the reconnection period if the reconnection failed.
- If you have multiple browser tabs with the editor open, you are not charged for each one. You are only charged for the amount of time at least one is connected.
- The editor can be connected even if there are no input datasets, tools, or outputs configured.
- When the editor shows a disconnected status, the following data is lost:
- Cached sources and datasets
- Long-running data pipeline jobs and any associated messages (warnings, errors, and results)
- When you return to the data pipeline editor after closing the tab, you will not have access to any warnings, errors, and results from the previous run.
For running jobs via scheduled data pipeline tasks or ArcGIS API for Python, each job run uses a dedicated compute resource.