Compute resources provide the processing power for the Data Pipelines editor and scheduled tasks. This includes running the data pipeline, inferring dataset schemas, generating data previews, caching inputs, and generating error and warning messages.
Status
The Data Pipelines editor connects to a dedicated compute resource to power your processing. The status of the compute resource is displayed at the top of the application on the connection details dialog box. Compute resources have the following status types:
- Connecting—The compute resource for the data pipeline is being provisioned and the connection is initiating.
- Connected—The compute resource has started and is currently active. To stop the compute resource, click the Disconnect all button on the connection details dialog box. This will disconnect all editors.
- Disconnected—The compute resource has stopped and is currently inactive. When the editor is disconnected, the following options are available in the disconnected modal:
- Leave—Leave the editor without saving changes made to the data pipeline. You will be returned to the Data Pipelines gallery page.
- Save and leave—Save any changes made to the data pipeline and leave. You will be returned to the Data Pipelines gallery page.
- Reconnect—Reconnect and continue working in the data pipeline editor.
- Reconnecting—The compute resource has stopped and the application is attempting to reconnect.
For scheduled data pipeline tasks, a compute resource is active when a task is running, and inactive when the task completes.
Credit consumption
Credits are consumed when a compute resource is active. Compute resources are active and credits are consumed in the following scenarios:
- Interactive—In the editor, credits are consumed while the connection status is Connected. Credits are not consumed when the status is Disconnected. If the status is Reconnecting, the compute resource is active, but credits will not be consumed for the reconnection period if the reconnection failed. If you have multiple browser tabs with the editor open, you are not charged for each one. You are only charged for the amount of time at least one is connected. Data pipelines can be connected even if there are no input datasets, tools, or outputs configured.
- Scheduled—For scheduled data pipeline tasks, credits are consumed while the data pipeline is running. Credits are charged per run for the time it takes to complete.
- After using the disconnect all button on the connection details dialog box. This disconnects all connected editors and credits will not be consumed until at least one editor is reconnected.
- After all browser tabs with connected editors have been closed for at least 10 minutes. Credits are not consumed for those 10 minutes.
- After 30 minutes of inactivity in all editor browser tabs. The status will be Disconnected.
- When a scheduled data pipeline task run is complete.
Considerations
For interactive editing, each user has at most one compute resource, which powers all browser tabs with an open data pipeline editor.
For scheduled tasks, each task run uses a dedicated compute resource.
When the editor shows a disconnected status, the following data is lost:
- Cached sources and datasets
- Long-running data pipeline jobs and any associated messages (warnings, errors, and results)
When you return to the data pipeline editor after closing the tab, you will not have access to any warnings, errors, and results from the previous run.