Compute resources provide the processing power for the Data Pipelines editor and scheduled tasks. This includes running the data pipeline, inferring dataset schemas, generating data previews, caching inputs, and generating error and warning messages.
Status
The Data Pipelines editor connects to a dedicated compute resource to power your processing. The status of the compute resource is displayed at the top of the application in the connection details dialog box. Compute resources have the following status types:
- Connecting—The compute resource for the data pipeline is being provisioned and the connection is initiating.
- Connected—The compute resource has started and is currently active.
- Disconnected—The compute resource has stopped and is currently inactive. To restart the compute resource, save the data pipeline and refresh the browser.
- Reconnecting—The compute resource has stopped and the application is attempting to reconnect.
For scheduled data pipeline tasks, a compute resource is active when a task is running, and inactive when the task completes.
Considerations
Each user has at most one compute resource, which powers all browser tabs with an open data pipeline editor.
When the editor shows a disconnected status, the following data is lost:
- Cached sources and datasets
- Long running data pipeline jobs and any associated messages (warnings, errors, and results)
When you return to the data pipeline editor after closing the tab, you will not have access to any warnings, errors, and results from the previous run.
Credit consumption
Credits are consumed when a compute resource is active. Compute resources are active in the following scenarios:
- Interactive—In the editor, credits are consumed while the connection status is Connected. Credits are not consumed when the status is Disconnected. If the status is Reconnecting, the compute resource is active, but credits will not be consumed for the reconnection period if the reconnection failed. If you have multiple browser tabs with the editor open, you are not charged for each one. You are only charged for the amount of time at least one is connected.
- Scheduled—For scheduled data pipeline tasks, credits are consumed while the data pipeline is running. Credits are charged per run for the time it takes to complete.
- After all browser tabs with connected editors have been closed for at least 10 minutes. Credits are not consumed for those 10 minutes.
- After 30 minutes of inactivity in all editor browser tabs. The status will be Disconnected.
- When a scheduled data pipeline task run is complete.