Answers to frequently asked questions about ArcGIS Data Pipelines are provided.
- What is ArcGIS Data Pipelines?
- Does Data Pipelines charge credits?
- Is Data Pipelines available in ArcGIS Enterprise?
- How do I access Data Pipelines?
- How can I get started with Data Pipelines?
- What data can I use in Data Pipelines?
- Can I use ArcGIS Living Atlas layers as input to my data pipeline?
- Can I connect to my datasets on the Google Cloud platform?
- My data was updated in its source location. How do I sync my dataset in my data pipeline?
- Where can I store my Data Pipelines results? Can I store them in Amazon S3?
- How many features can I write to a feature layer or table using Data Pipelines?
- Can I geocode addresses using Data Pipelines?
- What tools are coming in future releases?
- Can I share a data pipeline?
- Is there a way to undo or redo an action in the Data Pipelines editor?
- Is there a way to copy and paste elements in a diagram?
- Can I schedule a data pipeline run?
- How is Data Pipelines different from ArcGIS Velocity?
- How is Data Pipelines different from ArcGIS Data Interoperability
Data Pipelines is an ArcGIS Online application that allows you to connect to, process, and integrate data from various sources. You can perform data preparation and save the results to your Web GIS to complete your organization's workflows. All of this is completed using an intuitive interface where you can construct, run, save, share, and reproduce your data preparation workflows.
Yes. Credit consumption is based on compute resource usage time. See Compute resources for more information.
Credits are consumed when a compute resource is active. Compute resources are active in the following scenarios:
- Interactive—In the editor, credits are consumed while the connection status is Connected. Credits are not consumed when the status is Disconnected. If the status is Reconnecting, the compute resource is active, but credits will not be consumed for the reconnection period if the reconnection failed. If you have multiple browser tabs with the editor open, you are not charged for each one. You are only charged for the amount of time at least one is connected.
- Scheduled—For scheduled data pipeline tasks, credits are consumed while the data pipeline is running. Credits are charged per run for the time it takes to complete.
- After all browser tabs with connected editors have been closed for at least 10 minutes. Credits are not consumed for those 10 minutes.
- After 30 minutes of inactivity in all editor browser tabs. The status will be Disconnected.
- When a scheduled data pipeline task run is complete.
You can access Data Pipelines by using the app launcher and choosing Data Pipelines.
To access Data Pipelines, your user account must have the required privileges. See Requirements to learn more about the privileges and requirements to access Data Pipelines.
If you are unsure whether you or your organization meets the requirements above, contact your organization administrator.
To get started with Data Pipelines, see Tutorial: Create a data pipeline. The tutorial outlines the key components for using Data Pipelines, including connecting to and processing data, running a data pipeline, and more.
For more resources to get started, see the Data Pipelines Community blog posts.
The following types of data are supported as input:
- Amazon S3
- Feature layers
- Files from public URLs
- Files uploaded to content
- Google BigQuery
- Microsoft Azure Storage
- Snowflake
See the linked input data type documentation to learn more about supported file types and how to connect to an input dataset.
Yes. You can use ArcGIS Living Atlas feature layers as input. To add a layer to a diagram, see Feature layer. By default, the feature layer browse dialog box opens to My content. To search for an ArcGIS Living Atlas layer, switch to Living Atlas on the dialog box.
No, not yet. In future releases, the following additional types of external data sources will be supported:
- Google Cloud platform
- Microsoft Azure Cosmos DB for PostgreSQL
- Data returned from API requests
The data sources in this list are not guaranteed for a specific release, and data sources that are not listed here may be added. If you have suggestions for data sources that will improve your workflows, leave a comment in the Data Pipelines Community forums.
If the data is regularly updating in the source location and you want to use it in a data pipeline, it is recommended that you do not use the Use caching parameter for inputs. If you do not use caching, Data Pipelines reads the latest data every time you request a preview or run. If you use caching, only the data available at the time you cached is used.
If you created an output feature layer and need to update it with the latest data, use the Replace or Add and update options in the Feature layer tool, and run the data pipeline again. You can automate rerunning a data pipeline by scheduling a task for the data pipeline item. To learn more about automating data pipeline workflows, see Schedule a data pipeline task.
The following tools may be included in future releases:
- Find and replace—Search fields for specific values and replace them with a new value.
- Geocode addresses—Use string addresses from a table or file to return the geocoded results.
The tools in this list are not guaranteed for any release, and tools that are not listed here may be added. If you have suggestions for tools that will improve your workflows, leave a comment in the Data Pipelines Community forums.
Yes. You can share data pipeline items with groups in your organization or with the public. Only the owner of the item can edit data pipeline items. Use shared update groups so everyone in the group can edit and save the data pipeline. If a data pipeline is shared with a group that does not have shared update capabilities, you can save the data pipeline as an editable copy in your content using the Save As option on the editor toolbar.
Yes. You can create tasks for data pipeline items to run your workflows on a schedule. To learn more about creating data pipeline tasks, see Schedule a data pipeline.
There are certain similarities between Data Pipelines and Velocity in ArcGIS Online. Both applications allow you to connect to external data sources and import the data into ArcGIS Online for use across the ArcGIS system. However, they serve distinct purposes. Velocity is specifically designed for real-time and big data processing, efficiently handling high-speed data streams from sensors and similar sources. It also is focused on enabling analytics such as device tracking, incident detection, and pattern analysis. Data Pipelines is primarily a data integration application that focuses on data engineering tasks, particularly for non-sensor-based data streams. While Velocity is used for handling real-time data, Data Pipelines is used for managing and optimizing data that requires updates on a less frequent basis.
First, let's start with how they are similar. Both are no-code ETL tools for ArcGIS, supporting data integration, transformation, and cleaning. However, they are very different in that Data Pipelines is a web-based application available immediately in ArcGIS Online, while Data Interoperability is an extension for ArcGIS Pro, requiring a separate license and installation. Data Pipelines is focused on data integration for ArcGIS Online, with results being written out to a hosted feature layer, while Data Interoperability supports a larger set of supported inputs and file types, and is able to write results back to the source.