ArcGIS Data Pipelines provides data integration with ArcGIS. With Data Pipelines, you can connect to and read data from where it is stored, perform data preparation operations, and write the data out to a feature layer that is available in ArcGIS. You can use the Data Pipelines interface to construct, run, and reproduce data preparation workflows. To automate your workflows, you can schedule data pipelines to run on regular intervals.
Data Pipelines works with vector data (for example, points, lines, and polygons) and tabular data (for example, data represented as a table). You can connect to a variety of data sources including Amazon S3, Google BigQuery, Snowflake, feature layers, and others. Once connected, you can use tools to blend, build, and integrate datasets for use in your workflows.
Data Pipelines tools are structured as toolsets with capabilities such as clean, construct, integrate, and format. For example, the following workflows are supported by Data Pipelines tools:
- Manipulate dataset schemas by updating field names or types.
- Select a subset of fields to extract targeted information.
- Filter by attribute or geometry values to clean the data.
- Combine datasets using join or merge functionality.
- Calculate fields using ArcGIS Arcade functions.
- Create geometry or time fields for use in spatial or temporal analysis.
When building a data pipeline and configuring tools, you can preview results. You can inspect and perfect the data in preparation for writing the final result. Once you’ve completed the data pipeline, you can run it to create or update an ArcGIS feature layer that will be available in your content. You can configure geometry and time properties for the output feature layer so it’s ready for use in additional workflows such as spatial or temporal analysis, dashboards, or web maps.