Data Pipelines provides integration of your data with ArcGIS. With Data Pipelines you can connect to and read data from where it is stored, perform data preparation operations, and write the data out to a feature layer that is readily available in ArcGIS. You can use the Data Pipelines interface to construct, run, and reproduce your data preparation workflows.
Data Pipelines works with vector data (for example, points, lines, and polygons) and tabular data (for example, data represented as a table). You can connect to a variety of data sources including Amazon S3, Google BigQuery, Snowflake, feature layers, and more. Once connected, you can use tools to blend, build, and integrate datasets for use in your workflows.
Data Pipelines tools are structured as toolsets with capabilities such as clean, construct, integrate, and format. For example, the following workflows are supported by Data Pipelines tools:
- Manipulate dataset schemas by updating field names or types.
- Select a subset of fields to extract targeted information.
- Find and replace attribute values to clean or simplify the data.
- Combine datasets using join or merge functionality.
- Calculate fields using Arcade functions.
- Create geometry or time fields for use in spatial or temporal analysis.
While building a data pipeline and configuring tools, you can preview results. You can inspect and perfect the data in preparation for writing the final result. Once you’ve completed the data pipeline, you can run it to create or update an ArcGIS feature layer that will be available in your content. You can configure geometry and time properties for the output feature layer so it’s ready for use in additional workflows such as spatial or temporal analysis, dashboards or web maps.