The Parquet feature layer (Beta) output writes data pipeline datasets to a Parquet feature layer in your content. You can create a new Parquet feature layer, or overwrite an existing one. Use Parquet feature layers to optimize storage, and for visualization workflows in Map Viewer and other supported applications.
To learn more about Parquet feature layers (Beta), see the ArcGIS Blog article Scaling your GIS workflows with the new Parquet feature layer (Beta) in ArcGIS Online.
Note:
This feature is currently in beta. Share your experience, request enhancements, and seek support through the Beta Features Feedback forum in the Data Pipelines Community. If your organization has blocked beta apps and capabilities, you will not have access to this feature.Parameters
The following table describes the parameters used in the Parquet feature layer (Beta) output:
| Parameter | Description |
|---|---|
Input dataset | The dataset that will be written as a file. |
Geometry field | The geometry field that will be represented in the resulting layer. |
Output method | Specifies the method that will be used to write the output results. The options are Create (the default) and Overwrite. |
Title | The title of the output item. |
Overwrite if item already exists | Specifies whether an existing item with the provided output name will be overwritten (checked). Overwrite is enabled by default. |
Folder | The output folder where the file will be saved. This option is only available if the Output method is set to Create. |
Parquet feature layer | An existing item to be overwritten the next time the data pipeline runs. This option is only available if the Output method is set to Overwrite. |
Usage notes
To run a data pipeline, at least one output must be configured.
Use the Input dataset parameter to identify the dataset that will be written as a Parquet feature layer.
The Geometry field parameter is used to determine the geometry of the resulting item. Parquet feature layers only support the WGS 1984 (4326) coordinate system. If the geometry field uses a coordinate system other than WGS 1984 (4326), it will be automatically projected before the data is written.
Use the Output method parameter to specify how the data pipeline results will be written. The options are as follows:
- Create—A new Parquet feature layer will be created in your content. You can optionally use the Overwrite if item already exists parameter to overwrite the item each time the data pipeline is run.
- Overwrite—An existing Parquet feature layer will be completely overwritten.
The following parameters are available when the Create output method is selected:
- The Title parameter specifies the name of the item that will be created or overwritten.
- The Overwrite if item already exists parameter allows you to rerun the data pipeline without changing the output name. This operation will completely overwrite a file with the specified title that is stored in the specified folder. This parameter is enabled by default.
- The Folder parameter determines which folder the newly created result will be stored in. If the Output method is set to Overwrite this parameter indicates which folder the item to overwrite is currently stored in and cannot be changed.
When the Overwrite output method is selected, the Parquet feature layer parameter is available. Use this parameter to browse to and select the item to overwrite. You can overwrite items you own, or items that are shared with you in a shared update group.
For scheduling automated updates to Parquet feature layers, it is recommended to use the Overwrite output method instead of the Overwrite if item already exists option.
To modify the item properties, such as the summary or tags, browse to your portal content page and edit the item directly. Learn more about item details in ArcGIS Online.
Limitations
The following are known limitations of the Parquet feature layer (Beta) output:
- This feature is currently in beta. Share your experience, request enhancements, and seek support through the Beta Features Feedback forum in the Data Pipelines Community. If your organization has blocked beta apps and capabilities, you will not have access to this feature.
- Parquet feature layers are optimized for storage, and for visualization workflows in Map Viewer. They are not currently supported for use as input to analysis tools, direct editing workflows, or for use in ArcGIS Pro.
- Parquet feature layers must have a geometry. Tables are not supported.
- The geometry for Parquet feature layers will always be stored in the WGS 1984 (4326) coordinate system.
- Some field types that are supported for input data sources are not supported when writing results to Parquet feature layers. Field types will be automatically converted for compatibility with the Parquet feature layer format.
- You cannot use the overwrite option for files you do not own unless you are an administrator, or the file is shared with a shared update group that you are a member of.
- When using a combination of the Create and Overwrite if item already exists output options, the following limitations apply:
- The item to overwrite must be owned by you and must be stored in the specified folder. If the item has the same name in another folder, it will not be overwritten.
- This option is not recommended for scheduled or automated runs. For more reliable scheduled updates, use the Overwrite output method and explicitly browse to the Parquet feature layer to overwrite.
Licensing requirements
The following licensing and configurations are required:
- Creator or Professional user type
- Publisher, Facilitator, or Administrator role, or an equivalent custom role
To learn more about Data Pipelines requirements, see Requirements.
Related topics
See the following for additional information:
- To learn more about Parquet feature layers and how they compare to other layer types, see Layers.
- To learn how to update items on an automated schedule, see Schedule a data pipeline task.
- To learn more about items, see Items supported in ArcGIS Online.