Use Snowflake records

Use records from a Snowflake table as input to ArcGIS Data Pipelines.

Usage notes

Keep the following in mind when working with Snowflake:

• To use a dataset from Snowflake, you must first create a data store item. Data store items securely store credentials and connection information so the data can be read by Data Pipelines. To create a data store, follow the steps in the Connect to Snowflake section below.
• To change the data store item you configured, use the Data store item parameter to remove the currently selected item, and choose one of the following options:
  • Add data store—Create a new data store item.
  • Select item—Browse your content to select an existing data store item.
• Use the Table parameter to specify the dataset you want to use. Table names are case sensitive. For example, if the table is named MyTable, you must specify it using the exact capitalization.
• To improve the performance of reading input datasets, consider the following options:
  • Use the Use caching parameter to store a copy of the dataset. The cached copy is only maintained while at least one browser tab open to the editor is connected. This may make it faster to access the data during processing. If the source data has been updated since it was cached, uncheck this parameter and preview or run the tool again.
  • After configuring an input dataset, configure any of the following tools that limit the amount of data being processed:
    • Filter by attribute—Maintain a subset of records that contain certain attribute values.
    • Filter by extent—Maintain a subset of records within a certain spatial extent.
    • Select fields—Maintain only the fields of interest.
    • Clip—Maintain a subset of records that intersect with specific geometries.

Connect to Snowflake

To use data stored in Snowflake, complete the following steps to create a data store item in the Data Pipelines editor:

1. On the Data Pipelines editor toolbar, click Inputs and choose Snowflake.

   The Select a data store connection dialog box appears.

2. Choose Add a new data store.
3. Click Next.

   The Add a connection to a data store dialog box appears.

4. Provide the server URL for the Snowflake account. Note that validation may fail if you specify https:// in the server URL.
   The following is an example: my_account.snowflakecomputing.com.
5. Choose one of the following authentication types:
   • Key-pair—Type the username and specify the key file (.p8) configured for the Snowflake user account.
   • User—Provide the username and password for the Snowflake user account.
6. If you provided an encrypted key file in the previous step, provide a pass code for the file using the Passcode parameter.
7. In the Database parameter, provide the name of the database that contains the datasets to connect to.
8. In the Schema parameter, provide the name of the schema used for the database.
9. In the Warehouse parameter, provide the name of the warehouse that will provide the compute resources for the connection.
10. In the Role parameter, provide the name of the role that defines the set of permissions for the connection.

    The role must have at least USAGE and CREATE STAGE privileges on the schema that contains the table that you will read from.

11. Click Next.

    The item details pane appears.

12. Provide a title for the new data store item.

    This title will appear in your content. You can also store the item in a specific folder and provide item tags or a summary.

13. Click Create connection to create the data store item.

    The Select datasets dialog box appears.

14. Provide the name of the table to use as an input dataset in the data pipeline.
15. Click Add.

    A Snowflake element is added to the canvas.

Limitations

The following are known limitations:

• User accounts that have multi-factor authentication (MFA) enabled cannot be used in Data Pipelines.
• Key-pair and user are the only authentication types supported for connecting to Snowflake in Data Pipelines.
• To use a data store item to connect to external data sources, you must be the owner of the data store item. Data store items are private and cannot be shared.

Licensing requirements

The following licensing and configurations are required:

• Creator or Professional user type
• Publisher, Facilitator, or Administrator role, or an equivalent custom role

To learn more about Data Pipelines requirements, see Requirements.