Amazon S3

Read records from files stored in an Amazon S3 bucket and perform analysis in ArcGIS Velocity.

Examples

  • A researcher wants to load hundreds of delimited text files stored in an Amazon S3 bucket to perform analysis in Velocity.
  • A GIS department stores commonly used boundary shapefiles in an Amazon S3 bucket and wants to load the county boundary shapefile into Velocity as an aggregation boundary.

Usage notes

  • All files identified in the Amazon S3 bucket by the naming pattern in the dataset property must have the same schema and geometry type.
  • The secret access key will be encrypted the first time the analytic is saved and is stored in an encrypted state.
  • When specifying the Folder path, use forward slashes (/).
  • After configuring source connection properties, see Configure input data to learn how to define the schema and the key properties.
  • Certain Amazon S3 actions are required for the user policy associated with the provided Amazon key in order for Velocity to be able to successfully connect to a Amazon S3 bucket and data in the provided bucket and folder path.
    • The s3:ListBucket action is required to be allowed for the specified bucket.
    • The s3:GetObject action is required to be allowed on the specified folder path and subresources (arn:aws:s3:::yourBucketName/*) for an Amazon S3 source to read data.

Parameters

ParameterExplanationData Type

Access key

Amazon access key ID for the S3 bucket.

For example: AKIAIOSFODNN7EXAMPLE.

Velocity will use the provided access keys to load specified data sources into the application.

For details on Amazon access keys, see Understanding and getting your AWS credentials in the AWS documentation.

String

Secret key

Amazon secret access key for the S3 bucket.

For example: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Velocity will use the provided access keys to load specified data sources into the application.

Your secret access key will be encrypted the first time the analytic is saved and will be stored in an encrypted state.

For details on Amazon secret access keys, see Understanding and getting your AWS credentials in the AWS documentation.

String

S3 bucket name

The name of the Amazon S3 bucket containing the files to read.

String

Folder path

The folder path of the folder containing the file(s) to be loaded into Velocity.

  • If loading the file(s) from the root level of an Amazon S3 bucket, enter a single forward slash. For example:
    • /
  • If loading the file(s) from a folder in the Amazon S3 bucket, enter a forward slash followed by the path to the folder. For example:
    • /gis_data_folder/folder_containing_desired_dataset

String

Dataset

The name of the file to read if loading a single file, or a pattern indicating a set of files, followed by the file type extension.

In order to build a pattern indicating a set of files an asterisk (*) can be used as a wildcard either on its own or in conjunction with a partial file name.

All files identified by the naming pattern must have the same schema and geometry type. For example:

  • If loading a single file in a folder:
    • filename.csv
  • If loading all files in a folder:
    • *.shp
  • If loading select files in a folder:
    • sensor_data_201*.json

String

Load recent files only

Specifies whether the Amazon S3 source should load all files or only the files created or modified since the last run of the analytic.

  • By default, the parameter is set to false, meaning each time the analytic runs, all files in the specified bucket and path with the provided dataset name will be loaded.
  • When set to true, only files that were modified or created will be loaded by each run of the analytic.

The parameter can only be set to true for scheduled big data analytics.

For the first run of a scheduled big data analytic with the parameter set to true, big data analytics will not load any files and the analytic run will complete. Subsequent analytic runs will load files with a last modified date since the last scheduled run of the analytic.

Boolean

Considerations and limitations

  • All files identified in the Amazon S3 bucket by the naming pattern in the dataset property must have the same schema and geometry type.
  • Ingesting JSON with an array of objects referenced by a root node is currently not supported for Amazon S3 or Azure Blob Store.