Supported data formats

Every feed or data source type in ArcGIS Analytics for IoT supports either an inherent data format or a variety of formats. When configuring input data, Analytics for IoT will automatically sample for messages or records and attempt to determine the format of your data.

The following data formats are supported:

  • Delimited
  • JSON
  • GeoJSON
  • EsriJSON
  • XML
  • RSS
  • GeoRSS
  • Shapefile
  • Parquet

Delimited

Delimited data is supported for most feed and data source types. When ingesting delimited data, you can specify the field separator and whether or not there is a header row.

It is important to pay close attention to the fields and field types derived by Analytics for IoT when delimited data is sampled. Delimited data is not strongly typed, meaning that field/attribute types are not specified explicitly within the data. For example, a small sample set may not reveal that a particular field should be Int64 as opposed to Int32.

Additionally, changing fields from a float type (Float32 or Float64) to an integer type (Int32 or Int64) is not recommended. Changing of field types is not intended for on-the-fly conversion of numerical values. For delimited data, downgrading from a float to an integer can cause the value to be skipped entirely. To convert or transform numerical values, use Calculate Field or Map Fields within an analytic.

JSON

JavaScript Object Notation (JSON) is a popular and lightweight data-interchange format. JSON data is supported for most feed and data source types. When ingesting JSON data, you can specify a root node containing the array of messages, whether or not the data has lines, as well as whether or not to flatten a JSON structure with multiple levels or exempt certain fields for flattening.

For more information, see JSON which includes detailed JSON samples and configuration parameter examples.

EsriJSON

EsriJSON (also referred to as a FeatureSet object) can be ingested by Analytics for IoT as either a FeatureSet (collection of features) or individual features separated by lines in a file. In either configuration, the EsriJSON data ingested should have a specific geometry type, fields, and a spatial reference. EsriJSON data is supported for most feed and data source types. When ingesting EsriJSON data, you can specify whether or not the data has lines.

For more information, see EsriJSON which includes EsriJSON samples and configuration parameter examples.

GeoJSON

GeoJSON is a format for encoding a variety of geographic data structures. GeoJSON data is supported for most feed and data source types. When ingesting GeoJSON data, you can specify whether or not the data has lines.

For more information, see GeoJSON which includes GeoJSON samples and configuration parameter examples.

XML

Extensible Markup Language (XML) is a widely used data format for both tabular and geographic data. XML data is supported for most feed and data source types in Analytics for IoT. When ingesting XML data, you can specify a root element as well as whether or not to flatten an XML structure with multiple levels or exempt certain elements for flattening.

For more information, see XML which includes XML samples and configuration parameter examples.

RSS

The RSS data format is supported specifically with the RSS feed and data source type.

Analytics for IoT supports RSS in either ATOM or RSS 2.0 formats.

GeoRSS

The GeoRSS data format is supported specifically with the RSS feed and data source type. GeoRSS is a data format used for live web feeds that include geographic features and locations.

Analytics for IoT supports GeoRSS in either GeoRSS Simple or GeoRSS GML encodings.

Shapefile

The shapefile data format is supported with the Amazon S3 and Azure Blob Store data source types only.

A shapefile is an Esri vector data storage format for storing the location, shape, and attributes of geographic features. It is stored as a set of related files and contains one feature class. Shapefiles often contain large features with a lot of associated data and historically have been used in GIS desktop applications such as ArcMap.

The path specified in your Amazon S3 or Azure Blob Store data source must include at least the .shp, .shx, .dbf, and .prj file components of the shapefile.

Analytics for IoT does not support shapefiles contained within .zip files.

Parquet

The Parquet data format is supported with the Amazon S3, Azure Blob Store, and Feature Layer (archive) data source types only.

A Parquet file is an open-source columnar oriented data storage format from Apache. For more information, see Apache Parquet.