JSON

JavaScript Object Notation (JSON) is a popular and lightweight data-interchange format. ArcGIS Analytics for IoT can ingest IoT observation data expressed as JSON from a variety of sources.

JSON is supported as a data format for the following feed and data source types:

  • Feeds—Azure Event Hub, Azure Service Bus, AWS IoT, Cisco Kinetic EFM, Website (Poll), Endpoint (Receive), Kafka, WebSocket, RabbitMQ, MQTT.
  • Data sources—Azure Blob Store, Amazon S3, Website (Poll).

Specify JSON configuration

When configuring a feed or data source, sampling occurs to determine the type of data being ingested. If sampling determines the data to be JSON, additional properties for the JSON configuration can be specified.

Root node

If applicable, specify the root node of your JSON structure in which messages are found. This should be left blank if your JSON structure contains all your messages in an array at the root level.

For example, consider the following sample JSON data with multiple levels and objects:

{
  "requestInfo": {
    "requestTime": 1573591345,
    "requestingUser": "Greg Smith"
  },
  "responseInfo": {
    "dataLastUpdated": 1573239432
  },
  "sensorData": [{
    "vehicleID": "17256",
    "driverName": "Peter Jones",
    "latitude": 35.21027,
    "longitude": -81.291326,
    "speed": 64.3,
    "heading": 161.34,
    "status": "Out for delivery",
    "lastBreak": 1573591581
  }, {
    "vehicleID": "11954",
    "driverName": "Frank Jenkins",
    "latitude": 35.23425,
    "longitude": -81.092657,
    "speed": 25.8,
    "heading": 54.64,
    "status": "Returning to warehouse",
    "lastBreak": 1573581253
  }]
}

If a root node of sensorData was specified, only the following data would be ingested into Analytics for IoT:

[{
  "vehicleID": "17256",
  "driverName": "Peter Jones",
  "latitude": 35.21027,
  "longitude": -81.291326,
  "speed": 64.3,
  "heading": 161.34,
  "status": "Out for delivery",
  "lastBreak": 1573591581
}, {
  "vehicleID": "11954",
  "driverName": "Frank Jenkins",
  "latitude": 35.23425,
  "longitude": -81.092657,
  "speed": 25.8,
  "heading": 54.64,
  "status": "Returning to warehouse",
  "lastBreak": 1573581253
}]

This would then result in the schema derivation below:

Resulting Confirm Schema page with

Flatten

The Flatten parameter determines whether or not any nested JSON should be broken out into separate fields. These new fields are named by appending lower level objects with an underscore to the higher level object. In the below example, there is a position object that contains latitude and longitude objects. If flattened, the position object is broken into two fields, position_latitude and position_longitude.

If flattening, the JSON can be nested with multiple levels of depth. The new flattened field names will simply append object names with an underscore until reaching the deepest level object.

For example, consider the following sample JSON, paying attention to the position and vehicleInformation objects:

[{
  "vehicleID": "17256",
  "driverName": "Peter Jones",
  "position": {
    "latitude": 35.21027,
    "longitude": -81.291326
  },
  "vehicleInformation": {
    "lastService": 1571431836,
    "warningCodes": 2
  },
  "speed": 64.3,
  "heading": 161.34,
  "status": "Out for delivery",
  "lastBreak": 1573591581
}, {
  "vehicleID": "11954",
  "driverName": "Frank Jenkins",
  "position": {
    "latitude": 35.23425,
    "longitude": -81.092657
  },
  "vehicleInformation": {
    "lastService": 1560531836,
    "warningCodes": 2
  },
  "speed": 25.8,
  "heading": 54.64,
  "status": "Returning to warehouse",
  "lastBreak": 1573581253
}]

If the Flatten checkbox is checked, the above sample JSON would instead be processed as follows where the elements in the position and vehicleInformation objects are broken out into respective fields:

[{
  "vehicleID": "17256",
  "driverName": "Peter Jones",
  "position_latitude": 35.21027,
  "position_longitude": -81.291326,
  "vehicleInformation_lastService": 1571431836,
  "vehicleInformation_warningCodes": 2,
  "speed": 64.3,
  "heading": 161.34,
  "status": "Out for delivery",
  "lastBreak": 1573591581
}, {
  "vehicleID": "11954",
  "driverName": "Frank Jenkins",
  "position_latitude": 35.23425,
  "position_longitude": -81.092657,
  "vehicleInformation_lastService": 1560531836,
  "vehicleInformation_warningCodes": 2,
  "speed": 25.8,
  "heading": 54.64,
  "status": "Returning to warehouse",
  "lastBreak": 1573581253
}]

And the schema would be ingested as illustrated below. Note that, while the vehicleInformation_lastService and vehicleInformation_warningCodes field names are cut off by the user interface, they would be left intact by the application:

Resulting Confirm Schema page from flattened JSON.

Flattening Exemptions

Flattening Exemptions allow you to specify one or more JSON element name(s) that should be left as a single string element and not broken out into separate fields. For example, a JSON object representing an array or an object containing other elements.

Consider the following sample JSON:

[{
  "ship_ID": "17256",
  "call_sign": "3FCB8",
  "ship_type": "Cargo",
  "last_known_position": {
    "timestamp": "2019-06-24T18:30:56+00:00",
    "geometry": {
      "type": "Point",
      "coordinates": [
        79.19315,
        7.18374
      ]
    },
    "heading": 109
  }
}, {
  "ship_ID": "29435",
  "call_sign": "9GEF3",
  "ship_type": "Passenger",
  "last_known_position": {
    "timestamp": "2019-06-24T18:30:47+00:00",
    "geometry": {
      "type": "Point",
      "coordinates": [
        73.24954,
        10.43512
      ]
    },
    "heading": 120
  }
}]

If the Flatten checkbox is checked and a Flattening Exemption of geometry was specified, the above sample JSON would be treated as illustrated below. Notice that in the last_known_position_geometry field, the JSON object has not been broken up by flattening. This is because the original JSON element called geometry was specified in the Flattening Exemptions parameter. This is useful in scenarios where you want to retain JSON objects. Often, this is desired for the geometry object representing the spatial location of features. In this case, this attribute is in the GeoJSON geometry format.

[{
  "ship_ID": "17256",
  "call_sign": "3FCB8",
  "ship_type": "Cargo",
  "last_known_position_timestamp": "2019-06-24T18:30:56+00:00",
  "last_known_position_geometry": {
    "type": "Point",
    "coordinates": [
      79.19315,
      7.18374
    ]
  },
  "last_known_position_heading": 109
}, {
  "ship_ID": "29435",
  "call_sign": "9GEF3",
  "ship_type": "Passenger",
  "last_known_position_timestamp": "2019-06-24T18:30:47+00:00",
  "last_known_position_geometry": {
    "type": "Point",
    "coordinates": [
      73.24954,
      10.43512
    ]
  },
  "last_known_position_heading": 120
}]

The schema would be ingested as illustrated below and the feed or data source location information could be set to be derived from a single field.

Resulting Confirm Schema page from flattened JSON with geometry field exempted from flattening.

Considerations and limitations

When working with data in the JSON format in Analytics for IoT, there are several important considerations and limitations.

Does your JSON data have arrays?

Arrays offer a lot of flexibility in JSON, however they pose a challenge when constructing ArcGIS features from JSON data given their indeterminate length. If the entire document or a defined root node is an array of objects, individual objects will be treated as separate features. Any arrays in the JSON object will derived as strings. Further processing on the strings can be performed using analytic tools.

Are the JSON samples representative of the entire schema?

In the JSON data format, data is represented with name value pairs. If a value is null, many applications creating data in JSON format will simply omit the name object entirely.

This has an implication when first configuring or editing a feed or data source. Analytics for IoT samples the data stream or files in order to determine its best guess for the data format and schema. If all of the samples returned in the sampling process are missing JSON elements because those samples have null values, then those fields would be omitted from the feed or source data schema.

As a result, messages that came in at a later time that did have data values for those JSON elements would be unable to be parsed into observations or records that can be processed by feeds or analytics in Analytics for IoT.

Are you changing the derived field types of the JSON data?

Generic JSON data is not strongly typed, so in some cases it may be necessary to correct the field types derived from the samples. However, use caution when changing field types, the recommended best practice is to make a field type more inclusive, not less. For example, a small sample set may not reveal that a particular field should be Float64 as opposed to Float32.

Additionally, changing fields from a float type (Float32 or Float64) to an integer type (Int32 or Int64) is not recommended. Changing of field types is not intended for on-the-fly conversion of numerical values. For JSON data, downgrading from a float to an integer will cause the decimal portion of the value to be dropped without rounding. To convert or transform numerical values, use Calculate Field or Map Fields in an analytic.

JSON file size

As a best practice, JSON files being ingested should be under 100 MB per file. If you have a larger amount of data to be ingested, it is recommended you break the files up into files under 100 MB per file.