JavaScript Object Notation (JSON) is a popular and lightweight data interchange format. ArcGIS Velocity can use JSON generated from Internet of Things (IoT) observation data from a variety of sources.
JSON is supported as a data format for the following feed and data source types:
- Feeds—Azure Event Hub, Azure Service Bus, AWS IoT, Cisco Edge Intelligence, HTTP Poller, HTTP Receiver, Kafka, WebSocket, RabbitMQ, MQTT
- Data sources—Azure Blob Storage, Amazon S3, HTTP Poller
Specify the JSON configuration
When configuring a feed or data source, sampling occurs to identify the type of data being used. If sampling identifies the data to be JSON, the following additional parameters for the JSON configuration can be specified:
Root node
If applicable, specify the root node of the JSON structure in which messages reside. Leave this blank if the JSON structure contains all the messages in an array at the root level. For example, the following sample JSON shows data with multiple levels and objects:
{
"requestInfo": {
"requestTime": 1573591345,
"requestingUser": "Greg Smith"
},
"responseInfo": {
"dataLastUpdated": 1573239432
},
"sensorData": [{
"vehicleID": "17256",
"driverName": "Peter Jones",
"latitude": 35.21027,
"longitude": -81.291326,
"speed": 64.3,
"heading": 161.34,
"status": "Out for delivery",
"lastBreak": 1573591581
}, {
"vehicleID": "11954",
"driverName": "Frank Jenkins",
"latitude": 35.23425,
"longitude": -81.092657,
"speed": 25.8,
"heading": 54.64,
"status": "Returning to warehouse",
"lastBreak": 1573581253
}]
}
If a root node of sensorData was specified, only the following data is used in Velocity:
[{
"vehicleID": "17256",
"driverName": "Peter Jones",
"latitude": 35.21027,
"longitude": -81.291326,
"speed": 64.3,
"heading": 161.34,
"status": "Out for delivery",
"lastBreak": 1573591581
}, {
"vehicleID": "11954",
"driverName": "Frank Jenkins",
"latitude": 35.23425,
"longitude": -81.092657,
"speed": 25.8,
"heading": 54.64,
"status": "Returning to warehouse",
"lastBreak": 1573581253
}]
The result is the schema derivation below:
Flatten
The Flatten parameter determines whether nested JSON is divided into separate fields. The new fields are named by appending lower level objects with an underscore to the higher level object. In the example below, the position object contains latitude and longitude objects. When flattened, the position object is divided into two fields, position_latitude and position_longitude.
When flattening, the JSON can be nested with multiple levels of depth. The new flattened field names append object names with an underscore until reaching the deepest level object.
For example, the following sample JSON shows the position and vehicleInformation objects:
[{
"vehicleID": "17256",
"driverName": "Peter Jones",
"position": {
"latitude": 35.21027,
"longitude": -81.291326
},
"vehicleInformation": {
"lastService": 1571431836,
"warningCodes": 2
},
"speed": 64.3,
"heading": 161.34,
"status": "Out for delivery",
"lastBreak": 1573591581
}, {
"vehicleID": "11954",
"driverName": "Frank Jenkins",
"position": {
"latitude": 35.23425,
"longitude": -81.092657
},
"vehicleInformation": {
"lastService": 1560531836,
"warningCodes": 2
},
"speed": 25.8,
"heading": 54.64,
"status": "Returning to warehouse",
"lastBreak": 1573581253
}]
When the Flatten parameter is checked, the sample JSON above is processed as follows in which the elements in the position and vehicleInformation objects are divided into respective fields:
[{
"vehicleID": "17256",
"driverName": "Peter Jones",
"position_latitude": 35.21027,
"position_longitude": -81.291326,
"vehicleInformation_lastService": 1571431836,
"vehicleInformation_warningCodes": 2,
"speed": 64.3,
"heading": 161.34,
"status": "Out for delivery",
"lastBreak": 1573591581
}, {
"vehicleID": "11954",
"driverName": "Frank Jenkins",
"position_latitude": 35.23425,
"position_longitude": -81.092657,
"vehicleInformation_lastService": 1560531836,
"vehicleInformation_warningCodes": 2,
"speed": 25.8,
"heading": 54.64,
"status": "Returning to warehouse",
"lastBreak": 1573581253
}]
The schema is used as illustrated below. While the vehicleInformation_lastService and vehicleInformation_warningCodes field names are cut off by the user interface, they are left intact by the application:
Flattening Exemptions
The Flattening Exemptions parameter allows you to specify one or more JSON element names that are to be left as a single string element instead of divided into separate fields, for example, a JSON object representing an array or an object containing other elements. Consider the following sample JSON:
[{
"ship_ID": "17256",
"call_sign": "3FCB8",
"ship_type": "Cargo",
"last_known_position": {
"timestamp": "2019-06-24T18:30:56+00:00",
"geometry": {
"type": "Point",
"coordinates": [
79.19315,
7.18374
]
},
"heading": 109
}
}, {
"ship_ID": "29435",
"call_sign": "9GEF3",
"ship_type": "Passenger",
"last_known_position": {
"timestamp": "2019-06-24T18:30:47+00:00",
"geometry": {
"type": "Point",
"coordinates": [
73.24954,
10.43512
]
},
"heading": 120
}
}]
When the Flatten parameter is checked and a Flattening Exemptions value of geometry is specified, the sample JSON above is treated as illustrated below. In the last_known_position_geometry field, the JSON object has not been divided by flattening. This is because the original geometry JSON element was specified in the Flattening Exemptions parameter. This is useful when you want to retain JSON objects. This is often the case when the geometry object represents the spatial location of features. In this case, this attribute is in the GeoJSON geometry format.
[{
"ship_ID": "17256",
"call_sign": "3FCB8",
"ship_type": "Cargo",
"last_known_position_timestamp": "2019-06-24T18:30:56+00:00",
"last_known_position_geometry": {
"type": "Point",
"coordinates": [
79.19315,
7.18374
]
},
"last_known_position_heading": 109
}, {
"ship_ID": "29435",
"call_sign": "9GEF3",
"ship_type": "Passenger",
"last_known_position_timestamp": "2019-06-24T18:30:47+00:00",
"last_known_position_geometry": {
"type": "Point",
"coordinates": [
73.24954,
10.43512
]
},
"last_known_position_heading": 120
}]
The schema is used as illustrated below and the feed or data source location information can be set to be derived from a single field.
Flatten arrays
The Flatten arrays parameter determines whether arrays are flattened into separate fields. New fields are named by appending the arrayName value with an underscore and the number of the array object. If the array is an array of objects, the field name also appends an underscore and the name associated with the value. When flattening arrays of objects, the JSON can be nested with multiple levels of depth. The new flattened field names append object names with an underscore until reaching the deepest object.
For example, the following sample JSON shows the cargoInformation array object:
{
"name": "iss",
"id": 25544,
"latitude": -13.210305423781,
"longitude": -118.47517068897,
"velocity": 27564.442053654,
"visibility": "daylight",
"cargoInformation": [{
"sensorsID": 1,
"age": 5432
}, {
"sensorsID": 2,
"age": 9876
}
],
"footprint": 4521.609615023,
"timestamp": 1654531549,
"daynum": 2459737.170706,
"solar_lat": 22.696540649846,
"solar_lon": 298.23268983658,
"units": "kilometers"
}
When the Flatten arrays parameter is checked, the sample JSON above is processed as follows in which the elements in the cargo information object are divided into respective fields:
{
"name": "iss",
"id": 25544,
"latitude": -13.210305423781,
"longitude": -118.47517068897,
"velocity": 27564.442053654,
"visibility": "daylight",
"cargoInformation_0_sensorsID": 1,
"cargoInformation_0_age": 5432,
"cargoInformation_1_sensorsID": 2,
"cargoInformation_1_age": 9876,
"footprint": 4521.609615023,
"timestamp": 1654531549,
"daynum": 2459737.170706,
"solar_lat": 22.696540649846,
"solar_lon": 298.23268983658,
"units": "kilometers"
}
The schema is used as illustrated below. While the cargoInformation field names are cut off in the user interface, they are left intact by the application.
Array flattening exemptions
The Array flattening exemptions parameter allows you to specify one or more JSON array names that are to be left as a single string element instead of divided into separate fields. Consider the following sample JSON:
{
"name": "iss",
"id": 25544,
"latitude": -13.210305423781,
"longitude": -118.47517068897,
"velocity": 27564.442053654,
"visibility": "daylight",
"cargoInformation": [{
"sensorsID": 1,
"age": 5432
}, {
"sensorsID": 2,
"age": 9876
}
],
"sensorValues": [1,2,3,4,5],
"footprint": 4521.609615023,
"timestamp": 1654531549,
"daynum": 2459737.170706,
"solar_lat": 22.696540649846,
"solar_lon": 298.23268983658,
"units": "kilometers"
}
If the Flatten arrays parameter is checked and an Array flattening exemptions value of sensorValues is specified, the sample JSON above is treated as illustrated below. In the sensorValues array, the array object is not divided by flattening and is left as a string. To indicate that multiple arrays are to be exempted from flattening, separate them with commas in the Array flattening exemptions parameter.
{
"name": "iss",
"id": 25544,
"latitude": -13.210305423781,
"longitude": -118.47517068897,
"velocity": 27564.442053654,
"visibility": "daylight",
"cargoInformation_0_sensorsID": 1,
"cargoInformation_0_age": 5432,
"cargoInformation_1_sensorsID": 2,
"cargoInformation_1_age": 9876,
"sensorValues": [1,2,3,4,5],
"footprint": 4521.609615023,
"timestamp": 1654531549,
"daynum": 2459737.170706,
"solar_lat": 22.696540649846,
"solar_lon": 298.23268983658,
"units": "kilometers"
}
The schema is used as illustrated below.
Considerations and limitations
When working with data in the JSON format in Velocity, consider the information and limitations described below.
JSON data containing arrays
Arrays offer flexibility in JSON; however they pose a challenge when constructing ArcGIS features from JSON data because of their indeterminate length. If the entire document or a defined root node is an array of objects, individual objects are treated as separate features. Arrays in the JSON object are derived as strings. You can perform further processing on the strings using analytic tools.
Null data values
In JSON format, data is represented with name value pairs. If a value is null, many applications creating data in JSON format omit the name object.
This has an implication when first configuring or editing a feed or data source. Velocity samples the data stream or files to determine the data format and schema. If all of the samples returned in the sampling process are missing JSON elements because those samples have null values, those fields are omitted from the feed or source data schema.
As a result, messages that came in at a later time that did have data values for those JSON elements cannot be parsed into observations or records that can be processed by feeds or analytics in Velocity.
Derived field types
Generic JSON data is not strongly typed; it may be necessary to correct the field types derived from the samples. Care should be taken when changing field types. A best practice is to make a field type more inclusive, not less. For example, a small sample set may not reveal that a particular field value should be Float64 as opposed to Float32.
Additionally, changing fields from a float type (Float64 or Float32) to an integer type (Int64 or Int32) is not recommended. Changing of field types is also not recommended for on-the-fly conversion of numerical values. For JSON data, downgrading from a float to an integer causes the decimal portion of the value to be dropped without rounding. To convert or transform numerical values, use Calculate Field or Map Fields.
JSON file size
As a best practice, keep JSON files under 100 MB per file in Velocity. If you have a larger amount of data, divide the files so each file is less than 100 MB in size.