XML—ArcGIS Velocity | Documentation

Extensible Markup Language (XML) is a common text-based markup language. ArcGIS Velocity can ingest IoT observation data expressed as XML from a variety of sources.

XML is supported as a data format for the following feed and data source types:

Feeds—Azure Event Hub, Azure Service Bus, Website (Poll), RabbitMQ, WebSocket, Kafka, MQTT, AWS IoT, Endpoint (Receive).
Data sources—Amazon S3, Azure Blob Store, Website (Poll).

Specify XML configuration

When configuring a feed or data source, sampling occurs to understand the type of data being ingested. If sampling determines the data to be XML, additional properties regarding the XML configuration can be specified.

Root element

If applicable, specify the Root element of your XML structure in which messages are found. The root element should identify an object whose sub-elements are attributes. This should be left blank if your XML structure contains all your messages beneath the root of the XML document.

For example, consider the following sample XML data with multiple elements:

<?xml version="1.0" encoding="UTF-8"?>
<flight>
  <aircraftData>
    <aircraftID>N-X-211</aircraftID>
    <pilot>Charles L.</pilot>
    <speed>212.28</speed>
    <altitude>20200</altitude>
    <heading>44.02</heading>
    <latitude>52.653726</latitude>
    <longitude>-21.885421</longitude>
    <status>In flight</status>
    <datetime>1589962024</datetime>
  </aircraftData>
  <aircraftData>
    <aircraftID>N02061</aircraftID>
    <pilot>Jane D.</pilot>
    <speed>132.00</speed>
    <altitude>13008</altitude>
    <heading>264.54</heading>
    <latitude>19.510083</latitude>
    <longitude>-153.950114</longitude>
    <status>In flight</status>
    <datetime>1589962042</datetime>
  </aircraftData>
  <airportData>
    <name>John F. Kennedy Int. Airport</name>
    <IATA>JFK</IATA>
    <ICAO>KJFK</ICAO>
    <date>1589932800</date>
    <location>
      <latitude>40.643585</latitude>
      <longitude>-73.781927</longitude>
    </location>
  </airportData>
</flight>

If a root element of aircraftData was specified, only the following data would be ingested into Velocity:

<aircraftData>
    <aircraftID>N-X-211</aircraftID>
    <pilot>Charles L.</pilot>
    <speed>212.28</speed>
    <altitude>20200</altitude>
    <heading>44.02</heading>
    <latitude>52.653726</latitude>
    <longitude>-21.885421</longitude>
    <status>In flight</status>
    <datetime>1589962024</datetime>
  </aircraftData>
  <aircraftData>
    <aircraftID>N02061</aircraftID>
    <pilot>Jane D.</pilot>
    <speed>132.00</speed>
    <altitude>13008</altitude>
    <heading>264.54</heading>
    <latitude>19.510083</latitude>
    <longitude>-153.950114</longitude>
    <status>In flight</status>
    <datetime>1589962042</datetime>
  </aircraftData>

This would result in the schema derivation illustrated below:

Flatten

The Flatten parameter determines whether any nested XML should be broken out into separate fields. These new fields are named by appending sub-elements with an underscore to the higher level, or parent, element. In the below example, there is a location element that contains two sub-elements: latitude and longitude. If flattened, the location element is broken into two fields, location_latitude and location_longitude.

If flattening, the XML can be nested with multiple levels of depth. The new flattened field names will simply append object names with an underscore until reaching the deepest level sub-element.

For example, consider the following sample XML, paying attention to the codes and location elements:

<?xml version="1.0" encoding="UTF-8"?>
<airportData>
  <airport>
    <name>John F. Kennedy Int. Airport</name>
    <codes>
      <IATA>JFK</IATA>
      <ICAO>KJFK</ICAO>
    </codes>
    <status>Operational</status>
    <date>1589932800</date>
    <location>
      <latitude>40.643585</latitude>
      <longitude>-73.781927</longitude>
    </location>
  </airport>
  <airport>
    <name>Los Angeles Int. Airport</name>
    <codes>
      <IATA>LAX</IATA>
      <ICAO>KLAX</ICAO>
    </codes>
    <status>Delayed</status>
    <date>1589932800</date>
    <location>
      <latitude>33.942619</latitude>
      <longitude>-118.420942</longitude>
    </location>
  </airport>
</airportData>

If airport was specified as the Root element and the Flatten checkbox was checked, the above sample XML would instead be processed as follows where the codes and location sub-elements are broken out into respective fields:

<airport>
    <name>John F. Kennedy Int. Airport</name>
    <codes_IATA>JFK</codes_IATA>
    <codes_ICAO>KJFK</codes_ICAO>
    <status>Operational</status>
    <date>1589932800</date>
    <location_latitude>40.643585</location_latitude>
    <location_longitude>-73.781927</location_longitude>
  </airport>
  <airport>
    <name>Los Angeles Int. Airport</name>
    <codes_IATA>LAX</codes_IATA>
    <codes_ICAO>KLAX</codes_ICAO>
    <status>Delayed</status>
    <date>1589932800</date>
    <location_latitude>33.942619</location_latitude>
    <location_longitude>-118.420942</location_longitude>
  </airport>

And the schema would be ingested as illustrated below:

Resulting Confirm Schema page from flattened XML.

Flattening Exemptions

The Flattening Exemptions parameter allow you to specify one or more XML element name(s) that should be left as a single string element and not broken out into separate fields.

Consider the following sample XML:

<?xml version="1.0" encoding="UTF-8"?>
<airportData>
  <airport>
    <name>John F. Kennedy Int. Airport</name>
    <codes>
      <IATA>JFK</IATA>
      <ICAO>KJFK</ICAO>
    </codes>
    <status>Operational</status>
    <date>1589932800</date>
    <location>
      <latitude>40.643585</latitude>
      <longitude>-73.781927</longitude>
    </location>
  </airport>
  <airport>
    <name>Los Angeles Int. Airport</name>
    <codes>
      <IATA>LAX</IATA>
      <ICAO>KLAX</ICAO>
    </codes>
    <status>Delayed</status>
    <date>1589932800</date>
    <location>
      <latitude>33.942619</latitude>
      <longitude>-118.420942</longitude>
    </location>
  </airport>
</airportData>

If airport was specified as the Root element, the Flatten checkbox is checked, and a Flattening Exemption of codes was specified, the above sample XML would be treated as illustrated below. Notice that the codes element has not been broken up by flattening. IATA and ICAO are maintained as sub-elements.

<airport>
    <name>John F. Kennedy Int. Airport</name>
    <codes>
      <IATA>JFK</IATA>
      <ICAO>KJFK</ICAO>
    </codes>
    <status>Operational</status>
    <date>1589932800</date>
    <location_latitude>40.643585</location_latitude>
    <location_longitude>-73.781927</location_longitude>
  </airport>
  <airport>
    <name>Los Angeles Int. Airport</name>
    <codes>
      <IATA>LAX</IATA>
      <ICAO>KLAX</ICAO>
    </codes>
    <status>Delayed</status>
    <date>1589932800</date>
    <location_latitude>33.942619</location_latitude>
    <location_longitude>-118.420942</location_longitude>
  </airport>

The schema would be ingested as illustrated below:

Resulting Confirm Schema page from flattened XML with codes field exempted from flattening.

Considerations and limitations

When working with XML formatted data in Velocity, there are several important considerations and limitations.

Does your XML data have nested sub-elements?

XML requires a hierarchy of elements that begins and ends with a root element. Below a root element can be an unspecified number of additional elements and sub-elements. Currently, Velocity derives its schema from the primary element(s) found beneath the specified root. Any additional sub-elements beneath the primary elements are handled as concatenated strings. Further processing on the strings can be performed using analytic tools. Consider using the Root element parameter to define a new root in your XML document.

Are the XML samples representative of the entire schema?

In an XML document, elements sharing an identical name do not always have the same sub-elements. This has an implication when first configuring, editing, or eventually consuming an XML based feed or data source. Velocity samples the data stream or files to determine its best guess for the data format and schema. If all the samples returned in the sample process are missing XML elements, because those samples have a different schema, then those fields would be omitted from the feed or source data schema.

XML file size

As a best practice, XML files being ingested should be under 100 MB per file. If you have a larger amount of data to be ingested, it is recommended you break the files up into files under 100 MB per file.

Feedback on this topic?