Design a real-time analytic

In this lesson, you will explore creating a real-time analytic on streaming data in ArcGIS Analytics for IoT. This lesson builds upon the Create a feed lesson in which you created a new feed to ingest AIS ship data. In this lesson, you will configure a real-time analytic to enrich the streaming AIS data with the U.S. Coast Guard District each ship is currently in, all in real-time.

In this lesson, you will learn the following skills: creating a real-time analytic, adding and configuring real-time tools, adding outputs that disseminate data, and storing features in a feature layer.

This lesson is designed for beginners. You must have an ArcGIS Online account with access to ArcGIS Analytics for IoT. Estimated time: 30 minutes.

Create a new real-time analytic

To begin, you will create a new real-time analytic on the AIS feed you created in the Create a feed lesson.

  1. Go to https://iot.arcgis.com and sign in with your ArcGIS Online credentials.

    For the best experience, use Google Chrome or Mozilla Firefox.

    Note:

    If you encounter issues signing in, contact your ArcGIS Online administrator. You may need to be assigned to an ArcGIS Online role with privileges to use ArcGIS Analytics for IoT.

  2. In the main menu on the left, click Real-Time to access the Real-Time Analytics page.

    Real-time analytics page

    From here, you can view and manage existing real-time analytics as well as create new real-time analytics.

  3. Click Create real-time analytic to launch the analytic configuration wizard.
  4. In the Create a new real-time analytic page, choose Existing feed.

    Create a real-time analytic using an existing feed

  5. In the Select existing feed window, choose the ship_positions_simulation feed and click Confirm.

    Choose an existing feed to perform real-time analytics

    Note:

    The ship_positions_simulation feed should be available if you completed the Create a feed lesson. If it is not, it is recommended you complete that lesson before continuing.

    step

    The real-time analytic editor opens, where you can configure additional tools and outputs. By default, the editor displays the chosen feed and all processing elements in a linear, sequential layout.

    Real-time analytic editor

Save the analytic

Although the analytic only contains the input feed at this point, you will save it so you can leave your work and return to it later if necessary.

  1. In the New Real-Time Analytic page, click Create Analytic to save the new real-time analytic.
  2. In the Create Analytic window, configure the title and description as follows
    1. For the Title, enter the following text:

      AIS Real Time Processing

    2. For the Description, enter the following text:

      Filter and modify incoming AIS data with streaming output and feature storage

      Save the new real-time analytic

  3. Click Create analytic to save the AIS Real Time Processing real-time analytic.

    Once the new real-time analytic is created, several new options are available in the analytic editor.

    Analytic editor after saving the real-time analytic

Add and configure an analytic tool

Next, you will add analytic tools to the AIS Real Time Processing analytic that will perform real-time analysis on the AIS data feed. With Analytics for IoT, you can build a sequence of successive analytical elements in a chain from your input(s) to output(s).

In this lesson, you will use the Join Features tool to enrich the inbound AIS data with the name of the U.S. Coast Guard District in which each ship is currently located. You will perform a spatial join between the ship features and the U.S. Coast Guard District layer.

  1. Click Toggle to model view to switch the analytic editor to the model view.

    Real-time analytic editor in model view

    This will change the analytic editor from a workflow view to model view. Model view displays a graphical canvas for configuring your analytic similar to ModelBuilder in ArcGIS Desktop or the GeoEvent Service editor in ArcGIS GeoEvent Server.

  2. Follow the steps below to add the Join Features tool to the AIS Real Time Processing analytic and connect to the ship_positions_simulation feed.
    1. Click the Summarize Data folder and choose Join Features to add the tool to the canvas.
    2. Click the Join Features tool and drag and drop it to the right of the ship_positions_simulation feed.
    3. Connect the ship_positions_simulation feed to the Join Features tool.

      Join Features tool added to apply real-time analysis

    Next, you will add the source of the enrichment data, specifically the U.S. Coast Guard District feature layer, and connect it to the Join Features tool.

  3. Click the Sources folder and choose Feature layer. Configure the feature layer source as follows:
    1. Click the My Content drop-down and choose By URL.
    2. In the Layer URL text box, enter the following URL:

      https://services2.arcgis.com/FiaPA4ga0iQKduv3/arcgis/rest/services/US_Coast_Guard_Districts/FeatureServer

    3. Select the US_Coast_Guard_Districts (0) sublayer and click Next.

      Configure the data source to the feature layer used for enrichment

    4. In the Filter Data step, accept the defaults and click Next.
    5. In the Confirm Schema step, accept the schema as sampled and click Next.

      Recall in the Create a feed lesson, you learned that Analytics for IoT will sample the data and generate a starting schema to work with.

    6. In the Identify Key Fields step, under Tracking, choose Data does not have a Track ID.

      Remember in the Create a feed lesson, you learned that this step of the configuration wizard is used to define how to construct geometry, date and time, and tracking information. Since the data source is a feature layer, the geometry and date and time information is automatically configured.

    7. Click Complete to create the new data source.

    With the new feature layer source added to the canvas, you will now connect it to the other components.

  4. Drag and drop the US_Coast_Guard_Districts source below the ship_positions_simulation feed and connect it to the Join Features tool.

    Inputs configured with a real-time analytic

    The Join Features tool now has two inputs connected to it, the ship positions feed and the U.S. Coast Guard District feature layer. Next, you will configure the Join Features tool to perform a spatial join and add the name of the U.S. Coast Guard Districts to the inbound event data.

  5. Double-click the Join Features tool to open its properties. Configure the tool as follows:
    1. For Relationship, check the Spatial checkbox and choose Intersects from the drop-down for the type of spatial relationship.

      Join Features tool with spatial relationship options

      In this lesson, you are detecting when any of the ships intersect a U.S. Coast Guard District and adding, or joining, that additional contextual information to the ship data which can then be used for increased situational awareness or further analysis. Next, you will configure the Summary fields.

    2. For Attribute, choose DistrictNa.
    3. For Statistic, choose Any.
    4. For the Output field name, change DistrictNa_Any to DistrictName.
    5. Click the Add field button to add the join operation.

      Join Features tool with summary fields defined

    6. Click Apply to apply the updated tool properties.

      Join Features tool with statistics configured

  6. In the analytic editor, click Save to save the analytic.

Configure a feature layer output

With the necessary real-time processing elements and an additional enrichment data source added, you will now add two outputs to store and visualize the data. You will start by adding and configuring an output to store the data in a new feature layer. In the next section, you will add another output to send the data to a stream layer for visualization in a web map.

  1. Click the Outputs folder and choose Feature Layer (new). Configure the feature layer output as follows:
    1. For Data storage method, choose Keep latest feature.

      This will retain only the latest observation for each track, in this case each ship. Subsequent observations will overwrite any previous observations for each track in the output feature layer.

      Note:

      The Add new features option retains all incoming data rather than just the most recently received features.

    2. For Each time the analytic runs, choose Replace existing features and schema.

      Feature Layer (new) output properties configured

      If Replace existing features and schema is checked, each time the analytic is started (or re-started), any records in the output feature layer will be deleted and the schema of the output feature layer will be regenerated. This is useful in cases where you are developing and testing your real-time analytic and adding, removing, or changing tools between runs.

      Note:

      In a production environment, you should always redefine your feature layer output and choose Keep existing features and schema so that existing data is retained if the analytic restarts.

    3. Click Next to proceed to the next step.
    4. For Feature layer name, enter the following text:

      Ships_Inside_USCG_Districts

    5. For Feature layer summary, enter the following text:

      Ships inside US Coast Guard Districts

      Feature layer (new) output layer name

    6. Click Complete to save the new feature layer output.

    The new feature layer output, Ships_Inside_USCG_Districts, will be added to the analytic editor.

  2. Connect the Join Features tool to the new Ships_Inside_USCG_Districts feature layer output.

    Join Features real-time analytic connected to the feature layer output

  3. Click Save to save the real-time analytic.

Configure a stream layer output

With the Ships_Inside_USCG_Districts feature layer output created, you will now add a second output to send the features to a stream layer that can then be visualized in a web map.

  1. Click the Outputs folder and choose Stream Layer. Configure the stream layer output as follows:
    1. Click Next to accept the defaults on the Stream Layer Configuration step.

      The options to Publish a feature layer which keeps the latest observation for each Track ID and Select a related feature layer to provide geometry when stream layer is drawn in a map are optional. Publishing a feature layer allows you to publish a separate feature layer which contains just the latest observations, useful if you want to symbolize the current position of an asset, in this case a ship, differently in a web map. Selecting a related feature layer allows you to choose a related feature layer so that it can be used to enrich the output stream layer with additional fields.

    2. For Stream layer name, enter the following text:

      Ships_Inside_USCG_Districts_stream

    3. For Stream layer summary, enter the following text:

      Ships inside US Coast Guard Districts stream

      Stream Layer output configuration

  2. Click Complete to create the new Stream Layer output.

    The new stream layer output, Ships_Inside_USCG_Districts_stream, is added to the analytic editor.

  3. Connect the Join Features tool to the Ships_Inside_USCG_Districts_stream stream layer output.

    Join Features real-time analytic connected to the stream layer output

  4. Click Save to save the real-time analytic.

Start the real-time analytic

The real-time analytic now has the necessary feeds and outputs and is ready to be started. When started, or running, the analytic will receive the simulated ship data, join to it the name of the U.S. Coast Guard District each ship is currently located, and will write the event data to the feature layer and stream layer outputs.

  1. At the top right of the analytic editor, click Start to start the real-time analytic.

    Like feeds, real-time analytics are long-running tasks and will continue to run until they are stopped.

  2. Notice the Start button transitions to Stop Initialization and then to Stop. This indicates the real-time analytic is currently running.
  3. You can monitor the status of your real-time and big data analytics from the ANALYSIS page in the Analytics for IoT application.

View the real-time analytic metrics

Metrics about running real-time analytics can be viewed in both the workflow and model views in the Analytics for IoT application. Metrics provide the average rate of events through each element and represent an average of the rate for each element across the past five minutes of runtime. Additionally, while a real-time analytic is running the metrics will represent the compute utilization of the analytic. For details, see Real-time analytic metrics.

  1. In the AIS Real Time Processing real-time analytic, click View metrics to turn on metrics for each element in the analytic.

    Turn on metrics for real-time analytics

  2. Explore the metrics for each element in the AIS Real Time Processing real-time analytic. Notice the events per second each is processing, the total number of observations in and out, as well as the compute utilization of this real-time analytic.

    Metrics for the AIS Real Time Processing real-time analytics

Examine the output layer

With the real-time analytic now running and the feature and stream layer outputs available, you will add these output layers to a web map directly from the analytic.

  1. In the main menu, click Feeds under INPUT to access the Feeds page.
  2. Click Start to start the ship_positions_simulation feed if it is not already started.
  3. In the main menu, click Real-Time under ANALYTICS to access the Real-Time Analytics page.
  4. Click Start to start the AIS Real Time Processing real-time analytic if it is not already started.
  5. Once the AIS Real Time Processing real-time analytic is running, click the edit button to open it.
  6. Right-click Ships_Inside_USCG_Districts feature layer and choose Open in Map Viewer.

    Open feature layer in a map viewer

    A new browser tab opens to the ArcGIS Online map viewer.

  7. Change the basemap to the Dark Gray Canvas basemap.

    Output feature layer in web map

  8. Optionally, click the Add button and add the U.S. Coast Guard Districts feature layer by using the URL below:

    https://services2.arcgis.com/FiaPA4ga0iQKduv3/arcgis/rest/services/US_Coast_Guard_Districts/FeatureServer

    Output feature layer in web map with U.S. Coast Guard Districts

  9. Select one of the ship position features to open the popup window and explore its attributes. Note that each observation has been enriched with the name of the U.S. Coast Guard District (DistrictName) in which the ship is currently located.

    Output feature layer in web map with popup

    The Join Features tool can be used to perform spatial enrichment and associate rich contextual information with incoming observation data.

Next steps

Congratulations! You have created a real-time analytic using ArcGIS Analytics for IoT and viewed the output features in a web map.

What's next? Take a look at other available lessons as you continue working with Analytics for IoT including Design a big data analytic and Use Arcade expressions.