Design a real-time analytic

In this lesson, you will create a real-time analytic on streaming data in ArcGIS Velocity. This lesson builds on the Create a feed lesson in which you created a new feed to collect AIS ship data. In this lesson, you will configure a real-time analytic to enrich the streaming AIS data with the U.S. Coast Guard District where each ship is located.

You will learn how to create a real-time analytic, add and configure real-time tools, add outputs that disseminate data, and store features in a feature layer.

This lesson is designed for beginners. You must have an ArcGIS Online account with access to ArcGIS Velocity. The estimated time to complete this lesson is 30 minutes.

Create a new real-time analytic

To begin, you will create a real-time analytic on the AIS feed you created in the Create a feed lesson.

1. In a web browser, open ArcGIS Velocity and sign in with your ArcGIS Online credentials.

   For the best experience, use Google Chrome or Mozilla Firefox.

   Note:

   If you encounter issues signing in, contact your ArcGIS Online administrator. You may need to be assigned an ArcGIS Online role with privileges to use ArcGIS Velocity. For more information, see the Create roles and assign users topic.

2. From the main menu on the left, click Real-Time under ANALYTICS to access the Real-Time Analytics page.

   On the Real-Time Analytics page, you can create new real-time analytics as well as view and manage existing real-time analytics.

3. Click Create real-time analytic to open the analytic configuration wizard.
4. On the Create a new real-time analytic page, click Existing feed.

   

5. In the Select existing feed window, choose the ship_positions_simulation feed and click Confirm.

   

   Note:

   The ship_positions_simulation feed is available if you completed the Create a feed lesson. If it is not available, complete that lesson before continuing.

   The real-time analytic editor opens, where you can configure additional tools and outputs. By default, the editor displays the chosen feed and all processing elements in a linear, sequential layout.

   

Create the analytic

Although the analytic only contains the input feed at this point, you will create it so you can return to it later if necessary.

1. On the New Real-Time Analytic page, click Create analytic to create the new real-time analytic.
2. On the Create Analytic window, set the Title and Summary as follows.
   1. For Title, type AIS Real Time Processing.
   2. For Summary, type Filter and modify incoming AIS data with streaming output and feature storage.
   3. For Folder, choose the folder you want to save the new analytic.
3. Click Create analytic to create the analytic.

   

   Once the analytic is created, the analytic editor opens, providing the ability to add new elements including feeds, sources, analytic tools, and outputs.

   

Add and configure an analytic tool

Next, you will add a tool to the AIS Real Time Processing analytic that will perform real-time analysis on the AIS data feed. With Velocity, you can build a sequence of successive analytic nodes that define the flow of data from inputs to outputs.

You will use the Join Features tool to enrich the inbound AIS data with the name of the U.S. Coast Guard District where each ship is located. Specifically, the tool will perform a spatial join between the ship point features and the U.S. Coast Guard District polygon layer.

1. First, you will add the Join Features tool to the AIS Real Time Processing analytic and connect it to the ship_positions_simulation feed.
   1. From the Add Node menu on the left, click the Summarize Data folder and choose the Join Features tool to add it to the editor.
   2. Drag and drop the Join Features tool to the right of the ship_positions_simulation feed.
   3. Connect the ship_positions_simulation feed to the Join Features tool.

      

   Next, you will add the source of the enrichment data—the U.S. Coast Guard District feature layer—and connect it to the Join Features tool.

2. Click the Sources folder, choose Feature Layer and configure the feature layer source as follows:
   1. Click the My Content drop-down menu and choose By URL.
   2. In the Layer URL text box, type https://services2.arcgis.com/FiaPA4ga0iQKduv3/arcgis/rest/services/US_Coast_Guard_Districts_new/FeatureServer.
   3. Select the USCG_Districts (0) sublayer and click Next.

      

   4. On the Filter Data step, set the Output spatial reference (optional) parameter to GCS WGS 1984 and click Next.
   5. On the Confirm Schema step, accept the schema as sampled and click Next.

      Velocity will sample the data and generate a schema to start with.

   6. On the Identify Key Fields step, under Tracking, choose Data does not have a Track ID.

      This step of the configuration wizard is used to define the construction of geometry, date and time, and tracking information. Since the data source is an ArcGIS feature layer, the geometry and date and time information is automatically configured.

   7. Click Complete to create and add the data source to the editor.

   With the new feature layer source added to the editor, you will now connect it to the other nodes.

3. Drag and drop the USCG_Districts source below the ship_positions_simulation feed and connect it to the Join Features tool.

   

   The Join Features tool now has two inputs connected to it; the ship positions feed and the U.S. Coast Guard District feature layer data source. Next, you will configure the Join Features tool to perform a spatial join and add the name of the U.S. Coast Guard Districts to the inbound event data.

4. Double-click the Join Features tool to open its properties and configure the tool as follows:
   1. For Join operation, click One-to-one.
   2. For Retain all features, click Only retain features that are joined.
   3. For Relationship, check the Spatial check box and choose Intersects from the drop-down menu for the type of spatial relationship.

      

      In this lesson, you are detecting when any of the ships intersect a U.S. Coast Guard District and adding, or joining, that contextual information to the ship data, which can then be used for increased situational awareness or further analysis. Next, you will set the Summary fields parameters.

   4. For Attribute, select DistrictName.
   5. For Statistic, select Any.
   6. For Output field name, delete DistrictName_Any and type DistrictName.
   7. Click Add to add the field to the join operation.
   8. Leave Change geometry of target feature to geometry of join feature set to No.
   9. Click Apply to save the properties.

      

5. In the analytic editor, click Save to save the AIS Real Time Processing analytic.

Configure a feature layer output

With the necessary real-time processing node and an enrichment data source added, you will now add two outputs to store and visualize the data. You will first add and configuring an output to store the data in a new feature layer. In the next section, you will add another output to send the data to a stream layer for visualization in a web map.

1. From the Add Node menu on the left, click the Outputs folder and choose Feature Layer (new) and configure the output as follows:
   1. For Data storage method, select Keep only latest feature for each Track ID value.

      This retains only the latest observation for each track, in this case each ship. Subsequent observations will overwrite any previous observations for each track in the output feature layer.

      Note:

      The Add all new features option retains all incoming data rather than only the most recently received features.

   2. For Each time the analytic starts, select Replace existing features and schema.

      

      With this configuration, each time the analytic is started (or restarted), all records in the output feature layer will be deleted and the schema of the output feature layer will be regenerated. This is useful when you are developing and testing a real-time analytic and adding, removing, or changing tools between analytic runs.

      Note:

      In a production environment, you can edit the feature layer output and select Keep existing features and schema so existing data is retained if the analytic restarts.

   3. Click Next to proceed to the next step.
   4. On the Save step, for Feature layer name, type Ships_Inside_USCG_Districts.
   5. For Feature layer summary (optional), type Ships inside US Coast Guard Districts.
   6. For Folder, choose a folder to save the feature layer in.
   7. Click Complete to save the new feature layer output.

   

   The new feature layer output, Ships_Inside_USCG_Districts, is added to the analytic editor.

2. Connect the Join Features tool to the new Ships_Inside_USCG_Districts feature layer output.

   

3. Click Save to save the real-time analytic.

Configure a stream layer output

With the new feature layer output added and the real-time analytic saved, next you will add a second output to send the features to a stream layer that you will visualize in a web map.

1. From the Add Node menu on the left, click the Outputs folder, choose Stream Layer, and configure the output as follows:
   1. On the Stream Layer Configuration step, click Next to accept the defaults parameters.

      The Also publish a feature layer which keeps the latest observation for each Track ID and Select a related feature layer to provide geometry when stream layer is drawn in a map parameters are not required. When you publish a feature layer, you can publish a separate feature layer that contains only the latest observations. This is useful when you want to symbolize the current position of an asset, in this case a ship, differently in a web map. You can choose a related feature layer that can be used to enrich the output stream layer with additional fields.

   2. For Stream layer name, type Ships_Inside_USCG_Districts_stream.
   3. For Stream layer summary (optional), type Ships inside US Coast Guard Districts stream.
   4. For Folder, choose the folder to save the stream layer.

   

2. Click Complete to create the stream layer output.

   The new stream layer output, Ships_Inside_USCG_Districts_stream, is added to the analytic editor.

3. Connect the Join Features tool to the Ships_Inside_USCG_Districts_stream stream layer output.

   

4. Click Save to save the real-time analytic.

Start the real-time analytic

The real-time analytic now includes the necessary feed, data source, tool, and outputs and is ready to be started. When started, or running, the analytic will receive the simulated ship data, join to it the name of the U.S. Coast Guard District where each ship is currently located, and write the event data to the feature layer and stream layer outputs.

1. In the upper right of the analytic editor, click Start.

   As with feeds, real-time analytics are long-running tasks and continue to run until they are stopped.

   The Start button text changes to Stop Initialization and then to Stop. This indicates that the real-time analytic is now running.

2. View the status and monitor the AIS Real Time Processing analytic on the Real-Time Analytics page.

View the real-time analytic metrics

Metrics about your running real-time analytics can be viewed in Velocity. Metrics provide the average rate of events through each element and represent an average of the rate for each element for the past five minutes of run time. Additionally, while a real-time analytic is running, the metrics represent the compute utilization of the analytic. For details, see Real-time analytic metrics.

1. In the AIS Real Time Processing real-time analytic, turn on the Metrics toggle to view metrics for each node in the analytic.
2. Review the metrics for each node in the real-time analytic. Notice the events per second that each node is processing, the total number of observations in and out, and the compute utilization.

   

Examine the output layer

With the real-time analytic now running and sending event data to the feature and stream layer outputs, you can now add these output layers to a web map directly in the real-time analytic.

1. From the main menu, click Feeds under INPUT to access the Feeds page.
2. If the ship_positions_simulation feed is not already started, click Start.
3. From the main menu, click Real-Time under ANALYTICS to access the Real-Time Analytics page.
4. If the AIS Real Time Processing real-time analytic is not already started, click Start.
5. Once the AIS Real Time Processing real-time analytic is running, click the pencil icon to edit the analytic.
6. Right-click the Ships_Inside_USCG_Districts feature layer output and select Open in Map Viewer.

   

   A new browser tab opens the Map Viewer in ArcGIS Online and adds the feature layer to the web map.

7. Change the basemap to the Dark Gray Canvas.

   

8. Click Add and choose Add Layer from Web.
9. Add the U.S. Coast Guard Districts feature layer using the URL https://services2.arcgis.com/FiaPA4ga0iQKduv3/arcgis/rest/services/US_Coast_Guard_Districts_new/FeatureServer.

   

10. Click one of the ship features to open the pop-up and explore its attributes. Each observation is enriched with the name of the U.S. Coast Guard District (DistrictName) the ship is currently inside.

    

Next steps

You created a real-time analytic using ArcGIS Velocity and viewed the output features in a web map. Then you added the Join Features tool to perform spatial enrichment and associated contextual information with the ingested observation data.

Next, explore the Design a big data analytic lesson that introduces you to working with big data analytics in Velocity. If interested in using Arcade expressions, see the Use Arcade expressions topic.