Design a real-time analytic

This lesson builds on the Create a feed lesson in which you created a new feed to collect AIS ship data. It demonstrates how to create a real-time analytic on streaming data in Velocity by configuring a real-time analytic to enrich the streaming AIS data with the U.S. Coast Guard District where each ship is located. It goes over adding and configuring real-time tools, adding outputs that disseminate data, and storing features in a feature layer.

This lesson is designed for beginners. You must have an ArcGIS Online account with access to ArcGIS Velocity. The estimated time to complete this lesson is 30 minutes.

Create a new real-time analytic

To begin, create a real-time analytic on the AIS feed that was created in the Create a feed lesson.

1. In a web browser, open ArcGIS Velocity and sign in with your ArcGIS Online credentials.

   Google Chrome or Mozilla Firefox is recommended.

   Note:

   If you encounter issues signing in, contact your ArcGIS Online administrator. You may need to be assigned an ArcGIS Online role with privileges to use Velocity.

   Learn more about creating roles and assigning users

2. From the main menu on the left, click Real-Time under ANALYTICS to access the Real-Time Analytics page.

   On the Real-Time Analytics page, you can create new real-time analytics as well as view and manage existing real-time analytics.

3. Click Create real-time analytic to open the analytic configuration wizard.
4. On the Create a new real-time analytic page, click Existing feed.

   

5. In the Select existing feed dialog box, choose the ship_positions_simulation feed and click Confirm.

   

   Note:

   The ship_positions_simulation feed is available if you completed the Create a feed lesson. If it is not available, complete that lesson before continuing.

   The real-time analytic editor appears, where you can configure additional tools and outputs. By default, the editor displays the chosen feed and all processing elements in a linear, sequential layout.

   

Create the analytic

Although the analytic only contains the input feed at this point, you create it so you can return to it later if necessary.

1. On the New Real-Time Analytic page, click Create analytic to create the new real-time analytic.
2. On the Create Analytic window, set the Title and Summary as follows.
   1. For Title, type AIS Real Time Processing.
   2. For Summary, type Filter and modify incoming AIS data with streaming output and feature storage.
   3. For Folder, choose the folder you want to save the new analytic.
3. Click Create analytic to create the analytic.

   

   The analytic editor appears where you can add new elements including feeds, sources, analytic tools, and outputs.

   

Add and configure an analytic tool

Next, add a tool to the AIS Real Time Processing analytic to perform real-time analysis on the AIS data feed. With Velocity, you can build a sequence of successive analytic nodes that define the flow of data from inputs to outputs.

Use the Join Features tool to enrich the inbound AIS data with the name of the U.S. Coast Guard District where each ship is located. The tool performs a spatial join between the ship point features and the U.S. Coast Guard District polygon layer.

1. Add the Join Features tool to the AIS Real Time Processing analytic and connect it to the ship_positions_simulation feed.
   1. From the Add Node menu on the left, click the Summarize Data folder and choose the Join Features tool to add it to the editor.
   2. Drag and drop the Join Features tool to the right of the ship_positions_simulation feed.
   3. Connect the ship_positions_simulation feed to the Join Features tool.

      

   Add the source of the enrichment data—the U.S. Coast Guard District feature layer—and connect it to the Join Features tool.

2. Click the Sources folder, choose Feature Layer and configure the feature layer source as follows:
   1. Click the My Content drop-down menu and choose By URL.
   2. In the Layer URL text box, type https://services2.arcgis.com/FiaPA4ga0iQKduv3/arcgis/rest/services/US_Coast_Guard_Districts_new/FeatureServer.
   3. Choose the USCG_Districts (0) sublayer and click Next.

      

   4. On the Filter Data step, set the Output spatial reference (optional) parameter to GCS WGS 1984 and click Next.
   5. On the Confirm Schema step, accept the schema as sampled and click Next.

      Velocity samples the data and generates a schema to start with.

   6. On the Identify Key Fields step, under Tracking, choose Data does not have a Track ID.

      This step of the configuration wizard is used to define the construction of geometry, date and time, and tracking information. Since the data source is an ArcGIS feature layer, the geometry and date and time information is automatically configured.

   7. Click Complete to create and add the data source to the editor.

   With the new feature layer source added to the editor, connect it to the other nodes.

3. Drag and drop the USCG_Districts source below the ship_positions_simulation feed and connect it to the Join Features tool.

   

   The Join Features tool now has two inputs connected to it; the ship positions feed and the U.S. Coast Guard District feature layer data source. Next, configure the Join Features tool to perform a spatial join and add the name of the U.S. Coast Guard Districts to the inbound event data.

4. Double-click the Join Features tool to open its properties and configure the tool as follows:
   1. For Join operation, click One-to-one.
   2. For Retain all features, click Only retain features that are joined.
   3. For Relationship, check the Spatial check box and choose Intersects from the drop-down menu for the type of spatial relationship.

      

      In this lesson, you are detecting when any of the ships intersect a U.S. Coast Guard District and adding, or joining, that contextual information to the ship data, which can then be used for increased situational awareness or further analysis. Next, set the Summary fields parameters.

   4. For Attribute, choose DistrictName.
   5. For Statistic, choose Any.
   6. For Output field name, delete DistrictName_Any and type DistrictName.
   7. Click Add to add the field to the join operation.
   8. Leave Change geometry of target feature to geometry of join feature set to No.
   9. Click Apply to save the properties.

      

5. In the analytic editor, click Save to save the AIS Real Time Processing analytic.

Configure a feature layer output

With the necessary real-time processing node and an enrichment data source added, now add two outputs to store and visualize the data. First add and configure an output to store the data in a new feature layer. In the next section, add another output to send the data to a stream layer for visualization in a web map.

1. From the Add Node menu on the left, click the Outputs folder and choose Feature Layer (new) and configure the output as follows:
   1. For Data storage method, choose Keep only latest feature for each Track ID value.

      This retains only the latest observation for each track, in this case each ship. Subsequent observations will overwrite any previous observations for each track in the output feature layer.

      Note:

      The Add all new features option retains all incoming data rather than only the most recently received features.

   2. For Each time the analytic starts, choose Replace existing features and schema.

      

      With this configuration, each time the analytic is started (or restarted), all records in the output feature layer will be deleted and the schema of the output feature layer will be regenerated. This is useful when you are developing and testing a real-time analytic and adding, removing, or changing tools between analytic runs.

      Note:

      In a production environment, you can edit the feature layer output and choose Keep existing features and schema so existing data is retained if the analytic restarts.

   3. Click Next to proceed to the next step.
   4. On the Save step, for Feature layer name, type Ships_Inside_USCG_Districts.
   5. For Feature layer summary (optional), type Ships inside US Coast Guard Districts.
   6. For Folder, choose a folder to save the feature layer in.
   7. Click Complete to save the new feature layer output.

   

   The new feature layer output, Ships_Inside_USCG_Districts, is added to the analytic editor.

2. Connect the Join Features tool to the new Ships_Inside_USCG_Districts feature layer output.

   

3. Click Save to save the real-time analytic.

Configure a stream layer output

With the new feature layer output added and the real-time analytic saved, next add a second output to send the features to a stream layer that you will visualize in a web map.

1. From the Add Node menu on the left, click the Outputs folder, choose Stream Layer, and configure the output as follows:
   1. On the Stream Layer Configuration step, click Next to accept the defaults parameters.

      The Also publish a feature layer which keeps the latest observation for each Track ID and Select a related feature layer to provide geometry when stream layer is drawn in a map parameters are not required. When you publish a feature layer, you can publish a separate feature layer that contains only the latest observations. This is useful when you want to symbolize the active position of an asset, in this case a ship, differently in a web map. You can choose a related feature layer that can be used to enrich the output stream layer with additional fields.

   2. For Stream layer name, type Ships_Inside_USCG_Districts_stream.
   3. For Stream layer summary (optional), type Ships inside US Coast Guard Districts stream.
   4. For Folder, choose the folder to save the stream layer.

   

2. Click Complete to create the stream layer output.

   The new stream layer output, Ships_Inside_USCG_Districts_stream, is added to the analytic editor.

3. Connect the Join Features tool to the Ships_Inside_USCG_Districts_stream stream layer output.

   

4. Click Save to save the real-time analytic.

Start the real-time analytic

The real-time analytic now includes the necessary feed, data source, tool, and outputs and is ready to be started. When started, or running, the analytic receives the simulated ship data, join it to the name of the U.S. Coast Guard District where each ship is currently located, and writes the event data to the feature layer and stream layer outputs.

1. In the upper right of the analytic editor, click Start.

   As with feeds, real-time analytics are long-running tasks and continue to run until they are stopped.

   The Start button text changes to Stop Initialization and then to Stop. This indicates that the real-time analytic is now running.

2. View the status and monitor the AIS Real Time Processing analytic on the Real-Time Analytics page.

View the real-time analytic metrics

Metrics about your running real-time analytics can be viewed in Velocity. Metrics provide the average rate of events through each element and represent an average of the rate for each element for the past five minutes of run time. Additionally, while a real-time analytic is running, the metrics represent the compute utilization of the analytic. For details, see Real-time analytic metrics.

1. In the AIS Real Time Processing real-time analytic, turn on the Metrics turn off/turn on button to view metrics for each node in the analytic.
2. Review the metrics for each node in the real-time analytic. Notice the events per second that each node is processing, the total number of observations in and out, and the compute utilization.

   

Examine the output layer

With the real-time analytic now running and sending event data to the feature and stream layer outputs, you can now add these output layers to a web map directly in the real-time analytic.

1. From the main menu, click Feeds under INPUT to access the Feeds page.
2. If the ship_positions_simulation feed is not already started, click Start.
3. From the main menu, click Real-Time under ANALYTICS to access the Real-Time Analytics page.
4. If the AIS Real Time Processing real-time analytic is not already started, click Start.
5. Once the AIS Real Time Processing real-time analytic is running, click the pencil button to edit the analytic.
6. Right-click the Ships_Inside_USCG_Districts feature layer output and choose Open in Map Viewer.

   

   A new browser tab opens the Map Viewer in ArcGIS Online and adds the feature layer to the web map.

7. Change the basemap to the Dark Gray Canvas.

   

8. Click Add and choose Add Layer from Web.
9. Add the U.S. Coast Guard Districts feature layer using the URL https://services2.arcgis.com/FiaPA4ga0iQKduv3/arcgis/rest/services/US_Coast_Guard_Districts_new/FeatureServer.

   

10. Click one of the ship features to open the pop-up and explore its attributes. Each observation is enriched with the name of the U.S. Coast Guard District (DistrictName) the ship is currently inside.

    

Next steps

You created a real-time analytic using Velocity and viewed the output features in a web map. Then you added the Join Features tool to perform spatial enrichment and associated contextual information with the ingested observation data.

Next, explore the Design a big data analytic lesson that introduces you to working with big data analytics in Velocity. If interested in using Arcade expressions, see the Use Arcade expressions topic.