Create a feed

This lesson will demonstrate how to create a feed in ArcGIS Analytics for IoT.

Feeds allow you rapidly ingest and visualize real-time data. Feeds typically connect to external sources of observational data such as Internet of Things (IoT) platforms, message brokers, or third-party API's. Feeds are also a type of stream layer and can be easily added to ArcGIS Online web maps and other applications.

In this quick lesson, you will be working with Marine Cadastre Automatic Identification System (AIS) data. You will create a feed to simulate historical AIS data (ship position observations) from a file.

The configuration wizard you will use for creating a feed is a common one throughout the Analytics for IoT application. It is good to familiarize yourself with the process as you will be working with the configuration wizard whenever you are configuring input or output data components.

This lesson is designed for beginners. You must have an ArcGIS Online account with access to ArcGIS Analytics for IoT. Estimated time: 30 minutes.

Explore options for creating feeds

There are several places in the ArcGIS Analytics for IoT application you can access the configuration wizard to begin creating a feed.

  1. Go to https://iot.arcgis.com and sign in with your ArcGIS Online credentials.

    For the best experience, use Google Chrome or Mozilla Firefox.

    Note:

    If you encounter issues signing in, contact your ArcGIS Online administrator. You may need to be assigned to an ArcGIS Online role with privileges to use ArcGIS Analytics for IoT.

  2. Once you're signed in, you can create a new feed by clicking Create Feed from the Home page.

    application home page

  3. In addition, by clicking Feeds on the menu on the left of the site, you access the Feeds page where you can view existing feeds and their status as well as create new feeds.

    Feeds page in the

    In the next section, you will create and configure a new feed.

Create a new feed

To begin, you will create a new Simulator feed type using the feed configuration wizard.

  1. From either the Home page or Feeds page you explored above, click Create Feed.
  2. In the Select a feed type window, you have access to groupings of the various feed types you can create in Analytics for IoT.

    Feed configuration wizard in the

  3. Click See More under Web & Messaging to access the feed types for web and messaging feeds. Similarly, you can search for simulator in the search box.

    Web and messaging feed types

  4. Click HTTP to access the HTTP related feed types.

    Web and messaging HTTP feed types

  5. Click Simulator to choose this feed type.

Configure the simulator feed

The first step in configuring a new feed is to define the connection properties to the data source. This can be, for example, defining a connection to a server with authentication credentials or simply browsing to a feature layer in ArcGIS Online.

For Simulator feeds, you will connect to a simulation file available at a URL.

  1. In the Simulator Options step, enter the connection properties as follows:
    1. For URL, enter the following URL:

      https://a4iot-public.s3-us-west-2.amazonaws.com/simulations/AIS_MarineCadastre_SelectedVessels.csv

    2. For Field Separator, accept the default comma (,), since this file you are connecting to is comma-delimited.
    3. For Features Per Execution, enter: 30

      This simulation file contains a set of observations for 30 ships, ordered by timestamp. By setting this property to 30, the data will be extracted 30 rows at a time, meaning there will be a new observation for every unique ship in the dataset.

    4. For Interval for Sending Events (milliseconds), enter: 1000

      This property controls the rate at which the specified number of rows is simulated. In this case, you will simulate 30 observations (one for each ship) every 1000 milliseconds (or 1 second).

    5. For Repeat Simulation, leave the checkbox checked.

      When checked, the simulator feed will loop the data when the end of the simulation file is reached.

    6. For Time Field Index, enter: 1

      This property sets the numerical index of the date field in the dataset, where the index starts at 0. For this dataset, the timestamp is in the second field, which means the index is 1.

    7. For Convert to Current Time, leave the checkbox checked.

      When checked, the time values in the data are converted to the current time as it is simulated. This allows applications to treat the data as current real-time data.

      Feed connection properties for the Simulator feed type

  2. Click Next to proceed to the next step.

    Analytics for IoT will test the connection to the data source and attempt to sample the data. If the connection is successful, it will then attempt to derive the schema of the data, which you will explore next.

Confirm the data schema

When configuring a feed, it is important to define the schema of the data you are receiving. Analytics for IoT makes a best attempt at defining that schema when it samples the data, estimating the data format, field delimiter, field type, and field name properties.

  1. In the Confirm Schema step, confirm the properties are configured as follows:

    Confirm schema in the feed configuration wizard

  2. Explore the Field Types returned from the sampled data records.

    For delimited data, Analytics for IoT makes an estimation of the field type based on the sampled records. It is important to review the field types and confirm they are correct. For example, if a field is derived to be a number, but you know the values in the field represent coded values, it is a best practice to change the field type to a string. In this lesson, you will accept the default field types that were returned.

  3. Explore the Field Name values returned.

    Analytics for IoT has derived field names when it sampled the file. Field names will be populated for delimited data when a header row appears to be present. You can rename fields now as well as uncheck any unnecessary fields.

  4. Click Next to confirm the schema of the data.

Identify the key fields

Next, you will configure some important fields so Analytics for IoT can properly construct geometry, date information, and a unique identifier for each ship.

  1. Define the Location properties as follows:
    1. For Location Type, choose X/Y Fields.
    2. For X (Longitude, choose the LON field.
    3. For Y (Latitude), choose the LAT field.
    4. For Spatial reference (WKID), enter: 4326
    5. Choose the GCS WGS 1984 spatial reference from the returned search results.

    Key fields location properties in the feed configuration wizard

  2. Define the Date & Time properties as follows:
    1. For Does your data have date fields?, choose Yes.
    2. For Date Format, choose Other (String).
    3. For Date Formatting String, enter the following text:

      yyyy-MM-dd HH:mm:ss.S

    4. For Start Time (optional), choose BaseDateTime.
    5. For End Time (optional), leave it set to None.

      This property defines the end time for data where time is represented as intervals. For this lesson, the ship observations only have a single timestamp.

      Key fields date and time properties in the feed configuration wizard

    The date and time properties allow Analytics for IoT to properly construct dates from the string values provided in the BaseDateTime field.

  3. Under Tracking, choose MMSI for the Track ID.

    Key fields tracking properties in the feed configuration wizard

    A Track ID is a unique identifier for an observation or feature in the data. It allows you to identify multiple records or events in the feed as being associated with a particular feature. For details, see Track ID. In the AIS data feed, the unique identifier is the Maritime Mobile Service Identity, or MMSI, which is unique to each ship.

  4. Click Next to confirm the key fields.

Save the feed

With the properties of the feed configured, you will now assign a name, an optional summary, and save the new feed.

  1. For Feed Name, enter the following text:

    ship_positions_simulation

    Note:

    Feed names must be unique across an organization.

  2. For Feed Summary, which is optional, enter the following text:

    Simulated AIS ship positions from the Marine Cadastre Automatic Identification System

  3. Click Complete to save the new feed.

    Save feed in the feed configuration wizard

  4. If this is the first item you are creating in Analytics for IoT, when you click Complete, a login screen will appear to verify your credentials. Re-enter your ArcGIS credentials and click Sign In. This verification is required and enables Analytics for IoT to maintain long-running tasks on your behalf. You should only have to login a second time once.

Review feed details

Once the feed is published, the feed Details page loads. Here you can review the details of the feed you just created as well as edit, clone, or delete the feed.

Feed details page in the

The feed should start automatically and the status button will change from Initializing to Stop, indicating the feed has started. If the feed does not start automatically, click Start to start it. Once the Start button changes from Initializing to Stop, the feed is running.

To view the new feed item in the ArcGIS organization, click View item details. Here you can add any additional metadata or information that might be necessary.

You can also immediately view your data in a web map by clicking Open in map viewer.

Ship positions in a web map

Next steps

Congratulations! You have successfully configured a feed in ArcGIS Analytics for IoT and are now ingesting and visualizing real-time data in a web map.

Next, explore the options for performing real-time analysis on this streaming data by creating a real-time analytic. For details, see the Design a real-time analytic quick lesson.