Schedule recurring big data analysis

In ArcGIS Velocity, big data analytics run when you start the analytic. Big data analytics can also be scheduled to run periodically or at a recurring time. Scheduling big data analytics can be done when editing an analytic using the Schedule options.

Available configurations

The following configurations can be defined in Velocity big data analytics:

  • Runs once:
    • Analytic runs when you click Start.
  • Run periodically:
    • Analytic runs every X minutes/hours. For example: every 5 minutes.
    • Analytic runs every X minutes/hours within a time frame. For example: every 15 minutes between 8 am and 5 pm.
    • Analytic runs every X minutes/hours on Y day(s). For example: every 2 hours on weekdays.
    • Analytic runs every X minutes/hours on Y day(s) within a time frame. For example: every 20 minutes on weekends from 6 pm to 11 pm.
  • Runs at a recurring time:
    • Analytic runs on X day(s) around Y time. For example: Mondays around 8 am.

Scheduling recurring big data analytics

To schedule recurring big data analytics that run periodically or at a recurring time, edit the big data analytic and click Schedule in the upper right. A scheduling pane will open that can be used to define a specific schedule for the big data analytic.

Important considerations

When scheduling recurring big data analytics, consider the following:

  • When scheduling big data analytics to run at a recurring time, the day and time is in UTC time.
  • The big data analytic should be able to finish running before the next run starts. For example, if your analytic takes roughly three minutes to complete, it is not recommended to configure the analytic to run every minute as multiple runs would be skipped and you will receive validation warnings.
  • When updating and saving a big data analytic that is already started, the next scheduled run of that analytic will use the latest saved configuration.

Why use recurring big data analytics?

Processing in near real-time

One of the most commons use cases for recurring big data analysis is to perform processing in near real-time. This is when a big data analytic is configured to run every X minutes/hours to process only the most recent features stored by a real-time analytic in that period of time.

For more information on performing near-real-time analysis, see Perform near-real-time analysis.

Perform analysis to generate analytic product(s) for certain time ranges

Often, organizations want perform repeated analysis on data collected within a certain time range. For example, consider an environmental organization with sensor data that requires analytic products and outputs to be generated and archived each week, representing the past week of data collection. The analysis being performed remains the same, however the data values being processed change week by week. To accomplish this, a big data analytic can be configured to run once a week, for example, every Monday morning at 12 am.

For more information on how to perform and when to use a recurring analytic to generate informational products, see Generate up-to-date informational products.