In a real-time, IoT system, spikes in the quantity, complexity, or velocity of data occur. Additionally, analytics can be configured that process millions or even billions of records with processing pipelines of varying complexity.
To address this, feeds, real-time analytics, and big data analytics in ArcGIS Velocity can dynamically allocate additional resources to maintain collection and real-time analysis velocity as well as achieve fast processing for big data analytics. Autoscaling scales resources up or down according to load.
Feed autoscaling
Feed autoscaling is performed incrementally following startup as incoming velocity is monitored. When starting a higher velocity feed, it may take 5–7 minutes until the desired velocity is sustained by the feed. You can determine whether or not a feed was scaled by inspecting the feed logs, which report a relevant message each time a feed is scaled.
Feeds autoscale differently based on the ArcGIS Velocity subscription license for your organization. This is because Standard and Advanced license levels have maximum supported velocities, and subscriptions are limited by the number of running items.
- Standard and Advanced
- Feeds will scale up once if needed to support data velocities that are close to the maximum allowed by the subscription.
- Feeds will continue to scale up as needed to support the detected velocity. Feeds can only scale to levels supported within the available compute of the subscription.
- If the subscription compute reaches full utilization, additional compute units can be purchased to provide additional capacity.
Real-time analytic autoscaling
Real-time analytic autoscaling is performed incrementally following startup as velocity is monitored to ensure incoming records from the feed are all processed with minimal latency. When starting a higher velocity or complex real-time analytic, it may take 5–7 minutes until the velocity is sustained by the analytic. You can determine whether or not a real-time analytic was scaled by inspecting the analytic logs, which will report a relevant message each time the analytic is scaled.
Real-time analytics autoscale differently based on the ArcGIS Velocity subscription license for your organization. This is because Standard and Advanced license types have maximum supported velocities and subscriptions are limited by the number of running items.
- Standard and Advanced
- Real-time analytics will scale up once if needed to support data velocities that are close to the maximum allowed by the subscription.
- Dedicated
- Real-time analytics will continue to scale up as needed to support the detected velocity and the latency introduced by the complexity of the analytic process. Real-time analytics can only scale to levels supported within the available compute of the subscription.
- If the subscription compute reaches full utilization, additional compute units can be purchased to provide additional capacity. For details on compute units in the Dedicated license, see Dedicated license.
Big data analytic autoscaling
Big data analytics apply certain kinds of autoscaling in all subscription license levels to accommodate large datasets or complex processing pipelines.
With the Dedicated license, additional autoscaling is present for scheduled big data analytics:
- If an insufficient run setting resource allocation plan was specified, additional resources will be allocated to complete the analysis for scheduled analytics.
- In a future release, additional autoscaling will be supported for scheduled big data analytics where big data analytics can autoscale between runs. For example, if an analytic run time begins to approach the time between scheduled runs, the analytic is autoscaled to run faster so that runs are not skipped.