Create and publish a mosaic dataset in ArcGIS Image Dedicated

Note:

This tutorial requires an ArcGIS Image Dedicated account with a Dynamic Imagery subscription. ArcGIS Pro 2.9+ is also required.

If you are not an Image Dedicated administrator, you will need to check with your organization's administrator to make sure you have access to the server access item.

The Dynamic Imagery subscription offered through ArcGIS Image Dedicated lets you publish mosaic datasets that reference rasters stored in cloud storage as dynamic image services. With dynamic image services, your users can access large collections of your imagery with on-the-fly processing and dynamic mosaicking. The collection of imagery is then accessible in a potentially wide range of products without the need to pre-process or store it. Dynamic image services can be published from a wide range of public datasets available through Microsoft Azure and Amazon Web Services (AWS), or from imagery stored in your own cloud storage within any AWS or Azure region. The Dynamic Imagery subscription also allows the serving of individual rasters as dynamic image services or as tiled imagery, although this is typically only used for single large datasets.

In this tutorial, you will publish a mosaic dataset as a dynamic image service using publicly accessible National Agricultural Imagery Program (NAIP) imagery from a public AWS S3 bucket provided by Image Dedicated. NAIP imagery is acquired by the US Department of Agriculture's Farm Service Agency during the growing season of various crops in the United States and is made available to the public, typically about a year after collection.

In this tutorial, you will first set up a server in the same region as the imagery. Next, you will create an empty mosaic dataset and add rasters to the mosaic dataset. Finally, you will publish your dataset as a dynamic image service.

Create a Dynamic Imagery Server

ArcGIS Image Dedicated uses imagery and raster data stored in cloud storage—from AWS S3 buckets or Microsoft Azure containers—as the data source to publish image services using servers. In this tutorial, you will be publishing imagery as image services using the dynamic imagery server so that your imagery is available as a service URL, or REST endpoint.

1. Sign into ArcGIS Image Dedicated with your ArcGIS Online credentials and click Dynamic Imagery from the site homepage. You will be redirected to a new page where the details of any existing servers will be listed. If the list is empty, no servers have been created within your organization.
2. Select + New Server to create a new dynamic imagery server within your organization. A dialog box appears with blank fields and drop-down menus.
   1. Order ID—Select your order ID or subscription from the drop-down menu. If you don't have an order ID listed, contact Esri.
   2. Cloud Provider—Select "aws" because this data is from a public AWS bucket. ArcGIS Image Dedicated also supports Azure containers, but for the purposes of this tutorial, you will be using data in AWS storage.
   3. Designated Cloud Region—Select "us-west-2." This is the region the tutorial data is located. To reduce egress costs, it is important to select the same region where the data is located in the cloud and where the server will be deployed.
   4. Data Storage Account Alias—For the purposes of the tutorial, it is not necessary to provide your data storage account alias. However, if you have a data storage account, you can provide one here. The data bucket, log bucket, and output bucket will auto populate based on the data storage account alias.
   5. DNS Prefix Preferences—Provide a unique prefix name. This prefix will appear as part of the dynamic imagery server and service URL associated with the image service you will eventually create. This will also auto populate the Server Alias. The service URL provides a way to access your image service on a dynamic imagery server or tiled imagery server.

      For example, if your prefix is "xyz," the service URL would be https://di-xyx.img.arcgis.com; you would use this URL in your browser to access the server. The alias will help you differentiate between servers on the Dynamic Imagery Active Servers tab within ArcGIS Image Dedicated. You can enter a maximum of three prefixes, in order of preference. The available prefix will be what's used as a DNS.

   6. Additional Comments—This field is typically used to provide additional information about the server, such as the server's purpose, which is for the tutorial in this case. Although optional, you are welcome to add additional context to the server you are creating as part of this tutorial.
   7. Create a Server Access Item on my ArcGIS Online Account—Check this box. Once the server is created, an access item will be created in your ArcGIS Online account that can be used to share with other members of the organization that would like to manage or publish image services to the server you are currently creating.

   

3. Press Submit Request. Once the request is submitted, the administrator will receive an email that will have details about the dynamic imagery server. It usually takes 24 hours to update the server URL. Once updated, you can use the dynamic imagery server to create services. The server will also be listed under Active Servers on the Dynamic Imagery Server page when it's ready. You will use this server to publish the imagery as an image service in a later step.

Note:

For more information on data storage accounts, you can review the Image Dedicated User Manual or visit the Downloads tile on the ArcGIS Image Dedicated website to download the instructions provided in the Create Storage Account document.

Create a Mosaic Dataset in ArcGIS Pro

Next, you will create an empty mosaic dataset container that will eventually contain the NAIP imagery. A mosaic dataset is a data model that enables you to manage and query small to vast collections of raster and image data along with properties and raster function to be applied and view them as a mosaicked image.

1. Start ArcGIS Pro. If necessary, sign in with your ArcGIS Online credentials.
2. Select Map under the Blank Templates list to create a new project with the Map template. In the Name box, type ImageDedicatedTutorial and save in your preferred file location. Check Create a new folder for this project and click Ok.
3. The project opens with the World Topographic basemap already in the display.
4. If the catalog pane is not already displayed, click the View tab on the ribbon above the map view. From there, click the Catalog Pane. The Catalog pane will now display.
   1. Within ArcGIS Pro, locate Databases from within the Catalog pane. Right-click on the Databases folder and select New File Geodatabase.

      

   2. Name your geodatabase in the File Name field in your designated folder. Press Save. Now, you have created a .gdb file, or geodatabase file, for the tutorial. This geodatabase will now display under the Databases folder within the Catalog.
5. Expand the Databases folder in the Catalog pane. Right-click on the new file geodatabase from the Databases folder and select New > Mosaic Dataset. This will open a new geoprocessing parameters page, where you will create a new mosaic dataset.
   1. Browse to your newly created geodatabase file under Output and select it.
   2. Name your mosaic dataset under Mosaic Dataset Name.
   3. You will then need to select an appropriate coordinate system. For the purposes of the tutorial, select WGS_1984_Web_Mercator_Auxiliary_Sphere as the Coordinate System. This coordinate system is used in ArcGIS Online.
   4. Leave Product Definition as None, because the tutorial data is just a dataset and is not from a specific sensor, therefore it is not included in this list.

   5. Press Run. You have now created an empty mosaic dataset container. To view, expand the Databases folder in the Catalog pane. Now, expand the geodatabase you just created. This is where the mosaic dataset displays.

Add Rasters Using ACS File

Now that you've created an empty mosaic dataset, you will need to add the tutorial raster data to your mosaic dataset. Currently, the raster data is located in a cloud-based AWS bucket. To access the cloud data in ArcGIS, you will connect to the cloud storage with an ArcGIS Cloud Storage Connection File, or .acs file.

1. From the Analysis tab in the ArcGIS Pro ribbon, select Tools under Geoprocessing.
2. A Geoprocessing pane appears. Navigate to the Toolboxes tab. In the search bar, type Create Cloud Storage Connection File. Select the Create Cloud Storage Connection File geoprocessing tool from the results.

   

   1. Under the Parameters tab for the Create Cloud Storage Connection File geoprocessing tool, select the icon under Connection File Location and browse to your current ArcGIS Pro project. Select the root folder of the project. This is where your file will be created.
   2. Give your ACS file a name under Connection File Name and select "Amazon" under Service Provider.
   3. Leave Access Key ID and Secret Access Key blank because the data is from a public S3 bucket.
   4. Under Bucket (Container) Name, type imagerytutorialdata because this is the name of the public S3 bucket.
   5. Leave Folder blank.
   6. Under Region (Environment), select US West (Oregon) because that is the region the S3 bucket is located.
   7. The Service End Point will populate with s3-us-west-2.amazonaws.com.
   8. You now need to clarify that a secret access key and key ID are not required, since this is a public AWS bucket that doesn't require a secret access key. Under Provider Options, select AWS_NO_SIGN_REQUEST from the Name drop-down list under Provider Options and type True as the Value.

3. Press Run. Your ACS connection file is now in the project files within File Explorer. The .acs file allows you connect to cloud stores directly, browse buckets/folders, and access imagery in your project.
4. From the Insert tab in the ArcGIS Pro ribbon, select Connections > Cloud Store > Add Cloud Storage Connection.
5. Select imagerytutorialdata.acs from within your current ArcGIS Pro project's folder and press Ok. Your .acs file that contains the NAIP folder you created along with the individual imagery files will now list under Cloud Stores in the Catalog pane.
6. From the Analysis tab in ArcGIS Pro, select Tools under Geoprocessing. The Geoprocessing pane will appear.
7. Navigate to the Toolboxes tab and select Data Management > Raster > Mosaic Dataset > Add Rasters to Mosaic Dataset.
   1. Under the Parameters tab for the Add Rasters to Mosaic Dataset geoprocessing tool, select the icon under Mosaic Dataset and select the mosaic dataset that you created from your current project.
   2. Under Raster Type, select Raster Dataset.
   3. Since you aren't performing any processing or analysis with the data as part of this tutorial, keep Processing Templates as Default.
   4. Select Folder as the Input Data Type because we will be adding multiple raster datasets from the ACS connection file. By selecting Folder, all rasters that exist in that folder will be added.
   5. Browse to the .acs file imagerytutorialdata.acs\naip\sourcetif and select it as input. This is the folder that contains the rasters as tif files. Leave the remaining fields as default.

      As part of this tutorial, you will also add mosaic dataset overviews as rasters to the mosaic dataset. Browse to the .acs file imagerytutorialdata.acs\naip\overviews and select as input. You will see both folders listed as input.

      

   6. Press Run. The rasters are now added to your mosaic dataset. In the Contents pane, you right click on Footprint and select Zoom to Layer. You will see the footprints of the rasters. Once zoomed into the rasters, the imagery will display on the map.

Note:

Similar to image pyramids, mosaic dataset overviews are a lower-resolution version of images that are created to help users envision the mosaicked imagery when zoomed out.

For the purposes of the tutorial, the overviews were already pre-generated and made available in the same bucket where the source imagery exists. Using the Copy Raster tool, overviews were generated in Cloud Raster Format (CRF) with the mosaic dataset as input. The .crf file is optimized for writing and reading large files in a distributed processing and storage environment. CRF data is designed for higher performance in cloud storage.

8. Next, you will enable the overviews that you added as rasters. To do this, right click on the Footprint in the Content pane and select the Attribute Table. A table appears with all of the rasters within the footprint, including the overviews. For the row with the Name "overviews," double click on the Category, which currently displays as "Primary" for the item.
9. A drop-down menu appears. From this drop-down menu, select "Overview."
10. For the MinPS value, type 1. This field reduces the overview to the minimum visibility of the raster, which is usually the LowPS.
11. Save your project.

Publish your Mosaic Dataset

Next, you will publish your mosaic dataset as a dynamic image service.

1. From the ArcGIS Image Dedicated website, navigate to the Downloads tile. From there, select the ArcGIS Image Dedicated Service Management GP Toolbox to download a zipped folder that contains the toolbox.
2. Save the zipped ArcGIS_Image_Dedicated_Service_Management folder in the Image_Mgmt_Workflows folder within the C: drive. If you do not have the Image_Mgmt_Workflows folder, create it as a new folder in the C: drive. It is important that this folder exists on your C: drive since most workflows associated with ArcGIS Image Dedicated default to this folder.
3. Unzip the ArcGIS_Image_Dedicated_Service_Management folder.
4. In the Catalog pane of ArcGIS Pro, right-click on Toolboxes and select Add Toolbox.
5. From your C: drive, navigate to the unzipped ArcGIS_Image_Dedicated_Service_Management folder and open its contents. Select the python toolbox within this folder and press Ok.
6. The toolbox will display under Toolboxes in the Catalog pane.
7. In order to publish your mosaic dataset as a dynamic image service, you will need to create a .zmd extension file, or image definition file, that aggregates and zips all relevant information needed to create a dynamic imagery service. To do this, under the ArcGIS Image Dedicated Service Management GP Toolbox, select Create Image Definition.
   1. Under the Parameters tab, select the icon under Input Mosaic Dataset/Raster Dataset/Tile Cache/Scene Layer. Browse to the mosaic dataset that was created as part of this tutorial and select it. Press Ok.
   2. Select a local storage location for the Image Definition.
   3. Check Copy Cloud Storage Connection File.
   4. Press Run.

Note:

You need to have MDTools already installed in order to create an image definition. You can download and install MDTools from the mdcs-py repo in GitHub.
8. Now that you've created your .zmd file, or image definition, you can publish your dynamic imagery service. To do this, select the Manage Dynamic Services geoprocessing tool under the ArcGIS Image Dedicated Service Management toolbox.
   1. Under the Parameters tab, select AWS for the Cloud Type.
   2. The Dynamic Imagery Server field will populate with the dynamic imagery servers that have been created in ArcGIS Image Dedicated. Select the server that was created in the same region as the tutorial data, which is us-west-2.
   3. Since this is the first time you are creating an image service for the server, select Create Service under Action.
   4. The Server Folder Name will auto-populate as the root folder; if you would like to save the service in a subfolder, provide the name here.
   5. Under Image Service Name, type a name for your image service. You can use the name "NAIP," because it is the dataset type for your dynamic imagery, or a unique name that you come up with.
   6. Select the icon under Image Definition and browse to the .zmd file, or image definition file that you've created.
   7. Select Dedicated Instance under Instance Type.
   8. Under Description, add text about the imagery data to provide context. The description will appear on the REST URL link associated with the image service after it is published.
   9. Under Copyright, type Esri, USDA Farm Service Agency.
   10. Check Create Portal Item. This will add the image service as an item in ArcGIS Online. Typically, this option is used to ensure users can access the imagery as an item and view and perform analysis in ArcGIS Online.
   11. Leave Enable WMS and Enable WCS unchecked. These are open-source formats that allow you to make your maps available in an open and recognized way across different platforms and clients. For the purposes of this tutorial, we do not need to enable the image service as a web coverage service or web map service.
   12. Check Set Service Permissions. For the purposes of the tutorial, add public for the User and select Disabled from the Permission detail. Now only the user can access the secured service. The displayed user names are associated with the Image Dedicated dynamic imagery subscription. If you do not set service permissions, the server will default to being publicly accessible.
9. Press Run. A Job Status ID window appears displaying the Job ID and reads as Successful. Copy the service URL link. Browse to the Map tab in ArcGIS Pro and select Add Data under Layer.

   Note:

   If your job status does not read as Successful, copy the Job ID and select Get Job Status in the ArcGIS Image Dedicated Service Management toolbox. Under the Parameters tab, you will provide the Cloud Type, Server Name, and the Job ID that you copied from the Job Status window.

10. A pop-up window will appear. Paste the service URL under Path and press Add. You are now displaying the data from the dynamic image service. Alternatively, you can use the URL and access it in the ArcGIS Online Map Viewer.

Now that you have published a mosaic dataset as a dynamic image service, you can use the image service URL to perform analyses. You can also add the image service as an item in ArcGIS Online or ArcGIS Enterprise portals. Learn more about the processing and analysis tools in ArcGIS Pro. For more information on the topics detailed in this tutorial, see the following resources:

• Key concepts for image services
• Create a mosaic dataset
• Publish an image service
• Connect to a cloud store
• Create a cloud storage connection file
• Create overviews of mosaic datasets
  • Define Overview geoprocessing tool
  • Build Overviews geoprocessing tool