This document explains how to use the Prompt Based Segmentation pretrained model available on ArcGIS Living Atlas of the World. This deep learning model is used to create a binary mask based on the provided prompt from high-resolution imagery.
This deep learning model is based on CLIPSeg, and is used to create a binary mask based on the provided prompt. This model works as a zero-shot segmentation model and can segment any target object without any fine-tuning. It uses CLIP's pretrained knowledge of image and prompt relationships to segment images based on a user-provided text or visual prompt. Visual prompting could be helpful, when it is hard to describe an image through text prompt due to the complex features in the image, by directly providing the image as a prompt for generating the segmentation mask.
To complete this workflow, the following are the license requirements:
- ArcGIS Desktop—ArcGIS Image Analyst extension for ArcGIS Pro
- ArcGIS Enterprise—ArcGIS Image Server
- ArcGIS Online—ArcGIS Pro or Professional Plus user type
Model details
This model has the following characteristics:
- Input—8-bit RGB oriented images, mosaic dataset, or image service.
- Output—A classified raster with the prompt as a class.
- Compute—This workflow is compute-intensive, and a GPU with minimum CUDA compute capability of 6.0 is recommended.
- Applicable geographies—The model is expected to work globally.
- Architecture—The implementation is based on the CLIPSeg.
Access and download the model
Download the Prompt Based Segmentation pretrained model from ArcGIS Living Atlas of the World. Alternatively, access the model directly from ArcGIS Pro 2.7 and later, or consume it in ArcGIS Image for ArcGIS Online.
- Browse to ArcGIS Living Atlas of the World.
- Sign in with your ArcGIS Online credentials.
- Search for Prompt Based Segmentation and open the item page from the search results. Alternatively, download the pretrained model from here.
- Click the Download button to download the model.
You can use the downloaded .dlpk file directly in ArcGIS Pro or upload it and use it in ArcGIS Enterprise. Additionally, you can fine-tune the pretrained model if necessary.
Release notes
The following are the release notes:
| Date | Description |
|---|---|
| December 2024 |
First release of Prompt Based Segmentation |