Data sources and formats

Note:

Check out the rest of the image management workflows to find more information about managing a specific type of imagery, such as high-resolution satellite imagery, imagery from frame cameras, or elevation. Each workflow will provide information about relevant types of data and how the imagery can be obtained, as well as recommendations concerning the format and suitable metadata.

Generally, it is best to leave imagery in its original form. When imagery is processed so that the pixels are sampled (for example, to change the projection), this leads to degradation in quality, possible artifacts, creation of NoData areas, and issues managing additional files. With imagery used for analysis or high-quality interpretation, it is preferable (and sometimes necessary) to ensure that the pixel values do not change.

In some cases, it is advisable to change the format of the imagery to make it faster to access. This does not involve sampling the imagery or changing the structure, but it does result in data duplication. The original data is often then archived. Including lossy or lossless compression in the format conversion is optional.

Many traditional workstation imagery applications first read the complete image into memory and then allow a user to make changes to the image before saving it. This is not a scalable approach. To enable scalability for large numbers of large images, ArcGIS reads on demand only the appropriate imagery from storage. The performance of the storage system as well as the format of the imagery can have a significant effect on the performance of the imaging system.

The following five factors have the greatest influence on the suitability of a file format:

1. Internal tiling of the rasters

   In many cases, rasters are stored as an array of pixels on disk. When the rasters are large and only a small extent of an image needs to be read, the system can skip quickly to the appropriate row in the file, but it will still need to read the complete row due to the way files are broken and read as blocks on disk.

   On larger files (more than 3,000 columns), you can use a tiled image format that breaks the image internally into smaller tiles (typically 256x256 pixels), making it faster to access a group of pixels representing a rectangular extent. Most modern file formats are either tiled or include options for tiling. The TIFF format can be tiled or non-tiled. By default, ArcGIS writes a tiled TIFF.

2. Volume of data required to be read

   For any group of pixels to be processed, those pixels must be read from storage. If the data volume to be read is reduced, this can improve performance. Imagery compression can substantially reduce the size of the data read. Typically, a natural color image can be compressed 5–10 times with negligible difference in image quality. Such compression substantially reduces the volume of data read from disk and can have a positive influence on performance, especially on systems with slower drive systems.

   Lossless compression algorithms, such as LZW, can be effectively used to compress imagery that contains a large number of NoData values, but generally provide minimal compression of optical imagery. Lossy compression algorithms can substantially reduce data volumes but add unacceptable artifacts to imagery, especially if it is to be used for analysis. Compressing imagery also increases the CPU load for reading the imagery, and some algorithms are much more CPU efficient than others, so some compression formats are better than others.

3. Amount of processing power required to decompress the image

   Compression requires that the system decompress the imagery before any processing can be applied. Some compression formats, such as JPEG 2000, are CPU-intensive to decompress. For workstation applications where the complete image is first read into memory, or in applications that just stream pixels (or wavelets) without any processing, such compressions are useful, but they are not recommended for use on servers that perform processing. Wavelet-based compression, like JPEG 2000, is also not recommended for streaming to web applications, as web browsers cannot natively read wavelet-compressed images—they require special plug-ins to reduce CPU usage.

   If imagery is suitable for lossy compression, then JPEG compression (not JPEG format), which is CPU-inexpensive to decompress due to the relative simplicity of the algorithms, is recommended. The ubiquitous use of JPEG compression also means the codecs are hardwired into many CPUs. Although JPEG compression does not provide quite as high compression as some of the wavelet compression algorithms, the difference is relatively minor. Typically, an 8-bit natural color image can be compressed about 8 times using JPEG compression, which gives similar quality to a wavelet-compressed image with 11 times compression. In relation to the 8 times compression, the additional 30% compression provided by wavelet compression algorithms is not warranted, since the CPU load to read formats such as JPEG 2000 can be 4–10 times greater.

   For lossless compressed imagery, LZW compression provides good compression with low CPU load. Other lossy compression algorithms can provide a bit more compression, but the additional CPU load generally outweighs the small increase in compression. ArcGIS includes support for Limited Error Raster Compression (LERC) that provides high compression of elevation data with control of the precision of the data returned and minimal CPU load. LERC is most valuable when working with elevation data. It is also very valuable when working with optical imagery, where it can be used as a lossless compression that provides 25% compression while being very fast to compress or decompress. LERC can be used for the transmission of data from the server to the client, and in the MRF and CRF file formats.

4. Existence of pyramids

   Pyramids are reduced-resolution datasets stored with imagery that are used to read imagery at lower resolutions. Pyramids are recommended, especially for larger files. Pyramids can often be compressed even if the base data is not compressed. More details about creating pyramids are defined in the preprocessing section below.

5. Location and type of metadata

   The way in which the metadata is stored with the file can have an influence on the access speed. When a file is opened to read the pixels, it is often necessary to read the file's metadata to obtain georeferencing information and access properties such as spatial reference. Formats such as GeoTIFF store metadata in tags that can be quickly accessed. ArcGIS writes metadata to a small .aux.xml file stored next to the file to enable faster access while providing an extensible method for storing additional metadata.

   When using mosaic datasets, the method by which metadata is stored in the source's files is less important, because the metadata is read during the 'add raster datasets' process and then embedded in the mosaic dataset for faster access when required.

Recommended imagery formats

There are three recommended file formats to use in ArcGIS:

• TIF that is internally tiled with overviews and suitable JPEG or LZW compression

  This is the most generic standard format that is most suitable for most applications.

• MRF using LERC or JPEG compression with overviews

  MRF is optimized especially for cloud storage, but also works well for enterprise storage. MRF splits a raster into three separate files. The main very small file includes key properties and references to separate files used to store the index and data tiles. This structure optimizes data access. MRF can include multiple compressions, including JPEG (8-bit and 12-bit), deflate, and LERC.

  Note:

  To learn more about converting data to MRF or TIF, see the OptimizeRasters GitHub repo.

• CRF

  CRF is a format used as part of Raster Analysis, but can also be used as a raster dataset. It is optimized for writing and reading large files. Internally, the large rasters are broken down into smaller bundles of tiles, which allows multiple processes to write simultaneously to a single raster.

If imagery is to be converted, follow these recommendations:

• 8-bit or 16-bit, 1-,3-, or 4-band imagery where lossy compression is not suitable

  Use MRF with LERC compression or TIF with LZW compression. These formats include tiling with tiles of size 512, 256 or 128. Smaller tile sizes work best for scientific data where access to temporal profiles is more common.

• 8-bit, 3-band natural color imagery already preprocessed by orthorectification, color balanced, mosaicked, and cut into tiles

  This imagery is generally used as background imagery, and should be converted directly to a tile cache or store as MRF or TIFF with JPEG YCbCr compression. Typically, a quality value of around 80 is used, which provides approximately 8-times compression. YCbCr-based JPEG compression internally converts the image to a different spectral domain, improving the compression.

• 16-bit or 32-bit 1-band elevation data

  Use MRF with LERC compression or TIFF, LZW compression, tiled 128 or 256. For 16-bit elevation, be sure that JPEG is not used.

• 8- or 16-bit imagery where lossy compression is suitable

  Use MRF or TIF with JPEG (YCbCr) compression. The quality should be checked by testing on some sample imagery. In many cases, a quality factor of 90 is suitable. Note that ArcGIS supports a 12-bit version JPEG. Therefore, when compressing 16-bit pan imagery using JPEG, only the first 12 bits of the imagery will be used. Many modern sensors have a sensitivity in the range of 11- to 14-bits, and using 12-bit compression maintains the majority of the image content but excludes the last (often noisy) bits.

• 8-bit or 16-bit, 3-band, non-natural-color imagery when lossy compression is suitable

  Examples of this kind of imagery include false color imagery or scanned maps. Use MRF or TIFF, with JPEG (RGB) compression. In RGB JPEG compression, each band is compressed separately.

• 8-bit or 16-bit 4-band RGBI

  This is the format often captured by modern digital sensors. If the data has been orthorectified and enhanced, then some of the original image information has been lost, potentially limiting its use for some forms of analysis. For such imagery, lossy compression may be suitable, but care should be taken to quantify the effects on any intended future analysis.

  It is then recommended to convert such imagery into a 3-band RGB and 1-band NIR image and use the above recommendations for compressing each. Splitting into a separate RGB image enables better compression, and most users will likely access the RGB component more than the NIR. In ArcGIS, one can easily virtually remerge the two files to create a RGBIR image suitable for displaying as false color or computing NDVI. Typically for such imagery, the compression quality is set higher, to 90 or 95, so that compression does not add significant artifacts to NDVI.

  When using JPEG compression, the recommended quality values to use can range from 80 to 95. It is best to try different factors on sample images and review the differences to determine an optimal value.

• 8-bit or 16-bit 5-band RGBI with a panchromatic band

  Many sensors include 4-band multispectral imagery and 1-band higher-resolution panchromatic imagery. If you are maintaining the IR band, it is recommended to not pre-pansharpen such imagery. The pan-image changes the multispectral properties of the bands and the pan-sharpening process will significantly increase the file sizes, reducing the suitability of the imagery for analysis. Instead, use the capability of ArcGIS to pan-sharpen on the fly, which is performed very fast and ensures that the integrity of the spectral bands is not lost. If you need to compress the imagery to reduce size, the panchromatic band should use JPEG compression, as the panchromatic band is typically much larger than the multispectral image and is not used for spectral analysis. Limited JPEG compression (for example, Q90) has minimal effect on visual interpretation or computation of tie points or DSM generation.

  When using JPEG compression, the recommended quality values to use can range from 80 to 95. It is best to try different factors on sample images and review the differences to determine an optimal value.

Working with large mosaics

Many organizations receive large, pre-processed mosaics created from collections of images that are orthorectified, color corrected, and mosaicked to cover a large area. In some cases, such data is split into rectangular tiles, while in others the data is combined into a single large file. The creation and use of a single large file can be problematic and so typically the data it tiled.

Within ArcGIS, the optimum way of working with such very large rasters is to use tile cache or CRF. Both formats are designed to handle very large datasets and internally split the virtual raster into multiple files.

If the data is not delivered as tile cache or CRF, the such data should be tiled into large TIF files where each file is about 1GB. There should also be some overlap (around 50 pixels) between the tiles. Such overlaps can reduce the potential of artifacts when the data is projected, especially at small scales.

Reformatting imagery

If imagery is not in an optimized format, reformatting the tiles is recommended if possible.

The following formats should be reformatted:

Format	Reformatting recommendations
.jpg	JPEG files larger than 3,000 columns are slower to read, because JPEG is not tiled; therefore, access to the last pixel of the file requires the complete file to be decompressed. When converting to TIFF with JPEG compression, try to use the same quality factor and type (YCBCR or RGB) as the original data.
.asc	ASCII text files, sometimes used to store elevation, are inherently slow to read as they are unnecessarily large and need to be interpreted.
.dem	Internally, some variations of the format store the numbers as ASCII.
.jp2	It is recommended to test difference of performance after converting a sample file to TIFF. There are many variants of JPEG 2000. Some can be very costly to decompress or to access pyramids. In some cases, it is advisable to leave the format as JP2, but create an additional set of pyramids(optionally skipping the first pyramid level).
.ecw	This proprietary format has limitations on use in a server environment. The format is often used for preprocessed imagery, so a more optimum format for storage and serving is a map cache. Conversion may result in the file size increasing by about 30-40 percent, but it will be in web optimal format. Since wavelet artifacts are different from JPEG artifacts, the conversion of highly compressed to ECW to JPEG often results in unnecessary additional artifacts. Where possible, it is advantageous to obtain and compress the original imagery if available.
.flt	Sometimes used for elevation data. Conversion is recommended if the number of columns is greater than 2,000.
.tif	If untiled (for example, the raw format from most data providers),it is advisable to convert to tiled TIFF. A typical example would be imagery from most satellite vendors, which generally deliver TIFF files as RAW and untiled to be compatible with legacy systems that may not be able to handle tiled TIFF. Tiling these files will improve performance and put reduced load on disk storage systems.

It is generally not necessary to reformat the following formats:

Format	Reformatting recommendations
.nitf	Generally read quickly, and extensive metadata support and complexity make other formats unsuitable. There are some formats of NITF that have forms of JPEG 2000 compression pyramids that are very slow to read, in which case consider creating additional pyramids. In some cases, to improve performance it is necessary to convert to a different compressed NITF format.
.sid (MrSID)	The MG2 and MG3 versions of the format are read relatively quickly and include pyramids. The MG4 format appears to be slower to read and it is advisable to test.

Reformatting imagery is not always necessary and may not be required at the start of a project, when time to load the imagery, additional data storage, or support by legacy systems is a concern. It is often advantageous to leave the imagery in its nonoptimized form initially, while making imagery accessibility faster, then at a later time optimize the format of the imagery. If the imagery is originally TIFF, this can be achieved without changing the mosaic datasets. If originally another format, then is it also possible to convert the format and change the name of the reference in the mosaic dataset so that other properties of the mosaic dataset need not be changed.

The best way to reformat imagery is usually to use the Copy Raster tool in batch (or ModelBuilder or Python). An alternative is to use the Compress Imagery tool available on ArcGIS Online.

Storage system performance

In an imagery environment with large volumes of imagery, it is not possible to read the complete imagery into memory,so it is necessary for the system to read the imagery as required from the disk system on demand. Therefore, the performance of the disk system is important and becomes more significant when using formats that are not tiled and not compressed, as they put more load on the disk. Naturally, a larger number of users simultaneously accessing the server puts additional load on it. When the imagery is stored on a different location to the server and connected via a network, the network can quickly become a significant bottleneck. Many issues related to poor performance of an image server implementation are related to poor storage systems. The performance of disk subsystems varies considerably, and in most cases the performance is either very good or very bad. Unlike CPU and memory performance, where the difference is generally measured in percentage, the performance of different storage systems can vary often by a factor of 10. Though, the price of the disk system is not a good measure of performance or suitability for image server tasks. The following are some general recommendations:

For smaller, dedicated systems—Direct Attached Storage (DAS) systems generally provide high performance at lowest cost, but provide limited scalability. In some cases, it can be advantageous to have a server configured specifically for imagery and have the imagery on a dedicated DAS. To scale by a factor of two and/or have 100 percent redundancy, duplicate the server and mirror the DAS. This method has limited scalability.

For larger systems—Using a NAS (or a SAN with a NAS head) is generally recommended. This enables simpler scaling in the number of servers. Often it is advisable to have a separate NAS dedicated to imagery. This enables the NAS to be connected to the servers using a dedicated switch and NICs or using a dedicated fiber channel, and also removes potential contention with other traffic on the network.

Cloud infrastructure offers blob storage such as Amazon Web Services S3 or Azure Blob storage. Such storage is low cost and very reliable, but has worse latency then direct access storage. This can be overcome by using appropriate data structures. When managing imagery using cloud infrastructure, you should do the following:

1. Convert the data to MRF format as the data is copied to the cloud. This can be done using OptimizeRasters.
2. Once the data is in cloud storage, local raster proxy files should be created that reference the rasters in the cloud.

   Note:

   Rasters proxies are small files that reference the source data in the cloud. They have the same extension as the original imagery, and are treated like any other raster file by ArcGIS, but they contain only key metadata. Raster proxies work best if the source data is an MRF file, but they can also reference other formats, such as TIF. Raster proxies can also be used to define local caches stored on fast ephemeral disks. This ensures that tiles of data read from the cloud storage are locally cached for subsequent access. This can increase the performance especially for dynamic image services.