Welcome to the Harris Geospatial product documentation center. Here you will find reference guides, help documents, and product libraries.


  >  Docs Center  >  Using ENVI  >  Minimum Noise Fraction Transform

ENVI

Minimum Noise Fraction Transform

Minimum Noise Fraction Transform

Use MNF Rotation transforms to determine the inherent dimensionality of image data, to segregate noise in the data, and to reduce the computational requirements for subsequent processing (Boardman and Kruse, 1994).

You can write scripts to perform an MNF transform using the ENVIForwardMNFTransformTask and ENVIInverseMNFTransformTask routines.

See the following sections:

Background


The MNF transform as modified from Green et al. (1988) and implemented in ENVI, is a linear transformation that consists of the following separate principal components analysis rotations:

  • The first rotation uses the principal components of the noise covariance matrix to decorrelate and rescale the noise in the data (a process known as noise whitening), resulting in transformed data in which the noise has unit variance and no band-to-band correlations.
  • The second rotation uses the principal components derived from the original image data after they have been noise-whitened by the first rotation and rescaled by the noise standard deviation. Since further spectral processing will occur, the inherent dimensionality of the data is determined by examining the final eigenvalues and the associated images. You can divide the data space into two parts: one part associated with large eigenvalues and coherent eigenimages, and a complementary part with near-unity eigenvalues and noise-dominated images. Using only the coherent portions separates the noise from the data, thus improving spectral processing results.

You can also use the MNF transform to remove noise from data by performing a forward transform, determining which bands contain the coherent images (by examining the images and eigenvalues), and running an inverse transform using a spectral subset to include only the good bands, or smoothing the noisy bands before the inverse.

You have the option to subset the output of the MNF rotation based on eigenvalues, which means (with hyperspectral data) that you do not have to generate an output cube of hundreds of bands (floating-point) when you only need tens of bands.

ENVI also has the option of applying an MNF transform to individual spectra. This transforms endmember spectra into MNF space for input into mixture tuned matched filtering (MTMF).

Note: You can use the Spectral Hourglass Wizard to guide you step-by-step through the ENVI hourglass processing flow, including MNF transforms, to find and map image spectral endmembers from hyperspectral or multispectral data.

References

Green, A. A., Berman, M., Switzer, P., and Craig, M. D., 1988, A transformation for ordering multispectral data in terms of image quality with implications for noise removal: IEEE Transactions on Geoscience and Remote Sensing, v. 26, no. 1, p. 65-74.

Boardman, J. W., and Kruse, F. A., 1994, Automated spectral analysis: a geological example using AVIRIS data, north Grapevine Mountains, Nevada: in Proceedings, ERIM Tenth Thematic Conference on Geologic Remote Sensing, Environmental Research Institute of Michigan, Ann Arbor, MI, pp. I-407 - I-418.

Statistics Files


Unlike principal components analysis, the forward MNF transform produces two separate statistics files: MNF noise statistics and MNF statistics. While these files appear to be ordinary ENVI statistics files, they contain information unique to MNF and omit data typically found in ENVI statistics files.

During a forward MNF rotation, ENVI computes the following statistics:

  • The mean for each band of the input image (to normalize the data)
  • The covariance statistics of the noise (for the noise rotation and normalization)
  • The covariance statistics of the noise-whitened and rescaled input image data

The first rotation stores its complete set of covariance statistics in the MNF noise statistics file. However, this file contains only the noise covariance statistics, not the data normally found in ENVI statistics files.

The second rotation stores its eigenvector matrix and eigenvalues in the MNF statistics file. However, the rest of the covariance statistics for the second rotation are not saved because the covariance placeholder in the MNF statistics file is used to store a special “composite” MNF transformation matrix. This matrix describes the net result of both principal components rotations as well as the band-independent scaling introduced by the noise normalization. This non-orthogonal, non-unit length, matrix allows an inverse MNF rotation to be applied in a single step.

The following table summarizes the contents of the statistics files generated by the MNF transform:

Statistic

MNF Noise Statistics File

MNF Statistics File

Mean

<null>

Of the original input bands

Maximum

<null>

<null>

Minimum

<null>

<null>

Standard Deviation

<null>

<null>

Eigenvalues

Of the noise covariance matrix

Of the second principal components rotation

Covariance matrix

Of the noise

Composite transformation matrix

Correlation matrix

Of the noise

Incorrect values. Do not use.

Eigenvectors

Of the noise covariance matrix

Of the second principal components rotation

Forward MNF Transforms


Use forward MNF transforms to estimate the noise statistics used in the first rotation (including estimating the noise from the input data), using noise statistics that were calculated previously, or calculating the noise statistics from a dark image associated with the dataset.

Estimate Noise Statistics from the Data

Use Forward MNF Estimate Noise Statistics to estimate noise when you do not have a dark current image for your data, which is usually the case. ENVI assumes that each pixel contains both signal and noise, and that adjacent pixels contain the same signal but different noise. A shift difference is performed on the data by differencing adjacent pixels to the right and above each pixel and averaging the results to obtain the “noise” value to assign to the pixel being processed. The best noise estimate is gathered using the shift-difference statistics from a homogeneous area rather than from the whole image. ENVI allows you to select the subset for statistics extraction.

  1. From the Toolbox, select Transform > MNF Rotation > Forward MNF Estimate Noise Statistics.
  2. The MNF Input File dialog appears.

  3. Select an input file and perform optional spatial and spectral subsetting, and/or masking, then click OK. The Forward MNF Transform Parameters dialog appears.
  4. Click Shift Diff Subset to select a spatial subset or an area under an ROI/EVF/and so forth on which to calculate the statistics. You can then apply the calculated results to the entire file (or to the file subset if you selected one when you selected the input file).
  5. When estimating noise from image data using the shift differencing approach, you should strongly consider choosing a spatial subset that is spectrally uniform. However, if any band in your dataset has zero variance, you may encounter a “singularity” problem where the covariance matrix cannot be inverted, and you may receive a “Too May Iterations in TQLI” error message.

    For example, if you use a deep, placid lake for your noise subset, most bands will have very little variation in the subset, which is what you want. However, if one band has the same pixel value throughout the entire subset, the band has zero variance and will cause a singularity in the covariance matrix.

    You can find bands that contain zero variance, large outliers, or statistics that are identical to adjacent bands, by generating and reviewing a statistics report on the same shift-difference subset of your input image that you plan to choose for an MNF transform. If you determine that any band has zero variance, you should choose a slightly different or larger subset for your noise estimate. The shift-difference spatial subset must contain more pixels than you have bands in the image you are transforming.

  6. In the Enter Output Noise Stats Filename [.sta] field, enter a filename for the noise statistics.
  7. To select a homogeneous area for calculating the noise statistics, click Spatial Subset to either manually enter a spatial subset or to graphically indicate the area for statistics extraction.

  8. In the Output MNF Stats Filename [.sta] field, enter an output file for the MNF statistics. Be sure that the MNF and noise statistics files have different names.
  9. Select output to File or Memory.
  10. If you selected a mask in Step 2, the Zero Out Masked Values toggle button appears. Masked pixels are excluded when calculating noise statistics. The statistics results are then applied to all pixels in the image when applying the forward MNF transform. The default value for this toggle button is Yes, which means that masked pixels will be assigned values of 0 in the MNF output. Setting this option to No means that the masked pixels will run through the same forward MNF transform as non-masked pixels.
  11. Select the number of output MNF bands by using one of the following options:
    • To select the number of output bands without examining the eigenvalues, select No from the Select Subset from Eigenvalues toggle button, then set the Number of Output MNF Bands.
    • To select the number of output MNF bands by examining the eigenvalues, use the following steps:
      1. Select Yes from the Select Subset from Eigenvalues toggle button.
      2. Click OK. ENVI calculates the statistics and the Select Output MNF Bands dialog appears, with each band listed with its corresponding eigenvalue. Also listed is the cumulative percentage of data variance contained in each MNF band for all bands.
      3. Set the Number of Output MNF Bands. For the best results, and to save disk space, output only those bands with high eigenvalues. Images with eigenvalues close to 1 are mostly noise.
  12. Click OK. When ENVI finishes processing, the MNF Eigenvalues plot window appears and the MNF bands are added to the Layer Manager.

Interpret the MNF Eigenvalues Plot

When ENVI finishes processing, it adds the MNF bands to the Layer Manager and displays the MNF Eigenvalues plot window. The output only contains the number of bands you selected for output. For example, if your input data contains 224 bands, but you selected only 50 bands for output, your output only contains the first 50 calculated MNF bands from you input file.

Bands with large eigenvalues (greater than 1) contain data, and bands with eigenvalues near 1 contain noise. Display the MNF bands from the Layer Manager and compare with the MNF Eigenvalue plot to determine which bands contain data and which bands contain predominantly noise. In subsequent processing of this data, spectrally subset the MNF bands to only include those bands where the images appear spatially coherent and the eigenvalues are above the break in slope of the MNF Eigenvalue plot.

Estimate from Previous Noise Statistics

Use Forward MNF Previous Noise Statistics to use noise statistics calculated during a previous MNF transformation. This allows you to use noise statistics from one image with a second image.

  1. From the Toolbox, select Transform > MNF Rotation > Forward MNF Previous Noise Statistics.
  2. The Input File dialog appears.

  3. Select an input file and perform optional spatial subsetting, and/or masking, then click OK. The Enter Noise Statistics Filename dialog appears with all of the existing statistics files in the current input data directory listed. The statistics files appear with the default file extension .sta.
  4. Select a noise statistics file from a previous MNF transform processing session and click Open. The Forward MNF Transform Parameters dialog appears.
  5. In the Enter Output MNF Stats Filename [.sta] field, enter an output filename for the MNF statistics files.
  6. Select output to File or Memory.
  7. Select the number of output MNF bands by using one of the following options:
    • To select the number of output bands without examining the eigenvalues, select No from the Select Subset from Eigenvalues toggle button, then set the Number of Output MNF Bands.
    • To select the number of output MNF bands by examining the eigenvalues, use the following steps:
      1. Select Yes from the Select Subset from Eigenvalues toggle button.
      2. Click OK. ENVI calculates the noise statistics and the first rotation. The Select Output MNF Bands dialog appears, with each band listed with its corresponding eigenvalue. Also listed is the cumulative percentage of data variance contained in each MNF band for all bands.
      3. Set the Number of Output MNF Bands. For the best results, and to save disk space, output only those bands with high eigenvalues. Images with eigenvalues close to 1 are mostly noise.
  8. Click OK. When ENVI finishes processing, the MNF Eigenvalues plot window appears and the MNF bands are added to the Layer Manager. See Interpret the MNF Eigenvalues Plot.

Estimate Noise Statistics from Dark Data

Use Forward MNF Noise Statistics from Dark Data to estimate noise statistics from a dark image. Some instruments collect a dark image as part of data calibration or collection. The dark image is the equivalent of a photograph with the lens cap on. In electronic instruments, there is still a signal that is actually the noise of the instrument. You can use the dark image to characterize the instrument noise.

  1. From the Toolbox, select Transform > MNF Rotation > Forward MNF Noise Statistics from Dark Data.
  2. The Input File dialog appears.

  3. Select an input file and optional spatial and spectral subsetting, and/or masking, then click OK. The Dark Current Input File dialog appears.
  4. Select the dark current file and perform spatial subsetting as required, then click OK. ENVI calculates the noise statistics using the dark current input file. The Forward MNF Transform Parameters dialog appears.
  5. In the Enter Output Noise Stats Filename [.sta] field, enter an output file for the noise statistics.
  6. In the Enter Output MNF Stats Filename [.sta] field, enter an output file for the MNF statistics. Be sure that the MNF and noise statistics files have different names.
  7. Select output to File or Memory.
  8. Select the number of output MNF bands by using one of the following options:
    • To select the number of output bands without examining the eigenvalues, select No from the Select Subset from Eigenvalues toggle button, then set the Number of Output MNF Bands.
    • To select the number of output MNF bands by examining the eigenvalues, use the following steps:
      1. Select Yes from the Select Subset from Eigenvalues toggle button.
      2. Click OK. ENVI calculates the noise statistics and the first rotation. The Select Output MNF Bands dialog appears, with each band listed with its corresponding eigenvalue. Also listed is the cumulative percentage of data variance contained in each MNF band for all bands.
      3. Set the Number of Output MNF Bands. For the best results, and to save disk space, output only those bands with high eigenvalues. Images with eigenvalues close to 1 are mostly noise.
  9. Click OK. When ENVI finishes processing, the MNF Eigenvalues plot window appears and the MNF bands are added to the Layer Manager. See Interpret the MNF Eigenvalues Plot.

Inverse MNF Transforms


Use Inverse MNF Transform to transform the MNF bands back to their original data space.

Note: Smooth noise-dominated images (higher band numbers) before performing the inverse transform or eliminate the noisy bands during the inverse transform using spectral subsetting. Smoothing or eliminating noisy bands reduces the noise in the original data space.

  1. From the Toolbox, select Transform > MNF Rotation > Inverse MNF Transform. The Inverse MNF Transform Input File dialog appears.
  2. Select an input file and perform optional spatial and spectral subsetting, then click OK. The Enter Forward MNF Stats Filename dialog appears.
  3. Enter the name of the forward MNF statistics file.
  4. Click OK. The Inverse MNF Transform Parameters dialog appears.
  5. Select output to File or Memory.
  6. From the Output Data Type drop-down list, select the data type.
  7. Click OK. ENVI adds the resulting output to the Layer Manager.

Apply Forward MNFs to Spectra


Use Apply Forward MNF to Spectra to transform endmember spectra into MNF space for use in mixture tuned matched filtering. Use the MNF statistics from the mixture tuned matched filtering input data file for the transform. You can collect the spectra to transform from plot windows, spectral libraries, ASCII files, ROIs, and statistics files using a dialog similar to the endmember collection dialog.

  1. From the Toolbox, select Transform > MNF Rotation > Apply Forward MNF to Spectra. The Forward MNF Statistics Filename dialog appears.
  2. Select the MNF statistics filename that was calculated for the mixture tuned matched filtering input data file, then click Open. The Forward MNF Convert Spectra dialog appears.
  3. Import spectra into the dialog by selecting Import > file_type from the dialog's menu bar, or by dragging and dropping the spectra from the Spectral Library Plots window to the Forward MNF Convert Spectra dialog (see Endmember Options for further details).
  4. If you are transforming library spectra into forward MNF space to use as endmembers in mixture tuned matched filtering, the input library spectra must be scaled identically to the image associated with the MNF statistics being used to do the forward transformation.

  5. To delete a spectrum from the dialog, select the spectrum and click Delete Spectrum.
  6. Click Apply. The transformed spectra appear in the Forward MNF Spectra plot window and you can input the spectra into the Mixture Tuned Matched Filtering Endmember Collection dialog.

Apply Inverse MNFs to Spectra


Use the Apply Inverse MNF to Spectra to transform MNF spectra back to the original data space. ENVI uses forward MNF statistics from a data file for the transform. You can collect the spectra to transform from plot windows, spectral libraries, ASCII files, ROIs, and statistics files using a dialog similar to the endmember collection dialog.

  1. From the Toolbox, select Transform > MNF Rotation > Apply Inverse MNF to Spectra. The Forward MNF Statistics Filename dialog appears.
  2. Select the forward MNF statistics filename, then click Open. The Inverse MNF Convert Spectra dialog appears.
  3. Import spectra into the dialog by selecting Import > file_type from the dialog's menu bar, or by dragging and dropping the spectra from the Spectral Library Plots window to the Forward MNF Convert Spectra dialog (see Endmember Options for further details).
  4. Click Apply. The transformed spectra appear in the Inverse MNF Spectra plot window.



© 2018 Harris Geospatial Solutions, Inc. |  Legal
My Account    |    Store    |    Contact Us