Since the creation of custom ENVITasks in ENVI 5.2 SP1, there has been the requirement that your procedure must commit all output objects to disk.  There was the rule that the procedure wrapped by the task must have an input keyword telling it the filename to use to write the object to.  In the task template, you would have an output parameter that mapped to that keyword, and then during task loading the framework would magically create an input parameter for you, mapped to the same keyword with TYPE set to “ENVIURI”. 

The automatic creation of the input parameter and the internal correlation of the two parameters were done with the best of intentions, to simplify the process of creating custom tasks.  Alas the user feedback on this feature wasn’t always as rosy as we hoped.

So in ENVI 5.4 we’re shaking things up and giving you the task developer more control.  If you still use the “version” property in the task template, and have it set to “5.2.1”, “5.3”, or “5.3.1”, then you’ll get the old behavior.  But if you switch to using “schema” set to the value “envitask_3.0”, then a new set of rules apply to the procedure, and what you can do inside it.  In the new paradigm, your procedure will have separate keywords for the input filename and output object reference.  If you like, you can skip the filename keyword completely and return an object that hasn’t been tethered to disk at all.  This makes life much easier for types like ENVIGCPSet and ENVITiePointSet, but also allows for a procedure that constructs a complex virtual raster chain based on the other input parameters.

You might be asking why you would want a task to return a virtual raster.  With the new Classification Framework that is part of ENVI 5.4, you need to make sure that you prepare the data you want to run through a trained classifier the same way you prepared your training data.  One way to do this is to create a task that returned the preprocessed data in a consistent way.  If you can get away with not having to save that preprocessed attribute raster to disk, why not take advantage of the time and space advantages of using a virtual raster.

The following example is a modified version of the new Custom Task Tutorial in the ENVI 5.4 release.  That task wraps a procedure that uses a number of other tasks internally to perform all the preprocessing.  Here I’ve modified it to use virtual rasters and other ENVI API function calls to avoid ever writing a file to disk.

The code goes through a number of steps, which I will describe after showing it:

pro SentinelVegIndices_blog, INPUT_RASTER_10M=raster10m, $
                             INPUT_RASTER_20M=raster20m, $
                             OUTPUT_RASTER=outputRaster
  compile_opt idl2, hidden
 
  ; Get the spatial reference of the 10-meter raster
  spatialRef = raster10m.SPATIALREF
  coordSys = ENVICoordSys(COORD_SYS_CODE=spatialRef.COORD_SYS_CODE)
 
  ; Create a spatial grid definition
  grid = ENVIGridDefinition(coordSys, $
                            PIXEL_SIZE=spatialRef.PIXEL_SIZE, $
                            NCOLUMNS = raster10m.NCOLUMNS, $
                            NROWS = raster10m.NROWS, $
                            TIE_POINT_MAP=spatialRef.TIE_POINT_MAP, $
                            TIE_POINT_PIXEL = spatialRef.TIE_POINT_PIXEL)
 
  ; Regrid the 20-meter bands to 10 meters
  regriddedRaster = ENVISpatialGridRaster(raster20m, GRID_DEFINITION=grid)
 
  ; Create a layer stack
  layerStack = ENVIMetaspectralRaster([raster10m, regriddedRaster], $
                                      SPATIALREF=spatialRef)
 
  ; Compute image statistics
  stats = ENVIRasterStatistics(layerStack)
 
  ; Perform dark subtraction as an alternative to atmospheric correction
  bandRasters = ObjArr(layerStack.nBands)
  for i = 1, layerStack.nBands do begin
    expression = 'b' + i.ToString() + ' - ' + stats.Min[i-1].ToString()
    bandRasters[i-1] = ENVIPixelwiseBandMathRaster(layerStack, expression)
  endfor
  bandStackRaster = ENVIMetaspectralRaster(bandRasters, $
                                           SPATIALREF=spatialRef)
 
  ; we need to put the wavelengths back into the band stack,
  ; they were removed by the band math
  metadata = ENVIRasterMetadata()
  metadata['wavelength'] = layerStack.Metadata['wavelength']
  metadata['wavelength units'] = layerStack.Metadata['wavelength units']
  correctedRaster = ENVIMetadataOverrideRaster(bandStackRaster, $
                                               METADATA_OVERRIDE=metadata)
 
  ; Scale pixel values from 0 to 1
  gains = MAKE_ARRAY(layerStack.NBANDS, /FLOAT, VALUE=0.0001)
  offsets = FLTARR(layerStack.NBANDS)
  scaledRaster = ENVIGainOffsetRaster(correctedRaster, gains, offsets)
 
 
  ; Create vegetation indices
  indices = [ 'Enhanced Vegetation Index', $
              'Global Environmental Monitoring Index', $
              'Leaf Area Index', $
              'Plant Senescence Reflectance Index', $
              'Red Edge Normalized Difference Vegetation Index' ]
  outputRaster = ENVISpectralIndexRaster(scaledRaster, indices)
end

The first step is to upsample the 20m raster to 10m pixels, which in the tutorial is performed using the ENVIRegridRasterTask.  This can be done with the ENVISpatialGridRaster virtual raster, once we have constructed an ENVIGridDefinition with the appropriate mixing of properties form the 10m and 20m rasters.

Next the tutorial uses the ENVIBuildBandStackTask to build a metaspectral stack of all the 10m bands.  Here we use the ENVIMetaspectralRaster virtual raster, though we have to pass in the original 10m raster’s spatial reference to keep this raster registered on the map.

Next is the dark subtraction.  The tutorial uses the ENVIRasterStatisticsTask, but here we just use the API function ENVIRasterStatistics() to accomplish the same thing.

The band minima values are used to perform dark object subtraction.  The tutorial uses the ENVIDarkSubtractionCorrectionTask, which handles this as a single raster.  Here I had to build separate band math equations for each band, as the ENVIPixelwiseBandMathRaster virtual raster always returns a single band output, so I have to build an array of band math expressions and then metaspectrally stack the results, again passing in the spatial reference.

One pitfall of the band math is that it removes most of the spectral metadata, so I have to put the wavelength metadata back into the raster so the spectral index calculations select the correct bands.  I do this with ENVIMetadataOverrideRaster(), using a copy of the original metaspectral raster’s metadata values.

The raster is then scaled down by a factor of 10000.0 with ENVIGainOffsetRaster, to simulate the atmospheric correction better and produce spectral index values that are more accurate.  Lastly, the scaled raster is passed into ENVISpectralIndexRaster to calculate 5 different spectral indices.

Once this has been written, we can use an almost identical .task file to wrap the procedure.  The main difference is that we no longer need to specify the OUTPUT_URI parameter, and I had a slightly different procedure name.  This task’s output raster is never commited to disk, but it can be used as input to another task by calling ENVIRaster::Dehydrate() on it, which yields a 25KB JSON representation.

Here is the updated .task file:

{
  "name": "SentinelVegIndices_blog",
  "base_class": "ENVITaskFromProcedure",
  "routine": "SentinelVegIndices_blog",
  "display_name": "Compute Sentinel-2A Vegetation Indices",
  "description": "This task regrids the visible and near-infrared bands of a Sentinel-2A Level-1C 20-meter image to 10 meters. It creates a layer stack of all bands. It applies dark-object subtraction as a simple alternative to atmospheric correction. It scales the reflectance pixel values from 0 to 1, then computes a select group of vegetation indices.",
  "schema": "envitask_3.0",
  "parameters": [
    {
    "name": "INPUT_RASTER_10M",
    "display_name": "Select a 10-meter image",
    "type": "ENVIRASTER",
    "direction": "input",
    "required": true,
    "description": "Select a Sentinel Level-1C 10-meter image."
    },
    {
    "name": "INPUT_RASTER_20M",
    "display_name": "Select a 20-meter image",
    "type": "ENVIRASTER",
    "direction": "input",
    "required": true,
    "description": "Select a Sentinel Level-1C 20-meter image."
    },
    {
    "name": "OUTPUT_RASTER",
    "display_name": "Output image",
    "type": "ENVIRASTER",
    "direction": "output",
    "required": true,
    "description": "This is a reference to the output raster."
    }
  ]
}

One of the greatest features of the Geospatial Services Framework (GSF), in my opinion, is the fact that it is built upon Node.js. There are many reasons why this is great, but I wanted to talk about one in particular which is the amount of customization this provides.

Below I will show a simple but powerful example of this customization, but before I get there, I want to give a quick overview of both GSF and Node.js. 

We have improved ENVI Service Engine by adding GSF – a lightweight, but powerful framework based on Node.js that can provide scalable geospatial intelligence for any size organization. I like to describe it as a “messenger” system that provides a way to communicate between the web and our various products or “engines” that provide analytics such as ENVI, IDL and more. 

//Import express, used by node.js to setup rest endpoints
var express = require('express');
//Import AWS, used to interact with Amazon Web Services including S3 storage
var aws = require('aws-sdk');
//Extra tools that should be included in request handlers
var extend = require('util')._extend;
var defaultConfig = require('./config.json');

/**
 * Dynamic Request Handler
 */

function DynUiHandler() {
    var s3Bucket, s3;
    //Setup config options
    var config = {};
    extend(config, defaultConfig);
    //Grab information for S3 from config.json
    s3Workspace = config.S3Root;
    s3Bucket = config.S3Bucket;

    // Setup S3
    s3 = new aws.S3(config);


    /**
     * List files in the s3 bucket.
     * @param {object} req - The Express request object.
     * @param {object} res - The Express response object.
     */
    function listS3Data(req, res) {
    //Take the parameter passed from the REST call and set that as the bucket to be accessed in the call to s3.
        var params = {
            Bucket: req.params.bucket
        }; 
        //Call the S3 package's built in listObjects function to return what is avaiable in the specified bucket

        s3.listObjects(params, function(err, data) {
        //check if error occured and if so, halt and report it to client.
            if (err) {
                var code = err.code || 500;
                var message = err.message || err;
                res.status(code).send({
                    error: message
                });
            } 
            // If no error, push every object found in the response with a '.dat' extension to an array that will be returned to client 
            else {
                var files = [];
                //Look at each file contained in data returned by s3.listObjects()
                data.Contents.forEach(function(file) {
                    //Searches for files with .dat in the bucket requested
                    if(file.Key.endsWith('.dat')){
                        //If found, store that file information in the files array.
                        files.push(file);
                    }
                });
            }
            //send the files array containing metadata of all .dat files found.
            res.send(files);
        });
    };

    //Initialize request handler, run when GSF starts up.
    this.init = function(app) {
        // Set up request handler to host the html subdirectory as a webpage.
        app.use('/dynui/', require('express').static(__dirname + '/html/'));
        // Set up a rest call that runs listS3Data and supplies the accomponying bucket parameter.
        app.get('/s3data/:bucket', listS3Data);
    };
}
//
module.exports = DynUiHandler;

After restarting the server I was able to hit my rest endpoint by pasting the following in my browser:
http://localhost:9191/s3data/mybucket

This would return me JSON of the contents of “mybucket”


 
Using this information I was able to give a user of my application real time information right inside of the interface about which data they have available to them. 
 



This is a simple example of how I customized a stock request handler to make my web demo more powerful. However, the ability to modify the source code of such a powerful system in a modular fashion provides a safe way to mold GSF into the exact form and shape you need to fit into your own architecture. 

It might just be me but I much prefer that over restructuring my own existing architecture to fit closed source code I know nothing about.

Recently, I was given the chance to practice some spectroscopy and in preparation for the project, I realized that I did not have a simple way to visualize the variations in different absorption features between very discreet wavelengths. The method that I elected to employ for this task is called continuum removal  (Kokaly, Despain, Clark, & Livo, 2007). This method allows you to compare different spectra and essentially normalize the data so that they can be more easily compared with one another.

To use the algorithm, you first select the region that you are interested in (for me this was between 550 nm and 700 nm -this is the region of my spectra that deals with chlorophyll absorption and pigment). Once the region is selected then a linear model is fit between the two points and this is called the continuum. The continuum is the hypothetical background absorption feature, or artifact or other absorption feature, which acts as the baseline for the target features to be compared against (Clark, 1999). Once the continuum has been set then the continuum removal process can be performed on all spectra in question using the following equation (Kokaly, Despain, Clark, & Livo, 2007).

 R_C is the resulting continuum removed spectra, R_L is the continuum line and, R_o is the original reflectance value.

Figure 1: Original spectra of two healthy plants. The dotted line denotes the continuum line. The x axis shows wavelengths in nm and the y axis represents reflectance.

Figure 2: The continuum removal for wavelengths 550 nm - 700 nm.

The resulting code gives you a tool that will take in two spectral libraries, with one spectra per library, and return two plots similar to what is shown in Figure 1 and Figure 2.

pro Continuum_Removal

compile_opt IDL2

Spectra_File_1  =

Spectra_File_2 =

; Find the bounds for the feature

FB_left = 550

FB_right =700

; Open Spectra 1

oSLI1 = ENVISpectralLibrary(Spectra_File_1)

spectra_name = oSLI1.SPECTRA_NAMES

Spectra_Info_1 = oSLI1.GetSpectrum(spectra_name)

; Open Spectra 2

oSLI2 = ENVISpectralLibrary(Spectra_File_2)

spectra_name = oSLI2.SPECTRA_NAMES

Spectra_Info_2 = oSLI2.GetSpectrum(spectra_name)

; Get the wavelengths

wl = Spectra_Info_1.wavelengths

; Create Bad Bands List (this removes some regions of the spectra associated with water vapor absorption)

bb_range = [[926,970],[1350,1432],[1796,1972],[2349,2500]]

bbl = fltarr(n_elements(wl))+1

dims = size(bb_range, /DIMENSIONS)

for i = 0 , dims[1]-1 do begin

  range = bb_range[*,i]

  p1 = where(wl eq range[0])

  p2 = where(wl eq range[1])

  bbl[p1:p2] = !VALUES.F_Nan

endfor

;Plot oSLI1 / oSLI2

p = plot(wl, Spectra_Info_1.spectrum*bbl, xrange = [min(wl, /nan),max(wl, /nan)],$

  yrange=[0,max([Spectra_Info_1.spectrum*bbl,Spectra_Info_2.spectrum*bbl], /nan)], thick = 2, color = 'blue')

p = plot(wl, Spectra_Info_2.spectrum*bbl, /overplot, thick = 2, color = 'green')

; create the linear segment

Spectra_1_y1 = Spectra_Info_1.spectrum[where( wl eq FB_left)]

Spectra_1_y2 = Spectra_Info_1.spectrum[where( wl eq FB_right)]

pl_1 = POLYLINE([FB_left,FB_right], [Spectra_1_y1, Spectra_1_y2], /overplot, /data, thick = 2, LINESTYLE = '--')

Spectra_2_y1 = Spectra_Info_2.spectrum[where( wl eq FB_left)]

Spectra_2_y2 = Spectra_Info_2.spectrum[where( wl eq FB_right)]

pl_2 = POLYLINE([FB_left,FB_right], [Spectra_2_y1, Spectra_2_y2], /overplot, /data, thick = 2, LINESTYLE = '--')

; Get the equation of the line

LF_1 = LINFIT([FB_left,FB_right], [Spectra_1_y1, Spectra_1_y2])

LF_2 = LINFIT([FB_left,FB_right], [Spectra_2_y1, Spectra_2_y2])

; Get the values between the lower and upper bounds

x_vals = wl [ where(wl eq FB_left) : where(wl eq FB_right)]

; Compute the continuum line

RL_1 = LF_1[0] + LF_1[1]* x_vals

RL_2 = LF_2[0] + LF_2[1]* x_vals

; Perform Continuum Removal

Ro_1 = Spectra_Info_1.spectrum[ where(wl eq FB_left) : where(wl eq FB_right)]

RC_1 =  Ro_1 / RL_1

Ro_2 = Spectra_Info_2.spectrum[ where(wl eq FB_left) : where(wl eq FB_right)]

RC_2 = Ro_2 / RL_2

; Plot the new Continuum Removal Spectra

pl_RC_1 = plot(x_vals, RC_1, color = 'Blue', xrange = [min(x_vals, /NAN), max(x_vals, /NAN)] )

pl_RC_2 = plot(x_vals, RC_2, color = 'Green', /overplot)

end

Kokaly, R. F., Despain, D. G., Clark, R. N., & Livo, K. E. (2007). Spectral analysis of absorption features for mapping vegetation cover and microbial communities in Yellowstone National Park using AVIRIS data.

Clark, R. N. (1999). Spectroscopy of rocks and minerals, and principles of spectroscopy. Manual of remote sensing, 3, 3-58.