27

Feb

2017

Using SAR Imagery for Road Maintenance

Author: Cherie Muleh

How do you go about trying to find whether a building has sunk enough to become unsafe and destabilized? That might not be as much as you think. Or what if you needed to know if there has been lateral movement of a railroad due to a landslide which could cause a car to derail? When we need to know precise, fine scale answers, we can turn to SAR data for this type of knowledge.

SAR data has the ability to detect millimeter or centimeter scale movements, so you can really understand complex infrastructure decisions. Check out this blogpost by Dipak Paudyal at Esri Australia to see an example of using SAR data to monitor road subsidence. The post is called “SAR imagery – the smart solution for road maintenance.” By using remote sensing to study these sorts of infrastructure projects, we can economically use data to look at our infrastructure on a regional level.  

Have you thought about using SAR data to help with your infrastructure needs?

Comments (0) Number of views (959) Article rating: No rating

Categories: ENVI Blog | Imagery Speaks

Tags:

22

Feb

2017

Machine Learning Training for Automatic Target Detection

Author: Pedro Rodriguez

This blog offers a deeper dive into the machine learning training process for performing automatic target detection. Samples of automatic target detection were recently presented at the Machine Learning: Automate Remote Sensing Analytics to Gain a Competitive Advantage webinar.

Machine learning (ML) applications, from object recognition and caption generation, to automatic language translation and driverless cars, have increased dramatically over the last few years, powered mainly by the increase of computing power (using GPUs), reduced cost of storage, wider availability of training data, and development of new training techniques for the machine learning models.

In the last five years, Harris Corporation has made a multi-million dollar investment into applying machine learning to solve customer challenges using remote sensing data. In response to the increased interest from our customers in evaluating how machine learning can solve their problems using geospatial data, I set out to train some of my coworkers on how to build a ML model to perform automatic feature detection on 2D overhead imagery. This training was crucial for our Solutions Engineers (SEs) to be able to prototype custom solutions for our customers and/or to integrate Machine Learning with our other powerful image analytics software like ENVI/IDL.

Figure 1: From left to right, Jeff McKissick, Zach Norman, Pedro Rodriguez, Rebecca Lasica, and Dan Platt are pictured here in the lobby of the Harris Broomfield, CO office

 

For this particular automatic target detection training, I chose to build a ML classifier to identify all the crosswalks in a subset image of São Paulo, Brazil. São Paulo is notoriously known for congested streets and woefully unfriendly streets for pedestrians, often lacking zebra-type crosswalk markings. This application demonstrates how city officials can use ML to automatically find all the crosswalks in the city for urban planning purposes. As an example, by knowing where crosswalks are it can be used to determine the number of missing crosswalks and accurately gauge the amount of labor and material required in order to increase pedestrian safety.

In just a few hours, and with each trainee using a small Red Hat Virtual Machine of 4GB RAM and 2 CPUs, we were able complete the entire process, from gathering the training data, building the ML model, and finally classifying a subset of the selected raster dataset.

For the raster dataset we used a high resolution satellite image from DigitalGlobe, Inc. (0.3 GSD, 4-band (RGBN), WorldView-3) from São Paulo, Brazil. To gather the training data (positives and negatives), we used a custom ENVI extension to chip 35x35 pixels samples and augment the training data as shown in Figure 2 below.

Figure 2: “Chip From Points” ENVI extension to gather training data

For data augmentation, we simply rotated each image chip 90 degrees (4 rotations). For the sake of time, we only selected 100 positives and 200 negatives, which after the data augmentation we had 1,200 training chips (400 positives and 800 negatives). From the 1,200 training chips 10% were used for validation, 20% for testing, and 70% for doing the actual training of ML model.  As seen in the heatmap shown in Figure 3 below, the ML classifier resulting from the limited training dataset (1,200 samples x 70% training = 840 training samples) performed very poorly as it contained many false negatives (missed detections) and some false positives (wrong detections).

Figure 3: Heatmap from a ML classifier using 100 positives and 200 negatives with 4 rotations

 In order to highlight the true potential of our ML technology, I decided to train the crosswalk classifier with a larger training data set. For this, I increased the training data by 5 times, so instead of just having 100 positives and 200 negatives, the new training set had 500 positives and 1,000 negatives. I also rotated each image chip by 10 degrees (36 rotations) instead of just every 90 degrees (4 rotations) which augmented the total image samples to 54,000. Table 1 below summarizes the data set used in both cases.

Table 1: Date Set Characteristics

Data Set

Positives

Negatives

Rotations

Total Samples

Training Samples (70%)

Small

100

200

4

1,200

840

Large

500

1,000

36

54,000

37,800

 

The next step was to determine the number of iterations (mini-batch updates) that were needed to complete an epoch. One epoch consists of one full training cycle on the training data set. To calculate iterations per epoch we use the following formula:

 where,

TS = Training Samples and,

BS = Batch Size

 

 

It’s difficult to prescribe a minimum number of epochs for training a new model since it will vary depending on the difficulty of the problem, quality of the data, chosen network architecture, etc. As a starting point, I began with 42 epochs and to calculate the total number of iterations I used the following formula:

The required number of epochs can be determined by watching the validation accuracy as the training proceeds with increasing number of iterations. This validation accuracy can be plotted with IDL as Receiver Operating Characteristic (ROC) curves as seen in Figure 4 below:

 

Figure 4: ROC Curves with 10k, 20k, and 30k iterations

ROC curves feature false positive rate on the X axis and true positive rate on the Y axis. This means that the top left corner of the plot represents the “ideal” Machine Learning classifier, which has a false positive rate of zero, and a true positive rate of one. It can be seen from Figure 4 above, that the accuracy of the crosswalk classifier increased with the number of iterations. At 30,000 iterations (about 126 epochs), the ROC indicated that enough training was achieved with an overall accuracy (ACC) of 98.84%. Figures 5 and Figure 6 below show the results of the 30k iteration classifier in a dense urban scene and in a highway scene, respectively. This crosswalk classifier proved to be very robust against confusers, like other similar street marking, and occlusions, like partially hidden crosswalks in the shadows. I challenge you to find the crosswalks manually in the “before” (left) scenes of Figures 5 and Figure 6. You can later validate your answers in the “after”(right) scene that was analyzed using ML. Can you imagine manually identifying all the crosswalks in the city of Sao Paulo?


Figure 5: Urban Scene, before and after crosswalk detection


Figure 6: Highway Scene, before and after crosswalk detection


Automatic target detection is one of our most basic ML solutions, which usually involves searching for particular features in a large dataset, therefore applicable to many real world challenges. This type of solution is even more relevant with the “Big Data” surge in which studies indicate that only 0.5% of all data generated gets ever used or analyzed (1). It is clear that future business advantages in every industry will arise when companies are able to automatically analyze this surge of data. Machine Learning is not meant to replace industry professionals, but to off load some of the tedious task to the computer, so they can focus their expert attention to analysis and not on “snailing” large datasets searching for particular features. ML can also run 24/7 and is highly scalable to available computing resources.

I want to emphasize that at Harris Corp. we are not merely delivering software on disk, but an end-to-end solution to deliver answers to specific industry problems. To answers questions like, “How many utility poles need servicing?” “Which blades in a wind farm have damage?” or “How are the road conditions near me?” All of these are questions we have been able to accurately answer for our customers.

If you would like to know more about how we have implemented Machine Learning to address other real-world problems, watch the webinar that my co-worker Will Rorrer and I hosted in January 2017: Machine Learning: Automate Remote Sensing Analytic s to Gain a Competitive Advantage

Download a printer-friendly PDF of this blog here.

For any other questions, please contact our Software Sales Manager:

Kevin Wells

Kevin.wells@harris.com

Office: 303-413-3954

 

References:

(1)    https://www.technologyreview.com/s/530371/big-data-creating-the-power-to-move-heaven-and-earth/

Comments (0) Number of views (1956) Article rating: 5.0

Categories: ENVI Blog | Imagery Speaks

Tags:

9

Feb

2017

First Valley Shadow Detection

Author: Austin Coates

Shadows can be difficult to deal with when introduced into an analysis pipeline. There are many different ways in which to deal with shadows, but I’ve found that one of the easiest ways is to simply mask them out. I work primarily with agricultural data and row-planted crops in particular. In this type of data, the shadows are cast on bare earth between rows, because of that they can be masked out without much heartache, assuming that your target material is the vegetation itself. Depending on what type of imagery is being used, shadows are normally one of the darkest, if not the darkest, surfaces in the image. This assumption can be used to our advantage—in a histogram of the imagery, the shadows should fall at the lower end.

For example, in this image of an orange orchard, the shadows are the darkest feature and thus fall at the bottom of the histogram.

Figure 1: NIR image of orange orchard

Figure 2: Histogram of figure 1

By looking at the imagery, you can see that there is a distinctive valley between two major features. Using the NIR First Valley Method (Jan-Chang, Yi-Ta, Chaur-Tzuhn, & Shou-Tsung, 2016), you can isolate the local minima then use that as the upper bound of a threshold used for shadow masking. The local minima can be easily derived by using the method presented in the support document entitled: “Finding multiple local max/min values for 2D plots with IDL”.

Figure 3: Shadows displayed in red

Figure 4: First valley identified

function local_max_finder, datax, datay, minima = minima
  compile_opt idl2
 
  ;initialize list
  max_points = list()
 
  data_x = datax
  data_y = datay
  
  ;check for keyword, flip the sign of the y values
  if keyword_set(minima) then data_y = -datay
 
  ;iterate through elements
  for i=1, n_elements(data_y)-2 do begin
    ;previous point less than i-th point and next point less than i-th point
    if ( (data_y[i-1] le data_y[i]) AND (data_y[i] ge data_y[i+1])) then max_      points.add, i
  endfor
 
  ;return an array of the indices where the extrema occur
  return, max_points.toarray()
 
end

        
pro ShadowFinder
  compile_opt IDL2
 
  ; Select your image
  e = envi(/current)
  oRaster = e.UI.SelectInputData(/Raster, bands = bands)
 
  ; Check for single band
  if N_ELEMENTS(bands) gt 1 then begin
    MESSAGE, 'Input raster may only contain 1 band', /INFORMATIONAL
    return
  endif
 
  ; Pull the data out of the image
  data = oRaster.GetData(bands = bands, pixel_state = pixel_state)
 
  ; Convert data to float
  data = float(bytscl(data))
 
  ; Mask all values in the image
  bkgrd_pos = where(pixel_state ne 0)
  data[bkgrd_pos] = !VALUES.F_NAN
 
  ; find all local minima based off the histogram of the image
  h = histogram(data, LOCATIONS=xbin)
 
  ; Remove zero values
  non_zeros = where(h ne 0)
  h = h[non_zeros]
  xbin = xbin[non_zeros]
 
  ; Get the number of points
  n_pts = n_elements(h)
 
  ; Smooth data (7 pixel moving average)
  boxcar = 7
  p1 = (boxcar - 1) / 2
  p2 = boxcar - p1
  for i = 0 , n_pts-1 do begin
    pos = [i-p1:i+p1]
    pos = pos[where((pos ge 0) and (pos le n_pts-1))]
    h[i] = mean(h[pos])
  endfor
 
  
  MINIMA = local_max_finder(xbin, h, /MINIMA)
  x_extrema = xbin[MINIMA]
  y_extrema = h[MINIMA]
 
  ; Plot all local minima
  p = plot(xbin, h)
  p3 = scatterplot(x_extrema, y_extrema, /current, /overplot, $
    symbol = 'o', sym_color = 'b', sym_thick = 2)
 
  ; Create a shadow mask
  mask = bytarr(oRaster.ns, oRaster.nl)
  mask[where(data le x_extrema[0])] = 1
 
  ; create a new metadata object
  metadata = ENVIRasterMetadata()
  metadata.AddItem, 'classes', 2
  metadata.AddItem, 'class names', ['Background', 'Shadow']
  metadata.AddItem, 'class lookup', [[0,0,0],[255,0,0]]
  metadata.AddItem, 'data ignore value', 0
  metadata.AddItem, 'band names', 'Shadows'
 
  ; Create a classification image
  oClass = e.CreateRaster(e.GetTemporaryFileName(), mask, SPATIALREF = oRaster.SPATIALREF, $
    data_type = 1, metadata = metadata)
  oClass.save
 
  ; Add the new class image to envi
  e.data.add, oClass
 
 
end

References

Jan-Chang, C., Yi-Ta, H., Chaur-Tzuhn, C., & Shou-Tsung, W. (2016). Evaluation of Automatic Shadow Detection Approaches Using ADS-40 High Radiometric Resolution Aerial Images at High Mountainous Region. Journal of Remote Sensing & GIS.

 

 

Comments (0) Number of views (741) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

2

Feb

2017

Base 60 encoding of positive floating point numbers in IDL

Author: Atle Borsholm

Here is an example of representing numbers efficiently using a restricted set of symbols. I am using a set of 60 symbols (or characters) to encode floating point numbers as strings of any selected length. The longer the strings are, the more precise the numbers will potentially be.
 
Here is an example of a representation, this is restricted to positive numbers, in order to keep the example short.
 
IDL> a=[14.33, 3.1415, 12345]
IDL> a
       14.330000       3.1415000       12345.000
IDL> base60(a)
FotV*
FdiDx
HdzS*
IDL> base60(a, precision=8)
FotV**aO
FdiDx*^c
HdzS****
IDL> base60(base60(a)) - a
 -4.5533356836102712e-006 -4.6258149324351905e-006    -0.016666666666424135
IDL> base60(base60(a, precision=8)) - a
 -9.2104102122902987e-012 -4.6052051061451493e-013 -7.7159711509011686e-008
 
In this example, it can be seen that the 5-digit representations are not as close to the original numbers as the 8-digit representations.
 
The code example for the base60 function is listed below.
;
; Converts from a numeric type to a base 60 representation
; Converts from a base 60 string to a floating point representation
; PRECISION is only used to determine how many symbols to use when encoding,
; and is ignored for decoding.
function Base60, input, precision=precision
  compile_opt idl2,logical_predicate
 
  ; set default precision of 5 digits for encoding only
  if ~keyword_set(precision) then precision = 5
 
  ; base 60 symbology
  symbols = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!@#$%^&*'
  base = strlen(symbols)
 
  ; fast conversion from symbol to value
  lut = bytarr(256)
  lut[byte(symbols)] = bindgen(base)
 
  if isa(input, /string) then begin
    ; convert from base60 string to float
    ; find exponent first
    scale = replicate(double(base),n_elements(input)) ^ $
      (lut[byte(strmid(input,0,1))] - base/2)
    res = dblarr(n_elements(input))
    for i=max(strlen(input))-1,1,-1 do begin
      dig = lut[byte(strmid(input,i,1))]
      res += dig
      res /= base
    endfor
    res *= scale
  endif else begin
    ; convert from float to base60 strings
    ; encode exponent(scale) first
    ex = intarr(n_elements(input))
    arr = input
    dbase = double(base)
    repeat begin
      dec = fix(arr ge 1)
      ex += dec
      arr *= dbase ^ (-dec)
      inc = fix(arr lt 1/dbase)
      ex -= inc
      arr *= dbase ^ inc
    endrep until array_equal(arr lt 1 and arr ge 1/dbase,1b)
    if max(ex) ge base/2 || min(ex) lt -base/2 then begin
      message, 'Number is outside representable range'
    endif
    bsym = byte(symbols)
    res = string(bsym[reform(ex+base/2,1,n_elements(ex))])
    for i=1,precision-1 do begin
      arr *= base
      fl = floor(arr)
      arr -= fl
      res += string(bsym[reform(fl,1,n_elements(fl))])
    endfor
  endelse
  return, res
end
 
 

Comments (0) Number of views (731) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

26

Jan

2017

When Might I Use An IDL Task? IDL As a Key to Data Analysis in a Heterogeneous Computing Environment

Author: Jim Pendleton

In IDL 8.6 we've exposed a new feature that standardizes a way for IDL to be called from any environment which can communicate between processes on a single operating system via standard input, standard output, and standard error. 

In our Harris Geospatial Custom Solutions Group, we look forward to deploying this new feature extensively to help our clients expose even more analysis capabilities into large, heterogeneous processing environments.

Although a programmer, in earlier IDL releases, could accomplish the goal of calling an IDL routine from an operating system-level script in an ad-hoc way using a combination of the IDL Runtime executable, the IDL function COMMAND_LINE_ARGS, and other techniques, the new IDL Task architecture adds a level of commonality and standardization.

In the past, you might have written individual IDL Runtime applications to execute atomic processes on data in this type of environment. Your architecture would package up arguments and make a call to the idlrt.exe with those arguments passed on the standard input command line, via a system(), fork(), or another language's equivalent to IDL's SPAWN, along with a path to the IDL SAVE file containing your "task" to execute.

With the IDL Task architecture, you write procedural wrappers to your functionality using standard IDL, in combination with a simple JSON file which defines the arguments for your task, their data types, transfer directions, etc.

Placing the compiled IDL task code along with the JSON in your IDL session's search path exposes the tasks to the IDL Task Engine. This is essentially a stateless application that wraps an IDL Runtime interpreter. It performs the essential bits of validating input and output arguments and packaging them up before calling your IDL routine.

Your job distribution system, such as Harris' Geospatial Framework, will call the IDL Task Engine with JSON that represents the name of the task to be executed along with the task's arguments, written to the task script's standard input.

The task engine starts an independent IDL interpreter process for each task, allowing multiple tasks to be executed in parallel, up to the number of available processing licenses.

The arguments to and from the IDL Task must use data types that can be represented in JSON.  That restriction precludes arguments that are disallowed from crossing process boundaries, such as references to objects or pointers, as defined either in IDL or in another language.

An Example - Generating a Summary Report From Multiple Images

Let's say through some mechanism outside IDL and ENVI you have generated a directory of image files. Perhaps you own a fleet of UAVs with sensors, or a satellite or two. These image files contain data that, when distilled through various processing algorithms, can produce a single intelligence report.

You want to fold this workflow into a larger processing sequence of services that consists of multiple steps, only one of which involves the generation of the reports.

For the IDL portion, let's say you already have an object class in IDL that takes as input the path to a directory of images, performs classifications and time series analysis and outputs a PDF report with a summary of the results. Because it's just that simple in IDL. 

Let's call this class IntelReportGenerator. We will look at this class first, outside the context of the IDL Task Engine. For simplicity, the class I will describe will only have two methods, an ::Init method and a ::GenerateReport method.

This class is super-efficient and only has a handful of member variables.

Pro IntelReportGenerator__Define
!null = {IntelReportGenerator,  $
    ImageDirectory : '', $ ; path to the images to be read, input
    OutputReport   : '', $ ; path to the report file to be written, input
    Author         : '', $ ; a name to be applied to the report, input
    Debug          : !false $ ; a flag to toggle more detailed debugging information
}
End

PSA: I highly recommend adding a debug flag to each class. Debugging might not be enabled in an operational environment, but it's always nice to know it can be turned on without a modification and redeployment of the code.

The ::Init method of the class is primarily used to populate the member variables with the keyword parameters.

Function IntelReportGenerator::Init, $
    Image_Directory = Image_Directory, $
    Author = Author, $
    Output_Report = Output_Report, $
    Debug = Debug, $
    Status = Status, $
    Error = Error
On_Error, 2
Status = !false ; Assume failure
Error = !null ; Clear any error string on input
Catch, ErrorNumber ; Handle any unexpected error conditions
If (ErrorNumber ne 0) then Begin
    Catch, /Cancel
    If (self.Debug) then Begin
        ; Return a complete traceback if debugging is enabled
        Help, /Last_Message, Output = Error
    EndIf Else Begin
        ; Return a summary error instead of a traceback
        Error = !error_state.msg
    EndElse
    Return, Status
EndIf
self.Debug = Keywod_Set(Debug)
self.Author = Author ne !null ? Author : 'UNKNOWN'
If (~File_Test(Image_Directory, /Dir)) then Message, 'Image directory does not exist.', /Traceback
self.ImageDirectory = Image_Directory
; ... More here.  you get the idea.
Status = !true
Return, 1
End

Next, let's consider the ::GenerateReport method. It's a simple matter of programming. We loop over the files in the input image directory, magic occurs, and an output file is generated. I relish the elegance of a simple design, don't you?

Pro IntelReportGenerator::GenerateReport, $
    Status = Status, $
    Error = Error
On_Error, 2
Status = !false
Error = !null
Catch, ErrorNumber
If (ErrorNumber ne 0) then Begin
    Catch, /Cancel
    If (self.Debug) then Begin
        Help, /Last_Message, Output = Error
    EndIf Else Begin
        Error = !error_state.msg
    EndElse
    Return
EndIf
Files = File_Search(self.ImageDirectory)
ForEach File, Files Do Begin
  ;... Magic analysis here.  Batteries not included.
EndFor
; Magic report-writing here.  Nope, still no batteries.
Status = !true
End

All this should look familiar to you thus far if you have written any IDL code, especially the magic bits.

In order to put this functionality into an IDL Task workflow, we will need to write a procedural wrapper for our class that will instantiate an object with the appropriate keywords, then execute the method to generate the report. We will name this new routine IntelReportTask.

Pro IntelReportTask, $
    Image_Directory = Image_Directory, $
    Author = Author, $
    Output_Report = Output_Report, $
    Debug = Debug, $
    Status = Status, $ ; an output from this procedure, 0 = failure, 1 = success
    Error = Error ; An error string if Status is 0, or null on return otherwise
On_Error, 2
Error = !null ; Clear any error string
Status = !false ; assume failure
; ALWAYS include a CATCH handler to manage unexpected
; exception conditions.
Catch, ErrorNumber
If (ErrorNumber ne 0) then Begin
    Catch, /Cancel
    If (self.Debug) then Begin
        ; Return a complete traceback if debugging is enabled
        Help, /Last_Message, Output = Error
    EndIf Else Begin
        ; Return only a summary error without traceback if debugging is off
        Error = !error_state.msg
    EndIf
    Return
EndIf
; Attempt to create the report-generation object, passing through the keywords.
o = IntelReportGenerator( $
    Image_Directory = Image_Directory, $
    Author = Author, $
    Output_Report = Output_Report)
    Status = Status, $
    Error = Error, $
    Debug = Debug)
If (Obj_Valid(o)) then Begin
    ; Call the method to generate the report
    o.GenerateReport, Status = Status, Error = Error
EndIf
End

An IDL Task routine definition is required to pass all its arguments via keywords. Other than that restriction, it is a standard IDL procedure. There is no magic required.

The new piece of functionality is the requirement of a JSON task definition file. Within this file we define the name of the task (which corresponds to the IDL procedure name) and the type definitions associated with each of the keywords.

The argument type definitions allow the IDL Task Engine itself to execute parameter type checking and validation before your procedure is even called, relieving you of the burden of writing code to ensure, for example, that a directory path that should be a string is not being populated by a floating point number, instead. For some pedants of certain schools of computer science thought, IDL's weak data type validation at compile time is a turn-off rather than a strength. Wrapping pure IDL in a task with stricter argument types enforced by the Task Engine is one way to assuage such opinions, perhaps as a stepping stone to more illuminated paths to consciousness.

Of course, it also means it makes your IDL Tasks less generic than they are within IDL itself.  A single IDL routine that may operate on any data type from byte values to double precision numbers may require two or more different IDL Task routines as wrappers if you want to expose more than one. Another option is to write your task with multiple keywords to accept different data types, then pass the input to a common processing algorithm.

The general JSON syntax of a Custom IDL Task is described here.

The JSON associated with the IDL task follows.

{
  "name": "IntelReportTask",
  "description": "Generates a report from a directory of images.",
  "base_class": "IDLTaskFromProcedure",
  "routine": "intelreporttask",
  "schema": "idltask_1.0",
  "parameters": [
    {
      "name": "IMAGE_DIRECTORY",
      "description": "URI to the directory containing image files",
      "type": "STRING",
      "direction": "input",
      "required": true
    },
    {
      "name": "AUTHOR",
      "description": "Label to apply as the author to the output report",
      "type": "STRING",
      "direction": "input",
      "required": false,
      "default": "UNKNOWN"
    },
    {
      "name": "OUTPUT_REPORT",
      "description": "URI to the output report file",
      "type": "STRING",
      "direction": "input",
      "required": true
    },
    {
      "name": "DEBUG",
      "description": "Flag to enable verbose debugging information during errors or processing",
      "type": "BOOLEAN",
      "direction": "input",
      "required": false,
      "default": false
    },
    {
      "name": "STATUS",
      "description": "Status of the report generation request at completion, or error.",
      "type": "BOOLEAN",
      "direction": "output",
      "required": false
    },
    {
      "name": "ERROR",
      "description": "Any error text generated during processing",
      "type": "STRINGARRAY",
      "dimensions": "[*]",
      "direction": "output",
      "required": false
    }
  ]
}

Here, we have identified optional and required keywords, their input/output directions, and data types, among other things.

In the IDL documentation, we show some examples for calling a procedure within the context of an IDLTask object within IDL itself.  In truth, this has limited utility outside of debugging. If you're even a semi-competent IDL wizard (which I assume you are if you have read this far), you will recognize that within the context of IDL, the IDLTask class and the task wrapper you have written is simply adding some overhead to a call you could make directly to your intended "worker" routine.

The real value of an IDL Task is shown when you insert your functionality into a heterogeneous workflow, outside of IDL itself.

In this environment, your framework will launch a command line-level script to execute your task.

On Windows, the default location for the script is in the installation directory, "C:\Program Files\Harris\idl86\bin\bin.x86_64\idltaskengine.bat".

On Linux, the default path is /usr/local/harris/idl/bin/idltaskengine.

The input to the idltaskengine script is JSON-format text that represents the name of the task along with the parameters.  The JSON may be passed to the script's standard input either through redirection from a file (<) or a pipe (|), for example,

<installpath>\idltaskengine.bat < <filepath>\my_intel_report_request.json

or

echo '{"taskName":"IntelReportTask","inputParameters":{"IMAGE_DIRECTORY":"<imagespath>"}, etc.}' | <installpath>/idltaskengine

It is the responsibility of your framework to construct the appropriate JSON object to be passed to the task engine script.

For our current example, the JSON might be constructed like this:

{
	"taskName": "IntelReportTask",
	"inputParameters": {
		"IMAGE_DIRECTORY": "/path-to-data/",
		"AUTHOR": "MELENDY",
		"OUTPUT_REPORT": "/path-to-report/myreport.pdf"
	}
}

Any parameters defined as having an output direction will be written to standard output in JSON format. In our example, the output might be returned in this general format if a handled error was encountered:

{
    "outputParameters": [{
        "STATUS": false
    }, {
        "ERROR": [
            "% SWAP_ENDIAN: Unable to swap object reference data type",
            "% Execution halted at: SWAP_ENDIAN        99 C:\\Program Files\\Harris\\IDL86\\lib\\swap_endian.pro",
            "%                      $MAIN$"
        ]
    }]
}

In the event of a truly wretched error, one that was unable to populate the JSON, the stderr return from the call to the IDL Task Engine script should be queried as well. See the "Exit Status" section of the online help topic, at the bottom of the page.

Your surrounding framework should be designed to validate the status return from the IDL Task Engine script on standard error first, then check for and parse any JSON returned on standard output.

More Examples

Additional IDL Task examples can be found here.

Geospatial Framework (GSF)

The Harris Geospatial Framework product (GSF) is just one example implementation of  a distributed processing architecture into which IDL Tasks might be "snapped".  Despite its marketing name, it is not limited to processing geospatial data only.



Comments (0) Number of views (784) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

12345678910Last

MONTHLY ARCHIVE

«March 2017»
SunMonTueWedThuFriSat
2627281234
567891011
12131415161718
19202122232425
2627282930311
2345678

MOST POPULAR POSTS

AUTHORS

Authors

Authors

Authors

Authors

Authors

Authors

Authors

Authors

Authors

Authors

GUEST AUTHORS

Authors

Authors

Authors

Authors

Authors

Authors

Authors



© 2017 Exelis Visual Information Solutions, Inc., a subsidiary of Harris Corporation