9

Feb

2017

Author: Austin Coates

Shadows can be difficult to deal with when introduced into an analysis pipeline. There are many different ways in which to deal with shadows, but I’ve found that one of the easiest ways is to simply mask them out. I work primarily with agricultural data and row-planted crops in particular. In this type of data, the shadows are cast on bare earth between rows, because of that they can be masked out without much heartache, assuming that your target material is the vegetation itself. Depending on what type of imagery is being used, shadows are normally one of the darkest, if not the darkest, surfaces in the image. This assumption can be used to our advantage—in a histogram of the imagery, the shadows should fall at the lower end.

For example, in this image of an orange orchard, the shadows are the darkest feature and thus fall at the bottom of the histogram.

Figure 1: NIR image of orange orchard

Figure 2: Histogram of figure 1

### By looking at the imagery, you can see that there is a distinctive valley between two major features. Using the NIR First Valley Method (Jan-Chang, Yi-Ta, Chaur-Tzuhn, & Shou-Tsung, 2016), you can isolate the local minima then use that as the upper bound of a threshold used for shadow masking. The local minima can be easily derived by using the method presented in the support document entitled: “Finding multiple local max/min values for 2D plots with IDL”.

Figure 3: Shadows displayed in red

Figure 4: First valley identified

```function local_max_finder, datax, datay, minima = minima
compile_opt idl2

;initialize list
max_points = list()

data_x = datax
data_y = datay
```
```  ;check for keyword, flip the sign of the y values
if keyword_set(minima) then data_y = -datay

;iterate through elements
for i=1, n_elements(data_y)-2 do begin
```
```    ;previous point less than i-th point and next point less than i-th point
if ( (data_y[i-1] le data_y[i]) AND (data_y[i] ge data_y[i+1])) then max_      points.add, i
endfor

;return an array of the indices where the extrema occur
return, max_points.toarray()

end

compile_opt IDL2

e = envi(/current)
oRaster = e.UI.SelectInputData(/Raster, bands = bands)

; Check for single band
if N_ELEMENTS(bands) gt 1 then begin
MESSAGE, 'Input raster may only contain 1 band', /INFORMATIONAL
return
endif

; Pull the data out of the image
data = oRaster.GetData(bands = bands, pixel_state = pixel_state)

; Convert data to float
data = float(bytscl(data))

; Mask all values in the image
bkgrd_pos = where(pixel_state ne 0)
data[bkgrd_pos] = !VALUES.F_NAN

; find all local minima based off the histogram of the image
h = histogram(data, LOCATIONS=xbin)

; Remove zero values
non_zeros = where(h ne 0)
h = h[non_zeros]
xbin = xbin[non_zeros]

; Get the number of points
n_pts = n_elements(h)

; Smooth data (7 pixel moving average)
boxcar = 7
p1 = (boxcar - 1) / 2
p2 = boxcar - p1
for i = 0 , n_pts-1 do begin
pos = [i-p1:i+p1]
pos = pos[where((pos ge 0) and (pos le n_pts-1))]
h[i] = mean(h[pos])
endfor

MINIMA = local_max_finder(xbin, h, /MINIMA)
x_extrema = xbin[MINIMA]
y_extrema = h[MINIMA]

; Plot all local minima
p = plot(xbin, h)
p3 = scatterplot(x_extrema, y_extrema, /current, /overplot, \$
symbol = 'o', sym_color = 'b', sym_thick = 2)

; create a new metadata object

; Create a classification image
oClass = e.CreateRaster(e.GetTemporaryFileName(), mask, SPATIALREF = oRaster.SPATIALREF, \$
oClass.save

; Add the new class image to envi

end

```

# References

Jan-Chang, C., Yi-Ta, H., Chaur-Tzuhn, C., & Shou-Tsung, W. (2016). Evaluation of Automatic Shadow Detection Approaches Using ADS-40 High Radiometric Resolution Aerial Images at High Mountainous Region. Journal of Remote Sensing & GIS.

Comments (0) Number of views (760) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

2

Feb

2017

### Base 60 encoding of positive floating point numbers in IDL

Author: Atle Borsholm

Here is an example of representing numbers efficiently using a restricted set of symbols. I am using a set of 60 symbols (or characters) to encode floating point numbers as strings of any selected length. The longer the strings are, the more precise the numbers will potentially be.

Here is an example of a representation, this is restricted to positive numbers, in order to keep the example short.

IDL> a=[14.33, 3.1415, 12345]
IDL> a
14.330000       3.1415000       12345.000
IDL> base60(a)
FotV*
FdiDx
HdzS*
IDL> base60(a, precision=8)
FotV**aO
FdiDx*^c
HdzS****
IDL> base60(base60(a)) - a
-4.5533356836102712e-006 -4.6258149324351905e-006    -0.016666666666424135
IDL> base60(base60(a, precision=8)) - a
-9.2104102122902987e-012 -4.6052051061451493e-013 -7.7159711509011686e-008

In this example, it can be seen that the 5-digit representations are not as close to the original numbers as the 8-digit representations.

The code example for the base60 function is listed below.
;
; Converts from a numeric type to a base 60 representation
; Converts from a base 60 string to a floating point representation
; PRECISION is only used to determine how many symbols to use when encoding,
; and is ignored for decoding.
function Base60, input, precision=precision
compile_opt idl2,logical_predicate

; set default precision of 5 digits for encoding only
if ~keyword_set(precision) then precision = 5

; base 60 symbology
symbols = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!@#\$%^&*'
base = strlen(symbols)

; fast conversion from symbol to value
lut = bytarr(256)
lut[byte(symbols)] = bindgen(base)

if isa(input, /string) then begin
; convert from base60 string to float
; find exponent first
scale = replicate(double(base),n_elements(input)) ^ \$
(lut[byte(strmid(input,0,1))] - base/2)
res = dblarr(n_elements(input))
for i=max(strlen(input))-1,1,-1 do begin
dig = lut[byte(strmid(input,i,1))]
res += dig
res /= base
endfor
res *= scale
endif else begin
; convert from float to base60 strings
; encode exponent(scale) first
ex = intarr(n_elements(input))
arr = input
dbase = double(base)
repeat begin
dec = fix(arr ge 1)
ex += dec
arr *= dbase ^ (-dec)
inc = fix(arr lt 1/dbase)
ex -= inc
arr *= dbase ^ inc
endrep until array_equal(arr lt 1 and arr ge 1/dbase,1b)
if max(ex) ge base/2 || min(ex) lt -base/2 then begin
message, 'Number is outside representable range'
endif
bsym = byte(symbols)
res = string(bsym[reform(ex+base/2,1,n_elements(ex))])
for i=1,precision-1 do begin
arr *= base
fl = floor(arr)
arr -= fl
res += string(bsym[reform(fl,1,n_elements(fl))])
endfor
endelse
return, res
end

Comments (0) Number of views (745) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

26

Jan

2017

### When Might I Use An IDL Task? IDL As a Key to Data Analysis in a Heterogeneous Computing Environment

Author: Jim Pendleton

In IDL 8.6 we've exposed a new feature that standardizes a way for IDL to be called from any environment which can communicate between processes on a single operating system via standard input, standard output, and standard error.

In our Harris Geospatial Custom Solutions Group, we look forward to deploying this new feature extensively to help our clients expose even more analysis capabilities into large, heterogeneous processing environments.

Although a programmer, in earlier IDL releases, could accomplish the goal of calling an IDL routine from an operating system-level script in an ad-hoc way using a combination of the IDL Runtime executable, the IDL function COMMAND_LINE_ARGS, and other techniques, the new IDL Task architecture adds a level of commonality and standardization.

In the past, you might have written individual IDL Runtime applications to execute atomic processes on data in this type of environment. Your architecture would package up arguments and make a call to the idlrt.exe with those arguments passed on the standard input command line, via a system(), fork(), or another language's equivalent to IDL's SPAWN, along with a path to the IDL SAVE file containing your "task" to execute.

With the IDL Task architecture, you write procedural wrappers to your functionality using standard IDL, in combination with a simple JSON file which defines the arguments for your task, their data types, transfer directions, etc.

Placing the compiled IDL task code along with the JSON in your IDL session's search path exposes the tasks to the IDL Task Engine. This is essentially a stateless application that wraps an IDL Runtime interpreter. It performs the essential bits of validating input and output arguments and packaging them up before calling your IDL routine.

Your job distribution system, such as Harris' Geospatial Framework, will call the IDL Task Engine with JSON that represents the name of the task to be executed along with the task's arguments, written to the task script's standard input.

The task engine starts an independent IDL interpreter process for each task, allowing multiple tasks to be executed in parallel, up to the number of available processing licenses.

The arguments to and from the IDL Task must use data types that can be represented in JSON.  That restriction precludes arguments that are disallowed from crossing process boundaries, such as references to objects or pointers, as defined either in IDL or in another language.

### An Example - Generating a Summary Report From Multiple Images

Let's say through some mechanism outside IDL and ENVI you have generated a directory of image files. Perhaps you own a fleet of UAVs with sensors, or a satellite or two. These image files contain data that, when distilled through various processing algorithms, can produce a single intelligence report.

You want to fold this workflow into a larger processing sequence of services that consists of multiple steps, only one of which involves the generation of the reports.

For the IDL portion, let's say you already have an object class in IDL that takes as input the path to a directory of images, performs classifications and time series analysis and outputs a PDF report with a summary of the results. Because it's just that simple in IDL.

Let's call this class IntelReportGenerator. We will look at this class first, outside the context of the IDL Task Engine. For simplicity, the class I will describe will only have two methods, an ::Init method and a ::GenerateReport method.

This class is super-efficient and only has a handful of member variables.

```Pro IntelReportGenerator__Define
!null = {IntelReportGenerator,  \$
ImageDirectory : '', \$ ; path to the images to be read, input
OutputReport   : '', \$ ; path to the report file to be written, input
Author         : '', \$ ; a name to be applied to the report, input
Debug          : !false \$ ; a flag to toggle more detailed debugging information
}
End```

PSA: I highly recommend adding a debug flag to each class. Debugging might not be enabled in an operational environment, but it's always nice to know it can be turned on without a modification and redeployment of the code.

The ::Init method of the class is primarily used to populate the member variables with the keyword parameters.

```Function IntelReportGenerator::Init, \$
Image_Directory = Image_Directory, \$
Author = Author, \$
Output_Report = Output_Report, \$
Debug = Debug, \$
Status = Status, \$
Error = Error
On_Error, 2
Status = !false ; Assume failure
Error = !null ; Clear any error string on input
Catch, ErrorNumber ; Handle any unexpected error conditions
If (ErrorNumber ne 0) then Begin
Catch, /Cancel
If (self.Debug) then Begin
; Return a complete traceback if debugging is enabled
Help, /Last_Message, Output = Error
EndIf Else Begin
; Return a summary error instead of a traceback
Error = !error_state.msg
EndElse
Return, Status
EndIf
self.Debug = Keywod_Set(Debug)
self.Author = Author ne !null ? Author : 'UNKNOWN'
If (~File_Test(Image_Directory, /Dir)) then Message, 'Image directory does not exist.', /Traceback
self.ImageDirectory = Image_Directory
; ... More here.  you get the idea.
Status = !true
Return, 1
End
```

Next, let's consider the ::GenerateReport method. It's a simple matter of programming. We loop over the files in the input image directory, magic occurs, and an output file is generated. I relish the elegance of a simple design, don't you?

```Pro IntelReportGenerator::GenerateReport, \$
Status = Status, \$
Error = Error
On_Error, 2
Status = !false
Error = !null
Catch, ErrorNumber
If (ErrorNumber ne 0) then Begin
Catch, /Cancel
If (self.Debug) then Begin
Help, /Last_Message, Output = Error
EndIf Else Begin
Error = !error_state.msg
EndElse
Return
EndIf
Files = File_Search(self.ImageDirectory)
ForEach File, Files Do Begin
;... Magic analysis here.  Batteries not included.
EndFor
; Magic report-writing here.  Nope, still no batteries.
Status = !true
End
```

All this should look familiar to you thus far if you have written any IDL code, especially the magic bits.

In order to put this functionality into an IDL Task workflow, we will need to write a procedural wrapper for our class that will instantiate an object with the appropriate keywords, then execute the method to generate the report. We will name this new routine IntelReportTask.

```Pro IntelReportTask, \$
Image_Directory = Image_Directory, \$
Author = Author, \$
Output_Report = Output_Report, \$
Debug = Debug, \$
Status = Status, \$ ; an output from this procedure, 0 = failure, 1 = success
Error = Error ; An error string if Status is 0, or null on return otherwise
On_Error, 2
Error = !null ; Clear any error string
Status = !false ; assume failure
; ALWAYS include a CATCH handler to manage unexpected
; exception conditions.
Catch, ErrorNumber
If (ErrorNumber ne 0) then Begin
Catch, /Cancel
If (self.Debug) then Begin
; Return a complete traceback if debugging is enabled
Help, /Last_Message, Output = Error
EndIf Else Begin
; Return only a summary error without traceback if debugging is off
Error = !error_state.msg
EndIf
Return
EndIf
; Attempt to create the report-generation object, passing through the keywords.
o = IntelReportGenerator( \$
Image_Directory = Image_Directory, \$
Author = Author, \$
Output_Report = Output_Report)
Status = Status, \$
Error = Error, \$
Debug = Debug)
If (Obj_Valid(o)) then Begin
; Call the method to generate the report
o.GenerateReport, Status = Status, Error = Error
EndIf
End```

An IDL Task routine definition is required to pass all its arguments via keywords. Other than that restriction, it is a standard IDL procedure. There is no magic required.

The new piece of functionality is the requirement of a JSON task definition file. Within this file we define the name of the task (which corresponds to the IDL procedure name) and the type definitions associated with each of the keywords.

The argument type definitions allow the IDL Task Engine itself to execute parameter type checking and validation before your procedure is even called, relieving you of the burden of writing code to ensure, for example, that a directory path that should be a string is not being populated by a floating point number, instead. For some pedants of certain schools of computer science thought, IDL's weak data type validation at compile time is a turn-off rather than a strength. Wrapping pure IDL in a task with stricter argument types enforced by the Task Engine is one way to assuage such opinions, perhaps as a stepping stone to more illuminated paths to consciousness.

Of course, it also means it makes your IDL Tasks less generic than they are within IDL itself.  A single IDL routine that may operate on any data type from byte values to double precision numbers may require two or more different IDL Task routines as wrappers if you want to expose more than one. Another option is to write your task with multiple keywords to accept different data types, then pass the input to a common processing algorithm.

The general JSON syntax of a Custom IDL Task is described here.

The JSON associated with the IDL task follows.

```{
"description": "Generates a report from a directory of images.",
"parameters": [
{
"name": "IMAGE_DIRECTORY",
"description": "URI to the directory containing image files",
"type": "STRING",
"direction": "input",
"required": true
},
{
"name": "AUTHOR",
"description": "Label to apply as the author to the output report",
"type": "STRING",
"direction": "input",
"required": false,
"default": "UNKNOWN"
},
{
"name": "OUTPUT_REPORT",
"description": "URI to the output report file",
"type": "STRING",
"direction": "input",
"required": true
},
{
"name": "DEBUG",
"description": "Flag to enable verbose debugging information during errors or processing",
"type": "BOOLEAN",
"direction": "input",
"required": false,
"default": false
},
{
"name": "STATUS",
"description": "Status of the report generation request at completion, or error.",
"type": "BOOLEAN",
"direction": "output",
"required": false
},
{
"name": "ERROR",
"description": "Any error text generated during processing",
"type": "STRINGARRAY",
"dimensions": "[*]",
"direction": "output",
"required": false
}
]
}```

Here, we have identified optional and required keywords, their input/output directions, and data types, among other things.

In the IDL documentation, we show some examples for calling a procedure within the context of an IDLTask object within IDL itself.  In truth, this has limited utility outside of debugging. If you're even a semi-competent IDL wizard (which I assume you are if you have read this far), you will recognize that within the context of IDL, the IDLTask class and the task wrapper you have written is simply adding some overhead to a call you could make directly to your intended "worker" routine.

The real value of an IDL Task is shown when you insert your functionality into a heterogeneous workflow, outside of IDL itself.

On Windows, the default location for the script is in the installation directory, "C:\Program Files\Harris\idl86\bin\bin.x86_64\idltaskengine.bat".

On Linux, the default path is /usr/local/harris/idl/bin/idltaskengine.

The input to the idltaskengine script is JSON-format text that represents the name of the task along with the parameters.  The JSON may be passed to the script's standard input either through redirection from a file (<) or a pipe (|), for example,

or

It is the responsibility of your framework to construct the appropriate JSON object to be passed to the task engine script.

For our current example, the JSON might be constructed like this:

```{
"inputParameters": {
"IMAGE_DIRECTORY": "/path-to-data/",
"AUTHOR": "MELENDY",
"OUTPUT_REPORT": "/path-to-report/myreport.pdf"
}
}```

Any parameters defined as having an output direction will be written to standard output in JSON format. In our example, the output might be returned in this general format if a handled error was encountered:

```{
"outputParameters": [{
"STATUS": false
}, {
"ERROR": [
"% SWAP_ENDIAN: Unable to swap object reference data type",
"% Execution halted at: SWAP_ENDIAN        99 C:\\Program Files\\Harris\\IDL86\\lib\\swap_endian.pro",
"%                      \$MAIN\$"
]
}]
}```

In the event of a truly wretched error, one that was unable to populate the JSON, the stderr return from the call to the IDL Task Engine script should be queried as well. See the "Exit Status" section of the online help topic, at the bottom of the page.

Your surrounding framework should be designed to validate the status return from the IDL Task Engine script on standard error first, then check for and parse any JSON returned on standard output.

### Geospatial Framework (GSF)

The Harris Geospatial Framework product (GSF) is just one example implementation of  a distributed processing architecture into which IDL Tasks might be "snapped".  Despite its marketing name, it is not limited to processing geospatial data only.

Comments (0) Number of views (802) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

15

Dec

2016

### Interactive Vector GUI

Author: Austin Coates

With holidays rapidly approaching, I thought it might be nice to present a light hearted demonstration of capabilities; something that you can delight friends and family with during cold winter nights.  This week’s post shows how to use a simple widget_window to create an interactive vector animation.  The code is broken up into four pieces VectorBlog, VectorBlog_Event, VectorBlogMouseEvent, and VectorBlogVectorBuilder. The VectorBlog creates the widget itself and initially displays the vector information.  VectorBlog_Event runs the events for the base widget.  This is mainly the closing of the widget. VectorBlogMouseEvent is the mechanism by which the mouse locations are recoded by the widget window. Last but not least VectorBlogVectorBuilder builds the two matrices containing the horizontal and vertical components of the vector.  When the following code is run the cursor location is recorded, a distance weighted vector diagram is created, and the resulting image is then displayed in the widget window.  The only tricky bits in this code relate to the fact that there are two event handlers, one for the base widget and one for the mouse events in the widget window.  The object reference for the image displayed must also be maintained and manipulated each time the cursor is move.

Figure 1: Example Vector Plot

```;+
; Build the vecort components
; This is distance weighting
;-
pro VectorBlogVectorBuilder, dims = dims, pt = pt, u = u, v = v
compile_opt IDL2

; Get all the x locations
xs = findgen(dims[0])
xs = xs#replicate(1,dims[1])

; Get all the y locations
ys = findgen(dims[1])
ys = transpose(ys# replicate(1,dims[0]))

; Build the horizontal components
u = pt[0] - xs

; Build the vertical components
v =  pt[1] - ys
end

;+
; The event handler called when the mouse is moved.
;-
FUNCTION VectorBlogMouseEvent, win, x, y, keymods
COMPILE_OPT idl2

; Set the window
win.SetCurrent

; Set the scale factor
sc = .05

; Get the size of the window
dims = win.DIMENSIONS * sc

; Get the vecotr information
VectorBlogVectorBuilder, dims = dims, pt = [x*sc,y*sc], \$
u = u , v = v

; Build the vecotr image
vector = vector(u,v,xrange=[0,dims[0]], yrange=[0,dims[1]], /current, AUTO_COLOR=1,  \$
RGB_TABLE=73, AXIS_STYLE = 0, LENGTH_SCALE=1.5, ARROW_THICK =3)

; Delete the old image
win.uvalue.delete

; Display the new image
win.uvalue = vector

END

;+
; The event handler for the widget
;-
pro VectorBlog_Event, ev
compile_opt IDL2

end

pro VectorBlog
compile_opt IDL2

; Setup the widget
tlb = widget_base()
wwindow = widget_window(tlb, MOUSE_MOTION_HANDLER='VectorBlogMouseEvent', \$
XSIZE=500, YSIZE=500)
widget_control, tlb, /REALIZE

; Set the scale factor
sc = .05

; Get infomation about the widget window
WIDGET_CONTROL, wwindow, GET_VALUE=w
dims = w.DIMENSIONS * sc

; Build the vecotr infomation
VectorBlogVectorBuilder, dims = dims, pt = [(dims[0])/2.,(dims[1])/2.], \$
u = u , v = v

; Display the vectors
vector = vector(u,v,xrange=[0,dims[0]], yrange=[0,dims[1]], /current, AUTO_COLOR=1,  \$
RGB_TABLE=73, AXIS_STYLE = 0, LENGTH_SCALE=1.5, ARROW_THICK =3)

; Record the vector object
w.uvalue = vector
WIDGET_CONTROL, tlb, SET_UVALUE=w

XMANAGER, 'VectorBlog', tlb

end```

Comments (1) Number of views (1036) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

7

Dec

2016

### Maintaining Backward Compatibility in IDL 8.6 - Keep Calm And Read Your Release Notes

Author: Jim Pendleton

One of the "benefits" of being a Harris Geospatial Solutions insider is access to pre-release candidates of the commercial products developed by our engineering group.

We in Custom Software Solutions are sometimes the canaries in the coal mine, learning on occasion that an "undocumented feature" that we had used benignly or even to our advantage has been removed from the language in a newer release. Generally, these changes are justifiable.

## Where to Find Release Notes

Sometimes it's not enough to look at the documentation center's What's New help page to learn of all these changes. An additional source of information, generally prepared after the "What's New" documentation has gone to press, is located in a place other than your IDL or ENVI installation's documentation subdirectory.

This file includes information about supported platforms and potential backward-compatibility issues.

If you have received a Harris product installation DVD, check the info subdirectory on the DVD itself for the release notes files.

After logging in, select the "Harris" link near the bottom of the web page, under "Browse My Software and Documentation".

Select the "IDL" link, that pops up in a new list in the "Product Lines" column.

Select the "IDL" link in the "Current Releases" tab.

On the "Product Download" page, select the appropriate item for the Release Notes document.

An Example Backward Compatibility Note

Recently, I discovered some of my routines were using an admittedly illegal syntax involving the "_REF_EXTRA" keyword passing mechanism.  The code wasn't so much illegal, as it was ignored. And one could argue the compiler should have complained about it from the time that _REF_EXTRA was added to the language. For example,

function MyRoutine, _REF_EXTRA=extra

MySubroutine, _REF_EXTRA=extra

end

Can you spot the problem?

The _REF_EXTRA keyword is only intended to appear in the declaration of a function or procedure.Within the body of the code, you should always use the _EXTRA keyword when making calls to other routines.

In the form shown above, the code is basically ignored at execution time.  It serves no useful purpose.

Up until IDL 8.6, illegal use of the _REF_EXTRA syntax would simply be skipped by the compiler and interpreter. In the example above, MySubroutine would be called without any keywords, regardless of what was passed to MyRoutine.

In IDL 8.6, the compiler has been beefed up to complain about the invalid syntax. If you have code that fits this pattern your code will not compile. You may want to simply remove the flagged code because it has never been operational. Or you may want to change the syntax.

function MyRoutine, _REF_EXTRA=extra

MySubroutine, _EXTRA=extra

end

Carefully consider the implications of changing the keyword, however. Modifying the syntax will also alter the behavior. You may end up modifying keywords on output that you hadn't intended to change!

Comments (0) Number of views (1226) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

## AUTHORS

 Industries Defense & Intelligence Environmental Monitoring Academic Precision Agriculture
 Learn Videos Blogs Events & Webinars Training Case Studies Whitepapers Resources

 Support Forums Help Articles Reference Guides Updates & Maintenance
 Company Mission Careers Press Room Legal Exelis, Inc
© 2017 Exelis Visual Information Solutions, Inc., a subsidiary of Harris Corporation