2967 Rate this article:
No rating

New Features Coming Soon to IDL - With a Few Brief Examples

Benjamin Foreback

IDL 8.4.1 is scheduled to release within the next few weeks. Although this is only a point release rather than a full new version, there are a few features that the IDL team has implemented that are very exciting to me.

HDF and NetCDF Updates

When working with scientific data, I often find myself using IDL to open and read HDF4 and HDF5 files. Although the design of HDF5 is quite a bit simpler than HDF4, working with the API for both formats can sometimes be rather painful, and I almost always need to reference the documentation examples just to get started. 

One of my favorite routines in IDL is H5_PARSE (which is a function in the IDL library and not part of the standard HDF5 interface). The reason I like this function so much is that I can use it to query datasets and attributes in the file without needing to write multiple lines of code to search for and access the desired information (and I can't forget to ensure the file gets closed even if an error occurs). Everything I want is simply dumped into a hierarchical structure.

My one complaint, however, about H5_PARSE is that it can be rather slow, especially with large files. The reason for this is that the structure is created dynamically as IDL descends through the file, using this logic:

sTree = CREATE_STRUCT(sTree, tagname, sMember)

Just like how appending an array is very inefficient, creating a structure in this fashion is slow and memory intensive, particularly when using the /READ_DATA keyword.

Luckily, IDL’s new OrderedHash data type provides a handy alternative; adding fields to a hash is much more efficient than appending a structure. Additionally, an ordered hash not only preserves the order, it also preserves names and cases (structures are not case-sensitive and special characters and spaces in object names must be converted to underscores to be valid structure tags).

Furthermore, what is cool about an ordered hash is that you can use Implied Print, and IDL will print out the file’s information in a hierarchical JSON format, mimicking the structure of the file. To use an ordered hash in place of a structure, use the /ORDEREDHASH keyword when calling H5_PARSE. Here is an example using an HDF5-EOS file:

file = 'C:\hdf5\hdfeos\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5'
result = H5_PARSE(file, /ORDEREDHASH)
result

IDL will print:

{
    "_NAME": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
    "_ICONTYPE": "hdf",
    "_TYPE": "GROUP",
    "_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
    "_PATH": "/",
    "_COMMENT": "",
    "HDFEOS": {
        "_NAME": "HDFEOS",
        "_ICONTYPE": "",
        "_TYPE": "GROUP",
        "_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
        "_PATH": "/",
        "_COMMENT": "",
        "ADDITIONAL": {
            "_NAME": "ADDITIONAL",
            "_ICONTYPE": "",
            "_TYPE": "GROUP",
            "_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
            "_PATH": "/HDFEOS",
            "_COMMENT": "",
            "FILE_ATTRIBUTES": {
                "_NAME": "FILE_ATTRIBUTES",
                "_ICONTYPE": "",
                "_TYPE": "GROUP",
                "_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
                "_PATH": "/HDFEOS/ADDITIONAL",
                "_COMMENT": "",
                "InstrumentName": {
                    "_NAME": "InstrumentName",
                    "_ICONTYPE": "text",
                    "_TYPE": "ATTRIBUTE",
                    "_NDIMENSIONS": 0,
                    "_DIMENSIONS": 0,
                    "_NELEMENTS": 1,
                    "_DATATYPE": "H5T_STRING",
                    "_STORAGESIZE": 3,
                    "_PRECISION": 24,
                    "_SIGN": "",
                    "_DATA": "OMI"


etc.

Another new function that IDL 8.4.1 will offer is HDF_PARSE. This is the equivalent to H5_PARSE but for HDF4 files. It will walk through a file and return information about groups, scientific datasets, images, palettes, annotations, and VData. This information will be returned in an ordered hash, following the same structure that the information is stored in the file. HDF_PARSE will always return an ordered hash; if you would like a structure instead, you can convert the hash to a structure using OrderedHash::ToStruct. Here is an example using a MODIS file:

file = 'C:\scratch\modisl2\MOD10_L2.A2002013.0930.003.2002017035237.hdf'
result = HDF_PARSE(file)
result

IDL prints:

{
    "_NAME": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
    "_TYPE": "GROUP",
    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
    "_PATH": "/",
    "MOD_Swath_Snow": {
        "_NAME": "MOD_Swath_Snow",
        "_TYPE": "GROUP",
        "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
        "_PATH": "/",
        "Geolocation Fields": {
            "_NAME": "Geolocation Fields",
            "_TYPE": "GROUP",
            "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
            "_PATH": "/MOD_Swath_Snow",
            "Latitude": {
                "_NAME": "Latitude",
                "_TYPE": "SD",
                "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                "_PATH": "/MOD_Swath_Snow/Geolocation Fields",
                "_NDIMENSIONS": 2,
                "_DIMENSIONS": [271, 406],
                "_IDL_DATATYPE": "FLOAT",
                "_DATA": "<unread>",
                "long_name": {
                    "_NAME": "long_name",
                    "_TYPE": "ATTRIBUTE",
                    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                    "_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
                    "_IDL_DATATYPE": "STRING",
                    "_DATA": "Coarse 5 km resolution latitude"
                },
                "units": {
                    "_NAME": "units",
                    "_TYPE": "ATTRIBUTE",
                    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                    "_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
                    "_IDL_DATATYPE": "STRING",
                    "_DATA": "degrees"
                },
                "valid_range": {
                    "_NAME": "valid_range",
                    "_TYPE": "ATTRIBUTE",
                    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                    "_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
                    "_IDL_DATATYPE": "FLOAT",
                    "_DATA": [-90.000000, 90.000000]
                },
                "_FillValue": {
                    "_NAME": "_FillValue",
                    "_TYPE": "ATTRIBUTE",
                    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                    "_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
                    "_IDL_DATATYPE": "FLOAT",
                    "_DATA": [-999.00000]
                },
                "source": {
                    "_NAME": "source",
                    "_TYPE": "ATTRIBUTE",
                    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                    "_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
                    "_IDL_DATATYPE": "STRING",
                    "_DATA": "MOD03 geolocation product; data read from center pixel in 5 km box"
                }
            },
            "Longitude": {

etc.

IDL 8.4.1 will also include new NetCDF routines for querying, retrieving and creating NetCDF files. These routines include NCDF_GET, NCDF_LIST, and NCDF_PUT

Overloading the ++ and -- Operators

I have always liked IDL’s ability to overload operators of classes that inherit IDL_Object. This gives me the ability to create objects that behave like their own data types.

One set of operators that couldn't be overloaded until now is the increment/decrement operator, ++/--.  I like this operator because not only is it easier to type var++ than var = var + 1, it is more efficient. There are also cases where “increment” has a meaning other than “add one,” such as if the value is not numeric. Here is an example of an alphabet class that increments through letters (all lower-case). Once the value reaches the letter “z,” the next value in the sequence will be “aa.” When it reaches “az,” incrementing it again will change the value to “ba,” etc.

pro SequentialLetters::_OverloadPlusPlus
  compile_opt idl2, hidden
  
  if (StrCmp(self.value, '')) then begin
    self.value = 'a'
    return
  endif
  
  byteValue = Byte(self.value)
  if (byteValue[-1] eq 122) then begin
    if (StrLen(self.value) eq 1) then begin
      self.value = 'aa'
    endif else begin
      ; Create a new SequentialLetters object to recursively handle all but the 
      ; last letter.
      newObj = Obj_New('SequentialLetters', VALUE=StrMid(self.value, 0, StrLen(self.value)-1))
      newObj++
      self.value = newObj.VALUE + 'a'
      Obj_Destroy, newObj
    endelse
  endif else begin
    byteValue[-1]++
    self.value = String(byteValue)
  endelse
  
end

;-------------------------------------------------------------------------------

pro SequentialLetters::_OverloadMinusMinus
  compile_opt idl2, hidden

  ; If there is only one character and it is 'a', then the value becomes
  ; an empty string.
  if (StrCmp(self.value, 'a')) then begin
    self.value = ''
    return
  endif
  
  byteValue = Byte(self.value)
  if (byteValue[-1] eq 97) then begin
    ; Create a new SequentialLetters object to recursively handle all but the 
    ; last letter.
    newObj = Obj_New('SequentialLetters', VALUE=StrMid(self.value, 0, StrLen(self.value)-1))
    newObj--
    self.value = newObj.VALUE + 'z'
    Obj_Destroy, newObj
  endif else begin
    byteValue[-1]--
    self.value = String(byteValue)
  endelse

end

;-------------------------------------------------------------------------------

pro SequentialLetters::SetProperty, VALUE=value, _REF_EXTRA=extra
  compile_opt idl2
  
  if (N_Elements(value) gt 0) then begin
    self.value = value
  endif
  
  if (ISA(extra)) then begin
    self.IDL_Object::SetProperty, _EXTRA=extra
  endif

end

;-------------------------------------------------------------------------------

pro SequentialLetters::GetProperty, VALUE=value, _REF_EXTRA=extra
  compile_opt idl2

  if (Arg_Present(value)) then begin
    value = self.value
  endif

  if (ISA(extra)) then begin
    self.IDL_Object::GetProperty, _EXTRA=extra
  endif

end

;-------------------------------------------------------------------------------


function SequentialLetters::Init, _REF_EXTRA=extra
  compile_opt idl2
  
  if (~self.IDL_Object::Init()) then return, 0
  
  if (ISA(extra)) then begin
    self.SetProperty, _EXTRA=extra
  endif
  
  return, 1
  
end

;-------------------------------------------------------------------------------

pro SequentialLetters__Define
  compile_opt idl2
  
  !null = {SequentialLetters, $
           inherits IDL_Object, $
           value: ''}

end

Here is an example of using this object:

myLetters = Obj_New('SequentialLetters', VALUE='z')
myLetters++
print, myLetters.VALUE

IDL prints:
aa

myLetters--
print, myLetters.VALUE

IDL prints:
z

Other Updates

A full list of new features and product improvements in IDL 8.4.1 will be published soon. Stay tuned!

Please login or register to post comments.