Welcome to the Harris Geospatial product documentation center. Here you will find reference guides, help documents, and product libraries.


Harris Geospatial / Docs Center / Geospatial Services Framework / GSF Tutorial: Custom Route Mapping

Geospatial Services Framework

GSF Tutorial: Custom Route Mapping

GSF - Tutorial - Custom Route Mapping

Route mappers are modules designed to help direct jobs to the desired node in a cluster. They are called when a job is submitted to make sure that job ends up where it needs to go. A route mapper can use any of the job submission parameters to decide where a job should be run. If the job does not get routed, it ends up on the default route.

Intensive jobs requiring specific hardware to run would be prime cases to take advantage of route mappers. In a cluster, the system administrator could make sure all of the nodes have the needed hardware, but that could be cost prohibitive if the needed hardware is expensive and only needed for a small percentage of jobs. For example, if one machine in the cluster has a solid state drive and one task needs to perform many IO operations, the administrator can use route mapping to increase performance and have that task always execute on the solid state drive machine.

Another application of route mapping is to establish priority and have a specific job always execute before any other submitted job. Order of route definition matters, and each node pulls jobs from each route based on the route order. Each node continues to pull jobs until it has emptied the first route and then move on to pull from the next route until that one is emptied as well or until a higher priority job presents.

The Follow-on Job Route Mapper


Disabled by default, the follow-on job route mapper is an example route mapper that can be used to send a new job to the same node that executed a previous job. This can be useful when the input of one job depends on the output of another. With a distributed workspace, the data would normally be copied on-the-fly to the whichever node picked up the new job. With the follow-on route mapper, the data copy step is eliminated by routing the job to the data instead of bringing the data to the job.

To enable the follow-on job route mapper, add it to route mappers in the server config.

  "routeMappers": [
    ...,
    {
      "type": "gsf-follow-on-job-route-mapper"
    }
  ],

When enabled, you can specify a follow-on job by adding the jobID in the submitJob REST path.

HTTP POST http://server:port/ese/services/ENVI/SpectralIndex/job31/SubmitJob

Create a Custom Route Mapper


This tutorial uses the SpectralIndex task as an example for creating a custom route mapper. The SpectralIndex task performance can greatly depend on the type or speed of a machine's hard disk because it performs many read/write operations. To increase this task's performance, this tutorial demonstrates how to create a new custom route mapper that maps this task to an 'IO' route that is only defined on nodes with a fast hard drive.

Create a new folder in the main GSFxx installation directory named custom-route-mapper. This folder will contain all the files created during this tutorial.

Create the following files:

  • package.json - Contains various metadata relevant to the route mapper.
  • config.json - Contains configuration options for the route mapper.
  • routeMapper.js - Contains the JavaScript to handle the route mapper.

Note: As in other tutorials, the full example code blocks are provided at the bottom of the page.

package.json

Create a file called package.json in the custom-route-mapper folder. Add the following contents to the file:

{
  "name": "custom-route-mapper",
  "version": "1.0.0",
  "description": "Custom route mapper. If the task name matches a name in the config, a new route will be assigned",
  "main": "routeMapper.js",
  "dependencies": {"gsf-server-logger": "latest"},
  "devDependencies": {},
  "scripts": {}
}

This file provides information about the route mapper and its dependencies.

config.json

Create a file called config.json in the custom-route-mapper folder. Add the following contents to the file:

{
  "taskRoutes": [
    {
      "tasks": ["SpectralIndex"],
      "route": "IO"
    }
  ]
}

taskRoutes

The taskRoutes property is an array of objects where each object defines an array of tasks and a single route.

If a job is submitted with one of the tasks in the array, it gets routed to the defined route.

routeMapper.js

Create a file called routeMapper.js in the custom-route-mapper folder.

Add the following JavaScript code to the routeMapper.js file.

Initialization

The first part of the code loads the logger, creates a class, and loads the configuration file. The init function is not used for this tutorial, but it is part of the interface and as such needed for the t=module to load properly.

/**
 * The CustomRouteMapper can modify the route based on the RequestHandlerAPI submitJob input
 */
var log = require('gsf-server-logger').logger;
function CustomRouteMapper() {
  // Load the configuration file
  var config = require('./config.json');
  // Init isn't needed for this tutorial, but we still need it to comply to the interface.
  this.init = function() {}; 
  <!-- TODO fill me in --> 
}

GetRoute

Every route mapper needs to define a getRoute function to implement the interface. The function passes in the parameters of the submitJob request. The callback function should only be called if the function translates the route.

The function loops through the configuration task routes, searching for the task to see if it needs routing. If the task has a custom route defined, the corresponding route is then used in place of the old route and is passed into the callback. The function then returns true to let the caller know a route mapper has handled the route.

Using process.nextTick allows the route mapper to wait until this function returns before calling the callback. This route mapper is not asynchronous by default, so this line is needed to implement the asynchronously-based interface correctly.

  this.getRoute = function(task, service, params, route, callback) {
    log.trace('Entered CustomRouteMapper getRoute', task, service, params, route);
    for (var i = 0; i < config.taskRoutes.length; i++) {
      if (config.taskRoutes[i].tasks.indexOf(task) >= 0) {
        // We found a route, so return the new route name.
        // This function should be async
        process.nextTick(function() {
          callback(null, config.taskRoutes[i].route);
        });
        return true;
      }
    }
    // Return false to indicate we are not routing the job.
    return false;
  };

It is important to export the new class so the server can instantiate it on load.

module.exports = CustomRouteMapper;

Configure The Server

The last step is to add the new custom-route-mapper to the list of route mappers. The custom-route-mapper must be placed before the gsf-basic-route-mapper (if it has been added), otherwise the gsf-basic-route-mapper will handle the route and the custom-route-mapper will not be called. The first route mapper to return true is used and no other route mappers get an opportunity route the job.

To add the custom route mapper from a command line, start a command prompt in the GSFxx directory and execute the following command:

node updateConfig.js config.json --arrayadd routeMappers={\"type\":\"./custom-route-mapper\"} --addbefore [type:gsf-basic-route-mapper]

Please note that running 'arrayadd' commands multiple times will add multiple elements to the config file.

The updateConfig.js script will automatically back up the original config.json file for you.

You may also manually configure the custom route mapper by editing the config.json as shown below. It is highly recommended to save a backup copy of the server-level config.json file before making any changes.

  "routeMappers": [
    {
      "type": "./custom-route-mapper" 
    },
    {
      "type": "gsf-basic-route-mapper" 
    }
  ],

Restart the server any time this file is changed so it reflects the new configuration.

Routes are also defined in the top-level server config.json file. Each node in the cluster can have its own set of routes defined, allowing the user to control where in the cluster a job is executed. This example routed the spectralIndex task to the "IO" route. Add "IO" to the routes array on all nodes in the cluster that have a fast hard drive so they pick up these jobs. If a job is on a route that does not have any configured nodes, the job waits in the queue until a node is added with that route - so be sure that each route has at least one node that can process the job.

To add a new route from a command line, start a command prompt in the GSFxx directory and execute the following command:

node updateConfig.js config.json --arrayadd routeInfo.routes=IO --addbefore default

You may also manually add a new route by editing the config.json as shown below. It is recommended that you back up your config file before making any changes manually.

  "routeInfo": {
    "routes": ["IO","default"],
    "defaultRoute":"default"
  },

Restart the server any time this file is changed so it reflects the new configuration.

Using The New Route Mapper

To use the new route mapper, just submit jobs normally. This route mapper uses the task name to ensure it is on the correct route, so no additional request information is needed.

To submit a spectral index job, use the jobRequest.js script in the scripts folder. From a command line in the installation folder, run:

node scripts/jobRequest.js

On the job status page, there is a jobRoute parameter whose value is always "IO" with this configuration. The job only executes on machines with this route defined.

Full Example Code Blocks

Place all of the following files in the custom-route-mapper folder and add the provided contents. Remember to Configure the Server before attempting to execute these processes.

package.json

{
  "name": "custom-route-mapper",
  "version": "1.0.0",
  "description": "Custom route mapper. If the task name matches a name in the config, a new route will be assigned",
  "main": "routeMapper.js",
  "dependencies": {"gsf-server-logger": "latest"},
  "devDependencies": {},
  "scripts": {}
}

config.json

{
  "taskRoutes": [
    {
      "tasks": ["SpectralIndex"],
      "route": "IO"
    }
  ]
}

routeMapper.js

/**
 * The CustomRouteMapper can modify the route based on the RequestHandlerAPI submitJob input
 */
var log = require('gsf-server-logger').logger;
function CustomRouteMapper() {
  // Load the configuration file
  var config = require('./config.json');
  // Init isn't needed for this tutorial, but we still need it to comply to the interface.
  this.init = function() {}; 
  this.getRoute = function(task, service, params, route, callback) {
    log.trace('Entered CustomRouteMapper getRoute', task, service, params, route);
    for (var i = 0; i < config.taskRoutes.length; i++) {
      if (config.taskRoutes[i].tasks.indexOf(task) >= 0) {
        // We found a route, so return the new route name.
        // This function should be async
        process.nextTick(function() {
          callback(null, config.taskRoutes[i].route);
        });
        return true;
      }
    }
    // Return false to indicate we are not routing the job.
    return false;
  };
}
module.exports = CustomRouteMapper;



© 2017 Exelis Visual Information Solutions, Inc. |  Legal
My Account    |    Buy    |    Contact Us