>  Docs Center  >  ENVI API  >  Classification Framework  >  Define and Train the Classifier

Define and Train the Classifier

Define and Train the Classifier

The next step in the classification process is to define a classification method, or classifier. Then pass the examples and class values to a trainer to iteratively minimize the classification error.

Define the Classifier


ENVI provides the classifiers listed below. The associated API objects, ENVITasks, and code examples are listed for each classifier.

Support Vector Machine (SVM)

API object: ENVISVMClassifier

ENVITask: ENVICreateSVMClassifierTask

Background topic: Support Vector Machine Background

Code examples:

Softmax Regression

API object: ENVISoftmaxRegressionClassifier

ENVITask: ENVICreateSoftmaxRegressionClassifierTask

Background topic: Softmax Regression Background

Code examples:

Create the classifier object. This example creates a SVM classifier and continues from the previous section, Prepare Data for Classification. From here on, code examples show how to use API objects rather than ENVITasks.

classifier = ENVISVMClassifier( $
  NATTRIBUTES=outExamples.NATTRIBUTES, $
  NCLASSES=outExamples.NCLASSES, $
  CLASS_NAMES=outExamples.CLASS_NAMES)

These steps do not run the classifier yet; they only define the input properties for the classifier.

Normalize the Examples


Classifiers are often sensitive to data values that widely vary. Use theENVIApplyGainOffsetToExamples procedure to scale the data to a consistent range of values.

normalizedExamples = ENVIApplyGainOffsetToExamples( $
  outExamples, OUTPUT_OFFSET=offset, OUTPUT_GAIN=gain)

Shuffle and Split the Examples


You should train a classifier on one set of examples and evaluate the classifier with another set.

First, shuffle the examples to create a random distribution of data. Use the ENVIShuffleExamples procedure as follows:

shuffledExamples = ENVIShuffleExamples(normalizedExamples)

Then use the ENVISplitExamples procedure to split an ENVIExamples object into two separate ENVIExample objects:

splitExamples = ENVISplitExamples(shuffledExamples, SPLIT_FRACTION=0.8)

The splitExamples object is a two-element array where the first array element, splitExamples[0], contains the examples that you will use to train the classifier. The second array element, splitExamples[1], contains the examples that you will evaluate the classifier with. The SPLIT_FRACTION keyword is set to 0.8 by default, which means that 80 percent of the original examples will be used for training and the other 20 percent will be used for evaluation.

Train the Classifier


A trainer is an algorithm that iteratively trains a classifier in order to minimize its error. It tries to adjust the classifier's internal parameters until the error (also called loss) converges on a minimum value. A loss function calculates the loss, which is a unitless number that indicates how closely the classifier fits the training data. A value of 0 represents a perfect fit. The further the value from 0, the less accurate the fit. A high-quality fit does not ensure that new data will be classified correctly. The classifier still must be evaluated with untrained data after training is complete.

A classifier can be trained once and used many times to classify similar datasets. It can also be trained in stages (for example, as more data become available) where subsequent training will further refine the classifier.

ENVI provides iterative and gradient descent trainers.

Iterative Trainer

An iterative trainer works with classifiers that know how to update themselves; for example, SVM. It does not tell the classifier how to change. It iterates until the loss function converges on a minimum value. It takes two input parameters:

  • Maximum iterations: The maximum number of iterations for which to compute the minimum, if the convergence criterion is not met.
  • Convergence criterion: Iterations stop when the change in loss value from the previous iteration falls below a specified threshold. This threshold is the convergence criterion.

Create the trainer object. This implementation of the SVM classifier converges in one iteration. See ENVISVMClassifier for more information.

trainer = ENVIIterativeTrainer( $
  CONVERGENCE_CRITERION=0.0001, $
  MAXIMUM_ITERATIONS=1)

Finally, train the classifier. The ENVITrainClassifier procedure takes in a classifier object, an examples object (the output from the splitting step earlier), and a trainer object. It updates the original classifier instead of creating a new output classifier.

ENVITrainClassifier, trainer, classifier, splitExamples[0]

You can skip directly to Evaluate the Classifier, or learn more about the gradient descent trainer next.

Gradient Descent Trainer

The gradient descent algorithm iteratively updates the classifier according to the classifier's gradient for that iteration.

Tip: The Softmax Regression classifier should use a gradient descent trainer. See Code Example: Softmax Regression Classification using API Objects.

The gradient descent trainer takes the same two input parameters as the iterative trainer, plus the following:

  • Learning rate: The step size to take downhill during each iteration.

First, set up the Softmax Regression classifier:

classifier = ENVISoftmaxRegressionClassifier( $
  NATTRIBUTES=outExamples.NATTRIBUTES, $
  NCLASSES=outExamples.NCLASSES, $
  CLASS_NAMES=outExamples.CLASS_NAMES)

Create the trainer object:

trainer = ENVIGradientDescentTrainer( $
  CONVERGENCE_CRITERION=1e-7, $
  LEARNING_RATE=100, $
  MAXIMUM_ITERATIONS=60)

Normalize the examples:

normalizedExamples = ENVIApplyGainOffsetToExamples( $
  outExamples, OUTPUT_OFFSET=offset, OUTPUT_GAIN=gain)

Shuffle the examples:

shuffledExamples = ENVIShuffleExamples(normalizedExamples)

Split the examples into training and evaluation sets:

splitExamples = ENVISplitExamples(shuffledExamples, SPLIT_FRACTION=0.8)

Finally, train the classifier. The ENVITrainClassifier procedure takes in a classifier object, an examples object (the output from the splitting step), and a trainer object.

ENVITrainClassifier, trainer, classifier, $
  splitExamples[0], LOSS_PROFILE=lossProfile

Loss Profile

Notice how this example specifies an output keyword called LOSS_PROFILE. You can plot the loss profile to evaluate how well the input parameters work together to converge on a minimum value.

Note: You only need to calculate a loss profile with the Softmax Regression classifier, not with SVM.

p = PLOT(lossProfile, $
  TITLE='Loss Profile', $
  XTITLE='Iterations', $
  YTITLE='Loss', $
  COLOR='red', $
  THICK=2)

The loss profile should look similar to the following, where the curve flattens and approaches a minimum value:

Trial-and-error is needed to find the optimal learning rate, maximum number of iterations, and convergence criterion.

If you set the learning rate to a smaller value (such as 10), you will need to increase the maximum number of iterations to approximately 800 for the loss function to converge. This can increase processing time.

If you set the learning rate to 10 without increasing the maximum number of iterations, the resulting loss profile will look similar to the following plot. The curve starts to approach a minimum but never reaches it.

If you set the learning rate too high, you can actually step over the minimum. The resulting loss profile reveals a large spike in the curve, which suggests oscillations and a learning rate that is too high. Increasing the maximum number of iterations shows repeated spikes and the solution never converges:

When the loss profile looks reasonable, the classifier is ready to evaluate.

Next Steps


The next step is to evaluate the performance of the classifier by calculating a confusion matrix and accuracy metrics. See Evaluate the Classifier.

 



© 2019 Harris Geospatial Solutions, Inc. |  Legal
My Account    |    Store    |    Contact Us