>  Docs Center  >  ENVI Server  >  GSF Tutorial: Amazon S3 Public Data Sets

GSF Tutorial: Amazon S3 Public Data Sets

GSF - Tutorial - Amazon S3 Public Data Sets


This tutorial describes how to configure a server to use public Landsat 8 data from Amazon S3 as input to tasks.

Landsat 8 data is provided for public use via Amazon Web Services Public Datasets. New Landsat scenes are added daily and are available as an Amazon S3 bucket. The following tutorial addresses how to get started using the Amazon S3 Parameter Mapper with Landsat 8 data.

Specifying an S3 URL as an input value triggers this parameter mapper to download the data locally for processing. This parameter mapper then translates the URL into the local file path so that an engine can read the data as a normal file on disk.

Configure the Server

Enable the Amazon S3 Parameter Mapper

First, the server must have the Amazon S3 Parameter Mapper enabled. To configure this, add "gsf-amazon-s3-parameter-mapper" to the "parameterMappers" section of the top-level config file (config.json).

To configure the Amazon S3 Parameter Mapper from a command line, start a command prompt in the GSFxx directory and execute the following commands (replace YourAccessKeyID and YourSecretKey with your personal keys):

node updateConfig.js config.json --arrayadd parameterMappers={\"type\":\"gsf-amazon-s3-parameter-mapper\"}
node updateConfig.js config.json --set parameterMappers[type:gsf-amazon-s3-parameter-mapper].accessKeyId=YourAccessKeyID
node updateConfig.js config.json --set parameterMappers[type:gsf-amazon-s3-parameter-mapper].secretAccessKey=YourSecretKey
node updateConfig.js config.json --set parameterMappers[type:gsf-amazon-s3-parameter-mapper].downloadAllFilesInDirectory=true
node updateConfig.js config.json --set parameterMappers[type:gsf-amazon-s3-parameter-mapper].dataDirectory=AWSData

The updateConfig.js script will automatically back up the original config.json file for you.

You may also edit the config.json file manually as shown below. It is highly recommended that you save a backup copy of the config.json file before making any changes.

Example config.json excerpt with modifications:

  "parameterMappers": [
      "accessKeyId": "Your Access Key ID",
      "secretAccessKey": "Your Secret Key",
      "downloadAllFilesInDirectory": true,
      "dataDirectory": "AWSData"

Restart the server any time this file is changed so it reflects the new configuration.

Enter an Access Key (if needed)

If a server is running from an Amazon EC2 instance, then it has permissions to use S3 by default.

If a server is running on a machine outside of the Amazon cloud, the users must provide Amazon access keys from their Amazon Web Services account. Add these values to the "accessKeyId" and "secretAccessKey" sections of the config.json file (shown above).

Select a Local Directory (optional)

The parameter mapper automatically downloads Amazon S3 data for use by the processing engine. Therefore, the user must specify either a fully-qualified local path or a relative path where the data should be downloaded. This destination folder is specified by the "dataDirectory" configuration option.

Additional Configuration Options

The parameter mapper has the ability to download one file or all the files from an S3 directory. This configuration option is called "downloadAllFilesInDirectory". This is useful for data that requires the existence of ancillary files. Landsat data is a multi file format and needs this option to work.


Below is an example of a request that creates an NDVI raster using Landsat 8 S3 data as input.

http://localhost:9191/ese/services/ENVI/SpectralIndex/submitJob?INPUT_RASTER={"url":"s3://landsat-pds/L8/139/045/LC81390452014295LGN00/LC81390452014295LGN00_MTL.txt","factory":"URLRaster","dataset_index": 0}&INDEX=Normalized Difference Vegetation Index

© 2020 Harris Geospatial Solutions, Inc. |  Legal
My Account    |    Store    |    Contact Us