Welcome to the Harris Geospatial product documentation center. Here you will find reference guides, help documents, and product libraries.


  >  Docs Center  >  IDL Reference  >  I/O - Network  >  IDLnetURL

IDLnetURL

IDLnetURL

An IDLnetURL object allows IDL to act as a client to an HTTP or FTP server. The object handles:

  • Opening a connection to a remote HTTP or FTP server.
  • Creating and sending HTTP requests to the remote HTTP or FTP server.
  • Receiving responses from the remote HTTP or FTP server.
  • Receiving and writing large data files to disk at a user-specified location.

For additional information, see the following sections:

Superclasses


None.

Creation


See IDLnetURL::Init.

Properties


Objects of this class have the following properties. See IDLnetURL Properties for details on individual properties.

AUTHENTICATION

CALLBACK_DATA

CALLBACK_FUNCTION

CONNECT_TIMEOUT

CONTENT_TYPE

ENCODE

FTP_CONNECTION_MODE

HEADERS

PROXY_AUTHENTICATION

PROXY_HOSTNAME

PROXY_PASSWORD

PROXY_PORT

PROXY_USERNAME

RESPONSE_CODE

RESPONSE_FILENAME

RESPONSE_HEADER

SSL_CERTIFICATE_FILE

SSL_VERIFY_HOST

SSL_VERIFY_PEER

SSL_VERSION

TIMEOUT

URL_HOSTNAME

URL_PASSWORD

URL_PATH

URL_PORT

URL_QUERY

URL_SCHEME

URL_USERNAME

VERBOSE

 

Methods


This class has the following methods:

Examples


See the following methods for examples:

Version History


6.4

Introduced

8.3

Added ::Delete method

Overview of the IDLnetURL Object


The following sections give a general overview of the IDLnetURL object’s capabilities.

HTTP Functionality

The IDLnetURL object enables the following HTTP-related tasks:

  • Allows IDL to act as a HTTP client node.
  • Allows IDL to act as a HTTP SSL client node.
  • Issues Get requests to retrieve data from a remote HTTP server and place it in a buffer or file.
  • Issues Put requests to move data from a buffer or file to a remote HTTP server.
  • Issues Post requests to move data from a buffer or file to a remote HTTP server.
  • Encrypts all data to and from a remote HTTP server with SSL.
  • Compresses and decompresses data to and from a remote HTTP server.
  • Supports Basic and Digest authentication modes.
  • Supports customizing HTTP headers that are later sent to the remote HTTP server.
  • Returns the HTTP response code and HTTP response header to the user.
  • Provides status callbacks so the user can monitor call progress.
  • Requests can be cancelled if the user chooses not to wait for a complete reply.

FTP Functionality

The IDLnetURL object enables the following FTP-related tasks:

  • Allows IDL to act as a FTP client node.
  • Allows IDL to act as a FTP SSL client node.
  • Issues Get requests to retrieve data from a remote FTP server to a buffer or file.
  • Issues Put requests to move data from a buffer or file to a remote FTP server.
  • Issues Dir requests to retrieve directory listings from a remote FTP server.
  • Issues commands (Delete, Make dir, and so on) to a remote FTP server.
  • Encrypts all data to and from a remote FTP server with SSL.
  • Supports authentication (logging into a remote FTP server).
  • Returns the FTP response code and FTP response replies to the user.
  • Provides status callbacks so the user can monitor call progress.
  • Requests can be cancelled if the user chooses not to wait for a complete reply.
  • Supports custom commands that are sent to the remote FTP server.
  • Supports both Active and Passive mode connections.

HTTP Authentication Modes

The IDLnetURL object supports Disabled, Basic, and Digest HTTP authentication modes. It also supports authenticating with a remote HTTP or FTP server, and with proxy servers. The username and password can be set for both a remote HTTP or FTP server, and for a proxy server.

FTP Authentication

The IDLnetURL object supports logging in to a remote FTP server. A username and password can be set prior to connecting to a remote FTP server.

Secure HTTP

The IDLnetURL object supports the HTTPS protocol by implementing a secure socket layer (SSL). Secure HTTP ensures that all requests and responses are encrypted. To activate SSL, set the URL_SCHEME property to ‘https’.

Secure FTP

The IDLnetURL object supports implicit and explicit SSL, which ensure that all requests and responses are encrypted. An FTP server may support implicit SSL, explicit SSL, or both.

Note: Different FTP servers provide different responses when SSL authentication is not available. IDL attempts to catch these responses and generate errors if the FTP server does not support SSL, but there may be cases when IDL makes a non-SSL connection even when an SSL connection is requested. If you are not sure that the FTP server you are connecting to supports SSL, view the complete response headers to check that the connection is being made using SSL.

Explicit SSL is activated by setting the FTP_EXPLICIT_SSL keyword in the Get, Put, GetFtpDirList, and FtpCommand methods. Implicit SSL is activated by setting the scheme to ‘ftps’.

Note: Implicit SSL has been deprecated.

SSL Protocol Versions

By default, the IDLnetURL object supports automatic determination of the SSL protocol that is supported by the remote HTTP or FTP server. If the remote server does not support automatic determination of the SSL protocol version, you can direct the object to use a specific version with the SSL_VERSION property. The IDLnetURL object supports protocol versions TLS v1, SSL v2, and SSL v3. TLS v1 supersedes all previous versions of the SSL protocols.

HTTP Encoding

The IDLnetURL object supports compressed/encoded responses from a remote HTTP server, as well as the deflate and GZIP encoding formats. Refer to the description of the ENCODE property for details.

Proxies

The IDLnetURL object supports proxies. When the object utilizes a proxy, the PROXY_HOSTNAME and PROXY_PORT properties must be set.

Connection Caching

The IDLnetURL object caches up to five existing connections to remote HTTP or FTP servers. Once a connection is established to a remote server, the connection is reused on subsequent transactions with the remote server. Reusing a connection can save time because the connect, login, and SSL handshake information does not need to be repeated.

Use the IDLnetURL::CloseConnections method to close a connection. Typically, a remote server closes an existing connection after a period of no activity. When this occurs, the closed connection is removed from the existing connections list. Closing a connection using the CloseConnections method is useful for changing login information, enabling or disabling SSL, starting a new set of transactions on the FTP or HTTP server, and so on for the next transaction with the server.

User Agent

If you receive an Http ErrCode(403) or Http Err(Forbidden) the url host may require 'User-Agent:' value in the http header.

A 'User Agent:' field can be added to the http header, for example:

Obj->SetProperty, HEADERS = 'User-Agent: <user agent value>'

Mapping URL Components to IDLnetURL Properties


The following table shows how the components of a URL map to IDLnetURL properties. Set these properties by calling the SetProperty method.

Note: You can use the PARSE_URL function to break up a URL into its component strings, which can then be passed to IDLnetURL::SetProperty method.

A URL has the following composition:

URL_SCHEME://URL_USERNAME:URL_PASSWORD@URL_HOST:URL_PORT/URL_PATH?URL_QUERY

For example:

http://me:mypw@host.com:8080/project/data/get_data.cgi?dataset=climate&date=01012006

IDLnetURL Property

Value

URL_SCHEME

‘http’ (the default), ‘https’, ‘ftp’, or ‘ftps’

(The string “://” is appended to URL_SCHEME)

URL_USERNAME

Any string.

The character “:” is appended if URL_PASSWORD is supplied; otherwise the character “@” is appended.

Empty string is the default.

URL_PASSWORD

Any string.

The character “@” is appended.

Empty string is the default.

URL_HOST

my.hostname.com

Empty string is the default.

URL_PORT

8080

The default is 80.

The “:” character is prepended to URL_PORT if it is not null.

URL_PATH

demo/index.html

Empty string is the default.

The '/' character is prepended to URL_PATH if it is not null.

URL_QUERY

Parm1=value1&Parm2=value2& (and so on).

Empty string is the default.

The '?' character is prepended to URL_QUERY if it is not null.

Using Callbacks with the IDLnetURL Object


Callbacks from the IDLnetURL object provide a way to transmit information back to the caller based on the status of a request made to a HTTP or FTP server. The value returned by the callback function is then used by the IDLnetURL object to determine whether to continue or abort the request.

The caller must set the callback function property to receive callbacks from a method. The callback is made to an IDL function, which takes IDL variables as input parameters.

Note: The callback method is intended to provide status and progress information. The callback should not be used to process the incoming or outgoing data. Any data processing must occur after the Get, Put, GetFtpDirList, or FtpCommand methods have returned.

The IDLnetURL object lets you define functions that are called during a Get or Put method call. If a callback function is specified with the CALLBACK_FUNCTION property, it is called in the following circumstances:

  • For the IDLnetURL::Get method, the callback function is called each time a packet is received from the HTTP or FTP server.
  • For the IDLnetURL::Put method, the callback function is called each time a packet of data is sent to the HTTP or FTP server.
  • For the IDLnetURL::GetFtpDirList method, the callback function is called each time a list of files is retrieved.
  • For the IDLnetURL::FtpCommand method, the callback function is called each time an FTP command is executed.

All methods invoke the callback function with an array of strings indicating status as the first parameter, an array of integers containing progress information as the second parameter, and with the value (if any) specified by the CALLBACK_DATA property as the third parameter. A callback function returns a value of type Int.

Cancelling a Request

The return value of the callback function should be an integer zero or one. If the return value is zero, a cancel message will be sent to the remote node. If the return value is one, the operation will continue.

Callback Routine Signature

A callback function is written in IDL and has the following format:

Function Callback_Function_Name, StatusInfo, ProgressInfo,    CallbackData
...
return vCancelFlag

where

  • Callback_Function_Name is the name of the callback function. This value is specified by the CALLBACK_FUNCTION property.
  • StatusInfo is the first input parameter. It is an array of IDL strings that contains status information about the Get or Put. When the caller is a user interface (UI), the UI typically displays the string array in a text widget.
  • ProgressInfo is the second input parameter, and is an array of signed 64-bit values (type Long64) that contains progress information:
  • ProgressInfo[0] – Contains 1 (one) when the ProgressInfo array contains valid data, or 0 (zero) when the array does not contain valid data.
  • ProgressInfo[1] – The download total used when doing a HTTP Get or FTP Get. This value is 0 when doing a HTTP Get and the transfer encoding is chunked.
  • ProgressInfo[2] – Contains the number of bytes that have been downloaded during the current HTTP or FTP get.
  • ProgressInfo[3] – Contains the total number of bytes to be uploaded during a HTTP Put, HTTP Post, or FTP Put.
  • ProgressInfo[4] – Contains the number of bytes currently uploaded during the HTTP Put, HTTP Post, or FTP Put.
  • CallbackData is the CALLBACK_DATA property that can be set prior to calling the Get, Put, GetFtpDirList, or FtpCommand method. This property is passed into the callback function unmodified, and the contents of the property are determined by the user. When the caller is a UI, the callback data typically contains the ID of a text widget that displays the status information strings. If the CALLBACK_DATA property is not set, its value is undefined.
  • The return value is of type Int, and can be used to cancel the Get, Put, GetFtpDirList, or FtpCommand operation. See Cancelling a Request for more information.

HTTP and FTP Response Codes


The following tables list the response codes stored in the RESPONSE_CODE property for the last HTTP or FTP request sent to the remote server. See RESPONSE_CODE for more information about the property that holds these values.

The following table lists the error response codes generated by the IDLnetURL object:

Code

Code Number

Description

UNSUPPORTED_PROTOCOL

1

The URL you passed uses an unsupported protocol. The problem might be an unused compile-time option or a misspelled protocol string.

FAILED_INIT

2

Very early initialization code failed. This is likely an internal error or problem.

URL_MALFORMAT

3

The URL was not properly formatted.

URL_MALFORMAT_USER

4

Not used.

COULDNT_RESOLVE_PROXY

5

The given proxy host could not be resolved.

COULDNT_RESOLVE_HOST

6

The given remote host was not resolved.

COULDNT_CONNECT

7

Failed to connect to host or proxy.

FTP_WEIRD_SERVER_REPLY

8

After connecting to an FTP server, the IDLnetURL object received a strange or bad reply. The remote server is probably not an OK FTP server.

FTP_ACCESS_DENIED

9

A service was denied by the FTP server due to lack of access. When a login fails, this is not returned.

FTP_USER_PASSWORD_INCORRECT

10

This is never returned.

FTP_WEIRD_PASS_REPLY

11

After sending the FTP password to the server, an unexpected code was received.

FTP_WEIRD_USER_REPLY

12

After sending a user name to the FTP server, an unexpected code was received.

FTP_WEIRD_PASV_REPLY

13

The IDLnetURL object did not receive a sensible result from the server in response to either a PASV or EPSV command.

FTP_WEIRD_227_FORMAT

14

FTP servers return a 227-line as a response to a PASV command. This code is returned if the IDLnetURL object fails to parse that line.

FTP_CANT_GET_HOST

15

Indicates an internal failure when looking up the host used for the new connection.

FTP_CANT_RECONNECT

16

A bad return code for either the PASV or EPSV command was sent by the FTP server, preventing the IDLnetURL object from continuing.

FTP_COULDNT_SET_BINARY

17

An error was received when trying to set the transfer mode to binary.

PARTIAL_FILE

18

A file transfer was shorter or larger than expected. This happens when the server first reports an expected transfer size, and then delivers data that doesn't match the previously-given size.

FTP_COULDNT_RETR_FILE

19

Either the server returned a weird reply to a RETR command, or a zero-byte transfer was completed.

FTP_WRITE_ERROR

20

After a completed file transfer, the FTP server did not send a proper "transfer successful" code.

FTP_QUOTE_ERROR

21

When sending custom QUOTE commands to the remote server, one of the commands returned an error code of 400 or higher.

HTTP_RETURNED_ERROR

22

This is returned if CURLOPT_FAILONERROR is TRUE and the HTTP server returns an error code that is >= 400.

WRITE_ERROR

23

An error occurred when writing received data to a local file, or an error was returned from a write callback.

MALFORMAT_USER

24

Not used

FTP_COULDNT_STOR_FILE

25

The server denied the STOR operation. The error buffer usually contains the server's explanation.

READ_ERROR

26

There was a problem reading a local file, or the read callback returned an error.

OUT_OF_MEMORY

27

A memory allocation request failed. This is not a good thing.

OPERATION_TIMEOUTED

28

The specified time-out period was exceeded.

FTP_COULDNT_SET_ASCII

29

Failed to set ASCII transfer type (TYPE A).

FTP_PORT_FAILED

30

The FTP PORT command returned an error. This often happens when the address is improper.

FTP_COULDNT_USE_REST

31

The FTP REST command failed.

FTP_COULDNT_GET_SIZE

32

The FTP SIZE command failed. SIZE is not a fundamental FTP command; it is an extension and not all servers support it. This is not a surprising error.

HTTP_RANGE_ERROR

33

The HTTP server does not support or accept range requests.

HTTP_POST_ERROR

34

This is an odd error that mainly occurs due to internal confusion.

SSL_CONNECT_ERROR

35

A problem occurred somewhere in the SSL/TLS handshake. Check the error buffer for more information.

AD_DOWNLOAD_RESUME

36

An FTP resume was attempted beyond the file size.

FILE_COULDNT_READ_FILE

37

A file in the format of “FILE://” couldn't be opened, most likely because the file path is invalid. File permissions may also be the culprit.

LDAP_CANNOT_BIND

38

The LDAP bind operation failed.

LDAP_SEARCH_FAILED

39

LDAP search failed.

LIBRARY_NOT_FOUND

40

The LDAP library was not found.

FUNCTION_NOT_FOUND

41

A required LDAP function was not found.

ABORTED_BY_CALLBACK

42

A callback returned an abort code.

BAD_FUNCTION_ARGUMENT

43

Internal error. A function was called with a bad parameter.

BAD_CALLING_ORDER

44

Not used.

INTERFACE_FAILED

45

A specified outgoing interface could not be used. Use CURLOPT_INTERFACE to set the interface for outgoing connections.

BAD_PASSWORD_ENTERED

46

Not used.

TOO_MANY_REDIRECTS 

47

Too many redirects. When following redirects, IDL hit the maximum amount. Set your limit with CURLOPT_MAXREDIRS.

UNKNOWN_TELNET_OPTION

48

An option set with CURLOPT_TELNETOPTIONS was not recognized.

TELNET_OPTION_SYNTAX 

49

A TELNET option string was malformed.

OBSOLETE

50

Not used.

SSL_PEER_CERTIFICATE

51

The remote server's SSL certificate is invalid.

GOT_NOTHING

52

The server returned nothing. In certain circumstances, getting nothing is considered an error.

SSL_ENGINE_NOTFOUND

53

The specified crypto engine wasn't found.

SSL_ENGINE_SETFAILED

 

Can not set the selected SSL crypto engine as the default.

SEND_ERROR

55

Sending network data failed.

RECV_ERROR

56

Failure in receiving network data.

SHARE_IN_USE

57

Share is in use.

SSL_CERTPROBLEM

58

There is a problem with the local certificate.

SSL_CIPHER

59

Could not use the specified cipher.

SSL_CACERT

60

The peer certificate cannot be authenticated with known CA certificates.

BAD_CONTENT_ENCODING

61

Unrecognized transfer encoding.

LDAP_INVALID_URL

62

Invalid LDAP URL.

FILESIZE_EXCEEDED

63

Maximum file size exceeded.

FTP_SSL_FAILED

 

Requested FTP SSL level failed.

SEND_FAIL_REWIND

65

Sending the data required rewinding the data to retransmit, but the rewind operation failed.

SSL_ENGINE_INITFAILED

66

Failed to initialize the SSL engine.

LOGIN_DENIED

67

The user password (or similar) was not accepted and the login failed.

TFTP_NOTFOUND

68

File not found on TFTP server.

TFTP_PERM

69

There is a permission problem on the TFTP server.

TFTP_DISKFULL

70

TFTP server is out of disk space.

TFTP_ILLEGAL

71

Illegal TFTP operation.

TFTP_UNKNOWNID

72

Unknown TFTP transfer ID.

TFTP_EXISTS

73

TFTP file already exists.

TFTP_NOSUCHUSER

74

No such TFTP user.

The following table lists all the HTTP response codes. Refer to http://www.w3.org/ for detailed specifications.

Code Number

Code

1xx

Informational                        

100

Continue

101

Switching Protocols

2xx

Success                        

200

OK

201

Created

202

Accepted

203

Non-Authoritative Information

204

No Content

205

Reset Content

206

Partial Content

207

Multi-Status

3xx

Redirection

300

Multiple Choices

301

Moved Permanently

302

Found

303

See Other (since HTTP/1.1)

304

Not Modified

305

Use Proxy (since HTTP/1.1)

306

 

307

Temporary Redirect (since HTTP/1.1)

4xx

Client Error                        

400

Bad Request

401

Unauthorized

402

Payment Required

403

Forbidden

404

Not Found

405

Method Not Allowed

406

Not Acceptable

407

Proxy Authentication Required

408

Request Timeout

409

Conflict

410

Gone

411

Length Required

412

Precondition Failed

413

Request Entity Too Large

414

Request-URI Too Long

415

Unsupported Media Type

416

Requested Range Not Satisfiable

417

Expectation Failed

449

Retry

5xx

Server Error                        

500

Internal Server Error

501

Not Implemented

502

Bad Gateway

503

Service Unavailable

504

Gateway Timeout

505

HTTP Version Not Supported

509

Bandwidth Limit Exceeded

The following table lists all the FTP response codes. Refer to http://www.w3.org/ for detailed specifications.

Code Number

Code

110

Restart marker reply.

120

Service ready in nnn minutes.

125

Data connection already open; transfer starting.

150

File status okay; about to open data connection.

200

Command okay.

202

Command not implemented, superfluous at this site.

211

System status, or system help reply.

212

Directory status.

213

File status.

214

Help message.

215

NAME system type.

220

Service ready for new user.

221

Service closing control connection.

225

Data connection open; no transfer in progress.

226

Closing data connection.

227

Entering Passive Mode (h1,h2,h3,h4,p1,p2).

230

User logged in, proceed.

250

Requested file action okay, completed.

257

"PATHNAME" created.

331

User name okay, need password.

332

Need account for login.

350

Requested file action pending further information.

421

Service not available, closing control connection.

425

Can't open data connection.

426

Connection closed; transfer aborted.

450

Requested file action not taken.

451

Requested action aborted: local error in processing.

452

Requested action not taken.

500

Syntax error, command unrecognized.

501

Syntax error in parameters or arguments.

502

Command not implemented.

503

Bad sequence of commands.

504

Command not implemented for that parameter.

530

Not logged in.

532

Need account for storing files.

550

Requested action not taken.

551

Requested action aborted: page type unknown.

552

Requested file action aborted.

553

Requested action not taken. File name not allowed.

Example of Using Widgets with IDLnetURL


The following example creates a widget-based dialog that connects to a OGC WCS server and allows you to download files located on the server. The dialog automatically points to the Harris Geospatial Solutions FTP server, which contains several sample files for download.

See idlneturl_widget.pro, located in the examples/doc/objects subdirectory of the IDL distribution, for the example code listing. View the file in an IDL editor window by entering .EDIT idlneturl_widget.pro. at the IDL Command Line.



© 2018 Harris Geospatial Solutions, Inc. |  Legal
My Account    |    Store    |    Contact Us