Module Documentation

pdsc.client

This module contains code to support various types of clients that can be used to query PDSC for metadata or find observations coincident with a point on the surface or another point.

pdsc.client.DATABASE_DIRECTORY_VAR = 'PDSC_DATABASE_DIR'

Environment variable that holds the PDSC ingested database directory

See Environment Variables for details.

pdsc.client.PORT_VAR = 'PDSC_SERVER_PORT'

Environment variable that holds the PDSC server port

See Environment Variables for details.

class pdsc.client.PdsClient(database_directory=None)

The PdsClient class handles queries to a local PDSC database for looking up observations based on id or location

Parameters:database_directory – location of the PDSC databases; if None, the PDSC_DATABASE_DIR environment variable is used to determine the database directory

find_observations_of_latlon(instrument, lat, lon, radius=0)

Find observations from a particular instrument that fall within a radius of the given location.

Parameters:

• instrument – PDSC instrument name
• lat – degrees latitude
• lon – degrees east longitude
• radius – query tolerance in meters

Returns:

a list of observation ids corresponding to observations within radius of the given location

>>> import pdsc
>>> client = pdsc.PdsClient()
>>> observation_ids = client.find_observations_of_latlon(
...     'hirise_rdr', -4.5, 137.4, radius=0
... )
>>> observation_ids # doctest: +ELLIPSIS
[u'ESP_018854_1755', u'ESP_018920_1755', ..., u'PSP_010639_1755']

find_overlapping_observations(instrument, observation_id, other_instrument)

Find observations from other_instrument that overlap observations with the given observation_id from instrument.

Parameters:

• instrument – PDSC instrument name for query observation
• observation_id – query observation id
• other_instrument – PDSC instrument name for target instrument

Returns:

a list of observation ids corresponding to observations overlapping the given observation

>>> import pdsc
>>> client = pdsc.PdsClient()
>>> observation_ids = client.find_overlapping_observations(
...     'ctx', 'P09_004477_1906_XN_10N100W', 'hirise_rdr'
... )
>>> observation_ids # doctest: +ELLIPSIS
[u'ESP_015909_1890', u'ESP_016832_1885', ..., u'PSP_007246_1890']

query(instrument, conditions=None)

Supports a generic query of observations based on metadata

Parameters:

• instrument – PDSC instrument name
• conditions –

  a collection of tuples indicating query constraints; each tuple should contain three entries:

  • metadata variable name
  • comparator ('=', '<', '>', '<=', '>=')
  • value

  A SQL-like query will be performed with a logical AND of the specified conditions

Returns:

a list of PdsMetadata objects corresponding to observations matching the specified query conditions

>>> import pdsc
>>> client = pdsc.PdsClient()
>>> metadata = client.query('hirise_rdr', [
...     ('corner1_latitude', '>', -0.5),
...     ('corner1_latitude', '<',  0.5)
... ])

Warning

This function currently assumes non-adversarial inputs; the current implementation allows a potential SQL injection attack.

query_by_observation_id(instrument, observation_ids)

Query observation matching any of the specified observation_ids

Parameters:

• instrument – PDSC instrument name
• observation_ids – either a collection of observation ids, or a single observation id

Returns:

a list of PdsMetadata objects corresponding to observations matching the specified observation_ids

>>> import pdsc
>>> client = pdsc.PdsClient()
>>> metadata = client.query_by_observation_id(
...     'hirise_rdr', 'PSP_005423_1780'
... )
>>> len(metadata)
2
>>> metadata[0].product_id
u'PSP_005423_1780_COLOR'
>>> metadata[1].product_id
u'PSP_005423_1780_RED'

>>> metadata = client.query_by_observation_id(
...     'hirise_rdr', ['PSP_010341_1775', 'PSP_010486_1775']
... )
>>> len(metadata)
4

Note

Some instruments generate multiple data products per observation, os a single observation id might correspond to multiple PdsMetadata objects for each data product.

class pdsc.client.PdsHttpClient(host=None, port=None)

The PdsHttpClient class handles queries to a remote PDSC database over HTTP for looking up observations based on id or location.

The interface for PdsHttpClient is the same as for PdsClient.

Parameters:

• host – the hostname of the PdsServer to query
• port – the port to use for queries

pdsc.client.SERVER_VAR = 'PDSC_SERVER_HOST'

Environment variable that holds the PDSC server host name

See Environment Variables for details.

pdsc.ingest

Injests PDS metadata into databases and index structures for quick querying

pdsc.ingest.CUMIDX_EXT_PAIRS = (('LBL', 'TAB'), ('lbl', 'tab'))

The list of valid cumulative index file extension pairs; the first is the extension of the label file and the second is the extension of the corresponding table file

pdsc.ingest.get_idx_file_pair(path)

Returns the pair of corresponding LBL and TAB files given the path to one file of the pair. The case of the extension does not matter as long as it is consitent (i.e., all upper-case or all lower-case). The corresponding case and path prefix is assumed for the matching file.

>>> get_idx_file_pair('cumindex.lbl')
('cumindex.lbl', 'cumindex.tab')
>>> get_idx_file_pair('CUMINDEX.TAB')
('CUMINDEX.LBL', 'CUMINDEX.TAB')

Parameters:path – path to one LBL or TAB file from a pair Returns:a pair of paths to corresponding LBL and TAB files

pdsc.ingest.ingest_idx(label_file, table_file, configpath, outputdir)

Ingests a PDS cumulative index into PDSC

Parameters:

• label_file – a PDS cumulative index LBL file path
• table_file – a PDS cumulative index TAB file path
• configpath – a configuration file or a directory containing the configuration file with name [instrument name]_metadata.yaml
• outputdir – the directory into which the ingested SQL databases and index structures will be stored

pdsc.ingest.store_metadata(outputfile, instrument, table, config)

Converts and stores metadata into a SQL database file in accordance with configuration, given a PDS cumulative index table

Parameters:

• outputfile – output location for SQL database
• instrument – PDSC instrument name
• table – PdsTable containing parsed metadata
• config –

  dict containing configuration; the entries used in this function are:

  • scale_factors: dictionary mapping PDS cumulative index field names to multiplicative scale factors (e.g., for unit conversion)
  • index: list containing PDSC metadata column names on which to build a SQL index
  • columns: list of lists; each sub-list corresponds to a column and is a triple containing:

    • PDS cumulative index field name
    • PDSC metadata column name
    • PDSC metadata column type (a valid SQL type)

Returns:

a list of PdsMetadata objects associated with every entry in the created metadata table

pdsc.ingest.store_segment_tree(outputfile, segments)

Constructs a ball tree index for segmented observations and saves the resulting data structure to the specified output file.

Parameters:

• outputfile – file to save pickled SegmentTree
• segments – a collection of TriSegment objects

pdsc.ingest.store_segments(outputfile, metadata, config)

Segments observations corresponding to each entry in metadata, and stores these segments in a SQL database

Parameters:

• outputfile – output location for SQL database
• metadata – list of PdsMetadata objects corresponding to observations to be segmented
• config –

  dict containing configuration; the entries used in this function are:

  • segmentation: dictionary containing segmentation-specific parameters:

    • resolution: the maximum size in meters of a side length in the triangular segmentation of an observation; a good heuristic is the average across-track width of an observation to produce isosceles triangles.
    • localizer_kwargs: the kwargs that will be supplied to the get_localizer() function for determining observation footprints

Returns:

a list of TriSegment objects for segments across all observations

pdsc.localization

Contains code that performs “localization,” or the mapping between observation pixel coordinates and real-world latitude/longitude coordinates.

PDSC uses the convention that pixel space coordinates have their origin at the top left of the image, with the positive \(x\)-direction corresponding to increasing columns and the positive \(y\)-direction corresponding to increasing rows. Row and column indices start at 0. The mapping from line/sample to row/column is instrument-specific, but typically lines are rows and samples are columns.

Warning

The localization provided by PDSC is not intended to be the most accurate localization possible, but rather the most accurate localization achievable using only information available within PDS cumulative index files. Therefore, PDSC localization often relies on assumptions that introduce errors whose magnitudes vary across instruments.

class pdsc.localization.CtxLocalizer(metadata)

A localizer for the CTX instrument (subclass of GeodesicLocalizer)

Parameters:metadata – “ctx” PdsMetadata object

BODY = <geographiclib.geodesic.Geodesic object>

Uses a Geodesic model for CTX that assumes Mars is spherical, which seems to work better in practice.

DEFAULT_RESOLUTION_M = 0.001

Sets the default resolution for CTX localization

class pdsc.localization.FourCornerLocalizer(corners, n_rows, n_cols, flight_direction)

The FourCornerLocalizer is a type of localizer that is used when observation locations are described using the latitude and longitude of the four corners of the observation.

Parameters:

• corners –

  an ordered collection of the four image corners, each a latitude and east longitude pair (in degrees); the order of the corners is:

  • top left
  • bottom left
  • bottom right
  • top right

• n_rows – the number of rows in the image
• n_cols – the number of columns in the image
• flight_direction – a multiplicative factor indicating the direction of flight relative to the \(y\)-direction in pixel space; if flight_direction=1, then the flight direction is from the the top down, whereas flight_direction=-1 indicates a bottom-up direction of flight

pixel_to_latlon(row, col)

Converts pixel coordinates to latitude and longitude coordinates within an observation

Parameters:

• row – image row, starting at 0 at the top of the image
• col – image column, starting at 0 at the left of the image

Note

If NORMALIZED_PIXEL_SPACE for this localizer is False, then the pixel coordinates range from zero to one less than the number of total rows/columns in the image. Otherwise, the pixel coordinates range from zero to one along each dimension.

class pdsc.localization.GeodesicLocalizer(center_row, center_col, center_lat, center_lon, n_rows, n_cols, pixel_height_m, pixel_width_m, north_azimuth_deg, flight_direction=1)

The GeodesicLocalizer is a type of localizer that is used when observation locations are described in terms of a center latitude/longitude and a line-of-flight direction. This localizer assumes that along-track pixels in the center column of the observaton roughly follow a geodesic path in the direction of flight and cross-track pixels in each row are perpendicular to this path.

Parameters:

• center_row – the center row of the observation
• center_col – the center column of the observation
• center_lat – the latitude (in degrees) of the pixel at the center of the observation
• center_lat – the longitude (east, in degrees) of the pixel at the center of the observation
• n_rows – the total number of rows in the observation
• n_cols – the total number of columns in the observation
• pixel_height_m – the pixel height (in meters)
• pixel_width_m – the pixel width (in meters)
• north_azimuth_deg – the clockwise angle (in degrees) from a vector that points 90 degrees counter-clockwise from the line-of-flight direction to north
• flight_direction – a multiplicative factor indicating the direction of flight relative to the \(y\)-direction in pixel space; if flight_direction=1, then the flight direction is from the the top down, whereas flight_direction=-1 indicates a bottom-up direction of flight

Note

If NORMALIZED_PIXEL_SPACE for this localizer is True, then the pixel coordinates range from zero to one. Consequently, the attributes n_cols and n_rows are both equal to one, and the attributes pixel_width_m and pixel_height_m are both the width and height of the entire observation.

BODY = <geographiclib.geodesic.Geodesic object>

A Geodesic object describing the target body

location_mask(subsample_rows=10, subsample_cols=25, reinterpolate=True, verbose=False)

Compute a latitude and longitude for every pixel in an observation (with subsampling/reinterpolation to increase efficiency)

Parameters:

• subsample_rows – only compute location once every this many rows
• subsample_cols – only compute location once every this many columns
• reinterpolate – if subsampling is used, reinterpolate values for skipped pixels
• verbose – if True, display a progress bar

Returns:

a numpy.array containing the latitude and east longitude (in degrees) along the last dimension for every pixel in the image, modulo subsampling

Warning

This function is experimental and the reinterpolation step does not correctly handle discontinuities that arise near the “date line” or the poles.

pixel_to_latlon(row, col)

Converts pixel coordinates to latitude and longitude coordinates within an observation

Parameters:

• row – image row, starting at 0 at the top of the image
• col – image column, starting at 0 at the left of the image

Note

If NORMALIZED_PIXEL_SPACE for this localizer is False, then the pixel coordinates range from zero to one less than the number of total rows/columns in the image. Otherwise, the pixel coordinates range from zero to one along each dimension.

class pdsc.localization.HiRiseLocalizer(metadata)

A localizer for the HiRISE EDR observations (subclass of GeodesicLocalizer)

Parameters:metadata – “hirise_edr” PdsMetadata object

CCD_TABLE = {'BG12': -1000, 'BG13': 1000, 'IR10': -1000, 'IR11': 1000, 'RED0': -9000, 'RED1': -7000, 'RED2': -5000, 'RED3': -3000, 'RED4': -1000, 'RED5': 1000, 'RED6': 3000, 'RED7': 5000, 'RED8': 7000, 'RED9': 9000}

A mapping from HiRISE CCDs to pixel offsets from the center of the observation. Each CCD is 2048 pixels across, but they overlap by 48 pixels.

See Figure 2.1.b in the HiRISE EDR SIS.

CHANNEL_OFFSET = {0: 512, 1: -512}

Each HiRISE CCD is split into two channels, each 1024 pixels of the full 2048-pixel CCD. Within a CCD, this dictionary defines a mapping from channel to the offset of the center pixel within that channel.

See Figure 2.1.b in the HiRISE EDR SIS.

DEFAULT_RESOLUTION_M = 1e-06

Sets the default resolution for HiRISE EDR localization

class pdsc.localization.HiRiseRdrBrowseLocalizer(metadata, browse_width)

A localizer for the HiRISE RDR (map-projected) “browse” images (subclass of HiRiseRdrLocalizer)

This classifier is included for convenience; it simply scales the pixel coordinates of the browse image to/from those of the full image before/after calling the super-class implementation.

Parameters:

• metadata – “hirise_rdr” PdsMetadata object
• browse_width – the width of the HiRISE browse image (if it varies from the default value)

HIRISE_BROWSE_WIDTH = 2048

The default width of HiRISE browse images

latlon_to_pixel(lat, lon)

Converts a latitude and longitude location to pixel coordinates within an observation

Parameters:

• lat – latitude (in degrees)
• lon – east longitude (in degrees)
• resolution_m – the resolution (in meters) when optimizing the mapping from latitude and longitude to pixel coordinates; if None, the value defaults to DEFAULT_RESOLUTION_M
• resolution_pix – the resolution (in pixels) when optimizing the mapping from latitude and longitude to pixel coordinates

Returns:

the tuple containing the row and column of the specified location to within the stricter of the two resolution requirements

pixel_to_latlon(row, col)

Converts pixel coordinates to latitude and longitude coordinates within an observation

Parameters:

• row – image row, starting at 0 at the top of the image
• col – image column, starting at 0 at the left of the image

Note

If NORMALIZED_PIXEL_SPACE for this localizer is False, then the pixel coordinates range from zero to one less than the number of total rows/columns in the image. Otherwise, the pixel coordinates range from zero to one along each dimension.

class pdsc.localization.HiRiseRdrLocalizer(metadata)

A localizer for the HiRISE RDR (map-projected) observations (subclass of MapLocalizer)

Parameters:metadata – “hirise_rdr” PdsMetadata object

DEFAULT_RESOLUTION_M = 1e-06

Sets the default resolution for HiRISE RDR localization, although this attribute is not used for the MapLocalizer

class pdsc.localization.HiRiseRdrNoMapLocalizer(metadata)

A localizer for the HiRISE RDR NOMAP observations (subclass of FourCornerLocalizer)

Parameters:metadata – “hirise_rdr” PdsMetadata object

DEFAULT_RESOLUTION_M = 1e-06

Sets the default resolution for HiRISE NOMAP localization

NORMALIZED_PIXEL_SPACE = True

The HiRISE RDR cumulative index metadata does not contain information about the size of the NOMAP data product. Therefore, we must use a normalized pixel space for these observations.

class pdsc.localization.Localizer

Base class for all localizers

Subclasses need only implement pixel_to_latlon() and the reverse translation latlon_to_pixel() is derived from it.

BODY_RADIUS = 3396200.0

Radius of the observed body (defaults to mean Mars equatorial radius)

DEFAULT_RESOLUTION_M = 0.1

Default resolution (in meters) when optimizing the reverse mapping from latitude and longitude back to pixel coordinates via latlon_to_pixel()

NORMALIZED_PIXEL_SPACE = False

If True, pixel coordinates are defined such that (0, 0) is the top left corner of the image and (1, 1) is the bottom right. Otherwise, (0, 0) is the top left and (rows, cols) is the bottom right

latlon_to_pixel(lat, lon, resolution_m=None, resolution_pix=0.1)

Converts a latitude and longitude location to pixel coordinates within an observation

Parameters:

• lat – latitude (in degrees)
• lon – east longitude (in degrees)
• resolution_m – the resolution (in meters) when optimizing the mapping from latitude and longitude to pixel coordinates; if None, the value defaults to DEFAULT_RESOLUTION_M
• resolution_pix – the resolution (in pixels) when optimizing the mapping from latitude and longitude to pixel coordinates

Returns:

the tuple containing the row and column of the specified location to within the stricter of the two resolution requirements

observation_length_m

Total observation length (along-track) in meters

observation_width_m

Total observation width (cross-track) in meters

pixel_to_latlon(row, col)

Converts pixel coordinates to latitude and longitude coordinates within an observation

Parameters:

• row – image row, starting at 0 at the top of the image
• col – image column, starting at 0 at the left of the image

Note

If NORMALIZED_PIXEL_SPACE for this localizer is False, then the pixel coordinates range from zero to one less than the number of total rows/columns in the image. Otherwise, the pixel coordinates range from zero to one along each dimension.

class pdsc.localization.MapLocalizer(proj_type, proj_latitude, proj_longitude, map_scale, row_offset, col_offset, lines, samples)

The MapLocalizer supports map-projected observations. Specifically, it supports the EQUIRECTANGULAR and POLAR STEREOGRAPHIC projection types used for HiRISE observations.

Parameters:

• proj_type – map projection type
• proj_latitude – projection center latitude
• proj_longitude – projection center longitude
• map_scale – projection map scale (meters)
• row_offset – projection row offset
• col_offset – projection col offset
• lines – total observation lines (rows)
• samples – total observation samples (columns)

See https://hirise-pds.lpl.arizona.edu/PDS/CATALOG/DSMAP.CAT for a further description of these parameters.

MARS_RADIUS_EQUATORIAL = 3396190

The Mars equatorial radius used for HiRISE map projections

MARS_RADIUS_POLAR = 3376200

The Mars polar radius used for HiRISE map projections

latlon_to_pixel(lat, lon)

Converts a latitude and longitude location to pixel coordinates within an observation

Parameters:

• lat – latitude (in degrees)
• lon – east longitude (in degrees)
• resolution_m – the resolution (in meters) when optimizing the mapping from latitude and longitude to pixel coordinates; if None, the value defaults to DEFAULT_RESOLUTION_M
• resolution_pix – the resolution (in pixels) when optimizing the mapping from latitude and longitude to pixel coordinates

Returns:

the tuple containing the row and column of the specified location to within the stricter of the two resolution requirements

pixel_to_latlon(row, col)

Converts pixel coordinates to latitude and longitude coordinates within an observation

Parameters:

• row – image row, starting at 0 at the top of the image
• col – image column, starting at 0 at the left of the image

Note

If NORMALIZED_PIXEL_SPACE for this localizer is False, then the pixel coordinates range from zero to one less than the number of total rows/columns in the image. Otherwise, the pixel coordinates range from zero to one along each dimension.

class pdsc.localization.MocLocalizer(metadata)

A localizer for the MOC observations (subclass of GeodesicLocalizer)

Parameters:metadata – “moc” PdsMetadata object

BODY = <geographiclib.geodesic.Geodesic object>

Uses a Geodesic model for MOC that assumes Mars is spherical, which seems to work better in practice.

DEFAULT_RESOLUTION_M = 0.001

Sets the default resolution for MOC localization

class pdsc.localization.ThemisLocalizer(metadata)

A localizer for the THEMIS VIS and IR instruments (subclass of GeodesicLocalizer)

Parameters:metadata – “themis_ir” or “themis_vis” PdsMetadata object

DEFAULT_RESOLUTION_M = 0.001

Sets the default resolution for THEMIS localization

pdsc.localization.geodesic_distance(latlon1, latlon2, radius=3396200.0)

Computes the geodesic distance on a spherical body between two points

Parameters:

• latlon1 – a pair containing the latitude and east longitude (in radians) of the first point
• latlon2 – a pair containing the latitude and east longitude (in radians) of the second point
• radius –

  the radius (in meters) of the spherical body for which distance is computed (defaults to mean Mars equatorial radius)

Returns:

geodesic distance (in meters) between the two given points

>>> import numpy as np
>>> geodesic_distance((0, 0), (0, np.pi))
10669476.970121656

pdsc.localization.get_localizer(metadata, *args, **kwargs)

Get a localizer for an observation corresponding to the provided metadata

Parameters:

• metadata – a PdsMetadata object for an observation
• *args – additional args provided to the localizer constructor
• **kwargs – additional kwargs provided to the localizer constructor

Returns:

a Localizer for the observation

Note

The get_localizer() method determines the appropriate localizer to use for the observation by looking for the class or function that was registered to the instrument using the register_localizer() decorator. See Extending PDSC for more details.

pdsc.localization.hirise_rdr_localizer(metadata, nomap=False, browse=False, browse_width=2048)

Constructs the appropriate HiRISE RDR localizer for the desired data product type

Parameters:

• metadata – “hirise_rdr” PdsMetadata object
• nomap – construct localizer for the NOMAP (non-map-projected) data product
• browse – construct localizer for the BROWSE data product
• browse_width – if browse=True, use this value as the width of the browse image

Returns:

a Localizer for the appropriate data product

pdsc.localization.latlon2unit(latlon)

Converts a latitude, longitude pair into a vector representing that point on a unit circle

Parameters:latlon – a pair containing the latitude and east longitude (in degrees) of a point on a unit sphere Returns:the Cartesian coordinates of the point on a unit sphere

pdsc.localization.register_localizer(instrument)

A decorator that can be used to register a class or function that constructs a localizer to a particular instrument.

Parameters:instrument – PDSC instrument name Returns:decorator that registers target to given instrument

See Extending PDSC for more details.

pdsc.localization.xyz2latlon(xyz)

Converts a point in Cartesian coordinates to the latitude and longitude of that point projected onto a unit sphere

Parameters:xyz – Cartesian coordinates of a nonzero point Returns:latitude and east longitude (in degrees) of the point projected onto a unit sphere

>>> xyz2latlon((0, 0, 1))
array([90.,  0.])
>>> xyz2latlon((1, 0, 0))
array([0., 0.])
>>> xyz2latlon((0, 0, 0))
Traceback (most recent call last):
 ...
ValueError: Point must be nonzero

pdsc.metadata

This module contains classes for representing and serializing PDS metadata

pdsc.metadata.METADATA_DB_SUFFIX = '_metadata.db'

The suffix used to save metadata SQL database files; the full filename for an instrument will be the instrument name followed by the suffix

class pdsc.metadata.PdsMetadata(instrument, **kwargs)

Represents PDS metadata associated with an observation. The metadata available to PDSC corresponds to the metadata in the cumulative index files, which is sometimes a subset of what is available in the EDR/RDR headers.

Parameters:

• instrument – PDSC instrument name
• **kwargs – a dictionary mapping metadata field names to values

class pdsc.metadata.PdsMetadataJsonEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, encoding='utf-8', default=None)

Overrides the json.JSONEncoder class to provide support for serializing PdsMetadata objects and date-valued metadata fields

Constructor for JSONEncoder, with sensible defaults.

If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, long, float or None. If skipkeys is True, such items are simply skipped.

If ensure_ascii is true (the default), all non-ASCII characters in the output are escaped with uXXXX sequences, and the results are str instances consisting of ASCII characters only. If ensure_ascii is False, a result may be a unicode instance. This usually happens if the input contains unicode strings or the encoding parameter is used.

If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an OverflowError). Otherwise, no such check takes place.

If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.

If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.

If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation. Since the default item separator is ‘, ‘, the output might include trailing whitespace when indent is specified. You can use separators=(‘,’, ‘: ‘) to avoid this.

If specified, separators should be a (item_separator, key_separator) tuple. The default is (‘, ‘, ‘: ‘). To get the most compact JSON representation you should specify (‘,’, ‘:’) to eliminate whitespace.

If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError.

If encoding is not None, then all input strings will be transformed into unicode using that encoding prior to JSON-encoding. The default is UTF-8.

default(obj)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

pdsc.metadata.TIME_FORMAT = '%Y-%m-%dT%H:%M:%S.%f'

The time format used by PDSC across all instruments

pdsc.metadata.date_decoder(obj)

Provides an object_hook to parse datetime objects out of JSON, assuming they were encoded using PdsMetadataJsonEncoder

Parameters:obj – JSON object (dict) Returns:obj if no date is detected; otherwise, return a datetime.datetime object

>>> json_str = (
...     '{ "foo": 5, "bar": { "__datetime__":'
...     '{ "__val__": "1985-10-26T01:20:00.000",'
...     '  "__fmt__": "%Y-%m-%dT%H:%M:%S.%f"}}}'
... )
>>> import json
>>> json.loads(json_str, object_hook=date_decoder)
{u'foo': 5, u'bar': datetime.datetime(1985, 10, 26, 1, 20)}

pdsc.metadata.json_dumps(obj)

Dumps a Python object to a JSON string, using the PdsMetadataJsonEncoder to encode objects

Parameters:obj – Python object to encode Returns:JSON string with encoded object

>>> metadata = PdsMetadata('hirise_rdr', rows=100, cols=20)
>>> json_dumps(metadata)
'{"instrument": "hirise_rdr", "rows": 100, "cols": 20}'

pdsc.metadata.json_loads(jstr)

Loads a list of PdsMetadata objects from a JSON string; uses the date_decoder() object hook to parse dates

Parameters:jstr – JSON string with list of PdsMetadata objects Returns:list of parsed PdsMetadata objects

>>> json_loads('[{"instrument": "hirise_rdr", "rows": 100, "cols": 20}]')
[PdsMetadata(instrument=u'hirise_rdr', cols=20, rows=100)]

pdsc.segment

Code for decomposing an observation footprint into triangular segments and storing these in a tree data structure for efficient querying

pdsc.segment.INCLUSION_EPSILON = 1e-10

Numerical precision for checking point inclusion in a segment; this is required due to floating point error, and corresponds to a roughly 1 mm error on the surface of Mars

class pdsc.segment.PointQuery(lat, lon, radius)

Encapsulates the information corresponding to a point inclusion query

Parameters:

• lat – latitude in degrees
• lon – east longitude in degrees
• radius – radius in meters

This query is for all observations that overlap with a circle of the given radius around the specified location on the surface

xyz

The point on a unit sphere expressed in Cartesian coordinates corresponding to the query point

pdsc.segment.SEGMENT_DB_SUFFIX = '_segments.db'

The suffix used to save segment SQL database files; the full filename for an instrument will be the instrument name followed by the suffix

pdsc.segment.SEGMENT_TREE_SUFFIX = '_segment_tree.pkl'

The suffix used to save segment tree index files; the full filename for an instrument will be the instrument name followed by the suffix

class pdsc.segment.SegmentTree(segments, verbose=True)

Encapsulates a ball tree data structure used to efficiently find all observation segments within some radius of a query point

Parameters:

• segments – collection of all observation segments
• verbose – if True display a progress bar as the index is being built

static load(inputfile)

Loads a SegmentTree from the specified file

Parameters:inputfile – path to pickled SegmentTree Returns:parsed SegmentTree object

query_point(point)

Queries the SegmentTree for all segments that potentially overlap the given query point

Parameters:point – a PointQuery Returns:a collection of segment ids for segments that satisfy the query

query_segment(segment)

Queries the SegmentTree for all segments that potentially overlap the given segment

Parameters:segment – a TriSegment Returns:a collection of segment ids for segments that satisfy the query

save(outputfile)

Saves this SegmentTree to the specified file

Parameters:outputfile – output file path for pickled SegmentTree

class pdsc.segment.SegmentedFootprint(metadata, resolution, localizer_kwargs)

Base class for segmenting an observation footprint

Parameters:

• metadata – a PdsMetadata object
• resolution – segmentation resolution (the maximum size of a segment edge)
• localizer_kwargs – the kwargs passed to the localizer used to convert observation pixel coordinates into real-world coordinates

class pdsc.segment.TriSegment(latlon0, latlon1, latlon2)

Represent a single triangular segment into which observations are decomposed for indexing and efficient querying.

The three points of the triangular segment are enumerated in counterclockwise order looking down on the surface. Each point is a pair of (latitude, longitude) in degrees (east longitude).

Parameters:

• latlon0 – represents the first point (index 0) in a triangular segment
• latlon1 – represents the second point (index 1) in a triangular segment
• latlon2 – represents the third point (index 2) in a triangular segment

center()

Computes the center of this TriSegment by converting the latitude and longitude of the vertices to Cartesian coordiantes, taking an average, then converting back to spherical coordinates

Returns:a numpy.array containing the latitude and east longitude (in degrees) of the center of this triangular segment

center_latitude

The center latitude as compuated via TriSegment.center()

center_longitude

The center longitude as compuated via TriSegment.center()

distance_to_point(xyz)

Computes the distance from the given point in Cartesian coordiantes to this TriSegment

If the point falls within the segment, the distance is zero; otherwise, the distance is the minimum geodesic distance between the point and any vertex or edge of the segment.

The geodesic distance between this point and an edge is approximated as the geodesic distance between this point expressed in spherical coordinates and this point projected onto the edge plane (the plane formed by the two vertices of the edge and the origin), expressed in spherical coordinates.

Parameters:xyz – query point in Cartesian coordinates Returns:distance (in meters) between this TriSegment and the query point

includes_point(point_query)

Determines whether the query point falls within this TriSegment

Parameters:point_query – a PointQuery Returns:True iff the query point falls within the specified radius of this TriSegment

is_inside(xyz)

Returns True iff the given vector falls within this TriSegment (i.e., it is on the positive side of every plane defined by TriSegment.normals())

normals

A 3-by-3 numpy.array where each row contains a normal vector to a plane that passes through two of the vertices of this TriSegment and the origin

overlaps_segment(other)

Determines whether the query segment overlaps with this TriSegment

Parameters:other – query TriSegment Returns:True iff the query segment overlaps with this segment

projection_plane

Returns a 2-by-3 numpy.array holding two orthonormal vectors defining a plane that is tangent the unit sphere at this segment’s center

radius

The radius of this TriSegment in meters: the maximum distance from the center to any vertex

xyz_points

A 3-by-3 numpy.array where each row contains a vertix of this TriSegment represented in Cartesian coordinates

class pdsc.segment.TriSegmentedFootprint(metadata, resolution, localizer_kwargs)

Segments a footprint using triangular segments

By using triangular segments, each segment is guaranteed to be convex, so determing whether a point falls within a segment can be done more efficiently than with rectangular segments.

Parameters:

• metadata – a PdsMetadata object
• resolution – segmentation resolution (the maximum size of a segment edge)
• localizer_kwargs – the kwargs passed to the localizer used to convert observation pixel coordinates into real-world coordinates

pdsc.server

Implmements an HTTP server that wraps client for remote calls to PDSC

pdsc.server.DEFAULT_SERVER_PORT = 7372

The default PDSC server port (7372 is P-D-S-C on a numeric keypad)

pdsc.server.DEFAULT_SOCKET_TIMEOUT = 10000

The default PDSC server socket timeout in seconds

class pdsc.server.PdsServer(database_directory=None, socket_host='0.0.0.0', port=7372)

Implements an HTTP server for PDSC using cherrypy

Warning

The PDSC server is not yet designed to be robust against malicious queries. While some care has been taken to properly parse arguments to avoid SQL injection attacks, for example, a thorough review of potential security vulnerabilities has not yet been performed.

Parameters:

• database_directory – location of the PDSC databases; if None, the PDSC_DATABASE_DIR environment variable is used to determine the database directory
• socket_host – specifies the network interface on which the server will listen
• port – specifies the port on which the server will listen

query(instrument, conditions=None)

Serves an interface to PdsClient.query

Parameters:

• instrument – PDSC instrument name
• conditions – JSON-encoded conditions of the kind described for PdsClient.query

Returns:

JSON-encoded list of PdsMetadata objects corresponding to observations matching the specified query conditions

queryByLatLon(instrument, lat, lon, radius=0)

Serves an interface to PdsClient.find_observations_of_latlon

Parameters:

• instrument – PDSC instrument name
• lat – degrees latitude
• lon – degrees east longitude
• radius – query tolerance in meters

Returns:

JSON-encoded list of observation ids corresponding to observations within radius of the given location

queryByObservationId(instrument, observation_ids)

Serves an interface to PdsClient.query_by_observation_id

Parameters:

• instrument – PDSC instrument name
• observation_ids – either a JSON-encoded list of observation ids, or a single observation id string

Returns:

JSON-encoded list of PdsMetadata objects corresponding to observations matching the specified observation_ids

queryByOverlap(instrument, observation_id, other_instrument)

Serves an interface to PdsClient.find_observations_of_latlon

Parameters:

• instrument – PDSC instrument name for query observation
• observation_id – query observation id
• other_instrument – PDSC instrument name for target instrument

Returns:

JSON-encoded list of observation ids corresponding to observations overlapping the given observation

start()

Starts the server; this function will block until the process is interrupted or killed

pdsc.server.content_type(t)

Creates a decorator to set the response Content-Type header for CherryPy response handlers

Parameters:t – content type Returns:content type decorator

pdsc.table

Parses PDS cumulative index files into an internal table representation

class pdsc.table.CtxTable(label_file, table_file)

A subclass of PdsTable for the CTX instrument that uses the custom CtxTableColumn class

Parameters:

• label_file – path to a PDS cumulative index LBL file
• table_file – path to a PDS cumulative index TAB file

COLUMN_CLASS

The CtxTable class should use CtxTableColumn for parsing columns

alias of CtxTableColumn

class pdsc.table.CtxTableColumn(fpointer)

A subclass of PdsTableColumn for the CTX instrument to define some special types

Parameters:fpointer – an open file object, pointing to the start of the column within the PDS index LBL file

SPECIAL_TYPES = {'IMAGE_TIME': <function themis_datetime>, 'SPACECRAFT_CLOCK_START_COUNT': <function ctx_sclk>}

Defines special types for the CTX instrument to parse observation and spacecraft clock times

class pdsc.table.HiRiseEdrTable(label_file, table_file)

A subclass of PdsTable for the HiRISE instrument that uses the custom HiRiseTableColumn class

Parameters:

• label_file – path to a PDS cumulative index LBL file
• table_file – path to a PDS cumulative index TAB file

CHECK_COLUMN_COUNT = False

Ignore the column count discrepancy for the HiRISE EDR table

COLUMN_CLASS

The HiRiseEdrTable class should use HiRiseTableColumn for parsing columns

alias of HiRiseTableColumn

TABLE_OBJECT_NAME = 'EDR_INDEX_TABLE'

The HiRISE EDR table has a custom name

class pdsc.table.HiRiseRdrTable(label_file, table_file)

A subclass of PdsTable for the HiRISE instrument that uses the custom HiRiseTableColumn class

Parameters:

• label_file – path to a PDS cumulative index LBL file
• table_file – path to a PDS cumulative index TAB file

COLUMN_CLASS

The HiRiseRdrTable class should use HiRiseTableColumn for parsing columns

alias of HiRiseTableColumn

TABLE_OBJECT_NAME = 'RDR_INDEX_TABLE'

The HiRISE RDR table has a custom name

class pdsc.table.HiRiseTableColumn(fpointer)

A subclass of PdsTableColumn for the HiRISE instrument to define some special types

Parameters:fpointer – an open file object, pointing to the start of the column within the PDS index LBL file

SPECIAL_TYPES = {'ADC_CONVERSION_SETTINGS': <type 'str'>, 'OBSERVATION_START_COUNT': <function ctx_sclk>, 'OBSERVATION_START_TIME': <function hirise_datetime>, 'SPACECRAFT_CLOCK_START_COUNT': <function ctx_sclk>, 'SPACECRAFT_CLOCK_STOP_COUNT': <function ctx_sclk>, 'START_TIME': <function hirise_datetime>, 'STOP_TIME': <function hirise_datetime>}

Defines special types for the HiRISE observation metadata

class pdsc.table.MocTable(label_file, table_file)

A subclass of PdsTable for the MOC instrument that uses the custom MocTableColumn class

Parameters:

• label_file – path to a PDS cumulative index LBL file
• table_file – path to a PDS cumulative index TAB file

COLUMN_CLASS

The MocTable class should use MocTableColumn for parsing columns

alias of MocTableColumn

class pdsc.table.MocTableColumn(fpointer)

A subclass of PdsTableColumn for the MOC instrument to define some special types

Parameters:fpointer – an open file object, pointing to the start of the column within the PDS index LBL file

SPECIAL_TYPES = {'IMAGE_TIME': <function themis_datetime>, 'PRODUCT_ID': <function moc_observation_id>, 'SPACECRAFT_CLOCK_START_COUNT': <function ctx_sclk>}

Defines special types for the MOC observation metadata

class pdsc.table.PdsColumnType(f)

Wraps a type used for PDS columns to ensure a deterministic representation that omits memory addresses. This is a workaroud for an issue in Sphinx.

>>> f = PdsColumnType(themis_datetime)
>>> repr(f)
'<function themis_datetime>'
>>> f('1985-10-26T01:20:00.000')
datetime.datetime(1985, 10, 26, 1, 20)

Parameters:f – type function to wrap

class pdsc.table.PdsTable(label_file, table_file)

Class for representing and parsing a PDS cumulative index table

Parameters:

• label_file – path to a PDS cumulative index LBL file
• table_file – path to a PDS cumulative index TAB file

CHECK_COLUMN_COUNT = True

Whether to check the number of columns parsed against the number of columns reported in the table metadata; for most instruments, these numbers match, but other instruments have columns with multiple fields so there is sometimes a discrepancy between the effective number of columns and the reported column count.

COLUMN_CLASS

The table column class used to parse columns in this table

alias of PdsTableColumn

COLUMN_OBJECT_NAME = 'COLUMN'

The name of a COLUMN object in the PDS cumulative index label

PARSE_TABLE = {'COLUMNS': ('n_columns', <type 'int'>), 'ROWS': ('n_rows', <type 'int'>), 'ROW_BYTES': ('row_bytes', <type 'int'>)}

Information for parsing the table object out of the PDS cumulative index label file

This dictionary maps the metadata identifier to a tuple containing the name and type of the PdsTable attribute that will be set when parsing this column.

TABLE_OBJECT_NAME = 'TABLE'

The name of a TABLE object in the PDS cumulative index label

get_column(column_name_or_idx, progress=True, cache=True)

Parses all column values out of a PDS cumulative index table

Parameters:

• column_name_or_idx – either an integer column index, or its name as given in the PDS label file
• progress – if True, displays a progress bar as the column is being read
• cache – if True, caches the result in memory so that subsequent calls do not have to read from the file

Returns:

a numpy.array containing values for every row of the specified column

get_column_idx(column_name)

Get numerical column index given column name

Parameters:column_name – PDS table column name Returns:index of column within table (raises IndexError if the column is not found)

class pdsc.table.PdsTableColumn(fpointer)

Class for representing and parsing a column from a PDS cumulative index table

Parameters:fpointer – an open file object, pointing to the start of the column within the PDS index LBL file

PARSE_TABLE = {'BYTES': ('length', <type 'int'>), 'COLUMN_NUMBER': ('number', <type 'int'>), 'DATA_TYPE': ('dtype', <type 'str'>), 'NAME': ('name', <type 'str'>), 'NOT_APPLICABLE_CONSTANT': ('unknown_constant', <type 'str'>), 'START_BYTE': ('start_byte', <type 'int'>)}

Information for parsing table columns; each column contains associated metadata such as the column number, the size of the column in bytes, or the fill value used when an entry is not applicable

This dictionary maps the metadata identifier to a tuple containing the name and type of the PdsTableColumn attribute that will be set when parsing this column.

SPECIAL_TYPES = {}

Contains a mapping from column names with non-standard column types to assocaited Python types; sub-classes should use this attribute to define custom column types for a particular instrument

TYPE_TABLE = {'ASCII_INTEGER': <type 'int'>, 'ASCII_REAL': <type 'float'>}

Contains a mapping of standard column data types to assocaited Python types

class pdsc.table.ThemisTable(label_file, table_file)

A subclass of PdsTable for the THEMIS instrument that uses the custom ThemisTableColumn class

Parameters:

• label_file – path to a PDS cumulative index LBL file
• table_file – path to a PDS cumulative index TAB file

COLUMN_CLASS

The ThemisTable class should use ThemisTableColumn for parsing columns

alias of ThemisTableColumn

class pdsc.table.ThemisTableColumn(fpointer)

A subclass of PdsTableColumn for the THEMIS instrument to override column metadata and define some special types

Parameters:fpointer – an open file object, pointing to the start of the column within the PDS index LBL file

PARSE_TABLE = {'BYTES': ('length', <type 'int'>), 'COLUMN_NUMBER': ('number', <type 'int'>), 'DATA_TYPE': ('dtype', <type 'str'>), 'NAME': ('name', <type 'str'>), 'START_BYTE': ('start_byte', <type 'int'>), 'UNKNOWN_CONSTANT': ('unknown_constant', <type 'str'>)}

Override column metadata, which follows a slightly different convention for THEMIS

SPECIAL_TYPES = {'BAND_NUMBER': <type 'int'>, 'LOCAL_TIME': <type 'float'>, 'SPACECRAFT_CLOCK_START_COUNT': <type 'float'>, 'SPACECRAFT_CLOCK_STOP_COUNT': <type 'float'>, 'START_TIME': <function themis_datetime>, 'START_TIME_ET': <type 'float'>, 'STOP_TIME': <function themis_datetime>, 'STOP_TIME_ET': <type 'float'>, 'UNCORRECTED_SCLK_START_COUNT': <type 'float'>}

Defines special types for the THEMIS observation metadata

pdsc.table.ctx_determiner(label_contents)

Determines whether a cumulative index file is for CTX products

Parameters:label_contents – PDS cumulative index LBL file contents Returns:True iff this label file is for CTX products

pdsc.table.ctx_sclk(s)

Converts the CTX SCLK representation with a colon into a fractional second with a decimal place

Parameters:s – CTX SCLK string Returns:floating-point fractional second

>>> ctx_sclk('10:1')
10.1

pdsc.table.determine_instrument(label_contents)

Determines the PDSC instrument name associated with a PDS cumulative index LBL file

Parameters:label_contents – contents of the PDS cumulative index LBL file Returns:the instrument name corresponding to the first registered “determiner” function that returns True; instruments are checked in alphabetical order by name

pdsc.table.generic_determiner(label_contents, instrument_name)

Determines whether a cumulative index file is for an instrument with the specified name

Parameters:

• label_contents – PDS cumulative index LBL file contents
• instrument_name – instrument name as reported in the cumulative index INSTRUMENT_NAME header

Returns:

True iff this label file is for the specified instrument

This determiner works for cumulative index files that have an explicit INSTRUMENT_NAME header.

pdsc.table.hirise_datetime(s)

Parses date/time format found in HiRISE cumulative index files

Parameters:s – datetime string Returns:datetime.datetime object

>>> hirise_datetime('1985-10-26T01:20:00')
datetime.datetime(1985, 10, 26, 1, 20)

pdsc.table.hirise_edr_determiner(label_contents)

Determines whether a cumulative index file is for HiRISE EDR products

Parameters:label_contents – PDS cumulative index LBL file contents Returns:True iff this label file is for HiRISE EDR products

pdsc.table.hirise_rdr_determiner(label_contents)

Determines whether a cumulative index file is for HiRISE RDR products

Parameters:label_contents – PDS cumulative index LBL file contents Returns:True iff this label file is for HiRISE RDR products

pdsc.table.moc_determiner(label_contents)

Determines whether a cumulative index file is for MOC products

Parameters:label_contents – PDS cumulative index LBL file contents Returns:True iff this label file is for MOC products

pdsc.table.moc_observation_id(s)

Remove the forward slash in MOC observation ids

Parameters:s – MOC observation id Returns:reformatted id

>>> moc_observation_id('FHA/00469')
'FHA00469'

pdsc.table.parse_simple_label(label_contents, key)

Retrieves the value of a “simple” PDS header entry corresponding to the given key. Simple entries are string-valued entries that do not split across lines.

Parameters:

• label_contents – string contents of the PDS LBL file
• key – entry key to search for in PDS label

Returns:

entry value string or None if not found

pdsc.table.parse_table(label_file, table_file)

Parses a PDS cumulative index table

Parameters:

• label_file – path to the PDS LBL file assocated with the cumulate index
• table_file – path to the PDS TAB file assocated with the cumulate index

Returns:

a PdsTable object containing parsed table metadata

This function first uses determine_instrument() to determine the instrument name associated with the label_file. Then, the function looks up the PdsTable subclass that has been registered to the instrument and uses this class to parse the table. See Extending PDSC for more details.

pdsc.table.register_determiner(instrument)

A decorator that can be used to register a function that determines whether a cumulative index file is for a particular instrument.

Parameters:instrument – PDSC instrument name Returns:decorator that registers target to given instrument

See Extending PDSC for more details.

pdsc.table.register_table(instrument)

A decorator that can be used to register a PdsTable subclass to a particular instrument.

Parameters:instrument – PDSC instrument name Returns:decorator that registers target to given instrument

See Extending PDSC for more details.

pdsc.table.themis_datetime(s)

Parses date/time format found in THEMIS cumulative index files

Parameters:s – datetime string Returns:datetime.datetime object

>>> themis_datetime('1985-10-26T01:20:00.000')
datetime.datetime(1985, 10, 26, 1, 20)

pdsc.table.themis_determiner(label_contents, detector_name)

Determines whether a cumulative index file is for generic THEMIS products

Parameters:

• label_contents – PDS cumulative index LBL file contents
• detector_name – THEMIS detector name (either 'VIS' or 'IR')

Returns:

True iff this label file is for THEMIS products with the specified detector

pdsc.table.themis_ir_determiner(label_contents)

Determines whether a cumulative index file is for THEMIS IR products

Parameters:label_contents – PDS cumulative index LBL file contents Returns:True iff this label file is for THEMIS IR products

pdsc.table.themis_vis_determiner(label_contents)

Determines whether a cumulative index file is for THEMIS VIS products

Parameters:label_contents – PDS cumulative index LBL file contents Returns:True iff this label file is for THEMIS VIS products

pdsc.tools

Implements command line tools for PDSC

pdsc.tools.fix_hirise_index(idx, outputfile, quiet)

Repairs HiRISE EDR cumulative index files for which the value in SCAN_EXPOSURE_DURATION exceeds the available number of bytes for that field

Parameters:

• idx – path to one LBL or TAB file from the index file pair
• outputfile – output file path for the corrected index TAB file; if None, overwrite the existing file
• quiet – if True, do not output progress or results

pdsc.util

Miscellaneous utilities

pdsc.util.registerer(registration_dict)

Creates a decorator that will “register” a method or class to a particular instrument. The registration decorators are used to extend PDSC to other instruments.

Parameters:registration_dict – a dictionary to which the decorated function will be registered Returns:registration decorator that assigns decorated function or class to the registration_dict keyed by a particular instrument, which is a required decorator argument

pdsc.util.standard_progress_bar(message, verbose=True)

Optionally constructs a standard progressbar.ProgressBar used through PDSC

Parameters:

• message – progress message to display
• verbose – whether to display progress

Returns:

a progressbar.ProgressBar object if verbose=True, or an identify function that can be used to omit progress