Guide to ISO Data Products

Last Update: 08 June 2004

The ISO Data Archive contains various sets of data products that the user can download for independent scientific analysis. These are the pipeline products, that are kept on disk at the ISO Data Center in Villafranca, Spain and accessible through the ISO Data Archive interface. The pipeline products are subject to a scientific validation by the Instrument Teams. In order to facilitate browsing through the archive and identifying data to download, the so-called browse products are generated by an additional software.

In addition, the ISO Data Archive contains the so-called 'Highly Processed Data Products' (HPDP). These products include DATA (images, spectra etc.), which have been processed beyond the pipeline and/or using new, refined algorithms and therefore have been improved to any degree compared to the pipeline products, as well as any resulting CATALOGUES and ATLASES.

Pipeline Products

Every observation is run through an automatic data-analysis pipeline called Off-Line Processing, or OLP, to produce three sets of data products, namely: The various types of analysis performed are described, as appropriate, in the ISO Handbook.

Detailed technical descriptions of the contents and formats of data product files are given in the ISO Data Product Document and in the corresponding sections of the ISO Handbook.

The type and sophistication of analysis performed for each product set is instrument dependent. In the Archive, products are further classified according to the following scheme:

- Raw Data Products
These are essentially unpacked telemetry in which no data reduction has taken place and thus form the starting point for data analysis performed by OLP or the user's own system. PHT ERD is an example.
- Basic Science Data Products
These data have been processed further to an intermediate level (with the use of calibration files where necessary), often containing physical units. They are therefore ready for assembly into structures of astronomical significance such as images or spectra by OLP or independently by observers themselves before scientific conclusions can be drawn. For example, the final result of the LWS pipeline is the LSAN file that contains a tabulation of flux in physical units measured during the observation.
- Fully Auto-processed Science Data Products
These data include a set of coherent, instrument-independent measurements of images or spectra designed to get as close as possible by automatic means to what could be produced by an astronomer using an interactive analysis system. Where possible, products follow FITS standard conventions to ease further analysis by standard high-level tools such as IDL or MIDAS. They allow the user to carry out initial survey programmes or to make an assessment of the relevance of a given observation to their scientific work, although the limitations must be emphasised of such an automatic procedure which takes no account of the individual circumstances of an observation and involves no scientific judgement. Significantly better results are likely to be possible using an interactive system. Nevertheless, the CAM pipeline for example produces individual FITS image in the CMAP file, that are combined where appropriate into mosaics in the CMOS file; gives a tabulation of sources detected in the images in the CPSL file; and gives individual source spectra as appropriate in the CSSP file.
For the spectrometers, LWS and SWS, the AAR products contain basic science data, for CAM and PHT these are the SPD products. An overview of the files per product level for each instrument can be found in list of files per product level. Each observation has a Data Quality Report including the quality information available.

The pipeline products undergo a scientific validation to establish confidence in the processing and calibration of the products. In the documents about the accuracy of pipeline products an overview of the calibration status, the accuracies of the current off-line products and the improvements one may expect with interactive analysis (IA) are given. The scientific validation per AOT summarizes the current overall status.

Browse Products: Icons, Postcards and Survey Products

From the standard pipeline products a number of products are generated to provide users with:
Survey Products
These are fully reduced standard data sets, either FITS images or ASCII FITS tables, for survey-type work. Although these products contain fully reduced data, it must be emphasised that the processing is done in a standard and automatic way which does not involve any scientific judgement. These products may serve for statistical or survey-type analysis of large samples.

Icons and Postcards
These are static representations in GIF-format of the survey products. Icons are small images intended to give an impression of the data: photometry, an image or a spectrum. Postcards are essentially enlarged icons with annotations added to give users an impression of the flux levels and wavelengths covered. Icons and postcards facilitate a quick-look to scan the data for their particular purposes and identify which data need to be retrieved. These should never be used for scientific work. The postcards are also used to give a quick look impression of the ISO data for a given observation when browsing through ISO information from other archives/databases, in the framework of the Virtual Observatories, within the so-called Postcard Server. Through calling a URL / Java Server Page (JSP) containing the ISO observation identifier (so called TDT number), it returns the ISO postcard (GIF image) of this observation and ancillary information embedded into an HTML page. The information embedded in the Postcard Server is dynamically generated via direct queries to the IDA database. This currently includes information related to the observation technical quality and the status of validation of the observation mode used. Links are also provided to specific sections of the ISO documentation (observation mode description and associated pipeline products accuracies and caveats). When relevant, specific caveats text is dynamically assembled depending on the observation parameters.

A short description follows of the main, instrument-dependent, characteristics of the browse products.

CAM
- Survey Products
These products are the equivalent of the AAR, giving the image from the CMOS file when it exists and CMAP otherwise.
- Icons and Postcards
The postcard is the image of the survey product projected in RA,DEC coordinates. It contains a grey-scale coded wedge to indicate flux levels. The icon is a small version of the CMAP/CMOS image shown in detector coordinates. For the CAM spectral observations (CAM04) the mean image of all wavelengths is displayed; the spectrum shown is that of the innermost 10x10 pixels. Up to four multi-filter or multi-PFOV measurements (for a given observation) are shown inside the corresponding icon.
LWS
- Survey Products
The following processing steps are applied to the standard AAR LSAN file:
  1. remove all data points with bad status
  2. clip outliers
  3. remove individual scans which are inconsistent with the majority of the scans
  4. for each detector the remaining scans are averaged
It should be noted that the resulting spectrum is not stiched together nor is it averaged across detectors. Also, no defringing is applied.
- Icons and Postcards
Postcard and icon are spectra. In the case of a raster map only the spectrum of the central point is presented.
PHT
- Survey Products
The survey product is the equivalent of AAR. No survey product is derived for polarization observations.
- Icons and Postcards
Depending on the observing mode, postcards and icons are:
SWS
- Survey Products
The following processing steps are applied to the standard AAR file:
  1. remove all data points with bad or unreliable status
  2. flatfield the data such that all detectors are scaled to the same level
  3. clip outliers using sigma clipping
  4. rebin the data to a grid with the nominal resolution
- Icons and Postcards
Postcard and icon are spectra which are presented in a mosaic form when the observation contains more than one wavelength range.

Highly Processed Data Products (HPDP)

The Off-Line Processing Pipeline copes well with a number of instrumental artifacts in an automatic fashion. The final products can however be improved by processing them further, in particular by means of the Interactive Analysis software packages, as documented in the ISO Handbook.

Many ISO papers are based on a systematic reduction of archive data, producing 'Highly Processed Data Products' (HPDP). These products include DATA (images, spectra etc.), which have been processed beyond the pipeline and/or using new, refined algorithms and therefore have been improved to any degree compared to the OLP 10 products, as well as any resulting CATALOGUES and ATLASES.

To query and download these data, access the ISO Data Archive and look at the specific help sections, at the top-right help menu or directly clicking on the "Highly Processed Data Products" or "Catalogues" panels.

Projects have been undertaken by the ISO Data Centre, in collaboration with the national instrument data centres, for systematic data reduction of specific instrument modes, that produce homogeneous sets of HPDP. We also invite and encourage your participation in the enrichment of the contents of the ISO Data Archive by providing the results of your personal ISO data reduction for ingestion as Highly Processed Data Products. If you are interested, just send an e-mail to the ISO helpdesk with a short description of your data set. The ISO Data Centre will contact you shortly after and take care of the details of the ingestion.