Appendix A: Data Structures and Files, an Overview

The PHT Interactive Analysis is not only designed to be able to reproduce all steps of the PHT Standard Products Generation (SPG), but also to provide users with all the versatility of a fully interactive data analysis package. In this sense several intermediate and further processing steps have been added to the steps defined for the SPG. This is not only reflected in the architectural design which varies quite a bit from the design of the SPG s/w, but also in the necessary file and data structures.

PIA recognizes the following types of data: Buffers, External FITS Product files, External FITS PIA files, Internal PIA files, Calibration Files (CAL-G, CAL-U, and CAL-Q), PIA Calibration Tables and PIA Configuration files. These files, data structures and their different flavours, will be described below.

PIA Buffers

While active the PHT Interactive Analysis software stores all information into "Buffers". These buffers are IDL structures (collection of scalars and arrays) which contain all the information that PIA requires in order to perform its task. IDL stores these structures in the memory of the computer that is running PIA.

The buffers are organized per processing level, and contain the data on a measurement basis, which is the PIA processing unit (thus differing to the SPG). A PHT-measurement is defined as a continuous observation of one sub-instrument configuration per raster point. Since the PHT sub-instruments have different data contents (given mainly by the different number of pixels) and the measurement have different lengths (thus producing different lenghts of data arrays), the only way to store them in IDL structures is by using pointers to the data variables. Every time data is read in or produced by reducing data from a lower level, the corresponding measurements are piled up to the existing ones on the corresponding level.

This has the advantage that correlations, comparisons, operations between different data sets are easily accessible and several capabilities are given to make use of this. Furthermore, after a data set has been read by PIA there is no further disk i/o and consequently all processing is very fast. It has, however, the disadvantage that PIA is "memory hungry", which might cause problems on computers with a small amount of RAM. To avoid such problems, it is possible to delete the contents of the PIA buffers at any time in order to release the memory again, after a certain processing step has been finished (cf. "Delete Buffers"). All buffers are automatically deleted, when the IDL session is closed and the blocked memory is released. The buffers are not deleted, however, if only PIA is quit. The buffers can then be made available under the standard IDL environment for manual analysis, which can be very useful for experienced users. In addition, because the information is not lost while IDL remains active, it is possible to re-enter PIA and continue the work. Re-entering PIA after the buffers have been modified using the IDL command interface is only recommended to real PIA and IDL experts as the results can be unpredictable, if all side effects are not properly taken into account.

In order to analyse and display the contents of the various PIA buffers a dedicated tool was written, which is called context sensitive from within PIA. For details on how to use this tool please refer to "The Show Structures - Plot Curves Menu". The details of the data structure definitions used within PHT Interactive Analysis:

are listed in "Appendix B: Internal PIA Buffer Structures".

External FITS Files

In order to ensure compatibility with other astronomical s/w packages, all ISO product files follow the Flexible Image Transport (FITS) standard. The only exceptions to this rules are those files which are used only internal to the ISO ground station, which are the Uplink calibration files (CAL-U) and the QLA/RTA calibration files (CAL-Q). The following file types are used and/or produced by PIA:
Data Products, i.e. the data products available in the ISO archive. These products are generated by the pipeline software and contain all information from the spacecraft and the instrument in its various stages of processing. For the user the most important filetypes are ERD, SPD and AAP, comprising the main instrument data files. All these stages of processing can be read by PIA for further processing and their generation by the pipeline s/w can be emulated using PIA. There are various auxiliary files containing e.g. the s/c pointing information AOCS data (IIPH & IRPH), data from the ISOPHOT housekeeping data channel (GEHK) or instrument status data (CSH). These files are automatically read by PIA, when the related ERD, SPD or AAP product is addressed. In this respectr the user does not have to bother with their filename conventions.

Calibration Files, there are four types of calibration files for PHT: FCS calibration files (CAL-A), Uplink Tables (CAL-U), RTA/QLA calibration files (CAL-Q) and General (pipeline) calibration files (CAL-G). CAL-A and CAL-G files are products and consequently follow the FITS standard. All calibration files can be generated or updated using PIA. The data analysis PIA is performing is based on the pipeline CAL-G files or on results obtained with PIA itself. The FCS calibration files (CAL-A) are generated by the pipeline software and can be used by PIA to perform responsivity calibrations. The Uplink and RTA/QLA files (CAL-U and CAL-Q) are not in FITS format, but in a dedicated format used by the related s/w systems. These files are not used by PIA for calibrations purposes, but are read for checking purposes and also derived by PIA for inclusion into the SOC software.

The detailed description of all product files is given in the IDPD [2]. For a good understanding of the PHT Interactive Analysis software the user is highly recommended to read this document to get familiar with the main product types and their contents. For a list of the available products and filenames, please refer to "Appendix E: Naming Conventions".

External PIA FITS files

PIA not only allows the standard product files to be handled as described in the previous section, but also offers the possibility of storing measurements on intermediate reduction levels. These data files are called External PIA files. The format chosen for these files is FITS.

Main files of this type are the Signal Reduced Data (SRD) and the Signal per Chopper Plateau Data (SCP). Both data types represent intermediate steps of processing between ERD and SPD pipeline products. It was found important to be able to keep these intermediate results, because the reduction step from ERD to SRD is by far the most CPU intensive one and considerably reduces the amount of data to be handled. In addition, the SRD and SCP levels of data processing allow the user to make a better judgement about the quality of the data than ERD and SPD level, for example w.r.t. drift behaviour, influence of glitches on signals, etc.

PIA SPD products can also be stored into FITS files. Although they are on the same level as SPD pipeline products, there are fundamental differences between these products. PIA products are measurement-based, e.g. one measurement on one file, while the SPG PHT-SPD products are per observation and detector, generally containing several measurements. The reduction parameters and corrections used for the derivation of the Pipeline products have to be extracted from the documentation of the used version, while the derivation of PIA products are documented on the file header itself by keywords, then they can be freely chosen by the user at any time. For an explanation of all the possible keywords please refer to "Appendix B: Internal Buffers Structures Definitions"

Also of fundamental importance, but for different reasons, is the possibility of storing highly reduced data in FITS format, on the 'Astrophysical Applications' level. This allows exporting data from PIA into other s/w packages for doing special processing, not contained in PIA. Up to now, only the possibility of producing mapping FITS files is given, but the future upgrades concerning polarimetry and (spectro-)photometry will be accompanied by an upgrade on this sector.

Nomenclature

Although PIA offers the user complete freedom to choose names for the files to be saved, as a first choice a fixed nomenclature is proposed. The use of this nomenclature facilitates searching for file types of a certain reduction level and detector, when using the PIA menues. Since every file represents one measurement on a certain processing level the nomenclature chosen is unique for every measurement which can be performed while ISO is operating on every processing level contained in PIA.

DDmmTTTTTTss.LLL is the general representation, where "DD" stands for detector ('P1', 'P2', 'P3', 'C1', 'C2', 'SS' and 'SL') followed by "mm" as the position of the measurement in the observation sequence for one detector (counting from 00 upwards). "TTTTTT" is the Target Dedicated Name (TDT) composed of 3 first digits for the orbit number and 3 last digits for the position within the orbit. "ss" is the sequence number as specified by the observer by observation planning. LLL finally gives the reduction level ('SRD', 'SCP', 'SPD', 'AAP', 'MAP'). Thus, a filename 'P20103700122.SRD' would represent the second (!) measurement with P2 on the first target during revolution 37. The number 22 could indicate a position within the proposal and the processing level is SRD, signal per ramp. Please note that a measurement with PHT-S is always an SS/SL measurement and up to the SPD level the information from all SS and SL detector pixels is kept together. In this case there are only 'SL...' files on the SRD and SCP level. On the SPD level they are separated (to correspond fully to the SPG definitions), thus 'SS....SPD' and 'SL....SPD' files exist.

Internal PIA files

The difference between External and Internal PIA files is meant with respect to the exportability to other systems and not to the fact of their physical independence from the s/w (the data is actually kept internally to PIA in "buffers"). Thus, internal or external files refers only to the format of these files. Internal files are not meant to be exported to other s/w systems and are consequently stored in a format that can only be read by IDL (full interpretation and accomodation within special buffers can be done only by the PHT Interactive Analysis). The advantage of this format is that it is much faster to write and read by means of PIA than the external FITS files and that these files use less disk space. This is especially important for large raw data files.

Nomenclature: for every External PIA Data file the corresponding internal file exists. The proposed nomenclature to the user (since he/she also has the freedom here to change the name) has got the same prefix (as described above for External PIA Files), but the suffix contains the letter "I" to indicate the internal format, e.g. *.ISCP would be an internal Signal per Chopper Plateau file.

PIA Configuration Files

In order to work properly PIA needs information about the PHT instrument, e.g. the specific details of the instrument model (FM or FS) providing the data. This information is provided to PIA via "Configuration" files. Currently all information required by PIA is contained in one file "pht_fm.idb_def" the flight model instrument data base file. It contains for example all information about data conversion (ADC reads), fixed calibration curves (Temperatures), constant parameters, etc. characterising the PHT Flight Model instrument. In order to work properly it is necessary for PIA to have this file on-line. There is no update planned for this file as it contains fixed information about the current PHT instrument. Only when analysing data from the Flight Spare model of PHT, must a different configuration file be used with PIA to ensure proper data conversion.

PIA Calibration Tables

NOTE: The calibration tables are only used in the 'full' PIA, the package used by experts for deriving calibration files, as is done in VILSPA during operations and in several associated institutes. In the PIA 'export' version the calibration menues are disabled, and so the calibration tables are not necessary.

For calibration purposes the PHT Interactive Analysis keeps the main results of several or all in-orbit calibration measurements in little "data bases" for every type of calibration observation. The files in which these data are stored are called "Calibration Tables". As these files are written and maintained only by PIA, they are kept in an internal format which is not compatible with other systems. As these database files are usually common to all users of a PIA installation on a specific site, strict configuration control procedures must be observed when updating these files via CFDPs.


 
Chapter history:
Date  Author  Description 
15/06/1996  Carlos GABRIEL (ESA/VILSPA-SAI)  First Version 
20/07/1996 Carlos GABRIEL  Update
10/06/1997 Carlos GABRIEL  Update (V6.3)