6.4 Auto Analysis Processing Steps

6.4.1 Dark Current/Background Stray-light Subtraction

Each observation contains at least two `closed' illuminator flashes. During these illuminator flashes the wheel is set to an opaque position, removing the flux contributions due to the source. This is achieved by placing one of the FPs in the beam with the etalons mis-aligned. For grating observations, at least the first and last flashes in the observation will be closed flashes. For FP observations, all flashes are closed flashes since the FP is already in the beam and the etalons are mis-aligned before the illuminator flash is taken.

The background measurements during these closed illuminator flashes are a measure of the `dark signal' at the time they were taken. (The `dark signal' is the sum of the dark current and straylight). There is a separate background value for each detector. The values for the background measurements for the closed illuminator flashes are calculated using the information in the LIPD file. See section  6.4.9.2 for details of how the background values are calculated. The backgrounds for each illuminator flash in the observation are written into the LIAC file. The closed illuminator flashes in the LIAC file are identified by the wheel absolute position field being set to either 0 (FPS) or 2 (FPL).

Between each pair of closed illuminator flashes in the observation a single darkcurrent/straylight value is calculated for each detector. This is done by taking the mean of the two values from the surrounding flashes. The uncertainty in this value is given by the maximum of the two uncertainties from the surrounding flashes.

The dark current/straylight value is then subtracted from all of the detector photocurrent values between the pair of illuminator flashes. The uncertainty in the photocurrent is calculated by adding the uncertainties in the dark current/straylight and input photocurrent in quadrature.

6.4.2 Grating Scan Wavelength Calibration

The grating mechanism position s (coded as LVDT values in the SPD product file) are converted to wavelengths using a lookup table of LVDT value versus wavelength (calibration file LCGW; see section 8.2.9). In this table for a set of LVDT values the corresponding wavelength for each detector is given. The processing looks up the current LVDT value in this table, and reads the wavelength corresponding to this LVDT value and the current detector.

6.4.3 Grating Spectral Responsivity Calibration

The efficiency of the LWS as a spectrometer varies with wavelength, mainly due to the bandpass filtering incorporated into each detector unit and the spectral response of the detector itself.

The Grating Relative Responsivity Wavelength Calibration File , LCGR (see section 8.2.9), contains a spectrum of the relative response of the instrument in grating mode. The wavelength for each point in the spectrum is looked up in this table, and the corresponding responsivity value read. The responsivity corrected flux then is calculated by dividing the flux by the responsivity value.

6.4.4 Spectral bandwidth correction

The correction is currently only done for grating scans.

The LCGB calibration file contains the spectral element size and uncertainty for each of the ten detectors. Auto Analysis simply divides the flux for each detector by the appropriate spectral element size to perform the correction. The new flux uncertainty is calculated using the standard error formula.

The values of the spectral element sizes and uncertainties are written as keywords into the header of the LSAN file. Keywords LCGBddd contain the spectral element size for detector ddd (ddd='SW1'...'LW5'), while keywords LCGBUddd contain the corresponding uncertainties.

6.4.5 Fabry Perot Scan Wavelength Calibration

The wavelength calibration of a FP scan is done using a parametrised algorithm for the FP wavelength calibration. The wavelength calibration for FP spectra is done as follows:

1. The grating position (LVDT value) is converted to wavelength using the algorithm specified above for the grating scan wavelength calibration (section 6.4.2).
2. For every point in the scan the position value of the FP is converted into a gap of the two Fabry Perot etalons. This is done using the third order polynomial:

   

   where d is the gap of the FP etalons, POS is the FP position value (as stored in the SPD product file (see section 8.2.4), and , , and are the FP wavelength calibration parameters read from calibration file LCFW (see section 8.2.9).

3. The first point of the scan is then used to determine the order the FP was working at for this scan. For this the wavelength determined from the grating position is taken as the approximate wavelength of the first point of the spectrum. The order is then calculated from:

   

   where m is the order of the scan, d is the gap of the FP etalons and is the wavelength. INT means the integer part of this division.

4. Using the order calculated in the previous step, all Fabry Perot gaps for the points in the spectrum are converted to wavelength using:

   

6.4.6 Fabry Perot Spectral Responsivity Calibration

This stage adjusts the flux of each point by the responsivity value at the current FP and grating position. The responsivity values are stored in the LCLRZ, LCSRZ, LCLR_n and LCSR_n files. The LCLR_n files contain the relative responsivity values for a range of relative FP positions for detector n (n=0-9) for FPL. The LCSR_n file contains the same information for FPS. The LCLRZ file contains the absolute FP zero position in the LCLR_n file for a range of grating positions for FPL. The LCSRZ file contains the same information for FPS. This allows the relative FP positions in the LCSR_n or LCLR_n files to be converted into absolute positions. Using these files it is possible to determine the responsivity value for each possible FP and grating position combination.

The correction applied by this stage simply consists of dividing the flux by the responsivity value.

The grating spectral responsivity calibration, described in section 6.4.3, is also applied to FP data.

6.4.7 Velocity correction to wavelength

The wavelengths calculated in the previous stages are corrected for the velocity of the spacecraft and earth towards the target. The header of the LSPD file contains keywords which specify this velocity at three points during the observation. These keywords are written by a subroutine written by ESA which is external to the LWS pipeline. See section 6.3 of the ISO Data Products Document (entitled `General FITS keywords for SPD') for details of these keywords.

The velocity at each mechanism position in a scan is calculated by interpolating in time between the three given values. A second order curve fit is used for the interpolation. Once calculated, the coefficients of this fit are written into the LSAN header as the keywords LVCOEFFn (n=0-2).

The wavelength at each mechanism position is then corrected using the following formula:

Where:

• is the wavelength to be corrected, in microns.
• V is the velocity of the spacecraft and earth towards the target, in Km/sec.
• C is the speed of light in Km/sec.

6.4.8 Write LSNR data product

At this stage the results are written into the first product file produced by Auto-Analysis, named LSNR. This file is identical in structure to the final LSAN file, apart from a few minor differences. This file is provided to give observers access to the data before the absolute responsivity correction and responsivity drift corrections are applied. In a few cases these corrections do not work sucessfully. The LSNR file provides an alternative product file for those cases.

6.4.9 Absolute responsivity correction and responsivity drift correction

The photoconducting detectors in the LWS drift in responsivity owing to the impact of ambient ionising radiation. This drift in responsivity must be corrected for to allow the co-adding of individual scans without introducing extraneous noise. This is referred to as the responsivity drift correction. The spectra must also be corrected to an absolute flux scale. This is referred to as the absolute responsivity correction. The following sections describe how these corrections are performed.

Note that L02 photometric observations are a special case. For these observations no responsivity drift correction is applied. Photometric observations can be identified by the keyword LPHOTOM in the LSAN header being set to `true'. The keyword LORELDN will also be set to `false' to indicate that the responsivity drift correction has not been performed.

6.4.9.1 Grouping of data

Before these corrections are applied, the data must be divided into `groups'. Each group will have a separate responsivity drift correction and absolute responsivity correction calculated and applied. The LGIF, group information file, contains one record for each group. The LGIF file identifies the start and end ITK of each group and also records information which is constant over the group. This includes the absolute responsivity and responsivity drift correction information for the group.

The grouping of data depends upon the AOT type. The easiest way of describing the grouping is to define the condition for the current group to end and a new group to start. A new group starts when:

1. An illuminator flash occurs.
2. A new raster position starts. This is checked for by looking for changes in the raster point ID. However, in the case of solar tracking observations the raster point ID is ignored as it can change even when the raster position is the same.
3. The observation is an L02 or L04 and a new line starts.
4. The observation is an L03 and the grating position changes. A small amount of variation in the grating position is allowed before it is regarded as `changed'. This is because only the grating measured position is available and this is subject to small fluctuations even when at the same nominal position.

Note that there is a special case for L02 photometric observations. In this case all data between each pair of closed illuminator flashes are grouped together.

For each group identified, a single reference time is calculated. This is point at which the absolute responsivity correction will be calculate for the group. It is also the point where the responsivity drift correction will be normalised.

The reference time is simply half way between the ITK of the start and end of the group. This reference time is written into the LGIF file.

6.4.9.2 Absolute responsivity correction

The first stage of the absolute responsivity correction is to process each illuminator flash in the observation. The aim is to find for each flash the ratio between the detector photocurrents from the flash and the reference photocurrents stored in the LCIR calibration file.

Only the `closed' illuminator flashes are used for the absolute responsivity correction. However, all illuminator flashes are first processed using the same method. The results of processing each flash are written into the LIAC file. This file contains one record for each flash in the observation.

The data for all illuminator flashes in each observation are read from the LIPD data file produced by SPL. This file contains the detector photocurrents for each ramp in each flash, plus the illuminator commanded status word and other information.

The first stage of processing an illuminator flash is to determine the background photocurrent for each detector. These backgrounds will also be used in the dark current/straylight subtraction stage (see section  6.4.1). The background value for each detector for each flash is written into the LIAC file.

The method for determining the background is as follows:

1. Extract the set of detector photo-currents in the LIPD file corresponding to the background measurement taken at the start of the flash.
2. Perform median clipping on the set of photo-currents for each detector This is to remove spurious values due to undetected glitches. See below for a description of median clipping. The value of the keyword LCIRNSDB in the LCIR file header is used for the NSD value for median clipping.
3. Average the set of photo-currents for each detector to determine a single background value for each detector.

   The uncertainty to be associated with this value is given by for the set of averaged photo-currents. If there are less than three photo-current values then the maximum of the individual photo-current uncertainties is used.

The purpose of median clipping is to remove any outlying values from a set of measurements of the same value.

There must be at least five values in order for median clipping to be performed.

The method for median clipping is as follows:

1. Calculate the median value of the set of points.
2. Calculate the standard deviation of the set of points, omitting the highest and lowest values in the set.
3. Check each point and reject any that are more than NSD standard deviations above or below the median value. The value of NSD depends upon the data which is being median clipped.

For each illuminator flash a single absolute responsivity ratio is calculated for each detector. This is done by ratioing the photocurrents in the illuminator flash against reference flash data in the LCIR calibration file. The final ratio for each detector is written into the LIAC file.

The method for calculating the ratio for each detector is as follows:

1. Determine the `type' of the illuminator flash for the current observation. The illuminator flash type is determined from the revolution number of the observation. The LCIR file header should describe each of the possible flash types and the range of revolution numbers in which they occur.
2. Locate the start of the data for the appropriate flash type in the LCIR file.
3. Locate the start of the illuminator flash data in the LIPD file. The data from the background measurement at the start of the flash are skipped.
4. For each photo-current value for each detector in the LIPD file, subtract the appropriate background (see section  6.4.9.2), then divide by the corresponding entry in the LCIR file. Continue until no more entries remain in the LCIR file.

   Skip any photocurrent values which are set to zero in the LIPD file or the LCIR file. Skip any values for which the status word in the LCIR file indicates that it should be ignored.

   If while doing this, data is found to be missing from the LIPD file then jump to the start of the next illuminator level in the LIPD and LCIR files. Missing data are detected by a mismatch between the illuminator commanded status value in the LIPD and LCIR records. The warning message `LIMM' is issued each time this occurs. Data may be missing from the LIPD file because of telemetry dropouts or frame checksum errors. The LCIR file should not have any missing data.

5. Perform median clipping on the set of ratios calculated for each detector. This is to discard outliers due to undetected glitches etc. See section  6.4.9.2 for a details of median clipping. The value of the keyword LCIRNSDF in the LCIR file header is used for the NSD value for median clipping.
6. Find the average of the remaining ratios for each detector. The result is a single responsivity correction factor for each detector. The uncertainty for each value is calculated using the standard error formula ( ).

Once all illuminator flashes have been processed the absolute responsivity can be performed.

For each group identified in the LGIF file the absolute responsivity ratio for each detector at the reference ITK time is calculated. This is done using the data from the two closed illuminator flashes which surround the reference ITK. For each detector the absolute responsivity correction factor is calculated by doing a linear interpolation in time between the values at the two surrounding closed illuminator flashes. The value of this correction factor is written into the LGIF file.

The absolute responsivity correction is then performed on all of the data within the group by simply dividing each flux value in the LSNR file by the absolute responsivity correction factor. The uncertainty in each flux value is not changed.

6.4.9.3 Responsivity drift correction

The responsivity drift correction corrects for the `drift' in responsivity during an observation. The drift is obtained from the information in the LSCA scan summary file. The responsivity drift is calculated separately for each group of data identified in the LGIF file.

The LSCA scan summary file contains summary information for every scan in the observation. This includes a value which represents the signal level over the whole scan. This is calculated by finding the average signal per point in the scan. The signal values used are the values from the LSPD file, before any further processing. Any values which are marked as `invalid' in the LSAN status word are not included in this average.

For each scan a reference ITK time is also calculated. This is simply the mid point between the ITK times of the start and end of the scan.

Note that for L02 photometric observations no LSCA file is produced. This is because no responsivity drift correction is needed on this data. Also, since AAL regards each point in a photometric observation as a single scan, the LSCA file would be very large and would contain the same information as the LSAN file.

For each group of data identified in the LGIF file a separate drift slope is calculated for each detector. The method is as follows:

1. Identify the data in the LSCA scan summary file which lie within the ITK time range of the group.
2. Discard any data from the LSCA file which does not correspond to a `full' scan. The last scan in a measurement is often a `short' scan where the mechanism only covers a fraction of the previous `full' scans.

   Short scans are identified by comparing the total number of points (ramps) in each scan with the number of points in the first scan in the group. The first scan of a group is assumed to be a full scan. If the number of points in a scan is below half of the number in a full scan then it is classified as a short scan and discarded.

   Note that the total number of ramps in a scan can also vary because of missing frames of telemetry data.

3. For each of the full scans identified in the LSCA file, for each detector, fit a first order polynomial to the set of average signal values against ITK reference times. This is done using a least squares fitting algorithm. The coefficients of the fitted slope are written into the LGIF file. The coefficients give the LSPD value at the ITK reference time for the group and the gradient of the slope in LSPD units per ITK unit.

Note that in certain cases there may be insufficient valid data to determine a drift slope. This can happen for the inactive detectors in FP observations. The flag LGIFRSTA in the LGIF file indicates when this happens. In this case no responsivity drift correction is performed.

Once the drift slopes have been calculated for each detector in each group the responsivity drift correction can be applied.

For each group identified in the LGIF file the corresponding flux data in the LSNR file are corrected. The method for correction is as follows:

1. Find the Y value of the drift slope for the appropriate detector at the ITK time of the point to be corrected.
2. Divide this by the Y value of the drift slope at the ITK reference time for the group. This gives the relative drift normalised to the ITK reference time of the group.
3. Divide the LSNR flux value by the ratio determined above. The uncertainty in flux value is not changed.