The inputs to Derive SPD (SPL) are the LSTA file, science ERD files, the housekeeping ERD file (LWHK), the Executed observation history files (EOHI and EOHA), and various calibration files. The science ERD files consist of LGER for grating scan data, LLER for FPL scan data, LSER for FPS scan data, and LIER for illuminator flash data.
The outputs of SPL are the standard processed data file (LSPD), the illuminator processed data file (LIPD), and the glitch history file (LWGH).
SPL is driven by the contents of the Compact status history file (CSH) for the selected observation. The LWS CSH file is named LSTA. The LSTA file identifies the regions of data taken in an observation with the grating, FP, or illuminator.
The first stage of SPL reads in all records from the currently open science ERD file that correspond to one ramp of data for all ten detectors. The LSTA file specifies which of the science ERD files the data is read from.
The start of a ramp is indicated by a detector readout which has its most significant bit set. The expected number of number of readouts per ramp is then read in. The expected number of readouts per ramp is obtained from the housekeeping ERD file (LWHK).
The ITK time key of each readout is checked as it is read in to identify any periods of missing data and to adjust the ramp contents appropriately.
After the ramp has been read in, some of the readouts have to be discarded for the following reasons:
It has been found that, for some detectors, the length of time needed to recover from a reset pulse is longer at low signal levels. The LCDT file therefore contains two discard times for each detector. The selection between the two times is made using the length of the ramp, assuming that longer ramps are used for low signal levels. See section 3.2.1.1 for details.
The number of points discarded for the above reasons are written as keywords into the header of the LSPD file. See section 8.2.4 for details.
Before the raw detector readouts are converted into voltages, any invalid points are discarded. Any points which are outside the valid range for the analogue amplification chain are discarded. The valid range is specified in the LCAL calibration file. Note that this is NOT the same as the saturation of the detector, which is corrected later in the processing chain.
The number of readouts discarded for this reason are written as keywords into the header of the LSPD file. See section 8.2.4.
For each readout the raw detector readout (in digital units; DN) the conversion to voltages is performed using the formula:
Where:
is an offset to the detector readout value.
is the analogue amplifier gain factor for the current
gain level and detector. The gain level is in the range 0-7. The level is
stored in bits 12-14 of the detector word.
can be found in calibration file 
LCVC
(see section 8.2.6.3 for details).
The gain for the analogue amplification
chain is read from calibration file LCGA (see
section 8.2.6.4) using
the gain level (0-7) read from the detector word.
Finally the voltage as derived using formula 6.1 is divided by the
gain factor of the JF4 for the appropriate detector to reconstruct the
voltage at the input of the JF4's. The JF4 gain factor can be found in
calibration file LCJF (see section 8.2.6.5).
As described in section 4.2, the present method of extracting the
detector signal involves fitting a second order polynomial to the output voltage
of the integrating amplifier; the coefficients of this fit
are then interpreted in terms of a second order approximation to the
behavior of the detector circuit. At a certain level 
of output
voltage, this approximation breaks down and all voltages above this
threshold are dicarded. The array of values for can be found
in the calibration file LCDB (see section 8.2.6.6).
As also explained in section 4.2, improved methods of extracting
the signal are being sought and, as a result, outputs above the threshold
may be processed in future.
The number of readouts discarded due to saturation are written as keywords into the header of the LSPD file. See section 8.2.4.
Glitches are caused by the effects of cosmic ray particles on the detectors. There is roughly one glitch per ten seconds per detector during the normal period of LWS operation. These energetic particles cause a sudden jump in the ramp voltage, due to a quantity of charge being dumped on the integrating amplifier. They also cause a change in the detector responsivity which affects the following ramps.
"Slow" glitches are glitches where the jump in voltage covers more than one readout value.
In addition to these "positive" glitches, "negative" glitches have also been found. These cause a sudden decrease in the ramp voltage, rather than an increase. They are thought to be due to hits on the FET. Negative glitches do not appear to affect the detector responsivity.
Before launch it was anticipated that "spikes" may also need to be located an removed. These are caused by spikes in the analogue amplification chain. They cause a single detector readout to have a much larger value than normal. Subsequent readouts are unaffected and there is no effect on subsequent ramps. However, no real spikes have yet been seen in the data since launch. The spike removal has been switched off as all of the "spikes" detected were actually caused by the effects of glitches. A modified spike detection is still operational, but it would be more accurate to describe it as an "anomaly" detector, rather than a spike detector. This anomaly could be cause by a real spike, but it is more likely to be caused by undetected glitches, or the effects of glitches which have been detected. These points will be examined to help improve the detection of real glitches. If any real spikes are found then it may be necessary for a modified spike removal stage to be added.
Statistics related to glitches and spikes are written into the header of the LSPD file. See section 8.2.4 for details.
The following list describes how glitches and "spikes" (see introduction) are detected. Note that glitch detection is only performed on the section of ramp after the discarding of data due to the reset pulse etc. Any glitches which occur in this discarded section of ramp are not currently detected. This problem may be addressed in future versions of SPL.
LCD1 calibration file.If a glitch is detected by this step then the next three points are not checked for glitches. This is because it has been found that the effects of a glitch often caused a second, false glitch, to be detected shortly afterwards.
No "spike" detection is done for the remainder of a ramp following a glitch. This is because it has been found that the effects of glitches caused lots of false "spikes" to be detected.
The height of a spike is estimated by subtracting the voltage of the previous point from the voltage of the spiked point. The expected voltage increment due to the ramp slope is then subtracted from this value.
There is a special case for the first point in the ramp, since there will be no previous point. In this case the spike height is obtained by subtracting the voltage of the point following the spike from the voltage of the spiked point. The expected voltage increment due to the ramp slope is then ADDED to this height.
The height of a glitch is estimated by finding the difference between the point at the glitch location and a point a few places ahead (3, at the time of writing). This is to cope with slow glitches, or glitches that have noise. If the second point is beyond the end of the ramp then the last point in the ramp is used.
The expected nominal ramp increment over the time period between these two points is calculated and subtracted from the glitch height.
For spikes the fractional height with respect to the height
of the ramp is calculated. The height of the ramp is simply the
voltage of the last point in the ramp minus the voltage of the first
point in the ramp. Only those spikes with fractional heights above the
threshold specified in the LCD1 calibration file are accepted.
For glitches the same procedure is performed, except that the
glitch height is also subtracted from the height of the ramp.
This should give the height of the ramp if no glitch has occurred.
There is a separate threshold level specified in the LCD1 file for
the fractional heights of glitches.
Note that these calculations have assumed that there is only one spike or glitch in the ramp.
Note that the `location' of a glitch is understood to mean the point before the outlying point(s):
*-*-*
/
*-*
^
Glitch location
Using test data it has been found that `slow' glitches are often detected only on the second point of the glitch:
*-*
/
* <--------- Glitch detected here
/
*-*
In order to cope with this, the point at the glitch location is always discarded. For normal `fast' glitches this will mean that one possibly good point is thrown away.
This section details the patterns which identify spikes and glitches. The following conventions are used:
Positive glitch at point n in ramp
==================================
ramp point n-1 n ramp point n-1 n
OUT1 * 1 OR OUT1 * 1
OUT2 1 * OUT2 * 1
The second of these checks tends to catch the `slow' glitches, which cover more than one point.
For the first point in the ramp only the second of these test for positive glitches is done, as there are no previous points to check.
Negative glitch at point n in ramp
==================================
ramp point n-1 n ramp point n-1 n
OUT1 * -1 OR OUT1 * -1
OUT2 -1 * OUT2 * -1
No check is done for negative glitches at the first point in the ramp. Negative glitches at the first point in the ramp will be recorded as a positive "spike". It remains to be checked whether this is the correct thing to do.
If the number of points in a ramp is NPOINTS, then these checks for glitches can only be done for n=1 to NPOINTS-2, as there are only NPOINTS-2 values for the second differential. This means that if the last point of a ramp is an outlier, then it will be reported as a spike, rather than a glitch. There is no way of telling the difference between a spike at the last point and a glitch.
Positive spike at point n in ramp
=================================
ramp point n-1 n
OUT1 1 -1
OUT2 * *
There is a special case for the first point in the ramp, as there is no previous point to check. In this case if OUT1 is -1 (ie. a negative outlier) then this is regarded as a positive spike at this point.
Negative spike at point n in ramp
=================================
ramp point n-1 n
OUT1 -1 1
OUT2 * *
It is not possible to distinguish between a negative spike at the first point in the ramp and a positive glitch at this point. Therefore, no check for negative spikes is performed for the first point. A negative spike at the first point will be reported as a positive glitch.
The glitches identified using the method described above are now removed. The way in which this is done is controlled by the values in the LCD1 calibration file. Note that the removal of glitched data is done after all glitches have been identified. This means that glitches which occur during ramps discarded because of glitches in previous ramps are still identified. The following description is based on the current contents of the LCD1 calibration file (version 8). If the LCD1 file is updated then this description may no longer be valid. Check the LCD1 file if you are in any doubt.
For positive glitches all of the ramp following the glitch is discarded, plus the two subsequent ramps. The section of ramp before the glitch occurred is still used, provided that it has at least the minimum number of points required for slope fitting (this value is specified in an include file and is currently set to 10).
Negative glitches are handled in the same way as positive glitches, except that no ramps are discarded following the glitched ramp.
The deglitching performed during illuminator flashes is slightly different from the above description. See section 6.3.7 for details.
The LSPD file also contains the "undeglitched" data, ie. the results when there is no discarding of data due to glitches.
As described in section 4.2, the signal currents are currently extracted by fitting a first or second order polynomial to each ramp for each detector. The longer the ramp or the higher the signal level, the more the ramp tends to deviate from a straight line.
The order of the fit is selected using two criteria, the ramp length and the ramp height. Ramps longerthan four seconds are always fitted using a second order polynomial. For ramps of less than four seconds a second order fit is used only if the height of the ramp exceeds a certain threshold level for that detector. These levels are specified in the LCDB calibration file in terms of the debiasing parameters, which are also specified in this file.
The numbers of first and second order fits performed are written into the header of the LSPD file (see section 8.2.4).
The polynomial fit gives the following parameters:
, equal to the
zeroth order
fit parameter) in Volts. This gives an indication of how well the reset
for this ramp worked. 
where 
is the voltage at time 
, 
is the value of
the fit at that time and 
is the number of points in the ramp.
The goodness of fit is used to calculate the uncertainty in the
photocurrent (see below). Note that G is not the usual measure of goodness
of fit; in fact, the smaller the goodness of fit the better the fit.
where 
is the capacitance of the JF4 for this detector, 
is the time constant of the high pass filter for this detector and DBIA is the
debiasing parameter for this detector and bias level. The JF4 capacitances
are contained in the file LCJF, the time contants of the high pass
filters are contained in file LCFP and the debiasing parameters are
contained in file LCDB.
The uncertainty in the photocurrent is then calculated as follows:
The calculated photocurrent and uncertainty are now written into the LSPD file, together with a time key, grating and FP positions and other information. The time key assigned is the ITK time value of the start of the ramp.
If for any reason the photocurrent has not been calculated for this ramp then both the photocurrent and uncertainty will be set to zero. The most common reason for this is a glitch which has caused all of the data to be discarded. The status word should also indicated that this point is not valid (see section 8.2.5).
For calibration purposes each observation includes two or more periods when the internal illuminators are used. The data from these `illuminator flashes' are identified by SPL, processed, and the results written into an LIPD file. This file is then used as an input to AAL.
Each illuminator flash consists of a dark current measurement, followed by a sequence in which different illuminators are flashed at one or more different levels, followed by another dark current measurement. For grating scans, at least two of the illuminator flashes are `closed' flashes, where the FP is moved into the beam. This removes the source from the beam and means that the dark current measurement during the illuminator flash is a measure of the dark current/straylight. For FP scans all flashes will be `closed' flashes.
The processing of the ramps in illuminator flashes is identical to the processing of ramps of science data, as previously described. The only difference is in the handling of glitches. The LCD1 file contains a separate set of parameters which control the handling of glitches during illuminator flashes. The current setting of these parameters (version 8 of the LCD1 file) means that the whole of a glitched ramp will be discarded, but not subsequent ramps are discarded.
After each ramp in the illuminator flash has been processed it is written into the LIPD file. The LIPD file is analogous to the LSPD file, except that it contains data from illuminator flashes rather than science data. The LIPD file contains the photocurrents for each ramp for each detector, plus auxiliary information such as the value of the illuminator commanded status word and the illuminator current.