ccdproc
reduce raw CCD images

Version 2.0 (2011 Nov 8)

ccdproc performs basic reductions on raw CCD images (overscan bias subtraction, trim, zero and/or dark subtraction, and flat fielding), creating a reduced image ready for subsequent analysis. A process control (proc) file contains the instructions for performing the reduction. ccdproc is designed to be used for fast pipeline reductions of CCD images, but requires that the necessary zero (2D Bias) and normalized flat-field images be already prepared using standard image reduction packages like IRAF, VISTA, IDL, etc.

You can jump a head to information on how to download the latest version (v2.0) of ccdproc, and see also the Release Notes for a brief summary of what is new in this version.

Index:

Usage:

  ccdproc -Pprocfile infile outfile
where:
-Pprocfile
Use "procfile" as the Process Control File. By default, ccdproc will use procfile.pro in the current working directory. See below for a description of the commands in the procfile.

infile
Name of the raw FITS file to reduce.

outfile
Name of the FITS file to contain the reduced image.
[Menu]

Process Control Files:

A Process Control (proc) File is an ASCII text file containing the processing commands as "KEYWORD value" pairs. Blank lines are ignored. Extraneous or unrecognized directives generate errors. The <>'s are used to delimit the arguments of the keywords but are not typed as the examples show.

The order of the functions performed are fixed and as follows: 

        OVERSCAN
        TRIM
        ZERO
        DARK
        FLAT
Even if any of these steps are skipped, the order remains the same. An additional function to estimate input parameters for DoPhot (sky level, star image FWHM, and min/max star thresholds), is available via the FWHMSKY function written by John Menzies (SAAO).

The zero, dark, and flat calibration images must have the same size and orientation as the input image after trimming and overscan, or if both overscan and trim are skipped, they must be the same size and orientation as the raw input image.

The input image must be valid FITS format, of any data common type (i.e., short integer, long integer, or float). All internal operations are done with floating arithmetic, and input type conversion is implicit. On output, ccdproc will generate floating (BITPIX=-32) images by default, but will generate either SHORT or LONG images, with appropriate scaling using BZERO and BSCALE keywords to preserve data precision. The output datatype is specified using the DTYPE flag in the procfile.

ccdproc can also keep a runtime log of all processing, using the LOGFILE command. This is useful for record-keeping if ccdproc is used as part of an automated image reduction pipeline.

[Menu]

Procfile Commands:

#: Insert a comment into the procfile

  # <comment text>
     Function: Insert a comment into the procfile.  This also provides
               a way to disable commands without deleting them.
     Notes: The # must appear only as the first character in a line.
            No in-line comments are allowed.
[Menu]

DTYPE: Define the datatype of the output FITS file

  DTYPE <data type>
     Function: Defines the data type of the output FITS file.  Valid
               data types are:
                   FLOAT (bitpix=-32): single-precision floats
                   LONG (bitpix=32): 32-bit signed integers
                   SHORT (bitpix=16): 16-bit signed integers
     Example: DTYPE LONG
     Default: FLOAT
[Menu]

LOGFILE: Keep a runtime log of image processing

  LOGFILE <name or code>
     Function: Keep a runtime log of all processing.  Also records errors
               that occur.  The argument of LOGFILE is either the path/name
               of the logfile to keep or one of the following special
               codes:
                   DATE: create a filename of the form ccdproc.YYMMDD.log
                         where YYMMDD is the date in local time.
                   UTDATE: same as DATE, but uses the UT time clock.
     Examples:
               LOGFILE /data/Logs/ccdproc.log
               LOGFILE UTDATE

     Default: None, keeps no runtime log if LOGFILE is absent.

     Note: At present, UTDATE and DATE only create process logs in the 
           current working directory.
[Menu]

Subtract overscan bias from Single-Amp CCDs:

  DATASEC [dsc:dec,dsr:der]
  BIASSEC [bsc:bec,bsr:ber]
     Function: Defines the data and overscan bias sections on a CCD
               image with a single readout amplifier (see below for
               multi-amp CCDs). 
               Coordinates are:
                     sc = starting pixel along the column (x) axis
                     ec = ending pixel along the column (x) axis
                     sr = starting pixel along the row (y) axis
                     er = ending pixel along the row (y) axis
              No trimming is performed after bias correction (see 
              TRIM.)
     Example: DATASEC [1:512,1:512]
              BIASSEC [513:544,1:512]
 
              512x512 CCD with 32 columns of overscan.
 
     Default: None.  If absent, no overscan bias subtraction will be done.
[Menu]

Subtract overscan bias from multiple readout CCDs:

  DATASECn [dsc:dec,dsr:der]
  BIASSECn [bsc:bec,bsr:ber]
     Function: Defines the data and overscan bias sections on a CCD
               image with a multiple readout amplifiers.
               Coordinates are:
                     sc = starting pixel along the column (x) axis
                     ec = ending pixel along the column (x) axis
                     sr = starting pixel along the row (y) axis
                     er = ending pixel along the row (y) axis
              No trimming is performed after bias correction.  
              See TRIM.
     Example: DATASEC1 [1:2048,1:1024]
              BIASSEC1 [2049:2080,1:1024]
              DATASEC2 [1:2048,1025:2048]
              BIASSEC2 [2049:2080,1025:2048]
 
              2048x2048 read with 2 amplifiers, 32 overscan columns/amp.
    
     Default: None.  If absent, no overscan bias subtraction will be done.
     Notes: ccdproc determines the number of readout amplifiers from the
            number of DATASECn/BIASSECn pairs in the proc file.  These
            must appear without gaps (e.g., you cannot have BIASSEC1,2,4
            and skip BIASSEC3.  All or nothing...)
[Menu]

TRIM: Trim the Image

  TRIM [sc:ec,sr:er]
     Function: Trim (crop) an image within the specified sub-array.
               Sub-array coordinates are:
                     sc = starting pixel along the column (x) axis
                     ec = ending pixel along the column (x) axis
                     sr = starting pixel along the row (y) axis
                     er = ending pixel along the row (y) axis
               Discards regions outside of the trim section. This is
               usually used in conjunction with the DATASEC/BIASSEC
               commands to trim away the bias sections after correction.
               The TRIM function will be executed before any of ZERO,
               DARK, or FLAT.
     Example: TRIM [17:528,1:512]
              Trims the image so that only the pixels in the specified
              sub-array remain.  The pixels will be renumbered to start
              with pixel (1,1). 
     Default: None.  If TRIM is absent, no trimming is done.
[Menu]

ZERO: Subtract a Zero (2D Bias) Image

  ZERO <FITS file with zero image>
     Function: Subtracts the named zero (2D bias) frame from the raw 
               image.
     Example: ZERO zero2.fits
              subtracts "zero2.fits" from the input image.
     Default: None.  If absent, no zero frame is subtracted.
     Warning: The raw image and zero image must be the same size.
              See TRIM to trim the input image
              prior to subtracting the ZERO image.
[Menu]

DARK: Subtract a Dark Frame

  DARK <FITS file with dark frame> <scale_factor>
     Function: Subtracts the named dark frame from the raw 
               image.  If an optional scale factor is given,
               the dark is multiplied by scale_factor before
               subtraction.
     Examples: 
          DARK dark60.fits
              subtracts "dark60.fits" from the input image.

          DARK dark60.fits 2.0
              multiplies dark60.fits by 2.0 and subtracts it
              from the input image.

    Defaults: None.  If absent, no dark frame is subtracted.
              If no scale factor is given, 1.0 (unscaled) is assumed.

     Warning: The raw image and dark image must be the same size.
              See TRIM to trim the input image
              prior to subtracting the DARK image.

       Notes: DARK assumes that the dark frame has been at least
              bias corrected, trimmed (as required), and zero
              subtracted (as required).  Dark subtraction comes
              between ZERO and FLAT in order of processing.

              The DARK function is available only in ccdproc v1.4
              and later.
[Menu]

FLAT: Divide by a Flat Field

  FLAT <FITS file with normalized flat-field image>
     Function: Divides the raw image by the named normalized
               flat-field image.
     Example: FLAT Vflat.fits
              divides the image by the flat-field frame "Vflat.fits"
     Default: None.  If absent, no flat-field division is performed.
     Warning: The raw image and flat-field image must be the same size.
              See TRIM to trim the input image
              prior to ZERO and FLAT processing.
     Notes: ccdproc assumes that the flat field image has already been
            normalized appropriately to a mean value of 1.0.  No 
            internal scaling is performed.
[Menu]

FWHMSKY: Estimate DoPhot Input Parameters

  FWHMSKY GAIN=<e-/adu>
     Function: Estimate input parameters for DoPhot using the final reduced
               image.  Computes the modal sky level, mean stellar FWHM,
               and min/max thresholds for DoPhot.  These values are 
               stored in the FITS header.  The optional GAIN= keyword
               is used to give the detector gain in e-/ADU.  If GAIN=
               is not given, it assumes a default of 1.0 e-/ADU.

               FITS header cards written are:
                   FWHM  = Mean FWHM in pixels.  -1 means no stars found
                   SKY   = Modal SKY level in ADU.
                   TSMIN = Minimum star threshold for DoPhot
                   TSMAX = Maximum star threshold for DoPhot

     Example: FWHMSKY GAIN=9.0
              Computes DoPhot input parameters assuming a detector gain
              of 9.0 e-/ADU.
     Default: GAIN=1 if not given.  If the FWHMSKY keyword is absent,
              no parameter estimation is done.
[Menu]

Examples:

Example 1:

The raw CCD image is a 512x512 B-band image read with a single amp named 980106.001 in the current working directory. Overscan is 32 columns, we have no zero frame, and a B-band flat field image previously prepared and stored in Bflat.fits in the current directory. The proc file is Bproc, also in the current directory. The reduced frame is to be written to the /data/980106/reduced/ directory with name ngc1068b1.fits as a floating-point FITS image. 
   % ccdproc -PBproc 980106.001 /data/980106/reduced/ngc1068b1.fits
The Bproc file looks like this: 
       DTYPE FLOAT
       BIASSEC [513:544,1:512]
       DATASEC [1:512,1:512]
       TRIM [1:512,1:512]
       FLAT Bflat.fits

Example 2:

A 512x512 CCD with 1 amp named b140405980003.fits in the current directory, prepared zero and flat frames named Zero.fits & VFlat.fits, respectively in /usr1/calib/. We need to trim pre/post scan columns that contains no useful overscan info (the zero bias dominates). The reduced frame is to be written to the /usr1/040598/reduced/ directory with name b14.0003.fits as a 32-bit integer FITS file. 
   % ccdproc -PVproc b140405980003.fits /usr1/940598/reduced/b14.0003.fits
The Vproc file looks like this: 
       DTYPE LONG
       TRIM [17:528,1:512]
       ZERO /usr1/calib/Zero.fits
       FLAT /usr1/calib/VFlat.fits
[Menu]

Example 3:

A 180-second 512x512 CCD I-band image named MB99047I001.fits is in the current directory. Pre-processd zero, dark and flat frames named Zero.fits, Dark60.fits & IFlat.fits, respectively are in /data/nt04/calib/. Dark60.fits is a mean of five 60-second dark frames that has previously been overscan corrected, trimmed, and zero subtracted using ccdproc, and combined using imcombine. It will need to be scaled by a factor of 3.0 to match the dark counts in the 180-second object frame. The reduced frame is to be written to the /data/nt04/reduced/ directory with name YMB99047I001.fits as a 16-bit integer FITS file. The CCD has one amp, and the overscan region runs between columns 515 and 544. 
   % ccdproc -PPreProcI MB99047I001.fits /data/nt04/reduced/YMB99047I001.fits
The PreProcI file looks like this: 
       DTYPE SHORT
       BIASSEC [515:544,1:512]
       DATASEC [1:512,1:512]
       TRIM [1:512,1:512]
       ZERO /data/nt04/calib/Zero.fits
       DARK /data/nt04/calib/Dark60.fits 3.0
       FLAT /data/nt04/calib/IFlat.fits
The five 60s dark frames were named dark001.fits through dark005.fits, and processed ahead of time using ccdproc and imcombine as follows: 
   % ccdproc -PDarkProc dark001.fits dark1.fits
   % ccdproc -PDarkProc dark002.fits dark2.fits
   % ccdproc -PDarkProc dark003.fits dark3.fits
   % ccdproc -PDarkProc dark004.fits dark4.fits
   % ccdproc -PDarkProc dark005.fits dark5.fits
   % imcombine -a /data/nt04/calib/Dark60.fits dark[1-5].fits
The DarkProc file looked like this: 
       BIASSEC [515:544,1:512]
       DATASEC [1:512,1:512]
       TRIM [1:512,1:512]
       ZERO /data/nt04/calib/Zero.fits
It uses the overscan columns 515-544 to remove the DC bias, trims the image to 512x512 pixels, and uses the Zero.fits frame to subtract the residual 2-D bias on the device. The output Dark60.fits file is a mean of the 5 frames, in floating (bitpix=-32) format.

[Menu]

Example 4:

A 30-second 4096x4096 CCD image named "k040426.0012.fits" is in the current directory. A normalized flat named "nflat040426.fits", zero-frame named "zero040426.fits", and 30-second dark frame named "dark040426.fits" are in the /data/kelt/calib/ directory. The output is to be a reduced image named "rk040426.0012.fits" in the /data/kelt/040426/ directory. The procfile is "kelt.pro" in the /data/kelt/proc/ directory. The raw images do not have overscan columns, so no overscan or trim processing is required. Reduced images are created in default floating-point (bitpix=-32) format.

Run ccdproc like this: 

   % ccdproc -P/data/kelt/proc/kelt.pro k040426.0012.fits /data/kelt/040426/rk040426.0012.fits
The kelt.pro file looks like this 
       LOGFILE /data/kelt/redux040426.log
       ZERO /data/kelt/calib/zero040426.fits
       DARK /data/kelt/calib/dark040426.fits
       FLAT /data/kelt/calib/nflat040426.fits
If you had a large number of such frames from a given night, it might make sense to write a simple csh script to run ccdproc over all raw fits images. Such a script might look like this: 
   #!/bin/csh
   set pfile = "/data/kelt/proc/kelt.pro"
   foreach infile ('k*.fits')
     set outfile = "/data/kelt/040426/r${infile}"
     ccdproc -P${pfile} ${infile} ${outfile}
   end
   exit
The progress of the script will be recorded in the logfile noted in the "kelt.pro" file example above.

The utility of ccdproc is that it can be run from any kind of script. For example, a more sophisticated semi-automated reduction pipeline written in Perl or Python could incorporate calls to ccdproc that create the necessary procfiles to reduce data from a given night automatically.

[Menu]

Requirements:

Version 2.031 or later of the HEASARC cfitsio library is needed to compile ccdproc (see Compiling below). Otherwise ccdproc is self-contained.

We require you to use version 2 or later of CFITSIO (the current version is 2.490, released on 2004 Feb 12). These versions of CFITSIO provide support for Year2000-compliant FITS date formats that became mandatory on 1999 January 1. We have tested all code with CFITSIO version 2.490 on RedHat Linux systems running RHv7.3

At runtime, you need procfile to be named on the command line using the -P flag and an input raw FITS image. All zero and flat-field images (as required) must be prepared in advance and in a valid FITS format. The output FITS file must not exist prior to running ccdproc, as it will prevent you from overwriting an existing image. It also prevents you from overwriting the raw image with the reduced image as a safety feature.

[Menu]

Errors:

All errors are reported to stderr and to the runtime logfile (if enabled). Errors that occur before ccdproc can open the logfile are, of course, only printed to stderr.

[Menu]

Bugs:

ccdproc will let you do some pretty goofy things if you are not careful. This program is meant to be used for well-defined pipeline data reduction, and is inadvisable for general use without care. IRAF, VISTA, etc. are better suited for general image reduction.

Please report bugs to either Rick Pogge or Paul Martini.

[Menu]

Downloading ccdproc:

ccdproc and imcombine are available as a single gzip-compressed tar archive (33KB). The latest version is v2.0 (2011 November 8):
ccdproc_v2.tgz
A suite of test FITS images (originally distributed with v1.0) is also available separately (2.4MB gzip compressed tar file):
ccdtest.tgz
See the 00README file provided with the ftp distribution for installation details.

[Menu]

Installation:

ccdproc must be compiled with the HEASARC cfitsio libraries, version 3.280 or later.

See the 00README file provided with the ftp distribution for installation instructions.

The most recent version of CFITSIO that we have used and tested with this code is v3.280 (2011 May 12), but any version later than v3.280 should work fine.
CFITSIO is freely available and very well supported and documented. It has emerged as the benchmark FITS generation/reading package in Astronomy. In general, it is senseless to use anything else to read or write FITS format images. See the FITSIO website
http://heasarc.gsfc.nasa.gov/docs/software/fitsio/.
for the latest version of cfitsio and the associated documentation.

[Menu]

Authors:

Richard Pogge (pogge@astronomy.ohio-state.edu),
Paul Martini (martini@astronomy.ohio-state.edu)
OSU Astronomy Department
1999 May 27
The FWHMSKY routines were written by John Menzies (jwm@saao.ac.za), SAAO.

Return to: [OSU Software Homepage]
Updated: 2011 Nov 8 [rwp]