Comments on "Description of 2D Spectroscopic Image Data"
From valdes@noao.edu Wed Feb 23 17:26:17 2000
Date: Wed, 23 Feb 2000 17:26:16 -0700 (MST)
From: Frank Valdes
To: ijorgensen@gemini.edu, J.E.H.Turner@durham.ac.uk
Subject: Re: IFU data - meeting etc.
Cc: tody
Hi Inger and James,
I have been working part-time on the changes needed to support IFU
data in IRAF. The first thing was to define a description of the observation
that is general and can be used to drive the extraction and
reconstruction of a data cube fairly automatically. I have a description
which I was awaiting comments internally. However I think it would be
good to start getting your comments too. The only question I think
might be raised from Doug Tody is whether we should put some effort into
supporting a table description which mention in my study. In
any case identifying what information would be needed to allow data
reductions to proceed more automatically is still valid. Also the
actual keyword names is certainly open to change if there is some
compelling reason or precedent that I don't know about.
The description is
http://iraf.noao.edu/projects/ccdmosaic/imagedef/spec2d.html
I would be very interested in your comments and whether you think it
would be feasible to provide the kinds of information described as part
of the data at the time it is taken. That is if the description could
be added by the data acquisition or immediate postproc addition. The
less desirable alternative is to provide the native format description
(such as a mask file or lenslet geometry description) as a file that we
could write a task to convert to the required information.
With the WCS and associated information I describe it would be possible
to reconstruct data cubes after extraction. Making something like
the prototype program from Matt Kenworthy would be easy and would not
need the description files currently required.
Cheers,
Frank Valdes
From tody@noao.edu Wed Feb 23 18:54:05 2000
Date: Wed, 23 Feb 2000 18:54:02 -0700 (MST)
From: Doug Tody
To: Frank Valdes
Subject: Re: IFU data - meeting etc.
The overall scheme looks fine. Extending the keyword mechanism to support
this was the right thing to do. The use of the defaulting mechanism is good.
Providing test case examples in the design specification is good.
From J.E.H.Turner@durham.ac.uk Thu Feb 24 08:13:42 2000
Date: Thu, 24 Feb 2000 15:06:38 +0000 (GMT)
From: James Turner
To: Frank Valdes
cc: ijorgensen@gemini.edu, tody@tucana.tuc.noao.edu
Subject: Re: IFU data - meeting etc.
Frank,
It seems that you have put quite a lot of work into assessing the
conceptual problem of IFU reduction! People at the [IFU Software] meeting
were a bit worried about IFS reduction being generic enough. The question
of FITS headers being interoperable was also stressed, although not in any
detail. It might be useful for you to discuss your ideas with Starlink
when you are ready, so that development can be compatible and
complementary. They seem committed to producing software which is
independent of, yet will happily run under IRAF. I would categorize
possible work as datacube analysis, infrastructure (eg. interpolation
algorithms), or data reconstruction, in order of likelihood. In the
meantime, it was suggested that Isobel, the Cambridge group, the Starlink
programmer and I should give some thought to headers, although I don't see
any need for these to be complicated at the datacube stage.
James.
From J.E.H.Turner@durham.ac.uk Sat Feb 26 11:32:17 2000
Date: Sat, 26 Feb 2000 18:25:03 +0000 (GMT)
From: James Turner
To: Frank Valdes
cc: ijorgensen@gemini.edu, tody@tucana.tuc.noao.edu
Subject: Re: IFU data - meeting etc.
Hello again,
In your "Description of 2D Spectroscopic Image Data", you are starting
from quite a different point of view from me! I suppose TEIFU is half way
between the prototype and common user generations of IFU's, and doesn't
have much of a control system. The acquisition is controlled by an EPICS
interface from the ELECTRA group, but there is currently no means of
writing information to the observation headers. It would actually be quite
difficult to automate everything because of the optical bench setup,
interchangeable moving fore-optics and pre-existing spectrograph control
system. The lack of header info doesn't present a problem, but does limit
the scope for automating reduction.
Jeremy thinks that the control system for GMOS is the responsibility of
the ATC at Edinburgh, and that the person dealing with it is Steven Beard
(smb@roe.ac.uk). Perhaps Inger / Isobel know more about this? I hope
something suitable can be agreed. I take it you are already communicating
with Rachel Johnson (raj@ast.cam.ac.uk) at Cambridge regarding FITS
headers for CIRPASS, since they want to adopt a pipeline system?
I like the way that your scheme allows for all 3 types of IFU. This
substantiates some vaguer ideas I had regarding how the instruments could
be described in a generic definition file, and provides a better starting
point for locating spectra. In general, it is useful to make the
distinction between the physical apertures in the system, which you refer
to, and spectra which may be extracted. With TEIFU, for instance, the
spacing between physical aperture centres on the detector is slightly LESS
than one pixel, so it is impossible to extract individual spectra.
Now a few more specific comments. Might not the limit of 10k apertures in
the header be a problem in the longer term? I believe that some groups are
working with industry to develop large fibre-IFU's, for example. For now,
probably only slicers will have 10k elements, but what if IFU's with many
fibres become cheap to make? Perhaps the FITS standard is inadequate or
the table method would be better? I suppose it all depends how
Gemini-specific you want to be.
There seems to be a lot of redundancy in the information, if it is all
written into every observation header, unless perhaps there is significant
flexure which can be modelled. Had you considered storing a single
supplementary header file for each particular run / grating angle etc? I
can't say I have given this a lot of thought. Perhaps that's what you're
getting at in section 3.
For GMOS, I suppose individual WCS could finally be associated with
fibres, blocks or pixels. For instance, with TEIFU, spatial and spectral
co-ordinates are not associated with apertures but pixels in the end.
However, the idea is to perform wavelength calibration at 2 or more
positions within each output block, which I suppose is more like a slit
problem. Densely packed blocks of spectra are like slits in some ways,
but the individual fibres cannot be ignored because of throughput
variations, multiple rows and the integration they perform.
Important point: You say that broken fibres should be removed from the
description, and rightly point out the importance of those at the ends.
However, I think it would be better still to leave them in and include a
throughput value for each aperture where known. Apart from possibly aiding
flat fielding, this ensures that all the information regarding the output
pattern is available when performing the pattern matching. For GMOS, I am
not completely confident that the MOS technique of tracing each aperture
will prove reliable. However, if the complete flat-field output pattern
can be reconstructed from positions and throughputs (AND maybe PSF),
perhaps a cross-correlation can be performed. I think you hinted at this;
it would represent a hybrid between the TEIFU and MOS approaches.
Obviously, it would be complicated by flexure since it is only really
valid for flat-field observations.
Another possible complication is, of course, that the separation of
elements on the detector may vary with position as well as focus, due to
the optics. This is one of the reasons that I haven't currently
incorporated a proper cross-correlation into the TEIFU code. However, I
think the GMOS optics will be nicer :)
Perhaps the cross-dispersion limit is too simple a concept for GMOS-IFU
since the spectra overlap? Some indication of the spectrum width is
required, but I assume that for MOS the cross-dispersion is the whole
width to be summed up for each spectrum, to the ends of the PSF?
Therefore, it would seem slightly inconsistent to use the limit to
specify, say, FWHM for GMOS. If, however, you just set the limit according
to the fibre pitch, then you don't have all the information that you might
need to determine the overlap for pattern matching. Also, going back to
the last point, the PSF may vary across the detector, ie. not be global.
Since you specify absolute sky co-ordinates for each fibre, are you
intending to reconstruct images which are tilted so as to point North, or
were you intending somehow to calculate which are the principal IFU axes,
by looking for co-linear points in the reconstruction? You can't use the
aperture PA, since it isn't there for fibre-only IFU's... I think that the
concept of a reference PA for an IFU as a whole is useful, particularly
since the observer will have to consider this anyway. For TEIFU/SMIRFS and
others, the IFU orientation (rather than the sky) is always the same with
respect to the reconstructed image. Mosaicing may lend some weight to a
"standard" orientation, but then masks are needed at an earlier stage to
show which pixels contain data, the image size will vary, and
interpolation may be awkward.
I don't know whether it will help much, but I'll attach a file describing
the TEIFU mapping. The input sections were generated by a task which
looks like this (defines a rectangular field):
outfile = "test2.def" Output text file with mapping
(field = 2) Input field number to add / replace
(rowax = "dec") Row axis at PA 0 (ra/dec)
(pattern = "raster") Input sampling pattern
(altlen = no) Row lengths alternating by 1el?
(stagger = "-") Sense of even row offset relative to odd
(startel = "+") Centre -> element 1 sense along rows
(startro = "+") Centre -> element 1 sense across rows
(slicel = 28) Sampling axis elements (max)
(nslice = 18) Reformatted axis elements
The reference PA is chosen above as one of the 2 principal axes.
Co-ordinates are relative to the field centre and are normalized to the
aperture width, since the image scale can be changed (specified in arcsec
when reconstructing). The throughputs won't tell you much. Pitch and FWHM
are just starting values. The output ordering is listed by field number
then element number(s). The sign of the co-ordinates depends on the
fore-optics. Fortunately, the user doesn't have to bother about this!
All in all, I think you have summarized the problem well ... I suppose
your group have experience of this sort of thing! Hope these comments are
helpful for both NOAO and Gemini, and not too rambling.
James.
------------------------------------------------------
James E.H. Turner (J.E.H.Turner@durham.ac.uk)
Durham University Astronomical Instrumentation Group
------------------------------------------------------
Dept. of Physics, Durham Univ. Science Laboratories,
South Road, Durham DH1 3LE, United Kingdom.
Tel. 0191 374 1171 (UK) / +44 191 374 1171 (Outside UK)
Fax. 0191 374 3749 (UK) / +44 191 374 3749
[Sample description file deleted]
From tody@noao.edu Sat Feb 26 12:19:46 2000
Date: Sat, 26 Feb 2000 12:19:40 -0700 (MST)
From: Doug Tody
To: Frank Valdes
Subject: Re: IFU data - meeting etc.
Turner's comments look very useful. The throughput field looks like a good
idea, rather than entirely omitting bad fibers from the list.
From valdes@noao.edu Mon Feb 28 16:21:51 2000
Date: Mon, 28 Feb 2000 16:21:49 -0700 (MST)
From: Frank Valdes
To: J.E.H.Turner@durham.ac.uk
Subject: Re: Comments on spectral description
Cc: ijorgensen, tody
Hi James,
Thanks for your detailed comments. I have also been looking at your web
page which is very helpful. I am rapidly learning about issues I was fuzzy
on.
> In your "Description of 2D Spectroscopic Image Data", you are starting
> from quite a different point of view from me! I suppose TEIFU is half way
> between the prototype and common user generations of IFU's, and doesn't
> have much of a control system. The acquisition is controlled by an EPICS
> interface from the ELECTRA group, but there is currently no means of
> writing information to the observation headers. It would actually be quite
> difficult to automate everything because of the optical bench setup,
> interchangeable moving fore-optics and pre-existing spectrograph control
> system. The lack of header info doesn't present a problem, but does limit
> the scope for automating reduction.
I'm not sure it was clear that the description of the spectra can be
supplied either at the time the data is acquired or at the time the data
reductions are done. I emphasized that it is better for several reasons if
the information is associated with the data right at the beginning. But I
realize that this may not be possible. Also the input information may not
be in the form I describe. Instead some other description might be used to
generate the data to be used by the extraction software using an IRAF task
(or other program) for the purpose.
> Jeremy thinks that the control system for GMOS is the responsibility of
> the ATC at Edinburgh, and that the person dealing with it is Steven Beard
> (smb@roe.ac.uk). Perhaps Inger / Isobel know more about this? I hope
> something suitable can be agreed. I take it you are already communicating
> with Rachel Johnson (raj@ast.cam.ac.uk) at Cambridge regarding FITS
> headers for CIRPASS, since they want to adopt a pipeline system?
There is certainly more chance to get additional information into other
instruments. Again this is a goal but not a requirement.
> I like the way that your scheme allows for all 3 types of IFU. This
> substantiates some vaguer ideas I had regarding how the instruments could
> be described in a generic definition file, and provides a better starting
> point for locating spectra. In general, it is useful to make the
> distinction between the physical apertures in the system, which you refer
> to, and spectra which may be extracted. With TEIFU, for instance, the
> spacing between physical aperture centres on the detector is slightly LESS
> than one pixel, so it is impossible to extract individual spectra.
I am now getting the idea that (some) fibre instruments will pack the
spectra so densely that individual fibre profiles will not be visible.
In this case it may be that the description will be more akin to an
image slicer provided the fibres are mapped to the detector in
contiguous blocks along one spatial axis. I do not see an obstacle to
treating such instruments like the slicer format.
An area which is not the domain of the "Description" document is what
happens with slit type data during extraction. What I forsee is that
the APEXTRACT software will extract 2D longslit subimages with the
trace component perpendicular to the dispersion. This is currently
possible with the "strip" output format. Alternatively, when there
is some distortion across the strip such that the path of constant
wavelength is not aligned with the image lines or columns, is to
use the "subaperture" approach. This divides up the "slit" or
block of fibres into some number of subapertures. These are extracted
as 1D spectra and wavelength calibrated separately to line up
the wavelengths. Which to use with TEIFU-like data I'm not sure.
Probably the subaperture approach where the expected pixel spacing
(which need not be integer) would determine the number of subapertures
to divide the "block" into.
> Now a few more specific comments. Might not the limit of 10k apertures in
> the header be a problem in the longer term? I believe that some groups are
> working with industry to develop large fibre-IFU's, for example. For now,
> probably only slicers will have 10k elements, but what if IFU's with many
> fibres become cheap to make? Perhaps the FITS standard is inadequate or
> the table method would be better? I suppose it all depends how
> Gemini-specific you want to be.
Yes the limit might require one to go to a table description rather
than a keyword description. Alternatively the mapping to FITS keywords
from the logical elements can be changed so that the prefixes are
three characters (100K indexed values) or even two characters (1M).
But at that level a keyword description is not really very appropriate.
On the other hand, it is not my thought that image slicer or TEFUI-like
blocks need to be handled as a large set of 1D spectra one-pixel wide, at
least at the input description stage. The building blocks for the software
and the reconstruction to a data cube would be slit images. The conversion
to a data cube would be more complex since the spatial planes would not be
a set of points with some aperture shape (like lenses) but some strip which
might not be a simple line. That is a detail at this point.
> There seems to be a lot of redundancy in the information, if it is all
> written into every observation header, unless perhaps there is significant
> flexure which can be modelled. Had you considered storing a single
> supplementary header file for each particular run / grating angle etc? I
> can't say I have given this a lot of thought. Perhaps that's what you're
> getting at in section 3.
Well one can always point to a redundancy. For instance why should every
exposure have NAXIS=2? I think that one can live with redundancy
under the consideration that each exposure should be self-contained and
self-describing.
But if the information is not associated with the data at the time it
is taken then the data file driving the reductions could indeed be
a single file describing a block of observations.
> For GMOS, I suppose individual WCS could finally be associated with
> fibres, blocks or pixels. For instance, with TEIFU, spatial and spectral
> co-ordinates are not associated with apertures but pixels in the end.
> However, the idea is to perform wavelength calibration at 2 or more
> positions within each output block, which I suppose is more like a slit
> problem. Densely packed blocks of spectra are like slits in some ways,
> but the individual fibres cannot be ignored because of throughput
> variations, multiple rows and the integration they perform.
What aspect of fibres run together would need to be treated differently
than a slit? The thing that occurs to me is only that when building
a data cube the "PSF" of the spatial cuts would be be different.
> Important point: You say that broken fibres should be removed from the
> description, and rightly point out the importance of those at the ends.
> However, I think it would be better still to leave them in and include a
> throughput value for each aperture where known. Apart from possibly aiding
> flat fielding, this ensures that all the information regarding the output
> pattern is available when performing the pattern matching. For GMOS, I am
> not completely confident that the MOS technique of tracing each aperture
> will prove reliable. However, if the complete flat-field output pattern
> can be reconstructed from positions and throughputs (AND maybe PSF),
> perhaps a cross-correlation can be performed. I think you hinted at this;
> it would represent a hybrid between the TEIFU and MOS approaches.
> Obviously, it would be complicated by flexure since it is only really
> valid for flat-field observations.
This is a good idea. This would be a logical element of the
aperture class that would generally apply only to fibre type apertures.
> Another possible complication is, of course, that the separation of
> elements on the detector may vary with position as well as focus, due to
> the optics. This is one of the reasons that I haven't currently
> incorporated a proper cross-correlation into the TEIFU code. However, I
> think the GMOS optics will be nicer :)
>
> Perhaps the cross-dispersion limit is too simple a concept for GMOS-IFU
> since the spectra overlap? Some indication of the spectrum width is
> required, but I assume that for MOS the cross-dispersion is the whole
> width to be summed up for each spectrum, to the ends of the PSF?
> Therefore, it would seem slightly inconsistent to use the limit to
> specify, say, FWHM for GMOS. If, however, you just set the limit according
> to the fibre pitch, then you don't have all the information that you might
> need to determine the overlap for pattern matching. Also, going back to
> the last point, the PSF may vary across the detector, ie. not be global.
My picture of spectra on the detector was colored by the idea that the
spectra would be clearly separate. As you say the CMIN/CMAX part of
the description was a way to propagate recommended extraction aperture
widths to the "aperture" extraction software. There is no reason these
limits cannot overlap. The description I prepared does not say anything
about PSFs and FWHMs. Keywords could be added but I think the extraction
software should be profile independent. The variation of the profile
with wavelength is another matter. The software can measure this if
desired but I don't think it is necessary in providing the software
approximate information that it can refine as needed.
> Since you specify absolute sky co-ordinates for each fibre, are you
> intending to reconstruct images which are tilted so as to point North, or
> were you intending somehow to calculate which are the principal IFU axes,
> by looking for co-linear points in the reconstruction? You can't use the
> aperture PA, since it isn't there for fibre-only IFU's... I think that the
> concept of a reference PA for an IFU as a whole is useful, particularly
> since the observer will have to consider this anyway. For TEIFU/SMIRFS and
> others, the IFU orientation (rather than the sky) is always the same with
> respect to the reconstructed image. Mosaicing may lend some weight to a
> "standard" orientation, but then masks are needed at an earlier stage to
> show which pixels contain data, the image size will vary, and
> interpolation may be awkward.
What I was thinking was that for IFUs that produce one spectrum sampling
one point on the sky where spatial information is absent within the
entrance aperture (this applies to both lenses and fibre only IFUs)
then the reconstruction of a spatial image would be described solely
by the sky coordinates (which are assumed accurate relative to each
other but may have an error in the zero point and possibly in PA)
and some information about the shape of the sample (circular, hexagonal,
with some dimensional information). As I noted a hexagonal aperture can
have a PA to orient it relative to the sky and its neighbors. For
fibre only there would be no need for an PA for the aperture. However
reconstructing a spatial image might have parameters to allow filling in
the picture or leave gaps between the fiber ends as desired.
The reason that I described in my other recent mail that one does not need
to build a data cube as a final product but instead keep the 1D spectra is
related to this. For fiber or lenslet data where you have a 1D spectrum at
some point which you smear out into a hexagon or circle on the sky with the
same value everywhere, it is redundant to actually build a cube. Instead
the visualization software can build a picture when requested at some
resolution.
> I don't know whether it will help much, but I'll attach a file describing
> the TEIFU mapping.
I'll need to study this a bit. I'll let you know if I have questions.
> All in all, I think you have summarized the problem well ... I suppose
> your group have experience of this sort of thing! Hope these comments are
> helpful for both NOAO and Gemini, and not too rambling.
Thank you. Your comments, web page, and the IFU meeting notes have expanded
my thinking considerably.
Cheers,
Frank
From sla@ucolick.org Wed Mar 1 13:33:25 2000
Date: Wed, 1 Mar 2000 12:33:14 -0800
From: Steve Allen
To: Frank Valdes
Subject: Re: Description of 2D spectra
At first glance I have merely questions.
What about echelle spectra? I don't see any indications of how
you would fit them into here. Perhaps that's an entirely different
document?
I share concerns of others that 10000 objects may not prove enough.
We will use separate tables rather than cramming the 8-char keyword
space so full.
I believe that rectangular limits on the WCS regions will not prove
adequate for DEIMOS. The combination of the 3-d grating equation and
the distortion of the camera lens are going to cause rectangular
pixel regions enclosing each spectrum to have significant overlap
with the neighbor.
We have been considering describing the WCS bounds as a curved box of
constant width whose center is described as a spline or low-order
polynomial.
--
Steve Allen UCO/Lick Observatory Santa Cruz, CA 95064
sla@ucolick.org Voice: +1 831 459 3046 http://www.ucolick.org/~sla
PGP: 1024/E46978C5 F6 78 D1 10 62 94 8F 2E 49 89 0E FE 26 B4 14 93
From valdes@noao.edu Thu Mar 2 11:53:06 2000
Date: Thu, 2 Mar 2000 11:53:05 -0700 (MST)
From: Frank Valdes
To: sla@ucolick.org
Subject: Re: Description of 2D spectra
Hi Steve,
Thanks for the comments.
> What about echelle spectra? I don't see any indications of how
> you would fit them into here. Perhaps that's an entirely different
> document?
You are right that I did not consider this case. I don't think there
would be any problem in fitting it into the scheme.
> I share concerns of others that 10000 objects may not prove enough.
> We will use separate tables rather than cramming the 8-char keyword
> space so full.
A table specification will need to be supported. The reason for
devoting more attention to the keywords right now is that I would like
to support the keyword description in APEXTRACT in the next few months.
This would be easy with keywords. With tables and table extensions
it would require getting into TABLES software and access to FITS
table extensions. Something that would be more work.
> I believe that rectangular limits on the WCS regions will not prove
> adequate for DEIMOS. The combination of the 3-d grating equation and
> the distortion of the camera lens are going to cause rectangular
> pixel regions enclosing each spectrum to have significant overlap
> with the neighbor.
Good point. However what may not be clear as it applies to APEXTRACT
as the data extraction tool is that the rectangular limits would be
sufficient to define the aperture width and find the profile center.
The tracing feature of APEXTRACT would still be used and that trace
would "curve" the rectangular aperture to fit the data. The rectangular
region to define wavelength limited regions for the extraction is also
all that is needed to support things like tiered multislit and objective
prism detector patterns of spectra.
> We have been considering describing the WCS bounds as a curved box of
> constant width whose center is described as a spline or low-order
> polynomial.
This could be done but it would certainly make the description much
more complex and voluminous (if there are many spectra). It would be
good for archival descriptive purposes but it is not really required
for data reduction as I noted previously.
Cheers,
Frank
From sla@ucolick.org Fri Mar 3 00:48:28 2000
Date: Thu, 2 Mar 2000 23:48:24 -0800
From: Steve Allen
To: Frank Valdes
Subject: Re: Description of 2D spectra
Hi Frank,
On Thu 2000-03-02T11:53:05 -0700, Frank Valdes hath writ:
> > We have been considering describing the WCS bounds as a curved box of
> > constant width whose center is described as a spline or low-order
> > polynomial.
>
> This could be done but it would certainly make the description much
> more complex and voluminous (if there are many spectra). It would be
> good for archival descriptive purposes but it is not really required
> for data reduction as I noted previously.
Well, in our case we're considering it for the case of quick
look reduction tools. The mouse cursor position would be tracked,
compared with an insidedness test for the slitlet bounding regions,
and then used to lookup the right WCS information for giving out
the object ID, sky position (along the slitlet), and wavelength
of the cursor.
That's the reason why we don't think rectangles will do our job.
When the observer is moving the mouse around the image we would
like to choose the slitlet correctly, even when the mouse is way
off toward the distorted edges.
--
Steve Allen UCO/Lick Observatory Santa Cruz, CA 95064
sla@ucolick.org Voice: +1 831 459 3046 http://www.ucolick.org/~sla
PGP: 1024/E46978C5 F6 78 D1 10 62 94 8F 2E 49 89 0E FE 26 B4 14 93