Fast (simple) spectral synthesis and emission-line fitting of DESI spectra.

Overview

FastSpecFit

Documentation Status

Introduction

This repository contains code and documentation to perform fast, simple spectral synthesis and emission-line fitting of DESI spectra.

Installation

To be written.

License

FastSpecFit is free software licensed under a 3-clause BSD-style license. For details see the LICENSE.rst file.

John Moustakas
Siena College
Winter 2020
Comments
  • photsys can be empty for some secondary targets

    photsys can be empty for some secondary targets

    Summarizing an off-list conversation with Raga, @geordie666, and @stephjuneau:

    To get PHOTSYS from TARGETID (thanks to @geordie666):

    from desitarget.io import releasedict
    from desitarget.targets import decode_targetid  
    _, _, releases, _, _, _ = decode_targetid(targets["TARGETID"])  
    photsys = [releasedict[release] if release >= 9000 else None for release in releases]
    

    This should return "N"/"S" as expected, or None if the target has no Legacy Surveys provenance (by which I mean that no valid TARGETID was propagated from dr9 for the target).

    But then Raga found:

    Adam's code works if the release is not equal to 11 - so, I am assuming it works for 
    the targets that have DR9 photometry. 
    
    Here are two targets for testing - 
    1. TARGETID: 39633489506601735, TILEID: 80696, RELEASE: 9011, PHOTSYS: 'N'
    2. TARGETID: 39632956502838508, TILEID: 80895, RELEASE: 9011, PHOTSYS: 'N'
    
    There is another thing that I have found in the fibermaps. In some cases, the flux_* values 
    are 0.0 - even when they have valid DR9 phtometry. And this is not always the case for just 
    secondary targets. I have seen ~10 BGS targets with FLUX_G = 0.0 - I still need to investigate 
    this further.  From the above two targets, the first target have FLUX_G, R, Z = 0 from the FIBERMAP - 
    but, it has valid values in DR9 photometry. 
    
    help wanted 
    opened by moustakas 42
  • Broad+Narrow Balmer line components

    Broad+Narrow Balmer line components

    Some Seyfert galaxies will likely have both a broad and a narrow Balmer line components. This example shows such a case with broad+narrow components on H-beta and H-alpha. However, the linewidth reported by fastspecfit is not larger for Halpha than for [OIII]5007, as I would have expected if fitting a single broad component to Halpha but a single narrow component to forbidden lines such as [OIII]5007.

         TARGETID     HPXPIXEL SURVEY FAPRGRM HALPHA_SIGMA OIII_5007_SIGMA
    ----------------- -------- ------ ------- ------------ ---------------
    39628517884825354     9730    sv1    dark    126.67458       139.04639
    
    enhancement 
    opened by stephjuneau 10
  • the unit 'nmgy' could not be saved in native FITS format

    the unit 'nmgy' could not be saved in native FITS format

    #69 propagates units through all the astropy tables using the QTable object, but when writing the files in the last step, the following warning is issued. Not sure how to deal with this at the moment but tagging @weaverba137 @geordie666 @dstndstn @sbailey as there's a similar discussion (I think?) going on with the imaging / targeting files.

    WARNING: The unit 'nmgy' could not be saved in native FITS format and cannot be recovered in reading. It can roundtrip within astropy by using QTable both to write and read back, though one has to enable the unit before reading. [astropy.io.fits.convenience]
    WARNING: The unit '1 / nmgy2' could not be saved in native FITS format and cannot be recovered in reading. It can roundtrip within astropy by using QTable both to write and read back, though one has to enable the unit before reading. [astropy.io.fits.convenience]
    
    opened by moustakas 7
  • band_shift the absolute magnitudes

    band_shift the absolute magnitudes

    In fastphot, consider band-shifting the ugriz photometry to z=0.1 so the results can be compared with SDSS more easily. The UBVW1 bands can be band-shifted to z=0.

    help wanted 
    opened by moustakas 5
  • Provide wrapper to instantiate model spectrum

    Provide wrapper to instantiate model spectrum

    It will be helpful to provide a wrapper to take a data table with the continuum coefficients as inputs and instantiate the best fit model spectrum for both photometry only (FASTPHOT) and spectra only (FASTSPEC) fits.

    This can help with science cases like (but not limited to):

    • Synthesize photometry in any arbitrary band
    • apply aperture corrections
    • change distance modulus to any arbitrary cosmology
    opened by biprateep 5
  • profile the code and see if there are some simple optimizations to be had

    profile the code and see if there are some simple optimizations to be had

    The most expensive part of the code is in the emission-line fitting which, unfortunately, relies on the astropy.fitting module. Do a profiling pass and see if there are some simple optimizations and speed-ups to be had.

    https://docs.nersc.gov/development/languages/python/profiling-debugging-python/

    opened by moustakas 5
  • fit for the doublet ratio as a free parameter (with physical priors)

    fit for the doublet ratio as a free parameter (with physical priors)

    The [OII] and [SII] doublet ratios depend on the electron density of the medium producing those lines, while the MgII doublet ratio can vary over a not-arbitrary range (for example, if one line is positive, the other line must be positive, too).

    To integrate these physical priors into fastspecfit, consider using the ratio of these lines as one of the free parameters (with a sensible prior range) rather than either fixing the ratio or letting the individual line-amplitudes to be optimized independently.

    Credit for this idea to Ben Weiner.

    enhancement 
    opened by moustakas 4
  • fixed spectroscopic DN4000 calculation bug

    fixed spectroscopic DN4000 calculation bug

    This pull request fixes issue #31 . The d4000 function was getting the incorrect keyword argument. Also added a fix to prevent unintentional modification of input array by the same function. This shows a comparison of calculations based on using the get_d4000 function directly on camera added spectra vs the output of fastspec. zoomed This shows a comparison between the fastphot catalog and the fastspec values. I think this fixes the bug. download (8)

    opened by biprateep 4
  • refactor fitting engine to not use fnnls or astropy.modeling

    refactor fitting engine to not use fnnls or astropy.modeling

    [WIP]

    Still WIP but opening a PR to track related issues.

    This PR is a fairly major rewrite of the emission-line fitting engine with an eye toward porting the code to the Perlmutter/GPUs (e.g., https://jaxopt.github.io/stable/constrained.html). Specifically, all the astropy.modeling routines have been replaced with a simpler table-based linemodel of parameters (including tied parameters), resulting in about 300 fewer lines of code and notable speed-ups.

    However, the speed-ups have been used to include one additional (third) round of fitting where emission lines are minimally constrained to one another, which leads to notable improvements in the line-fitting results for systems with complex line-kinematics (and to account for any cross-camera wavelength-calibration issues).

    Second, I've moved away from the home-grown fnnls algorithm for continuum fitting and am now just using scipy.optimize.nnls which I also hope to be able to port to a GPU.

    opened by moustakas 3
  • Fujilupe release candidate

    Fujilupe release candidate

    Backwards incompatible updates in preparation of generating fastspec and fastphot catalogs for Fuji+Guadalupe. Together with #55, #61, and #64, this PR contains many significant improvements, including:

    • All non-sky spectra at z>0.001 are now fit by default, irrespective of ZWARN or COADD_FIBERSTATUS (#45).
    • Balmer and Helium lines now optionally include both broad and narrow components (#47).
    • The [OII] 3726,29, [SII] 6716,31, and MgII 2796,2803 doublets are now fit using a more robust model which optimizes the ratio of the doublet lines (#39).
    • The velocity dispersion is now estimated for spectra with enough signal-to-noise (#67).
    • More robust initialization of parameters to help ensure numerical convergence (#52).
    • A simple stellar mass estimate is now computed (#70).
    • Minimum photometric uncertainties included when fitting photometry (#22). In addition, W3 and W4 fluxes and ivar fluxes have been propagated and now included in the fastphot QA.
    • Numerous bug fixes and many data model improvements and changes.

    Still on the critical path at the moment:

    • [ ] Profile the code to identify speed-ups.
    • [ ] Test the feasibility of writing out the best-fitting model spectra.
    • [ ] Consider gzipping the output catalogs, in order to save disk space.
    • [ ] Fix the failing unit tests.
    • [ ] Update the documentation and installation instructions.
    opened by moustakas 3
  • Unreal fits because the fit does not converge to a final solution

    Unreal fits because the fit does not converge to a final solution

    I found one galaxy (low-mass AGN from Reines+2013) where the fit does not converge to a solution leading to wrong emission line fluxes. The fastspecfit-qa for this sources is as follows -

    image

    Source Information: image

    SDSS Information if needed: image

    opened by Ragadeepika-Pucha 3
  • Stacked spectra input option

    Stacked spectra input option

    An option to input spectra for which we just have a flux + ivar + wl_bins would be really useful! For stacking mostly( but maybe someone wants to throw some other spectra in as well).

    In this case MW-foreground correction etc would already be done before entering fastspecfit and cameras coadded into a single spectrum. Once the input format is determined we can write an output option in the stacking code that can write out the stacked spectrum in that format.

    I know you already pointed me to some code for this in the templates folder - I had a bit of a struggle getting it to work. 😅

    opened by dirkscholte 0
  • Biases of model magnitude for red galaxies

    Biases of model magnitude for red galaxies

    @moustakas It seems that the colors based on the best-fit model are biased blue for red galaxies. This bias becomes more pronounced for low redshift galaxies.

    The following plot, based on the fastphot-fuji-sv1-dark.fits catalog, demonstrates this finding. I first cut out galaxies with DES>0 to make sure DECAM filter is used for calculations. The y-axis is the fractional error of color based on FLUX_SYNTH_MODEL_band and color based on Flux_band. Triangles with the error bar show the median and error of the median. The blue band shows the histogram of the samples going into the median calculation. The top panel shows low redshift galaxies while the bottom panel shows the full samples.

    Since the width of the redsequence is ~0.025, it would be great to have biases smaller than this value.

    download - 2022-11-22T153850 701

    templates 
    opened by chto 1
  • nb for k-correction calculation

    nb for k-correction calculation

    Following our meeting, it would be useful to have a notebook that could give an example of how to use the fastspecfit code to calculate k-corrections given a spectrum.

    documentation 
    opened by SgmAstro 0
  • consider adopting the fiducial DESI cosmology

    consider adopting the fiducial DESI cosmology

    fastphot currently uses a vanilla LCDM cosmology with h=0.7 to compute luminosity distances. Consider adopting the fiducial DESI cosmology to make it easier to map between data and mocks-- https://desi.lbl.gov/trac/wiki/keyprojects#Fiducialcosmology

    opened by moustakas 1
  • Objects with odd ABSMAG_SDSS_R

    Objects with odd ABSMAG_SDSS_R

    A handful of objects from the LOW-Z program with unexpectedly bright ABSMAG_SDSS_R

    TARGETID

    39633481701000832 39627862432552794 39633141308069648 39633343209279504 39633428097794736 39633507869262245 39627717636787810 39628023342826572 39628071015284948 39628159053726478 39628182306947825 39628135825672502 39628211436393100 39628199897861563 39628245884208887 39628188023785809 39628357876318744 39632941596281448 39628512897797887 39633072903160512

    HEALPIX

    11615 12320 15025 15279 16006 16106 19075 31655 31685 31725 31738 32298 32311 32311 32358 8521 8575 9397 9411 9868

    PROGRAM | SURVEY -- | -- bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main bright | main

    opened by edarragh 9
Releases(v1.0.1)
Owner
Public code associated with the Dark Energy Spectroscopic Instrument (DESI).
Paddle Graph Learning (PGL) is an efficient and flexible graph learning framework based on PaddlePaddle

DOC | Quick Start | 中文 Breaking News !! 🔥 🔥 🔥 OGB-LSC KDD CUP 2021 winners announced!! (2021.06.17) Super excited to announce our PGL team won TWO

1.5k Jan 06, 2023
UCSD Oasis platform

oasis UCSD Oasis platform Local project setup Install Docker Compose and make sure you have Pip installed Clone the project and go to the project fold

InSTEDD 4 Jun 16, 2021
A trashy useless Latin programming language written in python.

Codigum! The first programming langage in latin! (please keep your eyes closed when if you read the source code) It is pretty useless though. Document

Bic 2 Oct 25, 2021
Config files for my GitHub profile.

Canalyst Candas Data Science Library Name Canalyst Candas Description Built by a former PM / analyst to give anyone with a little bit of Python knowle

Canalyst Candas 13 Jun 24, 2022
Management Dashboard for Torchserve

Torchserve Dashboard Torchserve Dashboard using Streamlit Related blog post Usage Additional Requirement: torchserve (recommended:v0.5.2) Simply run:

Ceyda Cinarel 103 Dec 10, 2022
Official repository for "Exploiting Session Information in BERT-based Session-aware Sequential Recommendation", SIGIR 2022 short.

Session-aware BERT4Rec Official repository for "Exploiting Session Information in BERT-based Session-aware Sequential Recommendation", SIGIR 2022 shor

Jamie J. Seol 22 Dec 13, 2022
Enhancing Column Generation by a Machine-Learning-BasedPricing Heuristic for Graph Coloring

Enhancing Column Generation by a Machine-Learning-BasedPricing Heuristic for Graph Coloring (to appear at AAAI 2022) We propose a machine-learning-bas

YunzhuangS 2 May 02, 2022
PyZebrascope - an open-source Python platform for brain-wide neural activity imaging in behaving zebrafish

PyZebrascope - an open-source Python platform for brain-wide neural activity imaging in behaving zebrafish

1 May 31, 2022
In-place Parallel Super Scalar Samplesort (IPS⁴o)

In-place Parallel Super Scalar Samplesort (IPS⁴o) This is the implementation of the algorithm IPS⁴o presented in the paper Engineering In-place (Share

82 Dec 22, 2022
make ASCII Art by Deep Learning

DeepAA This is convolutional neural networks generating ASCII art. This repository is under construction. This work is accepted by NIPS 2017 Workshop,

OsciiArt 1.4k Dec 28, 2022
Implementation of paper "Graph Condensation for Graph Neural Networks"

GCond A PyTorch implementation of paper "Graph Condensation for Graph Neural Networks" Code will be released soon. Stay tuned :) Abstract We propose a

Wei Jin 66 Dec 04, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF

Google 625 Dec 30, 2022
This is the repo for the paper `SumGNN: Multi-typed Drug Interaction Prediction via Efficient Knowledge Graph Summarization'. (published in Bioinformatics'21)

SumGNN: Multi-typed Drug Interaction Prediction via Efficient Knowledge Graph Summarization This is the code for our paper ``SumGNN: Multi-typed Drug

Yue Yu 58 Dec 21, 2022
🔮 A refreshing functional take on deep learning, compatible with your favorite libraries

Thinc: A refreshing functional take on deep learning, compatible with your favorite libraries From the makers of spaCy, Prodigy and FastAPI Thinc is a

Explosion 2.6k Dec 30, 2022
Repository for MuSiQue: Multi-hop Questions via Single-hop Question Composition

🎵 MuSiQue: Multi-hop Questions via Single-hop Question Composition This is the repository for our paper "MuSiQue: Multi-hop Questions via Single-hop

21 Jan 02, 2023
Scripts and misc. stuff related to the PortSwigger Web Academy

PortSwigger Web Academy Notes Mostly scripts to automate the exploits. Going in the order of the recomended learning path - starting with SQLi. Commun

pageinsec 17 Dec 30, 2022
A python interface for training Reinforcement Learning bots to battle on pokemon showdown

The pokemon showdown Python environment A Python interface to create battling pokemon agents. poke-env offers an easy-to-use interface for creating ru

Haris Sahovic 184 Dec 30, 2022
Contrastive Learning with Non-Semantic Negatives

Contrastive Learning with Non-Semantic Negatives This repository is the official implementation of Robust Contrastive Learning Using Negative Samples

39 Jul 31, 2022
Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation.

Understanding Minimum Bayes Risk Decoding This repo provides code and documentation for the following paper: Müller and Sennrich (2021): Understanding

ZurichNLP 13 May 01, 2022
Deep Dual Consecutive Network for Human Pose Estimation (CVPR2021)

Beanie - is an asynchronous ODM for MongoDB, based on Motor and Pydantic. It uses an abstraction over Pydantic models and Motor collections to work wi

295 Dec 29, 2022