【SDSS】SQL search through SDSS website

This is just a note from my work.

SDSS search website:
      http://www.sdss3.org/dr10/
      http://classic.sdss.org/dr7/
  DR10包括DR8和DR9, 但是DR7以及之前的结果都需要在DR7里面搜索。
(1) search particular objects
  http://data.sdss3.org/bulkSpectra/
(2) search objects with certain criterion
  http://skyservice.pha.jhu.edu/casjobs/default.aspx


********************************************************************************
          DR10
********************************************************************************
commands
search at website http://skyserver.sdss3.org/CasJobs/MyDB.aspx or from individual sites.
********************************************************************************
http://skyserver.sdss3.org/dr10/en/help/docs/realquery.aspx
-- a) Finding galaxies by their emission lines:
-- This query selects galaxy spectra with high internal reddening,
-- as measured by the standard Balmer decrement technique. It
-- makes use of the galSpec tables for the measurements of
-- galaxy lines. In this case we use galSpecLine, which has
-- emission line measurements.

SELECT
s.specObjID, s.plate, s.fiberid, s.mjd, s.z, s.zwarning,
g.h_beta_flux, g.h_beta_flux_err,
g.h_alpha_flux, g.h_alpha_flux_err,
g.oiii_5007_flux, g.oiii_5007_flux_err,
g.oiii_5007_eqw
FROM GalSpecLine AS g
  JOIN SpecObj AS s ON s.specobjid = g.specobjid
WHERE
  --h_alpha_flux > h_alpha_flux_err*5
  --AND h_beta_flux > h_beta_flux_err*5
  --AND h_beta_flux_err > 0
  --AND h_alpha_flux > 10.*h_beta_flux
  oiii_5007_flux > oiii_5007_flux_err*5
  AND oiii_5007_eqw > 600.
  AND s.class = 'GALAXY'
  AND s.zwarning = 0

********************************************************************************
          DR7
********************************************************************************
http://cas.sdss.org/dr7/en/tools/search/sql.asp

SELECT
G.objId, S.plate, S.fiberID, S.mjd, S.z,  L1.ew,L1.ewerr, L2.ew,L2.ewerr,L1.ew*L1.continuum,L1.ewerr*L1.continuum

FROM Galaxy as G
   JOIN SpecObj as S ON G.objId=S.bestObjId
   JOIN SpecLine as hL1 ON S.specObjId=L1.specObjId
   JOIN SpecLine as L2 ON S.specObjId=L2.specObjId  
   JOIN SpecLine as Lb ON S.specObjId=Lb.specObjId
WHERE
     L1.LineId = 5008
     and L1.ew > 1000.
     and L1.ew/L1.ewerr > 3.
     and L2.LineId =6565
     and Lb.LineId =4863
     --and log10( (L1.ew*L1.continuum)/(Lb.ew*Lb.continuum) ) < 0.75


********************************************************************************
********************************************************************************
Check image:
   http://cas.sdss.org/dr7/en/tools/chart/list.asp
   http://skyserver.sdss3.org/public/en/tools/chart/listinfo.aspx

********************************************************************************
# Example script for looking at BOSS spectra and redshift fits via Python.
#
# Written by Adam S. Bolton, University of Utah, Oct. 2009
#

# Imports:
import numpy as n
import pyfits as pf
import matplotlib as mpl
mpl.use('TkAgg')
mpl.interactive(True)
from matplotlib import pyplot as p

# Set topdir:
topdir = '/data/BOSS/spectro/redux/v5_5_12/'

# Pick your plate/mjd and read the data:
plate = '3621'
mjd = '55104'
spfile = topdir + plate + '/spPlate-' + plate + '-' + mjd + '.fits'
zbfile = topdir + plate + '/spZbest-' + plate + '-' + mjd + '.fits'
hdulist = pf.open(spfile)
c0 = hdulist[0].header['coeff0']
c1 = hdulist[0].header['coeff1']
npix = hdulist[0].header['naxis1']
wave = 10.**(c0 + c1 * n.arange(npix))
# Following commented-out bit was needed for some of the early redux:
#bzero = hdulist[0].header['bzero']
bunit = hdulist[0].header['bunit']
flux = hdulist[0].data
ivar = hdulist[1].data
hdulist.close()
hdulist = 0
hdulist = pf.open(zbfile)
synflux = hdulist[2].data
zstruc = hdulist[1].data
hdulist.close()
hdulist = 0

i = 499
# Set starting fiber point (above), then copy and paste
# the following repeatedly to loop over spectra:
i+=1
# Following commented-out bit was needed for some of the early redux:
#p.plot(wave, (flux[i,:]-bzero) * (ivar[i,:] > 0), 'k', hold=False)
p.plot(wave, flux[i,:] * (ivar[i,:] > 0), 'k', hold=False)
p.plot(wave, synflux[i,:], 'g', hold=True)
p.xlabel('Angstroms')
p.ylabel(bunit)
p.title(zstruc[i].field('class') + ', z = ' + str(zstruc[i].field('z')))




********************************************************************************
  SDSS get the images and the spectra
********************************************************************************
One example from Peter Erwin:   http://www.mpe.mpg.de/~erwin/code/telarchive/
# This is the URL of the SDSS images ( from astroML)
http://www.astroml.org/sklearn_tutorial/auto_examples/plot_sdss_images.html

url = ("http://casjobs.sdss.org/ImgCutoutDR7/"
           "getjpeg.aspx?ra=%.8f&dec=%.8f&scale=%.2f&width=%i&height=%i"
           % (RA, DEC, scale, width, height))

However, this link does not work now. Please use :
SDSS_URL = ('http://skyservice.pha.jhu.edu/DR7/ImgCutout/getjpeg.aspx?'
            'ra=%(RA).8f&dec=%(DEC).8f&scale=%(scale).2f&width=%(width)i&height=%(height)i&opt=&query=')

# This is the URL of the sdss fits spectra (from astroML)
FITS_FILENAME = 'spSpec-%(mjd)05i-%(plate)04i-%(fiber)03i.fit'
SDSS_URL = ('http://das.sdss.org/spectro/1d_26/%(plate)04i/1d/spSpec-%(mjd)05i-%(plate)04i-%(fiber)03i.fit')



********************************************************************************
  DR7   about the spectra files 
********************************************************************************
http://classic.sdss.org/dr7/products/spectra/read_spSpec.html
The spSpec file's header contains several parameters such as object identification and coordinates, observing information, spectral classification, redshift, etc..
The fits image contains a 4x(roughly 4000) image whose
  first row is the spectrum,
  second the continuum subtracted spectrum,
  third the error,
  forth a bitmask.
  fifth ?   sky light ?
Notice that the wavelength vector is not contained in the image, but must be generated from parameters in the header.
SDSS spectra are binned in constant Log(Λ ) and the wavelength can be obtained from the header parameters COEFF0 and COEFF1 (or alternatively CRVAL1 and CD1_1) as follows:

lambda = 10**(COEFF0 + COEFF1*i), where i denotes the (zero indexed) pixel number.

Six binary tables are included, most importantly,
 HDU 2 which contains a table of gaussian fits to line positions,
 HDU 3 which contains a table on line index parameters. Also included, but of less general interest, are:
 HDU 1 which contains line parameters used in the emission-line redshift determination;
 HDU 3 which contains information on emission-line redshifts;
 HDU 4 which contains a table of cross-correlation redshifts with all the templates, and
 HDU 6 which has additional mask information as well as the spectral resolution as a function of wavelength.

【PYTHON】Python中filter、map、reduce、lambda 的用法

filter(function, sequence)：对sequence中的item依次执行function(item)，将执行结果为True的item组成一个List/String/Tuple（取决于sequence的类型）返回：

>>> def f(x): return x % 2 != 0 and x % 3 != 0 
>>> filter(f, range(2, 25)) 

[5, 7, 11, 13, 17, 19, 23]
>>> def f(x): return x != 'a' 
>>> filter(f, "abcdef") 
'bcdef'

map(function, sequence) ：对sequence中的item依次执行function(item)，见执行结果组成一个List返回：
>>> def cube(x): return x*x*x 
>>> map(cube, range(1, 11)) 
[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]
>>> def cube(x) : return x + x 
... 
>>> map(cube , "abcde") 
['aa', 'bb', 'cc', 'dd', 'ee']
另外map也支持多个sequence，这就要求function也支持相应数量的参数输入：
>>> def add(x, y): return x+y 
>>> map(add, range(8), range(8)) 
[0, 2, 4, 6, 8, 10, 12, 14]

reduce(function, sequence, starting_value)：对sequence中的item顺序迭代调用function，如果有starting_value，还可以作为初始值调用，例如可以用来对List求和：
>>> def add(x,y): return x + y 
>>> reduce(add, range(1, 11)) 
55 （注：1+2+3+4+5+6+7+8+9+10）
>>> reduce(add, range(1, 11), 20) 
75 （注：1+2+3+4+5+6+7+8+9+10+20）

lambda：这是Python支持一种有趣的语法，它允许你快速定义单行的最小函数，类似与C语言中的宏，这些叫做lambda的函数，是从LISP借用来的，可以用在任何需要函数的地方： 
>>> g = lambda x: x * 2 
>>> g(3) 
6 
>>> (lambda x: x * 2)(3) 
6


我们也可以把filter map reduce 和lambda结合起来用，函数就可以简单的写成一行。
例如
kmpathes = filter(lambda kmpath: kmpath,                  
map(lambda kmpath: string.strip(kmpath),
string.split(l, ':')))              
看起来麻烦，其实就像用语言来描述问题一样，非常优雅。
对 l 中的所有元素以':'做分割，得出一个列表。对这个列表的每一个元素做字符串strip，形成一个列表。对这个列表的每一个元素做直接返回操作(这个地方可以加上过滤条件限制)，最终获得一个字符串被':'分割的列表，列表中的每一个字符串都做了strip，并可以对特殊字符串过滤。

[转] http://hi.baidu.com/black/item/307001d18715fc322a35c747

---------------------------------------------------------------

lambda表达式返回一个函数对象

例子：

func = lambda x,y:x+y

func相当于下面这个函数

def func(x,y):

    return x+y

 

注意def是语句而lambda是表达式

下面这种情况下就只能用lambda而不能用def

[(lambda x:x*x)(x) for x in range(1,11)]

 

map，reduce，filter中的function都可以用lambda表达式来生成！

 

map(function,sequence)

把sequence中的值当参数逐个传给function，返回一个包含函数执行结果的list。

如果function有两个参数，即map(function,sequence1,sequence2)。

 

例子：

求1*1,2*2,3*3,4*4

map(lambda x:x*x,range(1,5))

返回值是[1,4,9,16]

 

reduce(function,sequence)

function接收的参数个数只能为2

先把sequence中第一个值和第二个值当参数传给function，再把function的返回值和第三个值当参数传给

function，然后只返回一个结果。

 

例子：

求1到10的累加

reduce(lambda x,y:x+y,range(1,11))

返回值是55。

 

filter(function,sequence)

function的返回值只能是True或False

把sequence中的值逐个当参数传给function，如果function(x)的返回值是True，就把x加到filter的返回值里面。一般来说filter的返回值是list，特殊情况如sequence是string或tuple，则返回值按照sequence的类型。

 

例子：

找出1到10之间的奇数

filter(lambda x:x%2!=0,range(1,11))

返回值

[1,3,5,7,9]

 

如果sequence是一个string

filter(lambda x:len(x)!=0,'hello')返回'hello'

filter(lambda x:len(x)==0,'hello')返回''

SCI常见经典句型[转]

Beginning 
    1. In this paper, we focus on the need for
    2. This paper proceeds as follow.
    3. The structure of the paper is as follows.
    4. In this paper, we shall first briefly introduce fuzzy sets and related concepts
    5. To begin with we will provide a brief background on the

    Introduction 
    1. This will be followed by a description of the fuzzy nature of the problem and a detailed presentation of how the required membership functions are defined.
    2. Details on xx and xx are discussed in later sections.
    3. In the next section, after a statement of the basic problem, various situations involving possibility knowledge are investigated: first, an entirely possibility model is proposed; then the cases of a fuzzy service time with stochastic arrivals and non fuzzy service rule is studied; lastly, fuzzy service rule are considered.

    Review 
    1. This review is followed by an introduction.
    2. A brief summary of some of the relevant concepts in xxx and xxx is presented in Section 2.
    3. In the next section, a brief review of the .... is given.
    4. In the next section, a short review of ... is given with special regard to ...
    5. Section 2 reviews relevant research related to xx.
    6. Section 1.1 briefly surveys the motivation for a methodology of action, while 1.2 looks at the difficulties posed by the complexity of systems and outlines the need for development of possibility methods.

    Body 
    1. Section 1 defines the notion of robustness, and argues for its importance.
    2. Section 1 devoted to the basic aspects of the FLC decision making logic.
    3. Section 2 gives the background of the problem which includes xxx
    4. Section 2 discusses some problems with and approaches to, natural language understanding.
    5. Section 2 explains how flexibility which often ... can be expressed in terms of fuzzy time window
    6. Section 3 discusses the aspects of fuzzy set theory that are used in the ...
    7. Section 3 describes the system itself in a general way, including the ….. and also discusses how to evaluate system performance.
    8. Section 3 describes a new measure of xx.
    9. Section 3 demonstrates the use of fuzzy possibility theory in the analysis of xx.
    10. Section 3 is a fine description of fuzzy formulation of human decision.
    11. Section 3, is developed to the modeling and processing of fuzzy decision rules
    12. The main idea of the FLC is described in Section 3 while Section 4 describes the xx strategies.
    13. Section 3 and 4 show experimental studies for verifying the proposed model.
    14. Section 4 discusses a previous fuzzy set based approach to cost variance investigation.
    15. Section 4 gives a specific example of xxx.
    16. Section 4 is the experimental study to make a fuzzy model of memory process.
    17. Section 4 contains a discussion of the implication of the results of Section 2 and 3.
    18. Section 4 applies this fuzzy measure to the analysis of xx and illustrate its use on experimental data.
    19. Section 5 presents the primary results of the paper: a fuzzy set model ..
    20. Section 5 contains some conclusions plus some ideas for further work.
    21. Section 6 illustrates the model with an example.
    22. Various ways of justification and the reasons for their choice are discussed very briefly in Section 2.
    23. In Section 2 are presented the block diagram expression of a whole model of human DM system
    24. In Section 2 we shall list a collection of basic assumptions which a ... scheme must satisfy.
    25. In Section 2 of this paper, we present representation and uniqueness theorems for the fundamental measurement of fuzziness when the domain of discourse is order dense.
    26. In Section 3, we describe the preliminary results of an empirical study currently in progress to verify the measurement model and to construct membership functions.
    27. In Section 5 is analyzed the inference process through the two kinds of inference experiments...
This Section
    1. In this section, the characteristics and environment under which MRP is designed are described.
    2. We will provide in this section basic terminologies and notations which are necessary for the understanding of subsequent results.Next Section
    2. The next section describes the mathematics that goes into the computer implementation of such fuzzy logic statements.
    3. However, it is cumbersome for this purpose and in practical applications the formulae were rearranged and simplified as discussed in the next section.
    4. The three components will be described in the next two section, and an example of xx analysis of a computer information system will then illustrate their use.
    5. We can interpret the results of Experiments I and II as in the following sections.
    6. The next section summarizes the method in a from that is useful for arguments based on xx

    Summary 
    1. This paper concludes with a discussion of future research consideration in section 5.
    2. Section 5 summarizes the results of this investigation.
    3. Section 5 gives the conclusions and future directions of research.
    4. Section 7 provides a summary and a discussion of some extensions of the paper.
    5. Finally, conclusions and future work are summarized
    6. The basic questions posed above are then discussed and conclusions are drawn.
    7. Section 7 is the conclusion of the paper.

    Chapter 0. Abstract 
    1. A basic problem in the design of xx is presented by the choice of a xx rate for the measurement of experimental variables.
    2. This paper examines a new measure of xx in xx based on fuzzy mathematics which overcomes the difficulties found in other xx measures.
    3. This paper describes a system for the analysis of the xx.
    4. The method involves the construction of xx from fuzzy relations.
    5. The procedure is useful in analyzing how groups reach a decision.
    6. The technique used is to employ a newly developed and versatile xx algorithm.
    7. The usefulness of xx is also considered.
    8. A brief methodology used in xx is discussed.
    9. The analysis is useful in xx and xx problem.
    10. A model is developed for a xx analysis using fuzzy matrices.
    11. Algorithms to combine these estimates and produce a xx are presented and justified.
    12. The use of the method is discussed and an example is given.
    13. Results of an experimental applications of this xx analysis procedure are given to illustrate the proposed technique.
    14. This paper analyses problems in
    15. This paper outlines the functions carried out by ...
    16. This paper includes an illustration of the ...
    17. This paper provides an overview and information useful for approaching
    18. Emphasis is placed on the construction of a criterion function by which the xx in achieving a hierarchical system of objectives are evaluated.
    19. The main emphasis is placed on the problem of xx
    20. Our proposed model is verified through experimental study.
    21. The experimental results reveal interesting examples of fuzzy phases of: xx, xx
    22. The compatibility of a project in terms of cost, and xx are likewise represented by linguistic variables.
    23. A didactic example is included to illustrate the computational procedure

    Chapter 1. Introduction 
    Time
    1. Over the course of the past 30 years, .. has emerged form intuitive
    2. Technological revolutions have recently hit the industrial world
    3. The advent of ... systems for has had a significant impact on the
    4. The development of ... is explored
    5. During the past decade, the theory of fuzzy sets has developed in a variety of directions
    6.The concept of xx was investigated quite intensively in recent years
    7. There has been a turning point in ... methodology in accordance with the advent of ...
    8. A major concern in ... today is to continue to improve...
    9. A xx is a latecomer in the part representation arena.
    10. At the time of this writing, there is still no standard way of xx
    11. Although a lot of effort is being spent on improving these weaknesses, the efficient and effective method has yet to be developed.
    12. The pioneer work can be traced to xx [1965].
    13. To date, none of the methods developed is perfect and all are far from ready to be used in commercial systems.

    Objective / Goal / Purpose 
    1. The purpose of the inference engine can be outlined as follows:
    2. The ultimate goal of the xx system is to allow the non experts to utilize the existing knowledge in the area of manual handling of loads, and to provide intelligent, computer aided instruction for xxx.
    3. The paper concerns the development of a xx
    4. The scope of this research lies in
    5. The main theme of the paper is the application of rule based decision making.
    6. These objectives are to be met with such thoroughness and confidence as to permit ...
    7. The objectives of the ... operations study are as follows:
    8. The primary purpose/consideration/objective of
    9. The ultimate goal of this concept is to provide
    10. The main objective of such a ... system is to
    11. The aim of this paper is to provide methods to construct such probability distribution.
    12. In order to achieve these objectives, an xx must meet the following requirements:
    13. In order to take advantage of their similarity
    14. more research is still required before final goal of ... can be completed
    15. In this trial, the objective is to generate...
    16. for the sake of concentrating on ... research issues
    17. A major goal of this report is to extend the utilization of a recently developed procedure for the xx.
    18. For an illustrative purpose, four well known OR problems are studied in presence of fuzzy data: xx.
    19. A major thrust of the paper is to discuss approaches and strategies for structuring ..methods
    20. This illustration points out the need to specify
    21. The ultimate goal is both descriptive and prescriptive.
    22. Chapter 2. Literature Review
    23. A wealth of information is to be found in the statistics literature, for example, regarding xx
    24. A considerable amount of research has been done .. during the last decade
    25. A great number of studies report on the treatment of uncertainties associated with xx.
    26. There is considerable amount of literature on planning
    27. However, these studies do not provide much attention to uncertainty in xx.
    28. Since then, the subject has been extensively explored and it is still under investigation as well in
    methodological aspects as in concrete applications.
    29. Many research studies have been carried out on this topic.
    30. Problem of xx draws recently more and more attention of system analysis.
    31. Attempts to resolve this dilemma have resulted in the development of
    32. Many complex processes unfortunately, do not yield to this design procedure and have, therefore, not yet been automated.
    33. Most of the methods developed so far are deterministic and /or probabilistic in nature.
    34. The central issue in all these studies is to
    35. The problem of xx has been studied by other investigators, however, these studies have been based upon classical statistical approaches.
    36. Applied ... techniques to
    37. Characterized the ... system as
    38. Developed an algorithm to
    39. Developed a system called ... which
    40. Uses an iterative algorithm to deduce
    41. Emphasized the need to
    42. Identifies six key issues surrounding high technology
    43. A comprehensive study of the... has been undertaken
    44. Much work has been reported recently in these filed
    45. Proposed/Presented/State that/Described/Illustrated/
    Indicated/Has shown / showed/Address/Highlights
    46. Point out that the problem of
    47. A study on ...was done / developed by []
    48. Previous work, such as [] and [], deal only with
    49. The approach taken by [] is
    50. The system developed by [] consists
    51. A paper relevant to this research was published by []
    52. []'s model requires consideration of...
    53. []' model draws attention to evolution in human development
    54. []'s model focuses on...
    55. Little research has been conducted in applying ... to
    56. The published information that is relevant to this research...
    57. This study further shows that
    58. Their work is based on the principle of
    59. More history of ... can be found in xx et al. [1979].
    60. Studies have been completed to established
    61. The ...studies indicated that
    62. Though application of xx in the filed of xx has proliferated in recent years, effort in analyzing xx, especially xx, is lacking.
    Problem / Issue / Question
    63. Unfortunately, real-world engineering problems such as manufacturing planning do not fit well with this narrowly defined model. They tend to span broad activities and require consideration of multiple aspects.
    64. Remedy / solve / alleviate these problems
    67. ... is a difficult problem, yet to be adequately resolved
    68. Two major problems have yet to be addressed
    69. An unanswered question
    70. This problem in essence involves using x to obtain a solution.
    71. An additional research issue to be tackled is ....
    72. Some important issues in developing a ... system are discussed
    73. The three prime issues can be summarized:
    74. The situation leads to the problem of how to determine the ...
    75. There have been many attempts to
    76. It is expected to be serious barrier to
    77. It offers a simple solution in a limited domain for a complex

Luminosity Fuction

Usually, people use Schechter parameterization to show the luminosity functions.

From WIKI:

Schechter luminosity function[edit]

The Schechter luminosity function provides a parametric description of the space density of galaxies as a function of their luminosity. The form of the function is

where , and  is a characteristic galaxy luminosity where the power-law form of the function cuts off. The parameter  has units of number density and provides the normalization. The galaxy luminosity function may have different parameters for different populations and environments; it is not a universal function. One measurement from field galaxies is .^[2]

It is often more convenient to rewrite the Schechter function in terms of magnitudes, rather than luminosities. In this case, the Schechter function becomes:

Note that because the magnitude system is logarithmic, the power law has logarithmic slope . This is why a Schechter function with  is said to be flat.

Notice, in the papers, people use a instead of a+1.

Some reference:

(1) In python. you can calculate the n(M) using the parameters of z=1.5 from Oesch et al. 2010

(2) More star formation may occur in even lower-mass galaxies having higher EWHα and that dominate the UV luminosity function at z ∼ 2 (Fumagalli et al. 2012; Alavi et al. 2013). 

Fumagalli et al. 2012   Hα Equivalent Widths from the 3D-HST Survey: Evolution with Redshift and Dependence on Stellar Mass 

Alavi et al. 2013   Ultra-faint Ultraviolet Galaxies at z ~ 2 behind the Lensing Cluster A1689: The Luminosity Function, Dust Extinction, and Star Formation Rate Density

1 2 3 4 5

phi = -2.64 # normalization M = -19.82 # the standard luminosity a = -1.46 # slope X= M # the luminosity you want to calculate 10**phi*(np.log(10)/2.5) * ((10**(-0.4*(X-M)))**(a)) * np.e**(-10**(-0.4*(X-M)))