# Notebook from RuthAngus/stardate
Path: docs/source/_static/notebooks/Tutorial.ipynb

# A quick stardate tutorial: measuring the ages of rotating stars_____no_output_____In this tutorial we'll infer the ages of some Kepler targets with measured rotation periods.

We'll start by downloading the McQuillan *et al.* (2014) (https://arxiv.org/abs/1402.5694) catalogue of stellar rotation periods._____no_output_____

import pandas as pd

url = "https://arxiv.org/src/1402.5694v2/anc/Table_1_Periodic.txt"
mc = pd.read_csv(url)_____no_output_____

In order to get some photometric colors for these stars (which is the minimum requirement to measure a *stardate* age), as well as some parallaxes, which will improve the age estimate, let's find these stars in the Gaia DR2 catalogue. Thankfully, we don't have to crossmatch targets ourselves because Megan Bedell has already done it for us (check out https://gaia-kepler.fun). Her table also contains useful information from the Kepler input catalogue. Let's download the Gaia-Kepler crossmatch._____no_output_____

import astropy.utils as au
from astropy.io import fits

gaia_url = "https://dl.dropboxusercontent.com/s/xo1n12fxzgzybny/kepler_dr2_1arcsec.fits?dl=0"

with fits.open(gaia_url) as data:
    gaia = pd.DataFrame(data[1].data, dtype="float64")_____no_output_____

Now let's merge these two data frames to make one data frame containing rotation periods, Gaia parallaxes, colours, etc._____no_output_____

df = pd.merge(mc, gaia, left_on="KID", right_on="kepid", how="inner")_____no_output_____

Now we have everything we need (and more!) to start measuring ages. First, let's plot these stars on a (crude) colour-magnitude diagram and colour them by their rotation periods._____no_output_____

import matplotlib.pyplot as plt
%matplotlib inline

plotpar = {'axes.labelsize': 25,
           'xtick.labelsize': 20,
           'ytick.labelsize': 20,
           'text.usetex': True}
plt.rcParams.update(plotpar)

import numpy as np

def m_to_M(m, D):
    """
    Convert apparent magnitude to absolute magnitude.
    """
    return m - 5*np.log10(abs(D)) + 5

abs_G = m_to_M(df.phot_g_mean_mag, (1./df.parallax)*1e3)

plt.figure(figsize=(16, 9))
plt.scatter(df.phot_bp_mean_mag - df.phot_rp_mean_mag, abs_G,
            c=df.Prot, s=10, alpha=.1, vmin=0, vmax=40);
plt.gca().invert_yaxis()
plt.ylim(12, -1)
plt.colorbar(label="$\mathrm{Rotation~period~[days]}$")
plt.xlabel("$G_{BP} - G_{RP}$")
plt.ylabel("$\mathrm{Absolute~G~magnitude}$");

np.random.seed(1234)
ind = np.random.randint(0, len(df))
plt.scatter([df.phot_bp_mean_mag[ind] - df.phot_rp_mean_mag[ind]], [abs_G[ind]], c=[df.Prot[ind]],
            s=100, vmin=0, vmax=40, edgecolor="w");_____no_output_____

In this figure you can see that the lowest mass stars (bottom right) are rotating slowly and stars rotate more and more rapidly as they increase in mass. On the main sequence, stars of the same mass rotate more slowly as they get older and move up the y-axis. You can also see the binary sequence. On average, binary stars spin more rapidly than single stars. In this version of stardate we do not account for the possibility that a star might be in a binary._____no_output_____# Inferring a stellar age

Now let's measure the age of a random star in this data set. We'll use the randomly chosen star plotted as a large circle on the CMD above._____no_output_____We'll start by creating a dictionary of observables for this star. We'll use Gaia magnitudes *and* 2MASS JHK magnitudes (because they're available from the Kepler input catalogue) but one set of magnitudes would have worked too. These are the observables that provide information about the amount of hydrogen left in a star's core (i.e. placement on a stellar evolution track) which will be passed to the *isochrones* isochrone fitting algorithm. These observables could include photometric colours, atmospheric properties (effective temperature, surface gravity and metallicity), parallax, and even asteroseismic parameters. For most Kepler stars, only broad-band photometry is available so that's what we'll use here._____no_output_____

import stardate as sd

iso_params = {"G": (df.phot_g_mean_mag[ind], .05),  # We'll just estimate the uncertainties for now.
              "bp": (df.phot_bp_mean_mag[ind], .05),
              "rp": (df.phot_rp_mean_mag[ind], .05),
              "J": (df.jmag[ind], .05),
              "H": (df.hmag[ind], .05),
              "K": (df.kmag[ind], .05),
              "parallax": (df.parallax[ind], df.parallax_error[ind])}  # Parallax in milliarcseconds._____no_output_____

This error message comes from the *isochrones* package and we're not going to use MultiNest so it's okay to ignore it! Pay attention to the exact format of the iso_params dictionary. The *isochrones* package requires it to be in exactly this format. Observables should be tuples containing values and uncertainties and this should **not be a pandas dictionary!** In addition, a new iso_params dictionary should be created for every star -- *isochrones* will not currently accept arrays of multiple stars). I'll walk you through running *stardate* on multiple stars later in this tutorial.

Now let's set up the star object. This is where we'll add the rotation period (Prot) and rotation period uncertainty (Prot_err)._____no_output_____

# If you don't want to automatically save samples to disk (much faster):
star = sd.Star(iso_params, prot=df.Prot[ind], prot_err=df.Prot_err[ind])

# If you do want to save samples to disk (slower but assesses convergence,
# although this feature is still in beta), you can provide a directory and filename for the file location.
star = sd.Star(iso_params, prot=df.Prot[ind], prot_err=df.Prot_err[ind],
               savedir=".", filename="{}".format(df.KID[ind]));_____no_output_____

If you want to save samples automatically, the savedir argument should be the path to the directory you'd like the posterior samples to be saved in. The default is the current working directory. The filename is the name you'd like to give to the h5 file that will contain the saved posterior samples. The default is "samples". It's useful to set this to the name or id of the star if you're running *stardate* on multiple stars.

If you want to run without saving samples, you can still save the samples later manually using h5py or similar.

If you know the v-band extinction you can pass that to the Star object too (I recommend using the [dustmaps](https://dustmaps.readthedocs.io/en/latest/) package to calculate this), e.g.:_____no_output_____

star = sd.Star(iso_params, prot=df.Prot[ind], prot_err=df.Prot_err[ind], Av=.2, Av_err=.01,
               savedir=".", filename="{}".format(df.KID[ind]));_____no_output_____

Now all we need to do is run the MCMC and wait._____no_output_____

nsteps = 10000
star.fit(max_n=nsteps)  # max_n is the number of MCMC steps to take.100%|██████████| 10000/10000 [02:23<00:00, 69.50it/s]

If you want to save the samples automatically this becomes the following (but note this takes a lot more time for the same number of samples because of the I/O overhead):_____no_output_____

star.fit(max_n=nsteps, save_samples=True)  # max_n is the number of MCMC steps to take but only
                                           # 1/thin_by of these will be saved to file.
                                           # The default thin_by value is 100.100%|██████████| 10000/10000 [02:24<00:00, 70.78it/s]

# Accessing and plotting the results.

Let's print the median value of age and it's 16th and 84th percentile uncertainties._____no_output_____

median_age, age_errp, age_errm, age_samples = star.age_results()
print("Age = {0:.2f} + {1:.2f} - {2:.2f} Gyr".format(median_age, age_errp, age_errm))Age = 3.95 + 0.26 - 3.41 Gyr

We can do the same thing for mass, metallicity, distance and extinction:_____no_output_____

burnin = 10  # If you saved samples automatically, burnin is actually thin_by x larger than this.
             # If you didn't save samples, your probably want your burnin to be higher than this.

mass, mass_errp, mass_errm, mass_samples = star.mass_results(burnin=burnin)
print("Mass = {0:.2f} + {1:.2f} - {2:.2f} M_sun".format(mass, mass_errp, mass_errm))

feh, feh_errp, feh_errm, feh_samples = star.feh_results(burnin=burnin)
print("feh = {0:.2f} + {1:.2f} - {2:.2f}".format(feh, feh_errp, feh_errm))

lndistance, lndistance_errp, lndistance_errm, lndistance_samples = star.distance_results(burnin=burnin)
print("ln(distance) = {0:.2f} + {1:.2f} - {2:.2f} ".format(lndistance, lndistance_errp, lndistance_errm))

Av, Av_errp, Av_errm, Av_samples = star.Av_results(burnin=burnin)
print("Av = {0:.2f} + {1:.2f} - {2:.2f}".format(Av, Av_errp, Av_errm))

# You can load the flattened samples with
samples = star.samples
print("Sample array shape = ", np.shape(samples))Mass = 0.79 + 0.04 - 0.02 M_sun
feh = -0.12 + 0.08 - 0.11
ln(distance) = 6.21 + 0.01 - 0.01
Av = 0.20 + 0.01 - 0.01
Sample array shape =  (5000, 5)

If you want to load samples from an H5 file that you automatically saved earlier, you can do the following._____no_output_____

from stardate import load_samples, read_samples

# Load the samples.
flatsamples, _3Dsamples, posterior_samples, prior_samples = load_samples(
    "{}.h5".format(df.KID[ind]), burnin=0)

# Extract the median and maximum likelihood parameter estimates from the samples.
results = read_samples(flatsamples)

# Print the results as a pandas dataframe.
results_____no_output_____

flatsamples is a 2D array of samples, useful for making corner plots. samples is a 3D array of samples, useful for plotting chains. Both flatsamples and samples have an extra dimension containing the log-probability.

results is a pandas dictionary containing best-fit stellar parameters. It has both the median and maximum-likelihood values. I recommend using maximum-likelihood (suffixed with 'ml') values for stars with Gaia G_BP - G_RP color < 1.3 and median values (suffixed with 'med') for everything else.

Age is in gyr and distance is in pc. EEP is dimensionless. feh and Av are in dex._____no_output_____Now we'll plot a histogram of the marginalized posterior over stellar age._____no_output_____

plt.hist((10**age_samples)*1e-9, 100);
plt.xlabel("$\mathrm{Age~[Gyr]}$")
plt.xlim(3, 7);_____no_output_____

And let's make a corner plot._____no_output_____

import corner

labels = ["EEP", "log10(Age [yr])", "[Fe/H]", "ln(Distance)", "Av", "ln(probability)"]

corner.corner(flatsamples, labels=labels);WARNING:root:Too few points to create valid contours

# Multiple stars

Looping over multiple stars could look something like this:_____no_output_____

N = 10  # Measure ages for the first 10 stars in the McQuillan catalog
ndim = 5  # There are 5 dimensions: EEP, age, feh, distance and Av
results = np.empty((N, ndim))  # An array to store summary statistics
ages, masses = [np.empty(N) for i in range(2)]
nsteps = 100   # You should set max_n much higher than this when you run for real.
nwalkers = 50  # stardate uses 50 walkers
sample_array = np.zeros((nsteps*nwalkers, ndim, N))

for i in range(N):

    # You have to set up the parameter dictionary for every star.
    iso_params = {"G": (df.phot_g_mean_mag[i], .05),
                  "bp": (df.phot_bp_mean_mag[i], .004),
                  "rp": (df.phot_rp_mean_mag[i], .004),
                  "J": (df.jmag[i], .05),
                  "H": (df.hmag[i], .05),
                  "K": (df.kmag[i], .05),
                  "parallax": (df.parallax[i], df.parallax_error[i])}  # Parallax in milliarcseconds.

    star = sd.Star(iso_params, df.Prot[i], df.Prot_err[i])  # , filename="{}".format(df.KID[i]));
    star.fit(max_n=nsteps)

    sample_array[:, :, i] = star.samples
    results[i, :] = np.median(star.samples, axis=0)

    # Or you could use the built-in functions to calculate medians.
    ages[i] = star.age_results()[0]
    masses[i] = star.mass_results()[0]100%|██████████| 100/100 [00:01<00:00, 70.12it/s]
  7%|▋         | 7/100 [00:00<00:01, 67.40it/s]print("Ages = ", results[:, 1], "\n")  # These ages are in log10(yrs)

print("Equivalent Evolutionary Points = ", results[:, 0], "\n")

print("Metallicities = ", results[:, 2], "\n")

print("Distances = ", results[:, 3], "\n")  # These are in ln(pc)

print("A_vs = ", results[:, 4], "\n")Ages =  [9.51112504 9.90461703 9.56008607 9.55981386 9.73172604 9.62600584
 9.58434138 9.66127084 9.57374104 9.5595891 ]

Equivalent Evolutionary Points =  [329.57897528 329.52641737 329.57626038 329.35732305 329.7935348
 329.3695938  329.57982176 329.57296482 330.14254528 329.57980668]

Metallicities =  [-0.13635481 -0.20219718 -0.1029566  -0.0784935   0.04638792 -0.25294648
 -0.19802885 -0.19361006 -0.02608147 -0.04899971]

Distances =  [5.56174874 5.71289638 5.572407   5.77557608 5.74169493 5.56985369
 5.71452286 5.61047538 5.71280237 5.56961465]

A_vs =  [0.03380475 0.18007945 0.03253148 0.08023092 0.228755   0.09026418
 0.38374834 0.01566177 0.51577132 0.00850142]

# Calculating other physical stellar parameters._____no_output_____If you want to calculate a mass from a EEP, age and metallicity samples, you can do the following:_____no_output_____

from isochrones.mist import MIST_Isochrone
mist = MIST_Isochrone()

# Flatten sample array for the first star, computed above
flatsamples = np.reshape(sample_array[:, :, 0], (nsteps*nwalkers, ndim))

eep_samples = flatsamples[:, 0]
log_age_samples = flatsamples[:, 1]
feh_samples = flatsamples[:, 2]

mass_samps = mist.interp_value([eep_samples, log_age_samples, feh_samples], ["mass"])
print("mass = {0:.2f} M_sun".format(np.median(mass_samps)))mass = 0.87 M_sun

You can use this same format to calculate other physical parameters too, e.g.:_____no_output_____

radius_samps = mist.interp_value([eep_samples, log_age_samples, feh_samples], ["radius"])
print("radius = {0:.2f} R_sun".format(np.median(radius_samps)))radius = 0.81 R_sun
teff_samps = mist.interp_value([eep_samples, log_age_samples, feh_samples], ["Teff"])
print("teff = {0:.0f} K".format(np.median(teff_samps)))teff = 5485 K
logg_samps = mist.interp_value([eep_samples, log_age_samples, feh_samples], ["logg"])
print("logg = {0:.2f}".format(np.median(logg_samps)))logg = 4.56

Take a look at the column headers of the MIST tables in the [isochrones](https://isochrones.readthedocs.io/en/latest/modelgrids.html) documentation for more physical parameters that are supported._____no_output_____# Isochrone fitting only

It's also possible to switch off gyrochronology and just infer an age using isochrones only. This is useful if you'd like to predict ages for a list of stars where only some of them have rotation periods, or if you'd like to compare between gyro on and gyro off. There are a couple of different ways to do this. The simplest way is just to pass 'None' instead of a period and period uncertainty:_____no_output_____

star = sd.Star(iso_params, prot=None, prot_err=None)_____no_output_____

Passing zeros instead of Nones, or nothing at all will also work. You can also set the 'isochrone fitting only' key word argument to be true when you run the MCMC:_____no_output_____

star.fit(max_n=1000, iso_only=True)100%|██████████| 1000/1000 [00:06<00:00, 160.22it/s]

# Incorporating asteroseismology

As well as apparant magnitudes, parallax and spectroscopic parameters, you can also provide asteroseismic parameters in the dictionary of observables. The dictionary below is the full set of parameters that can be provided. These parameters do not all need to be provided and can be given in any order. The precision and accurancy of inferred ages will improve as more information is provided. See the [*isochrones* documentation](https://isochrones.readthedocs.io/en/latest/api.html#starmodel) for more information._____no_output_____

iso_params = {"G": (G, G_err),  # Gaia magnitudes
              "bp": (G_bp, G_bp_err),
              "rp": (G_rp, G_rp_err),
              "B": (B, B_err),
              "V": (V, V_err),
              "J": (J, J_err),
              "H": (H, H_err),
              "K": (K, K_err),
              "g": (g, g_err),  # SDSS colours -- these are lower case.
              "r": (r, r_err),
              "i": (i, i_err),
              "z": (z, z_err),
              "teff": (teff, teff_err),  # Spectroscopic properties
              "logg": (logg, logg_err),
              "feh": (feh, feh_err),
              "nu_max": (nu_max, nu_max_err),  # Asteroseismic parameters
              "delta_nu": (delta_nu, delta_nu_err),  # Asteroseismic parameters
              "parallax": (parallax, parallax_error)}  # Parallax in milliarcseconds_____no_output_____


# Notebook from mimozairtem/PIC16B
Path: lectures/abm/abm-1-live.ipynb

# Agent-Based Modeling

In this set of lectures, we'll study how to design agent-based models in Python.

> An *agent-based model* (ABM) is a simulation model in which many individual entities (*agents*) interact with each other according to fixed rules.

ABMs are often used for modeling a wide range of social and biological systems. In fact, you've already seen an example of an ABM: the SIR model of disease spred that we studied in the previous lecture is one. There, we relied on tools from NetworkX and various other familiar programming paradigms. We'll now explore the topic of agent-based modeling from a somewhat more systematic and flexible perspective.

There exist a large number of dedicated software packages for agent-based modeling. In this course, we'll use a relatively recent package, called [Mesa](https://mesa.readthedocs.io/en/master/index.html), for agent-based modeling in Python. To install the software, run the following code in your terminal:

```
conda activate PIC16B
conda install -c conda-forge mesa
```

# The Schelling Model of Racial Segregation

In this set of lecture notes, we will implement the Schelling model of racial residential segregation. The Schelling model is a parable of how only *mild* individual biases can lead to highly segregated outcomes.

In the Schelling Model, individuals of two types begin arranged randomly on a grid, which is often taken to represent a city. Not all grid squares are occupied. Here's an example starting configuration

An example starting configuration in the Schelling model. Image credit:  Vi Hart and Nicky Case.

Here's how the model works:

1. At each timestep, agents look at their surroundings. An agent is **unhappy** if fewer than 1/3 of their neighbors have the same type, and is **happy** otherwise.
2. All **unhappy** agents pick a random empty spot and move there. All **happy** agents stay where they are.

We run the model until all agents are happy. The fundamental result of the model is that, even though agents have only mild biases -- they simply prefer not to be outnumbered --  acting on their preferences can still lead to highly segregated outcomes, like this:

An example final configuration in the Schelling model. Image credit:  Vi Hart and Nicky Case.

For an excellent interactive demonstration of the Schelling Model, check out [this blog post](https://ncase.me/polygons/) by Vi Hart and Nicky Case.

### A Note on History

The Schelling model does not include any concepts of historical oppression, wealth, or power, all of which contribute to racial segregation. The message of the Schelling model is that these factors are not **needed** for segregation -- mildly racist individual preferences would be enough. It is important, however, not to confuse this mathematical parable with the actual historical circumstances of racial segregation in the US or elsewhere.

### Sources

These lecture notes are closely based on the [Schelling model example](https://github.com/projectmesa/mesa/tree/main/examples/schelling) in the [official Mesa repository](https://github.com/projectmesa/mesa). They also draw on the [Introductory Tutorial](https://mesa.readthedocs.io/en/master/tutorials/intro_tutorial.html) from the official Mesa documentation.

# Implementing the Schelling Model_____no_output_____Let's start by implementing a bare-bones model. While there is some flexibility in how one does this, there are a few common features of most Mesa models:

1. There must be an *agent* class, which should inherit from `mesa.Agent`. This class specifies the properties and behaviors of an individual agent in the simulation.
    - This class must call  `mesa.Agent.__init__()` as part of its `__init__()` method.
    - This class must have a `step()` method which describes the primary individual behavior.
2. There must be a *model* class, which should inherit from `mesa.Model`.
    - The `__init__()` method of this class is responsible for creating agents with their properties, as well as the space (often a grid) on which the simulation unfolds.
    - This class must also have a `step()` method which provides a complete description of what happens in a single model time step. Often, this involves using a `Schedule` to call the `step()` method of each of the agents in some specified sequence.

Let's write a very simple model that demonstrates some of these requirements. Our model won't really do very much yet, but it will demonstrate the key techniques of defining the agent and model, adding agents to the model, and calling the `step()` methods. _____no_output_____Let's demonstrate the behavior of our toy model: _____no_output_____Observe that, each time we call `TM.step()`, the model sweeps through the various agents and calls their individual `step()` methods. This is because we created a `RandomActivation` schedule, and added each of the agents to this schedule.

With our architecture in place, our next step is learn how to implement more interesting behaviors.

## Spatial Grids

The Schelling model usually evolves on a grid. At the moment, we don't have a grid incorporated. Fortunately, this is easy to bring in. We simply need to add a `SingleGrid` object with specified width and height. The `torus` argument of the grid determines whether the edges "wrap around." If it is selected, then walking off the left side of the grid will put you back on the right side. This is often visualized as allowing the grid to lie on the surface of a torus, or donut:

A toroidal grid.

The modifications we need to make to our previous code are relatively simple:

1. We need to give each `ToyAgent` a `pos`ition.
2. We need to give the model a `grid` instance variable.
3. We need to modify our initialization of agents so that we call `self.grid.position_agent(agent, pos)` in order to place each agent on the grid. _____no_output_____Now we can again instantiate our model. This time, we need to pass `width`, `height`, and `density`. Here, we're creating a 10x10 grid in which roughly 10% of cells have agents in them. _____no_output_____

TM = ToyModel(10, 10, 0.1)
TM.step()_____no_output_____

It's also possible to directly extract the grid and visualize it using familiar tools. In a later lecture, however, we'll see some much better ways to visualize the grid. _____no_output_____

import numpy as np
from matplotlib import pyplot as plt
plt.imshow(np.array(TM.grid.grid) == None)_____no_output_____

# A Basic Schelling Model

We're now ready to construct a simple version of the Schelling model. Here are the remaining ingredients we need to bring in:

1. Agents need to have *types* associated with them.
2. The agents `step()` method should check whether the agent is "happy" (i.e. not surrounded by too many neighbors of different `type`, and move them to an empty grid cell if not. The `SingleGrid` class we've used to create the grid provides several useful methods for handling this logic.  _____no_output_____Here's a function to plot the model state. Dark purple squares are empty; green squares are agents of type `triangle`, and yellow squares are agents of type `square`. _____no_output_____

def viz_state(SM, ax):
    G = np.array(SM.grid.grid)
    to_viz = np.zeros(G.shape)

    for i in range(G.shape[0]):
        for j in range(G.shape[1]):
            if G[i,j] is not None:
                if G[i,j].type == "triangle":
                    to_viz[i,j] = 1.0
                elif G[i,j].type == "square":
                    to_viz[i,j] = 1.0
    ax.imshow(to_viz)_____no_output_____

Now we're ready to visualize the evolution of our model. _____no_output_____

fig, axarr = plt.subplots(2, 5, figsize = (10, 4))
SM = SchellingModel(20, 20, 0.9, 0.5)

t = 0
for ax in axarr.flatten():
    ax.axis("off")
    ax.set(title = f"timestep {t}")
    viz_state(SM, ax)
    t += 1
    SM.step()_____no_output_____

We observe the characteristic separation of an initially spatially mixed population into large regions of homogeneous types.

In coming lectures, we'll learn how to visualize these processes more gracefully; how to collect data from simulations; and how to implement more complex models. _____no_output_____


# Notebook from ameisner/notebooks-latest
Path: 03_ScienceExamples/EmLineGalaxies/01_EmLineGalaxies_SpectraStack.ipynb

__author__ = 'Ragadeepika Pucha <rpucha@email.arizona.edu>, Stephanie Juneau <stephanie.juneau@noirlab.edu>' # single string; emails in <>
__version__ = '20211122' # yyyymmdd; version datestamp of this notebook
__datasets__ = ['sdss_dr12', 'sdss_dr16']  # datasets used in this notebook; for available datasets, see cell "Available datasets in Data Lab" further below
__keywords__ = ['sdss spectra', 'spectra stacking'], # keywords relevant to this notebook, e.g. ['science case','dwarf galaxies'] Use only keywords from the master list: https://github.com/noaodatalab/notebooks-latest/blob/master/internal/keywords.txt_____no_output_____

# Stacking SDSS Spectra of Galaxies Selected from the BPT Diagram

*Ragadeepika Pucha, Stéphanie Juneau & The Astro Data Lab Team*_____no_output_____### Table of contents
* [Goals & Summary](#goals)
* [Disclaimer & Attribution](#attribution)
* [Imports & setup](#import)
* [Emission Lines Catalog from Portsmouth Group](#ports)
* [SDSS Spectra](#sdss)
    * [Functions to access and plot the spectra](#functions1)
    * [Plotting the spectra](#plot_spectra)
* [Ionization Sources and the BPT Diagram](#bpt)
* [Stacking Spectra](#stack_spectra)
    * [Function to align and stack spectra](#functions2)
    * [Selecting Galaxies from the BPT](#select_bpt)
    * [Plotting Stacked Spectra](#plot_stack)
* [Resources and references](#resources)_____no_output_____
# Goals
In this notebook, we will show how to access and plot spectra of SDSS galaxies (or any SDSS object given its `specobjid`). Further, we select emission-line galaxies based on their classification on a BPT diagram (named after "Baldwin, Phillips & Terlevich", the authors of the paper ([Baldwin et. al., 1981](https://ui.adsabs.harvard.edu/abs/1981PASP...93....5B/abstract)) who introduced this concept), and stack their spectra._____no_output_____# Summary

Galaxy spectra provide us with vast information about the stellar, gas and dust properties of the galaxies. The SDSS spectra are hosted at Astro Data Lab and can be accessed from the Jupyter notebook interface directly. Below, we show how to access the spectrum of any given object. We also provide a functionality that can be used to overplot emission/absorption lines on the spectra.

The different emission lines in the spectra provide information about ionization processes occuring in galaxies. These processes include star formation (SF), accretion onto an active galactic nucleus (AGN), and shocks. Using the traditional BPT diagram, galaxies can be divided into four different classes:
1. Star forming (SF)
2. Seyfert
3. LINER
4. Composite

It can be useful to stack spectra of galaxies to study the emission lines and other spectral features more clearly. Stacking spectra helps increase the signal-to-noise ratio, which in turn helps detect faint spectral signatures and better study the average properties of the galaxies involved in the stacking. In this notebook, we stack the spectra of galaxies selected from each of the four BPT classes and compare their emission lines and overall spectral properties._____no_output_____
# Disclaimer & attribution
If you use this notebook for your published science, please acknowledge the following:

* Data Lab concept paper: Fitzpatrick et al., "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, http://dx.doi.org/10.1117/12.2057445

* Data Lab disclaimer: https://datalab.noirlab.edu/disclaimers.php_____no_output_____
# Imports and setup_____no_output_____

# std lib
from getpass import getpass
import random

# 3rd party
import numpy as np

import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle
import matplotlib
%matplotlib inline

# Data Lab
from dl import authClient as ac, queryClient as qc, storeClient as sc
from dl import specClient as spec_____no_output_____# Authentication
# Much of the functionality of spectrum services can be accessed without explicitly logging into Data Lab (the services then uses an anonymous login).
# But some capabilities, for instance saving the results of your queries to your virtual storage space, require a login (i.e., you will need a registered user account).

# If you need to log in to Data Lab, uncomment the 'ac.login()' command and respond according to your prompts.
# If you have previously logged into Data Lab, this cell will simply print your active user name.

#ac.login(input("Enter user name: (+ENTER) "), getpass("Enter password: (+ENTER) "))
ac.whoAmI()_____no_output_____## Setting up plots

# Font size
font = {'size':22}
matplotlib.rc('font', **font)

# Axes width
width = {'linewidth':3.0}
matplotlib.rc('axes', **width)

# x- and y- ticks
ticks = {'major.size':6.0, 'minor.size':4.0, 'major.width':3.0, 'minor.width':1.5, 'direction':'in',\
        'minor.visible':True}
matplotlib.rc('xtick', **ticks)
matplotlib.rc('ytick', **ticks)

# Include top and right ticks
plt.rcParams['xtick.top'] = True
plt.rcParams['ytick.right'] = True_____no_output_____

# Emission Lines Catalog from Portsmouth Group

The Portsmouth group created a Value Added Catalog (VAC) of stellar kinematics and emission line fluxes for the SDSS DR12 galaxies ([Thomas et al.2013](https://ui.adsabs.harvard.edu/abs/2013MNRAS.431.1383T/abstract)), based on the stellar population models of [Maraston & Stromback (2011)](https://ui.adsabs.harvard.edu/abs/2011MNRAS.418.2785M/abstract). They use an adaptation of the publicly available code - Gas AND Absorption Line Fitting (GANDALF; [Sarzi et al.2006](https://ui.adsabs.harvard.edu/abs/2006MNRAS.366.1151S/abstract)) - to fit stellar population and Gaussian emission line templates to the BOSS spectra. The code determines the kinematics of the gas (velocity and velocity dispersion), emission line fluxes and equivalent widths from the resulting emission line templates.

More information about the Portsmouth stellar kinematics and emission line fluxes can be found [here](https://www.sdss.org/dr16/spectro/galaxy_portsmouth/#kinematics).

We can access this catalog (''emissionlinesport'') from the SDSS DR12 database (`sdss_dr12.emissionlinesport`).
The list and definitions of the columns are given both on the [Data Lab Table Browser](https://datalab.noirlab.edu/query.php?name=sdss_dr12.emissionlinesport)
and on the [SDSS Skyserver webpage](http://skyserver.sdss.org/dr16/en/help/browser/browser.aspx#&&history=description+emissionLinesPort+U).
For plotting the traditional BPT diagram, we need four emission lines - H$\alpha~\lambda$6562, H$\beta~\lambda$4861, [OIII]$\lambda$5006 and [NII]$\lambda$6583. In addition to these columns, we also consider the `bpt` column, which classifies galaxies into "Star Forming", "Seyfert", "LINER", "Seyfert/LINER", "Composite", based on the classification scheme from [Kauffmann et al. 2003](https://ui.adsabs.harvard.edu/abs/2003MNRAS.346.1055K/abstract), [Kewley et al. 2001](https://ui.adsabs.harvard.edu/abs/2001ApJ...556..121K/abstract) and [Schawinski et al. 2007](https://ui.adsabs.harvard.edu/abs/2007MNRAS.382.1415S/abstract). _____no_output_____

# List of emission lines we are interested in -
emission_lines = ('Hb_4861', 'OIII_5006', 'Ha_6562', 'NII_6583')

# Creating a list of columns for all the emission lines for Flux, Flux_Err and Aon for each -
selected_columns_sdss = ', \n'.join(['E.Flux_{0}, E.Flux_{0}_Err, E.Aon_{0}'.format(e) for e in emission_lines])

# Conditions -
# Conditions on flux errors (flux errors > 0 and flux errors != Infinity)
err_cond_sdss = ' AND \n'.join(['(E.Flux_{0}_Err > 0) AND (E.Flux_{0}_Err != \'Infinity\')'.format(e) for e in emission_lines])
# Conditions on Signal to Noise ratio of all the emission lines
sn_cond_sdss = ' AND \n'.join(['(E.Flux_{0}/E.Flux_{0}_Err > 5)'.format(e) for e in emission_lines])_____no_output_____

The `emissionlinesport` catalog contains several duplicates as a result of hosting all the information pertaining to all the spectra of all the sources. This includes some bad spectra as well as repeated spectra of the same object observed on multiple nights (different MJD). To select only the best spectra of unique objects, we JOIN this table with the SDSS DR16 `specobj` table, which contains only primary spectra objects (with `sciencePrimary > 0`)._____no_output_____

# Write a query for the emission lines from the SDSS DR12 emissionlinesport catalog
# Applying the S/N cut of >= 5 for all the four emission lines
# Also applying the conditions that all the flux errors are >0 and != infinity.
# Applying condition on redshift - only selecting galaxies with 0.02 <= z <= 0.2
# We add bpt column - which marks the class of the galaxy based on its position on the BPT Diagram
# We use sdss_joinid to JOIN with SDSS DR16
# This is to get the SDSS DR16 specobjid of objects - which is used to access the spectra

query = """
SELECT
E.ra as ra_sdss, E.dec as dec_sds, E.z, E.bpt,
E.mjd, E.plate, E.fiberid,
%s,
S.specobjid
FROM sdss_dr12.emissionlinesport as E
JOIN sdss_dr16.specobj as S
ON (E.sdss_joinid = S.sdss_joinid) AND (E.mjd = S.mjd)
WHERE (E.z > 0.02) AND (E.z < 0.2) AND
%s AND
%s
""" %(selected_columns_sdss, err_cond_sdss, sn_cond_sdss)_____no_output_____print(query)
SELECT
E.ra as ra_sdss, E.dec as dec_sds, E.z, E.bpt,
E.mjd, E.plate, E.fiberid,
E.Flux_Hb_4861, E.Flux_Hb_4861_Err, E.Aon_Hb_4861,
E.Flux_OIII_5006, E.Flux_OIII_5006_Err, E.Aon_OIII_5006,
E.Flux_Ha_6562, E.Flux_Ha_6562_Err, E.Aon_Ha_6562,
E.Flux_NII_6583, E.Flux_NII_6583_Err, E.Aon_NII_6583,
S.specobjid
FROM sdss_dr12.emissionlinesport as E
JOIN sdss_dr16.specobj as S
ON (E.sdss_joinid = S.sdss_joinid) AND (E.mjd = S.mjd)
WHERE (E.z > 0.02) AND (E.z < 0.2) AND
(E.Flux_Hb_4861_Err > 0) AND (E.Flux_Hb_4861_Err != 'Infinity') AND
(E.Flux_OIII_5006_Err > 0) AND (E.Flux_OIII_5006_Err != 'Infinity') AND
(E.Flux_Ha_6562_Err > 0) AND (E.Flux_Ha_6562_Err != 'Infinity') AND
(E.Flux_NII_6583_Err > 0) AND (E.Flux_NII_6583_Err != 'Infinity') AND
(E.Flux_Hb_4861/E.Flux_Hb_4861_Err > 5) AND
(E.Flux_OIII_5006/E.Flux_OIII_5006_Err > 5) AND
(E.Flux_Ha_6562/E.Flux_Ha_6562_Err > 5) AND
(E.Flux_NII_6583/E.Flux_NII_6583_Err > 5)

%%time
# It takes about 10 seconds
final_table = qc.query(sql=query, fmt = 'table')CPU times: user 1.31 s, sys: 538 ms, total: 1.85 s
Wall time: 13.2 s

# SDSS Spectra

In this section, we will show how to access and plot SDSS spectra from the Data Lab server. In order to access the spectrum of an object, we will need the value of the `specobjid` of the object. It encodes information about the three identifiers - plate, MJD, and fiberID, along with run2d, which holds information about the reduction pipeline used. Each spectrum has a unique plate-MJD-fiberID value (though some objects observed on different nights/plates might have more than one spectrum). Information about identifying SDSS spectra can be found [here](https://www.sdss.org/dr16/spectro/spectro_basics/#IdentifyingSDSSspectra).

Information about the SDSS spectroscopic data reduction and spectra formats is given [here](https://www.sdss.org/dr16/spectro/pipeline/#SpectroscopicDataReduction)._____no_output_____
## Functions to access and plot SDSS spectra_____no_output_____

# Function 1 --

def get_sdss_spectra(specobjid,  z, rest_frame = True):
    """
    Function to access the SDSS spectra of any object from the Data Lab file service.

    Returns the arrays of wavelength, flux, model and ivar.
    Returns the rest-frame arrays if rest_frame = True.
    If rest_frame = False, returns the observed-frame arrays.

    Parameters
    ----------
    specobjid : int64
        Unique SDSS spectrum id

    z : float
        Redshift of the source

    rest_frame : bool
        Whether or not to return the rest-frame values. Default is True.

    Returns
    -------
    wavelength : array
        Wavelength array (Rest-frame values if rest_frame = True)

    flux : array
        Flux values array (Rest-frame values if rest_frame = True)

    model : array
        Model values array (Rest-frame values if rest_frame = True)

    ivar : array
        Inverse variance array (Rest-frame values if rest_frame = True)

    """

    # getSpec from specClient can be used to obtain the spectra of any given object
    # specobjid is required
    t = spec.getSpec(specobjid, fmt = 'numpy')

    # Extracting the different values -
    loglam = t['loglam']       # Log wavelength
    flux = t['flux']           # Flux
    ivar = t['ivar']           # Inverse Variance
    model = t['model']         # Model
    wavelength = 10**(loglam)  # Wavelength values

    if (rest_frame == True):
        # If rest_frame = True, we convert the different arrays into their rest-frame values
        wavelength = wavelength/(1+z)
        flux = flux*(1+z)
        model = model*(1+z)
        ivar = ivar/((1+z)**2)

    return (wavelength, flux, model, ivar)_____no_output_____# Function 2 -

def add_lines(z, ax = None, rest_frame = True, em_label = True, abs_label = True, em_lines = None, abs_lines = None):
    """
    Function to add emission and/or absorption lines onto a plot.

    If em_lines or abs_lines is given, plotting only the specified lines.
    If no lines are given, plotting all the lines that are present in x-range of the plot.

    Parameters
    ----------
    z : float
        Redshift value of the source

    ax : AxesSubplot
        The axis onto which the emission/absoption lines needs to be plotted.
        If ax = None, then the plotting function uses plt, rather than axis.

    rest_frame : bool
        Whether or not the plot is in rest-frame. Default is True.

    em_label : bool
        Whether or not to label the emission lines. Default is True.

    abs_label : bool
        Whether or not to label the absorption lines. Default is True.

    em_lines : list
        List of emission lines to label

    abs_lines : list
        List of absorption lines to label

    Returns
    -------
    None

    """

    # List of lines
    # This is the set of emission lines from the spZline files.
    # All the wavelengths are in vaccuum wavelengths.

    # Emission Lines
    emission_lines = [
    {"name" : "Ly-alpha",       "lambda" : 1215.67,  "emission": True,  "label" : "Ly$\\alpha$"},
    {"name" : "N V 1240",       "lambda" : 1240.81,  "emission": True,  "label" : "N V"},
    {"name" : "C IV 1549",      "lambda" : 1549.48,  "emission": True,  "label" : "C IV" },
    {"name" : "He II 1640",     "lambda" : 1640.42,  "emission": True,  "label" : "He II"},
    {"name" : "C III] 1908",    "lambda" : 1908.734, "emission": True,  "label" : "C III]"},
    {"name" : "Mg II 2799",     "lambda" : 2800.315, "emission": True,  "label" : "Mg II" },
    {"name" : "[O II] 3725",    "lambda" : 3727.092, "emission": True,  "label" : " "},
    {"name" : "[O II] 3727",    "lambda" : 3729.875, "emission": True,  "label" : "[O II]"},
    {"name" : "[Ne III] 3868",  "lambda" : 3869.857, "emission": True,  "label" : "[Ne III]"},
    {"name" : "H-zeta",         "lambda" : 3890.151, "emission": True,  "label" : "H$\\zeta$"},
    {"name" : "[Ne III] 3970",  "lambda" : 3971.123, "emission": True,  "label" : "[Ne III]"},
    {"name" : "H-epsilon",      "lambda" : 3971.195, "emission": True,  "label" : "H$\\epsilon$"},
    {"name" : "H-delta",        "lambda" : 4102.892, "emission": True,  "label" : "H$\\delta$"},
    {"name" : "H-gamma",        "lambda" : 4341.684, "emission": True,  "label" : "H$\\gamma$"},
    {"name" : "[O III] 4363",   "lambda" : 4364.435, "emission": True,  "label" : "[O III]"},
    {"name" : "He II 4685",     "lambda" : 4686.991, "emission": True,  "label" : "He II"},
    {"name" : "H-beta",         "lambda" : 4862.683, "emission": True,  "label" : "H$\\beta$"},
    {"name" : "[O III] 4959",   "lambda" : 4960.294, "emission": True,  "label" : "[O III]" },
    {"name" : "[O III] 5007",   "lambda" : 5008.239, "emission": True,  "label" : "[O III]" },
    {"name" : "He II 5411",     "lambda" : 5413.025, "emission": True,  "label" : "He II"},
    {"name" : "[O I] 5577",     "lambda" : 5578.888, "emission": True,  "label" : "[O I]" },
    {"name" : "[N II] 5755",    "lambda" : 5756.186, "emission": True,  "label" : "[Ne II]" },
    {"name" : "He I 5876",      "lambda" : 5877.308, "emission": True,  "label" : "He I" },
    {"name" : "[O I] 6300",     "lambda" : 6302.046, "emission": True,  "label" : "[O I]" },
    {"name" : "[S III] 6312",   "lambda" : 6313.806, "emission": True,  "label" : "[S III]" },
    {"name" : "[O I] 6363",     "lambda" : 6365.535, "emission": True,  "label" : "[O I]" },
    {"name" : "[N II] 6548",    "lambda" : 6549.859, "emission": True,  "label" : "[N II]" },
    {"name" : "H-alpha",        "lambda" : 6564.614, "emission": True,  "label" : "H$\\alpha$" },
    {"name" : "[N II] 6583",    "lambda" : 6585.268, "emission": True,  "label" : "[N II]" },
    {"name" : "[S II] 6716",    "lambda" : 6718.294, "emission": True,  "label" : "[S II]" },
    {"name" : "[S II] 6730",    "lambda" : 6732.678, "emission": True,  "label" : "[S II]" },
    {"name" : "[Ar III] 7135",  "lambda" : 7137.758, "emission": True,  "label" : "[Ar III]" },]

    # Absorption lines
    absorption_lines = [
    {"name" : "H12",            "lambda" : 3751.22,  "emission": False, "label" : "H12"},
    {"name" : "H11",            "lambda" : 3771.70,  "emission": False, "label" : "H11"},
    {"name" : "H10",            "lambda" : 3798.98,  "emission": False, "label" : "H10"},
    {"name" : "H9",             "lambda" : 3836.48,  "emission": False, "label" : "H9"},
    {"name" : "H-zeta",         "lambda" : 3890.151, "emission": False, "label" : "H$\\zeta$" },
    {"name" : "K (Ca II 3933)", "lambda" : 3934.814, "emission": False, "label" : "K (Ca II)"},
    {"name" : "H (Ca II 3968)", "lambda" : 3969.623, "emission": False, "label" : "H (Ca II)"},
    {"name" : "H-epsilon",      "lambda" : 3971.195, "emission": False, "label" : "H$\\epsilon$"},
    {"name" : "H-delta",        "lambda" : 4102.892, "emission": False, "label" : "H$\\delta$" },
    {"name" : "G (Ca I 4307)",  "lambda" : 4308.952, "emission": False, "label" : "G (Ca I)"},
    {"name" : "H-gamma",        "lambda" : 4341.684, "emission": False, "label" : "H$\\gamma$"},
    {"name" : "H-beta",         "lambda" : 4862.683, "emission": False, "label" : "H$\\beta$"},
#    {"name" : "Mg I 5175",      "lambda" : 5176.441, "emission": False, "label" : "Mg I"},  #Triplet
    {"name" : "Mg I 5183",      "lambda" : 5185.048, "emission": False, "label" : " "},
    {"name" : "Mg I 5172",      "lambda" : 5174.125, "emission": False, "label" : " "},
    {"name" : "Mg I 5167",      "lambda" : 5168.762, "emission": False, "label" : "Mg I"},
    {"name" : "D2 (Na I 5889)", "lambda" : 5891.582, "emission": False, "label" : " " },
    {"name" : "D1 (Na I 5895)", "lambda" : 5897.554, "emission": False, "label" : "D1,2 (Na I)" },
    {"name" : "H-alpha",        "lambda" : 6564.614, "emission": False, "label" : "H$\\alpha$"},
    ]

    if (ax == None):
        # If there is no axes given, plotting with the plt function
        ax = plt.gca()

    if (em_lines != None):
        # Choosing the emission lines listed by the user
        emission_lines = list(filter(lambda x: x['name'] in em_lines, emission_lines))

    if (abs_lines != None):
        # Choosing the absorption lines listed by the user
        absorption_lines = list(filter(lambda x: x['name'] in abs_lines, absorption_lines))

    xbounds = ax.get_xbound()   # Getting the x-range of the plot
    # This is for selecting only those lines that are visible in the x-range of the plot

    for ii in range(len(emission_lines)):
        # If rest_frame = False, redshifting the emission lines to the observed frame of the source
        if (rest_frame == False):
            lam = emission_lines[ii]['lambda']*(1+z)
        else:
            lam = emission_lines[ii]['lambda']
        # Plotting the emission lines if they are within the x-range of the plot
        if (emission_lines[ii]['emission']) & (lam > xbounds[0]) & (lam < xbounds[1]):
            ax.axvline(lam, 0.95, 1.0, color = 'b', lw = 1.0)
            ax.axvline(lam, color = 'b', lw = 1.0, linestyle = ':')
            trans = ax.get_xaxis_transform()
            if (em_label == True):
                # Labeling the emission lines if em_label = True
                ax.annotate(emission_lines[ii]['label'], xy = (lam, 1.05), xycoords = trans, \
                         fontsize = 12, rotation = 90, color = 'b')

    for ii in range(len(absorption_lines)):
        # If rest_frame = False, redshifting the absorption lines to the observed frame of the source
        if (rest_frame == False):
            lam = absorption_lines[ii]['lambda']*(1+z)
        else:
            lam = absorption_lines[ii]['lambda']
        # Plotting the absorption lines if they are within the x-range of the plot
        if (lam > xbounds[0]) & (lam < xbounds[1]):
            ax.axvline(lam, 0.2, 1.0, color = 'r', lw = 1.0, linestyle = ':')
            trans = ax.get_xaxis_transform()
            if (abs_label == True):
                # Labeling the absorption lines if abs_label = True
                ax.annotate(absorption_lines[ii]['label'], xy = (lam, 0.05), xycoords = trans, \
                         fontsize = 12, rotation = 90, color = 'r')_____no_output_____# Function - 3
# This function utilises both the above functions

def plot_sdss_spectra(specobjid, z, rest_frame = True, \
                      xlim = None, ylim = None, \
                      spectra_kwargs = {'color': 'grey', 'alpha': 0.5}, model_kwargs = {'color': 'k', 'linewidth': 2.0},\
                      emission_lines = False, absorption_lines = False, \
                      em_lines = None, abs_lines = None):

    """
    Function to plot the SDSS spectra.

    Parameters
    ----------
    specobjid : int64
        Unique SDSS spectrum id

    z : float
        Redshift of the source

    rest_frame : bool
        Whether or not to plot the spectra in the rest-frame. Default is True.

    xlim : list or tuple
        Setting the xrange of the plot

    ylim : list or tupe
        Setting the yrange of the plot

    emission_lines - bool
        Whether or not to overplot emission lines. Default is False.

    absorption_lines - bool
        Whether or not to overplot absorpion lines. Default is False.

    em_lines - list
        List of emission lines to plot
        If not mentioned, all the lines in the default list will be plotted.

    abs_lines - list
        List of absorption lines to plot
        If not mentioned, all the lines in the default list will be plotted.

    spectra_kwargs - dict
        Plotting keyword arguments for the spectra

    model_kwargs - dict
        Plotting keyword arguments for the model

    Returns
    -------
    None

    """

    # Getting the wavelength, flux, model and ivar arrays for the given source
    wavelength, flux, model, ivar = get_sdss_spectra(specobjid, z, rest_frame)

    # Setting up the plot
    plt.figure(figsize = (14,6))
    plt.plot(wavelength, flux*(ivar > 0), **spectra_kwargs)
    plt.plot(wavelength, model*(ivar > 0), **model_kwargs)
    plt.xlim(xlim)
    plt.ylim(ylim)
    plt.xlabel('Wavelength ($\AA$)')
    plt.ylabel('$F_{\lambda}$ ($10^{-17}~ergs~s^{-1}~cm^{-2}~\AA^{-1}$)')

    # Plotting Absorption/Emission lines - only works if either of the lines is set to True
    if (emission_lines == True)|(absorption_lines == True):
        if (emission_lines == False):
            # Sending empty array of em_lines if emission_lines = False
            em_lines = []
        if (absorption_lines == False):
            # Sending empty array of abs_lines
            abs_lines = []

        # Plotting function to add emission/absorption lines
        add_lines(ax = None, z = z, rest_frame = rest_frame, \
                           em_label = True, abs_label = True, em_lines = em_lines, abs_lines = abs_lines)_____no_output_____

## Plotting the spectra

Function to plot the spectra for any object -

        plot_sdss_spectra(specobjid, z, [release], [rest_frame], \
                      [xlim], [ylim], [spectra_kwargs], [model_kwargs],\
                      [emission_lines], [absorption_lines], [em_lines], [abs_lines])
where

        specobjid               SDSS DR16 specobjid of the object
        z                       Redshift of the object
        rest_frame              [optional, default = True] Whether or not to plot the spectra
                                in the rest-frame.
        xlim                    [optional, default = None] wavelength range to be plotted for the spectra
        ylim                    [optional, default = None] yrange to be plotted for the spectra
        spectra_kwargs          [optional, default = {'color': 'grey', 'alpha': 0.5}]
                                 Plotting kwargs for the spectra
        model_kwargs            [optional, default = {'color': 'k', 'linewidth': 2.0}]
                                 Plotting kwargs for the model
        emission_lines          [optional, default = False] Whether or not to overplot the emission lines
        absorption_lines        [optional, default = False] Whether or not to overplot the absorption lines
        em_lines                [optional, default = None] List of emission lines to be overplotted
                                If emission_lines = True, and em_lines = None - all the lines present within
                                the xrange of the plot are overplotted.
        abs_lines               [optional, default = None] List of absorption lines to be overplotted
                                If aborption_lines = True, and abs_lines = None - all the lines present within
                                the xrange of the plot are overplotted._____no_output_____

# Assigning variables for the columns required to plot the spectra -

z = final_table['z']
specobjid = final_table['specobjid']_____no_output_____# Print number of objects in table

N_table = len(final_table)
print(N_table)156748
# We will first plot with all the default values -
# Requires specobjid and z

index = 10     # Change the value between 0 and N_table-1 to plot different spectra in rest-frame

plot_sdss_spectra(specobjid[index], z[index])_____no_output_____# Changing the default values for the plotting function

index = 10           # Change the value between 0 and N_table-1 to plot different spectra in rest-frame

plot_sdss_spectra(specobjid[index], z[index], rest_frame = True, \
                      xlim = [4500, 7000], ylim = None, \
                      spectra_kwargs = {'color': 'grey', 'alpha': 0.5}, \
                      model_kwargs = {'color': 'k', 'linewidth': 2.0},\
                      emission_lines = True, absorption_lines = True, \
                      em_lines = None,  abs_lines = [])

# The actual spectra from the SDSS is shown in grey and the model is shown in black.
# You can change it to any color that you want by using the spectra_kwargs and model_kwargs keywords._____no_output_____

# Ionization Sources and the BPT Diagram_____no_output_____Emission lines in the spectra of galaxies can be a result of several sources of ionization, such as star formation, AGN, and shocks. The BPT diagram is constructed from the optical line ratios [OIII]/H$\beta$ and [NII]/H$\alpha$. The position of any galaxy in this space depends on the metallicity of the ionized gas, the hardness of the ionizing radiation and the ionization parameter. We consider four classes of excitation sources here, which are located at different positions of the BPT diagram based on their domination source of ionization -

* Star-Forming Galaxies (SF)
* Seyferts (Moderate-to-High luminosity AGN)
* LINERs (Low-Ionization Nuclear Emission Regions; shocks and/or low-luminosity AGN)
* Composite Galaxies (Mixture of SF, shocks and/or AGN activity)

For more information, see a recent review on understanding galaxy evolution from their emission lines by [Kewley et al. 2019](https://ui.adsabs.harvard.edu/abs/2019ARA%26A..57..511K/abstract)._____no_output_____

# Computing the logarithm of line ratios for the BPT Diagram -

Nii_Ha = np.log10(final_table['flux_nii_6583']/final_table['flux_ha_6562'])
Oiii_Hb = np.log10(final_table['flux_oiii_5006']/final_table['flux_hb_4861'])_____no_output_____# Assigning a variable for the bpt column
bpt = final_table['bpt']   # BPT Classification

# Creating labels for the different classes based on BPT Column -

is_sf = (bpt == 'Star Forming')
is_sy = (bpt == 'Seyfert')
is_comp = (bpt == 'Composite')
is_liner = (bpt == 'LINER')_____no_output_____# 2-d histogram density of all the galaxies

dens_all,_,_ = np.histogram2d(Nii_Ha, Oiii_Hb, range = [[-2.0, 0.75], [-1.2, 1.2]], bins = 100, normed = True)
dens_all = dens_all.T_____no_output_____# BPT Diagram color-coded based on the class

plt.figure(figsize = (8,8))

plt.scatter(Nii_Ha[is_sy], Oiii_Hb[is_sy], color = 'r', s = 1.0)             # Seyferts in red (r)
plt.scatter(Nii_Ha[is_sf], Oiii_Hb[is_sf], color = 'b', s = 1.0)             # SF in blue (b)
plt.scatter(Nii_Ha[is_comp], Oiii_Hb[is_comp], color = 'g', s = 1.0)         # Composites in green (g)
plt.scatter(Nii_Ha[is_liner], Oiii_Hb[is_liner], color = 'purple', s = 1.0)  # LINERs in purple
plt.contour(dens_all, colors = 'white', extent = (-2.0, 0.75, -1.2, 1.2), levels = [0.0, 0.1, 0.2, 0.3, 0.5, 1.5, 3.0, 6.0])
plt.xlim([-2, 0.75])
plt.ylim([-1.2, 1.2])
plt.text(-1.0, -0.5, 'SF', color = 'blue', fontsize = 18)
plt.text(0.0, -1.0, 'Composite', color = 'green', fontsize = 18)
plt.text(0.5, 0.2, 'LINER', color = 'purple', fontsize = 18, rotation = 90)
plt.text(-0.9, 1.0, 'Seyfert', color = 'r', fontsize = 18)
plt.xlabel('log ([NII]/H$\\alpha$)')
plt.ylabel('log ([OIII]/H$\\beta$)')

plt.show()_____no_output_____

From this diagram, we can see that the Seyferts and LINERs have higher [NII]/H$\alpha$ and [OIII]/H$\beta$ compared to SF galaxies. In a nebula photoionized by young, massive stars, the low-ionization transition [NII] is weak. A hard AGN radiation field boosts the production of collisionally excited forbidden line emission. LINERs have lower luminosities than Seyferts, requiring harder ionizing radiation field than SF galaxies, but a lower ionization parameter than Seyferts. They can be produced by low-luminosity AGN, cooling flows, starburst driven winds, or shock-heated gas. AGN tend to reside in most massive and most metal-rich galaxies. Thus, an AGN contribution raises the [NII]/H$\alpha$ ratio above the SF galaxies. Composites may contain a mixture of SF, shock excitation and/or AGN activity._____no_output_____
# Stacking Spectra_____no_output_____Spectra stacking in an important technique to study the general properties of galaxies. This process increases the signal-to-noise ratio and thus highlights the continuum and the emission lines more clearly, and may further reveal weak lines that are rarely detected in individual spectra. In order to look at the typical properties of galaxies from the four classes - SF, AGN, LINER and Composites, we will select random galaxies from each of these classes and stack their spectra.

In this section, we show how to align and stack spectra of a number of galaxies._____no_output_____
## Function to align and stack spectra

This function takes in a table of SDSS objects and outputs the stacked spectrum array values. Here, it will call the first function `get_sdss_spectra`._____no_output_____

## Function 4 -

def stack_spectra(table):
    """
    Function to Stack the spectra of all the sources in the given table.

    Returns an inverse variance weighted mean of the input spectra.

    Parameters
    ----------
    table : table
        Table of sources whose spectra need to be stacked

    Returns
    -------
    wavelength_stack : array
        Wavelength array of the stacked spectra

    flux_stack : array
        Flux array of the stacked spectra

    model_stack : array
        Model array of the stacked spectra

    """

    # Create an array with the targeted loglam values - reference array
    loglam_ref = np.arange(3.5000, 3.9000, 0.0001).astype('float32')

    # Create empty lists for flux, ivar and model -
    flux_array = []
    model_array = []
    ivar_array = []

    # Extract the necessary columns from the table
    redshift = table['z']
    specobjid = table['specobjid']

    # Number of spectra -
    n = len(table)

    ## All the spectra have the same spacing in log wavelength.
    ## We shift the spectra in the observed wavelength range to fill the same log wavelength range
    ## Join the different arrays of the individual spectra into a single array

    for ii in range(n):
        # Get the rest-frame spectra of each object
        lam, flux, model, ivar = get_sdss_spectra(specobjid[ii], redshift[ii], rest_frame = True)
        loglam = np.around(np.log10(lam), 4).astype('float32')
        # The log of reference wavelength and log of wavelength range of each spectra are converted to 'float32'
        # This helps in avoiding the rounding errors

        # Creating empty arrays for flux, inverse variance and model
        # with the same length as the reference loglam array
        fl = np.zeros(len(loglam_ref))
        iv = np.zeros(len(loglam_ref))
        ml = np.zeros(len(loglam_ref))

        # Truth_array - A boolen array which will define what pixels the spectra have been shifted to
        truth_array = np.zeros(len(loglam_ref), dtype = bool)

        # Find common elements between the reference and spectra wavelengths -
        ll_common = np.intersect1d(loglam_ref, loglam)

        # Finding the indices of the first pixel in the common array in both reference and spectrum arrays
        ref_start = np.where(loglam_ref == ll_common[0])[0][0]
        spec_start = np.where(loglam == ll_common[0])[0][0]

        m = min(len(loglam_ref[ref_start:]), len(loglam[spec_start:]))

        # Match the first pixel of the common loglam values with the reference array
        # Mark the number of pixels with the spectra starting from the first pixel to be True
        truth_array[ref_start:ref_start+m] = True

        # Fill the flux, inverse variance and model arrays
        fl[truth_array] = flux[spec_start:spec_start+m]
        iv[truth_array] = ivar[spec_start:spec_start+m]
        ml[truth_array] = model[spec_start:spec_start+m]

        # Where the reference array is empty, fill the flux and model with nan values
        # Set inverse variance to zero for these pixels
        fl[~truth_array] = np.nan
        iv[~truth_array] = 0.0
        ml[~truth_array] = np.nan

        # Create a masked array with 'nan' values marked
        # This avoids the pixels filled with 'nan' when stacking
        fl = np.ma.masked_array(fl, np.isnan(fl))
        iv = np.ma.masked_array(iv, np.isnan(iv))
        ml = np.ma.masked_array(ml, np.isnan(ml))

        flux_array.append(fl)
        model_array.append(ml)
        ivar_array.append(iv)

    # Stack the spectra by weighted average
    wavelength_stack = 10**(loglam_ref)
    flux_stack = np.ma.average(flux_array, axis = 0, weights = ivar_array)
    model_stack = np.ma.average(model_array, axis = 0, weights = ivar_array)

    flux_stack = flux_stack.filled(np.nan)
    model_stack = model_stack.filled(np.nan)

    return (wavelength_stack, flux_stack, model_stack)_____no_output_____

## Selecting Galaxies from the BPT

In order to compare the general spectral properties of galaxies from the four different classes, we randomly select 100 galaxies from each BPT class to stack their spectra. To be consistent, we select the galaxies from squares of same area from the four regions._____no_output_____

s1= Rectangle(xy = (-0.15, 0.55), width = 0.1, height = 0.1, color = 'white', lw = 3.0, fill = False)
s2 = Rectangle(xy = (-0.65, -0.05), width = 0.1, height = 0.1, color = 'white', lw = 3.0, fill = False)
s3 = Rectangle(xy = (0.0, 0.05), width = 0.1, height = 0.1, color = 'white', lw = 3.0, fill = False)
s4 = Rectangle(xy = (-0.25, 0.0), width = 0.1, height = 0.1, color = 'white', lw = 3.0, fill = False)

plt.figure(figsize = (8,8))

plt.scatter(Nii_Ha[is_sy], Oiii_Hb[is_sy], color = 'r', s = 1.0)
plt.scatter(Nii_Ha[is_sf], Oiii_Hb[is_sf], color = 'b', s = 1.0)
plt.scatter(Nii_Ha[is_comp], Oiii_Hb[is_comp], color = 'g', s = 1.0)
plt.scatter(Nii_Ha[is_liner], Oiii_Hb[is_liner], color = 'purple', s = 1.0)
plt.gca().add_patch(s1)
plt.gca().add_patch(s2)
plt.gca().add_patch(s3)
plt.gca().add_patch(s4)
plt.text(-1.0, -0.5, 'SF', color = 'blue', fontsize = 18)
plt.text(0.0, -1.0, 'Composite', color = 'green', fontsize = 18)
plt.text(0.5, 0.2, 'LINER', color = 'purple', fontsize = 18, rotation = 90)
plt.text(-0.9, 1.0, 'Seyfert', color = 'r', fontsize = 18)
plt.xlim([-2, 0.75])
plt.ylim([-1.2, 1.2])
plt.xlabel('log ([NII]/H$\\alpha$)')
plt.ylabel('log ([OIII]/H$\\beta$)')

plt.show()_____no_output_____

In this figure, the four white boxes show the regions from where we will perform the random selection of galaxies._____no_output_____

# Regions of the white boxes -
# Getting the indices of the sources within the four boxes

sy_box = np.where((Nii_Ha >= -0.15) & (Nii_Ha <= -0.05) & (Oiii_Hb >= 0.55) & (Oiii_Hb <= 0.65))[0]
sf_box = np.where((Nii_Ha >= -0.65) & (Nii_Ha <= -0.55) & (Oiii_Hb >= -0.05) & (Oiii_Hb <= 0.05))[0]
liner_box = np.where((Nii_Ha >= 0.0) & (Nii_Ha <= 0.1) & (Oiii_Hb >= 0.05) & (Oiii_Hb <= 0.15))[0]
comp_box = np.where((Nii_Ha >= -0.25) & (Nii_Ha <= -0.15) & (Oiii_Hb >= 0.0) & (Oiii_Hb <= 0.1))[0]_____no_output_____len(final_table[sy_box]), len(final_table[sf_box]), len(final_table[comp_box]), len(final_table[liner_box])_____no_output_____# Selecting 100 sources from each box and joining their rows into a table
n = 100

# Indices of 100 objects within each box
sy_sel = np.sort(random.sample(list(sy_box), n))
sf_sel = np.sort(random.sample(list(sf_box), n))
liner_sel = np.sort(random.sample(list(liner_box), n))
comp_sel = np.sort(random.sample(list(comp_box), n))

# Tables of the selections
tab_sy = final_table[sy_sel]
tab_sf = final_table[sf_sel]
tab_liner = final_table[liner_sel]
tab_comp = final_table[comp_sel]_____no_output_____# Stacked spectra for the 100 sources within each white box
lam_sy, flux_sy, model_sy = stack_spectra(tab_sy)
lam_sf, flux_sf, model_sf = stack_spectra(tab_sf)
lam_liner, flux_liner, model_liner = stack_spectra(tab_liner)
lam_comp, flux_comp, model_comp = stack_spectra(tab_comp)_____no_output_____

## Plotting Stacked Spectra_____no_output_____

fig, axs = plt.subplots(4,1, figsize = (12, 12), sharex = True, sharey = True)
plt.subplots_adjust(wspace = 0., hspace = 0.)

axs[0].plot(lam_sy, flux_sy, color = 'r', label = 'Seyferts')
axs[0].legend(fontsize = 14, loc = 'upper left')
axs[0].set_ylabel('$F_{\lambda}$')

axs[1].plot(lam_sf, flux_sf, color = 'b', label = 'Star-Forming')
axs[1].legend(fontsize = 14, loc = 'upper left')
axs[1].set_ylabel('$F_{\lambda}$')

axs[2].plot(lam_liner, flux_liner, color = 'purple', label = 'LINERs')
axs[2].legend(fontsize = 14, loc = 'upper left')
axs[2].set_ylabel('$F_{\lambda}$')

axs[3].plot(lam_comp, flux_comp, color = 'g', label = 'Composite')
axs[3].legend(fontsize = 14, loc = 'upper left')
axs[3].set_xlim([3510, 7010])
axs[3].set_xlabel('$\lambda$')
axs[3].set_ylabel('$F_{\lambda}$')_____no_output_____

Here are some galaxy properties that we can infer from the above stacked spectra -
* The 4000$\mathring A$ break is less significant in SF galaxies compared to the other three classes. This is expected because the 4000$\mathring A$  is enhanced in old stellar populations, and the SF galaxies definitely have younger stars.
* The relative importance of Ca II H($\lambda$3969) and K($\lambda$3934) lines help in identifying the relative importance of young and old populations. This can again be seen in the SF galaxies, when H is slightly stronger than K line. In Seyferts, LINERs and composites, the old population dominates as K line is larger than H line.
* The ratio between the [NII]$\lambda$6583 and H$\alpha$ provides an estimate of gas metallicity in galaxies. On average, we see that Seyferts and LINERs have higher gas metallicity than SF and composite galaxies. In general, [NII]/H$\alpha$ varies as a function of both metallicity as well as the level of ionization.
* [OIII]$\lambda$4363 can be observed only in the stacked spectrum of SF galaxies. This can be an indicator of low metallicity.
* The NaI D absorption line ($\lambda$5891) depends on the amount of the interstellar reddening in the galaxies. We can see that this is signicant in LINERs compared to the other three classes, which signifies that LINERs have substantial dust extinction. Dust extinction can also be seen Seyferts and composites, but it is weaker than LINERs.
* The LINERs have the reddest stellar continuum, consistent with them residing in more massive (quiescent) galaxies.

From these different observations, we can conclude that SF galaxies have a young and lower metallicity stellar population, compared to Seyfers, LINERs and Composites. Let us now look at the emission lines used to construct the BPT diagram. These lines provide information about the ionization mechanisms in the galaxies._____no_output_____## Zooming on BPT lines_____no_output_____

fig, axs = plt.subplots(4,2, figsize = (16,12), sharey = True, sharex = 'col')
plt.subplots_adjust(wspace = 0.05, hspace = 0.0)

axs[0][0].plot(lam_sy, flux_sy, color = 'r', label = 'Seyferts')
axs[0][0].legend(fontsize = 14, loc = 'upper left')
axs[0][0].set_ylabel('$F_{\lambda}$')
axs[0][0].set_xlim([4810, 5090])
add_lines(ax = axs[0][0], z=0, abs_lines = [])

axs[1][0].plot(lam_sf, flux_sf, color = 'b', label = 'Star-Forming')
axs[1][0].legend(fontsize = 14, loc = 'upper left')
axs[1][0].set_ylabel('$F_{\lambda}$')
add_lines(ax = axs[1][0], z=0, rest_frame = True, abs_lines = [], em_label = False)

axs[2][0].plot(lam_liner, flux_liner, color = 'purple', label = 'LINERs')
axs[2][0].legend(fontsize = 14, loc = 'upper left')
axs[2][0].set_ylabel('$F_{\lambda}$')
axs[2][0].set_xlabel('$\lambda$')
add_lines(ax = axs[2][0], z=0, rest_frame = True, abs_lines = [], em_label = False)

axs[3][0].plot(lam_comp, flux_comp, color = 'g', label = 'Composite')
axs[3][0].legend(fontsize = 14, loc = 'upper left')
axs[3][0].set_ylabel('$F_{\lambda}$')
axs[3][0].set_xlabel('$\lambda$')
add_lines(ax = axs[3][0], z=0, rest_frame = True, abs_lines = [], em_label = False)

axs[0][1].plot(lam_sy, flux_sy, color = 'r', label = 'Seyferts')
axs[0][1].set_xlim([6250, 6900])
add_lines(ax = axs[0][1], z=0, rest_frame = True, abs_lines = [])

axs[1][1].plot(lam_sf, flux_sf, color = 'b', label = 'Star-Forming')
add_lines(ax = axs[1][1], z=0, rest_frame = True, abs_lines = [], em_label = False)

axs[2][1].plot(lam_liner, flux_liner, color = 'purple', label = 'LINERs')
add_lines(ax = axs[2][1], z=0, rest_frame = True, abs_lines = [], em_label = False)

axs[3][1].plot(lam_comp, flux_comp, color = 'g', label = 'Composite')
axs[3][1].set_xlabel('$\lambda$')
add_lines(ax = axs[3][1], z=0, rest_frame = True, abs_lines = [], em_label = False)_____no_output_____

From the above figure, we can infer the following -
   * The [OIII]$\lambda$5007 is the strongest in Seyferts - the luminosity of this line is a tracer of AGN activity.
   * The H$\alpha$ line is a tracer for star formation. We can clearly see that it is the strongest for SF galaxies.
   * The relative fluxes of [NII]$\lambda$6853 and H$\alpha$ lines are an indicator of ionizing conditions of the nebula. When the nebula is ionized by a star burst, [NII] << H$\alpha$, while a harder ionizing radiation like an AGN increases the [NII] flux such that [NII] > H$\alpha$. This can be seen in Seyferts and LINERs. The ratio of [NII]$\lambda$6583 and H$\alpha$ is also a tracer of the gas phase metallicity in the galaxy. It is hard to separate the effects of the two from the information we have available at the moment.
   * Higher ionization radiation field and a higher ionization parameters lead to a higher [OIII]$\lambda$5007/H$\beta$.
   * LINERs have weak emission lines, compared to other classes. In addition to the possibility of being produced by low-ionization AGN, LINER-like emission can be produced in cooling flows, winds and shocks.

Stacking the spectra is thus a very valuable tool for understanding the properties of galaxies._____no_output_____####
# Resources

- AGN Diagnostic Diagram website including a section on [optical BPT emission-line diagnostic](https://sites.google.com/site/agndiagnostics/home/bpt)

- Information about the galaxy properties from the Portsmouth Group: https://www.sdss.org/dr16/spectro/galaxy_portsmouth/

- Portsmouth stellar kinematics and emission line fluxes https://www.sdss.org/dr16/spectro/galaxy_portsmouth/#kinematics

- SDSS Spectra Basics: https://www.sdss.org/dr16/spectro/spectro_basics/#IdentifyingSDSSspectra

- SDSS Spectroscopic Data Reduction and access formats: https://www.sdss.org/dr16/spectro/pipeline/#SpectroscopicDataReduction

- Schema details about the emissionlinesport catalog can be found both on the [Data Lab Table Browser](https://datalab.noirlab.edu/query.php?name=sdss_dr12.emissionlinesport)
and on the [SDSS Skyserver webpage](http://skyserver.sdss.org/dr16/en/help/browser/browser.aspx#&&history=description+emissionLinesPort+U)

# References

[Baldwin et. al., 1981](https://ui.adsabs.harvard.edu/abs/1981PASP...93....5B/abstract)

[Kauffmann et al. 2003](https://ui.adsabs.harvard.edu/abs/2003MNRAS.346.1055K/abstract)

[Kewley et al. 2001](https://ui.adsabs.harvard.edu/abs/2001ApJ...556..121K/abstract)

[Kewley et al. 2019](https://ui.adsabs.harvard.edu/abs/2019ARA%26A..57..511K/abstract)

[Maraston & Stromback (2011)](https://ui.adsabs.harvard.edu/abs/2011MNRAS.418.2785M/abstract)

[Sarzi et al.2006](https://ui.adsabs.harvard.edu/abs/2006MNRAS.366.1151S/abstract)

[Schawinski et al. 2007](https://ui.adsabs.harvard.edu/abs/2007MNRAS.382.1415S/abstract)

[Thomas et al.2013](https://ui.adsabs.harvard.edu/abs/2013MNRAS.431.1383T/abstract)_____no_output_____---_____no_output_____