# Notebook from cstubens/notebooks-latest
Path: 04_HowTos/FileService/How_to_use_SDSS_Data.ipynb

__author__ = 'Benjamin Alan Weaver <baweaver@lbl.gov>'
__version__ = '20190109'
__datasets__ = ['sdss_dr13', 'sdss_dr14']
__keywords__ = ['extragalactic', 'spectroscopic redshift']_____no_output_____

# How to use SDSS Data in NOAO Data Lab

*Benjamin Alan Weaver & NOAO Data Lab Team*

### Table of Contents

* [Goals & Notebook Summary](#Goals)
* [Disclaimer & Attribution](#Disclaimer-&-Attribution)
* [Imports & Setup](#Imports-&-Setup)
* [BOSS Galaxies (photometry)](#BOSS-Galaxies-in-the-SDSS-Photometric-Catalog)
* [BOSS Galaxies (spectroscopy)](#BOSS-Galaxies-in-the-SDSS-Spectroscopic-Catalog)
* [Plotting SDSS Spectra](#Plotting-SDSS-Spectra)
* [Display SDSS Images](#Display-SDSS-Images)
* [Resources & References](#Resources-&-References)

## Goals

Work with SDSS tables in the Data Lab database.

## Notebook Summary

We're bringing the SDSS-III/IV (DR8-DR14) data archive to NOAO, and some of the data are already available in Data Lab.  This notebook demonstrates some queries and plots of SDSS data.

The SDSS website is [www.sdss.org](https://www.sdss.org).

The examples below are primarily focused on target selection and spectroscopy for the [BOSS survey](https://www.sdss.org/surveys/boss/).  This notebook makes extensive use of the target selection algorithms described by [Reid et al. (2016)](http://adsabs.harvard.edu/abs/2016MNRAS.455.1553R).

## Disclaimer & Attribution

If you use this notebook for your published science, please acknowledge the following:

* Data Lab concept paper: Fitzpatrick *et al.*, "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, http://dx.doi.org/10.1117/12.2057445
* Data Lab disclaimer: http://datalab.noao.edu/disclaimers.php_____no_output_____## Imports & Setup_____no_output_____

# Standard library
from contextlib import contextmanager
from getpass import getpass
from io import BytesIO
from bz2 import BZ2File
from time import sleep
# matplotlib, etc.
%matplotlib inline
import matplotlib
matplotlib.rcParams['figure.figsize'] = (10.0, 10.0)
import matplotlib.pyplot as plt
from astropy.io import fits
from astropy.visualization import astropy_mpl_style
plt.style.use(astropy_mpl_style)
from matplotlib.font_manager import fontManager, FontProperties
import numpy as np
from astropy.io.votable import parse_single_table
# Data Lab
from dl import queryClient as qc, storeClient as sc

def sdss_objid(data):
    """Convert SDSS photometric identifiers into CAS-style ObjID.

    Bits are assigned in objid thus:

    ===== ========== ===============================================
    Bits  Name       Comment
    ===== ========== ===============================================
    63    empty      unassigned
    59-62 skyVersion resolved sky version (0-15)
    48-58 rerun      number of pipeline rerun
    32-47 run        run number
    29-31 camcol     camera column (1-6)
    28    firstField [is this the first field in segment?] 0 for now
    16-27 field      field number within run
    0-15  object     object number within field
    ===== ========== ===============================================

    Parameters
    ----------
    data : :class:`numpy.ndarray`
        Array containing 'rerun', 'run', 'camcol', 'field', 'id'.

    Returns
    -------
    :class:`numpy.ndarray` of :class:`numpy.uint64`
        The ObjIDs of the objects.

    Raises
    ------
    ValueError
        If the sizes of the arrays don't match or if the array values are
        out of bounds.

    Notes
    -----
    * ``firstField`` flag never set.
    * On 32-bit systems, makes sure to explicitly declare all inputs as
      64-bit integers.
    """
    skyversion = np.zeros(data.shape, dtype=np.uint64) + 2
    rerun = np.array(list(map(int, data['rerun'].tolist())), dtype=np.uint64)
    run = data['run'].astype(np.uint64)
    camcol = data['camcol'].astype(np.uint64)
    firstfield = np.zeros(data.shape, dtype=np.uint64)
    field = data['field'].astype(np.uint64)
    objnum = data['id'].astype(np.uint64)
    #
    # Check ranges of parameters
    #
    if ((skyversion < 0) | (skyversion >= 16)).any():
        raise ValueError("skyversion values are out-of-bounds!")
    if ((rerun < 0) | (rerun >= 2**11)).any():
        raise ValueError("rerun values are out-of-bounds!")
    if ((run < 0) | (run >= 2**16)).any():
        raise ValueError("run values are out-of-bounds!")
    if ((camcol < 1) | (camcol > 6)).any():
        raise ValueError("camcol values are out-of-bounds!")
    if ((field < 0) | (field >= 2**12)).any():
        raise ValueError("field values are out-of-bounds!")
    if ((objnum < 0) | (objnum >= 2**16)).any():
        raise ValueError("id values are out-of-bounds!")
    #
    # Compute the objid
    #
    objid = ((skyversion << 59) |
             (rerun << 48) |
             (run << 32) |
             (camcol << 29) |
             (firstfield << 28) |
             (field << 16) |
             (objnum))
    return objid

def asinh_mag(data, magtype='model', bands='ugriz', deredden=False):
    """Convert fluxes to asinh magnitudes.

    Parameters
    ----------
    data : :class:`numpy.ndarray`
        Array containing fluxes in units of nanomaggies.
    magtype : :class:`str`, optional
        Name of the type of magnitude to compute, *e.g.*, 'model' or 'psf'.
        'model' is the default.
    bands : :class:`str`, optional
        Filter bands to compute.  Default is 'ugriz'.
    deredden : :class:`bool`, optional
        If ``True``, also apply extinction correction to the magnitude.
        Extinction values must be present in `data`.

    Returns
    -------
    :class:`numpy.ndarray`
        Asinh magnitudes with columns renamed to, *e.g.*, ``modelflux_g`` to ``modelmag_g``.
    """
    dt = data['{0}flux_{1}'.format(magtype, bands[0])].dtype
    C = -2.5/np.log(10.0)
    b = np.array([(1.4e-10, 0.9e-10, 1.2e-10, 1.8e-10, 7.4e-10)], dtype=[(str(band), dt) for band in bands])
    mag = np.zeros(data.shape, dtype=[(str('{0}mag_{1}'.format(magtype, band)), dt) for band in bands])
    for i, band in enumerate(bands):
        mag['{0}mag_{1}'.format(magtype, band)] = C*(np.arcsinh((data["{0}flux_{1}".format(magtype, band)]*1.0e-9)/(2.0*b[band])) + np.log(b[band]))
        if deredden:
            mag['{0}mag_{1}'.format(magtype, band)] -= data['extinction_{0}'.format(band)]
    return mag

def auxilliary_colors(data, bands='ugriz'):
    """Compute some artificial color quantities useful for target selection.

    Parameters
    ----------
    data : :class:`numpy.ndarray`
        Array containing MODEL fluxes.
    bands : :class:`str`, optional
        Filter bands to compute.  Default is 'ugriz'.

    Returns
    -------
    :class:`numpy.ndarray`
        Array containing auxilliary colors.
    """
    dt = data['modelmag_{0}'.format(bands[0])].dtype
    aux = np.zeros(data.shape, dtype=[(str('c_par'), dt), (str('c_perp'), dt), (str('d_perp'), dt)])
    aux['c_par'] = (0.7 * (data['modelmag_g'] - data['modelmag_r']) +
                    1.2 * (data['modelmag_r'] - data['modelmag_i'] - 0.18))
    aux['c_perp'] = ((data['modelmag_r'] - data['modelmag_i']) -
                     (data['modelmag_g'] - data['modelmag_r'])/4.0 - 0.18)
    aux['d_perp'] = ((data['modelmag_r'] - data['modelmag_i']) -
                     (data['modelmag_g'] - data['modelmag_r'])/8.0)
    return aux

def extract_spectrum(plate, mjd, fiber, run2d='v5_10_0', release=14, sas='sdss_dr14://'):
    """Extract a spectrum from a spPlate file.

    Parameters
    ----------
    plate : :class:`int`
        Plate of observation.
    mjd : :class:`int`
        MJD of observation.
    fiber : :class:`int`
        Fiber number.
    run2d : :class:`str`, optional
        Reduction id.
    release : :class:`int`, optional
        Data release number.
    sas : :class:`str`, optional
        Root path to SDSS data.

    Returns
    -------
    :func:`tuple`
        A tuple containing wavelength, flux, inverse variance of flux, and sky spectrum.
    """
    vopath = "{0}sdss/spectro/redux/{1}/{2:04d}/spPlate-{2:04d}-{3:05d}.fits".format(sas, run2d, plate, mjd)
    with fits.open(sc.get(vopath, mode='fileobj')) as hdulist:
        hdr = hdulist[0].header
        flux = hdulist[0].data[fiber-1, :]
        ivar = hdulist[1].data[fiber-1, :]
        sky = hdulist[6].data[fiber-1, :]
    loglam = hdr['COEFF0'] + hdr['COEFF1']*np.arange(hdr['NAXIS1'], dtype=flux.dtype)
    wavelength = 10.0**loglam
    return wavelength, flux, ivar, sky

def plot_spectrum(plate, mjd, fiber, showsky=False, **kwargs):
    """Simple SDSS/BOSS spectrum plot.

    Parameters
    ----------
    plate : :class:`int`
        Plate of observation.
    mjd : :class:`int`
        MJD of observation.
    fiber : :class:`int`
        Fiber number.
    showsky : :class:`bool`, optional
        If ``True``, plot the sky spectrum
    """
    wavelength, flux, ivar, sky = extract_spectrum(plate, mjd, fiber, **kwargs)
    fig = plt.figure(dpi=100)
    ax = fig.add_subplot(111)
    p0 = ax.plot(wavelength, flux, 'k-', linewidth=1, label='OBJECT')
    if showsky:
        p1 = ax.plot(wavelength, sky, 'm-', linewidth=1, label='SKY')
    foo = ax.set_title("{0:04d}-{1:05d}-{2:04d}".format(plate, mjd, fiber))
    foo = ax.set_xlabel("Wavelength [Å]")
    foo = ax.set_ylabel(r"Flux [$10^{-17} \; \mathrm{erg} \; \mathrm{cm}^{-2} \; \mathrm{s}^{-1} \; \mathrm{\AA}^{-1}$]")
    l = ax.legend()_____no_output_____

## BOSS Galaxies in the SDSS Photometric Catalog

Currently, NOAO Data Lab has loaded a subset of the full SDSS photometric catalog that contains the most commonly used data columns from the full catalog.  This subset is actually provided by SDSS for convenience.  This subset is colloquially known as the "Sweep files".  The contents of these files is described in this [data model](https://data.sdss.org/datamodel/files/PHOTO_SWEEP/RERUN/calibObj.html).

There are several types of flux measurements reported by SDSS.  See the [SDSS Magnitudes page](http://www.sdss.org/dr13/algorithms/magnitudes/) for further details.  The "Sweep files" only contain *fluxes* not magnitudes, so we have to convert the fluxes to (asinh) magnitudes offline.

We're going to select several measurements of flux, and choose objects marked as `SURVEY_PRIMARY` in the [RESOLVE_STATUS bitmask](http://www.sdss.org/dr13/algorithms/bitmasks/#RESOLVE_STATUS) (see the [resolve](http://www.sdss.org/dr13/algorithms/resolve/) documentation for further details.

Note that bitwise operations that are normally part of the SQL standard are forbidden by [ADQL](http://www.ivoa.net/documents/latest/ADQL.html) (even though the `&` operator is part of the ADQL standard)._____no_output_____

release = 13
select_magnitudes = ', '.join([', '.join(['{0}flux_{1}'.format(magtype, b)
                                          for b in 'ugriz'])
                               for magtype in ('model', 'dev', 'exp', 'psf', 'fiber2')])
select_cmodel = ', '.join(['((fracdev_{0} * devflux_{0}) + (1.0 - fracdev_{0}) * expflux_{0}) AS cmodelflux_{0}'.format(b)
                           for b in 'ugriz'])
select_other = ', '.join([', '.join(['{0}_{1}'.format(col, b) for b in 'ugriz'])
                          for col in ('fracdev', 'theta_dev', 'extinction')])
q = ("SELECT rerun, run, camcol, field, id, ra, dec, " +
     select_magnitudes + ", " +
     select_cmodel + ", " +
     select_other +
     " FROM sdss_dr{0:d}.calibobj_gal WHERE (resolve_status & 256) != 0 LIMIT 100000".format(release))
     # " FROM sdss_dr{0:d}.calibobj_gal LIMIT 10".format(release))
print(q)_____no_output_____%%time
try:
    vot = qc.query(sql=q, fmt='votable', async=False)
except Exception as e:
    print(e.message)
else:
    print(vot[:50])_____no_output_____result = parse_single_table(BytesIO(vot.encode('utf-8')))
objID = sdss_objid(result.array)
psfmag = asinh_mag(result.array, magtype='psf', deredden=True)
cmodelmag = asinh_mag(result.array, magtype='cmodel', deredden=True)
modelmag = asinh_mag(result.array, magtype='model', deredden=True)
fib2mag = asinh_mag(result.array, magtype='fiber2', deredden=True)
rdev_i = result.array['theta_dev_i']/0.396  # convert back to pixels
auxcolor = auxilliary_colors(modelmag)_____no_output_____LOWZ = ((cmodelmag['cmodelmag_r'] < (13.5 + auxcolor['c_par']/0.3)) &
        (np.abs(auxcolor['c_perp']) < 0.2) &
        (cmodelmag['cmodelmag_r'] > 16.0) & (cmodelmag['cmodelmag_r'] < 19.6) &
        ((psfmag['psfmag_r'] - cmodelmag['cmodelmag_r']) > 0.3))
CMASS = ((cmodelmag['cmodelmag_i'] < (19.86 + 1.6*(auxcolor['d_perp'] - 0.8))) &
         (cmodelmag['cmodelmag_i'] > 17.5) & (cmodelmag['cmodelmag_i'] < 19.9) &
         (auxcolor['d_perp'] > 0.55) &
         ((psfmag['psfmag_i'] - modelmag['modelmag_i']) > (0.2 + 0.2*(20.0 - modelmag['modelmag_i']))) &
         ((psfmag['psfmag_z'] - modelmag['modelmag_z']) > (9.125 - 0.46*modelmag['modelmag_z'])) &
         ((modelmag['modelmag_r'] - modelmag['modelmag_i']) < 2) &
         (fib2mag['fiber2mag_i'] < 21.5) &
         (rdev_i < 20))_____no_output_____print(LOWZ.sum())
print(CMASS.sum())_____no_output_____fig = plt.figure(dpi=100)
ax = fig.add_subplot(111)
p1 = ax.plot(modelmag['modelmag_g'] - modelmag['modelmag_r'], modelmag['modelmag_r'], 'k,', label='model')
p2 = ax.plot(modelmag['modelmag_g'][LOWZ] - modelmag['modelmag_r'][LOWZ], modelmag['modelmag_r'][LOWZ], 'gs', label='LOWZ')
p3 = ax.plot(modelmag['modelmag_g'][CMASS] - modelmag['modelmag_r'][CMASS], modelmag['modelmag_r'][CMASS], 'ro', label='CMASS')
foo = p2[0].set_markeredgecolor('g')
foo = p2[0].set_markerfacecolor('none')
foo = p3[0].set_markeredgecolor('r')
foo = p3[0].set_markerfacecolor('none')
foo = ax.set_xlim([-5, 10])
foo = ax.set_ylim([24, 15])
foo = ax.set_xlabel('g - r')
foo = ax.set_ylabel('r')
foo = ax.set_title('Color-Magnitude of SDSS (DR13) Galaxies')
l = ax.legend(numpoints=1)_____no_output_____fig = plt.figure(dpi=100)
ax = fig.add_subplot(111)
p1 = ax.plot(modelmag['modelmag_g'] - modelmag['modelmag_r'],
             modelmag['modelmag_r'] - modelmag['modelmag_i'], 'k,', label='model')
p2 = ax.plot(modelmag['modelmag_g'][LOWZ] - modelmag['modelmag_r'][LOWZ],
             modelmag['modelmag_r'][LOWZ] - modelmag['modelmag_i'][LOWZ], 'gs', label='LOWZ')
p3 = ax.plot(modelmag['modelmag_g'][CMASS] - modelmag['modelmag_r'][CMASS],
             modelmag['modelmag_r'][CMASS] - modelmag['modelmag_i'][CMASS], 'ro', label='CMASS')
foo = p2[0].set_markeredgecolor('g')
foo = p2[0].set_markerfacecolor('none')
foo = p3[0].set_markeredgecolor('r')
foo = p3[0].set_markerfacecolor('none')
foo = ax.set_xlim([-5, 15])
foo = ax.set_ylim([-2, 4])
foo = ax.set_xlabel('g - r')
foo = ax.set_ylabel('r - i')
foo = ax.set_title('Colors of SDSS (DR13) Galaxies')
l = ax.legend(numpoints=1)_____no_output_____

## BOSS Galaxies in the SDSS Spectroscopic Catalog

### Number of Objects

How many objects are in the specObj table?_____no_output_____

%%time
q = "SELECT COUNT(*) AS nrows FROM sdss_dr{0:d}.specobj".format(release)
result = qc.query(adql=q, fmt='csv')
print(result)_____no_output_____

### Select LOWZ and CMASS spectra.

The [BOSS_TARGET1 bitmask](http://www.sdss.org/dr13/algorithms/bitmasks/#BOSS_TARGET1) makes this very easy.  SDSS [uses bitmasks extensively](http://www.sdss.org/dr13/algorithms/bitmasks/)!_____no_output_____

%%time
q = ("SELECT boss_target1 FROM sdss_dr{0:d}.specobj WHERE boss_target1 != 0 LIMIT 100000".format(release))
try:
    vot = qc.query(sql=q, fmt='votable')
except Exception as e:
    print(e.message)
else:
    result = parse_single_table(BytesIO(vot.encode('utf-8')))
    boss_target1 = result.array['boss_target1']
    print(boss_target1)_____no_output_____for mask in (1, 5, 6, 9, 14, 15, 33, 41, 133, 134, 135, 140, 141, 142):
    print(mask, (boss_target1 == mask).sum())_____no_output_____%%time
q_lowz = ("SELECT specobjid, fluxobjid, bestobjid, plate, mjd, fiberid, z_noqso FROM sdss_dr{0:d}.specobj " +
          "WHERE (boss_target1 & {1:d}) != 0 AND specprimary = 1 AND " +
          "zwarning_noqso = 0 AND class_noqso = 'GALAXY' LIMIT 100000").format(release, 1)
# jobId = qc.query(sql=q, fmt='votable', async=True)
result = parse_single_table(BytesIO(qc.query(sql=q_lowz, fmt='votable').encode('utf-8')))
plate_lowz = result.array['plate']
mjd_lowz = result.array['mjd']
fiber_lowz = result.array['fiberid']
redshift_lowz = result.array['z_noqso']
print(len(redshift_lowz))_____no_output_____%%time
q_cmass = ("SELECT specobjid, fluxobjid, bestobjid, plate, mjd, fiberid, z_noqso FROM sdss_dr{0:d}.specobj " +
           "WHERE (boss_target1 & {1:d}) != 0 AND specprimary = 1 AND " +
           "zwarning_noqso = 0 AND class_noqso = 'GALAXY' LIMIT 100000").format(release, 2)
result = parse_single_table(BytesIO(qc.query(sql=q_cmass, fmt='votable').encode('utf-8')))
plate_cmass = result.array['plate']
mjd_cmass = result.array['mjd']
fiber_cmass = result.array['fiberid']
redshift_cmass = result.array['z_noqso']
print(len(redshift_cmass))_____no_output_____

### Redshift Distribution

Here's the redshift distribution of LOWZ and CMASS galaxies.  Note that the number of bins for CMASS is chosen so the bin *sizes* are approximately the same as the LOWZ sample (when the entire data set can be queried)._____no_output_____

fig = plt.figure(dpi=100)
ax = fig.add_subplot(111)
n_lowz, bins_lowz, patches_lowz = ax.hist(redshift_lowz, 50, facecolor='green', alpha=0.5, label='LOWZ')
n_cmass, bins_cmass, patches_cmass = ax.hist(redshift_cmass, 100, facecolor='red', alpha=0.5, label='CMASS')
foo = ax.set_yscale('log')
foo = ax.grid(True)
foo = ax.set_xlabel("redshift")
foo = ax.set_ylabel("N")
foo = ax.set_title("BOSS redshifts")
foo = ax.legend()_____no_output_____

### What's that? Some CMASS Galaxies have negative redshift?

Let's actually take a look at these spectra._____no_output_____

rerun = 'v5_10_0'
from IPython.core.display import display, HTML
negative_z = (redshift_cmass > bins_cmass[0]) & (redshift_cmass < bins_cmass[1])
display(HTML('
'))
for p, m, f in zip(plate_cmass[negative_z], mjd_cmass[negative_z], fiber_cmass[negative_z]):
    display(HTML('
{2}, {3}, {4}
'.format(release, rerun, p, m, f)))
display(HTML('
'))_____no_output_____

Hmmm, after looking at a few of these, a number of them are point-like, red objects that are close to a brighter object.  A plausible hypothesis is that CMASS color selection doesn't work so great when a nearby, brighter object might be contaminating the photometry, making it harder to distinguish LRGs from M stars._____no_output_____## Plotting SDSS Spectra

In addition to databases that contain the SDSS catalog data, NOAO Data Lab will also contain the images and spectra themselves.  In fact, some spectroscopic data is already in the NOAO archive, so we can plot it directly from a Data Lab notebook!

First, some spectra from the previous list.  You can get a better view of these spectra from the links above, but this demonstrates the fun fact that many BOSS spectra are *fainter* than the sky!_____no_output_____

plot_spectrum(4202, 55445, 3, showsky=True)_____no_output_____

Here's another from the list above.  If you look in the 6000 - 8000 Å range, you can see some features that resemble certain M stars._____no_output_____

plot_spectrum(4203, 55447, 651)_____no_output_____

Here's a "classic" LRG at redshift 0.67 (which was actually targeted as LOWZ!)._____no_output_____

plot_spectrum(4055, 55359, 408)_____no_output_____

## Display SDSS Images

Practice plotting corrected frame images in Data Lab.  The original image is [here](https://dr12.sdss.org/fields/runCamcolField?field=13&camcol=1&run=6122)._____no_output_____

def asinh_image(image, band='r'):
    """Convert an image in nanomaggies into asinh magnitudes.

    Parameters
    ----------
    image : :class:`numpy.ndarray`
        Image data, read from *e.g.* a SDSS-III corrected frame file.
    band : :class:`str`, optional
        Filter band, default *r*.

    Returns
    -------
    :class:`numpy.ndarray`
        Image data converted to asinh magnitudes.
    """
    C = -2.5/np.log(10.0)
    b = dict(u=1.4e-10, g=0.9e-10, r=1.2e-10, i=1.8e-10, z=7.4e-10)
    mag = C*(np.arcsinh((image*1.0e-9)/(2.0*b[band])) + np.log(b[band]))
    return mag

def extract_frame(run, camcol, field, band, sas='sdss_dr14://'):
    """Obtain image data from a SDSS corrected frame, which may be at a remote location.

    Parameters
    ----------
    run : :class:`int`
        Run number.
    camcol : :class:`int`
        Column number.
    field : :class:`int`
        Field number
    band : :class:`str`
        Filter band.
    sas : :class:`str`, optional
        Root path to SDSS data.

    Returns
    -------
    :class:`numpy.ndarray`
        Image data.
    """
    fm = '{0}eboss/photoObj/frames/301/{1:d}/{2:d}/frame-{3}-{1:06d}-{2:d}-{4:04d}.fits.bz2'.format(sas, run, camcol, band, field)
    with BZ2File(BytesIO(sc.get(fm, mode='binary'))) as cf:
        with fits.open(cf) as hdulist:
            image = hdulist[0].data
    return image

def display_frame(run, camcol, field, band, invert=False, **kwargs):
    """Display a SDSS corrected frame.

    Parameters
    ----------
    run : :class:`int`
        Run number.
    camcol : :class:`int`
        Column number.
    field : :class:`int`
        Field number.
    band : :class:`str`
        Filter band.
    invert : :class:`bool`, optional
        If ``True`` display an inverted grayscale image.
    sas : :class:`str`, optional
        Root path to SDSS data.

    Returns
    -------
    :func:`tuple`
         Handles to the figure, image and colorbar.
    """
    color_map = 'gray_r'
    if invert:
        color_map = 'gray'
    image = extract_frame(run, camcol, field, band, **kwargs)
    mag_image = asinh_image(image, band=band)
    fig = plt.figure(dpi=100)
    im = plt.imshow(mag_image, cmap=color_map, clim=(26, 18), origin='lower')
    foo = im.axes.set_title('frame-{0}-{1:06d}-{2:d}-{3:04d}.fits'.format(band, run, camcol, field))
    foo = im.axes.grid(True)
    cb = plt.colorbar()
    return fig, im, cb_____no_output_____image = extract_frame(6122, 1, 13, 'r')
fig = plt.figure(dpi=100)
im = plt.imshow(image, cmap='gray', clim=(0.0, 1.0), origin='lower')
foo = im.axes.set_title('frame-r-006122-1-0013.fits')
foo = im.axes.grid(True)
cb = plt.colorbar()_____no_output_____fig = plt.figure(dpi=100)
ax = fig.add_subplot(111)
n, bins, patches = ax.hist(image.flatten(), bins=1000, facecolor='k', edgecolor='k')
foo = ax.set_xscale('log')
foo = ax.set_yscale('log')
foo = ax.grid(True)
foo = ax.set_xlabel('Pixel Flux [nanomaggies]')
foo = ax.set_ylabel('N')
foo = ax.set_title('frame-r-006122-1-0013.fits')_____no_output_____mag_image = asinh_image(image, band='r')
fig = plt.figure(dpi=100)
ax = fig.add_subplot(111)
n, bins, patches = ax.hist(mag_image.flatten(), bins=1000, facecolor='k', edgecolor='k')
# foo = ax.set_xscale('log')
foo = ax.set_yscale('log')
foo = ax.grid(True)
foo = ax.set_xlabel('Pixel Magnitude [asinh magnitude]')
foo = ax.set_ylabel('N')
foo = ax.set_title('frame-r-006122-1-0013.fits')# foo = ax.set_xlabel("redshift")_____no_output_____fig = plt.figure(dpi=100)
im = plt.imshow(mag_image, cmap='gray', clim=(26, 18), origin='lower')
foo = im.axes.set_title('frame-r-006122-1-0013.fits')
foo = im.axes.grid(True)
cb = plt.colorbar()_____no_output_____fig = plt.figure(dpi=100)
im = plt.imshow(mag_image, cmap='gray_r', clim=(26, 18), origin='lower')
foo = im.axes.set_title('frame-r-006122-1-0013.fits')
foo = im.axes.grid(True)
cb = plt.colorbar()_____no_output_____from matplotlib.colors import LogNorm
fig = plt.figure(dpi=100)
im = plt.imshow(image, cmap='gray', norm=LogNorm(), origin='lower')
foo = im.axes.set_title('frame-r-006122-1-0013.fits')
foo = im.axes.grid(True)
cb = plt.colorbar()_____no_output_____foo = display_frame(6122, 1, 13, 'u')_____no_output_____foo = display_frame(6122, 1, 13, 'g')_____no_output_____foo = display_frame(6122, 1, 13, 'r')_____no_output_____foo = display_frame(6122, 1, 13, 'i')_____no_output_____foo = display_frame(6122, 1, 13, 'g', invert=True)_____no_output_____foo = display_frame(6122, 1, 13, 'z')_____no_output_____

## Resources & References

* BOSS: [Kyle S. Dawson, *et al.*, "The Baryon Oscillation Spectroscopic Survey of SDSS-III," *Astron. J.* **145** (2013) 10.](http://adsabs.harvard.edu/abs/2013AJ....145...10D)
* eBOSS: [Kyle S. Dawson, *et al.*, "The SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Overview and Early Data," *Astron. J.* **151** (2016) 44.](http://adsabs.harvard.edu/abs/2016AJ....151...44D)
* SDSS-IV: [Michael R. Blanton, *et al.*, "Sloan Digital Sky Survey IV: Mapping the Milky Way, Nearby Galaxies and the Distant Universe," *Astron. J.* **154** (2017) 28.](http://adsabs.harvard.edu/abs/2017AJ....154...28B)
* DR13: [Franco D. Albareti, *et al.*, "The 13th Data Release of the Sloan Digital Sky Survey: First Spectroscopic Data from the SDSS-IV Survey Mapping Nearby Galaxies at Apache Point Observatory," *Astrophys. J. Supp.* **233** (2017) 25.](http://adsabs.harvard.edu/abs/2017ApJS..233...25A)
* DR14: [Bela Abolfathi, *et al.*, "The Fourteenth Data Release of the Sloan Digital Sky Survey: First Spectroscopic Data from the extended Baryon Oscillation Spectroscopic Survey and from the Second Phase of the Apache Point Observatory Galactic Evolution Experiment," *Astrophys. J. Supp.* **235** (2018) 42.](http://adsabs.harvard.edu/abs/2018ApJS..235...42A)_____no_output_____


# Notebook from naheld/Held2020_TrichoDiel
Path: .ipynb_checkpoints/Held2020_TrichodesmiumDiel_AnalysesandPlots-checkpoint.ipynb

# Analyses for Held et al., 2020 "Dynamic proteome and diel nitrogenase activity supports buoyancy in Trichodesmium erythraeum"_____no_output_____## Dependencies_____no_output_____

# all easily acquired from conda (https://anaconda.org/anaconda/conda)
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
import networkx as nx
from scipy.optimize import curve_fit
from scipy import stats_____no_output_____

## Figure 1. Clustered heatmap of Trichodesmium diel proteome_____no_output_____

# using protein data mapped to KO values
KOoutput = pd.read_csv('Held2020_trichodiel_proteindata_withKOvalues.csv')
KOgrouped = KOoutput.groupby('children__children__children__name').mean()

KOgroupedt = KOgrouped.transpose()
KOgroupedt['tod'] = KOgroupedt.index.values
KOgroupedt['tod'] = KOgroupedt['tod'].str.split('l').apply(lambda x: x[1][:-2]).astype(float)
KOgroupedtg = KOgroupedt.groupby('tod').mean()

# take the log to make the visualization easier to understand
KOgroupedtglog = KOgroupedtg.apply(lambda x: np.log(x + 1))

sns.clustermap(KOgroupedtglog.transpose(), cmap="RdPu", standard_scale=0, figsize=(8,12))/Users/noelle/miniconda3/lib/python3.6/site-packages/matplotlib/font_manager.py:1331: UserWarning: findfont: Font family ['normal'] not found. Falling back to DejaVu Sans
  (prop.get_family(), self.defaultFamily[fontext]))
# Line plot, colored by the cluster

# normalize to mean
# divide by first value
KOgroupedt = KOgrouped.transpose()
KOgroupedt['tod'] = KOgroupedt.index.values
KOgroupedt['tod'] = KOgroupedt['tod'].str.split('l').apply(lambda x: x[1][:-2]).astype(float)
KOgroupedtg = KOgroupedt.groupby('tod').mean()

divideby = KOgroupedtg.iloc[0:-1,1:-1].mean()
KOgroupedtg.iloc[:,1:-2] = KOgroupedtg.iloc[:,1:-2].div(divideby, axis='columns')
#KOgroupedtg = KOgroupedtg.apply(lambda x: (np.log(x+1)))

KOgroupedtg['color'] = 0
pinklist = ['DNA polymerase', 'Carbon fixation', 'ATP synthesis', 'Photosynthesis', 'Cysteine and methionine metabolism', 'Nitrogen metabolism']
greenlist = ['Central carbohydrate metabolism', 'Ribosome', 'Methane metabolism', 'RNA Processing']
for module in KOgroupedtg.columns:
    print(module)
    if module in pinklist:
        KOgroupedtg.loc['color', module] = 'darkblue'
    elif module in greenlist:
        KOgroupedtg.loc['color', module] = 'magenta'

    else:
        KOgroupedtg.loc['color', module] = 'mediumseagreen'
KOgroupedtg.drop(['Two-component regulatory system', 'ABC-2 type and other transport systems','DNA polymerase','Lipopolysaccharide metabolism', 'Biosynthesis of secondary metabolites', 'Other terpenoid biosynthesis', 'Terpenoid backbone biosynthesis','Peptide and nickel transport system','Saccharide, polyol, and lipid transport system','Histidine metabolism','Metallic cation, iron-siderophore and vitamin B12 transport system'], inplace=True, axis=1)
fig, ax = plt.subplots(figsize=(10,5))
#ax.set_ylim([0,2])
for column in KOgroupedtg:
    ax.plot(KOgroupedtg.index[:-1], KOgroupedtg[column][:-1], color = KOgroupedtg[column][-1])

plt.rcParams.update({'font.size': 22})ABC-2 type and other transport systems
ATP synthesis
Aminoacyl tRNA
Arginine and proline metabolism
Aromatic amino acid metabolism
Bacterial secretion system
Biosynthesis of secondary metabolites
Branched-chain amino acid metabolism
Carbon fixation
Central carbohydrate metabolism
Cofactor and vitamin biosynthesis
Cysteine and methionine metabolism
DNA polymerase
Drug resistance
Fatty acid metabolism
Histidine metabolism
Lipid metabolism
Lipopolysaccharide metabolism
Lysine metabolism
Metallic cation, iron-siderophore and vitamin B12 transport system
Methane metabolism
Mineral and organic ion transport system
Nitrogen metabolism
Nucleotide sugar
Other amino acid metabolism
Other carbohydrate metabolism
Other terpenoid biosynthesis
Peptide and nickel transport system
Phosphate and amino acid transport system
Photosynthesis
Polyamine biosynthesis
Protein processing
Purine metabolism
Pyrimidine metabolism
RNA polymerase
RNA processing
Ribosome
Saccharide, polyol, and lipid transport system
Serine and threonine metabolism
Sulfur metabolism
Terpenoid backbone biosynthesis
Two-component regulatory system
color

# Figure 2. Line plots with confidence intervals_____no_output_____

proteinsforlineplots = KOoutput.transpose()
proteinsforlineplots.rename(columns=proteinsforlineplots.iloc[0],inplace=True)
dft = proteinsforlineplots.drop('trichotimeofday').reset_index(0)
dft = dft.iloc[:40]
dft['tod'] = dft['index'].str.split('l').apply(lambda x: x[1][:-2]).astype(float)
dft['rep'] = dft['index'].str.split('_').apply(lambda x: x[1]).astype(float)
dft.sort_values(by=["tod","rep"], inplace=True, ascending=True)
dft.reset_index(inplace=True)

# Here also loading in CHN and acetylene reduction data
chn = pd.read_csv('chndatanew_military_permL.csv')
n2fixdf = pd.read_csv("n2fixratesnew_military.csv")
chngrouped = chn.groupby('hourspostdusk').mean()
n2fixmean = n2fixdf.groupby('hourspastdusk').mean()

# divide by the protein's mean
divideby = dft.iloc[:,2:-3].mean() + 0.001
dft.iloc[:,2:-3] = dft.iloc[:,2:-3].div(divideby, axis='columns').astype(float)
dftgrouped = dft.groupby('tod').mean()

# Here I provide an example but this could be adapted to display any protein of interest

# Matplotlib global parameters
font = {'family' : 'normal',
        'weight' : 'normal',
        'size'   : 16}

plt.rc('font', **font)

fig, axes = plt.subplots(5,1,figsize=(6,12))

plt.xlim((0,24))

# draw the line plots on indiivdual axes - doing this manually here
sns.lineplot(ax = axes[0], x=chn['hourspostdusk'], y=chn['c:n'], c='purple',  ci=90, marker="o")
sns.lineplot(ax = axes[1], x=chn['hourspostdusk'], y=chn['c'], c='yellowgreen',  ci=90, marker="o")
sns.lineplot(ax=axes[2], x='tod', y='Tery_4669 psaA photosystem I core protein PsaA [Trichodesmium erythraeum IMS101: NC_008312]',data=dft,label='PsaA', ci=90, c='green',marker="o")
sns.lineplot(ax=axes[3],x=dft['tod'], y=dft['Tery_4668 psaB photosystem I core protein PsaB [Trichodesmium erythraeum IMS101: NC_008312]'],label='PsaB', c='green',  ci=90,marker="o")
sns.lineplot(ax=axes[4],x=dft['tod'], y=dft['Tery_3791  photosystem I protein PsaD [Trichodesmium erythraeum IMS101: NC_008312]'], label='PsaD', c='green',  ci=90,marker="o")

# axes labels - here setting manually
axes[0].set_ylabel('POC:PON\nratio', fontsize=14)
axes[1].set_ylabel('POC\n(ug/mL)', fontsize=14)
axes[2].set_ylabel('Protein fold\nchg. from avg.', fontsize=14)
axes[3].set_ylabel('Protein fold\nchg. from avg.', fontsize=14)
axes[4].set_ylabel('Protein fold\nchg. from avg.', fontsize=14)

axes[0].set_xlabel('', fontsize=16)
axes[1].set_xlabel('', fontsize=16)
axes[2].set_xlabel('', fontsize=16)
axes[3].set_xlabel('', fontsize=16)
axes[4].set_xlabel('Time of day (hours)', fontsize=16)

axes[2].set_ylim([0,3])
axes[3].set_ylim([0,3])
axes[4].set_ylim([0,3])

axes[2].legend(loc=1)
axes[3].legend(loc=1)
axes[4].legend(loc=1)

axes[2].get_legend().remove()
axes[3].get_legend().remove()
axes[4].get_legend().remove()

for ax in axes:
    ax.set_xlim([0,25])

plt.setp(axes[0].get_xticklabels(), visible=False)
plt.setp(axes[1].get_xticklabels(), visible=False)
plt.setp(axes[2].get_xticklabels(), visible=False)
plt.setp(axes[3].get_xticklabels(), visible=False)

# display in the notebook
plt.show()/Users/noelle/miniconda3/lib/python3.6/site-packages/matplotlib/cbook/__init__.py:2448: UserWarning: Saw kwargs ['c', 'color'] which are all aliases for 'color'.  Kept value from 'color'
  seen=seen, canon=canonical, used=seen[-1]))
/Users/noelle/miniconda3/lib/python3.6/site-packages/matplotlib/font_manager.py:1331: UserWarning: findfont: Font family ['normal'] not found. Falling back to DejaVu Sans
  (prop.get_family(), self.defaultFamily[fontext]))

## Figure 2. Changes in POC and PON content, colored by inferred process_____no_output_____

plt.rcParams.update({'font.size': 14})

# calculate change in C:N ratio over time
#plt.rcparams(fontsize=13)
differencechn = chngrouped.diff()
for index, row in differencechn.iterrows():
    # excess C production
    if row['c:n'] > 0 and row['c'] > 0:
        differencechn.loc[index,'color'] = 'darkgreen'
    # excess N consumption
    elif row['c:n'] > 0 and row['n'] < 0:
        differencechn.loc[index,'color'] = 'pink'
    # excess C production and N consumption
    elif row['c:n'] > 0 and row['n'] < 0 and row['c'] > 0 :
        differencechn.loc[index,'color'] = 'purple'
    # excess N production
    elif row['c:n'] < 0 and row['n'] > 0:
        differencechn.loc[index,'color'] = 'r'
    # excess C respiration
    elif row['c:n'] < 0 and row['c'] < 0:
        differencechn.loc[index,'color'] = 'darkseagreen'
    # excess N production and C respiration
    elif row['c:n'] < 0 and row['n'] > 0 and row['c'] < 0 :
        differencechn.loc[index,'color'] = 'grey'
    else:
        differencechn.loc[index, 'color'] = 'black'

differencechn['positive'] = differencechn['c:n'] > 0
differencechn['cpositive'] = differencechn['c:n'] > 0
differencechn['tod'] = chngrouped['tod']

# draw the plot
fig, ax = plt.subplots(figsize=(6,4))
plt.bar(differencechn['tod'], differencechn['c:n'], color=differencechn['color'])
plt.xlabel('Time of day (hours)')
plt.ylabel('Change POC:PON ratio\nfrom previous time point')
plt.xlim([0,25])_____no_output_____

## Figure 3. Network analyses_____no_output_____

# proteins of interest
# I selected all N2 fix, photosynthesis and various regulatory proteins

protDict = {'NifD' : 'Tery_4137 nifD Mo-nitrogenase MoFe protein subunit NifD precursor (EC 1.18.6.1) [Trichodesmium erythraeum IMS101: NC_008312]',
'NifK' : 'Tery_4138 nifK Mo-nitrogenase MoFe protein subunit NifK (EC 1.18.6.1) [Trichodesmium erythraeum IMS101: NC_008312]',
'NifH' : 'Tery_4136 nifH Mo-nitrogenase iron protein subunit NifH (EC 1.18.6.1) [Trichodesmium erythraeum IMS101: NC_008312]',
'PsaA' : 'Tery_4669 psaA photosystem I core protein PsaA [Trichodesmium erythraeum IMS101: NC_008312]',
'PsaB' : 'Tery_4668 psaB photosystem I core protein PsaB [Trichodesmium erythraeum IMS101: NC_008312]',
'PsaD' : 'Tery_3791  photosystem I protein PsaD [Trichodesmium erythraeum IMS101: NC_008312]',
'PsbC' : 'Tery_0513 psbC photosystem II 44 kDa subunit reaction center protein [Trichodesmium erythraeum IMS101: NC_008312]',
'BPhycoeryth' : 'Tery_0983  Phycobilisome protein [Trichodesmium erythraeum IMS101: NC_008312]',
'Phycocyanin' : 'Tery_5048  phycocyanin, alpha subunit [Trichodesmium erythraeum IMS101: NC_008312]',
'RPhycoeryth' : 'Tery_0996  Phycobilisome protein [Trichodesmium erythraeum IMS101: NC_008312]',
'Allophyco' : 'Tery_3649  allophycocyanin beta subunit apoprotein [Trichodesmium erythraeum IMS101: NC_008312]',
'GlcG' : 'Tery_4395  glucose-1-phosphate adenylyltransferase [Trichodesmium erythraeum IMS101: NC_008312]',
'Glyco\nPhospho.': "Tery_3694  glycogen/starch/alpha-glucan phosphorylases [Trichodesmium erythraeum IMS101: NC_008312]",
'GlgA' : 'Tery_2147 glgA glycogen/starch synthases, ADP-glucose type [Trichodesmium erythraeum IMS101: NC_008312]',
'NtcA' : 'Tery_2023  transcriptional regulator, Crp/Fnr family [Trichodesmium erythraeum IMS101: NC_008312]',
'PII' : 'Tery_2842  nitrogen regulatory protein P-II family [Trichodesmium erythraeum IMS101: NC_008312]',
#'Cyanophycin_synth' : 'Tery_1965  cyanophycin synthetase [Trichodesmium erythraeum IMS101: NC_008312]',
#'Cyanophycinase' : 'Tery_1964  cyanophycinase [Trichodesmium erythraeum IMS101: NC_008312]',

'RpaA' : 'Tery_4937 rpaA two component transcriptional regulator, winged helix family [Trichodesmium erythraeum IMS101: NC_008312]',
'RpaB' : 'Tery_0954 ycf27 two component transcriptional regulator, winged helix family [Trichodesmium erythraeum IMS101: NC_008312]',
'Fur' : 'Tery_1902  response regulator receiver protein [Trichodesmium erythraeum IMS101: NC_008312]',
'KaiB' : 'Tery_3804 kaiB circadian clock protein KaiB [Trichodesmium erythraeum IMS101: NC_008312]',
'KaiC' : 'Tery_3805 kaiC circadian clock protein KaiC [Trichodesmium erythraeum IMS101: NC_008312]',

#'Rubisco': 'Tery_4410 cbbL ribulose 1,5-bisphosphate carboxylase large subunit (EC 4.1.1.39) [Trichodesmium erythraeum IMS101: NC_008312]',
'GroEL': 'Tery_4327 groL2 chaperonin GroEL [Trichodesmium erythraeum IMS101: NC_008312]',
'Ef-Tu': 'Tery_0476 tuf translation elongation factor 1A (EF-1A/EF-Tu) [Trichodesmium erythraeum IMS101: NC_008312]',
'ATP_Synth': 'Tery_3385 atpD ATP synthase F1 subcomplex beta subunit [Trichodesmium erythraeum IMS101: NC_008312]',
'Thioredoxin': 'Tery_3312  thioredoxin [Trichodesmium erythraeum IMS101: NC_008312]',

'Ferredoxin PsaC' : 'Tery_0454 psaC 4Fe-4S ferredoxin, iron-sulfur binding [Trichodesmium erythraeum IMS101: NC_008312]',
'Ferredoxin\n4145' : 'Tery_4145 fdxH ferredoxin (2Fe-2S) [Trichodesmium erythraeum IMS101: NC_008312]',
'Flavodoxin' : 'Tery_1666 fld flavodoxin [Trichodesmium erythraeum IMS101: NC_008312]',
'GvpLGvpF' : 'Tery_2339  Gas vesicle synthesis GvpLGvpF [Trichodesmium erythraeum IMS101: NC_008312]',
'GvpA' : 'Tery_2325 gvpA gas vesicle protein GVPa [Trichodesmium erythraeum IMS101: NC_008312]',
'SOD' : 'Tery_0891  putative nickel-containing superoxide dismutase precursor (NISOD) [Trichodesmium erythraeum IMS101: NC_008312]',
#'NADH\nRed.': 'Tery_1578 ndhI NADH-plastoquinone oxidoreductase, I subunit [Trichodesmium erythraeum IMS101: NC_008312]',
'NADP+\nRed.': "Tery_3658  oxidoreductase FAD/NAD(P)-binding [Trichodesmium erythraeum IMS101: NC_008312]",
'Thioredoxin': 'Tery_3312  thioredoxin [Trichodesmium erythraeum IMS101: NC_008312]',
'Thioredoxin\n331':'Tery_3311  thioredoxin [Trichodesmium erythraeum IMS101: NC_008312]'}
_____no_output_____# generate the network edges (proteins are the nodes)

proteinsforlineplots = KOoutput.transpose()
proteinsforlineplots.rename(columns=proteinsforlineplots.iloc[0],inplace=True)
dft = proteinsforlineplots.drop('trichotimeofday').reset_index(0)
dft = dft.iloc[:40]
dft['tod'] = dft['index'].str.split('l').apply(lambda x: x[1][:-2]).astype(float)
dft['rep'] = dft['index'].str.split('_').apply(lambda x: x[1]).astype(float)
dft.sort_values(by=["tod","rep"], inplace=True, ascending=True)
dft.reset_index(inplace=True)

sdictneg = {}
sdictall = {}
G_symmetricneg = nx.Graph()
G_symmetricall = nx.Graph()
for name, entry in protDict.items():
    for name2, entry2 in protDict.items():
        s,p = stats.spearmanr(dft[entry], dft[entry2])
        if s < -0.8 and p < 0.05:
            sdictneg[name, name2] = s
            sdictall[name, name2] = s
            G_symmetricneg.add_edge(name, name2)
            G_symmetricall.add_edge(name, name2)
sdictpos = {}

G_symmetricpos = nx.Graph()
for name, entry in protDict.items():
    for name2, entry2 in protDict.items():
        s,p = stats.spearmanr(dft[entry], dft[entry2])
        if s > 0.8 and p < 0.05:
            if name != name2:
                sdictpos[name, name2] = s
                sdictall[name, name2] = s
                G_symmetricpos.add_edge(name, name2)
                G_symmetricall.add_edge(name, name2)   _____no_output_____# Number of times a node is present in the shortest path between 2 other nodes - positive correlation
betCent = nx.closeness_centrality(G_symmetricpos)

node_color = [100.0 * G_symmetricpos.degree(v) for v in G_symmetricpos]
node_size =  [v * 1500 for v in betCent.values()]

pos = nx.spring_layout(G_symmetricpos)
plt.figure(figsize=(9,9))
nx.draw_networkx(G_symmetricpos, pos=pos, with_labels=True,
                 node_color=node_color,
                 node_size=node_size,
                  font_size = 13,
                cmap='cool')
plt.axis('off')/Users/noelle/miniconda3/lib/python3.6/site-packages/networkx/drawing/nx_pylab.py:522: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
  if not cb.is_string_like(edge_color) \
/Users/noelle/miniconda3/lib/python3.6/site-packages/networkx/drawing/nx_pylab.py:543: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
  if cb.is_string_like(edge_color) or len(edge_color) == 1:
/Users/noelle/miniconda3/lib/python3.6/site-packages/networkx/drawing/nx_pylab.py:724: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
  if not cb.is_string_like(label):
# Number of times a node is present in the shortest path between 2 other nodes  - negative correlations
betCent = nx.closeness_centrality(G_symmetricneg)

node_color = [100.0 * G_symmetricneg.degree(v) for v in G_symmetricneg]
node_size =  [v * 1500 for v in betCent.values()]

pos = nx.spring_layout(G_symmetricneg)
plt.figure(figsize=(8,8))
G = nx.draw_networkx(G_symmetricneg, pos=pos, with_labels=True,
                 node_color=node_color,
                 node_size=node_size,
                 font_size = 13,
                cmap = 'cool')
plt.axis('off')/Users/noelle/miniconda3/lib/python3.6/site-packages/networkx/drawing/nx_pylab.py:522: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
  if not cb.is_string_like(edge_color) \
/Users/noelle/miniconda3/lib/python3.6/site-packages/networkx/drawing/nx_pylab.py:543: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
  if cb.is_string_like(edge_color) or len(edge_color) == 1:
/Users/noelle/miniconda3/lib/python3.6/site-packages/networkx/drawing/nx_pylab.py:724: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
  if not cb.is_string_like(label):

## Figure 3. Scatter plots and curve fits, to emphasize the positive and negative correlation relationships_____no_output_____

# Defining functions for plots - linear and non-linear fits

font = {'family' : 'normal',
        'weight' : 'normal',
        'size'   : 16}

plt.rc('font', **font)

def plotandoptimize(xs,ys):

    dfz = dft.sort_values(by=protDict[xs])
    xdata = dfz[protDict[xs]]
    ydata = dfz[protDict[ys]]

    fig, axes = plt.subplots(figsize=(4,4))
    sns.scatterplot(xdata,ydata)

    plt.xlabel(xs)
    plt.ylabel(ys)
    plt.tight_layout()

    # fit an exponential curve

    def func(x, a, b, c):
        #return a * np.log(x) + c
        return a * np.exp(-b * x) + c

    popt, pcov = curve_fit(func, xdata, ydata)
    residuals = ydata- func(xdata, *popt)
    ss_res = np.sum(residuals**2)
    ss_tot = np.sum((ydata-np.mean(ydata))**2)
    r_squared = 1 - (ss_res / ss_tot)

    plt.plot(xdata, func(xdata, *popt), 'r-', ms=3, label='a=%5.3f,\nb=%5.3f,\nc=%5.3f' % tuple(popt))

    print(r_squared)
    plt.legend(fontsize=14)

def plotandoptimizelinear(xs,ys):

    xdata = dft[protDict[xs]]
    ydata = dft[protDict[ys]]

    fig, axes = plt.subplots(figsize=(4,4))
    sns.scatterplot(xdata,ydata)

    plt.xlabel(xs)
    plt.ylabel(ys)
    plt.tight_layout()

    # fit an exponential curve

    def func(x, a, b):
        #return a * np.log(x) + c
        return a*x + b

    popt, pcov = curve_fit(func, xdata, ydata)
    residuals = ydata- func(xdata, *popt)
    ss_res = np.sum(residuals**2)
    ss_tot = np.sum((ydata-np.mean(ydata))**2)
    r_squared = 1 - (ss_res / ss_tot)
    print(r_squared)

    plt.plot(xdata, func(xdata, *popt), 'r-', ms=3, label='a=%5.3f,\nb=%5.3f' % tuple(popt))
    plt.legend(fontsize=14)
                                _____no_output_____# then can plot relationships between individual proteins simply by calling the desired function

plotandoptimizelinear('NifH','NifD')/Users/noelle/miniconda3/lib/python3.6/site-packages/matplotlib/font_manager.py:1331: UserWarning: findfont: Font family ['normal'] not found. Falling back to DejaVu Sans
  (prop.get_family(), self.defaultFamily[fontext]))
/Users/noelle/miniconda3/lib/python3.6/site-packages/matplotlib/font_manager.py:1331: UserWarning: findfont: Font family ['normal'] not found. Falling back to DejaVu Sans
  (prop.get_family(), self.defaultFamily[fontext]))

# Modeled protein abundances_____no_output_____

import numpy as np
from scipy.optimize import leastsq
import pylab as plt

fig, ax = plt.subplots(figsize=(12,2.2))

N = 24 # number of data points
t = np.linspace(0, 4*np.pi, N)

# average fold change relative to mean value for all proteins in the dark blue clustr
data = [
1.045624,
1.027394,
np.NaN,
0.809967,
np.NaN,
1.232183,
np.NaN,
0.918981,
0.930507,
1.267087,
1.073016,
1.175116,
np.NaN,
0.975502,
np.NaN,
0.878217,
np.NaN,
np.NaN,
1.081296,
np.NaN,
np.NaN,
0.864874,
0.841574,
1.053778]

nonans = [x for x in data if x > 0]

guess_mean = np.mean(nonans)
guess_std = 3*np.std(nonans)/(2**0.5)/(2**0.5)
guess_phase = 0
guess_freq = 2
guess_amp = 1

# we'll use this to plot our first estimate. This might already be good enough for you
data_first_guess = guess_std*np.sin(t+guess_phase) + guess_mean

# Define the function to optimize, in this case, we want to minimize the difference
# between the actual data and our "guessed" parameters
optimize_func = lambda x: x[0]*np.sin(x[1]*t+x[2]) + x[3] - data
est_amp, est_freq, est_phase, est_mean = leastsq(optimize_func, [guess_amp, guess_freq, guess_phase, guess_mean])[0]

# recreate the fitted curve using the optimized parameters
data_fit = est_amp*np.sin(est_freq*t+est_phase) + est_mean

# recreate the fitted curve using the optimized parameters

fine_t = np.arange(0,max(t),0.1)

data_fit=est_amp*np.sin(est_freq*fine_t+est_phase)+est_mean

#plt.plot(t, data, '.')
#plt.plot(t, data_first_guess, label='first guess')
plt.plot(fine_t, data_fit, label='Nit. fix mode', c='darkblue')

N = 24 # number of data points
t = np.linspace(0, 4*np.pi, N)
# average fold change relative to mean value for all proteins in the green clustr
data = [
0.758547,
0.797299,
np.NaN,
1.297316,
np.NaN,
0.662274,
np.NaN,
1.108425,
1.118096,
0.679664,
0.790268,
0.768502,
np.NaN,
1.010798,
np.NaN,
1.185657,
np.NaN,
np.NaN,
0.772739,
np.NaN,
np.NaN,
1.230235,
1.169635,
0.835714]

nonans = [x for x in data if x > 0]

guess_mean = np.mean(nonans)
guess_std = 3*np.std(nonans)/(2**0.5)/(2**0.5)
guess_phase = 3
guess_freq = 2
guess_amp = 0.5

# we'll use this to plot our first estimate. This might already be good enough for you
data_first_guess = guess_std*np.sin(t+guess_phase) + guess_mean

# Define the function to optimize, in this case, we want to minimize the difference
# between the actual data and our "guessed" parameters
optimize_func = lambda x: x[0]*np.sin(x[1]*t+x[2]) + x[3] - data
est_amp, est_freq, est_phase, est_mean = leastsq(optimize_func, [guess_amp, guess_freq, guess_phase, guess_mean])[0]

# recreate the fitted curve using the optimized parameters
data_fit = est_amp*np.sin(est_freq*t+est_phase) + est_mean

# recreate the fitted curve using the optimized parameters
# note this assumes that the data points are evenly spread apart, which they are not

fine_t = np.arange(0,max(t),0.1)
data_fit=est_amp*np.sin(est_freq*fine_t+est_phase)+est_mean

#plt.plot(t, data, '.')
#plt.plot(t, data_first_guess, label='first guess')
plt.plot(fine_t, data_fit, label='Met. mode', c='mediumseagreen')
#plt.ylim(0.5,1.5)
#plt.xticks([])
#plt.yticks([])

#plt.legend()
plt.legend(['Active photosynthesis/Nit. Fix', 'Inactive photosynthesis/Nit. fix'], fontsize=12, loc=4)
plt.show()_____no_output_____data_fit_____no_output_____


# Notebook from xdata-skylark/libskylark
Path: python-skylark/skylark/notebooks/Sketch and Solve.ipynb

<code>
import El
from mpi4py import MPI
import skylark.io
import skylark.sketch as sketch
import numpy as np
import urllib
import sys

D = 500
T = 10000;

RFT = sketch.GaussianRFT
RFT_PARAMS = { 'sigma' : 10 }
RPT = sketch.CWT

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

data_file = '../datasets/usps.t'

# If on rank 0 read data
if rank == 0:
    print "Loading training data..."
    trnfile = skylark.io.libsvm(data_file)
    data = trnfile.read()

# Now broadcast the sizes
shape_X = data[0].shape if rank == 0 else None
shape_X = MPI.COMM_WORLD.bcast(shape_X, root=0)

# Transfer to Elemental format and redistribute the matrix and labels
if rank == 0 : print "Distributing the matrix..."
X_cc = El.DistMatrix(colDist = El.CIRC, rowDist = El.CIRC)
El.Zeros(X_cc, shape_X[0], shape_X[1])
Y_cc = El.DistMatrix(colDist = El.CIRC, rowDist = El.CIRC)
El.Zeros(Y_cc, shape_X[0], 1)
if rank == 0:
    X_ll = X_cc.Matrix()
    s = 0
    row = 0
    for e in data[0].indptr[1:]:
        for idx in range(s, e):
            col = data[0].indices[idx]
            val = data[0].data[idx]
            X_ll.Set(row, col, val)
        s = e
        row = row + 1

    Y_ll = Y_cc.Matrix()
    for j in range(shape_X[0]):
        Y_ll.Set(j, 0, data[1][j] - 1)

X = El.DistMatrix(colDist = El.VC, rowDist = El.STAR)
El.Copy(X_cc, X)
Y = El.DistMatrix(colDist = El.VC, rowDist = El.STAR)
El.Copy(Y_cc, Y);

if rank == 0 : print "Doing the regression..."

n = X.Height()
d = X.Width()

# Create right-hand side for the regression
k = int(El.Max(Y)[0] + 1)
rY = El.DistMatrix(colDist = El.VC, rowDist = El.STAR)
rY.Resize(Y.Height(), k)
El.Fill(rY, -1.0);
for i in range(rY.LocalHeight()): rY.SetLocal(i, int(Y.GetLocal(i, 0).value), 1.0)

# (in the last line we take advantage of the fact that
# the local matrices of X, Y and RY correspond to the same row.

# Apply random feature transform
R = RFT(d, D, **RFT_PARAMS)
XR = R / X    # <-------- Apply on the rows

# Reduce number of rows by sketching on the colums
S = RPT(n, T, defouttype = "RootMatrix")
SXR = S * XR
SY = S * rY

# Solve the regression

# Solve the regression: SXR and SY reside on rank zero, so solving the equation
# is done there.
if (rank == 0):
    # Solve using NumPy
    [W, res, rk, s] = np.linalg.lstsq(SXR.Matrix().ToNumPy(), SY.Matrix().ToNumPy())
else:
    W = None

# Test the solution -- we need to disribute the solution since
# we need to apply R in a distributed manner (this should change in future
# versions of Skylark)

# Note: instead we could have sketched to [STAR, STAR] and did local solves

# Distribute the solution
W = MPI.COMM_WORLD.bcast(W, root=0)

# If on rank 0 read test set from the web
if rank == 0:
    print "Loading test data..."
    tstfile = skylark.io.libsvm(data_file)
    data = tstfile.read()

# Now broadcast the sizes
shape_Xt = data[0].shape if rank == 0 else None
shape_Xt = comm.bcast(shape_Xt, root=0)

# Redistribute the matrix and labels
if rank == 0: print "Distributing the test matrix..."
Xt_cc = El.DistMatrix(colDist = El.CIRC, rowDist = El.CIRC)
El.Zeros(Xt_cc, shape_Xt[0], shape_Xt[1])
Yt_cc = El.DistMatrix(colDist = El.CIRC, rowDist = El.CIRC)
El.Zeros(Yt_cc, shape_Xt[0], 1)
if rank == 0:
    Xt_ll = Xt_cc.Matrix()
    s = 0
    row = 0
    for e in data[0].indptr[1:]:
        for idx in range(s, e):
            col = data[0].indices[idx]
            val = data[0].data[idx]
            Xt_ll.Set(row, col, val)
        s = e
        row = row + 1

    Yt_ll = Yt_cc.Matrix()
    for j in range(shape_Xt[0]):
        Yt_ll.Set(j, 0, data[1][j] - 1)

Xt = El.DistMatrix(colDist = El.VC, rowDist = El.STAR)
El.Copy(Xt_cc, Xt)
Yt = El.DistMatrix(colDist = El.VC, rowDist = El.STAR)
El.Copy(Yt_cc, Yt);

# Apply random features to Xt
if rank == 0: print "Doing the prediciton..."
XtR = R / Xt

# Elemental does not have a DistMatrix * LocalMatrix operation.
# But luckly for [VC, STAR] this operation is very simple
Yp_local = np.dot(XtR.Matrix().ToNumPy(), W).argmax(axis=1)

# Now evaluate solution
correct = np.equal(Yp_local, Yt.Matrix().ToNumPy().flatten()).sum()
correct = comm.reduce(correct)
if rank == 0:
    accuracy = (100.0 * correct) / Xt.Height()
    print "Accuracy rate is %.2f%%" % accuracy_____no_output_____
</code>


# Notebook from pckSF/pandas
Path: doc/source/user_guide/style.ipynb

# Table Visualization

This section demonstrates visualization of tabular data using the [Styler][styler]
class. For information on visualization with charting please see [Chart Visualization][viz]. This document is written as a Jupyter Notebook, and can be viewed or downloaded [here][download].

[styler]: ../reference/api/pandas.io.formats.style.Styler.rst
[viz]: visualization.rst
[download]: https://nbviewer.ipython.org/github/pandas-dev/pandas/blob/master/doc/source/user_guide/style.ipynb_____no_output_____## Styler Object and HTML

Styling should be performed after the data in a DataFrame has been processed. The [Styler][styler] creates an HTML `<table>` and leverages CSS styling language to manipulate many parameters including colors, fonts, borders, background, etc. See [here][w3schools] for more information on styling HTML tables. This allows a lot of flexibility out of the box, and even enables web developers to integrate DataFrames into their exiting user interface designs.

The `DataFrame.style` attribute is a property that returns a [Styler][styler] object. It has a `_repr_html_` method defined on it so they are rendered automatically in Jupyter Notebook.

[styler]: ../reference/api/pandas.io.formats.style.Styler.rst
[w3schools]: https://www.w3schools.com/html/html_tables.asp_____no_output_____

import matplotlib.pyplot
# We have this here to trigger matplotlib's font cache stuff.
# This cell is hidden from the output_____no_output_____import pandas as pd
import numpy as np

df = pd.DataFrame([[38.0, 2.0, 18.0, 22.0, 21, np.nan],[19, 439, 6, 452, 226,232]],
                  index=pd.Index(['Tumour (Positive)', 'Non-Tumour (Negative)'], name='Actual Label:'),
                  columns=pd.MultiIndex.from_product([['Decision Tree', 'Regression', 'Random'],['Tumour', 'Non-Tumour']], names=['Model:', 'Predicted:']))
df.style_____no_output_____

The above output looks very similar to the standard DataFrame HTML representation. But the HTML here has already attached some CSS classes to each cell, even if we haven't yet created any styles. We can view these by calling the  [.render()][render] method, which returns the raw HTML as string, which is useful for further processing or adding to a file - read on in [More about CSS and HTML](#More-About-CSS-and-HTML). Below we will show how we can use these to format the DataFrame to be more communicative. For example how we can build `s`:

[render]: ../reference/api/pandas.io.formats.style.Styler.render.rst_____no_output_____

# Hidden cell to just create the below example: code is covered throughout the guide.
s = df.style\
      .hide_columns([('Random', 'Tumour'), ('Random', 'Non-Tumour')])\
      .format('{:.0f}')\
      .set_table_styles([{
        'selector': '',
        'props':  'border-collapse: separate;'
      },{
        'selector': 'caption',
        'props': 'caption-side: bottom; font-size:1.3em;'
      },{
        'selector': '.index_name',
        'props': 'font-style: italic; color: darkgrey; font-weight:normal;'
      },{
        'selector': 'th:not(.index_name)',
        'props': 'background-color: #000066; color: white;'
      },{
        'selector': 'th.col_heading',
        'props': 'text-align: center;'
      },{
        'selector': 'th.col_heading.level0',
        'props': 'font-size: 1.5em;'
      },{
        'selector': 'th.col2',
        'props': 'border-left: 1px solid white;'
      },{
        'selector': '.col2',
        'props': 'border-left: 1px solid #000066;'
      },{
        'selector': 'td',
        'props': 'text-align: center; font-weight:bold;'
      },{
        'selector': '.true',
        'props': 'background-color: #e6ffe6;'
      },{
        'selector': '.false',
        'props': 'background-color: #ffe6e6;'
      },{
        'selector': '.border-red',
        'props': 'border: 2px dashed red;'
      },{
        'selector': '.border-green',
        'props': 'border: 2px dashed green;'
      },{
        'selector': 'td:hover',
        'props': 'background-color: #ffffb3;'
      }])\
      .set_td_classes(pd.DataFrame([['true border-green', 'false', 'true', 'false border-red', '', ''],
                                    ['false', 'true', 'false', 'true', '', '']],
                                    index=df.index, columns=df.columns))\
      .set_caption("Confusion matrix for multiple cancer prediction models.")\
      .set_tooltips(pd.DataFrame([['This model has a very strong true positive rate', '', '', "This model's total number of false negatives is too high", '', ''],
                                    ['', '', '', '', '', '']],
                                    index=df.index, columns=df.columns),
                   css_class='pd-tt', props=
    'visibility: hidden; position: absolute; z-index: 1; border: 1px solid #000066;'
    'background-color: white; color: #000066; font-size: 0.8em;'
    'transform: translate(0px, -24px); padding: 0.6em; border-radius: 0.5em;')
_____no_output_____s_____no_output_____

## Formatting the Display

### Formatting Values

Before adding styles it is useful to show that the [Styler][styler] can distinguish the *display* value from the *actual* value. To control the display value, the text is printed in each cell, and we can use the [.format()][formatfunc] method to manipulate this according to a [format spec string][format] or a callable that takes a single value and returns a string. It is possible to define this for the whole table or for individual columns.

Additionally, the format function has a **precision** argument to specifically help formatting floats, an **na_rep** argument to display missing data, and an **escape** argument to help displaying safe-HTML. The default formatter is configured to adopt pandas' regular `display.precision` option, controllable using `with pd.option_context('display.precision', 2):`

Here is an example of using the multiple options to control the formatting generally and with specific column formatters.

[styler]: ../reference/api/pandas.io.formats.style.Styler.rst
[format]: https://docs.python.org/3/library/string.html#format-specification-mini-language
[formatfunc]: ../reference/api/pandas.io.formats.style.Styler.format.rst_____no_output_____

df.style.format(precision=0, na_rep='MISSING',
                formatter={('Decision Tree', 'Tumour'): "{:.2f}",
                           ('Regression', 'Non-Tumour'): lambda x: "$ {:,.1f}".format(x*-1e3)
                          })_____no_output_____

### Hiding Data

The index can be hidden from rendering by calling [.hide_index()][hideidx], which might be useful if your index is integer based.

Columns can be hidden from rendering by calling [.hide_columns()][hidecols] and passing in the name of a column, or a slice of columns.

Hiding does not change the integer arrangement of CSS classes, e.g. hiding the first two columns of a DataFrame means the column class indexing will start at `col2`, since `col0` and `col1` are simply ignored.

We can update our `Styler` object to hide some data and format the values.

[hideidx]: ../reference/api/pandas.io.formats.style.Styler.hide_index.rst
[hidecols]: ../reference/api/pandas.io.formats.style.Styler.hide_columns.rst_____no_output_____

s = df.style.format('{:.0f}').hide_columns([('Random', 'Tumour'), ('Random', 'Non-Tumour')])
s_____no_output_____# Hidden cell to avoid CSS clashes and latter code upcoding previous formatting
s.set_uuid('after_hide')_____no_output_____

## Methods to Add Styles

There are **3 primary methods of adding custom CSS styles** to [Styler][styler]:

- Using [.set_table_styles()][table] to control broader areas of the table with specified internal CSS. Although table styles allow the flexibility to add CSS selectors and properties controlling all individual parts of the table, they are unwieldy for individual cell specifications. Also, note that table styles cannot be exported to Excel.
- Using [.set_td_classes()][td_class] to directly link either external CSS classes to your data cells or link the internal CSS classes created by [.set_table_styles()][table]. See [here](#Setting-Classes-and-Linking-to-External-CSS). These cannot be used on column header rows or indexes, and also won't export to Excel.
- Using the [.apply()][apply] and [.applymap()][applymap] functions to add direct internal CSS to specific data cells. See [here](#Styler-Functions). These cannot be used on column header rows or indexes, but only these methods add styles that will export to Excel. These methods work in a similar way to [DataFrame.apply()][dfapply] and [DataFrame.applymap()][dfapplymap].

[table]: ../reference/api/pandas.io.formats.style.Styler.set_table_styles.rst
[styler]: ../reference/api/pandas.io.formats.style.Styler.rst
[td_class]: ../reference/api/pandas.io.formats.style.Styler.set_td_classes.rst
[apply]: ../reference/api/pandas.io.formats.style.Styler.apply.rst
[applymap]: ../reference/api/pandas.io.formats.style.Styler.applymap.rst
[dfapply]: ../reference/api/pandas.DataFrame.apply.rst
[dfapplymap]: ../reference/api/pandas.DataFrame.applymap.rst_____no_output_____## Table Styles

Table styles are flexible enough to control all individual parts of the table, including column headers and indexes.
However, they can be unwieldy to type for individual data cells or for any kind of conditional formatting, so we recommend that table styles are used for broad styling, such as entire rows or columns at a time.

Table styles are also used to control features which can apply to the whole table at once such as creating a generic hover functionality. The `:hover` pseudo-selector, as well as other pseudo-selectors, can only be used this way.

To replicate the normal format of CSS selectors and properties (attribute value pairs), e.g.

```
tr:hover {
  background-color: #ffff99;
}
```

the necessary format to pass styles to [.set_table_styles()][table] is as a list of dicts, each with a CSS-selector tag and CSS-properties. Properties can either be a list of 2-tuples, or a regular CSS-string, for example:

[table]: ../reference/api/pandas.io.formats.style.Styler.set_table_styles.rst_____no_output_____

cell_hover = {  # for row hover use
 instead of
    'selector': 'td:hover',
    'props': [('background-color', '#ffffb3')]
}
index_names = {
    'selector': '.index_name',
    'props': 'font-style: italic; color: darkgrey; font-weight:normal;'
}
headers = {
    'selector': 'th:not(.index_name)',
    'props': 'background-color: #000066; color: white;'
}
s.set_table_styles([cell_hover, index_names, headers])_____no_output_____# Hidden cell to avoid CSS clashes and latter code upcoding previous formatting
s.set_uuid('after_tab_styles1')_____no_output_____

Next we just add a couple more styling artifacts targeting specific parts of the table. Be careful here, since we are *chaining methods* we need to explicitly instruct the method **not to** ``overwrite`` the existing styles._____no_output_____

s.set_table_styles([
    {'selector': 'th.col_heading', 'props': 'text-align: center;'},
    {'selector': 'th.col_heading.level0', 'props': 'font-size: 1.5em;'},
    {'selector': 'td', 'props': 'text-align: center; font-weight: bold;'},
], overwrite=False)_____no_output_____# Hidden cell to avoid CSS clashes and latter code upcoding previous formatting
s.set_uuid('after_tab_styles2')_____no_output_____

As a convenience method (*since version 1.2.0*) we can also pass a **dict** to [.set_table_styles()][table] which contains row or column keys. Behind the scenes Styler just indexes the keys and adds relevant `.col<m>` or `.row<n>` classes as necessary to the given CSS selectors.

[table]: ../reference/api/pandas.io.formats.style.Styler.set_table_styles.rst_____no_output_____

s.set_table_styles({
    ('Regression', 'Tumour'): [{'selector': 'th', 'props': 'border-left: 1px solid white'},
                               {'selector': 'td', 'props': 'border-left: 1px solid #000066'}]
}, overwrite=False, axis=0)_____no_output_____# Hidden cell to avoid CSS clashes and latter code upcoding previous formatting
s.set_uuid('xyz01')_____no_output_____

## Setting Classes and Linking to External CSS

If you have designed a website then it is likely you will already have an external CSS file that controls the styling of table and cell objects within it. You may want to use these native files rather than duplicate all the CSS in python (and duplicate any maintenance work).

### Table Attributes

It is very easy to add a `class` to the main `<table>` using [.set_table_attributes()][tableatt]. This method can also attach inline styles - read more in [CSS Hierarchies](#CSS-Hierarchies).

[tableatt]: ../reference/api/pandas.io.formats.style.Styler.set_table_attributes.rst_____no_output_____

out = s.set_table_attributes('class="my-table-cls"').render()
print(out[out.find('<table'):][:109])_____no_output_____

### Data Cell CSS Classes

*New in version 1.2.0*

The [.set_td_classes()][tdclass] method accepts a DataFrame with matching indices and columns to the underlying [Styler][styler]'s DataFrame. That DataFrame will contain strings as css-classes to add to individual data cells: the `<td>` elements of the `<table>`. Rather than use external CSS we will create our classes internally and add them to table style. We will save adding the borders until the [section on tooltips](#Tooltips).

[tdclass]: ../reference/api/pandas.io.formats.style.Styler.set_td_classes.rst
[styler]: ../reference/api/pandas.io.formats.style.Styler.rst_____no_output_____

s.set_table_styles([  # create internal CSS classes
    {'selector': '.true', 'props': 'background-color: #e6ffe6;'},
    {'selector': '.false', 'props': 'background-color: #ffe6e6;'},
], overwrite=False)
cell_color = pd.DataFrame([['true ', 'false ', 'true ', 'false '],
                           ['false ', 'true ', 'false ', 'true ']],
                          index=df.index,
                          columns=df.columns[:4])
s.set_td_classes(cell_color)_____no_output_____# Hidden cell to avoid CSS clashes and latter code upcoding previous formatting
s.set_uuid('after_classes')_____no_output_____

## Styler Functions

We use the following methods to pass your style functions. Both of those methods take a function (and some other keyword arguments) and apply it to the DataFrame in a certain way, rendering CSS styles.

- [.applymap()][applymap] (elementwise): accepts a function that takes a single value and returns a string with the CSS attribute-value pair.
- [.apply()][apply] (column-/row-/table-wise): accepts a function that takes a Series or DataFrame and returns a Series, DataFrame, or numpy array with an identical shape where each element is a string with a CSS attribute-value pair. This method passes each column or row of your DataFrame one-at-a-time or the entire table at once, depending on the `axis` keyword argument. For columnwise use `axis=0`, rowwise use `axis=1`, and for the entire table at once use `axis=None`.

This method is powerful for applying multiple, complex logic to data cells. We create a new DataFrame to demonstrate this.

[apply]: ../reference/api/pandas.io.formats.style.Styler.apply.rst
[applymap]: ../reference/api/pandas.io.formats.style.Styler.applymap.rst_____no_output_____

np.random.seed(0)
df2 = pd.DataFrame(np.random.randn(10,4), columns=['A','B','C','D'])
df2.style_____no_output_____

For example we can build a function that colors text if it is negative, and chain this with a function that partially fades cells of negligible value. Since this looks at each element in turn we use ``applymap``._____no_output_____

def style_negative(v, props=''):
    return props if v < 0 else None
s2 = df2.style.applymap(style_negative, props='color:red;')\
              .applymap(lambda v: 'opacity: 20%;' if (v < 0.3) and (v > -0.3) else None)
s2_____no_output_____# Hidden cell to avoid CSS clashes and latter code upcoding previous formatting
s2.set_uuid('after_applymap')_____no_output_____

We can also build a function that highlights the maximum value across rows, cols, and the DataFrame all at once. In this case we use ``apply``. Below we highlight the maximum in a column._____no_output_____

def highlight_max(s, props=''):
    return np.where(s == np.nanmax(s.values), props, '')
s2.apply(highlight_max, props='color:white;background-color:darkblue', axis=0)_____no_output_____# Hidden cell to avoid CSS clashes and latter code upcoding previous formatting
s2.set_uuid('after_apply')_____no_output_____

We can use the same function across the different axes, highlighting here the DataFrame maximum in purple, and row maximums in pink._____no_output_____

s2.apply(highlight_max, props='color:white;background-color:pink;', axis=1)\
  .apply(highlight_max, props='color:white;background-color:purple', axis=None)_____no_output_____

This last example shows how some styles have been overwritten by others. In general the most recent style applied is active but you can read more in the [section on CSS hierarchies](#CSS-Hierarchies). You can also apply these styles to more granular parts of the DataFrame - read more in section on [subset slicing](#Finer-Control-with-Slicing).

It is possible to replicate some of this functionality using just classes but it can be more cumbersome. See [item 3)  of Optimization](#Optimization)

*Debugging Tip*: If you're having trouble writing your style function, try just passing it into ``DataFrame.apply``. Internally, ``Styler.apply`` uses ``DataFrame.apply`` so the result should be the same, and with ``DataFrame.apply`` you will be able to inspect the CSS string output of your intended function in each cell.

_____no_output_____## Tooltips and Captions

Table captions can be added with the [.set_caption()][caption] method. You can use table styles to control the CSS relevant to the caption.

[caption]: ../reference/api/pandas.io.formats.style.Styler.set_caption.rst_____no_output_____

s.set_caption("Confusion matrix for multiple cancer prediction models.")\
 .set_table_styles([{
     'selector': 'caption',
     'props': 'caption-side: bottom; font-size:1.25em;'
 }], overwrite=False)_____no_output_____# Hidden cell to avoid CSS clashes and latter code upcoding previous formatting
s.set_uuid('after_caption')_____no_output_____

Adding tooltips (*since version 1.3.0*) can be done using the [.set_tooltips()][tooltips] method in the same way you can add CSS classes to data cells by providing a string based DataFrame with intersecting indices and columns. You don't have to specify a `css_class` name or any css `props` for the tooltips, since there are standard defaults, but the option is there if you want more visual control.

[tooltips]: ../reference/api/pandas.io.formats.style.Styler.set_tooltips.rst_____no_output_____

tt = pd.DataFrame([['This model has a very strong true positive rate',
                    "This model's total number of false negatives is too high"]],
                  index=['Tumour (Positive)'], columns=df.columns[[0,3]])
s.set_tooltips(tt, props='visibility: hidden; position: absolute; z-index: 1; border: 1px solid #000066;'
                         'background-color: white; color: #000066; font-size: 0.8em;'
                         'transform: translate(0px, -24px); padding: 0.6em; border-radius: 0.5em;')_____no_output_____# Hidden cell to avoid CSS clashes and latter code upcoding previous formatting
s.set_uuid('after_tooltips')_____no_output_____

The only thing left to do for our table is to add the highlighting borders to draw the audience attention to the tooltips. We will create internal CSS classes as before using table styles. **Setting classes always overwrites** so we need to make sure we add the previous classes._____no_output_____

s.set_table_styles([  # create internal CSS classes
    {'selector': '.border-red', 'props': 'border: 2px dashed red;'},
    {'selector': '.border-green', 'props': 'border: 2px dashed green;'},
], overwrite=False)
cell_border = pd.DataFrame([['border-green ', ' ', ' ', 'border-red '],
                           [' ', ' ', ' ', ' ']],
                          index=df.index,
                          columns=df.columns[:4])
s.set_td_classes(cell_color + cell_border)_____no_output_____# Hidden cell to avoid CSS clashes and latter code upcoding previous formatting
s.set_uuid('after_borders')_____no_output_____

## Finer Control with Slicing

The examples we have shown so far for the `Styler.apply` and `Styler.applymap` functions have not demonstrated the use of the ``subset`` argument. This is a useful argument which permits a lot of flexibility: it allows you to apply styles to specific rows or columns, without having to code that logic into your `style` function.

The value passed to `subset` behaves similar to slicing a DataFrame;

- A scalar is treated as a column label
- A list (or Series or NumPy array) is treated as multiple column labels
- A tuple is treated as `(row_indexer, column_indexer)`

Consider using `pd.IndexSlice` to construct the tuple for the last one. We will create a MultiIndexed DataFrame to demonstrate the functionality._____no_output_____

df3 = pd.DataFrame(np.random.randn(4,4),
                   pd.MultiIndex.from_product([['A', 'B'], ['r1', 'r2']]),
                   columns=['c1','c2','c3','c4'])
df3_____no_output_____

We will use subset to highlight the maximum in the third and fourth columns with red text. We will highlight the subset sliced region in yellow._____no_output_____

slice_ = ['c3', 'c4']
df3.style.apply(highlight_max, props='color:red;', axis=0, subset=slice_)\
         .set_properties(**{'background-color': '#ffffb3'}, subset=slice_)_____no_output_____

If combined with the ``IndexSlice`` as suggested then it can index across both dimensions with greater flexibility._____no_output_____

idx = pd.IndexSlice
slice_ = idx[idx[:,'r1'], idx['c2':'c4']]
df3.style.apply(highlight_max, props='color:red;', axis=0, subset=slice_)\
         .set_properties(**{'background-color': '#ffffb3'}, subset=slice_)_____no_output_____

This also provides the flexibility to sub select rows when used with the `axis=1`._____no_output_____

slice_ = idx[idx[:,'r2'], :]
df3.style.apply(highlight_max, props='color:red;', axis=1, subset=slice_)\
         .set_properties(**{'background-color': '#ffffb3'}, subset=slice_)_____no_output_____

There is also scope to provide **conditional filtering**.

Suppose we want to highlight the maximum across columns 2 and 4 only in the case that the sum of columns 1 and 3 is less than -2.0 *(essentially excluding rows* `(:,'r2')`*)*._____no_output_____

slice_ = idx[idx[(df3['c1'] + df3['c3']) < -2.0], ['c2', 'c4']]
df3.style.apply(highlight_max, props='color:red;', axis=1, subset=slice_)\
         .set_properties(**{'background-color': '#ffffb3'}, subset=slice_)_____no_output_____

Only label-based slicing is supported right now, not positional, and not callables.

If your style function uses a `subset` or `axis` keyword argument, consider wrapping your function in a `functools.partial`, partialing out that keyword.

```python
my_func2 = functools.partial(my_func, subset=42)
```_____no_output_____## Optimization

Generally, for smaller tables and most cases, the rendered HTML does not need to be optimized, and we don't really recommend it. There are two cases where it is worth considering:

 - If you are rendering and styling a very large HTML table, certain browsers have performance issues.
 - If you are using ``Styler`` to dynamically create part of online user interfaces and want to improve network performance.

Here we recommend the following steps to implement:_____no_output_____### 1. Remove UUID and cell_ids

Ignore the `uuid` and set `cell_ids` to `False`. This will prevent unnecessary HTML._____no_output_____

This is sub-optimal:

_____no_output_____

df4 = pd.DataFrame([[1,2],[3,4]])
s4 = df4.style_____no_output_____

This is better:

_____no_output_____

from pandas.io.formats.style import Styler
s4 = Styler(df4, uuid_len=0, cell_ids=False)_____no_output_____

### 2. Use table styles

Use table styles where possible (e.g. for all cells or rows or columns at a time) since the CSS is nearly always more efficient than other formats._____no_output_____

This is sub-optimal:

_____no_output_____

props = 'font-family: "Times New Roman", Times, serif; color: #e83e8c; font-size:1.3em;'
df4.style.applymap(lambda x: props, subset=[1])_____no_output_____

This is better:

_____no_output_____

df4.style.set_table_styles([{'selector': 'td.col1', 'props': props}])_____no_output_____

### 3. Set classes instead of using Styler functions

For large DataFrames where the same style is applied to many cells it can be more efficient to declare the styles as classes and then apply those classes to data cells, rather than directly applying styles to cells. It is, however, probably still easier to use the Styler function api when you are not concerned about optimization._____no_output_____

This is sub-optimal:

_____no_output_____

df2.style.apply(highlight_max, props='color:white;background-color:darkblue;', axis=0)\
         .apply(highlight_max, props='color:white;background-color:pink;', axis=1)\
         .apply(highlight_max, props='color:white;background-color:purple', axis=None)_____no_output_____

This is better:

_____no_output_____

build = lambda x: pd.DataFrame(x, index=df2.index, columns=df2.columns)
cls1 = build(df2.apply(highlight_max, props='cls-1 ', axis=0))
cls2 = build(df2.apply(highlight_max, props='cls-2 ', axis=1, result_type='expand').values)
cls3 = build(highlight_max(df2, props='cls-3 '))
df2.style.set_table_styles([
    {'selector': '.cls-1', 'props': 'color:white;background-color:darkblue;'},
    {'selector': '.cls-2', 'props': 'color:white;background-color:pink;'},
    {'selector': '.cls-3', 'props': 'color:white;background-color:purple;'}
]).set_td_classes(cls1 + cls2 + cls3)_____no_output_____

### 4. Don't use tooltips

Tooltips require `cell_ids` to work and they generate extra HTML elements for *every* data cell._____no_output_____### 5. If every byte counts use string replacement

You can remove unnecessary HTML, or shorten the default class names with string replace functions._____no_output_____

html = Styler(df4, uuid_len=0, cell_ids=False)\
      .set_table_styles([{'selector': 'td', 'props': props},
                         {'selector': '.col1', 'props': 'color:green;'},
                         {'selector': '.level0', 'props': 'color:blue;'}])\
      .render()\
      .replace('blank', '')\
      .replace('data', '')\
      .replace('level0', 'l0')\
      .replace('col_heading', '')\
      .replace('row_heading', '')

import re
html = re.sub(r'col[0-9]+', lambda x: x.group().replace('col', 'c'), html)
html = re.sub(r'row[0-9]+', lambda x: x.group().replace('row', 'r'), html)
print(html)_____no_output_____from IPython.display import HTML
HTML(html)_____no_output_____

## Builtin Styles_____no_output_____Some styling functions are common enough that we've "built them in" to the `Styler`, so you don't have to write them and apply them yourself. The current list of such functions is:

 - [.highlight_null][nullfunc]: for use with identifying missing data.
 - [.highlight_min][minfunc] and [.highlight_max][maxfunc]: for use with identifying extremeties in data.
 - [.highlight_between][betweenfunc] and [.highlight_quantile][quantilefunc]: for use with identifying classes within data.
 - [.background_gradient][bgfunc]: a flexible method for highlighting cells based or their, or other, values on a numeric scale.
 -  [.bar][barfunc]: to display mini-charts within cell backgrounds.

The individual documentation on each function often gives more examples of their arguments.

[nullfunc]: ../reference/api/pandas.io.formats.style.Styler.highlight_null.rst
[minfunc]: ../reference/api/pandas.io.formats.style.Styler.highlight_min.rst
[maxfunc]: ../reference/api/pandas.io.formats.style.Styler.highlight_max.rst
[betweenfunc]: ../reference/api/pandas.io.formats.style.Styler.highlight_between.rst
[quantilefunc]: ../reference/api/pandas.io.formats.style.Styler.highlight_quantile.rst
[bgfunc]: ../reference/api/pandas.io.formats.style.Styler.background_gradient.rst
[barfunc]: ../reference/api/pandas.io.formats.style.Styler.bar.rst_____no_output_____### Highlight Null_____no_output_____

df2.iloc[0,2] = np.nan
df2.iloc[4,3] = np.nan
df2.loc[:4].style.highlight_null(null_color='yellow')_____no_output_____

### Highlight Min or Max_____no_output_____

df2.loc[:4].style.highlight_max(axis=1, props='color:white; font-weight:bold; background-color:darkblue;')_____no_output_____

### Highlight Between
This method accepts ranges as float, or NumPy arrays or Series provided the indexes match._____no_output_____

left = pd.Series([1.0, 0.0, 1.0], index=["A", "B", "D"])
df2.loc[:4].style.highlight_between(left=left, right=1.5, axis=1, props='color:white; background-color:purple;')_____no_output_____

### Highlight Quantile
Useful for detecting the highest or lowest percentile values_____no_output_____

df2.loc[:4].style.highlight_quantile(q_left=0.85, axis=None, color='yellow')_____no_output_____

### Background Gradient_____no_output_____You can create "heatmaps" with the `background_gradient` method. These require matplotlib, and we'll use [Seaborn](https://stanford.edu/~mwaskom/software/seaborn/) to get a nice colormap._____no_output_____

import seaborn as sns
cm = sns.light_palette("green", as_cmap=True)

df2.style.background_gradient(cmap=cm)_____no_output_____

[.background_gradient][bgfunc] has a number of keyword arguments to customise the gradients and colors. See its documentation.

[bgfunc]: ../reference/api/pandas.io.formats.style.Styler.background_gradient.rst_____no_output_____Use `Styler.set_properties` when the style doesn't actually depend on the values. This is just a simple wrapper for `.applymap` where the function returns the same properties for all cells._____no_output_____

df2.loc[:4].style.set_properties(**{'background-color': 'black',
                           'color': 'lawngreen',
                           'border-color': 'white'})_____no_output_____

### Bar charts_____no_output_____You can include "bar charts" in your DataFrame._____no_output_____

df2.style.bar(subset=['A', 'B'], color='#d65f5f')_____no_output_____

In version 0.20.0 the ability to customize the bar chart further was given. You can now have the `df.style.bar` be centered on zero or midpoint value (in addition to the already existing way of having the min value at the left side of the cell), and you can pass a list of `[color_negative, color_positive]`.

Here's how you can change the above with the new `align='mid'` option:_____no_output_____

df2.style.bar(subset=['A', 'B'], align='mid', color=['#d65f5f', '#5fba7d'])_____no_output_____

The following example aims to give a highlight of the behavior of the new align options:_____no_output_____

# Hide the construction of the display chart from the user
import pandas as pd
from IPython.display import HTML

# Test series
test1 = pd.Series([-100,-60,-30,-20], name='All Negative')
test2 = pd.Series([10,20,50,100], name='All Positive')
test3 = pd.Series([-10,-5,0,90], name='Both Pos and Neg')

head = """

        Align
        All Negative
        All Positive
        Both Neg and Pos

"""

aligns = ['left','zero','mid']
for align in aligns:
    row = "
{}	".format(align)
    for series in [test1,test2,test3]:
        s = series.copy()
        s.name=''
        row += "{}	".format(s.to_frame().style.bar(align=align,
                                                           color=['#d65f5f', '#5fba7d'],
                                                           width=100).render()) #testn['width']
    row += '
'
    head += row

head+= """

"""_____no_output_____HTML(head)_____no_output_____

## Sharing styles_____no_output_____Say you have a lovely style built up for a DataFrame, and now you want to apply the same style to a second DataFrame. Export the style with `df1.style.export`, and import it on the second DataFrame with `df1.style.set`_____no_output_____

style1 = df2.style.applymap(style_negative, props='color:red;')\
         .applymap(lambda v: 'opacity: 20%;' if (v < 0.3) and (v > -0.3) else None)_____no_output_____style2 = df3.style
style2.use(style1.export())
style2_____no_output_____

Notice that you're able to share the styles even though they're data aware. The styles are re-evaluated on the new DataFrame they've been `use`d upon._____no_output_____## Limitations

- DataFrame only `(use Series.to_frame().style)`
- The index and columns must be unique
- No large repr, and construction performance isn't great; although we have some [HTML optimizations](#Optimization)
- You can only style the *values*, not the index or columns (except with `table_styles` above)
- You can only apply styles, you can't insert new HTML entities

Some of these might be addressed in the future. _____no_output_____## Other Fun and Useful Stuff

Here are a few interesting examples._____no_output_____### Widgets

`Styler` interacts pretty well with widgets. If you're viewing this online instead of running the notebook yourself, you're missing out on interactively adjusting the color palette._____no_output_____

from ipywidgets import widgets
@widgets.interact
def f(h_neg=(0, 359, 1), h_pos=(0, 359), s=(0., 99.9), l=(0., 99.9)):
    return df2.style.background_gradient(
        cmap=sns.palettes.diverging_palette(h_neg=h_neg, h_pos=h_pos, s=s, l=l,
                                            as_cmap=True)
    )_____no_output_____

### Magnify_____no_output_____

def magnify():
    return [dict(selector="th",
                 props=[("font-size", "4pt")]),
            dict(selector="td",
                 props=[('padding', "0em 0em")]),
            dict(selector="th:hover",
                 props=[("font-size", "12pt")]),
            dict(selector="tr:hover td:hover",
                 props=[('max-width', '200px'),
                        ('font-size', '12pt')])
]_____no_output_____np.random.seed(25)
cmap = cmap=sns.diverging_palette(5, 250, as_cmap=True)
bigdf = pd.DataFrame(np.random.randn(20, 25)).cumsum()

bigdf.style.background_gradient(cmap, axis=1)\
    .set_properties(**{'max-width': '80px', 'font-size': '1pt'})\
    .set_caption("Hover to magnify")\
    .format(precision=2)\
    .set_table_styles(magnify())_____no_output_____

### Sticky Headers

If you display a large matrix or DataFrame in a notebook, but you want to always see the column and row headers you can use the following CSS to make them stick. We might make this into an API function later._____no_output_____

bigdf = pd.DataFrame(np.random.randn(15, 100))
bigdf.style.set_table_styles([
    {'selector': 'thead th', 'props': 'position: sticky; top:0; background-color:salmon;'},
    {'selector': 'tbody th', 'props': 'position: sticky; left:0; background-color:lightgreen;'}
])_____no_output_____

### Hiding Headers

We don't yet have any API to hide headers so a quick fix is:_____no_output_____

df3.style.set_table_styles([{'selector': 'thead tr', 'props': 'display: none;'}])  # or 'thead th'_____no_output_____

### HTML Escaping

Suppose you have to display HTML within HTML, that can be a bit of pain when the renderer can't distinguish. You can use the `escape` formatting option to handle this, and even use it within a formatter that contains HTML itself._____no_output_____

df4 = pd.DataFrame([['

', '"&other"', '']])
df4.style_____no_output_____df4.style.format(escape=True)_____no_output_____df4.style.format('{}', escape=True)_____no_output_____

## Export to Excel

Some support (*since version 0.20.0*) is available for exporting styled `DataFrames` to Excel worksheets using the `OpenPyXL` or `XlsxWriter` engines. CSS2.2 properties handled include:

- `background-color`
- `color`
- `font-family`
- `font-style`
- `font-weight`
- `text-align`
- `text-decoration`
- `vertical-align`
- `white-space: nowrap`

- Currently broken: `border-style`, `border-width`, `border-color` and their {`top`, `right`, `bottom`, `left` variants}

- Only CSS2 named colors and hex colors of the form `#rgb` or `#rrggbb` are currently supported.
- The following pseudo CSS properties are also available to set excel specific style properties:
    - `number-format`

Table level styles, and data cell CSS-classes are not included in the export to Excel: individual cells must have their properties mapped by the `Styler.apply` and/or `Styler.applymap` methods._____no_output_____

df2.style.\
    applymap(style_negative, props='color:red;').\
    highlight_max(axis=0).\
    to_excel('styled.xlsx', engine='openpyxl')_____no_output_____

A screenshot of the output:

![Excel spreadsheet with styled DataFrame](../_static/style-excel.png)
_____no_output_____## More About CSS and HTML

Cascading Style Sheet (CSS) language, which is designed to influence how a browser renders HTML elements, has its own peculiarities. It never reports errors: it just silently ignores them and doesn't render your objects how you intend so can sometimes be frustrating. Here is a very brief primer on how ``Styler`` creates HTML and interacts with CSS, with advice on common pitfalls to avoid._____no_output_____### CSS Classes and Ids

The precise structure of the CSS `class` attached to each cell is as follows.

- Cells with Index and Column names include `index_name` and `level<k>` where `k` is its level in a MultiIndex
- Index label cells include
  + `row_heading`
  + `level<k>` where `k` is the level in a MultiIndex
  + `row<m>` where `m` is the numeric position of the row
- Column label cells include
  + `col_heading`
  + `level<k>` where `k` is the level in a MultiIndex
  + `col<n>` where `n` is the numeric position of the column
- Data cells include
  + `data`
  + `row<m>`, where `m` is the numeric position of the cell.
  + `col<n>`, where `n` is the numeric position of the cell.
- Blank cells include `blank`

The structure of the `id` is `T_uuid_level<k>_row<m>_col<n>` where `level<k>` is used only on headings, and headings will only have either `row<m>` or `col<n>` whichever is needed. By default we've also prepended each row/column identifier with a UUID unique to each DataFrame so that the style from one doesn't collide with the styling from another within the same notebook or page. You can read more about the use of UUIDs in [Optimization](#Optimization).

We can see example of the HTML by calling the [.render()][render] method.

[render]: ../reference/api/pandas.io.formats.style.Styler.render.rst_____no_output_____

print(pd.DataFrame([[1,2],[3,4]], index=['i1', 'i2'], columns=['c1', 'c2']).style.render())_____no_output_____

### CSS Hierarchies

The examples have shown that when CSS styles overlap, the one that comes last in the HTML render, takes precedence. So the following yield different results:_____no_output_____

df4 = pd.DataFrame([['text']])
df4.style.applymap(lambda x: 'color:green;')\
         .applymap(lambda x: 'color:red;')_____no_output_____df4.style.applymap(lambda x: 'color:red;')\
         .applymap(lambda x: 'color:green;')_____no_output_____

This is only true for CSS rules that are equivalent in hierarchy, or importance. You can read more about [CSS specificity here](https://www.w3schools.com/css/css_specificity.asp) but for our purposes it suffices to summarize the key points:

A CSS importance score for each HTML element is derived by starting at zero and adding:

 - 1000 for an inline style attribute
 - 100 for each ID
 - 10 for each attribute, class or pseudo-class
 - 1 for each element name or pseudo-element

Let's use this to describe the action of the following configurations_____no_output_____

df4.style.set_uuid('a_')\
         .set_table_styles([{'selector': 'td', 'props': 'color:red;'}])\
         .applymap(lambda x: 'color:green;')_____no_output_____

This text is red because the generated selector `#T_a_ td` is worth 101 (ID plus element), whereas `#T_a_row0_col0` is only worth 100 (ID), so is considered inferior even though in the HTML it comes after the previous._____no_output_____

df4.style.set_uuid('b_')\
         .set_table_styles([{'selector': 'td', 'props': 'color:red;'},
                            {'selector': '.cls-1', 'props': 'color:blue;'}])\
         .applymap(lambda x: 'color:green;')\
         .set_td_classes(pd.DataFrame([['cls-1']]))_____no_output_____

In the above case the text is blue because the selector `#T_b_ .cls-1` is worth 110 (ID plus class), which takes precendence._____no_output_____

df4.style.set_uuid('c_')\
         .set_table_styles([{'selector': 'td', 'props': 'color:red;'},
                            {'selector': '.cls-1', 'props': 'color:blue;'},
                            {'selector': 'td.data', 'props': 'color:yellow;'}])\
         .applymap(lambda x: 'color:green;')\
         .set_td_classes(pd.DataFrame([['cls-1']]))_____no_output_____

Now we have created another table style this time the selector `T_c_ td.data` (ID plus element plus class) gets bumped up to 111.

If your style fails to be applied, and its really frustrating, try the `!important` trump card._____no_output_____

df4.style.set_uuid('d_')\
         .set_table_styles([{'selector': 'td', 'props': 'color:red;'},
                            {'selector': '.cls-1', 'props': 'color:blue;'},
                            {'selector': 'td.data', 'props': 'color:yellow;'}])\
         .applymap(lambda x: 'color:green !important;')\
         .set_td_classes(pd.DataFrame([['cls-1']]))_____no_output_____

Finally got that green text after all!_____no_output_____## Extensibility

The core of pandas is, and will remain, its "high-performance, easy-to-use data structures".
With that in mind, we hope that `DataFrame.style` accomplishes two goals

- Provide an API that is pleasing to use interactively and is "good enough" for many tasks
- Provide the foundations for dedicated libraries to build on

If you build a great library on top of this, let us know and we'll [link](https://pandas.pydata.org/pandas-docs/stable/ecosystem.html) to it.

### Subclassing

If the default template doesn't quite suit your needs, you can subclass Styler and extend or override the template.
We'll show an example of extending the default template to insert a custom header before each table._____no_output_____

from jinja2 import Environment, ChoiceLoader, FileSystemLoader
from IPython.display import HTML
from pandas.io.formats.style import Styler_____no_output_____

We'll use the following template:_____no_output_____

with open("templates/myhtml.tpl") as f:
    print(f.read())_____no_output_____

Now that we've created a template, we need to set up a subclass of ``Styler`` that
knows about it._____no_output_____

class MyStyler(Styler):
    env = Environment(
        loader=ChoiceLoader([
            FileSystemLoader("templates"),  # contains ours
            Styler.loader,  # the default
        ])
    )
    template_html = env.get_template("myhtml.tpl")_____no_output_____

Notice that we include the original loader in our environment's loader.
That's because we extend the original template, so the Jinja environment needs
to be able to find it.

Now we can use that custom styler. It's `__init__` takes a DataFrame._____no_output_____

MyStyler(df3)_____no_output_____

Our custom template accepts a `table_title` keyword. We can provide the value in the `.render` method._____no_output_____

HTML(MyStyler(df3).render(table_title="Extending Example"))_____no_output_____

For convenience, we provide the `Styler.from_custom_template` method that does the same as the custom subclass._____no_output_____

EasyStyler = Styler.from_custom_template("templates", "myhtml.tpl")
EasyStyler(df3)_____no_output_____

Here's the template structure:_____no_output_____

with open("templates/template_structure.html") as f:
    structure = f.read()

HTML(structure)_____no_output_____

See the template in the [GitHub repo](https://github.com/pandas-dev/pandas) for more details._____no_output_____

# # Hack to get the same style in the notebook as the
# # main site. This is hidden in the docs.
# from IPython.display import HTML
# with open("themes/nature_with_gtoc/static/nature.css_t") as f:
#     css = f.read()

# HTML(''.format(css))_____no_output_____