# Notebook from ChrisKeefe/Network-Analysis-Made-Simple
Path: notebooks/04-advanced/03-stats.ipynb

%load_ext autoreload
%autoreload 2
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import warnings
warnings.filterwarnings('ignore')_____no_output_____

## Introduction_____no_output_____

from IPython.display import YouTubeVideo

YouTubeVideo(id="P-0CJpO3spg", width="100%")_____no_output_____

In this chapter, we are going to take a look at how to perform statistical inference on graphs._____no_output_____## Statistics refresher

Before we can proceed with statistical inference on graphs,
we must first refresh ourselves with some ideas from the world of statistics.
Otherwise, the methods that we will end up using
may seem a tad _weird_, and hence difficult to follow along.

To review statistical ideas,
let's set up a few statements and explore what they mean._____no_output_____## We are concerned with models of randomness

As with all things statistics, we are concerned with models of randomness.
Here, probability distributions give us a way to think about random events
and how to assign credibility points to them._____no_output_____### In an abstract fashion...

The supremely abstract way of thinking about a probability distribution
is that it is the space of all possibilities of "stuff"
with different credibility points _distributed_ amongst each possible "thing"._____no_output_____### More concretely: the coin flip

A more concrete example is to consider the coin flip.
Here, the space of all possibilities of "stuff" is the set of "heads" and "tails".
If we have a fair coin, then we have 0.5 credibility points _distributed_
to each of "heads" and "tails"._____no_output_____### Another example: dice rolls

Another concrete example is to consider the six-sided dice.
Here, the space of all possibilities of "stuff" is the set of numbers in the range $[1, 6]$.
If we have a fair dice, then we have 1/6 credibility points assigned
to each of the numbers.
(Unfair dice will have an unequal _distribution_ of credibility points across each face.)_____no_output_____### A graph-based example: social networks

If we receive an undirected social network graph with 5 nodes and 6 edges,
we have to keep in mind that this graph with 6 edges
was merely one of $15 \choose 6$ ways to construct 5 node, 6 edge graphs.
(15 comes up because there are 15 edges that can be constructed in a 5-node undirected graph.)_____no_output_____## Hypothesis Testing

A commonplace task in statistical inferences
is calculating the probability of observing a value or something more extreme
under an assumed "null" model of reality.
This is what we commonly call "hypothesis testing",
and where the oft-misunderstood term "p-value" shows up._____no_output_____### Hypothesis testing in coin flips, by simulation

As an example, hypothesis testing in coin flips follows this logic:

- I observe that 8 out of 10 coin tosses give me heads, giving me a probability of heads $p=0.8$ (a summary statistic).
- Under a "null distribution" of a fair coin, I simulate the distribution of probability of heads (the summary statistic) that I would get from 10 coin tosses.
- Finally, I use that distribution to calculate the probability of observing $p=0.8$ or more extreme._____no_output_____### Hypothesis testing in graphs

The same protocol applies when we perform hypothesis testing on graphs.

Firstly, we calculate a _summary statistic_ that describes our graph.

Secondly, we propose a _null graph model_, and calculate our summary statistic under simulated versions of that null graph model.

Thirdly, we look at the probability of observing the summary statistic value that we calculated in step 1 or more extreme, under the assumed graph null model distribution._____no_output_____## Stochastic graph creation models

Since we are going to be dealing with models of randomness in graphs,
let's take a look at some examples._____no_output_____### Erdos-Renyi (a.k.a. "binomial") graph

On easy one to study is the Erdos-Renyi graph, also known as the "binomial" graph.

The data generation story here is that we instantiate an undirected graph with $n$ nodes,
giving $\frac{n^2 - n}{2}$ possible edges.
Each edge has a probability $p$ of being created._____no_output_____

import networkx as nx

G_er = nx.erdos_renyi_graph(n=30, p=0.2)_____no_output_____nx.draw(G_er)_____no_output_____

You can verify that there's approximately 20% of $\frac{30^2 - 30}{2} = 435$._____no_output_____

len(G_er.edges())_____no_output_____len(G_er.edges()) / 435_____no_output_____

We can also look at the degree distribution:_____no_output_____

import pandas as pd
from nams.functions import ecdf
import matplotlib.pyplot as plt

x, y = ecdf(pd.Series(dict(nx.degree(G_er))))
plt.scatter(x, y)_____no_output_____

### Barabasi-Albert Graph

The data generating story of this graph generator is essentially that nodes that have lots of edges preferentially get new edges attached onto them.
This is what we call a "preferential attachment" process._____no_output_____

G_ba = nx.barabasi_albert_graph(n=30, m=3)
nx.draw(G_ba)_____no_output_____len(G_ba.edges())_____no_output_____

And the degree distribution:_____no_output_____

x, y = ecdf(pd.Series(dict(nx.degree(G_ba))))
plt.scatter(x, y)_____no_output_____

You can see that even though the number of edges between the two graphs are similar,
their degree distribution is wildly different._____no_output_____## Load Data

For this notebook, we are going to look at a protein-protein interaction network,
and test the hypothesis that this network was _not_ generated by the data generating process
described by an Erdos-Renyi graph.

Let's load a [protein-protein interaction network dataset](http://konect.uni-koblenz.de/networks/moreno_propro).

> This undirected network contains protein interactions contained in yeast.
> Research showed that proteins with a high degree
> were more important for the surivial of the yeast than others.
> A node represents a protein and an edge represents a metabolic interaction between two proteins.
> The network contains loops._____no_output_____

from nams import load_data as cf

G = cf.load_propro_network()
for n, d in G.nodes(data=True):
    G.nodes[n]["degree"] = G.degree(n)_____no_output_____

As is always the case, let's make sure we know some basic stats of the graph._____no_output_____

len(G.nodes())_____no_output_____len(G.edges())_____no_output_____

Let's also examine the degree distribution of the graph._____no_output_____

x, y = ecdf(pd.Series(dict(nx.degree(G))))
plt.scatter(x, y)_____no_output_____

Finally, we should visualize the graph to get a feel for it._____no_output_____

import nxviz as nv
from nxviz import annotate

nv.circos(G, sort_by="degree", node_color_by="degree", node_enc_kwargs={"size_scale": 10})
annotate.node_colormapping(G, color_by="degree")_____no_output_____

One thing we might infer from this visualization
is that the vast majority of nodes have a very small degree,
while a very small number of nodes have a high degree.
That would prompt us to think:
what process could be responsible for generating this graph?_____no_output_____## Inferring Graph Generating Model

Given a graph dataset, how do we identify which data generating model provides the best fit?

One way to do this is to compare characteristics of a graph generating model against the characteristics of the graph.
The logic here is that if we have a good graph generating model for the data,
we should, in theory, observe the observed graph's characteristics
in the graphs generated by the graph generating model._____no_output_____### Comparison of degree distribution

Let's compare the degree distribution between the data, a few Erdos-Renyi graphs, and a few Barabasi-Albert graphs._____no_output_____### Comparison with Barabasi-Albert graphs_____no_output_____

from ipywidgets import interact, IntSlider

m = IntSlider(value=2, min=1, max=10)

@interact(m=m)
def compare_barabasi_albert_graph(m):
    fig, ax = plt.subplots()
    G_ba = nx.barabasi_albert_graph(n=len(G.nodes()), m=m)
    x, y = ecdf(pd.Series(dict(nx.degree(G_ba))))
    ax.scatter(x, y, label="Barabasi-Albert Graph")

    x, y = ecdf(pd.Series(dict(nx.degree(G))))
    ax.scatter(x, y, label="Protein Interaction Network")
    ax.legend()_____no_output_____

### Comparison with Erdos-Renyi graphs_____no_output_____

from ipywidgets import FloatSlider
p = FloatSlider(value=0.6, min=0, max=0.1, step=0.001)

@interact(p=p)
def compare_erdos_renyi_graph(p):
    fig, ax = plt.subplots()
    G_er = nx.erdos_renyi_graph(n=len(G.nodes()), p=p)
    x, y = ecdf(pd.Series(dict(nx.degree(G_er))))
    ax.scatter(x, y, label="Erdos-Renyi Graph")

    x, y = ecdf(pd.Series(dict(nx.degree(G))))
    ax.scatter(x, y, label="Protein Interaction Network")
    ax.legend()
    ax.set_title(f"p={p}")_____no_output_____

Given the degree distribution only, which model do you think better describes the generation of a protein-protein interaction network?_____no_output_____## Quantitative Model Comparison

Each time we plug in a value of $m$ for the Barabasi-Albert graph model, we are using one of many possible Barabasi-Albert graph models, each with a different $m$.
Similarly, each time we choose a different $p$ for the Erdos-Renyi model, we are using one of many possible Erdos-Renyi graph models, each with a different $p$.

To quantitatively compare degree distributions, we can use the [Wasserstein distance][wasd] between the data.
Let's see how to implement this.

[wasd]: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wasserstein_distance.html_____no_output_____

from scipy.stats import wasserstein_distance

def erdos_renyi_degdist(n, p):
    """Return a Pandas series of degree distribution of an Erdos-Renyi graph."""
    G = nx.erdos_renyi_graph(n=n, p=p)
    return pd.Series(dict(nx.degree(G)))

def barabasi_albert_degdist(n, m):
    """Return a Pandas series of degree distribution of an Barabasi-Albert graph."""
    G = nx.barabasi_albert_graph(n=n, m=m)
    return pd.Series(dict(nx.degree(G)))_____no_output_____deg = pd.Series(dict(nx.degree(G)))

er_deg = erdos_renyi_degdist(n=len(G.nodes()), p=0.001)
ba_deg = barabasi_albert_degdist(n=len(G.nodes()), m=1)
wasserstein_distance(deg, er_deg), wasserstein_distance(deg, ba_deg)_____no_output_____

Notice that because the graphs are instantiated in a non-deterministic fashion, re-running the cell above will give you different values for each new graph generated.

Let's now plot the wasserstein distance to our graph data for the two particular Erdos-Renyi and Barabasi-Albert graph models shown above._____no_output_____

import matplotlib.pyplot as plt
from tqdm.autonotebook import tqdm

er_dist = []
ba_dist = []
for _ in tqdm(range(100)):
    er_deg = erdos_renyi_degdist(n=len(G.nodes()), p=0.001)
    er_dist.append(wasserstein_distance(deg, er_deg))

    ba_deg = barabasi_albert_degdist(n=len(G.nodes()), m=1)
    ba_dist.append(wasserstein_distance(deg, ba_deg))

# er_degs = [erdos_renyi_degdist(n=len(G.nodes()), p=0.001) for _ in range(100)]_____no_output_____import seaborn as sns
import janitor

data = (
    pd.DataFrame(
        {
            "Erdos-Renyi": er_dist,
            "Barabasi-Albert": ba_dist,
        }
    )
    .melt(value_vars=["Erdos-Renyi", "Barabasi-Albert"])
    .rename_columns({"variable": "Graph Model", "value": "Wasserstein Distance"})
)
sns.swarmplot(data=data, x="Graph Model", y="Wasserstein Distance")_____no_output_____

From this, we might conclude that the Barabasi-Albert graph with $m=1$ has the better fit to the protein-protein interaction network graph._____no_output_____## Interpretation

That statement, accurate as it might be, still does not connect the dots to _biology_.

Let's think about the generative model for this graph.
The Barabasi-Albert graph gives us a model for "rich gets richer".
Given the current state of the graph,
if we want to add a new edge, we first pick a node with probability proportional to
the number of edges it already has.
Then, we pick another node with probability proportional to the number of edges that it has too.
Finally, we add an edge there.
This has the effect of "enriching" nodes that have a large number of edges with more edges.

How might this connect to biology?

We can't necessarily provide a concrete answer, but this model might help raise new hypotheses.

For example, if protein-protein interactions of the "binding" kind
are driven by subdomains, then proteins that acquire a domain through recombination
may end up being able to bind to everything else that the domain was able to.
In this fashion, proteins with that particular binding domain
gain new edges more readily.

Testing these hypotheses would be a totally different matter, and at this point,
I submit the above hypothesis with a large amount of salt thrown over my shoulder.
In other words, the hypothesized mechanism could be completely wrong.
However, I hope that this example illustrated that
the usage of a "graph generative model" can help us narrow down hypotheses about the observed world._____no_output_____


# Notebook from quangphu1912/Py-data-analysis-McKinney
Path: 09_Plotting and Visualization.ipynb

# Plotting and Visualization_____no_output_____

%matplotlib inline
#@quangphu1912: this is to show the plot in the notebook _____no_output_____import numpy as np
import pandas as pd
PREVIOUS_MAX_ROWS = pd.options.display.max_rows
pd.options.display.max_rows = 20
np.random.seed(12345)
import matplotlib.pyplot as plt
import matplotlib
plt.rc('figure', figsize=(10, 6))
np.set_printoptions(precision=4, suppress=True)_____no_output_____

Making informative visualizations (sometimes called plots) is one of the most important tasks in data analysis. It may be a part of the exploratory process — for example, to help identify outliers or needed data transformations, or as a way of generating ideas for models. For others, building an interactive visualization for the web may be the end goal. Python has many add-on libraries for making static or dynamic visualizations, but I’ll be mainly focused on matplotlib and libraries that build on top of it.

matplotlib is a desktop plotting package designed for creating (mostly twodimensional) publication-quality plots. The project was started by John Hunter in 2002 to enable a MATLAB-like plotting interface in Python. The matplotlib and IPython communities have collaborated to simplify interactive plotting from the IPython shell (and now, Jupyter notebook). matplotlib supports various GUI backends on all operating systems and additionally can export visualizations to all of the common vector and raster graphics formats (PDF, SVG, JPG, PNG, BMP, GIF, etc.). With the exception of a few diagrams, nearly all of the graphics in this book were produced using matplotlib. Over time, matplotlib has spawned a number of add-on toolkits for data visualization that use matplotlib for their underlying plotting. One of these is seaborn, which we explore later in this chapter. The simplest way to follow the code examples in the chapter is to use interactive plotting in the Jupyter notebook. To set this up, execute the following statement in a Jupyter notebook:_____no_output_____

%matplotlib notebook_____no_output_____

## A Brief matplotlib API Primer_____no_output_____With matplotlib, we use the following import convention:_____no_output_____

import matplotlib.pyplot as plt_____no_output_____

After running %matplotlib notebook in Jupyter (or simply %matplotlib in IPython), we can try creating a simple plot. If everything is set up right, a line plot like Figure 9-1 should appear:_____no_output_____

import numpy as np
data = np.arange(10)
data_____no_output_____plt.plot(data)_____no_output_____

While libraries like seaborn and pandas’s built-in plotting functions will deal with many of the mundane details of making plots, should you wish to customize them beyond the function options provided, you will need to learn a bit about the matplotlib API._____no_output_____**NOTE**

There is not enough room in the book to give a comprehensive treatment to the breadth and depth of functionality in matplotlib. It should be enough to teach you the ropes to get up and running. The matplotlib gallery and documentation are the best resource for learning advanced features._____no_output_____### 9.1.1. Figures and Subplots_____no_output_____Plots in matplotlib reside within a `Figure` object. You can create a new figure with `plt.figure:`_____no_output_____

fig = plt.figure()_____no_output_____

In IPython, an empty plot window will appear, but in Jupyter nothing will be shown until we use a few more commands. `plt.figure` has a number of options; notably, `figsize` will guarantee the figure has a certain size and aspect ratio if saved to disk._____no_output_____You can’t make a plot with a blank figure. You have to create one or more `subplots` using `add_subplot`:_____no_output_____

ax1 = fig.add_subplot(2, 2, 1)_____no_output_____

This means that the figure should be 2 × 2 (so up to four plots in total), and we’re selecting the first of four subplots (numbered from 1). If you create the next two subplots, you’ll end up with a visualization that looks like Figure 9-2:_____no_output_____

ax2 = fig.add_subplot(2, 2, 2)
ax3 = fig.add_subplot(2, 2, 3)_____no_output_____

**TIP**

One nuance of using Jupyter notebooks is that plots are reset after each cell is evaluated, so for more complex plots you must put all of the plotting commands in a single notebook cell._____no_output_____

fig = plt.figure()
ax1 = fig.add_subplot(2, 2, 1)
ax2 = fig.add_subplot(2, 2, 2)
ax3 = fig.add_subplot(2, 2, 3)
#P note: there are 4 plots in the figure (2x2) and this syntax is to specify the location of the first 3 plots (identical plots)_____no_output_____

When you issue a plotting command like plt.plot([1.5, 3.5, -2, 1.6]), matplotlib draws on the last figure and subplot used (creating one if necessary), thus hiding the figure and subplot creation. So if we add the following command, you’ll get something like Figure 9-3:_____no_output_____

plt.plot(np.random.randn(50).cumsum(), 'k--')_____no_output_____

The 'k--' is a style option instructing matplotlib to plot a black dashed line.

The objects returned by `fig.add_subplot` here are `AxesSubplot` objects, on which you can directly plot on the other empty subplots by calling each one’s instance method (see Figure 9-4):_____no_output_____

_ = ax1.hist(np.random.randn(100), bins=20, color='k', alpha=0.3)
ax2.scatter(np.arange(30), np.arange(30) + 3 * np.random.randn(30))_____no_output_____plt.close('all')_____no_output_____

Creating a figure with a grid of subplots is a very common task, so matplotlib includes a convenience method, plt.subplots, that creates a new figure and returns a NumPy array containing the created subplot objects:_____no_output_____

fig, axes = plt.subplots(2, 3)
axes_____no_output_____

This is very useful, as the `axes` array can be easily indexed like a two dimensional array; for example, axes[0, 1]. You can also indicate that subplots should have the same x- or y-axis using `sharex` and `sharey`, respectively. This is especially useful when you’re comparing data on the same scale; otherwise, matplotlib autoscales plot limits independently. See Table 9-1 for more on this method_____no_output_____#### Adjusting the spacing around subplots_____no_output_____By default matplotlib leaves a certain amount of padding around the outside of the subplots and spacing between subplots. This spacing is all specified relative to the height and width of the plot, so that if you resize the plot either programmatically or manually using the GUI window, the plot will dynamically adjust itself. You can change the spacing using the `subplots_adjust` method on `Figure` objects, also available as a top-level function:_____no_output_____subplots_adjust(left=None, bottom=None, right=None, top=None,
                wspace=None, hspace=None)_____no_output_____wspace and hspace controls the percent of the figure width and figure height, respectively, to use as spacing between subplots. Here is a small example where I shrink the spacing all the way to zero (see Figure 9-5):_____no_output_____

fig, axes = plt.subplots(2, 2, sharex=True, sharey=True)
for i in range(2):
    for j in range(2):
        axes[i, j].hist(np.random.randn(500), bins=50, color='k', alpha=0.5)
plt.subplots_adjust(wspace=0, hspace=0)_____no_output_____

You may notice that the axis labels overlap. matplotlib doesn’t check whether the labels overlap, so in a case like this you would need to fix the labels yourself by specifying explicit tick locations and tick labels (we’ll look at how to do this in the following sections)._____no_output_____### 9.1.2. Colors, Markers, and Line Styles_____no_output_____Matplotlib’s main `plot` function accepts arrays of x and y coordinates and optionally a string abbreviation indicating color and line style. For example, to plot x versus y with green dashes, you would execute:_____no_output_____ax.plot(x, y, 'g--')_____no_output_____This way of specifying both color and line style in a string is provided as a convenience; in practice if you were creating plots programmatically you might prefer not to have to munge strings together to create plots with the desired style. The same plot could also have been expressed more explicitly as:_____no_output_____ax.plot(x, y, linestyle='--', color='g')_____no_output_____There are a number of color abbreviations provided for commonly used colors, but you can use any color on the spectrum by specifying its hex code (e.g., '#CECECE'). You can see the full set of line styles by looking at the docstring for plot (use plot? in IPython or Jupyter)._____no_output_____

plt.plot??Signature: plt.plot(*args, scalex=True, scaley=True, data=None, **kwargs)
Docstring:
Plot y versus x as lines and/or markers.

Call signatures::

    plot([x], y, [fmt], *, data=None, **kwargs)
    plot([x], y, [fmt], [x2], y2, [fmt2], ..., **kwargs)

The coordinates of the points or line nodes are given by *x*, *y*.

The optional parameter *fmt* is a convenient way for defining basic
formatting like color, marker and linestyle. It's a shortcut string
notation described in the *Notes* section below.

>>> plot(x, y)        # plot x and y using default line style and color
>>> plot(x, y, 'bo')  # plot x and y using blue circle markers
>>> plot(y)           # plot y using x as index array 0..N-1
>>> plot(y, 'r+')     # ditto, but with red plusses

You can use `.Line2D` properties as keyword arguments for more
control on the appearance. Line properties and *fmt* can be mixed.
The following two calls yield identical results:

>>> plot(x, y, 'go--', linewidth=2, markersize=12)
>>> plot(x, y, color='green', marker='o', linestyle='dashed',
...      linewidth=2, markersize=12)

When conflicting with *fmt*, keyword arguments take precedence.

**Plotting labelled data**

There's a convenient way for plotting objects with labelled data (i.e.
data that can be accessed by index ``obj['y']``). Instead of giving
the data in *x* and *y*, you can provide the object in the *data*
parameter and just give the labels for *x* and *y*::

>>> plot('xlabel', 'ylabel', data=obj)

All indexable objects are supported. This could e.g. be a `dict`, a
`pandas.DataFrame` or a structured numpy array.

**Plotting multiple sets of data**

There are various ways to plot multiple sets of data.

- The most straight forward way is just to call `plot` multiple times.
  Example:

  >>> plot(x1, y1, 'bo')
  >>> plot(x2, y2, 'go')

- If *x* and/or *y* are 2D arrays a separate data set will be drawn
  for every column. If both *x* and *y* are 2D, they must have the
  same shape. If only one of them is 2D with shape (N, m) the other
  must have length N and will be used for every data set m.

  Example:

  >>> x = [1, 2, 3]
  >>> y = np.array([[1, 2], [3, 4], [5, 6]])
  >>> plot(x, y)

  is equivalent to:

  >>> for col in range(y.shape[1]):
  ...     plot(x, y[:, col])

- The third way is to specify multiple sets of *[x]*, *y*, *[fmt]*
  groups::

  >>> plot(x1, y1, 'g^', x2, y2, 'g-')

  In this case, any additional keyword argument applies to all
  datasets. Also this syntax cannot be combined with the *data*
  parameter.

By default, each line is assigned a different style specified by a
'style cycle'. The *fmt* and line property parameters are only
necessary if you want explicit deviations from these defaults.
Alternatively, you can also change the style cycle using
:rc:`axes.prop_cycle`.

Parameters
----------
x, y : array-like or scalar
    The horizontal / vertical coordinates of the data points.
    *x* values are optional and default to ``range(len(y))``.

    Commonly, these parameters are 1D arrays.

    They can also be scalars, or two-dimensional (in that case, the
    columns represent separate data sets).

    These arguments cannot be passed as keywords.

fmt : str, optional
    A format string, e.g. 'ro' for red circles. See the *Notes*
    section for a full description of the format strings.

    Format strings are just an abbreviation for quickly setting
    basic line properties. All of these and more can also be
    controlled by keyword arguments.

    This argument cannot be passed as keyword.

data : indexable object, optional
    An object with labelled data. If given, provide the label names to
    plot in *x* and *y*.

    .. note::
        Technically there's a slight ambiguity in calls where the
        second label is a valid *fmt*. ``plot('n', 'o', data=obj)``
        could be ``plt(x, y)`` or ``plt(y, fmt)``. In such cases,
        the former interpretation is chosen, but a warning is issued.
        You may suppress the warning by adding an empty format string
        ``plot('n', 'o', '', data=obj)``.

Returns
-------
list of `.Line2D`
    A list of lines representing the plotted data.

Other Parameters
----------------
scalex, scaley : bool, default: True
    These parameters determine if the view limits are adapted to the
    data limits. The values are passed on to `autoscale_view`.

**kwargs : `.Line2D` properties, optional
    *kwargs* are used to specify properties like a line label (for
    auto legends), linewidth, antialiasing, marker face color.
    Example::

    >>> plot([1, 2, 3], [1, 2, 3], 'go-', label='line 1', linewidth=2)
    >>> plot([1, 2, 3], [1, 4, 9], 'rs', label='line 2')

    If you specify multiple lines with one plot call, the kwargs apply
    to all those lines. In case the label object is iterable, each
    element is used as labels for each set of data.

    Here is a list of available `.Line2D` properties:

    Properties:
    agg_filter: a filter function, which takes a (m, n, 3) float array and a dpi value, and returns a (m, n, 3) array
    alpha: scalar or None
    animated: bool
    antialiased or aa: bool
    clip_box: `.Bbox`
    clip_on: bool
    clip_path: Patch or (Path, Transform) or None
    color or c: color
    contains: unknown
    dash_capstyle: `.CapStyle` or {'butt', 'projecting', 'round'}
    dash_joinstyle: `.JoinStyle` or {'miter', 'round', 'bevel'}
    dashes: sequence of floats (on/off ink in points) or (None, None)
    data: (2, N) array or two 1D arrays
    drawstyle or ds: {'default', 'steps', 'steps-pre', 'steps-mid', 'steps-post'}, default: 'default'
    figure: `.Figure`
    fillstyle: {'full', 'left', 'right', 'bottom', 'top', 'none'}
    gid: str
    in_layout: bool
    label: object
    linestyle or ls: {'-', '--', '-.', ':', '', (offset, on-off-seq), ...}
    linewidth or lw: float
    marker: marker style string, `~.path.Path` or `~.markers.MarkerStyle`
    markeredgecolor or mec: color
    markeredgewidth or mew: float
    markerfacecolor or mfc: color
    markerfacecoloralt or mfcalt: color
    markersize or ms: float
    markevery: None or int or (int, int) or slice or list[int] or float or (float, float) or list[bool]
    path_effects: `.AbstractPathEffect`
    picker: float or callable[[Artist, Event], tuple[bool, dict]]
    pickradius: float
    rasterized: bool
    sketch_params: (scale: float, length: float, randomness: float)
    snap: bool or None
    solid_capstyle: `.CapStyle` or {'butt', 'projecting', 'round'}
    solid_joinstyle: `.JoinStyle` or {'miter', 'round', 'bevel'}
    transform: `matplotlib.transforms.Transform`
    url: str
    visible: bool
    xdata: 1D array
    ydata: 1D array
    zorder: float

See Also
--------
scatter : XY scatter plot with markers of varying size and/or color (
    sometimes also called bubble chart).

Notes
-----
**Format Strings**

A format string consists of a part for color, marker and line::

    fmt = '[marker][line][color]'

Each of them is optional. If not provided, the value from the style
cycle is used. Exception: If ``line`` is given, but no ``marker``,
the data will be a line without markers.

Other combinations such as ``[color][marker][line]`` are also
supported, but note that their parsing may be ambiguous.

**Markers**

=============   ===============================
character       description
=============   ===============================
``'.'``         point marker
``','``         pixel marker
``'o'``         circle marker
``'v'``         triangle_down marker
``'^'``         triangle_up marker
``'<'``         triangle_left marker
``'>'``         triangle_right marker
``'1'``         tri_down marker
``'2'``         tri_up marker
``'3'``         tri_left marker
``'4'``         tri_right marker
``'8'``         octagon marker
``'s'``         square marker
``'p'``         pentagon marker
``'P'``         plus (filled) marker
``'*'``         star marker
``'h'``         hexagon1 marker
``'H'``         hexagon2 marker
``'+'``         plus marker
``'x'``         x marker
``'X'``         x (filled) marker
``'D'``         diamond marker
``'d'``         thin_diamond marker
``'|'``         vline marker
``'_'``         hline marker
=============   ===============================

**Line Styles**

=============    ===============================
character        description
=============    ===============================
``'-'``          solid line style
``'--'``         dashed line style
``'-.'``         dash-dot line style
``':'``          dotted line style
=============    ===============================

Example format strings::

    'b'    # blue markers with default shape
    'or'   # red circles
    '-g'   # green solid line
    '--'   # dashed line with default color
    '^k:'  # black triangle_up markers connected by a dotted line

**Colors**

The supported color abbreviations are the single letter codes

=============    ===============================
character        color
=============    ===============================
``'b'``          blue
``'g'``          green
``'r'``          red
``'c'``          cyan
``'m'``          magenta
``'y'``          yellow
``'k'``          black
``'w'``          white
=============    ===============================

and the ``'CN'`` colors that index into the default property cycle.

If the color is the only part of the format string, you can
additionally use any  `matplotlib.colors` spec, e.g. full names
(``'green'``) or hex strings (``'#008000'``).
Source:
@_copy_docstring_and_deprecators(Axes.plot)
def plot(*args, scalex=True, scaley=True, data=None, **kwargs):
    return gca().plot(
        *args, scalex=scalex, scaley=scaley,
        **({"data": data} if data is not None else {}), **kwargs)
File:      d:\users\phu.le2\anaconda3\lib\site-packages\matplotlib\pyplot.py
Type:      function

Line plots can additionally have `markers` to highlight the actual data points. Since matplotlib creates a continuous line plot, interpolating between points, it can occasionally be unclear where the points lie. The marker can be part of the style string, which must have color followed by marker type and line style (see Figure 9-6):_____no_output_____

plt.figure()_____no_output_____

This could also have been written more explicitly as:_____no_output_____

from numpy.random import randn
plt.plot(randn(30).cumsum(), 'ko--')
plt.show() #P added_____no_output_____

plot(randn(30).cumsum(), color='k', linestyle='dashed', marker='o')_____no_output_____

plt.close('all')_____no_output_____

For line plots, you will notice that subsequent points are linearly interpolated by default. This can be altered with the drawstyle option (Figure 9-7):_____no_output_____

data = np.random.randn(30).cumsum()
plt.plot(data, 'k--', label='Default')
plt.plot(data, 'k-', drawstyle='steps-post', label='steps-post')
plt.legend(loc='best')
plt.show() #P added_____no_output_____

You may notice output like <matplotlib.lines.Line2D at ...> when you run this. matplotlib returns objects that reference the plot subcomponent that was just added. A lot of the time you can safely ignore this output. Here, since we passed the label arguments to plot, we are able to create a plot legend to identify each line using plt.legend._____no_output_____**NOTE**

You must call plt.legend (or ax.legend, if you have a reference to the axes) to create the legend, whether or not you passed the label options when plotting the data._____no_output_____### 9.1.3. Ticks, Labels, and Legends_____no_output_____#### Setting the title, axis labels, ticks, and ticklabels_____no_output_____For most kinds of plot decorations, there are two main ways to do things:

 * using the procedural `pyplot` interface (i.e., `matplotlib.pyplot`) and
 * the more object-oriented native matplotlib API._____no_output_____The `pyplot` interface, designed for interactive use, consists of methods like `xlim`, `xticks`, and `xticklabels`. These control the plot range, tick locations, and tick labels, respectively. They can be used in two ways:
* Called with no arguments returns the current parameter value (e.g., plt.xlim() returns the current x-axis plotting range)
* Called with parameters sets the parameter value (e.g., plt.xlim([0, 10]), sets the x-axis range to 0 to 10)_____no_output_____All such methods act on the active or most recently created `AxesSubplot`. Each of them corresponds to two methods on the subplot object itself; in the case of `xlim` these are `ax.get_xlim` and `ax.set_xlim`. I prefer to use the subplot instance methods myself in the interest of being explicit (and especially when working with multiple subplots), but you can certainly use whichever you find more convenient._____no_output_____To illustrate customizing the axes, I’ll create a simple figure and plot of a random walk (see Figure 9-8):_____no_output_____

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.plot(np.random.randn(1000).cumsum())_____no_output_____

To change the x-axis ticks, it’s easiest to use `set_xticks` and `set_xticklabels`. The former instructs matplotlib where to place the ticks along the data range; by default these locations will also be the labels. But we can set any other values as the labels using set_xticklabels:_____no_output_____

ticks = ax.set_xticks([0, 250, 500, 750, 1000])
labels = ax.set_xticklabels(['one', 'two', 'three', 'four', 'five'],
                            rotation=30, fontsize='small')_____no_output_____

The rotation option sets the x tick labels at a 30-degree rotation. Lastly, set_xlabel gives a name to the x-axis and set_title the subplot title (see Figure 9-9 for the resulting figure):_____no_output_____

ax.set_title('My first matplotlib plot')
ax.set_xlabel('Stages')_____no_output_____ax.show()_____no_output_____

Modifying the y-axis consists of the same process, substituting y for x in the above. The axes class has a set method that allows batch setting of plot properties. From the prior example, we could also have written:_____no_output_____```py
props = {
    'title': 'My first matplotlib plot',
    'xlabel': 'Stages'
}
ax.set(**props)
```_____no_output_____#### Adding legends_____no_output_____Legends are another critical element for identifying plot elements. There are a couple of ways to add one. The easiest is to pass the label argument when adding each piece of the plot:_____no_output_____

from numpy.random import randn
fig = plt.figure(); ax = fig.add_subplot(1, 1, 1)
ax.plot(randn(1000).cumsum(), 'k', label='one')
ax.plot(randn(1000).cumsum(), 'k--', label='two')
ax.plot(randn(1000).cumsum(), 'k.', label='three')_____no_output_____

Once you’ve done this, you can either call ax.legend() or plt.legend() to automatically create a legend. The resulting plot is in Figure 9-10:_____no_output_____

ax.legend(loc='best')_____no_output_____#P checking type of ax to see why I could not call the show func
ax?Type:            AxesSubplot
String form:     AxesSubplot(0.125,0.125;0.775x0.755)
File:            d:\users\phu.le2\anaconda3\lib\site-packages\matplotlib\axes\_subplots.py
Docstring:       <no docstring>
Class docstring:
Base class for subplots, which are :class:`Axes` instances with
additional methods to facilitate generating and manipulating a set
of :class:`Axes` within a figure.
Init docstring:
Parameters
----------
fig : `matplotlib.figure.Figure`

*args : tuple (*nrows*, *ncols*, *index*) or int
    The array of subplots in the figure has dimensions ``(nrows,
    ncols)``, and *index* is the index of the subplot being created.
    *index* starts at 1 in the upper left corner and increases to the
    right.

    If *nrows*, *ncols*, and *index* are all single digit numbers, then
    *args* can be passed as a single 3-digit number (e.g. 234 for
    (2, 3, 4)).

**kwargs
    Keyword arguments are passed to the Axes (sub)class constructor.

The legend method has several other choices for the location loc argument. See the docstring (with ax.legend?) for more information.

The loc tells matplotlib where to place the plot. *If you aren’t picky, `'best'` is a good option, as it will choose a location that is most out of the way*. To exclude one or more elements from the legend, pass no label or label='_nolegend_'._____no_output_____### 9.1.4. Annotations and Drawing on a Subplot_____no_output_____In addition to the standard plot types, *you may wish to draw your own plot annotations, which could consist of text, arrows, or other shapes*. You can add annotations and text using the `text`, `arrow`, and `annotate` functions. text draws text at given coordinates (x, y) on the plot with optional custom styling:_____no_output_____```py
ax.text(x, y, 'Hello world!',
        family='monospace', fontsize=10)
```_____no_output_____Annotations can draw both text and arrows arranged appropriately. As an example, let’s plot the closing S&P 500 index price since 2007 (obtained from Yahoo! Finance) and annotate it with some of the important dates from the 2008–2009 financial crisis. You can most easily reproduce this code example in a single cell in a Jupyter notebook. See Figure 9-11 for the result:_____no_output_____

from datetime import datetime

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)

data = pd.read_csv('examples/spx.csv', index_col=0, parse_dates=True)
spx = data['SPX']

spx.plot(ax=ax, style='k-')

crisis_data = [
    (datetime(2007, 10, 11), 'Peak of bull market'),
    (datetime(2008, 3, 12), 'Bear Stearns Fails'),
    (datetime(2008, 9, 15), 'Lehman Bankruptcy')
]

for date, label in crisis_data:
    ax.annotate(label, xy=(date, spx.asof(date) + 75),
                xytext=(date, spx.asof(date) + 225),
                arrowprops=dict(facecolor='black', headwidth=4, width=2,
                                headlength=4),
                horizontalalignment='left', verticalalignment='top')

# Zoom in on 2007-2010
ax.set_xlim(['1/1/2007', '1/1/2011'])
ax.set_ylim([600, 1800])

ax.set_title('Important dates in the 2008-2009 financial crisis')_____no_output_____

There are a couple of important points to highlight in this plot: the ax.annotate method can draw labels at the indicated x and y coordinates.

We use the `set_xlim` and `set_ylim` methods to manually set the start and end boundaries for the plot rather than using matplotlib’s default. Lastly, `ax.set_title` adds a main title to the plot.

See the online matplotlib gallery for many more annotation examples to learn from._____no_output_____Drawing shapes requires some more care. matplotlib has objects that represent many common shapes, referred to as `patches`. Some of these, like `Rectangle` and `Circle`, are found in matplotlib.pyplot, but the full set is located in `matplotlib.patches`.

To add a shape to a plot, you create the patch object shp and add it to a subplot by calling ax.add_patch(shp) (see Figure 9-12):_____no_output_____

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)

rect = plt.Rectangle((0.2, 0.75), 0.4, 0.15, color='k', alpha=0.3)
circ = plt.Circle((0.7, 0.2), 0.15, color='b', alpha=0.3)
pgon = plt.Polygon([[0.15, 0.15], [0.35, 0.4], [0.2, 0.6]],
                   color='g', alpha=0.5)

ax.add_patch(rect)
ax.add_patch(circ)
ax.add_patch(pgon)_____no_output_____

If you look at the implementation of many familiar plot types, you will see that they are assembled from patches._____no_output_____

fig = plt.figure(figsize=(12, 6)); ax = fig.add_subplot(1, 1, 1)
rect = plt.Rectangle((0.2, 0.75), 0.4, 0.15, color='k', alpha=0.3)
circ = plt.Circle((0.7, 0.2), 0.15, color='b', alpha=0.3)
pgon = plt.Polygon([[0.15, 0.15], [0.35, 0.4], [0.2, 0.6]],
                   color='g', alpha=0.5)
ax.add_patch(rect)
ax.add_patch(circ)
ax.add_patch(pgon)_____no_output_____

### 9.1.5. Saving Plots to File_____no_output_____You can save the active figure to file using `plt.savefig`. This method is equivalent to the figure object’s `savefig` instance method. For example, to save an SVG version of a figure, you need only type:_____no_output_____```py
plt.savefig('figpath.svg')
```_____no_output_____The file type is inferred from the file extension. So if you used .pdf instead, you would get a PDF. There are a couple of important options that I use frequently for publishing graphics:
* `dpi`, which controls the dots-per-inch resolution, and
* `bbox_inches`, which can trim the whitespace around the actual figure.
To get the same plot as a PNG with minimal whitespace around the plot and at 400 DPI, you would do:_____no_output_____```py
plt.savefig('figpath.png', dpi=400, bbox_inches='tight')
```_____no_output_____`savefig` doesn’t have to write to disk; it can also write to any file-like object, such as a BytesIO:_____no_output_____```py
from io import BytesIO
buffer = BytesIO()
plt.savefig(buffer)
plot_data = buffer.getvalue()
```_____no_output_____### 9.1.6. matplotlib Configuration_____no_output_____matplotlib comes configured with color schemes and defaults that are geared primarily toward preparing figures for publication. Fortunately, nearly all of the default behavior can be customized via an extensive `set of global parameters` governing figure size, subplot spacing, colors, font sizes, grid styles, and so on. One way to modify the configuration programmatically from Python is to use the `rc method`; for example, to set the *global default figure size to be 10 × 10*, you could enter:_____no_output_____```py
plt.rc('figure', figsize=(10, 10))
```_____no_output_____The first argument to rc is the component you wish to customize, such as 'figure', 'axes', 'xtick', 'ytick', 'grid', 'legend', or many others. After that can follow a sequence of keyword arguments indicating the new parameters. An easy way to write down the options in your program is as a dict:_____no_output_____```py
font_options = {'family' : 'monospace',
                'weight' : 'bold',
                'size'   : 'small'}
plt.rc('font', **font_options)
```_____no_output_____For more extensive customization and to see a list of all the options, matplotlib comes with a configuration file matplotlibrc in the matplotlib/mpl-data directory. If you customize this file and place it in your home directory titled .matplotlibrc, it will be loaded each time you use matplotlib.

As we’ll see in the next section, the seaborn package has several built-in plot themes or styles that use matplotlib’s configuration system internally._____no_output_____## 9.2. Plotting with pandas and seaborn_____no_output_____matplotlib can be a fairly low-level tool. You assemble a plot from its base components: the data display (i.e., the type of plot: line, bar, box, scatter, contour, etc.), legend, title, tick labels, and other annotations.

In pandas we may have multiple columns of data, along with row and column labels. pandas itself has built-in methods that simplify creating visualizations from DataFrame and Series objects. Another library is `seaborn`, a statistical graphics library created by Michael Waskom. Seaborn simplifies creating many common visualization types._____no_output_____**TIP**

Importing seaborn modifies the default matplotlib color schemes and plot styles to improve readability and aesthetics. Even if you do not use the seaborn API, you may prefer to import seaborn as a simple way to improve the visual aesthetics of general matplotlib plots_____no_output_____### 9.2.1. Line Plots_____no_output_____

plt.close('all')_____no_output_____

Series and DataFrame each have a plot attribute for making some basic plot types. *By default, `plot()` makes line plots* (see Figure 9-13):_____no_output_____

s = pd.Series(np.random.randn(10).cumsum(), index=np.arange(0, 100, 10))
s.plot()_____no_output_____

The Series object’s index is passed to matplotlib for plotting on the x-axis, though you can disable this by passing `use_index=False`. The x-axis ticks and limits can be adjusted with the `xticks` and `xlim` options, and y-axis respectively with `yticks` and `ylim`. See Table 9-3 for a full listing of plot options. I’ll comment on a few more of them throughout this section and leave the rest to you to explore._____no_output_____Most of pandas’s plotting methods accept an optional ax parameter, which can be a matplotlib subplot object. This gives you more flexible placemen tof subplots in a grid layout.

DataFrame’s plot method plots each of its columns as a different line on the same subplot, creating a legend automatically (see Figure 9-14):_____no_output_____

df = pd.DataFrame(np.random.randn(10, 4).cumsum(0),
                  columns=['A', 'B', 'C', 'D'],
                  index=np.arange(0, 100, 10))
df_____no_output_____df.plot()_____no_output_____

The plot attribute contains a “family” of methods for different plot types.
For example, `df.plot()` is equivalent to *df.plot.line()*. We’ll explore some of these methods next._____no_output_____### 9.2.2. Bar Plots_____no_output_____The `plot.bar()` and `plot.barh()` make vertical and horizontal bar plots, respectively. In this case, the Series or DataFrame index will be used as the x (bar) or y (barh) ticks (see Figure 9-15):_____no_output_____

fig, axes = plt.subplots(2, 1)
data = pd.Series(np.random.rand(16), index=list('abcdefghijklmnop'))
data.plot.bar(ax=axes[0], color='k', alpha=0.7)
data.plot.barh(ax=axes[1], color='k', alpha=0.7)
#The options color='k' and alpha=0.7 set the color of the plots to black and use partial transparency on the filling. (author note)_____no_output_____

With a DataFrame, bar plots group the values in each row together in a group in bars, side by side, for each value. See Figure 9-16:_____no_output_____

np.random.seed(12348)_____no_output_____df = pd.DataFrame(np.random.rand(6, 4),
                  index=['one', 'two', 'three', 'four', 'five', 'six'],
                  columns=pd.Index(['A', 'B', 'C', 'D'], name='Genus'))
df_____no_output_____df.plot.bar()
#Note that the name “Genus” on the DataFrame’s columns is used to title the legend (author note)._____no_output_____plt.figure()_____no_output_____

We create stacked bar plots from a DataFrame by passing `stacked=True`, resulting in the value in each row being stacked together (see Figure 9-17):_____no_output_____

df.plot.barh(stacked=True, alpha=0.5)_____no_output_____

**NOTE**

A useful recipe for bar plots is to visualize a Series’s value frequency using *value_counts*: `s.value_counts().plot.bar()`._____no_output_____

plt.close('all')_____no_output_____

Returning to the tipping dataset used earlier in the book, suppose we wanted to make a stacked bar plot showing the percentage of data points for each party size on each day. I load the data using read_csv and make a crosstabulation by day and party size:_____no_output_____

tips = pd.read_csv('examples/tips.csv')
party_counts = pd.crosstab(tips['day'], tips['size'])
party_counts
# Not many 1- and 6-person parties_____no_output_____party_counts = party_counts.loc[:, 2:5]_____no_output_____

Then, normalize so that each row sums to 1 and make the plot (see Figure 9-18):_____no_output_____

# Normalize to sum to 1
party_pcts = party_counts.div(party_counts.sum(1), axis=0)
party_pcts_____no_output_____party_pcts.plot.bar()_____no_output_____

So you can see that party sizes appear to increase on the weekend in this dataset._____no_output_____

plt.close('all')_____no_output_____

With data that requires aggregation or summarization before making a plot, using the `seaborn` package can make things much simpler. Let’s look now at the tipping percentage by day with seaborn (see Figure 9-19 for the resulting plot):_____no_output_____

import seaborn as sns
tips['tip_pct'] = tips['tip'] / (tips['total_bill'] - tips['tip'])
tips.head()_____no_output_____sns.barplot(x='tip_pct', y='day', data=tips, orient='h')_____no_output_____

Plotting functions in seaborn take a data argument, which can be a pandas DataFrame. The other arguments refer to column names. Because there are multiple observations for each value in the day, *the bars are the average value of tip_pct*. The *black lines drawn on the bars represent the 95% confidence interval* (this can be configured through optional arguments)._____no_output_____

plt.close('all')_____no_output_____

seaborn.barplot has a `hue` option that enables us to split by an additional categorical value (Figure 9-20):_____no_output_____

sns.barplot(x='tip_pct', y='day', hue='time', data=tips, orient='h')_____no_output_____plt.close('all')_____no_output_____

Notice that seaborn has automatically changed the aesthetics of plots: the default color palette, plot background, and grid line colors. You can switch between different plot appearances using `seaborn.set`:_____no_output_____

sns.set(style="whitegrid")_____no_output_____

### 9.2.3. Histograms and Density Plots_____no_output_____

plt.figure()_____no_output_____

A histogram is a kind of bar plot that gives a discretized display of value frequency. The data points are split into discrete, evenly spaced bins, and the number of data points in each bin is plotted.

Using the tipping data from before, we can make a histogram of tip percentages of the total bill using the `plot.hist` method on the Series (see Figure 9-21):_____no_output_____

tips['tip_pct'].plot.hist(bins=50)_____no_output_____plt.figure()_____no_output_____

A related plot type is a `density plot`, which is formed by *computing an estimate of a continuous probability distribution that might have generated the observed data.* The usual procedure is to approximate this distribution as a mixture of “kernels” — that is, simpler distributions like the normal distribution. Thus, density plots are also known as `kernel density estimate (KDE)` plots. Using plot.kde makes a density plot using the conventional mixture-of-normals estimate (see Figure 9-22):_____no_output_____

tips['tip_pct'].plot.density()_____no_output_____plt.figure()_____no_output_____

Seaborn makes histograms and density plots even easier through its `distplot` method, which can plot both a histogram and a continuous density estimate simultaneously. As an example, consider a bimodal distribution consisting of draws from two different standard normal distributions (see Figure 9-23):_____no_output_____

comp1 = np.random.normal(0, 1, size=200)
comp2 = np.random.normal(10, 2, size=200)
values = pd.Series(np.concatenate([comp1, comp2]))
# sns.distplot(values, bins=100, color='k')
sns.distplot(values, bins=100, color='k')D:\Users\phu.le2\Anaconda3\lib\site-packages\seaborn\distributions.py:2557: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)

### 9.2.4. Scatter or Point Plots_____no_output_____Point plots or scatter plots can be a useful way of examining the relationship between two one-dimensional data series. For example, here we load the macrodata dataset from the statsmodels project, select a few variables, then compute log differences:_____no_output_____

macro = pd.read_csv('examples/macrodata.csv')
data = macro[['cpi', 'm1', 'tbilrate', 'unemp']]
trans_data = np.log(data).diff().dropna()
trans_data[-5:]_____no_output_____plt.figure()_____no_output_____

We can then use seaborn’s regplot method, which makes a *scatter plot* and *fits a linear regression line* (see Figure 9-24):_____no_output_____

sns.regplot('m1', 'unemp', data=trans_data)
plt.title('Changes in log %s versus log %s' % ('m1', 'unemp'))D:\Users\phu.le2\Anaconda3\lib\site-packages\seaborn\_decorators.py:36: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(

In exploratory data analysis it’s helpful to be able to look at all the scatter plots among a group of variables; this is known as a `pairs plot` or `scatter plot matrix`. Making such a plot from scratch is a bit of work, so seaborn has a convenient pairplot function, which supports placing histograms or density estimates of each variable along the diagonal (see Figure 9-25 for the resulting plot):_____no_output_____

sns.pairplot(trans_data, diag_kind='kde', plot_kws={'alpha': 0.2})_____no_output_____

You may notice the plot_kws argument. This enables us to pass down configuration options to the individual plotting calls on the off-diagonal elements. Check out the seaborn.pairplot docstring for more granular configuration options._____no_output_____### 9.2.5. Facet Grids and Categorical Data_____no_output_____What about datasets where we have additional grouping dimensions? One way to visualize data with many categorical variables is to use a `facet grid`. Seaborn has a useful built-in function `factorplot` that simplifies making many kinds of faceted plots (see Figure 9-26 for the resulting plot):_____no_output_____

sns.factorplot(x='day', y='tip_pct', hue='time', col='smoker',
               kind='bar', data=tips[tips.tip_pct < 1])D:\Users\phu.le2\Anaconda3\lib\site-packages\seaborn\categorical.py:3714: UserWarning: The `factorplot` function has been renamed to `catplot`. The original name will be removed in a future release. Please update your code. Note that the default `kind` in `factorplot` (`'point'`) has changed `'strip'` in `catplot`.
  warnings.warn(msg)
#P update after the error message above
sns.catplot(x='day', y='tip_pct', hue='time', col='smoker',
               kind='bar', data=tips[tips.tip_pct < 1])_____no_output_____

Instead of grouping by 'time' by different bar colors within a facet, we can also expand the facet grid by adding one row per time value (Figure 9-27):_____no_output_____

sns.factorplot(x='day', y='tip_pct', row='time',
               col='smoker',
               kind='bar', data=tips[tips.tip_pct < 1])D:\Users\phu.le2\Anaconda3\lib\site-packages\seaborn\categorical.py:3714: UserWarning: The `factorplot` function has been renamed to `catplot`. The original name will be removed in a future release. Please update your code. Note that the default `kind` in `factorplot` (`'point'`) has changed `'strip'` in `catplot`.
  warnings.warn(msg)
#P update after the error message above
sns.catplot(x='day', y='tip_pct', row='time',
               col='smoker',
               kind='bar', data=tips[tips.tip_pct < 1])_____no_output_____

`factorplot` supports other plot types that may be useful depending on what you are trying to display. For example, box plots (which show the median, quartiles, and outliers) can be an effective visualization type (Figure 9-28):_____no_output_____

sns.factorplot(x='tip_pct', y='day', kind='box',
               data=tips[tips.tip_pct < 0.5])D:\Users\phu.le2\Anaconda3\lib\site-packages\seaborn\categorical.py:3714: UserWarning: The `factorplot` function has been renamed to `catplot`. The original name will be removed in a future release. Please update your code. Note that the default `kind` in `factorplot` (`'point'`) has changed `'strip'` in `catplot`.
  warnings.warn(msg)
sns.catplot(x='tip_pct', y='day', kind='box',
               data=tips[tips.tip_pct < 0.5])_____no_output_____

You can create your own facet grid plots using the more general `seaborn.FacetGrid` class. See the seaborn documentation for more_____no_output_____## 9.3. Other Python Visualization Tools_____no_output_____

pd.options.display.max_rows = PREVIOUS_MAX_ROWS_____no_output_____

## Conclusion_____no_output_____


# Notebook from mrixlam/inst_rad_primer_g2
Path: G2 - Radiation Primer Lab.ipynb

# Group Member Names
(Double-click here and update your group member names below. Remember to keep the `*` symbols for bullet points. Hit  shift+enter when you're done to render the Markdown and make things look pretty again.)
* Rubaiat
* Joe
* Bronlyn
* Molly
_____no_output_____

'''
Your first cell of code. Notice that the pull-down menu says "Code"
when you double-click on this cell. If you double-click the first cell again,
you'll see "Markdown".
In code, stuff that comes between triplets of apostrophes or after a hashtag symbol
is a comment, and is ignored.

This cell needs to be run for the rest of this lab to work.
Be sure to run this cell by hitting shift+enter like you did above for the Markdown cell,
or by clicking the "Run" button above.'''

# These lines load special libraries that help python do sciency things.
# They don't come pre-loaded with "vanilla" python because not everyone uses them, they take
# up space on your drive, and importing them uses up some RAM.
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import cm
from netCDF4 import Dataset

# This next line makes it so plots will work in this notebook
%matplotlib inline

# Here we'll define some constants that the rest of the notebook uses.
# (It's nice to declare constants ahead of time. If you just start putting numbers
# in places where you want them, it can keep people from understanding your reasoning.)
speed_of_light = 3e8 # ms^-1
planck_constant = 6.626e-34 # J s
ceres_aperture = 18e-3 # 18 mm entrance pupil diameter (Smith et al., 2002)
ceres_altitude = 705e3 # 705 km nominal altitude for Aqua and Terra satellites_____no_output_____

# Problem 1 - Convert frequency to wavelength
Petty problem 2.1 wasn't too bad, but we can do better by automating it.

## Name some frequencies
Run the cell below to create some variables and display them in a table._____no_output_____

band_names = ['WFIU1', 'Ka band police radar', 'red', 'violet', 'UV-c', 'x-ray', 'gamma ray']
band_frequencies = np.array([101.9e6, 35e9, 4.5e14, 7.5e14, 1.5e15, 3e17, 3e19]) # in Hz or s^-1
print('-----------------------------------')
print("{:23}| Frequency".format('Description'))
print('-----------------------------------')
for iName, iFreq in zip(band_names,band_frequencies):
    iFreqSci="{:.1e} Hz".format(iFreq)
    iNameLong ="{:23}| ".format(iName)
    print(iNameLong + iFreqSci)
print('-----------------------------------')-----------------------------------
Description            | Frequency
-----------------------------------
WFIU1                  | 1.0e+08 Hz
Ka band police radar   | 3.5e+10 Hz
red                    | 4.5e+14 Hz
violet                 | 7.5e+14 Hz
UV-c                   | 1.5e+15 Hz
x-ray                  | 3.0e+17 Hz
gamma ray              | 3.0e+19 Hz
-----------------------------------

## Fix the converter
The frequency-to-wavelength converter `freq_to_wave` below is broken. Please fix it and run it so we can use it later if we need. Also, carefully fix the table header._____no_output_____

# Here's the function that should take frequency and phase speed
# and spit out wavelength.
def freq_to_wave(input_frequency, phase_speed):
    output_wavelength = phase_speed/input_frequency
    #output_wavelength = np.arange(len(input_frequency))+1 # replace the right-hand-side with an expression for wavelength
    return(output_wavelength)

# Here we run that function on all of the frequencies we defined in the previous cell.
band_wavelengths = freq_to_wave(band_frequencies, speed_of_light)

# Below we print a table of the frequencies in scientific notation.
# Carefully change the heading over the 2nd column.
print('-----------------------------------')
print("{:23}| Wavelength".format('Description'))
print('-----------------------------------')
for iName, iWave in zip(band_names,band_wavelengths):
    iWaveSci="{:.1e} m".format(iWave)
    iNameLong ="{:23}| ".format(iName)
    print(iNameLong + iWaveSci)
print('-----------------------------------')-----------------------------------
Description            | Wavelength
-----------------------------------
WFIU1                  | 2.9e+00 m
Ka band police radar   | 8.6e-03 m
red                    | 6.7e-07 m
violet                 | 4.0e-07 m
UV-c                   | 2.0e-07 m
x-ray                  | 1.0e-09 m
gamma ray              | 1.0e-11 m
-----------------------------------

## Plot your results
The cell below makes a scatter plot of wavelength versus frequency. Once you have fixed the frequency-to-wavelength converter above and calculated the correct wavelengths, run this plotter again.

Also, make the change suggested below so that you get a plot that's more useful for dealing with quantities orders of magnitude apart._____no_output_____

plt.plot(band_frequencies, band_wavelengths, 'o')
for iName, iFreq, iWave in zip(band_names, band_frequencies, band_wavelengths):
    plt.text(iFreq, iWave, "  {}".format(iName))
plt.title("Scatter plot of wavelength versus frequency")
plt.xlabel("Frequency (Hz)")
plt.ylabel("Wavelength (m)")

# Uncomment the lines below to change this to a log-log plot
plt.yscale("log")
plt.xscale("log")_____no_output_____

## Just Checking
You can check your answers against this image below.
![](http://www.sengpielaudio.com/SpectrumOfElectromagneticRadiation.gif)
_(c) Philip Ronan / Wikimedia Commons / CC-BY-SA-3.0______no_output_____# Problem 2 - Convert wavelength to frequency
## A little background
Light interacts with objects differently, depending on their size relative to the wavelength of the light. If an object is _much_ larger than the wavelength, geometric objects come into play—the kind of behavior we are familiar with in our macroscale world lit by visible light. If an object is _much_ smaller than the wavelength, it does what we call Rayleigh (pronounced ray-lee) scattering. If an object is similar in size to the wavelength, it does what we call mie (pronounced "me") scattering. The figure below has some nice examples. In it, $x$ is the ratio of object size to wavelength.

![Optical Regimes](http://2016.igem.org/wiki/images/4/46/T--TU_Delft--Scattering_regimes.png)

Using the cell below, print the frequency of light for which each of the object is clearly within its mie scattering regime. _Hint: You don't need the figure above. You just need to calculate the frequency of light corresponding to a wavelength as big as the object._

Run the cell below to define some length scales (in powers of ten)._____no_output_____## Doppler shifts and radar
Radars use doppler shifting to calculate how quickly rain droplets are moving radially relative to the radar station. Petty #2.2 - the doppler shift for a stationary transmitter and a moving observer - gets you halfway there (from the radar to the raindrop). The other half of the problem (the return trip to the radar) is the doppler shift for a moving transmitter and a stationary observer. The expression is slightly different:
$$f_o=\frac{f_t}{\left ( 1+\frac{\nu}{c} \right)}$$

## An James Webb Space Telescope example
The James Webb Space Telescope operates in the infrared to capture details from the farthest reachest of observable space. Since the universe is expanding, everything is moving apart. Stars farther away are moving faster. The resulting doppler shift makes stars appear redder, and is often called "red shift." Hubble used the red shift of stars to figure out how quickly the universe is expanding, and from there figured out how old the universe is. (No wonder there's a telescope named after him.)

The thing is, objects _very_ far away are moving _very_ quickly — at relativistic speeds compared to us. That's fast enough to red-shift their light right out of the visible and into the infrared. (The highest frequency waves, x-rays and gamma rays, are attenuated as they cross the universe; similar attenuation of relatively short waves occurs in a bunch of settings, as you'll see elsewhere in this course.)

By using infrared light, the James Webb Space Telescope will be able to see light that was given off by stars right after the universe was born. This light has been traveling across the universe for billions of years, and will teach us about the earliest moments of the universe. (And there's lots of amazing questions to be answered with infrared light from objects more nearby, like, say, exoplanets. Sometimes it's just great having a giant telescope out in our solar system.)

In the example below, we'll define some approximate wavelengths of near-infrared light, visible light, and ultraviolet. Then we'll put our doppler-shift expressions into code so we can see how different colors are red-shifted at different speeds._____no_output_____

obj_names = ['Aerosol Particle', 'Cloud Droplet', 'Rain Drop', 'Military Drone']
obj_magnitudes = np.array([-7, -5, -3, 1])
print('-----------------------------')
print("{:17}| Magnitude".format('Object'))
print('-----------------------------')
for i_name, i_mag in zip(obj_names, obj_magnitudes):
    i_mag_sci="10^({:}) m".format(i_mag)
    i_name_long ="{:17}| ".format(i_name)
    print(i_name_long + i_mag_sci)
print('-----------------------------')-----------------------------
Object           | Magnitude
-----------------------------
Aerosol Particle | 10^(-7) m
Cloud Droplet    | 10^(-5) m
Rain Drop        | 10^(-3) m
Military Drone   | 10^(1) m
-----------------------------

 We'll come back to scattering again soon. For now, just know that scattering regimes are important, and for a given kind of object (and object size), the choice of wavelength for a transmitter or sensor is important._____no_output_____## Time to fix a function again
Oh no! The wave-to-frequency converter `wave_to_freq` below is broken! (How does this keep happening!?) Please fix it, fix the table header, and run the cell to display a table of frequencies corresponding to a wavelength as wide as the object._____no_output_____

#  def wave_to_freq(input_wavelength, phase_speed):
    # output_frequency = np.arange(len(input_wavelength)) # replace the right-hand-side with an expression for frequency
    # return(output_frequency)

def wave_to_freq(input_wavelength, speed_of_light):
    output_frequency = speed_of_light/input_wavelength
    return(output_frequency)

obj_frequencies = wave_to_freq(10.**obj_magnitudes, speed_of_light)
print('--------------------------------')
print("{:17}| Frequency".format('Object'))
print('--------------------------------')
for iName, iFreq in zip(obj_names,obj_frequencies):
    iFreqSci="{:.1e} Hz".format(iFreq)
    iNameLong ="{:17}| ".format(iName)
    print(iNameLong + iFreqSci)
print('--------------------------------')--------------------------------
Object           | Frequency
--------------------------------
Aerosol Particle | 3.0e+15 Hz
Cloud Droplet    | 3.0e+13 Hz
Rain Drop        | 3.0e+11 Hz
Military Drone   | 3.0e+07 Hz
--------------------------------

# Problem 3 - Calculating doppler shift_____no_output_____

color_names = ['near-infrared', 'red', 'orange', 'yellow', 'green', 'blue', 'indigo', 'violet', 'ultraviolet']
color_plotnames = color_names.copy()
color_plotnames[0] = 'black'
color_plotnames[-1] = 'gray'
color_wavelengths = np.array([900, 710, 600, 550, 510, 450, 410, 390, 200])*1e-9
color_frequencies = wave_to_freq(color_wavelengths, speed_of_light)
star_speeds = np.array([0.1, 1, 2, 2.5, 2.75, 2.875, 2.9, 2.99, 2.997])*1e8_____no_output_____

### Moving transmitter, stationary observer
Let's treat the light from distant stars as being emitted by a moving transmitter, and make a contour plot of the color of observed light versus the transmitted frequency and transmitter velocity.

Oh, and the function below is broken. Again. (_Seriously, Dr. Staten!?_)_____no_output_____

# def doppler_moving_trns(transmitted_frequency, transmitter_speed):
    # observed_frequency = transmitted_frequency*np.ones_like(transmitter_speed) # FIXME
    # return(observed_frequency)

def doppler_moving_trns(transmitted_frequency, transmitter_speed):
    observed_frequency = transmitted_frequency/(1+(transmitter_speed/speed_of_light))
    return(observed_frequency)

observed_frequencies1 = doppler_moving_trns(color_frequencies[np.newaxis,:], star_speeds[:, np.newaxis])
plt.contourf(color_frequencies, star_speeds, observed_frequencies1, levels=color_frequencies, colors=color_plotnames, extend='both')
plt.xscale("log")
plt.title("Observed colors, stars moving")
plt.xlabel("Frequency (Hz)")
plt.ylabel("Transmitter speed ($10^8$ms$^{-1}$)")_____no_output_____

## Written Response
Based on the figure above, how fast does the transmitting star need to be moving away from the earth in order for all visible light to be shifted to infrared? Does this make sense? Why or why not? Enter your answer in the cell below (as Markdown)._____no_output_____`Double-click this cell to use for your response`_____no_output_____## Moving observer, stationary transmitter

Since motion in space is relative, anyway, can we just treat the observer as moving instead, and use the equation above? Let's use the moving observer formula from Petty #2.2 and see what happens._____no_output_____

def doppler_moving_rcvr(transmitted_frequency, receiver_speed):
    observed_frequency = transmitted_frequency*(1-(receiver_speed/speed_of_light))
    return(observed_frequency)

# def doppler_moving_rcvr(transmitted_frequency, receiver_speed):
    # observed_frequency = transmitted_frequency * np.ones_like(receiver_speed)
    # return(observed_frequency)

observed_frequencies2 = doppler_moving_rcvr(color_frequencies[np.newaxis,:], star_speeds[:, np.newaxis])
plt.contourf(color_frequencies, star_speeds, observed_frequencies2, levels=color_frequencies, colors=color_plotnames, extend='both')
plt.xscale("log")
plt.title("Observed colors, earth")
plt.xlabel("Transmitted Frequency (Hz)")
plt.ylabel("Speed of the moving Earth ($10^8$ms$^{-1}$)")_____no_output_____

## Written Response

Based on the figure above, how fast does the earth need to be moving away from the stars in order for all visible light to be shifted to infrared? Does this make sense? Why or why not? Enter your answer in the cell below (as Markdown)._____no_output_____`Double-click this cell to use for your response`_____no_output_____## Maybe _both_ are moving away from each other at similar speeds?
Now that you fixed both of the functions above, I'll assume that, since there's no absolute reference point anyway, we can just treat the transmitter and receiver as moving apart at equal and opposite velocities. Run the cell below to apply the moving transmitter function (at half-speed) to the transmitted frequencies and the moving observer equation (also at half-speed) and see how that does. Nothing for you to change; just run it._____no_output_____

observed_frequencies3 = doppler_moving_rcvr(doppler_moving_trns(color_frequencies[np.newaxis,:], star_speeds[:,np.newaxis]/2.), star_speeds[:, np.newaxis]/2.)
plt.contourf(color_frequencies, star_speeds, observed_frequencies3, levels=color_frequencies, colors=color_plotnames, extend='both')
plt.xscale("log")
plt.title("Observed colors, both moving")
plt.xlabel("Frequency (Hz)")
plt.ylabel("Summed speed of Stars and Earth ($10^8$ms$^{-1}$)")_____no_output_____

## Written Response

Based on the figure above, how fast would the earth and stars need to be moving away each other in order for all visible light to be shifted to infrared? Does this make sense? Why or why not? Enter your answer in the cell below (as Markdown)._____no_output_____`Double-click this cell to use for your response`_____no_output_____# Problem 4 - photons from earth...in space
Photodetectors (i.e. cameras, spectrometers, etc.) often work by accumulating a charge each time photons hit a chip. (In the case of a digital camera, this is called a charged couple device, or CCD, which contains a whole grid of tiny photo-receptors.) Every so often, this charge is recorded and dumped so the process can start over (and a new image or measurement can be taken).

The Clouds and the Earth's Radiant Energy System (CERES) instrument suite has several sensors designed to measure the longwave radiance of the earth from space. Some of these operate over a narrow spectral window. In contrast, the total infrared ("TOT") sensor captures everything between 0.3 $\mu$m to 200 $\mu$m, which is pretty much all that earth emits.

But CERES has a tiny telescope, it's hundreds of miles away, and it only samples one 10-km-wide scene, or "footprint," at a time. How many photons of outgoing longwave radiation (OLR) actually hit the sensor each second?

To figure this out, we'll first want to pick a representative wavelength for OLR. Using the spectrum in the image below, choose one such wavelength, and enter it into the `representative_wavelength` variable in the cell below. (Note that the x-axis does not show frequency $c/\lambda$ but the wavenumber $1/\lambda$. Engineers and scientists like the wavenumber because it comes in numbers that are easy to put our heads around.)
![Earth's blackbody spectrum](https://www.researchgate.net/profile/KW_Jucks/publication/11002997/figure/fig1/AS:601756643885074@1520481372523/IR-portion-of-the-Earth-model-spectrum-showing-the-blackbody-spectrum-of-the-Earths.png)_____no_output_____

indata = Dataset('Aqua_CERES_47min_subsampled_from_2021-11-02.nc')
ceres_lon = indata.variables['lon'][:]
ceres_lat = indata.variables['lat'][:]
ceres_irr = indata.variables['CERES_LW_TOA_flux___upwards'][:]
ceres_rad = indata.variables['CERES_LW_radiance___upwards'][:]

representative_wavelength = 1 # CHOOSE A GOOD VALUE based on the figure above_____no_output_____

Now fix the formula below by using what you learned from Petty #2.9. But be careful! (Should you use $\pi$, $2\pi$, or $4\pi$ in your denominator?)_____no_output_____

ceres_footprint = 10e3*np.pi # m^2
watts_at_earth = ceres_irr * ceres_footprint # watts per meter squared * square meters = watts
watts_at_ceres = watts_at_earth * (np.pi * ceres_aperture) / (2*np.pi*ceres_altitude**2.) # ENTER A GOOD FORMULA HERE
photons_at_ceres = watts_at_ceres * representative_wavelength / planck_constant / speed_of_light
plt.set_cmap('viridis')
median_photons = np.median(photons_at_ceres)
pho_colors = photons_at_ceres / np.percentile(photons_at_ceres, 90)
plt.scatter(ceres_lon, ceres_lat, c=pho_colors, s=3)
plt.title('photons hitting CERES detector (# s$^{-1}$) ')
plt.xlabel('$^\circ$lon')
plt.ylabel('$^\circ$lat')
plt.text(30, -76, "median {:.1e} photons".format(median_photons))_____no_output_____

## Written Response

How does the number of electrons compare to what you expected? Is this good or bad? What changes could be made to the instrument, do you think, based on the number of photons observed every second?_____no_output_____`Double-click this cell to use for your response`_____no_output_____# Problem 5 - Radiance vs Irradiance
In Petty #2.16, you proved a certain relationship between the radiance (which Petty calls intensity) of light coming from a surface, and the irradiance (which Petty and climate scientists often just call "flux") coming out of that surface.

Enter the coefficient from this relationship as the `best_guess_factor` below, and run the code to plot the Irradiance, Radiance, and the error of the conversion between the two using the constant factor from your homework._____no_output_____

best_guess_factor = 1 # use the value here from Petty #2.16
plt.set_cmap('inferno')
irr_colors = ceres_irr/np.percentile(ceres_irr, 90)
fig, ax = plt.subplots(1, 3, sharex=True, sharey=True)
ax[0].scatter(ceres_lon, ceres_lat, c=irr_colors, s=3)
ax[0].set_title('(a) irr (Wm$^{-2}$)')
ax[1].set_xlabel('$^\circ$lon')
ax[0].set_ylabel('$^\circ$lat')

rad_colors = ceres_rad/np.percentile(ceres_irr, 90)*best_guess_factor
ax[1].scatter(ceres_lon, ceres_lat, c=rad_colors, s=3)
ax[1].set_title('(b) rad (Wm$^{-2}$sr$^{-1}$)')

plt.set_cmap('RdBu_r')
dif_colors = rad_colors - irr_colors
ax[2].scatter(ceres_lon, ceres_lat, c=dif_colors, s=3)
ax[2].set_title('(c) error?')
ax[1].set_xlabel('$^\circ$lon')
ax[0].set_ylabel('$^\circ$lat')_____no_output_____

## Written Response
How does the accuracy of the irradiance-to-flux conversion from Petty #2.16 compare to what you expected?

Looking at the error plot, what do you think is the primary source of error in the textbook's conversion factor? (What's the faulty assumption?)

What evidence do you have for your claim? Explain._____no_output_____`Double-click this cell to use for your response`_____no_output_____# References
* G. Louis Smith, Gary L. Peterson, Robert Benjamin Lee III, Bruce R. Barkstrom, "Optical design of the CERES telescope," Proc. SPIE 4483, Earth Observing Systems VI, (18 January 2002); http://doi.org/10.1117/12.453463
* David J. Des Marais, Martin O. Harwit, Kenneth W. Jucks, James F. Kasting, Douglas N.C. Lin, Jonathan I. Lunine, Jean Schneider, Sara Seager, Wesley A. Traub, and Neville J. Woolf.Astrobiology.Jun 2002.153-181.http://doi.org/10.1089/15311070260192246
* V. I. Kostylev, "Scattering Fundamentals," Bistatic Radar Principles Practice, pp. 193–223, 2007.

# Acknowledgements
* Data downloaded from https://ceres-tool.larc.nasa.gov/ord-tool/
* Binder service by https://MyBinder.org_____no_output_____