# Notebook from rjenc29/numerical
Path: course/matplotlib/2. Plotting methods.ipynb

%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt_____no_output_____

# Visual Overview of Plotting Functions

We've talked a lot about laying things out, etc, but we haven't talked about actually plotting data yet. Matplotlib has a number of different plotting functions -- many more than we'll cover here, in fact. There's a more complete list in the pyplot documentation, and matplotlib gallery is a great place to get examples of all of them.

However, a full list and/or the gallery can be a bit overwhelming at first. Instead we'll condense it down and give you a look at some of the ones you're most likely to use, and then go over a subset of those in more detail.

Here's a simplified visual overview of matplotlib's most commonly used plot types.  Let's browse through these, and then we'll go over a few in more detail. Clicking on any of these images will take you to the code that generated them. We'll skip that for now, but feel browse through it later._____no_output_____## The Basics: 1D series/points
### What we've mentioned so far

### Other common plot types

## 2D Arrays and Images

## Vector Fields

## Data Distributions
_____no_output_____# Detailed Examples (of a few of these)_____no_output_____# Input Data: 1D Series

We've briefly mentioned `ax.plot(x, y)` and `ax.scatter(x, y)` to draw lines and points, respectively.  We'll cover some of their options (markers, colors, linestyles, etc) in the next section. Let's move on to a couple of other common plot types.

### Bar Plots: `ax.bar(...)` and `ax.barh(...)`

Bar plots are one of the most common plot types.  Matplotlib's `ax.bar(...)` method can also plot general rectangles, but the default is optimized for a simple sequence of x, y values, where the rectangles have a constant width.  There's also `ax.barh(...)` (for horizontal), which makes a constant-height assumption instead of a constant-width assumption._____no_output_____

np.random.seed(1)
x = np.arange(5)
y = np.random.randn(5)

fig, axes = plt.subplots(ncols=2, figsize=plt.figaspect(1./2))

vert_bars = axes[0].bar(x, y, color='lightblue', align='center')
horiz_bars = axes[1].barh(x, y, color='lightblue', align='center')

# I'll also introduce axhline & axvline to draw a line all the way across the axes
# This can be a quick-n-easy way to draw an axis "spine".
axes[0].axhline(0, color='gray', linewidth=2)
axes[1].axvline(0, color='gray', linewidth=2)

plt.show()_____no_output_____

Note that we held on to what `ax.bar(...)` returned.  Matplotlib plotting methods return an `Artist` or a sequence of artists.  Anything you can see in a matplotlib figure/axes/etc is an `Artist` of some sort. Most of the time, you will not need to retain these returned objects. You will want to capture them for special customizing that may not be possible through the normal plotting mechanism.

Let's re-visit that last example and modify what's plotted.  In the case of `bar`, a container artist is returned, so we'll modify its contents instead of the container itself (thus `for bar in vert_bars`)._____no_output_____

fig, ax = plt.subplots()
vert_bars = ax.bar(x, y, color='lightblue', align='center')

# We could have also done this with two separate calls to `ax.bar` and numpy boolean indexing.
for bar in vert_bars:
    if bar.xy[1] < 0:
        bar.set(edgecolor='darkred', color='salmon', linewidth=3)

plt.show()_____no_output_____

Keep in mind that any plotting method in matplotlib returns the artists that are plotted. We'll use it again, particularly when we get to adding colorbars to images._____no_output_____### Filled Regions: `ax.fill(x, y)`, `fill_between(...)`, etc

Of these functions, `ax.fill_between(...)` is probably the one you'll use the most often.  In its most basic form, it fills between the given y-values and 0:_____no_output_____

np.random.seed(1)
y = np.random.randn(100).cumsum()
x = np.linspace(0, 10, 100)

fig, ax = plt.subplots()
ax.fill_between(x, y, color='lightblue')
plt.show()_____no_output_____

However, it can also be used to fill between two curves.  This is particularly useful when you want to show an envelope of some sort (e.g. error, confidence, amplitude, etc)._____no_output_____

x = np.linspace(0, 10, 200)

y1 = 2 * x + 1
y2 = 3 * x + 1.2

y_mean = 0.5 * x * np.cos(2*x) + 2.5 * x + 1.1

fig, ax = plt.subplots()

# Plot the envelope with `fill_between`
ax.fill_between(x, y1, y2, color='yellow')

# Plot the "centerline" with `plot`
ax.plot(x, y_mean, color='black')

plt.show()_____no_output_____

# Exercise 2.1:

Now let's try combining `bar` and `fill_between`:

Can you reproduce the figure below?
_____no_output_____

import numpy as np
import matplotlib.pyplot as plt
np.random.seed(1)

# Generate data...
y_raw = np.random.randn(1000).cumsum() + 15
x_raw = np.linspace(0, 24, y_raw.size)

# Get averages of every 100 samples...
x_pos = x_raw.reshape(-1, 100).min(axis=1)
y_avg = y_raw.reshape(-1, 100).mean(axis=1)
y_err = y_raw.reshape(-1, 100).ptp(axis=1)

bar_width = x_pos[1] - x_pos[0]

# Make a made up future prediction with a fake confidence
x_pred = np.linspace(0, 30)
y_max_pred = y_avg[0] + y_err[0] + 2.3 * x_pred
y_min_pred = y_avg[0] - y_err[0] + 1.2 * x_pred

# Just so you don't have to guess at the colors...
barcolor, linecolor, fillcolor = 'wheat', 'salmon', 'lightblue'

# Now you're on your own!_____no_output_____

# Input Data: 2D Arrays or Images

There are several options for plotting 2D datasets.  `imshow`, `pcolor`, and `pcolormesh` have a lot of overlap, at first glance.  (The example image below is meant to clarify that somewhat.)

In short, `imshow` can interpolate and display large arrays very quickly, while `pcolormesh` and `pcolor` are much slower, but can handle flexible (i.e. more than just rectangular) arrangements of cells.

We won't dwell too much on the differences and overlaps here.  They have overlapping capabilities, but different default behavior because their primary use-cases are a bit different (there's also `matshow`, which is `imshow` with different defaults).

Instead we'll focus on what they have in common.

`imshow`, `pcolor`, `pcolormesh`, `scatter`, and any other matplotlib plotting methods that map a range of data values onto a colormap will return artists that are instances of `ScalarMappable.`  In practice, what that means is a) you can display a colorbar for them, and b) they share several keyword arguments._____no_output_____### Colorbars

Let's add a colorbar to the figure to display what colors correspond to values of `data` we've plotted. _____no_output_____

from matplotlib.cbook import get_sample_data
data = np.load(get_sample_data('axes_grid/bivariate_normal.npy'))

fig, ax = plt.subplots()
im = ax.imshow(data, cmap='gist_earth')
fig.colorbar(im)
plt.show()_____no_output_____

You may notice that `colorbar` is a `Figure` method and not an `Axes` method.  That's because `colorbar` doesn't operate on the axes. Instead, it shrinks the current axes by a bit, adds a _new_ axes to the figure, and places the colorbar on that axes.

The new axes that `fig.colorbar` creates is fairly limited in where it can be positioned.   For example, it's always outside the axes it "steals" room from. Sometimes you may want to avoid "stealing" room from an axes or maybe even have the colorbar _inside_ another axes.  In that case, you can manually create the axes for the colorbar and position it where you'd like:_____no_output_____

fig, ax = plt.subplots()
cax = fig.add_axes([0.27, 0.8, 0.5, 0.05])

im = ax.imshow(data, cmap='gist_earth')
fig.colorbar(im, cax=cax, orientation='horizontal')
plt.show()_____no_output_____

One note: In the last module in this tutorial, we'll briefly cover `axes_grid`, which is very useful for aligning colorbars and/or other axes with images displayed with `imshow`.  _____no_output_____  ### Shared parameters for `imshow`, `pcolormesh`, `contour`, `scatter`, etc

  As we mentioned earlier, any plotting method that creates a `ScalarMappable` will have some common kwargs.  The ones you'll use the most frequently are:

  * `cmap` : The colormap (or name of the colormap) used to display the input.  (We'll go over the different colormaps in the next section.)
  * `vmin` : The minimum data value that will correspond to the "bottom" of the colormap (defaults to the minimum of your input data).
  * `vmax` : The maximum data value that will correspond to the "top" of the colormap (defaults to the maximum of your input data).
  * `norm` : A `Normalize` instance to control how the data values are mapped to the colormap. By default, this will be a linear scaling between `vmin` and `vmax`, but other norms are available (e.g. `LogNorm`, `PowerNorm`, etc).

`vmin` and `vmax` are particularly useful.  Quite often, you'll want the colors to be mapped to a set range of data values, which aren't the min/max of your input data. For example, you might want a symmetric ranges of values around 0.

As an example of that, let's use a divergent colormap with the data we showed earlier. We'll also use `interpolation="nearest"` to "turn off" interpolation of the cells in the input dataset:_____no_output_____

from matplotlib.cbook import get_sample_data
data = np.load(get_sample_data('axes_grid/bivariate_normal.npy'))

fig, ax = plt.subplots()
im = ax.imshow(data, cmap='seismic', interpolation='nearest')
fig.colorbar(im)
plt.show()_____no_output_____

In this case, we'd really like the white in the colormap to correspond to 0.  A quick way to do this is to make the `vmin` equal to the negative of the `vmax`.  _____no_output_____

fig, ax = plt.subplots()
im = ax.imshow(data, cmap='seismic', interpolation='nearest',
               vmin=-2, vmax=2)
fig.colorbar(im)
plt.show()_____no_output_____

`vmin` and `vmax` are also very useful when we want multiple plots to share one colorbar, as our next exercise will do.

## Exercise 2.2:

Can you reproduce the figure below?
_____no_output_____

%load exercises/2.2-vmin_vmax_imshow_and_colorbars.py_____no_output_____import numpy as np
import matplotlib.pyplot as plt
np.random.seed(1)

# Generate random data with different ranges...
data1 = np.random.random((10, 10))
data2 = 2 * np.random.random((10, 10))
data3 = 3 * np.random.random((10, 10))

# Set up our figure and axes...
fig, axes = plt.subplots(ncols=3, figsize=plt.figaspect(0.5))
fig.tight_layout() # Make the subplots fill up the figure a bit more...
cax = fig.add_axes([0.25, 0.1, 0.55, 0.03]) # Add an axes for the colorbar

# Now you're on your own!_____no_output_____


# Notebook from Beilinson/course-content
Path: tutorials/W3D5_NetworkCausality/W3D5_Tutorial2.ipynb

   _____no_output_____# Tutorial 2: Correlations
**Week 3, Day 5: Network Causality**

**By Neuromatch Academy**

**Content creators**: Ari Benjamin, Tony Liu, Konrad Kording

**Content reviewers**: Mike X Cohen, Madineh Sarvestani, Yoni Friedman, Ella Batty, Michael Waskom
_____no_output_____**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

_____no_output_____---
# Tutorial objectives

*Estimated timing of tutorial: 45 min*

This is tutorial 2 on our day of examining causality. Below is the high level outline of what we'll cover today, with the sections we will focus on in this tutorial in bold:

1. Master definitions of causality
2. Understand that estimating causality is possible
3. Learn 4 different methods and understand when they fail
    1. perturbations
    2. **correlations**
    3. simultaneous fitting/regression
    5. instrumental variables

### Tutorial 2 objectives

In tutorial 1, we implemented and explored the dynamical system of neurons we will be working with throughout all of the tutorials today. We also learned about the "gold standard" of measuring causal effects through random perturbations. As random perturbations are often not possible, we will now turn to alternative methods to attempt to measure causality. We will:

- Learn how to estimate connectivity from observations assuming **correlations approximate causation**
- Show that this only works when the network is small

### Tutorial 2 setting

Often, we can't force neural activites or brain areas to be on or off. We just have to observe. Maybe we can get the correlation between two nodes -- is that good enough? The question we ask in this tutorial is **when is correlation a "good enough" substitute for causation?**

The answer is not "never", actually, but "sometimes"._____no_output_____

# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/gp4m9/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)_____no_output_____

---
# Setup_____no_output_____

# Imports

import numpy as np
import matplotlib.pyplot as plt_____no_output_____#@title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")_____no_output_____#@title Plotting Functions

def see_neurons(A, ax):
    """
    Visualizes the connectivity matrix.

    Args:
        A (np.ndarray): the connectivity matrix of shape (n_neurons, n_neurons)
        ax (plt.axis): the matplotlib axis to display on

    Returns:
        Nothing, but visualizes A.
    """
    A = A.T  # make up for opposite connectivity
    n = len(A)
    ax.set_aspect('equal')
    thetas = np.linspace(0, np.pi * 2, n, endpoint=False)
    x, y = np.cos(thetas), np.sin(thetas),
    ax.scatter(x, y, c='k', s=150)
    A = A / A.max()
    for i in range(n):
        for j in range(n):
            if A[i, j] > 0:
                ax.arrow(x[i], y[i], x[j] - x[i], y[j] - y[i], color='k', alpha=A[i, j], head_width=.15,
                        width = A[i,j] / 25, shape='right', length_includes_head=True)
    ax.axis('off')

def plot_estimation_quality_vs_n_neurons(number_of_neurons):
    """
    A wrapper function that calculates correlation between true and estimated connectivity
    matrices for each number of neurons and plots

    Args:
      number_of_neurons (list): list of different number of neurons for modeling system
      corr_func (function): Function for computing correlation
    """
    corr_data = np.zeros((n_trials, len(number_of_neurons)))
    for trial in range(n_trials):
      print("simulating trial {} of {}".format(trial + 1, n_trials))
      for j, size in enumerate(number_of_neurons):
          corr = get_sys_corr(size, timesteps, trial)
          corr_data[trial, j] = corr

    corr_mean = corr_data.mean(axis=0)
    corr_std = corr_data.std(axis=0)

    plt.plot(number_of_neurons, corr_mean)
    plt.fill_between(number_of_neurons,
                     corr_mean - corr_std,
                     corr_mean + corr_std,
                     alpha=.2)
    plt.xlabel("Number of neurons")
    plt.ylabel("Correlation")
    plt.title("Similarity between A and R as a function of network size")
    plt.show()

def plot_connectivity_matrix(A, ax=None):
  """Plot the (weighted) connectivity matrix A as a heatmap

    Args:
      A (ndarray): connectivity matrix (n_neurons by n_neurons)
      ax: axis on which to display connectivity matrix
  """
  if ax is None:
    ax = plt.gca()
  lim = np.abs(A).max()
  im = ax.imshow(A, vmin=-lim, vmax=lim, cmap="coolwarm")
  ax.tick_params(labelsize=10)
  ax.xaxis.label.set_size(15)
  ax.yaxis.label.set_size(15)
  cbar = ax.figure.colorbar(im, ax=ax, ticks=[0], shrink=.7)
  cbar.ax.set_ylabel("Connectivity Strength", rotation=90,
                     labelpad= 20, va="bottom")
  ax.set(xlabel="Connectivity from", ylabel="Connectivity to")

def plot_true_vs_estimated_connectivity(estimated_connectivity, true_connectivity, selected_neuron=None):
  """Visualize true vs estimated connectivity matrices

  Args:
    estimated_connectivity (ndarray): estimated connectivity (n_neurons by n_neurons)
    true_connectivity (ndarray): ground-truth connectivity (n_neurons by n_neurons)
    selected_neuron (int or None): None if plotting all connectivity, otherwise connectivity
      from selected_neuron will be shown

  """
  fig, axs = plt.subplots(1, 2, figsize=(10, 5))

  if selected_neuron is not None:
    plot_connectivity_matrix(np.expand_dims(estimated_connectivity, axis=1), ax=axs[0])
    plot_connectivity_matrix(true_connectivity[:, [selected_neuron]], ax=axs[1])
    axs[0].set_xticks([0])
    axs[1].set_xticks([0])
    axs[0].set_xticklabels([selected_neuron])
    axs[1].set_xticklabels([selected_neuron])
  else:
    plot_connectivity_matrix(estimated_connectivity, ax=axs[0])
    plot_connectivity_matrix(true_connectivity, ax=axs[1])

  axs[1].set(title="True connectivity")
  axs[0].set(title="Estimated connectivity")_____no_output_____# @title Helper Functions

def sigmoid(x):
    """
    Compute sigmoid nonlinearity element-wise on x.

    Args:
        x (np.ndarray): the numpy data array we want to transform
    Returns
        (np.ndarray): x with sigmoid nonlinearity applied
    """
    return 1 / (1 + np.exp(-x))

def create_connectivity(n_neurons, random_state=42, p=0.9):
    """
    Generate our nxn causal connectivity matrix.

    Args:
        n_neurons (int): the number of neurons in our system.
        random_state (int): random seed for reproducibility

    Returns:
        A (np.ndarray): our 0.1 sparse connectivity matrix
    """
    np.random.seed(random_state)
    A_0 = np.random.choice([0, 1], size=(n_neurons, n_neurons), p=[p, 1 - p])

    # set the timescale of the dynamical system to about 100 steps
    _, s_vals, _ = np.linalg.svd(A_0)
    A = A_0 / (1.01 * s_vals[0])

    # _, s_val_test, _ = np.linalg.svd(A)
    # assert s_val_test[0] < 1, "largest singular value >= 1"

    return A

def simulate_neurons(A, timesteps, random_state=42):
    """
    Simulates a dynamical system for the specified number of neurons and timesteps.

    Args:
        A (np.array): the connectivity matrix
        timesteps (int): the number of timesteps to simulate our system.
        random_state (int): random seed for reproducibility

    Returns:
        - X has shape (n_neurons, timeteps).
    """
    np.random.seed(random_state)

    n_neurons = len(A)
    X = np.zeros((n_neurons, timesteps))

    for t in range(timesteps - 1):
        # solution
        epsilon = np.random.multivariate_normal(np.zeros(n_neurons), np.eye(n_neurons))
        X[:, t + 1] = sigmoid(A.dot(X[:, t]) + epsilon)

        assert epsilon.shape == (n_neurons,)
    return X

def get_sys_corr(n_neurons, timesteps, random_state=42, neuron_idx=None):
    """
    A wrapper function for our correlation calculations between A and R.

    Args:
        n_neurons (int): the number of neurons in our system.
        timesteps (int): the number of timesteps to simulate our system.
        random_state (int): seed for reproducibility
        neuron_idx (int): optionally provide a neuron idx to slice out

    Returns:
        A single float correlation value representing the similarity between A and R
    """

    A = create_connectivity(n_neurons, random_state)
    X = simulate_neurons(A, timesteps)

    R = correlation_for_all_neurons(X)

    return np.corrcoef(A.flatten(), R.flatten())[0, 1]_____no_output_____

The helper functions defined above are:
- `sigmoid`: computes sigmoid nonlinearity element-wise on input, from Tutorial 1
- `create_connectivity`: generates nxn causal connectivity matrix., from Tutorial 1
- `simulate_neurons`: simulates a dynamical system for the specified number of neurons and timesteps, from Tutorial 1
- `get_sys_corr`: a wrapper function for correlation calculations between A and R._____no_output_____---
# Section 1: Small systems

_____no_output_____

# @title Video 1: Correlation vs causation
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Ak4y1m7kk", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="vjBO-S7KNPI", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____

## Coding Exercise 1: Try to approximate causation with correlation

In small systems, correlation can look like causation. Let's attempt to recover the true connectivity matrix (A) just by correlating the neural state at each timestep with the previous state: $C=\vec{x_t}{\vec{x_{t+1}}^T}$.

Complete this function to estimate the connectivity matrix of a single neuron by calculating the correlation coefficients with every other neuron at the next timestep. That is, please correlate two vectors: 1) the activity of a selected neuron at time $t$ 2) The activity of all other neurons at time $t+1$._____no_output_____

def compute_connectivity_from_single_neuron(X, selected_neuron):
    """
    Computes the connectivity matrix from a single neuron neurons using correlations

    Args:
        X (ndarray): the matrix of activities
        selected_neuron (int): the index of the selected neuron

    Returns:
        estimated_connectivity (ndarray): estimated connectivity for the selected neuron, of shape (n_neurons,)
    """

    # Extract the current activity of selected_neuron, t
    current_activity = X[selected_neuron, :-1]

    # Extract the observed outcomes of all the neurons
    next_activity = X[:, 1:]

    # Initialize estimated connectivity matrix
    estimated_connectivity = np.zeros(n_neurons)

    # Loop through all neurons
    for neuron_idx in range(n_neurons):

        # Get the activity of neuron_idx
        this_output_activity = next_activity[neuron_idx]

        ########################################################################
        ## TODO: Estimate the neural correlations between
        ## this_output_activity        and       current_activity
        ## -------------------                  ----------------
        ##
        ## Note that np.corrcoef returns the full correlation matrix; we want the
        ## top right corner, which we have already provided.
        ## FIll out function and remove
        raise NotImplementedError('Compute neural correlations')
        ########################################################################
        # Compute correlation
        correlation = np.corrcoef(...)[0, 1]

        # Store this neuron's correlation
        estimated_connectivity[neuron_idx] = correlation

    return estimated_connectivity

# Simulate a 6 neuron system for 5000 timesteps again.
n_neurons = 6
timesteps = 5000
selected_neuron = 1

# Invoke a helper function that generates our nxn causal connectivity matrix
A = create_connectivity(n_neurons)

# Invoke a helper function that simulates the neural activity
X = simulate_neurons(A, timesteps)

# Estimate connectivity
estimated_connectivity = compute_connectivity_from_single_neuron(X, selected_neuron)

# Visualize
plot_true_vs_estimated_connectivity(estimated_connectivity, A, selected_neuron)_____no_output_____# to_remove solution
def compute_connectivity_from_single_neuron(X, selected_neuron):
    """
    Computes the connectivity matrix from a single neuron neurons using correlations

    Args:
        X (ndarray): the matrix of activities
        selected_neuron (int): the index of the selected neuron

    Returns:
        estimated_connectivity (ndarray): estimated connectivity for the selected neuron, of shape (n_neurons,)
    """

    # Extract the current activity of selected_neuron, t
    current_activity = X[selected_neuron, :-1]

    # Extract the observed outcomes of all the neurons
    next_activity = X[:, 1:]

    # Initialize estimated connectivity matrix
    estimated_connectivity = np.zeros(n_neurons)

    # Loop through all neurons
    for neuron_idx in range(n_neurons):

        # Get the activity of neuron_idx
        this_output_activity = next_activity[neuron_idx]

        # Compute correlation
        correlation = np.corrcoef(this_output_activity, current_activity)[0, 1]

        # Store this neuron's correlation
        estimated_connectivity[neuron_idx] = correlation

    return estimated_connectivity

# Simulate a 6 neuron system for 5000 timesteps again.
n_neurons = 6
timesteps = 5000
selected_neuron = 1

# Invoke a helper function that generates our nxn causal connectivity matrix
A = create_connectivity(n_neurons)

# Invoke a helper function that simulates the neural activity
X = simulate_neurons(A, timesteps)

# Estimate connectivity
estimated_connectivity = compute_connectivity_from_single_neuron(X, selected_neuron)

# Visualize
with plt.xkcd():
  plot_true_vs_estimated_connectivity(estimated_connectivity, A, selected_neuron)_____no_output_____

Hopefully you saw that it pretty much worked. We wrote a function that does what you just did but in matrix form, so it's a little faster. It also does all neurons at the same time (helper function `correlation_for_all_neurons`)._____no_output_____

# @markdown Execute this cell get helper function `correlation_for_all_neurons`

def correlation_for_all_neurons(X):
  """Computes the connectivity matrix for the all neurons using correlations

    Args:
        X: the matrix of activities

    Returns:
        estimated_connectivity (np.ndarray): estimated connectivity for the selected neuron, of shape (n_neurons,)
  """
  n_neurons = len(X)
  S = np.concatenate([X[:, 1:], X[:, :-1]], axis=0)
  R = np.corrcoef(S)[:n_neurons, n_neurons:]
  return R_____no_output_____# @markdown Execute this cell to visualize full estimated vs true connectivity

R = correlation_for_all_neurons(X)

fig, axs = plt.subplots(1, 2, figsize=(10, 5))
see_neurons(A, axs[0])
plot_connectivity_matrix(A, ax=axs[1])
plt.suptitle("True connectivity matrix A")

fig, axs = plt.subplots(1, 2, figsize=(10, 5))
see_neurons(R, axs[0])
plot_connectivity_matrix(R, ax=axs[1])
plt.suptitle("Estimated connectivity matrix R");_____no_output_____

That pretty much worked too. Let's quantify how much it worked.

We'll calculate the correlation coefficient between the true connectivity and the actual connectivity;_____no_output_____

print("Correlation matrix of A and R:", np.corrcoef(A.flatten(), R.flatten())[0, 1])_____no_output_____

It *appears* in our system that correlation captures causality._____no_output_____

# @title Video 2: Correlation ~ causation for small systems
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1XZ4y1u7FR", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="eWLOnTUe9SM", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____

**Video correction**: the connectivity graph plots and associated explanations in this and other videos show the wrong direction of connectivity (the arrows should be pointing the opposite direction). This has been fixed in the figures above._____no_output_____---
# Section 2: Large systems

*Estimated timing to here from start of tutorial: 15 min*

As our system becomes more complex however, correlation fails to capture causality._____no_output_____## Section 2.1: Failure of correlation in complex systems

Let's jump to a much bigger system. Instead of 6 neurons, we will now use 100 neurons. How does the estimation quality of the connectivity matrix change? _____no_output_____

# @markdown Execute this cell to simulate large system, estimate connectivity matrix with correlation and return estimation quality

# Simulate a 100 neuron system for 5000 timesteps.
n_neurons = 100
timesteps = 5000
random_state = 42

A = create_connectivity(n_neurons, random_state)
X = simulate_neurons(A, timesteps)
R = correlation_for_all_neurons(X)

print("Correlation matrix of A and R:", np.corrcoef(A.flatten(), R.flatten())[0, 1])

fig, axs = plt.subplots(1, 2, figsize=(16, 8))
plot_connectivity_matrix(A, ax=axs[0])
axs[0].set_title("True connectivity matrix A")
plot_connectivity_matrix(R, ax=axs[1])
axs[1].set_title("Estimated connectivity matrix R");_____no_output_____# @title Video 3: Correlation vs causation in large systems
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1uC4y1b76C", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="U4sV-7g8T08", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____

## Section 2.2: Correlation as a function of network size

_____no_output_____### Interactive Demo 2.2.1: Connectivity estimation as a function of number of neurons

Instead of looking at just a few neurons (6) or a lot of neurons (100), as above, we will now systematically vary the number of neurons and plot the resulting changes in correlation coefficient between the true and estimated connectivity matrices. _____no_output_____

# @markdown Execute this cell to enable demo
@widgets.interact(n_neurons=(6, 42, 3))
def plot_corrs(n_neurons=6):
  fig, axs = plt.subplots(1, 3, figsize=(15, 5))
  timesteps = 2000
  random_state = 42
  A = create_connectivity(n_neurons, random_state)
  X = simulate_neurons(A, timesteps)
  R = correlation_for_all_neurons(X)
  corr = np.corrcoef(A.flatten(), R.flatten())[0, 1]
  plot_connectivity_matrix(A, ax=axs[0])
  plot_connectivity_matrix(R, ax=axs[1])
  axs[0].set_title("True connectivity")
  axs[1].set_title("Estimated connectivity")
  axs[2].text(0, 0.5, "Correlation : {:.2f}".format(corr), size=15)
  axs[2].axis('off')_____no_output_____

Of course there is some variability due to randomness in $A$. Let's average over a few trials and find the relationship._____no_output_____

# @markdown Execute this cell to plot connectivity estimation as a function of network size

n_trials = 5
timesteps = 1000  # shorter timesteps for faster running time
number_of_neurons = [5, 10, 25, 50, 100]
plot_estimation_quality_vs_n_neurons(number_of_neurons)_____no_output_____

### Interactive Demo 2.2.2: Connectivity estimation as a function of the sparsity of $A$

You may rightly wonder if correlation only fails for large systems for certain types of $A$. In this interactive demo, you can examine connectivity estimation as a function of the sparsity of $A$. Does connectivity estimation get better or worse with less sparsity?_____no_output_____

# @markdown Execute this cell to enable demo
@widgets.interact(sparsity=(0.01, 0.99, .01))
def plot_corrs(sparsity=0.9):
  fig, axs = plt.subplots(1, 3, figsize=(15, 5))
  timesteps = 2000
  random_state = 42
  n_neurons = 25
  A = create_connectivity(n_neurons, random_state, sparsity)
  X = simulate_neurons(A, timesteps)
  R = correlation_for_all_neurons(X)
  corr=np.corrcoef(A.flatten(), R.flatten())[0, 1]
  plot_connectivity_matrix(A, ax=axs[0])
  plot_connectivity_matrix(R, ax=axs[1])
  axs[0].set_title("True connectivity")
  axs[1].set_title("Estimated connectivity")
  axs[2].text(0, 0.5, "Correlation : {:.2f}".format(corr), size=15)
  axs[2].axis('off')_____no_output_____

---
# Section 3: Reflecting on causality

*Estimated timing to here from start of tutorial: 34 min*_____no_output_____ ## Think! 3: Reflecting on causality

Please discuss the following questions within groups for around 10 minutes.

* Think of a research paper you've written. Did it use previous causal knowledge (e.g. a mechanism), or ask a causal question? Try phrasing that causal relationship in the language of an intervention. ("*If I were to force $A$ to be $A'$, $B$ would...*")
* What methods for interventions exist in your area of neuroscience?
* Think about these common "filler" words. Do they imply a causal relationship, in its interventional definition? (*regulates, mediates, generates, modulates, shapes, underlies, produces, encodes, induces, enables, ensures, supports, promotes, determines*)
* What dimensionality would you (very roughly) estimate the brain to be? Would you expect correlations between neurons to give you their connectivity? Why?_____no_output_____---
# Summary

*Estimated timing of tutorial: 45 min*_____no_output_____

# @title Video 4: Summary
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1KK4y1x74w", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="hRyAN3yak_U", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____

Now for the takeaway. We know that for large systems correlation ≠ causation. But what about when we coarsely sample the large system? Do we get better at estimating the *effective* causal interaction between groups (=average of weights) from the correlation between the groups?

From our simulation above, the answer appears to be no: as the number of neurons per group increases, we don't see any significant increase in our ability to estimate the causal interaction between groups.

If you have time after completing all tutorials and are interested, please see Bonus Section 1 below for a brief discussion of correlation as a similarity metric and Bonus Section 2 to learn about causality and correlation in low resolution systems._____no_output_____---
# Bonus

_____no_output_____---
## Bonus Section 1: Correlation as similarity metric

We'd like to note here that though we make use of Pearson correlation coefficients throughout all of our tutorials to measure similarity between our estimated connectivity matrix $R$ and the ground truth connectivity $A$, this is not strictly correct usage of Pearson correlations as elements of $A$ are not normally distributed (they are in fact binary).

We use Pearson correlations as they are quick and easy to compute within the Numpy framework and provide qualitatively similar results to other correlation metrics. Other ways to compute similarities:
- [Spearman rank correlations](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient), which does not require normally distributed data
- dichotomizing our estimated matrix $R$ by the median and then running concordance analysis, such as computing [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa)

Another thing to consider: all we want is some measure of the similarity between $A$ and $R$. Element-wise comparisons are one way to do this, but are there other ways you can think of? What about matrix similarities?_____no_output_____---
## Bonus Section 2: Low resolution systems

A common situation in neuroscience is that you observe the *average* activity of large groups of neurons. (Think fMRI, EEG, LFP, etc.)
We're going to simulate this effect, and ask if correlations work to recover the average causal effect of groups of neurons or areas.

**Note on the quality of the analogy**: This is not intended as a perfect analogy of the brain or fMRI. Instead, we want to ask: *in a big system in which correlations fail to estimate causality, can you at least recover average connectivity between groups?*

**Some brainy differences to remember**:
We are assuming that the connectivity is random. In real brains, the neurons that are averaged have correlated input and output connectivities. This will improve the correspondence between correlations and causality for the average effect because the system has a lower true dimensionality. However, in real brains the system is also order of magnitudes larger than what we examine here, and the experimenter never has the fully-observed system.

## Simulate a large system

Execute the next cell to simulate a large system of 256 neurons for 10000 timesteps - it will take a bit of time to finish so move on as it runs._____no_output_____

# @markdown Execute this cell to simulate a large system

n_neurons = 256
timesteps = 10000
random_state = 42

A = create_connectivity(n_neurons, random_state)
X = simulate_neurons(A, timesteps)_____no_output_____

### Bonus Section 1.1: Coarsely sample the system

We don't observe this system. Instead, we observe the average activity of groups.
_____no_output_____#### Bonus Coding Exercise 1.1: Compute average activity across groups and compare resulting connectivity to the truth

Let's get a new matrix `coarse_X` that has 16 groups, each reflecting the average activity of 16 neurons (since there are 256 neurons in total).

We will then define the true coarse connectivity as the average of the neuronal connection strengths between groups. We'll compute the correlation between our coarsely sampled groups to estimate the connectivity and compare with the true connectivity._____no_output_____

def get_coarse_corr(n_groups, X):
    """
    A wrapper function for our correlation calculations between coarsely sampled
    A and R.

    Args:
        n_groups (int): the number of groups. should divide the number of neurons evenly
        X: the simulated system

    Returns:
        A single float correlation value representing the similarity between A and R
        ndarray: estimated connectivity matrix
        ndarray: true connectivity matrix
    """

    ############################################################################
    ## TODO: Insert your code here to get coarsely sampled X
    # Fill out function then remove
    raise NotImplementedError('Student exercise: please complete get_coarse_corr')
    ############################################################################
    coarse_X = ...

    # Make sure coarse_X is the right shape
    assert coarse_X.shape == (n_groups, timesteps)

    # Estimate connectivity from coarse system
    R = correlation_for_all_neurons(coarse_X)

    # Compute true coarse connectivity
    coarse_A = A.reshape(n_groups, n_neurons // n_groups, n_groups, n_neurons // n_groups).mean(3).mean(1)

    # Compute true vs estimated connectivity correlation
    corr = np.corrcoef(coarse_A.flatten(), R.flatten())[0, 1]

    return corr, R, coarse_A

n_groups = 16

# Call function
corr, R, coarse_A = get_coarse_corr(n_groups, X)

# Visualize
plot_true_vs_estimated_connectivity(R, coarse_A)_____no_output_____# to_remove solution

def get_coarse_corr(n_groups, X):
    """
    A wrapper function for our correlation calculations between coarsely sampled
    A and R.

    Args:
        n_groups (int): the number of groups. should divide the number of neurons evenly
        X: the simulated system

    Returns:
        A single float correlation value representing the similarity between A and R
        ndarray: estimated connectivity matrix
        ndarray: true connectivity matrix
    """

    coarse_X = X.reshape(n_groups, n_neurons // n_groups, timesteps).mean(1)

    # Make sure coarse_X is the right shape
    assert coarse_X.shape == (n_groups, timesteps)

    # Estimate connectivity from coarse system
    R = correlation_for_all_neurons(coarse_X)

    # Compute true coarse connectivity
    coarse_A = A.reshape(n_groups, n_neurons // n_groups, n_groups, n_neurons // n_groups).mean(3).mean(1)

    # Compute true vs estimated connectivity correlation
    corr = np.corrcoef(coarse_A.flatten(), R.flatten())[0, 1]

    return corr, R, coarse_A

n_groups = 16

# Call function
corr, R, coarse_A = get_coarse_corr(n_groups, X)

# Visualize
with plt.xkcd():
  plot_true_vs_estimated_connectivity(R, coarse_A)_____no_output_____

How close is the estimated coarse connectivity matrix to the truth?_____no_output_____

print("Correlation: {}".format(corr))_____no_output_____

We will now look at the estimation quality for different levels of coarseness when averaged over 3 trials._____no_output_____

#@markdown Execute this cell to visualize plot

n_neurons = 128
timesteps = 5000
n_trials = 3
groups = [2 ** i for i in range(2, int(np.log2(n_neurons)))]

corr_data = np.zeros((n_trials, len(groups)))

for trial in range(n_trials):
    print("Trial {} out of {}".format(trial, n_trials))
    A = create_connectivity(n_neurons, random_state=trial)
    X = simulate_neurons(A, timesteps, random_state=trial)
    for j, n_groups in enumerate(groups):
        corr_data[trial, j], _, _ = get_coarse_corr(n_groups, X)

corr_mean = corr_data.mean(axis=0)
corr_std = corr_data.std(axis=0)

plt.plot(np.divide(n_neurons, groups), corr_mean)
plt.fill_between(np.divide(n_neurons, groups),
                  corr_mean - corr_std,
                  corr_mean + corr_std,
                  alpha=.2)

plt.ylim([-0.2, 1])
plt.xlabel("Number of neurons per group ({} total neurons)".format(n_neurons),
        fontsize=15)
plt.ylabel("Correlation of estimated effective connectivity")
plt.title("Connectivity estimation performance vs coarseness of sampling")
plt.show()_____no_output_____

We know that for large systems correlation $\neq$ causation. Here, we have looked at what happens we coarsely sample a large system. Do we get better at estimating the *effective* causal interaction between groups (=average of weights) from the correlation between the groups?

From our simulation above, the answer appears to be no: as the number of neurons per group increases, we don't see any significant increase in our ability to estimate the causal interaction between groups._____no_output_____


# Notebook from gmstanle/msc_aging
Path: 2_aging_signature/archive_initial_submission/.ipynb_checkpoints/DE_tissue_cell_FACS-checkpoint.ipynb

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style="whitegrid")
import numpy as np
import scanpy.api as sc
from anndata import read_h5ad
from anndata import AnnData
import scipy as sp
import scipy.stats
from gprofiler import GProfiler
import pickle
# Other specific functions
from itertools import product
from statsmodels.stats.multitest import multipletests
import util
# R related packages
import rpy2.rinterface_lib.callbacks
import logging
from rpy2.robjects import pandas2ri
import anndata2ri/home/martin/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)
# Ignore R warning messages
#Note: this can be commented out to get more verbose R output
rpy2.rinterface_lib.callbacks.logger.setLevel(logging.ERROR)
# Automatically convert rpy2 outputs to pandas dataframes
pandas2ri.activate()
anndata2ri.activate()
%load_ext rpy2.ipython
# autoreload
%load_ext autoreload
%autoreload 2
# logging
sc.logging.print_versions()scanpy==1.4.3 anndata==0.6.20 umap==0.3.8 numpy==1.16.4 scipy==1.2.1 pandas==0.25.0 scikit-learn==0.21.1 statsmodels==0.9.0 python-igraph==0.7.1 louvain==0.6.1
%%R
library(MAST)_____no_output_____

## Load data_____no_output_____

# Data path
data_path = '/data3/martin/tms_gene_data'
output_folder = data_path + '/DE_result'_____no_output_____# Load the data
adata_combine = util.load_normalized_data(data_path)_____no_output_____temp_facs = adata_combine[adata_combine.obs['b_method']=='facs',]
temp_droplet = adata_combine[adata_combine.obs['b_method']=='droplet',]_____no_output_____

## Generate a list of tissue-cell types for DE testing_____no_output_____

cell_type_list = list(set(temp_facs.obs['cell_ontology_class']))
tissue_list = list(set(temp_facs.obs['tissue']))
min_cell_number = 25
analysis_list = []
analysis_info = {}
# for cell_type in cell_type_list:
for tissue,cell_type in product(tissue_list, cell_type_list):
    analyte = '%s.%s'%(tissue,cell_type)
    ind_select = (temp_facs.obs['cell_ontology_class'] == cell_type) & \
                    (temp_facs.obs['tissue'] == tissue)
    n_young = (temp_facs.obs['age'][ind_select].isin(['1m', '3m'])).sum()
    n_old = (temp_facs.obs['age'][ind_select].isin(['18m', '21m',
                                                   '24m', '30m'])).sum()
    analysis_info[analyte] = {}
    analysis_info[analyte]['n_young'] = n_young
    analysis_info[analyte]['n_old'] = n_old
    if (n_young>min_cell_number) & (n_old>min_cell_number) & (cell_type!='nan'):
        print('%s, n_young=%d, n_old=%d'%(analyte, n_young, n_old))
        analysis_list.append(analyte)Pancreas.pancreatic ductal cell, n_young=163, n_old=150
Pancreas.pancreatic D cell, n_young=123, n_old=68
Pancreas.pancreatic B cell, n_young=522, n_old=820
Pancreas.leukocyte, n_young=54, n_old=66
Pancreas.pancreatic A cell, n_young=364, n_old=157
Pancreas.endothelial cell, n_young=86, n_old=116
Pancreas.pancreatic acinar cell, n_young=191, n_old=385
Brain_Myeloid.macrophage, n_young=44, n_old=243
Brain_Myeloid.microglial cell, n_young=4488, n_old=8642
Skin.basal cell of epidermis, n_young=678, n_old=994
Skin.epidermal cell, n_young=260, n_old=232
Skin.bulge keratinocyte, n_young=1370, n_old=1234
Marrow.precursor B cell, n_young=489, n_old=367
Marrow.late pro-B cell, n_young=276, n_old=135
Marrow.CD4-positive, alpha-beta T cell, n_young=29, n_old=251
Marrow.macrophage, n_young=182, n_old=255
Marrow.basophil, n_young=29, n_old=42
Marrow.mature alpha-beta T cell, n_young=99, n_old=231
Marrow.early pro-B cell, n_young=29, n_old=43
Marrow.granulocyte, n_young=742, n_old=2150
Marrow.NK cell, n_young=158, n_old=315
Marrow.naive B cell, n_young=675, n_old=2284
Marrow.promonocyte, n_young=343, n_old=672
Marrow.granulocyte monocyte progenitor cell, n_young=70, n_old=180
Marrow.immature B cell, n_young=383, n_old=206
Marrow.hematopoietic stem cell, n_young=1363, n_old=1943
Marrow.granulocytopoietic cell, n_young=197, n_old=286
Brain_Non-Myeloid.oligodendrocyte precursor cell, n_young=206, n_old=106
Brain_Non-Myeloid.oligodendrocyte, n_young=1396, n_old=698
Brain_Non-Myeloid.brain pericyte, n_young=179, n_old=305
Brain_Non-Myeloid.endothelial cell, n_young=723, n_old=1509
Brain_Non-Myeloid.interneuron, n_young=83, n_old=157
Brain_Non-Myeloid.neuron, n_young=154, n_old=330
Brain_Non-Myeloid.astrocyte, n_young=454, n_old=138
Trachea.T cell, n_young=61, n_old=222
Trachea.fibroblast, n_young=563, n_old=673
Trachea.macrophage, n_young=137, n_old=293
Trachea.endothelial cell, n_young=116, n_old=125
Trachea.basal epithelial cell of tracheobronchial tree, n_young=83, n_old=321
Trachea.smooth muscle cell of trachea, n_young=78, n_old=47
Diaphragm.mesenchymal stem cell, n_young=243, n_old=251
Diaphragm.skeletal muscle satellite cell, n_young=441, n_old=451
Diaphragm.macrophage, n_young=31, n_old=45
Diaphragm.B cell, n_young=75, n_old=48
Diaphragm.endothelial cell, n_young=92, n_old=126
Spleen.CD4-positive, alpha-beta T cell, n_young=197, n_old=217
Spleen.NK cell, n_young=51, n_old=36
Spleen.proerythroblast, n_young=44, n_old=63
Spleen.B cell, n_young=1248, n_old=1603
Spleen.CD8-positive, alpha-beta T cell, n_young=153, n_old=201
Lung.neutrophil, n_young=29, n_old=94
Lung.endothelial cell of lymphatic vessel, n_young=51, n_old=48
Lung.type II pneumocyte, n_young=96, n_old=71
Lung.fibroblast of lung, n_young=129, n_old=354
Lung.smooth muscle cell of the pulmonary artery, n_young=55, n_old=56
Lung.vein endothelial cell, n_young=39, n_old=113
Lung.adventitial cell, n_young=140, n_old=149
Lung.myeloid dendritic cell, n_young=52, n_old=321
Lung.B cell, n_young=55, n_old=221
Lung.non-classical monocyte, n_young=56, n_old=93
Lung.bronchial smooth muscle cell, n_young=420, n_old=1697
Lung.classical monocyte, n_young=68, n_old=134
Tongue.basal cell of epidermis, n_young=1070, n_old=1472
Tongue.keratinocyte, n_young=348, n_old=845
Liver.mature NK T cell, n_young=31, n_old=145
Liver.endothelial cell of hepatic sinusoid, n_young=189, n_old=428
Liver.B cell, n_young=34, n_old=176
Liver.hepatocyte, n_young=401, n_old=761
Liver.Kupffer cell, n_young=47, n_old=215
Kidney.kidney loop of Henle ascending limb epithelial cell, n_young=75, n_old=71
Kidney.macrophage, n_young=42, n_old=110
Kidney.epithelial cell of proximal tubule, n_young=46, n_old=347
Kidney.kidney collecting duct epithelial cell, n_young=124, n_old=321
Kidney.kidney collecting duct principal cell, n_young=55, n_old=102
Kidney.mesangial cell, n_young=32, n_old=61
Kidney.fenestrated cell, n_young=113, n_old=155
Thymus.thymocyte, n_young=734, n_old=891
Thymus.DN4 thymocyte, n_young=581, n_old=1175
Bladder.bladder cell, n_young=650, n_old=362
Bladder.bladder urothelial cell, n_young=656, n_old=764
BAT.T cell, n_young=80, n_old=75
BAT.B cell, n_young=198, n_old=331
BAT.endothelial cell, n_young=166, n_old=247
BAT.mesenchymal stem cell of adipose, n_young=82, n_old=123
BAT.myeloid cell, n_young=168, n_old=592
GAT.T cell, n_young=26, n_old=58
GAT.B cell, n_young=49, n_old=267
GAT.endothelial cell, n_young=160, n_old=181
GAT.mesenchymal stem cell of adipose, n_young=689, n_old=873
GAT.myeloid cell, n_young=461, n_old=257
SCAT.epithelial cell, n_young=56, n_old=124
SCAT.B cell, n_young=178, n_old=145
SCAT.endothelial cell, n_young=246, n_old=243
SCAT.mesenchymal stem cell of adipose, n_young=737, n_old=833
SCAT.myeloid cell, n_young=359, n_old=498
Large_Intestine.intestinal crypt stem cell, n_young=853, n_old=726
Large_Intestine.large intestine goblet cell, n_young=692, n_old=895
Large_Intestine.enterocyte of epithelium of large intestine, n_young=1176, n_old=1366
Large_Intestine.epithelial cell of large intestine, n_young=878, n_old=826
Large_Intestine.secretory cell, n_young=256, n_old=307
Large_Intestine.enteroendocrine cell, n_young=65, n_old=96
Large_Intestine.Brush cell of epithelium proper of large intestine, n_young=67, n_old=108
Aorta.fibroblast of cardiac tissue, n_young=66, n_old=149
Aorta.aortic endothelial cell, n_young=189, n_old=278
Aorta.professional antigen presenting cell, n_young=56, n_old=74
Mammary_Gland.luminal epithelial cell of mammary gland, n_young=452, n_old=148
Mammary_Gland.basal cell, n_young=1186, n_old=426
Mammary_Gland.stromal cell, n_young=262, n_old=53
Heart.fibroblast of cardiac tissue, n_young=2118, n_old=1593
Heart.smooth muscle cell, n_young=65, n_old=38
Heart.T cell, n_young=37, n_old=160
Heart.valve cell, n_young=158, n_old=157
Heart.atrial myocyte, n_young=159, n_old=244
Heart.endocardial cell, n_young=164, n_old=211
Heart.ventricular myocyte, n_young=46, n_old=101
Heart.endothelial cell of coronary artery, n_young=1179, n_old=1515
Heart.monocyte, n_young=490, n_old=791
MAT.CD4-positive, alpha-beta T cell, n_young=65, n_old=110
MAT.T cell, n_young=27, n_old=38
MAT.macrophage, n_young=119, n_old=109
MAT.B cell, n_young=116, n_old=254
MAT.endothelial cell, n_young=117, n_old=136
MAT.CD8-positive, alpha-beta T cell, n_young=38, n_old=171
MAT.mesenchymal stem cell of adipose, n_young=601, n_old=795
MAT.myeloid cell, n_young=74, n_old=147
Limb_Muscle.mesenchymal stem cell, n_young=264, n_old=671
Limb_Muscle.T cell, n_young=38, n_old=98
Limb_Muscle.skeletal muscle satellite cell, n_young=535, n_old=1298
Limb_Muscle.macrophage, n_young=54, n_old=257
Limb_Muscle.B cell, n_young=70, n_old=117
Limb_Muscle.endothelial cell, n_young=141, n_old=312

### DE using R package MAST _____no_output_____

## DE testing
gene_name_list = np.array(temp_facs.var_names)
DE_result_MAST = {}
for i_analyte,analyte in enumerate(analysis_list):
    print(analyte, '%d/%d'%(i_analyte, len(analysis_list)))
    tissue,cell_type = analyte.split('.')
    ind_select = (temp_facs.obs['cell_ontology_class'] == cell_type) & \
                    (temp_facs.obs['tissue'] == tissue)
    adata_temp = temp_facs[ind_select,]
    # reformatting
    adata_temp.X = np.array(adata_temp.X.todense())
    adata_temp.obs['condition'] = [int(x[:-1]) for x in adata_temp.obs['age']]
    adata_temp.obs = adata_temp.obs[['condition', 'sex']]
    if len(set(adata_temp.obs['sex'])) <2:
        covariate = ''
    else:
        covariate = '+sex'
#     # toy example
#     covariate = ''
#     np.random.seed(0)
#     ind_select = np.random.permutation(adata_temp.shape[0])[0:100]
#     ind_select = np.sort(ind_select)
#     adata_temp = adata_temp[ind_select, 0:3]
#     adata_temp.X[:,0] = (adata_temp.obs['sex'] == 'male')*3
#     adata_temp.X[:,1] = (adata_temp.obs['condition'])*3
    # DE using MAST
    R_cmd = util.call_MAST_age()
    get_ipython().run_cell_magic(u'R', u'-i adata_temp -i covariate -o de_res', R_cmd)
    de_res.columns = ['gene', 'raw-p', 'coef', 'bh-p']
    de_res.index = de_res['gene']
    DE_result_MAST[analyte] = pd.DataFrame(index = gene_name_list)
    DE_result_MAST[analyte] = DE_result_MAST[analyte].join(de_res)
    # fc between yound and old
    X = adata_temp.X
    y = (adata_temp.obs['condition']>10)
    DE_result_MAST[analyte]['fc'] = X[y,:].mean(axis=0) - X[~y,:].mean(axis=0)
#     breakPancreas.pancreatic ductal cell 0/131
Pancreas.pancreatic D cell 1/131
Pancreas.pancreatic B cell 2/131
Pancreas.leukocyte 3/131
Pancreas.pancreatic A cell 4/131
Pancreas.endothelial cell 5/131
Pancreas.pancreatic acinar cell 6/131
Brain_Myeloid.macrophage 7/131
Brain_Myeloid.microglial cell 8/131
Skin.basal cell of epidermis 9/131
Skin.epidermal cell 10/131
Skin.bulge keratinocyte 11/131
Marrow.precursor B cell 12/131
Marrow.late pro-B cell 13/131
Marrow.CD4-positive, alpha-beta T cell 14/131
Marrow.macrophage 15/131
Marrow.basophil 16/131
Marrow.mature alpha-beta T cell 17/131
Marrow.early pro-B cell 18/131
Marrow.granulocyte 19/131
Marrow.NK cell 20/131
Marrow.naive B cell 21/131
Marrow.promonocyte 22/131
Marrow.granulocyte monocyte progenitor cell 23/131
Marrow.immature B cell 24/131
Marrow.hematopoietic stem cell 25/131
Marrow.granulocytopoietic cell 26/131
Brain_Non-Myeloid.oligodendrocyte precursor cell 27/131
Brain_Non-Myeloid.oligodendrocyte 28/131
Brain_Non-Myeloid.brain pericyte 29/131
Brain_Non-Myeloid.endothelial cell 30/131
Brain_Non-Myeloid.interneuron 31/131
Brain_Non-Myeloid.neuron 32/131
Brain_Non-Myeloid.astrocyte 33/131
Trachea.T cell 34/131
Trachea.fibroblast 35/131
Trachea.macrophage 36/131
Trachea.endothelial cell 37/131
Trachea.basal epithelial cell of tracheobronchial tree 38/131
Trachea.smooth muscle cell of trachea 39/131
Diaphragm.mesenchymal stem cell 40/131
Diaphragm.skeletal muscle satellite cell 41/131
Diaphragm.macrophage 42/131
Diaphragm.B cell 43/131
Diaphragm.endothelial cell 44/131
Spleen.CD4-positive, alpha-beta T cell 45/131
Spleen.NK cell 46/131
Spleen.proerythroblast 47/131
Spleen.B cell 48/131
Spleen.CD8-positive, alpha-beta T cell 49/131
Lung.neutrophil 50/131
Lung.endothelial cell of lymphatic vessel 51/131
Lung.type II pneumocyte 52/131
Lung.fibroblast of lung 53/131
Lung.smooth muscle cell of the pulmonary artery 54/131
Lung.vein endothelial cell 55/131
Lung.adventitial cell 56/131
Lung.myeloid dendritic cell 57/131
Lung.B cell 58/131
Lung.non-classical monocyte 59/131
Lung.bronchial smooth muscle cell 60/131
Lung.classical monocyte 61/131
Tongue.basal cell of epidermis 62/131
Tongue.keratinocyte 63/131
Liver.mature NK T cell 64/131
Liver.endothelial cell of hepatic sinusoid 65/131
Liver.B cell 66/131
Liver.hepatocyte 67/131
Liver.Kupffer cell 68/131
Kidney.kidney loop of Henle ascending limb epithelial cell 69/131
Kidney.macrophage 70/131
Kidney.epithelial cell of proximal tubule 71/131
Kidney.kidney collecting duct epithelial cell 72/131
Kidney.kidney collecting duct principal cell 73/131
Kidney.mesangial cell 74/131
Kidney.fenestrated cell 75/131
Thymus.thymocyte 76/131
Thymus.DN4 thymocyte 77/131
Bladder.bladder cell 78/131
Bladder.bladder urothelial cell 79/131
BAT.T cell 80/131
BAT.B cell 81/131
BAT.endothelial cell 82/131
BAT.mesenchymal stem cell of adipose 83/131
BAT.myeloid cell 84/131
GAT.T cell 85/131
GAT.B cell 86/131
GAT.endothelial cell 87/131
GAT.mesenchymal stem cell of adipose 88/131
GAT.myeloid cell 89/131
SCAT.epithelial cell 90/131
SCAT.B cell 91/131
SCAT.endothelial cell 92/131
SCAT.mesenchymal stem cell of adipose 93/131
SCAT.myeloid cell 94/131
Large_Intestine.intestinal crypt stem cell 95/131
Large_Intestine.large intestine goblet cell 96/131
Large_Intestine.enterocyte of epithelium of large intestine 97/131
Large_Intestine.epithelial cell of large intestine 98/131
Large_Intestine.secretory cell 99/131
Large_Intestine.enteroendocrine cell 100/131
Large_Intestine.Brush cell of epithelium proper of large intestine 101/131
Aorta.fibroblast of cardiac tissue 102/131
Aorta.aortic endothelial cell 103/131
Aorta.professional antigen presenting cell 104/131
Mammary_Gland.luminal epithelial cell of mammary gland 105/131
Mammary_Gland.basal cell 106/131
Mammary_Gland.stromal cell 107/131
Heart.fibroblast of cardiac tissue 108/131
Heart.smooth muscle cell 109/131
Heart.T cell 110/131
Heart.valve cell 111/131
Heart.atrial myocyte 112/131
Heart.endocardial cell 113/131
Heart.ventricular myocyte 114/131
Heart.endothelial cell of coronary artery 115/131
Heart.monocyte 116/131
MAT.CD4-positive, alpha-beta T cell 117/131
MAT.T cell 118/131
MAT.macrophage 119/131
MAT.B cell 120/131
MAT.endothelial cell 121/131
MAT.CD8-positive, alpha-beta T cell 122/131
MAT.mesenchymal stem cell of adipose 123/131
MAT.myeloid cell 124/131
Limb_Muscle.mesenchymal stem cell 125/131
Limb_Muscle.T cell 126/131
Limb_Muscle.skeletal muscle satellite cell 127/131
Limb_Muscle.macrophage 128/131
Limb_Muscle.B cell 129/131
Limb_Muscle.endothelial cell 130/131

### Save DE results_____no_output_____

with open(output_folder+'/DE_tissue_cell_FACS.pickle', 'wb') as handle:
    pickle.dump(DE_result_MAST, handle)
    pickle.dump(analysis_list, handle)
    pickle.dump(analysis_info, handle)_____no_output_____

### Validation_____no_output_____

# Load DE result
with open(output_folder+'_old/DE.pickle', 'rb') as handle:
    DE_result_MAST_temp = pickle.load(handle)
    analysis_list_temp = pickle.load(handle)
for analyte in analysis_list:
    if analyte in analysis_list_temp:
        bh_p = DE_result_MAST[analyte]['bh-p']
        bh_p_temp = DE_result_MAST_temp[analyte]['bh-p']
        print('%s, New:%d, Old:%d, Overlap:%d'%(analyte, np.sum(bh_p<0.01),
                                            np.sum(bh_p_temp<0.01),
                                            np.sum((bh_p<0.01) & (bh_p_temp<0.01))))Trachea.endothelial cell, New:2437, Old:2312, Overlap:2177
Trachea.fibroblast, New:9847, Old:8395, Overlap:8242
Trachea.basal epithelial cell of tracheobronchial tree, New:1539, Old:1895, Overlap:1169
Trachea.macrophage, New:2225, Old:1221, Overlap:1159
Lung.non-classical monocyte, New:83, Old:86, Overlap:65
Lung.lymphatic endothelial cell, New:54, Old:75, Overlap:54
Lung.classical monocyte, New:121, Old:107, Overlap:72
Lung.fibroblast of lung, New:717, Old:478, Overlap:384
Lung.dendritic cell, New:1710, Old:1900, Overlap:1188
Lung.mature natural killer T cell, New:1513, Old:3384, Overlap:1385
Lung.capillary endothelial cell, New:5690, Old:6824, Overlap:5079
Pancreas.pancreatic acinar cell, New:1396, Old:244, Overlap:223
Pancreas.endothelial cell, New:120, Old:97, Overlap:83
Pancreas.pancreatic D cell, New:1332, Old:715, Overlap:650
Pancreas.pancreatic A cell, New:5651, Old:2568, Overlap:2482
Pancreas.leukocyte, New:482, Old:48, Overlap:48
Pancreas.pancreatic ductal cell, New:3781, Old:3219, Overlap:2990
Pancreas.pancreatic B cell, New:11813, Old:9561, Overlap:9468
Kidney.kidney loop of Henle ascending limb epithelial cell, New:66, Old:55, Overlap:48
Kidney.fenestrated cell, New:319, Old:177, Overlap:117
Kidney.kidney collecting duct principal cell, New:47, Old:3, Overlap:3
Kidney.macrophage, New:30, Old:35, Overlap:24
Kidney.kidney collecting duct epithelial cell, New:150, Old:113, Overlap:92
Kidney.epithelial cell of proximal tubule, New:302, Old:308, Overlap:204
Heart.fibroblast of cardiac tissue, New:2920, Old:5052, Overlap:2705
Heart.leukocyte, New:3616, Old:3213, Overlap:3080
Heart.endocardial cell, New:60, Old:21, Overlap:13
Heart.valve cell, New:93, Old:98, Overlap:77
Heart.endothelial cell of coronary artery, New:269, Old:233, Overlap:194
Bladder.bladder cell, New:4118, Old:6069, Overlap:4074
Bladder.bladder urothelial cell, New:10536, Old:10756, Overlap:10187
Brain_Myeloid.microglial cell, New:12161, Old:11629, Overlap:11183
Brain_Myeloid.macrophage, New:245, Old:181, Overlap:141
Limb_Muscle.endothelial cell, New:2548, Old:2137, Overlap:1889
Limb_Muscle.B cell, New:1257, Old:1413, Overlap:1081
Limb_Muscle.mesenchymal stem cell, New:9737, Old:9819, Overlap:9297
Limb_Muscle.skeletal muscle satellite cell, New:9507, Old:9398, Overlap:8905
Limb_Muscle.macrophage, New:100, Old:72, Overlap:49
Limb_Muscle.T cell, New:632, Old:674, Overlap:553
Thymus.thymocyte, New:8429, Old:7320, Overlap:7114
Thymus.DN4 thymocyte, New:4124, Old:3830, Overlap:2949
Skin.basal cell of epidermis, New:9372, Old:9252, Overlap:8762
Skin.epidermal cell, New:5383, Old:4844, Overlap:4487
Skin.keratinocyte stem cell, New:11477, Old:11526, Overlap:11238
Fat.endothelial cell, New:7416, Old:8220, Overlap:7273
Fat.epithelial cell, New:6230, Old:5567, Overlap:4889
Fat.B cell, New:3171, Old:3370, Overlap:2417
Fat.mesenchymal stem cell of adipose, New:12732, Old:12900, Overlap:12310
Fat.T cell, New:4922, Old:5468, Overlap:4445
Fat.myeloid cell, New:10606, Old:10278, Overlap:9512
Fat.NK cell, New:26, Old:16, Overlap:9
Marrow.precursor B cell, New:2091, Old:2463, Overlap:1909
Marrow.immature B cell, New:6659, Old:5986, Overlap:5835
Marrow.granulocyte, New:11933, Old:11866, Overlap:11695
Marrow.MPP Fraction C, New:2199, Old:1730, Overlap:1658
Marrow.granulocytopoietic cell, New:7005, Old:7140, Overlap:6869
Marrow.Marrow Cd4 T cell, New:1846, Old:1830, Overlap:1577
Marrow.HSC enriched fraction, New:7496, Old:7269, Overlap:7078
Marrow.nan-marrow-needs-subclustering, New:6004, Old:6491, Overlap:5245
Marrow.Cd4+ macrophage, New:1088, Old:570, Overlap:539
Marrow.naive B cell, New:5902, Old:5675, Overlap:5121
Marrow.late pro-B cell, New:1910, Old:1758, Overlap:1660
Marrow.NK cell, New:6616, Old:6168, Overlap:5992
Spleen.B cell, New:5400, Old:5405, Overlap:4650
Spleen.T cell, New:980, Old:710, Overlap:605
Spleen.NK cell, New:259, Old:167, Overlap:161
Spleen.proerythroblast, New:117, Old:52, Overlap:51
Brain_Non-Myeloid.endothelial cell, New:3485, Old:5042, Overlap:3345
Brain_Non-Myeloid.oligodendrocyte, New:8865, Old:8979, Overlap:8809
Brain_Non-Myeloid.brain pericyte, New:4654, Old:4995, Overlap:4558
Brain_Non-Myeloid.astrocyte, New:1318, Old:1992, Overlap:1309
Brain_Non-Myeloid.neuron, New:10287, Old:10211, Overlap:10028
Brain_Non-Myeloid.interneuron, New:163, Old:172, Overlap:135
Large_Intestine.enterocyte of epithelium of large intestine, New:10880, Old:10057, Overlap:9273
Large_Intestine.secretory cell, New:7897, Old:8122, Overlap:6903
Large_Intestine.large intestine goblet cell, New:4456, Old:1452, Overlap:1076
Large_Intestine.epithelial cell of large intestine, New:6113, Old:6069, Overlap:5472
Large_Intestine.enteroendocrine cell, New:9, Old:28, Overlap:4
Large_Intestine.intestinal crypt stem cell, New:10511, Old:9276, Overlap:9014
Large_Intestine.Brush cell of epithelium proper of large intestine, New:215, Old:17, Overlap:17
Liver.B cell, New:35, Old:35, Overlap:35
Liver.hepatocyte, New:7297, Old:4665, Overlap:4532
Liver.endothelial cell of hepatic sinusoid, New:465, Old:465, Overlap:465
Liver.NK cell, New:562, Old:562, Overlap:562