# Notebook from bentrevett/py-algorithms
Path: advent-of-code-2018/1.ipynb

--- Day 1: Chronal Calibration ---

"We've detected some temporal anomalies," one of Santa's Elves at the Temporal Anomaly Research and Detection Instrument Station tells you. She sounded pretty worried when she called you down here. "At 500-year intervals into the past, someone has been changing Santa's history!"

"The good news is that the changes won't propagate to our time stream for another 25 days, and we have a device" - she attaches something to your wrist - "that will let you fix the changes with no such propagation delay. It's configured to send you 500 years further into the past every few days; that was the best we could do on such short notice."

"The bad news is that we are detecting roughly fifty anomalies throughout time; the device will indicate fixed anomalies with stars. The other bad news is that we only have one device and you're the best person for the job! Good lu--" She taps a button on the device and you suddenly feel like you're falling. To save Christmas, you need to get all fifty stars by December 25th.

Collect stars by solving puzzles. Two puzzles will be made available on each day in the advent calendar; the second puzzle is unlocked when you complete the first. Each puzzle grants one star. Good luck!

After feeling like you've been falling for a few minutes, you look at the device's tiny screen. "Error: Device must be calibrated before first use. Frequency drift detected. Cannot maintain destination lock." Below the message, the device shows a sequence of changes in frequency (your puzzle input). A value like +6 means the current frequency increases by 6; a value like -3 means the current frequency decreases by 3.

For example, if the device displays frequency changes of +1, -2, +3, +1, then starting from a frequency of zero, the following changes would occur:

    Current frequency  0, change of +1; resulting frequency  1.
    Current frequency  1, change of -2; resulting frequency -1.
    Current frequency -1, change of +3; resulting frequency  2.
    Current frequency  2, change of +1; resulting frequency  3.

In this example, the resulting frequency is 3.

Here are other example situations:

    +1, +1, +1 results in  3
    +1, +1, -2 results in  0
    -1, -2, -3 results in -6

Starting with a frequency of zero, what is the resulting frequency after all of the changes in frequency have been applied?
_____no_output_____

#read file
with open('1.txt', 'r') as f:
    changes = f.readlines()

#get rid of newlines and convert to int
changes = [int(c.strip()) for c in changes]

print(f'Resulting frequency: {sum(changes)}')Resulting frequency: 435

--- Part Two ---

You notice that the device repeats the same frequency change list over and over. To calibrate the device, you need to find the first frequency it reaches twice.

For example, using the same list of changes above, the device would loop as follows:

    Current frequency  0, change of +1; resulting frequency  1.
    Current frequency  1, change of -2; resulting frequency -1.
    Current frequency -1, change of +3; resulting frequency  2.
    Current frequency  2, change of +1; resulting frequency  3.
    (At this point, the device continues from the start of the list.)
    Current frequency  3, change of +1; resulting frequency  4.
    Current frequency  4, change of -2; resulting frequency  2, which has already been seen.

In this example, the first frequency reached twice is 2. Note that your device might need to repeat its list of frequency changes many times before a duplicate frequency is found, and that duplicates might be found while in the middle of processing the list.

Here are other examples:

    +1, -1 first reaches 0 twice.
    +3, +3, +4, -2, -4 first reaches 10 twice.
    -6, +3, +8, +5, -6 first reaches 5 twice.
    +7, +7, -2, -7, -4 first reaches 14 twice.

What is the first frequency your device reaches twice?_____no_output_____

#read file
with open('1.txt', 'r') as f:
    changes = f.readlines()

#get rid of newlines and convert to int
changes = [int(c.strip()) for c in changes]

current_frequency = 0
unique_frequencies = {0}
found = False

while not found:

    for c in changes:
        current_frequency = c + current_frequency
        if current_frequency in unique_frequencies:
            print(f'Repeating frequency: {current_frequency}')
            found = True
            break
        else:
            unique_frequencies.add(current_frequency)Repeating frequency: 245


# Notebook from mmmarchetti/project_network_analysis_game_of_thrones
Path: notebook.ipynb

## 1. Winter is Coming. Let's load the dataset ASAP!

If you haven't heard of Game of Thrones, then you must be really good at hiding. Game of Thrones is the hugely popular television series by HBO based on the (also) hugely popular book series A Song of Ice and Fire by George R.R. Martin. In this notebook, we will analyze the co-occurrence network of the characters in the  Game of Thrones books. Here, two characters are considered to co-occur if their names appear in the vicinity of 15 words from one another in the books.

This dataset constitutes a network and is given as a text file describing the edges between characters, with some attributes attached to each edge. Let's start by loading in the data for the first book A Game of Thrones and inspect it.
_____no_output_____

# Importing modules
import pandas as pd

# Reading in datasets/book1.csv
book1 = pd.read_csv('datasets/book1.csv')

# Printing out the head of the dataset
print(book1.head())                            Source              Target        Type  weight  \
0                   Addam-Marbrand     Jaime-Lannister  Undirected       3
1                   Addam-Marbrand     Tywin-Lannister  Undirected       6
2                Aegon-I-Targaryen  Daenerys-Targaryen  Undirected       5
3                Aegon-I-Targaryen        Eddard-Stark  Undirected       4
4  Aemon-Targaryen-(Maester-Aemon)      Alliser-Thorne  Undirected       4

   book
0     1
1     1
2     1
3     1
4     1

## 2. Time for some Network of Thrones

The resulting DataFrame book1 has 5 columns: Source, Target, Type, weight, and book. Source and target are the two nodes that are linked by an edge. A network can have directed or undirected edges and in this network all the edges are undirected. The weight attribute of every edge tells us the number of interactions that the characters have had over the book, and the book column tells us the book number.

Once we have the data loaded as a pandas DataFrame, it's time to create a network. We will use networkx, a network analysis library, and create a graph object for the first book.
_____no_output_____

# Importing modules
import networkx as nx

# Creating an empty graph object
G_book1 = nx.Graph()_____no_output_____

## 3. Populate the network with the DataFrame

Currently, the graph object G_book1 is empty. Let's now populate it with the edges from book1. And while we're at it, let's load in the rest of the books too!
_____no_output_____

# Iterating through the DataFrame to add edges
for _, edge in book1.iterrows():
    G_book1.add_edge(edge['Source'], edge['Target'], weight=edge['weight'])

# Creating a list of networks for all the books
books = [G_book1]
book_fnames = ['datasets/book2.csv', 'datasets/book3.csv', 'datasets/book4.csv', 'datasets/book5.csv']
for book_fname in book_fnames:
    book = pd.read_csv(book_fname)
    G_book = nx.Graph()
    for _, edge in book.iterrows():
        G_book.add_edge(edge['Source'], edge['Target'], weight=edge['weight'])
    books.append(G_book)_____no_output_____

## 4. The most important character in Game of Thrones

Is it Jon Snow, Tyrion, Daenerys, or someone else? Let's see! Network science offers us many different metrics to measure the importance of a node in a network. Note that there is no "correct" way of calculating the most important node in a network, every metric has a different meaning.

First, let's measure the importance of a node in a network by looking at the number of neighbors it has, that is, the number of nodes it is connected to. For example, an influential account on Twitter, where the follower-followee relationship forms the network, is an account which has a high number of followers. This measure of importance is called degree centrality.

Using this measure, let's extract the top ten important characters from the first book (book[0]) and the fifth book (book[4]).
_____no_output_____

# Calculating the degree centrality of book 1
deg_cen_book1 = nx.degree_centrality(books[0])

# Calculating the degree centrality of book 5
deg_cen_book5 = nx.degree_centrality(books[4])

# Sorting the dictionaries according to their degree centrality and storing the top 10
sorted_deg_cen_book1 =  sorted(deg_cen_book1.items(), key=lambda x:x[1], reverse=True)[0:10]

# Sorting the dictionaries according to their degree centrality and storing the top 10
sorted_deg_cen_book5 =  sorted(deg_cen_book5.items(), key=lambda x:x[1], reverse=True)[0:10]

# Printing out the top 10 of book1 and book5
print(sorted_deg_cen_book1)
print(sorted_deg_cen_book5)[('Eddard-Stark', 0.3548387096774194), ('Robert-Baratheon', 0.2688172043010753), ('Tyrion-Lannister', 0.24731182795698928), ('Catelyn-Stark', 0.23118279569892475), ('Jon-Snow', 0.19892473118279572), ('Sansa-Stark', 0.18817204301075272), ('Robb-Stark', 0.18817204301075272), ('Bran-Stark', 0.17204301075268819), ('Cersei-Lannister', 0.16129032258064518), ('Joffrey-Baratheon', 0.16129032258064518)]
[('Jon-Snow', 0.1962025316455696), ('Daenerys-Targaryen', 0.18354430379746836), ('Stannis-Baratheon', 0.14873417721518986), ('Tyrion-Lannister', 0.10443037974683544), ('Theon-Greyjoy', 0.10443037974683544), ('Cersei-Lannister', 0.08860759493670886), ('Barristan-Selmy', 0.07911392405063292), ('Hizdahr-zo-Loraq', 0.06962025316455696), ('Asha-Greyjoy', 0.056962025316455694), ('Melisandre', 0.05379746835443038)]

## 5. The evolution of character importance

According to degree centrality, the most important character in the first book is Eddard Stark but he is not even in the top 10 of the fifth book. The importance of characters changes over the course of five books because, you know, stuff happens... ;)

Let's look at the evolution of degree centrality of a couple of characters like Eddard Stark, Jon Snow, and Tyrion, which showed up in the top 10 of degree centrality in the first book.
_____no_output_____

%matplotlib inline

# Creating a list of degree centrality of all the books
evol = [nx.degree_centrality(book) for book in books]

# Creating a DataFrame from the list of degree centralities in all the books
degree_evol_df = pd.DataFrame.from_records(evol)

# Plotting the degree centrality evolution of Eddard-Stark, Tyrion-Lannister and Jon-Snow
degree_evol_df[['Eddard-Stark', 'Tyrion-Lannister', 'Jon-Snow']].plot()_____no_output_____

## 6. What's up with Stannis Baratheon?

We can see that the importance of Eddard Stark dies off as the book series progresses. With Jon Snow, there is a drop in the fourth book but a sudden rise in the fifth book.

Now let's look at various other measures like betweenness centrality and PageRank to find important characters in our Game of Thrones character co-occurrence network and see if we can uncover some more interesting facts about this network. Let's plot the evolution of betweenness centrality of this network over the five books. We will take the evolution of the top four characters of every book and plot it.
_____no_output_____

# Creating a list of betweenness centrality of all the books just like we did for degree centrality
evol = [nx.betweenness_centrality(book, weight='weight') for book in books]

# Making a DataFrame from the list
betweenness_evol_df = pd.DataFrame.from_records(evol)

# Finding the top 4 characters in every book
set_of_char = set()
for i in range(5):
    set_of_char |= set(list(betweenness_evol_df.T[i].sort_values(ascending=False)[0:4].index))
list_of_char = list(set_of_char)

# Plotting the evolution of the top characters
betweenness_evol_df[list_of_char].plot(figsize=(13, 7))_____no_output_____

## 7. What does Google PageRank tell us about GoT?

We see a peculiar rise in the importance of Stannis Baratheon over the books. In the fifth book, he is significantly more important than other characters in the network, even though he is the third most important character according to degree centrality.

PageRank was the initial way Google ranked web pages. It evaluates the inlinks and outlinks of webpages in the world wide web, which is, essentially, a directed network. Let's look at the importance of characters in the Game of Thrones network according to PageRank.
_____no_output_____

# Creating a list of pagerank of all the characters in all the books
evol = [nx.pagerank(book) for book in books]

# Making a DataFrame from the list
pagerank_evol_df = pd.DataFrame.from_records(evol)

# Finding the top 4 characters in every book
set_of_char = set()
for i in range(5):
    set_of_char |= set(list(pagerank_evol_df.T[i].sort_values(ascending=False)[0:4].index))
list_of_char = list(set_of_char)

# Plotting the top characters
pagerank_evol_df[list_of_char].plot(figsize=(13, 7))_____no_output_____

## 8. Correlation between different measures

Stannis, Jon Snow, and Daenerys are the most important characters in the fifth book according to PageRank. Eddard Stark follows a similar curve but for degree centrality and betweenness centrality: He is important in the first book but dies into oblivion over the book series.

We have seen three different measures to calculate the importance of a node in a network, and all of them tells us something about the characters and their importance in the co-occurrence network. We see some names pop up in all three measures so maybe there is a strong correlation between them?

Let's look at the correlation between PageRank, betweenness centrality and degree centrality for the fifth book using Pearson correlation.
_____no_output_____

# Creating a list of pagerank, betweenness centrality, degree centrality
# of all the characters in the fifth book.
measures = [nx.pagerank(books[4]),
            nx.betweenness_centrality(books[4], weight='weight'),
            nx.degree_centrality(books[4])]

# Creating the correlation DataFrame
cor = pd.DataFrame.from_records(measures)

# Calculating the correlation
cor.T.corr()_____no_output_____

## 9. Conclusion

We see a high correlation between these three measures for our character co-occurrence network.

So we've been looking at different ways to find the important characters in the Game of Thrones co-occurrence network. According to degree centrality, Eddard Stark is the most important character initially in the books. But who is/are the most important character(s) in the fifth book according to these three measures?
_____no_output_____

# Finding the most important character in the fifth book,
# according to degree centrality, betweenness centrality and pagerank.
p_rank, b_cent, d_cent = cor.idxmax(axis=1)

# Printing out the top character accoding to the three measures
print(p_rank, b)_____no_output_____


# Notebook from dongxulee/lifeCycle
Path: 20200907/renting.ipynb

%pylab inline
from scipy.interpolate import interpn
from helpFunctions import surfacePlot
import numpy as np
from multiprocessing import Pool
from functools import partial
import warnings
import math
warnings.filterwarnings("ignore")
np.printoptions(precision=2)Populating the interactive namespace from numpy and matplotlib

### The value of renting
Assuming we obtain the value: $\tilde{V}_{t+1}(x_{t+1})$ where:
$x_{t+1} = [w_{t+1}, n_{t+1}, M_{t+1}, g_{t+1} = 0, e_{t+1}, s_{t+1}, (H)]$  from interpolation. We know $H$ and $M_t$ from the action taken and we could calculate mortgage payment $m$ and $rh$ (now treated as constant) is observed from the market. _____no_output_____* To start with we have state variable: $x_t = [w_t, n_t, e_t, s_t]$
* Housing choice is limited: $H_{\text{choice}} = \{1000, 2000\}$
* Mortgage choice is also limitted to discrete values $M_{t} = [0.2H, 0.5H, 0.8H]$
* Action: continue to rent: $a = (c, b, k, h)$ switch to owning a house: $a = (c, b, k, M, H)$
* Buying house activities can only happend during the age of 10 and 25._____no_output_____

# time line
T_min = 0
T_max = 70
T_R = 45
# discounting factor
beta = 1/(1+0.02)
# utility function parameter
gamma = 2
# relative importance of housing consumption and non durable consumption
alpha = 0.8
# parameter used to calculate the housing consumption
kappa = 0.3
# depreciation parameter
delta = 0.025
# housing parameter
chi = 0.3
# uB associated parameter
B = 2
# constant cost
c_h = 0.5
# All the money amount are denoted in thousand dollars
earningShock = [0.8,1.2]
# Define transition matrix of economical states
# GOOD -> GOOD 0.8, BAD -> BAD 0.6
Ps = np.array([[0.6, 0.4],[0.2, 0.8]])
# current risk free interest rate
r_b = np.array([0.03 ,0.05])
# stock return depends on current and future econ states
# r_k = np.array([[-0.2, 0.15],[-0.15, 0.2]])
r_k = np.array([[-0.15, 0.20],[-0.15, 0.20]])
# expected return on stock market
# r_bar = 0.0667
r_bar = 0.02
# probability of survival
Pa = np.load("prob.npy")
# deterministic income
detEarning = np.load("detEarning.npy")
# probability of employment transition Pe[s, s_next, e, e_next]
Pe = np.array([[[[0.3, 0.7], [0.1, 0.9]], [[0.25, 0.75], [0.05, 0.95]]],
               [[[0.25, 0.75], [0.05, 0.95]], [[0.2, 0.8], [0.01, 0.99]]]])
# tax rate before and after retirement
tau_L = 0.2
tau_R = 0.1

# some variables associate with 401k amount
Nt = [np.sum(Pa[t:]) for t in range(T_max-T_min)]
Dt = [np.ceil(((1+r_bar)**N - 1)/(r_bar*(1+r_bar)**N)) for N in Nt]
# income fraction goes into 401k
yi = 0.005
# mortgate rate
rh = 0.036
# this discount is used to calculate mortgage payment m
D = [((1+rh)**N - 1)/(rh*(1+rh)**N) for N in range(T_max-T_min)]
# housing price constant
pt = 2*250/1000
# renting price constant
pr = 2*30/1000_____no_output_____#Define the utility function
def u(c):
    # shift utility function to the left, so it only takes positive value
    return (np.float_power(c, 1-gamma) - 1)/(1 - gamma)

#Define the bequeath function, which is a function of wealth
def uB(tb):
    return B*u(tb)

#Calculate TB_rent
def calTB_rent(x):
    # change input x as numpy array
    # w, n, e, s = x
    TB = x[:,0] + x[:,1]
    return TB

#Calculate TB_own
def calTB_own(x):
    # change input x as numpy array
    # transiton from (w, n, e, s) -> (w, n, M, 0, e, s, H)
    TB = x[:,0] + x[:,1] + x[:,6]*pt - x[:,2]
    return TB

def u_rent(a):
    '''
    Input:
        action a: c, b, k, h = a
    Output:
        reward value: the length of return should be equal to the length of a
    '''
    c = a[:,0]
    h = a[:,3]
    C = np.float_power(c, alpha) * np.float_power(h, 1-alpha)
    return u(C)

def u_own(a):
    '''
    Input:
        action a: c, b, k, M, H = a
    Output:
        reward value: the length of return should be equal to the length of a
    '''
    c = a[:,0]
    H = a[:,4]
    C = np.float_power(c, alpha) * np.float_power((1+kappa)*H, 1-alpha)
    return u(C)

#Define the earning function, which applies for both employment and unemployment, good econ state and bad econ state
def y(t, x):
    w, n, e, s = x
    if t <= T_R:
        welfare = 5
        return detEarning[t] * earningShock[int(s)] * e + (1-e) * welfare
    else:
        return detEarning[t]

#Earning after tax and fixed by transaction in and out from 401k account
def yAT(t,x):
    yt = y(t, x)
    w, n, e, s = x
    if t <= T_R and e == 1:
        # 5% of the income will be put into the 401k
        return (1-tau_L)*(yt * (1-yi))
    if t <= T_R and e == 0:
        return yt
    else:
        # t > T_R, n/discounting amount will be withdraw from the 401k
        return (1-tau_R)*yt + n/Dt[t]

#Define the evolution of the amount in 401k account
def gn(t, n, x, s_next):
    w, n, e, s = x
    if t <= T_R and e == 1:
        # if the person is employed, then 5 percent of his income goes into 401k
        n_cur = n + y(t, x) * yi
    elif t <= T_R and e == 0:
        # if the perons is unemployed, then n does not change
        n_cur = n
    else:
        # t > T_R, n/discounting amount will be withdraw from the 401k
        n_cur = n - n/Dt[t]
    return (1+r_k[int(s), s_next])*n_cur _____no_output_____def transition_to_rent(x,a,t):
    '''
        imput a is np array constains all possible actions
        from x = [w, n, e, s] to x = [w, n, e, s]
    '''
    w, n, e, s = x
    # variables used to collect possible states and probabilities
    x_next = []
    prob_next = []
    for aa in a:
        c, b, k, h = aa
        for s_next in [0,1]:
                w_next =  b*(1+r_b[int(s)]) + k*(1+r_k[int(s), s_next])
                n_next = gn(t, n, x, s_next)
                if t >= T_R:
                    e_next = 0
                    x_next.append([w_next, n_next, e_next, s_next])
                    prob_next.append(Ps[int(s),s_next])
                else:
                    for e_next in [0,1]:
                        x_next.append([w_next, n_next, e_next, s_next])
                        prob_next.append(Ps[int(s),s_next] * Pe[int(s),s_next,int(e),e_next])
    return np.array(x_next), np.array(prob_next)

def transition_to_own(x,a,t):
    '''
        imput a is np array constains all possible actions
        from x = [w, n, e, s] to x = [w, n, M, g=0, e, s, H]
    '''
    w, n, e, s = x
    # variables used to collect possible states and probabilities
    x_next = []
    prob_next = []
    for aa in a:
        c, b, k, M, H = aa
        M_next = M*(1+rh)
        for s_next in [0,1]:
                w_next =  b*(1+r_b[int(s)]) + k*(1+r_k[int(s), s_next])
                n_next = gn(t, n, x, s_next)
                if t >= T_R:
                    e_next = 0
                    x_next.append([w_next, n_next, M_next, 0, e_next, s_next, H])
                    prob_next.append(Ps[int(s),s_next])
                else:
                    for e_next in [0,1]:
                        x_next.append([w_next, n_next, M_next, 0, e_next, s_next, H])
                        prob_next.append(Ps[int(s),s_next] * Pe[int(s),s_next,int(e),e_next])
    return np.array(x_next), np.array(prob_next)_____no_output_____# used to calculate dot product
def dotProduct(p_next, uBTB, t):
    if t >= T_R:
        return (p_next*uBTB).reshape((len(p_next)//2,2)).sum(axis = 1)
    else:
        return (p_next*uBTB).reshape((len(p_next)//4,4)).sum(axis = 1)

# Value function is a function of state and time, according to the restriction transfer from renting to ownning can only happen
# between the age: 10 - 25
def V(x, t, NN):
    w, n, e, s = x
    yat = yAT(t,x)

    # first define the objective function solver and then the objective function
    def obj_solver_rent(obj_rent):
    # a = [c, b, k, h]
    # Constrain: yat + w = c + b + k + pr*h
    # h_portion takes [0:0.05:0.95]
    # c_portion takes remaining [0:0.05:0.95]
    # b_portion takes reamining [0:0.05:0.95]
    # k is the remainder
        actions = []
        for hp in np.linspace(0,0.99,20):
            budget1 = yat + w
            h = budget1 * hp/pr
            budget2 = budget1 * (1-hp)
            for cp in np.linspace(0,1,11):
                c = budget2*cp
                budget3 = budget2 * (1-cp)
                for bp in np.linspace(0,1,11):
                    b = budget3* bp
                    k = budget3 * (1-bp)
                    # q = 1 not renting in this case
                    actions.append([c,b,k,h])

        actions = np.array(actions)
        values = obj_rent(actions)
        fun = np.max(values)
        ma = actions[np.argmax(values)]
        return fun, ma

    def obj_solver_own(obj_own):
    # a = [c, b, k, M, H]
    # possible value of H = {750, 1000, 1500, 2000} possible value of [0.2H, 0.4H, 0.6H, 0.8H]]*pt
    # (M, t, rh) --> m
    # Constrain: yat + w = c + b + k + (H*pt - M) + ch
        actions = []
        for H in H_options:
            for mp in M_options:
                M = mp*H*pt
                m = M/D[T_max - t - 1]
                # 5 is the welfare income which is also the minimum income
                if (H*pt - M) + c_h <= yat + w and m < pr*H + 5:
                    budget1 = yat + w - (H*pt - M) - c_h
                    # c_portion takes remaining [0:0.05:0.95]
                    # b_portion takes reamining [0:0.05:0.95]
                    # k is the remainder
                    for cp in np.linspace(0,1,11):
                        c = budget1*cp
                        budget2 = budget1 * (1-cp)
                        for bp in np.linspace(0,1,11):
                            b = budget2* bp
                            k = budget2 * (1-bp)
                            actions.append([c,b,k,M,H])
        if len(actions) == 0:
            return -np.inf, [0,0,0,0,0]
        else:
            actions = np.array(actions)
            values = obj_own(actions)
            fun = np.max(values)
            ma = actions[np.argmax(values)]
            return fun, ma

    if t == T_max-1:
        # The objective function of renting
        def obj_rent(actions):
            # a = [c, b, k, h]
            x_next, p_next  = transition_to_rent(x, actions, t)
            uBTB = uB(calTB_rent(x_next))
            return u_rent(actions) + beta * dotProduct(uBTB, p_next, t)

        fun, action = obj_solver_rent(obj_rent)
        return np.array([fun, action])

    elif t < 10 or t > 25:
        # The objective function of renting
        def obj_rent(actions):
            # a = [c, b, k, h]
            x_next, p_next  = transition_to_rent(x, actions, t)
            V_tilda = NN.predict(x_next) # V_rent_{t+1} used to approximate, shape of x is [w,n,e,s]
            uBTB = uB(calTB_rent(x_next))
            return u_rent(actions) + beta * (Pa[t] * dotProduct(V_tilda, p_next, t) + (1 - Pa[t]) * dotProduct(uBTB, p_next, t))

        fun, action = obj_solver_rent(obj_rent)
        return np.array([fun, action])

    else:
        # The objective function of renting
        def obj_rent(actions):
            # a = [c, b, k, h]
            x_next, p_next  = transition_to_rent(x, actions, t)
            V_tilda = NN.predict(x_next) # V_rent_{t+1} used to approximate, shape of x is [w,n,e,s]
            uBTB = uB(calTB_rent(x_next))
            return u_rent(actions) + beta * (Pa[t] * dotProduct(V_tilda, p_next, t) + (1 - Pa[t]) * dotProduct(uBTB, p_next, t))
        # The objective function of owning
        def obj_own(actions):
            # a = [c, b, k, M, H]
            x_next, p_next  = transition_to_own(x, actions, t)
            V_tilda = NN.predict(x_next) # V_own_{t+1} used to approximate, shape of x is [w,n,0,e,s,H,M]
            uBTB = uB(calTB_own(x_next))
            return u_own(actions) + beta * (Pa[t] * dotProduct(V_tilda, p_next, t) + (1 - Pa[t]) * dotProduct(uBTB, p_next, t))

        fun1, action1 = obj_solver_rent(obj_rent)
        fun2, action2 = obj_solver_own(obj_own)
        if fun1 > fun2:
            return np.array([fun1, action1])
        else:
            return np.array([fun2, action2])_____no_output_____# wealth discretization
ws = np.array([10,25,50,75,100,125,150,175,200,250,500,750,1000,1500,3000])
w_grid_size = len(ws)
# 401k amount discretization
ns = np.array([1, 5, 10, 15, 25, 40, 65, 100, 150, 300, 400,1000])
n_grid_size = len(ns)

xgrid = np.array([[w, n, e, s]
                            for w in ws
                            for n in ns
                            for e in [0,1]
                            for s in [0,1]
                            ]).reshape((w_grid_size, n_grid_size,2,2,4))

Vgrid = np.zeros((w_grid_size, n_grid_size,2,2, T_max-T_min))
cgrid = np.zeros((w_grid_size, n_grid_size,2,2, T_max-T_min))
bgrid = np.zeros((w_grid_size, n_grid_size,2,2, T_max-T_min))
kgrid = np.zeros((w_grid_size, n_grid_size,2,2, T_max-T_min))
hgrid = np.zeros((w_grid_size, n_grid_size,2,2, T_max-T_min))
# Policy function of buying a house
Mgrid = np.zeros((w_grid_size, n_grid_size,2,2, T_max-T_min))
Hgrid = np.zeros((w_grid_size, n_grid_size,2,2, T_max-T_min))_____no_output_____# Define housing choice part: Housing unit options and Mortgage amount options
V1000 = np.load("Vgrid1000.npy")
V1500 = np.load("Vgrid1500.npy")
V2000 = np.load("Vgrid2000.npy")
V750 = np.load("Vgrid750.npy")
H_options = [750, 1000, 1500, 2000]
M_options = [0.2, 0.5, 0.8]
Vown = [V750, V1000, V1500, V2000]
Hs = [750, 1000, 1500, 2000]_____no_output_____class iApproxy(object):
    def __init__(self, pointsRent, Vrent, Vown, t):
        self.Vrent = Vrent
        self.Vown = Vown
        self.Prent = pointsRent
        self.t = t
    def predict(self, xx):
        if xx.shape[1] == 4:
            # x = [w, n, e, s]
            pvalues = np.zeros(xx.shape[0])
            index00 = (xx[:,2] == 0) & (xx[:,3] == 0)
            index01 = (xx[:,2] == 0) & (xx[:,3] == 1)
            index10 = (xx[:,2] == 1) & (xx[:,3] == 0)
            index11 = (xx[:,2] == 1) & (xx[:,3] == 1)
            pvalues[index00]=interpn(self.Prent, self.Vrent[:,:,0,0], xx[index00][:,:2], bounds_error = False, fill_value = None)
            pvalues[index01]=interpn(self.Prent, self.Vrent[:,:,0,1], xx[index01][:,:2], bounds_error = False, fill_value = None)
            pvalues[index10]=interpn(self.Prent, self.Vrent[:,:,1,0], xx[index10][:,:2], bounds_error = False, fill_value = None)
            pvalues[index11]=interpn(self.Prent, self.Vrent[:,:,1,1], xx[index11][:,:2], bounds_error = False, fill_value = None)
            return pvalues
        else:
            # x = w, n, M, g=0, e, s, H
            pvalues = np.zeros(xx.shape[0])
            for i in range(len(Hs)):
                H = Hs[i]
                # Mortgage amount, * 0.25 is the housing price per unit
                Ms = np.array([0.01*H,0.05*H,0.1*H,0.2*H,0.3*H,0.4*H,0.5*H,0.6*H,0.7*H,0.8*H]) * pt
                points = (ws,ns,Ms)
                index00 = (xx[:,4] == 0) & (xx[:,5] == 0) & (xx[:,6] == H)
                index01 = (xx[:,4] == 0) & (xx[:,5] == 1) & (xx[:,6] == H)
                index10 = (xx[:,4] == 1) & (xx[:,5] == 0) & (xx[:,6] == H)
                index11 = (xx[:,4] == 1) & (xx[:,5] == 1) & (xx[:,6] == H)
                pvalues[index00]=interpn(points, self.Vown[i][:,:,:,0,0,0,self.t], xx[index00][:,:3], method = "nearest",bounds_error = False, fill_value = None)
                pvalues[index01]=interpn(points, self.Vown[i][:,:,:,0,0,1,self.t], xx[index01][:,:3], method = "nearest",bounds_error = False, fill_value = None)
                pvalues[index10]=interpn(points, self.Vown[i][:,:,:,0,1,0,self.t], xx[index10][:,:3], method = "nearest",bounds_error = False, fill_value = None)
                pvalues[index11]=interpn(points, self.Vown[i][:,:,:,0,1,1,self.t], xx[index11][:,:3], method = "nearest",bounds_error = False, fill_value = None)
            return pvalues_____no_output_____%%time
# value iteration part
xs = xgrid.reshape((w_grid_size*n_grid_size*2*2,4))
pool = Pool()
pointsRent = (ws, ns)
for t in range(T_max-1,T_min, -1):
    print(t)
    if t == T_max - 1:
        f = partial(V, t = t, NN = None)
        results = np.array(pool.map(f, xs))
    else:
        approx = iApproxy(pointsRent,Vgrid[:,:,:,:,t+1], Vown, t+1)
        f = partial(V, t = t, NN = approx)
        results = np.array(pool.map(f, xs))

    # here results need some clean up due to different length of the actions taken.
    # a = [c,b,k,h] or a = [c,b,k,M,H]
    Vgrid[:,:,:,:,t] = results[:,0].reshape((w_grid_size,n_grid_size,2,2))
    cgrid[:,:,:,:,t] = np.array([r[0] for r in results[:,1]]).reshape((w_grid_size,n_grid_size,2,2))
    bgrid[:,:,:,:,t] = np.array([r[1] for r in results[:,1]]).reshape((w_grid_size,n_grid_size,2,2))
    kgrid[:,:,:,:,t] = np.array([r[2] for r in results[:,1]]).reshape((w_grid_size,n_grid_size,2,2))
    # if a = [c, b, k, h]
    hgrid[:,:,:,:,t] = np.array([r[3] if len(r) == 4 else r[4] for r in results[:,1]]).reshape((w_grid_size,n_grid_size,2,2))
    # if a = [c, b, k, M, H]
    Mgrid[:,:,:,:,t] = np.array([r[3] if len(r) == 5 else 0 for r in results[:,1]]).reshape((w_grid_size,n_grid_size,2,2))
    Hgrid[:,:,:,:,t] = np.array([r[4] if len(r) == 5 else 0 for r in results[:,1]]).reshape((w_grid_size,n_grid_size,2,2))
pool.close()

np.save("Vgrid_renting",Vgrid)
np.save("cgrid_renting",cgrid)
np.save("bgrid_renting",bgrid)
np.save("kgrid_renting",kgrid)
np.save("hgrid_renting",hgrid)
np.save("Mgrid_renting",Mgrid)
np.save("Hgrid_renting",Hgrid) 69
68
67
66
65
64
63
62
61
60
59
58
57
56
55
54
53
52
51
50
49
48
47
46
45
44
43
42
41
40
39
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
CPU times: user 11min 18s, sys: 17min 29s, total: 28min 48s
Wall time: 38min 17s
for tt in range(10,25):
    print(Hgrid[:,1,1,1,tt])[   0.    0.  750.    0. 1000.    0. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.    0. 1000.    0. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.    0. 1000.    0. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.    0. 1000. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.  750.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.    0.
  750.  750.  750.]
[   0.  750.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.  750.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.  750.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.  750.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.  750.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.  750.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
[   0.  750.  750.    0.    0. 1500. 1500. 1500. 2000. 2000. 2000.  750.
  750.  750.  750.]
for tt in range(10,25):
    print(Hgrid[:,1,0,1,tt])[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000.    0. 1500. 1500. 2000.  750.
  750.  750.  750.]
for tt in range(10,25):
    print(Hgrid[:,1,1,0,tt])[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.  750.  750. 1000. 1000. 1500. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
for tt in range(10,25):
    print(Hgrid[:,1,0,0,tt])[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1500. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1000. 1500. 2000. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1000. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1000. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1000. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1000. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1000. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1000. 1500. 1500. 2000.  750.
  750.  750.  750.]
[   0.    0.    0.    0.  750.  750. 1000. 1000. 1500. 1500. 2000.  750.
  750.  750.  750.]


# Notebook from tallamjr/spark_stfcsummerschool2019
Path: notebooks/solutions/mini-projects-images_solutions.ipynb

# Mini-Project 2: Source extraction

**Context**

This notebook illustrates the basics of distributing image data, and process them separately. In this notebook, we will load MegaCam data, and extract source positions in the CCDs. For more about Apache Spark in the context of astronomy, connect to [AstroLab Software](https://astrolabsoftware.github.io/)!

**Learning objectives**

After going through this notebook, you should be able to:

- Load and efficiently access astronomical images with Apache Spark
- Interface and use your favourite image processing package.

For this project, you will use the data at `data/images`:_____no_output_____

df = spark.read.format("fits").option("hdu", 1).load("../../data/images")_____no_output_____df.show(3)+--------------------+
|               Image|
+--------------------+
|[0.0, 0.0, 0.0, 0...|
|[0.0, 0.0, 0.0, 0...|
|[0.0, 0.0, 0.0, 0...|
+--------------------+
only showing top 3 rows

Spark SQL does not have the sense of what an image is, he knows only about rows.

**Exercise (£):** Find a way to isolate each image into one partition. Hint: each image comes from its own file._____no_output_____

from pyspark.sql.functions import input_file_name

df_images = df.withColumn("filename",input_file_name()).orderBy("filename")_____no_output_____df_images.show(1)+--------------------+--------------------+
|               Image|            filename|
+--------------------+--------------------+
|[0.0, 0.0, 0.0, 0...|file:/home/jovyan...|
+--------------------+--------------------+
only showing top 1 row

**Exercise (££ to ££££):** Using photutils or whatever you prefer, extract position of sources in images whose peak value is 50-sigma above the background._____no_output_____

from astropy.stats import sigma_clipped_stats
from astropy.visualization import SqrtStretch
from astropy.visualization.mpl_normalize import ImageNormalize

import numpy as np
import pandas as pd

from photutils import DAOStarFinder
from photutils import CircularAperture

from pyspark.sql.types import FloatType, ArrayType
from pyspark.sql.functions import pandas_udf, PandasUDFType

def get_stat(data, sigma=3.0, iters=3):
    """ Estimate the background and background noise using
    sigma-clipped statistics.

    Parameters
    ----------
    data : 2D array
        2d array containing the data.
    sigma : float
        sigma.
    iters : int
        Number of iteration to perform to get accurate estimate.
        The higher the better, but it will be longer.
    """
    mean, median, std = sigma_clipped_stats(data, sigma=sigma, iters=iters)
    return mean, median, std

def process_me_map(part):
    """ Use photutils to extract source information
    from image (one image per partition).
    """
    data = [*part]

    # Reshape images for photutils
    image = np.reshape(data, (4644, 2112))

    # Get background statistics
    mean, median, std = get_stat(image)

    # Use star finder
    sf = DAOStarFinder(fwhm=10.0, threshold=50.*std)
    cat = sf(image - median)

    yield catWARNING: AstropyDeprecationWarning: Composition of model classes will be removed in 4.0 (but composition of model instances is not affected) [astropy.modeling.core]
# Use mapPartitions (NB: we could have used pandas_udf)
rdd_catalogs = df_images.select("Image").rdd.glom().mapPartitions(process_me_map)_____no_output_____# get first image and its associated catalog
first_image = df_images.select('Image').rdd.glom().take(1)
first_image = np.reshape(first_image, (4644, 2112))

first_catal = rdd_catalogs.take(1)[0]_____no_output_____# Overplot detections on the CCD
import matplotlib.pyplot as plt

fig = plt.figure(0, (15, 15))
positions = (first_catal['ycentroid'], first_catal['xcentroid'])
apertures = CircularAperture(positions, r=30.)
norm = ImageNormalize(stretch=SqrtStretch())
plt.imshow(np.transpose(first_image), cmap='binary', origin="lower", norm=norm)
plt.show()

fig = plt.figure(0, (15, 15))
plt.imshow(np.transpose(first_image), cmap='binary', origin="lower", norm=norm)
apertures.plot(color='blue', lw=1.0, alpha=0.5)_____no_output_____


# Notebook from JWarmenhoven/pymc3
Path: docs/source/notebooks/multilevel_modeling.ipynb

# A Primer on Bayesian Methods for Multilevel Modeling_____no_output_____Hierarchical or multilevel modeling is a generalization of regression modeling.

*Multilevel models* are regression models in which the constituent model parameters are given **probability models**. This implies that model parameters are allowed to **vary by group**.

Observational units are often naturally **clustered**. Clustering induces dependence between observations, despite random sampling of clusters and random sampling within clusters.

A *hierarchical model* is a particular multilevel model where parameters are nested within one another.

Some multilevel structures are not hierarchical.

* e.g. "country" and "year" are not nested, but may represent separate, but overlapping, clusters of parameters

We will motivate this topic using an environmental epidemiology example._____no_output_____### Example: Radon contamination (Gelman and Hill 2006)

Radon is a radioactive gas that enters homes through contact points with the ground. It is a carcinogen that is the primary cause of lung cancer in non-smokers. Radon levels vary greatly from household to household.

![radon](http://www.fix-your-radon.com/images/how_radon_enters.jpg)

The EPA did a study of radon levels in 80,000 houses. Two important predictors:

* measurement in basement or first floor (radon higher in basements)
* county uranium level (positive correlation with radon levels)

We will focus on modeling radon levels in Minnesota.

The hierarchy in this example is households within county. _____no_output_____### Data organization_____no_output_____First, we import the data from a local file, and extract Minnesota's data._____no_output_____

%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_context('notebook')
sns.set_style('white')

from pymc3 import get_data

# Import radon data
srrs2 = pd.read_csv(get_data('srrs2.dat'))
srrs2.columns = srrs2.columns.map(str.strip)
srrs_mn = srrs2[srrs2.state=='MN'].copy()_____no_output_____

Next, obtain the county-level predictor, uranium, by combining two variables._____no_output_____

srrs_mn['fips'] = srrs_mn.stfips*1000 + srrs_mn.cntyfips
cty = pd.read_csv(get_data('cty.dat'))
cty_mn = cty[cty.st=='MN'].copy()
cty_mn[ 'fips'] = 1000*cty_mn.stfips + cty_mn.ctfips_____no_output_____

Use the `merge` method to combine home- and county-level information in a single DataFrame._____no_output_____

srrs_mn = srrs_mn.merge(cty_mn[['fips', 'Uppm']], on='fips')
srrs_mn = srrs_mn.drop_duplicates(subset='idnum')
u = np.log(srrs_mn.Uppm)

n = len(srrs_mn)_____no_output_____srrs_mn.head()_____no_output_____

We also need a lookup table (`dict`) for each unique county, for indexing._____no_output_____

srrs_mn.county = srrs_mn.county.map(str.strip)
mn_counties = srrs_mn.county.unique()
counties = len(mn_counties)
county_lookup = dict(zip(mn_counties, range(len(mn_counties))))_____no_output_____

Finally, create local copies of variables._____no_output_____

county = srrs_mn['county_code'] = srrs_mn.county.replace(county_lookup).values
radon = srrs_mn.activity
srrs_mn['log_radon'] = log_radon = np.log(radon + 0.1).values
floor_measure = srrs_mn.floor.values_____no_output_____

Distribution of radon levels in MN (log scale):_____no_output_____

srrs_mn.activity.apply(lambda x: np.log(x+0.1)).hist(bins=25);_____no_output_____

## Conventional approaches

The two conventional alternatives to modeling radon exposure represent the two extremes of the bias-variance tradeoff:

***Complete pooling***:

Treat all counties the same, and estimate a single radon level.

$$y_i = \alpha + \beta x_i + \epsilon_i$$

***No pooling***:

Model radon in each county independently.

$$y_i = \alpha_{j[i]} + \beta x_i + \epsilon_i$$

where $j = 1,\ldots,85$

The errors $\epsilon_i$ may represent measurement error, temporal within-house variation, or variation among houses._____no_output_____Here are the point estimates of the slope and intercept for the complete pooling model:_____no_output_____

from pymc3 import Model, sample, Normal, HalfCauchy, Uniform

floor = srrs_mn.floor.values
log_radon = srrs_mn.log_radon.values

with Model() as pooled_model:

    beta = Normal('beta', 0, sd=1e5, shape=2)
    sigma = HalfCauchy('sigma', 5)

    theta = beta[0] + beta[1]*floor

    y = Normal('y', theta, sd=sigma, observed=log_radon)
    _____no_output_____with pooled_model:
    pooled_trace = sample(2000, n_init=50000, tune=1000)Auto-assigning NUTS sampler...
Initializing NUTS using ADVI...
Average Loss = 1,121.1:  53%|█████▎    | 26617/50000 [00:03<00:03, 7258.34it/s]
Convergence archived at 27000
Interrupted at 27,000 [54%]: Average Loss = 1,201.3
100%|██████████| 2000/2000 [00:02<00:00, 798.65it/s]
b0, m0 = pooled_trace['beta', 1000:].mean(axis=0)_____no_output_____plt.scatter(srrs_mn.floor, np.log(srrs_mn.activity+0.1))
xvals = np.linspace(-0.2, 1.2)
plt.plot(xvals, m0*xvals+b0, 'r--');_____no_output_____

Estimates of county radon levels for the unpooled model:_____no_output_____

with Model() as unpooled_model:

    beta0 = Normal('beta0', 0, sd=1e5, shape=counties)
    beta1 = Normal('beta1', 0, sd=1e5)
    sigma = HalfCauchy('sigma', 5)

    theta = beta0[county] + beta1*floor

    y = Normal('y', theta, sd=sigma, observed=log_radon)
    _____no_output_____with unpooled_model:
    unpooled_trace = sample(2000, n_init=50000, tune=1000)Auto-assigning NUTS sampler...
Initializing NUTS using ADVI...
Average Loss = 2,069.3: 100%|██████████| 50000/50000 [00:11<00:00, 4504.30it/s]
Finished [100%]: Average Loss = 2,069.3
100%|██████████| 2000/2000 [00:13<00:00, 145.16it/s]

Here are the unpooled county expected values_____no_output_____

from pymc3 import forestplot

plt.figure(figsize=(6,14))
forestplot(unpooled_trace, varnames=['beta0']);_____no_output_____unpooled_estimates = pd.Series(unpooled_trace['beta0'].mean(axis=0), index=mn_counties)
unpooled_se = pd.Series(unpooled_trace['beta0'].std(axis=0), index=mn_counties)_____no_output_____

We can plot the ordered estimates to identify counties with high radon levels:_____no_output_____

order = unpooled_estimates.order().index

plt.scatter(range(len(unpooled_estimates)), unpooled_estimates[order])
for i, m, se in zip(range(len(unpooled_estimates)), unpooled_estimates[order], unpooled_se[order]):
    plt.plot([i,i], [m-se, m+se], 'b-')
plt.xlim(-1,86); plt.ylim(-1,4)
plt.ylabel('Radon estimate');plt.xlabel('Ordered county');/Users/fonnescj/anaconda3/envs/dev/lib/python3.6/site-packages/ipykernel_launcher.py:1: FutureWarning: order is deprecated, use sort_values(...)
  """Entry point for launching an IPython kernel.

Here are visual comparisons between the pooled and unpooled estimates for a subset of counties representing a range of sample sizes._____no_output_____

sample_counties = ('LAC QUI PARLE', 'AITKIN', 'KOOCHICHING',
                    'DOUGLAS', 'CLAY', 'STEARNS', 'RAMSEY', 'ST LOUIS')

fig, axes = plt.subplots(2, 4, figsize=(12, 6), sharey=True, sharex=True)
axes = axes.ravel()
m = unpooled_trace['beta1'].mean()
for i,c in enumerate(sample_counties):
    y = srrs_mn.log_radon[srrs_mn.county==c]
    x = srrs_mn.floor[srrs_mn.county==c]
    axes[i].scatter(x + np.random.randn(len(x))*0.01, y, alpha=0.4)

    # No pooling model
    b = unpooled_estimates[c]

    # Plot both models and data
    xvals = np.linspace(-0.2, 1.2)
    axes[i].plot(xvals, m*xvals+b)
    axes[i].plot(xvals, m0*xvals+b0, 'r--')
    axes[i].set_xticks([0,1])
    axes[i].set_xticklabels(['basement', 'floor'])
    axes[i].set_ylim(-1, 3)
    axes[i].set_title(c)
    if not i%2:
        axes[i].set_ylabel('log radon level')_____no_output_____

Neither of these models are satisfactory:

* if we are trying to identify high-radon counties, pooling is useless
* we do not trust extreme unpooled estimates produced by models using few observations_____no_output_____## Multilevel and hierarchical models

When we pool our data, we imply that they are sampled from the same model. This ignores any variation among sampling units (other than sampling variance):

![pooled](http://f.cl.ly/items/0R1W063h1h0W2M2C0S3M/Screen%20Shot%202013-10-10%20at%208.22.21%20AM.png)

When we analyze data unpooled, we imply that they are sampled independently from separate models. At the opposite extreme from the pooled case, this approach claims that differences between sampling units are to large to combine them:

![unpooled](http://f.cl.ly/items/38020n2t2Y2b1p3t0B0e/Screen%20Shot%202013-10-10%20at%208.23.36%20AM.png)

In a hierarchical model, parameters are viewed as a sample from a population distribution of parameters. Thus, we view them as being neither entirely different or exactly the same. This is ***parital pooling***.

![hierarchical](http://f.cl.ly/items/1B3U223i002y3V2W3r0W/Screen%20Shot%202013-10-10%20at%208.25.05%20AM.png)_____no_output_____We can use PyMC to easily specify multilevel models, and fit them using Markov chain Monte Carlo._____no_output_____## Partial pooling model

The simplest partial pooling model for the household radon dataset is one which simply estimates radon levels, without any predictors at any level. A partial pooling model represents a compromise between the pooled and unpooled extremes, approximately a weighted average (based on sample size) of the unpooled county estimates and the pooled estimates.

$$\hat{\alpha} \approx \frac{(n_j/\sigma_y^2)\bar{y}_j + (1/\sigma_{\alpha}^2)\bar{y}}{(n_j/\sigma_y^2) + (1/\sigma_{\alpha}^2)}$$

Estimates for counties with smaller sample sizes will shrink towards the state-wide average.

Estimates for counties with larger sample sizes will be closer to the unpooled county estimates._____no_output_____

with Model() as partial_pooling:

    # Priors
    mu_a = Normal('mu_a', mu=0., sd=1e5)
    sigma_a = HalfCauchy('sigma_a', 5)

    # Random intercepts
    a = Normal('a', mu=mu_a, sd=sigma_a, shape=counties)

    # Model error
    sigma_y = HalfCauchy('sigma_y',5)

    # Expected value
    y_hat = a[county]

    # Data likelihood
    y_like = Normal('y_like', mu=y_hat, sd=sigma_y, observed=log_radon)
    _____no_output_____with partial_pooling:
    partial_pooling_trace = sample(2000, n_init=50000, tune=1000)Auto-assigning NUTS sampler...
Initializing NUTS using ADVI...
Average Loss = 1,115.5: 100%|██████████| 50000/50000 [00:13<00:00, 3590.55it/s]
Finished [100%]: Average Loss = 1,115.5
100%|██████████| 2000/2000 [00:07<00:00, 253.79it/s]
sample_trace = partial_pooling_trace['a', 1000:]

fig, axes = plt.subplots(1, 2, figsize=(14,6), sharex=True, sharey=True)
samples, counties = sample_trace.shape
jitter = np.random.normal(scale=0.1, size=counties)

n_county = srrs_mn.groupby('county')['idnum'].count()
unpooled_means = srrs_mn.groupby('county')['log_radon'].mean()
unpooled_sd = srrs_mn.groupby('county')['log_radon'].std()
unpooled = pd.DataFrame({'n':n_county, 'm':unpooled_means, 'sd':unpooled_sd})
unpooled['se'] = unpooled.sd/np.sqrt(unpooled.n)

axes[0].plot(unpooled.n + jitter, unpooled.m, 'b.')
for j, row in zip(jitter, unpooled.iterrows()):
    name, dat = row
    axes[0].plot([dat.n+j,dat.n+j], [dat.m-dat.se, dat.m+dat.se], 'b-')
axes[0].set_xscale('log')
axes[0].hlines(sample_trace.mean(), 0.9, 100, linestyles='--')

samples, counties = sample_trace.shape
means = sample_trace.mean(axis=0)
sd = sample_trace.std(axis=0)
axes[1].scatter(n_county.values + jitter, means)
axes[1].set_xscale('log')
axes[1].set_xlim(1,100)
axes[1].set_ylim(0, 3)
axes[1].hlines(sample_trace.mean(), 0.9, 100, linestyles='--')
for j,n,m,s in zip(jitter, n_county.values, means, sd):
    axes[1].plot([n+j]*2, [m-s, m+s], 'b-')_____no_output_____

Notice the difference between the unpooled and partially-pooled estimates, particularly at smaller sample sizes. The former are both more extreme and more imprecise._____no_output_____## Varying intercept model

This model allows intercepts to vary across county, according to a random effect.

$$y_i = \alpha_{j[i]} + \beta x_{i} + \epsilon_i$$

where

$$\epsilon_i \sim N(0, \sigma_y^2)$$

and the intercept random effect:

$$\alpha_{j[i]} \sim N(\mu_{\alpha}, \sigma_{\alpha}^2)$$

As with the the “no-pooling” model, we set a separate intercept for each county, but rather than fitting separate least squares regression models for each county, multilevel modeling **shares strength** among counties, allowing for more reasonable inference in counties with little data._____no_output_____

with Model() as varying_intercept:

    # Priors
    mu_a = Normal('mu_a', mu=0., tau=0.0001)
    sigma_a = HalfCauchy('sigma_a', 5)

    # Random intercepts
    a = Normal('a', mu=mu_a, sd=sigma_a, shape=counties)
    # Common slope
    b = Normal('b', mu=0., sd=1e5)

    # Model error
    sd_y = HalfCauchy('sd_y', 5)

    # Expected value
    y_hat = a[county] + b * floor_measure

    # Data likelihood
    y_like = Normal('y_like', mu=y_hat, sd=sd_y, observed=log_radon)
    _____no_output_____

We can fit the above model using MCMC._____no_output_____

with varying_intercept:
    varying_intercept_trace = sample(2000, n_init=50000, tune=1000)Auto-assigning NUTS sampler...
Initializing NUTS using ADVI...
Average Loss = 1,077.4: 100%|██████████| 50000/50000 [00:11<00:00, 4206.38it/s]
Finished [100%]: Average Loss = 1,077.5
100%|██████████| 2000/2000 [00:06<00:00, 296.67it/s]
from pymc3 import forestplot, traceplot, plot_posterior

plt.figure(figsize=(6,14))
forestplot(varying_intercept_trace[1000:], varnames=['a'])_____no_output_____plot_posterior(varying_intercept_trace[1000:], varnames=['sigma_a', 'b'])_____no_output_____

The estimate for the `floor` coefficient is approximately -0.66, which can be interpreted as houses without basements having about half ($\exp(-0.66) = 0.52$) the radon levels of those with basements, after accounting for county._____no_output_____

from pymc3 import summary

summary(varying_intercept_trace, varnames=['b'])
b:

  Mean             SD               MC Error         95% HPD interval
  -------------------------------------------------------------------

  -0.660           0.068            0.002            [-0.794, -0.532]

  Posterior quantiles:
  2.5            25             50             75             97.5
  |--------------|==============|==============|--------------|

  -0.787         -0.706         -0.661         -0.615         -0.525

xvals = np.arange(2)
bp = varying_intercept_trace[a, 1000:].mean(axis=0)
mp = varying_intercept_trace[b, 1000:].mean()
for bi in bp:
    plt.plot(xvals, mp*xvals + bi, 'bo-', alpha=0.4)
plt.xlim(-0.1,1.1);_____no_output_____

It is easy to show that the partial pooling model provides more objectively reasonable estimates than either the pooled or unpooled models, at least for counties with small sample sizes._____no_output_____

fig, axes = plt.subplots(2, 4, figsize=(12, 6), sharey=True, sharex=True)
axes = axes.ravel()
for i,c in enumerate(sample_counties):

    # Plot county data
    y = srrs_mn.log_radon[srrs_mn.county==c]
    x = srrs_mn.floor[srrs_mn.county==c]
    axes[i].scatter(x + np.random.randn(len(x))*0.01, y, alpha=0.4)

    # No pooling model
    m,b = unpooled_estimates[['floor', c]]

    xvals = np.linspace(-0.2, 1.2)
    # Unpooled estimate
    axes[i].plot(xvals, m*xvals+b)
    # Pooled estimate
    axes[i].plot(xvals, m0*xvals+b0, 'r--')
    # Partial pooling esimate
    axes[i].plot(xvals, mp*xvals+bp[county_lookup[c]], 'k:')
    axes[i].set_xticks([0,1])
    axes[i].set_xticklabels(['basement', 'floor'])
    axes[i].set_ylim(-1, 3)
    axes[i].set_title(c)
    if not i%2:
        axes[i].set_ylabel('log radon level')_____no_output_____

## Varying slope model

Alternatively, we can posit a model that allows the counties to vary according to how the location of measurement (basement or floor) influences the radon reading.

$$y_i = \alpha + \beta_{j[i]} x_{i} + \epsilon_i$$
_____no_output_____

with Model() as varying_slope:

    # Priors
    mu_b = Normal('mu_b', mu=0., sd=1e5)
    sigma_b = HalfCauchy('sigma_b', 5)

    # Common intercepts
    a = Normal('a', mu=0., sd=1e5)
    # Random slopes
    b = Normal('b', mu=mu_b, sd=sigma_b, shape=counties)

    # Model error
    sigma_y = HalfCauchy('sigma_y',5)

    # Expected value
    y_hat = a + b[county] * floor_measure

    # Data likelihood
    y_like = Normal('y_like', mu=y_hat, sd=sigma_y, observed=log_radon)
    _____no_output_____with varying_slope:
    varying_slope_trace = sample(2000, n_init=50000, tune=1000)Auto-assigning NUTS sampler...
Initializing NUTS using ADVI...
Average Loss = 1,124.6: 100%|██████████| 50000/50000 [00:13<00:00, 3759.25it/s]
Finished [100%]: Average Loss = 1,124.6
100%|█████████▉| 1998/2000 [00:12<00:00, 172.69it/s]/Users/fonnescj/Repos/pymc3/pymc3/step_methods/hmc/nuts.py:268: UserWarning: The acceptance probability in chain 0 does not match the target. It is 0.655412161099, but should be close to 0.8. Try to increase the number of tuning steps.
  % (chain, mean_accept, target_accept))
100%|██████████| 2000/2000 [00:12<00:00, 160.72it/s]
plt.figure(figsize=(6,14))
forestplot(varying_slope_trace[1000:], varnames=['b'])_____no_output_____xvals = np.arange(2)
b = varying_slope_trace['a', 1000:].mean()
m = varying_slope_trace['b', 1000:].mean(axis=0)
for mi in m:
    plt.plot(xvals, mi*xvals + b, 'bo-', alpha=0.4)
plt.xlim(-0.2, 1.2);_____no_output_____

## Varying intercept and slope model

The most general model allows both the intercept and slope to vary by county:

$$y_i = \alpha_{j[i]} + \beta_{j[i]} x_{i} + \epsilon_i$$
_____no_output_____

with Model() as varying_intercept_slope:

    # Priors
    mu_a = Normal('mu_a', mu=0., sd=1e5)
    sigma_a = HalfCauchy('sigma_a', 5)
    mu_b = Normal('mu_b', mu=0., sd=1e5)
    sigma_b = HalfCauchy('sigma_b', 5)

    # Random intercepts
    a = Normal('a', mu=mu_a, sd=sigma_a, shape=counties)
    # Random slopes
    b = Normal('b', mu=mu_b, sd=sigma_b, shape=counties)

    # Model error
    sigma_y = Uniform('sigma_y', lower=0, upper=100)

    # Expected value
    y_hat = a[county] + b[county] * floor_measure

    # Data likelihood
    y_like = Normal('y_like', mu=y_hat, sd=sigma_y, observed=log_radon)_____no_output_____with varying_intercept_slope:
    varying_intercept_slope_trace = sample(2000, n_init=50000, tune=1000)Auto-assigning NUTS sampler...
Initializing NUTS using ADVI...
Average Loss = 1,093.1: 100%|██████████| 50000/50000 [00:19<00:00, 2551.95it/s]
Finished [100%]: Average Loss = 1,093.1
100%|██████████| 2000/2000 [00:17<00:00, 116.26it/s]
plt.figure(figsize=(6,16))
forestplot(varying_intercept_slope_trace[1000:], varnames=['a','b'])_____no_output_____xvals = np.arange(2)
b = varying_intercept_slope_trace['a', 1000:].mean(axis=0)
m = varying_intercept_slope_trace['b', 1000:].mean(axis=0)
for bi,mi in zip(b,m):
    plt.plot(xvals, mi*xvals + bi, 'bo-', alpha=0.4)
plt.xlim(-0.1, 1.1);_____no_output_____

## Adding group-level predictors

A primary strength of multilevel models is the ability to handle predictors on multiple levels simultaneously. If we consider the varying-intercepts model above:

$$y_i = \alpha_{j[i]} + \beta x_{i} + \epsilon_i$$

we may, instead of a simple random effect to describe variation in the expected radon value, specify another regression model with a county-level covariate. Here, we use the county uranium reading $u_j$, which is thought to be related to radon levels:

$$\alpha_j = \gamma_0 + \gamma_1 u_j + \zeta_j$$

$$\zeta_j \sim N(0, \sigma_{\alpha}^2)$$

Thus, we are now incorporating a house-level predictor (floor or basement) as well as a county-level predictor (uranium).

Note that the model has both indicator variables for each county, plus a county-level covariate. In classical regression, this would result in collinearity. In a multilevel model, the partial pooling of the intercepts towards the expected value of the group-level linear model avoids this.

Group-level predictors also serve to reduce group-level variation $\sigma_{\alpha}$. An important implication of this is that the group-level estimate induces stronger pooling._____no_output_____

from pymc3 import Deterministic

with Model() as hierarchical_intercept:

    # Priors
    sigma_a = HalfCauchy('sigma_a', 5)

    # County uranium model for slope
    gamma_0 = Normal('gamma_0', mu=0., sd=1e5)
    gamma_1 = Normal('gamma_1', mu=0., sd=1e5)

    # Uranium model for intercept
    mu_a = gamma_0 + gamma_1*u
    # County variation not explained by uranium
    eps_a = Normal('eps_a', mu=0, sd=sigma_a, shape=counties)
    a = Deterministic('a', mu_a + eps_a[county])

    # Common slope
    b = Normal('b', mu=0., sd=1e5)

    # Model error
    sigma_y = Uniform('sigma_y', lower=0, upper=100)

    # Expected value
    y_hat = a + b * floor_measure

    # Data likelihood
    y_like = Normal('y_like', mu=y_hat, sd=sigma_y, observed=log_radon)
    _____no_output_____with hierarchical_intercept:
    hierarchical_intercept_trace = sample(2000, n_init=50000, tune=1000)Auto-assigning NUTS sampler...
Initializing NUTS using ADVI...
Average Loss = 1,081.6: 100%|██████████| 50000/50000 [00:12<00:00, 3866.91it/s]
Finished [100%]: Average Loss = 1,081.6
100%|██████████| 2000/2000 [00:08<00:00, 228.08it/s]
a_means = hierarchical_intercept_trace['a', 1000:].mean(axis=0)
plt.scatter(u, a_means)
g0 = hierarchical_intercept_trace['gamma_0'].mean()
g1 = hierarchical_intercept_trace['gamma_1'].mean()
xvals = np.linspace(-1, 0.8)
plt.plot(xvals, g0+g1*xvals, 'k--')
plt.xlim(-1, 0.8)

a_se = hierarchical_intercept_trace['a', 1000:].std(axis=0)
for ui, m, se in zip(u, a_means, a_se):
    plt.plot([ui,ui], [m-se, m+se], 'b-')
plt.xlabel('County-level uranium'); plt.ylabel('Intercept estimate');_____no_output_____

The standard errors on the intercepts are narrower than for the partial-pooling model without a county-level covariate._____no_output_____### Correlations among levels

In some instances, having predictors at multiple levels can reveal correlation between individual-level variables and group residuals. We can account for this by including the average of the individual predictors as a covariate in the model for the group intercept.

$$\alpha_j = \gamma_0 + \gamma_1 u_j + \gamma_2 \bar{x} + \zeta_j$$

These are broadly referred to as ***contextual effects***._____no_output_____

# Create new variable for mean of floor across counties
xbar = srrs_mn.groupby('county')['floor'].mean().rename(county_lookup).values_____no_output_____with Model() as contextual_effect:

    # Priors
    sigma_a = HalfCauchy('sigma_a', 5)

    # County uranium model for slope
    gamma = Normal('gamma', mu=0., sd=1e5, shape=3)

    # Uranium model for intercept
    mu_a = Deterministic('mu_a', gamma[0] + gamma[1]*u.values + gamma[2]*xbar[county])

    # County variation not explained by uranium
    eps_a = Normal('eps_a', mu=0, sd=sigma_a, shape=counties)
    a = Deterministic('a', mu_a + eps_a[county])

    # Common slope
    b = Normal('b', mu=0., sd=1e15)

    # Model error
    sigma_y = Uniform('sigma_y', lower=0, upper=100)

    # Expected value
    y_hat = a + b * floor_measure

    # Data likelihood
    y_like = Normal('y_like', mu=y_hat, sd=sigma_y, observed=log_radon)
    _____no_output_____with contextual_effect:
    contextual_effect_trace = sample(2000, n_init=50000, tune=1000)Auto-assigning NUTS sampler...
Initializing NUTS using ADVI...
Average Loss = 1,116.3: 100%|██████████| 50000/50000 [00:12<00:00, 4108.64it/s]
Finished [100%]: Average Loss = 1,116.3
100%|██████████| 2000/2000 [00:11<00:00, 168.51it/s]
forestplot(contextual_effect_trace[1000:], varnames=['gamma'])_____no_output_____summary(contextual_effect_trace[1000:], varnames=['gamma'])
gamma:

  Mean             SD               MC Error         95% HPD interval
  -------------------------------------------------------------------

  1.424            0.047            0.003            [1.337, 1.519]
  0.697            0.081            0.003            [0.537, 0.853]
  0.395            0.183            0.010            [0.026, 0.754]

  Posterior quantiles:
  2.5            25             50             75             97.5
  |--------------|==============|==============|--------------|

  1.335          1.392          1.424          1.456          1.519
  0.537          0.645          0.696          0.752          0.853
  0.015          0.281          0.396          0.513          0.753

So, we might infer from this that counties with higher proportions of houses without basements tend to have higher baseline levels of radon. Perhaps this is related to the soil type, which in turn might influence what type of structures are built._____no_output_____### Prediction

Gelman (2006) used cross-validation tests to check the prediction error of the unpooled, pooled, and partially-pooled models

**root mean squared cross-validation prediction errors**:

* unpooled = 0.86
* pooled = 0.84
* multilevel = 0.79

There are two types of prediction that can be made in a multilevel model:

1. a new individual within an existing group
2. a new individual within a new group

For example, if we wanted to make a prediction for a new house with no basement in St. Louis county, we just need to sample from the radon model with the appropriate intercept._____no_output_____

county_lookup['ST LOUIS']_____no_output_____

That is,

$$\tilde{y}_i \sim N(\alpha_{69} + \beta (x_i=1), \sigma_y^2)$$

This is simply a matter of adding a single additional line in PyMC:_____no_output_____

with Model() as contextual_pred:

    # Priors
    sigma_a = HalfCauchy('sigma_a', 5)

    # County uranium model for slope
    gamma = Normal('gamma', mu=0., sd=1e5, shape=3)

    # Uranium model for intercept
    mu_a = Deterministic('mu_a', gamma[0] + gamma[1]*u.values + gamma[2]*xbar[county])

    # County variation not explained by uranium
    eps_a = Normal('eps_a', mu=0, sd=sigma_a, shape=counties)
    a = Deterministic('a', mu_a + eps_a[county])

    # Common slope
    b = Normal('b', mu=0., sd=1e15)

    # Model error
    sigma_y = Uniform('sigma_y', lower=0, upper=100)

    # Expected value
    y_hat = a + b * floor_measure

    # Data likelihood
    y_like = Normal('y_like', mu=y_hat, sd=sigma_y, observed=log_radon)

    # St Louis county prediction
    stl_pred = Normal('stl_pred', mu=a[69] + b, sd=sigma_y)
    _____no_output_____with contextual_pred:
    contextual_pred_trace = sample(2000, n_init=50000, tune=1000)Auto-assigning NUTS sampler...
Initializing NUTS using ADVI...
Average Loss = 1,116.3: 100%|██████████| 50000/50000 [00:12<00:00, 3890.22it/s]
Finished [100%]: Average Loss = 1,116.3
100%|██████████| 2000/2000 [00:15<00:00, 131.23it/s]
plot_posterior(contextual_pred_trace[1000:], varnames=['stl_pred'])_____no_output_____

## Benefits of Multilevel Models

Accounting for natural hierarchical structure of observational data

Estimation of coefficients for (under-represented) groups

Incorporating individual- and group-level information when estimating group-level coefficients

Allowing for variation among individual-level coefficients across groups

_____no_output_____## References

Gelman, A., & Hill, J. (2006). Data Analysis Using Regression and Multilevel/Hierarchical Models (1st ed.). Cambridge University Press.

Gelman, A. (2006). Multilevel (Hierarchical) modeling: what it can and cannot do. Technometrics, 48(3), 432–435._____no_output_____


# Notebook from AvijeetPrasad/laputas-blog
Path: book/notebooks/.ipynb_checkpoints/negatons-checkpoint.ipynb

_____no_output_____# Gravity of Negative Mass_____no_output_____## Summary
The gravitational interaction has mass of only one sign as the source that is conventionally taken to be positive. The existence of negative mass is not precluded in either Newtonian gravity or general relativity. Negative masses have been invoked in many recent papers to understand the dominance of repulsive dark energy that is accelerating the universe. Here we discuss the various gravitating properties of negative mass particles and their implications for black holes, cosmology, and fundamental physics. Inconsistencies in the earlier works are also discussed. We study the possibility of such negative mass particles accounting for accelerated expansion of the universe. We set constraints on the density of these negative mass particles based on astrophysical observations._____no_output_____

# Import relevant packages and constants
from astropy import units as u
from astropy.constants.si import c, G, m_p, M_sun, hbar,h, k_B
from astropy.cosmology import WMAP9 as cosmo
import math
import matplotlib.pyplot as plt
import numpy as np
#c = speed of light
#m_p =  rest mass of proton
#G =  universal gravatational constant
#M_sun = Solar mass
# Hubble's constant = cosmo.H(0)
# hbar = Reduced Planck's constant_____no_output_____

## 1. Introduction
It is an empirical fact that unlike in electromagnetic interactions where we have both positive and negative electric charges, the gravitational interaction has mass of only one sign as the source. This mass (or energy) is taken to be positive, implying that there are no negative masses in nature. In electromagnetism there is a similar asymmetry in the absence of a monopole which has not been observed, which is not prohibited by theory. Negative mass is quite consistent with gravitational physics [1-3], both Newtonian gravity and general relativity do not preclude its existence. There has not been much discussion about the gravitational effects of negative masses and their other properties. However many recent papers have invoked the possible presence of negative masses to understand cosmic conundrums like the dominance of repulsive dark energy (DE) accelerating the universe [4-6].
_____no_output_____## 2. Implications of negative mass sources
Negative mass particles with a number density $n$, of mass $m$, moving with velocity $v$ would exert a negative pressure, $P\sim -nmv^2$. It has been pointed out that Einstein himself interpreted the cosmological constant as perhaps due to empty space consisting of gravitating negative masses distributed all over interstellar space [4, 7]. As other fundamental interactions involve charges of both signs as sources of the field, some authors have felt it odd that gravitational charges, conventionally called masses, appear to consist of only positive sign.
The gravitating properties of negative mass particles (negatons), assuming they exist, would be very interesting and in contrast to the usual positive mass particles. To begin with, the equivalence principle would imply that all test particles whether of positive or negative mass would fall in the field of a positive mass source (like the Earth). This is because we have $-ma=GM(-m)/r^2$ , with -m cancelling out (for negatons both inertial and gravitational masses are negative). However two negatons would gravitationally repel each other so that a collection of such particles would distribute themselves, almost homogenously in the Universe. A negative mass source would repel all particles with both positive and negative mass, whereas positive mass source will attract both. A pair consisting of a positive and negative mass would try to chase each other, but problem is more involved if the particles are moving. In this case it has been shown [2] that the particles move away from each other and the proper separation as measured by an observer commoving with one of the particles increases roughly as $\gamma d$, where $\gamma$ is the Lorentz factor, the separation becoming indefinitely large as $v\rightarrow c$.
There is symmetry between positive and negative masses in many equations in physics. For instance, relativistic wave equations for particles of any spin are form invariant for particles with both positive and negative mass. This is true for Klein-Gordon, Dirac equations as well as linearized massive spin-2 Pauli-Fierz equation. Dirac equation, well known to have negative energy solutions, interpreted as positrons in negative energy sea. In quantum mechanics it is known that the centre of a wave packet of negative energy solutions describes a uniform motion with velocity in opposite direction to momentum thus behaving like negatons [8]. As far as spin-1/2 particles are concerned, the chirality transformation described by $\psi=\gamma_5 \psi$, converts the Dirac equation for positive mass into one for negative mass. If $\psi (x)$ is the solution for the positive mass case, $\gamma_5 \psi(x)$ is the solution for the negative mass case, the chirality transformation being effectively equivalent to mass inversion, $m\rightarrow -m$ [9].
Moreover, negative effective masses occur in many physical contexts like semiconductors [10], Casimir effect [11], Hawking radiation [12] modelled as virtual negatons falling into black hole (BH), excitation of phantom DE as negatons [13], Bose-Einstein condensate [14]. However here we shall mainly be concerned with the gravitating properties of negatons and their implications for BH physics and cosmology.
_____no_output_____## 3. Sign of G for negative mass sources
A common error made in several discussions of the gravity of negative masses, including recent papers [4-6, 15] is to retain the same sign for the gravitational constant G. This would lead to inconsistent results. This is easily seen for instance in considering the gravitational self-energy of mass M with a spatial extent $R$, i.e.

$E=-\frac{(GM^2)}{R} \quad (1)$

If $M$ is positive, $E$ is the negative gravitational binding energy implying an attractive force. However if $M$ is negative, $M^2$ is positive, but to have a positive binding energy appropriate for a repulsive interaction, $G$ must also change sign. Thus we should have a negative $G$ in discussing the gravitational interaction of negatons.
_____no_output_____### 3.1 Einstein’s field equations with negative mass
The change in sign of $G$ is also to be carried over in discussing the general relativity field equations. Contrary to the inconsistent discussion in recent literature [4, 15] the GR field equations for matter consisting of both positive and negative masses is:

$R_{\mu \nu} - \frac{1}{2} g_{\mu \nu} R = \frac{8\pi G}{c^4}T^+_{\mu \nu} - \frac{8\pi G}{c^4}T^-_{\mu \nu} = \kappa T^+_{\mu\nu} - \kappa T^-_{\mu\nu} \quad (2)$

where $T^+_{\mu \nu}$ and $T^-_{\mu \nu}$ are the energy-momentum tensors for positive and negative masses. The pressure term would be $P=+nmv^2$ or $P=-nmv^2$. The Friedmann equation for the scale factor $a(t)$ would be (for $k=0$):

$\left(\frac{\dot{a}}{a}\right)^2=\frac{8\pi G}{3}\rho_+-\frac{8\pi G}{3}\rho_- \quad (3)$

(Equal energy densities of positive and negative masses implies a static Universe)
As is known, inflation scenarios with initial exponential expansion are driven by negative pressure of quantum vacuum. The quantum vacuum has an equation of state for positive mass particles,

$P=-\rho c^2 \quad (4)$
This gives rise to an accelerating universe, as seen from:

$\ddot{a}=-\frac{GM}{a^2} =-\frac{G}{a^2}\left(\frac{4\pi}{3} \left(\rho+\frac{3P}{c^2} \right) a^3 \right) \quad (5)$

For equation of state given by equation (4), $\ddot{a}$ reverses sign.
_____no_output_____### 3.2 Inflation with negative mass

Can a quantum vacuum of negatons give rise to an accelerated expansion? Here conservation of energy would imply,

$P-\rho c^2=0 \Rightarrow P=+\rho c^2 \quad (6)$

However if sign of $G$ is also reversed in equation (5) it is seen that $\ddot{a}$ is positive. This implies that in the initial epoch, the quantum vacuum energy of either positive or negative mass particles or both (if present) can give rise to an inflationary scenario. An effective negative energy (mass) can arise in the early Universe by considering torsion in the early Universe via Einstein-Cartan theory [16, 17, 19] and vortex solutions of this theory could lead to DE [18].

One can discuss the gravitational interactions of both positive and negative mass particles consistently if the sign of $G$ is opposite for negatons. This leads to reasonable results. If we consider a body of mass $M$, with a small particle of mass $\Delta m$ slowly brought from infinity to the surface of $M$, net energy is:

$M+\Delta m - \frac{M(\Delta m) G}{r c^2} = M' \quad (7)$

In all the different cases: $M$, $\Delta m$ positive, $M$, $\Delta m$ negative, or $M$ positive, $\Delta m$ negative, it turns out that Buchdahl's theorem is satisfied, i.e. the maximum energy which can be extracted is either $+Mc^2$ or $-Mc^2$.

Negatons will have a negative kinetic energy, which in the relativistic case is $-mc^2/\sqrt{1-v^2/c^2}$, but a positive gravitational potential energy. Thus in the Newtonian case:

$-\frac{1}{2} mv^2 + \frac{GMm}{R}=0 \quad (8)$

i.e. negative masses can escape from positive or negative mass sources with escape velocity and have trajectories prescribed by conics. Negatons always have $v\leq c$, like usual positive mass particles but with $m$ replaced by $-m$, a real but negative proper mass. Tachyons in contrast would have imaginary positive proper mass, with $v \geq c$.
_____no_output_____

# plot the kinetic energy for different mass
ke = lambda v: -m_p*c**2/np.sqrt(1-(v**2/c**2))
v = np.linspace(.1,.99)*c
plt.plot(v/c,ke(v),'b')
plt.xlabel('v/c')
plt.ylabel('KE')
plt.grid(True)
plt.show()_____no_output_____

### 3.3 Negatons as possible candidates for dark matter and dark energy

Gravitationally, dark matter (DM) behaves like baryonic matter, i.e. they have $P=+nm_{DM} v^2$. Therefore DM cannot consist dominantly of negatons. In any case the flux of negatons required to significantly affect masses and evolution of supermassive black holes and even large objects (large capture cross section) like red supergiants is very large. For a billion solar mass black hole, the flux required to significantly alter its mass over the Hubble time ($t_H \approx 4.45\times10^{17} s$) is given as:

$f=\frac{M_{BH}}{(m \times t_H \times A_{BH})}  \approx 2.44\times10^{22} m^{-2} s^{-1} \quad (9)$

where, $A_{BH} \approx 10^{26} m^2$ is the horizon area of the black hole, and the negaton mass, $m$ is taken to be of the order of the proton mass. And similarly we can work out the flux required to alter the evolution of red supergiants as, $f \approx 10^{16} m^{-2} s^{-1}$. These are much higher than the expected DM density of $\approx 2\times 10^{-27} kgm^{-3}$. So negative mass particles cannot be a source of DM. And in any case their energy density being negative, they will not be suitable candidates.  _____no_output_____

m_bh = 1.e9 * M_sun
a_bh = (16 * math.pi * G**2 * m_bh**2) / c**4
t_h = 1/cosmo.H(0).decompose()
flux = m_bh / (t_h * a_bh * m_p)
print("Hubble constant = {0:.2e}".format(cosmo.H(0)))
print("Hubble time = {0:.2e}".format(t_h))
print("flux = {0:.2e}".format(flux))Hubble constant = 6.93e+01 km / (Mpc s)
Hubble time = 4.45e+17 s
flux = 2.44e+22 1 / (m2 s)

But dark energy (DE) can result from negatons through $P=-nmc^2$ (negative $\rho$ implying repulsive gravity). So density of such particles giving rise to DE can be estimated as follows. Assuming a separation between the particles of a Compton length, i.e. $\hbar/mc$, which is also positive for negatons (since both ℏ and m are negative) (negatons also obey the quantum uncertainty principle), the density is given by, $\frac{m}{(\hbar/mc)^3} =\frac{m^4 c^3}{\hbar ^3}$ , which is negative (since $m^4$ is positive and $\hbar^3$ is negative for negatons). Equating this to the observed DE density, $\rho_{DE}\approx 7\times 10^{-27} kgm^{-3}$, we can estimate the mass of these particles as:

$m=\left(\frac{\rho_{DE} \hbar^3}{c^3}\right)^{1/4}\approx 4\times 10^{-39} kg \quad (10)$

For these negative mass particles to make up dark energy of density $\approx 7\times 10^{-27} kgm^{-3}$, the number density is given by, $n \times m \approx 7\times 10^{-27} kgm^{-3}$. This gives:

$n\approx 10^{12} m^{-3} \quad (11)$

At earlier epochs (high $z$), the matter (and DM) densities would have been higher and as the Universe expands, the negative pressure due to the negatons starts to dominate, hence accounting for the accelerated expansion of the Universe observed at the present epoch.
_____no_output_____

rho_de = 7.e-27*u.kg*u.m**-3
m = ((rho_de * hbar**3) / c**3)**(1/4)
print("mass of negatons = {0:.2e}".format(m.decompose()))mass of negatons = 4.18e-39 kg
n = rho_de / m
print("number density of negatons = {0:.2e}".format(n.decompose()))number density of negatons = 1.68e+12 1 / m3

## 4. Consequences of negative mass
Flux of negatons falling on a compact positive mass can reduce its mass. For instance a positive mass black hole (BH) accreting negatons could presumably end up having a negative mass. Here we have some consistency with cosmic censorship.
_____no_output_____### 4.1 Consequences for black holes
The effective mass of a Kerr-Newman BH is, $M_{eff}=M-a^2/r-Q^2/r$, so $M_{eff}$ can become negative if $a^2$, $Q^2$ reach extremal values. So also for charged BH. Extremal BH have $M^2=a^2+Q^2$, etc. By accreting negatons, the BH can become of effective negative mass. However once we have a negative mass BH it would not further capture any particles (both positive and negative) as a negative mass source repels all particles. Such a negative mass BH would also be described by the Schwarzschild solution (as both $G$,$M$ are reversed in sign). So it would have a horizon radius, $R=2GM/c^2$ . This would ensure symmetry of GR (and its solutions) under mass reversal, $M\rightarrow -M$.

Can we distinguish a negative mass BH from a positive mass one? A negative mass BH cannot gather or accrete matter. So there is no quasar like activity. But it can still emit negative energy Hawking radiation. The Hawking temperature would be $T=(hc^3)/8\pi k_B GM$, but the quantum of action is negative for negatons. So it is seen $T$ is negative. Indeed to avoid a Boltzmann exponential blow up, negatons must have a negative temperature. However the entropy of a negative mass BH would be positive as, $S\propto (GM^2)/\hbar c$ ($G$,$\hbar$ negative, $M^2$ positive). This is consistent with second law of thermodynamics which cannot be violated. The evaporation lifetime would also be positive, given by $t=(G^2 M^3)/(\hbar c^4)$. With a negative T, black body emission energy is negative as $E=aT^4,a=k^5/(45\hbar^3 c^3)$, $k$ being positive and $\hbar$ negative. The entropy of the black body radiation is however positive ($\propto aT^3$, both negative). Again consistent with thermodynamics.
_____no_output_____

# Calculate the Hawking temperature
T = lambda M: (h*c**3) / (8*np.pi*k_B*G*M)
n = np.linspace(24,40)
M = 10.**n*u.kg

# Calculate the evapouration timescale
t = lambda M: (G**2*M**3)/(hbar*c**4)

# Hawking tempertaure vs Mass
fig, (ax1, ax2) = plt.subplots(1, 2,figsize=(10,5))
ax1.plot(n,np.log10(T(M).value),lw=2,c='b')
ax1.set_xlabel('log10 M (kg)')
ax1.set_ylabel('log10 T (K)')
ax1.grid(True)

# Evapouration timescale vs Mass
ax2.plot(n,np.log10(t(M).value),lw=2,c='r')
ax2.set_xlabel('log10 M (kg)')
ax2.set_ylabel('log10 t (s)')
ax2.grid(True)
plt.show()_____no_output_____

### 4.2 Charge on negatons

If negatons have an electric charge, the charge squared, i.e. $e^2$ is negative. With this charge, negatons can form atoms (with the usual Bohr radii) but with negative energy levels. Also Larmor radiation energy would be negative (proportional to $a^2 e^2$, $a$ the acceleration) which implies that the energy would not be unbounded from below which is unphysical.
In order to avoid reducing significantly the mass of SMBH or red supergiant stars (over a billion years) the limit background energy density in interstellar space of such negatons is easily shown to be $\rho <|10^{-25} g cm^{-3}|$, consistent with DM density. Like in an electrostatic plasma, a background density of both positons and negatons can give rise to a long range gravitational screening analogous to Debye screening. In Newtonian case, such screening can be described by a modified Poisson equation,

$\nabla ^2 \phi - \frac{1}{\lambda_D^2}\phi=-4\pi G nm \delta(r) \quad (12)$

This will give rise to $\lambda_D \approx 2\pi \nu / (4 \pi Gnm)^{1/2}$ , which would be expected to be of the order of several kiloparsec, with $nm=\rho \approx 10^{-25} gcm^{-3}$. Such a modified potential (over kiloparsec distance) was used in [17, 18] to obtain flat rotation curves. Presence of such a distribution of negatons over megaparsec scales could prevent formation of galaxies, possibly accounting for presence of large voids free of galactic systems.
_____no_output_____## 5. Negaton bound states

Even negative mass can form bound states similar to Bohr's atom. The radius of the orbit is given as

$r = \frac{4\pi\epsilon_0 n^2 \hbar^2}{m e^2} \quad (13)$.

Here the mass $m$  is negative and $\hbar^2$ is positive. So, for the radius to be positve, equation (13) implies $e^2$ is negative. Therefore the chrage on the negaton $e_{neg} = i e$.

For the bound state we have the centripetal force balancing the Coloumb force ($F_{cen} = F_{Col}$) , i.e.,

$ \frac{m v^2}{r} = \frac{e^2}{4\pi \epsilon_0 r^2} \quad (14).$

For negatons, the negative sign of $m$ will cancel out the negative sign of $e^2$, hence we have the usual expression for kinetic energy from equation (14) as

$\frac{1}{2}mv^2= \frac{e^2}{8 \pi \epsilon_0 r} \quad (15)$.

The energy of the bound state is then given by

$E = \frac{m e^4}{8\epsilon_0^2n^2h^2} \quad (16)$.

Here, $e^4$ is positive, $h^2$ is positive and $m$ is negative._____no_output_____##6. Conclusion and future directions

Many recent papers have invoked negative mass as a possible model for dark energy. Here we have brought out some of the inconsistencies in these earlier work while considering the various gravitating properties of negative mass particles, and show that for a negative mass source (to have a positive binding energy associated with repulsive interaction) $G$ must change to $-G$. These discussions are carried over to the field equations in general relativity, making the field equations – for matter consisting of both positive and negative masses – consistent. We have also shown that the quantum vacuum of negatons, in the initial epoch, also gives rise to an accelerated expansion (inflationary scenario).

In short, the presence of negative masses (not ruled out by gravitational theories including GR) and their interesting gravitational interactions opens up a whole host of exciting possibilities for astrophysics, cosmology and expands our basic concepts in theoretical physics [17-19]. In particular, the implications for formation of black holes and their associated thermodynamics (including Hawking radiation), consequences for other compact objects (like white dwarfs and neutron stars), as well as for cosmic structure formation would be of much interest.
_____no_output_____## References:
[1] Jammer, M.: Concepts of Mass in Classical and Modern Physics. Harvard Univ. Press, Cambridge (1961)

[2] Bondi, H.: Negative mass in general relativity. Rev. Mod. Phys. 29, 423 (1957)

[3] Terletsky, Ya.P.: Positive, negative and imaginary rest masses. J. Phys. Radium, 23, 910 (1963)

[4] Farnes, J.S.: A unifying theory of dark energy and dark matter: Negative masses and matter creation within a modified ΛCDM framework. Astron. Astrophys. 620, A92 (2018)

[5] Petit, J.P., d’Agostini, G.: Negative mass hypothesis in cosmology and the nature of dark energy. Astrophys. Space Sci. 354, 611 (2014)

[6] Benoit-Lévy, A., Chardin, G.: Introducing the Dirac-Milne universe. Astron. Astrophys. 537, A78 (2012)

[7] Einstein, A.: Comment on Schrödinger’s Note “On a System of Solutions for the Generally Covariant Gravitational Field Equations”. Physikalische Zeitschrift, 19, 165 (1918). Translated by Engel A. in The Collected Papers of Albert Einstein, Vol. 7, The Berlin Years: Writings, 1918–1921 (Princeton University Press).

[8] Messiah, A.: Quantum mechanics. Interscience Publishers, New York (1961)

[9] Sakurai, J.J.: Mass reversal and weak interactions. Il Nuovo Cimento. 7, 649 (1958)

[10] Ashcroft, N.W., Mermin, N.D.: Solid State Physics. Harcourt, New York (1976)

[11] Morris, M.S. et al.: Wormholes, time machines, and the weak energy condition. Phys. Rev. Lett. 61, 1446 (1988)

[12] Hawking, S.W.: Particle creation by black holes. Comm. Math. Phys. 43, 199 (1975)

[13] Caldwell, R.R.: A phantom menace? Cosmological consequences of a dark energy component with super-negative equation of state. Phys. Lett. B. 545, 23 (2002)

[14] Winterberg, F.: Remark concerning the gravitational interaction of matter and anti-matter. Il Nuovo Cimento. 19, 186 (1961)

[15] Forward, R.L.: Negative matter propulsion. J. Propul. Power. 6, 28 (1990)

[16] Buchdahl, H.A.: General relativistic fluid spheres. Phys. Rev. 116, 1027 (1959)

[17] de Sabbata, V., Sivaram, C.: Spin and Torsion in Gravitation. World Scientific, Singapore, (1994)

[18] Sivaram, C., Arun, K.: Some enigmatic aspects of the early universe. Astrophys. Space Sci. 334, 225 (2011)

[19] Sivaram, C., Arun, K.: Primordial rotation of the universe, hydrodynamics, vortices and angular momenta of celestial objects. Open Astron. J. 5, 7 (2012)
_____no_output_____


# Notebook from cnyambura/NovoNordisk_Capstone
Path: archive/dev_jay/aa_one_hot_and_features_encode-classify.ipynb

import pandas as pd
import numpy as np
import os, sys, subprocess
import h5py
import json
import matplotlib.pyplot as plt_____no_output_____#import data from Sastry et. al.
DATA_DIR = '../dataframes/'

DF_prest = pd.concat([pd.read_csv(DATA_DIR+'DF_prest_features_1.csv',index_col=0),
                      pd.read_csv(DATA_DIR+'DF_prest_features_2.csv',index_col=0),
                      pd.read_csv(DATA_DIR+'DF_prest_features_3.csv',index_col=0),
                      pd.read_csv(DATA_DIR+'DF_prest_features_4.csv',index_col=0)])
print ('Number of peptides:', len(DF_prest))
Number of peptides: 45206
DF_prest.head(5)_____no_output_____aa_features = ['list_comp_A','list_comp_R','list_comp_N',
               'list_comp_D','list_comp_C','list_comp_Q','list_comp_E','list_comp_G',
               'list_comp_H','list_comp_I','list_comp_L','list_comp_K','list_comp_M',
               'list_comp_F','list_comp_P','list_comp_S','list_comp_T','list_comp_W',
               'list_comp_Y','list_comp_V',
               # Amino acid types
               'frac_aliphatic', 'frac_uncharged_polar', 'frac_polar',
               'frac_hydrophobic', 'frac_positive', 'frac_sulfur', 'frac_negative', 'frac_amide',
               'frac_alcohol']

disorder_features = ['disembl_COILS_frac','disembl_REM465_frac','disembl_HOTLOOPS_frac','ronn_avg','ronn_frac',
                    'disopred_avg','disopred_pb_avg','disopred_frac','disopred_pb_frac']

phys_features = ['aa_len',
                # Physical properties
                 'bio_pI','bio_mW','bio_aromaticity','bio_instability','bio_gravy','abs_avg_charge','abs_charge',
                 'avg_charge']

rna_features = ['AAA','AAC','AAT','AAG','ACA','ACC','ACT','ACG','ATA','ATC','ATT','ATG','AGA','AGC','AGT',
                'AGG','CAA','CAC','CAT','CAG','CCA','CCC','CCT','CCG','CTA','CTC','CTT','CTG','CGA','CGC',
                'CGT','CGG','TAA','TAC','TAT','TAG','TCA','TCC','TCT','TCG','TTA','TTC','TTT','TTG','TGA',
                'TGC','TGT','TGG','GAA','GAC','GAT','GAG','GCA','GCC','GCT','GCG','GTA','GTC','GTT','GTG',
                'GGA','GGC','GGT','GGG',
                'GC_content','list_nuc_A','list_nuc_C','list_nuc_G','list_nuc_T',
                'GC30',
                # SD sequences
                'sd_like_fwd','sd_like_fwd_frac','sd_like_rev','sd_like_rev_frac','sd_seq_fwd',
                'sd_seq_fwd_frac','sd_seq_rev','sd_seq_rev_frac',
                # RNA folding
                'RNA_folding_energy','RNA_40_energy','tAI']

ss_features = ['acc20_mean','ss_helix', 'ss_ext', 'ss_c', 'ss8_helix', 'ss8_ext', 'ss8_turn', 'ss8_helix3',
               'ss8_pi_helix', 'ss8_bridge', 'ss8_bend', 'ss8_coil', 'acc_frac',
               'in_gravy','out_gravy', 'acc_hydrophilic_in','acc_hydrophilic_out',
               'acc_hydrophobic_in','acc_hydrophobic_out']

list_of_features = aa_features+disorder_features+phys_features+rna_features+ss_features_____no_output_____

# Do this with the top 11 features according to Pearson Correlation instead_____no_output_____

# list_of_features = ['frac_polar', 'ss8_ext', 'ss_ext', 'list_comp_Y',
#        'frac_uncharged_polar', 'list_comp_L', 'ss8_helix', 'ss_helix', 'in_gravy', 'frac_hydrophobic',
#        'bio_pI']_____no_output_____df_features= pd.DataFrame()
df_features['prest_id'] = DF_prest['prest_id']
df_features['conc_cf'] = DF_prest['conc_cf']
df_features[list_of_features] = DF_prest[list_of_features]
df_features.head()_____no_output_____

# One hot encoding of AA sequence_____no_output_____

# list of amino acid single letter abbreviations from https://en.wikipedia.org/wiki/Amino_acid
#aa_list = ['A', 'R', 'N', 'D', 'C', 'E', 'Q', 'G', 'H', 'I', 'L', 'K', 'M', 'F', 'P', 'S', 'T', 'W', 'Y', 'V']

# amino acid list in order of hydropathy score from https://en.wikipedia.org/wiki/Hydrophilicity_plot
aa_list = ['I', 'V', 'L', 'F', 'C', 'M', 'A', 'G', 'T', 'S', 'W', 'Y', 'P', 'H', 'E', 'Q', 'D', 'N', 'K', 'R']_____no_output_____

Make a dictionary for the encoding (with abbreviations sorted in alpabetical order)_____no_output_____

# create a dictionary
features = {}
for idx, aa in enumerate(aa_list):
    one_hot = np.zeros(len(aa_list))
    one_hot[idx] = 1
    features[aa] = list(one_hot)  _____no_output_____

In order to avoid the need to pad the data later we will make all one hot encodings the same length_____no_output_____

max(DF_prest['aa_len'])_____no_output_____def aa_to_one_hot_matrix(aa_seq):

    one_hot_matrix = np.zeros([149, 20]) # maximum aa_seq length X number of amino acids
    for idx, aa in enumerate(aa_seq):
        one_hot_matrix[idx] = np.array(features[aa])

    return one_hot_matrix_____no_output_____

Make a new dataframe that only has the info needed for the one hot encoding and apply the encoding function to the entire dataset_____no_output_____

df_features['aa_one_hot'] = DF_prest['aa_seq'].apply(aa_to_one_hot_matrix)_____no_output_____

Vizulaize an example sequence.  Note that the right hand side is empty (this is padding so all data is the same shape)_____no_output_____

ind = 5000
plt.figure(figsize=(12, 8))
plt.imshow(np.rot90(df_features['aa_one_hot'][ind]))_____no_output_____

Split the data into high and low expression_____no_output_____

df_features['conc_cf'].hist(bins=100)
plt.axvline(df_features['conc_cf'].quantile(.75), color='k', linestyle='dashed', linewidth=3)
plt.axvline(df_features['conc_cf'].quantile(.25), color='r', linestyle='dashed', linewidth=3)_____no_output_____

# Classify as high or low expression_____no_output_____

# identify high and low classes by conc_cf quantiles
low_cut = df_features['conc_cf'].quantile(0.25)
high_cut = df_features['conc_cf'].quantile(0.75)

low_exp_df = df_features[df_features['conc_cf'] < low_cut]
high_exp_df = df_features[df_features['conc_cf'] > high_cut]

# set classes for high vs. low expression# set cl
low_exp_df.insert(0, 'class', 0)
high_exp_df.insert(0, 'class', 1)
#doing it this was causes a warning
# low_exp_df['class'] = 0
# high_exp_df['class'] = 1

print(low_exp_df.shape, high_exp_df.shape)

features_highlow = pd.concat([low_exp_df,high_exp_df])
(11302, 151) (11301, 151)
features_highlow.head()_____no_output_____features_highlow.to_hdf('features_highlow.h5', 'data')/Users/prg/miniconda3/lib/python3.6/site-packages/pandas/core/generic.py:1471: PerformanceWarning:
your performance may suffer as PyTables will pickle object types that it cannot
map directly to c-types [inferred_type->mixed,key->block2_values] [items->['aa_one_hot']]

  return pytables.to_hdf(path_or_buf, key, self, **kwargs)