# Notebook from jtiagosg/batch3-students
Path: S04 - Text Classification/BLU08 - Dimensionality Reduction/BLU08 - Learning Notebook - Part 1 of 3 - Dimensionality Reduction and Feature Selection.ipynb

## Part 1 - Dimensionality Reduction and Feature Selection

Thinking in high dimensions is particularly hard. A 1-D dot would hardly be able to imagine a 2D world. In the same way, the square below struggles to think of our 3D world. For us, it is very difficult to imagine dimensions above the 3rd. We have proxies for thinking of a 4th dimension - passing of time, a 3D surface with some extra measure represented by color, etc - but this starts to become really difficult to work above the 4th dimension.

![dimensionality](./media/flatland.png)

Thankfully, when dealing with ML problems, you can go above three features without trying to imagine how they would look like (Lucky you!). However, you still need to understand how the number of dimensions might affect - either positively or negatively - your models and how to work with high dimensional spaces. Don't worry, we will guide you through it._____no_output_____

import string
import re
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pylab import barh,plot,yticks,show,grid,xlabel,figure,cla,close
from nltk.tokenize import WordPunctTokenizer

from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn import svm
from sklearn.metrics import accuracy_score
from sklearn.feature_selection import SelectFromModel
from sklearn.feature_selection import SelectKBest, chi2_____no_output_____

## 1. The curse of dimensionality

So far you learned how to handle text data by transforming it into a vectorized feature space. Namely, we mostly used preprocessing tricks and some sort of count over words - remember the CountVectorizer and TF-IDF - to generate our feature space. However, as you might have realized, the number of features for these problems is huge, in particular if you want to cover all language. In the limit, your feature space will cover the entire vocabulary!

The effect of high dimensionality features when modelling these problems is called the **curse of dimensionality**. You can probably already see some of the effects of these curse, right?

* To start with, more features, will obviously mean a longer training process
* On another hand, a higher dimensionality can actually hurt the classifier performance

#### But I thought "more features" meant "more accuracy"...

This is not always true. When you have high dimensionality problems, the use of all the features might actually hurt your model. The model becomes more prone to overfitting and might have worse accuracy in points outside your training data._____no_output_____
## 2.  Basic Feature Selection

One of the ways you might think of to reduce your feature dimensionality is by performing feature selection. We are going to walk you through some methods with an actual example. Let's start by loading some data - we are going to use the twitter dataset of republican and democrat tweets_____no_output_____### 2.1 - Get your dataset

Start by importing the dataset. _____no_output_____

df = pd.read_csv('./datasets/twitter_rep_dem_data_small.csv')_____no_output_____

### 2.2 - Get to know our problem

We'll first learn our categories and see a few examples of how our training data looks like. _____no_output_____

print('Categories:')
print(', '.join(set(df.Party)))Categories:
Democrat, Republican

Check class balances_____no_output_____

df.Party.value_counts()_____no_output_____

Pretty balanced!_____no_output_____

df.head(10)_____no_output_____

You'll notice that our tweets are just text with some particularities. For example, it is common to have twitter handles in the text, defined by the "@" character. Our data also has three columns, but we are going to ignore the *Handle* column for now and just focus on classifying *Tweets* with *Party* labels._____no_output_____### 2.3 - Feature Extraction

Our data is simply raw text, each element a document to be classified, with a corresponding label, which is the Party of the tweet. Pretty simple, right?

Since you are a great student, you went thoroughly through BLU07 and you already know how to prepare, handle text and extract some simple features. So let's process our data and use TfidfVectorizer with a range of 1-2 ngrams to get us some simple features.

First let's apply simple tokenization and remove punctuation. To avoid overfitting to twitter related information, like the handles tagged in the messages, let's remove those, so our focus is only on the language._____no_output_____

handle_removal = lambda doc: re.subn(r'@\w+','', doc.lower())[0]
df['Tweet'] = df['Tweet'].map(handle_removal)

simple_tokenizer = lambda doc: " ".join(WordPunctTokenizer().tokenize(doc))
df['Tweet'] = df['Tweet'].map(simple_tokenizer)

df.head(10)_____no_output_____

Now let's split our data and apply some vectorization. Let's pick a random seed so the results are replicable._____no_output_____

seed = 42_____no_output_____

_____no_output_____

train_data, test_data = train_test_split(df, test_size=0.3, random_state=seed)

print('Training examples: {}'.format(train_data.size))
print('Test examples: {}\n'.format(test_data.size))

# important: only train the vectorizer on the training data
vectorizer = TfidfVectorizer(ngram_range=(1,2))
%timeit vectorizer.fit(train_data.Tweet)

X_train = vectorizer.transform(train_data.Tweet)
X_test = vectorizer.transform(test_data.Tweet)

y_train = train_data.Party
y_test = test_data.PartyTraining examples: 38889
Test examples: 16668

1.2 s ± 183 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

### 2.4 - Getting our baseline

Let's get our baseline accuracy and measure the time it takes to fit a Naive Bayes Model, a model you should be familiar with and which in NLP comes hand in hand with the Bag Of Words representation (if you don't get the joke below, eventually you should go read about naive bayes).

_____no_output_____

clf =  MultinomialNB()
%timeit clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)
print('Accuracy: {}'.format(accuracy_score(y_pred, y_test)))47.1 ms ± 4.42 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7595392368610511

### 2.5 - Feature selection

Now that we have our baseline, let's start by looking into the number of features used in our classifier:_____no_output_____

X_train.shape_____no_output_____

So far so good, and our classifier even trained pretty fast, but a 140K-dimensional space is obviously very difficult to interpret (think back to the fact that even 4D is difficult for us to grasp). Let's instead try to extract our K most important words. One way that you might think to do this is actually to just get the features corresponding to the most frequent terms. In fact, our TfidfVectorizer already has an option for that. Let's see the impact on our training speed and accuracy._____no_output_____

for k in [10, 100, 1000, 5000, 10000, 50000, 100000]:
    print('Using {} features'.format(k))
    print('----'.format(k))

    X_train = train_data.Tweet
    X_test = test_data.Tweet

    vectorizer_truncated = TfidfVectorizer(ngram_range=(1,2), max_features=k)
    vectorizer_truncated.fit(X_train)

    X_train_truncated = vectorizer_truncated.transform(X_train)
    X_test_truncated = vectorizer_truncated.transform(X_test)

    clf =  MultinomialNB()
    %timeit clf.fit(X_train_truncated, y_train)
    y_pred = clf.predict(X_test_truncated)

    print('Accuracy: {}\n'.format(accuracy_score(y_pred, y_test)))Using 10 features
----
39.3 ms ± 7.6 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.5259179265658748

Using 100 features
----
30.6 ms ± 953 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.5815334773218143

Using 1000 features
----
30.8 ms ± 1.19 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.6637868970482361

Using 5000 features
----
38.9 ms ± 8.89 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7177825773938085

Using 10000 features
----
32.8 ms ± 3.98 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7341612670986322

Using 50000 features
----
33.2 ms ± 1.88 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7543196544276458

Using 100000 features
----
32.7 ms ± 357 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7543196544276458

Ok, so no amazing effects. Let's look into the actual top K-features to see if they make sense._____no_output_____

K=10
vectorizer_truncated = TfidfVectorizer(ngram_range=(1,2), max_features=K)
vectorizer_truncated.fit(X_train)
feature_names = vectorizer_truncated.get_feature_names()
for f in feature_names:
    print(f)and
co
for
https
https co
in
of
rt
the
to

As you can see, the top 10 features are basically meaningless when thinking of our classes. In the next cell you will see that the counts of these words are balanced between classes._____no_output_____

sub_punct = lambda s: re.sub(r'\s+\W*[a-z]?\W*\s+', ' ', s)_____no_output_____for feature in feature_names:
    print('Documents that contain the word "{}"'.format(feature))
    print('----')
    docs = X_train.apply(sub_punct).str.lower().str.contains(feature)
    print(str(y_train[docs].value_counts()) + '\n\n')Documents that contain the word "and"
----
Democrat      2410
Republican    2334
Name: Party, dtype: int64

Documents that contain the word "co"
----
Democrat      5571
Republican    5493
Name: Party, dtype: int64

Documents that contain the word "for"
----
Republican    2307
Democrat      2188
Name: Party, dtype: int64

Documents that contain the word "https"
----
Democrat      4818
Republican    4702
Name: Party, dtype: int64

Documents that contain the word "https co"
----
Democrat      4775
Republican    4616
Name: Party, dtype: int64

Documents that contain the word "in"
----
Republican    5103
Democrat      5101
Name: Party, dtype: int64

Documents that contain the word "of"
----
Democrat      2294
Republican    2235
Name: Party, dtype: int64

Documents that contain the word "rt"
----
Republican    2755
Democrat      2670
Name: Party, dtype: int64

Documents that contain the word "the"
----
Democrat      4191
Republican    4139
Name: Party, dtype: int64

Documents that contain the word "to"
----
Democrat      4287
Republican    4182
Name: Party, dtype: int64

Tip: Notice that most of these words are normally considered stopwords. Try to exclude stopwords from the TfidfVectorizer and see what new top features you obtain. You will probably see that there will be some common words and some "political discourse" words, but not really democrat/republican specific.

But let's move on to more meaningful approaches._____no_output_____## 3.  Feature Selection through statistical analysis

Basic feature selection methods might actually work sometimes, in particular if you pick a reasonable heuristic to decide on which features to choose. In our case, obviously, just picking the higher counts is not that good, since it does not provide any useful information on the labels. But you could imagine for example trying to pick as features words that appear only (or almost only) on one of our classes.

Although this might seem a good idea, depending on your problem, you would probably not want to lose that much time thinking about heuristics, implementing them and comparing them. This is where statistical tests are useful. You don't have to reason on the features you are using, these tests use your data to provide insights on your features.

##### Chi-squared test

The chi-squared test is one of these tests. The chi-squared formula measures how much expected counts and observed counts of variables/distributions deviate from each other. It can be used to test for independce between two variables, like defined by the equation below, where $O_{x_1x_2}$ is the observation of the conjuction of variables and $E_{x_1x_2}$ the corresponding expected value, this is, the expected value given our hypothesis *$H_0$: the variables are independent*.

$$\chi^2 = \sum{\frac{(O_{x_1x_2} - E_{x_1x_2})^2}{E_{x_1x_2}}}$$

For feature selection we want to test the independence of our features from the class labels. In our particular case, we define $x_1=t$ as our term or word and $x_2=c$ as our class label. A small chi-squared value will mean that the term is closer to independence from the class, and a big value that it is very dependent on the class.

Knowing the details of the chi-squared can be useful for you, but in the context of our BLU is not our primary goal and there are more useful methods for text features that you will learn about in the following notebooks. However, we provide at the end of this BLU a more detailed explanation of this test, and some examples, if you wish to understand it better (see Annex A).

We will lean on the previous example and show you how chi-squared would help us select more meaningful features._____no_output_____## 3.1 - Setup problem

Like before, let's fetch our data, extract its features, run a baseline, and move from there._____no_output_____

stat_df = pd.read_csv('./datasets/twitter_rep_dem_data_small.csv')

stat_df['Tweet'] = stat_df['Tweet'].map(handle_removal)
stat_df['Tweet'] = stat_df['Tweet'].map(simple_tokenizer)

stat_train_data, stat_test_data = train_test_split(stat_df, test_size=0.3, random_state=seed)

stat_vectorizer = TfidfVectorizer(ngram_range=(1,2))
%timeit stat_vectorizer.fit(stat_train_data.Tweet)

stat_X_train = stat_vectorizer.transform(stat_train_data.Tweet)
stat_X_test = stat_vectorizer.transform(stat_test_data.Tweet)

stat_y_train = stat_train_data.Party
stat_y_test = stat_test_data.Party

stat_clf =  MultinomialNB()
%timeit stat_clf.fit(stat_X_train, stat_y_train)

stat_pred = stat_clf.predict(stat_X_test)

print('Accuracy: {}'.format(accuracy_score(stat_pred, stat_y_test)))985 ms ± 60.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
34.6 ms ± 899 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7595392368610511

## 3.2 - Feature Selection from chi-squared

We will now use the chi2 and obtain some chi-squared values for our features. _____no_output_____

chi_values, p_values = chi2(stat_X_train, stat_y_train)_____no_output_____

We can plot the most dependent features from the chi-squared values._____no_output_____

feature_names = stat_vectorizer.get_feature_names()

cla()   # Clear axis
close() # Close a figure window

figure(figsize=(12,10))
zipped_chi_squared = zip(feature_names, chi_values)
sorted_chi_values = sorted(zipped_chi_squared, key=lambda x:x[1])
top_chi_values = list(zip(*sorted_chi_values[-30:]))

x = range(len(top_chi_values[1]))
labels = top_chi_values[0]
barh(x, list(top_chi_values)[1], align='center', alpha=.2, color='g')
yticks(x, labels)
xlabel('$\chi^2$')
show()_____no_output_____

Actually, scikit-learn already provides a function to directly select the K-best features for our model, so we are going to use that to extract our most important features. We can confirm that the top features selected match the ones with the highest chi-values_____no_output_____

ch2 = SelectKBest(chi2, k=10)
ch2.fit(stat_X_train, stat_y_train)

most_important_features = [feature_names[i] for i in ch2.get_support(indices=True)]
for f in most_important_features:
    print(f)chairman
code
hearing
reform
tax
tax reform
taxcutsandjobsact
taxreform
texas
the taxcutsandjobsact

Now we're getting somewhere, these new features are starting to make sense. We can look into their distribution of  over our documents to get a sense of the relation they share with the labels of the dataset._____no_output_____

for feature in most_important_features:
    print('Documents that contain the word "{}"'.format(feature))
    print('----')
    docs = stat_train_data['Tweet'].apply(sub_punct).str.contains(feature)
    print(str(stat_y_train[docs].value_counts()) + '\n\n')Documents that contain the word "chairman"
----
Republican    196
Democrat        8
Name: Party, dtype: int64

Documents that contain the word "code"
----
Republican    107
Democrat        8
Name: Party, dtype: int64

Documents that contain the word "hearing"
----
Republican    264
Democrat       62
Name: Party, dtype: int64

Documents that contain the word "reform"
----
Republican    426
Democrat       34
Name: Party, dtype: int64

Documents that contain the word "tax"
----
Republican    800
Democrat      217
Name: Party, dtype: int64

Documents that contain the word "tax reform"
----
Republican    126
Democrat        6
Name: Party, dtype: int64

Documents that contain the word "taxcutsandjobsact"
----
Republican    102
Name: Party, dtype: int64

Documents that contain the word "taxreform"
----
Republican    227
Democrat        2
Name: Party, dtype: int64

Documents that contain the word "texas"
----
Republican    121
Democrat        1
Name: Party, dtype: int64

Documents that contain the word "the taxcutsandjobsact"
----
Republican    86
Name: Party, dtype: int64

As you can see, some words are much more common in Republican tweets compared to Democrat tweets. These features are thus much more interpretable and might have a better impact on training._____no_output_____

for k in [10, 100, 1000, 5000, 10000, 50000, 100000, 'all']:
    print('Using {} features'.format(k))
    print('----'.format(k))

    ch2_train = SelectKBest(chi2, k=k)
    ch2_train.fit(stat_X_train, stat_y_train)
    X_train_chi = ch2_train.transform(stat_X_train)
    X_test_chi = ch2_train.transform(stat_X_test)

    clf = MultinomialNB()
    %timeit clf.fit(X_train_chi, stat_y_train)

    y_pred = clf.predict(X_test_chi)

    print('Accuracy: {}\n'.format(accuracy_score(y_pred, stat_y_test)))

Using 10 features
----
29.8 ms ± 648 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.4744420446364291

Using 100 features
----
32 ms ± 8.22 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.5984521238300936

Using 1000 features
----
32.9 ms ± 8.05 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.709143268538517

Using 5000 features
----
30.2 ms ± 1.57 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7321814254859611

Using 10000 features
----
30.4 ms ± 2.38 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7368610511159107

Using 50000 features
----
31.3 ms ± 1.81 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7496400287976962

Using 100000 features
----
31.9 ms ± 1.01 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7566594672426206

Using all features
----
33.9 ms ± 1.05 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Accuracy: 0.7595392368610511

Using 100000 features we get only slightly better results. However, looking at the features themselves, we can clearly see this is a better feature selection method than the previous one, but we could still use a bit more gain and speed improvements.

One thing that feature selection does not take into consideration is feature interaction, which can limit a bit the gains in performance. In the remaining of this BLU you will learn about more elaborate methods to perform dimensionality reduction, in particular some very recent methods that are the standard in text-related tasks. _____no_output_____## 4. Final remarks

After reading this notebook you should understand:

 - What is the curse of dimensionality
 - How can you perform simple feature selection by reasoning about your problem
 - How to apply statistical methods for feature selection by finding dependencies between features and labels

Keep in mind that predicting in the real world is much less theoretical. The performance of these methods will depend a lot on your problem, the size of your dataset, your model choice, your preprocessing. You can use feature selection to improve speed, avoid overfitting, or even just to interpret your features and how they interact with your classes.

-----

**Suggestion**: try to vary the following options/parameters and analyze its impact:

- Experiment a bit more with your preprocessing and see its impact
- Try other feature extraction options such as the simple CountVectorizer
- Use smaller slices of the dataset to see how the dataset size impacts both baseline and feature selected results
- Experiment with other classifiers and the impact of the dimensionality reduction on these
- Try to use the model to evaluate other text data, search for republican/democrat posts/news/blogs and see how well your model classifies each, for example political speeches.

-----

Although we focused on simple heuristics and in the chi-squared method, there are other features selection methods that you might find useful. Some examples of these are:

- Feature selection through variance (sklearn.feature_selection.VarianceThreshold)
- Feature selection through mutual information
- Recursive feature elimination
- Tree-based feature selection

**And remember, these methods just tell us that there is a relation between labels and features, but not the nature of that relation.** Now go and apply these methods!

![correlation](./media/correlation.png)

_____no_output_____-------------

## Annex A.  Details on chi-squared

Let's do a really quick example for you to understand how this works. Let's say we are modeling how characteristics from star trek characters that appear in an episode are related to their death, and among our features we have one particular called "has red t-shirt" which can take only two categorical values: Yes/No.

Let's build a table representing this scenario:

|  Has red t-shirt    | Dies   | Does not die | Total |
|----------------------|--------|--------------|-------|
|         Yes          |   63   |      9       |  72   |
|         No           |   13   |     40       |  53   |
|         total        |   76   |     49       |  125  |

This is what we call a contigency table, and it contains our observed values. Testing for independence of the variables, our expected value is computed from the probabilities $N*P(x_1x_2) = N*P(x_1)P(x_2) $, and we get:

|  Has red t-shirt    | Dies   | Does not die |
|----------------------|--------|--------------|
|         Yes          |  43.78 $$(125 * \frac{76}{125}\frac{72}{125})$$ | 28.22 $$(125 * \frac{49}{125}\frac{72}{125})$$ |
|         No           |  32.22 $$(125 * \frac{76}{125}\frac{53}{125})$$ | 20.78 $$(125 * \frac{49}{125}\frac{53}{125})$$ |

Now we can use these expected values to compute chi-squared using the formula $\chi^2 = \sum{\frac{(O_{x_1x_2} - E_{x_1x_2})^2}{E_{x_1x_2}}}$

|  Has red t-shirt    | Dies   | Does not die |
|----------------------|--------|--------------|
|         Yes          |  8.44 $$(\frac{(63-43.78)^2}{43.78})$$ | 13.09 $$(\frac{(9-28.22)^2}{28.22})$$ |
|         No           |  11.47 $$(\frac{(13-32.22)^2}{32.22})$$ | 17.79 $$(\frac{(40-20.78)^2}{20.78})$$ |

Summing these values, we get a chi-squared of 50.79, which corresponds to a p-value < .00001, which means we are dealing with dependent variables.

_____no_output_____### Chi-squared in BoW context

But how about our context? Our variables are not categorical, so how do we compute this? Actually the chi-squared value can be extended to frequencies, building a contingency table from the feature values and class label. Starting from a table of word frequencies:

|                      |            $C_0$              | ... |             $C_j $               | ... |                         Total               |
|----------------------|---------------------|-----|-------------------------|-----|----------------------------|
|   Word 0             |  $$C_{t_0, c_0}$$   | ... |  $$C_{t_0, c_1}$$       | ... | $$ \sum_j{C_{t_0, c_j}} $$ |
|   Word 1             |  $$C_{t_1, c_0}$$   | ... |  $$C_{t_1, c_1}$$       | ... | $$ \sum_j{C_{t_1, c_j}} $$ |
|    ...               |   ...               | ... |     ...                 | ... |        ...                 |
|   Word i             |  $$C_{t_i, c_0}$$   | ... |  $$C_{t_i, c_1}$$       | ... | $$ \sum_j{C_{t_i, c_j}} $$ |
|    ...               |   ...               | ... |     ...                 | ... |        ...                 |
|   Total       | $$ \sum_i{C_{t_i, c_0}} $$ | ... |$$\sum_i{C_{t_i, c_j}} $$| ... | $$ N = \sum_{i,j}{C_{t_i, c_j}} $$|

We can take each feature ($t=x_i$) and class ($c=c_j$) and assume a table of the form:

|                      | Class j   | Not Class j |  Total |
|----------------------|-----------|-------------|--------|
|   Word i             |  $C_{tc}$ |   $C_{tx}$  |  $C_{tc}$ + $C_{tx}$ |
|   Not Word i         |  $C_{xc}$ |   $C_{xx}$  |  $C_{xc}$ + $C_{xx}$ |
|   Total      |  $C_{tc}$ + $C_{xc}$ |   $C_{tx}$ + $C_{xx}$  |  N = $C_{tc}$ + $C_{xc}$ + $C_{tx}$ + $C_{xx}$ |

Where:

- $C_{tc}$ : counts of co-ocurrences of the term and class
- $C_{tx}$ : counts of ocurrences of the term but not the class
- $C_{xc}$ : counts of ocurrences of the class but not the term
- $C_{xx}$ : counts of ocurrences outside the class and without the term

Notice that you can compute your negative word counts ("Not Word i") by using the totals:

$$C_{xc} = \sum_i{C_{t_i, c}} - C_{tc}  \quad\quad C_{tx} = \sum_j{C_{t, c_j}} - C_{tc} \quad\quad C_{xx} = N - C_{tc} - C_{tx} - C_{xc} $$

The expression can be expanded to the following, for each term $t$ and class $c$:

$$\chi^2(t, c) = \frac{N(C_{tc}C_{xx}-C_{tx}C_{xc})^2}{(C_{tc}+C_{xc})(C_{tx}+C_{xx})(C_{tc}+C_{tx})(C_{xc}+C_{xx})}$$

-----

**Sugestion**: If, like me, you can't move forward without understanding the origin of these expressions, try to get from the initial expression to this expanded form. However, if you get stuck on the math, just go into the notebook **Learning Notebook - Optional - Chi-squared math** and follow the demonstration.

-----

And we will get for all our terms and classes the chi-squared values indication correlation

|                      | Republican ($C_0$)  | Democrat ($C_1$) |
|----------------------|-----------|-------------|
|   Word 0             |  $$\chi^2(t_0, c_0)$$ |  $$\chi^2(t_0, c_1)$$ |
|   Word 1             |  $$\chi^2(t_1, c_0)$$ |  $$\chi^2(t_1, c_1)$$ |
|    ...               |   ...     |     ...     |
|   Word i             |  $$\chi^2(t_i, c_0)$$ |  $$\chi^2(t_i, c_1)$$ |
|    ...               |   ...     |     ...     |

### Chi-squared and TF-IDF

You've seen how to apply chi-squared for categorical values and now for frequencies, more specifically for word frequencies. But we were applying it to TF-IDF values, values that seem to violate the chi-squared rules. The reason why we can apply the chi-squared to TF-IDF values is actually because these are just weighted/scaled frequencies and the probabilities and totals should add up.

### Implementation, finally!

Let's apply this and write a function that receives a matrix with term counts for each label._____no_output_____

def chi_squared(counts):
    """
    Non vectorized version of chi squared function - the idea is that you see the relation with the formula above,
    but you should never use such an inefficient version when actually performing a chi-squared analysis
    """
    print("Applying chi-squared to {} feature and {} classes".format(counts.shape[0], counts.shape[1]))
    chi_values = np.zeros(counts.shape)
    for i in range(counts.shape[0]):
        for j in range(counts.shape[1]):
            n = counts.sum()
            c_tc = counts[i,j]
            c_tx = counts.sum(axis=1)[i,0]-c_tc
            c_xc = counts.sum(axis=0)[0,j]-c_tc
            c_xx = n-c_tc-c_tx-c_xc
            chi_values[i,j] = n*(((c_tc*c_xx)-(c_tx*c_xc))**2)/((c_tc+c_xc)*(c_tx+c_xx)*(c_tc+c_tx)*(c_xc+c_xx))
    return chi_values

def chi_squared_vect(counts):
    """
    Vectorized version of chi squared function - this is still a non-optimized version, but it should run faster than
    the previous function
    """
    print("Applying chi-squared to {} feature and {} classes".format(counts.shape[0], counts.shape[1]))
    n = counts.sum()
    c_tc = counts
    c_tx = counts.sum(axis=1)-counts
    c_xc = counts.sum(axis=0)-counts
    c_xx = n * np.ones(counts.shape) - counts - c_tx - c_xc
    num = n * np.square(np.multiply(c_tc, c_xx)-np.multiply(c_tx, c_xc))
    den = np.multiply(np.multiply(np.multiply(c_tc+c_xc, c_tx+c_xx), c_tc+c_tx), c_xc+c_xx)
    chi_values = np.divide(num, den)
    return chi_values_____no_output_____

### Applying to our previous example

Now we'll apply our functions to a small portion of our data (first 100 tweets) since our implementation is not optimized, in particular the non-vectorized example. If we find the features with higher chi-values, we find our more important features, which are the ones with less independence._____no_output_____

small_vectorizer = TfidfVectorizer(ngram_range=(1,2))
small_vectorizer.fit(train_data.Tweet)
small_X_train = small_vectorizer.transform(train_data.Tweet)
small_y_train = train_data.Party

idx_rep = np.where(small_y_train=='Republican')
idx_dem = np.where(small_y_train=='Democrat')

counts_rep = small_X_train[idx_rep[0], :].sum(axis=0)
counts_dem = small_X_train[idx_dem[0], :].sum(axis=0)
counts = np.concatenate((counts_rep, counts_dem))

# chi_values = chi_squared(counts.transpose())
chi_values_vect = chi_squared_vect(counts.transpose())

feature_names = small_vectorizer.get_feature_names()

best_features = chi_values_vect.argsort(axis=0).tolist()

print("Most important features:\n")
for idx in sorted(best_features[-10:]):
    print(u"{}, value: {}".format(feature_names[idx[0]], chi_values_vect[idx[0], 0]))
Applying chi-squared to 138409 feature and 2 classes
Most important features:

chairman, value: 24.846272877542923
code, value: 12.933225558706729
hearing, value: 14.19673500702519
reform, value: 14.104207101906178
tax, value: 22.33857645946327
tax reform, value: 14.347639988872228
taxcutsandjobsact, value: 14.522494937205447
taxreform, value: 26.959234731744832
texas, value: 15.279411340041444
the taxcutsandjobsact, value: 12.734023819258736

Awesome, we got to the same results as the scikit-learn! Time to move on to the next notebook!_____no_output_____


# Notebook from rsinghlab/SCOT
Path: .ipynb_checkpoints/unbalanced_GW_SNAREseq-checkpoint.ipynb

## Unbalanced Gromov-Wasserstein for SNARE-seq example

To simulate imbalance between cell types, we subsample the ATAC dataset and try to align to the full RNA dataset._____no_output_____

import src.utils as ut
import src.evals as evals
from src.scot import *
from sklearn.cluster import SpectralClustering
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA_____no_output_____X=np.load("data/scatac_feat.npy")
y=np.load("data/scrna_feat.npy")
print("Dimensions of input datasets are: ", "X= ", X.shape, " y= ", y.shape)Dimensions of input datasets are:  X=  (1047, 19)  y=  (1047, 10)
X=ut.unit_normalize(X)
y=ut.unit_normalize(y)_____no_output_____

Can't find the cell-type information for now, so assign cluster labels based on spectral clustering. This seems relatively clean..._____no_output_____

pca=PCA(n_components=2)
y_pca = pca.fit_transform(y)

clustering = SpectralClustering(n_clusters = 4, assign_labels = "discretize", random_state = 0).fit(y_pca)
plt.scatter(y_pca[:, 0], y_pca[:, 1], s = 4, c = clustering.labels_)_____no_output_____

Now subsample the ATAC data according to class proportions `props`_____no_output_____

props = [0.5, 1, 0.75, 0.5]
# props = [1, 1, 1, 1]
index = np.concatenate([np.where(clustering.labels_ == i)[0][0:int((clustering.labels_ == i).sum() * props[i])] for i in range(len(props))])
X_subsamp = X[index, :]

X_subsamp_pca = pca.fit_transform(X_subsamp)
plt.scatter(X_subsamp_pca[:, 0], X_subsamp_pca[:, 1], s = 4, c = clustering.labels_[index])_____no_output_____

Align using balanced and unbalanced Gromov-Wasserstein_____no_output_____

k=50
e=0.0005
gamma_bal, _ = scot(X_subsamp, y, k, e, mode="connectivity", metric="correlation", returnCoupling = True)It.  |Err
-------------------
    0|2.512857e-03|
   10|9.632699e-04|
   20|2.201648e-04|
   30|9.146903e-05|
   40|5.387046e-05|
   50|4.225666e-05|
   60|3.867948e-05|
   70|3.756788e-05|
   80|3.603410e-05|
   90|3.265532e-05|
  100|2.762011e-05|
  110|2.198783e-05|
  120|1.672659e-05|
  130|1.233497e-05|
  140|8.915730e-06|
  150|6.364020e-06|
  160|4.507601e-06|
  170|3.177430e-06|
  180|2.233025e-06|
  190|1.566256e-06|
It.  |Err
-------------------
  200|1.097156e-06|
  210|7.678714e-07|
  220|5.370787e-07|
  230|3.754859e-07|
  240|2.624269e-07|
  250|1.833662e-07|
  260|1.281013e-07|
  270|8.948096e-08|
  280|6.249785e-08|
  290|4.364833e-08|
  300|3.048220e-08|
  310|2.128664e-08|
  320|1.486465e-08|
  330|1.037988e-08|
  340|7.248070e-09|
  350|5.061123e-09|
  360|3.534005e-09|
  370|2.467654e-09|
  380|1.723055e-09|
  390|1.203129e-09|
It.  |Err
-------------------
  400|8.400864e-10|
gamma_unbal, _ = scot(X_subsamp, y, k, 1e-3, rho = 0.05, mode="connectivity", metric="correlation", returnCoupling = True, balanced = False)_____no_output_____X_new_unbal = ut.transport_data(X_subsamp, y, gamma_unbal, transposeCoupling=False)
X_new_bal = ut.transport_data(X_subsamp, y, gamma_bal, transposeCoupling=False)
y_new = y_____no_output_____

We have aligned the ATAC dataset (subsampled) onto the RNA dataset (full)._____no_output_____

pca=PCA(n_components=2)
Xy_pca_bal=pca.fit_transform(np.concatenate((X_new_bal, y_new), axis=0))
X_pca_bal=Xy_pca_bal[0:X_new_bal.shape[0],]
y_pca_bal=Xy_pca_bal[X_new_bal.shape[0]:,]
Xy_pca_unbal=pca.fit_transform(np.concatenate((X_new_unbal, y_new), axis=0))
X_pca_unbal=Xy_pca_unbal[0:X_new_unbal.shape[0],]
y_pca_unbal=Xy_pca_unbal[X_new_unbal.shape[0]:,]

### Read in cell type information:
# Xlabels=np.genfromtxt("data/SNAREseq_typesChromatin.txt")
# ylabels=np.genfromtxt("data/SNAREseq_typesExpression.txt")

fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize = (15, 5))
ax1.scatter(y_pca_bal[:,0], y_pca_bal[:,1], s=15, label="Gene Expression", alpha = 0.5, c = clustering.labels_, marker = "+")
ax1.set_title("Expression")

ax2.scatter(X_pca_unbal[:,0], X_pca_unbal[:,1], s=15, label="Chromatin Accessibility", alpha = 0.5, c = clustering.labels_[index], marker = "^")
ax2.set_title("ATAC, unbalanced")

ax3.scatter(X_pca_bal[:,0], X_pca_bal[:,1], s=15, label="Chromatin Accessibility", alpha = 0.5, c = clustering.labels_[index], marker = "^")
ax3.set_title("ATAC, balanced")_____no_output_____

For a (row-normalised) coupling $P$, we can compute the expected class membership error:
$$\mathbf{E}_{(x, y) \sim P} \mathbf{1}\{ \mathrm{class}(x) \ne \mathrm{class}(y) \}$$_____no_output_____

P_unbal = (gamma_unbal.T/gamma_unbal.sum(1)).T
mean_err_unbal = (P_unbal*(clustering.labels_[index, None] != clustering.labels_[None, :])).sum(1).mean()
P_bal = (gamma_bal.T/gamma_bal.sum(1)).T
mean_err_bal = (P_bal*(clustering.labels_[index, None] != clustering.labels_[None, :])).sum(1).mean()_____no_output_____print("unbalanced: %f, balanced: %f" % (mean_err_unbal, mean_err_bal))unbalanced: 0.138904, balanced: 0.254218


# Notebook from zeochoy/RNASeq_CollaborativeFilter
Path: ipynb/tcga_training_normal.ipynb

# TCGA RNA-Seq RSEM normalized Collaborative Filtering_____no_output_____

%reload_ext autoreload
%autoreload 2
%matplotlib inline

from fastai.learner import *
from fastai.column_data import *

import pandas as pd
import numpy as np_____no_output_____path='~/data/tcga/'_____no_output_____

## Preprocessing_____no_output_____### subset_____no_output_____

df = pd.read_csv(path+'full.csv', sep=',', low_memory=False, index_col=[0], error_bad_lines=False); df.head()_____no_output_____dupabbr = ['KIPAN', 'COADREAD', 'STES', 'GBMLGG']
df = df.loc[~df['cancer'].isin(dupabbr)]
sids = list(df.index)
df = df.assign(sid=sids)
df = df.loc[df.sid.str.split('-', expand=True).loc[:,3].str[0] == str(1)]_____no_output_____df = df.reset_index(); df = df.drop(columns=['index']); df.head()_____no_output_____len(df)_____no_output_____

### Melt_____no_output_____

mdf = pd.melt(df, id_vars=['sid', 'cancer'], var_name='gene', value_name='log2exp'); mdf.head()_____no_output_____len(mdf)_____no_output_____#mdf.to_csv(path+'melted.csv')_____no_output_____

## Collaborative filtering_____no_output_____

#mdf = pd.read_feather(path+'melted')_____no_output_____#mdf.head()_____no_output_____val_idxs = get_cv_idxs(len(mdf))
wd=2e-4
n_factors = 50_____no_output_____cd = CollabFilterDataset.from_data_frame(path, mdf, 'sid', 'gene', 'log2exp')
#cf = CollabFilterDataset.from_csv(path, 'melted.csv', 'sid', 'gene', 'log2exp')
learn = cd.get_learner(n_factors, val_idxs, 64, opt_fn=optim.Adam)_____no_output_____learn.lr_find()_____no_output_____learn.sched.plot(100)_____no_output_____lr=1e-4
learn.fit(lr, 2, cycle_len=1, cycle_mult=2)_____no_output_____preds = learn.predict()_____no_output_____y=learn.data.val_y
# import seaborns as sns
sns.jointplot(preds, y, kind='hex', stat_func=None);/home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "
/home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "
learn.save('tcga_collab_normal_new')_____no_output_____#learn.load('tcga_collab_normal')_____no_output_____

## Analyze results_____no_output_____#### shortcut to retrieve learner, model, df_____no_output_____

df = pd.read_csv(path+'full.csv', sep=',', low_memory=False, index_col=[0], error_bad_lines=False); df.head()
dupabbr = ['KIPAN', 'COADREAD', 'STES', 'GBMLGG']
df = df.loc[~df['cancer'].isin(dupabbr)]
sids = list(df.index)
df = df.assign(sid=sids)
df = df.loc[df.sid.str.split('-', expand=True).loc[:,3].str[0] == str(1)]
df = df.reset_index(); df = df.drop(columns=['index']); df.head()
mdf = pd.melt(df, id_vars=['sid', 'cancer'], var_name='gene', value_name='log2exp'); mdf.head()
val_idxs = get_cv_idxs(len(mdf))
wd=2e-4
n_factors = 50
cd = CollabFilterDataset.from_data_frame(path, mdf, 'sid', 'gene', 'log2exp')
learn = cd.get_learner(n_factors, val_idxs, 64, opt_fn=optim.Adam)
learn.load('tcga_collab_normal_new')_____no_output_____genes = list(df.columns[:-2])
sids = list(df['sid'])
cancers = list(df['cancer'])
sid_ca = {s:c for s, c in zip(sids, cancers)}_____no_output_____

### Embedding interpretation_____no_output_____### retrieve embeddings _____no_output_____

m=learn.model; m.cuda()_____no_output_____

#### gene embedding_____no_output_____

geneidx = np.array([cd.item2idx[g] for g in genes])
gene_emb = to_np(m.i(V(geneidx)))
gene_emb.shape_____no_output_____gene_emb_df = pd.DataFrame(gene_emb, index=genes); gene_emb_df.head()_____no_output_____gene_emb_df.to_csv(path+'gene_emb_normal_new.csv', sep=',')_____no_output_____gene_emb_bias = to_np(m.ib(V(geneidx)))
gene_emb_bias_df = pd.DataFrame(gene_emb_bias, index=genes); gene_emb_bias_df.head()
gene_emb_bias_df.to_csv(path+'gene_emb_normal_new_bias.csv')_____no_output_____

#### sample embedding _____no_output_____

sampleidx = np.array([cd.user2idx[sid] for sid in sids])
samp_emb = to_np(m.u(V(sampleidx)))
samp_emb.shape_____no_output_____samp_emb_df = pd.DataFrame(samp_emb, index=sids); samp_emb_df.head()_____no_output_____samp_emb_df.to_csv(path+'samp_emb_normal_new.csv', sep=',')_____no_output_____samp_emb_bias = to_np(m.ub(V(sampleidx)))
samp_emb_bias_df = pd.DataFrame(samp_emb_bias, index=sids); samp_emb_bias_df.head()
samp_emb_bias_df.to_csv(path+'samp_emb_normal_new_bias.csv')_____no_output_____


# Notebook from drammock/mne-tools.github.io
Path: stable/_downloads/c3c186a71be1cfa94a34ecef5331099f/plot_brainstorm_phantom_elekta.ipynb

%matplotlib inline_____no_output_____

# Brainstorm Elekta phantom dataset tutorial

Here we compute the evoked from raw for the Brainstorm Elekta phantom
tutorial dataset. For comparison, see [1]_ and:

    https://neuroimage.usc.edu/brainstorm/Tutorials/PhantomElekta

References
----------
.. [1] Tadel F, Baillet S, Mosher JC, Pantazis D, Leahy RM.
       Brainstorm: A User-Friendly Application for MEG/EEG Analysis.
       Computational Intelligence and Neuroscience, vol. 2011, Article ID
       879716, 13 pages, 2011. doi:10.1155/2011/879716
_____no_output_____

# sphinx_gallery_thumbnail_number = 9

# Authors: Eric Larson <larson.eric.d@gmail.com>
#
# License: BSD (3-clause)

import os.path as op
import numpy as np
import matplotlib.pyplot as plt

import mne
from mne import find_events, fit_dipole
from mne.datasets.brainstorm import bst_phantom_elekta
from mne.io import read_raw_fif

from mayavi import mlab
print(__doc__)_____no_output_____

The data were collected with an Elekta Neuromag VectorView system at 1000 Hz
and low-pass filtered at 330 Hz. Here the medium-amplitude (200 nAm) data
are read to construct instances of :class:`mne.io.Raw`.

_____no_output_____

data_path = bst_phantom_elekta.data_path(verbose=True)

raw_fname = op.join(data_path, 'kojak_all_200nAm_pp_no_chpi_no_ms_raw.fif')
raw = read_raw_fif(raw_fname)_____no_output_____

Data channel array consisted of 204 MEG planor gradiometers,
102 axial magnetometers, and 3 stimulus channels. Let's get the events
for the phantom, where each dipole (1-32) gets its own event:

_____no_output_____

events = find_events(raw, 'STI201')
raw.plot(events=events)
raw.info['bads'] = ['MEG1933', 'MEG2421']_____no_output_____

The data have strong line frequency (60 Hz and harmonics) and cHPI coil
noise (five peaks around 300 Hz). Here we plot only out to 60 seconds
to save memory:

_____no_output_____

raw.plot_psd(tmax=30., average=False)_____no_output_____

Our phantom produces sinusoidal bursts at 20 Hz:

_____no_output_____

raw.plot(events=events)_____no_output_____

Now we epoch our data, average it, and look at the first dipole response.
The first peak appears around 3 ms. Because we low-passed at 40 Hz,
we can also decimate our data to save memory.

_____no_output_____

tmin, tmax = -0.1, 0.1
bmax = -0.05  # Avoid capture filter ringing into baseline
event_id = list(range(1, 33))
epochs = mne.Epochs(raw, events, event_id, tmin, tmax, baseline=(None, bmax),
                    preload=False)
epochs['1'].average().plot(time_unit='s')_____no_output_____

Let's use a `sphere head geometry model <ch_forward_spherical_model>`
and let's see the coordinate alignment and the sphere location. The phantom
is properly modeled by a single-shell sphere with origin (0., 0., 0.).

_____no_output_____

sphere = mne.make_sphere_model(r0=(0., 0., 0.), head_radius=0.08)

mne.viz.plot_alignment(epochs.info, subject='sample', show_axes=True,
                       bem=sphere, dig=True, surfaces='inner_skull')_____no_output_____

Let's do some dipole fits. We first compute the noise covariance,
then do the fits for each event_id taking the time instant that maximizes
the global field power.

_____no_output_____

# here we can get away with using method='oas' for speed (faster than "shrunk")
# but in general "shrunk" is usually better
cov = mne.compute_covariance(epochs, tmax=bmax)
mne.viz.plot_evoked_white(epochs['1'].average(), cov)

data = []
t_peak = 0.036  # true for Elekta phantom
for ii in event_id:
    # Avoid the first and last trials -- can contain dipole-switching artifacts
    evoked = epochs[str(ii)][1:-1].average().crop(t_peak, t_peak)
    data.append(evoked.data[:, 0])
evoked = mne.EvokedArray(np.array(data).T, evoked.info, tmin=0.)
del epochs
dip, residual = fit_dipole(evoked, cov, sphere, n_jobs=1)_____no_output_____

Do a quick visualization of how much variance we explained, putting the
data and residuals on the same scale (here the "time points" are the
32 dipole peak values that we fit):

_____no_output_____

fig, axes = plt.subplots(2, 1)
evoked.plot(axes=axes)
for ax in axes:
    ax.texts = []
    for line in ax.lines:
        line.set_color('#98df81')
residual.plot(axes=axes)_____no_output_____

Now we can compare to the actual locations, taking the difference in mm:

_____no_output_____

actual_pos, actual_ori = mne.dipole.get_phantom_dipoles()
actual_amp = 100.  # nAm

fig, (ax1, ax2, ax3) = plt.subplots(nrows=3, ncols=1, figsize=(6, 7))

diffs = 1000 * np.sqrt(np.sum((dip.pos - actual_pos) ** 2, axis=-1))
print('mean(position error) = %0.1f mm' % (np.mean(diffs),))
ax1.bar(event_id, diffs)
ax1.set_xlabel('Dipole index')
ax1.set_ylabel('Loc. error (mm)')

angles = np.rad2deg(np.arccos(np.abs(np.sum(dip.ori * actual_ori, axis=1))))
print(u'mean(angle error) = %0.1f°' % (np.mean(angles),))
ax2.bar(event_id, angles)
ax2.set_xlabel('Dipole index')
ax2.set_ylabel(u'Angle error (°)')

amps = actual_amp - dip.amplitude / 1e-9
print('mean(abs amplitude error) = %0.1f nAm' % (np.mean(np.abs(amps)),))
ax3.bar(event_id, amps)
ax3.set_xlabel('Dipole index')
ax3.set_ylabel('Amplitude error (nAm)')

fig.tight_layout()
plt.show()_____no_output_____

Let's plot the positions and the orientations of the actual and the estimated
dipoles

_____no_output_____

def plot_pos_ori(pos, ori, color=(0., 0., 0.), opacity=1.):
    """Plot dipole positions and orientations in 3D."""
    x, y, z = pos.T
    u, v, w = ori.T
    mlab.points3d(x, y, z, scale_factor=0.005, opacity=opacity, color=color)
    q = mlab.quiver3d(x, y, z, u, v, w,
                      scale_factor=0.03, opacity=opacity,
                      color=color, mode='arrow')
    q.glyph.glyph_source.glyph_source.shaft_radius = 0.02
    q.glyph.glyph_source.glyph_source.tip_length = 0.1
    q.glyph.glyph_source.glyph_source.tip_radius = 0.05

mne.viz.plot_alignment(evoked.info, bem=sphere, surfaces='inner_skull',
                       coord_frame='head', meg='helmet', show_axes=True)

# Plot the position and the orientation of the actual dipole
plot_pos_ori(actual_pos, actual_ori, color=(0., 0., 0.), opacity=0.5)
# Plot the position and the orientation of the estimated dipole
plot_pos_ori(dip.pos, dip.ori, color=(0.2, 1., 0.5))
mlab.view(70, 80, distance=0.5)_____no_output_____


# Notebook from ferngonzalezp/deep_learning_lab
Path: session1/DL_labSession1_part_1.ipynb

_____no_output_____#Introduction to Deep Learning with Python: session 1_____no_output_____Welcome to this introduction of Deep Learning with Python. This lab sessions are designed to give the student a practical introduction to the world of Deep Learning along with some theory. After the course completion you will have learned:

*   What is Deep Learning.
*   What is a Nerual Network.
*   How Nerual Networks are trained, basics of Stochastic Gradient descent.
*   Handwritten digit Classification.
*   Convolutional Neural Networks, uses in Computer Vision.
*   Detection of diseases in medical images.
*   Recurrent Neural Networks.
*   Examples in Natural Language processing.
*   How to train and build your own models.

---

_____no_output_____#Inital Motivation for Deep Learning_____no_output_____Deep Learning has gained recent Interest in the last years, and for good reason. It has allowed scientist and engineers to achieve tasks with greater degree of accuracy that couldn't be achieved before with other methods. Right now deep learning has been used extensively in the following tasks in which is the best method there is right now:

* Natural language processing (NLP): Answering questions; speech recognition; summarizing documents; classifying documents; finding names, dates, etc. in documents; searching for articles mentioning a concept
*   Computer vision:: Satellite and drone imagery interpretation (e.g., for disaster resilience); face recognition; image captioning; reading traffic signs; locating pedestrians and vehicles in autonomous vehicles
*    Medicine:: Finding anomalies in radiology images, including CT, MRI, and X-ray images; counting features in pathology slides; measuring features in ultrasounds; diagnosing diabetic retinopathy
*    Biology:: Folding proteins; classifying proteins; many genomics tasks, such as tumor-normal sequencing and classifying clinically actionable genetic mutations; cell classification; analyzing protein/protein interactions
*    Image generation:: Colorizing images; increasing image resolution; removing noise from images; converting images to art in the style of famous artists
*    Recommendation systems:: Web search; product recommendations; home page layout
*    Playing games:: Chess, Go, most Atari video games, and many real-time strategy games
*    Robotics:: Handling objects that are challenging to locate (e.g., transparent, shiny, lacking texture) or hard to pick up
*    Other applications:: Financial and logistical forecasting, text to speech, and much more...
_____no_output_____## Some cool examples_____no_output_____* [DALL-E](https://openai.com/blog/dall-e/): Image generation from text prompts.

![Dall-E](https://analyticsindiamag.com/wp-content/uploads/2021/01/DALLE_AIM.jpg)

* [AlphaFold](https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology): Solution of protein folding problem, an ongoing challenge during the past 50 years.

* [AlphaGo](https://www.alphagomovie.com/): An AI capable of beating the best Go players, there is even a movie about it._____no_output_____# What is Machine Learning?_____no_output_____We'll let machine learning respond that question. For that reason we wrote a custom prompt in this [web app](https://app.inferkit.com/demo) that said "**what is machine learning? Machine learning is**" and got this answer:

\\
`What is machine learning? Machine learning is a form of artificial intelligence where computers are given experiences and the computer "learns" from the experience and improves its own future predictions. Essentially, it lets the computer draw conclusions from the patterns it has observed in the past. This is useful for all sorts of reasons, such as automatic speech recognition (for example) or weather forecasts. It is an area of computing that is set to dominate the future and has been given a lot of attention in the tech industry.`

\\
That is a very good description. As the statement implies, machine learning is basically a set of methods to program computers in order that they can learn from examples. For that model to be able to reproduce a text as the previous ones it required lots and lots of text data and adjust itself internally based on a score of some sort, we'll later talk about that. In this picture is illustrated the building blocks of a machine learning model.

\\

Deep learning is the field inside of machine learning that uses Neural Networks as its principal building block for models. Deep learning also introduces a paradigm shift on how machine learning is done: In traditional machine learning, the input data has to be pre-processed extensively and the features to be learned by the algorithm has to be manually defined. In Deep Learning, these features or representations are learned automatically by the model, in exchange, deep learning models tendo to need more data and computational resources in most cases.

\\

_____no_output_____## Introduction to Neural Networks_____no_output_____In 1943 Warren McCulloch, a neurophysiologist, and Walter Pitts, a logician, teamed up to develop a mathematical model of an artificial neuron. In their [paper](https://link.springer.com/article/10.1007/BF02478259) "A Logical Calculus of the Ideas Immanent in Nervous Activity" they declared that:

`Because of the “all-or-none” character of nervous activity, neural events and the relations among them can be treated by means of propositional logic. It is found that the behavior of every net can be described in these terms.`

McCulloch and Pitts realized that a simplified model of a real neuron could be represented using simple addition and thresholding, as shown here:

Rosenblatt further developed the artificial neuron to give it the ability to learn. Even more importantly, he worked on building the first device that actually used these principles, the Mark I Perceptron. In "The Design of an Intelligent Automaton" Rosenblatt wrote about this work: "We are now about to witness the birth of such a machine–-a machine capable of perceiving, recognizing and identifying its surroundings without any human training or control." The perceptron was built, and was able to successfully recognize simple shapes.

An MIT professor named Marvin Minsky (who was a grade behind Rosenblatt at the same high school!), along with Seymour Papert, wrote a book called Perceptrons (MIT Press), about Rosenblatt's invention. They showed that a single layer of these devices was unable to learn some simple but critical mathematical functions (such as XOR). In the same book, they also showed that using multiple layers of the devices would allow these limitations to be addressed. Unfortunately, only the first of these insights was widely recognized. As a result, the global academic community nearly entirely gave up on neural networks for the next two decades.

\\

\\
So in theory by coupling many layers of perceptrons, one could achieve an universal function approximator, in theory. But at that time these networks were very slow due to hardware and algorithms limitations. It was not until the last decade that the use of Neural Networks has been gaining exponential interest thanks to the availability of data, hardware development and new algorithms that improved that training in parallel of these models.

_____no_output_____## The Multi-Layer Perceptron_____no_output_____So, the first and most basic neural network model that we will study is the Multi-layer perceptron architecture. And as the name implies, is just a bunch of perceptrons stacked together in many layers.

Oh and by the way, it is useful now to introduce some of the Deep learning jargon employed by the community:

*   The functional form of the model is called its architecture (but be careful—sometimes people use model as a synonym of architecture, so this can get confusing).
*    The weights are called parameters.
*   The predictions are calculated from the independent variable, which is the data not including the labels.
*   The results of the model are called predictions.
*   The measure of performance is called the loss.
*    The loss depends not only on the predictions, but also the correct labels (also known as targets or the dependent variable); e.g., "dog" or "cat."

Remember the Machine learning diagrama from before? Now it looks like this using this terminology:

 \\

_____no_output_____## How these models learn?_____no_output_____After we feed the model data and it outputs a prediction, we need to pass that value to a score we want to optimize, this score is usually called the "loss" of a model, and the training process is basically the optimization of this value.

\\

The optimization process for Neural Networks is based on [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent).

\\

_____no_output_____________________________

Now, we will see a gradient descent eaxmple using python. Using python and numpy._____no_output_____

import numpy as np

def gradient_descent(

    gradient, start, learn_rate, n_iter=50, tolerance=1e-06

):

    vector = start

    for _ in range(n_iter):

        diff = -learn_rate * gradient(vector)

        if np.all(np.abs(diff) <= tolerance):

            break

        vector += diff

    return vector_____no_output_____

*    **gradient** is the function or any Python callable object that takes a vector and returns the gradient of the function you’re trying to minimize.
*    **start** is the point where the algorithm starts its search, given as a sequence (tuple, list, NumPy array, and so on) or scalar (in the case of a one-dimensional problem).
*    **learn_rate** is the learning rate that controls the magnitude of the vector update.
*   **n_iter** is the number of iterations.

\\

We will use this algorithm to find the minimum of a simple convex function:

$y = x^2$

which we know it's gradient is:

$y' = 2x$
_____no_output_____

y_prime = lambda x: 2*x_____no_output_____

Here, we use gradient descent to find the minimum of a function. And plot the different steps it takes until it finds the minimum. Try to adjust the learning rate to see what happens._____no_output_____

%matplotlib inline
import matplotlib.pyplot as plt

#Parameters for Gradient descent
start = 10
learn_rate = 0.2
tol = 1e-6
n_steps = 50

#Plot of the function
x = np.linspace(-10,10)
y = lambda x: x**2
plt.plot(x,y(x))

#The optimization loop, we do it in such a way to obtain the result in each step.
results =[start]
for step in range(n_steps):

  min = gradient_descent(y_prime,start,learn_rate,1)
  results.append(min)
  if np.abs(start-min) < tol:
    break
  start = min

results = np.array(results)
plt.plot(results, y(results), '-*g')_____no_output_____

Try to change the function to a non convex one and it's gradient to see how it behaves!_____no_output______________________________________________
Gradient Descent is a simple and fast way to find a minimum, but when the number of parameters of a function increases and when there is non-convexity then the algorithm requires some modifications to work properly._____no_output_____## Stochastic Gradient Descent_____no_output_____Stochastic gradient descent algorithms are a modification of gradient descent. In stochastic gradient descent, you calculate the gradient using just a random small part of the observations instead of all of them. In some cases, this approach can reduce computation time.

Online stochastic gradient descent is a variant of stochastic gradient descent in which you estimate the gradient of the cost function for each observation and update the decision variables accordingly. This can help you find the global minimum, especially if the objective function is convex.

Batch stochastic gradient descent is somewhere between ordinary gradient descent and the online method. The gradients are calculated and the decision variables are updated iteratively with subsets of all observations, called minibatches. This variant is very popular for training neural networks.

You can imagine the online algorithm as a special kind of batch algorithm in which each minibatch has only one observation. Classical gradient descent is another special case in which there’s only one batch containing all observations.

From now on we will refer to the minibatch stocasthic gradient descent as stochastic gradient descent (SGD)._____no_output_____### Linear Regression with SGD_____no_output_____#### Simple Regression_____no_output_____We will start by a [simple Linear regression problem](https://en.wikipedia.org/wiki/Simple_linear_regression). In this problem we want to fit a distribution of points in 2D space with a curve. This curve can be linear, a polynomial of any order or even an exponential; the selection of curve depends on the nature of the data we have at hand, and to some extent this represents a limititation of doing regression in this fashion, we need certain prior information of the data befor applying the model, as opposed to the use of neural networks where we can learn that this distribution (but let's leave this to a further topic).

\\

First, we will do Linear regression in a classical way using the Least squares method. In linear regresion we want to the determine the slope and intercept (weight and bias) of a line equation:

\begin{equation}
  \hat{y} = b + w x
\end{equation}

\\

In the least squares method, we want to minimize the squared error between predictions and observations:

\begin{equation}
  error = \frac{1}{n} \sum^n{(y-\hat{y})^2}
\end{equation}

\\

We then can find the weight and bias just by taking the derivative of the error w.r.t. the parameters of our curve and equal to zero. We now are going to derive the formula for finding these values but first we will reformulate or curve equation in a matrix way:

\begin{equation}
  \hat{y} = \begin{bmatrix}1 & x\end{bmatrix} . \begin{bmatrix}b\\w\end{bmatrix}
\end{equation}
 \\
Now this reduces to a linear system, that can be rewritten as  $\hat{y}=X\theta$, where $\theta$ is the parameters vector of our model and X is the set of independent variables. We can now rewrite the error as:

\begin{equation}
  error = (Y-X\theta)^2\\
  error = (Y-X\theta)^T(Y-X\theta)\\
  error = Y^TY - \theta^TX^TY - Y^TX\theta + \theta^TX^TX\theta\\
  error = Y^TY - 2\theta^TX^TY + \theta^TX^TX\theta
\end{equation}

\\

This reduction is possible because $\theta^TX^TY = Y^TX\theta$, which are scalars and their transpose is the same scalar:

\begin{equation}
  (\theta^TX^TY)^T = (X^TY)^T\theta = Y^TX\theta
\end{equation}

\\

The the derivative of the error w.r.t. the parameters is:

\begin{equation}
  \partial{error}/\partial{\theta} = -2X^TY + X^TX\theta + \theta^TX^TX = -2X^TY + 2X^TX\theta
\end{equation}

\\

Equalizing the derivative to zero when then have:

\begin{equation}
  X^TY = X^TX\theta\\
  \theta = (X^TX)^{-1}X^TY
\end{equation}

\\

Now we have a general formula for fitting the parameters, for any linear function!

Now let's solve a simple problems, first we need data to fit. For this reason we will generate some points from a cuadractic function, but we will add some random noise:_____no_output_____

n_samples = 20
X = np.sort(np.random.uniform(0,10,(n_samples)))
y = lambda x: x**2 + np.random.uniform(-10,10,(len(x)))
Y = y(X)
plt.scatter(X,Y)_____no_output_____

Here we make a function to calculate the vector of parameters based on the previous formula:_____no_output_____

def lssr(Y, X):
  X = np.vstack((np.ones(np.size(X,axis=-1)),X)).transpose()
  B = np.matmul(X.transpose(),X)
  B = np.matmul(np.linalg.inv(B),X.transpose())
  return np.matmul(B,Y)_____no_output_____

Determine the parameters of our curve:_____no_output_____

Theta = lssr(Y,X)_____no_output_____

Now visualize the fitted line and the observable points._____no_output_____

plt.scatter(X,Y)
fit = Theta[0] + Theta[1]*X
plt.plot(X,fit,'r')_____no_output_____

Now we are goind to try to fit the points but with a cuadratic function: $y = b + w_1x + w_2x^2$. Use the function we created before to obtain the parameters of this curve._____no_output_____

x2 = X**2
X = np.vstack((X,x2))
Theta_cuadratic = lssr(Y,X)_____no_output_____

Now plot the results, is this model better suited to the data?_____no_output_____

plt.scatter(X[0],Y)
fit = Theta_cuadratic[0] + Theta_cuadratic[1]*X[0] + Theta_cuadratic[2]*X[1]
plt.plot(X[0],fit,'r')_____no_output_____

Now we are going to use gradient descent, to solve this same problem. First we have to create a function ta updates the coefficients based on a gradient descent step. Remember that we are trying to minimize de error function, in gradient descent we don't need an analytic formula of the gradient so we can simply take as gradient:
\begin{equation}
\partial{error}/\partial{\theta} = \frac{1}{n}\sum^n 2(\hat{Y}-Y)X
\end{equation}
\\
being $n$ the number of observations. Now we write the function:_____no_output_____

def sgd_update(X, Y, Theta, l_rate):
      X = np.vstack((np.ones(np.size(X,axis=-1)),X)).transpose()
      Yhat = np.matmul(X,Theta)
      grad_error = np.mean(np.einsum('i,ij->ij',2*(Yhat-Y),X), axis = 0)
      Theta  = Theta - l_rate * grad_error
      return Theta_____no_output_____

In this part we find the optimun using SGD:_____no_output_____

Theta_sgd = np.zeros(2)
n_epochs = 2000
lr = 0.01
for i in range(n_epochs):
  y = sgd_update(X[0],Y,Theta_sgd,lr)
  diff = np.abs(np.sum(y-Theta_sgd))
  Theta_sgd = y
  if diff < 1e-6:
    break_____no_output_____

Now we plot the results:_____no_output_____

plt.scatter(X[0],Y)
fit = Theta_sgd[0] + Theta_sgd[1]*X[0]
plt.plot(X[0],fit,'r')
error_linear = np.mean((Y - fit)**2)_____no_output_____print(error_linear)_____no_output_____

We compare the results obtained with the analytical way and using SGD:_____no_output_____

print('result with SGD:')
print(Theta_sgd)
print('result in analytical way:')
print(Theta)_____no_output_____

Now we are going to fit a line with some added gaussian noise:_____no_output_____

n_samples = 50
X = np.sort(np.random.uniform(-10,10,(n_samples)))
y = lambda x: x + np.random.normal(0,1,n_samples)
Y = y(X)
plt.scatter(X,Y)_____no_output_____coef_sgd = np.zeros(2)
n_epochs = 2000
lr = 0.01
for i in range(n_epochs):
  y = sgd_update(X,Y,coef_sgd,lr)
  diff = np.abs(np.sum(y-coef_sgd))
  coef_sgd = y
  if diff < 1e-6:
    break

plt.scatter(X,Y)
fit = coef_sgd[0] + coef_sgd[1]*X
plt.plot(X,fit,'r')
error_linear = np.mean((Y - fit)**2)_____no_output_____print(error_linear)_____no_output_____

But this time we are going to use a linear polynomial or third order and use SGD:_____no_output_____

coef_sgd = np.zeros(4)
#Based on what you have learned so far, fit the points using a third order polynomial: y = b + w1*x + w2*x**2 + w3*x**3 _____no_output_____

Now we plot the results, what happended?_____no_output_____

#plot the points and the fitted polynomial
#Also, calculate de Mean Squared error
error_cubic = _____no_output_____print(error_cubic)_____no_output_____

This phenomenon is called [overfitting](https://en.wikipedia.org/wiki/Overfitting). Here it happened because our model has more parameters than need for the data, it fitted exactly some points of the distribution but in certain areas the error may be to high, resulting in a higher MSE._____no_output_____#### Multi_Dimensional Linear Regression_____no_output_____In this part we will try to solve a [Linear regression](https://en.wikipedia.org/wiki/Linear_regression) problem using SGD programmed from scratch.
After we develop our linear regression algorithm with stochastic gradient descent, we will use it to model the wine quality dataset.

This dataset is comprised of the details of 4,898 white wines including measurements like acidity and pH. The goal is to use these objective measures to predict the wine quality on a scale between 0 and 10.

Each columm of the dataset represent the following attributes:

Attribute information:

   Input variables (based on physicochemical tests):
   1. fixed acidity
   2. volatile acidity
   3. citric acid
   4. residual sugar
   5. chlorides
   6. free sulfur dioxide
   7. total sulfur dioxide
   8. density
   9. pH
   10. sulphates
   11. alcohol
   Output variable (based on sensory data):
   12. quality (score between 0 and 10)

Click [here](https://drive.google.com/file/d/1KKvWK7pmDtJXN5zEA7w0Xje-qbP2FrJI/view?usp=sharing) to download the dataset and the [description](https://drive.google.com/file/d/1dhXvJgR5TRWCk69VTpOznrlyIyTfG3JE/view?usp=sharing).

the function we are trying to fit with this method it is:

\begin{equation}
  \hat{Y} = b_0 + W^T X = X\theta
\end{equation}

Where b_0 is a bias, W is the weight vector and X is the vector of independent variables.

\\

In this case we will use mini-batches in stochastic gradient descent for calculating the gradients and updating, the motivation for this is that by doing it this way we are dealing with matrixes of reduced size. The forumula for mini-batch of SGD changes to:

\begin{equation}
  \theta_k = \theta_{k-1} - lr * \frac{1}{b}\sum_1^b 2(\hat{Y}-Y)X \\
  b < size \ of \ the \ dataset
\end{equation}

With this two terms arise:

* Iteration: Is a SGD update step with a batch.
* Epoch: an epoch completes when we iterate throug the whole training dataset. For better performance, the batches elements are picked randomly from the dataset, this helps with convergence.
_____no_output_____

# Linear Regression With Stochastic Gradient Descent for Wine Quality
from random import seed
from random import randrange
from csv import reader
from math import sqrt
from math import ceil
from numpy import genfromtxt
from random import shuffle

# Find the min and max values for each column
def dataset_minmax(dataset):
	minmax = list()
	for i in range(len(dataset[0])):
		col_values = dataset[:,i]
		value_min = col_values.min()
		value_max = col_values.max()
		minmax.append([value_min, value_max])
	return minmax

# Rescale dataset columns to the range 0-1
def normalize_dataset(dataset, minmax):
  for i in range(len(dataset[0])):
			dataset[:,i] = (dataset[:,i] - minmax[i][0]) / (minmax[i][1] - minmax[i][0])
  return dataset

# Make a prediction with coefficients
def predict(batch, coefficients):
    yhat = np.matmul(batch[:,:-1],coefficients[1:]) + coefficients[0]
    return yhat

# Estimate linear regression coefficients using stochastic gradient descent
def sgd_update(batch, Theta, l_rate):
      X = batch[:,:-1].transpose()
      Y = batch[:,-1]
      X = np.vstack((np.ones(np.size(X,axis=-1)),X)).transpose()
      Yhat = np.matmul(X,Theta)
      grad_error = np.mean(np.einsum('i,ij->ij',2*(Yhat-Y),X), axis = 0)
      Theta  = Theta - l_rate * grad_error
      return Theta_____no_output_____#Evaluation Metric
def mse_metric(actual, predicted):
  return np.mean((actual-predicted)**2)_____no_output_____# Linear Regression on wine quality dataset
seed(1)
# load and prepare data
filename = 'winequality-white.csv'
#dataset = load_csv(filename)
dataset = genfromtxt(filename, delimiter=';')
# normalize
minmax = dataset_minmax(dataset)
dataset = normalize_dataset(dataset, minmax)
train_data = dataset[:4000]
test_data = dataset[4000:]
#Training paramateres
n_epochs = 500
batch_size = 500
l_rate = 0.1
#Initialize Coefficients with zeros
coef = np.zeros((len(dataset[0])))
#indexes of rows
idx = list(range(len(train_data[:,0])))
n_iterations = ceil(len(train_data[:,0])/batch_size)
#training loop
loss = []
for i in range(n_epochs):
  shuffle(idx)
  for j in range(n_iterations):
    if j==n_iterations-1:
      batch = train_data[idx[j*batch_size:]]
    else:
      batch = train_data[idx[j*batch_size:(j+1)*batch_size]]
    coef = sgd_update(batch, coef, l_rate)
    prediction = predict(batch, coef)
    mse = mse_metric(batch[:,-1],prediction)
    loss.append(mse)
    if j%50 == 0 :
      print("epoch: %d Iteration: %d/%d MSE: %.4f \n" % ((i+1), (j+1), n_iterations, mse))_____no_output_____plt.plot(loss)
plt.title("MSE")_____no_output_____

Now we test the coefficients with the test data we didn't used for training._____no_output_____

test_predictions = predict(test_data,coef)
print(mse_metric(test_data[:,-1],test_predictions))_____no_output_____idxs = np.random.randint(0,np.size(test_data,0),5)
print('Wine Quality:')
print(test_data[idxs,-1]*(minmax[-1][1]-minmax[-1][0]) + minmax[-1][0])
print('Predicted Wine Quality:')
print(test_predictions[idxs]*(minmax[-1][1]-minmax[-1][0]) + minmax[-1][0])_____no_output_____

We obtained a fit for the data with a Mean squared error of 0.014, try changing the batch size and learning rate to see what effect it gives to the optimization process. Maybe the advantage of SGD cannot be seen in this problem but as dimensionality increases it becomes a really useful tool for training Deep learning models._____no_output_____


# Notebook from davidthomas5412/PanglossNotebooks
Path: MassLuminosityProject/DataAndMassPrior_2017_02_08.ipynb

## Generate Mock Data_____no_output_____

from IPython.display import Image
Image(filename='pgm_mock_data.png')_____no_output_____

The distributions are

\begin{align*}
P(L^i|\alpha^*, S^*, M_h^i, z^i) &= \text{Lognormal}(\mu_L^i, S^*)\\
\mu_L^i &= \alpha_1^* + \alpha_2^* \ln(M_h^i/\alpha_3^*) + \alpha_4^*(1+z^i)\\
P(L^i_{obs}|L^i) &= \text{Lognormal}(L^i, \sigma_L)\\
\end{align*}

The mass-luminosity parameters from Reddick's thesis [1]:

\begin{align*}
\alpha_1 &= \ln L_{c0} &= 10.709^{+0.021}_{-0.023} \log L_{\odot} / h^2\\
\alpha_2 &= A_L &= 0.359\pm 0.009\\
\alpha_3 &= M_{piv} &= 2.35 \times 10^{14} M_{\odot}\\
\alpha_4 &= B_L &= 1.10 \pm 0.06\\
S &= \sigma_L &= 0.155 \pm 0.009 \log L_{\odot}/h^2\\
\end{align*}

We fix $\sigma_L \sim S / 3 \sim 0.05$. Now we draw a random assortment of $\alpha, S$ from normal distributions characterized by Reddick's numbers:_____no_output_____

from scipy.stats import norm
import numpy as np

np.random.seed(1)

alpha1 = norm(10.709, 0.022).rvs()
alpha2 = norm(0.359, 0.009).rvs()
alpha3 = 2.35e14
alpha4 = norm(1.10, 0.06).rvs()
S = norm(0.155, 0.0009).rvs()
sigma_L = 0.05

print ' alpha1 = {}\n alpha2 = {}\n alpha3 = {}\n alpha4 = {}\n S = {}\n sigma_L = {}'\
    .format(alpha1, alpha2, alpha3, alpha4, S, sigma_L) alpha1 = 10.744735598
 alpha2 = 0.353494192277
 alpha3 = 2.35e+14
 alpha4 = 1.06830969486
 S = 0.15403432824
 sigma_L = 0.05

Next we load data from the Millennium Simulation and extract a $60 \times 60 \text{ arcmin}^2$ field of view._____no_output_____

import pandas as pd
from massinference.angle import Angle

# opening, white-listing, renaming
usecols = ['GalID', 'pos_0[rad]', 'pos_1[rad]', 'M_Subhalo[M_sol/h]', 'z_spec']
guo = pd.read_table(
    '/Users/user/Code/Pangloss/data/GGL_los_8_0_0_0_0_N_4096_ang_4_Guo_galaxies_on_plane_27_to_63.images.txt',
    usecols=usecols)
guo.rename(
    columns={'GalID': 'gal_id',
             'M_Subhalo[M_sol/h]': 'mass_h',
             'z_spec': 'z'
            },
    inplace=True)
guo = guo[guo['mass_h'] > 0]

guo['mass_h'] = guo['mass_h'] * 0.73 #remove h^{-1} from mass units, use h from WMAP

# convert to arcmin
guo['ra'] = Angle.radian_to_arcmin(guo['pos_0[rad]'])
guo['dec'] = Angle.radian_to_arcmin(guo['pos_1[rad]'])

# field of view bounds
ra_i = guo['ra'].min()
dec_i = guo['dec'].min()
ra_f = ra_i + 40
dec_f = dec_i + 40
z_i = guo['z'].min()
z_f = guo['z'].max()

# clip data, fov = field of view
fov = guo[(guo['ra'] >= ra_i)
           & (guo['ra'] < ra_f)
           & (guo['dec'] >= dec_i)
           & (guo['dec'] < dec_f)].copy(deep=True)

fov.drop(['pos_0[rad]', 'pos_1[rad]'], axis=1, inplace=True)_____no_output_____fov.columns_____no_output_____

Finally we sample luminosity and observed luminosity with the hyperparameters we drew above. Then we save the dataset.  _____no_output_____

from scipy.stats import lognorm

mu_lum = np.exp(alpha1 + alpha2 * np.log(fov.mass_h / alpha3) + alpha4 * (1 + fov.z))
lum = lognorm(S, scale=mu_lum).rvs()
lum_obs = lognorm(sigma_L, loc=lum).rvs()

fov['lum'] = lum
fov['lum_obs'] = lum_obs

fov.to_csv('mock_data.csv')_____no_output_____

## Mass Prior_____no_output_____First we find the min and max halo mass over the whole healpix of the Millennium Survey [2]._____no_output_____

from math import log

# NOTE: hmf uses units of M_{\odot}/h for mass
h = 0.73

Mmin = log((guo['mass_h'] / h).min()) / log(10)
Mmax = log((guo['mass_h'] / h).max()) / log(10)

print ' Mmin = {}\n Mmax = {}'\
    .format(Mmin, Mmax) Mmin = 10.2358590918
 Mmax = 14.3277327776

Next we use the python package hmf [3] to get the mass function (Tinker 2010) for our cosmology (WMAP)._____no_output_____

%matplotlib inline

import matplotlib.pyplot as plt
from matplotlib import rc
import hmf
from scipy.stats import rv_discrete
import scipy.interpolate as interpolate

rc('text', usetex=True)

mf = hmf.MassFunction(Mmin=Mmin, Mmax=Mmax, cosmo_model=hmf.cosmo.WMAP5, hmf_model=hmf.fitting_functions.Tinker10)
pdf = mf.dndm / sum(mf.dndm)
cum_values = np.cumsum(pdf)
inv_cdf = interpolate.interp1d(cum_values, mf)

plt.figure(figsize=(8,4))

plt.subplot(311)
plt.ylabel('Density')
plt.hist(np.log(fov.mass_h) / np.log(10), alpha=0.5, normed=True, label='millennium')
plt.plot(np.log(mf.m * h) /np.log(10), 50*pdf, color='red', label='hmf')
plt.legend()

plt.subplot(312)
plt.ylabel('Density')
ms_mass = np.log(fov.mass_h) / np.log(10)
ms_mass_cut = ms_mass[ms_mass >= 11.]
plt.hist(ms_mass_cut, alpha=0.5, normed=True, label='millennium')
cut_ind = 90
m_cut = (np.log(mf.m * h) /np.log(10))[cut_ind:]
pdf_cut = (1.85e3*pdf)[cut_ind:]
plt.plot(m_cut, pdf_cut, color='red', label='hmf')
plt.legend()

plt.subplot(313)
plt.xlabel('$\log_{10}M_{\odot}$')
plt.ylabel('Density')
ms_mass = np.log(fov.mass_h) / np.log(10)
ms_mass_cut = ms_mass[ms_mass >= 12.]
plt.hist(ms_mass_cut, alpha=0.5, normed=True, label='millennium')
cut_ind = 190
m_cut = (np.log(mf.m * h) /np.log(10))[cut_ind:]
pdf_cut = (2e5*pdf)[cut_ind:]
plt.plot(m_cut, pdf_cut, color='red', label='hmf')
plt.suptitle('PDF Cut Overlays')
plt.legend()_____no_output_____

The issue here is that the hmf distribution does not match the mass distribution from the millennium simulation. This makes sense for the bottom end of our mass range where there are likely many halos on the edge that the halo finder failed to recognize. The part that I am more concerned about is that the decay as we move to high mass is MUCH greater in the halo mass function than we see from the dataset._____no_output_____

import scipy.interpolate as interpolate

class MassPrior():
    def __init__(self, mass, prob):
        self.mass = mass
        self.prob = prob
        self.min = mass.min()
        self.max = mass.max()
        # have to add 0,1 samples for interpolation bounds
        cumsum = np.append(np.append(np.array([0]), np.cumsum(prob)), np.array([1]))
        masses = np.append(np.append(np.array([self.min-1]), self.mass), np.array([self.max+1]))
        self.inv_cdf = interpolate.interp1d(cumsum, masses)

    def pdf(self, mass):
        if np.any(mass < self.min) or np.any(mass > self.max):
            raise Exception('out of range')
        right_ind = np.searchsorted(self.mass, mass)
        left_ind = right_ind - 1
        # find where we fall in interval between masses
        f = (mass - self.mass[left_ind]) / (self.mass[right_ind] - self.mass[left_ind])
        return f * self.prob[right_ind] + (1-f) * self.prob[left_ind]

    def rvs(self, *args, **kwargs):
        return self.inv_cdf(np.random.rand(size))_____no_output_____

## Bias Distribution Test_____no_output_____See [Proposal](Proposal_2017_01_10.ipynb) for more information on the biased distribution we use for importance sampling. Below we compare the numerator in our likelihood integrand, $P(L_{obs}^i| L^i)P(L^i|\alpha^*, S^*, M_h^i, z^i)P(M_h^i)$ when sampled from $P(M_h)$ and when sampled from $Q(M_h,L)$._____no_output_____

from scipy.stats import lognorm

mp = MassPrior(mf.m*h, mf.dndm / sum(mf.dndm))

def log_P_m(mass_h):
    return np.log(mp.pdf(mass_h))

def log_P_l_given_m_z(mass_h, z, lum):
    mu_l = alpha1 + alpha2 * np.log(mass_h / alpha3) + alpha4 * (1+z)
    return lognorm(S, loc=mu_l).logpdf(lum)

def log_P_lobs_given_l(lum, lum_obs):
    return lognorm(sigma_L, loc=lum).logpdf(lum_obs)

def log_weight(mass_h, lum, lum_obs, z):
    print log_P_m(mass_h)
    print log_P_l_given_m_z(mass_h, z, lum)
    print log_P_lobs_given_l(lum, lum_obs)
    return log_P_m(mass_h) + log_P_l_given_m_z(mass_h, z, lum) + log_P_lobs_given_l(lum, lum_obs)

def draw_m(size):
    return mp.rvs(size=size)

def draw_lobs_given_l(lum):
    return lognorm(sigma_L, loc=lum).rvs()

def draw_l_given_m_z(m, z):
    mu_l = np.exp(alpha1 + alpha2 * np.log(m / alpha3) + alpha4 * (1+z))
    return lognorm(S, loc=mu_l).rvs()    _____no_output_____

To test our biased importance sampling distribution we randomly draw new hyperparameters._____no_output_____

np.random.seed(2)

alpha1b = norm(10.709, 0.022).rvs()
alpha2b = norm(0.359, 0.009).rvs()
alpha3b = 2.35e14
alpha4b = norm(1.10, 0.06).rvs()
Sb = norm(0.155, 0.0009).rvs()
sigma_Lb = 0.05

print ' alpha1b = {}\n alpha2b = {}\n alpha3b = {}\n alpha4b = {}\n Sb = {}\n sigma_Lb = {}'\
    .format(alpha1b, alpha2b, alpha3b, alpha4b, Sb, sigma_Lb) alpha1b = 10.6998313274
 alpha2b = 0.358493598555
 alpha3b = 2.35e+14
 alpha4b = 0.97182823426
 Sb = 0.156476243728
 sigma_Lb = 0.05
data = pd.read_csv('mock_data.csv')

def draw_biased_m_l(lobs, z):
    l = lognorm(sigma_Lb, loc=lobs).rvs()
    mu_m = alpha3b * (l / (np.exp(alpha1b) * (1 + z) ** alpha4b)) ** (1 / alpha2b)
    m = lognorm(Sb, loc=mu_m).rvs()
    return (m,l)_____no_output_____nsamples = 100

# data
z_data = data['z'].as_matrix()[0:nsamples]
lobs_data = data['lum_obs'].as_matrix()[0:nsamples]

# biased samples
m_biased, l_biased = draw_biased_m_l(lobs_data, z_data)

# unbiased samples
m = draw_m(nsamples)
l = draw_l_given_m_z(m, z_data)

log_weight(m[0], l_biased[0], lobs_data[0], z_data[0])-3.72509689076
-2696.39035571
-inf

## Questions
- Reddick's function fits the luminosity of centrals. Should we be worried about using it in our case for subhalos?
- What should we do about mass function vs millennium mass distribution?_____no_output_____## References_____no_output_____[1] Rachel Marie Reddick. ReddiCosmology and galaxy formation using the galaxy-halo connection to probe cosmology. https://searchworks.stanford.edu/view/10531737. 2014.

[2] V. Springel, S. D. M. White, A. Jenkins, C. S. Frenk, N. Yoshida, L. Gao, J. Navarro, R. Thacker, D. Croton, J. Helly, J. A. Peacock, S. Cole, P. Thomas, H. Couchman, A. Evrard, J. Colberg, and F. Pearce. Simulations of the formation, evolution and clustering of galaxies and quasars. http://www.nature.com/nature/journal/v435/n7042/full/nature03597.html. 2005.

[3] Steven Murray, Chris Power, Aaron Robotham . HMFcalc: An Online Tool for Calculating Dark Matter Halo Mass Functions. https://arxiv.org/abs/1306.6721. 2013._____no_output_____


# Notebook from prisae/aoc20
Path: 2021/Day-20.ipynb

# Day 20_____no_output_____

import numpy as np
from aocd.models import Puzzle_____no_output_____

## Data_____no_output_____

puzzle = Puzzle(year=2021, day=20)
data = puzzle.input_data.replace('.', '0').replace('#', '1').split('\n\n')_____no_output_____alg, im = data
alg = np.array([a for a in alg], dtype=int)
im = np.array([[i for i in di] for di in im.split('\n')], dtype=int)_____no_output_____

## Part One_____no_output_____

def apply(img, N):

    dat = np.zeros((img.shape[0]+2*N+4, img.shape[1]+2*N+4), dtype=int)
    dat[N+2:-N-2, N+2:-N-2] = im

    for i in range(N):
        ndat = dat.copy()
        for x in range(1, dat.shape[0]-1):
            for y in range(1, dat.shape[1]-1):

                bn = (str(dat[x-1, y-1]) + str(dat[x-1, y]) + str(dat[x-1, y+1]) +
                      str(dat[x  , y-1]) + str(dat[x  , y]) + str(dat[x  , y+1]) +
                      str(dat[x+1, y-1]) + str(dat[x+1, y]) + str(dat[x+1, y+1]))

                ndat[x, y] = alg[int(bn, base=2)]

        ndat[0, :] = ndat[1, :]
        ndat[-1, :] = ndat[-2, :]
        ndat[:, 0] = ndat[:, 1]
        ndat[:, -1] = ndat[:, -2]

        dat = ndat

    return dat_____no_output_____answer_a = apply(im, 2).sum()
answer_a_____no_output_____puzzle.answer_a = answer_aThat's the right answer!  You are one gold star closer to finding the sleigh keys. [Continue to Part Two]

## Part Two_____no_output_____

answer_b = apply(im, 50).sum()
answer_b_____no_output_____puzzle.answer_b = answer_bThat's the right answer!  You are one gold star closer to finding the sleigh keys.You have completed Day 20! You can [Shareon
  Twitter
Mastodon] this victory or [Return to Your Advent Calendar].
import scooby
scooby.Report('aocd')_____no_output_____