# Notebook from gilberto-BE/aifi-pytorch
Path: notebooks/.ipynb_checkpoints/6. Ensemble Methods-checkpoint.ipynb

# Ensemble Methods
The goal of **ensemble methods** is to combine the predictions of several base estimators built with a given learning algorithm in order to improve generalizability / robustness over a single estimator.

Two families of ensemble methods are usually distinguished:

* In **averaging methods**, the driving principle is to build several estimators independently and then to average their predictions. On average, the combined estimator is usually better than any of the single base estimator because its variance is reduced.

**Examples**: Bagging methods, Forests of randomized trees, …

* By contrast, in **boosting methods**, base estimators are built sequentially and one tries to reduce the bias of the combined estimator. The motivation is to combine several weak models to produce a powerful ensemble.

**Examples**: AdaBoost, Gradient Tree Boosting, …_____no_output_____# Bagging Meta-Estimator

In ensemble algorithms, bagging methods form a class of algorithms which build several instances of a black-box estimator on random subsets of the original training set and then aggregate their individual predictions to form a final prediction. These methods are used as a way to reduce the variance of a base estimator (e.g., a decision tree), by introducing randomization into its construction procedure and then making an ensemble out of it. In many cases, bagging methods constitute a very simple way to improve with respect to a single model, without making it necessary to adapt the underlying base algorithm. As they provide a way to reduce overfitting, bagging methods work best with strong and complex models (e.g., fully developed decision trees), in contrast with boosting methods which usually work best with weak models (e.g., shallow decision trees).

Bagging methods come in many flavours but mostly differ from each other by the way they draw random subsets of the training set:

* When random subsets of the dataset are drawn as random subsets of the samples, then this algorithm is known as Pasting.
* When samples are drawn with replacement, then the method is known as Bagging.
* When random subsets of the dataset are drawn as random subsets of the features, then the method is known as Random Subspaces.
* Finally, when base estimators are built on subsets of both samples and features, then the method is known as Random Patches.

In scikit-learn, bagging methods are offered as a unified BaggingClassifier meta-estimator (resp. BaggingRegressor), taking as input a user-specified base estimator along with parameters specifying the strategy to draw random subsets. In particular, max_samples and max_features control the size of the subsets (in terms of samples and features), while bootstrap and bootstrap_features control whether samples and features are drawn with or without replacement. When using a subset of the available samples the generalization accuracy can be estimated with the out-of-bag samples by setting oob_score=True. As an example, the snippet below illustrates how to instantiate a bagging ensemble of KNeighborsClassifier base estimators, each built on random subsets of 50% of the samples and 50% of the features._____no_output_____

from sklearn.ensemble import GradientBoostingClassifier_____no_output_____

# Forest of Randomized Trees
The sklearn.ensemble module includes two averaging algorithms based on randomized decision trees: the RandomForest algorithm and the Extra-Trees method. Both algorithms are perturb-and-combine techniques [B1998] specifically designed for trees. This means a diverse set of classifiers is created by introducing randomness in the classifier construction. The prediction of the ensemble is given as the averaged prediction of the individual classifiers.

As other classifiers, forest classifiers have to be fitted with two arrays: a sparse or dense array X of size [n_samples, n_features] holding the training samples, and an array Y of size [n_samples] holding the target values (class labels) for the training samples:_____no_output_____# Random Forests
In random forests (see RandomForestClassifier and RandomForestRegressor classes), each tree in the ensemble is built from a sample drawn with replacement (i.e., a bootstrap sample) from the training set. In addition, when splitting a node during the construction of the tree, the split that is chosen is no longer the best split among all features. Instead, the split that is picked is the best split among a random subset of the features. As a result of this randomness, the bias of the forest usually slightly increases (with respect to the bias of a single non-random tree) but, due to averaging, its variance also decreases, usually more than compensating for the increase in bias, hence yielding an overall better model.

In contrast to the original publication [B2001], the scikit-learn implementation combines classifiers by averaging their probabilistic prediction, instead of letting each classifier vote for a single class.
# Extremly Randomized Trees
In extremely randomized trees (see ExtraTreesClassifier and ExtraTreesRegressor classes), randomness goes one step further in the way splits are computed. As in random forests, a random subset of candidate features is used, but instead of looking for the most discriminative thresholds, thresholds are drawn at random for each candidate feature and the best of these randomly-generated thresholds is picked as the splitting rule. This usually allows to reduce the variance of the model a bit more, at the expense of a slightly greater increase in bias:_____no_output_____

from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_blobs
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.tree import DecisionTreeClassifier
import numpy as np

X, y = make_blobs(n_samples=10000, n_features=10, centers=100,
    random_state=0)

print(X.shape)
print(y.shape)
print('np.unique()')
print(np.unique(y))
print('len of unique y')
print(len(np.unique(y)))

# Homework use the classification report for each of the classifiers.
# what happens if you change from cross_val_score(...) and use instead clf.fit(x_train, y_train)
# clf.predict(x_test)?
# What happens when you try both cross_val_scor() and clf.fit() on the breast cancer data set?

clf = DecisionTreeClassifier(max_depth=None, min_samples_split=2,
    random_state=0)
scores = cross_val_score(clf, X, y, cv=5)
print('Scores DecisionTreeClassifier:', scores.mean())

clf = RandomForestClassifier(n_estimators=10, max_depth=None,
    min_samples_split=2, random_state=0)
scores = cross_val_score(clf, X, y, cv=5)
print('Score RandomForestClassifier:', scores.mean())

clf = ExtraTreesClassifier(n_estimators=10, max_depth=None,
    min_samples_split=2, random_state=0)
scores = cross_val_score(clf, X, y, cv=5)

print('Scores for ExtraTreeClassifier:', scores.mean())
(10000, 10)
(10000,)
np.unique()
[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
 96 97 98 99]
len of unique y
100
Scores DecisionTreeClassifier: 0.9823000000000001
Score RandomForestClassifier: 0.9997
Scores for ExtraTreeClassifier: 1.0

# AdaBoost
The module sklearn.ensemble includes the popular boosting algorithm AdaBoost, introduced in 1995 by Freund and Schapire [FS1995].

The core principle of AdaBoost is to fit a sequence of weak learners (i.e., models that are only slightly better than random guessing, such as small decision trees) on repeatedly modified versions of the data. The predictions from all of them are then combined through a weighted majority vote (or sum) to produce the final prediction. The data modifications at each so-called boosting iteration consist of applying weights , , …,  to each of the training samples. Initially, those weights are all set to , so that the first step simply trains a weak learner on the original data. For each successive iteration, the sample weights are individually modified and the learning algorithm is reapplied to the reweighted data. At a given step, those training examples that were incorrectly predicted by the boosted model induced at the previous step have their weights increased, whereas the weights are decreased for those that were predicted correctly. As iterations proceed, examples that are difficult to predict receive ever-increasing influence. Each subsequent weak learner is thereby forced to concentrate on the examples that are missed by the previous ones in the sequence [HTF].

_____no_output_____

from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris
from sklearn.ensemble import AdaBoostClassifier

iris = load_iris()
print(iris.data.shape)
print(iris.target.shape)
clf = AdaBoostClassifier(n_estimators=100)
scores = cross_val_score(clf, iris.data, iris.target, cv=5)
scores.mean()
(150, 4)
(150,)

# Gradient Tree Boosting

Gradient Tree Boosting or Gradient Boosted Regression Trees (GBRT) is a generalization of boosting to arbitrary differentiable loss functions. GBRT is an accurate and effective off-the-shelf procedure that can be used for both regression and classification problems. Gradient Tree Boosting models are used in a variety of areas including Web search ranking and ecology.

The advantages of GBRT are:

* Natural handling of data of mixed type (= heterogeneous features)
* Predictive power
* Robustness to outliers in output space (via robust loss functions)

The disadvantages of GBRT are:

* Scalability, due to the sequential nature of boosting it can hardly be parallelized.

The module sklearn.ensemble provides methods for both classification and regression via gradient boosted regression trees._____no_output_____## Classification

GradientBoostingClassifier supports both binary and multi-class classification. The following example shows how to fit a gradient boosting classifier with 100 decision stumps as weak learners:_____no_output_____$\text{precision} = \frac{tp}{tp + fp},$

$\text{recall} = \frac{tp}{tp + fn},$

$F_\beta = (1 + \beta^2) \frac{\text{precision} \times \text{recall}}{\beta^2 \text{precision} + \text{recall}}.$_____no_output_____

from sklearn.datasets import make_hastie_10_2
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import classification_report, confusion_matrix

X, y = make_hastie_10_2(random_state=0)
X_train, X_test = X[:2000], X[2000:]
y_train, y_test = y[:2000], y[2000:]

print(X_train.shape)
print(y_train.shape)

clf = GradientBoostingClassifier(n_estimators=500, learning_rate=1.0,
    max_depth=1, random_state=0).fit(X_train, y_train)
clf.score(X_test, y_test)

y_pred = clf.predict(X_test)
y_pred_train = clf.predict(X_train)
print('Classification report for test data set:')
print(classification_report(y_test, y_pred))
print()
print('Classification report for train data set:')
print(classification_report(y_train, y_pred_train))

clf = GradientBoostingClassifier(n_estimators=50, learning_rate=1.0,
                                 max_depth=1, random_state=0)

scores = cross_val_score(clf, X, y, cv=5)
print(round(scores.mean(), 2))
print(scores)(2000, 10)
(2000,)
Classification report for test data set:
              precision    recall  f1-score   support

        -1.0       0.94      0.96      0.95      5049
         1.0       0.95      0.93      0.94      4951

    accuracy                           0.95     10000
   macro avg       0.95      0.95      0.95     10000
weighted avg       0.95      0.95      0.95     10000

Classification report for train data set:
              precision    recall  f1-score   support

        -1.0       1.00      1.00      1.00      1019
         1.0       1.00      1.00      1.00       981

    accuracy                           1.00      2000
   macro avg       1.00      1.00      1.00      2000
weighted avg       1.00      1.00      1.00      2000

0.87
[0.86838817 0.88004998 0.8675     0.88870363 0.86702793]

The number of weak learners (i.e. regression trees) is controlled by the parameter n_estimators; The size of each tree can be controlled either by setting the tree depth via max_depth or by setting the number of leaf nodes via max_leaf_nodes. The learning_rate is a hyper-parameter in the range (0.0, 1.0] that controls overfitting via shrinkage ._____no_output_____## Regression

GradientBoostingRegressor supports a number of different loss functions for regression which can be specified via the argument loss; the default loss function for regression is least squares ('ls').
_____no_output_____

import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn.datasets import make_friedman1
from sklearn.ensemble import GradientBoostingRegressor

X, y = make_friedman1(n_samples=1200, random_state=0, noise=1.0)
X_train, X_test = X[:200], X[200:]
y_train, y_test = y[:200], y[200:]
est = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1,
    max_depth=1, random_state=0, loss='ls').fit(X_train, y_train)
mean_squared_error(y_test, est.predict(X_test))
_____no_output_____

The figure below shows the results of applying GradientBoostingRegressor with least squares loss and 500 base learners to the Boston house price dataset (sklearn.datasets.load_boston). The plot on the left shows the train and test error at each iteration. The train error at each iteration is stored in the train_score_ attribute of the gradient boosting model. The test error at each iterations can be obtained via the staged_predict method which returns a generator that yields the predictions at each stage. Plots like these can be used to determine the optimal number of trees (i.e. n_estimators) by early stopping. The plot on the right shows the feature importances which can be obtained via the feature_importances_ property.

Both **GradientBoostingRegressor** and **GradientBoostingClassifier** support warm_start=True which allows you to add more estimators to an already fitted model.
_____no_output_____GBRT

$F(x) = \sum_{m=1}^{M} \gamma_m h_m(x)$

Similar to other boosting algorithms, GBRT builds the additive model in a greedy fashion:

$F_m(x) = F_{m-1}(x) + \gamma_m h_m(x)$,

where the newly added tree $h_m$ tries to minimize the loss $L$, given the previous ensemble $F_{m-1}$:

$h_m =  \arg\min_{h} \sum_{i=1}^{n} L(y_i,
F_{m-1}(x_i) + h(x_i)).$_____no_output_____Gradient Boosting attempts to solve this minimization problem numerically via steepest descent: The steepest descent direction is the negative gradient of the loss function evaluated at the current model  which can be calculated for any differentiable loss function:

$F_m(x) = F_{m-1}(x) - \gamma_m \sum_{i=1}^{n} \nabla_F L(y_i,
F_{m-1}(x_i))$

Where the step length $\gamma_m$ is chosen using line search:

$\gamma_m = \arg\min_{\gamma} \sum_{i=1}^{n} L(y_i, F_{m-1}(x_i)
- \gamma \frac{\partial L(y_i, F_{m-1}(x_i))}{\partial F_{m-1}(x_i)})$

The algorithms for regression and classification only differ in the concrete loss function used._____no_output_____## Regularization
$F_m(x) = F_{m-1}(x) + \nu \gamma_m h_m(x)$

The parameter $\nu$ is also called the learning rate because it scales the step length the gradient descent procedure; it can be set via the learning_rate parameter._____no_output_____## Voting Classifier_____no_output_____

from sklearn import datasets
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from itertools import product
from sklearn.ensemble import VotingClassifier

# Loading some example data
iris = datasets.load_iris()
X = iris.data[:, [0, 2]]
y = iris.target

# Training classifiers
clf1 = DecisionTreeClassifier(max_depth=4)
clf2 = KNeighborsClassifier(n_neighbors=7)
clf3 = SVC(gamma='scale', kernel='rbf', probability=True)
eclf = VotingClassifier(estimators=[('dt', clf1), ('knn', clf2), ('svc', clf3)],
                        voting='soft', weights=[2, 1, 2])

clf1 = clf1.fit(X, y)
clf2 = clf2.fit(X, y)
clf3 = clf3.fit(X, y)
eclf = eclf.fit(X, y)_____no_output_____

# Check feature selection
https://scikit-learn.org/stable/modules/feature_selection.html_____no_output_____

from sklearn.svm import LinearSVC
from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectFromModel
iris = load_iris()
X, y = iris.data, iris.target
print(X.shape)

lsvc = LinearSVC(C=0.01, penalty="l1", dual=False).fit(X, y)
model = SelectFromModel(lsvc, prefit=True)
X_new = model.transform(X)
print(X_new.shape)(150, 4)
(150, 3)
from sklearn import pipeline

clf = pipeline.Pipeline([
  ('feature_selection', SelectFromModel(LinearSVC(penalty="l1"))),
  ('classification', RandomForestClassifier())])
_____no_output_____# Grid search
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB

clf1 = LogisticRegression(solver='lbfgs', multi_class='multinomial',
                          random_state=1)
clf2 = RandomForestClassifier(random_state=1)
clf3 = GaussianNB()
eclf = VotingClassifier(estimators=[('lr', clf1), ('rf', clf2), ('gnb', clf3)], voting='soft')

params = {'lr__C': [1.0, 100.0], 'rf__n_estimators': [20, 200]}

grid = GridSearchCV(estimator=eclf, param_grid=params, cv=5)
grid = grid.fit(iris.data, iris.target)D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
D:\anaconda\lib\site-packages\sklearn\linear_model\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)

## Voting Regressor_____no_output_____

from sklearn import datasets
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import VotingRegressor
import matplotlib.pyplot as plt

import matplotlib.pyplot as plt

from sklearn import datasets
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import VotingRegressor

# Loading some example data
X, y = datasets.load_boston(return_X_y=True)

# Training classifiers
reg1 = GradientBoostingRegressor(random_state=1, n_estimators=10)
reg2 = RandomForestRegressor(random_state=1, n_estimators=10)
reg3 = LinearRegression()
ereg = VotingRegressor([('gb', reg1), ('rf', reg2), ('lr', reg3)])
reg1.fit(X, y)
reg2.fit(X, y)
reg3.fit(X, y)
ereg.fit(X, y)

xt = X[:20]

plt.figure(figsize=(15, 8))
plt.plot(y[:20], 'b', label='y_true')
plt.plot(reg1.predict(xt), 'gd', label='GradientBoostingRegressor')
plt.plot(reg2.predict(xt), 'b^', label='RandomForestRegressor')
plt.plot(reg3.predict(xt), 'ys', label='LinearRegression')
plt.plot(ereg.predict(xt), 'r*', label='VotingRegressor')
plt.tick_params(axis='x', which='both', bottom=False, top=False,
                labelbottom=False)
plt.ylabel('predicted')
plt.xlabel('training samples')
plt.legend(loc="best")
plt.title('Comparison of individual predictions with averaged')
plt.show()_____no_output_____


# Notebook from ajlee21/generic-expression-patterns
Path: tests/2_identify_generic_genes_pathways.ipynb

# Test: Identify generic human genes on test set

This notebook performs the following steps to identify generic genes
1. Simulates N gene expression experiments using [ponyo](https://github.com/ajlee21/ponyo)
2. Perform DE analysis to get association statistics for each gene

In this case the DE analysis is based on the experimental design of the template experiment, described in the previous [notebook](1_process_recount2_data.ipynb). The template experiment is [SRP012656](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE37764), which contains primary non-small cell lung adenocarcinoma tumors and adjacent normal tissues of 6 never-smoker Korean female patients. So the DE analysis is comparing tumor vs normal in this case.

3. For each gene, aggregate statsitics across all simulated experiments
4. Rank genes based on this aggregated statistic

**Evaluation:**
We want to compare our ranking using ponyo, compared to the ranking found from Crow et. al._____no_output_____

%load_ext autoreload
%load_ext rpy2.ipython
%autoreload 2
%matplotlib inline

import os
import pandas as pd
import numpy as np
import pickle
from rpy2.robjects import pandas2ri
from ponyo import utils, simulate_expression_data
from generic_expression_patterns_modules import process, stats, ranking

pandas2ri.activate()
np.random.seed(123)Using TensorFlow backend.
/home/alexandra/anaconda3/envs/generic_expression_new/lib/python3.7/site-packages/matplotlib/__init__.py:886: MatplotlibDeprecationWarning:
examples.directory is deprecated; in the future, examples will be found relative to the 'datapath' directory.
  "found relative to the 'datapath' directory.".format(key))
# Read in config variables
base_dir = os.path.abspath(os.path.join(os.getcwd(), "../"))

config_filename = os.path.abspath(os.path.join(base_dir, "configs", "config_test.tsv"))

params = utils.read_config(config_filename)_____no_output_____# Load params
local_dir = params["local_dir"]
dataset_name = params["dataset_name"]
NN_architecture = params["NN_architecture"]
latent_dim = params["latent_dim"]
num_runs = params["num_simulated"]
project_id = params["project_id"]
metadata_col_id = params["metadata_colname"]
mapped_template_filename = params["mapped_template_filename"]
processed_template_filename = params["processed_template_filename"]
normalized_compendium_filename = params["normalized_compendium_filename"]
scaler_filename = params["scaler_filename"]
col_to_rank_genes = params["rank_genes_by"]
col_to_rank_pathways = params["rank_pathways_by"]
statistic = params["gsea_statistic"]
count_threshold = params["count_threshold"]
logFC_name = params["DE_logFC_name"]
pvalue_name = params["DE_pvalue_name"]

# Load metadata file with grouping assignments for samples
sample_id_metadata_filename = os.path.join(
    base_dir, dataset_name, "data", "metadata", f"{project_id}_process_samples.tsv"
)

# Load metadata file with grouping assignments for samples
metadata_filename = os.path.join(
    base_dir, dataset_name, "data", "metadata", f"{project_id}_groups.tsv"
)

# Load metadata file with mapping between experiments and associated samples
metadata_simulate_filename = os.path.join(
    base_dir, dataset_name, "data", "metadata", "recount2_metadata.tsv"
)
metadata_delimiter = "\t"
experiment_id_colname = "project"

# Load pickled file
with open(scaler_filename, "rb") as scaler_fh:
    scaler = pickle.load(scaler_fh)_____no_output_____

## Test: Simulation_____no_output_____

# Simulate multiple experiments
# This step creates the following files in "<local_dir>/pseudo_experiment/" directory:
#   - selected_simulated_data_SRP012656_<n>.txt
#   - selected_simulated_encoded_data_SRP012656_<n>.txt
#   - template_normalized_data_SRP012656_test.txt
# in which "<n>" is an integer in the range of [0, num_runs-1]
os.makedirs(os.path.join(local_dir, "pseudo_experiment"), exist_ok=True)
simulate_expression_data.shift_template_experiment(
    normalized_compendium_filename,
    NN_architecture,
    latent_dim,
    dataset_name,
    scaler,
    metadata_simulate_filename,
    metadata_delimiter,
    experiment_id_colname,
    metadata_col_id,
    project_id,
    local_dir,
    base_dir,
    num_runs,
)WARNING:tensorflow:From /home/alexandra/anaconda3/envs/generic_expression_new/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /home/alexandra/anaconda3/envs/generic_expression_new/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

## Test: Processing template experiment

`processed_template_filename`: template data with some sample rows dropped_____no_output_____

if not os.path.exists(sample_id_metadata_filename):
    sample_id_metadata_filename = None

stats.process_samples_for_DESeq(
    mapped_template_filename,
    metadata_filename,
    processed_template_filename,
    count_threshold,
    sample_id_metadata_filename,
)sample ids are ordered correctly
# Read data
template_data = pd.read_csv(
    processed_template_filename, header=0, sep="\t", index_col=0
)

# Check samples dropped
print(template_data.shape)
assert template_data.shape[0] == 24
template_data.head()(24, 17788)

## Test: Processing simulation experiments_____no_output_____

# This step modifies the following files:
# "<local_dir>/pseudo_experiments/selected_simulated_data_SRP012656_<n>.txt"
for i in range(num_runs):
    simulated_filename = os.path.join(
        local_dir, "pseudo_experiment", f"selected_simulated_data_{project_id}_{i}.txt"
    )
    stats.process_samples_for_DESeq(
        simulated_filename,
        metadata_filename,
        None,
        count_threshold,
        sample_id_metadata_filename,
    )sample ids are ordered correctly
sample ids are ordered correctly
# Check simulated files were created
sim_output1 = os.path.join(
    local_dir, "pseudo_experiment", "selected_simulated_data_SRP012656_0.txt"
)
sim_output2 = os.path.join(
    local_dir, "pseudo_experiment", "selected_simulated_data_SRP012656_1.txt"
)
assert os.path.exists(sim_output1) and os.path.exists(sim_output2)_____no_output_____# Check that simulated files are non-empty
assert os.path.getsize(sim_output1) > 0 and os.path.getsize(sim_output2) > 0_____no_output_____# Check that simulated experiments are different
sim1 = pd.read_csv(sim_output1, sep="\t", index_col=0, header=0)
sim2 = pd.read_csv(sim_output2, sep="\t", index_col=0, header=0)

assert sim1.equals(sim2) == False_____no_output_____

## Test: Differential expression analysis_____no_output_____

# Create subdirectory: "<local_dir>/DE_stats/"
os.makedirs(os.path.join(local_dir, "DE_stats"), exist_ok=True)_____no_output_____%%R -i metadata_filename -i project_id -i processed_template_filename -i local_dir -i base_dir

source(paste0(base_dir, '/generic_expression_patterns_modules/DE_analysis.R'))

# File created: "<local_dir>/DE_stats/DE_stats_template_data_SRP012656_real.txt"
get_DE_stats_DESeq(metadata_filename,
                   project_id,
                   processed_template_filename,
                   "template",
                   local_dir,
                   "real")R[write to console]: Loading required package: S4Vectors

R[write to console]: Loading required package: stats4

R[write to console]: Loading required package: BiocGenerics

R[write to console]: Loading required package: parallel

R[write to console]:
Attaching package: ‘BiocGenerics’

R[write to console]: The following objects are masked from ‘package:parallel’:

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

R[write to console]: The following object is masked from ‘package:limma’:

    plotMA

R[write to console]: The following objects are masked from ‘package:stats’:

    IQR, mad, sd, var, xtabs

R[write to console]: The following objects are masked from ‘package:base’:

    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
    union, unique, unsplit, which, which.max, which.min

R[write to console]:
Attaching package: ‘S4Vectors’

R[write to console]: The following object is masked from ‘package:base’:

    expand.grid

R[write to console]: Loading required package: IRanges

R[write to console]: Loading required package: GenomicRanges

R[write to console]: Loading required package: GenomeInfoDb

R[write to console]: Loading required package: SummarizedExperiment

R[write to console]: Loading required package: Biobase

R[write to console]: Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

R[write to console]: Loading required package: DelayedArray

R[write to console]: Loading required package: matrixStats

R[write to console]:
Attaching package: ‘matrixStats’

R[write to console]: The following objects are masked from ‘package:Biobase’:

    anyMissing, rowMedians

R[write to console]: Loading required package: BiocParallel

R[write to console]:
Attaching package: ‘DelayedArray’

R[write to console]: The following objects are masked from ‘package:matrixStats’:

    colMaxs, colMins, colRanges, rowMaxs, rowMins, rowRanges

R[write to console]: The following objects are masked from ‘package:base’:

    aperm, apply, rowsum

%%R -i metadata_filename -i project_id -i base_dir -i local_dir -i num_runs -i base_dir

source(paste0(base_dir, '/generic_expression_patterns_modules/DE_analysis.R'))

# Files created: "<local_dir>/DE_stats/DE_stats_simulated_data_SRP012656_<n>.txt"
for (i in 0:(num_runs-1)){
    simulated_data_filename <- paste(local_dir,
                                     "pseudo_experiment/selected_simulated_data_",
                                     project_id,
                                     "_",
                                     i,
                                     ".txt",
                                     sep = "")

    get_DE_stats_DESeq(metadata_filename,
                       project_id,
                       simulated_data_filename,
                       "simulated",
                       local_dir,
                       i)
}[1] "Checking sample ordering..."
[1] TRUE
# Check DE stats files were created
DE_output1 = os.path.join(
    local_dir, "DE_stats", "DE_stats_simulated_data_SRP012656_0.txt"
)
DE_output2 = os.path.join(
    local_dir, "DE_stats", "DE_stats_simulated_data_SRP012656_1.txt"
)
assert os.path.exists(DE_output1) and os.path.exists(DE_output2)_____no_output_____# Check that DE stats files are non-empty
assert os.path.getsize(DE_output1) > 0 and os.path.getsize(DE_output2) > 0_____no_output_____

### Rank genes_____no_output_____

analysis_type = "DE"
template_DE_stats_filename = os.path.join(
    local_dir, "DE_stats", f"DE_stats_template_data_{project_id}_real.txt"
)

template_DE_stats, simulated_DE_summary_stats = ranking.process_and_rank_genes_pathways(
    template_DE_stats_filename,
    local_dir,
    num_runs,
    project_id,
    analysis_type,
    col_to_rank_genes,
    logFC_name,
    pvalue_name,
)/home/alexandra/anaconda3/envs/generic_expression_new/lib/python3.7/site-packages/pandas/core/reshape/merge.py:544: UserWarning: merging between different levels can give an unintended result (2 levels on the left, 1 on the right)
  warnings.warn(msg, UserWarning)

### Gene summary table_____no_output_____

summary_gene_ranks = ranking.generate_summary_table(
    template_DE_stats_filename,
    template_DE_stats,
    simulated_DE_summary_stats,
    col_to_rank_genes,
    local_dir,
    "gene",
    params,
)_____no_output_____summary_gene_ranks.head()_____no_output_____# Some genes will have NaN's in the simulated statistics columns. These are genes that were filtered
# due to low expression and therefore the corresponding Z-score for this gene is NaN
summary_gene_ranks.isna().any()_____no_output_____summary_gene_ranks[summary_gene_ranks.isna().any(axis=1)]_____no_output_____# Create `gene_summary_filename`
gene_summary_filename = os.path.join(local_dir, "gene_summary_table.tsv")
summary_gene_ranks.to_csv(gene_summary_filename, sep="\t")_____no_output_____

## Test: Compare gene ranking
Studies have found that there are some genes that are more likely to be differentially expressed even across a wide range of experimental designs. These *generic genes* are not necessarily specific to the biological process being studied but instead represents a more systematic change.

We want to compare the ability to detect these generic genes using our method vs those found by [Crow et. al. publication](https://www.pnas.org/content/pnas/116/13/6491.full.pdf). Their genes are ranked 0 = not commonly DE; 1 = commonly DE. Genes by the number differentially expressed gene sets they appear in and then ranking genes by this score._____no_output_____

# Get generic genes identified by Crow et. al.
DE_prior_file = params["reference_gene_filename"]
ref_gene_col = params["reference_gene_name_col"]
ref_rank_col = params["reference_rank_col"]

figure_filename = f"gene_ranking_{col_to_rank_genes}.svg"

corr_stats, shared_ranking = ranking.compare_gene_ranking(
    summary_gene_ranks, DE_prior_file, ref_gene_col, ref_rank_col, figure_filename
)r = 0.22078516078709132
p = 1.3138111306669223e-170
ci_low = 0.20724082206538047
ci_high = 0.23641242185968384

## Test: GSEA_____no_output_____

# Create "<local_dir>/GSEA_stats/" subdirectory
os.makedirs(os.path.join(local_dir, "GSA_stats"), exist_ok=True)_____no_output_____# Load pathway data
hallmark_DB_filename = os.path.join(
    base_dir, dataset_name, "data", "metadata", "hallmark_DB.gmt"
)_____no_output_____%%R -i base_dir -i template_DE_stats_filename -i hallmark_DB_filename -i statistic -o template_enriched_pathways
source(paste0(base_dir, '/generic_expression_patterns_modules/GSEA_analysis.R'))

out_file <- paste(local_dir,
                  "GSA_stats/GSEA_stats_template_data_",
                  project_id,
                  "_test.txt",
                  sep = "")

template_enriched_pathways <- find_enriched_pathways(template_DE_stats_filename, hallmark_DB_filename, statistic)

template_enriched_pathways <- as.data.frame(template_enriched_pathways[1:7])

write.table(template_enriched_pathways, file = out_file, row.names = F, sep = "\t")R[write to console]: Loading required package: Rcpp

%%R -i project_id -i local_dir -i hallmark_DB_filename -i num_runs -i statistic -i base_dir

source(paste0(base_dir,'/generic_expression_patterns_modules/GSEA_analysis.R'))

# New files created: "<local_dir>/GSEA_stats/GSEA_stats_simulated_data_<project_id>_<n>.txt"
for (i in 0:(num_runs-1)) {
    simulated_DE_stats_file <- paste(local_dir,
                                     "DE_stats/DE_stats_simulated_data_",
                                     project_id,
                                     "_",
                                     i,
                                     ".txt",
                                     sep = "")

    out_file <- paste(local_dir,
                     "GSA_stats/GSEA_stats_simulated_data_",
                     project_id,
                     "_",
                     i,
                     ".txt",
                     sep = "")

    enriched_pathways <- find_enriched_pathways(simulated_DE_stats_file, hallmark_DB_filename, statistic)

    # Remove column with leading edge since its causing parsing issues
    write.table(as.data.frame(enriched_pathways[1:7]), file = out_file, row.names = F, sep = "\t")
}_____no_output_____# Check GSEA stats files were created
GSEA_output1 = os.path.join(
    local_dir, "GSA_stats", "GSEA_stats_simulated_data_SRP012656_0.txt"
)
GSEA_output2 = os.path.join(
    local_dir, "GSA_stats", "GSEA_stats_simulated_data_SRP012656_1.txt"
)
assert os.path.exists(DE_output1) and os.path.exists(DE_output2)_____no_output_____# Check that GSEA stats files are non-empty
assert os.path.getsize(GSEA_output1) > 0 and os.path.getsize(GSEA_output2) > 0_____no_output_____

### Rank pathways_____no_output_____

analysis_type = "GSA"
template_GSEA_stats_filename = os.path.join(
    local_dir, "GSA_stats", f"GSEA_stats_template_data_{project_id}_test.txt"
)

(
    template_GSEA_stats,
    simulated_GSEA_summary_stats,
) = ranking.process_and_rank_genes_pathways(
    template_GSEA_stats_filename,
    local_dir,
    num_runs,
    project_id,
    analysis_type,
    col_to_rank_pathways,
    logFC_name,
    pvalue_name,
    "GSEA",
)_____no_output_____

### Pathway summary table_____no_output_____

# Create intermediate file: "<local_dir>/gene_summary_table_<col_to_rank_pathways>.tsv"
summary_pathway_ranks = ranking.generate_summary_table(
    template_GSEA_stats_filename,
    template_GSEA_stats,
    simulated_GSEA_summary_stats,
    col_to_rank_pathways,
    local_dir,
    "pathway",
    params,
)/home/alexandra/anaconda3/envs/generic_expression_new/lib/python3.7/site-packages/pandas/core/reshape/merge.py:544: UserWarning: merging between different levels can give an unintended result (1 levels on the left, 2 on the right)
  warnings.warn(msg, UserWarning)

## Test: Compare pathway ranking_____no_output_____

# Load Powers et. al. results file
powers_rank_filename = os.path.join(
    base_dir, dataset_name, "data", "metadata", "Hallmarks_qvalues_GSEAPreranked.csv"
)_____no_output_____# Read Powers et. al. data
# This file contains qvalue results for hallmark pathways across ~400 experiments
powers_rank_df = pd.read_csv(powers_rank_filename, header=0, index_col=0)
powers_rank_df.drop(["Category"], axis=1, inplace=True)_____no_output_____# Count the number of experiments where a given pathway was found to be enriched (qvalue < 0.05)
total_num_experiments = powers_rank_df.shape[1]
frac_enriched_pathways = (powers_rank_df < 0.05).sum(axis=1) / total_num_experiments

# Rank pathways from 0-50, 50 indicating that the pathways was frequently enriched
pathway_ranks = frac_enriched_pathways.rank()

powers_rank_stats_df = pd.DataFrame(
    data={
        "Fraction enriched": frac_enriched_pathways.values,
        "Powers Rank": pathway_ranks.values,
    },
    index=powers_rank_df.index,
)_____no_output_____# Save reference file for input into comparison
powers_rank_processed_filename = os.path.join(
    base_dir,
    dataset_name,
    "data",
    "metadata",
    "Hallmarks_qvalues_GSEAPreranked_processed.tsv",
)

powers_rank_stats_df.to_csv(
    powers_rank_processed_filename,
    sep="\t",
)_____no_output_____figure_filename = f"pathway_ranking_{col_to_rank_pathways}.svg"

corr_stats = ranking.compare_pathway_ranking(
    summary_pathway_ranks, powers_rank_processed_filename, figure_filename
)r = 0.15704404920248013
p = 0.276084382283898
ci_low = -0.16380348646531837
ci_high = 0.4134132111726299
{'r': 0.15704404920248013, 'p': 0.276084382283898, 'ci_low': -0.16380348646531837, 'ci_high': 0.4134132111726299}


# Notebook from FRBs/PreFRBLE
Path: notebooks/model/Rodrigues18.ipynb

import matplotlib
%matplotlib inline
import numpy as np, matplotlib.pyplot as plt, h5py as h5
from matplotlib.cm import rainbow
from PreFRBLE.convenience import *
from PreFRBLE.likelihood import *
from PreFRBLE.physics import *
from PreFRBLE.plot import *
_____no_output_____

# Rodrigues model for host and intervening galaxies (Rodrigues et al. 2018)

Rodrigues et al. provide a set of axisymmetric galaxy ensembles, obtained via GALFORM parameters derived by Lacey et al. 2016, which represent the population of galaxies in the Universe according to galaxy formation theory.
From these, we obtain the expected contribution to DM, RM and SM of FRBs via a Monte-Carlo simulation, probing LoS with random geometry through random galaxy models.
Here we translate the output of the Monte-Carlo simulation performed with magnetizer code by Luiz F. S. Rodrigues to format used in PrEFRBLE.
These results are given for different redshift bins than used in PrEFRBLE.
We obtain the likelihood functions used in PrEFRBLE by redshifting the average of nearest neighbours (comoving), transformed to proper values at the given redshift._____no_output_____

## model nmnemonic
model = 'Rodrigues18'

## file containing raw data
Rodrigues_datafile = '../../likelihood/Rodrigues_galaxy_raw.h5'

## file to contain average r_gal and n_gal
Rodrigues_file_rgal = '../../likelihood/Rodrigues_galaxy_radius.dat'

## number of redshift bins and bins in likelihood functions
N_z, N_bins = 60, 100

## redshift bins in PreFRBLE
zs0 = redshift_bins
h = 0.703
_____no_output_____def NearestNeighbor( x=1., xs=np.array([0.,1.,2.]) ):
    """ find nearst neighbor of x in xs """
    #print( np.diff( xs, axis=1, prepend=x) )
    dx = (xs - x)**2
    i_NN = np.where( dx == dx.min() )[0][0]
    return i_NN

def NearestNeighbors( x=1., xs=np.array([0.,1.,2.]) ):
    """ find nearst neighbors of x in xs """
    try:
        i_NN = np.where( xs >= x )[0][0]
    except:
        return np.array([-1])
    return np.array([-1,0]) + i_NN

def  LikelihoodStandardize( P=[], x=[], redshift=0.0, measure='DM' ):
    """ transform raw data format to standard format in PrEFRBLE """

    ## translate logarithmic bin central points to bin ranges
    dx = np.mean(np.diff(x))
    x = 10.**np.append( x[0]-dx/2, x+dx/2 )

    ## raw data is normalized to 1 = sum( P * diff( log10(DM) ) ), renormalize to 1 = sum( P * diff(DM) )
    P *= np.diff( np.log10(x) ) / np.diff( x )
    P /= np.sum( P*np.diff(x)) ## make it exact

    ## transform to proper values (data is written in residual frame)
    P, x = LikelihoodShift( x=x, P=P, shift=(1+redshift)**-scale_factor_exponent[measure] )

    ## use correct number of bins
    P, x = LikelihoodShrink( P=P, x=x, bins=N_bins )

    return P, x_____no_output_____

### Standardize Likelihood
Here we read the raw output of Monte-Carlo simulation performed by Luiz F. S. Rodrigues (*Rodrigues_datafile*) and transform to the standards used in PrEFRBLE (*likelihood_file_galaxy*), i. e. number of bins in P, renormalization and redshift bins. Also saves average half-mass radius and number density of galaxies as function of redshift (*Rodrigues_file_rgal*)._____no_output_____

n_gals, r_gals = [], []

with h5.File(Rodrigues_datafile,'r') as f:

    ## redshift bins of raw results
    zs = list(f.keys())
    zs_float = np.array(zs).astype('float')
    colors = list(Rainbow(zs0))
    for z0, c in zip(zs0, colors):

        ## find nearst neighbour redshift
        iz = NearestNeighbor( x=z0, xs=zs_float )

        izs = NearestNeighbors( x=z0, xs=zs_float )
        ## linear approximation
        weights = 1 - np.abs( zs_float[izs] - z0 )/np.diff(zs_float[izs])
        if izs[0] == [-1]:
            weights = [1]

        ## first, prepare correct redshift binning of r_gal and n_gal
        ## use approximation between nearest neighbours
        r_gal = np.sum( [ w*f['/'.join([zs[iz],"avg_radius"])][()] for iz, w in zip(izs, weights)] )  ## kpc
        n_gal = np.sum( [ w*f['/'.join([zs[iz],"number_density"])][()] for iz, w in zip(izs, weights)] )  ## Mpc^-3 comoving
        #'''
        n_gal *= (1+z0)**3 ## Mpc^-3

        n_gals.append(n_gal)
        r_gals.append(r_gal)

#        """
        for measure in ['DM','RM','SM'][1:]:
            for host in [True,False]:
                ## read raw data of nearest neighbours
                Ps, xs = [], []
                for iz in izs:
                    x = f['/'.join([zs[iz],"FRB_{}".format(measure) if host else measure])]['bins'][()]
                    P = f['/'.join([zs[iz],"FRB_{}".format(measure) if host else measure])]['data'][()]

                    ## bring to standard form used in PrEFRBLE
                    P, x = LikelihoodStandardize( P=P, x=x, redshift=z0, measure=measure)
                    Ps.append(P)
                    xs.append(x)

                ## find the correspoding mean
                P, x = LikelihoodsAdd( Ps=Ps, xs=xs, log=True, weights=weights, renormalize=1)

                ## write to file
                if host:
                    keys = [ KeyHost( model=model, redshift=z0, measure=measure, axis=axis ) for axis in ['P','x'] ]
                else:
                    keys = [ Keyinter( model=model, redshift=z0, measure=measure, axis=axis ) for axis in ['P','x'] ]
                Write2h5( likelihood_file_galaxy, datas=[P,x], keys=keys )

with open( Rodrigues_file_rgal, 'w' ) as f:
    f.write( 'redshift\tr_gal\tn_gal\n' )
    for z, r, n in zip( zs0, r_gals, n_gals):
        f.write( "{0}\t{1}\t{2}\n".format(z,r, n) )_____no_output_____## this function delivers the expected number of galaxies along each LoS as function of source redshift
NInter(1.5)_____no_output_____

### plot redshift evolution of $n_{\rm gal}$ and $r_{\rm gal}$_____no_output_____

files = ['../../../PreFRBLE_2nd/data/galaxy_data_centrals.hdf5','../../../PreFRBLE_2nd/data/galaxy_data.hdf5']
h = 0.703

files= [Rodrigues_datafile]

fig, axs = plt.subplots( 2, 1, figsize=(6,5), sharex='all' )

for fn in files[:]:
    zs = []
    ns= []
    rs = []
    r_devs = []
    with h5.File(fn) as f:
#with h5.File('../../../PreFRBLE_2nd/data/galaxy_data.hdf5') as f:
        for k in f.keys():
            n=f[k+'/number_density'][()]
            r=f[k+'/avg_radius'][()]
            r_dev = f[k+'/std_radius'][()]
#            r=np.sqrt(f[k+'/avg_radius_squared'][()])
#            r_dev = f[k+'/std_radius_squared'][()]
            z = float(k)
            zs.append(z)
            ns.append(n) #*(1+z)**3)
#            ns.append(n*h**3)
            rs.append(r)
            r_devs.append(r_dev)
    axs[0].plot(zs,ns, lw=3)
    axs[1].errorbar(zs,rs, yerr=r_dev, lw=3)
axs[0].set_yscale('log')
#plt.plot(z_gal, n_gal)
axs[0].set_ylabel(r"$n_{\rm gal}$ / Mpc$^{-3}$", fontsize=16)
axs[1].set_ylabel(r"$\langle r_{1/2}\rangle$ / kpc", fontsize=16)
axs[1].set_xlabel('redshift', fontsize=16)
axs[0].set_yscale('log')
for ax in axs:
    ax.tick_params(axis='both', which='major', labelsize=14)
plt.subplots_adjust(hspace=0.0)/data/PreFRBLE/venv_PreFRBLE/lib/python3.4/site-packages/ipykernel_launcher.py:13: H5pyDeprecationWarning: The default file mode will change to 'r' (read-only) in h5py 3.0. To suppress this warning, pass the mode you need to h5py.File(), or set the global default h5.get_config().default_file_mode, or set the environment variable H5PY_DEFAULT_READONLY=1. Available modes are: 'r', 'r+', 'w', 'w-'/'x', 'a'. See the docs for details.
  del sys.path[0]

### Intervening Galaxies_____no_output_____We combine the likelihood functions of intervening galaxies $P_{\rm inter}$ at redshift $z$ to likelihood function for contribution of intervening galaxies $P_{\rm Inter}$ of unknown redshift to LoS of source at redshift $z_s$. This is done by adding probabilities of different bins, weighted by the probability to intersect galaxy in that bin, $n_{\rm inter}(z)$. Here we compute that probability as well as the average number of intervening galaxies as function of source redshift
$$
N_{\rm Inter}(z_s) = \int\limits_0^{z_s} \pi r_{\rm gal}^2 n_{\rm gal} \frac{d_H(z)}{1+z} \text{d}z = \int \pi(z) \text{d}z = \sum n_{\rm inter}(z)
$$
which is used to renormalize $N_{\rm Inter}(z) = \int P_{\rm Inter}(\text{DM}|z) \text{d}\text{DM}$.
$d_H$ is Hubble length._____no_output_____

### plot likelihood for galaxies at redshift to intersect LoS, n_inter, which is used to weigh the redshift bins
### as well as expectet number of galaxies in LoS to source, N_inter

fig, ax = plt.subplots( figsize=(5,3) )

model = 'Rodrigues18'

#pi_inter = PriorInter( redshift_bins[-1], model=model )
n_inter = nInter( redshift_bins[-1], model=model )
N_inter = np.cumsum( n_inter)

plt.plot( redshift_bins, N_inter, label=r"$N_{Inter}(z_{FRB})$", lw=3 )
plt.plot( redshift_bins, n_inter, linestyle=':', label=r"$\pi_{Inter}(z)$d$z$", color='green', lw=3 )

model = 'Macquart13'
n_inter = nInter( redshift_bins[-1], model=model )
N_inter = np.cumsum( n_inter)
plt.plot( redshift_bins, N_inter, label=r"$N_{Inter}(z_{FRB})$ MK13", alpha=0.5, color='orange' )

plt.xlabel('redshift', fontsize=16)
plt.ylabel(r"$P$", fontsize=16)
plt.yscale( 'log')
plt.legend( fontsize=16)
AllSidesTicks(ax)
_____no_output_____

Finally, compute the full likelihood $P_{\rm Inter}$ for contribution of interverning galaxies of unknown type at unknown redshift to measures along LoS to source at redshift $z_s$ by summing the contribution of individual redshifts $P_{\rm inter}$ weighed by the intersection probability $\pi_{\rm inter}$.
$$
P_{\rm Inter} (DM|z_s) = \int P_{\rm inter}(DM|z) \pi_{\rm inter}(z) \text{d}z = \sum P_{\rm inter}( DM|z) n_{\rm inter}(z)
$$_____no_output_____

measures = [ 'DM', 'RM', 'SM' ]  ## scattering time needs special treatment in own notebook

fig, axs1 = plt.subplots( 1, 2, figsize=(8,3))
Colorbar(redshift_bins)
fig.tight_layout()
fig, axs2 = plt.subplots( 1, 2, figsize=(8,3))
Colorbar(redshift_bins)
fig.tight_layout()
fig, axs3 = plt.subplots( 1, 2, figsize=(8,3))
Colorbar(redshift_bins)
fig.tight_layout()
axs = [ axs1, axs2, axs3 ]

model='Rodrigues18'

weights = nInter( redshift_bins[-1], model=model )
## redshifts of intervening galaxy chosen in center of bin
redshift_L = redshift_bins - np.diff(redshift_range)/2

for measure, axs in zip( measures, axs ):
    for i_s, (z_s, color) in enumerate( zip( redshift_bins, rainbow( np.linspace(0,1,len(redshift_bins)) ) ) ):
        ## for source redshift
        Ps, xs = [], []
        for i_L , (z_L, color_L) in enumerate( zip( redshift_L[:i_s+1], rainbow( np.linspace(0,1,len(redshift_bins)) ) ) ):
            ### read and collect P_inter(DM|z_L) for all z_L <= z_s
            P, x = GetLikelihood_inter( redshift=redshift_bins[i_L], model=model, measure=measure)
            if i_s == len(redshift_bins) - 1:
                PlotLikelihood( x=x, P=P, measure=measure, color=color_L, ax=axs[0] )
            Ps.append(P)
            xs.append(x)
        ### integrate results for all intervening redshifts, weighing with differential intersection probability
        ### result is renormalized to 1, can be read out renormalized to N_inter with GetLikelihood_Inter( ..., N_inter=True )
        P, x = LikelihoodsAdd( Ps, xs, weights=weights[:i_s+1], renormalize=1 )
        PlotLikelihood( x=x, P=P, measure=measure, color=color, ax=axs[1] )
        Write2h5( filename=likelihood_file_galaxy, datas=[P,x],
                 keys=[ KeyInter( redshift=z_s, model=model, measure=measure, axis=axis) for axis in ['P','x'] ]
                )

plt.show()
_____no_output_____

### compare with Kierdorf et al.
depreceated for now, as waiting for data of prediction from single galaxy model with comparable global parameters in _Rodrigues_ sample_____no_output_____

RM_M51 = np.genfromtxt('../../downloads/M51_RM_Sband.txt')
P, x = Histogram( np.abs(RM_M51)[RM_M51 != 0.0], log=True, bins=20, density=True, range=[1,200] )
PlotLikelihood( x, P, measure='RM' )
plt.title( 'M51 observed S-band')_____no_output_____

### Compute galaxy number density from galaxy mass function
This is not needed, since all the info is already encoded in results of Rodrigues et al._____no_output_____

### GSMF from other works

def Schechter( M, M_star=1, phi=1, alpha=1):
    """ Schechter function, Masses in log10(M_sun), phi in Mpc^-3 h^3 (determines unit of result) """
    M_ = 10.**M / 10.**M_star
    return phi * ( M_ )**alpha * np.exp( -M_ ) / M_star

def SchechterDouble( M, M_star=1, phi1=1, alpha1=1, phi2=1, alpha2=1):
    """ double Schechter function, Masses in log10(M_sun), phi in Mpc^-3 h^3 (determines unit of result) """
    M_ = 10.**M / 10.**M_star
    return ( phi1 * ( M_ )**alpha1 + phi2 * ( M_ )**alpha2 ) * np.exp( -M_ ) / M_star  ## M_ is ratio of M_sun, while M_star is log10(M_sun)

## Henriques et al 2015

def PlotGSMFHenriques( ax=None, **kwargs_plot ):
    """ Plot GSMF according to Henriques et al 2015 """
    if ax is None:
        fig, ax = plt.subplots()

    d = np.genfromtxt( '../GSMF_Henriques2015.txt ', names=True, skip_header=1 )
    M = 10.**np.mean([d['M_lo'],d["M_hi"]], axis=0) ## M_sun h^-2
    M /= h*h ## M_sun
    phi = d['Phi'] ## h^3 Mpc^-3 dex^-1
    phi_err = d['Phi_err'] ## h^3 Mpc^-3 dex^-1
    ax.errorbar( M, phi, phi_err, label='Henriques15', **kwargs_plot)

## COSMOS2015 and Davidzon et al. 2017, Tab. 1

## Best fit parameters from Davidzon et al. 2017, Tab. 1, redshifts are extended to z=0 and z=6
parameters = np.genfromtxt( '../COSMOS2015_Schechter_fit.txt', names=True,  delimiter='\t')
#parameters['M_star'] = 10.**parameters['M_star']

def PlotGSMFCOSMOS2015( ax=None, Ms = np.linspace(9,12,10), redshift_evolution=False, **kwargs_plot ):
    if ax is None:
        fig, ax = plt.subplots()
    colors = rainbow( np.linspace( 0, 1, len(parameters) ) )

    for p, c in zip( parameters, colors ):
        norm_phi = 1e-3/h**3  ## phi are written in 10^-3 Mpc^-3, this tranforms to Mpc^-3 h^3
#        print(p)
        phi = SchechterDouble( Ms, M_star=p['M_star'], phi1=p['phi1'], alpha1=p['alpha1'], phi2=p['phi2'], alpha2=p['alpha2'])
        phi *= norm_phi ## Mpc^-3 h^3

        if redshift_evolution:
            ax.plot( 10.**Ms, phi, label='%.1f < z < %.1f' % ( p['z0'], p['z1'] ), color=c, **kwargs_plot )
        else:
            ## consider deviation according to Tab. 1 in Davidzon et al. 2017
            phi_hi = norm_phi*SchechterDouble( Ms, M_star=p['M_star']+0.13, phi1=p['phi1']+0.633, alpha1=p['alpha1']+0.08, phi2=p['phi2']+0.73, alpha2=p['alpha2']+0.62)
            phi_lo = norm_phi*SchechterDouble( Ms, M_star=p['M_star']-0.14, phi1=p['phi1']-0.969, alpha1=p['alpha1']-0.25, phi2=p['phi2']-0.78, alpha2=p['alpha2']-0.60)
            ax.errorbar( 10.**Ms, phi, [phi-phi_lo, phi_hi-phi], label='Davidzon17', **kwargs_plot )

            break

### parameters for Doble Schechter functions
### sorted as read y function: log10(M_star), phi1, alpha1, phi2, alpha2
### M_star in M_sun, phi in Mpc^-3 h^-3
schechter_double_parameters = {  ## as listed in Wright et al. 2017 Tab 2
    'Wright17' : {
        'mean' : np.array([ 10.78, 2.93e-3, -0.62, 0.63e-3, -1.50 ]),
        'err_lo'  : np.array([ 0.21, 0.4e-3, 0.18, 0.1e-3, 0.16 ]),
        'err_hi'  : np.array([ 0.21, 0.4e-3, 0.18, 0.1e-3, 0.16 ])
    },
    'Eckert16' : {
        'mean' : np.array([ 10.87, 9.0e-3, -0.52, 3.25e-3, -1.38 ]),
        'err_lo' : np.array([ 0.47, 8.47e-3, 0.64, 2.81e-3, 0.5]),
        'err_hi' : np.array([ 0.53, 6.36e-3, 1.02, 3.e-3, 0.28 ])
    },
    'Weigel16' : {
        'mean' : np.array([ 10.79, 3.35e-3, -0.79, 0.17e-3, -1.69 ]),
        'err_lo' : np.array([ 0.21, 2.31e-3, 0.19, 0.01e-3, 0.2  ]),
        'err_hi' : np.array([ 0.21, 2.31e-3, 0.19, 0.01e-3, 0.2  ]),
    },
    'Baldry12' : {
        'mean' : np.array([ 10.66, 3.96e-3, -0.35, 0.79e-3, -1.47 ]),
        'err_lo' : np.array([ 0.25, 0.34e-3, 0.33, 0.23e-3, 0.2 ]),
        'err_hi' : np.array([ 0.25, 0.34e-3, 0.33, 0.23e-3, 0.2 ]),
    },
    'Peng10' : {
        'mean' : np.array([ 10.67, 4.03e-3, -0.52, 0.66e-3, -1.56 ]),
        'err_lo' : np.array([ 0.21, 0.12e-3, 0.19, 0.09e-3, 0.27 ]),
        'err_hi' : np.array([ 0.21, 0.12e-3, 0.19, 0.09e-3, 0.27 ])
    }

}

##  Wrigt et al. 2017
def PlotGSMFDoubleSchecter( ax=None, model='Wright17', **kwargs_plot ):
    if ax is None:
        fig, ax = plt.subplots()

    Ms = np.linspace(7.5,12,10)
    p = schechter_double_parameters[model]
    phi = SchechterDouble( Ms, *p['mean'] )
    phi_lo = SchechterDouble( Ms, *(p['mean']-p['err_lo']) )
    phi_hi = SchechterDouble( Ms, *(p['mean']+p['err_hi']) )
    ax.errorbar( 10.**Ms, phi, [phi-phi_lo, phi_hi-phi], label=model, **kwargs_plot )

PlotGSMF_model = {
    'Henriques15' : PlotGSMFHenriques,
    'Davidzon17' : PlotGSMFCOSMOS2015,

}

def PlotGSMF( model='Davidzon17', **kwargs_GSMF):
    try:
        PlotGSMFDoubleSchecter( model=model, **kwargs_GSMF )
        return;
    except:
        pass
    PlotGSMF_model[model]( **kwargs_GSMF )

GSMF_models = np.append( list(PlotGSMF_model.keys()), list(schechter_double_parameters.keys()) )

            _____no_output_____fig, ax = plt.subplots()
#for model in ['Henriques15','Davidzon17' ]:
for model in GSMF_models[:]:
    print(model)
    PlotGSMF( model, ax=ax )
plt.legend()
plt.loglog()
#plt.ylim(1e-4,1)
_____no_output_____source = '../../../PreFRBLE_2nd/data/galaxy_data.hdf5'
fig, axs = plt.subplots(2,1, figsize=(6,9), sharex='all')

## parameters used by Rodrigues18
bin_width = 0.1
subvol = 500**3/512 ## Mpc^3 h^-3
N_subvol = 41
volume = subvol*N_subvol ## Mpc^3 h^-3

ns = []
with h5.File(source, 'r') as f:
    ## only for z=0
    zs = list(f.keys())
    zs_float = np.array(zs).astype('f')
    colors = list(Rainbow(zs_float))[::-1]
#    colors = ['blue']
    for z in zs[:1]:

        GSMF = f[z]['GSMF']['data'][()]  ## log10( h^3 Mpc^-3 dex^1 comoving )
        M = f[z]['GSMF']['bins'][()]
        bin_width = M[1]-M[0] ## log10( M_sun )
        GSMF = 10.**GSMF  ## h^3 Mpc^-3 dex^1 comoving
        M = 10.**M ## M_sun

        color=colors.pop()
        axs[0].loglog(M, GSMF, color=color)

        GDMF = f[z]['cumGSMF']['data'][()] ## N_galaxies
        Mmin = f[z]['cumGSMF']['bins'][()] ## log10( M_sun )

        GDMF /= volume ## ## h^3 Mpc^-3 comoving
        GDMF *= h**3 ## Mpc^-3 comoving
        Mmin = 10.** Mmin ## M_sun

        axs[1].loglog( Mmin, GDMF, color=color, ls='-')

        n = GDMF[0] ## Mpc^-3 comoving
        ns.append(n)

        ## write correct n to Rodrigues_datafile
#        with h5.File(Rodrigues_datafile,'a') as ff:
#            ff[z]['number_density'][()] = n

    axs[0].set_ylabel(r"$\Phi /  ( h^3 {\rm Mpc}^{-3} {\rm dex}^{-1})$", fontsize=16 )
    axs[1].set_ylabel(r"$n_{\rm gal} (\geq M)/  {\rm Mpc}^{-3}$", fontsize=16 )
    axs[1].set_xlabel(r"$M_\star/ M_\odot$", fontsize=16 )
#    axs[1].set_yscale('linear')
    for ax in axs:
        ax.tick_params(axis='both', which='major', labelsize=14)

    ### plot GSMF constraints
    for model in GSMF_models[1:2]:
        PlotGSMF( model, ax=axs[0], ls=':' )

    plt.subplots_adjust(hspace=0.0)
axs[0].legend( fontsize=14)
axs[0].set_ylim(1e-6,5e-1)
plt.show()

plt.plot(zs_float, ns)
#plt.plot(zs_float, ns*(1+zs_float)**-3)
plt.yscale('log')

## here is some mistake in unit conversion and computation of GSMF constraints. correct shape of GSMF is confirmed in Lacey et al. 2016_____no_output_____


# Notebook from MusicalInformatics/miws2021env
Path: key_meter_estimation/hidden_markov_models.ipynb

# Hidden Markov Models

## Quick Recap of Probability

* Probability: measure of the **likelihood** of an event

* $0\leq p(x) \leq 1$,
    * $p(x) = 0$ indicates that the event is very unlikely to occur
    * $p(x) = 1$ indicates that the event will most likely occur

## Quick Recap: Graph Theory

* Graphs: A tuple $\mathcal{G}=(\mathbf{X}, \mathbf{E})$ consisting in a set of nodes $\mathbf{X} = \{X_1,\dots,X_N\}$ and a set of edges $\mathbf{E}$ connecting the nodes.

* Edges can be *directed* $(X_i \rightarrow X_j)$ or *undirected* $(X_i - X_j)$. If the nodes of a graph are directed, we call it a *directed Graph*, otherwise is an *undirected grap*
* Parent and Children nodes: In directed graphs, if $(X_i \rightarrow X_j)\in \mathbf{E}$ then $X_i$ is a parent of $X_j$ and $X_j$ is a child of $X_i$.
    * $\mathbf{Pa}(X_i)$ is the set of parents of $X_i$
    * $\mathbf{Ch}(X_i)$ is the set of children of $X_i$

Let's consider this graph

* the parents of $X_3$ are $\mathbf{Pa}(X_3) = \{X_1, X_2\}$.
* Which are the parents of $X_4$?
* Which are the parents of $X_1$?

## Quick Recap: Probabilistic Graphical Models

* Probabilistic graphical models (PGMs) provide a way to visualize the structure of a probabilisitc model

* Easy and elegant way to represent conditional independence properties

* **Bayesian Networks**: Nodes in a graph represent *random variable* and edges specify conditional independence properties:

$$p(X_1, \dots, X_N) = \prod_{i=1}^{N} p(X_i \mid \mathbf{Pa}(X_i))$$

For the example above

$$p(X_1, X_2, X_3, X_4, X_5, X_6) = p(X_1) p(X_2) p(X_3\mid X_1, X_2) p(X_4 \mid X_3) p(X_5 \mid X_3) p(X_6 \mid X_3)$$_____no_output_____## Markov Models

* The simplest way of modeling a sequence of observations is to treat them as independent_____no_output_____

_____no_output_____$$p(\mathbf{x}_1\dots, \mathbf{x}_N) = \prod_{n=1}^{N} p(\mathbf{x}_i)$$

but this is a poor assumption for inherently sequential data (like music!)_____no_output_____* Easy way to model sequential data: the conditional distribution of each variable is independent of all previous observations except for the most recent: *first-order Markov chain*_____no_output_____

_____no_output_____$$p(\mathbf{x}_1, \dots, \mathbf{x}_N) = p(\mathbf{x}_1)\prod_{n=2}^{N}p(\mathbf{x}_n \mid \mathbf{x}_{n-1})$$_____no_output_____## Hidden Markov Models

### Model Definition

A hidden Markov Model (HMM) is a *state space model* that is not limited by the Markov assumption to any order. To do this, for each observation $\mathbf{x}_n$, we have a corresponding *hidden* (latent) variable $\mathbf{z}_n$ that satisfies the conditional independence property that $\mathbf{z}_{n-1}$ and $\mathbf{z}_{n+1}$ are independent give $\mathbf{z}_n$._____no_output_____

_____no_output_____* Since an HMM is a Bayesian network, its joint probability distribution is given by
$$
p(\mathbf{x}_1, \dots, \mathbf{x}_{N-1}, \mathbf{z}_{1},\dots \mathbf{z}_{N-1})=p(\mathbf{z}_1) \left[ \prod_{n=2}^N p(\mathbf{z}_n \mid \mathbf{z}_{n-1})\right]\prod_{n=1}^N p(\mathbf{x}_n \mid \mathbf{z}_n)
$$

* In HMMs, the hidden variables $\mathbf{z}_n$ are discrete and are typically represented by multinomial variables. Therefore, it is convenient to use a 1-of-$K$ coding scheme.

* The conditional distribution $p(\mathbf{z}_n \mid \mathbf{z}_{n -1})$ can be represented by a matrix $\mathbf{A}$, commonly referred to as *transition probabilities*.

$$A_{ij} = p(z_{ni} = 1 \mid  z_{n-1,j} = 1)$$

where $0\leq A_ij \leq 1$ and $\sum_j A_{ij} = 1$.

$$p(\mathbf{z}_n \mid \mathbf{z}_{n-1}) = \prod_{i=1}^K \prod_{j=1}^K A_{ij}^{z_{n-1,j} \cdot z_{n,i}}$$

* Since $\mathbf{z}_1$ has no parents, the marginal distribution $p(\mathbf{z}_1)$ is represented by a vector of probabilities $\mathbf{\pi}$ with elements $\pi_k = p(z_{1k} = 1)$

$$p(\mathbf{z}_1) = \prod_{k=1}^K \pi_k^{z_{1k}}$$

* The distribution of the observed variables (*emission probabilities*) is modeled by $p(\mathbf{x}_n \mid \mathbf{z}_n, \mathbf{\phi})$, where $\mathbf{\phi}$ is the set of parameters of this distribution.

$$ p(\mathbf{x}_n \mid \mathbf{z}_n) = \prod_{k=1}^K p(\mathbf{x}_n \mid  \mathbf{\phi}_k)^{z_{nk}} $$

* The set of parameters of the HMM is then $\mathbf{\theta} = \{\mathbf{\pi}, \mathbf{A}, \mathbf{\phi}\}$_____no_output_____### Three Inference Problems with HMMs

1. Given $\mathbf{\theta}$ and a sequence of observations, find the most likely sequence of hidden varibles
    * Viterbi Algorithm

2. Given $\mathbf{\theta}$ and a sequence of observations, find the probability of the observed sequence
    * Forward algorithm (not covered here)

3. Given sequences of observations, learn the model parameters $\mathbf{\theta}$
    * Maximum likelihood Using Expectation-Maximization (also not covered here!)

For a more detailed (and formal) description, see the tutorial by [Pernkopf et al., 2013](https://www2.spsc.tugraz.at/www-archive/downloads/PGM.pdf)._____no_output_____### Viterbi Algorithm

* In many applications, the hidden variables have some meaningful interpretation:
    * In speech recognition: find the most probable sequence of phonemes
    * In bioinformatics: Aligning DNA/RNA sequences
    * In MIR: Music alignment, chord recognition, key identification...

* Formally, we would like to find

$$\hat{\mathbf{Z}} = \arg \max_\mathbf{Z} p(\mathbf{X}, \mathbf{Z})$$

where $\mathbf{X} = \{\mathbf{x}_1, \dots, \mathbf{x}_N\}$ and $\mathbf{Z} = \{\mathbf{z}_1,\dots, \mathbf{z}_N \}$

* Direct optimization of the joint distribution might not be feasible!

* Viterbi Algorithm: Use dynamic programming (like in DTW!)

* We define
$$\omega(\mathbf{z}_n) = \max_{\mathbf{z}_1, \dots, \mathbf{z}_{n - 1}}\log p(\mathbf{x}_1,\dots, \mathbf{x}_{n-1}, \mathbf{z}_1,\dots, \mathbf{z}_{n-1})$$

(compare to the Dynamic Time Warping distance!)

#### Algorithm

**Inputs**

* Sequence of observations $\mathbf{X}=\{\mathbf{x}_1, \dots, \mathbf{x}_N\}$
* Input probabilities $\mathbf{\pi}$
* Transition Matrix $\mathbf{A}$
* Observation model (to compute $p(\mathbf{x}_{n} \mid \mathbf{z}_n)$)

**Initialization**

We want to compute

$$\omega(\mathbf{z}_1) =  \log p(\mathbf{x}_1 \mid \mathbf{z}_1) + \log p(\mathbf{z}_1)$$

* For $k \in [1, K]$
    $$ \omega_{1k} = p(\mathbf{x}_n \mid  \mathbf{\phi}_k) + \log \pi_k $$

**Viterbi interation**

We want to compute

$$\omega(\mathbf{z}_n) = \log p(\mathbf{x}_{n} \mid \mathbf{z}_{n}) + \max_{\mathbf{z}_n} \left\{\log p(\mathbf{z}_{n} \mid \mathbf{z}_{n-1}) + \omega(\mathbf{z}_{n - 1})  \right\}$$

* For $n\in[2, N]$
    * For $k \in [1, K]$
        $$\omega_{nk} = p(\mathbf{x}_n \mid  \mathbf{\phi}_k) + \max_i\{ A_{ik} + \omega_{n-1, i} \}$$_____no_output_____

import numpy as np
from collections import defaultdict_____no_output_____def viterbi_algorithm(observations, init_distribution,
                      observation_model, transition_probabilities,
                      state_space=None):
    """
    Find the most probable sequence of latent variables given
    a sequence of observations

    Parameters
    ----------
    observations: iterable
       An iterable containing observations. The type of each
       element depends on input types accepted by the
       `observation_model`
    init_distribution: np.array
        A 1D vector of length n_states defining the initial
        probabilities of each state
    observation_model : callable
        A method for computiong the observation (emission) probabilities.
    transition_probabilities: np.ndarray
        A (n_states, n_states) matrix where component
        [i, j] represents the probability of going to state j
        from state i.

    Returns
    -------
    path: np.ndarray
        The most probable sequence of latent variables
    likelihood: float
        The log-likelihood of the best sequence.
    """
    # Initialize matrix for holding the best sub-sequence
    # log-likelihood
    n_states = len(transition_probabilities)
    omega = np.zeros((len(observations), n_states))
    # Initialize dictionary for tracking the best paths
    path = defaultdict(lambda: list())

    # Initiate for i == 0
    obs_prob = observation_model(observations[0])
    omega[0, :] = obs_prob + init_distribution
    # omega[0, :] = 0

    # Initialize path
    for i in range(n_states):
        path[i].append(i)

    # Viterbi recursion
    for i, obs in enumerate(observations[1:], 1):
        obs_prob = observation_model(obs)
        for j in range(n_states):
            # prob, state = 0, 0
            prob, state = max(
                [(omega[i - 1, k] + transition_probabilities[k, j], k)
                 for k in range(n_states)],
                key=lambda x: x[0]
                )
            omega[i, j] = obs_prob[j] + prob

            # keep track of the best state
            path[j].append(state)

    # Get index of the best state
    best_sequence_idx = omega[-1, :].argmax()
    # Get best path (backtracking!)
    best_sequence = np.array(path[best_sequence_idx][::-1], dtype=int)
    if state_space is not None:
        best_sequence = state_space[best_sequence]
    # likelihood of the path
    path_likelihood = omega[-1, best_sequence_idx]

    return best_sequence, path_likelihood_____no_output_____

Here is an example observation model, for the case that $\mathbf{x}_i$ can take $M$ different states, and that the probability $p(\mathbf{x}_n = m \mid \mathbf{z}_n = k) = c_{n,k}$ is constant._____no_output_____

class CategoricalObservationModel(object):

        def __init__(
                self,
                observation_probabilities,
                observations=None,
                use_log_probabilities=True
        ):
            self.use_log_probabilities = use_log_probabilities

            self.observation_probabilities = observation_probabilities

            if observations is not None:
                self.observations = list(observations)
            else:
                self.observations = [
                    str(i) for i in len(observation_probabilities)
                ]

        @property
        def observation_probabilities(self):
            if self.use_log_probabilities:
                return self._log_obs_prob
            else:
                return self._obs_prob

        @observation_probabilities.setter
        def observation_probabilities(self, observation_probabilities):
            self._obs_prob = observation_probabilities
            self._log_obs_prob = np.log(self._obs_prob)

        def __call__(self, observation, *args, **kwargs):
            idx = self.observations.index(observation)
            return self.observation_probabilities[idx]_____no_output_____obs = ("normal", "jacket", "umbrella")
observations = ("normal", "umbrella", "jacket", "umbrella", "normal")
states = ("Sunny", "Raining")
observation_probabilities = np.array([[0.5, 0.1],
                                      [0.4, 0.3],
                                      [0.1, 0.6]])
transition_probabilities = np.array([[0.7, 0.4],
                                     [0.3, 0.6]])

observation_model = CategoricalObservationModel(
    observation_probabilities, obs
)

init_distribution = np.array([0.6, 0.4])

best_sequence, log_likelihood = viterbi_algorithm(
    obs, init_distribution,
    observation_model, transition_probabilities,
    state_space=np.array(states)
)

print(best_sequence)['Sunny' 'Sunny' 'Raining']
np.exp(observation_model("normal"))_____no_output_____