# Notebook from Exia-Aix-2016/Projet-Data
Path: ProjetDATA.ipynb

# PROJET DATA
_____no_output_____## Contexte

Depuis les années 90, il y a eu une véritable prise de conscience mondiale de la nécessité de réduire la consommation d’énergie et des émissions de gaz à effet de serre. Les premiers engagements sont apparus lors de la signature du protocole de Kyoto en 1997. Mais son entrée en vigueur n’a finalement eu lieu qu’en 2005 et de nombreux scientifiques ont jugé les efforts insuffisants pour ralentir le réchauffement climatique. Depuis, d’autres engagements plus ambitieux ont vus le jour (division par 4 des émissions d’ici 2050 pour la France par exemple, engagements de certaines grandes villes comme Paris). Mais la tâche est compliquée. Les pouvoirs publics et les collectivités territoriales n’ont pas la possibilité d’obliger les entreprises et les particuliers à changer leurs habitudes pour atteindre ces objectifs. L’action se porte donc avant tout à faire évoluer les comportements. L’économie et le recyclage des matières premières, l’amélioration des modes de transports et des performances énergétiques des bâtiments doivent devenir des priorités.

Dans ce sens, L’ADEME (Agence de l’Environnement et de la Maîtrise de l’Energie) a récemment lancé un appel à manifestation d’intérêt pour promouvoir la réalisation de démonstrateurs et d’expérimentations de nouvelles solutions de mobilité pour les personnes et les marchandises adaptées à différents types de territoires. Votre structure CesiCDP est déjà bien implantée dans le domaine. Aidé de nombreux partenaires, vous avez réalisé plusieurs études sur le thème de la Mobilité Multimodale Intelligente. Les nouvelles technologies de transport, plus économiques et moins polluantes ne sont pas sans poser de nouveaux défis notamment d’un point de vue de l’optimisation de la gestion des ressources. Mais ces problèmes de logistique du transport présentent un enjeu majeur pour l’avenir : ses applications sont nombreuses (distribution du courrier, livraison de produits, traitement du réseau routier, ramassage des ordures) et leur impact sur l’environnement peut être véritablement significatif. Votre étude s’inscrit donc dans le cadre d’une réponse à l’appel de l’ADEME.
_____no_output_____## Sujet

Le but de votre étude est de générer une tournée de livraison (problème du VRP). Le problème algorithmique consiste donc à calculer sur un réseau routier une tournée permettant de relier entre elles un sous-ensemble de villes, puis de revenir à son point de départ, de manière à minimiser la distance totale parcourue.

Vous devrez proposer une méthode issue de la Recherche Opérationnelle pour générer une tournée de livraison correspondant à ce problème. L’implémentation se fera sur une version de base du problème, à laquelle vous pourrez ajouter des contraintes supplémentaires, rendant le problème plus réaliste, mais aussi plus dur à traiter.

Par ailleurs, vous devrez effectuer une étude statistique du comportement de votre méthode de résolution, faisant apparaitre ses performances (qualité de solution, temps de convergence). Idéalement, des statistiques prédictives permettent d’extrapoler ce comportement sur des cas d’usages que vos ordinateurs seuls ne pourraient traiter._____no_output_____## Spécificités techniques

Dans le cas de notre résolution, nous avons décidé de mettre en place les contraintes suivantes dans le cas du sujet :

*   Le graphique est constitué d'un nombre donné de nodes (qui représenteront les villes)
*   Chaque ville attend un nombre aléatoire de produits
*  Le nombre de camions disponibles est défini par l'utilisateur
*  Les camions ont une capacité de stockage définie par l'utilisateur
*  Tous les camions partent de la même ville
*  La distance entre les villes est définies par leur position géographique

_____no_output_____# Présentation de la solution_____no_output_____## Bibliothéques Python

Pour répondre à nos besoins dans le cadre du développement de la solution, nous avons eu recours à plusieurs bibliothéques python._____no_output_____### Google OR-Tools

Pour la mise en place de la solution, nous avons choisi d'utiliser bibliothéque Google OR-Tools, qui permet d'implémenter plusieurs algorithmes de résolution, voici son implémentation :
_____no_output_____

<code>
# Create and register a transit callback.
    def distance_callback(from_index, to_index):
        """Returns the distance between the two nodes."""
        # Convert from routing variable Index to distance matrix NodeIndex.
        from_node = manager.IndexToNode(from_index)
        to_node = manager.IndexToNode(to_index)
        return data['distance_matrix'][from_node][to_node]

    transit_callback_index = routing.RegisterTransitCallback(distance_callback)

    # Define cost of each arc.
    routing.SetArcCostEvaluatorOfAllVehicles(transit_callback_index)

    # Add Distance constraint.
    dimension_name = 'Distance'
    routing.AddDimension(
        transit_callback_index,
        0,  # no slack
        3000,  # vehicle maximum travel distance
        True,  # start cumul to zero
        dimension_name)
    distance_dimension = routing.GetDimensionOrDie(dimension_name)
    distance_dimension.SetGlobalSpanCostCoefficient(100)

    # Setting first solution heuristic.
    search_parameters = pywrapcp.DefaultRoutingSearchParameters()
    search_parameters.first_solution_strategy = (
        routing_enums_pb2.FirstSolutionStrategy.PATH_CHEAPEST_ARC)

    # Solve the problem.
    solution = routing.SolveWithParameters(search_parameters)

    # Print solution on console.
    if solution:
        print_solution(data, manager, routing, solution)
_____no_output_____
</code>

Ici, on peut voir que l'on choisir le dataset qui va être utilisé, ainsi que le type d'algorithme qui va être implémenter (avec les particularités s'il est heuristique ou méta-heuristique)_____no_output_____## Description des algorithmes

Dans la résolution de notre de problème, nous avons décidé de choisir 3 algorithmes de résolution de problème pour répondre à la problèmatique. Le but de ce document est donc de faire une description de ces algorithmes, de présenter leurs spécificités, ainsi que les résultats que l'on a obtenu lors de leur utilisation.
_____no_output_____### Algorithme de recherche locale

La recherche locale est une méthode générale utilisée pour résoudre des problèmes d'optimisation, c'est-à-dire des problèmes où l'on cherche la meilleure solution dans un ensemble de solutions candidates. La recherche locale consiste à passer d'une solution à une autre solution proche dans l'espace des solutions candidates (l'espace de recherche) jusqu'à ce qu'une solution considérée comme optimale soit trouvée, ou que le temps imparti soit dépassé._____no_output_____#### Implémentation

Voici le schéma de fonctionnement de l'algorithme :
![recherche locale](http://www.tony-lambert.fr/these/Images/ls4.png)_____no_output_____####Solution
Voici la représentation graphique de la solution de cet algorithme, pour plus de lisibilité nous avons supprimé les liens vers le dépot
![GARL](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/graphs_no_depot/path_cheapest_arc.png)
_____no_output_____### Algorithme de recherche tabou

L'idée de la recherche tabou consiste, à partir d'une position donnée, à en explorer le voisinage et à choisir la position dans ce voisinage qui minimise la fonction objectif.

Il est essentiel de noter que cette opération peut conduire à augmenter la valeur de la fonction (dans un problème de minimisation) : c'est le cas lorsque tous les points du voisinage ont une valeur plus élevée. C'est à partir de ce mécanisme que l'on sort d'un minimum local.

Le risque cependant est qu'à l'étape suivante, on retombe dans le minimum local auquel on vient d'échapper. C'est pourquoi il faut que l'heuristique ait de la mémoire : le mécanisme consiste à interdire (d'où le nom de tabou) de revenir sur les dernières positions explorées.

Les positions déjà explorées sont conservées dans une file FIFO (appelée souvent liste tabou) d'une taille donnée, qui est un paramètre ajustable de l'heuristique. Cette file doit conserver des positions complètes, ce qui dans certains types de problèmes, peut nécessiter l'archivage d'une grande quantité d'informations. Cette difficulté peut être contournée en ne gardant en mémoire que les mouvements précédents, associés à la valeur de la fonction à minimiser._____no_output_____#### Implémentation

Voici le schéma de l'implémentation de l'algorithme :
![recherche tabout](https://slideplayer.fr/slide/1752884/7/images/6/2.+Algorithme+de+recherche+tabou.jpg)_____no_output_____#### Solution
![GTR](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/graphs_no_depot/tabu_search.png)_____no_output_____### Algorithme du gradient

L'algorithme du gradient est également connu sous le nom d'algorithme de la plus forte pente ou de la plus profonde descente (steepest descent, en anglais) parce que le gradient est la pente de la fonction linéarisée au point courant et est donc, localement, sa plus forte pente (notion qui dépend du produit scalaire)._____no_output_____#### Implémentation

Voici le schéma de l'implémentation de l'algorithme :
![gradient descendant](https://www.charlesbordet.com/assets/images/gradient-descent/gradientdescent-f3.gif)_____no_output_____#### Solution
![ADGG](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/graphs_no_depot/path_cheapest_arc.png)_____no_output_____### Etude de la complexité

Le diagramme suivant présente la compléxité de nos différents algorithmes :

![complexite](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/complexity.png)

On retrouve alors 2 cas :
* Le cas de l'algorithme heuristique (PATH_CHEAPEST_ARC) qui dispose de tout le temps qu'il désire pour trouver une solution qui sera donc plus précise. , On va pouvoir par la suite dégager une courbe représentation de la compléxité, qui sera utilisable dans le cas où l'on va vouloir prédire le comportement de l'algorithme
* Les cas des algorithmes méta-heuristiques ( et ). En effet, de part leur nature méta-heuristique, nous sommes obligés d'imposer un temps limite à l'exécution de l'agoithme, ce pourquoi l'on va retrouver un temps d'exécution constant, quel que soit la taille du problème passé en paramétre._____no_output_____## Etude Statistique

Dans cette partie seront présentées les différentes études statistiques qui ont été réalisées. Elles seront sous forme de graphiques, qui seront décrits et interprétés._____no_output_____## Uniformité
L'uniformité montrera la stabilité du temps d'execution selon un même dataset, en effet pour une configuration  donné par exemple avec n villes, n camions, n packet p des dataset seront générés avec des distances entre les villes différentes puis l'algorithme sera executé sur chacun des ces datasets ainsi créés pour en sortir l'uniformité.

On peut uniquement mesurer l'algorithme heuristique, étant donné que les méta-heuristique s'arrête à la limite imposé.

visualisation avec 100 villes :
(axe des ordonnées correspond au nombre d'itération)

![Uniformité_100](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/uniformity/path_cheapest_arc_100.png)

Visualisation avec 500 Villes :

![Uniformité_500](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/uniformity/path_cheapest_arc_500.png)

On constate donc que pour peu de ville le temps d'execution est chaotique, cependant plus le nombre de ville augmente plus l'uniformité est prédictible._____no_output_____# Statistiques descriptives_____no_output_____#Evolution des variables selon le nombre de villes
Nous allons représenter l'évolution des variables suivante : nombre de camions, temps d'exécution et la distance parcourues
cette représentation est sous forme de nuage de points._____no_output_____## Temps d'execution
Nous rappelons que pour les méta-heurisque atant donné leur cap en temps de recherche imposé, il n'est pas pertinent de le représenter.
Voici la représentation pour l'heuristique :
![executiontimeScatter](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/scatter_plot/execution_time/path_cheapest_arc.png)
Chaque points rouge corresponds a une execution de l'algorithme sur un même dataset (comme pour l'uniformité) et la courbe bleu représente une fonction d'adoucissement sur la moyenne des points.
Nous pouvons observer que plus le nombre de ville augmente plus le temps d'execution augmente et ainsi via la courbe prédir l'évolution du temps d'execution qui est comme vue ci-dessus ext extrêmement croissant à partir de 1000 villes. Nous recommandons encore une fois l'utilisation d'une méta-heuristique._____no_output_____##Distance parcourue par les camions et nombre de camions utilisés
Dans cette section verrons qu'il est intéressant de comparer l'évolution de la distance parcourue par les camions dans un dataset donné pour chaque algorithme qu'il soit heuristique ou méta-heuristique_____no_output_____### Algorithme de recherche locale

![ARLD](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/scatter_plot/distance/guided_local_search.png)
*Distance*
![azer](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/scatter_plot/trucks/guided_local_search.png)
*Nombre de camions*

Dans ce cas on peu observer que la distance parcourue est relative au nombre de villes générée, or contrairement à ce qui est présenté sur la courbe on peu suposer que la croissance exponantielle à partir de l'interprétation mathématique peut être fausse (besoin de plus de calcul)_____no_output_____### Algorithme  de recherche tabou

![ART](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/scatter_plot/distance/tabu_search.png)
*Distance*
![bla](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/scatter_plot/trucks/tabu_search.png)
*Nombre de camions*

Dans ce cas on peu observer que la distance parcourue est relative au nombre de villes générée, or contrairement à ce qui est présenté sur la courbe on peu suposer que la croissance exponantielle à partir de l'interprétation mathématique peut être fausse (besoin de plus de calcul)

En compararant avec l'algorithme de recherche locale on peu voir que les courbes sont similaire. On peu donc déduire que cette statistique descriptive est relative à la nature méta-heuristique de l'algorithme._____no_output_____###Algorithme du gradient

![AG](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/scatter_plot/distance/path_cheapest_arc.png)
*Distance*
![dqs](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/scatter_plot/trucks/guided_local_search.png)
*Nombre de camions*

Dans ce cas on peu observer que la distance parcourue est relative au nombre de villes générée, or contrairement à ce qui est présenté sur la courbe on peu suposer que la croissance exponantielle à partir de l'interprétation mathématique peut être fausse (besoin de plus de calcul)

Dans ce cas-ci l'algorithme utilisé est heuristique, de ce fait l'hypothèse précédente est fausse, en effet on retrouve à nouveau les même similitude que les deux précédent algorithmes (tendance de la courbe). On peu donc en déduire que celle-ci n'est pas relative à la nature de l'algorithme, on pourra donc supposer de nouvelles hypothèses dans le cas d'une analyse plus poussée._____no_output_____## Comparaison des algorithmes

L'histogramme de qualité :

L'histogramme de qualité nous permet de visualiser intuitivement la performance de chaque algorithme, en effet nous allons pouvoir voir la différence entre le temps d'execution, le nombre de camion utilisé ainsi que la distance total parcourues par les camions ainsi qu'un ratio représetant la qualité et qui se calcule comme suit :

$qualité = (\frac{distance_{parcourue}}{distance_{arrêtes}}) \times (\frac{camions_{utilisés}}{camions_{disponibles}})$

![quality](https://raw.githubusercontent.com/Exia-Aix-2016/Data-Project/master/screenshots/chart_quality.png)

En rouge : Algorithme de recherche locale ; en vert : l'algorithme de recherche tabou ; en bleu l'algorithme du gradient.

On peut voir que les deux algorithmes méta-heuristique, malgré un temps d'execution limité on une variation de qualité assez faible, on peut donc en déduire que sur de très gros dataset, l'utilisation d'une métat-heuristique est recommandée pour une application industrielle (Important nombre de ville important, une grande flotte de véhicule et un besoin d'exécution rapide).

_____no_output_____# CONCLUSION
D'après nos études poussées on peut en conclure  les éléments suivant :
* L'utilisation d'algorithme heuristique est pertinente dans le cas de jeux de données petit (Moins de 1000 villes ou noeuds).
* L'utilisation des algorithmes méta-heuristique est pertinente dans le cas de jeux de données grans (plus de 1000 villes ou noeuds).
* L'étude de complexité n'est pas relatives à la nature de l'algorithme.
* L'étude de la qualité a permis de démontrer que dans un dataset donné, méta-heuristiques et heuristiques ont des caractèristiques d'execution similiraire (Sauf le temps d'exécution).

On peu donc suposer que les meilleurs algorithmes reposent sur un calcule en amont par l'utilisation d'heuristiques qui réalise un pré-traitement sur les donnés qui sont ensuite analysable pour les méta-heuristique._____no_output_____


# Notebook from Gordonias/Cogsci
Path: Group Project - Final Assignment - Q1-3 (1).ipynb

# Cognitive Neuroscience: Group Project

## Final Group Project Code Instructions

Marijn van Wingerden, Department of Cognitive Science and Artificial Intelligence – Tilburg University Academic Year 2020-2021_____no_output_____In this Jupyter Notebook, you will find the programmatic instructions to complete the Group Project.

You will analyse a subset of the trials and conditions from the RITA dataset, as introduced by dr. Roncaglia in the second week of class. Your time-resolved analysis, using the FFT and baseline averaging, will be performed across ALL subjects in a group (Monolinguals, Early Learners, Late Learners) in the dataset, and look for oscillatory activity in relation to the onset of the **critical item** in each sentence (the auxilary verb).

You will inspect activity in the following frequency bands:
- Delta (1-4 Hz)
- Theta (4-8 Hz)
- Alpha (8-12 Hz)
- Beta (15-25 Hz)
- low Gamma (30-60 Hz)
- high Gamma (60-100 Hz)
(different cutoffs points can be found in the literature, we are sticking with these for convenience)

All relevant methods have been covered in Worksheets 1-7, with the exception of loading multiple datafiles. Whenever a new method/function is introduced here, it will come with an example.
_____no_output_____## Datafiles - assignment per group

Each group will analyze one set of filler sentences (NA: Non-Ambiguous sentences) and a set of experimental sentences (AM: Ambiguous sentences). The conditions in the dataset are split over groups in the following way:

- SF: subject-first
- OF: object-first
- IR: irregular rhythm
- RR: regular rhythm

- Monolinguals: RM
    - Groups 01 + 02: SFIR triggers XX1 and XX5
    - Groups 03 + 04: SFRR triggers XX2 and XX6
    - Groups 05 + 06: OFIR triggers XX3 and XX7
    - Groups 07 + 08: OFRR triggers XX4 and XX8
- Early Learners: RB
    - Groups 09 + 10: SFIR triggers XX1 and XX5
    - Groups 11 + 12: SFRR triggers XX2 and XX6
    - Groups 13 + 14: OFIR triggers XX3 and XX7
    - Groups 15 + 16: OFRR triggers XX4 and XX8
- Late Learners: RL
    - Groups 17 + 18: SFIR triggers XX1 and XX5
    - Groups 19 + 20: SFRR triggers XX2 and XX6
    - Groups 21 + 22: OFIR triggers XX3 and XX7
    - Groups 23 + 24: OFRR triggers XX4 and XX8

- Odd groups:
analyse the odd trials in the dataset
- Even groups:
analyse the even trials in the dataset

You can download the datafiles from: https://surfdrive.surf.nl/files/index.php/s/JcA9speED020q4p
_____no_output_____## Handing in of your code

You can adapt this script template and hand it in as the code component of your Group Assignment Report.

Whenever you are asked to make a plot, it should be completed with a meaningful plot title, xlabel and ylabel texts. Figures are started with a Matplotlib figure handle: "fig_Q2A, ax = plt.subplots;". This indicates that a link (called handle) to your figure will be saved in the variable, so we can easily check it when checking your scripts. Whenever a naming convention for a variable is given, use it, because it will allow semi-automatic grading of your project script.

### Intermediate hand-in

You will be able to hand in a script/notebook with your solutions to Q1-3 after the midterms, and the correct solution will be discussed in class to give you some feedback on how the Group Project is evaluated. Groups that do not hand in their solutions to Q1-3 will receive half of the total points available for these questions by default. _____no_output_____## Group members:

Please list the contributors and their U-numbers here in comments:

- Emil Anker Wiik
- Gordon Arscott
- Nina Ella Gjessing
- Louis Dowden
- Babette Scheepers_____no_output_____## Setting up: list your modules to import
For loading/saving puroposes, we will make use of the **os** package.
An example worksheet with instructions on how to use the os package will be provided_____no_output_____

#%matplotlib inline

import numpy as np
import os

#The OS module in Python provides functions for interacting with the operating system.
#OS comes under Python's standard utility modules.
#This module provides a portable way of using operating system-dependent functionality.
#The *os* and *os. path* modules include many functions to interact with the file system

#os.getcwd() #check the directory
#os.mkdir("C:\MyPythonProject") #create a directory
#os.chdir("C:\MyPythonProject") #change the directory
#os.listdir("c:\python37") #returns list of files in the directory

#os.path.join() method in Python join one or more path components intelligently.
#This method concatenates various path components with exactly one directory separator (‘/’)
#following each non-empty part except the last path component.
#If the last path component to be joined is empty then a directory seperator (‘/’) is put at the end._____no_output_____

## Data loading
We will need to load the datafiles from all participants and add them all together so that we end up with a matrix that has nChannels x nTime x nParticipants (instead of trials). You can make your work easier by organising the datafiles in such a way that you put the control.npy files in their own subdirectory, and the experimental.npy files as well.

In order to load the files, we can use the os package.

Adapt the following so that it works on your machine:_____no_output_____

path_control = "./group_12_control"
path_experimental = "./group_12_experimental"

control_files = os.listdir(path_control)
experimental_files = os.listdir(path_experimental)

# check that the length of your files list matches the provided datafiles, and contains only .npy datafiles

if len(control_files) and len(experimental_files) == 22:
    print(True)

os.listdir(path_control), os.listdir(path_experimental)True

## Combining data and matrix pre-allocation
next, you will need to load these files one by one and extract the data for this participant.
The data in the NumPy arrays are stored as Trials x Channels x Time. To aggregate across participants, you will thus need to add a 4th dimension to store the data.

To be able to adequately pre-allocate the data from the different subjects, we will load one trial subject manually to have a look at the shape/dimensionality of the data:_____no_output_____

EEG = np.load(os.path.join(path_control,control_files[0]))

# control_files is a list of strings, so indexing its first element returns a string
# in this case, we are loading the first entry of control_files, i.e. participant 1

# verify that the number of trials equals 22,
# verify that the number of channels equals 64 or 65
# and verify that there are 751 samples per trace

np.shape(EEG)
#(Trials, Channels, Time)
_____no_output_____

## Q1 - setting up the data structure and loading data from all participants

The EEG data is currently stored as a 3-dimensional NumPy array. But to run our time-frequency analysis, we need some more information like the sampling rate and the time axis that corresponds to the stimulus-locked analysis window. In order to set up (=pre-allocate) a matrix that will hold all traces for all participants, we need to know the sizes of the dimensions of this 4-dimensional matrix, and fill up this matrix by looping over participants:_____no_output_____

# There are 64 or 65 channels in the dataset. Only channels up to channel 59 are EEG channels
# the remaining channels are EMG and EOG channels that we will ignore in this analysis
# subset your EEG array so that only the EEG channels remain

EEG = EEG[:, 0:59]
np.shape(EEG)

# Define nTrials, nChan, nSamples and nPart. Then, pre-allocate a matrix
# named comb_data, filled with zeros and with size nTrials, nChans x nSamples x nParticipants

nTrials = 22
nChans = 59 # number of channels
nSamples = 751 #time samples?
nParts = 22 # number of participants

comb_data = np.zeros((nTrials, nChans, nSamples, nParts))
np.shape(comb_data)

# next, we need to loop over all participant datafiles and add them to the appropriate slice in your 4-D array
# For this, you need to use specific array indexing to indicate where in comb_data each participant's data
# needs to go. You can and should reuse the data-reading code above.

# loop over participants, and wihtin each iteration of the loop, load the
# next datafile and fill comb_data with the EEG traces (nTrials x nChans x nSamples)

print(type(comb_data))

for each_participant in range(nParts):
    EEG = np.load(os.path.join(path_control,control_files[each_participant]))
    comb_data.append[EEG]
    ... # caution - multiple lines of code might be needed

#EXAMPLE FROM LECTURE 1 FOR FILLING IN ZERO MATRIX IN 2D

#freq_mat = np.zeros((3,1000))

#freqs = np.array([2,5,7])
#thetas = np.array([0.5, 1, 1.5])*np.pi

#for iMat in range(freq_mat.shape[0]):
#    freq_mat[iMat,:] = np.sin(2*np.pi*freqs[iMat]*time + thetas[iMat])
<class 'numpy.ndarray'>

## Q2 - explore the data

Let's explore this newly combined dataset a little bit. This sections fo EEG traces in your dataset have been taken with a [-0.5s, 1s] window aroun the relevant event. What's more, each trace has been averaged to its baseline period, so that the mean amplitude should be 0 (with some rounding error).

To verify, first, determine the mean for the time period of -0.5 to 0 seconds. Given that the srate = 500 Hz, the baseline period corresponds to the first 250 samples
- subset your combined data to only the first 250 samples
- select a random participant and sunset the data further to only this participant
- select a random EEG channel and subset the data further to only this channel

This should leave you with a nTrials x 250 (samples) matrix. Create a similar evoked matrix with the remainder of the samples. With these matrices:
- plot the traces for all trials in the **baseline** matrix (use transpose if necessary).
- calculate the mean for each trace:
    - once for the baseline period
    - once for the remainder of the trial
- plot these values (N= nTrials, check) in a histogram, each in their own subplot

Refer to Worksheet 1 for example uses of **np.mean**. np.std works in a similar way_____no_output_____

baseline_mean = ...
evoked_mean = ...

# now plot these three histograms using the subplot given
fig_Q2A, ax = plt.subplots(figsize=(8,3), nrows=1, ncols=3) # 1x3 graph

# plot the traces in ax[0]

##
## your code here
##

# plot the baseline mean in ax[1] with this code
ax[1].hist(baseline_mean)
plt.axes(ax[1])
plt.title('Baseline means')
ax[1].set_xlabel('Mean voltage')
ax[1].set_ylabel('count')

# plot the evoked mean in ax[2] with this code
ax[2].hist(evoked_mean)
plt.axes(ax[2])
plt.title('Evoked means')
ax[2].set_xlabel('Mean voltage')
ax[2].set_ylabel('count')_____no_output_____

Looking at theses histograms, you should see that the distribution of evoked mean values can vary: while EEG recordings are referenced to a common ground, and usualy then re-referenced to the global average, we are dealing here with specific cutouts of EEG traces around specific events in the dataset. The data has been normalised to the baseline window for each trial, but the mean of the evoked part is not controlled. In addition, there might be differences in the size of the amplitudes between channels and between participants due to differences in conductivity.

Illustrate that this global averaging does not guarantee equal variance. Reuse the baseline and evoked subsets:
- Compute, instead of the mean across samples, the standard deviation (numpy.std or variants)
- calculate the mean for each trace:
    - once for the baseline period
    - once for the remainder of the trial
- Adapt the plotting code for Q2A and plot these distributions of standard deviations in figure Q2B._____no_output_____

baseline_std = ...
evoked_std = ...

fig_Q2B, ax = plt.subplots(figsize=(8,3), nrows=1, ncols=3) # don't forget proper plot annotation

##
## your code here
##

_____no_output_____

## Q3 - normalizing the data

So, if there are large differences in mean and/or standard deviation between channels or participants, we can implement some standard scaling. First, let think about why this "standard scaling" (substracting the mean, dividing by std) is important? You will be combining data from different participants for these exercises. They have been possibly recorded at different days, with different gels or electrodes, and thus with different conductivity between participants. We will assume that the recording across videos within one participant remains stable. So, in order to compare and average the recordings from different participants "fairly", we want them to be on more or less the same scale.

We can thus attempt to normalize the signal per participant, by dividing all data per participant by its standard deviation. Let's show the extent of the problem by plotting the participants with the lowest and highest std side by side. Re-create your matrix of std values for the evoked period, but:
- do not subset one participant, but retain all participants (still only selecting 1 channel)
- average the std values for each trace over trials, save in part_std (you will retain one value per participant)
- use np.argmax (and variants) to create two indexes, min_std and max_std that point to the participants with the lowest and highest standard deviations
- calculate and plot the average over trials for these two participants, using different line colors and proper line labeling.
    - Plot them in the first subplot of a 1x2 subplot
- Observe the scaling difference

_____no_output_____

##
## your code here
##

part_std =
min_std =
max_std =

fig_Q3, ax = plt.subplots(figsize=(8,3), ...) # don't forget proper plot annotation
..._____no_output_____

Next, we want to use the standard deviation for each channel to normalize (i.e. divide) the data by this value.
- Re-use the standard deviation per participants calculated above
- Normalize the evoked matrix (per participant) by the participant std (save as evoked_norm)
- plot the **normalized** average over trials for these two participants, using different line colors and proper line labeling.
    - Plot them in the second subplot of fig Q3
    - The data should now be more or less on the same scale_____no_output_____

##
## your code here
##

evoked_norm = ..._____no_output_____

We now have to apply this normalization across all channels. The fairest way is to calculate the grand standard deviation per participant (over all their channels, so the relative scaling between channels remains intact). This will normalize the range of the signal within one participants, so that they will be comparable between participants.

In order to do this:
- first preallocate a normalized matrix with the same size as comb_data, called comb_data_norm.
- Next, create a loop to go over Participants, and inside the loop:
    - calculate the grand standard deviation per participant
    - normalization all values by this grand standard deviation
        - Examine the numpy.std documentation to get a single std value across a 3D matrix
- Then, recreate the plot with the trial averages as made for the first subplot of fig_Q2
    - It should now rather resemble the normalized plot as made in the second subplot
- Save this figure as Figure1

_____no_output_____

##
## your code here
##

comb_data_norm = ...

for
    ..
end

fig_Q3, ax = plt.subplots(figsize=(8,3), ...) # 1x1 fig, don't forget proper plot annotation
...

# save Figure Q3C as your Figure 1 for your report
Figure1 = fig_Q3;_____no_output_____


# Notebook from DAWells/LitMD
Path: litmd_writer.ipynb

# LitMD writer
An interface for writing paper summaries for litmd._____no_output_____

from pathlib import Path
import os
import pandas as pd
import numpy as np

# import itertools

# %load_ext rpy2.ipython_____no_output_____papers = os.listdir("papers")_____no_output_____

## Tag summary
Print a list of all tags and the number of papers with that tag._____no_output_____

tag_list = list()

for paper in papers:
#     print(paper)
    file = "papers/" + paper
    with open(file, "r") as f:
        lines=f.readlines()
    #Index of tag start and end
    tagi = [i for i,line in enumerate(lines) if "tags" in line]
    tags = lines[tagi[0]+1 : tagi[-1]]
    #Remove new line character
    tags = [tag[:-1] for tag in tags]
    tag_list.append(tags)# = tag_list + tags

#Flatten tag_list
flat_tag_list = [tag for paper_tags in tag_list for tag in paper_tags]

alphabetised_list = pd.Series(flat_tag_list).value_counts().sort_index()
print(alphabetised_list)                                          5
ADE                                       2
MHC processing                            3
MHC_processing                            1
ancestral_reconstruction                  8
antibody                                 10
antigen                                  12
assay_prediction                          6
coronavirus                              74
coronavirus, vaccine, immune response     1
crossreactive                             9
epitope                                  15
epitope prediction                        6
epitope_prediction                        1
flu-vaccine                              16
flu_evolution                             6
hemagglutinin                             6
immune correlates                         2
immune response                          14
indels                                    4
machine learning                          3
machine_learning                         22
machines learning                         1
phylogenetic non-independence             1
phylogenetics                             7
sequence_alignment                        3
statistics                                8
substitution_matrix                       2
vaccine                                  16
vaccine design                           25
virus entry                               1
virus evolution                          12
zoonotic disease                          7
dtype: int64
title = ""

year = 2021

authors = """"""

#No new lines before, between, or after tags
tags = """"""

#When using mathjax \t must be \\t eg \\theta rather than \theta
text = """
"""_____no_output_____#Remove bad characters from title and make into file name

#Shorten title to 100 characters for file name
file = "papers/" + title[:100].replace(":", "") + ".md"

#If file already exists don't overwrite
if os.path.exists(file):
    print("File already exists")
else:
    with open(file, 'w') as f:
        #Write metadata
        f.write("# " + title + "\n")
        f.write("## %s\n" %year)
        f.write("## Authors\n" + authors + "\n")

        #write tags
        f.write("### tags\n")
        f.write(tags)
        f.write("\n### tags\n")

        #Write summary of paper
        f.write(text)_____no_output_____title = ""

year = 2021

authors = """"""

#No new lines before, between, or after tags
tags = """"""

#When using mathjax \t must be \\t eg \\theta rather than \theta
text = """
"""_____no_output_____


# Notebook from ckrilow/dev-ckrilow
Path: studies/chen-t1d/stimulation_collins/analysis/diffxpy.ipynb

%load_ext autoreload
%autoreload 2

import scanpy as sc
import matplotlib.pyplot as plt
import seaborn as sns
import logging
import numpy as np
import pandas as pd
import scipy.stats_____no_output_____import diffxpy.api as de_____no_output_____

# Read data:_____no_output_____Read data. This data is result from using the Parker lab thresholds in Anderson pipeline._____no_output_____

data = sc.read_h5ad("../results/nf-sample_qc=parker_singlet-parameter_sweep_v001_harmony/normalize=total_count.vars_to_regress=none.hvg_exclude=data-variable_gene_filter_ribo_mt.scores=data-gene_scores/reduced_dims-vars_to_regress=none-harmony.n_pcs=23.variables=experiment_id.thetas=1.0/cluster.number_neighbors=30.method=leiden.resolution=0.25/adata-normalized_pca-umap-clustered.h5ad")_____no_output_____

# Running cluster-specific testing_____no_output_____We can now partition the data set by cluster and conduct a test across conditions in each cell type._____no_output_____

part = de.test.partition(
    data=data,
    parts="cluster"
)
test_part = part.wald(
    formula_loc="~ 1 + condition",
    factor_loc_totest="condition",
    size_factors=data.obs["normalization_factor"]
)training location model: True
training scale model: True
iter   0: ll=220871631505.539398
iter   1: ll=220871631505.539398, converged: 0.00% (loc: 100.00%, scale update: False), in 89.17sec
Fitting 22741 dispersion models: (progress not available with multiprocessing)

Note that there is one test and p-value for each partition for each gene now. We can summarize test statistics across partitions using summary:_____no_output_____

test_part.summary().iloc[:10, :]_____no_output_____

Or look the results of a single partition:_____no_output_____

test_part.tests[test_part.partitions.index("c1")].summary().iloc[:10, :]_____no_output_____

# Find cluster-specific gene progression across condition_____no_output_____First, get the conditions as a continuous value (0, 1, 2) to use in the model. We also want to set up a partition object, so we can test each cluster. _____no_output_____

data.obs["condition_cont"] = [["Normal", "preT1D", "T1D"].index(x) for x in data.obs["detailed_cond"]]_____no_output_____part = de.test.partition(
    data=data,
    parts="cluster"
)_____no_output_____

Now run the model.

For each cluster, look at the DE when looking at the continouous variable, `condition_cont` which is a 3 stage progression into _____no_output_____

part_det = part.wald(
    formula_loc="~ 1 + condition_cont",
    factor_loc_totest="condition_cont",
    as_numeric="condition_cont",
    size_factors=data.obs["normalization_factor"]
)/usr/local/lib/python3.7/site-packages/dask/core.py:121: RuntimeWarning: divide by zero encountered in log
  return func(*(_execute_task(a, cache) for a in args))
part_det.partitions_____no_output_____part_det.tests[1].summary().sort_values(["log2fc", "pval"], ascending=[False, True])_____no_output_____help(part_det.tests[1])Help on DifferentialExpressionTestWald in module diffxpy.testing.det object:

class DifferentialExpressionTestWald(_DifferentialExpressionTestSingle)
 |  DifferentialExpressionTestWald(model_estim: batchglm.models.base.estimator.EstimatorBaseTyping, col_indices: numpy.ndarray, noise_model: str, sample_description: pandas.core.frame.DataFrame)
 |
 |  Single wald test per gene.
 |
 |  Method resolution order:
 |      DifferentialExpressionTestWald
 |      _DifferentialExpressionTestSingle
 |      _DifferentialExpressionTest
 |      builtins.object
 |
 |  Methods defined here:
 |
 |  __init__(self, model_estim: batchglm.models.base.estimator.EstimatorBaseTyping, col_indices: numpy.ndarray, noise_model: str, sample_description: pandas.core.frame.DataFrame)
 |      :param model_estim:
 |      :param col_indices: indices of coefs to test
 |
 |  log_fold_change(self, base=2.718281828459045, **kwargs)
 |      Returns one fold change per gene
 |
 |      Returns coefficient if only one coefficient is testeed.
 |      Returns the coefficient that is the maximum absolute coefficient if multiple coefficients are tested.
 |
 |  plot_comparison_ols_coef(self, size=20, show: bool = True, save: Union[str, NoneType] = None, suffix: str = '_ols_comparison_coef.png', ncols=3, row_gap=0.3, col_gap=0.25, return_axs: bool = False)
 |      Plot location model coefficients of inferred model against those obtained from an OLS model.
 |
 |      Red line shown is the identity line.
 |      Note that this comparison only seems to be useful if the covariates are zero centred. This is
 |      especially important for continuous covariates.
 |
 |      :param size: Size of points.
 |      :param show: Whether (if save is not None) and where (save indicates dir and file stem) to display plot.
 |      :param save: Path+file name stem to save plots to.
 |          File will be save+suffix. Does not save if save is None.
 |      :param suffix: Suffix for file name to save plot to. Also use this to set the file type.
 |      :param ncols: Number of columns in plot grid if multiple genes are plotted.
 |      :param row_gap: Vertical gap between panel rows relative to panel height.
 |      :param col_gap: Horizontal gap between panel columns relative to panel width.
 |      :param return_axs: Whether to return axis objects.
 |
 |      :return: Matplotlib axis objects.
 |
 |  plot_comparison_ols_pred(self, size=20, log1p_transform: bool = True, show: bool = True, save: Union[str, NoneType] = None, suffix: str = '_ols_comparison_pred.png', row_gap=0.3, col_gap=0.25, return_axs: bool = False)
 |      Compare location model prediction of inferred model with one obtained from an OLS model.
 |
 |      Red line shown is the identity line.
 |
 |      :param size: Size of points.
 |      :param log1p_transform: Whether to log1p transform the data.
 |      :param show: Whether (if save is not None) and where (save indicates dir and file stem) to display plot.
 |      :param save: Path+file name stem to save plots to.
 |          File will be save+suffix. Does not save if save is None.
 |      :param suffix: Suffix for file name to save plot to. Also use this to set the file type.
 |      :param row_gap: Vertical gap between panel rows relative to panel height.
 |      :param col_gap: Horizontal gap between panel columns relative to panel width.
 |      :param return_axs: Whether to return axis objects.
 |
 |      :return: Matplotlib axis objects.
 |
 |  plot_gene_fits_boxplots(self, gene_names: Tuple, covariate_x: str = None, covariate_hue: str = None, log1p_transform: bool = False, incl_fits: bool = True, show: bool = True, save: Union[str, NoneType] = None, suffix: str = '_genes_boxplot.png', ncols=3, row_gap=0.3, col_gap=0.25, xtick_rotation=0, legend: bool = True, return_axs: bool = False, **kwargs)
 |      Plot gene-wise model fits and observed distribution by covariates.
 |
 |      Use this to inspect fitting performance on individual genes.
 |
 |      :param gene_names: Genes to generate plots for.
 |      :param covariate_x: Covariate in location model to partition x-axis by.
 |      :param covariate_hue: Covariate in location model to stack boxplots by.
 |      :param log1p_transform: Whether to log transform observations
 |          before estimating the distribution with boxplot. Model estimates are adjusted accordingly.
 |      :param incl_fits: Whether to include fits in plot.
 |      :param show: Whether (if save is not None) and where (save indicates dir and file stem) to display plot.
 |      :param save: Path+file name stem to save plots to.
 |          File will be save+suffix. Does not save if save is None.
 |      :param suffix: Suffix for file name to save plot to. Also use this to set the file type.
 |      :param ncols: Number of columns in plot grid if multiple genes are plotted.
 |      :param row_gap: Vertical gap between panel rows relative to panel height.
 |      :param col_gap: Horizontal gap between panel columns relative to panel width.
 |      :param xtick_rotation: Angle to rotate x-ticks by.
 |      :param legend: Whether to show legend.
 |      :param return_axs: Whether to return axis objects.
 |
 |      :return: Matplotlib axis objects.
 |
 |  plot_gene_fits_violins(self, gene_names: Tuple, covariate_x: str = None, log1p_transform: bool = False, show: bool = True, save: Union[str, NoneType] = None, suffix: str = '_genes_violin.png', ncols=3, row_gap=0.3, col_gap=0.25, xtick_rotation=0, return_axs: bool = False, **kwargs)
 |      Plot gene-wise model fits and observed distribution by covariates as violins.
 |
 |      Use this to inspect fitting performance on individual genes.
 |
 |      :param gene_names: Genes to generate plots for.
 |      :param covariate_x: Covariate in location model to partition x-axis by.
 |      :param log1p_transform: Whether to log transform observations
 |          before estimating the distribution with boxplot. Model estimates are adjusted accordingly.
 |      :param show: Whether (if save is not None) and where (save indicates dir and file stem) to display plot.
 |      :param save: Path+file name stem to save plots to.
 |          File will be save+suffix. Does not save if save is None.
 |      :param suffix: Suffix for file name to save plot to. Also use this to set the file type.
 |      :param ncols: Number of columns in plot grid if multiple genes are plotted.
 |      :param row_gap: Vertical gap between panel rows relative to panel height.
 |      :param col_gap: Horizontal gap between panel columns relative to panel width.
 |      :param xtick_rotation: Angle to rotate x-ticks by.
 |      :param return_axs: Whether to return axis objects.
 |
 |      :return: Matplotlib axis objects.
 |
 |  plot_vs_ttest(self, log10=False, show: bool = True, save: Union[str, NoneType] = None, suffix: str = '_plot_vs_ttest.png', return_axs: bool = False)
 |      Normalizes data by size factors if any were used in model.
 |
 |      :param log10:
 |      :param show: Whether (if save is not None) and where (save indicates dir and file stem) to display plot.
 |      :param save: Path+file name stem to save plots to.
 |          File will be save+suffix. Does not save if save is None.
 |      :param suffix: Suffix for file name to save plot to. Also use this to set the file type.
 |      :param return_axs: Whether to return axis objects.
 |
 |      :return:
 |
 |  summary(self, qval_thres=None, fc_upper_thres=None, fc_lower_thres=None, mean_thres=None, **kwargs) -> pandas.core.frame.DataFrame
 |      Summarize differential expression results into an output table.
 |
 |      :param qval_thres: Upper bound of corrected p-values for gene to be included.
 |      :param fc_upper_thres: Upper bound of fold-change for gene to be included.
 |      :param fc_lower_thres: Lower bound of fold-change p-values for gene to be included.
 |      :param mean_thres: Lower bound of average expression for gene to be included.
 |      :return: Summary table of differential expression test.
 |
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |
 |  gene_ids
 |
 |  model_gradient
 |
 |  x
 |
 |  ----------------------------------------------------------------------
 |  Data and other attributes defined here:
 |
 |  __abstractmethods__ = frozenset()
 |
 |  __annotations__ = {'_error_codes': <class 'numpy.ndarray'>, '_niter': ...
 |
 |  ----------------------------------------------------------------------
 |  Methods inherited from _DifferentialExpressionTest:
 |
 |  log10_fold_change(self, **kwargs)
 |      Calculates the log_10 fold change(s) for this DifferentialExpressionTest.
 |
 |  log10_pval_clean(self, log10_threshold=-30)
 |      Return log10 transformed and cleaned p-values.
 |
 |      NaN p-values are set to one and p-values below log10_threshold
 |      in log10 space are set to log10_threshold.
 |
 |      :param log10_threshold: minimal log10 p-value to return.
 |      :return: Cleaned log10 transformed p-values.
 |
 |  log10_qval_clean(self, log10_threshold=-30)
 |      Return log10 transformed and cleaned q-values.
 |
 |      NaN p-values are set to one and q-values below log10_threshold
 |      in log10 space are set to log10_threshold.
 |
 |      :param log10_threshold: minimal log10 q-value to return.
 |      :return: Cleaned log10 transformed q-values.
 |
 |  log2_fold_change(self, **kwargs)
 |      Calculates the pairwise log_2 fold change(s) for this DifferentialExpressionTest.
 |
 |  plot_ma(self, corrected_pval=True, log2_fc_threshold=10, min_mean=0.0001, alpha=0.05, size=20, highlight_ids: Union[List, Tuple] = (), highlight_size: float = 30, highlight_col: str = 'red', show: bool = True, save: Union[str, NoneType] = None, suffix: str = '_ma_plot.png', return_axs: bool = False)
 |      Returns an MA plot of mean expression vs. log fold change with significance
 |      super-imposed.
 |
 |      :param corrected_pval: Whether to use multiple testing corrected
 |          or raw p-values.
 |      :param log2_fc_threshold: Negative lower and upper bound of
 |          log2 fold change displayed in plot.
 |      :param min_mean:
 |          Lower bound for mean expression of plot. All values below this threshold
 |          are updated to this threshold.
 |      :param alpha: p/q-value lower bound at which a test is considered
 |          non-significant. The corresponding points are colored in grey.
 |      :param size: Size of points.
 |      :param highlight_ids: Genes to highlight in volcano plot.
 |      :param highlight_size: Size of points of genes to highlight in volcano plot.
 |      :param highlight_col: Color of points of genes to highlight in volcano plot.
 |      :param show: Whether (if save is not None) and where (save indicates dir and file stem) to display plot.
 |      :param save: Path+file name stem to save plots to.
 |          File will be save+suffix. Does not save if save is None.
 |      :param suffix: Suffix for file name to save plot to. Also use this to set the file type.
 |      :param return_axs: Whether to return axis objects.
 |
 |      :return: Tuple of matplotlib (figure, axis)
 |
 |  plot_volcano(self, corrected_pval=True, log10_p_threshold=-30, log2_fc_threshold=10, alpha=0.05, min_fc=1, size=20, highlight_ids: Union[List, Tuple] = (), highlight_size: float = 30, highlight_col: str = 'red', show: bool = True, save: Union[str, NoneType] = None, suffix: str = '_volcano.png', return_axs: bool = False)
 |      Returns a volcano plot of p-value vs. log fold change
 |
 |      :param corrected_pval: Whether to use multiple testing corrected
 |          or raw p-values.
 |      :param log10_p_threshold: lower bound of log10 p-values displayed in plot.
 |      :param log2_fc_threshold: Negative lower and upper bound of
 |          log2 fold change displayed in plot.
 |      :param alpha: p/q-value lower bound at which a test is considered
 |          non-significant. The corresponding points are colored in grey.
 |      :param min_fc: Fold-change lower bound for visualization,
 |          the points below the threshold are colored in grey.
 |      :param size: Size of points.
 |      :param highlight_ids: Genes to highlight in volcano plot.
 |      :param highlight_size: Size of points of genes to highlight in volcano plot.
 |      :param highlight_col: Color of points of genes to highlight in volcano plot.
 |      :param show: Whether (if save is not None) and where (save indicates dir and file stem) to display plot.
 |      :param save: Path+file name stem to save plots to.
 |          File will be save+suffix. Does not save if save is None.
 |      :param suffix: Suffix for file name to save plot to. Also use this to set the file type.
 |      :param return_axs: Whether to return axis objects.
 |
 |      :return: Tuple of matplotlib (figure, axis)
 |
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from _DifferentialExpressionTest:
 |
 |  __dict__
 |      dictionary for instance variables (if defined)
 |
 |  __weakref__
 |      list of weak references to the object (if defined)
 |
 |  log_likelihood
 |
 |  mean
 |
 |  pval
 |
 |  qval

# part_det.tests[7].plot_volcano(corrected_pval=True, min_fc=1.05, alpha=0.05, size=20)
part_det.tests[1].plot_ma()_____no_output_____


# Notebook from indirajith/2019-camerino-advanced-numpy
Path: Advanced NumPy Patterns.ipynb

# About me

Juan Nunez-Iglesias
CZI Imaging Software Fellow
Monash University  _____no_output_____http://elegant-scipy.org

_____no_output_____# Quick example: gene expression, without numpy_____no_output_____| Gene   | Cell type A | Cell type B | Cell type C | Cell type D |
|--------|-------------|-------------|-------------|-------------|
| Gene 0 | 100         | 200         | 50          | 400         |
| Gene 1 | 50          | 0           | 0           | 100         |
| Gene 2 | 350         | 100         | 50          | 200         |_____no_output_____

gene0 = [100, 200, 50, 400]
gene1 = [50, 0, 0, 100]
gene2 = [350, 100, 50, 200]
expression_data = [gene0, gene1, gene2]_____no_output_____

Why is this a bad idea?_____no_output_____# Now with NumPy_____no_output_____

import numpy as np
a = np.array(expression_data)
print(a)_____no_output_____

We are going to:

* Obtain an *RPKM* expression matrix
* Quantile normalize the data

using the awesome power of NumPy_____no_output_____# Inside a numpy ndarray_____no_output_____

def print_info(a):
    print('number of elements:', a.size)
    print('number of dimensions:', a.ndim)
    print('shape:', a.shape)
    print('data type:', a.dtype)
    print('strides:', a.strides)
    print('flags:')
    print(a.flags)

print_info(a)_____no_output_____print(a.data)_____no_output_____abytes = a.ravel().view(dtype=np.uint8)_____no_output_____print_info(abytes)_____no_output_____print(abytes[:24])_____no_output_____

### Example: take the transpose of `a`_____no_output_____

print_info(a)_____no_output_____print_info(a.T)_____no_output_____

### Example: skipping rows and columns with *slicing*_____no_output_____

print_info(a.T)_____no_output_____print_info(a.T[::2])_____no_output_____print_info(a.T[::2, ::2])_____no_output_____

### Getting a copy_____no_output_____

b = a_____no_output_____print(b)_____no_output_____a[0, 0] = 5
print(b)
a[0, 0] = 100_____no_output_____

# Advanced operations: axis-wise evaluation_____no_output_____

expr = np.load('expr.npy')_____no_output_____print_info(expr)_____no_output_____

This has the raw read count data. However, each sample gets a different number of reads, so we want to normalize by the *library size*, which is the total number of reads across a column.

The `np.sum` function returns the sum of all the elements of an array. With the `axis` argument, you can take the sum *along the given axis*._____no_output_____

lib_size = np.sum(expr, axis=0)_____no_output_____

### Exercise

Generate a 10 x 3 array of random numbers. From each row, pick the column containing the number closest to 0.75.

*Hint:* use of `np.abs` and `np.argmin` to find the column j that contains the closest element in each row i. The final result should be a vector of integers of shape (10,)._____no_output_____### Exercise

Some applications, such as clustering, are computationally expensive, and wouldn't work without first doing some form of *feature selection*, where we discard most of the data and keep only what we think will be most useful. One simple version is to keep only the genes with the most variance (as these will be more informative than genes that don't vary between patients).

- Find the variance across patients of all the genes (rows) in the expression dataset.
- Use `np.argsort` to find the location of the 1,500 most variable genes.
- Use these indices to produce a shape (1500, 375) matrix containing only the most variable genes._____no_output_____# Advanced operations: broadcasting_____no_output_____In order to normalize every column by its corresponding library size, we have to *align* the two arrays' axes: each dimension must be either the same size, or one of the arrays must have size 1. Use `np.newaxis` to match the dimensions. But let's first do some simple examples:_____no_output_____

a + 5  # simplest "broadcasting": scalar - array operations_____no_output_____b = np.array([1, 2, 3, 4])
a + b  # broadcasting: coerce arrays to same shape by repeating as needed_____no_output_____b = np.array([1, 2, 3])
a + b  # broadcasting: not just magic!_____no_output_____b = np.array([[1], [2], [3]])
a + b  # broadcasting: shape compatibility_____no_output_____

Now, back to our expression data._____no_output_____

print(expr.shape)
print(lib_size.shape)
print(lib_size[np.newaxis, :].shape)_____no_output_____

However, NumPy will automatically prepend singleton dimensions until the array shapes match or there is an error:_____no_output_____

np.all(expr / lib_size ==
       expr / lib_size[np.newaxis, :])_____no_output_____expr_lib = expr / lib_size_____no_output_____

We also multiply by $10^6$ in order to keep the numbers on a readable scale (reads per million reads)._____no_output_____

expr_lib *= 1e6_____no_output_____

Finally, longer genes are more likely to produce reads. So we must normalize by the gene length (in kb) to produce a measure of expression called Reads Per Kilobase per Million reads (RPKM). We start by loading the gene lengths in *bases*. (1 kilobase = 1,000 bases.)_____no_output_____

gene_len = np.load('gene-lens.npy')
print(gene_len.shape)_____no_output_____

### Exercise: broadcast `expr`, `lib_size` and `gene_len` together to produce RPKM_____no_output_____Reminder:

$RPKM_{i, j} = \frac{C_{i, j}}{N_j \times 10^{-6} \times L_i \times 10^{-3}} = \frac{10^9C_{i, j}}{N_jL_i}$

where $C$ is the raw counts (`expr`), $N$ is the library size (in reads) (`lib_size`) and $L$ is the gene length (in bases) (`gene_len`). _____no_output_____

rpkm = ...  # FIX THIS_____no_output_____

In order to admire our handywork, we will use a custom plotting function:_____no_output_____

from matplotlib import pyplot as plt
from scipy import stats

def plot_col_density(data, xlim=None, ax=None, *args, **kwargs):
    """Use gaussian smoothing to estimate the density.

    Parameters
    ----------
    data : 2D array
        The data to be plotted. Each column will result in a
        different KDE line.
    xlim : tuple of int, optional
        The range of the x-axis. Useful when the output has a
        few outlier values, squishing the bulk of the
        distribution to a small range.
    ax : matplotlib Axes object, optional
        The axes on which to plot. If None, new axes will be
        created.
    *args, **kwargs
        Extra arguments to ax.plot.

    Returns
    -------
    ax : matplotlib Axes object
        The axes on which the plot was made.
    """
    density_per_col = [stats.kde.gaussian_kde(col) for col in data.T]
    if xlim is not None:
        m, M = xlim
    else:
        m, M = np.min(data), np.max(data)
    x = np.linspace(m, M, 100)

    if ax is None:
        fig, ax = plt.subplots()
    for density in density_per_col:
        ax.plot(x, density(x), *args, **kwargs)
    ax.set_xlabel('log-counts')
    ax.set_ylabel('frequency')
    if xlim is not None:
        ax.set_xlim(xlim)
    return ax
_____no_output_____%matplotlib inline_____no_output_____plot_col_density(np.log(expr+1));_____no_output_____plot_col_density(np.log(rpkm + 1), xlim=(0, 6));_____no_output_____

You should see that the most "disparate" data column is now a much better fit with the rest of the data. This should improve downstream scientific analysis._____no_output_____### Exercise: 3D broadcasting

Below, using broadcasting, produce the array containing the sum of every element in `x` with every element in `y`. That is, produce an array `z` such that `z[i, j, k]` contains either the sum of `x[i, j]` and `y[k]` OR the sum of `y[i]` and `x[j, k]`._____no_output_____

x = np.random.random(size=(3, 5))
y = np.random.randint(10, size=8)
z = ... # FIX THIS_____no_output_____

### Exercise: explicit broadcasting and stride tricks

Now, use `np.broadcast_arrays` to generate `xbroad` and `ybroad` that are the same shape as `z` (so that a simple element-wise addition will give `z`). Then use `print_info` on `xbroad` and `ybroad`. Notice anything weird?_____no_output_____## Stride tricks

By manipulating the shape and strides of an array, we can produce a "virtual" array much bigger than its memory usage:_____no_output_____

def repeat(arr, n):
    return np.lib.stride_tricks.as_strided(arr,
                                           shape=(n,) + arr.shape,
                                           strides=(0,) + arr.strides)_____no_output_____repeated_row = repeat(np.random.random(size=5), 4)
repeated_row_____no_output_____

Be careful, though: some operations, such as `np.copy`, actually materialize the much bigger array!_____no_output_____

print_info(repeated_row)_____no_output_____print_info(np.copy(repeated_row))_____no_output_____

### Exercise_____no_output_____

x = np.random.random((3, 2)).astype(np.float32)_____no_output_____

Try to answer these without looking at `x`. Then, try them out with the `print_info` function.

- What is the shape of `x`?
- What are the strides of `x`?
- Is `x` C-contiguous, F-contiguous, or neither?

Now let `y = repeat(x, 4)`. What is its shape? What are its strides? Is it contiguous?_____no_output_____### Exercise: `np.lib.stride_tricks.as_strided`

Use `as_strided` to produce a sliding-window view of a 1D array._____no_output_____

def sliding_window(arr, size=2):
    """Produce an array of sliding window views of `arr`

    Parameters
    ----------
    arr : 1D array, shape (N,)
        The input array.
    size : int, optional
        The size of the sliding window.

    Returns
    -------
    arr_slide : 2D array, shape (N - size + 1, size)
        The sliding windows of size `size` of `arr`.

    Examples
    --------
    >>> a = np.array([0, 1, 2, 3])
    >>> sliding_window(a, 2)
    array([[0, 1],
           [1, 2],
           [2, 3]])
    """
    return ...  # fix this_____no_output_____# test your code here
sliding_window(np.arange(8), 3)_____no_output_____

### Exercise: mean filtering

Use `sliding_window` to implement mean filtering, in which every value in an array is replaced by the mean of it and its neighbours. This is a basic operation in signal processing._____no_output_____

def mean_filter(signal, window_size=3):
    """Apply a mean filter to the input with the desired window size.

    Parameters
    ----------
    signal : 1D array, shape (M,)
        The input signal.
    window_size : int, optional
        The size of the window along which to compute the mean.

    Returns
    -------
    filtered : 1D array, shape (M - window_size + 1,)
        The filtered signal.
    """
    return ...  # FIX THIS_____no_output_____

To test your function, we will use the example of a *difference filter*, which finds the location of changes in a signal using *convolution*. When the signal is perfectly noiseless, it works great:_____no_output_____

signal = np.zeros(100, np.float)
signal[30:60] = 1

diff = np.array([1, 0, -1])
from scipy import ndimage as ndi
dsignal = ndi.convolve(signal, diff)_____no_output_____fig, ax = plt.subplots(1, 2)
ax[0].plot(signal)
ax[0].set_title('signal')
ax[1].plot(dsignal)
ax[1].set_title('change')
fig.tight_layout()_____no_output_____

However, if the signal is corrupted by noise, a standard difference filter convolution doesn't work:_____no_output_____

np.random.seed(0)
signal_noisy = signal + np.random.normal(0, 0.3, size=signal.shape)
dsignal_noisy = ndi.convolve(signal_noisy, diff)

fig, ax = plt.subplots(1, 2)
ax[0].plot(signal_noisy)
ax[0].set_title('signal')
ax[1].plot(dsignal_noisy)
ax[1].set_title('change')
fig.tight_layout()_____no_output_____

### Exercise

Try mean filtering with different window sizes to see whether the change signal becomes more apparent._____no_output_____### Exercise: padding

What is the shape of your mean-filtered signal?

...

Oops! We've shortened the signal, which means that our indices have changed: `signal_filtered[i]` does not correspond to the signal around `signal[i]`.

Use `np.pad` to add some "fake" data around `signal` before filtering, so that the filtered result has the same shape as the input._____no_output_____### Exercise: Gaussian filtering

It turns out that mean filtering is not the "optimal" way to recover your signal, assuming certain properties of the noise. For that, we use *Gaussian* filtering, which uses a *weighted* mean of the sliding window elements. The weights are given by the famous Gaussian bell-shaped distribution. For example, here are the weights for a window size of 17 for a particular sigma:_____no_output_____

weight = np.exp(-(np.arange(-8, 9) / (8/3))**2)
weight /= np.sum(weight)  ## ensure overall intensity of signal doesn't change
fig, ax = plt.subplots()
ax.plot(weight)_____no_output_____

Write a function that uses sliding windows, broadcasting, and axis-wise operations to compute the Gaussian filter of a signal for a given window size. (You should also pad your input.)_____no_output_____# Fancy indexing_____no_output_____You can index arrays with slicing, but also with boolean arrays (including broadcasting!), integer arrays, and individual indices along multiple dimensions._____no_output_____

values = np.array([0, 5, 99])
selector = np.random.randint(0, 3, size=(3, 4))
print(selector)
print(values[selector])_____no_output_____relabeled = values[selector]
has_large_cols = np.any(relabeled > 10, axis=1)
print(relabeled[has_large_cols])_____no_output_____

### Exercise

Use boolean indexing and broadcasting to select the columns of `relabeled` that do not contain 99._____no_output_____### Example: quantile normalization

Quantile Normalization (https://en.wikipedia.org/wiki/Quantile_normalization) is a method to align distributions. Here we implement it using NumPy axis-wise operations and fancy indexing._____no_output_____

def qnorm(X):
    """Quantile normalize an input matrix.

    Parameters
    ----------
    X : 2D array of float, shape (M, N)
        The input data, with each column being a
        distribution to normalize.

    Returns
    -------
    Xn : 2D array of float, shape (M, N)
        The normalized data.
    """
    ranks =
    return Xn_____no_output_____logexpr = np.log(expr + 1)
logrpkm = np.log(rpkm + 1)_____no_output_____logexprn = qnorm(logexpr)
logrpkmn = qnorm(logrpkm)_____no_output_____plot_col_density(logexprn)_____no_output_____plot_col_density(logrpkmn, xlim=(0, 0.25))_____no_output_____

## Fancy indexing along multiple dimensions

Combining fancy indexing and slicing selects entire rows/columns:_____no_output_____

relabeled_____no_output_____relabeled[[1, 1, 2], :]_____no_output_____relabeled[:, [1, 3, 1]]_____no_output_____

To select individual elements for a new array shape, we must use as many fancy indices as the array has dimensions:_____no_output_____

selector_rows = [[0, 0],
                 [1, 2]]
selector_cols = [[0, 3],
                 [1, 2]]

arr = np.arange(12).reshape((3, 4))
print(arr)_____no_output_____print(arr[selector_rows, selector_cols])_____no_output_____

One way to think about this is:
- make a "coordinate array", of the shape that you want plus one more axis, to hold the coordinates of each point (see below),
- transpose that final axis to the front, and
- convert to tuple

For the above example, perhaps you find this "notation", with the individual coordinates in the final axis, more intuitive:_____no_output_____

selector_t = [[ [0, 0], [0, 3] ],
              [ [1, 1], [2, 2] ]] _____no_output_____

That is, we want element (0, 0) in the top left corner, (0, 3) in the top right, (1, 1) in the bottom left, and (2, 3) in the bottom right. However, *rows and columns* must be in the first dimension and presented as a tuple to index the original array. So, to use this notation, we can use np.transpose and cast the result to `tuple`._____no_output_____

selector = tuple(np.transpose(selector_t, (2, 0, 1)))
print(selector[0], selector[1], sep='\n')_____no_output_____

For technical reasons that one might grasp for fleeting moments, the "tuple of index arrays" format is most consistent with other forms of multi-dimensional indexing in NumPy. It is a widespread convention (see e.g. `scipy.ndimage.map_coordinates`), so it's worth practicing._____no_output_____### Exercise

What happens when you make `selector_col`:
- a single number?
- a 1D array with two elements?
- a 2D array of shape (1, 2)?
- a 1D array with three elements?

Repeat similar experiments with `selector_row`.

Does this remind you of any other NumPy feature we may have seen?_____no_output_____## Advanced exercise: Jack's dilemma

(If time permits.)

```email
Date: Wed, 16 Jul 2008 16:45:37 -0500
From: Jack Cook
To: <numpy-discussion@scipy.org>
Subject: Numpy Advanced Indexing Question

Greetings,

I have an I,J,K 3D volume of amplitude values at regularly sampled
time intervals. I have an I,J 2D slice which contains a time (K)
value at each I, J location. What I would like to do is extract a
subvolume at a constant +/- K window around the slice. Is there an
easy way to do this using advanced indexing or some other method?
Thanks in advanced for your help.

-- Jack
```_____no_output_____

# In the command-line, run `python tools/generate-volume.py`

data = np.load('geo.npz')  # npz file with multiple arrays, keyed like dict
strata, density = data['strata'], data['density']_____no_output_____row = 256

fig, (ax0, ax1) = plt.subplots(1, 2, sharey=True, figsize=(7.2, 3.6))
ax0.imshow(strata[:, row, :], cmap='gray')
ax0.set_ylabel('depth')
ax0.set_xlabel('x')
ax0.set_title(f'strata at row {row}')
ax1.imshow(density[:, 256, :], cmap='magma')
ax1.set_xlabel('x')
ax1.set_title(f'fossil density at row {row}') _____no_output_____

### Exercise

We want to quantify the apparent extinction event near a particular stratum. The strata, however, are not perfectly horizontal, and there is a large break in the rock, too. Therefore, we want to align the strata by using the solution to Jack's dilemma (which you must solve for him).

Using a window size of 120 (so the half-width is K=60):
- find the stratum of maximum intensity along the depth axis. This is a 2D slice of integers measuring the depth of the maximum intensity stratum at each (row, column) coordinate.
- using broadcasting, create a volume of shape (2K, Nrow, Ncol) containing the depth coordinate at each (row, column) of your desired window.
- create matching row and column index volumes to perform fancy indexing of the fossil density array.
- extract the subvolume of fossil density around the stratum using fancy indexing
- compute the mean fossil density at each depth.

**-->Advice<--**: play around with a much tinier subset of the volume, say, a subset of size (10, 15, 20), and window size of 3. Evaluating the wrong expression can result in a giant dataset that blows up your memory and crashes your computer. (Yay Science!)_____no_output_____## Linear algebra with NumPy_____no_output_____Since version 3.5, Python supports the matrix multiplication operator, denoted by the `@` symbol. This makes it a new powerhouse for linear algebra._____no_output_____

M = np.array([[0, 1],
              [1, 1],
              [1, 0]])
v = np.array([9, 2])

print(M @ v)_____no_output_____print(M.T @ M)_____no_output_____

Now consider the rotation matrix

$
R = \begin{bmatrix}
  \cos \theta &  -\sin \theta & 0 \\
  \sin \theta & \cos \theta & 0\\
  0 & 0 & 1\\
\end{bmatrix}
$

When $R$ is multiplied with a 3-dimensional column-vector $p =
\left[ x\, y\, z \right]^T$, the resulting vector $R p$ is rotated
by $\theta$ degrees around the z-axis._____no_output_____

theta_deg = 45
theta = np.deg2rad(theta_deg)

c = np.cos(theta)
s = np.sin(theta)

R = np.array([[c, -s, 0],
              [s,  c, 0],
              [0,  0, 1]]) _____no_output_____x_axis = np.array([1, 0, 0]) _____no_output_____R @ x_axis_____no_output_____rotated = R @ x_axis
R @ rotated_____no_output_____

### Exercise: rotation matrices

If you know some linear algebra, try to answer the following questions about $R$ in your head, before trying them out in Python. If you're totally stuck, just give them a go!

- What does the matrix $S = RR$ do? (Remember, `S = R @ R` in code.)
- What does $R^4 = RRRR$ do?
- The inverse $R^{-1}$ of $R$ is defined as the matrix such that $R^{-1}R = I$, where $I$ is the identity matrix. What does $Q = R^{-1}$ do? (Note: NumPy provides matrix inverse computation in `np.linalg.inv`.)
- What does $R$ do to the vector [0, 0, 1]?
- What does that make the vector [0, 0, 1]?
- Verify this with some function in `np.linalg`. (Look at the online documentation for this module.)
- What does $R^8$ do? What does that make $R^8$?_____no_output_____### Exercise: revisiting Gaussian filtering

Above, we performed Gaussian filtering with a broadcast product and a axis-wise sum. Use a matrix multiplication instead._____no_output_____## Neural network with NumPy_____no_output_____Inspired by https://iamtrask.github.io/2015/07/12/basic-python-network/_____no_output_____

n = 20_____no_output_____np.random.seed(128) _____no_output_____X = np.random.random(size=(n, 3)) _____no_output_____

- Make a vector y of length n that is 0 at position i if `X[i, 0] + X[i, 2] < 1`, and 1 otherwise._____no_output_____*Hint:* How do I get the 0th column of X?

*Hint 2:* How can I convert an array of (True/False) to an array of float?_____no_output_____

X[2, 1]_____no_output_____y = (X[:, 0] + X[:, 2] > 1).astype(float) _____no_output_____plt.scatter(X[:, 2], X[:, 0], color=plt.cm.viridis(y)) _____no_output_____X.shape_____no_output_____X.size_____no_output_____Xb = np.column_stack((X, np.ones(X.shape[0])))_____no_output_____W = 2 * np.random.random(size=Xb.shape[1]) - 1_____no_output_____def sigmoid(x):
    return 1 / (1 + np.exp(-x))_____no_output_____pred = sigmoid(Xb @ W)_____no_output_____pred_____no_output_____fig, ax = plt.subplots()
ax.scatter(y, pred)_____no_output_____def dsigmoid(x):
    return x * (1 - x)_____no_output_____def train(X, y, W, n_iter=100):
    W_history = np.empty((n_iter, W.size), dtype=float)
    error_history = np.empty(n_iter, dtype=float)
    W_out = np.copy(W)
    for i in range(n_iter):
        W_history[i] = W_out
        predictions = sigmoid(X @ W_out)
        derror = y - predictions
        gradient = derror * dsigmoid(predictions)
        update = X.T @ gradient
        error_history[i] = np.sum(np.abs(error))
        W_out += update
    return W_out, W_history, error_history_____no_output_____Wout, Whist, errhist = train(Xb, y, W, 10000)_____no_output_____fig, ax = plt.subplots()
ax.semilogx(errhist)_____no_output_____np.mgrid[0:1:0.3, 0:1:0.3] _____no_output_____decision_space = np.mgrid[0:1:0.01, 0:1:1, 0:1:0.01, 1:2:1] _____no_output_____decision_space.shape_____no_output_____prediction_image = sigmoid(decision_space.T @ Wout).T_____no_output_____prediction_image.shape_____no_output_____prediction_squeezed = np.squeeze(prediction_image) _____no_output_____fig, ax = plt.subplots()
ax.imshow(prediction_squeezed, cmap='magma', origin='lower',
          extent=[0, 1, 0, 1])
ax.scatter(X[:, 2], X[:, 0], color=plt.cm.viridis(y)) _____no_output_____

### Exercise

Why do some points appear misclassified?

*Hint:* Look at `Wout`._____no_output_____# References and sources

In addition to original content, these notes use materials from the following sources:

- [100 NumPy Exercises](http://www.labri.fr/perso/nrougier/teaching/numpy.100/), compiled by Nicolas Rougier, used here under the terms of the MIT License.
- [SciPy Lecture Notes](http://www.scipy-lectures.org/), compiled by Gaël Varoquaux, Emmanuelle Gouillart, and Olav Vahtras, used under the terms of the CC-BY license.
- [Elegant SciPy](http://elegant-scipy.org), written by Juan Nunez-Iglesias, Stéfan van der Walt, and Harriet Dashnow, used with permission from O'Reilly.
- [iamtrask blog](https://iamtrask.github.io/2015/07/12/basic-python-network/), for a neural network from scratch that inspired this one.

They are all freely available online and worth checking out in their entirety!_____no_output_____