# Notebook from henryeahhh/scanpy_template
Path: notebooks/ML-2020-11-04_sample_notebook.ipynb

Title of the Analysis
----
Short description of what is being done here. _____no_output_____# Preliminaries_____no_output_____## Dependecy notebooks_____no_output_____TODO: define here links to other notebooks which need to be run in the correct order prior
to running this notebook (e.g. data preprocessing/data generation).
If there aren't any, remove this section._____no_output_____## Import packages_____no_output_____If you want modules to be automatically reloaded when you call them, use `autoreload`:_____no_output_____

%load_ext autoreload
%autoreload 2_____no_output_____

Import packages_____no_output_____

# import standard packages
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns

import os
import sys

# import single-cell packages
import scanpy as sc
import scanpy.external as sce
import scvelo as scv
import cellrank as cr

# set verbosity levels
sc.settings.verbosity = 2
cr.settings.verbosity = 2
scv.settings.verbosity = 3 _____no_output_____

## Print package versions for reproducibility_____no_output_____If you want to exactly reproduce the results shown here, please make sure that your package versions match what is printed below. _____no_output_____

cr.logging.print_versions()cellrank==1.0.0 scanpy==1.6.0 anndata==0.7.4 numpy==1.19.2 numba==0.51.2 scipy==1.5.3 pandas==1.1.4 scikit-learn==0.23.2 statsmodels==0.12.1 scvelo==0.2.2 pygam==0.8.0 matplotlib==3.3.2 seaborn==0.11.0
cr.logging.print_version_and_date()Running CellRank 1.0.0, on 2020-11-04 17:44.

## Set up paths_____no_output_____Define the paths to load data, cache results and write figure panels._____no_output_____

sys.path.insert(0, "../")  # this depends on the notebook depth and must be adapted per notebook

from paths import DATA_DIR, CACHE_DIR, FIG_DIR_____no_output_____

Set up the paths to save figures._____no_output_____

scv.settings.figdir = str(FIG_DIR)
sc.settings.figdir = str(FIG_DIR)
cr.settings.figdir = str(FIG_DIR)_____no_output_____

## Set up caching_____no_output_____Note: we use a caching extension called `scachepy` for this analysis, see [here](https://github.com/theislab/scachepy). We do this to speed up the runtime of this notebook by avoiding the most expensive computations. Below, we check whether you have scachepy installed and if you don't, then we automatically recompute all results. _____no_output_____

try:
    import scachepy
    c = scachepy.Cache(CACHE_DIR / "sample_notebook", separate_dirs=True)
    print('Sucessfully imported scachepy. Expensive computations will be cached. ')
except ImportError:
    print('Failed to import scachepy. Consider installing it from `https://github.com/theislab/scachepy`. ')
    c = None

use_caching = c is not None
cSucessfully imported scachepy. Expensive computations will be cached.

## Set global parameters_____no_output_____Set some plotting parameters._____no_output_____

scv.settings.set_figure_params('scvelo', dpi_save=400, dpi=80, transparent=True, fontsize=20, color_map='viridis')
scv.settings.plot_prefix = ""_____no_output_____

Set other global parameters_____no_output_____

# should figures just be displayed or also saved?
save_figure = True

# should cashed values be used, or recompute?
force_recompute = False_____no_output_____

If there are other global parameters for this analysis, put them here as well. _____no_output_____## Define utility functions_____no_output_____Any utility functions you may need in this notebook go here. _____no_output_____## Load the data_____no_output_____Load the AnnData object from the CellRank software package (or from whereever you would like to load your data from)_____no_output_____

adata = cr.datasets.pancreas(DATA_DIR / "pancreas" / "pancreas.h5ad")
adata_____no_output_____

# Quality control and pre-processing_____no_output_____Do your QC and data pre-processing here. _____no_output_____## Pre-process the data_____no_output_____Often, we want to process the raw data a bit differently than the data in `.X`._____no_output_____### Raw data (for plotting)_____no_output_____

adata_raw = adata.copy()
sc.pp.filter_genes(adata_raw, min_cells=10)
scv.pp.normalize_per_cell(adata_raw)
sc.pp.log1p(adata_raw)

# annotate highly variable genes, but don't filter them out
sc.pp.highly_variable_genes(adata_raw)
print(f"This detected {np.sum(adata_raw.var['highly_variable'])} highly variable genes. ")
adata.raw = adata_rawfiltered out 15011 genes that are detected in less than 10 cells
Normalized count data: X, spliced, unspliced.
extracting highly variable genes
    finished (0:00:00)
This detected 1480 highly variable genes.

### Data for velocity computation_____no_output_____

# filter, normalise, log transform
scv.pp.filter_and_normalize(adata, min_shared_counts=20, log=True, n_top_genes=2000)

# compute pca, knn graph and scvelo's moments
sc.tl.pca(adata)
sc.pp.neighbors(adata, n_pcs=30, n_neighbors=30)
scv.pp.moments(adata, n_pcs=None, n_neighbors=None)Filtered out 22024 genes that are detected 20 counts (shared).
Normalized count data: X, spliced, unspliced.
Exctracted 2000 highly variable genes.
Logarithmized X.
computing PCA
    on highly variable genes
    with n_comps=50
    finished (0:00:00)
computing neighbors
    using 'X_pca' with n_pcs = 30
    finished (0:00:02)
computing moments based on connectivities
    finished (0:00:00) --> added
    'Ms' and 'Mu', moments of un/spliced abundances (adata.layers)

# Main analysis part 1_____no_output_____Structure your analysis by sections and subsections. For me, the first part of the analysis is often about computing embeddings and clustering the data. _____no_output_____## Visualize annotations_____no_output_____Often, we want to show UMAP plots with cluster annotation of marker genes here. This also serves as an example of figure saving. _____no_output_____

fig_kwargs = {'color': 'clusters', 'dpi': 100}
if save_figure: fig_kwargs['save'] = "sample_notebook/pancreas_umap.pdf"
scv.pl.scatter(adata, **fig_kwargs)saving figure to file /Users/marius/Projects/sc_analysis_template/figures/sample_notebook/pancreas_umap.pdf

# scVelo analysis_____no_output_____## Compute velocities using scVelo_____no_output_____

# compute/load from cache the dyn. model params and compute velocities
if use_caching:
    c.tl.recover_dynamics(adata, fname='2020-11-04_recover_dynamics', force=force_recompute)
else:
    scv.tl.recover_dynamics(adata)

scv.tl.velocity(adata, mode='dynamical')Loading data from: `2020-11-04_recover_dynamics.pickle`.
computing velocities
    finished (0:00:02) --> added
    'velocity', velocity vectors for each individual cell (adata.layers)

## Project velocities onto the embedding_____no_output_____Often, we want to show velocities in the UMAP embedding here._____no_output_____

scv.tl.velocity_graph(adata)
scv.pl.velocity_embedding_stream(adata)computing velocity graph
    finished (0:00:03) --> added
    'velocity_graph', sparse matrix with cosine correlations (adata.uns)
computing velocity embedding
    finished (0:00:00) --> added
    'velocity_umap', embedded velocity vectors (adata.obsm)

# CellRank analysis_____no_output_____Most of the analysis I do usually involve CellRank at some stage. _____no_output_____## Initialise kernel and estimator_____no_output_____This serves as an example of how to cach a transition matrix and of the default values we should be using. _____no_output_____

from cellrank.tl.estimators import GPCCA

# set the parameters we want to use to compute the transition matrix
weight_connectivities=0.2
mode="stochastic"
n_jobs=8
softmax_scale=None

if use_caching:
    c.tl.transition_matrix(adata,
                           weight_connectivities=weight_connectivities,
                           mode=mode,
                           n_jobs=n_jobs,
                           softmax_scale=softmax_scale,
                           fname='2020-11-04_transition_matrix_stochastic',
                           force=force_recompute)
    g_fwd = GPCCA(adata, obsp_key='T_fwd')
else:
    kernel = cr.tl.transition_matrix(adata,
                                     weight_connectivities=weight_connectivities,
                                     mode=mode,
                                     n_jobs=n_jobs,
                                     softmax_scale=softmax_scale)
    g_fwd = GPCCA(kernel)

print(g_fwd)Loading data from: `2020-11-04_transition_matrix_stochastic.pickle`.
GPCCA[n=2531, kernel=<Precomputed[origin='array']>]

## Print transition matrix statistics for reproducibility_____no_output_____To make sure that we're using the same transition matrix, show some stats. _____no_output_____

# just for consistency, display some summary staes about the transition matrix
T = adata.obsp['T_fwd']
T_med, T_mean, T_var = np.median(T.data), np.mean(T.data), np.var(T.data)
T_min, T_max = np.min(T.data), np.max(T.data)

print(f'med = {T_med:.6f},\nmean = {T_mean:.6f},\nvar = {T_var:.6f},\nmin = {T_min:.6f},\nmax = {T_max:.6f}')med = 0.018748,
mean = 0.024591,
var = 0.000466,
min = 0.000058,
max = 0.426351

## Compute the real Schur decomposition_____no_output_____

g_fwd.compute_schur()
g_fwd.plot_spectrum(real_only=True)Computing Schur decomposition
When computing macrostates, choose a number of states NOT in `[5, 9]`
Adding `.eigendecomposition`
       `adata.uns['eig_fwd']`
       `.schur`
       `.schur_matrix`
    Finish (0:00:00)


# Notebook from ckmah/cell2cell
Path: examples/cell2cell/Toy-Example.ipynb

import cell2cell as c2c

%matplotlib inline_____no_output_____

# Analysis Setup

Here, all inputs and parameters needed for the analysis are defined_____no_output_____**Cutoffs setup for binarizing gene expression**

- This dictionary contains a **'parameter'** and a **'type'** key. The first one is the respective parameter associated to the type used.

    - Information according to the 'type' used:
        - 'local_percentile' : computes the value of a given percentile, for each gene independently. In this case, the parameter corresponds to the percentile to compute, as a float value between 0 and 1.

        - 'global_percentile' : computes the value of a given percentile from all genes and samples simultaneously. In this case, the parameter corresponds to the percentile to compute, as a float value between 0 and 1. All genes have the same cutoff.

        - 'file' : load a cutoff table from a file. Parameter in this case is the path of that file. It must contain the same genes as index and same samples as columns.

        - 'multi_col_matrix' : a dataframe must be provided, containing a cutoff for each gene in each sample. This allows to use specific cutoffs for each sample. The columns here must be the same as the ones in the rnaseq_data.

        - 'single_col_matrix' : a dataframe must be provided, containing a cutoff for each gene in only one column. These cutoffs will be applied to all samples.

        - 'constant_value' : binarizes the expression. Evaluates whether expression is greater than the value input in the parameter.

***If used a DE analysis and got DE genes for each sample/cell type, a binary matrix can be used as rnaseq data. Then use a cutoff type of 'constant_value' and a parameter of 0 (so it evaluates expression >0 and will keep the DE genes as ones).***_____no_output_____

cutoff_setup = dict()
cutoff_setup['type'] = 'constant_value'
cutoff_setup['parameter'] = 10 # TPM in this example_____no_output_____

**General Analysis setups**

- 'communication_score' is the type of communication score used to detect active pathways between each pair of cell. It can be:
    - 'expression_thresholding'
    - 'expression_product'
    - 'expression_correlation'
    - 'differential_combinations'
- 'cci_score' is the scoring function to aggregate the communication scores. It can be:
    - 'bray_curtis'
    - 'jaccard'
    - 'count'
- 'cci_type' is the type of interaction between two cells. If it is undirected, ligand and receptor are considered from both cells. If it is directed, ligands from one cell and receptors from the other are considered separately with respect to ligands from the second cell and receptor from the first one. So, it can be:
    - 'undirected'
    - 'directed
- 'complex_sep' is the caracter separating protein subunits in a multimeric complex in the list of PPIs or LR pairs. See PPI data for further details. In this case the separator is '&'. If complexes are not used, use None instead._____no_output_____

analysis_setup = dict()
analysis_setup['communication_score'] = 'expression_thresholding'
analysis_setup['cci_score'] = 'bray_curtis'
analysis_setup['cci_type'] = 'undirected'
analysis_setup['complex_sep'] = '&'_____no_output_____

# Use toy data_____no_output_____**RNA-seq data**

In this case is a 6x5 matrix (6 genes/proteins and 5 samples/cell types). The values are arbitrary; for simplicity we will use them as TPMs. The indexes of this DataFrame are the names of the genes/proteins, while the columns are the names of samples/cell types._____no_output_____

rnaseq = c2c.datasets.generate_toy_rnaseq()_____no_output_____rnaseq_____no_output_____

**Protein-Protein Interactions or Ligand-Receptor Pairs**

In this case, we use a list of LR pairs where some proteins also are ***complexes of multiple subunits***. The complexes are represented as a joint name of ***all subunits*** composing each complex, but ***separated by a separator '&'***. The ligands are in the column 'A', while the receptors in the column 'B'. For these purposes we do not use the column 'score'.

**The names of genes/proteins have to match those in the rnaseq dataset.**_____no_output_____

ppi = c2c.datasets.generate_toy_ppi(prot_complex=True)_____no_output_____ppi_____no_output_____

**Metadata**

The metadata contains the extra information for the samples/cell types in the RNA-seq data. This dataframe must contain a column where the samples/cell types are specified (#SampleID) and another with the new information, for example major groups for the samples/cell types (Groups)_____no_output_____

meta = c2c.datasets.generate_toy_metadata()_____no_output_____meta_____no_output_____

# Cell-cell Interactions and Communication Analysis_____no_output_____**Adapting PPI data**

If we are going to use a undirected CCI score, we have to append the list of PPIs but in the opposite direction in order to have both directions in a cell pair. When using a directed CCI score, we just copy the original data.

In the first case, we also keep the original PPI as a reference for further analysis related with cell-cell communication scores._____no_output_____

if analysis_setup['cci_type'] == 'undirected':
    bi_ppi_data = c2c.preprocessing.bidirectional_ppi_for_cci(ppi_data=ppi,
                                                              verbose=True)
    ref_ppi = ppi
else:
    bi_ppi_data = ppi.copy()
    ref_ppi = NoneMaking bidirectional PPI for CCI.
Removing duplicates in bidirectional PPI network.

**Generate the Interaction Space**

The interaction space integrates the RNA-seq and PPI datasets by using the analysis setups. It generates an instance for each sample/cell type, containing the values assigned to each protein in the PPI list given the setups for computing the CCI and CCC scores._____no_output_____

interaction_space = c2c.core.InteractionSpace(rnaseq_data=rnaseq,
                                              ppi_data=ppi,
                                              gene_cutoffs=cutoff_setup,
                                              communication_score=analysis_setup['communication_score'],
                                              cci_score=analysis_setup['cci_score'],
                                              cci_type=analysis_setup['cci_type'],
                                              complex_sep=analysis_setup['complex_sep'],
                                              verbose=True
                                             )Calculating cutoffs for gene abundances
Creating Interaction Space
Generating objects according to RNAseq datasets provided
New cell instance created for C1
New cell instance created for C2
New cell instance created for C3
New cell instance created for C4
New cell instance created for C5

**Compute communication scores for each PPI or LR pair**

It is computed according to the analysis setups. The interaction_columns are those in our PPI data. Here we want to se a directed interaction, so we need the reference PPI (the one that is not bidirectional for computing undirected CCI scores later)._____no_output_____

interaction_space.compute_pairwise_communication_scores(ref_ppi_data=ref_ppi,
                                                        cci_type='directed',
                                                        interaction_columns=('A', 'B'))Computing pairwise communication
Computing communication score between C4 and C5
Computing communication score between C1 and C1
Computing communication score between C4 and C2
Computing communication score between C4 and C1
Computing communication score between C3 and C4
Computing communication score between C2 and C1
Computing communication score between C4 and C3
Computing communication score between C5 and C3
Computing communication score between C4 and C4
Computing communication score between C1 and C2
Computing communication score between C2 and C4
Computing communication score between C2 and C3
Computing communication score between C2 and C5
Computing communication score between C3 and C2
Computing communication score between C3 and C1
Computing communication score between C5 and C5
Computing communication score between C5 and C2
Computing communication score between C1 and C3
Computing communication score between C1 and C4
Computing communication score between C2 and C2
Computing communication score between C3 and C3
Computing communication score between C3 and C5
Computing communication score between C5 and C1
Computing communication score between C1 and C5
Computing communication score between C5 and C4

**Compute CCI scores for each pair of cells**

It is computed according to the analysis setups. We computed an undirected Bray-Curtis-like score for each pair, meaning that score(C1, C2) = score(C2, C1). We do not weight the scores for each PPI or LR pair, so use_ppi_score is False (the one in the column score of the PPI data would be used here if this is true).

**Notice that our score is undirected, so for C1 and C2 was not necessary to compute C2 and C1 as happened for the communication scores**_____no_output_____

interaction_space.compute_pairwise_cci_scores(use_ppi_score=False,
                                              verbose=True)Computing pairwise interactions
Computing interaction score between C4 and C5
Computing interaction score between C1 and C4
Computing interaction score between C2 and C4
Computing interaction score between C1 and C1
Computing interaction score between C2 and C3
Computing interaction score between C2 and C5
Computing interaction score between C3 and C4
Computing interaction score between C3 and C5
Computing interaction score between C2 and C2
Computing interaction score between C3 and C3
Computing interaction score between C5 and C5
Computing interaction score between C1 and C5
Computing interaction score between C4 and C4
Computing interaction score between C1 and C2
Computing interaction score between C1 and C3

# Visualizations

If we want to save the figure as a vector figure, we can pass a pathname into the filename input to save it. E.g. ***filename='/Users/cell2cell/CommScores.svg'***_____no_output_____**Generate colors for the groups in the metadata**

It returns a dictionary with the colors._____no_output_____

colors = c2c.plotting.get_colors_from_labels(labels=meta['Groups'].unique().tolist(),
                                             cmap='tab10'
                                            )_____no_output_____colors_____no_output_____

**Visualize communication scores for each LR pair and each cell pair**_____no_output_____

interaction_clustermap = c2c.plotting.clustermap_ccc(interaction_space,
                                                     metric='jaccard',
                                                     method='complete',
                                                     metadata=meta,
                                                     sample_col='#SampleID',
                                                     group_col='Groups',
                                                     colors=colors,
                                                     row_fontsize=14,
                                                     title='Active ligand-receptor pairs for interacting cells',
                                                     filename=None,
                                                     cell_labels=('SENDER-CELLS', 'RECEIVER-CELLS'),
                                                     **{'figsize' : (10,9)}
                                                     )

# Add a legend to know the groups of the sender and receiver cells:
l1 = c2c.plotting.generate_legend(color_dict=colors,
                                  loc='center left',
                                  bbox_to_anchor=(20, -2), # Indicated where to include it
                                  ncol=1, fancybox=True,
                                  shadow=True,
                                  title='Groups',
                                  fontsize=14,
                                 )Interaction space detected as an InteractionSpace class

**Circos plot**

It generates a circos plot, showing the cells producing ligands (sender_cells), according to a list of specific ligands (ligands) to the cells producing receptors (receiver_cells), according to a list of specific receptors (receptors). The order of these lists is preserved in the visualization. Elements that are not used are omitted and therefore not plotted (in this case those with a communication score of 0, specified in 'excluded_score')._____no_output_____

sender_cells = ['C1', 'C2', 'C5']
receiver_cells = ['C2', 'C3', 'C4', 'C5']
ligands = ['Protein-C',
           'Protein-E',
           'Protein-C&Protein-E',
           'Protein F'
          ]
receptors = ['Protein-B',
             'Protein-C',
             'Protein-F',
             'Protein-A'
            ]_____no_output_____c2c.plotting.circos_plot(interaction_space=interaction_space,
                         sender_cells=sender_cells,
                         receiver_cells=receiver_cells,
                         ligands=ligands,
                         receptors=receptors,
                         excluded_score=0,
                         metadata=meta,
                         sample_col='#SampleID',
                         group_col='Groups',
                         colors=colors,
                         fontsize=15,
                        )_____no_output_____

If we do not pass metadata info, the samples/cell types are only plotted.

We can also change the label colors of ligands and receptors_____no_output_____

c2c.plotting.circos_plot(interaction_space=interaction_space,
                         sender_cells=sender_cells,
                         receiver_cells=receiver_cells,
                         ligands=ligands,
                         receptors=receptors,
                         excluded_score=0,
                         fontsize=20,
                         ligand_label_color='orange',
                         receptor_label_color='brown',
                        )_____no_output_____

**Visualize CCI scores**

Since this case is undirected, a triangular heatmap is plotted instead of the complete one._____no_output_____

cm = c2c.plotting.clustermap_cci(interaction_space,
                                 method='complete',
                                 metadata=meta,
                                 sample_col="#SampleID",
                                 group_col="Groups",
                                 colors=colors,
                                 title='CCI scores for cell-types',
                                 cmap='Blues'
                                 )

# Add a legend to know the groups of the sender and receiver cells:
l1 = c2c.plotting.generate_legend(color_dict=colors,
                                  loc='center left',
                                  bbox_to_anchor=(20, -2), # Indicated where to include it
                                  ncol=1, fancybox=True,
                                  shadow=True,
                                  title='Groups',
                                  fontsize=14,
                                 )Interaction space detected as an InteractionSpace class

**Project samples/cells with PCoA into a Euclidean space**

We can project the samples/cells given their CCI scores with other cells and see how close they are given their potential of interaction.

**THIS ONLY WORKS WITH UNDIRECTED CCI SCORES**_____no_output_____

if (interaction_space.interaction_elements['cci_matrix'].values.transpose() == interaction_space.interaction_elements['cci_matrix'].values).all():

    pcoa = c2c.plotting.pcoa_3dplot(interaction_space,
                                    metadata=meta,
                                    sample_col="#SampleID",
                                    group_col="Groups",
                                    title='PCoA based on potential CCIs',
                                    colors=colors,
                                    )Interaction space detected as an InteractionSpace class


# Notebook from greenelab/Pseudomonas_latent_spaces
Path: exploration/scripts/generate_input_continuum.ipynb

<code>
%matplotlib inline_____no_output_____#-------------------------------------------------------------------------------------------------------------------------------
# By Alexandra Lee (July 2018)
#
# Generate input files
#
# Dataset: Pseudomonas aeruginosa gene expression compendium referenced in https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5069748/
#
# Use map_file to group samples into phenotype groups (condition A and B) based on experimental design annotations
# Example: control vs treatment with antibiotics
#
# Then group samples into training and test sets
#
# Generate offset vector using gene expression data in the original space (train_offset_original):
# average gene expression for condition A - average gene expression for condition B using all genes/dimensions
#-------------------------------------------------------------------------------------------------------------------------------
import os
import pandas as pd
import numpy as np
from scipy.stats import variation
import seaborn as sns
import matplotlib.pyplot as plt

randomState = 123
from numpy.random import seed
seed(randomState)_____no_output_____# load arguments
data_file = os.path.join(os.path.dirname(os.getcwd()), "data", "all-pseudomonas-gene-normalized.zip")  # repo file is zipped
map_file = os.path.join(os.path.dirname(os.getcwd()), "metadata", "mapping_PA1673.txt")

# output
train_max_file = os.path.join(os.path.dirname(os.getcwd()), "data", "PA1673", "train_minExp.txt")
train_min_file = os.path.join(os.path.dirname(os.getcwd()), "data", "PA1673", "train_maxExp.txt")
train_input_file = os.path.join(os.path.dirname(os.getcwd()), "data", "PA1673", "train_model_input.txt.xz")
original_offset_file = os.path.join(os.path.dirname(os.getcwd()), "data", "PA1673", "train_offset_original.txt")_____no_output_____# read in data
data = pd.read_table(data_file, header = 0, sep = '\t', index_col = 0, compression='zip')
X = data.transpose()
X.head(5)_____no_output_____# read in metadata file containing grouping of each sample into training/test and phenotypic group
grp = pd.read_table(map_file, header=0, sep='\t', index_col=None)
grp_____no_output_____# Group samples into training and test sets
# Training: min and max levels of O2
# Test: all intermediate levels

maxO2 = pd.DataFrame()
minO2 = pd.DataFrame()
intermediate = pd.DataFrame()

for index, row in grp.iterrows():
    if row['Group'] == 'Train':
        if row['Phenotype'] == "minExp":
            sample = str(row['Sample ID'])
            maxO2 = maxO2.append(X[X.index.str.contains(sample, regex=False)])
        else:
            sample = str(row['Sample ID'])
            minO2 = minO2.append(X[X.index.str.contains(sample, regex=False)])
    if row['Group'] == 'Test':
        sample = str(row['Sample ID'])
        intermediate = intermediate.append(X[X.index.str.contains(sample, regex=False)])

#maxO2
#minO2
intermediate_____no_output_____# Create input holding out test test (intermediate time points)
input_holdout = X.drop(intermediate.index)

input_holdout.shape
#X.shape_____no_output_____# Generate offset using average gene expression in original dataset
train_offset_original = minO2.values - maxO2.values
train_offset_original = pd.DataFrame(train_offset_original, columns = minO2.columns)
train_offset_original_____no_output_____# Output training and test sets

# training data
maxO2.to_csv(train_max_file, sep='\t')
minO2.to_csv(train_min_file, sep='\t')
input_holdout.to_csv(train_input_file, sep='\t', compression='xz')

# test data
for index, row in grp.iterrows():
    if row['Group'] == 'Test':
        sample = str(row['Sample ID'])
        df = pd.DataFrame(intermediate.loc[sample]).transpose()
        df.to_csv(os.path.join(os.path.dirname(os.getcwd()), "data", "PA1673", "test_"+row['Phenotype']+".txt"), sep='\t')

# original offset
train_offset_original.to_csv(original_offset_file, sep='\t')
_____no_output_____
</code>


# Notebook from kazewong/nrpytutorial
Path: notebook/Tutorial-GiRaFFE_Higher_Order_v2.ipynb

# $\texttt{GiRaFFE}$: General Relativistic Force-Free Electrodynamics

## Authors: Patrick Nelson, Zachariah Etienne, George Vopal & Maria Babiuc-Hamilton
### Formatting improvements courtesy Brandon Clark

## GiRaFFE_HO_v2 is an experiment that omits the analytic derivatives of $\partial_j T^j_{{\rm EM} i}$

**Module Status:**  In progress

**Validation Notes:** This module is under active development -- do ***not*** use the resulting C code output for doing science.

### NRPy+ Source Code for this module: [GiRaFFE_HO/GiRaFFE_Higher_Order_v2.py](../edit/GiRaFFE_HO/GiRaFFE_Higher_Order_v2.py)

## Introduction:
The original $\texttt{GiRaFFE}$ code, as presented in [the original paper](https://arxiv.org/pdf/1704.00599.pdf), exists as a significant modification to $\texttt{IllinoisGRMHD}$. As such, it used a third-order reconstruction algorithm with a slope limiter (Colella et al's piecewise parabolic method, or PPM) to handle spatial derivatives in the general relativistic force-free electrodynamics (GRFFE) equations. However the GRFFE equations do not generally permit shocks, so a more optimal approach would involve finite differencing all derivatives in the GRFFE equations. As it happens, NRPy+ was designed to generate C codes involving complex tensorial expressions and finite difference spatial derivatives, with finite-differencing order a freely-specifiable parameter.

The purpose of this notebook is to rewrite the equations of GRFFE as used in the original $\texttt{GiRaFFE}$ code so that all derivatives that appear are represented numerically as finite difference derivatives.

The GRFFE evolution equations (from eq. 13 of the [original paper](https://arxiv.org/pdf/1704.00599.pdf)) we wish to encode in the NRPy+ version of $\texttt{GiRaFFE}$ are as follows:

* $\partial_t \tilde{S}_i = - \partial_j \left( \alpha \sqrt{\gamma} T^j_{{\rm EM} i} \right) + \frac{1}{2} \alpha \sqrt{\gamma} T^{\mu \nu}_{\rm EM} \partial_i g_{\mu \nu}$: [Link to Step 3.0](#step7)
* $\partial_t A_i = \epsilon_{ijk} v^j B^k - \partial_i (\alpha \Phi - \beta^j A_j)$: [Link to Step 4.0](#step8)
* $\partial_t [\sqrt{\gamma} \Phi] = -\partial_j (\alpha\sqrt{\gamma}A^j - \beta^j [\sqrt{\gamma} \Phi]) - \xi \alpha [\sqrt{\gamma} \Phi]$: [Link to Step 4.0](#step8)

Here, the densitized spatial Poynting flux one-form $\tilde{S}_i = \sqrt{\gamma} S_i$ (and $S_i$ comes from $S_{\mu} -n_{\nu} T^{\nu}_{{\rm EM} \mu}$), and $(\Phi, A_i)$ is the vector potential. We will solve these PDEs using the method of lines, where the right-hand sides of these equations (involving no explicit time derivatives) will be constructed using NRPy+.

### A Note on Notation:

As is standard in NRPy+,

* Greek indices refer to four-dimensional quantities where the zeroth component indicates temporal (time) component.
* Latin indices refer to three-dimensional quantities. This is somewhat counterintuitive since Python always indexes its lists starting from 0. As a result, the zeroth component of three-dimensional quantities will necessarily indicate the first *spatial* direction.

For instance, in calculating the first term of [$T_{\rm EM}^{\mu\nu}$](#em_tensor) (specifically, ${\rm Term\ 1} = b^2 u^\mu u^\nu$), we use Greek indices:

```
T4EMUU = ixp.zerorank2(DIM=4)
for mu in range(4):
    for nu in range(4):
        # Term 1: b^2 u^{\mu} u^{\nu}
        T4EMUU[mu][nu] = smallb2*u4U[mu]*u4U[nu]\
```

When we calculate [$\beta_i = \gamma_{ij} \beta^j$](#4metric), we use Latin indices:
```
betaD = ixp.zerorank1()
for i in range(DIM):
    for j in range(DIM):
        betaD[i] += gammaDD[i][j] * betaU[j]
```

As a corollary, any expressions involving mixed Greek and Latin indices will need to offset one set of indices by one: A Latin index in a four-vector will be incremented and a Greek index in a three-vector will be decremented (however, the latter case does not occur in this tutorial notebook). This can be seen when we handle the second term of [$\partial_t \tilde{S}_i$](#construct_si) (or, more specifically, the second term thereof: $\frac{1}{2} \alpha \sqrt{\gamma} T^{\mu \nu}_{\rm EM} \partial_i g_{\mu \nu}$):
```
# The second term: \alpha \sqrt{\\gamma} T^{\mu \nu}_{\rm EM} \partial_i g_{\mu \nu} / 2
for i in range(DIM):
    for mu in range(4):
        for nu in range(4):
            Stilde_rhsD[i] += alpsqrtgam * T4EMUU[mu][nu] * g4DDdD[mu][nu][i+1] / 2
```_____no_output_____

# Table of Contents:
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Set up the needed NRPy+ infrastructure and declare core gridfunctions used by $\texttt{GiRaFFE}$
1. [Step 2](#4metric): Build the four metric $g_{\mu\nu}$, its inverse $g^{\mu\nu}$ and spatial derivatives $g_{\mu\nu,i}$ from ADM 3+1 quantities $\gamma_{ij}$, $\beta^i$, and $\alpha$
1. [Step 3](#t_derivatives): $T^{\mu\nu}_{\rm EM}$ and its derivatives
    1. [Step 3.a](#uibi): $u^i$ and $b^i$ and related quantities
    1. [Step 3.b](#em_tensor): Construct all components of the electromagnetic stress-energy tensor $T^{\mu \nu}_{\rm EM}$
1. [Step 4](#construct_si): Construct the evolution equation for $\tilde{S}_i$
1. [Step 5](#construct_ai): Construct the evolution equations for $A_i$ and $[\sqrt{\gamma}\Phi]$
    1. [Step 5.a](#aux): Construct some useful auxiliary gridfunctions for the other evolution equations
    1. [Step 5.b](#complete_construct_ai): Complete the construction of the evolution equations for $A_i$ and $[\sqrt{\gamma}\Phi]$
1. [Step 6](#code_validation): Code Validation against `GiRaFFE_HO.GiRaFFE_Higher_Order_v2` NRPy+ Module
1. [Step 7](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file_____no_output_____

# Step 1: Set up the needed NRPy+ infrastructure and declare core gridfunctions used by $\texttt{GiRaFFE}$ \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

1. Set some basic NRPy+ parameters. E.g., set the spatial dimension parameter to 3 and the finite differencing order to 4.
1. Next, declare some gridfunctions that are provided as input to the equations:
    1. $\alpha$, $\beta^i$, and $\gamma_{ij}$: These ADM 3+1 metric quantities are declared in the ADMBase Einstein Toolkit thorn, and are assumed to be made available to $\texttt{GiRaFFE}$ at this stage.
    1. The Valencia 3-velocity $v^i_{(n)}$ and vector potential $A_i$: Declared by $\texttt{GiRaFFE}$, and will have their initial values set in the separate thorn **GiRaFFEfood_HO**.
    1. The magnetic field as measured by a normal observer $B^i$: The quantities evolved forward in time in $\texttt{GiRaFFE}$ do not include the Valencia 3-velocity, so this quantity is not automatically updated. Instead, we compute it based on the evolved quantity $\tilde{S}_i$ and $B^i = \epsilon^{ijk} \partial_j A_k$ (where $A_k$ is another evolved quantity and $\epsilon^{ijk}$ is the Levi-Civita tensor). $B^i$ is evaluated using finite differences of $A_k$ in a separate function, though it can only be evaluated consistently on the interior of the grid. In the ghost zones, we will have to use lower-order derivatives.
_____no_output_____

import NRPy_param_funcs as par
import indexedexp as ixp
import grid as gri
import finite_difference as fin
from outputC import *

#Step 1.0: Set the spatial dimension parameter to 3.
par.set_parval_from_str("grid::DIM", 3)
DIM = par.parval_from_str("grid::DIM")

# Step 1.1: Set the finite differencing order to 4.
par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER", 4)

thismodule = "GiRaFFE_NRPy"

# M_PI will allow the C code to substitute the correct value
M_PI = par.Cparameters("#define",thismodule,"M_PI","")
# ADMBase defines the 4-metric in terms of the 3+1 spacetime metric quantities gamma_{ij}, beta^i, and alpha
gammaDD = ixp.register_gridfunctions_for_single_rank2("AUX","gammaDD", "sym01",DIM=3)
betaU   = ixp.register_gridfunctions_for_single_rank1("AUX","betaU",DIM=3)
alpha   = gri.register_gridfunctions("AUX","alpha")
# GiRaFFE uses the Valencia 3-velocity and A_i, which are defined in the initial data module(GiRaFFEfood)
ValenciavU = ixp.register_gridfunctions_for_single_rank1("AUX","ValenciavU",DIM=3)
AD = ixp.register_gridfunctions_for_single_rank1("EVOL","AD",DIM=3)
# B^i must be computed at each timestep within GiRaFFE so that the Valencia 3-velocity can be evaluated
BU = ixp.register_gridfunctions_for_single_rank1("AUX","BU",DIM=3)_____no_output_____

# Step 2: Build the four metric $g_{\mu\nu}$, its inverse $g^{\mu\nu}$ and spatial derivatives $g_{\mu\nu,i}$ from ADM 3+1 quantities $\gamma_{ij}$, $\beta^i$, and $\alpha$ \[Back to [top](#toc)\]
$$\label{4metric}$$

Notice that the time evolution equation for $\tilde{S}_i$
$$
\partial_t \tilde{S}_i = - \partial_j \left( \alpha \sqrt{\gamma} T^j_{{\rm EM} i} \right) + \frac{1}{2} \alpha \sqrt{\gamma} T^{\mu \nu}_{\rm EM} \partial_i g_{\mu \nu}
$$
contains $\partial_i g_{\mu \nu} = g_{\mu\nu,i}$. We will now focus on evaluating this term.

The four-metric $g_{\mu\nu}$ is related to the three-metric $\gamma_{ij}$, index-lowered shift $\beta_i$, and lapse $\alpha$ by
$$
g_{\mu\nu} = \begin{pmatrix}
-\alpha^2 + \beta^k \beta_k & \beta_j \\
\beta_i & \gamma_{ij}
\end{pmatrix}.
$$
This tensor and its inverse have already been built by the u0_smallb_Poynting__Cartesian.py module ([documented here](Tutorial-u0_smallb_Poynting-Cartesian.ipynb)), so we can simply load the module and import the variables._____no_output_____

# Step 1.2: import u0_smallb_Poynting__Cartesian.py to set
#           the four metric and its inverse. This module also sets b^2 and u^0.
import u0_smallb_Poynting__Cartesian.u0_smallb_Poynting__Cartesian as u0b
u0b.compute_u0_smallb_Poynting__Cartesian(gammaDD,betaU,alpha,ValenciavU,BU)

betaD = ixp.zerorank1()
for i in range(DIM):
    for j in range(DIM):
        betaD[i] += gammaDD[i][j] * betaU[j]
# Error check: fixed = to +=

# We will now pull in the four metric and its inverse.
g4DD = ixp.zerorank2(DIM=4)
g4UU = ixp.zerorank2(DIM=4)
for mu in range(4):
    for nu in range(4):
        g4DD[mu][nu] = u0b.g4DD[mu][nu]
        g4UU[mu][nu] = u0b.g4UU[mu][nu]_____no_output_____

Next we compute spatial derivatives of the metric, $\partial_i g_{\mu\nu} = g_{\mu\nu,i}$, written in terms of the three-metric, shift, and lapse. Simply taking the derivative of the expression for $g_{\mu\nu}$ above, we find
$$
g_{\mu\nu,l} = \begin{pmatrix}
-2\alpha \alpha_{,l} + \beta^k_{\ ,l} \beta_k + \beta^k \beta_{k,l} & \beta_{j,l} \\
\beta_{i,l} & \gamma_{ij,l}
\end{pmatrix}.
$$

Notice the derivatives of the shift vector with its indexed lowered, $\beta_{i,j} = \partial_j \beta_i$. This can be easily computed in terms of the given ADMBase quantities $\beta^i$ and $\gamma_{ij}$ via:
\begin{align}
\beta_{i,j} &= \partial_j \beta_i \\
            &= \partial_j (\gamma_{ik} \beta^k) \\
            &= \gamma_{ik} \partial_j\beta^k + \beta^k \partial_j \gamma_{ik} \\
\beta_{i,j} &= \gamma_{ik} \beta^k_{\ ,j} + \beta^k \gamma_{ik,j}.
\end{align}

Since this expression mixes Greek and Latin indices, we will declare this as a 4 dimensional quantity, but only set the three spatial components of its last index (that is, leaving $l=0$ unset).

So, we will first set
$$ g_{00,l} = \underbrace{-2\alpha \alpha_{,l}}_{\rm Term\ 1} + \underbrace{\beta^k_{\ ,l} \beta_k}_{\rm Term\ 2} + \underbrace{\beta^k \beta_{k,l}}_{\rm Term\ 3} $$_____no_output_____

# Step 1.2, cont'd: Build spatial derivatives of the four metric
# Step 1.2.a: Declare derivatives of grid functions. These will be handled by FD_outputC
alpha_dD   = ixp.declarerank1("alpha_dD")
betaU_dD   = ixp.declarerank2("betaU_dD","nosym")
gammaDD_dD = ixp.declarerank3("gammaDD_dD","sym01")

# Step 1.2.b: These derivatives will be constructed analytically.
betaDdD    = ixp.zerorank2()
g4DDdD     = ixp.zerorank3(DIM=4)

for i in range(DIM):
    for j in range(DIM):
        for k in range(DIM):
            # \gamma_{ik} \beta^k_{,j} + \beta^k \gamma_{ik,j}
            betaDdD[i][j] += gammaDD[i][k] * betaU_dD[k][j] + betaU[k] * gammaDD_dD[i][k][j]
#Error check: changed = to += above

# Step 1.2.c: Set the 00 components
# Step 1.2.c.i: Term 1: -2\alpha \alpha_{,l}
for l in range(DIM):
    g4DDdD[0][0][l+1] = -2*alpha*alpha_dD[l]

# Step 1.2.c.ii: Term 2: \beta^k_{\ ,l} \beta_k
for l in range(DIM):
    for k in range(DIM):
        g4DDdD[0][0][l+1] += betaU_dD[k][l] * betaD[k]

# Step 1.2.c.iii: Term 3: \beta^k \beta_{k,l}
for l in range(DIM):
    for k in range(DIM):
        g4DDdD[0][0][l+1] += betaU[k] * betaDdD[k][l]_____no_output_____

Now we will contruct the other components of $g_{\mu\nu,l}$. We will first construct
$$ g_{i0,l} = g_{0i,l} = \beta_{i,l}, $$
then
$$ g_{ij,l} = \gamma_{ij,l} $$_____no_output_____

# Step 1.2.d: Set the i0 and 0j components
for l in range(DIM):
    for i in range(DIM):
        # \beta_{i,l}
        g4DDdD[i+1][0][l+1] = g4DDdD[0][i+1][l+1] = betaDdD[i][l]

#Step 1.2.e: Set the ij components
for l in range(DIM):
    for i in range(DIM):
        for j in range(DIM):
            # \gamma_{ij,l}
            g4DDdD[i+1][j+1][l+1] = gammaDD_dD[i][j][l]_____no_output_____

# Step 3: $T^{\mu\nu}_{\rm EM}$ and its derivatives \[Back to [top](#toc)\]
$$\label{t_derivatives}$$

Now that the metric and its derivatives are out of the way, let's return to the evolution equation for $\tilde{S}_i$,
$$
\partial_t \tilde{S}_i = - \partial_j \left( \alpha \sqrt{\gamma} T^j_{{\rm EM} i} \right) + \frac{1}{2} \alpha \sqrt{\gamma} T^{\mu \nu}_{\rm EM} \partial_i g_{\mu \nu}.
$$
We turn our focus now to $T^j_{{\rm EM} i}$ and its derivatives. To this end, we start by computing $T^{\mu \nu}_{\rm EM}$ (from eq. 27 of [Paschalidis & Shapiro's paper on their GRFFE code](https://arxiv.org/pdf/1310.3274.pdf)):

$$\boxed{T^{\mu \nu}_{\rm EM} = b^2 u^{\mu} u^{\nu} + \frac{b^2}{2} g^{\mu \nu} - b^{\mu} b^{\nu}.}$$

Notice that $T^{\mu\nu}_{\rm EM}$ is written in terms of

* $b^\mu$, the 4-component magnetic field vector, related to the comoving magnetic field vector $B^i_{(u)}$
* $u^\mu$, the 4-velocity
* $g^{\mu \nu}$, the inverse 4-metric

However, $\texttt{GiRaFFE}$ has access to only the following quantities, requiring in the following sections that we write the above quantities in terms of the following ones:

* $\gamma_{ij}$, the 3-metric
* $\alpha$, the lapse
* $\beta^i$, the shift
* $A_i$, the vector potential
* $B^i$, the magnetic field (we assume only in the grid interior, not the ghost zones)
* $\left[\sqrt{\gamma}\Phi\right]$, the zero-component of the vector potential $A_\mu$, times the square root of the determinant of the 3-metric
* $v_{(n)}^i$, the Valencia 3-velocity
* $u^0$, the zero-component of the 4-velocity_____no_output_____

## Step 3.a: $u^i$ and $b^i$ and related quantities \[Back to [top](#toc)\]
$$\label{uibi}$$

We begin by importing what we can from [u0_smallb_Poynting__Cartesian.py](../edit/u0_smallb_Poynting__Cartesian/u0_smallb_Poynting__Cartesian.py). We will need the four-velocity $u^\mu$, which is related to the Valencia 3-velocity $v^i_{(n)}$ used directly by $\texttt{GiRaFFE}$ (see also [Duez, et al, eqs. 53 and 56](https://arxiv.org/pdf/astro-ph/0503420.pdf))
\begin{align}
u^i &= u^0 (\alpha v^i_{(n)} - \beta^i), \\
u_j &= \alpha u^0 \gamma_{ij} v^i_{(n)},
\end{align}
where $v^i_{(n)}$ is the Valencia three-velocity. These have already been constructed in terms of the Valencia 3-velocity and other 3+1 ADM quantities by the [u0_smallb_Poynting__Cartesian.py](../edit/u0_smallb_Poynting__Cartesian/u0_smallb_Poynting__Cartesian.py) module, so we can simply import these variables:
_____no_output_____

# Step 2.0: u^i, b^i, and related quantities
# Step 2.0.a: import the four-velocity, as written in terms of the Valencia 3-velocity
uD = ixp.zerorank1()
uU = ixp.zerorank1()
u4upperZero = gri.register_gridfunctions("AUX","u4upperZero")

for i in range(DIM):
    uD[i] = u0b.uD[i].subs(u0b.u0,u4upperZero)
    uU[i] = u0b.uU[i].subs(u0b.u0,u4upperZero)
_____no_output_____

We also need the magnetic field 4-vector $b^{\mu}$, which is related to the magnetic field by [eqs. 23, 24, and 31 in Duez, et al](https://arxiv.org/pdf/astro-ph/0503420.pdf):
\begin{align}
b^0 &= \frac{1}{\sqrt{4\pi}} B^0_{\rm (u)} = \frac{u_j B^j}{\sqrt{4\pi}\alpha}, \\
b^i &= \frac{1}{\sqrt{4\pi}} B^i_{\rm (u)} = \frac{B^i + (u_j B^j) u^i}{\sqrt{4\pi}\alpha u^0}, \\
\end{align}
where $B^i$ is the variable tracked by the HydroBase thorn in the Einstein Toolkit. Again, these have already been built by the [u0_smallb_Poynting__Cartesian.py](../edit/u0_smallb_Poynting__Cartesian/u0_smallb_Poynting__Cartesian.py), so we can simply import the variables._____no_output_____

# Step 2.0.b: import the small b terms
smallb4U = ixp.zerorank1(DIM=4)
smallb4D = ixp.zerorank1(DIM=4)
for mu in range(4):
    smallb4U[mu] = u0b.smallb4U[mu].subs(u0b.u0,u4upperZero)
    smallb4D[mu] = u0b.smallb4D[mu].subs(u0b.u0,u4upperZero)

smallb2 = u0b.smallb2etk.subs(u0b.u0,u4upperZero)
_____no_output_____

## Step 3.b: Construct all components of the electromagnetic stress-energy tensor $T^{\mu \nu}_{\rm EM}$ \[Back to [top](#toc)\]
$$\label{em_tensor}$$

We now have all the pieces to calculate the stress-energy tensor,
$$T^{\mu \nu}_{\rm EM} = \underbrace{b^2 u^{\mu} u^{\nu}}_{\rm Term\ 1} +
\underbrace{\frac{b^2}{2} g^{\mu \nu}}_{\rm Term\ 2}
- \underbrace{b^{\mu} b^{\nu}}_{\rm Term\ 3}.$$
Because $u^0$ is a separate variable, we will create a temporary variable $u^\mu=\left( u^0, u^i \right)$,_____no_output_____

# Step 2.1: Construct the electromagnetic stress-energy tensor
# Step 2.1.a: Set up the four-velocity vector
u4U = ixp.zerorank1(DIM=4)
u4U[0] = u4upperZero
for i in range(DIM):
    u4U[i+1] = uU[i]

# Step 2.1.b: Build T4EMUU itself
T4EMUU = ixp.zerorank2(DIM=4)
for mu in range(4):
    for nu in range(4):
        # Term 1: b^2 u^{\mu} u^{\nu}
        T4EMUU[mu][nu] = smallb2*u4U[mu]*u4U[nu]

for mu in range(4):
    for nu in range(4):
        # Term 2: b^2 / 2 g^{\mu \nu}
        T4EMUU[mu][nu] += smallb2*g4UU[mu][nu]/2

for mu in range(4):
    for nu in range(4):
        # Term 3: -b^{\mu} b^{\nu}
        T4EMUU[mu][nu] += -smallb4U[mu]*smallb4U[nu]_____no_output_____

# Step 4: Construct the evolution equation for $\tilde{S}_i$ \[Back to [top](#top)\]
$$\label{construct_si}$$

Finally, we will return our attention to the time evolution equation (from eq. 13 of the [original paper](https://arxiv.org/pdf/1704.00599.pdf)),
\begin{align}
\partial_t \tilde{S}_i &= - \partial_j \underbrace{\left( \alpha \sqrt{\gamma} T^j_{{\rm EM} i} \right)}_{\rm SevolParenUD} + \frac{1}{2} \alpha \sqrt{\gamma} T^{\mu \nu}_{\rm EM} \partial_i g_{\mu \nu} \\
                       &= - \partial_j{\rm SevolParenUD[j][i]} + \frac{1}{2} \alpha \sqrt{\gamma} T^{\mu \nu}_{\rm EM} \partial_i g_{\mu \nu} .
\end{align}
We will first construct `SevolParenUD`, then use its derivatives to construct the evolution equation. Note that
\begin{align}
{\rm SevolParenUD[j][i]} &= \alpha \sqrt{\gamma} T^j_{{\rm EM} i} \\
                             &= \alpha \sqrt{\gamma} g_{\mu i} T^{\mu j}_{\rm EM}.
\end{align}_____no_output_____

# Step 3.0: Construct the evolution equation for \tilde{S}_i
# Here, we set up the necessary machinery to take FD derivatives of alpha * sqrt(gamma)
gammaUU = ixp.register_gridfunctions_for_single_rank2("AUX","gammaUU","sym01")
gammadet = gri.register_gridfunctions("AUX","gammadet")
gammaUU, gammadet = ixp.symm_matrix_inverter3x3(gammaDD)

alpsqrtgam = alpha*sp.sqrt(gammadet)

SevolParenUD = ixp.register_gridfunctions_for_single_rank2("AUX","SevolParenUD","nosym")
SevolParenUD = ixp.zerorank2()

for i in range(DIM):
    for j in range (DIM):
        for mu in range(4):
            SevolParenUD[j][i] += alpsqrtgam * g4DD[mu][i+1] * T4EMUU[mu][j+1]

SevolParenUD_dD = ixp.declarerank3("SevolParenUD_dD","nosym")

Stilde_rhsD = ixp.zerorank1()
# The first term: \alpha \sqrt{\gamma} \partial_j T^j_{{\rm EM} i}
for i in range(DIM):
    for j in range(DIM):
        Stilde_rhsD[i] += -SevolParenUD_dD[j][i][j]

# The second term: \alpha \sqrt{\gamma} T^{\mu \nu}_{\rm EM} \partial_i g_{\mu \nu} / 2
for i in range(DIM):
    for mu in range(4):
        for nu in range(4):
            Stilde_rhsD[i] += alpsqrtgam * T4EMUU[mu][nu] * g4DDdD[mu][nu][i+1] / 2
_____no_output_____

# Step 5: Construct the evolution equations for $A_i$ and $[\sqrt{\gamma}\Phi]$ \[Back to [top](#toc)\]
$$\label{construct_ai}$$

We will also need to evolve the vector potential $A_i$. This evolution is given as eq. 17 in the [$\texttt{GiRaFFE}$](https://arxiv.org/pdf/1704.00599.pdf) paper:
$$\boxed{\partial_t A_i = \epsilon_{ijk} v^j B^k - \partial_i (\underbrace{\alpha \Phi - \beta^j A_j}_{\rm AevolParen}),}$$
where $\epsilon_{ijk} = [ijk] \sqrt{\gamma}$ is the antisymmetric Levi-Civita tensor, the drift velocity $v^i = u^i/u^0$, $\gamma$ is the determinant of the three metric, $B^k$ is the magnetic field, $\alpha$ is the lapse, and $\beta$ is the shift.

We also need the scalar electric potential $\Phi$, which is evolved by eq. 19:
$$\boxed{\partial_t [\sqrt{\gamma} \Phi] = -\partial_j (\underbrace{\alpha\sqrt{\gamma} \gamma^{ij} A_i - \beta^j [\sqrt{\gamma} \Phi]}_{\rm PevolParenU[j]}) - \xi \alpha [\sqrt{\gamma} \Phi],}$$
with $\xi$ chosen as a damping factor. _____no_output_____

## Step 5.a: Construct some useful auxiliary gridfunctions for the other evolution equations \[Back to [top](#toc)\]
$$\label{aux}$$

After declaring a  some needed quantities, we will also define the parenthetical terms (underbrace above) that we need to take derivatives of. That way, we can take finite-difference derivatives easily. Note that the above equations incorporate the fact that $\gamma^{ij}$ is the appropriate raising operator for $A_i$: $A^j = \gamma^{ij} A_i$. This is correct since $n_\mu A^\mu = 0$, where $n_\mu$ is a normal to the hypersurface, so $A^0=0$ (according to Sec. II, subsection C of [the "Improved EM gauge condition" paper of Etienne *et al*](https://arxiv.org/pdf/1110.4633.pdf))._____no_output_____

# Step 4.0: Construct the evolution equations for A_i and sqrt(gamma)Phi
# Step 4.0.a: Construct some useful auxiliary gridfunctions for the other evolution equations
xi = par.Cparameters("REAL",thismodule,"xi",0.1) # The damping factor

# Define sqrt(gamma)Phi as psi6Phi
psi6Phi = gri.register_gridfunctions("EVOL","psi6Phi")
Phi = psi6Phi / sp.sqrt(gammadet)

# We'll define a few extra gridfunctions to avoid complicated derivatives
AevolParen  = gri.register_gridfunctions("AUX","AevolParen")
PevolParenU = ixp.register_gridfunctions_for_single_rank1("AUX","PevolParenU")

# {\rm AevolParen} = \alpha \Phi - \beta^j A_j
AevolParen = alpha*Phi
for j in range(DIM):
    AevolParen += -betaU[j] * AD[j]

# {\rm PevolParenU[j]} = \alpha\sqrt{\gamma} \gamma^{ij} A_i - \beta^j [\sqrt{\gamma} \Phi]
for j in range(DIM):
    PevolParenU[j] = -betaU[j] * psi6Phi
    for i in range(DIM):
        PevolParenU[j] += alpsqrtgam * gammaUU[i][j] * AD[i]

AevolParen_dD  = ixp.declarerank1("AevolParen_dD")
PevolParenU_dD = ixp.declarerank2("PevolParenU_dD","nosym")
_____no_output_____

## Step 5.b: Complete the construction of the evolution equations for $A_i$ and $[\sqrt{\gamma}\Phi]$ \[Back to [top](#toc)\]
$$\label{complete_construct_ai}$$

Now to set the evolution equations ([eqs. 17 and 19](https://arxiv.org/pdf/1704.00599.pdf)), recalling that the drift velocity $v^i = u^i/u^0$:
\begin{align}
\partial_t A_i &= \epsilon_{ijk} v^j B^k - \partial_i (\alpha \Phi - \beta^j A_j) \\
               &= \epsilon_{ijk} \frac{u^j}{u^0} B^k - {\rm AevolParen\_dD[i]} \\
\partial_t [\sqrt{\gamma} \Phi] &= -\partial_j \left(\left(\alpha\sqrt{\gamma}\right)A^j - \beta^j [\sqrt{\gamma} \Phi]\right) - \xi \alpha [\sqrt{\gamma} \Phi] \\
                                &= -{\rm PevolParenU\_dD[j][j]} - \xi \alpha [\sqrt{\gamma} \Phi]. \\
\end{align}_____no_output_____

# Step 4.0.b: Construct the actual evolution equations for A_i and sqrt(gamma)Phi
A_rhsD = ixp.zerorank1()
psi6Phi_rhs = sp.sympify(0)

# We already have a handy function to define the Levi-Civita symbol in WeylScalars
import WeylScal4NRPy.WeylScalars_Cartesian as weyl
# Initialize the Levi-Civita tensor by setting it equal to the Levi-Civita symbol
LeviCivitaSymbolDDD = weyl.define_LeviCivitaSymbol_rank3()
LeviCivitaTensorDDD = ixp.zerorank3()
#LeviCivitaTensorUUU = ixp.zerorank3()

for i in range(DIM):
    for j in range(DIM):
        for k in range(DIM):
            LeviCivitaTensorDDD[i][j][k] = LeviCivitaSymbolDDD[i][j][k] * sp.sqrt(gammadet)
            #LeviCivitaTensorUUU[i][j][k] = LeviCivitaSymbolDDD[i][j][k] / sp.sqrt(gammadet)

for i in range(DIM):
    A_rhsD[i] = -AevolParen_dD[i]
    for j in range(DIM):
        for k in range(DIM):
            A_rhsD[i] += LeviCivitaTensorDDD[i][j][k]*(uU[j]/u4upperZero)*BU[k]

psi6Phi_rhs = -xi*alpha*psi6Phi
for j in range(DIM):
    psi6Phi_rhs += -PevolParenU_dD[j][j]_____no_output_____

# Step 6:  Code Validation against `GiRaFFE_HO.GiRaFFE_Higher_Order_v2` NRPy+ Module \[Back to [top](#toc)\]
$$\label{code_validation}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for the $\texttt{GiRaFFE}$ evolution equations and auxiliary quantities we intend to use between
1. this tutorial and
2. the NRPy+ [GiRaFFE_HO.GiRaFFE_Higher_Order_v2](../edit/GiRaFFE_HO/GiRaFFE_Higher_Order_v2.py) module.
_____no_output_____

# Reset the list of gridfunctions, as registering a gridfunction
#   twice will spawn an error.
gri.glb_gridfcs_list = []

import GiRaFFE_HO.GiRaFFE_Higher_Order_v2 as gho
gho.GiRaFFE_Higher_Order_v2()

print("Consistency check between GiRaFFE_Higher_Order tutorial and NRPy+ module: ALL SHOULD BE ZERO.")

print("AevolParen - gho.AevolParen = " + str(AevolParen - gho.AevolParen))
print("psi6Phi_rhs - gho.psi6Phi_rhs = " + str(psi6Phi_rhs - gho.psi6Phi_rhs))
for i in range(DIM):
    print("PevolParenU["+str(i)+"] - gho.PevolParenU["+str(i)+"] = " + str(PevolParenU[i] - gho.PevolParenU[i]))
    print("Stilde_rhsD["+str(i)+"] - gho.Stilde_rhsD["+str(i)+"] = " + str(Stilde_rhsD[i] - gho.Stilde_rhsD[i]))
    print("A_rhsD["+str(i)+"] - gho.A_rhsD["+str(i)+"] = " + str(A_rhsD[i] - gho.A_rhsD[i]))
    for j in range(DIM):
        print("SevolParenUD["+str(i)+"]["+str(j)+"] - gho.SevolParenUD["+str(i)+"]["+str(i)+"] = "\
              + str(SevolParenUD[i][j] - gho.SevolParenUD[i][j]))Consistency check between GiRaFFE_Higher_Order tutorial and NRPy+ module: ALL SHOULD BE ZERO.
AevolParen - gho.AevolParen = 0
psi6Phi_rhs - gho.psi6Phi_rhs = 0
PevolParenU[0] - gho.PevolParenU[0] = 0
Stilde_rhsD[0] - gho.Stilde_rhsD[0] = 0
A_rhsD[0] - gho.A_rhsD[0] = 0
SevolParenUD[0][0] - gho.SevolParenUD[0][0] = 0
SevolParenUD[0][1] - gho.SevolParenUD[0][0] = 0
SevolParenUD[0][2] - gho.SevolParenUD[0][0] = 0
PevolParenU[1] - gho.PevolParenU[1] = 0
Stilde_rhsD[1] - gho.Stilde_rhsD[1] = 0
A_rhsD[1] - gho.A_rhsD[1] = 0
SevolParenUD[1][0] - gho.SevolParenUD[1][1] = 0
SevolParenUD[1][1] - gho.SevolParenUD[1][1] = 0
SevolParenUD[1][2] - gho.SevolParenUD[1][1] = 0
PevolParenU[2] - gho.PevolParenU[2] = 0
Stilde_rhsD[2] - gho.Stilde_rhsD[2] = 0
A_rhsD[2] - gho.A_rhsD[2] = 0
SevolParenUD[2][0] - gho.SevolParenUD[2][2] = 0
SevolParenUD[2][1] - gho.SevolParenUD[2][2] = 0
SevolParenUD[2][2] - gho.SevolParenUD[2][2] = 0

# Step 7: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-GiRaFFE_Higher_Order_v2.pdf](Tutorial-GiRaFFE_Higher_Order_v2.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)_____no_output_____

!jupyter nbconvert --to latex --template latex_nrpy_style.tplx Tutorial-GiRaFFE_Higher_Order_v2.ipynb
!pdflatex -interaction=batchmode Tutorial-GiRaFFE_Higher_Order_v2.tex
!pdflatex -interaction=batchmode Tutorial-GiRaFFE_Higher_Order_v2.tex
!pdflatex -interaction=batchmode Tutorial-GiRaFFE_Higher_Order_v2.tex
!rm -f Tut*.out Tut*.aux Tut*.log[NbConvertApp] Converting notebook Tutorial-GiRaFFE_Higher_Order_v2.ipynb to latex
[NbConvertApp] Writing 76275 bytes to Tutorial-GiRaFFE_Higher_Order_v2.tex
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode


# Notebook from willfrank98/3100-Models_of_Computation
Path: asgjove/asg1/J1_U1067292.ipynb

# Asg1 problems for you to solve

 _____no_output_____

from jove.DotBashers import *
from jove.Def_md2mc import *
from jove.Def_DFA   import *
from jove.LangDef   import *_____no_output_____

# Q8_____no_output_____

Le = {"0"*(2*i) for i in range(6)}_____no_output_____Lo = {"0"*(2*i + 1) for i in range(5)}_____no_output_____star = lstar({'0'}, 10)_____no_output_____union = lunion(Le, Lo)_____no_output_____print(star == union) #python provides functionality for checking if two sets contain the same elements
print(lint(star, union) == star and lint(star, union) == union) #using lint to check if the sets completely overlap each otherTrue
True

# Q9_____no_output_____## Q9(a) Listing strings in numeric order_____no_output_____

help(nthnumeric)Help on function nthnumeric in module jove.LangDef:

nthnumeric(N, Sigma={'b', 'a'})
    Assume Sigma is a 2-sized list/set of chars (default {'a','b'}).
    Produce the Nth string in numeric order, where N >= 0.
    Idea : Given N, get b = floor(log_2(N+1)) - need that
    many places; what to fill in the places is the binary
    code for N - (2^b - 1) with 0 as Sigma[0] and 1 as Sigma[1].

firstTwelve = {nthnumeric(i) for i in range(12)}
print(firstTwelve){'', 'aa', 'a', 'abb', 'bab', 'ab', 'bba', 'baa', 'b', 'bb', 'ba', 'bbb'}

## Q9(b) Counting strings in L1L2_____no_output_____

Sigma1 = {'a', 'b', 'c'}
Sigma2 = {'0', '1'}

L1 = lstar(Sigma1, 8)
L2 = lstar(Sigma2, 8)

L1L2 = lcat(L1, L2)

#..now check that the len(L1L2) makes sense to you..
#..how must len(L1L2) relate to len(L1) and len(L2)? ..
#..introduce the required code cells below and show that the expected identity holds..
print(len(L1L2) == (len(L1) * len(L2)))True

# Q10_____no_output_____

# Essential part of a DFA for even # of 0s

D10 = md2mc('''
DFA
I : 0 -> BH !! no acceptable strings start with 0
BH : 0 -> BH
BH : 1 -> BH
BH : 2 -> BH
I : 1 -> F1 !! send strings start with 1 to odd length checking
F1 : 0 -> S1 !! if the DFA ends at S1 the string is even length
F1 : 1 -> S1
F1 : 2 -> S1
S1 : 0 -> F1 !! if the DFA ends at F1 the string is odd length
S1 : 1 -> F1
S1 : 2 -> F1
I : 2 -> S2 !! send strings starting with 2 to even length checking
S2 : 0 -> F2 !! if the DFA ends at F2 the string is even length
S2 : 1 -> F2
S2 : 2 -> F2
F2 : 0 -> S2 !! if the DFA ends at S2 the string is odd length
F2 : 1 -> S2
F2 : 2 -> S2
''')

#..use the right dotObj command of the DFA with the right FuseEdges boolean value..
dotObj_dfa_w_bh(D10, True)_____no_output_____

# Show it accepts only the requested strings_____no_output_____

accepts_dfa(D10,"111")_____no_output_____TestStrs = input = lstar({'0', '1', '2'}, 4)_____no_output_____TestStrs_____no_output_____# Test the DFA

for x in TestStrs :
    if accepts_dfa(D10, x) :
        print(x, "was accepted")

print("All other strings are rejected")2201 was accepted
120 was accepted
102 was accepted
2220 was accepted
22 was accepted
110 was accepted
2211 was accepted
2021 was accepted
2001 was accepted
100 was accepted
2212 was accepted
2120 was accepted
2011 was accepted
2111 was accepted
2222 was accepted
2110 was accepted
112 was accepted
20 was accepted
2002 was accepted
2102 was accepted
2101 was accepted
2122 was accepted
2000 was accepted
101 was accepted
2010 was accepted
122 was accepted
2100 was accepted
2012 was accepted
21 was accepted
2022 was accepted
121 was accepted
2020 was accepted
2112 was accepted
2210 was accepted
1 was accepted
2221 was accepted
111 was accepted
2200 was accepted
2121 was accepted
2202 was accepted
All other strings are rejected


# Notebook from abhisekp/Reference-Guide-For-Quantum-Computing-A-Microsoft-Garage-Project
Path: 1-Basic_Quantum_Concepts/2-Gates.ipynb

# 1.2 Gates

* [Unitary matrix](./1-Basic_Quantum_Concepts/2-Gates.ipynb#unitarymatrix)
* [The Control-NOT Gate (CNOT)](./1-Basic_Quantum_Concepts/2-Gates.ipynb#cnot)
    * [Q# exercise: CNOT gate](./1-Basic_Quantum_Concepts/2-Gates.ipynb#qcnot)
* [Circuit Representation](./1-Basic_Quantum_Concepts/2-Gates.ipynb#circuit)
* [Hadamard Gate](./1-Basic_Quantum_Concepts/2-Gates.ipynb#hadamard)
    * [Q# exercise: H gate](./1-Basic_Quantum_Concepts/2-Gates.ipynb#qh)
* [Bloch Sphere](./1-Basic_Quantum_Concepts/2-Gates.ipynb#boch)
* [Pauli Gates](./1-Basic_Quantum_Concepts/2-Gates.ipynb#pauli)
    * [Q# exercise: Pauli gates](./1-Basic_Quantum_Concepts/2-Gates.ipynb#qpauli)
* [Measurement](./1-Basic_Quantum_Concepts/2-Gates.ipynb#measurement)
    * [Q# exercise: Measurement using M operation](./1-Basic_Quantum_Concepts/2-Gates.ipynb#qmeasurement)
* [Interference](./1-Basic_Quantum_Concepts/2-Gates.ipynb#interference)

### Unitary matrix

Since qubit states are represented with vectors, the way to manipulate them is through matrix multiplications, which transforms a vector into a different vector. Such matrix operations applied onto the states are called "gates". In quantum mechanics, matrices representing gates are required to be unitary (see _Math insert - unitary, adjoint or Hermitian conjugate_ ). This is to ensure normalization condition, i.e. probabilities add up to 1, mentioned in Section 1.1. Unitarity also constraints the operation to be reversible. This is intuitive as one can find an operation that reverts the gate and returns a system back to its original state.

![Figure 1.2 Gate](media/gate.png)

_Math insert – unitary, adjoint or Hermitian conjugate_ --------------------------------------------

>A matrix $U$ is unitary if $U^{\dagger}U=I$, where $I$ is the identity matrix and the "$\dagger$" symbol (reads
>"dagger") means adjoint or Hermitian conjugate of matrix $U$. It can be further written as
>$U^{\dagger}=(U*)^{T}=(U^{T})*$, where "$T$" denotes transpose and "*" complex conjugate:
>
>$\begin{pmatrix} U_1 \\ U_2 \\...\\U_N \end{pmatrix}^T = \begin{pmatrix} U_1 & U_2 & ... & U_N \end{pmatrix}$
>and if $a=a_0+ia_1$, then $a*=a_0-ia_1$ by definition. Therefore,
>$\begin{pmatrix} a & b \\c & d \end{pmatrix}^{\dagger}=\begin{pmatrix} a* & b* \\c* & d* \end{pmatrix}$.
>
>Let's take a complex number $c=a+ib$; its complex conjugate will be $c*=a-ib$. Multiplying them:
>$c*c=a^2+b^2=|c|^2$.
>
>Now, let's take a matrix $x$ as follows:
>
$x=\begin{bmatrix} a_1+ib_1 \\a_2+ib_2 \\a_3+ib_3 \\a_4+ib_4 \end{bmatrix}$.
>
>The adjoint $x^{\dagger}$ will be:
>$x^{\dagger}=\begin{bmatrix} a_1-ib_1 & a_2-ib_2 & a_3-ib_3 & a_4-ib_4 \end{bmatrix}$.
>
>Multiplying them:
>$x^{\dagger}x=\begin{bmatrix} a_1-ib_1 & a_2-ib_2 & a_3-ib_3 & a_4-ib_4 \end{bmatrix}*\begin{bmatrix} a_1+ib_1 \\a_2+ib_2 \\a_3+ib_3 \\a_4+ib_4 \end{bmatrix}$.
>$=[a_1^2+b_1^2+a_2^2+b_2^2+a_3^2+b_3^2+a_4^2+b_4^2]$.
>
>If $x$ represents an amplitude matrix in a 2-qubit system, then from the concepts you
>have learned from the previous sessions, we can conclude the following:
>$Pr(|00\rangle)=|a_1+ib_1|^2=a_1^2+b_1^2$
>$Pr(|01\rangle)=|a_2+ib_2|^2=a_2^2+b_2^2$
>$Pr(|10\rangle)=|a_3+ib_3|^2=a_3^2+b_3^2$
>$Pr(|11\rangle)=|a_4+ib_4|^2=a_4^2+b_4^2$.
>And $x^{\dagger}x= sum of probablities of all basis states = 1$.
>
>Now, let's say we apply a gate whose matrix is $U$ on $x$ and the resultant state is
>$y$ as follows:
>$y=Ux$
>Applying conjugate transpose on both sides:
>$y^{\dagger}y=(Ux)^{\dagger}$
>After multiplication:
>$y^{\dagger}y=(Ux)^{\dagger}Ux$
>$=x^{\dagger}U^{\dagger}Ux$ (because $(AB)^{\dagger}=B^{\dagger}A{\dagger}$; you can easily find the proog for this online).
>
>Now, if $U$ is unitary, then by definition $U^{\dagger}=U^{-1}$. It implies:
>$y^{\dagger}y=x^{\dagger}A^{-1}Ax$,
>$y^{\dagger}y=x^{\dagger}Ix$,
>$y^{\dagger}y=x^{\dagger}x$,
>$y^{\dagger}y=1$.
>
>It implies that $y$ is also a valid quantum state. This is possible only because we assume
>that $U^{\dagger}=U^{-1}$ ($U$ is unitary). The bottom line here is that all the matrices (gates) we
>use in quantum computing must be unitary.

### The Control-NOT Gate (CNOT)

Let's introduce a commonly used gate, CNOT. One may know that NOT gate in classical computing is an operation that flips 0s and 1s. The CNOT gate acts on a two-qubit state. The two-qubit states is represented by a four-element vector, with the first qubit represented by the first two elements of the vector, and the second qubit by the last two elements. A CNOT gate flips the second qubit based on the state of the first qubit. Thus, the first qubit is called the "control" qubit, while the second one is called the "target" qubit. The CNOT gate takes the form of a 4 x 4 matrix.

$CNOT=\begin{bmatrix} 1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\ 0 & 0 & 0 & 1\\ 0 & 0 & 1 & 0 \end{bmatrix}$.
One can see that the first quadrant is a 2 x 2 identity matrix. It keeps the control qubit intact after
multiplication. The fourth quadrant is the 2 x 2 NOT gate. It flips the target qubit from "0" to be "1" and
from "1" to be "0" if the control qubit is "1" but does nothing if the control qubit is "0". Here's an example
to examine this behavior.

$CNOT|10\rangle = \begin{bmatrix} 1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 0 & 1 \\0 & 0 & 1 & 0 \end{bmatrix} \begin{pmatrix} 0 \\ 0 \\ 1 \\ 0 \end{pmatrix}$
$=\begin{pmatrix} 0 \\ 0 \\ 0 \\ 1 \end{pmatrix} = |11\rangle$. eq. (1.2.1)

Similarly, $C|00\rangle=|00\rangle$, $C|01\rangle=|01\rangle$ and $C|11\rangle=|10\rangle$. Use _Math insert – matrix multiplication_ to show
these equations.

_Math insert - matrix multiplication_ -------------------------------------------------------------------
>
>Gates are N by N matrices that multiply to state with $2^N$ vector elements. They follow
>the rules such that
>$\begin{bmatrix} a & b \\c & d \end{bmatrix}\begin{pmatrix} x \\y \end{pmatrix}=\begin{pmatrix} ax+by \\ cx+dy \end{pmatrix}$,
>$\begin{bmatrix} a & b & c \\d & e & f \\g & h & i \end{bmatrix}\begin{pmatrix} x \\ y \\ z \end{pmatrix} = \begin{pmatrix} ax+by+cs \\dx+ey+fz \\gx+hy+iz \end{pmatrix}$,
>and so on.

### Q# exercise: CNOT gate

Local:
1. Go to Quantum Katas: https://github.com/Microsoft/QuantumKatas Download or clone.

2. In Visual Studio (Code) open folder "BasicGates". The CNOT exercise is in Tasks.qs

3. Read about CNOT in Microsoft.Quantum.Intrinsic Reference Documentation
**Link:** https://docs.microsoft.com/en-us/qsharp/api/qsharp/microsoft.quantum.intrinsic

4. The command for CNOT is

operation CNOT (control : Qubit, target : Qubit) : ()

5. Don't change anything and run dotnet test in the integrated terminal. This should automatically build the kata project and run all unit tests; initially, all unit tests should fail. You should get

![Figure CNOT Gate1](media/CNOTgate_excercise.png)

6. Open Tasks.qs file. I n "Task 2.1. Two-qubit gate – 1" in the body, add the appropriate command inside the operation and run dotnet test again.

![Figure CNOT Gate1](media/CNOTgate_excercise2.png)

7. If your answer is correct, you should get

![Figure CNOT Gate1](media/CNOTgate_excercise3.png)

8. The Katas has many projects in this exercise. Feel free to try out the other ones. As you answer more questions correctly, the number of passed tests will increase accordingly.

Jupyter Notebook:

1. Go to https://github.com/microsoft/QuantumKatas/tree/master/BasicGates and follow link to Jupyter Notebook to the BasicGates exercise.

2. Run the package and jump to Task 2.1. It is the same exercise as above.

### Circuit Representation

Matrix calculations can sometimes get messy. To make quantum operations easier to follow, a graphic representation below is used to illustrate quantum operations. Because the diagram looks like a circuit, it is called a circuit representation. A qubit is denoted by a horizontal line:

![Figure Circuit Representation of Qubits](media/circuitrepresentationqubit.png)

Figure 1.2.1 Circuit representation of qubits.

The purpose of using horizontal lines is to provide a visualization of qubits' evolution over time. Individual qubits are placed in parallel so that gates can connect them vertically in the diagram. What happens to each qubit is apparent. For example, CNOT introduced above can be captured in the following circuit diagram:

![Figure Circuit Representation of CNOT Gate](media/circuitrepresentationcnot.png)

Figure 1.2.2 Circuit representation of CNOT gate.

Control qubit |$A$⟩ is unchanged after CNOT but target qubit |$B$⟩ becomes |$B$⊕$A$⟩ based on the control.

### Hadamard Gate

Another frequently used gate is the Hadamard gate, H. Its matrix form is written as

$H = \begin{bmatrix} \frac{1}{\sqrt{2}} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} {-\frac{1}{\sqrt{2}}} \end{bmatrix}$

The Hadamard gate puts states |0⟩ and |1⟩ into a superposition of the two with equal probability, i.e.

$H|0⟩ = \begin{bmatrix} \frac{1}{\sqrt{2}} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} {-\frac{1}{\sqrt{2}}} \end{bmatrix} \begin{pmatrix} 1 \\ 0 \end{pmatrix}$

$= \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \end{pmatrix} = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 \\ 0 \end{pmatrix} + \frac{1}{\sqrt{2}} \begin{pmatrix} 0 \\ 1 \end{pmatrix}$

$= \frac{1}{\sqrt{2}} \begin{pmatrix} |0⟩ + |1⟩ \end{pmatrix} ≡ |+⟩ $, eq. (1.2.2)

and

$H|1⟩ = \begin{bmatrix} \frac{1}{\sqrt{2}} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} {-\frac{1}{\sqrt{2}}} \end{bmatrix} \begin{pmatrix} 0 \\ 1 \end{pmatrix}$

$= \frac{1}{\sqrt{2}} \begin{pmatrix} |0⟩ + |1⟩ \end{pmatrix} ≡ |-⟩ $, eq. (1.2.3)

The symbol "≡" here means "defined as" or "equivalent to". In other words, the Hadamard gate converts classical states, i.e. |0⟩ and |1⟩, into quantum states |+⟩ and |−⟩. It also converts quantum states quantum states |+⟩ and |−⟩ back to classical states |0⟩ and |1⟩. Readers should prove $H$|+⟩= |0⟩ and $H$|−⟩= |1⟩ as an exercise. The negative sign in the fourth element in the $H$ matrix highlighted in eq. (1.2.3) has the significance. It makes the operation reversible. Without it, applying $H$ to the states |+⟩ and |−⟩ would give indistinguishable results. The circuit representation of a Hadamard gate is:

![Figure Circuit Representation of Hadamard Gate](media/circuitrepresentationhadamard.png)

Figure 1.2.3 Circuit representation of the Hadamard gate.

It acts on a single qubit.

### Q# exercise: H gate

1. In Quantum Katas Basic Gate is designed to implement H gate. Try the exercise on Jupiter Notebook.

2. To further your familiarity, let's do another exercise in QuantumComputingViaQSharpSolution introduced in session 1.1.

3. Open 07_Demo H Operation.qs in Visual Studio (Code) and run dotnet run.

4. "M" here is a measurement operation. We will talk about it at the end of the session.

M(qubits$[0]$)

5. If the measured result is 0, it'll output "False"; if "1", it'll output "true". Run the script several times, you'll see the output changes randomly because H puts the qubit into a state with equal probability of getting "0" and "1".

6. Also run 07_Demo H count-mutable-set-for-ToStringI. Half of the time, you will measure the state to be in |0⟩ and half of the time in |1⟩.

### Bloch Sphere

There is yet another graphic representation to visualize a qubit. It gives more information about a qubit than the circuit representation. It is called a Bloch sphere - a 3D sphere drawn in 2D with the North and South poles representing states |0⟩ and |1⟩, respectively.

![Figure Bloch Sphere States Zero and One](media/blochspherezeroone.png)

Figure 1.2.4 Bloch sphere with states |0⟩ and |1⟩ labelled by red dots.

A Hadamard gate applied to the state |0⟩ or |1⟩ moves the state to |+⟩ or |−⟩ which is a superposition

of |0⟩ and |1⟩ with the same amplitude $\frac{1}{\sqrt{2}}$. This means the state |+⟩ or |−⟩ is on the equator on the Bloch sphere.

![Figure Bloch Sphere States Plus Minus](media/blochsphereplusminus.png)

Figure 1.2.5 Bloch sphere with states |0⟩ and |1⟩ labelled by red dots and states |+⟩ and |−⟩ by blue
dots.

This is illustrating eq. (1.2.2 and 1.2.3) graphically on a Bloch sphere. It looks as if the states are rotated along
the sphere about the y-axis by 90 degrees. But it is actually not a one-step rotation. This will be explained
later in Pauli Gates.

In general, an arbitrary one-qubit state $|\psi⟩$ can be represented by a point anywhere on a Bloch sphere. $|\psi⟩$ is on the sphere with angle $\phi$ away from the x-axis and angle $\theta$ away from the z-axis. Thus it can be written as |$\psi⟩=\cos\frac{\theta}{2}|0⟩+e^{-i\phi}\sin\frac{\theta}{2}|1⟩$, which is a superposition of states |0⟩ and |1⟩ with the angles defining the amplitudes. This further makes it clear that the states |0⟩ and |1⟩ are just two special cases with $\theta$ = 0° and 180°, respectively.

![Figure Bloch Sphere Arbitrary](media/blochspherearbitrary.png)

Figure 1.2.6 Bloch sphere with an arbitrary state $|\psi⟩$ labelled by a red dot.

### Pauli Gates

Since a general one-qubit state can be represented on a Bloch sphere with relative angles to the axes,
what the gates do is to changes the angles, altering the position on the sphere that represents the state.
We introduce three special rotational gates below named after physicist Wolfgang Pauli.

Pauli X gate is the NOT gate that rotates the state about the x-axis by 180 degrees.

$X = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}$,

so that

$X\begin{pmatrix} \alpha \\ \beta \end{pmatrix} = \begin{pmatrix} \beta \\ \alpha \end{pmatrix}$.

It has the effect of swapping the coefficients, $\alpha$ and $\beta$ (Figure 1.2.7). Thus, a |0⟩ state "flips" to a |1⟩ and a
|1⟩ state "flips" to a |0⟩.

Pauli Y gate is a rotation about the y-axis by 180°. Not only does it flip 0s and 1s, the complex numbers $i$ and $-i$ change the phase of the vector (Figure 1.7).

$Y = \begin{bmatrix} 0 & -i \\ i & 0 \end{bmatrix}$,

so that

$Y\begin{pmatrix} \alpha \\ \beta \end{pmatrix} = i\begin{pmatrix} -\beta \\ \alpha \end{pmatrix}$.

Pauli Z gate rotates about the z-axis by 180°. Without the minus sign in the fourth element, it
would be an identity matrix, but the minus sign provides a phase shift to |1⟩ (Figure 1.2.7).

$Z = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}$,

so that

$Z\begin{pmatrix} \alpha \\ \beta \end{pmatrix} = \begin{pmatrix} \alpha \\ -\beta \end{pmatrix}$.

![Figure Bloch Sphere Pauli Interactions](media/blochspherepauli.png)

Figure 1.2.7 Bloch sphere representation of the Pauli gate interactions. Graph copied from Microsoft quantum computing.

In general, rotation gates, R, about an axis can be described by the angles $\phi$ and $\theta$ in Figure 1.2.6:

$R_z(\phi)=\begin{pmatrix} e^{i\phi/2} & 0 \\ 0 & e^{-i\phi/2} \end{pmatrix}$,

$R_y(\theta)=\begin{pmatrix} \cos\frac{\theta}{2} & -\sin\frac{\theta}{2} \\ \sin\frac{\theta}{2} & \cos\frac{\theta}{2} \end{pmatrix}$,

and

$R_x(\theta)=\begin{pmatrix} \cos\frac{\theta}{2} & -i\sin\frac{\theta}{2} \\ -i\sin\frac{\theta}{2} & \cos\frac{\theta}{2} \end{pmatrix}$

$= R_z(\frac{\pi}{2})R_y(\theta)R_z(-\frac{\pi}{2})$

In fact, any arbitrary single quantum logic gate can be decomposed into a series of rotation matrices:

$U=e^{i\gamma} \begin{bmatrix} e^{-i\phi/2} & 0 \\ 0 & e^{i\phi/2} \end{bmatrix} \begin{bmatrix} \cos(\frac{\theta}{2}) & -\sin(\frac{\theta}{2}) \\ \sin(\frac{\theta}{2}) & \cos(\frac{\theta}{2}) \end{bmatrix}$

with the only constraint on the gate being unitary. Here, $e^{i\gamma}$ is a global phase shift that can be added without affecting the behavior.

Now, if we revisit the Hadamard gate, it can be decomposed as a rotation about the x-axis by 180° followed by a clockwise rotation about the y-axis by 90°. The negative sign in the Hadamard gate determines the direction of rotation. |+⟩ and |−⟩ have opposite signs that put them on opposite sides of the Bloch sphere.

![Figure Bloch Sphere Hadamard](media/blochspherehadamard.png)

Figure 1.2.8 Hadamard gate on Block sphere using |0⟩ as the initial state as an example. Graph copied from Microsoft quantum computing.

The circuit representation of the Pauli gates and general rotation gates are

![Figure Bloch Sphere Hadamard](media/circuitrepresentationpauli.png)

Figure 1.2.9 Circuit representation of the Pauli X, Y, Z and R gates. Pauli X gate is the same as a NOT gate.

### Q# exercise: Pauli gates

1. Go to Quantum Katas: https://github.com/Microsoft/QuantumKatas introduced earlier in the CNOT excercise.

2. In Visual Studio (Code) open folder "BasicGates" and then Tasks.qs. Or go to Basic Gates in Quantum Katas Jupiter Notebook.

3. Try Task 1.1, 1.3, 1.4 and 1.6.

4. Read more about other gates:

    https://docs.microsoft.com/en-us/qsharp/api/prelude/microsoft.quantum.primitive?view=qsharp-preview

### Measurement – not a gate

Measurements are different from gates. Gates are unitary which makes operations reversible. Gates acting upon a quantum system do not destroy the system or make connections between the system and the outside world. Applying a gate does not lose information. However, when we in the outside world want to obtain information from the system, we need to measure it. Measuring a system brings it into contact with the outside world, irrevocably destroying information. Thus, measurement is, in general, not reversible – the system cannot be brought back to its state before the measurement. One consequence of this difference is that a measurement cannot be executed by a gate. It is an interaction between the system and its environment. A parameter or physical variable measured from a quantum system is called an "observable" (because we can observe it). Momentum, mass, velocity and energy are all observables.

![Figure Measurement](media/measurement.png)

>_Physics insert – measurement_ -------------------------------------------------------------------------
>
>If we use the wavefunction approach (session 1.1), we can derive the value we would expect to measure for a large number of measurements of a given observable, $M$. The expectation >value can be obtained as
>
>$〈M〉=〈\psi|M|\psi〉=\sum_jm_j|c_j|^2$,
>
>for state $|M⟩=\sum_jc_j|\psi_j〉$,
>
>where $m_j$ is each measurement result of $M$, and $|c_j|^2$ = $P(m_j)$ is the probability of getting result $m_j$. Obtaining $m_j$ leaves the system in the state $|\psi_j〉$. This >unavoidable disturbance of the system caused by the measurement process is often described as a "collapse," a "projection" or a "reduction" of the wavefunction.
>
In our qubit system, recall that with two qubits (eq. (1.1.7) ), a state is described by a superposition of four basis states, i.e.

$|\psi⟩=c_{00}|00⟩+c_{01}|01⟩+c_{10}|10⟩+c_{11}|11⟩$

with coefficients, $c_{ij}$. Now, what happens if we do a measurement on the system – say, if we measure the first qubit and find it to be 0? There are two possible states with the first qubit being 0. The probability of getting a 0 in the first qubit is

$P=|c_{00}|^2 + |c_{01}|^2$.

After such a measurement, there are only two possible states that can exist in the system. The state becomes

$|\psi'⟩=\frac{c_{00}|00}{\sqrt{P}}$. eq. (1.2.8)

The denominator is there to keep the probability normalized (try the math yourself). A measurement projects a state $|\psi⟩$ into a basis state with a definite value of the observable (or "operator") being measured.

>_Math insert – Hermitian operator_ -----------------------------------------------------
>
>Recall that a gate is a unitary matrix, $U$, such that $U^\dagger U$=I or U^\dagger=U^{−1}. A measurement is not a unitary matrix. An observable corresponds to a Hermitian matrix, >$M$, such that $M^\dagger=M$. (Technically, the basis states that have a definite value of the observable $M$ are the "eigenstates" of $M$, and the observed values of the operator >are the "eigenvalues.") Operating with the matrix $M$ on an eigenstate yields a number $m$ times the eigenstate:
>$M|\psi⟩=m|\psi⟩$.
>
>This number, $m$, is the value of the observable $M$ that one would measure if the
>system were in state $|\psi〉$.
>The operator corresponding to an observable is required to be Hermitian, because measurement results or observables need to be real numbers. Here's the proof that the measurement >operator must be Hermitian:
>$〈M〉=〈\psi|M|\psi〉$
>If 〈$M$〉 is real, then $〈M〉^\dagger=〈M〉$. Therefore,
>$〈M〉^\dagger=〈\psi|M^\dagger|\psi〉=〈\psi|M|\psi〉$.
>This must hold for any state |$\psi$〉. Thus, $M^\dagger=M$.

### Q# exercise: Measurement using M operation

1. Go to Quantum Katas: https://github.com/Microsoft/QuantumKatas introduced earlier in the CNOT exercise.
2. In Visual Studio (Code) open folder "Measurements" and then Tasks.qs. Or use the Measurements exercise in Quantum Katas Jupyter Notebook.
3. Look at Task 1.1. It only needs an operation M(q) to measure the state of the qubit q.

![Figure Task 1.1](media/measurement_exercise.png)

4. Now use M(q) to finish as many tasks as you can. These tasks need other gates you learned in this session to put the qubits into the states you can measure. Tasks 1.1-1.4 all follow the same process. Task 1.4 is also an interesting exercise as one can use the Bloch Sphere to visualize where the arbitrary state is relative to the |0⟩ state. Then use the rotation gate to move it back to the |0⟩ basis. The later tasks require additional concepts that we have not covered. You can skip them or read further on your own.

### Interference

In the end of session 1.1, we introduced a generalized probability theory. The key is that probability is equal to the _square of magnitude of amplitude_, so the square of amplitude must be a positive number between 0 and 1. Here, we're going to look at the idea of interference between different possibilities, which is something that is typical of quantum mechanics, but not of classical probability theory. Interference occurs because we use amplitudes, rather than probabilities, and amplitudes can be negative or complex. Let's use a metaphorical example to illustrate this abstract idea.

Say Bob is someone who may like sports and/or computer games. He has a 70% chance of liking
sports ($S$) and a 30% chance of not liking sports ($\overline{S}$). If he likes sports, there is an 80% chance that he likes
computer games ($C$) and a 20% chance that he does not like computer games ($\overline{C}$). If he does not like sports,
there is a 10% chance he likes computer games, and a 90% chance that he doesn't. This situation is
depicted as a tree on the left of Figure 1.2.9.

What is the chance that Bob likes computer games? We can compute this by aid of the probability tree. There are two paths we can go down the tree to end up at $C:S \to C$ and $\overline{S} \to C$. We can multiply the branching probabilities down each path, and then add the paths together. In this case, the probability of $C$ is given by

$P(C) = 0.7 \times 0.8 + 0.3 \times 0.1 = 0. 59$, or 59%.

But what if Bob were a quantum system? In that case, we would work with amplitudes. The tree
on the right of Figure 1.2.9 shows one possible set of amplitudes, which give the same branching possibilities.
Instead of calculating the probability of Bob liking computer games, we'll first calculate the amplitude. If
we switch the word "probability" to "amplitude" in the above calculation, then everything is the same.
Because in a quantum system, the amplitude could be a positive, negative or a complex number, let's
assign a negative amplitude to one of the branches in Figure 1.2.9 to prove our point next. We find all the
paths to get to $C$, and multiply the amplitudes along the path, and then sum the amplitudes of the paths:

$a_c = \sqrt{0.7} \times \sqrt{0.8} - \sqrt{0.3} \times \sqrt{0.1}$.

To get the probability, we take the modulus squared of the amplitude:

$P(C) = |a_c|^2 \approx$ 0.548, or 54.8%.

The second path actually subtracts from the first path, causing the overall probability to decrease. Very
strangely, the fact that there's a second way for Bob to like computer games _decreases_ the probability
that he likes them! This is because the amplitude of the second path has the opposite sign from the
amplitude for the first path. We call this "destructive interference." If the amplitudes had the same sign,
the second path would reinforce the first, and we would get "constructive interference." Interference is
fundamentally caused by the fact that amplitudes can be negative, or even complex, allowing them to
cancel one another out. Interference is at the heart of much of the "strangeness" of quantum mechanics.
Conceptually, however, it's rather simple, as we can see from the trees in Figure 1.2.9.

![Figure Measurement](media/probabilitytrees.png)

Figure 1.2.9. Probability trees in classical probability theory according to 1-norm (left) and in quantum
mechanics according to 2-norm (right). We begin at the top of the tree, and ask, "Does Bob like sports ($S$)
or not like sports ($\overline{S}$)?" Then we get to the next bifurcation, and ask, "Does Bob like computer games ($C$).
or not like computer games ($\overline{C}$).?" On the left, we show the probabilities for each option. On the right, we
show the amplitudes.

![Figure Measurement](media/constructiveinterference.png)

In quantum mechanics, why do we multiply the amplitudes, rather than the probabilities? Let's
check with an arbitrary qubit state, $|psi⟩=c_0|0⟩+c_1|1⟩$. It has amplitude $c_0$ of being in |0⟩ and $c_1$ of being
in |1⟩. If we then want to measure the presence of |+⟩ and |−⟩ in $|\psi⟩$, what probably will be measured?
Knowing $c_j=〈\phi_j|\psi〉$ from _Physics insert – Superposition_ in session 1.1, we would do an overlap integral
$〈+|\psi〉$ and $〈−|\psi〉$ to get the corresponding amplitudes and square them to get the probabilities.

$c_{\pm}=〈\pm|\psi〉$

$c_0〈\pm|0〉 + c_1〈\pm|1〉$

$P(|\pm〉) = |c_{\pm}|^2$.

($\pm$ means + or −) The square happens after the amplitudes add up. This turns out to be equivalent to
multiplying down a 2-norm probability tree in Figure 1.2.10, consistent with our metaphorical example for
Bob above. Quantum states are intrinsically described with amplitudes.

![Figure Multiplying Amplitudes](media/multiplyingamplitudes.png)

Figure 1.2.10 Multiplying amplitudes down the 2-norm probability tress is equivalent to calculating the overlap integrals.

The 2-norm in quantum mechanics can reduce to 1-norm in classical probability theory when constructive interference (paths with amplitudes of the same complex phase) dominates and destructive interference (paths with amplitudes of different complex phases) eliminate each other. This is explained very nicely in Richard Feynman's lecture on quantum electrodynamics. We are more likely to get a particular measurement if there are many paths to it that constructively interfere.

Interference can also be seen in a gate operation (unitary matrix). As Scott Aaronson showed in his lecture (https://www.scottaaronson.com/democritus/lec9.html ), a randomizing matrix

$\begin{bmatrix} \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \end{bmatrix}$.

can give equal probability of getting |0⟩ and |1⟩. But if the starting point is |0⟩, applying the matrix twice would give positive amplitudes to both |1⟩ states in either path, but a negative amplitude to one of the |0⟩ states. Thus, states |1⟩ constructively interfere, contributing to the outcome, while the states |0⟩ destructively interfere and cancel each other out (Figure 1.2.11).

![Figure Applying Randomizing Matrix](media/applyingrandomizingmatrix.png)

Figure 1.2.11 The outcome of applying randomizing matrix to the states twice depends on the interference between the paths.

_____no_output_____


# Notebook from ravinpoudel/ml-training-site
Path: assets/nb-lessons/.ipynb_checkpoints/06_training_and_datasets-Copy1-checkpoint.ipynb

# Training and Data Sets
Author: Ravin Poudel_____no_output_____Partitioning Data Sets in to train and test. If we have only one data,how are we going to evaluate the build model. One way to generate moredata, adn use that for model validation. However, in such senarious you can split/partitioned the your one data, and then use one part to construct/train the model, and then use the remaining portion of the data for testing or evaluation the performace of your model.
But how to split a single data into two parts?
Convention: such as 75%/ 25% train/ test scheme
Convention: such as 90%/ 10% train/ test scheme

Key to rememeber: if the dataset is not normally distributed then spliting might get tricky? Specially is the case of soem of the genomcis data such as microbiome.

Randomization: So that our model is not bias towards to one kind of data, and represent the whole data.

Larger the training dataset, better the model we are going to make.
Larger the test dataset, more confidence we are in our model evaluation.

If we  have a large data sets, then
Convention: such as 75%/ 25% train/ test scheme
Convention: such as 90%/ 10% train/ test scheme

But if we have smaller data set, then we might need to think about other sophisticated method such as

    cross validatiion
    cross fold validation?

NOTE: Never train model on your test dataset.

What could happen?
Be suspesious: If you ever happen to have 100% accuracy __over-fitting__ in youe model with test data, be suspecious and double check if you have not used test dataset for traning your model.

Train data set: A train dataset is a data set used to __construct/train__ the model.

Test data set: A test set is a data set used to __evaluate__ the model developed from a training set._____no_output_____Train-Test

Crossfold Validation

Crossfold Validation

_____no_output_____If the model performs very well on the training data but poorly on the test data, then it’s overfit._____no_output_____Validation Set: Another Partition

_____no_output_____When we build a statistial or machine learning model, we usually fit the model on the traning dataset to make a predicitons on the new data or the data that has not been used to fit the model. Getting the new or general data to test the model predictions often requires more experiments. Howver, to get around this, in machine learning or statistical modeling, we can first devide the dataset that we have into two sets: one for fitting the model and the next one for testing the model.When we deivide the data we want find the balance between model fiting and predictibility.

_____no_output_____

import pandas as pd
from sklearn import datasets, linear_model
from sklearn.model_selection import train_test_split
from matplotlib import pyplot as plt_____no_output_____# Load the Diabetes dataset
columns = "age sex bmi map tc ldl hdl tch ltg glu".split() # Declare the columns names
diabetes = datasets.load_diabetes() # Call the diabetes dataset from sklearn
df = pd.DataFrame(diabetes.data, columns=columns) # load the dataset as a pandas data frame
y = diabetes.target # define the target variable (dependent variable) as y_____no_output_____df.shape_____no_output_____df.head()_____no_output_____# create training and testing vars
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2)
print (X_train.shape, y_train.shape)
print (X_test.shape, y_test.shape)(353, 10) (353,)
(89, 10) (89,)
# fit a model
lm = linear_model.LinearRegression()
model = lm.fit(X_train, y_train)
predictions = lm.predict(X_test)_____no_output_____predictions[0:5]_____no_output_____## The line / model
plt.scatter(y_test, predictions)
plt.xlabel("True Values")
plt.ylabel("Predictions")_____no_output_____print ("Score:", model.score(X_test, y_test))
Score: 0.4936764592574261

# Cross Validation

- K-Folds Cross Validation
- Leave One Out Cross Validation (LOOCV)
_____no_output_____#### K-Folds Cross Validation
In K-Folds Cross Validation, first we divide the dataset randomly into k subset/bins. One of the subset/bin is used to validate the model, whereas the rest of bins for training model. We repeat the process for multiple rounds, at each round the `data-fold` used for test and train are randomized._____no_output_____

_____no_output_____

Resources:
- https://towardsdatascience.com/train-test-split-and-cross-validation-in-python-80b61beca4b6
- https://scikit-learn.org/stable/modules/cross_validation.html
_____no_output_____