# Notebook from sagnik1511/Binary-Classification-in-Deep-Learning
Path: Car or Truck/Car_or_Truck.ipynb

Copy and paste the following 3 cells and run it to gather the data and unzip it without downloading it locally._____no_output_____

!pip install -q kaggle

!mkdir -p ~/.kaggle

!cp kaggle.json ~/.kaggle/      #   (!cp [   API token file name   ] ~/.kaggle/)
## paste your kaggle API JSON file in the directory before running code cells

!chmod 600 ~/.kaggle/kaggle.json_____no_output_____!kaggle datasets download -d ryanholbrook/car-or-truckcar-or-truck.zip: Skipping, found more recently modified local copy (use --force to force download)
from zipfile import ZipFile
zf = ZipFile('car-or-truck.zip', 'r')
zf.extractall('/content')
zf.close()_____no_output_____

# Libraries :
---
As this is a deep learning binary classification task we need some data rocessing libraries , and the deep learning framworks and the confusion matrix plotter libraries._____no_output_____

# for data processing

import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import pandas as pd
import os

# for data pipeline --------------------

from sklearn.model_selection import train_test_split
from sklearn.metrics import*
import matplotlib.pyplot as plt
from sklearn.model_selection import*
from skimage import data, color
from skimage.transform import rescale, resize, downscale_local_mean

#deep learning libraries

import tensorflow as tf
from tensorflow.keras import Sequential

from tensorflow.keras.layers import Dense,Conv2D,MaxPool2D,Flatten,Dropout,BatchNormalization,GlobalAveragePooling2D
from tensorflow.keras.optimizers import Adam
print('version of tensorflow',tf.__version__)

# for confusion matrix plotting
from mlxtend.plotting import plot_confusion_matrix
from sklearn.metrics import multilabel_confusion_matrix,confusion_matrixversion of tensorflow 2.4.0

# Primary data visualization:
---_____no_output_____

os.listdir('train/')_____no_output_____os.listdir('valid/')_____no_output_____for x in ['train/','valid/']:
  for l in ['Truck/','Car/']:
    for ll in os.listdir(x+l):
      link=x+l+ll
      img=mpimg.imread(link)
      plt.title(l[:-1])
      plt.imshow(img)
      plt.show()
      break
    _____no_output_____

Dataframe Generator :
---_____no_output_____

x='train/'
arr=[]
df=[]
for l in ['Truck/','Car/']:
  for ll in os.listdir(x+l):
    link=x+l+ll
    img=mpimg.imread(link)
    if l== 'Truck/':
      arr.append(int(1))
    else :
      arr.append(int(0))
    img=resize(img,(64,64),anti_aliasing=True)  # resizes the images in 64x64 shape
    img=img.reshape(4096*3)    # flattens the image to append in the array
    df.append(img)
df=pd.DataFrame(df)
arr=pd.DataFrame({'label':arr})

df=pd.concat([arr,df],axis=1)      # creates the final dataframe
df.head()_____no_output_____train=df_____no_output_____x='valid/'
arr=[]
df=[]
for l in ['Truck/','Car/']:
  for ll in os.listdir(x+l):
    link=x+l+ll
    img=mpimg.imread(link)
    if l== 'Truck/':
      arr.append(int(1))
    else :
      arr.append(int(0))
    img=resize(img,(64,64),anti_aliasing=True)
    img=img.reshape(4096*3)
    df.append(img)
df=pd.DataFrame(df)
arr=pd.DataFrame({'label':arr})

df=pd.concat([arr,df],axis=1)
df.head()_____no_output_____test=df_____no_output_____

# Trainable data processing :
---_____no_output_____

train.shape_____no_output_____test.shape_____no_output_____X_train=train.drop('label',1)
y_train=train['label']_____no_output_____X_test=test.drop('label',1)
y_test=test['label']

X_test,X_val,y_test,y_val=train_test_split(X_test,y_test,random_state=0,test_size=0.5,stratify=y_test)

# splits the dataframe into test and validation for checking overfitting/ underfitting nature of model

X_test.shape,X_val.shape,y_test.shape,y_val.shape_____no_output_____y_train.value_counts()_____no_output_____y_val.value_counts()_____no_output_____y_test.value_counts()_____no_output_____

we can see the car and trucks are quite same in all the dataframes and this can help us gather better model performance._____no_output_____

X_train=X_train.to_numpy()
X_val=X_val.to_numpy()
X_test=X_test.to_numpy()
y_train=y_train.to_numpy()
y_val=y_val.to_numpy()
y_test=y_test.to_numpy()_____no_output_____X_train=X_train.reshape(5117,64,64,3)
X_val=X_val.reshape(2526,64,64,3)
X_test=X_test.reshape(2525,64,64,3)   # changing them into 4d to feed into the model.
y_train=y_train.reshape(5117,1)
y_val=y_val.reshape(2526,1)
y_test=y_test.reshape(2525,1)_____no_output_____

# Model Generation and Compiling :
---_____no_output_____

model=Sequential()
model.add(Conv2D(32,(3,3),activation='relu',input_shape=X_train[0].shape))
model.add(Conv2D(32,(3,3),activation='relu'))
model.add(MaxPool2D(2,2))
model.add(Dropout(0.25))

model.add(Conv2D(64,(3,3),activation='relu'))
model.add(Conv2D(64,(3,3),activation='relu'))
model.add(MaxPool2D(2,2))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(128,activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1,activation='sigmoid'))_____no_output_____model.summary()Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_8 (Conv2D)            (None, 62, 62, 32)        896
_________________________________________________________________
conv2d_9 (Conv2D)            (None, 60, 60, 32)        9248
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 30, 30, 32)        0
_________________________________________________________________
dropout_6 (Dropout)          (None, 30, 30, 32)        0
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 28, 28, 64)        18496
_________________________________________________________________
conv2d_11 (Conv2D)           (None, 26, 26, 64)        36928
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 13, 13, 64)        0
_________________________________________________________________
dropout_7 (Dropout)          (None, 13, 13, 64)        0
_________________________________________________________________
flatten_2 (Flatten)          (None, 10816)             0
_________________________________________________________________
dense_4 (Dense)              (None, 128)               1384576
_________________________________________________________________
dropout_8 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 129
=================================================================
Total params: 1,450,273
Trainable params: 1,450,273
Non-trainable params: 0
_________________________________________________________________
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['accuracy'])_____no_output_____

Model Fitting :
---

1. Batch size : 20
2. Epoch : 10
3. Optimizer : Adam
4. Loss function : BCE-Loss
5. Metrics : Accuracy
7. Validation set taken : YES_____no_output_____

hist=model.fit(X_train,y_train,batch_size=20,epochs=10,verbose=1,validation_data=(X_val,y_val))Epoch 1/10
256/256 [==============================] - 94s 363ms/step - loss: 0.6780 - accuracy: 0.5809 - val_loss: 0.6465 - val_accuracy: 0.6271
Epoch 2/10
256/256 [==============================] - 92s 360ms/step - loss: 0.6385 - accuracy: 0.6293 - val_loss: 0.6087 - val_accuracy: 0.6631
Epoch 3/10
256/256 [==============================] - 92s 360ms/step - loss: 0.5995 - accuracy: 0.6783 - val_loss: 0.5516 - val_accuracy: 0.7257
Epoch 4/10
256/256 [==============================] - 93s 364ms/step - loss: 0.5607 - accuracy: 0.7253 - val_loss: 0.5141 - val_accuracy: 0.7589
Epoch 5/10
256/256 [==============================] - 93s 362ms/step - loss: 0.4845 - accuracy: 0.7781 - val_loss: 0.4260 - val_accuracy: 0.8084
Epoch 6/10
256/256 [==============================] - 92s 358ms/step - loss: 0.4129 - accuracy: 0.8115 - val_loss: 0.4106 - val_accuracy: 0.8100
Epoch 7/10
256/256 [==============================] - 93s 363ms/step - loss: 0.3443 - accuracy: 0.8544 - val_loss: 0.3554 - val_accuracy: 0.8535
Epoch 8/10
256/256 [==============================] - 92s 360ms/step - loss: 0.3226 - accuracy: 0.8610 - val_loss: 0.3309 - val_accuracy: 0.8555
Epoch 9/10
256/256 [==============================] - 91s 354ms/step - loss: 0.2638 - accuracy: 0.8968 - val_loss: 0.3178 - val_accuracy: 0.8682
Epoch 10/10
256/256 [==============================] - 90s 353ms/step - loss: 0.2433 - accuracy: 0.8978 - val_loss: 0.3253 - val_accuracy: 0.8705

# Final visualizations :
---
1. checking the accuracy and the loss curve to find model's preformance._____no_output_____

plt.title('Model Accuracy')
plt.plot(hist.history['accuracy'],label='train')
plt.plot(hist.history['val_accuracy'],label='validation')
plt.legend()
plt.show()
plt.title('Model Loss')
plt.plot(hist.history['loss'],label='train')
plt.plot(hist.history['val_loss'],label='validation')
plt.legend()
plt.show()_____no_output_____

Model looks well tuned so we can move further._____no_output_____

pred=model.predict(X_test)_____no_output_____

As the final output is a probablity so we need to categorize it._____no_output_____

for i in pred:
  print(i)
  print(type(i))
  break[0.03084722]
<class 'numpy.ndarray'>
y_pred=[]
for i in pred:
  if i>=0.5:
    y_pred.append(int(1))
  else:
    y_pred.append(int(0))_____no_output_____print('test data accuracy : ',accuracy_score(y_test,y_pred))test data accuracy :  0.862970297029703
decoder=['Car','Truck']_____no_output_____

### Checking with the prediction with the actual pictures._____no_output_____

for i in range(10):
  plt.title(decoder[y_pred[i*200]])
  plt.imshow(X_test[i*200].reshape(64,64,3))
  plt.show()_____no_output_____

# Confusion Matrix :
---_____no_output_____

train_mat=confusion_matrix(y_test,y_pred)
plot_confusion_matrix(train_mat,figsize=(3,3),colorbar=True)_____no_output_____

Checking the wrong predicted ones and their quality._____no_output_____

x=0
y=0
for i in range(len(y_pred)):
  if y_pred[i]==1 and y_test[i]==0 and x==0:
    plt.title('was CAR , predicted TRUCK')
    plt.imshow(X_test[i].reshape(64,64,3))
    plt.show()
    x+=1
  elif y_pred[i]==0 and y_test[i]==1 and y==0:
    plt.title('was TRUCK , predicted CAR')
    plt.imshow(X_test[i].reshape(64,64,3))
    plt.show()
    y+=1
  elif x>0 and y>0:
    break
  _____no_output_____

we can see that the model is well tuned , the wrong predicted images are tougher to predict . So we can say the model works optimally._____no_output_____## HURRAH !
---
We've completed the binary classification task with a very good accuracy.

You can visit my other works at [github](https://github.com/sagnik1511) or in [kaggle](https://kaggle.com/sahnik1511).

Do **UPVOTE** in kaggle and **STAR** in github my projects to keep me going :)_____no_output_____


# Notebook from borisz264/NMD_screen_2020
Path: F4_profiling/8_4E2_GIGY_target_TPM_analysis/.ipynb_checkpoints/20200907_4E2_GIGY_target_fold_changes-checkpoint.ipynb

# 20200907 - ribosome density distributions of SMG1i, GIGYF2, EIF4E2, NMD targets_____no_output_____First I need to assemble the data from xtail (fold changes and p values)_____no_output_____

#set up imports, define some functions
import pandas as pd
import numpy as np
import math
import sys
pipeline_dir = '../ribo_seq/'
if pipeline_dir not in sys.path:
    sys.path.append(pipeline_dir)
import ribo_utils
comprehensive_gtf_data = ribo_utils.gtf_data('../annotations/gencode.v35.annotation.pBZ105.no_nonstop.gtf', add_3_for_stop=True)
genome_seq = ribo_utils.genome_sequence('../annotations/genome/GRCh38.primary_assembly.genome.pBZ105.fa')
def extract_gene_id(x):
    info_parts = x.split('|')
    return info_parts[0]

def extract_gene_name(x):
    info_parts = x.split('|')
    return info_parts[1]

def get_gene_id(x):
    if '|' in x:
        return get_gene_id(x.split('|')[0])
    return comprehensive_gtf_data.tx_to_genes[x]

def get_gene_name(x):
    if '|' in x:
        return get_gene_name(x.split('|')[0])
    return comprehensive_gtf_data.tx_to_gene_names[x]

def log_trans(x):
    try:
        return math.log(x, 2)
    except:
        return float('NaN')

def get_stop_codon(x, upstream=0, downstream=0):
    transcript_seq = comprehensive_gtf_data.transcript_sequence(genome_seq, x, exon_type='exon')
    cds_start, cds_end = comprehensive_gtf_data.cds_boundaries(x)
    stop_seq = transcript_seq[cds_end-(2+upstream): cds_end+1+downstream] #have to point 1 past end positions
    return stop_seq_____no_output_____#matplotlib setup
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
import scipy.stats as stats
import numpy as np
from adjustText import adjust_text
import math

mpl.rcParams['pdf.fonttype'] = 42 #leaves most text as actual text in PDFs, not outlines
mpl.rcParams['font.sans-serif'] = "Arial"
mpl.rcParams['font.family'] = "sans-serif"

black = (0,0,0)
gray = (0.5,0.5,0.5)
orange = (230/255.0,159/255.0,0)
skyBlue = (86/255.0,180/255.0,233/255.0)
bluishGreen = (0,158/255.0,115/255.0)
yellow = (240/255.0,228/255.0,66/255.0)
blue = (0,114/255.0,178/255.0)
vermillion = (213/255.0,94/255.0,0)
reddishPurple = (204/255.0,121/255.0,167/255.0)
colors = [black, orange, skyBlue, bluishGreen, vermillion, blue, reddishPurple, yellow]
rainbow = [black, vermillion, orange, bluishGreen, blue, reddishPurple, 'violet']_____no_output_____#get data from xtail analysis
#KO effects in DMSO
SMG1i_vs_DMSO = pd.read_csv('../5_run_xtail/xtail_results/xtail_SMG1i_mono_RNA.tsv', sep='\t')
EIF4E2_in_DMSO = pd.read_csv('../5_run_xtail/xtail_results/xtail_eIF4E2_in_DMSO_mono_RNA.tsv', sep='\t')
GIGYF2_in_DMSO = pd.read_csv('../5_run_xtail/xtail_results/xtail_GIGYF2_in_DMSO_mono_RNA.tsv', sep='\t')

significance_datasets = [SMG1i_vs_DMSO, EIF4E2_in_DMSO, GIGYF2_in_DMSO]
for dataset in significance_datasets:
    dataset.rename(columns={'Unnamed: 0': 'tx_id'}, inplace=True)
    dataset['gene_name'] = dataset['tx_id'].apply(get_gene_name)
    #dataset['minus logpadj'] = dataset['pvalue.adjust'].apply(log_trans)*-1_____no_output_____SMG1i_vs_DMSO_____no_output_____

### import TPMs_____no_output_____

all_tpms = pd.read_csv('../4_salmon_quant/all_datasets_stop_collapsed_tpm.tsv', sep='\t')_____no_output_____samples = ['WT_DMSO_1', 'WT_DMSO_2', 'WT_SMG1i_1', 'WT_SMG1i_2',
                    'eIF4E2_4_DMSO_1', 'eIF4E2_4_DMSO_2', 'eIF4E2_4_SMG1i_1', 'eIF4E2_4_SMG1i_2',
                    'GIGYF2_4_DMSO_1', 'GIGYF2_4_DMSO_2', 'GIGYF2_4_SMG1i_1', 'GIGYF2_4_SMG1i_2']
#average replicates
for i in range(0, len(samples), 2):
    rep1 = samples[i]
    rep2 = samples[i+1]
    name = rep1[:-2]
    all_tpms['%s_mono/RNA' % (name)] = (all_tpms['%s_mono' % (rep1)]/all_tpms['%s_RNA' % (rep1)] +  all_tpms['%s_mono' % (rep2)]/all_tpms['%s_RNA' % (rep2)])/2.
    all_tpms['%s_mono/RNA' % (name)]=all_tpms['%s_mono/RNA' % (name)].apply(log_trans)_____no_output_____#import NMD targets
colombo_p_values = pd.read_csv('../3_call_NMD_targets/colombo_NMD_meta_pvalues.tsv', sep = '\t')
colombo_p_values.sort_values(by=['meta_meta'], inplace=True)_____no_output_____

## Plot densities_____no_output_____

samples = ['WT_DMSO_1', 'WT_DMSO_2', 'WT_SMG1i_1', 'WT_SMG1i_2',
                    'eIF4E2_4_DMSO_1', 'eIF4E2_4_DMSO_2', 'eIF4E2_4_SMG1i_1', 'eIF4E2_4_SMG1i_2',
                    'GIGYF2_4_DMSO_1', 'GIGYF2_4_DMSO_2', 'GIGYF2_4_SMG1i_1', 'GIGYF2_4_SMG1i_2']
#average replicates
for i in range(0, len(samples), 2):
    rep1 = samples[i]
    rep2 = samples[i+1]
    name = rep1[:-2]

    #plot_colors = [gray, bluishGreen, blue, reddishPurple, vermillion]
    plot_colors = colors

    plot_index = 1
    num_plots_wide = 1
    num_plots_high = 1
    fig = plt.figure(figsize=(3*num_plots_wide, 3*num_plots_high))
    plot = fig.add_subplot(num_plots_high, num_plots_wide, plot_index)

    all_TEs =  all_tpms['%s_mono/RNA' % name].replace([np.inf, -np.inf], np.nan).dropna()
    plot.hist(all_TEs, bins=5000, density=True, histtype='step', cumulative=-1, label='all transcripts (%d)' %(len(all_TEs)), color = black)

    NMD_targets = colombo_p_values[colombo_p_values['meta_meta']<0.05]
    NMD_EJC_targets = colombo_p_values[(colombo_p_values['meta_meta']<0.05) & (colombo_p_values['GENCODE nmd annotation'])]
    NMD_no_EJC_targets = colombo_p_values[(colombo_p_values['meta_meta']<0.05) & (~colombo_p_values['GENCODE nmd annotation'])]

    reporter_TE = all_tpms[all_tpms['tx_id']=='UGAC_reporter']['WT_DMSO_mono/RNA'].replace([np.inf, -np.inf], np.nan).dropna()

    NMD_no_EJC_TEs = all_tpms[all_tpms['tx_id'].isin(NMD_no_EJC_targets['tx_id'])]['%s_mono/RNA' % name].replace([np.inf, -np.inf], np.nan).dropna()
    NMD_EJC_TEs = all_tpms[all_tpms['tx_id'].isin(NMD_EJC_targets['tx_id'])]['%s_mono/RNA' % name].replace([np.inf, -np.inf], np.nan).dropna()

    u, p = stats.mannwhitneyu(all_TEs, NMD_no_EJC_TEs)
    plot.hist(NMD_no_EJC_TEs, bins=5000, density=True, histtype='step', cumulative=-1, label='NMD, no EJC (%d, %.3g)'%(len(NMD_no_EJC_TEs), p), color = vermillion)
    u, p = stats.mannwhitneyu(all_TEs, NMD_EJC_TEs)
    plot.hist(NMD_EJC_TEs, bins=5000, density=True, histtype='step', cumulative=-1, label='NMD, EJC (%d, %.3g)'%(len(NMD_EJC_TEs), p), color = vermillion, linestyle ='dashed')

    SMG1i_hits = SMG1i_vs_DMSO[(SMG1i_vs_DMSO['pvalue.adjust']<0.1) & (SMG1i_vs_DMSO['log2FC_TE_final']>0)]
    GIGYF2_hits = GIGYF2_in_DMSO[(GIGYF2_in_DMSO['pvalue.adjust']<0.1) & (GIGYF2_in_DMSO['log2FC_TE_final']>0)]
    EIF4E2_hits = EIF4E2_in_DMSO[(EIF4E2_in_DMSO['pvalue.adjust']<0.1) & (EIF4E2_in_DMSO['log2FC_TE_final']>0)]

    SMG1i_TEs = all_tpms[all_tpms['tx_id'].isin(SMG1i_hits['tx_id'])]['%s_mono/RNA' % (name)].replace([np.inf, -np.inf], np.nan).dropna()
    GIGYF2_TEs = all_tpms[all_tpms['tx_id'].isin(GIGYF2_hits['tx_id'])]['%s_mono/RNA' % (name)].replace([np.inf, -np.inf], np.nan).dropna()
    EIF4E2_TEs = all_tpms[all_tpms['tx_id'].isin(EIF4E2_hits['tx_id'])]['%s_mono/RNA' % (name)].replace([np.inf, -np.inf], np.nan).dropna()

    u, p = stats.mannwhitneyu(all_TEs, SMG1i_TEs)
    plot.hist(SMG1i_TEs, bins=5000, density=True, histtype='step', cumulative=-1, label='SMG1i translation up (%d, %.3g)'%(len(SMG1i_TEs), p), color = orange)

    u, p = stats.mannwhitneyu(all_TEs, EIF4E2_TEs)
    plot.hist(EIF4E2_TEs, bins=5000, density=True, histtype='step', cumulative=-1, label= u'EIF4E2Δ translation up (%d, %.3g)'%(len(EIF4E2_TEs), p), color = bluishGreen)

    u, p = stats.mannwhitneyu(all_TEs, GIGYF2_TEs)
    plot.hist(GIGYF2_TEs, bins=5000, density=True, histtype='step', cumulative=-1, label= u'GIGYF2Δ translation up (%d, %.3g)'%(len(GIGYF2_TEs), p), color = blue)

    plot.scatter(reporter_TE,[0]*len(reporter_TE), s=30, color=blue)

    plot.set_ylabel('transcript fraction', fontsize=15)
    plot.set_xlabel('ribosome density', fontsize=15)
    plot.set_title(name)
    plot.set_ylim(0, 1.01)
    plot.set_xlim(-7, 7)

    plot.spines['right'].set_visible(False)
    plot.spines['top'].set_visible(False)
    #plt.xticks(rotation=45, ha='right')
    lg=plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
    lg.draw_frame(False)
    #plt.subplots_adjust(wspace = 0.3)
    #plt.tight_layout()
    plt.savefig('ribosome_densities_subsets_%s.pdf' % name, transparent='True', format='pdf', bbox_inches='tight')_____no_output_____


# Notebook from pachterlab/GRNP_2020
Path: notebooks/R_processing/ProcessR_EVALPBMC_DS.ipynb

_____no_output_____**EVALPBMC_DS dataset: Processes the BUG files into files prepared for use in R**

This notebook processes the output from the fastq file processing for this dataset. The data produced here is pre-generated and downloaded by the figure generation code. The purpose of this processing step is to prepare the data for figure generation, by filtering the data and producing downsampled datasets in addition to the original one.

Steps:
1. Clone the code repo and download data to process
2. Prepare the R environment
3. Process the data
4. Generate statistics for the dataset

The data used in this processing step is produced by the following notebook:

https://github.com/pachterlab/GRNP_2020/blob/master/notebooks/FASTQ_processing/ProcessEVALPBMC_DS.ipynb
_____no_output_____**1. Clone the code repo and download data to process**_____no_output_____

![ -d "GRNP_2020" ] && rm -r GRNP_2020

!git clone https://github.com/pachterlab/GRNP_2020.git
Cloning into 'GRNP_2020'...
remote: Enumerating objects: 23, done.
remote: Counting objects: 100% (23/23), done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 1741 (delta 6), reused 10 (delta 1), pack-reused 1718
Receiving objects: 100% (1741/1741), 9.43 MiB | 21.42 MiB/s, done.
Resolving deltas: 100% (1191/1191), done.
#download BUG data from Zenodo
!mkdir data
!cd data && wget https://zenodo.org/record/4661569/files/EVALPBMC_DS.zip?download=1 && unzip 'EVALPBMC_DS.zip?download=1' && rm 'EVALPBMC_DS.zip?download=1'--2021-04-03 14:36:09--  https://zenodo.org/record/4661569/files/EVALPBMC_DS.zip?download=1
Resolving zenodo.org (zenodo.org)... 137.138.76.77
Connecting to zenodo.org (zenodo.org)|137.138.76.77|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 123937246 (118M) [application/octet-stream]
Saving to: ‘EVALPBMC_DS.zip?download=1’

EVALPBMC_DS.zip?dow 100%[===================>] 118.20M  9.33MB/s    in 14s

2021-04-03 14:36:25 (8.36 MB/s) - ‘EVALPBMC_DS.zip?download=1’ saved [123937246/123937246]

Archive:  EVALPBMC_DS.zip?download=1
   creating: EVALPBMC_DS/
   creating: EVALPBMC_DS/bus_output/
  inflating: EVALPBMC_DS/bus_output/bug.txt
  inflating: EVALPBMC_DS/bus_output/coll.genes.txt
  inflating: EVALPBMC_DS/bus_output/transcripts_to_genes.txt
#Check that download worked
!cd data && ls -l && cd EVALPBMC_DS/bus_output && ls -ltotal 4
drwxr-xr-x 3 root root 4096 Jul  1  2020 EVALPBMC_DS
total 498464
-rw-r--r-- 1 root root 501340444 Jul  1  2020 bug.txt
-rw-r--r-- 1 root root    738211 Jul  1  2020 coll.genes.txt
-rw-r--r-- 1 root root   8335642 Jul  1  2020 transcripts_to_genes.txt

**2. Prepare the R environment**_____no_output_____

#switch to R mode
%reload_ext rpy2.ipython
_____no_output_____#install the R packages
%%R
install.packages("qdapTools")
install.packages("dplyr")
install.packages("stringdist")
install.packages("stringr")
R[write to console]: Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)

R[write to console]: also installing the dependencies ‘bitops’, ‘chron’, ‘data.table’, ‘RCurl’, ‘XML’

R[write to console]: trying URL 'https://cran.rstudio.com/src/contrib/bitops_1.0-6.tar.gz'

R[write to console]: Content type 'application/x-gzip'
R[write to console]:  length 8734 bytes

R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]:

R[write to console]: downloaded 8734 bytes

R[write to console]: trying URL 'https://cran.rstudio.com/src/contrib/chron_2.3-56.tar.gz'

R[write to console]: Content type 'application/x-gzip'
R[write to console]:  length 37657 bytes (36 KB)

R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]:

R[write to console]: downloaded 36 KB

R[write to console]: trying URL 'https://cran.rstudio.com/src/contrib/data.table_1.14.0.tar.gz'

R[write to console]: Content type 'application/x-gzip'
R[write to console]:  length 5303849 bytes (5.1 MB)

R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]:

R[write to console]: downloaded 5.1 MB

R[write to console]: trying URL 'https://cran.rstudio.com/src/contrib/RCurl_1.98-1.3.tar.gz'

R[write to console]: Content type 'application/x-gzip'
R[write to console]:  length 728630 bytes (711 KB)

R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]:

R[write to console]: downloaded 711 KB

R[write to console]: trying URL 'https://cran.rstudio.com/src/contrib/XML_3.99-0.6.tar.gz'

R[write to console]: Content type 'application/x-gzip'
R[write to console]:  length 968773 bytes (946 KB)

R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]:

R[write to console]: downloaded 946 KB

R[write to console]: trying URL 'https://cran.rstudio.com/src/contrib/qdapTools_1.3.5.tar.gz'

R[write to console]: Content type 'application/x-gzip'
R[write to console]:  length 36880 bytes (36 KB)

R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]:

R[write to console]: downloaded 36 KB

R[write to console]:

R[write to console]:
R[write to console]: The downloaded source packages are in
	‘/tmp/RtmpNMBLWn/downloaded_packages’
R[write to console]:
R[write to console]:

R[write to console]: Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)

R[write to console]: trying URL 'https://cran.rstudio.com/src/contrib/dplyr_1.0.5.tar.gz'

R[write to console]: Content type 'application/x-gzip'
R[write to console]:  length 949019 bytes (926 KB)

R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]:

R[write to console]: downloaded 926 KB

R[write to console]:

R[write to console]:
R[write to console]: The downloaded source packages are in
	‘/tmp/RtmpNMBLWn/downloaded_packages’
R[write to console]:
R[write to console]:

R[write to console]: Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)

R[write to console]: trying URL 'https://cran.rstudio.com/src/contrib/stringdist_0.9.6.3.tar.gz'

R[write to console]: Content type 'application/x-gzip'
R[write to console]:  length 839683 bytes (820 KB)

R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]:

R[write to console]: downloaded 820 KB

R[write to console]:

R[write to console]:
R[write to console]: The downloaded source packages are in
	‘/tmp/RtmpNMBLWn/downloaded_packages’
R[write to console]:
R[write to console]:

R[write to console]: Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)

R[write to console]: trying URL 'https://cran.rstudio.com/src/contrib/stringr_1.4.0.tar.gz'

R[write to console]: Content type 'application/x-gzip'
R[write to console]:  length 135777 bytes (132 KB)

R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]: =
R[write to console]:

R[write to console]: downloaded 132 KB

R[write to console]:

R[write to console]:
R[write to console]: The downloaded source packages are in
	‘/tmp/RtmpNMBLWn/downloaded_packages’
R[write to console]:
R[write to console]:

**3. Process the data**

Here we discard multimapped UMIs and all UMIs belonging to cells with fewer than 200 UMIs. We also precalculate gene expression, fraction of single-copy molecules etc. and save as stats (statistics). These can later be used when generating figures. We also generate down-sampled BUGs._____no_output_____

#create output directory
!mkdir figureData_____no_output_____#First set some path variables
%%R
source("GRNP_2020/RCode/pathsGoogleColab.R")
_____no_output_____#Process and filter the BUG file
%%R
source(paste0(sourcePath, "BUGProcessingHelpers.R"))
createStandardBugsData(paste0(dataPath,"EVALPBMC_DS/"), "EVALPBMC_DS", c(0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 1))

R[write to console]:
Attaching package: ‘dplyr’

R[write to console]: The following objects are masked from ‘package:stats’:

    filter, lag

R[write to console]: The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

R[write to console]:
Attaching package: ‘qdapTools’

R[write to console]: The following object is masked from ‘package:dplyr’:

    id

**4. Generate statistics for the dataset**

Here we create a file with various statistics for the dataset, which is used for generating table S3. It also contains some additional information about the dataset. Generation of this file may take several hours._____no_output_____

%%R
source(paste0(sourcePath, "GenBugSummary.R"))
genBugSummary("EVALPBMC_DS", "FGF23", "RPS10", 10)[1] "Down-sampling to 2000 UMIs"
[1] "Will process 2000 UMIs"
[1] 100
[1] 200
[1] 300
[1] 400
[1] 500
[1] 600
[1] 700
[1] 800
[1] 900
[1] 1000
[1] 1100
[1] 1200
[1] 1300
[1] 1400
[1] 1500
[1] 1600
[1] 1700
[1] 1800
[1] 1900
[1] 2000
[1] "Down-sampling to 2000 UMIs"
[1] "Will process 2000 UMIs"
[1] 100
[1] 200
[1] 300
[1] 400
[1] 500
[1] 600
[1] 700
[1] 800
[1] 900
[1] 1000
[1] 1100
[1] 1200
[1] 1300
[1] 1400
[1] 1500
[1] 1600
[1] 1700
[1] 1800
[1] 1900
[1] 2000
[1] "Down-sampling to 2000 UMIs"
[1] "Will process 2000 UMIs"
[1] 100
[1] 200
[1] 300
[1] 400
[1] 500
[1] 600
[1] 700
[1] 800
[1] 900
[1] 1000
[1] 1100
[1] 1200
[1] 1300
[1] 1400
[1] 1500
[1] 1600
[1] 1700
[1] 1800
[1] 1900
[1] 2000
[1] "Down-sampling to 2000 UMIs"
[1] "Will process 2000 UMIs"
[1] 100
[1] 200
[1] 300
[1] 400
[1] 500
[1] 600
[1] 700
[1] 800
[1] 900
[1] 1000
[1] 1100
[1] 1200
[1] 1300
[1] 1400
[1] 1500
[1] 1600
[1] 1700
[1] 1800
[1] 1900
[1] 2000
[1] "Down-sampling to 2000 UMIs"
[1] "Will process 2000 UMIs"
[1] 100
[1] 200
[1] 300
[1] 400
[1] 500
[1] 600
[1] 700
[1] 800
[1] 900
[1] 1000
[1] 1100
[1] 1200
[1] 1300
[1] 1400
[1] 1500
[1] 1600
[1] 1700
[1] 1800
[1] 1900
[1] 2000
!cd figureData/EVALPBMC_DS && ls -l && more ds_summary.txttotal 210824
-rw-r--r-- 1 root root 39106135 Apr  3 14:40 Bug_100.RData
-rw-r--r-- 1 root root 23469947 Apr  3 14:43 Bug_10.RData
-rw-r--r-- 1 root root 28181079 Apr  3 14:43 Bug_20.RData
-rw-r--r-- 1 root root 32714034 Apr  3 14:42 Bug_40.RData
-rw-r--r-- 1 root root 18097923 Apr  3 14:43 Bug_5.RData
-rw-r--r-- 1 root root 35466112 Apr  3 14:42 Bug_60.RData
-rw-r--r-- 1 root root 37415976 Apr  3 14:41 Bug_80.RData
-rw-r--r-- 1 root root      945 Apr  3 14:56 ds_summary.txt
-rw-r--r-- 1 root root  1415239 Apr  3 14:44 Stats.RData
Dataset: EVALPBMC_DS

totUMIs: 4015392
totCells: 6862
totCounts: 87378182
countsPerUMI: 21.7608099034914
UMIsPerCell: 585.163509180997
countsPerCell: 12733.6318857476
totFracOnes: 0.199118043767582
FracMolWithUMIDistToNeighborH: 901, 1015, 84, 0, 0, 0, 0, 0, 0, 0
FracMolWithUMIDistToNeighborL: 791, 1057, 151, 1, 0, 0, 0, 0, 0, 0
FracMolWithUMIDistToNeighborHFrac: 0.4505, 0.5075, 0.042, 0, 0, 0, 0, 0, 0, 0
FracMolWithUMIDistToNeighborLFrac: 0.3955, 0.5285, 0.0755, 5e-04, 0, 0, 0, 0, 0,
 0
FracMolWithUMIDistToNeighbor1cpy: 979, 910, 111, 0, 0, 0, 0, 0, 0, 0
FracMolWithUMIDistToNeighbor2cpy: 882, 991, 127, 0, 0, 0, 0, 0, 0, 0
FracMolWithUMIDistToNeighbor>=3cpy: 913, 987, 100, 0, 0, 0, 0, 0, 0, 0
FracMolWithUMIDistToNeighbor1cpyFrac: 0.4895, 0.455, 0.0555, 0, 0, 0, 0, 0, 0, 0
FracMolWithUMIDistToNeighbor2cpyFrac: 0.441, 0.4955, 0.0635, 0, 0, 0, 0, 0, 0, 0
FracMolWithUMIDistToNeighbor>=3cpyFrac: 0.4565, 0.4935, 0.05, 0, 0, 0, 0, 0, 0,
0


# Notebook from khanparwaz/PythonProjects
Path: notebooks/chapter15_symbolic/07_lotka.ipynb

> This is one of the 100 recipes of the [IPython Cookbook](http://ipython-books.github.io/), the definitive guide to high-performance scientific computing and data science in Python.
_____no_output_____# 15.7. Analyzing a nonlinear differential system: Lotka-Volterra (predator-prey) equations_____no_output_____Here, we conduct a brief analytical study of a famous nonlinear differential system: the Lotka-Volterra equations, also known as predator-prey equations. This simple model describes the evolution of two interacting populations (e.g. sharks and sardines), where the predators eat the preys. This example illustrates how we can use SymPy to obtain exact expressions and results for fixed points and their stability._____no_output_____

from sympy import *
init_printing()_____no_output_____var('x y')
var('a b c d', positive=True)_____no_output_____

The variables x and y represent the populations of the preys and predators, respectively. The parameters a, b, c and d are positive parameters (described more precisely in "How it works..."). The equations are:

$$\begin{align}
\frac{dx}{dt} &= f(x) = x(a-by)\\
\frac{dy}{dt} &= g(x) = -y(c-dx)
\end{align}$$_____no_output_____

f = x * (a - b*y)
g = -y * (c - d*x)_____no_output_____

Let's find the fixed points of the system (solving f(x,y) = g(x,y) = 0)._____no_output_____

solve([f, g], (x, y))_____no_output_____(x0, y0), (x1, y1) = ______no_output_____

Let's write the 2D vector with the two equations._____no_output_____

M = Matrix((f, g)); M_____no_output_____

Now we can compute the Jacobian of the system, as a function of (x, y)._____no_output_____

J = M.jacobian((x, y)); J_____no_output_____

Let's study the stability of the two fixed points by looking at the eigenvalues of the Jacobian at these points._____no_output_____

M0 = J.subs(x, x0).subs(y, y0); M0_____no_output_____M0.eigenvals()_____no_output_____

The parameters a and c are strictly positive, so the eigenvalues are real and of opposite signs, and this fixed point is a saddle point. Since this point is unstable, the extinction of both populations is unlikely in this model._____no_output_____

M1 = J.subs(x, x1).subs(y, y1); M1_____no_output_____M1.eigenvals()_____no_output_____

The eigenvalues are purely imaginary so this fixed point is not hyperbolic, and we cannot draw conclusions about the qualitative behavior of the system around this fixed point from this linear analysis. However, one can show with other methods that oscillations occur around this point._____no_output_____> You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).

> [IPython Cookbook](http://ipython-books.github.io/), by [Cyrille Rossant](http://cyrille.rossant.net), Packt Publishing, 2014 (500 pages)._____no_output_____


# Notebook from misialq/paycheck
Path: eval/figure2.ipynb

<code>
from os.path import join
import json
import csv
import glob
import os

import rpy2.rinterface
from pandas import read_csv, DataFrame, Series
from qiime2 import Artifact
from scipy.stats import t, sem
from IPython.display import Image_____no_output_____%reload_ext rpy2.ipython_____no_output_____%%R
library(ggplot2)
library(grid)_____no_output_____empo3_dir = '/Users/benkaehler/Data/empo_3/'_____no_output_____columns = ['sample type', 'sample', 'fold']
data = {c:[] for c in columns}
for type_dir in glob.glob(join(empo3_dir, '*')):
    if not os.path.exists(join(type_dir, 'results', 'weights.qza')):
        continue
    type_ = os.path.basename(type_dir)
    if type_ in ('sterile-water-blank', 'single-strain', 'mock-community', 'nick', 'plant-surface'):
        continue
    for fold_dir in glob.glob(join(type_dir, 'tmp', 'fold-*')):
        _, fold = fold_dir.rsplit('-', 1)
        with open(join(fold_dir, 'sample_test.json')) as fh:
            try:
                samples = json.load(fh)
            except UnicodeDecodeError:
                print(join(fold_dir, 'sample_test.json'), 'is corrupted')
        data['sample'].extend(samples)
        data['sample type'].extend([type_]*len(samples))
        data['fold'].extend([fold]*len(samples))
folds = DataFrame(data)_____no_output_____abundances = read_csv(join(empo3_dir, 'abundances.tsv'), sep='\t')
abundances = abundances.set_index(['sample type', 'sample'])_____no_output_____eval_comp = read_csv(join(empo3_dir, 'eval_comp_results.tsv'), sep='\t')
eval_comp = eval_comp.join(abundances, ['sample type', 'sample'])_____no_output_____folds = folds.set_index(['sample type', 'sample'])
eval_comp = eval_comp.join(folds, ['sample type', 'sample'])_____no_output_____level7 = eval_comp[eval_comp['level'] == 7]
level7 = level7[level7['abundance'] >= 1000]_____no_output_____level7.head()_____no_output_____grouped = level7.groupby(['sample type', 'class weights', 'fold'])
def weighted_stats(x):
    d1 = x['Bray-Curtis'].mean()
    return Series([d1], index=['mean'])
figure2 = grouped.apply(weighted_stats)
figure2.reset_index(inplace=True)_____no_output_____grouped = figure2.groupby(['sample type', 'class weights'])
def weighted_stats(x):
    mu = x['mean'].mean()
    se = sem(x['mean'])
    lower, upper = t.interval(0.95, 5, mu, se)
    return Series([mu, lower, upper], index=['mean', 'lower', 'upper'])
figure2 = grouped.apply(weighted_stats)
figure2.reset_index(inplace=True)_____no_output_____old_labels = [
    'animal-corpus',
    'animal-distal-gut',
    'animal-proximal-gut',
    'animal-secretion',
    'animal-surface',
    'plant-corpus',
    'plant-rhizosphere',
    'sediment-non-saline',
    'soil-non-saline',
    'surface-non-saline',
    'water-non-saline',
    'sediment-saline',
    'surface-saline',
    'water-saline',
    'average', 'uniform70', 'bespoke70']
new_labels = [
    'Other'] * 14 + ['Average', 'Uniform', 'Bespoke']
for old, new in zip(old_labels, new_labels):
    figure2.loc[figure2['class weights'] == old, 'class weights'] = new
old_labels = [
    'animal-corpus',
    'animal-distal-gut',
    'animal-proximal-gut',
    'animal-secretion',
    'animal-surface',
    'plant-corpus',
    'plant-rhizosphere',
    'sediment-non-saline',
    'soil-non-saline',
    'surface-non-saline',
    'water-non-saline',
    'sediment-saline',
    'surface-saline',
    'water-saline'
]
new_labels = [
    'Animal corpus',
    'Animal distal gut',
    'Animal proximal gut',
    'Animal secretion',
    'Animal surface',
    'Plant corpus',
    'Plant rhizosphere',
    'Sediment (non-saline)',
    'Soil (non-saline)',
    'Surface (non-saline)',
    'Water (non-saline)',
    'Sediment (saline)',
    'Surface (saline)',
    'Water (saline)'
]
for old, new in zip(old_labels, new_labels):
    figure2.loc[figure2['sample type'] == old, 'sample type'] = new_____no_output_____%%R -i figure2
figure2$sample.type = factor(figure2$sample.type, levels=c(
 'Water (saline)',
 'Surface (saline)',
 'Sediment (saline)',
 'Water (non-saline)',
 'Surface (non-saline)',
 'Soil (non-saline)',
 'Sediment (non-saline)',
 'Plant rhizosphere',
 'Plant corpus',
 'Animal surface',
 'Animal secretion',
 'Animal proximal gut',
 'Animal distal gut',
 'Animal corpus'))
figure2$class.weights = factor(figure2$class.weights, levels=c(
    'Other', 'Average', 'Uniform', 'Bespoke'
))
cbPalette <- c("#999999", "#56B4E9", "#E69F00", "#009E73", "#F0E442", "#0072B2", "#D55E00")
ggplot(data=figure2, aes(x=sample.type, y=mean, fill=class.weights)) +
  geom_bar(stat="identity", width=0.8, position=position_dodge(width=0.9), aes(colour=class.weights, alpha=(class.weights != "Other"))) +
  geom_errorbar(aes(ymin=lower, ymax=upper, alpha=(class.weights != "Other")), width=0.5, position=position_dodge(width=0.9)) +
  coord_flip(clip="off") +
  theme_bw() +
  labs(x='EMPO3 Habitat', y='Bray-Curtis dissimilarity') +
  scale_alpha_manual(values=c(0, 1), guide=F) +
  scale_fill_manual(values=cbPalette) +
  scale_colour_manual(values=cbPalette, guide=F) +
  guides(fill=guide_legend(title="Taxonomic Weights", reverse=T)) +
  theme(plot.margin = unit(c(1, 1, 2, 1), "lines"),
        axis.text.y = element_text(angle = 45)) +
  annotation_custom(
    grob=grid::textGrob(label = "Better Performance"),
    xmin=-0.5, xmax=-0.5, ymin=0.2, ymax=0.2
  ) +
  annotation_custom(
    grob=linesGrob(arrow=arrow(type="open", ends="first", length=unit(2,"mm"))),
    xmin=-0.3, xmax=-0.3, ymin=0., ymax=0.4
  )

ggsave(file="figure2.png", width=5, height=10, dpi=300)_____no_output_____Image("figure2.png")_____no_output_____max(figure2['upper'] - figure2['lower'])_____no_output_____figure2_____no_output_____bespoke = figure2[figure2['class weights'] == 'Bespoke']['mean']
print(min(bespoke), max(bespoke))
uniform = figure2[figure2['class weights'] == 'Uniform']['mean']
print(min(uniform), max(uniform))0.077939392151066 0.1887800108306231
0.1238677996316758 0.41385360682300887
len(set(level7['sample']))_____no_output_____
</code>


# Notebook from sdpython/ensae_teaching_cs
Path: _doc/notebooks/td1a_home/2020_covid.ipynb

# Algo - simulation COVID

Ou comment utiliser les mathématiques pour comprendre la propagation de l'épidémie._____no_output_____

from jyquickhelper import add_notebook_menu
add_notebook_menu()_____no_output_____%matplotlib inline_____no_output_____

## Enoncé
_____no_output_____### On récupère les données du COVID par région et par âge et premier graphe

A cette adresse : [Données relatives à l’épidémie de COVID-19 en France : vue d’ensemble](https://www.data.gouv.fr/en/datasets/donnees-relatives-a-lepidemie-de-covid-19-en-france-vue-densemble/)_____no_output_____

from pandas import read_csv, to_datetime
url = "https://www.data.gouv.fr/en/datasets/r/d3a98a30-893f-47f7-96c5-2f4bcaaa0d71"
covid = read_csv(url, sep=",")
covid['date'] = to_datetime(covid['date'])
covid.tail()_____no_output_____ax = covid.set_index("date").plot(
    title="Evolution des hospitalisations par jour", figsize=(14, 4))
ax.set_yscale("log");_____no_output_____

Il y a quelques valeurs manquantes même pour les séries aggrégées... Comme je n'ai pas le courage de corriger les valeurs unes à unes, je prends un autre fichier et quelques aberrations comme le nombre de décès qui décroît, ce qui est rigoureusement impossible._____no_output_____

from pandas import concat, to_datetime

def extract_data(kind='deaths', country='France'):
    url = (
        "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/"
        "master/csse_covid_19_data/"
        "csse_covid_19_time_series/time_series_covid19_%s_global.csv" %
        kind)
    df = read_csv(url)
    eur = df[df['Country/Region'].isin([country])
             & df['Province/State'].isna()]
    tf = eur.T.iloc[4:]
    tf.columns = [kind]
    return tf

def extract_whole_data(kind=['deaths', 'confirmed', 'recovered'],
                       country='France'):
    population = {
        'France': 67e6,
    }

    total = population[country]
    dfs = []
    for k in kind:
        df = extract_data(k, country)
        dfs.append(df)
    conc = concat(dfs, axis=1)
    conc['infected'] = conc['confirmed'] - (conc['deaths'] + conc['recovered'])
    conc['safe'] = total - conc.drop('confirmed', axis=1).sum(axis=1)
    conc.index = to_datetime(conc.index)

    return conc

covid = extract_whole_data()
covid.tail()_____no_output_____import matplotlib.pyplot as plt
fig, ax = plt.subplots(1, 2, figsize=(14, 4))
covid[['confirmed', 'infected']].plot(title="Evolution de l'épidémie par jour", ax=ax[0])
covid[['deaths', 'recovered']].plot(title="Evolution de l'épidémie par jour", ax=ax[1]);_____no_output_____

Même aberration, un nombre décès qui décroît... Il faudrait comprendre pourquoi pour savoir comment réparer les données. Ou on improvise. Pour chaque observation $X_t < X_{t-1}$, on calcule le ratio $\frac{X_{t}}{X_{t-1}}$ et on multiplie toutes les observations $i<t$ par ce ratio._____no_output_____

from tqdm import tqdm  # pour avoir une barre de progression

def correct_series(X):
    for t in range(1, X.shape[0]):
        if X[t-1] > 0 and X[t] == 0:
            X[t] = X[t-1]
            continue
        if X[t] >= X[t-1] and X[t] < X[t-1] + 200000:
            continue
        ratio = X[t] / X[t-1]
        for i in range(0, t):
            X[i] *= ratio

covid_modified = covid.copy()
for c in tqdm(covid.columns):
    values = covid_modified[c].values
    correct_series(values)
    covid_modified[c] = values

covid_modified.tail()100%|██████████| 5/5 [00:00<00:00, 63.46it/s]
fig, ax = plt.subplots(1, 2, figsize=(14, 4))
covid_modified[['confirmed', 'infected']].plot(title="Evolution de l'épidémie par jour", ax=ax[0])
covid_modified[['deaths', 'recovered']].plot(title="Evolution de l'épidémie par jour", ax=ax[1]);_____no_output_____

C'est mieux._____no_output_____

covid = covid_modified_____no_output_____

On lisse._____no_output_____

lisse = covid.rolling(7).mean()
fig, ax = plt.subplots(1, 2, figsize=(14, 4))
lisse[['confirmed', 'infected']].plot(title="Evolution de l'épidémie par jour", ax=ax[0])
lisse[['deaths', 'recovered']].plot(title="Evolution de l'épidémie par jour", ax=ax[1]);_____no_output_____

La série des *recovered* ne compte vraisemblablement que ceux qui sont passés à l'hôpital. Il faudrait recouper avec d'autres données pour être sûr. Ce sera pour un autre jour._____no_output_____### Modèle [SIRD](http://www.xavierdupre.fr/app/aftercovid/helpsphinx/model_covidsir.html)

Pour en savoir plus [Modèles compartimentaux en épidémiologie](https://fr.wikipedia.org/wiki/Mod%C3%A8les_compartimentaux_en_%C3%A9pid%C3%A9miologie). On classe la population en quatre catégories :

* S : personnes non contaminées
* I : nombre de personnes malades ou contaminantes
* R : personnes guéries (recovered)
* D : personnes décédées

Les gens changent de catégorie en fonction de l'évolution de l'épidémie selon les équations qui suivent :

* $\frac{dS}{dt} = - \beta \frac{S I}{N}$
* $\frac{dI}{dt} = \frac{\beta S I}{N} - \mu I - \nu I$
* $\frac{dD}{dt} = \nu I$
* $\frac{dR}{dt} = \mu I$

$\beta$ est lié au taux de transmission, $\frac{1}{\mu}$ est la durée moyenne jusqu'à guérison, $\frac{1}{\nu}$ est la durée moyenne jusqu'au décès._____no_output_____### Q0 : une petite fonction pour dessiner

Cette fonction servira à représenter graphiquement les résultats._____no_output_____

from datetime import datetime, timedelta

def plot_simulation(sim, day0=datetime(2020, 1, 1), safe=True,
                    ax=None, title=None, logy=False, two=False,
                    true_data=None):
    """
    On suppose que sim est une matrice (days, 4).
    :param sim: la simulation
    :param day0: le premier jour de la simulation (une observation par jour)
    :param safe: ajouter les personnes *safe* (non infectées), comme elles sont nombreuses,
        il vaut mieux aussi cocher *logy=True* pour que cela soit lisible
    :param ax: axes existant (utile pour superposer), None pour un créer un nouveau
    :param title: titre du graphe
    :param logy: échelle logarithmique sur l'axe des y
    :param two: faire deux graphes plutôt qu'un seul pour plus de visibilité
    :param true_data: vraies données à tracer également en plus de celle de la simulation
    :return: ax
    """
    df = DataFrame(sim, columns=['S', 'I', 'R', 'D'])
    # On ajoute des dates.
    df["date"] = [day0 + timedelta(d) for d in range(0, df.shape[0])]
    df = df.set_index("date")
    if true_data is None:
        tdf = None
    else:
        tdf = DataFrame(true_data, columns=['Sobs', 'Iobs', 'Robs', 'Dobs'])
        tdf["date"] = [day0 + timedelta(d) for d in range(0, tdf.shape[0])]
        tdf = tdf.set_index("date")

    if two:
        if ax is None:
            fig, ax = plt.subplots(1, 2, figsize=(14, 4))
        if safe:
            if tdf is not None:
                tdf.drop(['Dobs'], axis=1).plot(ax=ax[0], logy=logy, linewidth=8)
            df.drop('D', axis=1).plot(ax=ax[0], title=title, logy=logy, linewidth=4)
        else:
            if tdf is not None:
                tdf.drop(['Sobs', 'Dobs'], axis=1).plot(ax=ax[0], logy=logy, linewidth=8)
            df.drop(['S', 'D'], axis=1).plot(ax=ax[0], title=title, logy=logy, linewidth=4)
        if tdf is not None:
            tdf['Dobs'].plot(ax=ax[1], title=title, logy=logy, linewidth=8)
        df[['D']].plot(ax=ax[1], title='Décès', logy=logy, linewidth=4)
        ax[0].legend()
        ax[1].legend()
    else:
        if ax is None:
            fig, ax = plt.subplots(1, 1, figsize=(14, 4))
        if safe:
            if tdf is not None:
                tdf.plot(ax=ax, title=title, logy=logy, linewidth=8)
            df.plot(ax=ax, title=title, logy=logy, linewidth=4)
        else:
            if tdf is not None:
                tdf.drop(['Sobs'], axis=1).plot(ax=ax, title=title, logy=logy, linewidth=8)
            df.drop(['S'], axis=1).plot(ax=ax, title=title, logy=logy, linewidth=4)
        ax.legend()
    return ax_____no_output_____

### Q1 : écrire une fonction qui calcule la propagation

On suppose que $\beta, \mu, \nu, S_0, I_0, R_0, D_0$ sont connus. On rappelle le modèle :

* $dS = - \beta \frac{S I}{N}$
* $dI = \frac{\beta S I}{N} - \mu I - \nu I$
* $dD = \nu I$
* $dR = \mu I$_____no_output_____

import numpy_____no_output_____beta = 0.5
mu = 1./14
nu = 1./21
S0 = 9990
I0 = 10
R0 = 0
D0 = 0_____no_output_____

Il faudra compléter le petit programme suivant :_____no_output_____

from pandas import DataFrame

def simulation(beta, mu, nu, S0, I0, R0, D0, days=14):
    res = numpy.empty((days+1, 4), dtype=numpy.float64)
    res[0, :] = [S0, I0, R0, D0]
    N = sum(res[0, :])
    for t in range(1, res.shape[0]):
        dR = res[t-1, 1] * mu
        # ....
    return res

sim = simulation(beta, mu, nu, S0, I0, R0, D0, 30)
plot_simulation(sim);_____no_output_____

### Q2 : on veut estimer les paramètres du modèle, une fonction d'erreur ?

C'est compliqué parce que... les paramètres évoluent au cours du temps, en fonction du comportement des gens, masque, pas masque, confinement, reconfinement, température, manque de tests également... Tout d'abord les vraies données._____no_output_____

lisse_mars = lisse[30:]
dates = lisse_mars.index
france = numpy.zeros((lisse_mars.shape[0], 4), dtype=numpy.dtype)
france[:, 3] = lisse_mars['deaths']
france[:, 2] = lisse_mars['recovered']
france[:, 0] = lisse_mars['safe']
france[:, 1] = lisse_mars['infected']

france[:5]_____no_output_____plot_simulation(france, dates[0], safe=False, logy=True, title="Vraies données");_____no_output_____

Et sur les derniers jours._____no_output_____

plot_simulation(france[-60:], dates[-60], two=True, safe=False, title="Vraies données, derniers mois");_____no_output_____

Bref, on part du principe que le modèle est plutôt fiable sur une courte période de temps, on tire plein de paramètres aléatoires et on regarde ce qui marche le mieux. Et pour comparer deux jeux de paramètres, il faut donc une fonction d'erreur qu'on prendra comme égal à la somme des erreurs de prédictions.

Maintenant il faut faire attention à ce qu'on compare. La simulation calcule les catégories de population au temps *t*, mais pas toujours les séries cumulées. La série des personnes contaminées est **transitoire dans la simulation** et **cumulées dans les données récupérées**. La première étape consiste à transformer les données simulées pour qu'elles soient comparables aux données collectées._____no_output_____

def simulation_cumulee(beta, mu, nu, S0, I0, R0, D0, days=14):
    # ...
    pass_____no_output_____

Maintenant la fonction d'erreur :_____no_output_____

def error(data, simulation):
    # ... à compléter
    return 0_____no_output_____

### Q3 : optimisation

Pour optimiser, on tire des paramètres de façon aléatoire dans un intervalle donné et on choisit ceux qui minimisent l'erreur._____no_output_____

from tqdm import tqdm  # pour avoir une barre de progression

def optimisation(true_data, i_range=(0, 0.2), beta_range=(0, 0.5),
                 mu_range=(0., 0.2), nu_range=(0., 0.2),
                 max_iter=1000, error_fct=error):
    N = sum(true_data[0, :])
    rnd = numpy.random.rand(max_iter, 4)
    for i, (a, b) in enumerate([i_range, beta_range, mu_range, nu_range]):
        rnd[:, i] = rnd[:, i] # à compléter ...

    err_min = None
    for it in tqdm(range(max_iter)):
        i, beta, mu, nu = rnd[it, :]

        D0 = true_data[0, 3]
        # dI0 =
        # S0 =
        # I0 =
        # R0 =
        sim = simulation_cumulee(beta, mu, nu, S0, I0, R0, D0, days=true_data.shape[0] - 1)
        err = error_fct(true_data, sim)
        if err_min is None or err < err_min:
            # à compléter
            pass
    return best_____no_output_____

### Q4 : dessiner les résultats_____no_output_____### Q5 : vérifier que cela marche sur des données synthétiques

On simule, on vérifie que l'optimisation retrouve les paramètres de la simulation._____no_output_____### Q6 : sur des données réelles_____no_output_____## Réponses_____no_output_____### Q1 : propagation

On l'applique aux données réelles._____no_output_____

from datetime import datetime, timedelta
from pandas import DataFrame

def simulation(beta, mu, nu, S0, I0, R0, D0, days=14):
    res = numpy.empty((days+1, 4), dtype=numpy.float64)
    res[0, :] = [S0, I0, R0, D0]
    N = sum(res[0, :])
    for t in range(1, res.shape[0]):
        dR = res[t-1, 1] * mu
        dD = res[t-1, 1] * nu
        dI = res[t-1, 0] * res[t-1, 1] / N * beta
        res[t, 0] = res[t-1, 0] - dI
        res[t, 1] = res[t-1, 1] + dI - dR - dD
        res[t, 2] = res[t-1, 2] + dR
        res[t, 3] = res[t-1, 3] + dD
    return res

beta = 0.5
mu = 1./14
nu = 1./21
S0 = 9990
I0 = 10
R0 = 0
D0 = 0

sim = simulation(beta, mu, nu, S0, I0, R0, D0, 60)
plot_simulation(sim, dates[60], safe=False, two=True,
                title="Simulation pour essayer");_____no_output_____

### Q2 : série cumulées et fonction erreur

On doit d'abord calculer la simulation qui modifie $I_t$ en $J_t$ qui correspond à l'ensemble des personnes contaminées jusqu'à présent._____no_output_____

def simulation_cumulee(beta, mu, nu, S0, I0, R0, D0, days=14):
    res = numpy.empty((days+1, 4), dtype=numpy.float64)
    cum = numpy.empty((days+1, 1), dtype=numpy.float64)
    res[0, :] = [S0, I0, R0, D0]
    cum[0, 0] = I0
    N = sum(res[0, :])
    for t in range(1, res.shape[0]):
        dR = res[t-1, 1] * mu
        dD = res[t-1, 1] * nu
        dI = res[t-1, 0] * res[t-1, 1] / N * beta
        res[t, 0] = res[t-1, 0] - dI
        res[t, 1] = res[t-1, 1] + dI - dR - dD
        res[t, 2] = res[t-1, 2] + dR
        res[t, 3] = res[t-1, 3] + dD
        cum[t, 0] = cum[t-1, 0] + dI
    res[:, 1] = cum[:, 0]
    return res

beta = 0.5
mu = 1./14
nu = 1./21
S0 = 9990
I0 = 10
R0 = 0
D0 = 0

sim = simulation_cumulee(beta, mu, nu, S0, I0, R0, D0, 60)
plot_simulation(sim, dates[60], safe=False, two=True,
                title="Simulation pour essayer");_____no_output_____

On compare avec les vraies données en gras._____no_output_____

beta = 0.04
mu = 0.03
nu = 0.0001
S0, I0, R0, D0 = france[120, :]
sim = simulation_cumulee(beta, mu, nu, S0, I0, R0, D0, 120)
plot_simulation(sim, dates[120], safe=False, two=True, true_data=france[120:240],
                title="Simulation et vraies données (en gros)");_____no_output_____

Pas simple de choisir des paramètres pour approximer la courbe._____no_output_____

def error(data, simulation):
    err = (data[:, 1] - simulation[:, 1]) ** 2 + (data[:, 3] - simulation[:, 3]) ** 2
    total = (numpy.sum(err) / data.shape[0]) ** 0.5
    return total

beta = 0.5
mu = 1./14
nu = 1./21
S0 = 9990
I0 = 10
R0 = 0
D0 = 0

sim = simulation_cumulee(beta, mu, nu, S0, I0, R0, D0, 30)
plot_simulation(sim);_____no_output_____

On regarde si on arrive à retrouver les paramètres de la simulation._____no_output_____### Q3, Q4, Q5 : optimisation sur des données synthétiques_____no_output_____

from tqdm import tqdm  # pour avoir une barre de progression

def optimisation(true_data, i_range=(0, 0.2), beta_range=(0, 0.5),
                 mu_range=(0., 0.2), nu_range=(0., 0.2),
                 max_iter=1000, error_fct=error):
    N = sum(true_data[0, :])
    rnd = numpy.random.rand(max_iter, 4)
    for i, (a, b) in enumerate([i_range, beta_range, mu_range, nu_range]):
        rnd[:, i] = rnd[:, i] * (b - a) + a
    err_min = None
    for it in tqdm(range(max_iter)):
        i, beta, mu, nu = rnd[it, :]
        dI0 = true_data[0, 0] * i
        D0 = true_data[0, 3]
        S0 = true_data[0, 0] - dI0
        I0 = true_data[0, 1] + dI0
        R0 = N - D0 - I0 - S0
        sim = simulation_cumulee(beta, mu, nu, S0, I0, R0, D0, days=true_data.shape[0] - 1)
        err = error_fct(true_data, sim)
        if err_min is None or err < err_min:
            err_min = err
            best = dict(beta=beta, mu=mu, nu=nu, I0=I0, i=i,
                        S0=S0, R0=R0, D0=D0, err=err, sim=sim)
    return best

beta = 0.04
mu = 0.07
nu = 0.04
S0 = 67e6
I0 = 100000
R0 = 10000
D0 = 10000
sim = simulation_cumulee(beta, mu, nu, S0, I0, R0, D0, 30)

res = optimisation(sim, max_iter=2000, error_fct=error, i_range=(0., 0.001))
sim_opt = res['sim']
del res['sim']
plot_simulation(sim_opt, dates[90], safe=False, two=True, true_data=sim,
                title="beta=%1.3f mu=%1.3f nu=%1.3f err=%1.3g i=%1.3g" % (
                    res['beta'], res['mu'], res['nu'], res['err'], res['i']));100%|██████████| 2000/2000 [00:00<00:00, 6170.32it/s]

Ca ne marche pas trop mal._____no_output_____### Q6 : sur des données réelles_____no_output_____

res = optimisation(france[180:], error_fct=error, i_range=(0, 0.0001))
sim = res['sim']
del res['sim']
plot_simulation(sim, dates[180], safe=False, two=True, true_data=france[180:],
                title="beta=%1.3f mu=%1.3f nu=%1.3f err=%1.3f i=%1.3f" % (
                    res['beta'], res['mu'], res['nu'], res['err'], res['i']));100%|██████████| 1000/1000 [00:01<00:00, 517.22it/s]

Ca ne marche pas super. On peut modifier l'erreur pour donner plus d'importance à la courbe des morts._____no_output_____

def error_norm(data, simulation):
    m1 = numpy.max(simulation[:, 1])
    m3 = numpy.max(simulation[:, 3])
    err = (data[:, 1] - simulation[:, 1]) ** 2 / m1 ** 2 + (data[:, 3] - simulation[:, 3]) ** 2 / m3 ** 2
    total = (numpy.sum(err) / data.shape[0]) ** 0.5
    return total

res = optimisation(france[150:], error_fct=error_norm, i_range=(0, 0.001))
sim = res['sim']
del res['sim']
plot_simulation(sim, dates[150], safe=False, two=True, true_data=france[150:],
                title="beta=%1.3f mu=%1.3f nu=%1.3f err=%1.3f i=%1.3f" % (
                    res['beta'], res['mu'], res['nu'], res['err'], res['i']));100%|██████████| 1000/1000 [00:02<00:00, 494.82it/s]

L'erreur est répartie sur l'ensemble de la courbe mais cela ne tient pas compte de la dynamique. Il faudrait prendre en compte les dérivées comme dans l'aticle [A Modified SIR Model for the COVID-19 Contagion in Italy](https://arxiv.org/pdf/2003.14391.pdf). Ou partir sur une approche plus directe comme celle proposée dans l'article [Estimating and Simulating a SIRD Model of COVID-19 for Many Countries, States, and Cities](https://web.stanford.edu/~chadj/sird-paper.pdf). $\frac{dS}{dt} = - \beta \frac{S I}{N}$ implique que $\Delta(S_t) = S_{t+1} - S_t = -\beta \frac{S_t I_t}{N}$. On peut alors calculer une sorte de $\beta_t$ dépendant du temps : $\beta_t = -N\frac{S_{t+1} - S_t}{S_t I_t}$. Le problème, c'est $I_t$ qu'on ne connaît pas puisque c'est le nombre de personnes contaminantes à un instant *t*. On ne connaît pas plus $S_t$ mais on peut le déduire si on connaît $I_t$. Cela dit, le paramètre $\nu$ est plus ou moins connu puisqu'il s'agit de l'inverse de la durée moyenne d'incubation jusqu'au décès. De là, on peut retrouver $\beta$. On poursuit en passant au logarithme $\ln \beta_t = \ln N + \ln (S_t - S_{t+1}) - \ln S_t - \ln I_t$. Le nombre de cas positifs découverts chaque jour correspond à $S_t - S_{t+1}$. On ajoute que $\forall t, N = S_t + I_t + R_t + D_t$ et $D_{t+1} - D_t = \nu I_t$. Donc :

$$\ln \beta_t = \ln N + \ln (S_t - S_{t+1}) - \ln (S_t - S_{t-1} + S_{t-1}) - \ln (D_{t+1} - D_t) + \ln \nu$$

Il y a deux inconnues, $\ln \beta - \ln \nu$ qu'on suppose constant pendant la période et $S_{t-1}$. Il suffit d'écrire ces équations sur quelques jours puis de résoudre le système d'équations.

$$\begin{array}{l}
\ln \beta - \ln \nu = \ln N + \ln (S_t - S_{t+1}) - \ln (S_t - S_{t-1} + S_{t-1}) - \ln (D_{t+1} - D_t) \\
\ln \beta - \ln \nu = \ln N + \ln (S_{t+1} - S_{t+2}) - \ln (S_{t+1} - S_{t} + S_{t} - S_{t-1} + S_{t-1}) - \ln (D_{t+2} - D_{t+1})
\end{array}$$

D'autres directions sont possibles comme [The Parameter Identification Problem for SIR Epidemic Models: Identifying Unreported Cases](https://www.math.u-bordeaux.fr/~pmagal100p/papers/MW-JMB-2018.pdf) qui propose d'estimer les paramètres sur une plus grande période.

D'autres articles : [Introduction to inference: parameter estimation](https://kingaa.github.io/clim-dis/parest/parest.html), [Data-based analysis, modelling and forecasting of the COVID-19 outbreak](https://europepmc.org/article/med/32231374), [Identification and estimation of the SEIRD epidemic model for COVID-19](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7392128/).

Quelques liens :

* [CovidTracker](https://covidtracker.fr/covidtracker-france/)
* [pyepidemics](https://github.com/collectif-codata/pyepidemics)
* [COVID-19 pandemics control](https://freakonometrics.hypotheses.org/61300)
* [modcov19](https://modcov19.math.cnrs.fr/)
* [aftercovid](http://www.xavierdupre.fr/app/aftercovid/helpsphinx/index.html)
* [COVID-19 - France](https://dashboard.covid19.data.gouv.fr/vue-d-ensemble?location=FRA)
* [Awesome COVID19 resources](https://github.com/collectif-codata/awesome-covid19)
_____no_output_____


# Notebook from Jarino/cgp-optimization
Path: RVCGP-solid-pso-coevol.ipynb

<code>
from Solid.ParticleSwarm import ParticleSwarm
from sklearn.metrics import mean_squared_error
from tengp.individual import IndividualBuilder, NPIndividual
from tengp import Parameters, FunctionSet
import numpy as np

np.warnings.filterwarnings('ignore')_____no_output_____def fitness(individual, x, y):
    output = individual.transform(x)
    try:
        return mean_squared_error(output, y)
    except ValueError:
        return 10e10

class Algorithm(ParticleSwarm):
    """
    Optimize the CCGP individual
    """
    def custom_init(self,
                    params,
                    bounds,
                    x_train,
                    y_train,
                    fitness_fn,
                    predictors, trainers):
        self.cache = {}
        self.params = params
        self.bounds = bounds
        self.x_train_full = x_train
        self.y_train_full = y_train
        self.fitness_fn = fitness_fn
        self.predictors = predictors
        self.trainers = trainers
        self.predictors.evaluate_fitness(self.trainers)
        self.x_train_pred, self.y_train_pred = self.predictors.best_predictors_data()

    def _objective(self, member):
        """
        Calculate the fitness (the less, the better)
        """
        member = np.clip(member, self.lower_bound, self.upper_bound)

        key = hash(member.tostring())

        if key in self.cache:
            return self.cache[key]

        individual = NPIndividual(member.tolist(), self.bounds, self.params)

        fitness = self.fitness_fn(individual, self.x_train_pred, self.y_train_pred)

        self.cache[key] = fitness

        return fitness

    def _generation_callback(self):
        # add new trainer
        if self.cur_steps % 50 == 0:
            #import pdb; pdb.set_trace()
            genes = np.clip(self.global_best[0], self.bounds[0], self.bounds[1]).tolist()
            individual = NPIndividual(genes, self.bounds, self.params)
            individual.fitness = self.fitness_fn(individual, self.x_train_full, self.y_train_full)

        # update random population
        if self.cur_steps % 50 == 0:
            ts.update_random_population()

        # evolution step of predictors
        if self.cur_steps % 50 == 0:
            self.predictors.predictors_evolution_step(self.trainers)
            self.x_train_pred, self.y_train_pred = self.predictors.best_predictors_data()

        #print(self.cur_steps)_____no_output_____from trainers import TrainersSet

ts = TrainersSet(ib, 8, fitness, x_train, y_train)_____no_output_____import ga

predictors = ga.GaPredictors(x_train, y_train, 4, 24)_____no_output_____# define CCGP system
from experiment_settings import nguyen7_funset
import symreg
import random

params = Parameters(2, 1, 1, 50, nguyen7_funset, real_valued=True, max_back=20)

ib = IndividualBuilder(params)
initial_individual = ib.create()
bounds = initial_individual.bounds
individual_size = len(initial_individual.genes)_____no_output_____x_train, y_train, _, _ = symreg.get_benchmark_nguyen7(random, None)
x_train = np.c_[np.ones(len(x_train)), x_train]_____no_output_____algo = Algorithm(swarm_size=50,
                member_size=individual_size,
                lower_bound=bounds[0],
                upper_bound=bounds[1],
                c1=1,
                c2=1,
                c3=1,
                max_steps=2000)

algo.custom_init(params, bounds, x_train, y_train, fitness, predictors, ts)_____no_output_____%%time

champion_x, champion_f = algo.run(verbose=False)TERMINATING - REACHED MAXIMUM STEPS
CPU times: user 5min 35s, sys: 8min 8s, total: 13min 44s
Wall time: 3min 44s
champion_i = NPIndividual(np.clip(champion_x, bounds[0], bounds[1]).tolist(), bounds, params)
exact_fitness = fitness(champion_i, x_train, y_train)
print('predicted fitness', champion_f)
print('exact fitness', exact_fitness)predicted fitness 0.00033810095699820223
exact fitness 1.6720174010314826

</code>


# Notebook from imppresser/Quantum-1
Path: tutorial/mbqc/QAOA_CN.ipynb

# 基于测量的量子近似优化算法

 Copyright (c) 2021 Institute for Quantum Computing, Baidu Inc. All Rights Reserved. _____no_output_____## 多项式无约束布尔优化问题

在应用数学和理论计算机科学中，**组合优化问题（combinatorial optimization problem）** 是在一个离散的解空间中寻找最优解的一类问题。在[量桨平台教程：量子近似优化算法](https://qml.baidu.com/tutorials/combinatorial-optimization/quantum-approximate-optimization-algorithm.html)中，已经介绍过了一般的组合优化问题。在这里，我们关注一类具体的问题：**多项式无约束布尔优化问题（polynomial unconstrained boolean optimization problem, PUBO）**。

给定一个变量为 $x = \{x_1,\cdots,x_n\}$ 的 $n$ 元多项式，

$$
C(x) = \sum_{\lambda \in \Lambda } \alpha_{\lambda} \prod_{j \in \lambda} x_j,\tag{1}
$$

其中每个变量 $x_i \in \{0,1\}$，$\underset{j \in \lambda}{\prod} x_j$  为一个单项式，$\lambda \subseteq [n]:= \{1, 2, ..., n\}$ 为一个指标集，$\Lambda$ 为指标集的集合，$\alpha_\lambda$ 为每个单项式对应的实系数。在 PUBO 中，$C(x)$ 称为目标多项式，我们需要寻找一组最优解 $x^* = \{x_1^*, x_2^*, ..., x_n^*\} $ 使得目标多项式的取值最大，即

$$
x^* = \underset{x}{\text{argmax}} \ C(x).\tag{2}
$$

多项式无约束布尔优化问题是一种应用极为广泛的数学优化模型，对这种模型的高效求解有助于解决很多现实中的问题。如果需要求解的目标多项式次数为二，则称为二次多项式组合优化。这类模型可以描述例如最大独立集（MIS）、最大割（Max-Cut）、最大点集覆盖（Max-Coverage）等很多图论中的组合优化问题，并在诸如统计物理，网络设计，超大规模集成电路（VLSI）设计，数据聚类分析，金融分析和机器调度等方面有广泛应用。如果目标多项式次数大于二，这样的多项式组合优化则在信号处理（SP）和计算机视觉（CV）中图像重构等领域有重要应用。但是一般 PUBO 的求解是 NP-困难的，目前，精确求解这类问题没有有效的多项式时间复杂度的算法。_____no_output_____## 量子近似优化算法

在 2014 年 Farhi 及其合作者通过经典计算与量子计算混合迭代的思路，提出了量子近似优化算法 **(quantum approximate optimization algorithm, QAOA)** [1]，一方面希望利用量子计算机的能力更好地解决组合优化问题，另一方面也希望通过此问题展现量子计算机的优势。关于 QAOA 原理的详细论述请参见[量桨平台教程：量子近似优化算法](https://qml.baidu.com/tutorials/combinatorial-optimization/quantum-approximate-optimization-algorithm.html)，此处我们简要回顾一下该算法的基本思想和实现步骤。

要将组合优化问题翻译为量子版本，我们需要先将待优化的变量编码为量子态，以及将目标多项式编码为系统的哈密顿量。接下来，我们对这两点逐一进行讲解。

### 变量编码为量子态

对于变量 $x$，根据定义，它的每个比特的取值均为 $0$ 或 $1$，这便很自然地与量子态的 $|0\rangle$, $|1\rangle$ 系统相对应。于是长度为 $n$ 的布尔变量 $x$ 可以对应于一个 $n$ 量子比特构成的量子态，即：

$$
|x\rangle = |x_1x_2...x_n\rangle,\tag{3}
$$

从而寻找原问题的最优解 $x^{*}$ 就相当于寻找某一个量子态 $|x^{*} \rangle$。

### 目标多项式编码为系统哈密顿量

对于目标多项式 $C(x)$，我们可以将其编码到一个系统哈密顿量 $H_C$ 的对角元上，并使其满足对任意的 $x$，

$$
H_C |x\rangle = C(x) |x\rangle.\tag{4}
$$

值得注意的是，假设原问题的一个最优解是 $x^{*}$，那么我们有：

$$
\langle x^{*} | H_{C} |x^{*} \rangle = C(x^{*}).\tag{5}
$$

于是寻找原组合优化问题的最优解 $x^{*}$ 等同于寻找系统哈密顿量 $H_C$ 的最大本征值对应的本征态 $|x^{*} \rangle$，即：

$$
|x^{*}\rangle = \underset{|x\rangle}{\text{argmax}} \ \langle x | H_C | x \rangle.\tag{6}
$$

**注意**：以上给出了编码目标多项式到系统哈密顿量的方式，但是如何更方便的找到 $H_C$ 的表达式呢？我们不妨先考虑一个简单的例子。假设目标多项式为 $C(x) = 1-2x$，即 $C(0) = 1, C(1) = -1$，则可以跟据定义快速找到对应的系统哈密顿量为 $H_C = Z$，其中 $Z$ 为 Pauli $Z$ 门。那么一般地，我们可以先对目标多项式 $C(x)$ 进行变量替换 $1-2x_i = z_i$ (即 $x_i = (1-z_i)/2$)，其中 $z_i \in \{-1, 1\} $，然后将 $z_i$ 变量替换为 Pauli $Z_i$ 算符，其中下角标 $i$ 表示对应的量子系统。可以验证这样构造的哈密顿量 $H_C$ 刚好满足 $H_C |x\rangle = C(x) |x\rangle$。

为了方便使用，我们在 `qaoa` 中，定义了 `get_cost_hamiltonian` 函数，用来获取给定多项式的系统哈密顿量，我们可以直接调用它。

```python
from paddle_quantum.mbqc.QAOA.qaoa import get_cost_hamiltonian
```_____no_output_____### QAOA 电路

在将变量和目标多项式分别编码到量子态和系统哈密顿量上之后，我们引入辅助哈密顿量 $H_B$，具有如下形式：

$$
H_B = \bigotimes_{j=1}^n X_j,\tag{7}
$$

其中，$X_j$ 为作用到第 $j$ 个比特上的 Pauli X 门。依据绝热定理 [2,3]，在 QAOA 中，我们希望将 $H_B$ 最大本征值对应的本征态 $|+\rangle^{\otimes n}$ 演化到 $H_C$ 最大本征值对应的本征态。因此，我们可以构造量子电路实现如下的量子态演化，

$$
|\gamma,\beta\rangle :=  \left(\prod_{i=1}^p U_B(\beta_i)U_{C}(\gamma_i)\right)|+\rangle^{\otimes n},\tag{8}
$$

其中 $U_{C}(\gamma) = e^{-i\gamma H_{C}}$，$U_B(\beta) = e^{-i \beta H_B}$，$\beta,\gamma$ 为待训练的参数 ，$p$ 为给定的算法深度，$p$ 越大，算法求得的解越精确，但是计算量也越大。

关于电路模型下 QAOA 的详细介绍，请参见[量桨平台教程：量子近似优化算法](https://qml.baidu.com/tutorials/combinatorial-optimization/quantum-approximate-optimization-algorithm.html)，我们这里不做赘述。_____no_output_____##  基于测量的量子近似优化算法

如前所述，[基于测量的量子计算](MBQC_CN.ipynb)提供了一种不同于量子电路模型的计算方式。由于该模型的通用性，任何量子电路模型都可以找到与其相对应的 MBQC 版本。文献 [4] 提出了一种基于测量的变分量子本征求解器（measurement-based variational quantum eigensolver, MB-VQE），类似地，我们在此给出**基于测量的量子近似优化算法（measurement-based quantum approximate optimization Algorithm, MB-QAOA）** 作为 MBQC 模型的入门算法教程之一。注意到量子电路模型中，不同的电路可以实现完全相同的演化效果。同样地，我们也可以找到很多种不同的 MBQC 算法完成相同的功能。在这里，我们给出一种较为简洁的 MBQC 版本的 QAOA 算法，并使用我们设计的 MBQC 模块对该算法进行模拟。

### 技术思路

注意到 QAOA 的核心是对初始态完成 $U_{C}$ 和 $U_B$ 的交替演化。我们先用下面两个引理分别讲解在 MBQC 模型中该如何简单地完成这两种演化过程。感兴趣的读者可以尝试自行证明或者参见 [5]。

**引理 1:** 假设 $|\psi\rangle_{1 \cdots k}$ 为一个 $k$ 比特的量子态，对其进行 $e^{i\theta Z_1Z_2\cdots Z_k}$ 的演化，可以通过如下测量方案实现：

$$
M_0^{YZ}(2\theta) \left(\prod_{j=1}^{k} CZ_{0,j}\right) \Big(|+\rangle_0 \otimes |\psi\rangle_{1 \cdots k}\Big) \longrightarrow \left(\prod_{j=1}^{k} Z_j\right)^{s_0}\, e^{i\theta Z_1Z_2\cdots Z_k}\, |\psi\rangle_{1 \cdots k},\tag{9}
$$

即我们先在 $0$ 系统上准备一个加态，然后对这个比特和输入态 $|\psi\rangle_{1 \cdots k}$ 的每一个比特做 CZ 门，最后用 YZ 平面上的投影测量来测量 $0$ 系统上的比特，测量角度为 $2\theta$，测量完成后，处于 $1,\cdots,k$ 系统上的量子态将演化为箭头右边的状态，也就是 $e^{i\theta Z_1Z_2\cdots Z_k}\, |\psi\rangle_{1 \cdots k}$ 附加上 $\left(\prod_{j=1}^{k} Z_j\right)^{s_0}$ 的副产品，其中 $s_0 \in \{0,1\}$ 为 $0$ 系统的测量结果。

注意到 $U_C$ 的本质是跟据目标函数连续完成多个不同的 $e^{i\theta Z_1Z_2\cdots Z_k}$ 的演化，因此将引理 1 中的实现方式重复使用多次，即可实现 $U_C$。

**引理 2:** 设 $1$ 系统的量子态 $|\psi\rangle_1$ 为输入的单比特量子态，对其进行 $R_x(\theta_2)R_z(\theta_1)$ 的演化，可以通过如下测量方案实现：

$$
M_2^{XY}\big((-1)^{1+s_1}\theta_2\big) M_1^{XY}(-\theta_1) \Big(CZ_{23} CZ_{12}\Big) \Big(|\psi\rangle_1 \otimes |+\rangle_2 \otimes |+\rangle_3 \Big) \longrightarrow Z^{s_1} X^{s_2} R_{x}(\theta_2) R_z(\theta_1) |\psi\rangle_3.\tag{10}
$$

即我们先在 $2$ 和 $3$ 系统上分别准备一个加态，对 $1$， $2$ 系统和 $2$， $3$ 系统分别作用 $CZ$ 门，然后用 $XY$ 平面上的投影测量来测量 $1$ 系统上的比特，测量角度为 $-\theta_1$， 记录测量结果为 $s_1$，再用 $XY$ 平面上的投影测量来测量 $2$ 系统上的比特，测量角度为 $(-1)^{1+s_1}\theta_2$， 记录测量结果为 $s_2$，测量完成后在 $3$ 系统上的量子态将演化为箭头右边的状态，也就是 $R_{x}(\theta_2) R_z(\theta_1) |\psi\rangle_3$ 附加上副产品 $Z^{s_1} X^{s_2}$。

注意到 $U_B$ 的本质是在不同比特上实现 $R_x$ 旋转门，因此我们可以利用引理 2 中的方式，令 $\theta_1 = 0$ 来实现 $U_B$。_____no_output_____通过以上两个引理，相信大家对 MB-QAOA 的实现有了一些想法。接下来我们跟据 MBQC 模型“三步走”流程（量子图态准备、单比特测量、副产品纠正）进行详细介绍。

### 量子图态准备

由于量子图态和图一一对应，所以我们给出对应的图即可，我们称这个图为 **MB-QAOA 图**。

#### 单层 MB-QAOA 图的构造

根据目标多项式 $C(x)$ 中变量的个数 $n$，依次纵向排列 $n$ 个绿色、 $n$ 个蓝色节点和 $n$ 个灰色节点，并依次记绿色节点为 $G^v$，蓝色节点为 $B^v$，灰色节点为 $H^v$，其中 $1 \leq v \leq n$。连接并排的绿色和蓝色节点，以及蓝色和灰色节点；
对目标多项式 $C(x)$ 做变量替换 $x = (1-z)/2$，得到新的目标多项式记为

$$
C(z) = c + \sum_{v} \eta_v z_v + \sum_{S} \eta_S \prod_{j \in S} z_j,\tag{11}
$$

其中 $c$ 为常数项，$1 \leq v \leq n$ 为线性项的指标，对应系数为 $\eta_v$，$S$ 为非线性项的指标集，对应系数为 $\eta_S$。我们要求补齐线性项中未出现的元素，并且设其对应系数为 $0$。那么对于 $C(z)$ 中除常数项外的每个非线性项的单项式 $\prod_{j \in S} z_j$，我们在绿色节点的左边添加一个新的红色节点，并记为 $R^S$，连接红色节点 $R^S$ 和绿色节点 $G^v$ ($ \forall v\in S$)。

以上构造出来的图称为单层 MB-QAOA 图，根据之前讨论，我们将会通过测量红色节点来完成 $U_C$ 的演化，通过测量绿色和蓝色节点完成 $U_B$ 的演化，并且演化后的态存储在灰色节点中。为了方便理解，以下图 1 给出了一个具体的单层 MB-QAOA 图的例子。

![QAOA graph](./figures/mbqc-fig-qaoa_graph_1.jpg)

图 1：一个单层 MB-QAOA 图的例子，其中变量替换后的目标多项式为 $C(z) = z_2 + z_1 z_3 + 5 z_3 z_4 - 2 z_1 z_2 z_4$。

注：单项式的系数对 MB-QAOA 图结构的构造没有影响，但这些系数会影响到测量节点时的角度（见下文）。

#### $p$ 层 MB-QAOA 图的构造

由于电路模型下的 QAOA 会对 $U_C$，$U_B$ 交替演化 $p$ 次，MBQC 模型下的 $p$ 层 QAOA 也是如此。图 $1$ 所示为 $p=1$ 情况，对于一般的 $p>1$，我们需要在右侧将单层 MB-QAOA 图重复 $p$ 次，并且下一层的绿色节点（对应输入态）要和上一层的灰色节点（对应输出态）重合，以此保证量子态可以进行连续的演化。最终的量子态会保存在最右端的灰色节点上。为了方便理解，以下图 2 给出了一个具体的 $p$ 层 MB-QAOA 图的例子。

![QAOA graph](./figures/mbqc-fig-qaoa_graph_p.jpg)

图 2：一个 $p$ 层 MB-QAOA 图的例子，其中变量替换后的目标多项式为$C(z) = z_2 + z_1 z_3 + 5 z_3 z_4 - 2 z_1 z_2 z_4$。

在 `qaoa` 中，我们定义了 `preprocess` 函数，用于生成上述的 MB-QAOA 图，我们可以直接调用：

```python
from paddle_quantum.mbqc.QAOA.qaoa import preprocess
```_____no_output_____### 单比特测量

量子图态构造好之后，下一步就是设计每个比特上的测量方式了。根据以上两个引理，我们可以计算出每一步测量对应的角度，在这之中要注意的是测量角度对前面测量结果的依赖关系。

假设 MB-QAOA 电路总深度为 $p$ 层，测量顺序按 MB-QAOA 图从左往右，从上到下的顺序按列进行测量，考虑第 $1 \leq l \leq p$ 层的 MB-QAOA图，具体每个节点的测量方式见表 1：

|节点|测量基|测量角度|测量结果|
|:---:|:---:|:---:|:---:|
|$$R_l^S$$|$$M^{YZ}$$|$$T \Big(1+\sum_{v \in S}\sum_{k=1}^{l-1}s(B_k^v)\Big) \cdot 2 \eta_{S} \gamma_{l} $$|$$s(R_l^{S})$$|
|$$G_l^v$$|$$M^{XY}$$|$$T \Big(1+\sum_{k=1}^{l-1}s(B_k^v)\Big) \cdot 2 \eta_v \gamma_l $$|$$s(G_l^{v})$$|
|$$B_l^v$$|$$M^{XY}$$|$$T \Big(1+\sum_{k=1}^{l}\sum_{S:v \in S}s(R_k^S) + \sum_{k=1}^{l}s(G_k^v)\Big) 2 \beta_l$$|$$s(B_l^{v})$$|

表 1：MB-QAOA 详细的测量方式

其中 $1 \leq v \leq n$ 为变量替换后目标多项式的线性项指标，对应系数为 $\eta_v$，$S$ 为变量替换后目标多项式的非线性项指标集，对应系数为 $\eta_S$，$\beta_l,\gamma_l$ （$1 \leq l \leq p$）为待训练的参数，$M^{XY}$ 表示在 $XY$ 平面内的测量，$M^{YZ}$ 表示在 $YZ$ 平面内的测量，求和 $\sum_{k=1}^0$ 约定为 $0$，函数 $T(x) = (-1)^x$。$R_k^S$ 代表图 2 中的红色节点，$s(R_k^S)$ 代表其对应测量结果；$G_k^v$ 代表图 2 中的绿色节点，$s(G_k^v)$ 代表其对应测量结果；$B_k^v$ 代表图 2 中的蓝色节点，$s(B_k^v)$ 代表其对应测量结果。

为了方便 MB-QAOA 中测量角度的计算，我们在 `qaoa` 中，定义了测量角度函数 `adaptive_angle`。我们只需输入测量结果的字典, 待测量的比特标签, 待训练的角度参数和多项式系数等信息，即可按上表自动计算出测量角度。

我们可以直接调用：

```python
from paddle_quantum.mbqc.QAOA.qaoa import adaptive_angle
```_____no_output_____### 副产品纠正

通过以上全部测量后（除最后一列灰色节点外），MB-QAOA 图中最末端灰色节点对应的量子态会演化为 $|\gamma,\beta\rangle$ 附加上测量中产生的一些副产品。在第 $v$ 个节点上的副产品为 $X^{x}Z^{z}  $，其中指数：

$$
x = \sum_{k=1}^{p} s(B_{k}^{v}), \quad z = \sum_{k=1}^{p} \sum_{S: v\in S} s(R_k^S) + \sum_{k=1}^{p} s(G_k^v).\tag{12}
$$

我们需要在测量完成后对这些副产品进行纠正，最后获得的量子态将为我们预期的 $|\gamma,\beta\rangle$。为了方便使用，我们在 `qaoa` 中定义了 `byproduct_power` 用于求解副产品纠正的指数。我们只需要输入待纠正的副产品项，待纠正的比特位置，MB-QAOA 图，测量结果的字典，以及电路深度，即可计算出对应纠正项的指数：

```python
from paddle_quantum.mbqc.QAOA.qaoa import byproduct_power
```_____no_output_____## 代码实现

下面我们用 MBQC 模块，完整地实现上述 MB-QAOA 算法。_____no_output_____### 量子态的演化_____no_output_____

# 引入模拟相关的模块
from paddle_quantum.mbqc.simulator import MBQC
from paddle_quantum.mbqc.utils import pauli_gate, kron, basis, permute_systems
from paddle_quantum.mbqc.QAOA.qaoa import preprocess, adaptive_angle, byproduct_power_____no_output_____# 定义 MBQC 模型下的 QAOA 函数
def mbqc_qaoa(poly_x, depth, gamma, beta):

    # 对目标多项式函数进行预处理，获得变量替换后的多项式 C(z) 和 MB-QAOA 图
    poly_classified, qaoa_graph = preprocess(poly_x, depth)
    var_num, cons_item, linear_items, non_linear_items = poly_classified

    # 实例化一个 MBQC 模型并设置图
    mbqc = MBQC()
    mbqc.set_graph(graph=qaoa_graph)

    # 测量每一个节点
    for i in range(1, depth + 1):

        # 测量红色节点
        for item in non_linear_items:
            angle_r = adaptive_angle(which_qubit=('R', item[0], i),
                                     graph=mbqc.get_graph(),
                                     outcome=mbqc.get_classical_output(),
                                     theta=gamma[i - 1],
                                     eta=to_tensor(item[1], dtype='float64')
                                     )
            mbqc.measure(which_qubit=('R', item[0], i), basis_list=basis('YZ', angle_r))

        # 测量绿色节点
        for v in range(1, var_num + 1):
            angle_g = adaptive_angle(which_qubit=('G', v, i),
                                     graph=mbqc.get_graph(),
                                     outcome=mbqc.get_classical_output(),
                                     theta=gamma[i - 1],
                                     eta=linear_items[v])
            mbqc.measure(which_qubit=('G', v, i), basis_list=basis('XY', angle_g))

        # 测量蓝色节点
        for v in range(1, var_num + 1):
            angle_b = adaptive_angle(which_qubit=('B', v, i),
                                     graph=mbqc.get_graph(),
                                     outcome=mbqc.get_classical_output(),
                                     theta=beta[i - 1],
                                     eta=to_tensor([1], dtype='float64'))
            mbqc.measure(which_qubit=('B', v, i), basis_list=basis('XY', angle_b))

    # 纠正副产品
    for v in range(1, var_num + 1):
        pow_x = byproduct_power(gate='X', v=v, graph=mbqc.get_graph(), outcome=mbqc.get_classical_output(), depth=depth)
        pow_z = byproduct_power(gate='Z', v=v, graph=mbqc.get_graph(), outcome=mbqc.get_classical_output(), depth=depth)
        mbqc.correct_byproduct(gate='X', which_qubit=('H', v, depth), power=pow_x)
        mbqc.correct_byproduct(gate='Z', which_qubit=('H', v, depth), power=pow_z)

    output_label = [('H', i, depth) for i in range(1, var_num + 1)]

    # 对输出量子态进行系统顺序调整并返回
    state_out = permute_systems(mbqc.get_quantum_output(), output_label)

    return state_out.vector_____no_output_____

### MB-QAOA 优化网络的搭建

优化网络搭建的流程与量桨上大部分机器学习的教程类似，唯一不同的是这里需要使用 `mbqc_qaoa` 函数作为前向传播函数。当算法完成后，我们会得到演化后的量子态，计算系统哈密顿量 $H_{C}$ 在该量子态下的期望值，将其作为损失函数接入飞桨优化器，对测量角度参数 $\gamma_1, ... \gamma_p, \beta_1, ... \beta_p$ 进行训练优化，最终得到优化后的参数。

在 `qaoa` 中，我们定义了期望函数 `expecval`，用来计算系统哈密顿量 $H_C$ 在演化后的量子态 $|\gamma,\beta\rangle$ 下的期望值 $\langle \gamma,\beta| H_C| \gamma,\beta\rangle$。

```python
from paddle_quantum.mbqc.QAOA.qaoa import expecval
```

MB-QAOA 优化网络搭建的代码实现如下：_____no_output_____

# 引入飞桨优化器模块
from paddle import nn
# 引入期望函数
from paddle_quantum.mbqc.QAOA.qaoa import expecval_____no_output_____# 定义 MB-QAOA 优化网络
class MB_QAOA_Net(nn.Layer):

    def __init__(self, depth, dtype="float64"):

        super(MBQC_QAOA_Net, self).__init__()

        self.depth = depth

        # 定义训练参数
        self.gamma = self.create_parameter(shape=[self.depth],
                                           default_initializer=nn.initializer.Uniform(low=0.0, high=2 * pi),
                                           dtype=dtype,
                                           is_bias=False)
        self.beta = self.create_parameter(shape=[self.depth],
                                          default_initializer=nn.initializer.Uniform(low=0.0, high=2 * pi),
                                          dtype=dtype,
                                          is_bias=False)

    # 定义优化网络的前向传播机制，输入为目标多项式函数
    def forward(self, poly):

        # 执行 MB-QAOA 算法并返回演化后的量子态
        vector_out = mbqc_qaoa(poly, self.depth, self.gamma, self.beta)

        # 获取系统哈密顿量
        HC = get_cost_hamiltonian(poly)

        # 计算系统哈密顿量在演化后量子态下的期望，作为损失函数
        loss = - expecval(vector_out, HC)

        # 返回损失函数和量子态
        return loss, vector_out_____no_output_____

### 答案解码

运行完 MB-QAOA 优化网络后，我们得到最优的参数 $\gamma^*,\beta^*$，以及对应的输出态 $|\gamma^*,\beta^*\rangle$，但是我们还需要从输出态中获取 PUBO 问题的最终解，所以我们需要对量子态 $|\gamma^*,\beta^*\rangle$ 重复测量多次后，统计对应结果的概率分布，并将最大概率对应的比特串作为原问题的最优解。在 `qaoa` 中我们定义了 `get_solution_string` 函数，用来解码量子答案，在具体的例子当中，我们可以根据自己的需要使用。

```python
from paddle_quantum.mbqc.QAOA.qaoa import get_solution_string
```_____no_output_____## 示例

在介绍完上述基于测量的量子近似优化算法之后，我们将该算法应用到两个具体的问题当中。在这两个例子中，我们可以直接调用 `qaoa` 中的 `MB_QAOA_Net` 运行 MB-QAOA 及参数的优化过程，具体示例请参见：

- [MBQC 模型下求解多项式组合优化问题](PUBO_CN.ipynb)_____no_output_____---

## 参考文献

[1] Farhi, Edward, et al. "A quantum approximate optimization algorithm." [arXiv preprint arXiv:1411.4028 (2014).](https://arxiv.org/abs/1411.4028)

[2] Farhi, Edward, et al. "Quantum computation by adiabatic evolution." [arXiv preprint quant-ph/0001106 (2000).](https://arxiv.org/abs/quant-ph/0001106)

[3] Duan, Runyao. "Quantum adiabatic theorem revisited." [arXiv preprint arXiv:2003.03063 (2020).](https://arxiv.org/abs/2003.03063)

[4] Ferguson, R. R., et al. "Measurement-based variational quantum eigensolver." [Physical Review Letters 126.22 (2021): 220501-220501.](https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.126.220501)

[5] Browne, Dan, and Hans Briegel. "One-way quantum computation." [Quantum Information: From Foundations to Quantum Technology Applications (2016): 449-473.](https://onlinelibrary.wiley.com/doi/abs/10.1002/9783527805785.ch21)_____no_output_____


# Notebook from dwengovzw/PythonNotebooks
Path: STEM/Klimaat/0500_SilvrettagletsjerOpdracht.ipynb

_____no_output_____

SMELTENDE GLETSJERS: DE SILVRETTAGLETSJER

_____no_output_____

In deze notebook maak je een puntenwolk van data uit een csv-bestand. Je berekent hoeveel de Silvrettagletsjer als is verdund sinds 1919.

_____no_output_____Sinds de industriële revolutie is de concentratie broeikasgassen in de atmosfeer stelselmatig toegenomen. Sinds 1880 is de gemiddelde globale temperatuur ongeveer met 0,85 °C gestegen. Deze opwarming gaat gepaard met een opwarming van de oceanen, een stijging van het zeeniveau met 20 cm, het meer voorkomen van extreme weersomstandigheden en een afname van 40 % van het Arctische zee-ijs. Ook het gletsjerijs smelt, bijna overal ter wereld.

Het smeltwater afkomstig van gebergtegletsjers zal in belangrijke mate bepalen hoeveel het zeeniveau in de toekomst zal stijgen. Mogelijke scenario's spreken van een stijging tot 30 cm door het afsmelten van de gebergtegletsjers.

Bovendien hebben gletsjers een impact op lokale watervoorraden en zijn ze belangrijk voor het toerisme.

De snelheid waarmee het volume van een gletsjer afneemt o.i.v. de globale temperatuurstijging verschilt van gletsjer tot gletsjer. Lokale factoren spelen hierin immers een rol: bv. de oriëntatie van de gletsjer, de mate waarin de gletsjer in de schaduw ligt ... [1]. _____no_output_____De 3 km lange Silvrettagletsjer ligt in het noordoosten van Zwitserland. Sinds 1954 heeft de gletsjer al 400 m aan lengte ingeboet [3]._____no_output_____Lander Van Tricht (VUB) verschafte ons data van zijn onderzoek naar hoe de Silvrettagletsjer evolueert. Sinds 1919 werd geregistreerd met hoeveel meter de gletsjer jaarlijks verdunt [2]. _____no_output_____

Om de volgende opdracht tot een goed einde te brengen, heb je best eerst de voorgaande notebooks onder de knie. Bekijk zeker eerst de notebook over de Morteratschgletsjer.

_____no_output_____## Opdracht

- Je vindt de data van de metingen in het bestand `silvretta.csv` in de map `data`.
- Maak een puntenwolk die de verdunning uitzet in functie van het jaartal. Zorg voor een aantrekkelijke grafiek.
- Bereken met Python de totale verdunning van de Silvrettagletsjer sinds 1919.
- Bereken met Python de totale verdunning van de Silvrettagletsjer sinds 1954.
- Bereken met Python hoeveel centimeter de gletsjer is verdund in de periode van 2006 t.e.m. 2016.    _____no_output_____

Referentielijst

_____no_output_____[1] Zekollari, H., Fürst, J. J., & Huybrechts, P. (2014). Modelling the evolution of Vadret da Morteratsch, Switzerland,
        since the little ice age and into the future. *Journal of Glaciology*.60, 244. doi:10.3189/2014JoG14J053.

[2] Van Tricht, L. (2019). Via e-mail.

[3] Schweizer Gletscher. Glacier Monitoring Switzerland (Glamos).
         Geraadpleegd op 18 augustus 2019 via https://www.glamos.ch/factsheet#/A10g%2F05._____no_output_____

Notebook Python en STEM, zie AI Op School, van F. wyffels & N. Gesquière is in licentie gegeven volgens een Creative Commons Naamsvermelding-NietCommercieel-GelijkDelen 4.0 Internationaal-licentie. _____no_output_____

Met steun van

_____no_output__________no_output_____