# Notebook from mguo123/pan_omics_psych
Path: notebooks/9A_make_mpra_tracks.ipynb

# 9A_make_mpra_tracks

2/16/2021

how to make rna tracks : will be displaying TPM values on exons in hg19

Option 1: Wiggle (not done b/c wiggle not easy to process)
1. make wiggle file (https://genome.ucsc.edu/goldenPath/help/wiggle.html)
2. convert wig to BigWig (https://genome.ucsc.edu/goldenpath/help/bigWig.html)

Option 2: Bedgraph
1. make bedgraph (https://genome.ucsc.edu/goldenpath/help/bedgraph.html)
2. convert bedgraph to BigWig (https://genome.ucsc.edu/goldenpath/help/bigWig.html)
    To create a bigWig track from a bedGraph file, follow these steps:

1. Create a bedGraph format file following the directions here. When converting a bedGraph file to a bigWig file, you are limited to one track of data in your input file; therefore, you must create a separate bedGraph file for each data track.
2. Remove any existing track or browser lines from your bedGraph file so that it contains only data.
3. Download the bedGraphToBigWig program from the binary utilities directory.
4. Use the fetchChromSizes script from the same directory to create the chrom.sizes file for the UCSC database with which you are working (e.g., hg19). If the assembly genNom is hosted by UCSC, chrom.sizes can be a URL like http://hgdownload.soe.ucsc.edu/goldenPath/genNom/bigZips/genNom.chrom.sizes
5. Use the bedGraphToBigWig utility to create a bigWig file from your bedGraph file:
    `bedGraphToBigWig in.bedGraph chrom.sizes myBigWig.bw`
(Note that the bedGraphToBigWig program DOES NOT accept gzipped bedGraph input files.)
6. Move the newly created bigWig file (myBigWig.bw) to a web-accessible http, https, or ftp location.
7. Paste the URL into the custom track entry form or construct a custom track using a single track line.
8. Paste the custom track line into the text box on the custom track management page.

_____no_output_____

import pandas as pd
import os, glob, re
import subprocess
import pybedtools
import numpy as np_____no_output_____

# 0. get files
- exon file
- rna seq tpm matrix file
- chrom size file_____no_output_____

save_dir = '../data/processed/mpra_bigwigs'
if not os.path.exists(save_dir):
    os.makedirs(save_dir)
if not os.path.exists(os.path.join(save_dir, 'bedgraphs')):
    os.makedirs(os.path.join(save_dir, 'bedgraphs'))
if not os.path.exists(os.path.join(save_dir, 'bigwigs')):
    os.makedirs(os.path.join(save_dir, 'bigwigs'))_____no_output_____size_file = '../data/external/chrom_hg19.sizes'
exon_bed = '../data/external/gencode.v19.exon.bed'_____no_output_____exon_df = pybedtools.BedTool(exon_bed).to_dataframe()
exon_df[:5]_____no_output_____# rna_file = '../data/interim/rna/tissue_tpm_sym.csv'
# rna_df = pd.read_csv(rna_file)
# print(rna_df.shape)
# rna_df[:5]_____no_output_____mpra_df = pd.read_excel('/Users/mguo123/Google Drive/1_khavari/noncancer_project/miseq/novogene_071420/figure_tables/lib_df_annon_mprasig_FINAL_v020521.xlsx')
mpra_df_sel = mpra_df[['rowname','Chr','Position','Ref/Alt','Linked_SNP','diseases',
        'mpra_tissue',	'mpra_pval_str',	'mpra_pval_mean',
         'mpra_logfc_str',	'mpra_logfc_mean',	'bool_mpra_sig']]
mpra_df_sel['end'] = mpra_df_sel.Position +1
mpra_df_sel['neglog10pval'] = mpra_df_sel.mpra_pval_mean.apply(lambda x:-np.log10(x))
mpra_df_sel = mpra_df_sel[['Chr','Position','end','neglog10pval', 'mpra_logfc_mean','mpra_tissue','Linked_SNP','diseases']]
mpra_df_sel.columns = ['chr','start','stop','name', 'mpra_logfc_mean','mpra_tissue','Linked_SNP','diseases']
mpra_df_sel.to_csv(os.path.join(save_dir, 'mpra_df_sel.csv'))
mpra_df_sel[:5]/Users/mguo123/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:5: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """
/Users/mguo123/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:6: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

tissues = ['AST', 'SL',  'SLC','H9D0', 'H9D2', 'H9D4','H9D10','HEK293']_____no_output_____# # bedgraph defintion lines
# definition_str = "track type=bedGraph name={track_label:s} description=center_label visibility=display_mode color=r,g,b altColor=r,g,b priority=priority autoScale=on|off alwaysZero=on|off gridDefault=on|off maxHeightPixels=max:default:min graphType=bar|points viewLimits=lower:upper yLineMark=real-value yLineOnOff=on|off windowingFunction=maximum|mean|minimum smoothingWindow=off|2-16"
# definition_str.format(track_label='test', )_____no_output_____# txt = "For only {price:.2f} dollars!"
# print(txt.format(price = 49))_____no_output_____

# 1A. make bedGraph (no brower or track lines) per tissue

space delimited
`chrom chromStart chromEnd dataValue`

example:

```
chr19 49302000 49302300 -1.0
chr19 49302300 49302600 -0.75
```_____no_output_____

for tissue in tissues:
    print('*****', tissue)
    mpra_df_tissue = mpra_df_sel[mpra_df_sel.mpra_tissue.str.contains(tissue)]

    mpra_df_tissue = mpra_df_tissue[['chr','start','stop','name']]
    mpra_df_bed = pybedtools.BedTool.from_dataframe(mpra_df_tissue).sort()#.merge(c=4,o='sum')
    mpra_df_tissue = mpra_df_bed.to_dataframe()
    print('size mpra_df_bed sorted ',mpra_df_tissue.shape)
    mpra_df_tissue.to_csv(os.path.join(save_dir,'bedgraphs', tissue+'_unsorted.bedGraph'),index=False, header=False,sep=' ')
    ***** AST
size mpra_df_bed sorted  (66, 4)
***** SL
size mpra_df_bed sorted  (355, 4)
***** SLC
size mpra_df_bed sorted  (335, 4)
***** H9D0
size mpra_df_bed sorted  (105, 4)
***** H9D2
size mpra_df_bed sorted  (86, 4)
***** H9D4
size mpra_df_bed sorted  (188, 4)
***** H9D10
size mpra_df_bed sorted  (153, 4)
***** HEK293
size mpra_df_bed sorted  (273, 4)
mpra_df_tissue[:5]_____no_output_____

# 1B. sort bedGraphs

`sort -k1,1 -k2,2n unsorted.bedGraph > sorted.bedGraph`_____no_output_____

for tissue in tissues:
    unsort_file = os.path.join(save_dir, 'bedgraphs',tissue+'_unsorted.bedGraph')
    sort_file = os.path.join(save_dir,'bedgraphs', tissue+'_sorted.bedGraph')
    sort_cmd = 'sort -k1,1 -k2,2n "{}" > "{}"'.format(unsort_file,sort_file)
    print(sort_cmd)
    subprocess.call(sort_cmd,shell=True)sort -k1,1 -k2,2n "../data/processed/mpra_bigwigs/bedgraphs/AST_unsorted.bedGraph" > "../data/processed/mpra_bigwigs/bedgraphs/AST_sorted.bedGraph"
sort -k1,1 -k2,2n "../data/processed/mpra_bigwigs/bedgraphs/SL_unsorted.bedGraph" > "../data/processed/mpra_bigwigs/bedgraphs/SL_sorted.bedGraph"
sort -k1,1 -k2,2n "../data/processed/mpra_bigwigs/bedgraphs/SLC_unsorted.bedGraph" > "../data/processed/mpra_bigwigs/bedgraphs/SLC_sorted.bedGraph"
sort -k1,1 -k2,2n "../data/processed/mpra_bigwigs/bedgraphs/H9D0_unsorted.bedGraph" > "../data/processed/mpra_bigwigs/bedgraphs/H9D0_sorted.bedGraph"
sort -k1,1 -k2,2n "../data/processed/mpra_bigwigs/bedgraphs/H9D2_unsorted.bedGraph" > "../data/processed/mpra_bigwigs/bedgraphs/H9D2_sorted.bedGraph"
sort -k1,1 -k2,2n "../data/processed/mpra_bigwigs/bedgraphs/H9D4_unsorted.bedGraph" > "../data/processed/mpra_bigwigs/bedgraphs/H9D4_sorted.bedGraph"
sort -k1,1 -k2,2n "../data/processed/mpra_bigwigs/bedgraphs/H9D10_unsorted.bedGraph" > "../data/processed/mpra_bigwigs/bedgraphs/H9D10_sorted.bedGraph"
sort -k1,1 -k2,2n "../data/processed/mpra_bigwigs/bedgraphs/HEK293_unsorted.bedGraph" > "../data/processed/mpra_bigwigs/bedgraphs/HEK293_sorted.bedGraph"

# 2. run `bedGraphToBigWig`
cmd: `bedGraphToBigWig in.bedGraph chrom.sizes myBigWig.bw`_____no_output_____

bedGraphToBigWig_cmd = '/Users/mguo123/Documents/genome_browser_tools/bedGraphToBigWig'
_____no_output_____for tissue in tissues:
    bedgraph_file = os.path.join(save_dir,'bedgraphs', tissue+'_sorted.bedGraph')
    bigwig_file = os.path.join(save_dir,'bigwigs', tissue+'.bw')
    cmd = '{} {} {} {}'.format(bedGraphToBigWig_cmd,bedgraph_file, size_file,bigwig_file )
    print(cmd)
    subprocess.call(cmd,shell=True)/Users/mguo123/Documents/genome_browser_tools/bedGraphToBigWig ../data/processed/mpra_bigwigs/bedgraphs/AST_sorted.bedGraph ../data/external/chrom_hg19.sizes ../data/processed/mpra_bigwigs/bigwigs/AST.bw
/Users/mguo123/Documents/genome_browser_tools/bedGraphToBigWig ../data/processed/mpra_bigwigs/bedgraphs/SL_sorted.bedGraph ../data/external/chrom_hg19.sizes ../data/processed/mpra_bigwigs/bigwigs/SL.bw
/Users/mguo123/Documents/genome_browser_tools/bedGraphToBigWig ../data/processed/mpra_bigwigs/bedgraphs/SLC_sorted.bedGraph ../data/external/chrom_hg19.sizes ../data/processed/mpra_bigwigs/bigwigs/SLC.bw
/Users/mguo123/Documents/genome_browser_tools/bedGraphToBigWig ../data/processed/mpra_bigwigs/bedgraphs/H9D0_sorted.bedGraph ../data/external/chrom_hg19.sizes ../data/processed/mpra_bigwigs/bigwigs/H9D0.bw
/Users/mguo123/Documents/genome_browser_tools/bedGraphToBigWig ../data/processed/mpra_bigwigs/bedgraphs/H9D2_sorted.bedGraph ../data/external/chrom_hg19.sizes ../data/processed/mpra_bigwigs/bigwigs/H9D2.bw
/Users/mguo123/Documents/genome_browser_tools/bedGraphToBigWig ../data/processed/mpra_bigwigs/bedgraphs/H9D4_sorted.bedGraph ../data/external/chrom_hg19.sizes ../data/processed/mpra_bigwigs/bigwigs/H9D4.bw
/Users/mguo123/Documents/genome_browser_tools/bedGraphToBigWig ../data/processed/mpra_bigwigs/bedgraphs/H9D10_sorted.bedGraph ../data/external/chrom_hg19.sizes ../data/processed/mpra_bigwigs/bigwigs/H9D10.bw
/Users/mguo123/Documents/genome_browser_tools/bedGraphToBigWig ../data/processed/mpra_bigwigs/bedgraphs/HEK293_sorted.bedGraph ../data/external/chrom_hg19.sizes ../data/processed/mpra_bigwigs/bigwigs/HEK293.bw

# 3. make hub.config.json

manually made

final folder for upload: `/Users/mguo123/Documents/pan_omics_psych/data/processed/mpra_bigwigs/bigwigs`_____no_output_____# 4. checking
_____no_output_____


# Notebook from pankaj-pundir/git
Path: openCV_tutorials/mini projedct (Shape identification).ipynb

# Shape identifiaction
## Mini project
>various basic shapes are loaded in file_____no_output_____

<code>
def im(image,name="image"):
    cv2.imshow(name,image)
    cv2.waitKey(0)
    cv2.destroyAllWindows()_____no_output_____import cv2
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
shape=cv2.imread("shapes_5.jpg")
im(shape)_____no_output_____def find_contours(shape):
    temp_shape=cv2.cvtColor(shape,cv2.COLOR_BGR2GRAY)
    can=cv2.Canny(temp_shape,30,200)
    con_img,contours,hierarchy=cv2.findContours(can,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
    return contours
    # cv2.drawContours(shape,contours,-1,(0,255,0),5)
# plt.imshow(shape)
_____no_output_____def center_of_contour(x_c):
    M=cv2.moments(x_c)
    return (int(M['m10']/M['m00']),int(M['m01']/M['m00']))
def x_cord_contour(x_c):
    if cv2.contourArea(x_c) >9:
        return center_of_contour(x_c)[0]_____no_output_____def shape_id(shape,contour,name,color=(0,255,0),magnifier=0.2):
        cv2.drawContours(shape,[c],0,color,-1)
        cv2.putText(shape,name,center_of_contour(c),cv2.FONT_HERSHEY_COMPLEX,0.2,(255,255,255),1)
_____no_output_____contours=find_contours(shape)_____no_output_____for c in contours:

    accuracy=0.01*cv2.arcLength(c,True)
    approx=cv2.approxPolyDP(c,accuracy,True)
    if len(approx)==3:
        shape_name="triangle"
        cv2.drawContours(shape,[c],0,(0,255,0),-1)
        cv2.putText(shape,shape_name,center_of_contour(c),cv2.FONT_HERSHEY_COMPLEX,0.2,(255,255,255),1)
    elif(len(approx)==4):
        x,y,w,h=cv2.boundingRect(c)
        if w==h:
            shape_id(shape,c,'square',color=(255,0,0))
        else:
            shape_id(shape,c,'rectangle',color=(2,100,50))
    elif(len(approx)==10):
        shape_id(shape,c,'star',color=(2,25,70))
    elif(len(approx)>20):
        shape_id(shape,c,'circle',color=(25,255,0))
    print(len(approx))
im(shape)
cv2.waitKey()
cv2.destroyAllWindows()10
15
3
4
4

</code>


# Notebook from pinselimo/AWS1_S22
Path: notebooks/3_1_Agent-based_Modelling.ipynb

# Agent-based Modelling (ABM)
### Angewandte Systemwissenschaften I
#### Python - Wonderland_____no_output_____# Semesterplan
---

1.  0. Anaconda und Pip
    1. Einführung in Python
    2. Matplotlib & Jupyter-Notebooks: Eine Einführung ins Wonderland - Modell
2.  1. Python-Pakete und Workflow
    2. Funktionale Programmierung & Python-Module
3.  1. **Agenten-basierte Modellierung**
    2. Objekt-orientierte Programmierung
4.  Projektpräsentationen_____no_output_____## Inhalt

+ Equation-based Modelling
+ Agent-based Modelling
+ Emergenz
+ Aufbau von ABMs
+ Was ist ein Agent?
+ ABM in Python
+ Pattern-oriented Modelling_____no_output_____# Gleichungsbasierte Modellierung

Gleichungsbasierte Modelle basieren, wie der Name bereits verrät auf **Gleichungssystemen**. Ihre **Definition** ist oft durch **gekoppelte, gewöhnliche Differentialgleichungen** gegeben. _____no_output_____+ Wonderland_____no_output_____+ SIR - Modell_____no_output_____+ Lotka-Volterra Modell (Räuber-Beute-Modell)_____no_output_____Räumliche Details können nur mittels partieller Differentialgleichungen mit einbezogen werden._____no_output_____Der Fokus dieser gleichungsbasierten Modelle liegt auf der **Beschreibung makroskopischer Phänomene**._____no_output_____## Darstellung

Zu einer Form der Darstellung von gleichungsbasierten Modellen wird gerne **System Dynamics** herangezogen._____no_output_____Oft dient SD auch dazu das gleichungsbasierte Modell zu **erstellen**. Spezielle Software wie **Vensim** übernimmt dann das **Lösen der Differentialgleichungen**._____no_output_____+ Stocks $\rightarrow$ Zustandsvariablen_____no_output_____+ Flows + Feedback Loops $\rightarrow$  Differentialgleichungen_____no_output_____## Zum Beispiel: Räuber-Beute-System

$\frac{dH}{dt}=H*(a-b*L)$

$\frac{dL}{dt}=-L*(c-d*H)$

![Predator-Prey-SD](http://systems-sciences.uni-graz.at/etextbook/assets/img/img_sw2/vensim_predprey.JPG)
(Source: *Manfred Füllsack*)_____no_output_____# Agenten-basierte Modellierung

Bei der Agenten-basierten Modellierung steht das **Verhalten von Agenten** im Mittelpunkt._____no_output_____+ Epidemische Modelle (ähnlich SIR)_____no_output_____+ Räuber-Beute Modelle (ähnlich Lotka-Volterra)_____no_output_____Der Fokus der Modellierung liegt auf **mikroskopischen Details**._____no_output_____Modelle können **räumlich explizit** aufgebaut sein._____no_output_____## Software

Zur Entwicklung von Agenten-basierten Modellen werden **oft entsprechende Programmierkenntnisse vorausgesetzt**.

Es gibt verschiedene **Plattformen** welche dies unterstützen und erleichtern sollen:_____no_output_____+ [NetLogo](https://ccl.northwestern.edu/netlogo/)_____no_output_____+ [GAMA](https://gama-platform.github.io/)_____no_output_____+ [Repast](https://repast.github.io/)_____no_output_____+ [AnyLogic](https://www.anylogic.com/)_____no_output_____Außerdem können auch **general-purpose Sprachen** wie zum Beispiel Python herangezogen werden. Dabei kann man entweder auf vorhandenen **Softwarebibliotheken** aufbauen (zB MESA), oder ein eigenes Framework erstellen._____no_output_____# Emergenz

Effekte welche durch das **Zusammenwirken** vieler **kleiner Systemteile** entstehen (welche selbst diese Effekte nicht zeigen) werden als **emergente Phänomene** bezeichnet._____no_output_____Die **Vielzahl** an **Systemteilen** wird in ABMs durch die **Agenten** verkörpert._____no_output_____Die **Beobachtungen** konzentrieren sich dabei meistens wieder auf die **makroskopische Ebene**,

wo **emergente Phänomene** oft das Zeil der Modellierung darstellen._____no_output_____Beispiele von Emergenz:_____no_output_____+ Nagel-Schreckenberg Modell: Bildung von Phantomstaus_____no_output_____+ Schelling Modell: Segregation_____no_output_____+ Räuber-Beute Modell: Effekte wie im ODE Modell_____no_output_____# Aufbau von ABMs

Dadurch dass es eine **Vielzahl von Anwendungen** Agenten-basierter Modelle gibt, gibt es nicht **die eine** Anleitung zum Erstellen von ABMs._____no_output_____Die meisten dieser ABMs haben aber einen **grundsätzlich ähnlichen** Aufbau der sich wie folgt beschreiben lässt:_____no_output_____+ Eine **Umwelt** beherbergt die Agenten. Diese Umwelt umfasst Eigenschaften auf welche die Agenten eventuell einwirken können, welche aber auf jeden Fall für alle Agenten gleich "aussehen".

  + zB *''Der Markt''* in der Ökonomie, oder *''das Straßennetz''* in einem Verkehrsmodell_____no_output_____+ Die **Agenten** verhalten sich entsprechend eines **Regelwerkes**. Dieses kann für **unterschiedliche Kategorien** von Agenten anders geartet sein. Typisch ist außerdem auch eine gewisse **Zufälligkeit** im Verhalten der Agenten._____no_output_____# Was ist ein Agent?_____no_output_____> What makes James Bond an Agent?
>
> \- *Grimm et al., 2005*_____no_output_____+ Hat ein klares Ziel_____no_output_____+ Trifft selbstständige Entscheidungen und erreicht eigenständig sein Ziel_____no_output_____+ Passt seine Entscheidung den rasch wechselnden Bedingungen an_____no_output_____In einem Punkt finde ich unterscheidet sich James Bond aber von Agenten in ABMs:

+ Agenten interagieren untereinander (und zwar nicht nur mit Waffen)_____no_output_____Außerdem macht es aus Sicht des Programmierens/Modellierens Sinn, noch einige **weitere Kriterien** zu definieren:_____no_output_____+ Jeder Agent muss eine eindeutige Identität besitzen._____no_output_____+ Ein Agent ist eine einmalige Entität._____no_output_____+ Agenten verfügen über einen Zustand._____no_output_____# ABM in Python

Diese Eigenschaften lassen sich in Python intuitiv durch **Object-Oriented Programming** (OOP) einbinden._____no_output_____Python-Objekte haben:

+ Einen Typen
+ Eine Identität
+ Einen Zustand
+ Eine Abstammung_____no_output_____Alle Python-Objekte stammen von ```object``` ab, es ist sozusagen die "Urgroßmutter" aller Dinge in Python:_____no_output_____

o = object()
o, type(o), id(o), dir(o)[:5] # 5 damit es nur eine Zeile braucht_____no_output_____

Mit ```isinstance``` können wir die Abstammung eines Objekts klären:_____no_output_____

x = 5
isinstance(x, object), isinstance(x, int), isinstance(x, float)_____no_output_____

Wollen wir neue Formen von Objekten einführen, benutzen wir das ```class``` Keyword:_____no_output_____

class Agent(object):
    def __init__(self, argument1, attribut2,):
        self.attribut1 = argument1
        self.attribut2 = attribut2

    def methode1(self, argument):
        return self.attribut1 + argument # Beispiel_____no_output_____

Die ```__init__``` Methode ist eine sogenannte **special Method**. Sie wird ausgeführt wenn ein neues Objekt **initialisiert** wird.

Ihre Argumente werden zur Initialisierung der **Objekte dieser Klasse** genutzt:_____no_output_____

james_bond = Agent("007", True)
james_blonde = Agent("007.1", False)_____no_output_____

## Attribute

Variablen die im Namespace ```self``` einer Klasse gespeichert sind, nennt man **Attribute**._____no_output_____

james_bond.attribut1, james_bond.attribut2, _____no_output_____

In den Attributen eines Objekts wird als der **mutable Zustand** des Objekts gespeichert._____no_output_____## Methoden

Wir haben ```def``` jetzt schon in zwei Rollen gesehen: Zur Erstellung von **Funktionen** und **Generators** (letztere mit ```yield```)

Nun folgt die **dritte Anwendung: Methoden**_____no_output_____Im **Unterschied** zu Funktionen haben Methoden als **erstes Argument** immer ```self```._____no_output_____Methoden können durch ```self``` am **Zustand** des Objekts operieren. Funktionen bearbeiten im **Idealfall** keinen Zustand (siehe Functional Programming)._____no_output_____```self``` beschreibt in der **Klasse** dabei eine **abstrakte Instanz** des Objekts. Die **Klasse** selbst ist keine Instanz seiner selbst sondern der **Bauplan für die Instanzen** und gleichzeitig **ihr Typ**._____no_output_____

james_bond, id(james_bond), type(james_bond), dir(james_bond)[-3:] # 3 zeigt die von uns erstellten Namen_____no_output_____james_bond.methode1(" - Bond, James Bond")_____no_output_____

In der nächsten Einheit werden wir uns genauer mit OOP auseinandersetzen._____no_output_____# Pattern-oriented Modelling
> *Grimm et al., 2005*_____no_output_____Makroskopische Modelle können **Details** oft nicht gut genug darstellen.

Mikroskopische Modelle sollen dies verbessern, aber dürfen **nicht zu komplex** werden._____no_output_____Wie wählt man also **die richtigen Details** für das Verhalten der Agenten aus?_____no_output_____Mit **Pattern-oriented Modelling** schlägt Grimm (2005) eine Lösung für dieses Dilemma vor._____no_output_____## Orienteren an Mustern

Wir wollen **Muster** die wir beobachten können in unserem Modell **darstellen**._____no_output_____+ Fisch oder Vogelschwärme und ihr Flugmuster_____no_output_____+ Staubildung in Städten oder Überschwemmungen in Kanälen_____no_output_____+ Muster die wir bereits von einem gleichungsbasierten Modell kennen *(zB Lotka-Volterra)*_____no_output_____+ Die Ausbreitung einer Laola-Welle_____no_output_____Diese Muster können auf **verschiedenen Ebenen** im Modell angetroffen werden:

+ Individuell
+ Global
+ Lokal
+ Gruppen
+ ..._____no_output_____Lässt sich mit den **Verhaltensregeln** unserer Agenten das Muster nicht erreichen, müssen andere Schlüsselfaktoren dafür verantwortlich sein.

Unterschiedliche Beschreibungen können so auch **verglichen** werden._____no_output_____# Literatur

Grimm, V., Revilla, E., Berger, U., Jeltsch, F., Mooij, W.M., Railsback, S.F., Thulke, H.-H., Weiner, J.,
Wiegand, T., DeAngelis, D.L., 2005. Pattern-Oriented Modeling of Agent-Based Complex Systems:
Lessons from Ecology. Science 310, 987–991. https://doi.org/10.1126/science.1116681_____no_output_____


# Notebook from research-licit/TRB2019-Energy-Impact-Platoon
Path: notebook/04-Operational-control.ipynb

### Control law

This notebook emulates a control laws where a set of 8 trucks are commanded to split at specific points in time. References to compute the platoon are in this case simulated not by a tactical strategy but virtually instead:

0. Setup model parameters (CAV)
1. Initial conditions for the model
2. Reference conditions for the truck
3. Launching the simulations for _____no_output_____

# %%writefile control.py
import numpy as np
import pandas as pd

# Plot
import matplotlib.pyplot as plt

import plotly.plotly as py
from plotly.offline import init_notebook_mode, plot, iplot, iplot_mpl
import plotly.graph_objs as go
init_notebook_mode(connected=True)

from ipywidgets import interactive, widgets
from IPython.display import display

iplot([{"x": [1, 2, 3], "y": [3, 1, 6]}])_____no_output_____

#### Parameters

Simulation parameters_____no_output_____

# %%writefile -a control.py
DT = 0.1 # Sample time _____no_output_____

Traffic parameters_____no_output_____

# %%writefile -a control.py

# Truck parameters
L_AVG = 18
G_X = 2
S_X = G_X + L_AVG
K_X = 1/S_X

# Physics
G = 9.81
MU = 0.02
M = 44.000
RHO = 1.2
A = 10
CD = 0.7

# Traffic
V_F = 25.0 # Max speed.
V_P = 20.0 # Platoon free flow
E  = 25.0*0.3 # Speed drop for relaxation

U_MAX = 1.5 # Max. Acceleration
U_MIN = -1.5 # Min. Acceleration_____no_output_____

Fundamental diagram settings, interactively done to illustrate behavior of influence of front-rear spacing `G_X` and capacity `C`_____no_output_____

# %%writefile -a control.py

def compute_parameters(G_X, C):
    """ Compute dynamically parameters based on G_X,C"""
    S_X = L_AVG + G_X
    K_X = 1 / S_X
    K_C = C / V_P
    W  = C / (K_X - K_C)
    return (S_X, K_X, K_C, W)

def compute_FD(G_X, C):
    """Compute data for some predefined parameters """

    def compute_FD(k):
        """Compute k,q vector: Piecewise FD"""
        free_branch = lambda x: V_P * x
        cong_branch = lambda x: - (x-K_X) * W
        flow = np.piecewise(k, [k < K_C, k >= K_C], [free_branch, cong_branch])

        return flow

    S_X, K_X, K_C, W  = compute_parameters(G_X, C)

    k = np.linspace(0, K_X, 100)
    q = compute_FD(k)
    return (k,q)

def create_layout(K_X, C):
    """ Create figure layout """
    layout = go.Layout(title = 'FUNDAMENTAL DIAGRAM',
                       yaxis = dict(title = 'Q',
                                    range = [0, C],
                                   ),
                       xaxis = dict(title = 'k',
                                    range = [0, K_X],
                                   ),
                       width = 600,
                       height = 600,
                    )
    return layout

layout = create_layout(0.05, 1 * 3600)

def update_plot(fr_space, capacity):
    """ Update plot within the figure"""

    k, q = compute_FD(fr_space, capacity/3600)

    data = []

    trace = go.Scatter(
        x = k,
        y = q * 3600,
        mode = 'lines',
        name = 'Fundamental diagram',
        line = dict(
            shape = 'spline'
        )
    )

    data.append(trace)

    fig = go.Figure(data = data, layout = layout)
    iplot(fig)

G_X_wg = widgets.FloatSlider(
        value=5,
        min=2,
        max=L_AVG-0.1,
        step=0.1,
        description='Front-rear Space gap',
        disabled=False,
        continuous_update=True,
        orientation='horizontal',
        readout=True,
        readout_format='.1f',
        )

C_wg  = widgets.FloatSlider(
        value=2400,
        min=1800,
        max=3600-10,
        step=5,
        description='Capacity',
        disabled=False,
        continuous_update=True,
        orientation='horizontal',
        readout=True,
        readout_format='.1f',
        )

interactive(update_plot, fr_space = G_X_wg, capacity = C_wg)_____no_output_____# %%writefile -a control.py

G_X, C = G_X_wg.value, C_wg.value/3600
S_X, K_X, K_C, W  = compute_parameters(G_X, C)

TAU = 1/(K_X*W) # Time headway CAV
RTE = TAU * (V_P+W) / V_P

# At capacity
K = C/V_P
S_D = 1/K-L_AVG
G_T = S_D/V_P

Tau_wg = widgets.BoundedFloatText(
    value=TAU,
    min=0,
    max=10.0,
    step=0.1,
    description='Tau:',
    disabled=True
)
Rat_wg = widgets.BoundedFloatText(
    value=RTE,
    min=0,
    max=10.0,
    step=0.1,
    description='Rate:',
    disabled=True
)
gt_wg = widgets.BoundedFloatText(
    value=G_T,
    min=0,
    max=10.0,
    step=0.1,
    description='Inter (t):',
    disabled=True
)

gx_wg = widgets.BoundedFloatText(
    value=S_D,
    min=0,
    max=30.0,
    step=0.1,
    description='Inter (x):',
    disabled=True
)

display(Tau_wg)
display(Rat_wg)
display(gt_wg)
display(gx_wg)_____no_output_____# %%writefile -a control.py
fSimTime = 10 * 60 # seconds
nSamples = int(fSimTime*1/DT)
aTime = np.arange(nSamples)*DT

nHorizon = 50 # samples horizon_____no_output_____

#### Initial conditions for the trucks _____no_output_____

# %%writefile -a control.py

nTrucks = 4
aDims = (nSamples, nTrucks)_____no_output_____

State initialization, space, speed, delta speed leader _____no_output_____

# %%writefile -a control.py

def set_initial_condition(mS0, mV0, mDV0):
    """ Setup initial conditions of experiment"""
    mS = np.zeros(aDims) # Spacing all trucks
    mV = np.zeros(aDims) # Speed
    mDV = np.zeros(aDims) # Speed diference
    mS[0,:] = mS0
    mV[0,:] = mV0
    mDV[0,:]= mDV0
    return (mS, mV, mDV)_____no_output_____# %%writefile -a control.py
mS0 = np.ones(nTrucks) * (S_D + L_AVG)
mV0 = np.ones(nTrucks) * V_P
mDV0 = np.zeros(nTrucks)

mS, mV, mDV = set_initial_condition(mS0, mV0, mDV0)_____no_output_____

#### Reference conditions for trucks
_____no_output_____

# %%writefile -a control.py

iYieldTruck = range(1, nTrucks)

# Splits are predefined at some specific points in time (merging times)
iIdxTimeSplit = [int(t*60*1/DT) for t in (3,)]
fTimeSplit = [aTime[i] for i in iIdxTimeSplit ]
fValueTimeHwyInitial = [G_T, G_T, G_T]
fValueTimeHwyEnd = [2*G_T, 3*G_T, 4*G_T]
fValueTimeHwy = [(st,ed) for st, ed in zip(fValueTimeHwyInitial,fValueTimeHwyEnd)]

mEvents = [{'id':i, 'tm': tm, 'tg': tg}  for tm in fTimeSplit for i in iYieldTruck for tg in fValueTimeHwy]
mEvents_____no_output_____

To do:

* Create a scenario (2 splits and reintegrations)  - Func
* Run a single simulation Func( initial_cond, leaders_acc, theta) _____no_output_____

# %%writefile -a control.py

def create_ref(dEvent, Teq):
    """Creates a reference matrix for the control"""

    def anticipation_time(T_0, T_F):
        """Computes the anticipation time according to TRB 2018"""
        T_a = E / 2 * (U_MIN-U_MAX)/ (U_MIN * U_MAX) + (V_P + W) / E * (T_F - T_0)
        return T_a

    def get_sigmoid(v0, vf, yld, ant):
        """ Computes a sigmoid function with rise time equivalent to anticipation time"""
        aNewTime = 8 * (aTime - (yld + ant/2)) / ant
        return v0 + (vf-v0) * 1 / (1 + np.exp(- aNewTime))

    mRef = np.ones(aDims) * Teq
    iIdTruck = dEvent['id']
    fMrgTime = dEvent['tm']
    T_0, T_X = dEvent['tg']
    _T_0, _T_X = (T_X, T_0) if T_0 > T_X else (T_0, T_X)
    fAntTime = anticipation_time(_T_0, _T_X)
    fYldTime = fMrgTime - fAntTime

    mRef[:,iIdTruck] = get_sigmoid(T_0, T_X, fYldTime, fAntTime)

    print(f'Anticipation time: {fAntTime}')
    print(f'Yielding time: {fYldTime}')

    return mRef_____no_output_____# %%writefile -a control.py

mRef = create_ref(mEvents[-1], G_T)Anticipation time: 25.57142857142857
Yielding time: 154.42857142857144

### Control design

Control parameters_____no_output_____

# %%writefile -a control.py

C1 = 0.1
C2 = 1
C3 = 0.5_____no_output_____

Internal functions for the control _____no_output_____

# %%writefile -a control.py

def initialize_mpc(mS0, mV0, mDV0):
    """ Initialize internal variables control"""
    aDimMPC = (nHorizon,nTrucks)
    m_S, m_V, m_DV, m_LS, m_LV, m_U = (np.zeros(aDimMPC) for i in range(6))
    m_S[0] = mS0
    m_V[0] = mV0
    m_DV[0] = mDV0
    return m_S, m_V, m_DV, m_LS, m_LV, m_U

def update_cd(s):
    """ Updates the drag coefficient"""
    s[0] = L_AVG # Accounts for leader not saving
    fCD = (1-np.exp(-2 * s/ L_AVG))/2 + 0.42
    return fCD

def update_grad_cd(s):
    """ Updates the drag coefficient"""
    s[0] = L_AVG # Accounts for leader not saving
    fCD = np.exp(-2 * s/ L_AVG)/ (2 * L_AVG)
    return fCD

def forward_evolution(X, U, D):
    """ Compute forward model evolution
        X : S, V, DV
        U : control
        D : slope
    """

    S, V, DV = X

    cordim = lambda x: x.shape if len(x.shape)>1 else (1,x.shape[0])

    U = U.reshape(cordim(U))

    DU = np.concatenate((np.zeros((U.shape[0],1)),
                         U[:,0:-1] + U[:,1:]),
                        axis = 1)

    run = zip(U, DU, D)

    K1 = G * MU
    K2 = G
    K3 = RHO * A * CD / (2 * M)

    for i, u in enumerate(run):
        u_s, du, theta  = u
        if i < len(S)-1:
            print(du)
            print(DV)
            DV[i+1] = DV[i] + DT * du
            S[i+1] = S[i] + DT * DV[i]
            mfac = u_s - K1 - K2 * theta\
                   - K3 * update_cd(S[i]) * V[i]**2
            V[i+1] = V[i] + DT * mfac
    return S, V, DV

def backward_evolution(X, Ref):
    """ Compute  bakckward costate evolution
        L : LS, LV
        X : S, V, DV
    """

    def reversedEnumerate(*args):
        """ Inverse enumeration iterator"""
        revArg = [np.flip(x, axis=0) for x in args]
        return zip(range(len(args[0])-1, -1, -1), *revArg)

    S, V, DV = X

    aDimMPC = (nHorizon,nTrucks)
    ls = np.zeros(aDimMPC)
    lv = np.zeros(aDimMPC)

    runinv = reversedEnumerate(S, V, DV, Ref)

    K3 = RHO * A * CD / (2 * M)

    for i, s, v, dv, tg in runinv:
        if i > 0:
            sref = v * tg + L_AVG
            lv[i-1] = lv[i] + DT * (-2 * C1 * (s-sref) * tg \
                                    - C2 * dv - ls[i]\
                                    - lv[i] * K3 * update_cd(s)
                                   )
            ls[i-1] = ls[i] + DT * (2 * C1 * (s-sref)\
                                    - lv[i] * K3 * update_grad_cd(s) * (v ** 2)
                                   )

    return ls, lv_____no_output_____

Control routine_____no_output_____

# %%writefile -a control.py

def compute_control(mX0, mRef, mTheta):
    """ Computes a control based on mX0 and the reference mRef"""

    _m_S, _m_V, _m_DV, _m_LS, _m_LV, m_U = initialize_mpc(*mX0)
    _X = (_m_S, _m_V, _m_DV)

    # Parameters

    ALPHA = 0.01
    EPS = 0.1

    # Convergence
    error = 100
    bSuccess = 2
    N = 10000  # number of iterations

    step = iter(range(N))
    n = 0
    n_prev = 0

    while (error > EPS) and (bSuccess > 0):
        try:
            next(step)

            U_star = -_m_LV /(2 * C3)

            U_star = np.clip(U_star, U_MIN, U_MAX)

            _m_S, _m_V, _m_DV = forward_evolution(_X, U_star, mTheta)

            _lS, _lV = backward_evolution(_X, mRef)

            _m_LS = (1 - ALPHA) * _m_LS + ALPHA * _lS
            _m_LV = (1 - ALPHA) * _m_LV + ALPHA * _lV

            error = np.linalg.norm(_m_LS - _lS) + \
                    np.linalg.norm(_m_LV - _lV)

            # print(f'Error:{error}')
            # Routine for changing convergence parameter

            if error > 10e5:
                raise AssertionError('Algorithm does not converge ')
            if n >= 500:
                ALPHA = max(ALPHA - 0.01, 0.01)
                print(f'Reaching {n} iterations: Reducing alpha: {ALPHA}')
                print(f'Error before update {error}')
                if n > 10000:
                    raise AssertionError(
                        'Maximum iterations reached by the algorithm')
                n_prev = n + n_prev
                n = 0
            if error <= EPS:
                bSuccess = 0

            n += 1

        except StopIteration:
            print('Stop by iteration')
            print('Last simulation step at iteration: {}'.format(n+n_prev))
            bSuccess = 0

    n = n + n_prev
    print(f'Total it:{n}')

    return U_star[0]_____no_output_____

## Control test

Verify the control test by launching the file `platoon-closed.py`_____no_output_____


# Notebook from KayvanShah1/training-data-analyst
Path: courses/machine_learning/deepdive2/production_ml/solutions/tfdv_advanced_taxi.ipynb

# Advanced Visualizations with TensorFlow Data Validaiton_____no_output_____## Learning Objectives

1. Install TFDV
2. Compute and visualize statistics
3. Infer a schema
4. Check evaluation data for errors
5. Check for evaluation anomalies and fix it
6. Check for drift and skew
7. Freeze the schema

## Introduction

This notebook illustrates how TensorFlow Data Validation (TFDV) can be used to investigate and visualize your dataset.  That includes looking at descriptive statistics, inferring a schema, checking for and fixing anomalies, and checking for drift and skew in our dataset.  It's important to understand your dataset's characteristics, including how it might change over time in your production pipeline.  It's also important to look for anomalies in your data, and to compare your training, evaluation, and serving datasets to make sure that they're consistent.

We'll use data from the [Taxi Trips dataset](https://data.cityofchicago.org/Transportation/Taxi-Trips/wrvz-psew) released by the City of Chicago.

Note: This site provides applications using data that has been modified for use from its original source, www.cityofchicago.org, the official website of the City of Chicago. The City of Chicago makes no claims as to the content, accuracy, timeliness, or completeness of any of the data provided at this site. The data provided at this site is subject to change at any time. It is understood that the data provided at this site is being used at one’s own risk.

[Read more](https://cloud.google.com/bigquery/public-data/chicago-taxi) about the dataset in [Google BigQuery](https://cloud.google.com/bigquery/). Explore the full dataset in the [BigQuery UI](https://bigquery.cloud.google.com/dataset/bigquery-public-data:chicago_taxi_trips).

Key Point: As a modeler and developer, think about how this data is used and the potential benefits and harm a model's predictions can cause. A model like this could reinforce societal biases and disparities. Is a feature relevant to the problem you want to solve or will it introduce bias? For more information, read about [ML fairness](https://developers.google.com/machine-learning/fairness-overview/).

Each learning objective will correspond to a __#TODO__ in the [student lab notebook](https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/deepdive2/production_ml/labs/tfdv_advanced_taxi.ipynb) -- try to complete that notebook first before reviewing this solution notebook._____no_output_____The columns in the dataset are:

pickup_community_area	fare	trip_start_month

trip_start_hour	trip_start_day	trip_start_timestamp

pickup_latitude	pickup_longitude	dropoff_latitude

dropoff_longitude	trip_miles	pickup_census_tract

dropoff_census_tract	payment_type	company

trip_seconds	dropoff_community_area	tips

_____no_output_____### Install Libraries_____no_output_____**Run the below cell. Restart the kernel (Kernel > Restart kernel > Restart).**
**Re-run the below cell and proceed further.**_____no_output_____

!pip install pyarrow==2.0.0
!pip install numpy==1.19.2
!pip install tensorflow-data-validation

## Install TFDV

This will pull in all the dependencies, which will take a minute.  Please ignore the warnings or errors regarding incompatible dependency versions.

_____no_output_____

import pandas as pd
import tensorflow_data_validation as tfdv
import sys
import warnings
warnings.filterwarnings('ignore')

print('Installing TensorFlow Data Validation')
!pip install -q tensorflow_data_validation[visualization]Installing TensorFlow Data Validation

## Load the Files
We will download our dataset from Google Cloud Storage._____no_output_____

import os
import tempfile, urllib, zipfile

# Set up some globals for our file paths
BASE_DIR = tempfile.mkdtemp()
DATA_DIR = os.path.join(BASE_DIR, 'data')
OUTPUT_DIR = os.path.join(BASE_DIR, 'chicago_taxi_output')
TRAIN_DATA = os.path.join(DATA_DIR, 'train', 'data.csv')
EVAL_DATA = os.path.join(DATA_DIR, 'eval', 'data.csv')
SERVING_DATA = os.path.join(DATA_DIR, 'serving', 'data.csv')

# Download the zip file from GCP and unzip it
zip, headers = urllib.request.urlretrieve('https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/chicago_data.zip')
zipfile.ZipFile(zip).extractall(BASE_DIR)
zipfile.ZipFile(zip).close()

print("Here's what we downloaded:")
!ls -R {os.path.join(BASE_DIR, 'data')}Here's what we downloaded:
/tmp/tmpgtsgp42w/data:
eval  serving  train

/tmp/tmpgtsgp42w/data/eval:
data.csv

/tmp/tmpgtsgp42w/data/serving:
data.csv

/tmp/tmpgtsgp42w/data/train:
data.csv

### Check the version_____no_output_____

import tensorflow_data_validation as tfdv
print('TFDV version: {}'.format(tfdv.version.__version__))TFDV version: 0.22.0

## Compute and visualize statistics

First we'll use [`tfdv.generate_statistics_from_csv`](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/generate_statistics_from_csv) to compute statistics for our training data. (ignore the snappy warnings)

TFDV can compute descriptive [statistics](https://github.com/tensorflow/metadata/blob/v0.6.0/tensorflow_metadata/proto/v0/statistics.proto) that provide a quick overview of the data in terms of the features that are present and the shapes of their value distributions.

Internally, TFDV uses [Apache Beam](https://beam.apache.org/)'s data-parallel processing framework to scale the computation of statistics over large datasets. For applications that wish to integrate deeper with TFDV (e.g., attach statistics generation at the end of a data-generation pipeline), the API also exposes a Beam PTransform for statistics generation._____no_output_____

# TODO
train_stats = tfdv.generate_statistics_from_csv(data_location=TRAIN_DATA)_____no_output_____

Now let's use [`tfdv.visualize_statistics`](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/visualize_statistics), which uses [Facets](https://pair-code.github.io/facets/) to create a succinct visualization of our training data:

* Notice that numeric features and categorical features are visualized separately, and that charts are displayed showing the distributions for each feature.
* Notice that features with missing or zero values display a percentage in red as a visual indicator that there may be issues with examples in those features.  The percentage is the percentage of examples that have missing or zero values for that feature.
* Notice that there are no examples with values for `pickup_census_tract`.  This is an opportunity for dimensionality reduction!
* Try clicking "expand" above the charts to change the display
* Try hovering over bars in the charts to display bucket ranges and counts
* Try switching between the log and linear scales, and notice how the log scale reveals much more detail about the `payment_type` categorical feature
* Try selecting "quantiles" from the "Chart to show" menu, and hover over the markers to show the quantile percentages_____no_output_____

# TODO
tfdv.visualize_statistics(train_stats)_____no_output_____

## Infer a schema

Now let's use [`tfdv.infer_schema`](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/infer_schema) to create a schema for our data.  A schema defines constraints for the data that are relevant for ML. Example constraints include the data type of each feature, whether it's numerical or categorical, or the frequency of its presence in the data.  For categorical features the schema also defines the domain - the list of acceptable values.  Since writing a schema can be a tedious task, especially for datasets with lots of features, TFDV provides a method to generate an initial version of the schema based on the descriptive statistics.

Getting the schema right is important because the rest of our production pipeline will be relying on the schema that TFDV generates to be correct.  The schema also provides documentation for the data, and so is useful when different developers work on the same data.  Let's use [`tfdv.display_schema`](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/display_schema) to display the inferred schema so that we can review it._____no_output_____

# Infers schema from the input statistics.
# TODO
schema = tfdv.infer_schema(statistics=train_stats)
tfdv.display_schema(schema=schema)_____no_output_____

## Check evaluation data for errors

So far we've only been looking at the training data.  It's important that our evaluation data is consistent with our training data, including that it uses the same schema.  It's also important that the evaluation data includes examples of roughly the same ranges of values for our numerical features as our training data, so that our coverage of the loss surface during evaluation is roughly the same as during training.  The same is true for categorical features.  Otherwise, we may have training issues that are not identified during evaluation, because we didn't evaluate part of our loss surface.

* Notice that each feature now includes statistics for both the training and evaluation datasets.
* Notice that the charts now have both the training and evaluation datasets overlaid, making it easy to compare them.
* Notice that the charts now include a percentages view, which can be combined with log or the default linear scales.
* Notice that the mean and median for `trip_miles` are different for the training versus the evaluation datasets.  Will that cause problems?
* Wow, the max `tips` is very different for the training versus the evaluation datasets.  Will that cause problems?
* Click expand on the Numeric Features chart, and select the log scale.  Review the `trip_seconds` feature, and notice the difference in the max.  Will evaluation miss parts of the loss surface?_____no_output_____

# Compute stats for evaluation data
eval_stats = tfdv.generate_statistics_from_csv(data_location=EVAL_DATA)

# Compare evaluation data with training data
tfdv.visualize_statistics(lhs_statistics=eval_stats, rhs_statistics=train_stats,
                          lhs_name='EVAL_DATASET', rhs_name='TRAIN_DATASET')_____no_output_____

## Check for evaluation anomalies

Does our evaluation dataset match the schema from our training dataset?  This is especially important for categorical features, where we want to identify the range of acceptable values.

Key Point: What would happen if we tried to evaluate using data with categorical feature values that were not in our training dataset?  What about numeric features that are outside the ranges in our training dataset?_____no_output_____

# Check eval data for errors by validating the eval data stats using the previously inferred schema.
# TODO
anomalies = tfdv.validate_statistics(statistics=eval_stats, schema=schema)
tfdv.display_anomalies(anomalies)_____no_output_____

## Fix evaluation anomalies in the schema

Oops!  It looks like we have some new values for `company` in our evaluation data, that we didn't have in our training data.  We also have a new value for `payment_type`.  These should be considered anomalies, but what we decide to do about them depends on our domain knowledge of the data.  If an anomaly truly indicates a data error, then the underlying data should be fixed.  Otherwise, we can simply update the schema to include the values in the eval dataset.

Key Point: How would our evaluation results be affected if we did not fix these problems?

Unless we change our evaluation dataset we can't fix everything, but we can fix things in the schema that we're comfortable accepting.  That includes relaxing our view of what is and what is not an anomaly for particular features, as well as updating our schema to include missing values for categorical features.  TFDV has enabled us to discover what we need to fix.

Let's make those fixes now, and then review one more time._____no_output_____

# Relax the minimum fraction of values that must come from the domain for feature company.
company = tfdv.get_feature(schema, 'company')
company.distribution_constraints.min_domain_mass = 0.9

# Add new value to the domain of feature payment_type.
payment_type_domain = tfdv.get_domain(schema, 'payment_type')
payment_type_domain.value.append('Prcard')

# Validate eval stats after updating the schema
# TODO
updated_anomalies = tfdv.validate_statistics(eval_stats, schema)
tfdv.display_anomalies(updated_anomalies)_____no_output_____

Hey, look at that!  We verified that the training and evaluation data are now consistent!  Thanks TFDV ;)_____no_output_____## Schema Environments_____no_output_____We also split off a 'serving' dataset for this example, so we should check that too.  By default all datasets in a pipeline should use the same schema, but there are often exceptions. For example, in supervised learning we need to include labels in our dataset, but when we serve the model for inference the labels will not be included. In some cases introducing slight schema variations is necessary.

**Environments** can be used to express such requirements. In particular, features in schema can be associated with a set of environments using `default_environment`, `in_environment` and `not_in_environment`.

For example, in this dataset the `tips` feature is included as the label for training, but it's missing in the serving data. Without environment specified, it will show up as an anomaly._____no_output_____

serving_stats = tfdv.generate_statistics_from_csv(SERVING_DATA)
serving_anomalies = tfdv.validate_statistics(serving_stats, schema)

tfdv.display_anomalies(serving_anomalies)_____no_output_____

We'll deal with the `tips` feature below.  We also have an INT value in our trip seconds, where our schema expected a FLOAT. By making us aware of that difference, TFDV helps uncover inconsistencies in the way the data is generated for training and serving. It's very easy to be unaware of problems like that until model performance suffers, sometimes catastrophically. It may or may not be a significant issue, but in any case this should be cause for further investigation.

In this case, we can safely convert INT values to FLOATs, so we want to tell TFDV to use our schema to infer the type.  Let's do that now._____no_output_____

options = tfdv.StatsOptions(schema=schema, infer_type_from_schema=True)
serving_stats = tfdv.generate_statistics_from_csv(SERVING_DATA, stats_options=options)
serving_anomalies = tfdv.validate_statistics(serving_stats, schema)

tfdv.display_anomalies(serving_anomalies)_____no_output_____

Now we just have the `tips` feature (which is our label) showing up as an anomaly ('Column dropped').  Of course we don't expect to have labels in our serving data, so let's tell TFDV to ignore that._____no_output_____

# All features are by default in both TRAINING and SERVING environments.
schema.default_environment.append('TRAINING')
schema.default_environment.append('SERVING')

# Specify that 'tips' feature is not in SERVING environment.
tfdv.get_feature(schema, 'tips').not_in_environment.append('SERVING')

serving_anomalies_with_env = tfdv.validate_statistics(
    serving_stats, schema, environment='SERVING')

tfdv.display_anomalies(serving_anomalies_with_env)_____no_output_____

## Check for drift and skew_____no_output_____In addition to checking whether a dataset conforms to the expectations set in the schema, TFDV also provides functionalities to detect drift and skew.  TFDV performs this check by comparing the statistics of the different datasets based on the drift/skew comparators specified in the schema.

### Drift

Drift detection is supported for categorical features and between consecutive spans of data (i.e., between span N and span N+1), such as between different days of training data.  We express drift in terms of [L-infinity distance](https://en.wikipedia.org/wiki/Chebyshev_distance), and you can set the threshold distance so that you receive warnings when the drift is higher than is acceptable.  Setting the correct distance is typically an iterative process requiring domain knowledge and experimentation._____no_output_____### Skew

TFDV can detect three different kinds of skew in your data - schema skew, feature skew, and distribution skew.

#### Schema Skew

Schema skew occurs when the training and serving data do not conform to the same schema. Both training and serving data are expected to adhere to the same schema. Any expected deviations between the two (such as the label feature being only present in the training data but not in serving) should be specified through environments field in the schema.

#### Feature Skew

Feature skew occurs when the feature values that a model trains on are different from the feature values that it sees at serving time. For example, this can happen when:

* A data source that provides some feature values is modified between training and serving time
* There is different logic for generating features between training and serving. For example, if you apply some transformation only in one of the two code paths.

#### Distribution Skew

Distribution skew occurs when the distribution of the training dataset is significantly different from the distribution of the serving dataset. One of the key causes for distribution skew is using different code or different data sources to generate the training dataset. Another reason is a faulty sampling mechanism that chooses a non-representative subsample of the serving data to train on._____no_output_____

# Add skew comparator for 'payment_type' feature.
payment_type = tfdv.get_feature(schema, 'payment_type')
payment_type.skew_comparator.infinity_norm.threshold = 0.01

# Add drift comparator for 'company' feature.
company=tfdv.get_feature(schema, 'company')
company.drift_comparator.infinity_norm.threshold = 0.001

# TODO
skew_anomalies = tfdv.validate_statistics(train_stats, schema,
                                          previous_statistics=eval_stats,
                                          serving_statistics=serving_stats)

tfdv.display_anomalies(skew_anomalies)_____no_output_____

In this example we do see some drift, but it is well below the threshold that we've set._____no_output_____## Freeze the schema

Now that the schema has been reviewed and curated, we will store it in a file to reflect its "frozen" state._____no_output_____

from tensorflow.python.lib.io import file_io
from google.protobuf import text_format

file_io.recursive_create_dir(OUTPUT_DIR)
schema_file = os.path.join(OUTPUT_DIR, 'schema.pbtxt')
tfdv.write_schema_text(schema, schema_file)

!cat {schema_file}feature {
  name: "payment_type"
  type: BYTES
  domain: "payment_type"
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  skew_comparator {
    infinity_norm {
      threshold: 0.01
    }
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "company"
  value_count {
    min: 1
    max: 1
  }
  type: BYTES
  domain: "company"
  presence {
    min_count: 1
  }
  distribution_constraints {
    min_domain_mass: 0.9
  }
  drift_comparator {
    infinity_norm {
      threshold: 0.001
    }
  }
}
feature {
  name: "pickup_community_area"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "fare"
  type: FLOAT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "trip_start_month"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "trip_start_hour"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "trip_start_day"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "trip_start_timestamp"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "pickup_latitude"
  type: FLOAT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "pickup_longitude"
  type: FLOAT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "dropoff_latitude"
  value_count {
    min: 1
    max: 1
  }
  type: FLOAT
  presence {
    min_count: 1
  }
}
feature {
  name: "dropoff_longitude"
  value_count {
    min: 1
    max: 1
  }
  type: FLOAT
  presence {
    min_count: 1
  }
}
feature {
  name: "trip_miles"
  type: FLOAT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "pickup_census_tract"
  type: BYTES
  presence {
    min_count: 0
  }
}
feature {
  name: "dropoff_census_tract"
  value_count {
    min: 1
    max: 1
  }
  type: INT
  presence {
    min_count: 1
  }
}
feature {
  name: "trip_seconds"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "dropoff_community_area"
  value_count {
    min: 1
    max: 1
  }
  type: INT
  presence {
    min_count: 1
  }
}
feature {
  name: "tips"
  type: FLOAT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  not_in_environment: "SERVING"
  shape {
    dim {
      size: 1
    }
  }
}
string_domain {
  name: "payment_type"
  value: "Cash"
  value: "Credit Card"
  value: "Dispute"
  value: "No Charge"
  value: "Pcard"
  value: "Unknown"
  value: "Prcard"
}
string_domain {
  name: "company"
  value: "0118 - 42111 Godfrey S.Awir"
  value: "0694 - 59280 Chinesco Trans Inc"
  value: "1085 - 72312 N and W Cab Co"
  value: "2733 - 74600 Benny Jona"
  value: "2809 - 95474 C & D Cab Co Inc."
  value: "3011 - 66308 JBL Cab Inc."
  value: "3152 - 97284 Crystal Abernathy"
  value: "3201 - C&D Cab Co Inc"
  value: "3201 - CID Cab Co Inc"
  value: "3253 - 91138 Gaither Cab Co."
  value: "3385 - 23210 Eman Cab"
  value: "3623 - 72222 Arrington Enterprises"
  value: "3897 - Ilie Malec"
  value: "4053 - Adwar H. Nikola"
  value: "4197 - 41842 Royal Star"
  value: "4615 - 83503 Tyrone Henderson"
  value: "4615 - Tyrone Henderson"
  value: "4623 - Jay Kim"
  value: "5006 - 39261 Salifu Bawa"
  value: "5006 - Salifu Bawa"
  value: "5074 - 54002 Ahzmi Inc"
  value: "5074 - Ahzmi Inc"
  value: "5129 - 87128"
  value: "5129 - 98755 Mengisti Taxi"
  value: "5129 - Mengisti Taxi"
  value: "5724 - KYVI Cab Inc"
  value: "585 - Valley Cab Co"
  value: "5864 - 73614 Thomas Owusu"
  value: "5864 - Thomas Owusu"
  value: "5874 - 73628 Sergey Cab Corp."
  value: "5997 - 65283 AW Services Inc."
  value: "5997 - AW Services Inc."
  value: "6488 - 83287 Zuha Taxi"
  value: "6743 - Luhak Corp"
  value: "Blue Ribbon Taxi Association Inc."
  value: "C & D Cab Co Inc"
  value: "Chicago Elite Cab Corp."
  value: "Chicago Elite Cab Corp. (Chicago Carriag"
  value: "Chicago Medallion Leasing INC"
  value: "Chicago Medallion Management"
  value: "Choice Taxi Association"
  value: "Dispatch Taxi Affiliation"
  value: "KOAM Taxi Association"
  value: "Northwest Management LLC"
  value: "Taxi Affiliation Services"
  value: "Top Cab Affiliation"
}
default_environment: "TRAINING"
default_environment: "SERVING"

## When to use TFDV

It's easy to think of TFDV as only applying to the start of your training pipeline, as we did here, but in fact it has many uses.  Here are a few more:

* Validating new data for inference to make sure that we haven't suddenly started receiving bad features
* Validating new data for inference to make sure that our model has trained on that part of the decision surface
* Validating our data after we've transformed it and done feature engineering (probably using [TensorFlow Transform](https://www.tensorflow.org/tfx/guide/transform)) to make sure we haven't done something wrong_____no_output_____


# Notebook from zjzh/alibi
Path: doc/source/examples/cfproto_mnist.ipynb

# Counterfactuals guided by prototypes on MNIST_____no_output_____This method is described in the [Interpretable Counterfactual Explanations Guided by Prototypes](https://arxiv.org/abs/1907.02584) paper and can generate counterfactual instances guided by class prototypes. It means that for a certain instance X, the method builds a prototype for each prediction class using either an [autoencoder](https://en.wikipedia.org/wiki/Autoencoder) or [k-d trees](https://en.wikipedia.org/wiki/K-d_tree). The nearest prototype class other than the originally predicted class is then used to guide the counterfactual search. For example, in MNIST the closest class to a 7 could be a 9. As a result, the prototype loss term will try to minimize the distance between the proposed counterfactual and the prototype of a 9. This speeds up the search towards a satisfactory counterfactual by steering it towards an interpretable solution from the start of the optimization. It also helps to avoid out-of-distribution counterfactuals with the perturbations driven to a prototype of another class.

The loss function to be optimized is the following:

$$
Loss = cL_{pred} + \beta L_{1} + L_{2} + L_{AE} + L_{proto}
$$

The first loss term relates to the model's prediction function, the following 2 terms define the elastic net regularization while the last 2 terms are optional. The aim of $L_{AE}$ is to penalize out-of-distribution counterfactuals while $L_{proto}$ guides the counterfactual to a prototype. When we only have acces to the model's prediction function and cannot fully enjoy the benefits of automatic differentiation, the prototypes allow us to drop the prediction function loss term $L_{pred}$ and still generate high quality counterfactuals. This drastically reduces the number of prediction calls made during the numerical gradient update step and again speeds up the search.

Other options include generating counterfactuals for specific classes or including trust score constraints to ensure that the counterfactual is close enough to the newly predicted class compared to the original class. Different use cases are illustrated throughout this notebook._____no_output_____

import tensorflow as tf
tf.get_logger().setLevel(40) # suppress deprecation messages
tf.compat.v1.disable_v2_behavior() # disable TF2 behaviour as alibi code still relies on TF1 constructs
from tensorflow.keras.layers import Conv2D, Dense, Dropout, Flatten, MaxPooling2D, Input, UpSampling2D
from tensorflow.keras.models import Model, load_model
from tensorflow.keras.utils import to_categorical

import matplotlib
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import os
from time import time
from alibi.explainers import CounterfactualProto

print('TF version: ', tf.__version__)
print('Eager execution enabled: ', tf.executing_eagerly()) # FalseTF version:  2.2.0
Eager execution enabled:  False

## Load and prepare MNIST data_____no_output_____

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
print('x_train shape:', x_train.shape, 'y_train shape:', y_train.shape)
plt.gray()
plt.imshow(x_test[1]);x_train shape: (60000, 28, 28) y_train shape: (60000,)

Prepare data: scale, reshape and categorize_____no_output_____

x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255
x_train = np.reshape(x_train, x_train.shape + (1,))
x_test = np.reshape(x_test, x_test.shape + (1,))
print('x_train shape:', x_train.shape, 'x_test shape:', x_test.shape)
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)
print('y_train shape:', y_train.shape, 'y_test shape:', y_test.shape)x_train shape: (60000, 28, 28, 1) x_test shape: (10000, 28, 28, 1)
y_train shape: (60000, 10) y_test shape: (10000, 10)
xmin, xmax = -.5, .5
x_train = ((x_train - x_train.min()) / (x_train.max() - x_train.min())) * (xmax - xmin) + xmin
x_test = ((x_test - x_test.min()) / (x_test.max() - x_test.min())) * (xmax - xmin) + xmin_____no_output_____

## Define and train CNN model_____no_output_____

def cnn_model():
    x_in = Input(shape=(28, 28, 1))
    x = Conv2D(filters=32, kernel_size=2, padding='same', activation='relu')(x_in)
    x = MaxPooling2D(pool_size=2)(x)
    x = Dropout(0.3)(x)

    x = Conv2D(filters=64, kernel_size=2, padding='same', activation='relu')(x)
    x = MaxPooling2D(pool_size=2)(x)
    x = Dropout(0.3)(x)

    x = Flatten()(x)
    x = Dense(256, activation='relu')(x)
    x = Dropout(0.5)(x)
    x_out = Dense(10, activation='softmax')(x)

    cnn = Model(inputs=x_in, outputs=x_out)
    cnn.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return cnn_____no_output_____cnn = cnn_model()
cnn.fit(x_train, y_train, batch_size=32, epochs=3, verbose=0)
cnn.save('mnist_cnn.h5', save_format='h5')_____no_output_____

Evaluate the model on test set_____no_output_____

cnn = load_model('mnist_cnn.h5')
score = cnn.evaluate(x_test, y_test, verbose=0)
print('Test accuracy: ', score[1])Test accuracy:  0.9871

## Define and train auto-encoder_____no_output_____

def ae_model():
    # encoder
    x_in = Input(shape=(28, 28, 1))
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(x_in)
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(x)
    x = MaxPooling2D((2, 2), padding='same')(x)
    encoded = Conv2D(1, (3, 3), activation=None, padding='same')(x)
    encoder = Model(x_in, encoded)

    # decoder
    dec_in = Input(shape=(14, 14, 1))
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(dec_in)
    x = UpSampling2D((2, 2))(x)
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(x)
    decoded = Conv2D(1, (3, 3), activation=None, padding='same')(x)
    decoder = Model(dec_in, decoded)

    # autoencoder = encoder + decoder
    x_out = decoder(encoder(x_in))
    autoencoder = Model(x_in, x_out)
    autoencoder.compile(optimizer='adam', loss='mse')

    return autoencoder, encoder, decoder_____no_output_____ae, enc, dec = ae_model()
ae.fit(x_train, x_train, batch_size=128, epochs=4, validation_data=(x_test, x_test), verbose=0)
ae.save('mnist_ae.h5', save_format='h5')
enc.save('mnist_enc.h5', save_format='h5')_____no_output_____

Compare original with decoded images_____no_output_____

ae = load_model('mnist_ae.h5')
enc = load_model('mnist_enc.h5', compile=False)

decoded_imgs = ae.predict(x_test)
n = 5
plt.figure(figsize=(20, 4))
for i in range(1, n+1):
    # display original
    ax = plt.subplot(2, n, i)
    plt.imshow(x_test[i].reshape(28, 28))
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    # display reconstruction
    ax = plt.subplot(2, n, i + n)
    plt.imshow(decoded_imgs[i].reshape(28, 28))
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()_____no_output_____

## Generate counterfactual guided by the nearest class prototype_____no_output_____Original instance:_____no_output_____

X = x_test[0].reshape((1,) + x_test[0].shape)
plt.imshow(X.reshape(28, 28));_____no_output_____

Counterfactual parameters:_____no_output_____

shape = (1,) + x_train.shape[1:]
gamma = 100.
theta = 100.
c_init = 1.
c_steps = 2
max_iterations = 1000
feature_range = (x_train.min(),x_train.max())_____no_output_____

Run counterfactual:_____no_output_____

# initialize explainer, fit and generate counterfactual
cf = CounterfactualProto(cnn, shape, gamma=gamma, theta=theta,
                         ae_model=ae, enc_model=enc, max_iterations=max_iterations,
                         feature_range=feature_range, c_init=c_init, c_steps=c_steps)
start_time = time()
cf.fit(x_train)  # find class prototypes
print('Time to find prototypes each class: {:.3f} sec'.format(time() - start_time))
start_time = time()
explanation = cf.explain(X)
print('Explanation took {:.3f} sec'.format(time() - start_time))Time to find prototypes each class: 14.580 sec
Explanation took 9.269 sec

Results:_____no_output_____

print('Counterfactual prediction: {}'.format(explanation.cf['class']))
print(f'Closest prototype class: {explanation.id_proto}')
plt.imshow(explanation.cf['X'].reshape(28, 28));Counterfactual prediction: 9
Closest prototype class: 9

The counterfactual starting from a 7 moves towards its closest prototype class: a 9. The evolution of the counterfactual during the first iteration can be seen below:_____no_output_____

iter_cf = 0
print(f'iteration c {iter_cf}')
n = len(explanation['all'][iter_cf])
plt.figure(figsize=(20, 4))
for i in range(n):
    ax = plt.subplot(1, n+1, i+1)
    plt.imshow(explanation['all'][iter_cf][i].reshape(28, 28))
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()iteration c 0

Typically, the first few iterations already steer the 7 towards a 9, while the later iterations make the counterfactual more sparse._____no_output_____## Prototypes defined by the $k$ nearest encoded instances_____no_output_____In the above example, the class prototypes are defined by the average encoding of all instances belonging to the specific class. Instead, we can also select only the $k$ nearest encoded instances of a class to the encoded instance to be explained and use the average over those $k$ encodings as the prototype._____no_output_____

# initialize explainer, fit and generate counterfactuals
cf = CounterfactualProto(cnn, shape, gamma=gamma, theta=theta,
                         ae_model=ae, enc_model=enc, max_iterations=max_iterations,
                         feature_range=feature_range, c_init=c_init, c_steps=c_steps)
cf.fit(x_train)
explanation_k1 = cf.explain(X, k=1, k_type='mean')
explanation_k20 = cf.explain(X, k=20, k_type='mean')_____no_output_____

Results for $k$ equals 1:_____no_output_____

print('Counterfactual prediction: {}'.format(explanation_k1.cf['class']))
print(f'Closest prototype class: {explanation.id_proto}')
plt.imshow(explanation_k1.cf['X'].reshape(28, 28));Counterfactual prediction: 9
Closest prototype class: 9

Results for $k$ equals 20:_____no_output_____

print('Counterfactual prediction: {}'.format(explanation_k20.cf['class']))
print(f'Closest prototype class: {explanation.id_proto}')
plt.imshow(explanation_k20.cf['X'].reshape(28, 28));Counterfactual prediction: 9
Closest prototype class: 9

A lower value of $k$ typically leads to counterfactuals that look more like the original instance and less like an average instance of the counterfactual class._____no_output_____## Remove the autoencoder loss term $L_{AE}$_____no_output_____In the previous example, we used both an autoencoder loss term to penalize a counterfactual which falls outside of the training data distribution as well as an encoder loss term to guide the counterfactual to the nearest prototype class. In the next example we get rid of the autoencoder loss term to speed up the counterfactual search and still generate decent counterfactuals:_____no_output_____

# initialize explainer, fit and generate counterfactuals
cf = CounterfactualProto(cnn, shape, gamma=gamma, theta=theta,
                         enc_model=enc, max_iterations=max_iterations,
                         feature_range=feature_range, c_init=c_init, c_steps=c_steps)
cf.fit(x_train)
start_time = time()
explanation = cf.explain(X, k=1)
print('Explanation took {:.3f} sec'.format(time() - start_time))Explanation took 6.443 sec

Results:_____no_output_____

print('Counterfactual prediction: {}'.format(explanation.cf['class']))
print(f'Closest prototype class: {explanation.id_proto}')
plt.imshow(explanation.cf['X'].reshape(28, 28));Counterfactual prediction: 9
Closest prototype class: 9

## Specify prototype classes_____no_output_____For multi-class predictions, we might be interested to generate counterfactuals for certain classes while avoiding others. The following example illustrates how to do this:_____no_output_____

X = x_test[12].reshape((1,) + x_test[1].shape)
plt.imshow(X.reshape(28, 28));_____no_output_____# initialize explainer, fit and generate counterfactuals
cf = CounterfactualProto(cnn, shape, gamma=gamma, theta=theta,
                         ae_model=ae, enc_model=enc, max_iterations=max_iterations,
                         feature_range=feature_range, c_init=c_init, c_steps=c_steps)
cf.fit(x_train)
explanation_1 = cf.explain(X, k=5, k_type='mean')
proto_1 = explanation_1.id_proto
explanation_2 = cf.explain(X, k=5, k_type='mean', target_class=[7])
proto_2 = explanation_2.id_proto_____no_output_____

The closest class to the 9 is 4. This is evident by looking at the first counterfactual below. For the second counterfactual, we specified that the prototype class used in the search should be a 7. As a result, a counterfactual 7 instead of a 4 is generated._____no_output_____

print('Counterfactual prediction: {}'.format(explanation_1.cf['class']))
print(f'Closest prototype class: {proto_1}')
plt.imshow(explanation_1.cf['X'].reshape(28, 28));Counterfactual prediction: 4
Closest prototype class: 4
print('Counterfactual prediction: {}'.format(explanation_2.cf['class']))
print(f'Closest prototype class: {proto_2}')
plt.imshow(explanation_2.cf['X'].reshape(28, 28));Counterfactual prediction: 7
Closest prototype class: 7

## Speed up the counterfactual search by removing the predict function loss term_____no_output_____We can also remove the prediction loss term and still obtain an interpretable counterfactual. This is especially relevant for fully black box models. When we provide the counterfactual search method with a Keras or TensorFlow model, it is incorporated in the TensorFlow graph and evaluated using automatic differentiation. However, if we only have access to the model's prediction function, the gradient updates are numerical and typically require a large number of prediction calls because of the prediction loss term $L_{pred}$. These prediction calls can slow the search down significantly and become a bottleneck. We can represent the gradient of the loss term as follows:

$$
\frac{\partial L_{pred}}{\partial x} = \frac{\partial L_{pred}}{\partial p} \frac{\partial p}{\partial x}
$$

where $L_{pred}$ is the prediction loss term, $p$ the prediction function and $x$ the input features to optimize. For a 28 by 28 MNIST image, the $^{\delta p}/_{\delta x}$ term alone would require a prediction call with batch size 28x28x2 = 1568. By using the prototypes to guide the search however, we can remove the prediction loss term and only make a single prediction at the end of each gradient update to check whether the predicted class on the proposed counterfactual is different from the original class. We do not necessarily need a Keras or TensorFlow auto-encoder either and can use k-d trees to find the nearest class prototypes. Please check out [this notebook](./cfproto_housing.ipynb) for a practical example.

The first example below removes $L_{pred}$ from the loss function to bypass the bottleneck. It illustrates the drastic speed improvements over the black box alternative with numerical gradient evaluation while still producing interpretable counterfactual instances._____no_output_____

plt.gray()
X = x_test[23].reshape(1, 28, 28, 1)
plt.imshow(X.reshape(28, 28));_____no_output_____c_init = 0.  # weight on prediction loss term set to 0
c_steps = 1  # no need to find optimal values for c_____no_output_____# define a black-box model
predict_fn = lambda x: cnn.predict(x)

# initialize explainer, fit and generate counterfactuals
cf = CounterfactualProto(predict_fn, shape, gamma=gamma, theta=theta,
                         ae_model=ae, enc_model=enc, max_iterations=max_iterations,
                         feature_range=feature_range, c_init=c_init, c_steps=c_steps)
cf.fit(x_train)
start_time = time()
explanation = cf.explain(X, k=1)
print('Explanation took {:.3f} sec'.format(time() - start_time))Explanation took 7.257 sec
print('Counterfactual prediction: {}'.format(explanation.cf['class']))
print(f'Closest prototype class: {explanation.id_proto}')
plt.imshow(explanation.cf['X'].reshape(28, 28));Counterfactual prediction: 6
Closest prototype class: 6

Let us know add the $L_{pred}$ loss term back in the objective function and observe how long it takes to generate a black box counterfactual:_____no_output_____

c_init = 1.
c_steps = 2_____no_output_____# define a black-box model
predict_fn = lambda x: cnn.predict(x)

# initialize explainer, fit and generate counterfactuals
cf = CounterfactualProto(predict_fn, shape, gamma=gamma, theta=theta,
                         ae_model=ae, enc_model=enc, max_iterations=max_iterations,
                         feature_range=feature_range, c_init=c_init, c_steps=c_steps)
cf.fit(x_train)
start_time = time()
explanation = cf.explain(X, k=1)
print('Explanation took {:.3f} sec'.format(time() - start_time))Explanation took 966.342 sec
print('Counterfactual prediction: {}'.format(explanation.cf['class']))
print(f'Closest prototype class: {explanation.id_proto}')
plt.imshow(explanation.cf['X'].reshape(28, 28));Counterfactual prediction: 6
Closest prototype class: 6

Clean up:_____no_output_____

os.remove('mnist_cnn.h5')
os.remove('mnist_ae.h5')
os.remove('mnist_enc.h5')_____no_output_____


# Notebook from nicolossus/neuromodels
Path: notebooks/Brunel.ipynb

<code>
#!/usr/env/bin python
# -*- coding: utf-8 -*-
#
# brunel_alpha_nest.py
#
# This file is part of NEST.
#
# Copyright (C) 2004 The NEST Initiative
#
# NEST is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# NEST is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with NEST.  If not, see <http://www.gnu.org/licenses/>.
#
# Modified as an example with hybridLFPy (https:/github.com/INM-6/hybridLFPy)
# for calculating local field potentials from spiking point-neuron networks

"""
Random balanced network (alpha synapses) connected with NEST
------------------------------------------------------------
This script simulates an excitatory and an inhibitory population on
the basis of the network used in
Brunel N, Dynamics of Sparsely Connected Networks of Excitatory and
Inhibitory Spiking Neurons, Journal of Computational Neuroscience 8,
183-208 (2000).
In contrast to brunel-alpha-numpy.py, this variant uses NEST's builtin
connection routines to draw the random connections instead of NumPy.
When connecting the network customary synapse models are used, which
allow for querying the number of created synapses. Using spike
detectors the average firing rates of the neurons in the populations
are established. The building as well as the simulation time of the
network are recorded.
"""

"""
Importing all necessary modules for simulation, analysis and plotting.
"""

'''
Definition of functions used in this example. First, define the
Lambert W function implemented in SLI. The second function
computes the maximum of the postsynaptic potential for a synaptic
input current of unit amplitude (1 pA) using the Lambert W
function. Thus function will later be used to calibrate the synaptic
weights.
'''

import os
import time

import nest
import nest.raster_plot
from numpy import exp, random
from scipy.optimize import fsolve

def LambertWm1(x):
    nest.ll_api.sli_push(x)
    nest.ll_api.sli_run('LambertWm1')
    y = nest.ll_api.sli_pop()
    return y

def ComputePSPnorm(tauMem, CMem, tauSyn):
    a = (tauMem / tauSyn)
    b = (1.0 / tauSyn - 1.0 / tauMem)

    # time of maximum
    t_max = 1.0 / b * (-LambertWm1(-exp(-1.0 / a) / a) - 1.0 / a)

    # maximum of PSP for current of unit amplitude
    return exp(1.0) / (tauSyn * CMem * b) * ((exp(-t_max / tauMem) -
                                              exp(-t_max / tauSyn)) / b - t_max * exp(-t_max / tauSyn))

'''
Assigning the current time to a variable in order to determine the
build time of the network.
'''

startbuild = time.time()

'''
Assigning the simulation parameters to variables.
'''

dt = 0.1    # the resolution in ms
simtime = 1000.0  # Simulation time in ms
delay = 1.5    # synaptic delay in ms

'''
Definition of the parameters crucial for asynchronous irregular firing
of the neurons.
'''

g = 6.0  # ratio inhibitory weight/excitatory weight
eta = 2.0  # external rate relative to threshold rate
epsilon = 0.1  # connection probability

'''
Definition of the number of neurons in the network and the number of
neuron recorded from.
Note: We record here all spike events.
'''

order = 100
NE = 4 * order  # number of excitatory neurons
NI = 1 * order  # number of inhibitory neurons
N_neurons = NE + NI   # number of neurons in total

'''
flag for using Poisson or equivalent DC current external drive
'''

Poisson = False

'''
Definition of connectivity parameter
'''

CE = int(epsilon * NE)  # number of excitatory synapses per neuron
CI = int(epsilon * NI)  # number of inhibitory synapses per neuron
C_tot = int(CI + CE)      # total number of synapses per neuron

'''
Initialization of the parameters of the integrate and fire neuron and
the synapses. The parameter of the neuron are stored in a dictionary.
The synaptic currents are normalized such that the amplitude of the
PSP is J.
'''

tauSyn = 0.5        # synaptic time constant in ms
tauMem = 20.0       # time constant of membrane potential in ms
CMem = 250.0        # capacitance of membrane in in pF
theta = 20.0       # membrane threshold potential in mV
J = 1.0        # postsynaptic amplitude in mV
J_unit = ComputePSPnorm(tauMem, CMem, tauSyn)
J_ex = J / J_unit  # amplitude of excitatory postsynaptic current
J_in = -g * J_ex    # amplitude of inhibitory postsynaptic current

'''
Definition of threshold rate, which is the external rate needed to fix
the membrane potential around its threshold, the external firing rate
and the rate of the poisson generator which is multiplied by the
in-degree CE and converted to Hz by multiplication by 1000.
'''

nu_th = (theta * CMem) / (J_ex * CE * exp(1) * tauMem * tauSyn)
nu_ex = eta * nu_th
p_rate = 1000.0 * nu_ex * CE

'''
Adjust parameters accordingly if Poisson background input is used
'''

if Poisson:
    # use zero dc input amplitude to the neurons
    dc_amplitude = 0.
else:
    # compute equivalent DC current amplitude instead of Poisson input assuming
    # alpha current synapse
    factor = 0.4  # CHECK
    dc_amplitude = p_rate * J_ex / 1000 / tauSyn * factor

'''
Parameters for "iaf_psc_alpha" cell model
'''

neuron_params = {"C_m": CMem,
                 "tau_m": tauMem,
                 "tau_syn_ex": tauSyn,
                 "tau_syn_in": tauSyn,
                 "t_ref": 2.0,
                 "E_L": 0.0,
                 "V_reset": 0.0,
                 "V_th": 20.0,
                 "V_m": 0.0,
                 "V_th": theta,
                 "I_e": dc_amplitude, }
'''
Destination of file output
'''

spike_output_path = os.path.join('simulation_output_example_brunel',
                                 'spiking_output_path')

'''
File prefix for spike detectors
'''

label = 'brunel-py'

'''
Main simulation procedure function
'''

def simulate():
    '''instantiate and execute network simulation'''
    # separate model execution from parameters for safe import from other files
    nest.ResetKernel()

    '''
    Configuration of the simulation kernel by the previously defined time
    resolution used in the simulation. Setting "print_time" to True prints
    the already processed simulation time as well as its percentage of the
    total simulation time.
    '''

    nest.SetKernelStatus({"resolution": dt, "print_time": True,
                          "overwrite_files": True})

    print("Building network")

    '''
    Configuration of the model `iaf_psc_alpha` and `poisson_generator`
    using SetDefaults(). This function expects the model to be the
    inserted as a string and the parameter to be specified in a
    dictionary. All instances of theses models created after this point
    will have the properties specified in the dictionary by default.
    '''

    nest.SetDefaults("iaf_psc_alpha", neuron_params)
    nest.SetDefaults("poisson_generator", {"rate": p_rate})

    '''
    Creation of the nodes using `Create`. We store the returned handles in
    variables for later reference. Here the excitatory and inhibitory, as
    well as the poisson generator and two spike detectors. The spike
    detectors will later be used to record excitatory and inhibitory
    spikes.
    '''

    nodes_ex = nest.Create("iaf_psc_alpha", NE)
    nodes_in = nest.Create("iaf_psc_alpha", NI)
    noise = nest.Create("poisson_generator")
    espikes = nest.Create("spike_recorder")
    ispikes = nest.Create("spike_recorder")

    print("first exc node: {}".format(nodes_ex[0]))
    print("first inh node: {}".format(nodes_in[0]))

    '''
    distribute membrane potentials
    '''
    nest.SetStatus(nodes_ex, "V_m",
                   random.rand(len(nodes_ex)) * neuron_params["V_th"])
    nest.SetStatus(nodes_in, "V_m",
                   random.rand(len(nodes_in)) * neuron_params["V_th"])

    '''
    Configuration of the spike detectors recording excitatory and
    inhibitory spikes using `SetStatus`, which expects a list of node
    handles and a list of parameter dictionaries. Setting the variable
    "to_file" to True ensures that the spikes will be recorded in a .gdf
    file starting with the string assigned to label. Setting "withtime"
    and "withgid" to True ensures that each spike is saved to file by
    stating the gid of the spiking neuron and the spike time in one line.
    '''

    nest.SetStatus(espikes, [{
        "label": os.path.join(spike_output_path, label + "-EX"),
        "record_to": 'ascii',
    }])

    nest.SetStatus(ispikes, [{
        "label": os.path.join(spike_output_path, label + "-IN"),
        "record_to": 'ascii', }])

    print("Connecting devices")

    '''
    Definition of a synapse using `CopyModel`, which expects the model
    name of a pre-defined synapse, the name of the customary synapse and
    an optional parameter dictionary. The parameters defined in the
    dictionary will be the default parameter for the customary
    synapse. Here we define one synapse for the excitatory and one for the
    inhibitory connections giving the previously defined weights and equal
    delays.
    '''

    nest.CopyModel(
        "static_synapse", "excitatory", {
            "weight": J_ex, "delay": delay})
    nest.CopyModel(
        "static_synapse", "inhibitory", {
            "weight": J_in, "delay": delay})

    '''
    Connecting the previously defined poisson generator to the excitatory
    and inhibitory neurons using the excitatory synapse. Since the poisson
    generator is connected to all neurons in the population the default
    rule ('all_to_all') of Connect() is used. The synaptic properties are
    inserted via syn_spec which expects a dictionary when defining
    multiple variables or a string when simply using a pre-defined
    synapse.
    '''

    if Poisson:
        nest.Connect(noise, nodes_ex, 'all_to_all', "excitatory")
        nest.Connect(noise, nodes_in, 'all_to_all', "excitatory")

    '''
    Connecting the first N_neurons nodes of the excitatory and inhibitory
    population to the associated spike detectors using excitatory
    synapses. Here the same shortcut for the specification of the synapse
    as defined above is used.
    '''

    nest.Connect(nodes_ex, espikes, 'all_to_all', "excitatory")
    nest.Connect(nodes_in, ispikes, 'all_to_all', "excitatory")

    print("Connecting network")

    print("Excitatory connections")

    '''
    Connecting the excitatory population to all neurons using the
    pre-defined excitatory synapse. Beforehand, the connection parameter
    are defined in a dictionary. Here we use the connection rule
    'fixed_indegree', which requires the definition of the indegree. Since
    the synapse specification is reduced to assigning the pre-defined
    excitatory synapse it suffices to insert a string.
    '''

    conn_params_ex = {'rule': 'fixed_indegree', 'indegree': CE}
    nest.Connect(nodes_ex, nodes_ex + nodes_in, conn_params_ex, "excitatory")

    print("Inhibitory connections")

    '''
    Connecting the inhibitory population to all neurons using the
    pre-defined inhibitory synapse. The connection parameter as well as
    the synapse paramtere are defined analogously to the connection from
    the excitatory population defined above.
    '''

    conn_params_in = {'rule': 'fixed_indegree', 'indegree': CI}
    nest.Connect(nodes_in, nodes_ex + nodes_in, conn_params_in, "inhibitory")

    '''
    Storage of the time point after the buildup of the network in a
    variable.
    '''

    endbuild = time.time()

    '''
    Simulation of the network.
    '''

    print("Simulating")

    nest.Simulate(simtime)

    '''
    Storage of the time point after the simulation of the network in a
    variable.
    '''

    endsimulate = time.time()

    '''
    Reading out the total number of spikes received from the spike
    detector connected to the excitatory population and the inhibitory
    population.
    '''

    events_ex = nest.GetStatus(espikes, "n_events")[0]
    events_in = nest.GetStatus(ispikes, "n_events")[0]

    '''
    Calculation of the average firing rate of the excitatory and the
    inhibitory neurons by dividing the total number of recorded spikes by
    the number of neurons recorded from and the simulation time. The
    multiplication by 1000.0 converts the unit 1/ms to 1/s=Hz.
    '''

    rate_ex = events_ex / simtime * 1000.0 / N_neurons
    rate_in = events_in / simtime * 1000.0 / N_neurons

    '''
    Reading out the number of connections established using the excitatory
    and inhibitory synapse model. The numbers are summed up resulting in
    the total number of synapses.
    '''

    num_synapses = nest.GetDefaults("excitatory")["num_connections"] +\
        nest.GetDefaults("inhibitory")["num_connections"]

    '''
    Establishing the time it took to build and simulate the network by
    taking the difference of the pre-defined time variables.
    '''

    build_time = endbuild - startbuild
    sim_time = endsimulate - endbuild

    '''
    Printing the network properties, firing rates and building times.
    '''

    print("Brunel network simulation (Python)")
    print("Number of neurons : {0}".format(N_neurons))
    print("Number of synapses: {0}".format(num_synapses))
    print("       Exitatory  : {0}".format(int(CE * N_neurons) + N_neurons))
    print("       Inhibitory : {0}".format(int(CI * N_neurons)))
    print("Excitatory rate   : %.2f Hz" % rate_ex)
    print("Inhibitory rate   : %.2f Hz" % rate_in)
    print("Building time     : %.2f s" % build_time)
    print("Simulation time   : %.2f s" % sim_time)

    '''
    Plot a raster of the excitatory neurons and a histogram.
    '''

    if False:
        nest.raster_plot.from_device(espikes, hist=True)
        nest.raster_plot.from_device(ispikes, hist=True)
        nest.raster_plot.show()
              -- N E S T --
  Copyright (C) 2004 The NEST Initiative

 Version: master@c4a0daab5
 Built: Aug 13 2021 01:17:10

 This program is provided AS IS and comes with
 NO WARRANTY. See the file LICENSE for details.

 Problems or suggestions?
   Visit https://www.nest-simulator.org

 Type 'nest.help()' to find out more about NEST.

simulate()Building network
Aug 13 01:41:45 SimulationManager::set_status [Info]:
    Temporal resolution changed.

first exc node: NodeCollection(metadata=None, model=iaf_psc_alpha, size=1, first=1)
first inh node: NodeCollection(metadata=None, model=iaf_psc_alpha, size=1, first=401)
Connecting devices
Connecting network
Excitatory connections
Inhibitory connections
Simulating

Aug 13 01:41:45 NodeManager::prepare_nodes [Info]:
    Preparing 503 nodes for simulation.

Aug 13 01:41:45 RecordingBackendASCII::prepare() [Error]:
    I/O error while opening file
    'simulation_output_example_brunel/spiking_output_path/brunel-py-EX-502-0.dat'.

</code>