# Notebook from KarrLab/schema
Path: examples/tutorials/1. Building and visualizing schemas.ipynb

# Introduction_____no_output_____This tutorial illustrates how to use *ObjTables* to define a schema for modeling data, as well as how to visualize schemas. The tutorial uses an address book as an example._____no_output_____# Ceate subclasses of `obj_tables.Model` to represent companies, their employees, and their addresses_____no_output_____First, use `obj_tables.Model` and subclasses of `obj_tables.Attribute` to create classes to represent companies, their employees, and their addresses._____no_output_____##### Import `obj_tables`_____no_output_____

import obj_tables_____no_output_____

##### Create a class `Address` to represent the address of companies and their employees_____no_output_____First, define attributes to represent the street, city, state, zip code, and country of an address.

Next, use `obj_tables.Model.Meta` to control how adresses are encoded into and decoded from tables.
* Use `table_format = obj_tables.TableFormat.multiple_cells` to indicate that addresses should be embedded into and extracted out of tables for companies and people.
* Use `attribute_order` to control the order in which the attributes are encoded into colums of tables.
* Use `verbose_name` and `verbose_name_plural` to control how address are labeled when embedded into other tables (heading above the address columns)._____no_output_____

class Address(obj_tables.Model):
    street = obj_tables.StringAttribute(unique=True, primary=True, verbose_name='Street')
    city = obj_tables.StringAttribute(verbose_name='City')
    state = obj_tables.StringAttribute(verbose_name='State')
    zip_code = obj_tables.StringAttribute(verbose_name='Zip code')
    country = obj_tables.StringAttribute(verbose_name='Country')

    class Meta(obj_tables.Model.Meta):
        table_format = obj_tables.TableFormat.multiple_cells
        attribute_order = ('street', 'city', 'state', 'zip_code', 'country',)
        verbose_name = 'Address'
        verbose_name_plural = 'Addresses'_____no_output_____

##### Create a class `Company` to represent companies_____no_output_____Next, create a class to represent companies.
* Use `unique=True` to indicate each company must have a unique name.
* Use `primary=True` to indicate that company names can be used as a foreign key to encode relationships to companies into cells of tables.
* Use `obj_tables.OneToOneAttribute` to indicate that each company can have an address._____no_output_____

class Company(obj_tables.Model):
    name = obj_tables.StringAttribute(unique=True, primary=True, verbose_name='Name')
    url = obj_tables.UrlAttribute(verbose_name='URL')
    address = obj_tables.OneToOneAttribute(Address, related_name='company', verbose_name='Address')

    class Meta(obj_tables.Model.Meta):
        table_format = obj_tables.TableFormat.column
        attribute_order = ('name', 'url', 'address',)
        verbose_name = 'Company'
        verbose_name_plural = 'Companies'_____no_output_____

##### Create a class `Person` to represent people_____no_output_____Next, create a class to represent people.
* Use `enum.Enum` and `obj_tables.EnumAttribute` to capture types of people.
* Use `obj_tables.ManyToOneAttribute` to indicate that each person can be employed by a company and have an address._____no_output_____

import enum

class PersonType(str, enum.Enum):
    family = 'family'
    friend = 'friend'
    business = 'business'

class Person(obj_tables.Model):
    name = obj_tables.StringAttribute(unique=True, primary=True, verbose_name='Name')
    type = obj_tables.EnumAttribute(PersonType, verbose_name='Type')
    company = obj_tables.ManyToOneAttribute(Company, related_name='employees', verbose_name='Company')
    email_address = obj_tables.EmailAttribute(verbose_name='Email address')
    phone_number = obj_tables.StringAttribute(verbose_name='Phone number')
    address = obj_tables.ManyToOneAttribute(Address, related_name='people', verbose_name='Address')

    class Meta(obj_tables.Model.Meta):
        table_format = obj_tables.TableFormat.row
        attribute_order = ('name', 'type', 'company', 'email_address', 'phone_number', 'address',)
        verbose_name = 'Person'
        verbose_name_plural = 'People'_____no_output_____

##### Using additional types of attributes_____no_output_____*ObjTables* supports a wide range of attributes in addition to those illustrated here. This includes attributes that can represent mathematical expressions, NumPy arrays, Pandas DataFrames, chemical structures, and biological sequences.

* Strings
  * `obj_tables.StringAttribute`: attribute for strings
  * `obj_tables.LongStringAttribute`: attribute for long strings; provided for compatibility with relational databases
  * `obj_tables.RegexAttribute`: attribute for strings which must match a regular expression
* Enumerations
  * `obj_tables.EnumAttribute`: attribute for enumerations (subclasses of `enum.Enum`)
* Numbers
  * `obj_tables.BooleanAttribute`: attribute for Boolean values
  * `obj_tables.IntegerAttribute`: attribute for integers
  * `obj_tables.PositiveIntegerAttribute`: attribute for positive integers
  * `obj_tables.FloatAttribute`: attribute for floats
  * `obj_tables.PositiveAttribute`: attribute for positive floats
* Internet addresses
  * `obj_tables.UrlAttribute`: attribute for URLs
  * `obj_tables.EmailAttribute`: attribute for email addresses
* Dates and times
  * `obj_tables.DateAttribute`: attribute for dates (`datetime.date`)
  * `obj_tables.TimeAttribute`: attribute for times (`datetime.time`)
  * `obj_tables.DateTimeAttribute`: attribute for date and times (`datetime.datetime`)
* Mathematics (`obj_tables.math`): attributes for NumPy arrays, Pandas DataFrames, and symbolic expressions
* Science (`obj_tables.sci`): attributes units, quantities, uncertainties, ontology terms, digital object identifiers, PubMed identifiers, and other identifiers
* Chemistry (`obj_tables.chem`): attributes for chemical structures and formulas
* Biology (`obj_table.bio`): attributes for DNA, RNA, and protein sequences, sequence features, and position-specific weight matrices
* Relationships
  * `obj_tables.OneToOneAttribute`: attribute for a one-to-one relationship to another object
  * `obj_tables.OneToManyAttribute`: attribute for a one-to-many relationship to multiple other objects
  * `obj_tables.ManyToOneAttribute`: attribute for a many-to-one relationship to another object
  * `obj_tables.ManyToManyAttribute`: attribute for a many-to-many relationship to multiple other objects

See https://objtables.org for more information._____no_output_____##### Customizing the validation of each attribute_____no_output_____Most attributes support optional arguments which can be used to define constraints on their values. For example,

* `obj_tables.StringAttribute`: supports two optional arguments, `min_length` and `max_length` which can be used to define the minimum and maximum valid string lengths.
* `obj_tables.RegexAttribute`: supports one optional argument, `pattern` which can be used to define a regular expression that each value must match.
* `obj_tables.IntegerAttribute`: supports two optional arguments, `min` and `max` which can be used to define the minimum and maximum valid values.
* `obj_tables.FloatAttribute`: supports two optional arguments, `min` and `max` which can be used to define the minimum and maximum valid values.

See https://objtables.org for more information._____no_output_____##### Encapsulate the classes into a schema module_____no_output_____For illustration purposes, we defined the classes for the address book schema in this Jupyter notebook. Generally, we recommend defining all of the classes for a schema in a Python module. This encapsulates schemas into a single (module) object, which makes it easy to perform operations on schemas such as generating UML diagrams. To illustrate the utility of schema modules, here we programmatically encapsulate the classes defined above into a module._____no_output_____

import types
schema = type('address_book', (types.ModuleType, ), {
    'Address': Address,
    'Company': Company,
    'Person': Person,
    'PersonType': PersonType,
})_____no_output_____

# Visualize the schema as a UML diagram_____no_output_____Second, use `obj_tables.utils.viz_schema` to visualize the schema for address books._____no_output_____##### Create a temporary file for the UML diagram of the schema_____no_output_____

import os
dir = 'Address book'
if not os.path.isdir(dir):
    os.makedirs(dir)
filename = os.path.join(dir, 'Address book.png')_____no_output_____

##### Generate a UML diagram of the schema_____no_output_____

from obj_tables.utils import viz_schema
viz_schema(schema, filename)_____no_output_____

##### Display the UML diagram_____no_output_____

from IPython.display import Image
Image(filename=filename)_____no_output_____


# Notebook from hanzopgp/TensorFlowTest
Path: Notebooks/PoemGenerationRNN/main.ipynb

# --> Importations_____no_output_____

import tensorflow as tf
import numpy as np
from sklearn.model_selection import train_test_split_____no_output_____

# --> Importation dataset poeme de Victor Hugo_____no_output_____

with open("../../Datasets/VictorHugoPoems/victorhugo.txt", "r", encoding='utf-8') as f:
    text = f.read()

print("Taille du text : ", len(text))
print("Texte avant preprocessing :\n", text[:100])Taille du text :  127286
Texte avant preprocessing :
 Parce que, jargonnant vêpres, jeûne et vigile,
Exploitant Dieu qui rêve au fond du firmament,
Vous a

# --> Preprocessing du dataset_____no_output_____

#Supprime les caracteres inutiles, les majuscules...
import unidecode
text = unidecode.unidecode(text)
text.lower()
text = text.replace("2", "")
text = text.replace("1", "")
text = text.replace("8", "")
text = text.replace("5", "")
text = text.replace(">", "")
text = text.replace("<", "")
text = text.replace("!", "")
text = text.replace("?", "")
text = text.replace("-", "")
text = text.replace("$", "")
text = text.replace(";", "")
text = text.strip()

#Supprime tous les doublons
vocab = set(text)

#Affichage resultat
print("Taille du vocabulaire : ", len(vocab))
print("Vocabulaire :\n", vocab)
print("Texte formate :\n", text[:100])Taille du vocabulaire :  57
Vocabulaire :
 {'H', 'f', ' ', 'n', 'm', 'e', 'h', '.', 'o', 'E', 'p', 'R', 'k', 'A', 'N', 'J', 'w', 'a', 'U', 'Y', 'B', 's', 'S', 'V', 'r', 'b', 'c', 'X', 't', 'K', 'G', 'P', 'i', 'v', '"', 'q', 'g', 'F', 'L', 'x', 'I', 'C', 'O', "'", 'l', 'Q', 'd', 'z', 'y', 'u', 'M', '\n', ':', ',', 'T', 'D', 'j'}
Texte formate :
 Parce que, jargonnant vepres, jeune et vigile,
Exploitant Dieu qui reve au fond du firmament,
Vous a
#On traduit maintenant tout le vocabulaire en nombre
vocab_size = len(vocab)
#Dictionnaire traduction
vocab_to_int = {l:i for i,l in enumerate(vocab)}
int_to_vocab = {i:l for i,l in enumerate(vocab)}
#Affichage
print("Vocab to int :\n", vocab_to_int)
print("Int to vocab :\n", int_to_vocab)Vocab to int :
 {'H': 0, 'f': 1, ' ': 2, 'n': 3, 'm': 4, 'e': 5, 'h': 6, '.': 7, 'o': 8, 'E': 9, 'p': 10, 'R': 11, 'k': 12, 'A': 13, 'N': 14, 'J': 15, 'w': 16, 'a': 17, 'U': 18, 'Y': 19, 'B': 20, 's': 21, 'S': 22, 'V': 23, 'r': 24, 'b': 25, 'c': 26, 'X': 27, 't': 28, 'K': 29, 'G': 30, 'P': 31, 'i': 32, 'v': 33, '"': 34, 'q': 35, 'g': 36, 'F': 37, 'L': 38, 'x': 39, 'I': 40, 'C': 41, 'O': 42, "'": 43, 'l': 44, 'Q': 45, 'd': 46, 'z': 47, 'y': 48, 'u': 49, 'M': 50, '\n': 51, ':': 52, ',': 53, 'T': 54, 'D': 55, 'j': 56}
Int to vocab :
 {0: 'H', 1: 'f', 2: ' ', 3: 'n', 4: 'm', 5: 'e', 6: 'h', 7: '.', 8: 'o', 9: 'E', 10: 'p', 11: 'R', 12: 'k', 13: 'A', 14: 'N', 15: 'J', 16: 'w', 17: 'a', 18: 'U', 19: 'Y', 20: 'B', 21: 's', 22: 'S', 23: 'V', 24: 'r', 25: 'b', 26: 'c', 27: 'X', 28: 't', 29: 'K', 30: 'G', 31: 'P', 32: 'i', 33: 'v', 34: '"', 35: 'q', 36: 'g', 37: 'F', 38: 'L', 39: 'x', 40: 'I', 41: 'C', 42: 'O', 43: "'", 44: 'l', 45: 'Q', 46: 'd', 47: 'z', 48: 'y', 49: 'u', 50: 'M', 51: '\n', 52: ':', 53: ',', 54: 'T', 55: 'D', 56: 'j'}
#Le dictionnaire nous permet de traduire notre text en nombre
encoded = [vocab_to_int[l] for l in text]
encoded_sentence = encoded[:100]
print(encoded_sentence)[31, 17, 24, 26, 5, 2, 35, 49, 5, 53, 2, 56, 17, 24, 36, 8, 3, 3, 17, 3, 28, 2, 33, 5, 10, 24, 5, 21, 53, 2, 56, 5, 49, 3, 5, 2, 5, 28, 2, 33, 32, 36, 32, 44, 5, 53, 51, 9, 39, 10, 44, 8, 32, 28, 17, 3, 28, 2, 55, 32, 5, 49, 2, 35, 49, 32, 2, 24, 5, 33, 5, 2, 17, 49, 2, 1, 8, 3, 46, 2, 46, 49, 2, 1, 32, 24, 4, 17, 4, 5, 3, 28, 53, 51, 23, 8, 49, 21, 2, 17]
decoded_sentence = [int_to_vocab[i] for i in encoded_sentence]
print(decoded_sentence)['P', 'a', 'r', 'c', 'e', ' ', 'q', 'u', 'e', ',', ' ', 'j', 'a', 'r', 'g', 'o', 'n', 'n', 'a', 'n', 't', ' ', 'v', 'e', 'p', 'r', 'e', 's', ',', ' ', 'j', 'e', 'u', 'n', 'e', ' ', 'e', 't', ' ', 'v', 'i', 'g', 'i', 'l', 'e', ',', '\n', 'E', 'x', 'p', 'l', 'o', 'i', 't', 'a', 'n', 't', ' ', 'D', 'i', 'e', 'u', ' ', 'q', 'u', 'i', ' ', 'r', 'e', 'v', 'e', ' ', 'a', 'u', ' ', 'f', 'o', 'n', 'd', ' ', 'd', 'u', ' ', 'f', 'i', 'r', 'm', 'a', 'm', 'e', 'n', 't', ',', '\n', 'V', 'o', 'u', 's', ' ', 'a']
decoded_sentence = "".join(decoded_sentence)
print(decoded_sentence)Parce que, jargonnant vepres, jeune et vigile,
Exploitant Dieu qui reve au fond du firmament,
Vous a

# --> Creation des batchs_____no_output_____

#Un batch = plusieurs sequences de mots
#Ce qu'on peut faire lorsqu'on a un dataset comme cela, on peut prendre une sequence de quelques mots
#Chaque lettre est une entree dont le target est la lettre suivante.
#Une incoherence peut arriver lors de l'analyse de la premiere lettre d'une sequence
#Car dans notre cellule RNN il n'a pas d'informations sur la lettre precedente car la memoire est nulle.
#Au lieu de lui mettre un etat nulle on lui mets l'etat retenu du batch precendent.
#On ne peut donc pas se permettre de selectionner des sequences aleatoires dans notre texte.
#On va donc seprarer notre texte en chunks
#Une epoch : un ensemble de batch
def gen_batch(inputs, targets, seq_len, batch_size, noise=0):

    chunk_size = (len(inputs) -1) // batch_size
    sequences_per_chunk = chunk_size // seq_len

    for seq in range(0, sequences_per_chunk):
        batch_inputs = np.zeros((batch_size, seq_len))
        batch_targets = np.zeros((batch_size, seq_len))
        for b in range(0, batch_size):
            fr = (b*chunk_size) + (seq*seq_len)
            to = fr + seq_len
            batch_inputs[b] = inputs[fr:to]
            batch_targets[b] = inputs[fr+1:to+1]

            if noise > 0: #"noise" aide le model a generaliser, evite l'overfitting
                noise_indices = np.random.choice(seq_len, noise)
                batch_inputs[b][noise_indices] = np.random.randint(0, vocab_size)

        yield batch_inputs, batch_targets #Permet d'appeler la fonction dans la boucle

inputs, targets = encoded, encoded[1:]
print("First inputs : ", inputs[:10])
print("First targets : ", targets[:10])First inputs :  [31, 17, 24, 26, 5, 2, 35, 49, 5, 53]
First targets :  [17, 24, 26, 5, 2, 35, 49, 5, 53, 2]
print("\n##################### Sans noise #####################")
i = 0
for batch_inputs, batch_targets in gen_batch(inputs, targets, seq_len=5, batch_size=32, noise=0): #Sequence de 5, batch de 32
    i += 1
    print("\n----------------------Step ", i, "----------------------")
    print("\nBatch input :\n", batch_inputs.shape, "\nBatch target shape :\n", batch_targets.shape)
    print("\nBatch input :\n", batch_inputs[0], "\nBatch target :\n", batch_targets[0])
    if i > 1:
        break
##################### Sans noise #####################

----------------------Step  1 ----------------------

Batch input :
 (32, 5)
Batch target shape :
 (32, 5)

Batch input :
 [31. 17. 24. 26.  5.]
Batch target :
 [17. 24. 26.  5.  2.]

----------------------Step  2 ----------------------

Batch input :
 (32, 5)
Batch target shape :
 (32, 5)

Batch input :
 [ 2. 35. 49.  5. 53.]
Batch target :
 [35. 49.  5. 53.  2.]
print("\n##################### Avec noise = 3 #####################")
i = 0
for batch_inputs, batch_targets in gen_batch(inputs, targets, seq_len=5, batch_size=32, noise=3): #Sequence de 5, batch de 32
    i += 1
    print("\n---------------------- Step ", i, " ----------------------")
    print("\nBatch input :\n", batch_inputs.shape, "\nBatch target shape :\n", batch_targets.shape)
    print("\nBatch input :\n", batch_inputs[0], "\nBatch target :\n", batch_targets[0])
    if i > 1:
        break
##################### Avec noise = 3 #####################

---------------------- Step  1  ----------------------

Batch input :
 (32, 5)
Batch target shape :
 (32, 5)

Batch input :
 [31. 17. 29. 29. 29.]
Batch target :
 [17. 24. 26.  5.  2.]

---------------------- Step  2  ----------------------

Batch input :
 (32, 5)
Batch target shape :
 (32, 5)

Batch input :
 [ 2. 13. 13. 13. 53.]
Batch target :
 [35. 49.  5. 53.  2.]

# --> One hot encoding_____no_output_____

#Les valeurs au dessus ne nous convienne pas pour entrainer un model il y a mieux.
#On va donc utiliser le one hot encoding pour simplifier la tache à notre model.
#Exemple de one hot encoding : a => 2 => [0, 1, 0, 0]
#Le one hot encoding est tres efficace lorsqu'on veut specifier des classes.
#En effet, il n'y a aucune raison qu'un nombres que nous donnons a un caractere
#ait un nombre plus eleve et donc avec plus de poids qu'un autre alors qu'il n'y
#a aucune hierarchie entre les caracteres.
class OneHot(tf.keras.layers.Layer): #On creer une custom layer OneHot

    def __init__(self, depth, **kwargs):
        super(OneHot, self).__init__(**kwargs)
        self.depth = depth

    def call(self, x, mask=None):
         #Transforme le x en int 32 et creer un vecteur one hot encoded
        return tf.one_hot(tf.cast(x, tf.int32), self.depth)

    def get_config(self): #Override get_config pour pouvoir save le model
        config = super().get_config().copy()
        config.update({
            'depth': self.depth,
        })
        return config_____no_output_____class RNNModel(tf.keras.Model):

    def __init__(self, vocab_size):
        super(RNNModel, self).__init__()
        self.one_hot = OneHot(len(vocab))

    def call(self, inputs):
        output_layer = self.one_hot(inputs)
        return output_layer

batch_inputs, batch_targets = next(gen_batch(inputs, targets, seq_len=50, batch_size=32)) #32 sequences, 50 elements
model = RNNModel(len(vocab))
output = model.predict(batch_inputs)[0][0]

print("Input letter :\n", batch_inputs[0][0])
print("Next letter prediction :\n", output)Input letter :
 31.0
Next letter prediction :
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0.]

# --> Creation du model RNN_____no_output_____

vocab_size = len(vocab)

#Input layer
#On ne set pas le nombre d'element dans les sequences
tf_inputs = tf.keras.Input(shape=(None,), batch_size=32)
#One hot layer
#En lui passant tf_inputs, on specifie la shape qu'on enverra dans la layer one_hot
one_hot = OneHot(vocab_size)(tf_inputs)
#LSTM layers
#"return_sequences" permet de specifier que l'on prend en compte plusieurs des anciennes
#cellules LSTM, si on met a False nous aurions que l'information de la derniere cellule LSTM
#"stateful" permet de specifier qu'a chaque appel on ne va pas reinitialiser les cellules.
#A chaque appel l'etat initial sera egal au dernier element de la sequence precedente
rnn_layer1 = tf.keras.layers.LSTM(128, return_sequences=True, stateful=True)(one_hot)
rnn_layer2 = tf.keras.layers.LSTM(128, return_sequences=True, stateful=True)(rnn_layer1)
#Dense layer
hidden_layer = tf.keras.layers.Dense(128, activation="relu")(rnn_layer2)
#Output layer
output_layer = tf.keras.layers.Dense(vocab_size, activation="softmax")(hidden_layer)

#Model
model = tf.keras.Model(inputs=tf_inputs, outputs=output_layer)_____no_output_____#Reset les cellules du RNN
model.reset_states()

#Creer un premier batch
batch_inputs, target_inputs = next(gen_batch(inputs, targets, seq_len=50, batch_size=32))

#Prediction pour un premier batch
outputs = model.predict(batch_inputs)
#Prediction de la premiere sortie
first_prediction = outputs[0][0]
print("First prediction :\n", first_prediction)

model.reset_states()

#Deuxieme prediction, c'est exactement la meme grace au stateful=True
outputs = model.predict(batch_inputs)
second_prediction = outputs[0][0]
print("Second prediction :\n", second_prediction)

#Check si les deux predictions sont egales avec un reset_state() entre les deux
assert(set(first_prediction)==set(second_prediction))First prediction :
 [0.0175922  0.01755079 0.01758404 0.01756127 0.01746607 0.01760784
 0.01756605 0.0175176  0.01746702 0.01754959 0.0174946  0.01754105
 0.01754688 0.01754523 0.01757365 0.01753175 0.01750153 0.01756241
 0.0175528  0.01750962 0.01759637 0.01758435 0.01750205 0.01752504
 0.01762402 0.01750478 0.01752348 0.01755448 0.01755507 0.01755614
 0.01752438 0.01752501 0.01755911 0.01761584 0.01760784 0.01751597
 0.0174827  0.01754175 0.01757086 0.01754703 0.01747712 0.01755028
 0.0175303  0.01752012 0.01752548 0.01755951 0.01750464 0.01755796
 0.01758248 0.01748202 0.01758216 0.01755241 0.0175486  0.01758036
 0.0175125  0.01750905 0.01758676]
Second prediction :
 [0.0175922  0.01755079 0.01758404 0.01756127 0.01746607 0.01760784
 0.01756605 0.0175176  0.01746702 0.01754959 0.0174946  0.01754105
 0.01754688 0.01754523 0.01757365 0.01753175 0.01750153 0.01756241
 0.0175528  0.01750962 0.01759637 0.01758435 0.01750205 0.01752504
 0.01762402 0.01750478 0.01752348 0.01755448 0.01755507 0.01755614
 0.01752438 0.01752501 0.01755911 0.01761584 0.01760784 0.01751597
 0.0174827  0.01754175 0.01757086 0.01754703 0.01747712 0.01755028
 0.0175303  0.01752012 0.01752548 0.01755951 0.01750464 0.01755796
 0.01758248 0.01748202 0.01758216 0.01755241 0.0175486  0.01758036
 0.0175125  0.01750905 0.01758676]
loss_object = tf.keras.losses.SparseCategoricalCrossentropy()
optimizer = tf.keras.optimizers.Adam(lr=0.001) #lr : learning rate
train_loss = tf.keras.metrics.Mean(name="train_loss")
train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()_____no_output_____@tf.function
def train_step(inputs, targets):
    with tf.GradientTape() as tape:
        #Fait une prediction sur le batch
        predictions = model(inputs)
        #Recupere l'erreur par rapport aux predictions faites
        loss = loss_object(targets, predictions)
    #Calcul du gradient
    gradients = tape.gradient(loss, model.trainable_variables)
    #Change les poids du model grace au gradient
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    #Garde l'information sur l'evolution de l'entrainement
    train_loss(loss)
    train_accuracy(targets, predictions)

@tf.function
def predict(inputs):
    # Fait une prediction sur tous le batch
    predictions = model(inputs)
    return predictions_____no_output_____

# --> Entrainement du model_____no_output_____

model.reset_states()

model.summary()

for epoch in range(10):
    #Pendant toute cette etape dans le for, on ne reinitialise pas les states
    for batch_inputs, batch_targets in gen_batch(inputs, targets, seq_len=100, batch_size=32, noise=13): #Sequence de taille 100, batch de 32
        train_step(batch_inputs, batch_targets)
    template = '\r Epoch {}, Train Loss: {}, Train Accuracy: {}'
    print(template.format(epoch,
                          train_loss.result(),
                          train_accuracy.result()*100), end="")
    model.reset_states() #On reinitialise le state pour la prochaine epochModel: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         [(32, None)]              0
_________________________________________________________________
one_hot_1 (OneHot)           (32, None, 57)            0
_________________________________________________________________
lstm (LSTM)                  (32, None, 128)           95232
_________________________________________________________________
lstm_1 (LSTM)                (32, None, 128)           131584
_________________________________________________________________
dense (Dense)                (32, None, 128)           16512
_________________________________________________________________
dense_1 (Dense)              (32, None, 57)            7353
=================================================================
Total params: 250,681
Trainable params: 250,681
Non-trainable params: 0
_________________________________________________________________
 Epoch 9, Train Loss: 2.650911808013916, Train Accuracy: 24.4056091308593755

# --> Sauveguarde du model_____no_output_____

import json
model.save("model_rnn.h5")

with open("model_rnn_vocab_to_int", "w") as f:
    f.write(json.dumps(vocab_to_int))
with open("model_rnn_int_to_vocab", "w") as f:
    f.write(json.dumps(int_to_vocab))_____no_output_____

# --> Generation de poeme_____no_output_____

import random

model.reset_states()

size_poetries = 300

poetries = np.zeros((32, size_poetries, 1))
sequences = np.zeros((32, 100))
for b in range(32):
    rd = np.random.randint(0, len(inputs) - 100)
    sequences[b] = inputs[rd:rd+100]

for i in range(size_poetries+1):
    if i > 0:
        poetries[:,i-1,:] = sequences
    softmax = predict(sequences)
    # Set the next sequences
    sequences = np.zeros((32, 1))
    for b in range(32):
        argsort = np.argsort(softmax[b][0])
        argsort = argsort[::-1]
        # Select one of the strongest 4 proposals
        sequences[b] = argsort[0]

for b in range(32):
    sentence = "".join([int_to_vocab[i[0]] for i in poetries[b]])
    print(sentence)
    print("\n=====================\n")e le cous le cous le coure et le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

 le coure et le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

es le mes le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le l

=====================

 le mes le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

es le poure et le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

e poure et le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le l

=====================

 le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous l

=====================

 le ces le ces le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le

=====================

 ces le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le

=====================

 cous le mes le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le l

=====================

e de ces le ces le poure et le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le c

=====================

es le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

 les le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le l

=====================

 de le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

e ces le ces le ces le poure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cou

=====================

 le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous l

=====================

es le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le

=====================

te de coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le

=====================

L de ces le ces le ces le poure et le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le l

=====================

 et le mes le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

t le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le l

=====================

 sour de poure et le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

 de ces le poure et le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le l

=====================

 les le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le l

=====================

 le ces le poure et le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le l

=====================

es le mes le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

e ces le ces le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le l

=====================

e le poure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le p

=====================

 ces le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

 de ces le ces le poure et le poure

Le cous le cous le cous le coure et le mes le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le

=====================

et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le cous le cous le cous le coure et le poure

Le c

=====================


# Notebook from dwitvliet/python-causality-handbook
Path: causal-inference-for-the-brave-and-true/09-Non-Compliance-and-LATE.ipynb

# Non Compliance and LATE

## Dipping our Toes into a Heterogeneous World

Previously, we've seen Instrumental Variables through a more traditional lens. IV was seen as some sort of natural experiment we can leverage. In contrast, modern IV practice draws a lot of insight from medical sciences. It partitions the world into 4 kinds of subjects, depending on how they respond to the instrumental variable.

1. Compliers
2. Never Takers
3. Always Takers
4. Defiers

This naming comes from pharmaceutical science. Imagine you are conducting an experiment to test the effect of a new drug on some illness. Each subject gets assigned to a treatment: a drug or a placebo. The compliers are the subjects that stick to what was assigned to them. If they get the placebo, they take it; if they get the drug, they also take it. The never takers are those subjects that refuse to take their medicine. Even if they are assigned the new drug, they won't take it. On the other hand, the always takers are those that can somehow get the new drug even if they were assigned to the placebo. Finally, Defiers are the ones that take the treatment if assigned to the control and take the control if assigned the treatment. You can think of them as that annoying child that does the opposite of what it is told. In practice, they are not so common (the defiers, not the kids), so we will often ignore them.

![img](./data/img/late/defiers.png)

Modern IV treats the instrument as a quasi-experimental design where compliance is not perfect. By doing so, it distinguishes internally from externally valid causal effects. Just as a reminder, an internally valid effect is one we are able to identify. It is valid in that particular setting, with that particular data. In IV, it would be the treatment effect on those where the instrument changes the treatment. External validity, on the other hand, would be concerned with the predictive power of that causal effect. It would ask the question of whether we could generalize the effect we found in this sample for other populations. For example, let's say you conducted an RCT in your university to figure out if people are generous or not, when given an incentive to donate. The experiment is well designed, but you only invite economics students to participate. You then find that all of them are selfish jerks. This is an internally valid conclusion. It's valid for those data points. But can you, from that experiment, infer that humanity is selfish? That's hardly the case. So we would question whether your experiment has external validity to generalize its results. Anyway, back to IV.

To make it more concrete, let's consider a case where you want to boost user engagement measured by in app purchase. One way to do that, is by asking your marketing department to come up with a push you can use to engage your users. They come up with a brilliant design and very fancy user interactions. With that push, you move on to design a randomised trial. You select 10000 random customers and, for each of them, you assign the push with 50% probability. However, when you execute the test, you notice that some customers who were assigned to receive the push are not receiving it. When you talk to the engineers, they say that it is because they probably have an older phone that doesn't support the kind of push the marketing team designed.

At first, you might think that this is no big deal. Instead of using treatment assigned as your treatment variable, you can just use the treatment received, right? It turns out it is not that simple. If you plot the causal graph of this whole situation, it would look something like this:_____no_output_____

import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
from scipy import stats
from matplotlib import style
import seaborn as sns
from matplotlib import pyplot as plt
import statsmodels.formula.api as smf
from linearmodels.iv import IV2SLS
import graphviz as gr

%matplotlib inline

style.use("fivethirtyeight")_____no_output_____g = gr.Digraph()

g.edge("push assigned", "push delivered")
g.edge("push delivered", "in app purchase")
g.edge("income", "in app purchase")
g.edge("income", "push delivered")
g.node("income", color="blue")
g_____no_output_____

Up on the causal graph, you have the push assignment. This is random by design, so nothing is causing it. Then, you have a node for if the push was delivered. Not everyone that was assigned to receive the push did it, so you have non compliance here. More specifically, you have some never takers: those that don't receive the treatment even if assigned to it. You also have reasons to suspect that this non compliance is not simply by chance. Since people with older phones are the one that don't get the push, you can argue that income is also causing push delivery. The richer the person, the more likely it is that he or she has a nicer phone, which in turn makes it more likely that he or she will receive the push. Finally, you have the outcome variable, in app purchase. Keep in mind that we don't know income, so we can't control it. With this in mind, let's examine what would happen if we simply used the push assigned as the treatment variable and if instead we used push delivered as the treatment.

In the first case, we would estimate the causal effect by the following difference in means:

$
ATE = E[Y | pushAssigned=1] - E[Y | pushAssigned=0]
$

As we are very aware by now, this is only unbiased estimate for \\(E[Y_1] - E[Y_0]\\) if the bias \\(E[Y_0|pushAssigned=0] - E[Y_0|pushAssigned=1]\\) is zero. Since `pushAssigned` is random, we know that the bias is zero. Does this settle it then? Not quite. You see, if we do this, we are actually answering a different question than the one we intended to. We would be finding the **causal effect of treatment assignment**, not of the treatment itself. But are they different or can we extrapolate the causal effect of treatment assignment to the ATE? In other words, is the causal effect of treatment assignment an unbiased estimate of the ATE?

As it turns out, it is not. Because of non compliance, the outcome of those assigned to the treatment will be pushed in the direction of the outcome of those assigned the control. Non compliance flips the treatment unintendedly, making the treatment and control more similar in the outcome. Don't confuse this with similarity on the variables. We want treated and control to be similar on the variables. That will make them comparable. What we don't want is for them to be similar in the outcome if there is indeed a treatment effect.

To see this, suppose first we have always takers. Some of them will be assigned to the control by chance. But those that are, will take the treatment anyway. This makes them essentially a treated group that is mixed up with the control. As a result of this mix, the causal effect will be harder to find when we have non compliance.

![img](./data/img/late/always_takers.png)

By the same reasoning, never takers will make those assigned to the treatment look a little bit like the untreated, because they don't take the treatment even if assigned to it. In this sense, **the causal effect of treatment assignment is biased towards zero** because non compliance shrinks the detectable impact. Another way to see this is to imagine an extreme case. Let's say that non compliance is really high. Treatment assignment says nothing about treatment received. Treatment received is purely random in this case. In IV language, that would be saying we have a very weak 1st stage. Using `Z` to denote the treatment assignment, we would have that

$
E[Y|Z=1] - E[Y|Z=0] = 0
$

There would be no more causal link of treatment assignment to the outcome. Z would be just a meaningless random variable floating there. _____no_output_____

g = gr.Digraph()

g.node("push assigned")
g.edge("push delivered", "in app purchase")
g.edge("income", "in app purchase")
g.edge("income", "push delivered")
g.node("income", color="blue")
g_____no_output_____

OK, so we've ruled out using the causal effect of the assignment as a way to estimate the causal effect of the treatment. So, how about just using the treatment received?

$
ATE = E[Y | push=1] - E[Y | push=0]
$

Once again, we need to think if this is biased or not, or if \\(E[Y_0|push=0] = E[Y_0|push=1]\\). Just by looking at the causal graph above, we know it is not the case. We have that unmeasured confounder, income, lurking around, and it is sure to mess things up. As we've said earlier, we know that failure in push delivery, in our case, is caused by customers having older phones. Which means that we probably have  \\(E[Y_0|push=0] < E[Y_0|push=1]\\). We think this is the case because customers with less money both have older phones which will lead to \\(push=0\\) and also a lower potential in app purchase \\(Y_0\\).

Bummer! We can't use treatment assigned nor treatment received to estimate our ATE. But fortunately, we know what we can use: Instrumental Variables. Here, treatment assigned is the perfect instrument for the treatment. It is as good as random and it only causes in-app purchase by the virtue of the treatment.

## Local Average Treatment Effect: LATE

Local average treatment effect makes it explicit the population for which we can estimate the causal effect. It is also another way to view IV, which gives other cool intuitions we can use. In modern IV, we see the instrument as starting a causal chain: Z causes T which causes Y. In this context, the exclusion restriction means that Z does not cause Y, unless by its effect on T. The 1st stage is now viewed as the causal effect of Z on T. We also rewrite potential outcomes with a double indexing notation, where the first index denotes the counterfactual for the instrument and the second index, for the treatment

$
\text{Potential Outcome}=\begin{cases}
Y_i(1, 1) \ \text{if } T_i=1, \ Z_i=1\\
Y_i(1, 0) \ \text{if } T_i=1, \ Z_i=0\\
Y_i(0, 1) \ \text{if } T_i=0, \ Z_i=1\\
Y_i(0, 0) \ \text{if } T_i=0, \ Z_i=0\\
\end{cases}
$

In a sense, the treatment becomes the outcome, at least in the first stage. This means we can also write it with potential outcome notation:

$
\text{Potential Treatment}=\begin{cases}
T_0 \ \text{if } Z_i=0 \\
T_1 \ \text{if } Z_i=1
\end{cases}
$

![img](./data/img/late/double_index.png)

Instrumental Variable assumptions can now be rewritten as follows

1. \\(T_{0i}, T_{1i} \perp Z_i \\) and \\(Y_i(T_{1i},1), Y_i(T_{0i},0) \perp Z_i \\). This is the independence Assumption. This says that the instrument is as good as randomly assigned. In other words, Z, the instrument, is not correlated with the potential treatments, which is the same as saying that people in different instrument groups are comparable.

2. \\(Y_i(1, 0)=Y_i(1, 1)=Y_{i1}\\) and \\(Y_i(1, 0)=Y_i(1, 1)=Y_{i0}\\). This is the exclusion restriction. It says that if I'm looking at the potential outcome for the treated, it is the same for both instrument groups. In other words, the instrument does not affect the potential outcome, which is the same as saying that the instrument only affects the outcome through the treatment.

3. \\(E[T_{1i}-T_{0i}] \neq 0\\). This is the existence of a 1st stage. It is saying that the potential outcome of the 1st stage, that is, the potential treatment, is NOT the same. Another way of saying this is that the instrument does affect the treatment.

4. \\(T_{i1} > T_{i0}\\). This is the monotonicity assumption. It is saying that if everyone had the instrument turned on, the treatment level would be higher than if everyone had the treatment turned off.

Now, let's review the Wald estimator to gain some further intuition on IV:

$
ATE = \dfrac{E[Y|Z=1]-E[Y|Z=0]}{E[T|Z=1]-E[T|Z=0]}
$

Let's take the first bit of it, \\(E[Y|Z=1]\\). Using the exclusion restriction, we can rewrite Y in terms of potential outcome like this.

$
E[Y_i|Z_i=1]=E[Y_{i0} + T_{i1}(Y_{i1} - Y_{i0})|Z=1]
$

Using independence, we can take out the conditioning on Z.

$
E[Y_i|Z_i=1]=E[Y_{i0} + T_{i1}(Y_{i1} - Y_{i0})]
$

With a similar argument, we get that

$
E[Y_i|Z_i=0]=E[Y_{i0} + T_{i0}(Y_{i1} - Y_{i0})]
$

We can now rewrite the numerator of the Wald estimator like this

$
E[Y|Z=1]-E[Y|Z=0] = E[(Y_{i1}-Y_{i0})(T_{i1}-T_{i0})]
$

Using monotonicity, we know that \\(T_{i1}-T_{i0}\\) is 0 or 1, so

$
E[(Y_{i1}-Y_{i0})(T_{i1}-T_{i0})] = E[(Y_{i1}-Y_{i0})|T_{i1}>T_{i0}]P(T_{i1}>T_{i0})
$

Using a similar argument to tackle the denominator, we get that

$
E[T|Z=1]-E[T|Z=0]=E[T_{i1}-T_{i0}]=P(T_{i1}>T_{i0})
$

All of this allows us to look at the Wald estimator like this:

$
ATE = \dfrac{E[(Y_{i1}-Y_{i0})|T_{i1}>T_{i0}]P(T_{i1}>T_{i0})}{P(T_{i1}>T_{i0})}=E[(Y_{i1}-Y_{i0})|T_{i1}>T_{i0}]
$

That is, the ATE as estimated by IV is the ATE on the subpopulation where \\(T_{i1}>T_{i0}\\). And if you think about compliance, which population is this? This is the population where those that get the instrument turned on have the treatment level higher than if they had the instrument turned off. In other words, this is the compiler population. Just so we can remember,

1. Compilers means that \\(T_{i1}>T_{i0}\\)
2. Never Takers \\(T_{i1}=T_{i0}=0\\)
3. Always Takers \\(T_{i1}=T_{i0}=1\\)

The conclusion of this is that IV says nothing about the effect on never takers, always takers or defiers, because the treatment is unchanged for them! **IV only finds the treatment effect for the compliers**.

## Effect on Engagement

Let's see how all of this plays out in the case study where we try to estimate the effect of a push on in-app purchase. The causal graph is the one we depicted above, so I won't repeat it here. The data we have is on both push assigned, the randomised instrument, and push delivered, the treatment variable._____no_output_____

data = pd.read_csv("./data/app_engagement_push.csv")

data.head()_____no_output_____

First, let's run OLS to see what it would give us._____no_output_____

ols = IV2SLS.from_formula("in_app_purchase ~ 1 + push_assigned + push_delivered", data).fit()
ols.summary.tables[1]_____no_output_____

OLS is saying that the treatment effect is R\$ 27.60, that is, the push increases in-app purchase by 27.6 reais. However, we have reasons to believe this is a biased estimate. We know that older phones are having trouble in receiving the push, so, probably, richer customers, with newer phones, are the compilers. Since the ones that get the treatment also have more money, we believe this bias is positive and the true impact of the push is lower. In other words, we probably have \\(E[Y_0|T=0] < E[Y_0|T=1]\\).

Now, let's try to estimate this effect with Instrumental Variables. First, let's run the first stage._____no_output_____

first_stage = IV2SLS.from_formula("push_delivered ~ 1 + push_assigned", data).fit()
first_stage.summary.tables[1]_____no_output_____

Looks like we have a strong first stage. Those that get assigned to get the push get it 71.8% of the time. This means that we have something like 28% of never takers. We also have strong reasons to believe there are no always takers, since the intercept parameter is estimated to be zero. This means that no one get's the push if it is not assigned to it. Given the design of our experiment, this is expected.

Let's now run the reduced form:_____no_output_____

reduced_form = IV2SLS.from_formula("in_app_purchase ~ 1 + push_assigned", data).fit()
reduced_form.summary.tables[1]_____no_output_____

The reduced form shows that the causal effect of treatment assignment is 2.36. This means that assigning someone to receive the push increases the in-app purchase by 2.36 reais.

If we divide the reduced form by the first stage, we scale the effect of the instrument by the units of the treatment, we get \\(2.3636/0.7176=3.29\\). Running the 2SLS, we get these same estimates, with the bonus of correct standard errors._____no_output_____

iv = IV2SLS.from_formula("in_app_purchase ~ 1 + [push_delivered ~ push_assigned]", data).fit()
iv.summary.tables[1]_____no_output_____

This shows that the result with 2SLS is much lower than the one we got with OLS: 3.29 against 27.60. This makes sense, since the causal effect estimated with OLS is positively biased. We also need to remember about LATE. 3.29 is the average causal effect on compilers. Unfortunately, we can't say anything about those never takers. This means that we are estimating the effect on the richer segment of the population that have newer phones.

## Key Ideas

Here we've looked at a more modern view of Instrumental Variables. We saw how IV can be seen as a causal chain where the instrument causes the treatment which causes the outcome. In this view, we took a look at compliance to understand ATE in IV estimation and we've figured out that it was the LATE on compilers.

## References

I like to think of this entire book as a tribute to Joshua Angrist, Alberto Abadie and Christopher Walters for their amazing Econometrics class. Most of the ideas here are taken from their classes at the American Economic Association. Watching them is what is keeping me sane during this tough year of 2020.
* [Cross-Section Econometrics](https://www.aeaweb.org/conference/cont-ed/2017-webcasts)
* [Mastering Mostly Harmless Econometrics](https://www.aeaweb.org/conference/cont-ed/2020-webcasts)

I'll also like to reference the amazing books from Angrist. They have shown me that Econometrics, or 'Metrics as they call it, is not only extremely useful but also profoundly fun.

* [Mostly Harmless Econometrics](https://www.mostlyharmlesseconometrics.com/)
* [Mastering 'Metrics](https://www.masteringmetrics.com/)

My final reference is Miguel Hernan and Jamie Robins' book. It has been my trustworthy companion in the most thorny causal questions I had to answer.

* [Causal Inference Book](https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/)

![img](./data/img/poetry.png)

## Contribute

Causal Inference for the Brave and True is an open-source material on causal inference, the statistics of science. It uses only free software, based in Python. Its goal is to be accessible monetarily and intellectually.
If you found this book valuable and you want to support it, please go to [Patreon](https://www.patreon.com/causal_inference_for_the_brave_and_true). If you are not ready to contribute financially, you can also help by fixing typos, suggesting edits or giving feedback on passages you didn't understand. Just go to the book's repository and [open an issue](https://github.com/matheusfacure/python-causality-handbook/issues). Finally, if you liked this content, please share it with others who might find it useful and give it a [star on GitHub](https://github.com/matheusfacure/python-causality-handbook/stargazers)._____no_output_____


# Notebook from RonnieGandhi/OLIVIA
Path: NLP-UI/TextSummarizer.ipynb

<code>
import torch
from summarizer import Summarizer
from summarizer.coreference_handler import CoreferenceHandler
from transformers import *

body = '''despite the fact that tony stark does not have any superpowers he has an almost superhuman ability to predict problems in the future and proactively form plans to solve those problems after loki was able to successfully manipulate the hulk into rampaging and nearly destroying the shield helle carrier during the events of avengers it became obvious that tony and the rest of the avengers needed a better contingency plan for the big guy if he ever got out of control again during iron man three we see that iron ban had begun to start developing a series of more specialized armors for tackling specific scenarios one of those souths is seen during the events of age of ultronn when scarlet which is able to force halt in a transforming and rampaging through to hannesburg south africa starkas force to deployed the mark forty four a k a the hullcluster armor today's bidil is going to cover everything we know about this thick armour and its capabilities now before we begin the version of the hawkbuster armor that we're discussing to day is the version from the m c u and not from the comics as with our other lower videos it's important to make this distinction because the m c u is not the main comic book six one six universe and as most of us longtime comic readers know the m c u takes creative liberties with a lot of the source material so jest keep that in mind were only talking about the halkbuster from the films and supplemental film material the mark forty four armour begain development presumably after the events of avengers as a collaborative effort between tony stark and bruce banner when bruce transforms into the hulk he has described the control of halks's actions as a steering wheel where hulk and bruce both have a hand on the wheel however when forced to change in the avengers by loki and again in age of altron by scarlet whitch hulk takes over and begins to rampage causing complete havoc bruce helps tony develop the mark forty four whose sole purpose is to take on the halt in the event that he becomes inconsolable the holtbuster armors deployed via a mobile service delivery module that can be deployed anywhere in the world named veronica director josweed named the system veronica after the archi comics character of the same name veronica was a rival to archie's girlfriend betty jos said i just decided to call the armor of veronica because banner used to be in love with a girl named betty and veronicas the opposite of that veronica is essentially a giant mobile floating armory that houses the mark forty four as well as replacement parts for the haltbuster armor unit when tony calls for veronicas aid the holkbuster armors deployed and rockets its way to him this is a clear evolution of the teck that tony was working on to some degree of success in ironman three in the mark forty two k a the prodigal sun armor the halbuster armour is considered modular in the sense that the armors parts can be switched and replaced on the fly it was presumably designed this way to account for the fact that any fight with the hulk would surely damage the unit to the point where it would need to have replacement parts swamped in and out this is evidenced in the fight when the halk rans a traffic light pole through the suits left arm just below the shoulder and the hawkbuster atjacks the armor and installs a new one in place continuing to allow it start to do battle at banner in order to generate the significantly improved strength at the unit boasted the armor had to be powered by multiple arcreactors there is one in the center of the chest one on the back of each leg two on the front of each leg four on the back near the propulsion system three on each lat one under each shoulder paldron and one on the palm of each hand all of this for a total of at least twenty one archreactors that we could count the strength of the armour was able to consistently match the rampaging halt strength as well as his striking power the suit was strong enough to send halk flying down a street in one punch as well as strong enough to knock his tooth out when combined with the propulsion system it's even able to body slam hulk hold him down firmly in place and land a series of consecutive plunches to the head ve a jack hammer like hydraulic battering rain mechanism embedded in the arms the suits enhanced durability is also consistently tested throughout the battle the suit features a reinforced exoskeleton that helps protect tony inside of the suit halk throws cars at it shoulder charges it punches it kicks it rips off or completely destroys multiple various pieces of the souit and hulk even throws it through various buildings and just generally wails on this thing pretty hard the suit is capable of flight through a mixture of the archreactors on its back as well as its jet propulsion system ar creactors on the palm of the hands are capable of firing unibeams similar to what iron men has been able to do with other armors except presumably much stronger it has a really cool locking mechanism built into the arms that allow the hand to retract and lock the halt's arm within it and use it for restraint the suit also has a high power chemical sendative spray built into it that did absolutely nothing against the hawk but hey it is the thought that counts the suit also has various missiles built into it and tony uses those missiles to weaken the structural integrity of a building that he ultimately destroys when crashing the hulk through it the suit also features an ai within it similar to jarvis but its full ability set is unknown at this point in the m c u jarvis was presumed destroyed by altron we know its capable of delivering commands to veronica feeding diagnostics to the suit backto tony compiling damage reports scanning large areas and structures for potential threats and casualties in purchasing whole buildings to destroy ultimately the suit proved successful in stopping the hulk from rampaging in johannesburg even though it came with a tremendous amount of damage in the city the mark forty four is eventually improved upon in the form of the mark forty eight or the halkbuster two point o the hullcluster two features the same modular design and multiple arch reactors with a somewhat more streamlined appearance the suit was first used in the battleo l conda where it destroyed a number of outriders as well as did battle with the mass of colebsidian this holtbuster was having a difficult time dealing with ubsidian but was able to defeat and kill him when banner tricked him in distabbing one of the halkbusters broken arms then triggered the propulsion system on the arm which forced the behemeth into a conda's protective dome causing a mass of explosion we can assume that the suit was physically stronger and more durable than the previous model because tony and bruce would have learned from the battle of johannesburg and improved upon the model in every way outside of that the second interation of the hawkbuster was only used one more time during the capture of danas on his vacation planet danos was significantly weakened at that point so the suit was more than capable of restraining him until ultimately he was killed the holkbuster suit is definitely one of the coolest armors that wehave seen in the m c u and considering the pantheon of awesome armors that tony has that sang quite a lot thank you gus for watching this video we hope you learned a lot about the thickest battle armour of them all the halkbuster and if you did please consider subscribing and dropping ha like on this video better yet drop a comment down below letting us know what other suits of armors you like us to cover on the channel and maybe we can eventually cover them all this has been nicke with key issues and remember the motto hawkbusters over everything'''

custom_config = AutoConfig.from_pretrained('allenai/scibert_scivocab_uncased')
custom_config.output_hidden_states=True
custom_tokenizer = AutoTokenizer.from_pretrained('allenai/scibert_scivocab_uncased')
custom_model = AutoModel.from_pretrained('allenai/scibert_scivocab_uncased', config=custom_config)

# handler = CoreferenceHandler(greedyness=.4)
model = Summarizer(custom_model = custom_model, custom_tokenizer=custom_tokenizer)

model(body)/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: spacy.morphology.Morphology size changed, may indicate binary incompatibility. Expected 104 from C header, got 112 from PyObject
  return f(*args, **kwds)
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: spacy.vocab.Vocab size changed, may indicate binary incompatibility. Expected 96 from C header, got 104 from PyObject
  return f(*args, **kwds)
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: spacy.tokens.span.Span size changed, may indicate binary incompatibility. Expected 72 from C header, got 80 from PyObject
  return f(*args, **kwds)
body = '''
The Chrysler Building, the famous art deco New York skyscraper, will be sold for a small fraction of its previous sales price.
The deal, first reported by The Real Deal, was for $150 million, according to a source familiar with the deal.
Mubadala, an Abu Dhabi investment fund, purchased 90% of the building for $800 million in 2008.
Real estate firm Tishman Speyer had owned the other 10%.
The buyer is RFR Holding, a New York real estate company.
Officials with Tishman and RFR did not immediately respond to a request for comments.
It's unclear when the deal will close.
The building sold fairly quickly after being publicly placed on the market only two months ago.
The sale was handled by CBRE Group.
The incentive to sell the building at such a huge loss was due to the soaring rent the owners pay to Cooper Union, a New York college, for the land under the building.
The rent is rising from $7.75 million last year to $32.5 million this year to $41 million in 2028.
Meantime, rents in the building itself are not rising nearly that fast.
While the building is an iconic landmark in the New York skyline, it is competing against newer office towers with large floor-to-ceiling windows and all the modern amenities.
Still the building is among the best known in the city, even to people who have never been to New York.
It is famous for its triangle-shaped, vaulted windows worked into the stylized crown, along with its distinctive eagle gargoyles near the top.
It has been featured prominently in many films, including Men in Black 3, Spider-Man, Armageddon, Two Weeks Notice and Independence Day.
The previous sale took place just before the 2008 financial meltdown led to a plunge in real estate prices.
Still there have been a number of high profile skyscrapers purchased for top dollar in recent years, including the Waldorf Astoria hotel, which Chinese firm Anbang Insurance purchased in 2016 for nearly $2 billion, and the Willis Tower in Chicago, which was formerly known as Sears Tower, once the world's tallest.
Blackstone Group (BX) bought it for $1.3 billion 2015.
The Chrysler Building was the headquarters of the American automaker until 1953, but it was named for and owned by Chrysler chief Walter Chrysler, not the company itself.
Walter Chrysler had set out to build the tallest building in the world, a competition at that time with another Manhattan skyscraper under construction at 40 Wall Street at the south end of Manhattan. He kept secret the plans for the spire that would grace the top of the building, building it inside the structure and out of view of the public until 40 Wall Street was complete.
Once the competitor could rise no higher, the spire of the Chrysler building was raised into view, giving it the title.
'''

model(body, min_length =150)_____no_output_____body1 = '''
Before the emergence of the Wakanda nation, mystic beings known as Originators were expelled from the region by the Heliopolitans.[9] This was later retconned as the Orisha,[10] the pantheon of Wakanda consisting of Thoth, Ptah, Mujaji, Kokou and Bast, the Panther Goddess.[11][12]

In the distant past, a massive meteorite made up of the element vibranium crashed in Wakanda. It was unearthed a generation before the events of the present day. T'Challa, the current Black Panther, is the son of T'Chaka, the Black Panther before him and a descendant of Bashenga. Knowing that others would attempt to manipulate and dominate Wakanda for this rare and valuable resource, T'Chaka conceals his country from the outside world. He sells off minute amounts of the valuable vibranium while surreptitiously sending the country's best scholars to study abroad, consequently turning Wakanda into one of the world's most technologically advanced nations. Eventually, however, the explorer Ulysses Klaue finds his way to Wakanda and covers up his work on a vibranium-powered, sound-based weapon. When exposed, Klaue kills T'Chaka, only to see his "sound blaster" turned on him by a grieving teenaged T'Challa. Klaue's right hand is destroyed, and he and his men flee.[13]

Wakanda has an unusually high rate of mutation due to the dangerously mutagenic properties of the Vibranium Mound. A large number of these Wakandan Mutates are working for Erik Killmonger.[13]

Vibranium radiation has permeated much of Wakanda's flora and fauna, including the Heart-Shaped Herb eaten by members of the Black Panther Cult (although T'Challa once allowed a dying Spider-Man to eat it in the hope that it would help him deal with a mysterious illness) and the flesh of the White Gorilla eaten by the members of the White Gorilla Cult.

In the 2008 "Secret Invasion" storyline, Skrull forces led by Commander K'vvvr invade Wakanda and engage Black Panther and his forces. Due to heavy resistance to the deployment of technological developments, both sides are forced to fight with swords and spears. The Wakandan forces voluntarily wear panther masks; this prevents the Skrulls from focusing attacks on their leader. Despite losses, the Wakandans defeat the Skrulls. They kill every single one, including K'vvvr, and send their ship back, packed with the bodies. A warning against invading Wakanda is left written on the wall of the ship's control center.[14]

While under the cosmic power of the Phoenix Force, Namor attacks Wakanda for hiding the Avengers and destroys much of the country with a tidal wave. After the attack, all mutants are banned from Wakanda as stated by Black Panther and attack on some students from the Jean Grey school by its people who barely flee with the help of Storm.[15]
'''

model(body1)_____no_output_____body2 = '''
One of the biggest challenges in natural language processing (NLP) is the shortage of training data. Because NLP is a diversified field with many distinct tasks, most task-specific datasets contain only a few thousand or a few hundred thousand human-labeled training examples. However, modern deep learning-based NLP models see benefits from much larger amounts of data, improving when trained on millions, or billions, of annotated training examples. To help close this gap in data, researchers have developed a variety of techniques for training general purpose language representation models using the enormous amount of unannotated text on the web (known as pre-training). The pre-trained model can then be fine-tuned on small-data NLP tasks like question answering and sentiment analysis, resulting in substantial accuracy improvements compared to training on these datasets from scratch.

This week, we open sourced a new technique for NLP pre-training called Bidirectional Encoder Representations from Transformers, or BERT. With this release, anyone in the world can train their own state-of-the-art question answering system (or a variety of other models) in about 30 minutes on a single Cloud TPU, or in a few hours using a single GPU. The release includes source code built on top of TensorFlow and a number of pre-trained language representation models. In our associated paper, we demonstrate state-of-the-art results on 11 NLP tasks, including the very competitive Stanford Question Answering Dataset (SQuAD v1.1).

What Makes BERT Different?
BERT builds upon recent work in pre-training contextual representations — including Semi-supervised Sequence Learning, Generative Pre-Training, ELMo, and ULMFit. However, unlike these previous models, BERT is the first deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus (in this case, Wikipedia).

Why does this matter? Pre-trained representations can either be context-free or contextual, and contextual representations can further be unidirectional or bidirectional. Context-free models such as word2vec or GloVe generate a single word embedding representation for each word in the vocabulary. For example, the word “bank” would have the same context-free representation in “bank account” and “bank of the river.” Contextual models instead generate a representation of each word that is based on the other words in the sentence. For example, in the sentence “I accessed the bank account,” a unidirectional contextual model would represent “bank” based on “I accessed the” but not “account.” However, BERT represents “bank” using both its previous and next context — “I accessed the ... account” — starting from the very bottom of a deep neural network, making it deeply bidirectional.

A visualization of BERT’s neural network architecture compared to previous state-of-the-art contextual pre-training methods is shown below. The arrows indicate the information flow from one layer to the next. The green boxes at the top indicate the final contextualized representation of each input word:

BERT is deeply bidirectional, OpenAI GPT is unidirectional, and ELMo is shallowly bidirectional.
The Strength of Bidirectionality
If bidirectionality is so powerful, why hasn’t it been done before? To understand why, consider that unidirectional models are efficiently trained by predicting each word conditioned on the previous words in the sentence. However, it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that’s being predicted to indirectly “see itself” in a multi-layer model.

To solve this problem, we use the straightforward technique of masking out some of the words in the input and then condition each word bidirectionally to predict the masked words. For example:

While this idea has been around for a very long time, BERT is the first time it was successfully used to pre-train a deep neural network.

BERT also learns to model relationships between sentences by pre-training on a very simple task that can be generated from any text corpus: Given two sentences A and B, is B the actual next sentence that comes after A in the corpus, or just a random sentence? For example:

Training with Cloud TPUs
Everything that we’ve described so far might seem fairly straightforward, so what’s the missing piece that made it work so well? Cloud TPUs. Cloud TPUs gave us the freedom to quickly experiment, debug, and tweak our models, which was critical in allowing us to move beyond existing pre-training techniques. The Transformer model architecture, developed by researchers at Google in 2017, also gave us the foundation we needed to make BERT successful. The Transformer is implemented in our open source release, as well as the tensor2tensor library.

Results with BERT
To evaluate performance, we compared BERT to other state-of-the-art NLP systems. Importantly, BERT achieved all of its results with almost no task-specific changes to the neural network architecture. On SQuAD v1.1, BERT achieves 93.2% F1 score (a measure of accuracy), surpassing the previous state-of-the-art score of 91.6% and human-level score of 91.2%:

BERT also improves the state-of-the-art by 7.6% absolute on the very challenging GLUE benchmark, a set of 9 diverse Natural Language Understanding (NLU) tasks. The amount of human-labeled training data in these tasks ranges from 2,500 examples to 400,000 examples, and BERT substantially improves upon the state-of-the-art accuracy on all of them:

Making BERT Work for You
The models that we are releasing can be fine-tuned on a wide variety of NLP tasks in a few hours or less. The open source release also includes code to run pre-training, although we believe the majority of NLP researchers who use BERT will never need to pre-train their own models from scratch. The BERT models that we are releasing today are English-only, but we hope to release models which have been pre-trained on a variety of languages in the near future.

The open source TensorFlow implementation and pointers to pre-trained BERT models can be found at http://goo.gl/language/bert. Alternatively, you can get started using BERT through Colab with the notebook “BERT FineTuning with Cloud TPUs.”

You can also read our paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" for more details.

'''

model(body2)_____no_output_____body3 = '''
We introduce a new neural architecture to learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence. Such problems cannot be trivially addressed by existent approaches such as sequence-to-sequence and Neural Turing Machines, because the number of target classes in each step of the output depends on the length of the input, which is variable. Problems such as sorting variable sized sequences, and various combinatorial optimization problems belong to this class. Our model solves the problem of variable size output dictionaries using a recently proposed mechanism of neural attention. It differs from the previous attention attempts in that, instead of using attention to blend hidden units of an encoder to a context vector at each decoder step, it uses attention as a pointer to select a member of the input sequence as the output. We call this architecture a Pointer Net (Ptr-Net). We show Ptr-Nets can be used to learn approximate solutions to three challenging geometric problems -- finding planar convex hulls, computing Delaunay triangulations, and the planar Travelling Salesman Problem -- using training examples alone. Ptr-Nets not only improve over sequence-to-sequence with input attention, but also allow us to generalize to variable size output dictionaries. We show that the learnt models generalize beyond the maximum lengths they were trained on. We hope our results on these tasks will encourage a broader exploration of neural learning for discrete problems.
'''

model(body3)_____no_output_____
</code>