# Notebook from Saharsh007/open-qas
Path: learning-phase/sequence-models/week1/saharsh/Dinosaur Island -- Character-level language model/Dinosaurus Island -- Character level language model final - v3.ipynb

# Character level language model - Dinosaurus land

Welcome to Dinosaurus Island! 65 million years ago, dinosaurs existed, and in this assignment they are back. You are in charge of a special task. Leading biology researchers are creating new breeds of dinosaurs and bringing them to life on earth, and your job is to give names to these dinosaurs. If a dinosaur does not like its name, it might go beserk, so choose wisely!

Luckily you have learned some deep learning and you will use it to save the day. Your assistant has collected a list of all the dinosaur names they could find, and compiled them into this [dataset](dinos.txt). (Feel free to take a look by clicking the previous link.) To create new dinosaur names, you will build a character level language model to generate new names. Your algorithm will learn the different name patterns, and randomly generate new names. Hopefully this algorithm will keep you and your team safe from the dinosaurs' wrath!

By completing this assignment you will learn:

- How to store text data for processing using an RNN
- How to synthesize data, by sampling predictions at each time step and passing it to the next RNN-cell unit
- How to build a character-level text generation recurrent neural network
- Why clipping the gradients is important

We will begin by loading in some functions that we have provided for you in `rnn_utils`. Specifically, you have access to functions such as `rnn_forward` and `rnn_backward` which are equivalent to those you've implemented in the previous assignment. _____no_output_____

import numpy as np
from utils import *
import random_____no_output_____

## 1 - Problem Statement

### 1.1 - Dataset and Preprocessing

Run the following cell to read the dataset of dinosaur names, create a list of unique characters (such as a-z), and compute the dataset and vocabulary size. _____no_output_____

data = open('dinos.txt', 'r').read()
data= data.lower()
chars = list(set(data))
data_size, vocab_size = len(data), len(chars)
print('There are %d total characters and %d unique characters in your data.' % (data_size, vocab_size))_____no_output_____

The characters are a-z (26 characters) plus the "\n" (or newline character), which in this assignment plays a role similar to the `<EOS>` (or "End of sentence") token we had discussed in lecture, only here it indicates the end of the dinosaur name rather than the end of a sentence. In the cell below, we create a python dictionary (i.e., a hash table) to map each character to an index from 0-26. We also create a second python dictionary that maps each index back to the corresponding character character. This will help you figure out what index corresponds to what character in the probability distribution output of the softmax layer. Below, `char_to_ix` and `ix_to_char` are the python dictionaries. _____no_output_____

char_to_ix = { ch:i for i,ch in enumerate(sorted(chars)) }
ix_to_char = { i:ch for i,ch in enumerate(sorted(chars)) }
print(ix_to_char)_____no_output_____

### 1.2 - Overview of the model

Your model will have the following structure:

- Initialize parameters
- Run the optimization loop
    - Forward propagation to compute the loss function
    - Backward propagation to compute the gradients with respect to the loss function
    - Clip the gradients to avoid exploding gradients
    - Using the gradients, update your parameter with the gradient descent update rule.
- Return the learned parameters

 **Figure 1**: Recurrent Neural Network, similar to what you had built in the previous notebook "Building a RNN - Step by Step".

At each time-step, the RNN tries to predict what is the next character given the previous characters. The dataset $X = (x^{\langle 1 \rangle}, x^{\langle 2 \rangle}, ..., x^{\langle T_x \rangle})$ is a list of characters in the training set, while $Y = (y^{\langle 1 \rangle}, y^{\langle 2 \rangle}, ..., y^{\langle T_x \rangle})$ is such that at every time-step $t$, we have $y^{\langle t \rangle} = x^{\langle t+1 \rangle}$. _____no_output_____## 2 - Building blocks of the model

In this part, you will build two important blocks of the overall model:
- Gradient clipping: to avoid exploding gradients
- Sampling: a technique used to generate characters

You will then apply these two functions to build the model._____no_output_____### 2.1 - Clipping the gradients in the optimization loop

In this section you will implement the `clip` function that you will call inside of your optimization loop. Recall that your overall loop structure usually consists of a forward pass, a cost computation, a backward pass, and a parameter update. Before updating the parameters, you will perform gradient clipping when needed to make sure that your gradients are not "exploding," meaning taking on overly large values.

In the exercise below, you will implement a function `clip` that takes in a dictionary of gradients and returns a clipped version of gradients if needed. There are different ways to clip gradients; we will use a simple element-wise clipping procedure, in which every element of the gradient vector is clipped to lie between some range [-N, N]. More generally, you will provide a `maxValue` (say 10). In this example, if any component of the gradient vector is greater than 10, it would be set to 10; and if any component of the gradient vector is less than -10, it would be set to -10. If it is between -10 and 10, it is left alone.

 **Figure 2**: Visualization of gradient descent with and without gradient clipping, in a case where the network is running into slight "exploding gradient" problems.

**Exercise**: Implement the function below to return the clipped gradients of your dictionary `gradients`. Your function takes in a maximum threshold and returns the clipped versions of your gradients. You can check out this [hint](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.clip.html) for examples of how to clip in numpy. You will need to use the argument `out = ...`._____no_output_____

### GRADED FUNCTION: clip

def clip(gradients, maxValue):
    '''
    Clips the gradients' values between minimum and maximum.

    Arguments:
    gradients -- a dictionary containing the gradients "dWaa", "dWax", "dWya", "db", "dby"
    maxValue -- everything above this number is set to this number, and everything less than -maxValue is set to -maxValue

    Returns:
    gradients -- a dictionary with the clipped gradients.
    '''

    dWaa, dWax, dWya, db, dby = gradients['dWaa'], gradients['dWax'], gradients['dWya'], gradients['db'], gradients['dby']

    ### START CODE HERE ###
    # clip to mitigate exploding gradients, loop over [dWax, dWaa, dWya, db, dby]. (≈2 lines)
    for gradient in [dWax, dWaa, dWya, db, dby]:
        np.clip(gradient,-maxValue,maxValue,out=gradient)
    ### END CODE HERE ###

    gradients = {"dWaa": dWaa, "dWax": dWax, "dWya": dWya, "db": db, "dby": dby}

    return gradients_____no_output_____np.random.seed(3)
dWax = np.random.randn(5,3)*10
dWaa = np.random.randn(5,5)*10
dWya = np.random.randn(2,5)*10
db = np.random.randn(5,1)*10
dby = np.random.randn(2,1)*10
gradients = {"dWax": dWax, "dWaa": dWaa, "dWya": dWya, "db": db, "dby": dby}
gradients = clip(gradients, 10)
print("gradients[\"dWaa\"][1][2] =", gradients["dWaa"][1][2])
print("gradients[\"dWax\"][3][1] =", gradients["dWax"][3][1])
print("gradients[\"dWya\"][1][2] =", gradients["dWya"][1][2])
print("gradients[\"db\"][4] =", gradients["db"][4])
print("gradients[\"dby\"][1] =", gradients["dby"][1])_____no_output_____

** Expected output:**

    **gradients["dWaa"][1][2] **

    10.0

    **gradients["dWax"][3][1]**

    -10.0

    **gradients["dWya"][1][2]**

0.29713815361

    **gradients["db"][4]**

[ 10.]

    **gradients["dby"][1]**

[ 8.45833407]

_____no_output_____### 2.2 - Sampling

Now assume that your model is trained. You would like to generate new text (characters). The process of generation is explained in the picture below:

 **Figure 3**: In this picture, we assume the model is already trained. We pass in $x^{\langle 1\rangle} = \vec{0}$ at the first time step, and have the network then sample one character at a time.

**Exercise**: Implement the `sample` function below to sample characters. You need to carry out 4 steps:

- **Step 1**: Pass the network the first "dummy" input $x^{\langle 1 \rangle} = \vec{0}$ (the vector of zeros). This is the default input before we've generated any characters. We also set $a^{\langle 0 \rangle} = \vec{0}$

- **Step 2**: Run one step of forward propagation to get $a^{\langle 1 \rangle}$ and $\hat{y}^{\langle 1 \rangle}$. Here are the equations:

$$ a^{\langle t+1 \rangle} = \tanh(W_{ax}  x^{\langle t \rangle } + W_{aa} a^{\langle t \rangle } + b)\tag{1}$$

$$ z^{\langle t + 1 \rangle } = W_{ya}  a^{\langle t + 1 \rangle } + b_y \tag{2}$$

$$ \hat{y}^{\langle t+1 \rangle } = softmax(z^{\langle t + 1 \rangle })\tag{3}$$

Note that $\hat{y}^{\langle t+1 \rangle }$ is a (softmax) probability vector (its entries are between 0 and 1 and sum to 1). $\hat{y}^{\langle t+1 \rangle}_i$ represents the probability that the character indexed by "i" is the next character.  We have provided a `softmax()` function that you can use.

- **Step 3**: Carry out sampling: Pick the next character's index according to the probability distribution specified by $\hat{y}^{\langle t+1 \rangle }$. This means that if $\hat{y}^{\langle t+1 \rangle }_i = 0.16$, you will pick the index "i" with 16% probability. To implement it, you can use [`np.random.choice`](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.random.choice.html).

Here is an example of how to use `np.random.choice()`:
```python
np.random.seed(0)
p = np.array([0.1, 0.0, 0.7, 0.2])
index = np.random.choice([0, 1, 2, 3], p = p.ravel())
```
This means that you will pick the `index` according to the distribution:
$P(index = 0) = 0.1, P(index = 1) = 0.0, P(index = 2) = 0.7, P(index = 3) = 0.2$.

- **Step 4**: The last step to implement in `sample()` is to overwrite the variable `x`, which currently stores $x^{\langle t \rangle }$, with the value of $x^{\langle t + 1 \rangle }$. You will represent $x^{\langle t + 1 \rangle }$ by creating a one-hot vector corresponding to the character you've chosen as your prediction. You will then forward propagate $x^{\langle t + 1 \rangle }$ in Step 1 and keep repeating the process until you get a "\n" character, indicating you've reached the end of the dinosaur name. _____no_output_____

# GRADED FUNCTION: sample

def sample(parameters, char_to_ix, seed):
    """
    Sample a sequence of characters according to a sequence of probability distributions output of the RNN

    Arguments:
    parameters -- python dictionary containing the parameters Waa, Wax, Wya, by, and b.
    char_to_ix -- python dictionary mapping each character to an index.
    seed -- used for grading purposes. Do not worry about it.

    Returns:
    indices -- a list of length n containing the indices of the sampled characters.
    """

    # Retrieve parameters and relevant shapes from "parameters" dictionary
    Waa, Wax, Wya, by, b = parameters['Waa'], parameters['Wax'], parameters['Wya'], parameters['by'], parameters['b']
    vocab_size = by.shape[0]
    n_a = Waa.shape[1]

    ### START CODE HERE ###
    # Step 1: Create the one-hot vector x for the first character (initializing the sequence generation). (≈1 line)
    x = np.zeros((vocab_size,1))   #don't know why this dimention:/
    # Step 1': Initialize a_prev as zeros (≈1 line)
    a_prev = np.zeros((n_a,1))     #don't know why 2nd dimention is 1:/

    # Create an empty list of indices, this is the list which will contain the list of indices of the characters to generate (≈1 line)
    indices = []

    # Idx is a flag to detect a newline character, we initialize it to -1
    idx = -1

    # Loop over time-steps t. At each time-step, sample a character from a probability distribution and append
    # its index to "indices". We'll stop if we reach 50 characters (which should be very unlikely with a well
    # trained model), which helps debugging and prevents entering an infinite loop.
    counter = 0
    newline_character = char_to_ix['\n']

    while (idx != newline_character and counter != 50):

        # Step 2: Forward propagate x using the equations (1), (2) and (3)
        a = np.tanh(np.dot(Wax,x)+ np.dot(Waa,a_prev) + b)
        z = np.dot(Wya,a) + by
        y = softmax(z)

        # for grading purposes
        np.random.seed(counter+seed)

        # Step 3: Sample the index of a character within the vocabulary from the probability distribution y
        #print(list(range(x.shape[0])))  #don't know why it's here
        idx = np.random.choice(list(range(x.shape[0])), p = y.ravel())

        # Append the index to "indices"
        indices.append(idx)

        # Step 4: Overwrite the input character as the one corresponding to the sampled index.
        x = np.zeros((vocab_size, 1))
        x[idx] = 1

        # Update "a_prev" to be "a"
        a_prev = a

        # for grading purposes
        seed += 1
        counter +=1

    ### END CODE HERE ###

    if (counter == 50):
        indices.append(char_to_ix['\n'])

    return indices_____no_output_____np.random.seed(2)
_, n_a = 20, 100
Wax, Waa, Wya = np.random.randn(n_a, vocab_size), np.random.randn(n_a, n_a), np.random.randn(vocab_size, n_a)
b, by = np.random.randn(n_a, 1), np.random.randn(vocab_size, 1)
parameters = {"Wax": Wax, "Waa": Waa, "Wya": Wya, "b": b, "by": by}

indices = sample(parameters, char_to_ix, 0)
print("Sampling:")
print("list of sampled indices:", indices)
print("list of sampled characters:", [ix_to_char[i] for i in indices])_____no_output_____

** Expected output:**

    **list of sampled indices:**

    [12, 17, 24, 14, 13, 9, 10, 22, 24, 6, 13, 11, 12, 6, 21, 15, 21, 14, 3, 2, 1, 21, 18, 24,

    7, 25, 6, 25, 18, 10, 16, 2, 3, 8, 15, 12, 11, 7, 1, 12, 10, 2, 7, 7, 11, 5, 6, 12, 25, 0, 0]

    **list of sampled characters:**

    ['l', 'q', 'x', 'n', 'm', 'i', 'j', 'v', 'x', 'f', 'm', 'k', 'l', 'f', 'u', 'o',

    'u', 'n', 'c', 'b', 'a', 'u', 'r', 'x', 'g', 'y', 'f', 'y', 'r', 'j', 'p', 'b', 'c', 'h', 'o',

    'l', 'k', 'g', 'a', 'l', 'j', 'b', 'g', 'g', 'k', 'e', 'f', 'l', 'y', '\n', '\n']

_____no_output_____## 3 - Building the language model

It is time to build the character-level language model for text generation.

### 3.1 - Gradient descent

In this section you will implement a function performing one step of stochastic gradient descent (with clipped gradients). You will go through the training examples one at a time, so the optimization algorithm will be stochastic gradient descent. As a reminder, here are the steps of a common optimization loop for an RNN:

- Forward propagate through the RNN to compute the loss
- Backward propagate through time to compute the gradients of the loss with respect to the parameters
- Clip the gradients if necessary
- Update your parameters using gradient descent

**Exercise**: Implement this optimization process (one step of stochastic gradient descent).

We provide you with the following functions:

```python
def rnn_forward(X, Y, a_prev, parameters):
    """ Performs the forward propagation through the RNN and computes the cross-entropy loss.
    It returns the loss' value as well as a "cache" storing values to be used in the backpropagation."""
    ....
    return loss, cache

def rnn_backward(X, Y, parameters, cache):
    """ Performs the backward propagation through time to compute the gradients of the loss with respect
    to the parameters. It returns also all the hidden states."""
    ...
    return gradients, a

def update_parameters(parameters, gradients, learning_rate):
    """ Updates parameters using the Gradient Descent Update Rule."""
    ...
    return parameters
```_____no_output_____

# GRADED FUNCTION: optimize

def optimize(X, Y, a_prev, parameters, learning_rate = 0.01):
    """
    Execute one step of the optimization to train the model.

    Arguments:
    X -- list of integers, where each integer is a number that maps to a character in the vocabulary.
    Y -- list of integers, exactly the same as X but shifted one index to the left.
    a_prev -- previous hidden state.
    parameters -- python dictionary containing:
                        Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)
                        Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)
                        Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
                        b --  Bias, numpy array of shape (n_a, 1)
                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
    learning_rate -- learning rate for the model.

    Returns:
    loss -- value  Noneof the loss function (cross-entropy)
    gradients -- python dictionary containing:
                        dWax -- Gradients of input-to-hidden weights, of shape (n_a, n_x)
                        dWaa -- Gradients of hidden-to-hidden weights, of shape (n_a, n_a)
                        dWya -- Gradients of hidden-to-output weights, of shape (n_y, n_a)
                        db -- Gradients of bias vector, of shape (n_a, 1)
                        dby -- Gradients of output bias vector, of shape (n_y, 1)
    a[len(X)-1] -- the last hidden state, of shape (n_a, 1)
    """

    ### START CODE HERE ###

    # Forward propagate through time (≈1 line)
    loss, cache = rnn_forward(X, Y, a_prev, parameters)

    # Backpropagate through time (≈1 line)
    gradients, a = rnn_backward(X, Y, parameters, cache)

    # Clip your gradients between -5 (min) and 5 (max) (≈1 line)
    gradients = clip(gradients,5)   #why isn't np.clip used

    # Update parameters (≈1 line)
    parameters = update_parameters(parameters, gradients, learning_rate)

    ### END CODE HERE ###

    return loss, gradients, a[len(X)-1]_____no_output_____np.random.seed(1)
vocab_size, n_a = 27, 100
a_prev = np.random.randn(n_a, 1)
Wax, Waa, Wya = np.random.randn(n_a, vocab_size), np.random.randn(n_a, n_a), np.random.randn(vocab_size, n_a)
b, by = np.random.randn(n_a, 1), np.random.randn(vocab_size, 1)
parameters = {"Wax": Wax, "Waa": Waa, "Wya": Wya, "b": b, "by": by}
X = [12,3,5,11,22,3]
Y = [4,14,11,22,25, 26]

loss, gradients, a_last = optimize(X, Y, a_prev, parameters, learning_rate = 0.01)
print("Loss =", loss)
print("gradients[\"dWaa\"][1][2] =", gradients["dWaa"][1][2])
print("np.argmax(gradients[\"dWax\"]) =", np.argmax(gradients["dWax"]))
print("gradients[\"dWya\"][1][2] =", gradients["dWya"][1][2])
print("gradients[\"db\"][4] =", gradients["db"][4])
print("gradients[\"dby\"][1] =", gradients["dby"][1])
print("a_last[4] =", a_last[4])_____no_output_____

** Expected output:**

    **Loss **

    126.503975722

    **gradients["dWaa"][1][2]**

    0.194709315347

    **np.argmax(gradients["dWax"])**

     93

    **gradients["dWya"][1][2]**

     -0.007773876032

    **gradients["db"][4]**

     [-0.06809825]

    **gradients["dby"][1]**

    [ 0.01538192]

    **a_last[4]**

     [-1.]

_____no_output_____### 3.2 - Training the model _____no_output_____Given the dataset of dinosaur names, we use each line of the dataset (one name) as one training example. Every 100 steps of stochastic gradient descent, you will sample 10 randomly chosen names to see how the algorithm is doing. Remember to shuffle the dataset, so that stochastic gradient descent visits the examples in random order.

**Exercise**: Follow the instructions and implement `model()`. When `examples[index]` contains one dinosaur name (string), to create an example (X, Y), you can use this:
```python
        index = j % len(examples)
        X = [None] + [char_to_ix[ch] for ch in examples[index]]
        Y = X[1:] + [char_to_ix["\n"]]
```
Note that we use: `index= j % len(examples)`, where `j = 1....num_iterations`, to make sure that `examples[index]` is always a valid statement (`index` is smaller than `len(examples)`).
The first entry of `X` being `None` will be interpreted by `rnn_forward()` as setting $x^{\langle 0 \rangle} = \vec{0}$. Further, this ensures that `Y` is equal to `X` but shifted one step to the left, and with an additional "\n" appended to signify the end of the dinosaur name. _____no_output_____

# GRADED FUNCTION: model

def model(data, ix_to_char, char_to_ix, num_iterations = 35000, n_a = 50, dino_names = 7, vocab_size = 27):
    """
    Trains the model and generates dinosaur names.

    Arguments:
    data -- text corpus
    ix_to_char -- dictionary that maps the index to a character
    char_to_ix -- dictionary that maps a character to an index
    num_iterations -- number of iterations to train the model for
    n_a -- number of units of the RNN cell
    dino_names -- number of dinosaur names you want to sample at each iteration.
    vocab_size -- number of unique characters found in the text, size of the vocabulary

    Returns:
    parameters -- learned parameters
    """

    # Retrieve n_x and n_y from vocab_size
    n_x, n_y = vocab_size, vocab_size

    # Initialize parameters
    parameters = initialize_parameters(n_a, n_x, n_y)

    # Initialize loss (this is required because we want to smooth our loss, don't worry about it)
    loss = get_initial_loss(vocab_size, dino_names)

    # Build list of all dinosaur names (training examples).
    with open("dinos.txt") as f:
        examples = f.readlines()
    examples = [x.lower().strip() for x in examples]

    # Shuffle list of all dinosaur names
    np.random.seed(0)
    np.random.shuffle(examples)

    # Initialize the hidden state of your LSTM
    a_prev = np.zeros((n_a, 1))

    # Optimization loop
    for j in range(num_iterations):

        ### START CODE HERE ###

        # Use the hint above to define one training example (X,Y) (≈ 2 lines)
        index = j % len(examples)
        X = [None] + [char_to_ix[k] for k in examples[index]]
        Y = X[1:]+ [char_to_ix["\n"]]

        # Perform one optimization step: Forward-prop -> Backward-prop -> Clip -> Update parameters
        # Choose a learning rate of 0.01
        curr_loss, gradients, a_prev = optimize(X, Y, a_prev, parameters, learning_rate = 0.01)

        ### END CODE HERE ###

        # Use a latency trick to keep the loss smooth. It happens here to accelerate the training.
        loss = smooth(loss, curr_loss)

        # Every 2000 Iteration, generate "n" characters thanks to sample() to check if the model is learning properly
        if j % 2000 == 0:

            print('Iteration: %d, Loss: %f' % (j, loss) + '\n')

            # The number of dinosaur names to print
            seed = 0
            for name in range(dino_names):

                # Sample indices and print them
                sampled_indices = sample(parameters, char_to_ix, seed)
                print_sample(sampled_indices, ix_to_char)

                seed += 1  # To get the same result for grading purposed, increment the seed by one.

            print('\n')

    return parameters_____no_output_____

Run the following cell, you should observe your model outputting random-looking characters at the first iteration. After a few thousand iterations, your model should learn to generate reasonable-looking names. _____no_output_____

parameters = model(data, ix_to_char, char_to_ix)_____no_output_____

## Conclusion

You can see that your algorithm has started to generate plausible dinosaur names towards the end of the training. At first, it was generating random characters, but towards the end you could see dinosaur names with cool endings. Feel free to run the algorithm even longer and play with hyperparameters to see if you can get even better results. Our implemetation generated some really cool names like `maconucon`, `marloralus` and `macingsersaurus`. Your model hopefully also learned that dinosaur names tend to end in `saurus`, `don`, `aura`, `tor`, etc.

If your model generates some non-cool names, don't blame the model entirely--not all actual dinosaur names sound cool. (For example, `dromaeosauroides` is an actual dinosaur name and is in the training set.) But this model should give you a set of candidates from which you can pick the coolest!

This assignment had used a relatively small dataset, so that you could train an RNN quickly on a CPU. Training a model of the english language requires a much bigger dataset, and usually needs much more computation, and could run for many hours on GPUs. We ran our dinosaur name for quite some time, and so far our favoriate name is the great, undefeatable, and fierce: Mangosaurus!

_____no_output_____## 4 - Writing like Shakespeare

The rest of this notebook is optional and is not graded, but we hope you'll do it anyway since it's quite fun and informative.

A similar (but more complicated) task is to generate Shakespeare poems. Instead of learning from a dataset of Dinosaur names you can use a collection of Shakespearian poems. Using LSTM cells, you can learn longer term dependencies that span many characters in the text--e.g., where a character appearing somewhere a sequence can influence what should be a different character much much later in ths sequence. These long term dependencies were less important with dinosaur names, since the names were quite short.

 Let's become poets!

We have implemented a Shakespeare poem generator with Keras. Run the following cell to load the required packages and models. This may take a few minutes. _____no_output_____

from __future__ import print_function
from keras.callbacks import LambdaCallback
from keras.models import Model, load_model, Sequential
from keras.layers import Dense, Activation, Dropout, Input, Masking
from keras.layers import LSTM
from keras.utils.data_utils import get_file
from keras.preprocessing.sequence import pad_sequences
from shakespeare_utils import *
import sys
import io_____no_output_____

To save you some time, we have already trained a model for ~1000 epochs on a collection of Shakespearian poems called [*"The Sonnets"*](shakespeare.txt). _____no_output_____Let's train the model for one more epoch. When it finishes training for an epoch---this will also take a few minutes---you can run `generate_output`, which will prompt asking you for an input (`<`40 characters). The poem will start with your sentence, and our RNN-Shakespeare will complete the rest of the poem for you! For example, try "Forsooth this maketh no sense " (don't enter the quotation marks). Depending on whether you include the space at the end, your results might also differ--try it both ways, and try other inputs as well.
_____no_output_____

print_callback = LambdaCallback(on_epoch_end=on_epoch_end)

model.fit(x, y, batch_size=128, epochs=1, callbacks=[print_callback])_____no_output_____# Run this cell to try with different inputs without having to re-train the model
generate_output()_____no_output_____

The RNN-Shakespeare model is very similar to the one you have built for dinosaur names. The only major differences are:
- LSTMs instead of the basic RNN to capture longer-range dependencies
- The model is a deeper, stacked LSTM model (2 layer)
- Using Keras instead of python to simplify the code

If you want to learn more, you can also check out the Keras Team's text generation implementation on GitHub: https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py.

Congratulations on finishing this notebook! _____no_output_____**References**:
- This exercise took inspiration from Andrej Karpathy's implementation: https://gist.github.com/karpathy/d4dee566867f8291f086. To learn more about text generation, also check out Karpathy's [blog post](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).
- For the Shakespearian poem generator, our implementation was based on the implementation of an LSTM text generator by the Keras team: https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py _____no_output_____


# Notebook from jasonrwang/pyNetLogo
Path: examples/SALib_sequential.ipynb

## Example 2: Sensitivity analysis on a NetLogo model with SALib

This notebook provides a more advanced example of interaction between NetLogo and a Python environment, using the SALib library (Herman & Usher, 2017; available through the pip package manager) to sample and analyze a suitable experimental design for a Sobol global sensitivity analysis. All files used in the example are available from the pyNetLogo repository at https://github.com/quaquel/pyNetLogo._____no_output_____

#Ensuring compliance of code with both python2 and python3

from __future__ import division, print_function
try:
    from itertools import izip as zip
except ImportError: # will be 3.x series
    pass_____no_output_____%matplotlib inline

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

import pyNetLogo

#Import the sampling and analysis modules for a Sobol variance-based sensitivity analysis
from SALib.sample import saltelli
from SALib.analyze import sobol_____no_output_____

SALib relies on a problem definition dictionary which contains the number of input parameters to sample, their names (which should here correspond to a NetLogo global variable), and the sampling bounds. Documentation for SALib can be found at https://salib.readthedocs.io/en/latest/._____no_output_____

problem = {
  'num_vars': 6,
  'names': ['random-seed',
            'grass-regrowth-time',
            'sheep-gain-from-food',
            'wolf-gain-from-food',
            'sheep-reproduce',
            'wolf-reproduce'],
  'bounds': [[1, 100000],
             [20., 40.],
             [2., 8.],
             [16., 32.],
             [2., 8.],
             [2., 8.]]
}_____no_output_____

We start by instantiating the wolf-sheep predation example model, specifying the _gui=False_ flag to run in headless mode._____no_output_____

import os

netlogo = pyNetLogo.NetLogoLink(gui=False)
model_file = os.path.join(netlogo.netlogo_home, 'models/Sample Models/Biology/Wolf Sheep Predation.nlogo')
netlogo.load_model(model_file)_____no_output_____

The SALib sampler will automatically generate an appropriate number of samples for Sobol analysis. To calculate first-order, second-order and total sensitivity indices, this gives a sample size of _n*(2p+2)_, where _p_ is the number of input parameters, and _n_ is a baseline sample size which should be large enough to stabilize the estimation of the indices. For this example, we use _n_ = 1000, for a total of 14000 experiments.

For more complex analyses, parallelizing the experiments can significantly improve performance. An additional notebook in the pyNetLogo repository demonstrates the use of the ipyparallel library; parallel processing for NetLogo models is also supported by the Exploratory Modeling Workbench (Kwakkel, 2017)._____no_output_____

n = 1000
param_values = saltelli.sample(problem, n, calc_second_order=True)_____no_output_____

The sampler generates an input array of shape (_n*(2p+2)_, _p_) with rows for each experiment and columns for each input parameter._____no_output_____

param_values.shape_____no_output_____

Assuming we are interested in the mean number of sheep and wolf agents over a timeframe of 100 ticks, we first create an empty dataframe to store the results._____no_output_____

results = pd.DataFrame(columns=['Avg. sheep', 'Avg. wolves'])_____no_output_____

We then simulate the model over the 14000 experiments, reading input parameters from the param_values array generated by SALib. The repeat_report command is used to track the outcomes of interest over time.

To later compare performance with the ipyparallel implementation of the analysis, we also keep track of the elapsed runtime._____no_output_____

import time

t0=time.time()

for run in range(param_values.shape[0]):

    #Set the input parameters
    for i, name in enumerate(problem['names']):
        if name == 'random-seed':
            #The NetLogo random seed requires a different syntax
            netlogo.command('random-seed {}'.format(param_values[run,i]))
        else:
            #Otherwise, assume the input parameters are global variables
            netlogo.command('set {0} {1}'.format(name, param_values[run,i]))

    netlogo.command('setup')
    #Run for 100 ticks and return the number of sheep and wolf agents at each time step
    counts = netlogo.repeat_report(['count sheep','count wolves'], 100)

    #For each run, save the mean value of the agent counts over time
    results.loc[run, 'Avg. sheep'] = counts['count sheep'].values.mean()
    results.loc[run, 'Avg. wolves'] = counts['count wolves'].values.mean()

elapsed=time.time()-t0 #Elapsed runtime in seconds_____no_output_____elapsed_____no_output_____

The "to_csv" dataframe method provides a simple way of saving the results to disk.

Pandas supports several more advanced storage options, such as serialization with msgpack, or hierarchical HDF5 storage._____no_output_____

results.to_csv('Sobol_sequential.csv')_____no_output_____results = pd.read_csv('Sobol_sequential.csv', header=0, index_col=0)_____no_output_____results.head(5)_____no_output_____

We can then proceed with the analysis, first using a histogram to visualize output distributions for each outcome:_____no_output_____

sns.set_style('white')
sns.set_context('talk')
fig, ax = plt.subplots(1,len(results.columns), sharey=True)

for i, n in enumerate(results.columns):
    ax[i].hist(results[n], 20)
    ax[i].set_xlabel(n)

ax[0].set_ylabel('Counts')

fig.set_size_inches(10,4)
fig.subplots_adjust(wspace=0.1)
#plt.savefig('JASSS figures/SA - Output distribution.pdf', bbox_inches='tight')
#plt.savefig('JASSS figures/SA - Output distribution.png', dpi=300, bbox_inches='tight')
plt.show()_____no_output_____

Bivariate scatter plots can be useful to visualize relationships between each input parameter and the outputs. Taking the outcome for the average sheep count as an example, we obtain the following, using the scipy library to calculate the Pearson correlation coefficient (r) for each parameter:_____no_output_____

%matplotlib
import scipy

nrow=2
ncol=3
fig, ax = plt.subplots(nrow, ncol, sharey=True)
sns.set_context('talk')
y = results['Avg. sheep']

for i, a in enumerate(ax.flatten()):
    x = param_values[:,i]
    sns.regplot(x, y, ax=a, ci=None, color='k',scatter_kws={'alpha':0.2, 's':4, 'color':'gray'})
    pearson = scipy.stats.pearsonr(x, y)
    a.annotate("r: {:6.3f}".format(pearson[0]), xy=(0.15, 0.85), xycoords='axes fraction',fontsize=13)
    if divmod(i,ncol)[1]>0:
        a.get_yaxis().set_visible(False)
    a.set_xlabel(problem['names'][i])
    a.set_ylim([0,1.1*np.max(y)])

fig.set_size_inches(9,9,forward=True)
fig.subplots_adjust(wspace=0.2, hspace=0.3)
#plt.savefig('JASSS figures/SA - Scatter.pdf', bbox_inches='tight')
#plt.savefig('JASSS figures/SA - Scatter.png', dpi=300, bbox_inches='tight')
plt.show()_____no_output_____

This indicates a positive relationship between the "sheep-gain-from-food" parameter and the mean sheep count, and negative relationships for the "wolf-gain-from-food" and "wolf-reproduce" parameters.

We can then use SALib to calculate first-order (S1), second-order (S2) and total (ST) Sobol indices, to estimate each input's contribution to output variance. By default, 95% confidence intervals are estimated for each index._____no_output_____

Si = sobol.analyze(problem, results['Avg. sheep'].values, calc_second_order=True, print_to_console=False)_____no_output_____

As a simple example, we first select and visualize the first-order and total indices for each input, converting the dictionary returned by SALib to a dataframe._____no_output_____

Si_filter = {k:Si[k] for k in ['ST','ST_conf','S1','S1_conf']}
Si_df = pd.DataFrame(Si_filter, index=problem['names'])_____no_output_____Si_df_____no_output_____sns.set_style('white')
fig, ax = plt.subplots(1)

indices = Si_df[['S1','ST']]
err = Si_df[['S1_conf','ST_conf']]

indices.plot.bar(yerr=err.values.T,ax=ax)
fig.set_size_inches(8,4)

#plt.savefig('JASSS figures/SA - Indices.pdf', bbox_inches='tight')
#plt.savefig('JASSS figures/SA - Indices.png', dpi=300, bbox_inches='tight')

plt.show()_____no_output_____

The "sheep-gain-from-food" parameter has the highest ST index, indicating that it contributes over 50% of output variance when accounting for interactions with other parameters. However, it can be noted that the confidence bounds are overly broad due to the small _n_ value used for sampling, so that a larger sample would be required for reliable results. For instance, the S1 index is estimated to be larger than ST for the "random-seed" parameter, which is an artifact of the small sample size.

We can use a more sophisticated visualization to include the second-order interactions between inputs._____no_output_____

import itertools
from math import pi

def normalize(x, xmin, xmax):
    return (x-xmin)/(xmax-xmin)

def plot_circles(ax, locs, names, max_s, stats, smax, smin, fc, ec, lw,
                 zorder):
    s = np.asarray([stats[name] for name in names])
    s = 0.01 + max_s * np.sqrt(normalize(s, smin, smax))

    fill = True
    for loc, name, si in zip(locs, names, s):
        if fc=='w':
            fill=False
        else:
            ec='none'

        x = np.cos(loc)
        y = np.sin(loc)

        circle = plt.Circle((x,y), radius=si, ec=ec, fc=fc, transform=ax.transData._b,
                            zorder=zorder, lw=lw, fill=True)
        ax.add_artist(circle)

def filter(sobol_indices, names, locs, criterion, threshold):
    if criterion in ['ST', 'S1', 'S2']:
        data = sobol_indices[criterion]
        data = np.abs(data)
        data = data.flatten() # flatten in case of S2
        # TODO:: remove nans

        filtered = ([(name, locs[i]) for i, name in enumerate(names) if
                     data[i]>threshold])
        filtered_names, filtered_locs = zip(*filtered)
    elif criterion in ['ST_conf', 'S1_conf', 'S2_conf']:
        raise NotImplementedError
    else:
        raise ValueError('unknown value for criterion')

    return filtered_names, filtered_locs

def plot_sobol_indices(sobol_indices, criterion='ST', threshold=0.01):
    '''plot sobol indices on a radial plot

    Parameters
    ----------
    sobol_indices : dict
                    the return from SAlib
    criterion : {'ST', 'S1', 'S2', 'ST_conf', 'S1_conf', 'S2_conf'}, optional
    threshold : float
                only visualize variables with criterion larger than cutoff

    '''
    max_linewidth_s2 = 15#25*1.8
    max_s_radius = 0.3

    # prepare data
    # use the absolute values of all the indices
    #sobol_indices = {key:np.abs(stats) for key, stats in sobol_indices.items()}

    # dataframe with ST and S1
    sobol_stats = {key:sobol_indices[key] for key in ['ST', 'S1']}
    sobol_stats = pd.DataFrame(sobol_stats, index=problem['names'])

    smax = sobol_stats.max().max()
    smin = sobol_stats.min().min()

    # dataframe with s2
    s2 = pd.DataFrame(sobol_indices['S2'], index=problem['names'],
                      columns=problem['names'])
    s2[s2<0.0]=0. #Set negative values to 0 (artifact from small sample sizes)
    s2max = s2.max().max()
    s2min = s2.min().min()

    names = problem['names']
    n = len(names)
    ticklocs = np.linspace(0, 2*pi, n+1)
    locs = ticklocs[0:-1]

    filtered_names, filtered_locs = filter(sobol_indices, names, locs,
                                           criterion, threshold)

    # setup figure
    fig = plt.figure()
    ax = fig.add_subplot(111, polar=True)
    ax.grid(False)
    ax.spines['polar'].set_visible(False)
    ax.set_xticks(ticklocs)

    ax.set_xticklabels(names)
    ax.set_yticklabels([])
    ax.set_ylim(ymax=1.4)
    legend(ax)

    # plot ST
    plot_circles(ax, filtered_locs, filtered_names, max_s_radius,
                 sobol_stats['ST'], smax, smin, 'w', 'k', 1, 9)

    # plot S1
    plot_circles(ax, filtered_locs, filtered_names, max_s_radius,
                 sobol_stats['S1'], smax, smin, 'k', 'k', 1, 10)

    # plot S2
    for name1, name2 in itertools.combinations(zip(filtered_names, filtered_locs), 2):
        name1, loc1 = name1
        name2, loc2 = name2

        weight = s2.ix[name1, name2]
        lw = 0.5+max_linewidth_s2*normalize(weight, s2min, s2max)
        ax.plot([loc1, loc2], [1,1], c='darkgray', lw=lw, zorder=1)

    return fig

from matplotlib.legend_handler import HandlerPatch
class HandlerCircle(HandlerPatch):
    def create_artists(self, legend, orig_handle,
                       xdescent, ydescent, width, height, fontsize, trans):
        center = 0.5 * width - 0.5 * xdescent, 0.5 * height - 0.5 * ydescent
        p = plt.Circle(xy=center, radius=orig_handle.radius)
        self.update_prop(p, orig_handle, legend)
        p.set_transform(trans)
        return [p]

def legend(ax):
    some_identifiers = [plt.Circle((0,0), radius=5, color='k', fill=False, lw=1),
                        plt.Circle((0,0), radius=5, color='k', fill=True),
                        plt.Line2D([0,0.5], [0,0.5], lw=8, color='darkgray')]
    ax.legend(some_identifiers, ['ST', 'S1', 'S2'],
              loc=(1,0.75), borderaxespad=0.1, mode='expand',
              handler_map={plt.Circle: HandlerCircle()})

sns.set_style('whitegrid')
fig = plot_sobol_indices(Si, criterion='ST', threshold=0.005)
fig.set_size_inches(7,7)
#plt.savefig('JASSS figures/Figure 8 - Interactions.pdf', bbox_inches='tight')
#plt.savefig('JASSS figures/Figure 8 - Interactions.png', dpi=300, bbox_inches='tight')
plt.show()_____no_output_____

In this case, the sheep-gain-from-food variable has strong interactions with the wolf-gain-from-food and sheep-reproduce inputs in particular. The size of the ST and S1 circles correspond to the normalized variable importances._____no_output_____Finally, the kill_workspace() function shuts down the NetLogo instance._____no_output_____

netlogo.kill_workspace()_____no_output_____


# Notebook from rrcook/fivestarreviews
Path: Load and Shuffle.ipynb

Optional check of timing after each cell_____no_output_____

%load_ext autotime_____no_output_____

Necessary imports_____no_output_____

import os
import gzip
from datetime import datetime
import numpy as np
import pandas as pd
import cudf
from pandas import Series, DataFrame
import sklearn.utilstime: 448 ms

Julian McAuley's code to unzip the gzip file and load it into a pandas dataframe._____no_output_____

def parse(path):
  g = gzip.open(path, 'rb')
  for l in g:
    yield eval(l)

def getDF(path):
  i = 0
  df = {}
  for d in parse(path):
    df[i] = d
    i += 1
  return pd.DataFrame.from_dict(df, orient='index')time: 883 µs

Set up the DATA_DIR environment variable to point to the review data. Used in all the code._____no_output_____

data_dir = os.environ['DATA_DIR'] + '/'time: 3.9 ms

Read the full gzipped data into a data frame._____no_output_____

jdf = getDF(data_dir + 'reviews_Books_5.json.gz') # 'json' dftime: 6min 39s
jdf_____no_output_____

We're keeping asin (amazon id), reviewText, and overall (the rating, 1.0 to 5.0).
There are enough reviews that we can take the smallest number of reviews, that for one star, and use that number
of the rest of the reviews to get a balance._____no_output_____

item_counts = jdf['overall'].value_counts()
item_counts_____no_output_____minimum_review_counts = item_counts[1]
minimum_review_counts_____no_output_____

Loading from gzip takes a long time and includes columns we don't need.
Save the relevant columns as a csv, to load and have around for later analysis._____no_output_____

jdf.to_csv(data_dir + 'reviews_1_to_5.csv', index=None, columns=['asin', 'reviewText', 'overall'])time: 1min 42s

We no longer need the dataframe from the json file._____no_output_____

del jdftime: 299 µs

Load the trimmed dataframe._____no_output_____

df = pd.read_csv(data_dir + 'reviews_1_to_5.csv')time: 51.7 s

Shuffle it._____no_output_____

df = sklearn.utils.shuffle(df)
df = df.reset_index(drop=True)time: 5.62 s

Use the same number of reviews from each category, which means the same number as the category with the least reviews._____no_output_____

df1 = df[df['overall'] == 1.0]
df1 = df1.reset_index(drop=True)
df2 = df[df['overall'] == 2.0]
df2 = df2.reset_index(drop=True)
df3 = df[df['overall'] == 3.0]
df3 = df3.reset_index(drop=True)
df4 = df[df['overall'] == 4.0]
df4 = df4.reset_index(drop=True)
df5 = df[df['overall'] == 5.0]
df5 = df5.reset_index(drop=True)time: 2.39 s
df2 = df2.truncate(before=0, after=minimum_review_counts - 1)
df3 = df3.truncate(before=0, after=minimum_review_counts - 1)
df4 = df4.truncate(before=0, after=minimum_review_counts - 1)
df5 = df5.truncate(before=0, after=minimum_review_counts - 1)time: 606 ms

Make a new dataframe from the right number of 1 to 5 reviews_____no_output_____

dfconcat = pd.concat([df1, df2, df3, df4, df5], ignore_index=True)time: 217 ms

Dataframe is currently all 1, 2, 3, 4, 5, so shuffle again_____no_output_____

dfconcat = sklearn.utils.shuffle(dfconcat)
dfconcat = dfconcat.reset_index(drop=True)time: 983 ms

It's ready to use, so write out the file._____no_output_____

dfconcat.to_csv(data_dir + 'reviews_1_to_5_shuffle.csv', index=None)time: 18.8 s


# Notebook from ajlee21/Batch_effects_simulation
Path: archive/scripts/experiment_1/test_svcca_and_transformations.ipynb

# SVCCA and affine transformations
We want to test that SVCCA is working as expected.  In other words, what is the SVCCA score when we compare two datasets that are 1) identical and 2) one is a contant scaled version of the other?_____no_output_____

%load_ext autoreload
%autoreload 2

import os
import pandas as pd
import numpy as np
import random
import glob
import warnings
warnings.filterwarnings("ignore")

from ggplot import *
from functions import cca_core

from numpy.random import seed
randomState = 123
seed(randomState)_____no_output_____# Parameters
analysis_name = 'experiment_0'
base_dir = os.path.abspath(os.path.join(os.getcwd(),"../.."))_____no_output_____# Load arguments
simulated_data_file = os.path.join(
    base_dir,
    "data",
    "simulated",
    analysis_name,
    "simulated_data.txt")_____no_output_____# Read in simulated gene expression data
simulated_data = pd.read_table(
    simulated_data_file,
    header=0,
    index_col=0,
    sep='\t')

simulated_data.head()_____no_output_____

## Similarity between input vs itself
We expect that the similarity of SVCCA comparing the input with itself should yeild an SVCCA score of 1.0_____no_output_____

%%time
# SVCCA
svcca_results = cca_core.get_cca_similarity(simulated_data.T,
                                      simulated_data.T,
                                      verbose=False)

print(np.mean(svcca_results["cca_coef1"]))0.8914584502967937
CPU times: user 26min 32s, sys: 5min 55s, total: 32min 27s
Wall time: 7min 7s

## Similarity between input vs scaled version of input¶
We expect that the similarity of SVCCA comparing the input with scaled version of itself to yield a high SVCCA score since this transformation is an affine transformation which SVCCA is supposed to be invariant to._____no_output_____

# Scale data by a constant
scaled_simulated_data = simulated_data.multiply(2)
scaled_simulated_data.head()_____no_output_____%%time
# SVCCA
svcca_results = cca_core.get_cca_similarity(simulated_data.T,
                                      scaled_simulated_data.T,
                                      verbose=False)

print(np.mean(svcca_results["cca_coef1"]))0.8914584502967937
CPU times: user 26min 17s, sys: 5min 51s, total: 32min 9s
Wall time: 6min 55s

## Similarity between input vs permuted input¶
We will use the similarity of between the input with permuted input as a negative control.  We would expect the SVCCA score to be fairly low for this comparison._____no_output_____

%%time
# Permute simulated data
shuffled_simulated_arr = []
num_samples = simulated_data.shape[0]

for i in range(num_samples):
    row = list(simulated_data.values[i])
    shuffled_simulated_row = random.sample(row, len(row))
    shuffled_simulated_arr.append(shuffled_simulated_row)CPU times: user 30.4 s, sys: 407 ms, total: 30.8 s
Wall time: 30.8 s
shuffled_simulated_data = pd.DataFrame(shuffled_simulated_arr, index=simulated_data.index, columns=simulated_data.columns)
shuffled_simulated_data.head()_____no_output_____%%time
# SVCCA
svcca_results = cca_core.get_cca_similarity(simulated_data.T,
                                      shuffled_simulated_data.T,
                                      verbose=False)

print(np.mean(svcca_results["cca_coef1"]))0.8764167125489775
CPU times: user 26min 10s, sys: 5min 48s, total: 31min 58s
Wall time: 6min 56s

## Toy_____no_output_____

df = pd.DataFrame({'A': range(5),'B': np.ones(5), 'C': [20,40,60,80,100], 'D': np.ones(5)*5})
df.head()_____no_output_____shuffled_arr = []

for i in range(len(df.values)):
    row = list(df.values[i])
    shuffled = random.sample(row, len(row))
    shuffled_arr.append(shuffled)
shuffled_df = pd.DataFrame(shuffled_arr, index=df.index, columns=df.columns)
shuffled_df.head()_____no_output_____


# Notebook from dwitvliet/python-causality-handbook
Path: causal-inference-for-the-brave-and-true/18-When-Prediction-Fails.ipynb

# When Prediction Fails

## When all you have is a Hammer...
Between 2015 and 2020, Machine Learning went through a massive surge. Its proven usefulness in the fields of computer vision and natural language understanding, coupled with an initial lack of professionals in the area, provided the perfect opportunity for a machine learning teaching industry. Figures like Andrew Ng and Sebastian Thrun managed to teach machine learning to the world at rock bottom prices. At the same time, on the software side, it became increasingly easier to fit a complex machine learning model (as you've already seen by the very few lines of code it took us to write an ML in the previous chapter). Tutorials about how to make intelligent systems sprung all over the internet. The cost of entry in ML plumend.

![img](./data/img/when-prediction-fails/ml-in-5.png)

Building ML became so simple that you didn't even need to know how to code very well (and I'm living evidence of that), nor the math behind the algorithms. In fact, you could build wonders with the following 5 lines of Python.

```python
X_train, y_train, X_test, y_test =  train_test_split(X, y)

## instantiate the machine learning model
model = MachineLearningModel()

## Fit the ML model
model.fit(X_train, y_train)

# Make predictions on unseen data
y_pred = model.predict(X_test)

# Evaluate the quality of predictions
print("Performance", metric(y_test, y_pred))
```

For the most part, this is an amazing thing! I'm all in for taking valuable content and making it available. However, there is also a dark side to all of this. This new wave of data scientists were trained mostly in predictive modeling, since that is what ML primarily focuses on solving. As a result, whenever those data scientists encountered a business problem, they tried to tackle it with, not surprisingly, predictive models. When they were indeed prediction problems, like the one we saw in the previous chapter, the data scientist usually succeeded and everyone got happy. However, there is an entire class of problems that are simply not solvable with prediction techniques. And when those appeared, the data scientists usually failed miserably. These are problems that are framed like "how much can I increase Y by changing X".

From my experience, this other type of problem is what management usually cares the most about. They often want to know how to increase sales, decrease cost or bring in more customers. Needless to say, they are not very happy when a data scientist comes up with an answer to how to predict sales instead of how to increase it. Sadly, when everything the data scientist knows is predictive models, this tends to happen a lot. As a boss of mine once told me: "when all you have is a hammer, everything starts to look like a thumb".

Like I've said, I'm all in for lowering the cost of knowledge, but the current Data Scientist curriculum has a huge gap. I think that my job here is to fill in that gap. Is to equip you with tools to solve this other class of problems, which are causal in nature.

What you are trying to do is estimate how something you can control (advertisement, price, customer service) affects or causes something you want to change, but can't control directly (sales, number of customers, PNL). But ,before showing you how to solve these problems, I want to show you what happens when you treat them like prediction tasks and try to solve them with the traditional ML toolkit. The reason for it is that data scientists often come to me and say "OK, but although tackling causal problems with prediction tools is not the best idea, it surely helps something, no? Imean, it couldn't hurt...". Well, as it turns out, it can. And you better understand this before you go on hammering your own thumb.

![img](./data/img/when-prediction-fails/horse-meme.png)
_____no_output_____

import pandas as pd
import numpy as np
from sklearn import ensemble
from sklearn.model_selection import train_test_split, cross_val_predict
from sklearn.ensemble import gradient_boosting
from sklearn.metrics import r2_score
import seaborn as sns
from matplotlib import pyplot as plt
from matplotlib import style
style.use("ggplot")

# helper functions for this notebook
from nb18 import ltv_with_coupons_____no_output_____

## Who Wants a Coupon?

To make matters more relatable, let's continue with the example we used in the previous chapter, but with a little twist to it. Before, we were trying to distinguish the profitable from the non profitable customers. We framed that as a prediction problem: predicting customer profitability. We could then build a machine learning model for this task and use it to choose who we would do business with: only the customers we predicted to be profitable. In other words, our goal was to separate the profitable from the non-profitable, which we could do with a predictive model.

Now, you have a new task. You suspect that giving coupons to new customers increases their engagement with your business and makes them more profitable in the long run. That is, they spend more and for a longer period. Your new assignment is to figure out how much the coupon value should be (zero included). Notice that, with coupons, you are essentially giving away money for people to spend on your business. For this reason, they enter as a cost in your book account. Notice that if the coupon value is too high, you will probably lose money, since customers will buy all they need using only the coupons. That's another way of saying that they will get your product for free. On the flip side, if coupon value is too low (or zero), you are not even giving coupons. This could be a valid answer, but it could also be that some discounts upfront, in the form of coupons, will be more profitable in the long run.

For reasons you will see later, we will use a data generating function instead of loading a static dataset. The function `ltv_with_coupons` generates transaction data for us. As you can see, they have the same format as the one we saw previously, with one row per customer, a column for the cost of acquisition and columns for the transactions between day 1 and 30. _____no_output_____

transactions, customer_features = ltv_with_coupons()

print(transactions.shape)
transactions.head()(10000, 32)

As for the other parts of the data, again, we have a customer identifier, the region the customer lives, the customer income and the customer age. In addition, we now have a variable that is `coupons`, which tells us how much we've given in coupons for that customer._____no_output_____

print(customer_features.shape)
customer_features.head()(10000, 5)

To process this data to a single dataframe, we will sum all the columns in the first table (that is, summing `CACQ` with the transactions).This will give us the `net_value` as it was computed in the previous chapter. After that, we will join in the features data and update the `net_value` to include the coupon cost._____no_output_____

def process_data(transactions, customer_data):

    profitable = (transactions[["customer_id"]]
                  .assign(net_value = transactions
                          .drop(columns="customer_id")
                          .sum(axis=1)))

    return (customer_data
            # join net_value and features
            .merge(profitable, on="customer_id")
            # include the coupons cost
            .assign(net_value = lambda d: d["net_value"] - d["coupons"]))

customer_features = process_data(transactions, customer_features)
customer_features.head()_____no_output_____

This processed data frame has all that we need. It has our target variable `net_value`, it has our customer features `region`, `income` and `age`, and it has the lever or treatment we want to optimise for: coupons. Just to begin understanding how coupons can increase `net_value`, let's look at how they were given away._____no_output_____

customer_features.groupby("coupons")["customer_id"].count()_____no_output_____

We can see that most of the coupons that were handed out had a value of 5 BRL, followed by the coupons with 10 BRL in value. We gave very few 15 BRL coupons or no coupons at all (zero value). This is indicative that they were **NOT** given randomly. To check that, let's see the correlation between the other variable and coupons._____no_output_____

customer_features.corr()[["coupons"]]_____no_output_____

That's interesting. It looks like the older the person, the higher the probability he or she will receive a coupon. This is some indication of bias in our data. We can also see a negative correlation between coupons and `net_value`: the more coupons we give, the smaller the net_value. This is hardly causal, since we already know that coupons were not randomly distributed. It could be that, say, older people spend less on our products and also receive higher coupon values, confounding the relationship between coupons and `net_value` to the point of making it negative.

The point being, we know there is bias. However, since there is already too much packed into this chapter, I'll ignore it for now (actually, I'll bypass the problem with an artifact you will see in just a moment). Just keep in mind it's something we will have to address sometime in the future.

At this point in the analysis, **if this was a prediction problem**, we would probably split the dataset into a training and a test set to build and evaluate some policies, respectively. But this is NOT a prediction problem. The final goal here is not to get a good prediction on customer profitability. Instead, it's to figure out the optimal coupon strategy. To evaluate this optimization, we would have to know how things would have played out if we have given different coupons than the ones that were given. This is the sort of counterfactual "what if'' question we've been studying under causality. Cross validation won't help us here because we simply can't observe counterfactuals. We can only see what happened for the coupons that were actually given, but we can't know what would have happened if customers received a different coupon value. Unless, we have simulated data!

If our data is simulated, we can generate the exact same data, only changing the coupon value parameters. This will allow us to see how `net_value` changes under different coupons strategies. We will then be able to calculate the treatment effect between different strategies \\(NetValue_{t=a} - NetValue_{t=b}\\). With the power of simulated data, understanding this chapter will be much easier. Oh yes, and this will also render the bias problem irrelevant, because we will observe the causal effect directly.

Nevertheless, always remember that this is a pedagogical artifact. In the real world, you don't have simulated data and you certainly can't see what would have happened under different treatment strategies. Individual causal effects remain hidden as they always have been. This poses an interesting problem. How can we evaluate our strategies for identifying causal effect if we can never see the actual causal effect? The real answer is very involved and so important that it deserves its own chapter. Rest assured that we will tackle it. For now, just enjoy the simplicity of simulated data. And speaking of simplicity...

## Simple Policy

As always, the first thing we should do whenever we encounter a new data problem is to ask ourselves "what is the simplest thing I can do that will already bring value?". For this specific case, the simplest thing is to look back on the data that we have and estimate the `net_value` for each coupon value. Then, check which coupon value is generating the highest `net_value` and give only that coupon value for every customer. _____no_output_____

sns.barplot(data=customer_features, x="coupons", y="net_value")
plt.title("Net Value by Coupon Value");_____no_output_____

Doing that analysis, we can see that, on average, we lose money when the coupon value is 0 or 15 and we gain money for coupons of 5 and 10 BLR. The highest average `net_income` appears when we have 5 BLR coupons, yielding us about 250 BLR in `net_value` per customer. Naturally then, the simplest thing we can try is to give everyone 5 BLR in coupons and see how that would play out. This completely disregards the possibility of bias but hey, we are talking simplicity here!

To do evaluate that policy, the function `ltv_with_coupons` accepts as argument a 10000 array that contains the desired coupon for each of the 10000 customers on our database. To create this array, we will generate an array of ones with `np.ones` the size of our `coupons` array (10000) and multiply it by 5. Then, we will pass this array to the `ltv_with_coupons`. This will generate a new dataset exactly like the one we had previously, but with every coupon value set to 5. We then process this data to get the net value under this newly proposed policy._____no_output_____

simple_policy = 5 * np.ones(customer_features["coupons"].shape)

transactions_simple_policy, customer_features_simple_policy = ltv_with_coupons(simple_policy)
customer_features_simple_policy = process_data(transactions_simple_policy, customer_features_simple_policy)

customer_features_simple_policy.head()_____no_output_____

Just as a sanity check, let's see if the features are indeed unchanged, considering the first few customers. Take the third one, for example (`customer_id` 2). For this customer, the region is 35, income is 2034 and the age is 33. If we scroll up a bit, we can see that it matches what we had before, so we are good here. Also, we can check that all the coupons are indeed 5 BRL. Finally, the `net_value` changes as expected. One reason for this is that the cost associated with coupons will change. For example, that customer had 15 BRL in coupons, but now it's 5. This would decrease the cost from 15 to 5 units. But notice that the `net_value` goes from -23 to 63, a 86 BRL increase in `net_value`. This is much larger than the 10 cost difference. Here giving less in the coupons made this particular customer much more profitable than he or she was before. Finally, to evaluate the policy, we can simply take the average `net_value`._____no_output_____

simple_policy_gain = customer_features_simple_policy["net_value"].mean()
simple_policy_gain_____no_output_____

As we can see, this simple policy is telling us we can get, on average, 253 BRL for each customer if we give them all a 5 BLR coupon. This is massive! But can we do better? What if we use our shiny machine learning hammer on this problem? Let's try this next.

## Policy With Model

To use ML, we will adapt what we did in the previous chapter. The idea is to build a ML model that predicts `net_value`, just like before, take those predictions and bins them into a defined number of bands. Then, we will partition the data into those bands. Essentially, we are splitting the customer by their predicted `net_value`. Customers that we think will generate roughly the same `net_value` will end up in the same bin or group. Finally, for each group, we will see which coupon value yields the maximum `net_value`. We are doing the same thing as in the simple policy, but now within the groups defined by a prediction band.

![img](./data/img/when-prediction-fails/partitions.png)

The intuition behind this is the following: we know that, on average, 5 BLR coupons performed better. However, it is possible that for some group of customers, another value is even better than 5 BRL. Maybe 5 BLR is the optimal strategy for most of the customers, but not for all of them.

![img](./data/img/when-prediction-fails/personalise.png)

If we can identify the ones where the optimal value is different, we can build a coupon strategy better than the simple one we did above.

This is what we call a personalisation problem. We can leverage personalisation when we have more than one strategy to choose from and at least one of them is not the overall best strategy, but it is the best in a subset of the targeted population. This definition is a bit convoluted, but the intuition is simple. If you have only one strategy, you are not personalising. You are doing the same thing for every customer. If you have more than one strategy, but one of them is better for every single customer, why will you personalise? You could just do that one best thing. You will only do personalisation if you have one strategy that works better at one subset of the population and another strategy that works best in another subset of the population.

But back to the example. The first thing we need is a function that fits our predictive model and also splits the predictions into the prediction bands. This function will return another function, a prediction function that will take a dataframe and add both predictions and band columns.
_____no_output_____

def model_bands(train_set, features, target, model_params, n_bands, seed=1):

    np.random.seed(seed)

    # train the ML model
    reg = ensemble.GradientBoostingRegressor(**model_params)
    reg.fit(train_set[features], train_set[target])

    # fit the bands
    bands = pd.qcut(reg.predict(train_set[features]), q=n_bands, retbins=True)[1]

    def predict(test_set):
        # make predictions with trained model
        predictions = reg.predict(test_set[features])

        # discretize predictions into bands.
        pred_bands = np.digitize(predictions, bands, right=False)
        return test_set.assign(predictions=predictions,
                               # cliping avoid creating new upper bands
                               pred_bands=np.clip(pred_bands, 1, n_bands))

    return predict_____no_output_____

To evaluate the quality of our predictions, we will split the dataset into a training and a testing set. Notice here that we are evaluating the quality of the prediction, NOT of the policy. This is just to see if our model is any good at doing what is supposed to do. _____no_output_____

train, test = train_test_split(customer_features, test_size=0.3, random_state=1)_____no_output_____

Now, let's train our model and make 10 bands with its predictions._____no_output_____

model_params = {'n_estimators': 150,
                'max_depth': 4,
                'min_samples_split': 10,
                'learning_rate': 0.01,
                'loss': 'ls'}

features = ["region", "income", "age"]
target = "net_value"

np.random.seed(1)
model = model_bands(train, features, target, model_params, n_bands=10)_____no_output_____

After training our model, we can use it to make predictions, passing it a dataframe. The result will also be a dataframe with 2 new columns: `predictions` and `pred_bands`._____no_output_____

model(train).head()_____no_output_____

To see the predictive power of our model, we can look at the \\(R^2\\) for both training and test sets._____no_output_____

print("Train Score:, ", r2_score(train["net_value"], model(train)["predictions"]))
print("Test Score:, ", r2_score(test["net_value"], model(test)["predictions"]))Train Score:,  0.5382953634651921
Test Score:,  0.504563847410434

Remember that this performance is only the predictive performance. What we really want to know is if this model can make us some money. Let's make a policy! The idea here is very similar to what we saw in the previous chapter. We will group the customers by model band. Then, for each type of customer (where type is defined by the bands) we'll see which decision - coupon value in our case - is the best one. To do so, we can group our data by prediction band and coupon value and plot the `net_value`._____no_output_____

plt.figure(figsize=(12,6))
sns.barplot(data=model(customer_features), x="pred_bands", y="net_value", hue="coupons")
plt.title("Net Value by Coupon Value");_____no_output_____

This plot is very interesting. Notice how the optimal decision changes across prediction bands. For instance, on bands like 1, 7 and 8, the best thing to do is to give 10 BLR in coupons. For bands like 3, 5 and 10, the best thing is 5 BRL in coupons. This means that this policy is very much like the simple one, except for the last band. This is evidence that personalisation might be possible, since the optimal decision changes across subpopulations.

We can code that policy with a couple of `if ... then ...` statements, but I'll show a more general approach that leverages dataframe operations.

![img](./data/img/when-prediction-fails/pandas-magic.png)

First, we will group our customers by band and coupon value and take the average `net_value` for each group, much like the plot above._____no_output_____

pred_bands = (model(customer_features)
              .groupby(["pred_bands", "coupons"])
              [["net_value"]].mean()
              .reset_index())

pred_bands.head(7)_____no_output_____

Then, we will group by band and take the `net_value` rank for each row. This will order the rows according to the average `net_value`, where 1 is the best `net_value` in that band._____no_output_____

pred_bands["max_net"] = (pred_bands
                         .groupby(['pred_bands'])
                         [["net_value"]]
                         .rank(ascending=False))

pred_bands.head(7)_____no_output_____

For example, for band one, the best coupon strategy is 10 BRL. Next, we will keep only the greatest `net_value` per band._____no_output_____

best_coupons_per_band = pred_bands.query("max_net==1")[["pred_bands", "coupons"]]

best_coupons_per_band_____no_output_____

To build our policy, we will take that small table above and join it back on the original table using the band as the key. This will pair each row in the original dataset with what we think is optimal coupon value, according to this policy. Then, we sort the rows according to the `customer_id` so that we keep the same ordering we had previously. This is important for evaluation, since `ltv_with_coupons` takes as argument the coupon value in the order of the original dataframe._____no_output_____

coupons_per_id = (model(customer_features)
                 .drop(columns=["coupons"])
                 .merge(best_coupons_per_band, on="pred_bands")
                 [["customer_id", "coupons"]]
                 .sort_values('customer_id'))

coupons_per_id.head()_____no_output_____

Finally, to evaluate the policy, we pass the `coupons` column as the coupon array to the `ltv_with_coupons` function. This will regenerate the data, now assuming the coupons were given as we defined by this policy. _____no_output_____

transactions_policy_w_model, customer_features_policy_w_model = ltv_with_coupons(
    coupons_per_id[["coupons"]].values.flatten()
)

customer_features_policy_w_model = process_data(transactions_policy_w_model, customer_features_policy_w_model)

customer_features_policy_w_model.head()_____no_output_____

Just doing a sanity check again, we can see that the third customer is still the one with region 35, income 2034 and age 33. It also has a coupon value of 5 BRL, just like we've established by our policy.

To check how much money this policy is making us, we can compute the average `net_value` mean for this new dataset._____no_output_____

policy_w_model_gain = customer_features_policy_w_model["net_value"].mean()
policy_w_model_gain_____no_output_____

Not bad! We can expect to get about 230 BRL per customer with this model policy. But wait a second! You remember how much we were making with the simple policy? Let's compare both of them side by side._____no_output_____

plt.figure(figsize=(10,6))
sns.histplot(data=customer_features_policy_w_model, bins=40,
             x="net_value", label="Policy W/ Model", color="C0")
sns.histplot(data=customer_features_simple_policy, bins=40,
             x="net_value", label="Simple Policy", color="C1")
plt.legend()
plt.title(f"Simple Policy Gain: {simple_policy_gain}; Policy w/ Model Gain: {policy_w_model_gain};");_____no_output_____

Here is where most Data Scientists fall off their chair. The policy with the model has an average `net_income` which is 20 BRL worse than a very simple policy. How can a model that is good at predicting `net_value` not be good for a strategy that aims to maximize `net_value`? Surely, there must be some bug in the code. You are clearly mistaken. This cannot be! Well, as it turns out, there is a perfectly reasonable and simple explanation for it. However, the answer to this question is so important that I think it's worth going a bit deeper into it.

## Hammering Your Thumb with Predictions

The short answer lies in understanding what we want with this policy, namely, to optimize `net_value` by playing with the coupon values. If we were to put it in an image, it's not crazy to think that `net_value` will have a quadratic shape on coupons: as we increase the coupon value `net_value` first increases, then it reaches a maximum point. After that, any additional coupon value will cost more than the value it brings.

![img](./data/img/when-prediction-fails/opt-deriv.png)

Finding the optimal coupon value is then equivalent to finding the maximum of the `net_value` function. We can do this by differentiating the function and setting it to zero (second plot). Economists might recognize this as a pricing problem.  The way they (we) would tackle this problem would be to assume a functional form for `net_value`, differentiate it and optimize it.

There are some great merits to this approach, but I feel is not that general and it requires a great deal of hypothesizing. Sadly, real world data doesn't come with an underlying function we can differentiate, so guesswork is often involved here. A more practical approach (and perhaps less elegant) is to test multiple coupon values and see which one yields the best `net_value`. This is exactly what our simple policy does. It looks at what happened in the past and repeats what the treatment that was shown to be the most promising one.

Contrast this to what the model based policy does. First, the model based policy fits a machine learning model to predict `net_value`. Then, it partitions the customer space according to the predictions. If the model is good, this is approximately equal to partitioning the space by `net_value` itself, just like in the following plot.

![img](./data/img/when-prediction-fails/model-opt.png)

The better the prediction, the more this partitioning of the space approaches partitioning on the target variable, `net_income`. Pay very close attention to what happens when you do that. Essentially, you are splitting the customer into sets where `net_value`, the thing you've predicted, doesn't change! And that makes total sense from a predictive standpoint. If your model is good at predicting, groups of points that have the same prediction will also have the same `net_value`.

So far, so good, but look at what this does to the perceived function of `net_value` on coupons (green lines). It flattens them to have no slope at all. From the predictive point of view, this is awesome. It means that your model has captured all the variation in `net_income`. However, from the policy perspective, this is terrible, because there is no variance left in `net_income` for us to see how it would change given different coupon values. Without this variation in `net_income`, it would look like changing the coupon values has no effect on `net_income` at all, leaving us no room for optimization. By the way, this is a general phenomenon that has nothing to do with the specific quadratic shape we are using here. I'm only using an example to make things more concrete.

![img](./data/img/when-prediction-fails/flat-curves.png)

To summarize it, whenever we want to optimise some \\(Y\\) variable using some \\(T\\) variable, predicting \\(Y\\) will not only not help, it will hurt our policy, since data partitions defined by the prediction will have limited \\(Y\\) variance, hindering our capacity to estimate how \\(T\\) changes \\(Y\\), that is, the elasticity \\(\frac{\delta Y}{\delta T} \\). This is the most important paragraph of this chapter. All of it is summarized here, so reread it if you didn't understood at first.

The key to fixing this mistake lies in adjusting our objective to what we really want. Instead of estimating \\(Y\\) out of \\(X\\), which is what prediction does, we need to estimate \\(\frac{\delta Y}{\delta T} \\) out of \\(X\\). Easier said than done. As you might have guessed already, this precisely what causal inference is all about. And, as it is natural of causal problems, we can't observe our quantity of interest \\(\frac{\delta Y}{\delta T} \\). You simply cannot observe how `net_income` would change if we changed the coupon value because we only observe one instance of a coupon per customer. We can never know what would have happened if some different coupon value had taken place (unless we use simulated data, of course. But that's only useful for teaching purposes).

This characteristic of causal problems leads to further questions: how can I know if my model is any good if I can't see what it is supposedly estimating? How can I validate a model like that? Besides, what do we do when data is not random? How can we estimate the best policy under biased data? Those are fair questions and we shall answer them in time. Meanwhile, keep in mind that, as we change our focus from estimating \\(Y\\) to estimating \\(\frac{\delta Y}{\delta T}\\), lots of things will have to change accordingly. The traditional ML toolkit will need some adaptation.

## When Might Prediction Helps

All this mess with machine learning hindering our ability to infer causal effects comes from mistaking the real goal. It comes from estimating \\(Y\\) instead of \\(\frac{\delta Y}{\delta T} \\). But sometimes, you might actually get away when using a prediction model to achieve a causal inference goal. But for this to happen, \\(Y\\) and \\(\frac{\delta Y}{\delta T} \\) must be somehow correlated. For example, consider the cases on the image below.

![img](./data/img/when-prediction-fails/waiting-time.png)

The first one is the problem we've seen before of understanding how coupons impact profitability. First, as we increase coupon value, profitability increases. If they go up together, we can see that they are positively correlated. Moreover, in the derivative plot, at the beginning, as we increase the coupon value, the elasticity \\(\frac{\delta Y}{\delta T} \\) decreases. If we put both of them together, in the first region of the plot, as net value increases, elasticity will decrease. This means that they are negatively correlated.

But that's only at the beginning. If we keep increasing the coupon value, profitability will decrease and elasticity profitability will also decrease. This means that \\(\frac{\delta Y}{\delta T} \\) and \\(Y\\) are now positively correlated. So, while outcome and elasticity are negatively correlated at the beginning, this correlation reverses as we increase coupon values. This means that a prediction model won't help us here because there isn't a direct relationship between outcome and elasticity of the outcome.

Another way of thinking about this is that prediction models make slices of the data on the \\(Y\\) axis. If we do those sorts of slices, we will mix units that are both in the positive elasticity region and in the negative elasticity region.

![img](./data/img/when-prediction-fails/slice-1.png)

Now, consider a second case where we want to see how waiting in line to get answered by customers services impact customer satisfaction. In this case, we can see that customer satisfaction drops pretty fast in the first few minutes of waiting time. Customers get really pissed off when they go from not having to wait much to having to wait just a little bit. However, as waiting time increases, customer satisfaction is already so low it doesn't drop much afterwards. It sorts of saturates at a lower level.

This case is interesting because the relationship between \\(\frac{\delta Y}{\delta T} \\) and \\(Y\\) doesn't change much. It is always negatively correlated. As waiting time increases, satisfaction decreases and satisfaction elasticity increases. Or, as \\(T\\) increases, \\(\frac{\delta Y}{\delta T} \\) also increases and  \\(Y\\) decreases. In this case, a prediction model can be useful. The reason being that, now, if we slice on the \\(Y\\) axis, we will group units that have similar elasticities.

![img](./data/img/when-prediction-fails/slice-2.png)

More generally, whenever we have these sorts of functional forms where elasticity changes ruthly in the same direction as the outcome, you might get away with prediction models. Now, be careful here. This doesn't mean they are your best bet nor that you should use them for causal problems. I'm simply stating this here so that when you come across a prediction model that works for a causal goal, you can understand what is happening. _____no_output_____## A Graphical Explanation

The fact that prediction models can be harmful for personalisation is so counterintuitive and strange that I feel I must tackle the problem from multiple perspectives. The following graphical explanation is thus a way of understanding what is happening from a different angle.

Consider the causal leftmost graph on the image below. It's a typical confounding graph, where we have features \\(X\\) causing both the outcome \\(Y\\) and the treatment \\(T\\). What happens if you build a predictive model and use it to segment the units by its predictions?

![img](./data/img/when-prediction-fails/graph-1.png)

You end up getting something like the graphs on the left (technically, they are not causal anymore, because the model obviously doesn't cause the outcome. It just learns the mapping function from features to outcomes, but that doesn't invalidate the points I'm about to make). Segmenting on this model is equivalent to conditioning on the model node. You are holding \\(M(T, X)\\) fixed on each segment. If you do that and try to learn the relationship between \\(T\\) and \\(Y\\) while conditioning on \\(M(T, X)\\), you will end up with the situation we've discussed in part I, about bad controls. And in case you don't remember, if you control for anything in the path between the treatment and the outcome, you block a path that the treatment is using to affect the outcome. As a consequence, the perceived effect ends up looking smaller than the actual effect. So there you have if. We used another explanation to get to the exact same conclusion as before: segmenting units by a predictive model hinders our ability to identify the causal effect.

Now, if you are smart, you will probably object with the following idea. What if we simply remove \\(T\\) from the model? That's a really good argument. In fact, it's so good it took me years to properly articulate why just trying to fool your model like that is probably not a smart thing to do. In fact, if you look back on the example we've explored in this chapter, notice that I didn't even give the predictive model the `coupon` treatment. That didn't solve the problem that the predictive model partitioning was squashing our measured elasticity to a horizontal line. The answer to why lies in the following image.

![img](./data/img/when-prediction-fails/graph-2.png)

Excluding the treatment \\(T\\) from your model \\(M(X)\\) is not enough to make the model ignore the relationship between \\(T\\) and \\(Y\\). Because \\(X\\) causes \\(T\\), the model can indirectly learn \\(T\\) through \\(X\\). Why will it do that? Because if \\(T\\) causes \\(Y\\) and \\(M\\) wants to predict \\(Y\\) as much as possible,  it will forcibly explore the pattern in \\(X\\) that links \\(T\\) to \\(Y\\). In other words, the information of how the treatment impacts the outcomes flows backwards to our model through the features. To give an example, consider that you only give discounts (\\(T\\)) to customers that live in a certain area of town (\\(X\\)). For the sake of the argument, suppose that the area doesn't cause sales (\\(Y\\)) in any way. Even if that is the case, the city area still has predictive power because it causes \\(T\\), which in turns causes \\(Y\\). Hence, the model will explore that fact in order to predict \\(Y\\).

To be fair to this argument, you actually can fool your model a little bit by not giving it \\(T\\). To see that, I recommend you rerun the code on our examples but this time include `coupons` in the list of features to the predictive model. You will see that the Model Policy will perform even worse. This means that NOT giving \\(T\\) to our model is doing some good, although not much.

Finally, there is one way out of this, which is when \\(T\\) is random. In that very particular case, a predictive model can actually help you with causal inference. But first, let's understand why it doesn't hurt. When \\(T\\) is random, this excludes any confounding that comes from \\(X\\). In this case, \\(X\\) have no information about how \\(T\\) comes to be.

![img](./data/img/when-prediction-fails/graph-3.png)

When that happens, even if you control for \\(M(X)\\), you are still allowing all the effect from \\(T\\) to flow onto \\(Y\\). In other words, even if you look inside segments where \\(M(X)\\) is fixed, \\(Y\\) will still vary due to \\(T\\). \\(M(X)\\) can no longer control that variability because it has no way of learning \\(T\\) anymore. Not only that, this sort of condition will actually help you figure out the Average Treatment Effect. We also saw that on Part I. If we control for variables that are good predictors of \\(Y\\) but that don't cause \\(T\\), we will reduce the variance of our causal estimates. The intuitive reason is that a lot of the variation on \\(Y\\) that is due to \\(X\\) is removed by conditioning on the model. Since \\(X\\) is then explained away, the reason why \\(Y\\) is still changing must be mostly due to \\(T\\) (or other unmeasured things, but you get the point that it helps).

Again, this isn't an argument in favor of using predictive models when the goal is causal inference. In fact, segments defined by a predictive model will help you identify the ATE, but that doesn't mean each partition will have a different ATE (or elasticity). Remember that not only do you want to estimate the elasticity, but you also want to find segments where it is higher or lower than average so that you can personalise the treatment. _____no_output_____## Key Ideas

We saw how there is a whole range of problems that are not solvable with traditional, predictive Machine Learning. Those problems are optimization ones, with a causal component, usually framed as "how should I set this thing I can control, T (price, number of phone calls, discount, ...), in order to optimize this metric, Y (profits, customer satisfaction, conversion, costs…)". When that is the case, predictive models can do more harm than good.

To prove this point, we walked through an example where we needed to find the best coupon value. The first thing that we tried was to look at past data and see which coupon value led to the highest revenue. Then, we took that supposedly optimal value and gave it to every customer. This approach answers what is the best coupon value (treatment, in causal terms), on average. That is, if we were to pick only one fixed coupon value, what should it be. But we don't have such a restriction. If it turns out to be the case that some customers respond better to lower valued coupons and some respond better to higher ones, we could personalise the coupon for each customer. To do so, we tried building a predictive ML model, a model that predicted the value each customer would bring. Then, we tried to personalise the coupon value based on the model's predictions. We grouped the customers according to the predictions and for each group, we found which coupon value was the best one. Then, we tried using that learned value as a coupon strategy. However, we found that it performed worse than the simple strategy of giving the same coupon value for everyone. In other words, our ML model failed to separate customers that responded better or worse to coupon value.

Notice that the problem we are trying to solve here is one of **personalisation**. In our case, it was trying to figure out which coupon value is best for each customer. More generally, it is framed as "which is the best decision \\(T\\) I can take for each customer \\(i\\) so that I can maximise \\(Y_i\\) across all customers". This is not an easy problem to solve. But we can solve it with the help of causal inference.

There is one important thing I want you to take out of this chapter. First, notice that causal inference as we saw it in part one helps you design policies like the simple one we've built. With them, we are trying to estimate the average causal effect for the treatment and find the treatment that works best on average. Now, we are getting more ambitious. We don't want the best treatment on average, we want the best treatment for each individual customer. That's what personalisation is all about and it relies heavily on causal inference methods.

## References

The things I've written here are mostly stuff from my head. I've learned them through experience. This means there isn't a direct reference I can point you to. It also means that the things I wrote here have **not** passed the academic scrutiny that good science often goes through. Instead, notice how I'm talking about things that work in practice, but I don't spend too much time explaining why that is the case. It's a sort of science from the streets, if you will. However, I am putting this up for public scrutiny, so, by all means, if you find something preposterous, open an issue and I'll address it to the best of my efforts.

For this chapter, there isn't much I can point to. There is only this one great article that formulates the problem of personalisation that we are trying to solve with causal inference: *Recursive Partitioning for Heterogeneous Causal Effects, Susan Athey and Guido Imbens, 2016*. Aslo, the first image is from a YouTube video from Siraj Raval.

## Contribute

Causal Inference for the Brave and True is an open-source material on causal inference, the statistics of science. It uses only free software, based in Python. Its goal is to be accessible monetarily and intellectually.
If you found this book valuable and you want to support it, please go to [Patreon](https://www.patreon.com/causal_inference_for_the_brave_and_true). If you are not ready to contribute financially, you can also help by fixing typos, suggesting edits or giving feedback on passages you didn't understand. Just go to the book's repository and [open an issue](https://github.com/matheusfacure/python-causality-handbook/issues). Finally, if you liked this content, please share it with others who might find it useful and give it a [star on GitHub](https://github.com/matheusfacure/python-causality-handbook/stargazers)._____no_output_____