# Notebook from adamvig96/John-Rust-1987-Python
Path: Data Generation and Likelihood Fit.ipynb

# Recreation of Rust 1987 - Quentin ANDRE_____no_output_____# I. Setting-up the environment

## 1°) Importation of the modules_____no_output_____

import pandas as pd
import numpy as np
import random as rnd
import scipy.stats as stats
import scipy.optimize as opt
import json as json
import matplotlib as mpl
from math import exp
from matplotlib import pyplot as plt
from IPython.core.pylabtools import figsize
from IPython.display import display
from IPython.core.display import HTML
rnd.seed(2)
import warnings
warnings.filterwarnings('ignore')_____no_output_____

# II. Notations and definitions

## 1°) Number of states and state transitions

In this paper, the agent observes each bus at a given time $t$, and makes a decision regarding the replacement of the engine or its maintenance as a function of the observed state of the engine (i.e. its mileage).

As in the original paper, we discretize the number of states into intervals of length 5000, and define a transition density which governs the evolution of mileage from one time period to another:

$$ p(x_{t+1} | x_{t} , i_{t} , \theta_{3})  = \left\{
  \begin{array}{l l}
    g(x_{t+1} - x_{t}, \theta_{3})   & \quad \text{if } i_t = 0\\
    g(x_{t+1} - 0, \theta_{3}) & \quad \text{if } i_t = 1
  \end{array} \right.$$

Here, the function g is defined by a multinomial distribution on the set {0, 1, 2}. In other words, the probability that a bus' mileage will increase by a value x between now and the next maintenance decision is:
* $\theta_{31}$ if $x \in [0, 5000[$
* $\theta_{32}$ if $x \in [5000, 10,000[$
* $\theta_{33} = 1 - (\theta_{31} + \theta_{32})$ if $x \in [10,000, +\infty[$

## 2°) Decision criteria

The utility function of the decision maker for a single time period is:

$$ u(x_{t}, i , \theta_{1})  = \left\{
  \begin{array}{l l}
    \qquad \enspace -c(x_t, \theta_1) + \epsilon_t(0)& \quad \text{if } i = 0\\
    -RC -c(0, \theta_1) + \epsilon_t(1) & \quad \text{if } i = 1
  \end{array} \right. \quad \text{(Errors are I.I.D. standard Gumbel)}$$

The agent will chose the decision which maximizes its utility. If the agent is forward looking (i.e. if $\beta \neq 0$), he does not only maximize his utility in the present time period, but also his future utility discounted by $\beta$.

Since $\beta$ cannot be estimated without knowing the utility function, we will later set $\beta$ to its true value.

## 3°) Cost function

A functional form has to be imposed on the cost function $c(x_t)$.

In this recreation, we assume that the replacement cost is a constant $\text{RC}$, and allow the maintenance cost $\text{MC}(s, \theta_1)$ to be a linear, exponential or logarithmic function of the state of the bus scaled by a parameter $\theta_1$_____no_output_____# III. Data-generating process

## 1°) Bus Mileage

### A. Introduction

The generation of a fake dataset can be done in two different ways:
* Generating monthly mileage increase from a continuous distribution, discretize the mileage, and recover the state transition parameters ($\theta_3$) using a maximum-likelihood estimation (as done in the paper).
* Set the vector $\theta_3$ to an arbitrary value, and generate discrete states transitions with corresponding probabilities.

The advantage of the first method is that this is closer to what is actually done in the paper: indeed, the focal goal of the paper is to recover RC and the cost scaling vector of parameters $\theta_1$.

If the decision of the agent is based on the discretized state (and not on the actual mileage), then this approach is not noisier than the second one.

### B. Mileage Transition Definition
For this reason, we will adopt this first approach, and assumes that the increase in mileage follows a truncated normal distribution bounded at [0, 15000]:

$$
(M_{t+1} - M_{t}) \sim \mathcal{N}_{Trunc}(6000, 4000)
$$

The parameters are arbitrarily chosen to ensure that the state transitions probabilities are plausible:_____no_output_____

lower, upper = 0, 15000
mu, sigma = 6000, 4000
mileage_dist = stats.truncnorm((lower - mu) / sigma, (upper - mu) / sigma, loc=mu, scale=sigma)_____no_output_____

Computing the state transitions probabilities:_____no_output_____

p_x0 = mileage_dist.cdf(5000)
p_x1 = mileage_dist.cdf(10000) - p_x0
p_x2 = 1 - p_x1 - p_x0
p = (p_x0, p_x1, p_x2)_____no_output_____

Plotting the state transitions probabilities:_____no_output_____

x = np.linspace(0,150001,150001)
y = mileage_dist.pdf(x)
plt.plot(x, y)
plt.title("State transitions probabilities", fontsize=25)
plt.xlabel("Mileage")
plt.ylabel("Probabilies")
plt.text(750, 0.00001, "$P(x_t = 0) = {:.2f}$".format(p_x0), fontsize=20, weight="bold")
plt.text(5750, 0.00001, "$P(x_t = 1) = {:.2f}$".format(p_x1), fontsize=20, weight="bold")
plt.text(10500, 0.00001, "$P(x_t = 2) =  {:.2f}$".format(p_x2), fontsize=20, weight="bold")
plt.fill_between(x, y, where=x>=0, facecolor='#F6E3CE', interpolate=True)
plt.fill_between(x, y, where=x>=5000, facecolor='#CEF6CE', interpolate=True)
plt.fill_between(x, y, where=x>=10000, facecolor='#CECEF6', interpolate=True)
plt.grid(False)
plt.xlim(0, 15000);_____no_output_____

Between each inspection, each bus has:
* A probability $P(x_t = 0) =  0.36$ of keeping the same mileage state (between 0 and 5,000 miles traveled)
* A probability $P(x_t = 1) =  0.48$ of increasing its mileage state by 1 (between 5,000 and 10,000 miles traveled)
* A probability $P(x_t = 2) =  0.16$ of increasing its mileage state by 2 (between 10,000 and 15,000 miles traveled) _____no_output_____## 2°) Agent's Decisions

### A. Initialization of the variables

The replacement cost, the maintenance cost function parameters, and the discount rate are defined in this section. We set up the discount rate to be lower than in the paper to speed up the convergence of the contraction mapping algorithm._____no_output_____

rc = 20
theta1_1 = 0.5
theta1_2 = 0.01
beta = 0.75_____no_output_____

### B. Definition of the cost function_____no_output_____

def myopic_costs(S, MF, params, p):
    """
    This function computes the myopic expected cost associated with each decision for each state,
    and returns an array of state/decision costs.

    Takes:
        * An integer S, describing the possible states of the bus
        * A maintenance cost function MF, which takes a vector of parameters and a `state' argument
        * A vector params, to be supplied to the maintenance cost function MF. The first element of
          the vector is the replacement cost rc.
        * A (3x1) vector p describing the state transitions probabilities

    Returns:
        * A (Nx2) array containing the maintenance and replacement costs for the N possible states of the bus
    """
    rc = params[0]
    maint_cost = [MF(s, params[1:]) for s in range(0, S)]
    repl_cost = [rc for state in range(0, S)]
    return np.vstack((maint_cost, repl_cost)).T_____no_output_____

We assume different possible forms for the maintenance cost function:

$$\text{A linear form: } \qquad \text{MF}(s, \theta) = \theta_1 s \\
\text{A quadratic form: } \qquad \text{MF}(s, \theta) = \theta_{1 1}  s + \theta_{1 2}s^2 \\
\text{An exponential form: } \qquad \text{MF}(s, \theta) = \theta_{1 1}\exp(\theta_{1 2}s) \\
\text{A logarithmic form: } \qquad \text{MF}(s, \theta) = \theta_{1 1}\log(\theta_{1 2}s)$$_____no_output_____

def lin_cost(s, params):
    try:
        theta1_1, = params
        return s*theta1_1
    except ValueError:
        raise ValueError
        print("Wrong number of parameters specified: expected 2, got {}".format(len(params)))_____no_output_____

### C. Definition of the choice probabilities, as a function of an array of costs_____no_output_____

def choice_prob(cost_array):
    """
    Returns the probability of each choice, conditional on an array of state/decision costs.
    """
    S = cost_array.shape[0]
    cost = cost_array - cost_array.min(1).reshape(S, -1)
    util = np.exp(-cost)
    pchoice = util/(np.sum(util, 1).reshape(S, -1))
    return pchoice_____no_output_____

### D. The Contraction Mapping Algorithm (generate the forward-looking choice probabilities)_____no_output_____

def contraction_mapping(
    S, p, MF, params, beta=0.75, threshold=1e-6, suppr_output=False
):
    """
    Compute the non-myopic expected value of the agent for each possible decision and each possible
    state of the bus.
    Iterate until the difference in the previously obtained expected value and the new expected value
    is smaller than a constant.
    Takes:
        * A finite number of states S
        * A state-transition probability vector p = [p(0), p(1), p(2), ..., p(k)] of length k < N
        * A maintenance cost function MF
        * A vector params for the cost function
        * A discount factor beta (optional)
        * A convergence threshold (optional)

    Returns:
        * The converged choice probabilities for the forward-looking and myopic agents for each state,
        conditional on `params'
    """
    achieved = True
    # Initialization of the state-transition matrices: describe the state-transition probabilities
    # if the maintenance cost is incurred, and regenerate the state to 0 if the replacement cost
    # is incurred.
    ST_mat = np.zeros((S, S))
    p = np.array(p)
    for i in range(S):
        for j, _p in enumerate(p):
            if i + j < S - 1:
                ST_mat[i + j][i] = _p

            elif i + j == S - 1:
                ST_mat[S - 1][i] = p[j:].sum()
            else:
                pass

    R_mat = np.vstack((np.ones((1, S)), np.zeros((S - 1, S))))

    # Initialization of the expected value (which is also the myopic
    # decision cost of the agent). Here, the forward-looking component is initialized at 0.
    k = 0
    EV = np.zeros((S, 2))
    EV_myopic = EV_new = myopic_costs(S, MF, params, p)
    # Contraction mapping loop
    while abs(EV_new - EV).max() > threshold:
        # Store the former expected value
        EV = EV_new
        # Obtained the probability of maintenance and replacement from the former expected value
        pchoice = choice_prob(EV)
        # Compute the expected cost for each state: Nx1 vector
        ecost = (pchoice * EV).sum(1)
        # Compute the two components of forward-looking utility: In case of maintenance,
        # utility of future states weighted by transition probabilities. In case of replacement,
        # the future utility is the utility of state 0
        futil_maint = np.dot(ecost, ST_mat)
        futil_repl = np.dot(ecost, R_mat)
        futil = np.vstack((futil_maint, futil_repl)).T
        # Future utility is discounted by beta, and added to the myopic cost.
        EV_new = EV_myopic + beta * futil
        k += 1
        if k == 1000:
            achieved = False
            break

    if not suppr_output:
        if achieved:
            print("Convergence achieved in {} iterations".format(k))
        else:
            print(
                "CM could not converge! Mean difference = {:.6f}".format(
                    (EV_new - EV).mean()
                )
            )

    return (choice_prob(EV_new), choice_prob(EV_myopic), EV_new)_____no_output_____

# IV. Generation of simulated data (using a linear cost specification)

## 1°) True coefficients

Here, we use the same coefficients as the one described earlier:
* Cost and discount coefficients:
    * $RC = 20$
    * $\theta_{11} = 0.5$
    * $\theta_{12} = 0.01$
    * $\beta = 0.75$
* State transition probabilities: $(\text{Mileage}_{t+1} - \text{Mileage}_{t}) \sim \mathcal{N}_{Trunc}(6000, 4000)$, such that:
    * $P(x_t = 0) =  0.36$
    * $P(x_t = 1) =  0.48$
    * $P(x_t = 2) =  0.16$

Finally, we assume that there are 70 discrete states for the bus' mileage (i.e. no bus has a mileage greater than 345,000 in the dataset)._____no_output_____## 2°) Choice probabilities, as a function of the bus' state:

Using the Contraction Mapping algorithm implemented earlier, we can obtain the probabibility of maintenance as a function of the bus' state for a forward-looking agent and a myopic agent._____no_output_____

params_lin = (rc, theta1_1)
p = (p_x0, p_x1, p_x2)
lin_forward, lin_myopic, lin_forward_util = contraction_mapping(
    S=70, p=p, MF=lin_cost, params=params_lin, beta=0.75
)
pchoice = lin_forward.T[0]Convergence achieved in 54 iterations

Plotting the choice probabilities of the Agents:_____no_output_____

plt.plot(lin_forward.T[0])
plt.plot(lin_myopic.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking", "Myopic"])
plt.xlim(5, 50)
plt.title("Action probabilities of the Agent (Lin. Spec.)");_____no_output_____

## 3°) The bus replacement dataset.

### A. Initialization
First, we decide on a number of bus, that we initialize in state 0.

We treat the dataset as an N_Bus x 4 array, containing:
* The identifier of the buses
* The choice made for the buses at this time
* The mileage
* The states_____no_output_____

[np.random.gumbel(),np.random.gumbel()]_____no_output_____np.array(([np.random.gumbel(),np.random.gumbel()]))_____no_output_____np.random.gumbel(size=(n_bus,2))_____no_output_____n_bus = 1
bus_array = np.hstack(
    (
        np.linspace(1, n_bus, n_bus).reshape(-1, 1),
        np.zeros((n_bus, 3)),
        np.random.gumbel(size=(n_bus, 2)),
    )
)_____no_output_____bus_array_____no_output_____

### B. State transition functions_____no_output_____The dataset of buses transition from one period to another in the following way:
* The agent observes the current state of the buses, and take a decision to maintain (0) or replace (1) each bus.*

* The mileage of each bus is updated a first time:
    * Kept at its current value if the agent decided to maintain the engine
    * Reset to 0 if the agent decided to replace the engine

* The mileage of each bus is updated a second time : a random draw is taken from the mileage increment random variable for each bus, and the corresponding mileage is added to the mileage of the bus.

* The state of each bus is updated to reflect its new mileage

We define below the transition function and the decision function_____no_output_____

def decide(s, pchoice):
    """
    Make a decision to maintain or replace the bus, based on a probability of choice p
    and a current state s
    """
    s = int(s)

    return np.random.choice([0, 1], p=[pchoice[s], 1 - pchoice[s]])

def transition(bus_array, p):
    """
    Return the updated bus dataset after one decision of our agent.
    Takes:
        * bus_array : An array of buses, containing the identifier of the buses, their mileages, and their current
                        state.
        * pchoice : The converged choice probabities of the agent making the decision

    Returns:
        * The updated dataset of buses, with the new decisions appended at the end of the dataframe.
    """
    # Recovering the number of buses, the previous mileage and the previous states of the buses
    n_bus = int(bus_array[:, 0].max())
    prev_mileage = bus_array[-n_bus:, 2]
    prev_states = bus_array[-n_bus:, 3]
    # Generating choices from choice probabilities, conditional on the state of each bus
    choices = np.array([decide(x, pchoice) for x in prev_states])

    # Generating the new mileage and state
    new_mileage = (1 - choices) * prev_mileage + mileage_dist.rvs(size=n_bus)
    new_states = np.floor(new_mileage / 5000)
    new_array = np.vstack(
        (
            bus_array[-n_bus:, 0],
            np.zeros(n_bus),
            new_mileage,
            new_states,
            np.random.gumbel(size=(2, n_bus)),
        )
    )
    bus_array[-n_bus:, 1] = choices
    return np.vstack((bus_array, new_array.T))_____no_output_____bus_array_____no_output_____

### C. Generation of the full dataset_____no_output_____The complete dataset is generated by making the agent take a serie of decisions. Here, we assume that the agent takes 1000 consecutive decisions. The total number of data points thus generated will be equal to 100,000 (1000 buses and 100 time periods)._____no_output_____

n_periods = 1000_____no_output_____for i in range(n_periods):
    bus_array = transition(bus_array, pchoice)_____no_output_____

Finally, the array is stored into a Dataframe, to make plot and data exports more convenient._____no_output_____

def calculate_utility(bus_array, lin_cost, params_lin, beta):
    lin_dataframe = pd.DataFrame(
        bus_array, columns=["Identifier", "Choice", "Mileage", "State", "ϵ0", "ϵ1"]
    )

    lin_dataframe["Relative_ERepl"] = (
        lin_dataframe.groupby("State")["Choice"]
        .transform("mean")
        .apply(lambda x: np.where(x == 0, -1 * np.inf, np.log(x)))
    )

    lin_dataframe["simulatied_choice"] = np.where(
        (lin_dataframe["Relative_ERepl"] + lin_dataframe["ϵ1"] - lin_dataframe["ϵ0"])
        > 0,
        1,
        0,
    )

    lin_dataframe["t"] = lin_dataframe.index

    maintenance_cost = myopic_costs(
        int(lin_dataframe["State"].max()), lin_cost, params_lin, _
    ).T[0]
    replacement_cost = myopic_costs(
        int(lin_dataframe["State"].max()), lin_cost, params_lin, _
    ).T[1]

    Maintenance_Cost = (
        pd.DataFrame(maintenance_cost)
        .reset_index()
        .rename(columns={"index": "State", 0: "maintenance_cost"})
    )
    Replacement_Cost = (
        pd.DataFrame(replacement_cost)
        .reset_index()
        .rename(columns={"index": "State", 0: "replacement_cost"})
    )

    lin_dataframe = lin_dataframe.merge(Maintenance_Cost, on="State", how="left").merge(
        Replacement_Cost, on="State", how="left"
    )

    lin_dataframe = lin_dataframe.assign(
        util=lambda x: np.where(
            x["simulatied_choice"] == 0,
            -1 * (x["maintenance_cost"]) + x["ϵ0"],
            -1 * (x["replacement_cost"]) + x["ϵ1"],
        )
    )

    lin_dataframe["β"] = beta
    u1=lin_dataframe.iloc[0, -2]
    v1_v0=lin_dataframe.iloc[0, -2]

    return ((
        (lin_dataframe["β"] ** (lin_dataframe["t"] - 1)) * lin_dataframe["util"]
    ).sum(),u1)  _____no_output_____calculate_utility(bus_array, lin_cost, params_lin, beta)_____no_output_____n_bus = 1
init_bus_array_0 = np.hstack(
    (
        np.linspace(1, n_bus, n_bus).reshape(-1, 1),
        np.zeros((n_bus, 3)),
        np.random.gumbel(size=(n_bus, 2)),
    )
)
init_bus_array_1 = np.hstack(
    (
        np.linspace(1, n_bus, n_bus).reshape(-1, 1),
        np.ones((n_bus, 1)),
        np.zeros((n_bus, 2)),
        np.random.gumbel(size=(n_bus, 2)),
    )
)_____no_output_____U0 = []
for i in range(1,101):
    n_periods = 1000
    bus_array=init_bus_array_0
    if i%10==0:
        print(i)
    for j in range(n_periods):
        bus_array = transition(bus_array, pchoice)
    u = calculate_utility(bus_array, lin_cost, params_lin, beta)
    U0.append(u[0])
u_1_0=u[1]10
20
30
40
50
60
70
80
90
100
u_1_0_____no_output_____U1 = []
for i in range(1,101):
    n_periods = 1000
    bus_array=init_bus_array_1
    if i%10==0:
        print(i)
    for j in range(n_periods):
        bus_array = transition(bus_array, pchoice)
    u = calculate_utility(bus_array, lin_cost, params_lin, beta)
    U1.append(u[0])
u_1_1=u[1]10
20
30
40
50
60
70
80
90
100
u_1_1_____no_output_____U0_mean=np.mean(U0)_____no_output_____U1_mean=np.mean(U1)_____no_output_____rel_EV_repl=u_1_1+U1_mean-u_1_0-U0_mean
exp(rel_EV_repl)_____no_output_____

### C. Plot of the data_____no_output_____

choice_freq = lin_dataframe.loc[:, ["Choice", "State"]].groupby(["State"]).mean().sort_values("Choice").reset_index()
plt.plot(choice_freq["State"], 1-choice_freq["Choice"])
plt.plot(pchoice, "--")
plt.xlabel("Observed state of the bus")
plt.ylabel("Probability of maintenance")
plt.legend(["Observed behavior", "True choice probabilities"])
plt.title("Behavior of forward-looking agent (Lin. Spec.)")
plt.xlim(5, 25)
plt.ylim(-0.05, 1.05);_____no_output_____

### C. Export of the data

We only keep the decision state and the choice, as the other variables will not be used in the estimation section._____no_output_____

lin_dataframe.loc[n_bus+1:,["Choice", "State"]].to_csv("Datasets/Lin_Dataset.csv", index=False)_____no_output_____

# V. Generation of simulated data (using other cost specifications)

## 1°) Quadratic form_____no_output_____

params_quad = (rc, theta1_1, theta1_2)
p = (p_x0, p_x1, p_x2)
quad_forward, quad_myopic = contraction_mapping(S=70, p=p, MF=quad_cost, params=params_quad, beta = 0.75)
pchoice = quad_forward.T[0]
plt.plot(quad_forward.T[0])
plt.plot(quad_myopic.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking", "Myopic"])
plt.xlim(5, 35)
plt.title("Action probabilities of the Agent (Quad. Spec.)");Convergence achieved in 54 iterations
n_bus = 1000
bus_array = np.hstack((np.linspace(1, n_bus, n_bus).reshape(-1, 1), np.zeros((n_bus, 3))))
for i in range(n_periods):
    bus_array = transition(bus_array, pchoice)
quad_dataframe = pd.DataFrame(bus_array, columns=["Identifier", "Choice", "Mileage", "State"])
choice_freq = quad_dataframe.loc[:, ["Choice", "State"]].groupby(["State"]).mean().sort_values("Choice").reset_index()
plt.plot(choice_freq["State"], 1-choice_freq["Choice"])
plt.plot(pchoice, "--")
plt.xlabel("Observed state of the bus")
plt.ylabel("Probability of maintenance")
plt.legend(["Observed behavior", "True choice probabilities"])
plt.title("Behavior of forward-looking agent (Quad. Spec.)")
plt.xlim(5, 20)
plt.ylim(-0.05, 1.05);
quad_dataframe.loc[n_bus+1:,["Choice", "State"]].to_csv("Datasets\Quad_Dataset.csv", index=False)_____no_output_____

# VI. Estimation

## 1°) Dynamic Utility Logit Model

In this section, we implement a Dynamic Utility Logit model, which will be our workbench for the estimation of the data. We wrap in this class the functions defined earlier (to avoid global references) and refactor some portions of the code (to make the model more flexible). The maintenance cost function is kept outside of the scope of the class, and must be supplied at the initialization of the object as an argument._____no_output_____

class DynamicLogit(object):
    def __init__(self, data, Y, X, p, MF, npars):
        """
        A statistics workbench used to evaluate the cost parameters underlying
        a bus replacement pattern by a forward-looking agent.

        Takes:
            * Data: a Pandas dataframe, which contains:
                -Y: the name of the column containing the dummy exogenous
                    variable (here, the choice)
                -X: the name of the column containing the endogenous variable
                    (here, the state of the bus)

            * p: The state-transition vector of endogenous variable.
                    For instance, p = [0, 0.6, 0.4] means that the bus will
                    transition to the next mileage state with probability 0.6,
                    and to the second next mileage state with probability 0.4.

            * MF: A function passed as an argument, which is the functional
                  form for the maintenance cost. This function must accept as
                  a first argument a state s, and as a second argument a vector
                  of parameters.

            * npars: The number of parameters to evalutate (i.e. the number of
                     parameters of the maintenance cost function, plus 1 for
                     the replacement cost)
        """

        self.endog = data.loc[:, Y].values
        self.exog = data.loc[:, X].values

        self.N = self.endog.shape[0]
        self.S = int(self.exog.max()*2) # Assumes that the true maximum number
                                         # states is twice the maximum observed
                                         # state.

        # Check that p is a correct vector of probabilities (i.e. sums to 1)
        p = np.array(p)
        if p.sum() == 1:
            self.p = p
        else:
            raise ValueError(("The probability of state transitions should add"
                              " up to 1!"))

        # Check that the stated number of parameters correspond to the
        # specifications of the maintenance cost function.
        try:
            MF(1, [0]*(npars-1))
        except ValueError:
            raise ValueError(("The number of parameters specified does not "
                              "match the specification of the maintenance cost"
                              " function!"))
        else:
            self.MF = MF
            self.npars = npars

        # To speed up computations and avoid loops when computing the log
        # likelihood, we create a few useful matrices here:
        S = self.S

        # A (SxN) matrix indicating the state of each observation
        self.state_mat = np.array([[self.exog[i]==s for i in range(self.N)]
                                                    for s in range(self.S)])

        # A (SxS) matrix indicating the probability of a bus transitioning
        # from a state s to a state s' (used to compute maintenance utility)

        self.trans_mat = np.zeros((S, S))
        for i in range(S):
            for j, _p in enumerate(self.p):
                if i + j < S-1:
                    self.trans_mat[i+j][i] = _p
                elif i + j == S-1:
                    self.trans_mat[S-1][i] = p[j:].sum()
                else:
                    pass

        # A second (SxS) matrix which regenerates the bus' state to 0 with
        # certainty (used to compute the replacement utility)
        self.regen_mat = np.vstack((np.ones((1, S)),np.zeros((S-1, S))))

        # A (2xN) matrix indicating with a dummy the decision taken by the agent
        # for each time/bus observation (replace or maintain)
        self.dec_mat = np.vstack(((1-self.endog), self.endog))

    def myopic_costs(self, params):
        S = self.S
        """
        This function computes the myopic expected cost associated with each
        decision for each state.

        Takes:
            * A vector params, to be supplied to the maintenance cost function
              MF. The first element of the vector is the replacement cost rc.

        Returns:
            * A (Sx2) array containing the maintenance and replacement costs
              for the S possible states of the bus
        """
        rc = params[0]
        thetas = params[1:]
        maint_cost = [self.MF(s, thetas) for s in range(0, S)]
        repl_cost = [rc for state in range(0, S)]
        return np.vstack((maint_cost, repl_cost)).T

    def fl_costs(self, params, beta=0.75, threshold=1e-6, suppr_output=False):
        """
        Compute the non-myopic expected value of the agent for each possible
        decision and each possible state of the bus, conditional on a vector of
        parameters and on the maintenance cost function specified at the
        initialization of the DynamicUtility model.

        Iterates until the difference in the previously obtained expected value
        and the new expected value is smaller than a constant.

        Takes:
            * A vector params for the cost function
            * A discount factor beta (optional)
            * A convergence threshold (optional)
            * A boolean argument to suppress the output (optional)

        Returns:
            * An (Sx2) array of forward-looking costs associated with each
              state and each decision.
        """
        achieved = True
        # Initialization of the contraction mapping
        k = 0
        EV = np.zeros((self.S, 2))
        self.EV_myopic = EV_new = self.myopic_costs(params)

        # Contraction mapping Loop
        while abs(EV_new-EV).max() > threshold:
            EV = EV_new
            pchoice = self.choice_prob(EV)
            ecost = (pchoice*EV).sum(1)
            futil_maint = np.dot(ecost, self.trans_mat)
            futil_repl = np.dot(ecost, self.regen_mat)
            futil = np.vstack((futil_maint, futil_repl)).T

            EV_new = self.EV_myopic + beta*futil
            k += 1
            if k == 1000:
                achieved = False
                break

        # Output:
        if not suppr_output:
            if achieved:
                print("Convergence achieved in {} iterations".format(k))
            else:
                print("CM could not converge! Mean difference = {:.6f}".format(
                                                            (EV_new-EV).mean())
                                                                              )

        self.EV_FL = EV_new
        return EV

    def choice_prob(self, cost_array):
        """
        Returns the probability of each choice for each observed state,
        conditional on an array of state/decision costs (generated by the
        myopic or forward-looking cost functions)
        """
        cost = cost_array - cost_array.min(1).reshape(self.S, -1)
        util = np.exp(-cost)
        pchoice = util/(np.sum(util, 1).reshape(self.S, -1))
        return pchoice

    def loglike(self, params):
        """
        The log-likelihood of the Dynamic model is estimated in several steps.
        1°) The current parameters are supplied to the contraction mapping
            function
        2°) The function returns a matrix of decision probabilities for each
            state.
        3°) This matrix is used to compute the loglikelihood of the
            observations
        4°) The log-likelihood are then summed accross individuals, and
            returned
        """
        util = self.fl_costs(params, suppr_output=True)
        pchoice = self.choice_prob(util)
        logprob = np.log(np.dot(pchoice.T, self.state_mat))
        return -np.sum(self.dec_mat*logprob)

    def fit_likelihood(self, x0=None, bounds=None):
        """
        Fit the parameters to the data.
        """
        if bounds == None:
            bounds = [(1e-6, None) for i in range(self.npars)]

        if x0 == None:
            x0 = [0.1 for i in range(self.npars)]

        self.fitted = opt.fmin_l_bfgs_b(self.loglike, x0=x0, approx_grad=True,
                                        bounds=bounds)

    def get_parameters(self):
        """
        Return the parameters obtained after fitting the likelihood function
        to the data.
        """
        return self.fitted[0]

    def print_parameters(self):
        loglike =  self.fitted[1]
        fit_params = self.get_parameters()
        RC, thetas = fit_params[0], fit_params[1:]
        logstring = "Log-likelihood = {0:.2f}".format(loglike)
        thetas_string = ["theta1_{0} = {1:.4f}".format(i+1, t) \
                                                for i, t in enumerate(thetas)]
        thetas_string = ", ".join(thetas_string)
        rc_string = "Parameters: RC = {0:.4f}".format(RC)
        print(logstring, rc_string + ", " + thetas_string)_____no_output_____

## 2°) Fitting the linear cost data

In this section, we fit the data generated by the linear cost function and recover the parameters $\text{RC}$ and $\theta$. We do it for different characterizations of the cost function, and thereby illustrate the consequences of a misspecification.

Recall that in the linear model:

$$\text{RC} = 20$$

$$\theta_{11} = 0.5$$

### A. Fitting the true linear cost function to the data
We start by initializing a DynamicLogit instance, and fit the likelihood function to the data. _____no_output_____

lin_to_lin = DynamicLogit(lin_dataframe, "Choice", "State", p, lin_cost, npars=2)
lin_to_lin.fit_likelihood()_____no_output_____

The results:_____no_output_____

lin_to_lin.print_parameters()Log-likelihood = 16326.12 Parameters: RC = 19.5287, theta1_1 = 0.4860
lin_forward_app, lin_myopic_app = contraction_mapping(S=70, p=p,
                                                      MF=lin_cost, params=lin_to_lin.get_parameters(),
                                                      beta = 0.75)Convergence achieved in 54 iterations
plt.plot(lin_forward.T[0])
plt.plot(lin_myopic.T[0])
plt.plot(lin_forward_app.T[0])
plt.plot(lin_myopic_app.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking (Lin)", "Myopic (Lin)", "Forward-Looking (Lin, Est.)", "Myopic (Lin, Est.)", ])
plt.xlim(0, 70)
plt.title("Action probabilities of the Agent (Linear Cost Specification)");_____no_output_____

### B. Fitting an incorrect quadratic cost model to the data

We initialize a second DynamicLogit instance, this time with our quadratic cost specification_____no_output_____

quad_to_lin = DynamicLogit(lin_dataframe, "Choice", "State", p, quad_cost, npars=3)
quad_to_lin.fit_likelihood()_____no_output_____

The results:_____no_output_____

quad_to_lin.print_parameters()Log-likelihood = 16326.05 Parameters: RC = 19.1321, theta1_1 = 0.4608, theta1_2 = 0.0012

The optimization procedures correctly brings $\theta_{12}$ to 0. As a consequence, this specification fits the data perfectly._____no_output_____

quad_forward_app, quad_myopic_app = contraction_mapping(S=70, p=p,
                                                      MF=quad_cost, params=quad_to_lin.get_parameters(),
                                                      beta = 0.75)Convergence achieved in 54 iterations
plt.plot(lin_forward.T[0])
plt.plot(lin_myopic.T[0])
plt.plot(quad_forward_app.T[0])
plt.plot(quad_myopic_app.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking (Lin)", "Myopic (Lin)", "Forward-Looking (Quad)", "Myopic (Quad)"])
plt.xlim(0, 70)
plt.title("Action probabilities of the Agent (Exp. Cost Specification)");_____no_output_____

### C. Fitting an incorrect exponential cost model to the data_____no_output_____We initialize a third DynamicLogit instance, this time with our exponential cost specification_____no_output_____

exp_to_lin = DynamicLogit(lin_dataframe, "Choice", "State", p, exp_cost, npars=2)
exp_to_lin.fit_likelihood(bounds=[(0, None), (0.1, None)])_____no_output_____

The results:_____no_output_____

exp_to_lin.print_parameters()Log-likelihood = 16375.61 Parameters: RC = 13.3726, theta1_1 = 0.1255

The myopic choice probabilities look different, but the forward-looking ones fit the data quite well._____no_output_____

exp_forward_app, exp_myopic_app = contraction_mapping(S=70, p=p,
                                                      MF=exp_cost, params=exp_to_lin.get_parameters(),
                                                      beta = 0.75)Convergence achieved in 53 iterations
plt.plot(lin_forward.T[0])
plt.plot(lin_myopic.T[0])
plt.plot(exp_forward_app.T[0])
plt.plot(exp_myopic_app.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking (Lin)", "Myopic (Lin)", "Forward-Looking (Exp)", "Myopic (Exp)"])
plt.xlim(0, 70)
plt.title("Action probabilities of the Agent (Exp. Cost Specification)");_____no_output_____

### D. Fitting an incorrect log cost function to the data._____no_output_____We initialize a fourth DynamicLogit instance, this time with our log cost specification_____no_output_____

log_to_lin = DynamicLogit(lin_dataframe, "Choice", "State", p, log_cost, npars=3)
log_to_lin.fit_likelihood(x0=[10, 10, 10])_____no_output_____

The results:_____no_output_____

log_to_lin.print_parameters()Log-likelihood = 20418.23 Parameters: RC = 23.4788, theta1_1 = 0.0040, theta1_2 = 47.1788
log_forward_app, log_myopic_app = contraction_mapping(S=70, p=p, MF=log_cost,
                                                      params=log_to_lin.get_parameters(), beta = 0.75)Convergence achieved in 56 iterations

In this case, both the myopic and forward looking choice probabilities are very different from the true ones. This shows the importance of using a correctly specified functional form for the model!_____no_output_____

plt.plot(lin_forward.T[0])
plt.plot(lin_myopic.T[0])
plt.plot(log_forward_app.T[0])
plt.plot(log_myopic_app.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking (Lin)", "Myopic (Lin)", "Forward-Looking (Log)", "Myopic (Log)"])
plt.xlim(0, 70)
plt.title("Action probabilities of the Agent (Log. Cost Specification)");_____no_output_____

## 2°) Fitting the other cost data

In this section, we fit the data generated by the quadratic, exponential, and logarithmic cost function and recover the parameters $\text{RC}$ and $\theta_{11}$ and $\theta_{12}$. We do not illustrate the impact of misspecification for those cost functions.

### A. Fitting the true quadratic cost function
We start by initializing a DynamicLogit instance, and fit the likelihood function to the data.

Recall that in the quadratic model:

$$\text{RC} = 20$$

$$\theta_{11} = 0.5$$

$$\theta_{12} = 0.01$$_____no_output_____

quad_to_quad = DynamicLogit(quad_dataframe, "Choice", "State", p, quad_cost, npars=3)
quad_to_quad.fit_likelihood(x0=[20, 1, 1])_____no_output_____

The results:_____no_output_____

quad_to_quad.print_parameters()Log-likelihood = 16821.39 Parameters: RC = 20.7029, theta1_1 = 0.5757, theta1_2 = 0.0048
quad_forward_app, quad_myopic_app = contraction_mapping(S=70, p=p, MF=quad_cost,
                                                      params=quad_to_quad.get_parameters(), beta = 0.75)Convergence achieved in 54 iterations
plt.plot(quad_forward.T[0])
plt.plot(quad_myopic.T[0])
plt.plot(quad_forward_app.T[0])
plt.plot(quad_myopic_app.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking (Quad)", "Myopic (Quad)", "Forward-Looking (Quad, Est.)", "Myopic (Quad, Est.)"])
plt.xlim(5, 40)
plt.title("Action probabilities of the Agent (Quad. Cost Specification)");_____no_output_____

### B. Fitting the exponential cost function_____no_output_____Recall that in the exponential model:

$$\text{RC} = 15$$

$$\theta_{11} = 0.05$$_____no_output_____

exp_to_exp = DynamicLogit(exp_dataframe, "Choice", "State", p, exp_cost, npars=2)
exp_to_exp.fit_likelihood()_____no_output_____

The results:_____no_output_____

exp_to_exp.print_parameters()Log-likelihood = 8478.18 Parameters: RC = 14.8325, theta1_1 = 0.0495
exp_forward_app, exp_myopic_app = contraction_mapping(S=70, p=p, MF=exp_cost,
                                                      params=exp_to_exp.get_parameters(), beta = 0.75)Convergence achieved in 55 iterations
plt.plot(exp_forward.T[0])
plt.plot(exp_myopic.T[0])
plt.plot(exp_forward_app.T[0])
plt.plot(exp_myopic_app.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking (Exp)", "Myopic (Exp)", "Forward-Looking (Exp, Est.)", "Myopic (Exp, Est.)"])
plt.xlim(20, 65)
plt.title("Action probabilities of the Agent (Exp. Cost Specification)");_____no_output_____

### C. Fitting the logarithmic cost function_____no_output_____Recall that in the logarithmic model:

$$\text{RC} = 4$$

$$\theta_{11} = 1$$

$$\theta_{12} = 10$$_____no_output_____

log_to_log = DynamicLogit(log_dataframe, "Choice", "State", p, log_cost, npars=3)
log_to_log.fit_likelihood()_____no_output_____

The results:_____no_output_____

log_to_log.print_parameters()Log-likelihood = 51738.04 Parameters: RC = 0.5247, theta1_1 = 0.0321, theta1_2 = 0.3004

The parameters look different from the one used to generate the data..._____no_output_____

log_forward_app, log_myopic_app = contraction_mapping(S=70, p=p, MF=log_cost,
                                                      params=log_to_log.get_parameters(), beta = 0.75)Convergence achieved in 50 iterations
plt.plot(log_forward.T[0])
plt.plot(log_myopic.T[0])
plt.plot(log_forward_app.T[0])
plt.plot(log_myopic_app.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking (Log)", "Myopic (Log)", "Forward-Looking (Log, Est.)", "Myopic (Log, Est.)"])
plt.xlim(0, 65)
plt.title("Action probabilities of the Agent (Log. Cost Specification)");_____no_output_____

... but the fit of the data is excellent. Why is that?

Remember that our (true) log-cost specification is: $\qquad \text{MF}(s, \theta) = 1*\log(10s)$, and $RC = 4$

We also know that $\log(a*b) = \log(a) + \log(b)$ and that $a*\log(b) = \log(b^a)$.

For this reason, we can rewrite the same cost function in different ways, by changing the relative value of $\theta_{11}$ and $\theta_{12}$. Here, the parameters recovered are algebrically equivalent to the initial parameters._____no_output_____


# Notebook from zenetio/deep-learning
Path: Sequence-Models/Dinosaurus+Island+--+Character+level+language+model+final+-+v3.ipynb

# Character level language model - Dinosaurus land

Welcome to Dinosaurus Island! 65 million years ago, dinosaurs existed, and in this assignment they are back. You are in charge of a special task. Leading biology researchers are creating new breeds of dinosaurs and bringing them to life on earth, and your job is to give names to these dinosaurs. If a dinosaur does not like its name, it might go beserk, so choose wisely!

Luckily you have learned some deep learning and you will use it to save the day. Your assistant has collected a list of all the dinosaur names they could find, and compiled them into this [dataset](dinos.txt). (Feel free to take a look by clicking the previous link.) To create new dinosaur names, you will build a character level language model to generate new names. Your algorithm will learn the different name patterns, and randomly generate new names. Hopefully this algorithm will keep you and your team safe from the dinosaurs' wrath!

By completing this assignment you will learn:

- How to store text data for processing using an RNN
- How to synthesize data, by sampling predictions at each time step and passing it to the next RNN-cell unit
- How to build a character-level text generation recurrent neural network
- Why clipping the gradients is important

We will begin by loading in some functions that we have provided for you in `rnn_utils`. Specifically, you have access to functions such as `rnn_forward` and `rnn_backward` which are equivalent to those you've implemented in the previous assignment. _____no_output_____

import numpy as np
from utils import *
import random_____no_output_____

## 1 - Problem Statement

### 1.1 - Dataset and Preprocessing

Run the following cell to read the dataset of dinosaur names, create a list of unique characters (such as a-z), and compute the dataset and vocabulary size. _____no_output_____

data = open('dinos.txt', 'r').read()
data= data.lower()
chars = list(set(data))
data_size, vocab_size = len(data), len(chars)
print('There are %d total characters and %d unique characters in your data.' % (data_size, vocab_size))There are 19909 total characters and 27 unique characters in your data.

The characters are a-z (26 characters) plus the "\n" (or newline character), which in this assignment plays a role similar to the `<EOS>` (or "End of sentence") token we had discussed in lecture, only here it indicates the end of the dinosaur name rather than the end of a sentence. In the cell below, we create a python dictionary (i.e., a hash table) to map each character to an index from 0-26. We also create a second python dictionary that maps each index back to the corresponding character character. This will help you figure out what index corresponds to what character in the probability distribution output of the softmax layer. Below, `char_to_ix` and `ix_to_char` are the python dictionaries. _____no_output_____

char_to_ix = { ch:i for i,ch in enumerate(sorted(chars)) }
ix_to_char = { i:ch for i,ch in enumerate(sorted(chars)) }
print(char_to_ix){'\n': 0, 'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26}

### 1.2 - Overview of the model

Your model will have the following structure:

- Initialize parameters
- Run the optimization loop
    - Forward propagation to compute the loss function
    - Backward propagation to compute the gradients with respect to the loss function
    - Clip the gradients to avoid exploding gradients
    - Using the gradients, update your parameter with the gradient descent update rule.
- Return the learned parameters

 **Figure 1**: Recurrent Neural Network, similar to what you had built in the previous notebook "Building a RNN - Step by Step".

At each time-step, the RNN tries to predict what is the next character given the previous characters. The dataset $X = (x^{\langle 1 \rangle}, x^{\langle 2 \rangle}, ..., x^{\langle T_x \rangle})$ is a list of characters in the training set, while $Y = (y^{\langle 1 \rangle}, y^{\langle 2 \rangle}, ..., y^{\langle T_x \rangle})$ is such that at every time-step $t$, we have $y^{\langle t \rangle} = x^{\langle t+1 \rangle}$. _____no_output_____## 2 - Building blocks of the model

In this part, you will build two important blocks of the overall model:
- Gradient clipping: to avoid exploding gradients
- Sampling: a technique used to generate characters

You will then apply these two functions to build the model._____no_output_____### 2.1 - Clipping the gradients in the optimization loop

In this section you will implement the `clip` function that you will call inside of your optimization loop. Recall that your overall loop structure usually consists of a forward pass, a cost computation, a backward pass, and a parameter update. Before updating the parameters, you will perform gradient clipping when needed to make sure that your gradients are not "exploding," meaning taking on overly large values.

In the exercise below, you will implement a function `clip` that takes in a dictionary of gradients and returns a clipped version of gradients if needed. There are different ways to clip gradients; we will use a simple element-wise clipping procedure, in which every element of the gradient vector is clipped to lie between some range [-N, N]. More generally, you will provide a `maxValue` (say 10). In this example, if any component of the gradient vector is greater than 10, it would be set to 10; and if any component of the gradient vector is less than -10, it would be set to -10. If it is between -10 and 10, it is left alone.

 **Figure 2**: Visualization of gradient descent with and without gradient clipping, in a case where the network is running into slight "exploding gradient" problems.

**Exercise**: Implement the function below to return the clipped gradients of your dictionary `gradients`. Your function takes in a maximum threshold and returns the clipped versions of your gradients. You can check out this [hint](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.clip.html) for examples of how to clip in numpy. You will need to use the argument `out = ...`._____no_output_____

### GRADED FUNCTION: clip

def clip(gradients, maxValue):
    '''
    Clips the gradients' values between minimum and maximum.

    Arguments:
    gradients -- a dictionary containing the gradients "dWaa", "dWax", "dWya", "db", "dby"
    maxValue -- everything above this number is set to this number, and everything less than -maxValue is set to -maxValue

    Returns:
    gradients -- a dictionary with the clipped gradients.
    '''

    dWaa, dWax, dWya, db, dby = gradients['dWaa'], gradients['dWax'], gradients['dWya'], gradients['db'], gradients['dby']

    ### START CODE HERE ###
    # clip to mitigate exploding gradients, loop over [dWax, dWaa, dWya, db, dby]. (≈2 lines)
    for gradient in [dWax, dWaa, dWya, db, dby]:
        np.clip(gradient, -maxValue, maxValue, out= gradient)
    ### END CODE HERE ###

    gradients = {"dWaa": dWaa, "dWax": dWax, "dWya": dWya, "db": db, "dby": dby}

    return gradients_____no_output_____np.random.seed(3)
dWax = np.random.randn(5,3)*10
dWaa = np.random.randn(5,5)*10
dWya = np.random.randn(2,5)*10
db = np.random.randn(5,1)*10
dby = np.random.randn(2,1)*10
gradients = {"dWax": dWax, "dWaa": dWaa, "dWya": dWya, "db": db, "dby": dby}
gradients = clip(gradients, 10)
print("gradients[\"dWaa\"][1][2] =", gradients["dWaa"][1][2])
print("gradients[\"dWax\"][3][1] =", gradients["dWax"][3][1])
print("gradients[\"dWya\"][1][2] =", gradients["dWya"][1][2])
print("gradients[\"db\"][4] =", gradients["db"][4])
print("gradients[\"dby\"][1] =", gradients["dby"][1])gradients["dWaa"][1][2] = 10.0
gradients["dWax"][3][1] = -10.0
gradients["dWya"][1][2] = 0.29713815361
gradients["db"][4] = [ 10.]
gradients["dby"][1] = [ 8.45833407]

** Expected output:**

    **gradients["dWaa"][1][2] **

    10.0

    **gradients["dWax"][3][1]**

    -10.0

    **gradients["dWya"][1][2]**

0.29713815361

    **gradients["db"][4]**

[ 10.]

    **gradients["dby"][1]**

[ 8.45833407]

_____no_output_____### 2.2 - Sampling

Now assume that your model is trained. You would like to generate new text (characters). The process of generation is explained in the picture below:

 **Figure 3**: In this picture, we assume the model is already trained. We pass in $x^{\langle 1\rangle} = \vec{0}$ at the first time step, and have the network then sample one character at a time.

**Exercise**: Implement the `sample` function below to sample characters. You need to carry out 4 steps:

- **Step 1**: Pass the network the first "dummy" input $x^{\langle 1 \rangle} = \vec{0}$ (the vector of zeros). This is the default input before we've generated any characters. We also set $a^{\langle 0 \rangle} = \vec{0}$

- **Step 2**: Run one step of forward propagation to get $a^{\langle 1 \rangle}$ and $\hat{y}^{\langle 1 \rangle}$. Here are the equations:

$$ a^{\langle t+1 \rangle} = \tanh(W_{ax}  x^{\langle t \rangle } + W_{aa} a^{\langle t \rangle } + b)\tag{1}$$

$$ z^{\langle t + 1 \rangle } = W_{ya}  a^{\langle t + 1 \rangle } + b_y \tag{2}$$

$$ \hat{y}^{\langle t+1 \rangle } = softmax(z^{\langle t + 1 \rangle })\tag{3}$$

Note that $\hat{y}^{\langle t+1 \rangle }$ is a (softmax) probability vector (its entries are between 0 and 1 and sum to 1). $\hat{y}^{\langle t+1 \rangle}_i$ represents the probability that the character indexed by "i" is the next character.  We have provided a `softmax()` function that you can use.

- **Step 3**: Carry out sampling: Pick the next character's index according to the probability distribution specified by $\hat{y}^{\langle t+1 \rangle }$. This means that if $\hat{y}^{\langle t+1 \rangle }_i = 0.16$, you will pick the index "i" with 16% probability. To implement it, you can use [`np.random.choice`](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.random.choice.html).

Here is an example of how to use `np.random.choice()`:
```python
np.random.seed(0)
p = np.array([0.1, 0.0, 0.7, 0.2])
index = np.random.choice([0, 1, 2, 3], p = p.ravel())
```
This means that you will pick the `index` according to the distribution:
$P(index = 0) = 0.1, P(index = 1) = 0.0, P(index = 2) = 0.7, P(index = 3) = 0.2$.

- **Step 4**: The last step to implement in `sample()` is to overwrite the variable `x`, which currently stores $x^{\langle t \rangle }$, with the value of $x^{\langle t + 1 \rangle }$. You will represent $x^{\langle t + 1 \rangle }$ by creating a one-hot vector corresponding to the character you've chosen as your prediction. You will then forward propagate $x^{\langle t + 1 \rangle }$ in Step 1 and keep repeating the process until you get a "\n" character, indicating you've reached the end of the dinosaur name. _____no_output_____

# GRADED FUNCTION: sample

def sample(parameters, char_to_ix, seed):
    """
    Sample a sequence of characters according to a sequence of probability distributions output of the RNN

    Arguments:
    parameters -- python dictionary containing the parameters Waa, Wax, Wya, by, and b.
    char_to_ix -- python dictionary mapping each character to an index.
    seed -- used for grading purposes. Do not worry about it.

    Returns:
    indices -- a list of length n containing the indices of the sampled characters.
    """

    # Retrieve parameters and relevant shapes from "parameters" dictionary
    Waa, Wax, Wya, by, b = parameters['Waa'], parameters['Wax'], parameters['Wya'], parameters['by'], parameters['b']
    vocab_size = by.shape[0]
    n_a = Waa.shape[1]

    ### START CODE HERE ###
    # Step 1: Create the one-hot vector x for the first character (initializing the sequence generation). (≈1 line)
    x = np.zeros((vocab_size, 1))
    # Step 1': Initialize a_prev as zeros (≈1 line)
    a_prev = np.zeros((n_a, 1))

    # Create an empty list of indices, this is the list which will contain the list of indices of the characters to generate (≈1 line)
    indices = []

    # Idx is a flag to detect a newline character, we initialize it to -1
    idx = -1

    # Loop over time-steps t. At each time-step, sample a character from a probability distribution and append
    # its index to "indices". We'll stop if we reach 50 characters (which should be very unlikely with a well
    # trained model), which helps debugging and prevents entering an infinite loop.
    counter = 0
    newline_character = char_to_ix['\n']

    while (idx != newline_character and counter != 50):

        # Step 2: Forward propagate x using the equations (1), (2) and (3)
        a = np.tanh(np.dot(Wax, x) + np.dot(Waa, a_prev) + b)
        z = np.dot(Wya, a) + by
        y = softmax(z)

        # for grading purposes
        np.random.seed(counter+seed)

        # Step 3: Sample the index of a character within the vocabulary from the probability distribution y
        idx = np.random.choice(list(range(vocab_size)), p = y.ravel())

        # Append the index to "indices"
        indices.append(idx)

        # Step 4: Overwrite the input character as the one corresponding to the sampled index.
        x = np.zeros((vocab_size, 1))
        x[idx] = 1

        # Update "a_prev" to be "a"
        a_prev = a

        # for grading purposes
        seed += 1
        counter +=1

    ### END CODE HERE ###

    if (counter == 50):
        indices.append(char_to_ix['\n'])

    return indices_____no_output_____np.random.seed(2)
_, n_a = 20, 100
Wax, Waa, Wya = np.random.randn(n_a, vocab_size), np.random.randn(n_a, n_a), np.random.randn(vocab_size, n_a)
b, by = np.random.randn(n_a, 1), np.random.randn(vocab_size, 1)
parameters = {"Wax": Wax, "Waa": Waa, "Wya": Wya, "b": b, "by": by}

indices = sample(parameters, char_to_ix, 0)
print("Sampling:")
print("list of sampled indices:", indices)
print("list of sampled characters:", [ix_to_char[i] for i in indices])Sampling:
list of sampled indices: [12, 17, 24, 14, 13, 9, 10, 22, 24, 6, 13, 11, 12, 6, 21, 15, 21, 14, 3, 2, 1, 21, 18, 24, 7, 25, 6, 25, 18, 10, 16, 2, 3, 8, 15, 12, 11, 7, 1, 12, 10, 2, 7, 7, 11, 17, 24, 12, 3, 1, 0]
list of sampled characters: ['l', 'q', 'x', 'n', 'm', 'i', 'j', 'v', 'x', 'f', 'm', 'k', 'l', 'f', 'u', 'o', 'u', 'n', 'c', 'b', 'a', 'u', 'r', 'x', 'g', 'y', 'f', 'y', 'r', 'j', 'p', 'b', 'c', 'h', 'o', 'l', 'k', 'g', 'a', 'l', 'j', 'b', 'g', 'g', 'k', 'q', 'x', 'l', 'c', 'a', '\n']

** Expected output:**

    **list of sampled indices:**

    [12, 17, 24, 14, 13, 9, 10, 22, 24, 6, 13, 11, 12, 6, 21, 15, 21, 14, 3, 2, 1, 21, 18, 24,

    7, 25, 6, 25, 18, 10, 16, 2, 3, 8, 15, 12, 11, 7, 1, 12, 10, 2, 7, 7, 11, 5, 6, 12, 25, 0, 0]

    **list of sampled characters:**

    ['l', 'q', 'x', 'n', 'm', 'i', 'j', 'v', 'x', 'f', 'm', 'k', 'l', 'f', 'u', 'o',

    'u', 'n', 'c', 'b', 'a', 'u', 'r', 'x', 'g', 'y', 'f', 'y', 'r', 'j', 'p', 'b', 'c', 'h', 'o',

    'l', 'k', 'g', 'a', 'l', 'j', 'b', 'g', 'g', 'k', 'e', 'f', 'l', 'y', '\n', '\n']

_____no_output_____## 3 - Building the language model

It is time to build the character-level language model for text generation.

### 3.1 - Gradient descent

In this section you will implement a function performing one step of stochastic gradient descent (with clipped gradients). You will go through the training examples one at a time, so the optimization algorithm will be stochastic gradient descent. As a reminder, here are the steps of a common optimization loop for an RNN:

- Forward propagate through the RNN to compute the loss
- Backward propagate through time to compute the gradients of the loss with respect to the parameters
- Clip the gradients if necessary
- Update your parameters using gradient descent

**Exercise**: Implement this optimization process (one step of stochastic gradient descent).

We provide you with the following functions:

```python
def rnn_forward(X, Y, a_prev, parameters):
    """ Performs the forward propagation through the RNN and computes the cross-entropy loss.
    It returns the loss' value as well as a "cache" storing values to be used in the backpropagation."""
    ....
    return loss, cache

def rnn_backward(X, Y, parameters, cache):
    """ Performs the backward propagation through time to compute the gradients of the loss with respect
    to the parameters. It returns also all the hidden states."""
    ...
    return gradients, a

def update_parameters(parameters, gradients, learning_rate):
    """ Updates parameters using the Gradient Descent Update Rule."""
    ...
    return parameters
```_____no_output_____

# GRADED FUNCTION: optimize

def optimize(X, Y, a_prev, parameters, learning_rate = 0.01):
    """
    Execute one step of the optimization to train the model.

    Arguments:
    X -- list of integers, where each integer is a number that maps to a character in the vocabulary.
    Y -- list of integers, exactly the same as X but shifted one index to the left.
    a_prev -- previous hidden state.
    parameters -- python dictionary containing:
                        Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)
                        Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)
                        Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
                        b --  Bias, numpy array of shape (n_a, 1)
                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
    learning_rate -- learning rate for the model.

    Returns:
    loss -- value of the loss function (cross-entropy)
    gradients -- python dictionary containing:
                        dWax -- Gradients of input-to-hidden weights, of shape (n_a, n_x)
                        dWaa -- Gradients of hidden-to-hidden weights, of shape (n_a, n_a)
                        dWya -- Gradients of hidden-to-output weights, of shape (n_y, n_a)
                        db -- Gradients of bias vector, of shape (n_a, 1)
                        dby -- Gradients of output bias vector, of shape (n_y, 1)
    a[len(X)-1] -- the last hidden state, of shape (n_a, 1)
    """

    ### START CODE HERE ###

    # Forward propagate through time (≈1 line)
    loss, cache = rnn_forward(X, Y, a_prev, parameters)

    # Backpropagate through time (≈1 line)
    gradients, a = rnn_backward(X, Y, parameters, cache)

    # Clip your gradients between -5 (min) and 5 (max) (≈1 line)
    gradients = clip(gradients, 5)

    # Update parameters (≈1 line)
    parameters = update_parameters(parameters, gradients, learning_rate)

    ### END CODE HERE ###

    return loss, gradients, a[len(X)-1]_____no_output_____np.random.seed(1)
vocab_size, n_a = 27, 100
a_prev = np.random.randn(n_a, 1)
Wax, Waa, Wya = np.random.randn(n_a, vocab_size), np.random.randn(n_a, n_a), np.random.randn(vocab_size, n_a)
b, by = np.random.randn(n_a, 1), np.random.randn(vocab_size, 1)
parameters = {"Wax": Wax, "Waa": Waa, "Wya": Wya, "b": b, "by": by}
X = [12,3,5,11,22,3]
Y = [4,14,11,22,25, 26]

loss, gradients, a_last = optimize(X, Y, a_prev, parameters, learning_rate = 0.01)
print("Loss =", loss)
print("gradients[\"dWaa\"][1][2] =", gradients["dWaa"][1][2])
print("np.argmax(gradients[\"dWax\"]) =", np.argmax(gradients["dWax"]))
print("gradients[\"dWya\"][1][2] =", gradients["dWya"][1][2])
print("gradients[\"db\"][4] =", gradients["db"][4])
print("gradients[\"dby\"][1] =", gradients["dby"][1])
print("a_last[4] =", a_last[4])Loss = 126.503975722
gradients["dWaa"][1][2] = 0.194709315347
np.argmax(gradients["dWax"]) = 93
gradients["dWya"][1][2] = -0.007773876032
gradients["db"][4] = [-0.06809825]
gradients["dby"][1] = [ 0.01538192]
a_last[4] = [-1.]

** Expected output:**

    **Loss **

    126.503975722

    **gradients["dWaa"][1][2]**

    0.194709315347

    **np.argmax(gradients["dWax"])**

     93

    **gradients["dWya"][1][2]**

     -0.007773876032

    **gradients["db"][4]**

     [-0.06809825]

    **gradients["dby"][1]**

    [ 0.01538192]

    **a_last[4]**

     [-1.]

_____no_output_____### 3.2 - Training the model _____no_output_____Given the dataset of dinosaur names, we use each line of the dataset (one name) as one training example. Every 100 steps of stochastic gradient descent, you will sample 10 randomly chosen names to see how the algorithm is doing. Remember to shuffle the dataset, so that stochastic gradient descent visits the examples in random order.

**Exercise**: Follow the instructions and implement `model()`. When `examples[index]` contains one dinosaur name (string), to create an example (X, Y), you can use this:
```python
        index = j % len(examples)
        X = [None] + [char_to_ix[ch] for ch in examples[index]]
        Y = X[1:] + [char_to_ix["\n"]]
```
Note that we use: `index= j % len(examples)`, where `j = 1....num_iterations`, to make sure that `examples[index]` is always a valid statement (`index` is smaller than `len(examples)`).
The first entry of `X` being `None` will be interpreted by `rnn_forward()` as setting $x^{\langle 0 \rangle} = \vec{0}$. Further, this ensures that `Y` is equal to `X` but shifted one step to the left, and with an additional "\n" appended to signify the end of the dinosaur name. _____no_output_____

# GRADED FUNCTION: model

def model(data, ix_to_char, char_to_ix, num_iterations = 35000, n_a = 50, dino_names = 7, vocab_size = 27):
    """
    Trains the model and generates dinosaur names.

    Arguments:
    data -- text corpus
    ix_to_char -- dictionary that maps the index to a character
    char_to_ix -- dictionary that maps a character to an index
    num_iterations -- number of iterations to train the model for
    n_a -- number of units of the RNN cell
    dino_names -- number of dinosaur names you want to sample at each iteration.
    vocab_size -- number of unique characters found in the text, size of the vocabulary

    Returns:
    parameters -- learned parameters
    """

    # Retrieve n_x and n_y from vocab_size
    n_x, n_y = vocab_size, vocab_size

    # Initialize parameters
    parameters = initialize_parameters(n_a, n_x, n_y)

    # Initialize loss (this is required because we want to smooth our loss, don't worry about it)
    loss = get_initial_loss(vocab_size, dino_names)

    # Build list of all dinosaur names (training examples).
    with open("dinos.txt") as f:
        examples = f.readlines()
    examples = [x.lower().strip() for x in examples]

    # Shuffle list of all dinosaur names
    np.random.seed(0)
    np.random.shuffle(examples)

    # Initialize the hidden state of your LSTM
    a_prev = np.zeros((n_a, 1))

    # Optimization loop
    for j in range(num_iterations):

        ### START CODE HERE ###

        # Use the hint above to define one training example (X,Y) (≈ 2 lines)
        index = j % len(examples)
        X = [None] + [char_to_ix[ch] for ch in examples[index]]
        Y = X[1:] + [char_to_ix["\n"]]

        # Perform one optimization step: Forward-prop -> Backward-prop -> Clip -> Update parameters
        # Choose a learning rate of 0.01
        curr_loss, gradients, a_prev = optimize(X, Y, a_prev, parameters, learning_rate = 0.01)

        ### END CODE HERE ###

        # Use a latency trick to keep the loss smooth. It happens here to accelerate the training.
        loss = smooth(loss, curr_loss)

        # Every 2000 Iteration, generate "n" characters thanks to sample() to check if the model is learning properly
        if j % 2000 == 0:

            print('Iteration: %d, Loss: %f' % (j, loss) + '\n')

            # The number of dinosaur names to print
            seed = 0
            for name in range(dino_names):

                # Sample indices and print them
                sampled_indices = sample(parameters, char_to_ix, seed)
                print_sample(sampled_indices, ix_to_char)

                seed += 1  # To get the same result for grading purposed, increment the seed by one.

            print('\n')

    return parameters_____no_output_____

Run the following cell, you should observe your model outputting random-looking characters at the first iteration. After a few thousand iterations, your model should learn to generate reasonable-looking names. _____no_output_____

parameters = model(data, ix_to_char, char_to_ix)Iteration: 0, Loss: 23.087336

Nkzxwtdmfqoeyhsqwasjkjvu
Kneb
Kzxwtdmfqoeyhsqwasjkjvu
Neb
Zxwtdmfqoeyhsqwasjkjvu
Eb
Xwtdmfqoeyhsqwasjkjvu

Iteration: 2000, Loss: 27.884160

Liusskeomnolxeros
Hmdaairus
Hytroligoraurus
Lecalosapaus
Xusicikoraurus
Abalpsamantisaurus
Tpraneronxeros

Iteration: 4000, Loss: 25.901815

Mivrosaurus
Inee
Ivtroplisaurus
Mbaaisaurus
Wusichisaurus
Cabaselachus
Toraperlethosdarenitochusthiamamumamaon

Iteration: 6000, Loss: 24.608779

Onwusceomosaurus
Lieeaerosaurus
Lxussaurus
Oma
Xusteonosaurus
Eeahosaurus
Toreonosaurus

Iteration: 8000, Loss: 24.070350

Onxusichepriuon
Kilabersaurus
Lutrodon
Omaaerosaurus
Xutrcheps
Edaksoje
Trodiktonus

Iteration: 10000, Loss: 23.844446

Onyusaurus
Klecalosaurus
Lustodon
Ola
Xusodonia
Eeaeosaurus
Troceosaurus

Iteration: 12000, Loss: 23.291971

Onyxosaurus
Kica
Lustrepiosaurus
Olaagrraiansaurus
Yuspangosaurus
Eealosaurus
Trognesaurus

Iteration: 14000, Loss: 23.382338

Meutromodromurus
Inda
Iutroinatorsaurus
Maca
Yusteratoptititan
Ca
Troclosaurus

Iteration: 16000, Loss: 23.268257

Mbutosaurus
Indaa
Iustolophulurus
Macagosaurus
Yusoclichaurus
Caahosaurus
Trodon

Iteration: 18000, Loss: 22.928870

Phytrogiaps
Mela
Mustrha
Pegamosaurus
Ytromacisaurus
Efanshie
Troma

Iteration: 20000, Loss: 23.008798

Onyusperchohychus
Lola
Lytrranfosaurus
Olaa
Ytrrcharomulus
Ehagosaurus
Trrcharonyhus

Iteration: 22000, Loss: 22.794515

Onyvus
Llecakosaurus
Mustodonosaurus
Ola
Yusodon
Eiadosaurus
Trodontorus

Iteration: 24000, Loss: 22.648635

Meutosaurus
Incaachudachus
Itntodon
Mecaessan
Yurong
Daadropachusaurus
Troenatheusaurosaurus

Iteration: 26000, Loss: 22.599152

Nixusehoenomulushapnelspanthuonathitalia
Jigaadroncansaurus
Kustodonis
Nedantrocantiteniupegyankuaeusalomarotimenmpangvin
Ytrodongoluctos
Eebdssaegoterichus
Trodolopiunsitarbilus

Iteration: 28000, Loss: 22.628455

Pnywrodilosaurus
Loca
Mustodonanethosaurus
Phabesceeatopsaurus
Ytrodonnoludosaurus
Elaishacaosaurus
Trrdilosaurus

Iteration: 30000, Loss: 22.587893

Piusosaurus
Locaadrus
Lutosaurus
Pacalosaurus
Yusochesaurus
Eg
Trraodon

Iteration: 32000, Loss: 22.314649

Nivosaurus
Jiacamisaurus
Kusplasaurus
Ncaadosaurus
Yusiandon
Eeaisilaanus
Trokalenator

Iteration: 34000, Loss: 22.445100

Mewsroengosaurus
Ilabafosaurus
Justoeomimavesaurus
Macaeosaurus
Yrosaurus
Eiaeosaurus
Trodondolus

## Conclusion

You can see that your algorithm has started to generate plausible dinosaur names towards the end of the training. At first, it was generating random characters, but towards the end you could see dinosaur names with cool endings. Feel free to run the algorithm even longer and play with hyperparameters to see if you can get even better results. Our implemetation generated some really cool names like `maconucon`, `marloralus` and `macingsersaurus`. Your model hopefully also learned that dinosaur names tend to end in `saurus`, `don`, `aura`, `tor`, etc.

If your model generates some non-cool names, don't blame the model entirely--not all actual dinosaur names sound cool. (For example, `dromaeosauroides` is an actual dinosaur name and is in the training set.) But this model should give you a set of candidates from which you can pick the coolest!

This assignment had used a relatively small dataset, so that you could train an RNN quickly on a CPU. Training a model of the english language requires a much bigger dataset, and usually needs much more computation, and could run for many hours on GPUs. We ran our dinosaur name for quite some time, and so far our favoriate name is the great, undefeatable, and fierce: Mangosaurus!

_____no_output_____## 4 - Writing like Shakespeare

The rest of this notebook is optional and is not graded, but we hope you'll do it anyway since it's quite fun and informative.

A similar (but more complicated) task is to generate Shakespeare poems. Instead of learning from a dataset of Dinosaur names you can use a collection of Shakespearian poems. Using LSTM cells, you can learn longer term dependencies that span many characters in the text--e.g., where a character appearing somewhere a sequence can influence what should be a different character much much later in ths sequence. These long term dependencies were less important with dinosaur names, since the names were quite short.

 Let's become poets!

We have implemented a Shakespeare poem generator with Keras. Run the following cell to load the required packages and models. This may take a few minutes. _____no_output_____

from __future__ import print_function
from keras.callbacks import LambdaCallback
from keras.models import Model, load_model, Sequential
from keras.layers import Dense, Activation, Dropout, Input, Masking
from keras.layers import LSTM
from keras.utils.data_utils import get_file
from keras.preprocessing.sequence import pad_sequences
from shakespeare_utils import *
import sys
import io_____no_output_____

To save you some time, we have already trained a model for ~1000 epochs on a collection of Shakespearian poems called [*"The Sonnets"*](shakespeare.txt). _____no_output_____Let's train the model for one more epoch. When it finishes training for an epoch---this will also take a few minutes---you can run `generate_output`, which will prompt asking you for an input (`<`40 characters). The poem will start with your sentence, and our RNN-Shakespeare will complete the rest of the poem for you! For example, try "Forsooth this maketh no sense " (don't enter the quotation marks). Depending on whether you include the space at the end, your results might also differ--try it both ways, and try other inputs as well.
_____no_output_____

print_callback = LambdaCallback(on_epoch_end=on_epoch_end)

model.fit(x, y, batch_size=128, epochs=1, callbacks=[print_callback])_____no_output_____# Run this cell to try with different inputs without having to re-train the model
generate_output()_____no_output_____

The RNN-Shakespeare model is very similar to the one you have built for dinosaur names. The only major differences are:
- LSTMs instead of the basic RNN to capture longer-range dependencies
- The model is a deeper, stacked LSTM model (2 layer)
- Using Keras instead of python to simplify the code

If you want to learn more, you can also check out the Keras Team's text generation implementation on GitHub: https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py.

Congratulations on finishing this notebook! _____no_output_____**References**:
- This exercise took inspiration from Andrej Karpathy's implementation: https://gist.github.com/karpathy/d4dee566867f8291f086. To learn more about text generation, also check out Karpathy's [blog post](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).
- For the Shakespearian poem generator, our implementation was based on the implementation of an LSTM text generator by the Keras team: https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py _____no_output_____