# Notebook from dibyanshushekhardey/Machine-Learning
Path: Google Cloud Data Science and Machine Learning/Perform Foundational Data, ML, and AI Tasks in Google Cloud/Reinforcement learning Qwik start/early_rl.ipynb

# Early Reinforcement Learning

With the advances of modern computing power, the study of Reinforcement Learning is having a heyday. Machines are now able to learn complex tasks once thought to be solely in the domain of humans, from controlling the [heating and cooling in massive data centers](https://www.technologyreview.com/s/611902/google-just-gave-control-over-data-center-cooling-to-an-ai/) to beating [grandmasters at Starcraft](https://storage.googleapis.com/deepmind-media/research/alphastar/AlphaStar_unformatted.pdf). As magnificent as it may seem today, it had humble roots many decades ago. Seeing how far it's come, it's a wonder to see how far it will go!

Let's take a step back in time to see how these early algorithms developed. Many of these algorithms make sense given the context of when they were created. Challenge yourself and see if you can come up with the same strategies given the right problem. Ok! Time to cozy up for a story.

This is the hero of our story, the gumdrop emoji. It was enjoying a cool winter day building a snowman when suddenly, it slipped and fell on a frozen lake of death.

The lake can be thought of as a 4 x 4 grid where the gumdrop can move left (0), down (1), right (2) and up (3). Unfortunately, this frozen lake of death has holes of death where if the gumdrop enters that square, it will fall in and meet an untimely demise. To make matters worse, the lake is surrounded by icy boulders that if the gumdrop attempts to climb, will have it slip back into its original position. Thankfully, at the bottom right of the lake is a safe ramp that leads to a nice warm cup of hot cocoa.

## Set Up

We can try and save the gumdrop ourselves! This is a common game people begin their Reinforcement Learning journey with, and is included in the OpenAI's python package [Gym](https://gym.openai.com/) and is aptly named [FrozenLake-v0](https://gym.openai.com/envs/FrozenLake-v0/) ([code](https://github.com/openai/gym/blob/master/gym/envs/toy_text/frozen_lake.py)). No time to waste, let's get the environment up and running. Run the below to install the needed libraries if they are not installed already._____no_output_____

# Ensure the right version of Tensorflow is installed.
!pip install tensorflow==2.1 --userRequirement already satisfied: tensorflow==2.1 in /home/jupyter/.local/lib/python3.7/site-packages (2.1.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (3.3.0)
Requirement already satisfied: protobuf>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (3.13.0)
Requirement already satisfied: absl-py>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (0.8.1)
Requirement already satisfied: astor>=0.6.0 in /home/jupyter/.local/lib/python3.7/site-packages (from tensorflow==2.1) (0.8.1)
Requirement already satisfied: keras-preprocessing>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (1.1.2)
Requirement already satisfied: tensorboard<2.2.0,>=2.1.0 in /home/jupyter/.local/lib/python3.7/site-packages (from tensorflow==2.1) (2.1.1)
Requirement already satisfied: six>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (1.15.0)
Requirement already satisfied: wrapt>=1.11.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (1.12.1)
Requirement already satisfied: grpcio>=1.8.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (1.32.0)
Requirement already satisfied: gast==0.2.2 in /home/jupyter/.local/lib/python3.7/site-packages (from tensorflow==2.1) (0.2.2)
Requirement already satisfied: keras-applications>=1.0.8 in /home/jupyter/.local/lib/python3.7/site-packages (from tensorflow==2.1) (1.0.8)
Requirement already satisfied: tensorflow-estimator<2.2.0,>=2.1.0rc0 in /home/jupyter/.local/lib/python3.7/site-packages (from tensorflow==2.1) (2.1.0)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (0.35.1)
Requirement already satisfied: scipy==1.4.1; python_version >= "3" in /home/jupyter/.local/lib/python3.7/site-packages (from tensorflow==2.1) (1.4.1)
Requirement already satisfied: google-pasta>=0.1.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (0.2.0)
Requirement already satisfied: numpy<2.0,>=1.16.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (1.18.5)
Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.1) (1.1.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.8.0->tensorflow==2.1) (50.3.0)
Requirement already satisfied: google-auth<2,>=1.6.3 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (1.21.1)
Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (1.0.1)
Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (3.2.2)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (0.4.1)
Requirement already satisfied: requests<3,>=2.21.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (2.24.0)
Requirement already satisfied: h5py in /opt/conda/lib/python3.7/site-packages (from keras-applications>=1.0.8->tensorflow==2.1) (2.10.0)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.5" in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (4.6)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (4.1.1)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (1.7.0)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (1.3.0)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (2020.6.20)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (1.25.10)
Requirement already satisfied: pyasn1>=0.1.3 in /opt/conda/lib/python3.7/site-packages (from rsa<5,>=3.1.4; python_version >= "3.5"->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (0.4.8)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (3.1.0)
Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1) (3.1.0)

**NOTE**: In the output of the above cell you may ignore any WARNINGS or ERRORS related to the following:  "witwidget", "pyarrow", "tensorflow-model-analysis", "tensorflow-data-validation" etc._____no_output_____If you get any related errors mentioned above please rerun the above cell._____no_output_____

!pip install gym==0.12.5 --userRequirement already satisfied: gym==0.12.5 in /home/jupyter/.local/lib/python3.7/site-packages (0.12.5)
Requirement already satisfied: pyglet>=1.2.0 in /home/jupyter/.local/lib/python3.7/site-packages (from gym==0.12.5) (1.5.7)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from gym==0.12.5) (1.15.0)
Requirement already satisfied: numpy>=1.10.4 in /opt/conda/lib/python3.7/site-packages (from gym==0.12.5) (1.18.5)
Requirement already satisfied: scipy in /home/jupyter/.local/lib/python3.7/site-packages (from gym==0.12.5) (1.4.1)

There are [four methods from Gym](http://gym.openai.com/docs/) that are going to be useful to us in order to save the gumdrop.
* `make` allows us to build the environment or game that we can pass actions to
* `reset` will reset an environment to it's starting configuration and return the state of the player
* `render` displays the environment for human eyes
* `step` takes an action and returns the player's next state.

Let's make, reset, and render the game. The output is an ANSI string with the following characters:
* `S` for starting point
* `F` for frozen
* `H` for hole
* `G` for goal
* A red square indicates the current position_____no_output_____**Note**: Restart the kernel if the above libraries needed to be installed
_____no_output_____

import gym
import numpy as np
import random
env = gym.make('FrozenLake-v0', is_slippery=False)
state = env.reset()
env.render()
SFFF
FHFH
FFFH
HFFG

If we print the state we'll get `0`. This is telling us which square we're in. Each square is labeled from `0` to `15` from left to right, top to bottom, like this:

| | | | |
|-|-|-|-|
|0|1|2|3|
|4|5|6|7|
|8|9|10|11|
|12|13|14|15|
_____no_output_____

print(state)0

We can make a simple print function to let us know whether it's game won, game over, or game on._____no_output_____

def print_state(state, done):
    statement = "Still Alive!"
    if done:
        statement = "Cocoa Time!" if state == 15 else "Game Over!"
    print(state, "-", statement)_____no_output_____

We can control the gumdrop ourselves with the `step` method. Run the below cell over and over again trying to move from the starting position to the goal. Good luck!_____no_output_____

#0 left
#1 down
#2 right
#3 up

# Uncomment to reset the game
env.reset()
action = 2  # Change me, please!
state, _, done, _ = env.step(action)
env.render()
print_state(state, done)  (Right)
SFFF
FHFH
FFFH
HFFG
1 - Still Alive!

Were you able to reach the hot chocolate? If so, great job! There are multiple paths through the maze. One solution is `[1, 1, 2, 2, 1, 2]`. Let's loop through our actions in order to get used to interacting with the environment programmatically._____no_output_____

def play_game(actions):
    state = env.reset()
    step = 0
    done = False

    while not done and step < len(actions):
        action = actions[step]
        state, _, done, _ = env.step(action)
        env.render()
        step += 1
        print_state(state, done)

actions = [1, 1, 2, 2, 1, 2]  # Replace with your favorite path.
play_game(actions)  (Down)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Right)
SFFF
FHFH
FFFH
HFFG
9 - Still Alive!
  (Right)
SFFF
FHFH
FFFH
HFFG
10 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
14 - Still Alive!
  (Right)
SFFF
FHFH
FFFH
HFFG
15 - Cocoa Time!

Nice, so we know how to get through the maze, but how do we teach that to the gumdrop? It's just some bytes in an android phone. It doesn't have our human insight.

We could give it our list of actions directly, but then it would be copying us and not really learning. This was a tricky one to the mathematicians and computer scientists originally trying to solve this problem. How do we teach a machine to do this without human insight?

## Value Iteration

Let's turn the clock back on our time machines to 1957 to meet Mr. [Richard Bellman](https://en.wikipedia.org/wiki/Richard_E._Bellman). Bellman started his academic career in mathematics, but due to World War II, left his postgraduate studies at John Hopkins to teach electronics as part of the war effort (as chronicled by J. J. O'Connor and E. F. Robertson [here](https://www-history.mcs.st-andrews.ac.uk/Biographies/Bellman.html)). When the war was over, and it came time for him to focus on his next area of research, he became fascinated with [Dynamic Programming](https://en.wikipedia.org/wiki/Dynamic_programming): the idea of breaking a problem down into sub-problems and using recursion to solve the larger problem.

Eventually, his research landed him on [Markov Decision Processes](https://en.wikipedia.org/wiki/Markov_decision_process). These processes are a graphical way of representing how to make a decision based on a current state. States are connected to other states with positive and negative rewards that can be picked up along the way.

Sound familiar at all? Perhaps our Frozen Lake?

In the lake case, each cell is a state. The `H`s and the `G` are a special type of state called a "Terminal State", meaning they can be entered, but they have no leaving connections. What of rewards? Let's say the value of losing our life is the negative opposite of getting to the goal and staying alive. Thus, we can assign the reward of entering a death hole as -1, and the reward of escaping as +1.

Bellman's first breakthrough with this type of problem is now known as Value Iteration ([his original paper](http://www.iumj.indiana.edu/IUMJ/FULLTEXT/1957/6/56038)). He introduced a variable, gamma (γ), to represent discounted future rewards. He also introduced a function of policy (π) that takes a state (s), and outputs corresponding suggested action (a). The goal is to find the value of a state (V), given the rewards that occur when following an action in a particular state (R).

Gamma, the discount, is the key ingredient here. If my time steps were in days, and my gamma was .9, `$100` would be worth `$100` to me today, `$90` tomorrow, `$81` the day after, and so on. Putting this all together, we get the Bellman Equation

source: [Wikipedia](https://en.wikipedia.org/wiki/Bellman_equation)

In other words, the value of our current state, `current_values`, is equal to the discount times the value of the next state, `next_values`, given the policy the agent will follow. For now, we'll have our agent assume a greedy policy: it will move towards the state with the highest calculated value. If you're wondering what P is, don't worry, we'll get to that later.

Let's program it out and see it in action! We'll set up an array representing the lake with -1 as the holes, and 1 as the goal. Then, we'll set up an array of zeros to start our iteration._____no_output_____

LAKE = np.array([[0,  0,  0,  0],
                 [0, -1,  0, -1],
                 [0,  0,  0, -1],
                 [-1, 0,  0,  1]])
LAKE_WIDTH = len(LAKE[0])
LAKE_HEIGHT = len(LAKE)

DISCOUNT = .9  # Change me to be a value between 0 and 1.
current_values = np.zeros_like(LAKE)_____no_output_____

The Gym environment class has a handy property for finding the number of states in an environment called `observation_space`. In our case, there a 16 integer states, so it will label it as "Discrete". Similarly, `action_space` will tell us how many actions are available to the agent.

Let's take advantage of these to make our code portable between different lakes sizes._____no_output_____

print("env.observation_space -", env.observation_space)
print("env.observation_space.n -", env.observation_space.n)
print("env.action_space -", env.action_space)
print("env.action_space.n -", env.action_space.n)

STATE_SPACE = env.observation_space.n
ACTION_SPACE = env.action_space.n
STATE_RANGE = range(STATE_SPACE)
ACTION_RANGE = range(ACTION_SPACE)env.observation_space - Discrete(16)
env.observation_space.n - 16
env.action_space - Discrete(4)
env.action_space.n - 4

We'll need some sort of function to figure out what the best neighboring cell is. The below function take's a cell of the lake, and looks at the current value mapping (to be called with `current_values`, and see's what the value of the adjacent state is corresponding to the given `action`._____no_output_____

def get_neighbor_value(state_x, state_y, values, action):
    """Returns the value of a state's neighbor.

    Args:
        state_x (int): The state's horizontal position, 0 is the lake's left.
        state_y (int): The state's vertical position, 0 is the lake's top.
        values (float array): The current iteration's state values.
        policy (int): Which action to check the value for.

    Returns:
        The corresponding action's value.
    """
    left = [state_y, state_x-1]
    down = [state_y+1, state_x]
    right = [state_y, state_x+1]
    up = [state_y-1, state_x]
    actions = [left, down, right, up]

    direction = actions[action]
    check_x = direction[1]
    check_y = direction[0]

    is_boulder = check_y < 0 or check_y >= LAKE_HEIGHT \
        or check_x < 0 or check_x >= LAKE_WIDTH

    value = values[state_y, state_x]
    if not is_boulder:
        value = values[check_y, check_x]

    return value_____no_output_____

But this doesn't find the best action, and the gumdrop is going to need that if it wants to greedily get off the lake. The `get_max_neighbor` function we've defined below takes a number corresponding to a cell as `state_number` and the same value mapping as `get_neighbor_value`._____no_output_____

def get_state_coordinates(state_number):
    state_x = state_number % LAKE_WIDTH
    state_y = state_number // LAKE_HEIGHT
    return state_x, state_y

def get_max_neighbor(state_number, values):
    """Finds the maximum valued neighbor for a given state.

    Args:
        state_number (int): the state to find the max neighbor for
        state_values (float array): the respective value of each state for
            each cell of the lake.

    Returns:
        max_value (float): the value of the maximum neighbor.
        policy (int): the action to take to move towards the maximum neighbor.
    """
    state_x, state_y = get_state_coordinates(state_number)

    # No policy or best value yet
    best_policy = -1
    max_value = -np.inf

    # If the cell has something other than 0, it's a terminal state.
    if LAKE[state_y, state_x]:
        return LAKE[state_y, state_x], best_policy

    for action in ACTION_RANGE:
        neighbor_value = get_neighbor_value(state_x, state_y, values, action)
        if neighbor_value > max_value:
            max_value = neighbor_value
            best_policy = action

    return max_value, best_policy_____no_output_____

Now, let's write our value iteration code. We'll write a function that comes out one step of the iteration by checking each state and finding its maximum neighbor. The values will be reshaped so that it's in the form of the lake, but the policy will stay as a list of ints. This way, when Gym returns a state, all we need to do is look at the corresponding index in the policy list to tell our agent where to go._____no_output_____

def iterate_value(current_values):
    """Finds the future state values for an array of current states.

    Args:
        current_values (int array): the value of current states.

    Returns:
        next_values (int array): The value of states based on future states.
        next_policies (int array): The recommended action to take in a state.
    """
    next_values = []
    next_policies = []

    for state in STATE_RANGE:
        value, policy = get_max_neighbor(state, current_values)
        next_values.append(value)
        next_policies.append(policy)

    next_values = np.array(next_values).reshape((LAKE_HEIGHT, LAKE_WIDTH))
    return next_values, next_policies

next_values, next_policies = iterate_value(current_values)_____no_output_____

This is what our values look like after one step. Right now, it just looks like the lake. That's because we started with an array of zeros for `current_values`, and the terminal states of the lake were loaded in._____no_output_____

next_values_____no_output_____

And this is what our policy looks like reshaped into the form of the lake. The `-1`'s are terminal states. Right now, the agent will move left in any non-terminal state, because it sees all of those states as equal. Remember, if the gumdrop is along the leftmost side of the lake, and tries to move left, it will slip on a boulder and return to the same position._____no_output_____

np.array(next_policies).reshape((LAKE_HEIGHT ,LAKE_WIDTH))_____no_output_____

There's one last step to apply the Bellman Equation, the `discount`! We'll multiply our next states by the `discount` and set that to our `current_values`. One loop done!_____no_output_____

current_values = DISCOUNT * next_values
current_values_____no_output_____

Run the below cell over and over again to see how our values change with each iteration. It should be complete after six iterations when the values no longer change. The policy will also change as the values are updated._____no_output_____

next_values, next_policies = iterate_value(current_values)
print("Value")
print(next_values)
print("Policy")
print(np.array(next_policies).reshape((4,4)))
current_values = DISCOUNT * next_valuesValue
[[ 0.   0.   0.   0. ]
 [ 0.  -1.   0.  -1. ]
 [ 0.   0.   0.  -1. ]
 [-1.   0.   0.9  1. ]]
Policy
[[ 0  0  0  0]
 [ 0 -1  1 -1]
 [ 0  0  0 -1]
 [-1  1  2 -1]]

Have a completed policy? Let's see it in action! We'll update our `play_game` function to instead take our list of policies. That way, we can start in a random position and still get to the end._____no_output_____

def play_game(policy):
    state = env.reset()
    step = 0
    done = False

    while not done:
        action = policy[state]  # This line is new.
        state, _, done, _ = env.step(action)
        env.render()
        step += 1
        print_state(state, done)

play_game(next_policies)  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Game Over!

Phew! Good job, team! The gumdrop made it out alive. So what became of our gumdrop hero? Well, the next day, it was making another snowman and fell onto an even more slippery and deadly lake. Doh! Turns out this story is part of a trilogy. Feel free to move onto the next section after your own sip of cocoa, coffee, tea, or poison of choice.

## Policy Iteration

You may have noticed that the first lake was built with the parameter `is_slippery=False`. This time, we're going to switch it to `True`._____no_output_____

env = gym.make('FrozenLake-v0', is_slippery=True)
state = env.reset()
env.render()
SFFF
FHFH
FFFH
HFFG

Hmm, looks the same as before. Let's try applying our old policy and see what happens._____no_output_____

play_game(next_policies)  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
12 - Game Over!

Was there a game over? There's a small chance that the gumdrop made it to the end, but it's much more likely that it accidentally slipped and fell into a hole. Oh no! We can try repeatedly testing the above code cell over and over again, but it might take a while. In fact, this is a similar roadblock Bellman and his colleagues faced.

How efficient is Value Iteration? On our modern machines, this algorithm ran fairly quickly, but back in 1960, that wasn't the case. Let's say our lake is a long straight line like this:

| | | | | | | |
|-|-|-|-|-|-|-|
|S|F|F|F|F|F|H|

This is the worst case scenario for value iteration. In each iteration, we look at every state (s) and each action per state (a), so one step of value iteration is O(s*a). In the case of our lake line, each iteration only updates one cell. In other words, the value iteration step needs to be run `s` times. In total, that's O(s2a).

Back in 1960, that was computationally heavy, and so [Ronald Howard](https://en.wikipedia.org/wiki/Ronald_A._Howard) developed an alteration of Value Iteration that mildly sacrificed mathematical accuracy for speed.

Here's the strategy: it was observed that the optimal policy often converged before value iteration was complete. To take advantage of this, we'll start with random policy. When we iterate over our values, we'll use this policy instead of trying to find the maximum neighbor. This has been coded out in `find_future_values` below._____no_output_____

def find_future_values(current_values, current_policies):
    """Finds the next set of future values based on the current policy."""
    next_values = []

    for state in STATE_RANGE:
        current_policy = current_policies[state]
        state_x, state_y = get_state_coordinates(state)

        # If the cell has something other than 0, it's a terminal state.
        value = LAKE[state_y, state_x]
        if not value:
            value = get_neighbor_value(
                state_x, state_y, current_values, current_policy)
        next_values.append(value)

    return np.array(next_values).reshape((LAKE_HEIGHT, LAKE_WIDTH))_____no_output_____

After we've calculated our new values, then we'll update the policy (and not the values) based on the maximum neighbor. If there's no change in the policy, then we're done. The below is very similar to our `get_max_neighbor` function. Can you see the differences?_____no_output_____

def find_best_policy(next_values):
    """Finds the best policy given a value mapping."""
    next_policies = []
    for state in STATE_RANGE:
        state_x, state_y = get_state_coordinates(state)

        # No policy or best value yet
        max_value = -np.inf
        best_policy = -1

        if not LAKE[state_y, state_x]:
            for policy in ACTION_RANGE:
                neighbor_value = get_neighbor_value(
                    state_x, state_y, next_values, policy)
                if neighbor_value > max_value:
                    max_value = neighbor_value
                    best_policy = policy

        next_policies.append(best_policy)
    return next_policies_____no_output_____

To complete the Policy Iteration algorithm, we'll combine the two functions above. Conceptually, we'll be alternating between updating our value function and updating our policy function._____no_output_____

def iterate_policy(current_values, current_policies):
    """Finds the future state values for an array of current states.

    Args:
        current_values (int array): the value of current states.
        current_policies (int array): a list where each cell is the recommended
            action for the state matching its index.

    Returns:
        next_values (int array): The value of states based on future states.
        next_policies (int array): The recommended action to take in a state.
    """
    next_values = find_future_values(current_values, current_policies)
    next_policies = find_best_policy(next_values)
    return next_values, next_policies_____no_output_____

Next, let's modify the `get_neighbor_value` function to now include the slippery ice. Remember the `P` in the Bellman Equation above? It stands for the probability of ending up in a new state given the current state and action taken. That is, we'll take a weighted sum of the values of all possible states based on our chances to be in those states.

How does the physics of the slippery ice work? For this lake, whenever the gumdrop tries to move in a particular direction, there are three possible positions that it could end up with. It could move where it was intending to go, but it could also end up to the left or right of the direction it was facing. For instance, if it wanted to move right, it could end up on the square above or below it! This is depicted below, with the yellow squares being potential positions after attempting to move right.

Each of these has an equal probability chance of happening. So since there are three outcomes, they each have about a 33% chance to happen. What happens if we slip in the direction of a boulder? No problem, we'll just end up not moving anywhere. Let's make a function to find what our possible locations could be given a policy and state coordinates._____no_output_____

def get_locations(state_x, state_y, policy):
    left = [state_y, state_x-1]
    down = [state_y+1, state_x]
    right = [state_y, state_x+1]
    up = [state_y-1, state_x]
    directions = [left, down, right, up]
    num_actions = len(directions)

    gumdrop_right = (policy - 1) % num_actions
    gumdrop_left = (policy + 1) % num_actions
    locations = [gumdrop_left, policy, gumdrop_right]
    return [directions[location] for location in locations]_____no_output_____

Then, we can add it to `get_neighbor_value` to find the weighted value of all the possible states the gumdrop can end up in._____no_output_____

def get_neighbor_value(state_x, state_y, values, policy):
    """Returns the value of a state's neighbor.

    Args:
        state_x (int): The state's horizontal position, 0 is the lake's left.
        state_y (int): The state's vertical position, 0 is the lake's top.
        values (float array): The current iteration's state values.
        policy (int): Which action to check the value for.

    Returns:
        The corresponding action's value.
    """
    locations = get_locations(state_x, state_y, policy)
    location_chance = 1.0 / len(locations)
    total_value = 0

    for location in locations:
        check_x = location[1]
        check_y = location[0]

        is_boulder = check_y < 0 or check_y >= LAKE_HEIGHT \
            or check_x < 0 or check_x >= LAKE_WIDTH

        value = values[state_y, state_x]
        if not is_boulder:
            value = values[check_y, check_x]
        total_value += location_chance * value

    return total_value_____no_output_____

For Policy Iteration, we'll start off with a random policy if only because the Gumdrop doesn't know any better yet. We'll reset our current values while we're at it._____no_output_____

current_values = np.zeros_like(LAKE)
policies = np.random.choice(ACTION_RANGE, size=STATE_SPACE)
np.array(policies).reshape((4,4))_____no_output_____

As before with Value Iteration, run the cell below multiple until the policy no longer changes. It should only take 2-3 clicks compared to Value Iteration's 6._____no_output_____

next_values, policies = iterate_policy(current_values, policies)
print("Value")
print(next_values)
print("Policy")
print(np.array(policies).reshape((4,4)))
current_values = DISCOUNT * next_valuesValue
[[ 0.  0.  0.  0.]
 [ 0. -1.  0. -1.]
 [ 0.  0.  0. -1.]
 [-1.  0.  0.  1.]]
Policy
[[ 0  3  0  3]
 [ 0 -1  0 -1]
 [ 3  1  0 -1]
 [-1  2  1 -1]]

Hmm, does this work? Let's see! Run the cell below to watch the gumdrop slip its way to victory._____no_output_____

play_game(policies)  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
9 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
10 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
9 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
13 - Still Alive!
  (Right)
SFFF
FHFH
FFFH
HFFG
13 - Still Alive!
  (Right)
SFFF
FHFH
FFFH
HFFG
9 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
9 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
9 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
10 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
14 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
15 - Cocoa Time!

So what was the learned strategy here? The gumdrop learned to hug the left wall of boulders until it was down far enough to make a break for the exit. Instead of heading directly for it though, it took advantage of actions that did not have a hole of death in them. Patience is a virtue!

We promised this story was a trilogy, and yes, the next day, the gumdrop fell upon a frozen lake yet again.

## Q Learning
Value Iteration and Policy Iteration are great techniques, but what if we don't know how big the lake is? With real world problems, not knowing how many potential states are can be a definite possibility.

Enter [Chris Watkins](http://www.cs.rhul.ac.uk/~chrisw/). Inspired by how animals learn with delayed rewards, he came up with the idea of [Q Learning](http://www.cs.rhul.ac.uk/~chrisw/new_thesis.pdf) as an evolution of [Richard Sutton's](https://en.wikipedia.org/wiki/Richard_S._Sutton) [Temporal Difference Learning](https://en.wikipedia.org/wiki/Temporal_difference_learning). Watkins noticed that animals learn from positive and negative rewards, and that they often make mistakes in order to optimize a skill.

From this emerged the idea of a Q table. In the lake case, it would look something like this.

| |Left|Down|Right|Up|
|-|-|-|-|-|
|0| | | | |
|1| | | | |
|...| | | | |

Here's the strategy: our agent will explore the environment. As the agent observes new states, we'll add more rows to our table. Whenever it moves from one state to the next, we'll update the cell corresponding to the old state based on the Bellman Equation. The agent doesn't need to know what the probabilities are between transitions. It'll learn the value of these as it experiments.

For Q learning, this works by looking at the row that corresponds to the agent's current state. Then, we'll select the action with the highest value. There are multiple ways to initialize the Q-table, but for us, we'll start with all zeros. In that case, when selecting the best action, we'll randomly select between tied max values. If we don't, the agent will favor certain actions which will limit its exploration.

To be able to handle an unknown number of states, we'll initialize our q_table as one row to represent our initial state. Then, we'll make a dictionary to map new states to rows in the table._____no_output_____

new_row = np.zeros((1, env.action_space.n))
q_table = np.copy(new_row)
q_map = {0: 0}

def print_q(q_table, q_map):
    print("mapping")
    print(q_map)
    print("q_table")
    print(q_table)

print_q(q_table, q_map)mapping
{0: 0}
q_table
[[0. 0. 0. 0.]]

Our new `get_action` function will help us read the `q_table` and find the best action.

First, we'll give the agent the ability to act randomly as opposed to choosing the best known action. This gives it the ability to explore and find new situations. This is done with a random chance to act randomly. So random!

When the Gumdrop chooses not to act randomly, it will instead act based on the best action recorded in the `q_table`. Numpy's [argwhere](https://docs.scipy.org/doc/numpy/reference/generated/numpy.argwhere.html) is used to find the indexes with the maximum value in the q-table row corresponding to our current state. Since numpy is often used with higher dimensional data, each index is returned as a list of ints. Our indexes are really one dimensional since we're just looking within a single row, so we'll use [np.squeeze](https://docs.scipy.org/doc/numpy/reference/generated/numpy.squeeze.html) to remove the extra brackets. To randomly select from the indexes, we'll use [np.random.choice](https://docs.scipy.org/doc/numpy-1.14.1/reference/generated/numpy.random.choice.html)._____no_output_____

def get_action(q_map, q_table, state_row, random_rate):
    """Find max-valued actions and randomly select from them."""
    if random.random() < random_rate:
        return random.randint(0, ACTION_SPACE-1)

    action_values = q_table[state_row]
    max_indexes = np.argwhere(action_values == action_values.max())
    max_indexes = np.squeeze(max_indexes, axis=-1)
    action = np.random.choice(max_indexes)
    return action_____no_output_____

Here, we'll define how the `q_table` gets updated. We'll apply the Bellman Equation as before, but since there is so much luck involved between slipping and random actions, we'll update our `q_table` as a weighted average between the `old_value` we're updating and the `future_value` based on the best action in the next state. That way, there's a little bit of memory between old and new experiences._____no_output_____

def update_q(q_table, new_state_row, reward, old_value):
    """Returns an updated Q-value based on the Bellman Equation."""
    learning_rate = .1  # Change to be between 0 and 1.
    future_value = reward + DISCOUNT * np.max(q_table[new_state_row])
    return old_value + learning_rate * (future_value - old_value)_____no_output_____

We'll update our `play_game` function to take our table and mapping, and at the end, we'll return any updates to them. Once we observe new states, we'll check our mapping and add then to the table if space isn't allocated for them already.

Finally, for every `state` - `action` - `new-state` transition, we'll update the cell in `q_table` that corresponds to the `state` and `action` with the Bellman Equation.

There's a little secret to solving this lake problem, and that's to have a small negative reward when moving between states. Otherwise, the gumdrop will become too afraid of slipping in a death hole to explore out of what is thought to be safe positions._____no_output_____

def play_game(q_table, q_map, random_rate, render=False):
    state = env.reset()
    step = 0
    done = False

    while not done:
        state_row = q_map[state]
        action = get_action(q_map, q_table, state_row, random_rate)
        new_state, _, done, _ = env.step(action)

        #Add new state to table and mapping if it isn't there already.
        if new_state not in q_map:
            q_map[new_state] = len(q_table)
            q_table = np.append(q_table, new_row, axis=0)
        new_state_row = q_map[new_state]

        reward = -.01  #Encourage exploration.
        if done:
            reward = 1 if new_state == 15 else -1
        current_q = q_table[state_row, action]
        q_table[state_row, action] = update_q(
            q_table, new_state_row, reward, current_q)

        step += 1
        if render:
            env.render()
            print_state(new_state, done)
        state = new_state

    return q_table, q_map_____no_output_____

Ok, time to shine, gumdrop emoji! Let's do one simulation and see what happens._____no_output_____

# Run to refresh the q_table.
random_rate = 1
q_table = np.copy(new_row)
q_map = {0: 0}_____no_output_____q_table, q_map = play_game(q_table, q_map, random_rate, render=True)
print_q(q_table, q_map)  (Right)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
1 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Right)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Down)
SFFF
FHFH
FFFH
HFFG
5 - Game Over!
mapping
{0: 0, 4: 1, 1: 2, 5: 3}
q_table
[[ 0.      -0.00271 -0.0019  -0.001  ]
 [-0.001   -0.1      0.      -0.001  ]
 [ 0.       0.       0.      -0.001  ]
 [ 0.       0.       0.       0.     ]]

Unless the gumdrop was incredibly lucky, chances were, it fell in some death water. Q-learning is markedly different from Value Iteration or Policy Iteration in that it attempts to simulate how an animal learns in unknown situations. Since the layout of the lake is unknown to the Gumdrop, it doesn't know which states are death holes, and which ones are safe. Because of this, it's going to make many mistakes before it can start making successes.

Feel free to run the above cell multiple times to see how the gumdrop steps through trial and error. When you're ready, run the below cell to have the gumdrop play 1000 times._____no_output_____

for _ in range(1000):
    q_table, q_map = play_game(q_table, q_map, random_rate)
    random_rate = random_rate * .99
print_q(q_table, q_map)
random_ratemapping
{0: 0, 4: 1, 1: 2, 5: 3, 2: 4, 3: 5, 6: 6, 8: 7, 12: 8, 9: 9, 13: 10, 7: 11, 10: 12, 14: 13, 11: 14, 15: 15}
q_table
[[-0.17859462 -0.21683258 -0.22707618 -0.22202052]
 [-0.06506833 -0.34027509 -0.36022658 -0.40212942]
 [-0.57599308 -0.40139265 -0.36359524 -0.08915388]
 [ 0.          0.          0.          0.        ]
 [-0.2917146  -0.24777764 -0.26932154 -0.09538552]
 [-0.26724347 -0.34082535 -0.29285415 -0.20776494]
 [-0.66344346 -0.67117004 -0.44894497 -0.67412802]
 [-0.41098224 -0.34409584 -0.23580235 -0.02739325]
 [ 0.          0.          0.          0.        ]
 [-0.27323516  0.11283202 -0.28284193 -0.40023591]
 [-0.3383272  -0.06760574  0.31610858 -0.04919265]
 [ 0.          0.          0.          0.        ]
 [ 0.16083802 -0.36816045 -0.32278753 -0.37812964]
 [ 0.07261658  0.63570504 -0.00396321  0.13897806]
 [ 0.          0.          0.          0.        ]
 [ 0.          0.          0.          0.        ]]

Cats have nine lives, our Gumdrop lived a thousand! Moment of truth. Can it get out of the lake now that it matters?_____no_output_____

q_table, q_map = play_game(q_table, q_map, 0, render=True)  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
8 - Still Alive!
  (Up)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
4 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Still Alive!
  (Left)
SFFF
FHFH
FFFH
HFFG
0 - Game Over!

Third time's the charm!

Each of these techniques has its pros and cons. For instance, while Value Iteration is the mathematically correct solution, it's not as time efficient at Policy Iteration or as flexible as Q-Learning.

| |Value Iteration|Policy Iteration|Q Tables|
|-|-|-|-|
|Avoids locally optimal routes|✓|x|x|
|On-policy (greedy)|✓|✓|x|
|Model Free|x|x|✓|
|Most time efficient|x|✓|x|

Congratulations on making it through to the end. Now if you ever fall on a Frozen Lake, you'll have many different ways to calculate your survival. The gumdrop thank you!

_____no_output_____Copyright 2020 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License._____no_output_____


# Notebook from duderstadt-lab/Born-to-slide
Path: Analysis_software/Figures_1_S1/Figures_1H_S1J+K.ipynb

# Born to slide: mobile origin licensing factors confer resistance to conflicts with RNA polymerase

## Jupyter notebooks

#### Figures:
+ 1H
+ S1J
+ S1K

#### Archives:
+ MCM_high-salt_diffusion.yama

#### Additional data:
+ none_____no_output_____### Initialize ImageJ_____no_output_____

import imagej
ij = imagej.init('/Applications/Fiji.app')
ij.getVersion()_____no_output_____

### Imports_____no_output_____

import sys
sys.path.insert(0, '..')
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import re
import awesome_data
from marspy.convert.archive import DnaMoleculeArchive, instantiate_archive, describe_archives
from marspy.stats import bootstrap, calc_ci
from tqdm.notebook import tqdm
from functools import reducemarspy initialized.

### Select archives_____no_output_____

selected_archives = ['MCM_high-salt_diffusion.yama']_____no_output_____

### Figure style and output directory_____no_output_____

from figure_style import *_____no_output_____set_style_paper()_____no_output_____dir_out = '/Volumes/pool-duderstadt/Matthias/Manuscript/Figures/SVGs/Figure1/'
dir_out_____no_output_____

### Instantiate selected archives_____no_output_____

for archive in tqdm(selected_archives):
    instantiate_archive(archive, awesome_data.datasets)_____no_output_____

### Archives Overview_____no_output_____

describe_archives(DnaMoleculeArchive.instances)_____no_output_____

### Functions_____no_output_____

def calc_msd(df, prefix, msd_col, time_col, tau=1, start=np.nan, end=np.nan):
    '''
    Calculates mean square displacement of molecules.
    Remove pairs with missing slices in interval, do not drop NaN values!
    *df: dataframe
    *prefix: protein prefix
    *msd_col: column in df to calculate msd (without protein prefix!)
    *time_col: time column in df (without protein prefix!)
    *tau: time delta used to calculate msd (without protein prefix!)

    returns (msd, dtime)
    msd: mean squared error
    dtime: mean time difference underlying msd calculation
    '''
    # if not start / end was specified by used, set to max range
    if np.isnan(start):
        start=df[prefix+time_col].iloc[0]

    if np.isnan(end):
        end=df[prefix+time_col].iloc[-1]

    # will be a more accurate way to calculate MSD due to existing NaN values
    diffs = []
    dtimes = []
    for row in df.index:
            try:
                # check if sliding window range
                if (df.loc[row, prefix+time_col] >= start and df.loc[row+tau, prefix+time_col] <= end):

                    # check if we have consecutive frames (slices) which are not null
                    if (((df.loc[row+tau, prefix+'slice'] - df.loc[row, prefix+'slice']) == tau) and not
                        ((np.isnan(df.loc[row+tau, prefix+msd_col]) or np.isnan(df.loc[row, prefix+msd_col])))):

                        diffs.append(df.loc[row+tau, prefix+msd_col] - df.loc[row, prefix+msd_col])
                        dtimes.append(df.loc[row+tau, prefix+time_col] - df.loc[row, prefix+time_col])

                    else:
                        #skip to next row
                        continue

            # end of df
            except KeyError:
                break

    return np.mean(np.square(diffs)), np.mean(dtimes)_____no_output_____

### Generate main df_____no_output_____

def setup_pandas_df(archive_instances):
    '''Sets up a 2-level df with all relevant information'''
    #indices will be UIDs
    #outside: set comprehension to cover all possible prefixes in all instances
    prefixes = set()
    for archive in archive_instances:
        prefixes.update(archive.prefixes)
    col_1 = sorted(list(prefixes))
    #inside
    col_2 = ['lifetime', 'initial_intensity', 'sliding_start', 'sliding_end', 'msd', 'dtime', 'd_coeff']

    hier_index = pd.MultiIndex.from_product([col_1,col_2],names=['molecule','properties'])

    return pd.DataFrame(columns=hier_index)
    _____no_output_____df = setup_pandas_df(DnaMoleculeArchive.instances)

# sort MultiIndexCols for performance
df.sort_index(axis=1, inplace=True)
df.head()
_____no_output_____# fill dataframe
for archive in tqdm(DnaMoleculeArchive.instances):

    for molecule in archive.molecules:
        # general columns first
        # this could be generalized with mol prefixes
        df.loc[molecule.uid,'number_mcm'] = molecule.params['Number_MCM']
        df.loc[molecule.uid,'tags'] = reduce(lambda tag1,tag2: tag1+','+tag2, molecule.tags)
        df.loc[molecule.uid,'nucleotide'] = archive.nucleotide
        df.loc[molecule.uid,'dna_length'] = molecule.calc_length_dna()

        for prefix in molecule.prefixes:

            # lifetime (in frames) defined on tracking length
            df.loc[molecule.uid,(prefix,'lifetime')] = len(molecule.df[prefix+'y'].dropna())

            # take intensity from mean of first 20 frames
            df.loc[molecule.uid,(prefix,'initial_intensity')] = molecule.df[prefix+'Intensity'].iloc[0:20].mean()

            # a specific region was defined
            if 'sliding_'+prefix in list(filter(lambda region: 'sliding' in region.name, molecule.regions))[0].name:

                # temporary variables
                start = archive.archive_link.get(molecule.uid).getRegion('sliding_'+prefix).getStart()
                end = archive.archive_link.get(molecule.uid).getRegion('sliding_'+prefix).getEnd()

                # sliding window start & end
                df.loc[molecule.uid,(prefix,'sliding_start')] = start
                df.loc[molecule.uid,(prefix,'sliding_end')] = end

                # MSD (kbp^2) & dtime (s)

                msd, dtime = calc_msd(df=molecule.df, prefix=prefix, msd_col='Position_on_DNA', time_col='Time (s)',
                                      tau=1, start=start, end=end)
                df.loc[molecule.uid,(prefix,'msd')] = msd/(1000**2)
                df.loc[molecule.uid,(prefix,'dtime')] = dtime

                # diffusion coefficient (kpb^2/s)
                df.loc[molecule.uid,(prefix,'d_coeff')] = df.loc[molecule.uid,(prefix,'msd')] / (2*df.loc[molecule.uid,(prefix,'dtime')])

            #no region => molecule was not sliding
            else:
                pass

# convert tags back to list
df['tags'] = df['tags'].apply(lambda tags: tags.split(','))

# again sort MultiIndexCols for performance
df.sort_index(axis=1, inplace=True)

# fix data types
df = df.infer_objects()
_____no_output_____df.head()_____no_output_____

## Figures_____no_output_____### Figure 1H: Diffusion coefficient salt-sliding MCMs_____no_output_____

protein = 'MCM'

fig,axes = plt.subplots(figsize=(0.5,2))

sns.barplot(y=df['MCM_1_']['d_coeff'],palette=palettes[protein][1:2], lw=1 , edgecolor='.15', capsize=.3,facecolor=(1, 1, 1, 0),
            errwidth=1,ax=axes,n_boot=10000, seed=42, ci=68, errcolor='.15')
sns.swarmplot(y=df['MCM_1_']['d_coeff'],palette=palettes[protein][1:2],ax=axes)

axes.set_ylim(0,50)
#axes.set_title('Observed MCM diffusion in 0.5 M NaCl')
axes.set_ylabel('D (kbp2/s)')
axes.text(0.5, 0.95, f"n = {len(df['MCM_1_']['d_coeff'])}",
        verticalalignment='top', horizontalalignment='center',
        transform=axes.transAxes)
axes.text(0.6, 0.7, f"D = {round(df['MCM_1_']['d_coeff'].mean(),1)} +/- \n{round(df['MCM_1_']['d_coeff'].sem(),1)} kbp2/s",
        verticalalignment='top', horizontalalignment='left',
        transform=axes.transAxes)
axes.set_xlabel('MCM diffusion\ncoefficient')
# tidy up
#plt.tight_layout()
sns.despine()

# save
plt.savefig(dir_out+"1H_MCM_diffusionCoeff.svg",transparent=True)_____no_output_____

### Figure S1J: Probability of MCM sliding in high salt_____no_output_____

import scyjava as sc_____no_output_____total_mcm_count = 0
for metadata in DnaMoleculeArchive.instances[0].metadata_uids:
    total_mcm_count += dict(sc.to_python(DnaMoleculeArchive.instances[0].archive_link.getMetadata(metadata).
                                         getParameters()))['TotalMcmCount']
total_mcm_count_____no_output_____df_pslide = pd.DataFrame(data={'non-diffusive':(total_mcm_count-len(DnaMoleculeArchive.instances[0])) / total_mcm_count,
                               'diffusive':len(DnaMoleculeArchive.instances[0]) / total_mcm_count,
                               'total':total_mcm_count}.items(),
                         columns = ['population','count'])
df_pslide.set_index('population',inplace=True)_____no_output_____df_pslide_____no_output_____protein = 'MCM'
fig,axes = plt.subplots()

df_pslide.iloc[0:-1].T.plot(kind='bar',stacked=True, figsize=(0.5,2),ax=axes,color=palettes['qualitative'][0::2],lw=0.5,edgecolor='.15')
axes.set_ylabel("Probability")
#axes.set_title("MCM populations")
axes.set_ylim([0,1.19])
axes.legend(title='MCM population',frameon=False, bbox_to_anchor=[1, 1], loc='upper left')
for item in axes.get_xticklabels():
    item.set_rotation(45)

labels = [int(df_pslide.loc['total',column]) for column in df_pslide.columns]

#calculate y offset to label bars on top
dy = []
for column in df_pslide.columns:
    temp_sum = 0
    for index,row in df_pslide.iterrows():
        if (index == 'total'):
            continue
        else:
            temp_sum+=df_pslide.loc[index,column]
    dy.append(temp_sum)
rects = axes.patches
counter = 0
for rect, label in zip(rects, labels):
    height = dy[counter]
    axes.text(rect.get_x() + rect.get_width()/2, height+0.05, s=f"n = {label}", ha='center', va='bottom')
    axes.text(rect.get_x() + rect.get_width()/2, 0.45, s=f"{round(df_pslide.loc['non-diffusive','count']*100,1)} %", ha='center', va='bottom')
    axes.text(rect.get_x() + rect.get_width()/2, 0.95, s=f"{round(df_pslide.loc['diffusive','count']*100,1)} %", ha='center', va='bottom')
    counter+=1

axes.set_xticklabels([])
#tidy up
#fig.tight_layout()
sns.despine()

#save
plt.savefig(dir_out+"S1J_MCM_Populations.svg",transparent=True)_____no_output_____# estimate standard error by bootstrapping (skip)_____no_output_____df_pslide.columns = ['probability']
df_pslide_____no_output_____for row in df_pslide.index[:-1]:
    bootstrap_data = ([0 for _ in range(int((1-df_pslide.loc[row,'probability'])*df_pslide.loc['total','probability']))] +
                          [1 for _ in range(int(df_pslide.loc[row,'probability']*df_pslide.loc['total','probability']))])
    bootstrap_means = bootstrap(bootstrap_data,n_boot=10000)
    df_pslide.loc[row,'bs_mean'] = np.mean(bootstrap_means)
    df_pslide.loc[row,'standard_error'] = np.std(bootstrap_means)
    df_pslide.loc[row,'68_ci_lower'], df_pslide.loc[row,'68_ci_upper'] = calc_ci(bootstrap_means,ci=68)
    df_pslide.loc[row,'95_ci_lower'], df_pslide.loc[row,'95_ci_upper'] = calc_ci(bootstrap_means,ci=95)_____no_output_____df_pslide_____no_output_____

### Figure S1K: MSD vs dtime_____no_output_____

# Generate separate dfs with MSD for different tau
dfs_msd = []
for molecule in DnaMoleculeArchive.instances[0].molecules:
    df_temp = pd.DataFrame()
    for i in range(0,11):

        msd, dtime = calc_msd(molecule.df, prefix='MCM_1_',
                              msd_col='Position_on_DNA', time_col='Time (s)',
                              tau=i, start=df.loc[molecule.uid,'MCM_1_']['sliding_start'],
                              end=df.loc[molecule.uid,'MCM_1_']['sliding_end'])
        df_temp.loc[i,'dtime'] = dtime
        df_temp.loc[i,'msd'] = msd/(1000**2)
    dfs_msd.append(df_temp)_____no_output_____sns.set_palette(palettes['MCM'])_____no_output_____protein = 'MCM'
fig,axes = plt.subplots()

for item in dfs_msd[:]:
    sns.lineplot(x='dtime', y='msd', data=item, ax=axes)
axes.set_xlim(0,1)
axes.set_ylim(0,70)
#axes.set_title('MSD vs $\Delta$ t')
axes.set_xlabel('dt (s)')
axes.set_ylabel('MSD (kbp2)')
axes.text(0.05, 0.95, f'n = {len(dfs_msd[:])}',
        verticalalignment='top', horizontalalignment='left',
        transform=axes.transAxes)

sns.despine(trim=True)

#save
plt.savefig(dir_out+"S1K_MSD_vs_time.svg",transparent=True)
_____no_output_____


# Notebook from leouieda/lithosphere
Path: practicals/practical5.ipynb

# Investigations of the lithosphere

**Goal:** Bring together all that we've learned so far to investigate the state of the lithosphere is different parts of the world.

So far we've learned how to:

1. Assess the overall state of isostatic equilibrium of the lithosphere using topography/bathymetry and gravity data
1. Model the thermal evolution of the oceanic lithosphere and compare model predictions against heat flow and bathymetry data
1. Model the bending of the lithosphere at subduction zones and make predictions of bathymetry and gravity disturbances along profiles
1. Model the intra-plate bending of the lithosphere under the weight of oceanic islands and seamounts and compare model predictions against observed gravity disturbances

In this practical, you will:

1. Investigate a particular region of the Earth using global topography/bathymetry, age, heat flow, and gravity data
1. Use all of the tools available to you to describe the overall state of the lithosphere (isostic equilibrium, flexure, thickness, age, fit to models, etc)
1. Work in groups to come up with a summary of your understanding of the lithosphere in that particular region_____no_output_____## The data

**Download** (if you haven't already): [global-geophysical-data.nc](https://github.com/leouieda/lithosphere/raw/main/data/global-geophysical-data.nc) (place it in the **same folder** as this notebook)

The data grids are stored in a [netCDF](https://en.wikipedia.org/wiki/NetCDF) file, which is a very common standard for storing and distributing gridded data. It's used throughout the Earth, ocean, and climate sciences and can be read an manipulated with most software platforms. It contains global grids of gravity, topography, and heat flow. All grids were interpolated onto the same resolution of 0.2 degrees (a compromise between resolution and size of the file) so that we can easily compare and use all of the data together.

* **Gravity**: generated from the [EIGEN-6C4 spherical harmonic model](https://doi.org/10.5880/icgem.2015.1) through the [ICGEM webservice](http://icgem.gfz-potsdam.de/home). It includes two grids: the gravity disturbance calculated with the WGS84 ellipsoid and the Bouguer gravity disturbance calculated using a topography density of 2670 kg/m³. The data for both grids was generated on top of the Earth's surface (the water surface in the oceans and topography on land).
* **Topography and bathymetry**: a downsampled version of [ETOPO1](https://doi.org/10.7289/V5C8276M) and includes a version smoothed using a Gaussian filter with 1 degree width (~100 km).
* **Heat flow**: an upsampled version of the compilation by [Lucazeau (2019)](https://doi.org/10.1029/2019GC008389) (originally 0.5 degree resolution).
* **Lithosphere age (oceanic)**: a downsampled version of the grid by [Müller et al. (2008)](https://doi.org/10.1029/2007GC001743) (originally 6 arc-minutes)._____no_output_____## Import the required libraries

Load the required software to load the data, make maps, and perform calculations.

Since we are reusing code and models from previous lessons, you may want to **add them to the `lithosphere.py` module** so that you can easily access your functions in this notebook._____no_output_____

# For arrays and linear algebra
import numpy as np
# To make figures and maps
import matplotlib.pyplot as plt
# To load and manipulate grids
import xarray as xr
# To insert a Google Map on the notebook
from IPython.display import IFrame
# Import our own module (lithosphere.py)
import lithosphere_____no_output_____

## Load the data grids

Once again, we'll use xarray to load the data from the netCDF file._____no_output_____

data = xr.load_dataset("global-geophysical-data.nc")
data_____no_output_____data.topography.plot(figsize=(18, 8))_____no_output_____

And let's also insert a Google Map so that we can easily look up the names of places and locate our study areas._____no_output_____

IFrame("https://www.google.com/maps/embed?pb=!1m14!1m12!1m3!1d68542618.09920536!2d1.6072808059942376!3d-3.737561027842355!2m3!1f0!2f0!3f0!3m2!1i1024!2i768!4f13.1!5e1!3m2!1sen!2suk!4v1621589542222!5m2!1sen!2suk", width=1000, height=600)_____no_output_____

---

## **Your turn!**

Each group will be assigned one of three study areas:

1. Hawaiian Islands: `longitude=slice(-170, -145), latitude=slice(13, 30)`
1. Mariana trench and the Philippine Sea: `longitude=slice(135, 163), latitude=slice(11, 25)`
1. Southern Ocean near the Scotia and Sandwich Plate and the Bouvet triple junction: `longitude=slice(-50, 15), latitude=slice(-65, -50)`

Your task is to:

* Gather the available data for your area
* Make plots, maps, and profiles as you see fit
* Think about what the data can tell you about the lithosphere and tectonics of the area
* Supplement the data investigation with some Google searches
* Use 1 or more of the models we've seen to try to explain the observed data
* Discuss your findings as a group: relate the data, model fit (or lack thereof), and background information

You will deliver back to the whole class:

* A bullet point summary of your findings
* Selected figures that support your ideas

Some tips:

* Start off by plotting the maps of all available data
* Divide the work between group members or pairs
* Take notes as you work instead of waiting until the end_____no_output_____