# Notebook from csdms/espin
Path: lessons/landlab/landlab-terrainbento/Welcome_to_Terrainbento_Solution.ipynb

![Changing_landscapes2021](../../media/Changing_landscapes2021_a.jpg)_____no_output_____# Welcome to terrainbento

If you are interested in reading about the details of terrainbento, this package is described in [Barnhart et al. (2019)](https://www.geosci-model-dev.net/12/1267/2019/).

The documentation for the package is provided [here](http://terrainbento.readthedocs.io/en/latest/).

If you are interested in the source code, you can find it [on GitHub](https://github.com/TerrainBento/terrainbento). If there is a feature that terrainbento does not have that you are interested in, if you have a clarification question, or if you find an error, please make an [Issue on GitHub](https://github.com/TerrainBento/terrainbento/issues) so we can improve the package.

This notebook exists to provide a hyperlinked guide to the supporting examples, tests, and tutorials we have created in support of this package.

## Introduction

terrainbento was designed to make it easy to create alternative models to be compared in Earth surface dynamics. The package has 28 model programs and a model base class that makes it possible to make additional models within the same framework.

The simplest model, called Basic, evolves topography using stream power and linear diffusion. It has the following governing equation:

$\frac{\partial \eta}{\partial t} = - KQ^{m}S^{n} + D\nabla^2 \eta$

where $K$ and $D$ are constants, $Q$ is discharge, $S$ is local slope, and $\eta$ is the topography. Other models modify Basic by adding or changing a process component and changing the governing equation. See the [model Basic documentation](https://terrainbento.readthedocs.io/en/latest/source/terrainbento.derived_models.model_basic.html) for additional information. _____no_output_____## Let's evolve a landscape
Let's start by simulating Landscape evolution on a simple grid using the Basic model_____no_output_____

from terrainbento import Basic

# Figure properties
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline_____no_output_____

What do we need to run the Basic model?

*Note that Basic is a class inheriting from the ErosionModel base class. If you want to learn more on classes, objects and instances, check out [this](https://csdms.colorado.edu/wiki/Presenters-0479) webinar.*_____no_output_____

# Show the documentation for the Basic model
Basic_____no_output_____

We need a [Clock](https://terrainbento.readthedocs.io/en/latest/source/terrainbento.clock.html) instance. The terrainbento Clock controls the timing of model runs.
_____no_output_____

from terrainbento import Clock_____no_output_____

* Figure out what input arguments are needed to create a Clock instance.
* Create a clock instance with:
        * a model start time of 0
        * a total model time of 103 years
        * a timestep of 100 years_____no_output_____

clock = Clock(stop=10e3, step = 100)_____no_output_____

* Make a Landlab raster grid of 15 rows, 20 columns, and 100-unit grid spacing.
* Add a field that holds values of topographic elevation at the grid nodes. Make it zero everywhere
_____no_output_____

# Create a Landlab raster grid
from landlab import RasterModelGrid
grid = RasterModelGrid((15,20),100)
grid.add_zeros('topographic__elevation',at='node')

# Make a figure of the grid
from landlab import imshow_grid
imshow_grid(grid, 'topographic__elevation')_____no_output_____

Create a randomized initial topography_____no_output_____

from landlab.values import random_____no_output_____random(grid, "topographic__elevation")
imshow_grid(grid, 'topographic__elevation')
grid.at_node.keys()_____no_output_____

Now that we have a clock and a grid, let's initialize
the model using the default Basic model constructor (the `__init__` method). _____no_output_____

model = Basic(clock, grid)_____no_output_____

Check all the fields associated to the grid_____no_output_____

grid.at_node.keys()_____no_output_____

Terainbento is adhering to the principles of the basic model interface (BMI)
To run the model as specified, we execute the following line:_____no_output_____

model.run()_____no_output_____

## What did actually happen?
Make a plot of the initial and final topography_____no_output_____

from matplotlib import pyplot as plt
plt.figure()
imshow_grid(grid, "initial_topographic__elevation")
plt.show()
plt.figure()
imshow_grid(grid, "topographic__elevation")
plt.show()_____no_output_____

### Challenge
Make a new Basic model instance, run an plot the initial and calculated topography assuming:
* a grid of 80 by 50 cells, and 100-unit grid spacing.
* an initial random topography
* a water_erodibility coefficient of 2e-4
* a regolith_transport_parameter of 0.2
* a total model run time of 1e5 years and a step of 2000 years_____no_output_____

from landlab import RasterModelGrid
from landlab.values import random
from terrainbento import Clock
from terrainbento import Basic
grid2 = RasterModelGrid((30,20),100)
random(grid2, 'topographic__elevation')
clock = Clock(stop = 1e6, step = 2000)
model2 = Basic(clock,grid2,water_erodibility=5e-4,regolith_transport_parameter=0.01)
model2.run()
plt.figure()
imshow_grid(model2.grid, "initial_topographic__elevation")
plt.show()
plt.figure()
imshow_grid(model2.grid, "topographic__elevation")
plt.show()_____no_output_____

Looks a little strange... why? How can we change this? _____no_output_____

# We need uplift
from terrainbento import ErosionModel
ErosionModel
#We need to adjust the boundary_handlers
# What is the default value of the boundary_handler? #None
model.boundary_handlers_____no_output_____

# Boundary Handlers_____no_output_____

#How do we set a boundary handler? Boundary condition handlers are similar to landlab components.
#They have an init method, and a run_one_step method
from terrainbento import NotCoreNodeBaselevelHandler
#Lowering rate of the outlet node.
grid3 = RasterModelGrid((5,5),10)
z = grid3.add_zeros('topographic__elevation',at='node')
bd_handler = NotCoreNodeBaselevelHandler(grid3,modify_core_nodes=False,lowering_rate=-0.1)

print(z.reshape(grid3.shape))
bd_handler.run_one_step(step=10)
print(z.reshape(grid3.shape))[[ 0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.]]
[[-1. -1. -1. -1. -1.]
 [-1.  0.  0.  0. -1.]
 [-1.  0.  0.  0. -1.]
 [-1.  0.  0.  0. -1.]
 [-1. -1. -1. -1. -1.]]

How to provide this info to a terrainbento model run?

From [documentation](https://terrainbento.readthedocs.io/en/latest/source/terrainbento.base_class.erosion_model.html):
boundary_handlers (dictionary, optional) – Dictionary with `name: instance` key-value pairs. Each entry must be a valid instance of a terrainbento boundary handler._____no_output_____

# Provide the information of the boundary handler as a dictionary
model_bd_handler ={"NotCoreNodeBaselevelHandler": bd_handler}_____no_output_____

### Challenge 1.2
Make a new Basic model instance, run an plot the initial and calculated topography assuming:
* a grid of 80 by 50 cells, and 100-unit grid spacing.
* an initial random topography
* a water_erodibility coefficient of 2e-4
* a regolith_transport_parameter of 0.2
* a total model run time of 1e5 years and a step of 2000 years
* an uplift rate of 0.002 m/year (sea level stays fixed at ca. 0m) _____no_output_____

from landlab import RasterModelGrid
from landlab.values import random
from terrainbento import Clock
from terrainbento import Basic
grid2 = RasterModelGrid((80,50),100)
random(grid2, 'topographic__elevation')
clock = Clock(stop = 1e5, step = 2000)
# Provide the information of the boundary handler as a dictionary
bd_handler = NotCoreNodeBaselevelHandler(grid2,modify_core_nodes=True,lowering_rate=-0.002)
bd_handler ={"NotCoreNodeBaselevelHandler": bd_handler}
model2 = Basic(clock,grid2,water_erodibility=2e-4,regolith_transport_parameter=0.2,
               boundary_handlers=bd_handler)
model2.run()
plt.figure()
imshow_grid(model2.grid, "initial_topographic__elevation")
plt.show()
plt.figure()
imshow_grid(model2.grid, "topographic__elevation")
plt.show()_____no_output_____

## Example usage

There are three additional introductory tutorials.

1) [Introduction terrainbento](example_usage/Introduction_to_terrainbento.ipynb)

2) [Introduction to boundary conditions in terrainbento](example_usage/introduction_to_boundary_conditions.ipynb)

3) [Introduction to output writers in terrainbento](example_usage/introduction_to_output_writers.ipynb).

## Example Coupled Models

This section provides links to five notebooks that show the usage of five example models provided in terrainbento. In each of these notebooks we provide the governing equation(s) for the model, initialize and run the model, make a slope-area plot, save a NetCDF of the final topography, and plot a image of the final topography.

1) [Basic](coupled_process_elements/model_basic_steady_solution.ipynb) the simplest landscape evolution model in the terrainbento package.

2) [BasicVm](coupled_process_elements/model_basic_var_m_steady_solution.ipynb) which permits the drainage area exponent to change

3) [BasicCh](coupled_process_elements/model_basicCh_steady_solution.ipynb) which uses a non-linear hillslope erosion and transport law

4) [BasicVs](coupled_process_elements/model_basicVs_steady_solution.ipynb) which uses variable source area hydrology

5) [BasisRt](coupled_process_elements/model_basicRt_steady_solution.ipynb) which allows for two lithologies with different K values

6) [RealDEM](coupled_process_elements/model_basic_realDEM.ipynb) Run the basic terrainbento model with a real DEM as initial condition.

## Terrainnbento challenge
- [Make your own gif](coupled_process_elements/Challenge.ipynb)_____no_output_____


# Notebook from andybyers21/box-office-web-scraper
Path: notebooks/2 - Combine Data.ipynb

<code>
import os
import pandas as pd_____no_output_____cwd = os.getcwd()
BASE_DIR = os.path.dirname(cwd)
DATA_DIR = os.path.join(BASE_DIR, 'data')
CACHE_DIR = os.path.join(BASE_DIR, 'cache')
os.makedirs(CACHE_DIR, exist_ok = True)_____no_output_____for filename in os.listdir(DATA_DIR):
    print(filename)1995_boxofficemojo_rankings.csv
1999_boxofficemojo_rankings.csv
1983_boxofficemojo_rankings.csv
1978_boxofficemojo_rankings.csv
2006_boxofficemojo_rankings.csv
2010_boxofficemojo_rankings.csv
2017_boxofficemojo_rankings.csv
.DS_Store
2001_boxofficemojo_rankings.csv
1984_boxofficemojo_rankings.csv
1988_boxofficemojo_rankings.csv
1992_boxofficemojo_rankings.csv
2016_boxofficemojo_rankings.csv
2000_boxofficemojo_rankings.csv
1985_boxofficemojo_rankings.csv
1989_boxofficemojo_rankings.csv
1993_boxofficemojo_rankings.csv
1994_boxofficemojo_rankings.csv
1998_boxofficemojo_rankings.csv
1979_boxofficemojo_rankings.csv
1982_boxofficemojo_rankings.csv
2007_boxofficemojo_rankings.csv
2011_boxofficemojo_rankings.csv
2002_boxofficemojo_rankings.csv
2018_boxofficemojo_rankings.csv
2014_boxofficemojo_rankings.csv
1991_boxofficemojo_rankings.csv
1987_boxofficemojo_rankings.csv
1977_boxofficemojo_rankings.csv
1980_boxofficemojo_rankings.csv
1996_boxofficemojo_rankings.csv
2013_boxofficemojo_rankings.csv
2009_boxofficemojo_rankings.csv
2005_boxofficemojo_rankings.csv
1981_boxofficemojo_rankings.csv
1997_boxofficemojo_rankings.csv
2012_boxofficemojo_rankings.csv
2008_boxofficemojo_rankings.csv
2004_boxofficemojo_rankings.csv
2003_boxofficemojo_rankings.csv
2019_boxofficemojo_rankings.csv
2015_boxofficemojo_rankings.csv
2020_boxofficemojo_rankings.csv
1990_boxofficemojo_rankings.csv
1986_boxofficemojo_rankings.csv
df = pd.read_csv(os.path.join(DATA_DIR, '2020_boxofficemojo_rankings.csv'))
df.head()_____no_output_____df['Year'] = 2020_____no_output_____df.head()_____no_output_____my_entire_data = pd.DataFrame

my_dataframes = []
csv_files = [x for x in os.listdir(DATA_DIR) if x.endswith(".csv")]

for filename in csv_files:
    print(filename)
    year = filename.replace("_boxofficemojo_rankings.csv", "")
    csv_path = os.path.join(DATA_DIR, filename)
    this_df = pd.read_csv(csv_path)
    this_df['filename'] = filename
    this_df['year'] = year
    print(this_df.head(n=1))
    my_dataframes.append(this_df)1995_boxofficemojo_rankings.csv
   Rank              Release Group     Worldwide      Domestic      %  \
0     1  Die Hard with a Vengeance  $366,101,666  $100,012,499  27.3%

        Foreign    %.1                         filename  year
0  $266,089,167  72.7%  1995_boxofficemojo_rankings.csv  1995
1999_boxofficemojo_rankings.csv
   Rank                              Release Group     Worldwide  \
0     1  Star Wars: Episode I - The Phantom Menace  $924,305,084

       Domestic      %       Foreign    %.1                         filename  \
0  $431,088,295  46.6%  $493,216,789  53.4%  1999_boxofficemojo_rankings.csv

   year
0  1999
1983_boxofficemojo_rankings.csv
   Rank                               Release Group     Worldwide  \
0     1  Star Wars: Episode VI - Return of the Jedi  $374,593,074

       Domestic      %       Foreign    %.1                         filename  \
0  $252,583,617  67.4%  $122,009,457  32.6%  1983_boxofficemojo_rankings.csv

   year
0  1983
1978_boxofficemojo_rankings.csv
   Rank Release Group     Worldwide      Domestic      %       Foreign    %.1  \
0     1        Grease  $366,178,870  $159,978,870  43.7%  $206,200,000  56.3%

                          filename  year
0  1978_boxofficemojo_rankings.csv  1978
2006_boxofficemojo_rankings.csv
   Rank                               Release Group       Worldwide  \
0     1  Pirates of the Caribbean: Dead Man's Chest  $1,066,179,725

       Domestic      %       Foreign    %.1                         filename  \
0  $423,315,812  39.7%  $642,863,913  60.3%  2006_boxofficemojo_rankings.csv

   year
0  2006
2010_boxofficemojo_rankings.csv
   Rank Release Group       Worldwide      Domestic      %       Foreign  \
0     1   Toy Story 3  $1,066,969,703  $415,004,880  38.9%  $651,964,823

     %.1                         filename  year
0  61.1%  2010_boxofficemojo_rankings.csv  2010
2017_boxofficemojo_rankings.csv
   Rank                            Release Group       Worldwide  \
0     1  Star Wars: Episode VIII - The Last Jedi  $1,332,539,889

       Domestic      %       Foreign    %.1                         filename  \
0  $620,181,382  46.5%  $712,358,507  53.5%  2017_boxofficemojo_rankings.csv

   year
0  2017
2001_boxofficemojo_rankings.csv
   Rank                          Release Group     Worldwide      Domestic  \
0     1  Harry Potter and the Sorcerer's Stone  $974,755,371  $317,575,550

       %       Foreign    %.1                         filename  year
0  32.6%  $657,179,821  67.4%  2001_boxofficemojo_rankings.csv  2001
1984_boxofficemojo_rankings.csv
   Rank                         Release Group     Worldwide      Domestic  \
0     1  Indiana Jones and the Temple of Doom  $333,107,271  $179,870,271

     %       Foreign  %.1                         filename  year
0  54%  $153,237,000  46%  1984_boxofficemojo_rankings.csv  1984
1988_boxofficemojo_rankings.csv
   Rank Release Group     Worldwide      Domestic      %       Foreign    %.1  \
0     1      Rain Man  $354,825,435  $172,825,435  48.7%  $182,000,000  51.3%

                          filename  year
0  1988_boxofficemojo_rankings.csv  1988
1992_boxofficemojo_rankings.csv
   Rank Release Group     Worldwide      Domestic      %       Foreign    %.1  \
0     1       Aladdin  $504,050,219  $217,350,219  43.1%  $286,700,000  56.9%

                          filename  year
0  1992_boxofficemojo_rankings.csv  1992
2016_boxofficemojo_rankings.csv
   Rank               Release Group       Worldwide      Domestic      %  \
0     1  Captain America: Civil War  $1,153,296,293  $408,084,349  35.4%

        Foreign    %.1                         filename  year
0  $745,211,944  64.6%  2016_boxofficemojo_rankings.csv  2016
2000_boxofficemojo_rankings.csv
   Rank           Release Group     Worldwide      Domestic      %  \
0     1  Mission: Impossible II  $546,388,108  $215,409,889  39.4%

        Foreign    %.1                         filename  year
0  $330,978,219  60.6%  2000_boxofficemojo_rankings.csv  2000
1985_boxofficemojo_rankings.csv
   Rank       Release Group     Worldwide      Domestic      %       Foreign  \
0     1  Back to the Future  $381,109,762  $210,609,762  55.3%  $170,500,000

     %.1                         filename  year
0  44.7%  1985_boxofficemojo_rankings.csv  1985
1989_boxofficemojo_rankings.csv
   Rank                       Release Group     Worldwide      Domestic  \
0     1  Indiana Jones and the Last Crusade  $474,171,806  $197,171,806

       %       Foreign    %.1                         filename  year
0  41.6%  $277,000,000  58.4%  1989_boxofficemojo_rankings.csv  1989
1993_boxofficemojo_rankings.csv
   Rank  Release Group     Worldwide      Domestic      %       Foreign  \
0     1  Jurassic Park  $912,667,947  $357,067,947  39.1%  $555,600,000

     %.1                         filename  year
0  60.9%  1993_boxofficemojo_rankings.csv  1993
1994_boxofficemojo_rankings.csv
   Rank  Release Group     Worldwide      Domestic    %       Foreign  %.1  \
0     1  The Lion King  $763,455,561  $312,855,561  41%  $450,600,000  59%

                          filename  year
0  1994_boxofficemojo_rankings.csv  1994
1998_boxofficemojo_rankings.csv
   Rank Release Group     Worldwide      Domestic      %       Foreign    %.1  \
0     1    Armageddon  $553,709,788  $201,578,182  36.4%  $352,131,606  63.6%

                          filename  year
0  1998_boxofficemojo_rankings.csv  1998
1979_boxofficemojo_rankings.csv
   Rank Release Group     Worldwide     Domestic      %       Foreign    %.1  \
0     1     Moonraker  $210,308,099  $70,308,099  33.4%  $140,000,000  66.6%

                          filename  year
0  1979_boxofficemojo_rankings.csv  1979
1982_boxofficemojo_rankings.csv
   Rank               Release Group     Worldwide      Domestic      %  \
0     1  E.T. the Extra-Terrestrial  $663,400,925  $359,197,037  54.1%

        Foreign    %.1                         filename  year
0  $304,203,888  45.9%  1982_boxofficemojo_rankings.csv  1982
2007_boxofficemojo_rankings.csv
   Rank                             Release Group     Worldwide      Domestic  \
0     1  Pirates of the Caribbean: At World's End  $960,996,492  $309,420,425

       %       Foreign    %.1                         filename  year
0  32.2%  $651,576,067  67.8%  2007_boxofficemojo_rankings.csv  2007
2011_boxofficemojo_rankings.csv
   Rank                                 Release Group       Worldwide  \
0     1  Harry Potter and the Deathly Hallows: Part 2  $1,341,511,219

       Domestic      %       Foreign    %.1                         filename  \
0  $381,011,219  28.4%  $960,500,000  71.6%  2011_boxofficemojo_rankings.csv

   year
0  2011
2002_boxofficemojo_rankings.csv
   Rank                          Release Group     Worldwide      Domestic  \
0     1  The Lord of the Rings: The Two Towers  $936,689,735  $339,789,881

       %       Foreign    %.1                         filename  year
0  36.3%  $596,899,854  63.7%  2002_boxofficemojo_rankings.csv  2002
2018_boxofficemojo_rankings.csv
   Rank           Release Group       Worldwide      Domestic      %  \
0     1  Avengers: Infinity War  $2,048,359,754  $678,815,482  33.1%

          Foreign    %.1                         filename  year
0  $1,369,544,272  66.9%  2018_boxofficemojo_rankings.csv  2018
2014_boxofficemojo_rankings.csv
   Rank                    Release Group       Worldwide      Domestic      %  \
0     1  Transformers: Age of Extinction  $1,104,054,072  $245,439,076  22.2%

        Foreign    %.1                         filename  year
0  $858,614,996  77.8%  2014_boxofficemojo_rankings.csv  2014
1991_boxofficemojo_rankings.csv
   Rank               Release Group     Worldwide      Domestic      %  \
0     1  Terminator 2: Judgment Day  $516,950,043  $204,843,345  39.6%

        Foreign    %.1                         filename  year
0  $312,106,698  60.4%  1991_boxofficemojo_rankings.csv  1991
1987_boxofficemojo_rankings.csv
   Rank     Release Group     Worldwide      Domestic      %       Foreign  \
0     1  Fatal Attraction  $320,145,693  $156,645,693  48.9%  $163,500,000

     %.1                         filename  year
0  51.1%  1987_boxofficemojo_rankings.csv  1987
1977_boxofficemojo_rankings.csv
   Rank                       Release Group     Worldwide      Domestic  \
0     1  Star Wars: Episode IV - A New Hope  $503,015,849  $307,263,857

       %       Foreign    %.1                         filename  year
0  61.1%  $195,751,992  38.9%  1977_boxofficemojo_rankings.csv  1977
1980_boxofficemojo_rankings.csv
   Rank                                   Release Group     Worldwide  \
0     1  Star Wars: Episode V - The Empire Strikes Back  $400,083,259

       Domestic      %       Foreign    %.1                         filename  \
0  $209,398,025  52.3%  $190,685,234  47.7%  1980_boxofficemojo_rankings.csv

   year
0  1980
1996_boxofficemojo_rankings.csv
   Rank     Release Group     Worldwide      Domestic      %       Foreign  \
0     1  Independence Day  $817,400,891  $306,169,268  37.5%  $511,231,623

     %.1                         filename  year
0  62.5%  1996_boxofficemojo_rankings.csv  1996
2013_boxofficemojo_rankings.csv
   Rank Release Group       Worldwide      Domestic      %       Foreign  \
0     1        Frozen  $1,280,802,282  $400,738,009  31.3%  $880,064,273

     %.1                         filename  year
0  68.7%  2013_boxofficemojo_rankings.csv  2013
2009_boxofficemojo_rankings.csv
   Rank Release Group       Worldwide      Domestic      %         Foreign  \
0     1        Avatar  $2,744,336,793  $749,766,139  27.3%  $1,994,570,654

     %.1                         filename  year
0  72.7%  2009_boxofficemojo_rankings.csv  2009
2005_boxofficemojo_rankings.csv
   Rank                        Release Group     Worldwide      Domestic  \
0     1  Harry Potter and the Goblet of Fire  $895,921,036  $290,013,036

       %       Foreign    %.1                         filename  year
0  32.4%  $605,908,000  67.6%  2005_boxofficemojo_rankings.csv  2005
1981_boxofficemojo_rankings.csv
   Rank            Release Group     Worldwide      Domestic    %  \
0     1  Raiders of the Lost Ark  $353,988,025  $212,222,025  60%

        Foreign  %.1                         filename  year
0  $141,766,000  40%  1981_boxofficemojo_rankings.csv  1981
1997_boxofficemojo_rankings.csv
   Rank Release Group       Worldwide      Domestic      %         Foreign  \
0     1       Titanic  $1,850,906,470  $600,788,188  32.5%  $1,250,118,282

     %.1                         filename  year
0  67.5%  1997_boxofficemojo_rankings.csv  1997
2012_boxofficemojo_rankings.csv
   Rank Release Group       Worldwide      Domestic    %       Foreign  %.1  \
0     1  The Avengers  $1,518,812,988  $623,357,910  41%  $895,455,078  59%

                          filename  year
0  2012_boxofficemojo_rankings.csv  2012
2008_boxofficemojo_rankings.csv
my_entire_dataframe = pd.concat(my_dataframes)_____no_output_____my_entire_dataframe.head()_____no_output_____my_entire_dataframe.tail()_____no_output_____dataset = os.path.join(CACHE_DIR, 'box-office-ranking-dataset.csv')
my_entire_dataframe.to_csv(dataset, index=False)_____no_output_____
</code>


# Notebook from duderstadt-lab/Born-to-slide
Path: Analysis_software/Figures_1_S1/Figures_1B_S1D.ipynb

# Born to slide: mobile origin licensing factors confer resistance to conflicts with RNA polymerase

## Jupyter notebooks

#### Figures:
+ 1B
+ S1D

#### Archives:
+ Transcription_licensed-DNA_ATP_LS.yama
+ Transcription_licensed-DNA_ATP_HS.yama
+ Transcription_licensed-DNA_ATPgS_LS.yama
+ Transcription_licensed-DNA_ATPgS_HS.yama
+ Transcription_low-chromatin-licensed-DNA_ATP_1.yama
+ Transcription_high-chromatin-licensed-DNA_ATP.yama
+ Transcription_low-chromatin-licensed-DNA_ATP_2.yama
+ Transcription_low-chromatin-licensed-DNA_ATPgS.yama
+ Photobleaching_licensed-DNA_ATP.yama
+ Photobleaching_licensed-DNA_ATPgS.yama
+ Transcription_licensed-DNA_T7T_ATP.yama
+ Transcription_licensed-DNA_T7T_ATPgS.yama
+ Transcription_ORC-DNA_ATP.yama
+ Transcription_ORC-DNA_T7T_ATP.yama

#### Additional data:
+ none_____no_output_____### Initialize ImageJ_____no_output_____

import imagej
ij = imagej.init('/Applications/Fiji.app')
ij.getVersion()_____no_output_____

### Imports_____no_output_____

import sys
sys.path.insert(0, '..')
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import re
import awesome_data
from marspy.convert.archive import DnaMoleculeArchive, instantiate_archive, describe_archives
from tqdm.notebook import tqdm
from functools import reducemarspy initialized.

### Select archives_____no_output_____

selected_archives = ['Transcription_licensed-DNA_ATP_LS.yama',
                     'Transcription_licensed-DNA_ATP_HS.yama',
                     'Transcription_licensed-DNA_ATPgS_LS.yama',
                     'Transcription_licensed-DNA_ATPgS_HS.yama',
                     'Transcription_low-chromatin-licensed-DNA_ATP_1.yama',
                     'Transcription_high-chromatin-licensed-DNA_ATP.yama',
                     'Transcription_low-chromatin-licensed-DNA_ATP_2.yama',
                     'Transcription_low-chromatin-licensed-DNA_ATPgS.yama',
                     'Photobleaching_licensed-DNA_ATP.yama',
                     'Photobleaching_licensed-DNA_ATPgS.yama',
                     'Transcription_licensed-DNA_T7T_ATP.yama',
                     'Transcription_licensed-DNA_T7T_ATPgS.yama',
                     'Transcription_ORC-DNA_ATP.yama',
                     'Transcription_ORC-DNA_T7T_ATP.yama']_____no_output_____

### Figure style and output directory_____no_output_____

from figure_style import *_____no_output_____set_style_paper()_____no_output_____# change to desired output directory
dir_out = '/Volumes/pool-duderstadt/Matthias/Manuscript/Figures/SVGs/Figure1/'
dir_out_____no_output_____

### Instantiate selected archives_____no_output_____

for archive in tqdm(selected_archives):
    instantiate_archive(archive, awesome_data.datasets)_____no_output_____

### Archives Overview_____no_output_____

describe_archives(DnaMoleculeArchive.instances)_____no_output_____

### Load segment tables, apply filter & detect pauses_____no_output_____

for index,archive in enumerate(tqdm(DnaMoleculeArchive.instances)):

    archive.add_segments_tables()
    archive.filter_segments(b_min=-10, sigma_b_max=10)
    archive.detect_pauses()
_____no_output_____

### Generate main df_____no_output_____

def setup_pandas_df(archive_instances):
    '''
    Sets up a multilevel df with all relevant information
    '''

    # indices will be UIDs
    prefixes = set()
    for archive in archive_instances:
        prefixes.update(archive.prefixes)
    col_1 = sorted(list(prefixes))
    #inside
    col_2 = ['initial_intensity', 'lifetime', 'start_site', 'stop_site', 'number_pauses']
    col_3 = ['']
    hier_index = pd.MultiIndex.from_product([col_1,col_2,col_3],names=['molecule','properties','properties_details'])
    print(prefixes)
    return pd.DataFrame(columns=hier_index)
    _____no_output_____df = setup_pandas_df(DnaMoleculeArchive.instances)

# sort MultiIndexCols for performance
df.sort_index(axis=1, inplace=True)
df.head()
_____no_output_____for archive in tqdm(DnaMoleculeArchive.instances):

    for molecule in archive.molecules:
        # general columns first
        df.loc[molecule.uid,'dna_length'] = molecule.calc_length_dna()
        df.loc[molecule.uid,'dna_count_valid'] = archive.dna_count_valid
        df.loc[molecule.uid,'hs_wash'] = archive.highsalt_wash
        df.loc[molecule.uid,'nucleotide'] = archive.nucleotide
        df.loc[molecule.uid,'tags'] = reduce(lambda tag1,tag2: tag1+','+tag2, molecule.tags)

        # general proteins
        for protein in molecule.proteins:
            #store protein number
            df.loc[molecule.uid,'number_'+protein] = molecule.params['Number_'+protein]
            #assign label for each protein
            df.loc[molecule.uid,'label_'+protein] = archive.labels[protein]

        # every specific protein
        for prefix in molecule.prefixes:

            #take mean intensity from first 10 frames (maybe need try block later if shorter traces)
            df.loc[molecule.uid,(prefix,'initial_intensity')] = molecule.df.iloc[:5][prefix+'Intensity'].mean()

            #lifetime (in frames) defined on tracking length
            df.loc[molecule.uid,(prefix,'lifetime')] = len(molecule.df[prefix+'y'].dropna())

            #start & stop site
            if re.match('T7',prefix):
                df.loc[molecule.uid,(prefix,'start_site')] = molecule.df[prefix+'Position_on_DNA'].min()
                df.loc[molecule.uid,(prefix,'stop_site')] = molecule.df[prefix+'Position_on_DNA'].max()
            else:
                df.loc[molecule.uid,(prefix,'start_site')] = molecule.df.iloc[:5][prefix+'Position_on_DNA'].mean()
                df.loc[molecule.uid,(prefix,'stop_site')] = molecule.df[prefix+'Position_on_DNA'].max()

# convert tags back to list
df['tags'] = df['tags'].apply(lambda tags: tags.split(','))

# again sort MultiIndexCols for performance
df.sort_index(axis=1, inplace=True)

# fix data types
df = df.infer_objects()
_____no_output_____df_____no_output_____

## Figures_____no_output_____### Figure 1B: ORC-MCM loading sites_____no_output_____

proteins = ['ORC','MCM']
pal_pos = 1

fig,axes = plt.subplots(len(proteins),1,sharex=True,sharey=True)

for i, protein in enumerate(proteins):
    sns.distplot(df.filter(regex=(protein+"_"),axis=1).xs('start_site',level='properties',axis=1)/1000,
                 label=f'{protein}',color=palettes[protein][pal_pos],bins = np.linspace(0, 25, 51), ax=axes[i])

    axes[i].text(0.95, 0.95, f"n = {len(pd.melt(df.filter(regex=(protein+'_'),axis=1).xs('start_site',level='properties',axis=1))['value'].dropna())}",
            verticalalignment='top', horizontalalignment='right',
            transform=axes[i].transAxes)

    axes[i].text(0.95, 0.95, f"n = {len(pd.melt(df.filter(regex=(protein+'_'),axis=1).xs('start_site',level='properties',axis=1))['value'].dropna())}",
            verticalalignment='top', horizontalalignment='right',
            transform=axes[i].transAxes)

for ax in axes:
    ax.set_xlim(0,24)
    ax.set_ylim(0,0.75)
    ax.set_ylabel(None)
    ax.legend(frameon=False,loc=9)

    sns.despine()

axes2 = fig.add_axes([.383, .62, .42, .2])
axes3 = fig.add_axes([.383, .21, .42, .2])

sns.distplot(df.filter(regex=(proteins[0]+"_"),axis=1).xs('start_site',level='properties',axis=1)/1000,
             color=palettes[proteins[0]][pal_pos],bins = np.linspace(0, 25, 51), kde=False, ax=axes2)

axes2.set_xlim(8,21)
axes2.set_ylim(0,12.5)
axes2.set_ylabel('Counts')
axes2.set_xticklabels([])
axes2.set_xticks([10,15,20])
axes2.set_yticks([0,5,10])
axes2.patch.set_alpha(0)
axes2.yaxis.tick_right()
axes2.yaxis.set_label_position("right")
axes2.spines['left'].set_visible(False)
axes2.spines['top'].set_visible(False)

sns.distplot(df.filter(regex=(proteins[1]+"_"),axis=1).xs('start_site',level='properties',axis=1)/1000,
             color=palettes[proteins[1]][pal_pos],bins = np.linspace(0, 25, 51), kde=False, ax=axes3)

axes3.set_xlim(8,21)
axes3.set_ylim(0,25)
axes3.set_ylabel('Counts')
axes3.set_xticklabels([])
axes3.set_xticks([10,15,20])
axes3.set_yticks([0,10,20])
axes3.patch.set_alpha(0)
axes3.yaxis.tick_right()
axes3.yaxis.set_label_position("right")
axes3.spines['left'].set_visible(False)
axes3.spines['top'].set_visible(False)

fig.text(0.5, -0.04, 'Position on DNA (kb)', ha='center', rotation='horizontal')
fig.text(-0.04, 0.5, 'Probability density', va='center', rotation='vertical')

#plt.tight_layout()

plt.savefig(dir_out+'1B_ORC-MCM_loading_sites_inset.svg',transparent=True)_____no_output_____

### Figure S1D: ORC-MCM loading sites - hue nucleotide_____no_output_____

_plot_df = pd.DataFrame(columns=['start_site','nucleotide','protein'])
proteins = ['ORC','MCM']
for protein in proteins:
    for nuc in df['nucleotide'].unique():
        _temp_df = pd.DataFrame(data=pd.melt(df[df['nucleotide']==nuc].filter(regex=(protein+"_"),axis=1).xs('start_site',level='properties',axis=1)/1000)['value'].dropna())
        _temp_df.columns = ['start_site']
        _temp_df['nucleotide'] = nuc
        _temp_df['protein'] = protein
        _plot_df = pd.concat([_plot_df,_temp_df])

fig,axes = plt.subplots(1,2,sharex=True,sharey=True)

sns.violinplot(x='start_site',y=['dummy_label']*len(_plot_df[_plot_df['protein']=='ORC']),hue='nucleotide',data=_plot_df[_plot_df['protein']=='ORC'],
                split=True,inner='quartile', scale='area',ax=axes[0], palette=[palettes['ORC'][1],palettes['ORC'][3]], hue_labels=[1,2])
sns.violinplot(x='start_site',y=['dummy_label']*len(_plot_df[_plot_df['protein']=='MCM']),hue='nucleotide',data=_plot_df[_plot_df['protein']=='MCM'],
                split=True,inner='quartile', scale='area',ax=axes[1], palette=[palettes['MCM'][1],palettes['OCCM'][1]], hue_labels=[1,2])

for i, protein in enumerate(proteins):
    axes[i].set_xlabel(None)
    axes[i].set_yticklabels([])
    axes[i].set_yticks([])
    axes[i].text(0.95, 0.75, f"n-ATP = {len(_plot_df[(_plot_df['protein']==protein) & (_plot_df['nucleotide']=='ATP')])}",
            verticalalignment='top', horizontalalignment='right',
            transform=axes[i].transAxes)
    axes[i].text(0.95, 0.65, f"n-ATPgS = {len(_plot_df[(_plot_df['protein']==protein) & (_plot_df['nucleotide']=='ATPgS')])}",
            verticalalignment='top', horizontalalignment='right',
            transform=axes[i].transAxes)

    axes[i].legend(frameon=False)

sns.despine()
fig.text(0.5, -0.04, 'Position on DNA (kb)', ha='center', rotation='horizontal')
fig.text(0.04, 0.5, 'Relative probability density', va='center', rotation='vertical')

#plt.tight_layout()

plt.savefig(dir_out+'S1D_ORC-MCM_loading_sites_hueNuc.svg',transparent=True)
_____no_output_____