# Notebook from MichielStock/SelectedTopicsOptimization
Path: Chapters/07.ShortestPathProblems/Chapter7.ipynb

# Shortest Path Problems

*Selected Topics in Mathematical Optimization*

**Michiel Stock** ([email](michiel.stock@ugent.be))

![](Figures/logo.png)_____no_output_____

import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
from shortestpaths import red, blue, green, yellow, orange
from sys import path
# add path of MST to load Tickets to Ride network
path.append('../06.MinimumSpanningTrees')_____no_output_____

## Representing graphs in Python (recap)

Graphs are discrete mathematical objects, represented as a tuple $G=(V, E)$ with $V$ the set of vertices (nodes, points) and $E$ the set of edges (links, arcs, lines) that connect the vertices.

Consider for example the graph below.

![A directed unweighted graph](Figures/unweighted_graph.png)

This graph contains five edges: A, B, C, D and E and nine vertices. Since each of the vertices has a direction from one node to another so we call this a **directed graph**.

Like most programming languages, Python does not provide direct support for graphs a data types. Graphs can however be represented by [**dictionaries**](https://docs.python.org/2/tutorial/datastructures.html#dictionaries). Dictionaries in Python store key-value pairs in a way such that accessing, insertion and deletion of new elements can be done very efficiently. The keys will represent the edges, while its associated values is the set of vertices to which depart from this node.

For example, the unweighted graph above is represented bij the following code._____no_output_____

unweighted_graph = {'A' : set(['B']),
                   'B' : set(['C']),
                   'C' : set(['A', 'B', 'E']),
                   'D' : set(['A']),
                   'E' : set(['B','D', 'D'])  # set data structure only contains unique elements
                   }                     # only one node will be retained_____no_output_____

Note that we store the outgoing vertices in a [**set**](https://docs.python.org/2/tutorial/datastructures.html#sets) data structure. This will allow us to perform certain operations faster. Note that of the two vertices starting for E only one is retained. _____no_output_____

unweighted_graph['E']_____no_output_____

In this chapter, we will focus on algorithms that process **weighted graphs**, i.e. graphs on which the vertices have a weight. For example:
![A weighted directed graph](Figures/weighted_graph.png)
We will store each vertex now as a tuple containing the weight and the outgoing edge it is connected to. For this example:_____no_output_____

weighted_graph = {'A' : set([(3, 'B')]),
                   'B' : set([(2, 'C')]),
                   'C' : set([(6, 'A'), (1, 'B'), (0.3, 'E')]),
                   'D' : set([(9, 'A')]),
                   'E' : set([(2.8, 'D'), (4.1, 'D'), (4.2, 'E')])  # two paths from E -> D
                    }_____no_output_____

Storing a weighted vertex as `(weight, outgoing node)` has the advantage that the native `sort()` function in Python can be used to order vertices by weight, something which will be useful in the algorithms we will implement._____no_output_____## Dijkstra's shortest path algorithm_____no_output_____Dijkstra's algorithm is a popular algorithm to find the shortest path between the nodes of a graph. The algorithm can be used in two ways:

- when both a source and a sink node are provided, the algorithm gives the list of nodes of the shortest path, together with the length (distance of the path).
- when only a source is given, the shortest distance between the source and all (accessable) nodes is returned together with a dictionary representing the tree of the shortest paths between the source and other nodes.

The pseudocode can be found below._____no_output_____### Pseudocode of Dijkstra's shortest path algorithm
Source: Wikipedia.org
```
1  function Dijkstra(Graph, source, sink (optional)):
2      distance[source] := 0  // initialize a dictionary with distance
3                             // to source
4      for each vertex V in Graph:
5          if V ≠ source
6              distance[V] := infinity
7      previous := empty dict   // stores the previous node in path
8
10     make empty priority queue Q for vertices to check
11     Q.add_with_priority(source, 0)  # first check the source
12
13
14     while Q is not empty:                     // The main loop
15         get node U with smallest distance from Q
16         if U is the sink (if provided)
17              break the loop
17         for every neighbor V of U:
18             alternative := distance[U] + dist(U, V)
18             if alternative < distance[V]
19                 distance[V] := alternative
20                 previous[V] := U
21                 Q.add_with_priority(V, alternative)
22
23     if sink is given
24         reconstruct shortest path
25         return shortest path and length of path
26     else
27         return distance, previous
```_____no_output_____### Pseudocode of path reconstruction  algorithm
```
1 function reconstruct_path(previous, source, sink)
2    if the source is not in previous
3          return empty path
4    V := sink   \\ path is reconstructed backwards
5    path := [V]  \\ path is a list of nodes
6    while V is not source
7         V := previous node in path
8         add V in beginning of the path
9    return the path
```_____no_output_____

from numpy import infty

def dijkstra(graph, source, sink=None):
    """
    Implementation of Dijkstra's shortest path algorithm.

    Inputs:
        - graph : dict representing the weighted graph
        - source : the source node
        - sink : the sink node (optional)

    Ouput:
            - distance : dict with the distances of the nodes to the source
            - previous : dict with for each node the previous node in the
                        shortest path from the source (if one is given)
    """
    # complete the algorithm
    # ...
    # ...

    if sink is None:
        return distance, previous
    else:
        return reconstruct_path(previous, source, sink), distance[sink]
_____no_output_____def reconstruct_path(previous, source, sink):
    """
    Reconstruct the path from the output of the Dijkstra algorithm.

    Inputs:
            - previous : a dict with the previous node in the path
            - source : the source node
            - sink : the sink node
    Ouput:
            - the shortest path from source to sink
    """
    if sink not in previous:
        return []
    # ... complete this!
    return path_____no_output_____

**Assignment 1: Dijkstra**

1. Complete the implementation for Dijstra's algorithm and the path reconstruction function
2. Find the shortest path from node A to I in the example graph below. Calculate the shortest distance from A to any other node. Compare both running times with `%timeit`.

![Example network](Figures/example_graph.png)_____no_output_____

# the above graph as an adjacency list
graph = {'A' : set([(2, 'B'), (3, 'D')]),
        'B' : set([(1, 'C'), (5, 'E')]),
        'C' : set([(2, 'D'), (1, 'E')]),
        'D' : set([(3, 'E')]),
        'E' : set([(2, 'B'), (3, 'F')]),
        'F' : set([(5, 'G'), (8, 'I')]),
        'G' : set([(2, 'H'), (5, 'I')]),
        'H' : set([(3, 'I')]),
        'I' : []}_____no_output_____dijkstra(graph, 'A', 'I')_____no_output_____dijkstra(graph, 'A')_____no_output_____%timeit dijkstra(graph, 'A', 'I')_____no_output_____%timeit dijkstra(graph, 'A')_____no_output_____

## A* shortest path algorithm_____no_output_____It is hard to explain the algorithm into more depth than [here](http://theory.stanford.edu/~amitp/GameProgramming/) and [here](http://www.redblobgames.com/pathfinding/a-star/introduction.html)._____no_output_____The A\* algorithm is exact (like Dijkstra's algorithm), but it can also use a **heuristic** to speed up the search. In each iteration, the next neighbor $v$ of the current vertex that the algorithm considers is chosen based on the heuristic

$$
f(v) = g(v) + h(v)
$$

with $g(v)$ the cost of the path so far and $h(v)$ a heuristic that estimates the cost of the cost of the shortest path from $v$ to the goal. The heuristic $h(v)$ should satify two properties:
1. To guaranty that a shortest path is found, the heuristic should be **admissible**, i.e. it should never overestimate the true distance to goal.
2. Evaluating the heuristic should be cheap (it is either pre-computed or is a function).

So the A\* algorithm is basically the same as Dijkstra's algorithm, but with the main difference that the the latter chooses nodes to explore based on the distance from the starting node, while the latter chooses nodes **based on an estimate of distance to the goal**._____no_output_____When using A\* to find the shortest path between two physical locations, each vertex corresponds to a point in space. A good heuristic is the (Euclidian) distance between the vertexs, as this will always be a lower bound for the actual distance to travel to the goal.

Consider the following example of finding the path from vertex $a$ to vertex $h$.

![Example how A\* uses the heuristics](Figures/astarexample.png)

The shortest path so far is $[a, e]$ and the algorithm needs to choose to explore vertex $d$ or vertex $g$. Since the path of $ed$ is shorter than the path $eg$, Dijkstra will choose this vertex, even though it is farther removed from the goal $h$ (Dijkstra chooses a neighbor $n$ only based on the current path length form the starting vertex $g(n)$. The A\* algorithm will chose vertex $g$ to explore, because the estimated path length $f(e) = g(e) + h(e) > f(d)$ and hence will approach the goal. _____no_output_____### Pseudocode of A\* shortest path algorithm
Source from Wikipedia (modified)
```
1  function Astar(Graph, source, sink, heuristic):
2      make empty priority queue Q for vertices to check
3
4      add source to Q with priority f(source)
5          // use the estimated path length for priority
6
7      distance[V] := 0   // initialize a dictionary with distance
8                           // to source
9      previous := empty dict   // stores the previous node in path
10
11     while Q is not empty:                     // The main loop
12         pop node U with lowest estimated path length to sink
13         if U is the sink
14              break the loop
15         for every neighbor V of U:
16             distance_source_U := distance[U] + dist(U, V)
17             if V not in distance or distance_source_U < distance[V]
18                 distance[V] := distance_source_U
19                 previous[V] := U
20                 heuristic_V_sink := distance[V] + heuristic(V)
21                 Q.add_with_priority(V, heuristic_V_sink)
22
23     reconstruct shortest path
24     return shortest path and length of path
```_____no_output_____#### Representation of the graphs for physical locations

For the assignments above, we will work with graphs representing paths on a two-dimensional plane.

Each node or location is a point on this plane, e.g. $(x_i, y_i)$. An edge or link between two locations is a tuple of two points: $((x_i, y_i),(x_j, y_j))$. In the link representation, the network is represented as just a list of these links. For example:_____no_output_____

from parcour_tools import add_path, load_links, plot_parcour, links_to_graph

links_example = [((1, 1), (1, 3)), ((1, 1), (2, 2)), ((1,3), (2, 2)),
                ((2, 2), (3, 1)), ((2, 2), (3, 3)), ((3, 1), (3, 3))]_____no_output_____

Using the function ```links_to_graph``` we can change these links in the more familiar dictionary-based representation._____no_output_____

graph_example = links_to_graph(links_example)
print(graph_example)_____no_output_____

Which can be visualized using ```plot_parcour``` and we can use the function ```add_path``` to draw a path._____no_output_____

fig, ax = plt.subplots()
plot_parcour(links_example, ax, line_width=20)  # uses the links representation as an input!
ax.set_ylim([0.5, 3.5])
ax.set_xlim([0.5, 3.5])

path_example = [(1, 1), (1, 3), (2, 2), (3, 1), (3, 3)]
add_path(path_example, ax, color=green)_____no_output_____

**Assignment 2: A\***

1. Complete the implementation for the A\* algorithm.
2. Some parcours and mazes are given. Use A\* and Dijkstra to find the shortest paths between the lower left corner to the upper right corner. Use the function ```add_path``` to add the path to the figure.
3. Compare running time of the two shortest path algorithms using ```%timeit```._____no_output_____

# functions for computing distances
manhattan_distance = lambda x, y : sum([abs(xi - yi) for xi, yi in zip(x, y)])
euclidean_distance = lambda x, y : sum([(xi - yi)**2 for xi, yi in zip(x, y)])**0.5_____no_output_____

def a_star(graph, source, sink, heuristic):
    """
    Implementation of the A* shortest path algorithm
    Inputs:
        - graph : dict representing the graph
        - source : the source node
        - sink : the sink node (optional)
        - heuristic : a function with the heuristic for the shortest path between two nodes
    Ouput:
        - distance : dict with the distances of the nodes to the source
        - came_from : dict with for each node the previous node in the shortest
                    path from the source
    """
    # COMPLETE THIS (you can start from dijkstra...)
    #...
    _____no_output_____# a parcour, white lines are the routes that can be followed
# find the shortest path from (0, 0) to (99, 99)
parcour = load_links('Data/large_parcour.csv')_____no_output_____fig, ax = plt.subplots(figsize=(15, 15))

plot_parcour(parcour, ax)
parcour_graph = links_to_graph(parcour)

path_dijkstra, length_dijstra = dijkstra(parcour_graph, (0, 0), (99, 99))
add_path(path_dijkstra, ax, orange)

path_astar, length_astar = a_star(parcour_graph, (0, 0), (99, 99), heuristic=euclidean_distance)
add_path(path_astar, ax, green)_____no_output_____%timeit path_dijkstra, length_dijstra = dijkstra(parcour_graph, (0, 0), (99, 99))_____no_output_____%timeit path_astar, length_astar = a_star(parcour_graph, (0, 0), (99, 99), manhattan_distance)_____no_output_____# a small maze, white lines are the routes that can be followed
# find the shortest path from (0, 0) to (99, 99)
small_maze = load_links('Data/small_maze.csv')

fig, ax = plt.subplots(figsize=(10, 10))

plot_parcour(small_maze, ax, line_width=15)
maze_graph = links_to_graph(small_maze)

path_dijkstra, length_dijstra = dijkstra(maze_graph, (0, 0), (19, 19))
add_path(path_dijkstra, ax, orange)

path_astar, length_astar = a_star(maze_graph, (0, 0), (19, 19), euclidean_distance)
add_path(path_astar, ax, green)_____no_output_____%timeit path_dijkstra, length_dijstra = dijkstra(maze_graph, (0, 0), (19, 19))_____no_output_____%timeit path_astar, length_astar = a_star(maze_graph, (0, 0), (19, 19), manhattan_distance)_____no_output_____# a small maze, white lines are the routes that can be followed
# find the shortest path from (0, 0) to (99, 99)
large_maze = load_links('Data/large_maze.csv')

fig, ax = plt.subplots(figsize=(15, 15))

plot_parcour(large_maze, ax)
maze_graph = links_to_graph(large_maze)

path_dijkstra, length_dijstra = dijkstra(maze_graph, (0, 0), (99, 99))
add_path(path_dijkstra, ax, orange)

path_astar, length_astar = a_star(maze_graph, (0, 0), (99, 99), euclidean_distance)
add_path(path_astar, ax, green)_____no_output_____%timeit path_dijkstra, length_dijstra = dijkstra(maze_graph, (0, 0), (99, 99))_____no_output_____%timeit path_astar, length_astar = a_star(maze_graph, (0, 0), (99, 99), manhattan_distance)_____no_output_____


# Notebook from aviadr1/nbdev
Path: nbs/03_export2html.ipynb

from nbdev import *
%nbdev_default_export export2html
%nbdev_default_class_level 3Cells will be exported to nbdev.export2html,
unless a different module is specified after an export flag: `%nbdev_export special.module`
#export
from nbdev.imports import *
from nbdev.sync import *
from nbdev.export import *
from nbdev.export import _mk_flag_re
from nbdev.showdoc import *
from nbdev.template import *

from html.parser import HTMLParser
from nbconvert.preprocessors import ExecutePreprocessor, Preprocessor
from nbconvert import HTMLExporter,MarkdownExporter
import traitlets_____no_output_____

# Convert to html

> The functions that transform the dev notebooks in the documentation of the library

- toc: true_____no_output_____The most important function defined in this module is `notebook2html`, so you may want to jump to it before scrolling though the rest, which explain the details behind the scenes of the conversion from notebooks to the html documentation. The main things to remember are:
- put a `#hide` or `%nbdev_hide` flag at the top of any cell you want to completely hide in the docs
- use the hide input [jupyter extension](https://github.com/ipython-contrib/jupyter_contrib_nbextensions) to hide the input of some cells (by default all `show_doc` cells have that marker added)
- you can define some jekyll metadata in the markdown cell with the title, see `get_metadata`
- use backsticks for terms you want automatic links to be found, but use `<code>` and `</code>` when you have homonyms and don't want those links
- you can define the default toc level of classes with `%nbdev_default_class_level` flag followed by a number (default is 2)
- you can add jekyll warnings, important or note banners with appropriate block quotes (see `add_jekyll_notes`)
- put any images you want to use in the images folder of your notebook folder, they will be automatically copied over to the docs folder
- put a `#hide_input` or `%nbdev_hide_input` flag at the top of a cell if you don't want code to be shown in the docs
    - cells containing `%nbdev_export` or `show_doc` have their code hidden automatically
- put a `#hide_output` or `%nbdev_hide_output` flag at the top of a cell if you don't want output to be shown in the docs
- use `%nbdev_collapse_input` or `%nbdev_collapse_output` to include code or output in the docs under a collapsable element_____no_output_____## Preprocessing notebook_____no_output_____### Cell processors_____no_output_____

#export
class HTMLParseAttrs(HTMLParser):
    "Simple HTML parser which stores any attributes in `attrs` dict"
    def handle_starttag(self, tag, attrs): self.tag,self.attrs = tag,dict(attrs)

    def attrs2str(self):
        "Attrs as string"
        return ' '.join([f'{k}="{v}"' for k,v in self.attrs.items()])

    def show(self):
        "Tag with updated attrs"
        return f'<{self.tag} {self.attrs2str()} />'

    def __call__(self, s):
        "Parse `s` and store attrs"
        self.feed(s)
        return self.attrs_____no_output_____h = HTMLParseAttrs()
t = h('')
test_eq(t['width'], '700')
test_eq(t['src'  ], 'src')_____no_output_____t['width'] = '600'
test_eq(h.show(), '')_____no_output_____t['max-width'] = t.pop('width')
test_eq(h.show(), '')_____no_output_____

The following functions are applied on individual cells as a preprocessing step before the conversion to html._____no_output_____

#export
def remove_widget_state(cell):
    "Remove widgets in the output of `cells`"
    if cell['cell_type'] == 'code' and 'outputs' in cell:
        cell['outputs'] = [l for l in cell['outputs']
                           if not ('data' in l and 'application/vnd.jupyter.widget-view+json' in l.data)]
    return cell_____no_output_____

Those outputs usually can't be rendered properly in html._____no_output_____

#export
# Note: `_re_show_doc` will catch show_doc even if it's commented out etc
_re_show_doc = re.compile(r"""
# Catches any show_doc and get the first argument in group 1
^\s*show_doc # line can start with any amount of whitespace followed by show_doc
\s*\(\s*     # Any number of whitespace, opening (, any number of whitespace
([^,\)\s]*)  # Catching group for any character but a comma, a closing ) or a whitespace
[,\)\s]      # A comma, a closing ) or a whitespace
""", re.MULTILINE | re.VERBOSE)

_re_show_doc_magic = _mk_flag_re(True, 'show_doc', -1,
    "# Catches a cell with %nbdev_show_doc \*\* and get that \*\* in group 1")

_re_hide_input = [
    _mk_flag_re(False, 'export', (0,1),
        "Matches any cell that has `#export in it"),
    _mk_flag_re(False, '(hide_input|hide-input)', 0,
        "Matches any cell that has `#hide_input` or `#hide-input` in it"),
    _mk_flag_re(True, 'export', (0,1),
        "Matches any cell that has `%nbdev_export` in it"),
    _mk_flag_re(True, 'hide_input', 0,
        "Matches any cell that has `%nbdev_hide_input` in it")]

_re_hide_output = [
    _mk_flag_re(False, '(hide_output|hide-output)', 0,
        "Matches any cell that has `#hide_output` or `#hide-output` in it"),
    _mk_flag_re(True, 'hide_output', 0,
        "Matches any cell that has `%nbdev_hide_output` in it")]_____no_output_____#export
def upd_metadata(cell, key, value=True):
    "Sets `key` to `value` on the `metadata` of `cell` without replacing metadata"
    cell.setdefault('metadata',{})[key] = value_____no_output_____#export
def hide_cells(cell):
    "Hide inputs of `cell` that need to be hidden"
    if check_re_multi(cell, [_re_show_doc, _re_show_doc_magic, *_re_hide_input]): upd_metadata(cell, 'hide_input')
    elif check_re_multi(cell, _re_hide_output): upd_metadata(cell, 'hide_output')
    return cell_____no_output_____

This concerns all the cells with `%nbdev_export` or `%nbdev_hide_input` flags and all the cell containing a `show_doc` for a function or class._____no_output_____

for source in ['show_doc(read_nb)', '# export\nfrom local.core import *', '# hide_input\n2+2',
              'line1\n show_doc (read_nb) \nline3', '# export with.mod\nfrom local.core import *',
              '%nbdev_export with.mod\nfrom local.core import *', 'line1\n%nbdev_export\nline3',
              '%nbdev_show_doc read_nb', '%nbdev_hide_input\n2+2']:
    cell = {'cell_type': 'code', 'source': source}
    cell1 = hide_cells(cell.copy())
    assert 'metadata' in cell1
    assert 'hide_input' in cell1['metadata']
    assert cell1['metadata']['hide_input']

for flag in ['# exports', '%nbdev_export_and_show']:
    cell = {'cell_type': 'code', 'source': f'{flag}\nfrom local.core2 import *'}
    test_eq(hide_cells(cell.copy()), cell)_____no_output_____

This concerns all the cells with a `%nbdev_hide_output` flag._____no_output_____

for source in ['# hide-output\nfrom local.core import *', '# hide_output\n2+2', '%nbdev_hide_output\na=b']:
    cell = {'cell_type': 'code', 'source': source}
    cell1 = hide_cells(cell.copy())
    assert 'metadata' in cell1
    assert 'hide_output' in cell1['metadata']
    assert cell1['metadata']['hide_output']

cell = {'cell_type': 'code', 'source': '# hide-outputs\nfrom local.core import *'}
test_eq(hide_cells(cell.copy()), cell)_____no_output_____#export
def clean_exports(cell):
    "Remove all flags from code `cell`s"
    if cell['cell_type'] == 'code':
        cell['source'] = split_flags_and_code(cell, str)[1]
    return cell_____no_output_____

The rest of the cell is displayed without any modification._____no_output_____

for flag in ['# exports', '%nbdev_export_and_show']:
    cell = {'cell_type': 'code', 'source': f'{flag}\nfrom local.core import *'}
    test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': 'from local.core import *'})
    cell['cell_type'] = 'markdown'
    test_eq(clean_exports(cell.copy()), cell)
    cell = {'cell_type': 'code', 'source': f'{flag} core\nfrom local.core import *'}
    test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': 'from local.core import *'})

cell = {'cell_type': 'code', 'source': f'# comment \n# exports\nprint("something")'}
test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': '# exports\nprint("something")'})
cell = {'cell_type': 'code', 'source': f'# comment \n%nbdev_export_and_show\nprint("something")'}
test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': 'print("something")'})_____no_output_____#export
def treat_backticks(cell):
    "Add links to backticks words in `cell`"
    if cell['cell_type'] == 'markdown': cell['source'] = add_doc_links(cell['source'])
    return cell_____no_output_____cell = {'cell_type': 'markdown', 'source': 'This is a `DocsTestClass`'}
test_eq(treat_backticks(cell), {'cell_type': 'markdown',
    'source': 'This is a [`DocsTestClass`](/export.html#DocsTestClass)'})_____no_output_____#export
_re_nb_link = re.compile(r"""
# Catches any link to a local notebook and keeps the title in group 1, the link without .ipynb in group 2
\[          # Opening [
([^\]]*)    # Catching group for any character except ]
\]\(        # Closing ], opening (
([^http]    # Catching group that must not begin by html (local notebook)
[^\)]*)     # and containing anything but )
.ipynb\)    # .ipynb and closing )
""", re.VERBOSE)_____no_output_____#export
_re_block_notes = re.compile(r"""
# Catches any pattern > Title: content with title in group 1 and content in group 2
^\s*>\s*     # > followed by any number of whitespace
([^:]*)      # Catching group for any character but :
:\s*         # : then any number of whitespace
([^\n]*)     # Catching group for anything but a new line character
(?:\n|$)     # Non-catching group for either a new line or the end of the text
""", re.VERBOSE | re.MULTILINE)_____no_output_____#export
def _to_html(text):
    return text.replace("'", "’")_____no_output_____#export
def add_jekyll_notes(cell):
    "Convert block quotes to jekyll notes in `cell`"
    styles = Config().get('jekyll_styles', 'note,warning,tip,important').split(',')
    def _inner(m):
        title,text = m.groups()
        if title.lower() not in styles: return f"> {title}:{text}"
        return '{% include '+title.lower()+".html content=\'"+_to_html(text)+"\' %}"
    if cell['cell_type'] == 'markdown':
        cell['source'] = _re_block_notes.sub(_inner, cell['source'])
    return cell_____no_output_____

Supported styles are `Warning`, `Note` `Tip` and `Important`:

Typing `> Warning: There will be no second warning!` will render in the docs:

> Warning: There will be no second warning!

Typing `> Important: Pay attention! It's important.` will render in the docs:

> Important: Pay attention! It's important.

Typing `> Tip: This is my tip.` will render in the docs:

> Tip: This is my tip.

Typing `> Note: Take note of this.` will render in the docs:

> Note: Take note of this.

Typing ``> Note: A doc link to `add_jekyll_notes` should also work fine.`` will render in the docs:

> Note: A doc link to `add_jekyll_notes` should also work fine._____no_output_____

%nbdev_hide
for w in ['Warning', 'Note', 'Important', 'Tip', 'Bla']:
    cell = {'cell_type': 'markdown', 'source': f"> {w}: This is my final {w.lower()}!"}
    res = '{% include '+w.lower()+'.html content=\'This is my final '+w.lower()+'!\' %}'
    if w != 'Bla': test_eq(add_jekyll_notes(cell), {'cell_type': 'markdown', 'source': res})
    else: test_eq(add_jekyll_notes(cell), cell)_____no_output_____%nbdev_hide
cell = {'cell_type': 'markdown', 'source': f"> This is a link, don't break me! https://my.link.com"}
test_eq(add_jekyll_notes(cell.copy()), cell)_____no_output_____#export
_re_image = re.compile(r"""
# Catches any image file used, either with `![alt](image_file)` or `<img src="image_file">`
^(!\[           #   Beginning of line (since re.MULTILINE is passed) followed by ![ in a catching group
[^\]]*          #   Anything but ]
\]\()           #   Closing ] and opening (, end of the first catching group
[ \t]*          #   Whitespace before the image path
([^\) \t]*)     #   Catching block with any character that is not ) or whitespace
(\)| |\t)       #   Catching group with closing ) or whitespace
|               # OR
^(]*>) #   Catching group with
""", re.MULTILINE | re.VERBOSE)_____no_output_____m=_re_image.search('![Alt](images/logo.png)')
test_eq(m.groups(), ('![Alt](', 'images/logo.png', ')', None))
# using ) or whitespace to close the group means we don't need a special case for captions
m=_re_image.search('![Alt](images/logo.png "caption (something)")')
test_eq(m.groups(), ('![Alt](', 'images/logo.png', '', None))_____no_output_____#export
def _img2jkl(d, h, jekyll=True):
    if not jekyll: return ''
    if 'width' in d: d['max-width'] = d.pop('width')
    if 'src' in d:   d['file'] = d.pop('src')
    return '{% include image.html ' + h.attrs2str() + ' %}'_____no_output_____#export
def _is_real_image(src):
    return not (src.startswith('http://') or src.startswith('https://') or src.startswith('data:image/'))_____no_output_____#export
def copy_images(cell, fname, dest, jekyll=True):
    "Copy images referenced in `cell` from `fname` parent folder to `dest` folder"
    def _rep_src(m):
        grps = m.groups()
        if grps[3] is not None:
            h = HTMLParseAttrs()
            dic = h(grps[3])
            src = dic['src']
        else: src = grps[1]
        if _is_real_image(src):
            os.makedirs((Path(dest)/src).parent, exist_ok=True)
            shutil.copy(Path(fname).parent/src, Path(dest)/src)
            src = Config().doc_baseurl + src
        if grps[3] is not None:
            dic['src'] = src
            return _img2jkl(dic, h, jekyll=jekyll)
        else: return f"{grps[0]}{src}{grps[2]}"
    if cell['cell_type'] == 'markdown': cell['source'] = _re_image.sub(_rep_src, cell['source'])
    return cell_____no_output_____

This is to ensure that all images defined in `nbs_folder/images` and used in notebooks are copied over to `doc_folder/images`._____no_output_____

dest_img = Config().doc_path/'images'/'logo.png'
cell = {'cell_type': 'markdown', 'source':'Text\n![Alt](images/logo.png)'}
try:
    copy_images(cell, Path('01_export.ipynb'), Config().doc_path)
    test_eq(cell["source"], 'Text\n![Alt](/images/logo.png)')
    #Image has been copied
    assert dest_img.exists()
    cell = {'cell_type': 'markdown', 'source':'Text\n![Alt](images/logo.png "caption (something)")'}
    copy_images(cell, Path('01_export.ipynb'), Config().doc_path)
    test_eq(cell["source"], 'Text\n![Alt](/images/logo.png "caption (something)")')
finally: dest_img.unlink()_____no_output_____%nbdev_hide
cell = {'cell_type': 'markdown', 'source':'Text\n![Alt](https://site.logo.png)'}
copy_images(cell, Path('01_export.ipynb'), Config().doc_path)
test_eq(cell["source"], 'Text\n![Alt](https://site.logo.png)')
cell = {'cell_type': 'markdown', 'source':'Text\n![Alt](https://site.logo.png "caption")'}
copy_images(cell, Path('01_export.ipynb'), Config().doc_path)
test_eq(cell["source"], 'Text\n![Alt](https://site.logo.png "caption")')_____no_output_____%nbdev_hide
cell = {'cell_type': 'markdown', 'source': 'Text\n'}
try:
    copy_images(cell, Path('01_export.ipynb'), Config().doc_path)
    test_eq(cell["source"], 'Text\n{% include image.html alt="alt" caption="cap" max-width="600" file="/images/logo.png" %}')
    assert dest_img.exists()
finally: dest_img.unlink()_____no_output_____%nbdev_hide
cell = {'cell_type': 'markdown', 'source': 'Text\n'}
copy_images(cell, Path('01_export.ipynb'), Config().doc_path)
test_eq(cell["source"], 'Text\n{% include image.html alt="alt" caption="cap" max-width="600" file="http://site.logo.png" %}')_____no_output_____#export
def _relative_to(path1, path2):
    p1,p2 = Path(path1).absolute().parts,Path(path2).absolute().parts
    i=0
    while i <len(p1) and i<len(p2) and p1[i] == p2[i]: i+=1
    p1,p2 = p1[i:],p2[i:]
    return os.path.sep.join(['..' for _ in p2] + list(p1))_____no_output_____%nbdev_hide
test_eq(_relative_to(Path('images/logo.png'), Config().doc_path), '../nbs/images/logo.png')
test_eq(_relative_to(Path('images/logo.png'), Config().doc_path.parent), 'nbs/images/logo.png')_____no_output_____#export
def adapt_img_path(cell, fname, dest, jekyll=True):
    "Adapt path of images referenced in `cell` from `fname` to work in folder `dest`"
    def _rep(m):
        gps = m.groups()
        if gps[0] is not None:
            start,img,end = gps[:3]
            if not (img.startswith('http:/') or img.startswith('https:/')):
                img = _relative_to(fname.parent/img, dest)
            return f'{start}{img}{end}'
        else:
            h = HTMLParseAttrs()
            dic = h(gps[3])
            if not (dic['src'].startswith('http:/') or dic['src'].startswith('https:/')):
                dic['src'] = _relative_to(fname.parent/dic['src'], dest)
            return _img2jkl(dic, h, jekyll=jekyll)
    if cell['cell_type'] == 'markdown': cell['source'] = _re_image.sub(_rep, cell['source'])
    return cell_____no_output_____

This function is slightly different as it ensures that a notebook convert to a file that will be placed in `dest` will have the images location updated. It is used for the `README.md` file (generated automatically from the index) since the images are copied inside the github repo, but in general, you should make sure your images are going to be accessible from the location your file ends up being._____no_output_____

cell = {'cell_type': 'markdown', 'source': 'Text\n![Alt](images/logo.png)'}
cell1 = adapt_img_path(cell, Path('01_export.ipynb'), Path('.').absolute().parent)
test_eq(cell1['source'], 'Text\n![Alt](nbs/images/logo.png)')

cell = {'cell_type': 'markdown', 'source': 'Text\n![Alt](http://site.logo.png)'}
cell1 = adapt_img_path(cell, Path('01_export.ipynb'), Path('.').absolute().parent)
test_eq(cell1['source'], 'Text\n![Alt](http://site.logo.png)')

cell = {'cell_type': 'markdown',
        'source': 'Text\n'}
cell1 = adapt_img_path(cell, Path('01_export.ipynb'), Path('.').absolute().parent)
test_eq(cell1['source'], 'Text\n{% include image.html alt="Logo" max-width="600" file="nbs/images/logo.png" %}')

cell = {'cell_type': 'markdown',
        'source': 'Text\n'}
cell1 = adapt_img_path(cell, Path('01_export.ipynb'), Path('.').absolute().parent)
test_eq(cell1['source'], 'Text\n{% include image.html alt="Logo" max-width="600" file="https://site.image.png" %}')_____no_output_____

Escape Latex in liquid_____no_output_____

#export
_re_latex = re.compile(r'^(\$\$.*\$\$)$', re.MULTILINE)_____no_output_____#export
def escape_latex(cell):
    if cell['cell_type'] != 'markdown': return cell
    cell['source'] = _re_latex.sub(r'{% raw %}\n\1\n{% endraw %}', cell['source'])
    return cell_____no_output_____cell = {'cell_type': 'markdown',
        'source': 'lala\n$$equation$$\nlala'}
cell = escape_latex(cell)
test_eq(cell['source'], 'lala\n{% raw %}\n$$equation$$\n{% endraw %}\nlala')_____no_output_____

### Collapsable Code Cells
_____no_output_____

#export
_re_cell_to_collapse_closed = [
    _mk_flag_re(False, '(collapse|collapse_hide|collapse-hide)', 0,
        "Matches any cell with #collapse or #collapse_hide"),
    _mk_flag_re(True, 'collapse_input', 0,
        "Matches any cell with %nbdev_collapse_input")]

_re_cell_to_collapse_open = [
    _mk_flag_re(False, '(collapse_show|collapse-show)', 0,
        "Matches any cell with #collapse_show"),
    _mk_flag_re(True, r'collapse_input[ \t]+open', 0,
        "Matches any cell with %nbdev_collapse_input open")]

_re_cell_to_collapse_output = [
    _mk_flag_re(False, '(collapse_output|collapse-output)', 0,
        "Matches any cell with #collapse_output"),
    _mk_flag_re(True, 'collapse_output', 0,
        "Matches any cell with %nbdev_collapse_output")]_____no_output_____#export
def collapse_cells(cell):
    "Add a collapse button to inputs or outputs of `cell` in either the open or closed position"
    if check_re_multi(cell, _re_cell_to_collapse_closed): upd_metadata(cell,'collapse_hide')
    elif check_re_multi(cell, _re_cell_to_collapse_open): upd_metadata(cell,'collapse_show')
    elif check_re_multi(cell, _re_cell_to_collapse_output): upd_metadata(cell,'collapse_output')
    return cell_____no_output_____%nbdev_hide
for flag in [
    ('collapse_hide', '#collapse'),
    ('collapse_hide', '# collapse_hide'),
    ('collapse_hide', ' # collapse-hide'),
    ('collapse_hide', '%nbdev_collapse_input'),
    ('collapse_show', '#collapse_show'),
    ('collapse_show', '#collapse-show'),
    ('collapse_show', '%nbdev_collapse_input    open'),
    ('collapse_output', ' #collapse_output'),
    ('collapse_output', '#collapse-output'),
    ('collapse_output', '%nbdev_collapse_output ')]:
    cell = nbformat.v4.new_code_cell(f'#comment\n{flag[1]} \ndef some_code')
    test_eq(True, collapse_cells(cell)['metadata'][flag[0]])_____no_output_____%nbdev_hide
# check that we can't collapse both input and output
cell = nbformat.v4.new_code_cell(f'%nbdev_collapse_input\n%nbdev_collapse_output \ndef some_code')
test_eq({'collapse_hide': True}, collapse_cells(cell)['metadata'])
cell = nbformat.v4.new_code_cell(f'%nbdev_collapse_input open\n#collapse_output \ndef some_code')
test_eq({'collapse_show': True}, collapse_cells(cell)['metadata'])
# check that we can hide input and collapse output
cell = nbformat.v4.new_code_cell(f'%nbdev_hide_input\n%nbdev_collapse_output \ndef some_code')
test_eq({'hide_input': True, 'collapse_output': True}, hide_cells(collapse_cells(cell))['metadata'])
cell = nbformat.v4.new_code_cell(f'#hide-input\n#collapse_output \ndef some_code')
test_eq({'hide_input': True, 'collapse_output': True}, hide_cells(collapse_cells(cell))['metadata'])_____no_output_____

- Placing `%nbdev_collapse_input open` in a code cell will inlcude your code under a collapsable element that is **open** by default._____no_output_____

%nbdev_collapse_input open
print('This code cell is not collapsed by default but you can collapse it to hide it from view!')
print("Note that the output always shows with `%nbdev_collapse_input`.")This code cell is not collapsed by default but you can collapse it to hide it from view!
Note that the output always shows with `%nbdev_collapse_input`.

- Placing `%nbdev_collapse_input` in a code cell will include your code in a collapsable element that is **closed** by default.  For example:_____no_output_____

%nbdev_collapse_input
print('The code cell that produced this output is collapsed by default but you can expand it!')The code cell that produced this output is collapsed by default but you can expand it!

- Placing `%nbdev_collapse_output` in a code cell will hide the output under a collapsable element that is **closed** by default._____no_output_____

%nbdev_collapse_output
print('The input of this cell is visible as usual.\nHowever, the OUTPUT of this cell is collapsed by default but you can expand it!')The input of this cell is visible as usual.
However, the OUTPUT of this cell is collapsed by default but you can expand it!

### Preprocessing the list of cells_____no_output_____The following functions are applied to the entire list of cells of the notebook as a preprocessing step before the conversion to html._____no_output_____

#export
_re_hide = [
    _mk_flag_re(False, 'hide', 0, 'Matches any cell with #hide'),
    _mk_flag_re(True, 'hide', 0, 'Matches any cell with %nbdev_hide')]
_re_all_flag = ReTstFlags(True)
_re_cell_to_remove = [
    _mk_flag_re(False, '(default_exp|exporti)', (0,1),
        'Matches any cell with #default_exp or #exporti'),
    _mk_flag_re(True, '(default_export|export_internal)', (0,1),
        'Matches any cell with %nbdev_default_export or %nbdev_export_internal')]
_re_default_cls_lvl = [
    _mk_flag_re(False, 'default_cls_lvl', 1, "Matches any cell with #default_cls_lvl"),
    _mk_flag_re(True, 'default_class_level', 1, "Matches any cell with %nbdev_default_class_level"),
]_____no_output_____#export
def remove_hidden(cells):
    "Remove in `cells` the ones with a flag `#hide`, `#default_exp`, `#default_cls_lvl` or `#exporti`"
    def _hidden(cell):
        "Check if `cell` should be hidden"
        if check_re_multi(cell, _re_hide, code_only=False): return True
        if check_re_multi(cell, [_re_all_flag, *_re_cell_to_remove, *_re_default_cls_lvl]): return True
        return False
    return [c for c in cells if not _hidden(c)]_____no_output_____cells = [{'cell_type': 'code', 'source': source, 'hide': hide} for hide, source in [
    (False, '# export\nfrom local.core import *'),
    (True, '# export\n%nbdev_default_class_level 4\nfrom local.core import *'),
    (True, ' # default_cls_lvl 2 \n# export\nfrom local.core import *'),
    (False, '%nbdev_export \nfrom local.core import *'),
    (False, '# exporti mod file'), # Note: this used to get removed but we're more strict now
    (False, '%nbdev_export_internal mod file'),
    (True, '# hide\nfrom local.core import *'),
    (True, '%nbdev_hide\nfrom local.core import *'),
    (False, '# hide_input\nfrom local.core import *'),
    (False, '%nbdev_hide_output\nfrom local.core import *'),
    (False, '#exports\nsuper code'),
    (False, '%nbdev_export_and_show\nsuper code'),
    (True, '#default_exp notebook.export'),
    (True, '%nbdev_default_export notebook.export'),
    (False, 'show_doc(read_nb)'),
    (True, '#default_cls_lvl 3'),
    (False, '#all_foo'), # foo is not in setting.ini tst_flags
    (False, '%nbdev_foo_test all'),
    (True, '#all_fastai'),
    (True, '%nbdev_fastai_test all'),
    (False, '#hide (last test of to_concat)'),
    (True, '# exporti\n1 + 1'),
    (True, '%nbdev_export_internal\n1 + 1')]] + [
    {'cell_type': 'markdown', 'source': source, 'hide': hide} for hide, source in [
    (False, '#hide_input\nnice'),
    (False, '%nbdev_hide_input\nnice'),
    (True, '#hide\n\nto hide'),
    (True, '#comment\n%nbdev_hide\nto hide')]]

for a,b in zip([cell for cell in cells if not cell['hide']], remove_hidden(cells)):
    test_eq(a,b)_____no_output_____#export
def find_default_level(cells):
    "Find in `cells` the default class level."
    for cell in cells:
        tst = check_re_multi(cell, _re_default_cls_lvl)
        if tst: return int(tst.groups()[0])
    return 2_____no_output_____tst_nb = read_nb('00_export.ipynb')
test_eq(find_default_level(tst_nb['cells']), 3)_____no_output_____#export
_re_export = _mk_flag_re(False, "exports?", (0,1),
    "Matches any line with #export or #exports with or without module name")
_re_export_magic = _mk_flag_re(True, "export(|_and_show)", (0,1),
    "Matches any line with %nbdev_export or %nbdev_export_and_show with or without module name")_____no_output_____#export
def nb_code_cell(source):
    "A code cell (as a dict) containing `source`"
    return {'cell_type': 'code', 'execution_count': None, 'metadata': {}, 'outputs': [], 'source': source}_____no_output_____#export
def _show_doc_cell(name, cls_lvl=None):
    return nb_code_cell(f"show_doc({name}{'' if cls_lvl is None else f', default_cls_level={cls_lvl}'})")

def add_show_docs(cells, cls_lvl=None):
    "Add `show_doc` for each exported function or class"
    res, documented, documented_wild = [], [], []
    for cell in cells:
        m = check_re_multi(cell, [_re_show_doc, _re_show_doc_magic])
        if not m: continue
        if m.re is _re_show_doc:
            documented.append(m.group(1))
        else:
            names, wild_names, kwargs = parse_nbdev_show_doc(m.group(1))
            documented.extend(names)
            documented_wild.extend(wild_names)

    def _documented(name):
        if name in documented: return True
        # assume that docs will have been shown for all members of everything in documented_wild
        if name.rfind('.') != -1 and name[0:name.rfind('.')] in documented_wild: return True

    for cell in cells:
        res.append(cell)
        if check_re_multi(cell, [_re_export, _re_export_magic]):
            for n in export_names(cell['source'], func_only=True):
                if not _documented(n): res.append(_show_doc_cell(n, cls_lvl=cls_lvl))
    return res_____no_output_____

This only adds cells with a `show_doc` for non-documented functions, so if you add yourself a `show_doc` cell (because you want to change one of the default argument), there won't be any duplicates._____no_output_____

for i,cell in enumerate(tst_nb['cells']):
    if cell['source'].startswith('#export\ndef read_nb'): break
tst_cells = [c.copy() for c in tst_nb['cells'][i-1:i+1]]
added_cells = add_show_docs(tst_cells, cls_lvl=3)
test_eq(len(added_cells), 3)
test_eq(added_cells[0], tst_nb['cells'][i-1])
test_eq(added_cells[1], tst_nb['cells'][i])
test_eq(added_cells[2], _show_doc_cell('read_nb', cls_lvl=3))
test_eq(added_cells[2]['source'], 'show_doc(read_nb, default_cls_level=3)')

for flag in ['#export', '%nbdev_export', '#exports', '%nbdev_export_and_show']:
    for show_doc_source in [
            ('show_doc(my_func)', 'show_doc(my_func, title_level=3)'),
            ('%nbdev_show_doc my_func', '%nbdev_show_doc my_func, title_level=3')]:
        #Check show_doc isn't added if it was already there.
        tst_cells1 = [{'cell_type':'code', 'source': f'{flag}\ndef my_func(x):\n    return x'},
                      {'cell_type':'code', 'source': show_doc_source[0]}]
        test_eq(add_show_docs(tst_cells1), tst_cells1)
        #Check show_doc is added
        test_eq(len(add_show_docs(tst_cells1[:-1])), len(tst_cells1))
        tst_cells1 = [{'cell_type':'code', 'source': f'{flag} with.mod\ndef my_func(x):\n    return x'},
                      {'cell_type':'markdown', 'source': 'Some text'},
                      {'cell_type':'code', 'source': show_doc_source[1]}]
        test_eq(add_show_docs(tst_cells1), tst_cells1)
        #Check show_doc is added when using mod export
        test_eq(len(add_show_docs(tst_cells1[:-1])), len(tst_cells1))_____no_output_____%nbdev_hide
# `add_show_docs` should understand wildcard and multi-element `%nbdev_show_doc` calls
test_code_source = """%nbdev_export
class A:
    def a(self): pass
@patch
def b(x:A): pass
"""
for show_doc_source in ["%nbdev_show_doc A *", "%nbdev_show_doc A . a b", "%nbdev_show_doc A A.b"]:
    tst_cells1 = [nbformat.v4.new_code_cell(test_code_source), nbformat.v4.new_code_cell(show_doc_source)]
    test_eq(add_show_docs(tst_cells1), tst_cells1)_____no_output_____#export
_re_fake_header = re.compile(r"""
# Matches any fake header (one that ends with -)
\#+    # One or more #
\s+    # One or more of whitespace
.*     # Any char
-\s*   # A dash followed by any number of white space
$      # End of text
""", re.VERBOSE)_____no_output_____#export
def remove_fake_headers(cells):
    "Remove in `cells` the fake header"
    return [c for c in cells if c['cell_type']=='code' or _re_fake_header.search(c['source']) is None]_____no_output_____

You can fake headers in your notebook to navigate them more easily with collapsible headers, just make them finish with a dash and they will be removed. One typical use case is to have a header of level 2 with the name of a class, since the `show_doc` cell of that class will create the same anchor, you need to have the one you created manually disappear to avoid any duplicate._____no_output_____

cells = [{'cell_type': 'markdown',
          'metadata': {},
          'source': '### Fake-'}] + tst_nb['cells'][:10]
cells1 = remove_fake_headers(cells)
test_eq(len(cells1), len(cells)-1)
test_eq(cells1[0], cells[1])_____no_output_____#export
def remove_empty(cells):
    "Remove in `cells` the empty cells"
    return [c for c in cells if len(c['source']) >0]_____no_output_____

### Grabbing metada_____no_output_____

#export
_re_title_summary = re.compile(r"""
# Catches the title and summary of the notebook, presented as # Title > summary, with title in group 1 and summary in group 2
^\s*       # Beginning of text followe by any number of whitespace
\#\s+      # # followed by one or more of whitespace
([^\n]*)   # Catching group for any character except a new line
\n+        # One or more new lines
>[ ]*       # > followed by any number of whitespace
([^\n]*)   # Catching group for any character except a new line
""", re.VERBOSE)

_re_title_only = re.compile(r"""
# Catches the title presented as # Title without a summary
^\s*       # Beginning of text followe by any number of whitespace
\#\s+      # # followed by one or more of whitespace
([^\n]*)   # Catching group for any character except a new line
(?:\n|$)    # New line or end of text
""", re.VERBOSE)

_re_properties = re.compile(r"""
^-\s+      # Beginnig of a line followed by - and at least one space
(.*?)      # Any pattern (shortest possible)
\s*:\s*    # Any number of whitespace, :, any number of whitespace
(.*?)$     # Any pattern (shortest possible) then end of line
""", re.MULTILINE | re.VERBOSE)

_re_mdlinks = re.compile(r"\[(.+)]\((.+)\)", re.MULTILINE)_____no_output_____#export
def _md2html_links(s):
    'Converts markdown links to html links'
    return _re_mdlinks.sub(r"\1", s)_____no_output_____#export
def get_metadata(cells):
    "Find the cell with title and summary in `cells`."
    for i,cell in enumerate(cells):
        if cell['cell_type'] == 'markdown':
            match = _re_title_summary.match(cell['source'])
            if match:
                cells.pop(i)
                attrs = {k:v for k,v in _re_properties.findall(cell['source'])}
                return {'keywords': 'fastai',
                        'summary' : _md2html_links(match.groups()[1]),
                        'title'   : match.groups()[0],
                        **attrs}
            elif _re_title_only.search(cell['source']) is not None:
                title = _re_title_only.search(cell['source']).groups()[0]
                cells.pop(i)
                attrs = {k:v for k,v in _re_properties.findall(cell['source'])}
                return {'keywords': 'fastai',
                        'title'   : title,
                        **attrs}

    return {'keywords': 'fastai',
            'title'   : 'Title'}_____no_output_____

In the markdown cell with the title, you can add the summary as a block quote (just put an empty block quote for an empty summary) and a list with any additional metadata you would like to add, for instance:
```
# Title

> Awesome summary
- toc: False
```

The toc: False metadata will prevent the table of contents from showing on the page._____no_output_____

tst_nb = read_nb('00_export.ipynb')
test_eq(get_metadata(tst_nb['cells']), {
    'keywords': 'fastai',
    'summary': 'The functions that transform notebooks in a library',
    'title': 'Export to modules'})

#The cell with the metada is popped out, so if we do it a second time we get the default.
test_eq(get_metadata(tst_nb['cells']), {'keywords': 'fastai', 'title'   : 'Title'})_____no_output_____%nbdev_hide
#test with title only
test_eq(get_metadata([{'cell_type': 'markdown', 'source': '# Awesome title'}]),
        {'keywords': 'fastai', 'title': 'Awesome title'})_____no_output_____%nbdev_hide
text = r"""
[This](https://nbdev.fast.ai) goes to docs.
This [one:here](00_export.ipynb) goes to a local nb.
\n[And-this](http://dev.fast.ai/) goes to fastai docs
"""
res = """
This goes to docs.\nThis one:here goes to a local nb. \n\\nAnd-this goes to fastai docs
"""
test_eq(_md2html_links(text), res)_____no_output_____%nbdev_hide
cells = [{'cell_type': 'markdown', 'source': "# Title\n\n> s\n\n- toc: false"}]
test_eq(get_metadata(cells), {'keywords': 'fastai', 'summary': 's', 'title': 'Title', 'toc': 'false'})_____no_output_____

## Executing show_doc cells_____no_output_____

#export
_re_mod_export = _mk_flag_re(False, "export[s]?", 1,
    "Matches any line with #export or #exports with a module name and catches it in group 1")
_re_mod_export_magic = _mk_flag_re(True, "export(?:|_and_show)", 1,
    "Matches any line with %nbdev_export or %nbdev_export_and_show catching module name in group 1")

def _gather_export_mods(cells):
    res = []
    for cell in cells:
        tst = check_re_multi(cell, [_re_mod_export, _re_mod_export_magic])
        if tst is not None: res.append(tst.groups()[0])
    return res_____no_output_____%nbdev_hide
cells = [
    {'cell_type': 'markdown', 'source': '#export ignored'},
    {'cell_type': 'code', 'source': '#export'},
    {'cell_type': 'code', 'source': '#export normal'},
    {'cell_type': 'code', 'source': '# exports show'},
    {'cell_type': 'code', 'source': '# exporti hidden'},
    {'cell_type': 'code', 'source': '#export\n@call_parse'},
    {'cell_type': 'code', 'source': '#export \n@delegates(concurrent.futures.ProcessPoolExecutor)'}
]

test_eq(_gather_export_mods(cells), ['normal', 'show'])
for cell in cells: cell['source'] = cell['source'].replace('#export', '%nbdev_export')
test_eq(_gather_export_mods(cells), ['normal', 'show'])
for cell in cells: cell['source'] = cell['source'].replace('# exports', '%nbdev_export_and_show')
test_eq(_gather_export_mods(cells), ['normal', 'show'])
for cell in cells: cell['source'] = cell['source'].replace('# exporti', '%nbdev_export_internal')
test_eq(_gather_export_mods(cells), ['normal', 'show'])_____no_output_____#export
# match any cell containing a zero indented import from the current lib
_re_lib_import = ReLibName(r"^from LIB_NAME\.", re.MULTILINE)
# match any cell containing a zero indented import
_re_import = re.compile(r"^from[ \t]+\S+[ \t]+import|^import[ \t]", re.MULTILINE)
# match any cell containing a zero indented call to notebook2script
_re_notebook2script = re.compile(r"^notebook2script\(", re.MULTILINE)_____no_output_____%nbdev_hide
for cell in [nbformat.v4.new_code_cell(s, metadata={'exp': exp}) for exp,s in [
    (True, 'show_doc(Tensor.p)'),
    (True, ' show_doc(Tensor.p)'),
    (True, '%nbdev_show_doc Tensor.p'),
    (True, 'if somthing:\n    show_doc(Tensor.p)'),
    (False, '# show_doc(Tensor.p)'),
    (True, '# comment \n show_doc(Tensor.p)'),
    (True, '"""\nshow_doc(Tensor.p)\n"""'),
    (True, 'import torch\nshow_doc(Tensor.p)'),
    (False,'class Ex(ExP):\n"An `ExP` that ..."\ndef preprocess_cell(self, cell, resources, index):\n'),
    (False, 'from somewhere import something'),
    (False, 'from '),
    (False, 'import re'),
    (False, 'import '),
    (False, 'try: from PIL import Image\except: pass'),
    (False, 'from PIL import Image\n@patch\ndef p(x:Image):\n    pass'),
    (False, '@patch\ndef p(x:Image):\n    pass\nfrom PIL import Image')]]:
    exp = cell.metadata.exp
    assert exp == bool(check_re_multi(cell, [_re_show_doc, _re_show_doc_magic, _re_lib_import.re])), f'expected {exp} for {cell}'_____no_output_____%nbdev_hide
for cell in [nbformat.v4.new_code_cell(s, metadata={'exp': exp}) for exp,s in [
    (False, 'show_doc(Tensor.p)'),
    (True, 'import torch\nshow_doc(Tensor.p)'),
    (False,'class Ex(ExP):\n"An `ExP` that ..."\ndef preprocess_cell(self, cell, resources, index):\n'),
    (False, ' from somewhere import something'),
    (True, 'from somewhere import something'),
    (False, 'from '),
    (False, 'select * \nfrom database'),
    (False, ' import re'),
    (True, 'import re'),
    (True, 'import '),
    (False, 'try: from PIL import Image\except: pass'),
    (True, 'from PIL import Image\n@patch\ndef p(x:Image):\n    pass'),
    (True, '@patch\ndef p(x:Image):\n    pass\nfrom PIL import Image')]]:
    exp = cell.metadata.exp
    assert exp == bool(check_re(cell, _re_import)), f'expected {exp} for {cell}'_____no_output_____%nbdev_hide
for cell in [nbformat.v4.new_code_cell(s, metadata={'exp': exp}) for exp,s in [
    (False, 'show_doc(Tensor.p)'),
    (False, 'notebook2script'),
    (False, '#notebook2script()'),
    (True, 'notebook2script()'),
    (True, 'notebook2script(anything at all)')]]:
    exp = cell.metadata.exp
    assert exp == bool(check_re(cell, _re_notebook2script)), f'expected {exp} for {cell}'_____no_output_____#export
def _non_comment_code(s):
    if re.match(r'\s*#', s): return False
    if _re_import.findall(s) or _re_lib_import.re.findall(s): return False
    return re.match(r'\s*\w', s)_____no_output_____#export
class ExecuteShowDocPreprocessor(ExecutePreprocessor):
    "An `ExecutePreprocessor` that only executes `show_doc` and `import` cells"
    def preprocess_cell(self, cell, resources, index):
        if not check_re(cell, _re_notebook2script):
            if check_re_multi(cell, [_re_show_doc, _re_show_doc_magic]):
                return super().preprocess_cell(cell, resources, index)
            elif check_re_multi(cell, [_re_import, _re_lib_import.re]):
                if check_re_multi(cell, [_re_export, _re_export_magic, 'show_doc', '^\s*#\s*import']):
#                     r = list(filter(_non_comment_code, cell['source'].split('\n')))
#                     if r: print("You have import statements mixed with other code", r)
                    return super().preprocess_cell(cell, resources, index)
#                 try: return super().preprocess_cell(cell, resources, index)
#                 except: pass
        return cell, resources_____no_output_____

Cells containing:
- a zero indented call to `notebook2script`

are not run while building docs. This avoids failures caused by importing empty or partially built modules.

Cells containing:
- `show_doc` (which could be indented) or
- a "library import" (zero indent import from current library) e.g. `from LIB_NAME.core import *`

are executed and must run without error. If running these cells raises an exception, the build will stop.

Cells containing zero indented imports. e.g.
- `from module import *` or
- `import module`

are executed but errors will not stop the build.

If you need to `show_doc` something, please make sure it's imported via a cell that does not depend on previous cells being run. The easiest way to do this is to use a cell that contains nothing but imports._____no_output_____

#export
def _import_show_doc_cell(mods=None):
    "Add an import show_doc cell."
    source = f"from nbdev.showdoc import show_doc"
    if mods is not None:
        for mod in mods: source += f"\nfrom {Config().lib_name}.{mod} import *"
    return {'cell_type': 'code',
            'execution_count': None,
            'metadata': {'hide_input': True},
            'outputs': [],
            'source': source}

def execute_nb(nb, mod=None, metadata=None, show_doc_only=True):
    "Execute `nb` (or only the `show_doc` cells) with `metadata`"
    mods = ([] if mod is None else [mod]) + _gather_export_mods(nb['cells'])
    nb['cells'].insert(0, _import_show_doc_cell(mods))
    ep_cls = ExecuteShowDocPreprocessor if show_doc_only else ExecutePreprocessor
    ep = ep_cls(timeout=600, kernel_name='python3')
    metadata = metadata or {}
    pnb = nbformat.from_dict(nb)
    ep.preprocess(pnb, metadata)
    return pnb_____no_output_____

## Converting bibtex citations_____no_output_____

#export
_re_cite = re.compile(r"(\\cite{)([^}]*)(})", re.MULTILINE | re.VERBOSE) # Catches citations used with `\cite{}`_____no_output_____#export
def _textcite2link(text):
    citations = _re_cite.finditer(text)
    out = []
    start_pos = 0
    for cit_group in citations:
        cit_pos_st =  cit_group.span()[0]
        cit_pos_fin =  cit_group.span()[1]
        out.append(text[start_pos:cit_pos_st])
        out.append('[')
        cit_group = cit_group[2].split(',')
        for i, cit in enumerate(cit_group):
            cit=cit.strip()
            out.append(f"""{cit}""")
            if i != len(cit_group) - 1:
                out.append(',')
        out.append(']')
        start_pos = cit_pos_fin
    out.append(text[start_pos:])
    return ''.join(out)_____no_output_____#export
def cite2link(cell):
    '''Creates links from \cite{} to Refenrence section generated by jupyter_latex_envs'''
    if cell['cell_type'] == 'markdown': cell['source'] = _textcite2link(cell['source'])
    return cell_____no_output_____

jupyter_latex_envs is a jupyter extension https://github.com/jfbercher/jupyter_latex_envs.

You can find relevant section [here](https://rawgit.com/jfbercher/jupyter_latex_envs/master/src/latex_envs/static/doc/latex_env_doc.html#Bibliography)
Note, that nbdev now only supports `\cite{}` conversion and not the rest, e.g., `\figure{}` and so on._____no_output_____

%nbdev_hide
cell = {'cell_type': 'markdown', 'source': r"""This is cited multireference \cite{Frob1, Frob3}.
And single \cite{Frob2}."""}
expected=r"""This is cited multireference [Frob1,Frob3].
And single [Frob2]."""
test_eq(cite2link(cell)["source"], expected)_____no_output_____

It's important to execute all `show_doc` cells before exporting the notebook to html because some of them have just been added automatically or others could have outdated links._____no_output_____

#slow
fake_nb = {k:v for k,v in tst_nb.items() if k != 'cells'}
fake_nb['cells'] = [tst_nb['cells'][0].copy()] + added_cells
fake_nb = execute_nb(fake_nb, mod='export')
assert len(fake_nb['cells'][-1]['outputs']) > 0_____no_output_____

## Filling templates_____no_output_____The following functions automatically adds jekyll templates if they are misssing._____no_output_____

#export
def write_tmpl(tmpl, nms, cfg, dest):
    "Write `tmpl` to `dest` (if missing) filling in `nms` in template using dict `cfg`"
    if dest.exists(): return
    vs = {o:cfg.d[o] for o in nms.split()}
    outp = tmpl.format(**vs)
    dest.write_text(outp)_____no_output_____#export
def write_tmpls():
    "Write out _config.yml and _data/topnav.yml using templates"
    cfg = Config()
    path = Path(cfg.get('doc_src_path', cfg.doc_path))
    write_tmpl(config_tmpl, 'user lib_name title copyright description', cfg, path/'_config.yml')
    write_tmpl(topnav_tmpl, 'host git_url', cfg, path/'_data'/'topnav.yml')
    write_tmpl(makefile_tmpl, 'nbs_path lib_name', cfg, cfg.config_file.parent/'Makefile')_____no_output_____

## Conversion_____no_output_____

__file__ = Config().lib_path/'export2html.py'_____no_output_____#export
def nbdev_exporter(cls=HTMLExporter, template_file=None):
    cfg = traitlets.config.Config()
    exporter = cls(cfg)
    exporter.exclude_input_prompt=True
    exporter.exclude_output_prompt=True
    exporter.anchor_link_text = ' '
    exporter.template_file = 'jekyll.tpl' if template_file is None else template_file
    exporter.template_path.append(str(Path(__file__).parent/'templates'))
    return exporter_____no_output_____#export
process_cells = [remove_fake_headers, remove_hidden, remove_empty]
process_cell  = [hide_cells, collapse_cells, remove_widget_state, add_jekyll_notes, escape_latex, cite2link]_____no_output_____#export
def _nb2htmlfname(nb_path, dest=None):
    if dest is None: dest = Config().doc_path
    return Path(dest)/re_digits_first.sub('', nb_path.with_suffix('.html').name)_____no_output_____%nbdev_hide
test_eq(_nb2htmlfname(Path('00a_export.ipynb')), Config().doc_path/'export.html')
test_eq(_nb2htmlfname(Path('export.ipynb')), Config().doc_path/'export.html')
test_eq(_nb2htmlfname(Path('00ab_export_module_1.ipynb')), Config().doc_path/'export_module_1.html')
test_eq(_nb2htmlfname(Path('export.ipynb'), '.'), Path('export.html'))_____no_output_____#export
def convert_nb(fname, cls=HTMLExporter, template_file=None, exporter=None, dest=None, execute=True):
    "Convert a notebook `fname` to html file in `dest_path`."
    fname = Path(fname).absolute()
#     os.chdir(fname.parent)
    nb = read_nb(fname)
    call_cb('begin_doc_nb', nb, fname, 'html')
    meta_jekyll = get_metadata(nb['cells'])
    meta_jekyll['nb_path'] = str(fname.relative_to(Config().lib_path.parent))
    cls_lvl = find_default_level(nb['cells'])
    mod = find_default_export(nb['cells'])
    nb['cells'] = compose(*process_cells,partial(add_show_docs, cls_lvl=cls_lvl))(nb['cells'])
    _func = compose(partial(copy_images, fname=fname, dest=Config().doc_path), *process_cell, treat_backticks)
    nb['cells'] = [_func(c) for c in nb['cells']]
    if execute: nb = execute_nb(nb, mod=mod)
    nb['cells'] = [clean_exports(c) for c in nb['cells']]
    call_cb('after_doc_nb_preprocess', nb, fname, 'html')
    if exporter is None: exporter = nbdev_exporter(cls=cls, template_file=template_file)
    with open(_nb2htmlfname(fname, dest=dest),'w') as f:
        f.write(exporter.from_notebook_node(nb, resources=meta_jekyll)[0])
    call_cb('after_doc_nb', fname, 'html')_____no_output_____#export
def _notebook2html(fname, cls=HTMLExporter, template_file=None, exporter=None, dest=None, execute=True):
    time.sleep(random.random())
    print(f"converting: {fname}")
    try:
        convert_nb(fname, cls=cls, template_file=template_file, exporter=exporter, dest=dest, execute=execute)
        return True
    except Exception as e:
        print(e)
        return False_____no_output_____#export
def notebook2html(fname=None, force_all=False, n_workers=None, cls=HTMLExporter, template_file=None,
                  exporter=None, dest=None, pause=0, execute=True):
    "Convert all notebooks matching `fname` to html files"
    if fname is None:
        files = [f for f in Config().nbs_path.glob('**/*.ipynb')
                 if not f.name.startswith('_') and not '/.' in f.as_posix()]
    else:
        p = Path(fname)
        files = list(p.parent.glob(p.name))
    if len(files)==1:
        force_all = True
        if n_workers is None: n_workers=0
    if not force_all:
        # only rebuild modified files
        files,_files = [],files.copy()
        for fname in _files:
            fname_out = _nb2htmlfname(Path(fname).absolute(), dest=dest)
            if not fname_out.exists() or os.path.getmtime(fname) >= os.path.getmtime(fname_out):
                files.append(fname)
    if len(files)==0: print("No notebooks were modified")
    else:
        passed = parallel(_notebook2html, files, n_workers=n_workers, cls=cls,
                          template_file=template_file, exporter=exporter, dest=dest, pause=pause, execute=execute)
        if not all(passed):
            msg = "Conversion failed on the following:\n"
            print(msg + '\n'.join([f.name for p,f in zip(passed,files) if not p]))_____no_output_____#hide
class TestCallbacks:
    def begin(self, nb, file_name, output_type):
        self.begin_data=dict(nb=nb,file_name=file_name,output_type=output_type)
        return nb
    def middle(self, nb, file_name, output_type):
        self.middle_data=dict(nb=nb,file_name=file_name,output_type=output_type)
        return nb
    def end(self, file_name, output_type):
        self.end_data=dict(file_name=file_name,output_type=output_type)
test_callbacks=TestCallbacks()
call_cb('this makes sure nbdev_callbacks is loaded from the right place')
import nbdev_callbacks as cbs
original_callbacks=cbs.begin_doc_nb,cbs.after_doc_nb_preprocess,cbs.after_doc_nb
try:
    cbs.begin_doc_nb=test_callbacks.begin
    cbs.after_doc_nb_preprocess=test_callbacks.middle
    cbs.after_doc_nb=test_callbacks.end
    assert not hasattr(test_callbacks,'begin_data')
    # Test when an argument is given to notebook2html
    p1 = Path('/tmp/sync.html')
    if p1.exists(): p1.unlink()
    Path('/tmp').mkdir(exist_ok=True)
    notebook2html('01_sync.ipynb', dest='/tmp');
    assert p1.exists()
    # Check that callback handlers have been called
    for d in [test_callbacks.begin_data, test_callbacks.middle_data, test_callbacks.end_data]:
        test_eq(d['file_name'], Path('01_sync.ipynb').absolute())
        test_eq(d['output_type'], 'html')
        if d is not test_callbacks.end_data: assert d['nb']
finally:
    cbs.begin_doc_nb,cbs.after_doc_nb_preprocess,cbs.after_doc_nb=original_callbacks_____no_output_____#slow
#hide
# Test when no argument is given to notebook2html
dest_files = [_nb2htmlfname(f, dest='/tmp') for f in Config().nbs_path.glob('*.ipynb') if not f.name.startswith('_')]
[f.unlink() for f in dest_files if f.exists()]
notebook2html(fname=None, dest='/tmp');
assert all([f.exists() for f in dest_files])_____no_output_____# %nbdev_hide
# # Test Error handling
# try: notebook2html('../README.md');
# except Exception as e: pass
# else: assert False, 'An error should be raised when a non-notebook file is passed to notebook2html!'_____no_output_____

Hide cells starting with `%nbdev_export` and only leaves the prose and the tests. If `fname` is not specified, this will convert all notebooks not beginning with an underscore in the `nb_folder` defined in `setting.ini`. Otherwise `fname` can be a single filename or a glob expression.

By default, only the notebooks that are more recent than their html counterparts are modified, pass `force_all=True` to change that behavior._____no_output_____

%nbdev_hide
#notebook2html(force_all=True)_____no_output_____#export
def convert_md(fname, dest_path, img_path='docs/images/', jekyll=True):
    "Convert a notebook `fname` to a markdown file in `dest_path`."
    fname = Path(fname).absolute()
    if not img_path: img_path = fname.stem + '_files/'
    Path(img_path).mkdir(exist_ok=True, parents=True)
    nb = read_nb(fname)
    call_cb('begin_doc_nb', nb, fname, 'md')
    meta_jekyll = get_metadata(nb['cells'])
    try: meta_jekyll['nb_path'] = str(fname.relative_to(Config().lib_path.parent))
    except: meta_jekyll['nb_path'] = str(fname)
    nb['cells'] = compose(*process_cells)(nb['cells'])
    nb['cells'] = [compose(partial(adapt_img_path, fname=fname, dest=dest_path, jekyll=jekyll), *process_cell)(c)
                   for c in nb['cells']]
    call_cb('after_doc_nb_preprocess', nb, fname, 'md')
    fname = Path(fname).absolute()
    dest_name = fname.with_suffix('.md').name
    exp = nbdev_exporter(cls=MarkdownExporter, template_file='jekyll-md.tpl' if jekyll else 'md.tpl')
    export = exp.from_notebook_node(nb, resources=meta_jekyll)
    md = export[0]
    for ext in ['png', 'svg']:
        md = re.sub(r'!\['+ext+'\]\((.+)\)', '!['+ext+'](' + img_path + '\\1)', md)
    with (Path(dest_path)/dest_name).open('w') as f: f.write(md)
    if hasattr(export[1]['outputs'], 'items'):
        for n,o in export[1]['outputs'].items():
            with open(Path(dest_path)/img_path/n, 'wb') as f: f.write(o)
    call_cb('after_doc_nb', fname, 'md')_____no_output_____

This is used to convert the index into the `README.md`._____no_output_____

%nbdev_hide
def _test_md(fn):
    fn,dest = Path(fn),Path().absolute().parent
    try: convert_md(fn, dest, jekyll=False)
    finally: (dest/f'{fn.stem}.md').unlink()_____no_output_____%nbdev_hide
_test_md('index.ipynb')
# `export[1]['outputs']` will be a `str` if the notebook has no markdown cells to convert.
# e.g. the nb could have a single jekyll markdown cell or just code cells ...
_test_md(f'../tests/single-cell-index.ipynb')_____no_output_____#export
_re_att_ref = re.compile(r' *!\[(.*)\]\(attachment:image.png(?: "(.*)")?\)')_____no_output_____t = '![screenshot](attachment:image.png)'
test_eq(_re_att_ref.match(t).groups(), ('screenshot', None))

t = '![screenshot](attachment:image.png "Deploying to Binder")'
test_eq(_re_att_ref.match(t).groups(), ('screenshot', "Deploying to Binder"))_____no_output_____#export
try: from PIL import Image
except: pass # Only required for _update_att_ref_____no_output_____#export
_tmpl_img = ''

def _update_att_ref(line, path, img):
    m = _re_att_ref.match(line)
    if not m: return line
    alt,title = m.groups()
    w = img.size[0]
    if alt=='screenshot': w //= 2
    if not title: title = "TK: add title"
    return _tmpl_img.format(title=title, width=str(w), id='TK: add it', name=str(path))_____no_output_____#export
def _nb_detach_cell(cell, dest, use_img):
    att,src = cell['attachments'],cell['source']
    mime,img = first(first(att.values()).items())
    ext = mime.split('/')[1]
    for i in range(99999):
        p = dest/(f'att_{i:05d}.{ext}')
        if not p.exists(): break
    img = b64decode(img)
    p.write_bytes(img)
    del(cell['attachments'])
    if use_img:  return [_update_att_ref(o,p,Image.open(p)) for o in src]
    else: return [o.replace('attachment:image.png', str(p)) for o in src]_____no_output_____#export
def nb_detach_cells(path_nb, dest=None, replace=True, use_img=False):
    "Export cell attachments to `dest` and update references"
    path_nb = Path(path_nb)
    if not dest: dest = f'{path_nb.stem}_files'
    dest = Path(dest)
    dest.mkdir(exist_ok=True, parents=True)
    j = json.load(path_nb.open())
    atts = [o for o in j['cells'] if 'attachments' in o]
    for o in atts: o['source'] = _nb_detach_cell(o, dest, use_img)
    if atts and replace: json.dump(j, path_nb.open('w'))
    if not replace: return j_____no_output_____

## Sidebar_____no_output_____

#export
import time,random,warnings_____no_output_____#export
def _leaf(k,v):
    url = 'external_url' if "http" in v else 'url'
    #if url=='url': v=v+'.html'
    return {'title':k, url:v, 'output':'web,pdf'}_____no_output_____#export
_k_names = ['folders', 'folderitems', 'subfolders', 'subfolderitems']
def _side_dict(title, data, level=0):
    k_name = _k_names[level]
    level += 1
    res = [(_side_dict(k, v, level) if isinstance(v,dict) else _leaf(k,v))
        for k,v in data.items()]
    return ({k_name:res} if not title
            else res if title.startswith('empty')
            else {'title': title, 'output':'web', k_name: res})_____no_output_____#export
_re_catch_title = re.compile('^title\s*:\s*(\S+.*)$', re.MULTILINE)_____no_output_____#export
def _get_title(fname):
    "Grabs the title of html file `fname`"
    with open(fname, 'r') as f: code = f.read()
    src =  _re_catch_title.search(code)
    return fname.stem if src is None else src.groups()[0]_____no_output_____%nbdev_hide
test_eq(_get_title(Config().doc_path/'export.html'), "Export to modules")_____no_output_____#export
def _create_default_sidebar():
    "Create the default sidebar for the docs website"
    dic = {"Overview": "/"}
    files = [f for f in Config().nbs_path.glob('*.ipynb') if not f.name.startswith('_')]
    fnames = [_nb2htmlfname(f) for f in sorted(files)]
    titles = [_get_title(f) for f in fnames if 'index' not in f.stem!='index']
    if len(titles) > len(set(titles)): print(f"Warning: Some of your Notebooks use the same title ({titles}).")
    dic.update({_get_title(f):f'{f.name}' for f in fnames if f.stem!='index'})
    return dic_____no_output_____#export
def create_default_sidebar():
    "Create the default sidebar for the docs website"
    dic = {Config().lib_name: _create_default_sidebar()}
    json.dump(dic, open(Config().doc_path/'sidebar.json', 'w'), indent=2)_____no_output_____

The default sidebar lists all html pages with their respective title, except the index that is named "Overview". To build a custom sidebar, set the flag `custom_sidebar` in your `settings.ini` to `True` then change the `sidebar.json` file in the `doc_folder` to your liking. Otherwise, the sidebar is updated at each doc build._____no_output_____

#export
def make_sidebar():
    "Making sidebar for the doc website form the content of `doc_folder/sidebar.json`"
    cfg = Config()
    if not (cfg.doc_path/'sidebar.json').exists() or cfg.get('custom_sidebar', 'False') == 'False':
        create_default_sidebar()
    sidebar_d = json.load(open(cfg.doc_path/'sidebar.json', 'r'))
    res = _side_dict('Sidebar', sidebar_d)
    res = {'entries': [res]}
    res_s = yaml.dump(res, default_flow_style=False)
    res_s = res_s.replace('- subfolders:', '  subfolders:').replace(' - - ', '   - ')
    res_s = f"""
#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# Instead edit {'../../sidebar.json'}
"""+res_s
    open(cfg.doc_path/'_data/sidebars/home_sidebar.yml', 'w').write(res_s)_____no_output_____

## Export-_____no_output_____

#hide
from nbdev.export import *
notebook2script()Converted 00_export.ipynb.
Converted 01_sync.ipynb.
Converted 02_showdoc.ipynb.
Converted 03_export2html.ipynb.
Converted 04_test.ipynb.
Converted 05_merge.ipynb.
Converted 05a_conda.ipynb.
Converted 06_cli.ipynb.
Converted 07_clean.ipynb.
Converted 99_search.ipynb.
Converted index.ipynb.
Converted magic_flags.ipynb.
Converted nbdev_callbacks.ipynb.
Converted tutorial.ipynb.


# Notebook from dongxulee/lifeCycle
Path: 20210706/.ipynb_checkpoints/variationPrice-checkpoint.ipynb

<code>
import numpy as np
import jax.numpy as jnp
from jax.scipy.ndimage import map_coordinates
import warnings
from jax import jit, partial, random, vmap
from tqdm import tqdm
warnings.filterwarnings("ignore")
np.printoptions(precision=2)_____no_output_____# time line
T_min = 0
T_max = 60
T_R = 45
# discounting factor
beta = 1/(1+0.02)
# utility function parameter
gamma = 2
# relative importance of housing consumption and non durable consumption
alpha = 0.7
# parameter used to calculate the housing consumption
kappa = 0.3
# uB associated parameter
B = 2
# social welfare after the unemployment
welfare = 20
# tax rate before and after retirement
tau_L = 0.2
tau_R = 0.1
# number of states S
nS = 8
# number of states e
nE = 2
# housing state
nO = 2
# experience state
nZ = 2_____no_output_____'''
    Economic state calibration
'''

# probability of survival
Pa = jnp.array(np.load("constant/prob.npy"))
# deterministic income
detEarning = jnp.array(np.load("constant/detEarningHigh.npy"))
# rescale the deterministic income
detEarning = detEarning
####################################################################################### high skill feature
detEarning = jnp.concatenate([detEarning[:46], detEarning[46:]-35])
# Define transition matrix of economical states S
Ps = np.genfromtxt('constant/Ps.csv',delimiter=',')
fix = (np.sum(Ps, axis = 1) - 1)
for i in range(nS):
    for j in range(nS):
        if Ps[i,j] - fix[i] > 0:
            Ps[i,j] = Ps[i,j] - fix[i]
            break
Ps = jnp.array(Ps)
# The possible GDP growth, stock return, bond return
gkfe = np.genfromtxt('constant/gkfe.csv',delimiter=',')
gkfe = jnp.array(gkfe)
# GDP growth depending on current S state
gGDP = gkfe[:,0]/100
# risk free interest rate depending on current S state
r_b = gkfe[:,1]/100
# stock return depending on current S state
r_k = gkfe[:,2]/100
# unemployment rate depending on current S state
Pe = gkfe[:,7:]/100
Pe = Pe[:,::-1]
# housing and renting price
Pt = gkfe[:,3]/100
Pr = gkfe[:,4]
'''
    401k related constants
'''
# some variables associated with 401k amount
r_bar = 0.02
Pa = Pa[:T_max]
Nt = [np.sum(Pa[t:]) for t in range(T_min,T_max)]
#Factor used to calculate the withdraw amount
Dn = [(r_bar*(1+r_bar)**N)/((1+r_bar)**N - 1) for N in Nt]
Dn[-1] = 1
Dn = jnp.array(Dn)
# income fraction goes into 401k
yi = 0.04WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
'''
    housing related constants
'''
# variable associated with housing and mortgage
# mortgage rate
rh = 0.045
# housing unit
H = 1000
# rent unit
Rl = 500
# housing price constant
pt = 2*250/1000
# 30k rent 1000 sf
pr = 2*10/1000 * 2
# constant cost
c_h = 5
c_s = H*pt*0.4
# Dm is used to update the mortgage payment
Dm = [(1+rh) - rh*(1+rh)**(T_max - t)/((1+rh)**(T_max-t)-1) for t in range(T_min, T_max)]
Dm[-1] = 0
Dm = jnp.array(Dm)

# 30 year mortgage
Ms = []
M = H*pt*0.8
m = M*(1+rh) - Dm[30]*M
for i in range(30, T_max):
    M = M*(1+rh) - m
    Ms.append(M)
Ms[-1] = 0
Ms = jnp.array(Ms)
############################################################################################################ high skill feature
# stock transaction fee
Kc = 0

# stock participation cost
c_k = 5_____no_output_____'''
    Discretize the state space
    Discretize the action space
'''
# actions dicretization(hp, cp, kp)
numGrid = 20
As = np.array(np.meshgrid(np.linspace(0.001,0.999,numGrid), np.linspace(0,1,numGrid), [0,1],[0,1])).T.reshape(-1,4)
As = jnp.array(As)
# wealth discretization
ws = np.linspace(0, 400, 20)
ns = np.linspace(0, 300, 10)
ms = np.linspace(0, 0.8*H*pt, 10)
# scales associated with discretization
scaleW = ws.max()/ws.size
scaleN = ns.max()/ns.size
scaleM = ms.max()/ms.size

# dimentions of the state
dim = (ws.size, ns.size, ms.size, nS, nE, nO, nZ)
dimSize = len(dim)

xgrid = np.array([[w,n,m,s,e,o,k] for w in ws
                            for n in ns
                            for m in ms
                            for s in range(nS)
                            for e in range(nE)
                            for o in range(nO)
                            for k in range(nZ)]).reshape(dim + (dimSize,))

Xs = xgrid.reshape((np.prod(dim),dimSize))
Xs = jnp.array(Xs)

Vgrid = np.zeros(dim + (T_max,))
cgrid = np.zeros(dim + (T_max,))
bgrid = np.zeros(dim + (T_max,))
kgrid = np.zeros(dim + (T_max,))
hgrid = np.zeros(dim + (T_max,))
agrid = np.zeros(dim + (T_max,))

# start of function definitions
nX = Xs.shape[0]
nA = As.shape[0]_____no_output_____#Define the earning function, which applies for both employment status and 8 econ states
@partial(jit, static_argnums=(0,))
def y(t, x):
    '''
        x = [w,n,m,s,e,o,z]
        x = [0,1,2,3,4,5,6]
    '''
    if t <= T_R:
        return detEarning[t] * (1+gGDP[jnp.array(x[3], dtype = jnp.int8)]) * x[4] + (1-x[4]) * welfare
    else:
        return detEarning[-1]

#Earning after tax and fixed by transaction in and out from 401k account
@partial(jit, static_argnums=(0,))
def yAT(t,x):
    yt = y(t, x)
    if t <= T_R:
        # yi portion of the income will be put into the 401k if employed
        return (1-tau_L)*(yt * (1-yi))*x[4] + (1-x[4])*yt
    else:
        # t > T_R, n/discounting amount will be withdraw from the 401k
        return (1-tau_R)*yt + x[1]*Dn[t]

#Define the evolution of the amount in 401k account
@partial(jit, static_argnums=(0,))
def gn(t, x, r = r_bar):
    if t <= T_R:
        # if the person is employed, then yi portion of his income goes into 401k
        n_cur = x[1] + y(t, x) * yi * x[4]
    else:
        # t > T_R, n*Dn amount will be withdraw from the 401k
        n_cur = x[1] - x[1]*Dn[t]
        # the 401 grow with the rate r
    return (1+r)*n_cur

#Define the utility function
@jit
def u(c):
    return (jnp.power(c, 1-gamma) - 1)/(1 - gamma)

#Define the bequeath function, which is a function of bequeath wealth
@jit
def uB(tb):
    return B*u(tb)

#Reward function depends on the housing and non-housing consumption
@jit
def R(x,a):
    '''
    Input:
        x = [w,n,m,s,e,o,z]
        x = [0,1,2,3,4,5,6]
        a = [c,b,k,h,action]
        a = [0,1,2,3,4]
    '''
    c = a[:,0]
    b = a[:,1]
    k = a[:,2]
    h = a[:,3]
    C = jnp.power(c, alpha) * jnp.power(h, 1-alpha)
    return u(C) + (-1/((c > 0) * (b >= 0) * (k >= 0) * (h > 0)) + 1)

_____no_output_____As.shape_____no_output_____# pc*qc / (ph*qh) = alpha/(1-alpha)
@partial(jit, static_argnums=(0,))
def feasibleActions(t, x):
    # owner
    sell = As[:,2]
    # pay for participation cost
    payParticipation = As[:,3]
    # last term is the tax deduction of the interest portion of mortgage payment
    payment = (x[2] > 0)*(((t<=T_R)*tau_L + (t>T_R)*tau_R)*x[2]*rh - m)

#     # if the agent is able to pay
#     if yAT(t,x) + x[0] + payment > 0:
#         sell = jnp.zeros(nA)
#         budget1 = yAT(t,x) + x[0] + (1-sell)*payment
#     # if the agent is not able to pay (force sell)
#     else:
#         sell = jnp.ones(nA)
#         budget1 = yAT(t,x) + x[0] + sell*(H*pt - x[2] - c_s)

    sell = (yAT(t,x) + x[0] + payment > 0)*jnp.zeros(nA) + (yAT(t,x) + x[0] + payment <= 0)*jnp.ones(nA)
    budget1 = yAT(t,x) + x[0] + (1-sell)*payment + sell*(H*pt - x[2] - c_s) - (1-x[6])*payParticipation
    # last term is the tax deduction of the interest portion of mortgage payment
    h = jnp.ones(nA)*H*(1+kappa)*(1-sell) + sell*jnp.clip(budget1*As[:,0]*(1-alpha)/pr, a_max = Rl)
    c = budget1*As[:,0]*(1-sell) + sell*(budget1*As[:,0] - h*pr)
    budget2 = budget1*(1-As[:,0])

    k = budget2*As[:,1]*x[6]
    b = budget2 - k
    k = k*(1-Kc)

    owner_action = jnp.column_stack((c,b,k,h,sell,payParticipation))

    # renter
    buy = As[:,2]*(t < 30)
    budget1 = yAT(t,x) + x[0] - buy*(H*pt*0.2 + c_h) - (1-x[6])*payParticipation
    h = jnp.clip(budget1*As[:,0]*(1-alpha)/pr, a_max = Rl)*(1-buy) + buy*jnp.ones(nA)*H*(1+kappa)
    c = (budget1*As[:,0] - h*pr)*(1-buy) + buy*budget1*As[:,0]
    budget2 = budget1*(1-As[:,0])

    k = budget2*As[:,1]*x[6]
    b = budget2 - k
    k = k*(1-Kc)

    renter_action = jnp.column_stack((c,b,k,h,buy,payParticipation))

    actions = x[5]*owner_action + (1-x[5])*renter_action
    return actions

@partial(jit, static_argnums=(0,))
def transition(t,a,x):
    '''
        Input:
            x = [w,n,m,s,e,o,z]
            x = [0,1,2,3,4,5]
            a = [c,b,k,h,action]
            a = [0,1,2,3,4]
        Output:
            w_next
            n_next
            m_next
            s_next
            e_next
            o_next
            z_next

            prob_next
    '''
    nA = a.shape[0]
    s = jnp.array(x[3], dtype = jnp.int8)
    e = jnp.array(x[4], dtype = jnp.int8)
    # actions taken
    b = a[:,1]
    k = a[:,2]
    action = a[:,4]
    # wether to pay for participation
    payForParticipation = a[:,5]
    w_next = ((1+r_b[s])*b + jnp.outer(k,(1+r_k)).T).T.flatten().repeat(nE)
    n_next = gn(t, x)*jnp.ones(w_next.size)
    s_next = jnp.tile(jnp.arange(nS),nA).repeat(nE)
    e_next = jnp.column_stack((e.repeat(nA*nS),(1-e).repeat(nA*nS))).flatten()
    z_next = x[6]*jnp.ones(w_next.size) + ((1-x[6]) * payForParticipation).repeat(nS*nE)
    # job status changing probability and econ state transition probability
    pe = Pe[s, e]
    ps = jnp.tile(Ps[s], nA)
    prob_next = jnp.column_stack(((1-pe)*ps,pe*ps)).flatten()

    # owner
    m_next_own = ((1-action)*jnp.clip(x[2]*(1+rh) - m, a_min = 0)).repeat(nS*nE)
    o_next_own = (x[5] - action).repeat(nS*nE)
    # renter
    m_next_rent = (action*H*pt*0.8).repeat(nS*nE)
    o_next_rent = action.repeat(nS*nE)

    m_next = x[5] * m_next_own + (1-x[5]) * m_next_rent
    o_next = x[5] * o_next_own + (1-x[5]) * o_next_rent
    return jnp.column_stack((w_next,n_next,m_next,s_next,e_next,o_next,z_next,prob_next))

# used to calculate dot product
@jit
def dotProduct(p_next, uBTB):
    return (p_next*uBTB).reshape((p_next.shape[0]//(nS*nE), (nS*nE))).sum(axis = 1)

# define approximation of fit
@jit
def fit(v, xp):
    return map_coordinates(v,jnp.vstack((xp[:,0]/scaleW,
                                                      xp[:,1]/scaleN,
                                                      xp[:,2]/scaleM,
                                                      xp[:,3],
                                                      xp[:,4],
                                                      xp[:,5],
                                                      xp[:,6])),
                                                     order = 1, mode = 'nearest')

@partial(jit, static_argnums=(0,))
def V(t,V_next,x):
    '''
    x = [w,n,m,s,e,o,z]
    x = [0,1,2,3,4,5]
    xp:
        w_next    0
        n_next    1
        m_next    2
        s_next    3
        e_next    4
        o_next    5
        z_next    6
        prob_next 7
    '''
    actions = feasibleActions(t,x)
    xp = transition(t,actions,x)
    # bequeath utility
    TB = xp[:,0]+x[1]*(1+r_bar)+xp[:,5]*(H*pt-x[2]*(1+rh)-25)
    bequeathU = uB(TB)
    if t == T_max-1:
        Q = R(x,actions) + beta * dotProduct(xp[:,7], bequeathU)
    else:
        Q = R(x,actions) + beta * dotProduct(xp[:,7], Pa[t]*fit(V_next, xp) + (1-Pa[t])*bequeathU)
    Q = jnp.nan_to_num(Q, nan = -jnp.inf)
    v = Q.max()
    cbkha = actions[Q.argmax()]
    return v, cbkha

# calculate the stationary distribution of econ state and employment state
S_distribution = jnp.ones(nS)/nS
for _ in range(100):
    S_distribution = jnp.matmul(S_distribution, Ps)

#P(0,1)
P01 = jnp.dot(Pe[:,0],S_distribution)
#P(1,0)
P10 = jnp.dot(Pe[:,1],S_distribution)
jnp.array([[1-P01, P01],[P10, 1-P10]])

E_distribution = jnp.ones(2)/2
for _ in range(100):
    E_distribution = jnp.matmul(E_distribution, jnp.array([[1-P01, P01],[P10, 1-P10]]))

####################################################################################### solving the model
for t in tqdm(range(T_max-1,T_min-1, -1)):
    if t == T_max-1:
        v,cbkha = vmap(partial(V,t,Vgrid[:,:,:,:,:,:,:,t]))(Xs)
    else:
        v,cbkha = vmap(partial(V,t,Vgrid[:,:,:,:,:,:,:,t+1]))(Xs)
    Vgrid[:,:,:,:,:,:,:,t] = v.reshape(dim)
    cgrid[:,:,:,:,:,:,:,t] = cbkha[:,0].reshape(dim)
    bgrid[:,:,:,:,:,:,:,t] = cbkha[:,1].reshape(dim)
    kgrid[:,:,:,:,:,:,:,t] = cbkha[:,2].reshape(dim)
    hgrid[:,:,:,:,:,:,:,t] = cbkha[:,3].reshape(dim)
    agrid[:,:,:,:,:,:,:,t] = cbkha[:,4].reshape(dim)  2%|▏         | 1/60 [00:05<05:10,  5.27s/it]np.save("richLow",Vgrid)_____no_output_____
</code>