# Notebook from AdeyemiA/pytorchfundamentals
Path: nlp-pytorch/5-generative-networks.ipynb

# Generative Networks

Recurrent Neural Networks (RNNs) and their gated cell variants such as Long Short Term Memory Cells (LSTMs) and Gated Recurrent Units (GRUs) provided a mechanism for language modeling, i.e. they can learn word ordering and provide predictions for next word in a sequence. This allows us to use RNNs for **generative tasks**, such as ordinary text generation, machine translation, and even image captioning.

In RNN architecture we discussed in the previous unit, each RNN unit produced next next hidden state as an output. However, we can also add another output to each recurrent unit, which would allow us to output a **sequence** (which is equal in length to the original sequence). Moreover, we can use RNN units that do not accept an input at each step, and just take some initial state vector, and then produce a sequence of outputs.

This allows for different neural architectures that are shown in the picture below:

![RNN paterns](../images/unreasonable-effectiveness-of-rnn.jpg)
*Image from blog post [Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) by [Andrej Karpaty](http://karpathy.github.io/)*

* **One-to-one** is a traditional neural network with one input and one output
* **One-to-many** is a generative architecture that accepts one input value, and generates a sequence of output values. For example, if we want to train **image captioning** network that would produce a textual description of a picture, we can a picture as input, pass it through CNN to obtain hidden state, and then have recurrent chain generate caption word-by-word
* **Many-to-one** corresponds to RNN architectures we described in the previous unit, such as text classification
* **Many-to-many**, or **sequence-to-sequence** corresponds to tasks such as **machine translation**, where we have first RNN collect all information from the input sequence into the hidden state, and another RNN chain unrolls this state into the output sequence.

In this unit, we will focus on simple generative models that help us generate text. For simplicity, let's build **character-level network**, which generates text letter by letter. During training, we need to take some text corpus, and split it into letter sequences. _____no_output_____

import torch
import torchtext
import numpy as np
from torchnlp import *
load_dataset() # we need this to make sure data is fetched120000lines [00:04, 26121.40lines/s]
120000lines [00:08, 14104.98lines/s]
7600lines [00:00, 14122.40lines/s]

## TorchText Datasets

In previous unit, we were using built-in AG_NEWS dataset. Now that we need more flexibility in loading text, we will use another mechanism for loading custom datasets.

We will still be using AG News, but let's load it from original data files on disk. From previous units, we shoild have source files for AG News dataset located in `data/ag_news_csv` directory. Training dataset is contained in `train.csv` file, which contains records like the following:
```
"3","Wall St. Bears Claw Back Into the Black (Reuters)","Reuters - Short-sellers, Wall Street's..."
"3","Carlyle Looks Toward Commercial Aerospace (Reuters)","Private investment firm Carlyle Group..."
"2","Nets get Carter from Raptors","INDIANAPOLIS -- All-Star Vince Carter was traded..."
```
The file is in so-called *comma separated values* (CSV) format, and each line consists of three fields separeate by comma:
* Class number (0-3)
* Title
* News text

To work with this file, we will define `Field` objects that specify how fields should be handled. In our case, there will be two objects:
* `TEXT` will define fields for news title and text. Because we need to split it into individual characters, we will specify custom tokenizer function `char_tokenier`
* `LABEL` will be used for the first field with class number, and we will indicate that it does not need to be tokenized (`use_vocab=False`)

After defining fields, we create `TabularDataset` object which points to the training CSV file:_____no_output_____

def char_tokenizer(words):
    return list(words) #[word for word in words]

TEXT = torchtext.data.Field(sequential=True, tokenize=char_tokenizer) #, lower=True)
LABEL = torchtext.data.Field(sequential=False, use_vocab=False)

train_dataset = torchtext.data.TabularDataset('./data/ag_news_csv/train.csv',
        format='csv',
        fields=[('Label', LABEL), ('Head', TEXT), ('Text', TEXT) ])_____no_output_____

Before we can start loading the dataset, we need to build the vocabulary by calling `build_vocab`. After that, we can use vocabulary dictionaries to map between characters and their numerical indices:_____no_output_____

TEXT.build_vocab(train_dataset)
vocab_size = len(TEXT.vocab)
print(f"Vocabulary size = {vocab_size}")
print(f"Encoding of 'a' is {TEXT.vocab.stoi['a']}")
print(f"Character with code 13 is {TEXT.vocab.itos[13]}")Vocabulary size = 84
Encoding of 'a' is 4
Character with code 13 is h

Now can access our dataset by iterating through `train_dataset.examples`. Each example contains named fields corresponding to the fields we have defined:_____no_output_____

print(train_dataset.examples[0].Text)['R', 'e', 'u', 't', 'e', 'r', 's', ' ', '-', ' ', 'S', 'h', 'o', 'r', 't', '-', 's', 'e', 'l', 'l', 'e', 'r', 's', ',', ' ', 'W', 'a', 'l', 'l', ' ', 'S', 't', 'r', 'e', 'e', 't', "'", 's', ' ', 'd', 'w', 'i', 'n', 'd', 'l', 'i', 'n', 'g', '\\', 'b', 'a', 'n', 'd', ' ', 'o', 'f', ' ', 'u', 'l', 't', 'r', 'a', '-', 'c', 'y', 'n', 'i', 'c', 's', ',', ' ', 'a', 'r', 'e', ' ', 's', 'e', 'e', 'i', 'n', 'g', ' ', 'g', 'r', 'e', 'e', 'n', ' ', 'a', 'g', 'a', 'i', 'n', '.']

To encode the text, we can use the following function:_____no_output_____

def encode_text(s):
    return torch.LongTensor([TEXT.vocab.stoi[t] for t in s])

encode_text(train_dataset.examples[0].Text)_____no_output_____

## Training Generative RNN

The way we will train RNN to generate text is the following. On each step, we will take a sequence of characters of length `nchars`, and ask the network to generate next output character for each input character:

![RNN Generation](../images/rnn-generate.png)

Depending on the actual scenario, we may also want to include some special characters, such as *end-of-sequence* `<eos>`. In our case, we just want to train the network for endless text generation, thus we will fix the size of each sequence to be equal to `nchars` tokens. Consequently, each training example will consist of `nchars` inputs and `nchars` outputs (which are input sequence shifted one symbol to the left). Minibatch will consist of several such sequences.

The way we will generate minibatches is to take each news text of length `l`, and generate all possible input-output combinations from it (there will be `l-nchars` such combinations). They will form one minibatch, and size of minibatches would be different at each training step. _____no_output_____

nchars = 100

def get_batch(s,nchars=nchars):
    ins = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)
    outs = torch.zeros(len(s)-nchars,nchars,dtype=torch.long,device=device)
    for i in range(len(s)-nchars):
        ins[i] = encode_text(s[i:i+nchars])
        outs[i] = encode_text(s[i+1:i+nchars+1])
    return ins,outs

get_batch(train_dataset.examples[1].Text)_____no_output_____

Now let's define generator network. It can be based on any recurrent cell which we discussed in the previous unit (simple, LSTM or GRU). In our exampe we will use LSTM.

Because the network takes characters as input, and vocabulary size is pretty small, we do not need embedding layer, one-hot-encoded input can directly go to LSTM cell. However, because we pass character numbers as input, we need to one-hot-encode them before passing to LSTM. This is done by calling `one_hot` function during `forward` pass. Output encoder would be a linear layer that will convert hidden state into one-hot-encoded output._____no_output_____

class LSTMGenerator(torch.nn.Module):
    def __init__(self, vocab_size, hidden_dim):
        super().__init__()
        self.rnn = torch.nn.LSTM(vocab_size,hidden_dim,batch_first=True)
        self.fc = torch.nn.Linear(hidden_dim, vocab_size)

    def forward(self, x, s=None):
        x = torch.nn.functional.one_hot(x,vocab_size).to(torch.float32)
        x,s = self.rnn(x,s)
        return self.fc(x),s_____no_output_____

During training, we want to be able to sample generated text. To do that, we will define `generate` function that will produce output string of length `size`, starting from the initial string `start`.

The way it works is the following. First, we will pass the whole start string through the network, and take output state `s` and next predicted character `out`. Since `out` is one-hot encoded, we take `argmax` to get the index of the character `nc` in the vocabulary, and use `itos` to figure out the actual character and append it to the resulting list of characters `chars`. This process of generating one character is repeated `size` times to generate required number of characters.  _____no_output_____

def generate(net,size=100,start='today '):
        chars = list(start)
        out, s = net(encode_text(chars).view(1,-1).to(device))
        for i in range(size):
            nc = torch.argmax(out[0][-1])
            chars.append(TEXT.vocab.itos[nc])
            out, s = net(nc.view(1,-1),s)
        return ''.join(chars)_____no_output_____

Now let's do the training! Training loop is almost the same as in all our previous examples, but instead of accuracy we print sampled generated text every 1000 epochs.

Special attention needs to be paid to the way we compute loss. We need to compute loss given one-hot-encoded output `out`, and expected text `text_out`, which is the list of character indices. Luckily, `cross_entropy` function expects unnormalized network output as first argument, and class number as the second, which is exactly what we have. It also performs automatic averaging over minibatch size.

We also limit the training by `samples_to_train` samples, in order not to wait for too long. We encourage you to experiment and try longer training, possibly for several epochs (in which case you would need to create another loop around this code)._____no_output_____

net = LSTMGenerator(vocab_size,64).to(device)

samples_to_train = 10000
optimizer = torch.optim.Adam(net.parameters(),0.01)
loss_fn = torch.nn.CrossEntropyLoss()
net.train()
for i,x in enumerate(train_dataset.examples):
    if len(x.Text)-nchars<10:
        continue
    samples_to_train-=1
    if not samples_to_train: break
    text_in, text_out = get_batch(x.Text)
    optimizer.zero_grad()
    out,s = net(text_in)
    loss = torch.nn.functional.cross_entropy(out.view(-1,vocab_size),text_out.flatten()) #cross_entropy(out,labels)
    loss.backward()
    optimizer.step()
    if i%1000==0:
        print(f"Current loss = {loss.item()}")
        print(generate(net))Current loss = 2.0068554878234863
today a resite a resite a resite a resite a resite a resite a resite a resite a resite a resite a resite a
Current loss = 1.6322784423828125
today and a reside and a reside and a reside and a reside and a reside and a reside and a reside and a res
Current loss = 2.3880646228790283
today and the company and the company and the company and the company and the company and the company and
Current loss = 1.6894474029541016
today and the company to the stack and the stack and the stack and the stack and the stack and the stack a
Current loss = 1.708423376083374
today and the battere and the battere and the battere and the battere and the battere and the battere and
Current loss = 1.8624305725097656
today and the first the first the first the first the first the first the first the first the first the fi
Current loss = 1.6230477094650269
today and the company and the company and the company and the company and the company and the company and
Current loss = 1.6095234155654907
today of the second the company and a second the company and a second the company and a second the company
Current loss = 1.465132474899292
today of the service of the service of the service of the service of the service of the service of the ser
Current loss = 1.610368013381958
today and the security to the security to the security to the security to the security to the security to

This example already generates some pretty good text, but it can be further improved in several ways:
* **Better minibatch generation**. The way we prepared data for training was to generate one minibatch from one sample. This is not ideal, because minibatches are all of different sizes, and some of them even cannot be generated, because the text is smaller than `nchars`. Also, small minibatches do not load GPU sufficiently enough. It would be wiser to get one large chunk of text from all samples, then generate all input-output pairs, shuffle them, and generate minibatches of equal size.
* **Multilayer LSTM**. It makes sense to try 2 or 3 layers of LSTM cells. As we mentioned in the previous unit, each layer of LSTM extracts certain patterns from text, and in case of character-level generator we can expect lower LSTM level to be responsible for extracting syllables, and higher levels - for words and word combinations. This can be simply implemented by passing number-of-layers parameter to LSTM constructor.
* You may also want to experiment with **GRU units** and see which ones perform better, and with **different hidden layer sizes**. Too large hidden layer may result in overfitting (e.g. network will learn exact text), and smaller size might not produce good result._____no_output_____## Soft Text Generation and Temperature

In the previous definition of `generate`, we were always taking the character with highest probability as the next character in generated text. This resulted in the fact that the text often "cycled" between the same character sequences again and again, like in this example:
```
today of the second the company and a second the company ...
```

However, if we look at the probability distribution for the next character, it could be that the difference between a few highest probabilities is not huge, eg. one character can have probability 0.2, another - 0.19, etc. For example, when looking for the next character in the sequence '*play*', next character can equally well be either space, or **e** (as in the word *player*).

This leads us to the conclusion that it is not always "fair" to select the character with higher probability, because chosing second highest might still lead us to meaningful text. It is more wise to **sample** characters from the probability distribution given by the network output.

This sampling can be done using `multinomial` function that implements so-called **muntinomial** distribution. A function that implements this **soft** text generation is defined below:_____no_output_____

def generate_soft(net,size=100,start='today ',temperature=1.0):
        chars = list(start)
        out, s = net(encode_text(chars).view(1,-1).to(device))
        for i in range(size):
            #nc = torch.argmax(out[0][-1])
            out_dist = out[0][-1].div(temperature).exp()
            nc = torch.multinomial(out_dist,1)[0]
            chars.append(TEXT.vocab.itos[nc])
            out, s = net(nc.view(1,-1),s)
        return ''.join(chars)

for i in [0.3,0.8,1.0,1.3,1.8]:
    print(f"--- Temperature = {i}\n{generate_soft(net,size=300,start='Today ',temperature=i)}\n")--- Temperature = 0.3
Today and the first of the profit at the be that the second the be workers and the recome were the string the second the second-released to the market that store the profit and the first a showed the most the signed in the were market in the American Aneenest The man state for the a the company and the ma

--- Temperature = 0.8
Today in a limater companie in the streal and the the an in were belost was dising weeks yesterday in the were the changions of out on Thursday specy to share inthers of a recond-fell defeater state destable is wield at the develop the rever that the Olympics will business was begonning proceldages sincea

--- Temperature = 1.0
Today of hold calling a gtriging to new down hoped begin athly engined frambiligation from the Gamb in Fivenrating An6 Explan-thay centing publlicated thris makeigre: Basrey in Athenss Concuren #39;s Veris Corp. shepting up hissing marker case were for a will manager wroper nest 24 thig week  prevangroney

--- Temperature = 1.3
Today to of cojemens, stabiled. The 180-K lo-Fildytey dows and Ladamatimmed Getera, twent yally at Iraq, fouthing more.  hig trifwer to it were a weveney Park 6-3 for 16The Sharctas Curra,. But most concluning Wingle Builitory diguseless daranipqionales warning ehard will hugtar to morint, to enterle init

--- Temperature = 1.8
Today !F'm JW2.jeslady-hcopred ar mujon uses\for no,NFidgC" wonm?se-Ijual C ne-Rok Joft\fictoreatcy. Has 16 TG:O/Y CD)"quot;lapiced? PC.A sGorile L2 drvicadaby Basod cycleats people wMidestmustmenty its I2ngf5/xal-ona on Whing for diside leagugnst Wolkay,matury greate, HNFV RB(3O(7L\Th1ffper-"oblove,wh#30

We have introduced one more parameter called **temperature**, which is used to indicate how hard we should stick to the highest probability. If temperature is 1.0, we do fair multinomial sampling, and when temperature goes to infinity - all probabilities become equal, and we randomly select next character. In the example below we can observe that the text becomes meaningless when we increase the temparature too much, and it resembles "cycled" hard-generated text when it becomes closer to 0. _____no_output_____


# Notebook from TerrainBento/examples_tests_and_tutorials
Path: coupled_process_elements/model_basicRt_steady_solution.ipynb

![terrainbento logo](../images/terrainbento_logo.png)

# terrainbento model BasicRt steady-state solution_____no_output_____This model shows example usage of the BasicRt model from the TerrainBento package.

BasicRt modifies Basic by allowing for two lithologies:

$\frac{\partial \eta}{\partial t} = - K(\eta,\eta_C) Q^{1/2}S + D\nabla^2 \eta$

$K(\eta, \eta_C ) = w K_1 + (1 - w) K_2$

$w = \frac{1}{1+\exp \left( -\frac{(\eta -\eta_C )}{W_c}\right)}$

where $Q$ is the local stream discharge, $S$ is the local slope, $W_c$ is the contact-zone width, $K_1$ and $K_2$ are the erodibilities of the upper and lower lithologies, and $D$ is the regolith transport parameter. $w$ is a weight used to calculate the effective erodibility $K(\eta, \eta_C)$ based on the depth to the contact zone and the width of the contact zone.

Refer to [Barnhart et al. (2019)](https://www.geosci-model-dev-discuss.net/gmd-2018-204/) for further explaination. For detailed information about creating a BasicRt model, see [the detailed documentation](https://terrainbento.readthedocs.io/en/latest/source/terrainbento.derived_models.model_basicRt.html).

This notebook (a) shows the initialization and running of this model, (b) saves a NetCDF file of the topography, which we will use to make an oblique Paraview image of the landscape, and (c) creates a slope-area plot at steady state._____no_output_____

<code>
# import required modules
import os
import numpy as np

import matplotlib.pyplot as plt
import matplotlib
matplotlib.rcParams["font.size"] = 20
matplotlib.rcParams["pdf.fonttype"] = 42
%matplotlib inline

from landlab import imshow_grid, RasterModelGrid

from landlab.io.netcdf import write_netcdf
from landlab.values import random

from terrainbento import BasicRt, Clock, NotCoreNodeBaselevelHandler

np.random.seed(4897)_____no_output_____# in this example we will create a grid, a clock, a boundary handler,
# and an output writer. We will then use these to construct the model.

grid = RasterModelGrid((100, 160),
                       xy_spacing=10)

z = random(grid,
           "topographic__elevation",
           where="CORE_NODE")

contact = grid.add_zeros("node", "lithology_contact__elevation")
contact[grid.node_y > 500.0] = 10.
contact[grid.node_y <= 500.0] = -100000000.

clock = Clock(start=0,
              step=10,
              stop=1e7)
ncnblh = NotCoreNodeBaselevelHandler(grid,
                                     modify_core_nodes=True,
                                     lowering_rate=-0.001)_____no_output_____# we can use an output writer to run until the model reaches steady state.
class run_to_steady(object):
    def __init__(self, model):
        self.model = model
        self.last_z = self.model.z.copy()
        self.tolerance = 0.1
    def run_one_step(self):
        if model.model_time > 0:
            diff = (self.model.z[model.grid.core_nodes]
                    - self.last_z[model.grid.core_nodes])
            if max(abs(diff)) <= self.tolerance:
                self.model.clock.stop = model._model_time
                print("Model reached steady state in " + str(model._model_time) + " time units\n")
            else:
                self.last_z = self.model.z.copy()
                if model._model_time <= self.model.clock.stop - self.model.output_interval:
                    self.model.clock.stop += self.model.output_interval_____no_output_____# initialize the model by passing the correct arguments and
# keyword arguments.

model = BasicRt(clock,
                grid,
                boundary_handlers={"NotCoreNodeBaselevelHandler": ncnblh},
                output_interval=1e4,
                save_first_timestep=True,
                output_prefix="output_netcdfs/basicRt.",
                fields=["topographic__elevation"],
                water_erodibility_lower=0.001,
                water_erodibility_upper=0.01,
                m_sp=0.5,
                n_sp=1.0,
                regolith_transport_parameter=0.1,
                contact_zone__width=1.0,
                output_writers={"class": [run_to_steady]})

# to run the model as specified, execute the following line:
model.run()Model reached steady state in 140000.0 time units

# MAKE SLOPE-AREA PLOT

# plot nodes that are not on the boundary or adjacent to it
core_not_boundary = np.array(model.grid.node_has_boundary_neighbor(model.grid.core_nodes)) == False
plotting_nodes = model.grid.core_nodes[core_not_boundary]

upper_plotting_nodes = plotting_nodes[model.grid.node_y[plotting_nodes] > 500.0]

lower_plotting_nodes = plotting_nodes[model.grid.node_y[plotting_nodes] < 500.0]

# assign area_array and slope_array for ROCK
area_array_upper = model.grid.at_node["drainage_area"][upper_plotting_nodes]
slope_array_upper = model.grid.at_node["topographic__steepest_slope"][upper_plotting_nodes]
# assign area_array and slope_array for TILL
area_array_lower = model.grid.at_node["drainage_area"][lower_plotting_nodes]
slope_array_lower = model.grid.at_node["topographic__steepest_slope"][lower_plotting_nodes]

# instantiate figure and plot
fig = plt.figure(figsize=(6, 3.75))
slope_area = plt.subplot()

#plot the data for ROCK
slope_area.scatter(area_array_lower, slope_array_lower, marker="s", edgecolor="0", color="1",
                   label = "Model BasicRt, Lower Layer")

#plot the data for TILL
slope_area.scatter(area_array_upper, slope_array_upper, color="k",
                   label = "Model BasicRt, Upper Layer")

#make axes log and set limits
slope_area.set_xscale("log")
slope_area.set_yscale("log")

slope_area.set_xlim(9*10**1, 3*10**5)
slope_area.set_ylim(1e-5, 1e-1)

#set x and y labels
slope_area.set_xlabel(r"Drainage area [m$^2$]")
slope_area.set_ylabel("Channel slope [-]")
slope_area.legend(scatterpoints=1,prop={"size":12})
slope_area.tick_params(axis="x", which="major", pad=7)

# save out an output figure
output_figure = os.path.join("output_figures/maintext_rock_till_slope_area.pdf")
fig.savefig(output_figure, bbox_inches="tight", dpi=1000) # save figure_____no_output_____# Save stack of all netcdfs for Paraview to use.
model.save_to_xarray_dataset(filename="output_netcdfs/basicRt.nc",
                             time_unit="years",
                             reference_time="model start",
                             space_unit="meters")

# remove temporary netcdfs
model.remove_output_netcdfs()_____no_output_____# make a plot of the final steady state topography
imshow_grid(model.grid, "topographic__elevation")_____no_output_____
</code>

## Next Steps

- We recommend you review the [terrainbento manuscript](https://www.geosci-model-dev-discuss.net/gmd-2018-204/).

**The links to other notebooks will not work on GitHub. To use them you'll need to clone this repository and open the notebooks using [`jupyter notebook`](https://jupyter.org).**

- There are three additional introductory tutorials:

    1) [Introduction terrainbento](http://localhost:8888/notebooks/example_usage/Introduction_to_terrainbento.ipynb)

    2) [Introduction to boundary conditions in terrainbento](http://localhost:8888/notebooks/example_usage/introduction_to_boundary_conditions.ipynb)

    3) [Introduction to output writers in terrainbento](http://localhost:8888/notebooks/example_usage/introduction_to_output_writers.ipynb).

- Five examples of steady state behavior in coupled process models can be found in the following notebooks:

    1) [Basic](http://localhost:8888/notebooks/coupled_process_elements/model_basic_steady_solution.ipynb) the simplest landscape evolution model in the terrainbento package.

    2) [BasicVm](http://localhost:8888/notebooks/coupled_process_elements/model_basic_var_m_steady_solution.ipynb) which permits the drainage area exponent to change

    3) [BasicCh](http://localhost:8888/notebooks/coupled_process_elements/model_basicCh_steady_solution.ipynb) which uses a non-linear hillslope erosion and transport law

    4) [BasicVs](http://localhost:8888/notebooks/coupled_process_elements/model_basicVs_steady_solution.ipynb) which uses variable source area hydrology

    5) **This Notebook**: [BasisRt](http://localhost:8888/notebooks/coupled_process_elements/model_basicRt_steady_solution.ipynb) which allows for two lithologies with different K values_____no_output_____


# Notebook from LSSTScienceCollaborations/DMStackClub
Path: Validation/Ingest_and_load_local_refcat_demo.ipynb

# Ingest and load local refcat demo using DELVE_DR1

Owner: **Peter Ferguson** ([@psferguson](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@psferguson))

Last Verified to Run: **2022-04-01**

Verified Stack Release: **w_2021_49**_____no_output_____### Learning Objectives

This notebook demonstrates how to:

1. Create an LSST-format reference catalog from an existing acii or fits reference catalog
2. Create an empty Gen3 butler repo
3. Ingest the LSST-format refcat into newly created repo
4. Load the new reference catalog with the butler_____no_output_____### Set Up
You can find the Stack version by using `eups list -s` on the terminal command line._____no_output_____

# Site, host, and stack version
! echo $EXTERNAL_INSTANCE_URL
! echo $HOSTNAME
! eups list -s | grep lsst_distrib_____no_output_____import subprocess
import numpy as np
import pylab as plt
import lsst.geom
import lsst.daf.butler  as dafButler_____no_output_____

### Create a gen3 reference catalog

For this example we will create a refcat from a DELVE (DEcam Local Volume Exploration survey [Website](https://delve-survey.github.io/)) DR1 healpixel located on NCSA, and import it into a gen3 repo.

Following the developer instructions [piplines refcat documentation](https://pipelines.lsst.io/v/daily/modules/lsst.meas.algorithms/creating-a-reference-catalog.html)

The first step is to use the `ConvertReferenceCatalogTask` along with a config we create for the conversion to create a catalog in LSST format. _____no_output_____

# setting names
refcatDir='custom_refcat_demo'
configFile="ingestConfigOverride.cfg"
inputFile="/project/shared/data/delve_dr1/cat/cat_hpx_07798.fits"_____no_output_____# This notebook will only run if refcatDir doesn't exist
if os.path.exists(refcatDir):
    msg = f"Please remove directory '{refcatDir}' to continue:\n! rm -r {refcatDir}"
    raise Exception(msg)_____no_output_____! mkdir {refcatDir}_____no_output_____

Below is the set of configs used in creating this refcat
 1. Since the refcat is in FITS format we retarget the file reader
 2. It is required to give a name to this refcat, in this case 'delve_dr1'
 3. We also need to specify ra, dec, mag, and mag_error columns
 4. Finally we can give the config a list of extra columns to include in the refcat (e.g., a star/galaxy classifier)

for this tutorial we will change the HTM depth to 4 to improve runtime but an HTM depth of 7 is default.

The `%%writefile` cell magic writes the contents of the following cell to `configFile`._____no_output_____

%%writefile {configFile}
from lsst.meas.algorithms.readFitsCatalogTask import ReadFitsCatalogTask

# Default is ReadTextCatalogTask
config.file_reader.retarget(ReadFitsCatalogTask)

# String to pass to the butler to retrieve persisted files.
config.dataset_config.ref_dataset_name='delve_dr1'

config.dataset_config.indexer.name='HTM'

# Depth of the HTM tree to make.  Default is depth=7 which gives ~ 0.3 sq. deg. per trixel.
# for this tutorial we will change the HTM depth to 4 to improve runtime
config.dataset_config.indexer['HTM'].depth=4

# Number of python processes to use when ingesting.
config.n_processes=5

# Name of RA column
config.ra_name='RA'

# Name of Dec column
config.dec_name='DEC'

# Name of column to use as an identifier (optional).
config.id_name='QUICK_OBJECT_ID'

# The values in the reference catalog are assumed to be in AB magnitudes. List of column names to use for
# photometric information.  At least one entry is required.
config.mag_column_list=['MAG_PSF_G', 'MAG_PSF_R','MAG_PSF_I', 'MAG_PSF_Z']

# A map of magnitude column name (key) to magnitude error column (value).
config.mag_err_column_map={'MAG_PSF_G':'MAGERR_PSF_G', 'MAG_PSF_R':'MAGERR_PSF_R','MAG_PSF_I':'MAGERR_PSF_I', 'MAG_PSF_Z':'MAGERR_PSF_Z'}

# Names of extra columns to include
config.extra_col_names=['SPREAD_MODEL_G','SPREAD_MODEL_R','SPREAD_MODEL_I','SPREAD_MODEL_Z',
                        'SPREADERR_MODEL_G', 'SPREADERR_MODEL_R', 'SPREADERR_MODEL_I', 'SPREADERR_MODEL_Z',
                        'EXTINCTION_G', 'EXTINCTION_R', 'EXTINCTION_I', 'EXTINCTION_Z']
_____no_output_____

### Convert Files to LSST format
We then use the `convertReferenceCatalog` command line tool to ingest the catalog, this takes a bit of time to run. _____no_output_____

! convertReferenceCatalog {refcatDir} {configFile} {inputFile}_____no_output_____

example output:

    lsst.ConvertReferenceCatalogTask INFO: Creating 2048 file locks.
    lsst.ConvertReferenceCatalogTask INFO: File locks created.
    lsst.ConvertReferenceCatalogTask INFO: Completed 1 / 1 files: 100 % complete
    Completed refcat conversion. Ingest the resulting files with the following commands, substituting the path to your butler repo for REPO:
        butler register-dataset-type REPO delve_dr1 SimpleCatalog htm7
        butler ingest-files -t direct REPO gaia_dr2 refcats custom_refcat_demo/filename_to_htm.ecsv_____no_output_____### Create a gen3 repo and load catalog into it
We now have a LSST format refcat, for this demo we will create a new gen3 repo to ingest the refcat into.

The creation of an empty repo needs a `butler.yaml` file. (e.g `./custom_refcat_demo/butler.yaml`)._____no_output_____

# Create this file using notebook shell commands
filepath = "./custom_refcat_demo/butler.yaml"
filecontent = """
datastore:
  cls: lsst.daf.butler.datastores.fileDatastore.FileDatastore
  root: <butlerRoot>
registry:
  db: sqlite:///<butlerRoot>/test.sqlite3
"""
! echo "$filecontent" > $filepath
! cat $filepath_____no_output_____

Now we can run the `butler create` command line task to create a new repo._____no_output_____

repoName="test_repo_gen3"
! mkdir {refcatDir}/{repoName}
! touch {refcatDir}/{repoName}/test.sqlite3_____no_output_____!butler create {refcatDir}/{repoName} --seed-config {refcatDir}/butler.yaml --override_____no_output_____

Now that we have an empty gen3 repo we can ingest the catalog into it with the following commands.

    butler register-dataset-type REPO RefcatName SimpleCatalog htm3
    butler ingest-files -t direct REPO RefcatName collectionName filename_to_htm.ecsv
note the collection name must be a RUN not CHAIN type collection.

Also note that the htm option (htm4) must match with the ingestion config. _____no_output_____

!butler register-dataset-type {refcatDir}/{repoName} delve_dr1 SimpleCatalog htm4
!butler ingest-files -t direct {refcatDir}/{repoName} delve_dr1 refcats {refcatDir}/filename_to_htm.ecsv_____no_output_____

### Loading the new refcat
We can now load this new repo, and check the "refcats" collection to see what it contains. _____no_output_____

butler = dafButler.Butler(refcatDir+"/"+repoName, writeable=True)
registry = butler.registry_____no_output_____[i for i in list(registry.queryCollections())]_____no_output_____registry.getCollectionSummary('refcats').datasetTypes.names_____no_output_____

We can set a docstring for this refcat collection_____no_output_____

registry.setCollectionDocumentation('refcats', "doc for delve dr1 refcat")_____no_output_____

No longer need this butler repo to be writeable _____no_output_____

butler = dafButler.Butler(refcatDir+"/"+repoName, writeable=False)
registry = butler.registry_____no_output_____registry.getCollectionDocumentation('refcats')_____no_output_____refDataset="delve_dr1"
refcatRefs = list(registry.queryDatasets(datasetType=refDataset,
                                          collections=["refcats"]).expanded())
refDataIds=[_.dataId for _ in refcatRefs]
refCatsDef = [butler.getDeferred(refDataset, __, collections=['refcats']) for __ in refDataIds]_____no_output_____refCats=[butler.getDirect(__) for __ in refcatRefs]_____no_output_____

Finally we can plot the loaded refcat. The two different colors arise because this DELVE DR1 healpixel has been sharded into two htm4 pixels._____no_output_____

import pylab as plt
fit,ax=plt.subplots()
for refCat in refCats:
    ax.scatter(refCat["coord_ra"], refCat["coord_dec"], label="refcat",s=0.01)
plt.xlabel("RA")
plt.ylabel("DEC")_____no_output_____


# Notebook from learningequality/BasicCrawler
Path: notebooks/Crawling.ipynb

# Crawling tutorial

A web scraper script is anlogous to a spider that crawls a website's graph of URLs.
In this tutorial, we'll look at some examples of Python libraries for downloading and undestanding website stucture.

For optimal viewing of this notebook, you should enable the custom "wide format":
`cat wide_notebooks.css >> ../../venv/lib/python3.X/site-packages/notebook/static/custom/custom.css` (replace X with 4/5/6 depending on your version of Python)_____no_output_____

import json
from basiccrawler.crawler import BasicCrawler

crawler = BasicCrawler(
        main_source_domain='http://chef-take-home-test.learningequality.org',
        start_page='http://chef-take-home-test.learningequality.org/')
crawler.CRAWLING_STAGE_OUTPUT = 'chefdata/trees/takehome_web_resource_tree.json'

channel_tree = crawler.crawl(limit=1000)
crawler.print_tree(channel_tree, print_depth=1)

################################################################################
# CRAWLER RECOMMENDATIONS BASED ON URLS ENCOUNTERED:
################################################################################

1. These URLs are very common and look like global navigation links:

2. These are common path fragments found in URLs paths, so could correspond to site struture:
  -  7 urls on site start with  /Math
  -  7 urls on site start with  /Science
  -  1 urls on site start with  /

################################################################################

  - path: /  (PageWebResource)
    children:
     - path: /Math/  (PageWebResource)
       children counts: {'PageWebResource': 6}
     - path: /Science/  (PageWebResource)
       children counts: {'PageWebResource': 6}
crawler.print_tree(channel_tree, print_depth=2)
  - path: /  (PageWebResource)
    children:
     - path: /Math/  (PageWebResource)
       children:
        - path: /Math/Numbers/  (PageWebResource)
          children counts: {'PageWebResource': 2}
        - path: /Math/Functions/  (PageWebResource)
          children counts: {'PageWebResource': 1}
        - path: /Math/Vectors_overview/  (PageWebResource)
     - path: /Science/  (PageWebResource)
       children:
        - path: /Science/Biology/  (PageWebResource)
          children counts: {'PageWebResource': 1}
        - path: /Science/Chemistry/  (PageWebResource)
          children counts: {'PageWebResource': 1}
        - path: /Science/Physics/  (PageWebResource)
          children counts: {'PageWebResource': 1}
print(json.dumps(channel_tree, indent =2)){
  "kind": "PageWebResource",
  "url": "http://chef-take-home-test.learningequality.org/",
  "children": [
    {
      "kind": "PageWebResource",
      "url": "http://chef-take-home-test.learningequality.org/Math/",
      "children": [
        {
          "kind": "PageWebResource",
          "url": "http://chef-take-home-test.learningequality.org/Math/Numbers/",
          "children": [
            {
              "kind": "PageWebResource",
              "url": "http://chef-take-home-test.learningequality.org/Math/Numbers/Numbers_Overview/",
              "children": []
            },
            {
              "kind": "PageWebResource",
              "url": "http://chef-take-home-test.learningequality.org/Math/Numbers/Classifying_numbers/",
              "children": []
            }
          ]
        },
        {
          "kind": "PageWebResource",
          "url": "http://chef-take-home-test.learningequality.org/Math/Functions/",
          "children": [
            {
              "kind": "PageWebResource",
              "url": "http://chef-take-home-test.learningequality.org/Math/Functions/Functions_overview/",
              "children": []
            }
          ]
        },
        {
          "kind": "PageWebResource",
          "url": "http://chef-take-home-test.learningequality.org/Math/Vectors_overview/",
          "children": []
        }
      ]
    },
    {
      "kind": "PageWebResource",
      "url": "http://chef-take-home-test.learningequality.org/Science/",
      "children": [
        {
          "kind": "PageWebResource",
          "url": "http://chef-take-home-test.learningequality.org/Science/Biology/",
          "children": [
            {
              "kind": "PageWebResource",
              "url": "http://chef-take-home-test.learningequality.org/Science/Biology/Whale_sounds/",
              "children": []
            }
          ]
        },
        {
          "kind": "PageWebResource",
          "url": "http://chef-take-home-test.learningequality.org/Science/Chemistry/",
          "children": [
            {
              "kind": "PageWebResource",
              "url": "http://chef-take-home-test.learningequality.org/Science/Chemistry/Organic_chemistry/",
              "children": []
            }
          ]
        },
        {
          "kind": "PageWebResource",
          "url": "http://chef-take-home-test.learningequality.org/Science/Physics/",
          "children": [
            {
              "kind": "PageWebResource",
              "url": "http://chef-take-home-test.learningequality.org/Science/Physics/Wave_particle_duality/",
              "children": []
            }
          ]
        }
      ]
    }
  ]
}
from basiccrawler.crawler import BasicCrawler

skills_crawler = BasicCrawler(
        main_source_domain='https://applieddigitalskills.withgoogle.com',
        start_page='https://applieddigitalskills.withgoogle.com/en/curriculum')
skills_crawler.CRAWLING_STAGE_OUTPUT = 'chefdata/trees/skills_web_resource_tree.json'

# undomment below
skills_channel_tree = skills_crawler.crawl(limit=100)
# skills_crawler.print_tree(skills_channel_tree, print_depth=2)Starting new HTTPS connection (1): applieddigitalskills.withgoogle.com
https://applieddigitalskills.withgoogle.com:443 "HEAD /en/curriculum HTTP/1.1" 200 0
Downloaded page https://applieddigitalskills.withgoogle.com/en/curriculum title:Curriculum - Applied Digital Skills from Google
in on_page https://applieddigitalskills.withgoogle.com/en/curriculum
a with no nohref found Menu keyboard_arrow_down
https://applieddigitalskills.withgoogle.com:443 "HEAD /apps HTTP/1.1" 302 0
https://applieddigitalskills.withgoogle.com:443 "HEAD /en/apps HTTP/1.1" 200 0
https://applieddigitalskills.withgoogle.com:443 "GET /apps HTTP/1.1" 302 0
Downloaded page https://applieddigitalskills.withgoogle.com/apps title:Free Technology Curriculum from Google - Applied Digital Skills
in on_page https://applieddigitalskills.withgoogle.com/en/apps
a with no nohref found Menu keyboard_arrow_down
https://applieddigitalskills.withgoogle.com:443 "HEAD /high-school HTTP/1.1" 302 0
https://applieddigitalskills.withgoogle.com:443 "HEAD /en/high-school HTTP/1.1" 200 0
https://applieddigitalskills.withgoogle.com:443 "GET /high-school HTTP/1.1" 302 0
Downloaded page https://applieddigitalskills.withgoogle.com/high-school title:Learn Digital Skills to Solve Everyday Tasks - Applied Digital Skills
in on_page https://applieddigitalskills.withgoogle.com/en/high-school
a with no nohref found Menu keyboard_arrow_down
https://applieddigitalskills.withgoogle.com:443 "HEAD /adults HTTP/1.1" 302 0
https://applieddigitalskills.withgoogle.com:443 "HEAD /en/adults HTTP/1.1" 200 0
https://applieddigitalskills.withgoogle.com:443 "GET /adults HTTP/1.1" 302 0
Downloaded page https://applieddigitalskills.withgoogle.com/adults title:Grow Your Career with Applied Digital Skills - Applied Digital Skills
in on_page https://applieddigitalskills.withgoogle.com/en/adults
a with no nohref found Menu keyboard_arrow_down
https://applieddigitalskills.withgoogle.com:443 "HEAD /educators HTTP/1.1" 302 0
https://applieddigitalskills.withgoogle.com:443 "HEAD /en/educators HTTP/1.1" 200 0
https://applieddigitalskills.withgoogle.com:443 "GET /educators HTTP/1.1" 302 0
Downloaded page https://applieddigitalskills.withgoogle.com/educators title:Digital Tools to Help Learners Succeed Online - Applied Digital Skills
in on_page https://applieddigitalskills.withgoogle.com/en/educators
a with no nohref found Menu keyboard_arrow_down
https://applieddigitalskills.withgoogle.com:443 "HEAD /curriculum HTTP/1.1" 302 0
https://applieddigitalskills.withgoogle.com:443 "HEAD /en/curriculum HTTP/1.1" 200 0
https://applieddigitalskills.withgoogle.com:443 "GET /curriculum HTTP/1.1" 302 0
Downloaded page https://applieddigitalskills.withgoogle.com/curriculum title:Curriculum - Applied Digital Skills from Google
in on_page https://applieddigitalskills.withgoogle.com/en/curriculum
a with no nohref found Menu keyboard_arrow_down
https://applieddigitalskills.withgoogle.com:443 "HEAD /apps/route HTTP/1.1" 302 0
https://applieddigitalskills.withgoogle.com:443 "HEAD /hybridauth/window/Google?destination=apps/role&destination_error=apps/role HTTP/1.1" 302 0
https://applieddigitalskills.withgoogle.com:443 "HEAD /hybridauth/endpoint?hauth.start=Google&hauth.time=1511672631 HTTP/1.1" 302 0
Starting new HTTPS connection (1): accounts.google.com
https://accounts.google.com:443 "HEAD /o/oauth2/auth?client_id=697756715164-tg5ir0ti7hocn6mbb5m390n3d6gqoskc.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fapplieddigitalskills.withgoogle.com%2Fhybridauth%2Fendpoint%3Fhauth.done%3DGoogle&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&access_type=online&prompt=select_account&hl=en HTTP/1.1" 302 0
https://accounts.google.com:443 "HEAD /ServiceLogin?passive=1209600&continue=https://accounts.google.com/o/oauth2/auth?client_id%3D697756715164-tg5ir0ti7hocn6mbb5m390n3d6gqoskc.apps.googleusercontent.com%26redirect_uri%3Dhttps://applieddigitalskills.withgoogle.com/hybridauth/endpoint?hauth.done%253DGoogle%26response_type%3Dcode%26scope%3Dhttps://www.googleapis.com/auth/userinfo.profile%2Bhttps://www.googleapis.com/auth/userinfo.email%26access_type%3Donline%26prompt%3Dselect_account%26hl%3Den%26from_login%3D1%26as%3D327d64101a0d1455&hl=en&oauth=1&sarp=1&scc=1 HTTP/1.1" 200 0
https://applieddigitalskills.withgoogle.com:443 "GET /apps/route HTTP/1.1" 302 0
https://applieddigitalskills.withgoogle.com:443 "GET /hybridauth/window/Google?destination=apps/role&destination_error=apps/role HTTP/1.1" 302 0
https://applieddigitalskills.withgoogle.com:443 "GET /hybridauth/endpoint?hauth.start=Google&hauth.time=1511672632 HTTP/1.1" 302 0
https://accounts.google.com:443 "GET /o/oauth2/auth?client_id=697756715164-tg5ir0ti7hocn6mbb5m390n3d6gqoskc.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fapplieddigitalskills.withgoogle.com%2Fhybridauth%2Fendpoint%3Fhauth.done%3DGoogle&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&access_type=online&prompt=select_account&hl=en HTTP/1.1" 302 0
https://accounts.google.com:443 "GET /ServiceLogin?passive=1209600&continue=https://accounts.google.com/o/oauth2/auth?client_id%3D697756715164-tg5ir0ti7hocn6mbb5m390n3d6gqoskc.apps.googleusercontent.com%26redirect_uri%3Dhttps://applieddigitalskills.withgoogle.com/hybridauth/endpoint?hauth.done%253DGoogle%26response_type%3Dcode%26scope%3Dhttps://www.googleapis.com/auth/userinfo.profile%2Bhttps://www.googleapis.com/auth/userinfo.email%26access_type%3Donline%26prompt%3Dselect_account%26hl%3Den%26from_login%3D1%26as%3D-351fe9e05fe0beb0&hl=en&oauth=1&sarp=1&scc=1 HTTP/1.1" 200 None
Downloaded page https://applieddigitalskills.withgoogle.com/apps/route title:Sign in - Google Accounts
in on_page https://accounts.google.com/ServiceLogin?passive=1209600&continue=https://accounts.google.com/o/oauth2/auth?client_id%3D697756715164-tg5ir0ti7hocn6mbb5m390n3d6gqoskc.apps.googleusercontent.com%26redirect_uri%3Dhttps://applieddigitalskills.withgoogle.com/hybridauth/endpoint?hauth.done%253DGoogle%26response_type%3Dcode%26scope%3Dhttps://www.googleapis.com/auth/userinfo.profile%2Bhttps://www.googleapis.com/auth/userinfo.email%26access_type%3Donline%26prompt%3Dselect_account%26hl%3Den%26from_login%3D1%26as%3D-351fe9e05fe0beb0&hl=en&oauth=1&sarp=1&scc=1
https://applieddigitalskills.withgoogle.com:443 "HEAD /user/logout?destination=/apps HTTP/1.1" 403 0
ERROR 403 when getting url=https://applieddigitalskills.withgoogle.com/user/logout?destination=/apps
HEAD request failed for url https://applieddigitalskills.withgoogle.com/user/logout?destination=/apps
HEAD https://applieddigitalskills.withgoogle.com/user/logout?destination=/apps did not return response.
Starting new HTTPS connection (2): applieddigitalskills.withgoogle.com
https://applieddigitalskills.withgoogle.com:443 "HEAD /apps/dashboard HTTP/1.1" 403 0
ERROR 403 when getting url=https://applieddigitalskills.withgoogle.com/apps/dashboard
HEAD request failed for url https://applieddigitalskills.withgoogle.com/apps/dashboard
HEAD https://applieddigitalskills.withgoogle.com/apps/dashboard did not return response.
Starting new HTTPS connection (3): applieddigitalskills.withgoogle.com
https://applieddigitalskills.withgoogle.com:443 "HEAD /applied-digital-skills/en/if-then-adventure-stories/overview.html HTTP/1.1" 200 0
Downloaded page https://applieddigitalskills.withgoogle.com/applied-digital-skills/en/if-then-adventure-stories/overview.html title:If-Then Adventure Stories - Applied Digital Skills from Google
in on_page https://applieddigitalskills.withgoogle.com/applied-digital-skills/en/if-then-adventure-stories/overview.html
a with no nohref found Menu keyboard_arrow_down
# le_crawler.print_tree(le_channel_tree, print_depth=5)_____no_output_____# le_crawler.infer_tree_structure(le_channel_tree)
_____no_output_____import re
from crawler import BasicCrawler

ict_crawler = BasicCrawler(
        main_source_domain='https://ict-essentials-for-teachers.moodlecloud.com',
        start_page='https://ict-essentials-for-teachers.moodlecloud.com/')
ict_crawler.IGNORE_URLS.append(re.compile('.*[\?&]lang=.*'))
ict_crawler.IGNORE_URLS.append(re.compile('.*calendar/set\.php.*'))
ict_crawler.IGNORE_URLS.append(re.compile('.*calendar/view\.php.*'))

# ict_channel_tree = ict_crawler.crawl(limit=1000)
_____no_output_____# ict_crawler.print_tree(ict_channel_tree, print_depth=2)_____no_output_____# ict_crawler.infer_tree_structure(ict_channel_tree)
_____no_output_____


# Notebook from zoltcom/machine-learning-notes
Path: svm/Classifying_Data_Using_a_Support_Vector_Machine/support_vector_machine_lesson.ipynb

# The Math of Intelligence (Week 1) - Support Vector Machine Classification

## What will we do?

We will build a Support Vector Machine that will find the optimal hyperplane that maximizes the margin between two toy data classes using gradient descent.

![alt text](http://opticalengineering.spiedigitallibrary.org/data/journals/optice/24850/oe_52_2_027003_f005.png "Logo Title Text 1")

## What are some use cases for SVMs?

-Classification, regression (time series prediction, etc) , outlier detection, clustering

## Other Examples

- Learning to use Scikit-learn's SVM function to classify images https://github.com/ksopyla/svm_mnist_digit_classification
- Pulse classification, more useful dataset
https://github.com/akasantony/pulse-classification-svm

## How does an SVM compare to other ML algorithms?

![alt text](https://image.slidesharecdn.com/mscpresentation-140722065852-phpapp01/95/msc-presentation-bioinformatics-7-638.jpg?cb=1406012610 "Logo Title Text 1")

- As a rule of thumb, SVMs are great for relatively small data sets with fewer outliers.
- Other algorithms (Random forests, deep neural networks, etc.) require more data but almost always come up with very robust models.
- The decision of which classifier to use depends on your dataset and the general complexity of the problem.
- "Premature optimization is the root of all evil (or at least most of it) in programming." - Donald Knuth, CS Professor (Turing award speech 1974)

## What is a Support Vector Machine?

It's a supervised machine learning algorithm which can be used for both classification or regression problems. But it's usually used for classification. Given 2 or more labeled classes of data, it acts as a discriminative classifier, formally defined by an optimal hyperplane that seperates all the classes. New examples that are then mapped into that same space can then be categorized based on on which side of the gap they fall.

## What are Support Vectors?

![alt text](https://www.dtreg.com/uploaded/pageimg/SvmMargin2.jpg "Logo Title Text 1")

Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane. Because of this, they can be considered the critical elements of a data set, they are what help us build our SVM.

## Whats a hyperplane?

![alt text](http://slideplayer.com/slide/1579281/5/images/32/Hyperplanes+as+decision+surfaces.jpg "Logo Title Text 1")

Geometry tells us that a hyperplane is a subspace of one dimension less than its ambient space. For instance, a hyperplane of an n-dimensional space is a flat subset with dimension n − 1. By its nature, it separates the space into two half spaces.

## Linear vs nonlinear classification?

Sometimes our data is linearly seperable. That means for N classes with M features, we can learn a mapping that is a linear combination. (like y = mx + b). Or even a multidimensional hyperplane (y = x + z + b + q). No matter how many dimensions/features a set of classes have, we can represent the mapping using a linear function.

But sometimes its not. Like if there was a quadratic mapping. Luckily for us SVMs can can efficiently perform a non-linear classification using what is called the kernel trick. I'll talk about that lter on in the course.

![alt text](https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAuSAAAAJDlhYzcwMzhlLTA0MjYtNDEyYS1hMWM4LTE3Zjk5NDlhNzVkMQ.png "Logo Title Text 1")

Alright let's get to building!_____no_output_____

#To help us perform math operations
import numpy as np
#to plot our data and model visually
from matplotlib import pyplot as plt
%matplotlib inline

#Step 1 - Define our data

#Input data - Of the form [X value, Y value, Bias term]
X = np.array([
    [-2,4,-1],
    [4,1,-1],
    [1, 6, -1],
    [2, 4, -1],
    [6, 2, -1],
])

#Associated output labels - First 2 examples are labeled '-1' and last 3 are labeled '+1'
y = np.array([-1,-1,1,1,1])

#lets plot these examples on a 2D graph!
#for each example
for d, sample in enumerate(X):
    # Plot the negative samples (the first 2)
    if d < 2:
        plt.scatter(sample[0], sample[1], s=120, marker='_', linewidths=2)
    # Plot the positive samples (the last 3)
    else:
        plt.scatter(sample[0], sample[1], s=120, marker='+', linewidths=2)

# Print a possible hyperplane, that is seperating the two classes.
#we'll two points and draw the line between them (naive guess)
plt.plot([-2,6],[6,0.5])_____no_output_____

## Let's define our loss function (what to minimize) and our objective function (what to optimize)

#### Loss function

We'll use the Hinge loss. This is a loss function used for training classifiers. The hinge loss is used for "maximum-margin" classification, most notably for support vector machines (SVMs).

![alt text](http://i.imgur.com/OzCwzyN.png "Logo Title Text 1")

c is the loss function, x the sample, y is the true label, f(x) the predicted label.

![alt text](http://i.imgur.com/FZ7JcG3.png "Logo Title Text 1")

#### Objective Function

![alt text](http://i.imgur.com/I5NNu44.png "Logo Title Text 1")

As you can see, our objective of a SVM consists of two terms. The first term is a regularizer, the heart of the SVM, the second term the loss. The regularizer balances between margin maximization and loss. We want to find the decision surface that is maximally far away from any data points.

How do we minimize our loss/optimize for our objective (i.e learn)?

We have to derive our objective function to get the gradients! Gradient descent ftw.  As we have two terms, we will derive them seperately using the sum rule in differentiation.

![alt text](http://i.imgur.com/6uK3BnH.png "Logo Title Text 1")

This means, if we have a misclassified sample, we update the weight vector w using the gradients of both terms, else if classified correctly,we just update w by the gradient of the regularizer.

Misclassification condition

![alt text](http://i.imgur.com/g9QLAyn.png "Logo Title Text 1")

Update rule for our weights (misclassified)

![alt text](http://i.imgur.com/rkdPpTZ.png "Logo Title Text 1")

including the learning rate η and the regularizer λ
The learning rate is the length of the steps the algorithm makes down the gradient on the error curve.
- Learning rate too high? The algorithm might overshoot the optimal point.
- Learning rate too low? Could take too long to converge. Or never converge.

The regularizer controls the trade off between the achieving a low training error and a low testing error that is the ability to generalize your classifier to unseen data. As a regulizing parameter we choose 1/epochs, so this parameter will decrease, as the number of epochs increases.
- Regularizer too high? overfit (large testing error)
- Regularizer too low? underfit (large training error)

Update rule for our weights (correctly classified)

![alt text](http://i.imgur.com/xTKbvZ6.png "Logo Title Text 1")
_____no_output_____

#lets perform stochastic gradient descent to learn the seperating hyperplane between both classes

def svm_sgd_plot(X, Y):
    #Initialize our SVMs weight vector with zeros (3 values)
    w = np.zeros(len(X[0]))
    #The learning rate
    eta = 1
    #how many iterations to train for
    epochs = 100000
    #store misclassifications so we can plot how they change over time
    errors = []

    #training part, gradient descent part
    for epoch in range(1,epochs):
        error = 0
        for i, x in enumerate(X):
            #misclassification
            if (Y[i]*np.dot(X[i], w)) < 1:
                #misclassified update for ours weights
                w = w + eta * ( (X[i] * Y[i]) + (-2  *(1/epoch)* w) )
                error = 1
            else:
                #correct classification, update our weights
                w = w + eta * (-2  *(1/epoch)* w)
        errors.append(error)

    #lets plot the rate of classification errors during training for our SVM
    plt.plot(errors, '|')
    plt.ylim(0.5,1.5)
    plt.axes().set_yticklabels([])
    plt.xlabel('Epoch')
    plt.ylabel('Misclassified')
    plt.show()

    return w_____no_output_____for d, sample in enumerate(X):
    # Plot the negative samples
    if d < 2:
        plt.scatter(sample[0], sample[1], s=120, marker='_', linewidths=2)
    # Plot the positive samples
    else:
        plt.scatter(sample[0], sample[1], s=120, marker='+', linewidths=2)

# Add our test samples
plt.scatter(2,2, s=120, marker='_', linewidths=2, color='yellow')
plt.scatter(4,3, s=120, marker='+', linewidths=2, color='blue')

#w = svm_sgd_plot(X,y)

# Print the hyperplane calculated by svm_sgd()
x2=[w[0],w[1],-w[1],w[0]]
x3=[w[0],w[1],w[1],-w[0]]

x2x3 =np.array([x2,x3])
X,Y,U,V = zip(*x2x3)
ax = plt.gca()
ax.quiver(X,Y,U,V,scale=1, color='blue')_____no_output_____w = svm_sgd_plot(X,y)
#they decrease over time! Our SVM is learning the optimal hyperplane/home/silver/miniconda3/lib/python3.7/site-packages/ipykernel_launcher.py:31: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.


# Notebook from znreza/PySyft
Path: examples/homomorphic-encryption/Tutorial_2_TenSEAL_Syft_Data_Scientist.ipynb

# Homomorphic Encryption using Duet: Data Owner
## Tutorial 2: Encrypted image evaluation

Welcome!
Welcome! This tutorial will show you how to evaluate Encrypted images using Duet and TenSEAL. This notebook shows the Data Scientist view on the operations.

We recommend going through Tutorial 0 and 1 before trying this one._____no_output_____## Setup

All modules are imported here, make sure everything is installed by running the cell below._____no_output_____

import pickle
import requests
import os

import syft as sy
import tenseal as ts

from typing import Dict

sy.load_lib("tenseal")
_____no_output_____

### Join the Duet_____no_output_____

duet = sy.join_duet(loopback=True)_____no_output_____

###  Checkpoint 1 : Now STOP and run the Data Owner notebook until the next checkpoint._____no_output_____### Define The Model

Here we define the model used by the Data Scientist. he model contains a convolution and two fully connected layers.

The input should be an encoded 28x28 matrix representing the image.
TenSEAL can be used for encoding `tenseal.im2col_encoding(ctx, input_matrix, 7, 7, 3)`
The input should also be normalized with a mean=0.1307 and an std=0.3081 before encryption._____no_output_____

# Load a pretrained model and adapt the forward call for encrypted input
class ConvMNIST():
    def __init__(self, parameters: Dict[str, list]):
        self.conv1_weight = parameters["conv1_weight"]
        self.conv1_bias = parameters["conv1_bias"]
        self.fc1_weight = parameters["fc1_weight"]
        self.fc1_bias = parameters["fc1_bias"]
        self.fc2_weight = parameters["fc2_weight"]
        self.fc2_bias = parameters["fc2_bias"]
        self.windows_nb = parameters["windows_nb"]

    def forward(self, enc_x: ts.CKKSVector) -> ts.CKKSVector:
        # conv layer
        channels = []
        for kernel, bias in zip(self.conv1_weight, self.conv1_bias):
            y = enc_x.conv2d_im2col(kernel, self.windows_nb) + bias
            channels.append(y)
        out = ts.CKKSVector.pack_vectors(channels)
        # squaring
        out.square_()
        # no need to flat
        # fc1 layer
        out = out.mm_(self.fc1_weight) + self.fc1_bias
        # squaring
        out.square_()
        # output layer
        out = out.mm_(self.fc2_weight) + self.fc2_bias
        return out

    def __call__(self, *args, **kwargs):
        return self.forward(*args, **kwargs)_____no_output_____

### Load the pretrained parameters_____no_output_____

model_path = "parameters/ConvMNIST-0.1.pickle"

def download_parameters():
    try:
        os.mkdir("parameters")
    except BaseException as e:
        pass

    url = "https://github.com/OpenMined/TenSEAL/raw/master/tutorials/parameters/ConvMNIST-0.1.pickle"

    r = requests.get(url)

    with open(model_path, 'wb') as f:
        f.write(r.content)

def load_parameters() -> dict:
    try:
        download_parameters()
        parameters = pickle.load(open(model_path, "rb"))
        print(f"Model loaded from '{model_path}'")
    except OSError as ose:
        print("error", ose)
        raise ose
    return parameters

parameters = load_parameters()
model = ConvMNIST(parameters)_____no_output_____

### Request the image for evaluation_____no_output_____

duet.store.pandas_____no_output_____ctx_ptr = duet.store["context"]
enc_image_ptr = duet.store["enc_image"]

ctx_ptr.request(reason="I would like to get the context")
enc_image_ptr.request(reason="I would like to get encrypted evaluation image")_____no_output_____duet.requests.pandas_____no_output_____

###  Checkpoint 2 : Now STOP and run the Data Owner notebook until the next checkpoint._____no_output_____### Evaluate the encrypted image_____no_output_____

enc_image = enc_image_ptr.get(delete_obj=False)
ctx = ctx_ptr.get(delete_obj=False)

enc_image.link_context(ctx)

encrypted_result = model(enc_image)_____no_output_____

### Send the encrypted result back to the DO instance_____no_output_____

encrypted_result_ptr = encrypted_result.send(duet, pointable=True, tags=["result"])

print(duet.store)_____no_output_____

###  Checkpoint 3 : Now STOP and run the Data Owner notebook until the next checkpoint._____no_output_____# Congratulations!!! - Time to Join the Community!

Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the movement toward privacy preserving, decentralized ownership of AI and the AI supply chain (data), you can do so in the following ways!

### Star PySyft and TenSEAL on GitHub

The easiest way to help our community is just by starring the Repos! This helps raise awareness of the cool tools we're building.

- [Star PySyft](https://github.com/OpenMined/PySyft)
- [Star TenSEAL](https://github.com/OpenMined/TenSEAL)

### Join our Slack!

The best way to keep up to date on the latest advancements is to join our community! You can do so by filling out the form at [http://slack.openmined.org](http://slack.openmined.org). #lib_tenseal and #code_tenseal are the main channels for the TenSEAL project.

### Donate

If you don't have time to contribute to our codebase, but would still like to lend support, you can also become a Backer on our Open Collective. All donations go toward our web hosting and other community expenses such as hackathons and meetups!

[OpenMined's Open Collective Page](https://opencollective.com/openmined)_____no_output_____


# Notebook from Shubhamdutta2000/semester-4-works
Path: FINAL_Practical/AL_ML_advanced/codes/Q3.ipynb

<code>
#i
from sklearn.datasets import load_digits
digits = load_digits()
print(digits.DESCR).. _digits_dataset:

Optical recognition of handwritten digits dataset
--------------------------------------------------

**Data Set Characteristics:**

    :Number of Instances: 5620
    :Number of Attributes: 64
    :Attribute Information: 8x8 image of integer pixels in the range 0..16.
    :Missing Attribute Values: None
    :Creator: E. Alpaydin (alpaydin '@' boun.edu.tr)
    :Date: July; 1998

This is a copy of the test set of the UCI ML hand-written digits datasets
https://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits

The data set contains images of hand-written digits: 10 classes where
each class refers to a digit.

Preprocessing programs made available by NIST were used to extract
normalized bitmaps of handwritten digits from a preprinted form. From a
total of 43 people, 30 contributed to the training set and different 13
to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of
4x4 and the number of on pixels are counted in each block. This generates
an input matrix of 8x8 where each element is an integer in the range
0..16. This reduces dimensionality and gives invariance to small
distortions.

For info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G.
T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C.
L. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469,
1994.

.. topic:: References

  - C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their
    Applications to Handwritten Digit Recognition, MSc Thesis, Institute of
    Graduate Studies in Science and Engineering, Bogazici University.
  - E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika.
  - Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin.
    Linear dimensionalityreduction using relevance weighted LDA. School of
    Electrical and Electronic Engineering Nanyang Technological University.
    2005.
  - Claudio Gentile. A New Approximate Maximal Margin Classification
    Algorithm. NIPS. 2000.
#ii
digits.data.shape_____no_output_____#ii
digits.target.shape_____no_output_____#iii
print(digits.data[204])
image = digits.data[204][ 0.  4. 16. 16. 16. 16.  5.  0.  0. 11. 16.  8.  5.  8.  3.  0.  0. 10.
 16.  2.  0.  0.  0.  0.  0.  3. 16.  6.  0.  0.  0.  0.  0.  0. 16.  9.
  0.  0.  0.  0.  0.  0. 12. 16.  2.  0.  0.  0.  0.  0.  6. 16. 11.  0.
  0.  0.  0.  4. 16. 12.  1.  0.  0.  0.]
#iv
import numpy as np
np.reshape(image,(8,8))_____no_output_____#v
import matplotlib.pyplot as plt
plt.imshow(np.reshape(image,(8,8)), cmap='gray')_____no_output_____#vi
from sklearn.model_selection import train_test_split
x_train_1,x_test_1,y_train_1,y_test_1=train_test_split(digits.data,digits.target,test_size=0.20)_____no_output_____
</code>

**Not done.**

![WhatsApp Image 2021-05-18 at 1.21.29 PM.jpeg](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/4QAiRXhpZgAATU0AKgAAAAgAAQESAAMAAAABAAEAAAAAAAD/2wBDAAIBAQIBAQICAgICAgICAwUDAwMDAwYEBAMFBwYHBwcGBwcICQsJCAgKCAcHCg0KCgsMDAwMBwkODw0MDgsMDAz/2wBDAQICAgMDAwYDAwYMCAcIDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAz/wAARCABlAkQDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD9/KKKKACiignBoAKKM0bqACijdRuoAKKN1G6gAoo3UbqACijdRuoAKKN1G6gAoo3UbqACijdRuoAKKN1G6gAoo3UbqACijdRuoAKKN1G6gAoo3UbqACijdRuoAKKN1G6gAoo3UbqACijdRuoAKKN1G6gAoo3UbqACijdRuoAKKN1G6gAoo3UbqACijdRuoAKKN1G6gAoo3UbqACijdRuoAKKN1G6gAoo3UbqACijdRuoAKKN1G6gAoo3UbqACijdSFsUALRQDmigAooooAKKKKACiiigArg/jr+0Jov7P2mWN3rUd7JHfymGMW6BiCBnnJFd5XzV/wUZ8MXfi/QPDFlYqsly11PKELYLBISzY98An8K4cwxE6NCVSG6Mq03GHMi8P+ClHgTP/AB6699Ps6H/2aj/h5T4EH/Lrrn/fhP8A4uvjrRvAdzd+H7y8+y/aI0tlmSRbtIzEGl2AsDyQTwOw696o+O9Abwt4qurNrWSyNuU/dPMJmQEAnLKMHrn/ABr4uXEmNUbtI8j+0Ku59qf8PKPAv/Ppr3/gOn/xVJ/w8p8C/wDPnrx9hbpz/wCPV8fv4T08fBmHXf33259WNg4MmYwnl784655/Kk1H4etqMFnJpsLRwtaxSTPPKAvmuGO0HGP4TgE9B71X+seNtflQ/wC0KvY+wf8Ah5V4Dz/x665/34T/AOLo/wCHlPgT/n017/wHT/4qvkBfhZqmsT20NjYhZW0s6m5e9jIliXcTIOQFG0fd68e9RzfCHXINUe1eCEMohIlEytC/nf6rD9Du7D2NT/rJjukUP69V7H2IP+Ck/gVjxaa9nIGPs6f/ABVJ/wAPKPAjdLXXm+lsp/8AZq+RPht8Ok1r4w6f4Y1pJYRdXX2W4EMi74zznDDI49j2q3a/D3T/ABp4W1660cXsOoeHUFzNa3DCVZ4DIqFkbAwyls46EfTFVHiHGzV0kH16s+h9Y/8ADyfwLn/j1178LdD/AOzUg/4KUeBf+fPXv/AZf/iq+PLn4Oa/DbJI1jmMyPFIQwP2dlXewkH8GF5ORgcjrxViL4SamukXX/Eu+1TedaQwTQXKMg8/OwbR97eMAE4Awe9R/rJjm7cqF9fqn14f+ClHgUf8uev/APgMv/xVIf8AgpT4EH/Lrr3/AIDp/wDFV8cWHw+ukmuEkt4bvy9Olvf3N0i+WqbkLH12spBXGeM4wQadp/wj1zVdiW9qkkjSRRyR+aqvAZeY94JG0Nxg9Oe1H+smO/lQ/r9XsfYv/DynwIR/x669/wCA6f8AxVIP+ClPgQ/8uuvH6Wyn/wBmr5BvPAMngeKC58QWkktveGSK3+y3SH96jKHVmG7BUNnBHcetbfjr4UWJ+Il14V8O297catb3CLGbi5Ty5UMW85zjBzjA74q/9Ysba7iifr9U+pP+HlHgUH/j017/AMBk/wDiqUf8FJ/Ap/5c9eP/AG7Lx/49Xxy/wj1oLcGO3gm8iN5v3U6SeaqgFim3O7aDyATjBHWrWhfA/XNb1jS7X7OsK6lepY+YcN9mlYBsOO3yZOD1x7Gp/wBYsc3blQ/r9U+ux/wUp8CH/l017/wGX/4r/OaP+HlHgX/n017/AMB0/wDiq+PhB4XuNWuLWFNS8qM+Xa3DSjdcFWA5XHy55wBnGBV7xf8AB66tfFmtQ6TD5lnps0yRiWVfMdIlDORxzhTk0/8AWLG9Ehf2hV6H1of+ClPgUf8ALpr3/gMv/wAVSH/gpT4EA/49Ne6Z4tl/+Kr4zt/hrq09vbSC3j3Xhj8uMyKJCHR3Vsf3SqMcjPA9xme3+H1x/ZN27QxuY57eJbpLlPLi80ErlepyCPp3xRHiTHN25UP+0KvY+xR/wUp8CE4+y691wP8AR05/8epf+HlHgX/nz178bZf/AIqvlnx18MbDQZdb02ysri6vNAYLcXK3yGPBdEU+XjOd2QcZxxWLa/BTxDeIhjso5AyzZPnIqo0K7pEYk4DBecE8jJGQDhviLGraKH/aFXsfYA/4KUeBSf8Aj017/wAB0/8Ai6T/AIeU+BMf8emvf+A6f/FV8eeCPhz/AMJH8RIdDuJrZBvcSSJMjRsEUsdrDhs46jNW/E3wpvBBp95p9ksNrqaRrbRG+SeW5kLNGSgBDMCyHjGR3xS/1kxrXMooX9oVT65/4eUeBf8An017/wAB0/8AiqQ/8FKvAY/5ddd/8B0/+Lr49s/g7rupIslvBb3ELQTTiZJ0Me2EZlGd33lHJA5xg9CCV1z4Pa54b0+4urqO1ENrFHO/l3cbExSY2yABslTkc0v9ZMcldxQ/7Qq9j7BH/BSnwISP9E17k4/490/+Ko/4eU+Bf+fPX+Bkj7Mv/wAVXxRpvhC71PS2vtsa2as4LSSBPM2IHdVJGNwVgcZzyOK6Px78M4fCvjiaGNJI9KW6ht41lkCzOWRGbGRk4yeegxyRVR4jxrjzKKD+0KvY+tT/AMFJ/Ao/5dNe/wDAZf8A4qgf8FKPAn/Prrv/AIDp/wDF18meKfgzfL401qz0W2aazsLmeO3DyKZJxHywX+8Qvp17ZrLk+FOuQra/6PETeXKWagTINkzpvRG5AUleRmh8RY7+VB/aFXsfY/8Aw8o8C/8APpr3/gOn/wAVR/w8o8C4/wCPTXvp9nT/AOKr5Jl/Z/8AEyXCxmDT2bz/ALKdt/A2yb5v3Zwxwx2NwetZPg74eXXjDX73TVkgs5tPtri5nMzYEfkozEHHuP0qf9Zcd/Kg/tCr2Psz/h5R4E/59de/8B0/+Ko/4eUeBf8An017/wAB0/8Aiq+Prn4M65a2UNw8dn5dxGs6BbuJmETKzCTaCTtwp56Z46mlHwV8RHVHs2sQLhceWGdVFxlN42E/eypB47Gn/rJjlvBB/aFXsfYH/DyjwL/z6a9/4Dp/8VQP+ClHgU/8uevHp0tl/wDiq+JdD8I3viSW8jtYkMljbyXMqM4Vgkf38DIyQOcDnAPpWnZfCvVBrtvazW8UvmJbzhBOF82OfHl7WPQkHNKPE2Neqihf2hVZ9jf8PKfAmP8Aj017/wABl/8AiqVv+ClPgRWx9l138bdR/wCzV8seGPgylz4u8O/2k0cen67qstj9njnVpotkgQ5bGD17elct4y8D3vgqaB7hY/st7va3kjmWYMqsVIJUnBBGDmqlxJjUr8qH/aFXsfZ4/wCClPgQj/j113/wHT/4ql/4eUeBc/8AHpr34Wyn/wBmr45034Ra5rktvHDbxtNcSRxCJp0jdDIpdNwbGNyjIrY0j4H6hqUNja+Xaxz318LYXgv4pIkzFv8ALKrkhgMnmpXEeOf2UL+0Kp9Xf8PKPAv/AD6a9/4DL/8AFUn/AA8p8CZ/49Ne/wDAZf8A4qvjRfhpq0mm3V1HDHJDaRvMxDrueJG2NIo7qD6eh9DjUf4Nato0V62oWdvK1vYtdeXFfw+ZCu1HEhVSSV2uO38jR/rJjtuVD/tCr2Prj/h5R4F/589e/wDAZf8A4qj/AIeUeBf+fTXvxt0/+Lr46svg/rmptHHFZxyTPNHA6eYoaBpeYw/QLuHT8uvFUvFPw/1LwXZWVxfQxJBqBkELxTLIrNGQHUlScMuRke4pf6y45bxQf2hV7H2j/wAPKfAv/Ppr3/gOn/xdKP8AgpP4FI/489e/8B0/+Kr4xs/htq+o6QL6G2WaNgjhVcGTa8nlq23qAXBHPeqXiLw1d+GZ4o7pYd0wJAR1bGGKkHjg5U8H8aX+s2NSu4oP7Qq9j7b/AOHlHgX/AJ9Ne/8AAdP/AIqk/wCHlPgMf8uuvf8AgOvP/j1fIVj8D/EmosqwWMbMzPG6mZVaJ0TzGRgSMMFycexHUYp1l8GdWmvLuGaSztfsumHVA5mDRzRA7RtYEg/Nke2Kr/WTHfyoP7Qq9j67H/BSnwL/AM+mvf8AgOn/AMVS/wDDynwJj/j113/vwn/xdfH+r/BzXNBWNruKziV4Bcti6jbyoiqsHYAkhTvGD3JwOamsvgN4k1G4eNLS3i2Tpb7pbqOJWd0MiYJIyGUZB75p/wCsmO/lQfX6vY+uv+HlHgX/AJ9dc/78J/8AF0f8PKPAv/Ppr3/gOn/xdfGlx8L9ZtLe3klt0jFxePp4DSjck6feVhnK465PGOaoaH4VuvEFzcRwfZ8WoBlkeVRGgLBB83uzAcevpk1H+s2N25UT/aFU+2h/wUn8Cn/l017/AMB1/wDiqP8Ah5P4Fz/x5696f8ey/wDxVfIMHwR8RTK2LKFWWa4g2NcxK7SQECVApOcjP41Y8A/Cr/hINN1K61KOSCFNIudQtR5ojeXy+A+MZK5rRcRY6+sUV/aFXsfWv/DynwL/AM+evf8AgOnH/j9B/wCClHgUf8uevf8AgOn/AMVXxh4Y+H2peMbSW4so7byIZ44JHlnjiVHk3FQdxHHynnpWhffBXxDYEedZwoVmlt5R58ZNu8SlnEgB+XCgsM9QOM1P+smO6RQf2hV7H1+P+ClPgX/n017/AMB0/wDiqP8Ah5T4F/589e/8Bl/+Kr5Ai+CviCaa6jS1h3WgieT98nEcuCkv/XMgg7ugzziuXntWgmeNtrMjFSQAQfoenNRPifGx3iif7RqLc+5x/wAFKfApP/Hnr3/gMv8A8VS/8PKPAv8Az569/wCAy/8AxVfJPgr4XLqnhnXL/UlaL7LpDajZKsoV2wwUOVwSVOcVX0f4GeJNe8n7Lp6SNPKkRXzUVo2ZCy71JyuVBI9cHHNX/rJjraRRX16ra9j6+/4eUeBR/wAuevf+Ay//ABVA/wCClHgU/wDLnr3/AIDp/wDFV8kw/AjWYrae6uG0+3t4bBtQWVbmORJ0VwjIrJkbgxwR1FSfEX4QT+H72S402NW0lVtF3S3MZkjaaJGBccFVLEgE4HFV/rFjrczig/tCr2PrL/h5R4FB/wCPTXvX/j3T/wCKoP8AwUn8Cgf8emvfX7OhH578V8Zt4EuLDy5tQa0FiroZjBfQSSBSQMqquSSMntjr2FdVa/DDQ/GGr6DJpMl7Z2upWt1dXsF2VkmgWDJYghRneA2AQDkGiPEeNltFAswqs+rNA/4KHeCvEWv2Wnw2uuLNfTx28Za3UKGdgoz83TkV71G29c1+aXh/wlBpXjf4f6tZ+cLPWr6J0jkbc0TR3ARxnjPQHpnk1+lkAxEK+hyPMK2JUvbJXR3YOtKpG8h9FFFe+dgUUUUAFFFFABXzT/wUY8Qz+FNL8GalbZ8+x1Rpk99qZI/EcfjX0tXzr+394NTxzpvhmxku1slWS8ufMKk58u2d8cZ9K8vOub6pPlOfFfw3Y+Sm+KVq114mKaSYrfXoVtoYYZ9q2UayK6qCUJcALjBxVL4k+ObHx9rl9qn9m3FnfXTxlC1ysqRIsexsjywWPCkHjGfbm5L8ILiG3smfVtIU31sLuNN7eY8R3fMBj1QjHXJHFXv+FJ6hYrfafIulm5XULG1S5eVlaMzhmTA/usOpOMYr80lCtL3Wj5/llazRj6J45tF8AyeHdSs7ia0N8L+KW3nVZIn2FCCGUggjHvkVp+F/jIvhbUbiVLOd7eaKO3+yNMskM8KqV2SqUIY5O4MANp6daxPG3gC68GWFrcSXdreRXE01tugctslhKiRSCB/eU+nNdboPwo02w1yC2urmDUV1Dw9Lqa7mZFtX2OVPGM42j8jRT9teyWwRcjMs/jJb2sluy6TNth0CfQ+L1f8AlqHHmj92cbRIfkOc4HIrqfBvxCsdc0eaWSGO3W1trGzS2l1i3jdjbhmE6iaPYSCBgAZGcZ5zXIxfBLUJdVEH2qza3KwNFdBj5MnnKWTkjI4Bznpisfw78PrrxHq+o2cctrDJpcE087SPhdsWS+CODjHb861jWrwdpIvmlezNTw545tPh58ZY/EEK3WsRafd/aQHmWKS4fkndIFYH5ickA55pNG+KKeF9B1Wz0awe0l1pkW6uJ5xO/lK4k8pAEQKCwXJwScCrU3wB1ZlmS0u9P1C6EUFwkFu5Z545WVEdeP7zKCCc89xVJvg/qUzW4tXt7wTXradIY24gnUbirZAwNoZt3TCt6VEZYiLskTGU09Dorv492F34uvtVOg3jLrUcw1S2k1EmNvNUhzCNn7r5mJGd3pWNo/xQ03w/puoWVvpN9HaX13aXIf8AtLZNCsDMxw/lglm3H5hjbgYBxWrq/wAJLbUfAHhe60s2jNcJdy394u7ZIqSxqhIPOctjAB6iqfiv4dw+E/hZPJcRQvq9rrP2VrmGVmWSMwBwBg7eSe4z61peulzMfNO92LqfxuXVdbOoT6VJNdPpE2mSXHmpFNctJuCzSFY8MyKwXplgoJOeBasvj6tj4gutUj0mRbnUxarfJ9p2xyCEoTs+T5C5jUnrjnHWvpn9j34A+DfG37PXh/U9U8P6bfXtz9o8yaSLc74uJVGT7AAfhXpv/DK/w9J/5FPR/wDvzXu0sjxdSnGaktdTuhg6klzXPz+8W/E238T+HbHTf7NnhSz1S61EuboOZPPMZK42DBXy+GyevSr83xrhm+NzeMP7JnCsd32L7Zkg+WI/9Zs9s/d7mvvL/hlX4e/9CppH/fij/hlb4ej/AJlTSP8AvwKuXDuMe80P6jV7o+ArX4n2q+CrXTZtPvPtemmQWk8V75SGN23bZUC/Pg55BXOeau658bFuvG1l4k03T7qy1aG7S+nEt4Zrd5V67E2rtVskkEn72Pr93/8ADK3w+z/yKmj/APfilP7K3w9/6FTSP+/FP/V/F/zIf1Gr3Pzt1bWtFuNXa8stIu7TzZBIYHvBJHCMg7VOwEj03Zx712g/aB09dWvrseHbhJdQku3k2agBuW4iMe1j5XO3JIxgc9DjNfb3/DK/w9x/yKej/wDfmkP7K/w9B/5FTR/xgpQ4dxcdVJai+o1Vs0fCVh8aI7DwzZaZ9jv5FsJYprac3gFxZMqMrGCQJuRWYhihyo2kYOciPXvjNDr8WpCbSf32pPaSSyRzJDvaDOWZVj27nzyRjHuck/eH/DK/w9x/yKuj/wDfgU4fsrfD4/8AMq6T+EOKv+wMb/Mh/UqvdHwbd/GiObxV4m1aHS5Fm8QMjxq92HW1ZZkk5+QeYDsAx8p9zU118Y9N/t++vrfRdSVdUtrqKeGTVTIsTzxtGTGTHgAbyfmDHgDIGc/dn/DKnw9/6FXSf+/NH/DKnw9z/wAirpH/AH5FL+wMa9HJB9Rq90fnX8PPFieBvF9rqktq14tssiiJZhCzBkZfvbW6Z6Y9s1tQfFsadd+Eri109lk8LK0eJLkSLdK0rSEEBBtzvI4J6Cvvr/hlX4ff9CrpP/fmgfsq/D7/AKFXSP8AvzUR4dxaVuZE/UKq6o+EtD+N1v4e0SbSbXSbp9LkW+KrJeDzUe5iEWd3l/dRVGAAMnJ9hBqXxht7/TriH+ybtWm0WHRw320fKI2VvMx5Y67R8vbnnmvvQ/sq/D3/AKFTR/8AvyKP+GVfh7/0Kej/APfmtJcP4tq3MivqNXuj8+/BfxNHhPwjqGlSWst5BqQlSaGR0Nu25AquFZCVkQjduUgngHgc7XxG+N9n8SJrW4uNDkgutPmQ2sy3asywDbmF/wB2A4ypIbgrkjkYA+6h+yp8Pc/8ipo//fmj/hlT4ff9Cpo//fmpXDuMUVFSQRwNW1ro+F/E/wAdbHxbBefaNEvoZmvZ7y1eDVCnl+bgtHLiMeYuVBGNp7Zp158ddOn0iCzh0G4tre31Sz1NY1vx5aNBGyFFXyvlDbic8nJOS3Wvub/hlX4e4/5FXSP+/Io/4ZU+Hv8A0Kmkf9+RT/sDG/zIPqNXuj4Nt/jHaw6hdXH9kXDC48QJrgQXirs2l/3WfL5++fmwPp3NXSfifY6P4yutXj0m53ahFew3MRvB0uAygxt5fylQ3cHPtX34P2VPh7/0Kuj/APfkUf8ADKvw9/6FTR/+/NL/AFexj1ckH1Gr3R8G2nxis49Ws2k0e4ksY9F/sWeFb7bJIo5EqSCMbGDYPQjI98BR8V9Nl8PSadNpOpMsM5uLKZNTKyoWQKyyts/eKcDgbcV94j9lX4fZ/wCRU0f/AL80H9lX4e5/5FPR/wDvzVf6v4t6cyF9Rqvqj85/Bvip/B/i211TyftCwsfNgLbRNGylXTdgnlWbnqOOtdBqXxnbV5dBkuNLhDaLcLITDIY/PhjcvDCODsEe91B5OCB2FffH/DKvw9/6FPR/+/NB/ZU+Hv8A0Kuj/wDfis48O4tK3MgWBqpbo+ErT43W1trui3zaTcSSaLqk+oxgXgUSCVw5jb92ejKAG9O1c/4s8bQ+JvDOl6dHYzW/9myXEhdp1k8zzn3kAbF249ec+1fob/wyp8Pv+hV0f/vxQP2VPh7/ANCpo/8A35qv9XsW1ZyQfUavdHwefjBpo8Z6br/9gXH9pW7QvdBL/bFO0SBAVGz5N2ASDuA5xUPg74wReDdLs7WPTJ5vsesf2qrNdAbhs2eXjZ6ZO7kZ/h6k/ex/ZV+Hv/QqaP8A9+aQfsqfD/8A6FXR/wDvzQuH8YnpJAsDV7nwJrPxMsNZ8KW1j/Zd9b3VlFJawSpqG2Iws7OvmRhAHZd7DIKg8EjjBsXPxgtp/FOpaoulzxnUNITSxF9s3eSVgSHzN2wbuEB24HU8mvvT/hlb4ej/AJlPR/8AvwKX/hlb4e/9Cpo//fgU3w/jf50P6jV7o+GbP9oFdO8UXGrw6S7XGpS2st8n2kBZDAyt8h2HbuZQcncRyB1rmvFfj2HxF4O03SY7Ga3Om3t5dCV7gP5i3HlkqQEHI8peTnPPAr9Cj+yr8Pc/8ipo/wD35FH/AAyp8Pf+hV0j/vyKT4fxrVudC+o1e6Pg7SPjHY6X4R/sv+xbj95apBO0V/5ayMswl8wDyyVY4Ck5PAHTFZXxG+JbePL3T7j7N5dxYRmNrqQIZ7obyyGQqqhmUEDcRk7ec9v0G/4ZU+Hv/Qq6R/34FH/DKnw+/wChU0f/AL8ih8O41xS5kP6jV7nwze/HDT7vxs2urod99ouvOku4G1LdCZJY3RjGpj+VcuWwd3QDI5Jz7T4sWsEMMT6TcvGuhvokuLwBnDOziQHy/lwWxtwc46197f8ADKnw9/6FTSP+/Io/4ZU+Hv8A0Kmkf9+RTfD+NatzoPqNXuj4P1f4zQ6xqt9JJpU0dnqWjW+kTxC6DSDyREFlVtgHWIHaV7nnnjUufiNZa58MzJeKu7TNSsI7W0W9RLgwwW8yh2BG5huIywHVuOlfbh/ZV+Hv/QqaP/35oP7K3w/yMeFdI4/6Zf5/yKuOQYtN+8g+o1e6Phc/Ee6k0LxNqt42nzT+J7hpreBJ1aa3nyQZBGpyo8tmGT6+1cz4G8X2/hCS+ebT5byW6jRYZ45/Jlt2WRXypKtjIBU4AJGQCMmv0O/4ZV+H/wA3/FK6Sd3UeT1+tC/sq/D4f8yro/8A35rF8OYy6amhfUar3aPh5v2gLX/hKLTUl0K4BtdSvdR2G/X5jcqoK58rjbtHPOR+dVLb4z2K6VaJNotx9stNHm0USxXgVHifdtkKmMneu7scH2xX3Yf2Vfh6f+ZV0j/vzQf2VPh8f+ZU0f8A781pLh/GtfGg+o1bbo/PHw141t9C8OXthJYy3DXV3bXQlWcR7PJLnbtKn728jIIxjODXXL+0Bp0l1rD3Hh2eddW1ObUxm+XdbvJG6DH7kglS+4EjHA4HWvt4/sofDw/8yno//fmhf2Ufh6P+ZT0f/vzRHh7FxStNB9Rq90fDrfH+1/4S0asuizPLNaQ6deRz34YXVusex0/1YIZ9qnf8xUjgVxo8KSarI11YLZ2tpKzNFFPqEHmRrk/KcspP1Kg+1foqP2Uvh6P+ZV0n1H7ql/4ZW+H+P+RU0f8A79VNThzFT+KSF/Z9R7s+FZfjDY2+mzQz6PcHUm0X+w5Jor1fJZQQVlA8s5IAAIDYP167ngP4taX4n+L2g6hcW7aPL5qvfzXGqBbIlYypZI2ACknHBY9enNfZo/ZU+Hw/5lTR/wAIKU/sq/D8/wDMq6OB0I8mqjw/jFpzKxSwNRLc+CbD4rW+h+H5dDXT7ibTWtbm2Ja7Hm75XjYuGCldo8pRtA55Oead4g+MsWr6frEMOlz2smrJZhZPtqnyDAqrnAjG7dj1GK+9P+GVPh7/ANCpo55ycw0D9lX4egf8ippH4RUPh/FvTmQfUavdH502/jK7urmNdUur7ULDeDPbi58szLkHG4hsdMZwf8d5PitY6R4stdU0vR7i3EcUltNDcX6zrJA8fl+UmIkCAKW5wTkg9a++f+GVvh9/0Kuk/wDfr/69If2V/h/n/kVdJ/79f/XrOPDeMirKaJ/s+r3Phz4a+MtL8TeLvC+n3VrcW8Gj3VvHpwW5GfOa5DlpPl+cMSBgbcAd6/SG3/1VcPp/7MvgXS7+G6g8M6XHPbyLLG6x4ZGU5BHPY13SDaK+gyfL6mFi1Ud2zuwtCVNNMdRRRXtHUFFFFABRRRQAV84f8FC/FsngrSvCt8kMNyFuriKSKTO2RHgKMMggjhj0r6Pr5v8A+ChnhWXxrpfhPToJoYria7nMQkziRhFnaMA8nGB715edX+pz5TnxV/ZOx8p23xlNtfWUx0XTWSz0xtKSMvJ8yMWIYHdkOpYkEVeuv2gri9vJrhtI04NcXFhdPtlk5NoCsY+8eG3c+vbvjE0bwNNL4e1K73afJ9ns0unE3mCS3VphGCoXjJPXOeD2rR8bfDq40Cx16R7fSI/7Nu7WGQQvKWhMiuy7NxwVbBznngYxX5wp1lqeBFztcx/FfxAfxRo0VnJYw26299c3wdGfdum2blOSRgbBgYrStfjJNGbKRtLs5ZrTSn0cuXk/exsCoJAbAYBj0ql8G9Bs/E/xT8PabfQLNZahfw20yHK5VmAIz1Heuo8OfDnTviZJ4p0vT9N+w6to3mT2LRSM0dwiy+WYpNx6kMCGB4IqaMqk/fQ4qT3MofGdbnSmsb7QdLv7H7JbwLG8j/I8CMqShg33tpII6HjpzWF4U8byeGNR1K4Frb3LalazWjoxZVjWQYYrg9cV0eh/Aea88QaXa3WpWsEOoTXEDuikyQtFGXOUwOGA4YcHB7g1m2PwbvtR0tbpZ7SFJYlniE+5BLGz7N6kjByfmwOcc47UpLEt3sHKzRsf2gL7SLsXVrp1lDeJYW9hFKGkJiEDq6PyfvZUZ/8Ar1XT413em67HfWWn2dqq6idUmg5aKeUoyMCCchCrycAjAc81LN8DJLb7Yz65o+3Tb2PT7vAl/wBHldnUA/J0Ow8jNIvwYvdI8Q2Fu11pd3J/bI0ieImQLFcBhhHIAO1gcZX8aq+Ie6Bc25bg+PiWumRabH4b059Jjjmje0lnlk81ZWVyC27OQyAgg8dKw7r4ji78N/2T/Y9jHZf2iuoBEaTdwoTy+p+XaAM8GtGH4GX2o2yzx3mmRyXQvHgtl8zcxtuZVGQenONxOfWs3xF8JNQ8M+HZr67a3ja3aNLi3bcso8wbl2gj5uOu3OKmpKu04taCcpnvfwI/bz0H4RfCzTdBuNB1B5LJ52PkyL5YDzu4ALHPAcDn0rsP+HnXhrGf+Ed1j/v5H/jXzr8L/hfpt34i8DyX0sN/D4kuJVnsyHQRqjbOoxyDz171kr8D9Qltba+guLW4025tzcG4jDssQ83ythGN2d/HQ/WvZo51jqdNRjsjrWMqxSR9Rf8ADzrw3jP/AAjutc9PnTn170D/AIKdeHG6eHtZ/wC/kQ/rXy34S8Ax2Xxx0fw3qaWl5FJqcFpceTLujljeRFJVgfQ8H/69bHxH8G6H4O0DSbi8sRaanNqEjG1trhmjubEHCyMxLbWLqV+U5IycVrHP8dKPNdfcV9crbn0Yf+CnnhvOP+Ed1n/vuPn9aB/wU68N9vDutf8AfyP/ABr5U+IsGkyaJoV5Y6auk395C7XVrFK8iKAxCONxLAsAeMnpnuBSQ/CbU7ix0uZWjEmsFTbKwO11beSwbp8oU7hnI4rP/WLH3to/kT/aFW9j6s/4ed+G8/8AIu61/wB9x/40H/gp14bP/Mu61/33H/jXybH8PfNsTfR6lYyabHEjzTxsW8gvJ5aq64ypJyeR05rSuPgfqGmW2qS6hfadZrpcsUTlmZvOMyF4iu1TkMBweKFxBj3svwH9erH2d8D/ANtnRfjj48j0Gy0jUbO4kheYSTOm3C8noc17avSviT9ib4YX3gv49W1xdSQnbFc2ksQJWSOQKD0YDKnplcjII4r7aQ5r6zJsXVxFHnrbnpYWpKcOaQ6ijPNGa9g6QoooBzQAUUUUAFFFFABRRSFsUALRTd9OByKACiiigAooooAKKM80ZoAKKKKACiiigAoozRmgAoooBzQAUUUUAFFBOKKACiiigAooozzQAUUZozQAUUUZoAKKCcGjNFwCiiigAoopC2KAFopN1LQAUUUUAFFFFABRRRQAV8x/8FKdWm0HQfB19bSeVcWeptLG+cbWCAj+VfTlfPf7d+ht4ibwbaJNHbySXlztaSHzkyLdjyp+nXtXl5zf6pOxz4r+Ez47uPirc3eo+I7iSys2/wCEiXZMi5RYF3h8IAeB8oGPTFWtb+Mc2vrqgutNsWTWLi1uLlQXVSLcFQvXoQxzjmob/wCG9vafDi18SQ301xBdN9m8tYeYbjOSjexXkHvnHaovgnbwaj8X/C9vdRQ3FrdapbwSJIoZJFeQAjB9Qa/NOardRl1Pn4uWxQ8IeMG8G+OLHXLezt2ksboXUFuwby1IJZRnOcDjvWtbfGDUNEbUG0m3t9Jm1KdJ7mWMszygOJAmWJwu4AkDriq9ttj1HxVmG32JDIEBiXbERKg47cDI/Kugsvg5No2s2Mf2jb9ut7iS3uDbLNa3SJAZDscHHIDD+8pXpzTjGqvdj0ZSUlsZl58ZGl8b2+vWui6XZXUMzz3Cwh9ty7ghiQSdoILcL0yaiuviv9r0jT4ptF06S50v93b3R3+YIhIXWI8kEDJAbGQD+VjVvhNbWXhe9vYdUa4uLLTrTUmha32KY55FixuzncGkBxjkc0eL/g+vg/wzeXs2oB7mxe0UIsQ8u68+LzMxvuwVA4JHtVSlXV0w9/qV5Pi7NcDVlawtSusajHqMgDN8rxlmCjB6ZY/nU1x8a7q41T7b/Z9n5i63/bvBcqZuPl6/d4ya7X4JeFLPW/BGhSXFrpcyjWZhdpPAJJrq2jiSRoowBud/vbQCMk1xmi/Dix8cXGpXGn3E1lB514bWKRN3lJDH5yq7dMsCRtXJG3nAIJu1ZJSiwcZLVEkHx3vLSWzZdLs99mL0ICzc/ahh884yAeKy/G/xJ/4TuFWm0vToNREaRzXkQbzLjYAq5GcA4wDjGa2LD4Ixano8MlvqXmX9xo66skBtwE2mVo2Qtu6gjI45rpNK+DkGp+BNV0K3kjm1Sx1+O3uLh4PmhRLed5dpByw+RsDGcqPXhP6zPSWwe+9GcT4W+LVz4XPh3yrG1nk8NzvNbtIzAtv5KtyOjEkH/CrmjfHOfTfD9tpEukWN1pcNrJaywy7/AN+ryiUNuzkMrjhl7Zo0j4TW+uw6bdQ6oy2OoWt5NueEM0Elsm942GeQVKtkE/exgYq1d/Bq1tNLlvLXVZJJl0ldbgiltdqvH5mxlb5iFIP1yMc9qmmsRZIOWbOX0Dxq3h/x3Y67b2VoJbG4ju7e2VisSshDKOuSAQOprSi+Kn2jTFsb7SdP1GzhvmvbRZS+bfdy6KwOfLJwSp74wRznsfjn4etbHxbfaTpsmg/6cbS2hsYoGjuLdpI1YvkqFAJGPlY8PWZ4Y/Z+j8VajNDHrBj/ALP1U6VeloPmSTZIyuozyjGJxjg8Djng5asZci+Ye+nZHH+NvFlv4wuVki0my02TJeQwySMZTxgfOxwBjgDgZrY8MfGvUvB+nWtvp8NusdvLDOY5WaWOV4887CdqFtxViPvAc06T4X2A0qO8/toW9vfW8s1lLcRpF5xjxnK7ieWDAYz2zitK1+CtvpvizS7PUri6a01FLmSKeFFMFwscJkVo5AcMrEEEHBG3pUxjXTugjGdzEl+KUcMOpWsOj2NvpOqIBNZpI4G9X3q4cnOc59sZ9afrHxj1DXdI1CzuI4WbULi2nEwY77fyEZIwo6YCn9KveN7W2+Hfh/wxHp1rDI2rWH9pXF1NCJHm3TPGI+QQFVUwQOpJ5q5eaRZeNvB/h547e10r+0NWuo5GggLFF2xtt+UEnknHHG7sBmrl7ZXi3qN860Nj4VftSW3gf4tx+KLnw7bLNNFIt39jOyS7lcABzu4XuSBwTXtA/wCCoOi7M/8ACL6mVxnP2mMZFeAwfs9ST30Ecd81x9osoL4W8CKbwRSNIsjCPdl/LMfO0k4YYB5xT0D4RzxyeH737Syx6pdwRJcCFZrdJGDZRhn5WVhgqwGffivQw2ZY7Dx5IbXNYYitDSJ9F/8AD0LRgf8AkVdU4/6eU/w/Wgf8FRNHz/yKuqfX7Sn+FeC+C9Hs9B8JXmpYjk1SPxDDp2+eySeEIVckbGOAD1I/2QKztV+FEY1nUG1DVrOzaS6vEt5cJHHK8LkY2ZyoZgwAUHbgetdf9uY+2jX3Gv1yufRg/wCCoejn/mVdU/8AAlP8KQ/8FRNHUn/ildT47C5Qn+VfLPxF8H2fgnVI7KDUpr688uKWUeQERFlijkGCCSSAxHTHFdndeC9O1d7XQ7Wf7Da/2AmsSyPZK8u6OIyEK2d2WBOeewqY8QY53V19wljK7PdB/wAFQtHJ/wCRU1Q84GLlOf0oH/BUPR2HHhXVOxz9pTp+VfN9p8GY9Vvo7Oy1GTz7zT5tUsBLBtWWGPeQG+Y7W2Ru2OnTkVoap8DdL0Zb9ptem8vS57OKfbaAsVuEDKV+bkjPOce2aI59mNun3B9crn0B/wAPQ9H/AOhV1T/wJj/wpp/4Kh6MSP8AildTJPAH2qOvla58CPZ/EyTw3Ldwo8V61o08jiNThiAcnAGeBzwM8kV3Gn/B64PhDxBp72dxDcWNza3FxJLb4uLeLy5Xl2gZ3LxkbchgOOOaVLPswl1X3B9cr+R7j/w9C0bH/Iq6mR7XSeuPSlP/AAVB0cH/AJFXVM5xj7SnXt2714DbeAYfFvws0WS1WG2W2uL6We5ZFV3hjWPG4k46njn+KotC+BVp4o1yGy0/XIrg3kkcMLxw5WF3ieQrI2du5WXbtUseDgGqlnmYdLfcH1yufQR/4Kh6MD/yKuqf+BEf+FKP+CoWj4/5FXVP/AlP8K+ePCvwa0/X9O8NvNrU9td+JPNEES2u5IzHJtbc24cY9ia53xn4Lj8O6do15Z3TXlvrVs80ZeLy2VldkII3HupOc1Es+zFa6B9crn1V/wAPQtH/AOhV1THr9oT/AApP+Homjj/mVdU/G5T/AA9x+deAaR+z/bXviddEn16C11SO8FpNAEErDKFi6BTnaGUD5gp+YYzk4zbb4OrqV9pK6dfNcLq9jJd28TRpFNI6SMhiVS2C2QSBnJxxzxVrPsxavp9wfXK59Jf8PRNH/wChV1T/AMCU/wAKP+HoOjsf+RU1TI/6eo6+cL74MQ6JCzalqi6b5yXEloLlFRmELMm10YhgzFSAMHkjJFWdR+Cul6TLfLNrdw39mzWkcxS0HK3CBlK/NzjPOccYxmj+3Mxfb7h/W659C/8AD0PRuP8AildT5/6ek/wpP+HoWi8/8UpqnyjJH2hP8K8Vt/hVpp8LQaFqF8ljfL4ln0uO8itd7TsFCoHIIYLux0z16dapeIvhgbzwfoTMtraNpOn3MuoTIFRpGS+aAfeIBbJHUjgH6Vf9tZglfT7hfXK/ke8f8PQNG/6FXVPT/j5T39vak/4eg6N/0Kmqf+BKf4V8j+LdFt/DfiG4tbW8TULaIgJcRn5XBA4P+0M4IHcHGcZrt/DPwa03xPoGgsuoXNrdahY317O7QiRALdS2ByOw/WsIcQ499V9xP16ufQP/AA9B0Yn/AJFTUyfQXUdO/wCHoOjgf8irqmD0/wBJTn9K+atI+Flr4i0fULqw1KZoLa3u7qLzbfygyWyCQhjnG5hkfLuwQPWtjU/g7azwyXd9q6xQ2ljps37mwUF47lRsGA2Ny5HJ5rT+3cx8vuH9cxB75/w9C0f/AKFXVP8AwKT/AAoP/BULR/8AoVdT/wDAqP8Awr5q0P4a26/Gl/DGpTzNDb3MsLzRL8z7FZlOOODtFbWpfDux8W+E9Ha1uFs7y18OXeoIPswC3KwTXBYu2fvFEwMZ5HbrShnmYSWjX3DjjK7R74P+Cn+j5/5FPVP/AAJT/Cmn/gqFo3/Qq6p/4Ep/hXzJ4n+FsfhLQppLrU4YtUhht7j7GSMypMquAuDncqupOQBzwTVvwj8JLXxLo2lXUmqSWs2rC6ESfZiyxvbpvOTuHBGecGpXEGYOXLp9wfXMQfSB/wCCoWij/mVNU/8AAlP8KX/h6HowOP8AhFdT/wDAlP8ACvnDUfhBBp3h6DXF1GaXRprKG5En2fbNvkneHZt3EABo3Oc9Md+K0NO+DP8Awh3i+zuNSuo59NXVLS2gdItwuhKFkVipxhfLYEjr7Vf9uZjezt9wfXMQe/f8PRNHwP8AildTxjP/AB9R8fpS/wDD0LRmGf8AhFNU/wDAlP8ACvnXxr8Nrew1PVby4vrfT/Ne7vLOJ9m2VIp5ECYzuy20gDGOBzVL4maFFc/GJrGNY7OO5NopCRFhGzwxk4VcknJJ4BNKWfY6Ora+4Prlc+lz/wAFQNH/AOhT1T/wJT/Cgf8ABUPRh/zKupY9ftUf+H4V89J8B4zqtlDJqbxQ32n3V8jNAS6+QXBRlDcbtvGeemcVDP8ACSHQ0t9Wa8luNHxZOjfZ/wB473G4rGUyRwI3744HrT/t3MfL7g+uYg+ix/wVD0dj/wAirqnTP/Hyn+FDf8FQdHA+bwnqox63CDH6V88SeGdP1b9qK50uaNhpcniGS32RLhfL89lAA6Y6VqX3gyP4gCzhW6tbK1j1G8sovL01EkXyY/M+d1ILfLjGT2pxz3HNuzX3D+uVz3If8FQtGx/yKmqdM/8AHyn+FH/D0LRs/wDIqap0J/4+kr5tf4QoohuoNQebSzop1qaV4dsiIJmhK7ckEmReDno1WvifoOnnx/4dtIF8uzvNNsDJ5cIRjvUbjtz97370nnmYJXbX3C+uV0rs+iT/AMFQtGA/5FXU+P8Ap5Tj9KP+HoOj7sf8IrqnP/T0nP6V40vwo0nxBZxaBb3L20reJbjTrW5a2DSSERgKrnP3d34+gNcTqfwyh0fw/cTXWpQQ6hHZxXyWrbcXCSYOxed27YQ3IAwCM8DKlnuYpX0+4X12ufTf/D0PR8f8irqgHqblPr6Uf8PQtHJx/wAIrqhb0+1R/wD6q8S+BXhaPxT8PmYR6R9qGvQQ+beRK3mQmJ3aIHBJJ2jA9frXN6frdjpXxS1BdH0ezksZ74iGHU4RIttbhyWV1bheOp6qAcEc1X9uY5QUrrXyL+t17XufSB/4KhaKOvhXVPXm5T/Cgf8ABUHRj/zKmqf+BKf4V8stodj4x+Jl1Z6Wy2OnXN1L9k8xwmEyxjTLYHzAADOPf1rS1D4Sr4dnLatd3el25uvsaNNanzI5BEJGLqMnaCwGRnIORkc1muIMw6NfcT9cr+R9KH/gqDowP/Iq6n/4Ep/hQP8AgqDo/wD0Kmqf+BCf4V81SfCGYWN9Gbr/AImWn6XFrD2/l/K0D7C209cqrhuccVp+IPgjZ6Dqd5CusSTLpkVvNeM0CRLGJ0jeMLl+T85znA+Xqar+3sxtfT7hfXMRY+6v2fPjva/tA+EJtYtbC40+OG4Nv5crhmYgKc5H1x+Fd+teCf8ABP8A0mLRPhhq9pBeQ38NvqsiR3EX3JV2phh9RXvg6V9tl9adXDxqT3aPXoScoKUtwooorsNgooooAKKKKACvnX/goFc6xpuleFb7RbSa6vLW8m5jtzNsDRFDkYP96voqo5YRMfmC4x6Vy4zDuvRlSTtczq0+eLifmTBN4yt9KuLFdEvGs7qBIHhbSi0eEbcCFKkBs55681nvoPihfES6pBoep2d1HMtxGbaweJY3U7lKgLgYYZ471+ogso8fdX8qPsUY/gX8q+XfCsnvU/A87+zfM/MbWrbxVra3Ibw9cwNePvnkg0p42m5zzx0zzgHBwPSrtprPjexkjaDRbyFYzI4RdKOwtJF5TttK4B2cZGM9a/S37HH/AM80/KkNlH/zzT8qqPC8lr7T8B/2c+kj8z5L3xld2U1rNol99nurKHTpUTSiD5ETq6AHZwQUU565UVtfEp7jxTYSw2vh3UJeI0td+iGCa1VMAAurHd8q4+6AcnpX6LCzjJ/1a/lS/Y4/7q/lVf6tTtZ1PwK+oO1mz8x9Pm8aaRoOn6daaZqltDpl59tt5I9PYTRTYALh9uc4Udcj8udNfF/jpJfMXRJFbzp5126MPkedNkuPk4DLwQP0r9JvscePur+VNNnGT9xfypf6sT29p+BP9nPfmPzTstc8cadDDHDpN9GINP8A7LQrprBlg3F9udvXcc57Zq+/xC+IX2hpotLube4kuY7ySaLSSjyzIrIHYgckqzA9mzyK/R37Gg/5Zr+VKLOPH3F/Kn/q1Usl7UP7PfWR+ZlrfeMrC9jmt9FvIVhimhSJdNYRqsylZTjH3mBxk9gPQCpoNc8bW9usS6PdeWth/ZmP7K6wby+0/Jz8xJr9Lfskf9xaX7HH/dX8qUeGai/5e/gH9n+Z+YPiuHxh4z8Sf2xe6VqX9ofu/wB5DYNFgoAqkALgEADnrwOla0Hi/wAfWustfQ6XeR3ElwbubbpbBbibDL5jjHJ+djjoCx+p/Sf7JH/cWl+xx/3V/Kl/qvO/N7X8A/s3zPzJtbjxha+Gf7J/sO5ltFdpITLpW+S2LY3eW23Kg9cDHSpl1nxtFf208WkX0H2WaS4SJNNbyjJIgSQlduDuUAY6V+mH2SP+6v5Un2RCfuL+VH+rE/8An7+Af2d5n5kmbxbPo8Nhc6DcXdrau72yzaWW+y7zlgh25Ck/w/d9ql0jV/G+inT/ACdKvv8AiWTSzwltNLHdIu18nbyCMAdgOgr9Mvscf91fypps4/7in8KX+q8783tPwD+z3vc/NmTxV44kewJ0ORTpYhFoyaQVa38pnZNpC9jI/B4IxnoKr2Os+N9Omjkg0m+ikWaO4dhpz4uJI87WcYwxyT1HNfpcbOP/AJ5r+VAs4/8Anmv5VT4ZqPeqH9nv+Y/MmCfxjb6XJZro155M1+NSYf2WcmcZw2dvTk8DA5q5deIvHF/ZXUNxo1xcLdXUl4HfSdz20kmPMaJipMZbCn5SOVB61+lX2OP/AJ5r+VKLOP8AuL+VEeGZr/l7+ALL5fzH5d+KdG8UeMdV+2XujaiZvLjjPl6e0YIRQi8BccKAPwrTj1TxpDeLcLo92sq6adK/5BZwbcpsKkbcZK5G7Ga/TD7HHn/Vr+VO+yoOir+VR/qrK9/a/gNZe/5j8yLa68aWmmW9rHpOoL9jt5LWCYae/nRwuWLRBsfcO9uO2fereoeI/HWqfbPO0m7b+0JLeS4/4lZ/eNAAsZ+7xgfnmv0s+zL/AHV/Kj7JGeqr+VV/qzP/AJ+/gH9nv+Y/L6/07xTqfiybWrjQ72S8uJWmlDac3luxznKbcHOT9K2LLxZ4+03y/s+n6hCLdoWixppYx+UGCDJXJUB24Oc7j61+lP2SP+6v5Un2SP8AuL+VEeF5q9qv4Cjl1up+a2leJ/HWitb/AGXR7iCO2knkWFNI/dMJgFkRlK4ZWAAwQQB9TUmkeMfHeh3MUlroskIt7kXVui6N8ts+3adg24GRjOQc9etfpL9kj/uLSGzj/wCea/lR/qzP/n7+A/7Pfc/NCz1fxtp76O0Oj3itoJkaz/4lX3DJy+fl5yagvm8Wajolvp82gXElvaQPbwZ0r5old97YO3IO7uOa/Tb7HH/zzX8qPscf/PNfyo/1Yn1qfgL+z3/Mfm7L428fS6vZ6i2kTNqVmABdto+Z5iq7VMj7cuQuep9KzfO8XNZ2dvJoNzLDYwNbwo+lkhEZ95/h67uQeor9NPscf/PNfypfscf9xar/AFZn/wA/PwD+z31kfmtqXinx5rOlSWd7pd7eI0jyI8+l+ZJb78bxGxXKA47Gor7XPG+oy3jTaReE6hJBLPjSiN7QcRYG3C4AGQOvNfpd9kj/ALi0fZI/7i0f6s1P+fgf2f5n5qyeJfHU880raTdmSa9OoAnSz+6nIwZF+XgkCjTPE3jrS0tY10m9mhtbaWz8qXTC6TwySeY6SAr82Xw2TyCARzX6V/ZI/wC6v5UfY4/7q/lR/q1U/wCfv4B/Z/mfmFdabr17peoQy+F52mvpI5FkTSvLNuF/hQqowpGBtGBx0Jq5oOs+NvDem21rbaPebbOKeGFpNLMjokw2yLuZc4IJ71+mP2OP+6v5UfZUHRV/Ks1wtJPSp+Af2b5n5n6DrHjTw7pMdnb6JceVGk8KmTSt7iOddsibiudpHbseadd+IfHV9ZSW8ukXbRzQW9u4/srlkt8eUM7e2Bz3r9Lvsy/3V/Kg2sZ/hX8q0/1an/z9/Ar+z2vtH5jGXxkfHbeI/wCyNRXVHmadnXT2VCzDByu3BGOKtW+ueNrUR+XolxH5dhPpgCaTtH2ebcZE+733Nz1Ga/S0Wcf/ADzX8qd9lQdFX8qmPC843tV/ASy9/wAx+ZPiC88ZeJ9FtrK90W6m+ywpbrcDSyty8aDCI0m3cVA4+go0O88ZeH7TToLfRbzy9LaZoA2l7sGVdr5+XnIwK/Tb7Mv91fyo+zL/AHV/KhcMTTv7T8B/2e9+Zn5nxa743j0OHTG0a6m0+G1+xi3k0olGQSmUZG3khySDwRnHSpNP8S+O7B7jdpd/eR3E8V15d1pplSOWIAIygrhSBxxgYAHav0s+zL/dX8qBbL/dX8qr/Vupe/tfwD+z3/Mz80dW17xtrukSWd5oc86s8jpM2k5nhEh3SKj7cqrEk4HqapXLeMLrxpD4gOkagupQNEyumnFVzGoVMrtxxtHHfH1Nfp39mT+6v5UG2T+6v5VL4Ym96v4C/s7+8z81n8VeOWkt2XRZ4/ssE9tGE0gKEjmz5i8Lzkkn2zTIPEfjaHTzZtolxNZtbw25gk0ncmIiTGcbfvLubB4POK/Sz7Mv91fyo+yxnqq/lVf6t1P+fn4D/s9/zH5ihvF6+P8A/hJl0W//ALW+1fbPM/s47fNzktt245NWtP1rxtpe3ydFvBsupbwbtLJHmyIUc/d7rxjtX6YfY4/7q/lR9kjH8K/lU/6rzW1T8Cf7PfSR+aFhr3jjTGt/J0i88u3sJNM8t9KLRy27sztG67cMNzMfUE+1UPENt4s8S6rZ3k+i38c1jHHDB5OnGNY0T7vAXBxgdfzr9Pvssf8AdX8qPs0Y/hX8qJcMzkrOr+BX1GWzmfmvF4v8eW+pW95HotxHcWt+dTiZdJ+7cEYL/d549aq6zqfjPX/D8OnXWh3UsdrH5EM/9lYuY4txYR+bt3bRk4GeOR3r9Mvsyf3V/Kg2kZ/hX8qf+rM2rOp+Af2e7Wcj8wbCPxhpfhltJt9H1GK1a6W93DT281ZlBVWD7cjAJ/E5rSOv+Mn1TUL2Tw4txcaooFy02iK/m45ycp1YgFj1OK/Ss2cf91fypRaR/wBxfypf6rztb2n4C/s/pc/LkaH4kXX5dR/4R+6aabeDH/ZZWEKylSoQKAFwcADpgelbcnibx1caZLZzaRdXVq+zy47jSjMtsUTy0aPcvynbwSK/Sv7JH/dX8qPsqD+FfyojwvNf8vPwEst6XPzHv7jxlqOnfZpNJ1Js2q2TTfYGE0sCkFY2bbkj5VH0GK0D4m8cnxBdahLo9xcPqFtHZ3MUulF4Zo41RYwUK4yoRcN14r9KBarn7q/lTWs0Y/dXpR/qzUt/E/Af9nu1uY8L/wCCf1tqEPwk1B9QtZrW4n1KRykkJhwNq9FPQfTiveVGBTI4RGPlwKkHFfTYOg6NKNNu9jvpw5IqIUUUV1GgUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAH/2Q==)_____no_output_____