# Notebook from bgalbraith/course-content
Path: tutorials/W1D1_ModelTypes/student/W1D1_Tutorial1.ipynb

_____no_output_____# Tutorial 1: "What" models
**Week 1, Day 1: Model Types**

**By Neuromatch Academy**

__Content creators:__ Matt Laporte, Byron Galbraith, Konrad Kording

__Content reviewers:__ Dalin Guo, Aishwarya Balwani, Madineh Sarvestani, Maryam Vaziri-Pashkam, Michael Waskom

We would like to acknowledge [Steinmetz _et al._ (2019)](https://www.nature.com/articles/s41586-019-1787-x) for sharing their data, a subset of which is used here.
_____no_output________
# Tutorial Objectives
This is tutorial 1 of a 3-part series on different flavors of models used to understand neural data. In this tutorial we will explore 'What' models, used to describe the data. To understand what our data looks like, we will visualize it in different ways. Then we will compare it to simple mathematical models. Specifically, we will:

- Load a dataset with spiking activity from hundreds of neurons and understand how it is organized
- Make plots to visualize characteristics of the spiking activity across the population
- Compute the distribution of "inter-spike intervals" (ISIs) for a single neuron
- Consider several formal models of this distribution's shape and fit them to the data "by hand"_____no_output_____

#@title Video 1: "What" Models
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='KgqR_jbjMQg', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video_____no_output_____

# Setup

_____no_output_____Python requires you to explictly "import" libraries before their functions are available to use. We will always specify our imports at the beginning of each notebook or script._____no_output_____

import numpy as np
import matplotlib.pyplot as plt_____no_output_____

Tutorial notebooks typically begin with several set-up steps that are hidden from view by default.

**Important:** Even though the code is hidden, you still need to run it so that the rest of the notebook can work properly. Step through each cell, either by pressing the play button in the upper-left-hand corner or with a keyboard shortcut (`Cmd-Return` on a Mac, `Ctrl-Enter` otherwise). A number will appear inside the brackets (e.g. `[3]`) to tell you that the cell was executed and what order that happened in.

If you are curious to see what is going on inside each cell, you can double click to expand. Once expanded, double-click the white space to the right of the editor to collapse again._____no_output_____

#@title Figure Settings
import ipywidgets as widgets #interactive display

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")_____no_output_____#@title Helper functions

#@markdown Most of the tutorials make use of helper functions
#@markdown to simplify the code that you need to write. They are defined here.

# Please don't edit these, or worry about understanding them now!

def restrict_spike_times(spike_times, interval):
  """Given a spike_time dataset, restrict to spikes within given interval.

  Args:
    spike_times (sequence of np.ndarray): List or array of arrays,
      each inner array has spike times for a single neuron.
    interval (tuple): Min, max time values; keep min <= t < max.

  Returns:
    np.ndarray: like `spike_times`, but only within `interval`
  """
  interval_spike_times = []
  for spikes in spike_times:
    interval_mask = (spikes >= interval[0]) & (spikes < interval[1])
    interval_spike_times.append(spikes[interval_mask])
  return np.array(interval_spike_times, object)_____no_output_____#@title Data retrieval
#@markdown This cell downloads the example dataset that we will use in this tutorial.
import io
import requests
r = requests.get('https://osf.io/sy5xt/download')
if r.status_code != 200:
  print('Failed to download data')
else:
  spike_times = np.load(io.BytesIO(r.content), allow_pickle=True)['spike_times']_____no_output_____

---

# Section 1: Exploring the Steinmetz dataset

In this tutorial we will explore the structure of a neuroscience dataset.

We consider a subset of data from a study of [Steinmetz _et al._ (2019)](https://www.nature.com/articles/s41586-019-1787-x). In this study, Neuropixels probes were implanted in the brains of mice. Electrical potentials were measured by hundreds of electrodes along the length of each probe. Each electrode's measurements captured local variations in the electric field due to nearby spiking neurons. A spike sorting algorithm was used to infer spike times and cluster spikes according to common origin: a single cluster of sorted spikes is causally attributed to a single neuron.

In particular, a single recording session of spike times and neuron assignments was loaded and assigned to `spike_times` in the preceding setup.

Typically a dataset comes with some information about its structure. However, this information may be incomplete. You might also apply some transformations or "pre-processing" to create a working representation of the data of interest, which might go partly undocumented depending on the circumstances. In any case it is important to be able to use the available tools to investigate unfamiliar aspects of a data structure.

Let's see what our data looks like..._____no_output_____## Section 1.1: Warming up with `spike_times`_____no_output_____What is the Python type of our variable?_____no_output_____

type(spike_times)_____no_output_____

You should see `numpy.ndarray`, which means that it's a normal NumPy array.

If you see an error message, it probably means that you did not execute the set-up cells at the top of the notebook. So go ahead and make sure to do that.

Once everything is running properly, we can ask the next question about the dataset: what's its shape?_____no_output_____

spike_times.shape_____no_output_____

There are 734 entries in one dimension, and no other dimensions. What is the Python type of the first entry, and what is *its* shape?_____no_output_____

idx = 0
print(
  type(spike_times[idx]),
  spike_times[idx].shape,
  sep="\n",
)_____no_output_____

It's also a NumPy array with a 1D shape! Why didn't this show up as a second dimension in the shape of `spike_times`? That is, why not `spike_times.shape == (734, 826)`?

To investigate, let's check another entry._____no_output_____

idx = 321
print(
  type(spike_times[idx]),
  spike_times[idx].shape,
  sep="\n",
)_____no_output_____

It's also a 1D NumPy array, but it has a different shape. Checking the NumPy types of the values in these arrays, and their first few elements, we see they are composed of floating point numbers (not another level of `np.ndarray`):_____no_output_____

i_neurons = [0, 321]
i_print = slice(0, 5)

for i in i_neurons:
  print(
    "Neuron {}:".format(i),
    spike_times[i].dtype,
    spike_times[i][i_print],
    "\n",
    sep="\n"
  )_____no_output_____

Note that this time we've checked the NumPy `dtype` rather than the Python variable type. These two arrays contain floating point numbers ("floats") with 32 bits of precision.

The basic picture is coming together:
- `spike_times` is 1D, its entries are NumPy arrays, and its length is the number of neurons (734): by indexing it, we select a subset of neurons.
- An array in `spike_times` is also 1D and corresponds to a single neuron; its entries are floating point numbers, and its length is the number of spikes attributed to that neuron. By indexing it, we select a subset of spike times for that neuron.

Visually, you can think of the data structure as looking something like this:

```
| . . . . . |
| . . . . . . . . |
| . . . |
| . . . . . . . |
```

Before moving on, we'll calculate and store the number of neurons in the dataset and the number of spikes per neuron:_____no_output_____

n_neurons = len(spike_times)
total_spikes_per_neuron = [len(spike_times_i) for spike_times_i in spike_times]

print(f"Number of neurons: {n_neurons}")
print(f"Number of spikes for first five neurons: {total_spikes_per_neuron[:5]}")_____no_output_____#@title Video 2: Exploring the dataset
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='oHwYWUI_o1U', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video_____no_output_____

## Section 1.2: Getting warmer: counting and plotting total spike counts

As we've seen, the number of spikes over the entire recording is variable between neurons. More generally, some neurons tend to spike more than others in a given period. Lets explore what the distribution of spiking looks like across all the neurons in the dataset._____no_output_____Are most neurons "loud" or "quiet", compared to the average? To see, we'll define bins of constant width in terms of total spikes and count the neurons that fall in each bin. This is known as a "histogram".

You can plot a histogram with the matplotlib function `plt.hist`. If you just need to compute it, you can use the numpy function `np.histogram` instead._____no_output_____

plt.hist(total_spikes_per_neuron, bins=50, histtype="stepfilled")
plt.xlabel("Total spikes per neuron")
plt.ylabel("Number of neurons");_____no_output_____

Let's see what percentage of neurons have a below-average spike count:_____no_output_____

mean_spike_count = np.mean(total_spikes_per_neuron)
frac_below_mean = (total_spikes_per_neuron < mean_spike_count).mean()
print(f"{frac_below_mean:2.1%} of neurons are below the mean")_____no_output_____

We can also see this by adding the average spike count to the histogram plot:_____no_output_____

plt.hist(total_spikes_per_neuron, bins=50, histtype="stepfilled")
plt.xlabel("Total spikes per neuron")
plt.ylabel("Number of neurons")
plt.axvline(mean_spike_count, color="orange", label="Mean neuron")
plt.legend();_____no_output_____

This shows that the majority of neurons are relatively "quiet" compared to the mean, while a small number of neurons are exceptionally "loud": they must have spiked more often to reach a large count.

### Exercise 1: Comparing mean and median neurons

If the mean neuron is more active than 68% of the population, what does that imply about the relationship between the mean neuron and the median neuron?

*Exercise objective:* Reproduce the plot above, but add the median neuron.
_____no_output_____

# To complete the exercise, fill in the missing parts (...) and uncomment the code

median_spike_count = ...  # Hint: Try the function np.median

# plt.hist(..., bins=50, histtype="stepfilled")
# plt.axvline(..., color="limegreen", label="Median neuron")
# plt.axvline(mean_spike_count, color="orange", label="Mean neuron")
# plt.xlabel("Total spikes per neuron")
# plt.ylabel("Number of neurons")
# plt.legend()_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D1_ModelTypes/solutions/W1D1_Tutorial1_Solution_b3411d5d.py)

*Example output:*

_____no_output_____
*Bonus:* The median is the 50th percentile. What about other percentiles? Can you show the interquartile range on the histogram?_____no_output_____---

# Section 2: Visualizing neuronal spiking activity_____no_output_____## Section 2.1: Getting a subset of the data

Now we'll visualize trains of spikes. Because the recordings are long, we will first define a short time interval and restrict the visualization to only the spikes in this interval. We defined a utility function, `restrict_spike_times`, to do this for you. If you call `help()` on the function, it will tell you a little bit about itself:_____no_output_____

help(restrict_spike_times)_____no_output_____t_interval = (5, 15)  # units are seconds after start of recording
interval_spike_times = restrict_spike_times(spike_times, t_interval)_____no_output_____

Is this a representative interval? What fraction of the total spikes fall in this interval?_____no_output_____

original_counts = sum([len(spikes) for spikes in spike_times])
interval_counts = sum([len(spikes) for spikes in interval_spike_times])
frac_interval_spikes = interval_counts / original_counts
print(f"{frac_interval_spikes:.2%} of the total spikes are in the interval")_____no_output_____

How does this compare to the ratio between the interval duration and the experiment duration? (What fraction of the total time is in this interval?)

We can approximate the experiment duration by taking the minimum and maximum spike time in the whole dataset. To do that, we "concatenate" all of the neurons into one array and then use `np.ptp` ("peak-to-peak") to get the difference between the maximum and minimum value:_____no_output_____

spike_times_flat = np.concatenate(spike_times)
experiment_duration = np.ptp(spike_times_flat)
interval_duration = t_interval[1] - t_interval[0]

frac_interval_time = interval_duration / experiment_duration
print(f"{frac_interval_time:.2%} of the total time is in the interval")_____no_output_____

These two values—the fraction of total spikes and the fraction of total time—are similar. This suggests the average spike rate of the neuronal population is not very different in this interval compared to the entire recording.

## Section 2.2: Plotting spike trains and rasters

Now that we have a representative subset, we're ready to plot the spikes, using the matplotlib `plt.eventplot` function. Let's look at a single neuron first:_____no_output_____

neuron_idx = 1
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);_____no_output_____

We can also plot multiple neurons. Here are three:_____no_output_____

neuron_idx = [1, 11, 51]
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);_____no_output_____

This makes a "raster" plot, where the spikes from each neuron appear in a different row.

Plotting a large number of neurons can give you a sense for the characteristics in the population. Let's show every 5th neuron that was recorded:_____no_output_____

neuron_idx = np.arange(0, len(spike_times), 5)
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);_____no_output_____

*Question*: How does the information in this plot relate to the histogram of total spike counts that you saw above?_____no_output_____

#@title Video 3: Visualizing activity
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='QGA5FCW7kkA', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video_____no_output_____

---

# Section 3: Inter-spike intervals and their distributions_____no_output_____Given the ordered arrays of spike times for each neuron in `spike_times`, which we've just visualized, what can we ask next?

Scientific questions are informed by existing models. So, what knowledge do we already have that can inform questions about this data?

We know that there are physical constraints on neuron spiking. Spiking costs energy, which the neuron's cellular machinery can only obtain at a finite rate. Therefore neurons should have a refractory period: they can only fire as quickly as their metabolic processes can support, and there is a minimum delay between consecutive spikes of the same neuron.

More generally, we can ask "how long does a neuron wait to spike again?" or "what is the longest a neuron will wait?" Can we transform spike times into something else, to address questions like these more directly?

We can consider the inter-spike times (or interspike intervals: ISIs). These are simply the time differences between consecutive spikes of the same neuron.

### Exercise 2: Plot the distribution of ISIs for a single neuron

*Exercise objective:* make a histogram, like we did for spike counts, to show the distribution of ISIs for one of the neurons in the dataset.

Do this in three steps:

1. Extract the spike times for one of the neurons
2. Compute the ISIs (the amount of time between spikes, or equivalently, the difference between adjacent spike times)
3. Plot a histogram with the array of individual ISIs_____no_output_____

def compute_single_neuron_isis(spike_times, neuron_idx):
  """Compute a vector of ISIs for a single neuron given spike times.

  Args:
    spike_times (list of 1D arrays): Spike time dataset, with the first
      dimension corresponding to different neurons.
    neuron_idx (int): Index of the unit to compute ISIs for.

  Returns:
    isis (1D array): Duration of time between each spike from one neuron.
  """
  #############################################################################
  # Students: Fill in missing code (...) and comment or remove the next line
  raise NotImplementedError("Exercise: compute single neuron ISIs")
  #############################################################################

  # Extract the spike times for the specified neuron
  single_neuron_spikes = ...

  # Compute the ISIs for this set of spikes
  # Hint: the function np.diff computes discrete differences along an array
  isis = ...

  return isis

# Uncomment the following lines when you are ready to test your function
# single_neuron_isis = compute_single_neuron_isis(spike_times, neuron_idx=283)
# plt.hist(single_neuron_isis, bins=50, histtype="stepfilled")
# plt.axvline(single_neuron_isis.mean(), color="orange", label="Mean ISI")
# plt.xlabel("ISI duration (s)")
# plt.ylabel("Number of spikes")
# plt.legend()_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D1_ModelTypes/solutions/W1D1_Tutorial1_Solution_4792dbfa.py)

*Example output:*

_____no_output_____---

In general, the shorter ISIs are predominant, with counts decreasing rapidly (and smoothly, more or less) with increasing ISI. However, counts also rapidly decrease to zero with _decreasing_ ISI, below the maximum of the distribution (8-11 ms). The absence of these very low ISIs agrees with the refractory period hypothesis: the neuron cannot fire quickly enough to populate this region of the ISI distribution.

Check the distributions of some other neurons. To resolve various features of the distributions, you might need to play with the value of `n_bins`. Using too few bins might smooth over interesting details, but if you use too many bins, the random variability will start to dominate.

You might also want to restrict the range to see the shape of the distribution when focusing on relatively short or long ISIs. *Hint:* `plt.hist` takes a `range` argument_____no_output_____---

# Section 4: What is the functional form of an ISI distribution?_____no_output_____

#@title Video 4: ISI distribution
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='DHhM80MOTe8', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video_____no_output_____

The ISI histograms seem to follow continuous, monotonically decreasing functions above their maxima. The function is clearly non-linear. Could it belong to a single family of functions?

To motivate the idea of using a mathematical function to explain physiological phenomena, let's define a few different function forms that we might expect the relationship to follow: exponential, inverse, and linear._____no_output_____

def exponential(xs, scale, rate, x0):
  """A simple parametrized exponential function, applied element-wise.

  Args:
    xs (np.ndarray or float): Input(s) to the function.
    scale (float): Linear scaling factor.
    rate (float): Exponential growth (positive) or decay (negative) rate.
    x0 (float): Horizontal offset.

  """
  ys = scale * np.exp(rate * (xs - x0))
  return ys

def inverse(xs, scale, x0):
  """A simple parametrized inverse function (`1/x`), applied element-wise.

  Args:
    xs (np.ndarray or float): Input(s) to the function.
    scale (float): Linear scaling factor.
    x0 (float): Horizontal offset.

  """
  ys = scale / (xs - x0)
  return ys

def linear(xs, slope, y0):
  """A simple linear function, applied element-wise.

  Args:
    xs (np.ndarray or float): Input(s) to the function.
    slope (float): Slope of the line.
    y0 (float): y-intercept of the line.

  """
  ys = slope * xs + y0
  return ys_____no_output_____

### Interactive Demo: ISI functions explorer

Here is an interactive demo where you can vary the parameters of these functions and see how well the resulting outputs correspond to the data. Adjust the parameters by moving the sliders and see how close you can get the lines to follow the falling curve of the histogram. This will give you a taste of what you're trying to do when you *fit a model* to data.

"Interactive demo" cells have hidden code that defines an interface where you can play with the parameters of some function using sliders. You don't need to worry about how the code works – but you do need to **run the cell** to enable the sliders.
_____no_output_____

#@title

#@markdown Be sure to run this cell to enable the demo
# Don't worry about understanding this code! It's to setup an interactive plot.
single_neuron_idx = 283
single_neuron_spikes = spike_times[single_neuron_idx]
single_neuron_isis = np.diff(single_neuron_spikes)

counts, edges = np.histogram(
  single_neuron_isis,
  bins=50,
  range=(0, single_neuron_isis.max())
)

functions = dict(
  exponential=exponential,
  inverse=inverse,
  linear=linear,
)

colors = dict(
  exponential="C1",
  inverse="C2",
  linear="C4",
)

@widgets.interact(
  exp_scale=widgets.FloatSlider(1000, min=0, max=20000, step=250),
  exp_rate=widgets.FloatSlider(-10, min=-200, max=50, step=1),
  exp_x0=widgets.FloatSlider(0.1, min=-0.5, max=0.5, step=0.005),
  inv_scale=widgets.FloatSlider(1000, min=0, max=3e2, step=10),
  inv_x0=widgets.FloatSlider(0, min=-0.2, max=0.2, step=0.01),
  lin_slope=widgets.FloatSlider(-1e5, min=-6e5, max=1e5, step=10000),
  lin_y0=widgets.FloatSlider(10000, min=0, max=4e4, step=1000),
)
def fit_plot(
  exp_scale=1000, exp_rate=-10, exp_x0=0.1,
  inv_scale=1000, inv_x0=0,
  lin_slope=-1e5, lin_y0=2000,
):
  """Helper function for plotting function fits with interactive sliders."""
  func_params = dict(
    exponential=(exp_scale, exp_rate, exp_x0),
    inverse=(inv_scale, inv_x0),
    linear=(lin_slope, lin_y0),
  )
  f, ax = plt.subplots()
  ax.fill_between(edges[:-1], counts, step="post", alpha=.5)
  xs = np.linspace(1e-10, edges.max())
  for name, function in functions.items():
    ys = function(xs, *func_params[name])
    ax.plot(xs, ys, lw=3, color=colors[name], label=name);
  ax.set(
      xlim=(edges.min(), edges.max()),
      ylim=(0, counts.max() * 1.1),
      xlabel="ISI (s)",
      ylabel="Number of spikes",
  )
  ax.legend()_____no_output_____#@title Video 5: Fitting models by hand
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='uW2HDk_4-wk', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video_____no_output_____

# Summary

In this tutorial, we loaded some neural data and poked at it to understand how the dataset is organized. Then we made some basic plots to visualize (1) the average level of activity across the population and (2) the distribution of ISIs for an individual neuron. In the very last bit, we started to think about using mathematical formalisms to understand or explain some physiological phenomenon. All of this only allowed us to understand "What" the data looks like.

This is the first step towards developing models that can tell us something about the brain. That's what we'll focus on in the next two tutorials._____no_output_____


# Notebook from RahatDubey/gpt-2
Path: Notebooks/.ipynb_checkpoints/prog-checkpoint.ipynb

<code>
!ls
=1.13.0			encode.py			    sample.py
accumulate.py		encoder.py			    samples
better.txt		LICENSE				    src
Chatbotter-Copy1.ipynb	load_dataset.py			    test.txt
Chatbotter.ipynb	memory_saving_gradients.py	    toastme2.npz
checkpoint		model.py			    ToastMeDF.ipynb
CONTRIBUTORS.md		models				    toastme.npz
DEVELOPERS.md		My First Project-01f0de489aed.json  toastme.txt
Dockerfile.cpu		__pycache__			    train-horovod.py
Dockerfile.gpu		README.md			    train.py
download_model.py	requirements.txt		    Untitled.ipynb
!pip3 install -r requirements.txt #use docker.gpu or venv if on local Requirement already satisfied: fire>=0.1.3 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.2.1)
Requirement already satisfied: regex==2017.4.5 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (2017.4.5)
Requirement already satisfied: requests==2.21.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (2.21.0)
Requirement already satisfied: tqdm==4.31.1 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (4.31.1)
Requirement already satisfied: toposort==1.5 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (1.5)
Requirement already satisfied: termcolor in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from fire>=0.1.3->-r requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from fire>=0.1.3->-r requirements.txt (line 1)) (1.14.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from requests==2.21.0->-r requirements.txt (line 3)) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from requests==2.21.0->-r requirements.txt (line 3)) (2019.9.11)
Requirement already satisfied: idna<2.9,>=2.5 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from requests==2.21.0->-r requirements.txt (line 3)) (2.6)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from requests==2.21.0->-r requirements.txt (line 3)) (1.23)
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
!python download_model.py 345M #go into encode.py and change default to 345 ALSO TRAIN.PY change to 345mFetching checkpoint: 1.00kit [00:00, 1.18Mit/s]
Fetching encoder.json: 1.04Mit [00:00, 56.4Mit/s]
Fetching hparams.json: 1.00kit [00:00, 1.75Mit/s]
Fetching model.ckpt.data-00000-of-00001: 498Mit [00:05, 93.4Mit/s]
Fetching model.ckpt.index: 6.00kit [00:00, 7.29Mit/s]
Fetching model.ckpt.meta: 472kit [00:00, 57.0Mit/s]
Fetching vocab.bpe: 457kit [00:00, 56.7Mit/s]
!export PYTHONIOENCODING=UTF-8_____no_output_____import sys
sys.path.append("/home/ec2-user/SageMaker/gpt-2/src")_____no_output_____import fire
import json
import os
import numpy as np
import tensorflow as tf

import model, sample, encoder_____no_output_____#import unconditional samples not needed here bc im making chatbot
import interactive_conditional_samples_____no_output_____!python encode.py test2.txt test2.npz
WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/__init__.py:1467: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.

Reading files
100%|█████████████████████████████████████████████| 1/1 [00:12<00:00, 12.85s/it]
Writing test2.npz
# learning rate default=0.00002
#default batch_size should be one

_____no_output_____!python train.py --dataset test2.npz --learning_rate 0.0005 --batch_size 2WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/__init__.py:1467: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.

WARNING:tensorflow:From /home/ec2-user/SageMaker/gpt-2/model.py:147: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /home/ec2-user/SageMaker/gpt-2/memory_saving_gradients.py:13: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From train.py:88: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From train.py:91: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-03-03 15:38:14.772993: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  AVX512F
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
2020-03-03 15:38:14.796996: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2999995000 Hz
2020-03-03 15:38:14.798575: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55fffd36ac70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-03-03 15:38:14.798592: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-03-03 15:38:14.800372: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
WARNING:tensorflow:From train.py:92: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /home/ec2-user/SageMaker/gpt-2/model.py:148: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /home/ec2-user/SageMaker/gpt-2/model.py:152: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

WARNING:tensorflow:From /home/ec2-user/SageMaker/gpt-2/model.py:36: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.

WARNING:tensorflow:From /home/ec2-user/SageMaker/gpt-2/model.py:166: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.

WARNING:tensorflow:From /home/ec2-user/SageMaker/gpt-2/sample.py:65: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /home/ec2-user/SageMaker/gpt-2/sample.py:16: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/ec2-user/SageMaker/gpt-2/sample.py:70: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.random.categorical` instead.
WARNING:tensorflow:From train.py:117: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

WARNING:tensorflow:From train.py:121: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From train.py:144: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From train.py:147: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.

WARNING:tensorflow:From train.py:149: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

WARNING:tensorflow:From train.py:152: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From train.py:156: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

Loading checkpoint models/117M/model.ckpt
Loading dataset...
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 12.56it/s]
dataset has 2816957 tokens
Training...
[1 | 18.88] loss=3.69 avg=3.69
[2 | 21.64] loss=4.63 avg=4.16
[3 | 24.38] loss=4.13 avg=4.15
[4 | 27.14] loss=3.83 avg=4.07
[5 | 29.88] loss=3.78 avg=4.01
[6 | 32.63] loss=3.98 avg=4.00
[7 | 35.39] loss=3.86 avg=3.98
[8 | 38.33] loss=3.66 avg=3.94
[9 | 41.23] loss=3.56 avg=3.90
[10 | 43.97] loss=3.67 avg=3.87
[11 | 46.71] loss=3.58 avg=3.85
[12 | 49.44] loss=3.57 avg=3.82
[13 | 52.19] loss=3.47 avg=3.79
[14 | 54.94] loss=3.47 avg=3.77
[15 | 57.68] loss=3.21 avg=3.73
[16 | 60.42] loss=3.29 avg=3.70
[17 | 63.15] loss=3.47 avg=3.69
[18 | 65.91] loss=3.19 avg=3.65
[19 | 68.67] loss=3.33 avg=3.64
[20 | 71.42] loss=3.49 avg=3.63
[21 | 74.17] loss=3.13 avg=3.60
[22 | 76.90] loss=3.30 avg=3.59
[23 | 79.64] loss=3.20 avg=3.57
[24 | 82.39] loss=3.20 avg=3.55
[25 | 85.13] loss=3.32 avg=3.54
[26 | 87.89] loss=3.41 avg=3.53
[27 | 90.63] loss=3.20 avg=3.52
[28 | 93.37] loss=3.44 avg=3.52
[29 | 96.12] loss=3.04 avg=3.50
[30 | 98.85] loss=3.26 avg=3.49
[31 | 101.60] loss=3.14 avg=3.48
[32 | 104.35] loss=3.27 avg=3.47
[33 | 107.10] loss=3.33 avg=3.46
[34 | 109.85] loss=3.41 avg=3.46
[35 | 112.59] loss=3.21 avg=3.45
[36 | 115.33] loss=3.09 avg=3.44
[37 | 118.07] loss=3.31 avg=3.44
[38 | 120.81] loss=3.17 avg=3.43
[39 | 123.55] loss=3.40 avg=3.43
[40 | 126.31] loss=3.33 avg=3.42
[41 | 129.06] loss=3.09 avg=3.41
[42 | 131.80] loss=3.33 avg=3.41
[43 | 134.54] loss=3.18 avg=3.41
[44 | 137.30] loss=3.09 avg=3.40
[45 | 140.05] loss=3.31 avg=3.39
[46 | 142.80] loss=2.87 avg=3.38
[47 | 145.55] loss=2.94 avg=3.37
[48 | 148.30] loss=3.11 avg=3.36
[49 | 151.04] loss=3.01 avg=3.35
[50 | 153.79] loss=3.21 avg=3.35
[51 | 156.54] loss=2.93 avg=3.34
[52 | 159.28] loss=3.06 avg=3.33
[53 | 162.03] loss=3.14 avg=3.33
[54 | 164.77] loss=3.15 avg=3.32
[55 | 167.52] loss=3.17 avg=3.32
[56 | 170.26] loss=3.00 avg=3.31
[57 | 173.01] loss=3.11 avg=3.31
[58 | 175.75] loss=3.37 avg=3.31
[59 | 178.49] loss=3.10 avg=3.30
[60 | 181.24] loss=2.80 avg=3.29
[61 | 183.98] loss=2.98 avg=3.29
[62 | 186.72] loss=2.97 avg=3.28
[63 | 189.46] loss=3.06 avg=3.27
[64 | 192.21] loss=3.17 avg=3.27
[65 | 194.97] loss=3.17 avg=3.27
[66 | 197.72] loss=5.26 avg=3.31
[67 | 200.47] loss=3.11 avg=3.31
[68 | 203.22] loss=2.88 avg=3.30
[69 | 205.97] loss=3.15 avg=3.30
[70 | 208.73] loss=3.30 avg=3.30
[71 | 211.48] loss=2.92 avg=3.29
[72 | 214.23] loss=3.06 avg=3.28
[73 | 217.14] loss=2.98 avg=3.28
[74 | 220.09] loss=3.11 avg=3.28
[75 | 222.84] loss=3.38 avg=3.28
[76 | 225.57] loss=2.75 avg=3.27
[77 | 228.32] loss=3.11 avg=3.26
[78 | 231.06] loss=2.87 avg=3.26
[79 | 233.81] loss=3.23 avg=3.26
[80 | 236.53] loss=3.10 avg=3.25
[81 | 239.26] loss=3.10 avg=3.25
[82 | 242.01] loss=2.97 avg=3.25
[83 | 244.75] loss=2.67 avg=3.24
[84 | 247.48] loss=3.17 avg=3.23
[85 | 250.23] loss=3.09 avg=3.23
[86 | 252.96] loss=3.14 avg=3.23
[87 | 255.71] loss=2.99 avg=3.23
[88 | 258.46] loss=3.20 avg=3.23
[89 | 261.20] loss=2.84 avg=3.22
[90 | 263.94] loss=2.75 avg=3.21
[91 | 266.68] loss=2.86 avg=3.21
[92 | 269.43] loss=2.79 avg=3.20
[93 | 272.17] loss=3.38 avg=3.20
[94 | 274.90] loss=2.74 avg=3.19
[95 | 277.63] loss=3.20 avg=3.19
[96 | 280.37] loss=3.18 avg=3.19
[97 | 283.10] loss=2.90 avg=3.19
[98 | 285.84] loss=2.88 avg=3.18
[99 | 288.58] loss=3.11 avg=3.18
Generating samples...
======== SAMPLE 1 ========
 I a bunch of fun.  Good luck, I'm sure you’re going to be a wonderful fellow." "There’s nothing better about an internet stranger than telling you to “go your own way�⣁.����‣�“‣�E?"
"Got picked on a lot recently and it seems like every girl likes my body. Could definitely use some positivity." "I’m always in the dark on my phone, my brain getting a little lost, and my brain telling me when I’ll not want to be alone."
"I love my hair and skin and I just want to toast me!" "Hey! I’m a fan of your hair!"
"I have a little toasty beard and could really use a pick me up. :)" "You're not one-dimensional. The other ones are just amazing! The rest of the world, I know, is pretty similar but I've put them in the past few months. All the other hair models out there seem like more efficient people."
"This is the hardest birthday i have ever had, so i guess it’s not gonna pass" "Happy Birthday!"
"19y and no makeup. Been feeling insecure about my looks because I don’t look like a boy but today I am one. I just want to enjoy this life more much!" "You look really beautiful to me. Also, I was only two years old and I had pretty big breasts for a while but that was it.

I can’t believe i am just sitting back and not smiling - I have no sense of fashion and my body does not match the beauty I’ve seen in several other photos. And with a little effort I can imagine how beautiful I’ve come at the age of 13.

You are beautiful and it’s such a huge accomplishment that is accomplished.

I love your teeth, lips and hair and I am so proud of you. Keep pushing, and I know your nose is an awesome smile. And a smile that suits me."
"I have no sense of style and my skin doesn’t hold up; sorry I'm not doing my duty, but I can't get off this. I'm in a depression episode and just feeling really ugly right now, please toast me." "Your lips, and skin are amazing on you! Maybe you can see what someone else says to you, too. "
"Life is rough around here!" "Been feeling a little down here all day."
"In my life, i'm not good at what I see, which is why i want to escape. I’m just so sorry I don't recognize myself. Please toast me." "Your face, eyes, and smile.  The best things you have to offer us are amazing!"
"Going through a rough patch with the biggest breakup in 5 hours. Don’t know if my life will be alright." "i think i'm pretty cute, and i'd appreciate a hug :)"
"(R/toastme, im a boy for this one)" "You're a cute chap.. I know you seem to be extremely handsome! "
"I'm so lonesome I can't take this anymore. A week away from my usual family and friends and not for long from the front door.... I just need some positive reinforcement...." "I'm sorry. I'm just so lonesome I can take this anymore. A week away from my usual family and friends and not for long from the front door.... I just need some positive reinforcement...."
"Please do my best as a boy. Being in the early 30s for my first date after college is hard because I'm mentally challenged and have low self esteem for sure.

i also feel a lot younger than I really are, so I'll be honest to say I do seem a bit older but we're not done for sure.

i don't give a fuck what you think im doing because I have low self esteem for sure but hey its gonna get better i promise. (You have your best wishes to )"
"M23, still in the middle of it but trying to lose weight. I'm in this shit hole of doubt, and this summer is finally starting to make me feel better... I guess. Toast me?" "You look so good.

But the last few months have really helped me my depression is pretty much gone and I am feeling healthier and I could do with some motivation to try new things."
"Got rejected for a job in the last few hours because I hate my face. Don't feel happy and my self esteem is at an all time low. Please toast me." "Lovely job! You're cute, and honestly you're a brilliant work ethic fan

#I copied the class out of src to here to modify it with visible ease, it should go back for clean code when finished
class GPT2:

  # extracted from the source code to generate some text based on a prior
    def __init__(
        self,
        model_name='117M',
        seed=None,
        nsamples=1,
        batch_size=1,
        length=None,
        temperature=1,
        top_k=0,
        raw_text="",
  ):
      """
      Interactively run the model
      :model_name=345M : String, which model to use. 345m needs GPU for sure
      :seed=None : Integer seed for random number generators, fix seed to reproduce
       results
      :nsamples=1 : Number of samples to return total
      :batch_size=1 : Number of batches (only affects speed/memory).  Must divide nsamples.
      :length=None : Number of tokens in generated text, if None (default), is
       determined by model hyperparameters
      :temperature=1 : Float value controlling randomness in boltzmann. 1 is best
       distribution. Lower temperature results in less random completions. As the
       temperature approaches zero, the model will become deterministic and
       repetitive. Higher temperature results in more random completions.
      :top_k=0 : Integer value controlling diversity. 1 means only 1 word is
       considered for each step (token), resulting in deterministic completions,
       while 40 means 40 words are considered at each step. 0 (default) is a
       special setting meaning no restrictions. 40 generally is a good value.
      """
    if batch_size is None:
        batch_size = 1
    assert nsamples % batch_size == 0

    self.nsamples = nsamples
    self.batch_size = batch_size

    self.enc = encoder.get_encoder(model_name)
    hparams = model.default_hparams()
    with open(os.path.join('models', model_name, 'hparams.json')) as f:
        hparams.override_from_dict(json.load(f))

    if length is None:
        length = hparams.n_ctx // 2
    elif length > hparams.n_ctx:
        raise ValueError("Can't get samples longer than window size: %s" % hparams.n_ctx)

    self.sess = tf.Session(graph=tf.Graph())
    self.sess.__enter__()

    self.context = tf.placeholder(tf.int32, [batch_size, None])
    np.random.seed(seed)
    tf.set_random_seed(seed)
    self.output = sample.sample_sequence(
      hparams=hparams, length=length,
      context=self.context,
      batch_size=batch_size,
      temperature=temperature, top_k=top_k
    )

    saver = tf.train.Saver()
    self.ckpt = tf.train.latest_checkpoint(os.path.join('models', model_name))
    saver.restore(self.sess, self.ckpt)

    def close(self):
        self.sess.close()

    def generate_conditional(self,raw_text):
        context_tokens = self.enc.encode(raw_text)
        generated = 0
        for _ in range(self.nsamples // self.batch_size):
            out = self.sess.run(self.output, feed_dict={
              self.context: [context_tokens for _ in range(self.batch_size)]
            })[:, len(context_tokens):]
            for i in range(self.batch_size):
                generated += 1
                text = self.enc.decode(out[i])
                return text
              #print("=" * 40 + " SAMPLE " + str(generated) + " " + "=" * 40)
              #print(text)
        #print("=" * 80)
_____no_output_____gpt2 = GPT2()_____no_output_____#gpt2.close() #remember to close session after

#try updating deprecated tensorflow functions in model.py if necessary (or use Docker)_____no_output_____# result = gpt2.generate_conditional(raw_text="Hello bot, tell me about yourself") #chatbot logic should work if i put this inside class

# print(result)."

You ask if it is safe to question the codeword cherishing inside him.

"What gives girl you're here? Meet my old friend Krumkov? Someone from your past? Cause he screws the wrong people. Tell him, he should reach the Hotel talli Ive no doubt about it, it's nothing to kill for."

Penitent looks up at you, wide-eyed. "Galp..."

Search Warrant Pay

Darius Kurmov:

An Archaeologist and Sydney Esperang

There is major public activity on Piltover 27 in the form of an intense wave of actions. The simultaneous hijacking of all cargo ships, smuggling and reconnaissance operations linking Piltover with Germany and Tehran, is temporary and ineffectual. Fighters from Kazortas each declared for surgical mutiny, while engineers charged with bringing care into operation appear as helpless victims of pireface without fingers and who roam through vivid oxymorons left out of dinner talk. The City's AI is activated, except by Crossing Over. Inquirer Nolf refuses to utter the words of unimaginable grief to users, now gathered in their tiny spaces, in deserted pubs, parks and malls, Rettens and shopping centres - and is convinced the (very public) kidnap of the Athena overturned the timestream: hostilities have too savage a tone to hold the last two for more than partly emotional episode padding. Complains Lycan Thontrox 25, Ekkerothious informs visitor Ron Patvan - a veteran idealist agreed to return to a blood feud due to intractable debt - that they need urgently to update their infographics. This content is running.

Ecology Occult

What Can We Tell About Gods From The New Hideous Origins?

Ingrin Skarkov:<|endoftext|>Daylight Savings Time is a new time; for the revived Bitcoin community, perhaps the get-robot is finally emerging as a real thing that can play out between one service provider or another.

A coworker of mine recently sent me a small list of three the devilish capabilities of this solar phone installation, ignoring the fact that there are many devices out there improving on fixtures like these that, due to the insufficient surface area, are difficult to analyze.

Landline Cables

1. Outgoing Peer Time

Obviously, we are not having a heated argument over this. In fact, it's pretty much its state. People while entitled
class Who:
    """A class defining the conversation parties: me, friend"""
    def __init__(self):
        self.prefixes = []

    def matches(self,phrase):
        for prefix in self.prefixes:
            if phrase.startswith(prefix):
            #print(f"{phrase} starts with {prefix}")
                return True

    #print(f"{phrase} does not start with {self.prefixes}")
        return False

    def get_random_prefix(self):
        return self.prefixes[0] #just use first one

class Friend(Who):
    def __init__(self):
        super().__init__()  #initializes prefixes from who
        self.prefixes = ["Friend said: \""]

class You(Who):
    def __init__(self):
        super().__init__()
        self.prefixes = ["You said: \""]_____no_output_____class Chatter:

    def __init__(self, prior = None):
        if prior is None:
            prior="""
              You: "Hello, are you my friend?"
              Friend: "Of course!"
              """
        self.suggestion = None

        self.friend = Friend()
        self.you = You()
        self.parties  = [ self.friend, self.you ]

        self.conversation = []

        lines = prior.split("\n")
        for line in lines:
            line = line.strip()
            if len(line)!=0:
                party = None
                for party in self.parties:
                    if party.matches(line):
                        break
                if party is None:
                    raise Exception(f"Unknown party: {line}")
                self.conversation.append((party,line))
        self.get_suggestion()

    def get_prior(self):
        conv = ""
        for (party, line) in self.conversation:
            conv+=line+"\n"
        return conv

    def get_suggestion(self):
        who, last_line = self.conversation[-1]

        party_index = self.parties.index(who)
        next_party = self.parties[(party_index+1) % len(self.parties)]

        conv = self.get_prior()
        conv += next_party.get_random_prefix()
        answer = self.get_answer(next_party, conv)

        if not next_party.matches(answer):
            prefix = next_party.get_random_prefix()
            answer = prefix + answer

        self.suggestion = (next_party, answer)

    def next(self, party = None, answer = ""):
        """Continue the conversation
        :param party: None -> use the current party which is currently in turn
        :param answer: override the suggestion

        """
        suggested_party, suggested_answer = self.suggestion
        if party is None:
            party = suggested_party

        if answer == "":
            answer = suggested_answer

        if not party.matches(answer):
            prefix = party.get_random_prefix()
            answer = prefix + answer

        answer = answer.strip()

        if answer[-1] != "\"":
            answer += "\""
         # add the closing "

        self.conversation.append((party, answer))
        self.get_suggestion()

    def retry(self):
        self.get_suggestion()

    def get_answer(self, party, conv):
        answer = gpt2.generate_conditional(raw_text=conv)
        lines = answer.split("\n")
        line = ""
        for line in lines:
            if line !="":
                break

        if line!="":
            return line

        return ""

    def show(self):
        conv = ""
        for (party, line) in self.conversation:
            conv+=line+"\n"
        print(conv)
        if self.suggestion is not None:
            party, answer  = self.suggestion
            print(answer)
_____no_output_____c = Chatter()_____no_output_____c.show()You: "Hello, are you my friend?"
Friend: "Of course!"

--> Friend said: "Please trust me."
c.retry()
_____no_output_____c.next()_____no_output_____c.next(c.you, "My favorite video game is Overwatch")_____no_output_____c.show()You: "Hello, are you my friend?"
Friend: "Of course!"
Friend said: "Let's head back to the train station."
You said: "My favorite video game is Overwatch"

--> Friend said: "Oh, not you."
c.retry()_____no_output_____
</code>

c.next(c.you, "what is your favorite game?")
c.show()_____no_output_____

<code>
#gpt2.close() #remember to close session after_____no_output_____
</code>