# Notebook from imsaksham-c/Udacity-ComputerVision-Nanodegree
Path: Lessons&CourseWorks/3.ObjectTracking&Localization/3. Histogram_Filter/writeup.ipynb

# Two Dimensional Histogram Filter - Your First Feature (and your first bug).
Writing code is important. But a big part of being on a self driving car team is working with a **large** existing codebase. On high stakes engineering projects like a self driving car, you will probably have to earn the trust of your managers and coworkers before they'll let you make substantial changes to the code base.

A typical assignment for someone new to a team is to make progress on a backlog of bugs. So with that in mind, that's what you will be doing for your first project in the Nanodegree.

You'll go through this project in a few parts:

1. **Explore the Code** - don't worry about bugs at this point. The goal is to get a feel for how this code base is organized and what everything does.
2. **Implement a Feature** - write code that gets the robot moving correctly.
3. **Fix a Bug** - Implementing motion will reveal a bug which hadn't shown up before. Here you'll identify what the bug is and take steps to reproduce it. Then you'll identify the cause and fix it.
_____no_output_____## Part 1: Exploring the code
In this section you will just run some existing code to get a feel for what this localizer does.

You can navigate through this notebook using the arrow keys on your keyboard. You can run the code in a cell by pressing **`Ctrl + Enter`**

Navigate through the cells below. In each cell you should

1. Read through the code. It's okay to not understand everything at this point.
2. Make a guess about what will happen when you run the code.
3. Run the code and compare what you see with what you expected.
4. When you get to a **TODO** read the instructions carefully and complete the activity.

_____no_output_____

# Don't worry too much about this code for now...
from __future__ import division, print_function
%load_ext autoreload
%autoreload 2

# This code "imports" code from some of the other files we've written
# in this directory. Specifically simulate.py and helpers.py
import simulate as sim
import helpers
import localizer
from importlib import reload_____no_output_____# This code defines a 5x5 robot world as well as some other parameters
# which we will discuss later. It then creates a simulation and shows
# the initial beliefs.
R = 'r'
G = 'g'
grid = [[R,G,G,G,R],
        [G,G,R,G,R],
        [G,R,G,G,G],
        [R,R,G,R,G],
        [R,G,R,G,R]]

blur = 0.05
p_hit = 200.0
simulation = sim.Simulation(grid, blur, p_hit)
simulation.show_beliefs()_____no_output_____

Run the code below multiple times by repeatedly pressing Ctrl + Enter.

After each run observe how the state has changed. _____no_output_____

simulation.run(1)
simulation.show_beliefs()_____no_output_____

What do you think this call to `run` is doing? Look at the code in **`simulate.py`** to find out (remember - you can see other files in the current directory by clicking on the `jupyter` logo in the top left of this notebook).

Spend a few minutes looking at the `run` method and the methods it calls to get a sense for what's going on._____no_output_____#### What am I looking at?

The red star shows the robot's true position. The blue circles indicate the strength of the robot's belief that it is at any particular location.

Ideally we want the biggest blue circle to be at the same position as the red star._____no_output_____

# We will provide you with the function below to help you look
# at the raw numbers.

def show_rounded_beliefs(beliefs):
    for row in beliefs:
        for belief in row:
            print("{:0.3f}".format(belief), end="  ")
        print()

# The {:0.3f} notation is an example of "string
# formatting" in Python. You can learn more about string
# formatting at https://pyformat.info/_____no_output_____show_rounded_beliefs(simulation.beliefs)0.002  0.069  0.003  0.069  0.002
0.002  0.070  0.070  0.069  0.002
0.069  0.070  0.004  0.070  0.003
0.069  0.003  0.069  0.070  0.070
0.002  0.002  0.069  0.003  0.069

__________no_output_____## Part 2: Implement a 2D sense function.
As you can see, the robot's beliefs aren't changing. No matter how many times we call the simulation's sense method, nothing happens. The beliefs remain uniform._____no_output_____### Instructions
1. Open `localizer.py` and complete the `sense` function.
3. Run the code in the cell below to import the localizer module (or reload it) and then test your sense function.
4. If the test passes, you've successfully implemented your first feature! Keep going with the project. If your tests don't pass (they likely won't the first few times you test), keep making modifications to the `sense` function until they do!_____no_output_____

reload(localizer)
def test_sense():
    R = 'r'
    _ = 'g'

    simple_grid = [
        [_,_,_],
        [_,R,_],
        [_,_,_]
    ]

    p = 1.0 / 9
    initial_beliefs = [
        [p,p,p],
        [p,p,p],
        [p,p,p]
    ]

    observation = R

    expected_beliefs_after = [
        [1/11, 1/11, 1/11],
        [1/11, 3/11, 1/11],
        [1/11, 1/11, 1/11]
    ]

    p_hit  = 3.0
    p_miss = 1.0
    beliefs_after_sensing = localizer.sense(
        observation, simple_grid, initial_beliefs, p_hit, p_miss)

    if helpers.close_enough(beliefs_after_sensing, expected_beliefs_after):
        print("Tests pass! Your sense function is working as expected")
        return

    elif not isinstance(beliefs_after_sensing, list):
        print("Your sense function doesn't return a list!")
        return

    elif len(beliefs_after_sensing) != len(expected_beliefs_after):
        print("Dimensionality error! Incorrect height")
        return

    elif len(beliefs_after_sensing[0] ) != len(expected_beliefs_after[0]):
        print("Dimensionality Error! Incorrect width")
        return

    elif beliefs_after_sensing == initial_beliefs:
        print("Your code returns the initial beliefs.")
        return

    total_probability = 0.0
    for row in beliefs_after_sensing:
        for p in row:
            total_probability += p
    if abs(total_probability-1.0) > 0.001:

        print("Your beliefs appear to not be normalized")
        return

    print("Something isn't quite right with your sense function")

test_sense()Tests pass! Your sense function is working as expected

## Integration Testing
Before we call this "complete" we should perform an **integration test**. We've verified that the sense function works on it's own, but does the localizer work overall?

Let's perform an integration test. First you you should execute the code in the cell below to prepare the simulation environment._____no_output_____

from simulate import Simulation
import simulate as sim
import helpers
reload(localizer)
reload(sim)
reload(helpers)

R = 'r'
G = 'g'
grid = [
    [R,G,G,G,R,R,R],
    [G,G,R,G,R,G,R],
    [G,R,G,G,G,G,R],
    [R,R,G,R,G,G,G],
    [R,G,R,G,R,R,R],
    [G,R,R,R,G,R,G],
    [R,R,R,G,R,G,G],
]

# Use small value for blur. This parameter is used to represent
# the uncertainty in MOTION, not in sensing. We want this test
# to focus on sensing functionality
blur = 0.1
p_hit = 100.0
simulation = sim.Simulation(grid, blur, p_hit)_____no_output_____
# Use control+Enter to run this cell many times and observe how
# the robot's belief that it is in each cell (represented by the
# size of the corresponding circle) changes as the robot moves.
# The true position of the robot is given by the red star.

# Run this cell about 15-25 times and observe the results
for _ in range(25):
    simulation.run(1)
    simulation.show_beliefs()

# If everything is working correctly you should see the beliefs
# converge to a single large circle at the same position as the
# red star. Though, if your sense function is implemented correctly
# and this output is not converging as expected.. it may have to do
# with the `move` function bug; your next task!
#
# When you are satisfied that everything is working, continue
# to the next section_____no_output_____

## Part 3: Identify and Reproduce a Bug
Software has bugs. That's okay.

A user of your robot called tech support with a complaint

> "So I was using your robot in a square room and everything was fine. Then I tried loading in a map for a rectangular room and it drove around for a couple seconds and then suddenly stopped working. Fix it!"

Now we have to debug. We are going to use a systematic approach.

1. Reproduce the bug
2. Read (and understand) the error message (when one exists)
3. Write a test that triggers the bug.
4. Generate a hypothesis for the cause of the bug.
5. Try a solution. If it fixes the bug, great! If not, go back to step 4._____no_output_____### Step 1: Reproduce the bug
The user said that **rectangular environments** seem to be causing the bug.

The code below is the same as the code you were working with when you were doing integration testing of your new feature. See if you can modify it to reproduce the bug._____no_output_____

from simulate import Simulation
import simulate as sim
import helpers
reload(localizer)
reload(sim)
reload(helpers)

R = 'r'
G = 'g'

grid = [
    [R,G,G,G,R,R,R],
    [G,G,R,G,R,G,R],
    [G,R,G,G,G,G,R],
    [R,R,G,R,G,G,G],
]

blur = 0.001
p_hit = 100.0
simulation = sim.Simulation(grid, blur, p_hit)

# remember, the user said that the robot would sometimes drive around for a bit...
# It may take several calls to "simulation.run" to actually trigger the bug.
simulation.run(1)
simulation.show_beliefs()_____no_output_____simulation.run(1)_____no_output_____

### Step 2: Read and Understand the error message
If you triggered the bug, you should see an error message directly above this cell. The end of that message should say:

```
IndexError: list index out of range
```

And just above that you should see something like

```
path/to/your/directory/localizer.pyc in move(dy, dx, beliefs, blurring)
     38             new_i = (i + dy ) % width
     39             new_j = (j + dx ) % height
---> 40             new_G[int(new_i)][int(new_j)] = cell
     41     return blur(new_G, blurring)
```

This tells us that line 40 (in the move function) is causing an `IndexError` because "list index out of range".

If you aren't sure what this means, use Google!

Copy and paste `IndexError: list index out of range` into Google! When I do that, I see something like this:

![Search Results](http://i.imgur.com/gleBmBy.png)

Browse through the top links (often these will come from stack overflow) and read what people have said about this error until you are satisfied you understand how it's caused._____no_output_____### Step 3: Write a test that reproduces the bug
This will help you know when you've fixed it and help you make sure you never reintroduce it in the future. You might have to try many potential solutions, so it will be nice to have a single function to call to confirm whether or not the bug is fixed_____no_output_____

# According to the user, sometimes the robot actually does run "for a while"
# - How can you change the code so the robot runs "for a while"?
# - How many times do you need to call simulation.run() to consistently
#   reproduce the bug?
# Modify the code below so that when the function is called
# it consistently reproduces the bug.
def test_robot_works_in_rectangle_world():
    from simulate import Simulation
    import simulate as sim
    import helpers
    reload(localizer)
    reload(sim)
    reload(helpers)

    R = 'r'
    G = 'g'

    grid = [
        [R,G,G,G,R,R,R],
        [G,G,R,G,R,G,R],
        [G,R,G,G,G,G,R],
        [R,R,G,R,G,G,G],
    ]

    blur = 0.001
    p_hit = 100.0
    for i in range(1000):
        simulation = sim.Simulation(grid, blur, p_hit)
        simulation.run(1)

test_robot_works_in_rectangle_world()_____no_output_____

### Step 4: Generate a Hypothesis
In order to have a guess about what's causing the problem, it will be helpful to use some Python debuggin tools

The `pdb` module (`p`ython `d`e`b`ugger) will be helpful here!

#### Setting up the debugger
1. Open `localizer.py` and uncomment the line to the top that says `import pdb`
2. Just before the line of code that is causing the bug `new_G[int(new_i)][int(new_j)] = cell`, add a new line of code that says `pdb.set_trace()`
3. Run your test by calling your test function (run the cell below this one)
4. You should see a text entry box pop up! For now, type `c` into the box and hit enter to **c**ontinue program execution. Keep typing `c` and enter until the bug is triggered again_____no_output_____

test_robot_works_in_rectangle_world()_____no_output_____

#### Using the debugger
The debugger works by pausing program execution wherever you write `pdb.set_trace()` in your code. You also have access to any variables which are accessible from that point in your code.

Try running your test again. This time, when the text entry box shows up, type `new_i` and hit enter. You will see the value of the `new_i` variable show up in the debugger window. Play around with the debugger: find the values of `new_j`, `height`, and `width`. Do they seem reasonable / correct?

When you are done playing around, type `c` to continue program execution. Was the bug triggered? Keep playing until you have a guess about what is causing the bug._____no_output_____### Step 5: Write a Fix
You have a hypothesis about what's wrong. Now try to fix it. When you're done you should call your test function again. You may want to remove (or comment out) the line you added to `localizer.py` that says `pdb.set_trace()` so your test can run without you having to type `c` into the debugger box._____no_output_____

for _ in range(10):
    test_robot_works_in_rectangle_world()_____no_output_____

## Congratulations!
You've implemented your first feature and successfully debugged a problem the robot was having with rectangular environments. Well done._____no_output_____


# Notebook from DPDmancul/fruit-or-vegetable
Path: linear.ipynb

# Fruit or Vegatable?
From the point of view of biology and botanic a tomato is a fruit, but from the point of view of food it is commonly considered a vegetable.

So how to distinguish between culinary fruit and vegetables? We will train a simple linear binary classification with the aim to distinguish culinary fruit from vegatables taking in account the fact it is botanically a fruit, the fact that it grows on tree or not, the fact that it grows underground or not, and nutritional informations._____no_output_____

import numpy as np
import pandas as pd
from sklearn.linear_model import RidgeClassifier
import matplotlib.pyplot as plt_____no_output_____

Read input database (nutritional informations taken from ANSES-CIQUAL 2020, https://data.europa.eu/data/datasets/5369a15fa3a729239d2065b7)_____no_output_____

df = pd.read_csv('db.csv')_____no_output_____

Print size of dataframe and training set_____no_output_____

(df.shape[0], training_size := int(.7 * df.shape[0]))_____no_output_____df.head()_____no_output_____

Prepare data to training:_____no_output_____

while True:
  df = df.sample(frac=1) #shuffle

  X = df[df.columns[2:]].to_numpy() # discard name
  y = df[df.columns[1]].map(lambda t: t == 'V').to_numpy() # cast Vegatable/Fruit enum to boolean

  X_training, X_test = X[:training_size], X[training_size:]
  y_training, y_test = y[:training_size], y[training_size:]

  n_veg = y_training.sum();
  if abs(training_size - 2*n_veg)/(training_size) < 1/4: # shuffle until fruits and vegetables are balanced
    break_____no_output_____

## Test 1
Train with linear binary classification_____no_output_____

clf = RidgeClassifier().fit(X_training, y_training)
clf.score(X_training, y_training) # training error_____no_output_____clf.score(X_test, y_test) # test error_____no_output_____def plot_coef(clf, n=5): # plot most influent features
  coef = abs(clf.coef_[0])
  idx = coef >= sorted(coef)[-n]

  fig,ax = plt.subplots()
  ax.bar(df.columns[2:][idx], clf.coef_[0][idx])
  ax.set_ylabel('← Fruit | Vegetable →')
  fig.autofmt_xdate(rotation=45)
  plt.show()_____no_output_____plot_coef(clf) # plot most influent features_____no_output_____

It seems that usually if it grows on a tree it is considered a fruit (obviously there are some exceptions, e.g. watermelon), and if it grows underground it is a vegetable._____no_output_____## Test 2
Remove (put to zero) all features except nutritional values_____no_output_____

X_training[:,:4] = 0_____no_output_____clf = RidgeClassifier().fit(X_training, y_training)
clf.score(X_training, y_training) # training error_____no_output_____clf.score(X_test, y_test) # test error_____no_output_____plot_coef(clf) # plot most important features_____no_output_____

If it has manganese it could be a vegetable, otherwise with iron a fruit.

I expected that fructose was an important feature to identify fruit and I am very surprised to not see it here._____no_output_____


# Notebook from Herikc2/Pneumonia-Prediction
Path: pneumonia_prediction.py.ipynb

# Definition of the Business Problem:

What is pneumonia?

    Pneumonia is an inflammatory condition of the lung that mainly affects the small air sacs known as alveoli. Symptoms usually include some combination of a productive or dry cough, chest pain, fever, and difficulty breathing. The severity of the condition is variable. Pneumonia is usually caused by infection by viruses or bacteria and less commonly by other microorganisms, certain medications or conditions such as autoimmune diseases. Risk factors include cystic fibrosis, chronic obstructive pulmonary disease (COPD), asthma, diabetes, heart failure, poor ability to cough, as after a stroke and a weak immune system. The diagnosis is usually based on symptoms and physical examination. Chest radiography, blood and sputum culture can help confirm the diagnosis. The disease can be classified according to the place where it was acquired, such as community-acquired or hospital-acquired pneumonia or associated with healthcare.

The business problem is dealing with lives, where a wrong prediction could be costing someone else's life. The goal is bring the highest possible precision to the model, analyzing models tirelessly until we create the best model possible. The goal in costume, it is obviously 100%, lives are priceless. However, Deep Learning is math and mistakes can happen, with that in mind. A target of at least 97% accuracy is set, with an ideal target of 98.5%.

As input the model will receive an x-ray of the chest, where it will classify whether that person has pneumonia or not.

Suggestions for Improvement:

    1 - Use Transfer-Learning (Insufficient Memory)

    2 - Use Tuning to find the best parameters (Using already in hyperparameters)

    3 - Use of larger images (Images above 248 make tests impossible)

    4 - Apply a new split, as it has a large imbalance between the datasets (Applied, observed significant improvement)

    5 - Pre-apply image filters

    6 - Reduce batch size (Batch size of 16 generated a gain in accuracy, but an increase in Loss.)

Note: For items 1 and 3, a computer with high availability by ram memory is required._____no_output_____

# Import from libraries

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, Conv2D , MaxPool2D , Flatten , Dropout , BatchNormalization, MaxPooling2D
from keras.preprocessing.image import ImageDataGenerator
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report,confusion_matrix
from keras.callbacks import ReduceLROnPlateau
import cv2
import os
import type_models.utils as u_____no_output_____

# FUNCTIONS_____no_output_____

# Function for reading images and applying filters
# Opencv grayscale applied and resized to IMG_SIZE
def get_data(data_dir):
    data = []
    for label in labels:
        path = os.path.join(data_dir, label)
        class_num = labels.index(label)
        for img in os.listdir(path):
            # Grayscale and image reshape
            img_arr = cv2.imread(os.path.join(path, img), cv2.IMREAD_GRAYSCALE)

            resized_arr = cv2.resize(img_arr, (IMG_SIZE, IMG_SIZE)) # COMMENT FOR EXPLORATORY ANALYSIS

            data.append([resized_arr, class_num])
    return np.array(data)_____no_output_____# Add row to dataframe
def add_row(df, row):
    df.loc[-1] = row
    df.index = df.index + 1
    return df.sort_index()  _____no_output_____####### DEFINITION OF GLOBAL VARIABLES FOR ENVIRONMENTAL EXECUTION #######

labels = ['PNEUMONIA', 'NORMAL']
IMG_SIZE = 150
BATCH_SIZE = 16
INPUT_SHAPE = (IMG_SIZE, IMG_SIZE, 1)_____no_output_____

After checking the average size of the images, the following statement arrives:

    max_height = 2916

    max_width = 2713

    media_height = 1327

    media_width = 970

    min_height = 384

    min_width = 127

However, viewing through a histogram, it is stated that the mean is not representative,
thus, several tests were carried out, until the IMG_SIZE was reached between 128 and 250.
After analyzing some cases, 150 were adopted, since 250 were at risk of decreasing
accuracy without increasing the complexity of the model._____no_output_____

train = get_data('Dataset/chest_xray/train')
test = get_data('Dataset/chest_xray/test')
val = get_data('Dataset/chest_xray/val')

x_train = []
y_train = []

x_val = []
y_val = []

x_test = []
y_test = []_____no_output_____# The dataset is separated into vectors with the images and labels, according to its definition
for array_image, label in train:
    x_train.append(array_image)
    y_train.append(label)

for array_image, label in test:
    x_test.append(array_image)
    y_test.append(label)

for array_image, label in val:
    x_val.append(array_image)
    y_val.append(label)_____no_output_____

# EXPLORATORY ANALYSIS_____no_output_____Analyzing the graph below, it can be seen that the dataset is totally unbalanced, has many more images of Pneumonia, for better performance it is necessary apply a new distribution on the dataset._____no_output_____

l = []
for i in y_train:
    if(i == 0):
        l.append("Pneumonia")
    else:
        l.append("Normal")
sns.countplot(l)
plt.savefig('count_normal_pneumo.png', format='png')   _____no_output_____

For better analysis, all exploratory analysis coding was separated in the utils.py file, it is recommended to read the utils.py file to understand the results and what they mean.

Brief analysis:

The main analysis was performed on the measures of width and height, this evaluation was carried out,
in order to reach an IMG_SIZE with greater representativeness. It was evaluated by the Histogram,
and Quartiles where there should be a greater focus. The box plot served as an analysis, to identify,
in general the view of the dataset.

A big problem with the analysis is that the informed outputs brought results that were difficult to simulate,
arriving at times when an implementation would become unviable or exhaustive.

The suggestion for training more complex models is to rent Clusters from Google._____no_output_____

#x_total = x_train + x_val + x_test

# Performed the analysis for each split, after pre-processing can still be performed again for a new comparison

'''
u.main(x_train)
u.main(x_val)
u.main(x_test)
'''_____no_output_____

# PRE-PROCESSING_____no_output_____After analyzing the inconsistency in the dataset, it is analyzed that a new split can help in the performance. The performance had an average increase of 5% in the analyzes performed._____no_output_____

### NEW SPLIT ###

# Joining all the data in the old list format in a single list of total x and y
x_total = x_train + x_val + x_test
y_total = y_train + y_val + y_test

# Converting the total list to data frame and filling in
total_df = pd.DataFrame(columns = ['Img', 'Label'])

for i in range(len(x_total)):
    add_row(total_df, [x_total[i], y_total[i]])

# Separating into training and test data

x_train, x_test, y_train, y_test = train_test_split(total_df.drop('Label',axis=1),
                                                    total_df['Label'],
                                                    test_size = 0.3)

# Separating the data again between testing and validation
x_test, x_val, y_test, y_val = train_test_split(x_test, y_test, test_size=0.05)

# Dataframe to list conversion performed again
x_train = x_train['Img'].values.tolist()
x_val = x_val['Img'].values.tolist()
x_test = x_test['Img'].values.tolist()

y_train = y_train.values.tolist()
y_val = y_val.values.tolist()
y_test = y_test.values.tolist()_____no_output_____# Pixel normalization is performed for faster execution
x_train = np.array(x_train) / 255
x_val = np.array(x_val) / 255
x_test = np.array(x_test) / 255

# Resizing is applied over the array, to a format that Keras accepts
x_train = x_train.reshape(-1, IMG_SIZE, IMG_SIZE, 1)
y_train = np.array(y_train)

x_val = x_val.reshape(-1, IMG_SIZE, IMG_SIZE, 1)
y_val = np.array(y_val)

x_test = x_test.reshape(-1, IMG_SIZE, IMG_SIZE, 1)
y_test = np.array(y_test)_____no_output_____# Augmentation is used to have an outlier reduction and greater precision
datagen = ImageDataGenerator(rotation_range = 30, zoom_range = 0.2, width_shift_range = 0.1, height_shift_range = 0.1,
                             horizontal_flip = True)
datagen.fit(x_train)_____no_output_____

# The Convolutional Neural Network is created_____no_output_____The neural network below was created in a block structure:

Conv2D - Each block has a Dense layer, to recalculate the weights from filters, to each layer
more filters are applied, so we can have more precision in the pixels.

BatchNormalization - After the new calculations, a new normalization of the output values is performed,
so we maintain efficiency in calculations and improving accuracy.

MaxPooling2D - Used to prioritize the highest output values, as these have the greatest impact.

Dropout - To avoid the outlier, Dropout is applied where values that differ many of the others are discarded._____no_output_____

model = Sequential()

# First Block (Input)
model.add(Conv2D(32 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu' , input_shape = INPUT_SHAPE))
model.add(BatchNormalization())
model.add(MaxPooling2D((2,2) , strides = 2 , padding = 'same'))

# Second Block
model.add(Conv2D(64 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))
model.add(Dropout(0.1))
model.add(BatchNormalization())
model.add(MaxPooling2D((2,2) , strides = 2 , padding = 'same'))

# Third Block
model.add(Conv2D(64 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D((2,2) , strides = 2 , padding = 'same'))

# Fourth Block
model.add(Conv2D(128 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(MaxPooling2D((2,2) , strides = 2 , padding = 'same'))

# Fifth Blocks
model.add(Conv2D(256 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(MaxPooling2D((2,2) , strides = 2 , padding = 'same'))

# Sixth Block - Flatten is used to convert pixels from matrix to array
model.add(Flatten())
model.add(Dense(units = 128 , activation = 'relu'))
model.add(Dropout(0.2))

# Output
model.add(Dense(units = 1 , activation = 'sigmoid'))_____no_output_____# Build using RMSPROP or ADAM
model.compile(optimizer = "rmsprop" , loss = 'binary_crossentropy' , metrics = ['accuracy'])
model.summary()

# A callback is created to reduce the training rate and we don't fall in minimum places
learning_rate_reduction = ReduceLROnPlateau(monitor='val_accuracy', patience = 2, verbose=1, factor=0.3, min_lr = 0.000001)

# The model is trained
model_history = model.fit(datagen.flow(x_train,y_train, batch_size = BATCH_SIZE), epochs = 13,
                    validation_data = datagen.flow(x_val, y_val), callbacks = [learning_rate_reduction])_____no_output_____# The model is applied on the test basis to measure its accuracy
print("Loss: " , model.evaluate(x_test,y_test)[0])
print("Accuracy: " , model.evaluate(x_test,y_test)[1]*100 , "%")_____no_output_____# Analysis of the evolution of accuracy throughout the times
plt.plot(model_history.history['accuracy'])
plt.plot(model_history.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epochs')
plt.legend(['train', 'val'])
plt.show()_____no_output_____# Analysis of loss over time
plt.plot(model_history.history['loss'])
plt.plot(model_history.history['val_loss'])
plt.title('Model Loss')
plt.ylabel('Loss')
plt.xlabel('Epochs')
plt.legend(['train', 'val'])
plt.show()_____no_output_____# The prediction value is stored
predictions = model.predict_classes(x_test)
predictions = predictions.reshape(1,-1)[0]_____no_output_____

The data about the model is displayed:

Accuracy: The hit rate minus error, placed in %

Recall: The rate of hit frequency, how much do you hit each error?

f1-score: F1 is the combination of precision and recall in a single score.
_____no_output_____

print(classification_report(y_test, predictions, target_names = ['Pneumonia (Class 0)','Normal (Class 1)']))_____no_output_____# The precision matrix is created to visualize the number of correct cases
cm = confusion_matrix(y_test,predictions)
cm = pd.DataFrame(cm , index = ['0','1'] , columns = ['0','1'])

plt.figure(figsize = (10,10))
sns.heatmap(cm,cmap= "Blues", linecolor = 'black' , linewidth = 1 , annot = True, fmt='', xticklabels = labels, yticklabels = labels)_____no_output_____# The model is saved its shape and weights, for future use
model_json = model.to_json()
model_name = 'classifier'
with open("models/complexity_model_advanced/" + model_name + ".json", "w") as json_file:
    json_file.write(model_json)
model.save_weights("models/complexity_model_advanced/" + model_name + ".h5")_____no_output_____


# Notebook from LITdrive/aadc2018
Path: src/aadcUserPython/dynamic_track_detection.ipynb

<code>
import cv2
import numpy as np_____no_output_____parkourX0 = 0
parkourX1 = 0

parkourY0 = 0
parkourY1 = 0_____no_output_____img = cv2.imread('/home/mohamed/star.jpg',0)
ret,thresh = cv2.threshold(img,127,255,0)
contours,hierarchy, rest = cv2.findContours(thresh, 1, 2)
cnt = contours[0]
M = cv2.moments(cnt)
print M_____no_output_____x,y,w,h = cv2.boundingRect(cnt)
cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,0),2)_____no_output_____im = cv2.imread('/home/mohamed/star.jpg')
gray=cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)
contours,hierarchy, rest = cv2.findContours(gray,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)
idx =0
for cnt in contours:
    idx += 1
    x,y,w,h = cv2.boundingRect(cnt)
    roi=im[y:y+h,x:x+w]
    cv2.imwrite(str(idx) + '.jpg', roi)
    #cv2.rectangle(im,(x,y),(x+w,y+h),(200,0,0),2)
cv2.imshow('img',im)
cv2.waitKey(0)  _____no_output_____from __future__ import print_function
import cv2 as cv
import numpy as np
import argparse
import random as rng
rng.seed(12345)

input_src = "/home/mohamed/Downloads/scaledMap.png"
def thresh_callback(val):
    threshold = val

    canny_output = cv.Canny(src_gray, threshold, threshold * 2)
    print(canny_output)

    _, contours, _ = cv.findContours(canny_output, cv.RETR_TREE, cv.CHAIN_APPROX_SIMPLE)

    contours_poly = [None]*len(contours)
    boundRect = [None]*len(contours)
    centers = [None]*len(contours)
    radius = [None]*len(contours)
    for i, c in enumerate(contours):
        contours_poly[i] = cv.approxPolyDP(c, 3, True)
        boundRect[i] = cv.boundingRect(contours_poly[i])
        centers[i], radius[i] = cv.minEnclosingCircle(contours_poly[i])

    drawing = np.zeros((canny_output.shape[0], canny_output.shape[1], 3), dtype=np.uint8)

    for i in range(len(contours)):
        color = (rng.randint(0,256), rng.randint(0,256), rng.randint(0,256))
        cv.drawContours(drawing, contours_poly, i, color)
        cv.rectangle(drawing, (int(boundRect[i][0]), int(boundRect[i][1])), \
          (int(boundRect[i][0]+boundRect[i][2]), int(boundRect[i][1]+boundRect[i][3])), color, 2)
        cv.circle(drawing, (int(centers[i][0]), int(centers[i][1])), int(radius[i]), color, 2)

    cv.imshow('Contours', drawing)

src = cv.imread(input_src)
if src is None:
    print('Could not open or find the image:', input_src)
    exit(0)
# Convert image to gray and blur it
src_gray = cv.cvtColor(src, cv.COLOR_BGR2GRAY)
src_gray = cv.blur(src_gray, (3,3))
source_window = 'Source'
cv.namedWindow(source_window)
cv.imshow(source_window, src)
max_thresh = 255
thresh = 50 # initial threshold
cv.createTrackbar('Canny thresh:', source_window, thresh, max_thresh, thresh_callback)
thresh_callback(thresh)
cv.waitKey()_____no_output_____img = cv2.imread('/home/mohamed/road_small')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

ret,thresh = cv2.threshold(gray,127,255,1)

_, contours, _ = cv2.findContours(thresh,1,2)_____no_output_____import cv2
import numpy as np

image = cv2.imread("/home/mohamed/road_small")
copy = image.copy()

gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
cv2.imshow('Gray', gray)
cv2.waitKey(0)

edged = cv2.Canny(gray, 10, 250)
cv2.imshow('Edged', edged)
cv2.waitKey(0)

kernel = np.ones((5, 5), np.uint8)

dilation = cv2.dilate(edged, kernel, iterations=1)
cv2.imshow('Dilation', dilation)
cv2.waitKey(0)

closing = cv2.morphologyEx(dilation, cv2.MORPH_CLOSE, kernel)
cv2.imshow('Closing', closing)
cv2.waitKey(0)

(image, cnts, hiers) = cv2.findContours(closing, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

cont = cv2.drawContours(copy, cnts, -1, (0, 0, 0), 1, cv2.LINE_AA)
cv2.imshow('Contours', cont)
cv2.waitKey(0)

mask = np.zeros(cont.shape[:2], dtype="uint8") * 255

# Draw the contours on the mask
cv2.drawContours(mask, cnts, -1, (255, 255, 255), -1)

# remove the contours from the image and show the resulting images
img = cv2.bitwise_and(cont, cont, mask=mask)
cv2.imshow("Mask", img)
cv2.waitKey(0)

for c in cnts:
    x, y, w, h = cv2.boundingRect(c)
    if w > 50 and h > 130:
        new_img = img[y:y + h, x:x + w]

        cv2.imwrite('Cropped.png', new_img)
        cv2.imshow("Cropped", new_img)
        cv2.waitKey(0)_____no_output_____import cv2
import numpy as np

filename = '/home/mohamed/road_small'
img = cv2.imread(filename)
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

gray = np.float32(gray)
dst = cv2.cornerHarris(gray,10,3,0.04)

#result is dilated for marking the corners, not important
dst = cv2.dilate(dst,None)

# Threshold for an optimal value, it may vary depending on the image.
img[dst>0.01*dst.max()]=[0,0,255]

cv2.imshow('dst',img)
if cv2.waitKey(0) & 0xff == 27:
    cv2.destroyAllWindows()_____no_output_____import cv2
import numpy as np

image = cv2.imread("/home/mohamed/rollover_gold_00235.jpg")
copy = image.copy()

gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
blur = cv2.blur(gray, (3,3))
edged = cv2.Canny(blur, 10, 250)
(image, cnts, hiers) = cv2.findContours(edged, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

biggest_cnt = -1
biggest_cnt_size = -1
for c in cnts:
    if (c.size > biggest_cnt_size):
        biggest_cnt_size = c.size
        biggest_cnt = c

hull = cv2.convexHull(biggest_cnt)
approx = cv2.approxPolyDP(hull,20,True)

cont = cv2.drawContours(copy, [approx], 0, (0,0,0), 3)
cv2.imshow('Contours', cont)
cv2.waitKey(0)
approx_____no_output_____from __future__ import division

height_tiles = 5
width_tiles = 4

import numpy as np
import cv2
img = cv2.imread('/home/mohamed/Downloads/scaledMap.png')
height, width, _ = img.shape

test_input = (533, 200)

test_output = (test_input[0] * height_tiles / height, test_input[1] * width_tiles / width)
test_output_____no_output_____img = cv2.imread('/home/mohamed/Downloads/scaledMap.png')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
edges = cv2.Canny(gray,50,150,apertureSize = 3)

lines = cv2.HoughLines(edges,1,np.pi/180,200)
for rho,theta in lines[0]:
    a = np.cos(theta)
    b = np.sin(theta)
    x0 = a*rho
    y0 = b*rho
    x1 = int(x0 + 1000*(-b))
    y1 = int(y0 + 1000*(a))
    x2 = int(x0 - 1000*(-b))
    y2 = int(y0 - 1000*(a))

    cv2.line(img,(x1,y1),(x2,y2),(0,0,255),2)

cv2.imwrite('houghlines3.jpg',img)_____no_output_____import numpy as np
import cv2

img = cv2.imread('/home/mohamed/rollover_gold_00235.jpg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
edges = cv2.Canny(gray,50,150,apertureSize = 3)
minLineLength = 10
maxLineGap = 20
lines = cv2.HoughLinesP(edges,1,np.pi/180,100,minLineLength,maxLineGap)
for onelines in lines:
    for x1,y1,x2,y2 in onelines:
        cv2.line(img,(x1,y1),(x2,y2),(0,255,0),2)

cv2.imwrite('houghlines5.jpg',img)_____no_output_____
</code>


# Notebook from SystemsBiologyUniandes/PyEcoLib
Path: development/examples/SizeStatistics/.ipynb_checkpoints/Pop_Size-gene-checkpoint.ipynb

<code>
import sys, os
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from scipy.stats import bayes_mvs as bayesest
import time

sys.path.insert(0, '../../PyEcoLib')
from PopSimulator import PopSimulator

%matplotlib inline_____no_output_____meanbirthsize = 1 # micron
doubling_time = 18 #min
tmax = 6*doubling_time #min
sample_time = 2 #min
div_steps = 10
ncells = 200_____no_output_____gr = np.log(2)/doubling_time _____no_output_____if not os.path.exists('./data'):
    os.makedirs('./data') #data path
if not os.path.exists('./figures'):
    os.makedirs('./figures') #Figures path_____no_output_____v0=meanbirthsize*np.ones(ncells)
sim = PopSimulator(ncells=ncells,gr = gr, sb=meanbirthsize, steps = div_steps,nu=2,V0array=v0) #Initializing the simulator_____no_output_____start = time.time()
sim.szdyn(tmax = 6*doubling_time, sample_time = 0.1*doubling_time, FileName=  "./data/data2Pop.csv", DivEventsFile="./data/DivEvents2.csv")
print('It took', np.int(time.time()-start), 'seconds.')It took 26 seconds.
data1=pd.read_csv("./data/data2Pop.csv")_____no_output_____data1=pd.read_csv("./data/data2Pop.csv")
smparr=data1.Sample.unique()
smparr.sort()
for smp in smparr:
    df=data1[data1.Sample==smp]
    tm,N=np.unique(df['Time'],return_counts=True)
    plt.plot(tm,N,c="#AAAAAA")
tm,N=np.unique(data1['Time'],return_counts=True)
plt.plot(tm,N/ncells,lw=3,c='k')
plt.yscale('log')_____no_output_____data1=pd.read_csv("./data/data2Pop.csv")

timearray=data1.Time.unique()

mnszarray=[]
cvszarray=[]
errcv2sz=[]
errmnsz=[]

for t in timearray:
    df=data1[data1.Time==t]
    szs=df.Size.values.tolist()
    mean_cntr, var_cntr, std_cntr = bayesest(szs,alpha=0.95)
    mnszarray.append(np.mean(szs))
    errmnsz.append(mean_cntr[1][1]-mean_cntr[0])
    cvszarray.append(np.var(szs)/np.mean(szs)**2)
    errv=(var_cntr[1][1]-var_cntr[0])/mean_cntr[0]**2+2*(mean_cntr[1][1]-mean_cntr[0])*var_cntr[0]/mean_cntr[0]**3
    errcv2sz.append(errv)
plt.plot(timearray/doubling_time,mnszarray)
plt.fill_between(np.array(timearray)/doubling_time,np.array(mnszarray)-np.array(errmnsz),np.array(mnszarray)+np.array(errmnsz),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0,label="SSA")C:\ProgramData\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:2023: RuntimeWarning: invalid value encountered in multiply
  lower_bound = _a * scale + loc
C:\ProgramData\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:2024: RuntimeWarning: invalid value encountered in multiply
  upper_bound = _b * scale + loc
data1=pd.read_csv("./data/data2Pop.csv")

timearray=data1.Time.unique()

mnsz2array=[]
cvszarray=[]
errcv2sz2arr=[]
errmnsz2arr=[]

for t in timearray:
    df=data1[data1.Time==t]
    mntemp=[]
    vartemp=[]
    smparray=df.Sample.unique()
    for s in smparray:
        dft=df[df.Sample==s]
        mntemp.append(np.mean(dft.Size.tolist()))
        vartemp.append(np.var(dft.Size.tolist())/np.mean(dft.Size.tolist())**2)

    mean_cntr, var_cntr, std_cntr = bayesest(mntemp,alpha=0.95)
    mnsz2array.append(np.mean(mntemp))
    errmnsz2arr.append(mean_cntr[1][1]-mean_cntr[0])
    cvszarray.append(np.var(szs)/np.mean(szs)**2)
    errv=(var_cntr[1][1]-var_cntr[0])/mean_cntr[0]**2+2*(mean_cntr[1][1]-mean_cntr[0])*var_cntr[0]/mean_cntr[0]**3
    errcv2sz2arr.append(errv)

plt.plot(timearray/doubling_time,mnszarray)
plt.fill_between(np.array(timearray)/doubling_time,np.array(mnszarray)-np.array(errmnsz),np.array(mnszarray)+np.array(errmnsz),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0,label="SSA")

plt.plot(timearray/doubling_time,mnsz2array)
plt.fill_between(np.array(timearray)/doubling_time,np.array(mnsz2array)-np.array(errmnsz2arr),np.array(mnsz2array)+np.array(errmnsz2arr),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0,label="SSA")C:\ProgramData\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:2023: RuntimeWarning: invalid value encountered in multiply
  lower_bound = _a * scale + loc
C:\ProgramData\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:2024: RuntimeWarning: invalid value encountered in multiply
  upper_bound = _b * scale + loc
data=pd.read_csv("./data/DivEvents2.csv")_____no_output_____mean_size=1.44*meanbirthsize
sampling_time = sample_time
rprom = 10/mean_size # RNA mean concentration (RNA/um)
pprom = 100/mean_size # prot mean concentration (RNA/um)
gammar = 5*gr # RNA Active degradation rate
kr = rprom*(gr+gammar) # RNA transcription rate
kp = pprom*gr/rprom # Protein translation rate_____no_output_____class Bacteria:
    def __init__(self, idx, V0, num_steps, gr, divpar, k,rna,prot):
        self.dp = divpar
        self.gr = gr
        self.idx = idx
        self.popidx = idx
        self.V = V0
        self.Vb = V0
        self.Vd = V0
        self.ndiv = 0
        self.rna = rna
        self.prot = prot
        self.k = k
        self.nextrb=0
        self.nextrd=0
        self.nextp=0
    def CalcTimeEvent(self,Nr,Np,s):#estimating reaction times from propensity function
        self.nextrb=(1/gr)*np.log(1-(gr/(kr*s))*np.log(np.random.rand())) #time to thenext rna creation
        self.nextrd=-np.log(np.random.rand())/(gammar*Nr)
        self.nextp=-np.log(np.random.rand())/(kp*Nr)_____no_output_____mnR=10
mnP=100
pop = []
df= data[data.BirthTime==0]
for m in range(len(df)):
    gr=df.iloc[m].GrowthRate
    sz=df.iloc[m].Sb
    dp=df.iloc[m].DivPar
    rna=mnR
    prot=mnP
    bct = Bacteria(idx=m, V0=sz, num_steps=div_steps, gr=gr, divpar=dp, k = gr,rna=rna,prot=prot)
    bct.CalcTimeEvent(rna,prot,sz)
    pop.append(bct)

#-------------------------------------------------------------------------------

t=0
tmax=6*doubling_time
data=data[data.BirthTime>0]
m=0
m=0
sample_time=0.1*doubling_time
tref=sample_time
GeneFile=open("./data/PopGene.csv","w")
output="Time,Sample,Cell,Nr,Np,Size\n"

nextdiv=data.iloc[m].BirthTime
popidx=data.iloc[m].Sample
idx=data.iloc[m].Cell
idx2=data.iloc[m+1].Cell
times=pd.DataFrame([[popidx,idx,nextdiv,0]],columns=['Sample','Cell','time','Type'])
for cell in pop:
    output+="0.00,"+str(cell.popidx)+","+str(cell.idx)+','+str(cell.rna)+','+str(cell.prot)+','+str(cell.V)+"\n"
    times=pd.concat([times,pd.DataFrame([[cell.popidx,cell.idx,cell.nextrb,1],\
                            [cell.popidx,cell.idx,cell.nextrd,2],\
                            [cell.popidx,cell.idx,cell.nextp,3]],columns=['Sample','Cell','time','Type'])])
times.reset_index(drop=True)
GeneFile.write(output)_____no_output_____while m<=len(data)-2 and t>=0:
    #print(t)
    nextt=np.min(times.time.tolist())
    if tref<nextt:

        dt=tref-t
        output=''
        for cell in pop:
            cell.V=cell.V*np.exp(cell.gr*dt)
            output+=str(tref)+','+str(int(cell.popidx))+','+str(int(cell.idx))+','+str(cell.rna)+','+str(cell.prot)+','+str(cell.V)+'\n'
        GeneFile.write(output)
        t=tref
        tref+=sample_time
        print(t)
    else:
        dt=nextt-t
        for cell in pop:
            cell.V=cell.V*np.exp(cell.gr*dt)
        t=nextt
        loc=np.argmin(times.time.tolist())
        if times.iloc[loc].Type==0:

            df=data.iloc[m]
            idx=int(data.iloc[m].Mother)
            cell=pop[idx]
            cell.V=df.Sb
            Nr=cell.rna
            Np=cell.prot
            newnr=np.random.binomial(Nr,df.Sb/df.MotherSize)
            newnp=np.random.binomial(Np,df.Sb/df.MotherSize)
            cell.rna=newnr
            cell.prot=newnp
            cell.CalcTimeEvent(Nr-newnr,Np,cell.V)
            cell.gr=df.GrowthRate
            df=data.iloc[m+1]#The sister cell
            bct = Bacteria(idx=df.Cell, V0=df.Sb, num_steps=div_steps, gr=df.GrowthRate, divpar=df.DivPar, k = df.GrowthRate,\
                       rna=Nr-newnr,prot=Np-newnp)
            bct.popidx=cell.popidx
            bct.CalcTimeEvent(Nr-newnr,Np-newnp,cell.V)
            pop.append(bct)
            times=pd.concat([times,pd.DataFrame([[bct.popidx,bct.idx,bct.nextrb+t,1],\
                            [bct.popidx,bct.idx,bct.nextrd+t,2],\
                            [bct.popidx,bct.idx,bct.nextp+t,3]],columns=['Sample','Cell','time','Type'])])
            times.reset_index(drop=True)
            times.iloc[0]=[data.iloc[m+2].Sample,data.iloc[m+2].Cell,data.iloc[m+2].BirthTime,0]
            m+=2
        else:
            idx=int(times.iloc[loc].Cell)
            cell=pop[idx]

            if times.iloc[loc].Type==1:
                cell.rna+=1

            elif times.iloc[loc].Type==2:
                cell.rna-=1
            else:
                cell.prot+=1
            cell.CalcTimeEvent(cell.rna,cell.prot,cell.V)
            if len(times[times.Cell==idx])==3:
                times.loc[times.Cell==idx,'time']=[cell.nextrb+t,cell.nextrd+t,cell.nextp+t]
            else:
                times.loc[times.Cell==idx,'time']=[times.iloc[0].time,cell.nextrb+t,cell.nextrd+t,cell.nextp+t]

GeneFile.close() 1.798544273543955
3.5906815352912473
5.3992491674974925
7.196983729743141
8.997405943773412
10.798393901033938
12.59942682651936
t_____no_output_____m_____no_output_____len(data)_____no_output_____data1=pd.read_csv("./data/PopGene.csv")

timearray=data1.Time.unique()

mnszarray=[]
cvszarray=[]
errcv2sz=[]
errmnsz=[]

for t in timearray:
    df=data1[data1.Time==t]
    szs=df.Np.values.tolist()
    mean_cntr, var_cntr, std_cntr = bayesest(szs,alpha=0.95)
    mnszarray.append(np.mean(szs))
    errmnsz.append(mean_cntr[1][1]-mean_cntr[0])
    cvszarray.append(np.var(szs)/np.mean(szs)**2)
    errv=(var_cntr[1][1]-var_cntr[0])/mean_cntr[0]**2+2*(mean_cntr[1][1]-mean_cntr[0])*var_cntr[0]/mean_cntr[0]**3
    errcv2sz.append(errv)
plt.plot(timearray/doubling_time,mnszarray)
plt.fill_between(np.array(timearray)/doubling_time,np.array(mnszarray)-np.array(errmnsz),np.array(mnszarray)+np.array(errmnsz),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0,label="SSA")_____no_output_____timearray=data1.Time.unique()

mnsz2array=[]
cvszarray=[]
errcv2sz2arr=[]
errmnsz2arr=[]

for t in timearray:
    df=data1[data1.Time==t]
    mntemp=[]
    vartemp=[]
    smparray=df.Sample.unique()
    for s in smparray:
        dft=df[df.Sample==s]
        mntemp.append(np.mean(dft.Np.tolist()))
        vartemp.append(np.var(dft.Np.tolist())/np.mean(dft.Np.tolist())**2)

    mean_cntr, var_cntr, std_cntr = bayesest(mntemp,alpha=0.95)
    mnsz2array.append(np.mean(mntemp))
    errmnsz2arr.append(mean_cntr[1][1]-mean_cntr[0])
    cvszarray.append(np.var(szs)/np.mean(szs)**2)
    errv=(var_cntr[1][1]-var_cntr[0])/mean_cntr[0]**2+2*(mean_cntr[1][1]-mean_cntr[0])*var_cntr[0]/mean_cntr[0]**3
    errcv2sz2arr.append(errv)

plt.plot(timearray/doubling_time,mnszarray)
plt.fill_between(np.array(timearray)/doubling_time,np.array(mnszarray)-np.array(errmnsz),np.array(mnszarray)+np.array(errmnsz),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0,label="SSA")

plt.plot(timearray/doubling_time,mnsz2array)
plt.fill_between(np.array(timearray)/doubling_time,np.array(mnsz2array)-np.array(errmnsz2arr),np.array(mnsz2array)+np.array(errmnsz2arr),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0,label="SSA")_____no_output_____meanbirthsize = 1 # micron
doubling_time = 18 #min
tmax = 6*doubling_time #min
sample_time = 2 #min
div_steps = 10
ncells = 1000_____no_output_____v0=meanbirthsize*np.ones(ncells)
sim = PopSimulator(ncells=ncells,gr = gr, sb=meanbirthsize, steps = div_steps,nu=1,V0array=v0) #Initializing the simulator_____no_output_____start = time.time()
sim.szdyn(tmax = 6*doubling_time, sample_time = 0.1*doubling_time, FileName=  "./data/dataPopMM.csv", DivEventsFile="./data/DivEventsMM.csv")
print('It took', np.int(time.time()-start), 'seconds.')_____no_output_____data=pd.read_csv("./data/DivEventsMM.csv")_____no_output_____mean_size=1.44*meanbirthsize
sampling_time = sample_time
rprom = 10/mean_size # RNA mean concentration (RNA/um)
pprom = 100/mean_size # prot mean concentration (RNA/um)
gammar = 5*gr # RNA Active degradation rate
kr = rprom*(gr+gammar) # RNA transcription rate
kp = pprom*gr/rprom # Protein translation rate_____no_output_____mnR=10
mnP=100
pop = []
df= data[data.BirthTime==0]
for m in range(len(df)):
    gr=df.iloc[m].GrowthRate
    sz=df.iloc[m].Sb
    dp=df.iloc[m].DivPar
    rna=mnR
    prot=mnP
    bct = Bacteria(idx=m, V0=sz, num_steps=div_steps, gr=gr, divpar=dp, k = gr,rna=rna,prot=prot)
    bct.CalcTimeEvent(rna,prot,sz)
    pop.append(bct)

#-------------------------------------------------------------------------------

t=0
tmax=6*doubling_time
data=data[data.BirthTime>0]
m=0
m=0
sample_time=0.1*doubling_time
tref=sample_time
GeneFile=open("./data/MMGene.csv","w")
output="Time,Sample,Cell,Nr,Np,Size\n"

nextdiv=data.iloc[m].BirthTime
popidx=data.iloc[m].Sample
idx=data.iloc[m].Cell
idx2=data.iloc[m+1].Cell
times=pd.DataFrame([[popidx,idx,nextdiv,0]],columns=['Sample','Cell','time','Type'])
for cell in pop:
    output+="0.00,"+str(cell.popidx)+","+str(cell.idx)+','+str(cell.rna)+','+str(cell.prot)+','+str(cell.V)+"\n"
    times=pd.concat([times,pd.DataFrame([[cell.popidx,cell.idx,cell.nextrb,1],\
                            [cell.popidx,cell.idx,cell.nextrd,2],\
                            [cell.popidx,cell.idx,cell.nextp,3]],columns=['Sample','Cell','time','Type'])])
times.reset_index(drop=True)
GeneFile.write(output)_____no_output_____while m<=len(data):
    #print(t)
    nextt=np.min(times.time.tolist())
    if tref<nextt:

        dt=tref-t
        output=''
        for cell in pop:
            cell.V=cell.V*np.exp(cell.gr*dt)
            output+=str(tref)+','+str(int(cell.popidx))+','+str(int(cell.idx))+','+str(cell.rna)+','+str(cell.prot)+','+str(cell.V)+'\n'
        GeneFile.write(output)
        t=tref
        tref+=sample_time
        print(t)
    else:
        dt=nextt-t
        for cell in pop:
            cell.V=cell.V*np.exp(cell.gr*dt)
        t=nextt
        loc=np.argmin(times.time.tolist())
        if times.iloc[loc].Type==0:

            df=data.iloc[m]
            idx=int(data.iloc[m].Mother)
            cell=pop[idx]
            cell.V=df.Sb
            Nr=cell.rna
            Np=cell.prot
            newnr=np.random.binomial(Nr,df.Sb/df.MotherSize)
            newnp=np.random.binomial(Np,df.Sb/df.MotherSize)
            cell.rna=newnr
            cell.prot=newnp
            cell.CalcTimeEvent(Nr-newnr,Np,cell.V)
            cell.gr=df.GrowthRate
            m+=1
        else:
            idx=int(times.iloc[loc].Cell)
            cell=pop[idx]

            if times.iloc[loc].Type==1:
                cell.rna+=1

            elif times.iloc[loc].Type==2:
                cell.rna-=1
            else:
                cell.prot+=1
            cell.CalcTimeEvent(cell.rna,cell.prot,cell.V)
            if len(times[times.Cell==idx])==3:
                times.loc[times.Cell==idx,'time']=[cell.nextrb+t,cell.nextrd+t,cell.nextp+t]
            else:
                times.loc[times.Cell==idx,'time']=[times.iloc[0].time,cell.nextrb+t,cell.nextrd+t,cell.nextp+t]

GeneFile.close() _____no_output_____data1=pd.read_csv("./data/MMGene.csv")

timearrayMM=data1.Time.unique()

mnszarrayMM=[]
cvszarrayMM=[]
errcv2szMM=[]
errmnszMM=[]

for t in timearray:
    df=data1[data1.Time==t]
    szs=df.Np.values.tolist()
    mean_cntr, var_cntr, std_cntr = bayesest(szs,alpha=0.95)
    mnszarrayMM.append(np.mean(szs))
    errmnszMM.append(mean_cntr[1][1]-mean_cntr[0])
    cvszarrayMM.append(np.var(szs)/np.mean(szs)**2)
    errv=(var_cntr[1][1]-var_cntr[0])/mean_cntr[0]**2+2*(mean_cntr[1][1]-mean_cntr[0])*var_cntr[0]/mean_cntr[0]**3
    errcv2szMM.append(errv)
plt.plot(timearrayMM/doubling_time,mnszarrayMM)
plt.fill_between(np.array(timearrayMM)/doubling_time,np.array(mnszarrayMM)-np.array(errmnszMM),np.array(mnszarrayMM)+np.array(errmnszMM),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0,label="SSA")
plt.plot(timearray/doubling_time,mnszarray)
plt.fill_between(np.array(timearray)/doubling_time,np.array(mnszarray)-np.array(errmnsz),np.array(mnszarray)+np.array(errmnsz),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0,label="SSA")

plt.plot(timearray/doubling_time,mnsz2array)
plt.fill_between(np.array(timearray)/doubling_time,np.array(mnsz2array)-np.array(errmnsz2arr),np.array(mnsz2array)+np.array(errmnsz2arr),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0,label="SSA")_____no_output_____data1=pd.read_csv("./data/data2Pop.csv")
smparr=data1.Sample.unique()
smparr.sort()
for smp in smparr:
    df=data1[data1.Sample==smp]
    var,freq=np.unique(df['Time'],return_counts=True)
    plt.plot(var,freq)
t=np.linspace(0,100,100)
plt.plot(t,2**(t/18),lw=4,c='k')

plt.yscale('log')_____no_output_____start = time.time()
sim.divstrat(tmax = tmax, sample_time = 0.1*doubling_time, nameDSM = "./data/dataDSM.csv") #Obtaining the simulation of Sd vs Sb
print('It took', np.int(time.time()-start), 'seconds.')_____no_output_____start = time.time()
sim.szdyn(tmax = tmax, sample_time= 0.1*doubling_time, nameCRM = "./data/dataCRM.csv") #Simulating the size for all the cells
print('It took', np.int(time.time()-start), 'seconds.')Cells initialized
1%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
It took 9 seconds.
start = time.time()
sim.szdynFSP(tmax = tmax, nameFSP = "./data/dataFSP.csv") #Obtaining trends using numerical FSP algorithm
print('It took', np.int(time.time()-start), 'seconds.')It took 115 seconds.
sbar=np.linspace(0.5,1.5,100)*mean_size
cv2sz=[]
deltsz=[]
for i in sbar:
    sd,cv2=sim.SdStat(i) #Obtaining trends in sd vs Sb using master equation formulation
    cv2sz.append(cv2)
    deltsz.append(sd-i)_____no_output_____data1=pd.read_csv("./data/dataCRM.csv")
timearray=data1.time.unique()

mnszarray=[]
cvszarray=[]
errcv2sz=[]
errmnsz=[]
df=data1
del df['time']
for m in range(len(df)):
    szs=df.loc[m, :].values.tolist()
    mean_cntr, var_cntr, std_cntr = bayesest(szs,alpha=0.95)
    mnszarray.append(np.mean(szs))
    errmnsz.append(mean_cntr[1][1]-mean_cntr[0])
    cvszarray.append(np.var(szs)/np.mean(szs)**2)
    errv=(var_cntr[1][1]-var_cntr[0])/mean_cntr[0]**2+2*(mean_cntr[1][1]-mean_cntr[0])*var_cntr[0]/mean_cntr[0]**3
    errcv2sz.append(errv)C:\ProgramData\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:1910: RuntimeWarning: invalid value encountered in multiply
  lower_bound = self.a * scale + loc
C:\ProgramData\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:1911: RuntimeWarning: invalid value encountered in multiply
  upper_bound = self.b * scale + loc
fig, ax = plt.subplots(1,2, figsize=(12,4))
data=pd.read_csv("./data/dataCRM.csv")
tt=data.time
del data['time']
for column in data.columns[0:10]:
    ax[0].plot(tt/doubling_time,data[column],c="#B9B9B9",label='_nolegend_')

ax[0].plot(np.array(timearray)/doubling_time,mnszarray)
ax[0].fill_between(np.array(timearray)/doubling_time,np.array(mnszarray)-np.array(errmnsz),np.array(mnszarray)+np.array(errmnsz),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0,label="SSA")

ax[1].plot(np.array(timearray)/doubling_time,cvszarray)
ax[1].fill_between(np.array(timearray)/doubling_time,np.array(cvszarray)-np.array(errcv2sz),np.array(cvszarray)+np.array(errcv2sz),
                 alpha=1, edgecolor='#4db8ff', facecolor='#4db8ff',linewidth=0)

data=pd.read_csv("./data/dataFSP.csv")
ax[0].plot(data.time/doubling_time,data.Meansize,ls='--',c='g',label="Numeric",lw=2)
ax[1].plot(data.time/doubling_time,data.VarSize/data.Meansize**2,ls='--',c='g',lw=2)

ax[0].set_ylabel("$s$ $(\mu m)$",size=20)
ax[1].set_ylabel("$C_V^2(s)$",size=20)
ax[0].set_xlabel(r"$t/\tau$",size=20)
ax[1].set_xlabel(r"$t/\tau$",size=20)
#ax[0].set_ylim([1,1.7])
#ax[1].set_ylim([0,0.15])
for l in [0,1]:
    ax[l].set_xlim([0,7])
    taqui=np.arange(0,8,step=1)
    ax[l].set_xticks(np.array(taqui))
    ax[l].grid()
    ax[l].tick_params(axis='x', labelsize=15)
    ax[l].tick_params(axis='y', labelsize=15)
    for axis in ['bottom','left']:
        ax[l].spines[axis].set_linewidth(2)
        ax[l].tick_params(axis='both', width=2,length=6)
    for axis in ['top','right']:
        ax[l].spines[axis].set_linewidth(0)
        ax[l].tick_params(axis='both', width=0,length=6)
plt.subplots_adjust(hspace=0.3,wspace=0.3)
taqui=np.arange(0,0.15,step=0.02)
ax[1].set_yticks(np.array(taqui))

ax[0].legend(fontsize=15)
plt.savefig('./figures/size_statistics.svg',bbox_inches='tight')
plt.savefig('./figures/size_statistics.png',bbox_inches='tight')_____no_output_____data2=pd.read_csv("./data/dataDSM.csv")
data2=data2[data2.time>3*doubling_time]
fig, ax = plt.subplots(1,2, figsize=(12,4))
ax[0].scatter(data2.S_b/np.mean(data2.S_b),(data2.S_d-data2.S_b)/np.mean(data2.S_b),s=2)

quantnumber=5
pvadd2=data2
CV2d=[]
delt=[]
sb=[]

errcv2d=[]
errdelt=[]
errsb=[]
for i in range(quantnumber):
    lperv0=np.percentile(pvadd2.S_b,i*100/quantnumber)
    hperv0=np.percentile(pvadd2.S_b,(i+1)*100/quantnumber)
    quanta1=pvadd2[pvadd2.S_b>lperv0]
    quanta2=quanta1[quanta1.S_b<hperv0]
    mean_cntr, var_cntr, std_cntr = bayesest((quanta2.S_d-quanta2.S_b)/np.mean(pvadd2.S_d-pvadd2.S_b),alpha=0.95)
    meanv0_cntr, varv0_cntr, stdv0_cntr = bayesest(quanta2.S_b/np.mean(pvadd2.S_b),alpha=0.95)
    CV2d.append(var_cntr[0]/mean_cntr[0]**2)
    delt.append(mean_cntr[0])
    sb.append(meanv0_cntr[0])
    errv=(var_cntr[1][1]-var_cntr[0])/mean_cntr[0]**2+2*(mean_cntr[1][1]-mean_cntr[0])*var_cntr[0]/mean_cntr[0]**3
    errcv2d.append(errv)
    errdelt.append(mean_cntr[1][1]-mean_cntr[0])
    errsb.append(meanv0_cntr[1][1]-meanv0_cntr[0])
ax[0].errorbar(np.array(sb),np.array(delt),xerr=errsb,yerr=errdelt, fmt='o',mec='k',capsize=5,markersize='8',elinewidth=3,c='#0075BD',label="SSA")
ax[1].errorbar(np.array(sb),CV2d,xerr=errsb,yerr=errcv2d, fmt='o',mec='k',capsize=5,markersize='8',elinewidth=3,c='#0075BD',label="SSA")
ax[1].set_ylim([0,0.3])

ax[0].set_xlabel("$s_b/\overline{s_b}$",size=20)
ax[1].set_xlabel("$s_b/\overline{s_b}$",size=20)
ax[0].set_ylabel("$\Delta/\overline{s_b}$",size=15)
ax[1].set_ylabel("$C_V^2(\Delta)$",size=15)
for l in [0,1]:
    ax[l].set_xlim([0.2,2])
    ax[l].grid()
    ax[l].tick_params(axis='x', labelsize=15)
    ax[l].tick_params(axis='y', labelsize=15)
    for axis in ['bottom','left']:
        ax[l].spines[axis].set_linewidth(2)
        ax[l].tick_params(axis='both', width=2,length=6)
    for axis in ['top','right']:
        ax[l].spines[axis].set_linewidth(0)
        ax[l].tick_params(axis='both', width=0,length=6)

ax[0].plot(np.array(sbar)/mean_size, np.array(deltsz)/mean_size, lw=2,c='k',label="Numeric")
ax[1].plot(np.array(sbar)/mean_size, cv2sz, lw=2,c='k',label="Numeric")
ax[1].legend(fontsize=15)

plt.savefig('./figures/div_strategy.eps',bbox_inches='tight')
plt.savefig('./figures/div_strategy.svg',bbox_inches='tight')
plt.savefig('./figures/div_strategy.png',bbox_inches='tight')_____no_output_____
</code>


# Notebook from DianaMosquera/course-content
Path: tutorials/W1D1_ModelTypes/W1D1_Tutorial1.ipynb

_____no_output_____
#  Neuromatch Academy: Week 1, Day 1, Tutorial 1
# Model Types: "What" models
__Content creators:__ Matt Laporte, Byron Galbraith, Konrad Kording

__Content reviewers:__ Dalin Guo, Aishwarya Balwani, Madineh Sarvestani, Maryam Vaziri-Pashkam, Michael Waskom

We would like to acknowledge [Steinmetz _et al._ (2019)](https://www.nature.com/articles/s41586-019-1787-x) for sharing their data, a subset of which is used here.
_____no_output________
# Tutorial Objectives
This is tutorial 1 of a 3-part series on different flavors of models used to understand neural data. In this tutorial we will explore 'What' models, used to describe the data. To understand what our data looks like, we will visualize it in different ways. Then we will compare it to simple mathematical models. Specifically, we will:

- Load a dataset with spiking activity from hundreds of neurons and understand how it is organized
- Make plots to visualize characteristics of the spiking activity across the population
- Compute the distribution of "inter-spike intervals" (ISIs) for a single neuron
- Consider several formal models of this distribution's shape and fit them to the data "by hand"_____no_output_____

#@title Video 1: "What" Models
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='KgqR_jbjMQg', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video_____no_output_____

# Setup

_____no_output_____Python requires you to explictly "import" libraries before their functions are available to use. We will always specify our imports at the beginning of each notebook or script._____no_output_____

import numpy as np
import matplotlib.pyplot as plt_____no_output_____

Tutorial notebooks typically begin with several set-up steps that are hidden from view by default.

**Important:** Even though the code is hidden, you still need to run it so that the rest of the notebook can work properly. Step through each cell, either by pressing the play button in the upper-left-hand corner or with a keyboard shortcut (`Cmd-Return` on a Mac, `Ctrl-Enter` otherwise). A number will appear inside the brackets (e.g. `[3]`) to tell you that the cell was executed and what order that happened in.

If you are curious to see what is going on inside each cell, you can double click to expand. Once expanded, double-click the white space to the right of the editor to collapse again._____no_output_____

#@title Figure Settings
import ipywidgets as widgets #interactive display

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")_____no_output_____#@title Helper functions

#@markdown Most of the tutorials make use of helper functions
#@markdown to simplify the code that you need to write. They are defined here.

# Please don't edit these, or worry about understanding them now!

def restrict_spike_times(spike_times, interval):
  """Given a spike_time dataset, restrict to spikes within given interval.

  Args:
    spike_times (sequence of np.ndarray): List or array of arrays,
      each inner array has spike times for a single neuron.
    interval (tuple): Min, max time values; keep min <= t < max.

  Returns:
    np.ndarray: like `spike_times`, but only within `interval`
  """
  interval_spike_times = []
  for spikes in spike_times:
    interval_mask = (spikes >= interval[0]) & (spikes < interval[1])
    interval_spike_times.append(spikes[interval_mask])
  return np.array(interval_spike_times, object)_____no_output_____#@title Data retrieval
#@markdown This cell downloads the example dataset that we will use in this tutorial.
import io
import requests
r = requests.get('https://osf.io/sy5xt/download')
if r.status_code != 200:
  print('Failed to download data')
else:
  spike_times = np.load(io.BytesIO(r.content), allow_pickle=True)['spike_times']_____no_output_____

---

# Section 1: Exploring the Steinmetz dataset

In this tutorial we will explore the structure of a neuroscience dataset.

We consider a subset of data from a study of [Steinmetz _et al._ (2019)](https://www.nature.com/articles/s41586-019-1787-x). In this study, Neuropixels probes were implanted in the brains of mice. Electrical potentials were measured by hundreds of electrodes along the length of each probe. Each electrode's measurements captured local variations in the electric field due to nearby spiking neurons. A spike sorting algorithm was used to infer spike times and cluster spikes according to common origin: a single cluster of sorted spikes is causally attributed to a single neuron.

In particular, a single recording session of spike times and neuron assignments was loaded and assigned to `spike_times` in the preceding setup.

Typically a dataset comes with some information about its structure. However, this information may be incomplete. You might also apply some transformations or "pre-processing" to create a working representation of the data of interest, which might go partly undocumented depending on the circumstances. In any case it is important to be able to use the available tools to investigate unfamiliar aspects of a data structure.

Let's see what our data looks like..._____no_output_____## Section 1.1: Warming up with `spike_times`_____no_output_____What is the Python type of our variable?_____no_output_____

type(spike_times)_____no_output_____

You should see `numpy.ndarray`, which means that it's a normal NumPy array.

If you see an error message, it probably means that you did not execute the set-up cells at the top of the notebook. So go ahead and make sure to do that.

Once everything is running properly, we can ask the next question about the dataset: what's its shape?_____no_output_____

spike_times.shape_____no_output_____

There are 734 entries in one dimension, and no other dimensions. What is the Python type of the first entry, and what is *its* shape?_____no_output_____

idx = 0
print(
  type(spike_times[idx]),
  spike_times[idx].shape,
  sep="\n",
)_____no_output_____

It's also a NumPy array with a 1D shape! Why didn't this show up as a second dimension in the shape of `spike_times`? That is, why not `spike_times.shape == (734, 826)`?

To investigate, let's check another entry._____no_output_____

idx = 321
print(
  type(spike_times[idx]),
  spike_times[idx].shape,
  sep="\n",
)_____no_output_____

It's also a 1D NumPy array, but it has a different shape. Checking the NumPy types of the values in these arrays, and their first few elements, we see they are composed of floating point numbers (not another level of `np.ndarray`):_____no_output_____

i_neurons = [0, 321]
i_print = slice(0, 5)

for i in i_neurons:
  print(
    "Neuron {}:".format(i),
    spike_times[i].dtype,
    spike_times[i][i_print],
    "\n",
    sep="\n"
  )_____no_output_____

Note that this time we've checked the NumPy `dtype` rather than the Python variable type. These two arrays contain floating point numbers ("floats") with 32 bits of precision.

The basic picture is coming together:
- `spike_times` is 1D, its entries are NumPy arrays, and its length is the number of neurons (734): by indexing it, we select a subset of neurons.
- An array in `spike_times` is also 1D and corresponds to a single neuron; its entries are floating point numbers, and its length is the number of spikes attributed to that neuron. By indexing it, we select a subset of spike times for that neuron.

Visually, you can think of the data structure as looking something like this:

```
| . . . . . |
| . . . . . . . . |
| . . . |
| . . . . . . . |
```

Before moving on, we'll calculate and store the number of neurons in the dataset and the number of spikes per neuron:_____no_output_____

n_neurons = len(spike_times)
total_spikes_per_neuron = [len(spike_times_i) for spike_times_i in spike_times]

print(f"Number of neurons: {n_neurons}")
print(f"Number of spikes for first five neurons: {total_spikes_per_neuron[:5]}")_____no_output_____#@title Video 2: Exploring the dataset
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='oHwYWUI_o1U', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video_____no_output_____

## Section 1.2: Getting warmer: counting and plotting total spike counts

As we've seen, the number of spikes over the entire recording is variable between neurons. More generally, some neurons tend to spike more than others in a given period. Lets explore what the distribution of spiking looks like across all the neurons in the dataset._____no_output_____Are most neurons "loud" or "quiet", compared to the average? To see, we'll define bins of constant width in terms of total spikes and count the neurons that fall in each bin. This is known as a "histogram".

You can plot a histogram with the matplotlib function `plt.hist`. If you just need to compute it, you can use the numpy function `np.histogram` instead._____no_output_____

plt.hist(total_spikes_per_neuron, bins=50, histtype="stepfilled")
plt.xlabel("Total spikes per neuron")
plt.ylabel("Number of neurons");_____no_output_____

Let's see what percentage of neurons have a below-average spike count:_____no_output_____

mean_spike_count = np.mean(total_spikes_per_neuron)
frac_below_mean = (total_spikes_per_neuron < mean_spike_count).mean()
print(f"{frac_below_mean:2.1%} of neurons are below the mean")_____no_output_____

We can also see this by adding the average spike count to the histogram plot:_____no_output_____

plt.hist(total_spikes_per_neuron, bins=50, histtype="stepfilled")
plt.xlabel("Total spikes per neuron")
plt.ylabel("Number of neurons")
plt.axvline(mean_spike_count, color="orange", label="Mean neuron")
plt.legend();_____no_output_____

This shows that the majority of neurons are relatively "quiet" compared to the mean, while a small number of neurons are exceptionally "loud": they must have spiked more often to reach a large count.

### Exercise 1: Comparing mean and median neurons

If the mean neuron is more active than 68% of the population, what does that imply about the relationship between the mean neuron and the median neuron?

*Exercise objective:* Reproduce the plot above, but add the median neuron.
_____no_output_____

# To complete the exercise, fill in the missing parts (...) and uncomment the code

median_spike_count = ...  # Hint: Try the function np.median

# plt.hist(..., bins=50, histtype="stepfilled")
# plt.axvline(..., color="limegreen", label="Median neuron")
# plt.axvline(mean_spike_count, color="orange", label="Mean neuron")
# plt.xlabel("Total spikes per neuron")
# plt.ylabel("Number of neurons")
# plt.legend()_____no_output_____# to_remove solution

median_spike_count = np.median(total_spikes_per_neuron)  # Hint: Try the function np.median

with plt.xkcd():
  plt.hist(total_spikes_per_neuron, bins=50, histtype="stepfilled")
  plt.axvline(median_spike_count, color="limegreen", label="Median neuron")
  plt.axvline(mean_spike_count, color="orange", label="Mean neuron")
  plt.xlabel("Total spikes per neuron")
  plt.ylabel("Number of neurons")
  plt.legend()_____no_output_____

*Bonus:* The median is the 50th percentile. What about other percentiles? Can you show the interquartile range on the histogram?_____no_output_____---

# Section 2: Visualizing neuronal spiking activity_____no_output_____## Section 2.1: Getting a subset of the data

Now we'll visualize trains of spikes. Because the recordings are long, we will first define a short time interval and restrict the visualization to only the spikes in this interval. We defined a utility function, `restrict_spike_times`, to do this for you. If you call `help()` on the function, it will tell you a little bit about itself:_____no_output_____

help(restrict_spike_times)_____no_output_____t_interval = (5, 15)  # units are seconds after start of recording
interval_spike_times = restrict_spike_times(spike_times, t_interval)_____no_output_____

Is this a representative interval? What fraction of the total spikes fall in this interval?_____no_output_____

original_counts = sum([len(spikes) for spikes in spike_times])
interval_counts = sum([len(spikes) for spikes in interval_spike_times])
frac_interval_spikes = interval_counts / original_counts
print(f"{frac_interval_spikes:.2%} of the total spikes are in the interval")_____no_output_____

How does this compare to the ratio between the interval duration and the experiment duration? (What fraction of the total time is in this interval?)

We can approximate the experiment duration by taking the minimum and maximum spike time in the whole dataset. To do that, we "concatenate" all of the neurons into one array and then use `np.ptp` ("peak-to-peak") to get the difference between the maximum and minimum value:_____no_output_____

spike_times_flat = np.concatenate(spike_times)
experiment_duration = np.ptp(spike_times_flat)
interval_duration = t_interval[1] - t_interval[0]

frac_interval_time = interval_duration / experiment_duration
print(f"{frac_interval_time:.2%} of the total time is in the interval")_____no_output_____

These two values—the fraction of total spikes and the fraction of total time—are similar. This suggests the average spike rate of the neuronal population is not very different in this interval compared to the entire recording.

## Section 2.2: Plotting spike trains and rasters

Now that we have a representative subset, we're ready to plot the spikes, using the matplotlib `plt.eventplot` function. Let's look at a single neuron first:_____no_output_____

neuron_idx = 1
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);_____no_output_____

We can also plot multiple neurons. Here are three:_____no_output_____

neuron_idx = [1, 11, 51]
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);_____no_output_____

This makes a "raster" plot, where the spikes from each neuron appear in a different row.

Plotting a large number of neurons can give you a sense for the characteristics in the population. Let's show every 5th neuron that was recorded:_____no_output_____

neuron_idx = np.arange(0, len(spike_times), 5)
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);_____no_output_____

*Question*: How does the information in this plot relate to the histogram of total spike counts that you saw above?_____no_output_____

#@title Video 3: Visualizing activity
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='QGA5FCW7kkA', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video_____no_output_____

---

# Section 3: Inter-spike intervals and their distributions_____no_output_____Given the ordered arrays of spike times for each neuron in `spike_times`, which we've just visualized, what can we ask next?

Scientific questions are informed by existing models. So, what knowledge do we already have that can inform questions about this data?

We know that there are physical constraints on neuron spiking. Spiking costs energy, which the neuron's cellular machinery can only obtain at a finite rate. Therefore neurons should have a refractory period: they can only fire as quickly as their metabolic processes can support, and there is a minimum delay between consecutive spikes of the same neuron.

More generally, we can ask "how long does a neuron wait to spike again?" or "what is the longest a neuron will wait?" Can we transform spike times into something else, to address questions like these more directly?

We can consider the inter-spike times (or interspike intervals: ISIs). These are simply the time differences between consecutive spikes of the same neuron.

### Exercise 2: Plot the distribution of ISIs for a single neuron

*Exercise objective:* make a histogram, like we did for spike counts, to show the distribution of ISIs for one of the neurons in the dataset.

Do this in three steps:

1. Extract the spike times for one of the neurons
2. Compute the ISIs (the amount of time between spikes, or equivalently, the difference between adjacent spike times)
3. Plot a histogram with the array of individual ISIs_____no_output_____

def compute_single_neuron_isis(spike_times, neuron_idx):
  """Compute a vector of ISIs for a single neuron given spike times.

  Args:
    spike_times (list of 1D arrays): Spike time dataset, with the first
      dimension corresponding to different neurons.
    neuron_idx (int): Index of the unit to compute ISIs for.

  Returns:
    isis (1D array): Duration of time between each spike from one neuron.
  """
  #############################################################################
  # Students: Fill in missing code (...) and comment or remove the next line
  raise NotImplementedError("Exercise: compute single neuron ISIs")
  #############################################################################

  # Extract the spike times for the specified neuron
  single_neuron_spikes = ...

  # Compute the ISIs for this set of spikes
  # Hint: the function np.diff computes discrete differences along an array
  isis = ...

  return isis

# Uncomment the following lines when you are ready to test your function
# single_neuron_isis = compute_single_neuron_isis(spike_times, neuron_idx=283)
# plt.hist(single_neuron_isis, bins=50, histtype="stepfilled")
# plt.axvline(single_neuron_isis.mean(), color="orange", label="Mean ISI")
# plt.xlabel("ISI duration (s)")
# plt.ylabel("Number of spikes")
# plt.legend()_____no_output_____# to_remove solution
def compute_single_neuron_isis(spike_times, neuron_idx):
  """Compute a vector of ISIs for a single neuron given spike times.

  Args:
    spike_times (list of 1D arrays): Spike time dataset, with the first
      dimension corresponding to different neurons.
    neuron_idx (int): Index of the unit to compute ISIs for.

  Returns:
    isis (1D array): Duration of time between each spike from one neuron.
  """
  # Extract the spike times for the specified neuron
  single_neuron_spikes = spike_times[neuron_idx]

  # Compute the ISIs for this set of spikes
  # Hint: the function np.diff computes discrete differences along an array
  isis = np.diff(single_neuron_spikes)

  return isis

single_neuron_isis = compute_single_neuron_isis(spike_times, neuron_idx=283)

with plt.xkcd():
  plt.hist(single_neuron_isis, bins=50, histtype="stepfilled")
  plt.axvline(single_neuron_isis.mean(), color="orange", label="Mean ISI")
  plt.xlabel("ISI duration (s)")
  plt.ylabel("Number of spikes")
  plt.legend()_____no_output_____

---

In general, the shorter ISIs are predominant, with counts decreasing rapidly (and smoothly, more or less) with increasing ISI. However, counts also rapidly decrease to zero with _decreasing_ ISI, below the maximum of the distribution (8-11 ms). The absence of these very low ISIs agrees with the refractory period hypothesis: the neuron cannot fire quickly enough to populate this region of the ISI distribution.

Check the distributions of some other neurons. To resolve various features of the distributions, you might need to play with the value of `n_bins`. Using too few bins might smooth over interesting details, but if you use too many bins, the random variability will start to dominate.

You might also want to restrict the range to see the shape of the distribution when focusing on relatively short or long ISIs. *Hint:* `plt.hist` takes a `range` argument _____no_output_____---

# Section 4: What is the functional form of an ISI distribution?_____no_output_____

#@title Video 4: ISI distribution
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='DHhM80MOTe8', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video_____no_output_____

The ISI histograms seem to follow continuous, monotonically decreasing functions above their maxima. The function is clearly non-linear. Could it belong to a single family of functions?

To motivate the idea of using a mathematical function to explain physiological phenomena, let's define a few different function forms that we might expect the relationship to follow: exponential, inverse, and linear._____no_output_____

def exponential(xs, scale, rate, x0):
  """A simple parametrized exponential function, applied element-wise.

  Args:
    xs (np.ndarray or float): Input(s) to the function.
    scale (float): Linear scaling factor.
    rate (float): Exponential growth (positive) or decay (negative) rate.
    x0 (float): Horizontal offset.

  """
  ys = scale * np.exp(rate * (xs - x0))
  return ys

def inverse(xs, scale, x0):
  """A simple parametrized inverse function (`1/x`), applied element-wise.

  Args:
    xs (np.ndarray or float): Input(s) to the function.
    scale (float): Linear scaling factor.
    x0 (float): Horizontal offset.

  """
  ys = scale / (xs - x0)
  return ys

def linear(xs, slope, y0):
  """A simple linear function, applied element-wise.

  Args:
    xs (np.ndarray or float): Input(s) to the function.
    slope (float): Slope of the line.
    y0 (float): y-intercept of the line.

  """
  ys = slope * xs + y0
  return ys_____no_output_____

### Interactive Demo: ISI functions explorer

Here is an interactive demo where you can vary the parameters of these functions and see how well the resulting outputs correspond to the data. Adjust the parameters by moving the sliders and see how close you can get the lines to follow the falling curve of the histogram. This will give you a taste of what you're trying to do when you *fit a model* to data.

"Interactive demo" cells have hidden code that defines an interface where you can play with the parameters of some function using sliders. You don't need to worry about how the code works – but you do need to **run the cell** to enable the sliders.
_____no_output_____

#@title

#@markdown Be sure to run this cell to enable the demo
# Don't worry about understanding this code! It's to setup an interactive plot.
single_neuron_idx = 283
single_neuron_spikes = spike_times[single_neuron_idx]
single_neuron_isis = np.diff(single_neuron_spikes)

counts, edges = np.histogram(
  single_neuron_isis,
  bins=50,
  range=(0, single_neuron_isis.max())
)

functions = dict(
  exponential=exponential,
  inverse=inverse,
  linear=linear,
)

colors = dict(
  exponential="C1",
  inverse="C2",
  linear="C4",
)

@widgets.interact(
  exp_scale=widgets.FloatSlider(1000, min=0, max=20000, step=250),
  exp_rate=widgets.FloatSlider(-10, min=-200, max=50, step=1),
  exp_x0=widgets.FloatSlider(0.1, min=-0.5, max=0.5, step=0.005),
  inv_scale=widgets.FloatSlider(1000, min=0, max=3e2, step=10),
  inv_x0=widgets.FloatSlider(0, min=-0.2, max=0.2, step=0.01),
  lin_slope=widgets.FloatSlider(-1e5, min=-6e5, max=1e5, step=10000),
  lin_y0=widgets.FloatSlider(10000, min=0, max=4e4, step=1000),
)
def fit_plot(
  exp_scale=1000, exp_rate=-10, exp_x0=0.1,
  inv_scale=1000, inv_x0=0,
  lin_slope=-1e5, lin_y0=2000,
):
  """Helper function for plotting function fits with interactive sliders."""
  func_params = dict(
    exponential=(exp_scale, exp_rate, exp_x0),
    inverse=(inv_scale, inv_x0),
    linear=(lin_slope, lin_y0),
  )
  f, ax = plt.subplots()
  ax.fill_between(edges[:-1], counts, step="post", alpha=.5)
  xs = np.linspace(1e-10, edges.max())
  for name, function in functions.items():
    ys = function(xs, *func_params[name])
    ax.plot(xs, ys, lw=3, color=colors[name], label=name);
  ax.set(
      xlim=(edges.min(), edges.max()),
      ylim=(0, counts.max() * 1.1),
      xlabel="ISI (s)",
      ylabel="Number of spikes",
  )
  ax.legend()_____no_output_____#@title Video 5: Fitting models by hand
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='uW2HDk_4-wk', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video_____no_output_____

# Summary

In this tutorial, we loaded some neural data and poked at it to understand how the dataset is organized. Then we made some basic plots to visualize (1) the average level of activity across the population and (2) the distribution of ISIs for an individual neuron. In the very last bit, we started to think about using mathematical formalisms to understand or explain some physiological phenomenon. All of this only allowed us to understand "What" the data looks like.

This is the first step towards developing models that can tell us something about the brain. That's what we'll focus on in the next two tutorials._____no_output_____


# Notebook from mm5110/PIC16A
Path: content/object_oriented_programming/inheritance_II.ipynb

# Inheritance II

- [Download the lecture notes](https://philchodrow.github.io/PIC16A/content/object_oriented_programming/inheritance_II.ipynb).

In this lecture, we'll code up a more complex example of inheritance. We'll define a general class, and then define several subclasses with different, distinctive behaviors.

We will model Starfleet Officers.

A Starfleet officer is a space-faring member of Starfleet, the military and scientific force of the United Federation of Planets. Starfleet officers all have names and a rank such as "Lieutenant" or "Captain." Additionally, Starfleet officers receive rudimentary combat training and can fire a phaser.

Finally, Starfleet officers are people, with mental and physical wellbeing. We'll model each of these as numbers between 0 to 1, with higher numbers indicating better mental or physical health. _____no_output_____

import random

class StarfleetOfficer:

    # class variable: shared across all StarfleetOfficers
    allegiance = "United Federation of Planets"

    def __init__(self, first, last, rank):
        self.rank = rank
        self.first = first
        self.last = last
        self.mental_health = random.random()
        self.physical_health = random.random()

    def introduce(self):
        return self.rank + " " + self.first + " " + self.last + " of the " + self.allegiance

    def fire_phaser(self):
        return "pew pew!"_____no_output_____burnham = StarfleetOfficer("Michael", "Burnham", "Commander")
burnham.introduce()_____no_output_____burnham.fire_phaser()_____no_output_____burnham.mental_health, burnham.physical_health_____no_output_____

Different kinds of `StarfleetOfficers` have different abilities. For example, medical officers use medical science to improve the health of their patients.

_____no_output_____

class MedicalOfficer(StarfleetOfficer):

    def operate_on(self, officer):
        officer.physical_health = random.uniform(officer.physical_health, 1)
        print("Is that better?")_____no_output_____bashir = MedicalOfficer("Julian", "Bashir", "Lieutenant")
bashir.introduce()_____no_output_____bashir.operate_on(burnham)
burnham.physical_health
# ---Is that better?

Ship counselors talk through problems with crew-members, helping those crew members to improve their mental well-being.

_____no_output_____

class ShipsCounselor(StarfleetOfficer):

    def talk_with(self, officer):
        officer.mental_health = random.uniform(officer.mental_health, 1)
        print("Thank you for sharing.")_____no_output_____troi = ShipsCounselor("Deanna", "Troi", "Lieutenant Commander")
troi.introduce()_____no_output_____troi.talk_with(burnham)

burnham.mental_health
# ---Thank you for sharing.

## Overriding

Suppose that we'd like to not just *add* a method to the `StarfleetOfficer` class, but that we'd actually like to *override* a method.

A *command* officer makes decisions of strategy and tactics. Suppose we'd like for command officers to introduce themselves differently.

_____no_output_____

class CommandOfficer(StarfleetOfficer):

    # overrides StarfleetOfficer.introduce()
    def introduce(self):
        return(self.first + " " + self.last + " and I call the shots around here!")_____no_output_____picard = CommandOfficer("Jean-Luc", "Picard", "Captain")
picard.introduce()_____no_output_____

We can still access the original method as well, albeit with some more complex syntax. We need to tell Python that we want to access the `full_name()` method associated with the `StarfleetOfficer` class. _____no_output_____

StarfleetOfficer.introduce(picard)_____no_output_____

A related and useful way to write new methods is to use the `super()` function, which will automatically identify the superclass of the subclass in which it is used. _____no_output_____

class CommandOfficer(StarfleetOfficer):

    # overrides StarfleetOfficer.full_name()
    def introduce(self):
        return(super().introduce() + " and I call the shots around here.")_____no_output_____