# Notebook from 00inboxtest/Stats-Maths-with-Python
Path: Brownian-motion-with-Python.ipynb

# Brownian motion with Python
## Tirthajyoti Sarkar, Fremont, CA, July 2020

## What is Brownian motion?

### Physical origin and properties
Brownian motion, or pedesis, is the random motion of particles suspended in a fluid (a liquid or a gas) resulting from their collision with the fast-moving molecules in the fluid.

This pattern describes a fluid at [thermal equilibrium](https://en.wikipedia.org/wiki/Thermal_equilibrium), defined by a given temperature. Within such a fluid, there exists no preferential direction of flow (as in [transport phenomena](https://en.wikipedia.org/wiki/Transport_phenomena)). More specifically, the fluid's overall linear and angular momenta remain null over time.

The kinetic energies of the molecular Brownian motions, together with those of molecular rotations and vibrations, sum up to the caloric component of a fluid's [internal energy](https://en.wikipedia.org/wiki/Internal_energy).

### Discovery and early explanations

This motion is named after the botanist [Robert Brown](https://en.wikipedia.org/wiki/Robert_Brown_(botanist,_born_1773)), who first described the phenomenon in 1827, while looking through a microscope at pollen of the plant Clarkia pulchella immersed in water.

In 1905, almost eighty years later, theoretical physicist **[Albert Einstein](https://en.wikipedia.org/wiki/Albert_Einstein) published [a seminal paper](http://www.maths.usyd.edu.au/u/UG/SM/MATH3075/r/Einstein_1905.pdf)** where he modeled the motion of the pollen as being moved by individual water molecules, making one of his first major scientific contributions. This explanation of Brownian motion **served as convincing evidence that atoms and molecules exist** and was further verified experimentally by [Jean Perrin](https://en.wikipedia.org/wiki/Jean_Baptiste_Perrin) in 1908.

Also read this [enjoyable paper commemorating the 100-year of Einstein's paper](https://arxiv.org/pdf/cond-mat/0504610.pdf).

## Brownian motion as a stochastic process
### Many-body interaction

The many-body interactions that yield the Brownian pattern cannot be solved by a model accounting for every involved molecule. In consequence, only probabilistic models applied to molecular populations can be employed to describe it. This is why it is described as a purely stochastic process in its modern form and applications.

### Wiener process
In mathematics, the Wiener process is a real valued continuous-time stochastic process named in honor of American mathematician [Norbert Wiener](https://en.wikipedia.org/wiki/Norbert_Wiener) for his investigations on the mathematical properties of the one-dimensional Brownian motion.

It is one of the best known stochastic processes with [stationary](https://en.wikipedia.org/wiki/Stationary_process) [independent increments](https://en.wikipedia.org/wiki/Independent_increments) and find frequent applications in a wide range of fields covering pure and applied mathematics, economics, quantitative finance, evolutionary biology, and physics.

For example, it plays a vital role in [stochastic calculus](https://en.wikipedia.org/wiki/Stochastic_calculus), [diffusion processes](https://en.wikipedia.org/wiki/Diffusion_process) and even [potential theory](https://en.wikipedia.org/wiki/Diffusion_process).  In applied mathematics, the Wiener process is used to represent the integral of a white noise [Gaussian process](https://en.wikipedia.org/wiki/Gaussian_process), and so is useful as a model of **noise in electronics engineering (see [Brownian noise](https://en.wikipedia.org/wiki/Brownian_noise))**, instrument errors in filtering theory and unknown forces in control theory.

In physics, it is used to study Brownian motion, the diffusion of minute particles suspended in fluid, and other types of diffusion via the [Fokker–Planck](https://en.wikipedia.org/wiki/Fokker%E2%80%93Planck_equation) and [Langevin equations](https://en.wikipedia.org/wiki/Langevin_equation). It also forms the basis for the rigorous path [integral formulation](https://en.wikipedia.org/wiki/Path_integral_formulation) of quantum mechanics (by the [Feynman–Kac formula](https://en.wikipedia.org/wiki/Feynman%E2%80%93Kac_formula), a solution to the Schrödinger equation can be represented in terms of the Wiener process) and the study of [eternal inflation](https://en.wikipedia.org/wiki/Eternal_inflation) in physical cosmology. It is also prominent in the mathematical theory of finance, in particular the **[Black–Scholes](https://en.wikipedia.org/wiki/Black%E2%80%93Scholes) option pricing model**._____no_output_____## A `Brownian` class

In the Python code below, we define a class `Brownian` with a few useful methods,

- `gen_random_walk()`: Generates motion from the Random Walk process
- `gen_normal()`: Generates motion by drawing from the Normal Distribution
- `stock_price()`: Models a stock price using so-called **'Geometric Brownian Motion'**_____no_output_____

import numpy as np
import matplotlib.pyplot as plt_____no_output_____class Brownian():
    """
    A Brownian motion class constructor
    """
    def __init__(self,x0=0):
        """
        Init class
        """
        assert (type(x0)==float or type(x0)==int or x0 is None), "Expect a float or None for the initial value"

        self.x0 = float(x0)

    def gen_random_walk(self,n_step=100):
        """
        Generate motion by random walk

        Arguments:
            n_step: Number of steps

        Returns:
            A NumPy array with `n_steps` points
        """
        # Warning about the small number of steps
        if n_step < 30:
            print("WARNING! The number of steps is small. It may not generate a good stochastic process sequence!")

        w = np.ones(n_step)*self.x0

        for i in range(1,n_step):
            # Sampling from the Normal distribution with probability 1/2
            yi = np.random.choice([1,-1])
            # Weiner process
            w[i] = w[i-1]+(yi/np.sqrt(n_step))

        return w

    def gen_normal(self,n_step=100):
        """
        Generate motion by drawing from the Normal distribution

        Arguments:
            n_step: Number of steps

        Returns:
            A NumPy array with `n_steps` points
        """
        if n_step < 30:
            print("WARNING! The number of steps is small. It may not generate a good stochastic process sequence!")

        w = np.ones(n_step)*self.x0

        for i in range(1,n_step):
            # Sampling from the Normal distribution
            yi = np.random.normal()
            # Weiner process
            w[i] = w[i-1]+(yi/np.sqrt(n_step))

        return w

    def stock_price(
                    self,
                    s0=100,
                    mu=0.2,
                    sigma=0.68,
                    deltaT=52,
                    dt=0.1
                    ):
        """
        Models a stock price S(t) using the Weiner process W(t) as
        `S(t) = S(0).exp{(mu-(sigma^2/2).t)+sigma.W(t)}`

        Arguments:
            s0: Iniital stock price, default 100
            mu: 'Drift' of the stock (upwards or downwards), default 1
            sigma: 'Volatility' of the stock, default 1
            deltaT: The time period for which the future prices are computed, default 52 (as in 52 weeks)
            dt (optional): The granularity of the time-period, default 0.1

        Returns:
            s: A NumPy array with the simulated stock prices over the time-period deltaT
        """
        n_step = int(deltaT/dt)
        time_vector = np.linspace(0,deltaT,num=n_step)
        # Stock variation
        stock_var = (mu-(sigma**2/2))*time_vector
        # Forcefully set the initial value to zero for the stock price simulation
        self.x0=0
        # Weiner process (calls the `gen_normal` method)
        weiner_process = sigma*self.gen_normal(n_step)
        # Add two time series, take exponent, and multiply by the initial stock price
        s = s0*(np.exp(stock_var+weiner_process))

        return s_____no_output_____

### Process with initial value of zero and using random walk

We can use a basic stochastic process such as **Random Walk**, to generate the data points for a Brownian motion. _____no_output_____

b = Brownian()_____no_output_____for i in range(4):
    plt.plot(b.gen_random_walk(1000))
plt.show()_____no_output_____

### Process with initial value of 20 and using Normal distribution

We can generate Brownian motion data by drawing from Normal distribution._____no_output_____

b = Brownian(20)_____no_output_____for i in range(4):
    plt.plot(b.gen_normal(1000))
plt.show()_____no_output_____

### Stock price simulation
Here we simulate multiple scenarios with for 52 time periods (imagining 52 weeks a year). Note, all the stock proces start at the same point but evolve randomly along different trajectories.

Note that the dynamics is controlled by the mean and variance parameters of the underlying Normal distribution, which, somehow emulate the growth trend and the 'volatility' of the stock. For example, a stock with positive growth trend will have a positive mean. For this particular simulation, the choice of mean (mu) is 0.2 and the choice of standard deviation (square root of the variance) is 0.68.

We define a utility function for plotting first._____no_output_____

def plot_stock_price(mu,sigma):
    """
    Plots stock price for multiple scenarios
    """
    plt.figure(figsize=(9,4))
    for i in range(5):
        plt.plot(b.stock_price(mu=mu,
                               sigma=sigma,
                               dt=0.1))
    plt.legend(['Scenario-'+str(i) for i in range(1,6)],
               loc='upper left')
    plt.hlines(y=100,xmin=0,xmax=520,
               linestyle='--',color='k')
    plt.show()_____no_output_____

Note that, although the scenarios look sufficiently stochastic, they have a downward trend. This is because even with a positive mean, we have a slightly high spread or volatility._____no_output_____

plot_stock_price(mu=0.2,sigma=0.7)_____no_output_____

### A slight change in the volatility

We simulate the stock price again with a **slightly less volatility (but with the same mean as before)** and get a completely different outcome this time. The trend looks neutral i.e. some scenario shows an increase in the stock price after the 52-week time period, whereas some show a decrease._____no_output_____

plot_stock_price(mu=0.2,sigma=0.65)_____no_output_____

### Lowering the volatility further down

If we lower the volatility even more, we get a clear positive trend._____no_output_____

plot_stock_price(mu=0.2,sigma=0.6)_____no_output_____

### Two-dimensional plot

In the following example, we show a two-dimensional Brownian motion much like the actual suspended particle in the fluid medium goes through._____no_output_____

b1 = Brownian()
b2 = Brownian()

x = b1.gen_normal(1000)
y = b2.gen_normal(1000)

plt.plot(x,y,c='b')
xmax,xmin,ymax,ymin = x.max(),x.min(),y.max(),y.min()
scale_factor = 1.25
xmax,xmin,ymax,ymin = xmax*scale_factor,xmin*scale_factor,ymax*scale_factor,ymin*scale_factor
plt.xlim(xmin,xmax)
plt.ylim(ymin,ymax)
plt.show()_____no_output_____


# Notebook from davidlmobley/drug-computing
Path: uci-pharmsci/lectures/QM/psi4_example.ipynb

# PharmSci 175/275 (UCI)
## What is this??
The material below is a supplement to the quantum mechanics (QM) lecture from Drug Discovery Computing Techniques, PharmSci 175/275 at UC Irvine.
Extensive materials for this course, as well as extensive background and related materials, are available on the course GitHub repository: [github.com/mobleylab/drug-computing](https://github.com/mobleylab/drug-computing)_____no_output_____# Using QM in Python

This material adapted (under CC-BY) from a [workshop example](https://github.com/QCMM/workshop2017/blob/master/Theory_electronic_structure_day3/i_inter_es.ipynb) of Stefano Vogt-Giesse (University of Concepcion) from the December, 2017 QCMM workshop in Chile, available under the [workshop repository](https://github.com/QCMM/workshop2017)

### Instructor: David L. Mobley

### Contributors to these materials:
- Stefano Vogt-Giesse
- David L. Mobley_____no_output_____# Exploring molecular interactions using electronic structure methods._____no_output_____For today's activity we will use the package of program psi4, so we will need to install it first. Assuming you already have anaconda/miniconda installed, you can install as follows (in a new conda environment, `p4env`:

**conda create -n psi4 psi4 psi4-rt jupyter matplotlib -c psi4/label/dev -c psi4**

(Note psi4 seems to be somewhat incompatible with the other software we are using in this course, so a separate environment is needed.)

This will install all the psi4 binaries and a python module which can be imported from the notebook. Then activate the environment via `conda activate psi4`.

You also need to ensure it works in your jupyter notebook, which you can do via (in the terminal, with your `psi4` environment active):
```
conda install ipykernel --name psi4
python -m ipykernel install --user
```

To finish the installation you need to provide a scratch directory in your `~/.bash_profile`, for example (assuming you want your scratch directory in this space):

**export PSI_SCRATCH=/home/user_name/scratch/psi4**

Then type `source ~/.bash_profile`. Now you may open jupyter-notebook and install psi4.

Every time you wish to use psi4 you will need to `conda activate psi4`._____no_output_____

import psi4
import numpy as np_____no_output_____

# 1. Compute the energy of a diatomic molecule

As a first example we will compute the scf energy of the diatómic molecule hydrogen flouride (HF):
_____no_output_____

# ==> Basic Psi4 options <==
# Memory
psi4.set_memory(int(5e8))
numpy_memory = 500

# Output
psi4.core.set_output_file('output.dat', False)

# Geometry input
hf_mol = psi4.geometry("""
0 1
H
F 1 0.917
""")

energy_hf_mol , wfn_hf_mol = psi4.energy('mp2/cc-pvtz', return_wfn=True)
print(energy_hf_mol) #Energy in Hartrees-100.3429063303609

This corresponds to the MP2/cc-pVTZ energy for this system (HF).  Input coordinates are given in [Z matrix format](https://en.wikipedia.org/wiki/Z-matrix_(chemistry)), using internal coordinates (in this case just a bond distance).

### Exercise: Compute the energy of other diatomic molecules using similar methods
You might consider computing the energy of of F$_2$ and N$_2$ using the cc-pVDZ and cc-pVTZ methods. _____no_output_____# 2. Compute the dipole and quadrupole moment of diatomic molecules_____no_output_____Since we are intrested in studying long range  molecular interactions using classical elctrodynamics, it is necessary
to compute the dipole and quadrupole moments. Quantum mechanically the dipole can be computed using the one electron dipole operator:

\begin{equation}
\hat{\mu} = \sum_i q_i r_i
\end{equation}

where $q_i$ is the charge of the particle and $r_i$ is the position vector of the particle.  The dipole moment can be computed using the wavefunction through the expectation value of the operator $\mu$.

\begin{equation}
\mu = <\psi|\hat{\mu}|\psi>
\end{equation}

In psi4 we can obtain the dipole moment from the wafefunction object that was defined above_____no_output_____

psi4.oeprop(wfn_hf_mol, 'DIPOLE', 'QUADRUPOLE', title='HF SCF')

mux = psi4.core.get_variable('HF SCF DIPOLE X') # in debye
muy = psi4.core.get_variable('HF SCF DIPOLE Y')
muz = psi4.core.get_variable('HF SCF DIPOLE Z')
quad_zz = psi4.core.get_variable('HF SCF QUADRUPOLE ZZ')_____no_output_____print(muz)
print(quad_zz)
mu = (np.sqrt(mux**2 + muy**2 + muz**2))
print(mu)-1.9411548174366635
-3.271015995238057
1.94115481744

# 3.  Compute a potential energy surface of HF dimer.

In order to study the physical interactions between two molecules it is convinient to draw
a potential energy surface along the interaction coordinate. In this section we will
obtain a potential energy profile for the most favorable dipole-dipole interaction, which is the
horizontal orientation with oposing dipole vectors, HF---FH. First we need to define a list containing
the distances between both dimers for which the energy will be obtained._____no_output_____

hf_dimer = psi4.geometry("""
        0 1
        H
        F 1 0.917
        H 2 R 1 180.0
        F 3 0.917 2 180.0 1 0.0
        """)_____no_output_____

Next, we write a loop and in each step of the loop we compute the energy at the mp4 level of theory. _____no_output_____

energy = []
dist   = []

Rval = np.arange(1.5,10.0,0.1)

for d in Rval:
    hf_dimer.R = d
    psi4.set_options({'freeze_core': 'True'})
    en = psi4.energy('scf/cc-pvtz')
    print(en)
    print(d)
    energy.append(en)
    dist.append(d)
-200.1159400762018
1.5
-200.1190819646694
1.6
-200.12084511069853
1.7
-200.1217418230204
1.8
-200.12210646265714
1.9
-200.12214740628218
2.0
-200.12199128622555
2.1
-200.12171855348862
2.2
-200.1213834825437
2.3
-200.12102248398264
2.4
-200.1206584434578
2.5
-200.12030471830317
2.6
-200.11996870223288
2.7
-200.11965427034008
2.8
-200.11936321363657
2.9
-200.11909600295286
3.0
-200.1188522860765
3.1
-200.11863115701004
3.2
-200.11843131717893
3.3
-200.11825114501002
3.4
-200.11808882589855
3.5
-200.11794245507383
3.6
-200.11781018000022
3.7
-200.11769030206548
3.8
-200.11758132579342
3.9
-200.11748198075108
4.0
-200.11739119292795
4.1
-200.11730805328622
4.2
-200.11723176731104
4.3
-200.11716164900196
4.4
-200.11709708263894
4.5
-200.1170375095619
4.6
-200.1169824408411
4.7
-200.11693143092367
4.8
-200.11688408496144
4.9
-200.11684005298991
5.0
-200.1167990260822
5.1
-200.1167607297471
5.2
-200.11672492543434
5.3
-200.11669140001007
5.4
-200.1166599601226
5.5
-200.11663044555814
5.6
-200.11660269950414
5.7
-200.11657658816026
5.8
-200.1165519929236
5.9
-200.1165288021483
6.0
-200.11650691710236
6.1
-200.11648624307801
6.2
-200.11646669629727
6.3
-200.11644820451244
6.4
-200.11643069564883
6.5
-200.11641410467115
6.6
-200.1163983745758
6.7
-200.11638344606672
6.8
-200.11636927063006
6.9
-200.116355801285
7.0
-200.1163429924834
7.1
-200.11633081020645
7.2
-200.11631921073243
7.3
-200.11630816039016
7.4
-200.11629762874782
7.5
-200.11628758588446
7.6
-200.1162780030017
7.7
-200.11626885594384
7.8
-200.116260117293
7.9
-200.11625176417053
8.0
-200.11624377692485
8.1
-200.1162361367973
8.2
-200.11622882502687
8.3
-200.1162218216226
8.4
-200.11621511335397
8.5
-200.11620868510147
8.6
-200.11620251717724
8.7
-200.11619660443722
8.8
-200.11619092565678
8.9
-200.11618547548989
9.0
-200.11618024015092
9.1
-200.11617521010604
9.2
-200.11617037397684
9.3
-200.11616572549198
9.4
-200.11616124982714
9.5
-200.1161569453809
9.6
-200.11615279951042
9.7
-200.1161488082223
9.8
-200.11614496206
9.9

Now we are ready to plot the potential energy profile. We will use the matplotlib python library for this
purpose. The function `ref_zero_kcal` transforms the energy which is in Hartee to kcal/mol and takes the
energy of the dimer with the farthest separation as the reference energy._____no_output_____

import matplotlib.pyplot as plt
%matplotlib inline_____no_output_____def ref_zero_kcal(en_list):
    energy_kcal = []
    for x in range(len(en_list)):
        energy_kcal.append((en_list[x] - en_list[-1])*627.51)
    return energy_kcal

energy_kcal = ref_zero_kcal(energy)
plt.plot(dist,energy_kcal)
plt.xlabel('Distance')
plt.ylabel('Energy (kcal/mol)')_____no_output_____

There are many more examples/tutorials in the Psi4 GitHub repositories, especially see `Tutorials` under the (psi4numpy repository](https://github.com/psi4/psi4numpy) for many Jupyter notebooks._____no_output_____


# Notebook from JanWeldert/freeDOM
Path: notebooks/ICU/upgrade_service_LLH_scans.ipynb

### Reconstruct an Upgrade event and conduct LLH scans using the LLH Service_____no_output_____

import pickle
import math
import os
from multiprocessing import Process
from copy import deepcopy

from matplotlib import pyplot as plt
import numpy as np

from freedom.llh_service.llh_client import LLHClient
from freedom.llh_service.service_utils import kill_service
from freedom.reco import crs_reco_____no_output_____

### Check for a consistent llh service / client installation_____no_output_____

import zmq
from freedom.llh_service import llh_cython
assert zmq.zmq_version() == llh_cython.zmq_version(), "Inconsistent zmq installations!"_____no_output_____params = {'legend.fontsize': 17,
          'figure.figsize': (15, 9.3),
          'axes.labelsize': 24,
          'axes.titlesize': 24,
          'xtick.labelsize': 22,
          'ytick.labelsize': 22}
plt.rcParams.update(params)_____no_output_____cvmfs_path = '/cvmfs/icecube.opensciencegrid.org/users/peller/freeDOM/resources'
model_path = f'{cvmfs_path}/Upgrade_NNs'_____no_output_____event_ind = 16256_____no_output_____with open(f'{cvmfs_path}/test_events_upgrade.pkl', 'rb') as f:
    event = pickle.load(f)[event_ind]_____no_output_____

### LLH service configuration:_____no_output_____

base_conf = {
        "poll_timeout": 1,
        "flush_period": 1,
        "n_hypo_params": 8,
        "n_hit_features": 10,
        "n_evt_features": 2,
        "send_hwm": 10000,
        "recv_hwm": 10000,
        "batch_size": {"n_hypos": 500},
        "req_addr": "tcp://127.0.0.1:*",
        "ctrl_addr": "tcp://127.0.0.1:12350"
}_____no_output_____def start_upgrade_service(base_conf, index, obs_per_hypo, hitnet_file, chargenet_file, cuda_device=0):
    conf = deepcopy(base_conf)
    conf["batch_size"]["n_observations"] = conf["batch_size"]["n_hypos"] * obs_per_hypo
    conf["ctrl_addr"] = crs_reco.adjust_addr_string(conf["ctrl_addr"], index)
    conf["hitnet_file"] = hitnet_file
    conf["chargenet_file"] = chargenet_file
    proc = Process(target=crs_reco.start_service,
                   args=(conf, conf["ctrl_addr"], conf["req_addr"], cuda_device))
    proc.start()

    return proc, conf["ctrl_addr"]_____no_output_____%%capture
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
dc_proc, dc_ctrl_addr = start_upgrade_service(base_conf, 0, 16,
                                              f"{model_path}/hitNet_DOMs.hdf5",
                                              f"{model_path}/chargeNet_DOMs.hdf5")
degg_proc, degg_ctrl_addr = start_upgrade_service(base_conf, 1, 32,
                                                  f"{model_path}/hitNet_DEggs.hdf5",
                                                  f"{model_path}/chargeNet_DEggs.hdf5")
mdom_proc, mdom_ctrl_addr = start_upgrade_service(base_conf, 2, 64,
                                                  f"{model_path}/hitNet_mDOMs.hdf5",
                                                  f"{model_path}/chargeNet_mDOMs.hdf5")_____no_output_____

### Construct the clients, one per DOM type_____no_output_____

%%time
dc_client = LLHClient(ctrl_addr=dc_ctrl_addr, conf_timeout=60000)
degg_client = LLHClient(ctrl_addr=degg_ctrl_addr, conf_timeout=60000)
mdom_client = LLHClient(ctrl_addr=mdom_ctrl_addr, conf_timeout=60000)CPU times: user 5.76 ms, sys: 11.1 ms, total: 16.8 ms
Wall time: 36.4 s

### Reconstruct the event_____no_output_____

rng = np.random.default_rng()

n_live_points = 97

event['hit_data'] = [event['hits_DOM'], event['hits_mDOM'], event['hits_DEgg']]
event['evt_data'] = [event['total_charge_DOM'], event['total_charge_mDOM'], event['total_charge_DEgg']]_____no_output_____%%time
fit_result = crs_reco.batch_crs_fit(event, [dc_client, mdom_client, degg_client], rng,
                                    n_live_points=97, do_postfit=True, store_all=True,
                                    spherical_indices=[[4, 5]], batch_size=12)
fit_result['n_calls']CPU times: user 409 ms, sys: 79.7 ms, total: 488 ms
Wall time: 19.6 s

### 1D scans_____no_output_____

true_params = event['params']

par_names = ['x', 'y', 'z', 'time',
             'azimuth', 'zenith',
             'cascade energy', 'track energy']

par_pairs = [(0, 1), (0, 2), (1, 2), (3, 2),
             (4, 5), (4, 2), (4, 0), (6, 7)
            ]

gridsize = 100
spacings = [np.linspace(true_params[0]-30, true_params[0]+30, 101),
            np.linspace(true_params[1]-30, true_params[1]+30, 101),
            np.linspace(true_params[2]-20, true_params[2]+20, 101),
            np.linspace(true_params[3]-80, true_params[3]+160, 101),
            np.linspace(0, 2*math.pi, 101),
            np.linspace(0, math.pi, 101),
            np.logspace(-1, 3, 101),
            np.logspace(-1, 3, 101)
           ]_____no_output_____def upgrade_llh(params):
    components = []
    for client, suffix in zip([dc_client, mdom_client, degg_client], ['DOM', 'mDOM', 'DEgg']):
        components.append(client.eval_llh(event[f'hits_{suffix}'], event[f'total_charge_{suffix}'], params))
    total = sum(components)

    return total, components

def scan_1d(par_ind, base_params):
    n_pts = len(spacings[par_ind])
    params = np.repeat(base_params[np.newaxis, :], n_pts, axis=0)
    params[:, par_ind] = spacings[par_ind]

    return upgrade_llh(params)_____no_output_____truth_llh = upgrade_llh(event['params'])[0]
print(f"-LLH at truth: {truth_llh:.3f}")
print(f"-LLH at best fit: {fit_result['fun']:.3f}")-LLH at truth: -467.645
-LLH at best fit: -469.479
n_DOM = len(event['hits_DOM'])
n_DEgg = len(event['hits_DEgg'])
n_mDOM = len(event['hits_mDOM'])_____no_output_____def plot_1d_scans(base_params):
    lab = ['total', 'DOM (%i)'%(n_DOM), 'mDOM (%i)'%(n_mDOM), 'DEgg (%i)'%(n_DEgg)]
    fig, axes = plt.subplots(figsize=(35, 12), nrows=2, ncols=4)

    truth = event['params']
    best_fit = fit_result['x']

    for i, ax in enumerate(axes.flat):
        total, comps = scan_1d(i, base_params)
        for curve, label in zip([total] + comps, lab):
            ax.plot(spacings[i], curve - curve.min(), label=label, linewidth=3)

        if truth[i] != 0:
            ax.axvline(truth[i], color='black', linestyle='--', label='truth')

        ax.axvline(best_fit[i], color='red', label='best fit')

        if i == 3:
            ax.legend()

        if i > 5:
            ax.set_xscale('log')

        ax.set_ylabel(r'$\Delta$LLH')
        ax.set_xlabel(par_names[i])_____no_output_____plot_1d_scans(event['params'])
plt.gcf().suptitle('scan around truth', y=0.95, fontsize=24)_____no_output_____plot_1d_scans(fit_result["x"])
plt.gcf().suptitle('scan around best fit', y=0.95, fontsize=24)_____no_output_____

### 2D scans_____no_output_____

def colorbar(mappable):
    # from https://joseph-long.com/writing/colorbars/
    from mpl_toolkits.axes_grid1 import make_axes_locatable
    import matplotlib.pyplot as plt
    last_axes = plt.gca()
    ax = mappable.axes
    fig = ax.figure
    divider = make_axes_locatable(ax)
    cax = divider.append_axes("right", size="5%", pad=0.05)
    cbar = fig.colorbar(mappable, cax=cax)
    plt.sca(last_axes)
    cbar.ax.tick_params(labelsize=12)
    return cbar

# utility function to map out a 2d NLL slice
def map_2d(inds, steps, event, base_params=event['params']):
    mg = np.meshgrid(*steps)

    n_hypotheses = mg[0].size

    # param table is a table of hypotheses to evaluate
    # each row contains a value for each of the 8 hypothesis parameters
    param_table = np.repeat(base_params[np.newaxis, :], n_hypotheses, axis=0)

    # load hypotheses to evaluate into the param table
    for ind, coord in zip(inds, mg):
        param_table[:, ind] = coord.flat

    return upgrade_llh(param_table)[0].reshape([step.size for step in steps[::-1]])

def plot_2d(inds, steps, true_params, base_params, ax):
    llh_map = map_2d(inds, steps, event, base_params)

    delta = llh_map - llh_map.min()

    im = ax.pcolormesh(*steps, delta, cmap='Spectral',
                       rasterized=True, linewidth=0, shading='auto')
    colorbar(im)

    # place crosshair at true parameters
    ax.axvline(true_params[0], label='truth')
    ax.axhline(true_params[1])

    # white star at best fit point
    best_fit = [fit_result["x"][i] for i in inds]
    ax.plot(*[[x] for x in best_fit], 'w*', markersize=10, label='best fit')_____no_output_____def plot_2d_scan(base_params):
    fig, axes = plt.subplots(nrows=2, ncols=4, figsize=(16,8))
    plt.subplots_adjust(wspace=0.5)
    plt.subplots_adjust(hspace=0.25)

    axiter = axes.flat
    for i, ax in enumerate(axiter):
        ind0, ind1 = par_pairs[i]
        grid_spacing = (spacings[ind0], spacings[ind1])
        plot_2d((ind0, ind1), grid_spacing, (true_params[ind0], true_params[ind1]), base_params, ax)
        ax.set_xlabel(par_names[ind0], fontsize=12)
        ax.set_ylabel(par_names[ind1], fontsize=12)
        ax.tick_params(labelsize=12)

        if par_names[ind0].endswith('energy'):
            ax.set_xscale('log')
        if par_names[ind1].endswith('energy'):
            ax.set_yscale('log')

        if i == 3:
            ax.legend()_____no_output_____%%time
plot_2d_scan(event['params'])
plt.gcf().suptitle('Scan around truth', fontsize=16, y=0.95)CPU times: user 409 ms, sys: 69.5 ms, total: 479 ms
Wall time: 17.1 s
%%time
plot_2d_scan(fit_result["x"])
plt.gcf().suptitle('Scan around best fit', fontsize=16, y=0.95)CPU times: user 410 ms, sys: 57.6 ms, total: 468 ms
Wall time: 17.1 s

### Kill the LLH services_____no_output_____

def kill_services(addrs, procs):
    for addr, proc in zip(addrs, procs):
        kill_service(addr)
        proc.join()_____no_output_____kill_services([dc_ctrl_addr, mdom_ctrl_addr, degg_ctrl_addr], [dc_proc, mdom_proc, degg_proc])_____no_output_____


# Notebook from Roberto919/Propedeutico
Path: Ejercicios_clase/R/1_introduccion/1_informacion_general.ipynb

**Notas para contenedor de docker:**_____no_output_____Comando de docker para ejecución de la nota de forma local:

nota: cambiar `dir_montar` por la ruta de directorio que se desea mapear a `/datos` dentro del contenedor de docker.

```
dir_montar=<ruta completa de mi máquina a mi directorio>#aquí colocar la ruta al directorio a montar, por ejemplo:
#dir_montar=/Users/erick/midirectorio.
```

Ejecutar:

```
$docker run --rm -v $dir_montar:/datos --name jupyterlab_prope_r_kernel_tidyverse -p 8888:8888 -d palmoreck/jupyterlab_prope_r_kernel_tidyverse:2.1.4

```_____no_output_____Ir a `localhost:8888` y escribir el password para jupyterlab: `qwerty`

Detener el contenedor de docker:

```
docker stop jupyterlab_prope_r_kernel_tidyverse
```
_____no_output_____Documentación de la imagen de docker `palmoreck/jupyterlab_prope_r_kernel_tidyverse:2.1.4` en [liga](https://github.com/palmoreck/dockerfiles/tree/master/jupyterlab/prope_r_kernel_tidyverse)._____no_output_____---_____no_output_____Para ejecución de la nota usar:

[docker](https://www.docker.com/) (instalación de forma **local** con [Get docker](https://docs.docker.com/install/)) y ejecutar comandos que están al inicio de la nota de forma **local**.

O bien dar click en alguno de los botones siguientes:_____no_output_____[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/palmoreck/dockerfiles-for-binder/jupyterlab_prope_r_kernel_tidyerse?urlpath=lab/tree/Propedeutico/R/clases/1_introduccion/1_informacion_general.ipynb) esta opción crea una máquina individual en un servidor de Google, clona el repositorio y permite la ejecución de los notebooks de jupyter._____no_output_____[![Run on Repl.it](https://repl.it/badge/github/palmoreck/dummy)](https://repl.it/languages/Rlang) esta opción no clona el repositorio, no ejecuta los notebooks de jupyter pero permite ejecución de instrucciones de Python de forma colaborativa con [repl.it](https://repl.it/). Al dar click se crearán nuevos ***repl*** debajo de sus users de ***repl.it***.
_____no_output_____### Lo siguiente está basado en:

### * [R: Un lenguaje para análisis de datos por E. B. Zamudio](http://allman.rhon.itam.mx/~ebarrios/docs/porqueR.pdf)

### * El prefacio y capítulo 1 de M.D. Ugarte, A. F. Militino, A. T. Arnholt, Probability and Statistics with R, **2ed** (que tiene github: [alanarnholt/PASWR2E-Book](https://github.com/alanarnholt/PASWR2E-Book))

### * [R-intro](https://cran.r-project.org/doc/manuals/R-intro.html)._____no_output_____# Información general_____no_output_____## ¿Qué es?_____no_output_____R es un lenguaje de programación basado en *S* el cual es un lenguaje y un ambiente de programación interactiva para el análisis de datos y graficación desarrollado por los *Bell laboratories* por Rick Becker, John Chambers y Allan Wilks. Dos dialectos del lenguaje S existen: *R* una implementación **open source** disponible en [r-project](http://www.r-project.org) y *S-PLUS* una implementación comercial. R es el resultado de esfuerzos de todo el mundo y fue escrito inicialmente por Robert Gentleman y Ross Ihaka del departamento de Estadística de la Universidad de Auckland en 1996._____no_output_____R es un lenguaje y ambiente para cómputo estadístico y graficación. El término ambiente lo caracteriza como un sistema completamente planeado y coherente en el que técnicas estadísticas clásicas y modernas han sido implementadas. Algunas forman parte de *R* las llamadas *built-in* pero otras se proveen vía paquetes. Alrededor de 25 paquetes son incluídas con la instalación de R (*standard* & *recommended*) y muchas más se encuentran disponibles en [CRAN family of Internet sites: Contributed Packages](https://cran.r-project.org/web/packages/index.html)_____no_output_____## ¿Qué provee?_____no_output_____El ambiente de R contiene software que permite la manipulación de datos, cálculos y visualización de gráficos y tiene:

* programas para almacenamiento y manejo de datos.
* una gama amplia de operadores para cálculos en arreglos, en particular matrices.
* una gran colección de herramientas intermedias correctamente integradas para análisis de datos.
* lenguaje de programación *S* para *loops*, funciones definidas por *users* y desarrollo para I/O.
_____no_output_____*R* es un vehículo para desarrollo de nuevos métodos de análisis de datos de forma interactiva y se extiende por medio de paquetes._____no_output_____## ¿Diferencia entre S (y por tanto R) y otros softwares de estadística?_____no_output_____(extraído de [R-intro: 1.3 R and statistics](https://cran.r-project.org/doc/manuals/R-intro.html)) "There is an important difference in philosophy between S (and hence R) and the other main statistical systems. In S a statistical analysis is normally done as a series of steps, with intermediate results being stored in objects. Thus whereas SAS and SPSS will give copious output from a regression or discriminant analysis, R will give minimal output and store the results in a fit object for subsequent interrogation by further R functions."_____no_output_____## Características de S _____no_output_____(extraído de [R: Un lenguaje para análisis de datos por E. B. Zamudio](http://allman.rhon.itam.mx/~ebarrios/docs/porqueR.pdf))_____no_output_____* *S* es sobre datos: provee de herramientas generales y fáciles de usar para la organización, almacenamiento y recuperación de varios tipos de datos.

* *S* es sobre análisis: es decir, cálculos necesarios para entender los datos. *S* provee de métodos numéricos y otras técnicas computacionales.

* *S* es sobre programación: usted puede programar funciones en el mismo lenguaje *S* aprovechando su poder y simplicidad. Si es necesario el lenguaje ofrece  interfaces  para  comunicación  con  el  sistema  operativo  o rutinas en C y Fortran.

* Especialmente, *S* es sobre graficación: ver a los datos de maneras interactivas, informativas y flexibles. Las capacidades de *S* están diseñadas para motivar la creación de nuevas herramientas e intentar nuevas ideas.
_____no_output_____## Ejemplo:_____no_output_____**En algunas líneas del código siguiente no es necesario colocar print (sólo se realiza por la integración entre Jupyterlab y R)**_____no_output_____Calcular la probabilidad de obtener sol al lanzar una moneda vía simulación._____no_output_____**Solución:** para este caso simulamos $1000$ lanzamientos con la función [sample](https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/sample) que se encuentra en el paquete [base](https://stat.ethz.ch/R-manual/R-devel/library/base/html/00Index.html) de R. Al final del ejemplo se muestra cómo acceder a su descripción y la misma._____no_output_____Por ejemplo si queremos simular un resultado de lanzamiento lo ejecutamos como:_____no_output_____- `sample` es un comando para hacer una simulación probabilística_____no_output_____

set.seed(2000) #for reproducibility
sample(c('aguila', 'sol'),1)_____no_output_____

Si queremos simular tres resultados de lanzamientos realizamos:_____no_output_____

set.seed(2000)
sample(c('aguila', 'sol'), 3, replace=T) #with replacement_____no_output_____

Creamos una función con la palabra `function` que recibe el número de lanzamientos a realizar:_____no_output_____- `function` es una palabra reservada en *R*_____no_output_____

throw_coin <- function(number_of_throwings){
                  set.seed(1989) #1989 randomly choosen
                  sample(c('aguila', 'sol'),numero_de_lanzamientos,replace=T)
              }_____no_output_____

**Obs:** obsérvese que se ha utilizado el operador de asignación `<-`. También es posible utilizar `=` pero hay situaciones cuando el uso del `=` crea confusión al leer (desde la perspectiva de user) las líneas por lo que se opta por `<-`._____no_output_____- el `=` se usa para parámetros_____no_output_____Por ejemplo 4 lanzamientos se realizan con:_____no_output_____

throw_coin(4)_____no_output_____

8 lanzamientos con:_____no_output_____

throw_coin(8)_____no_output_____

Entonces nuestra simulación de lanzamientos se ejecuta como sigue:_____no_output_____

num_tosses <- 1000
sum(throw_coin(num_tosses)=='sol')/num_tosses_____no_output_____

el cual es un valor cercano a la respuesta $0.5$._____no_output_____y obsérvese que con un número menor de lanzamientos nuestra estimación no es buena:_____no_output_____

num_tosses <- 10
sum(throw_coin(num_tosses)=='sol')/num_tosses_____no_output_____

- Este es un modelo frecuentista para modelar probabilidades_____no_output_____### ¿Y si quisiéramos usar un paquete para este ejemplo? ..._____no_output_____Un paquete es una colección de funciones de R, datos (*datasets*) y código compilado que tienen una organización uniforme._____no_output_____Se pueden instalar con la línea `install.packages("<aquí colocar el nombre del paquete>", dependencies=TRUE)` que utiliza por default el repositorio [The Comprehensive R Archive Network: CRAN](https://cran.r-project.org/) (pero hay otros repositorios como: [Bioconductor](https://www.bioconductor.org/), [Omegahat](http://www.omegahat.net/) o [R-Forge](http://r-forge.r-project.org/))._____no_output_____**Obs:** Un paquete sólo se instala una vez excepto si se tiene una nueva versión de R._____no_output_____## ¿Para instalar paquetes en los jupyter notebooks...?_____no_output_____Podemos usar en una celda:

```
install.packages("<aquí colocar el nombre del paquete>",lib="/usr/local/lib/R/site-library/",
                repos="https://cran.itam.mx/",verbose=TRUE)
```_____no_output_____## Ejemplo de paquete que ya está en la imagen de docker instalado_____no_output_____En la imagen de docker `palmoreck/jupyterlab_prope_r_kernel_tidyverse:2.1.4` cuya documentación está en [liga](https://github.com/palmoreck/dockerfiles/tree/master/jupyterlab/prope_r_kernel_tidyverse) se tienen paquetes ya instalados que utilizaremos en el curso._____no_output_____El paquete `prob` es un framework para calcular probabilidades de experimentos sencillos (finitos) y generar las posibilidades de resultados. Ver: [Package prob](https://cran.r-project.org/web/packages/prob/prob.pdf)_____no_output_____Una vez instalado el paquete se debe hacer disponibles todos los contenidos del paquete en el ambiente de trabajo, esto es, se debe *load a package*, para esto tenemos a la función `library`:_____no_output_____**Comentario:** si se ejecuta la función library sin argumentos: `library()` se enlistan todos los paquetes instalados_____no_output_____

library(prob)Loading required package: combinat

Attaching package: ‘combinat’

The following object is masked from ‘package:utils’:

    combn

Loading required package: fAsianOptions

Loading required package: timeDate

Loading required package: timeSeries

Loading required package: fBasics

Loading required package: fOptions

Attaching package: ‘prob’

The following objects are masked from ‘package:base’:

    intersect, setdiff, union

**Obs:** obsérvese que al ejecutar `library` el output indica que está ennmascarados algunos objetos como `union`. Esto indica que en `prob` existe un objeto con nombre `union` y también en `base` existe el mismo nombre de `union`. Para ejecutar una u otra función se puede: usar una sintaxis de la forma `base::union` para uso de la función `union` del paquete `base` o bien `prob::union` para uso de la función `union` del paquete `prob` aunque también **muchas de las funciones se *comportan* de forma distinta dependiendo del tipo de argumentos que reciben**. _____no_output_____Usamos la función find para observar el enmascaramiento antes descrito_____no_output_____

find('union') _____no_output_____

Se utiliza la función [tosscoin](https://www.rdocumentation.org/packages/prob/versions/1.0-1/topics/tosscoin) para resolver el problema y se puede acceder a los argumentos de la misma con la función `args`:_____no_output_____

args(tosscoin)_____no_output_____

#### 1 lanzamiento de una moneda, dos posibilidades_____no_output_____

tosscoin(1)_____no_output_____

#### 1 lanzamiento de una moneda, dos posibilidades y probabilidad de .5_____no_output_____

tosscoin(1, makespace = T)_____no_output_____

#### Lanzamiento de tres monedas, 8 posibilidades y probabilidad de 1/8_____no_output_____A manera de ejemplo de uso de esta función, se muestra cómo se puede realizar diferentes cálculos de probabilidades_____no_output_____

tosscoin(3, makespace = T)_____no_output_____

## Funciones en R para obtener información o ayuda de paquetes o funciones mismas_____no_output_____La función [sample](https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/sample) se encuentra en el paquete *base* de R y se puede obtener la ayuda de este paquete con la sintaxis `?<package>` o `help(<package>)`._____no_output_____

?base_____no_output_____

Sin embargo como se hizo `library(prob)` anteriormente (y se tiene enmascaramiento de algunos objetos) si ejecutamos la ayuda para la función de `sample` (la cual también se puede obtener con `?<func>` o `help(func)`) se obtiene:_____no_output_____**Obs:** obsérvese que sample aparece en el paquete timeDate y no en el paquete base ..._____no_output_____

?sample _____no_output_____

Y observando que el paquete `timeDate` también tiene una función `sample` en el output de arriba o con:_____no_output_____

find('sample')_____no_output_____

entonces se tiene un enmascaramiento entre `sample` del paquete `base` y el paquete `timeDate`. Por lo que se puede acceder a la ayuda de `sample` del paquete `timeDate` con:_____no_output_____

?timeDate::sample_____no_output_____

y para la función `sample` del `base package` que nos interesa debemos hacer:_____no_output_____

?base::sample _____no_output_____

Si queremos ejecutar los ejemplos de la ayuda anterior, se utiliza la función `example`. A continuación se muestran los argumentos que recibe la función `example`:_____no_output_____

args(example)_____no_output_____

y se ejecutan los ejemplos de la función `sample` como sigue:_____no_output_____

example(sample, package='base')
sample> x <- 1:12

sample> # a random permutation
sample> sample(x)
 [1]  5  7  4  1  3  8  9 12 11  6  2 10

sample> # bootstrap resampling -- only if length(x) > 1 !
sample> sample(x, replace = TRUE)
 [1]  2  6  6 12  2  6 10  8  2  7  7  1

sample> # 100 Bernoulli trials
sample> sample(c(0,1), 100, replace = TRUE)
  [1] 1 1 1 0 1 0 0 0 0 0 1 0 1 0 1 1 1 1 0 0 1 1 1 0 1 0 1 0 0 0 1 1 1 0 1 1 1
 [38] 1 1 0 0 0 1 0 0 1 0 1 1 0 1 0 0 1 1 0 0 0 0 0 1 1 1 1 1 0 1 0 1 0 0 1 1 0
 [75] 1 0 1 1 0 0 1 0 1 1 1 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0

sample> ## More careful bootstrapping --  Consider this when using sample()
sample> ## programmatically (i.e., in your function or simulation)!
sample>
sample> # sample()'s surprise -- example
sample> x <- 1:10

sample>     sample(x[x >  8]) # length 2
[1] 10  9

sample>     sample(x[x >  9]) # oops -- length 10!
 [1]  3  6  1  7  8  2  9  5  4 10

sample>     sample(x[x > 10]) # length 0
integer(0)

sample> ## safer version:
sample> resample <- function(x, ...) x[sample.int(length(x), ...)]

sample> resample(x[x >  8]) # length 2
[1] 10  9

sample> resample(x[x >  9]) # length 1
[1] 10

sample> resample(x[x > 10]) # length 0
integer(0)

sample> ## R 3.x.y only
sample> sample.int(1e10, 12, replace = TRUE)
 [1] 2575566849 3514429428 2827840570 6308005517  811253583 6982960333
 [7] 9251408822 8413115991 8011349018 2787805302 7411643999 9521768691

sample> sample.int(1e10, 12) # not that there is much chance of duplicates
 [1] 2078891680 9333817626 6049975312 4416776733 3467884371 3240276955
 [7] 9839126466 8630997200 7205736564 3237221598 1269177447 9814225164

Para la función `tosscoin` se tiene la siguiente ayuda:_____no_output_____

?tosscoin_____no_output_____

## Para saber más sobre R_____no_output_____* Ir a la documentación oficial: https://cran.r-project.org/doc/manuals/R-intro.html
* Ir [aquí](https://github.com/ITAM-DS/Propedeutico/tree/master/R) para más referencias sobre R.

Sobre paquetes:
* En Bioconductor (y otros repositorios) encontrarás paquetes que no están en CRAN.
* La mejor manera de conocer qué hace y usar un paquete es ir a su documentación oficial y seguir un tutorial o vignette:

```
#vignette() muestra documentación de todos los paquetes instalados
vignette('prob') #muestra documentación del paquete prob
```

* Para instalación de paquetes desde diferentes fuentes (incluyendo CRAN y otros repositorios) dar click en [devtools](https://devtools.r-lib.org/) que es una colección de herramientas para desarrollo de paquetes (pero también tiene funcionalidad para instalación de paquetes) y su correspondiente [Package Development:: Cheat Sheet](https://rawgit.com/rstudio/cheatsheets/master/package-development.pdf).

* Para buscar paquetes que fueron instalados usar:
_____no_output_____

help.search('prob') #or: ??prob_____no_output_____

Sobre funciones para encontrar objetos:

* La función `apropos` también ayuda a encontrar objetos. Nos devuelve objetos (incluídas funciones) en la sesión actual de R que tienen nombres que incluyen un *string* en específico.  Por ejemplo en la siguiente ejecución obtenemos los objetos que tienen el nombre `union`:_____no_output_____

apropos('union')_____no_output_____

`apropos` y cómo buscar ayuda en R se encuentra en [Getting Help with R](https://www.r-project.org/help.html)._____no_output_____La función de `library` también nos proporciona ayuda de un paquete:_____no_output_____

library(help=prob)_____no_output_____?prob_____no_output_____sessionInfo()_____no_output_____


# Notebook from senthilprabhut/DataScience
Path: 00 Statistical Learning/Medicon_dose_testing.ipynb

# **Context:**

Pharmaceutical company Medicon has manufactured the sixth batch (40,000 units) of COVID-19 vaccine doses. This vaccine was clinically tested last quarter and around 2,00,000 doses of this vaccine have already been given to the people, in five batches. Now, this sixth batch of doses needs to be tested for their time of effect (which is measured as the time taken for the dose to completely cure COVID), as well as for quality assurance (which tells you whether the dose will be able to do a satisfactory job or not). [Please note that this is not a clinical trial, the vaccine has already cleared clinical trials. This batch test is to ensure the quality and effectiveness of the sixth batch] The company wants to plan for the next batch of doses. Let's understand the quality of this(sixth) batch. You are working with the quality assurance team of Medicon and want to understand the quality of the sixth batch.

**Note:** Here, satisfactory job means that the dose is able to completely cure covid and it does not have any side effects.
_____no_output_____## **Problem statement**:

From the previous analysis, it is found that a dose is 2 times more likely to produce a satisfactory result than not.

1) The quality assurance team has collected 10 volunteers who were ready for the trial of this new batch and given one dose to each volunteer. Help the team to answer the following questions

a) Plot the probability distribution for the number of unsatisfactory doses.

b) Find the probability that exactly 3 doses will not be able to do a satisfactory job.

c) Find the probability that at most 3 doses will not do the satisfactory job out of 10 selected.

2) The New York city administration wants to buy 20 Doses for Health care workers who were COVID-Positive. They have contacted the company and requested 20 Doses. What is the probability that at least 11 doses will not do a satisfactory job out of 20 doses?

3) Quality assurance team wants to analyze the time of effect for doses. So, they have collected the 50 volunteers with the help of the city administration and given one dose to each volunteer. Data of these 50 doses are in the 'doses.csv' file. Based on this sample, answer the following questions:

a) What is the probability that a dose will take less than 11.5 hours?

b) What is the 90th percentile of time of effect for the doses?

c) For the effectiveness test of the vaccine. Estimate the range in which the population mean (effective time) will lie with a 95% confidence level._____no_output_____## Import the libraries _____no_output_____

#import the important packages
import pandas as pd #library used for data manipulation and analysis
import numpy as np # library used for working with arrays.
import matplotlib.pyplot as plt # library for plots and visualisations
import seaborn as sns # library for visualisations
%matplotlib inline

import scipy.stats as stats # this library contains a large number of probability distributions as well as a growing library of statistical functions._____no_output_____

**The quality assurance team has collected the 10 volunteers who were ready for the trial of this new batch and given one dose to each volunteer. What is the probability that k doses will do a satisfactory job out of 10 doses given to the volunteers?**

The above situation follows the Binomial distribution because we are taking the random sample of 10 doses and checking the probability of exactly 3 will not do a satisfactory job.

Let's check first whether the assumptions of the binomial distribution are satisfied or not.

*  There are only two possible outcomes (success or failure) for each trial – A dose will do a satisfactory job or not (yes or no).

* Number of trials (n) is fixed - A  sample of size 10 is taken. Each dose in the sample represents a trial.

* Each trial is independent of the other trials - It is reasonable to assume that giving dose to each person is independent.

* The probability of success (p) is the same for each trial - The probability of success for each trial (a dose will not do a satisfactory job) is not known.

Let's find the probability of success of each trial (A dose will not do a satisfactory job)

As we already know, the quality assurance teams check on the fifth batch of doses and found that - it is 2 times more likely that a dose will be able to produce a satisfactory result than not.

Let p be the probability that a dose will not do a satisfactory job.

It is 2 times more likely that a dose will be able to produce a satisfactory result than not. So, the probability that a dose will do a satisfactory job is 2p.

According to the rules of probability,

p + 2p = 1

=> 3p = 1

=> p = 1/3

=> p = 0.33

Hence, The probability of success for each trial (a dose will not do a satisfactory job) is 0.33._____no_output_____## Lets estimate the probability distribution of doses
_____no_output_____

# declare the sample size in variable n which represents the number of doses given to the volunteers
n=10_____no_output_____# declare p which represents the probability of success, i.e., the probability that a dose will not do a satisfactory job
p=0.33_____no_output_____# declare different number of doses selected in a numpy array
k=np.arange(0,11)
k_____no_output_____# import the required function
from scipy.stats import binom
# use the binom.pmf() function to generate the probability distribution
binomial = binom.pmf(k = k,n = n, p = p)_____no_output_____binomial_____no_output_____

As you can see **binomial** is the array of probabilities for the different number of doses that will not do a satisfactory job. Let us visualize the same in the following plot._____no_output_____**a) Plot the  probability distribution**
_____no_output_____

# plot the distribution
plt.bar(k,binomial)
plt.title('Binomial: n=%i , p=%.2f' % (n,p), fontsize=15)
plt.xlabel('Number of Doses')
plt.ylabel('Probability that doses will not do a satisfactory job')
plt.show()_____no_output_____

**b) Find the probability that exactly, 3 doses will not be able to do a satisfactory job.**_____no_output_____

binomial[3]_____no_output_____

* The probability of exactly 3 doses will not do a satisfactory job is 26.14%._____no_output_____**c) Find the probability that at most 3 doses will not do the satisfactory job out of 10 selected.**

First, we need to calculate P(X<=3). Here, X is the random variable that represents the number of doses that will not do a satisfactory job.

CDF is used to calculate the cumulative probability.
_____no_output_____

# plot the probability distribution
# We are plotting the distributions here to better visualize the calculations.
# Of course you do not 'need' to create the following visualization to answer the question above.
# You can directly use the cdf function for probability calculations.
barl=plt.bar(k,binomial)
plt.title('Binomial: n=%i , p=%.2f' % (n,p), fontsize=15)
plt.xlabel('Number of Doses')
plt.ylabel('Probability that doses will not do a satisfactory job')
for i in range(0,4):
   barl[i].set_color('r')
plt.show()_____no_output_____

In the above graph, the red region represents P(X<=3).

Let's calculate the probability that out of 10 randomly selected doses, at most 3 will not do a satisfactory job. We will use binom.cdf() for this._____no_output_____

# calculate cdf
prob_atmost3 = binom.cdf(k = 3 ,n = n, p = p)
print(prob_atmost3)0.5683679584925136

*  56.84% is the probability that at most 3 doses will not do a satisfactory job._____no_output_____**2) The New York city administration wants to buy 20 Doses for Health care workers who were COVID-Positive. They have contacted the company and requested 20 Doses. What is the probability that at least 11 doses will not do a satisfactory job out of 20 doses?**

Let's calculate the probability that at least 11 doses will not do a satisfactory job out of 20 selected.

First, we need to calculate the probability for P(X>=11). Here, X is the random variable that represents doses that will not do a satisfactory job. CDF is used to calculate the cumulative probability._____no_output_____

# declare the sample size in variable n which represents the number of doses selected randomly
n=20
# declare p which represents the probability of success, i.e., the probability that a dose will not do satisfactory job
p=0.33
# declare different number of doses selected in a numpy array
k=np.arange(0,21)
k_____no_output_____# import the required function
from scipy.stats import binom
# use the binom.pmf() function to generate the probability distribution
binomial = binom.pmf(k = k,n = n, p = p)_____no_output_____# plot the probability distribution
# We are plotting the distributions here to better visualize the calculations.
# Of course you do not 'need' to create the following visualization to answer the question above.
# You can directly use the cdf function for probability calculations.
barl=plt.bar(k,binomial)
plt.title('Binomial: n=%i , p=%.2f' % (n,p), fontsize=15)
plt.xlabel('Number of Doses')
plt.ylabel('Probability that doses will not do a satisfactory job')
for i in range(11,21):
   barl[i].set_color('r')
plt.show()_____no_output_____

In the above graph, the red region represents P(X>=11).

Let's calculate the probability that out of 20 randomly selected doses, at least 11 will not do a satisfactory job. We will use binom.cdf() for this.

P(X>=11) = 1 - P(X<=10)_____no_output_____

# calculate cdf
prob_atleast11 = 1 - binom.cdf(k = 10, n = n, p = p)
print(prob_atleast11)0.03499740152648401

* There is 3.5 % chance that at least 11 doses will not do a satisfactory job out of 20 doses_____no_output_____**3) Quality assurance team has collected 50 volunteers with the help of the city administration. Each volunteer is given a dose and the effective time of curing the covid is also monitored. Data for 50 doses is in 'doses.csv'.Based on this sample, Answer the following questions**
_____no_output_____## **Data Dictionary**:

We are using the dataset which is randomly generated and having two features

**drung_serial_number**: It represents the serial number of each dose and discrete in nature

**time_of_effect**: It represents the time taken by dose to cure the COVID. It is in hours and continuous in nature._____no_output_____### Reading the Sample Data into the DataFrame_____no_output_____

drug = pd.read_csv('doses.csv')
drug.head()_____no_output_____

## Overview of the dataset_____no_output_____

drug.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 50 entries, 0 to 49
Data columns (total 2 columns):
 #   Column              Non-Null Count  Dtype
---  ------              --------------  -----
 0   drug_serial_number  50 non-null     int64
 1   time_of_effect      50 non-null     float64
dtypes: float64(1), int64(1)
memory usage: 928.0 bytes

* As you can see from the result of the info function there no null values present in any of the columns_____no_output_____Let's calculate the mean and standard deviation of time of effect for the sample doses

_____no_output_____

# import the required function
from scipy.stats import norm
# estimate the mean and standard deviation of the time of effect for the doses
mu = drug['time_of_effect'].mean()
sigma = drug['time_of_effect'].std()
print('The estimated mean is', round(mu,2))
print('The estimated standard deviation is', round(sigma,2))The estimated mean is 13.44
The estimated standard deviation is 4.75

### Plotting the Distribution

It will help us analyze the shape of the distribution of time of effect for doses. _____no_output_____

import seaborn as sns
sns.displot(drug['time_of_effect'],kind="kde")_____no_output_____

* As you can see in the above plot, the shape of the distribution is not perfectly normal because we just have 50 observations in the sample. But, we can assume this data distribution to be normal and perform our calculations based on the normality assumption.

_____no_output_____**a) Calculate the probability that a dose will take  less than 11.5 hours**_____no_output_____

# find the cumulative probability
# norm.cdf() calculates the cumulative probability
prob = norm.cdf(11.5, mu, sigma)
print('The probability that a dose will take less than 11.5 is', round(prob, 4))The probability that a dose will take less than 11.5 is 0.3412

**b) What is the 90th percentile of the time of effect for doses?**_____no_output_____

perc_90th = norm.ppf(0.90, mu, sigma)_____no_output_____print('The 90th percentile of the time of effect for doses is', round(perc_90th, 2))The 90th percentile of the time of effect for doses is 19.52

**c) For the effectiveness test of the vaccine, based on this sample of 50 doses taken by the quality assurance team. Estimate the range in which the population mean (effective time) will lie with a 95% confidence level.**

A sample of size (n) 50 is taken which is sufficiently large (n>30). According to the central limit theorem, sampling distribution of sufficiently large distribution will follow a normal distribution.

_____no_output_____

#import the required function
from scipy.stats import t

#set the values of sample mean and sample standard deviation
x_bar, s = mu, sigma

# set the value of sample size and degrees of freedom
n = 50
k = n - 1

# construct the confidence interval
np.round(t.interval(0.95, df = k, loc = x_bar, scale = s/np.sqrt(n)), 2)_____no_output_____

* 95% of the time, the mean effective time  of doses will be between 12.09 hours and 14.79 hours._____no_output_____