# Notebook from kammerje/pynrc
Path: notebooks/Basic_Usage.ipynb

# Basic Usage

This tutorial walks through the basic usage of the `pynrc` package to calculate sensitivities and saturation limits for NIRCam in a variety of modes. _____no_output_____

# Import the usual libraries
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

# Enable inline plotting at lower left
%matplotlib inline

from IPython.display import display, Latex, clear_output_____no_output_____# jedi 0.14.1 tab completion fails; will be fixed in 0.14.2
import jedi
if jedi.__version__ == '0.14.1':
    %config Completer.use_jedi = False

# Progress bar
from tqdm.auto import trange, tqdm_____no_output_____

## Getting Started

We assume you have already installed `pynrc` as outlined in the documentation._____no_output_____

# import main module
import pynrc
from pynrc import nrc_utils

# import pysynphot instance
#from pynrc.nrc_utils import S_____no_output_____

Log messages for `pynrc` follow the same the logging functionality included in `webbpsf`. Logging levels include `DEBUG`, `INFO`, `WARN`, and `ERROR`. _____no_output_____

pynrc.setup_logging()pyNRC log messages of level INFO and above will be shown.
pyNRC log outputs will be directed to the screen.

If you get tired of the `INFO` level messages, simply type:
```python
pynrc.setup_logging('WARN', verbose=False)
```    _____no_output_____## First NIRCam Observation

The basic NIRCam object consists of all the instrument settings one would specify for a JWST observation, including filter, pupil, and coronagraphic mask selections along with detector subarray settings and ramp sampling cadence (i.e., `MULTIACCUM`).

The NIRCam class makes use of high order polynomial coefficient maps to quickly generate large numbers of monochromatic PSFs that can be convolved with arbitrary spectra and collapsed into a final broadband PSF (or dispersed with NIRCam's slitless grisms). The PSF coefficients are calculated from a series of WebbPSF monochromatic PSFs and saved to disk. These polynomial coefficients are further modifed based on focal plane position and drift in the wavefront error relative to nominal OPD mao.

There are a multitude of posssible keywords one can pass upon initialization, including certain detector settings and PSF generation parameters. If not passed initially, then defaults are assumed. The user can update these parameters at any time by either setting attributes directly (e.g., `filter`, `mask`, `pupil`, etc.) along with using the `update_detectors()` and `update_psf_coeff()` methods.

For instance,
```python
nrc = pynrc.NIRCam('F210M')
nrc.module = 'B'
nrc.update_detectors(read_mode='DEEP8', nint=10, ngroup=5)
```
is the same as:
```python
nrc = pynrc.NIRCam('F210M', module='B', read_mode='DEEP8', nint=10, ngroup=5)
```

To start, we'll set up a simple observation using the `F430M` filter. Defaults will be populated for unspecified attributes such as `module`, `pupil`, `mask`, etc.

**Check the function docstrings for more detailed information**_____no_output_____

nrc = pynrc.NIRCam(filter='F430M')
print('Filter: {};  Pupil: {};  Mask: {};  Module: {}'\
      .format(nrc.filter, nrc.pupil, nrc.mask, nrc.module))[     pynrc:INFO] Initializing SCA 485/A5
[     pynrc:INFO] RAPID readout mode selected.
[     pynrc:INFO] Setting nf=1, nd1=0, nd2=0, nd3=0.
[     pynrc:INFO] Updating PSF coeff with fov_pix=33 and oversample=4
[     pynrc:INFO] Generating and saving new PSF coefficient
Filter: F430M;  Pupil: CLEAR;  Mask: None;  Module: A

Keyword information for detector and PSF settings are stored in the `det_info` and `psf_info` dictionaries. These cannot be modified directly, but instead are updated via the `update_detectors()` and `update_psf_coeff()` methods._____no_output_____

print('Detector Info Keywords:')
print(nrc.det_info)
print('')
print('PSF Info Keywords:')
print(nrc.psf_info)Detector Info Keywords:
{'wind_mode': 'FULL', 'nout': 4, 'xpix': 2048, 'ypix': 2048, 'x0': 0, 'y0': 0, 'read_mode': 'RAPID', 'nint': 1, 'ngroup': 1, 'nf': 1, 'nd1': 0, 'nd2': 0, 'nd3': 0}

PSF Info Keywords:
{'fov_pix': 33, 'oversample': 4, 'quick': False, 'nproc': None, 'offset_r': 0, 'offset_theta': 0, 'tel_pupil': None, 'save': True, 'force': False, 'use_legendre': True, 'include_si_wfe': True, 'opd': ('OPD_RevW_ote_for_NIRCam_requirements.fits', 0), 'jitter': 'gaussian', 'jitter_sigma': 0.007}

PSF coefficient information is stored in the `psf_coeff` attribute. This data is accessed by many of the NIRCam class functions to generate PSFs with arbitrary wavelength weights, such as the `gen_psf()` function._____no_output_____

# Demonstrate the color difference of the PSF for different spectral types, same magnitude
sp_M0V = pynrc.stellar_spectrum('M0V', 10, 'vegamag', nrc.bandpass)
sp_A0V = pynrc.stellar_spectrum('A0V', 10, 'vegamag', nrc.bandpass)

# Generate oversampled PSFs (counts/sec)
_, psf_M0V = nrc.gen_psf(sp_M0V, return_oversample=True)
_, psf_A0V = nrc.gen_psf(sp_A0V, return_oversample=True)

fig, axes = plt.subplots(1,3, figsize=(12,4))

axes[0].imshow(psf_M0V**0.5)
axes[0].set_title('M0V PSF ({})'.format(nrc.filter))
axes[1].imshow(psf_A0V**0.5)
axes[1].set_title('A0V PSF ({})'.format(nrc.filter))

diff = psf_M0V - psf_A0V

minmax = np.abs(diff).max() / 2
axes[2].imshow(diff, cmap='RdBu', vmin=-minmax, vmax=minmax)
axes[2].set_title('Difference')

fig.tight_layout()_____no_output_____

Bandpass information is stored in the `bandpass` attribute and can be plotted with the convenience function `plot_bandpass()`._____no_output_____

nrc.plot_bandpass()_____no_output_____

## 1. Saturation Limits

One of the most basic functions is to determine the saturation limit of a CDS observation, so let's try this for the current filter selection. Generally, saturation is considered to be 80% of the full well._____no_output_____

# Turn off those pesky informational texts
pynrc.setup_logging('WARN', verbose=False)

# Configure the observation for CDS frames (ngroup=2)
# Print out frame and ramp information using verbose=True
nrc.update_detectors(ngroup=2, verbose=True)New Ramp Settings:
  read_mode :    RAPID
  nf        :        1
  nd2       :        0
  ngroup    :        2
  nint      :        1
New Detector Settings
  wind_mode :     FULL
  xpix      :     2048
  ypix      :     2048
  x0        :        0
  y0        :        0
New Ramp Times
  t_group   :   10.737
  t_frame   :   10.737
  t_int     :   21.474
  t_int_tot1 :   32.210
  t_int_tot2 :    0.000
  t_exp     :   21.474
  t_acq     :   32.216

The `sat_limits()` function returns a dictionary of results. There's the option in include a Pysynphot spectrum, but if None is specificed then it defaults to a G2V star._____no_output_____

# Set verbose=True to print results in a user-friendly manner
sat_lims = nrc.sat_limits(verbose=True)

# Dictionary information
print("\nDictionary Info:", sat_lims)F430M Saturation Limit assuming G2V source (point source): 12.18 vegamag
F430M Saturation Limit assuming G2V source (extended): 8.50 vegamag/arcsec^2

Dictionary Info: ({'satlim': 12.182163337498555, 'units': 'vegamag', 'bp_lim': 'F430M', 'Spectrum': 'G2V'}, {'satlim': 8.502480823570135, 'units': 'vegamag/arcsec^2', 'bp_lim': 'F430M', 'Spectrum': 'G2V'})

By default, the function `sat_limits()` uses a G2V stellar spectrum, but any arbritrary spectrum can be passed via the `sp` keyword. In addition, using the `bp_lim` keyword, you can use spectral information to determine the brightness in some other bandpass that saturates the source within the NIRCam filter. _____no_output_____

# Spectrum of an M0V star (not normalized)
sp_M0V = pynrc.stellar_spectrum('M0V')
# 2MASS Ks Bandpass
bp_k = pynrc.bp_2mass('K')

sat_lims = nrc.sat_limits(sp=sp_M0V, bp_lim=bp_k, verbose=True)Ks-Band Saturation Limit for F430M assuming M0V source (point source): 12.37 vegamag
Ks-Band Saturation Limit for F430M assuming M0V source (extended): 8.69 vegamag/arcsec^2

Now, let's get the same saturation limit assuming a 128x128 subarray._____no_output_____

nrc.update_detectors(wind_mode='WINDOW', xpix=128, ypix=128)
sat_lims = nrc.sat_limits(sp=sp_M0V, bp_lim=bp_k, verbose=True)Ks-Band Saturation Limit for F430M assuming M0V source (point source): 7.94 vegamag
Ks-Band Saturation Limit for F430M assuming M0V source (extended): 4.26 vegamag/arcsec^2

You can also use the `saturation_levels()` function to generate an image of a point source indicating the fractional well fill level._____no_output_____

# Spectum of A0V star with Ks = 8 mag
sp = pynrc.stellar_spectrum('M0V', 8, 'vegamag', bp_k)
sat_levels = nrc.saturation_levels(sp, full_size=False, ngroup=nrc.det_info['ngroup'])

print('Max Well Fraction: {:.2f}'.format(sat_levels.max()))Max Well Fraction: 0.76
# Plot the well fill levels for each pixel
fig, axes = plt.subplots(1,2, figsize=(12,5))

for i,ax in enumerate(axes):
    extent = 0.5 * nrc.psf_info['fov_pix'] * np.array([-1,1,-1,1])
    if i==0:
        cax = ax.imshow(sat_levels, extent=extent, vmin=0, vmax=1)
    else:
        norm = matplotlib.colors.LogNorm(vmin=0.001, vmax=1)
        cax = ax.imshow(sat_levels, extent=extent, norm=norm)
    ax.set_xlabel('Pixels')
    ax.set_ylabel('Pixels')
    ax.set_title('Well Fraction in {} of $K_s = 5$ M0V star'.format(nrc.filter))

    cbar = fig.colorbar(cax, ax=ax)
    cbar.set_label('Well Fill Fraction')

    ax.tick_params(axis='both', color='white', which='both')
    for k in ax.spines.keys():
        ax.spines[k].set_color('white')

fig.tight_layout()_____no_output_____

Information for slitless grism observations show wavelength-dependent results._____no_output_____

nrc = pynrc.NIRCam('F444W', pupil='GRISM0', ngroup=2, wind_mode='STRIPE', ypix=128)
sat_lims = nrc.sat_limits(sp=sp_M0V, bp_lim=bp_k, verbose=True)Ks-Band Saturation Limit for F444W assuming M0V source:
   Wave   Sat Limit (vegamag)
--------- -------------------
     3.90                4.50
     4.00                4.49
     4.10                4.38
     4.20                4.26
     4.30                4.11
     4.40                3.87
     4.50                3.69
     4.60                3.53
     4.70                3.35
     4.80                3.18
     4.90                2.97
     5.00                2.38

## 2. Sensitivity Limits

Similarly, we can determine sensitivity limits of point sources (and extended sources) for the defined instrument configuration. By default, the `sensitivity()` function uses a flat spectrum. In this case, let's find the sensitivities NIRCam can reach in a single ~1000sec integration with the F430M filter. Noise values will depend on the exact `MULTIACCUM` settings._____no_output_____

nrc = pynrc.NIRCam('F430M')
nrc.update_detectors(read_mode='MEDIUM8', ngroup=10)

# The multiaccum_times attribute describes the various timing information
print(nrc.multiaccum_times){'t_frame': 10.73677, 't_group': 107.3677, 't_int': 1052.20346, 't_exp': 1052.20346, 't_acq': 1062.94547, 't_int_tot1': 1062.94023, 't_int_tot2': 0.0}
sens = nrc.sensitivity(nsig=5, units='vegamag', verbose=True)Point Source Sensitivity (5-sigma): 23.32 vegamag
Surface Brightness Sensitivity (5-sigma): 21.22 vegamag/arcsec^2

The sensitivity function also includes a keyword `forwardSNR`, which allows the user to pass a normalized spectrum and estimate the SNR For some extraction aperture._____no_output_____

sp = pynrc.stellar_spectrum('M0V', 20, 'vegamag', nrc.bandpass)
snr = nrc.sensitivity(sp=sp, forwardSNR=True, units='vegamag', verbose=True)Point Source SNR (20.00 vegamag): 75.57 sigma
Surface Brightness SNR (20.00 vegamag/arcsec^2): 14.47 sigma

## 3. Ramp Optimization

Armed with these two basic functions, we can attempt to determine the best instrument settings to optimize for SNR and efficiency. In these types of optimizations, we must consider observational constraints such as saturation levels, SNR requirements, and limits on acquisition time.

**Note**: The reported acquisition times do not include obsevatory and instrument-level overheads, such as slew times, filter changes, script compilations, etc. It only includes detector readout times (including reset frames and Fast Row Resets).

For instance, we want to observe an M-Dwarf (K=18 mag) in the F430M filter. What is the most efficient configuration to obtain an SNR of 100?_____no_output_____

# Setup observation
nrc = pynrc.NIRCam('F430M', wind_mode='WINDOW', xpix=160, ypix=160)

# Spectrum of an M2V star
bp_k = pynrc.bp_2mass('K')
sp_M0V = pynrc.stellar_spectrum('M2V', 18, 'vegamag', bp_k)_____no_output_____# Run optimizer. Result is a sorted by efficiency.
tbl = nrc.ramp_optimize(sp_M0V, snr_goal=100, ng_min=5, nint_min=10, verbose=True)BRIGHT1
BRIGHT2
DEEP2
DEEP8
MEDIUM2
MEDIUM8
RAPID
SHALLOW2
SHALLOW4
 Pattern   NGRP NINT   t_int     t_exp     t_acq     SNR      Well     eff
---------- ---- ---- --------- --------- --------- -------- -------- --------
DEEP8         5   10     24.52    245.20    248.04    124.6    0.006    7.909
MEDIUM8       6   12     16.16    193.93    197.34    100.3    0.004    7.138
MEDIUM8       6   13     16.16    210.09    213.78    104.4    0.004    7.138
MEDIUM2       8   10     20.06    200.62    203.46     98.7    0.005    6.918
MEDIUM2       8   11     20.06    220.68    223.80    103.5    0.005    6.918
SHALLOW4     10   15     13.65    204.80    209.06     99.5    0.003    6.880
SHALLOW4     10   16     13.65    218.45    222.99    102.7    0.003    6.880
DEEP2         5   10     22.85    228.48    231.32    103.0    0.005    6.770
MEDIUM8       5   18     13.37    240.74    245.85    104.4    0.003    6.658
SHALLOW4      9   20     12.26    245.20    250.88    104.3    0.003    6.587
...         ...  ...       ...       ...       ...      ...      ...      ...
RAPID        10  512      2.79   1426.64   1571.94    101.2    0.001    2.553
RAPID        10  513      2.79   1429.42   1575.01    101.3    0.001    2.553
RAPID        10  514      2.79   1432.21   1578.08    101.4    0.001    2.553
RAPID        10  515      2.79   1435.00   1581.15    101.5    0.001    2.553
RAPID        10  516      2.79   1437.78   1584.22    101.6    0.001    2.553
RAPID        10  517      2.79   1440.57   1587.29    101.7    0.001    2.553
RAPID        10  518      2.79   1443.36   1590.36    101.8    0.001    2.553
RAPID        10  519      2.79   1446.14   1593.43    101.9    0.001    2.553
RAPID        10  520      2.79   1448.93   1596.50    102.0    0.001    2.553
RAPID         9  696      2.51   1745.40   1942.93    102.0    0.001    2.314
RAPID         7 1416      1.95   2761.88   3163.74    102.0    0.000    1.813
Length = 50 rows

For a slightly more complicated scenario, consider an additional foreground source. In this scenario, the F0V star will saturate much more quickly compared to the fainter M2V, so it limits which ramp settings we may want to use (assuming we want unsaturated frames, which isn't always necessarily true)._____no_output_____

sp_F0V = pynrc.stellar_spectrum('F0V', 10, 'vegamag', bp_k)
tbl = nrc.ramp_optimize(sp_M0V, sp_bright=sp_F0V,
                        snr_goal=100, ng_min=5, nint_min=10, verbose=True)BRIGHT1
BRIGHT2
DEEP2
DEEP8
MEDIUM2
MEDIUM8
RAPID
SHALLOW2
SHALLOW4
 Pattern   NGRP NINT   t_int     t_exp     t_acq     SNR      Well     eff
---------- ---- ---- --------- --------- --------- -------- -------- --------
RAPID        10  499      2.79   1390.41   1532.03     99.9    0.779    2.553
RAPID        10  500      2.79   1393.20   1535.10    100.0    0.779    2.553
RAPID        10  501      2.79   1395.99   1538.17    100.1    0.779    2.553
RAPID        10  502      2.79   1398.77   1541.24    100.2    0.779    2.553
RAPID        10  503      2.79   1401.56   1544.31    100.3    0.779    2.553
RAPID        10  504      2.79   1404.35   1547.38    100.4    0.779    2.553
RAPID        10  505      2.79   1407.13   1550.45    100.5    0.779    2.553
RAPID        10  506      2.79   1409.92   1553.52    100.6    0.779    2.553
RAPID        10  507      2.79   1412.70   1556.59    100.7    0.779    2.553
RAPID        10  508      2.79   1415.49   1559.66    100.8    0.779    2.553
...         ...  ...       ...       ...       ...      ...      ...      ...
BRIGHT1       5  959      2.51   2404.94   2677.11    101.5    0.701    1.962
BRIGHT1       5  960      2.51   2407.45   2679.90    101.6    0.701    1.962
BRIGHT1       5  961      2.51   2409.96   2682.69    101.6    0.701    1.962
BRIGHT1       5  962      2.51   2412.47   2685.48    101.7    0.701    1.962
BRIGHT1       5  963      2.51   2414.97   2688.27    101.7    0.701    1.962
BRIGHT1       5  964      2.51   2417.48   2691.06    101.8    0.701    1.962
BRIGHT1       5  965      2.51   2419.99   2693.86    101.8    0.701    1.962
BRIGHT1       5  966      2.51   2422.50   2696.65    101.9    0.701    1.962
BRIGHT1       5  967      2.51   2425.00   2699.44    102.0    0.701    1.962
BRIGHT1       5  968      2.51   2427.51   2702.23    102.0    0.701    1.962
RAPID         7 1416      1.95   2761.88   3163.74    102.0    0.545    1.813
Length = 86 rows

If there are no objections to saturating the bright source, then we can set the `well_frac_max` parameter to something like 5 times the hard saturation limit. This allows for more efficient exposure settings._____no_output_____

tbl = nrc.ramp_optimize(sp_M0V, sp_bright=sp_F0V, well_frac_max=5,
                        snr_goal=100, ng_min=5, nint_min=10, verbose=True)BRIGHT1
BRIGHT2
DEEP2
DEEP8
MEDIUM2
MEDIUM8
RAPID
SHALLOW2
SHALLOW4
 Pattern   NGRP NINT   t_int     t_exp     t_acq     SNR      Well     eff
---------- ---- ---- --------- --------- --------- -------- -------- --------
MEDIUM8       6   12     16.16    193.93    197.34    100.3    4.516    7.138
MEDIUM8       6   13     16.16    210.09    213.78    104.4    4.516    7.138
SHALLOW4     10   15     13.65    204.80    209.06     99.5    3.816    6.880
SHALLOW4     10   16     13.65    218.45    222.99    102.7    3.816    6.880
MEDIUM8       5   18     13.37    240.74    245.85    104.4    3.738    6.658
SHALLOW4      9   20     12.26    245.20    250.88    104.3    3.426    6.587
MEDIUM2       7   13     17.28    224.58    228.27     98.4    4.828    6.510
MEDIUM2       7   14     17.28    241.86    245.83    102.1    4.828    6.510
SHALLOW2     10   19     13.10    248.83    254.22    100.3    3.660    6.287
SHALLOW2     10   20     13.10    261.92    267.60    102.9    3.660    6.287
...         ...  ...       ...       ...       ...      ...      ...      ...
RAPID        10  512      2.79   1426.64   1571.94    101.2    0.779    2.553
RAPID        10  513      2.79   1429.42   1575.01    101.3    0.779    2.553
RAPID        10  514      2.79   1432.21   1578.08    101.4    0.779    2.553
RAPID        10  515      2.79   1435.00   1581.15    101.5    0.779    2.553
RAPID        10  516      2.79   1437.78   1584.22    101.6    0.779    2.553
RAPID        10  517      2.79   1440.57   1587.29    101.7    0.779    2.553
RAPID        10  518      2.79   1443.36   1590.36    101.8    0.779    2.553
RAPID        10  519      2.79   1446.14   1593.43    101.9    0.779    2.553
RAPID        10  520      2.79   1448.93   1596.50    102.0    0.779    2.553
RAPID         9  696      2.51   1745.40   1942.93    102.0    0.701    2.314
RAPID         7 1416      1.95   2761.88   3163.74    102.0    0.545    1.813
Length = 49 rows


# Notebook from kostasx/lab-workshops
Path: python-for-humanists/PartIII/PythonTextComplete.ipynb

# Working with Text in Python_____no_output_____### Examples of text analysis

* Word frequency
* Word associations (collocates)
* Tracking words over time

### [Quantitative Analysis of Culture Using Millions of Digitized Books](https://science.sciencemag.org/content/331/6014/176)
* Using a corpus of 5 million English language books from 1800-200, all which were digitized by Google books.

![](flu.png)
![](greatwar.png)
![](evolution.png)

### Yale Daily News Corpus
* Digitized YDN from 1878-1996.
* Looking at terms around coeducation.
    * Yale College became co-ed in 1969.
    * Subset of corpus looking at 1959-1979.

![](coed.png)
![](man_woman.png)_____no_output_____### Types of text data

* The easiest method for working with textual data in python is to use plain-text files.
    * Common plain-text files include:
        * .txt
        * .xml
        * .csv
        * .json
        * .html
        * .tsv
        * more...
* Word Docs & Excel spreadsheets are not plain-text files.

### Creating raw data
* The first step in most text based research will be creating text files. These files can come from a variety of sources:
    * Archival documents
    * Government documents
    * Web pages
    * Newspapers & periodicals
    * Twiiter
* It's common to start with a PDF or other scan of a physical work.
* **Optical Character Recognition** (OCR) is the process of extracting text from an image file or PDF.
    * OCR can create plain-text files that can be used in many types of analysis.
* This workshop will not cover OCR, but we can recommend software for OCR:
    * ABBYY FInereader
    * Adobe Acrobat
    * Python-tesseract_____no_output_____## Preparing text files
### Opening text files

* There are a few methods for opening files in python. We wil focus on using the built-in python function `open()`.
    * Other libraries, like `pandas` and `NLTK` have their own functions for reading text files.
* `dir()` will list every method or function available to call upon an object.
* `.read()` will read the contents of a text file. We can print the reasult or save the read text to another variable.
* `.readlines()` will split our file into a list. Each item in this list is one line of our text file. _____no_output_____

infile = open('./rawText/obama_speeches_000.txt','r')
print(dir(infile))_____no_output_____help(open)_____no_output_____title = infile.readline()
date = infile.readline()
text = infile.read()
print(title, date, text)_____no_output_____

### Creating Sub-Strings
* Python lets us slice strings based on the index of each character
    * We use square brackets `[:]` to designate the dimensions of a slice. _____no_output_____

title = title[8:-3]
date = date[7:-3]

print(title,date)_____no_output_____

### Cleaning strings
* We can slice our `title` and `date` varibales to remove erroneous characters.
* The `replace()` function will remove spaces from the speech title and reaplce them with an underscore.
    * It's best practice not to have spaces in a filename._____no_output_____

title = title.replace(' ','_')

print(title,date)_____no_output_____

### Writing files

* The `open()` function is also used to write new files.
* The new filename and path is set in the first arguement.
    * If the file does not exist, python will create it.
    * If the file akready exists, python will overwrite the file._____no_output_____

outfile = open('./cleanText/' + date + '_' + title + '.txt', 'w')
outfile.write(text)_____no_output_____

### Cleaning files in batches

* The power of python is more apparent when working with large amounts of data.
* The code above works to clean one text file in a collection.
    * By adding this code to a `for` loop, python can batch process our text files.
* `glob.glob` creates a list of filenames based on a pattern.
* The `.close()` method will close the file before moving on to the next file in the loop._____no_output_____

from datetime import datetime
import glob

filenames = glob.glob('./rawText/*.txt')
for filename in filenames:
    with open(filename, 'r') as infile:
        title = infile.readline().replace(' ','_')[8:-3]
        date = infile.readline()[7:-3]
        text = infile.read()
        outfile = open('./cleanText/' + title + '.txt', 'w')
        outfile.write(text)
        outfile.close()_____no_output_____

## Working with XML
### Converting XML to TXT using BeautifulSoup
* The `BeautifulSoup` library contains functions for working with XML & HTML files.
    * `BeautifulSoup` is commonly used when web-scraping in python to parse HTML.
* Using `with open() as :` automatically closes the file after the indented lines are complete. Meaning the `.close()` method is not necessary.
* BeautifulSoup will read our file object and return parsed XML._____no_output_____

from bs4 import BeautifulSoup

with open('./NYHT_raw/1322266375.xml','r') as infile:
    soup = BeautifulSoup(infile.read(), 'xml')
    print(type(soup))
    print(dir(soup))
    print(soup)_____no_output_____

### Find XML Elements by Name
* The `soup` variable conatains the XML contents of the file
* The `.find()` funciton searches the XML for an element name that matches the arguement string.
    * `find()` only matches the first instance of the element.
    * Use `find_all()` to return a list of *all* matching elements_____no_output_____

date = soup.find('NumericPubDate').text
recordid = soup.find('RecordID').text
print(date, recordid)

new_file = date + '_' + recordid + '.txt'
print(new_file)_____no_output_____text = soup.find('FullText').text
print(text)_____no_output_____

### Writing a new file
* Python can write our new file to a new directory with our clean data._____no_output_____

with open('./NYHT_clean/' + new_file, 'w') as outfile:
    outfile.write(text)_____no_output_____

### Batch convert XML files to TXT_____no_output_____

import glob

filenames = glob.glob('./NYHT_raw/*.xml')

for filename in filenames:
    with open(filename,'r') as infile:
        soup = BeautifulSoup(infile.read(), 'xml')
        date = soup.find('NumericPubDate').text
        text = soup.find('FullText').text
        recordid = soup.find('RecordID').text
        with open('./NYHT_clean/' + date + '_' + recordid + '.txt', 'w') as outfile:
            outfile.write(text)_____no_output_____

## Non-Latin Text

utf-8 encoding allows us to work with scripts outside of the latin alphabet. See the full Unicode character list here: [all unicode characters](https://www.utf8-chartable.de/)_____no_output_____

rus = open('./rawText/anna_k.txt', encoding='utf-8')
print(rus.read())_____no_output_____

## Cleaning the Text_____no_output_____### Built-in string functions
* Python includes many built-in [string methods](https://docs.python.org/3/library/stdtypes.html#string-methods) helpful in editing text data.
* In order to compare words, it's best to standardize them into once case.
    * The `lower()` method will convert all characters in a string to lowercase.
    * `strip()` by default will remove leading and trailing whitespace (spaces & tabs)._____no_output_____

infile = open('./NYHT_clean/19520926_1322266375.txt', 'r')

text = infile.read().lower()
print(text)

text = text.strip()
print(text)_____no_output_____

### Create a custom list of special charcters

* The `string` library inclides pre-build strings containing letters, numbers, and punctuation.
* The `list()` function breaks a string into a list of characters.
* The `.replace()` function will replace one string with another. In this case, we replace each special charcter in our list with an empty string. This is equivalent to deleting the special characters._____no_output_____

import string

special = list(string.punctuation)
print(special)

for x in special:
    text = text.replace(x,'')
print(text)_____no_output_____

### Removing stopwords
* Stopwords is a term given to any words in a corpus that should be ignored in the analysis.
* Generally, stopwords are common words in a language that do not give insight into the text.
    * This includes words like: *the, and, or, a, an, if,* and many more.
* Libraries like `NLTK` will provide built-in stopwords
    * The code below manually creates a dictionary of stopwords_____no_output_____

stopwords = {'the', 'do', 'in', 'can', "aren't", 'needn', "mightn't", 'it', 'you', 'through', 'her', 'against', 'because', 'into', "hasn't", 'down', 'why', "don't", 'again', 'than', "you're", 'themselves', 'does', 'weren', 'this', 'll', 'don', 'they', "didn't", 'as', 'there', 'too', 'off', "haven't", "mustn't", 'shan', 'all', 'such', 're', 'have', 'that', 'over', "it's", 'most', 'those', 'and', 'yours', "couldn't", 'under', 'didn', 'so', 'myself', 'itself', 'herself', "that'll", 'theirs', 'should', 'how', 'am', "you've", 'be', 'will', 'd', "needn't", 'i', 'is', 'couldn', 'or', 'himself', 'other', 'your', 'below', 'our', 'by', 'before', 'between', 'just', 'hasn', 'ain', "hadn't", 'not', 'did', 'his', 'shouldn', "wasn't", 'to', "isn't", 'wouldn', 'haven', 'wasn', 'ma', 'some', 'a', 'their', 'them', 'up', 'few', 'yourselves', 'doesn', 'm', 't', 'at', 'during', 'then', 'when', 'me', 'while', 'aren', 'more', 'isn', 'we', 'above', 'after', 'where', 'until', 'whom', 'o', "shan't", 'once', 'being', 'him', 'hers', 'an', 's', 'but', 'further', 'ours', 'both', "you'll", 'from', 'own', 'nor', 'who', 'been', "wouldn't", 'very', "won't", 'on', 'same', 'no', 'doing', 'what', 'she', 'mightn', 'ourselves', 'if', "should've", 'hadn', 'was', 'of', "you'd", 'are', "she's", 'yourself', 'out', 'he', 'mustn', "doesn't", 'which', 'having', "weren't", 'won', 'for', 'its', 'had', 'has', 'about', 'here', 'only', 'y', 'now', "shouldn't", 'my', 'each', 'were', 'with', 'these', 've', 'any'}
print(stopwords)

for stop in stopwords:
    text = text.replace(' '+stop+' ',' ')
print(text)_____no_output_____

### Creating a word list

* `split()` will split our string into a list of strings. This method splits the string based on the arguement of the method. _____no_output_____

word_list = text.split()

print(word_list)_____no_output_____

### Cleaning individual words
* The methods `.isalpha()`, `.isnumeric()`, `.isalnum()`, `.isascii()` return True if *all* characters in the string match the criteria.
    * `.isalpha()` returns True if there is at least one character & all characters are letters (latin alphabet).
    * `.isnumeric()` returns True if there is at least one character & all characters are digits.
    * `.isalnum()` returns True if there is at least one character & all characters are letters or digits.
    * `.isascii()` returns True if all characters are ascii letters, digits, or special characters. See [ASCII Table](https://www.cs.cmu.edu/~pattis/15-1XX/common/handouts/ascii.html)._____no_output_____

w = ['abc','123','!?-','a2?']

for x in w:
    print(x, x.isalpha(), x.isnumeric(), x.isalnum(), x.isascii()) _____no_output_____final_words = []

for word in word_list:
    if word.isalpha() == True:
        final_words.append(word)

print(final_words)_____no_output_____

### A little fun with ASCII
* Run the cell below to make some ASCII art.
* We are using an image calle 'logo.jog'. Change the image name or path to convert a differnt image into "art".
* [Source](https://www.geeksforgeeks.org/converting-image-ascii-image-python/)
![Handome Dan Yale logo](./logo.jpg)_____no_output_____

import os

cmd = 'python ./ascii_art.py --file logo.jpg --cols 120'
os.system(cmd)
open('out.txt', 'r').read()_____no_output_____

## Counting Words

### Word frequency
* The `Counter` library is an easy method for counting unique items in a list.
    * We can use counter on our `final_words` list to find word frequencies._____no_output_____

from collections import Counter

pets = ["dog", "cat", "bird", "gnu", "dog", "dog", "cat"]

pet_counts = Counter(pets)
print(pet_counts)_____no_output_____word_counts = Counter(final_words)

print(word_counts)
print(len(word_counts))
print(word_counts.most_common(10))_____no_output_____

### Saving our results
* Our `word_dict` can be saved as a JSON file.
    * python dictioanries follow the JSON structure._____no_output_____

import json

word_dict = dict(word_counts)
print(word_dict)

with open('./output/19511216_1322285142.json', 'w') as outfile:
    json.dump(word_dict, outfile)_____no_output_____

* We can also save our results as a CSV file.
* A CSV file can be used by many programs like excel. _____no_output_____

import csv

with open('./output/19511216_1322285142.csv', 'w') as f:
    w = csv.writer(f)
    w.writerow(['word','count'])
    for word in word_dict.keys():
        row = [word, word_dict[word]]
        print(row)
        w.writerow(row)_____no_output_____

## Working at Scale

### Word frequencies over time

* Our goal is to end up with one file containing word frequencies grouped by the article date._____no_output_____

stopwords = {'the', 'do', 'in', 'can', "aren't", 'needn', "mightn't", 'it', 'you', 'through', 'her', 'against', 'because', 'into', "hasn't", 'down', 'why', "don't", 'again', 'than', "you're", 'themselves', 'does', 'weren', 'this', 'll', 'don', 'they', "didn't", 'as', 'there', 'too', 'off', "haven't", "mustn't", 'shan', 'all', 'such', 're', 'have', 'that', 'over', "it's", 'most', 'those', 'and', 'yours', "couldn't", 'under', 'didn', 'so', 'myself', 'itself', 'herself', "that'll", 'theirs', 'should', 'how', 'am', "you've", 'be', 'will', 'd', "needn't", 'i', 'is', 'couldn', 'or', 'himself', 'other', 'your', 'below', 'our', 'by', 'before', 'between', 'just', 'hasn', 'ain', "hadn't", 'not', 'did', 'his', 'shouldn', "wasn't", 'to', "isn't", 'wouldn', 'haven', 'wasn', 'ma', 'some', 'a', 'their', 'them', 'up', 'few', 'yourselves', 'doesn', 'm', 't', 'at', 'during', 'then', 'when', 'me', 'while', 'aren', 'more', 'isn', 'we', 'above', 'after', 'where', 'until', 'whom', 'o', "shan't", 'once', 'being', 'him', 'hers', 'an', 's', 'but', 'further', 'ours', 'both', "you'll", 'from', 'own', 'nor', 'who', 'been', "wouldn't", 'very', "won't", 'on', 'same', 'no', 'doing', 'what', 'she', 'mightn', 'ourselves', 'if', "should've", 'hadn', 'was', 'of', "you'd", 'are', "she's", 'yourself', 'out', 'he', 'mustn', "doesn't", 'which', 'having', "weren't", 'won', 'for', 'its', 'had', 'has', 'about', 'here', 'only', 'y', 'now', "shouldn't", 'my', 'each', 'were', 'with', 'these', 've', 'any'}
punct = {'!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~'}
digits = ['0','1','2','3','4','5','6','7','8','9']_____no_output_____import csv, json, glob
from collections import Counter

cbd = open('./output/counts_by_date.csv', 'w')
w = csv.writer(cbd)
w.writerow(['word','count','date'])
for filename in glob.glob('./NYHT_clean/*.txt'):
    with open(filename, 'r') as infile:
        text = infile.read().lower()
        for p in punct:
            text = text.replace(p,'')
        for d in digits:
            text = text.replace(d,'')
        for stop in stopwords:
            text = text.replace(' '+stop+' ',' ')
        words_by_file = Counter(text.split())
        word_dict = dict(words_by_file)
        for key in word_dict.keys():
            row = [key, word_dict[key], filename[13:17]]
            print(row)
            w.writerow(row)
cbd.close()_____no_output_____

### Total word counts
* Now let's work with all of our newspaper articles.
* We can use much of the same code as above, but we need to structure it in a loop to work on each file.
* Our goal is to produce a single file, with the total count of each word across all documents._____no_output_____

import csv, json, glob
from collections import Counter

total_words = Counter()

for filename in glob.glob('./NYHT_clean/*.txt'):
    with open(filename, 'r') as infile:
        text = infile.read().lower()
        for stop in stopwords:
            text = text.replace(' '+stop+' ',' ')
        for p in punct:
            text = text.replace(p,'')
        for d in digits:
            text = text.replace(d,'')
        total_words.update(text.split())

print(total_words)

with open('./output/total_counts.csv', 'w') as outfile:
    w = csv.writer(outfile)
    w.writerow(['word','count'])
    word_dict = dict(total_words)
    for key in word_dict.keys():
        row = [key, word_dict[key]]
        print(row)
        w.writerow(row)_____no_output_____import csv, json, glob, pickle
from collections import Counter

c = Counter()

for filename in glob.glob('./NYHT_clean/*.txt'):
    with open(filename, 'r') as infile:
        text = infile.read().lower()
        for stop in stopwords:
            text = text.replace(' '+stop+' ',' ')
        for p in punct:
            text = text.replace(p,'')
        for d in digits:
            text = text.replace(d,'')
        c.update(text.split())
c = dict(c)
print(c)_____no_output_____with open('NYHT.pickle', 'wb') as outfile:
    pickle.dump(c,outfile)_____no_output_____with open('NYHT.pickle', 'rb') as infile:
    counts = pickle.load(infile)
print(counts)_____no_output_____

## Plotting Data

### Plotting basics
* `matplotlib` is a frequently used graphing library for python.
* The most basic plot requires data for the x and y axis.
    * `plt.plot()` takes two or more arguements. The data for the two axis is the minimal requirement.
    * The axis labels and title are set using methods `.xlabel()`, `.ylabel()`, and `.title()`.
* Many other style and graph options are possible with `matplotlib`. See the full documentation: [Matplotlib User Guide](https://matplotlib.org/3.1.0/users/index.html)_____no_output_____

%matplotlib inline
import matplotlib.pyplot as plt

year = [91,92,93,94,95,96,97,98,99]
count = [1,2,3,4,5,6,7,8,9]

plt.plot(year, count)
plt.xlabel('Year')
plt.ylabel('Word Count')
plt.title('Frequency of Word x')_____no_output_____

### Reading data using pandas_____no_output_____

import pandas as pd

df = pd.read_csv('./output/counts_by_date.csv', index_col='date')
totals = df.groupby('word').sum().sort_values(by=['count'],ascending=False)
print(totals,df)_____no_output_____mrs = df[df.word == 'mrs'].groupby('date').sum()
mrs.plot.bar()_____no_output_____terms = totals.loc[['mrs','mr','dr','prof']]
terms.plot.bar()_____no_output_____totals.plot.hist(figsize=(20,5), bins=[x for x in range(0,200,10)])_____no_output_____

### Plotting word frequencies
* There are many ways to plot word frequencies. The code below loads in a JSON file with word frequencies of certain terms from Yale Daily News from 1959-1979._____no_output_____

import json

data = open('YDN_counts.json','r')
count = json.load(data)
type(count)_____no_output_____x = list(count['(gender|sex)'].keys())
y = list(count['(gender|sex)'].values())
#print(x,y)

plt.figure(figsize=(20,5))
plt.plot(x,y,'g--')
plt.xlabel('Year')
plt.ylabel('Frequency')
plt.title('Frequency of "gender" and "sex" in Yale Daily News (1959-1979)')
plt.savefig('YDN_gender.png', dpi=300)_____no_output_____x2 = list(count['(coed|coeducation)'].keys())
y2 = list(count['(coed|coeducation)'].values())

plt.figure(figsize=(20,5))
plt.plot(x,y,'g--', label = "SEX")
plt.plot(x2,y2,'b', label = "COED")
plt.legend(loc='upper left')
plt.xlabel('Year')
plt.ylabel('Frequency')
plt.title('Frequency of Words in Yale Daily News (1959-1979)')
plt.savefig('YDN_terms.png', dpi=300)_____no_output_____

## Text Analysis Tools

### Working with Voyant
* Now that we have text files, we can take our data to other software.
* [Voyant](https://voyant-tools.org/) is a free, online tool for text analysis._____no_output_____## Working with Dates

* The `datetime` library in python adds functions for working with dates.
* It can convert strings to ISO dates and vice-versa:
    * `strptime()` converts a string date into an ISO date object
    * `strftime()` converts a ISO date object to a string_____no_output_____

from datetime import datetime

date = 'November 8, 2008'

isoDate = datetime.strptime(date, '%B %d, %Y')
print(isoDate, type(isoDate))

strDate = isoDate.strftime('%Y%m%d')
print(isoDate, strDate)_____no_output_____


# Notebook from Jaycob-jh/course-content
Path: tutorials/W1D1_ModelTypes/W1D1_Tutorial1.ipynb

_____no_output_____
#  Neuromatch Academy: Week 1, Day 1, Tutorial 1
# Model Types: "What" models
__Content creators:__ Matt Laporte, Byron Galbraith, Konrad Kording

__Content reviewers:__ Dalin Guo, Aishwarya Balwani, Madineh Sarvestani, Maryam Vaziri-Pashkam, Michael Waskom

We would like to acknowledge [Steinmetz _et al._ (2019)](https://www.nature.com/articles/s41586-019-1787-x) for sharing their data, a subset of which is used here.
_____no_output________
# Tutorial Objectives
This is tutorial 1 of a 3-part series on different flavors of models used to understand neural data. In this tutorial we will explore 'What' models, used to describe the data. To understand what our data looks like, we will visualize it in different ways. Then we will compare it to simple mathematical models. Specifically, we will:

- Load a dataset with spiking activity from hundreds of neurons and understand how it is organized
- Make plots to visualize characteristics of the spiking activity across the population
- Compute the distribution of "inter-spike intervals" (ISIs) for a single neuron
- Consider several formal models of this distribution's shape and fit them to the data "by hand"_____no_output_____

#@title Video 1: "What" Models
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "//player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1mz4y1X7ot', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
videoVideo available at https://www.bilibili.com/video/BV1mz4y1X7ot

# Setup

_____no_output_____Python requires you to explictly "import" libraries before their functions are available to use. We will always specify our imports at the beginning of each notebook or script._____no_output_____

import numpy as np
import matplotlib.pyplot as plt_____no_output_____

Tutorial notebooks typically begin with several set-up steps that are hidden from view by default.

**Important:** Even though the code is hidden, you still need to run it so that the rest of the notebook can work properly. Step through each cell, either by pressing the play button in the upper-left-hand corner or with a keyboard shortcut (`Cmd-Return` on a Mac, `Ctrl-Enter` otherwise). A number will appear inside the brackets (e.g. `[3]`) to tell you that the cell was executed and what order that happened in.

If you are curious to see what is going on inside each cell, you can double click to expand. Once expanded, double-click the white space to the right of the editor to collapse again._____no_output_____

#@title Figure Settings
import ipywidgets as widgets #interactive display

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
_____no_output_____#@title Helper functions

#@markdown Most of the tutorials make use of helper functions
#@markdown to simplify the code that you need to write. They are defined here.

# Please don't edit these, or worry about understanding them now!

def restrict_spike_times(spike_times, interval):
  """Given a spike_time dataset, restrict to spikes within given interval.

  Args:
    spike_times (sequence of np.ndarray): List or array of arrays,
      each inner array has spike times for a single neuron.
    interval (tuple): Min, max time values; keep min <= t < max.

  Returns:
    np.ndarray: like `spike_times`, but only within `interval`
  """
  interval_spike_times = []
  for spikes in spike_times:
    interval_mask = (spikes >= interval[0]) & (spikes < interval[1])
    interval_spike_times.append(spikes[interval_mask])
  return np.array(interval_spike_times, object)_____no_output_____#@title Data retrieval
#@markdown This cell downloads the example dataset that we will use in this tutorial.
import io
import requests
r = requests.get('https://osf.io/sy5xt/download')
if r.status_code != 200:
  print('Failed to download data')
else:
  spike_times = np.load(io.BytesIO(r.content), allow_pickle=True)['spike_times']_____no_output_____

---

# Section 1: Exploring the Steinmetz dataset

In this tutorial we will explore the structure of a neuroscience dataset.

We consider a subset of data from a study of [Steinmetz _et al._ (2019)](https://www.nature.com/articles/s41586-019-1787-x). In this study, Neuropixels probes were implanted in the brains of mice. Electrical potentials were measured by hundreds of electrodes along the length of each probe. Each electrode's measurements captured local variations in the electric field due to nearby spiking neurons. A spike sorting algorithm was used to infer spike times and cluster spikes according to common origin: a single cluster of sorted spikes is causally attributed to a single neuron.

In particular, a single recording session of spike times and neuron assignments was loaded and assigned to `spike_times` in the preceding setup.

Typically a dataset comes with some information about its structure. However, this information may be incomplete. You might also apply some transformations or "pre-processing" to create a working representation of the data of interest, which might go partly undocumented depending on the circumstances. In any case it is important to be able to use the available tools to investigate unfamiliar aspects of a data structure.

Let's see what our data looks like..._____no_output_____## Section 1.1: Warming up with `spike_times`_____no_output_____What is the Python type of our variable?_____no_output_____

type(spike_times)_____no_output_____

You should see `numpy.ndarray`, which means that it's a normal NumPy array.

If you see an error message, it probably means that you did not execute the set-up cells at the top of the notebook. So go ahead and make sure to do that.

Once everything is running properly, we can ask the next question about the dataset: what's its shape?_____no_output_____

spike_times.shape_____no_output_____

There are 734 entries in one dimension, and no other dimensions. What is the Python type of the first entry, and what is *its* shape?_____no_output_____

idx = 0
print(
  type(spike_times[idx]),
  spike_times[idx].shape,
  sep="\n",
)<class 'numpy.ndarray'>
(826,)

It's also a NumPy array with a 1D shape! Why didn't this show up as a second dimension in the shape of `spike_times`? That is, why not `spike_times.shape == (734, 826)`?

To investigate, let's check another entry._____no_output_____

idx = 321
print(
  type(spike_times[idx]),
  spike_times[idx].shape,
  sep="\n",
)<class 'numpy.ndarray'>
(9723,)

It's also a 1D NumPy array, but it has a different shape. Checking the NumPy types of the values in these arrays, and their first few elements, we see they are composed of floating point numbers (not another level of `np.ndarray`):_____no_output_____

i_neurons = [0, 321]
i_print = slice(0, 5)

for i in i_neurons:
  print(
    "Neuron {}:".format(i),
    spike_times[i].dtype,
    spike_times[i][i_print],
    "\n",
    sep="\n"
  )
Neuron 0:
float32
[ 0.8149   14.822467 24.9646   25.1436   38.8709  ]

Neuron 321:
float32
[1.0698667 1.1536334 1.2403667 1.7072    1.799    ]

Note that this time we've checked the NumPy `dtype` rather than the Python variable type. These two arrays contain floating point numbers ("floats") with 32 bits of precision.

The basic picture is coming together:
- `spike_times` is 1D, its entries are NumPy arrays, and its length is the number of neurons (734): by indexing it, we select a subset of neurons.
- An array in `spike_times` is also 1D and corresponds to a single neuron; its entries are floating point numbers, and its length is the number of spikes attributed to that neuron. By indexing it, we select a subset of spike times for that neuron.

Visually, you can think of the data structure as looking something like this:

```
| . . . . . |
| . . . . . . . . |
| . . . |
| . . . . . . . |
```

Before moving on, we'll calculate and store the number of neurons in the dataset and the number of spikes per neuron:_____no_output_____

n_neurons = len(spike_times)
total_spikes_per_neuron = [len(spike_times_i) for spike_times_i in spike_times]

print(f"Number of neurons: {n_neurons}")
print(f"Number of spikes for first five neurons: {total_spikes_per_neuron[:5]}")Number of neurons: 734
Number of spikes for first five neurons: [826, 2818, 3953, 646, 1115]
#@title Video 2: Exploring the dataset
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "//player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1Hp4y1S7Au', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
videoVideo available at https://www.bilibili.com/video/BV1Hp4y1S7Au

## Section 1.2: Getting warmer: counting and plotting total spike counts

As we've seen, the number of spikes over the entire recording is variable between neurons. More generally, some neurons tend to spike more than others in a given period. Lets explore what the distribution of spiking looks like across all the neurons in the dataset._____no_output_____Are most neurons "loud" or "quiet", compared to the average? To see, we'll define bins of constant width in terms of total spikes and count the neurons that fall in each bin. This is known as a "histogram".

You can plot a histogram with the matplotlib function `plt.hist`. If you just need to compute it, you can use the numpy function `np.histogram` instead._____no_output_____

plt.hist(total_spikes_per_neuron, bins=50)
plt.xlabel("Total spikes per neuron")
plt.ylabel("Number of neurons");_____no_output_____

Let's see what percentage of neurons have a below-average spike count:_____no_output_____

mean_spike_count = np.mean(total_spikes_per_neuron)
frac_below_mean = (total_spikes_per_neuron < mean_spike_count).mean()
print(f"{frac_below_mean:2.1%} of neurons are below the mean")68.0% of neurons are below the mean

We can also see this by adding the average spike count to the histogram plot:_____no_output_____

plt.hist(total_spikes_per_neuron, bins=50)
plt.xlabel("Total spikes per neuron")
plt.ylabel("Number of neurons")
plt.axvline(mean_spike_count, color="orange", label="Mean neuron")
plt.legend();_____no_output_____

This shows that the majority of neurons are relatively "quiet" compared to the mean, while a small number of neurons are exceptionally "loud": they must have spiked more often to reach a large count.

### Exercise 1: Comparing mean and median neurons

If the mean neuron is more active than 68% of the population, what does that imply about the relationship between the mean neuron and the median neuron?

*Exercise objective:* Reproduce the plot above, but add the median neuron.
_____no_output_____

# To complete the exercise, fill in the missing parts (...) and uncomment the code

median_spike_count = ...  # Hint: Try the function np.median

# plt.hist(..., bins=50)
# plt.axvline(..., color="limegreen", label="Median neuron")
# plt.axvline(mean_spike_count, color="orange", label="Mean neuron")
# plt.xlabel("Total spikes per neuron")
# plt.ylabel("Number of neurons")
# plt.legend()_____no_output_____# to_remove solution

median_spike_count = np.median(total_spikes_per_neuron)

with plt.xkcd():
  # XKCD mode forces white outlines on a bar histogram, which is confusing
  # because you wouldn't see them in a "correct" student exercise.
  # Draw with stepfilled so this looks like what we want the students to make,
  # but it's fine for students to leave that parameter out.
  plt.hist(total_spikes_per_neuron, bins=50, histtype="stepfilled")
  plt.axvline(median_spike_count, color="limegreen", label="Median neuron")
  plt.axvline(mean_spike_count, color="orange", label="Mean neuron")
  plt.xlabel("Total spikes per neuron")
  plt.ylabel("Number of neurons")
  plt.legend()_____no_output_____

*Bonus:* The median is the 50th percentile. What about other percentiles? Can you show the interquartile range on the histogram?_____no_output_____---

# Section 2: Visualizing neuronal spiking activity_____no_output_____## Section 2.1: Getting a subset of the data

Now we'll visualize trains of spikes. Because the recordings are long, we will first define a short time interval and restrict the visualization to only the spikes in this interval. We defined a utility function, `restrict_spike_times`, to do this for you. If you call `help()` on the function, it will tell you a little bit about itself:_____no_output_____

help(restrict_spike_times)Help on function restrict_spike_times in module __main__:

restrict_spike_times(spike_times, interval)
    Given a spike_time dataset, restrict to spikes within given interval.

    Args:
      spike_times (sequence of np.ndarray): List or array of arrays,
        each inner array has spike times for a single neuron.
      interval (tuple): Min, max time values; keep min <= t < max.

    Returns:
      np.ndarray: like `spike_times`, but only within `interval`

t_interval = (5, 15)  # units are seconds after start of recording
interval_spike_times = restrict_spike_times(spike_times, t_interval)_____no_output_____

Is this a representative interval? What fraction of the total spikes fall in this interval?_____no_output_____

original_counts = sum([len(spikes) for spikes in spike_times])
interval_counts = sum([len(spikes) for spikes in interval_spike_times])
frac_interval_spikes = interval_counts / original_counts
print(f"{frac_interval_spikes:.2%} of the total spikes are in the interval")0.33% of the total spikes are in the interval

How does this compare to the ratio between the interval duration and the experiment duration? (What fraction of the total time is in this interval?)

We can approximate the experiment duration by taking the minimum and maximum spike time in the whole dataset. To do that, we "concatenate" all of the neurons into one array and then use `np.ptp` ("peak-to-peak") to get the difference between the maximum and minimum value:_____no_output_____

spike_times_flat = np.concatenate(spike_times)
experiment_duration = np.ptp(spike_times_flat)
interval_duration = t_interval[1] - t_interval[0]

frac_interval_time = interval_duration / experiment_duration
print(f"{frac_interval_time:.2%} of the total time is in the interval")0.37% of the total time is in the interval

These two values—the fraction of total spikes and the fraction of total time—are similar. This suggests the average spike rate of the neuronal population is not very different in this interval compared to the entire recording.

## Section 2.2: Plotting spike trains and rasters

Now that we have a representative subset, we're ready to plot the spikes, using the matplotlib `plt.eventplot` function. Let's look at a single neuron first:_____no_output_____

neuron_idx = 1
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);_____no_output_____

We can also plot multiple neurons. Here are three:_____no_output_____

neuron_idx = [1, 11, 51]
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);_____no_output_____

This makes a "raster" plot, where the spikes from each neuron appear in a different row.

Plotting a large number of neurons can give you a sense for the characteristics in the population. Let's show every 5th neuron that was recorded:_____no_output_____

neuron_idx = np.arange(0, len(spike_times), 5)
plt.eventplot(interval_spike_times[neuron_idx], color=".2")
plt.xlabel("Time (s)")
plt.yticks([]);_____no_output_____

*Question*: How does the information in this plot relate to the histogram of total spike counts that you saw above?_____no_output_____

#@title Video 3: Visualizing activity
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "//player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1dt4y1Q7C5', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
videoVideo available at https://www.bilibili.com/video/BV1dt4y1Q7C5

---

# Section 3: Inter-spike intervals and their distributions_____no_output_____Given the ordered arrays of spike times for each neuron in `spike_times`, which we've just visualized, what can we ask next?

Scientific questions are informed by existing models. So, what knowledge do we already have that can inform questions about this data?

We know that there are physical constraints on neuron spiking. Spiking costs energy, which the neuron's cellular machinery can only obtain at a finite rate. Therefore neurons should have a refractory period: they can only fire as quickly as their metabolic processes can support, and there is a minimum delay between consecutive spikes of the same neuron.

More generally, we can ask "how long does a neuron wait to spike again?" or "what is the longest a neuron will wait?" Can we transform spike times into something else, to address questions like these more directly?

We can consider the inter-spike times (or interspike intervals: ISIs). These are simply the time differences between consecutive spikes of the same neuron.

### Exercise 2: Plot the distribution of ISIs for a single neuron

*Exercise objective:* make a histogram, like we did for spike counts, to show the distribution of ISIs for one of the neurons in the dataset.

Do this in three steps:

1. Extract the spike times for one of the neurons
2. Compute the ISIs (the amount of time between spikes, or equivalently, the difference between adjacent spike times)
3. Plot a histogram with the array of individual ISIs_____no_output_____

def compute_single_neuron_isis(spike_times, neuron_idx):
  """Compute a vector of ISIs for a single neuron given spike times.

  Args:
    spike_times (list of 1D arrays): Spike time dataset, with the first
      dimension corresponding to different neurons.
    neuron_idx (int): Index of the unit to compute ISIs for.

  Returns:
    isis (1D array): Duration of time between each spike from one neuron.
  """
  #############################################################################
  # Students: Fill in missing code (...) and comment or remove the next line
  raise NotImplementedError("Exercise: compute single neuron ISIs")
  #############################################################################

  # Extract the spike times for the specified neuron
  single_neuron_spikes = ...

  # Compute the ISIs for this set of spikes
  # Hint: the function np.diff computes discrete differences along an array
  isis = ...

  return isis

# Uncomment the following lines when you are ready to test your function
# single_neuron_isis = compute_single_neuron_isis(spike_times, neuron_idx=283)
# plt.hist(single_neuron_isis, bins=50, histtype="stepfilled")
# plt.axvline(single_neuron_isis.mean(), color="orange", label="Mean ISI")
# plt.xlabel("ISI duration (s)")
# plt.ylabel("Number of spikes")
# plt.legend()_____no_output_____# to_remove solution
def compute_single_neuron_isis(spike_times, neuron_idx):
  """Compute a vector of ISIs for a single neuron given spike times.

  Args:
    spike_times (list of 1D arrays): Spike time dataset, with the first
      dimension corresponding to different neurons.
    neuron_idx (int): Index of the unit to compute ISIs for.

  Returns:
    isis (1D array): Duration of time between each spike from one neuron.
  """
  # Extract the spike times for the specified neuron
  single_neuron_spikes = spike_times[neuron_idx]

  # Compute the ISIs for this set of spikes
  isis = np.diff(single_neuron_spikes)

  return isis

single_neuron_isis = compute_single_neuron_isis(spike_times, neuron_idx=283)

with plt.xkcd():
  plt.hist(single_neuron_isis, bins=50, histtype="stepfilled")
  plt.axvline(single_neuron_isis.mean(), color="orange", label="Mean ISI")
  plt.xlabel("ISI duration (s)")
  plt.ylabel("Number of spikes")
  plt.legend()_____no_output_____

---

In general, the shorter ISIs are predominant, with counts decreasing rapidly (and smoothly, more or less) with increasing ISI. However, counts also rapidly decrease to zero with _decreasing_ ISI, below the maximum of the distribution (8-11 ms). The absence of these very low ISIs agrees with the refractory period hypothesis: the neuron cannot fire quickly enough to populate this region of the ISI distribution.

Check the distributions of some other neurons. To resolve various features of the distributions, you might need to play with the value of `n_bins`. Using too few bins might smooth over interesting details, but if you use too many bins, the random variability will start to dominate.

You might also want to restrict the range to see the shape of the distribution when focusing on relatively short or long ISIs. *Hint:* the third argument to `plt.hist` sets the interval to define bins over._____no_output_____---

# Section 4: What is the functional form of an ISI distribution?_____no_output_____

#@title Video 4: ISI distribution
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "//player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1ov411B7Pm', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
videoVideo available at https://www.bilibili.com/video/BV1ov411B7Pm

The ISI histograms seem to follow continuous, monotonically decreasing functions above their maxima. The function is clearly non-linear. Could it belong to a single family of functions?

To motivate the idea of using a mathematical function to explain physiological phenomena, let's define a few different function forms that we might expect the relationship to follow: exponential, inverse, and linear._____no_output_____

def exponential(xs, scale, rate, x0):
  """A simple parametrized exponential function, applied element-wise.

  Args:
    xs (np.ndarray or float): Input(s) to the function.
    scale (float): Linear scaling factor.
    rate (float): Exponential growth (positive) or decay (negative) rate.
    x0 (float): Horizontal offset.

  """
  ys = scale * np.exp(rate * (xs - x0))
  return ys

def inverse(xs, scale, x0):
  """A simple parametrized inverse function (`1/x`), applied element-wise.

  Args:
    xs (np.ndarray or float): Input(s) to the function.
    scale (float): Linear scaling factor.
    x0 (float): Horizontal offset.

  """
  ys = scale / (xs - x0)
  return ys

def linear(xs, slope, y0):
  """A simple linear function, applied element-wise.

  Args:
    xs (np.ndarray or float): Input(s) to the function.
    slope (float): Slope of the line.
    y0 (float): y-intercept of the line.

  """
  ys = slope * xs + y0
  return ys_____no_output_____

### Interactive Demo: ISI functions explorer

Here is an interactive demo where you can vary the parameters of these functions and see how well the resulting outputs correspond to the data. Adjust the parameters by moving the sliders and see how close you can get the lines to follow the falling curve of the histogram. This will give you a taste of what you're trying to do when you *fit a model* to data.

"Interactive demo" cells have hidden code that defines an interface where you can play with the parameters of some function using sliders. You don't need to worry about how the code works – but you do need to **run the cell** to enable the sliders.
_____no_output_____

#@title

#@markdown Be sure to run this cell to enable the demo
# Don't worry about understanding this code! It's to setup an interactive plot.
single_neuron_idx = 283
single_neuron_spikes = spike_times[single_neuron_idx]
single_neuron_isis = np.diff(single_neuron_spikes)

counts, edges = np.histogram(
  single_neuron_isis,
  bins=50,
  range=(0, single_neuron_isis.max())
)

functions = dict(
  exponential=exponential,
  inverse=inverse,
  linear=linear,
)

colors = dict(
  exponential="C1",
  inverse="C2",
  linear="C4",
)

@widgets.interact(
  exp_scale=widgets.FloatSlider(1000, min=0, max=20000, step=250),
  exp_rate=widgets.FloatSlider(-10, min=-200, max=50, step=1),
  exp_x0=widgets.FloatSlider(0.1, min=-0.5, max=0.5, step=0.005),
  inv_scale=widgets.FloatSlider(1000, min=0, max=3e2, step=10),
  inv_x0=widgets.FloatSlider(0, min=-0.2, max=0.2, step=0.01),
  lin_slope=widgets.FloatSlider(-1e5, min=-6e5, max=1e5, step=10000),
  lin_y0=widgets.FloatSlider(10000, min=0, max=4e4, step=1000),
)
def fit_plot(
  exp_scale=1000, exp_rate=-10, exp_x0=0.1,
  inv_scale=1000, inv_x0=0,
  lin_slope=-1e5, lin_y0=2000,
):
  """Helper function for plotting function fits with interactive sliders."""
  func_params = dict(
    exponential=(exp_scale, exp_rate, exp_x0),
    inverse=(inv_scale, inv_x0),
    linear=(lin_slope, lin_y0),
  )
  f, ax = plt.subplots()
  ax.fill_between(edges[:-1], counts, step="post", alpha=.5)
  xs = np.linspace(1e-10, edges.max())
  for name, function in functions.items():
    ys = function(xs, *func_params[name])
    ax.plot(xs, ys, lw=3, color=colors[name], label=name);
  ax.set(
      xlim=(edges.min(), edges.max()),
      ylim=(0, counts.max() * 1.1),
      xlabel="ISI (s)",
      ylabel="Number of spikes",
  )
  ax.legend()_____no_output_____#@title Video 5: Fitting models by hand
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "//player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1w54y1S7Eb', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
videoVideo available at https://www.bilibili.com/video/BV1w54y1S7Eb

# Summary

In this tutorial, we loaded some neural data and poked at it to understand how the dataset is organized. Then we made some basic plots to visualize (1) the average level of activity across the population and (2) the distribution of ISIs for an individual neuron. In the very last bit, we started to think about using mathematical formalisms to understand or explain some physiological phenomenon. All of this only allowed us to understand "What" the data looks like.

This is the first step towards developing models that can tell us something about the brain. That's what we'll focus on in the next two tutorials._____no_output_____


# Notebook from Hsu-Che-Wei/COPILOT
Path: jupyter_notebook/6-CytoTRACE.ipynb

# CytoTRACE_____no_output_____#### Working directory should contain root atlas Seurat object, which is available on GEO under record GSE152766:

    Root_Atlas.rds (or get it by running through notebook 2)_____no_output_____

rm(list=ls())
# Set the working directory to where folders named after the samples are located.
# The folder contains spliced.mtx, unspliced.mtx, barcodes and gene id files, and json files produced by scKB that documents the sequencing stats.
setwd("/scratch/AG_Ohler/CheWei/scKB")_____no_output_____# Load libraries
suppressMessages(library(Seurat))
suppressMessages(library(CytoTRACE))
suppressMessages(library(RColorBrewer))
suppressMessages(library(ggplot2))_____no_output_____sessionInfo()_____no_output_____# Read in the atlas
rc.integrated <- readRDS('./Root_Atlas.rds')_____no_output_____

## 1. CytoTRACE_____no_output_____

# Since the atlas contains too many cells to be processed by CytoTRACE, we decided to perform CytoTRACE on clusters instead of cells
# Find nearest neighbors in 50 UMAP dimensions
rc.integrated <- FindNeighbors(rc.integrated, reduction = "umap_50",dims = 1:50)Computing nearest neighbor graph

Computing SNN

# Clustering with SLM algorithm with resolution 500, which produces ~ 3000 clusters
rc.integrated <- FindClusters(rc.integrated, resolution = 500, algorithm = 3)Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 110427
Number of edges: 2611498

Running smart local moving algorithm...
Maximum modularity in 10 random starts: 0.5994
Number of communities: 2998
Elapsed time: 41 seconds
# Extract integrated (batch-corrected) expression matrix
afm <- rc.integrated@assays$integrated@data

# Pool (average) expression values of each cluster
pooled <- matrix(nrow=nrow(afm), ncol = 0)
for (i in 0:(length(unique(rc.integrated@meta.data$seurat_clusters))-1)) {
m <- afm[,which(rc.integrated@meta.data$seurat_clusters==i)]
pooled <- cbind(pooled, rowSums(m)/ncol(m))
}_____no_output_____# Manually make the expression value non-negative
pooled[which(pooled < 0)]=0_____no_output_____# Set colnames
colnames(pooled) <- as.character(0:(length(unique(rc.integrated@meta.data$seurat_clusters))-1))
str(pooled) num [1:17513, 1:2998] 0 0 0 1.68 0 ...
 - attr(*, "dimnames")=List of 2
  ..$ : chr [1:17513] "AT1G05260" "AT3G59370" "AT2G36100" "AT1G12080" ...
  ..$ : chr [1:2998] "0" "1" "2" "3" ...
# Run CytoTRACE
results <- CytoTRACE(pooled, ncores = 16, subsamplesize = 1000)The number of cells in your dataset is less than 3,000. Fast mode has been disabled.

CytoTRACE will be run on 1 sub-sample(s) of approximately 2998 cells each using 1 / 16 core(s)

Pre-processing data and generating similarity matrix...

Calculating gene counts signature...

Smoothing values with NNLS regression and diffusion...

Calculating genes associated with CytoTRACE...

Done

# Store CytoTRACE results
new.cluster.ids <- results$CytoTRACE
names(new.cluster.ids) <- levels(rc.integrated)
rc.integrated <- RenameIdents(object = rc.integrated, new.cluster.ids)
rc.integrated$CytoTRACE <- as.numeric(levels(rc.integrated@active.ident)[rc.integrated@active.ident])_____no_output_____# Plot CytoTRACE
options(repr.plot.width=8, repr.plot.height=8)
FeaturePlot(rc.integrated, features = "CytoTRACE", pt.size=0.5)+ scale_colour_gradientn(colours = rev(brewer.pal(11,"Spectral")))Scale for 'colour' is already present. Adding another scale for 'colour',
which will replace the existing scale.

# Save Seurat object
saveRDS(rc.integrated, file='./Root_Atlas.rds')_____no_output_____


# Notebook from bharris12/pyMN
Path: notebooks/protocol1_cluster_replicability.ipynb

# Protocol 1: assessment of cell type replicability with unsupervised MetaNeighbor

Protocol 1 demonstrates how to compute and visualize cluster replicability across 4 human pancreas datasets. We will show steps detailing how to install MetaNeighbor, how to compute and interpret MetaNeighbor AUROCs. All code blocks can be run in a Jupyter notebook, python or ipython interpretor or in a script run at the command line.

## Step 0: Installation of MetaNeighbor and packages used in the protocol

1. We start by installing the latest MetaNeighbor package from the Gillis lab GitHub page if you have not done so already.

Uncomment the line in the cell below and run it, if you run into errors with the installation, refer to the installation information in the [README](https://github.com/gillislab/pymn)
_____no_output_____

#!pip install git+https://github.com/gillislab/pyMN#egg=pymn_____no_output_____

2. Next we need to import other packages needed to run this analysis, all of these are required by pyMN anyway so you should have them if it successfully installs_____no_output_____

import numpy as np
import pandas as pd
import scanpy as sc
import matplotlib.pyplot as plt
import seaborn as sns
import pymn_____no_output_____%matplotlib inline_____no_output_____

3. Here we set up our plotting settings, these are our preferences_____no_output_____

#These save characters as text in PDFs
import matplotlib
matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42

#These change plot aesthetics

sns.set(style='white', font_scale=1.25)
plt.rc("axes.spines", top=False, right=False)
plt.rc('xtick', bottom=True)
plt.rc('ytick', left=True)_____no_output_____

## Step 1: Downloading the Hemberg scRNAseq data

We consider 4 pancreatic datasets along with their independent annotation (from the original publication). MetaNeighbor expects a gene x cell matrix encapsulated in a anndata format. In Python we recommend using the anndata object. It works well with the popular scRNAseq analysis package Scanpy. The 4 datasets are best acquired using the R package scRNAseq. In the equivalent protocol using the R version of MetaNeighbor we outline how to acquire and preprocess each individual dataset.

Merging the R SingleCellExperiment objects requires multiple steps shown in the R version of the protocol. In comparison, merging Python anndata objects is relatively straightforward, but if you have difficulties the [anndata documentation](https://anndata.readthedocs.io/en/stable/concatenation.html) has a section dedicated to it. For the BICCN data used in Protocols 2 and 3, when processing the data we merged 7 Anndata objects. To follow how the data was processed you can refer to this [R script](https://github.com/gillislab/pyMN/blob/master/data/hemberg.R) and this [Python script](https://github.com/gillislab/pyMN/blob/master/data/hemberg.py). Running the R script requires installing the R version of [MetaNeigbhor](https://github.com/gillislab/MetaNeighbor) and the Bioconductor package [scRNAseq](https://bioconductor.org/packages/3.12/scRNAseq).

4. First we download the data using `curl`_____no_output_____

!curl -L -o hemberg.h5ad https://ndownloader.figshare.com/files/24928574  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 97.2M  100 97.2M    0     0  3634k      0  0:00:27  0:00:27 --:--:-- 3461k

## Step 2: Hierarchical cell type replicability analysis

5. We load the previously created pancreas dataset._____no_output_____

adata = sc.read_h5ad('hemberg.h5ad') #Temporary until hosted file_____no_output_____adata.obs.columns = adata.obs.columns.astype(str)_____no_output_____

Here you can see that we have an AnnData object with 15,793 cells and 15,295 genes

It has two observation features: `cell.type` and `study_id`_____no_output_____

adata_____no_output_____

6. Study and Cell Type Data must be passed as a string or integer to MetaNeigbhor fast version to improve computation times._____no_output_____

adata.obs['cell.type'] = adata.obs['cell.type'].astype(str)
adata.obs['study_id'] = adata.obs['study_id'].astype(str)_____no_output_____

7. To perform neighbor voting, MetaNeighbor builds a cell-cell similarity network, which we defined as the Spearman correlation over a user-defined set of genes. We found that we obtained best results by picking genes that are highly variable across datasets, which can be picked using the "variableGenes" function.

>**NOTE**: This is the one function that differs slightly from the R version. The quantile function in R defaults to a behavoir that is not native to any standard python packages, thus the variable gene list is slightly different between the python and R versions. Overall this does not appear to affect the results significantly. While the AUROCs will not be identical they are extremely similar. In Section 1.4 there are a few differences in connectivity between individual clusters, but the neighborhood structure of the cell-type graph is consistent_____no_output_____

pymn.variableGenes(adata, study_col='study_id')/home/bharris/miniconda3/lib/python3.7/site-packages/anndata/_core/anndata.py:1094: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  if not is_categorical(df_full[k]):

The function returns a list of 736 (600 in the R version) genes that were detected as highly variable in each of the 4 datasets. By default the `variableGenes` functions stores the highly variable genes under the `highly_variable` column in `.var` of the passed AnnData object

8. The data and a set of biological meaningful genes is all we need to run MetaNeighbor and obtain cluster similarities.

>**NOTE**: Most functions do not return anything by default. Rather they save the output of the function in the `.uns` part of the anndata object. Additionally, by default the functions search for input data from the `.uns` data. This functionality is highly customizable, you can pass objects directly if you want and return the results instead of saving them from the object. Additionally, you can define your own keys to save results as instead of the defaults. __EX__: The output of the below `MetaNeighborUS` run is save in `adata.uns['MetaNeighborUS']` and the parameters uses are saved under `adata.uns['MetaNeighborUS_params']`_____no_output_____

pymn.MetaNeighborUS(adata,
                    study_col='study_id',
                    ct_col='cell.type',
                    fast_version=True)_____no_output_____

Cluster similarities are defined as an Area Under the ROC curve (AUROC), which range between 0 and 1, where 0 indicates low similarity and 1 high similarity.

9. For ease of interpretation, we visualize AUROCs as a heatmap, where rows and columns are cell types from all the datasets._____no_output_____

pymn.plotMetaNeighborUS(adata, figsize=(10, 10), cmap='coolwarm', fontsize=10)/home/bharris/miniconda3/lib/python3.7/site-packages/seaborn/matrix.py:1215: UserWarning: ``square=True`` ignored in clustermap
  warnings.warn(msg)

In the heatmap, the color of each square indicates the proximity of a pair of cluster, ranging from blue (low similarity) to red (high similarity). For example, "serger|gamma cell" (bottom row) is highly similar to "muraro|pp" (2nd column from the right) but very dissimilar from "muraro|ductal" (middle column). To group similar clusters together, "plotHeatmap" applies hierarchical clustering on the AUROC matrix. On the heatmap, we see two red blocks that indicate clear hierarchical structure in the data, with endocrine cell types clustering together (e.g., alpha, beta, gamma) and mesenchymal cells on the other side (e.g., amacrine, ductal, endothelial). Note that each red block is composed of smaller red blocks, indicating that clusters can be matched at an even higher resolution. We also see some off-diagonal patterns (e.g., lawlor|Gamma/PP, lawlor|Delta), which generally indicate the presence of doublets or contamination (presence of cells from other cell types), but what matters here is the clear presence of red blocks, which is a strong indicator of replicability.

10. To identify pairs of replicable clusters, we rely on a simple heuristics: a pair of cluster is replicable if they are reciprocal top hits (they preferentially vote for each other) and the AUROC exceeds a given threshold value (in our experience, 0.9 is a good heuristic value)._____no_output_____

pymn.topHits(adata, threshold=0.9)

adata.uns['MetaNeighborUS_topHits']_____no_output_____

We find a long list of replicable endocrine cell types (e.g., epsilon, alpha and beta cells) and non-endocrine cell types (e.g. mast, endothelial or acinar cells). This list provides strong evidence that these cell types are robust, as they are identified across all datasets with high AUROC.

11. In the case where there is a clear structure in the data (endocrine vs non-endocrine here), we can refine AUROCs by splitting the data. AUROCs have a simple interpretation: an AUROC of 0.6 indicates that cells from a given cell type are ranked in front of 60% of other test cells. However, this interpretation is outgroup dependent: because endocrine cells represent ~65% of cells, even an unrelated pair of non-endocrine cell types will have an AUROC > 0.65, because non-endocrine cells will always be ranked in front of endocrine cells.

By starting with the full datasets, we uncovered the global structure in the data (endocrine vs non-endocrine). However, to evaluate replicability of endocrine cell types and reduce dataset composition effects, we can make the assessment more stringent by restricting the outgroup to close cell types, i.e. by keeping only endocrine subtypes. We split cell types in two by using the splitClusters function and retain only endocrine cell types._____no_output_____

level1_split = pymn.splitClusters(
    adata, k=2,
    save_uns=False)  #Returning the splitClusters instead of saving them
level1_split_____no_output_____first_split = level1_split[1]_____no_output_____

By outputting “level1_split”, we found that the cell types were nicely split between non-endocrine and endocrine, and that endocrine cell types where in the second element of the list. Note that splitClusters applies a simple hierarchical clustering algorithm to separate cell types, cell types can be selected manually in more complex scenarios.

12. We repeat the MetaNeighbor analysis on endocrine cells only. First, we subset the data to the endocrine cell types (stored in "first_split")._____no_output_____

to_keep = np.in1d(
    pymn.join_labels(adata.obs['study_id'].values,
                     adata.obs['cell.type'].values), first_split)
subdata = adata[to_keep, :]
subdata.shape/home/bharris/miniconda3/lib/python3.7/site-packages/anndata/_core/anndata.py:1094: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  if not is_categorical(df_full[k]):

The new dataset contains the 9341 putative endocrine cells.

13. To focus on variability that is specific to endocrine cells, we re-pick highly variable genes:
_____no_output_____

pymn.variableGenes(subdata, study_col='study_id') Trying to set attribute `.var` of view, copying.

14. Finally we recompute cluster similarities and visualize AUROCs._____no_output_____

res = pymn.MetaNeighborUS(subdata,
                          study_col='study_id',
                          ct_col='cell.type',
                          fast_version=True)
_____no_output_____pymn.plotMetaNeighborUS(subdata,cmap='coolwarm')/home/bharris/miniconda3/lib/python3.7/site-packages/seaborn/matrix.py:1215: UserWarning: ``square=True`` ignored in clustermap
  warnings.warn(msg)

The resulting heatmap illustrates an example of a strong set of replicating cell types: when the assessment becomes more stringent (restriction to closely related cell types), the similarity of replicating cell types remains strong (AUROC~1 for alpha, beta, gamma, delta and epsilon cells) while the cross-cell-type similarity decreases (shift from red to blue, e.g. similarity of alpha and beta cell types has shifted from orange/red in the global heatmap to dark blue in the endocrine heatmap) by virtue of zooming in on a subpart of the dataset.

15. We can continue to zoom in as long as there are at least two cell types per dataset:_____no_output_____

level2_split = pymn.splitClusters(subdata, k=3, save_uns=False)
my_split = level2_split[2]
keep_cell = np.in1d(
    pymn.join_labels(adata.obs['study_id'], adata.obs['cell.type']), my_split)
subdata = adata[keep_cell, ]
pymn.MetaNeighborUS(subdata,
                          study_col='study_id',
                          ct_col='cell.type',
                          fast_version=True)/home/bharris/miniconda3/lib/python3.7/site-packages/anndata/_core/anndata.py:1094: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  if not is_categorical(df_full[k]):
Trying to set attribute `.uns` of view, copying.
pymn.plotMetaNeighborUS(subdata, cmap='coolwarm')/home/bharris/miniconda3/lib/python3.7/site-packages/seaborn/matrix.py:1215: UserWarning: ``square=True`` ignored in clustermap
  warnings.warn(msg)

Here we removed the alpha and beta cells (representing close to 85% of endocrine cells) and validate that, even when restricting to neighboring cell types, there is still a clear distinction between delta, gamma and epsilon cells (AUROC ~ 1).

## Step 3: stringent assessment of replicability with one-vs-best AUROCs

In the previous section, we created progressively more stringent replicability assessments by selecting more and more specific subsets of related cell types. As an alternative, we provide the “one_vs_best” parameter, which offers similar results without having to restrict the dataset manually. In this scoring mode, MetaNeighbor will automatically identify the two closest matching cell types in each test dataset and compute an AUROC based on the voting result for cells from the closest match against cells from the second closest match. Essentially, we are asking how easily a cell type can be distinguished from its closest neighbor.

16. To obtain one-vs-best AUROCs, we run the same command as before with two additional parameters: "one_vs_best = TRUE" and "symmetric_output = FALSE"._____no_output_____

pymn.MetaNeighborUS(adata,
                    study_col='study_id',
                    ct_col='cell.type',
                    fast_version=True,
                    symmetric_output=False,
                    one_vs_best=True)_____no_output_____pymn.plotMetaNeighborUS(adata,
                        cmap='coolwarm',
                        figsize=(10, 10),
                        mn_key='MetaNeighborUS_1v1',
                        xticklabels=True,
                        yticklabels=True,
                        fontsize=7)_____no_output_____

The interpretation of the heatmap is slightly different compared to one-vs-all AUROCs. First, since we only compare the two closest cell types, most cell type combinations are not tested (NAs, shown in white on the heatmap). Second, by setting “symmetric_output=FALSE”, we broke the symmetry of the heatmap: train cell types are shown as columns and test cell types are shown as rows. Since each cell type is only tested against two cell types in each test dataset (closest and second closest match), we have 8 values per column (2 per dataset).

This representation helps to rapidly identify a cell type’s closest hits as well as their closest outgroup. For example, ductal cells (1st red square from the top left) strongly match with each other (one-vs-best AUROC>0.8) and acinar cells are their closest outgroup (blue segments in the same column). The non-symmetric view makes it clear when best hits are not reciprocal. For example, mast cells (5th square from the left) heavily vote for “lawlor|Stellate” and “muraro|mesenchymal”, but this vote is not reciprocal. This pattern indicates that the mast cell type is missing in the Lawlor and Muraro datasets: because mast cells have no natural match in these datasets, they vote for the next closest cell type (stellate cells). The lack of reciprocity in voting is an important tool to detect imbalances in dataset composition.

17.When using one-vs-best AUROCs, we recommend extracting replicating cell types as meta-clusters. Cell types are part of the same meta-cluster if they are reciprocal best hits. Note that if cell type A is the reciprocal best hit of B and C, all three cell types are part of the same meta-cluster, even if B and C are not reciprocal best hits. To further filter for strongly replicating cell types, we specify an AUROC threshold (in our experience, 0.7 is a strong one-vs-best AUROC threshold)._____no_output_____

pymn.extractMetaClusters(adata, threshold=.7)
pymn.score_meta_clusters(adata)
mcsummary = adata.uns['MetaNeighborUS_metacluster_scores']
mcsummary[mcsummary.index != 'outliers']_____no_output_____

The *scoreMetaClusters* provides a good summary of meta-clusters, ordering cell types by the number of datasets in which they replicate, then by average AUROC. We find 12 cell types that have strong support across at least 2 datasets, with 7 cell types replicating across all 4 datasets. 8 cell types are tagged as "outlier", as they had no strong match. These cell types usually contain doublets, low quality cells or contaminated cell types. The replicability structure described here can be summarized as an Upset plot._____no_output_____

pymn.plotUpset(adata)_____no_output_____

To further investigate the robustness of meta-clusters, they can be visualized as heatmaps (called “cell-type badges”) with the plotMetaClusters function. Because the function generates one heatmap per meta-cluster, we save the output to a PDF file to facilitate investigation. Each badge shows an AUROC heatmap restricted to one specific meta-cluster. These badges help diagnose cases where AUROCs are lower in a specific train or test dataset. For example, the “muraro|duct” cell type has systematically lower AUROCs, suggesting the presence of contaminating cells in another cell type (probably in the “muraro|unclear” cell type).

> **Note** This is not yet implemented _____no_output_____

####### Not yet implemented #########

# pdf("meta_clusters.pdf")
# plotMetaClusters(mclusters, best_hits)
# dev.off()_____no_output_____

18. The last visualization is an alternative representation of the AUROC heatmap as a graph, which is particularly useful for large datasets. In this graph, top votes (AUROC > 0.5) are shown in gray, while outgroup votes (AUROC < 0.5) are shown in orange. To highlight close calls, we recommend keeping only strong outgroup votes, here with AUROC >= 0.3._____no_output_____

pymn.makeClusterGraph(adata, low_threshold=.3)
pymn.plotClusterGraph(adata, font_size=9, figsize=(10, 10))/home/bharris/pyMN/pymn/plotting.py:383: RuntimeWarning: invalid value encountered in greater
  (best_hits.values < low_threshold)] = 0
/home/bharris/pyMN/pymn/plotting.py:383: RuntimeWarning: invalid value encountered in less
  (best_hits.values < low_threshold)] = 0

We note that there are several orange edges, indicating that some cell types had two close matches. To investigate the origin of these close calls, we can focus on a cluster of interest (coi). Here we take a closer look at “baron|epsilon”, query its closest neighbors in the graph with *extendClusterSet*, then zoom in on its subgraph by passing the list of clusters to the *plotClusterGraph* function in order to subset the graph and plot it._____no_output_____

coi = "baron|epsilon"
coi = pymn.extendClusterSet(adata=adata, coi=coi, max_neighbor_distance=2)
pymn.plotClusterGraph(adata, node_list=coi, font_size=10, figsize=(5, 5))_____no_output_____

In the “baron|epsilon” case, we find that the epsilon cell type is missing in the Lawlor dataset, so there is no natural match for the Baron epsilon cell type. In such cases, votes are frequently non-reciprocal and equally split between two unrelated cell types, here “Lawlor|Gamma/PP” and “Lawlor|Delta”. In general, the cluster graph can be used to understand how meta-clusters are extracted, why some clusters are tagged as outliers and diagnose problems where the resolution of cell types differs across datasets._____no_output_____


# Notebook from bbglab/adventofcode
Path: 2020/loris/notebooks/day_1.ipynb

# Day 1_____no_output_____

data = [int(line.strip()) for line in open('../data/input_1.txt', 'r')]_____no_output_____def answer(result=2020):
    for number in data:
        other_number = result - number
        if other_number in data:
            return number, other_number, number * other_number_____no_output_____answer()_____no_output_____

def answer2(result=2020):
    for number in data:
        other_number = result - number
        res = answer(result=other_number)
        if res is not None:
            return number, res[0], res[1], number * res[0] * res[1]_____no_output_____answer2()_____no_output_____