# Notebook from PeerHerholz/Cog_Com_Neuro_ML_DL
Path: lecture/introduction/notebooks/neuroscience/mri_intro.ipynb

(mri-chapter)=
# Magnetic Resonance Imaging (MRI)

## Objectives 📍
This part of the course aims to introduce you to the `physical principles` of `magnetic resonance imaging`. During this part of the course, we will cover four fundamental principles of `MRI`:
  * `Magnetic resonance`
  * `Relaxation`: parameters `T1` and `T2`
  * `Imaging`
  * `MRI sequences`

## Anatomy of an MRI

```{figure} ../../../static/neuroscience/mri_schema.png
---
width: 800px
name: mri-schema-fig
---
Schematic illustration of the main components of an `MRI` machine. Figure generated by [P. Bellec](), 2021, licensed under CC-BY.
```
{numref}`mri-schema-fig` allows us to observe the main elements of an `MRI` device, and in particular:
  * The **magnet**: this is a `coil` that generates a very strong `magnetic field`. This `coil` is immersed in `liquid helium`, close to absolute zero, which makes it `superconductive`: the `electric current` which crosses it does not undergo any loss of energy, and can continue to circulate for a very long time. For this reason, the `MRI magnet` continues to operate continuously, even when the machine is not in use.

  * The **gradient coils**: allow to vary the `intensity` of the `magnetic field` in space. During `image acquisition`, the `gradients` are activated and then stopped several times. `Gradients` can be produced in any direction.

  * The **radio-frequency antenna**: makes it possible to (1) `excite` `matter` using `transmitters`, and (2) measure the `response` of these `biological tissues` to `excitation` using `receivers`. The `radio-frequency pulses` generated by the `antenna` create a weak `magnetic field` `perpendicular` to the main `magnetic field` generated by the `magnet`. Receiving `antennas` can also be placed in specific headgear.

We'll talk more about how all of these work in the next few sections.

```{warning}
`MRI` is very sensitive to `head movements`! Cushions or other devices can be used to reduce `movement`.
```

```{note} **Coil + current = magnetic field**!
By creating a ring with `electric wire` and passing an `electric current`, we produce a `magnetic field`. In the video below we can see the `magnetic field` lines being drawn when the `magnetic field` is `activated`. `Field lines` are straight as they pass through the `center` of the ring, but they `propagate` in `circles` away from the `center` of the ring. To obtain a constant `magnetic field` inside the ring, we can thicken the ring so as to form a `cylinder`. It's the same principle that we find in an `MRI machine`!
```_____no_output_____

from IPython.display import HTML
import warnings
warnings.filterwarnings("ignore")

# Youtube
HTML('')_____no_output_____

## Magnetic spin and B0 field
```{figure} ../../../static/neuroscience/spin.png
---
width: 500px
name: spin-fig
---
A `proton` is like a small `magnet`, whose `magnetic field` `oscillates` around a given position (`precessional motion`), and is characterized by `angular momentum`, or `spin`. The right part of the figure was generated by [P. Bellec](), 2021, under CC-BY license. The left part of the figure is adapted from an image [shutterstock](https://www.shutterstock.com) ID `130826045`, used under standard shutterstock license.
```

The `protons` that make up part of the `atoms` behave like little `magnets` that `spin` around their own `axis`, similar to a `spinning top` (see {numref}`spin-fig`). This `rotation` of the `magnetic moment` is called the `precession movement` and depends among other things on the `composition` of the `core`. Thus, each type of `nucleus` has a characteristic *Larmor frequency*.

```{note} Larmor frequency of a hydrogen atom
:class: dropdown
A `hydrogen atom` has a `Larmor frequency` of `42.58 MHz/Tesla`. This `frequency` is therefore not fixed, but **depends on the strength of the magnetic field!** Placed in an `external magnetic field` of `1T`, a `hydrogen atom` rotates `42580000` times per second. The stronger the `magnetic field` in which a `proton` is located, the more the speed at which the `magnetic moment` of this `proton` rotates will increase.
```

The `MRI magnet` helps align the `magnetic moment` of the `protons` along the same `axis` as the `main magnetic field`, called `B0`. This `B0` `field` goes from the `feet` to the `head`. The `strength` of the `main magnet` is measured in `Teslas (T)`. `1.5T` devices are mainly used for clinical purposes, while in research, the standard is rather `3T`, which is about `60,000` times more powerful than the Earth's `magnetic field`! The `7T` devices represent today the new frontier used in research, and some `10T`+ devices exist in the world. But why would we want to increase the `strength` of the `magnetic field`? By `increasing` the `strength` of the `magnetic field`, we can gain `spatial` and `temporal resolution`. On the other hand, `increasing` the `strength` of the `magnetic field` can also introduce `artifacts`!

## Magnetic resonance
```{note} Resonance... not just magnetic
We find `resonance` phenomena in many situations. A known example is the `resonance` between the `wind` and the `Tacoma Bridge`, which led to the collapse of the bridge (see the video below).
```_____no_output_____

from IPython.display import HTML
import warnings
warnings.filterwarnings("ignore")

# Youtube
HTML('')_____no_output_____

We can think of `resonance` as a `seesaw motion`. If we push the `swing` `randomly`, it won't `swing` much. To have an `amplified swing motion`, we need to push the `swing` at the `same frequency as the natural frequency of the swing`. We will then **enter into resonance** with the `swing`, and its `movement` will `amplify`. We can therefore see the `swing` as a `phenomenon` of `resonance` between the `object` that is `swinging` and the `person` who gives `impetus` to this `object`.

`MRI` exploits this `resonance phenomenon`. **The radio-frequency (RF) antenna** creates a series of **radio-frequency waves** in the `direction` `perpendicular` to the `B0 field`, ie in the `direction` of the `B1 field`. By producing a `series` of `pulses` following the `Larmor frequency of hydrogen`, the `hydrogen atoms` come into `resonance` and tilt in the `perpendicular direction`.

By stopping the `pulses`, the `hydrogen atoms` enter into `relaxation`, that is to say that their `magnetic moment` will return to the `initial direction B0`. In other words, the `magnetic moment` in `direction B1` decreases to return in `direction B0`. This `relaxation phenomenon` is very important, because the speed of `relaxation` will depend on the `characteristics` of the `tissues` that have been excited. The `relaxation` speed is measured by the receiving `antennas` placed in the `helmet` around the `participant`'s head!

```{note} Phasing
:class: tip

It is important to understand that the `signal` we measure in `MRI` **does not** come from a `single proton`. For reference, `18 grams` of `water` contains one [mole](https://fr.wikipedia.org/wiki/Mole_(unit)) of `H2O molecules`, or about $10^{24}$ `hydrogen atoms`... The `signal` that we are measuring comes from the `juxtaposition` of the `spins` of all of these `atoms`. A `radio-frequency wave` which `resonates` with `hydrogen` will not only tilt the `spins`, but also bring them into `phase`. Imagine that you have a thousand `swings`, which you push at the same time (at the right `frequency`). Not only will the `movement` of the `swings` gain in `amplitude`, but all the `swings` will be at the same point in their `trajectory` at the same time. It is the same for `spins` after an `excitement`.
```

```{note} Why radio?
As we have seen, the `Larmor frequency` of `hydrogen` is `42.58 MHz/Tesla`. In a `3T MRI`, we are therefore going to excite with a `wave` at a `frequency` of approximately `120 MHz`, or `120 million waves` per second (!). This type of `frequency` falls into the domain of [radio waves](https://fr.wikipedia.org/wiki/Onde_radio).
```

## Section selection and image formation

We have seen how a `radio-frequency wave` can excite `hydrogen nuclei` and measure the `response` to this `excitation` to examine the `characteristics` of `tissues`. But how to make an `image`? The **gradient coils** make it possible to vary the `amplitude` of the `magnetic field` in `three directions`:
 * `Direction z`: from `feet` to `head`
 * `Direction x`: from `left` to `right`
 * `Direction y`: from the `back` of the `head` to the `nose`

These variations are much weaker than the `B0 field`, and only represent a fraction of a `Tesla`, but this will allow us to `extract spatial information` in a `resonance process`. Using these `gradients`, it is possible to measure the `magnetic properties` of `tissues` located at a `specific point` in `space`, and therefore to make a `(3D) image`. This process is complex, but the first step is relatively simple to understand: it is the cut selection.

We remember that the `Larmor frequency` of a `particle` depends on the `strength` of the `magnetic field` in which it finds itself. By changing the `strength` of the `magnetic field` in a given `direction` thanks to the `gradient coils`, we are going to modify the `Larmor frequency` of the `hydrogen atoms` at a `specific point` in the `gradient`. The `radio-frequency pulses` will only `excite` the `hydrogen atoms` in the `cut` where the `magnetic field` has the `strength` that corresponds to the `excitation` `frequency`. In this way, instead of `receiving signal` from the `whole brain`, we only `receive signal` from the `selected slice`, because only the `hydrogen atoms` in this `slice` will `resonate`.

We still have to cut our section into `pixels`... But that is largely beyond the context of this introductory chapter. To learn more about `spatial encoding` in `MRI`, you can consult this [resource](https://www.imaios.com/en/e-Courses/e-MRI/Signal-spatial-encoding/Spatial-encoding-intro).

```{note} Field of View (FOV)
When we make a `3D image` of the `brain`, we will `slice` the `brain` into a `series` of `slices`. By knowing the `size` of the `cuts` as well as the `number of cuts`, we can deduce the `size` of the `3D cube` that corresponds to the `image`. This `size` is called `field of view`, or `FOV`.
```

```{note} The MRI: a noisy machine
The `acquisition` of an `image` requires `modifying` the `gradients` quickly. The `rapid current changes` in the `gradient coils`, as well as in the `radio-frequency wave emitting coils`, cause the `coils` to `expand` and `contract` rapidly. These `movements` create `significant noise`. Each `type` of `image` has its own "music", which depends on the nature and order of the `excitations` and `gradients`. See excerpts below.
```_____no_output_____

from IPython.display import HTML
import warnings
warnings.filterwarnings("ignore")

# Youtube
HTML('')_____no_output_____

## $T_1$ and $T_2$ contrasts

The `contrasts` $T_1$ and $T_2$ are the main parameters acquired during an `MRI` session. Initially, the `hydrogen proton` `spins` are aligned with the $B_0$ field. The application of `radio-frequency pulses` causes the `spins` to tilt along the $B_1$ axis, an `axis` `perpendicular` to $B_0$. Once the `radio-frequency pulses` have stopped, the `spins` `realign` with the $B_0$ field. This `realignment` is characterized by two distinct dynamics, linked to the `time constants` $T_1$ and $T_2$.

**Contrast in $T_1$**. The `increase` in the component according to $B_0$ (component $M_z$), or _longitudinal relaxation_, follows an `increasing exponential function`. The characteristic time of this growth (the speed of growth) is called $T_1$. The time $T_1$ corresponds to the time elapsed to obtain `63%` of the `equilibrium value` of the `magnetic moment` contribution along the `z axis` ($M_0$). For those who are comfortable with mathematical expressions, the regrowth in $B_0$ follows the equation $M_z(t) = M_0 ( 1 - e^{-t / T_1})$._____no_output_____

# ignore warnings
import warnings
warnings.filterwarnings("ignore")

# import necessary libraries
import matplotlib.pyplot as plt
import numpy as np
from myst_nb import glue

# setup figures
fig = plt.figure(figsize=(15, 5))

# Exponential functions for T1 curves (for example only)
t = np.linspace(0,5,100)
y1 = 70 * (1 - np.exp(-t / 1.2))
y2 = 85 * (1 - np.exp(-t / 1.3))
y3 = 100 * (1 - np.exp(-t / 1.5))

# Trace the figure
ax_plot = plt.subplot(1, 3, 1)
plt.plot(t, y1, label="Cerebrospinal fluid")
plt.plot(t, y2, label="grey matter")
plt.plot(t, y3, label="white matter")
plt.vlines(1, 0, 60, colors="black", linestyles="--")
plt.text(1, 60, "TE")
plt.vlines(4, 0, 60, colors="black", linestyles="--")
plt.text(4, 60, "TR")
plt.xlabel("Time", loc="right")
plt.ylabel("mz", loc="top")
plt.title("Curve in T1")
plt.legend()
plt.gca().axes.yaxis.set_ticklabels([])
plt.gca().axes.xaxis.set_ticklabels([])

# Import required modules and dataset
from nilearn.datasets import fetch_icbm152_2009
from nilearn.plotting import plot_anat

data_mri = fetch_icbm152_2009()

# display T1-weighted image
ax_plot = plt.subplot2grid((1, 3), (0, 1), colspan=2)
plot_anat(data_mri.t1, figure=fig, title="MRI - T1 contrast", axes=ax_plot,
    cut_coords=[-17, 0, 17])

glue("relax-t1-fig", fig, display=False)_____no_output_____

```{glue:figure} relax-t1-fig
:figwidth: 800px
:name: "relax-t1-fig"
`Longitudinal relaxation` and `contrast` $T_1$. Left image: growth of the `magnetic field` along the $B_0$ `axis`, called $M_{z}$. Note that different `tissue types` exhibit distinct `longitudinal relaxation curves`. Right image: an `image` generated by reading at time $TE$ presents a `contrast` between the different `tissue types`. This figure is generated by python code, click on the `+` to see the code.
```

```{note} M0
:class: tip
:name: M0-tip
At `equilibrium`, the contribution of the `magnetic moment` along the $B_0$ `axis` is called $M_0$. This value depends on the `density` of `protons` in the `tissues`, i.e. the `number` of `hydrogen atoms` present in the `tissue`. Thus, from one `voxel` to another, we do not necessarily obtain the same value of $M_0$. It is possible to image this parameter, and we then speak of an image in `proton density`.
```

**Contrast in $T_2$**. The `decrease` in the component according to $B_1$ (component $M_{xy}$), or _transverse relaxation_, follows a `decreasing exponential function`. The characteristic time of this `decrease` (the speed of `decrease`) is called $T_2$. The time $T_2$ corresponds to the `time` elapsed to obtain `37%` of the value of the contribution of the `initial magnetic moment` along the axis $B_1$. For those who are comfortable with mathematical expressions, the decrease in $B_1$ follows the equation $M_{xy}(t) = M_1 e^{-t / T_2}$. The constant $M_1$ will depend, among other things, on the `density` of `protons`, like $M_0$, and will vary from one `tissue` to another._____no_output_____

# ignore warnings
import warnings
warnings.filterwarnings("ignore")

# import necessary functions
import matplotlib.pyplot as plt
import numpy as np
from myst_nb import glue

# setup the figure
fig = plt.figure(figsize=(15, 5))

# Exponential functions for T2 curves (for example only)
t = np.linspace(0, 5, 100)
y1 = 100 * np.exp(-t / 1.5)
y2 = 85 * np.exp(-t / 1.1)
y3 = 70 * np.exp(-t / 0.8)

# Tracer la figure
ax_plot = plt.subplot(1, 3, 1)
plt.plot(t, y1, label="Cerebrospinal fluid")
plt.plot(t, y2, label="grey matter")
plt.plot(t, y3, label="white matter")
plt.vlines(1, 0, 60, colors="black", linestyles="--")
plt.text(1.1, 60, "TE")
plt.vlines(4, 0, 60, colors="black", linestyles="--")
plt.text(4, 60, "TR")
plt.xlabel("Time", loc="right")
plt.ylabel("Mz", loc="top")
plt.title("Curve in T2")
plt.legend()
plt.gca().axes.yaxis.set_ticklabels([])
plt.gca().axes.xaxis.set_ticklabels([])

# Import required modules and dataset
from nilearn.datasets import fetch_icbm152_2009
from nilearn.plotting import plot_anat

data_mri = fetch_icbm152_2009()

# display the T2-weighted image
ax_plot = plt.subplot2grid((1, 3), (0, 1), colspan=2)
plot_anat(data_mri.t2, figure=fig, title="MRI - T2 contrast", axes=ax_plot,
    cut_coords=[-17, 0, 17])

glue("relax-t2-fig", fig, display=False)_____no_output_____

```{glue:figure} relax-t2-fig
:figwidth: 800px
:name: "relax-t2-fig"
`Transverse relaxation` and contrast $T_2$. Left image: `decrease` of the `magnetic field` along the $B_1$ `axis`, called $M_{xy}$. Note that different `tissue types` exhibit `distinct transverse relaxation curves`. Right image: an `image` generated by reading at time $TE$ presents a `contrast` between the `different tissue types`. This `contrast` is essentially reversed with respect to `contrast` $T_1$. This figure is generated by python code, click on the `+` to see the code.
```
```{note} $TE$
When we `acquire` `MRI` `data`, we generally do not measure the entire `relaxation curve`, but simply a `measurement point` at time $TE$. By choosing the $TE$ appropriately, we will obtain very different reading values for the different `tissues`. The time $TE$ will be different for a `contrast` $T_1$ and a `contrast` $T_2$.
```

```{note} $TR$
We call $TR$ the time which separates two series of `excitations`. This value will correspond to the `acquisition time` of a `section` for a `structural MRI`, and the `acquisition time` of a `complete cerebral volume` in `fMRI`. It's a weird convention, but widely used by `MRI physicists`.
```
```{note} flip angle
If we're interested in the end of the `relaxation process`, we don't need to `flip` the `spins` completely in the $B_1$ `direction`, just a certain number of `degrees` from $B_0$. This parameter is called `flip angle`.
```
```{note} $T_1$ vs $T_2$: find the difference
:class: tip
:name: diff-t1-t2-tip
Why bother `contrasting` $T_1$ and $T_2$ when one seems to be the exact opposite of the other? Take a good look at the {numref}`relax-t1-fig` and {numref}`relax-t2-fig` `images`. Are there certain `regions` of the `brain` where `image` $T_2$ is not the exact opposite of image $T_1$?
```

## $T_2^*$, fMRI, dMRI
**Phase shift.** As we saw in the sidebar {ref}`on phase <phase-tip>`, the `radio-frequency pulses` will not only `flip` the `spins`, but also bring them into `phase`. When we stop the `impulses`, the `spins` will gradually go out of `phase`. This `phase shift` is due to `micro-interactions` between `protons` as well as `tissue molecules` that exhibit `magnetic properties`. The `relaxation curve` will have the same `shape`, but with `modified characteristic times`, which we call $T_1^*$ and $T_2^*$.

**Functional MRI.** The `inhomogeneities` in the `magnetic field` which cause the `phase shift` can in particular be created by the `deoxyhemoglobin` that we find in the `blood`. We will see in more detail how `oxyhemoglobin` and `deoxyhemoglobin` `disturb` the `magnetic field` in the chapter on [functional MRI](). In `functional MRI`, we use `T2*-weighted sequences`.

**Diffusion MRI.** In `diffusion MRI`, we also use the `T2* contrast`. On the other hand, in `diffusion MRI`, we measure `inhomogeneities` by `alternating` the `direction` of the `pulses` (eg by giving a `pulse` along the `xy axis`, then by giving a `pulse` along the `-xy axis`). By taking `several images` with `different directions` of `excitation`, we can get an idea of the `direction` of `water diffusion`. This operation allows us in the end to know the `direction` of the `fibers` of `white matter`, because the more a `fiber` points towards a given `direction`, the greater the `diffusion` will be in this `direction`. We will come back to this subject in the chapter on [diffusion MRI]().

## Console and acquisition sequences

```{figure} ../../../static/neuroscience/mri_console.png
---
width: 500px
name: irm-console-fig
---
Connections between the console and the different parts of an `MRI` system. Figure adapted by [P. Bellec](), 2021, licensed under CC-BY. The original Figure is taken from the article by {cite:p}`gruber_rf_2018`, licensed under CC-BY-NC.
```

All elements of the `MRI` device can be controlled by the console (see {numref}`irm-console-fig`). We have seen together the key principles of `MRI`, but an actual `image` is `acquired` with a complex series of `excitations` and `measurements`, which is called an _acquisition sequence_. Once the `sequence` has been programmed, we can always modify certain `parameters` of the `sequence`, such as:
* $TE$
* $TR$
* `field of view` (`FOV`)
* `Number` of `cuts`
* `Thickness` of `cuts`
* `Voxel size`

### Conclusions
This chapter introduced you to the `physical principles` of `MRI`. We have seen the different components of an `MRI machine`, the different `magnetic phenomena` allowing us to `acquire` `images`, as well as some `parameters` that we can modify during the `acquisition` of `MRI data`. In the next chapter, we will talk about `morphometry` using `structural MRI`.

## References

```{bibliography}
:filter: docname in docnames
:style: plain
```

## Exercices

In the following we created a few exercices that aim to recap core aspects of this part of the course and thus should allow you to assess if you understood the main points.

```{admonition} Exercice 2.1
:class: dropdown
Rank the `characteristic` `times` in `ascending order`, if possible.
  * `TE` vs `TR`
  * `TR` vs. `T1`
  * `T1` vs. `TE`
```

```{admonition} Exercice 2.2
:class: dropdown
True or false? The `field strength` of an `MRI` is related to the `size` of the `MRI`.
```

```{admonition} Exercice 2.3
:class: dropdown

Choose between the correct answer from 1, 2 or 3. the `spin` of a `hydrogen proton` has...

  1. A fixed `rotational frequency` is the `Larmor frequency`
  2. A `variable rotation frequency`
  3. A `rotational frequency` that depends on the `strength` of the `magnetic field` in the `MRI`.
```

```{admonition} Exercice 2.4
:class: dropdown
What produces `noise` in an `MRI` `acquisition`?
  * The `B0` `field`
  * `Air conditioning` to `cool` the `magnet`
  * `Gradient coils`
  * The `radio-frequency antenna`
```

```{admonition} Exercice 2.5
:class: dropdown
True or false? The `MRI magnet` consumes a lot of `electricity`.
```

```{admonition} Exercice 2.6
:class: dropdown
True or false? `Functional MRI` and `diffusion MRI` both use `T2* contrast`.
```

```{admonition} Exercice 2.7
:class: dropdown
In an `anatomical image`, we see `white ventricles` on a `black background`. Is this a `T1`- or `T2`-`weighted` `acquisition`? Explain why.
```

```{admonition} Exercice 2.8
:class: dropdown
We decide to modify an `MRI sequence` to reduce the `flip angle`: the `spins` will `tilt` by `70 degrees`, instead of `90 degrees`. What will be the effect on the `TR` of this change?
```

```{admonition} Exercice 2.9
:class: dropdown
A `T1 acquisition` is performed with a `field of view` of `210 mm x 210 mm` `in-plane`, and a `resolution` of `1 mm x 1 mm` in the `slice`. What is the `cut size` (`number` of `pixels x number` of `pixels`)?
```

```{admonition} Exercice 2.10
:class: dropdown
An `fMRI acquisition` is carried out with a `resolution` of `3 mm ×3 mm` in the `slice`, a `slice` of `dimension 64×64`, a `slice thickness` of `3.4 mm` with `31 slices`. We have a `TR` of `2 seconds`, and we acquire `150 volumes`.
What is the `size` of the `3D` `field of view`, knowing that the `slices` are acquired in the `axial plane`?
What is the `duration` of the `acquisition`?
```

```{admonition} Exercice 2.11
:class: dropdown
To answer this question, read the article by Shukla et al, "_Aberrant Frontostriatal Connectivity in Negative Symptoms of Schizophrenia_", published in Schizophrenia Bulletin (2019, 45(5): 1051-59) and available open access at this [ address](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6737477/). The following questions require short answers.

 * What is the `strength` of the `MRI magnet`?
 * How many `channels` are present in the `head antenna`?
 * What is the `TR` of `structural acquisition`? and `functional acquisition`? Comparing these two times with each other, does it make sense that one is bigger than the other?
 * What is the name of the `sequence` used for `structural acquisition`?
 * What is the name of the `sequence` used for `functional acquisition`?
 * What is the `TE` of the `structural acquisition`? and `functional acquisition`? Comparing these two times with each other, does it make sense that one is bigger than the other?
 * What is the `size` of the `functional acquisition` `field of view`, in `cm`?
 * How many `brain volumes` are `acquired` during the `functional sequence`?
```

```{admonition} Exercice 2.12
:class: dropdown

We want to `isolate` the `thalamus` on an `individual` `anatomical image`. Which `contrast` should we use: `T1`, `T2` or both? Please justify your answer.
```_____no_output_____


# Notebook from ScienceOctopus/octopus-core
Path: merge_csvs.ipynb

import pandas as pd
import glob

path = "downloads/pubmed_meta" # use your path
all_files = glob.glob(path + "/*.csv")

li = []

for filename in all_files:
    df = pd.read_csv(filename, index_col=None, header=0)
    li.append(df)

frame = pd.concat(li, axis=0, ignore_index=True)
frame.to_csv("downloads/concated.csv")_____no_output_____!pip install biopython spacy en_core_web_md
Requirement already satisfied: biopython in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (1.79)
Requirement already satisfied: spacy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (3.2.1)
ERROR: Could not find a version that satisfies the requirement en_core_web_md (from versions: none)
ERROR: No matching distribution found for en_core_web_md
!sh setup.shCollecting nbdev
  Using cached nbdev-1.1.23-py3-none-any.whl (46 kB)
Collecting nbconvert<6
  Downloading nbconvert-5.6.1-py2.py3-none-any.whl (455 kB)
     |████████████████████████████████| 455 kB 28.7 MB/s
Requirement already satisfied: jupyter-client<7.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (6.1.11)
Requirement already satisfied: pip in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (21.3.1)
Requirement already satisfied: packaging in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (21.3)
Requirement already satisfied: pyyaml in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (5.4.1)
Collecting fastrelease
  Using cached fastrelease-0.1.12-py3-none-any.whl (14 kB)
Requirement already satisfied: nbformat>=4.4.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (5.1.2)
Requirement already satisfied: jupyter in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (1.0.0)
Requirement already satisfied: ipykernel in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (5.5.0)
Collecting fastcore>=1.3.21
  Using cached fastcore-1.3.27-py3-none-any.whl (56 kB)
Collecting ghapi
  Using cached ghapi-0.1.19-py3-none-any.whl (51 kB)
Requirement already satisfied: jupyter-core>=4.6.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter-client<7.0->nbdev) (4.7.1)
Requirement already satisfied: tornado>=4.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter-client<7.0->nbdev) (6.1)
Requirement already satisfied: traitlets in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter-client<7.0->nbdev) (4.3.3)
Requirement already satisfied: pyzmq>=13 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter-client<7.0->nbdev) (22.0.3)
Requirement already satisfied: python-dateutil>=2.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter-client<7.0->nbdev) (2.8.1)
Requirement already satisfied: jinja2>=2.4 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (2.11.3)
Requirement already satisfied: bleach in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (4.1.0)
Requirement already satisfied: pandocfilters>=1.4.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (1.4.2)
Requirement already satisfied: testpath in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (0.4.4)
Requirement already satisfied: defusedxml in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (0.6.0)
Requirement already satisfied: entrypoints>=0.2.2 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (0.3)
Requirement already satisfied: mistune<2,>=0.8.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (0.8.4)
Requirement already satisfied: pygments in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (2.8.0)
Requirement already satisfied: ipython-genutils in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbformat>=4.4.0->nbdev) (0.2.0)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbformat>=4.4.0->nbdev) (3.2.0)
Requirement already satisfied: ipython>=5.0.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipykernel->nbdev) (7.16.1)
Requirement already satisfied: qtconsole in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter->nbdev) (5.0.2)
Requirement already satisfied: jupyter-console in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter->nbdev) (6.4.0)
Requirement already satisfied: notebook in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter->nbdev) (6.4.6)
Requirement already satisfied: ipywidgets in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter->nbdev) (7.6.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from packaging->nbdev) (2.4.7)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (3.0.5)
Requirement already satisfied: setuptools>=18.5 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (49.6.0.post20210108)
Requirement already satisfied: decorator in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (4.4.2)
Requirement already satisfied: backcall in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (0.2.0)
Requirement already satisfied: pickleshare in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (0.7.5)
Requirement already satisfied: jedi>=0.10 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (0.17.2)
Requirement already satisfied: pexpect in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (4.8.0)
Requirement already satisfied: MarkupSafe>=0.23 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jinja2>=2.4->nbconvert<6->nbdev) (1.1.1)
Requirement already satisfied: pyrsistent>=0.14.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (0.17.3)
Requirement already satisfied: importlib-metadata in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (3.7.0)
Requirement already satisfied: attrs>=17.4.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (20.3.0)
Requirement already satisfied: six>=1.11.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (1.15.0)
Requirement already satisfied: webencodings in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from bleach->nbconvert<6->nbdev) (0.5.1)
Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipywidgets->jupyter->nbdev) (1.0.0)
Requirement already satisfied: widgetsnbextension~=3.5.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipywidgets->jupyter->nbdev) (3.5.1)
Requirement already satisfied: Send2Trash>=1.8.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from notebook->jupyter->nbdev) (1.8.0)
Requirement already satisfied: argon2-cffi in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from notebook->jupyter->nbdev) (20.1.0)
Requirement already satisfied: nest-asyncio>=1.5 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from notebook->jupyter->nbdev) (1.5.4)
Requirement already satisfied: terminado>=0.8.3 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from notebook->jupyter->nbdev) (0.9.2)
Requirement already satisfied: prometheus-client in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from notebook->jupyter->nbdev) (0.9.0)
Requirement already satisfied: qtpy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from qtconsole->jupyter->nbdev) (1.9.0)
Requirement already satisfied: parso<0.8.0,>=0.7.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jedi>=0.10->ipython>=5.0.0->ipykernel->nbdev) (0.7.0)
Requirement already satisfied: wcwidth in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.0.0->ipykernel->nbdev) (0.2.5)
Requirement already satisfied: ptyprocess in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from terminado>=0.8.3->notebook->jupyter->nbdev) (0.7.0)
Requirement already satisfied: cffi>=1.0.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from argon2-cffi->notebook->jupyter->nbdev) (1.14.5)
Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from importlib-metadata->jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (3.4.0)
Requirement already satisfied: typing-extensions>=3.6.4 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from importlib-metadata->jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (3.10.0.2)
Requirement already satisfied: pycparser in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from cffi>=1.0.0->argon2-cffi->notebook->jupyter->nbdev) (2.20)
Installing collected packages: nbconvert, fastcore, ghapi, fastrelease, nbdev
  Attempting uninstall: nbconvert
    Found existing installation: nbconvert 6.0.7
    Uninstalling nbconvert-6.0.7:
      Successfully uninstalled nbconvert-6.0.7
Successfully installed fastcore-1.3.27 fastrelease-0.1.12 ghapi-0.1.19 nbconvert-5.6.1 nbdev-1.1.23
Collecting fastai
  Using cached fastai-2.5.3-py3-none-any.whl (189 kB)
Collecting flatdict
  Using cached flatdict-4.0.1.tar.gz (8.3 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: spacy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (3.2.1)
Collecting bertopic
  Downloading bertopic-0.9.3-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 767 kB/s
Requirement already satisfied: matplotlib in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (3.3.4)
Requirement already satisfied: pydash in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from -r requirements.txt (line 6)) (5.1.0)
Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from -r requirements.txt (line 7)) (1.1.5)
Requirement already satisfied: biopython in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from -r requirements.txt (line 8)) (1.79)
Collecting spacytextblob
  Using cached spacytextblob-3.0.1-py3-none-any.whl (4.1 kB)
Collecting sentence-transformers
  Using cached sentence-transformers-2.1.0.tar.gz (78 kB)
  Preparing metadata (setup.py) ... done
Collecting top2vec
  Using cached top2vec-1.0.26-py3-none-any.whl (23 kB)
Requirement already satisfied: scikit-learn in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (0.24.1)
Requirement already satisfied: pyyaml in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (5.4.1)
Collecting fastprogress>=0.2.4
  Using cached fastprogress-1.0.0-py3-none-any.whl (12 kB)
Requirement already satisfied: requests in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (2.26.0)
Requirement already satisfied: packaging in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (21.3)
Collecting torchvision>=0.8.2
  Downloading torchvision-0.11.2-cp36-cp36m-manylinux1_x86_64.whl (23.3 MB)
     |████████████████████████████████| 23.3 MB 93.9 MB/s
Requirement already satisfied: scipy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (1.5.3)
Collecting fastdownload<2,>=0.0.5
  Using cached fastdownload-0.0.5-py3-none-any.whl (13 kB)
Requirement already satisfied: pip in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (21.3.1)
Requirement already satisfied: pillow>6.0.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (8.4.0)
Collecting torch<1.11,>=1.7.0
  Downloading torch-1.10.1-cp36-cp36m-manylinux1_x86_64.whl (881.9 MB)
     |████████████                    | 327.9 MB 129.1 MB/s eta 0:00:05from octoflow.core import get_pubmed_records, get_segment_____no_output_____import pandas as pd
df = pd.read_csv("downloads/concated.csv")_____no_output_____df["PMID"] = df["PMID"].astype(str)
len(df)
_____no_output_____a = [1]
a += [2]
a_____no_output_____records = []
step = 10000
for idx in range(step, 100001, step):
    recs = get_pubmed_records(df[idx-step:idx]["PMID"])
    records += recs["PubmedArticle"]_____no_output_____len(records)_____no_output_____from pydash import get
get(records[1], "MedlineCitation.Article.Abstract.AbstractText")
_____no_output_____df["abstract"] = [str(get(record, "MedlineCitation.Article.Abstract.AbstractText.0")) for record in records]_____no_output_____df = df.rename(columns={"abstract": "Abstract"})_____no_output_____import json
import re

alphabets= "([A-Za-z])"
prefixes = "(Mr|St|Mrs|Ms|Dr)[.]"
suffixes = "(Inc|Ltd|Jr|Sr|Co)"
starters = "(Mr|Mrs|Ms|Dr|He\s|She\s|It\s|They\s|Their\s|Our\s|We\s|But\s|However\s|That\s|This\s|Wherever)"
acronyms = "([A-Z][.][A-Z][.](?:[A-Z][.])?)"
websites = "[.](com|net|org|io|gov)"

def split_into_sentences(text):
    text = " " + text + "  "
    text = text.replace("\n"," ")
    text = re.sub(prefixes,"\\1<prd>",text)
    text = re.sub(websites,"<prd>\\1",text)
    if "Ph.D" in text: text = text.replace("Ph.D.","Ph<prd>D<prd>")
    text = re.sub("\s" + alphabets + "[.] "," \\1<prd> ",text)
    text = re.sub(acronyms+" "+starters,"\\1<stop> \\2",text)
    text = re.sub(alphabets + "[.]" + alphabets + "[.]" + alphabets + "[.]","\\1<prd>\\2<prd>\\3<prd>",text)
    text = re.sub(alphabets + "[.]" + alphabets + "[.]","\\1<prd>\\2<prd>",text)
    text = re.sub(" "+suffixes+"[.] "+starters," \\1<stop> \\2",text)
    text = re.sub(" "+suffixes+"[.]"," \\1<prd>",text)
    text = re.sub(" " + alphabets + "[.]"," \\1<prd>",text)
    if "”" in text: text = text.replace(".”","”.")
    if "\"" in text: text = text.replace(".\"","\".")
    if "!" in text: text = text.replace("!\"","\"!")
    if "?" in text: text = text.replace("?\"","\"?")
    text = text.replace(".",".<stop>")
    text = text.replace("?","?<stop>")
    text = text.replace("!","!<stop>")
    text = text.replace("<prd>",".")
    sentences = text.split("<stop>")
    sentences = sentences[:-1]
    sentences = [s.strip() for s in sentences]
    return sentences

def get_first(p):
    return get(split_into_sentences(p), 0)_____no_output_____df["sentences"] = df["Abstract"].map(split_into_sentences)_____no_output_____df["first_sent"] = df["sentences"].map(lambda s: get(s,0))_____no_output_____

df.head(5)_____no_output_____# for index, r in [*df.iterrows()][:10]:
#     #print(row)
#     print(r["Title"], r["abstract"])_____no_output_____!sh setup.shCollecting nbdev
  Using cached nbdev-1.1.23-py3-none-any.whl (46 kB)
Collecting nbconvert<6
  Downloading nbconvert-5.6.1-py2.py3-none-any.whl (455 kB)
     |████████████████████████████████| 455 kB 26.1 MB/s
Collecting fastrelease
  Using cached fastrelease-0.1.12-py3-none-any.whl (14 kB)
Requirement already satisfied: jupyter in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (1.0.0)
Requirement already satisfied: ipykernel in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (5.5.0)
Requirement already satisfied: pip in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (21.3.1)
Requirement already satisfied: pyyaml in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (5.4.1)
Requirement already satisfied: jupyter-client<7.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (6.1.11)
Collecting fastcore>=1.3.21
  Using cached fastcore-1.3.27-py3-none-any.whl (56 kB)
Requirement already satisfied: nbformat>=4.4.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (5.1.2)
Requirement already satisfied: packaging in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbdev) (21.3)
Collecting ghapi
  Using cached ghapi-0.1.19-py3-none-any.whl (51 kB)
Requirement already satisfied: python-dateutil>=2.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter-client<7.0->nbdev) (2.8.1)
Requirement already satisfied: tornado>=4.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter-client<7.0->nbdev) (6.1)
Requirement already satisfied: pyzmq>=13 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter-client<7.0->nbdev) (22.0.3)
Requirement already satisfied: jupyter-core>=4.6.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter-client<7.0->nbdev) (4.7.1)
Requirement already satisfied: traitlets in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter-client<7.0->nbdev) (4.3.3)
Requirement already satisfied: bleach in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (4.1.0)
Requirement already satisfied: testpath in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (0.4.4)
Requirement already satisfied: jinja2>=2.4 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (2.11.3)
Requirement already satisfied: entrypoints>=0.2.2 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (0.3)
Requirement already satisfied: pygments in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (2.8.0)
Requirement already satisfied: mistune<2,>=0.8.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (0.8.4)
Requirement already satisfied: defusedxml in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (0.6.0)
Requirement already satisfied: pandocfilters>=1.4.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbconvert<6->nbdev) (1.4.2)
Requirement already satisfied: ipython-genutils in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbformat>=4.4.0->nbdev) (0.2.0)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from nbformat>=4.4.0->nbdev) (3.2.0)
Requirement already satisfied: ipython>=5.0.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipykernel->nbdev) (7.16.1)
Requirement already satisfied: notebook in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter->nbdev) (6.4.6)
Requirement already satisfied: qtconsole in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter->nbdev) (5.0.2)
Requirement already satisfied: ipywidgets in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter->nbdev) (7.6.3)
Requirement already satisfied: jupyter-console in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jupyter->nbdev) (6.4.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from packaging->nbdev) (2.4.7)
Requirement already satisfied: pickleshare in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (0.7.5)
Requirement already satisfied: backcall in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (0.2.0)
Requirement already satisfied: pexpect in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (4.8.0)
Requirement already satisfied: decorator in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (4.4.2)
Requirement already satisfied: setuptools>=18.5 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (49.6.0.post20210108)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (3.0.5)
Requirement already satisfied: jedi>=0.10 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbdev) (0.17.2)
Requirement already satisfied: MarkupSafe>=0.23 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jinja2>=2.4->nbconvert<6->nbdev) (1.1.1)
Requirement already satisfied: importlib-metadata in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (3.7.0)
Requirement already satisfied: attrs>=17.4.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (20.3.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (0.17.3)
Requirement already satisfied: six>=1.11.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (1.15.0)
Requirement already satisfied: webencodings in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from bleach->nbconvert<6->nbdev) (0.5.1)
Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipywidgets->jupyter->nbdev) (1.0.0)
Requirement already satisfied: widgetsnbextension~=3.5.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from ipywidgets->jupyter->nbdev) (3.5.1)
Requirement already satisfied: prometheus-client in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from notebook->jupyter->nbdev) (0.9.0)
Requirement already satisfied: Send2Trash>=1.8.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from notebook->jupyter->nbdev) (1.8.0)
Requirement already satisfied: argon2-cffi in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from notebook->jupyter->nbdev) (20.1.0)
Requirement already satisfied: terminado>=0.8.3 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from notebook->jupyter->nbdev) (0.9.2)
Requirement already satisfied: nest-asyncio>=1.5 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from notebook->jupyter->nbdev) (1.5.4)
Requirement already satisfied: qtpy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from qtconsole->jupyter->nbdev) (1.9.0)
Requirement already satisfied: parso<0.8.0,>=0.7.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from jedi>=0.10->ipython>=5.0.0->ipykernel->nbdev) (0.7.0)
Requirement already satisfied: wcwidth in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.0.0->ipykernel->nbdev) (0.2.5)
Requirement already satisfied: ptyprocess in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from terminado>=0.8.3->notebook->jupyter->nbdev) (0.7.0)
Requirement already satisfied: cffi>=1.0.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from argon2-cffi->notebook->jupyter->nbdev) (1.14.5)
Requirement already satisfied: typing-extensions>=3.6.4 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from importlib-metadata->jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (3.10.0.2)
Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from importlib-metadata->jsonschema!=2.5.0,>=2.4->nbformat>=4.4.0->nbdev) (3.4.0)
Requirement already satisfied: pycparser in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from cffi>=1.0.0->argon2-cffi->notebook->jupyter->nbdev) (2.20)
Installing collected packages: nbconvert, fastcore, ghapi, fastrelease, nbdev
  Attempting uninstall: nbconvert
    Found existing installation: nbconvert 6.0.7
    Uninstalling nbconvert-6.0.7:
      Successfully uninstalled nbconvert-6.0.7
Successfully installed fastcore-1.3.27 fastrelease-0.1.12 ghapi-0.1.19 nbconvert-5.6.1 nbdev-1.1.23
Collecting fastai
  Using cached fastai-2.5.3-py3-none-any.whl (189 kB)
Collecting flatdict
  Using cached flatdict-4.0.1.tar.gz (8.3 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: spacy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (3.2.1)
Collecting bertopic
  Downloading bertopic-0.9.3-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 6.4 MB/s
Requirement already satisfied: matplotlib in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (3.3.4)
Collecting pydash
  Using cached pydash-5.1.0-py3-none-any.whl (84 kB)
Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from -r requirements.txt (line 7)) (1.1.5)
Requirement already satisfied: biopython in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from -r requirements.txt (line 8)) (1.79)
Collecting spacytextblob
  Using cached spacytextblob-3.0.1-py3-none-any.whl (4.1 kB)
Collecting sentence-transformers
  Using cached sentence-transformers-2.1.0.tar.gz (78 kB)
  Preparing metadata (setup.py) ... done
Collecting top2vec
  Using cached top2vec-1.0.26-py3-none-any.whl (23 kB)
Requirement already satisfied: pip in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (21.3.1)
Requirement already satisfied: pyyaml in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (5.4.1)
Collecting fastprogress>=0.2.4
  Using cached fastprogress-1.0.0-py3-none-any.whl (12 kB)
Requirement already satisfied: fastcore<1.4,>=1.3.22 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (1.3.27)
Requirement already satisfied: scikit-learn in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (0.24.1)
Collecting torchvision>=0.8.2
  Downloading torchvision-0.11.2-cp36-cp36m-manylinux1_x86_64.whl (23.3 MB)
     |████████████████████████████████| 23.3 MB 95.7 MB/s
Requirement already satisfied: requests in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (2.26.0)
Collecting fastdownload<2,>=0.0.5
  Using cached fastdownload-0.0.5-py3-none-any.whl (13 kB)
Requirement already satisfied: scipy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (1.5.3)
Requirement already satisfied: packaging in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from fastai->-r requirements.txt (line 1)) (21.3)
Collecting torch<1.11,>=1.7.0
  Downloading torch-1.10.2-cp36-cp36m-manylinux1_x86_64.whl (881.9 MB)
     |████████████▊                   | 349.4 MB 83.4 MB/s eta 0:00:07 import pandas as pd
import glob
from octoflow.core import replace_outof_vocab_words, vocab30k
_____no_output_____df = pd.read_csv("downloads/pm_abstracts_100k.csv")_____no_output_____len(df)_____no_output_____
#df = df.dropna(subset=["labels"])
df["text"] = df["first_sent"] #backup

df["text_length"]= df["text"].map(lambda t: len(t.split(" ")))

df = df[df["text_length"] > 7]

len(df)

#df["labels"] = df["labels"].astype(int)_____no_output_____df["text"] = df["text"].map(lambda t: replace_outof_vocab_words(t, vocab30k))
_____no_output_____df["text"] = df["text"].astype(str)_____no_output_____df.to_csv("downloads/pm_abstracts_100k_masked.csv", index = False, header=True)_____no_output_____


# Notebook from cvsekhar/hmm-tagger
Path: HMM Tagger.ipynb

# Project: Part of Speech Tagging with Hidden Markov Models
---
### Introduction

Part of speech tagging is the process of determining the syntactic category of a word from the words in its surrounding context. It is often used to help disambiguate natural language phrases because it can be done quickly with high accuracy. Tagging can be used for many NLP tasks like determining correct pronunciation during speech synthesis (for example, _dis_-count as a noun vs dis-_count_ as a verb), for information retrieval, and for word sense disambiguation.

In this notebook, you'll use the [Pomegranate](http://pomegranate.readthedocs.io/) library to build a hidden Markov model for part of speech tagging using a "universal" tagset. Hidden Markov models have been able to achieve [>96% tag accuracy with larger tagsets on realistic text corpora](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf). Hidden Markov models have also been used for speech recognition and speech generation, machine translation, gene recognition for bioinformatics, and human gesture recognition for computer vision, and more.

![](_post-hmm.png)

The notebook already contains some code to get you started. You only need to add some new functionality in the areas indicated to complete the project; you will not need to modify the included code beyond what is requested. Sections that begin with **'IMPLEMENTATION'** in the header indicate that you must provide code in the block that follows. Instructions will be provided for each section, and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully!_____no_output_____

**Note:** Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You must then **export the notebook** by running the last cell in the notebook, or by using the menu above and navigating to **File -> Download as -> HTML (.html)** Your submissions should include both the `html` and `ipynb` files.

_____no_output_____

**Note:** Code and Markdown cells can be executed using the `Shift + Enter` keyboard shortcut. Markdown cells can be edited by double-clicking the cell to enter edit mode.

_____no_output_____### The Road Ahead
You must complete Steps 1-3 below to pass the project. The section on Step 4 includes references & resources you can use to further explore HMM taggers.

- [Step 1](#Step-1:-Read-and-preprocess-the-dataset): Review the provided interface to load and access the text corpus
- [Step 2](#Step-2:-Build-a-Most-Frequent-Class-tagger): Build a Most Frequent Class tagger to use as a baseline
- [Step 3](#Step-3:-Build-an-HMM-tagger): Build an HMM Part of Speech tagger and compare to the MFC baseline
- [Step 4](#Step-4:-[Optional]-Improving-model-performance): (Optional) Improve the HMM tagger_____no_output_____

# Jupyter "magic methods" -- only need to be run once per kernel restart
%load_ext autoreload
%aimport helpers, tests
%autoreload 1_____no_output_____# import python modules -- this cell needs to be run again if you make changes to any of the files
import matplotlib.pyplot as plt
import numpy as np
import collections

from itertools import chain
from collections import Counter, defaultdict
from helpers import show_model, Dataset
from pomegranate import State, HiddenMarkovModel, DiscreteDistribution
from IPython.core.display import HTML
from operator import itemgetter_____no_output_____

## Step 1: Read and preprocess the dataset
---
We'll start by reading in a text corpus and splitting it into a training and testing dataset. The data set is a copy of the [Brown corpus](https://en.wikipedia.org/wiki/Brown_Corpus) (originally from the [NLTK](https://www.nltk.org/) library) that has already been pre-processed to only include the [universal tagset](https://arxiv.org/pdf/1104.2086.pdf). You should expect to get slightly higher accuracy using this simplified tagset than the same model would achieve on a larger tagset like the full [Penn treebank tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), but the process you'll follow would be the same.

The `Dataset` class provided in helpers.py will read and parse the corpus. You can generate your own datasets compatible with the reader by writing them to the following format. The dataset is stored in plaintext as a collection of words and corresponding tags. Each sentence starts with a unique identifier on the first line, followed by one tab-separated word/tag pair on each following line. Sentences are separated by a single blank line.

Example from the Brown corpus.
```
b100-38532
Perhaps	ADV
it	PRON
was	VERB
right	ADJ
;	.
;	.

b100-35577
...
```_____no_output_____

data = Dataset("tags-universal.txt", "brown-universal.txt", train_test_split=0.8)

print("There are {} sentences in the corpus.".format(len(data)))
print("There are {} sentences in the training set.".format(len(data.training_set)))
print("There are {} sentences in the testing set.".format(len(data.testing_set)))

assert len(data) == len(data.training_set) + len(data.testing_set), \
       "The number of sentences in the training set + testing set should sum to the number of sentences in the corpus"There are 57340 sentences in the corpus.
There are 45872 sentences in the training set.
There are 11468 sentences in the testing set.

### The Dataset Interface

You can access (mostly) immutable references to the dataset through a simple interface provided through the `Dataset` class, which represents an iterable collection of sentences along with easy access to partitions of the data for training & testing. Review the reference below, then run and review the next few cells to make sure you understand the interface before moving on to the next step.

```
Dataset-only Attributes:
    training_set - reference to a Subset object containing the samples for training
    testing_set - reference to a Subset object containing the samples for testing

Dataset & Subset Attributes:
    sentences - a dictionary with an entry {sentence_key: Sentence()} for each sentence in the corpus
    keys - an immutable ordered (not sorted) collection of the sentence_keys for the corpus
    vocab - an immutable collection of the unique words in the corpus
    tagset - an immutable collection of the unique tags in the corpus
    X - returns an array of words grouped by sentences ((w11, w12, w13, ...), (w21, w22, w23, ...), ...)
    Y - returns an array of tags grouped by sentences ((t11, t12, t13, ...), (t21, t22, t23, ...), ...)
    N - returns the number of distinct samples (individual words or tags) in the dataset

Methods:
    stream() - returns an flat iterable over all (word, tag) pairs across all sentences in the corpus
    __iter__() - returns an iterable over the data as (sentence_key, Sentence()) pairs
    __len__() - returns the nubmer of sentences in the dataset
```

For example, consider a Subset, `subset`, of the sentences `{"s0": Sentence(("See", "Spot", "run"), ("VERB", "NOUN", "VERB")), "s1": Sentence(("Spot", "ran"), ("NOUN", "VERB"))}`. The subset will have these attributes:

```
subset.keys == {"s1", "s0"}  # unordered
subset.vocab == {"See", "run", "ran", "Spot"}  # unordered
subset.tagset == {"VERB", "NOUN"}  # unordered
subset.X == (("Spot", "ran"), ("See", "Spot", "run"))  # order matches .keys
subset.Y == (("NOUN", "VERB"), ("VERB", "NOUN", "VERB"))  # order matches .keys
subset.N == 7  # there are a total of seven observations over all sentences
len(subset) == 2  # because there are two sentences
```

**Note:** The `Dataset` class is _convenient_, but it is **not** efficient. It is not suitable for huge datasets because it stores multiple redundant copies of the same data.

_____no_output_____#### Sentences

`Dataset.sentences` is a dictionary of all sentences in the training corpus, each keyed to a unique sentence identifier. Each `Sentence` is itself an object with two attributes: a tuple of the words in the sentence named `words` and a tuple of the tag corresponding to each word named `tags`._____no_output_____

key = 'b100-38532'
print("Sentence: {}".format(key))
print("words:\n\t{!s}".format(data.sentences[key].words))
print("tags:\n\t{!s}".format(data.sentences[key].tags))Sentence: b100-38532
words:
	('Perhaps', 'it', 'was', 'right', ';', ';')
tags:
	('ADV', 'PRON', 'VERB', 'ADJ', '.', '.')

**Note:** The underlying iterable sequence is **unordered** over the sentences in the corpus; it is not guaranteed to return the sentences in a consistent order between calls. Use `Dataset.stream()`, `Dataset.keys`, `Dataset.X`, or `Dataset.Y` attributes if you need ordered access to the data.

#### Counting Unique Elements

You can access the list of unique words (the dataset vocabulary) via `Dataset.vocab` and the unique list of tags via `Dataset.tagset`._____no_output_____

print("There are a total of {} samples of {} unique words in the corpus."
      .format(data.N, len(data.vocab)))
print("There are {} samples of {} unique words in the training set."
      .format(data.training_set.N, len(data.training_set.vocab)))
print("There are {} samples of {} unique words in the testing set."
      .format(data.testing_set.N, len(data.testing_set.vocab)))
print("There are {} words in the test set that are missing in the training set."
      .format(len(data.testing_set.vocab - data.training_set.vocab)))

assert data.N == data.training_set.N + data.testing_set.N, \
       "The number of training + test samples should sum to the total number of samples"There are a total of 1161192 samples of 56057 unique words in the corpus.
There are 928458 samples of 50536 unique words in the training set.
There are 232734 samples of 25112 unique words in the testing set.
There are 5521 words in the test set that are missing in the training set.

#### Accessing word and tag Sequences
The `Dataset.X` and `Dataset.Y` attributes provide access to ordered collections of matching word and tag sequences for each sentence in the dataset._____no_output_____

# accessing words with Dataset.X and tags with Dataset.Y
for i in range(2):
    print("Sentence {}:".format(i + 1), data.X[i])
    print()
    print("Labels {}:".format(i + 1), data.Y[i])
    print()Sentence 1: ('Mr.', 'Podger', 'had', 'thanked', 'him', 'gravely', ',', 'and', 'now', 'he', 'made', 'use', 'of', 'the', 'advice', '.')

Labels 1: ('NOUN', 'NOUN', 'VERB', 'VERB', 'PRON', 'ADV', '.', 'CONJ', 'ADV', 'PRON', 'VERB', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence 2: ('But', 'there', 'seemed', 'to', 'be', 'some', 'difference', 'of', 'opinion', 'as', 'to', 'how', 'far', 'the', 'board', 'should', 'go', ',', 'and', 'whose', 'advice', 'it', 'should', 'follow', '.')

Labels 2: ('CONJ', 'PRT', 'VERB', 'PRT', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'ADP', 'ADV', 'ADV', 'DET', 'NOUN', 'VERB', 'VERB', '.', 'CONJ', 'DET', 'NOUN', 'PRON', 'VERB', 'VERB', '.')

#### Accessing (word, tag) Samples
The `Dataset.stream()` method returns an iterator that chains together every pair of (word, tag) entries across all sentences in the entire corpus._____no_output_____

# use Dataset.stream() (word, tag) samples for the entire corpus
print("\nStream (word, tag) pairs:\n")
for i, pair in enumerate(data.stream()):
    print("\t", pair)
    if i > 5: break
Stream (word, tag) pairs:

	 ('Mr.', 'NOUN')
	 ('Podger', 'NOUN')
	 ('had', 'VERB')
	 ('thanked', 'VERB')
	 ('him', 'PRON')
	 ('gravely', 'ADV')
	 (',', '.')

For both our baseline tagger and the HMM model we'll build, we need to estimate the frequency of tags & words from the frequency counts of observations in the training corpus. In the next several cells you will complete functions to compute the counts of several sets of counts. _____no_output_____## Step 2: Build a Most Frequent Class tagger
---

Perhaps the simplest tagger (and a good baseline for tagger performance) is to simply choose the tag most frequently assigned to each word. This "most frequent class" tagger inspects each observed word in the sequence and assigns it the label that was most often assigned to that word in the corpus._____no_output_____### IMPLEMENTATION: Pair Counts

Complete the function below that computes the joint frequency counts for two input sequences._____no_output_____

def pair_counts(sequences_A, sequences_B):
    """Return a dictionary keyed to each unique value in the first sequence list
    that counts the number of occurrences of the corresponding value from the
    second sequences list.

    For example, if sequences_A is tags and sequences_B is the corresponding
    words, then if 1244 sequences contain the word "time" tagged as a NOUN, then
    you should return a dictionary such that pair_counts[NOUN][time] == 1244
    """
    # TODO: Finish this function!
    d ={}
    for i in range(len(sequences_A)):
        z = zip(sequences_A[i], sequences_B[i])
        for t, w in z:
            if t in d:
                if w in d[t]:
                    d[t][w] += 1
                else:
                    d[t][w] = 1
            else:
                d[t] = {}
                d[t][w] = 1
    return d

# Calculate C(t_i, w_i)
emission_counts = pair_counts(data.Y, data.X)

assert len(emission_counts) == 12, \
       "Uh oh. There should be 12 tags in your dictionary."
assert max(emission_counts["NOUN"], key=emission_counts["NOUN"].get) == 'time', \
       "Hmmm...'time' is expected to be the most common NOUN."
HTML('
Your emission counts look good!
')_____no_output_____

### IMPLEMENTATION: Most Frequent Class Tagger

Use the `pair_counts()` function and the training dataset to find the most frequent class label for each word in the training data, and populate the `mfc_table` below. The table keys should be words, and the values should be the appropriate tag string.

The `MFCTagger` class is provided to mock the interface of Pomegranite HMM models so that they can be used interchangeably._____no_output_____

# Create a lookup table mfc_table where mfc_table[word] contains the tag label most frequently assigned to that word
from collections import namedtuple

FakeState = namedtuple("FakeState", "name")

class MFCTagger:
    # NOTE: You should not need to modify this class or  any of its methods
    missing = FakeState(name="<MISSING>")

    def __init__(self, table):
        self.table = defaultdict(lambda: MFCTagger.missing)
        self.table.update({word: FakeState(name=tag) for word, tag in table.items()})

    def viterbi(self, seq):
        """This method simplifies predictions by matching the Pomegranate viterbi() interface"""
        return 0., list(enumerate(["<start>"] + [self.table[w] for w in seq] + ["<end>"]))

# TODO: calculate the frequency of each tag being assigned to each word (hint: similar, but not
# the same as the emission probabilities) and use it to fill the mfc_table

word_counts = pair_counts(data.training_set.Y, data.training_set.X)

mfc_table = {}

tags = word_counts.keys()

for word in data.training_set.vocab:
    tag_count = [(tag, word_counts[tag][word]) for tag in tags if word in word_counts[tag]]
    tag = max(tag_count,key=itemgetter(1))[0]
    mfc_table[word] = tag

# DO NOT MODIFY BELOW THIS LINE
mfc_model = MFCTagger(mfc_table) # Create a Most Frequent Class tagger instance

assert len(mfc_table) == len(data.training_set.vocab), ""
assert all(k in data.training_set.vocab for k in mfc_table.keys()), ""
assert sum(int(k not in mfc_table) for k in data.testing_set.vocab) == 5521, ""
HTML('
Your MFC tagger has all the correct words!
')_____no_output_____

### Making Predictions with a Model
The helper functions provided below interface with Pomegranate network models & the mocked MFCTagger to take advantage of the [missing value](http://pomegranate.readthedocs.io/en/latest/nan.html) functionality in Pomegranate through a simple sequence decoding function. Run these functions, then run the next cell to see some of the predictions made by the MFC tagger._____no_output_____

def replace_unknown(sequence):
    """Return a copy of the input sequence where each unknown word is replaced
    by the literal string value 'nan'. Pomegranate will ignore these values
    during computation.
    """
    return [w if w in data.training_set.vocab else 'nan' for w in sequence]

def simplify_decoding(X, model):
    """X should be a 1-D sequence of observations for the model to predict"""
    _, state_path = model.viterbi(replace_unknown(X))
    return [state[1].name for state in state_path[1:-1]]  # do not show the start/end state predictions_____no_output_____

### Example Decoding Sequences with MFC Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, mfc_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', '<MISSING>', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADV', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

### Evaluating Model Accuracy

The function below will evaluate the accuracy of the MFC tagger on the collection of all sentences from a text corpus. _____no_output_____

def accuracy(X, Y, model):
    """Calculate the prediction accuracy by using the model to decode each sequence
    in the input X and comparing the prediction with the true labels in Y.

    The X should be an array whose first dimension is the number of sentences to test,
    and each element of the array should be an iterable of the words in the sequence.
    The arrays X and Y should have the exact same shape.

    X = [("See", "Spot", "run"), ("Run", "Spot", "run", "fast"), ...]
    Y = [(), (), ...]
    """
    correct = total_predictions = 0
    for observations, actual_tags in zip(X, Y):

        # The model.viterbi call in simplify_decoding will return None if the HMM
        # raises an error (for example, if a test sentence contains a word that
        # is out of vocabulary for the training set). Any exception counts the
        # full sentence as an error (which makes this a conservative estimate).
        try:
            most_likely_tags = simplify_decoding(observations, model)
            correct += sum(p == t for p, t in zip(most_likely_tags, actual_tags))
        except:
            pass
        total_predictions += len(observations)
    return correct / total_predictions_____no_output_____

#### Evaluate the accuracy of the MFC tagger
Run the next cell to evaluate the accuracy of the tagger on the training and test corpus._____no_output_____

mfc_training_acc = accuracy(data.training_set.X, data.training_set.Y, mfc_model)
print("training accuracy mfc_model: {:.2f}%".format(100 * mfc_training_acc))

mfc_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, mfc_model)
print("testing accuracy mfc_model: {:.2f}%".format(100 * mfc_testing_acc))

assert mfc_training_acc >= 0.955, "Uh oh. Your MFC accuracy on the training set doesn't look right."
assert mfc_testing_acc >= 0.925, "Uh oh. Your MFC accuracy on the testing set doesn't look right."
HTML('
Your MFC tagger accuracy looks correct!
')training accuracy mfc_model: 95.72%
testing accuracy mfc_model: 93.02%

## Step 3: Build an HMM tagger
---
The HMM tagger has one hidden state for each possible tag, and parameterized by two distributions: the emission probabilties giving the conditional probability of observing a given **word** from each hidden state, and the transition probabilities giving the conditional probability of moving between **tags** during the sequence.

We will also estimate the starting probability distribution (the probability of each **tag** being the first tag in a sequence), and the terminal probability distribution (the probability of each **tag** being the last tag in a sequence).

The maximum likelihood estimate of these distributions can be calculated from the frequency counts as described in the following sections where you'll implement functions to count the frequencies, and finally build the model. The HMM model will make predictions according to the formula:

$$t_i^n = \underset{t_i^n}{\mathrm{argmin}} \prod_{i=1}^n P(w_i|t_i) P(t_i|t_{i-1})$$

Refer to Speech & Language Processing [Chapter 10](https://web.stanford.edu/~jurafsky/slp3/10.pdf) for more information._____no_output_____### IMPLEMENTATION: Unigram Counts

Complete the function below to estimate the co-occurrence frequency of each symbol over all of the input sequences. The unigram probabilities in our HMM model are estimated from the formula below, where N is the total number of samples in the input. (You only need to compute the counts for now.)

$$P(tag_1) = \frac{C(tag_1)}{N}$$_____no_output_____

def unigram_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequence list that
    counts the number of occurrences of the value in the sequences list. The sequences
    collection should be a 2-dimensional array.

    For example, if the tag NOUN appears 275558 times over all the input sequences,
    then you should return a dictionary such that your_unigram_counts[NOUN] == 275558.
    """
    # TODO: Finish this function!
    d = {}
    # YOu can use a counter ...
    for sequence in sequences:
        for word in sequence:
            if word in d:
                d[word] += 1
            else:
                d[word] =  1

    return d

# TODO: call unigram_counts with a list of tag sequences from the training set
tag_unigrams = unigram_counts(data.training_set.Y)

assert set(tag_unigrams.keys()) == data.training_set.tagset, \
       "Uh oh. It looks like your tag counts doesn't include all the tags!"
assert min(tag_unigrams, key=tag_unigrams.get) == 'X', \
       "Hmmm...'X' is expected to be the least common class"
assert max(tag_unigrams, key=tag_unigrams.get) == 'NOUN', \
       "Hmmm...'NOUN' is expected to be the most common class"
HTML('
Your tag unigrams look good!
')_____no_output_____

### IMPLEMENTATION: Bigram Counts

Complete the function below to estimate the co-occurrence frequency of each pair of symbols in each of the input sequences. These counts are used in the HMM model to estimate the bigram probability of two tags from the frequency counts according to the formula: $$P(tag_2|tag_1) = \frac{C(tag_2|tag_1)}{C(tag_2)}$$
_____no_output_____

def bigram_counts(sequences):
    """Return a dictionary keyed to each unique PAIR of values in the input sequences
    list that counts the number of occurrences of pair in the sequences list. The input
    should be a 2-dimensional array.

    For example, if the pair of tags (NOUN, VERB) appear 61582 times, then you should
    return a dictionary such that your_bigram_counts[(NOUN, VERB)] == 61582
    """
    d = collections.Counter()
    # TODO: Finish this function!
    for sequence in sequences:
        for (p,c) in zip(sequence[:-1], sequence[1:]):
            d[(p,c)] += 1

    return dict(d)

# TODO: call bigram_counts with a list of tag sequences from the training set
tag_bigrams = bigram_counts(data.training_set.Y)

assert len(tag_bigrams) == 144, \
       "Uh oh. There should be 144 pairs of bigrams (12 tags x 12 tags)"
assert min(tag_bigrams, key=tag_bigrams.get) in [('X', 'NUM'), ('PRON', 'X')], \
       "Hmmm...The least common bigram should be one of ('X', 'NUM') or ('PRON', 'X')."
assert max(tag_bigrams, key=tag_bigrams.get) in [('DET', 'NOUN')], \
       "Hmmm...('DET', 'NOUN') is expected to be the most common bigram."
HTML('
Your tag bigrams look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Starting Counts
Complete the code below to estimate the bigram probabilities of a sequence starting with each tag._____no_output_____

def starting_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the beginning of
    a sequence.

    For example, if 8093 sequences start with NOUN, then you should return a
    dictionary such that your_starting_counts[NOUN] == 8093
    """
    # TODO: Finish this function!
    d = collections.Counter()
    for sequence in sequences:
        d[sequence[0]] += 1
    return dict(d)

# TODO: Calculate the count of each tag starting a sequence
tag_starts = starting_counts(data.training_set.Y)

assert len(tag_starts) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_starts, key=tag_starts.get) == 'X', "Hmmm...'X' is expected to be the least common starting bigram."
assert max(tag_starts, key=tag_starts.get) == 'DET', "Hmmm...'DET' is expected to be the most common starting bigram."
HTML('
Your starting tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Ending Counts
Complete the function below to estimate the bigram probabilities of a sequence ending with each tag._____no_output_____

def ending_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the end of
    a sequence.

    For example, if 18 sequences end with DET, then you should return a
    dictionary such that your_starting_counts[DET] == 18
    """
    # TODO: Finish this function!
    d = collections.Counter()
    for sequence in sequences:
        d[sequence[len(sequence)-1]] += 1
    return dict(d)

# TODO: Calculate the count of each tag ending a sequence
tag_ends = ending_counts(data.training_set.Y)

assert len(tag_ends) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_ends, key=tag_ends.get) in ['X', 'CONJ'], "Hmmm...'X' or 'CONJ' should be the least common ending bigram."
assert max(tag_ends, key=tag_ends.get) == '.', "Hmmm...'.' is expected to be the most common ending bigram."
HTML('
Your ending tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Basic HMM Tagger
Use the tag unigrams and bigrams calculated above to construct a hidden Markov tagger.

- Add one state per tag
    - The emission distribution at each state should be estimated with the formula: $P(w|t) = \frac{C(t, w)}{C(t)}$
- Add an edge from the starting state `basic_model.start` to each tag
    - The transition probability should be estimated with the formula: $P(t|start) = \frac{C(start, t)}{C(start)}$
- Add an edge from each tag to the end state `basic_model.end`
    - The transition probability should be estimated with the formula: $P(end|t) = \frac{C(t, end)}{C(t)}$
- Add an edge between _every_ pair of tags
    - The transition probability should be estimated with the formula: $P(t_2|t_1) = \frac{C(t_1, t_2)}{C(t_1)}$_____no_output_____

basic_model = HiddenMarkovModel(name="base-hmm-tagger")

# TODO: create states with emission probability distributions P(word | tag) and add to the model
# (Hint: you may need to loop & create/add new states)
states = {}
for tag in word_counts.keys():
    dd = {}
    for word in word_counts[tag].keys():
        dd[word] = 1.0  * (word_counts[tag][word]/tag_unigrams[tag])
    word_dist = DiscreteDistribution(dd)
    word_state = State(word_dist, name=tag)
    states[tag] = word_state

basic_model.add_states(list(states.values()))

# TODO: add edges between states for the observed transition frequencies P(tag_i | tag_i-1)
# (Hint: you may need to loop & add transitions
for start_tag in tag_starts.keys():
    basic_model.add_transition(basic_model.start, states[start_tag], tag_starts[start_tag] / (1.0 *len(data.training_set.X)))

for end_tag in tag_ends.keys():
    basic_model.add_transition(states[end_tag], basic_model.end, tag_ends[end_tag] / (1.0 * tag_unigrams[end_tag]))

for (prev, curr) in tag_bigrams.keys():
    basic_model.add_transition(states[prev], states[curr], tag_bigrams[(prev, curr)] / (1.0 * tag_unigrams[prev]))

# NOTE: YOU SHOULD NOT NEED TO MODIFY ANYTHING BELOW THIS LINE
# finalize the model
basic_model.bake()

assert all(tag in set(s.name for s in basic_model.states) for tag in data.training_set.tagset), \
       "Every state in your network should use the name of the associated tag, which must be one of the training set tags."
assert basic_model.edge_count() == 168, \
       ("Your network should have an edge from the start node to each state, one edge between every " +
        "pair of tags (states), and an edge from each state to the end node.")
HTML('
Your HMM network topology looks good!
')_____no_output_____hmm_training_acc = accuracy(data.training_set.X, data.training_set.Y, basic_model)
print("training accuracy basic hmm model: {:.2f}%".format(100 * hmm_training_acc))

hmm_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, basic_model)
print("testing accuracy basic hmm model: {:.2f}%".format(100 * hmm_testing_acc))

assert hmm_training_acc > 0.97, "Uh oh. Your HMM accuracy on the training set doesn't look right."
assert hmm_testing_acc > 0.955, "Uh oh. Your HMM accuracy on the training set doesn't look right."
HTML('
Your HMM tagger accuracy looks correct! Congratulations, you\'ve finished the project.
')training accuracy basic hmm model: 97.54%
testing accuracy basic hmm model: 95.95%

### Example Decoding Sequences with the HMM Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, basic_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

## Finishing the project
---

**Note:** **SAVE YOUR NOTEBOOK**, then run the next cell to generate an HTML copy. You will zip & submit both this file and the HTML copy for review.

_____no_output_____

!!jupyter nbconvert *.ipynb_____no_output_____

## Step 4: [Optional] Improving model performance
---
There are additional enhancements that can be incorporated into your tagger that improve performance on larger tagsets where the data sparsity problem is more significant. The data sparsity problem arises because the same amount of data split over more tags means there will be fewer samples in each tag, and there will be more missing data  tags that have zero occurrences in the data. The techniques in this section are optional.

- [Laplace Smoothing](https://en.wikipedia.org/wiki/Additive_smoothing) (pseudocounts)
    Laplace smoothing is a technique where you add a small, non-zero value to all observed counts to offset for unobserved values.

- Backoff Smoothing
    Another smoothing technique is to interpolate between n-grams for missing data. This method is more effective than Laplace smoothing at combatting the data sparsity problem. Refer to chapters 4, 9, and 10 of the [Speech & Language Processing](https://web.stanford.edu/~jurafsky/slp3/) book for more information.

- Extending to Trigrams
    HMM taggers have achieved better than 96% accuracy on this dataset with the full Penn treebank tagset using an architecture described in [this](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf) paper. Altering your HMM to achieve the same performance would require implementing deleted interpolation (described in the paper), incorporating trigram probabilities in your frequency tables, and re-implementing the Viterbi algorithm to consider three consecutive states instead of two.

### Obtain the Brown Corpus with a Larger Tagset
Run the code below to download a copy of the brown corpus with the full NLTK tagset. You will need to research the available tagset information in the NLTK docs and determine the best way to extract the subset of NLTK tags you want to explore. If you write the following the format specified in Step 1, then you can reload the data using all of the code above for comparison.

Refer to [Chapter 5](http://www.nltk.org/book/ch05.html) of the NLTK book for more information on the available tagsets._____no_output_____

import nltk
from nltk import pos_tag, word_tokenize
from nltk.corpus import brown

nltk.download('brown')
training_corpus = nltk.corpus.brown
training_corpus.tagged_sents()[0]_____no_output_____