# Notebook from joeweaver/get_to_know_phyloseq
Path: phyloseq_notebooks/phylo_demo.ipynb

# Getting to know Phyloseq
An interactive Jupyter-based tutorial prepared by __[Joseph E. Weaver](jeweave4@ncsu.edu)__ for __[BIT 477/577 Metagenomics](http://biotech.ncsu.edu/system/courses/pdfs/000/000/037/original/BIT477_F17_Metagenomics_slide.pdf)__ at North Carolina State University.

**Note: This is the interactive version. You can always refer to the filled in version to check your work.**_____no_output_____## Loading packages
`R` scripts usually call upon multiple packages which you have already installed through `CRAN` or with `bioconductor`.  This is how we tell the script what packages we'll be using._____no_output_____

library('phyloseq')
library('ggplot2')    #make good figures
library('assertthat') #used to test code in this interactive demo_____no_output_____

## Built-in Datasets
We need some example data to play with. Phyloseq comes with a few datasets:
* GlobalPatterns
* esophagus
* enterotype
* soilrep

If you're curious, you can use ```?``` to get more info about each dataset. For example, `?GlobalPatterns`. We can load any of them with `data(*data_name*)`, like below. _____no_output_____

data("GlobalPatterns")_____no_output_____

## Reading your own data
With phyloseq, if you can read the file, [you can import it](https://joey711.github.io/phyloseq/import-data.html). But, generally the path of least resistance is to have a biom file, metadata mapping file, and (ideally) a tree file.  A commented example is below.  Here, we'll be using the built-in data to keep the example portable._____no_output_____

#biom_file <- "otu_table_mc2_w_tax.biom"
#map_file <- "mapping_w_metadata.txt"
#tree_file <- "rep_set.tre"

##BIOM files used to be text based but have moved on to a binary format
##Older phyloseq versions only understand the text based format
##If you can't upgrad phloseq, search for 'read_hdf5_biom'

##actually import the biom, metadata, and tree file

##parseFunction changes Rank names by getting rid of greengenes prefixes
##e.g. from "g_Vibrio" to "vibrio"

#fullscale<-import_biom2b,parseFunction = parse_taxonomy_greengenes)
#map <-import_qiime_sample_data(map_file)
#treefile <- read_tree(tree_file)

##merge all the data into a phyloseq object, creatively named 'run1'
#run1 <-merge_phyloseq(fullscale,map,treefile)
_____no_output_____

## Getting to know your data
In many cases, you'll want to *get to know your data and then transform it to get the most illustrative graphs*. `Phyloseq` objects have a lot of *accessors* and *processors* for just these situations.  As a matter of fact, most of your time creating a data visualizing will usually be spent arranging data. The final graphics are pretty simple to generate, apart from aesthetic tweaking. Typing the name of the dataset (once loaded) will give you a high level summary._____no_output_____

GlobalPatterns_____no_output_____

### Accesors

An *accessor* lets you find out basic things about your data set.

For example, ```ntaxa``` will return the number of taxa in the taxonomy table.  Edit the code below with your guess for what   what you think ```ntaxa(GlobalPatterns)``` would return, based on the summary above._____no_output_____

#In R, you can assign values to a variable using the '<-' operator
# For example, to assign the value of '2' to the variable 'myVar':
# myVar <- 2

my_guess <- #<- Change 0 to your guess.
real_value <- ntaxa(GlobalPatterns)
assert_that(my_guess==real_value)_____no_output_____

Can you guess what accessor might list the number of samples?  It's listed in this image:

Go ahead and alter the code below to use the correct accessor so  that ```number_of_samples``` is correct._____no_output_____

number_of_samples <- #YOUR CODE HERE. Yes, reading ahead and typing '26' is cheating.
assert_that(26==number_of_samples)_____no_output_____

What if you wanted to know what those samples were named?_____no_output_____

sample_names(GlobalPatterns)_____no_output_____

The names themselve are not immediately informative in this case, but each sample has a bunch of other variables associated with it. You can get a list of those variables like so:_____no_output_____

sample_variables(GlobalPatterns)_____no_output_____

`'Description'` and `'SampleType'` Both look useful, we can get values store din those variables like so: _____no_output_____

get_variable(GlobalPatterns)$Description_____no_output_____

`'Description'` gives you a lot more information, but what if you were interested in the type of sample (`SampleType`)?_____no_output_____

sampleTypes <- # insert your call to an accessor here

expected <- factor(c("Soil", "Soil", "Soil", "Feces",  "Feces", "Skin", "Skin", "Skin", "Tongue", "Tongue", "Freshwater", "Freshwater", "Freshwater (creek)", "Freshwater (creek)", "Freshwater (creek)", "Ocean", "Ocean", "Ocean", "Sediment (estuary)", "Sediment (estuary)", "Sediment (estuary)", "Feces", "Feces", "Mock", "Mock", "Mock"))

are_equal(expected,sampleTypes)_____no_output_____

You may have noticed a few things in that test.  First, a lot of the samples have the same type, so there's a lot of repeated words. R is smart enough to understand that these are experimental factors and can interpret lists of *strings* (fancy word for text) as factors. Sometimes this gets in the way, but sometimes it is also useful.  For example, if you just wanted to see the levels for a factor:_____no_output_____

levels(get_variable(GlobalPatterns)$SampleType)_____no_output_____

### Processors

A *processor* allows you to manipulate your data in many ways, often slicing and dicing it or reducing in other ways, such as merging similar taxa together.

Imagine you were interested in only *Ocean* samples. You can use `subset_samples` to limit your dataset to only those samples which you want to analyze. `subset_samples` and other similar functions can take a phyloseq dataset, apply a *logical expression* to it, and return a new, reduced data set.  For *Ocean* samples, the logical expression is straightforward: `SampleType =="Ocean"`_____no_output_____

ocean_only<-subset_samples(GlobalPatterns, SampleType=="Ocean")_____no_output_____

You can get pretty fancy with your logical expression, for example, if  you wanted to get either samples with Skin or Soil_____no_output_____

skin_or_soil<-subset_samples(GlobalPatterns, SampleType %in% c("Skin","Soil"))_____no_output_____

The `%in%` operator is pretty powerful. A loosely wordered interpretation is that it tells you if a thing matches any of the stuff listed in the ```c()``` portion. If you haven't seen it before `c("fro","baz","buz")` is the `R`-ish way of **c**ombining a bunch of things into a vector or list. So, `skin_or_soil` is based on checking if the contents of each `SampleType` entry match either `"Skin"` or `"Soil"`.   As with most of the `R` stuff, for more explanations of `c` or `%in%` you can always use the inline help:_____no_output_____

?c_____no_output_____#being tricky here  %in% is weird, so we want
?match_____no_output_____#I happened to know match includes %in%, if we didn't know this, we'd get a nice hint with
#help.search(apropos="%in%")_____no_output_____

You can also exclude stuff, for example, if you wanted to exclude any creeks:_____no_output_____

no_creeks <-subset_samples(GlobalPatterns, !(SampleType %in% c("Freshwater (creek)")))_____no_output_____

Going back to our "ocean samples only" subset, if you type just the name of the new object, you'll see some interesting things. _____no_output_____

ocean_only_____no_output_____

 You'll notice the number of samples has decreased to just 3, which is what you'd expect.  But things like the number of taxa and the size of the phylogenetic tree are unchanged. This is because phyloseq is conversative and doesn't prune more than it has to. Among other things, this makes it really nice if you want to do a few subsets and then merge the results later. So, there's two things to handle here; first we'll do the simple part, verify that Ocean is the only level for your samples ocean_only.  _____no_output_____

ooLevels <- levels( #fill in the rest
are_equal('Ocean',ooLevels)_____no_output_____

Ok great, now let's handle all those empty taxa. We're going to do this in two parts. First we'll write the bit that does the work, then realizing that we'll want to do this a lot, I'll show you how to wrap it up into a fucntion._____no_output_____To remove empty taxa, we're going to use two built in phyloseq functions ```prune_taxa``` and ```taxa_sums```.

```taxa_sums``` is a really handy convenience method. It tells you the per-OTU total across all samples.  This is just a rowsum or colsum, but it takes advantage of knowing how your OTU table is laid out.  So, if you wanted to generate a list of all non-0 taxa in ```x```, you would do something like:

```taxa_sums(x) > 0```

```prune_taxa``` is a way of saying 'apart from my list of exceptions, remove all the taxa from this object. So, to keep only the taxa within x that are not empty, you could write:

```prune_taxa(taxa_sums(x) > 0, x)```

However, you're going to be doing something like this a lot, and one of the really good rules in writing code is Don't Repeat Yourself.  One way to avoid that is to wrap up code in reusable functions.  Below, were' going to wrap up our empty-taxa-removing code into a function called ```removeEmptyTaxa```._____no_output_____

removeEmptyTaxa <- function(x) {
   return(prune_taxa(taxa_sums(x) > 0, x))
}_____no_output_____

You can see how it works by running the code below._____no_output_____

removeEmptyTaxa(ocean_only)_____no_output_____

Now we still have only 3 samples, and it appears that only 5669 taxa were not empty.  One important point, look what happens if you get a summary of ```ocean_only```._____no_output_____

ocean_only_____no_output_____

We're back to 19216 taxa.  Actually, we're not back, this is the way ```ocean_only``` has always been.  This is because the way we wrote removeEmptyTaxa intentionally does not modify the data you pass to it as ```x```.  Sometimes you may want a different behaviour, but this is a really good way to avoid shooting yourself in the foot.  To 'preserve' the value from `removeEmptyTaxa`, you can assign it to a variable._____no_output_____

ocean_only.noEmpties <- removeEmptyTaxa(ocean_only)
#let's check
ocean_only.noEmpties_____no_output_____

Subsetting just doesn't have to be on metadata though, you can also use `subset_taxa` to llimit things to a particular taxon at any rank.  Why don't you go ahead and get a sample that's just the Firmicutes Phylum from Tongues and Feces and store it as 'GP.tf.firmOnly' after removing all empty taxa.  Hint, you can subset subsets._____no_output_____

# code here
# code here
# code here
GP.tf.firmOnly <-  #last bit of code here
GP.tf.firmOnly_____no_output_____are_equal(6,nsamples(GP.tf.firmOnly))
are_equal(6947,ntaxa(GP.tf.firmOnly))_____no_output_____

Two final things.  Right now all of our counts in raw abundances.  We'd often like to work with relative abundance. Here's a quick and dirty way to do that._____no_output_____

GP.tf.firmOnly  = transform_sample_counts(GP.tf , function(x) 100 * x/sum(x))_____no_output_____

Also, just to make things run quickly and not overload some graphs, we're going to restrict ourselves to OTUs with 25% or greater relative abdundance.  Note, this a horrible thing to do in a real analysis without a lot  justification._____no_output_____

GP.tf.firmOnly.top75<-prune_taxa(taxa_sums(GP.tf.firmOnly)>0.25,GP.tf.firmOnly)_____no_output_____

# Graphing (non-ordination)
Finally, we've looked at our data and manipulated it in a bunch of different ways. Let's make some pretty pictures.
```plot_bar``` is a built-in method that we're using here to split things up by their class and color code by order.  We're going ahead and giving it a title and telling it to break apart the graph into multiple graphs sorted by Sample and SampleType. We're also suppressing the legend for the fill colors because it's not very useful in this case._____no_output_____

#ggplot lets you play with plots nicely
theme_update(legend.position = "none")
plot_bar(GP.tf.firmOnly.top75 ,  x="Class",  fill="Order"  ,    title="Firmicutes only", facet_grid="Sample~SampleType")_____no_output_____

This is nice, but there's a lot you could do to make it more aesthetically pleasing. An example of this is remove the lines which split up OTUs within the class.  You can take advantage of ```tax_glom``` to do just this. The line below tells phyloseq to roll up anything which shares the same Order into one taxon._____no_output_____

glommedUp <- tax_glom(GP.tf.firmOnly.top75,taxrank = "Order")_____no_output_____plot_bar(glommedUp ,  x="Class",  fill="Order"  ,    title="Firmicutes only", facet_grid="Sample~SampleType")_____no_output_____

Let's say you wanted to dig deeper into just the Orders 'Clostridiales' and 'Bacteroidales'.  You could do this in two steps, first subset ```GP.tf.firmOnly.top75```, remove empty taxa, and save it as ```clostrid_bact```  Second, glom the taxa at the Family level, and plot the glommed data with by Order, with a fill color denoting Family._____no_output_____

#in this case, we still want a legend
theme_update(legend.position = "right")

clostrid_bact <- #code here

clostrid_bact_glom <- #code here
plot_bar(clostrid_bact_glom,  #code here_____no_output_____

The graph you just generated above should look something like this:
_____no_output_____## Ordination_____no_output_____Finally, we can get to those ordination plots everyone loves.  For an ordination, you need a distance measure and an ordiantion method, phyloseq supports many with the ```ordinate``` function.  The choice of distance and ordination method is highly dependent on the goals of the analysis and, for better or worse, what people in your discipline are expecting.  My go-to references for choosing methods are the  __[excellent overview by Palmer](http://ordination.okstate.edu/overview.htm)__ and Legendre, P., and L. Legendre. 1998. Numerical Ecology, 2nd English Edition. Elsevier, Amsterdam.

If you wanted to make an NMDS plot of the entire GlobalPatterns dataset, using Bray-Curtis distances:_____no_output_____

bray.pcoa <- ordinate(GlobalPatterns, distance = "bray", method = "NMDS")_____no_output_____

To actually get the plot, you need to send the results of your ordination *and* the dataset itself to ```plot_ordination```. Here, we're plotting the ordination we just did, split up by samples, and color-coded by SampleType._____no_output_____

plot_ordination(GlobalPatterns,bray.pcoa,"samples",color="SampleType")_____no_output_____

The great thing about these plots are that they can be tweaked after generation using standard methods the excellent ggplot2 package.

Let's increase the point sizes and make them a little transparent._____no_output_____

p<-plot_ordination(GlobalPatterns,bray.pcoa,"samples",color="SampleType")
p<-p+ geom_point(size=7, alpha=0.75)
p_____no_output_____

Go ahead and plot a PCoA graph using the unifrac distance metric, color coded by SampleType, with no alpha/transparency._____no_output_____

#code here_____no_output_____

You should've gotten something resembling this:
_____no_output_____## Making new categories
One of the great things about having control of the data is that you can reorganize categories.  What if you were just intersted in differences between water and non-water samples.  You can do that by using ```get_variable``` to find all the SampleTypes which match your criteria.  Then you can set upa  new variable, here called water.  To make it show up in the graph, you can still color code by SampleType, but you can set the shape to show if it's water or not.  _____no_output_____

water = get_variable(GlobalPatterns, "SampleType") %in% c("Freshwater", "Freshwater (creek)", "Ocean")
sample_data(GlobalPatterns)$water <- factor(water)

p<-plot_ordination(GlobalPatterns,unifrac.pcoa ,"samples",color="SampleType",shape="water")
p<-p+ geom_point(size=7, alpha=0.75)
p_____no_output_____

You could do the reverse (set the shape by SampleType and color by Water), but phyloseq will give you a hint why this is not a great idea with this particular data set. feel free to try that by modifing the above code if you're interested._____no_output_____One final excercise.  Try making a new category called 'human', which is TRUE when the sample is human derived. Indicate 'human' by shape and SampleType by color._____no_output_____

# code here_____no_output_____

You should've gotten something like this:
_____no_output_____# Learning more.

This is barely scratching the surface, but hopefully you've become comfortable with importing your data into phyloseq, examining and modifying it, and seeing the basic workflow for generating figures.

To learn more about phyloseq and the figures it generates, a great starting off point is the tutorial section straight from __[the creator's website](https://joey711.github.io/phyloseq/index.html)__ .  If you want to continue for refine your figures, the best place to keep learning is at the __[companion website](http://www.cookbook-r.com/Graphs/)__ for the _R Graphics Cookbook_._____no_output_____# Feedback
I would enjoy hearing that you liked this and would really love to hear about parts that need improvment.  Please don't hesistate to let me know at __[jeweave4@ncsu.edu](mailto:jeweave4@ncsu.edu?Subject=Phyloseq Jupyter Tutorial)_______no_output_____

Bioconductor phyloseq metagenomics ecology


# Notebook from 876lkj/APARENT
Path: analysis/folding/fold_array_seqs.ipynb

<code>
import sys
sys.path.append('/home/jlinder2/multistrand')

import nupack

import pandas as pd
import numpy as np

def get_struct(seq, temperature=25) :
    return nupack.mfe([seq], T=temperature, material='rna', sodium=0.05, magnesium=0.002)[0][0]

def get_mfe(seq, temperature=25) :
    return float(nupack.mfe([seq], T=temperature, material='rna', sodium=0.05, magnesium=0.002)[0][1])_____no_output_____seq_df = pd.read_csv('array_maxcut_seqs.csv', sep=',')_____no_output_____
mfes = []
mfes_w_pas = []
structs = []
structs_w_pas = []

for index, row in seq_df.iterrows() :
    if index % 1000 == 0 :
        print('Folding sequence ' + str(index))

    seq = row['master_seq']
    fold_pos = row['fold_pos']

    struct = get_struct(seq[56:fold_pos])
    struct_w_pas = get_struct(seq[50:fold_pos])
    mfe = get_mfe(seq[56:fold_pos])
    mfe_w_pas = get_mfe(seq[50:fold_pos])

    mfes.append(mfe)
    mfes_w_pas.append(mfe)
    structs.append(struct)
    structs_w_pas.append(struct_w_pas)

seq_df['mfe_w_pas'] = mfes_w_pas
seq_df['mfe'] = mfes
seq_df['struct'] = structs
seq_df['struct_w_pas'] = structs_w_pasFolding sequence 0
Folding sequence 1000
seq_df.to_csv('array_maxcut_seqs_folded.csv', sep=',', index=False)_____no_output_____df_var = pd.read_csv('array_cut_var_seqs.csv', sep=',')_____no_output_____mfes_var = []
structs_var = []

mfes_ref = []
structs_ref = []

for index, row in df_var.iterrows() :
    if index % 1000 == 0 :
        print('Folding sequence ' + str(index))

    var_seq = row['master_seq']
    ref_seq = row['wt_seq']

    struct_var = get_struct(var_seq[50:130].replace('T', 'U'))
    struct_ref = get_struct(ref_seq[50:130].replace('T', 'U'))
    mfe_var = get_mfe(var_seq[50:130].replace('T', 'U'))
    mfe_ref = get_mfe(ref_seq[50:130].replace('T', 'U'))

    mfes_var.append(mfe_var)
    mfes_ref.append(mfe_ref)
    structs_var.append(struct_var)
    structs_ref.append(struct_ref)

df_var['mfe_var'] = mfes_var
df_var['mfe_wt'] = mfes_ref
df_var['struct_var'] = structs_var
df_var['struct_wt'] = structs_refFolding sequence 0
Folding sequence 1000
Folding sequence 2000
Folding sequence 3000
Folding sequence 4000
Folding sequence 5000
Folding sequence 6000
Folding sequence 7000
Folding sequence 8000
Folding sequence 9000
Folding sequence 10000
df_var['delta_mfe'] = df_var['mfe_var'] - df_var['mfe_wt']_____no_output_____df_var.to_csv('array_cut_var_seqs_folded_long.csv', sep=',')_____no_output_____
</code>


# Notebook from sblunt/orbitize
Path: docs/tutorials/show-me-the-orbit.ipynb

# Show Me the Orbit

Sarah Blunt (2019)

### Read this First:

This is some code I wrote to define the `orbitize!` coordinate system and help users visualize orbits. To run it, first [install orbitize on your computer](https://orbitize.readthedocs.io/en/latest/installation.html), then run the following cell and scroll to the bottom of the page to interact with the visualization.

If you have questions, comments, or suggestions, please [raise an issue](https://github.com/sblunt/orbitize/issues) on the `orbitize!` GitHub page._____no_output_____

<code>
%matplotlib notebook

import numpy as np
import time

import astropy.units as u
import astropy.constants as consts

import matplotlib as mpl
mpl.style.use('bmh')
mpl.rcParams['figure.dpi'] = 250

import matplotlib.pyplot as plt
from matplotlib import animation, rcParams
from matplotlib.widgets import Slider
from matplotlib.collections import LineCollection

import orbitize.kepler

class Orbit(object):
    """ Initialize an interactive matplotlib figure illustrating how eccentricity,
        argument of periastron, inclination, and position angle of nodes affect an orbit.

    Args:
        sma: semimajor axis (default: 10) [au]
        tau: epoch of periastron passage, expressed as a fraction of orbital period past MJD=0 (default: 0.3)
        plx: parallax (default: 30) [mas]
        mtot: total mass of system (default: 1.3) [Msol]
        time_divs: number of epochs at which to compute position & radial velocity (default: 200)
        mplanet: planet mass (default: 0.01) [Msol]
    """

    def __init__(
        self, sma=10, tau=0.3, plx=30, mtot=1.3, time_divs=200, mplanet=0.01
    ):
        self.sma = sma
        self.ecc = 0.0
        self.inc = 0
        self.aop = 0
        self.pan = 0
        self.tau = tau
        self.plx = plx
        self.mtot = mtot
        self.mplanet = mplanet
        self.time_divs = time_divs

        max_ecc = 0.95 # max eccentrity for slider (for aesthetic purposes)

        # calculate related quantities
        self.per_yr = np.sqrt(self.sma**3/self.mtot)
        self.per_days = self.per_yr*365.25
        self.max_sep_mas = self.sma*(1+max_ecc)*self.plx
        self.min_sep_mas = self.sma*(1-max_ecc)*self.plx
        self.maxK = (
            np.sqrt(consts.G / (1.0 - max_ecc**2)) *
            (self.mplanet * u.Msun) / np.sqrt(self.mtot * u.Msun) /
            np.sqrt(self.sma * u.au)
        ).to(u.km/u.s).value

        # number of epochs at which to solve the orbit
        self.epochs = np.linspace(0, self.per_days, self.time_divs)

        # calculate orbital track
        self.ra, self.dec, self.rv = orbitize.kepler.calc_orbit(
            self.epochs, self.sma, self.ecc, np.radians(self.inc), np.radians(self.aop),
            np.radians(self.pan), self.tau, self.plx, self.mtot, mass_for_Kamp=self.mplanet
        )
        self.rv *= -1 # stellar RVs must be multiplied by -1

        # RA/Dec for inc=aop=pan=0 (useful for calculating z(t))
        self.ra0 = self.ra
        self.dec0 = self.dec

        # calculate z(t)
        self.z = np.sin(np.radians(self.inc)) * (
            np.cos(np.radians(self.aop))*self.ra0 +
            np.sin(np.radians(self.aop))*self.dec0
        ) * self.plx

        # set up plot
        rcParams.update({'font.size': 5})
        self.fig, self.ax = plt.subplots(figsize=(3,4), dpi=250)
        self.ax.set_ylim(-1.1*self.max_sep_mas, 1.1*self.max_sep_mas)
        self.ax.set_xlim(1.1*self.max_sep_mas, -1.1*self.max_sep_mas)
        self.ax.set_ylabel('$\Delta$Dec [mas]')
        self.ax.set_xlabel('$\Delta$RA [mas]')
        plt.subplots_adjust(left=0.33, bottom=0.5)

        # define RGB tuples for colors to use
        self.purple = (0.5019607843137255, 0.0, 0.5019607843137255, 1.0)
        self.lightpurp = (0.5019607843137255, 0.0, 0.5019607843137255, 0.1)

        # plot orbit track
        points = np.array([self.ra, self.dec]).T.reshape(-1, 1, 2)
        segments = np.concatenate([points[:-1], points[1:]], axis=1)
        self.color = np.empty(len(self.epochs), dtype=(float,4))
        self.color[self.z < 0] = self.lightpurp
        self.color[self.z >= 0] = self.purple

        self.lc = LineCollection(segments, color=self.color)
        _ = self.ax.add_collection(self.lc)

        # plot location of star
        self.star = self.ax.plot([0],[0], marker='*',
            color='violet', markersize=10, zorder=5, markeredgecolor='black', markeredgewidth=0.5
        )[0]

        # plot line of nodes
        lineofnodes_x = [
            -np.sin(np.radians(self.pan))*self.max_sep_mas,
            np.sin(np.radians(self.pan))*self.max_sep_mas
        ]
        lineofnodes_y = [
            -np.cos(np.radians(self.pan))*self.max_sep_mas,
            np.cos(np.radians(self.pan))*self.max_sep_mas
        ]
        self.lineofnodes = self.ax.plot(
            lineofnodes_x, lineofnodes_y,
            linestyle='--', color='k'
        )[0]

        # plot location of planet
        self.planet_loc = self.ax.plot(
            self.ra[0], self.dec[0],
            marker='o', markersize=5, zorder=5, color='violet', markeredgecolor='black', markeredgewidth=0.5
        )[0]

        # plot RV vs time
        self.ax_rv = plt.axes([0.33, .27, 0.57, 0.15])
        self.ax_rv.set_xlim(self.epochs[0]/365.25, self.epochs[-1]/365.25)
        self.ax_rv.set_ylim(-1.5*self.maxK, 1.5*self.maxK)
        self.ax_rv.set_xlabel('time [yr]')
        self.ax_rv.set_ylabel('Stellar RV [km s$^{-1}$]')
        self.planet_rv_track = self.ax_rv.plot(self.epochs/365.25, self.rv, color='purple', zorder=4)[0]
        self.ax_rv.plot([self.epochs[0]/365.25, self.epochs[-1]/365.25], [0,0], linestyle='--', color='k')
        self.planet_rv = self.ax_rv.plot(
            self.epochs[0]/365.25, self.rv[0], marker='o', markersize=5, zorder=5,
            color='violet', markeredgecolor='black', markeredgewidth=0.5
        )[0]

        # add inc slider
        ax_inc = plt.axes([0.35, .11, 0.52, 0.02])
        self.inc_slider = Slider(
            ax_inc, '$i$ [$\degree$]', 0, 180, valinit=0, valstep=1, facecolor='purple'
        )
        self.inc_slider.on_changed(self.update_inc)

        # add PAN slider
        ax_pan = plt.axes([0.35, .14, 0.52, 0.02])
        self.pan_slider = Slider(
            ax_pan, '$\Omega$ [$\degree$]', 0, 360, valinit=0, valstep=1, facecolor='purple'
        )
        self.pan_slider.on_changed(self.update_pan)

        # add AOP slider
        ax_aop = plt.axes([0.35, .17, 0.52, 0.02])
        self.aop_slider = Slider(
            ax_aop, '$\omega$ [$\degree$]', 0, 360, valinit=0, valstep=1, facecolor='purple'
        )
        self.aop_slider.on_changed(self.update_aop)

        # add eccentricity slider
        ax_ecc = plt.axes([0.35, .08, 0.52, 0.02])
        self.ecc_slider = Slider(
            ax_ecc, '$e$', 0, max_ecc, valinit=0, valstep=.01, facecolor='purple'
        )
        self.ecc_slider.on_changed(self.update_ecc)

    def update_orbit(self):

        # resolve Kepler's Eq
        self.ra, self.dec, self.rv = orbitize.kepler.calc_orbit(
            self.epochs, self.sma, self.ecc, np.radians(self.inc), np.radians(self.aop),
            np.radians(self.pan), self.tau, self.plx, self.mtot, mass_for_Kamp=self.mplanet
        )
        self.rv *= -1 # stellar RVs must be multiplied by -1

        # calculate z(t)
        self.z = np.sin(np.radians(self.inc)) * (
            -np.cos(np.radians(self.aop))*self.ra0 -
            np.sin(np.radians(self.aop))*self.dec0
        ) * self.plx

        # update orbit track
        points = np.array([self.ra, self.dec]).T.reshape(-1, 1, 2)
        segments = np.concatenate([points[:-1], points[1:]], axis=1)
        self.color[self.z < 0] = self.lightpurp
        self.color[self.z >= 0] = self.purple

        self.lc.set_segments(segments)
        self.lc.set_color(self.color)

    def update_ecc(self, val):

        self.ecc = val

        # recalculate ra0 & dec0
        self.ra0, self.dec0, _ = orbitize.kepler.calc_orbit(
            self.epochs, self.sma, self.ecc, 0, 0,
            0, self.tau, self.plx, self.mtot, mass_for_Kamp=self.mplanet
        )

        # update orbit
        self.update_orbit()
        self.planet_rv_track.set_data(self.epochs/365.25, self.rv)

    def update_inc(self, val):

        self.inc = val

        # update orbit
        self.update_orbit()
        self.planet_rv_track.set_data(self.epochs/365.25, self.rv)

    def update_aop(self, val):

        self.aop = val

        # update orbit
        self.update_orbit()
        self.planet_rv_track.set_data(self.epochs/365.25, self.rv)

    def update_pan(self, val):

        self.pan = val

        # update orbit
        self.update_orbit()

        lineofnodes_x = [
            -np.sin(np.radians(self.pan))*self.max_sep_mas,
            np.sin(np.radians(self.pan))*self.max_sep_mas
        ]
        lineofnodes_y = [
            -np.cos(np.radians(self.pan))*self.max_sep_mas,
            np.cos(np.radians(self.pan))*self.max_sep_mas
        ]

        # update line of nodes
        self.lineofnodes.set_data(lineofnodes_x, lineofnodes_y)

    def advance_orbit_pos(self, index):
        """ Moves the planet along its orbit to the next epoch.

        Args:
            index: new epoch index
        """

        # advance orbit position
        self.planet_loc.set_data(
            self.ra[index], self.dec[index]
        )

        self.planet_rv.set_data(
            self.epochs[index]/365.25, self.rv[index]
        )

        if self.z[index] < 0:
            self.planet_loc.set_alpha(0.1)
        else:
            self.planet_loc.set_alpha(1)

    def animate(self, interval=5):
        """ Continuously advance the planet's orbital location.

        Args:
            interval: delay between frames [milliseconds]
        """

        frames=np.arange(self.time_divs)
        self.anim = animation.FuncAnimation(
            self.fig, self.advance_orbit_pos, frames=np.arange(self.time_divs),
            interval=interval
        )

myOrbit = Orbit()
myOrbit.animate()_____no_output_____
</code>

# The Orbital Elements in Words:

- $\omega$ is the argument of periastron. This parameter rotates the vector connecting the star and the planet's periastron **within the orbital plane**.
- $\Omega$ is the position angle of nodes. This parameter rotates the line of nodes within the plane of the sky.
- $i$ is the orbit's inclination. This parameter rotates the orbital plane about its line of nodes (dashed black line in top panel).
- $e$ is the orbit's eccentricity. This parameter controls the orbit's shape and the planet's speed over time.
### In summary, $\Omega$ and $i$ define the orientation of the orbital plane in space. $\omega$ orients the orbit within the orbital plane, and $e$ controls the orbit's shape.

# What the Symbols and Colors Mean:
- The pink circle tracks the planet's position, and the pink star shows the primary's location. The top panel is shown in the reference frame where the star is stationary. In other words, the orbital track plotted shows the **relative RA and decl** of the planet and star over time.
- The darker portion of the orbital arc is in front of the plane of the sky (closer to Earth), and the lighter portion is behind the plane of the sky (farther from Earth). When the planet symbol becomes transparent, the planet is behind the plane of the sky.
- The black dashed line in the top panel is the line of nodes, or the line in the plane of the sky that intersects the orbit.

# A Caution to RV Astronomers:
- ``orbitize!`` defines $\omega$ to be the argument of periastron of the **planet's** orbit, wheras most RV codes (e.g. ``radvel``) define $\omega$ to be the argument of periastron of the **star's** orbit. In practice, this means that you need to add 180$^{\circ}$ to $\omega$ when comparing the outputs of RV codes and ``orbitize!``._____no_output_____


# Notebook from audreyemmely/pln
Path: pln_lista5.ipynb

_____no_output_____# **Resolução Lista 5**

---

Audrey Emmely Rodrigues Vasconcelos

Karen Nayara Gomes da Silva_____no_output_____

from google.colab import files
uploaded = files.upload()
# upando train_data e test_data_solution_____no_output_____from google.colab import drive
drive.mount('/content/drive')Mounted at /content/drive
!pip install umap-learnCollecting umap-learn
  Downloading umap-learn-0.5.2.tar.gz (86 kB)
     |████████████████████████████████| 86 kB 3.3 MB/s
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from umap-learn) (1.19.5)
Requirement already satisfied: scikit-learn>=0.22 in /usr/local/lib/python3.7/dist-packages (from umap-learn) (1.0.2)
Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from umap-learn) (1.4.1)
Requirement already satisfied: numba>=0.49 in /usr/local/lib/python3.7/dist-packages (from umap-learn) (0.51.2)
Collecting pynndescent>=0.5
  Downloading pynndescent-0.5.6.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 16.7 MB/s
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from umap-learn) (4.62.3)
Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.49->umap-learn) (0.34.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.49->umap-learn) (57.4.0)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from pynndescent>=0.5->umap-learn) (1.1.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.22->umap-learn) (3.0.0)
Building wheels for collected packages: umap-learn, pynndescent
  Building wheel for umap-learn (setup.py) ... done
  Created wheel for umap-learn: filename=umap_learn-0.5.2-py3-none-any.whl size=82708 sha256=fb415a4d2495b890f7b62af831b58edabacf113f5b2e4acdb669ad0aba03e8fe
  Stored in directory: /root/.cache/pip/wheels/84/1b/c6/aaf68a748122632967cef4dffef68224eb16798b6793257d82
  Building wheel for pynndescent (setup.py) ... done
  Created wheel for pynndescent: filename=pynndescent-0.5.6-py3-none-any.whl size=53943 sha256=49c54554470d560f3bf54394265f1f3947f17a7ff6b37af9ed9b2afca5f61f1a
  Stored in directory: /root/.cache/pip/wheels/03/f1/56/f80d72741e400345b5a5b50ec3d929aca581bf45e0225d5c50
Successfully built umap-learn pynndescent
Installing collected packages: pynndescent, umap-learn
Successfully installed pynndescent-0.5.6 umap-learn-0.5.2
import pandas as pd
import numpy as np
import random
import nltk
import re
import time
import umap
from IPython.display import HTML, display
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from matplotlib import pyplot as plt
from sklearn.metrics import plot_confusion_matrix, accuracy_score, precision_score, recall_score, f1_score
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.svm import SVC
from sklearn.decomposition import PCA
from bokeh.io import output_notebook, show
from bokeh.plotting import figure
from sklearn.manifold import TSNE
from sklearn.cluster import KMeans
from yellowbrick.cluster import KElbowVisualizer
from bokeh.palettes import Category20
from imblearn.under_sampling import RandomUnderSampler

import warnings
warnings.filterwarnings('ignore')_____no_output_____output_notebook() _____no_output_____nltk.download('stopwords');
nltk.download('punkt');[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
df = pd.read_csv('train_data.txt', sep =':::', header = None, engine = 'python', nrows = 10000)
df.columns = ['id', 'title', 'genre', 'description']
df2 = pd.read_csv('test_data_solution.txt', sep =':::', header = None, engine = 'python', nrows = 10000)
df2.columns = ['id', 'title', 'genre', 'description']_____no_output_____df3 = pd.concat([df, df2], ignore_index=True)
df3.drop('id', axis=1, inplace=True)
comedy = df3.loc[df3['genre'].str.contains('comedy')]
drama = df3.loc[df3['genre'].str.contains('drama')]
dataset = pd.concat([comedy, drama], ignore_index=True)_____no_output_____def preprocess(description):
  description = re.sub(r'\w*\d\w*', '', description) #remove todas as palavras que contêm números
  description = re.sub(r'[^a-zA-Z ]', '', description.lower())
  return re.sub(r'\s+', ' ', description) #retira espaços repetidos_____no_output_____dataset['processed_description'] = dataset.description.apply(preprocess)_____no_output_____stop_words = set(nltk.corpus.stopwords.words('english'))
def remove_stopwords(description):
    tokenized_text = nltk.word_tokenize(description, language='english')
    return " ".join([token for token in tokenized_text if token not in stop_words])_____no_output_____dataset['processed_description'] = dataset.processed_description.apply(remove_stopwords)_____no_output_____dataset_____no_output_____

## Questão 1

O objetivo dessa questão é desenvolver um buscador de documentos.

a) escolha e aplique um modelo do tipo word2vec a seus textos, compatível com o idioma de seus textos (inglês ou português).

b) escolha 5 palavras de consulta que não estejam em nenhum dos textos. Para cada palavra de consulta, encontre as 3 palavras de seu conjunto de textos mais parecidas com cada uma das palavras de consulta e exiba os documentos onde estas palavras aparecem.

c) Seja d um documento da base e w uma palavra de consulta. Implemente o
seguinte algoritmo para buscar documentos:
1. Encontre d10(w): a lista com as 10 palavras mais parecidas com w em
um certo documento d.
2. Para cada documento d, calcule a distância média DM10(w) entre w e
as palavras de d10(w).
3. Recupere os 3 documentos da base cuja DM10(w) é menor.

d) aplique o algoritmo para buscar documentos em 5 palavras distintas, e exiba
os 3 documentos mais próximos de cada um._____no_output_____

import gensim
from gensim.models import KeyedVectors_____no_output_____def vocabulary(descriptions):
    vocab = ' '.join([description for description in descriptions])
    return sorted(set(vocab.split(' ')))

processed_descriptions_vocab = vocabulary(dataset.processed_description.values)_____no_output_____import urllib.request

word_url = "https://www.mit.edu/~ecprice/wordlist.10000"
response = urllib.request.urlopen(word_url)
long_txt = response.read().decode()
words = long_txt.splitlines()

get_words = [word for word in words if word not in processed_descriptions_vocab]_____no_output_____search_five_words = random.sample(get_words, 5)
search_five_words_____no_output_____print(search_five_words in processed_descriptions_vocab)False
modelPath = open('/content/drive/MyDrive/skip_s300.txt', 'r', encoding='iso8859-1')_____no_output_____def sim_words(search_word, n):
  similar = []
  for word in processed_descriptions_vocab:
    try:
      similarity = embedding.similarity(word, search_word)
    except:
      similarity = -1
    similar.append(similarity)
  similar = np.argsort(similar)[-(n+1):-1]
  similar = similar[::-1]
  return [processed_descriptions_vocab[i] for i in similar]_____no_output_____print('Modelo SKIP-GRAM 300')
embedding = KeyedVectors.load_word2vec_format(modelPath)
print('Palavras mais similares')
similar_words = []
for search_word in search_five_words:
  sim_search_word = sim_words(search_word, 3)
  print(f'\t{search_word} -> {str(sim_search_word)}')
  similar_words.append({'search_word': search_word, 'similar_words': sim_search_word})Modelo SKIP-GRAM 300
Palavras mais similares
	sprint -> ['challenge', 'volvo', 'multimedia']
	mongolia -> ['laos', 'ghana', 'yunnan']
	cdna -> ['nested', 'bacterial', 'antagonist']
	projectors -> ['chopped', 'noses', 'besieged']
	refine -> ['struggled', 'withstand', 'consented']
def words_list(search_word):
  for word_list in similar_words:
    if search_word == word_list['search_word']:
      return word_list['similar_words']

def documents(word_list):
  df = []
  for word in word_list:
    dataframe = dataset[dataset.processed_description.apply(lambda description: word in description.split(' '))]
    df.append(dataframe[['title', 'description']])
  return pd.concat(df).drop_duplicates().reset_index(drop = True)_____no_output_____for search_word in search_five_words:
  word_list = words_list(search_word=search_word)
  search_docs = documents(word_list)
  search_docs['description'] = search_docs['description'].str[:100] + '...'
  display(HTML(f'
Modelo SKIP-GRAM 300

Palavra de busca [{search_word}]

Palavras similares {word_list}
'))
  display(HTML(search_docs.head().to_html()))_____no_output_____mean_vectors = []
for idx, row in dataset.iterrows():
  vectors = []
  for word in row['processed_description']:
    if word in embedding.wv.vocab:
      vectors.append(embedding.get_vector(word))
  mean_vector = np.sum(vectors, axis=0) / len(vectors)
  mean_vectors.append(mean_vector)

dataset['mean_vector'] = mean_vectors_____no_output_____print(dataset['mean_vector'])_____no_output_____for word in search_five_words:
  results = cosine_similarity([embedding.get_vector(word)] + mean_vectors)[0]
  results = results[1:]
  results[results.argmax()]

  print('\nPalavra <{}>: '.format(word), dataset['description'][results.argmax()])
  print()
Palavra <sprint>:   In 1924 Estie comes to Hollywood to become an actress, but the dog, that followed her becomes the star. But Hollywood has its own rules of sucess.

Palavra <mongolia>:   When Giacomo D'Amato is released from jail he is immediately employed by Magda, in a plot to secure her the heritage of the count. But there are other players in this game too, and Giacomo has his own agenda.

Palavra <cdna>:   Mail to author for translation Karasz Nelli apja halala utan hozzamegy Takaro Sanyihoz, a falu legmodosabb gazdajanak fiahoz. Bar nem szereti a ferfit, a hazassagtol tamaszt remel. Az evek multaval azonban ferje kozeledeseit egyre nehezebben viseli el, s olyan tarthatatlanna valik kapcsolatuk, hogy egy izben el is hagyja kislanyaval egyutt. Az iszony vegul vegzetes helyzetbe taszitja az asszonyt.

Palavra <projectors>:   In Episode one we are introduced to the contestants, and it turns out not all of them know what they are getting themselves into. From a mother and son team where the son bears an uncanny resemblance to Justin Bieber to a boy band trying to make it big, these contestants from all over North America tell us if they think they have what it takes to be famous.

Palavra <refine>:   Hilarious sequel to the movies 'Yes Sir' and 'Yes Sir 2,' about the Nationalist soldiers of the Taiwan Army. This is equivalent in the US as M*A*S*H 3. Only this time, instead of being obedient as they were in 'Yes Sir' and 'Yes Sir 2,' it's just the opposite.

## Questão 2
Aplicando a representação Doc2Vec:

a) Resolva a segunda questão da 3a lista com esta nova representação e compare com os resultados obtidos anteriormente.

b) Resolva a segunda e terceira questão da 4a lista com esta nova representação e compare com os resultados obtidos anteriormente.
_____no_output_____

from gensim.models.doc2vec import Doc2Vec, TaggedDocument_____no_output_____paragraphs = dataset.processed_description.values
paragraphs = [paragraph.split(' ') for paragraph in paragraphs]_____no_output_____docs = [TaggedDocument(doc, [i]) for i, doc in enumerate(paragraphs)]_____no_output_____doc_embedding = Doc2Vec(docs, vector_size = 300, window = 10, min_count = 2, workers = 8)_____no_output_____# X = np.array([doc_embedding[i] for i, description in enumerate(dataset.processed_description.values)])
X = [doc_embedding.infer_vector(u) for u in dataset['processed_description']]_____no_output_____lbl_to_ind = {
    ' comedy ': 0, ' drama ': 1,
}
y = [lbl_to_ind[l] for l in dataset.genre.values[:7739]]_____no_output_____X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.3)_____no_output_____rus = RandomUnderSampler()
X_res, y_res = rus.fit_resample(X_train, y_train)_____no_output_____print(pd.Series(y_res).value_counts())1    1917
0    1917
dtype: int64
classifier = LogisticRegression().fit(X_res, y_res)_____no_output_____y_pred = classifier.predict(X_test)_____no_output_____plot_confusion_matrix(classifier, X_test, y_test)
plt.show()_____no_output_____accuracy_lr = accuracy_score(y_test, y_pred)
precision_lr = precision_score(y_test, y_pred)
recall_lr = recall_score(y_test, y_pred)
f1_lr = f1_score(y_test, y_pred)

print("Modelo Regressão Logística\nAcurácia: {}\nPrecisão: {}\nRecall: {}\nF1 Score: {}\n".format(accuracy_lr, precision_lr, recall_lr, f1_lr))Modelo Regressão Logística
Acurácia: 0.512489233419466
Precisão: 0.6998982706002035
Recall: 0.45114754098360654
F1 Score: 0.5486443381180224

classifier = SVC().fit(X_res, y_res)_____no_output_____y_pred = classifier.predict(X_test)_____no_output_____plot_confusion_matrix(classifier, X_test, y_test)
plt.show()_____no_output_____accuracy_svm = accuracy_score(y_test, y_pred)
precision_svm = precision_score(y_test, y_pred)
recall_svm = recall_score(y_test, y_pred)
f1_svm = f1_score(y_test, y_pred)

print("Modelo SVM\nAcurácia: {}\nPrecisão: {}\nRecall: {}\nF1 Score: {}\n".format(accuracy_svm, precision_svm, recall_svm, f1_svm))Modelo SVM
Acurácia: 0.49138673557278206
Precisão: 0.7014051522248244
Recall: 0.39278688524590166
F1 Score: 0.5035729298024381

lr_results = [accuracy_lr, precision_lr, recall_lr, f1_lr]
svm_results = [accuracy_svm,precision_svm, recall_svm,f1_svm]

data_results2 = {
    'Regressão Logística': lr_results,
    'SVM': svm_results,
}

index_values2 = ['accuracy', 'precision', 'recall', 'f1']
pd.set_option('display.max_columns', None)

df_results2 = pd.DataFrame(data_results2, index=index_values2)

df_results2_____no_output_____X_proj = PCA(n_components=.95).fit_transform(X)
print(X_proj.shape[1])2
visualizer = KElbowVisualizer(KMeans(), k=(10, 25), metric='distortion')
visualizer.fit(X_proj)
visualizer.show() _____no_output_____k = 15
kmeans = KMeans(n_clusters=k)
kmeans.fit(X_proj)_____no_output_____start = time.time()
X_tsne = TSNE(n_components=2).fit_transform(X_proj)
duration = time.time() - start_____no_output_____print('%s segundos' % duration)90.64402151107788 segundos
palette = Category20[k]_____no_output_____def scatter_plot(X, labels):
    scatter_plot = figure(plot_width=1000, plot_height=500)
    scatter_plot.circle(X[:, 0], X[:, 1], size=10, line_color=[palette[l] for l in labels], fill_color=[palette[l] for l in labels], fill_alpha=.8)
    show(scatter_plot)

scatter_plot(X_tsne, kmeans.labels_)_____no_output_____start = time.time()
X_umap = umap.UMAP().fit_transform(X_proj)
duration = time.time() - start_____no_output_____print('%s segundos' % duration)42.73649334907532 segundos
scatter_plot(X_umap, kmeans.labels_)_____no_output_____

**Referências**

[Repositório de Word Embeddings do NILC](http://www.nilc.icmc.usp.br/embeddings)

[Word2Vec — CBOW & Skip-gram : Algorithmic Optimizations](https://medium.com/analytics-vidhya/word2vec-cbow-skip-gram-algorithmic-optimizations-921d6f62d739)

[Documentation - gensim](https://radimrehurek.com/gensim/auto_examples/index.html)

[Pln repository](https://github.com/valeriojr/ecom063-pln/blob/master/Processamento%20de%20linguagem%20natural%20-%205%C2%AA%20lista%20de%20exerc%C3%ADcios.ipynb)

[NLP: Word Embedding Techniques Demystified](https://towardsdatascience.com/nlp-embedding-techniques-51b7e6ec9f92#:~:text=Doc2Vec%20is%20another%20widely%20used,every%20document%20in%20the%20corpus.)

[A gentle introduction to Doc2Vec](https://medium.com/wisio/a-gentle-introduction-to-doc2vec-db3e8c0cce5e)

[Como lidar com dados desbalanceados?](https://sigmoidal.ai/como-lidar-com-dados-desbalanceados/)_____no_output_____


# Notebook from aniellodesanto/Utah_5300
Path: notebooks_first_steps.ipynb

# Congratulations!

You have successfully loaded a Python notebook.
Click on the cell below and press the *Run* button on the left of the cell to execute the Python code._____no_output_____

print("Your Python setup is working correctly!")_____no_output_____

# Learning about Python notebooks

Before you can start to learn about Python, you have to master the usage of Python notebooks.
Don't worry, it's very easy and will be like second nature to you after a week or two.
This notebook covers everything you need to know to get going._____no_output_____## What is a Python notebook, and why do we use it?

A Python notebook is very different from the environments professional programmers use to develop software.
It is more like a scratch pad where code can be interwoven with normal text, images, links to videos, and so on.
While a programmer would rarely need those things, they are very useful for a beginner who is still learning the ropes.
For this course, you can think of the notebooks as chapters of a highly interactive textbook.
Any code in this "textbook" can be run to see what it does.
And you can add your own code to run it.
That's the really cool thing about Python notebooks.
It's as if you could scribble your code directly on the pages of a textbook and run it right there.

## How will notebooks be used in this course?

The notebooks are both your **primary Python reference** and **your Python homeworks**.
Each notebook contains a number of exercises.
For each exercise, you write down your answer right below it in the notebook and then save the notebook.
That's it.

As mentioned in the Syllabus, you can either run them locally on your computers, or load them remotely using Google Colab.
As you are reading this, you should have got the general gist of one of the two options.
 But if you want, you can check out this short introductory video about how Colab works (if you don't see the youtube window, run the cell below with Python code)._____no_output_____

from IPython.display import IFrame
IFrame('https://www.youtube.com/embed/inN8seMm7UI', width=700, height=350)_____no_output_____

Since we are using Colab, all your files are saved authomatically in the cloud.
I still recommend that you manually save before closing the notebook, just to be sure nothing is lost.
You can also create copies of your notebook, if you want to be creative and experiment without losing some finished solution.

Once you are done with an assignment, you can download your finished product and email it to me.
You just write your answers directly into the notebook, save it, and submit via email.

## Notebooks: How exactly do they work?

Jupyter notebooks mix text ("markdown") with code boxes. You can write and run Python code in code boxes, and you can write and format text in markdown boxes.

To execute Python code in a code box, click on a code box and hit "Run", for example on the box below. _____no_output_____

print("Hit 'Run' to execute this Python code.")_____no_output_____

You can also use a keyboard shortcut to run the code in the box you are currently in. The command for that is

Shift+Enter._____no_output_____## Getting a new empty box

You can always get additional boxes in a Jupyter Notebook. Hit ESC B to get a new box below the one you are currently in. (Or use ESC A to get a new box above.) Then hit "Enter" to get out of command mode.

(More info: ESC gets you into command mode, and then B means "make a new box below." Enter gets you out of command mode, and into the mode where you can type in the current box.)_____no_output_____## Code versus Markdown

You can see in the top menu whether the box you are currently in is "Code" or "Markdown". In that drop-down menu, you can switch the current box between Code and Markdown.

Or you can hit ESC M to make the current box Markdown. (Then remember to hit Enter to get out of command mode). ESC Y makes the current box Code. _____no_output_____## Prettier Markdown

In Markdown mode, ```*surrounding text by a single star on either side*``` makes it *italic*. ```**Two stars**``` make it **boldface.**

Putting triple back quotes around text makes it look like ```code```.

When you put a single hash mark at the beginning of the line, the following text will be formatted as a big heading. With two hash marks, you get a slightly less big heading, and so on.
_____no_output_____## More Jupyter Notebook tricks

... can be found here: https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/_____no_output_____## Try it Yourself!

**Exercise.**
The cell below contains a short piece of Python code. Do not worry about understanding the code! We are justy practicing the environment.

Click on the cell to highlight it, then run it to see the output.
If you are in Colab, you can also run a cell by clicking on the *RUN* icon left of the code, or with the keyboard combination *shift* + *return*.

Now click it again, move the blinking cursor to the line with `x = 5`, and change the value from 5 to 10.
Run the cell again to see the new output._____no_output_____

x = 5
print(x*2 + 1)_____no_output_____

If you've done everything correctly, you should now see 21 under the code cell.
At this point, save your progress by clicking on save under the File drop down menu._____no_output_____To edit a text cell (like the one you're currently reading, you have to double-click it (in CoLab, you can also push the edit button next to it).
Guess what's your next exercise?_____no_output_____**Exercise.**
Double click the text cell below to make it editable.
Enter a short piece of text of your choice (a particularly creative one might brighten the day of your Instructor).
Then run the cell like a code cell, and observe how it now looks a lot nicer now.
Don't forget to save your notebook._____no_output_____*Replace this text by your own message*_____no_output_____**First Submission** Now that you are done, try to submit this as an assignment. Download the notebook on your computer: go to File, and select *Download .pynb*. I suggest saving all your local files in the same folder, for  your records. Then, upload the resulting file in Canvas under Assignment 0.

Note: this won't be graded. I just need to make sure we are all on the same page re: accessing and submitting notebooks._____no_output_____


# Notebook from stu1130/djl
Path: jupyter/rank_classification_using_BERT_on_Amazon_Review.ipynb

# Rank Classification using BERT on Amazon Review dataset

## Introduction

In this tutorial, you learn how to train a rank classification model using [Transfer Learning](https://en.wikipedia.org/wiki/Transfer_learning). We will use a pretrained DistilBert model to train on the Amazon review dataset.

## About the dataset and model

[Amazon Customer Review dataset](https://s3.amazonaws.com/amazon-reviews-pds/readme.html) consists of all different valid reviews from amazon.com. We will use the "Digital_software" category that consists of 102k valid reviews. As for the pre-trained model, use the DistilBERT[[1]](https://arxiv.org/abs/1910.01108) model. It's a light-weight BERT model already trained on [Wikipedia text corpora](https://en.wikipedia.org/wiki/List_of_text_corpora), a much larger dataset consisting of over millions text. The DistilBERT served as a base layer and we will add some more classification layers to output as rankings (1 - 5).

Amazon Review example

We will use review body as our data input and ranking as label.

## Pre-requisites
This tutorial assumes you have the following knowledge. Follow the READMEs and tutorials if you are not familiar with:
1. How to setup and run [Java Kernel in Jupyter Notebook](https://github.com/awslabs/djl/blob/master/jupyter/README.md)
2. Basic components of Deep Java Library, and how to [train your first model](https://github.com/awslabs/djl/blob/master/jupyter/tutorial/02_train_your_first_model.ipynb).

## Getting started
Load the Deep Java Libarary and its dependencies from Maven. In here, you can choose between MXNet or PyTorch. MXNet is enabled by default. You can uncomment PyTorch dependencies and comment MXNet ones to switch to PyTorch._____no_output_____

// %mavenRepo snapshots https://oss.sonatype.org/content/repositories/snapshots/

%maven ai.djl:api:0.10.0
%maven ai.djl:basicdataset:0.10.0
%maven org.slf4j:slf4j-api:1.7.26
%maven org.slf4j:slf4j-simple:1.7.26

// See https://github.com/awslabs/djl/blob/master/mxnet/mxnet-engine/README.md
// MXNet
%maven ai.djl.mxnet:mxnet-model-zoo:0.10.0
%maven ai.djl.mxnet:mxnet-native-auto:1.7.0-backport

// PyTorch
// %maven ai.djl.pytorch:pytorch-model-zoo:0.10.0
// %maven ai.djl.pytorch:pytorch-native-auto:1.7.1_____no_output_____

Now let's import the necessary modules:_____no_output_____

import ai.djl.*;
import ai.djl.engine.Engine;
import ai.djl.basicdataset.tabular.CsvDataset;
import ai.djl.basicdataset.utils.DynamicBuffer;
import ai.djl.inference.Predictor;
import ai.djl.metric.Metrics;
import ai.djl.modality.Classifications;
import ai.djl.modality.nlp.SimpleVocabulary;
import ai.djl.modality.nlp.bert.BertFullTokenizer;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.nn.Activation;
import ai.djl.nn.Block;
import ai.djl.nn.SequentialBlock;
import ai.djl.nn.core.Linear;
import ai.djl.nn.norm.Dropout;
import ai.djl.repository.zoo.*;
import ai.djl.training.*;
import ai.djl.training.dataset.Batch;
import ai.djl.training.dataset.RandomAccessDataset;
import ai.djl.training.evaluator.Accuracy;
import ai.djl.training.listener.SaveModelTrainingListener;
import ai.djl.training.listener.TrainingListener;
import ai.djl.training.loss.Loss;
import ai.djl.training.util.ProgressBar;
import ai.djl.translate.*;
import java.io.IOException;
import java.nio.file.Paths;
import java.util.List;
import org.apache.commons.csv.CSVFormat;

System.out.println("You are using: " + Engine.getInstance().getEngineName() + " Engine");_____no_output_____

## Prepare Dataset

First step is to prepare the dataset for training. Since the original data was in TSV format, we can use CSVDataset to be the dataset container. We will also need to specify how do we want to preprocess the raw data. For BERT model, the input data are required to be tokenized and mapped into indices based on the inputs. In DJL, we defined an interface called Fearurizer, it is designed to allow user customize operation on each selected row/column of a dataset. In our case, we would like to clean and tokenize our sentencies. So let's try to implement it to deal with customer review sentencies._____no_output_____

final class BertFeaturizer implements CsvDataset.Featurizer {

    private final BertFullTokenizer tokenizer;
    private final int maxLength; // the cut-off length

    public BertFeaturizer(BertFullTokenizer tokenizer, int maxLength) {
        this.tokenizer = tokenizer;
        this.maxLength = maxLength;
    }

    /** {@inheritDoc} */
    @Override
    public void featurize(DynamicBuffer buf, String input) {
        SimpleVocabulary vocab = tokenizer.getVocabulary();
        // convert sentence to tokens (toLowerCase for uncased model)
        List<String> tokens = tokenizer.tokenize(input.toLowerCase());
        // trim the tokens to maxLength
        tokens = tokens.size() > maxLength ? tokens.subList(0, maxLength) : tokens;
        // BERT embedding convention "[CLS] Your Sentence [SEP]"
        buf.put(vocab.getIndex("[CLS]"));
        tokens.forEach(token -> buf.put(vocab.getIndex(token)));
        buf.put(vocab.getIndex("[SEP]"));
    }
}_____no_output_____

Once we got this part done, we can apply the `BertFeaturizer` into our Dataset. We take `review_body` column and apply the Featurizer. We also pick `star_rating` as our label set. Since we go for batch input, we need to tell the dataset to pad our data if it is less than the `maxLength` we defined. `PaddingStackBatchifier` will do the work for you._____no_output_____

CsvDataset getDataset(int batchSize, BertFullTokenizer tokenizer, int maxLength) {
    String amazonReview =
            "https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Software_v1_00.tsv.gz";
    float paddingToken = tokenizer.getVocabulary().getIndex("[PAD]");
    return CsvDataset.builder()
            .optCsvUrl(amazonReview) // load from Url
            .setCsvFormat(CSVFormat.TDF.withQuote(null).withHeader()) // Setting TSV loading format
            .setSampling(batchSize, true) // make sample size and random access
            .addFeature(
                    new CsvDataset.Feature(
                            "review_body", new BertFeaturizer(tokenizer, maxLength)))
            .addLabel(
                    new CsvDataset.Feature(
                            "star_rating", (buf, data) -> buf.put(Float.parseFloat(data) - 1.0f)))
            .optDataBatchifier(
                    PaddingStackBatchifier.builder()
                            .optIncludeValidLengths(false)
                            .addPad(0, 0, (m) -> m.ones(new Shape(1)).mul(paddingToken))
                            .build()) // define how to pad dataset to a fix length
            .build();
}_____no_output_____

## Construct your model

We will load our pretrained model and prepare the classification. First construct the `criteria` to specify where to load the embedding (DistiledBERT), then call `loadModel` to download that embedding with pre-trained weights. Since this model is built without classification layer, we need to add a classification layer to the end of the model and train it. After you are done modifying the block, set it back to model using `setBlock`.

### Load the word embedding

We will download our word embedding and load it to memory (this may take a while)_____no_output_____

// MXNet base model
String modelUrls = "https://resources.djl.ai/test-models/distilbert.zip";
if ("PyTorch".equals(Engine.getInstance().getEngineName())) {
    modelUrls = "https://resources.djl.ai/test-models/traced_distilbert_wikipedia_uncased.zip";
}

Criteria<NDList, NDList> criteria = Criteria.builder()
        .optApplication(Application.NLP.WORD_EMBEDDING)
        .setTypes(NDList.class, NDList.class)
        .optModelUrls(modelUrls)
        .optProgress(new ProgressBar())
        .build();
ZooModel<NDList, NDList> embedding = ModelZoo.loadModel(criteria);_____no_output_____

### Create classification layers

Then let's build a simple MLP layer to classify the ranks. We set the output of last FullyConnected (Linear) layer to 5 to get the predictions for star 1 to 5. Then all we need to do is to load the block into the model. Before applying the classification layer, we also need to add text embedding to the front. In our case, we just create a Lambda function that do the followings:

1. batch_data (batch size, token indices) -> batch_data + max_length (size of the token indices)
2. generate embedding_____no_output_____

Predictor<NDList, NDList> embedder = embedding.newPredictor();
Block classifier = new SequentialBlock()
        // text embedding layer
        .add(
            ndList -> {
                NDArray data = ndList.singletonOrThrow();
                NDList inputs = new NDList();
                long batchSize = data.getShape().get(0);
                float maxLength = data.getShape().get(1);

                if ("PyTorch".equals(Engine.getInstance().getEngineName())) {
                    inputs.add(data.toType(DataType.INT64, false));
                    inputs.add(data.getManager().full(data.getShape(), 1, DataType.INT64));
                    inputs.add(data.getManager().arange(maxLength)
                               .toType(DataType.INT64, false)
                               .broadcast(data.getShape()));
                } else {
                    inputs.add(data);
                    inputs.add(data.getManager().full(new Shape(batchSize), maxLength));
                }
                // run embedding
                try {
                    return embedder.predict(inputs);
                } catch (TranslateException e) {
                    throw new IllegalArgumentException("embedding error", e);
                }
            })
        // classification layer
        .add(Linear.builder().setUnits(768).build()) // pre classifier
        .add(Activation::relu)
        .add(Dropout.builder().optRate(0.2f).build())
        .add(Linear.builder().setUnits(5).build()) // 5 star rating
        .addSingleton(nd -> nd.get(":,0")); // Take [CLS] as the head
Model model = Model.newInstance("AmazonReviewRatingClassification");
model.setBlock(classifier);_____no_output_____

## Start Training

Finally, we can start building our training pipeline to train the model.

### Creating Training and Testing dataset

Firstly, we need to create a voabulary that is used to map token to index such as "hello" to 1121 (1121 is the index of "hello" in dictionary). Then we simply feed the vocabulary to the tokenizer that used to tokenize the sentence. Finally, we just need to split the dataset based on the ratio.

Note: we set the cut-off length to 64 which means only the first 64 tokens from the review will be used. You can increase this value to achieve better accuracy._____no_output_____

// Prepare the vocabulary
SimpleVocabulary vocabulary = SimpleVocabulary.builder()
        .optMinFrequency(1)
        .addFromTextFile(embedding.getArtifact("vocab.txt"))
        .optUnknownToken("[UNK]")
        .build();
// Prepare dataset
int maxTokenLength = 64; // cutoff tokens length
int batchSize = 8;
BertFullTokenizer tokenizer = new BertFullTokenizer(vocabulary, true);
CsvDataset amazonReviewDataset = getDataset(batchSize, tokenizer, maxTokenLength);
// split data with 7:3 train:valid ratio
RandomAccessDataset[] datasets = amazonReviewDataset.randomSplit(7, 3);
RandomAccessDataset trainingSet = datasets[0];
RandomAccessDataset validationSet = datasets[1];_____no_output_____

### Setup Trainer and training config

Then, we need to setup our trainer. We set up the accuracy and loss function. The model training logs will be saved to `build/modlel`._____no_output_____

SaveModelTrainingListener listener = new SaveModelTrainingListener("build/model");
        listener.setSaveModelCallback(
            trainer -> {
                TrainingResult result = trainer.getTrainingResult();
                Model model = trainer.getModel();
                // track for accuracy and loss
                float accuracy = result.getValidateEvaluation("Accuracy");
                model.setProperty("Accuracy", String.format("%.5f", accuracy));
                model.setProperty("Loss", String.format("%.5f", result.getValidateLoss()));
            });
DefaultTrainingConfig config = new DefaultTrainingConfig(Loss.softmaxCrossEntropyLoss()) // loss type
        .addEvaluator(new Accuracy())
        .optDevices(Device.getDevices(1)) // train using single GPU
        .addTrainingListeners(TrainingListener.Defaults.logging("build/model"))
        .addTrainingListeners(listener);_____no_output_____

### Start training

We will start our training process. Training on GPU will takes approximately 10 mins. For CPU, it will take more than 2 hours to finish._____no_output_____

int epoch = 2;

Trainer trainer = model.newTrainer(config);
trainer.setMetrics(new Metrics());
Shape encoderInputShape = new Shape(batchSize, maxTokenLength);
// initialize trainer with proper input shape
trainer.initialize(encoderInputShape);
EasyTrain.fit(trainer, epoch, trainingSet, validationSet);
System.out.println(trainer.getTrainingResult());_____no_output_____

### Save the model_____no_output_____

model.save(Paths.get("build/model"), "amazon-review.param");_____no_output_____

## Verify the model

We can create a predictor from the model to run inference on our customized dataset. Firstly, we can create a `Translator` for the model to do preprocessing and post processing. Similar to what we have done before, we need to tokenize the input sentence and get the output ranking._____no_output_____

class MyTranslator implements Translator<String, Classifications> {

    private BertFullTokenizer tokenizer;
    private SimpleVocabulary vocab;
    private List<String> ranks;

    public MyTranslator(BertFullTokenizer tokenizer) {
        this.tokenizer = tokenizer;
        vocab = tokenizer.getVocabulary();
        ranks = Arrays.asList("1", "2", "3", "4", "5");
    }

    @Override
    public Batchifier getBatchifier() { return new StackBatchifier(); }

    @Override
    public NDList processInput(TranslatorContext ctx, String input) {
        List<String> tokens = tokenizer.tokenize(input);
        float[] indices = new float[tokens.size() + 2];
        indices[0] = vocab.getIndex("[CLS]");
        for (int i = 0; i < tokens.size(); i++) {
            indices[i+1] = vocab.getIndex(tokens.get(i));
        }
        indices[indices.length - 1] = vocab.getIndex("[SEP]");
        return new NDList(ctx.getNDManager().create(indices));
    }

    @Override
    public Classifications processOutput(TranslatorContext ctx, NDList list) {
        return new Classifications(ranks, list.singletonOrThrow().softmax(0));
    }
}_____no_output_____

Finally, we can create a `Predictor` to run the inference. Let's try with a random customer review:_____no_output_____

String review = "It works great, but it takes too long to update itself and slows the system";
Predictor<String, Classifications> predictor = model.newPredictor(new MyTranslator(tokenizer));
System.out.println(predictor.predict(review));_____no_output_____