# Notebook from Arun-kc/awesome-notebooks
Path: Plotly/Plotly_Vertical_Barchart_stacked.ipynb

# Plotly - Vertical Barchart stacked
<img src="https://img.shields.io/badge/-Open%20in%20Naas-success?labelColor=000000&logo=data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHN2ZyB3aWR0aD0iMTAyNHB4IiBoZWlnaHQ9IjEwMjRweCIgdmlld0JveD0iMCAwIDEwMjQgMTAyNCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgdmVyc2lvbj0iMS4xIj4KIDwhLS0gR2VuZXJhdGVkIGJ5IFBpeGVsbWF0b3IgUHJvIDIuMC41IC0tPgogPGRlZnM+CiAgPHRleHQgaWQ9InN0cmluZyIgdHJhbnNmb3JtPSJtYXRyaXgoMS4wIDAuMCAwLjAgMS4wIDIyOC4wIDU0LjUpIiBmb250LWZhbWlseT0iQ29tZm9ydGFhLVJlZ3VsYXIsIENvbWZvcnRhYSIgZm9udC1zaXplPSI4MDAiIHRleHQtZGVjb3JhdGlvbj0ibm9uZSIgZmlsbD0iI2ZmZmZmZiIgeD0iMS4xOTk5OTk5OTk5OTk5ODg2IiB5PSI3MDUuMCI+bjwvdGV4dD4KIDwvZGVmcz4KIDx1c2UgaWQ9Im4iIHhsaW5rOmhyZWY9IiNzdHJpbmciLz4KPC9zdmc+Cg=="/>_____no_output_____**Tags:** #plotly #chart #verticalbarchart #group #dataviz_____no_output_____## Input_____no_output_____### Import libraries_____no_output_____

import naas
import plotly.graph_objects as go_____no_output_____

### Variables_____no_output_____

title = "Sales Evolution (group)"

# Output paths
output_image = f"{title}.png"
output_html = f"{title}.html"_____no_output_____

### Get data_____no_output_____

x_axis = [1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012]

y_axis_value1 = [219, 146, 112, 127, 124, 180, 236, 207, 236, 263, 350, 430, 474, 526, 488, 537, 500, 439]

y_axis_value2 = [16, 13, 10, 11, 28, 37, 43, 55, 56, 88, 105, 156, 270, 299, 340, 403, 549, 499]_____no_output_____

## Model_____no_output_____### Create Vertical Barchart stacked_____no_output_____

fig = go.Figure()

Value1 = "USA"
Value2 = "Rest of world"

fig.add_trace(go.Bar(x=x_axis,
                y= y_axis_value1,
                name= Value1,
                marker_color='#5ee290'
                ))
fig.add_trace(go.Bar(x=x_axis,
                y= y_axis_value2,
                name= Value2,
                marker_color='#3f3f3f'
                ))

#fig.update_layout( hovermode="x", title=title)

fig.update_layout(
    title=title ,
    plot_bgcolor="#ffffff",
    #width=800,
    height=500,
    xaxis_tickfont_size=14,
    yaxis=dict(
        title='USD (millions)',
        titlefont_size=16,
        tickfont_size=14,
    ),
    legend=dict(
        x=0,
        y=1.0,
        bgcolor='white',
        bordercolor='white'
    ),
    barmode='stack',
    bargap=0.1, # gap between bars of adjacent location coordinates.
    bargroupgap=0.1 # gap between bars of the same location coordinate.
)
fig.show()_____no_output_____

## Output_____no_output_____### Export in PNG and HTML_____no_output_____

fig.write_image(output_image, width=1200)
fig.write_html(output_html)_____no_output_____

### Generate shareable assets_____no_output_____

link_image = naas.asset.add(output_image)
link_html = naas.asset.add(output_html, {"inline":True})_____no_output_____


# Notebook from Chandrima-04/gimmebio
Path: notebooks/how_many_reads_do_you_need.ipynb

## How Many Reads Do You Need?

Metagenomics uses short-read DNA sequencing to measure the microbes in a sample. Typically, short-read DNA comes in pairs of sequences which are 100-200 base pairs long each. These reads are long enough to be matched to a catalog of genes or genomes allowing researchers to estimate the taxonomic composition of their sample. Unlike amplicon sequencing (which only sequences a particular marker gene) metagenomics allows more specific classification and accurate estimates of the genetic composition of a sample.

Though powerful short read DNA is expensive. Even with state of the art Illumina machines it can cost upward of several hundred dollars just for a few million reads. So the question naturally arises how many reads do you actually need?

There's no simple answer to this question in metagenomics: it will always depend on the parameters of your experiment and what you're trying to detect. If cost were not a factor there would never be a downside to more reads. However, since resources are constrained in the real world we can estimate how many reads we need to accurately detect low abundance taxa in our sample._____no_output_____### Setup the notebook_____no_output_____

from random import random, randrange
from math import log
import pandas as pd

%load_ext rpy2.ipythonThe rpy2.ipython extension is already loaded. To reload it, use:
  %reload_ext rpy2.ipython
%%R

library(ggplot2)_____no_output_____

### Detecting Low Abundance Microbes

We can model the chance of detecting a low abundance microbe in our samples based on a given read depth. Let $p_i$ be the proportion of DNA unique to taxa $i$ in our sample. We can then model the number of reads we need to draw $R$ before drawing $n$ reads from taxa $i$ as a negative binomial distribution.

$ R \sim NB(n, p_i) $
$ Pr(R = r) = {r + n - 1 \choose r} p_i^n (1 - p_i)^{r} $

If we assume we must detect two reads from a given taxa to detect that taxa we can say.

$ Pr(R = r) = (r + 1) (1 - p_i)^{r} p_i^2 $

Using the CDF we can state the probability that $R$ is less than a specified number of reads $r$.

$
    Pr(R \leq r) = \sum_{j=2}^{r}
        (j + 1) (1 - p_i)^j (p_i)^2
$

The proportion of DNA which is uniquely assignable to a specific taxa varies by taxa rank and by clade. However, the typical definition specifies that a species has at least 3\% unique DNA. We can use this to state $p_i$ in terms of the abundance of taxa $i$, $t_i$.

$ p_i = 0.03 t_i $

We can simulate this process to get an idea of the results. in the code below we simulate 4 taxa at abundances of 1 in 10,000; 1 in 1,000; 1 in 100; and 1 in 10. We then measure how many reads we need to select from an imaginary infinitely deep sample before we would get a certain number of reads that could be specifically assigned to the species._____no_output_____

def drawn_to_hit(n, p_i, proportion_uniq=0.03):
    """Return the number of reads necessary to find n reads from a particular taxa"""
    hits, total = 0, 0
    while hits < n:
        total += 1
        hits += 1 if random() < (proportion_uniq * p_i) else 0
    return total

NUM_TO_CALL = [1, 2, 4, 8]
ABS_ABUNDS, TOTAL_READS = [1, 10, 100, 1000], 10 * 1000
REL_ABUNDS = [abund / TOTAL_READS for abund in ABS_ABUNDS]
NTRIALS = 10

tbl = []
for _ in range(NTRIALS):
    for n in NUM_TO_CALL:
        for pi in REL_ABUNDS:
            tbl.append([n, pi, drawn_to_hit(n, pi)])

tbl = pd.DataFrame(tbl, columns=['num_hits', 'rel_abund', 'total_reads'])
_____no_output_____%%R -i tbl  --height 1080 --width 1920
mytbl = tbl
mytbl$num_hits = as.factor(mytbl$num_hits)
mytbl$basis_pts = jitter(log10(10 * 1000 * mytbl$rel_abund))
ggplot(mytbl, aes(x=basis_pts, y=total_reads, color=num_hits)) +
    geom_point(size=10) +
    geom_smooth(method='lm', alpha=0.01) +
    theme_minimal() +
    theme(text = element_text(size=40)) +
    scale_y_log10(breaks=c(10, 100, 1000, 10000, 100000, 1000000, 10000000)) +
    xlab('Log10 Relative Abundance (Basis Points)') +
    ylab('Number of Reads')_____no_output_____

### The Takeaway

With 5,000,000 reads we will nearly always detect species at 1 part in 10,000, even if we require a relatively large number of reads as our minumum threshold of detection.

### The Caveat
However, detection does not mean that we will accurately quantify the abundance of each taxa. Particularly for low abundance taxa we run the risk of misestimating the actual abundance. We can quantify this risk by simulating how many reads we draw from taxa of known abundances. The most important factor here is the expected number of reads from a taxa which is given by the unique proportion of the species, the relative abundance of the species in the sample, and the total number of reads in the sample._____no_output_____

NTRIALS, PROPORTION_UNIQ = 1, 0.03

abs_abunds = [1000, 10 * 1000, 100 * 1000]
num_reads = [
    abund * multiplier
    for multiplier in (25, 50, 75, 100)
    for abund in abs_abunds
]
rel_abunds = [abund / (1000 * 1000) for abund in abs_abunds]

tbl = []
for _ in range(NTRIALS):
    for rel_abund in rel_abunds:
        for nreads in num_reads:
            expected = PROPORTION_UNIQ * nreads * rel_abund
            #if expected < 0.1:
                #continue  # do not bother with very low abundance
            observed = 1 / (1000 * 1000 * 1000)  # pseudocount
            for _ in range(nreads):
                if random() < (PROPORTION_UNIQ * rel_abund):
                    observed += 1

            tbl.append([rel_abund, nreads, expected, observed, observed / expected])

tbl = pd.DataFrame(
    tbl,
    columns=['rel_abund', 'reads_in_sample', 'expected_reads', 'observed_reads', 'ratio']
)
_____no_output_____%%R -i tbl  --height 1080 --width 1920

tbl$lr = abs(log2(tbl$ratio))
ggplot(tbl, aes(x=expected_reads, y=lr)) +
    geom_point(size=10, aes(color=rel_abund)) +
    scale_x_log10(breaks=c(1, 10, 100, 1000, 10000, 100000)) +
    geom_smooth(method='loess') +
    theme_minimal() +
    theme(text = element_text(size=40)) +
    xlab('Expected Reads') +
    ylab('Abs Log2 (Observed / True)')_____no_output_____

Once we have about 100 expected reads our observed abundance starts to line up with the actual abundance. With 10 expected reads or fewer the error becomes extreme.

There is also an issue with dropout. If a species is low abundance we may miss it completely. We can simulate the chance of missing a taxa of aparticular relative abundance. _____no_output_____

NUM_TAXA = [10, 100, 1000, 10 * 1000]
READ_DEPTH = [10 * 1000, 100 * 1000, 1000 * 1000, 5 * 1000 * 1000, 10 * 1000 * 1000]
NTRIALS = 3
DETECT=[1,2,4,8]

def sim_seq(read_depth, num_taxa, proportion_uniq=0.03, detect=2):
    taxa = {}
    for _ in range(read_depth):
        if random() < proportion_uniq:
            taxon = randrange(num_taxa)
            taxa[taxon] = 1 + taxa.get(taxon, 0)
    taxa = [val / read_depth for val in taxa.values() if val >= detect]
    num_missing = num_taxa - len(taxa)
    taxa += [0] * num_missing
    taxa_frac = proportion_uniq / num_taxa
    l1 = sum([abs(val - taxa_frac) for val in taxa])
    return num_missing, l1

tbl = []
for _ in range(NTRIALS):
    for read_depth in READ_DEPTH:
        for num_taxa in NUM_TAXA:
            for detect in DETECT:
                num_missing, l1 = sim_seq(read_depth, num_taxa, detect=detect)
                tbl.append([read_depth, num_taxa, detect, num_missing, l1])

tbl = pd.DataFrame(
    tbl,
    columns=['read_depth', 'num_taxa', 'detect', 'num_missing', 'l1_dist']
)
_____no_output_____%%R -i tbl  --height 1080 --width 1920

mytbl = tbl[tbl$num_taxa >= 100 & tbl$detect <=4,]
mytbl$rel_abund = 1 / mytbl$num_taxa
mytbl$frac_missing = mytbl$num_missing / mytbl$num_taxa
mytbl$detect = as.factor(mytbl$detect)
mytbl$num_taxa = as.factor(mytbl$num_taxa)
mytbl$rel_abund = as.factor(mytbl$rel_abund)
ggplot(mytbl, aes(x=read_depth, y=frac_missing, color=rel_abund, shape=detect)) +
    geom_point(size=10) +
    geom_smooth(method='loess', alpha=0.01) +
    theme_minimal() +
    theme(text = element_text(size=40)) +
    #scale_y_log10(breaks=c(10, 100, 1000, 10000)) +
    scale_y_sqrt(breaks=c(1:4/50, 1:10/10)) +
    scale_x_log10(breaks=c(10000,100000,1000000,10000000)) +
    xlab('Read Depth') +
    ylab('Fraction of Dropouts')_____no_output_____

### Takeaway

If we set 2 reads as the minimum detection threshold A 5M read sample will be missing very few of its (1 / 10,000) abundance species but a 1M read sample will be missing ~20% of these taxa.

We can take this further and plot the distance between simulated samples that are known to have the same actual composition. _____no_output_____

%%R -i tbl  --height 1080 --width 1920
mytbl = tbl[tbl$num_taxa >= 100 & tbl$detect <=4,]
mytbl$rel_abund = 1 / mytbl$num_taxa
mytbl$frac_missing = mytbl$num_missing / mytbl$num_taxa
mytbl$detect = as.factor(mytbl$detect)
mytbl$num_taxa = as.factor(mytbl$num_taxa)
mytbl$rel_abund = as.factor(mytbl$rel_abund)
ggplot(mytbl, aes(x=read_depth, y=l1_dist, color=rel_abund, shape=detect)) +
    geom_point(size=10) +
    geom_smooth(method='loess', alpha=0.01) +
    theme_minimal() +
    theme(text = element_text(size=40)) +
    scale_y_sqrt(breaks=c(1:10/200)) +
    scale_x_log10(breaks=c(10000,100000,1000000,10000000)) +
    xlab('Read Depth') +
    ylab('L1 Distance')_____no_output_____

This is a little fuzzier because real microbiomes are not even abundance (this is a worst case) but the rough takeaway is that two 5M read microbiomes within an L1 distance of 0.01 cannot be distinguished._____no_output_____### Final notes

To effectively detect taxa that occur at one part in 10,000 in a sample should have 5,000,000 reads or more. For low abundance taxa, analyses should distinguish between abundances that can merely be detected and abundances that can be reliably estimated._____no_output_____


# Notebook from khiron/epithelial
Path: buildITP/PBSM.ipynb

<code>
import datetime
lipidname = "PBSM"
tail = "TCC CCCCC"
link = "A A"
head = "C P"

description = ";   A general model sphingomyelin (SM) lipid \n;      C16:0 palmitic acid , C20:0 arachidoyl   \n"
modeledOn=";   This topology follows the standard Martini 2.0 lipid definitions and building block rules.\n; Reference(s): \n;   S.J. Marrink, A.H. de Vries, A.E. Mark. Coarse grained model for semi-quantitative lipid simulations. JPC-B, 108:750-760, \n;   2004. doi:10.1021/jp036508g \n;   S.J. Marrink, H.J. Risselada, S. Yefimov, D.P. Tieleman, A.H. de Vries. The MARTINI force field: coarse grained model for \n;   biomolecular simulations. JPC-B, 111:7812-7824, 2007. doi:10.1021/jp071097f \n;   T.A. Wassenaar, H.I. Ingolfsson, R.A. Bockmann, D.P. Tieleman, S.J. Marrink. Computational lipidomics with insane: a versatile \n;   tool for generating custom membranes for molecular simulations. JCTC, 150410125128004, 2015. doi:10.1021/acs.jctc.5b00209\n; Created: "

now = datetime.datetime.now()
membrane="testmembrane"
insane="../insane+SF.py"
mdparams="../test.mdp"
martinipath="../martini.ff/"
ITPCatalogue="./epithelial.cat"
ITPMasterFile="martini_v2_epithelial.itp"

modeledOn+= now.strftime("%Y.%m.%d")+"\n"

# Cleaning up intermediate files from previous runs
!rm -f *#*
!rm -f *step*
!rm -f {membrane}*_____no_output_____import fileinput
import os.path

print("Create itp")
!python {martinipath}/lipid-martini-itp-v06.py -o {lipidname}.itp -alname {lipidname} -name {lipidname} -alhead '{head}' -allink '{link}' -altail '{tail}'

#update description and parameters
with fileinput.FileInput(lipidname+".itp", inplace=True) as file:
    for line in file:
        if line == ";   This is a ...\n":
            print(description, end='')
        elif line == ";   Was modeled on ...\n":
            print(modeledOn, end='')
        else:
            print(line, end='')

Create itp
The Martini lipid itp generator version 0.6  Args are: -o PBSM.itp -alname PBSM -alhead 'C P' -allink 'A A' -altail 'TCC CCCCC'
#Add this ITP file to the catalogue file
if not os.path.exists(ITPCatalogue):
    ITPCatalogueData = []
else:
    with open(ITPCatalogue, 'r') as file :
        ITPCatalogueData = file.read().splitlines()

    ITPCatalogueData = [x for x in ITPCatalogueData if not x==lipidname+".itp"]

ITPCatalogueData.append(lipidname+".itp")

with open(ITPCatalogue, 'w') as file :
    file.writelines("%s\n" % item for item in ITPCatalogueData)

#build ITPFile
with open(martinipath+ITPMasterFile, 'w') as masterfile:
    for ITPfilename in ITPCatalogueData:
        with open(ITPfilename, 'r') as ITPfile :
            for line in ITPfile:
                masterfile.write(line)

print("Done")Done
# build a simple membrane to visualize this species
!python2 {insane} -o {membrane}.gro -p {membrane}.top -d 0 -x 3 -y 3 -z 3 -sol PW -center -charge 0 -orient -u {lipidname}:1 -l {lipidname}:1 -itpPath {martinipath}; X: 3.000 (4 lipids) Y: 3.000 (4 lipids)
; 16 lipids in upper leaflet, 16 lipids in lower leaflet
; NDX Solute 1 0
; Charge of protein: 0.000000
; NDX Membrane 1 384
; Charge of membrane: 0.000000
; Total charge: 0.000000
; NDX Solvent 385 0
; NDX System 1 384
import os #Operating system specific commands
import re #Regular expression library

print("Test")
print("Grompp")
grompp = !gmx grompp -f {mdparams} -c {membrane}.gro -p {membrane}.top -o  {membrane}.tpr
success=True
for line in grompp:
    if re.search("ERROR", line):
        success=False
    if re.search("Fatal error", line):
        success=False
    #if not success:
    print(line)

if success:
    print("Run")
    !export GMX_MAXCONSTRWARN=-1
    !export GMX_SUPPRESS_DUMP=1
    run = !gmx mdrun -v -deffnm {membrane}
    summary=""
    logfile = membrane+".log"
    if not os.path.exists(logfile):
        print("no log file")
        print("== === ====")
        for line in run:
            print(line)
    else:
        try:
            file = open(logfile, "r")
            fe = False
            for line in file:
                if fe:
                    success=False
                    summary=line
                elif re.search("^Steepest Descents.*converge", line):
                    success=True
                    summary=line
                    break
                elif re.search("Fatal error", line):
                    fe = True
        except IOError as exc:
            sucess=False;
            summary=exc;
    if success:
        print("Success")
    else:
        print(summary)Test
Grompp
                      :-) GROMACS - gmx grompp, 2018.1 (-:

                            GROMACS is written by:
     Emile Apol      Rossen Apostolov      Paul Bauer     Herman J.C. Berendsen
    Par Bjelkmar    Aldert van Buuren   Rudi van Drunen     Anton Feenstra
  Gerrit Groenhof    Aleksei Iupinov   Christoph Junghans   Anca Hamuraru
 Vincent Hindriksen Dimitrios Karkoulis    Peter Kasson        Jiri Kraus
  Carsten Kutzner      Per Larsson      Justin A. Lemkul    Viveca Lindahl
  Magnus Lundborg   Pieter Meulenhoff    Erik Marklund      Teemu Murtola
    Szilard Pall       Sander Pronk      Roland Schulz     Alexey Shvetsov
   Michael Shirts     Alfons Sijbers     Peter Tieleman    Teemu Virolainen
 Christian Wennberg    Maarten Wolf
                           and the project leaders:
        Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel

Copyright (c) 1991-2000, University of Groningen, The Netherlands.
Copyright (c) 2001-2017, The GROMACS development team at
Uppsala University, Stockholm University and
the Royal Institute of Technology, Sweden.
check out http://www.gromacs.org for more information.

GROMACS is free software; you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License
as published by the Free Software Foundation; either version 2.1
of the License, or (at your option) any later version.

GROMACS:      gmx grompp, version 2018.1
Executable:   /usr/bin/gmx
Data prefix:  /usr
Working dir:  /home/richard/projects/epithelial/buildITP
Command line:
  gmx grompp -f ../test.mdp -c testmembrane.gro -p testmembrane.top -o testmembrane.tpr

Ignoring obsolete mdp entry 'title'
Setting the LD random seed to 338658698
Generated 1 of the 741 non-bonded parameter combinations
Excluding 1 bonded neighbours molecule type 'PBSM'
Excluding 1 bonded neighbours molecule type 'PBSM'
Excluding 1 bonded neighbours molecule type 'PW'
Removing all charge groups because cutoff-scheme=Verlet
Number of degrees of freedom in T-Coupling group System is 1149.00

GROMACS reminds you: "If you don't know what you're doing, use a (M)BAR-based method" (Erik Lindahl)

Analysing residue names:
There are:    32      Other residues
Analysing residues not classified as Protein/DNA/RNA/Water and splitting into groups...
This run will generate roughly 0 Mb of data
Run
Success

</code>


# Notebook from niccololampa/data-science-projects
Path: machine-learning-projects/natural-language-processing/nlp_project.ipynb

# Natural Language Processing Project

This NLP project will be attempting to classify Yelp Reviews into 1 star or 5 star categories based off the text content in the reviews.

The "stars" column is the number of stars (1 through 5) assigned by the reviewer to the business. (Higher stars is better.) In other words, it is the rating of the business by the person who wrote the review.

The "cool" column is the number of "cool" votes this review received from other Yelp users.

All reviews start with 0 "cool" votes, and there is no limit to how many "cool" votes a review can receive. In other words, it is a rating of the review itself, not a rating of the business.

The "useful" and "funny" columns are similar to the "cool" column._____no_output_____## Imports
_____no_output_____

import numpy as np
import pandas as pd_____no_output_____

## The Data
_____no_output_____

yelp = pd.read_csv('yelp.csv')_____no_output_____yelp.head()_____no_output_____yelp.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 10 columns):
business_id    10000 non-null object
date           10000 non-null object
review_id      10000 non-null object
stars          10000 non-null int64
text           10000 non-null object
type           10000 non-null object
user_id        10000 non-null object
cool           10000 non-null int64
useful         10000 non-null int64
funny          10000 non-null int64
dtypes: int64(4), object(6)
memory usage: 781.3+ KB
yelp.describe()_____no_output_____yelp['text length'] = yelp['text'].apply(len)_____no_output_____

## Imports_____no_output_____

import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('white')
%matplotlib inline_____no_output_____g = sns.FacetGrid(yelp,col='stars')
g.map(plt.hist,'text length')_____no_output_____sns.boxplot(x='stars',y='text length',data=yelp,palette='rainbow')_____no_output_____sns.countplot(x='stars',data=yelp,palette='rainbow')_____no_output_____stars = yelp.groupby('stars').mean()
stars_____no_output_____stars.corr()_____no_output_____sns.heatmap(stars.corr(),cmap='coolwarm',annot=True)_____no_output_____

## NLP Classification Task
**Create a dataframe called yelp_class that contains the columns of yelp dataframe but for only the 1 or 5 star reviews.**_____no_output_____

yelp_class = yelp[(yelp.stars==1) | (yelp.stars==5)]_____no_output_____

** Create two objects X and y. X will be the 'text' column of yelp_class and y will be the 'stars' column of yelp_class. (Your features and target/labels)**_____no_output_____

X = yelp_class['text']
y = yelp_class['stars']_____no_output_____

**Import CountVectorizer and create a CountVectorizer object.**_____no_output_____

from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer()_____no_output_____

** Use the fit_transform method on the CountVectorizer object and pass in X (the 'text' column). Save this result by overwriting X.**_____no_output_____

X = cv.fit_transform(X)_____no_output_____

## Train Test Split
_____no_output_____

from sklearn.model_selection import train_test_split_____no_output_____X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.3,random_state=101)_____no_output_____

## Training a Model
_____no_output_____

from sklearn.naive_bayes import MultinomialNB
nb = MultinomialNB()_____no_output_____nb.fit(X_train,y_train)_____no_output_____

## Predictions and Evaluations_____no_output_____

predictions = nb.predict(X_test)_____no_output_____from sklearn.metrics import confusion_matrix,classification_report_____no_output_____print(confusion_matrix(y_test,predictions))
print('\n')
print(classification_report(y_test,predictions))[[159  69]
 [ 22 976]]

             precision    recall  f1-score   support

          1       0.88      0.70      0.78       228
          5       0.93      0.98      0.96       998

avg / total       0.92      0.93      0.92      1226

# Using Text Processing_____no_output_____

from sklearn.feature_extraction.text import  TfidfTransformer_____no_output_____from sklearn.pipeline import Pipeline_____no_output_____

** Now create a pipeline with the following steps:CountVectorizer(), TfidfTransformer(),MultinomialNB()**_____no_output_____

pipeline = Pipeline([
    ('bow', CountVectorizer()),  # strings to token integer counts
    ('tfidf', TfidfTransformer()),  # integer counts to weighted TF-IDF scores
    ('classifier', MultinomialNB()),  # train on TF-IDF vectors w/ Naive Bayes classifier
])_____no_output_____

## Using the Pipeline_____no_output_____### Train Test Split

**Redo the train test split on the yelp_class object.**_____no_output_____

X = yelp_class['text']
y = yelp_class['stars']
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.3,random_state=101)_____no_output_____pipeline.fit(X_train,y_train)_____no_output_____

### Predictions and Evaluation

** Now use the pipeline to predict from the X_test and create a classification report and confusion matrix. _____no_output_____

predictions = pipeline.predict(X_test)_____no_output_____print(confusion_matrix(y_test,predictions))
print(classification_report(y_test,predictions))_____no_output_____


# Notebook from daoktar/ThreeBodyBot_for_collision_visuals
Path: 3BodyAnalysis.ipynb

# Analysis of random three-body problems generated by @ThreeBodyBot

This notebook generates a pretty suite of plots based on the animations posted to the Twitter account. Each time the script is run it tries to find a solution that lasts at least 15 years, with a maximum length of 60 years, with a maximum of 2000 iterations given to find a solution that satisfies that length criteria. If it can't find a solution (which is pretty rare) it makes it anyways, so extremely short animations (with length < 15 years) are the result of this rare condition. Right now I'm currently tracking the date the simulation was run, the simulation lengths, the masses, radii, how it ended (collision, ejection, timed out), which things collided (if there was a collision), the initial conditions, and the number of iterations it took to randomly arrive at that solution. This tracking started on 1/10/2021, but the notebook is only periodically updated with the latest results to Github. _____no_output_____

<code>
using DataFrames, Dates, Plots, Plots.Measures_____no_output_____function getDF(file = "3BodyStats.txt") #reads in data output by threeBodyProb.jl each night and puts into DF
    lines = readlines(file)
    Types = [Date,Float64,Float64,Float64,Float64,Float64,Float64,Float64,Bool,Int64,Int64,Float64,Float64,Float64,Float64,Float64,Float64,Float64,Float64,Float64,Float64,Float64,Float64,Int64]
    names = [:Date,:Length,:m1,:m2,:m3,:r1,:r2,:r3,:collisionBool,:collision1,:collision2,:x1,:y1,:x2,:y2,:x3,:y3,:vx1,:vy1,:vx2,:vy2,:vx3,:vy3,:iter]
    df = DataFrame(names .=> [Vector{type}(undef,length(lines)) for type in Types])
    for i = 1:length(lines)
        s = split(lines[i],",")
        if length(s) == length(names) #sometimes lines get weird stuff written to them, so only put stuff in df if it's a valid line
            df.Date[i] = parse(Date,s[1])
            df.Length[i] = parse(Float64,s[2])
            df.m1[i] = parse(Float64,s[3]); df.m2[i] = parse(Float64,s[4]); df.m3[i] = parse(Float64,s[5])
            df.r1[i] = parse(Float64,s[6]); df.r2[i] = parse(Float64,s[7]); df.r3[i] = parse(Float64,s[8])
            df.collisionBool[i] = parse(Bool,s[9]); df.collision1[i] = parse(Int64,s[10]); df.collision2[i] = parse(Int64,s[11])
            df.x1[i] = parse(Float64,s[12]); df.y1[i] = parse(Float64,s[13]); df.x2[i] = parse(Float64,s[14]); df.y2[i] = parse(Float64,s[15]); df.x3[i] = parse(Float64,s[16]); df.y3[i] = parse(Float64,s[17])
            df.vx1[i] = parse(Float64,s[18]); df.vy1[i] = parse(Float64,s[19]); df.vx2[i] = parse(Float64,s[20]); df.vy2[i] = parse(Float64,s[21]); df.vx3[i] = parse(Float64,s[22]); df.vy3[i] = parse(Float64,s[23])
            df.iter[i] = parse(Int64,s[24])
        end
    end
    return df
end        _____no_output_____df = getDF() #get the data -- this only shows first 30 entries and only the first 9 columns in preview below_____no_output_____display(df[end-29:end,:]) #show the last 30 just to check that they're there, can modify this to show any range you like, can access columns by name, ie display(df.iter)_____no_output_____N=size(df)[1]; cFrac=size(df[df.collisionBool .== true,:])[1]/N; tFrac = size(df[(df.collisionBool .== false) .& (df.Length .== 50.0),:])[1]/N; eFrac = 1.0 - tFrac - cFrac
collidedX=vcat(df[df.collisionBool .== true, :].x1, df[df.collisionBool .== true, :].x2, df[df.collisionBool .== true, :].x3)
collidedY=vcat(df[df.collisionBool .== true, :].y1, df[df.collisionBool .== true, :].y2, df[df.collisionBool .== true, :].y3)
timedOutX=vcat(df[(df.collisionBool .== false) .& (df.Length .== 60.0),:].x1,df[(df.collisionBool .== false) .& (df.Length .== 50.0),:].x2,df[(df.collisionBool .== false) .& (df.Length .== 50.0),:].x3)
timedOutY=vcat(df[(df.collisionBool .== false) .& (df.Length .== 60.0),:].y1,df[(df.collisionBool .== false) .& (df.Length .== 50.0),:].y2,df[(df.collisionBool .== false) .& (df.Length .== 50.0),:].y3)
escapedX=vcat(df[(df.collisionBool .== false) .& (df.Length .!= 60.0),:].x1,df[(df.collisionBool .== false) .& (df.Length .!= 50.0),:].x2,df[(df.collisionBool .== false) .& (df.Length .!= 50.0),:].x3)
escapedY=vcat(df[(df.collisionBool .== false) .& (df.Length .!= 60.0),:].y1,df[(df.collisionBool .== false) .& (df.Length .!= 50.0),:].y2,df[(df.collisionBool .== false) .& (df.Length .!= 50.0),:].y3)

default(fontfamily="times",minorgrid=true,widen=false,tickdirection=:out)
default(size=(1200,1000), xtickfontsize=10, ytickfontsize=10, xguidefontsize=12, yguidefontsize=12, legendfontsize=6, dpi=100)
p1=histogram(df.Length,label="",xlabel="Length [years]",ylabel="Counts",title="Length distribution",color=:4,bins=50)
p1=annotate!(((xlims(p1)[1]+xlims(p1)[2])/2,ylims(p1)[2]+0.05*ylims(p1)[2],Plots.text("Average = $(round(sum(df.Length)/N,sigdigits=3)) years","times",8)))
p1=pie!(["Collision","Escape","Time"],[cFrac,eFrac,tFrac],inset=(1,bbox(16/16,17/16,0.4,0.4)),title="",subplot=2,titlefontsize=8,legendfontsize=6,legend=false,
    foreground_color_legend=:transparent,background_color_legend=:transparent,background_color=:transparent,background_color_outside=:transparent,legendfontcolor=:white)
p1=plot!(p1[2],annotations=(0,0.4,Plots.text("$(round(cFrac*100,sigdigits=3)) %","times","white",8)))
p1=plot!(p1[2],annotations=(-0.15,-0.45,Plots.text("$(round(eFrac*100,sigdigits=3)) %","times","white",8)))
p1=plot!(p1[2],annotations=(0.68,-0.07,Plots.text("$(round(tFrac*100,sigdigits=3)) %","times","white",8)))
p1=plot!(p1[2],annotations=(-0.1,1.9,Plots.text("N = $N","times",8)))
p2=scatter(collidedX,collidedY,label="Collision",title="Initial positions",legendtitle="Ended because of:",legendtitlefontsize=8,
    foreground_color_legend=:transparent,background_color_legend=:transparent,xlabel="x [AU]",ylabel="y [AU]",legend=(-0.27,0.07),
    aspect_ratio=:equal,xlims=(-30,30),ylims=(-30,30),markersize=3,markershape=:star,markerstrokewidth=0)
p2=scatter!(escapedX,escapedY,label="Escape",markersize=3,markershape=:star,markerstrokewidth=0)
p2=scatter!(timedOutX,timedOutY,label="Time",markersize=3,markershape=:star,markerstrokewidth=0)
p2=annotate!((0.0,ylims(p2)[2]+0.05*ylims(p2)[2],Plots.text("Average number of iterations required to find solution = $(round(sum(df.iter)/N,sigdigits=3))","times",8)))
p3=histogram(vcat(df.vx1,df.vy1,df.vx2,df.vy2,df.vx3,df.vy3),label="",title="Initial velocity distribution",xlabel="Velocity [km/s]",ylabel="Counts",color=5,bins=50)
p3=annotate!((0.0,ylims(p3)[2]+0.05*ylims(p3)[2],Plots.text("Average = $(round(sum(vcat(df.vx1,df.vy1,df.vx2,df.vy2,df.vx3,df.vy3))/(6*N),sigdigits=3)) km/s","times",8)))
p4=histogram(vcat(df.m1,df.m2,df.m3),label="",title="Mass distribution",xlabel="Mass [solar masses]",ylabel="Counts",color=:6,bins=50)
p4=annotate!(((xlims(p4)[1]+xlims(p4)[2])/2,ylims(p4)[2]+0.05*ylims(p4)[2],Plots.text("Average = $(round(sum(vcat(df.m1,df.m2,df.m3))/(3*N),sigdigits=3)) solar masses","times",8)))
plot(p1,p2,p3,p4,layout=@layout[A B ; C D],size=(900,900),left_margin=5mm,right_margin=5mm,top_margin=5mm,bottom_margin=5mm)_____no_output_____
</code>


# Notebook from yilmazff/firebase_art
Path: notebooks/firebase_examples.ipynb

# Usage

Follow these steps:

1. Import and run the initial cells
2. Choose a document and run `crawl_doc` and `generate_keys` cells for the document:
    - There are 9 examples (documents) in the notebook: [Doc1](#doc1), [Doc2](#doc2), [Doc3](#doc3), [Doc4](#doc4), [Doc5](#doc5), [Doc6](#doc6), [Doc7](#doc7), [Doc8](#doc8), [Doc9](#doc9). Just pick one to get the keywords, then proceed to the next step.
3. Next, execute [FIREBASE pipeline](#fatih) for the selected document.
4. Finally, you can [visualize the results](#visualize) and inspect all the obtained images together with the computed semantic similarity scores._____no_output_____

%load_ext autoreload
%autoreload 2_____no_output_____import warnings
warnings.filterwarnings('ignore')

import pathlib
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from IPython.display import display

import sys
repo_root = str((pathlib.Path.cwd().parent / 'code').resolve())
sys.path.append(repo_root)

from firebase_utils import firebase
from wiki_utils import save_wiki_image, get_wiki_images
from bing_utils import get_bing_results_per_paragraph
from img_utils import get_image_with_url
from doc_utils import crawl_doc_____no_output_____

# Initialize FIREBASE instance_____no_output_____

firebase_model = firebase()_____no_output_____

# Doc1. Human digestive system_____no_output_____

url = 'https://www.asge.org/home/about-asge/newsroom/media-backgrounders-detail/human-digestive-system'
pars = crawl_doc(url,th=25)"par1" The human digestive system, (also known as the digestive tract, the GI tract, the alimentary canal) is a series of connected organs leading from the mouth to the anus. The digestive system allows us to break down the food we eat to obtain energy and nourishment.

"par2" The digestive system -- which can be up to 30 feet in length in adults -- is usually divided into eight parts: the mouth, the esophagus, the stomach, the small intestine (or "small bowel") and the large intestine (also called "large bowel" or "colon") with the liver, pancreas, and gallbladder adding secretions to help digestion. These organs combine to perform six tasks: ingestion, secretion, propulsion, digestion, absorption, and defecation.

"par3" The mouth starts the process by ingesting and mechanically breaking down the food we eat into a swallowable form, adding some early secretions to start the process of digestion. The esophagus is the muscular tube connecting the mouth to the stomach. A ring-like muscle at the end of the esophagus controls the passage of food into the stomach. This ring-like muscle also controls the food from going back up into the esophagus. The stomach then further liquefies the food and adds digestive acids.

"par4" From there the stomach contents are slowly released into the small intestine. The small intestine is where most of the digestion and absorption of nutrients takes place. There are millions of tiny finger-like projections lining the small intestine called villi (pronounced: "VILL-ee"). These villi aid in the absorption of nutrients. Secretions from the liver, pancreas, and gallbladder are emptied into the small intestine and also aid in the digestive process.

"par5" After traveling through the nearly 20 feet of small intestine, unabsorbed material moves into the colon (large bowel or large intestine). Here the remaining liquids and salts (electrolytes) are absorbed from the digested material. Bacteria further break down the undigested material, which continues to solidify and eventually passes from the body as feces through the rectum and anus during a bowel movement.

keys, par_embeddings = firebase_model.generate_keys(pars, num_words=2, thresh=0.51, top_n=1)
keys_____no_output_____

# Notes
- threshold on paragraphs (I put a 25-word limit)
- A single word works, but with similarity threshold = 0.42, we get two images for the document, which is fine because those images cover most of that "short" article._____no_output_____
# Doc2. Human visual system_____no_output_____

url = 'https://blogs.aalto.fi/neuroscience/2016/10/15/visual-system/'
pars = crawl_doc(url,th=35)"par1" The visual system of any specie is a sensory system developed to perceive the environment through the light: it uses light to form images of the surroundings. Light in the form of electromagnetic radiation propagates as a wave with a certain amplitude, frequency and wavelength. The human visual system has evolved to transduce the electromagnetic waves to meaningful information about the world.

"par2" Vision begins with the eye. Images are formed in the eye by refraction, the bending of light rays that can occur when they travel from one transparent medium to another. Lightwaves enter the eye through cornea, refracts, goes through the pupil and reaches the retina. The pigmentation of the eye is provided by the iris; the white part of the eye is called sclera and it forms the eyeball; extraocular muscles move the eye and the eye’s orbit. The axons exit the retina thought the optic nerve in the back of the eye. Image formation in the eye happens in the retina according to basic optical properties.

"par3" The conversion of light energy into neural activity starts in the retina. Retina has a laminar structure: it consists of several layers of cells. The lightwave information propagates through these layers as follows: photoreceptors respond to light and influence membrane potential of the bipolar cells, ganglion cells fire action potentials which propagate through the optic nerve to the rest of the brain. Other cells of the retina are horizontal cells and amarcine cells.

"par4" The only cells which respond to light are rod and cone photoreceptors. The mechanisms of phototransduction in rods and cones differ and define human color vision and light sensitivity. The ganglion cells are the only source of output from the retina and the only retinal neurons that fire action potentials. Ganglion cells (P-type, M-type, nonM–nonP type) and bipolar cells (ON-cell, OFF-cell) have their receptive fields, which give the ability to perceive complex combinations of colors, as well as light and dark regions. The key structure-function relationships defined here in terms of color opponency and center-surrounding receptive fields.

"par5" The neural pathway from optic nerve to the rest of the brain is called retinofugal projection and includes optic nerve, optict chiasm and optic tract. At the optic chiasm happens decussation: the crossing of fibers from one brain hemisphere to another. This implies the concept of the visual hemifields, nicely explained in the Book.

"par6" The vision involves the perception of numerous different properties of objects, including color, form, and movement; and these properties are processed in parallel by different cells of the visual system. Therefore the nature of the cortical processing of visual information is distributed. The main structures of the brain involved in processing of the visual information are the lateral geniculate nucleus; the striate cortex and superior colliculus.

keys, par_embeddings = firebase_model.generate_keys(pars, num_words=2, thresh=0.58, top_n=1)
keys_____no_output_____

# Notes
- threshold on paragraphs (I put a 35-word limit), because this article has longer paragraphs compared to the first one.
- A single word doesn't work, so I usea keyword of size 2. With similarity threshold = 0.58, we get three images for the document, which is again fine because those images cover most of the article._____no_output_____
# Doc3. Basic components of a building_____no_output_____

url = 'https://theconstructor.org/building/12-basic-components-building-structure/34024/'
pars = crawl_doc(url,th=30)Paragraph contains special characters. Skipping...
"par1" The basic components of a building structure are the foundation, floors, walls, beams, columns, roof, stair, etc. These elements serve the purpose of supporting, enclosing and protecting the building structure.

"par2" The roof forms the topmost component of a building structure. It covers the top face of the building. Roofs can be either flat or sloped based on the location and weather conditions of the area.

"par3" Lintels are constructed above the wall openings like doors, windows, etc. These structures support the weight of the wall coming over the opening. Normally, lintels are constructed by reinforced cement concrete. In residential buildings, lintels can be either constructed from concrete or from bricks.

"par4" Beams and slabs form the horizontal members in a building. For a single storey building, the top slab forms the roof. In case of a multi-storey building, the beam transfers the load coming from the floor above the slab which is in turn transferred to the columns. Beams and slabs are constructed by reinforced cement concrete (R.C.C).

"par5" Columns are vertical members constructed above the ground level. Columns can be of two types: Architectural columns and structural columns. Architectural columns are constructed to improve the building's aesthetics while a structural column takes the load coming from the slab above and transfers safely to the foundation.

"par6" Walls are vertical elements which support the roof. It can be made from stones, bricks, concrete blocks, etc. Walls provide an enclosure and protect against wind, sunshine, rain etc. Openings are provided in the walls for ventilation and access to the building.

"par7" The floor is the surface laid on the plinth level. Flooring can be done by a variety of materials like tiles, granites, marbles, concrete, etc. Before flooring, the ground has to be properly compacted and leveled.

"par8" A stair is a sequence of steps that connects different floors in a building structure. The space occupied by a stair is called as the stairway. There are different types of stairs like a wooden stair, R.C.C stair etc.

"par9" The Foundation is a structural unit that uniformly distributes the load from the superstructure to the underlying soil. This is the first structural unit to be constructed for any building construction. A good foundation prevents settlement of the building.

keys, par_embeddings = firebase_model.generate_keys(pars, num_words=2, thresh=0.72, top_n=1)
keys_____no_output_____

# Notes
- threshold on paragraphs = 30 words --> but we get a dummy paragraph at the beginning.
- bigrams are needed as keywords (i.e., l=2).
- to embed these keywords, we face a problem because these bigrams are out-of-vocab for Word2Vec, I searched the keyword of each paragraph manually and observed that the baseline (just picking the first image of google) works well._____no_output_____
# Doc4. Volcano eruption_____no_output_____

url = 'https://www.dogonews.com/2021/1/6/hawaiis-kilauea-volcano-eruption-creates-spectacular-lava-lake'
pars = crawl_doc(url,th=25)"par1" After erupting almost continuously for over three decades — from 1983 to 2018 —  Hawaii's Kilauea volcano finally seemed to lose steam, producing no lava for nearly two years. The slumber ended on the night of December 20, 2020, when the active volcano began spewing out dramatic lava fountains and giant puffs of gas and steam from a fissure in the northwest wall of the Halemaʻumaʻu crater.

"par2" As of December 31, 2020, the volcano had ejected over 27 million cubic meters (953 million cubic feet) of molten rock — enough to fill more than 8,000 Olympic-sized swimming pools — and replaced the existing water lake with a nearly 600-foot-deep lava lake.  Fortunately, the magma is contained inside the volcano's crater and does not pose a risk to people or property as it did in 2018, when the molten rock flowed through a residential neighborhood, destroying over 700 homes.

"par3" Residents have, however, been asked to limit outdoor activities in areas with high volcanic smog levels. Commonly known as vog, it is a form of air pollution created when sulfur dioxide and other pollutants emitted from volcanic activity interact chemically with atmospheric moisture, oxygen, dust, and sunlight. While not deadly, it can irritate skin and the tissues and mucous membranes of the eyes, nose, and throat, and it can be particularly harmful to individuals with preexisting respiratory conditions.

"par4" While Kilauea is now recognized as one of the world's youngest and most active volcanoes, it was initially believed to be an offshoot of a bigger volcano. It was only after researchers discovered a magma plumbing system extending over 37 miles (60 km) inside its crater in 1884 that the volcano was finally accepted as a separate entity.

"par5" Many locals believe that Kilauea’s eruptions are caused by Pele, a Hawaiian volcano goddess who resides in the mountain.  According to popular folklore, the deity — one of six daughters and seven sons born to ancient Earth goddess Haumea and universe creator Kane Milohai — was exiled to Hawaii from Tahiti.  Upon arrival, she went on a rampage, creating destructive fire pits and volcanoes. While Pele was killed during an epic battle with her sister Nāmaka on the island of Maui, her spirit lives on inside the Halemaʻumaʻu crater.

Paragraph contains special characters. Skipping...
keys, par_embeddings = firebase_model.generate_keys(pars, num_words=1, thresh=0.37)
keys_____no_output_____

# Notes
- threshold on paragraphs = 25 words --> but we get a dummy paragraph at the end.
- unigrams are fine as keywords (i.e., l=1) because the whole article is just about volcanos.
- for the dummy paragraph (par6), the keyword is of course out-of-vocab of the embedding models but the baseline (i.e., returning the first result of bing) returns no images which is good._____no_output_____
# Doc5. NASA finds water on the Moon_____no_output_____

url = 'https://www.dogonews.com/2020/11/6/nasas-sofia-finds-water-on-the-moons-sunlit-side'
pars = crawl_doc(url,th=25)"par1" The presence of ice in the permanently shadowed craters around the Moon's poles has been known for some time. However, researchers had been unsure if the hydration detected on the satellite's sunlit areas was "molecular" water (H2O), or hydroxyl (OH), a molecule that's one hydrogen atom shy of becoming water. On October 26, 2020, NASA confirmed that the liquid was indeed water.

"par2" “We had indications that H2O – the familiar water we know – might be present on the sunlit side of the Moon,” said Paul Hertz,  Director of Astrophysics at NASA.  “Now we know it is there. This discovery challenges our understanding of the lunar surface and raises intriguing questions about resources relevant for deep space exploration.”

"par3" The discovery was made using data collected by NASA’s Stratospheric Observatory for Infrared Astronomy (SOFIA). The modified Boeing 747 aircraft, which can fly its large telescope at altitudes of up to 45,000 feet, allows researchers to clearly observe space objects. The water molecules — detected using a special infrared camera capable of discerning between the wavelengths of H2O and OH — were found in the Clavius Crater.  Located on the Moon's Southern Hemisphere, the large crater can be clearly seen from Earth.

"par4" However, before you start packing your bags for our satellite, be warned that the water concentration is extremely low — about 100 to 412 parts per million. To put it in perspective,  the Sahara Desert has about 100 times more water! "This is not puddles of water but instead water molecules that are so spread apart that they do not form ice or liquid water," said NASA researcher Casey Honniball, who led the study. Though not enough to sustain humans, the discovery is significant because water is a rare and precious resource in deep space.

"par5" The scientists speculate that the H2O molecules may have been deposited by the tiny meteorites that are continually bombarding the Moon's surface. Another plausible explanation involves a multi-step process where the Sun's solar wind brings hydrogen to the Moon, which then reacts with oxygen-bearing minerals in the soil to create hydroxyl. This hydroxyl could later be transformed into water from the micrometeorites' radiation.

"par6" Also perplexing is how the water escapes evaporation. The liquid could be trapped in tiny bead-like structures in the soil that form from the intense heat generated from the micrometeorite impacts. Alternatively, the H2O molecules may be hidden inside the lunar soil grains and sheltered from the Sun.

"par7"  “Without a thick atmosphere, water on the sunlit lunar surface should just be lost to space,” said Honniball. “Yet somehow we’re seeing it. Something is generating the water, and something must be trapping it there.”

"par8" NASA plans to conduct follow-up missions to determine how widespread the water is across the lunar surface. The findings will allow the US space agency to find the perfect landing for its Artemis mission that will take the first woman and next man to explore the Moon more extensively in 2024.

Paragraph contains special characters. Skipping...
keys, par_embeddings = firebase_model.generate_keys(pars, num_words=1, thresh=0.35)
keys_____no_output_____

# Notes
- threshold on paragraphs = 25 words --> but we get a dummy paragraph again at the end (because the sources of Doc5 and Doc4 are the same website).
- unigrams are fine as keywords (i.e., l=1) because the whole article is just about the moon.
- for the dummy paragraph (par6), the keyword is of course out-of-vocab of the embedding models but the baseline (i.e., returning the first result of bing) returns no images which is good._____no_output_____
# Doc6. Human auditory system. An example of complete failure

This document essentially doesn't have any paragraphs with more than one sentence._____no_output_____

url = 'https://w3.ual.es/~vruiz/Docencia/Apuntes/Perception/Sistema_Auditivo/index.html'
pars = crawl_doc(url,th=25,disp=False)Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
Paragraph contains special characters. Skipping...
keys, par_embeddings = firebase_model.generate_keys(pars, num_words=2, thresh=0.4)
keys_____no_output_____

# Notes
- If you set disp=True when crawling the data above, you'll see that the article is written in a weird way that the whole document will be collected as one paragraph, and you'll see multiple repetitive chunks as the extracted paragraphs. So our framework doesn't work on this document.
- [This article](http://www.cs.tut.fi/~ypsilon/80545/HumanAuditorySystem.html) is another example of our framework's failure, because our framework is not suited for bulletpoint-like articles, it is meant to be used for well-written (i.e., structured in organized (and relatively long) paragraphs) articles._____no_output_____
# Doc7. BERT explained

- This is a rather failure case, because the keyBERT extracts keywords related to Google or researchers rather than anything about NLP and BERT._____no_output_____

url = 'https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270'
pars = crawl_doc(url,th=45)"par1" BERT (Bidirectional Encoder Representations from Transformers) is a recent paper published by researchers at Google AI Language. It has caused a stir in the Machine Learning community by presenting state-of-the-art results in a wide variety of NLP tasks, including Question Answering (SQuAD v1.1), Natural Language Inference (MNLI), and others.

"par2" BERT’s key technical innovation is applying the bidirectional training of Transformer, a popular attention model, to language modelling. This is in contrast to previous efforts which looked at a text sequence either from left to right or combined left-to-right and right-to-left training. The paper’s results show that a language model which is bidirectionally trained can have a deeper sense of language context and flow than single-direction language models. In the paper, the researchers detail a novel technique named Masked LM (MLM) which allows bidirectional training in models in which it was previously impossible.

"par3" In the field of computer vision, researchers have repeatedly shown the value of transfer learning — pre-training a neural network model on a known task, for instance ImageNet, and then performing fine-tuning — using the trained neural network as the basis of a new purpose-specific model. In recent years, researchers have been showing that a similar technique can be useful in many natural language tasks.

"par4" BERT makes use of Transformer, an attention mechanism that learns contextual relations between words (or sub-words) in a text. In its vanilla form, Transformer includes two separate mechanisms — an encoder that reads the text input and a decoder that produces a prediction for the task. Since BERT’s goal is to generate a language model, only the encoder mechanism is necessary. The detailed workings of Transformer are described in a paper by Google.

"par5" As opposed to directional models, which read the text input sequentially (left-to-right or right-to-left), the Transformer encoder reads the entire sequence of words at once. Therefore it is considered bidirectional, though it would be more accurate to say that it’s non-directional. This characteristic allows the model to learn the context of a word based on all of its surroundings (left and right of the word).

"par6" The chart below is a high-level description of the Transformer encoder. The input is a sequence of tokens, which are first embedded into vectors and then processed in the neural network. The output is a sequence of vectors of size H, in which each vector corresponds to an input token with the same index.

"par7" When training language models, there is a challenge of defining a prediction goal. Many models predict the next word in a sequence (e.g. “The child came home from ___”), a directional approach which inherently limits context learning. To overcome this challenge, BERT uses two training strategies:

"par8" Before feeding word sequences into BERT, 15% of the words in each sequence are replaced with a [MASK] token. The model then attempts to predict the original value of the masked words, based on the context provided by the other, non-masked, words in the sequence. In technical terms, the prediction of the output words requires:

"par9" In the BERT training process, the model receives pairs of sentences as input and learns to predict if the second sentence in the pair is the subsequent sentence in the original document. During training, 50% of the inputs are a pair in which the second sentence is the subsequent sentence in the original document, while in the other 50% a random sentence from the corpus is chosen as the second sentence. The assumption is that the random sentence will be disconnected from the first sentence.

"par10" In the fine-tuning training, most hyper-parameters stay the same as in BERT training, and the paper gives specific guidance (Section 3.5) on the hyper-parameters that require tuning. The BERT team has used this technique to achieve state-of-the-art results on a wide variety of challenging natural language tasks, detailed in Section 4 of the paper.

"par11" BERT is undoubtedly a breakthrough in the use of Machine Learning for Natural Language Processing. The fact that it’s approachable and allows fast fine-tuning will likely allow a wide range of practical applications in the future. In this summary, we attempted to describe the main ideas of the paper while not drowning in excessive technical details. For those wishing for a deeper dive, we highly recommend reading the full article and ancillary articles referenced in it. Another useful reference is the BERT source code and models, which cover 103 languages and were generously released as open source by the research team.

"par12" Training the language model in BERT is done by predicting 15% of the tokens in the input, that were randomly picked. These tokens are pre-processed as follows — 80% are replaced with a “[MASK]” token, 10% with a random word, and 10% use the original word. The intuition that led the authors to pick this approach is as follows (Thanks to Jacob Devlin from Google for the insight):

keys, par_embeddings = firebase_model.generate_keys(pars, num_words=3, thresh=0.6, top_n=1)
keys_____no_output_____

# Doc8. WHO - Coronavirus disease (COVID-19): Herd immunity, lockdowns and COVID-19

- This is another failure case. We do find good images, but our scoring system selects the worst one._____no_output_____

url = 'https://www.who.int/news-room/q-a-detail/herd-immunity-lockdowns-and-covid-19'
pars = crawl_doc(url,th=45)Paragraph contains special characters. Skipping...
"par1" 'Herd immunity', also known as 'population immunity', is the indirect protection from an infectious disease that happens when a population is immune either through vaccination or immunity developed through previous infection. WHO supports achieving 'herd immunity' through vaccination, not by allowing a disease to spread through any segment of the population, as this would result in unnecessary cases and deaths.

"par2" Vaccines train our immune systems to create proteins that fight disease, known as ‘antibodies’, just as would happen when we are exposed to a disease but – crucially – vaccines work without making us sick. Vaccinated people are protected from getting the disease in question and passing on the pathogen, breaking any chains of transmission. Visit our webpage on COVID-19 and vaccines for more detail.

"par3" To safely achieve herd immunity against COVID-19, a substantial proportion of a population would need to be vaccinated, lowering the overall amount of virus able to spread in the whole population. One of the aims with working towards herd immunity is to keep vulnerable groups who cannot get vaccinated (e.g. due to health conditions like allergic reactions to the vaccine) safe and protected from the disease. Read our Q&A on vaccines and immunization for more information.

"par4" The percentage of people who need to be immune in order to achieve herd immunity varies with each disease. For example, herd immunity against measles requires about 95% of a population to be vaccinated. The remaining 5% will be protected by the fact that measles will not spread among those who are vaccinated. For polio, the threshold is about 80%. The proportion of the population that must be vaccinated against COVID-19 to begin inducing herd immunity is not known. This is an important area of research and will likely vary according to the community, the vaccine, the populations prioritized for vaccination, and other factors.

Paragraph contains special characters. Skipping...
"par5" We are still learning about immunity to COVID-19. Most people who are infected with COVID-19 develop an immune response within the first few weeks, but we don’t know how strong or lasting that immune response is, or how it differs for different people. There have also been reports of people infected with COVID-19 for a second time.

"par6" Until we better understand COVID-19 immunity, it will not be possible to know how much of a population is immune and how long that immunity last for, let alone make future predictions. These challenges should preclude any plans that try to increase immunity within a population by allowing people to get infected.

"par7" Finally, while most infected people get mild or moderate forms of COVID-19 and some experience no disease, many become seriously ill and must be admitted into hospital. We are only beginning to understand the long-term health impacts among people who have had COVID-19, including what is being described as ‘Long COVID.’ WHO is working with clinicians and patient groups to better understand the long term effects of COVID-19.

Paragraph contains special characters. Skipping...
"par8" Research is still ongoing into how strong that protection is and how long it lasts. WHO is also looking into whether the strength and length of immune response depends on the type of infection a person has: without symptoms (‘asymptomatic’), mild or severe. Even people without symptoms seem to develop an immune response.

"par9" For other coronaviruses – such as the common cold, SARS-CoV-1 and Middle East Respiratory Syndrome (MERS) – immunity declines over time, as is the case with other diseases. While people infected with the SARS-CoV-2 virus develop antibodies and immunity, we do not yet know how long it lasts.

Paragraph contains special characters. Skipping...
"par10" However, these measures can have a profound negative impact on individuals, communities, and societies by bringing social and economic life to a near stop. Such measures disproportionately affect disadvantaged groups, including people in poverty, migrants, internally displaced people and refugees, who most often live in overcrowded and under resourced settings, and depend on daily labour for subsistence.

"par11" Governments must make the most of the extra time granted by ‘lockdown’ measures by doing all they can to build their capacities to detect, isolate, test and care for all cases; trace and quarantine all contacts; engage, empower and enable populations to drive the societal response and more.

keys, par_embeddings = firebase_model.generate_keys(par_merger(pars), num_words=3, thresh=0.3, top_n=1)
keys_____no_output_____

# Healthy Habits We Can Learn From Our Birds_____no_output_____

url = 'https://lafeber.com/pet-birds/healthy-habits-can-learn-birds/'
pars = crawl_doc(url,th=25)"par1" Looking for some healthy habits to try in the year ahead? Good news! Chances are you have your very own health coach living with you right now — look no further than your feathered friend! Here are a few of your bird’s healthy habits that you should follow, too.

"par2" Eat the rainbow. This trending phrase is a simple way to remind us to eat a variety of fruits and vegetables to help get the vitamins and minerals our bodies need. The same goes for our pet birds. Parrots cannot live on seed alone — they thrive when fed a healthy diet that is supplemented by a variety of fruits and veggies. Likewise, their human stewards cannot and should not consume a diet devoid of healthy selections. In the wild, parrots are natural-born foragers — they eat what’s in season, which adds variety; we can reap health benefits by doing the same. Try some of these winter fruits and veggies.

"par3" Eat slow, and enjoy your food. Ever noticed that your bird takes his/her time to thoroughly enjoy a favorite food? Birds eat different than our canine companions. A dog is more inclined to gulp down a prized treat, while a parrot might very well take a a good five minutes to eat a single blueberry. We humans can take a cue from our birds and take the time to enjoy our food. Research shows that chewing until “no lumps remain” increases the number of calories the human body burns during digestion: about 10 extra calories for a 300-calorie meal. Extra chewing also slows the pace of eating, which may be another reason why it’s a healthier way to eat. Slow eating may also help people eat more mindfully and tune in to their feelings of fullness.

"par4" Eat breakfast. Imagine your bird starting the day without breakfast. In the wild, the flock’s foraging begins soon after the sun rises. Chances are your bird wakes in morning waiting patiently (or impatiently) for his or her food. Birds are breakfast eaters for sure … if you asked your bird, he or she probably couldn’t fathom why their favorite humans would forgo breakfast. Birds don’t need a doctor to tell them that breakfast does a body good, but us silly humans sometimes need a little reminding, so here are some good reasons to eat breakfast, according to WebMD. Breakfast kick-starts your metabolism, helping to burn calories throughout the day, and it helps you focus at work or at school. Many studies link eating breakfast to better memory and concentration, lower levels of “bad” LDL cholesterol, and lower chances of getting diabetes, heart disease, and being overweight — no wonder breakfast is often referred to as the most important meal of the day.

keys, par_embeddings = firebase_model.generate_keys(pars, num_words=2, thresh=0.4, top_n=1)
keys_____no_output_____

# Final notes

In the above documents, we get a pretty good performance in terms of the quality of the images. Failure cases are (1) getting dummy paragraphs (which as shown in Doc 4 and Doc 5, their keywords typically result in no Bing images), (2) getting irrelevant keywords (which doesn't occur in the above documents, but by changing the threshold of keyword extraction, we can squeeze irrelevant terms out the returned keywords), and (3) when keywords are out-of-vocab of the embedding model, then we need to resort to the baseline (i.e., returning the first Bing result)._____no_output_____
# Run FIREBASE for the selected document_____no_output_____

results = get_bing_results_per_paragraph(firebase_model, keys, par_embeddings)[INFO] Total of 35 images found.
Obtaining paragraphs for the bing images for the keyword electromagnetic radiation...
This is an image tag, but does not contain an image source:
DONE!
Computing scores for the images...
DONE!
[INFO] Total of 35 images found.
Obtaining paragraphs for the bing images for the keyword eyeball extraocular...
This is an image tag, but does not contain an image source:
This is an image tag, but does not contain an image source: <img class="j-tooltip-thumb tooltip-thumb" onerror="this.src=''" slide-thumb-1="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-1-320.jpg?cb=1508905076" slide-thumb-10="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-10-320.jpg?cb=1508905076" slide-thumb-11="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-11-320.jpg?cb=1508905076" slide-thumb-12="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-12-320.jpg?cb=1508905076" slide-thumb-13="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-13-320.jpg?cb=1508905076" slide-thumb-14="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-14-320.jpg?cb=1508905076" slide-thumb-15="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-15-320.jpg?cb=1508905076" slide-thumb-16="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-16-320.jpg?cb=1508905076" slide-thumb-17="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-17-320.jpg?cb=1508905076" slide-thumb-18="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-18-320.jpg?cb=1508905076" slide-thumb-19="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-19-320.jpg?cb=1508905076" slide-thumb-2="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-2-320.jpg?cb=1508905076" slide-thumb-20="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-20-320.jpg?cb=1508905076" slide-thumb-21="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-21-320.jpg?cb=1508905076" slide-thumb-3="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-3-320.jpg?cb=1508905076" slide-thumb-4="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-4-320.jpg?cb=1508905076" slide-thumb-5="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-5-320.jpg?cb=1508905076" slide-thumb-6="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-6-320.jpg?cb=1508905076" slide-thumb-7="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-7-320.jpg?cb=1508905076" slide-thumb-8="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-8-320.jpg?cb=1508905076" slide-thumb-9="https://image.slidesharecdn.com/lacrimalapparatuseyelidandexternalfeaturesofeyeball-161021005848/85/lacrimal-apparatus-eye-lid-and-external-features-of-eye-ball-9-320.jpg?cb=1508905076"/>
This is an image tag, but does not contain an image source:

Anatomy of the eyelids
by optometry student
5290 views

This is an image tag, but does not contain an image source:
This is an image tag, but does not contain an image source:
This is an image tag, but does not contain an image source:
This is an image tag, but does not contain an image source: <img class="j-tooltip-thumb tooltip-thumb" onerror="this.src=''" slide-thumb-1="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-1-320.jpg?cb=1422177529" slide-thumb-10="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-10-320.jpg?cb=1422177529" slide-thumb-11="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-11-320.jpg?cb=1422177529" slide-thumb-12="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-12-320.jpg?cb=1422177529" slide-thumb-13="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-13-320.jpg?cb=1422177529" slide-thumb-14="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-14-320.jpg?cb=1422177529" slide-thumb-15="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-15-320.jpg?cb=1422177529" slide-thumb-16="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-16-320.jpg?cb=1422177529" slide-thumb-17="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-17-320.jpg?cb=1422177529" slide-thumb-18="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-18-320.jpg?cb=1422177529" slide-thumb-19="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-19-320.jpg?cb=1422177529" slide-thumb-2="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-2-320.jpg?cb=1422177529" slide-thumb-3="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-3-320.jpg?cb=1422177529" slide-thumb-4="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-4-320.jpg?cb=1422177529" slide-thumb-5="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-5-320.jpg?cb=1422177529" slide-thumb-6="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-6-320.jpg?cb=1422177529" slide-thumb-7="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-7-320.jpg?cb=1422177529" slide-thumb-8="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-8-320.jpg?cb=1422177529" slide-thumb-9="https://image.slidesharecdn.com/2wedhojjtzghupapyw18-signature-e86948ceefe84d6283e50b325b325ecc8a75e67c392d032e47fe9080affa6e22-poli-150125091804-conversion-gate01/85/movement-of-eyeball-9-320.jpg?cb=1422177529"/>
This is an image tag, but does not contain an image source:

Lymphatic Drainage Of Breast
by Aayush Rai
3965 views

This is an image tag, but does not contain an image source:
This is an image tag, but does not contain an image source:
This is an image tag, but does not contain an image source:
This is an image tag, but does not contain an image source:
This is an image tag, but does not contain an image source:

# Visualize the results_____no_output_____

from IPython.core.display import HTML
from IPython.display import Javascript, display
from ipywidgets import widgets
import tqdm
import numpy as np
from matplotlib import image
import os
import json
import operator
from PIL import Image
import io
from IPython.core.debugger import set_trace

import random
import pathlib
import pickle_____no_output_____num_images_to_show_most = 9
img_offset = 0

assert img_offset >= -1

num_cols = 3
num_rows = int(min(3, np.ceil(num_images_to_show_most / num_cols)))
num_per_tab = num_cols * num_rows
num_tabs = len(list(results.keys()))
scale = 3

checkboxes = {}

tab_contents = []
for kk, cur_par in tqdm.tqdm(enumerate(results.keys()), desc='Setting up scored images'):
    cur_keyword = results[cur_par]['keyword']
    cur_images = results[cur_par]['images']
    cur_scores = results[cur_par]['scores']
    num_images_to_show = int(min(num_images_to_show_most, len(cur_images)))
    num_rows = int(min(25, np.ceil(num_images_to_show / num_cols)))
    num_per_tab = num_cols * num_rows

    cur_sort_idx = list(np.argsort(cur_scores)[::-1])

    cur_images = [cur_images[i] for i in cur_sort_idx]
    cur_scores = [cur_scores[i] for i in cur_sort_idx]

    print('cur_keyword: {}\nnum_rows: {}\nnum_images_to_show: {}'.format(cur_keyword, num_rows, num_images_to_show))

    rows = []
    cur_num_rows = num_rows
    for ii in range(cur_num_rows):
        cur_row = []
        cur_num_cols = int(min(num_cols, num_images_to_show))
        if ii == cur_num_rows - 1:
            cur_num_cols = num_images_to_show - (cur_num_rows - 1) * num_cols
        for jj in range(cur_num_cols):
            cur_index = img_offset + ii * num_cols + jj
            cur_img = widgets.Image(value=get_image_with_url(cur_images[cur_index], binary=True, save_image=True, title=cur_keyword, size=(512,512), score=cur_scores[cur_index]), width=256, height=256)
            cur_checkbox = widgets.Checkbox(disabled=True, description=str(cur_scores[cur_index]), indent=False, layout=widgets.Layout(width='100px', height='28'))
            cur_checkbox.width = '90px'
            # checkboxes[random_imgs[cur_index].stem] = cur_checkbox
            cur_box = widgets.VBox([cur_img, cur_checkbox])
            cur_box.layout.align_items = 'center'
            cur_box.layout.padding = '6px'
            cur_row.append(cur_box)
        cur_hbox = widgets.HBox(cur_row)
        rows.append(cur_hbox)
    tab_contents.append(widgets.VBox(rows))

tab = widgets.Tab()
tab.children = tab_contents
for i in range(len(tab.children)):
    tab.set_title(i, list(results.values())[i]['keyword'])

display(tab)
Setting up scored images: 0it [00:00, ?it/s]