# Notebook from JLDaniel77/DS-Unit-1-Sprint-1-Dealing-With-Data
Path: module2-loadingdata/LS_DS_112_Loading_Data.ipynb

# Lambda School Data Science - Loading Data

Data comes in many shapes and sizes - we'll start by loading tabular data, usually in csv format.

Data set sources:

- https://archive.ics.uci.edu/ml/datasets.html
- https://github.com/awesomedata/awesome-public-datasets
- https://registry.opendata.aws/ (beyond scope for now, but good to be aware of)

Let's start with an example - [data about flags](https://archive.ics.uci.edu/ml/datasets/Flags)._____no_output_____## Lecture example - flag data_____no_output_____

# Step 1 - find the actual file to download

# From navigating the page, clicking "Data Folder"
flag_data_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/flags/flag.data'

# You can "shell out" in a notebook for more powerful tools
# https://jakevdp.github.io/PythonDataScienceHandbook/01.05-ipython-and-shell-commands.html

# Funny extension, but on inspection looks like a csv
!curl https://archive.ics.uci.edu/ml/machine-learning-databases/flags/flag.data

# Extensions are just a norm! You have to inspect to be sure what something is_____no_output_____# Step 2 - load the data

# How to deal with a csv? 🐼
import pandas as pd
flag_data = pd.read_csv(flag_data_url)_____no_output_____# Step 3 - verify we've got *something*
flag_data.head()_____no_output_____# Step 4 - Looks a bit odd - verify that it is what we want
flag_data.count()_____no_output_____!curl https://archive.ics.uci.edu/ml/machine-learning-databases/flags/flag.data | wc_____no_output_____# So we have 193 observations with funny names, file has 194 rows
# Looks like the file has no header row, but read_csv assumes it does
help(pd.read_csv)_____no_output_____# Alright, we can pass header=None to fix this
flag_data = pd.read_csv(flag_data_url, header=None)
flag_data.head()_____no_output_____flag_data.count()_____no_output_____flag_data.isna().sum()_____no_output_____

### Yes, but what does it *mean*?

This data is fairly nice - it was "donated" and is already "clean" (no missing values). But there are no variable names - so we have to look at the codebook (also from the site).

```
1. name: Name of the country concerned
2. landmass: 1=N.America, 2=S.America, 3=Europe, 4=Africa, 4=Asia, 6=Oceania
3. zone: Geographic quadrant, based on Greenwich and the Equator; 1=NE, 2=SE, 3=SW, 4=NW
4. area: in thousands of square km
5. population: in round millions
6. language: 1=English, 2=Spanish, 3=French, 4=German, 5=Slavic, 6=Other Indo-European, 7=Chinese, 8=Arabic, 9=Japanese/Turkish/Finnish/Magyar, 10=Others
7. religion: 0=Catholic, 1=Other Christian, 2=Muslim, 3=Buddhist, 4=Hindu, 5=Ethnic, 6=Marxist, 7=Others
8. bars: Number of vertical bars in the flag
9. stripes: Number of horizontal stripes in the flag
10. colours: Number of different colours in the flag
11. red: 0 if red absent, 1 if red present in the flag
12. green: same for green
13. blue: same for blue
14. gold: same for gold (also yellow)
15. white: same for white
16. black: same for black
17. orange: same for orange (also brown)
18. mainhue: predominant colour in the flag (tie-breaks decided by taking the topmost hue, if that fails then the most central hue, and if that fails the leftmost hue)
19. circles: Number of circles in the flag
20. crosses: Number of (upright) crosses
21. saltires: Number of diagonal crosses
22. quarters: Number of quartered sections
23. sunstars: Number of sun or star symbols
24. crescent: 1 if a crescent moon symbol present, else 0
25. triangle: 1 if any triangles present, 0 otherwise
26. icon: 1 if an inanimate image present (e.g., a boat), otherwise 0
27. animate: 1 if an animate image (e.g., an eagle, a tree, a human hand) present, 0 otherwise
28. text: 1 if any letters or writing on the flag (e.g., a motto or slogan), 0 otherwise
29. topleft: colour in the top-left corner (moving right to decide tie-breaks)
30. botright: Colour in the bottom-left corner (moving left to decide tie-breaks)
```

Exercise - read the help for `read_csv` and figure out how to load the data with the above variable names. One pitfall to note - with `header=None` pandas generated variable names starting from 0, but the above list starts from 1..._____no_output_____## Your assignment - pick a dataset and do something like the above

This is purposely open-ended - you can pick any data set you wish. It is highly advised you pick a dataset from UCI or a similar "clean" source.

If you get that done and want to try more challenging or exotic things, go for it! Use documentation as illustrated above, and follow the 20-minute rule (that is - ask for help if you're stuck).

If you have loaded a few traditional datasets, see the following section for suggested stretch goals._____no_output_____

# TODO your work here!
# And note you should write comments, descriptions, and add new
# code and text blocks as needed
import pandas as pd

# Assign data url to variable
mpg_data_url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data'

# load data
df = pd.read_csv(mpg_data_url)

# inspect data
df.head()_____no_output_____# Data is missing headers, reload data with column names
df = pd.read_csv(mpg_data_url, header=None, names=['mpg', 'cylinders', 'displacement',
                                                  'horsepower', 'weight', 'acceleration',
                                                  'model year', 'origin', 'car name'])

# Inspect data
df.head()_____no_output_____# Data is combined into 1 column, reload data
df = pd.read_csv(mpg_data_url, sep='\s+', header=None, names=['mpg', 'cylinders', 'displacement',
                                                  'horsepower', 'weight', 'acceleration',
                                                  'model year', 'origin', 'car name'])

# Inspect data
df.head(50)_____no_output_____import numpy as np

df_nan_values = df.replace('?', np.NaN)
df_nan_values.head(50)_____no_output_____# Check for missing data
df_nan_values.isnull().sum()
_____no_output_____print(df_nan_values.dtypes)_____no_output_____# Fill forward to replace nan values
df_no_nan = df_nan_values.fillna(method='ffill')
df_no_nan.head(50)_____no_output_____print(df_no_nan.dtypes)
print(df_no_nan.isnull().sum())mpg             float64
cylinders         int64
displacement    float64
horsepower       object
weight          float64
acceleration    float64
model year        int64
origin            int64
car name         object
dtype: object
mpg             0
cylinders       0
displacement    0
horsepower      0
weight          0
acceleration    0
model year      0
origin          0
car name        0
dtype: int64
# Horsepower still showing as type object, cast feature as float
df_no_nan['horsepower'] = df_no_nan['horsepower'].astype(float)_____no_output_____# Check data types
df_no_nan.dtypes_____no_output_____# Use one hot encoding on column 'car name' to convert categorical data to boolean
df_cleaned = pd.get_dummies(df_no_nan, columns=['car name'])
df_cleaned.head()_____no_output_____

## Stretch Goals - Other types and sources of data

Not all data comes in a nice single file - for example, image classification involves handling lots of image files. You still will probably want labels for them, so you may have tabular data in addition to the image blobs - and the images may be reduced in resolution and even fit in a regular csv as a bunch of numbers.

If you're interested in natural language processing and analyzing text, that is another example where, while it can be put in a csv, you may end up loading much larger raw data and generating features that can then be thought of in a more standard tabular fashion.

Overall you will in the course of learning data science deal with loading data in a variety of ways. Another common way to get data is from a database - most modern applications are backed by one or more databases, which you can query to get data to analyze. We'll cover this more in our data engineering unit.

How does data get in the database? Most applications generate logs - text files with lots and lots of records of each use of the application. Databases are often populated based on these files, but in some situations you may directly analyze log files. The usual way to do this is with command line (Unix) tools - command lines are intimidating, so don't expect to learn them all at once, but depending on your interests it can be useful to practice.

One last major source of data is APIs: https://github.com/toddmotto/public-apis

API stands for Application Programming Interface, and while originally meant e.g. the way an application interfaced with the GUI or other aspects of an operating system, now it largely refers to online services that let you query and retrieve data. You can essentially think of most of them as "somebody else's database" - you have (usually limited) access.

*Stretch goal* - research one of the above extended forms of data/data loading. See if you can get a basic example working in a notebook. Image, text, or (public) APIs are probably more tractable - databases are interesting, but there aren't many publicly accessible and they require a great deal of setup._____no_output_____


# Notebook from pints-team/electrochemistry_course
Path: Lecture-1-Bayesian-Inference.ipynb

import numpy as np

##################################################
##### Matplotlib boilerplate for consistency #####
##################################################
from ipywidgets import interact
from ipywidgets import FloatSlider
from matplotlib import pyplot as plt

%matplotlib inline

from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')

global_fig_width = 10
global_fig_height = global_fig_width / 1.61803399
font_size = 14

plt.rcParams['axes.axisbelow'] = True
plt.rcParams['axes.edgecolor'] = '0.8'
plt.rcParams['axes.grid'] = True
plt.rcParams['axes.labelpad'] = 8
plt.rcParams['axes.linewidth'] = 2
plt.rcParams['axes.titlepad'] = 16.0
plt.rcParams['axes.titlesize'] = font_size * 1.4
plt.rcParams['figure.figsize'] = (global_fig_width, global_fig_height)
plt.rcParams['font.sans-serif'] = ['Computer Modern Sans Serif', 'DejaVu Sans', 'sans-serif']
plt.rcParams['font.size'] = font_size
plt.rcParams['grid.color'] = '0.8'
plt.rcParams['grid.linestyle'] = 'dashed'
plt.rcParams['grid.linewidth'] = 2
plt.rcParams['lines.dash_capstyle'] = 'round'
plt.rcParams['lines.dashed_pattern'] = [1, 4]
plt.rcParams['xtick.labelsize'] = font_size
plt.rcParams['xtick.major.pad'] = 4
plt.rcParams['xtick.major.size'] = 0
plt.rcParams['ytick.labelsize'] = font_size
plt.rcParams['ytick.major.pad'] = 4
plt.rcParams['ytick.major.size'] = 0
##################################################_____no_output_____

# Lecture 1: Bayesian Inference and PINTS

by Martin Robinson

- Oxford Research Software Engineering group (http://www.cs.ox.ac.uk/projects/RSE)
- Department of Computer Science, University of Oxford
_____no_output_____![](fig/about_me.svg)

- PhD in Mathematical Science at Monash University
- Postdoc at the University of Twente, Netherlands
- Currently:
   - Senior Research Software Engineer in the [Oxford RSE group](http://www.cs.ox.ac.uk/projects/RSE)
   - Co-director of EPSRC & MRC [Sustainable Approaches to Biomedical Science Centre for Doctoral Training: Responsible and Reproducible Research](https://sabsr3.web.ox.ac.uk), or SABS:R3
  _____no_output_____## Research Interests include:

- numerical modelling and simulation
- particle-based methods
- Bayesian inference
- developing robust and reliable software for research
     - [Aboria](https://github.com/aboria/Aboria) (C++ library for particle-based modelling methods)
     - [Chaste](https://chaste.cs.ox.ac.uk) (C++ library for electrophysiology and discrete cellular modelling)
     - [PINTS](https://github.com/pints-team/pints) (Python library for Parameter Inference for Time-Series Models)
     - [Smoldyn](http://www.smoldyn.org/) (computer program for cell-scale biochemical simulations)
     - [SPH-DEM](https://github.com/martinjrobins/SPH-DEM) (C++ library for coupled Smoothed Particle Hydrodynamics and Discrete Elements Modelling)
     - [PyBaMM](https://github.com/tinosulzer/PyBaMM/tree/master/pybamm) (Python library for modelling batteries)
     - [Trase](https://github.com/trase-cpp/trase) (C++ animated plotting library)
    _____no_output_____# Course Structure

- **Lecture 1: Introduction to Bayesian Inference and Pints**
   - **What is Bayesian Inference?**
   - **Bayes Theorem: Priors, Likelihood functions and Posteriors**
   - **Introduction to PINTS**
   - **Using your own models in PINTS**
- Lecture 2: Maximum Likelihood Estimation
- Lecture 3: MCMC sampling
- Lecture 4: Hierarchical models_____no_output_____### Target audience:

- Researchers who want to apply statistical inference in their work.
- toy models used to illustrate concepts, but with an emphisis on *time-series* models

### Prerequisites:

- A basic knowledge of mathematical programming in Python, or similar.
- some basic statistics (probability distributions - continuous & discrete) and calculus _____no_output_____## Slides

- Lecture slides written as a Jupyter notebook (https://jupyter.org/)
- Converted to slides using the RISE plugin (https://github.com/damianavila/RISE)
- *Almost all* plots in these lectures generated using Python code embedded in the notebook
- Some example code is shown in the slides, for the rest you can look in the notebook
   - Get all material here: https://github.com/pints-team/electrochemistry_course
   - Need to install PINTs to execute the notebooks, instructions here: https://github.com/pints-team/pints_____no_output_____# Acknowledgements

- Most of the material for this course was taken from the textbook and lecture series by Dr Ben Lambert
   - *Lambert, B. (2018). A Student’s Guide to Bayesian Statistics. Sage.*
   - [https://ben-lambert.com](https://ben-lambert.com)
   - Ben Lambert is also a fellow [PINTS](https://github.com/pints-team/pints) developer

![](fig/lambert_book.jpg)

_____no_output_____# A conceptual introduction to inference

### The Big world
1. Consider an observable characteristic we are trying to
explain, for example the heights of 5 randomly chosen
individuals.
2. Assume that there exists a true process $T$ that generates
the heights of all individuals in our sample.
3. There is variability in the observables outputted by $T$; this
can due to the inherent
variability or because we lack knowledge
of the genetics and environmental factors that affect
growth.
4. Imagine a set of all conceivable processes that could result
in our sample of height observations, which we call the
“Big World”._____no_output_____

![](fig/big_world.svg)

*Images adapted from “A Technical Introduction to Probability and
Bayesian Inference for Stan Users”, Stan Development Team, 2016.*_____no_output_____## What is inference?

- **Motivation:** update our knowledge of $T$ in light of data,
and use the updated knowledge to estimate quantities of
interest.
  - In our height example we might want to estimate the
mean height of the entire population having witnessed our
sample of 5 individuals.

### Method:
1. Find areas of the Big World that are closest to $T$; ideally
we would find $T$ itself!
2. Estimate quantities of interest using these subsets of the
*Small World*._____no_output_____## The Small World

1. The infinity of the Big World is too large to be useful.
2. Instead we first consider a subset of possible data
generating processes which we call the “Small World”, or
$\Omega$.
3. The Small World corresponds to a single probability model
framework; in our height example we might suppose that
$H \sim  N(\mu, \sigma)$, where $\mu$ is the mean height, and $\sigma$ is their
standard deviation.
4. By varying our parameters $\theta = (\mu, \sigma)$ we get different data
generating processes.
5. The collection of probability distributions we get by varying
$\theta \in \Omega$ in the Small World is known as the *Likelihood*._____no_output_____

![](fig/small_world.svg)

*An unlikely Small World*

![](fig/small_world2.svg)

*A Boxian Small World: “All models are wrong but some are useful” (George Box, 1976)*
_____no_output_____## The Prior

1. The Small World is still too big for our purposes.
2. We usually have some knowledge about which areas of the
Small World are nearest to $T$. For example we don’t
believe that $\mu = 100$m and $\mu = 1.5$m are equally probable.
3. As such, in *Bayesian* inference we define a prior probability
density that gives a weighting to all $\theta \in \Omega$ reflecting our
beliefs.
4. *Frequentist* inference does not require us to specify a prior_____no_output_____

![](fig/the_prior.svg)
*A Gaussian prior for $\Omega$*

_____no_output_____## The data

1. Inference is the process of updating our prior knowledge in
light of data.
2. In Bayesian inference with a likelihood and our prior
knowledge explicitly stated we use *Bayes’ rule* to find our
posterior probability density over $\theta \in \Omega$.

_____no_output_____

![](fig/posterior.svg)
_____no_output_____## Summary of the inference process

1. Define the observables: The Big World
2. Specify a likelihood (Small World)
3. Specify a prior
4. Input the data, calculate the posterior_____no_output_____# Tangible benefits of Bayesian inference

- Straightforward application to scientific modelling and experimentatal data analysis
- Simple and intuitive model building (unlike frequentist statistics there is no need to remember lots of specific formulae).
- Exhaustive and creative model testing.
- Straightforward interpretation of results.
_____no_output_____## Frequentist and Bayesian world views

The two paradigms differ in their definition of probability:

**Frequentist:** $\;P(A)\;$ describes the limiting frequency of an event $A$.
 - there is a fixed value of $\;P(A)\;$ that must be calculated
   - e.g. proportion of heads from a fair coin toss will approach 0.5 after a large number of trials.
 - Does not make sense to “update” probabilities.

**Bayesian:** $\;P(A)\;$ is a measure of centainty, quantification of investigators belief that $\;A\;$ is true.
 - A fixed value of $\;P(A)\;$ is not neccessary, nor desirable.
 - Prior information must be used to augment sample data
 - $\rightarrow$ free to update our beliefs using *Bayes’ rule*!_____no_output_____## Bayes' Rule

- Inference involves finding parameter values, or distributions of parameters values, for which model outputs are consistent with observations

- Bayesian inference uses Bayes' rule to update prior beliefs after obtaining new data $y$

```
                     Bayes' Rule
likelihood + prior -------------------> posterior
```

 - **Likelihood function:** the probability of obtaining the data $y$, given a set of parameters $\theta$
 - **Prior probability distribution:** encodes your uncertainty in the parameters before the data $y$ has been obtained
 - **Posterior distribution:** updated probability distribution of $\theta$, given the new data $y$
_____no_output_____## Bayes' Rule:

An Essay towards solving a Problem in the Doctrine of Chances (1763)

![Thomas Bayes](fig/225px-Thomas_Bayes.gif)
*Reverend Thomas Bayes (1701?–1761)*
_____no_output_____# Likelihoods

$$P(\theta | data) = \frac{\color{red}{{P(data|\theta)}} P(\theta)}{P(data)}$$

- common to both Frequentist and Bayesian analyses
- probability of generating the particular sample of data, given the model parameters $\theta$
- normally easy to obtain, *given a good statistical model*
- often only interested in the *most likely* parameters (Maximum Likelihood Estimation, see Lecture 2)
- **The aim of inference:** inverting the likelihood
 - Both Frequentists and Bayesians essentially invert:

$$P(data|\theta) \rightarrow P(\theta | data).$$

 - This amounts to going from an ’effect’ back to a ’cause’._____no_output_____# Example: flipping a coin

![](fig/coin.jpeg)

Take the classic example of tossing a fair coin that has a probability landing heads up of $\theta = 0.5$

If we flip the coin 2 times, we have the set of possible outcomes:

$$P(H, H | \theta = 0.5) = P(H|\theta=0.5) P(H|\theta=0.5) = 0.25$$
$$P(H, T | \theta = 0.5) = P(H|\theta=0.5) P(T|\theta=0.5) = 0.25$$
$$P(T, H | \theta = 0.5) = P(T|\theta=0.5) P(H|\theta=0.5) = 0.25$$
$$P(T, T | \theta = 0.5) = P(T|\theta=0.5) P(T|\theta=0.5) = 0.25$$

This is a valid probability distribution:

$$ P(H, H | \theta = 0.5) + P(H, T | \theta = 0.5) + P(T, H | \theta = 0.5)+ P(T, T | \theta = 0.5) = 1.0$$_____no_output_____# Calculating the likelihood

- Hold the **data constant**, and find the likelihood of this data given a certain $\theta$
- Provides an infinite number of possibilities to arrive at the given data
- Take for example the case of $\text{data} = H, H$
- The likelihood function is given by $P(data|\theta) = \theta^2$

_____no_output_____

def show_coin_likelihood_hh():
    theta = np.linspace(0,1,100)
    likelihood = theta*theta
    plt.plot(theta,likelihood)
    plt.xlabel(r'$\theta$')
    plt.ylabel('likelihood')
    plt.fill_between(theta,likelihood,alpha=0.2)
    plt.show()
    return likelihood,theta

def show_coin_likelihood_ht():
    theta = np.linspace(0,1,100)
    likelihood = theta*(1-theta)
    plt.plot(theta,likelihood)
    plt.xlabel(r'$\theta$')
    plt.ylabel('likelihood')
    plt.fill_between(theta,likelihood,alpha=0.2)
    plt.show()
    return likelihood,theta_____no_output_____likelihood,theta = show_coin_likelihood_hh()_____no_output_____

This is **not** a valid probability distribution_____no_output_____

print('integral under the likelihood = ',np.trapz(likelihood,x=theta))integral under the likelihood =  0.33335033840084355

## Different data this time

- Take for example the case of $\text{data} = H, T$
- The likelihood function is given by $P(data|\theta) = \theta (1 - \theta)$ _____no_output_____

likelihood,theta = show_coin_likelihood_ht()_____no_output_____

# Priors

$$P(\theta | data) = \frac{{P(data|\theta)} \color{red}{P(\theta)}}{P(data)}$$

- This is particular to Bayesian inference, where we always update our *prior* beliefs using the new data
- This is a function of all the possible parameter values $\theta$_____no_output_____# Example Prior

Back to the coin example, we know that the possible domain for our parameter is $\theta \in [0, 1]$.
1. One option is the consider all the possible values of $\theta$ to be equally likely (i.e. a Uniform prior).
2. Another is to use our previous knowledge that most coins are likely to be fair (i.e. a Gaussian prior around $\theta = 0.5$)_____no_output_____

import numpy as np
import scipy
import scipy.stats
import math

def show_priors():
    x = np.linspace(0, 1, 100)
    f, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 8/1.618))
    ax1.plot(x,scipy.stats.uniform.pdf(x, loc=0, scale=1), label='Uniform prior')
    ax1.fill_between(x,scipy.stats.uniform.pdf(x, loc=0, scale=1),alpha=0.2)

    ax2.plot(x,scipy.stats.norm.pdf(x, 0.5, 0.1), label='Gaussian prior')
    ax2.fill_between(x,scipy.stats.norm.pdf(x, 0.5, 0.1),alpha=0.2)

    ax1.set_xlabel(r'$\theta$')
    ax1.set_ylabel('probability density')
    ax2.set_xlabel(r'$\theta$')
    ax2.set_ylabel('probability density')_____no_output_____show_priors()_____no_output_____

# The devil's in the denominator

$$P(\theta | data) = \frac{P(data|\theta) P(\theta)}{\color{red}{P(data)}}$$

- represents the probability of obtaining our particular sample of data, given a particular model (i.e. likelihood) and prior
- **normalises** the posterior so the area under the probability distribution is 1

\begin{align*}
P(data) &= \int_{all \theta} P(data, \theta) d\theta \\
        &= \int_{all \theta} P(data|\theta) P(\theta) d\theta
\end{align*}

_____no_output_____## Calculating the denominator in 1 dimension

- For our coin example there is a single parameter $\theta \rightarrow P(H,H |\theta) = \theta^2$.
- For this example we will use the Uniform prior

$$
P(H,H) = \int_0^1 P(H,H |\theta) P(\theta) d \theta= \int_0^1 \theta^2 d \theta = \frac{1}{3} \theta^3
$$

This is equivalent to working out an area under a curve._____no_output_____## Calculating the denominator in 2 dimension

If we considered a different model where there were two
parameters $\theta_1 \in (0, 1), \theta_2 \in (0, 1) \rightarrow$:

$$
P(data) = \int_0^1 \int_0^1 P(data |\theta) P(\theta) d \theta_1 d \theta_2
$$

This is equivalent to working out a **volume** contained within a surface. _____no_output_____## Calculating the denominator in d dimensions

If we considered a different model where there were $d$
parameters ($\theta_1, \theta_2, ..., \theta_d) \in (0, 1) \rightarrow$:

$$
P(data) = \int_0^1 ... \int_0^1 P(data |\theta) P(\theta) d \theta_1 ... d \theta_d
$$

This is equivalent to working out a ($d$ + 1)-dimensional **volume** contained within a $d$-dimensional (hyper-surface)!

_____no_output_____## The difficult denominator

- Calculating the denominator possible for $d < 10$ using computers.
- Numerical quadrature and many other approximate schemes struggle for larger $d$.
- Many models have thousands of parameters.

Arrrghhh!

### Solutions:

- **Non-linear optimisation methods**: find the maximum of the posterior, either ignoring the rest of the posterior distribution or making a Gaussian assumption around the maximum point (Lecture 2)

- **Sampling:** understand a distribution by sampling from it, rather than exact calculation (Lecture 3).

- **Conjugate priors:** use certain pairs of distributions where we can follow basic rules to find posteriors, no calculation necessary! (Lecture 4)_____no_output_____# Posteriors: the goal of Bayesian inference

$$\color{red}{P(\theta | data)} = \frac{P(data|\theta) P(\theta)}{P(data)}$$

**For example:**

![](fig/coin.jpeg)

- How likely is our coin to be biased?
- We perform an experiment of 10 flips, and it lands heads up 7 times
- We will use the Uniform prior given earlier
- Likelihood is:

$$P(7\times H | \theta) = \theta^7 (1-\theta)^3$$

_____no_output_____- Lets run a number of different experiments, each using a different number of flips $f$, giving a different random number of heads $h$_____no_output_____

# experiment - f coin flips
def run_coin_experiment(f):
    theta = np.linspace(0, 1, 1000)
    h = np.random.poisson(f/2)

    # Uniform prior
    prior = np.ones(len(theta))

    # Likelihood function
    likelihood = theta**h * (1-theta)**(f-h)

    # Normalising factor (numerical integration)
    denominator = np.trapz(likelihood*prior, x=theta)

    # Bayes rule
    posterior = likelihood * prior / denominator

    plt.plot(theta,posterior, label='h = %s'%str(h))
    plt.fill_between(theta,posterior,alpha=0.2)
    plt.xlabel(r'$\theta$')
    plt.ylabel('posterior')
    plt.title('area under posterior is %f'%np.trapz(posterior, x=theta))
    plt.legend()
    _____no_output_____f_widget = FloatSlider(value=7.0, min=1.0, max=200.0, step=1.0, continuous_update=False)_____no_output_____
interact(run_coin_experiment, f=f_widget, continuous_update=False);_____no_output_____

## Summary

- All methods of inference involve the subjective decision of defining the boundaries of the Small World (**likelihood**).
- Likelihood determined by a combination of the *statistical model* **and** the *data*
- Small World inference involves *inversion of the likelihood*.
- Bayesians use **Bayes’ rule**, which requires us to specify a **prior**._____no_output_____# Probabilistic Inference on Noisy Time Series (PINTS)

- **Authors:** Michael Clerx, Martin Robinson, Ben Lambert, Chon Lok Lei, Sanmitra Ghosh, Gary R. Mirams, David J. Gavaghan
- An **open-source** (BSD 3-clause license) **Python** library that provides researchers with a broad suite of non-linear optimisation and sampling methods
- Uses a **Bayesian framework**: Many different Priors, Likelihood functions, MCMC samplers and non-linear optimisers available
- Use your *own pre-build model* for inference: users wrap their model and data in a transparent and straightforward interface
- Pre-print available on [arXiv](https://arxiv.org/abs/1812.07388)
_____no_output_____# Implementation and architecture

PINTS is designed around two core ideas:

1. PINTS should work with a wide range of time series models, and make no demands on how they are implemented other than a minimal input/output interface.
2. It is assumed that model evaluation (simulation) is the most costly step in any optimisation or sampling routine._____no_output_____![Class Hierarchy](fig/class-hierarchy-eps-converted-to.svg)_____no_output_____# Writing a model

- Pints is intended to work with a wide range of models, and assumes as little as possible about the model's form.
- a "model" in Pints is anything that implements the `ForwardModel` interface:
    - take a parameter vector $(\boldsymbol{\theta})$ and a sequence of times $(\mathbf{t})$ as an input,
    - and then return a vector of simulated values $(\mathbf{y})$:

$$f(\boldsymbol{\theta}, \mathbf{t}) \rightarrow \mathbf{y}$$_____no_output_____# Example Model

In the example below, we define a system of ODEs (modelling a simple chemical reaction) and use SciPy to solve it. We then wrap everything in a `pints.ForwardModel class`, and use a Pints optimisation to find the best matching parameters.

In this example we'll use a model of a reversible chemical reaction:

$$\dot{y}(t) = k_1 (1 - y) - k_2 y,$$

where $k_1$ represents a forward reaction rate, $k_2$ is the backward reaction rate, and $y$ represents the concentration of a chemical solute.

The next slide shows how you would implement this model using the standard Python package SciPy_____no_output_____

import numpy as np
from scipy.integrate import odeint

# Define the right-hand side of a system of ODEs
def r(y, t, p):
    k1 = p[0] # Forward reaction rate
    k2 = p[1] # Backward reaction rate
    dydt = k1 * (1 - y) - k2 * y
    return dydt

# Run an example simulation
p = [5, 3]    # parameters
y0 = 0.1      # initial conditions

# Call odeint, with the parameters wrapped in a tuple
times = np.linspace(0, 1, 1000)
values = odeint(r, y0, times, (p,))_____no_output_____# Plot the results
plt.figure()
plt.xlabel('Time')
plt.ylabel('Concentration')
plt.plot(times, values)
plt.show()_____no_output_____

# Writing a wrapper class for Pints

Now we'll wrap the model in a class that extends `pints.ForwardModel`.

It should have two methods:
  - `simulate()`: Run a simulation with the given parameters for the given times and return the simulated values
  - `n_parameters()`: Return the dimension of the parameter vector_____no_output_____

import pints

class ExampleModel(pints.ForwardModel):

    def simulate(self, parameters, times):
        y0 = 0.1
        def r(y, t, p):
            dydt = (1 - y) * p[0] - y * p[1]
            return dydt
        return odeint(r, y0, times, (parameters,)).reshape(times.shape)

    def n_parameters(self):
        return 2

# Then create an instance of our new model class
model = ExampleModel()

# Run the same simulation using our new model wrapper
values = model.simulate([5, 3], times)_____no_output_____# Plot the results
plt.figure()
plt.xlabel('Time')
plt.ylabel('Concentration')
plt.plot(times, values)
plt.show()_____no_output_____

# Running an optimisation problem

Now that our model implements the `pints.ForwardModel` interface, we can use it with Pints tools such as optimisers or MCMC.

First, we use the model to generate test data by adding some generated noise_____no_output_____

# Define the 'true' parameters
true_parameters = [5, 3]

# Run a simulation to get test data
values = model.simulate(true_parameters, times)

# Add some noise
values += np.random.normal(0, 0.02, values.shape)_____no_output_____# Show the test data
plt.figure()
plt.xlabel('Time')
plt.ylabel('Concentration')
plt.plot(times, values)
plt.show()_____no_output_____

# Error function

We then define a score function that characterises the mismatch between model predictions and data.

We will use the classic sum of squares error for this (This is related to maximising a Bayesian Likelihood function with indepedent Gaussian Noise, see next Lecture).

$$\sum_{i=0}^N (f(\boldsymbol{\theta}, t_i) - f^d_i)^2$$

where $\mathbf{f}^d$ is the vector of test data. We can tell Pints about the the test data $\mathbf{f}^d$ by wrapping it and the model $f(\boldsymbol{\theta}, \mathbf{t})$ in a `pints.SingleOutputProblem` object

We then use the SNES optimiser to estimate the model parameters from the data._____no_output_____

# Create an object with links to the model and time series
problem = pints.SingleOutputProblem(model, times, values)

# Select a score function
score = pints.SumOfSquaresError(problem)

# Select some boundaries
boundaries = pints.RectangularBoundaries([0.1, 0.1], [10, 10])

# Select a starting point
x0 = [1, 1]

# Perform an optimization using SNES.
found_parameters, found_value = pints.optimise(score, x0, boundaries=boundaries, method=pints.SNES)
print('Score at true solution:')
print(score(true_parameters))Minimising error measure
using Seperable Natural Evolution Strategy (SNES)
Running in sequential mode.
Population size: 6
Iter. Eval. Best      Time m:s
0     6      18.50209   0:00.0
1     12     12.05861   0:00.0
2     18     5.356118   0:00.0
3     24     3.722737   0:00.0
20    126    0.403      0:00.1
40    246    0.403      0:00.1
60    366    0.402      0:00.2
80    486    0.402      0:00.3
100   606    0.402      0:00.4
120   726    0.402      0:00.5
140   846    0.402      0:00.5
160   966    0.402      0:00.6
180   1086   0.402      0:00.6
200   1206   0.402      0:00.7
220   1326   0.402      0:00.8
240   1446   0.402      0:00.8
260   1566   0.402      0:00.9
280   1686   0.402      0:01.0
300   1806   0.402      0:01.0
320   1926   0.402      0:01.1
340   2046   0.402      0:01.2
360   2166   0.402      0:01.2
380   2286   0.402      0:01.3
385   2310   0.402      0:01.3
Halting: No significant change for 200 iterations.
Score at true solution:
0.4024944688284413
# Plot the results
plt.figure()
plt.xlabel('Time')
plt.ylabel('Concentration')
plt.plot(times, values, alpha=0.5, label='noisy signal')
plt.plot(times, problem.evaluate(found_parameters), label='recovered signal')
plt.legend()
plt.show()_____no_output_____

## Summary

- If you have a time-series model that can be wrapped in a python function, you can start using **PINTS**
- Future lectures illustrate using PINTS for:
    - *Maximum Likelihood* and *Maximum a Postiori Estimation* (Lecture 2)
    - *Markov Chain Monte Carlo* sampling (Lecture 3) and
    - *Hierarchical models* (Lecture 4)
- I will be at Monash for the next 2 weeks. *Very happy* to help if you need help getting PINTS working with your own models!
_____no_output_____


# Notebook from mguidoti/DSND-p6-capstone
Path: sp_nov.ipynb

# Udacity's Data Scientist ND Capstone

In this project, I connected five different data providers using different methodologies to asnwer some pending questions on the biological sciences, more specifically, the zoological systematics, regarding journals' contributions to the discovery of new species, and their respective open-access policies.

As a PhD in the field, it was quite excting and fun to work on this project.
_____no_output_____## Importing Libraries and Modules_____no_output_____

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

import pickle

from sqlalchemy import create_engine

import etl.extract.tb as tb_proc
import etl.extract.wos as wos_parser
import etl.transform.transform as transform
import etl.transform.doaj as doaj
import etl.transform.sherpa_romeo as sherpa
import etl.load as load
import test_units.test_extract as test_ext
import test_units.test_transform as test_tf

%matplotlib inline_____no_output_____

## CRISP-DM

This process was carefully conducted in order to decide the questions I would try to answer.

**Business Understanding** came from years of experience in the field. As I said before, I'm a PhD in this particular field.

**Data Understanding** came from the previously cited experience, and experiments with the data done in the scope of this project. All sources were known for me, which helped a lot in order to define the goals for this project.

**Prepare Data**, the ETL process, was the most time-consuming part of the project. Data from five different sources were combined to answer the proposed questions.

The **Results** were obtained, focusing solely on counts (descriptive stats) and data visualization. The questions allowed for this simplistic approach, but they are not, at all, simple to answer due to the complexity of their nature and the disperse, somehow low-quality, data.

No model or deployment were conduct, as this was either out of scope of this particular project or the project's rubric.

### Target questions
- How much of the newly discovered data end up behind paywalls?
- Who are the biggest journals in the science of discovering and describing things?
- How does the sources related to the Zoological Sciences compare to each other in this context?_____no_output_____## ETL

Data from five different providers were combined in order to addressed the proposed questions. Below I describe which provider and the method used to extract the data._____no_output_____### (E)xtraction

The goal is to have a dataset, per source, with all articles that described one or more species. This means that on the source-specific dataframes the rows will be articles, and the columns, their respective metadata. Here are the sources and the applied method of extraction:

|Source|Description|Method of Extraction|
|--|--|--|
|Web of Science/Zoological Record|According to their website, the "world's leading taxonomic reference and oldest continuing database of animal biology"|Parser on downloaded data|
|TreatmentBank|Taxonomic tretments repository by Plazi|API|
|Zoobank|Official repository for registering scientific names according to the International Code of Zoological Nomenclature|Web scraping|
|Directory of Open-Access Journals (DOAJ)|Directory dedicated to list open-access journals, regardless of their type|API|
|SHERPA/RoMEO|API that brings a variety of information about journals' open-access policies|API|_____no_output_____#### Web of Science/Zoological Records
WoS do not provide an API, as far as I know, and has strict access policies across the board of all of their products. Thus, I downloaded the search results using their GUI, manually, and coded a parser from scratch that loads the data from the 150+ files into a SQLite database.

The specifics on how I queried this system is in the report, Methods section._____no_output_____

# Parse data, creates df
# I commented out this command when I was working on the notebook, to accelerate the process of reloading this dataset
wos = wos_parser.create_df()

# This is commented out because it generates a over 200Mb file that I couldn't push to GitHub, so I'm leaving it here
#wos_parser.to_pickle(wos, 'data/wos.p')

# Use the non-provided pickle file to reload this, if necessary (faster than re-parsing it!)
#wos = pd.read_pickle('data/wos.p')_____no_output_____# Checking if it loaded
wos.head()_____no_output_____

#### TreatmentBank (Plazi)
TreatmentBank has a powerful API with GUI. I used it to built the target query, and copied/pasted their URL into my code. From there I processed the results, adding them into a SQLite database. _____no_output_____

# Process the hard-coded API call and creates a database from it
# I'm commenting out because the SQLite db was provided and it contains a snapshot of the data I analyzed. As this includes an API call, running this now might get new data, changing my results

#tb_proc.create_db(tb_proc.get_data())

# Alternatively, I can simply load the .db
db_connect = create_engine('sqlite:///data/tb.db')

tb = pd.read_sql("docs", con=db_connect)_____no_output_____# Checking if it loaded
tb.head()_____no_output_____

#### Zoobank
As explained, I performed a web scraping to collect data from Zoobank. For this, I used a Scrapy project, with custom Item, ItemLoader, in- and output processors and ItemPipeline. These can't quite be run from here, as far as I know, I can only create a CrawlerProcess that won't have these functionalities. This was then saved into a SQLite database, which is alternatively loaded here._____no_output_____

# Load the .db created from the web scraping
db_connect = create_engine('sqlite:///data/zb.db')

zb = pd.read_sql("docs", con=db_connect)_____no_output_____# Checking if it loaded
zb.head()_____no_output_____

#### Exploratory Analyses: Extraction Tests
Here I will make a series of tests to check if I'm getting all the data that I was supposed to be getting. For that, I'm going to use Python's assert, and some manual observations.
_____no_output_____##### Web of Science/Zoological Records
These were the observed results from the WoS GUI, per year:

- 2010: 7207 results
- 2011: 7480 results
- 2012: 7604 results
- 2013: 7655 results
- 2014: 7456 results
- 2015: 7983 results
- 2016: 7859 results
- 2017: 8023 results
- 2018: 7973 results
- 2019: 7664 results
- 2020: 3548 results
_____no_output_____

# Create a dict with year: official results, to be used on a data consistency test
wos_results = {
    '2010': 7207,
    '2011': 7480,
    '2012': 7604,
    '2013': 7655,
    '2014': 7456,
    '2015': 7983,
    '2016': 7859,
    '2017': 8023,
    '2018': 7973,
    '2019': 7664,
    '2020': 3548,
}_____no_output_____# Checkout column names to get the right one: 'Year Published'
wos.columns_____no_output_____# Create a pandas.Series with the ordered counts of the 'Year Published' column
wos_years_count = wos['Year Published'].value_counts().sort_index()_____no_output_____# Calling the test with WoS/Zoological Record data
test_ext.check_consistency("WoS/Zoological Records", wos_years_count, wos_results)--------------------------------------------------------------------------
Testing for... WOS/ZOOLOGICAL RECORDS
--------------------------------------------------------------------------
SUCCESS! Total count was equal between the processed and official results!
--------------------------------------------------------------------------
SUCCESS! the year 2010 seems to be fine!
SUCCESS! the year 2011 seems to be fine!
SUCCESS! the year 2012 seems to be fine!
SUCCESS! the year 2013 seems to be fine!
SUCCESS! the year 2014 seems to be fine!
SUCCESS! the year 2015 seems to be fine!
SUCCESS! the year 2016 seems to be fine!
SUCCESS! the year 2017 seems to be fine!
SUCCESS! the year 2018 seems to be fine!
SUCCESS! the year 2019 seems to be fine!
SUCCESS! the year 2020 seems to be fine!
--------------------------------------------------------------------------

##### TreatmentBank
I can't make new API calls. The reason why is that as the data is live I might risk to get more data than I have got when I first run this, which could cause different results.

That's why I saved a .pickle file with the entire response of the API on the routine call, besides the .db resulting of the processing of this data. As such, I can compare the total number of records in both (and per year!) to check if my processing missed something or not, using the same function applied to WoS data._____no_output_____

# Creating a dict with the counts of records per year, based on the total response saved from the API call to TreatmentBank

# Open .pickle
with open('data/tb_response.p', 'rb') as p_file:
        tb_response = pickle.load(p_file)

tb_response_years_count = dict()

# Iterate over the full response, and create the counting
for result in tb_response['data']:
    try:
        tb_response_years_count[result['BibYear']] += 1
    except:
        tb_response_years_count[result['BibYear']] = 1_____no_output_____# Create a pandas.Series with the ordered counts of the 'year' column
tb_years_count = tb['year'].value_counts().sort_index()_____no_output_____# Calling the test with TreatmentBank data
test_ext.check_consistency("TreatmentBank", tb_years_count, tb_response_years_count)--------------------------------------------------------------------------
Testing for... TREATMENTBANK
--------------------------------------------------------------------------
SUCCESS! Total count was equal between the processed and official results!
--------------------------------------------------------------------------
SUCCESS! the year 2014 seems to be fine!
SUCCESS! the year 2018 seems to be fine!
SUCCESS! the year 2015 seems to be fine!
SUCCESS! the year 2016 seems to be fine!
SUCCESS! the year 2013 seems to be fine!
SUCCESS! the year 2017 seems to be fine!
SUCCESS! the year 2011 seems to be fine!
SUCCESS! the year 2012 seems to be fine!
SUCCESS! the year 2020 seems to be fine!
SUCCESS! the year 2010 seems to be fine!
SUCCESS! the year 2019 seems to be fine!
--------------------------------------------------------------------------

##### Zoobank
These were the observed results from the Zoobank, per year:

- 2010: 1128 results
- 2011: 1430 results
- 2012: 2482 results
- 2013: 4293 results
- 2014: 4829 results
- 2015: 5552 results
- 2016: 5732 results
- 2017: 6098 results
- 2018: 6401 results
- 2019: 6386 results
- 2020: 4554 results_____no_output_____

# Create a dict with year: official results, to be used on a data consistency test
zb_results = {
    '2010': 1128,
    '2011': 1430,
    '2012': 2482,
    '2013': 4293,
    '2014': 4829,
    '2015': 5552,
    '2016': 5732,
    '2017': 6098,
    '2018': 6401,
    '2019': 6386,
    '2020': 4554,
}_____no_output_____# Check the existing columns in this dataframe
list(zb.columns)_____no_output_____

Here there is one interesting caveat. Zoobank allow users to register papers from the past, and when queried by year, it shows not the publications published on that year, but the publications **registred** on that year.

As such, we might have all sort of years in the column 'year'._____no_output_____

# Create a pandas.Series with the ordered counts of the 'year' column
zb_years_count = zb['year'].value_counts().sort_index()

# Check results
zb_years_count_____no_output_____

To address this situation, I included a column on zb.db called 'from_year', which extracts the year being queried on the response.url, inside the parser() of the spider used to scrape Zoobank, and add to this column.

This means that we can get the total (included) per year, to compare with the official results._____no_output_____

# Create a pandas.Series with the ordered counts of the 'Year Published' column
zb_years_count = zb['from_year'].value_counts().sort_index()

# Check results
zb_years_count_____no_output_____# Now that I have both dictionary with official results and the pandas.Series with the processed results, I can call the unit test function
test_ext.check_consistency("Zoobank", zb_years_count, zb_results) --------------------------------------------------------------------------
Testing for... ZOOBANK
--------------------------------------------------------------------------
SUCCESS! Total count was equal between the processed and official results!
--------------------------------------------------------------------------
SUCCESS! the year 2010 seems to be fine!
SUCCESS! the year 2011 seems to be fine!
SUCCESS! the year 2012 seems to be fine!
SUCCESS! the year 2013 seems to be fine!
SUCCESS! the year 2014 seems to be fine!
SUCCESS! the year 2015 seems to be fine!
SUCCESS! the year 2016 seems to be fine!
SUCCESS! the year 2017 seems to be fine!
SUCCESS! the year 2018 seems to be fine!
SUCCESS! the year 2019 seems to be fine!
SUCCESS! the year 2020 seems to be fine!
--------------------------------------------------------------------------

It seems that the data I processed into databases are consistent with the data observed on the sources and that the processing were completed flawlessly.

It's time to move on._____no_output_____### (T)ransform

I'm aiming one single pandas.DataFrame with the counts of papers for each one of the three data sources, grouped per journal, with years on the columns. Thus, it's a MultiIndex dataframe (journal x Source/Year). But to achieve this, I want first check the following aspects of these data for all three sources:

1. Check for duplicates
2. Check the amount of NaN on the to-be-used 'year' columns
3. Group by journal and check the data quality of the journals' names
_____no_output_____#### Exploratory Analyses: Before Transform
_____no_output_____##### Web of Science/Zoological Records_____no_output_____

# First, check the available columns again
wos.columns_____no_output_____

###### Duplicates

With this I can see that the column 'Digital Object Identifier (DOI)' might contain a persistent, unique identifier, but not all papers will have it. So I'll perform two different checks for duplicates for this source:

1. Based on 'Digital Object Identifier (DOI)'
2. Based on the columns: 'Source', 'Year Published', 'Volume', 'Issue', 'Page Span'_____no_output_____

# Method 01 - DOI

# First, select the ones that has DOIs
wos_duplicates = wos.dropna(subset=['Digital Object Identifier (DOI)'], how='all', inplace=False)

# Checks if there were any NaNs in this column
wos_duplicates.shape[0] - wos.shape[0]_____no_output_____

This high number simply means that there are a lot of NaNs in the column 'Digital Object Identifier (DOI) - which is actually expected. Moving on..._____no_output_____

# Then, search for duplicates. 'True' will indicate the existence of them.
wos_duplicates.duplicated(subset=['Digital Object Identifier (DOI)'], keep=False).value_counts()_____no_output_____# Checking these four duplicates...
wos_duplicates[wos_duplicates.duplicated(subset=['Digital Object Identifier (DOI)'], keep=False)]_____no_output_____# They doesn't seem to be duplicates. Check their DOIs
wos_duplicates[wos_duplicates.duplicated(subset=['Digital Object Identifier (DOI)'], keep=False)]['Digital Object Identifier (DOI)']_____no_output_____

They have the same DOI, but they are clearly different papers (if you look into title, authors, and everything else!) So, **false-positives**, despite the same DOIs. Moving on..._____no_output_____

# Method 02

# Stores records that is not NaN on all selected fields
wos_duplicates = wos.dropna(subset=['Source', 'Year Published', 'Volume', 'Issue', 'Page Span'], how='all', inplace=False)

# Compare the lenght of the original df with this recently created one. Zero means no record that met the criteria was found
wos_duplicates.shape[0] - wos.shape[0]_____no_output_____# If there are no all-NaNs, then, check for duplicates
wos_duplicates.duplicated(subset=['Source', 'Year Published', 'Volume', 'Issue', 'Page Span'], keep=False).value_counts()_____no_output_____# A lot! Checking these 258...
wos_duplicates[wos_duplicates.duplicated(subset=['Source', 'Year Published', 'Volume', 'Issue', 'Page Span'], keep=False)]_____no_output_____# Checking of there are real titles on these 258 'duplicates'
wos_duplicates[wos_duplicates.duplicated(subset=['Source', 'Year Published', 'Volume', 'Issue', 'Page Span'], keep=False)]['Title'].value_counts()_____no_output_____

Some indeed seems to be duplicates... with more than one occurrence of the title. But a lot seems to be metadata issues. Better investigate._____no_output_____

# Checking how many of these have no repeated titles, meaning, could be metadata issues alone
wos_duplicates_title_counts = wos_duplicates[wos_duplicates.duplicated(subset=['Source', 'Year Published', 'Volume', 'Issue', 'Page Span'], keep=False)]['Title'].value_counts()

wos_duplicates_single_titles = wos_duplicates_title_counts[wos_duplicates_title_counts.where(wos_duplicates_title_counts < 2).notna()]

wos_duplicates_single_titles_____no_output_____# How many of these 'duplicates' have repeated titles?
wos_duplicates.duplicated(subset=['Source', 'Year Published', 'Volume', 'Issue', 'Page Span'], keep=False).value_counts()[1] - wos_duplicates_single_titles.shape[0]_____no_output_____# Getting info on one of the duplicates with non-repeated titles
wos_duplicates[wos_duplicates['Title'] == 'New nomenclatural and taxonomic acts, and Comments (2015).'][['Title', 'Source', 'Year Published', 'Volume', 'Issue', 'Page Span']]_____no_output_____# Checking if this duplicate with non-repeated titles have instead some metadata problems
wos_duplicates[wos_duplicates['Source'] == 'Snudebiller'][wos_duplicates['Year Published'] == '2015'][wos_duplicates['Volume'] == '16'][wos_duplicates['Page Span'] == '1-8']_____no_output_____

Though call! They have the same author and metadata, but not the same title._____no_output_____

# Checking for DOIs...
wos_duplicates[wos_duplicates['Source'] == 'Snudebiller'][wos_duplicates['Year Published'] == '2015'][wos_duplicates['Volume'] == '16'][wos_duplicates['Page Span'] == '1-8']['Digital Object Identifier (DOI)']_____no_output_____

No DOIs, and, this situation where everything else but title is the same. I'm making a tough call here, but I'll decide to exclude these potential duplicates too._____no_output_____###### 'year' NaNs

This is a more straight forward check... simply count NaNs on a specific column, 'Year Published'._____no_output_____

# Counting NaNs on 'Year Published'
wos['Year Published'].isna().value_counts()_____no_output_____

Wow! Seems that there is not a single row without the year of publication. That's quit amazing.

![](https://i.imgur.com/uM1TOIt.png)_____no_output_____###### journals' names

I just want to have a quick look into the (possible) mess that the journals' names in this source might be. I'm not specting that much of mess, to be honest.
_____no_output_____

# Group the wos dataframe by group, getting only 'Year Published' as return
wos_journals = wos.groupby('Source').count()['Year Published'].sort_values(ascending=False)

# Check the number of journals
wos_journals.shape[0]_____no_output_____

This is a LOT OF JOURNALS! I'm surprised. So, decided to check a bit further by:

1. Check the ones with one single occurrence, to see if there are any non-journals included in the dataset
2. Check if there are typos involving one of the most, if not the most popular journal: Zootaxa_____no_output_____

# Take a look at the journals with less than 2 occurrences as we might see some non-journals here
list(wos_journals[~wos_journals.where(wos_journals < 2).isna()].sort_values().index)_____no_output_____# Checking for possible typos involving Zootaxa's name
wos_journals[['Zoot'.lower() in s.lower() for s in wos_journals.index]]_____no_output_____

We can see that there are a LOT of different journals in this dataset, but not all of them are actual journals, there are mistakes, random enough to be extremely hard -- or impossible -- to come up with an automate way of cleaning at this point.

On the bright side, there aren't many typos with the most common journal on the dataset, which is a good indication that we are not missing a lot of data due to this issue._____no_output_____Summary of transformations needed in this dataset (Web of Science/Zoological Records):

1. Remove duplicates based on 'Source', 'Year Published', 'Volume', 'Issue', 'Page Span' columns
2. Select only 'Source' and 'Year Published' columns
3. Rename these two columns to 'Journal' and 'Year'
4. Add a column named 'Source', with all rows = 'WoS/Zoological Records'_____no_output_____##### TreatmentBank (Plazi)_____no_output_____

# First, check the available columns again
tb.columns_____no_output_____

###### Duplicates

There aren't many things that we can easily compare other than adopting the second strategy applied for WoS dataset. Thus:

1. Based on the columns: 'journal', 'year', 'volume', 'issue', 'start_page'

It's worth to mention that I'm deliberately avoiding using 'title' for this because I strongly believe that the more complex, variable, a field is, the more likely to have typos and other variance. As such, 'title' isn't a good fit to check for duplicates, for instance._____no_output_____

# Check for duplicates using the defined strategy

# First, remove possible rowns with NaNs in all of these fields:
tb_duplicates = tb.dropna(subset=['journal', 'year', 'volume', 'issue', 'start_page'], how='all', inplace=False)

# Checks if there were any NaNs in all of these columns
tb_duplicates.shape[0] - tb.shape[0]_____no_output_____# Then, applied strategy
tb_duplicates.duplicated(subset=['journal', 'year', 'volume', 'issue', 'start_page'], keep=False).value_counts()_____no_output_____# Take a carefull look into some of these so-called duplicates
tb_duplicates[tb_duplicates.duplicated(subset=['journal', 'year', 'volume', 'issue', 'start_page'], keep=False)]_____no_output_____# Looking the first record on that list, to see if we can find its duplicate
tb_duplicates[tb_duplicates['journal'] == 'Zootaxa'][tb_duplicates['volume'] == '3693'][tb_duplicates['issue'] == '1'][tb_duplicates['start_page'] == '0']_____no_output_____

Ok, this is clearly NOT a duplication, but a metadata issue. Let's check another one._____no_output_____

# Looking the first record on that list, to see if we can find its duplicate
tb_duplicates[tb_duplicates['journal'] == 'Journal of Natural History'][tb_duplicates['volume'] == '50'][tb_duplicates['issue'] == '29'][tb_duplicates['start_page'] == '1']_____no_output_____

Again, very likely another metadata issue, but two different entities. I think we should keep the so-called 'duplicates' for this dataset._____no_output_____###### 'year' NaNs

This is a more straight forward check... simply count NaNs on a specific column, 'year'._____no_output_____

# Check the amount of NaN on column 'year'
tb['year'].isna().value_counts()_____no_output_____

Again! Not a problematic field here!_____no_output_____###### journals' names

I just want to have a quick look into the (possible) mess that the journals' names in this source might be. I think this should be the most accurated source._____no_output_____

# Group the wos dataframe by group, getting only 'Year Published' as return
tb_journals = tb.groupby('journal').count()['year'].sort_values(ascending=False)_____no_output_____# Check the number of journals
tb_journals.shape[0]_____no_output_____

BIG difference here. From over 2500 to 196. TreatmentBank seems to be not only a much smaller database but less diverse too._____no_output_____

# Take a look at the journals with less than 2 occurrences as we might see some non-journals here
list(tb_journals[~tb_journals.where(tb_journals < 2).isna()].sort_values().index)_____no_output_____

There are fewer entries that are not real journals (for example, 'Handbook of the Birds of the Wold Special volume: new species and global index') than on wos dataset. It also doesn't seem to be a big problem in this dataset.
_____no_output_____

# Checking for possible typos involving Zootaxa's name
tb_journals[['Zoot'.lower() in s.lower() for s in tb_journals.index]]_____no_output_____# Definitely non-significant, but I decided to check 'ZooKeys' too
tb_journals[['Zook'.lower() in s.lower() for s in tb_journals.index]]_____no_output_____# And European Journal of Taxonomy
tb_journals[['Taxon'.lower() in s.lower() for s in tb_journals.index]]_____no_output_____

It's clear to me that journals' names is not a problem in this dataset._____no_output_____Summary of transformations needed in this dataset (TreatmentBank):

1. Select only 'journal' and 'year' columns
2. Rename these two columns to 'Journal' and 'Year', with capital initial letters
3. Add a column named 'Source', with all rows = 'TreatmentBank'_____no_output_____##### Zoobank
_____no_output_____

# First, let's check columns' names
zb.columns_____no_output_____

###### Duplicates

This dataset is very similar, in terms of structure, than the previous one. There aren't many things that we can easily compare other than adopting the second strategy applied for WoS dataset. Thus:

1. Based on the columns: 'journal', 'bibref_details'

The column 'bibref_details' carries the info for colume, issue, start and end_pages combined. Thus, it's a good place to check for duplicates._____no_output_____

# Check for duplicates using the defined strategy

# First, remove possible rowns with NaNs in all of these fields:
#zb_duplicates = zb[~zb[['journal', 'bibref_details']].isna()]
zb_duplicates = zb.dropna(subset=['journal', 'bibref_details'], how='all', inplace=False)

_____no_output_____# Checking if there were any NaNs in both columns
zb_duplicates.shape[0] - zb.shape[0]_____no_output_____# Then, applied strategy
zb_duplicates.duplicated(subset=['journal', 'bibref_details'], keep=False).value_counts()_____no_output_____

Wow. That's a LOT of potential duplicates. Better check this more carefully.

![](https://i.imgur.com/dG0WSvp.png)_____no_output_____

# Take a carefull look into some of these so-called duplicates
zb_duplicates[zb_duplicates.duplicated(subset=['journal', 'bibref_details'], keep=False)]_____no_output_____

I can already see a real duplication on the IDs 48884 and 48885. As I have the 'from_year' column, I actually went back to [Zoobank#2019](http://zoobank.org/Search?search_term=2019) and found these two guys duplicated...

![](https://i.imgur.com/HHyM254.png)

So we have duplicates. Let's check one more._____no_output_____

# Randomnly checking for another case of duplication...
zb_duplicates[zb_duplicates['bibref_details'] == '3768(1): 88–94']_____no_output_____

This one is even more interesting as their 'from_year' are different! Let me see if I can find them on Zoobank...

[Zoobank#2010](http://zoobank.org/Search?search_term=2010)

![](https://i.imgur.com/qEV24g4.png)

[Zoobank#2014](http://zoobank.org/Search?search_term=2014)

![](https://i.imgur.com/EFY6Cxn.png)

Yep, duplicate! And a weird one! Because how come a paper that was only published in 2014 appeared registred in 2010? **Better remove them all!**_____no_output_____###### 'year' NaNs

This is a more straight forward check... simply count NaNs on a specific column, 'year'._____no_output_____

# Check the amount of NaN on column 'year'
zb['year'].isna().value_counts()_____no_output_____

For a dataset built on a massive web scraping effort on a not reliably structured website, that just made my day.

![](https://media.giphy.com/media/OK27wINdQS5YQ/giphy-downsized.gif)
_____no_output_____###### journals' names

I just want to have a quick look into the (possible) mess that the journals' names in this source might be. I think this must be the least accurate as the data wasn't structured enough for a safe extraction of the journals' names._____no_output_____

# Group the wos dataframe by group, getting only 'Year Published' as return
zb_journals = zb.groupby('journal').count()['year'].sort_values(ascending=False)_____no_output_____# Check the number of journals
zb_journals.shape[0]_____no_output_____

Wayy too many. Probably there are a LOT of errors. As this was a result of web scraping on a badly structured HTML, I probably sill have highly ranked mistakes too._____no_output_____

# Take a look at the journals with less than 2 occurrences as we might see some non-journals here
list(zb_journals[~zb_journals.where(zb_journals < 2).isna()].sort_values().index)_____no_output_____

As expected, most of these are not journals, which is a different situation found on the two precedent datasets._____no_output_____

# Take a look at the top 100 journals in this dataset
zb_journals.head(40)_____no_output_____

Well, the fact that the journal 'blank' got 347 hits is not a good sign. As expected, many issues here too.

![](https://media.giphy.com/media/3oEjI80DSa1grNPTDq/giphy.gif)_____no_output_____

# Checking for possible typos involving Zootaxa's name. Also expecting many...
zb_journals[['Zoot'.lower() in s.lower() for s in zb_journals.index]]_____no_output_____# Checking for possible typos involving ZooKeys's name.
zb_journals[['Zook'.lower() in s.lower() for s in zb_journals.index]]_____no_output_____# Checking for possible typos involving European Journal of Taxonomy's name.
zb_journals[['Taxon'.lower() in s.lower() for s in zb_journals.index]]_____no_output_____

As expected, we do have more problems with typos in this database - but I guess not enough to be a concern if we are looking at the top 10, 20 journals only. Plus, after removing duplicates, we might get rid of a lot of these nonsense. So, moving on!_____no_output_____Summary of transformations needed in this dataset (Zoobank):

1. Remove duplicates based on 'journal' and 'bibref_details' columns
2. Rename some high-ranked journals
3. Remove rows that contains higher ranked non-jounals, such as "nov", "n", etc.
4. Remove years out of the 2010-2020 range
5. Select only 'journal' and 'year' columns
6. Rename these two columns to 'Journal' and 'Year', with capital initial letters
7. Add a column named 'Source', with all rows = 'Zoobank'_____no_output_____#### Transform

_____no_output_____

wos_final = transform.wos(wos)

wos_final.head()_____no_output_____test_tf.wos(wos_final)SUCCESS! You correctly selected the columns!
SUCCESS! You correctly filled the collumn 'Source'!
tb_final = transform.tb(tb)

tb_final.head()_____no_output_____test_tf.tb(tb_final)SUCCESS! You correctly selected the columns!
SUCCESS! You correctly filled the collumn 'Source'!
zb_final = transform.zb(zb)

zb_final.head()_____no_output_____test_tf.zb(zb_final)SUCCESS! You correctly fixed all pre-determined values!
SUCCESS! You correctly removed '' and ','!
SUCCESS! You correctly removed all years out of the 2010-2020 range!
SUCCESS! You correctly selected the columns!
SUCCESS! You correctly filled the collumn 'Source'!
# Combine all transformed dataframes into one, grouped and sorted multindex dataframe
df = transform.combine(wos_final, tb_final, zb_final)

df.head()_____no_output_____test_tf.final(df)SUCCESS! You successfully created the Sources-specific Sum columns!
SUCCESS! The ('All', 'Sum') function was correctly set!
SUCCESS! The final dataframe seems to be correctly sorted!

#### Is it open?_____no_output_____

# Adding a new column to the DF based on open-access policy of each journal
# I'm comming this out because it involves an API call and I've removed my API key
#df = transform.add_oa(df)

# Alternativelly, I'll add the following:
df = pd.read_pickle('data/df_final.p')_____no_output_____# Checking if everything went OK
df[('All', 'OA')]_____no_output_____

### (L)oad

Probably the most useless function I wrote for this project, but to follow a pattern, as I modularized most of stuff here... here it is._____no_output_____

# I'll comment this out because it clearly doesn't need to be re-run
#load.to_pickle(df, 'data/df_final.p')_____no_output_____

## Analysis

As said previously, analyses focused on descriptive stats (counts) and data visualization only. One might say that's a fairly simplistic methodology, but it's what is needed to answer these important questions on my field (zoological systematics).

In order to build charts easier, I create several chart-specific dataframes, wrangling the data to the point that they are ready to be ploted. _____no_output_____### How many top journals are open-access?
_____no_output_____

# Extract top wos journals sorted by its sum and get only 'Sum' and ('All', 'OA') columns
top_wos_oa = df[['WoS/Zoological Records', 'All']].sort_values(by=('WoS/Zoological Records', 'Sum'), ascending=False)[[('WoS/Zoological Records', 'Sum'),('All', 'OA')]].head(100)

# Get count, the % and change the name of the series
top_wos_oa_count = top_wos_oa[('All', 'OA')].value_counts(dropna=False)

top_wos_oa_count_rel = (top_wos_oa_count / top_wos_oa_count.sum() * 100).round(2)

top_wos_oa_count_rel.name = 'WoS/Zoological Records'_____no_output_____# Extract top tb journals sorted by its sum and get only 'Sum' and ('All', 'OA') columns
top_tb_oa = df[['TreatmentBank', 'All']].sort_values(by=('TreatmentBank', 'Sum'), ascending=False)[[('TreatmentBank', 'Sum'),('All', 'OA')]].head(100)

# Get count, the % and change the name of the series
top_tb_oa_count = top_tb_oa[('All', 'OA')].value_counts(dropna=False)

top_tb_oa_count_rel = (top_tb_oa_count / top_tb_oa_count.sum() * 100).round(2)

top_tb_oa_count_rel.name = 'TreatmentBank'_____no_output_____# Extract top zb journals sorted by its sum and get only 'Sum' and ('All', 'OA') columns
top_zb_oa = df[['Zoobank', 'All']].sort_values(by=('Zoobank', 'Sum'), ascending=False)[[('Zoobank', 'Sum'),('All', 'OA')]].head(100)

# Get count, the % and change the name of the series
top_zb_oa_count = top_zb_oa[('All', 'OA')].value_counts(dropna=False)

top_zb_oa_count_rel = (top_zb_oa_count / top_zb_oa_count.sum() * 100).round(2)

top_zb_oa_count_rel.name = 'Zoobank'_____no_output_____# Get the general count, the % and change the name of the series
overall_oa_count = df[('All', 'OA')].value_counts(dropna=False)

overall_oa_count_rel = (overall_oa_count / overall_oa_count.sum() * 100).round(2)

overall_oa_count_rel.name = 'All'_____no_output_____#au = overall_oa_count.plot.bar(xlabel='Open-access policies', ylabel='Number of Articles', rot=1, color=['black', 'blue', 'red', 'gray', 'yellow'], fontsize=14)_____no_output_____# BAR PLOT on Open-Access Distribution on complet dataset
fig = plt.figure()

ax = fig.add_axes([0, 0, 1, 1])

# Color system
colors = ['#dcedc8', '#795548', '#ffee58', '#78909c', '#b0bec5']

# Replace np.nan to str
data = overall_oa_count
data.index = data.index.map(str)

# Create all bars in the data
for i in range(len(data)):
    ax.bar(data.index[i], data[i], width=.5, color=colors[i])

plt.xlabel('Open-Access Policies', fontsize='x-large')
plt.ylabel('Number of Articles', fontsize='x-large')
plt.title('Open-Access in the Complete Dataset', fontsize=18)
_____no_output_____# Build the OA counts dataframe to ease the chart process
oa_counts = pd.DataFrame([top_wos_oa_count_rel, top_tb_oa_count_rel, top_zb_oa_count_rel, overall_oa_count_rel]).transpose()

oa_counts_____no_output_____data = oa_counts
data.index = data.index.map(str)

for each in data:
    print(data[each].name)WoS/Zoological Records
TreatmentBank
Zoobank
All
# BAR PLOT on Open-Access Distribution on complet dataset
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15,15), constrained_layout=True)

# Replace np.nan to str
data = oa_counts
data.index = data.index.map(str)

# Color system
colors = ['#dcedc8', '#795548', '#ffee58', '#78909c', '#b0bec5']

# Create all three pies
ax2.pie(data['WoS/Zoological Records'], colors=colors, autopct='%.1f%%')
ax1.pie(data['TreatmentBank'], colors=colors, autopct='%.1f%%')
ax3.pie(data['Zoobank'], colors=colors, autopct='%.1f%%')

ax2.legend(data.index, loc=8, ncol=5, fontsize='large', borderaxespad=-2)

ax1.set_title('WoS/Zoological Records', fontsize=18)
ax2.set_title('TreatmentBank', fontsize=18)
ax3.set_title('Zoobank', fontsize=18)

fig.suptitle("Observed journals' open-access policies in all datasets", y=0.68, fontsize=20)

fig.subplots_adjust()

plt.show()_____no_output_____

### Who are the biggest contributors?
_____no_output_____

# Prepare a wos-isolated df to ease manipulation
# Make a copy of entire df
top_wos = df

# Create a '%' column based on the records present on this already filtered (top 100) df. This will have the relative contribution of a journal within this dataset.
top_wos[('WoS/Zoological Records', '%')] = (top_wos['WoS/Zoological Records']['Sum'] / top_wos['WoS/Zoological Records']['Sum'].sum() * 100).round(2)

# Get only the data related to wos
top_wos = top_wos['WoS/Zoological Records']

# Sort the resulting filtered down dataset based on sum
top_wos = top_wos.sort_values(by='Sum', ascending=False)

top_wos_____no_output_____# This will calculates the real relative contribution of journals based on the original, non-filtered wos dataframe
# The steps to group, sum and sort are the same from transform.combine()
wos_contrib = wos_final.groupby(['Journal', 'Source', 'Year'])['Journal'].count().unstack(['Source','Year'])

wos_contrib = wos_contrib.reindex(sorted(wos_contrib.columns), axis=1)

wos_contrib['Sum'] = wos_contrib.iloc[:, 0:12].sum(axis=1)

wos_contrib.sort_values(by='Sum', ascending=False, inplace=True)

# Calculate the relative importance on the overall dataset, not the filtered one
wos_contrib['%'] = (wos_contrib['Sum'] / wos_contrib['Sum'].sum() * 100).round(2)

# Map the findings to the filtered dataframe
top_wos['% Real'] = top_wos.index.map(wos_contrib['%'].to_dict())

top_wos_____no_output_____# Get the 'Sum' column
top_wos.head(25)[['Sum', '% Real']]_____no_output_____# Prepare a wos-isolated df to ease manipulation
# Make a copy of entire df
top_tb = df

# Create a '%' column based on the records present on this already filtered (top 100) df. This will have the relative contribution of a journal within this dataset.
top_tb[('TreatmentBank', '%')] = (top_tb['TreatmentBank']['Sum'] / top_tb['TreatmentBank']['Sum'].sum() * 100).round(2)

# Get only the data related to tb
top_tb = top_tb['TreatmentBank']

# Sort the resulting filtered down dataset based on sum
top_tb = top_tb.sort_values(by='Sum', ascending=False)

top_tb_____no_output_____# This will calculates the real relative contribution of journals based on the original, non-filtered wos dataframe
# The steps to group, sum and sort are the same from transform.combine()
tb_contrib = tb_final.groupby(['Journal', 'Source', 'Year'])['Journal'].count().unstack(['Source','Year'])

tb_contrib = tb_contrib.reindex(sorted(tb_contrib.columns), axis=1)

tb_contrib['Sum'] = tb_contrib.iloc[:, 0:12].sum(axis=1)

tb_contrib.sort_values(by='Sum', ascending=False, inplace=True)

# Calculate the relative importance on the overall dataset, not the filtered one
tb_contrib['%'] = (tb_contrib['Sum'] / tb_contrib['Sum'].sum() * 100).round(2)

# Map the findings to the filtered dataframe
top_tb['% Real'] = top_tb.index.map(tb_contrib['%'].to_dict())

top_tb_____no_output_____# Get the 'Sum' column
top_tb.head(25)[['Sum', '% Real']]_____no_output_____# Prepare a wos-isolated df to ease manipulation
# Make a copy of entire df
top_zb = df

# Create a '%' column based on the records present on this already filtered (top 100) df. This will have the relative contribution of a journal within this dataset.
top_zb[('Zoobank', '%')] = (top_zb['Zoobank']['Sum'] / top_zb['Zoobank']['Sum'].sum() * 100).round(2)

# Get only the data related to tb
top_zb = top_zb['Zoobank']

# Sort the resulting filtered down dataset based on sum
top_zb = top_zb.sort_values(by='Sum', ascending=False)

top_zb_____no_output_____# This will calculates the real relative contribution of journals based on the original, non-filtered wos dataframe
# The steps to group, sum and sort are the same from transform.combine()
zb_contrib = zb_final.groupby(['Journal', 'Source', 'Year'])['Journal'].count().unstack(['Source','Year'])

zb_contrib = zb_contrib.reindex(sorted(zb_contrib.columns), axis=1)

zb_contrib['Sum'] = zb_contrib.iloc[:, 0:12].sum(axis=1)

zb_contrib.sort_values(by='Sum', ascending=False, inplace=True)

# Calculate the relative importance on the overall dataset, not the filtered one
zb_contrib['%'] = (zb_contrib['Sum'] / zb_contrib['Sum'].sum() * 100).round(2)

# Map the findings to the filtered dataframe
top_zb['% Real'] = top_zb.index.map(zb_contrib['%'].to_dict())

top_zb_____no_output_____# Get the 'Sum' column
top_zb.head(25)[['Sum', '% Real']]_____no_output_____

### How do the sources compare to each other?_____no_output_____#### Total number of records_____no_output_____

# Creates a dataframe with both total and filtered total records counts
total_count = pd.DataFrame([
    ['WoS/Zoological Records', wos_final.shape[0], top_wos['Sum'].sum()],
    ['TreatmentBank', tb_final.shape[0], top_tb['Sum'].sum()],
    ['Zoobank', zb_final.shape[0], top_zb['Sum'].sum()]],
    columns=['Source', 'Total Records', 'Filtered Records'],
    index=[1, 2, 3]).sort_values(by='Total Records', ascending=False)

# TODO: Change 'Filtered Records' type to integer

total_count_____no_output_____# Prepare the diff data, the part that will be on top of the stacked bar chart
wos_diff = total_count[total_count['Source'] == 'WoS/Zoological Records']['Total Records'] - total_count[total_count['Source'] == 'WoS/Zoological Records']['Filtered Records']

tb_diff = total_count[total_count['Source'] == 'TreatmentBank']['Total Records'] - total_count[total_count['Source'] == 'TreatmentBank']['Filtered Records']

zb_diff = total_count[total_count['Source'] == 'Zoobank']['Total Records'] - total_count[total_count['Source'] == 'Zoobank']['Filtered Records']

diff = (
    wos_diff.loc[1],
    tb_diff.loc[2],
    zb_diff.loc[3]
    )

# Prepare the filtered data, the bottom part
filtered = (
    int(total_count[total_count['Source'] == 'WoS/Zoological Records']['Filtered Records'].loc[1]),
     int(total_count[total_count['Source'] == 'TreatmentBank']['Filtered Records'][2]),
     int(total_count[total_count['Source'] == 'Zoobank']['Filtered Records'][3])
    )

ind = np.arange(3)
width = 0.35

plot1 = plt.bar(ind, filtered, width, color='#424242', label='Used')
plot2 = plt.bar(ind, diff, width, bottom=filtered, color='#2e7d32', label='Discarted')

plt.xticks(ind, ('WoS/ZR', 'TreatmentBank', 'Zoobank'), fontsize='large')
plt.ylabel('Number of Articles', fontsize='x-large')
plt.xlabel('Data Sources', fontsize='x-large')
plt.title('Used and Discarted Articles per Source', fontsize=18)
plt.legend()_____no_output_____total_count[total_count['Source'] == 'TreatmentBank']['Filtered Records']_____no_output_____

#### Comparison between the accumulative sums in all datasets_____no_output_____

# Gets the accumulative real contribution of top 25 and 100 journals in all sources
acc_contrib = [['WoS/Zoological Records', top_wos.head(25)['% Real'].sum().round(2), top_wos.head(100)['% Real'].sum().round(2)], ['TreatmentBank', top_tb.head(25)['% Real'].sum().round(2), top_tb.head(100)['% Real'].sum().round(2)], ['Zoobank', top_zb.head(25)['% Real'].sum().round(2), top_zb.head(100)['% Real'].sum().round(2)]]

acc_contrib_____no_output_____# Creates the dataframe with the data combined in the previous cell
acc_df = pd.DataFrame(acc_contrib, columns=['Source', 'Top25', 'Top100'])

# Adds another column, with the difference between Top100 contribution and Top25
acc_df['Diff'] = acc_df['Top100'] - acc_df['Top25']

acc_df_____no_output_____

#### Plot over time, per source_____no_output_____

# Extract only year columns from top_wos, and add name to the series
wos_yr = top_wos[[str(yr) for yr in range(2010, 2021)]].sum()

wos_yr.name = 'WoS/Zoological Records'

wos_yr_____no_output_____# Extract only year columns from top_tb, and add name to the series
tb_yr = top_tb[[str(yr) for yr in range(2010, 2021)]].sum()

tb_yr.name = 'TreatmentBank'

tb_yr_____no_output_____# Extract only year columns from top_zb, and add name to the series
zb_yr = top_zb[[str(yr) for yr in range(2010, 2021)]].sum()

zb_yr.name = 'Zoobank'

zb_yr_____no_output_____# Build the Source-per-Year dataframe
docs_per_year = pd.DataFrame([wos_yr, tb_yr, zb_yr])

docs_per_year_____no_output_____# Prepare data for plotting
data = docs_per_year.T
index = data.index

line1 = plt.plot(index, data['WoS/Zoological Records'], label='WoS/Zoological Records', color='#f9a825')
line2 = plt.plot(index, data['TreatmentBank'], label='TreatmentBank', color='#4caf50')
line3 = plt.plot(index, data['Zoobank'], label='Zoobank', color='blue')

plt.ylabel('Number of Articles', fontsize='x-large')
plt.xlabel('Years', fontsize='x-large')
plt.title('Number of Articles per Year', fontsize=18)

plt.legend()_____no_output_____

## Conclusion_____no_output_____All major points were addressed and discussed on the [project deliverable]((https://hackmd.io/@mguidoti/S1veVwNHv), but here are the highlights:

- The data on open-access policy is sparsed through two different providers, and fairly incomplete;
- There are less concern about open-access in this zoological taxonomic community than I would have hoped for;
- Zootaxa and ZooKeys are the two most prolific journals for new species descriptions publication, with Zootaxa taking, by far, the largest share;
- From #3 beyond the sources vary a lot;
- Web of Science/Zoological Records remain as the most complete dataset on zoological taxonomic research, regardless the efforts on promoting Zoobank;
- Zoobank data is the least curated, with more real duplicates and lack of data consistency, in addition to the somehow unstructured HTML;
- TreatmentBank is the most reliable source in terms of data consistency, but it's the smallest dataset, concentrated over 69% of its data from a single journal.

_____no_output_____## References
I have included it all on [README.md]() and on the [blog post](https://hackmd.io/@mguidoti/S1veVwNHv)._____no_output_____


# Notebook from imagej/imagej.py
Path: doc/10-Troubleshooting.ipynb

PyImageJ Tutorial
===

This notebook covers how to use ImageJ as a library from Python. A major advantage of this approach is the ability to combine ImageJ with other tools available from the Python software ecosystem, including NumPy, SciPy, scikit-image, CellProfiler, OpenCV, ITK and more.

This notebook assumes familiarity with the ImageJ API. Detailed tutorials in that regard can be found in the other notebooks._____no_output_____## 9 Troubleshooting

This notebook covers some basic troubleshooting with PyImageJ. Let's initalize ImageJ before exploring some troublehsooting options.
_____no_output_____

import imagej

# initialize ImageJ
ij = imagej.init()
print(f"ImageJ2 version: {ij.getVersion()}")log4j:WARN No appenders could be found for logger (org.bushe.swing.event.EventService).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

### 9.1 I can't pass my numpy image to an `Op`

`ij.py` is really good at converting numpy images into `RandomAccessibleInterval`s. However many `Op`s, like `addPoissonNoise`, take other forms of ImageJ images, like `IterableInterval`._____no_output_____

print(ij.op().help('filter.addPoissonNoise'))Available operations:
	(RealType out) =
	net.imagej.ops.filter.addPoissonNoise.AddPoissonNoiseRealType(
		RealType out,
		RealType in,
		long seed?)
	(IterableInterval out) =
	net.imagej.ops.filter.addPoissonNoise.AddPoissonNoiseMap(
		IterableInterval out,
		IterableInterval in)

We can't call this `Op` on a numpy array since it is a specialized type of `RandomAccessibleInterval`, which does not extend `IterableInterval`._____no_output_____

from skimage import io

# Create a numpy image using scikit
img = io.imread('https://imagej.net/images/clown.jpg')

ij.py.show(img)
print(type(ij.py.to_java(img)))_____no_output_____

We can fix this by using `transform.flatIterableView` on both the input and output, which will convert the `ReferenceGuardedRandomAccessibleInterval`s into `IterableInterval`s, allowing us to pass our numpy image into `addPoissonNoise`:_____no_output_____

import numpy as np

result = np.zeros(img.shape) # HINT: Uses float dtype, for more accurate noising.

imgIterable = ij.op().transform().flatIterableView(ij.py.to_java(img))
resIterable = ij.op().transform().flatIterableView(ij.py.to_java(result))

ij.op().filter().addPoissonNoise(resIterable, imgIterable)

ij.py.show(result)Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).

Uh oh. Something's off with the data values—see next section!_____no_output_____If you need to troubleshoot op workings, look for implementations that use only `IterableInterval`s or `RandomAccessibleInterval`s. To find the implementations use the `print(ij.op().help())` function.

For the multiply function the implementation we used is second to last (`net.imagej.ops.math.IIToRAIOutputII$Multiply`)_____no_output_____If you need to troubleshoot op workings, look for implementations that use only `IterableInterval`s or `RandomAccessibleInterval`s. To find the implementations use the `print(ij.op().help())` function.

For the multiply function the implementation we used is second to last (`net.imagej.ops.math.IIToRAIOutputII$Multiply`)_____no_output_____### 9.2 Clipping input data to the valid range for imshow with RGB data ([0..1] for floats of [0..255] for integers

This message is given either because (as described in the warning)

* The data consists of floats that are outside the range of [0..1]. This is pretty common and definitely is the case as to why the above data looks so bad.
* The data consists of ints that are outside the range [0..255].

Let's be certain which is the culprit by checking some values of `result`:_____no_output_____

# grab the RGB values in a line from [0][5] to [0][10] in our image
print(result[0][5:10])[[261. 179.  38.]
 [232. 185.  59.]
 [242. 136.  45.]
 [227. 173.  38.]
 [262. 158.  40.]]

Thus we not only have floats outside of [0..1] but also values outside of [0..255]; we are faulty of both points in the warning. We can fix this by first [clipping](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.clip.html) the entire array within the integer range, then cast to `uint8` so that the float range no longer applies:_____no_output_____

ij.py.show(img.astype(int))

result = np.clip(result, 0, 255)
ij.py.show(result.astype(np.uint8))_____no_output_____

Now our noisy image displays nicely alongside the original. Note that depending on your data this might not be the right path for you, but using `clip` and `astype` are great tools for rearranging your data within these bounds._____no_output_____If you need to troubleshoot op workings, look for implementations that use only `IterableInterval`s or `RandomAccessibleInterval`s. To find the implementations use the `print(ij.op().help())` function.

For the multiply function the implementation we used is second to last (`net.imagej.ops.math.IIToRAIOutputII$Multiply`)_____no_output_____

# print is required to render new lines
print(ij.op().help('multiply'))Available operations:
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImageP$MultiplyByte(
		ArrayImg arg,
		byte value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImageP$MultiplyDouble(
		ArrayImg arg,
		double value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImageP$MultiplyFloat(
		ArrayImg arg,
		float value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImageP$MultiplyInt(
		ArrayImg arg,
		int value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImageP$MultiplyLong(
		ArrayImg arg,
		long value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImageP$MultiplyShort(
		ArrayImg arg,
		short value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImageP$MultiplyUnsignedByte(
		ArrayImg arg,
		byte value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImageP$MultiplyUnsignedInt(
		ArrayImg arg,
		int value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImageP$MultiplyUnsignedLong(
		ArrayImg arg,
		long value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImageP$MultiplyUnsignedShort(
		ArrayImg arg,
		short value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImage$MultiplyByte(
		ArrayImg arg,
		byte value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImage$MultiplyDouble(
		ArrayImg arg,
		double value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImage$MultiplyFloat(
		ArrayImg arg,
		float value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImage$MultiplyInt(
		ArrayImg arg,
		int value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImage$MultiplyLong(
		ArrayImg arg,
		long value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImage$MultiplyShort(
		ArrayImg arg,
		short value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImage$MultiplyUnsignedByte(
		ArrayImg arg,
		byte value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImage$MultiplyUnsignedInt(
		ArrayImg arg,
		int value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImage$MultiplyUnsignedLong(
		ArrayImg arg,
		long value)
	(ArrayImg arg) =
	net.imagej.ops.math.ConstantToArrayImage$MultiplyUnsignedShort(
		ArrayImg arg,
		short value)
	(IterableInterval out?) =
	net.imagej.ops.math.IIToIIOutputII$Multiply(
		IterableInterval out?,
		IterableInterval in1,
		IterableInterval in2)
	(NumericType out?) =
	net.imagej.ops.math.NumericTypeBinaryMath$Multiply(
		NumericType out?,
		NumericType in1,
		NumericType in2)
	(int result) =
	net.imagej.ops.math.PrimitiveMath$IntegerMultiply(
		int a,
		int b)
	(long result) =
	net.imagej.ops.math.PrimitiveMath$LongMultiply(
		long a,
		long b)
	(float result) =
	net.imagej.ops.math.PrimitiveMath$FloatMultiply(
		float a,
		float b)
	(double result) =
	net.imagej.ops.math.PrimitiveMath$DoubleMultiply(
		double a,
		double b)
	(RealType out) =
	net.imagej.ops.math.BinaryRealTypeMath$Multiply(
		RealType out,
		RealType in1,
		RealType in2)
	(IterableInterval out?) =
	net.imagej.ops.math.ConstantToIIOutputII$Multiply(
		IterableInterval out?,
		IterableInterval in,
		NumericType value)
	(PlanarImg arg) =
	net.imagej.ops.math.ConstantToPlanarImage$MultiplyByte(
		PlanarImg arg,
		byte value)
	(PlanarImg arg) =
	net.imagej.ops.math.ConstantToPlanarImage$MultiplyDouble(
		PlanarImg arg,
		double value)
	(PlanarImg arg) =
	net.imagej.ops.math.ConstantToPlanarImage$MultiplyFloat(
		PlanarImg arg,
		float value)
	(PlanarImg arg) =
	net.imagej.ops.math.ConstantToPlanarImage$MultiplyInt(
		PlanarImg arg,
		int value)
	(PlanarImg arg) =
	net.imagej.ops.math.ConstantToPlanarImage$MultiplyLong(
		PlanarImg arg,
		long value)
	(PlanarImg arg) =
	net.imagej.ops.math.ConstantToPlanarImage$MultiplyShort(
		PlanarImg arg,
		short value)
	(PlanarImg arg) =
	net.imagej.ops.math.ConstantToPlanarImage$MultiplyUnsignedByte(
		PlanarImg arg,
		byte value)
	(PlanarImg arg) =
	net.imagej.ops.math.ConstantToPlanarImage$MultiplyUnsignedInt(
		PlanarImg arg,
		int value)
	(PlanarImg arg) =
	net.imagej.ops.math.ConstantToPlanarImage$MultiplyUnsignedLong(
		PlanarImg arg,
		long value)
	(PlanarImg arg) =
	net.imagej.ops.math.ConstantToPlanarImage$MultiplyUnsignedShort(
		PlanarImg arg,
		short value)
	(IterableInterval out?) =
	net.imagej.ops.math.IIToRAIOutputII$Multiply(
		IterableInterval out?,
		IterableInterval in1,
		RandomAccessibleInterval in2)
	(RandomAccessibleInterval out) =
	net.imagej.ops.math.ConstantToIIOutputRAI$Multiply(
		RandomAccessibleInterval out,
		IterableInterval in,
		NumericType value)


# Notebook from ilam56/wg-evolution
Path: implementations/notebooks/reviews_declined_github.ipynb

## Reviews_Declined
This is the reference implementation for [Reviews Declined](https://github.com/chaoss/wg-evolution/blob/master/metrics/Reviews_Declined.md),
a metric specified by the
[Evolution Working Group](https://github.com/chaoss/wg-evolution) of the
[CHAOSS project](https://chaoss.community).
This implementation is specific to Git repositories.

Have a look at [README.md](../README.md) to find out how to run this notebook (and others in this directory) as well as to get a better understanding of the purpose of the implementations.

The implementation is described in two parts (see below):

* Class for computing Reviews Declined
* An explanatory analysis of the class' functionality

Some more auxiliary information in this notebook:

* Examples of the use of the implementation_____no_output_____As discussed in the [README](../README.md) file, the scripts required to analyze the data fetched by Perceval are located in the `scripts` package. Due to python's import system, to import modules from a package which is not in the current directory, we have to either add the package to `PYTHONPATH` or simply append a `../..` to `sys.path`, so that `scripts` can be successfully imported. _____no_output_____

from datetime import datetime
import matplotlib.pyplot as plt
%matplotlib inline

import sys
sys.path.append('../..')

from implementations.scripts.pullrequest_github import PullRequestGitHub
from implementations.scripts.utils import read_json_file_____no_output_____class ReviewsDeclinedGitHub(PullRequestGitHub):
    """
    Class for the Reviews Declined metric (non-pandas)
    """

    def compute(self):
        """
        Compute the total number of reviews which were declined, from the
        Perceval data. Declined reviews are in the "closed" state but
        are not merged.

        :returns: The total number of reviews declined
        """

        pull_ids = {item['hash'] for item in self.items
                    if item['merged'] is False and item['current_status'] == 'closed'}

        return len(pull_ids)

    def __str__(self):
        return "Reviews Declined"_____no_output_____

## Performing the Analysis
Let's compute the the number of reviews declined in a given period_____no_output_____### Counting the total number of reviews declined
We first read the JSON file containing Perceval data using the `read_json_file` utility function. _____no_output_____

items = read_json_file('../pull_requests.json')_____no_output_____

We'll use the `compute` method to count the total number of declined reviews. First, we will do it without passing any since and until dates.
Next, we can pass in the start and end dates as a tuple. The format would be `%Y-%m-%d`.
Let's also find the number of declined reviews created after a certain date. Here, we would pass a valid `datetime` object for the `since` part of `date_range`, leaving the `until` part as `None`._____no_output_____

reviews_declined = ReviewsDeclinedGitHub(items)
print("The total number of reviews declined "
      "in the data is {}.".format(reviews_declined.compute()))

date_since = datetime.strptime("2018-01-01", "%Y-%m-%d")
date_until = datetime.strptime("2018-07-01", "%Y-%m-%d")

reviews_declined_dated = ReviewsDeclinedGitHub(items,
                        date_range=(date_since, date_until))

print("The total number of reviews created between "
      "2018-01-01 and 2018-07-01 that were declined is {}.".format(reviews_declined_dated.compute()))

reviews_declined_since = ReviewsDeclinedGitHub(items,
                        date_range=(date_since, None))

print("The total number of reviews created after "
      "2018-01-01 that were declined is {}.".format(reviews_declined_since.compute()))The total number of reviews declined in the data is 18.
The total number of reviews created between 2018-01-01 and 2018-07-01 that were declined is 5.
The total number of reviews created after 2018-01-01 that were declined is 12.