# Notebook from csavur/biosignalsnotebooks
Path: biosignalsnotebooks_notebooks/unpublished_notebooks/Pre-Process/temporal_statistical_parameters.ipynb

<link rel="stylesheet" href="../../styles/theme_style.css">

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">

         Parameter Extraction - Temporal/Statistical Parameters

_____no_output_____

        Difficulty Level:

                    Tags
                    extract|statistics|temporal signals

_____no_output_____When we are working with a signal recording, it is important to perform an preliminar analysis to understand how it behaves and obtain its main characteristics. One of the most prevalent ways to do this is by extracting descriptive parameters. There are different ways to approach this, and this notebook will cover some of the most common statistical and temporal parameters that can be used to analyze time series.

The use of these descriptive charateristics will help us decide the most effective strategy to follow during data preparation, model selection and model tuning depending on the use we will give to the data.

**List of statistical parameters:**
+ Maximum, minimum and range
+ Mean
+ Mode
+ Median and quantiles
+ Variance and standard deviation
+ Skewness and kurtosis

**List of temporal parameters:**
+ Autocorrelation
+ Stationarity
+ Seasonal decomposition_____no_output_____
1 - Importation of the needed packages
_____no_output_____

# biosignalsnotebooks own package for loading and plotting the acquired data
import biosignalsnotebooks as bsnb

# Scientific packages
import numpy
import math
import scipy.integrate as integrate
import scipy.stats as stats
import statsmodels.tsa.stattools as stattools
import statsmodels.tsa.seasonal as seasonal
import pandas as pd_____no_output_____# Base packages used in OpenSignals Tools Notebooks for plotting data
from bokeh.plotting import figure, output_file, show, curdoc
from bokeh.io import output_notebook
from bokeh.layouts import column
from bokeh.models import ColumnDataSource, Plot, LinearAxis, BoxAnnotation, Arrow, VeeHead, LinearAxis, Range1d
output_notebook(hide_banner=True)_____no_output_____

2 - Load of sample signal data
_____no_output_____

# Load of data
data, header = bsnb.load("../../signal_samples/ecg_20_sec_100_Hz.h5", get_header=True)
channel = list(data.keys())[0]
print(header){'channels': array([1]), 'comments': '', 'date': b'2018-9-28', 'device': 'biosignalsplux', 'device connection': b'BTH00:07:80:D8:A7:F9', 'device name': b'00:07:80:D8:A7:F9', 'digital IO': array([0, 1]), 'firmware version': 773, 'resolution': array([16]), 'sampling rate': 100, 'sync interval': 2, 'time': b'14:36:39.886', 'sensor': [b'ECG'], 'column labels': {1: 'channel_1'}}

3 - Storage of sampling frequency and acquired data inside variables

Since in this case we know that the original signal is a ECG recording, we can also convert its units to mV following the method explained in the ECG Sensor - Unit Conversion  notebook._____no_output_____

# Sampling frequency and acquired data
fs = header["sampling rate"]

# Signal Samples
signal_raw = data[channel]
time = numpy.linspace(0, len(signal_raw) / fs, len(signal_raw))

# Let's convert the signal's units, since we know it is a ECG signal
vcc = 3000 # mV
gain = 1000
ch = "CH1" # Channel
sr = header['sampling rate'] # Sampling rate
resolution = header['resolution'] # Resolution (number of available bits)
signal = (((numpy.array(signal_raw) / 2**resolution) - 0.5) * vcc) / gain

# Let's plot our raw signal
p = figure(plot_width=1000, plot_height=200)
p.ygrid.grid_line_alpha=0.5

# add a circle renderer with x and y coordinates, size, color, and alpha
p.line(time, signal)

show(p) # show the results_____no_output_____

4 - Extraction of statistical parameters in a time signal

A fundamental task in many statistical analysis is to characterize the location (a central value that best describes the data) and variability (or spread) of a dataset, in order to describe and understand it better. These descriptors are independent of the domain of the signal, and can be used to describe any type of information or dataset. Due to their simplicity and the fact that they are easy to calculate, they are widely used during statistical analyses.

4.1 - Minimum, maximum and range values

The minimum and maximum values are, respectively, the lowest and highest power values across the signal duration. There may be more than one occurrence of the minimum and maximum value across the time series.

The range is the difference between the maximum and the minimum value._____no_output_____

# Let's calculate the minimum, maximum and range of our signal
max_value = max(signal)
min_value = min(signal)
signal_range = max_value - min_value

print("Maximum: ", max_value, "\nMinimum: ", min_value, "\nRange (maximum - minimum): ", signal_range)Maximum:  1.10101318359375
Minimum:  -0.170013427734375
Range (maximum - minimum):  1.271026611328125
# Let's draw this values in our signal
index_max = numpy.where(signal == max_value)[0]
index_min = numpy.where(signal == min_value)[0]

# Let's plot our raw signal
p_maxmin = figure(plot_width=1000, plot_height=200)
p_maxmin.ygrid.grid_line_alpha=0.5

p_maxmin.line(time, signal)
p_maxmin.circle([time[i] for i in index_max],[max_value for i in range(len(index_max))], size=15, line_color="navy", fill_color="orange", fill_alpha=0.5, legend_label='Maximum')
p_maxmin.circle([time[i] for i in index_min],[min_value for i in range(len(index_min))], size=15, line_color="orange", fill_color="navy", fill_alpha=0.5, legend_label='Minimum')

show(p_maxmin)_____no_output_____

4.2 - Mean

The arithmetic mean ($\bar{x}$) of a temporal data set represents the average power value across the signal duration.

\begin{align}
\bar{x}=\frac{1}{n}\sum_{i=1}^n x_i=\frac{x_1+x_2+\cdots+x_n}{n} \\
\end{align}
With $x_1, x_2, \ldots, x_n$ being the values in the dataset and $n$ the number of elements in the dataset._____no_output_____

# Let's calculate the mean value of our signal
mean = numpy.mean(signal)

print('Mean:', mean)Mean: 0.00021564978679627862
# Let's draw the mean line in our plot
p.line(time, mean, line_alpha=0.5, color="firebrick", line_width=3, legend_label='Mean')

show(p)_____no_output_____

4.3 - Mode

The mode represents the power value with the highest number of occurrences in the dataset across the signal duration.

There may be more than one mode, if two or more different values have the same number of occurrences._____no_output_____

# Let's find the mode (or modes) in our signal
mode, occurrences = stats.mode(signal)

print('Mode: ', mode, '. Occurrences: ', occurrences)Mode:  [-0.04655457] . Occurrences:  [9]
# Let's print the mode values in a separate new graph
index_mode = numpy.where(signal == mode)

p_modes = figure(plot_width=1000, plot_height=200)
p_modes.line(time, signal)
p_modes.ygrid.grid_line_alpha=0.5

index_aux = 0
for i in index_mode:
    p_modes.line(time, mode[index_aux], color="green", line_width=3, line_alpha=0.5) # we can draw the line marking the mode value in the y-axis
    p_modes.circle([time[idx] for idx in i], mode[index_aux],
             size=10, line_color="green", fill_color="green", fill_alpha=0.5) # we can also mark the points where the function has the mode value
    index_aux += 1

show(p_modes)_____no_output_____

4.4 - Median value and quantiles

The median is the value that separates the top half from the lower half in a dataset. It is a popular summary statistic because it is simple to calculate and gives a measure that is more robust than the mean in the presence of outlier values.

[//]: <> (\begin{align})
[//]: <> (\mathrm{median}(a) = \frac{a_{\lfloor(\#x+1) \div 2\rfloor} + a_{\lceil(\#x+1) \div 2\rceil}}{2} \\)
[//]: <> (\end{align})

The median is also known as the 2-quantile because it marks the middle point of the dataset. However, other quantiles  can also be explored.

For example, 4-quantiles (or quartiles) divide the dataset in four sections:
- the first quartile (Q1), also known as the 25th percentile, splits off the lowest 25% of data from the highest 75%.
- the second quartile (Q2), also known as the 50th percentile, is the same as the median.
- the third quartile (Q3), also known as the 75th percentile splits off the lowest 75% of data from the highest 25%._____no_output_____

# Let's calculate the median, 25 and 75 quartiles
median = numpy.median(signal) # 50% of the values of the signal will be lower than the median or 50th quantile.
Q1 = numpy.quantile(signal, 0.25) # 25% of the values of the signal will be lower than the 25th quantile
Q3 = numpy.quantile(signal, 0.75) # 75% of the values of the signal will be lower than the 75th quantile

print('Mean: ', mean)
print('| Q1 |\t\t\t| Q2 (Median) |\t\t\t| Q3 |\n', Q1, '\t', median, '\t\t', Q3)Mean:  0.00021564978679627862
| Q1 |			| Q2 (Median) |			| Q3 |
 -0.0703582763671875 	 -0.0411529541015625 		 0.026275634765625
# Let's see them in a separated new graph to avoid cluttering the main one
p_quartiles = figure(plot_width=1000, plot_height=200)
p_quartiles.line(time, signal)
p_quartiles.ygrid.grid_line_alpha=0.5

# Redrawing the mean line...
p_quartiles.line(time, mean, color="red", line_width=2, legend_label='Mean')

p_quartiles.line(time, Q3, color="yellow", line_width=2, line_dash='dashed', legend_label='Q3')
p_quartiles.line(time, median, color="deeppink", line_width=2, line_dash='dashed', legend_label='Q2 (Median)')
p_quartiles.line(time, Q1, color="cyan", line_width=2, line_dash='dashed', legend_label='Q1')

p_quartiles.legend.location = "top_left"
p_quartiles.y_range = Range1d(-0.15, 0.15)
p_quartiles.x_range = Range1d(-1, 6)

show(p_quartiles)_____no_output_____

In this example we can observe that the distance between the Q3 and Q2 is bigger than the distance between Q2 and Q1.
Quartiles give us and idea about how the signal's values are distributed.

Even though the signal has extreme values (peaks), their frequency of occurrence is low, and that is why the distance between the Q3 line and the top of the graph is much bigger than the distance between any of the other quartiles._____no_output_____
4.5 - Standard deviation and variance

Both the standard deviation and the variance are statistics determined by using the mean, and used to measure the degree of dispersion in a dataset. Informally, they measure how close a group of numbers tend to be to the mean.

The variance ($\sigma^2$) is defined as the average of the squared differences from the mean of the dataset. It is obtained by calculating the difference between each point of the dataset and the mean, squaring and averaging the results. Squaring the distance from the mean gives greater weight to values that are further from the mean.
However, because of the squaring, the variance is no longer in the same measurement units as the dataset. This makes measuring variance a bit difficult, so many times the standard deviation is used instead.

The standard deviation ($\sigma$) is the square root of the variation. A low standard deviation value indicates that the values tend to be more concentrated and closer to the mean, whilst a high value indicates that the values tend to spread out over a wider range.

\begin{align}
\sigma = \sqrt{\frac{1}{N-1}\sum_{i=1}^N (x_i - \bar{x})^2 }, \\
\end{align}_____no_output_____

# Let's calculate the standard deviation and the variance
std = numpy.std(signal)
variance = numpy.var(signal)

print('Mean: ', mean)
print('Standard deviation: ', std, '\nVariance: ', variance)Mean:  0.00021564978679627862
Standard deviation:  0.16587464942332372
Variance:  0.02751439932131055
# Let's draw the std line in our graph
p.line(x=time, y=mean+std, line_color='orange', line_dash='dashed', line_width=2, legend_label='Standard deviation')
p.line(x=time, y=mean-std, line_color='orange', line_dash='dashed', line_width=2)

show(p)_____no_output_____

4.6 - Skewness and kurtosis

The skewness is a metric of symmetry (or the lack of) in a dataset. A dataset is symmetric if it looks the same to the left and right of the center point. A normal distribution has a skewness of 0.

- Negative (<0) skew value: longer left tail; distribution concentrates on the right.
- Positive (>=0) skew value: longer right tail; distribution concentrates on the left.

Kurtosis indicates if the dataset is heavy-tailed or light-tailed in comparison to a normal distribution. Datasets with high kurtosis ('heavy-tailed') tend to have outliers. The lower the kurtosis value, the less common it is to find outliers in a dataset._____no_output_____

# Let's calculate the skewness and kurtosis
skewness = stats.skew(signal)
kurtosis = stats.kurtosis(signal)

print('Skewness: ', skewness, '\nKurtosis: ', kurtosis)Skewness:  4.271322272366648
Kurtosis:  21.250186573425474
arr_hist, edges = numpy.histogram(signal, bins = int(180/5))

# Put the information in a dataframe
delays = pd.DataFrame({'arr_delay': arr_hist, 'left': edges[:-1], 'right': edges[1:]})

p_histogram = figure(plot_height = 250, plot_width = 1000, x_axis_label="mV", y_axis_label="bins")

# Add a quad glyph
p_histogram.quad(bottom=0, top=delays['arr_delay'],
       left=delays['left'], right=delays['right'],
       fill_color='red', line_color='black')

# Show the plot
show(p_histogram) _____no_output_____

The histogram shows clearly that the time series is right tailed, which is consistent with the positive skew value obtained.
Some outliers are also present on the right side of the histogram, around the 1mv value. This is consistent with the kurtosis result, because as we previously mentioned in the analysis of the quartiles, the highest peaks in the heartbeat cycle are, althought with a low frequency, common across the time signal._____no_output_____
5 - Temporal parameters

Parameters for analyzing time series and extract meaningful statistics or other characteristics of the signal.
These are parameters specific for data series located in the time domain. They extract relevant and descriptive information about a dataset in virtue of its temporal nature, which should be taken into account when analyzing temporal series.

5.1 - Autocorrelation

Autocorrelation, or serial correlation, is often used in time domain signals. It is the relationship of a signal with a delayed copy of itself as a function of delay. The analysis of autocorrelation finds repeating patterns, such as the presence of a periodic signal obscured by noise.

The ACF (autocorrelation function) shows the correlation between points separated by various time lags. So, in other words, it is the degree of association between points based on how many time steps apart they are. Normally, the autocorrelation function falls towards 0 as points become more separated, because the bigger the separation, the less correlation between the points. This is not a rule, but it is the most typical scenario._____no_output_____

# Let's obtain the autocorrelation function for our signal
autocorrelation_function = stattools.acf(signal, fft=True, nlags=round(len(signal)*0.75))_____no_output_____def get_autocorrelation_plot_params(series):
    n = len(series)
    data = numpy.asarray(series)
    mean = numpy.mean(data)
    c0 = numpy.sum((data - mean) ** 2) / float(n)

    def r(h):
        return ((data[:n - h] - mean) *
                (data[h:] - mean)).sum() / float(n) / c0
    x = numpy.arange(n) + 1
    y = list(map(r, x))
    z95 = 1.959963984540054 # confidence interval 95%
    z99 = 2.5758293035489004 # confidence interval 99%
    return n, x, y, z95, z99

n, x, y, z95, z99 = get_autocorrelation_plot_params(autocorrelation_function)
x = x/100
auto_correlation_plot2 = figure(title='Time Series Auto-Correlation', plot_width=1000,
                                plot_height=400, x_axis_label="Lag (s)", y_axis_label="Autocorrelation")

auto_correlation_plot2.line(x, y=z99 / numpy.sqrt(n), line_dash='dashed', line_color='red', legend_label='99% confidence band')
auto_correlation_plot2.line(x, y=z95 / numpy.sqrt(n), line_color='red', legend_label='95% confidence band')
auto_correlation_plot2.line(x, y=0.0, line_color='black')
auto_correlation_plot2.line(x, y=-z95 / numpy.sqrt(n), line_color='red')
auto_correlation_plot2.line(x, y=-z99 / numpy.sqrt(n), line_dash='dashed', line_color='red')

auto_correlation_plot2.line(x, y, line_width=1)
#auto_correlation_plot2.circle(x, y, fill_color="white", size=8)  # optional

curdoc().add_root(column(auto_correlation_plot2))

show(auto_correlation_plot2)_____no_output_____

As expected, the ACF tends to 0 the longer the distance between the points. However the analysis highlights the presence of a periodic pattern, with high correlation, characterized by the sucesion of a negative, positive and negative peak with a period of less than 1 second.

In this case, since this signal is a ECG recording, it makes sense that there is a periodicy associated with the heartbeat._____no_output_____
5.2 - Stationarity

A time series is stationary when its statistic properties, such as the mean and variance, do not change over time.

The stationarity of a dataset can usually be predicted by looking at the signal's plot, histogram or ACF. It can also be checked by calculating the mean and variance of the signal in different time intervals; if the results are similar then the signal is probably stationary. There are also statistical tests to determine if a time series is stationary, for instance the Dickey-Fuller test or the KPSS (Kwiatkowski-Phillips-Schmidt-Shin) test._____no_output_____

# Let's try to check the stationarity of our signal
print('Length of the signal: ', len(signal))
# We can measure the mean and variance of different intervals of the signal and see if they are different
mean_1 = numpy.mean(signal[1:300])
mean_2 = numpy.mean(signal[800:1100])
mean_3 = numpy.mean(signal[1600:1900])

var_1 = numpy.var(signal[1:300])
var_2 = numpy.var(signal[800:1100])
var_3 = numpy.var(signal[1600:1900])

print('Means: ', mean_1, mean_2, mean_3)
print('Variances: ', var_1, var_2, var_3, '\n')

# Let's try the Dickey-Fuller test
from statsmodels.tsa.stattools import adfuller

adf_test = adfuller(signal)

# Let's try the KPSS test
from statsmodels.tsa.stattools import kpss

kpsstest = kpss(signal, regression='c')Length of the signal:  1965
Means:  -0.0026015980187865805 0.006810760498046875 -0.00485137939453125
Variances:  0.0260274864826267 0.03005196664656978 0.025713346455339344

# Using pandas Series, we can print the ADF results in a readable way.
print ('Results of ADF Test:')
dfoutput = pd.Series(adf_test[0:4], index=['Test Statistic','p-value','#Lags Used','Number of Observations Used'])
for key,value in adf_test[4].items():
   dfoutput['Critical Value (%s)'%key] = value
print (dfoutput)

# Using pandas Series, we can print the KPSS results in a readable way.
print ('\nResults of KPSS Test:')
kpss_output = pd.Series(kpsstest[0:3], index=['Test Statistic','p-value','Lags Used'])
for key,value in kpsstest[3].items():
    kpss_output['Critical Value (%s)'%key] = value
print (kpss_output)Results of ADF Test:
Test Statistic                -9.868093e+00
p-value                        4.063043e-17
#Lags Used                     2.600000e+01
Number of Observations Used    1.938000e+03
Critical Value (1%)           -3.433729e+00
Critical Value (5%)           -2.863033e+00
Critical Value (10%)          -2.567565e+00
dtype: float64

Results of KPSS Test:
Test Statistic            0.014125
p-value                   0.100000
Lags Used                26.000000
Critical Value (10%)      0.347000
Critical Value (5%)       0.463000
Critical Value (2.5%)     0.574000
Critical Value (1%)       0.739000
dtype: float64

Although the means remain pretty similar in the three chosen intervals, the variances do show some significant changes.

Let's analyze the results of the statistical tests:

- The Dickey-Fuller test returned a very small p-value. Since this value is much smaller than 0.05 (95% confidence interval) and 0.01 (99% confidence interval), then we can reject the null hypothesis of non-stationarity. This test deems the signal as stationary.
- In the case of the KPSS test, the results have to be interpreted in the opposite way: a high p-value would point towards a stationary signal. In our case, however, the p-value is 0.1, which means that the signal may be non-stationary.

So we got opposite results in each test. What does this mean?
In reality, there is more than one type of non-stationarity. As a short introduction, there are three possible types of signals based in stationarity:

- Strict stationary: signals where the mean, variance and covariance do not vary with time. The aim is to convert non-stationary series into strict stationary series for making predictions.
- Trend stationary: a time series that can be made strict stationary by removing its trend. KPSS is a trend stationarity test.
- Difference stationary: a time series that can be made strict stationary by differencing. ADF is a difference stationarity test.

Therefore, the results of the KPSS and ADF tests are useful to test wether a signal es stationary, trend stationary or difference stationary:

- If both tests conclude that the series is not stationary: the series is not stationary.
- If both tests conclude that the series is stationary: the series is stationary.
- If KPSS = stationary and ADF = not stationary: the signal is trend stationary.
- If KPSS = not stationary and ADF = stationary: the signal difference stationary.

So in this case our signal would fall under the difference stationarity definition. To understand more about signal stationarity, there are some interesting online resources  that go deeper on how to transform non-stationary signals and different ways to handle them._____no_output_____
5.3 - Seasonality decomposition

The seasonality decomposition of a signal allows dividing the main time series in different components, each of which has a specific characteristic. This decomposition can be done following two different models:

- additive, for linear distributions, where the time signal is decomposed following the formula: $Signal=trend+seasonality+residue$
- multiplicative, for exponential series, where the time signal is decomposed following the formula: $Signal=trend*seasonality*residue$

The resulting components are the following:
- Trend: increasing or decreasing tendency in the time series.
- Seasonality: the repeating short-term cycles in the series.
- Residue: random noise present in the signal that cannot be attributed to trend or seasonality_____no_output_____

# Let's decompose our signal into trend, seasonal and residual components using seasonal decomposition
signal = numpy.array(signal) - numpy.mean(signal)
filter_signal_1 = bsnb.lowpass(signal, f=40, order=1, fs=sr)
filter_signal_2 = bsnb.lowpass(filter_signal_1, f=40, order=3, fs=sr)
results = seasonal.seasonal_decompose(filter_signal_2, model='additive', freq=sr, extrapolate_trend='freq')_____no_output_____decomposition_plot2 = figure(title='Seasonal decomposition - Observed and trend', plot_width=1200, plot_height=200)
decomposition_plot2.line(numpy.arange(0,len(results.observed)), results.observed, legend_label='Observed')
decomposition_plot2.line(numpy.arange(0,len(results.trend)), results.trend, line_color='red', legend_label='Trend')
show(decomposition_plot2)

decomposition_plot3 = figure(title='Seasonal decomposition - Seasonality and residuals', plot_width=1200, plot_height=200)
decomposition_plot3.line(numpy.arange(0,len(results.seasonal)), results.seasonal, line_color='orange', legend_label='Seasonality')
decomposition_plot3.line(numpy.arange(0,len(results.resid)), results.resid, line_color='green', legend_label='Residuals')
show(decomposition_plot3)_____no_output_____

The decomposition in this case is unable to separate the cyclic element (the heartbeat) from the signal. That is why the most significant part of the signal ends up as residues. This is not a good result, and it is probably caused by the naive approach of the seasonal_decompose function in the statsmodels package.

There are more complex and robust approaches to this problem, like Loess or STL decomposition, which would however require a more in depth explanation._____no_output_____**Auxiliary Code Segment (should not be replicated by the user)**_____no_output_____

bsnb.css_style_apply().................... CSS Style Applied to Jupyter Notebook .........................


# Notebook from vravishankar/Jupyter-Books
Path: Decorators.ipynb

# Decorators

Decorators can be thought of as functions which modify the functionality of another function. This is also called metaprogramming as one part of the program tries to modify another part of program at compile time. It can also be used to add functionality to the existing code.

There are 2 types of decorators:

* Function Decorators
* Class Decorators

To understand decorators better lets start with some important aspects of functions. First you have to know or remember that function names are references to functions and that we can assign multiple names to the same function:
_____no_output_____

def func(n):
    return n + 1

x = func(1)
print(x)
y = func(2)
print(y)2
3

 Here x and y refers to the same function object.

 The next important fact is that we can delete either "succ" or "successor" without deleting the function itself._____no_output_____

del y
print(x)2
print(y)_____no_output_____

## Functions within functions_____no_output_____

def f():

    def g():
        print("Hi, it's me 'g'")
        print("Thanks for calling me")

    print("This is the function 'f'")
    print("I am calling 'g' now:")
    g()

f()This is the function 'f'
I am calling 'g' now:
Hi, it's me 'g'
Thanks for calling me
def temperature(t):
    def celsiusToFarhenheit(x):
        return 9 * x / 5 + 32
    result = "It's " + str(celsiusToFarhenheit(t)) + " degrees"
    return result

temperature(10)_____no_output_____

## Functions as Parameters

Functions can be passed as arguments to another function. Such functions that take other functions as arguments are called **Higher Order Functions**._____no_output_____

def inc(x):
    return x + 1
def dec(x):
    return x - 1
def operate(func,x):
    result = func(x)
    return result

operate(inc,1)_____no_output_____operate(dec,2)_____no_output_____

## Functions returning functions

The output of a function is also a reference to an object._____no_output_____

def f(x):
    def g(y):
        return y + x + 3
    return g

nf1 = f(1)
nf2 = f(3)

print(nf1(1))
print(nf2(1))5
7
def make_pretty(func):
    def inner():
        print("I got decorated")
        func()
    return inner

def ordinary():
    print("I am ordinary")

ordinary()I am ordinary
pretty = make_pretty(ordinary)
pretty()I got decorated
I am ordinary

In the example shown above the function make_pretty() is a decorator. The function ordinary() got decorated and the returned function was given the name pretty.

We can see that the decorator function added some new functionality to the original function. This is similar to packing a gift. The decorator acts as a wrapper. The nature of the object that got decorated does not alter. But now, it looks pretty.

Generally we decorate a function and reassign it as

```python
ordinary = make_pretty(ordinary)
```

This is a common construct and for this reason, Python has a syntax to simplify this. We can use the @ symbol along with the name of the decorator function and place it above the definition of the function to be decorated. For example,

```python
@make_pretty
def ordinary():
    print("I am ordinary")
```

is equivalent to

```python
def ordinary():
    print('I am ordinary')
ordinary = make_pretty(ordinary)
```

This is just a syntactic sugar to implement decorators._____no_output_____

@make_pretty
def ordinary():
    print("I am ordinary")

ordinary()I got decorated
I am ordinary

## Decorating Functions with Parameters

The above decorator was simple and it only worked with functions that did not have any parameters. What if we had functions that took in parameters like below?

```python
def divide(a,b):
    return a/b
```

The function has 2 parameters a and b. We know, it will give error if we pass in b as 0._____no_output_____

def divide(a,b):
    return a/b

divide(2,5)
divide(2,0)_____no_output_____

To avoid this error lets make a decorator to check for this case._____no_output_____

def smart_divide(func):
    def inner(a,b):
        if b == 0:
            print('Cannot divide by zero')
            return
        return func(a,b)
    return inner

@smart_divide
def divide(a,b):
    return a / b_____no_output_____divide(5,2)_____no_output_____divide(5,0)Cannot divide by zero

If you observered closely the number of parameters of the nested inner() function inside the decorator is same as the parameters of functions it decorates. Taking this into account, now we can make general decorators that work with any number of parameter.

This magic is done as function(\*args, \**kwargs). In this way, args will be the tuple of positional arguments and kwargs will be the dictionary of keyword arguments. An example of such decorator will be._____no_output_____

def works_for_all(func):
    def inner(*args, **kwargs):
        print('I can decorate any function...')
        return func(*args, **kwargs)
    return inner_____no_output_____

## Chaining Decorators in Python

Multiple decorators can be chained in python._____no_output_____

def star(func):
    def inner(*args, **kwargs):
        print('*'* 30)
        func(*args, **kwargs)
        print('*'* 30)
    return inner

def percent(func):
    def inner(*args, **kwargs):
        print('%'*30)
        func(*args,**kwargs)
        print('%'*30)
    return inner

@star
@percent
def printer(msg):
    print(msg)

printer("Hello")******************************
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Hello
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
******************************

This is equivalent to:_____no_output_____

def printer(msg):
    print(msg)

printer = star(percent(printer))
printer('Hello')******************************
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Hello
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
******************************

Please note the order in which we chain the decorators also matter._____no_output_____

@percent
@star
def printer(msg):
    print(msg)

printer('Hello')%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
******************************
Hello
******************************
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


# Notebook from Djamil17/rna-seq-analysis
Path: notebooks/Normalization .ipynb

<code>
import numpy as np
import pandas as pd

# Import TCGA melanoma data
## Rna read data
file='../data/counts.txt'
with open(file, 'rt') as f:
    read_counts=pd.read_csv(f,index_col=0)

## Reference genome
file2='../data/genes.csv'
with open(file2, 'rt') as f:
    gene_reference=pd.read_csv(f,index_col=0) _____no_output_____read_counts_____no_output_____gene_reference_____no_output_____matched_index=pd.Index.intersection(read_counts.index, gene_reference.index)_____no_output_____matched_index_____no_output_____counts=np.asarray(read_counts.loc[matched_index], dtype=int)_____no_output_____counts_____no_output_____genes_name=np.array(matched_index)_____no_output_____genes_length=np.asarray(gene_reference.loc[matched_index]['GeneLength'], dtype=int)_____no_output_____print(f'{counts.shape[0]} genes measured in {counts.shape[1]} individuals') 20500 genes measured in 375 individuals
print(counts.shape, genes_name.shape)(20500, 375) (20500,)
import matplotlib.pyplot as plt

plt.style.use('ggplot')_____no_output_____total_counts=np.sum(counts, axis=0)_____no_output_____from scipy import stats

density=stats.kde.gaussian_kde(total_counts)
_____no_output_____density

x=np.arange(min(total_counts),max(total_counts), 10000)

fig,ax=plt.subplots()
ax.plot(x,density(x))_____no_output_____np.random.seed(seed=17)
samples_index=np.random.choice(range(counts.shape[1]), size=70, replace=False)
counts_subset=counts[:,samples_index]
_____no_output_____def reduce_x_axis(ax,factor):
    plt.setp(ax.xaxis.get_ticklabels(),visible=False)
    for label in ax.xaxis.get_ticklabels()[factor-1::factor]:
        label.set_visible(True)

fig,ax= plt.subplots(figsize=(20,8))

with plt.style.context('ggplot'):
    ax.boxplot(counts_subset)
    ax.set_xlabel("Individuals")
    ax.set_ylabel("Gene Expression Count")
    reduce_x_axis(ax,5)_____no_output_____fig,ax= plt.subplots(figsize=(20,8))

with plt.style.context('ggplot'):
    ax.boxplot(np.log(counts_subset +1 ))
    ax.set_xlabel("Individuals")
    ax.set_ylabel("Gene Expression Count")
    reduce_x_axis(ax,5)_____no_output_____counts_lib_norm=counts/total_counts * 1000000
counts_lib_subset=counts_lib_norm[:,samples_index]_____no_output_____fig,ax= plt.subplots(figsize=(20,8))

with plt.style.context('ggplot'):
    ax.boxplot(np.log(counts_lib_subset +1 ))
    ax.set_xlabel("Individuals")
    ax.set_ylabel("Gene Expression Count")
    reduce_x_axis(ax,5)_____no_output_____def binned_boxplots(x,y,**kwargs):
    x_hist, x_bins=np.histogram(x,bins='auto')
    x_bin_ids=np.digitize(x,x_bins[:-1])
    binned_y=[y[x_bin_ids==i]
            for i in range(np.max(x_bin_ids))]
    fig,ax=plt.subplots(figsize=(20,8))

    x_bin_centers=(x_bins[1:] + x_bins[:-1]) /2
    x_ticklabels=np.round(np.exp(x_bin_centers)).astype(int)

    ax.boxplot(binned_y,labels=x_ticklabels)
    reduce_x_axis(ax,10)
    dict_values=kwargs.values()
    list_values=list(dict_values)
    ax.set_xlabel(list_values[0])
    ax.set_ylabel(list_values[1])

log_counts=np.log(counts_lib_norm +1)
mean_log_counts=np.mean(log_counts,axis=1)
log_gene_lengths=np.log(genes_length)
    _____no_output_____with plt.style.context('ggplot'):
    binned_boxplots(x=log_gene_lengths, y=mean_log_counts, x_label='log_gene_length',y_label='mean_log_counts')
_____no_output_____C=counts
N=counts.sum(axis=0)
L=genes_length_____no_output_____C_9=counts * 10 ^9_____no_output_____C_9.shape
L.shape_____no_output_____L=L[:,np.newaxis]_____no_output_____C_tmp=C_9 / L
C_tmp_____no_output_____N=N[np.newaxis,:]_____no_output_____rpkm=C_tmp/N_____no_output_____def rpkm(counts, length):
    """

    RPKM= (10^9 * C) / (N * L)

    C= Number of reads mapped to gene
    N=Total mapped reads in the experiment
    L=Exon length in base pairs for a gene

    Parameters

    ..........

    counts: array, shape (N_genes,N_samples)
    lengths: array , shape (N_genes, )

    Returns

    ..........

    N=np.sum(counts, axis=0)

    """

    N=np.sum(counts, axis=0)
    L=length
    C=counts

    N_broadcast=N[np.newaxis,:]
    L_broadcast=L[:,np.newaxis]

    normed= (10 ^ 9 * C ) / (N_broadcast * L_broadcast)

    return normed

counts_rpkm=rpkm(counts,genes_length)

        _____no_output_____log_counts_rpkm=np.log(counts_rpkm + 1 )
mean_log_counts_rpkm=np.mean(log_counts_rpkm,axis=1)
log_length_rpkm=np.log(genes_length)

with plt.style.context('ggplot'):
    binned_boxplots(x=log_length_rpkm, y=mean_log_counts_rpkm, x_label='log_gene_length',y_label='mean_log_counts')_____no_output_____
</code>


# Notebook from kfuku52/amalgk
Path: util/update_sra_table.ipynb

<code>
import pandas, numpy, os, seaborn, matplotlib

%matplotlib inline
pandas.options.display.max_rows=100
pandas.options.display.max_columns=1000
seaborn.set_style("white")
seaborn.set_style("ticks")
font_size = 7
matplotlib.rcParams['svg.fonttype'] = 'none' # none, path, or svgfont
matplotlib.rcParams['axes.titlepad'] = 3
matplotlib.rcParams['axes.labelpad'] = 3
matplotlib.rcParams['font.size'] = font_size
matplotlib.rcParams['legend.fontsize'] = 'medium'
matplotlib.rcParams['figure.titlesize'] = 'medium'
matplotlib.rcParams['svg.fonttype'] = 'none' # none, path, or svgfont_____no_output_____sra_date='2018_5_1'

os.chdir('/Users/kf/Dropbox/kfdata/02_Data/my_db/Ensembl/release-91/sra/'+sra_date)

file_sra="/Users/kf/Dropbox/kfdata/02_Data/my_db/Ensembl/release-91/sra/"+sra_date+"/sra_table_curated_"+sra_date+".tsv"
file_sra_amalgamated="/Users/kf/Dropbox/kfdata/02_Data/my_db/Ensembl/release-91/sra/"+sra_date+"/sra_table_amalgamated_"+sra_date+".tsv"

dir_kallisto='/Users/kf/Dropbox/kfdata/02_Data/my_db/Ensembl/release-91/kallisto_summary/'+sra_date+'/'

dir_tpm_unfiltered=dir_kallisto+'tpm.unfiltered.kallisto.tsv/'
dir_tpm_fastp=dir_kallisto+'tpm.fastp.kallisto.tsv/'
dir_tpm_masked=dir_kallisto+'tpm.masked.kallisto.tsv/'

dir_ec_unfiltered=dir_kallisto+'est_counts.unfiltered.kallisto.tsv/'
dir_ec_fastp=dir_kallisto+'est_counts.fastp.kallisto.tsv/'
dir_ec_masked=dir_kallisto+'est_counts.masked.kallisto.tsv/'

dir_fatt=dir_kallisto+'fatt.tsv/'_____no_output_____sra = pandas.read_csv(file_sra, sep='\t')

steps = ['unfiltered','fastp','masked']
col1 = ['run', 'layout']
col2 = [ c1+c2 for c1 in ['num_read_','total_nucl_','avg_read_len_','min_read_len_','max_read_len_'] for c2 in steps ]
cols = col1 + col2

# Annotate fatt statistics
exts = {'unfiltered':'.*(_1|_2|[0-9]{2,})\.fastq$',
        'fastp':'.*[0-9]\.fastp\.fastq$',
        'masked':'.*[0-9]*\.masked\..*fastq$',
       }
df_fatt1 = pandas.DataFrame()
df_in = pandas.DataFrame()
files = os.listdir(dir_fatt)
for file in files:
    if (os.path.getsize(dir_fatt+file)>10)&(file!='.DS_Store'):
        df_in = df_in.append(pandas.read_csv(dir_fatt+file, sep='\t'))
print('Number of fatt record before dropping duplicates:', df_in.shape[0])
df_in = df_in.drop_duplicates()
print('Number of fatt record after dropping duplicates:', df_in.shape[0])
for item in exts.items():
    a = item[0]
    ext = item[1]
    df1 = df_in.loc[df_in['FILE'].str.match(ext),:]
    df1.loc[:,'FILE'] = df1.loc[:,'FILE'].replace('_.*', '', regex=True).replace('\..*', '', regex=True)
    if a=='unfiltered':
        tmp = pandas.DataFrame(df1.groupby('FILE')['NUM_READS'].count()); tmp.columns = ['layout']
        df_fatt1 = pandas.merge(df_fatt1, tmp, left_index=True, right_index=True, how='outer')
    tmp = pandas.DataFrame(df1.groupby('FILE')['NUM_READS'].mean()); tmp.columns = ['num_read_'+a]
    df_fatt1 = pandas.merge(df_fatt1, tmp, left_index=True, right_index=True, how='outer')
    tmp = pandas.DataFrame(df1.groupby('FILE')['NUM_NUCLS'].sum()); tmp.columns = ['total_nucl_'+a]
    df_fatt1 = pandas.merge(df_fatt1, tmp, left_index=True, right_index=True, how='outer')
    tmp = pandas.DataFrame(df1.groupby('FILE')['AVG_READ_LEN'].mean()); tmp.columns = ['avg_read_len_'+a]
    df_fatt1 = pandas.merge(df_fatt1, tmp, left_index=True, right_index=True, how='outer')
    tmp = pandas.DataFrame(df1.groupby('FILE')['MIN_READ_LEN'].min()); tmp.columns = ['min_read_len_'+a]
    df_fatt1 = pandas.merge(df_fatt1, tmp, left_index=True, right_index=True, how='outer')
    tmp = pandas.DataFrame(df1.groupby('FILE')['MAX_READ_LEN'].max()); tmp.columns = ['max_read_len_'+a]
    df_fatt1 = pandas.merge(df_fatt1, tmp, left_index=True, right_index=True, how='outer')
df_fatt1['run'] = df_fatt1.index
df_fatt1 = df_fatt1.loc[~df_fatt1['run'].duplicated(),cols]
sra = pandas.merge(sra, df_fatt1, on='run', how='left')

# Annotate kallisto statistics
indirs = [dir_ec_unfiltered,dir_ec_fastp,dir_ec_masked]
for indir,ext in zip(indirs,steps):
    files = os.listdir(indir)
    mapped_reads = pandas.Series()
    for file in files:
        if os.path.getsize(indir+file)>10:
            df = pandas.read_csv(indir+file, sep='\t')
            sra_cols = df.columns[~df.columns.isin(['target_id','length'])]
            tmp_mapped_reads = df.loc[:,sra_cols].sum(axis=0)
            mapped_reads = mapped_reads.append(tmp_mapped_reads)
    mapped_reads = pandas.DataFrame(mapped_reads)
    mapped_reads.columns = ['mapped_read_'+ext,]
    mapped_reads['run'] = mapped_reads.index
    sra = pandas.merge(sra, mapped_reads, on='run', how='left')

for ext in steps:
    sra.loc[(sra['num_read_'+ext]==0),'num_read_'+ext] = numpy.nan
    sra['mapping_rate_'+ext] = sra['mapped_read_'+ext] / sra['num_read_'+ext]
/Users/kf/anaconda/envs/ete3/lib/python3.5/site-packages/IPython/core/interactiveshell.py:2698: DtypeWarning: Columns (34) have mixed types. Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)
seaborn.regplot(x='num_read_unfiltered', y='num_read_fastp', data=sra, fit_reg=False, scatter_kws={'alpha':0.1}) # blue
seaborn.regplot(x='num_read_unfiltered', y='num_read_masked', data=sra, fit_reg=False, scatter_kws={'alpha':0.1}) # orange
seaborn.regplot(x='num_read_fastp', y='num_read_masked', data=sra, fit_reg=False, scatter_kws={'alpha':0.1}) # green
seaborn.regplot(x=numpy.arange(0,0.2,0.01), y=numpy.arange(0,0.2,0.01), scatter=False)_____no_output_____seaborn.regplot(x='mapping_rate_unfiltered', y='mapping_rate_fastp', data=sra, fit_reg=False, scatter_kws={'alpha':0.3}) # blue
#seaborn.regplot(x='mapping_rate_unfiltered', y='mapping_rate_masked', data=sra, fit_reg=False, scatter_kws={'alpha':0.1}) # orange
#seaborn.regplot(x='mapping_rate_fastp', y='mapping_rate_masked', data=sra, fit_reg=False, scatter_kws={'alpha':0.1}) # green
seaborn.regplot(x=numpy.arange(0,0.2,0.01), y=numpy.arange(0,0.2,0.01), scatter=False)_____no_output_____def plot_transcriptome_curation(sra, spp=[]):
    cols = ['num_read', 'total_nucl', 'avg_read_len', 'mapping_rate']
    ylabs = ['Number of reads', 'Total nucleotides (nt)', 'Mean read length (nt)', 'Mapping rate']
    ylims = [[10**6, (10**9)*0.3],[10**8, 10**11],[0, 260],[0,1]]
    num_row = len(cols)
    num_col = 1
    # full figure size = 9.7 length x 7.2 width in inches
    fig,axes = matplotlib.pyplot.subplots(nrows=num_row, ncols=num_col, figsize=(7.2,1.5*num_row), sharex=True, sharey=False)
    axes = axes.flat
    ind = 0
    for col in cols:
        if len(spp):
            df_plot = sra.loc[sra.scientific_name.isin(spp),:]
        else:
            df_plot = sra
        df_plot.index = df_plot['scientific_name']
        df_plot = df_plot.loc[:,[col+'_unfiltered', col+'_fastp', col+'_masked']]
        df_plot = df_plot.stack().reset_index()
        df_plot.columns = ['scientific_name', 'filter', 'value']
        #ax = seaborn.stripplot(x='scientific_name', y='value', hue='filter', data=df_plot, ax=axes[ind], jitter=True, dodge=True, alpha=0.2, edgecolor='gray')
        #ax = seaborn.violinplot(x='scientific_name', y='value', hue='filter', data=df_plot, ax=axes[ind])
        ax = seaborn.boxplot(x='scientific_name', y='value', hue='filter', data=df_plot, ax=axes[ind])
        xlabels = ax.get_xmajorticklabels()
        for xlabel in xlabels:
            xlabel.set_rotation(45)
            xlabel.set_horizontalalignment("right")
        if col in ['num_read', 'total_nucl']:
            ax.set(yscale="log")
        ax.set_xlabel('')
        ax.set_ylabel(ylabs[ind])
        ax.set_ylim(ylims[ind])
        if ind==0:
            ax.legend_.loc = 3
        else:
            ax.legend_ = None
        ind += 1
    return fig,axes

spp = ['Bos taurus','Homo sapiens','Gallus gallus','Macaca mulatta','Canis lupus','Sus scrofa','Ovis aries','Mus musculus','Callithrix jacchus',
'Rattus norvegicus','Monodelphis domestica','Oreochromis niloticus','Oryctolagus cuniculus','Danio rerio','Anolis carolinensis','Chinchilla lanigera',
'Astyanax mexicanus','Lepisosteus oculatus','Xenopus tropicalis','Oryzias latipes','Gadus morhua','Ornithorhynchus anatinus']

fig,axes = plot_transcriptome_curation(sra, spp)

fig.tight_layout()
outbase = 'mapping_rate_curated'
fig.savefig(outbase+".pdf", format='pdf', transparent=True)
fig.savefig(outbase+".svg", format='svg', transparent=True)_____no_output_____sra.to_csv(file_sra.replace('curated', 'mapped'), sep='\t', index=False)_____no_output_____# Run after an amalgamated table is created.

sra_amalgamated = pandas.read_csv(file_sra_amalgamated, sep='\t')
sra_amalgamated = sra_amalgamated.loc[(sra_amalgamated['exclusion']=='no'),:]

fig,axes = plot_transcriptome_curation(sra_amalgamated, spp)

fig.tight_layout()
outbase = 'mapping_rate_amalgamated'
fig.savefig(outbase+".pdf", format='pdf', transparent=True)
fig.savefig(outbase+".svg", format='svg', transparent=True)

def make_cross_tables(sra_table, n_sp_cutoff=5, suppress_tissue=True, sample_upper=True):
    print('Running: make_cross_tables')
    if suppress_tissue:
        sra_table = sra_table.loc[sra_table['is_selected_tissue']=='Yes',:]
    if sample_upper:
        sra_table = sra_table.loc[sra_table['is_sampled']=='Yes',:]
    sra_reduced = sra_table[['scientific_name', 'biosample', 'tissue']]
    sra_pivot = sra_reduced.pivot_table(columns='tissue',index='scientific_name', aggfunc='count')
    sra_pivot.columns = sra_pivot.columns.get_level_values(1)
    column_sort = sra_pivot.count(axis='index').sort_values(ascending=False).index
    index_sort = sra_pivot.count(axis='columns').sort_values(ascending=False).index
    sra_pivot = sra_pivot.loc[index_sort,column_sort]
    sra_pivot_reduced = sra_pivot.loc[:,sra_pivot.count(axis='index') >= n_sp_cutoff]
    column_sort = sra_pivot_reduced.count(axis='index').sort_values(ascending=False).index
    index_sort = sra_pivot_reduced.count(axis='columns').sort_values(ascending=False).index
    sra_pivot_reduced = sra_pivot_reduced.loc[index_sort,column_sort]
    return sra_pivot_reduced

d = '2018_5_1'
sra_amalgamated_pivot = make_cross_tables(sra_amalgamated, n_sp_cutoff=5, suppress_tissue=False, sample_upper=False)
sra_amalgamated_pivot.to_csv('sra_table_pivot_amalgamated_'+d+'.tsv', sep='\t')Running: make_cross_tables
# Run after an amalgamated table is created.

spp = ['Bos taurus','Homo sapiens','Gallus gallus','Macaca mulatta','Canis lupus','Sus scrofa','Ovis aries','Mus musculus','Callithrix jacchus',
'Rattus norvegicus','Monodelphis domestica','Oreochromis niloticus','Oryctolagus cuniculus','Danio rerio','Anolis carolinensis','Chinchilla lanigera',
'Astyanax mexicanus','Lepisosteus oculatus','Xenopus tropicalis','Oryzias latipes','Gadus morhua','Ornithorhynchus anatinus']

sra_amalgamated = pandas.read_csv(file_sra_amalgamated, sep='\t')
piv = sra_amalgamated.loc[:,['scientific_name','exclusion','run']].groupby(['scientific_name','exclusion']).count().unstack().fillna(0).T.astype(int)
piv = piv.loc[:,sorted(spp)]
piv.to_csv('sample_exclusion.tsv', sep='\t', index=True)

#seaborn.barplot(x='exclusion',)_____no_output_____
</code>