# Notebook from bmcandr/website
Path: book/tutorials/microstructure/microstructure-tutorial.ipynb

# Microstructure Tutorial

by Mike Durand, School of Earth Sciences and Byrd Polar & Climate Research Center durand.8@osu.edu

## Learning Objectives

At the end of this tutorial you should be able to...

* Explain why microstructure is important for remote sensing
* Define measures of microstructure, especially specific surface area
* Access and visualize tree different microstructure measurements from SnowEx Grand Mesa 2020

## Acknowledgments

Contributions from: Micah Johnson, Mike Durand, HP Marshall, Tate Meehan, Megan Mason, Scott Henderson. This relies heavily on the snowexsqul database and example scripts created by Micah Johnson.

## Caveats

The integrating sphere and the SMP data are published at NSIDC; you can read the pages there for documentation etc. However the microCT data are not yet published; please contact Lauren Farnsworth (lauren.b.farnsworth@usace.army.mil) for questions on the CT data.

## Fun Facts About Snow Microstructure

Snow microstructure plays a super important role in snow physics and snow remote sensing, so a lot of effort went towards measuring it in SnowEx 2020!

There are several different quantities that are used to measure snow microstructure, including "grain size". Grain size measurements are challenging to make in a repeatable way, and are also challenging to relate to the physical quantities that control remote sensing measurements. In the last ~15 years or so, a lot of effort has gone into more objective ways to measure microstructure.

Snow microstructure  governs response of remote sensing to snow cover for visible, near-infrared and high-frequency microwave wavelengths. See Figure 1, below, and read {cite:p}`Dozier2009`, for more information.

Snow microstructure governs visible and near-infrared reflectance. This is figure 2 from {cite:p}`Dozier2009`

Radar measurements such as those made by the Ku-band SWEARR instrument are also very sensitive to snow microstructure.

Modeled response of radar backscatter to SWE and single-scatter abledo (which in turn is a function of snow microstructure), based on a simple model suggested by {cite:p}`Ulaby2014`

Snow microstructure is super important to efforts to launch a Ku-band SAR to measure global snow water equivalent (SWE). An important area of research right now is exploring how to use estimates of microstructure (e.g. from snowpack evolution models) to improve SWE retrievals.

Snow microstructure evolves through the season, and varies a lot with depth. Snow microstructure evoluation is controlled by other snow properties, such as snow temperature, snow height and snow liquid water content. A really great resource on snow microstructure is Kelly Elder's recent talks:
* [Snow Metamorphism](https://boisestate.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=9a051fe4-db09-4f1d-92db-acd1003cc5cc)
* [Snow Grain Identification](https://boisestate.hosted.panopto.com/Panopto/Pages/Embed.aspx?id=136f7580-c1e2-4c55-ab33-acca003c23d6)_____no_output_____## SnowEx Microstructure Measurement Background

### Basic Microstructure Definitions

Microstructure definitions take a bit of getting used to. It's very easy to get confused. Specific surface area (SSA) is one of the most important quantity used to measure snow microstructure, so that's the focus here. Note that SSA is not the be-all and end-all, so there's a short table describing how to relate SSA to other quantities just below.  A couple of good reads on all of this is {cite:p}`Matzler2002` and {cite:p}`Matzl2010`.

Coarse and fine snow microstructure revealed by microCT. The microCT snow renderings on the left are Figure 2 from {cite:p}`Lowe2011`. The colorbars indicate that fine-grained snow (a) has high SSA and low Deq, whereas coarse-grained snow (b) has low SSA and high Deq.

Use Figure 3 above to ground these definitions: SSA is the surface area of the ice-air interface, normalized in some way. Confusingly, SSA is defined in a couple of different ways in the literature: sometimes, surface area within a particular volume of interest (VOI) is normalized by the mass of the ice in the VOI. As defined in this way, SSA has units of length squared per mass, usually expressed as m2/kg. Instead of normalizing by mass, SSA is sometimes defined by normalizing by the volume of the VOI (this is SSAv in {cite:p}`Matzl2010`), and sometimes by normalizing by the volume of the ice in the VOI (this is SSAi in {cite:p}`Matzl2010`, and q in {cite:p}`Matzler2002`). Here let's just go with the first definition I mentioned:

$$
SSA = \frac{\text{Surface area of ice-air interface}}{\text{Mass of ice}} \quad
$$

SSA tends to take values between 5 and 150 m2/kg: fresh, fine-grained snow has high SSA, and coarse snow has low SSA. Because it takes a little while for SSA values to become intuitive, a useful derived metric is the equivalent grain diameter (Deq; note that this is identical to Dq in {cite:p}`Matzler2002`), which by definition is the diameter that a sphere would have if it had a particular value of SSA. This is a one-to-one relationship, so there are no assumptions involved.

$$
D_{eq} = \frac{6}{SSA \rho_i} \quad
$$ _____no_output_____Relationships of specific surface area to other metrics are given in this list if you're curious but otherwise just skip past this bit
* Sometimes people refer to the "optical grain diameter", which is the same as Deq. The "optical" refers to {cite:p}`Grenfell1999`, who showed that any snow with a particular SSA had similar (not identical) radiative transfer properties regardless of particle shape in the visible and near-infrared parts of the spectrum. But note the same is not true in the microwave spectrum.
* Autocorrelation length is usually one of two metrics that summarize the two-point microstructure autocorelation function of the three-dimensional ice-air matrix. Think of the probability that you change media (from ice to air or vice versa) as you move a certain distance within the snow microstructure. The length that defines the likelihood that you'll change media is (an approximation of the correlation length). SSA is by definition (with almost no assumptions) equal to the slope of the autocorrelation function at the origin. But microwave scattering is controlled by correlations at longer lags. For more check out {cite:p}`Matzler2002`. The lack of closing the loop between SSA and correlation length is a significant issue when we have measurements of SSA and microwaves as we do in SnowEx.
* Geometric grain size is what we usually measure when we measure with a hand lens. You can try to relate it to SSA or corelation length, but it is not always possible, and will change with different observers.

Time to stop this list but there are many other metrics as well._____no_output_____### Microstructure Instruments

Now that we know what we're trying to measure (SSA, or correlation length) how do we actually measure? let's talk just about three techniques used in SnowEx 2020 Grand Mesa.

Left: Lauren Farnsworth transports microCT samples from field sites back to Grand Mesa Lodge in a cold storage container. Right: the microCT machine in the lab at CRREL.

Micro-computed tomography (microCT) is the only laboratory-based method used here, and it is the gold standard, although it does still come with caveats. The idea of microCT is to remove a sample of snow from a snow pit face, and either cast it with a compound such as diethyl pthalate that is still a liquid at 0° C, or preserve the same at a very cold temperature. Then the sample is sent back to the laboratory, and bombared with x-rays, similar to how you get x-rays to see if a bone is broken at the doctor. For much more on microCT, check out {cite:p}`Heggli2011`. microCT can be used to extract a ton of information about snow microstructure, including SSA, correlation length and many others.

Left: Kehan Yang operates an IceCube unit at the Grand Mesa Lodge intercomparison snowpit. Top right: schematic showing the integrating sphere measurement principle, from {cite:p}`Gallet2009`. Bottom right: snow in the IceCube sampling container, from {cite:p}`Leppanen2018`.

Integrating spheres are field-based and you make the measurements on samples extracted from the snowpit face. The principle of the measurement is based on firing a laser at the snow sample, within a special reflective hollow sphere, where one side is filled by the snow sample, and measuring how much of the laser is reflected at a sensor at a known geometry. For more information, check out {cite:p}`Gallet2009`.  Most integrating sphere measurements are either made by a commercial firm (A2 Photonics) known as the [IceCube](https://a2photonicsensors.com/icecube-ssa/) or a version constructed at the University of Sherbrooke known as the IRIS {cite:p}`Montpetit2012`. These approaches are set up to measure SSA only. There were three of these at Grand Mesa - one of the Sherbrooke IRIS units, and two IceCubes, one from Finnish Meteorological Institute, and one from Ohio State University.

Left: Megan Mason operates the SMP. Right: Closeup of the SMP sensor tip.

Snow micropenetrometers are also a field-based approach, but they do not require a snowpit, enabling far more observations to be made. Instead, an automated motor pushes a probe vertically downwards into the snowpack. The probe measures the force required to break snow microstructure, yielding a wealth of information. Snow density, specific surface area and correlation length can be retrieved; for more results see {cite:p}`Lowe2012` and {cite:p}`Proksch2015`. The micropen effort at Grand Mesa was led by Boise State University.

These methods are not the only ways to measure microstructure! There are several others not mentioned here, but not used at Grand Mesa 2020. Ask if interested.

## SnowEx Microstructure Measurement Data Overview

Of the three methods described above, microCT is by far the most expensive and most time consuming. Samples have to be transported back to the laboratory and the processing time requires a microCT machine. Thus the fewest CT sapmles are taken.

The integrating spheres require a snowpit to be dug, so we have an intermediate number of them: ~100.

The micropen measurements are by far the fastest to make, so a cross pattern of SMP measurements was made on orthogonal directions intersecting at the snowpit. There are thousands of SMP profiles from Grand Mesa 2020._____no_output_____## Working with the data

We're going to do two things! First, we'll intercompare the three different integrating sphere instruments at four different pits where we had multiple instruments operating. We'd expect these data to be fairly self-consistent. Second, we'll compare all three methods (integrating sphere, SMP and microCT) at a single pit where we had all of these measurements present. Here especially with the SMP we would expect to need to intercalibrate the data to match local conditions; so far SSA has only been fit to SMP force measurements in one study, and we should assume we'll need a local calibration to get a tight fit._____no_output_____### 0. Load needed modules_____no_output_____

# Modules needed to access snowexsql: SnowEx field data database
from snowexsql.db import get_db
from snowexsql.data import LayerData, PointData
from snowexsql.conversions import points_to_geopandas, query_to_geopandas

# Modules needed to work with data
import geoalchemy2.functions as gfunc
import geopandas as gpd
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np

import warnings
warnings.filterwarnings('ignore')

#note - this cell does not return any output_____no_output_____

### 1. Intercompare Integrating Sphere Datasets_____no_output_____There were three integrating spheres. The IRIS unit from University of Sherbrooke was operated by Celine Vargel. The IceCube unit from the Finnish Meteorological Institute was operated by Juha Lemmetyinen. And the IceCube unit from Ohio State was operated by Kehan Yang and Kate Hale. Carefully read the [documentation page](https://nsidc.org/data/SNEX20_SSA/versions/1) at NSIDC if you are interested in the data. If you are using the data for a project, please contact the authors and mention what you're doing - they'll appreciate it! Contact for SSA is Mike Durand (durand.8@osu.edu).

See Micah's tutorial on datasets for more on this! Won't explain too much here_____no_output_____

db_name = 'snow:hackweek@52.32.183.144/snowex'
engine, session = get_db(db_name)

# Grab all the equivalent diameter profiles
q = session.query(LayerData).filter(LayerData.type == 'specific_surface_area')
df = query_to_geopandas(q, engine)

# End our database session to avoid hanging transactions
session.close()

df.head() #check out the results of the query_____no_output_____

Since we want to intercompare integrating spheres, we need to isolate only the sites that actually had multiple integrating spheres measuring the same snow._____no_output_____

# Grab all the sites with equivalent diameter data (set reduces a list to only its unique entries)
sites = df['site_id'].unique()

# Store all site names that have mulitple SSA instruments
multi_instr_sites = []
instruments = []

for site in sites:

    # Grab all the layers associated to this site
    site_data = df.loc[df['site_id'] == site]

    # Do a set on all the instruments used here
    instruments_used = site_data['instrument'].unique()

    if len(instruments_used) > 1:
        multi_instr_sites.append(site)

# Get a unqique list of SSA instruments that were colocated
instruments = df['instrument'].unique()

instruments #check out the list of instruments. note that the IceCube values are displayed as serial numbers_____no_output_____

Finally, plot all Integrating Sphere SSA profiles at all Multi-Integrating Sphere Sites_____no_output_____

# Setup the subplot for each site for each instrument
fig, axes = plt.subplots(1, len(multi_instr_sites), figsize=(4*len(multi_instr_sites), 8))

# Establish plot colors unique to the instrument
c = ['k', 'm', 'c']
colors = {inst:c[i] for i,inst in enumerate(instruments)}

# Loop over all the multi-instrument sites
for i, site in enumerate(multi_instr_sites):

    # Grab the plot for this site
    ax = axes[i]

    # Loop over all the instruments at this site
    for instr in instruments:

        # Grab our profile by site and instrument
        ind = df['site_id'] == site
        ind2 = df['instrument'] == instr
        profile = df.loc[ind & ind2].copy()

        # Don't plot it unless there is data
        if len(profile.index) > 0:

            # Sort by depth so samples that are take out of order won't mess up the plot
            profile = profile.sort_values(by='depth')

            # Layer profiles are always stored as strings.
            profile['value'] = profile['value'].astype(float)

            # Plot our profile
            ax.plot(profile['value'], profile['depth'], colors[instr], label=instr)

    # Labeling and plot style choices
    ax.legend()
    ax.set_xlabel('SSA [m^2/kg]')
    ax.set_ylabel('Height above snow-soil interface [cm]')
    ax.set_title('Site {}'.format(site.upper()))

    # Set the x limits to show more detail
    ax.set_xlim((8, 75))

plt.tight_layout()
plt.show()
_____no_output_____

### 2. Pull the snowmicropenetrometer data and compute SSA

The next step is to grab some SMP data to compare to. We're going to get the SMP at site 2N13, where we have a copule of SSA profiles from interating spheres (as well as microCT data, to be looked at in the next step!).

The SMP measurements for SnowEx 2020 GrandMesa were all made by Megan Mason. If you're interested in working with the SMP data, please carefully read the NSIDC [documentation page](https://nsidc.org/data/SNEX20_SMP/versions/1). If you're planning to work with the data, please reach out to the author; the contact is (Megan Mason meganmason491@u.boisestate.edu). If you use a profile, consider checking out the comments which are described in the [Excel sheet linked from the Technical References part of the NSIDC documentation](https://nsidc.org/sites/nsidc.org/files/technical-references/SNEX20_SMP_FieldNotes.xlsx), where there are some really useful comments.

There are a few steps here, and one reason for that is just that the SMP data is quite large, and so the full-resolution SMP could not be included in Micah's database. The full resolution profile from SMP is resolved ever 1.25 mm! Instead, the SMP data in Micah's database is sampled to only every 100th datapoint, so it's every 12.5 cm. But the database is still very useful! What we'll do is use the database to find the right profile, then go and download that full resolution dataset from the NSIDC. Easy-peasey!

First up, we'll visualize the location of the SMP profiles, along with the snowpit location._____no_output_____

site = '2N13'
engine_smp, session_smp = get_db(db_name)
q_smp = session_smp.query(LayerData).filter(LayerData.type == 'force').filter(LayerData.site_id.contains(site) )
df_smp = query_to_geopandas(q_smp, engine_smp)

q_pit=session_smp.query(LayerData).filter(LayerData.type == 'hand_hardness').filter(LayerData.site_id.contains(site) )
df_pit = query_to_geopandas(q_pit, engine_smp)

session_smp.close()

# Plot SMP profile locations with colored by the time they were taken using upside down triangles
ax = df_smp.plot(column='time', cmap='jet', marker='v', label='SMP', figsize=(5,5), markersize=100, edgecolor='black')

ax.plot(df_pit.easting, df_pit.northing, color='black', marker='s', markersize=15, label='Pit ({})'.format(site))

# Add important labels
ax.set_xlabel('Easting [m]')
ax.set_ylabel('Northing [m]')
plt.suptitle('SMP Locations at Site {} Showing Acquisition Order'.format(site), fontsize=16)

# Avoid using Scientific notation for coords.
ax.ticklabel_format(style='plain', useOffset=False)
ax.legend()
# plt.tight_layout()
plt.show()_____no_output_____

Next up, let's find the closest SMP profile to the snowpit, and then find the profile ID of that profile, which is in the comments in the database._____no_output_____

# find closest SMP profile to the pit

# No profile is taken at the same time, so we grab all the unique times and sort them
times = sorted(df_smp['time'].unique())

nprofiles=len(times)

ids=np.empty(nprofiles)

p=0
for t in times:
    ind = df_smp['time'] == t
    data = df_smp.loc[ind].copy()
    ids[p]=data.iloc[0].id
    p+=1

i_dists=df_smp['id'].isin(ids)

df_smp_dists=df_smp.loc[i_dists]
df_smp_dists=df_smp_dists.assign(dists=-1)
df_smp_dists['dists']=np.sqrt((df_smp_dists['easting']-df_pit.iloc[0].easting)**2+(df_smp_dists['northing']-df_pit.iloc[0].northing)**2)

df_smp_dists.sort_values(by='dists')[['comments','dists']].head() #check out the list of profiles sorted by distance to pit_____no_output_____

So the id of the closest SMP profile is S19M1174.  I went to the [SMP page on NSIDC](https://nsidc.org/data/SNEX20_SMP/versions/1), and went to "Download" and searched for this ID, downloaded the profile, and then re-uploaded to my home directory here in the Jupyter hub.

Ok next up, we have to compute SSA from the SMP data. For this, we'll use the "snowmicropyn" modules created by the Swiss SLF. You can read more about them [at this site](https://snowmicropyn.readthedocs.io/en/latest/). The software is a little out of date on Python versions; just ignore any warnings that pop up below!

The next cell pulls in the needed modules, and then plots the profile of force measurements needed to break through the snow microstructure._____no_output_____

from snowmicropyn import Profile
from snowmicropyn import proksch2015_____no_output_____# Pull in some tutorial datasets
!aws s3 sync --quiet s3://snowex-data/tutorial-data/microstructure/ /tmp/microstructure_____no_output_____p = Profile.load('/tmp/microstructure/SMP/SNEX20_SMP_S19M1174_2N13_20200206.PNT',)
plt.plot(p.samples.distance, p.samples.force)
# Prettify our plot a bit
plt.title(p.name)
plt.ylabel('Force [N]')
plt.xlabel('Depth [mm]')
plt.show()_____no_output_____

The next step is the actual calculation of SSA from the force data. It then displays the data and lets you see that there is now a column called SSA! Note that this function is "proksch2015". You can read about how it works in Martin Proksch's paper {cite:p}`Proksch2015`._____no_output_____

# the window size is a parameter you can play with. default is 2.5 mm setting to 10 mm does some smoothing
p2015 = proksch2015.calc(p.samples,window=10)
p2015.head() #check out the first few values of SSA_____no_output_____

### 3. Read microCT data, and compare intergrating sphere, SMP and CT data_____no_output_____The microCT samples were extracted in the field and processed at CRREL by Lauren Farnsworth, and is not yet published at NSIDC. Please contact her with questions (lauren.b.farnsworth@usace.army.mil)!

This module reads in microCT datafiles which are stored as text. Some additional data are available, showing the computer generated slices through the ice-air interface: contact Mike (durand.8@osu.edu) if you want to look at a subset of these data that Lauren has shared.

Equivalent grain size is a useful quantity to compare: because it's proportional to 1/SSA, and because after a point as you increase SSA more and more, all fine-grained snow acts more-or-less the same (converging to e.g. the "fine-grained" curve in Figure 1, above), we'll look at equivalent diameter instead of SSA in this comparison._____no_output_____

from read_CT_txt_files import read_CT_txt_files_____no_output_____# read micro CT for 2N13
data_dir='/tmp/microstructure/microCT/txt/'
[SSA_CT,height_min,height_max]=read_CT_txt_files(data_dir)

SSA_CT #chck out the SSA values read in from MicroCT_____no_output_____# get data integrating sphere data for 2N13 and plot it
site='2N13'
engine_is, session_is = get_db(db_name)
q_is = session_is.query(LayerData).filter(LayerData.type == 'specific_surface_area').filter(LayerData.site_id.contains(site) )
df_is = query_to_geopandas(q_is, engine_is)
instruments_site = df_is['instrument'].unique()

# Loop over all the integrating sphere instruments at this site. plot equivalent diameter
fig,ax = plt.subplots()
for instr in instruments_site:

    # Grab our profile by site and instrument
    ind = df['site_id'] == site
    ind2 = df['instrument'] == instr
    profile = df.loc[ind & ind2].copy()

    # Don't plot it unless there is data
    if len(profile.index) > 0:

        # Sort by depth so samples that are take out of order won't mess up the plot
        profile = profile.sort_values(by='depth')

        # Layer profiles are always stored as strings.
        profile['value'] = 6/917/profile['value'].astype(float)*1000

        # Plot our profile
        ax.plot(profile['value'], profile['depth'], colors[instr], label=instr)

#All that's left to do is plot the CT and the SMP and label the plot!
ax.plot(6/917/SSA_CT*1000,height_min,label='microCT')        #CT data

ax.plot(6/917/p2015.P2015_ssa*1000,(max(p2015.distance)-p2015.distance)/10,label='SMP') #SMP data

# Labeling and plot style choices
ax.legend()
ax.set_xlabel('Equivalent diameter, mm')
ax.set_ylabel('Height above snow-soil interface [cm]')
ax.set_title('Site {}'.format(site.upper()))

plt.tight_layout()
plt.show()_____no_output_____

Wow, so the datasets are so very different, with the SMP being by far the most different. Because we might expect that the SMP is quite a ways off, actually, as the relationships between the force profile and SSA are empirical, and thus might change for different types of snow. There's also significant differences between the microCT and the two integrating spheres. This is science - sometimes when we start intercomparing these quantities, we do not get a perfect match. This data hasn't had kinks worked out yet - this is all hot off the press and ready for HackWeek!_____no_output_____Some of the ways that you could imagine connecting microstructure measurements to other quantities would be with the SWESARR radar data. Although the radar data does seem to have some orthorectification issues that haven't been fully worked out, I can imagine these being worked around by careful choice of places you match up the microstructure to the radar. Note that places that are shallower tend to have larger Deq and vice versa, and the spatial variability in SSA was fairly low in general in Grand Mesa 2020, so looking at multiple SSA vs radar samples might not yield a great correlation. But you never know, could be fun to try! Generaly speaking, we don't expect a ton of impact of the microstructure on L-band (UAVSAR), but it would be interesting to explore that.

One thing that could be of great value is to calibrate the SMP estimates of SSA to the integrating spheres. If you're interested in doing that, do reach out first. This could be a really interesting thing to explore!

It might also be interesting to compare the data to hand hardness measured in the snowpit, and to traditional hand lens measurements. _____no_output_____


# Notebook from zomansud/ml-graphlab-boilerplate
Path: notebooks/03 - Analyzing Product Sentiment.ipynb

import os, sys
import graphlab as gl
import graphlab.aggregate as agg
from tqdm import tqdm_notebook as tqdm

# set canvas path
# gl.canvas.set_target('ipynb')

%matplotlib inline
import matplotlib
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as snsThis non-commercial license of GraphLab Create for academic use is assigned to sudhanshu.shekhar.iitd@gmail.com and will expire on August 27, 2018.

# Read some product review data_____no_output_____

products = gl.SFrame('data/amazon_baby.gl/')_____no_output_____

# Explore this data_____no_output_____

products.head()_____no_output_____

# Build a word count vector for each review_____no_output_____

products['word_count'] = gl.text_analytics.count_words(products['review'])_____no_output_____products.head()_____no_output_____

# Explore Villi Sophie_____no_output_____

giraffe_reviews = products[products['name'] == 'Vulli Sophie the Giraffe Teether']_____no_output_____len(giraffe_reviews)_____no_output_____sns.countplot(x="rating", data=products.to_dataframe())_____no_output_____

# Build a sentiment classifier_____no_output_____## Define what a positive and negative sentiment_____no_output_____

#ignore all 3 star reviews_____no_output_____products = products[products['rating'] != 3]_____no_output_____len(products)_____no_output_____#positive sentiment is 4/5 star ; negative sentiment is 1/2 star_____no_output_____products['sentiment'] = products['rating'] >= 4_____no_output_____products.head()_____no_output_____

## Lets train the sentiment classifier_____no_output_____

train_data, test_data = products.random_split(0.8, seed=0)_____no_output_____sentiment_model = gl.logistic_classifier.create(train_data,
                                                      target='sentiment',
                                                      features=['word_count'],
                                                      validation_set=test_data)_____no_output_____

## Evaluate the sentiment model_____no_output_____

sentiment_model.evaluate(test_data, metric='roc_curve')_____no_output_____

# Apply the learned model to understand the sentiment for giraffe_____no_output_____

giraffe_reviews['predicted_sentiment'] = sentiment_model.predict(giraffe_reviews, output_type='probability')_____no_output_____giraffe_reviews.head()_____no_output_____

## Sort the reviews based on the predicted sentiment and explore_____no_output_____

giraffe_reviews = giraffe_reviews.sort(
    'predicted_sentiment',
    ascending=False
)_____no_output_____giraffe_reviews.head()_____no_output_____giraffe_reviews[0]['review']_____no_output_____giraffe_reviews[1]['review']_____no_output_____

## Show some negative reviews_____no_output_____

giraffe_reviews[-1]['review']_____no_output_____giraffe_reviews[-2]['review']_____no_output_____


# Notebook from jgvhabets/brady_reallife
Path: code/brady_2020_descriptives_class-results_figures.ipynb

# load packages
import itertools
from itertools import compress
import pdb

import time
from datetime import timedelta, datetime
from os.path import join
import os
from os import listdir
from os.path import isfile, join
import csv
from timeit import default_timer as timer

import numpy as np
import numpy.matlib
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import figure
from matplotlib.ticker import PercentFormatter
%matplotlib inline

import statsmodels.api as sm
from statsmodels.formula.api import ols
import scipy
from scipy.signal import welch, decimate, periodogram, find_peaks
from scipy import signal
from scipy import stats
from scipy.stats import pearsonr, mannwhitneyu, spearmanr, ranksums, ttest_ind, f_oneway
from scipy.ndimage.filters import uniform_filter1d
from statsmodels.stats.anova import AnovaRM
from statsmodels.stats.multitest import multipletests
from statsmodels.multivariate.manova import MANOVA
from statistics import mode
import math
import random
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler , MinMaxScaler
from sklearn.kernel_ridge import KernelRidge
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier, GradientBoostingClassifier
from sklearn import tree
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.svm import LinearSVC, SVC
from sklearn.model_selection import LeaveOneGroupOut, cross_val_predict, cross_val_score, KFold, train_test_split, StratifiedKFold, GridSearchCV
from sklearn.model_selection import permutation_test_score
from sklearn.metrics import roc_auc_score, roc_curve, classification_report, confusion_matrix, accuracy_score
from sklearn.metrics import precision_score, recall_score, auc, f1_score, precision_recall_curve
from sklearn.manifold import MDS
import seaborn as sns

import sys_____no_output_____# # Version check of used packages
print('Pandas:',pd.__version__)
print('Numpy:',np.__version__)
print('SciPy:',scipy.__version__)
# Pandas: 1.1.3
# Numpy: 1.19.1
# SciPy: 1.5.2Pandas: 1.1.3
Numpy: 1.19.1
SciPy: 1.5.2
# define path, working directory which contains filtered accelerometer files

# path="/Users/jeroenhabets/Desktop/radboud data/data/"
notebook_path = os.getcwd()
path = os.path.dirname(notebook_path)
# images_path = "/Users/jeroenhabets/Starr Lab Dropbox/jeroen habets/PHD werkmap/UCSF time/Sensor-project/analysis images/"

os.chdir(path)
print('Path and working directory defined as %s' %path)
Path and working directory defined as /Users/jeroenhabets/Research/pd_sensors/brady_reallife

# Real-life ON- vs OFF-bradykinesia detection using a wrist-accelerometer  - Patient population description and classification results (incl. significance and clinical correlations)

Prep: Loading in extracted accelerometer bradykinesia features (one value/ 60 seconds, medState 0 is pre-medication, 1 is post-medication), and outcomes from classification analyses.

Part 1: Descriptive statistics of population.

Part 2: Analyzing population for differences in medication-states based on 4 main features, using MANOVA + post-hoc ANOVA analyses.

Part 3a: Comparing classification results with barplots (varying models and data inclusion approaches).

Part 3b: Comparing classification results with equivalent plots.

Part 4: Correlating predictive performance with clinical characteristics.
_____no_output_____## Preparation: Load clinical info, features, and classification reults
Previsouly filtered data is loaded in to dataframes per patient-side._____no_output_____

def loadUPDRSscores():
    '''
    Input:
    Load in existing updrsScores table

    Returns:
    - updrsScores: dataframe with clinical scores of all patients

    PM: Differences are calculated OFF minus ON, which means clinical IMPROVEMENT (decrease in UPDRS in ON
    compared to OFF) results in POSITIVE diff-scores. (A negative diff-score means a clinical worsening
    with a higher UPDRS-score in ON compared to OFF)

    '''
    # Read in updrs file
    updrsScores = pd.read_csv(os.path.join(path,'data','updrsScores.csv'))
    # convert int to strings for PtId's
    updrsScores['PtId'] = updrsScores['PtId'].astype(str)
    # add missing zeros in front of PtId
    for row in np.arange(updrsScores.shape[0]):
        updrsScores.at[row,'PtId'] = updrsScores.loc[row]['Record Id'][3:]
    del(updrsScores['Record Id'])
    updrsScores = updrsScores.set_index('PtId')

    # create list for ON and OFF for subscore-lists
    OFF_sublists = {
        'leftHandBradyOFF': ['OFF_UPDRS_3_3c','OFF_UPDRS_3_4b','OFF_UPDRS_3_5b','OFF_UPDRS_3_6b',],
        'rightHandBradyOFF': ['OFF_UPDRS_3_3b','OFF_UPDRS_3_4a','OFF_UPDRS_3_5a','OFF_UPDRS_3_6a',],
        'bradyBodyOFF': ['OFF_UPDRS_3_14'],
        'leftHandTremorOFF': ['OFF_UPDRS_3_15b','OFF_UPDRS_3_16b','OFF_UPDRS_3_17b',],
        'rightHandTremorOFF': ['OFF_UPDRS_3_15a','OFF_UPDRS_3_16a','OFF_UPDRS_3_17a',],
        'legsOFF': ['OFF_UPDRS_3_7a','OFF_UPDRS_3_7b','OFF_UPDRS_3_8a','OFF_UPDRS_3_8b'],
        'gaitOFF': ['OFF_UPDRS_3_9','OFF_UPDRS_3_10','OFF_UPDRS_3_11'],
        'postureOFF': ['OFF_UPDRS_3_12','OFF_UPDRS_3_13'],
        'facialOFF': ['OFF_UPDRS_3_1','OFF_UPDRS_3_2',]}

    ON_sublists = {
        'leftHandBradyON': ['ON_UPDRS_3_3c', 'ON_UPDRS_3_4b','ON_UPDRS_3_5b','ON_UPDRS_3_6b',],
        'rightHandBradyON': ['ON_UPDRS_3_3b','ON_UPDRS_3_4a','ON_UPDRS_3_5a','ON_UPDRS_3_6a',],
        'bradyBodyON': ['ON_UPDRS_3_14'],
        'leftHandTremorON': ['ON_UPDRS_3_15b','ON_UPDRS_3_16b','ON_UPDRS_3_17b',],
        'rightHandTremorON': ['ON_UPDRS_3_15a','ON_UPDRS_3_16a','ON_UPDRS_3_17a',],
        'legsON': ['ON_UPDRS_3_7a','ON_UPDRS_3_7b','ON_UPDRS_3_8a','ON_UPDRS_3_8b'],
        'gaitON': ['ON_UPDRS_3_9','ON_UPDRS_3_10','ON_UPDRS_3_11'],
        'postureON': ['ON_UPDRS_3_12','ON_UPDRS_3_13'],
        'facialON': ['ON_UPDRS_3_1','ON_UPDRS_3_2',]}

    for l,(off, on) in enumerate(zip(OFF_sublists.keys(),ON_sublists.keys())):
        offList = []
        diffList = []
        relative_diffList = []
        for pt in updrsScores.index:
            offList.append(np.sum(updrsScores.loc[pt][OFF_sublists[off]]))
            diffScore = np.sum(updrsScores.loc[pt][OFF_sublists[off]]) - np.sum(updrsScores.loc[pt][ON_sublists[on]]) # calculate difference between off and on scores
            diffList.append(diffScore)
            relative_diffList.append(diffScore / (len(OFF_sublists[off])*4)) # take sum-difference, normalize to list-length by dividing by potential total score
        # create name for column
        off_colName = off[:-3] + '_off'
        rel_colName = off[:-3] + '_%diff'
        colName = off[:-3] + '_diff'
        updrsScores.insert(loc=l*2, value=diffList, column=colName)
        updrsScores.insert(loc=l*2 +1, value=relative_diffList, column=rel_colName)
        updrsScores.insert(loc=l*2 +2, value=offList, column=off_colName)

    # indicate which side had more arm-bradykinesia fluctuation
    affectedSides = []
    for pt in updrsScores.index:
        if updrsScores.loc[pt]['leftHandBrady_diff'] > updrsScores.loc[pt]['rightHandBrady_diff']:
            affectedSides.append('Left')
        else:
            affectedSides.append('Right')
    updrsScores.insert(loc=0, value=affectedSides, column='Side')
    # create uniform columns for bradykinesia and tremor on included side for analysis
    for symp in ['Tremor','Brady']:
        for var in ['_off','_diff','_%diff']:
            updrsScores['inclSideHand%s%s' % (symp,var)] = [float(99)]*updrsScores.shape[0]
    # fill in with values from correct sides
    for symp in ['Tremor','Brady']:
        for var in ['_off','_diff','_%diff']:
            for pt in updrsScores.index:
                if updrsScores.loc[pt]['Side'] == 'Left':
                    updrsScores.at[pt,'inclSideHand%s%s' % (symp,var)] = updrsScores.loc[pt]['leftHand%s%s' % (symp,var)]
                elif updrsScores.loc[pt]['Side'] == 'Right':
                    updrsScores.at[pt,'inclSideHand%s%s' % (symp,var)] = updrsScores.loc[pt]['rightHand%s%s' % (symp,var)]

    # selecting patient with handBrady improvement on vs off, and enough data
    for pt in updrsScores.index:
        if updrsScores.loc[pt]['inclSideHandBrady_diff'] < 0.5:
            updrsScores = updrsScores.drop(labels=[pt], axis=0)
    for pt in ['022','080']: # removing 022 and 080 due to not enough data for holdout
        updrsScores = updrsScores.drop(labels=[pt], axis=0)

    return updrsScores_____no_output_____def loadFeatures(minBradyDiff, updrsScores, windowLen=60):
    '''
    Input:
    - minBradyDiff = minimal difference in brady-updrs-subscores to include in analysis
    - updrsScores: dataframe with clinical scores of all patients

    Returns:
    - accData: one dictionary with accData, each patient has a dictionary per side,
    containing pre and post session.

    '''

    # select patients to involve in analysis
    selectedIDs = []
    for pt in updrsScores.index:
        if updrsScores.loc[pt]['leftHandBrady_diff'] > minBradyDiff:
            selectedIDs.append(pt)
        elif updrsScores.loc[pt]['rightHandBrady_diff'] > minBradyDiff:
            selectedIDs.append(pt)
        else:
            continue
    # removing 022 and 080 if included based on clinical difference, because data is not large enough for holdout
    for pt in ['022','080']:
        if pt in selectedIDs:
            selectedIDs.remove(pt)
    # dictionary for all acc data
    features = {}
    for pt in selectedIDs:
        features[pt] = pd.read_csv(os.path.join(
            path,'data','features_oct20','%s_%isec_features.csv' % (pt,windowLen)))

    return features
        _____no_output_____updrsScores = loadUPDRSscores()
features = loadFeatures(minBradyDiff=0.5, updrsScores=updrsScores, windowLen=60)_____no_output_____def create_result_dicts():
    '''Creates all different outcome dictionaries'''
    dictout = {}
    resultsfiles = listdir(os.path.join(path,'results'))
    for model in ['indiv','group']:
        for cls in ['SV','RF']:
            sel = [np.logical_and(model in name,cls in name) for name in resultsfiles]
            mod_files = list(compress(resultsfiles, sel))
            for fts in ['allfts','4fts']:
                sel = [fts in name for name in mod_files]
                sel_files = list(compress(mod_files,sel))
                for act_filter in [True,False]:
                    if act_filter:
                        sel = ['ACTF' in name for name in sel_files]
                        file = list(compress(sel_files,sel))
                        if len(file) == 1:
                            file = file[0]
                            dat = pd.read_csv(os.path.join(path,'results',file), index_col=0)
                            if model == 'indiv': # indiv models need mean,sd calculation over 41 splits
                                temp = pd.DataFrame(index=['pred'])
                                for col in dat.keys():
                                    temp.at['pred',col] = np.mean(dat[col])
                                    temp.at['pred',col+'_sd'] = np.std(dat[col])
                                dat = temp
                            dictout[model+'_'+cls+'_'+fts+'_actfilter'] = dat.append(pd.Series(name='p', dtype='object')) # add empty row for p
                        else:
                            print('No or multiple files for combination:%s, %s, %s, %s' %(model,cls,fts,act_filter))
                    elif act_filter == False:
                        sel = ['ACTF' not in name for name in sel_files]
                        file = list(compress(sel_files, sel))
                        if len(file) == 1:
                            file = file[0]
                            dat = pd.read_csv(os.path.join(path,'results',file), index_col=0)
                            if model == 'indiv': # indiv models need mean,sd calculation over 41 splits
                                temp = pd.DataFrame(index=['pred'])
                                for col in dat.keys():
                                    temp.at['pred',col] = np.mean(dat[col])
                                    temp.at['pred',col+'_sd'] = np.std(dat[col])
                                dat = temp
                            dictout[model+'_'+cls+'_'+fts] = dat.append(pd.Series(name='p', dtype='object')) # add empty row for p
                        else:
                            print('No or multiple files for combination:%s, %s, %s, %s' %(model,cls,fts,act_filter))

    ### ADD SIGNIFIANCE TESTING WHEN PERMUTATIONS ARE READY!
    permfilelist = listdir(os.path.join(path,'results/perms')) #enter perm folder
    for model in dictout.keys():
        if model+'_5000perms.csv' in permfilelist: # go further for models with perm-file
            prm = pd.read_csv(os.path.join(path,'results/perms',model+'_5000perms.csv'), index_col=0)
            for col in dictout[model].keys():
                if col[-2:] != 'sd': # exclude sd columns
                    v = dictout[model].loc['pred'][col] # determine value of pt+metric
                    p = sum(prm[col]>v)/len(prm[col]) # calculate part of perms higher
                    dictout[model].at['p',col] = p # add p value to outcome dataframe

    return dictout
_____no_output_____dictout = create_result_dicts()
# to check dict out shapes
keys=list(dictout.keys())
for key in keys:
    print(key,dictout[key].shape)indiv_SV_allfts_actfilter (2, 240)
indiv_SV_allfts (2, 240)
indiv_SV_4fts_actfilter (2, 240)
indiv_SV_4fts (2, 240)
indiv_RF_allfts_actfilter (2, 240)
indiv_RF_allfts (2, 240)
indiv_RF_4fts_actfilter (2, 240)
indiv_RF_4fts (2, 240)
group_SV_allfts_actfilter (2, 120)
group_SV_allfts (2, 120)
group_SV_4fts_actfilter (2, 120)
group_SV_4fts (2, 120)
group_RF_allfts_actfilter (2, 120)
group_RF_allfts (2, 120)
group_RF_4fts_actfilter (2, 120)
group_RF_4fts (2, 120)

## Part 1: Descriptives of patient population_____no_output_____

clin_vars = ['totalUPDRS3_On', 'totalUPDRS3_Off', 'totalUPDRS3_Diff',
             'inclSideHandBrady_off',  'inclSideHandTremor_off',
             'inclSideHandBrady_diff',  'inclSideHandTremor_diff',
       'bradyBody_diff','legs_diff', 'gait_diff', 'posture_diff', 'facial_diff',]
with open('results/updrs_descriptives_n%i.txt' % updrsScores.shape[0], 'w') as f:
            print('UPDRS Descriptives\n\n', file=f)
print('Number of included patients: %i' % updrsScores.shape[0])
print()
for var in clin_vars:
    print(var)
    print('Mean: %.2f (sd: %.2f)' % (np.mean(updrsScores[var]),np.std(updrsScores[var])))
    print('Min: %.2f, max: %.2f' % (np.min(updrsScores[var]),np.max(updrsScores[var])))
    print()
    with open('results/updrs_descriptives_n%i.txt' % updrsScores.shape[0], 'a') as f:
        print('%s:\nMean: %.2f (sd: %.2f)\n' %
              (var,np.mean(updrsScores[var]),np.std(updrsScores[var])), file=f)
        Number of included patients: 20

totalUPDRS3_On
Mean: 27.10 (sd: 9.60)
Min: 11.00, max: 42.00

totalUPDRS3_Off
Mean: 43.75 (sd: 11.63)
Min: 29.00, max: 67.00

totalUPDRS3_Diff
Mean: 16.65 (sd: 8.57)
Min: -4.00, max: 38.00

inclSideHandBrady_off
Mean: 8.85 (sd: 2.26)
Min: 6.00, max: 13.00

inclSideHandTremor_off
Mean: 3.85 (sd: 2.67)
Min: 0.00, max: 9.00

inclSideHandBrady_diff
Mean: 3.90 (sd: 1.95)
Min: 1.00, max: 7.00

inclSideHandTremor_diff
Mean: 2.10 (sd: 2.32)
Min: 0.00, max: 9.00

bradyBody_diff
Mean: 0.85 (sd: 0.57)
Min: 0.00, max: 2.00

legs_diff
Mean: 0.85 (sd: 3.09)
Min: -6.00, max: 6.00

gait_diff
Mean: 0.95 (sd: 0.97)
Min: 0.00, max: 3.00

posture_diff
Mean: 0.45 (sd: 1.12)
Min: -1.00, max: 3.00

facial_diff
Mean: 1.15 (sd: 0.91)
Min: -1.00, max: 3.00

## Part 2: Differences in 4 candidate movement features between pre- and post-med states

Analyze variance in features between pre and post-medication features.

Standardised features will be used, just as in predictive analysis. Features are standardised individually, only pre-medication features are used as reference for the standardisation function.

4 Main features pragmatically chosen based on importance in literature. Variance analysis will be conducted with multi-variate Analysis Of Variance (MANOVA), and 4 seperate repeated measures ANOVA as post-hoc test.
_____no_output_____### Standardisation, normalisation of features_____no_output_____

def zScoreFeatures(features, scaling):

    '''
    Features is dict with feature dataframe for every patient separate. Inlcudes all on and off features per minute.
    '''
    ## Analyze z-scored features, on vs off
    zFeats = {}
    firstPt = list(features.keys())[0]
    allFts = [f for f in features[firstPt].keys().tolist() if f != 'medState']
    ## Empirical choise of 3 main features:
    mainFts = ['SVM_maxAcc','SVM_coefVar','SVM_RMS','SVM_specPow_totalu4Hz']

    z_total = pd.DataFrame(columns = allFts)
    z_pre = pd.DataFrame(columns = allFts)
    z_post = pd.DataFrame(columns = allFts)
    group_ft = pd.DataFrame(columns = allFts)

    list_pt = []

    for pt in list(features.keys()):

        # minSize is shortest length of samples(rows) per pt-side of dataframe, on or off samples
        minSize = min(sum(features[pt]['medState']==0),sum(features[pt]['medState']==1))
        # create predData per pt, per side, with balanced rows pre and post med (minSize)
        predData = pd.concat([features[pt][features[pt]['medState']==0][:minSize],features[pt][features[pt]['medState']==1][:minSize]],axis=0,sort=False).reset_index(drop=True)
        # add un-standarized features to group_z, standarization follows later on full group
        group_ft = pd.concat([group_ft,predData],axis=0,sort=False).reset_index(drop=True)
        # collect pt-numbers
        list_pt.extend([pt]*minSize*2)

        # standarize data or normalize data
        if scaling == 'Stand':
            scaler = StandardScaler()
        elif scaling == 'Norm':
            scaler = MinMaxScaler()

        if scaling != 'None':
            scaler.fit(predData[predData['medState']==0][allFts]) # standardized only on pre-medication data
    #         scaler.fit(predData[allFts]) # use pre and post data for standarization (total mean=0, sd=1)

            # create x and y dataset for this patient
            x = pd.DataFrame(data = scaler.transform(predData[allFts]), columns = allFts)
            y = predData['medState']
            # adding x and y again to one df
            x['medState'] = y
            zFeats[pt] = x
            z_total = pd.concat([z_total,x],axis=0,sort=False).reset_index(drop=True)
            z_pre = pd.concat([z_pre,x[x['medState']==0]],axis=0,sort=False).reset_index(drop=True)
            z_post = pd.concat([z_post,x[x['medState']==1]],axis=0,sort=False).reset_index(drop=True)

        elif scaling == 'None':
            z_total = pd.concat([z_total,predData],axis=0,sort=False).reset_index(drop=True)

    z_total['pt'] = list_pt
    print(z_total.shape, z_pre.shape, z_post.shape)

    return z_total, z_pre, z_post, zFeats
    _____no_output_____mainFts = ['SVM_maxAcc','SVM_coefVar','SVM_RMS','SVM_specPow_totalu4Hz']
z_total, z_pre, z_post, zFeatsPt = zScoreFeatures(features, 'Stand')
n_total, n_pre, n_post, nFeatsPt = zScoreFeatures(features, 'Norm')
# raw_total,_,_,_ = zScoreFeatures(features, 'None')
(2380, 105) (1190, 104) (1190, 104)
(2380, 105) (1190, 104) (1190, 104)

### MANOVA - Multifactorial Analysis of Variance
To test if the 20 patients differ in pre-medication and post-medication state based on 4 main features which are averaged per patient, per medication state.
Features are standardised individually. Only pre-medication data is used to fit standardisation formula._____no_output_____

def manova(z_or_n, save):
    '''
    # Literature on MANOVA:
    # https://www.marsja.se/python-manova-made-easy-using-statsmodels/
    # https://www.real-statistics.com/multivariate-statistics/multivariate-analysis-of-variance-manova/two-way-manova/
    # https://github.com/statsmodels/statsmodels/issues/6464
    # https://statistics.laerd.com/spss-tutorials/two-way-manova-using-spss-statistics.php

    For unbalanced groups" `typ=3`. Type 3 sums of squares (SS) is
    recommended for an unbalanced design for multifactorial ANOVA.
    ols = Ordinary Least Squares (OLS) model
    error: https://stackoverflow.com/questions/53489106/statsmodels-ols-inf-or-nan-error-when-there-is-none-in-dataset'''
### Create (M)ANOVA table in long data form with mean values per patient
    if z_or_n == 'std':
        z_or_n = z_total # define to use stand. or norm. data
    elif z_or_n == 'norm':
        z_or_n = n_total # define to use stand. or norm. data
    anova_data = pd.DataFrame(columns=['pt','medState']+mainFts)
    dat={}
    for ft in mainFts:
        dat['pt']=[]
        dat['medState']=[]
        dat[ft]=[]
        for p,pt in enumerate(features.keys()):
            for med in [0,1]:
                dat['pt'].append(p)
                dat['medState'].append(med)
                m = np.mean(
                    z_or_n[np.logical_and(
                        z_or_n['pt']==pt,z_or_n['medState']==med)]
                    [ft])
                dat[ft].append(m)
    for col in anova_data.keys():
        anova_data[col] = dat[col]

# anova_data.to_csv(os.path.join(path,'manova_data.csv'))
    maov = MANOVA(endog= anova_data[mainFts], exog=anova_data['medState']) #
    print(maov.mv_test())
    if save:
        with open('results/manova_table.txt', 'w') as f:
            print(maov.mv_test(), file=f)
    return anova_data   _____no_output_____anova_data = manova('std',False)

                 Multivariate linear model
============================================================

------------------------------------------------------------
           x0           Value  Num DF  Den DF F Value Pr > F
------------------------------------------------------------
          Wilks' lambda 0.3878 4.0000 36.0000 14.2080 0.0000
         Pillai's trace 0.6122 4.0000 36.0000 14.2080 0.0000
 Hotelling-Lawley trace 1.5787 4.0000 36.0000 14.2080 0.0000
    Roy's greatest root 1.5787 4.0000 36.0000 14.2080 0.0000
============================================================

### Post-Hoc Analysis
1: Repeated Measures ANOVA to find individual p-values of 4 features  leading to signicicant difference in MANOVA.

2: Correct p-value for multiple comparison with False Detection Rate. _____no_output_____

def posthoc(anova_data, save):
    '''
    Analyse single features with repeated measures ANOVA (equal to t-test)
    to find which features causes signifcant difference found in MANOVA.
    '''
    if save:
    # to write text result in to a text-file
        with open('results/posthoc_rm_anova.txt', 'w') as f:
            print('Post-Hoc analysis after MANOVA, with repeated measures ANOVA (4 features)\n', file=f)

    for ft in mainFts:
        rm1way = AnovaRM(anova_data, ft, 'pt',within=['medState'])
        # AnovaRM: first data, then independet var, subject ident, within=independet var
        res = rm1way.fit()
        if save:
            with open('results/posthoc_rm_anova.txt', 'a') as f:
                print(ft,':', file=f)
                print(res, file=f)
        print(ft,':')
        print(res)

    ## FDR corrected significancies
    p_fdr,_,_,_ = multipletests([0.2501,0.0042,0.8538,0.3231],
                                    alpha=0.05, method='fdr_bh')
    sign_main_fts = list(compress(mainFts,p_fdr))
    print()
    print('Feature(s) with signficant difference:',sign_main_fts)
    if save:
        with open('results/posthoc_rm_anova.txt', 'a') as f:
            print('\nFeature(s) with signficant difference:',sign_main_fts, file=f)

    return sign_main_fts_____no_output_____sign_feats = posthoc(anova_data, False)SVM_maxAcc :
                Anova
======================================
         F Value Num DF  Den DF Pr > F
--------------------------------------
medState  1.4074 1.0000 19.0000 0.2501
======================================

SVM_coefVar :
                Anova
======================================
         F Value Num DF  Den DF Pr > F
--------------------------------------
medState 10.5744 1.0000 19.0000 0.0042
======================================

SVM_RMS :
                Anova
======================================
         F Value Num DF  Den DF Pr > F
--------------------------------------
medState  0.0349 1.0000 19.0000 0.8538
======================================

SVM_specPow_totalu4Hz :
                Anova
======================================
         F Value Num DF  Den DF Pr > F
--------------------------------------
medState  1.0293 1.0000 19.0000 0.3231
======================================

Feature(s) with signficant difference: ['SVM_coefVar']

## Visualization of 4 candidate features with violin plot_____no_output_____

def plotViolins(zmethod, sign_feats, save):
    feats = ['SVM_maxAcc','SVM_coefVar','SVM_RMS','SVM_specPow_totalu4Hz']
    ls = 20 # labelsize
    ts = 24 # titlesize
    # colors to use for violinplot
    col1,col2 = list(sns.color_palette("Set2"))[1],list(sns.color_palette("Paired"))[2]
    if zmethod == 'std':
        z_or_n = z_total # define to use stand. or norm. data
        y_label='Patient mean of \nstandarized feature'
    elif zmethod == 'nrm':
        z_or_n = n_total
        y_label='Patient mean of \nnormalized feature'
#     elif zmethod == 'None':
#         z_or_n = raw_total
#         y_label='Patient mean of feature'
    anova_data = pd.DataFrame(columns=['pt','medState']+mainFts)
    dat={}
    for ft in mainFts:
        dat['pt']=[]
        dat['medState']=[]
        dat[ft]=[]
        for p,pt in enumerate(features.keys()):
            for med in [0,1]:
                dat['pt'].append(p)
                dat['medState'].append(med)
                m = np.mean(
                    z_or_n[np.logical_and(
                        z_or_n['pt']==pt,z_or_n['medState']==med)]
                    [ft])
                dat[ft].append(m)
    for col in anova_data.keys():
        anova_data[col] = dat[col]

    violin_data = pd.melt(anova_data, id_vars=['medState'],
            value_vars=mainFts)
    violin_data.loc[violin_data['medState']==0,'medState'] = 'Pre-med (OFF)'
    violin_data.loc[violin_data['medState']==1,'medState'] = 'Post-med (ON)'
    violin_data = violin_data.rename(columns={'variable':'Features',
        'medState':'Medication Status','value':y_label})

    # plot violin
    fig, ax = plt.subplots(figsize=(12, 8))
    a = sns.violinplot(ax=ax, x='Features',y=y_label,
            hue='Medication Status',
        data=violin_data, palette=[col1,col2], split=True,scale='count',
                        inner='stick', figsize=(24,8))
    # Add significancy stars
    for n,ft in enumerate(mainFts):
        if ft  in sign_feats:
            if zmethod=='std':
                ax.annotate('*', xy= (n, 2.1), fontsize=34, color='red')
            elif zmethod=='nrm':
                ax.annotate('*', xy= (n, .9), fontsize=34, color='red')
#     ax.plot([],[],'',c='white',label='* = Significancy (alpha = 0.05')

    ax.set_title('Main Features during Pre- vs Post-medication States', size=ts)
    if zmethod=='std':
        ax.set_ylim(-2,2.5)
        ax.set_yticks(np.arange(-2,2.5,1))
        ax.set_yticklabels(np.arange(-2,2.5,1),size=ls)
        (xmin,xmax) = ax.get_xlim()
        ax.hlines(np.linspace(-2,2,5), xmin= xmin, xmax= xmax, color='gray', alpha=0.5)
    elif zmethod=='nrm':
        ax.set_ylim(0,1.1)
        ax.set_yticks(np.linspace(0,1,5))
        ax.set_yticklabels(np.linspace(0,1,5),size=ls)
        (xmin,xmax) = ax.get_xlim()
        ax.hlines(np.linspace(0,1,5), xmin= xmin, xmax= xmax, color='gray', alpha=0.5)
    ax.set_xticklabels(['Maximum\nAcceleration','Coefficient\nof Variance',
                        'Root Mean\nSquare','Spectral Power\nbelow 4 Hz'], size=ls)
    ax.set_xlabel('Features', size=ls)
    ax.set_ylabel(y_label, size=ls)
    ax.legend(fontsize=ls, )
    fig = ax.get_figure()
    if save:
        fig.savefig(os.path.join(path,'figures/%s_violin_4fts_sig.png' %zmethod), dpi=300)

    _____no_output_____# plotViolins('std', sign_feats , True)

_____no_output_____# plotViolins('nrm', sign_feats, True)

_____no_output_____

## Part 3: Classification of Pre- and Post-med states

Results and figures are saved to repective folders in main path. Each can be created for every model (SV vs RF, group vs individual), and data selection approach (4 vs all features, with vs without activity filtering).

_____no_output_____### 3a: Barplots to visualize classification accuracy and auroc for two selected models _____no_output_____

def predictionBars(two_models, model_labels, fig_title, save, save_filename):

    '''Always shows Accuracy and AUROC, number of lists to compare decides number of figures.
    Lists to compare has to be list of lists of two groups.[[A,B], [C,D], [A,C]]
    Colors to visualize.
    '''
    pts = np.unique([col[:3] for col in dictout[two_models[0]].keys()])
    # settings
    ls= 20 # standard labelsize
    ts= 24 # standard titlesize
    fig,axes = plt.subplots(2,1, figsize=(20,12))
    colors = ['green','salmon'] # colors in respective order
    widths = [-0.3,0.3] # position on x-axis for bars
    annpos = [-0.2,0.1] # corrections to plosition the annotation stars correctly
    for fig_row, metric in enumerate(['Accuracy','AUROC']):
        # show individual means with significancy (n=20) per outcome-metric
        for d,dct in enumerate(two_models):
            heights = []
            sds = []
            for key in dictout[dct].keys():
                if np.logical_and(metric in key,len(key) == len(metric)+4): # filters out XXX_METRIC keys
                    heights.append(dictout[dct].loc['pred'][key])
                if np.logical_and(metric in key, 'sd' in key): # filters out all '_sd' keys
                    sds.append(dictout[dct].loc['pred'][key])
            if 'group' in dct:
                sds = [0]*len(pts)
#             print('Analysis: %s (n=%i), mean %s = %.3f, sd = %.2f' %
#                   (dct,len(heights),metric,np.mean(heights),np.std(heights)))
            heights = [np.mean(heights)]+heights
            list_sd = [np.std(heights)]+sds
            axes[fig_row].bar(x=np.arange(21), height=heights, yerr=list_sd,
                    width=widths[d], align='edge', color=colors[d], label= model_labels[d] )

#                 # ADD LATER: significancy level with star annotation, and FDR correction!
#                 for x_pos,pt in enumerate(table.index):
#                     if table.loc[pt][methods[method]+metric+'_sign'] == 1:
#                         star_heigth = heights[x_pos+1] + list_errs[x_pos+1]/2 + 0.1
#                         axes[fig_row].annotate('*', xy=(x_pos+1 +annpos[m], star_heigth), fontsize=20, color=colors[m])
            # legend item for sign-asterixes
#             axes[fig_row].plot([],[],'',c='white',label='* = Significancy, p = 0.05')

        # figure design/layout
        axes[fig_row].tick_params(labelsize=ls)
        x_labels = ['Mean']
        x_labels.extend(pts)
        axes[fig_row].set_xticklabels(x_labels)
        axes[fig_row].set_xticks(np.arange(len(x_labels)))
        axes[fig_row].set_title(metric, fontsize=ts)
        axes[fig_row].axhline(y=0.5, ls='--', c='k', lw=1.5, label='Chance-level')
        for yline in [0.2,0.4,0.6,0.8]:
            axes[fig_row].axhline(y=yline, ls='-', c='gray', lw=0.5, )
        for yline in [0.1,0.3,0.7,0.9]:
            axes[fig_row].axhline(y=yline, ls='--', c='gray', lw=0.3, )
        axes[fig_row].axhline(y=1, ls='-', c='k', lw=0.5, )
        axes[fig_row].set_ylim(0, 1.1)
        axes[fig_row].set_yticks(np.arange(0,1.1,0.2))
        axes[fig_row].legend(fontsize=ls, loc='upper right', ncol=4)
    plt.suptitle(fig_title, fontsize=ts+6, y=.98)
    plt.tight_layout(h_pad=1, pad=2)

    if save == True:
        plt.savefig(os.path.join(path,'figures',save_filename), dpi=300)

    plt.show()

    _____no_output_____print('List of groups to pick from for comparison:\n')
for key in list(dictout.keys()):
    print('%s' % key)List of groups to pick from for comparison:

indiv_SV_allfts_actfilter
indiv_SV_allfts
indiv_SV_4fts_actfilter
indiv_SV_4fts
indiv_RF_allfts_actfilter
indiv_RF_allfts
indiv_RF_4fts_actfilter
indiv_RF_4fts
group_SV_allfts_actfilter
group_SV_allfts
group_SV_4fts_actfilter
group_SV_4fts
group_RF_allfts_actfilter
group_RF_allfts
group_RF_4fts_actfilter
group_RF_4fts
# predictionBars(['group_SV_4fts_actfilter','indiv_SV_allfts_actfilter'],
#                         ['4 feature model (SV group, filtered)',
#                          '103 feature model (SV individual, filtered)'],
#                'Models based on 4 vs 103 features',
#                True, 'bars_4_vs_103fts_(best)')

_____no_output_____## significance to add
# predictionBars(['indiv_SV_allfts_actfilter','group_SV_allfts_actfilter'],
#                ['Individual','Group'],
#                'Individual vs Group Model (both SV and activity filtered)',
#                False, 'bars_GroupVsIndiv_(SV_actfilter)')

_____no_output_____updrsScores[['inclSideHandBrady_diff','inclSideHandTremor_diff']].T_____no_output_____

### 3b: Equality-plots to compare classification results between two approaches per participant (e.g. individual vs group)_____no_output_____

def equality_plots(models,mod_labels,blackwhite,fig_title,save,save_filename):
    '''
    models = list of two models from dictout.
    mod_labels= list of the figure labels corresponding to the two models.
    blackwhite = Boolean
    fig_title = string for title in fig.
    save=Boolean, save_filename = string to store figure-file.

    P-value correction (and choice of type) for multiple comparison is done within script.

    Makes two ewuality plots, AUROC (l) and Accuracy (r),
    with indications of significances.
    First model in list will be X-AXIS, second will be Y-AXIS.'''

    metrics = ['AUROC','Accuracy']
    p_tresh = 0.05
    cols = ['%s_%s'%(mod,metr) for mod in ['1','2'] for metr in metrics]
    cols = cols+ [col+'_sign' for col in cols]+['AUROC_sign_comb','Accuracy_sign_comb']
    eq_dat = pd.DataFrame(index=updrsScores.index, columns=cols)

    for n,mod in enumerate(models): # create eq_dat
        for met in metrics:
            for pt in eq_dat.index:
                eq_dat.at[pt,'%i_%s' % (n+1,met) ] = dictout[mod].loc['pred']['%s_%s' % (pt,met)]
# #                 # signifiance without correction:
#                 eq_dat.at[pt,'%i_%s_sign' % (n+1,met) ] = dictout[mod].loc['p']['%s_%s' % (pt,met)] < p_tresh
#             # significance with multicomparison p-correction
            ps = [dictout[mod].loc['p']['%s_%s' % (pt,met)] for pt in eq_dat.index]
            ps_fdr,_,_,_ = multipletests(ps,alpha=p_tresh, method='fdr_bh')
            eq_dat['%i_%s_sign' % (n+1,met)] = ps_fdr.astype(int)
    for pt in eq_dat.index:
        for met in metrics: # combined sign: 1= only 1, 2=only 2, 3=both not, 4 BOTH SIGN
            if np.logical_and(eq_dat.loc[pt]['1_%s_sign' % met] == 1,eq_dat.loc[pt]['2_%s_sign' % met] == 0):
                eq_dat.at[pt,'%s_sign_comb' % met] = 1
            elif np.logical_and(eq_dat.loc[pt]['1_%s_sign' % met] == 0,eq_dat.loc[pt]['2_%s_sign' % met] == 1):
                eq_dat.at[pt,'%s_sign_comb' % met] = 2
            elif np.logical_and(eq_dat.loc[pt]['1_%s_sign' % met] == 0,eq_dat.loc[pt]['2_%s_sign' % met] == 0):
                eq_dat.at[pt,'%s_sign_comb' % met] = 3
            elif np.logical_and(eq_dat.loc[pt]['1_%s_sign' % met] == 1,eq_dat.loc[pt]['2_%s_sign' % met] == 1):
                eq_dat.at[pt,'%s_sign_comb' % met] = 4
            else:
                print('why')

    fig, axes = plt.subplots(1,len(metrics),figsize=(26,12))
    dotSize = 250
    fs=30
    ts=36
    for n,metric in enumerate(metrics):
        x_ax = '1_%s' % metric
        y_ax = '2_%s' % metric
        if blackwhite:
            # use triangle for ind-sig, round for ind-non-sig
            # use filled for group-sig, non-filled for group-non-sig
            scatter = axes[n].scatter(eq_dat[eq_dat['%s_sign_comb' % met]==4][x_ax],eq_dat[eq_dat['%s_sign_comb' % met]==4][y_ax],
                                      s=dotSize, lw=2, c='k',marker='^',  label='Both significant') #edgecolors='k', facecolors='none',
            scatter = axes[n].scatter(eq_dat[eq_dat['%s_sign_comb' % met]==1][x_ax],eq_dat[eq_dat['%s_sign_comb' % met]==1][y_ax],
                                      s=dotSize, lw=2,marker='^', label='Only %s significant' % mod_labels[0],edgecolors='k', facecolors='none',)
            scatter = axes[n].scatter(eq_dat[eq_dat['%s_sign_comb' % met]==2][x_ax],eq_dat[eq_dat['%s_sign_comb' % met]==2][y_ax], s=dotSize, lw=2,c='k',
                                      marker='o',  label= 'Only %s significant' % mod_labels[1],   ) #edgecolors='k', facecolors='none',
            scatter = axes[n].scatter(eq_dat[eq_dat['%s_sign_comb' % met]==3][x_ax],eq_dat[eq_dat['%s_sign_comb' % met]==3][y_ax], s=dotSize, lw=2,
                                      marker='o', label='None significant',edgecolors='k', facecolors='none',)
        else:
            # use triangle for ind-sig, round for ind-non-sig
            # use filled for group-sig, non-filled for group-non-sig
            scatter = axes[n].scatter(eq_dat[eq_dat['%s_sign_comb' % met]==4][x_ax],eq_dat[eq_dat['%s_sign_comb' % met]==4][y_ax],
                                      s=dotSize, lw=2, color='green',marker='o',  label='Both significant')
            scatter = axes[n].scatter(eq_dat[eq_dat['%s_sign_comb' % met]==1][x_ax],eq_dat[eq_dat['%s_sign_comb' % met]==1][y_ax],
                                      s=dotSize, lw=2,color='darkblue',marker='o', label='Only %s \nsignificant' % mod_labels[0],)
            scatter = axes[n].scatter(eq_dat[eq_dat['%s_sign_comb' % met]==2][x_ax],eq_dat[eq_dat['%s_sign_comb' % met]==2][y_ax],
                                      s=dotSize, lw=2,color='goldenrod',marker='o',  label='Only %s \nsignificant' % mod_labels[1],   ),
            scatter = axes[n].scatter(eq_dat[eq_dat['%s_sign_comb' % met]==3][x_ax],eq_dat[eq_dat['%s_sign_comb' % met]==3][y_ax],
                                      s=dotSize, lw=2,edgecolors='k',facecolors='lightgray',marker='o', label='None significant',)

        # plot equality line
        xStart, xEnd, yStart, yEnd = 0.2, 1.0, 0.2, 1.0
        axes[n].axis([xStart, xEnd, yStart, yEnd])
        axes[n].plot(np.linspace(xStart,xEnd,10),np.linspace(yStart,yEnd,10), c='k',
                         ls='dotted',lw=5, label='Equality of models',alpha=0.7)
        # details for main figure
        axes[n].hlines([0.4,0.6,0.8], xmin=0.2, xmax=1, color='gray', alpha=0.5)
        axes[n].vlines([0.4,0.6,0.8], ymin=0.2, ymax=1, color='gray', alpha=0.5)
        axes[n].set_xlabel('%s score' % mod_labels[0], size=fs)
        axes[n].set_ylabel('%s score' % mod_labels[1], size=fs)
        axes[n].set_title('%s' % (metric), size=ts)
#         axes[n].legend(loc='lower left',fontsize=24, )
        axes[n].tick_params(labelsize=fs,direction='out', length=6, width=2, colors='k',
               grid_color='r', grid_alpha=0.5)
    plt.legend(loc='upper left',fontsize=24, bbox_to_anchor=(1.05, 1),)
    plt.suptitle(fig_title, fontsize=ts+6, y=.98)
    plt.tight_layout(h_pad=1, pad=2)
    if save:
        plt.savefig(os.path.join(path,'figures',save_filename), dpi=150)
    plt.show()
_____no_output_____# # Best models based on 4 vs 103 features
# equality_plots(['group_SV_4fts_actfilter','indiv_SV_allfts_actfilter'],
#                         ['4 feature model \n(SV group, filtered)',
#                          '103 feature model \n(SV individual, filtered)'],
#                         False, 'Models with 4 vs 103 features (both group SV, activity filtered)',
#                         True,'EQplot_4_vs_103fts_(best)(fdr)')

_____no_output_____# # Best individual vs best group model
# equality_plots(['indiv_SV_allfts_actfilter','group_SV_allfts_actfilter'],
#                         ['Individual model','Group model'],
#                         False, 'Individual vs Group models (both SV, activity filtered)',
#                         True,'EQplot_Indiv_vs_Group_SV_actfilter(fdr)')
_____no_output_____# Best model without vs with filtering
# equality_plots(['indiv_SV_allfts','indiv_SV_allfts_actfilter'],
#                         ['Not filtered\n(individual SV)','Activity filtered\n(individual SV)'],
#                         False, 'Best model without vs best model with activity filtering',
#                         True,'EQplot_W_vs_WO_actfilter_SV_indiv(fdr)')

_____no_output_____

## Part 4: Correlating predictive outcomes with clinical scores_____no_output_____

## correlations updrs sub-scores with outcome, new datasplitting analyses

def plot_clin_corr(models,mod_labels,save,save_filename):
    '''
    models: defines one or two models which are compared with subscores
    '''
    subscores = ['inclSideHandBrady_%diff','inclSideHandTremor_%diff',
                     'bradyBody_%diff','legs_%diff','gait_%diff',
                     'posture_%diff','facial_%diff' ]
    subscorelabels = ['Hand brady','Hand tremor','Body brady','Legs','Gait',
                     'Posture','Face' ]
    metrics = ['AUROC', 'Accuracy']
    corr_dat = pd.DataFrame(index=updrsScores.index) # make df for this plot
    for sub in subscores:
        corr_dat[sub] = updrsScores[sub] # import subscores per pt
    for n,mod in enumerate(models):
        for met in metrics: # import metric scores, number 1, 2
            corr_dat['%i_%s'%(n+1,met)] = [dictout[mod].loc['pred']['%s_%s' % (pt,met)] for pt in pts]

    plt.figure(figsize=(8*len(models),8))
    corr_list = {}
    for clinVar in subscores:
        corr_list[clinVar] = [] # create per subscore list of two corr's per model in models-list
        for n,mod in enumerate(models):
            for met in metrics:
                r,p = spearmanr(corr_dat[clinVar], corr_dat['%i_%s'%(n+1,met)]) # calculate corr
                corr_list[clinVar].append(r) # add r-value to list per subscore

    xgroups = []
    for modlab in mod_labels:
        xgroups.extend(['%s\nAUROC'%modlab,'%s\nAccuracy'%modlab])
    x = np.arange(len(xgroups))
    width = 0.1
    lw = 0.2
    cs = 10
    for n,clinVar in enumerate(subscores):
        plt.bar(x+ 0.1*n, corr_list[clinVar], label=subscorelabels[n],
                capsize=cs,linewidth=lw, width=width, )#color='green' , yerr=stdsOFF,

    plt.ylabel('Spearman Correlation R', size=24)
    plt.title('Clinical MDS-UPDRS III subscores vs. Predictive performance with activity filtering', size=28)

    plt.xticks(np.arange(0.3,len(xgroups)+.3,1),xgroups, rotation = 45, fontsize =24)
    plt.yticks(fontsize=20)
    plt.ylim(-.5,0.5)
    plt.legend(loc='lower center',ncol=4,prop={'size': 24})
    plt.rcParams["axes.grid"] = False
    for h in np.arange(-0.3,0.5,0.1):
        plt.axhline(y=h, zorder=0, color='lightgray')
    plt.tight_layout()
    if save:
        plt.savefig(os.path.join(path,'figures','%s.png' % save_filename), dpi=300)

    plt.show()

_____no_output_____# Paper FIG5
# plot_clin_corr(['indiv_SV_allfts_actfilter','group_SV_allfts_actfilter'],
#                ['Individual SV','Group SV'],
#                True,'subscores_vs_performance_SVs_filtered')
_____no_output_____# ## SUPPL FIG 6 (pm adjust title in function for w or w/o filter)
# plot_clin_corr(['indiv_SV_allfts_actfilter','indiv_RF_allfts_actfilter',
#  'group_SV_allfts_actfilter','group_RF_allfts_actfilter'],
# ['Individual SV','Individual RF',
#  'Group SV','Group RF'],
#              True,'subscores_vs_performance_actfilter')

_____no_output_____

## Part 5: Visualizing effect of different number of trainings patients on group model
_____no_output_____

def plot_training_pts(actfilter,save):
    fig,ax=plt.subplots(1,1,figsize=(16,8))
    colors=['purple','orange']
    lstyles=['-','--']
    for c,cls in enumerate(['SV','RF']):
        if actfilter:
            dat=pd.read_csv(os.path.join(path,'results','preds_%s_nr_train_pts_actfilter.csv'%cls), index_col=0)
        else:
            dat=pd.read_csv(os.path.join(path,'results','preds_%s_nr_train_pts.csv'%cls), index_col=0)
        scores = pd.DataFrame(index=np.arange(1,20), columns=['AUROC','AUROC_sd','Accuracy','Accuracy_sd'])
        for n_pts in scores.index:
            for met in ['AUROC','Accuracy']:
                scores.at[n_pts,met] = np.mean(dat['%s_mean_n%i'%(met,n_pts)]) # mean and std's over tested patients
                scores.at[n_pts,met+'_sd'] = np.std(dat['%s_mean_n%i'%(met,n_pts)])
        for m,met in enumerate(['AUROC','Accuracy']):
            x_plot = scores.index
            ax.plot(x_plot, scores[met], label='Mean %s (%s)' % (met,cls), linestyle=lstyles[m],
                    c=colors[c], lw=2)
#             high = np.array(scores[met]-scores['%s_sd' %met]).astype(float)
#             low = np.array(scores[met]+scores['%s_sd' %met]).astype(float)
#             ax.fill_between(x_plot, low,high, color= colors[m], alpha=0.1, label='Std dev %s'%met)
    # config figure
    ax.legend(fontsize=20)
    ax.set_xlabel('Number of patients used for training', fontsize=20)
    ax.set_xticks(np.arange(1,20,2))
    ax.tick_params(labelsize=20)
    ax.grid(True, axis='both', which='major')
    if actfilter:
        ax.set_title('Effect of number of training patients in group models, with activity filter', fontsize=24)
    else:
        ax.set_title('Effect of number of training patients in group models', fontsize=24)

    plt.tight_layout()
    if save:
        if actfilter:
            plt.savefig(os.path.join(path,'figures','Effect_numTrainingPtsGroupModels_actfilter.png'), dpi=300)
        else:
            plt.savefig(os.path.join(path,'figures','Effect_numTrainingPtsGroupModels.png'), dpi=300)
    plt.show()

_____no_output_____# plot_training_pts(actfilter=False,save=True)
# plot_training_pts(actfilter=True,save=True)_____no_output_____