# Notebook from neurogenomics/scCODA
Path: tests/microbiome_testing/MovingPictures_Analysis.ipynb

# qiime2's "Moving Picutres" tutorial - an analysis in SCDCdm

This notebook takes the microbial count data from qiime2's "Moving Pictures" tutorial and compares different methods of
differential testing for microbial populations on it.

The aim is to assess the performance of the SCDCdm-model (built for single-cell data) on higher-dimensional and more sparse data._____no_output_____

# Setup

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import importlib
import arviz as az
import pickle as pkl
from matplotlib.pyplot import cm

from scdcdm.util import comp_ana as mod
from scdcdm.util import result_classes as res
from scdcdm.util import cell_composition_data as dat
from scdcdm.util import data_visualization as viz

pd.options.display.float_format = '{:10,.3f}'.format
pd.set_option('display.max_columns', None)C:\Users\Johannes\Anaconda3\envs\Helmholtz\lib\site-packages\anndata\_core\anndata.py:21: FutureWarning: pandas.core.index is deprecated and will be removed in a future version.  The public classes are available in the top-level namespace.
  from pandas.core.index import RangeIndex

## The data

We run the same processing steps used in the [Tutorial](https://docs.qiime2.org/2020.6/tutorials/moving-pictures/) in qiime2,
until the "Differential abundance testing with ANCOM" section. We then collapse the feature table on different taxonomic
levels (phylum and genus), and export it as a .tsv file.

To import the data into SCDCdm, we use the exported count table as the count data, and the metadata table as the covariate data.
The metadata has 4 interesting covariates:
subject (binary), reported antibiotic use (binary), days since start (semi-continuous), body site (discrete w/ 4 levels).

### Data on the phylum-level

We first look at the data on a phylum-level._____no_output_____

data_path = "C:/Users/Johannes/AppData/Local/Packages/CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc/LocalState/rootfs/home/johannes/qiime2_projects/moving-pictures-tutorial"

# read phylum-level data from tsv

with open(data_path+"/exported_data/feature-table-l2.tsv", "rb") as f:
    biom_data = pd.read_csv(f, sep="\t", header=1, index_col="#OTU ID")

biom_data = biom_data.transpose()

# read metadata
with open(data_path+"/sample-metadata.tsv", "rb") as f:
    metadata = pd.read_csv(f, sep="\t", index_col="sample-id").iloc[1:, :]

metadata_columns = ["subject", "reported-antibiotic-usage", "days-since-experiment-start", "body-site"]

# add metadata to count data
biom_data = pd.merge(biom_data, metadata[metadata_columns], left_index=True, right_index=True)

# Create SCDCdm-compatible data file
data_phy = dat.from_pandas(biom_data, metadata_columns)
data_phy.obs = data_phy.obs.rename(columns={"reported-antibiotic-usage": "antibiotic", "body-site": "site",
                                    "days-since-experiment-start": "days_since_start"})

print(data_phy)AnnData object with n_obs × n_vars = 34 × 21
    obs: 'subject', 'antibiotic', 'days_since_start', 'site'

The data has 34 samples with 21 recorded phyla.
For some visualization, we can look at stacked barplots on the covariate levels.

On a subject and antibiotic level, the differences are rather subtle.
The gut microbiome differs vastly from the other sites, while the two palms are almost identical._____no_output_____

viz.plot_feature_stackbars(data_phy, ["samples", "subject", "site", "antibiotic"])_____no_output_____

### Data on the genus-level

We now proceed equally with the genus-level data. We get 34 observations of 287 genera._____no_output_____

# read genus-level data from tsv

with open(data_path+"/exported_data/l6-table.tsv", "rb") as f:
    biom_data = pd.read_csv(f, sep="\t", header=1, index_col="#OTU ID")

biom_data = biom_data.transpose()

# read metadata
with open(data_path+"/sample-metadata.tsv", "rb") as f:
    metadata = pd.read_csv(f, sep="\t", index_col="sample-id").iloc[1:, :]

metadata_columns = ["subject", "reported-antibiotic-usage", "days-since-experiment-start", "body-site"]

# add metadata to count data
biom_data = pd.merge(biom_data, metadata[metadata_columns], left_index=True, right_index=True)

# Create SCDCdm-compatible data file
data_gen = dat.from_pandas(biom_data, metadata_columns)
data_gen.obs = data_gen.obs.rename(columns={"reported-antibiotic-usage": "antibiotic", "body-site": "site",
                                    "days-since-experiment-start": "days_since_start"})

print(data_gen)AnnData object with n_obs × n_vars = 34 × 287
    obs: 'subject', 'antibiotic', 'days_since_start', 'site'

## The SCDCdm model

SCDCdm is a fully Bayesian model for detecting compositional changes in single-cell data.

Conventions:
- Samples: $n=1 \ldots N$
- Classes: $k=1 \ldots K$
- Covariates: $d=1 \ldots D$
- Count matrix: $Y$
- Covariate matrix: $X$

### Model specification

At its core, it uses a Dirichlet-multinomial model with a log-linear covariate association to directly model cell (or microbial) counts:

$
\begin{align*}
\pmb{y_n|x_n} &\sim DirMult(\pmb{a(x_n)}, \pmb{\bar{y}}) \\
 \log(\pmb{a(x_n)}) &= \pmb{\alpha} + \pmb{x_n \cdot \beta} \\
\end{align*}
$

Effect selection is performed via a spike-and-slab prior on every effect $\beta_{d, k}.$
We use a continuous approximation of the spike-and-slab prior via the Logit-Normal distribution to ease HMC sampling:

$
\begin{align*}
\beta_{d, k} &= \tau_{d, k} \cdot \tilde{\beta}_{d, k} \\
\tau_{d, k} &\sim LogitNormal(0, 50)\\
\end{align*}
$

Priors on the intercepts $\alpha_k$ and raw effects $\tilde{\beta}_{d, k}$ are normally distributed,
with the effect prior having a noncentered reparametrization:

$
\begin{align*}
\alpha_k &\sim N(0, 5) \quad \forall k \in [K] \\
\tilde{\beta}_{d, k} &\sim N(\tilde{\mu}_{d, k}, \tilde{\sigma}^2_{d, k}) \quad \\
 \tilde{\mu}_{d, k} &\sim N(0, 1) \\
 \tilde{\sigma}^2_{d, k} &\sim HalfCauchy(0, 1) \quad \forall k \in [K], d \in [D] \\
\end{align*}
$

### Inference

SCDCdm is implemented in *tensorflow 2* and uses HMC sampling to infer parameters.
The *Edward2* framework is used to automatically calculate the log-prob function.

We sample one chain of 20,000 samples, and discard the first 5,000 samples as burnin.
Tensorflow uses a Leapfrog integrator; we set 10 leapfrog steps and an ititial step size of 0.01.
The step size is adapted over the first 4,000 chain states via a simple adaptation based on the log-acceptance-probability,
using a target acceptance probability of 0.8

_____no_output_____## SCDCdm on microbiome data - phylum level

We can now proceed to analyze the microbial count data with SCDCdm. First, we look at the data on a phylum level.
This data closely resembles single-cell RNA-seq data in terms of dimensions. The only difference is the presence of many zero entries.
While scRNA-seq data usually has close to no zero entries, the microbiome data has almost one third zeroes._____no_output_____

# Calculate proportion of zero entries

n_zeroes = np.sum(np.where(data_phy.X == 0, 0, 1))
n_total = data_phy.X.size

print(f"Zero entries: {n_zeroes} of {n_total} - {n_zeroes/n_total*100}%")Zero entries: 224 of 714 - 31.372549019607842%

### Comparison of subjects

Running SCDCdm on the "subject"-covariate, we see that no changes were found to be significant.
Judging from the inclusion probabilities, all phyla were also not close to being deemed significant
(inc. prob <= 0.55 << 0.81 (threshold)).

Note that compositional effects are always relative to the unchanged values.
We can therefore use one phylum as a baseline value (effect 0) to maybe get some effects in relation to this phylum.
Firmicutes as a baseline also reveals no effects, though._____no_output_____

# Model with subject as covariate
model_subject = mod.CompositionalAnalysis(data_phy, "subject", baseline_index=None)

result_subject = model_subject.sample_hmc(num_results=int(20000), n_burnin=5000)

result_subject.summary_extended(hdi_prob=0.95)MCMC sampling finished. (203.492 sec)
Acceptance rate: 67.5%
Compositional Analysis summary (extended):

Data: 34 samples, 21 cell types
Baseline index: None
Formula: subject
Spike-and-slab threshold: 0.810

MCMC Sampling: Sampled 20000 chain states (5000 burnin samples) in 203.492 sec. Acceptance rate: 67.5%

Intercepts:
                                Final Parameter   HDI 2.5%  HDI 97.5%  \
Cell Type
Unassigned;__                            -5.570     -7.576     -3.881
k__Archaea;p__Crenarchaeota              -4.903     -6.256     -3.585
k__Bacteria;__                           -2.992     -3.629     -2.290
k__Bacteria;p__Acidobacteria             -4.878     -6.374     -3.432
k__Bacteria;p__Actinobacteria            -1.208     -1.617     -0.800
k__Bacteria;p__Bacteroidetes              0.062     -0.288      0.404
k__Bacteria;p__Chloroflexi               -4.877     -6.393     -3.445
k__Bacteria;p__Cyanobacteria             -2.275     -2.822     -1.723
k__Bacteria;p__FBP                       -5.555     -7.759     -3.785
k__Bacteria;p__Firmicutes                 0.509      0.160      0.834
k__Bacteria;p__Fusobacteria              -1.313     -1.743     -0.899
k__Bacteria;p__GN02                      -4.068     -5.154     -2.979
k__Bacteria;p__Planctomycetes            -4.847     -6.266     -3.580
k__Bacteria;p__Proteobacteria             0.113     -0.227      0.475
k__Bacteria;p__SR1                       -4.405     -5.733     -3.267
k__Bacteria;p__Spirochaetes              -4.539     -5.858     -3.328
k__Bacteria;p__Synergistetes             -5.087     -6.922     -3.607
k__Bacteria;p__TM7                       -5.527     -7.434     -3.937
k__Bacteria;p__Tenericutes               -4.064     -5.160     -3.113
k__Bacteria;p__Verrucomicrobia           -2.869     -3.488     -2.215
k__Bacteria;p__[Thermi]                  -3.799     -4.819     -2.931

                                       SD  Expected Sample
Cell Type
Unassigned;__                       0.995            3.629
k__Archaea;p__Crenarchaeota         0.720            7.070
k__Bacteria;__                      0.341           47.793
k__Bacteria;p__Acidobacteria        0.746            7.249
k__Bacteria;p__Actinobacteria       0.212          284.542
k__Bacteria;p__Bacteroidetes        0.176        1,013.214
k__Bacteria;p__Chloroflexi          0.768            7.256
k__Bacteria;p__Cyanobacteria        0.280           97.894
k__Bacteria;p__FBP                  1.049            3.684
k__Bacteria;p__Firmicutes           0.174        1,584.275
k__Bacteria;p__Fusobacteria         0.214          256.181
k__Bacteria;p__GN02                 0.557           16.295
k__Bacteria;p__Planctomycetes       0.688            7.477
k__Bacteria;p__Proteobacteria       0.178        1,066.228
k__Bacteria;p__SR1                  0.632           11.633
k__Bacteria;p__Spirochaetes         0.673           10.174
k__Bacteria;p__Synergistetes        0.882            5.882
k__Bacteria;p__TM7                  0.919            3.788
k__Bacteria;p__Tenericutes          0.538           16.361
k__Bacteria;p__Verrucomicrobia      0.327           54.049
k__Bacteria;p__[Thermi]             0.484           21.325

Effects:
                                                     Final Parameter  \
Covariate            Cell Type
subject[T.subject-2] Unassigned;__                             0.000
                     k__Archaea;p__Crenarchaeota               0.000
                     k__Bacteria;__                            0.000
                     k__Bacteria;p__Acidobacteria              0.000
                     k__Bacteria;p__Actinobacteria             0.000
                     k__Bacteria;p__Bacteroidetes              0.000
                     k__Bacteria;p__Chloroflexi                0.000
                     k__Bacteria;p__Cyanobacteria              0.000
                     k__Bacteria;p__FBP                        0.000
                     k__Bacteria;p__Firmicutes                 0.000
                     k__Bacteria;p__Fusobacteria               0.000
                     k__Bacteria;p__GN02                       0.000
                     k__Bacteria;p__Planctomycetes             0.000
                     k__Bacteria;p__Proteobacteria             0.000
                     k__Bacteria;p__SR1                        0.000
                     k__Bacteria;p__Spirochaetes               0.000
                     k__Bacteria;p__Synergistetes              0.000
                     k__Bacteria;p__TM7                        0.000
                     k__Bacteria;p__Tenericutes                0.000
                     k__Bacteria;p__Verrucomicrobia            0.000
                     k__Bacteria;p__[Thermi]                   0.000

                                                      HDI 2.5%  HDI 97.5%  \
Covariate            Cell Type
subject[T.subject-2] Unassigned;__                      -0.748      0.832
                     k__Archaea;p__Crenarchaeota        -0.667      0.924
                     k__Bacteria;__                     -0.781      0.473
                     k__Bacteria;p__Acidobacteria       -0.785      0.862
                     k__Bacteria;p__Actinobacteria      -0.347      0.437
                     k__Bacteria;p__Bacteroidetes       -0.463      0.320
                     k__Bacteria;p__Chloroflexi         -0.741      0.806
                     k__Bacteria;p__Cyanobacteria       -0.659      0.381
                     k__Bacteria;p__FBP                 -0.865      0.646
                     k__Bacteria;p__Firmicutes          -0.293      0.473
                     k__Bacteria;p__Fusobacteria        -0.587      0.320
                     k__Bacteria;p__GN02                -0.672      0.613
                     k__Bacteria;p__Planctomycetes      -0.643      0.945
                     k__Bacteria;p__Proteobacteria      -0.560      0.162
                     k__Bacteria;p__SR1                 -0.728      0.624
                     k__Bacteria;p__Spirochaetes        -0.578      1.444
                     k__Bacteria;p__Synergistetes       -0.598      1.247
                     k__Bacteria;p__TM7                 -0.657      0.806
                     k__Bacteria;p__Tenericutes         -0.815      0.423
                     k__Bacteria;p__Verrucomicrobia     -0.755      0.423
                     k__Bacteria;p__[Thermi]            -0.903      0.470

                                                            SD  \
Covariate            Cell Type
subject[T.subject-2] Unassigned;__                       0.378
                     k__Archaea;p__Crenarchaeota         0.366
                     k__Bacteria;__                      0.286
                     k__Bacteria;p__Acidobacteria        0.413
                     k__Bacteria;p__Actinobacteria       0.167
                     k__Bacteria;p__Bacteroidetes        0.162
                     k__Bacteria;p__Chloroflexi          0.350
                     k__Bacteria;p__Cyanobacteria        0.234
                     k__Bacteria;p__FBP                  0.348
                     k__Bacteria;p__Firmicutes           0.170
                     k__Bacteria;p__Fusobacteria         0.203
                     k__Bacteria;p__GN02                 0.275
                     k__Bacteria;p__Planctomycetes       0.384
                     k__Bacteria;p__Proteobacteria       0.175
                     k__Bacteria;p__SR1                  0.311
                     k__Bacteria;p__Spirochaetes         0.517
                     k__Bacteria;p__Synergistetes        0.521
                     k__Bacteria;p__TM7                  0.327
                     k__Bacteria;p__Tenericutes          0.292
                     k__Bacteria;p__Verrucomicrobia      0.269
                     k__Bacteria;p__[Thermi]             0.356

                                                     Inclusion probability  \
Covariate            Cell Type
subject[T.subject-2] Unassigned;__                                   0.520
                     k__Archaea;p__Crenarchaeota                     0.522
                     k__Bacteria;__                                  0.503
                     k__Bacteria;p__Acidobacteria                    0.500
                     k__Bacteria;p__Actinobacteria                   0.402
                     k__Bacteria;p__Bacteroidetes                    0.446
                     k__Bacteria;p__Chloroflexi                      0.519
                     k__Bacteria;p__Cyanobacteria                    0.538
                     k__Bacteria;p__FBP                              0.546
                     k__Bacteria;p__Firmicutes                       0.412
                     k__Bacteria;p__Fusobacteria                     0.474
                     k__Bacteria;p__GN02                             0.441
                     k__Bacteria;p__Planctomycetes                   0.505
                     k__Bacteria;p__Proteobacteria                   0.417
                     k__Bacteria;p__SR1                              0.539
                     k__Bacteria;p__Spirochaetes                     0.543
                     k__Bacteria;p__Synergistetes                    0.526
                     k__Bacteria;p__TM7                              0.508
                     k__Bacteria;p__Tenericutes                      0.487
                     k__Bacteria;p__Verrucomicrobia                  0.551
                     k__Bacteria;p__[Thermi]                         0.528

                                                     Expected Sample  \
Covariate            Cell Type
subject[T.subject-2] Unassigned;__                             3.629
                     k__Archaea;p__Crenarchaeota               7.070
                     k__Bacteria;__                           47.793
                     k__Bacteria;p__Acidobacteria              7.249
                     k__Bacteria;p__Actinobacteria           284.542
                     k__Bacteria;p__Bacteroidetes          1,013.214
                     k__Bacteria;p__Chloroflexi                7.256
                     k__Bacteria;p__Cyanobacteria             97.894
                     k__Bacteria;p__FBP                        3.684
                     k__Bacteria;p__Firmicutes             1,584.275
                     k__Bacteria;p__Fusobacteria             256.181
                     k__Bacteria;p__GN02                      16.295
                     k__Bacteria;p__Planctomycetes             7.477
                     k__Bacteria;p__Proteobacteria         1,066.228
                     k__Bacteria;p__SR1                       11.633
                     k__Bacteria;p__Spirochaetes              10.174
                     k__Bacteria;p__Synergistetes              5.882
                     k__Bacteria;p__TM7                        3.788
                     k__Bacteria;p__Tenericutes               16.361
                     k__Bacteria;p__Verrucomicrobia           54.049
                     k__Bacteria;p__[Thermi]                  21.325

                                                     log2-fold change
Covariate            Cell Type
subject[T.subject-2] Unassigned;__                              0.000
                     k__Archaea;p__Crenarchaeota                0.000
                     k__Bacteria;__                             0.000
                     k__Bacteria;p__Acidobacteria               0.000
                     k__Bacteria;p__Actinobacteria              0.000
                     k__Bacteria;p__Bacteroidetes               0.000
                     k__Bacteria;p__Chloroflexi                 0.000
                     k__Bacteria;p__Cyanobacteria               0.000
                     k__Bacteria;p__FBP                         0.000
                     k__Bacteria;p__Firmicutes                  0.000
                     k__Bacteria;p__Fusobacteria                0.000
                     k__Bacteria;p__GN02                        0.000
                     k__Bacteria;p__Planctomycetes              0.000
                     k__Bacteria;p__Proteobacteria              0.000
                     k__Bacteria;p__SR1                         0.000
                     k__Bacteria;p__Spirochaetes                0.000
                     k__Bacteria;p__Synergistetes               0.000
                     k__Bacteria;p__TM7                         0.000
                     k__Bacteria;p__Tenericutes                 0.000
                     k__Bacteria;p__Verrucomicrobia             0.000
                     k__Bacteria;p__[Thermi]                    0.000
# Model with subject as covariate, Actinobacteria as baseline
model_subject_2 = mod.CompositionalAnalysis(data_phy, "subject", baseline_index="k__Bacteria;p__Firmicutes")

result_subject_2 = model_subject_2.sample_hmc(num_results=int(20000), n_burnin=5000)

result_subject_2.summary()MCMC sampling finished. (228.082 sec)
Acceptance rate: 54.3%
Compositional Analysis summary:

Data: 34 samples, 21 cell types
Baseline index: 9
Formula: subject

Intercepts:
                                Final Parameter  Expected Sample
Cell Type
Unassigned;__                            -5.578            3.586
k__Archaea;p__Crenarchaeota              -4.917            6.946
k__Bacteria;__                           -2.969           48.723
k__Bacteria;p__Acidobacteria             -4.958            6.667
k__Bacteria;p__Actinobacteria            -1.231          277.035
k__Bacteria;p__Bacteroidetes              0.057        1,004.399
k__Bacteria;p__Chloroflexi               -4.870            7.280
k__Bacteria;p__Cyanobacteria             -2.294           95.693
k__Bacteria;p__FBP                       -5.760            2.990
k__Bacteria;p__Firmicutes                 0.530        1,611.861
k__Bacteria;p__Fusobacteria              -1.321          253.191
k__Bacteria;p__GN02                      -4.101           15.708
k__Bacteria;p__Planctomycetes            -4.984            6.496
k__Bacteria;p__Proteobacteria             0.113        1,062.250
k__Bacteria;p__SR1                       -4.348           12.270
k__Bacteria;p__Spirochaetes              -4.485           10.699
k__Bacteria;p__Synergistetes             -5.013            6.310
k__Bacteria;p__TM7                       -5.814            2.832
k__Bacteria;p__Tenericutes               -4.021           17.016
k__Bacteria;p__Verrucomicrobia           -2.903           52.047
k__Bacteria;p__[Thermi]                  -3.764           22.002

Effects:
                                                     Final Parameter  \
Covariate            Cell Type
subject[T.subject-2] Unassigned;__                             0.000
                     k__Archaea;p__Crenarchaeota               0.000
                     k__Bacteria;__                            0.000
                     k__Bacteria;p__Acidobacteria              0.000
                     k__Bacteria;p__Actinobacteria             0.000
                     k__Bacteria;p__Bacteroidetes              0.000
                     k__Bacteria;p__Chloroflexi                0.000
                     k__Bacteria;p__Cyanobacteria              0.000
                     k__Bacteria;p__FBP                        0.000
                     k__Bacteria;p__Firmicutes                 0.000
                     k__Bacteria;p__Fusobacteria               0.000
                     k__Bacteria;p__GN02                       0.000
                     k__Bacteria;p__Planctomycetes             0.000
                     k__Bacteria;p__Proteobacteria             0.000
                     k__Bacteria;p__SR1                        0.000
                     k__Bacteria;p__Spirochaetes               0.000
                     k__Bacteria;p__Synergistetes              0.000
                     k__Bacteria;p__TM7                        0.000
                     k__Bacteria;p__Tenericutes                0.000
                     k__Bacteria;p__Verrucomicrobia            0.000
                     k__Bacteria;p__[Thermi]                   0.000

                                                     Expected Sample  \
Covariate            Cell Type
subject[T.subject-2] Unassigned;__                             3.586
                     k__Archaea;p__Crenarchaeota               6.946
                     k__Bacteria;__                           48.723
                     k__Bacteria;p__Acidobacteria              6.667
                     k__Bacteria;p__Actinobacteria           277.035
                     k__Bacteria;p__Bacteroidetes          1,004.399
                     k__Bacteria;p__Chloroflexi                7.280
                     k__Bacteria;p__Cyanobacteria             95.693
                     k__Bacteria;p__FBP                        2.990
                     k__Bacteria;p__Firmicutes             1,611.861
                     k__Bacteria;p__Fusobacteria             253.191
                     k__Bacteria;p__GN02                      15.708
                     k__Bacteria;p__Planctomycetes             6.496
                     k__Bacteria;p__Proteobacteria         1,062.250
                     k__Bacteria;p__SR1                       12.270
                     k__Bacteria;p__Spirochaetes              10.699
                     k__Bacteria;p__Synergistetes              6.310
                     k__Bacteria;p__TM7                        2.832
                     k__Bacteria;p__Tenericutes               17.016
                     k__Bacteria;p__Verrucomicrobia           52.047
                     k__Bacteria;p__[Thermi]                  22.002

                                                     log2-fold change
Covariate            Cell Type
subject[T.subject-2] Unassigned;__                              0.000
                     k__Archaea;p__Crenarchaeota                0.000
                     k__Bacteria;__                             0.000
                     k__Bacteria;p__Acidobacteria               0.000
                     k__Bacteria;p__Actinobacteria              0.000
                     k__Bacteria;p__Bacteroidetes               0.000
                     k__Bacteria;p__Chloroflexi                 0.000
                     k__Bacteria;p__Cyanobacteria               0.000
                     k__Bacteria;p__FBP                         0.000
                     k__Bacteria;p__Firmicutes                  0.000
                     k__Bacteria;p__Fusobacteria                0.000
                     k__Bacteria;p__GN02                        0.000
                     k__Bacteria;p__Planctomycetes              0.000
                     k__Bacteria;p__Proteobacteria              0.000
                     k__Bacteria;p__SR1                         0.000
                     k__Bacteria;p__Spirochaetes                0.000
                     k__Bacteria;p__Synergistetes               0.000
                     k__Bacteria;p__TM7                         0.000
                     k__Bacteria;p__Tenericutes                 0.000
                     k__Bacteria;p__Verrucomicrobia             0.000
                     k__Bacteria;p__[Thermi]                    0.000

### Comparison of body sites

We now switch towards looking at the four body sites. For discrete covariates, SCDCdm always takes one level as a reference and compares the other levels to it.
Taking the gut as a reference level, we find numerous significances:_____no_output_____

# Model with body site as covariate, no baseline
model_gut = mod.CompositionalAnalysis(data_phy, "site", baseline_index=None)

result_gut = model_gut.sample_hmc(num_results=int(20000), n_burnin=5000)

result_gut.summary_extended(hdi_prob=0.95)MCMC sampling finished. (291.258 sec)
Acceptance rate: 62.9%
Compositional Analysis summary (extended):

Data: 34 samples, 21 cell types
Baseline index: None
Formula: site
Spike-and-slab threshold: 0.810

MCMC Sampling: Sampled 20000 chain states (5000 burnin samples) in 291.258 sec. Acceptance rate: 62.9%

Intercepts:
                                Final Parameter   HDI 2.5%  HDI 97.5%  \
Cell Type
Unassigned;__                            -5.922     -8.605     -3.591
k__Archaea;p__Crenarchaeota              -5.489     -7.302     -3.613
k__Bacteria;__                           -4.583     -7.136     -3.038
k__Bacteria;p__Acidobacteria             -5.998     -8.662     -3.430
k__Bacteria;p__Actinobacteria            -2.430     -3.391     -1.493
k__Bacteria;p__Bacteroidetes              2.465      1.834      3.196
k__Bacteria;p__Chloroflexi               -4.770     -6.499     -3.072
k__Bacteria;p__Cyanobacteria             -3.719     -5.024     -2.453
k__Bacteria;p__FBP                       -6.052     -9.339     -3.736
k__Bacteria;p__Firmicutes                 1.848      1.322      2.636
k__Bacteria;p__Fusobacteria              -2.553     -3.575     -1.503
k__Bacteria;p__GN02                      -5.151     -7.043     -3.074
k__Bacteria;p__Planctomycetes            -5.957     -8.688     -3.741
k__Bacteria;p__Proteobacteria            -0.043     -0.690      0.516
k__Bacteria;p__SR1                       -5.388     -8.014     -3.197
k__Bacteria;p__Spirochaetes              -4.933     -6.920     -3.221
k__Bacteria;p__Synergistetes             -5.306     -7.735     -3.365
k__Bacteria;p__TM7                       -5.896     -8.129     -3.715
k__Bacteria;p__Tenericutes               -5.196     -7.333     -3.492
k__Bacteria;p__Verrucomicrobia           -2.307     -3.079     -1.500
k__Bacteria;p__[Thermi]                  -4.476     -6.405     -2.876

                                       SD  Expected Sample
Cell Type
Unassigned;__                       1.344            0.624
k__Archaea;p__Crenarchaeota         1.014            0.962
k__Bacteria;__                      1.113            2.381
k__Bacteria;p__Acidobacteria        1.324            0.578
k__Bacteria;p__Actinobacteria       0.511           20.504
k__Bacteria;p__Bacteroidetes        0.335        2,739.699
k__Bacteria;p__Chloroflexi          0.879            1.975
k__Bacteria;p__Cyanobacteria        0.657            5.650
k__Bacteria;p__FBP                  1.572            0.548
k__Bacteria;p__Firmicutes           0.318        1,478.234
k__Bacteria;p__Fusobacteria         0.534           18.131
k__Bacteria;p__GN02                 1.008            1.349
k__Bacteria;p__Planctomycetes       1.430            0.603
k__Bacteria;p__Proteobacteria       0.312          223.096
k__Bacteria;p__SR1                  1.320            1.065
k__Bacteria;p__Spirochaetes         1.009            1.678
k__Bacteria;p__Synergistetes        1.213            1.156
k__Bacteria;p__TM7                  1.138            0.641
k__Bacteria;p__Tenericutes          0.976            1.290
k__Bacteria;p__Verrucomicrobia      0.391           23.187
k__Bacteria;p__[Thermi]             0.946            2.650

Effects:
                                                  Final Parameter   HDI 2.5%  \
Covariate         Cell Type
site[T.left palm] Unassigned;__                             0.000     -2.552
                  k__Archaea;p__Crenarchaeota               0.000     -2.417
                  k__Bacteria;__                            3.017      1.113
                  k__Bacteria;p__Acidobacteria              0.000     -3.468
                  k__Bacteria;p__Actinobacteria             3.273      2.233
...                                                           ...        ...
site[T.tongue]    k__Bacteria;p__Synergistetes              0.000     -3.292
                  k__Bacteria;p__TM7                        0.000     -0.385
                  k__Bacteria;p__Tenericutes                0.000     -2.198
                  k__Bacteria;p__Verrucomicrobia           -2.507     -4.732
                  k__Bacteria;p__[Thermi]                   0.000     -3.667

                                                  HDI 97.5%         SD  \
Covariate         Cell Type
site[T.left palm] Unassigned;__                       1.875      0.950
                  k__Archaea;p__Crenarchaeota         1.858      0.891
                  k__Bacteria;__                      5.493      1.157
                  k__Bacteria;p__Acidobacteria        1.922      1.181
                  k__Bacteria;p__Actinobacteria       4.422      0.577
...                                                     ...        ...
site[T.tongue]    k__Bacteria;p__Synergistetes        1.393      1.094
                  k__Bacteria;p__TM7                  3.696      1.193
                  k__Bacteria;p__Tenericutes          2.458      0.965
                  k__Bacteria;p__Verrucomicrobia      0.013      1.412
                  k__Bacteria;p__[Thermi]             1.533      1.327

                                                  Inclusion probability  \
Covariate         Cell Type
site[T.left palm] Unassigned;__                                   0.355
                  k__Archaea;p__Crenarchaeota                     0.489
                  k__Bacteria;__                                  1.000
                  k__Bacteria;p__Acidobacteria                    0.454
                  k__Bacteria;p__Actinobacteria                   1.000
...                                                                 ...
site[T.tongue]    k__Bacteria;p__Synergistetes                    0.553
                  k__Bacteria;p__TM7                              0.570
                  k__Bacteria;p__Tenericutes                      0.469
                  k__Bacteria;p__Verrucomicrobia                  0.946
                  k__Bacteria;p__[Thermi]                         0.586

                                                  Expected Sample  \
Covariate         Cell Type
site[T.left palm] Unassigned;__                             0.684
                  k__Archaea;p__Crenarchaeota               1.055
                  k__Bacteria;__                           53.304
                  k__Bacteria;p__Acidobacteria              0.634
                  k__Bacteria;p__Actinobacteria           593.197
...                                                           ...
site[T.tongue]    k__Bacteria;p__Synergistetes              1.100
                  k__Bacteria;p__TM7                        0.610
                  k__Bacteria;p__Tenericutes                1.228
                  k__Bacteria;p__Verrucomicrobia            1.800
                  k__Bacteria;p__[Thermi]                   2.524

                                                  log2-fold change
Covariate         Cell Type
site[T.left palm] Unassigned;__                              0.132
                  k__Archaea;p__Crenarchaeota                0.132
                  k__Bacteria;__                             4.484
                  k__Bacteria;p__Acidobacteria               0.132
                  k__Bacteria;p__Actinobacteria              4.855
...                                                            ...
site[T.tongue]    k__Bacteria;p__Synergistetes              -0.071
                  k__Bacteria;p__TM7                        -0.071
                  k__Bacteria;p__Tenericutes                -0.071
                  k__Bacteria;p__Verrucomicrobia            -3.688
                  k__Bacteria;p__[Thermi]                   -0.071

[63 rows x 7 columns]

We can also compare both palm microbiomes. We see 3 significances: Actinobacteria, Fusobacteria, Proteobacteria.
Looking at the boxplots, one can indeed see some change in these types.
Note that these are also 3 of the 4 most abundant types._____no_output_____

# Model with body site as covariate, only data from both palms, no baseline
model_palm = mod.CompositionalAnalysis(data_phy[data_phy.obs["site"].isin(["left palm", "right palm"])], "site", baseline_index=None)

result_palm = model_palm.sample_hmc(num_results=int(20000), n_burnin=5000)

print(result_palm.effect_df.loc[result_palm.effect_df["Final Parameter"] != 0])MCMC sampling finished. (216.915 sec)
Acceptance rate: 41.1%
                                                  Final Parameter     HDI 3%  \
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria           -0.879     -1.420
                   k__Bacteria;p__Fusobacteria             -0.850     -1.382
                   k__Bacteria;p__Proteobacteria           -0.844     -1.321

                                                    HDI 97%         SD  \
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria      0.000      0.375
                   k__Bacteria;p__Fusobacteria        0.000      0.402
                   k__Bacteria;p__Proteobacteria      0.001      0.358

                                                  Inclusion probability  \
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria                  0.952
                   k__Bacteria;p__Fusobacteria                    0.912
                   k__Bacteria;p__Proteobacteria                  0.948

                                                  Expected Sample  \
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria          276.299
                   k__Bacteria;p__Fusobacteria            141.310
                   k__Bacteria;p__Proteobacteria          685.298

                                                  log2-fold change
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria            -0.774
                   k__Bacteria;p__Fusobacteria              -0.732
                   k__Bacteria;p__Proteobacteria            -0.723
print("Significant changes versus the gut microbiome:")
print(result_gut.effect_df.loc[result_gut.effect_df["Final Parameter"] != 0, "Final Parameter"])

sns.set(style="ticks", font_scale=2)
viz.boxplot_facets(data_phy, "site", False)
plt.show()Significant changes versus the gut microbiome:
Covariate           Cell Type
site[T.left palm]   k__Bacteria;__                        3.017
                    k__Bacteria;p__Actinobacteria         3.273
                    k__Bacteria;p__Bacteroidetes         -1.946
                    k__Bacteria;p__Cyanobacteria          2.967
                    k__Bacteria;p__Fusobacteria           2.629
                    k__Bacteria;p__GN02                   2.348
                    k__Bacteria;p__Proteobacteria         1.718
                    k__Bacteria;p__Tenericutes            2.477
site[T.right palm]  k__Bacteria;__                        2.120
                    k__Bacteria;p__Acidobacteria          2.479
                    k__Bacteria;p__Actinobacteria         1.713
                    k__Bacteria;p__Bacteroidetes         -2.559
                    k__Bacteria;p__Cyanobacteria          2.162
                    k__Bacteria;p__Firmicutes            -1.191
                    k__Bacteria;p__Fusobacteria           1.295
site[T.tongue]      k__Bacteria;p__Actinobacteria         2.477
                    k__Bacteria;p__Bacteroidetes         -1.429
                    k__Bacteria;p__Fusobacteria           3.297
                    k__Bacteria;p__Proteobacteria         2.120
                    k__Bacteria;p__Verrucomicrobia       -2.507
Name: Final Parameter, dtype: float64

We observe multiple problems when dealing with very heterogenous data like here:

Repeatedly fitting the site-dependent model gives inconsistent results (not pictured in the notebook).
There are multiple phyla, where only two or three samples from the right or left palm are nonzero.
This leads to these samples being sometimes included, sometimes not.
Thus, some kind of sparsity correction should be considered in the future.

Also, types with high abundance seem to be prioritised over low-abundance types (see palm comparison).
Can we somehow mitigate this effect as well?

_____no_output_____### Comparison to other methods

We now run the palm comparison with ancom and ALDEx2 in qiime2.
Both models do not find any significant differences between the two palms on the phylum-level._____no_output_____## Analysis on the genus-level

We now want to see whether a more fine-grained analysis gives new insights. For this, we use the genus-level data.

First, we take a look at the between-subject comparison. We get a total of 9 significances._____no_output_____

# Model with subject as covariate
model_subject_gen = mod.CompositionalAnalysis(data_gen, "subject", baseline_index=None)

result_subject_gen = model_subject_gen.sample_hmc(num_results=int(20000), n_burnin=5000)MCMC sampling finished. (1082.019 sec)
Acceptance rate: 55.9%
print(result_subject_gen.effect_df.loc[result_subject_gen.effect_df["Final Parameter"] != 0])                                                                         Final Parameter  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Actinobacteria;c__Actinobacteria...            0.169
                     k__Bacteria;p__Actinobacteria;c__Coriobacteriia...            0.538
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...            0.409
                     k__Bacteria;p__Firmicutes;c__Bacilli;o__Bacilla...            0.249
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            0.337
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            0.532
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            0.369
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            0.274
                     k__Bacteria;p__Proteobacteria;c__Epsilonproteob...            0.449

                                                                            HDI 3%  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Actinobacteria;c__Actinobacteria...     -0.267
                     k__Bacteria;p__Actinobacteria;c__Coriobacteriia...      0.000
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...     -0.001
                     k__Bacteria;p__Firmicutes;c__Bacilli;o__Bacilla...     -0.093
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.000
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.148
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -0.034
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -0.032
                     k__Bacteria;p__Proteobacteria;c__Epsilonproteob...      0.000

                                                                           HDI 97%  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Actinobacteria;c__Actinobacteria...      0.521
                     k__Bacteria;p__Actinobacteria;c__Coriobacteriia...      0.976
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...      0.860
                     k__Bacteria;p__Firmicutes;c__Bacilli;o__Bacilla...      0.579
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.734
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.897
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.883
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.691
                     k__Bacteria;p__Proteobacteria;c__Epsilonproteob...      0.892

                                                                                SD  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Actinobacteria;c__Actinobacteria...      0.215
                     k__Bacteria;p__Actinobacteria;c__Coriobacteriia...      0.285
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...      0.261
                     k__Bacteria;p__Firmicutes;c__Bacilli;o__Bacilla...      0.172
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.203
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.209
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.248
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.210
                     k__Bacteria;p__Proteobacteria;c__Epsilonproteob...      0.276

                                                                         Inclusion probability  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Actinobacteria;c__Actinobacteria...                  0.985
                     k__Bacteria;p__Actinobacteria;c__Coriobacteriia...                  0.967
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...                  0.963
                     k__Bacteria;p__Firmicutes;c__Bacilli;o__Bacilla...                  0.992
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...                  0.960
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...                  1.000
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...                  0.951
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...                  0.954
                     k__Bacteria;p__Proteobacteria;c__Epsilonproteob...                  0.962

                                                                         Expected Sample  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Actinobacteria;c__Actinobacteria...            1.918
                     k__Bacteria;p__Actinobacteria;c__Coriobacteriia...           19.463
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...           15.016
                     k__Bacteria;p__Firmicutes;c__Bacilli;o__Bacilla...            3.918
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           57.755
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           76.684
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            8.158
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           46.823
                     k__Bacteria;p__Proteobacteria;c__Epsilonproteob...           80.049

                                                                         log2-fold change
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Actinobacteria;c__Actinobacteria...             0.210
                     k__Bacteria;p__Actinobacteria;c__Coriobacteriia...             0.743
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...             0.556
                     k__Bacteria;p__Firmicutes;c__Bacilli;o__Bacilla...             0.326
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...             0.452
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...             0.733
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...             0.498
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...             0.362
                     k__Bacteria;p__Proteobacteria;c__Epsilonproteob...             0.615

We now take only samples from the gut (same thing as in the "Moving Pictures"-tutorial) and compare the two subjects.

SCDCdm finds 10 significant effects, while ancom only finds 2 and ALDEx2 finds three_____no_output_____

# Model with subject as covariate
model_gut_gen = mod.CompositionalAnalysis(data_gen[data_gen.obs["site"] == "gut"], "subject", baseline_index=None)

result_gut_gen = model_gut_gen.sample_hmc(num_results=int(20000), n_burnin=5000)MCMC sampling finished. (246.680 sec)
Acceptance rate: 51.6%
print(result_gut_gen.effect_df.loc[result_gut_gen.effect_df["Final Parameter"] != 0])                                                                         Final Parameter  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...           -0.905
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...           -2.795
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           -1.255
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           -1.220
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           -1.261
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           -1.379
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           -1.645
                     k__Bacteria;p__Proteobacteria;c__Deltaproteobac...           -1.107
                     k__Bacteria;p__Proteobacteria;c__Gammaproteobac...           -1.591
                     k__Bacteria;p__Verrucomicrobia;c__Verrucomicrob...           -2.011

                                                                            HDI 3%  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...     -1.257
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...     -4.176
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -2.065
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -1.755
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -2.018
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -2.008
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -3.253
                     k__Bacteria;p__Proteobacteria;c__Deltaproteobac...     -1.733
                     k__Bacteria;p__Proteobacteria;c__Gammaproteobac...     -2.509
                     k__Bacteria;p__Verrucomicrobia;c__Verrucomicrob...     -2.806

                                                                           HDI 97%  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...     -0.539
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...     -1.438
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -0.468
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -0.000
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -0.000
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -0.823
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -0.218
                     k__Bacteria;p__Proteobacteria;c__Deltaproteobac...      0.000
                     k__Bacteria;p__Proteobacteria;c__Gammaproteobac...     -0.000
                     k__Bacteria;p__Verrucomicrobia;c__Verrucomicrob...     -1.272

                                                                                SD  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...      0.194
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...      0.784
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.444
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.429
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.543
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.317
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.852
                     k__Bacteria;p__Proteobacteria;c__Deltaproteobac...      0.485
                     k__Bacteria;p__Proteobacteria;c__Gammaproteobac...      0.692
                     k__Bacteria;p__Verrucomicrobia;c__Verrucomicrob...      0.433

                                                                         Inclusion probability  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...                  1.000
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...                  1.000
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...                  1.000
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...                  0.962
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...                  0.958
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...                  1.000
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...                  0.998
                     k__Bacteria;p__Proteobacteria;c__Deltaproteobac...                  0.960
                     k__Bacteria;p__Proteobacteria;c__Gammaproteobac...                  0.953
                     k__Bacteria;p__Verrucomicrobia;c__Verrucomicrob...                  1.000

                                                                         Expected Sample  \
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...        3,404.823
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...           10.852
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           79.139
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           53.433
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           22.617
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...          189.030
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            2.527
                     k__Bacteria;p__Proteobacteria;c__Deltaproteobac...           43.622
                     k__Bacteria;p__Proteobacteria;c__Gammaproteobac...            0.212
                     k__Bacteria;p__Verrucomicrobia;c__Verrucomicrob...           19.354

                                                                         log2-fold change
Covariate            Cell Type
subject[T.subject-2] k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...            -0.418
                     k__Bacteria;p__Bacteroidetes;c__Bacteroidia;o__...            -3.145
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            -0.923
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            -0.872
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            -0.932
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            -1.102
                     k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            -1.486
                     k__Bacteria;p__Proteobacteria;c__Deltaproteobac...            -0.709
                     k__Bacteria;p__Proteobacteria;c__Gammaproteobac...            -1.408
                     k__Bacteria;p__Verrucomicrobia;c__Verrucomicrob...            -2.014
print(result_gut_gen.effect_df.loc[result_gut_gen.effect_df["Inclusion probability"] == 0])Empty DataFrame
Columns: [Final Parameter, HDI 3%, HDI 97%, SD, Inclusion probability, Expected Sample, log2-fold change]
Index: []

Finally, we take a look at how the two palms differ in SCDCdm. We find 7 significantly differing genera.
Here, neither ancom, nor ALDEx2 find any significant changes.

Astoundingly, no genus from the Fusobacteria phylum was detected despite Fusobacteria being detected as differing on the phylum level.
For the other two differing phyla (Actinobacteria, Proteobacteria), we have two differing genera each, while 3 more genera from other phyla were selected._____no_output_____

# Model with body site as covariate, only data from both palms, no baseline
model_palm_gen = mod.CompositionalAnalysis(data_gen[data_gen.obs["site"].isin(["left palm", "right palm"])], "site", baseline_index=None)

result_palm_gen = model_palm_gen.sample_hmc(num_results=int(20000), n_burnin=5000)MCMC sampling finished. (432.620 sec)
Acceptance rate: 60.1%
print(result_palm_gen.effect_df.loc[result_palm_gen.effect_df["Final Parameter"] != 0])

                                                                       Final Parameter  \
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria;c__Actinobacteria...           -0.620
                   k__Bacteria;p__Actinobacteria;c__Coriobacteriia...           -0.528
                   k__Bacteria;p__Bacteroidetes;c__Flavobacteriia;...           -0.584
                   k__Bacteria;p__Firmicutes;c__Bacilli;o__Lactoba...           -0.569
                   k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...           -0.552
                   k__Bacteria;p__Proteobacteria;c__Epsilonproteob...           -0.538
                   k__Bacteria;p__Proteobacteria;c__Gammaproteobac...           -0.554

                                                                          HDI 3%  \
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria;c__Actinobacteria...     -0.976
                   k__Bacteria;p__Actinobacteria;c__Coriobacteriia...     -0.897
                   k__Bacteria;p__Bacteroidetes;c__Flavobacteriia;...     -1.120
                   k__Bacteria;p__Firmicutes;c__Bacilli;o__Lactoba...     -1.002
                   k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -0.793
                   k__Bacteria;p__Proteobacteria;c__Epsilonproteob...     -0.774
                   k__Bacteria;p__Proteobacteria;c__Gammaproteobac...     -0.901

                                                                         HDI 97%  \
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria;c__Actinobacteria...     -0.297
                   k__Bacteria;p__Actinobacteria;c__Coriobacteriia...     -0.141
                   k__Bacteria;p__Bacteroidetes;c__Flavobacteriia;...     -0.257
                   k__Bacteria;p__Firmicutes;c__Bacilli;o__Lactoba...     -0.259
                   k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...     -0.000
                   k__Bacteria;p__Proteobacteria;c__Epsilonproteob...     -0.317
                   k__Bacteria;p__Proteobacteria;c__Gammaproteobac...     -0.000

                                                                              SD  \
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria;c__Actinobacteria...      0.185
                   k__Bacteria;p__Actinobacteria;c__Coriobacteriia...      0.180
                   k__Bacteria;p__Bacteroidetes;c__Flavobacteriia;...      0.212
                   k__Bacteria;p__Firmicutes;c__Bacilli;o__Lactoba...      0.196
                   k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...      0.195
                   k__Bacteria;p__Proteobacteria;c__Epsilonproteob...      0.137
                   k__Bacteria;p__Proteobacteria;c__Gammaproteobac...      0.234

                                                                       Inclusion probability  \
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria;c__Actinobacteria...                  0.962
                   k__Bacteria;p__Actinobacteria;c__Coriobacteriia...                  0.972
                   k__Bacteria;p__Bacteroidetes;c__Flavobacteriia;...                  0.975
                   k__Bacteria;p__Firmicutes;c__Bacilli;o__Lactoba...                  1.000
                   k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...                  0.977
                   k__Bacteria;p__Proteobacteria;c__Epsilonproteob...                  0.987
                   k__Bacteria;p__Proteobacteria;c__Gammaproteobac...                  0.957

                                                                       Expected Sample  \
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria;c__Actinobacteria...          192.959
                   k__Bacteria;p__Actinobacteria;c__Coriobacteriia...            1.681
                   k__Bacteria;p__Bacteroidetes;c__Flavobacteriia;...           17.921
                   k__Bacteria;p__Firmicutes;c__Bacilli;o__Lactoba...          253.870
                   k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            0.789
                   k__Bacteria;p__Proteobacteria;c__Epsilonproteob...           19.494
                   k__Bacteria;p__Proteobacteria;c__Gammaproteobac...            2.016

                                                                       log2-fold change
Covariate          Cell Type
site[T.right palm] k__Bacteria;p__Actinobacteria;c__Actinobacteria...            -0.747
                   k__Bacteria;p__Actinobacteria;c__Coriobacteriia...            -0.614
                   k__Bacteria;p__Bacteroidetes;c__Flavobacteriia;...            -0.695
                   k__Bacteria;p__Firmicutes;c__Bacilli;o__Lactoba...            -0.674
                   k__Bacteria;p__Firmicutes;c__Clostridia;o__Clos...            -0.649
                   k__Bacteria;p__Proteobacteria;c__Epsilonproteob...            -0.629
                   k__Bacteria;p__Proteobacteria;c__Gammaproteobac...            -0.652


# Notebook from rcurrie/variant2vec
Path: train.ipynb

# variant2vec

Learn an emedding of variants annotations (HGVS, BIC etc...), Hugo gene symbols, and classifications (pathogenic, benign...) from the text of pubmed articles.

Goal to see if something like [this](https://www.nature.com/articles/s41586-019-1335-8) paper can be applied to determine characteristics of a variant, or at lease identify the gene associated with the variant from the paper where multiple variants may be mentioned with no canonical gene or transcript qualifier._____no_output_____

import os
import sys
import re
import sqlite3
import pandas as pd
import gzip
import glob
import datetime

def log(text):
    # Print to jupyter notebook and console so we can monitor progress in k8s
    print(text)
    sys.__stdout__.write(text + "\n")
    sys.__stdout__.flush()

# Syntatic sugar for debug vs. train parameters
def debug(debug_param, no_debug_param):
    return debug_param if os.environ.get("DEBUG") == "True" else no_debug_param
log(debug("DEBUG: ON", "DEBUG: OFF"))

crawl_path = "/public/groups/brcaexchange/literature-search/crawl"DEBUG: ON

## Ingest
Load the text from the pubMunch [pubStore](https://github.com/maximilianh/pubMunch/blob/master/lib/pubStore.py) generated in a BRCA Exchange literature search crawl_____no_output_____

connection = sqlite3.connect(f"file:{crawl_path}/text/articles.db?mode=ro", uri=True)
articles = pd.read_sql_query("SELECT * FROM articles", connection)
articles.pmid = articles.pmid.astype(str)
print("{} articles loaded from the articles sqlite database".format(articles.shape[0]))
articles.head()15079 articles loaded from the articles sqlite database
%%time
log("Loading documents...")
articles = pd.DataFrame()
for path in glob.glob(f"{crawl_path}/text/*.files.gz"):
    articles = articles.append(pd.read_csv(path, sep="\t", encoding="utf-8"))

print("{} article text blocks consuming {:.2f} megabytes".format(
    articles.shape[0], articles.memory_usage(index=True).sum() / 2**20))28938 article text blocks consuming 2.43 megabytes
CPU times: user 28 s, sys: 2.87 s, total: 30.8 s
Wall time: 31.4 s
articles.head()_____no_output_____%%time
variants = set()
pattern = re.compile("c\.\d+[atcgATCG]+>[atcgATCG]+")
for article in articles.itertuples():
    variants = variants.union(set(re.findall(pattern, article.content)))
print("Found {} variants".format(len(variants)))Found 6460 variants
CPU times: user 7.32 s, sys: 373 ms, total: 7.69 s
Wall time: 7.69 s

## Clean
https://machinelearningmastery.com/clean-text-machine-learning-python/_____no_output_____

!pip3 install --user --upgrade --quiet unidecode
import unidecode

!pip3 install --user --upgrade --quiet nltk
import nltk
import nltk.tokenize
import nltk.stem.porter

nltk.download("stopwords")
stop_words = nltk.corpus.stopwords.words('english')

stemmer = nltk.stem.porter.PorterStemmer()[nltk_data] Downloading package stopwords to /tf/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
log("Cleaning...")
word_regex = re.compile("\s+|[,\(\)]")

def clean(text):
    # Convert to ascii and replace returns with spaces and lower case
    text = re.sub(r"[\x07]+", ' ', unidecode.unidecode(text)).lower()

    # split into words
#     words = nltk.tokenize.word_tokenize(text)

    words = word_regex.split(text)

    # Drop all single character words and stop words - effectively removes punctuation
    words = [w for w in words if len(w) > 3 and w not in stop_words]

    # Stem (lowercase's as a side affect as well)
    words = [stemmer.stem(w) for w in words]

    return words

print(clean(articles.iloc[42].content[0:2000]))['cancer:', 'breast', 'introduct', 'genet', 'test', 'famili', 'multipl', 'case', 'breast', 'and/or', 'ovarian', 'cancer', 'often', 'target', 'youngest', 'affect', 'woman', 'index', 'case', 'family.', 'clinic', 'genet', 'test', 'context', 'larg', 'limit', 'brca1', 'brca2', 'gene', 'unless', 'addit', 'indic', 'present', 'recently.', 'women', 'breast', 'cancer', 'test', 'uninform', 'identifi', 'clearli', 'pathogen', 'mutat', 'either', 'gene.1', 'mani', 'put', 'breast', 'cancer', 'suscept', 'gene', 'identifi', 'vari', 'level', 'evid', 'associ', 'breast', 'cancer.', 'today', 'diagnost', 'test', 'facil', 'includ', 'larg', 'number', 'gene', 'singl', 'panel', 'test', 'use', 'massiv', 'parallel', 'next', 'gener', 'sequenc', 'consider', 'reduc', 'cost.', 'howev', 'gene', 'panel', 'test', 'pose', 'consider', 'challeng', 'clinic', 'genet', 'servic', 'mani', 'gene', 'valid', 'breast', 'cancer', 'suscept', 'gene', 'even', 'risk', 'associ', 'differ', 'type', 'mutat', 'poorli', 'defined.2', 'test', 'use', 'estim', 'reduct', 'risk', 'mutat', 'carrier', 'known', 'estim', 'risk', 'cancer', 'non-mut', 'carrier', 'family.', 'gene', 'current', 'includ', 'commerci', 'breast', 'cancer', 'suscept', 'gene', 'panel', 'addit', 'brca1', 'brca2', 'vari', 'laboratori', 'compani', 'rang', 'reason', 'well-characteris', 'breast', 'cancer', 'predisposit', 'gene', 'tp53', 'palb2', 'chek2', 'cdh1', 'pten', 'stk11', 'recur', 'mutat', 'nbn3', 'gene', 'limit', 'evid', 'mutat', 'confer', 'elev', 'breast', 'cancer', 'risk', 'bard1', 'brip1', 'mre11', 'rad50', 'rad51c', 'rad51d', 'cdkn2a', 'xrcc2', 'gene', 'select', 'panel', 'appear', 'driven', 'commerci', 'interest', 'rather', 'strength', 'evid', 'role', 'breast', 'cancer', 'susceptibility.', 'vital', 'verifi', 'brea']
%%time
texts = [clean(article.content) for article in articles.iloc[0:None:debug(100,None)].itertuples()]
print("Cleaned {} texts".format(len(texts)))Cleaned 290 texts
CPU times: user 37.5 s, sys: 966 ms, total: 38.5 s
Wall time: 38.5 s
print("Before:")
print(articles.iloc[2].content[0:2000])

print("After:")
print(texts[2][0:250])Before:
PRECLINICAL STUDYAn integrative genomic and transcriptomic analysis revealsmolecular pathways and networks regulated by copy numberaberrations in basal-like, HER2 and luminal cancersRachael Natrajan Æ Britta Weigelt Æ Alan Mackay Æ Felipe C. Geyer ÆAnita Grigoriadis Æ David S. P. Tan Æ Chris Jones Æ Christopher J. Lord ÆRadost Vatcheva Æ Socorro M. Rodriguez-Pinilla Æ Jose Palacios ÆAlan Ashworth Æ Jorge S. Reis-FilhoReceived: 24 July 2009 / Accepted: 28 July 2009 / Published online: 18 August 2009  Springer Science+Business Media, LLC. 2009Abstract Breast cancer is a heterogeneous diseasecaused by the accumulation of genetic changes in neo-plastic cells. We hypothesised that molecular subtypes ofbreast cancer may be driven by specific constellations ofgenes whose expression is regulated by gene copynumber aberrations. To address this question, we ana-lysed a series of 48 microdissected grade III ductalcarcinomas using high resolution microarray comparativegenomic hybridisation and mRNA expression arrays.There were 5,931 genes whose expression significantlycorrelates with copy number identified; out of these,1,897 genes were significantly differentially expressedbetween basal-like, HER2 and luminal tumours. Inge-nuity Pathway Analysis (IPA) revealed that ‘G1/S cellcycle regulation’ and ‘BRCA1 in DNA damage control’pathways were significantly enriched for genes whoseexpression correlates with copy number and are differ-entially expressed between the molecular subtypes ofbreast cancer. IPA of genes whose expression signifi-cantly correlates with copy number in each molecularsubtype individually revealed that canonical pathwaysinvolved in oestrogen receptor (ER) signalling and DNArepair are enriched for these genes. We also identified32, 157 and 265 genes significantly overexpressed whenamplified in basal-like, HER2 and luminal cancers,respectively. These lists include known and novelpotential therapeutic targets (e.g. HER2 and PPM1D
After:
['cancer', 'letter', '2001', '87+-94', 'www.elsevier.com/locate/canlet', 'germlin', 'brca1', 'mutat', 'iranian', 'women', 'breast', 'cancer', 'ghaderi', 'talei', 'farjadian', 'mosalaei', 'doroudchi', 'kimura', 'depart', 'immunolog', 'medic', 'school', 'shiraz', 'univers', 'medic', 'scienc', 'p.o.', '71345-1798', 'shiraz', 'iran', 'depart', 'surgeri', 'shiraz', 'univers', 'medic', 'scienc', 'shiraz', 'iran', 'depart', 'radiat', 'therapi', 'shiraz', 'univers', 'medic', 'scienc', 'shiraz', 'iran', 'depart', 'forens', 'medicin', 'kurum', 'univers', 'kurum', 'japan', 'receiv', 'novemb', '2000;', 'receiv', 'revis', 'form', 'januari', '2001;', 'accept', 'januari', '2001', 'abstract', 'brca1', 'known', 'famili', 'earli', 'onset', 'breast', 'cancer', 'gene', 'locat', 'long', 'chromosom', 'alter', 'gene', 'report', 'differ', 'popul', 'population-speci', 'mutat', 'show', 'founder', 'effect.', 'studi', 'report', 'genet', 'test', 'iranian', 'women', 'exon-2', 'brca1', 'gene', 'sequenc', 'iranian', 'breast', 'cancer', 'patient', 'none', 'patient', 'show', '185del', 'mutat', 'chang', 'sequenc', 'exon.', 'among', 'popul', 'patient', 'kind', 'tumor', 'rst-degre', 'rel', 'select', 'entir', 'brca1', 'gene', 'code', 'region', 'sequenc', 'direct', 'pcr-sequencing.', 'novel', 'point', 'mutat', 'exon-16', 'unknown', 'signi', 'canc', 'observ', 'among', 'breast', 'cancer', 'patient', 'control', 'subjects.', 'mutat', 'caus', 'substitut', 'glutamin', '1612', 'glycin', 'allel', 'frequenc', '38.6', '52.8%', 'patient', 'control', 'respectively.', 'addit', 'point', 'mutat', 'exon-15', 'eight', 'polymorph', 'alter', 'detect', 'report', 'previously.', 'three', 'polymorph', 'site', 'place', 'intron', 'part', 'gene.', 'understand', 'signi', 'canc', 'contribut', 'brca1', 'gene', 'breast', 'cancer', 'among', 'iranian', 'investig', 'needed.', '2001', 'elsevi', 'scienc', 'ireland', 'ltd.', 'right', 'reserved.', 'keywords:', 'brca1;', 'breast', 'cancer;', '185del', 'mutation;', 'iranian;', 'polymorph', 'introduct', 'brca1', 'known', 'potenti', 'tumor', 'suppressor', 'gene', 'length', 'approxim', 'locat', 'long', 'chromosom', 'made', 'exon', 'code', 'polypeptid', '1863', 'amino', 'acid', 'residu', '[1].', 'zinc', 'nger', 'motif', 'encod', 'exon', 'respons', 'transcript', 'activ', 'genes.', 'middl', 'part', 'protein', 'encod', 'exon-11', 'correspond', 'author.', 'tel.:', '198-71-334589;', 'fax:', '198-71334589.', 'e-mail', 'address:', 'immunol@sums.ac.ir', 'ghaderi', 'interact', 'rad51', 'molecul', 'human', 'counterpart', 'bacteri', 'reca', 'protein', '[2].']

## Train

References:

https://radimrehurek.com/gensim/tut1.html

https://machinelearningmastery.com/develop-word-embeddings-python-gensim/_____no_output_____

!pip3 install --user --upgrade --quiet gensim

import gensim
import gensim.models_____no_output_____# model = gensim.models.Word2Vec(documents, size=150, window=10, min_count=2, workers=10, iter=10)
log("Training...")
model = gensim.models.Word2Vec(texts, max_vocab_size=5000)_____no_output_____# Timestamp for output files
log("Saving...")
timestamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
print("Training run at {}".format(timestamp))

path = os.path.expanduser("~/data/variante2vec/{}.model").format(debug("debug", timestamp))
print("Saved to:",path)

model.save(path)
model = gensim.models.Word2Vec.load(path)Training run at 20190913-145341
/tf/data/variante2vec/debug.model


# Notebook from celinedurniak/test_nbsphinx
Path: docs/examples/bumps_ChudleyElliotDiff_fit.ipynb

Disclaimer:

The main objective of the Jupyter notebooks is to show how to use the models of the QENS library by

- building a fitting model: composition of models, convolution with a resolution function
- setting and running the fit
- extracting and displaying information about the results

These steps have a minimizer-dependent syntax. That's one of the reasons why different minimizers have been used in the notebooks provided as examples.
But, the initial guessed parameters might not be optimal, resulting in a poor fit of the reference data.

# Chudley-Elliot diffusion with bumps

## Table of Contents

- [Introduction](#introduction)
- [Import and install required libraries](#Import-and-install-required-libraries)
- [Setting of fitting](#Setting-of-fitting)
- [Running the fit](#Running-the-fit)
- [Showing the results](#Showing-the-results)_____no_output_____[Top](#Table-of-Contents)

## Introduction

The objective of this notebook is to show how to use the Chudley Elliot diffusion model to perform some
fits using bumps .

_____no_output_____[Top](#Table-of-Contents)

## Import and install required libraries _____no_output_____

# Imported required libraries
from __future__ import print_function
import numpy as np_____no_output_____import ipywidgets_____no_output_____# install bumps (if not already installed)
import sys
import os
import pkgutil
if not pkgutil.find_loader("bumps"):
    bumpsY = ipywidgets.Button(description='Yes', button_style='success')
    bumpsN = ipywidgets.Button(description='No', button_style='danger')
    choice_installation = ipywidgets.VBox(
    [ipywidgets.Label("Do you want to install bumps?"), ipywidgets.HBox([bumpsY, bumpsN])],
    layout=ipywidgets.Layout(width='30%', height='80px'))

    display(choice_installation)

    def on_bumpsY_clicked(b):
        !{sys.executable} -m pip install bumps==0.7.12
    def on_bumpsN_clicked(b):
        print("You will not be able to run some of the remaining parts of this notebook")

    bumpsY.on_click(on_bumpsY_clicked)
    bumpsN.on_click(on_bumpsN_clicked)_____no_output_____import bumps

# check version of bumps installed
# Information message if installed version not recent enough
from distutils.version import StrictVersion
if StrictVersion(bumps.__version__) <= StrictVersion('0.7.6'):
    print("""The version of bumps installed is not recent
    enough to run the examples.
    Please update bumps. The minimum version required is 0.7.8""")_____no_output_____import bumps.names as bmp
from bumps.fitters import fit_____no_output_____

[Top](#Table-of-Contents)

## Setting of fitting

### install QENSmodels (if not already installed)_____no_output_____

if not pkgutil.find_loader("QENSmodels"):
    buttonY = ipywidgets.Button(description='Yes', button_style='success')
    buttonN = ipywidgets.Button(description='No', button_style='danger')
    choice_installation = ipywidgets.VBox(
    [ipywidgets.Label("Do you want to install the QENSmodels' library?"), ipywidgets.HBox([buttonY, buttonN])],
    layout=ipywidgets.Layout(width='50%', height='80px'))
    display(choice_installation)

    def on_buttonY_clicked(b):
        !{sys.executable} -m pip install git+https://github.com/QENSlibrary/QENSmodels#egg=QENSmodels

    def on_buttonN_clicked(b):
        print("You will not be able to run some of the remaining parts of this notebook")

    buttonY.on_click(on_buttonY_clicked)
    buttonN.on_click(on_buttonN_clicked) _____no_output_____import QENSmodels_____no_output_____

###  create reference data_____no_output_____

nb_points = 500
xx = np.linspace(-4, 4, nb_points)
q = np.linspace(0.2, 2, 10)
added_noise = 0.1 * np.random.normal(0, 1, nb_points)
chudley_elliot_noisy = QENSmodels.sqwChudleyElliotDiffusion(xx,
                                                            q,
                                                            scale=1.,
                                                            center=0.,
                                                            D=0.23,
                                                            L=1.) * (1. + added_noise)
chudley_elliot_noisy += 0.1 * added_noise_____no_output_____

### Physical units
For information about unit conversion, please refer to the jupyter notebook called `Convert_units.ipynb` in the `tools` folder.

The dictionary of units defined in the cell below specify the units of the refined parameters adapted to the convention used in the experimental datafile._____no_output_____

# Units of parameters for selected QENS model and experimental data
dict_physical_units = {'omega': "1/ps",
                       'q': "1/Angstrom",
                       'D': "ps.Angstrom^2",
                       'L': "Angstrom",
                       'scale': "unit_of_signal/ps",
                       'center': "1/ps"}_____no_output_____

### create fitting model_____no_output_____

M = []
for i in range(len(q)):
    # Bumps fitting model
    Mq = bmp.Curve(QENSmodels.sqwChudleyElliotDiffusion,
                   xx,
                   chudley_elliot_noisy[i],
                   q[i],
                   scale=1,
                   center=0,
                   D=0.2,
                   L=0.7)

    Mq.scale.range(0.1, 1e5)
    Mq.center.range(-0.1, 0.1)
    Mq.D.range(0.1, 1)
    Mq.L.range(0.1, 3)

    # Q-independent parameters
    if i == 0:
        QD = Mq.D
        QL = Mq.L
    else:
        Mq.D = QD
        Mq.L = QL

    M.append(Mq)

problem = bmp.FitProblem(M)_____no_output_____

### Choice of minimizer for bumps_____no_output_____

options_dict={'Levenberg-Marquardt': "lm",
             'Nelder-Mead Simplex': "amoeba",
             'DREAM': "dream",
             'Differential Evolution': "de",
             'Quasi-Newton BFGS': "newton",
             'Random Lines (experimental)': "rl",
             'Particle Swarm (experimental)': "ps",
             'Parallel Tempering (experimental)': "pt"}

w_choice_minimizer = ipywidgets.Dropdown(
    options=list(options_dict.keys()),
    value='Levenberg-Marquardt',
    description='Minimizer:',
    layout=ipywidgets.Layout(height='40px')
)
w_choice_minimizer_____no_output_____

### Setting for running bumps_____no_output_____

 steps_fitting = ipywidgets.IntText(
    value=100,
    description='Number of steps when fitting',
    style={'description_width': 'initial'})

steps_fitting_____no_output_____# Preview of the settings
print('Initial chisq', problem.chisq_str())_____no_output_____problem.plot()_____no_output_____

[Top](#Table-of-Contents)

## Running the fit

Run the fit using the *minimizer* defined above with a number of *steps* also specified above_____no_output_____

result = fit(problem,
             method=options_dict[w_choice_minimizer.value],
             steps=int(steps_fitting.value),
             verbose=True)_____no_output_____

[Top](#Table-of-Contents)

## Showing the results_____no_output_____

problem.plot()_____no_output_____# Print chi**2 and parameters' values after fit
print("final chisq", problem.chisq_str())

for k, v, dv in zip(problem.labels(), result.x, result.dx):
    if k in dict_physical_units.keys():
        print(k, ":", v, dv, dict_physical_units[k])
    else:
        print(k, ":", v, dv)_____no_output_____