# Notebook from Jayshil/pycheops-tutorials
Path: WASP-189/Phase_curves/p1_wasp189_phase_curve.ipynb

# Analysis of WASP-189 (Phase Curve) using `pycheops`

In the following notebook, we will analyse the data from `CHEOPS` visit 1 data of HD106315b using `pycheops`.

The method is similar to that used in the analysis of KELT-11b data ([https://github.com/Jayshil/pycheops-tutorials/blob/main/KELT-11/p2_kelt11_extra_decorr_parm.ipynb](https://github.com/Jayshil/pycheops-tutorials/blob/main/KELT-11/p2_kelt11_extra_decorr_parm.ipynb))._____no_output_____

import numpy as np
import matplotlib.pyplot as plt
from astropy.io import fits
from dace.cheops import Cheops
import pycheops
from uncertainties import ufloat
from uncertainties.umath import sqrt as usqrt
from pycheops.models import ThermalPhaseModel, ReflectionModel
import astropy.units as u
from lmfit import minimize
from uncertainties import UFloat
import re
from pycheops.utils import phaser_____no_output_____

## Downloading the data_____no_output_____

# Downloading data
dd = pycheops.Dataset('CH_PR100036_TG000701_V0200')
tim, fl, fle = dd.get_lightcurve(aperture='DEFAULT', decontaminate=True)#, reject_highpoints=True)

# To clip outliers (I would, in general, not prefer using this)
tim, fl, fle = dd.clip_outliers(verbose=True);

# Plotting the data
plt.figure(figsize=(16,9))
plt.errorbar(tim, fl, yerr=fle, fmt='.', c='orangered')
plt.xlabel('Time (BJD)')
plt.ylabel('Normalised Flux')
plt.title('Transit lightcurve for WASP-189b (Aperture: DEFAULT)')
plt.grid()Found archive tgzfile /home/jayshil/.local/share/pycheops/CH_PR100036_TG000701_V0200.tgz
 PI name     : Brice-Olivier DEMORY
 OBS ID      : 1145201
 Target      : WASP-189
 Coordinates : 15:02:44.87 -03:01:53.0
 Spec. type  : A6V
 G magnitude : 6.55 +- 0.00
Light curve data loaded from  /home/jayshil/.local/share/pycheops/CH_PR100036_TG000701_V0200-DEFAULT.fits
Time stored relative to BJD = 2459016
Aperture radius used = 25 arcsec
UTC start:  2020-06-15T18:07:40
UTC end:    2020-06-18T21:07:01
Visit duration: 269961 s
Exposure time: 7 x 4.8 s
Number of non-flagged data points: 4758
Efficiency (non-flagged data): 59.2 %
Light curve corrected for flux from background stars
Mean counts = 140556171.2
Median counts = 140629390.3
RMS counts = 222219.7 [1580 ppm]
Median standard error = 11901.2 [85 ppm]
Median background = 262452 e-/pxl
Mean contamination = 733.0 ppm
Mean smearing correction = 60.4 ppm

Rejected 12 points more than 5.0 x MAD = 356 ppm from the median

### Planetary check

We may want to check the position of various planetary bodies with respect to the target star, because if some of the bodies is very near to the target star then we may want to correct for their effects._____no_output_____

dd.planet_check()BJD = 2459016.260447496
Body     R.A.         Declination  Sep(deg)
-------------------------------------------
Moon     01:28:57.05  +04:05:47.0      42.4
Mars     23:35:07.42  -05:41:00.0      82.0
Jupiter  19:49:36.55  -21:18:28.3      66.3
Saturn   20:11:47.02  -20:10:17.7      73.7
Uranus   02:26:57.74  +14:03:48.3     165.0
Neptune  23:27:22.67  -04:40:49.8     123.8

## Stellar and Planetary properties

A knowledge of the stellar and planetary property of the system under investigation can proved to be useful while analysing the system. We can use some of the properties as priors in our analysis. In `pycheops` there are some in-build tools  available which ca go through the available stellar and planetary catalogues (like SWEET-Cat, or DACE) to give us the properties. Below, we show how to access such properties._____no_output_____

# For planetary properties
# This cell is copied from pycheops examples notebook
# The planet properties will not provide good results
# Hence, we will use ExoCTK to get planetary parameters.
"""
try:
    hd106315b = pycheops.PlanetProperties('HD 106315b',query_dace=True)
except:
    hd106315b = pycheops.PlanetProperties('HD 106315b',query_dace=False, query_tepcat=False,
                                    depth = ufloat(298.909, 20.791),   # PPM
                                    width = ufloat(0.1572,0.0034),  # days (that means that is transit duration -- in days)
                                    ecosw=0.,#ufloat(-0.0040, 0.0490),
                                    esinw=0.,#ufloat(0.0310, 0.0680),
                                    T0 = ufloat(2457586.5487, 0.0029),
                                    P = ufloat(9.55237, 0.00089))

print(hd106315b)
"""
# Using ExoCTK to get planetary properties
rprs = np.random.normal(0.060907361661750425, 0.003144594526637345, 10000)
dep, dep_err = np.mean((rprs**2)*1e6), np.std((rprs**2)*1e6)_____no_output_____

Well..., apparently, I cannot (even after studying two semesters at Geneva) access DACE-Planets. No problem --- we have TEPCat at our service. We derived parameters from there. We can now try searching for the host star._____no_output_____

"""
star = pycheops.StarProperties(dd.target)
print(star)
"""
teff = np.random.normal(8000, 80, 10000)
logg = np.random.normal(3.9, 0.2, 10000)
mh = np.random.normal(0.29, 0.13, 10000)
h1, h2 = np.zeros(10000), np.zeros(10000)
cdc = pycheops.ld.stagger_power2_interpolator('CHEOPS')
for i in range(len(teff)):
    c1, c2, h1[i], h2[i] = cdc(teff[i], logg[i], mh[i])_____no_output_____

It seems that the program would return nan to h1 and h2 because the temperature is out of range of the grid. So, we will let free the h1 and h2 in the analysis._____no_output_____We have it! Here `h_1` and `h_2` are power2 law limb darkening coefficients._____no_output_____## Renormalising the lightcurve

It already seems pretty flatten -- and applying the function would only reduce the quality of the data._____no_output_____

# This code is taken from the pycheops example notebook
P = 2.7240338
BJD_0 = 2456706.4558
cycle = round((dd.bjd_ref-BJD_0)/P)
T_0 = BJD_0 - dd.bjd_ref + (cycle+0)*P
T_1 = BJD_0 - dd.bjd_ref + (cycle+1)*P
T_ec = T_0 + (P/2)

D = dep/1e6  # Depth stored in ppm
W = 0.1819/P    # Width stored in days

try:
    f_c = 0.#np.sqrt(0.093)*np.sin(67.0*np.pi/180)
    f_s = 0.#np.sqrt(0.093)*np.cos(67.0*np.pi/180)
except:
    # From Pepper et al., 2017
    ecosw = 0.#ufloat(-0.004,0.05099)
    esinw = 0.#ufloat(0.031,0.055)
    ecc = usqrt(ecosw**2+esinw**2)
    f_s = 0.#esinw/usqrt(ecc)  # f_s = sqrt(e)sin(omega) = e.sin(omega)/sqrt(e)
    f_c = 0.#ecosw/usqrt(ecc)  # f_c = sqrt(e)cos(omega) = e.cos(omega)/sqrt(e)

# Let's take data from first transit only
"""
mask1 = tim > T_0-(W*P/2)
mask2 = tim < T_0+(W*P/2)
mask3 = tim > T_ec - (W*P/2)
mask4 = tim < T_ec + (W*P/2)
mask5 = tim > T_1-(W*P/2)
mask6 = tim < T_1+(W*P/2)
"""
mask = np.zeros(len(tim), dtype=bool)
for i in range(len(tim)):
    if (tim[i] > T_0-(W*P/2)) and (tim[i] < T_0+(W*P/2)):
        mask[i] = True
    elif (tim[i]>T_ec-(W*P/2)) and (tim[i]<T_ec+(W*P/2)):
        mask[i] = True
    elif (tim[i]>T_1-(W*P/2)) and (tim[i]<T_1+(W*P/2)):
        mask[i] = True

print(mask)

tim, fl, fle = dd.mask_data(mask)
# Plotting the data
plt.figure(figsize=(16,9))
plt.errorbar(tim, fl, yerr=fle, fmt='.', c='orangered')
plt.axvline(T_ec, c='k', lw=3)
plt.axvline(x=T_0, c='k', lw=3)
plt.axvline(x=T_1, c='k', lw=3)
plt.xlabel('Time (BJD)')
plt.ylabel('Normalised Flux')
plt.title('Phase curve for WASP-189b (Aperture: DEFAULT)')
plt.grid()[False False False ... False False False]

Masked 816 points
tim, fl, fle = dd.flatten(T_0, P*W)
# Plotting the data
plt.figure(figsize=(16,9))
plt.errorbar(tim, fl, yerr=fle, fmt='.', c='orangered')
plt.axvline(T_ec, c='k', lw=3)
plt.axvline(x=T_0, c='k', lw=3)
plt.axvline(x=T_1, c='k', lw=3)
plt.xlabel('Time (BJD)')
plt.ylabel('Normalised Flux')
plt.title('Phase curve for HD106315b (Aperture: DEFAULT)')
plt.grid()_____no_output_____

## Detrending (or, decorrelating) the dataset

To perform this operation of detrending we may want to look at diagnostic report for this observations. The diagnostic report consist of various plots of flux as a function of several instrumental properties like roll angle of the spacecraft, centroid position etc. This would enable us to see if there is any trend going on with instruments so that we can take care of it. So, first let's see the diagnostic report..._____no_output_____

dd.diagnostic_plot()_____no_output_____dd.should_I_decorr()Decorrelate in dfdt dfdx dfdy dfdsinphi dfdcosphi dfdbg dfdcontam dfdsmear d2fdt2 d2fdx2 d2fdy2 dfdsin2phi dfdcos2phi using decorr, lmfit_transt, or lmfit_eclipse functions.
dd.decorr(dfdx=True, dfdy=True, dfdt=True, dfdbg=True, dfdsinphi=True, dfdcosphi=True)Fit Report
[[Model]]
    Model(factor)
[[Fit Statistics]]
    # fitting method   = leastsq
    # function evals   = 17
    # data points      = 3786
    # variables        = 7
    chi-square         = 4.3226e-05
    reduced chi-square = 1.1439e-08
    Akaike info crit   = -69224.8388
    Bayesian info crit = -69181.1653
[[Variables]]
    c:           1.00004167 +/- 2.6486e-06 (0.00%) (init = 1)
    dfdt:       -7.9223e-07 +/- 2.0359e-06 (256.98%) (init = 0)
    d2fdt2:      0 (fixed)
    dfdbg:       2.5143e-05 +/- 6.7652e-05 (269.07%) (init = 0)
    dfdcontam:   0 (fixed)
    dfdsmear:    0 (fixed)
    ramp:        0 (fixed)
    dfdx:       -5.9666e-06 +/- 5.6118e-06 (94.05%) (init = 0)
    dfdy:       -2.4573e-05 +/- 5.2427e-06 (21.34%) (init = 0)
    d2fdxdy:     0 (fixed)
    d2fdx2:      0 (fixed)
    d2fdy2:      0 (fixed)
    dfdcosphi:  -1.7471e-05 +/- 2.1772e-06 (12.46%) (init = 0)
    dfdsinphi:   1.0039e-04 +/- 4.4811e-06 (4.46%) (init = 0)
    dfdcos2phi:  0 (fixed)
    dfdsin2phi:  0 (fixed)
    dfdcos3phi:  0 (fixed)
    dfdsin3phi:  0 (fixed)
[[Correlations]] (unreported correlations are < 0.100)
    C(c, dfdsinphi)         =  0.724
    C(c, dfdbg)             = -0.349
    C(dfdbg, dfdsinphi)     = -0.297
    C(dfdcosphi, dfdsinphi) =  0.173
    C(c, dfdcosphi)         =  0.167
    C(dfdbg, dfdcosphi)     = -0.139

Compare the lightcurve RMS before and after decorrelation
RMS before = 116.7 ppm
RMS after = 106.9 ppm

# Modelling the phasecurve_____no_output_____

# Defining a chi square minimization function
def _chisq_prior(pars, *args):
    r = (fl - Model1.eval(pars, t = tim) - 1)/fle
    for p in pars:
        u = pars[p].user_data
        if isinstance(u, UFloat):
            r = np.append(r, (u.n - pars[p].value)/u.s)
    return r_____no_output_____# Creating a model
Model1 = ReflectionModel()

# Adding parameters to the model
pars = Model1.make_params()
pars.add("P", value = 2.7240338, vary=False)
pars.add("T_0", value = T_0, vary=False)
pars.add("A_g", value = 0.1, min=0., max=1., vary = True)
pars.add("r_p", value = 1/4.70267804, vary = False)
pars.add("f_c", value = 0., vary = False)
pars.add("f_s", value = 0., vary = False)
pars.add("sini", value = 0.99568523, vary=False)_____no_output_____result = minimize(_chisq_prior, pars, nan_policy = "propagate", args = (Model1, tim, fl, fle))
flux_model = Model1.eval(result.params, t = tim)
residuals = result.residual

figure, ax = plt.subplots(2, 1, sharex = True, figsize=(16,9))
ax[0].scatter(tim, fl, marker='.')
ax[0].plot(tim, flux_model + 1, "r")
ax[1].scatter(tim, residuals, marker='.')
ax[0].set_ylabel("Flux")
ax[1].set_ylabel("Residuals")
ax[1].set_xlabel("Time")
ax[1].grid()
ax[0].grid()_____no_output_____result.params_____no_output_____


# Notebook from pseudoPixels/bokehBot
Path: examples/notebooks/barPlots.ipynb

### 1.0 Import bokehBorn_____no_output_____

import iSeaborn as isn

from bokeh.io import output_notebook
from bokeh.plotting import show, figure_____no_output_____output_notebook()_____no_output_____

### 2.0 Load Dataset_____no_output_____

tips = isn.load_dataset("tips")
tips.head()_____no_output_____

### 3.0 bokeBorn Bar Plot Examples_____no_output_____#### Draw Vertical Bar
Draw a set of vertical bar plots grouped by a categorical variable:_____no_output_____

fig = isn.barplot(x="day", y="total_bill", data=tips)
show(fig)_____no_output_____

### Choose from Interactive Tools
Choose the required interactive tools for visualization by passing value to `tools` options. Default tools are `"pan,box_select,wheel_zoom,box_zoom,reset,save"` . Example:_____no_output_____

fig = isn.barplot(x="day", y="total_bill", data=tips, tools="pan, save")
show(fig)_____no_output_____

#### Draw Horizontal Bar
Draw a set of horizontal bars automatically with change of axis_____no_output_____

fig = isn.barplot(x="tip", y="day", data=tips)
show(fig)_____no_output_____

#### Control Orders
Control bar order by passing an explicit order:_____no_output_____

fig = isn.barplot(x="time", y="tip", data=tips,order=["Lunch", "Dinner"])
show(fig)_____no_output_____

#### Set Desired Estimator
For example, use median as the estimate of central tendency:_____no_output_____

from numpy import median
fig = isn.barplot(x="day", y="tip", data=tips, estimator=median)
show(fig)_____no_output_____

#### Choose Palettes
Use a different color palette for the bars:_____no_output_____

fig = isn.barplot(x= "day", y="total_bill", data=tips, palette="Blues_d")
show(fig)_____no_output_____

#### Set Specific Color
Plot all bars in a single color:_____no_output_____

fig = isn.barplot(x= "day", y="total_bill", data=tips, color="salmon")
show(fig)_____no_output_____

#### Use Hue for Visualization
Draw a set of vertical bars with nested grouping by a two variables:

__Note__: Click the legend text to view only a selected category._____no_output_____

fig = isn.barplot(x="day", y="total_bill", hue="sex", data=tips)
show(fig)_____no_output_____fig = isn.barplot(x="day", y="total_bill", hue="smoker", data=tips)
show(fig)_____no_output_____fig = isn.barplot(x="tip", y="day", data=tips, hue="sex")
show(fig)_____no_output_____

#### Change Plot Properties
Change different plot properties such as plot_width:_____no_output_____

fig = isn.barplot(x="day", y="total_bill", data=tips, plot_width=800)
show(fig)_____no_output_____fig = isn.barplot(x="day", y="total_bill", data=tips, plot_width=600, plot_height=200, plot_title="Awesome Plot Title")
show(fig)_____no_output_____

#### And More
Change the other aesthetics of the plot as key word arguments as available in `bokeh.plotting.figure.vbar`, such as changing `alpha` of the plot. List of all the aesthetics properties :: https://docs.bokeh.org/en/latest/docs/reference/plotting.html#bokeh.plotting.figure.Figure.vbar_____no_output_____

fig = isn.barplot(x="day", y="total_bill", data=tips, alpha=0.3)
show(fig)_____no_output_____fig = isn.barplot(x="smoker", y="total_bill", hue="sex", data=tips)
show(fig)_____no_output_____


# Notebook from risdenk/hello-ltr
Path: notebooks/solr/tmdb/ai-powered-search-ch-10.ipynb

To get to know Solr _____no_output_____# Solr Client_____no_output_____

from ltr.client import SolrClient
client = SolrClient()_____no_output_____

# Download & Build Index (run once)

If you don't already have the downloaded dependencies; if you don't have TheMovieDB data indexed run this_____no_output_____

from ltr import download

tmdb_corpus='http://es-learn-to-rank.labs.o19s.com/tmdb_ai_pow_search.json'
judgments='http://es-learn-to-rank.labs.o19s.com/title_judgments_binary.txt'
download([tmdb_corpus, judgments]);_____no_output_____from ltr.index import rebuild
from ltr.helpers.movies import indexable_movies
movies=indexable_movies(movies='data/tmdb_ai_pow_search.json')
rebuild(client, index='tmdb', doc_src=movies)_____no_output_____

## Features for movie titles

We'll be searching movie titles (think searching for a specific movie on Netflix). And we have a set of judgments around the appropriatte movie to return. IE search for "Star Wars" return good star wars matches, in quality order...

These cover various aspects of the problem (searching title by phrase, title bm25 score, release date, etc). We'll use this to explore and analyze a simple model_____no_output_____

client.reset_ltr(index='tmdb')

ftr_config = [
    #1
    {
      "name" : "title_bm25",
      "store": "title",
      "class" : "org.apache.solr.ltr.feature.SolrFeature",
      "params" : {
        "q" : "title:(${keywords})"
      }
    },
    #2
    {
      "name" : "overview_bm25",
      "store": "title",
      "class" : "org.apache.solr.ltr.feature.SolrFeature",
      "params" : {
        "q" : "overview:(${keywords})"
      }
    },
    {#3
      "name" : "release_year",
      "store": "title",
      "class" : "org.apache.solr.ltr.feature.SolrFeature",
      "params" : {
        "q" : "{!func}def(release_year,2000)"
      }
    }

]

client.create_featureset(index='tmdb', name='title', ftr_config=ftr_config)_____no_output_____

## Training Set Generation

Log out features for each of the above queries out to a training set file_____no_output_____

from ltr.judgments import judgments_open
from ltr.log import FeatureLogger
from itertools import groupby

ftr_logger=FeatureLogger(client, index='tmdb', feature_set='title')
with judgments_open('data/title_judgments_binary.txt') as judgments:
    for qid, query_judgments in groupby(judgments, key=lambda j: j.qid):
        ftr_logger.log_for_qid(qid=qid,
                               judgments=query_judgments,
                               keywords=judgments.keywords(qid))_____no_output_____import numpy as np
from ltr.judgments import judgments_to_nparray

def pairwise_transform(features, predictors):
    """ Informed by
        https://gist.github.com/agramfort/2071994

        """
    GRADE = 0
    QID = 1

    assert features.shape[0] == predictors.shape[0]
    assert predictors.shape[1] == 2
    assert features.shape[1] > 0

    num_samples = features.shape[0]

    transformed_predictors = []
    transformed_features = []

    for i in range(num_samples):
        for j in range(num_samples):
            if (predictors[i][GRADE] != predictors[j][GRADE] and \
                predictors[i][QID] == predictors[j][QID]):

                transformed_predictors.append([predictors[i][GRADE] - predictors[j][GRADE]])
                transformed_features.append(features[i, :] - features[j, :])
    return np.array(transformed_features), np.array(transformed_predictors)

def samples_from_training_data(training_set):
    features, predictors = judgments_to_nparray(training_set)

    # Scale data
    print("Scaling")
    from sklearn.preprocessing import StandardScaler
    scaler = StandardScaler()

    scaler.fit(features)
    features = scaler.transform(features)

    print("Pairwise Transform")
    features, predictors = pairwise_transform(features, predictors)
    return features, predictors.ravel(), scaler

features, predictors, scaler = samples_from_training_data(ftr_logger.logged)
features_____no_output_____from sklearn import svm, linear_model
model = svm.LinearSVC(max_iter=1000, verbose=1)
model.fit(features, predictors)_____no_output_____model.coef______no_output_____linear_model = {
  "store": "title",
  "class": "org.apache.solr.ltr.model.LinearModel",
  "name": "movie_titles",
  "features": [
  ],
  "params": {
      "weights": {
      }
  }
}

import math
ftr_model = {}
ftr_names = [ftr['name'] for ftr in ftr_config]
for idx, ftr_name in enumerate(ftr_names):
    config = {
        "name": ftr_name,
        "norm": {
            "class": "org.apache.solr.ltr.norm.StandardNormalizer",
            "params": {
                "avg": str(scaler.mean_[idx]),
                "std": str(math.sqrt(scaler.var_[idx]))
            }
        }
    }
    linear_model['features'].append(config)
    linear_model['params']['weights'][ftr_name] =  model.coef_[0][idx]

linear_model_____no_output_____import json
client.submit_model(featureset='title',
                    index='tmdb',
                    model_name='movie_titles',
                    solr_model=linear_model)_____no_output_____from ltr import search
search(client, keywords='rambo', modelName='movie_titles')_____no_output_____


# Notebook from GTPB/3DAROC21
Path: assets/material/Notebooks/Day1/04-Mapping.ipynb

## Iterative vs fragment-based mapping_____no_output_______running time__: < 15 min

Iterative mapping first proposed by [(Imakaev et al., 2012)](#cite-Imakaev2012a), allows to map usually a high number of reads. However other methodologies, less "brute-force" can be used to take into account the chimeric nature of the Hi-C reads.

A simple alternative is to allow split mapping.

Another way consists in _pre-truncating_ [(Ay and Noble, 2015)](#cite-Ay2015a) reads that contain a ligation site and map only the longest part of the read [(Wingett et al., 2015)](#cite-Wingett2015).

Finally, an intermediate approach, _fragment-based_, consists in mapping full length reads first, and than splitting unmapped reads at the ligation sites [(Serra et al. 2017)](#cite-Serra2017)._____no_output_____![mapping strategies](images/mapping.png)_____no_output_____### Advantages of iterative mapping_____no_output_____- It's the only solution when no restriction enzyme has been used (i.e. micro-C)
- Can be faster when few windows (2 or 3) are used_____no_output_____### Advantages of fragment-based mapping _____no_output_____- Generally faster
- Safer: mapped reads are generally larger than 25-30 nt (the largest window used in iterative mapping). Less reads are mapped, but the difference is usually canceled or reversed when looking for "valid-pairs"._____no_output_____## Mapping_____no_output_____`tadbit map` tool can be used to perform either iterative or fragment-based mapping, or a combination of both.

In the course we will use fragment-based mapping which is generally more efficient for a conventional Hi-C experiment. If we choose to use the iterative mapping strategy the `tadbit map` command will be exactly the same but adding the `--iterative` option: _____no_output_____### Fragment-based mapping_____no_output_____The fragment-based mapping strategy works in 2 steps:
1. The read-ends are mapped entirely, assuming that no ligation occurred in them.
2. For unmapped read-ends, the function searches for a ligation site (e.g. in the case of MboI this would correspond to `GATCGATC` and in the case of HindIII to `AAGCTAGCTT`). The read-end is split accordingly replacing the ligation site by two RE sites:

```
                        read-end-part-one---AAGCTAGCTT----read-end-part-two
     will be split in:
                        read-end-part-one---AAGCTT
     and:
                                                AAGCTT----read-end-part-two
```

_Note: __if no ligation site is found__, step two will be repeated using digested RE site as split point (`AAGCT` in the case of HindIII). This is done in order to be protected against sequencing errors. When this path is followed the digested RE site is removed, but not replaced. __If digested RE sites are not found either, the read will be classified as unmapped__._

_Note: __both mapping strategies can be combined__, for example defining the windows as previously (iterative mapping), but also give a RE name_ `r_enz=MboI` _and setting_ `frag_map=True` _like this if a read has not been mapped in any window, TADbit will also try to apply the fragment-based strategy.______no_output_____

%%bash

tadbit map -w ../results/PSC_rep1 \
    --fastq ../FASTQs/PSC/PSC_HiC_rep1_subset_chr3_1.fastq.gz \
    --read 1 \
    --index ../refGenome/mm39_chr3.gem \
    --renz MboI \
    --cpus 12Writing log to ../results/PSC_rep1/process.log
Preparing FASTQ file
  - conversion to MAP format
Mapping reads in window 1-end_e06c0841c2...
TO GEM 3 /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_c1j0cpu2
/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper -I /home/3DAROC21/3DAROC21/refGenome/mm39_chr3.gem -t 6 -F SAM -o /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_c1j0cpu2_full_1-end_e06c0841c2.map -i /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_c1j0cpu2
Parsing result...
   x removing GEM input /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_c1j0cpu2
   x removing map /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_c1j0cpu2_full_1-end_e06c0841c2.map
  - splitting into restriction enzyme (RE) fragments using ligation sites
  - ligation sites are replaced by RE sites to match the reference genome
    * enzymes: MboI & MboI, ligation site: GATCGATC, RE site: GATC & GATC
Preparing MAP file
   x removing pre-GEM input /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_c1j0cpu2_filt_1-end_e06c0841c2.map
Mapping fragments of remaining reads...
TO GEM 3 /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_hgvvsuag
/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper -I /home/3DAROC21/3DAROC21/refGenome/mm39_chr3.gem -t 6 -F SAM -o /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_hgvvsuag_frag_1-end_e06c0841c2.map -i /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_hgvvsuag
Parsing result...
   x removing GEM input /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_hgvvsuag
   x removing failed to map /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_c1j0cpu2_fail_e06c0841c2.map
   x removing tmp mapped /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r1_e06c0841c2/PSC_HiC_rep1_subset_chr3_1_hgvvsuag_frag_1-end_e06c0841c2.map
,---------------.
| MAPPED_INPUTs |
,----.--------.------------.------.------.------.--------.---------------.-------------------.----------.-----------------.---------.
| Id | PATHid |    Entries | Trim | Frag | Read | Enzyme | Dangling_Ends |    Ligation_Sites | WRKDIRid | MAPPED_OUTPUTid | INDEXid |
|----+--------+------------+------+------+------+--------+---------------+-------------------+----------+-----------------+---------|
|  1 |      2 | 10,410,291 | None | full |    1 |   MboI |   MboI:1.340% | MboI-MboI:35.204% |        1 |               6 |       3 |
|  2 |      2 |  1,078,595 | None | frag |    1 |   MboI |   MboI:1.340% | MboI-MboI:35.204% |        1 |               7 |       3 |
'----^--------^------------^------^------^------^--------^---------------^-------------------^----------^-----------------^---------'
,-------.
| PATHs |
,----.-------.-------------------------------------------------------------------.--------------.
| Id | JOBid |                                                              Path |         Type |
|----+-------+-------------------------------------------------------------------+--------------|
|  1 |     1 |                          /home/3DAROC21/3DAROC21/results/PSC_rep1 |      WORKDIR |
|  2 |     1 |              ../../FASTQs/PSC/PSC_HiC_rep1_subset_chr3_1.fastq.gz | MAPPED_FASTQ |
|  3 |     1 |                                     ../../refGenome/mm39_chr3.gem |        INDEX |
|  4 |     1 |           PSC_HiC_rep1_subset_chr3_1.fastq.gz_MboI_6389769ac5.png |       FIGURE |
|  5 |     2 |           PSC_HiC_rep1_subset_chr3_1.fastq.gz_MboI_e06c0841c2.png |       FIGURE |
|  6 |     2 | 01_mapped_r1/PSC_HiC_rep1_subset_chr3_1_full_1-end_e06c0841c2.map |      SAM/MAP |
|  7 |     2 | 01_mapped_r1/PSC_HiC_rep1_subset_chr3_1_frag_1-end_e06c0841c2.map |      SAM/MAP |
'----^-------^-------------------------------------------------------------------^--------------'
,------.
| JOBs |
,----.----------------------------------------------------------------------------------------------------------------------------------------------------------------------------.---------------------.---------------------.------.----------------.
| Id |                                                                                Parameters |         Launch_time |         Finish_time | Type | Parameters_md5 |
|----+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+---------------------+------+----------------|
|  1 | skip_mapping:1 read:1 chr_name:[] noX:0 fast_fragment:0 cpus:6 mapper:gem mapper_binary:/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper mapper_param:{} gem_version:3 | 04/10/2021 14:01:11 | 04/10/2021 14:01:15 |  Map |     6389769ac5 |
|  2 | skip_mapping:0 read:1 chr_name:[] noX:0 fast_fragment:0 cpus:6 mapper:gem mapper_binary:/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper mapper_param:{} gem_version:3 | 04/10/2021 14:03:07 | 04/10/2021 14:07:50 |  Map |     e06c0841c2 |
'----^----------------------------------------------------------------------------------------------------------------------------------------------------------------------------^---------------------^---------------------^------^----------------'
,---------------.
| MAPPED_INPUTs |
,----.--------.------------.------.------.------.--------.---------------.-------------------.----------.-----------------.---------.
| Id | PATHid |    Entries | Trim | Frag | Read | Enzyme | Dangling_Ends |    Ligation_Sites | WRKDIRid | MAPPED_OUTPUTid | INDEXid |
|----+--------+------------+------+------+------+--------+---------------+-------------------+----------+-----------------+---------|
|  1 |      2 | 10,410,291 | None | full |    1 |   MboI |   MboI:1.340% | MboI-MboI:35.204% |        1 |               6 |       3 |
|  2 |      2 |  1,078,595 | None | frag |    1 |   MboI |   MboI:1.340% | MboI-MboI:35.204% |        1 |               7 |       3 |
'----^--------^------------^------^------^------^--------^---------------^-------------------^----------^-----------------^---------'
,-------.
| PATHs |
,----.-------.-------------------------------------------------------------------.--------------.
| Id | JOBid |                                                              Path |         Type |
|----+-------+-------------------------------------------------------------------+--------------|
|  1 |     1 |                          /home/3DAROC21/3DAROC21/results/PSC_rep1 |      WORKDIR |
|  2 |     1 |              ../../FASTQs/PSC/PSC_HiC_rep1_subset_chr3_1.fastq.gz | MAPPED_FASTQ |
|  3 |     1 |                                     ../../refGenome/mm39_chr3.gem |        INDEX |
|  4 |     1 |           PSC_HiC_rep1_subset_chr3_1.fastq.gz_MboI_6389769ac5.png |       FIGURE |
|  5 |     2 |           PSC_HiC_rep1_subset_chr3_1.fastq.gz_MboI_e06c0841c2.png |       FIGURE |
|  6 |     2 | 01_mapped_r1/PSC_HiC_rep1_subset_chr3_1_full_1-end_e06c0841c2.map |      SAM/MAP |
|  7 |     2 | 01_mapped_r1/PSC_HiC_rep1_subset_chr3_1_frag_1-end_e06c0841c2.map |      SAM/MAP |
'----^-------^-------------------------------------------------------------------^--------------'
,------.
| JOBs |
,----.----------------------------------------------------------------------------------------------------------------------------------------------------------------------------.---------------------.---------------------.------.----------------.
| Id |                                                                                Parameters |         Launch_time |         Finish_time | Type | Parameters_md5 |
|----+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+---------------------+------+----------------|
|  1 | skip_mapping:1 read:1 chr_name:[] noX:0 fast_fragment:0 cpus:6 mapper:gem mapper_binary:/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper mapper_param:{} gem_version:3 | 04/10/2021 14:01:11 | 04/10/2021 14:01:15 |  Map |     6389769ac5 |
|  2 | skip_mapping:0 read:1 chr_name:[] noX:0 fast_fragment:0 cpus:6 mapper:gem mapper_binary:/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper mapper_param:{} gem_version:3 | 04/10/2021 14:03:07 | 04/10/2021 14:07:50 |  Map |     e06c0841c2 |
'----^----------------------------------------------------------------------------------------------------------------------------------------------------------------------------^---------------------^---------------------^------^----------------'

And for the other read-end we should change the `--fastq` file and the `--read` parameter to `2`_____no_output_____

%%bash

tadbit map -w ../results/PSC_rep1 \
    --fastq ../FASTQs/PSC/PSC_HiC_rep1_subset_chr3_2.fastq.gz \
    --read 2 \
    --index ../refGenome/mm39_chr3.gem \
    --renz MboI \
    --cpus 12Writing log to ../results/PSC_rep1/process.log
Preparing FASTQ file
  - conversion to MAP format
Mapping reads in window 1-end_dbbd3e8e31...
TO GEM 3 /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_2sbavg3r
/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper -I /home/3DAROC21/3DAROC21/refGenome/mm39_chr3.gem -t 6 -F SAM -o /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_2sbavg3r_full_1-end_dbbd3e8e31.map -i /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_2sbavg3r
Parsing result...
   x removing GEM input /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_2sbavg3r
   x removing map /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_2sbavg3r_full_1-end_dbbd3e8e31.map
  - splitting into restriction enzyme (RE) fragments using ligation sites
  - ligation sites are replaced by RE sites to match the reference genome
    * enzymes: MboI & MboI, ligation site: GATCGATC, RE site: GATC & GATC
Preparing MAP file
   x removing pre-GEM input /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_2sbavg3r_filt_1-end_dbbd3e8e31.map
Mapping fragments of remaining reads...
TO GEM 3 /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_tcn500zr
/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper -I /home/3DAROC21/3DAROC21/refGenome/mm39_chr3.gem -t 6 -F SAM -o /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_tcn500zr_frag_1-end_dbbd3e8e31.map -i /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_tcn500zr
Parsing result...
   x removing GEM input /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_tcn500zr
   x removing failed to map /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_2sbavg3r_fail_dbbd3e8e31.map
   x removing tmp mapped /home/3DAROC21/3DAROC21/results/PSC_rep1_tmp_r2_dbbd3e8e31/PSC_HiC_rep1_subset_chr3_2_tcn500zr_frag_1-end_dbbd3e8e31.map
,---------------.
| MAPPED_INPUTs |
,----.--------.------------.------.------.------.--------.---------------.-------------------.----------.-----------------.---------.
| Id | PATHid |    Entries | Trim | Frag | Read | Enzyme | Dangling_Ends |    Ligation_Sites | WRKDIRid | MAPPED_OUTPUTid | INDEXid |
|----+--------+------------+------+------+------+--------+---------------+-------------------+----------+-----------------+---------|
|  1 |      2 | 10,410,291 | None | full |    1 |   MboI |   MboI:1.340% | MboI-MboI:35.204% |        1 |               6 |       3 |
|  2 |      2 |  1,078,595 | None | frag |    1 |   MboI |   MboI:1.340% | MboI-MboI:35.204% |        1 |               7 |       3 |
|  3 |      8 | 10,410,291 | None | full |    2 |   MboI |   MboI:1.434% | MboI-MboI:35.818% |        1 |              10 |       3 |
|  4 |      8 |  1,087,028 | None | frag |    2 |   MboI |   MboI:1.434% | MboI-MboI:35.818% |        1 |              11 |       3 |
'----^--------^------------^------^------^------^--------^---------------^-------------------^----------^-----------------^---------'
,-------.
| PATHs |
,----.-------.-------------------------------------------------------------------.--------------.
| Id | JOBid |                                                              Path |         Type |
|----+-------+-------------------------------------------------------------------+--------------|
|  1 |     1 |                          /home/3DAROC21/3DAROC21/results/PSC_rep1 |      WORKDIR |
|  2 |     1 |              ../../FASTQs/PSC/PSC_HiC_rep1_subset_chr3_1.fastq.gz | MAPPED_FASTQ |
|  3 |     1 |                                     ../../refGenome/mm39_chr3.gem |        INDEX |
|  4 |     1 |           PSC_HiC_rep1_subset_chr3_1.fastq.gz_MboI_6389769ac5.png |       FIGURE |
|  5 |     2 |           PSC_HiC_rep1_subset_chr3_1.fastq.gz_MboI_e06c0841c2.png |       FIGURE |
|  6 |     2 | 01_mapped_r1/PSC_HiC_rep1_subset_chr3_1_full_1-end_e06c0841c2.map |      SAM/MAP |
|  7 |     2 | 01_mapped_r1/PSC_HiC_rep1_subset_chr3_1_frag_1-end_e06c0841c2.map |      SAM/MAP |
|  8 |     3 |              ../../FASTQs/PSC/PSC_HiC_rep1_subset_chr3_2.fastq.gz | MAPPED_FASTQ |
|  9 |     3 |           PSC_HiC_rep1_subset_chr3_2.fastq.gz_MboI_dbbd3e8e31.png |       FIGURE |
| 10 |     3 | 01_mapped_r2/PSC_HiC_rep1_subset_chr3_2_full_1-end_dbbd3e8e31.map |      SAM/MAP |
| 11 |     3 | 01_mapped_r2/PSC_HiC_rep1_subset_chr3_2_frag_1-end_dbbd3e8e31.map |      SAM/MAP |
'----^-------^-------------------------------------------------------------------^--------------'
,------.
| JOBs |
,----.----------------------------------------------------------------------------------------------------------------------------------------------------------------------------.---------------------.---------------------.------.----------------.
| Id |                                                                                Parameters |         Launch_time |         Finish_time | Type | Parameters_md5 |
|----+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+---------------------+------+----------------|
|  1 | skip_mapping:1 read:1 chr_name:[] noX:0 fast_fragment:0 cpus:6 mapper:gem mapper_binary:/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper mapper_param:{} gem_version:3 | 04/10/2021 14:01:11 | 04/10/2021 14:01:15 |  Map |     6389769ac5 |
|  2 | skip_mapping:0 read:1 chr_name:[] noX:0 fast_fragment:0 cpus:6 mapper:gem mapper_binary:/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper mapper_param:{} gem_version:3 | 04/10/2021 14:03:07 | 04/10/2021 14:07:50 |  Map |     e06c0841c2 |
|  3 | skip_mapping:0 read:2 chr_name:[] noX:0 fast_fragment:0 cpus:6 mapper:gem mapper_binary:/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper mapper_param:{} gem_version:3 | 04/10/2021 14:07:52 | 04/10/2021 14:12:15 |  Map |     dbbd3e8e31 |
'----^----------------------------------------------------------------------------------------------------------------------------------------------------------------------------^---------------------^---------------------^------^----------------'
,---------------.
| MAPPED_INPUTs |
,----.--------.------------.------.------.------.--------.---------------.-------------------.----------.-----------------.---------.
| Id | PATHid |    Entries | Trim | Frag | Read | Enzyme | Dangling_Ends |    Ligation_Sites | WRKDIRid | MAPPED_OUTPUTid | INDEXid |
|----+--------+------------+------+------+------+--------+---------------+-------------------+----------+-----------------+---------|
|  1 |      2 | 10,410,291 | None | full |    1 |   MboI |   MboI:1.340% | MboI-MboI:35.204% |        1 |               6 |       3 |
|  2 |      2 |  1,078,595 | None | frag |    1 |   MboI |   MboI:1.340% | MboI-MboI:35.204% |        1 |               7 |       3 |
|  3 |      8 | 10,410,291 | None | full |    2 |   MboI |   MboI:1.434% | MboI-MboI:35.818% |        1 |              10 |       3 |
|  4 |      8 |  1,087,028 | None | frag |    2 |   MboI |   MboI:1.434% | MboI-MboI:35.818% |        1 |              11 |       3 |
'----^--------^------------^------^------^------^--------^---------------^-------------------^----------^-----------------^---------'
,-------.
| PATHs |
,----.-------.-------------------------------------------------------------------.--------------.
| Id | JOBid |                                                              Path |         Type |
|----+-------+-------------------------------------------------------------------+--------------|
|  1 |     1 |                          /home/3DAROC21/3DAROC21/results/PSC_rep1 |      WORKDIR |
|  2 |     1 |              ../../FASTQs/PSC/PSC_HiC_rep1_subset_chr3_1.fastq.gz | MAPPED_FASTQ |
|  3 |     1 |                                     ../../refGenome/mm39_chr3.gem |        INDEX |
|  4 |     1 |           PSC_HiC_rep1_subset_chr3_1.fastq.gz_MboI_6389769ac5.png |       FIGURE |
|  5 |     2 |           PSC_HiC_rep1_subset_chr3_1.fastq.gz_MboI_e06c0841c2.png |       FIGURE |
|  6 |     2 | 01_mapped_r1/PSC_HiC_rep1_subset_chr3_1_full_1-end_e06c0841c2.map |      SAM/MAP |
|  7 |     2 | 01_mapped_r1/PSC_HiC_rep1_subset_chr3_1_frag_1-end_e06c0841c2.map |      SAM/MAP |
|  8 |     3 |              ../../FASTQs/PSC/PSC_HiC_rep1_subset_chr3_2.fastq.gz | MAPPED_FASTQ |
|  9 |     3 |           PSC_HiC_rep1_subset_chr3_2.fastq.gz_MboI_dbbd3e8e31.png |       FIGURE |
| 10 |     3 | 01_mapped_r2/PSC_HiC_rep1_subset_chr3_2_full_1-end_dbbd3e8e31.map |      SAM/MAP |
| 11 |     3 | 01_mapped_r2/PSC_HiC_rep1_subset_chr3_2_frag_1-end_dbbd3e8e31.map |      SAM/MAP |
'----^-------^-------------------------------------------------------------------^--------------'
,------.
| JOBs |
,----.----------------------------------------------------------------------------------------------------------------------------------------------------------------------------.---------------------.---------------------.------.----------------.
| Id |                                                                                Parameters |         Launch_time |         Finish_time | Type | Parameters_md5 |
|----+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+---------------------+------+----------------|
|  1 | skip_mapping:1 read:1 chr_name:[] noX:0 fast_fragment:0 cpus:6 mapper:gem mapper_binary:/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper mapper_param:{} gem_version:3 | 04/10/2021 14:01:11 | 04/10/2021 14:01:15 |  Map |     6389769ac5 |
|  2 | skip_mapping:0 read:1 chr_name:[] noX:0 fast_fragment:0 cpus:6 mapper:gem mapper_binary:/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper mapper_param:{} gem_version:3 | 04/10/2021 14:03:07 | 04/10/2021 14:07:50 |  Map |     e06c0841c2 |
|  3 | skip_mapping:0 read:2 chr_name:[] noX:0 fast_fragment:0 cpus:6 mapper:gem mapper_binary:/home/3DAROC21/miniconda2/envs/tadbit/bin/gem-mapper mapper_param:{} gem_version:3 | 04/10/2021 14:07:52 | 04/10/2021 14:12:15 |  Map |     dbbd3e8e31 |
'----^----------------------------------------------------------------------------------------------------------------------------------------------------------------------------^---------------------^---------------------^------^----------------'

## Map files logs and format_____no_output_____TADbit tools uses a SQLite database file where all the commands executed by the user as well as all relevant statistics of the analysis are stored. An output of the database is shown at the end of each command.

In this case we can see useful statistics of the mapping step, for example how many reads are mapped at full length, after splitting and the percentages of dangling-ends and ligation sites:_____no_output_____

%%bash

tadbit describe -w ../results/PSC_rep1 -t 4,---------------.
| MAPPED_INPUTs |
,----.--------.------------.------.------.------.--------.---------------.-------------------.----------.-----------------.---------.
| Id | PATHid |    Entries | Trim | Frag | Read | Enzyme | Dangling_Ends |    Ligation_Sites | WRKDIRid | MAPPED_OUTPUTid | INDEXid |
|----+--------+------------+------+------+------+--------+---------------+-------------------+----------+-----------------+---------|
|  1 |      2 | 10,410,291 | None | full |    1 |   MboI |   MboI:1.340% | MboI-MboI:35.204% |        1 |               6 |       3 |
|  2 |      2 |  1,078,595 | None | frag |    1 |   MboI |   MboI:1.340% | MboI-MboI:35.204% |        1 |               7 |       3 |
|  3 |      8 | 10,410,291 | None | full |    2 |   MboI |   MboI:1.434% | MboI-MboI:35.818% |        1 |              10 |       3 |
|  4 |      8 |  1,087,028 | None | frag |    2 |   MboI |   MboI:1.434% | MboI-MboI:35.818% |        1 |              11 |       3 |
'----^--------^------------^------^------^------^--------^---------------^-------------------^----------^-----------------^---------'

We can list the generated map files in the working directory in the subfolder `01_mapped_r1` and `01_mapped_r2` and inspect one of the files to see its structure:_____no_output_____

%%bash

ls ../results/PSC_rep1/01_mapped_r1/
ls ../results/PSC_rep1/01_mapped_r2/PSC_HiC_rep1_subset_chr3_1_frag_1-end_e06c0841c2.map
PSC_HiC_rep1_subset_chr3_1_full_1-end_e06c0841c2.map
PSC_HiC_rep1_subset_chr3_2_frag_1-end_dbbd3e8e31.map
PSC_HiC_rep1_subset_chr3_2_full_1-end_dbbd3e8e31.map
%%bash

head ../results/PSC_rep1/01_mapped_r1/PSC_HiC_rep1_subset_chr3_1_frag_1-end_e06c0841c2.map -n 2SRR5344969.sra.180~2~	GATCATATCTACAGTCATTTTTTAAAATA	HHHHGGGGGGGGGGGGGGGGGGGGCCCCC	1	chr3:+:5690818:29
SRR5344969.sra.578	GATCCACAAAGACAAATTTGGTATGT	HHHHGGGGGGGGGGGGGGGGGCCCCC	1	chr3:-:29696661:26

A map file is a tsv file storing information of a sequence mapped to a reference genome. It is composed by the following columns:

1. Sequence identifier of the read in the FASTQ.
2. The sequence.
3. The base call quality scores, Phred +33 encoded using ASCII characters.
4. Reserved for mapper information.
5. The information of the region where the read was mapped as chromosome:strand:position:length
_____no_output_____### Questions

- True or False? In case it is false try to correct the	sentence:
 1. The iterative mapping works only if a restriction enzyme has been used in the experiment.
 2. Iterative and fragment-based mapping are incompatible between each other.
 3. If a read has been mapped, the .map files contains all the information in a .fastq file for this read.
 4. If a sequence is not mapped, it doesn't appear in a .map file.
- Would you be confortable to explain to your collegues the iterative mapping strategy? What about the fragment-based mapping?
- Would you be confortable to explain the difference between a read and restriction-enzyme fragment? Are they the same thing?_____no_output_____### References

[^ref1](#ref-1) Imakaev, Maxim V and Fudenberg, Geoffrey and McCord, Rachel Patton and Naumova, Natalia and Goloborodko, Anton and Lajoie, Bryan R and Dekker, Job and Mirny, Leonid A. 2012. _Iterative correction of Hi-C data reveals hallmarks of chromosome organization._. [URL](http://www.ncbi.nlm.nih.gov/pubmed/22941365)

[^ref2](#ref-2) Ay, Ferhat and Noble, William Stafford. 2015. _Analysis methods for studying the 3D architecture of the genome_. [URL](http://genomebiology.com/2015/16/1/183)

[^ref3](#ref-3) Wingett, Steven and Ewels, Philip and Furlan-Magaril, Mayra and Nagano, Takashi and Schoenfelder, Stefan and Fraser, Peter and Andrews, Simon. 2015. _HiCUP: pipeline for mapping and processing Hi-C data._. [URL](http://f1000research.com/articles/4-1310/v1)

[^ref4](#ref-4) François Serra, Davide Baù, Mike Goodstadt, David Castillo,  Guillaume Filion and Marc A. Marti-Renom. 2017. _Automatic analysis and 3D-modelling of Hi-C data using TADbit reveals structural features of the fly chromatin colors_. [URL](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005665)

[^ref5](#ref-5) Marco-Sola, Santiago and Sammeth, Michael and Guigó, Roderic and Ribeca, Paolo. 2012. _The GEM mapper: fast, accurate and versatile alignment by filtration_.

_____no_output_____


# Notebook from osterEWU/mathemathemical_computing_with_Python
Path: computing-primer/jupyter/markdown.ipynb

# Markdown

[Markdown](https://en.wikipedia.org/wiki/Markdown) is a simple text-to-HTML markup language written in plain text. Jupyter notebook recognizes markdown and renders markdown code as HTML. In this section, we present the basic features of markdown.

![Markdown in Jupyter notebook](../img/jupyter/markdown.gif)

See [Markdown](https://daringfireball.net/projects/markdown/) (by John Gruber) and [GitHub Markdown Help](https://help.github.com/articles/basic-writing-and-formatting-syntax/) for more information._____no_output_____## Text

| Output | Syntax |
| :---: | :---: |
| *emphasis* | `*emphasis*` |
| **strong** | `**strong**` |
| `code` | ``  `code` `` |_____no_output_____## Headings

| Output | Syntax |
| :---: | :---: |
|
Heading 1
 | `# Heading 1` |
|
Heading 2
 | `## Heading 2` |
|
Heading 3
 | `### Heading 3` |
|
Heading 4
 | `#### Heading 4` |
|
Heading 5
 | `##### Heading 5` |
|
Heading 6
 | `###### Heading 6` |_____no_output_____## Lists

Create an ordered list using numbers:

```plaintext
1. Number theory
2. Algebra
3. Partial differential equations
4. Probability
```

1. Number theory
2. Algebra
3. Partial differential equations
4. Probability

Create an unordered list using an asterisk * for each item:

```plaintext
* Number theory
* Algebra
* Partial differential equations
* Probability
```

* Number theory
* Algebra
* Partial differential equations
* Probability

Use indentation to create nested lists:

```plaintext
1. Mathematics
  * Calculus
  * Linear Algebra
  * Probability
2. Physics
  * Classical Mechanics
  * Relativity
  * Thermodynamics
3. Biology
  * Diffusion and Osmosis
  * Homeostasis
  * Immunology
```

1. Mathematics
    * Calculus
    * Linear Algebra
    * Probability
2. Physics
    * Classical Mechanics
    * Relativity
    * Thermodynamics
3. Biology
    * Diffusion and Osmosis
    * Homeostasis
    * Immunology_____no_output_____## Links

Create a link with the syntax `[description](url)`. For example:

```plaintext
[UBC Math](http://www.math.ubc.ca)
```

creates the link [UBC Math](http://www.math.ubc.ca)._____no_output_____## Images

Include an image using the syntax `![description](url)`. For example:

```plaintext
![Jupyter logo](http://jupyter.org/assets/nav_logo.svg)
```

displays the image

![Jupyter logo](http://jupyter.org/assets/nav_logo.svg)_____no_output_____## Tables

Create a table by separating entries by pipe characters |:

```plaintext
| Python Operator | Description  |
| :---: | :---: |
| `+` | addition |
| `-` | subtraction |
| `*` | multiplication |
| `/` | division |
| `**` | power |
```

| Python Operator | Description  |
| :---: | :---: |
| `+` | addition |
| `-` | subtraction |
| `*` | multiplication |
| `/` | division |
| `**` | power |

The syntax `:---:` specifies the alignment (centered in this case) of the columns. See more about [GitHub flavoured markdown](https://help.github.com/articles/organizing-information-with-tables/)._____no_output_____## Exercises

1. Create a numbered list of the top 5 websites you visit most often and include a link for each site.
2. Write a short biography of your favourite mathematician, provide a link to their Wikipedia page and include an image (with a link and description of the source).
3. Create a table of all the courses that you have taken in university. Include the columns: course number, course title, year (that you took the class), and instructor name._____no_output_____


# Notebook from ddkom/IBM-Data-Science-Certification
Path: 1.Data-Science-Methodology/c.Understanding-to-Preparation.ipynb

From Understanding to Preparation
_____no_output_____## Introduction

In this lab, we will continue learning about the data science methodology, and focus on the **Data Understanding** and the **Data Preparation** stages._____no_output_____## Table of Contents

1. [Recap](#0)

2. [Data Understanding](#2)

3. [Data Preparation](#4)

_____no_output_____# Recap _____no_output_____In Lab **From Requirements to Collection**, we learned that the data we need to answer the question developed in the business understanding stage, namely *can we automate the process of determining the cuisine of a given recipe?*, is readily available. A researcher named Yong-Yeol Ahn scraped tens of thousands of food recipes (cuisines and ingredients) from three different websites, namely:_____no_output_____

www.allrecipes.com

www.epicurious.com

www.menupan.com_____no_output_____For more information on Yong-Yeol Ahn and his research, you can read his paper on [Flavor Network and the Principles of Food Pairing](http://yongyeol.com/papers/ahn-flavornet-2011.pdf)._____no_output_____We also collected the data and placed it on an IBM server for your convenience.

------------_____no_output_____# Data Understanding _____no_output__________no_output_____ Important note: Please note that you are not expected to know how to program in Python. The following code is meant to illustrate the stages of data understanding and data preparation, so it is totally fine if you do not understand the individual lines of code. We have a full course on programming in Python, Python for Data Science, which is also offered on Coursera. So make sure to complete the Python course if you are interested in learning how to program in Python._____no_output_____### Using this notebook:

To run any of the following cells of code, you can type **Shift + Enter** to excute the code in a cell._____no_output_____Get the version of Python installed._____no_output_____

# check Python version
!python -VPython 3.6.10

Download the library and dependencies that we will need to run this lab._____no_output_____

import pandas as pd # import library to read data into dataframe
pd.set_option('display.max_columns', None)
import numpy as np # import numpy library
import re # import library for regular expression_____no_output_____

Download the data from the IBM server and read it into a *pandas* dataframe._____no_output_____

recipes = pd.read_csv("https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DS0103EN/labs/data/recipes.csv")

print("Data read into dataframe!") # takes about 30 secondsData read into dataframe!

Show the first few rows._____no_output_____

recipes.head()_____no_output_____

Get the dimensions of the dataframe._____no_output_____

recipes.shape_____no_output_____

So our dataset consists of 57,691 recipes. Each row represents a recipe, and for each recipe, the corresponding cuisine is documented as well as whether 384 ingredients exist in the recipe or not, beginning with almond and ending with zucchini._____no_output_____We know that a basic sushi recipe includes the ingredients:
* rice
* soy sauce
* wasabi
* some fish/vegetables_____no_output_____Let's check that these ingredients exist in our dataframe:_____no_output_____

ingredients = list(recipes.columns.values)

print([match.group(0) for ingredient in ingredients for match in [(re.compile(".*(rice).*")).search(ingredient)] if match])
print([match.group(0) for ingredient in ingredients for match in [(re.compile(".*(wasabi).*")).search(ingredient)] if match])
print([match.group(0) for ingredient in ingredients for match in [(re.compile(".*(soy).*")).search(ingredient)] if match])['brown_rice', 'licorice', 'rice']
['wasabi']
['soy_sauce', 'soybean', 'soybean_oil']

Yes, they do!

* rice exists as rice.
* wasabi exists as wasabi.
* soy exists as soy_sauce.

So maybe if a recipe contains all three ingredients: rice, wasabi, and soy_sauce, then we can confidently say that the recipe is a **Japanese** cuisine! Let's keep this in mind!

----------------_____no_output_____# Data Preparation _____no_output__________no_output_____In this section, we will prepare data for the next stage in the data science methodology, which is modeling. This stage involves exploring the data further and making sure that it is in the right format for the machine learning algorithm that we selected in the analytic approach stage, which is decision trees._____no_output_____First, look at the data to see if it needs cleaning._____no_output_____

recipes["country"].value_counts() # frequency table_____no_output_____

By looking at the above table, we can make the following observations:

1. Cuisine column is labeled as Country, which is inaccurate.
2. Cuisine names are not consistent as not all of them start with an uppercase first letter.
3. Some cuisines are duplicated as variation of the country name, such as Vietnam and Vietnamese.
4. Some cuisines have very few recipes._____no_output_____#### Let's fixes these problems._____no_output_____Fix the name of the column showing the cuisine._____no_output_____

column_names = recipes.columns.values
column_names[0] = "cuisine"
recipes.columns = column_names

recipes_____no_output_____

Make all the cuisine names lowercase._____no_output_____

recipes["cuisine"] = recipes["cuisine"].str.lower()_____no_output_____

Make the cuisine names consistent._____no_output_____

recipes.loc[recipes["cuisine"] == "austria", "cuisine"] = "austrian"
recipes.loc[recipes["cuisine"] == "belgium", "cuisine"] = "belgian"
recipes.loc[recipes["cuisine"] == "china", "cuisine"] = "chinese"
recipes.loc[recipes["cuisine"] == "canada", "cuisine"] = "canadian"
recipes.loc[recipes["cuisine"] == "netherlands", "cuisine"] = "dutch"
recipes.loc[recipes["cuisine"] == "france", "cuisine"] = "french"
recipes.loc[recipes["cuisine"] == "germany", "cuisine"] = "german"
recipes.loc[recipes["cuisine"] == "india", "cuisine"] = "indian"
recipes.loc[recipes["cuisine"] == "indonesia", "cuisine"] = "indonesian"
recipes.loc[recipes["cuisine"] == "iran", "cuisine"] = "iranian"
recipes.loc[recipes["cuisine"] == "italy", "cuisine"] = "italian"
recipes.loc[recipes["cuisine"] == "japan", "cuisine"] = "japanese"
recipes.loc[recipes["cuisine"] == "israel", "cuisine"] = "jewish"
recipes.loc[recipes["cuisine"] == "korea", "cuisine"] = "korean"
recipes.loc[recipes["cuisine"] == "lebanon", "cuisine"] = "lebanese"
recipes.loc[recipes["cuisine"] == "malaysia", "cuisine"] = "malaysian"
recipes.loc[recipes["cuisine"] == "mexico", "cuisine"] = "mexican"
recipes.loc[recipes["cuisine"] == "pakistan", "cuisine"] = "pakistani"
recipes.loc[recipes["cuisine"] == "philippines", "cuisine"] = "philippine"
recipes.loc[recipes["cuisine"] == "scandinavia", "cuisine"] = "scandinavian"
recipes.loc[recipes["cuisine"] == "spain", "cuisine"] = "spanish_portuguese"
recipes.loc[recipes["cuisine"] == "portugal", "cuisine"] = "spanish_portuguese"
recipes.loc[recipes["cuisine"] == "switzerland", "cuisine"] = "swiss"
recipes.loc[recipes["cuisine"] == "thailand", "cuisine"] = "thai"
recipes.loc[recipes["cuisine"] == "turkey", "cuisine"] = "turkish"
recipes.loc[recipes["cuisine"] == "vietnam", "cuisine"] = "vietnamese"
recipes.loc[recipes["cuisine"] == "uk-and-ireland", "cuisine"] = "uk-and-irish"
recipes.loc[recipes["cuisine"] == "irish", "cuisine"] = "uk-and-irish"

recipes_____no_output_____

Remove cuisines with < 50 recipes._____no_output_____

# get list of cuisines to keep
recipes_counts = recipes["cuisine"].value_counts()
cuisines_indices = recipes_counts > 50

cuisines_to_keep = list(np.array(recipes_counts.index.values)[np.array(cuisines_indices)])_____no_output_____rows_before = recipes.shape[0] # number of rows of original dataframe
print("Number of rows of original dataframe is {}.".format(rows_before))

recipes = recipes.loc[recipes['cuisine'].isin(cuisines_to_keep)]

rows_after = recipes.shape[0] # number of rows of processed dataframe
print("Number of rows of processed dataframe is {}.".format(rows_after))

print("{} rows removed!".format(rows_before - rows_after))Number of rows of original dataframe is 57691.
Number of rows of processed dataframe is 57403.
288 rows removed!

Convert all Yes's to 1's and the No's to 0's_____no_output_____

recipes = recipes.replace(to_replace="Yes", value=1)
recipes = recipes.replace(to_replace="No", value=0)_____no_output_____

#### Let's analyze the data a little more in order to learn the data better and note any interesting preliminary observations._____no_output_____Run the following cell to get the recipes that contain **rice** *and* **soy** *and* **wasabi** *and* **seaweed**._____no_output_____

recipes.head()_____no_output_____check_recipes = recipes.loc[
    (recipes["rice"] == 1) &
    (recipes["soy_sauce"] == 1) &
    (recipes["wasabi"] == 1) &
    (recipes["seaweed"] == 1)
]

check_recipes_____no_output_____

Based on the results of the above code, can we classify all recipes that contain **rice** *and* **soy** *and* **wasabi** *and* **seaweed** as **Japanese** recipes? Why?_____no_output_____

Let's count the ingredients across all recipes._____no_output_____

# sum each column
ing = recipes.iloc[:, 1:].sum(axis=0)
ing_____no_output_____# define each column as a pandas series
ingredient = pd.Series(ing.index.values, index = np.arange(len(ing)))
count = pd.Series(list(ing), index = np.arange(len(ing)))

# create the dataframe
ing_df = pd.DataFrame(dict(ingredient = ingredient, count = count))
ing_df = ing_df[["ingredient", "count"]]
print(ing_df.to_string())                  ingredient  count
0                     almond   2306
1                   angelica      1
2                      anise    223
3                 anise_seed     87
4                      apple   2422
5               apple_brandy     37
6                    apricot    620
7                   armagnac     11
8                  artemisia     13
9                  artichoke    391
10                 asparagus    460
11                   avocado    660
12                     bacon   2169
13              baked_potato      9
14                      balm      3
15                    banana    989
16                    barley    266
17             bartlett_pear     23
18                     basil   3842
19                       bay   1463
20                      bean   1992
21                     beech      1
22                      beef   4902
23                beef_broth    845
24                beef_liver     10
25                      beer    307
26                      beet    233
27               bell_pepper   5979
28                  bergamot      7
29                     berry    183
30             bitter_orange     85
31                black_bean    495
32             black_currant     11
33    black_mustard_seed_oil     30
34              black_pepper   9828
35           black_raspberry      8
36         black_sesame_seed     26
37                 black_tea     44
38                blackberry    170
39         blackberry_brandy      4
40               blue_cheese    396
41                 blueberry    466
42                  bone_oil     50
43           bourbon_whiskey    156
44                    brandy    395
45                  brassica    114
46                     bread   4571
47                  broccoli    929
48                brown_rice    346
49           brussels_sprout     92
50                 buckwheat     90
51                    butter  20719
52                buttermilk   1634
53                   cabbage   1026
54   cabernet_sauvignon_wine     17
55                     cacao     35
56          camembert_cheese     12
57             cane_molasses   7741
58                   caraway    233
59                  cardamom    352
60                 carnation      3
61                     carob      7
62                    carrot   3689
63                    cashew    208
64                   cassava     19
65                   catfish     71
66               cauliflower    332
67                    caviar     28
68                   cayenne   8254
69                    celery   3625
70                celery_oil   1005
71                    cereal    204
72                 chamomile      3
73            champagne_wine    100
74                   chayote     27
75            cheddar_cheese   3027
76                    cheese   3278
77                    cherry   1082
78             cherry_brandy     32
79                   chervil     52
80                   chicken   5436
81             chicken_broth   3604
82             chicken_liver     52
83                  chickpea    403
84                   chicory    156
85           chinese_cabbage    166
86                     chive   1333
87                     cider   1132
88                  cilantro   2474
89                  cinnamon   5594
90                    citrus    167
91               citrus_peel      4
92                      clam    476
93                     clove     10
94                     cocoa   4799
95                   coconut   1803
96               coconut_oil     17
97                       cod    180
98                    coffee    719
99                    cognac     67
100            concord_grape     12
101                condiment      9
102                coriander   1648
103                     corn   4828
104               corn_flake    225
105                corn_grit    163
106           cottage_cheese    347
107                     crab    574
108                cranberry    920
109                    cream  10171
110             cream_cheese   2840
111                 cucumber   1895
112                    cumin   3276
113               cured_pork    315
114                  currant    241
115                     date    378
116                     dill   1105
117                   durian      0
118                      eel     20
119                      egg  21025
120               egg_noodle    317
121               elderberry      5
122          emmental_cheese      1
123                   endive    115
124                enokidake    106
125                   fennel    912
126                fenugreek    924
127              feta_cheese    623
128                      fig    139
129                     fish   2110
130                   flower     32
131              frankfurter     37
132                    fruit    480
133                  galanga     49
134                 gardenia      9
135                   garlic  17353
136                  gelatin   1417
137                 geranium      1
138                      gin     68
139                   ginger   4358
140              goat_cheese    260
141                    grape    346
142             grape_brandy      8
143              grape_juice    824
144               grapefruit    121
145        green_bell_pepper   2582
146                green_tea     35
147           gruyere_cheese     45
148                    guava     13
149                  haddock     31
150                      ham   1300
151                 hazelnut    284
152                  herring     10
153               holy_basil      3
154                    honey   2552
155                      hop      3
156              horseradish    396
157              huckleberry     10
158             jamaican_rum      1
159            japanese_plum     13
160                  jasmine      8
161              jasmine_tea      2
162            juniper_berry     33
163              kaffir_lime      1
164                     kale    100
165              katsuobushi     63
166                     kelp    179
167              kidney_bean    442
168                     kiwi    109
169                 kohlrabi      6
170                  kumquat     33
171                     lamb    483
172                     lard   3052
173                   laurel      2
174                 lavender     62
175                     leaf      9
176                     leek    422
177                    lemon   3044
178              lemon_juice   5065
179               lemon_peel    729
180               lemongrass    217
181                   lentil    247
182                  lettuce   1206
183                 licorice     21
184         lilac_flower_oil      1
185                lima_bean    149
186                     lime   1160
187               lime_juice   1618
188            lime_peel_oil    108
189              lingonberry      9
190                   litchi     12
191                    liver     42
192                  lobster    131
193              long_pepper      2
194                   lovage    142
195            macadamia_nut    102
196                 macaroni   3115
197                     mace    117
198                 mackerel     44
199                     malt     37
200                 mandarin    279
201            mandarin_peel     15
202                    mango    418
203              maple_syrup    478
204                 marjoram    527
205                     mate      1
206                matsutake     57
207                     meat    987
208                    melon    163
209                     milk  12870
210                 milk_fat    959
211                     mint   1013
212        mozzarella_cheese   1289
213                mung_bean     24
214           munster_cheese     27
215             muscat_grape      1
216                 mushroom   3370
217                   mussel    168
218                  mustard   4119
219                   mutton      3
220                nectarine     51
221                     nira     67
222                      nut   1256
223                   nutmeg   2506
224                      oat   1265
225                  oatmeal     61
226                  octopus     45
227                     okra    102
228                    olive   1798
229                olive_oil   9876
230                    onion  18080
231                   orange   1724
232            orange_flower     17
233             orange_juice   1726
234              orange_peel    596
235                  oregano   3180
236                     ouzo      9
237                   oyster    406
238                     palm     46
239                   papaya     57
240          parmesan_cheese   3173
241                  parsley   5552
242                  parsnip    139
243            passion_fruit     20
244                      pea   1180
245                    peach    531
246                   peanut    509
247            peanut_butter   1014
248               peanut_oil    308
249                     pear    484
250              pear_brandy     11
251                    pecan   2177
252              pelargonium      1
253                   pepper   9230
254               peppermint    142
255           peppermint_oil      8
256                  pimenta      2
257                  pimento    270
258                pineapple   1638
259                pistachio    219
260                     plum    288
261                  popcorn     97
262                  porcini    106
263                     pork   2056
264               pork_liver      5
265             pork_sausage   1369
266                port_wine     49
267                   potato   3528
268              potato_chip     65
269                    prawn     24
270             prickly_pear     20
271         provolone_cheese    168
272                  pumpkin    803
273                   quince     29
274                   radish    525
275                   raisin   1890
276                 rapeseed      3
277                raspberry    784
278                 raw_beef      2
279                red_algae      2
280                 red_bean     33
281          red_kidney_bean     59
282                 red_wine   1395
283                  rhubarb    169
284                     rice   3857
285           roasted_almond      3
286             roasted_beef    227
287         roasted_hazelnut      1
288             roasted_meat     15
289              roasted_nut      1
290           roasted_peanut    202
291            roasted_pecan      1
292             roasted_pork    124
293      roasted_sesame_seed    594
294            romano_cheese    275
295                     root    101
296         roquefort_cheese     23
297                     rose     56
298                 rosemary   1892
299                      rum    599
300                 rutabaga     34
301                rye_bread     92
302                rye_flour    131
303                  saffron    234
304                     sage    904
305                     sake    680
306                   salmon    451
307               salmon_roe     15
308                sassafras     18
309               sauerkraut    185
310                   savory    128
311                 scallion   4782
312                  scallop    300
313                sea_algae      4
314                  seaweed    215
315                     seed   1340
316               sesame_oil   1693
317              sesame_seed    780
318                  shallot   1304
319             sheep_cheese      2
320                shellfish     27
321                   sherry    706
322                 shiitake    595
323                   shrimp   1679
324                    smoke    463
325              smoked_fish      6
326            smoked_salmon    100
327           smoked_sausage    268
328              sour_cherry     50
329                sour_milk     46
330                soy_sauce   3799
331                  soybean   1195
332              soybean_oil      2
333                spearmint      6
334                   squash    572
335                    squid    238
336               star_anise    131
337                   starch   2731
338               strawberry   1080
339           strawberry_jam      1
340         strawberry_juice      2
341          sturgeon_caviar      1
342                    sumac     11
343            sunflower_oil      8
344             sweet_potato    529
345             swiss_cheese    519
346           tabasco_pepper    976
347                 tamarind   1672
348                tangerine     52
349                 tarragon    478
350                      tea    108
351                  tequila    142
352              thai_pepper    136
353                    thyme   3043
354                   tomato   9920
355             tomato_juice    176
356                  truffle     52
357                     tuna    463
358                   turkey    901
359                 turmeric   1291
360                   turnip    188
361                  vanilla   9010
362                     veal    197
363                vegetable   1703
364            vegetable_oil  11105
365                  vinegar   8060
366                   violet      5
367                   walnut   2729
368                   wasabi    135
369               watercress    150
370               watermelon    110
371                    wheat  20781
372              wheat_bread     82
373                  whiskey    148
374              white_bread    370
375               white_wine   2205
376  whole_grain_wheat_flour    731
377                     wine   1026
378                     wood     33
379                      yam     85
380                    yeast   3385
381                   yogurt   1033
382                 zucchini   1102

Now we have a dataframe of ingredients and their total counts across all recipes. Let's sort this dataframe in descending order._____no_output_____

ing_df.sort_values(["count"], ascending=False, inplace=True)
ing_df.reset_index(inplace=True, drop=True)

print(ing_df)          ingredient  count
0                egg  21025
1              wheat  20781
2             butter  20719
3              onion  18080
4             garlic  17353
..               ...    ...
378   strawberry_jam      1
379  sturgeon_caviar      1
380      kaffir_lime      1
381            beech      1
382           durian      0

[383 rows x 2 columns]

#### What are the 3 most popular ingredients?_____no_output_____

However, note that there is a problem with the above table. There are ~40,000 American recipes in our dataset, which means that the data is biased towards American ingredients._____no_output_____**Therefore**, let's compute a more objective summary of the ingredients by looking at the ingredients per cuisine._____no_output_____#### Let's create a *profile* for each cuisine.

In other words, let's try to find out what ingredients Chinese people typically use, and what is **Canadian** food for example._____no_output_____

cuisines = recipes.groupby("cuisine").mean()
cuisines.head()_____no_output_____

As shown above, we have just created a dataframe where each row is a cuisine and each column (except for the first column) is an ingredient, and the row values represent the percentage of each ingredient in the corresponding cuisine.

**For example**:

* *almond* is present across 15.65% of all of the **African** recipes.
* *butter* is present across 38.11% of all of the **Canadian** recipes._____no_output_____Let's print out the profile for each cuisine by displaying the top four ingredients in each cuisine._____no_output_____

num_ingredients = 4 # define number of top ingredients to print

# define a function that prints the top ingredients for each cuisine
def print_top_ingredients(row):
    print(row.name.upper())
    row_sorted = row.sort_values(ascending=False)*100
    top_ingredients = list(row_sorted.index.values)[0:num_ingredients]
    row_sorted = list(row_sorted)[0:num_ingredients]

    for ind, ingredient in enumerate(top_ingredients):
        print("%s (%d%%)" % (ingredient, row_sorted[ind]), end=' ')
    print("\n")

# apply function to cuisines dataframe
create_cuisines_profiles = cuisines.apply(print_top_ingredients, axis=1)AFRICAN
onion (53%) olive_oil (52%) garlic (49%) cumin (42%)

AMERICAN
butter (41%) egg (40%) wheat (39%) onion (29%)

ASIAN
soy_sauce (49%) ginger (48%) garlic (47%) rice (41%)

CAJUN_CREOLE
onion (69%) cayenne (56%) garlic (48%) butter (36%)

CANADIAN
wheat (39%) butter (38%) egg (35%) onion (34%)

CARIBBEAN
onion (51%) garlic (50%) vegetable_oil (31%) black_pepper (31%)

CENTRAL_SOUTHAMERICAN
garlic (56%) onion (54%) cayenne (51%) tomato (41%)

CHINESE
soy_sauce (68%) ginger (53%) garlic (52%) scallion (48%)

EAST_ASIAN
garlic (55%) soy_sauce (50%) scallion (49%) cayenne (47%)

EASTERN-EUROPE
wheat (53%) egg (52%) butter (48%) onion (45%)

EASTERNEUROPEAN_RUSSIAN
butter (60%) egg (50%) wheat (49%) onion (38%)

ENGLISH_SCOTTISH
butter (67%) wheat (62%) egg (53%) cream (41%)

FRENCH
butter (50%) egg (44%) wheat (37%) olive_oil (27%)

GERMAN
wheat (64%) egg (60%) butter (47%) onion (34%)

GREEK
olive_oil (76%) garlic (44%) onion (36%) lemon_juice (33%)

INDIAN
cumin (60%) turmeric (50%) onion (49%) coriander (47%)

ITALIAN
olive_oil (60%) garlic (52%) tomato (39%) onion (32%)

JAPANESE
soy_sauce (56%) rice (44%) vinegar (36%) vegetable_oil (35%)

JEWISH
egg (58%) wheat (49%) butter (31%) onion (30%)

KOREAN
garlic (59%) scallion (52%) cayenne (51%) soy_sauce (49%)

MEDITERRANEAN
olive_oil (79%) garlic (50%) onion (38%) tomato (34%)

MEXICAN
cayenne (73%) onion (68%) garlic (62%) tomato (58%)

MIDDLEEASTERN
olive_oil (60%) garlic (46%) wheat (37%) lemon_juice (35%)

MOROCCAN
olive_oil (72%) cumin (54%) onion (49%) garlic (45%)

NORTH-AFRICAN
onion (55%) olive_oil (50%) cumin (48%) garlic (46%)

SCANDINAVIAN
butter (64%) wheat (57%) egg (52%) cream (28%)

SOUTH-AMERICA
onion (42%) garlic (36%) egg (34%) milk (31%)

SOUTHERN_SOULFOOD
butter (57%) wheat (48%) egg (41%) corn (29%)

SOUTHWESTERN
cayenne (81%) garlic (62%) onion (61%) cilantro (51%)

SPANISH_PORTUGUESE
olive_oil (57%) garlic (54%) onion (46%) bell_pepper (35%)

THAI
garlic (59%) fish (52%) cayenne (47%) cilantro (41%)

UK-AND-IRISH
butter (59%) wheat (58%) egg (48%) milk (33%)

VIETNAMESE
fish (73%) garlic (72%) rice (49%) cayenne (43%)

WESTERN
egg (51%) wheat (46%) butter (46%) black_pepper (36%)

At this point, we feel that we have understood the data well and the data is ready and is in the right format for modeling!

-----------_____no_output_____### Thank you for completing this lab!

This notebook was created by [Alex Aklson](https://www.linkedin.com/in/aklson/). We hope you found this lab session interesting. Feel free to contact us if you have any questions!_____no_output_____This notebook is part of a course on **Coursera** called *Data Science Methodology*. If you accessed this notebook outside the course, you can take this course, online by clicking [here](https://cocl.us/DS0103EN_Coursera_LAB3)._____no_output_____

Copyright © 2019 [Cognitive Class](https://cognitiveclass.ai/?utm_source=bducopyrightlink&utm_medium=dswb&utm_campaign=bdu). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/)._____no_output_____


# Notebook from sarodz/MLOps
Path: notebooks/feature_store.ipynb

 Made With ML

Applied ML · MLOps · Production

Join 30K+ developers in learning how to responsibly deliver value with ML.

    🔥  Among the top MLOps repositories on GitHub

_____no_output_____# Set up_____no_output_____

# Install Feast
!pip install feast==0.10.5 -q
!pip freeze | grep feastfeast==0.10.5

We're going to create a feature repository at the root of our project. [Feast](https://feast.dev/) will create a configuration file for us and we're going to add an additional [features.py](https://github.com/GokuMohandas/MLOps/blob/main/features/features.py) file to define our features.

> Traditionally, the feature repository would be it's own isolated repository that other services will use to read/write features from but we're going to simplify it and create it directly in our application's repository._____no_output_____

%%bash
cd ../
feast init --minimal --template local features
cd features
touch features.py
Creating a new Feast repository in /Users/goku/Documents/madewithml/mlops/features.

```bash
features/
├── feature_store.yaml  - configuration
└── features.py         - feature definitions
```_____no_output_____We're going to configure the locations for our registry and online store (SQLite) in our [feature_store.yaml](https://github.com/GokuMohandas/MLOps/blob/main/features/feature_store.yaml) file.
- **registry**: contains information about our feature repository, such as data sources, feature views, etc. Since it's in a DB, instead of a Python file, it can very quickly be accessed in production.
- **online store**: DB (SQLite for local) that stores the (latest) features for defined entites to be used for online inference.

If all definitions look valid, Feast will sync the metadata about Feast objects to the registry. The registry is a tiny database storing most of the same information you have in the feature repository. This step is necessary because the production feature serving infrastructure won't be able to access Python files in the feature repository at run time, but it will be able to efficiently and securely read the feature definitions from the registry.

```yaml
project: features
registry: ../stores/feature/registry.db
provider: local
online_store:
    path: ../stores/feature/online_store.db
```_____no_output_____# Data_____no_output_____Feast requires it's [data sources](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/data_source.py) to either come from a file ([Parquet](https://databricks.com/glossary/what-is-parquet)), data warehouse ([BigQuery](https://cloud.google.com/bigquery)) or data stream ([Kafka](https://kafka.apache.org/) / [Kinesis](https://aws.amazon.com/kinesis/)). We'll convert our generated features file (`features.json`) into a Parquet file.

> Read more about these data sources in our [pipelines](https://madewithml.com/courses/mlops/pipelines/#data) and [deployment](https://madewithml.com/courses/mlops/deployment/#batch-processing) lessons._____no_output_____

import pandas as pd
from pathlib import Path
from config import config
from tagifai import utils_____no_output_____# Load features to df
features_fp = Path(config.DATA_DIR, "features.json")
features = utils.load_dict(filepath=features_fp)
df = pd.DataFrame(features)_____no_output_____# Format timestamp
df.created_on = pd.to_datetime(df.created_on)_____no_output_____# Convert to parquet
df.to_parquet(
    Path(config.DATA_DIR, "features.parquet"),
    compression=None,
    allow_truncated_timestamps=True,
)_____no_output_____

# Feature definitions_____no_output_____Now that we have our data source prepared, we can define our features for the feature store._____no_output_____

from datetime import datetime
from feast import Entity, Feature, FeatureView, ValueType
from feast.data_source import FileSource
from google.protobuf.duration_pb2 import Duration
from config import config_____no_output_____

The first step is to define the location of the features (FileSource in our case) and the timestamp column for each data point._____no_output_____

# Read data
START_TIME = "2020-02-17"
project_details = FileSource(
    path=str(Path(config.DATA_DIR, "features.parquet")),
    event_timestamp_column="created_on",
)/Users/goku/Documents/madewithml/mlops/venv/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)

Next, we need to define the main entity that each data point pertains to. In our case, each project has a unique ID with features such as text and tags._____no_output_____

# Define an entity
project = Entity(
    name="id",
    value_type=ValueType.INT64,
    description="project id",
)_____no_output_____

Finally, we're ready to create a [FeatureView](https://docs.feast.dev/concepts/feature-views) that loads specific features (`features`), of various [value types](https://api.docs.feast.dev/python/feast.html?highlight=valuetype#feast.value_type.ValueType), from a data source (`input`) for a specific period of time (`ttl`)._____no_output_____

# Define a Feature View for each project
project_details_view = FeatureView(
    name="project_details",
    entities=["id"],
    ttl=Duration(
        seconds=(datetime.today() - datetime.strptime(START_TIME, "%Y-%m-%d")).days * 24 * 60 * 60
    ),
    features=[
        Feature(name="text", dtype=ValueType.STRING),
        Feature(name="tags", dtype=ValueType.STRING_LIST),
    ],
    online=True,
    input=project_details,
    tags={},
)_____no_output_____

Once we've defined our feature views, we can `apply` it to push a version controlled definition of our features to the registry for fast access. It will also configure our registry and online stores that we've defined in our [feature_store.yaml](https://github.com/GokuMohandas/MLOps/blob/main/features/feature_store.yaml)_____no_output_____

%%bash
cd ../features
feast applyRegistered entity id
Registered feature view project_details
Deploying infrastructure for project_details

# Historical features_____no_output_____Once we've registered our feature definition, along with the data source, entity definition, etc., we can use it to fetch historical features. This is done via joins using the provided timestamps using pandas (local) or BigQuery (production)._____no_output_____

import pandas as pd
from feast import FeatureStore/Users/goku/Documents/madewithml/mlops/venv/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)
# Identify entities
project_ids = [1, 2, 3]
now = datetime.now()
timestamps = [datetime(now.year, now.month, now.day)]*len(project_ids)
entity_df = pd.DataFrame.from_dict({"id": project_ids, "event_timestamp": timestamps})
entity_df.head()_____no_output_____# Get historical features
store = FeatureStore(repo_path=Path(config.BASE_DIR, "features"))
training_df = store.get_historical_features(
    entity_df=entity_df,
    feature_refs=["project_details:text", "project_details:tags"],
).to_df()
training_df.head()/Users/goku/Documents/madewithml/mlops/venv/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)

# Materialize_____no_output_____For online inference, we want to retrieve features very quickly via our online store, as opposed to fetching them from slow joins. However, the features are not in our online store just yet, so we'll need to [materialize](https://docs.feast.dev/quickstart#4-materializing-features-to-the-online-store) them first._____no_output_____

%%bash
cd ../features
CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
feast materialize-incremental $CURRENT_TIME/Users/goku/Documents/madewithml/mlops/venv/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)

This has moved the features for all of our projects into the online store since this was first time materializing to the online store. When we subsequently run the [`materialize-incremental`](https://docs.feast.dev/how-to-guides/load-data-into-the-online-store#2-b-materialize-incremental-alternative) command, Feast keeps track of previous materializations and so we'll only materialize the new data since the last attempt._____no_output_____# Online features_____no_output_____

# Get online features
store = FeatureStore(repo_path=Path(config.BASE_DIR, "features"))
feature_vector = store.get_online_features(
    feature_refs=["project_details:text", "project_details:tags"],
    entity_rows=[{"id": 1000}],
).to_dict()
feature_vector/Users/goku/Documents/madewithml/mlops/venv/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)