# Notebook from runbikeswim/RKI-COVID-15-Data-Sources-Analysis
Path: RKI_COVID_Data_Comparison.ipynb

# Compare and Analyze Data on COVID-19 Infections Provided by the [Robert Koch Institute (RKI)](https://www.rki.de/EN/Home/homepage_node.html)_____no_output_____The [Robert Koch Institute (RKI)](https://www.rki.de/EN/Home/homepage_node.html) is the federal government agency responsible for disease control and prevention in Germany. Is publishes data about COVID-19 for all of Germany and uses various channels for that (see [this](https://www.rki.de/EN/Content/infections/epidemiology/outbreaks/COVID-19/COVID19.html) page for an overview).

This notebook has been created for analyzing and comparing data from two different sources that are updated by the RKI daily, but have a different level of detail. The main objective is to understand how the very fine grained numbers provided via github can be aggregated such that they match what is shown in the
[RKI's COVID-19 dashboard](https://corona.rki.de/)._____no_output_____## Preliminaries_____no_output_____

import json
import humanize

import datetime as dt
import numpy as np
import pandas as pd
import plotly.express as px

import local_constants as LC

from math import sqrt
from urllib.request import urlopen
from IPython.display import Image_____no_output_____

## Load and Analyze Data From the [NPGEO Corona Hub 2020](https://npgeo-corona-npgeo-de.hub.arcgis.com/)_____no_output_____For all German districts, up-to-date COVID-19 data is available via [this](https://opendata.arcgis.com/datasets/917fc37a709542548cc3be077a786c17_0) page. This data appears to be the basis for the [COVID-19 dashboard](https://corona.rki.de/)._____no_output_____### Read Data_____no_output_____

# load main data, but restrict the created dataframe to the most relevant columns
RKI_ARCGIS_COVID_BY_DISTRICT = \
    pd.read_csv(LC.RKI_ARCGIS_URL, usecols=list(LC.RKI_ARCGIS_COLUMN_NAME_MAPPER.keys()))\
        .rename(columns=LC.RKI_ARCGIS_COLUMN_NAME_MAPPER)\
        .sort_values(by="district ID")\
        .set_index("district ID")

# fill missing values
RKI_ARCGIS_COVID_BY_DISTRICT.fillna(method="pad", axis="index", inplace=True)

# convert to datetime64[ns]
RKI_ARCGIS_COVID_BY_DISTRICT["update time"] = pd.to_datetime(RKI_ARCGIS_COVID_BY_DISTRICT["update time"], format="%d.%m.%Y, %H:%M Uhr")

# add column for the number of deaths within the last seven days adjusted to a population size of 100.000 people
RKI_ARCGIS_COVID_BY_DISTRICT["deaths last 7 days per 100k"] = \
    10**5 * RKI_ARCGIS_COVID_BY_DISTRICT["deaths last 7 days"] \
    / RKI_ARCGIS_COVID_BY_DISTRICT["population"]_____no_output_____# print date of most recent entry in the data

print(RKI_ARCGIS_COVID_BY_DISTRICT["update time"].max().strftime("last update is from %Y-%m-%d"))last update is from 2022-04-18

## Load and Analyze RKI's Data on COVID-19 From [GitHub](https://github.com/robert-koch-institut/SARS-CoV-2_Infektionen_in_Deutschland)_____no_output_____Repository ["SARS-CoV-2 Infektionen in Deutschland"](https://github.com/robert-koch-institut/SARS-CoV-2_Infektionen_in_Deutschland) (SARS-CoV-2 Infections in Germany) contains
up-to-date numbers of COVID-19 cases in Germany. The data appears to be what is reported by the districts to the RKI as it lists new cases based on the reporting date, beginning of the disease (reference date), age group, sex and district.

In [Readme.md](https://github.com/robert-koch-institut/SARS-CoV-2_Infektionen_in_Deutschland/blob/master/Readme.md), an explanation for the data is provided (in German). _____no_output_____### Read Metadata_____no_output_____The RKI publishes metadata based on [zenodo's](https://about.zenodo.org/) JSON format. Here, it is used to detect the publication date of the data. Typically, this is shortly after 3 AM of the current day (local time in Germany). _____no_output_____

RKI_GITHUB_METADATA = json.loads(urlopen(LC.RKI_GITHUB_RAW_DATA_BASE_URL + LC.RKI_GITHUB_ZENODO_REL_URL).read())
RKI_GITHUB_PUBLICATION_DATE = pd.to_datetime(RKI_GITHUB_METADATA["publication_date"])
print(f"publication date is {RKI_GITHUB_PUBLICATION_DATE:%Y-%m-%d %H:%M:%S%z}")publication date is 2022-04-18 03:01:57+0200

### Read Data_____no_output_____Load data describing COVID-19 infections and deaths etc._____no_output_____

RKI_GITHUB_COVID_INFECTIONS = pd.read_csv(LC.RKI_GITHUB_RAW_DATA_BASE_URL + LC.RKI_GITHUB_COVID_INFECTIONS_REL_URL,
                                          converters=LC.RKI_GITHUB_VALUE_CONVERTERS)\
                                .rename(columns=LC.RKI_GITHUB_COLUMN_NAME_MAPPER)\
                                .astype(LC.RKI_GITHUB_COLUMN_TYPES_MAPPER)_____no_output_____

### Trying to Understand [Readme.md](https://github.com/robert-koch-institut/SARS-CoV-2_Infektionen_in_Deutschland/blob/master/Readme.md)_____no_output_____The text in `Readme.md` states that negative values in column `cases` (`AnzahlFall` in the original data) are for correcting positives cases that have been falsely reported in the past. The same is described for columns
`deaths` (`AnzahlTodesfall` in the original data) and `recovered` (`AnzahlGenesen` in the original data).

This can be interpreted in a way that the correct number of total cases can be computed by simply adding all values of column `cases`, such that negative values "delete" the cases reported erroneously before. _____no_output_____

print("total cases for Germany considering also negative values:")
print(f' GitHub: {RKI_GITHUB_COVID_INFECTIONS["cases"].sum().sum():,d}')

# sum of all cases in the data from ARCGIS
print(f' ARCGIS: {RKI_ARCGIS_COVID_BY_DISTRICT["cases"].sum():,d}')total cases for Germany considering also negative values:
 GitHub: 23,436,950
 ARCGIS: 23,437,145

... but this number is **too small**.

The number that is shown on the dashboard can be retrieved from the data by adding _all positive values_ in column `cases`. _____no_output_____

print("total cases for Germany:")
# print the sum for all positive values in column "cases"
print(f' GitHub: {RKI_GITHUB_COVID_INFECTIONS.loc[RKI_GITHUB_COVID_INFECTIONS["cases"] > 0, "cases"].sum():,d}')
# sum of all cases in the data from ARCGIS
print(f' ARCGIS: {RKI_ARCGIS_COVID_BY_DISTRICT["cases"].sum():,d}')total cases for Germany:
 GitHub: 23,437,145
 ARCGIS: 23,437,145

### Compute Totals_____no_output_____Based on the interpretation of the data described above, the sum of columns `cases`, `deaths` and `recovered` is calculated for each district.

For the sake of convenience, a new dataframe is defined, in which negative values for the number of COVID-19 cases, deaths and recovered patients are set to zero. This is used later for the aggregation of data._____no_output_____

# define dataframe that without the negative values in RKI_GITHUB_COVID_INFECTIONS

RKI_GITHUB_COVID_INFECTIONS_WITHOUT_NEGATIVES = \
    pd.concat([RKI_GITHUB_COVID_INFECTIONS[["district ID", "age group", "sex", "reporting date", "reference date"]],
              RKI_GITHUB_COVID_INFECTIONS[["cases", "deaths", "recovered"]].apply(lambda a: np.maximum(a,0))],
              axis="columns")_____no_output_____#  sum up "cases", "deaths", "recovered" for each district
RKI_GITHUB_COVID_BY_DISTRICT_TOTALS = \
    RKI_GITHUB_COVID_INFECTIONS_WITHOUT_NEGATIVES[["district ID", "cases", "deaths", "recovered"]].groupby(by="district ID").sum()

# copy population size for each district from RKI_ARCGIS_COVID_BY_DISTRICT
RKI_GITHUB_COVID_BY_DISTRICT_TOTALS["population"] = RKI_ARCGIS_COVID_BY_DISTRICT["population"]_____no_output_____

### Compute Numbers per 100K People _____no_output_____The sums in columns `cases`, `deaths` and `recovered` are normalized by the population size of the district.
For the district's population size, the data from the [NPGEO Corona Hub 2020](https://npgeo-corona-npgeo-de.hub.arcgis.com/) is used._____no_output_____

# define a new dataframe by dividing the totals by the population size of each district and multiplying that with 100,000

RKI_GITHUB_COVID_BY_DISTRICT_PER_100K = \
    pd.DataFrame(data=10**5 * RKI_GITHUB_COVID_BY_DISTRICT_TOTALS[["cases", "deaths", "recovered"]].values \
                 / np.array(3 * [RKI_GITHUB_COVID_BY_DISTRICT_TOTALS["population"].values]).T,
                 index=RKI_GITHUB_COVID_BY_DISTRICT_TOTALS.index,
                 columns=["cases per 100k", "deaths per 100k", "recovered per 100k"])_____no_output_____

### Compute Totals for the Last Seven Days_____no_output_____The values in columns `cases`, `deaths` and `recovered` of the last seven days are summed up._____no_output_____

# define the date of the earlist data that should be considered
number_of_days = 7
cut_off_date = np.datetime64(RKI_GITHUB_PUBLICATION_DATE.date() - dt.timedelta(days=number_of_days))

# compute sums for data that has been reported on or after the cut_off_date
RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_TOTALS = \
        RKI_GITHUB_COVID_INFECTIONS_WITHOUT_NEGATIVES\
                .loc[RKI_GITHUB_COVID_INFECTIONS_WITHOUT_NEGATIVES["reporting date"] >= cut_off_date]\
                .groupby(by="district ID").sum()\
                .rename(columns={"cases": "cases last 7 days", "deaths": "deaths last 7 days", "recovered": "recovered last 7 days"})

# ensure that there is data for each district by filling missing data with zeros
# define a dataframe containing the most recent data of each district
df = RKI_GITHUB_COVID_INFECTIONS_WITHOUT_NEGATIVES[["district ID", "reporting date"]].sort_values(by=["district ID", "reporting date"])\
        .groupby(["district ID"]).last()

missing_rows = df.loc[df["reporting date"] < cut_off_date].index.shape[0]
if  missing_rows > 0:
    # there are districts that have not reported data within the last 7 days
    # create dataframe containing the zeroes filling the missing data
    ddf = pd.DataFrame(data=np.zeros((missing_rows, RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_TOTALS.shape[1]), dtype=np.int64),
                       index=df.loc[df["reporting date"] < cut_off_date].index,
                       columns=RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_TOTALS.columns)
    # append zeros
    RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_TOTALS = RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_TOTALS.append(ddf)_____no_output_____

### Compute Numbers for the Last Seven Days per 100K People _____no_output_____normalize the sums for the last seven days by the districts' population size _____no_output_____

# define a new dataframe computing values by dividing the totals for the last seven days by the population size
# of each district and multiplying it with 100,000

RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_PER_100K = \
    pd.DataFrame(data=10**5 * RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_TOTALS.values / np.array(3 * \
        [RKI_GITHUB_COVID_BY_DISTRICT_TOTALS.loc[RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_TOTALS.index, "population"]]).T,
        index=RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_TOTALS.index,
        columns=["cases last 7 days per 100k", "deaths last 7 days per 100k", "recovered last 7 days per 100k"])_____no_output_____

### Create a DataFrame Containing all Values Derived From the COVID-19 Data from GitHub_____no_output_____

# combine all of the dataframes with data that has been derived from RKI_GITHUB_COVID_INFECTIONS into one dataframe,
# only columns "district name" and "state name" are taken from RKI_ARCGIS_COVID_BY_DISTRICT

RKI_GITHUB_COVID_BY_DISTRICT = \
    pd.concat([RKI_ARCGIS_COVID_BY_DISTRICT[["district name", "state name"]],
               RKI_GITHUB_COVID_BY_DISTRICT_TOTALS,
               RKI_GITHUB_COVID_BY_DISTRICT_PER_100K,
               RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_TOTALS,
               RKI_GITHUB_COVID_BY_DISTRICT_LAST_7_DAYS_PER_100K], axis="columns")_____no_output_____

## Verify that Derived Data From the [COVID-19 Data from GitHub](https://github.com/robert-koch-institut/SARS-CoV-2_Infektionen_in_Deutschland) and the Data From [NPGEO Corona Hub 2020](https://npgeo-corona-npgeo-de.hub.arcgis.com/) are (More or Less) Identical_____no_output_____

EPSILON = 10**-10 # threshold for treating float values as zero
common_columns = list(set(RKI_ARCGIS_COVID_BY_DISTRICT.columns) & set(RKI_GITHUB_COVID_BY_DISTRICT.columns))
common_numerical_columns = [c for c in common_columns if RKI_GITHUB_COVID_BY_DISTRICT[c].dtypes != object]

# return True, if all the differences between the absolute values in the common numerical columns is smaller than the threshold (EPSILON)

np.amax(np.abs(RKI_ARCGIS_COVID_BY_DISTRICT[common_numerical_columns].values \
                 - RKI_GITHUB_COVID_BY_DISTRICT[common_numerical_columns].values)) < EPSILON_____no_output_____

## Show Some Data_____no_output_____Based on the result of the verification above, it seems that the interpretation of the data that is provided via GitHub is correct ... or at least identical with what is shown
in the COVID-19 Dashboard. Hence, the following list of the most affected districts in Germany can be assumed to be correct._____no_output_____

n = 30
RKI_GITHUB_COVID_BY_DISTRICT.sort_values(by="cases last 7 days per 100k", ascending=False)\
    .head(n).style.hide(axis="index").format(LC.FORMAT_MAPPER)_____no_output_____

Show sums by state_____no_output_____

base_columns = ["cases", "deaths", "recovered"]
last_7_days_columns = [c + " last 7 days" for c in base_columns]
index_column = "state name"
columns = [index_column, "population"] + base_columns + last_7_days_columns
RKI_GITHUB_COVID_BY_STATE = RKI_GITHUB_COVID_BY_DISTRICT[columns].groupby(by=index_column).sum()

per_100k_columns = [c + " per 100k" for c in base_columns]
last_7_days_per_100k_columns = [c + " per 100k" for c in last_7_days_columns]
RKI_GITHUB_COVID_BY_STATE[per_100k_columns + last_7_days_per_100k_columns] = \
    10**5 * RKI_GITHUB_COVID_BY_STATE[base_columns + last_7_days_columns].values\
        / np.array(6 * [RKI_GITHUB_COVID_BY_STATE["population"]]).T
RKI_GITHUB_COVID_BY_STATE.style.format(LC.FORMAT_MAPPER)_____no_output_____

equally, the following totals for all of Germany appear correct_____no_output_____

columns = ["population", "cases", "cases last 7 days", "deaths", "deaths last 7 days", "recovered", "recovered last 7 days"]
RKI_GITHUB_COVID_BY_DISTRICT[columns].sum().to_frame().T.style.hide(axis="index").format(LC.FORMAT_MAPPER)_____no_output_____

likewise, the sums for the last 7 days per 100,000 people can be computed_____no_output_____

columns = ["cases last 7 days", "deaths last 7 days", "recovered last 7 days"]
(10**5 * RKI_GITHUB_COVID_BY_DISTRICT[columns].sum() / RKI_GITHUB_COVID_BY_DISTRICT["population"].sum())\
    .to_frame().T.rename(columns={c:c+" per 100k" for c in columns}).style.hide(axis="index").format(LC.FORMAT_MAPPER)_____no_output_____

For the increase in the numbers of cases, deaths and recovered people within the last day, the data for the previous day is loaded and subtracted from the totals of the current day._____no_output_____

PREVIOUS_DAY = pd.Timestamp(RKI_GITHUB_PUBLICATION_DATE.date() - dt.timedelta(days=1))
RKI_GITHUB_COVID_INFECTIONS_PREVIOUS_DAY \
    = pd.read_csv(LC.RKI_GITHUB_RAW_DATA_BASE_URL + "/Archiv" + \
                  PREVIOUS_DAY.strftime("/%Y-%m-%d_Deutschland_SarsCov2_Infektionen.csv"),
                  converters=LC.RKI_GITHUB_VALUE_CONVERTERS)\
        .rename(columns=LC.RKI_GITHUB_COLUMN_NAME_MAPPER)\
        .astype(LC.RKI_GITHUB_COLUMN_TYPES_MAPPER)

RKI_GITHUB_COVID_INFECTIONS_PREVIOUS_DAY_WITHOUT_NEGATIVES = \
    pd.concat([RKI_GITHUB_COVID_INFECTIONS_PREVIOUS_DAY[["district ID", "age group", "sex", "reporting date", "reference date"]],
               RKI_GITHUB_COVID_INFECTIONS_PREVIOUS_DAY[["cases", "deaths", "recovered"]].apply(lambda a: np.maximum(a,0))],
               axis="columns")

(RKI_GITHUB_COVID_INFECTIONS_WITHOUT_NEGATIVES[["cases", "deaths", "recovered"]].sum()
    - RKI_GITHUB_COVID_INFECTIONS_PREVIOUS_DAY_WITHOUT_NEGATIVES[["cases", "deaths", "recovered"]].sum())\
        .to_frame().T.style.hide(axis="index").format(LC.FORMAT_MAPPER)_____no_output_____

### Compute Totals by Age Group and Sex_____no_output_____In order to close this notebook with something that is not mysterious, the total number of cases, deaths and recovered people is computed by age group and sex.
This is again in sync with the data of the COVID-19 dashboard._____no_output_____

RKI_GITHUB_COVID_BY_AGE_GROUP_AND_SEX_TOTALS = \
    RKI_GITHUB_COVID_INFECTIONS_WITHOUT_NEGATIVES[["age group", "sex", "cases", "deaths", "recovered"]]\
        .groupby(by=["age group", "sex"]).sum()
RKI_GITHUB_COVID_BY_AGE_GROUP_AND_SEX_TOTALS.style.format(LC.FORMAT_MAPPER)_____no_output_____

## Compare RKI Data with WHO Data_____no_output_____The World Health Organization (WHO) publishes data on COVID-19 for a large number of countries (see [covid19.who.int](https://covid19.who.int/)), including Germany. Apparently, these numbers differ from each other.
The following analysis quantifies those differences._____no_output_____

# load data from WHO
WHO_COVID_WORLDWIDE = pd.read_csv("https://covid19.who.int/WHO-COVID-19-global-data.csv", parse_dates=["Date_reported"])\
                        .rename(columns=LC.WHO_COLUMN_NAME_MAPPER).set_index("reporting date")_____no_output_____last_update = WHO_COVID_WORLDWIDE.loc[WHO_COVID_WORLDWIDE["country"] == "Germany"].index.max()
print(f"most recent data for Germany is from {last_update.strftime('%Y-%m-%d')}")most recent data for Germany is from 2022-04-14
# verify that the cumulative sums for Germany are correct
assert (WHO_COVID_WORLDWIDE[WHO_COVID_WORLDWIDE["country"] == "Germany"][["cases", "deaths"]]\
            .cumsum().rename(columns={"cases":"cumulative cases", "deaths":"cumulative deaths"}) == \
            WHO_COVID_WORLDWIDE[WHO_COVID_WORLDWIDE["country"] == "Germany"][["cumulative cases", "cumulative deaths"]]).all().all()_____no_output_____# define a start and end date that covers all the data available from RKI and WHO

START_DATE = min(RKI_GITHUB_COVID_INFECTIONS_WITHOUT_NEGATIVES["reporting date"].min(),
                 WHO_COVID_WORLDWIDE[WHO_COVID_WORLDWIDE["country"] == "Germany"].index.min())
END_DATE = max(RKI_GITHUB_COVID_INFECTIONS_WITHOUT_NEGATIVES["reporting date"].max(),
                 WHO_COVID_WORLDWIDE[WHO_COVID_WORLDWIDE["country"] == "Germany"].index.max())
DATE_RANGE = pd.date_range(start=START_DATE, end=END_DATE)_____no_output_____# define DataFrames with identical structure for the data from RKI and WHO on COVID-19 cases and deaths in Germany

RKI_COVID_GERMANY = RKI_GITHUB_COVID_INFECTIONS_WITHOUT_NEGATIVES[["reporting date", "cases", "deaths"]]\
                        .groupby(by="reporting date").sum().reindex(index=DATE_RANGE)
WHO_COVID_GERMANY = WHO_COVID_WORLDWIDE[WHO_COVID_WORLDWIDE["country"] == "Germany"][["cases", "deaths"]]\
                        .reindex(index=DATE_RANGE)_____no_output_____

Due to possible delays in the transmission and processing of data, it is likely that the time axis (index `reporting date`) is not in aligned for the two sets of data (i.e. RKI and WHO).
Hence a optimal time shift of the WHO's data is computed based on the euclidean distance of the two vectors._____no_output_____

# determine best shift (as multiplies of one day) for the WHO-data, so that the euclidean distance for cases
# and deaths combined is minimal
IMAGE_WIDTH, IMAGE_HEIGHT = 1500, 700
window, best_shift, min_distance, x, y = 7, None, float("Inf"), [], []
for shift in range(-window,window+1):
    distance = sqrt(np.nansum(np.square((RKI_COVID_GERMANY \
                                         - WHO_COVID_GERMANY.shift(periods=shift, freq=dt.timedelta(days=1))).values)))
    x.append(shift)
    y.append(distance)
    if distance < min_distance:
        best_shift, min_distance = shift, distance

fig = px.line(pd.DataFrame(data = {"x": x, "y":y}),
              x="x", y="y",
              title = f"Best Shift in Days is {best_shift:+3d} With Minimal Euclidian Distance of {min_distance:11,.2f}",
              labels = {"y": "euclidean distance", "x": "shift in days"},
              markers = True)

img_bytes = fig.to_image(format = "png",  width=IMAGE_WIDTH, height=IMAGE_HEIGHT)
display(Image(img_bytes))

WHO_SHIFT = dt.timedelta(days=best_shift)
WHO_COVID_GERMANY_SHIFTED = WHO_COVID_GERMANY.shift(periods=best_shift, freq=dt.timedelta(days=1)).reindex(index=DATE_RANGE)_____no_output_____

after aligning the data as good as possible, there should be difference in the total number of COVID-19 cases and deaths - unfortunately, this is not the case ..._____no_output_____

print("Difference in Totals Between Data From RKI and WHO for the Timeframe From"\
      + f"{humanize.naturaldate(START_DATE)} Until {humanize.naturaldate(END_DATE + WHO_SHIFT)}")

df = pd.concat([pd.DataFrame(data=RKI_COVID_GERMANY.loc[START_DATE:END_DATE + WHO_SHIFT].sum().to_dict(), index=["RKI"]),
                pd.DataFrame(data=WHO_COVID_GERMANY_SHIFTED.loc[START_DATE:END_DATE + WHO_SHIFT].sum().to_dict(), index=["WHO"]),],
               axis="index")
df = pd.concat([df, pd.DataFrame((df.iloc[0] - df.iloc[1]).to_dict(), index=["difference"])], axis="index")
df.style.format("{:,.0f}")Difference in Totals Between Data From RKI and WHO for the Timeframe FromJan 02 2020 Until Apr 16

maybe there is some obvious flaw, which can be see in the diagram showing the differences of a daily basis ..._____no_output_____

fig = px.line((RKI_COVID_GERMANY
               - WHO_COVID_GERMANY_SHIFTED).reset_index(), x="index", y=["cases", "deaths"],
               title="Differences in Daily Numbers Provided by RKI and WHO for COVID Cases and Deaths in Germany",
               labels={"value": "difference", "index": "time", "variable": ""})

img_bytes = fig.to_image(format="png",  width=IMAGE_WIDTH, height=IMAGE_HEIGHT)
Image(img_bytes)_____no_output_____

compare "smoothed" data_____no_output_____

window = '14d'
fig = px.line((RKI_COVID_GERMANY.rolling(window, center=True, min_periods=1).mean()
               - WHO_COVID_GERMANY_SHIFTED.rolling(window, center=True, min_periods=1).mean()).reset_index(),
               x="index", y=["cases", "deaths"],
               title=f"Differences in Moving Averages ({window}-Window) of Daily Number "+
                      "Provided by RKI and WHO for COVID Cases and Deaths in Germany",
               labels={"value": "difference", "index": "time", "variable": ""})
img_bytes = fig.to_image(format="png",  width=IMAGE_WIDTH, height=IMAGE_HEIGHT)
Image(img_bytes)_____no_output_____

show differences in the data sets based on the cumulative sum of cases and deaths_____no_output_____

column = "cases"
df = pd.concat([RKI_COVID_GERMANY[[column]].cumsum().rename(columns={column: column + " - RKI"}),\
                WHO_COVID_GERMANY_SHIFTED[[column]].cumsum().rename(columns={column: column + " - WHO"})], axis="columns").reset_index()
fig = px.line(df, x="index", y=[column + " - RKI", column + " - WHO"],
                title=f"Cumulative Sums in COVID {column.capitalize()} in Germany from RKI and WHO",
        labels={"value": column, "index": "time", "variable": ""})
img_bytes = fig.to_image(format="png",  width=IMAGE_WIDTH, height=IMAGE_HEIGHT)
Image(img_bytes)_____no_output_____column = "deaths"
df = pd.concat([RKI_COVID_GERMANY[[column]].cumsum().rename(columns={column: column + " - RKI"}),\
                WHO_COVID_GERMANY_SHIFTED[[column]].cumsum().rename(columns={column: column + " - WHO"})], axis="columns").reset_index()
fig = px.line(df, x="index", y=[column + " - RKI", column + " - WHO"],
              title=f"Cumulative Sums in COVID {column.capitalize()} in Germany from RKI and WHO",
              labels={"value": column, "index": "time", "variable": ""})
img_bytes = fig.to_image(format="png",  width=IMAGE_WIDTH, height=IMAGE_HEIGHT)
Image(img_bytes)_____no_output_____


# Notebook from kfollette/AST337-Fall2017
Path: Labs/Lab2/Lab2.ipynb

# AST 337 In-Class Lab #2
### Wednesday, September 13, 2017

### _Names: [insert your names here]_

***

In this lab, you will (1) work directly with published astronomical data from the _VizieR_ database, (2) continue using pandas and matplotlib to read, manipulate, and plot datasets, and (3) gain experience writing functions.

On the science end, you will create color-magnitude diagrams -- the observer's H-R diagram -- for various stellar populations, compare the clusters, and relate these to the results from last week. _____no_output_____

# Load packages
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline_____no_output_____

The first step is to download the datasets we need from VizieR for the following clusters:
* M22 (globular cluster) -- _BVI photometry of M22_ (Monaco+, 2004) -- **we'll download this one together as a class**
* M4 (globular cluster) -- _M4 UBV color-magnitude diagrams_ (Mochejska+, 2002)
* M67 (open cluster) -- _BVI photometry in M67_ (Stassun+, 2002)
* NGC 188 (open cluster) -- _A star catalog for the open cluster NGC 188_ (Stetson+, 2004)

Save each of these as semicolon delimited .tsv files in the directory with this notebook._____no_output_____First, we'll need to read the datasets into pandas, as we did in last week's lab. However, VizieR gives us more information than just the raw data to look at, so we'll need to give pd.read_csv additional information so it can parse the datafile.

Let's first look at the actual M22 datafile, by opening the file in Jupyter's text editor mode. Go back to the browser tab with the directory containing this notebook, and double-click on the file you just saved, M22.tsv. Take a look at the contents of the file, then come back to this notebook._____no_output_____#### Questions:

1) What information is contained in the header?

-

2) How many commented "#" lines are there before the dataset begins? (see the line numbers)

-_____no_output_____As useful as the header information is for us, pandas only needs to know where the data values start and what the column headers are.

To help pandas parse the data easily, edit the text file in the other Jupyter tab to add the '#' symbol at the beginnings of the two rows describing the column units and dashed lines. Be sure to save the text file!

We will now tell pandas to skip any commented rows and that the file is semicolon delimited by adding parameters to the read_csv function, separated by commas:
* comment = '#'
* delimiter = ';'

**EDIT** the cell below to add these parameters to the regular pd.read_csv command, then run the cell to read in the file._____no_output_____

m22 = pd.read_csv('M22.tsv')_____no_output_____

As before, we would like to check the table and its contents: _____no_output_____

m22_____no_output_____

And we would like to check the column headers of the dataset and the data types (dtypes). Do so in the following two cells for the M22 dataset. _____no_output_____

# Check the columns here_____no_output_____# Check the datatypes here. Do any columns need to be converted from object to float?_____no_output_____

## Calculating values for color-magnitude diagrams (CMDs)

We are most interested in the measured B and V magnitude columns for the stars in these clusters. However, these are both _apparent_ magnitudes, and we want to use absolute magnitude for the V vs. B-V color-magnitude diagram.

Therefore, for each dataset we downloaded, we will need to do four things:

* Read in the datafiles.

* Ensure the data columns we want to manipulate have the appropriate data type. (Depending on the dataset, we might need to use the pd.to_numeric function, as in Lab1.)

* Use the apparent magnitude and distance to calculate the absolute V magnitude (Y-axis proxy for luminosity).

* Use the B and V magnitudes to calculate the color (X-axis proxy for temperature).

In the next steps, we are going to calculate new pandas series for absolute V magnitude and B-V color and add them to our existing M22 dataframe._____no_output_____#### Questions:

3) What quantities are needed to calculate an absolute magnitude?

    -

4) Do we need to use apparent magnitudes or absolute magnitudes to calculate the B-V color? Why?

    -_____no_output_____### Adding a new column to an existing pandas dataframe

For the X-axis of our CMD, we want to add a column to our existing data frame with the new calculated B-V value. With pandas, this is very simple -- all we have to do is define a new column label, and subtract the existing columns!

Let's first remind ourselves what the M22 dataframe/table looks like right now. We can view a snippet of the top of the table by using the following method, which shows the first five rows only:_____no_output_____

m22.head()_____no_output_____

In pandas, dataframes have both "heads" and "tails". What do you expect the "tail" method to do? Try in the cell below._____no_output_____Now add a new column for B-V color to the existing M22 dataframe. We can label this column whatever we like, _as long as the label isn't already used for a different column_ -- if the column label already exists, it will overwrite it!

Let's label the new column "BVcolor", the values of which are the differences between values in the existing B and V columns:_____no_output_____

m22['BVcolor'] = m22['Bmag'] - m22['Vmag']_____no_output_____

Did that work? Check by viewing the table in the cell below. _____no_output_____

m22.head() # Could also do m22.columns_____no_output_____

What if we wanted to calculate the V-I color instead of B-V? Add a new V-I column to the dataframe, and check to ensure the dataframe has updated:_____no_output_____

# Calculate a new column labeled "VIcolor"_____no_output_____# Check for the updated dataframe column_____no_output_____

### A brief overview of functions in python_____no_output_____Programs often involve tasks that must be done repetitively, or there are tasks that we want to perform that are common to many programs. We will write a function that uses the distance modulus equation and calculates absolute magnitudes.

An example of a standard python function is the logarithm function, which is built into python within the _numpy_ package (short for "numerical python"). We saw last week that `np.log10` takes the base 10 logarithm of the input, and returns the corresponding power:_____no_output_____

input_value = 1000.0 # define a variable
return_value = np.log10(input_value) # use that variable within a function
print(return_value) # print the output of the function, which has been saved to the new variable "return_value"_____no_output_____

There are many, many reasons why one might want to take the log of something, so it is useful to have the log function defined once and for all in a standard python package. This way, any program that needs to take the log can do so, rather than having the user come up with it again and again. But what if the function we want to use does not exist in python?

The capability that we are seeking is provided by defining new functions. This allows us to make our own functions that are just like the log function, and can be called in a similar way. Functions are defined by the following syntax:_____no_output_____

def myfunc(arg1, arg2):
    """
    This is a function that does nothing in particular
    """
    print("I am a function! Here are my arguments:")
    print(arg1)
    print(arg2)
    print("I am returning my first argument now!")
    return(arg1)_____no_output_____

This defines a very simple function. Let's walk through this declaration step by step:

The first line begins with _def_, then the name of the function, and then in parentheses a list of arguments for the function, then a colon. Arguments are inputs to the function. For example, the `np.sin` function takes an angle as an argument, and calculates the sine of that angle. In this example our function has two arguments. The number of arguments is arbitrary, and can be zero, in which case the parentheses are just left empty. It is also possible to write functions where the number of arguments is variable, and need not be the same every time the function is called.

After the define line, we begin the body of the function. Note that all the lines in the function body are indented. This indentation is IMPORTANT. In python, indentation is used to indicate that a particular line belongs to a particular function, loop, or other block of code. All the lines of the function are indented four spaces. If you're using entering this manually in ipython, either at the command line or in the notebook, you don't need to type in those four spaces by hand; the ipython shell will automatically enter them for you after seeing the def line. If you're using emacs or another text editor, you can just hit the tab key and the correct number of spaces will be entered for you.

Within the body of the function, we can enter whatever commands we like. We can print things, for example. Or do a calculation. The arguments in that appeared in parentheses in the definition are accessible within the function, and can be manipulated however we like.

At the end of the function, we have a statement that begins return. A return function causes the function to give back a value, which the calling program can print, assign to a variable, or do something else with. For example, the `np.log10` function returns the base 10 log for any positive number (or **array** of numbers!). Return values are optional: functions don't have to return anything, and can just end.

OK, with that in mind, let's run the `myfunc` function above, with a string and a float number as input arguments:_____no_output_____

myfunc('star', 3.14159265)_____no_output_____

Note that `myfunc` includes something called a _"docstring"_ (denoted with the triple quotations at the start and end). This is a description of what the function does and is visible when call that function with a question mark (as below). Many companies (e.g. Google) have extensive rules about what should be included in a docstring. For example, here is a sample Google docstring.

Generally speaking, docstrings should include a description of what the function does, some notes about the input and output, and specifics about any optional inputs ("keywords") and what they do. Keep your eye out for these as we proceed as we'll be asking you to include docstrings with all of the functions that you write this semester.

You can view the docstring for any function in python (built-in ones, and ones you write yourself!) using the question mark. Try it below:_____no_output_____

myfunc?_____no_output_____

Try writing a simple function below called `test_function` that takes two numbers _a_ and _b_ as input, multiplies them togther, then divides the product by 2.0, and returns the answer._____no_output_____

def test_function(a,b):
    # Your docstring goes here, in triple quotations
    # Your code goes here
    # Your return statement goes here_____no_output_____

If all went according to plan, the following should return the value 42:_____no_output_____

answer_to_life_the_universe_and_everything = test_function(28,3)

print(answer_to_life_the_universe_and_everything)_____no_output_____

### Using a function to calculate absolute magnitude

Recall that the distance modulus equation is as follows:

$M - m = -5 log10(d) + 5$_____no_output_____In the cell below, write a function called `absmagcalc` that takes in two variables (distance in parsecs and apparent magnitude), calculates the absolute magnitude, then returns the value of the absolute magnitude._____no_output_____

# Write your function here (don't forget a docstring!):
_____no_output_____

Let's test your new function with the Sun, which has an apparent magnitude of -26.74. The Sun is, on average, located at a distance of 4.848e-6 pc from Earth.

#### Question:
5) What is the absolute magnitude of the Sun? _____no_output_____

# Use your new function here:
_____no_output_____

Now that we have a handy function to calculate absolute magnitudes from apparent ones, we can add a new column for absolute magnitude to our existing dataframe. First, we'll need the approximate distances to each of the clusters, provided here._____no_output_____

# all values in parsecs
dist_m22 = 3000.0
dist_ngc188 = 1770.0
dist_m67 = 850.0
dist_m4 = 1904.5 _____no_output_____

Now we will add a new column for absolute magnitude, `Mv` to our existing M22 dataframe. Use your new `absmagcalc` function to calculate the absolute magnitudes from the distance and existing apparent V magnitude column, and provide the output for this new column below:_____no_output_____

# Edit to continue this calculation using the absmagcalc function you defined
m22['Mv'] = _____no_output_____

In the cell below, check your dataframe to see if it has been updated with the new column:_____no_output_____

# Check the dataframe again -- do you have all the columns you need?
_____no_output_____

We are now ready to plot!

### Plotting from a pandas dataframe

Using the _matplotlib.pyplot_ skills we learned last week, we can now plot our first color magnitude diagram. One convenient aspect of pandas is that we can plot columns taken directly from the dataframe itself. For example, for M22:
* the X-axis is the series: `m22['BVcolor']`
* the Y-axis is the series: `m22['Mv']`

In the following exercises, you will plot a color-magnitude diagram for M22, and then load and manipulate new pandas dataframes for two open clusters and a globular cluster. _____no_output_____# Exercise 1

Plot the V vs. B-V color-magnitude diagram for M22. Scale the plot as necessary to show all of the data clearly (there are a lot of data points, so you may want to make the symbols small using _markersize = 3_ or another size). Don't forget to add axes labels+units and a title.

Hint #1: When scaling the axes, think about how this plot is analogous to the H-R diagrams from last week. Which way should the axes go?

Hint #2: Using a subscript for the Y-axis, you can use M$_{V}$ to indicate absolute magnitude (double-click the cell to see).

(For plotting methods, you may find it useful to refer back to Homework0 and Lab1.)_____no_output_____

# Plot your data directly from the m22 dataframe below:
_____no_output_____

# Exercise 2 (Additional Clusters)

Now you will read in the data for the other clusters you downloaded from VizieR at the beginning of the lab and plot these data. For each cluster:
1. Comment the datafile as needed
2. Load the datafile using pandas (pd.read_csv)
3. Calculate the B-V color
4. Use your absolute magnitude function to calculate M$_{V}$ from the V mag and the distance to each cluster
5. Make plots! (multipanel and overlay)_____no_output_____

# Load datafiles into new pandas dataframes. Be sure to check the datatypes -- if necessary, use pd.to_numeric.

_____no_output_____# Calculate B-V colors for NGC 188 and M67.

# Freebie! For M4, the B-V values are already provided in the table -- no need to calculate this one.

_____no_output_____# Calculate absolute V magnitudes (recall that the distances to each cluster are given earlier in the lab)

_____no_output_____# Make a multipanel plot showing each of the four clusters.

# In each of the panel titles, put the name of the cluster and its type (e.g., "M22 - globular"):

fig,((ax1,ax2),(ax3,ax4)) = plt.subplots(2, 2, figsize=(10,10))
fig.suptitle('Globular and Open Cluster Comparison')
# Continue the rest of the plotting below
_____no_output_____# Make an overlay plot showing all four clusters on the same axes.

# Hint: You may want to try plotting the datasets in various orders to make sure all datasets are visible.

_____no_output_____

# Exercise 3 (Comprehension Questions)

1) How do the color-magnitude diagrams (CMDs) you have plotted compare to the H-R diagrams from last week? What features are similar? What features are different?

-

2) Why do you think there is so much scatter in the observational data? Do you think this is an astrophysical effect or an instrumental one? What are some potential sources of error?

-

3) Which clusters do you think are older or younger? Rank NGC 188, M4, M22, and M67 by relative age. How can you tell?

-

4) Why might the main sequences be offset for each of the clusters? (Hint: How would uncertainty in a measured/estimated value shift values up/down on the Y-axis?)

-

5) Bonus question, if there's time: Earlier, we also calculated a V-I color column for M22. Plot the V vs. V-I CMD for M22. Why do you think the plot looks different from the V vs. B-V one?

-_____no_output_____

# Bonus Plot_____no_output_____


# Notebook from stared/piterpy-matrix
Path: slides.ipynb

# Matrices and tensors - named, visualized

* Piotr Migdał
    * founder at [Quantum Flytrap](https://quantumgame.io/) / AI researcher at ECC Games
    * [p.migdal.pl](https://p.migdal.pl/), [@pmigdal](https://twitter.com/pmigdal), [github.com/stared](https://github.com/stared)
* [PiterPy Online, 3-6 Aug 2020](https://piterpy.com/en)
    * [talk link](https://piterpy.com/en/materials/2831#2831)
    * [talk source code](https://github.com/stared/piterpy-matrix)_____no_output_____## Hi! / Привет!

_____no_output_____## Abstract

In data science, we often work with numeric arrays: for signals, images, accounting data - input, output, everything. When we use Pandas (vs raw NumPy) we have named dimensions (rows and columns) plus an easy way to plot the numerical values.
In this talk, I will show how to go beyond that - how to name dimensions in 3 and more dimensional arrays for deep learning (with Named Tensors in PyTorch), how to visualize advanced operations in a simple way (using tensor diagrams) and how to plot complex numbers (for quantum computing).
I will give examples from two open-source projects I develop: https://github.com/Quantum-Game/bra-ket-vue and https://github.com/stared/pytorch-named-dims.

## Bio

Piotr Migdal

A deep learning specialist with a Ph.D. in quantum physics (from ICFO, Castelldefels).
He works on AI for content design and physics engine optimization in ECC Games, and develops Quantum Game with Photons - an open-source in-browser game with real quantum mechanics.
Piotr enjoys explaining complicated things in simple ways, ideally with interactive data visualizations. He develops livelossplot - a Python package for visualizing the training process in Jupyter Notebook.

## PiterPy 2018

[Learning neural networks within Jupyter Notebook](https://github.com/stared/keras-interactively-piterpy2018)_____no_output_____## What is the matrix?_____no_output_____> *“The Matrix is everywhere. It's all around us,”* he explains to Neo.

_____no_output_____**In plain English?** Well, it is a table with numbers._____no_output_____## Outline

(warning: spoilers)

* Pure Python -> NumPy -> Pandas
* Some cool visualizations
* Descent into deep learning
* A pinch of quantum
* Applause (or awkward silence)_____no_output_____

%%html

One (of many) amusing socially awkward remote conference call interactions is when the speaker makes a joke. Everyone is mutated so it seems like it awkwardly falls flat, and noone wants to unmute just to say "haha". Calls still need many more features to bridge the real life gap
— Andrej Karpathy (@karpathy) May 15, 2020
 _____no_output_____

I hope that you learn a tool or two, and how to approach _____no_output_____## Pure Python

Does Python have support for matrices?_____no_output_____

lol = [[1., 3., -1., 0.5, 2., 0.],
       [4., 0., 2.5, -4., 2., 1.],
       [1., 1., -1., 2, 2., 0.]]_____no_output_____

`lol` stands for a list of lists. Nothing to laugh about._____no_output_____

# entry
lol[1][3]_____no_output_____# row
lol[2]_____no_output_____# column
[row[2] for row in lol]_____no_output_____

### So

* You can use it
* But you shouldn't_____no_output_____### Because

* Slow
* No easy way for typical operations
* No checks on numeric types
* Or even if it is a table or not_____no_output_____## NumPy

Or **the** numerics backbone for Python._____no_output_____

import numpy as np

arr = np.array(lol)
arr_____no_output_____# entry
arr[1, 3]_____no_output_____# row
arr[1]_____no_output_____# columns
arr[:, 1]_____no_output_____# operations
arr[2] - arr[0]_____no_output_____# type
arr.dtype_____no_output_____

### Good parts

* Fast (don't ever believe "but C++")
* A lot of numeric features (using `scipy`)_____no_output_____### But

* What the hell the columns mean?!_____no_output_____## Pandas

(Note to oneself: resist any **pand**emia puns!)_____no_output_____

import pandas as pd

df = pd.DataFrame(arr,
                  index=['Sasha', 'Alex', 'Kim'],
                  columns=['sweet', 'sour', 'salty', 'bitter', 'spicy', 'sugar'])
df _____no_output_____df.loc['Alex'].mean()_____no_output_____df.loc['Alex', 'sweet']_____no_output_____df.loc[:, 'sweet']_____no_output_____df.sweet_____no_output_____# change to
df.sweet - df.bitter_____no_output_____

### Do we need anything more?_____no_output_____#### Hint

> You think you own whatever data you loaded
The Matrix is just a dead thing you can claim
But I know every frame and row and column
Has a type, has an API, has a name  _____no_output_____#### Answer

**Colors** (of the wind)!_____no_output_____## Seaborn_____no_output_____

import seaborn as sns

sns.heatmap(df, cmap='coolwarm');_____no_output_____sns.heatmap(df, cmap='coolwarm', annot=True);_____no_output_____sns.clustermap(df, cmap='coolwarm', annot=True);_____no_output_____

## Or even interactively

Using [clustergrammer](https://clustergrammer.readthedocs.io/) - interactive hierarchically clustered heatmaps, mostly for biology._____no_output_____

from clustergrammer2 import Network, CGM2

net = Network(CGM2)
net.load_df(df)_____no_output_____net.widget()_____no_output_____

## So, we are done?

_____no_output_____

(from [Deep Learning with PyTorch](https://www.manning.com/books/deep-learning-with-pytorch))_____no_output_____### OK, but do we actually need it?

(except for some super-advanced physics)_____no_output_____Yes, for colors! RGB(A) channels.

* width, height, channel_____no_output_____In deep learning, we process a few images at the same time:

* sample, height, width, channel_____no_output_____* sample, channel, height, width
* sample, channel, width, height
* ???_____no_output_____## So, let's memorize that!

Well, for sure we can learn the order of dimensions. Practice makes perfect!_____no_output_____Then see it:

* OpenCV: `HWC` / `HW`
* Pillow:  `HWC` / `HW`
* Matplotlib: `HWC` / `HW`
* Theano: `NCHW`
* TensorFlow: `NHWC`
* PyTorch: `NCHW`

Is it only me, or does the **Theano tensor dimension order** sounds like some secret convent?

In PyTorch there are [Inconsistent dimension ordering for 1D networks - NCL vs NLC vs LNC](https://discuss.pytorch.org/t/inconsistent-dimension-ordering-for-1d-networks-ncl-vs-nlc-vs-lnc/14807)._____no_output_____## Tensors in PyTorch

Full disclaimer: I love PyTorch._____no_output_____

import torch

x = torch.tensor([[
    [[1., 0., 1.],
     [0., 1., 0.],
     [1., 0., 1.]],
    [[0., 1., 0.],
     [1., 1., 1.],
     [0., 1., 0.]],
    [[1., 1., 1.],
     [1., 0., 1.],
     [1., 1., 1.]],
]])_____no_output_____x_____no_output_____x.dtype_____no_output_____x.size()_____no_output_____x.mean(1)_____no_output_____

## Named tensors

Starting from `PyTorch 1.4.0`, there is [support for dimension names in tensors](https://pytorch.org/docs/stable/named_tensor.html).

However, still at `PyTorch 1.6.0` it is an experimental feature._____no_output_____

x = x.rename('N', 'C', 'H', 'W')
x_____no_output_____x.mean('C')_____no_output_____x.mean(('H', 'W'))_____no_output_____x.transpose('H', 'W')_____no_output_____x.flatten(['C', 'H', 'W'], 'features')_____no_output_____m1 = torch.tensor([[1., 2.], [3., 4.]], names=('H', 'W'))
m2 = torch.tensor([[0, 1.], [1., 0.]], names=('W', 'H'))_____no_output_____m1 + m2_____no_output_____# We can fix that!
m1 + m2.transpose('W', 'H')_____no_output_____

## Awesome! Let's use the names in neural networks!_____no_output_____

from torch import nn_____no_output_____x_input =  torch.rand((4, 4), names=('N', 'C'))
x_input_____no_output_____# a dense (fully-connected) layer
fc = nn.Linear(in_features=4, out_features=2)
fc(x_input)_____no_output_____

It only sort of works. And it gets worse._____no_output_____

x_input_2d =  torch.rand((4, 3, 4, 4), names=('N', 'C', 'H', 'W'))
conv = nn.Conv2d(in_channels=3, out_channels=2, kernel_size=3)
conv(x_input_2d)_____no_output_____

> RuntimeError: aten::mkldnn_convolution is not yet supported with named tensors. Please drop names via `tensor = tensor.rename(None)`, call the op with an unnamed tensor, and set names on the result of the operation.

#### My first response

😞

#### My second response

What, there is an instruction how to solve it!_____no_output_____## Named neural networks

A very experimental package: [github.com/stared/pytorch-named-dims](https://github.com/stared/pytorch-named-dims) by Bartłomiej Olechno and me, from ECC Games.

Inspired by:

* [Quantum Tensors JS](https://github.com/Quantum-Game/quantum-tensors) by Piotr Migdał
* [Tensor Considered Harmful](http://nlp.seas.harvard.edu/NamedTensor) by Alexander Rush

Not yet on PyPI, but you can install directly from the repo:

`pip install git+git://github.com/stared/pytorch-named-dims.git`_____no_output_____

import torch
from torch import nn
from pytorch_named_dims import nm

convs = nn.Sequential(
    nm.Conv2d(3, 5, kernel_size=3, padding=1),
    nn.ReLU(),  # preserves dims on its own
    nm.MaxPool2d(2, 2),
    nm.Conv2d(5, 2, kernel_size=3, padding=1)
)_____no_output_____x_input = torch.rand((2, 3, 2, 2), names=('N', 'C', 'H', 'W'))
convs(x_input)_____no_output_____x_input = torch.rand((2, 3, 2, 2), names=('N', 'C', 'W', 'H'))
convs(x_input)_____no_output_____

Still, it is as permissive and accepting as Python! 🐍❤_____no_output_____

x_input = torch.rand((2, 3, 2, 2), names=('N', 'C', None, None))
convs(x_input)_____no_output_____

## Quantum mechanics

Deep learning (so called "AI") is difficult?

Let's try quantum mechanics!

It is not thaaat bad - see [Quantum mechanics for high-school students](https://p.migdal.pl/2016/08/15/quantum-mechanics-for-high-school-students.html) and play the [Quantum Game with Photons](https://quantumgame.io/)._____no_output_____In quantum mechanics, we

* use complex numbers,
* there are as many dimensions as particles,
* for operations - twice as much._____no_output_____

# BTW: complex numbers are even in plain Python
(1 - 2j) * (3 + 1j)_____no_output_____

## QuTiP

[QuTiP: Quantum Toolbox in Python](http://qutip.org/), perhaps the easiest way to approach quantum mechanics with Python.

A few examples of [Qubism visualizations](https://nbviewer.jupyter.org/github/qutip/qutip-notebooks/blob/master/examples/qubism-and-schmidt-plots.ipynb) to show quantum states, based on **Qubism: self-similar visualization of many-body wavefunctions**, New J. Phys. 14 053028 (2012)._____no_output_____

from qutip import ket, plot_schmidt
singlet = (ket('01') - ket('10')).unit()
plot_schmidt(singlet, figsize=(3, 3));_____no_output_____separable = (ket('01') - ket('00')).unit()
plot_schmidt(separable, figsize=(3, 3));_____no_output_____ghz4 = (ket('0000') + ket('1111')).unit()
plot_schmidt(ghz4, figsize=(4, 4));_____no_output_____state = (ket('0101') + ket('1011')).unit()
plot_schmidt(state, figsize=(4, 4));_____no_output_____

##  Bra Ket Vue  `⟨𝜑|𝜓⟩.vue`

[Bra Ket Vue](https://github.com/Quantum-Game/bra-ket-vue) a Vue-based visualization of quantum states and operations.

A very new one - we released that with Klem Jankiewicz this May!_____no_output_____This slide is intentionally empty.

Tried to use JavaScript code in Jupyter, but I failed (again)._____no_output_____Instead: I will use [JSFiddle](https://jsfiddle.net/user/fiddles/all/)._____no_output_____

%%html
_____no_output_____%%html
_____no_output_____%%html
_____no_output_____%%html
_____no_output_____%%html
_____no_output_____

## Thanks!  Спасибо!

[p.migdal.pl](https://p.migdal.pl/), [@pmigdal](https://twitter.com/pmigdal), [github.com/stared](https://github.com/stared)

* training neural networks? use [livelossplot](https://github.com/stared/livelossplot/) for interactive charts (over 170k downloads!)
* tensors in JavaScript and their vis: [github.com/Quantum-Game](https://github.com/Quantum-Game)
* from Zen of Python and Jupyter Notebook to TypeScript and tests: [How I Learned to Stop Worrying and Love the Types & Tests](https://p.migdal.pl/2020/03/02/types-tests-typescript.html)
* [Bra Ket Vue in Distill RMarkdown](https://p.migdal.pl/bra-ket-vue-art/) (RMarkdown is like Jupyter but better)
* interactive slides created with [Jupyter Notebook & RISE](https://github.com/damianavila/RISE)_____no_output_____


# Notebook from greywormz/Astrophysics
Path: Interactives/SmallAngleEquation.ipynb

## Small Angle Approximation Interactive

This interactive can be used to explore the relationship between an object's size, its distance, and its observed angular size.  When an object is far away compared to its size, astronomers use the *small angle approximation* to simplify the relationship.

There are two control sliders: the first for the size of the object (s) and the second for the distance from Earth to the object (d). The interactive uses both the "exact" equation and small angle approximation to estimate the angular size of the object:

$$\theta_{exact} = 2\arctan\left(\frac{s}{2d}\right) \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \theta_{approx} = \frac{s}{d}$$

**Note**: The above espressions give the angle $\theta$ in radians. To get an angle in degrees, we must multiply by the conversion factor $\frac{180^{\circ}}{\pi}$ (because there are $2\pi$ radians or $360^{\circ}$ in a circle)._____no_output_____

# Originally created on June 13, 2018 by Samuel Holen

import ipywidgets as widgets
import numpy as np
import bqplot.pyplot as bq
from IPython.display import display_____no_output_____def approx_theta(s,d):
    # Small angle approximation equation
    return s/d

def exact_theta(s,d):
    # Exact equation for theta
    return 2*np.arctan(s/(2*d))

def circle(r,d=0.):
    # Creates a circle given a radius r and displacement along the
    # x-axis d
    theta = np.linspace(0,2*np.pi,1000)
    return (r*np.cos(theta)+d,r*np.sin(theta))

def par_circ(r,theta0,theta1,d=0.,h=0.):
    theta = np.linspace(theta0,theta1,1000)
    return (r*np.cos(theta)+d,r*np.sin(theta)+h)

def ellipse(a,b,d=0.):
    # Creates an ellipse centered at (d,0) with a semimajor axis of 'a'
    # in the x direction and 'b' in the y direction
    theta = np.linspace(0,2*np.pi,1000)
    return (a*np.cos(theta)+d,b*np.sin(theta))

def update(change=None):
    # Update the display.
    D1.y = [0,h_slider.value/2]
    D2.y = [0,-h_slider.value/2]
    D1.x = [0,d_slider.value]
    D2.x = [0,d_slider.value]
    ref.x = [0,d_slider.value]
    # Note that the ellipse is used so that the display needn't be a square.
    X_new, Y_new = circle(h_slider.value/2, d_slider.value)
    Object.x = X_new
    Object.y = Y_new

    # Update the resulting angles
    theta_approx = 180/np.pi*approx_theta(h_slider.value,d_slider.value)
    theta_exact = 180/np.pi*exact_theta(h_slider.value,d_slider.value)

    approx_eqn.value='
"Small Angle"  Equation:
 {:.5f}° = (180°/π) * ({} / {})
'.format(theta_approx,h_slider.value,d_slider.value)
    exact_eqn.value ='
"Exact" Equation:
 {:.5f}° = 2 arctan( {} / 2*{} )
'.format(theta_exact,h_slider.value,d_slider.value)
    difference_eqn.value='
Difference:
 {:.5f}° ({:.2f}%)
'.format(abs(theta_exact-theta_approx), 100*(abs(theta_exact-theta_approx)/theta_exact))

    arc_loc = (d_slider.value - h_slider.value/2)/3
    angle_loc = exact_theta(h_slider.value,d_slider.value)/2
    xc,yc = par_circ(r=arc_loc, theta0=2*np.pi-angle_loc, theta1=2*np.pi+angle_loc)

    angle_ex.x = xc
    angle_ex.y = yc_____no_output_____h_slider = widgets.FloatSlider(
    value=5,
    min=0.1,
    max=30.05,
    step=0.1,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True,
    readout_format='.2f',
)
d_slider = widgets.FloatSlider(
    value=50,
    min=20.,
    max=100,
    step=0.1,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True
)

theta_approx = 180/np.pi*approx_theta(h_slider.value,d_slider.value)
theta_exact = 180/np.pi*exact_theta(h_slider.value,d_slider.value)

# Create labels for sliders
h_label = widgets.Label(value='Size of star (s)')
d_label = widgets.Label(value='Distance to star (d)')

# Create text display
approx_eqn = widgets.HTML(value='
"Small Angle" Equation:
 {:.5f}° = (180°/π) * ({} / {})
'.format(theta_approx,h_slider.value,d_slider.value))
exact_eqn = widgets.HTML(value='
"Exact" Equation:
 {:.5f}° = 2 arctan( {} / 2*{} )
'.format(theta_exact,h_slider.value,d_slider.value))
difference_eqn = widgets.HTML(value='
Difference:
 {:.5f}° ({:.2f}%)
'.format(abs(theta_exact-theta_approx), 100*(abs(theta_exact-theta_approx)/theta_exact)))

blank = widgets.Label(value='')

_____no_output_____## PLOT/FIGURE ##

# Sets axis scale for x and y to
sc_x = bq.LinearScale(min=-5,max=115)
sc_y = bq.LinearScale(min=-26,max=26)

# Get the range to work with
x_range = sc_x.max - sc_x.min
y_range = sc_y.max - sc_y.min

# Initial height and distance of star
init_h = h_slider.value
init_d = d_slider.value

# Note that the ellipse is used so that the display needn't be a square.
# Creates a circular 'star'
X,Y = circle(r=init_h/2, d=init_d)

# Sets up the axes, grid-lines are set to black so that they blend in with the background.
ax_x = bq.Axis(scale=sc_x, grid_color='white', num_ticks=0)
ax_y = bq.Axis(scale=sc_y, orientation='vertical', grid_color='white', num_ticks=0)

# Draws the lines to the top and bottom of the star respectively
D1 = bq.Lines(x=[0,init_d], y=[0,init_h/2], scales={'x': sc_x, 'y': sc_y}, colors=['white'])
D2 = bq.Lines(x=[0,init_d], y=[0,-init_h/2], scales={'x': sc_x, 'y': sc_y}, colors=['white'])

# Creates the star
Object = bq.Lines(scales={'x': sc_x, 'y': sc_y}, x=X, y=Y, colors=['blue'],
                  fill='inside', fill_colors=['blue'])

# Creates a reference line.
ref = bq.Lines(x=[0,init_d], y=[0,0], scales={'x': sc_x, 'y': sc_y}, colors=['white'], line_style='dashed')

arc_loc = (init_d - init_h/2)/3
angle_loc = exact_theta(init_h,init_d)/2
xc,yc = par_circ(r=arc_loc, theta0=2*np.pi-angle_loc, theta1=2*np.pi+angle_loc)

angle_ex = bq.Lines(x=xc, y=yc, scales={'x': sc_x, 'y': sc_y}, colors=['white'])

angle_label = bq.Label(x=[2], y=[0], scales={'x': sc_x, 'y': sc_y},
                   text=[r'$$\theta$$'], default_size=15, font_weight='bolder',
                   colors=['white'], update_on_move=False)

# Update the the plot/display
h_slider.observe(update, names=['value'])
d_slider.observe(update, names=['value'])

# Creates the figure. The background color is set to black so that it looks like 'space.' Also,
# removes the default y padding.
fig = bq.Figure(title='Small Angle Approximation', marks=[Object,D1,D2,angle_ex], axes=[ax_x, ax_y],
                padding_x=0, padding_y=0, animation=100, background_style={'fill' : 'black'})

# Display to the screen

# Set up the figure
fig_width = 750
fig.layout.width = '{:.0f}px'.format(fig_width)
fig.layout.height = '{:.0f}px'.format(fig_width/2)

# Set up the bottom part containing the controls and display of equations
h_box = widgets.VBox([h_label, h_slider])
d_box = widgets.VBox([d_label, d_slider])
slide_box = widgets.HBox([h_box, d_box])
h_box.layout.width = '{:.0f}px'.format(fig_width/2)
d_box.layout.width = '{:.0f}px'.format(fig_width/2)

eqn_box = widgets.HBox([exact_eqn, approx_eqn, difference_eqn])
exact_eqn.layout.width = '{:.0f}px'.format(fig_width/3)
approx_eqn.layout.width = '{:.0f}px'.format(fig_width/3)
difference_eqn.layout.width = '{:.0f}px'.format(fig_width/3)

BOX = widgets.VBox([fig, slide_box, eqn_box])
display(BOX)_____no_output_____


# Notebook from ctb/BioBombe
Path: 8.gtex-interpret/2.apply-signatures.ipynb

# Application of Compressed Gene Expression Signatures

**Gregory Way, 2018**

We previously identified a specific VAE feature (z = 3) that captured blood signatures.
These signatures were not captured in VAE z = 2, and, when present, contributed to a rapid increase in the ability to capture the signal in GTEX blood tissues.

The primary differences between the two VAE models appeared to be related to neutrophil and monocyte signatures. Here, we test the ability of these signatures to generalize to external datasets.

## Part 1:

### Enrichment of Neutrophil Signatures

Here, we apply the VAE feature enriched for neutrophil genes to a publicly available dataset capturing neutrophil differentiation in two leukemia cell lines.

![cell_type_Neutrophils_HPCA_2.png](https://github.com/greenelab/BioBombe/raw/master/6.biobombe-projection/figures/GTEX/signal/GpXCELL/gene_set_Neutrophils_HPCA_2.png)

## Part 2:

### Enrichment of Monocyte Signatures

Here, we apply the VAE features enriched for monocyte genes to a different publicly available dataset that captures various cell-types undergoing hematopoiesis.

![cell_type_Monocytes_FANTOM_2.png](https://github.com/greenelab/BioBombe/raw/master/6.biobombe-projection/figures/GTEX/signal/GpXCELL/gene_set_Monocytes_FANTOM_2.png)

## Output

In both cases, various scores are output that will be visualized in a separate notebook._____no_output_____

import os
import sys
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from scripts.utils import load_weight_matrix, apply_signature, load_enrichment_results

sys.path.append('../scripts')
from latent import parse_gmt_____no_output_____%matplotlib inline_____no_output_____np.random.seed(123)_____no_output_____# First, load the XCELL dataset and extract genes
geneset_file = os.path.join('..', '3.build-hetnets', 'data', 'xcell_all_entrez.gmt')

xcell_genesets_gmt = parse_gmt(gene_sets=[geneset_file])
len(xcell_genesets_gmt)_____no_output_____

## 1.0. Load and Process External Neutrophil Dataset_____no_output_____

file = os.path.join('data', 'GSE103706_processed_matrix.tsv.gz')
geo_scaled_zeroone_df = pd.read_table(file, index_col=0)

print(geo_scaled_zeroone_df.shape)
geo_scaled_zeroone_df.head(2)(14, 18606)

## 1.1. Apply Signature from VAE z = 3 (feature 0)

We are using feature 0 from VAE z = 3 because we previously observed an enrichment of a Neutrophil signature in this specific feature._____no_output_____

vae_z3_seed = 908341
vae_z3_feature = 'vae_0'_____no_output_____weight_z3_df = load_weight_matrix(dataset='GTEX',
                                  z_dim=3,
                                  seed=vae_z3_seed)

result_vae_3_feat0, neutrophil_missing_genes = (
    apply_signature(weight_df=weight_z3_df,
                    other_df=geo_scaled_zeroone_df,
                    feature=vae_z3_feature,
                    align=True)
)

use_genes = weight_z3_df.shape[0] - len(neutrophil_missing_genes)
print('{} ({}%) of genes are used'.format(use_genes, use_genes / weight_z3_df.shape[0] * 100 ))7664 (95.8%) of genes are used
neutrophil_hpca_genes = xcell_genesets_gmt['Neutrophils_HPCA_2']
neutrophil_hpca_genes_____no_output_____# But how many of the missing genes belong to the specific Neutrophil signature
neutrophil_hpca_genes_missing = [x for x in neutrophil_hpca_genes if int(x) in neutrophil_missing_genes]
neutrophil_hpca_genes_missing_____no_output_____

## 1.2. Apply Signature from VAE z = 14 (feature 10)

We are using VAE z = 14 because, for some reason, this dimension was best able to capture the `Neutrophil_HPCA_2` xCell geneset.
This geneset was the same geneset that was enriched in feature 0 for VAE z = 3.

We identified feature 10 as the feature with the greatest enrichment by visually scanning the biobombe results in `results/gtex/gpxcell/signal/gtex_z_14_GpXCELL__geneset_scores.tsv`_____no_output_____

z_14_df = load_enrichment_results(dataset="GTEX",
                                  metaedge="GpXCELL",
                                  z_dim=14)

z_14_df = (
    z_14_df
    .query("variable == 'Neutrophils_HPCA_2'")
    .assign(abs_z_score=z_14_df.z_score.abs())
    .sort_values(by='abs_z_score', ascending=False)
)

z_14_df.head(1)_____no_output_____vae_z14_seed = z_14_df.head(1).seed.values[0]
vae_z14_feature = '{}_{}'.format(z_14_df.head(1).algorithm.values[0],
                                 z_14_df.head(1).feature.values[0])

print(vae_z14_seed)
vae_z14_feature486191
weight_z14_df = load_weight_matrix(dataset='GTEX',
                                   z_dim=14,
                                   seed=vae_z14_seed)

result_vae_14_feat10, _ = apply_signature(weight_df=weight_z14_df,
                                          other_df=geo_scaled_zeroone_df,
                                          feature=vae_z14_feature,
                                          align=True)_____no_output_____

## 1.3. Combine Data and Add Phenotype Information_____no_output_____

full_neutrophil_results_df = result_vae_14_feat10.merge(result_vae_3_feat0,
                                                        left_index=True,
                                                        right_index=True)_____no_output_____# Process phenotype data
cell_line = [x[0] for x in result_vae_3_feat0.index.str.split(',')]
treatment = [x[1] for x in result_vae_3_feat0.index.str.split(',')]
day = [x[2].strip(' ') if 'replicate' not in x[2] else 'day 0'
       for x in result_vae_3_feat0.index.str.split(',')]_____no_output_____full_neutrophil_results_df = (
    full_neutrophil_results_df
    .assign(cell_line=cell_line,
            treatment=treatment,
            day=day)
    .reset_index()
    .rename(columns={'index': 'full_id'})
)

recode_labels = {' not differentiated': 'Not Differentiated',
                 ' DMSO': 'DMSO',
                 ' DMSO+Nutridoma': 'DMSO+Nutridoma'}

full_neutrophil_results_df.treatment = full_neutrophil_results_df.treatment.replace(recode_labels)

file = os.path.join('results', 'neutrophil_data_biobombe_results.tsv')
full_neutrophil_results_df.to_csv(file, index=False, sep='\t')

full_neutrophil_results_df_____no_output_____

## 1.4. Quickly Visualize Signature Applied to External Dataset

Note the final figure is compiled in an alternative notebook_____no_output_____

# Quickly visualize results
plt.rcParams['figure.figsize'] = 5, 3
ax = sns.stripplot(x='cell_line',
                   y='vae_0',
                   data=full_neutrophil_results_df,
                   hue='treatment',
                   jitter=0.1)

handles, labels = ax.get_legend_handles_labels()

l = plt.legend(handles,
               labels,
               bbox_to_anchor=(1.02, 0.8),
               loc=2,
               borderaxespad=0.)
l.set_title("Treatment")

ax.set_ylabel('VAE Feature 0 (z = 3)')
ax.set_xlabel('Cell Lines')

plt.tight_layout()_____no_output_____# Quickly visualize results
plt.rcParams['figure.figsize'] = 5, 3
ax = sns.stripplot(x='cell_line',
                   y='vae_10',
                   data=full_neutrophil_results_df,
                   hue='treatment',
                   jitter=0.1)

handles, labels = ax.get_legend_handles_labels()

l = plt.legend(handles,
               labels,
               bbox_to_anchor=(1.02, 0.8),
               loc=2,
               borderaxespad=0.)
l.set_title("Treatment")

ax.set_ylabel('VAE Feature 10 (z = 14)')
ax.set_xlabel('Cell Lines')

plt.tight_layout()_____no_output_____

## 1.5. Compare the two signatures derived from both models

Use the function `load_enrichment_results` to retrieve and subset previously compiled BioBombe results.
The files are located in `6.biobombe-projection/results/`._____no_output_____

# What other genesets are enriched in VAE z = 3 feature?
vae_z3_p0 = load_enrichment_results(dataset='GTEX',
                                    z_dim=3,
                                    metaedge='GpXCELL',
                                    algorithm='vae',
                                    feature=0,
                                    seed=vae_z3_seed,
                                    shuffled=False)

# What other genesets are enriched in VAE z = 14 feature?
vae_z14_p10 = load_enrichment_results(dataset='GTEX',
                                      z_dim=14,
                                      metaedge='GpXCELL',
                                      algorithm='vae',
                                      feature=10,
                                      seed=vae_z14_seed,
                                      shuffled=False)

full_test_df = vae_z3_p0.merge(vae_z14_p10, on='variable', suffixes=('_z3', '_z14'))

file = os.path.join('results', 'latent_feature_enrichment_comparison_neutrophil_genesets.tsv')
full_test_df.to_csv(file, index=False, sep='\t')

full_test_df.head(3)_____no_output_____# Save dataframe for better plotting in R, but visualize quickly here
sns.scatterplot(data=full_test_df, x='z_score_z3', y='z_score_z14');_____no_output_____

### 1.5.1 Determine Gene Weights across the Two Features

Also assign labels to which genesets the genes contribute to_____no_output_____

unique_genes = []
for geneset_name, geneset in xcell_genesets_gmt.items():
    for gene in geneset:
        unique_genes.append(gene)

unique_genes = set(unique_genes)

classification_genes = []
for geneset_name, geneset in xcell_genesets_gmt.items():
    if 'neutrophil' in geneset_name.lower():
        classification = 'Neutrophils'
    elif 'keratinocytes' in geneset_name.lower():
        classification = 'Keratinocytes'
    elif 'neurons' in geneset_name.lower():
        classification = 'Neurons'
    elif 'skeletal' in geneset_name.lower():
        classification = 'Skeletal Muscle'
    elif 'monocytes' in geneset_name.lower():
        classification = 'Monocytes'
    else:
        classification = 'Other Geneset'
    for gene in geneset:
        if gene in weight_z14_df.index:
            classification_genes.append([classification, gene, geneset_name])_____no_output_____result_df = (
    pd.DataFrame(classification_genes, columns=['classification', 'gene', 'gene_set'])
    .sort_values(by='classification')
    .reset_index(drop=True)
    .drop_duplicates(subset='gene', keep='first')
)

result_df.gene = result_df.gene.astype(str)
result_df.index = result_df.gene
result_df.head()_____no_output_____result_df.classification.value_counts()_____no_output_____both_weight_df = (
    weight_z3_df.merge(weight_z14_df,
                       left_index=True,
                       right_index=True,
                       suffixes=('_3', '_14'))
    .merge(result_df, left_index=True,
           right_index=True, how='left')
    .fillna('No Geneset')
)

file = os.path.join('results', 'latent_feature_enrichment_comparison_neutrophil_genes.tsv')
both_weight_df.to_csv(file, index=False, sep='\t')_____no_output_____

## 2.0. Load and Process External Hematopoietic Dataset_____no_output_____

file = os.path.join('data', 'GSE24759_processed_matrix.tsv.gz')
heme_zeroone_df = pd.read_table(file, index_col=0)

print(heme_zeroone_df.shape)
heme_zeroone_df.head(2)(211, 8190)
heme_z3_seed = 908341
heme_z3_feature = 'vae_2'_____no_output_____# Transform the external dataset with this learned feature
weight_heme_z3_df = load_weight_matrix(dataset='GTEX',
                                       z_dim=3,
                                       seed=heme_z3_seed)

result_heme_vae_3_feat2, monocyte_missing_genes = (
    apply_signature(weight_df=weight_heme_z3_df,
                    other_df=heme_zeroone_df,
                    feature=heme_z3_feature,
                    align=True)
)

use_genes = weight_heme_z3_df.shape[0] - len(monocyte_missing_genes)
print('{} ({:.2f}%) of genes are used'.format(use_genes, use_genes / weight_heme_z3_df.shape[0] * 100 ))4645 (58.06%) of genes are used
monocyte_fantom_genes = xcell_genesets_gmt['Monocytes_FANTOM_2']
print(len(monocyte_fantom_genes))
monocyte_fantom_genes178
# But how many of the missing genes belong to the specific Monocyte signature
monocyte_fantom_genes_missing = [x for x in monocyte_fantom_genes if int(x) in monocyte_missing_genes]
print(len(monocyte_fantom_genes_missing))
monocyte_fantom_genes_missing10
print('{:.2f}% of monocyte genes are missing'.format(len(monocyte_fantom_genes_missing) / len(monocyte_fantom_genes) * 100 ))5.62% of monocyte genes are missing
# Additionall, the top scoring feature for Monocytes_FANTOM_2 is in the nmf model with 200 features
file = os.path.join('..', '6.biobombe-projection', 'results', 'gtex',
                    'gpxcell', 'signal',
                    'gtex_z_200_GpXCELL__geneset_scores.tsv.gz')

gtex_z200_scores_df = (
    pd.read_table(file)
    .query('variable == "Monocytes_FANTOM_2"'))

gtex_z200_scores_df = (
    gtex_z200_scores_df
    .assign(abs_z_score = gtex_z200_scores_df.z_score.abs())
    .sort_values(by='abs_z_score', ascending=False)
    .head(1)
)

gtex_z200_scores_df_____no_output_____heme_z200_feature = '{}_{}'.format(gtex_z200_scores_df.algorithm.values[0],
                                   gtex_z200_scores_df.feature.values[0])

heme_z200_feature_____no_output_____# Obtain this transformation too
weight_heme_z200_df = load_weight_matrix(dataset='GTEX',
                                         z_dim=200,
                                         seed=gtex_z200_scores_df.seed.values[0])

result_heme_nmf_200_feat6, _ = apply_signature(weight_df=weight_heme_z200_df,
                                            other_df=heme_zeroone_df,
                                            feature=heme_z200_feature,
                                            align=True)_____no_output_____# Combine the full scores and output for downstream visualization
full_heme_result_df = (
    result_heme_vae_3_feat2
    .merge(result_heme_nmf_200_feat6, left_index=True, right_index=True)
    .reset_index().rename(columns={'index': 'cell'})
)_____no_output_____heme_cell_type_recode_df = (
    pd.DataFrame(full_heme_result_df.cell.str.split('_').values.tolist(),
                 columns = ['cell_type', 'replicate', 'additional'])
)

heme_cell_type_recode_df.loc[~heme_cell_type_recode_df.additional.isna(), 'cell_type'] = "PRE_BCELL2"_____no_output_____full_heme_result_df = (
    pd.concat([heme_cell_type_recode_df.drop(['additional'], axis='columns'),
               full_heme_result_df], axis='columns')
)_____no_output_____# Recode cell-type into larger classification
file = os.path.join('results', 'cell-type-classification.tsv')
cell_class_df = pd.read_table(file)

cell_updater = dict(zip(cell_class_df.label, cell_class_df.classification))

cell_class_df.head()_____no_output_____full_heme_result_df = (
    full_heme_result_df
    .assign(cell_class = full_heme_result_df.cell_type.replace(cell_updater))
)

file = os.path.join('results', 'hematopoietic_data_biobombe_results.tsv')
full_heme_result_df.to_csv(file, index=False, sep='\t')

full_heme_result_df.head()_____no_output_____# Quickly plot results for both features
plt.rcParams['figure.figsize'] = 5, 3
ax = sns.stripplot(y='vae_2', x = 'cell_class', data = full_heme_result_df, hue = 'cell_class', jitter=0.1)

handles, labels = ax.get_legend_handles_labels()

l = plt.legend(handles,
               labels,
               bbox_to_anchor=(1.02, 0.8), loc=2, borderaxespad=0.)
_____no_output_____plt.rcParams['figure.figsize'] = 5, 3
ax = sns.stripplot(y='nmf_6', x = 'cell_class', data = full_heme_result_df, hue = 'cell_class', jitter=0.1)

handles, labels = ax.get_legend_handles_labels()

l = plt.legend(handles,
               labels,
               bbox_to_anchor=(1.02, 0.8), loc=2, borderaxespad=0.)_____no_output_____

## Generate Supplementary Table of Neutrophil and Monocyte Signatures_____no_output_____

# Load curated gene names from versioned resource
commit = '721204091a96e55de6dcad165d6d8265e67e2a48'
url = 'https://raw.githubusercontent.com/cognoma/genes/{}/data/genes.tsv'.format(commit)
gene_df = pd.read_table(url)

# Only consider protein-coding genes
gene_df = (
    gene_df.query("gene_type == 'protein-coding'")
)

entrez_to_symbol = dict(zip(gene_df.entrez_gene_id,
                            gene_df.symbol))_____no_output_____neutrophil_df = pd.DataFrame(neutrophil_hpca_genes, columns=['entrez_gene_id'])
neutrophil_df = (
    neutrophil_df.assign(
        gene_symbol=neutrophil_df.entrez_gene_id.astype(int).replace(entrez_to_symbol),
        signature='Neutrophil_HPCA_2',
        in_external_dataset='Yes')
)
neutrophil_df.head()_____no_output_____monocyte_df = pd.DataFrame(monocyte_fantom_genes, columns=['entrez_gene_id'])
monocyte_df = (
    monocyte_df.assign(
        gene_symbol=monocyte_df.entrez_gene_id.astype(int).replace(entrez_to_symbol),
        signature='Monocyte_FANTOM_2',
        in_external_dataset='Yes')
)

monocyte_df.loc[monocyte_df.entrez_gene_id.isin(monocyte_fantom_genes_missing), 'in_external_dataset'] = "No"
monocyte_df.head()_____no_output_____sup_table_df = pd.concat([neutrophil_df, monocyte_df], axis='rows').reset_index(drop=True)

file = os.path.join("results", "neutrophil_and_monocyte_signature_genes.tsv")
sup_table_df.to_csv(file, sep='\t', index=False)_____no_output_____


# Notebook from huoww07/StackOverflowSurveyAnalysis
Path: 2019_survey_DS_analysis_version 2.ipynb

# Analysis of  2019 Stack Overflow public developer survey

## 1. Data preparation: Import necessary libraries_____no_output_____

# import necessary libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)_____no_output_____

## 1. Data preparation: Read in the data and select the columns of interest for this study to reduce the memory usage

The survey covered many different aspects, in this study, I want to focus on education levels, occupation, job satisfaction and work environment related topics. Hence, a subset of the data is sliced and used for furthur analysis.
_____no_output_____

# read in the survey data
df = pd.read_csv('./survey_results_public.csv')_____no_output_____# Get the smaller data table that contains columns of interest
df_small = df[['Employment', 'EdLevel', 'UndergradMajor', 'EduOther',
               'DevType', 'YearsCode', 'Age1stCode', 'CareerSat', 'JobSat',
               'ConvertedComp', 'WorkWeekHrs', 'WorkChallenge', 'WorkRemote']]_____no_output_____

## 1. Data preparation: Get some insights about the data_____no_output_____

# Number of NaN values and their percentages
pd.concat([df_small.isnull().sum(), df_small.isnull().sum()*100/len(df_small)],
          axis=1, keys=['count', 'percentage'])_____no_output_____# What about the general descriptive summary for numerical columns?
df_small.describe()_____no_output_____# for categorical columns, what are the total number of categories and the distribution?
column_list = list(df_small.columns)
column_list.remove('ConvertedComp')
column_list.remove('WorkWeekHrs')
for column in column_list:
    print('-----------------------------------------------------------------------------------------------')
    total_num = len(df_small[column].value_counts())
    print('the column %s has %i categoris' % (column, total_num))
    print(df_small[column].value_counts().sort_values(ascending=False).head(5))
    -----------------------------------------------------------------------------------------------
the column Employment has 6 categoris
Employed full-time                                      64440
Independent contractor, freelancer, or self-employed     8511
Not employed, but looking for work                       5600
Employed part-time                                       4469
Not employed, and not looking for work                   3803
Name: Employment, dtype: int64
-----------------------------------------------------------------------------------------------
the column EdLevel has 9 categoris
Bachelor’s degree (BA, BS, B.Eng., etc.)                                              39134
Master’s degree (MA, MS, M.Eng., MBA, etc.)                                           19569
Some college/university study without earning a degree                                10502
Secondary school (e.g. American high school, German Realschule or Gymnasium, etc.)     8642
Associate degree                                                                       2938
Name: EdLevel, dtype: int64
-----------------------------------------------------------------------------------------------
the column UndergradMajor has 12 categoris
Computer science, computer engineering, or software engineering          47214
Another engineering discipline (ex. civil, electrical, mechanical)        6222
Information systems, information technology, or system administration     5253
Web development or web design                                             3422
A natural science (ex. biology, chemistry, physics)                       3232
Name: UndergradMajor, dtype: int64
-----------------------------------------------------------------------------------------------
the column EduOther has 492 categoris
Taught yourself a new language, framework, or tool without taking a formal course                                                                                6699
Taken an online course in programming or software development (e.g. a MOOC);Taught yourself a new language, framework, or tool without taking a formal course                                                         6012
Taught yourself a new language, framework, or tool without taking a formal course;Contributed to open source software                                                                                3810
Taken an online course in programming or software development (e.g. a MOOC);Received on-the-job training in software development;Taught yourself a new language, framework, or tool without taking a formal course    3104
Taken an online course in programming or software development (e.g. a MOOC)                                                                                2820
Name: EduOther, dtype: int64
-----------------------------------------------------------------------------------------------
the column DevType has 13141 categoris
Developer, full-stack                                             8433
Developer, back-end                                               4913
Developer, back-end;Developer, front-end;Developer, full-stack    2399
Developer, front-end                                              2212
Developer, mobile                                                 2172
Name: DevType, dtype: int64
-----------------------------------------------------------------------------------------------
the column YearsCode has 52 categoris
5     7047
10    6777
6     6179
4     5729
8     5361
Name: YearsCode, dtype: int64
-----------------------------------------------------------------------------------------------
the column Age1stCode has 69 categoris
15    8681
16    8633
14    8452
18    7945
12    7735
Name: Age1stCode, dtype: int64
-----------------------------------------------------------------------------------------------
the column CareerSat has 5 categoris
Very satisfied                        29173
Slightly satisfied                    25018
Slightly dissatisfied                  7670
Neither satisfied nor dissatisfied     7252
Very dissatisfied                      3734
Name: CareerSat, dtype: int64
-----------------------------------------------------------------------------------------------
the column JobSat has 5 categoris
Slightly satisfied                    24207
Very satisfied                        22452
Slightly dissatisfied                 10752
Neither satisfied nor dissatisfied     8720
Very dissatisfied                      4857
Name: JobSat, dtype: int64
-----------------------------------------------------------------------------------------------
the column WorkChallenge has 129 categoris
Being tasked with non-development work;Distracting work environment;Meetings                              2318
Being tasked with non-development work;Meetings;Not enough people for the workload                        1741
Being tasked with non-development work;Distracting work environment;Not enough people for the workload    1511
Distracting work environment;Meetings;Not enough people for the workload                                  1472
Not enough people for the workload                                                                        1420
Name: WorkChallenge, dtype: int64
-----------------------------------------------------------------------------------------------
the column WorkRemote has 7 categoris
Less than once per month / Never                           30220
A few days each month                                      17242
All or almost all the time (I'm full-time remote)           8465
Less than half the time, but at least one day each week     6320
It's complicated                                            3675
Name: WorkRemote, dtype: int64
# let's look at Employment distribution
df_small['Employment'].value_counts().sort_values(ascending=False)_____no_output_____# Focus on the full-time employees
full_time = df_small[df_small['Employment'] == 'Employed full-time']_____no_output_____# let's look at the devtype distribution
print(full_time['DevType'].value_counts().sort_values(ascending=False).sum())
print(full_time['DevType'].value_counts().sort_values(ascending=False).head(10))63192
Developer, full-stack                                                                                7387
Developer, back-end                                                                                4467
Developer, back-end;Developer, front-end;Developer, full-stack                                                  2050
Developer, front-end                                                                                1920
Developer, mobile                                                                                1907
Developer, back-end;Developer, full-stack                                                                       1830
Developer, front-end;Developer, full-stack                                                                       922
Developer, desktop or enterprise applications                                                                    880
Developer, back-end;Developer, desktop or enterprise applications;Developer, front-end;Developer, full-stack     784
Developer, back-end;Developer, desktop or enterprise applications                                                759
Name: DevType, dtype: int64
# it seems that the reported DevType from each respondent tend to be a combination of multiple things
# Here I will define a function to clean the combinations so that it shows the distribution of each category
def clean_combination(data, ascending=False):
    '''
    Input: the value_counts() table with indexes being items to be sorted
    Output: updated value counts table with each unique item as index
    Note: the deliminater is ';' in this case
    '''
    data_updated = {}
    for idx in data.index:
        if ';' in idx:
            items = idx.split(';')
            for item in items:
                if item in data_updated:
                    data_updated[item] += data.loc[idx]
                else:
                    data_updated[item] = data.loc[idx]
        else:
            if idx in data_updated:
                data_updated[idx] += data.loc[idx]
            else:
                data_updated[idx] = data.loc[idx]
    data_updated = pd.DataFrame.from_dict(data_updated, orient='index', columns=['count'])

    return data_updated.sort_values(by='count',ascending=ascending)_____no_output_____# clean up devtypes
devtype = full_time['DevType'].value_counts()
devtype = clean_combination(devtype)_____no_output_____# Plot the distribution of devtypes among all full_time employees
devtype.plot(kind='barh');_____no_output_____

## Split the full_time employees data into DS (Data scientist, data analyst, data engineer) and non DS (the rest)

Since the focus of the study is to compare those with DS jobs and those with other job types, the respondents who didn't report DevType would give less to no information here. Also the number of respondents without providing DevType accounts for less than 2% of all respondents, hence those null values are dropped using dropna and axis 0._____no_output_____

# Split the full_time employees data into DS (Data scientist, data analyst, data engineer) and non DS (the rest)
DS = pd.DataFrame()
DS_row = []
# drop na
full_time = full_time.dropna(subset=['DevType'], axis=0)
full_time['DS'] = ''
for i in range(len(full_time)):
    if 'Data or business analyst' in full_time.iloc[i,4]:
        DS_row.append(i)
        full_time.iloc[i,13] = 'DS'
    elif 'Data scientist' in full_time.iloc[i,4]:
        DS_row.append(i)
        full_time.iloc[i,13] = 'DS'
    elif 'Engineer, data' in full_time.iloc[i,4]:
        DS_row.append(i)
        full_time.iloc[i,13] = 'DS'
    else:
        full_time.iloc[i,13] = 'non DS'
        continue
DS = full_time.iloc[DS_row,:]
non_DS = full_time.drop(full_time.index[DS_row])
print('total number of full-time employed DS is ', len(DS))
print('total number of full-time employed non DS is ', len(non_DS))/Users/Wenwen/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:5: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """
/Users/Wenwen/anaconda3/lib/python3.6/site-packages/pandas/core/indexing.py:494: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self.obj[item] = s

### Save the DS and non_DS data table for easier import next time_____no_output_____

DS.to_csv('DS.csv')
non_DS.to_csv('non_DS.csv')_____no_output_____

### load the DS and non_DS from previous session_____no_output_____

DS= pd.read_csv('DS.csv',index_col =0)
non_DS = pd.read_csv('non_DS.csv', index_col = 0)
DS.head()_____no_output_____

## First question, what is the education background of all DS full-time employees?

Here, I'm interested in the education background of DS employees in the aspects of undergraduate studies, degree earnd as well as other education experiences. This shows the composition of employees in DS field._____no_output_____

# First question, what is the education background of all DS full-time employees?
# The education background has three aspect: degree earned, undergraduate major, and other education sources
degrees = DS['EdLevel'].value_counts().sort_values(ascending=False)
major = DS['UndergradMajor'].value_counts().sort_values(ascending=False)
other = DS['EduOther'].value_counts().sort_values(ascending=False)
other_clean= clean_combination(other)_____no_output_____# plot
plt.figure(figsize=(5,12))
plt.subplot(3, 1, 1)
degrees.plot(kind='barh')
plt.title('Degrees')

plt.subplot(3, 1, 2)
major.plot(kind='barh')
plt.title('Major')

plt.subplot(3, 1, 3)
other_clean['count'].plot(kind='barh')
plt.title('Other Education')

plt.show()_____no_output_____

## Second question, what is the work hours per week for DS v.s. non_DS?

When it comes to job, the work life balance is one of the big things people often talk about or pay attention to. Compared to traditional technology jobs, data scientsits are relatively new field. so what the work life balance is like when compared to other jobs?_____no_output_____

# Using violin plot to visualize the distribution of workweek hours
work_hour = pd.concat([DS[DS['WorkWeekHrs'] <= 100][['WorkWeekHrs','DS']],
                       non_DS[non_DS['WorkWeekHrs'] <=100][['WorkWeekHrs','DS']]], axis=0)

plt.figure(figsize=(12,5))
plt.subplot(1,2,1)
sns.violinplot(x='DS', y='WorkWeekHrs', data=work_hour)
plt.xlabel('Categories')

plt.subplot(1,2,2)
plt.hist(DS[DS['WorkWeekHrs'] <= 100]['WorkWeekHrs'], bins=100, density=True, label='DS',alpha=0.6)
plt.hist(non_DS[non_DS['WorkWeekHrs'] <=100]['WorkWeekHrs'], bins=100, density=True, label='non DS', alpha=0.6)
plt.legend()
plt.xlim([0,100])
plt.xlabel('Work week hours')
plt.ylabel('Percent of participants')
plt.yscale('log')

plt.show()/Users/Wenwen/anaconda3/lib/python3.6/site-packages/seaborn/categorical.py:598: FutureWarning: remove_na is deprecated and is a private function. Do not use.
  kde_data = remove_na(group_data)
/Users/Wenwen/anaconda3/lib/python3.6/site-packages/seaborn/categorical.py:826: FutureWarning: remove_na is deprecated and is a private function. Do not use.
  violin_data = remove_na(group_data)

However, we should keep in mind that the percentage of people not reporting work week hours is quite high (10%)._____no_output_____

# Number of NaN values and their percentages
pd.concat([DS.isnull().sum(), DS.isnull().sum()*100/len(DS),
          non_DS.isnull().sum(), non_DS.isnull().sum()*100/len(non_DS)],
          axis=1, keys=['DS count', 'DS percentage', 'non DS count', 'non DS percentage'])_____no_output_____

## Third question, What challenges will you face as a DS employee?

To have another peek into the field of DS is through what people are complaining about it. Here, I wanted to take a look at work challenges that DS employees complain about their work environment. From this, DS candidates can be more prepared for the potential job challenge._____no_output_____

satisfaction_combine = pd.concat([DS['JobSat'].value_counts()/len(DS),
                                 non_DS['JobSat'].value_counts()/len(non_DS)]
                                 ,axis=1,keys=['DS', 'non DS'])_____no_output_____satisfaction_combine.plot(kind='barh')
plt.show()_____no_output_____DS_workchallenge = clean_combination(DS['WorkChallenge'].value_counts())/np.sum(clean_combination(DS['WorkChallenge'].value_counts()))
non_DS_workchallenge = clean_combination(non_DS['WorkChallenge'].value_counts())/np.sum(clean_combination(non_DS['WorkChallenge'].value_counts()))
work_challenge = pd.concat([DS_workchallenge, non_DS_workchallenge], axis=1, keys=['DS', 'non DS'])/Users/Wenwen/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:3: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass 'sort=False'.

To retain the current behavior and silence the warning, pass 'sort=True'.

  This is separate from the ipykernel package so we can avoid doing imports until
work_challenge.plot(kind='barh')
plt.show()_____no_output_____