# Notebook from chengsoonong/mclass-sky
Path: projects/alasdair/notebooks/01_dataset_prepration.ipynb

# Dataset Prepration_____no_output_____

import numpy as np
import pandas as pd
from astropy.io import fits
from sklearn.preprocessing import StandardScaler
from mclearn.preprocessing import csv_to_hdf
from mclearn.photometry import (reddening_correction_w14,
                                correct_magnitudes,
                                compute_colours)
from mclearn.tools import save_results, results_exist_____no_output_____

## SDSS Dataset_____no_output_____### Getting the Labelled SDSS Dataset_____no_output_____The Sloan Digital Sky Survey (SDSS) is a comprehensive survey of the northern sky. We are interested in a subset of this survey, namely the photometric measurements and spectroscopic labels of around 2.8 million objects. Given the size of the dataset (385MB), we will need to manually query the SDSS server. To do this, create an account on the SDSS CasJobs site and submit the following SQL query to the DR12 catalog:_____no_output_____        SELECT
        -- right ascension and declination in degrees
        p.ra, p.dec,

        -- class of object, expert opinion (galaxy, star, or quasar)
        CASE s.class WHEN 'GALAXY' THEN 'Galaxy'
                     WHEN 'STAR' THEN 'Star'
                     WHEN 'QSO' THEN 'Quasar'
                     END AS class,

        -- subclass of object
        s.subclass,

        -- redshift of object from spectrum with error, expert opnion
        s.z AS redshift,
        s.zErr AS redshiftErr,

        -- 0 if spectrum is ok
        s.zWarning,

        -- PSF magnitude measurements in 5 bands (ugriz) with error, assuming object is a point souce
        p.psfMag_u, p.psfMagErr_u,
        p.psfMag_g, p.psfMagErr_g,
        p.psfMag_r, p.psfMagErr_r,
        p.psfMag_i, p.psfMagErr_i,
        p.psfMag_z, p.psfMagErr_z,

        -- Petrosian magnitude measurements in 5 bands (ugriz) with error, assuming object is an extended souce
        p.petroMag_u, p.petroMagErr_u,
        p.petroMag_g, p.petroMagErr_g,
        p.petroMag_r, p.petroMagErr_r,
        p.petroMag_i, p.petroMagErr_i,
        p.petroMag_z, p.petroMagErr_z,

        -- extinction values
        p.extinction_u, p.extinction_g, p.extinction_r, p.extinction_i, p.extinction_z,

        -- size measurement in r-band in arc seconds
        p.petroRad_r, p.petroRadErr_r

    FROM PhotoObj AS p
       JOIN SpecObj AS s
       ON s.bestobjid = p.objid

    WHERE
        -- only include objects with complete and reasonably accurate data
        p.psfMagErr_u BETWEEN 0 AND 3
        AND p.psfMagErr_g BETWEEN 0 AND 3
        AND p.psfMagErr_r BETWEEN 0 AND 3
        AND p.psfMagErr_i BETWEEN 0 AND 3
        AND p.psfMagErr_z BETWEEN 0 AND 3
        AND p.petroMagErr_u BETWEEN 0 AND 3
        AND p.petroMagErr_g BETWEEN 0 AND 3
        AND p.petroMagErr_r BETWEEN 0 AND 3
        AND p.petroMagErr_i BETWEEN 0 AND 3
        AND p.petroMagErr_z BETWEEN 0 AND 3
        AND p.petroRadErr_r BETWEEN 0 AND 3
        AND s.zErr BETWEEN 0 AND 0.1
        AND s.zWarning = 0    -- spectrum is ok_____no_output_____The WHERE conditions ensure that we only select the best possible data. Alternatively, you can download the dataset from [here](http://filestore.nicta.com.au/mlrg-data/astro/sdss_dr7_photometry_source.csv.gz). From now, we assume the dataset is stored in **`data/sdss_dr7_photometry_source.csv.gz`**._____no_output_____### About the Columns_____no_output_____The downloaded dataset is a compressed csv file and can be loaded as a pandas DataFrame._____no_output_____

sdss = pd.read_csv("../data/sdss_dr7_photometry_source.csv.gz", compression="gzip")_____no_output_____

Here are the first four rows of the table. Note that the DataFrame has been transposed below for easier viewing. Without the transpose, each row is an object is the sky and each column is a feature/input._____no_output_____

sdss.head(4).transpose()_____no_output_____

The first two columns (**`ra`** and **`dec`**) are the right ascension and the declination of the object in degrees. The third column (**`class`**) is the spectroscopic class (Star, Galaxy, and Quasar) as determined by expert opnion. This will be the target vector in the classficiation. Some objects are also further divided into subclasses, as indicated by the fourth column.  The columns (**`redshift`** and **`redshiftErr`**) are the redshift (with errror) of the object, also determined by expert opinion.

There are 11 columns that we can use as feature vectors. These are the different PSF and Petrosian magnitude measurements:

* **`psfMag_u`**: PSF magnitude measurement in u-band, assuming the object is a point souce
* **`psfMag_g`**: PSF magnitude measurement in g-band, assuming the object is a point souce
* **`psfMag_r`**: PSF magnitude measurement in r-band, assuming the object is a point souce
* **`psfMag_i`**: PSF magnitude measurement in i-band, assuming the object is a point souce
* **`psfMag_z`**: PSF magnitude measurement in z-band, assuming the object is a point souce
* **`petroMag_u`**: Petrosian magnitude measurement in u-band, assuming the object is an extended souce
* **`petroMag_g`**: Petrosian magnitude measurement in g-band, assuming the object is an extended souce
* **`petroMag_r`**: Petrosian magnitude measurement in r-band, assuming the object is an extended souce
* **`petroMag_i`**: Petrosian magnitude measurement in i-band, assuming the object is an extended souce
* **`petroMag_z`**: Petrosian magnitude measurement in z-band, assuming the object is an extended souce
* **`petroRad_r`**: size measurement of the object in r-band in arc seconds

Each of these 11 measurements also has an associated error.

In addition, there are 4 columns that contain extinction values. These values should be subtracted from the magnitude measurements to correct for the scattering of light by the galactic dust:

* **`extinction_u`**: Extinction value in the u-band
* **`extinction_g`**: Extinction value in the g-band
* **`extinction_r`**: Extinction value in the r-band
* **`extinction_i`**: Extinction value in the i-band
* **`extinction_z`**: Extinction value in the z-band_____no_output_____### Getting the Full SDSS Dataset_____no_output_____In a few notebooks we will also use the full dataset which contains photometric measurements of 800,000 million objects. To obtain these data, remove all the WHERE conditions and LEFT JOIN (instead of JOIN) PhotoObj with SpecObj. Note that since the full set is extremly large (around 200GB), you will not be able to use CasJobs. Instead, you need to email the SDSS Help Desk directly for a custom transfer. Here is the required query:_____no_output_____        SELECT
        p.ra, p.dec,
        CASE s.class WHEN 'GALAXY' THEN 'Galaxy'
                     WHEN 'STAR' THEN 'Star'
                     WHEN 'QSO' THEN 'Quasar'
                     END AS class,
        s.subclass,
        s.z AS redshift,
        s.zErr AS redshiftErr,
        s.zWarning,
        p.psfMag_u, p.psfMagErr_u,
        p.psfMag_g, p.psfMagErr_g,
        p.psfMag_r, p.psfMagErr_r,
        p.psfMag_i, p.psfMagErr_i,
        p.psfMag_z, p.psfMagErr_z,
        p.petroMag_u, p.petroMagErr_u,
        p.petroMag_g, p.petroMagErr_g,
        p.petroMag_r, p.petroMagErr_r,
        p.petroMag_i, p.petroMagErr_i,
        p.petroMag_z, p.petroMagErr_z,
        p.extinction_u, p.extinction_g, p.extinction_r, p.extinction_i, p.extinction_z,
        p.petroRad_r, p.petroRadErr_r

    FROM PhotoObj AS p
       LEFT JOIN SpecObj AS s
       ON s.bestobjid = p.objid_____no_output_____Assuming the files are split into 100 files with names `ANUdata00.csv`, `ANUdata01.csv`, etc., we now combine them into one HDF5 table._____no_output_____

csv_path = '../data/sdss_full/ANUdata{0:02d}.csv'
hdf_path = '../data/sdss_full.h5'
sdss_cols = ['objID', 'ra', 'dec', 'specobjid', 'class', 'subclass', 'redshift',
             'redshiftErr', 'zWarning', 'type', 'clean', 'flags', 'probPSF', 'psfMag_u',
             'psfMagErr_u', 'psfMag_g', 'psfMagErr_g', 'psfMag_r', 'psfMagErr_r',
             'psfMag_i', 'psfMagErr_i', 'psfMag_z', 'psfMagErr_z', 'petroMag_u',
             'petroMagErr_u', 'petroMag_g', 'petroMagErr_g', 'petroMag_r', 'petroMagErr_r',
             'petroMag_i', 'petroMagErr_i', 'petroMag_z', 'petroMagErr_z', 'extinction_u',
             'extinction_g', 'extinction_r', 'extinction_i', 'extinction_z', 'petroRad_r',
             'petroRadErr_r']
csv_to_hdf(csv_path, no_files=100, hdf_path=hdf_path, data_cols=sdss_cols,
           expectedrows=7569900, min_itemsize=40, table_name='sdss_full')_____no_output_____

### Extracting Colour Features_____no_output_____Extract the colour features and scale the data_____no_output_____

# compute reddening correction
A_u_w14, A_g_w14, A_r_w14, A_i_w14, A_z_w14 = reddening_correction_w14(sdss['extinction_r'])

# useful variables
psf_magnitudes = ['psfMag_u', 'psfMag_g', 'psfMag_r', 'psfMag_i', 'psfMag_z']
petro_magnitudes = ['petroMag_u', 'petroMag_g', 'petroMag_r', 'petroMag_i', 'petroMag_z']
w14_corrections = [A_u_w14, A_g_w14, A_r_w14, A_i_w14, A_z_w14]
colours = [('psfMag_u', 'psfMag_g'), ('psfMag_g', 'psfMag_r'), ('psfMag_r', 'psfMag_i'), ('psfMag_i', 'psfMag_z'),
           ('petroMag_u', 'petroMag_g'), ('petroMag_g', 'petroMag_r'), ('petroMag_r', 'petroMag_i'), ('petroMag_i', 'petroMag_z')]

# calculate the corrected magnitudes
correct_magnitudes(sdss, psf_magnitudes, w14_corrections, '_w14')
correct_magnitudes(sdss, petro_magnitudes, w14_corrections, '_w14')

# calculate the corrected magnitudes
compute_colours(sdss, colours, '_w14')

# scale data to zero mean and unit variance
scaler = StandardScaler()
w14_feature_cols = ['psfMag_r_w14', 'psf_u_g_w14', 'psf_g_r_w14', 'psf_r_i_w14',
                'psf_i_z_w14', 'petroMag_r_w14', 'petro_u_g_w14', 'petro_g_r_w14',
                'petro_r_i_w14', 'petro_i_z_w14', 'petroRad_r']
sdss[w14_feature_cols] = scaler.fit_transform(sdss[w14_feature_cols])
save_results(scaler, '../pickle/01_dataset_prepration/sdss_scaler.pickle')

# save as HDF5 table
relevant_cols = ['ra', 'dec', 'class', 'psfMag_r_w14', 'psf_u_g_w14', 'psf_g_r_w14', 'psf_r_i_w14',
                'psf_i_z_w14', 'petroMag_r_w14', 'petro_u_g_w14', 'petro_g_r_w14',
                'petro_r_i_w14', 'petro_i_z_w14', 'petroRad_r']
sdss[relevant_cols].to_hdf('../data/sdss.h5', 'sdss')_____no_output_____

### Extracting Dust Extinction Vector_____no_output_____

ebv_path = '../data/sdss_ebv.h5'
if not results_exist(ebv_path):
    sdss_full = pd.read_hdf('../data/sdss_full.h5', 'sdss_full', columns=['ra', 'dec', 'extinction_r'], chunksize=100000)
    store = pd.HDFStore(ebv_path, complevel=9, complib='zlib', fletcher32=True)
    for i, chunk in enumerate(sdss_full):
        chunk['ebv'] = chunk['extinction_r'] / 2.751
        store.append('sdss_ebv', chunk[['ra', 'dec', 'ebv']], index=False, expectedrows=7569900)
        if i % 10 == 0: print(i // 10, end=' ')
    store.close()_____no_output_____

## VST ATLAS Dataset_____no_output_____

sources = ['../data/vstatlas/lib_gal__AT.fits',
           '../data/vstatlas/lib_qso__AT.fits',
           '../data/vstatlas/lib_star_AT.fits',
           '../data/vstatlas/lib_WD___AT.fits']

classes = ['Galaxy', 'Quasar', 'Star', 'White Dwarf']
vstatlas_features = ['rmagC', 'umg', 'gmr', 'rmi', 'imz', 'rmw1', 'w1m2']

dfs = []
for source, obj in zip(sources, classes):
    data = fits.open(source)
    df = pd.DataFrame(data[1].data)
    df['class'] = obj
    dfs.append(df)

vstatlas = pd.concat(dfs, axis=0, ignore_index=True)
vstatlas = vstatlas[['class', 'rmagC', 'umg', 'gmr', 'rmi', 'imz', 'rmw1', 'w1m2']]
vstatlas = vstatlas.iloc[np.random.permutation(len(vstatlas))]
vstatlas = vstatlas.reset_index(drop=True)

scaler = StandardScaler()
vstatlas[vstatlas_features] = scaler.fit_transform(vstatlas[vstatlas_features])

vstatlas.to_hdf('../data/vstatlas.h5', 'vstatlas')_____no_output_____

To represent a random sky_____no_output_____

# a-priori likelihood for a random object in the sky
counts = np.array([6559, 2303, 25604, 590])
likelihoods = [6559. / 127. * 2496,
               2303. / 5. * 16,
               25604. / 697. * 14868,
               590. / 3. * 38]

np.array(np.sum(counts) * (np.array(likelihoods) / np.sum(likelihoods)), dtype=int)_____no_output_____


# Notebook from ajinkyasheth/solutions-to-coding-problems
Path: msci/NLP_coding_test.ipynb

# Drug Review Dataset (Druglib.com) Data Set

Please visit the UCI Machine Learning Repository to download the data set: http://archive.ics.uci.edu/ml/datasets/Drug+Review+Dataset+%28Druglib.com%29

Summary of Data Set:

The dataset provides patient reviews on specific drugs along with related conditions. Furthermore, reviews are grouped into reports on the three aspects benefits, side effects and overall comment. Additionally, ratings are available concerning overall satisfaction as well as a 5 step side effect rating and a 5 step effectiveness rating.

The UCI Repository contains additional information about the dataset.

Citation:

Surya Kallumadi

Kansas State University

Manhattan, Kansas, USA

surya '@' ksu.edu

Felix Gräßer

Institut für Biomedizinische Technik

Technische Universität Dresden

Dresden, Germany

felix.graesser '@' tu-dresden.de_____no_output_____# Q. The Task
1. Exploratory data analysis - show how the various aspects of the data are related. For example, are drugs with high ratings also highly effective?
2. Train a model that predicts the drug rating.
3. Explore the model results. For example, why might a drug be highly rated? What sort of side effects contribute to a low rating?_____no_output_____

import pandas as pd
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer_____no_output_____import re
import numpy as np_____no_output_____df = pd.read_csv('drugLibTrain_raw.tsv', sep='\t')
print(len(df), "observations in train dataset")
df.head()3107 observations in train dataset

**Attribute Information:**

1. urlDrugName (categorical): name of drug
2. condition (categorical): name of condition
3. benefitsReview (text): patient on benefits
4. sideEffectsReview (text): patient on side effects
5. commentsReview (text): overall patient comment
6. rating (numerical): 10 star patient rating
7. sideEffects (categorical): 5 step side effect rating
8. effectiveness (categorical): 5 step effectiveness rating_____no_output_____# 1 Exploratory Analysis_____no_output_____## 1.1 Distribution of Ratings
We begin out exploratory analysis by checking how the ratings are skewed in distribution_____no_output_____

plt.hist(df['rating'])
plt.xlabel('Ratings')
plt.ylabel('Frequency')
plt.title('Distribution of ratings')
plt.show()_____no_output_____

## 1.2 ratings vs effectiveness and side effects
It will be interested to see how ratings are affected by side effects and effectiveness

For this representation, side effects and effectiveness are represented numerically as follows:

**Effectiveness**
- "Ineffective":0,
- "Marginally Effective":1,
- "Moderately Effective":2,
- "Considerably Effective":3,
- "Highly Effective":4

**Side Effects**
- "Extremely Severe Side Effects":0,
- "Severe Side Effects":1,
- "Moderate Side Effects":2,
- "Mild Side Effects":3,
- "No Side Effects":4_____no_output_____

rating_exp=df[['rating','effectiveness','sideEffects']]
rating_grouped=rating_exp.groupby(['effectiveness','sideEffects']).mean().reset_index()_____no_output_____rating_grouped=rating_exp.groupby(['effectiveness','sideEffects']).mean().reset_index()_____no_output_____effectiveness_dict={"Ineffective":0,
                    "Marginally Effective":1,
                    "Moderately Effective":2,
                    "Considerably Effective":3,
                    "Highly Effective":4
                   }
sideEffects_dict={"Extremely Severe Side Effects":0,
                    "Severe Side Effects":1,
                    "Moderate Side Effects":2,
                    "Mild Side Effects":3,
                    "No Side Effects":4
                   }_____no_output_____rating_grouped['effectiveness']=rating_grouped['effectiveness'].map(lambda l: effectiveness_dict[l])
rating_grouped['sideEffects']=rating_grouped['sideEffects'].map(lambda l: sideEffects_dict[l])_____no_output_____fig = plt.figure(figsize=(10,8))
ax = plt.axes(projection='3d')
ax.plot_trisurf(rating_grouped['sideEffects'], rating_grouped['effectiveness'], rating_grouped['rating'], cmap='viridis', edgecolor='none')
ax.set_xlabel('sideEffects')
ax.set_ylabel('effectiveness')
ax.set_zlabel('rating')
fig.tight_layout()
ax.view_init(30, 235)
plt.title('Ratings vs Side Effects & Effectiveness')
plt.show()

_____no_output_____

**Observations:**

It is clear from the above 3D plot that highly effective drugs with negligible side effects are rated very highly._____no_output_____## 1.2 Exploratory Analysis using VaderSentiment
In the Sentiment Analysis area, there are excellent packages available in Python which have been trained over a large corpus of words. One such toolkit is VaderSentiment. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media.

The idea behind using VaderSentiment is to use pretrained packages to extract sentiment and check if this sentiment can be a good predictor for ratings of the drugs. Additionally, prediction achieved using these packages create a good baseline to validate the performance for model which we implement later. However, before employing such packages blindly, it is worthwhile to conduct a brief exploratory analysis.

*Some part of description borrowed from https://medium.com/analytics-vidhya/simplifying-social-media-sentiment-analysis-using-vader-in-python-f9e6ec6fc52f*
_____no_output_____

analyser = SentimentIntensityAnalyzer()
def sentiment_scores(sentence):
    score = analyser.polarity_scores(sentence)
    return score
def mean(x1,x2,x3):
    return (x1+x2+x3)/3_____no_output_____df['sideeffects_sentiment']=df['sideEffectsReview'].map(lambda l: sentiment_scores(l)['compound'] if type(l) == str else 0)
df['benefits_sentiment']=df['benefitsReview'].map(lambda l: sentiment_scores(l)['compound'] if type(l) == str else 0)
df['comments_sentiment']=df['commentsReview'].map(lambda l: sentiment_scores(l)['compound'] if type(l) == str else 0)_____no_output_____df['overall_sentiment']=df[['sideeffects_sentiment','benefits_sentiment','comments_sentiment']].mean(axis = 1, skipna = True) _____no_output_____rating_senti=df[['rating','overall_sentiment']]
rating_senti_avg=rating_senti.groupby('rating').mean().reset_index()_____no_output_____

VaderSentiment extracts positive, negative, neutral and compound sentiment from the text. For exploratory analysis, we have used the Compound sentiment only. Furthermore, the sentiment of three review columns **sideEffectsReview**, **benefitsReview** and **commentsReview** are averaged out.

Fig 1.2.1 represents overall sentiment vs ratings with sentiment been extracted directly using vader sentiment package
Fig 1.2.2 represents overall sentiment vs ratings with sentiment been extracted after removing medical terms appearing in columns **urlDrugName** and **condition**

In the second case, we are removing medical conditions as they may pose a unintentional bias

*For example, Kindey Failure is a medical condition which may add to a negative sentiment*_____no_output_____

fig, ax=plt.subplots(1,1,figsize=(8,6))
ax.set_ylabel("Overall Sentiment")
ax.set_xlabel("Ratings")
ax.scatter(rating_senti['rating'],rating_senti['overall_sentiment'],alpha=0.1, c='orange', label='Sentiment')
ax.plot(rating_senti_avg['rating'],rating_senti_avg['overall_sentiment'],lw='2',c='purple',label='Mean of sentiment')
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
leg = ax.legend()
ax.set_title("Fig 1.2.1 : Overall sentiment vs ratings")
plt.show()_____no_output_____

**Observations:** Higher ratings on average have higher mean sentiment_____no_output_____

def remove_drug_condition(query,drug_name,condition):
    stopwords=[]
    if type(drug_name) == str:
        stopwords = stopwords + re.sub('[^a-zA-Z ]+', '', drug_name).lower().split()
    if type(condition) == str:
        stopwords = stopwords + re.sub('[^a-zA-Z ]+', '', condition).lower().split()
    result=query
    if len(stopwords)>0:
        querywords = query.split()
        resultwords  = [word for word in querywords if word.lower() not in stopwords]
        result = ' '.join(resultwords)
    return result_____no_output_____df2=pd.DataFrame(df)_____no_output_____for i in df2.index:
    if type(df2['sideEffectsReview'].at[i]) == str:
        df2['sideEffectsReview'].at[i]=remove_drug_condition(df2['sideEffectsReview'].at[i],
                                                             df2['urlDrugName'].at[i],
                                                             df2['condition'].at[i])
for i in df2.index:
    if type(df2['benefitsReview'].at[i]) == str:
        df2['benefitsReview'].at[i]=remove_drug_condition(df2['benefitsReview'].at[i],
                                                             df2['urlDrugName'].at[i],
                                                             df2['condition'].at[i])
for i in df2.index:
    if type(df2['commentsReview'].at[i]) == str:
        df2['commentsReview'].at[i]=remove_drug_condition(df2['commentsReview'].at[i],
                                                             df2['urlDrugName'].at[i],
                                                             df2['condition'].at[i])_____no_output_____df2['sideeffects_sentiment']=df2['sideEffectsReview'].map(lambda l: sentiment_scores(l)['compound'] if type(l) == str else 0)
df2['benefits_sentiment']=df2['benefitsReview'].map(lambda l: sentiment_scores(l)['compound'] if type(l) == str else 0)
df2['comments_sentiment']=df2['commentsReview'].map(lambda l: sentiment_scores(l)['compound'] if type(l) == str else 0)_____no_output_____df2['overall_sentiment']=df2[['sideeffects_sentiment','benefits_sentiment','comments_sentiment']].mean(axis = 1, skipna = True) _____no_output_____rating_senti=df2[['rating','overall_sentiment']]
rating_senti_avg=rating_senti.groupby('rating').mean().reset_index()
fig, ax=plt.subplots(1,1,figsize=(8,6))
ax.set_ylabel("Overall Sentiment")
ax.set_xlabel("Ratings")
ax.scatter(rating_senti['rating'],rating_senti['overall_sentiment'],alpha=0.1, c='orange', label='Sentiment')
ax.plot(rating_senti_avg['rating'],rating_senti_avg['overall_sentiment'],lw='2',c='purple',label='Mean of sentiment')
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
leg = ax.legend()
ax.set_title("Fig 1.2.2 : Overall sentiment vs ratings")
plt.show()_____no_output_____

**Observations:** No significant difference from Fig 1.2.1_____no_output_____

plt.hist(df2['overall_sentiment'])
plt.xlabel('Sentiment')
plt.ylabel('Frequency')
plt.title('Fig 1.2.3 : Distribution of sentiment')
plt.show()_____no_output_____

**Observations:** Sentiments are nomally distributed, we can generate a model to predict ratings based on sentiments using Gaussian Naive Bayes_____no_output_____# 2 Machine Learning Models_____no_output_____## 2.1 Gaussian Naive Bayes
Using sentiments extracted from VaderSentiment as features
_____no_output_____### 2.1.1 Preprocessing_____no_output_____

drugs_train = pd.read_csv('drugLibTrain_raw.tsv', sep='\t')
drugs_test = pd.read_csv('drugLibTest_raw.tsv', sep='\t')_____no_output_____'''
Remove drug names and word in medical condition as it may pose a unintentional bias

For example, Kindey Failure is a medical condition which may add to a negative sentiment
'''
def remove_drug_condition(query,drug_name,condition):
    stopwords=[]
    if type(drug_name) == str:
        stopwords = stopwords + re.sub('[^a-zA-Z ]+', '', drug_name).lower().split()
    if type(condition) == str:
        stopwords = stopwords + re.sub('[^a-zA-Z ]+', '', condition).lower().split()
    result=query
    if len(stopwords)>0:
        querywords = query.split()
        resultwords  = [word for word in querywords if word.lower() not in stopwords]
        result = ' '.join(resultwords)
    return result
def clean_drug_review(df,review_field):
    for i in df.index:
        if type(df[review_field].at[i]) == str:
            df[review_field].at[i]=remove_drug_condition(df[review_field].at[i],
                                                             df['urlDrugName'].at[i],
                                                             df['condition'].at[i])_____no_output_____clean_drug_review(drugs_train,'sideEffectsReview')
clean_drug_review(drugs_train,'benefitsReview')
clean_drug_review(drugs_train,'commentsReview')
clean_drug_review(drugs_test,'sideEffectsReview')
clean_drug_review(drugs_test,'benefitsReview')
clean_drug_review(drugs_test,'commentsReview')_____no_output_____

### 2.1.2 Set up Vader Sentiment_____no_output_____

analyser = SentimentIntensityAnalyzer()
def sentiment_scores(sentence):
    score = analyser.polarity_scores(sentence)
    return score_____no_output_____

### 2.1.3 Extract sentiments from reviews using VaderSentiment
Feature Extraction_____no_output_____

def extract_sentiment(df,review_field):
    df[review_field+"_pos"]=df[review_field].map(lambda l: sentiment_scores(l)['pos'] if type(l) == str else 0)
    df[review_field+"_neg"]=df[review_field].map(lambda l: sentiment_scores(l)['neg'] if type(l) == str else 0)
    df[review_field+"_neu"]=df[review_field].map(lambda l: sentiment_scores(l)['neu'] if type(l) == str else 0)
    df[review_field+"_compound"]=df[review_field].map(lambda l: sentiment_scores(l)['compound'] if type(l) == str else 0)_____no_output_____extract_sentiment(drugs_train,'sideEffectsReview')
extract_sentiment(drugs_train,'benefitsReview')
extract_sentiment(drugs_train,'commentsReview')
extract_sentiment(drugs_test,'sideEffectsReview')
extract_sentiment(drugs_test,'benefitsReview')
extract_sentiment(drugs_test,'commentsReview')_____no_output_____columns_drop=['Unnamed: 0','urlDrugName','effectiveness','sideEffects','condition','benefitsReview','sideEffectsReview','commentsReview']
drugs_train_clean=drugs_train.drop(columns_drop,axis=1)
drugs_test_clean=drugs_test.drop(columns_drop,axis=1)_____no_output_____

### 2.1.4  Model to predict ratings 1 to 10_____no_output_____

from sklearn.naive_bayes import GaussianNB,MultinomialNB, ComplementNB
from sklearn.metrics import roc_curve, auc, confusion_matrix
from sklearn import svm
from sklearn.metrics import accuracy_score_____no_output_____model = GaussianNB()
X_train=drugs_train_clean[drugs_train_clean.columns.difference(['rating'])]
y_train=drugs_train_clean['rating']

X_test=drugs_test_clean[drugs_train_clean.columns.difference(['rating'])]
y_test=drugs_test_clean['rating']_____no_output_____model.fit(X_train,y_train)
y_pred_nb = model.predict(X_test)_____no_output_____print(accuracy_score(y_test, y_pred_nb))0.25386100386100385

**Observations**: With 10 labels accuracy we are getting is 0.25 which is way less than desired. As expected predicting 10 labels is too much of a task for this model. However, we can categorize the performance of drugs in less labels._____no_output_____In the next part, we reduce the number of labels to 3 with -1 indicating poorly rated drugs, 1 indicating highly rated drugs and 0 being avarage rating._____no_output_____

def reduce_label(rating):
    """ maps 0-10 rating to Positive (1), Neutral (0), or Negative (-1)"""
    if rating > 6:
        return 1
    elif rating > 4:
        return 0
    else:
        return -1_____no_output_____

### 2.1.5  Model to predict if drugs is a poor, average or a high performer_____no_output_____

X_train=drugs_train_clean[drugs_train_clean.columns.difference(['rating'])]
y_train=drugs_train_clean['rating'].map(lambda l:reduce_label(l))

X_test=drugs_test_clean[drugs_train_clean.columns.difference(['rating'])]
y_test=drugs_test_clean['rating'].map(lambda l:reduce_label(l))_____no_output_____model = GaussianNB()
model.fit(X_train,y_train)
y_pred_nb = model.predict(X_test)
y_pred_tr_nb = model.predict(X_train)_____no_output_____print('Testing Accuracy: '+str(accuracy_score(y_test, y_pred_nb)))
print('Training Accuracy: '+str(accuracy_score(y_train, y_pred_tr_nb)))Testing Accuracy: 0.6476833976833977
Training Accuracy: 0.6717090440939814

**Observations:** Accuracy is much better._____no_output_____## 2.2 Predicting drug rating without extracting sentiment
Here we attempt to predict drug rating using the words from three review sets directly_____no_output_____

from nltk.corpus import stopwords
from sklearn.feature_extraction.text import CountVectorizer
import itertools_____no_output_____stopwords_set=set(stopwords.words('english'))_____no_output_____drugs_train[['rating','benefitsReview','sideEffectsReview','commentsReview']].head()_____no_output_____

### 2.2.1 Combine three sets of reviews into one for both training and testing data set_____no_output_____

drug_reviews_train=pd.concat(
    [drugs_train['rating'],
     drugs_train['commentsReview'].replace(np.nan,"") \
     .str.cat(drugs_train['sideEffectsReview'].replace(np.nan,""),sep=" ") \
     .str.cat(drugs_train['benefitsReview'].replace(np.nan,""),sep=" ")],
    axis=1)
_____no_output_____drug_reviews_test=pd.concat(
    [drugs_test['rating'],
     drugs_test['commentsReview'].replace(np.nan,"") \
     .str.cat(drugs_test['sideEffectsReview'].replace(np.nan,""),sep=" ") \
     .str.cat(drugs_test['benefitsReview'].replace(np.nan,""),sep=" ")],
    axis=1)_____no_output_____def make_xy(reviews, vectorizer=None):
    verdict=reviews[['rating','commentsReview']]
    if vectorizer == None:
        vectorizer=CountVectorizer(min_df=0,stop_words=stopwords_set)
        vectorizer.fit(np.array(verdict['commentsReview']))
    X = vectorizer.transform(np.array(verdict['commentsReview']))
    rating = np.array(verdict['rating'])
    Y=[1 if i>6 else 0 if i>4 else -1 for i in rating]
    return X, Y
_____no_output_____

### 2.2.2 Extract tf-idf features and generate train and test data_____no_output_____

vectorizer=CountVectorizer(min_df=0,stop_words=stopwords_set)
vectorizer.fit(np.array(drug_reviews_train['commentsReview']))
X_train, y_train = make_xy(drug_reviews_train, vectorizer)
X_test, y_test = make_xy(drug_reviews_test, vectorizer)_____no_output_____X_train=X_train.toarray()
X_test=X_test.toarray()_____no_output_____

### 2.2.3 Set up model and compare performance_____no_output_____

model = GaussianNB()
model.fit(X_train,y_train)
y_pred_nb = model.predict(X_test)
y_pred_tr_nb = model.predict(X_train)_____no_output_____print('Testing Accuracy: '+str(accuracy_score(y_test, y_pred_nb)))
print('Training Accuracy: '+str(accuracy_score(y_train, y_pred_tr_nb)))Testing Accuracy: 0.5694980694980695
Training Accuracy: 0.939491470872224

**Observations:**

Since testing accuracy is much lesser than the training accuracy, this is a case of over fitting. Overfitting may happen for various reasons. In the further sections, we not only try to deal with overfitting using common techniques but also try a different approach as it is possible the model used currently is not suited for this kind of problem._____no_output_____### 2.3 Deal with overfitting_____no_output_____**2.3.1 Add drugnames and conditions to stopwords set**_____no_output_____

condition_words=list(drugs_train['condition'].map(lambda l: re.sub('[^a-zA-Z ]+', ' ', l) if type(l)==str else "").map(lambda l:l.split(" ")))
condition_words=list(set(itertools.chain.from_iterable(condition_words)))
condition_words=[w if len(w)>=2 else '#None#' for w in condition_words]
condition_words.remove('#None#')_____no_output_____drugnames_words=list(drugs_train['urlDrugName'].map(lambda l: re.sub('[^a-zA-Z ]+', ' ', l) if type(l)==str else "").map(lambda l:l.split(" ")))
drugnames_words=list(set(itertools.chain.from_iterable(drugnames_words)))
drugnames_words=[w if len(w)>=2 else '#None#' for w in drugnames_words]
drugnames_words.remove('#None#')_____no_output_____stopwords_set_custom=stopwords_set.union(set(condition_words)).union(set(drugnames_words))_____no_output_____

**2.3.2 Remove execessively large reviews**_____no_output_____

reviews_by_length=np.array(drug_reviews_train['commentsReview'].map(lambda l:len(l)))_____no_output_____plt.hist(reviews_by_length)
plt.show()_____no_output_____drug_reviews_train2=pd.DataFrame(drug_reviews_train)
drug_reviews_train2['review_length']=drug_reviews_train2['commentsReview'].map(lambda l:len(l))
drug_reviews_train2=drug_reviews_train2[drug_reviews_train2['review_length']<=2000]_____no_output_____vectorizer=CountVectorizer(min_df=0,stop_words=stopwords_set_custom)
vectorizer.fit(np.array(drug_reviews_train['commentsReview']))
X_train, y_train = make_xy(drug_reviews_train, vectorizer)
X_test, y_test = make_xy(drug_reviews_test, vectorizer)_____no_output_____X_train=X_train.toarray()
X_test=X_test.toarray()_____no_output_____model = GaussianNB()
model.fit(X_train,y_train)
y_pred_nb = model.predict(X_test)
y_pred_tr_nb = model.predict(X_train)_____no_output_____print('Testing Accuracy: '+str(accuracy_score(y_test, y_pred_nb)))
print('Training Accuracy: '+str(accuracy_score(y_train, y_pred_tr_nb)))Testing Accuracy: 0.5472972972972973
Training Accuracy: 0.9279047312520116

**Observations:**

Even after trying varous approaches to deal with over fitting, there is not significant difference in the accuracy. In the next section, we use Multinomoal Naive Bayes._____no_output_____### 2.4 Multinomial Naive Bayes
_____no_output_____#### 2.4.1 Model Fitting and Accuracy Evaluation_____no_output_____

model = MultinomialNB()
model.fit(X_train,y_train)
y_pred_nb = model.predict(X_test)
y_pred_tr_nb = model.predict(X_train)_____no_output_____print('Testing Accuracy: '+str(accuracy_score(y_test, y_pred_nb)))
print('Training Accuracy: '+str(accuracy_score(y_train, y_pred_tr_nb)))Testing Accuracy: 0.6978764478764479
Training Accuracy: 0.8751206952043772

The testing accuracy we achieve is about 70%, however, we can improve this accuracy by using regularization_____no_output_____#### 2.4.2 Estimating alpha_____no_output_____

alphas=np.arange(1,20)*0.1
accuracy_train=[]
accuracy_test=[]
for alpha in alphas:
    model = ComplementNB(alpha=alpha)
    model.fit(X_train,y_train)
    y_pred_nb = model.predict(X_test)
    y_pred_tr_nb = model.predict(X_train)
    accuracy_test.append(accuracy_score(y_test, y_pred_nb))
    accuracy_train.append(accuracy_score(y_train, y_pred_tr_nb))_____no_output_____plt.plot(alphas,accuracy_train,label='Train Accuracy')
plt.plot(alphas,accuracy_test,label='Test Accuracy')
plt.xlabel('alpha')
plt.ylabel('accuracy')
plt.legend()
plt.title('Alpha vs Accuracy')
plt.show()_____no_output_____

### 2.5 Final Model
Multinomoal Naive Bayes with alpha=0.75 gives the best testing accuracy
_____no_output_____

model = MultinomialNB(alpha=0.75)
model.fit(X_train,y_train)
y_pred_nb = model.predict(X_test)
y_pred_tr_nb = model.predict(X_train)_____no_output_____print('Testing Accuracy: '+str(accuracy_score(y_test, y_pred_nb)))
print('Training Accuracy: '+str(accuracy_score(y_train, y_pred_tr_nb)))Testing Accuracy: 0.7123552123552124
Training Accuracy: 0.9008690054715159

**Observations**:

So far we are able to achieve an accuracy of 0.71 which is decent however much lesser than the training accuracy. This could be a potential case of overfitting. Analyzing it further as to which words could be better predictors of classes can perhaps give us more insights._____no_output_____## 3 Result Explorations
List of words along with their corresponding probabilities which best predict three classes_____no_output_____

import scipy.sparse as sp_____no_output_____identity=sp.identity(X_train.shape[1])
probs=model.predict_proba(identity)
vec=pd.DataFrame(vectorizer.get_feature_names(), columns=['Features'])_____no_output_____print('10 words that best predicts bad drug rating')
vec['probs']=probs[:,0]
vec.sort_values(by=['probs'],ascending=False).head(10)10 words that best predicts bad drug rating
print('10 words that best predicts neutral drug rating')
vec['probs']=probs[:,1]
vec.sort_values(by=['probs'],ascending=False).head(10)10 words that best predicts neutral drug rating
print('10 words that best predicts good drug rating')
vec['probs']=probs[:,2]
vec.sort_values(by=['probs'],ascending=False).head(10)10 words that best predicts good drug rating

**Observations**:

We do see words like *excellent*, *consistently*, etc being predictors of good ratings. While words like *unsurprizingly* predicting  neutral ratings and *ached* predicting bad ratings. However, in the above tables, more often than not we see words such as rivotril, percocette, etc which are drug/chemical compound names and ideally shouldn't be an indicator whether the drug is performing good or not. We may need to perform a better text preprocessing to counter this issue. It is also possible some of the chemical compund / drug names are consistently associated with poor drugs. In this case, some domain expertise may be needed to evaluate it further._____no_output_____## 4 Next Steps
While this model gives decent accuracy in predicting the drug rating using the sentiment/content of the reviews, it can be improved upon. A more robust text preprocessing may increase the overall accuracy metric and perhaps employing deep learning techniques may give a further boost to the model performance.

With due respect in an interest of time, this will be my final submission for the coding round.

*- AJ (Ajinkya)
ajinkya@uw.edu*_____no_output_____## Thank You_____no_output_____


# Notebook from kojikoji/deepcolor
Path: tutorial/deepcolor_tutorial.ipynb

# Installation
Install required packages for deepCOLOR analysis_____no_output_____

! pip install deepcolor imgaug==0.2.5_____no_output_____

# Preprocessing data
Although deepCOLOR estimate the spatial distribution of single cells based on raw count matrix of single cell and spatail transcriptomes, it requires normalized expression for downstream analysis such as DEG detection. Also, we exclude lowly expressed cells and spots from folowing analysis. In this tutorial, we use pre-processed dataset of squamous cartinoma from [Ji et al. 2020](https://www.sciencedirect.com/science/article/pii/S0092867420306723?via%3Dihub).
_____no_output_____

! mkdir -p data/tutorial/
! wget -O data/tutorial/scc_sc.h5ad https://www.dropbox.com/s/cx2wx67cpqpu167/scc_sc.h5ad?dl=0
! wget -O data/tutorial/scc_sp.h5ad https://www.dropbox.com/s/as4v1yh4yrezrc6/scc_sp.h5ad?dl=0
import torch
import scanpy as sc
import numpy as np
np.random.seed(1)
torch.manual_seed(1)_____no_output_____sc_adata = sc.read_h5ad('data/tutorial/scc_sc.h5ad')
sp_adata = sc.read_h5ad('data/tutorial/scc_sp.h5ad')_____no_output_____

You need to align gene names for both sinlge cell and spatial `AnnData` objects. We also excluede genes lowly expressed._____no_output_____

import numpy as np
sc_adata = sc_adata[:, sc_adata.layers['count'].toarray().sum(axis=0) > 10]
sp_adata = sp_adata[:, sp_adata.layers['count'].toarray().sum(axis=0) > 10]
common_genes = np.intersect1d(sc_adata.var_names, sp_adata.var_names)
sc_adata = sc_adata[:, common_genes]
sp_adata = sp_adata[:, common_genes]_____no_output_____

# Estimate Spatial distribution
Estimation of spatial distribution is conduted for each singel cell. For this process, we strongly recommend you to conduct the caluculation in GPU availabel environments._____no_output_____

# conduct spatial estimation
import deepcolor
import importlib
importlib.reload(deepcolor)
sc_adata, sp_adata = deepcolor.estimate_spatial_distribution(sc_adata, sp_adata, param_save_path='data/tutorial/opt_params.pt', first_epoch=500, second_epoch=500)
Loss: 15761.6572265625
Start first opt
loss at epoch 0 is 14228.947265625
loss at epoch 10 is 12343.0185546875
loss at epoch 20 is 11043.0380859375
loss at epoch 30 is 10304.349609375
loss at epoch 40 is 9866.2666015625
loss at epoch 50 is 9583.560546875
loss at epoch 60 is 9384.0888671875
loss at epoch 70 is 9230.7578125
loss at epoch 80 is 9087.322265625
loss at epoch 90 is 8979.146484375
loss at epoch 100 is 8888.1865234375
loss at epoch 110 is 8807.9755859375
loss at epoch 120 is 8738.755859375
loss at epoch 130 is 8677.3427734375
loss at epoch 140 is 8617.3212890625
loss at epoch 150 is 8569.28125
loss at epoch 160 is 8526.8701171875
loss at epoch 170 is 8485.912109375
loss at epoch 180 is 8451.341796875
loss at epoch 190 is 8416.1298828125
loss at epoch 200 is 8384.6728515625
loss at epoch 210 is 8353.6142578125
loss at epoch 220 is 8323.42578125
loss at epoch 230 is 8294.5341796875
loss at epoch 240 is 8263.7646484375
loss at epoch 250 is 8230.5927734375
loss at epoch 260 is 8197.173828125
loss at epoch 270 is 8166.50390625
loss at epoch 280 is 8138.2705078125
loss at epoch 290 is 8112.22900390625
loss at epoch 300 is 8082.3740234375
loss at epoch 310 is 8058.859375
loss at epoch 320 is 8035.6884765625
loss at epoch 330 is 8014.42431640625
loss at epoch 340 is 7994.841796875
loss at epoch 350 is 7976.83154296875
loss at epoch 360 is 7959.3291015625
loss at epoch 370 is 7945.46435546875
loss at epoch 380 is 7928.8359375
loss at epoch 390 is 7914.91748046875
loss at epoch 400 is 7901.5048828125
loss at epoch 410 is 7885.7509765625
loss at epoch 420 is 7872.88330078125
loss at epoch 430 is 7860.99951171875
loss at epoch 440 is 7847.84716796875
loss at epoch 450 is 7837.0859375
loss at epoch 460 is 7823.966796875
loss at epoch 470 is 7813.330078125
loss at epoch 480 is 7803.15576171875
loss at epoch 490 is 7793.119140625
Done first opt
Loss: 8372.0625
Start second opt
loss at epoch 0 is 310742.34375
loss at epoch 10 is 307759.9375
loss at epoch 20 is 304705.5625
loss at epoch 30 is 301811.78125
loss at epoch 40 is 299031.5
loss at epoch 50 is 296557.625
loss at epoch 60 is 294494.28125
loss at epoch 70 is 292427.34375
loss at epoch 80 is 290646.0
loss at epoch 90 is 288718.90625
loss at epoch 100 is 287085.09375
loss at epoch 110 is 285561.46875
loss at epoch 120 is 283842.84375
loss at epoch 130 is 282291.9375
loss at epoch 140 is 280862.8125
loss at epoch 150 is 279306.0
loss at epoch 160 is 277867.1875
loss at epoch 170 is 276626.3125
loss at epoch 180 is 275127.25
loss at epoch 190 is 273976.5625
loss at epoch 200 is 272326.71875
loss at epoch 210 is 271211.28125
loss at epoch 220 is 270102.90625
loss at epoch 230 is 268787.75
loss at epoch 240 is 267585.1875
loss at epoch 250 is 266699.9375
loss at epoch 260 is 265555.625
loss at epoch 270 is 264225.625
loss at epoch 280 is 263090.15625
loss at epoch 290 is 263003.90625
loss at epoch 300 is 260666.28125
loss at epoch 310 is 259951.140625
loss at epoch 320 is 258948.296875
loss at epoch 330 is 257705.546875
loss at epoch 340 is 256665.71875
loss at epoch 350 is 255908.65625
loss at epoch 360 is 254948.15625
loss at epoch 370 is 253819.6875
loss at epoch 380 is 252760.796875
loss at epoch 390 is 252524.296875
loss at epoch 400 is 251504.734375
loss at epoch 410 is 250914.765625
loss at epoch 420 is 249765.609375
loss at epoch 430 is 248463.265625
loss at epoch 440 is 247445.609375
loss at epoch 450 is 247060.28125
loss at epoch 460 is 246079.515625
loss at epoch 470 is 245274.703125
loss at epoch 480 is 244920.984375
loss at epoch 490 is 244133.046875
Done second opt
Loss: 243013.78125

Here, we will display imputed expression patterns, combining estiamted spatial distribution of single cells with its expression profiles._____no_output_____

from matplotlib import pyplot as plt
sp_adata = deepcolor.calculate_imputed_spatial_expression(sc_adata, sp_adata)
sc.pl.spatial(sp_adata, color="CD8A", layer='imputed_exp')
_____no_output_____sc.pl.spatial(sp_adata, color="CD8A", layer='count')_____no_output_____

The first panel represents the imputed CD8A expression, while the second panel represents original expression. It is shown that imputed expression is more smooth than original one._____no_output_____While deepCOLOR estimate spatial distributions in single cell level, it can estimate the spatial distribution of spcified cell popluation by aggregating the distribution of belongin cells._____no_output_____

sp_adata = deepcolor.calculate_clusterwise_distribution(sc_adata, sp_adata, 'level3_celltype')
sc.pl.spatial(sp_adata, color='CD8_EM')some of cluster names in `cluster_label` is overlapped with `sp_adata.obs.columns`

# Colocalization based ligand receptor interaction_____no_output_____Using ligand-target regulatory potentianl derived from NicheNet ([Browaeys et al. 2020](https://www.nature.com/articles/s41592-019-0667-5)), we can estimate the ligand activity between spatially proximal cells._____no_output_____

! wget -O data/tutorial/ligand_target_df.csv https://www.dropbox.com/s/2z7ogbks4504iya/ligand_target_df.csv?dl=0
import pandas as pd
importlib.reload(deepcolor)
lt_df = pd.read_csv('data/tutorial/ligand_target_df.csv', index_col=0)
fig, coexp_cc_df = deepcolor.calculate_proximal_cell_communications(sc_adata, 'level1_celltype', lt_df, ["PDC", "CD1C", "ASDC", "MDSC", "Mac", "MDSC"], celltype_sample_num=500, ntop_genes=4000, each_display_num=3, role="sender", edge_thresh=1)
fig/home/tiisaishima/mambaforge/envs/deepcolor/lib/python3.8/site-packages/anndata/utils.py:111: UserWarning:

Suffix used (-[0-9]+) to deduplicate index values may make index values difficult to interpret. There values with a similar suffixes in the index. Consider using a different delimiter by passing `join={delimiter}`Example key collisions generated by the make_index_unique algorithm: ['P2_Tumor_TTCCCAGGTCTCTTTA-1', 'P2_Tumor_TTCCCAGGTCTCTTTA-2', 'P2_Tumor_TTCCCAGGTCTCTTTA-3', 'P2_Tumor_TTCCCAGGTCTCTTTA-4', 'P7_Tumor_ATGCGATGTCCAAGTT-1']

Observation names are not unique. To make them unique, call `.obs_names_make_unique`.
Observation names are not unique. To make them unique, call `.obs_names_make_unique`.

# Colocalization based clustering
Clustering cell pairs with high coloalization scores, we can obtain cell population pairs characterized by its colocalization._____no_output_____

dual_adata = deepcolor.analyze_pair_cluster(sc_adata, sp_adata, ["Epithelial"], ["Fibroblast"], "level1_celltype", max_pair_num=30000)
sc_adata.obs['coloc_cluster'] = 'None'
sc_adata.obs.loc[dual_adata.obs.cell1_obsname, 'coloc_cluster'] = dual_adata.obs.leiden.values
sc_adata.obs.loc[dual_adata.obs.cell2_obsname, 'coloc_cluster'] = dual_adata.obs.leiden.values
pair_adata = sc_adata[sc_adata.obs['coloc_cluster'] != 'None']
sc.pl.umap(pair_adata, color=['level1_celltype', 'coloc_cluster'])/home/tiisaishima/packages/deepcolor/src/deepcolor/workflow.py:306: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

/home/tiisaishima/packages/deepcolor/src/deepcolor/workflow.py:307: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

/home/tiisaishima/mambaforge/envs/deepcolor/lib/python3.8/site-packages/anndata/_core/anndata.py:1236: ImplicitModificationWarning:

Initializing view as actual.

Trying to set attribute `.obs` of view, copying.
... storing 'large_class' as categorical
Trying to set attribute `.obs` of view, copying.
... storing 'coloc_cluster' as categorical