# Notebook from mohityogesh44/ds-seed
Path: Regression/Linear Models/LassoLars_StandardScaler_QuantileTransformer.ipynb

# LassoLars with StandardScaler & Quantile Transformer
_____no_output_____**This Code template is for the regression analysis using a LassoLars Regression and the feature rescaling technique StandardScaler along with Quantile Transformer as a feature transformation technique in a pipeline** _____no_output_____### Required Packages_____no_output_____

import warnings as wr
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler,QuantileTransformer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LassoLars
from sklearn.metrics import mean_squared_error, r2_score,mean_absolute_error
wr.filterwarnings('ignore')_____no_output_____

### Initialization

Filepath of CSV file_____no_output_____

#filepath
file_path= ""_____no_output_____

List of features which are  required for model training ._____no_output_____

#x_values
features=[]_____no_output_____

Target feature for prediction._____no_output_____

#y_value
target=''_____no_output_____

### Data Fetching

Pandas is an open-source, BSD-licensed library providing high-performance, easy-to-use data manipulation and data analysis tools.

We will use panda's library to read the CSV file using its storage path.And we use the head function to display the initial row or entry._____no_output_____

df=pd.read_csv(file_path) #reading file
df.head()#displaying initial entries_____no_output_____print('Number of rows are :',df.shape[0], ',and number of columns are :',df.shape[1])Number of rows are : 400 ,and number of columns are : 9
df.columns.tolist()
_____no_output_____

### Data Preprocessing

Since the majority of the machine learning models in the Sklearn library doesn't handle string category data and Null value, we have to explicitly remove or replace null values. The below snippet have functions, which removes the null value if any exists. And convert the string classes data in the datasets by encoding them to integer classes.
_____no_output_____

def NullClearner(df):
    if(isinstance(df, pd.Series) and (df.dtype in ["float64","int64"])):
        df.fillna(df.mean(),inplace=True)
        return df
    elif(isinstance(df, pd.Series)):
        df.fillna(df.mode()[0],inplace=True)
        return df
    else:return df
def EncodeX(df):
    return pd.get_dummies(df)_____no_output_____

#### Correlation Map

In order to check the correlation between the features, we will plot a correlation matrix. It is effective in summarizing a large amount of data where the goal is to see patterns._____no_output_____

plt.figure(figsize = (15, 10))
corr = df.corr()
mask = np.triu(np.ones_like(corr, dtype = bool))
sns.heatmap(corr, mask = mask, linewidths = 1, annot = True, fmt = ".2f")
plt.show()_____no_output_____correlation = df[df.columns[1:]].corr()[target][:]
correlation_____no_output_____

### Feature Selections

It is the process of reducing the number of input variables when developing a predictive model. Used to reduce the number of input variables to both reduce the computational cost of modelling and, in some cases, to improve the performance of the model.

We will assign all the required input features to X and target/outcome to Y._____no_output_____

#spliting data into X(features) and Y(Target)

X=df[features]
Y=df[target]  _____no_output_____

Calling preprocessing functions on the feature and target set._____no_output_____

x=X.columns.to_list()
for i in x:
    X[i]=NullClearner(X[i])
X=EncodeX(X)
Y=NullClearner(Y)
X.head()_____no_output_____

### Data Splitting

The train-test split is a procedure for evaluating the performance of an algorithm. The procedure involves taking a dataset and dividing it into two subsets. The first subset is utilized to fit/train the model. The second subset is used for prediction. The main motive is to estimate the performance of the model on new data._____no_output_____

#we can choose randomstate and test_size as over requerment
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.2, random_state = 1) #performing datasplitting_____no_output_____

### Data Scaling
**Used StandardScaler**

* Standardize features by removing the mean and scaling to unit variance The standard score of a sample x is calculated as:

z = (x - u) / s

* Where u is the mean of the training samples or zero if with_mean=False, and s is the standard deviation of the training samples or one if with_std=False.

### Feature Transformation
**QuantileTransformer :**

This method transforms the features to follow a uniform or a normal distribution. Therefore, for a given feature, this transformation tends to spread out the most frequent values. It also reduces the impact of (marginal) outliers: this is therefore a robust preprocessing scheme.

More about QuantileTransformer module https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html

## Model
**LassoLars :**

LassoLars is a lasso model implemented using the LARS algorithm, and unlike the implementation based on coordinate descent, this yields the exact solution, which is piecewise linear as a function of the norm of its coefficients.

**Tuning parameters:**

* fit_intercept -> whether to calculate the intercept for this model. If set to false, no intercept will be used in calculations

* alpha -> Constant that multiplies the penalty term. Defaults to 1.0. alpha = 0 is equivalent to an ordinary least square, solved by LinearRegression. For numerical reasons, using alpha = 0 with the LassoLars object is not advised and you should prefer the LinearRegression object.

* eps -> The machine-precision regularization in the computation of the Cholesky diagonal factors. Increase this for very ill-conditioned systems. Unlike the tol parameter in some iterative optimization-based algorithms, this parameter does not control the tolerance of the optimization.

* max_iter -> Maximum number of iterations to perform.

* positive -> Restrict coefficients to be >= 0. Be aware that you might want to remove fitintercept which is set True by default. Under the positive restriction the model coefficients will not converge to the ordinary-least-squares solution for small values of alpha. Only coefficients up to the smallest alpha value (alphas[alphas_ > 0.].min() when fit_path=True) reached by the stepwise Lars-Lasso algorithm are typically in congruence with the solution of the coordinate descent Lasso estimator.

* precompute -> Whether to use a precomputed Gram matrix to speed up calculations.
_____no_output_____

#training the LassoLars

model = make_pipeline(StandardScaler(),QuantileTransformer(),LassoLars(random_state=1))
model.fit(X_train,y_train)_____no_output_____

#### Model Accuracy
score() method return the mean accuracy on the given test data and labels.

In multi-label classification, this is the subset accuracy which is a harsh metric since you require for each sample that each label set be correctly predicted._____no_output_____

print("Accuracy score {:.2f} %\n".format(model.score(X_test,y_test)*100))Accuracy score -0.34 %

#prediction on testing set
prediction=model.predict(X_test)_____no_output_____

### Model evolution

**r2_score:** The r2_score function computes the percentage variablility explained by our model, either the fraction or the count of correct predictions.

**MAE:** The mean abosolute error function calculates the amount of total error(absolute average distance between the real data and the predicted data) by our model.

**MSE:** The mean squared error function squares the error(penalizes the model for large errors) by our model._____no_output_____

print('Mean Absolute Error:', mean_absolute_error(y_test, prediction))
print('Mean Squared Error:', mean_squared_error(y_test, prediction))
print('Root Mean Squared Error:', np.sqrt(mean_squared_error(y_test, prediction)))Mean Absolute Error: 0.122846875
Mean Squared Error: 0.023206203124999998
Root Mean Squared Error: 0.15233582351174
print("R-squared score : ",r2_score(y_test,prediction))R-squared score :  -0.0034057220995624693
#ploting actual and predicted
red = plt.scatter(np.arange(0,80,5),prediction[0:80:5],color = "red")
green = plt.scatter(np.arange(0,80,5),y_test[0:80:5],color = "green")
plt.title("Comparison of Regression Algorithms")
plt.xlabel("Index of Candidate")
plt.ylabel("target")
plt.legend((red,green),('LassoLars', 'REAL'))
plt.show()
_____no_output_____

### Prediction Plot¶
First, we make use of a plot to plot the actual observations, with x_train on the x-axis and y_train on the y-axis. For the regression line, we will use x_train on the x-axis and then the predictions of the x_train observations on the y-axis._____no_output_____

plt.figure(figsize=(10,6))
plt.plot(range(20),y_test[0:20], color = "green")
plt.plot(range(20),model.predict(X_test[0:20]), color = "red")
plt.legend(["Actual","prediction"])
plt.title("Predicted vs True Value")
plt.xlabel("Record number")
plt.ylabel(target)
plt.show()_____no_output_____

#### Creator: Vipin Kumar , Github: [Profile](https://github.com/devVipin01)_____no_output_____


# Notebook from cabb99/open3spn2
Path: examples/from_sequence/Simulation_from_sequence.ipynb

Table of Contents

1  Generate atomistic model

_____no_output_____## Generate atomistic model_____no_output_____

import open3SPN2

# Initialize the DNA from a sequence.
# DNA type can be changed to 'A' or 'B'

seq='ATACAAAGGTGCGAGGTTTCTATGCTCCCACG'
dna=open3SPN2.DNA.fromSequence(seq, dna_type='B_curved',output_pdb=f'initial.pdb')

# Compute the topology for the DNA structure.
# Since the dna was generated from the sequence using X3DNA,
# it is not necesary to recompute the geometry.

dna.computeTopology(template_from_X3DNA=False)

# Create the system.
# To set periodic boundary conditions (periodicBox=[50,50,50]).
# The periodic box size is in nanometers.

dna.periodic=False
s=open3SPN2.System(dna, periodicBox=None)

#Add 3SPN2 forces

s.add3SPN2forces(verbose=True)
Bond
Angle
Stacking
Dihedral
BasePair
CrossStacking
Exclusion
Electrostatics
import sys
import simtk.openmm
import simtk.openmm.app
import simtk.unit
import numpy as np

#Initialize Molecular Dynamics simulations

simulation_platform='OpenCL' #['Reference', 'CPU', 'OpenCL', 'CUDA']
temperature=300 * simtk.unit.kelvin
s.initializeMD(temperature=temperature,platform_name=simulation_platform)
simulation=s.simulation

#Set initial positions

simulation.context.setPositions(s.coord.getPositions())_____no_output_____#Obtain total energy

energy_unit=simtk.openmm.unit.kilojoule_per_mole
state = simulation.context.getState(getEnergy=True)
energy = state.getPotentialEnergy().value_in_unit(energy_unit)
print('TotalEnergy',round(energy,6),energy_unit.get_symbol())

#Obtain detailed energy

energies = {}
for force_name, force in s.forces.items():
    group=force.getForceGroup()
    state = simulation.context.getState(getEnergy=True, groups=2**group)
    energies[force_name] =state.getPotentialEnergy().value_in_unit(energy_unit)

for force_name in s.forces.keys():
    print(force_name, round(energies[force_name],6),energy_unit.get_symbol())TotalEnergy -2250.551514 kJ/mol
Bond 2.1e-05 kJ/mol
Angle 0.002319 kJ/mol
Stacking -793.345703 kJ/mol
Dihedral -839.999756 kJ/mol
BasePair -522.301514 kJ/mol
CrossStacking -138.427765 kJ/mol
Exclusion 0.219572 kJ/mol
Electrostatics 43.301292 kJ/mol
#Add simulation reporters

dcd_reporter=simtk.openmm.app.DCDReporter(f'output.dcd', 1000)
energy_reporter=simtk.openmm.app.StateDataReporter(sys.stdout, 1000, step=True,time=True,
                                                   potentialEnergy=True, temperature=True)
simulation.reporters.append(dcd_reporter)
simulation.reporters.append(energy_reporter)_____no_output_____#Run simulation
simulation.minimizeEnergy()
simulation.context.setVelocitiesToTemperature(temperature)
simulation.step(10000)#"Step","Time (ps)","Potential Energy (kJ/mole)","Temperature (K)"
1000,2.0000000000000013,-1679.2578125,322.2481053243356
2000,3.999999999999781,-1682.40673828125,306.9846769155262
3000,5.999999999999561,-1700.3797607421875,305.51271863081064
4000,7.999999999999341,-1677.521240234375,309.2116127323974
5000,10.000000000000009,-1655.54248046875,313.8007790242212
6000,12.000000000000677,-1669.7579345703125,284.2262067969661
7000,14.000000000001345,-1639.3497314453125,290.03726981450706
8000,16.00000000000201,-1720.05810546875,286.10434070687154
9000,18.000000000000902,-1680.2728271484375,276.95129035343757
10000,19.999999999999794,-1691.1907958984375,297.7951922786541


# Notebook from C-Aguirre017/DeepMultiSurveyClassificationOfVariableStars
Path: 3.- Experiments RF/Random Forrest.ipynb

from sklearn.ensemble import RandomForestClassifier
import glob, os
import itertools
import numpy as np
from sklearn.model_selection import StratifiedKFold
import pandas as pd
from tqdm import tqdm_____no_output_____

# Settings_____no_output_____

# Technical Setting
limit = 8000 # Maximum amount of Star Per Class
permutation = True # Permute Files
n = 1000
folds = 10

# Files Setting
base_path = '/Users/Carlos/Desktop/Magister/3*/'
regular_exp1 = base_path + 'Data/Corot/**/*.npy'
regular_exp2 = base_path + 'Data/**/OGLE-*.npy'
regular_exp3 = base_path + 'Data/VVV/**/*.npy'_____no_output_____## Methods
subclasses = ['cepDiez', 'cepEfe', 'RRab', 'RRc', 'nonEC', 'EC', 'Mira', 'SRV', 'Osarg']
# subclasses = ['lpv','cep','rrlyr','ecl']

def get_survey(path):
    if 'Corot' in path:
        return 'Corot'
    elif 'VVV' in path:
        return 'VVV'
    elif 'OGLE' in path:
        return 'OGLE'
    else:
        return 'err'

def get_name(path):
    for subclass in subclasses:
        if subclass in path:
            return subclass
    return 'err'

def get_name_with_survey(path):
    for subclass in subclasses:
        if subclass in path:
            survey = get_survey(path)
            return survey + '_' + subclass
    return 'err'

def get_files(permutation=False):
    files1 = np.array(list(glob.iglob(regular_exp1, recursive=True)))
    files2 = np.array(list(glob.iglob(regular_exp2, recursive=True)))
    files3 = np.array(list(glob.iglob(regular_exp3, recursive=True)))

    print('[!] Files in Memory')

    # Permutations
    if permutation:
        files1 = files1[np.random.permutation(len(files1))]
        files2 = files2[np.random.permutation(len(files2))]
        files3 = files3[np.random.permutation(len(files3))]

        print('[!] Permutation applied')

    aux_dic = {}
    corot = {}
    vvv = {}
    ogle = {}
    for subclass in subclasses:
        aux_dic[subclass] = []
        corot[subclass] = 0
        vvv[subclass] = 0
        ogle[subclass] = 0

    new_files = []
    for idx in tqdm(range(len(files2))):
        foundCorot = False
        foundVista = False
        foundOgle = False

        for subclass in subclasses:
            # Corot
            if not foundCorot and corot[subclass] < limit and idx < len(files1) and subclass in files1[idx]:
                new_files += [files1[idx]]
                corot[subclass] += 1
                foundCorot = True

            # Ogle
            if not foundOgle and ogle[subclass] < limit and subclass in files2[idx]:
                new_files += [files2[idx]]
                ogle[subclass] += 1
                foundOgle = True

            # VVV
            if not foundVista and vvv[subclass] < limit and idx < len(files3) and subclass in files3[idx]:
                new_files += [files3[idx]]
                vvv[subclass] += 1
                foundVista = True

    del files1, files2, files3

    print('[!] Loaded Files')

    files = np.array(new_files)

    # Permutation
    for i in range(10):
        files = files[np.random.permutation(len(files))]

    # Class Names
    Y = np.array([get_name(i) for i in files])
    mask = np.where(~(Y == 'err'))
    Y, files = Y[mask], files[mask]

    return files, Y

def load_dic(files, Y):

    df = pd.DataFrame([])
    new_y = []
    new_subclass = []
    survey = []
    for idx, i in enumerate(tqdm(files)):

        dic = np.load(i, encoding='latin1').item()

        ############################
        ### Eliminate Inf Values ###
        ############################

        df_aux = pd.DataFrame.from_dict(dic, orient='index')
        if len(np.where(np.isinf(df_aux))[0]) == 0:
            if len(df) == 0:
                df = df_aux.T
            else:
                df = df.append(df_aux.T)
            new_y.append(get_name_with_survey(i))
            new_subclass.append( Y[idx])
            survey.append(get_survey(i))

    return df, np.array(new_y), np.array(new_subclass), np.array(survey)

files, Y = get_files(permutation)
dic, Y, ySubClass, survey = load_dic(files[:n], Y[:n])

########################
## Replace Nan Values ##
########################

dic = dic.apply(lambda x: x.fillna(x.mean()), axis=0)
  0%|          | 0/143771 [00:00<?, ?it/s]

# Random Forrest_____no_output_____

yReal = []
yPred = []
sReal = []

X = dic.values
skf = StratifiedKFold(n_splits=folds, shuffle =True)
for train_index, test_index in skf.split(X, Y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = ySubClass[train_index], ySubClass[test_index]
    s_test = survey[test_index]

    ########################
    #### Random Forrest ####
    ########################

    clf = RandomForestClassifier(min_samples_leaf=100, random_state=0)
    clf.fit(X_train, y_train)

    yPred = np.append(yPred, clf.predict(X_test))
    yReal = np.append(yReal, y_test)
    sReal = np.append(sReal, s_test)/Users/Carlos/tensorflow/lib/python3.6/site-packages/sklearn/model_selection/_split.py:581: Warning: The least populated class in y has only 1 members, which is too few. The minimum number of groups for any class cannot be less than n_splits=10.
  % (min_groups, self.n_splits)), Warning)
directory = './Resultados'
if not os.path.exists(directory):
    print('[+] Creando Directorio \n\t ->', directory)
    os.mkdir(directory)

filename_exp =  directory + '/data'
print('\n \t\t\t [+] Saving Results in', filename_exp)
np.save(filename_exp, [yReal, yPred, sReal])[+] Creando Directorio
	 -> ./Resultados

 			 [+] Saving Results in ./Resultados/data


# Notebook from byrne0319/2019-CS109A
Path: GhostNorm_233077_8729764_cs109a_HW4.ipynb

#  CS109A Introduction to Data Science

## Homework 4: Logistic Regression

**Harvard University**

**Fall 2019**

**Instructors**: Pavlos Protopapas, Kevin Rader, and Chris Tanner

_____no_output_____

#RUN THIS CELL
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)_____no_output_____

### INSTRUCTIONS

- **This is an individual homework. No group collaboration.**
- To submit your assignment follow the instructions given in Canvas.
- Restart the kernel and run the whole notebook again before you submit.
- As much as possible, try and stick to the hints and functions we import at the top of the homework, as those are the ideas and tools the class supports and is aiming to teach. And if a problem specifies a particular library you're required to use that library, and possibly others from the import list.
- Please use .head() when viewing data. Do not submit a notebook that is excessively long because output was not suppressed or otherwise limited. _____no_output_____

import numpy as np
import pandas as pd

from sklearn.linear_model import LinearRegression
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import LogisticRegressionCV
from sklearn.linear_model import LassoCV

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.metrics import accuracy_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

import zipfile

import seaborn as sns
sns.set()
_____no_output_____

 Cancer Classification from Gene Expressions

In this problem, we will build a classification model to distinguish between two related classes of cancer, acute lymphoblastic leukemia (ALL) and acute myeloid leukemia (AML), using gene expression measurements. The data set is provided in the file `data/dataset_hw4.csv`. Each row in this file corresponds to a tumor tissue sample from a patient with one of the two forms of Leukemia. The first column contains the cancer type, with **0 indicating the ALL** class and **1 indicating the AML** class. Columns 2-7130 contain expression levels of 7129 genes recorded from each tissue sample.

In the following questions, we will use linear and logistic regression to build classification models for this data set.
_____no_output_____
 Question 1 [20 pts]: Data Exploration

First step is to  split  the observations into an approximate 80-20 train-test split.  Below is some code to do this for you (we want to make sure everyone has the same splits). Print dataset shape before splitting and after splitting. `Cancer_type` is our target column.

**1.1** Take a peek at your training set (show a glimpse of what you did): you should notice the severe differences in the measurements from one gene to the next (some are negative, some hover around zero, and some are well into the thousands).  To account for these differences in scale and variability, normalize each predictor to vary between 0 and 1.

**1.2** The training set contains more predictors than observations. What problem(s) can this lead to in fitting a classification model to such a data set? Explain in 3 or fewer sentences.

**1.3** Identify and report which 10 genes individually discriminate between the two cancer classes the best (consider every gene in the data set).  Note: it wil lbe useful to save this list for future parts.

Plot two histograms ofyour best predictor - one using training and another for the testing dataset. Each histogram should clearly distinguish two different `Cancer_type` classes.

Hint: You may use t-testing to make this determination: #https://en.wikipedia.org/wiki/Welch%27s_t-test.

**1.4** Using your top gene from the previous part (call it  `best_predictor`), create a classification model by manually eye-balling a value for this gene that would discriminate the two classes the best. Justify your choice in 1-2 sentences. Report the accuracy of this hand-chosen model on the test set.

_____no_output_____

### Solutions_____no_output_____**First step is to split the observations into an approximate 80-20 train-test split. Below is some code to do this for you (we want to make sure everyone has the same splits). Print dataset shape before splitting and after splitting. `Cancer_type` is our target column.**_____no_output_____

np.random.seed(10)
df = pd.read_csv('data/hw4_enhance.csv', index_col=0)

X_train, X_test, y_train, y_test =train_test_split(df.loc[:, df.columns != 'Cancer_type'],
                                                         df.Cancer_type, test_size=0.2,
                                                         random_state = 109,
                                                         stratify = df.Cancer_type)_____no_output_____
print(df.shape)

print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
print(df.Cancer_type.value_counts(normalize=True))
(752, 7130)
((601, 7129), (151, 7129), (601L,), (151L,))
0.0    0.511968
1.0    0.488032
Name: Cancer_type, dtype: float64

**1.1 Take a peek at your training set (show a glimpse of what you did): you should notice the severe differences in the measurements from one gene to the next (some are negative, some hover around zero, and some are well into the thousands).  To account for these differences in scale and variability, normalize each predictor to vary between 0 and 1.**
_____no_output_____

#your code here
display(X_train.head())_____no_output_____#your code here
scaler=MinMaxScaler().fit(X_train)  ##https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html

X_train_scaled=pd.DataFrame(scaler.transform(X_train),index=X_train.index,columns=X_train.columns)
X_test_scaled=pd.DataFrame(scaler.transform(X_test),index=X_test.index,columns=X_test.columns)

X_train_scaled.head()_____no_output_____

**1.2 The training set contains more predictors than observations. What problem(s) can this lead to in fitting a classification model to such a data set? Explain in 3 or fewer sentences.**_____no_output_____- When a training set contains more predictors than observations we cannot use the Least Square Regression. Therefore, in such cases we need to use regularization methods and introduce penalty terms
_____no_output_____**1.3** **Identify and report which 10 genes individually discriminate between the two cancer classes the best (consider every gene in the data set).  Note: it wil lbe useful to save this list for future parts.**

**Plot two histograms of your best predictor - one using training and another for the testing dataset. Each histogram should clearly distinguish two different `Cancer_type` classes.**

**Hint: You may use t-testing to make this determination: #https://en.wikipedia.org/wiki/Welch%27s_t-test.**_____no_output_____

Cancer0 = df[df['Cancer_type']==0]
Cancer1 = df[df['Cancer_type']==1]
All_Genes = df.columns[1:]

ttest_List = []

for i in range(len(All_Genes)-1):
    ind_gene = All_Genes[i]
    mean_diff = Cancer0[ind_gene].mean() - Cancer1[ind_gene].mean()
    C0_var = Cancer0[ind_gene].std()**2/Cancer0[ind_gene].count()
    C1_var = Cancer1[ind_gene].std()**2/Cancer1[ind_gene].count()
    denominator = np.sqrt(C0_var + C1_var)
    ttest = mean_diff/denominator

    ttest_List.append(ttest)
#ttest_List_____no_output_____top10_gene_index = np.argsort(ttest_List)[-10:]
top10_gene = [X_train.columns[i] for i in top10_gene_index]

top_gene_index = np.argsort(ttest_List)[-1:]
top_gene = X_train.columns[top_gene_index][0]_____no_output_____#your code here
#logist_train_model1 = LogisticRegression(C=100000).fit(X_train_scaled,y_train)
#model1_abs_coef = np.abs(logist_train_model1.coef_)

#top10_gene_index = np.argsort(model1_abs_coef)[0][-10:]
#top10_gene = [X_train.columns[i] for i in top10_gene_index]

#top_gene_index = np.argsort(model1_abs_coef)[0][-1:]
#top_gene = X_train.columns[top_gene_index][0]_____no_output_____#your code here
print("Top Gene Predictor: \t{}\n".format(top_gene))
print("Top 10 Gene Predictors: \n{}".format(top10_gene));
Top Gene Predictor: 	M31523_at

Top 10 Gene Predictors:
['U72936_s_at', 'M31211_s_at', 'X80230_at', 'U29175_at', 'X53742_at', 'J04615_at', 'X76648_at', 'J04029_s_at', 'Y08612_at', 'M31523_at']
#0 indicating the ALL class and 1 indicating the AML

X_train_df = pd.DataFrame(list(zip(X_train[top_gene], y_train)),columns =[top_gene, 'Cancer_type'])
TopGene_Cancer0_train = X_train_df[(X_train_df.Cancer_type == 0)][top_gene]
TopGene_Cancer1_train = X_train_df[(X_train_df.Cancer_type == 1)][top_gene]

X_test_df = pd.DataFrame(list(zip(X_test[top_gene], y_test)),columns =[top_gene, 'Cancer_type'])
TopGene_Cancer0_test = X_test_df[(X_test_df.Cancer_type == 0)][top_gene]
TopGene_Cancer1_test = X_test_df[(X_test_df.Cancer_type == 1)][top_gene]

f,ax = plt.subplots(1,2, figsize=(14,6))
ax[0].hist(TopGene_Cancer0_train, alpha=0.5, label = 'Cancer0_ALL');
ax[0].hist(TopGene_Cancer1_train, alpha=0.5, label = 'Cancer1_AML');
ax[0].set_title('Train Data')
ax[0].set_xlabel(top_gene)
ax[0].set_ylabel('Frequency')
ax[0].legend()

ax[1].hist(TopGene_Cancer0_test, alpha=0.5, label = 'Cancer0_ALL');
ax[1].hist(TopGene_Cancer1_test, alpha=0.5, label = 'Cancer1_AML');
ax[1].set_title('Test Data')
ax[1].set_xlabel(top_gene)
ax[1].set_ylabel('Frequency')
ax[1].legend()_____no_output_____# Using Scaled Data
X_train_df = pd.DataFrame(list(zip(X_train_scaled[top_gene], y_train)),columns =[top_gene, 'Cancer_type'])
TopGene_Cancer0_train = X_train_df[(X_train_df.Cancer_type == 0)][top_gene]
TopGene_Cancer1_train = X_train_df[(X_train_df.Cancer_type == 1)][top_gene]

X_test_df = pd.DataFrame(list(zip(X_test_scaled[top_gene], y_test)),columns =[top_gene, 'Cancer_type'])
TopGene_Cancer0_test = X_test_df[(X_test_df.Cancer_type == 0)][top_gene]
TopGene_Cancer1_test = X_test_df[(X_test_df.Cancer_type == 1)][top_gene]

f,ax = plt.subplots(1,2, figsize=(14,6))
ax[0].hist(TopGene_Cancer0_train, alpha=0.5, label = 'Cancer0_ALL');
ax[0].hist(TopGene_Cancer1_train, alpha=0.5, label = 'Cancer1_AML');
ax[0].set_title('Train Data')
ax[0].set_xlabel(top_gene)
ax[0].set_ylabel('Frequency')
ax[0].legend()

ax[1].hist(TopGene_Cancer0_test, alpha=0.5, label = 'Cancer0_ALL');
ax[1].hist(TopGene_Cancer1_test, alpha=0.5, label = 'Cancer1_AML');
ax[1].set_title('Test Data')
ax[1].set_xlabel(top_gene)
ax[1].set_ylabel('Frequency')
ax[1].legend()_____no_output_____

**1.4 Using your top gene from the previous part (call it  `best_predictor`), create a classification model by eye-balling a value for this gene that would discriminate the two classes the best. Justify your choice in 1-2 sentences. Report the accuracy of this hand-chosen model on the test set.**
_____no_output_____

#your code here
best_predictor = top_gene

X_test_df['y_pred_test'] = ((X_test_df[top_gene] >= 0) * (X_test_df[top_gene] <= 0.5)).astype(float)
test_score = accuracy_score(y_test, X_test_df['y_pred_test'])*100

print("Training Set Accuracy: {}% \n".format(test_score))

print("It appears that scaled values between 0 and 0.5 of the {} gene predicts the existace of Cancer1 (AML)."
       "Though it may not be wise to make this predictions based on one gene, if we have are given only this piece"
      "of information it's reasonable make these assumptions based on the probability at each gene level".format(top_gene))Training Set Accuracy: 66.8874172185%

It appears that scaled values between 0 and 0.5 of the M31523_at gene predicts the existace of Cancer1 (AML).Though it may not be wise to make this predictions based on one gene, if we have are given only this pieceof information it's reasonable make these assumptions based on the probability at each gene level

 Question 2 [25 pts]: Linear and Logistic Regression

In class we discussed how to use both linear regression and logistic regression for classification. For this question, you will work with a single gene predictor that you identify as the best predictor above to explore these two methods.

**2.1** Fit a simple linear regression model to the training set using the single gene predictor `best_predictor` to predict cancer type. The scores predicted by the regression model for a patient could be interpreted as an estimate of the probability that the patient has Cancer_type=1 (AML). Is there a problem with this interpretation?

Create a figure with following items displayed on the same plot (use training data):
 - the predicted quantitative response from the linear regression model as a function of the best gene predictor
 - the true binary response.

**2.2** Use your estimated linear regression model to classify observations into 0 and 1 using the standard Bayes' classifier.  Evaluate the classification accuracy of this classification model on both the training and test sets.

**2.3** Next, fit a simple logistic regression model to the training set. How do the training and test classification accuracies of this model compare with the linear regression model?

Remember, you need to set the regularization parameter for sklearn's logistic regression function to be a very large value in order to **not** regularize (use 'C=100000').

**2.4**
Print and interpret the logistic regression coefficients: the 'slope' and intercept.

Create 2 plots (one each for training and testing data) with 4 items displayed on each plot.
- the predicted quantitative response from the linear regression model as a function of the best gene predictor.
- the predicted probabilities of the logistic regression model as a function of the best gene predictor.
- the true binary response.
- a horizontal line at $y=0.5$.

Based on these plots, does one of the models appear better suited for binary classification than the other?  Explain in 3 sentences or fewer.

_____no_output_____

### Solutions_____no_output_____**2.1** **Fit a simple linear regression model to the training set using the single gene predictor `best_predictor` to predict cancer type. The scores predicted by the regression model for a patient could be interpreted as an estimate of the probability that the patient has Cancer_type=1 (AML).  Is there a problem with this interpretation?**

_____no_output_____

import statsmodels.api as sm_____no_output_____# your code here

X_train_cst = sm.add_constant(X_train_scaled[best_predictor])
X_test_cst = sm.add_constant(X_test_scaled[best_predictor])

model_linear_sklearn = LinearRegression()
model_linear_sklearn.fit(X_train_cst, y_train)

y_pred_train = model_linear_sklearn.predict(X_train_cst)
y_pred_test = model_linear_sklearn.predict(X_test_cst)
_____no_output_____# your code here
#the predicted quantitative response from the linear regression model as a function of the best gene predictor
#the true binary response

plt.scatter(X_train_scaled[best_predictor], y_pred_train, label = 'Predicted Y value')
plt.scatter(X_train_scaled[best_predictor], y_train, label = 'True Response')
plt.xlabel(best_predictor)
plt.ylabel("Cancer Type")
plt.title("Training Data")
plt.legend();
_____no_output_____plt.hist(y_pred_train, alpha=0.6, label = 'Predicted Y value')
plt.xlabel(best_predictor)
plt.ylabel("Frequency")
plt.title("Training Data")
plt.legend();_____no_output_____

- The limitation of using the predicted y values as the pprobability that patient has cancer is that there's no clear cut off point. Furthermore, since the existance of the cancer doesn't solely depend on this gene. Therefore, there's a chance that even though the predicted y value indicates that the probability of this patient having cancer as 0.3, However in reality the cancer can be present in that patient
_____no_output_____**2.2** **Use your estimated linear regression model to classify observations into 0 and 1 using the standard Bayes' classifier.  Evaluate the classification accuracy of this classification model on both the training and test sets.**_____no_output_____

# your code here
classfy_train = (y_pred_train > 0.5).astype(int)
classfy_test = (y_pred_test > 0.5).astype(int)

print("Accuracy for Training Data\t: {}".format(accuracy_score(y_train, classfy_train) * 100))
print("Accuracy for Testing Data\t: {}".format(accuracy_score(y_test, classfy_test) * 100))Accuracy for Training Data	: 70.8818635607
Accuracy for Testing Data	: 68.8741721854

**2.3** **Next, fit a simple logistic regression model to the training set. How do the training and test classification accuracies of this model compare with the linear regression model? Are the classifications substantially different?  Explain why this is the case.**

**Remember, you need to set the regularization parameter for sklearn's logistic regression function to be a very large value in order to **not** regularize (use 'C=100000')._____no_output_____

# your code here
Log_model1 = LogisticRegression(C=100000).fit(X_train_cst,y_train)

Log_model1_pred_train = Log_model1.predict(X_train_cst)
Log_model1_pred_test = Log_model1.predict(X_test_cst)

print("Logistic Regression")
print("Accuracy for Training Data\t: {}".format(accuracy_score(y_train, Log_model1_pred_train) * 100))
print("Accuracy for Testing Data\t: {}".format(accuracy_score(y_test, Log_model1_pred_test) * 100))

print("\nLinear Regression")
print("Accuracy for Training Data\t: {}".format(accuracy_score(y_train, classfy_train) * 100))
print("Accuracy for Testing Data\t: {}".format(accuracy_score(y_test, classfy_test) * 100))Logistic Regression
Accuracy for Training Data	: 70.7154742097
Accuracy for Testing Data	: 70.8609271523

Linear Regression
Accuracy for Training Data	: 70.8818635607
Accuracy for Testing Data	: 68.8741721854

- The accuracy scores from both models are extremely similar. This is because we are using only one predictor to perform the model. Therefore, our linear estimates are pretty much the same
_____no_output_____**2.4 Print and interpret the logistic regression coefficients: the 'slope' and the intercept.**

**Create 2 plots (with training and test data) with 4 items displayed on each plot.**
- the predicted quantitative response from the linear regression model as a function of the best gene predictor.
- the predicted probabilities of the logistic regression model as a function of the best gene predictor.
- the true binary response.
- a horizontal line at $y=0.5$.

**Based on these plots, does one of the models appear better suited for binary classification than the other?  Explain in 3 sentences or fewer.**
_____no_output_____

print("Logistic Regression")
print("Intercept\t: {}".format(Log_model1.coef_[0][0]))
print("Slope\t\t: {}".format(Log_model1.coef_[0][1]))Logistic Regression
Intercept	: 2.03863636916
Slope		: -8.75711199029
# your code here

f,ax = plt.subplots(1,2, figsize=(14,6))

#the predicted quantitative response from the linear regression model as a function of the best gene predictor.
ax[0].plot(X_train_scaled[best_predictor], y_pred_train, c="b", label = 'Linear regression Prediction')
ax[1].plot(X_test_scaled[best_predictor], y_pred_test, c="b", label = 'Linear regression Prediction')

#the predicted probabilities of the logistic regression model as a function of the best gene predictor.
Log_model1_predproba_train = Log_model1.predict_proba(X_train_cst)[:,1]
Log_model1_predproba_test = Log_model1.predict_proba(X_test_cst)[:,1]

ax[0].scatter(X_train_scaled[best_predictor], Log_model1_predproba_train, c="g", alpha = 0.5, label = 'Logistic regression Prediction')
ax[1].scatter(X_test_scaled[best_predictor], Log_model1_predproba_test, c="g", alpha = 0.5, label = 'Logistic regression Prediction')

#the true binary response.
ax[0].scatter(X_train_scaled[best_predictor], y_train, c="orange")
ax[1].scatter(X_test_scaled[best_predictor], y_test, c="orange")

#a horizontal line at  𝑦=0.5 .
ax[0].axhline(y=.5, c="r")
ax[1].axhline(y=.5, c="r")

# labels
ax[0].set_title("Train Set using " + best_predictor + " gene")
ax[0].set_xlabel(best_predictor + "expression level")
ax[0].set_ylabel("Cancer type")
ax[0].legend();

ax[1].set_title("Test Set using " + best_predictor + " gene")
ax[1].set_xlabel(best_predictor + "expression level")
ax[1].set_ylabel("Cancer type")
ax[1].legend();_____no_output_____

- Though Both the regressions appear to be extremely similar Logistic regression might be the best option because it has a boundary of 0-1._____no_output_____
  Question 3 [20pts]: Multiple Logistic Regression

**3.1** Next, fit a multiple logistic regression model with all the gene predictors from the data set.  How does the classification accuracy of this model compare with the models fitted in question 2 with a single gene (on both the training and test sets)?

**3.2** How many of the coefficients estimated by this multiple logistic regression in the previous part are significantly different from zero at a *significance level of 5%*? Use the same value of C=100000 as before.

**Hint:** To answer this question, use *bootstrapping* with 100 bootstrap samples/iterations.

**3.3** Comment on the classification accuracy of training and test set? Given the results above how would you assess the generalization capacity of your trained model? What other tests would you suggest to better guard against false sense of security on the accuracy of the model as a whole?

**3.4** Now use regularization to improve predictions from the multiple logistic regression model.  Use LASSO-like regularization and cross-validation within the training set to tune the model.  Report the classification accuracy on both the training and test set.

**3.5** Do the 10 best predictors from Q1 hold up as important features in this regularized model?  If not, explain why this is the case (feel free to use the data to support your explanation)._____no_output_____

### Solutions_____no_output_____**3.1** **Next, fit a multiple logistic regression model with all the gene predictors from the data set.  How does the classification accuracy of this model compare with the models fitted in question 2 with a single gene (on both the training and test sets)?**
_____no_output_____

# your code here
X_train_cst = sm.add_constant(X_train)
X_test_cst = sm.add_constant(X_test)

Multi_Logistic_model = LogisticRegression(C=100000).fit(X_train_cst, y_train)

Multi_Logistic_pred_train = Multi_Logistic_model.predict(X_train_cst)
Multi_Logistic_pred_test = Multi_Logistic_model.predict(X_test_cst)

print("Multiple Logistic Regression")
print("Accuracy for Training Data\t: {}".format(accuracy_score(y_train, Multi_Logistic_pred_train) * 100))
print("Accuracy for Testing Data\t: {}".format(accuracy_score(y_test, Multi_Logistic_pred_test) * 100))

print("\nLogistic Regression")
print("Accuracy for Training Data\t: {}".format(accuracy_score(y_train, Log_model1_pred_train) * 100))
print("Accuracy for Testing Data\t: {}".format(accuracy_score(y_test, Log_model1_pred_test) * 100))Multiple Logistic Regression
Accuracy for Training Data	: 100.0
Accuracy for Testing Data	: 74.1721854305

Logistic Regression
Accuracy for Training Data	: 70.7154742097
Accuracy for Testing Data	: 70.8609271523

- Multiple logistic regression shows a significantly higher Accuracy score than the logistic regression performed with the single gene. However, this could be due to over fitting
_____no_output_____**3.2** **How many of the coefficients estimated by this multiple logistic regression in the previous part are significantly different from zero at a *significance level of 5%*? Use the same value of C=100000 as before.**

**Hint:** **To answer this question, use *bootstrapping* with 1000 bootstrap samples/iterations.**
_____no_output_____

train_data = pd.concat([X_train,y_train], axis=1)_____no_output_____# your code here
model = LogisticRegression(C=100000).fit(X_train, y_train)
bootstrapping = 100
coef = np.zeros((bootstrapping, train_data.shape[1]-1))

for i in range(bootstrapping):
    coef[i,:] = model.coef_

coef_count = 0
for i in range(coef.shape[1]):
    coeff_samples = coef[:,i]
    lower_bound = np.percentile(coeff_samples, 2.5)
    upper_bound = np.percentile(coeff_samples, 97.5)
    if lower_bound>0 or upper_bound<0:
        coef_count += 1

print('Number coefficients different from zero at a significance level of 5%\t: {}'.format(coeffs_count))Number coefficients different from zero at a significance level of 5%	: 546

**3.3 Open question: Comment on the classification accuracy of training and test set? Given the results above how would you assest the generalization capacity of your trained model? What other tests would you suggest to better guard against false sense of security on the accuracy of the model as a whole.**_____no_output_____- When performing Multiple logicstic Regression, we reached 100% accuracy with the training set. This suggest that we may have overfit the model. If we have more independent dataset we can run this model against that data to guage the accuracy. However, in order to better guard against the false sense of security on accuracy, we can plot the ROC curve_____no_output_____**3.4 Now use regularization to improve predictions from the multiple logistic regression model.  Use LASSO-like regularization and cross-validation within the training set to tune the model.  Report the classification accuracy on both the training and test set.**_____no_output_____

# your code here
Log_RegCV=LogisticRegressionCV(multi_class='auto', penalty='l1', solver='liblinear').fit(X_train_cst,y_train)
Log_RegCV_predproba_train = Log_RegCV.predict_proba(X_train_cst)[:,1]_____no_output_____Log_RegCV_pred_train = Log_RegCV.predict(X_train_cst)
Log_RegCV_pred_test = Log_RegCV.predict(X_test_cst)

print("Logistic Regression CV")
print("Accuracy for Training Data\t: {}".format(accuracy_score(y_train, Log_RegCV_pred_train) * 100))
print("Accuracy for Testing Data\t: {}".format(accuracy_score(y_test, Log_RegCV_pred_test) * 100))Logistic Regression CV
Accuracy for Training Data	: 100.0
Accuracy for Testing Data	: 76.821192053

**3.5: Do the 10 best predictors from Q1 hold up as important features in this regularized model?  If not, explain why this is the case (feel free to use the data to support your explanation).**_____no_output_____

# your code here
print("Top 10 Gene Predictors: \n{}".format(top10_gene));Top 10 Gene Predictors:
['U72936_s_at', 'M31211_s_at', 'X80230_at', 'U29175_at', 'X53742_at', 'J04615_at', 'X76648_at', 'J04029_s_at', 'Y08612_at', 'M31523_at']
# your code here
reg_top10_gene_index = np.argsort(np.abs(Log_RegCV.coef_[0]))[-10:]
reg_top10_gene_list = [X_train.columns[i] for i in reg_top10_gene_index]
print("Top 10 Gene Predictors using Regularized model: \n{}".format(reg_top10_gene_list));Top 10 Gene Predictors using Regularized model:
['HG881-HT881_at', 'M96980_at', 'HG4411-HT4681_at', 'M35296_at', 'X77794_at', 'L31881_at', 'X85753_at', 'M13232_s_at', 'U60325_at', 'X79204_at']
# your code here
priority_index_top10_gene = []
for i in top10_gene_index:
    x = len(Log_RegCV.coef_[0])-list(np.argsort(np.abs(Log_RegCV.coef_[0]))).index(i)
    priority_index_top10_gene.append(x)

print("Importance rating of the previously predicted Top10 genes with respect to the Regularized model : \n{}".format(priority_index_top10_gene));Importance rating of the previously predicted Top10 genes with respect to the Regularized model :
[2271, 5899, 2371, 2093, 6670, 922, 1857, 5740, 5060, 5553]
reg_top_gene_index = np.argsort(np.abs(Log_RegCV.coef_[0]))[-1:]
reg_top_gene_list = [X_train.columns[i] for i in reg_top_gene_index][0]
print("Top Gene Predictor using Regularized model: \t{}".format(reg_top_gene_list));Top Gene Predictor using Regularized model: 	X79204_at
X_train_df = pd.DataFrame(list(zip(X_train_scaled[reg_top_gene_list], y_train)),columns =[reg_top_gene_list, 'Cancer_type'])
TopGene_Cancer0_train = X_train_df[(X_train_df.Cancer_type == 0)][reg_top_gene_list]
TopGene_Cancer1_train = X_train_df[(X_train_df.Cancer_type == 1)][reg_top_gene_list]

X_test_df = pd.DataFrame(list(zip(X_test_scaled[reg_top_gene_list], y_test)),columns =[reg_top_gene_list, 'Cancer_type'])
TopGene_Cancer0_test = X_test_df[(X_test_df.Cancer_type == 0)][reg_top_gene_list]
TopGene_Cancer1_test = X_test_df[(X_test_df.Cancer_type == 1)][reg_top_gene_list]

f,ax = plt.subplots(1,2, figsize=(14,6))
ax[0].hist(TopGene_Cancer0_train, alpha=0.5, label = 'Cancer0_ALL');
ax[0].hist(TopGene_Cancer1_train, alpha=0.5, label = 'Cancer1_AML');
ax[0].set_title('Train Data')
ax[0].set_xlabel(reg_top_gene_list)
ax[0].set_ylabel('Frequency')
ax[0].legend()

ax[1].hist(TopGene_Cancer0_test, alpha=0.5, label = 'Cancer0_ALL');
ax[1].hist(TopGene_Cancer1_test, alpha=0.5, label = 'Cancer1_AML');
ax[1].set_title('Test Data')
ax[1].set_xlabel(reg_top_gene_list)
ax[1].set_ylabel('Frequency')
ax[1].legend();_____no_output_____

- Previously predicted top10 Gene predictors are not equivalent to the Top10 predictors obtained from the regularized model. In fact, importance rating of the previously predicted Top10 genes with respect to the resularized model indicated that none of the previous predectors falls with the top 100 predictors.
- Based on the accuracy scores of the initial model, I believe the regularized model predictors might actually be better at Cancer type prediction
- This might be due to making predictions using the linear regression model.
_____no_output_____
  Question 4 [25pts]: Multi Class Log Regression
_____no_output_____**4.1** Load the data from `hw4_mc_enhance.csv.zip` and examine its structure. How many instances of each class are there in our dataset?

**4.2** Split the dataset into train and test, 80-20 split, random_state = 8.

We are going to utilize these two features - 'M31523_at', 'X95735_at'.  Create a scatter plot of these two features using the training dataset.  It should be easily discernable via labeling/marking in the plot which observations belong to which `cancer_type`.

**4.3** Fit the following two models using crossvalidation:
- Logistic Regression Multiclass model with linear features.
- Logistic Regression Multiclass model with Polynomial features, degree = 2.

**4.4** Plot the decision boundaries for each model and interpret the results. Hint: You may utilize the function `overlay_decision_boundary`.

**4.5** Report and plot the CV scores for the two models and interpret.
_____no_output_____

### Solutions_____no_output_____**4.1 Load the data from `hw4_mc_enhance.csv.zip` and examine its structure. How many instances of each class are there in our dataset?**_____no_output_____

#your code here
mc_data = pd.read_csv('data/hw4_mc_enhance.csv', index_col=0)
mc_data.head()_____no_output_____#your code here
mc_data.describe()
mc_data_isna_count = pd.DataFrame(mc_data.isna().sum())

print("Count of Null entries in the dataset:")
mc_data_isna_count[mc_data_isna_count[0]>0].count()Count of Null entries in the dataset:
display(mc_data.shape)
mc_data.groupby('cancer_type').count()_____no_output_____

**4.2 Split the dataset into train and test, 80-20 split, random_state = 8.**

**We are going to utilize these two features - 'M31523_at', 'X95735_at'.  Create a scatter plot of these two features using training dataset.  We should be able to discern from the plot which sample belongs to which `cancer_type`.** _____no_output_____

# your code here
#train_data, test_data = train_test_split(mc_data, test_size=.2, random_state=8)
subset_data = pd.concat([mc_data['M31523_at'],mc_data['X95735_at'],mc_data['cancer_type']], axis=1)

X_train, X_test, y_train, y_test =train_test_split(subset_data.loc[:, subset_data.columns != 'cancer_type'],
                                                         subset_data.cancer_type, test_size=0.2,
                                                         random_state = 8,
                                                         stratify = subset_data.cancer_type)_____no_output_____def create_scatterplot(dataset):
    ax=plt.gca()
    ax.set_xlabel("M31523_at")
    ax.set_ylabel("X95735_at")
    for i in range(0,4):
        dataset_i = dataset[dataset['cancer_type']==i]
        ax.scatter(dataset_i["M31523_at"],dataset_i["X95735_at"])
    ax.legend(["0", "1", "2"],loc="right")
    return ax_____no_output_____# your code here
f,ax = plt.subplots(1,2, figsize=(14,6))
ax[0].scatter(X_train['M31523_at'],y_train, alpha=0.5, label = 'M31523_at')
ax[0].scatter(X_train['X95735_at'],y_train, alpha=0.5, label = 'X95735_at')
ax[0].set_title('Train Data')
ax[0].set_xlabel('Gene Level')
ax[0].set_ylabel('Cancer Type')
ax[0].legend()

xy_train = pd.concat([X_train['M31523_at'],X_train['X95735_at'],y_train], axis=1)
#ax[1] = sns.scatterplot(x="M31523_at", y="X95735_at", hue=xy_train['cancer_type'], data=xy_train)
ax[1] = create_scatterplot(xy_train)
ax[1].set_title('Train Data');

#legend = ax[1].legend_
#labels=['0', '1', '2','2']
#for i, label in enumerate(labels):
#    legend.get_texts()[i+1].set_text(label)_____no_output_____#sns.pairplot(xy_train, hue = 'cancer_type')
g  = sns.pairplot(xy_train, x_vars=["M31523_at"], y_vars=["X95735_at"], hue = 'cancer_type')
g.fig.set_size_inches(8,6)_____no_output_____

**4.3  Fit the following two models using crossvalidation:**

**Logistic Regression Multiclass model with linear features.**

**Logistic Regression Multiclass model with Polynomial features, degree = 2.**
_____no_output_____

# your code here
scaler=MinMaxScaler().fit(X_train)
X_train_scaled=pd.DataFrame(scaler.transform(X_train),index=X_train.index,columns=X_train.columns)
X_test_scaled=pd.DataFrame(scaler.transform(X_test),index=X_test.index,columns=X_test.columns)

X_train_cst = sm.add_constant(X_train_scaled)
X_test_cst = sm.add_constant(X_test_scaled)

Multi_Log_RegCV=LogisticRegressionCV().fit(X_train_cst,y_train)
Multi_Log_RegCV_predproba_train = Multi_Log_RegCV.predict_proba(X_train_cst)[:,1]

Multi_Log_RegCV_pred_train = Multi_Log_RegCV.predict(X_train_cst)
Multi_Log_RegCV_pred_test = Multi_Log_RegCV.predict(X_test_cst)

print("Multiclass Logistic Regression CV")
print("Accuracy for Training Data\t: {}".format(accuracy_score(y_train, Multi_Log_RegCV_pred_train) * 100))
print("Accuracy for Testing Data\t: {}".format(accuracy_score(y_test, Multi_Log_RegCV_pred_test) * 100))Multiclass Logistic Regression CV
Accuracy for Training Data	: 84.8333333333
Accuracy for Testing Data	: 86.0
# your code here
X_train_poly_cst=X_train_cst.copy()
X_train_poly_cst['M31523_at^2'] = X_train_cst['M31523_at']**2
X_train_poly_cst['X95735_at^2'] = X_train_cst['X95735_at']**2

X_test_poly_cst=X_test_cst.copy()
X_test_poly_cst['M31523_at^2'] = X_test_poly_cst['M31523_at']**2
X_test_poly_cst['X95735_at^2'] = X_test_poly_cst['X95735_at']**2

Multi_Log_RegCV_poly=LogisticRegressionCV().fit(X_train_poly_cst,y_train)
Multi_Log_RegCV_poly_predproba_train = Multi_Log_RegCV_poly.predict_proba(X_train_poly_cst)[:,1]

Multi_Log_RegCV_poly_pred_train = Multi_Log_RegCV_poly.predict(X_train_poly_cst)
Multi_Log_RegCV_poly_pred_test = Multi_Log_RegCV_poly.predict(X_test_poly_cst)

print("Multiclass Logistic Regression CV with Polynomial features, degree = 2")
print("Accuracy for Training Data\t: {}".format(accuracy_score(y_train, Multi_Log_RegCV_poly_pred_train) * 100))
print("Accuracy for Testing Data\t: {}".format(accuracy_score(y_test, Multi_Log_RegCV_poly_pred_test) * 100))Multiclass Logistic Regression CV with Polynomial features, degree = 2
Accuracy for Training Data	: 88.8333333333
Accuracy for Testing Data	: 90.6666666667
# your code here
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

polynomial_logreg_estimator = make_pipeline(
    PolynomialFeatures(degree=2, include_bias=False),
    LogisticRegression())

X_train_poly_cst=X_train_cst.copy()
X_train_poly_cst['M31523_at^2'] = X_train_cst['M31523_at']**2
X_train_poly_cst['X95735_at^2'] = X_train_cst['X95735_at']**2

X_test_poly_cst=X_test_cst.copy()
X_test_poly_cst['M31523_at^2'] = X_test_poly_cst['M31523_at']**2
X_test_poly_cst['X95735_at^2'] = X_test_poly_cst['X95735_at']**2

#Multi_Log_RegCV_poly.fit(X_train_poly_cst, y_train)
Multi_Log_RegCV_poly = polynomial_logreg_estimator.fit(X_train_poly_cst, y_train)

Multi_Log_RegCV_poly_pred_train = Multi_Log_RegCV_poly.predict(X_train_poly_cst)
Multi_Log_RegCV_poly_pred_test = Multi_Log_RegCV_poly.predict(X_test_poly_cst)

print("Multiclass Logistic Regression CV with Polynomial features, degree = 2 using make_pipeline")
print("Accuracy for Training Data\t: {}".format(accuracy_score(y_train, Multi_Log_RegCV_poly_pred_train) * 100))
print("Accuracy for Testing Data\t: {}".format(accuracy_score(y_test, Multi_Log_RegCV_poly_pred_test) * 100))Multiclass Logistic Regression CV with Polynomial features, degree = 2 using make_pipeline
Accuracy for Training Data	: 83.1666666667
Accuracy for Testing Data	: 84.6666666667

**4.4 Plot the decision boundary and interpret results. Hint: You may utilize the function `overlay_decision_boundary`**
_____no_output_____

def overlay_decision_boundary(ax, model, colors=None, nx=200, ny=200, desaturate=.5, xlim=None, ylim=None):
    """
    A function that visualizes the decision boundaries of a classifier.

    ax: Matplotlib Axes to plot on
    model: Classifier to use.
     - if `model` has a `.predict` method, like an sklearn classifier, we call `model.predict(X)`
     - otherwise, we simply call `model(X)`
    colors: list or dict of colors to use. Use color `colors[i]` for class i.
     - If colors is not provided, uses the current color cycle
    nx, ny: number of mesh points to evaluated the classifier on
    desaturate: how much to desaturate each of the colors (for better contrast with the sample points)
    xlim, ylim: range to plot on. (If the default, None, is passed, the limits will be taken from `ax`.)
    """
    # Create mesh.
    xmin, xmax = ax.get_xlim() if xlim is None else xlim
    ymin, ymax = ax.get_ylim() if ylim is None else ylim
    xx, yy = np.meshgrid(
        np.linspace(xmin, xmax, nx),
        np.linspace(ymin, ymax, ny))
    X = np.c_[xx.flatten(), yy.flatten()]

    # Predict on mesh of points.
    model = getattr(model, 'predict', model)
    y = model(X)
    #print("Do I predict" , y)

#     y[np.where(y=='aml')]=3
#     y[np.where(y=='allT')]=2
#     y[np.where(y=='allB')]=1

    y = y.astype(int) # This may be necessary for 32-bit Python.
    y = y.reshape((nx, ny))

    # Generate colormap.
    if colors is None:
        # If colors not provided, use the current color cycle.
        # Shift the indices so that the lowest class actually predicted gets the first color.
        # ^ This is a bit magic, consider removing for next year.
        colors = (['white'] * np.min(y)) + sns.utils.get_color_cycle()

    if isinstance(colors, dict):
        missing_colors = [idx for idx in np.unique(y) if idx not in colors]
        #assert len(missing_colors) == 0, f"Color not specified for predictions {missing_colors}"

        # Make a list of colors, filling in items from the dict.
        color_list = ['white'] * (np.max(y) + 1)
        for idx, val in colors.items():
            color_list[idx] = val
    else:
        assert len(colors) >= np.max(y) + 1, "Insufficient colors passed for all predictions."
        color_list = colors
    color_list = [sns.utils.desaturate(color, desaturate) for color in color_list]
    cmap = matplotlib.colors.ListedColormap(color_list)

    # Plot decision surface
    ax.pcolormesh(xx, yy, y, zorder=-2, cmap=cmap, norm=matplotlib.colors.NoNorm(), vmin=0, vmax=y.max() + 1)
    xx = xx.reshape(nx, ny)
    yy = yy.reshape(nx, ny)
    if len(np.unique(y)) > 1:
        ax.contour(xx, yy, y, colors="black", linewidths=1, zorder=-1)
    else:
        print("Warning: only one class predicted, so not plotting contour lines.")_____no_output_____# your code here
Multi_Log_RegCV=LogisticRegressionCV().fit(x_train,y_train)

ax=create_scatterplot(xy_train)
ax.set_title("Logistic Linear Regression")
overlay_decision_boundary(ax, Multi_Log_RegCV)_____no_output_____# your code here
polynomial_logreg_estimator = make_pipeline(
    PolynomialFeatures(degree=2, include_bias=False),
    LogisticRegressionCV())

Multi_Log_RegCV_poly = polynomial_logreg_estimator.fit(x_train, y_train)

ax=create_scatterplot(xy_train)
ax.set_title("Logistic Regression with Polynomial Features")

overlay_decision_boundary(ax, Multi_Log_RegCV_poly)_____no_output_____

**4.5 Report and plot the CV scores for the two models and interpret.**_____no_output_____

print("Cross Validation Score of Linear Logistic Regression: \n {}"
      .format(cross_val_score(Multi_Log_RegCV, X_train_cst, y_train)))
print("\nCross Validation Score of Polynomial Logistic Regression: \n {}"
      .format(cross_val_score(Multi_Log_RegCV_poly, X_train_poly_cst, y_train)))Cross Validation Score of Linear Logistic Regression:
 [0.80597015 0.82089552 0.86868687]

Cross Validation Score of Polynomial Logistic Regression:
 [0.91542289 0.86567164 0.94444444]
# your code here
x_2_range  = np.arange(x_2_min, x_2_max, 0.003)

f,ax = plt.subplots(1,1, figsize=(6,6))
ax.hist(cross_val_score(Multi_Log_RegCV, X_train, y_train), label= 'Logistic Linear Regression - CV Score')
ax.hist(cross_val_score(Multi_Log_RegCV_poly, X_train_poly_cst, y_train), label= 'Logistic Polynomial Regression - CV Score')
ax.set_title('Train Set')
ax.legend();_____no_output_____

- Cross Validation scores of both models appear to be very similar. However, I failed to see the value in plotting the Cross Validation Scores_____no_output_____
 Question 5: [10 pts] Including an 'abstain' option

One of the reasons a hospital might be hesitant to use your cancer classification model is that a misdiagnosis by the model on a patient can sometimes prove to be very costly (e.g. if the patient were to file a law suit seeking a compensation for damages). One way to mitigate this concern is to allow the model to 'abstain' from making a prediction: whenever it is uncertain about the diagnosis for a patient. However, when the model abstains from making a prediction, the hospital will have to forward the patient to a specialist, which would incur additional cost.  How could one design a cancer classification model with an abstain option, such that the cost to the hospital is minimized?

*Hint:* Think of ways to build on top of the logistic regression model and have it abstain on patients who are difficult to classify._____no_output_____**5.1** More specifically, suppose the cost incurred by a hospital when a model mis-predicts on a patient is $\$5000$ , and the cost incurred when the model abstains from making a prediction is \$1000. What is the average cost per patient for the OvR logistic regression model (without quadratic or interaction terms) from **Question 4**.  Note that this needs to be evaluated on the patients in the test set. _____no_output_____**5.2** Design a classification strategy (into the 3 groups plus the *abstain* group) that has as low cost as possible per patient (certainly lower cost per patient than the logistic regression model).   Give a justification for your approach._____no_output_____

### Solutions_____no_output_____**5.1 More specifically, suppose the cost incurred by a hospital when a model mis-predicts on a patient is $\$5000$ , and the cost incurred when the model abstains from making a prediction is \$1000. What is the average cost per patient for the OvR logistic regression model (without quadratic or interaction terms) from Question 4.  Note that this needs to be evaluated on the patients in the test set.**
**
...
**_____no_output_____*your answer here*
_____no_output_____

from sklearn.metrics import confusion_matrix_____no_output_____# your code here
Multi_Log_RegCV=LogisticRegressionCV(multi_class = "ovr").fit(X_train_cst,y_train)
Multi_Log_RegCV_predproba_test = Multi_Log_RegCV.predict_proba(X_test_cst)[:,1]

Multi_Log_RegCV_pred_train = Multi_Log_RegCV.predict(X_train_cst)
Multi_Log_RegCV_pred_test = Multi_Log_RegCV.predict(X_test_cst)

confusion_matrix = pd.crosstab(  y_test.values.flatten(),
                                          Multi_Log_RegCV_pred_test.flatten(),
                                          rownames=['Actual Class'],
                                          colnames=['Predicted Class'])
display(confusion_matrix)

incorrectPred = len(y_test) - (confusion_matrix[0][0] + confusion_matrix[1][1] + confusion_matrix[2][2])

print("Using Test Data:")
print("Avg Cost per Patient using Accuracy Score\t: {}".format((1-(accuracy_score(y_test, Multi_Log_RegCV_pred_test))) * 5000))
print("Avg Cost per Patient using Confusion Matrix\t: {}".format(incorrectPred*5000/len(y_test)))_____no_output_____

**5.2 Design a classification strategy (into the 3 groups plus the *abstain* group) that has as low cost as possible per patient (certainly lower cost per patient than the logistic regression model).   Give a justification for your approach.**_____no_output_____

cost_abstain = 1000
cost_missPred = 5000
def cost_basedOn_prob(y_prob, y_values, p):
    y_thresh = []
    for prob in y_prob:
        if prob[0] >= p:
            y_thresh.append(1) #CancerType=0
        elif prob[1] >= p:
            y_thresh.append(2) #CancerType=1
        elif prob[2] >= p:
            y_thresh.append(3) #CancerType=2
        else:
            y_thresh.append(0) #abstain
    cost = 0
    for y_t, y in zip(y_thresh, y_values):
        if y_t == y:
            cost += 0
        elif y_t == 0:
            cost += cost_abstain
        else:
            cost += cost_missPred
    return cost / len(y_values) _____no_output_____def probability_threshold(model, x, y_true):
    min_avg_cost = np.infty
    y_prob = model.predict_proba(x)

    best_prob = 0.5
    p = 0.5
    while p <= 1:
        avg_cost = cost_basedOn_prob(y_prob, y_true, p)
        if avg_cost < min_avg_cost:
            min_avg_cost = avg_cost
            best_prob = p

        p += 0.1
    return best_prob, min_avg_cost_____no_output_____# your code here
Multi_Log_RegCV_predproba_test = Multi_Log_RegCV.predict_proba(X_test_cst)
best_probability, min_avg_cost = probability_threshold(Multi_Log_RegCV, X_test_cst, y_test)
AvgCost_usingLinearLog_test = cost_basedOn_prob(Multi_Log_RegCV_predproba_test, y_test, best_probability)

print("Min avg. Cost per Patient using Linear Log. Reg\t: {}".format(AvgCost_usingLinearLog_test))Min avg. Cost per Patient using Linear Log. Reg	: 666
Multi_Log_RegCV_poly=LogisticRegressionCV(multi_class = "ovr").fit(X_train_poly_cst,y_train)
Multi_Log_RegCV_poly_predproba_test = Multi_Log_RegCV_poly.predict_proba(X_test_poly_cst)

best_probability, min_avg_cost = probability_threshold(Multi_Log_RegCV_poly, X_test_poly_cst, y_test)
AvgCost_usingPolyLog_test = cost_basedOn_prob(Multi_Log_RegCV_poly_predproba_test, y_test, best_probability)
print("Min avg. Cost per Patient using Poly. Log. Reg\t: {}".format(AvgCost_usingPolyLog_test))Min avg. Cost per Patient using Poly. Log. Reg	: 666
kNN = KNeighborsClassifier(n_neighbors=144).fit(X_train_cst, y_train)
kNN_predproba_test = kNN.predict_proba(X_test_cst)

best_probability, min_avg_cost = probability_threshold(kNN, X_test_cst, y_test)
AvgCost_usingkNN_test = cost_basedOn_prob(kNN_predproba_test, y_test, best_probability)
print("Min avg. Cost per Patient using kNN\t\t: {}".format(AvgCost_usingkNN_test))Min avg. Cost per Patient using kNN		: 666

- In this method, we compared 3 models Linear Logistic Regression, Poly Logistic Regression and kNN model. Based on our calculations all three models returned the 666 as the minimum Avg cost per patient.

- These models were created by predicting the best probability of each model as a cut off threshold. I do suspect these can result in overfitting the models. However, in theoratical stand point these appear to be the min cost per patients _____no_output_____