# Notebook from jakesiegel/RNASeq_cell_cycle
Path: notebooks/05 Unsupervised clustering.ipynb

# 05 - Unsupervised clustering via SOM

author: jake siegel

date: 1.22.2018

The purpose of this notebook is to explore methods for unsupervised clustering of the Karlsson TPM data with the goal of determining whether there is a more natural number of cell cycle stages in trancriptome data than the 3 cytological stages.

In the Karlsson paper, they did tSNE of all genes observed in at least 15 cells to make a timeline and did pairwise correlations.  They determined that there early and late G1 are effectively unique and that there are essentially 6 classes of gene expression through a cell cycle.  Leng et al. perform a similar analysis with a custom algorithm on a transcriptome dataset of 490 asychronous hESC cells.

The next few notebooks ivestigate how many cell clusters the transcriptome data naturally form using tSNE, self organizing maps and hdbscan.  An autoencoder will be implemented to reduce dimensionality for tSNE and HDBCSAN.

Initially, the Karlsson data will be used, however the Leng data should also be utilized.

This notebook is to investigate unsupervised clustering via SOM.

### Status: Ongoing 1/26
I still need to figure out how to build the neuron space into an edgeless toroid and display the relative distances between neurons.  Alterantively, I could implement the U* Matrix R package from Alfred Ultsch, which is supposed to be able to do both._____no_output_____

%matplotlib inline
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import datetime_____no_output_____

# Self Organizing Map:

This section contains the code to build a SOM in tensorflow.  Adapted from:

https://github.com/spiglerg/Kohonen_SOM_Tensorflow/blob/master/som.py

https://github.com/BinRoot/TensorFlow-Book/blob/master/ch05_clustering/som.py

https://wonikjang.github.io/deeplearning_unsupervised_som/2017/06/30/som.html

https://codesachin.wordpress.com/2015/11/28/self-organizing-maps-with-googles-tensorflow/_____no_output_____

# My SOM

class SOM:
    def __init__(self, width, height, dim, n_iter, alpha = None, sigma = None):
        #         assign parameters:
        self.n_iter = n_iter
        self.width = width
        self.height = height
        self.dim = dim # dimension of input data
#         self.locs = self.get_locs()

        #         setting defaults if not specified
        if alpha is None: # learning rate
            self.alpha = 0.2
        else:
            self.alpha = tf.to_float(alpha)
        if sigma is None: # neighborhood influence
            self.sigma = max(width, height) / 2.0
        else:
            self.sigma = tf.to_float(sigma)

        #         Initialize graph:
        self.initialize_graph()

    def initialize_graph(self):
        #   initialize weight vectors:
        self.weights = tf.Variable(tf.random_normal([self.width*self.height,self.dim]))

#         self.locations = tf.constant(np.array(list(self.locs(self.width,self.height))))
        self.locations = tf.constant(np.array(list(self.locs())))
        #         self.locations = tf.constant(self.locs)

        #   initialize placeholders for training vector and current iteration
        self.input_vector = tf.placeholder(tf.float32, [self.dim])
        self.curr_iter = tf.placeholder(tf.float32)

        #   The training operation:
        #   Compute the closest weight vector to each input for each neuron
        self.bmu_index = tf.argmin(tf.sqrt(tf.reduce_sum(
            tf.pow(tf.subtract(self.weights,tf.stack([self.input_vector for
                                               i in range(self.width*self.height)])),2),1)),0)
#         self.bmu_index = tf.argmin(tf.sqrt(tf.reduce_sum(
#             tf.pow(tf.subtract(self.weights,tf.stack([self.input_vector for
#                                                i in range(600)])),2),1)),0)

        #   Get the location of the BMU
        self.bmu_sliced = tf.pad(tf.reshape(self.bmu_index,[1]),np.array([[0,1]]))
        self.bmu_loc = tf.reshape(tf.slice(self.locations, self.bmu_sliced,
                                          tf.constant(np.array([1,2]))),[2])

        #   update learning rate & neighborhood influence:
        self.lr_update = tf.subtract(1.0, tf.divide(self.curr_iter, self.n_iter))
        self.alpha_update = tf.multiply(self.alpha, self.lr_update)
        self.sigma_update = tf.multiply(self.sigma, self.lr_update)

        #   Generate learning rate vectors for all neurons
        self.bmu_dist_sq = tf.cast(tf.reduce_sum(tf.pow(tf.subtract(self.locations,
                                    tf.stack([self.bmu_loc for i in range(self.width*self.height)])),
                                                        2),1),"float32")
        self.neighborhood = tf.exp(tf.negative(tf.divide(self.bmu_dist_sq, tf.pow(self.sigma_update,2))))
        self.lr_update = tf.multiply(self.alpha_update, self.neighborhood)

        #   Update the weights vector:
        #   w_delta = L(t) * ( V(t)-W(t) )
        self.lr_multiplier = tf.stack([tf.tile(tf.slice(
            self.lr_update, np.array([i]), np.array([1])),[self.dim])
                                     for i in range(self.width* self.height)])
        self.w_delta = tf.multiply(self.lr_multiplier, tf.subtract(tf.stack(
                [self.input_vector for i in range(self.width*self.height)]),
                self.weights))
        self.weights_update = tf.assign_add(self.weights, self.w_delta)

        #   Initialize:
        self.sess = tf.Session()
        self.sess.run(tf.global_variables_initializer())

    def locs(self):
#         locs = [[x, y]
#                 for y in range(self.height)
#                 for x in range(self.width)]
#         return tf.to_float(locs)
        for i in range(self.width):
            for j in range(self.height):
                yield np.array([i, j])

    def train(self,x_inputs):
        for iter_num in range(self.n_iter):
            for x_input in x_inputs:
                self.sess.run(self.weights_update,
                              {self.input_vector: x_input, self.curr_iter: iter_num})

        #  Store the centroid grid to map inputs to their closest neurons at the end
        centroid_grid = [[] for i in range(self.width)]
        self.weight_vectors = list(self.sess.run(self.weights))
        self.loc_vectors = list(self.sess.run(self.locations))
        for i,loc in enumerate(self.loc_vectors):
            centroid_grid[loc[0]].append(self.weight_vectors[i])
        self.centroids = centroid_grid

    def get_centroids(self):
        if not self.centroids:
            raise ValueError('Not trained yet')
        return self.centroids

    def map_to_vector(self, x_inputs):
        if not self.centroids:
            raise ValueError('Not trained yet')

        output = []
        for vec in x_inputs:
            neuron = min([i for i in range(len(self.weight_vectors))],
                         key = lambda x: np.linalg.norm(vec-self.weight_vectors[x]))
            output.append(self.locations[neuron].eval(session=self.sess))

        return output_____no_output_____# Testing cell

#Training inputs for RGBcolors
colors = np.array(
     [[0., 0., 0.],
      [0., 0., 1.],
      [0., 0., 0.5],
      [0.125, 0.529, 1.0],
      [0.33, 0.4, 0.67],
      [0.6, 0.5, 1.0],
      [0., 1., 0.],
      [1., 0., 0.],
      [0., 1., 1.],
      [1., 0., 1.],
      [1., 1., 0.],
      [1., 1., 1.],
      [.33, .33, .33],
      [.5, .5, .5],
      [.66, .66, .66]])
color_names = ['black', 'blue', 'darkblue', 'skyblue', 'greyblue', 'lilac', 'green', 'red',
               'cyan', 'violet', 'yellow', 'white', 'darkgrey', 'mediumgrey', 'lightgrey']

print str(datetime.datetime.now()),'Training SOM'
#Train a 20x30 SOM with 400 iterations
som = SOM(20, 30, 3, 400)
som.train(colors)

print str(datetime.datetime.now()),'Getting output grid'
#Get output grid
image_grid = som.get_centroids()

print str(datetime.datetime.now()),'Mapping inputs'
#Map colours to their closest neurons
mapped = som.map_to_vector(colors)

print str(datetime.datetime.now()),'Plotting'
#Plot
plt.imshow(image_grid)
plt.title('Color SOM')
for i, m in enumerate(mapped):
    plt.text(m[1], m[0], color_names[i], ha='center', va='center',
             bbox=dict(facecolor='white', alpha=0.5, lw=0))2018-01-24 12:31:24.858337 Training SOM
2018-01-24 12:32:21.895603 Getting output grid
2018-01-24 12:32:21.896046 Mapping inputs
2018-01-24 12:32:34.674275 Plotting
image_grid[0][0]_____no_output_____

The training took about 60s.  From the Peng & Nie paper, they suggest the time complexity is O(N_samples*N_neurons*Dimension_sample*Iterations)

For this example, that should be O(15*600*3*400) = O(10.8e6)

so as a rough guide, I'd estimate O(200,000)/s.

For the Karlsson data, it would be:
N_sample = 96
Dimension_sample = ~13566 (at the 8-cell detection cut-off)
Iterations = Keep this as a multiplicative factor since I have no idea how many it will take
N_neurons = 100 to 400 (rough guide is 10 per expected class)

So, that's 130,000,000 * (1 to 4) * Iterations = 600s * (1 to 4) * Iterations
so roughly 10 to 40 min per iteration, which is multiple days.

I should get about 960 minutes if I start around 3pm.  So, that would be:
O(1e10)= O(1e8=Dim*N_Neurons*Iter)
To keep 10,000 transcripts, leaves me with 1e4 to split between N_Neurons and Iter, which is not many
ideally, I'd keep a 15x15 network = 400 neurons
so 225 neurons, 250 iterations, 2000 dimensions

## Testing on the Karlsson TPM data:_____no_output_____

import pandas as pd

data_src = '../data/processed/KarlssonTPM.csv'
df = pd.read_csv(data_src, sep = '\t')
df = df.set_index(list(df.columns)[0])
df = df.drop(['bp'],axis=0)
df.Stage = pd.factorize(df.Stage)[0]_____no_output_____# I think the cut to 1000 genes is about 92 cells

keepcols = []

for column in df.iloc[:,:-1]:
    if df[column].astype(bool).sum(axis=0)>92:
        keepcols.append(column)

print len(keepcols)

y = df['Stage']
X = df[keepcols]

X.head()2230
# Training the Karlsson TPM data (transcripts found in >92 cells)

print str(datetime.datetime.now()),'Training SOM'
#Train a 15x15 SOM with 250 iterations
som = SOM(15, 15, 2230, 250)
som.train(X.values)

print str(datetime.datetime.now()),'Getting output grid'
#Get output grid
image_grid = som.get_centroids()

print str(datetime.datetime.now()),'Mapping inputs'
#Map colours to their closest neurons
mapped = som.map_to_vector(X.values)

# print str(datetime.datetime.now()),'Plotting'
# #Plot
# plt.imshow(image_grid)
# plt.title('Color SOM')
# for i, m in enumerate(mapped):
#     plt.text(m[1], m[0], color_names[i], ha='center', va='center',
#              bbox=dict(facecolor='white', alpha=0.5, lw=0))2018-01-24 14:37:48.858783 Training SOM
2018-01-24 14:43:58.439592 Getting output grid
2018-01-24 14:43:58.441176 Mapping inputs
colormap = ['blue','green','red']
cyt = ['G1','S','G2/M']
fig, ax = plt.subplots(figsize=[12,8])
for i,m in enumerate(mapped):
    ax = plt.scatter(m[1], m[0], color = colormap[y.values[i]], marker = 'x')
    ax = plt.text(m[1]+0.1,m[0]+0.1, cyt[y.values[i]],verticalalignment='bottom', horizontalalignment='left',
        color=colormap[y.values[i]], fontsize=8)

# y.values[12]
# colormap[y.values[40]]_____no_output_____

Interesting, that only took 5 minutes.  I guess training takes less time than setting up the SOM.  Let's test a few things (number of neurons, iterations) and then try it with the big dataset:_____no_output_____

# I think the cut to 1000 genes is about 92 cells
def columns_min_detect(df_temp,minimum_cells):
    keepcols = []
    for column in df_temp.iloc[:,:-1]:
        if df_temp[column].astype(bool).sum(axis=0)>minimum_cells:
            keepcols.append(column)
    return keepcols

# print len(keepcols)

# y = df['Stage']
# X = df[keepcols]

# X.head()_____no_output_____y = df['Stage']
use_cols = columns_min_detect(df,92)
X = df[use_cols]

X.head()_____no_output_____# Training the Karlsson TPM data (transcripts found in >92 cells, 400 neurons, 250 iterations)

print str(datetime.datetime.now()),'Training SOM'
#Train a 15x15 SOM with 250 iterations
som = SOM(20, 20, X.shape[1], 250)
som.train(X.values)

print str(datetime.datetime.now()),'Getting output grid'
#Get output grid
image_grid = som.get_centroids()

print str(datetime.datetime.now()),'Mapping inputs'
#Map colours to their closest neurons
mapped = som.map_to_vector(X.values)

colormap = ['blue','green','red']
cyt = ['G1','S','G2/M']
fig, ax = plt.subplots(figsize=[12,8])
for i,m in enumerate(mapped):
    ax = plt.scatter(m[1], m[0], color = colormap[y.values[i]], marker = 'x')
    ax = plt.text(m[1]+0.1,m[0]+0.1, cyt[y.values[i]],verticalalignment='bottom', horizontalalignment='left',
        color=colormap[y.values[i]], fontsize=8)
2018-01-24 15:12:37.621812 Training SOM
2018-01-24 15:23:54.717603 Getting output grid
2018-01-24 15:23:54.718761 Mapping inputs
# Training the Karlsson TPM data (transcripts found in >92 cells, 400 neurons, 400 iterations)

print str(datetime.datetime.now()),'Training SOM'
#Train a 20x20 SOM with 400 iterations
som = SOM(20, 20, X.shape[1], 400)
som.train(X.values)

print str(datetime.datetime.now()),'Getting output grid'
#Get output grid
image_grid = som.get_centroids()

print str(datetime.datetime.now()),'Mapping inputs'
#Map colours to their closest neurons
mapped = som.map_to_vector(X.values)

colormap = ['blue','green','red']
cyt = ['G1','S','G2/M']
fig, ax = plt.subplots(figsize=[12,8])
for i,m in enumerate(mapped):
    ax = plt.scatter(m[1], m[0], color = colormap[y.values[i]], marker = 'x')
    ax = plt.text(m[1]+0.1,m[0]+0.1, cyt[y.values[i]],verticalalignment='bottom', horizontalalignment='left',
        color=colormap[y.values[i]], fontsize=8)

2018-01-24 15:28:42.129754 Training SOM
2018-01-24 15:47:09.708838 Getting output grid
2018-01-24 15:47:09.710286 Mapping inputs
# Doesn't seem to have changed anything at all.  OK, now let's try it on the full dataset:
# transcripts found in > 8 cells, 20 x 20 ANN, 400 iterations

y = df['Stage']
use_cols = columns_min_detect(df,8)
X = df[use_cols]

print str(datetime.datetime.now()),'Training SOM'
#Train a 20x20 SOM with 400 iterations
som = SOM(20, 20, X.shape[1], 400)
som.train(X.values)

print str(datetime.datetime.now()),'Getting output grid'
#Get output grid
image_grid = som.get_centroids()

print str(datetime.datetime.now()),'Mapping inputs'
#Map colours to their closest neurons
mapped = som.map_to_vector(X.values)

colormap = ['blue','green','red']
cyt = ['G1','S','G2/M']
fig, ax = plt.subplots(figsize=[12,8])
for i,m in enumerate(mapped):
    ax = plt.scatter(m[1], m[0], color = colormap[y.values[i]], marker = 'x')
    ax = plt.text(m[1]+0.1,m[0]+0.1, cyt[y.values[i]],verticalalignment='bottom', horizontalalignment='left',
        color=colormap[y.values[i]], fontsize=8)2018-01-24 15:53:54.012657 Training SOM
2018-01-24 17:20:47.586914 Getting output grid
2018-01-24 17:20:47.592125 Mapping inputs
# Maybe I just have too many neurons.
# I am effectively able to map each data point to an individual neuron and allow space between them.
# Let's try this with 36 neurons: 6x6

y = df['Stage']
use_cols = columns_min_detect(df,8)
X = df[use_cols]

print str(datetime.datetime.now()),'Training SOM'
#Train a 6x6 SOM with 500 iterations
som = SOM(6, 6, X.shape[1], 500)
som.train(X.values)

print str(datetime.datetime.now()),'Getting output grid'
#Get output grid
image_grid = som.get_centroids()

print str(datetime.datetime.now()),'Mapping inputs'
#Map colours to their closest neurons
mapped = som.map_to_vector(X.values)

colormap = ['blue','green','red']
cyt = ['G1','S','G2/M']
fig, ax = plt.subplots(figsize=[12,8])
for i,m in enumerate(mapped):
    ax = plt.scatter(m[1], m[0], color = colormap[y.values[i]], marker = 'x')
    ax = plt.text(m[1]+0.1,m[0]+0.1, cyt[y.values[i]],verticalalignment='bottom', horizontalalignment='left',
        color=colormap[y.values[i]], fontsize=8)2018-01-25 10:33:41.471790 Training SOM
2018-01-25 10:43:37.885909 Getting output grid
2018-01-25 10:43:37.892655 Mapping inputs
colormap = ['blue','green','red']
marks = ['.','None','None']
cyt = ['G1','S','G2/M']
fig, ax = plt.subplots(figsize=[12,8])
for i,m in enumerate(mapped):
    ax = plt.scatter(m[1], m[0], color = colormap[y.values[i]], alpha = 0.3,
                     marker = marks[y.values[i]])
#     ax = plt.text(m[1]+0.1,m[0]+0.1, cyt[y.values[i]],verticalalignment='bottom', horizontalalignment='left',
#         color=colormap[y.values[i]], fontsize=8)_____no_output_____cols = ['X','Y','Stage']
temp = pd.DataFrame()
for i,m in enumerate(mapped):
    temp = temp.append(pd.Series([m[1],m[0],y.values[i]]), ignore_index=True)

temp.columns = cols
# temp
import seaborn as sns
sns.lmplot(x='X',y='Y',data = temp, fit_reg=False,hue='Stage',x_jitter=0.15, y_jitter=0.15, scatter_kws={'alpha':0.3},palette = 'dark')
_____no_output_____

# Summary

Unsurprisingly, the simple SOM does not seem to cluster wonderfully.  The biggest limitations are the edge effects and not having figured out yet how to map the distance between neurons onto a 2D visualization.

To solve the first problem, people use toroidal shaped spaces (i.e. no edges).  I need to look into how to implement this.

For the second problem, from the image_grid, I get a position of each neuron in n_transcript dimensions.  It's pretty easy to get the euclidean distance between any 2, but then I need to figure out how to display the relative distances between 3 neurons and then how to do it for the 4 neighboring neurons.

Alternatively, there is an R-package from Alfred Ultsch's group in Marburg called U*matrix that should handle both issues. _____no_output_____

print len(image_grid[0][0])

image_grid13566

##  Scratch space_____no_output_____

# OK, let's try it on scaled data now.
from sklearn.preprocessing import RobustScaler
y = df['Stage']
use_cols = columns_min_detect(df,8)
X = df[use_cols]

X_scaled = pd.DataFrame(RobustScaler(quantile_range=(25, 75)).fit_transform(X))

print str(datetime.datetime.now()),'Training SOM'
#Train a 20x20 SOM with 400 iterations
som = SOM(20, 20, X.shape[1], 400)
som.train(X.values)

print str(datetime.datetime.now()),'Getting output grid'
#Get output grid
image_grid = som.get_centroids()

print str(datetime.datetime.now()),'Mapping inputs'
#Map colours to their closest neurons
mapped = som.map_to_vector(X.values)

colormap = ['blue','green','red']
cyt = ['G1','S','G2/M']
fig, ax = plt.subplots(figsize=[12,8])
for i,m in enumerate(mapped):
    ax = plt.scatter(m[1], m[0], color = colormap[y.values[i]], marker = 'x')
    ax = plt.text(m[1]+0.1,m[0]+0.1, cyt[y.values[i]],verticalalignment='bottom', horizontalalignment='left',
        color=colormap[y.values[i]], fontsize=8)2018-01-24 23:03:01.086521 Training SOM
2018-01-25 00:30:33.141982 Getting output grid
2018-01-25 00:30:33.142619 Mapping inputs
X.head()_____no_output_____pd.DataFrame(X_scaled)_____no_output_____class SOM(object):
    """
    2-D Self-Organizing Map with Gaussian Neighbourhood function
    and linearly decreasing learning rate.
    """

    #To check if the SOM has been trained
    _trained = False

    def __init__(self, m, n, dim, n_iterations=100, alpha=None, sigma=None):
        """
        Initializes all necessary components of the TensorFlow
        Graph.

        m X n are the dimensions of the SOM. 'n_iterations' should
        should be an integer denoting the number of iterations undergone
        while training.
        'dim' is the dimensionality of the training inputs.
        'alpha' is a number denoting the initial time(iteration no)-based
        learning rate. Default value is 0.3
        'sigma' is the the initial neighbourhood value, denoting
        the radius of influence of the BMU while training. By default, its
        taken to be half of max(m, n).
        """

        #Assign required variables first
        self._m = m
        self._n = n
        if alpha is None:
            alpha = 0.3
        else:
            alpha = float(alpha)
        if sigma is None:
            sigma = max(m, n) / 2.0
        else:
            sigma = float(sigma)
        self._n_iterations = abs(int(n_iterations))

        ##INITIALIZE GRAPH
        self._graph = tf.Graph()

        ##POPULATE GRAPH WITH NECESSARY COMPONENTS
        with self._graph.as_default():

            ##VARIABLES AND CONSTANT OPS FOR DATA STORAGE

            #Randomly initialized weightage vectors for all neurons,
            #stored together as a matrix Variable of size [m*n, dim]
            self._weightage_vects = tf.Variable(tf.random_normal(
                [m*n, dim]))

            #Matrix of size [m*n, 2] for SOM grid locations
            #of neurons
            self._location_vects = tf.constant(np.array(
                list(self._neuron_locations(m, n))))

            ##PLACEHOLDERS FOR TRAINING INPUTS
            #We need to assign them as attributes to self, since they
            #will be fed in during training

            #The training vector
            self._vect_input = tf.placeholder("float", [dim])
            #Iteration number
            self._iter_input = tf.placeholder("float")

            ##CONSTRUCT TRAINING OP PIECE BY PIECE
            #Only the final, 'root' training op needs to be assigned as
            #an attribute to self, since all the rest will be executed
            #automatically during training

            #To compute the Best Matching Unit given a vector
            #Basically calculates the Euclidean distance between every
            #neuron's weightage vector and the input, and returns the
            #index of the neuron which gives the least value
            bmu_index = tf.argmin(tf.sqrt(tf.reduce_sum(
                tf.pow(tf.sub(self._weightage_vects, tf.pack(
                    [self._vect_input for i in range(m*n)])), 2), 1)),
                                  0)

            #This will extract the location of the BMU based on the BMU's
            #index
            slice_input = tf.pad(tf.reshape(bmu_index, [1]),
                                 np.array([[0, 1]]))
            bmu_loc = tf.reshape(tf.slice(self._location_vects, slice_input,
                                          tf.constant(np.array([1, 2]))),
                                 [2])

            #To compute the alpha and sigma values based on iteration
            #number
            learning_rate_op = tf.sub(1.0, tf.div(self._iter_input,
                                                  self._n_iterations))
            _alpha_op = tf.mul(alpha, learning_rate_op)
            _sigma_op = tf.mul(sigma, learning_rate_op)

            #Construct the op that will generate a vector with learning
            #rates for all neurons, based on iteration number and location
            #wrt BMU.
            bmu_distance_squares = tf.reduce_sum(tf.pow(tf.sub(
                self._location_vects, tf.pack(
                    [bmu_loc for i in range(m*n)])), 2), 1)
            neighbourhood_func = tf.exp(tf.neg(tf.div(tf.cast(
                bmu_distance_squares, "float32"), tf.pow(_sigma_op, 2))))
            learning_rate_op = tf.mul(_alpha_op, neighbourhood_func)

            #Finally, the op that will use learning_rate_op to update
            #the weightage vectors of all neurons based on a particular
            #input
            learning_rate_multiplier = tf.pack([tf.tile(tf.slice(
                learning_rate_op, np.array([i]), np.array([1])), [dim])
                                               for i in range(m*n)])
            weightage_delta = tf.mul(
                learning_rate_multiplier,
                tf.sub(tf.pack([self._vect_input for i in range(m*n)]),
                       self._weightage_vects))
            new_weightages_op = tf.add(self._weightage_vects,
                                       weightage_delta)
            self._training_op = tf.assign(self._weightage_vects,
                                          new_weightages_op)

            ##INITIALIZE SESSION
            self._sess = tf.Session()

            ##INITIALIZE VARIABLES
            init_op = tf.initialize_all_variables()
            self._sess.run(init_op)

    def _neuron_locations(self, m, n):
        """
        Yields one by one the 2-D locations of the individual neurons
        in the SOM.
        """
        #Nested iterations over both dimensions
        #to generate all 2-D locations in the map
        for i in range(m):
            for j in range(n):
                yield np.array([i, j])

    def train(self, input_vects):
        """
        Trains the SOM.
        'input_vects' should be an iterable of 1-D NumPy arrays with
        dimensionality as provided during initialization of this SOM.
        Current weightage vectors for all neurons(initially random) are
        taken as starting conditions for training.
        """

        #Training iterations
        for iter_no in range(self._n_iterations):
            #Train with each vector one by one
            for input_vect in input_vects:
                self._sess.run(self._training_op,
                               feed_dict={self._vect_input: input_vect,
                                          self._iter_input: iter_no})

        #Store a centroid grid for easy retrieval later on
        centroid_grid = [[] for i in range(self._m)]
        self._weightages = list(self._sess.run(self._weightage_vects))
        self._locations = list(self._sess.run(self._location_vects))
        for i, loc in enumerate(self._locations):
            centroid_grid[loc[0]].append(self._weightages[i])
        self._centroid_grid = centroid_grid

        self._trained = True

    def get_centroids(self):
        """
        Returns a list of 'm' lists, with each inner list containing
        the 'n' corresponding centroid locations as 1-D NumPy arrays.
        """
        if not self._trained:
            raise ValueError("SOM not trained yet")
        return self._centroid_grid

    def map_vects(self, input_vects):
        """
        Maps each input vector to the relevant neuron in the SOM
        grid.s
        'input_vects' should be an iterable of 1-D NumPy arrays with
        dimensionality as provided during initialization of this SOM.
        Returns a list of 1-D NumPy arrays containing (row, column)
        info for each input vector(in the same order), corresponding
        to mapped neuron.
        """

        if not self._trained:
            raise ValueError("SOM not trained yet")

        to_return = []
        for vect in input_vects:
            min_index = min([i for i in range(len(self._weightages))],
                            key=lambda x: np.linalg.norm(vect-
                                                         self._weightages[x]))
            to_return.append(self._locations[min_index])

        return to_return_____no_output_____class SOM:
    def __init__(self, width, height, dim):
        self.num_iters = 100
        self.width = width
        self.height = height
        self.dim = dim
        self.node_locs = self.get_locs()

        # Each node is a vector of dimension `dim`
        # For a 2D grid, there are `width * height` nodes
        nodes = tf.Variable(tf.random_normal([width*height, dim]))
        self.nodes = nodes

        # These two ops are inputs at each iteration
        x = tf.placeholder(tf.float32, [dim])
        iter = tf.placeholder(tf.float32)

        self.x = x
        self.iter = iter

        # Find the node that matches closest to the input
        bmu_loc = self.get_bmu_loc(x)

        self.propagate_nodes = self.get_propagation(bmu_loc, x, iter)

    def get_propagation(self, bmu_loc, x, iter):
        num_nodes = self.width * self.height
        rate = 1.0 - tf.div(iter, self.num_iters)
        alpha = rate * 0.5
        sigma = rate * tf.to_float(tf.maximum(self.width, self.height)) / 2.
        expanded_bmu_loc = tf.expand_dims(tf.to_float(bmu_loc), 0)
        sqr_dists_from_bmu = tf.reduce_sum(tf.square(tf.sub(expanded_bmu_loc, self.node_locs)), 1)
        neigh_factor = tf.exp(-tf.div(sqr_dists_from_bmu, 2 * tf.square(sigma)))
        rate = tf.mul(alpha, neigh_factor)
        rate_factor = tf.pack([tf.tile(tf.slice(rate, [i], [1]), [self.dim]) for i in range(num_nodes)])
        nodes_diff = tf.mul(rate_factor, tf.sub(tf.pack([x for i in range(num_nodes)]), self.nodes))
        update_nodes = tf.add(self.nodes, nodes_diff)
        return tf.assign(self.nodes, update_nodes)

    def get_bmu_loc(self, x):
        expanded_x = tf.expand_dims(x, 0)
        sqr_diff = tf.square(tf.sub(expanded_x, self.nodes))
        dists = tf.reduce_sum(sqr_diff, 1)
        bmu_idx = tf.argmin(dists, 0)
        bmu_loc = tf.pack([tf.mod(bmu_idx, self.width), tf.div(bmu_idx, self.width)])
        return bmu_loc

    def get_locs(self):
        locs = [[x, y]
                for y in range(self.height)
                for x in range(self.width)]
        return tf.to_float(locs)

    def train(self, data):
        with tf.Session() as sess:
            sess.run(tf.initialize_all_variables())
            for i in range(self.num_iters):
                for data_x in data:
                    sess.run(self.propagate_nodes, feed_dict={self.x: data_x, self.iter: i})
            centroid_grid = [[] for i in range(self.width)]
            self.nodes_val = list(sess.run(self.nodes))
            self.locs_val = list(sess.run(self.node_locs))
            for i, l in enumerate(self.locs_val):
                centroid_grid[int(l[0])].append(self.nodes_val[i])
            self.centroid_grid = centroid_grid_____no_output_____class SOM(object):

    # To check if the SOM has been trained
    trained = False

    def __init__(self, m, n, dim, n_iterations=100, alpha=None, sigma=None):

        # Assign required variables first
        self.m = m; self.n = n
        if alpha is None:
            alpha = 0.2
        else:
            alpha = float(alpha)
        if sigma is None:
            sigma = max(m, n) / 2.0
        else:
            sigma = float(sigma)
        self.n_iterations = abs(int(n_iterations))

        self.graph = tf.Graph()

        with self.graph.as_default():

            # To save data, create weight vectors and their location vectors

            self.weightage_vects = tf.Variable(tf.random_normal( [m * n, dim]) )

            self.location_vects = tf.constant(np.array(list(self.neuron_locations(m, n))))

            # Training inputs

            # The training vector
            self.vect_input = tf.placeholder("float", [dim])
            # Iteration number
            self.iter_input = tf.placeholder("float")

            # Training Operation  # tf.pack result will be [ (m*n),  dim ]

            bmu_index = tf.argmin(tf.sqrt(tf.reduce_sum(
                tf.pow(tf.sub(self.weightage_vects, tf.pack(
                    [self.vect_input for _ in range(m * n)])), 2), 1)), 0)

            slice_input = tf.pad(tf.reshape(bmu_index, [1]), np.array([[0, 1]]))
            bmu_loc = tf.reshape(tf.slice(self.location_vects,
                                    slice_input, tf.constant(np.array([1, 2]))), [2])

            # To compute the alpha and sigma values based on iteration number
            learning_rate_op = tf.sub(1.0, tf.div(self.iter_input, self.n_iterations))
            alpha_op = tf.mul(alpha, learning_rate_op)
            sigma_op = tf.mul(sigma, learning_rate_op)

            # learning rates for all neurons, based on iteration number and location w.r.t. BMU.
            bmu_distance_squares = tf.reduce_sum(tf.pow(tf.sub(
                self.location_vects, tf.pack( [bmu_loc for _ in range(m * n)] ) ) , 2 ), 1)

            neighbourhood_func = tf.exp(tf.neg(tf.div(tf.cast(
                bmu_distance_squares, "float32"), tf.pow(sigma_op, 2))))
            learning_rate_op = tf.mul(alpha_op, neighbourhood_func)

            # Finally, the op that will use learning_rate_op to update the weightage vectors of all neurons
            learning_rate_multiplier = tf.pack([tf.tile(tf.slice(
                learning_rate_op, np.array([i]), np.array([1])), [dim]) for i in range(m * n)] )

            ### Strucutre of updating weight ###
            ### W(t+1) = W(t) + W_delta ###
            ### wherer, W_delta = L(t) * ( V(t)-W(t) ) ###

            # W_delta = L(t) * ( V(t)-W(t) )
            weightage_delta = tf.mul(
                learning_rate_multiplier,
                tf.sub(tf.pack([self.vect_input for _ in range(m * n)]), self.weightage_vects))

            # W(t+1) = W(t) + W_delta
            new_weightages_op = tf.add(self.weightage_vects, weightage_delta)

            # Update weightge_vects by assigning new_weightages_op to it.
            self.training_op = tf.assign(self.weightage_vects, new_weightages_op)

            self.sess = tf.Session()
            init_op = tf.global_variables_initializer()
            self.sess.run(init_op)

    def neuron_locations(self, m, n):

        for i in range(m):
            for j in range(n):
                yield np.array([i, j])

    def train(self, input_vects):

        # Training iterations
        for iter_no in range(self.n_iterations):
            # Train with each vector one by one
            for input_vect in input_vects:
                self.sess.run(self.training_op,
                        feed_dict={self.vect_input: input_vect, self.iter_input: iter_no})

        # Store a centroid grid for easy retrieval later on
        centroid_grid = [[] for i in range(self.m)]
        self.weightages = list(self.sess.run(self.weightage_vects))
        self.locations = list(self.sess.run(self.location_vects))
        for i, loc in enumerate(self.locations):
            centroid_grid[loc[0]].append(self.weightages[i])

        self.centroid_grid = centroid_grid

        self.trained = True

    def get_centroids(self):

        if not self.trained:
            raise ValueError("SOM not trained yet")
        return self.centroid_grid

    def map_vects(self, input_vects):

        if not self.trained:
            raise ValueError("SOM not trained yet")

        to_return = []
        for vect in input_vects:
            min_index = min( [i for i in range(len(self.weightages))],
                            key=lambda x: np.linalg.norm(vect - self.weightages[x]) )
            to_return.append(self.locations[min_index])

        return to_return_____no_output_____# Testing cell for data in pandas df

#Training inputs for RGBcolors
colorDF = pd.DataFrame(
     [[0., 0., 0.],
      [0., 0., 1.],
      [0., 0., 0.5],
      [0.125, 0.529, 1.0],
      [0.33, 0.4, 0.67],
      [0.6, 0.5, 1.0],
      [0., 1., 0.],
      [1., 0., 0.],
      [0., 1., 1.],
      [1., 0., 1.],
      [1., 1., 0.],
      [1., 1., 1.],
      [.33, .33, .33],
      [.5, .5, .5],
      [.66, .66, .66]])
color_names = ['black', 'blue', 'darkblue', 'skyblue', 'greyblue', 'lilac', 'green', 'red',
               'cyan', 'violet', 'yellow', 'white', 'darkgrey', 'mediumgrey', 'lightgrey']

print str(datetime.datetime.now()),'Training SOM'
#Train a 20x30 SOM with 400 iterations
som = SOM(20, 30, 3, 400)
som.train(colorDF.values)

print str(datetime.datetime.now()),'Getting output grid'
#Get output grid
image_grid = som.get_centroids()

print str(datetime.datetime.now()),'Mapping inputs'
#Map colours to their closest neurons
mapped = som.map_to_vector(colorDF.values)

print str(datetime.datetime.now()),'Plotting'
#Plot
plt.imshow(image_grid)
plt.title('Color SOM')
for i, m in enumerate(mapped):
    plt.text(m[1], m[0], color_names[i], ha='center', va='center',
             bbox=dict(facecolor='white', alpha=0.5, lw=0))2018-01-24 14:30:51.213953 Training SOM
2018-01-24 14:31:52.547706 Getting output grid
2018-01-24 14:31:52.548605 Mapping inputs
2018-01-24 14:32:07.122769 Plotting
colorDF = pd.DataFrame(
     [[0., 0., 0.],
      [0., 0., 1.],
      [0., 0., 0.5],
      [0.125, 0.529, 1.0],
      [0.33, 0.4, 0.67],
      [0.6, 0.5, 1.0],
      [0., 1., 0.],
      [1., 0., 0.],
      [0., 1., 1.],
      [1., 0., 1.],
      [1., 1., 0.],
      [1., 1., 1.],
      [.33, .33, .33],
      [.5, .5, .5],
      [.66, .66, .66]])

colorDF_____no_output_____


# Notebook from fseidl-gdit/SL-Cloud
Path: first_notebook.ipynb

## First notebook
```
Title:   First notebook
Author:  Bahar Tercan
Created: 02-09-2022
Purpose: Retrieve data succesfully from BigQuery tables
Notes: Runs in MyBinder
```_____no_output_____

# this code block only needs to be run once after loading
!pip3 install google.cloud
!pip3 install pandasRequirement already satisfied: google.cloud in /usr/local/lib/python3.9/site-packages (0.34.0)
WARNING: You are using pip version 20.3.3; however, version 22.0.3 is available.
You should consider upgrading via the '/usr/local/opt/python@3.9/bin/python3.9 -m pip install --upgrade pip' command.
Requirement already satisfied: pandas in /usr/local/lib/python3.9/site-packages (1.4.0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/site-packages (from pandas) (2021.3)
Requirement already satisfied: numpy>=1.18.5 in /usr/local/lib/python3.9/site-packages (from pandas) (1.22.2)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.9/site-packages (from pandas) (2.8.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)
WARNING: You are using pip version 20.3.3; however, version 22.0.3 is available.
You should consider upgrading via the '/usr/local/opt/python@3.9/bin/python3.9 -m pip install --upgrade pip' command.
# To be able to execute queries we need to import the BQ python library
from google.cloud import bigquery
import pandas as pd_____no_output_____

## Google Authentication
Running the BigQuery cells in this notebook requires a Google Cloud Project, instructions for creating a project can be found in the [Google Documentation](https://cloud.google.com/resource-manager/docs/creating-managing-projects#console). The instance needs to be authorized to bill the project for querying, for more information on getting started in the cloud see ['Quick Start Guide to ISB-CGC'](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowToGetStartedonISB-CGC.html) and alternative authentication methods can be found in the [Google Documentation](https://googleapis.dev/python/google-api-core/latest/auth.html)._____no_output_____

# Please make sure that you have installed Cloud SDK.
# See support from https://cloud.google.com/sdk/docs/install

!gcloud auth application-default login_____no_output_____project_id='syntheticlethality' #please replace syntheticlethality with your own projectid
client = bigquery.Client(project_id)
#client = bigquery.Client(credentials=credentials, project=credentials.project_id)_____no_output_____# Retrieve BRCA1 gene effect score of the first data entry for gene BRCA1 from CRISPR dataset
crispr_query="""
SELECT * FROM
`isb-cgc-bq.DEPMAP.Achilles_gene_effect_DepMapPublic_current`
WHERE Hugo_Symbol='BRCA1'
LIMIT 10
"""
brca1_gene_effect=client.query(crispr_query).result().to_dataframe()
brca1_gene_effect_____no_output_____# Retrieve the first 10 records for BRCA1 gene expression from pancancer atlas dataset
pancancer_atlas_query="""
SELECT * FROM
`pancancer-atlas.Filtered.EBpp_AdjustPANCAN_IlluminaHiSeq_RNASeqV2_genExp_filtered`
WHERE Symbol='BRCA1'
LIMIT 10
"""
brca1_gene_exp=client.query(pancancer_atlas_query).result().to_dataframe()
brca1_gene_exp_____no_output_____


# Notebook from DataCrunchIO/course20
Path: index.ipynb

# Practical Deep Learning for Coders
> Deep Learning for Coders with fastai and PyTorch: AI Applications Without a PhD - the book and the course

- hide_colab_badge:true
- image:images/myths.png_____no_output_____Welcome to *Practical Deep Learning for Coders*. This web site covers the book and the 2020 version of the course, which are designed to work closely together. If you haven't yet got the book, you can [buy it here](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527). It's also [freely available](https://github.com/fastai/fastbook) as interactive Jupyter Notebooks; read on to learn how to access them.._____no_output_____## How do I get started?_____no_output_____If you're ready to dive in right now, here's how to get started. If you want to know more about this course, read the next sections, and then come back here.

To watch the videos, click on the *Lessons* section in the navigation sidebar. The lessons all have searchable transcripts; click "Transcript Search" in the top right panel to search for a word or phrase, and then click it to jump straight to video at the time that appears in the transcript. The videos are all captioned and also translated into Chinese (简体中文) and Spanish; while watching the video click the "CC" button to turn them on and off, and the setting button to change the language.

Each video covers a chapter from the book. The entirety of every chapter of the book is available as an interactive Jupyter Notebook. [Jupyter Notebook](https://jupyter.org/) is the most popular tool for doing data science in Python, for good reason. It is powerful, flexible, and easy to use. We think you will love it! Since the most important thing for learning deep learning is writing code and experimenting, it's important that you have a great platform for experimenting with code.

To get started, we recommend using a Jupyter Server from one of the recommended online platforms (click the links for instructions on how to use these for the course):
- [Colab](/start_colab): A popular free service from Google. Google have changed the notebook platform quite a lot, so keyboard shortcuts are different, and not everything works (e.g. much of chapter 2 doesn't work because Colab doesn't support creating web app GUIs)
- [Gradient](/start_gradient): Unlike Colab, this is a "real" Jupyter Notebook so everything in the course works. It also provides space to save your notebooks and models. However, sometimes the free servers get over-loaded, and when that happens it's impossible to connect.

If you are interested in the experience of running a full Linux server, you can consider [DataCrunch.io](start_datacrunch) (very new service so we don't know how good it is, no setup required, extremely good value and extremely fast GPUs), or [Google Cloud](start_gcp) (extremely popular service, very reliable, but the fastest GPUs are far more expensive). We strongly suggest using one of the recommended online platforms for running the notebooks, and to *not* use your own computer, unless you're very experienced with Linux system adminstration and handling GPU drivers, CUDA, and so forth.

If you need help, there's a [wonderful online community](https://forums.fast.ai/c/part1-v4/46) ready to help you at forums.fast.ai. Before asking a question on the forums, search carefully to see if your question has been answered before. (The forum system won't let you post until you've spent a few minutes on the site reading existing topics.) One bit that many students find tricky is getting signed up for the Bing API for the image download task in lesson 2; here's a helpful [forum post](https://forums.fast.ai/t/getting-the-bing-image-search-key/67417) explaining how to get the Bing API key you'll need for downloading images._____no_output_____## Is this course for me?_____no_output_____Thank you for letting us join you on your deep learning journey, however far along that you may be! Previous fast.ai courses have been studied by hundreds of thousands of students, from all walks of life, from all parts of the world. Many students have told us about how they've become [multiple gold medal winners](https://forums.fast.ai/t/my-first-gold-medal/54237) of [international machine learning competitions](https://towardsdatascience.com/my-3-year-journey-from-zero-python-to-deep-learning-competition-master-6605c188eec7), [received offers](https://forums.fast.ai/t/how-has-your-journey-been-so-far-learners/6480/2) from top companies, and having [research](https://ui.adsabs.harvard.edu/abs/2020EGUGA..2221465A/abstract) [papers](http://www.ieomsociety.org/ieom2020/papers/37.pdf) [published](https://pubs.rsna.org/doi/abs/10.1148/ryai.2019190113?journalCode=ai). For instance, Isaac Dimitrovsky [told us](https://forums.fast.ai/t/thanks-ra2-dream-challenge-win/76875) that he had "*been playing around with ML for a couple of years without really grokking it... [then] went through the fast.ai part 1 course late last year, and it clicked for me*". He went on to achieve first place in the prestigious international [RA2-DREAM Challenge](https://www.synapse.org/#!Synapse:syn20545111/wiki/594083) competition! He developed a [multistage deep learning method](https://www.synapse.org/#!Synapse:syn21478998/wiki/604432) for scoring radiographic hand and foot joint damage in rheumatoid arthritis, taking advantage of the fastai library.

It doesn't matter if you don't come from a technical or a mathematical background (though it's okay if you do too!); we wrote this course to make deep learning accessible to as many people as possible. The only prerequisite is that you know how to code (a year of experience is enough), preferably in Python, and that you have at least followed a high school math course. The first three chapters have been explicitly written in a way that will allow executives, product managers, etc. to understand the most important things they'll need to know about deep learning -- if that's you, just skip over the code in those sections.

Deep learning is a computer technique to extract and transform data–-with use cases ranging from human speech recognition to animal imagery classification–-by using multiple layers of neural networks. A lot of people assume that you need all kinds of hard-to-find stuff to get great results with deep learning, but as you'll see in this course, those people are wrong. Here's a few things you *absolutely don't need* to do world-class deep learning:

| Myth (don't need) | Truth
|---|---|
| Lots of math | Just high school math is sufficient
| Lots of data | We've seen record-breaking results with <50 items of data
| Lots of expensive computers | You can get what you need for state of the art work for free_____no_output_____Deep learning has power, flexibility, and simplicity. That's why we believe it should be applied across many disciplines. These include the social and physical sciences, the arts, medicine, finance, scientific research, and many more. Here's a list of some of the thousands of tasks in different areas at which deep learning, or methods heavily using deep learning, is now the best in the world:

- **Natural language processing (NLP)** Answering questions; speech recognition; summarizing documents; classifying documents; finding names, dates, etc. in documents; searching for articles mentioning a concept
- **Computer vision** Satellite and drone imagery interpretation (e.g., for disaster resilience); face recognition; image captioning; reading traffic signs; locating pedestrians and vehicles in autonomous vehicles
- **Medicine** Finding anomalies in radiology images, including CT, MRI, and X-ray images; counting features in pathology slides; measuring features in ultrasounds; diagnosing diabetic retinopathy
- **Biology** Folding proteins; classifying proteins; many genomics tasks, such as tumor-normal sequencing and classifying clinically actionable genetic mutations; cell classification; analyzing protein/protein interactions
- **Image generation** Colorizing images; increasing image resolution; removing noise from images; converting images to art in the style of famous artists
- **Recommendation systems** Web search; product recommendations; home page layout
- **Playing games** Chess, Go, most Atari video games, and many real-time strategy games
- **Robotics** Handling objects that are challenging to locate (e.g., transparent, shiny, lacking texture) or hard to pick up
- **Other applications** Financial and logistical forecasting, text to speech, and much more..._____no_output_____## Who we are_____no_output_____We are Sylvain Gugger and Jeremy Howard, your guides on this journey. We're the co-authors of fastai, the software that you'll be using throughout this course.

Jeremy has been using and teaching machine learning for around 30 years. He started using neural networks 25 years ago. During this time, he has led many companies and projects that have machine learning at their core, including founding the first company to focus on deep learning and medicine, Enlitic, and taking on the role of President and Chief Scientist of the world's largest machine learning community, Kaggle. He is the co-founder, along with Dr. Rachel Thomas, of fast.ai, the organization that built the course this course is based on.

Sylvain has written 10 math textbooks, covering the entire advanced French maths curriculum! He is now a researcher at Hugging Face, and was previously a researcher at fast.ai.

We care a lot about teaching. In this course, we start by showing how to use a complete, working, very usable, state-of-the-art deep learning network to solve real-world problems, using simple, expressive tools. And then we gradually dig deeper and deeper into understanding how those tools are made, and how the tools that make those tools are made, and so on… We always teaching through examples. We ensure that there is a context and a purpose that you can understand intuitively, rather than starting with algebraic symbol manipulation._____no_output_____## The software you'll be using_____no_output_____In this course, you'll be using [PyTorch](https://pytorch.org/) and [fastai](https://docs.fast.ai).

We've completed hundreds of machine learning projects using dozens of different packages, and many different programming languages. At fast.ai, we have written courses using most of the main deep learning and machine learning packages used today. We spent over a thousand hours testing PyTorch before deciding that we would use it for future courses, software development, and research. PyTorch is now the world's fastest-growing deep learning library and is already used for most research papers at top conferences.

PyTorch works best as a low-level foundation library, providing the basic operations for higher-level functionality. The fastai library is the most popular library for adding this higher-level functionality on top of PyTorch. In this course, as we go deeper and deeper into the foundations of deep learning, we will also go deeper and deeper into the layers of fastai. This course covers version 2 of the fastai library, which is a from-scratch rewrite providing many unique features._____no_output_____## What you'll learn_____no_output_____After finishing this course you will know:

- How to train models that achieve state-of-the-art results in:
  - Computer vision, including image classification (e.g., classifying pet photos by breed), and image localization and detection (e.g., finding where the animals in an image are)
  - Natural language processing (NLP), including document classification (e.g., movie review sentiment analysis) and language modeling
  - Tabular data (e.g., sales prediction) with categorical data, continuous data, and mixed data, including time series
  - Collaborative filtering (e.g., movie recommendation)
- How to turn your models into web applications, and deploy them
- Why and how deep learning models work, and how to use that knowledge to improve the accuracy, speed, and reliability of your models
- The latest deep learning techniques that really matter in practice
- How to implement stochastic gradient descent and a complete training loop from scratch
- How to think about the ethical implications of your work, to help ensure that you're making the world a better place and that your work isn't misused for harm

Here are some of the techniques covered (don't worry if none of these words mean anything to you yet--you'll learn them all soon):

- Random forests and gradient boosting
- Affine functions and nonlinearities
- Parameters and activations
- Random initialization and transfer learning
- SGD, Momentum, Adam, and other optimizers
- Convolutions
- Batch normalization
- Dropout
- Data augmentation
- Weight decay
- Image classification and regression
- Entity and word embeddings
- Recurrent neural networks (RNNs)
- Segmentation
- And much more_____no_output_____