# Notebook from virgesmith/deploying-machine-learning-models
Path: section-04-research-and-development/01-machine-learning-pipeline-data-analysis.ipynb

# Machine Learning Pipeline - Data Analysis

> #### Note: running in a virtualenv in vscode
>
> After creating and activating the virtualenv and installing requirements:
> ```sh
> pip install ipykernel
> ipython kernel install --user --name=<projectname>
> ```

In the following notebooks, we will go through the implementation of each of the steps in the Machine Learning Pipeline.

We will discuss:

1. **Data Analysis**
2. Feature Engineering
3. Feature Selection
4. Model Training
5. Obtaining Predictions / Scoring

We will use the house price dataset available on [Kaggle.com](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data). See below for more details.

===================================================================================================

## Predicting Sale Price of Houses

The aim of the project is to build a machine learning model to predict the sale price of homes based on different explanatory variables describing aspects of residential houses.

### Why is this important?

Predicting house prices is useful to identify fruitful investments or to determine whether the price advertised for a house is over or under-estimated.

### What is the objective of the machine learning model?

We aim to minimise the difference between the real price and the price estimated by our model. We will evaluate model performance with the:

1. mean squared error (mse)
2. root squared of the mean squared error (rmse)
3. r-squared (r2).

### How do I download the dataset?

**Instructions also in the lecture "Download Dataset" in section 1 of the course**

- Visit the [Kaggle Website](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data).

- Remember to **log in**.

- Scroll down to the bottom of the page, and click on the link **'train.csv'**, and then click the 'download' blue button towards the right of the screen, to download the dataset.

- The download the file called **'test.csv'** and save it in the directory with the notebooks.

**Note the following:**

-  You need to be logged in to Kaggle in order to download the datasets.
-  You need to accept the terms and conditions of the competition to download the dataset
-  If you save the file to the directory with the jupyter notebook, then you can run the code as it is written here._____no_output_____# Data Analysis

Let's go ahead and load the dataset._____no_output_____

# to handle datasets
import pandas as pd
import numpy as np

# for plotting
import matplotlib.pyplot as plt
import seaborn as sns

# for the yeo-johnson transformation
import scipy.stats as stats

# to display all the columns of the dataframe in the notebook
pd.pandas.set_option('display.max_columns', None)_____no_output_____# load dataset
data = pd.read_csv('../data/house_prices/train.csv', index_col='Id')

# rows and columns of the data
print(data.shape)

# visualise the dataset
data.head()_____no_output_____

The house price dataset contains 1460 rows, that is, houses, and 80 columns, i.e., variables.

79 are predictive variables and 1 is the target variable: SalePrice

## Analysis

**We will analyse the following:**

1. The target variable
2. Variable types (categorical and numerical)
3. Missing data
4. Numerical variables
    - Discrete
    - Continuous
    - Distributions
    - Transformations

5. Categorical variables
    - Cardinality
    - Rare Labels
    - Special mappings

6. Additional Reading Resources

## Target

Let's begin by exploring the target distribution._____no_output_____

# histogran to evaluate target distribution

data['SalePrice'].hist(bins=50, density=True)
plt.ylabel('Number of houses')
plt.xlabel('Sale Price')
plt.show()_____no_output_____

We can see that the target is continuous, and the distribution is skewed towards the right.

We can improve the value spread with a mathematical transformation._____no_output_____

# let's transform the target using the logarithm

np.log(data['SalePrice']).hist(bins=50, density=True)
plt.ylabel('Number of houses')
plt.xlabel('Log of Sale Price')
plt.show()_____no_output_____

Now the distribution looks more Gaussian._____no_output_____## Variable Types

Next, let's identify the categorical and numerical variables_____no_output_____

# let's identify the categorical variables
# we will capture those of type *object*

cat_vars = [var for var in data.columns if data[var].dtype == 'O']

# MSSubClass is also categorical by definition, despite its numeric values
# (you can find the definitions of the variables in the data_description.txt
# file available on Kaggle, in the same website where you downloaded the data)

# lets add MSSubClass to the list of categorical variables
cat_vars = cat_vars + ['MSSubClass']

# number of categorical variables
len(cat_vars)_____no_output_____# cast all variables as categorical
data[cat_vars] = data[cat_vars].astype('O')_____no_output_____# now let's identify the numerical variables

num_vars = [
    var for var in data.columns if var not in cat_vars and var != 'SalePrice'
]

# number of numerical variables
len(num_vars)_____no_output_____

# Missing values

Let's go ahead and find out which variables of the dataset contain missing values._____no_output_____

# make a list of the variables that contain missing values
vars_with_na = [var for var in data.columns if data[var].isnull().sum() > 0]

# determine percentage of missing values (expressed as decimals)
# and display the result ordered by % of missin data

data[vars_with_na].isnull().mean().sort_values(ascending=False)_____no_output_____

Our dataset contains a few variables with a big proportion of missing values (4 variables at the top). And some other variables with a small percentage of missing observations.

This means that to train a machine learning model with this data set, we need to impute the missing data in these variables.

We can also visualize the percentage of missing values in the variables as follows:_____no_output_____

# plot

data[vars_with_na].isnull().mean().sort_values(
    ascending=False).plot.bar(figsize=(10, 4))
plt.ylabel('Percentage of missing data')
plt.axhline(y=0.90, color='r', linestyle='-')
plt.axhline(y=0.80, color='g', linestyle='-')

plt.show()_____no_output_____# now we can determine which variables, from those with missing data,
# are numerical and which are categorical

cat_na = [var for var in cat_vars if var in vars_with_na]
num_na = [var for var in num_vars if var in vars_with_na]

print('Number of categorical variables with na: ', len(cat_na))
print('Number of numerical variables with na: ', len(num_na))_____no_output_____num_na_____no_output_____cat_na_____no_output_____

## Relationship between missing data and Sale Price

Let's evaluate the price of the house in those observations where the information is missing. We will do this for each variable that shows missing data._____no_output_____

def analyse_na_value(df, var):

    # copy of the dataframe, so that we do not override the original data
    # see the link for more details about pandas.copy()
    # https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.copy.html
    df = df.copy()

    # let's make an interim variable that indicates 1 if the
    # observation was missing or 0 otherwise
    df[var] = np.where(df[var].isnull(), 1, 0)

    # let's compare the median SalePrice in the observations where data is missing
    # vs the observations where data is available

    # determine the median price in the groups 1 and 0,
    # and the standard deviation of the sale price,
    # and we capture the results in a temporary dataset
    tmp = df.groupby(var)['SalePrice'].agg(['mean', 'std'])

    # plot into a bar graph
    tmp.plot(kind="barh", y="mean", legend=False,
             xerr="std", title="Sale Price", color='green')

    plt.show()_____no_output_____# let's run the function on each variable with missing data

for var in vars_with_na:
    analyse_na_value(data, var)_____no_output_____

In some variables, the average Sale Price in houses where the information is missing, differs from the average Sale Price in houses where information exists. This suggests that data being missing could be a good predictor of Sale Price._____no_output_____# Numerical variables

Let's go ahead and find out what numerical variables we have in the dataset_____no_output_____

print('Number of numerical variables: ', len(num_vars))

# visualise the numerical variables
data[num_vars].head()_____no_output_____

## Temporal variables

We have 4 year variables in the dataset:

- YearBuilt: year in which the house was built
- YearRemodAdd: year in which the house was remodeled
- GarageYrBlt: year in which a garage was built
- YrSold: year in which the house was sold

We generally don't use date variables in their raw format. Instead, we extract information from them. For example, we can capture the difference in years between the year the house was built and the year the house was sold._____no_output_____

# list of variables that contain year information

year_vars = [var for var in num_vars if 'Yr' in var or 'Year' in var]

year_vars_____no_output_____# let's explore the values of these temporal variables

for var in year_vars:
    print(var, data[var].unique())
    print()_____no_output_____

As expected, the values are years.

We can explore the evolution of the sale price with the years in which the house was sold:_____no_output_____

# plot median sale price vs year in which it was sold

data.groupby('YrSold')['SalePrice'].median().plot()
plt.ylabel('Median House Price')_____no_output_____

There has been a drop in the value of the houses. That is unusual, in real life, house prices typically go up as years go by.

Let's explore a bit further.

Let's plot the price of sale vs year in which it was built_____no_output_____

# plot median sale price vs year in which it was built

data.groupby('YearBuilt')['SalePrice'].median().plot()
plt.ylabel('Median House Price')_____no_output_____

We can see that newly built / younger houses tend to be more expensive.

Could it be that lately older houses were sold? Let's have a look at that._____no_output_____For this, we will capture the elapsed years between the Year variables and the year in which the house was sold:_____no_output_____

def analyse_year_vars(df, var):

    df = df.copy()

    # capture difference between a year variable and year
    # in which the house was sold
    df[var] = df['YrSold'] - df[var]

    df.groupby('YrSold')[var].median().plot()
    plt.ylabel('Time from ' + var)
    plt.show()

for var in year_vars:
    if var !='YrSold':
        analyse_year_vars(data, var)_____no_output_____

From the plots, we see that towards 2010, the houses sold had older garages, and had not been remodelled recently, that might explain why we see cheaper sales prices in recent years, at least in this dataset.

We can now plot instead the time since last remodelled, or time since built, and sale price, to see if there is a relationship._____no_output_____

def analyse_year_vars(df, var):

    df = df.copy()

    # capture difference between a year variable and year
    # in which the house was sold
    df[var] = df['YrSold'] - df[var]

    plt.scatter(df[var], df['SalePrice'])
    plt.ylabel('SalePrice')
    plt.xlabel(var)
    plt.show()

for var in year_vars:
    if var !='YrSold':
        analyse_year_vars(data, var)_____no_output_____

We see that there is a tendency to a decrease in price, with older houses. In other words, the longer the time between the house was built or remodeled and sale date, the lower the sale Price.

Which makes sense, cause this means that the house will have an older look, and potentially needs repairs._____no_output_____## Discrete variables

Let's go ahead and find which variables are discrete, i.e., show a finite number of values_____no_output_____

#  let's male a list of discrete variables
discrete_vars = [var for var in num_vars if len(
    data[var].unique()) < 20 and var not in year_vars]

print('Number of discrete variables: ', len(discrete_vars))_____no_output_____# let's visualise the discrete variables

data[discrete_vars].head()_____no_output_____

These discrete variables tend to be qualifications (Qual) or grading scales (Cond), or refer to the number of rooms, or units (FullBath, GarageCars), or indicate the area of the room (KitchenAbvGr).

We expect higher prices, with bigger numbers.

Let's go ahead and analyse their contribution to the house price.

MoSold is the month in which the house was sold._____no_output_____

for var in discrete_vars:
    # make boxplot with Catplot
    sns.catplot(x=var, y='SalePrice', data=data, kind="box", height=4, aspect=1.5)
    # add data points to boxplot with stripplot
    sns.stripplot(x=var, y='SalePrice', data=data, jitter=0.1, alpha=0.3, color='k')
    plt.show()_____no_output_____

For most discrete numerical variables, we see an increase in the sale price, with the quality, or overall condition, or  number of rooms, or surface.

For some variables, we don't see this tendency. Most likely that variable is not a good predictor of sale price._____no_output_____## Continuous variables

Let's go ahead and find the distribution of the continuous variables. We will consider continuous variables to all those that are not temporal or discrete._____no_output_____

# make list of continuous variables
cont_vars = [
    var for var in num_vars if var not in discrete_vars+year_vars]

print('Number of continuous variables: ', len(cont_vars))_____no_output_____# let's visualise the continuous variables

data[cont_vars].head()_____no_output_____# lets plot histograms for all continuous variables

data[cont_vars].hist(bins=30, figsize=(15,15))
plt.show()_____no_output_____

The variables are not normally distributed. And there are a particular few that are extremely skewed like 3SsnPorch, ScreenPorch and MiscVal.

Sometimes, transforming the variables to improve the value spread, improves the model performance. But it is unlikely that a transformation will help change the distribution of the super skewed variables dramatically.

We can apply a Yeo-Johnson transformation to variables like LotFrontage, LotArea, BsmUnfSF, and a binary transformation to variables like 3SsnPorch, ScreenPorch and MiscVal.

Let's go ahead and do that._____no_output_____

# first make a list with the super skewed variables
# for later

skewed = [
    'BsmtFinSF2', 'LowQualFinSF', 'EnclosedPorch',
    '3SsnPorch', 'ScreenPorch', 'MiscVal'
]_____no_output_____# capture the remaining continuous variables

cont_vars = [
    'LotFrontage',
    'LotArea',
    'MasVnrArea',
    'BsmtFinSF1',
    'BsmtUnfSF',
    'TotalBsmtSF',
    '1stFlrSF',
    '2ndFlrSF',
    'GrLivArea',
    'GarageArea',
    'WoodDeckSF',
    'OpenPorchSF',
]_____no_output_____

### Yeo-Johnson transformation_____no_output_____

# Let's go ahead and analyse the distributions of the variables
# after applying a yeo-johnson transformation

# temporary copy of the data
tmp = data.copy()

for var in cont_vars:

    # transform the variable - yeo-johsnon
    tmp[var], param = stats.yeojohnson(data[var])

# plot the histograms of the transformed variables
tmp[cont_vars].hist(bins=30, figsize=(15,15))
plt.show()_____no_output_____

For LotFrontage and MasVnrArea the transformation did not do an amazing job.

For the others, the values seem to be spread more evenly in the range.

Whether this helps improve the predictive power, remains to be seen. To determine if this is the case, we should train a model with the original values and one with the transformed values, and determine model performance, and feature importance. But that escapes the scope of this course.

Here, we will do a quick visual exploration here instead:_____no_output_____

# let's plot the original or transformed variables
# vs sale price, and see if there is a relationship

for var in cont_vars:

    plt.figure(figsize=(12,4))

    # plot the original variable vs sale price
    plt.subplot(1, 2, 1)
    plt.scatter(data[var], np.log(data['SalePrice']))
    plt.ylabel('Sale Price')
    plt.xlabel('Original ' + var)

    # plot transformed variable vs sale price
    plt.subplot(1, 2, 2)
    plt.scatter(tmp[var], np.log(tmp['SalePrice']))
    plt.ylabel('Sale Price')
    plt.xlabel('Transformed ' + var)

    plt.show()_____no_output_____

By eye, the transformations seems to improve the relationship only for LotArea.

Let's try a different transformation now. Most variables contain the value 0, and thus we can't apply the logarithmic transformation, but we can certainly do that for the following variables:

 ["LotFrontage", "1stFlrSF", "GrLivArea"]

 So let's do that and see if that changes the variable distribution and its relationship with the target.

 ### Logarithmic transformation_____no_output_____

# Let's go ahead and analyse the distributions of these variables
# after applying a logarithmic transformation

tmp = data.copy()

for var in ["LotFrontage", "1stFlrSF", "GrLivArea"]:

    # transform the variable with logarithm
    tmp[var] = np.log(data[var])

tmp[["LotFrontage", "1stFlrSF", "GrLivArea"]].hist(bins=30)
plt.show()_____no_output_____

The distribution of the variables are now more "Gaussian" looking.

Let's go ahead and evaluate their relationship with the target._____no_output_____

# let's plot the original or transformed variables
# vs sale price, and see if there is a relationship

for var in ["LotFrontage", "1stFlrSF", "GrLivArea"]:

    plt.figure(figsize=(12,4))

    # plot the original variable vs sale price
    plt.subplot(1, 2, 1)
    plt.scatter(data[var], np.log(data['SalePrice']))
    plt.ylabel('Sale Price')
    plt.xlabel('Original ' + var)

    # plot transformed variable vs sale price
    plt.subplot(1, 2, 2)
    plt.scatter(tmp[var], np.log(tmp['SalePrice']))
    plt.ylabel('Sale Price')
    plt.xlabel('Transformed ' + var)

    plt.show()_____no_output_____

The transformed variables have a better spread of the values, which may in turn, help make better predictions.

## Skewed variables

Let's transform them into binary variables and see how predictive they are:_____no_output_____

for var in skewed:

    tmp = data.copy()

    # map the variable values into 0 and 1
    tmp[var] = np.where(data[var]==0, 0, 1)

    # determine mean sale price in the mapped values
    tmp = tmp.groupby(var)['SalePrice'].agg(['mean', 'std'])

    # plot into a bar graph
    tmp.plot(kind="barh", y="mean", legend=False,
             xerr="std", title="Sale Price", color='green')

    plt.show()_____no_output_____

There seem to be a difference in Sale Price in the mapped values, but the confidence intervals overlap, so most likely this is not significant or predictive._____no_output_____# Categorical variables

Let's go ahead and analyse the categorical variables present in the dataset._____no_output_____

print('Number of categorical variables: ', len(cat_vars))_____no_output_____# let's visualise the values of the categorical variables
data[cat_vars].head()_____no_output_____

## Number of labels: cardinality

Let's evaluate how many different categories are present in each of the variables._____no_output_____

# we count unique categories with pandas unique()
# and then plot them in descending order

data[cat_vars].nunique().sort_values(ascending=False).plot.bar(figsize=(12,5))_____no_output_____

All the categorical variables show low cardinality, this means that they have only few different labels. That is good as we won't need to tackle cardinality during our feature engineering lecture.

## Quality variables

There are a number of variables that refer to the quality of some aspect of the house, for example the garage, or the fence, or the kitchen. I will replace these categories by numbers increasing with the quality of the place or room.

The mappings can be obtained from the Kaggle Website. One example:

- Ex = Excellent
- Gd = Good
- TA = Average/Typical
- Fa =	Fair
- Po = Poor_____no_output_____

# re-map strings to numbers, which determine quality

qual_mappings = {'Po': 1, 'Fa': 2, 'TA': 3, 'Gd': 4, 'Ex': 5, 'Missing': 0, 'NA': 0}

qual_vars = ['ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond',
             'HeatingQC', 'KitchenQual', 'FireplaceQu',
             'GarageQual', 'GarageCond',
            ]

for var in qual_vars:
    data[var] = data[var].map(qual_mappings)_____no_output_____exposure_mappings = {'No': 1, 'Mn': 2, 'Av': 3, 'Gd': 4, 'Missing': 0, 'NA': 0}

var = 'BsmtExposure'

data[var] = data[var].map(exposure_mappings)_____no_output_____finish_mappings = {'Missing': 0, 'NA': 0, 'Unf': 1, 'LwQ': 2, 'Rec': 3, 'BLQ': 4, 'ALQ': 5, 'GLQ': 6}

finish_vars = ['BsmtFinType1', 'BsmtFinType2']

for var in finish_vars:
    data[var] = data[var].map(finish_mappings)_____no_output_____garage_mappings = {'Missing': 0, 'NA': 0, 'Unf': 1, 'RFn': 2, 'Fin': 3}

var = 'GarageFinish'

data[var] = data[var].map(garage_mappings)_____no_output_____fence_mappings = {'Missing': 0, 'NA': 0, 'MnWw': 1, 'GdWo': 2, 'MnPrv': 3, 'GdPrv': 4}

var = 'Fence'

data[var] = data[var].map(fence_mappings)_____no_output_____# capture all quality variables

qual_vars  = qual_vars + finish_vars + ['BsmtExposure','GarageFinish','Fence']_____no_output_____# now let's plot the house mean sale price based on the quality of the
# various attributes

for var in qual_vars:
    # make boxplot with Catplot
    sns.catplot(x=var, y='SalePrice', data=data, kind="box", height=4, aspect=1.5)
    # add data points to boxplot with stripplot
    sns.stripplot(x=var, y='SalePrice', data=data, jitter=0.1, alpha=0.3, color='k')
    plt.show()_____no_output_____

For most attributes, the increase in the house price with the value of the variable, is quite clear._____no_output_____

# capture the remaining categorical variables
# (those that we did not re-map)

cat_others = [
    var for var in cat_vars if var not in qual_vars
]

len(cat_others)_____no_output_____

## Rare labels:

Let's go ahead and investigate now if there are labels that are present only in a small number of houses:_____no_output_____

def analyse_rare_labels(df, var, rare_perc):
    df = df.copy()

    # determine the % of observations per category
    tmp = df.groupby(var)['SalePrice'].count() / len(df)

    # return categories that are rare
    return tmp[tmp < rare_perc]

# print categories that are present in less than
# 1 % of the observations

for var in cat_others:
    print(analyse_rare_labels(data, var, 0.01))
    print()_____no_output_____

Some of the categorical variables show multiple labels that are present in less than 1% of the houses.

Labels that are under-represented in the dataset tend to cause over-fitting of machine learning models.

That is why we want to remove them.

Finally, we want to explore the relationship between the categories of the different variables and the house sale price:_____no_output_____

for var in cat_others:
    # make boxplot with Catplot
    sns.catplot(x=var, y='SalePrice', data=data, kind="box", height=4, aspect=1.5)
    # add data points to boxplot with stripplot
    sns.stripplot(x=var, y='SalePrice', data=data, jitter=0.1, alpha=0.3, color='k')
    plt.show()_____no_output_____

Clearly, the categories give information on the SalePrice, as different categories show different median sale prices._____no_output_____**Disclaimer:**

There is certainly more that can be done to understand the nature of this data and the relationship of these variables with the target, SalePrice. And also about the distribution of the variables themselves.

However, we hope that through this notebook we gave you a flavour of what data analysis looks like._____no_output_____# Additional Resources

- [Feature Engineering for Machine Learning](https://www.udemy.com/course/feature-engineering-for-machine-learning/?referralCode=A855148E05283015CF06) - Online Course
- [Packt Feature Engineering Cookbook](https://www.packtpub.com/data/python-feature-engineering-cookbook) - Book
- [Predict house price with Feature-engine](https://www.kaggle.com/solegalli/predict-house-price-with-feature-engine) - Kaggle kernel
- [Comprehensive data exploration with Python](https://www.kaggle.com/pmarcelino/comprehensive-data-exploration-with-python) - Kaggle kernel
- [How I made top 0.3% on a Kaggle competition](https://www.kaggle.com/lavanyashukla01/how-i-made-top-0-3-on-a-kaggle-competition) - Kaggle kernel_____no_output_____


# Notebook from mashhype/LearnAI-DesigningandArchitectingIntelligentAgents
Path: 03-luis/1_session.ipynb

### 3.1: LUIS Design Principles_____no_output_____This session focuses on the design aspects of LUIS with an emphasis on schema design that has been established by conducting engagements with partners and customers. At the end of this session you will be able to:

1. Design Domains and Intents
2. Design Entities
3. Manage Utterances
4. Follow Best Practices_____no_output_____### Section 1: Design Domains and Intents_____no_output_____In this section, you will explore the steps required to design LUIS Domains and Intents, including:

1. Understanding LUIS Domain Options
2. Determining Intents
3. Bot Logic Flow – Intents
4. LUIS Intent Evolution/Roadmap_____no_output_____### Section 1.1: Understanding LUIS Domain Options._____no_output_____![Understanding LUIS Domain Options](./resources/assets/sess_3.1_sect_1.1.jpg)_____no_output_____A LUIS app is usually centered around a domain-specific topic. For example, you may have a travel app that performs booking of tickets, flights, hotels, and rental cars. Another app may provide content related to exercising, tracking fitness efforts and setting goals.

Language Understanding (LUIS) provides prebuilt domains, which are prebuilt sets of intents and entities that work together for domains or common categories of client applications. The prebuilt domains have been pre-trained and are ready for you to add to your LUIS app. You can find a full listing of the prebuilt domains in the Prebuilt domains reference. The intents and entities in a prebuilt domain are fully customizable once you've added them to your app - you can train them with utterances from your system so they work for your users. You can use an entire prebuilt domain as a starting point for customization, or just borrow a few intents or entities from a prebuilt domain.

The intents and entities in a prebuilt domain work best together. It's better to combine intents and entities from the same domain when possible. A best practice is to use related intents from the same domain. For example, if you are customizing the MakeReservation intent in the Places domain, then select the Cancel intent from the Places domain instead of the Cancel intent in the Events or Utilities domains.

* Tip:
Check out the Utilities domain for prebuilt intents that you can customize for use in any domain. For example, you can add Utilities.Repeat to your app and train it recognize whatever actions user might want to repeat in your application.

Alternatively, you can create your own domain with their own set of intents and entities._____no_output_____### Section 1.2: Determining Intents._____no_output_____![Determining Intents](./resources/assets/sess_3.1_sect_1.2.jpg)_____no_output_____Think about the intents that are important to your application’s task. Let's take the example of a conference app, with functions to take bookings and check the agenda at the user's conference. You can define the "BookConference" and "GetAgenda" intents for these actions. In a more complex app with more functions, you will have more intents, and you should define them carefully so as to not be too specific. For example, “GetBookingBalance" and “GetBookStatus" may need to be separate intents, but “GetBookingBalance" and “GetBookStatus" may be too similar.

Using the None Intent.

The None intent is a catch-all or fallback intent. It is used to teach LUIS utterances that are not important in the app domain (subject area).
None intent helps conversation direction. When an utterance is predicted as the None intent and returned to the bot with that prediction, the bot can ask more questions or provide a menu to direct the user to valid choices in the bot.
No utterances in None intent skews predictions. If you do not add any utterances for the None intent, LUIS forces an utterance that is outside the domain into one of the domain intents. This will skew the prediction scores by teaching LUIS the wrong intent for the utterance.

* NOTE
It is a best practice to use only as many intents as you need to perform the functions of your app. If you define too many intents, it becomes harder for LUIS to classify utterances correctly. If you define too few, they may be so general as to be overlapping._____no_output_____### Section 1.3: Bot Logic Flow - Intents._____no_output_____![Bot Logic Flow - Intents](./resources/assets/sess_3.1_sect_1.3.jpg)_____no_output_____The image shows an example of how you can map the use case intents to a conversation flow in more detail. The starting point for the this diagram is from the Bot Logic Flow – High Level design. The intention here is to focus on the conversational flow and you can start to add the supporting technology. In addition, you can annotate the diagram to outline any logic for determining responses and personalization. The Logic Flow does not have to use this format, but it is important to ensure that the format is understood by the intended audience._____no_output_____### Section 1.4: LUIS Intent Evolution/Roadmap._____no_output_____![LUIS Intent Evolution/Roadmap.](./resources/assets/sess_3.1_sect_1.4.jpg)_____no_output_____With a more detailed discovery of the business intents, you can augment the high level bot roadmap with greater detail to include the LUIS intents. _____no_output_____### Section 1.5: Discussion. One intent, or multiple intents. That is the question._____no_output_____![Discussion](./resources/assets/sess_3.1_sect_1.5.jpg)_____no_output_____What would be the pro’s and cons of using two LUIS intents versus one in the example on the slide?

There is no right or wrong answer here.

The key points to pick up from the discussion is that students understand that:

An intent represents the actions that the user wants to perform.

So the question that needs to be answered in the context of the discussion example is whether or not “Get Booking Balance” and “Get Booking Status” are two separate actions or could they be merged as a single action “Get Booking Information”, and secondly what is the pro and con of each approach.

Encourage the students to discuss both options and come to a conclusion._____no_output_____### Section 2: Designing Entities_____no_output_____In this section, you will explore the steps required to design entities, including:

1. Understanding Entities
2. Entity types_____no_output_____### Section 2.1: Understanding Entities._____no_output_____![Understanding Entities](./resources/assets/sess_3.1_sect_2.1.jpg)_____no_output_____* Entities provide the context required to act on an intent

While intents are required, entities are optional. You do not need to create entities for every concept in your app, but only for those where specific information is required for the app to take action. If your utterances do not have details your bot needs to continue, you do not need to add them. As your app matures, you can add them later if necessary. If you are not sure how you would use the information, check out a few of the common prebuilt entities such as datetimeV2, ordinal, email, and phone number.

* Entities represent data

Entities are data you want to pull from the utterance. This can be a name, date, product name, or any group of words. For example in the utterance “Buy a ticket from New York to London on March 5”, three entities are used. Location.Origin, Location.Destination and Prebuilt datetimeV2 which would contain the value “New York”, “London” and “March 5” respectively.

* Entities are shared across intents

Entities are shared among intents. They don't belong to any single intent. Intents and entities can be semantically associated but it is not an exclusive relationship.
In the utterance "Book me a ticket to Paris", "Paris" is an entity of type location. By recognizing the entities that are mentioned in the user’s input, LUIS helps you choose the specific actions to take to fulfill an intent.

* Types of entities

LUIS offers many types of entities as shown in the next graphic._____no_output_____### Section 2.2: Understanding Entity Types._____no_output_____![Understanding Entity Types](./resources/assets/sess_3.1_sect_2.2.jpg)_____no_output_____Entities are important for accomplishing an intent. When you determine which entities to use in your app, keep in mind that there are different types of entities for capturing relationships between types of objects. The following entities exist:

* Simple entity:
A simple entity describes a single concept.

* Hierarchical entity:
A hierarchical entity represents a category and its members.

* Composite entity:
A composite entity is made up of other entities that form parts of a whole.

* List entity:
A list entity is an explicitly specified list of values. Each value consists of one or more synonyms. In a travel app, you might choose to create a list entity to represent airport names.

* Prebuilt entity:
LUIS provides prebuilt entities for common types like Number, which you can use for the number of tickets in a ticket order.

Machine-learned entities perform best when you use Active Learning. You can test via endpoint queries and review endpoint utterances to determine what your model might need more utterances for.
Regular expression entities use the open-source Recognizers-Text project. There are many examples of the regular expressions in the /Specs directory for the supported cultures. If your specific culture or regular expression isn't currently supported, contribute to the project.
Exact-match entities use the text provided in the entity to make an exact text match.

Always think of the users' perceived access for completing a task with your application or bot. Choosing a correct entity type could help your applications fulfill a user request more quickly and fulfill their needs. _____no_output_____### Section 2.3: Discussion - Entities._____no_output_____![Discussion - Entities](./resources/assets/sess_3.1_sect_2.3.jpg)_____no_output_____How would entities help with the above example?

There is no right or wrong answer here.

The key points to pick up from the discussion is that students understand that:

Entities represent the data or parameter aspects of the utterance that will feed the intent.
Entities are optional.

Direct the students to define useful entities that surround booking information and booking status

What other entities could be used in the area?  Booking fees, Booking prices etc._____no_output_____### Section 3: Managing Utterances_____no_output_____In this section, you will explore the practices for managing utterances, including:

1. Utterance Design Guidance
2. Utterance Review Guidance_____no_output_____### Section 3.1: Utterance Design Guidance._____no_output_____![Utterance Design Guidance](./resources/assets/sess_3.1_sect_3.1.jpg)_____no_output_____Utterances are inputs from the user that your app needs to interpret. To train LUIS to extract intents and entities from them, it's important to capture a variety of different inputs for each intent. Active learning, or the process of continuing to train on new utterances, is essential to machine-learned intelligence that LUIS provides.

Collect phrases that you think users will say, and include utterances that mean the same thing but are constructed differently.

* How to choose varied utterances
When you first get started by adding example utterances to your LUIS model, here are some principles to keep in mind.
Utterances aren't always well formed. It may be a sentence, like "Book me a ticket to Paris", or a fragment of a sentence, like "Booking" or "Paris flight." Users often make spelling mistakes. When planning your app, consider whether or not you spell-check user input before passing it to LUIS. The Bing Spell Check API integrates with LUIS. You can associate your LUIS app with an external key for the Bing Spell Check API when you publish it. If you do not spell check user utterances, you should train LUIS on utterances that include typos and misspellings.

* Use the representative language of the user
When choosing utterances, be aware that what you think is a common term or phrase might not be to the typical user of your client application. They may not have domain experience. So be careful when using terms or phrases that a user would only say if they were an expert.

* Choose varied terminology as well as phrasing
You will find that even if you make efforts to create varied sentence patterns, you will still repeat some vocabulary.
Take the following example utterances:

how do I get a computer?
Where do I get a computer?
I want to get a computer, how do I go about it?
When can I have a computer?

The core term here, "computer", is not varied. They could say desktop computer, laptop, workstation, or even just machine. LUIS intelligently infers synonyms from context, but when you create utterances for training, it's still better to vary them.

* Example utterances in each intent
Each intent needs to have example utterances. If you have an intent but do not have any example utterances in that intent, you will not be able to train LUIS. If you have an intent with one or very few example utterances, LUIS will not be able to give accurate predictions.

* Training utterances
Training is non-deterministic: the utterance prediction could vary slightly across versions or apps.

_____no_output_____### Section 3.2: Utterance Review Guidance._____no_output_____![Utterance Review Guidance](./resources/assets/sess_3.1_sect_3.2.jpg)_____no_output_____* Review utterances
After your model is trained, published, and receiving endpoint queries, review the utterances suggested by LUIS. LUIS selects endpoint utterances that have low scores for either the intent or entity. _____no_output_____### Section 3.3: Discussion. Defining Utterances_____no_output_____![Discussion. Defining Utterances](./resources/assets/sess_3.1_sect_3.3.jpg)_____no_output_____Define a set of utterances for each of the intents in the example?

There is no right or wrong answer here.

The key points to pick up from the discussion is that students understand that:

Utterances are input from the user that your app needs to interpret.
Utterances aren’t always well formed
Use the language that represents the users
Choose varied terminology as well as phrasing

Answers will vary, but should consider the issue described above_____no_output_____### Section 4: Best Practices_____no_output_____In this section, you will explore a summary of the following:

1. Best Practices
2. Enterprise Considerations_____no_output_____### Section 4.1: Best Practices._____no_output_____![Best Practices](./resources/assets/sess_3.1_sect_4.1.jpg)_____no_output_____Consider the following best practices:

* Intents
	Create an intent when this intent would trigger an action.
	Intents should be specific.
	If intents are semantically close, consider merging them.

* Entities
	Create when bot needs some parameters or data from the utterance.

* Utterances
	Begin with 10-15 utterances per intent.
	Each utterance should be contextually different.
	The None intent should have between 10-20% of the total utterances_____no_output_____### Section 4.2: Enterprise Considerations._____no_output_____![Enterprise Considerations](./resources/assets/sess_3.1_sect_4.2.jpg)_____no_output_____* LUIS app requests exceeding quotas.

If your LUIS app request rate exceeds the allowed quota rate, spread the load to more LUIS apps with the same app definition. Export the original LUIS app, then import the app back into separate apps. Each app has its own app ID. When you publish, instead of using the same key across all apps, create a separate key for each app. Balance the load across all apps so that no single app is overwhelmed. Add Application Insights to monitor usage.

In order to get the same top intent between all the apps, make sure the intent prediction between the first and second intent is wide enough that LUIS is not confused, giving different results between apps for minor variations in utterances.

Designate a single app as the master. Any utterances that are suggested for review should be added to the master app then moved back to all the other apps. This is either a full export of the app, or loading the labeled utterances from the master to the children. Loading can be done from either the LUIS website or the authoring API for a single utterance or for a batch.
Schedule a periodic review of endpoint utterances for active learning, such as every two weeks, then retrain and republish.

* Wrong intents are returned

if your app is meant to predict a wide variety of user utterances, consider implementing the dispatcher model. The parent app indicates top-level categories of questions. Create a child app for each subcategory. The child app breaks up the subcategory into relevant intents. Breaking up a monolithic app allows LUIS to focus detection between intents successfully instead of getting confused between intents across the top level and intents between the top level and sublevels.
Schedule a periodic review of endpoint utterances for active learning, such as every two weeks, then retrain and republish. _____no_output_____


# Notebook from TonysCousin/Udacity
Path: embedding/Skip_Grams_Exercise.ipynb

# Skip-gram Word2Vec

In this notebook, I'll lead you through using PyTorch to implement the [Word2Vec algorithm](https://en.wikipedia.org/wiki/Word2vec) using the skip-gram architecture. By implementing this, you'll learn about embedding words for use in natural language processing. This will come in handy when dealing with things like machine translation.

## Readings

Here are the resources I used to build this notebook. I suggest reading these either beforehand or while you're working on this material.

* A really good [conceptual overview](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/) of Word2Vec from Chris McCormick
* [First Word2Vec paper](https://arxiv.org/pdf/1301.3781.pdf) from Mikolov et al.
* [Neural Information Processing Systems, paper](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) with improvements for Word2Vec also from Mikolov et al.

---
## Word embeddings

When you're dealing with words in text, you end up with tens of thousands of word classes to analyze; one for each word in a vocabulary. Trying to one-hot encode these words is massively inefficient because most values in a one-hot vector will be set to zero. So, the matrix multiplication that happens in between a one-hot input vector and a first, hidden layer will result in mostly zero-valued hidden outputs.

To solve this problem and greatly increase the efficiency of our networks, we use what are called **embeddings**. Embeddings are just a fully connected layer like you've seen before. We call this layer the embedding layer and the weights are embedding weights. We skip the multiplication into the embedding layer by instead directly grabbing the hidden layer values from the weight matrix. We can do this because the multiplication of a one-hot encoded vector with a matrix returns the row of the matrix corresponding the index of the "on" input unit.

Instead of doing the matrix multiplication, we use the weight matrix as a lookup table. We encode the words as integers, for example "heart" is encoded as 958, "mind" as 18094. Then to get hidden layer values for "heart", you just take the 958th row of the embedding matrix. This process is called an **embedding lookup** and the number of hidden units is the **embedding dimension**.

There is nothing magical going on here. The embedding lookup table is just a weight matrix. The embedding layer is just a hidden layer. The lookup is just a shortcut for the matrix multiplication. The lookup table is trained just like any weight matrix.

Embeddings aren't only used for words of course. You can use them for any model where you have a massive number of classes. A particular type of model called **Word2Vec** uses the embedding layer to find vector representations of words that contain semantic meaning._____no_output_____---
## Word2Vec

The Word2Vec algorithm finds much more efficient representations by finding vectors that represent the words. These vectors also contain semantic information about the words.

Words that show up in similar **contexts**, such as "coffee", "tea", and "water" will have vectors near each other. Different words will be further away from one another, and relationships can be represented by distance in vector space.

There are two architectures for implementing Word2Vec:
>* CBOW (Continuous Bag-Of-Words) and
* Skip-gram

In this implementation, we'll be using the **skip-gram architecture** because it performs better than CBOW. Here, we pass in a word and try to predict the words surrounding it in the text. In this way, we can train the network to learn representations for words that show up in similar contexts._____no_output_____---
## Loading Data

Next, we'll ask you to load in data and place it in the `data` directory

1. Load the [text8 dataset](https://s3.amazonaws.com/video.udacity-data.com/topher/2018/October/5bbe6499_text8/text8.zip); a file of cleaned up *Wikipedia article text* from Matt Mahoney.
2. Place that data in the `data` folder in the home directory.
3. Then you can extract it and delete the archive, zip file to save storage space.

After following these steps, you should have one file in your data directory: `data/text8`._____no_output_____

# read in the extracted text file
with open('data/text8') as f:
    text = f.read()

# print out the first 100 characters
print(text[:100]) anarchism originated as a term of abuse first used against early working class radicals including t

## Pre-processing

Here I'm fixing up the text to make training easier. This comes from the `utils.py` file. The `preprocess` function does a few things:
>* It converts any punctuation into tokens, so a period is changed to ` <PERIOD> `. In this data set, there aren't any periods, but it will help in other NLP problems.
* It removes all words that show up five or *fewer* times in the dataset. This will greatly reduce issues due to noise in the data and improve the quality of the vector representations.
* It returns a list of words in the text.

This may take a few seconds to run, since our text file is quite large. If you want to write your own functions for this stuff, go for it!_____no_output_____

import utils

# get list of words
words = utils.preprocess(text)
print(words[:30])_____no_output_____# print some stats about this word data
print("Total words in text: {}".format(len(words)))
print("Unique words: {}".format(len(set(words)))) # `set` removes any duplicate words_____no_output_____

### Dictionaries

Next, I'm creating two dictionaries to convert words to integers and back again (integers to words). This is again done with a function in the `utils.py` file. `create_lookup_tables` takes in a list of words in a text and returns two dictionaries.
>* The integers are assigned in descending frequency order, so the most frequent word ("the") is given the integer 0 and the next most frequent is 1, and so on.

Once we have our dictionaries, the words are converted to integers and stored in the list `int_words`._____no_output_____

vocab_to_int, int_to_vocab = utils.create_lookup_tables(words)
int_words = [vocab_to_int[word] for word in words]

print(int_words[:30]) #this is the entire text, converted to indices_____no_output_____

## Subsampling

Words that show up often such as "the", "of", and "for" don't provide much context to the nearby words. If we discard some of them, we can remove some of the noise from our data and in return get faster training and better representations. This process is called subsampling by Mikolov. For each word $w_i$ in the training set, we'll discard it with probability given by

$$ P(w_i) = 1 - \sqrt{\frac{t}{f(w_i)}} $$

where $t$ is a threshold parameter and $f(w_i)$ is the frequency of word $w_i$ in the total dataset.

$$ P(0) = 1 - \sqrt{\frac{1*10^{-5}}{1*10^6/16*10^6}} = 0.98735 $$

I'm going to leave this up to you as an exercise. Check out my solution to see how I did it.

> **Exercise:** Implement subsampling for the words in `int_words`. That is, go through `int_words` and discard each word given the probablility $P(w_i)$ shown above. Note that $P(w_i)$ is the probability that a word is discarded. Assign the subsampled data to `train_words`._____no_output_____

from collections import Counter
import random
import numpy as np

threshold = 1e-5
word_counts = Counter(int_words)
print("word_counts size = ", len(word_counts), ", first items = ", word_counts.most_common(3), ", sample = ", word_counts[2])
print(list(word_counts.items())[0])  # dictionary of int_words, how many times they appear

# discard some frequent words, according to the subsampling equation
# create a new list of words for training
train_words = []
num_words = len(words)
for wi in int_words:
    p_discard = 1.0 - np.sqrt(threshold / (word_counts[wi]/num_words))
    r = random.uniform(0, 1)
    if r > p_discard:
        train_words.append(wi)

print("full text size = ", num_words, ", training words size = ", len(train_words))
print(train_words[:30])_____no_output_____

## Making batches_____no_output_____Now that our data is in good shape, we need to get it into the proper form to pass it into our network. With the skip-gram architecture, for each word in the text, we want to define a surrounding _context_ and grab all the words in a window around that word, with size $C$.

From [Mikolov et al.](https://arxiv.org/pdf/1301.3781.pdf):

"Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples... If we choose $C = 5$, for each training word we will select randomly a number $R$ in range $[ 1: C ]$, and then use $R$ words from history and $R$ words from the future of the current word as correct labels."

> **Exercise:** Implement a function `get_target` that receives a list of words, an index, and a window size, then returns a list of words in the window around the index. Make sure to use the algorithm described above, where you chose a random number of words to from the window.

Say, we have an input and we're interested in the idx=2 token, `741`:
```
[5233, 58, 741, 10571, 27349, 0, 15067, 58112, 3580, 58, 10712]
```

For `R=2`, `get_target` should return a list of four values:
```
[5233, 58, 10571, 27349]
```_____no_output_____

def get_target(words, idx, window_size=5):
    ''' Get a list of words in a window around an index. '''
    ''' The window will be randomly sized, up to window_size. '''

    window = random.randint(1, window_size+1)
    start = idx - window
    end = idx + window + 1

    #handle if we're near the beginning or end of the word list
    if start < 0:
        start = 0
    if end > len(words):
        end = len(words)

    tgt = []
    for i in range(start, idx):
        tgt.append(words[i])
    for i in range(idx+1, end): #skip the word of interest itself
        tgt.append(words[i])

    return tgt_____no_output_____# test your code!

# run this cell multiple times to check for random window selection
int_text = [i for i in range(10)]
print('Input: ', int_text)
idx=4 # word index of interest

target = get_target(int_text, idx=idx, window_size=5)
print('Target: ', target)  # you should get some indices around the idx_____no_output_____

### Generating Batches

Here's a generator function that returns batches of input and target data for our model, using the `get_target` function from above. The idea is that it grabs `batch_size` words from a words list. Then for each of those batches, it gets the target words in a window._____no_output_____

def get_batches(words, batch_size, window_size=5):
    ''' Create a generator of word batches as a tuple (inputs, targets) '''

    n_batches = len(words)//batch_size

    # only full batches
    words = words[:n_batches*batch_size]

    for idx in range(0, len(words), batch_size):
        x, y = [], []
        batch = words[idx:idx+batch_size]
        for ii in range(len(batch)):
            batch_x = batch[ii]
            batch_y = get_target(batch, ii, window_size)
            y.extend(batch_y)
            x.extend([batch_x]*len(batch_y))
        yield x, y
    _____no_output_____int_text = [i for i in range(20)]
x,y = next(get_batches(int_text, batch_size=4, window_size=5))

print('x\n', x)
print('y\n', y)_____no_output_____

## Building the graph

Below is an approximate diagram of the general structure of our network.

>* The input words are passed in as batches of input word tokens.
* This will go into a hidden layer of linear units (our embedding layer).
* Then, finally into a softmax output layer.

We'll use the softmax layer to make a prediction about the context words by sampling, as usual.

The idea here is to train the embedding layer weight matrix to find efficient representations for our words. We can discard the softmax layer because we don't really care about making predictions with this network. We just want the embedding matrix so we can use it in _other_ networks we build using this dataset._____no_output_____---
## Validation

Here, I'm creating a function that will help us observe our model as it learns. We're going to choose a few common words and few uncommon words. Then, we'll print out the closest words to them using the cosine similarity:

$$
\mathrm{similarity} = \cos(\theta) = \frac{\vec{a} \cdot \vec{b}}{|\vec{a}||\vec{b}|}
$$

We can encode the validation words as vectors $\vec{a}$ using the embedding table, then calculate the similarity with each word vector $\vec{b}$ in the embedding table. With the similarities, we can print out the validation words and words in our embedding table semantically similar to those words. It's a nice way to check that our embedding table is grouping together words with similar semantic meanings._____no_output_____

def cosine_similarity(embedding, valid_size=16, valid_window=100, device='cpu'):
    """ Returns the cosine similarity of validation words with words in the embedding matrix.
        Here, embedding should be a PyTorch embedding module.
    """

    # Here we're calculating the cosine similarity between some random words and
    # our embedding vectors. With the similarities, we can look at what words are
    # close to our random words.

    # sim = (a . b) / |a||b|

    embed_vectors = embedding.weight

    # magnitude of embedding vectors, |b|
    magnitudes = embed_vectors.pow(2).sum(dim=1).sqrt().unsqueeze(0)

    # pick N words from our ranges (0,window) and (1000,1000+window). lower id implies more frequent
    valid_examples = np.array(random.sample(range(valid_window), valid_size//2))
    valid_examples = np.append(valid_examples,
                               random.sample(range(1000,1000+valid_window), valid_size//2))
    valid_examples = torch.LongTensor(valid_examples).to(device)

    valid_vectors = embedding(valid_examples)
    similarities = torch.mm(valid_vectors, embed_vectors.t())/magnitudes

    return valid_examples, similarities_____no_output_____

## SkipGram model

Define and train the SkipGram model.
> You'll need to define an [embedding layer](https://pytorch.org/docs/stable/nn.html#embedding) and a final, softmax output layer.

An Embedding layer takes in a number of inputs, importantly:
* **num_embeddings** – the size of the dictionary of embeddings, or how many rows you'll want in the embedding weight matrix
* **embedding_dim** – the size of each embedding vector; the embedding dimension_____no_output_____

import torch
from torch import nn
import torch.optim as optim_____no_output_____class SkipGram(nn.Module):
    def __init__(self, n_vocab, n_embed):
        super().__init__()

        # complete this SkipGram model
        print("Entering SkipGram.init: n_vocab = ", n_vocab, ", n_embed = ", n_embed)
        self.emb = nn.Embedding(n_vocab, n_embed)
        self.output = nn.Linear(n_embed, n_vocab)
        self.ls = nn.LogSoftmax(dim=1)

    def forward(self, x):

        # define the forward behavior
        #print("Entering SkipGram.forward. input x shape = ", x.shape)
        x = self.emb(x)    #no activation on this layer; output is n_vocab x n_embed
        #print("    Embedding layer complete. x shape = ", x.shape)
        out = self.output(x)
        #print("    FC layer complete. out shape = ", out.shape)
        x = self.ls(out) #output is same dimensions as input
        #print("    returning x. size = ", x.shape)

        return x_____no_output_____

### Training

Below is our training loop, and I recommend that you train on GPU, if available.

**Note that, because we applied a softmax function to our model output, we are using NLLLoss** as opposed to cross entropy. This is because Softmax  in combination with NLLLoss = CrossEntropy loss ._____no_output_____

# check if GPU is available
device = 'cuda' if torch.cuda.is_available() else 'cpu'

embedding_dim=300 #jas - small to check for memory errors

model = SkipGram(len(vocab_to_int), embedding_dim).to(device)
print("Model structure is:\n", model, end="\n")

criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003) #jas torch docs for Embedding warn against using Adam

print_every = 500
steps = 0
epochs = 4 #limiting cuz this trains for a long time!

# train for some number of epochs
for e in range(epochs):

    # get input and target batches
    for inputs, targets in get_batches(train_words, 512):
        steps += 1
        inputs, targets = torch.LongTensor(inputs), torch.LongTensor(targets)
        inputs, targets = inputs.to(device), targets.to(device)
        #print("inputs/targets shapes = ", inputs.shape, targets.shape)

        log_ps = model(inputs)
        #print("log_ps type/shape = ", type(log_ps))
        loss = criterion(log_ps, targets)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if steps % print_every == 0:
            print("\n///// Step ", steps)
            # getting examples and similarities
            valid_examples, valid_similarities = cosine_similarity(model.emb, device=device)
            _, closest_idxs = valid_similarities.topk(6) # topk highest similarities

            valid_examples, closest_idxs = valid_examples.to('cpu'), closest_idxs.to('cpu')
            for ii, valid_idx in enumerate(valid_examples):
                closest_words = [int_to_vocab[idx.item()] for idx in closest_idxs[ii]][1:]
                print(int_to_vocab[valid_idx.item()] + " | " + ', '.join(closest_words))
            print("...")Entering SkipGram.init: n_vocab =  63641 , n_embed =  300

///// Step  300
all | tem, heijenoort, moviegoers, conceptions, popolare
state | chani, yah, azathoth, bastien, erroneous
their | derisively, plenty, manche, kraepelin, nutritionists
th | teague, iab, vy, corp, ratzenberger
zero | garagiola, moise, va, decompressed, sorensen
over | gildas, guessing, usgs, departure, smoother
these | ahenobarbus, occupants, skim, noche, savarkar
states | herald, cleve, few, motilal, willow
channel | evolutionarily, sanctioning, zara, auditory, rebelling
cost | medicine, gracen, mingle, marinated, bata
marriage | heartbeat, paramount, confucianist, dugan, lunatics
centre | baasha, wavre, jurors, sterg, faa
existence | spatially, eerdmans, nodal, heist, rainbow
professional | falkland, michael, nonstandard, harlequin, busta
behind | planck, flashpoint, beavers, polemical, depleted
award | reverted, referred, niv, pantheist, sandilands
...

///// Step  600
had | voip, mand, devils, mush, supporters
world | tzow, substitutes, hanasi, audiogalaxy, herrmann
all | tem, heijenoort, moviegoers, conceptions, popolare
can | kn, alkane, fra, perpetrator, arnhem
system | mex, kindness, sain, falklands, aspx
into | centigrade, aluminium, speciesism, emitter, sharply
only | summoner, divers, amathus, mechanically, callimico
however | chivalry, zy, look, ugandan, pasts
police | fluorspar, sigurd, okinawan, starship, messiah
construction | extremities, hewson, hoa, adelaide, hallucinogenic
older | ltspkr, unwieldy, probability, madhya, expect
road | msp, immediate, emptiness, coldness, albertine
additional | gopala, tugs, poll, patrons, grandparent
experience | elf, carcinogenic, goncharov, wits, sunflower
something | butler, winfield, niro, disinterested, arian
consists | trivia, bundesverfassungsgericht, compl, semiconducting, cone
...

///// Step  900
when | hoop, molars, cordite, sheave, putsch
war | decorating, debbie, musical, lothal, sahrawi
the | benito, adolph, leicestershire, facilitate, sula
b | joliet, restaurateur, nucleon, clementi, jawed
all | tem, heijenoort, moviegoers, conceptions, popolare
with | vres, gestae, silica, incompatibilities, synods
if | imprisoned, seton, metta, lexicographic, leyland
its | fittingly, experimentation, elias, integrative, cryptanalyst
scale | recognitions, paralleled, holders, graphics, anglicans
road | emptiness, msp, gail, immediate, stomach
bbc | harar, famicom, mobilize, merlini, racehorse
liberal | tracking, repay, cohesive, residents, stacy
instance | suzy, timor, multipliers, denotes, su
mathematics | miquelon, cambrian, jagiellonian, homeowners, registering
behind | planck, flashpoint, polemical, beavers, females
existence | spatially, eerdmans, nodal, arb, heist
...

///// Step  1200
b | joliet, armin, jawed, nucleon, restaurateur
all | tem, heijenoort, moviegoers, conceptions, rate
they | proteus, comma, jovi, leninist, humphry
not | canaris, lambasted, constructs, vestments, bollywood
years | aztlan, researched, cisternae, texture, bans
most | romeo, bundy, cma, corda, downer
may | special, strikingly, bartley, auditions, bengals
i | lore, roz, wc, brownlow, reddish
marriage | heartbeat, paramount, receipts, dugan, lunatics
account | gemara, novorossiysk, mmu, kendo, gana
discovered | vines, ape, homicides, erlewine, kwacha
mainly | niro, tremolo, presley, unintended, lup
mean | tenochtitl, protects, ek, immigrating, socialista
construction | extremities, hewson, hallucinogenic, lagrange, hoa
freedom | ballcarrier, gorges, collide, stews, qubit
alternative | manga, krajina, dyson, uneasy, joschka
...

///// Step  1500
new | surrounding, formations, ceeac, vocoder, peculiarly
years | aztlan, researched, cisternae, texture, bans
not | canaris, lambasted, constructs, bollywood, reflection
people | capi, stein, perpignan, contractually, disarm
may | strikingly, special, taner, bartley, auditions
had | voip, mand, tyrolean, devils, habitual
s | summers, cline, hanafi, porfirio, leven
or | rainstorm, guerillas, eph, terminally, opencourseware
instance | suzy, multipliers, aisle, timor, primitivism
quite | schuler, zooming, chobert, newborns, railways
articles | killian, absinthium, oates, novello, sofie
units | enda, lutenists, sponsored, expandability, oh
freedom | ballcarrier, gorges, collide, stews, qubit
running | positivists, postulated, yue, kickoff, diez
arts | icq, tractor, fascists, barbarians, wolfowitz
prince | oregon, authoring, decreasing, rgy, prompt
...

///// Step  1800
during | forestalled, geometer, pans, slag, vestiges
d | nss, farmhouse, schisms, popularising, ritually
if | leyland, imprisoned, lexicographic, metta, seton
these | skim, occupants, ahenobarbus, ritual, savarkar
many | lsap, intersects, brindisi, calico, minotaur
not | canaris, lambasted, bollywood, constructs, reflection
years | aztlan, cisternae, researched, texture, bans
they | proteus, jovi, comma, leninist, kb
proposed | wakko, collectively, syst, hydrodynamic, rod
something | niro, butler, winfield, hive, arian
freedom | ballcarrier, gorges, stews, collide, qubit
stage | annette, revels, nilgiri, singularities, eridanus
except | pedophile, bornemisza, seine, column, oiled
quite | schuler, zooming, chobert, newborns, spoke
engine | pmsd, fsn, quartic, piron, anka
powers | saint, shanty, slender, jobs, businesses
...

///// Step  2100
on | yttrium, negate, lan, galilean, anni
can | kn, perpetrator, iceland, alkane, arnhem
five | bohemia, serially, three, geller, normanby
american | abbr, duvalier, apa, firmer, lithograph
called | circulation, irritability, lonely, andover, hyundai
war | decorating, debbie, lothal, sahrawi, dandyism
where | sihamoni, esb, panel, disparage, fastened
as | sextilis, cyndi, diam, nikolai, undetermined
recorded | dancers, anomalistic, bunter, distributing, joel
pre | dih, quine, kimi, balkan, authors
shows | bodhisattvas, dubliners, isp, kassovitz, hci
operations | prophase, longevity, beckenbauer, mechanisation, zoo
governor | badly, kerala, gift, urges, attica
units | enda, lutenists, sponsored, expandability, oh
shown | outlines, auckland, liff, browns, kernewek
older | ltspkr, unwieldy, probability, scalia, sophists
...

///// Step  2400
time | bottle, codebreaking, dagon, cds, cripple
all | tem, heijenoort, moviegoers, sparingly, misadventures
three | five, casuistry, wynn, christa, solvers
nine | hammadi, five, mailbox, lewiston, costa
in | enlarged, intersects, blitzkrieg, puzzle, gottlob
use | bracelet, foreskin, connolly, ensign, tasman
had | voip, mand, tyrolean, supporters, suitor
years | aztlan, cisternae, researched, bans, texture
older | ltspkr, probability, unwieldy, sophists, scalia
except | pedophile, bornemisza, column, oiled, seine
square | curtiss, barratry, cosines, comoving, irreparable
universe | phases, diphtheria, cole, triad, fmd
smith | illyrian, tuff, hennepin, haggard, husayni
operating | nationalized, drag, matchups, handfasting, transmissive
engineering | bending, mood, terrorized, btr, blocker
ice | stereogram, empted, bologna, grandparent, accumulate
...

///// Step  2700
be | plenary, protectors, pardo, cried, answered
five | three, bohemia, geller, marin, normanby
from | ashore, assemblage, reliever, yasuo, verner
was | matches, emigrated, corfu, cmt, educate
often | bunch, disenchanted, asis, protagonists, howerd
its | fittingly, experimentation, elias, cryptanalyst, integrative
first | feet, azo, repayments, hopf, webmasters
so | pseudepigrapha, disgusting, amphibian, stint, temair
additional | gopala, grandparent, shamanistic, voir, rowed
bible | crematoria, corrino, slapstick, hostname, christians
resources | hearses, anxious, subsidize, vacated, mitigated
quite | zooming, chobert, schuler, newborns, ssl
prince | authoring, oregon, rgy, lyne, prodigal
taking | awarding, asahi, plastids, precarious, susumu
ocean | pili, opens, ien, regional, untested
mathematics | miquelon, cambrian, vivisection, watford, plows
...

///// Step  3000
b | attainable, vba, joliet, jawed, armin
most | romeo, corda, bundy, devonian, shillings
however | zy, chivalry, look, ugandan, chara
from | ashore, assemblage, reliever, verner, yasuo
he | proclaims, mcfarlane, scouting, barry, usury
or | variation, guerillas, rainstorm, opencourseware, spacesuit
been | nod, hilliard, wordsworth, assess, compaction
this | outta, respite, cfo, distracted, hedda
nobel | vibe, observances, transoxiana, parkinson, playwright
pre | dih, quine, balkan, kimi, assemble
mathematics | miquelon, cambrian, vivisection, plows, superset
institute | mcpherson, zen, mxy, wrists, boka
taking | awarding, asahi, plastids, precarious, susumu
issue | restyled, trim, intentionalists, durr, refurbishment
construction | extremities, trilobites, hewson, ict, hoa
scale | recognitions, paralleled, graphics, holders, novelties
...

///// Step  3300
over | gildas, departure, smoother, gendarmerie, guessing
where | sihamoni, panel, furrows, disparage, esb
a | prefects, meconium, dispelled, folds, poker
war | lothal, debbie, decorating, dandyism, sahrawi
first | feet, hopf, azo, repayments, webmasters
no | gastronomique, overpower, fibers, samuelson, hybrids
he | proclaims, mcfarlane, scouting, having, perceive
zero | va, garagiola, driest, moise, nonnus
nobel | vibe, observances, transoxiana, parkinson, playwright
primarily | prioritize, oracle, prototypes, benefiting, lilongwe
pope | pleats, pius, theological, negates, systran
egypt | kayaks, behaviorism, leprosy, aronson, ordered
know | progress, grabs, torvalds, autocode, commutative
units | enda, sponsored, lutenists, expandability, unsteady
cost | medicine, gracen, kyushu, tonnes, vaccinated
police | philanthropy, fluorspar, oscillatory, sigurd, canton
...

///// Step  3600
an | unleashes, tanlin, goodwrench, cistern, sparrows
has | tty, buda, obedience, porridge, icelandic
from | ashore, reliever, assemblage, verner, petroleum
by | peak, glutathione, pretext, restart, chaim
a | prefects, meconium, poker, carver, dispelled
had | mand, tyrolean, devils, voip, supporters
on | yttrium, negate, novak, galilean, anni
into | leveraged, centigrade, aluminium, oeuvre, speciesism
taking | awarding, plastids, asahi, precarious, promotional
gold | heterotic, mahavira, carnivals, osceola, nickel
stage | annette, singularities, revels, graphemes, eridanus
prince | oregon, authoring, rgy, lyne, prodigal
applications | tanzania, generally, osmotic, dri, transmissive
accepted | groups, ancient, iglesias, monastics, cundinamarca
additional | gopala, voir, grandparent, patrons, shamanistic
experience | elf, carcinogenic, goncharov, orthonormal, wits
...

///// Step  3900
was | convalescence, matches, soreness, obj, evidentiary
american | illinois, duvalier, abbr, lithograph, firmer
also | twiddle, mastelotto, quarto, attain, soliciting
four | crashed, two, gerlach, pcs, necked
from | ashore, reliever, assemblage, guerra, petroleum
who | jeffreys, mosques, persians, cubana, robe
their | zeev, plenty, hoops, gonna, pots
united | rastafari, rrs, astronautica, northwest, xf
behind | polemical, females, planck, flashpoint, depleted
instance | denotes, multipliers, mayne, radiating, wood
assembly | libertine, needless, dissent, aage, vinge
joseph | teaching, jade, convertibility, dukes, ayatollah
notes | automobili, discuss, besieges, gorgosaurus, lytic
mainly | unintended, niro, tremolo, streets, pares
units | enda, sponsored, lutenists, expandability, unsteady
numerous | infirmity, cybernetica, disappeared, priming, recombinant
...

///// Step  4200
there | blog, homeschooling, centerline, redrawing, friesian
this | outta, unadorned, expresses, respite, cfo
people | capi, responded, illyrians, perpignan, disarm
and | seen, blockade, antisymmetric, stereogram, nubians
three | five, christa, wynn, solvers, fujitsu
its | cryptanalyst, integrative, fittingly, pullback, experimentation
that | subtract, they, deleted, playboys, pneumonic
system | falklands, attuned, kindness, disperse, dicaprio
gold | mahavira, heterotic, nickel, carnivals, osceola
hold | audiences, danmark, muons, pythagoreans, curd
bbc | harar, famicom, merlini, mobilize, geology
question | formalism, alomari, financiers, calmed, slammed
alternative | uneasy, diverged, manga, dct, joschka
test | renaissance, conomique, localized, mentioned, diophantus
behind | females, polemical, depleted, planck, flashpoint
troops | trumbo, sneeze, flourished, exactions, sanderson
...

///// Step  4500
but | agrarianism, sophia, muzzleloaders, sited, concentrating
have | oshii, bisexuals, scientism, cannibalistic, parishes
b | historian, earl, vba, edo, attainable
can | perpetrator, kn, uneven, encyclopedic, para
all | tem, heijenoort, generate, moviegoers, popolare
years | aztlan, cisternae, dubs, researched, bans
zero | five, nine, one, va, driest
over | departure, gildas, guessing, cumbersome, gendarmerie
units | enda, sponsored, lutenists, expandability, sulfur
http | chinaman, merrimac, wanted, ipod, signer
gold | mahavira, heterotic, nickel, osceola, carnivals
defense | moussa, leathers, freedoms, plantation, hainan
question | alomari, formalism, slammed, unpaid, calmed
numerous | infirmity, disappeared, cybernetica, priming, beasts
square | curtiss, cosines, barratry, comoving, lugano
frac | slam, simplified, mycobacterium, exp, briand
...

///// Step  4800
a | meconium, prefects, dispelled, poker, dannebrog
state | bureau, chani, yah, azathoth, visitation
new | edmonton, sergio, hoppers, ferroelectric, surrounding
four | two, crashed, five, six, three
about | americium, hoax, biscuits, duomo, airs
no | overpower, samuelson, gastronomique, hybrids, macrobiotics
for | capacities, bypassing, czes, dachshund, tamer
i | lore, roz, brownlow, delia, ndp
numerous | infirmity, disappeared, cybernetica, beasts, priming
rise | iberians, merciful, carnivorous, roi, after
ice | stereogram, accumulate, programmability, tertiary, positivists
pope | pleats, theological, pius, negates, drugged
engine | fsn, canute, pmsd, tar, quartic
engineering | mood, btr, bending, overcharged, hierarchs
file | newsted, redeemer, duane, regionally, murdoc
brother | atreus, amn, bushido, sahrawi, hallowicked
...

///// Step  5100
at | hike, hummingbird, lapland, arras, falsifying
between | maneaba, infielders, bedin, uncomfortable, usually
he | mcfarlane, proclaims, disclose, perceive, scouting
has | tty, buda, porridge, icelandic, and
than | implicated, reasonably, unions, saarc, funding
be | plenary, actual, answered, defining, protectors
which | creation, kept, many, slugged, hawkeye
new | edmonton, ferroelectric, hoppers, sergio, surrounding
event | scalemajor, reasserted, hash, ayeyarwady, noriega
hit | ornament, cancelled, tracker, shorebirds, staten
instance | denotes, multipliers, mayne, radiating, wood
alternative | diverged, joschka, uneasy, manga, accidently
mainly | unintended, tremolo, streets, lup, pares
units | enda, sponsored, lutenists, unsteady, corps
joseph | teaching, jade, convertibility, dukes, ayatollah
centre | wavre, baasha, faa, airs, misinterpretation
...

///// Step  5400
th | teague, iab, liking, vy, asturias
were | melzi, lucasarts, surrender, aurelius, unification
two | four, six, comorian, queso, spartan
war | debbie, lothal, sahrawi, forces, dandyism
some | cosmos, exemplars, cute, among, lethargy
may | special, strikingly, padma, notated, taner
three | five, christa, six, four, nine
eight | six, usurper, alliance, five, klimt
channel | evolutionarily, sanctioning, lakeside, callistus, zara
discovered | erlewine, vines, ape, homicides, preventable
professional | soccer, romanizations, harlequin, michael, farmington
question | alomari, formalism, slammed, unpaid, computus
san | cessna, sledges, afghan, aborigines, puritanism
governor | parliamentary, rodgers, kerala, urges, dred
liberal | cohesive, repay, deed, aurelianus, stacy
units | enda, sponsored, lutenists, unsteady, kaohsiung
...

///// Step  5700
at | hummingbird, hike, dynastic, arras, lapland
some | among, lethargy, cosmos, exemplars, adjuncts
united | astronautica, rastafari, rrs, xf, refuses
up | cranes, baar, oases, arabica, spat
however | zy, chivalry, look, dcl, ugandan
if | must, musicologists, materialize, quo, lexicographic
for | capacities, bypassing, stepfather, afrique, dachshund
are | paddler, so, derivation, biking, stabilizes
governor | parliamentary, rodgers, dred, kerala, urges
stage | annette, graphemes, hopping, singularities, pile
frac | slam, exp, endomorphisms, mycobacterium, briand
institute | mcpherson, moshiach, weinstein, redeveloped, mxy
applications | tanzania, osmotic, transmissive, generally, dri
notes | automobili, besieges, hyginus, gch, discuss
assembly | dissent, libertine, right, austrians, needless
centre | wavre, baasha, faa, airs, misinterpretation
...

///// Step  6000
or | variation, guerillas, rainstorm, policemen, ethnic
so | are, necho, combatants, jurors, amphibian
for | capacities, bypassing, dachshund, competitions, stepfather
may | special, merely, harding, freshly, irony
no | overpower, prix, samuelson, ambulance, outweighs
time | bottle, codebreaking, mogadishu, dagon, shedding
he | proclaims, delian, scouting, disclose, picchi
use | bracelet, tierney, megabits, blackheath, demonstrably
woman | humility, born, avocats, unpaved, deathbed
event | scalemajor, reasserted, hash, noriega, ayeyarwady
question | alomari, slammed, formalism, unpaid, attic
bill | challenges, klaviertrio, disjointed, borland, grateful
operations | prophase, longevity, chamorro, restorative, zoo
pressure | instinctively, presupposes, fraser, archipelago, skeletal
report | chupacabra, ensures, medallist, unsuited, tridactyla
pre | dih, balkan, sicilian, prokaryote, assemble
...

///// Step  6300
may | special, merely, harding, arbitrary, taner
time | bottle, dagon, cripple, codebreaking, shedding
their | plenty, zeev, undermining, disgruntled, competence
b | edo, astrophysical, earl, vba, attainable
so | are, necho, combatants, amphibian, jurors
united | astronautica, rastafari, refuses, rrs, xf
all | tem, heijenoort, generate, astbury, moviegoers
about | duomo, hoax, biscuits, americium, tedious
shows | isp, bodhisattvas, hassle, dubliners, oliver
cost | gracen, medicine, cern, chadic, kyushu
pressure | instinctively, presupposes, fraser, archipelago, crystallizes
ice | stereogram, tertiary, accumulate, programmability, awash
behind | females, polemical, gnaeus, pattern, beavers
pre | dih, balkan, assemble, sicilian, prokaryote
placed | erect, nanette, flannel, serene, dom
older | unwieldy, chronology, expect, probability, localization
...

///// Step  6600
nine | five, zero, three, one, four
some | lethargy, exemplars, among, restrict, diverse
be | actual, answered, case, plenary, distinguishes
s | homomorphism, gilbert, summers, czechs, stowaway
zero | five, one, nine, four, two
many | lsap, which, intersects, diets, silo
if | must, lexicographic, quo, neighbor, additive
will | spacesuit, phine, methylated, mimics, boiling
units | sponsored, enda, kaohsiung, corps, unsteady
lived | stewie, sublayer, hayt, lurker, affection
behind | females, polemical, gnaeus, fulness, occasions
discovered | erlewine, homicides, vines, ape, discovery
police | sociolinguistic, arafat, canton, incensed, brigade
engineering | research, international, bm, btr, babbage
report | ensures, medallist, chupacabra, competence, unsuited
bbc | harar, merlini, famicom, mobilize, protocol
...

///// Step  6900
nine | one, five, three, zero, four
had | lucio, tyrolean, mand, italy, steinberg
two | four, six, three, five, zero
th | teague, iab, asturias, tte, reorganisation
its | borough, integrative, fiedler, fittingly, pullback
states | herald, tyrants, candidates, exorcism, presidency
five | three, zero, nine, four, two
new | surrounding, edmonton, hoppers, kline, gavin
event | scalemajor, reasserted, hash, ayeyarwady, noriega
versions | tp, rhapsody, dorian, configurable, wiki
units | sponsored, enda, lutenists, beeb, kaohsiung
numerous | infirmity, disappeared, priming, cybernetica, satyrs
know | grabs, torvalds, martyrdom, cakes, pronounces
primarily | prototypes, prioritize, oracle, benefiting, appreciable
pressure | fraser, presupposes, instinctively, archipelago, crystallizes
animals | crops, oratio, slavoj, fk, jadavpur
...

///// Step  7200
american | abbr, illinois, hathaway, crouch, gravitated
were | lucasarts, aurelius, midwestern, melzi, unhappy
up | cranes, oases, baar, arabica, concluded
called | lonely, griseus, desargues, prophets, luanda
than | implicated, reasonably, flatness, aux, jains
on | the, first, israel, negate, novak
not | bollywood, deliberately, eagerness, that, traditions
war | lothal, debbie, sahrawi, forces, nesbitt
animals | crops, slavoj, jadavpur, mankind, fungus
pre | dih, balkan, narrative, brittany, sicilian
placed | erect, nanette, whichever, dom, welland
powers | saint, slender, redefine, held, ouverture
behind | females, gnaeus, polemical, pattern, beavers
operating | port, nationalized, matchups, machines, kuroda
gold | nickel, osceola, carnivals, heterotic, mahavira
applied | stereochemistry, onslaught, denotation, concluding, comit
...

///// Step  7500
they | that, leninist, puritanical, seeing, cetera
one | nine, zero, four, two, three
who | mosques, jeffreys, persians, marriage, watani
as | diam, metaphor, westerners, banded, nikolai
its | borough, kalb, fiedler, tightly, stereotyping
american | abbr, illinois, hathaway, inventors, gravitated
called | luanda, griseus, insurgencies, irritability, borwein
while | offices, misunderstood, duty, acrobatics, drover
stage | annette, graphemes, chest, hopping, animated
marriage | daughter, caeu, lunatics, who, annuals
versions | tp, dorian, rhapsody, configurable, wiki
know | grabs, torvalds, martyrdom, pronounces, cakes
bill | grateful, iai, klaviertrio, challenges, borland
proposed | syst, collectively, acrimonious, auburn, voyaging
troops | trumbo, pows, sneeze, indians, rugova
centre | wavre, airs, baasha, backwaters, misinterpretation
...

///// Step  7800
its | borough, tightly, kalb, fiedler, neighborhood
can | uneven, perpetrator, conserved, or, lambar
these | frud, skim, nursultan, agnosticism, traditional
time | bottle, stunned, started, maybe, mogadishu
four | two, three, six, zero, five
been | reappear, hilliard, purposes, testified, nod
two | four, six, three, zero, seven
more | violent, corinthians, accolade, sonata, hooters
hit | cancelled, subgenre, tracker, grooves, rebar
ocean | ien, archipelago, pili, agnarr, opens
stage | annette, graphemes, chest, hopping, animated
magazine | tootsie, prequels, siskel, lou, sponsorships
egypt | aronson, ordered, diplomats, kayaks, colonised
older | unwieldy, chronology, localization, expect, household
bible | corrino, preach, gaulish, statement, christians
pre | dih, balkan, prokaryote, assemble, brittany
...

///// Step  8100
were | lucasarts, midwestern, imposed, melzi, aurelius
i | brownlow, lore, jn, letters, roz
been | reappear, hilliard, nod, purposes, lump
that | they, subtract, argument, not, possible
three | five, two, six, four, nine
five | three, zero, nine, two, four
for | competitions, capacities, trabzon, a, choose
d | farmhouse, eds, l, heywood, nss
versions | tp, configurable, dorian, rhapsody, wiki
magazine | tootsie, prequels, siskel, lou, sponsorships
liberal | cohesive, repay, aurelianus, brinton, marquis
ice | awash, stereogram, programmability, unnamed, druyan
bill | grateful, iai, challenges, borland, klaviertrio
channel | evolutionarily, lakeside, bharatiya, broadcast, determinants
writers | births, deaths, macleod, marsupial, fiction
stage | annette, chest, hopping, condemning, graphemes
...

///// Step  8400
by | lucian, chaim, labour, responsible, schulze
a | for, risch, diez, kaifu, seen
into | centigrade, leveraged, pediment, river, speciesism
where | emmerdale, democr, panel, carpio, brow
seven | two, six, four, three, nine
system | tradeoff, deckers, postgresql, attuned, network
new | edmonton, preventing, hoppers, kline, breast
is | roughly, superior, frac, every, city
except | pedophile, testify, guyana, column, maxims
award | mertens, ursus, idol, haine, grammy
animals | jadavpur, slavoj, crops, mankind, bakker
creation | ellipsoid, rout, sitter, production, platonic
engine | fsn, radiative, underwing, photochemistry, multiprocessor
derived | freed, reservists, associativity, homeomorphic, millennium
applications | tanzania, transmissive, dri, usually, osmotic
know | grabs, torvalds, martyrdom, pronounces, hermetic
...

///// Step  8700
was | he, ardant, reduce, corfu, amoxicillin
are | butyric, same, so, derivation, paddler
one | nine, zero, two, four, three
these | viagra, metals, frud, nursultan, indians
d | b, prize, farmhouse, six, chess
which | many, kept, outcome, goeldi, mussar
on | internetworking, nipple, opposite, first, mornington
history | availible, oran, inks, memel, expenditures
operating | machines, port, gdb, nationalized, rummy
governor | parliamentary, dred, rodgers, hussite, philbin
versions | tp, configurable, dorian, wiki, posix
instance | multipliers, mayne, radiating, jumbo, makes
behind | gnaeus, females, marlow, cpl, pattern
assembly | dissent, libertine, right, vinge, austrians
consists | bundesverfassungsgericht, rotor, crowded, dissatisfaction, trivia
mean | tenochtitl, dulcian, haben, finally, protects
...

///// Step  9000
or | variation, can, policemen, traditions, licensees
be | answered, actual, plenary, defining, thinkpad
s | stowaway, series, finances, summers, fi
may | notated, pohl, arbitrary, irony, harding
used | rom, use, polluted, magnetization, telecoms
there | signature, goode, adiabatic, centerline, theme
while | offices, misunderstood, duty, transpired, cures
however | dcl, zy, prop, seat, atypical
grand | cray, decommissioning, dynamism, pbx, inquisitors
existence | abauzit, eerdmans, arb, prioritize, spatially
cost | peotone, chadic, kyushu, sales, gracen
instance | multipliers, mayne, radiating, denotes, jumbo
ice | awash, stereogram, programmability, unnamed, druyan
police | incensed, brigade, safely, ominous, arafat
except | pedophile, column, tula, guyana, maxims
applied | stereochemistry, vacuum, comit, onslaught, emulators
...

///// Step  9300
had | lucio, habitual, avoid, steinberg, italy
or | variation, policemen, can, guerillas, traditions
their | albums, plenty, waging, stepney, dispersed
d | farmhouse, chess, prize, b, poet
who | mosques, men, jeffreys, he, tiu
american | abbr, hathaway, inventors, hines, laureate
united | refuses, republic, gonzales, trabajo, rastafari
s | stowaway, summers, finances, series, sampler
numerous | infirmity, vernet, priming, isla, disappeared
arts | spiritual, freddie, caucasians, barbarians, kallisti
mainly | lup, unintended, offical, mixture, presley
units | corps, enda, sponsored, kaohsiung, ketos
alternative | dct, joschka, accidently, patenting, manga
consists | bundesverfassungsgericht, crowded, dissatisfaction, trivia, rotor
woman | born, avocats, deathbed, humility, unpaved
heavy | awarding, hsieh, seaports, stamp, hub
...

///// Step  9600
they | that, immunology, ploughing, puritanical, reflector
about | quintuple, ragga, americium, rain, rajonas
who | he, mosques, men, persians, herself
state | bureau, recombination, liberty, vermont, united
use | efficiently, sasl, flexible, rett, misdiagnosed
s | stowaway, summers, finances, two, winsor
new | hoppers, surrounding, arkansas, turboexpress, ferroelectric
with | incompatibilities, tempura, gestae, paraphrasing, punic
bible | corrino, statement, christians, preach, gaulish
pressure | fraser, instinctively, pelvic, sunyaev, crystallizes
http | chinaman, www, merrimac, org, ftp
additional | gopala, flatulence, guebuza, shamanistic, patrons
marriage | daughter, lunatics, caeu, her, radon
existence | abauzit, inquire, religion, eerdmans, arb
test | outgrown, footwork, conomique, apc, diophantus
scale | recognitions, leaderships, graphics, bhishma, paralleled
...

///// Step  9900
three | five, one, four, two, nine
eight | six, five, one, nine, three
and | in, being, wetter, fulfilled, wahhabi
was | he, corfu, basilicas, had, convalescence
at | hike, nchen, the, falsifying, lapland
years | year, three, male, dubs, bachchan
th | teague, nd, dictionary, tte, asturias
which | many, abbasids, in, kept, outcome
except | column, pedophile, maxims, guyana, cantos
stage | chest, hopping, annette, overactive, condemning
instance | multipliers, mayne, radiating, duelling, denotes
behind | females, gnaeus, interlocking, lot, pattern
assembly | dissent, reuptake, needless, vinge, duisburg
san | angeles, aborigines, sledges, auditioned, cessna
professional | soccer, harlequin, farmington, wwe, volleyball
paris | tommaso, hillside, braddock, meza, wesley
...

///// Step  10200
th | teague, nd, tte, asturias, bahri
been | reappear, hilliard, purposes, likelihood, regulate
as | tinkering, aml, metaphor, dione, particulars
people | deaths, benefits, illyrians, disarm, epidemic
other | headings, consist, refer, integrative, imported
zero | two, one, five, nine, six
time | bottle, transmeta, expense, started, sensing
these | metals, nursultan, workflow, viagra, credence
professional | soccer, harlequin, volleyball, farmington, rickover
question | willfully, marketers, formalism, unpaid, subjective
operating | port, machines, gdb, rummy, java
mean | tenochtitl, haben, polyalphabetic, dulcian, conciliation
magazine | prequels, tootsie, lou, fan, nigger
experience | past, diminution, that, immanent, orthonormal
operations | paramilitary, restorative, prophase, idempotent, liguria
accepted | cundinamarca, indications, monastics, spina, organisation
...

///// Step  10500
at | hike, higher, the, glazed, noodle
their | were, flee, dispersed, albums, waging
so | offers, needlessly, notice, necho, glaciers
his | he, son, boomers, him, death
that | they, keel, redacted, happiness, possible
will | methylated, me, spacesuit, tripping, understands
other | headings, imported, consist, turkish, integrative
only | squash, consecutive, tamper, hindemith, mechanically
additional | gopala, flatulence, rollercoaster, rowed, patrons
ocean | ien, archipelago, agnarr, broadcast, atlantic
proposed | syst, acrimonious, antagonistic, collectively, hydrodynamic
notes | sacr, dodging, rotations, automobili, besieges
creation | ellipsoid, remind, sitter, marketplace, collaboratively
joseph | teaching, moritz, shaper, jade, izzard
orthodox | believers, pentecostal, cheek, empower, zy
existence | inquire, eerdmans, abauzit, spatially, prioritize
...

///// Step  10800
to | the, not, on, strike, moved
war | forces, sahrawi, guerrilla, jacqueline, totalitarian
can | uneven, radially, perpetrator, or, number
s | two, vampire, parents, series, leven
would | to, crazed, crawler, possibly, correspondingly
on | first, to, the, nipple, day
an | knowingly, only, respect, facetiously, red
by | glutathione, lucian, implosion, chaim, restart
nobel | prize, physiology, pianist, parkinson, recipient
stage | chest, gene, hopping, cavers, overactive
magazine | fan, prequels, tootsie, lou, nigger
grand | cray, dynamism, arndt, orm, palazzo
question | willfully, how, subjective, marketers, symonds
instance | multipliers, altruistic, mayne, radiating, duelling
consists | bundesverfassungsgericht, crowded, shell, dissatisfaction, compl
defense | freedoms, hainan, teammate, yao, prosecutors
...

///// Step  11100
the | of, in, has, and, to
will | shall, methylated, me, thine, must
there | to, quorum, malvern, afternoons, incipit
called | simple, suitable, imperialistic, prophets, irritability
while | duty, acrobatics, misunderstood, offices, drover
of | the, and, a, since, in
first | on, title, th, gerd, joel
other | consist, headings, imported, race, iditarod
know | grabs, autocode, think, pronounces, cakes
lived | stewie, hayt, sublayer, youngest, stamens
behind | females, gnaeus, interlocking, borg, lot
woman | born, humility, avocats, deathbed, oversimplified
cost | chadic, peotone, sales, medicine, kyushu
award | awards, medal, idol, grammy, mertens
placed | nanette, erect, vertically, binitarians, whichever
instance | multipliers, denotes, radiating, counterexamples, mayne
...

///// Step  11400
such | as, monopoly, supersymmetry, boneyard, everett
american | native, nine, illinois, abbr, dept
can | uneven, or, sure, perpetrator, kinship
other | are, consist, headings, integrative, refer
the | of, in, has, and, its
s | finances, hemphill, pli, parents, stowaway
who | men, he, mosques, persians, behalf
however | dcl, tetrapods, identical, zy, decades
bbc | protocol, mobilize, harar, famicom, merlini
shown | outlines, borzois, dyad, harper, louie
bible | statement, corrino, gaulish, preach, athanasius
writers | births, deaths, fiction, hermeticism, macleod
know | grabs, autocode, think, pronounces, cakes
file | february, blog, regionally, sasl, coq
square | northeast, militia, irreparable, cosines, majuro
egypt | selene, colonised, kayaks, aronson, ordered
...

## Visualizing the word vectors

Below we'll use T-SNE to visualize how our high-dimensional word vectors cluster together. T-SNE is used to project these vectors into two dimensions while preserving local stucture. Check out [this post from Christopher Olah](http://colah.github.io/posts/2014-10-Visualizing-MNIST/) to learn more about T-SNE and other ways to visualize high-dimensional data._____no_output_____

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
from sklearn.manifold import TSNE_____no_output_____# getting embeddings from the embedding layer of our model, by name
embeddings = model.embed.weight.to('cpu').data.numpy()_____no_output_____viz_words = 600
tsne = TSNE()
embed_tsne = tsne.fit_transform(embeddings[:viz_words, :])_____no_output_____fig, ax = plt.subplots(figsize=(16, 16))
for idx in range(viz_words):
    plt.scatter(*embed_tsne[idx, :], color='steelblue')
    plt.annotate(int_to_vocab[idx], (embed_tsne[idx, 0], embed_tsne[idx, 1]), alpha=0.7)_____no_output_____