# Notebook from abhisheksharma026/Apriori
Path: apriori-association-analysis.ipynb

- Authors : Abhishek_____no_output_____An organization wanted to mine association rules of frequently bought items from its stores and suggest some recommendations to its customers.
As a data scientist, you are required to recognize patterns from the available data and evaluate efficacy of methods to obtain patterns.

Your activities should include - performing various activities pertaining to the data such as, preparing the dataset for analysis; investigating the relationships in the data set with visualization; identify frequent patterns; formulate association rules and evaluate quality of rules.

Demonstrate KDD process with following activities:
- Problem statement
- Perform exploratory data analysis
- Preprocess the data. Identify relevant & irrelevant attributes for the problem.
- Propose parameters such as support, confidence etc.
- Discover frequent patterns
- Iterate previous steps by varying parameters
- Formulate association rules
- Compare association rules
- Briefly explain importance of discovered rules

Following are some points for you to take note of, while doing the assignment:
- The data in some of the rows in the data set may be noisy, sparse.
- State all your assumptions clearly
- Provide clear explanations to explain your stand
_____no_output_____

!pip install apyori
!pip install researchpy
!pip install openpyxl
!pip install xlrdCollecting apyori
  Downloading apyori-1.1.2.tar.gz (8.6 kB)
Building wheels for collected packages: apyori
  Building wheel for apyori (setup.py) ... - \ done
  Created wheel for apyori: filename=apyori-1.1.2-py3-none-any.whl size=5974 sha256=cf074e10ddf754ee137bd20556994c09d1ec15bcd47d0ae19327c91ce1556591
  Stored in directory: /root/.cache/pip/wheels/cb/f6/e1/57973c631d27efd1a2f375bd6a83b2a616c4021f24aab84080
Successfully built apyori
Installing collected packages: apyori
Successfully installed apyori-1.1.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Collecting researchpy
  Downloading researchpy-0.3.2-py3-none-any.whl (15 kB)
Requirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (from researchpy) (1.3.2)
Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from researchpy) (1.19.5)
Requirement already satisfied: patsy in /opt/conda/lib/python3.7/site-packages (from researchpy) (0.5.1)
Requirement already satisfied: scipy in /opt/conda/lib/python3.7/site-packages (from researchpy) (1.7.1)
Requirement already satisfied: statsmodels in /opt/conda/lib/python3.7/site-packages (from researchpy) (0.12.2)
Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from pandas->researchpy) (2.8.0)
Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.7/site-packages (from pandas->researchpy) (2021.1)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas->researchpy) (1.15.0)
Installing collected packages: researchpy
Successfully installed researchpy-0.3.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Collecting openpyxl
  Downloading openpyxl-3.0.9-py2.py3-none-any.whl (242 kB)
     |████████████████████████████████| 242 kB 278 kB/s
Collecting et-xmlfile
  Downloading et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)
Installing collected packages: et-xmlfile, openpyxl
Successfully installed et-xmlfile-1.1.0 openpyxl-3.0.9
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Collecting xlrd
  Downloading xlrd-2.0.1-py2.py3-none-any.whl (96 kB)
     |████████████████████████████████| 96 kB 376 kB/s
Installing collected packages: xlrd
Successfully installed xlrd-2.0.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
import pandas as pd
pd.options.display.float_format = "{:,.2f}".format

import numpy as np
import math
import scipy.stats as ss
import matplotlib.pyplot as plt
import seaborn as sns
import random

from apyori import apriori
from mlxtend.frequent_patterns import apriori, association_rules
import networkx as nx
import os
import json
from datetime import datetime
from itertools import combinations, groupby
from collections import Counter

import itertools
import researchpy
import networkx as nx

import time
import gc
import collections

begin = time.time() # Mark the start
gc.collect()

pd.pandas.set_option("display.max_columns", None)
pd.set_option("display.max_colwidth", None)
pd.set_option("precision", 3)
pd.options.display.float_format = '{:,.3f}'.format_____no_output_____df = pd.read_csv(r'../input/market-basket-analysis/apriori.csv', header=None)
df.head(3)_____no_output_____df.shape # 7501 Transactions # Max no. of items purchased = 20_____no_output_____df.info() # Original dataset has multiple instances as nan as individual transactions might not hold all possible item list<class 'pandas.core.frame.DataFrame'>
RangeIndex: 7501 entries, 0 to 7500
Data columns (total 20 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   0       7501 non-null   object
 1   1       5747 non-null   object
 2   2       4389 non-null   object
 3   3       3345 non-null   object
 4   4       2529 non-null   object
 5   5       1864 non-null   object
 6   6       1369 non-null   object
 7   7       981 non-null    object
 8   8       654 non-null    object
 9   9       395 non-null    object
 10  10      256 non-null    object
 11  11      154 non-null    object
 12  12      87 non-null     object
 13  13      47 non-null     object
 14  14      25 non-null     object
 15  15      8 non-null      object
 16  16      4 non-null      object
 17  17      4 non-null      object
 18  18      3 non-null      object
 19  19      1 non-null      object
dtypes: object(20)
memory usage: 1.1+ MB
records = []
for i in range(0, df.shape[0]):
    records.append([str(df.values[i,j]) for j in range(0, df.shape[1])])

print([x for x in records[1] if str(x) != 'nan']) # Second Transaction Data['burgers', 'meatballs', 'eggs']
# Unique items # To create the Transactional Encoding of the data in a binary representation

unq = []
for col in df.columns:
    unq.append(list(set(df[col].unique().tolist())))

flattenItemList = [item for sublist in unq for item in sublist]_____no_output_____# Repeated items # one of the items is 'nan' -> needs to be removed too.

# Certain spelling errors are seen : 'whole weat flour'
print([item for item, count in collections.Counter(flattenItemList).items() if count > 1])['whole wheat pasta', 'shallot', 'light cream', 'herb & pepper', 'butter', 'strawberries', 'cauliflower', 'french wine', 'frozen vegetables', 'meatballs', 'sandwich', 'body spray', 'candy bars', 'mushroom cream sauce', 'frozen smoothie', 'champagne', 'cooking oil', 'antioxydant juice', 'energy drink', 'soda', 'yogurt cake', 'eggs', 'energy bar', 'whole wheat rice', 'spinach', 'almonds', 'pepper', 'oatmeal', 'nonfat milk', 'burgers', 'cider', 'flax seed', 'eggplant', 'sparkling water', 'grated cheese', 'mint', 'olive oil', 'salad', 'hot dogs', 'dessert wine', 'pet food', 'soup', 'chicken', 'french fries', 'green tea', 'mint green tea', 'tomato sauce', 'honey', 'vegetables mix', 'salmon', 'whole weat flour', 'melons', 'spaghetti', 'fresh bread', 'blueberries', 'cake', 'babies food', 'parmesan cheese', 'muffins', 'fromage blanc', 'chocolate', 'carrots', 'cream', 'brownies', 'mayonnaise', 'fresh tuna', 'rice', 'burger sauce', 'chocolate bread', 'escalope', 'green grapes', 'bramble', 'black tea', 'tomatoes', 'milk', 'strong cheese', 'yams', 'cookies', 'red wine', 'tomato juice', 'bug spray', 'corn', 'oil', 'ground beef', 'ketchup', 'bacon', 'chili', 'gums', 'chutney', 'turkey', 'gluten free bar', 'magazines', 'low fat yogurt', 'clothes accessories', 'toothpaste', 'mashed potato', 'cottage cheese', 'white wine', 'shampoo', 'pancakes', 'hand protein bar', 'asparagus', 'ham', 'salt', 'extra dark chocolate', 'barbecue sauce', 'pickles', 'pasta', 'green beans', 'cereals', 'shrimp', 'light mayo', 'protein bar', 'avocado', 'mineral water', nan, 'water spray', 'tea', 'zucchini', 'napkins']
uniqueItems = list(set(flattenItemList)) # extracting only the unique items for building the DataFrame of transactions
uniqueItems[10] # example _____no_output_____uniqueItems = [x for x in uniqueItems if str(x) != 'nan'] # datasets contained 'nan' as string
len(uniqueItems)_____no_output_____%%time
def buildDataset(df, uniqueItems):
    """
    Transactional Encoding of the DataSet.
    Building a dataframe of zeros to accomodate the items by replacing 0's with 1's
    Input : List of unique items
    Output : DataFrame of 0's & 1's with columns as unique items in the list
    and no. of observations as rows filled with occurences of items.

    """
    dfCleaned = pd.DataFrame(np.zeros((df.shape[0], len(uniqueItems))), columns=uniqueItems, dtype=np.int32)

    for colIdx, item in enumerate(dfCleaned.columns.tolist()): # Take one column at a time
        for rowIdx, row in df.iterrows(): # Take items from the first row
            rowVals = row.values # Compare with uniqueItems
            if item in rowVals:
                dfCleaned.iloc[rowIdx, colIdx] = 1

    return dfCleaned

df = buildDataset(df, uniqueItems)
df.head(3)CPU times: user 54.9 s, sys: 4.4 ms, total: 54.9 s
Wall time: 54.9 s
df.tail(3)_____no_output_____df.sum()[:5] # Occurences of items (just the top 5 in sequence)_____no_output_____df.describe().loc[['count', 'mean', 'std']].sort_values(by = 'mean', axis=1, ascending=False) # To check the mean of the items across all 7501 transactions ;
# spaghetti ; french fries ; eggs ; ground beef ; chocolate ; milk are the most frequently bought items._____no_output_____def plotOccurrence(df, n=20, ascending=False):
    """
    Plot of top support values : item's frequency of occurrence

    """
    plt.rcParams['figure.figsize'] = (18, 7)
    color = plt.cm.coolwarm(np.linspace(0, 1, 40))
    (df.sum() / df.shape[0]).sort_values(ascending=ascending).head(n).plot.bar(color = color)
    plt.title('frequency of most popular items', fontsize = 20)
    plt.xticks(rotation = 90 )
    plt.grid()
    plt.show()

plotOccurrence(df, n=20) # Mineral Water has the highest percentage of occurrence _____no_output_____plotOccurrence(df, n=20, ascending=True) # Water Spray has the lowest percentage of occurrence _____no_output_____def cramers_v(confusion_matrix):
    """
    Calculate Cramers V statistic for categorial-categorial association.
    Considering items & their occurences here as categories.

    """
    chi2 = ss.chi2_contingency(confusion_matrix)[0]
    n = confusion_matrix.sum()
    phi2 = chi2 / n
    r, k = confusion_matrix.shape
    phi2corr = max(0, phi2 - ((k-1)*(r-1))/(n-1))
    rcorr = r - ((r-1)**2)/(n-1)
    kcorr = k - ((k-1)**2)/(n-1)
    return np.sqrt(phi2corr / min((kcorr-1), (rcorr-1)))_____no_output_____def find_association(method, data, columns, plot = True, returnCorr= False):
    """
    Finding associations b/w different metrics based on Cramer's rule.
    """

    df = data
    cols = columns
    corrM = np.zeros((len(cols),len(cols)))
    np.fill_diagonal(corrM, 1)

    for col1, col2 in itertools.combinations(cols, 2):
        idx1, idx2 = cols.index(col1), cols.index(col2)
        corrM[idx1, idx2] = cramers_v((pd.crosstab(df[col1], df[col2])).values)
        corrM[idx2, idx1] = corrM[idx1, idx2]

    corr = pd.DataFrame(corrM, index=cols, columns=cols)

    corr = round(corr, 2)
    mask = np.triu(np.ones_like(corr, dtype=bool))

    if plot:
        fig, ax = plt.subplots(figsize=(20, 20))
        ax = sns.heatmap(corr, cmap='coolwarm', annot=True, ax=ax, mask=mask,
                         vmax=1.0, vmin=-1.0, linewidths=0.1,
                        annot_kws={"size": 9}, square=True, cbar=True);
        ax.set_title("Correlation between Variables");

        plt.savefig("association.png", dpi = 300);
        plt.show()

    if returnCorr:
        return corr_____no_output_____cols = (df.sum() / df.shape[0]).sort_values(ascending=False).head(30).index.tolist()
find_association("cramers_v", df, cols)_____no_output_____def drawNetworkGraph(df, startNode, nCols = 6):
    """
    Takes input DataFrame & plots network graph based on Association.
    Edges are weighted using correlation.
    df : Transactional DataFrame
    StartNode :

    """
    weightDict = {}
    cols = (df.sum() / df.shape[0]).sort_values(ascending=False).head(6).index.tolist()
    corrDf = find_association("cramers_v", df, cols, plot=False, returnCorr=True)
    weights = corrDf[startNode].sort_values(ascending=False)

    for key, value in zip(weights.index.tolist(), weights.values.tolist()):
        weightDict[key] = value

    combs = [x for x in itertools.combinations(weightDict.keys(), 2)]

    edge_width = [round(45 * corrDf[u][v] , 3) for u, v in combs]

    G = nx.Graph()
    plt.figure(figsize =(10, 7))
    G.add_node(startNode)
    for items in weightDict.keys():
        G.add_node(items)

    for items in combs:
        G.add_edge(items[0], items[1])

    pos = nx.fruchterman_reingold_layout(G)

    nx.draw_networkx(G, pos, with_labels = True, arrowstyle='-|>',
                     alpha = 0.7, width = edge_width,
                     arrows=True, node_size = weights * 500,
                     edge_color ='.7', cmap = plt.cm.Blues, node_color ='green')

    plt.axis('off')
    plt.tight_layout();

drawNetworkGraph(df, startNode = 'mineral water', nCols = 6)_____no_output_____def getSupportData(df, min_support=0.05, length=1):
    """
    Returns DataFrame based on min_support & specified length
    """

    frequent_itemsets = apriori(df, min_support = min_support, use_colnames=True)
    frequent_itemsets['length'] = frequent_itemsets['itemsets'].apply(lambda x: len(x))
    frequent_itemsets.sort_values(by = ['length'], ascending=False).head(3)
    return frequent_itemsets[(frequent_itemsets['length'] == length) & (frequent_itemsets['support'] >= min_support)]

getSupportData(df, min_support=0.05, length=2)_____no_output_____

### Metrics to understand the rules :

- confidence(A→C) = $ \frac{support(A→C)} {support(A)}$ , range: [0,1]

- lift(A→C) = $ \frac{confidence(A→C)}{support(C)}$ , range: [0,∞]

- leverage(A→C) = $ support(A→C) - support(A) × support(C)$ , range: [−1,1]

- conviction(A→C) = $ \frac{1 − support(C)}{1−confidence(A→C)}$ , range: [0,∞]_____no_output_____

def getDataBasisMetric(df, min_support=0.02, length=2, min_threshold=0.5, metric='lift', minAntecedentLength=1, ascending=False):
    """
    Rules do not extract an individual’s preference, rather find relationships between set of elements of every distinct transaction.

    Association Rule: Ex. {X → Y} is a representation of finding Y on the basket which has X in it.
    X-> Antecedent ; Y-> Consequent
    Itemset: Ex. {X,Y} is a representation of the list of all items which form the association rule.

    Support: Fraction of transactions containing the itemset.

    Confidence: Probability of occurrence of {Y} given {X} is present (Non-Directional)

    Lift: Ratio of confidence to baseline probability of occurrence of {Y}.
    Also used to measure how much more often the antecedent and consequent of a rule A->C occur together
    than we would expect if they were statistically independent. For independent, lift = 1

    Leverage : It computes the difference between the observed frequency of A and C appearing together
    and the frequency that would be expected if A and C were independent.
    A leverage value of 0 indicates total independence.

    Conviction : A high conviction value means that the consequent is highly depending on the antecedent.
    If items are independent, the conviction is 1.

    Example : Support says that 67% of customers purchased milk and cheese.
    Confidence is that 100% of the customers that bought milk also bought cheese.
    Lift represents the 28% increase in expectation that someone will buy cheese, when we know that they bought milk.
    This is the conditional probability.

    """
    frequent_itemsets = apriori(df, min_support=min_support, use_colnames=True)
    rules = association_rules(frequent_itemsets, metric=metric, min_threshold=min_threshold)
    rules["antecedentLength"] = rules["antecedents"].apply(lambda x: len(x))
    return rules[rules["antecedentLength"] >= minAntecedentLength].sort_values(by=metric, ascending=ascending)

# Without rules for pruning, we will need to brute force for 119 items, which is (2^119 - 1) unique itemsets.
# Minimum support : probability of T(i) containing X itemset.
# Here, the item needs to be in 7501 * 0.04 transactions.
# min_threshold : Minimal threshold for the evaluation metric; in this case for lift

getDataBasisMetric(df, min_support=0.04, length=1, min_threshold=0.5, metric='lift')_____no_output_____

- For the above case of (mineral water) & (ground beef) : leverage is close to 0 which means independence; conviction being close to 1 states the same. Lift shows positive correlation( >1 ) though. We need to deep dive using other metrics.

- (spaghetti) & (mineral water) has higher confidence stating probability of seeing mineral water in a transaction given that it also contains spaghetti._____no_output_____

data = getDataBasisMetric(df, min_support=0.04, length=1, min_threshold=0.25, metric='lift')
data[data['antecedents'] == {'mineral water'}].sort_values(by=['lift'], ascending=False) # This confirms the correlation aspect._____no_output_____data = getDataBasisMetric(df, min_support=0.04, length=1, min_threshold=0.25, metric='lift')
data[data['antecedents'] == {'mineral water'}].sort_values(by=['support'], ascending=False) # This confirms the correlation aspect._____no_output_____getDataBasisMetric(df, min_support=0.05, length=1, min_threshold=0.3, metric='confidence')_____no_output_____def buildContingency(df, antecedant, precedant):
    data_crosstab = pd.crosstab(df[antecedant], df[precedant], rownames = [antecedant.upper()],
                                colnames=[precedant.upper()], margins = True)

    print("\nPivot :")
    print(data_crosstab)

    crosstab, resChi , expected = researchpy.crosstab(df[antecedant], df[precedant],
                                                      test= "chi-square", expected_freqs= True);

    print('\nExpected Values')
    print(expected)

    print("\nPearson Chi-square :", round(resChi.values[0][1], 3))
    _____no_output_____def validateMetric(df, antecedents, consequents, min_support=0.01, length=1, min_threshold=0.01, metric='lift'):
    """
    If Kulczynski is near 0 or 1, then we have an interesting rule that is negatively or positively associated respectively.
    If Kulczynski is near 0.5, then we may or may not have an interesting rule.
    Imbalance Ratio where 0 is perfectly balanced and 1 is very skewed.
    So completely uninteresting rules would have both Kulczynski=0.5 and IR=0.

    Input :
    df : Original Dataframe of Encoded Transactions
    antecedents : Antecedent(if) item
    consequents : Consequent(then) item
    min_support : Value in order to prune candidate rules by specifying a minimum lower bound for the Support.
    Length : Antecedent Length
    min_threshold : Threshold for Pruning
    metric : Confidence/Lift

    """
    data = getDataBasisMetric(df, min_support=0.01, length=1, min_threshold=0.01, metric='lift')
    PA = data[(data['antecedents'] == {antecedents}) & (data['consequents'] == {consequents})]

    kulczynski = round(0.5 * PA['support'].values[0] * \
                       ((1/PA['antecedent support'].values[0]) + (1/PA['consequent support'].values[0])), 4)

    IR = round((PA['antecedent support'].values[0] - PA['consequent support'].values[0])/ \
    (PA['antecedent support'].values[0] + PA['consequent support'].values[0] - PA['support'].values[0]) , 3)

    print(f"Kulczynski : {kulczynski} ; Imbalance Ratio : {IR}")
_____no_output_____buildContingency(df, antecedant = 'mineral water', precedant = 'ground beef')
Pivot :
GROUND BEEF       0    1   All
MINERAL WATER
0              5283  430  5713
1              1481  307  1788
All            6764  737  7501

Expected Values
              ground beef
ground beef             0       1
mineral water
0               5,151.677 561.323
1               1,612.323 175.677

Pearson Chi-square : 142.933

##### (ground beef)(mineral water) is much rarer than (~ ground beef)(mineral water) & (ground beef)(~ mineral water) but there are many (~ ground beef)(~ mineral water) -> Case of high null-invariant

- But lift = 1.75 > 1 showing +ve correlation.
- χ2 = 142.93 , observed[mineral water & ground beef] (307) > expected (175.68) ; +ve correlation

 It is unlikely that Ground Beef & Mineral Water will happen together, hence, both measure seem to be skewed due to null transactions._____no_output_____**Note** : Both Lift & χ2 are not null-invariant metrics, hence, we should use Kulczynski with Imbalance Ratio to compensate for the presence of null transactions.
- If Kulczynski is near 0 or 1, then we have an interesting rule that is negatively or positively associated respectively. If Kulczynski is near 0.5, then we may or may not have an interesting rule.
- Imbalance Ratio where 0 is perfectly balanced and 1 is very skewed.
- So completely uninteresting rules would have both Kulczynski=0.5 and IR=0.

##### Kulczynski= $ 1/2( \frac{S(A U B)}{S(A)} + \frac{S(A U B)}{S(B)} ) $

##### IR= $ \frac{|support(A) - support(B)|} {|support(A)+support(B)- support(A∪B)|} $_____no_output_____

validateMetric(df, antecedents='mineral water', consequents='ground beef',
               min_support=0.01, length=1, min_threshold=0.5, metric='lift')Kulczynski : 0.2941 ; Imbalance Ratio : 0.474
buildContingency(df, antecedant = 'mineral water', precedant = 'spaghetti')
Pivot :
SPAGHETTI         0     1   All
MINERAL WATER
0              4855   858  5713
1              1340   448  1788
All            6195  1306  7501

Expected Values
              spaghetti
spaghetti             0       1
mineral water
0             4,718.309 994.691
1             1,476.691 311.309

Pearson Chi-square : 95.416

- lift = 1.44 > 1 showing +ve correlation.
- χ2 = 95.416 , observed[mineral water & spaghetti] (448) > expected (311) ; +ve correlation

 It is likely that Ground Beef & Spaghetti will happen together(high confidence/ average support), but, both measure seem to be skewed due to null transactions.

Using Kulczynski & Imbalance Ratio to validate._____no_output_____

validateMetric(df, antecedents='mineral water', consequents='spaghetti',
               min_support=0.01, length=1, min_threshold=0.5, metric='lift')

# Higher Balance as ratio is closer to 0.Kulczynski : 0.2968 ; Imbalance Ratio : 0.182

#### Conclusion

- Lift, χ2 and cosine are good measures if null transactions are not predominant
- Otherwise, Kulczynski + Imbalance Ratio should be used to judge the interestingness of a pattern_____no_output_____

def metricsPlot(rules, alpha=0.95):

    titles = ["Support vs Confidence", "Support vs Lift",
              "Confidence vs Lift", "Antecedent/Consequent Support"]

    plt.style.use('ggplot')
    plt.subplot(221)
    plt.title(titles[0])
    sns.regplot(x="support", y="confidence", data=rules)
    plt.subplot(222)
    plt.title(titles[1])
    sns.regplot(x="support", y="lift", data=rules)
    plt.subplot(223)
    plt.title(titles[2])
    sns.regplot(x="confidence", y="lift", data=rules)
    plt.subplot(224)
    plt.title(titles[3])
    sns.scatterplot(x="antecedent support", y="consequent support", data=rules, alpha=alpha)
    plt.tight_layout()

rules = getDataBasisMetric(df, min_support=0.01, length=1, min_threshold=0.01, metric='lift')
metricsPlot(rules, alpha=0.75)_____no_output_____

#### Support & confidence seem to be positively related for the given set of rules & so is lift. _____no_output_____


# Notebook from CraigStacey1/CV_P2_ImageCaptioning
Path: 2_Training.ipynb

# Computer Vision Nanodegree

## Project: Image Captioning

---

In this notebook, you will train your CNN-RNN model.

You are welcome and encouraged to try out many different architectures and hyperparameters when searching for a good model.

This does have the potential to make the project quite messy!  Before submitting your project, make sure that you clean up:
- the code you write in this notebook.  The notebook should describe how to train a single CNN-RNN architecture, corresponding to your final choice of hyperparameters.  You should structure the notebook so that the reviewer can replicate your results by running the code in this notebook.
- the output of the code cell in **Step 2**.  The output should show the output obtained when training the model from scratch.

This notebook **will be graded**.

Feel free to use the links below to navigate the notebook:
- [Step 1](#step1): Training Setup
- [Step 2](#step2): Train your Model
- [Step 3](#step3): (Optional) Validate your Model_____no_output_____
## Step 1: Training Setup

In this step of the notebook, you will customize the training of your CNN-RNN model by specifying hyperparameters and setting other options that are important to the training procedure.  The values you set now will be used when training your model in **Step 2** below.

You should only amend blocks of code that are preceded by a `TODO` statement.  **Any code blocks that are not preceded by a `TODO` statement should not be modified**.

### Task #1

Begin by setting the following variables:
- `batch_size` - the batch size of each training batch.  It is the number of image-caption pairs used to amend the model weights in each training step.
- `vocab_threshold` - the minimum word count threshold.  Note that a larger threshold will result in a smaller vocabulary, whereas a smaller threshold will include rarer words and result in a larger vocabulary.
- `vocab_from_file` - a Boolean that decides whether to load the vocabulary from file.
- `embed_size` - the dimensionality of the image and word embeddings.
- `hidden_size` - the number of features in the hidden state of the RNN decoder.
- `num_epochs` - the number of epochs to train the model.  We recommend that you set `num_epochs=3`, but feel free to increase or decrease this number as you wish.  [This paper](https://arxiv.org/pdf/1502.03044.pdf) trained a captioning model on a single state-of-the-art GPU for 3 days, but you'll soon see that you can get reasonable results in a matter of a few hours!  (_But of course, if you want your model to compete with current research, you will have to train for much longer._)
- `save_every` - determines how often to save the model weights.  We recommend that you set `save_every=1`, to save the model weights after each epoch.  This way, after the `i`th epoch, the encoder and decoder weights will be saved in the `models/` folder as `encoder-i.pkl` and `decoder-i.pkl`, respectively.
- `print_every` - determines how often to print the batch loss to the Jupyter notebook while training.  Note that you **will not** observe a monotonic decrease in the loss function while training - this is perfectly fine and completely expected!  You are encouraged to keep this at its default value of `100` to avoid clogging the notebook, but feel free to change it.
- `log_file` - the name of the text file containing - for every step - how the loss and perplexity evolved during training.

If you're not sure where to begin to set some of the values above, you can peruse [this paper](https://arxiv.org/pdf/1502.03044.pdf) and [this paper](https://arxiv.org/pdf/1411.4555.pdf) for useful guidance!  **To avoid spending too long on this notebook**, you are encouraged to consult these suggested research papers to obtain a strong initial guess for which hyperparameters are likely to work best.  Then, train a single model, and proceed to the next notebook (**3_Inference.ipynb**).  If you are unhappy with your performance, you can return to this notebook to tweak the hyperparameters (and/or the architecture in **model.py**) and re-train your model.

### Question 1

**Question:** Describe your CNN-RNN architecture in detail.  With this architecture in mind, how did you select the values of the variables in Task 1?  If you consulted a research paper detailing a successful implementation of an image captioning model, please provide the reference.

**Answer:**

### (Optional) Task #2

Note that we have provided a recommended image transform `transform_train` for pre-processing the training images, but you are welcome (and encouraged!) to modify it as you wish.  When modifying this transform, keep in mind that:
- the images in the dataset have varying heights and widths, and
- if using a pre-trained model, you must perform the corresponding appropriate normalization.

### Question 2

**Question:** How did you select the transform in `transform_train`?  If you left the transform at its provided value, why do you think that it is a good choice for your CNN architecture?

**Answer:**

### Task #3

Next, you will specify a Python list containing the learnable parameters of the model.  For instance, if you decide to make all weights in the decoder trainable, but only want to train the weights in the embedding layer of the encoder, then you should set `params` to something like:
```
params = list(decoder.parameters()) + list(encoder.embed.parameters())
```

### Question 3

**Question:** How did you select the trainable parameters of your architecture?  Why do you think this is a good choice?

**Answer:**

### Task #4

Finally, you will select an [optimizer](http://pytorch.org/docs/master/optim.html#torch.optim.Optimizer).

### Question 4

**Question:** How did you select the optimizer used to train your model?

**Answer:** _____no_output_____

import torch
import torch.nn as nn
from torchvision import transforms
import sys
sys.path.append('/opt/cocoapi/PythonAPI')
from pycocotools.coco import COCO
from data_loader import get_loader
from model import EncoderCNN, DecoderRNN
import math
!pip install nltk
import nltk
nltk.download('punkt')

## TODO #1: Select appropriate values for the Python variables below.
batch_size = 128          # batch size
vocab_threshold = 4        # minimum word count threshold
vocab_from_file = True     # if True, load existing vocab file
embed_size = 128           # dimensionality of image and word embeddings
hidden_size = 256          # number of features in hidden state of the RNN decoder
num_epochs = 3             # number of training epochs
save_every = 1             # determines frequency of saving model weights
print_every = 100          # determines window for printing average loss
log_file = 'training_log.txt'       # name of file with saved training loss and perplexity

# (Optional) TODO #2: Amend the image transform below.
transform_train = transforms.Compose([
    transforms.Resize(256),                          # smaller edge of image resized to 256
    transforms.RandomCrop(224),                      # get 224x224 crop from random location
    transforms.RandomHorizontalFlip(),               # horizontally flip image with probability=0.5
    transforms.ToTensor(),                           # convert the PIL Image to a tensor
    transforms.Normalize((0.485, 0.456, 0.406),      # normalize image for pre-trained model
                         (0.229, 0.224, 0.225))])_____no_output_____

## Step 2: Train your Model

Once you have executed the code cell in **Step 1**, the training procedure below should run without issue.

It is completely fine to leave the code cell below as-is without modifications to train your model.  However, if you would like to modify the code used to train the model below, you must ensure that your changes are easily parsed by your reviewer.  In other words, make sure to provide appropriate comments to describe how your code works!

You may find it useful to load saved weights to resume training.  In that case, note the names of the files containing the encoder and decoder weights that you'd like to load (`encoder_file` and `decoder_file`).  Then you can load the weights by using the lines below:

```python
# Load pre-trained weights before resuming training.
encoder.load_state_dict(torch.load(os.path.join('./models', encoder_file)))
decoder.load_state_dict(torch.load(os.path.join('./models', decoder_file)))
```

While trying out parameters, make sure to take extensive notes and record the settings that you used in your various training runs.  In particular, you don't want to encounter a situation where you've trained a model for several hours but can't remember what settings you used :).

### A Note on Tuning Hyperparameters

To figure out how well your model is doing, you can look at how the training loss and perplexity evolve during training - and for the purposes of this project, you are encouraged to amend the hyperparameters based on this information.

However, this will not tell you if your model is overfitting to the training data, and, unfortunately, overfitting is a problem that is commonly encountered when training image captioning models.

For this project, you need not worry about overfitting. **This project does not have strict requirements regarding the performance of your model**, and you just need to demonstrate that your model has learned **_something_** when you generate captions on the test data.  For now, we strongly encourage you to train your model for the suggested 3 epochs without worrying about performance; then, you should immediately transition to the next notebook in the sequence (**3_Inference.ipynb**) to see how your model performs on the test data.  If your model needs to be changed, you can come back to this notebook, amend hyperparameters (if necessary), and re-train the model.

That said, if you would like to go above and beyond in this project, you can read about some approaches to minimizing overfitting in section 4.3.1 of [this paper](http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7505636).  In the next (optional) step of this notebook, we provide some guidance for assessing the performance on the validation dataset._____no_output_____

import torch.utils.data as data
import numpy as np
import os
import requests
import time

# Open the training log file.
f = open(log_file, 'w')

old_time = time.time()
response = requests.request("GET",
                            "http://metadata.google.internal/computeMetadata/v1/instance/attributes/keep_alive_token",
                            headers={"Metadata-Flavor":"Google"})

for epoch in range(1, num_epochs+1):

    for i_step in range(1, total_step+1):

        if time.time() - old_time > 60:
            old_time = time.time()
            requests.request("POST",
                             "https://nebula.udacity.com/api/v1/remote/keep-alive",
                             headers={'Authorization': "STAR " + response.text})

        # Randomly sample a caption length, and sample indices with that length.
        indices = data_loader.dataset.get_train_indices()
        # Create and assign a batch sampler to retrieve a batch with the sampled indices.
        new_sampler = data.sampler.SubsetRandomSampler(indices=indices)
        data_loader.batch_sampler.sampler = new_sampler

        # Obtain the batch.
        images, captions = next(iter(data_loader))

        # Move batch of images and captions to GPU if CUDA is available.
        images = images.to(device)
        captions = captions.to(device)

        # Zero the gradients.
        decoder.zero_grad()
        encoder.zero_grad()

        # Pass the inputs through the CNN-RNN model.
        features = encoder(images)
        outputs = decoder(features, captions)

        # Calculate the batch loss.
        loss = criterion(outputs.view(-1, vocab_size), captions.view(-1))

        # Backward pass.
        loss.backward()

        # Update the parameters in the optimizer.
        optimizer.step()

        # Get training statistics.
        stats = 'Epoch [%d/%d], Step [%d/%d], Loss: %.4f, Perplexity: %5.4f' % (epoch, num_epochs, i_step, total_step, loss.item(), np.exp(loss.item()))

        # Print training statistics (on same line).
        print('\r' + stats, end="")
        sys.stdout.flush()

        # Print training statistics to file.
        f.write(stats + '\n')
        f.flush()

        # Print training statistics (on different line).
        if i_step % print_every == 0:
            print('\r' + stats)

    # Save the weights.
    if epoch % save_every == 0:
        torch.save(decoder.state_dict(), os.path.join('./models', 'decoder-%d.pkl' % epoch))
        torch.save(encoder.state_dict(), os.path.join('./models', 'encoder-%d.pkl' % epoch))

# Close the training log file.
f.close()_____no_output_____

## Step 3: (Optional) Validate your Model

To assess potential overfitting, one approach is to assess performance on a validation set.  If you decide to do this **optional** task, you are required to first complete all of the steps in the next notebook in the sequence (**3_Inference.ipynb**); as part of that notebook, you will write and test code (specifically, the `sample` method in the `DecoderRNN` class) that uses your RNN decoder to generate captions.  That code will prove incredibly useful here.

If you decide to validate your model, please do not edit the data loader in **data_loader.py**.  Instead, create a new file named **data_loader_val.py** containing the code for obtaining the data loader for the validation data.  You can access:
- the validation images at filepath `'/opt/cocoapi/images/train2014/'`, and
- the validation image caption annotation file at filepath `'/opt/cocoapi/annotations/captions_val2014.json'`.

The suggested approach to validating your model involves creating a json file such as [this one](https://github.com/cocodataset/cocoapi/blob/master/results/captions_val2014_fakecap_results.json) containing your model's predicted captions for the validation images.  Then, you can write your own script or use one that you [find online](https://github.com/tylin/coco-caption) to calculate the BLEU score of your model.  You can read more about the BLEU score, along with other evaluation metrics (such as TEOR and Cider) in section 4.1 of [this paper](https://arxiv.org/pdf/1411.4555.pdf).  For more information about how to use the annotation file, check out the [website](http://cocodataset.org/#download) for the COCO dataset._____no_output_____

# (Optional) TODO: Validate your model._____no_output_____