# Notebook from chriskok/MLAI-CoP
Path: NLP_Workshop_May2020/completed/HandsOnWorkshop_May2020-Complete.ipynb

**Notes to self**: Remind the audience here that jupyter notebook is sequential (which is why there are numbers by it's side), so we can't skip ahead for example (going to the next block printing X before the previous of reading X).

Also, if they ever get in trouble and aren't sure where they are, just refresh the notebook and run all. there's also a completed notebook in the /completed/ folder that you can follow if you're newer to all this (show them where to get it)_____no_output_____# Text Classification Tutorial
Credit to: https://stackabuse.com/text-classification-with-python-and-scikit-learn/_____no_output_____

import numpy as np
import re
import nltk
from sklearn.datasets import load_files
import pickle
from nltk.corpus import stopwords_____no_output_____

Dataset: Cornell Natural Language Processing Group. The dataset consists of a total of 2000 documents

Check out what the data looks like in the review_polarity/txt_sentoken directory. Particularly cv001 in NEG reviews; it contains words like GOOD._____no_output_____

movie_data = load_files(r"../review_polarity/txt_sentoken")
X, y = movie_data.data, movie_data.target_____no_output_____

We print the length of the array "X" below and notice that we start with 2000 documents read. On the other hand, "y" contains all the labels (either negative or positive based on the folders they were in 'neg' or 'pos').

Remember that someone had to go through all these reviews and label them into these two folders manually. They read through all these reviews and placed them nicely in folders for us based on their sentiment accordingly.

Please note that these are case sensitive when programming with them later on.

**Notes to self**: Can talk about reason for uppercase X and lowercase y notation: https://stats.stackexchange.com/questions/389395/why-uppercase-for-x-and-lowercase-for-y. Essentially X is used in math for Matrices and y for Vectors.

Remember to tell people that the documents are reviews from a bunch of different movies (not just one).

Lastly, slow down. You'll be fine :)

**Disclaimer**: Binder is a free tool that's publically available and we highly suggest not using it with any proprietary data - unsure about the security plus you may lose data when logging out. There are plenty of tools and people around the company who can assist you with any of these needs if you're interested in using it for actual projects (just reach out!). _____no_output_____

len(X)_____no_output_____

Printing out the first review in X just to see the data we've pulled:_____no_output_____

print(X[0])b"arnold schwarzenegger has been an icon for action enthusiasts , since the late 80's , but lately his films have been very sloppy and the one-liners are getting worse . \nit's hard seeing arnold as mr . freeze in batman and robin , especially when he says tons of ice jokes , but hey he got 15 million , what's it matter to him ? \nonce again arnold has signed to do another expensive blockbuster , that can't compare with the likes of the terminator series , true lies and even eraser . \nin this so called dark thriller , the devil ( gabriel byrne ) has come upon earth , to impregnate a woman ( robin tunney ) which happens every 1000 years , and basically destroy the world , but apparently god has chosen one man , and that one man is jericho cane ( arnold himself ) . \nwith the help of a trusty sidekick ( kevin pollack ) , they will stop at nothing to let the devil take over the world ! \nparts of this are actually so absurd , that they would fit right in with dogma . \nyes , the film is that weak , but it's better than the other blockbuster right now ( sleepy hollow ) , but it makes the world is not enough look like a 4 star film . \nanyway , this definitely doesn't seem like an arnold movie . \nit just wasn't the type of film you can see him doing . \nsure he gave us a few chuckles with his well known one-liners , but he seemed confused as to where his character and the film was going . \nit's understandable , especially when the ending had to be changed according to some sources . \naside form that , he still walked through it , much like he has in the past few films . \ni'm sorry to say this arnold but maybe these are the end of your action days . \nspeaking of action , where was it in this film ? \nthere was hardly any explosions or fights . \nthe devil made a few places explode , but arnold wasn't kicking some devil butt . \nthe ending was changed to make it more spiritual , which undoubtedly ruined the film . \ni was at least hoping for a cool ending if nothing else occurred , but once again i was let down . \ni also don't know why the film took so long and cost so much . \nthere was really no super affects at all , unless you consider an invisible devil , who was in it for 5 minutes tops , worth the overpriced budget . \nthe budget should have gone into a better script , where at least audiences could be somewhat entertained instead of facing boredom . \nit's pitiful to see how scripts like these get bought and made into a movie . \ndo they even read these things anymore ? \nit sure doesn't seem like it . \nthankfully gabriel's performance gave some light to this poor film . \nwhen he walks down the street searching for robin tunney , you can't help but feel that he looked like a devil . \nthe guy is creepy looking anyway ! \nwhen it's all over , you're just glad it's the end of the movie . \ndon't bother to see this , if you're expecting a solid action flick , because it's neither solid nor does it have action . \nit's just another movie that we are suckered in to seeing , due to a strategic marketing campaign . \nsave your money and see the world is not enough for an entertaining experience . \n"

y labels: 0 is NEGATIVE, 1 is POSITIVE_____no_output_____

y[0]_____no_output_____

## Preprocessing The Data_____no_output_____

documents = []

from nltk.stem import WordNetLemmatizer

stemmer = WordNetLemmatizer()

for sen in range(0, len(X)):
    # Remove all the special characters
    document = re.sub(r'\W', ' ', str(X[sen]))

    # remove all single characters
    document = re.sub(r'\s+[a-zA-Z]\s+', ' ', document)

    # Remove single characters from the start
    document = re.sub(r'\^[a-zA-Z]\s+', ' ', document)

    # Substituting multiple spaces with single space
    document = re.sub(r'\s+', ' ', document, flags=re.I)

    # Removing prefixed 'b'
    document = re.sub(r'^b\s+', '', document)

    # Converting to Lowercase
    document = document.lower()

    # Lemmatization
    document = document.split()

    document = [stemmer.lemmatize(word) for word in document]
    document = ' '.join(document)

    documents.append(document)
_____no_output_____X[0]_____no_output_____documents[0]_____no_output_____

^ cuts down all the noise_____no_output_____Other measures of NLP preprocessing: n-grams, stopwords, lemmatize conversion vs stemming_____no_output_____## Feature Engineering_____no_output_____**Notes to self**: Visually represent each part of feature engineering

- each individual review into bag of words
- from bag of words into TFIDF
- finally, splitting the data_____no_output_____

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(max_features=1500, min_df=5, max_df=0.7, stop_words=stopwords.words('english'))
X1 = vectorizer.fit_transform(documents).toarray()_____no_output_____

CountVectorizer makes very unique word a feature for the ml model. Parameters:

- We take a max of 1500 features (the most common, highest occuring 1500 unique words)
- min_df is number of documents they appear in at minimum
- max_df is the maximum percentage of documents containing this word
- Finally, stop_words removes any very commons words in the english language_____no_output_____

print(vectorizer.get_feature_names())['000', '10', '13', '1997', '1998', '1999', '20', '80', '90', 'ability', 'able', 'absolutely', 'academy', 'accent', 'accident', 'across', 'act', 'acting', 'action', 'actor', 'actress', 'actual', 'actually', 'adam', 'adaptation', 'add', 'addition', 'admit', 'adult', 'adventure', 'affair', 'affleck', 'age', 'agent', 'ago', 'air', 'alan', 'alien', 'alive', 'allen', 'allows', 'almost', 'alone', 'along', 'already', 'also', 'although', 'always', 'amazing', 'america', 'american', 'among', 'amount', 'amusing', 'anderson', 'angel', 'angle', 'angry', 'animal', 'animated', 'animation', 'annoying', 'another', 'answer', 'anti', 'anyone', 'anything', 'anyway', 'apart', 'apartment', 'ape', 'apparent', 'apparently', 'appeal', 'appealing', 'appear', 'appearance', 'appears', 'appreciate', 'approach', 'appropriate', 'arm', 'army', 'around', 'art', 'artist', 'aside', 'ask', 'asked', 'asks', 'aspect', 'atmosphere', 'attack', 'attempt', 'attention', 'attitude', 'audience', 'average', 'award', 'away', 'awful', 'baby', 'back', 'background', 'bad', 'ball', 'band', 'bank', 'bar', 'barely', 'based', 'basic', 'basically', 'batman', 'battle', 'beat', 'beautiful', 'beauty', 'became', 'become', 'becomes', 'becoming', 'begin', 'beginning', 'behind', 'belief', 'believable', 'believe', 'ben', 'best', 'better', 'beyond', 'big', 'biggest', 'bill', 'billy', 'bit', 'bizarre', 'black', 'blade', 'blair', 'blame', 'blood', 'blow', 'blue', 'board', 'boat', 'bob', 'body', 'bond', 'book', 'boring', 'born', 'bos', 'box', 'boy', 'boyfriend', 'brain', 'break', 'brief', 'bright', 'brilliant', 'bring', 'brings', 'british', 'brother', 'brought', 'brown', 'bruce', 'buddy', 'budget', 'bug', 'build', 'building', 'bunch', 'burton', 'business', 'buy', 'cage', 'call', 'called', 'came', 'cameo', 'camera', 'cameron', 'camp', 'campbell', 'cannot', 'captain', 'capture', 'car', 'care', 'career', 'carrey', 'carry', 'carter', 'cartoon', 'case', 'cash', 'cast', 'casting', 'catch', 'caught', 'cause', 'center', 'central', 'century', 'certain', 'certainly', 'chan', 'chance', 'change', 'characterization', 'charm', 'charming', 'chase', 'cheap', 'chemistry', 'child', 'chinese', 'choice', 'chris', 'christmas', 'christopher', 'cinema', 'cinematic', 'cinematography', 'city', 'claim', 'class', 'classic', 'clear', 'clearly', 'clever', 'cliche', 'climax', 'close', 'club', 'co', 'cold', 'college', 'color', 'come', 'comedic', 'comedy', 'comic', 'coming', 'commercial', 'common', 'company', 'comparison', 'complete', 'completely', 'complex', 'computer', 'concept', 'conclusion', 'conflict', 'confused', 'confusing', 'consider', 'considered', 'considering', 'constantly', 'contact', 'contains', 'continues', 'control', 'convincing', 'cool', 'cop', 'costume', 'could', 'count', 'country', 'couple', 'course', 'cover', 'crap', 'crash', 'crazy', 'create', 'created', 'creates', 'creating', 'creature', 'credit', 'creepy', 'crew', 'crime', 'criminal', 'critic', 'cross', 'crowd', 'cry', 'culture', 'cut', 'cute', 'dance', 'danny', 'dark', 'date', 'daughter', 'david', 'day', 'de', 'dead', 'deal', 'death', 'debut', 'decade', 'decent', 'decide', 'decided', 'decides', 'decision', 'deep', 'definitely', 'deliver', 'delivers', 'depth', 'deserves', 'design', 'desire', 'desperate', 'despite', 'detail', 'detective', 'developed', 'development', 'device', 'devil', 'dialogue', 'die', 'difference', 'different', 'difficult', 'dimensional', 'directed', 'directing', 'direction', 'director', 'disappointing', 'disaster', 'discover', 'discovers', 'disney', 'display', 'disturbing', 'doctor', 'documentary', 'doe', 'dog', 'dollar', 'done', 'door', 'double', 'doubt', 'douglas', 'dozen', 'dr', 'drag', 'drama', 'dramatic', 'draw', 'dream', 'drive', 'driver', 'drug', 'dude', 'due', 'dull', 'dumb', 'dy', 'earlier', 'early', 'earth', 'easily', 'easy', 'ed', 'eddie', 'edge', 'editing', 'edward', 'effect', 'effective', 'effort', 'either', 'element', 'elizabeth', 'else', 'emotion', 'emotional', 'encounter', 'end', 'ending', 'energy', 'english', 'enjoy', 'enjoyable', 'enjoyed', 'enough', 'entertaining', 'entertainment', 'entire', 'entirely', 'epic', 'episode', 'equally', 'era', 'escape', 'especially', 'etc', 'even', 'event', 'eventually', 'ever', 'every', 'everyone', 'everything', 'evil', 'ex', 'exactly', 'example', 'excellent', 'except', 'exception', 'exciting', 'expect', 'expectation', 'expected', 'experience', 'explain', 'explosion', 'extra', 'extremely', 'eye', 'face', 'fact', 'failed', 'fails', 'failure', 'fair', 'fairly', 'fake', 'fall', 'familiar', 'family', 'famous', 'fan', 'fantasy', 'far', 'fare', 'fascinating', 'fashion', 'fast', 'fate', 'father', 'fault', 'favorite', 'fbi', 'fear', 'feature', 'feel', 'feeling', 'fellow', 'felt', 'female', 'fi', 'fiction', 'field', 'fight', 'fighting', 'figure', 'fill', 'filled', 'filmed', 'filmmaker', 'filmmaking', 'final', 'finally', 'find', 'fine', 'fire', 'first', 'fit', 'five', 'flashback', 'flat', 'flaw', 'flick', 'floor', 'fly', 'flying', 'focus', 'folk', 'follow', 'following', 'follows', 'food', 'foot', 'footage', 'force', 'forced', 'forever', 'forget', 'form', 'former', 'formula', 'forward', 'found', 'four', 'fox', 'frame', 'frank', 'free', 'french', 'fresh', 'friend', 'front', 'full', 'fully', 'fun', 'funny', 'future', 'gag', 'game', 'gang', 'gangster', 'gave', 'gay', 'general', 'generally', 'generation', 'genius', 'genre', 'genuine', 'george', 'get', 'getting', 'ghost', 'giant', 'girl', 'girlfriend', 'give', 'given', 'giving', 'go', 'goal', 'god', 'godzilla', 'going', 'gone', 'good', 'gore', 'got', 'government', 'grace', 'graphic', 'great', 'greatest', 'green', 'ground', 'group', 'guard', 'guess', 'gun', 'guy', 'hair', 'half', 'hall', 'hand', 'handle', 'hank', 'happen', 'happened', 'happens', 'happy', 'hard', 'hardly', 'harry', 'hate', 'head', 'hear', 'heard', 'heart', 'heavy', 'hell', 'help', 'henry', 'hero', 'high', 'highly', 'hilarious', 'hill', 'history', 'hit', 'hold', 'hole', 'hollywood', 'home', 'hong', 'hope', 'horrible', 'horror', 'horse', 'hot', 'hotel', 'hour', 'house', 'however', 'huge', 'human', 'humanity', 'humor', 'humorous', 'hundred', 'husband', 'ice', 'idea', 'ii', 'image', 'imagination', 'imagine', 'immediately', 'impact', 'important', 'impossible', 'impression', 'impressive', 'include', 'includes', 'including', 'incredible', 'incredibly', 'indeed', 'individual', 'industry', 'information', 'innocent', 'inside', 'inspired', 'instance', 'instead', 'intelligence', 'intelligent', 'intended', 'interest', 'interested', 'interesting', 'intriguing', 'introduced', 'involved', 'involves', 'involving', 'island', 'issue', 'jack', 'jackie', 'jackson', 'james', 'jason', 'jean', 'jeff', 'jennifer', 'jerry', 'jim', 'job', 'joe', 'john', 'join', 'joke', 'jones', 'journey', 'jr', 'julia', 'julie', 'jump', 'keep', 'kelly', 'kept', 'kevin', 'key', 'kid', 'kill', 'killed', 'killer', 'killing', 'kind', 'king', 'kiss', 'knew', 'know', 'known', 'kong', 'la', 'lack', 'lady', 'lame', 'land', 'language', 'large', 'last', 'late', 'later', 'latest', 'latter', 'laugh', 'laughing', 'law', 'lawyer', 'le', 'lead', 'leader', 'leading', 'leaf', 'learn', 'learns', 'least', 'leave', 'leaving', 'led', 'lee', 'left', 'legend', 'length', 'let', 'level', 'lie', 'life', 'light', 'liked', 'likely', 'limited', 'line', 'liner', 'list', 'literally', 'little', 'live', 'living', 'local', 'long', 'longer', 'look', 'looked', 'looking', 'lose', 'lost', 'lot', 'loud', 'love', 'loved', 'lover', 'low', 'lucas', 'machine', 'mad', 'made', 'magic', 'main', 'major', 'make', 'making', 'male', 'man', 'manages', 'manner', 'many', 'mar', 'mark', 'marriage', 'married', 'martial', 'martin', 'mary', 'master', 'masterpiece', 'match', 'material', 'matrix', 'matter', 'matthew', 'max', 'may', 'maybe', 'mean', 'meaning', 'meant', 'medium', 'meet', 'member', 'memorable', 'memory', 'men', 'menace', 'mention', 'mentioned', 'merely', 'mess', 'message', 'michael', 'middle', 'might', 'mike', 'mile', 'military', 'million', 'mind', 'minor', 'minute', 'miss', 'missing', 'mission', 'mistake', 'mix', 'model', 'modern', 'moment', 'money', 'monster', 'month', 'mood', 'moral', 'mostly', 'mother', 'motion', 'mouth', 'move', 'moving', 'mr', 'much', 'murder', 'murphy', 'music', 'musical', 'must', 'mysterious', 'mystery', 'na', 'nafter', 'nall', 'nalso', 'nalthough', 'name', 'named', 'nan', 'nand', 'nanother', 'nat', 'natural', 'nature', 'nbecause', 'nboth', 'nbut', 'nby', 'ndespite', 'ndirector', 'ndon', 'near', 'nearly', 'necessary', 'need', 'needed', 'neither', 'neven', 'never', 'nevery', 'new', 'news', 'next', 'nfirst', 'nfor', 'nfrom', 'nhe', 'nher', 'nhere', 'nhis', 'nhow', 'nhowever', 'ni', 'nice', 'nicely', 'nick', 'nif', 'night', 'nin', 'ninstead', 'nit', 'njust', 'nlet', 'nlike', 'nmaybe', 'nmeanwhile', 'nmost', 'nmy', 'nno', 'nnot', 'nnow', 'nof', 'noh', 'non', 'nonce', 'none', 'normal', 'note', 'nothing', 'notice', 'novel', 'nowhere', 'nperhaps', 'nshe', 'nsince', 'nso', 'nsome', 'nstill', 'nsure', 'nthat', 'ntheir', 'nthen', 'nthere', 'nthese', 'nthey', 'nthis', 'nthough', 'nto', 'number', 'numerous', 'nunfortunately', 'nwe', 'nwell', 'nwhat', 'nwhen', 'nwhile', 'nwhy', 'nwith', 'nyes', 'nyou', 'obvious', 'obviously', 'occasionally', 'odd', 'offer', 'office', 'officer', 'often', 'oh', 'old', 'older', 'onto', 'open', 'opening', 'opinion', 'opportunity', 'opposite', 'order', 'original', 'oscar', 'others', 'otherwise', 'outside', 'overall', 'owner', 'pace', 'pain', 'pair', 'parent', 'park', 'parody', 'part', 'particular', 'particularly', 'partner', 'party', 'pas', 'past', 'pathetic', 'patient', 'patrick', 'paul', 'pay', 'people', 'perfect', 'perfectly', 'performance', 'performer', 'perhaps', 'period', 'person', 'personal', 'personality', 'peter', 'phantom', 'phone', 'physical', 'pick', 'picture', 'piece', 'place', 'plan', 'plane', 'planet', 'play', 'played', 'player', 'playing', 'pleasure', 'plenty', 'plot', 'point', 'pointless', 'police', 'political', 'poor', 'poorly', 'pop', 'popular', 'portrayal', 'possible', 'possibly', 'potential', 'power', 'powerful', 'predictable', 'premise', 'presence', 'present', 'presented', 'president', 'pretty', 'previous', 'price', 'princess', 'prison', 'prisoner', 'private', 'probably', 'problem', 'process', 'produced', 'producer', 'product', 'production', 'prof', 'professional', 'project', 'promise', 'protagonist', 'prove', 'provide', 'provides', 'public', 'pull', 'pulp', 'pure', 'purpose', 'put', 'quality', 'queen', 'question', 'quick', 'quickly', 'quite', 'race', 'rare', 'rarely', 'rate', 'rated', 'rather', 'rating', 'reach', 'read', 'reading', 'ready', 'real', 'realistic', 'reality', 'realize', 'realized', 'really', 'reason', 'recent', 'recently', 'recommend', 'record', 'red', 'reference', 'relationship', 'release', 'released', 'relief', 'remains', 'remake', 'remember', 'rescue', 'respect', 'rest', 'result', 'return', 'revenge', 'review', 'rich', 'richard', 'ride', 'ridiculous', 'right', 'rise', 'rising', 'road', 'robert', 'robin', 'rock', 'roger', 'role', 'romance', 'romantic', 'room', 'rule', 'run', 'running', 'rush', 'russell', 'ryan', 'sad', 'said', 'sam', 'satire', 'save', 'saving', 'saw', 'say', 'saying', 'scare', 'scary', 'scene', 'school', 'sci', 'science', 'scientist', 'score', 'scott', 'scream', 'screen', 'screenplay', 'screenwriter', 'script', 'sean', 'search', 'seat', 'second', 'secret', 'see', 'seeing', 'seem', 'seemed', 'seemingly', 'seems', 'seen', 'self', 'sense', 'sent', 'sequel', 'sequence', 'series', 'serious', 'seriously', 'set', 'setting', 'seven', 'several', 'sex', 'sexual', 'shakespeare', 'shallow', 'shame', 'share', 'ship', 'shock', 'shoot', 'shooting', 'short', 'shot', 'show', 'showing', 'shown', 'side', 'sight', 'sign', 'silent', 'silly', 'similar', 'simon', 'simple', 'simply', 'since', 'singer', 'single', 'sister', 'sit', 'sitting', 'situation', 'six', 'skill', 'slasher', 'slave', 'slightly', 'slow', 'slowly', 'small', 'smart', 'smile', 'smith', 'snake', 'social', 'society', 'soldier', 'solid', 'somehow', 'someone', 'something', 'sometimes', 'somewhat', 'somewhere', 'son', 'song', 'soon', 'sort', 'soul', 'sound', 'soundtrack', 'space', 'speak', 'special', 'specie', 'speech', 'speed', 'spend', 'spends', 'spent', 'spielberg', 'spirit', 'spot', 'stage', 'stand', 'standard', 'star', 'starring', 'start', 'state', 'station', 'stay', 'steal', 'step', 'stephen', 'steve', 'steven', 'stick', 'still', 'stone', 'stop', 'store', 'story', 'storyline', 'straight', 'strange', 'street', 'strike', 'strong', 'struggle', 'stuck', 'student', 'studio', 'stuff', 'stunning', 'stunt', 'stupid', 'style', 'sub', 'subject', 'subplot', 'subtle', 'success', 'successful', 'suddenly', 'suit', 'summer', 'superb', 'superior', 'support', 'supporting', 'suppose', 'supposed', 'sure', 'surprise', 'surprised', 'surprising', 'surprisingly', 'suspect', 'suspense', 'sweet', 'system', 'take', 'taken', 'taking', 'tale', 'talent', 'talented', 'talk', 'talking', 'tarantino', 'target', 'tarzan', 'task', 'teacher', 'team', 'technical', 'teen', 'teenage', 'teenager', 'television', 'tell', 'telling', 'ten', 'tension', 'term', 'terrible', 'terrific', 'test', 'thanks', 'theater', 'theme', 'thin', 'thing', 'think', 'thinking', 'third', 'thomas', 'though', 'thought', 'three', 'thrill', 'thriller', 'throughout', 'throw', 'thrown', 'thus', 'tim', 'titanic', 'title', 'today', 'together', 'told', 'tom', 'tone', 'took', 'top', 'total', 'totally', 'touch', 'touching', 'tough', 'toward', 'towards', 'town', 'toy', 'track', 'trailer', 'train', 'travel', 'travolta', 'treat', 'trek', 'trick', 'tried', 'trip', 'trooper', 'trouble', 'true', 'truly', 'truman', 'truth', 'try', 'trying', 'turn', 'turned', 'tv', 'twenty', 'twice', 'twist', 'two', 'type', 'typical', 'ultimately', 'understand', 'unfortunately', 'unfunny', 'unique', 'unlike', 'upon', 'us', 'use', 'used', 'using', 'usual', 'usually', 'value', 'vampire', 'van', 'various', 'vega', 'vehicle', 'version', 'victim', 'video', 'view', 'viewer', 'viewing', 'villain', 'violence', 'violent', 'virtually', 'vision', 'visit', 'visual', 'voice', 'wait', 'waiting', 'walk', 'wall', 'want', 'wanted', 'war', 'warrior', 'waste', 'wasted', 'watch', 'watching', 'water', 'way', 'weak', 'weapon', 'wear', 'wedding', 'week', 'weird', 'well', 'went', 'west', 'whatever', 'whether', 'white', 'whole', 'whose', 'wide', 'wife', 'wild', 'william', 'williams', 'willis', 'win', 'wind', 'window', 'winner', 'winning', 'wise', 'wish', 'wit', 'witch', 'within', 'without', 'witty', 'woman', 'wonder', 'wonderful', 'woo', 'wood', 'woody', 'word', 'work', 'worked', 'worker', 'working', 'world', 'worse', 'worst', 'worth', 'worthy', 'would', 'write', 'writer', 'writing', 'written', 'wrong', 'wrote', 'year', 'yes', 'yet', 'york', 'young', 'younger']
np.set_printoptions(threshold=np.inf)
print(X1[0])[0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 5 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 1 0 0 0 0 1 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 1 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
 0 1 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 6 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 2 3 0 0 0 0 0 2 1 0 0 0 0
 0 0 0 0 2 0 2 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0
 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 0 1 1 0 0
 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 2 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0
 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 2 0 1 0 1 0 0 0 0 2 0 0 0 0
 0 0 1 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 2 0 0 2 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0
 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 1 0
 6 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 1 0 0 0 2 0 0 0 0
 0 0 0 0 0 0 0 2 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0
 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0
 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0
 0 0 0 0 4 2 2 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0
 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1
 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0
 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0
 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
 0 0 4 1 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0]
from sklearn.feature_extraction.text import TfidfTransformer
tfidfconverter = TfidfTransformer()
X2 = tfidfconverter.fit_transform(X1).toarray()_____no_output_____

TfidfTransformer turns the count array earlier into an array of weights for each unique word based on the document:
- TF-IDF weight is a weight often used in information retrieval and text mining
- This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus
- The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus
- Variations of the tf-idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query

Why we use TFIDF?
- It's so that we have a better metric to know what represents each document, what the machine learning algorithm will do is looks for patterns among these different words, how they're weighted and what makes up a good or bad review. For example, it is not enough to have 'beautiful' as highly representative of this document. Oftentime it's much better to have many different words that work together to make up a positive review that truly represent this document alone. _____no_output_____

print(X2[0])[0.         0.         0.         0.         0.         0.
 0.         0.07599077 0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.20052791 0.         0.         0.         0.04253558 0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.03153765 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.07461927 0.         0.         0.
 0.         0.06965278 0.         0.         0.         0.
 0.06772453 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.03756588 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.06633847 0.07983243
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.07763989 0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.1321944  0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.0523999  0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.03216359 0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.07770022 0.         0.07538821 0.         0.         0.
 0.         0.         0.         0.         0.         0.0660972
 0.         0.         0.03373273 0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.07885446
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.06001851 0.         0.         0.         0.03912609 0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.06733614
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.52958963 0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.03233802 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.06438116 0.
 0.         0.         0.         0.         0.06291102 0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.05200282 0.         0.         0.         0.06767596 0.16056766
 0.         0.         0.         0.         0.         0.07808419
 0.05655962 0.         0.         0.         0.         0.
 0.         0.         0.         0.10067936 0.         0.0564945
 0.         0.         0.         0.04031382 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.05774422
 0.         0.07933701 0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.04353999 0.         0.         0.         0.
 0.         0.         0.         0.05714248 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.0726378
 0.         0.         0.         0.         0.06036738 0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.0599323  0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.1384448  0.         0.         0.         0.
 0.         0.         0.         0.         0.02630558 0.0518352
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.06352689 0.         0.039783
 0.06449076 0.         0.         0.05059806 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.04273898 0.         0.
 0.         0.         0.         0.         0.         0.
 0.05933981 0.         0.04646309 0.07387087 0.         0.
 0.         0.         0.         0.         0.         0.
 0.08988821 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.08113206 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.05346174 0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.05578692
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.06562387 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.03366986 0.05766793 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.05984649 0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.08644485 0.
 0.         0.         0.         0.         0.         0.
 0.10289977 0.         0.0674647  0.         0.05942331 0.
 0.         0.         0.         0.15198154 0.         0.
 0.         0.         0.         0.         0.03992894 0.
 0.03544673 0.07405389 0.04948713 0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.07346671 0.         0.         0.
 0.05272392 0.         0.         0.07043589 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.05194678 0.         0.         0.         0.06384266
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.06241294 0.         0.
 0.04110836 0.         0.         0.         0.         0.
 0.         0.         0.         0.0504426  0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.06183235 0.05904896 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.07442545 0.         0.         0.         0.         0.
 0.06658291 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.09156438 0.
 0.         0.         0.         0.         0.03218289 0.
 0.15151403 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.07213342 0.         0.         0.
 0.08427067 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.07578774 0.
 0.         0.         0.06967072 0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.08679637 0.         0.         0.05257271
 0.07442545 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.05491333 0.         0.
 0.         0.         0.         0.         0.         0.
 0.03563181 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.04318748 0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.06438116
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.06241294 0.         0.
 0.         0.         0.         0.         0.         0.03424347
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.08196777 0.         0.         0.         0.
 0.21690009 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.07492591 0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.08453559
 0.         0.         0.         0.         0.         0.12533443
 0.10982666 0.09284296 0.06352689 0.         0.         0.
 0.         0.         0.         0.         0.         0.05257271
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.04437711 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.13148216 0.
 0.         0.         0.         0.06504868 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.03838346 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.03932357 0.         0.05537862
 0.         0.         0.         0.         0.         0.06438116
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.05065017 0.         0.         0.
 0.         0.         0.         0.         0.         0.03206738
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.03259291 0.         0.         0.         0.         0.
 0.         0.         0.         0.05868504 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.06720846 0.0518909
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.05286435 0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.05984649 0.
 0.         0.         0.         0.         0.         0.
 0.05900954 0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.06670634
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.07578774
 0.         0.         0.         0.         0.         0.03138903
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.04343353 0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.16234008 0.06126937 0.         0.05585585 0.         0.02914348
 0.         0.         0.         0.         0.         0.
 0.03146319 0.         0.         0.         0.         0.        ]

We find that the word that is highly weighted in this first document happens to be at the 18th index from the above print. Looking into it further, the word at the 18th index is 'action' which occurs alot in that review._____no_output_____

print(vectorizer.get_feature_names()[18])action

## Splitting The Data (Train-Test Split)_____no_output_____

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test, indices_train, indices_test = train_test_split(X2, y, [i for i in range(2000)], test_size=0.2, random_state=0)_____no_output_____

Notice that the indices are randomly shuffled, the X_test that we have is filled with randomly selected items from the documents array but we can tell which ones are where since we saved the shuffled indices. For example, the **first item in X_test** is the **405th item in documents**._____no_output_____

print(indices_test)[405, 1190, 1132, 731, 1754, 1178, 1533, 1303, 1857, 18, 1266, 1543, 249, 191, 721, 1896, 452, 1947, 1544, 1205, 1905, 1235, 799, 1594, 1091, 1357, 479, 1148, 511, 1931, 1257, 1105, 1938, 1750, 1058, 427, 1025, 156, 491, 995, 27, 1326, 1239, 17, 1622, 519, 361, 289, 1790, 1135, 1549, 1327, 76, 579, 279, 935, 475, 1786, 1023, 1556, 842, 1522, 1108, 1369, 47, 794, 918, 1220, 506, 353, 276, 881, 1835, 1703, 333, 1616, 135, 182, 1761, 1501, 930, 107, 390, 1165, 1698, 262, 1732, 260, 487, 303, 53, 1015, 148, 1442, 1580, 1999, 1927, 80, 1854, 264, 1979, 1273, 538, 1529, 37, 896, 1129, 1617, 1276, 878, 386, 1820, 977, 1074, 1542, 648, 1922, 77, 689, 9, 861, 6, 1083, 161, 1600, 1110, 962, 1385, 775, 1174, 823, 145, 240, 215, 1889, 465, 987, 1238, 760, 252, 1633, 781, 1451, 572, 85, 634, 317, 118, 1784, 1876, 443, 1900, 516, 1646, 30, 713, 254, 900, 587, 1317, 1887, 1811, 1121, 1334, 1918, 906, 1635, 1858, 796, 1076, 522, 1411, 124, 187, 517, 1945, 686, 342, 1787, 200, 1421, 1511, 1954, 233, 1620, 1344, 1295, 887, 229, 1893, 1426, 682, 440, 557, 899, 1363, 1981, 384, 1158, 1604, 1018, 655, 436, 921, 1340, 1781, 1610, 1621, 453, 1361, 294, 744, 310, 1188, 220, 376, 1406, 259, 1341, 658, 402, 706, 814, 530, 564, 1666, 1955, 1798, 459, 1834, 654, 34, 1808, 1985, 1717, 1147, 1124, 1557, 629, 1356, 512, 597, 999, 955, 610, 1299, 458, 385, 892, 175, 1991, 1715, 1756, 1694, 1744, 61, 1867, 1632, 1271, 227, 1093, 14, 521, 1150, 501, 1203, 1246, 1957, 1618, 1785, 574, 746, 839, 609, 546, 1005, 745, 223, 1677, 1039, 1492, 1196, 1507, 152, 1003, 688, 773, 677, 1367, 55, 141, 1644, 1256, 202, 933, 575, 982, 723, 1487, 801, 1452, 1044, 1279, 674, 1200, 1901, 393, 217, 1704, 1713, 39, 1088, 66, 1034, 1642, 981, 206, 399, 1471, 1695, 651, 1480, 5, 58, 1775, 1528, 898, 1440, 1358, 92, 1217, 1741, 1654, 1050, 831, 278, 793, 133, 1476, 144, 1906, 1386, 758, 1127, 795, 1926, 1724, 19, 1502, 880, 54, 1892, 1437, 789, 943, 1189, 1281, 1929, 1, 762, 1460, 113, 1222, 1051, 438, 1478, 82, 379, 1212, 1446, 326, 811, 322, 716, 1355, 1579, 461, 425, 1101, 122, 883, 446, 48, 1197, 866, 1374, 1263, 503, 1366, 1752, 298, 52, 1859, 1337, 1336, 638, 360, 1810, 1743, 563]

## Training The Model_____no_output_____

from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(n_estimators=1000, random_state=0)
classifier.fit(X_train, y_train) _____no_output_____

**Note to self**: Draw out a single movie review tree here (if the review contains >3 'bad' words, it's negative)_____no_output_____Random Forest example:

_____no_output_____Pros of random forest:
- Great predictive performance for binary classification
- They provide a reliable feature importance estimate
- They offer efficient estimates of the test error without incurring the cost of repeated model training associated with cross-validation
- Handles thousands of input variables without variable deletion

Cons of random forest:
- An ensemble model is inherently less interpretable than an individual decision tree
- Training a large number of deep trees can have high computational costs (but can be parallelized) and use a lot of memory
- Predictions are slower, which may create challenges for applications

More considerations:
- https://github.com/TayariAmine/ML_cheat_sheet/wiki/Random-forest-Pros-and-Cons
- https://www.oreilly.com/library/view/hands-on-machine-learning/9781789346411/e17de38e-421e-4577-afc3-efdd4e02a468.xhtml_____no_output_____

y_pred = classifier.predict(X_test)_____no_output_____len(X_train)_____no_output_____X[405]_____no_output_____y_pred[0]_____no_output_____

## Evaluating The Model_____no_output_____

from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

print(confusion_matrix(y_test,y_pred))
print("\nACCURACY: {}".format(accuracy_score(y_test, y_pred)))[[180  28]
 [ 30 162]]

ACCURACY: 0.855

_____no_output_____

print(classification_report(y_test,y_pred))              precision    recall  f1-score   support

           0       0.86      0.87      0.86       208
           1       0.85      0.84      0.85       192

    accuracy                           0.85       400
   macro avg       0.85      0.85      0.85       400
weighted avg       0.85      0.85      0.85       400

# Finding False Positives (actual = 0, pred = 1)
for i in range(len(y_pred)):
    if (y_pred[i]==1 and y[indices_test[i]]!=y_pred[i]):
        print(y[indices_test[i]], y_pred[i], indices_test[i], i)
        break0 1 1266 10

Did some digging to find the first false positive (index 1266 in 'documents' and 10th predictions in 'pred')_____no_output_____

print(documents[1266])the camera zoom in incredibly close nit focus on the closed eye of person presumed dead nall of sudden with thunderous sound effect and an eerie sounding musical pulse his eyelid open nthe camera zoom in incredibly close nit focus on one of the character desperately running for her life nthen with thunderous sound effect and an eerie sounding musical pulse she bump into the killer nthe camera zoom in incredibly close nit focus on door from where seemingly strange sound emanate nall of sudden with thunderous sound effect and an eerie sounding musical pulse the door swing open to reveal what inside nclose up thrill shot such a these seemed to elicit more laughter from the audience rather than genuine fear which is pretty good sign that you re not watching the thriller that the director had originally envisioned nrather you adopt the opinion that the character are so goofy and so unaware of the trouble that they re in that you stop caring about who life who dy and where the story go ninstead you become more interested in how the character get killed off nthe four young teenager who are the potential sacrificial lamb for this movie are helen and barry and julie and ray two couple and dear friend to one another nduring the summer just after high school graduation they take fateful drive down dark mountain highway nsuddenly without warning their car hit someone whom the teen presume wa killed a result of that collision nfearing jail time and possible manslaughter charge if they confess they instead decide to dump the body into the ocean and make pact never to discus the episode again na year go by and thing among the four friend begin to change nrelationships fizzle future dream crumble and attitude change nbut they can not forget what they did nand apparently neither can someone else nsomeone begin to send letter to all four of the teen with the frightening message know what you did last summer nthe letter writer figure in rubber slicker wielding large metal hook soon make his presence known and begin to hunt down the four teen nit now up to the four of them to try to figure out who the killer is before it too late nwhile this movie ha some nice visually eerie effect too much of it impact had to be generated with extreme close ups while the level of suspense wa choppy at best nthere is so much more that this movie could have been but decided to give u nothing but cheap thrill nit could have been clever whodunit but discovering who ultimately wa the killer made me just shrug my shoulder nit could have explored the changed friendship after year of adulthood but seemed to only include that aspect to add 15 more minute to the film nit could have given u smarter character who knew that anytime you re alone and you see shadowy figure moving about you don walk towards them yelling hello nonce you yell hello you can expect an extreme close up thunderous sound effect and an eerie sounding musical pulse nit difficult to be in suspense when you know what about to happen nthe only redeeming aspect of the movie is that now have neat idea for halloween costume but m not sure if the local costume store sell rubber slicker and hook nthe horror that experienced from this film is realizing that actually went to see it
print(y[1266])
print(y_pred[10])0
1

## Saving The Model_____no_output_____

with open('text_classifier', 'wb') as picklefile:
    pickle.dump(classifier,picklefile)_____no_output_____with open('text_classifier', 'rb') as training_model:
    model = pickle.load(training_model)_____no_output_____y_pred2 = model.predict(X_test)

print(confusion_matrix(y_test, y_pred2))
print(classification_report(y_test, y_pred2))
print(accuracy_score(y_test, y_pred2)) [[180  28]
 [ 30 162]]
              precision    recall  f1-score   support

           0       0.86      0.87      0.86       208
           1       0.85      0.84      0.85       192

    accuracy                           0.85       400
   macro avg       0.85      0.85      0.85       400
weighted avg       0.85      0.85      0.85       400

0.855

## Try It Yourself!_____no_output_____

import numpy as np
import re
import nltk
from sklearn.datasets import load_files
import pickle
from nltk.corpus import stopwords

from sklearn.feature_extraction.text import CountVectorizer
from nltk.stem import WordNetLemmatizer
from sklearn.feature_extraction.text import TfidfTransformer

movie_data = load_files(r"../review_polarity/txt_sentoken")
X, y = movie_data.data, movie_data.target

# Preprocessing functions
def preprocessing(input_X):

    stemmer = WordNetLemmatizer()
    documents = []

    for sen in range(0, len(input_X)):
        # Remove all the special characters
        document = re.sub(r'\W', ' ', str(input_X[sen]))

        # remove all single characters
        document = re.sub(r'\s+[a-zA-Z]\s+', ' ', document)

        # Remove single characters from the start
        document = re.sub(r'\^[a-zA-Z]\s+', ' ', document)

        # Substituting multiple spaces with single space
        document = re.sub(r'\s+', ' ', document, flags=re.I)

        # Removing prefixed 'b'
        document = re.sub(r'^b\s+', '', document)

        # Converting to Lowercase
        document = document.lower()

        # Lemmatization
        document = document.split()

        document = [stemmer.lemmatize(word) for word in document]
        document = ' '.join(document)

        documents.append(document)

    vectorizer = CountVectorizer(max_features=1500, min_df=5, max_df=0.7, stop_words=stopwords.words('english'))
    X1 = vectorizer.fit_transform(documents).toarray()

    tfidfconverter = TfidfTransformer()
    X2 = tfidfconverter.fit_transform(X1).toarray()

    return X2, documents

with open('text_classifier', 'rb') as training_model:
    model = pickle.load(training_model)_____no_output_____new_X_test = ['I loved this movie so much!', 'This movie was bad and had terrible actors']
count = len(new_X_test)
new_X_test.extend(X)

new_X2, new_documents = preprocessing(new_X_test)_____no_output_____for i in range(count):
    review = new_documents[i]
    prediction = model.predict([new_X2[i]])
    print("review: {}, pred: {}".format(review, prediction))
    review: i loved this movie so much, pred: [1]
review: this movie wa bad and had terrible actor, pred: [0]


# Notebook from jpmaldonado/packt-rl
Path: .ipynb_checkpoints/03 blackbox-checkpoint.ipynb

## RL as a blackbox optimization problem

Instead of looking at the value functions $Q$ and $V$ we can look directly at the policy. We assume our policy is parameterized by some vector $\theta$ and try to optimize the reward by changing the values u

Gradients, however, are usually bad news for reinforcement learning for a number of reasons: two issues are that rewards come far in the future, so it is not obvious how to calculate derivatives, and the other issue is the existence of local optima.

Luckily, there is a large class of optimization methods that do not use of the gradients (the expensive part of the computation), and are easy to parallelize. We will discuss two methods which have shown promising results.

_____no_output_____### Cross-Entropy Method

* Blackbox optimization for $f(w)$.
* Start with $\mu, \sigma$, `batch_size`, `elite_frac`.
* Do forever:
  * Sample `batch_size` possible values for $w$.
  * Evaluate $f$ on the sampled values.
  * Keep the top `elite_frac` of those. These are `elite_set`.
  * $\mu \leftarrow$ `np.mean(elite_set)`
  * $\sigma \leftarrow$ `np.std(elite_set)`

_____no_output_____### Natural Evolution Strategies

It might not be good idea to take some elite solutions to resample and throw away the rest, since "weak" parameters are valuable because they tell us what *not* to do. An alternative approach from CEM is to maximise the expected value of the reward, so that when we would sample from that population, we would very likely get lucky and pick a good parameter.

[It has been shown](https://blog.openai.com/evolution-strategies/) that NES is comparable results to fancier algorithms in a number of tasks

### NES Algorithm

* Start with $\sigma$, $\alpha$, `n_estimators`.
* Sample $\epsilon_1, \epsilon_2$ from a normal distribution with mean $0$ and variance $1$. These are **vectors**.
* Calculate return $f(w+\sigma \cdot \epsilon_i)$.
*  $w \leftarrow w + \alpha\cdot \frac{1}{N}\sum_{i=1}^n\left(\frac{f(w+\sigma\epsilon_i)-f(w)}{\sigma}\epsilon_i\right)$
* Stopping criteria: function changes below a threshold, or, in the case of OpenAI Gym, performance benchmark is reached._____no_output_____

## Code sample: NES for FrozenLake
import numpy as np
import gym
np.random.seed(0)

env= gym.make('CartPole-v0')

# the function we want to optimize
def f(w):
    n_iter = 200
    total_reward = 0
    for it in range(n_iter):
        state = env.reset()
        done = False
        while not done:
            action = 1 if np.matmul(state,w) < 0 else 0
            new_state, reward, done, _ = env.step(action)
            total_reward+=reward
            state = new_state
    return total_reward/(1+it)

# hyperparameters
npop = 100 # population size
sigma = 0.1 # noise standard deviation
alpha = 0.001 # learning rate

w = 2*np.random.random(4)-1 # our initial guess is random

i = 0

while f(w) <50:
    i+=1
  # print current fitness of the most likely parameter setting
    if i % 5 == 0:
        print('iter %d. w: %s, reward: %f' %(i, str(w),  f(w)))

  # initialize memory for a population of w's, and their rewards
    N = np.random.randn(npop, 4) # samples from a normal distribution N(0,1)
    R = np.zeros(npop)
    for j in range(npop):
        w_try = w + sigma*N[j] # jitter jw using gaussian of sigma 0.1
        R[j] = f(w_try) # evaluate the jittered version

    # standardize the rewards to have a gaussian distribution
    A = (R - np.mean(R)) / np.std(R)
    # perform the parameter update. The matrix multiply below
    # is just an efficient way to sum up all the rows of the noise matrix N,
    # where each row N[j] is weighted by A[j]
    w = w + alpha/(npop*sigma+1) * np.dot(N.T, A)

env.close()  _____no_output_____## Code sample: Cross entropy method for CartPole

import numpy as np
import gym

# Task settings:
env = gym.make('CartPole-v0') # Change as needed
num_steps = 500 # maximum length of episode
# Alg settings:
n_iter = 100 # number of iterations of CEM
batch_size = 25 # number of samples per batch
elite_frac = 0.2 # fraction of samples used as elite set

# Initialize mean and standard deviation
dim_theta = (env.observation_space.shape[0]+1) * env.action_space.n
theta_mean = np.zeros(dim_theta)
theta_std = np.ones(dim_theta)

def make_policy(theta):
    n_states = env.observation_space.shape[0]
    n_actions = env.action_space.n
    W = theta[0 : n_states * n_actions].reshape(n_states, n_actions)
    b = theta[n_states * n_actions : None].reshape(1, n_actions)

    def policy_fn(state):
        y = state.dot(W) + b
        action = y.argmax()
        return action

    return policy_fn

def run_episode(theta, num_steps, render=False):
    total_reward = 0
    state = env.reset()
    policy = make_policy(theta)
    for t in range(num_steps):
        a = policy(state)
        state, reward, done, _ = env.step(a)
        total_reward += reward
        if render and t%3==0: env.render()
        if done: break
    return total_reward

def noisy_evaluation(theta):
    total_reward = run_episode(theta, num_steps)
    return total_reward

# Now, for the algorithms
for it in range(n_iter):
    # Sample parameter vectors
    thetas = np.random.normal(theta_mean, theta_std, (batch_size,dim_theta))
    rewards = [noisy_evaluation(theta) for theta in thetas]
    # Get elite parameters
    n_elite = int(batch_size * elite_frac)
    elite_inds = np.argsort(rewards)[batch_size - n_elite:batch_size]
    elite_thetas = [thetas[i] for i in elite_inds]
    # Update theta_mean, theta_std
    theta_mean = np.mean(elite_thetas,axis=0)
    theta_std = np.std(elite_thetas,axis=0)
    #print("Mean reward f: {}. \
    # Max reward: {}".format(np.mean(rewards), np.max(rewards)))
    run_episode(theta_mean, num_steps, render=True)_____no_output_____

## CEM in steroids: DeepCEM

Instead of generating the samples from a normal distribution, we can use an arbitrary (and unknown) distribution approximated by neural networks! Each step of the optimization generates a sample produced by a neural network and adjusts its weights based on the outcome. Example from [Yandex Data School](https://github.com/yandexdataschool/Practical_RL)

_____no_output_____

import seaborn as sns
from sklearn.neural_network import MLPClassifier
import matplotlib.pyplot as plt
import gym
from tqdm import tqdm
import numpy as np

class DeepCEM():
    def __init__(self, initial_state, \
                 n_actions, \
                 clf=MLPClassifier(hidden_layer_sizes=(20,20), \
                                   activation='tanh', \
                                  warm_start=True, #keep progress between .fit(...) calls
                                  max_iter=1 #make only 1 iteration on each .fit(...)
                             )):
        self.clf = clf

        #initialize clf
        clf.fit([initial_state]*n_actions,range(n_actions))

    def policy(self, state):
        probs = self.clf.predict_proba([state])[0]
        action = np.random.choice(n_actions, p=probs)
        return action

    def train(self,n_iter):
        n_episodes = 100
        percentile = 75

        for i in tqdm(range(n_iter)):
            #generate new episodes
            episodes = [play_episode(self) \
                        for _ in range(n_episodes)]

            batch_states, batch_actions, batch_rewards = \
            map(np.array,zip(*episodes))

            reward_threshold = np.percentile(batch_rewards,70)
            idxs = [i for \
                    i in range(len(batch_rewards)) \
                    if batch_rewards[i]>reward_threshold]

            elite_states, elite_actions = \
            np.concatenate(batch_states[idxs],axis=0), \
            np.concatenate(batch_actions[idxs],axis=0)
            self.clf.fit(elite_states, elite_actions)

            if np.mean(batch_rewards)> 195:
                print("Solved.")

def play_episode(agent, max_iter=1000,render=False):
    states,actions = [],[]
    total_reward = 0
    state = env.reset()
    for _ in range(max_iter):
        # choose the action according to the policy
        action = agent.policy(state)
        new_state,reward,done,info = env.step(action)
        if render:
            env.render()
        # record sessions
        states.append(state)
        actions.append(action)
        total_reward+=reward
        state = new_state
        if done:
            break
    return states,actions,total_reward

env = gym.make("CartPole-v0").env
env.reset()
n_actions = env.action_space.n

agent = DeepCEM(initial_state=env.reset(), n_actions=2)
agent.train(n_iter=100)
states, actions, rewards= play_episode(agent, render=True)

_____no_output_____


# Notebook from Atashnezhad/Natural_language_processing_Project
Path: Codes/.ipynb_checkpoints/P05_HP_Tunning_DEAgorithm-checkpoint.ipynb

# !pip install regex
# !pip3 install eli5
# !python -m pip install regex
# !python -m pip install eli5
# !pip install scikit-learn==0.18.2_____no_output_____#imports
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns
import eli5

import regex as re
import nltk
#nltk.download('vader_lexicon')
from nltk.sentiment.vader import SentimentIntensityAnalyzer
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.model_selection import cross_val_score, train_test_split, GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix, plot_confusion_matrix
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier

from sklearn.ensemble import GradientBoostingClassifier, AdaBoostClassifier, VotingClassifier

from nltk.corpus import stopwords

import pickle

import warnings
warnings.filterwarnings('ignore')
warnings.simplefilter(action='ignore', category=FutureWarning)_____no_output_____# !pip install eli5
# !python.exe -m pip install --upgrade pip_____no_output_____

## Import data saved from EDA section_____no_output_____

df_nasa_keep_colmn = pickle.load(open('../DataSet/df_nasa_keep_colmn.pkl', 'rb'))
(df_nasa_keep_colmn.head(3))
# print(df_nasa_keep_colmn.shape)_____no_output_____df_space_keep_colmn = pickle.load(open('../DataSet/df_space_keep_colmn.pkl', 'rb'))
(df_space_keep_colmn.head(3))
# print(df_space_keep_colmn.shape)_____no_output_____

### All data_____no_output_____

df_reddit = pickle.load(open('../DataSet/df_reddit_for_model.pkl', 'rb'))
df_reddit.head()_____no_output_____df_reddit.shape_____no_output_____

Note that we have balanced data._____no_output_____

df_reddit['target'].value_counts(normalize=False)_____no_output_____plt.hist(df_reddit['target'])
plt.xlabel('target'); plt.ylabel('counts')
plt.show()_____no_output_____# fig = plt.figure(figsize=(18,5))
a = df_reddit.hist(figsize=(15,15), bins=50)_____no_output_____

No need for balancing classes number._____no_output_____## Defining X and Y variables and use train test split function_____no_output_____

X = df_reddit['text_merged']
y = df_reddit['target']

X_train, X_test, y_train, y_test = train_test_split(X,
                                                    y,
                                                    test_size = 0.25,
                                                    stratify=y,
                                                    random_state = 101)_____no_output_____print(y.value_counts(normalize=False))
print(y_train.value_counts(normalize=False))
print(y_test.value_counts(normalize=False))1    3000
0    3000
Name: target, dtype: int64
1    2250
0    2250
Name: target, dtype: int64
1    750
0    750
Name: target, dtype: int64

Looking for more stop words by vectorizing_____no_output_____

vectorizer = CountVectorizer(analyzer = "word",
                             tokenizer = None,
                             preprocessor = None,
                             stop_words = None,
                             max_features = 1000)

X_train_features = vectorizer.fit_transform(X_train)_____no_output_____X_train_features_df = pd.DataFrame(X_train_features.toarray(),
                          columns=vectorizer.get_feature_names())
X_train_features_df.head(2)_____no_output_____# plt.plot(X_train_features_df.iloc[2])_____no_output_____# import numpy as np

# k = np.array(X_train_features_df.iloc[2])
# k.reshape([54,54],order='F')
# pd.DataFrame(k)_____no_output_____# X_train_features_df.iloc[0][0:3000].plot()_____no_output_____

Looking into which words are used alot:_____no_output_____

# get count of top-occurring words

# empty dictionary
top_words = {}

# loop through columns
for i in X_train_features_df.columns:
    # save sum of each column in dictionary
    top_words[i] =  X_train_features_df[i].sum()

# top_words to dataframe sorted by highest occurance
most_freq = pd.DataFrame(sorted(top_words.items(), key = lambda x: x[1], reverse = True))
# most_freq.head(10)_____no_output_____# print(most_freq.head(40))_____no_output_____plt.figure(figsize = (18, 4))

# visualize top 10 words
plt.bar(most_freq[0][:30], most_freq[1][:30]);
plt.ylabel('counts')_____no_output_____my_stop_words = ["http","would","would","year","like","new", "first","could","one",
                "get","com","know","go","look","see","work","use"]
list_stop_words = nltk.corpus.stopwords.words('english')
list_stop_words.extend(my_stop_words)_____no_output_____

let's start applying logistic regression and use it as a baseline for other ML algorithm's performance._____no_output_____

vectorizer = CountVectorizer(analyzer = "word",
                             ngram_range=(1,10),
                             tokenizer = None,
                             preprocessor = None,
                             stop_words = list_stop_words,
                             max_features = 10000)

X_train_features = vectorizer.fit_transform(X_train)
X_test_features = vectorizer.transform(X_test)_____no_output_____lr = LogisticRegression(solver='newton-cg',
                        max_iter=1000)

lr_M1 = LogisticRegression(solver='newton-cg',
                        max_iter=1000)_____no_output_____lr.fit(X_train_features, y_train);
lr_M1.fit(X_train_features, y_train);_____no_output_____lr.score(X_train_features, y_train)_____no_output_____lr.score(X_test_features, y_test)_____no_output_____

### Lets find the overlap words and see if it will helps the algorithm._____no_output_____

# The cvec parameters obtained from previous gridsearch

cvec = CountVectorizer(stop_words=list_stop_words, min_df=4, max_df=1.0)
term_mat = cvec.fit_transform(df_reddit['text_merged'])
len(cvec.get_feature_names())

term_df = pd.DataFrame(term_mat.toarray(),
                       columns=cvec.get_feature_names())

term_df.insert(0, 'targets', df_reddit["target"])
term_df.head(2)

top_words_NASA = list(term_df.groupby('subreddit').
     mean().T.sort_values(1, ascending=False).head(200).index)

top_words_SPACE = list(term_df.groupby('subreddit').
     mean().T.sort_values(0, ascending=False).head(200).index)

top_words_overlap = [word for word in top_words_NASA if word in top_words_SPACE]_____no_output_____print(top_words_overlap)['ask', 'pleas', 'targets', 'anoth', 'two', 'tri', 'school', 'may', 'away', 'link', 'week', 'good', 'feel', 'shuttl', 'appli', 'sure', 'anyth', 'buy', 'current', 'decemb', 'els', 'give', 'hope', 'job', 'inform', 'seem', 'believ', 'advanc', 'project', 'seen', 'read', 'real', 'world', 'right', 'love', 'lot', 'life', 'last', 'idea', 'histori', 'high', 'site', 'view', 'video', 'pilot', 'pictur', 'png', 'posit', 'start', 'still', 'org', 'sun', 'test', 'took', 'travel', 'true', 'understand', 'place', 'complet', 'flat', 'experi', 'almost', 'train', 'month', 'way', 'appreci', 'never', 'top', 'curiou', 'sky', 'thought', 'sinc', 'gener', 'share', 'chang', 'name', 'differ', 'degre', 'theori', 'oc', 'star', 'point', 'cloud', 'sourc', 'person', 'answer', 'planet', 'someth', 'creat', 'technolog', 'old', 'someon', 'career', 'come', 'origin', 'preview', 'activ', 'intern', 'back', 'jupit', 'keep', 'eye', 'launch', 'let', 'explor', 'long', 'found', 'great']
len(top_words_overlap)_____no_output_____my_stop_words = ['ask', 'pleas', 'targets', 'anoth', 'two', 'tri', 'school',
                 'may', 'away', 'link', 'week', 'good', 'feel', 'shuttl',
                 'appli', 'sure', 'anyth', 'buy', 'current', 'decemb', 'els',
                 'give', 'hope', 'job', 'inform', 'seem', 'believ',
                 'advanc', 'project', 'seen', 'read', 'real', 'world',
                 'right', 'love', 'lot', 'life', 'last', 'idea', 'histori',
                 'high', 'site', 'view', 'video', 'pilot', 'pictur', 'png',
                 'posit', 'start', 'still', 'org', 'sun', 'test', 'took',
                 'travel', 'true', 'understand', 'place', 'complet', 'flat',
                 'experi', 'almost', 'train', 'month', 'way', 'appreci', 'never',
                 'top', 'curiou', 'sky', 'thought', 'sinc', 'gener', 'share',
                 'chang', 'name', 'differ', 'degre', 'theori', 'oc', 'star',
                 'point', 'cloud', 'sourc', 'person', 'answer', 'planet',
                 'someth', 'creat', 'technolog', 'old', 'someon', 'career',
                 'come', 'origin', 'preview', 'activ', 'intern', 'back', 'jupit',
                 'keep', 'eye', 'launch', 'let', 'explor', 'long', 'found', 'great']
list_stop_words = nltk.corpus.stopwords.words('english')
list_stop_words.extend(my_stop_words)_____no_output_____vectorizer = CountVectorizer(analyzer = "word",
                             ngram_range=(1,10),
                             tokenizer = None,
                             preprocessor = None,
                             stop_words = list_stop_words,
                             max_features = 10000)

X_train_features = vectorizer.fit_transform(X_train)
X_test_features = vectorizer.transform(X_test)_____no_output_____lr = LogisticRegression(solver='newton-cg',
                        max_iter=1000)

lr_M1 = LogisticRegression(solver='newton-cg',
                        max_iter=1000)_____no_output_____lr.fit(X_train_features, y_train);
lr_M1.fit(X_train_features, y_train);_____no_output_____lr.score(X_train_features, y_train)_____no_output_____lr.score(X_test_features, y_test)_____no_output_____

Save the model as M1 which later we use for ensemble prediction._____no_output_____

PRED1 = lr_M1.predict(X_test_features)_____no_output_____

check out the cofficients of lr model_____no_output_____

df_coeffs = pd.DataFrame()
df_coef= pd.DataFrame(lr.coef_).T
df_feat = pd.DataFrame(vectorizer.get_feature_names())

df_lr_coefs=pd.concat([df_feat, df_coef], axis=1)
df_lr_coefs.columns = ["feature_names", "LR_coefficients"]
df_lr_coefs_sorted = df_lr_coefs.sort_values(by =["LR_coefficients"], ascending=False, ignore_index=True)
df_lr_coefs_sorted_____no_output_____print(df_lr_coefs_sorted.head(10),
      "\n\n",
     df_lr_coefs_sorted.tail(10))  feature_names  LR_coefficients
0          nasa         2.067562
1  earth apollo         1.426832
2    earth moon         1.392540
3  space center         1.386831
4        apollo         1.360542
5            sl         1.242140
6         patch         1.193547
7        saturn         1.166894
8       transit         1.157967
9     halloween         1.141496

      feature_names  LR_coefficients
9990         potti        -1.186036
9991        review        -1.210565
9992    nasa rover        -1.280748
9993      nasa say        -1.314107
9994        spacex        -1.377168
9995       stellar        -1.398957
9996     betelgeus        -1.450943
9997      backyard        -1.457199
9998        discov        -1.463035
9999        secret        -1.658118

### logistic regression
let's find the best parameterms for count vectorizing and conduct a grid search._____no_output_____

# vectorizer = CountVectorizer(analyzer = "word",
#                              ngram_range=(1,10),
#                              tokenizer = None,
#                              preprocessor = None,
#                              stop_words = list_stop_words,
#                              max_features = 10000)

# cvec = CountVectorizer(stop_words=list_stop_words, min_df=4, max_df=1.0)_____no_output_____pipe = Pipeline([
    ('cvec', CountVectorizer()),
    ('logreg', LogisticRegression())
])_____no_output_____pipe_params = {
    'cvec__max_features': [2_000, 3_000, 4_000],
    'cvec__min_df': [2, 3],
    'cvec__max_df': [.9, .95],
    'cvec__stop_words': [list_stop_words],
    'cvec__ngram_range': [(1,1), (1,2)]
}_____no_output_____gs = GridSearchCV(pipe, # what object are we optimizing?
                  param_grid=pipe_params, # what parameters values are we searching?
                  cv=5) # 5-fold cross-validation.
gs_M2 = GridSearchCV(pipe, # what object are we optimizing?
                  param_grid=pipe_params, # what parameters values are we searching?
                  cv=5) # 5-fold cross-validation._____no_output_____gs.fit(X_train, y_train);
gs_M2.fit(X_train, y_train);_____no_output_____gs.best_params_;_____no_output_____# Score model on training set.
gs.score(X_train, y_train)_____no_output_____# Score model on testing set.
gs.score(X_test, y_test)_____no_output_____preds = gs.predict(X_test)
# Save confusion matrix values
tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()
Accuracy = (tp + tn) / (tp + tn + fp + fn)
Precision = tp / (tp + fp)
Recall = tp / (tp + fn)
Accuracy, Precision, Recall_____no_output_____preds_____no_output_____PRED2 = gs_M2.predict(X_test)_____no_output_____print(PRED1)
print(PRED2)[1 0 0 ... 0 1 1]
[1 0 0 ... 0 1 1]

### Naïve Bayes
#### using MultinomialNB_____no_output_____

# Let's set a pipeline up with two stages:
# 1. CountVectorizer (transformer)
# 2. Multinomial Naive Bayes (estimator)

pipe = Pipeline([
    ('cvec', CountVectorizer()),
    ('nb', MultinomialNB())
])_____no_output_____pipe_params = {
    'cvec__max_features': [2_000, 3_000, 4_000],
    'cvec__min_df': [2, 3],
    'cvec__max_df': [.9, .95],
    'cvec__stop_words': [list_stop_words],
    'cvec__ngram_range': [(1,1), (1,2)]
}_____no_output_____gs = GridSearchCV(pipe, # what object are we optimizing?
                  param_grid=pipe_params, # what parameters values are we searching?
                  cv=5) # 5-fold cross-validation.
gs_M3 = GridSearchCV(pipe, # what object are we optimizing?
                  param_grid=pipe_params, # what parameters values are we searching?
                  cv=5) # 5-fold cross-validation._____no_output_____gs.fit(X_train, y_train);
gs_M3.fit(X_train, y_train);
gs.best_params_;_____no_output_____# Score model on training set.
gs.score(X_train, y_train)_____no_output_____# Score model on testing set.
gs.score(X_test, y_test)_____no_output_____preds = gs.predict(X_test)

# Save confusion matrix values
tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()

# View confusion matrix

#plot_confusion_matrix(gs, X_test_features, y_test, cmap='Blues', values_format='d');

Accuracy = (tp + tn) / (tp + tn + fp + fn)
Precision = tp / (tp + fp)
Recall = tp / (tp + fn)

Accuracy, Precision, Recall_____no_output_____PRED3 = gs_M3.predict(X_test)_____no_output_____print(PRED1)
print(PRED2)
print(PRED3)[1 0 0 ... 0 1 0]
[1 0 1 ... 0 1 1]
[1 0 0 ... 0 1 0]

### Naïve Bayes
#### TFIDF_____no_output_____

# Instantiate the transformer.
tvec = TfidfVectorizer()_____no_output_____df = pd.DataFrame(tvec.fit_transform(X_train).toarray(),
                  columns=tvec.get_feature_names())
df.head()_____no_output_____

grid search/ need a pipeline_____no_output_____

# Set up a pipeline with tf-idf vectorizer and multinomial naive bayes

pipe_tfidf = Pipeline([
    ('tfidf', TfidfVectorizer()),
    ('nb', MultinomialNB())
])_____no_output_____# Search over the following values of hyperparameters:
# Source: http://freediscovery.io/doc/stable/python/examples/optimize_tfidf.html

pipe_tf_idf_params = {
              'tfidf__use_idf': (True, False),}_____no_output_____# Instantiate GridSearchCV.

gs = GridSearchCV(pipe_tfidf, # what object are we optimizing?
                        param_grid = pipe_tf_idf_params, # what parameters values are we searching?
                        cv=5) # 5-fold cross-validation.
# Instantiate GridSearchCV.

gs_M4 = GridSearchCV(pipe_tfidf, # what object are we optimizing?
                        param_grid = pipe_tf_idf_params, # what parameters values are we searching?
                        cv=5) # 5-fold cross-validation._____no_output_____# Fit GridSearch to training data.
gs.fit(X_train, y_train);
gs_M4.fit(X_train, y_train);_____no_output_____# Score model on training set.
gs.score(X_train, y_train)_____no_output_____# Score model on testing set.
gs.score(X_test, y_test)_____no_output_____preds = gs.predict(X_test)

# Save confusion matrix values
tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()

# View confusion matrix

#plot_confusion_matrix(gs, X_test_features, y_test, cmap='Blues', values_format='d');

Accuracy = (tp + tn) / (tp + tn + fp + fn)
Precision = tp / (tp + fp)
Recall = tp / (tp + fn)

Accuracy, Precision, Recall_____no_output_____# visualize this
plt.figure(figsize=(20,16))

plot_confusion_matrix(gs, X_test, y_test, cmap='Blues', values_format='d');_____no_output_____PRED4 = gs_M4.predict(X_test)_____no_output_____print(PRED1)
print(PRED2)
print(PRED3)
print(PRED4)[1 0 0 ... 0 1 0]
[1 0 1 ... 0 1 1]
[1 0 0 ... 0 1 0]
[0 0 0 ... 0 1 0]

### Random forest_____no_output_____

rf = RandomForestClassifier()

rf_params = {
    'n_estimators': [100, 150, 200],
    'max_depth': [None, 1, 2, 3, 4, 5],
}
gs = GridSearchCV(rf, param_grid=rf_params, cv=5)
gs.fit(X_train_features, y_train)

gs_M5 = GridSearchCV(rf, param_grid=rf_params, cv=5)
gs_M5.fit(X_train_features, y_train)

print(gs.best_score_)
gs.best_params_0.7297777777777779
# Score model on training set.
gs.score(X_train_features, y_train)_____no_output_____# Score model on testing set.
gs.score(X_test_features, y_test)_____no_output_____preds = gs.predict(X_test_features)

# Save confusion matrix values
tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()

# View confusion matrix

#plot_confusion_matrix(gs, X_test_features, y_test, cmap='Blues', values_format='d');

Accuracy = (tp + tn) / (tp + tn + fp + fn)
Precision = tp / (tp + fp)
Recall = tp / (tp + fn)

Accuracy, Precision, Recall_____no_output_____# visualize this
plt.figure(figsize=(20,16))

plot_confusion_matrix(gs, X_test_features, y_test, cmap='Blues', values_format='d');_____no_output_____PRED5 = gs_M5.predict(X_test_features)_____no_output_____print(PRED1)
print(PRED2)
print(PRED3)
print(PRED4)
print(PRED5)[1 0 0 ... 0 1 0]
[1 0 1 ... 0 1 1]
[1 0 0 ... 0 1 0]
[0 0 0 ... 0 1 0]
[0 0 0 ... 0 1 1]

### extra trees_____no_output_____

et = ExtraTreesClassifier(n_estimators=100)

rf_params = {
    'n_estimators': [100, 150, 200],
    'max_depth': [None, 1, 2, 3, 4, 5],
}
gs = GridSearchCV(et, param_grid=rf_params, cv=5)
gs.fit(X_train_features, y_train)

gs_M6 = GridSearchCV(et, param_grid=rf_params, cv=5)
gs_M6.fit(X_train_features, y_train)

print(gs.best_score_)
gs.best_params_0.7195555555555556
gs.score(X_train_features, y_train)_____no_output_____gs.score(X_test_features, y_test)_____no_output_____preds = gs.predict(X_test_features)

# Save confusion matrix values
tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()

# View confusion matrix

#plot_confusion_matrix(gs, X_test_features, y_test, cmap='Blues', values_format='d');

Accuracy = (tp + tn) / (tp + tn + fp + fn)
Precision = tp / (tp + fp)
Recall = tp / (tp + fn)

Accuracy, Precision, Recall_____no_output_____PRED6 = gs_M6.predict(X_test_features)_____no_output_____print(PRED1)
print(PRED2)
print(PRED3)
print(PRED4)
print(PRED5)
print(PRED6)[1 0 0 ... 0 1 0]
[1 0 1 ... 0 1 1]
[1 0 0 ... 0 1 0]
[0 0 0 ... 0 1 0]
[0 0 0 ... 0 1 1]
[0 0 0 ... 0 1 0]

### gradient boosting_____no_output_____

gboost = GradientBoostingClassifier()
gboost_params = {
    'max_depth': [2,3,4],
    'n_estimators': [100, 125, 150],
    'learning_rate': [.08, .1, .12]
}
gs = GridSearchCV(gboost, param_grid=gboost_params, cv=3)
gs.fit(X_train_features, y_train)

gs_M7 = GridSearchCV(gboost, param_grid=gboost_params, cv=3)
gs_M7.fit(X_train_features, y_train)

print(gs.best_score_)
gs.best_params_0.7217777777777777
gs.score(X_train_features, y_train)_____no_output_____gs.score(X_test_features, y_test)_____no_output_____preds = gs.predict(X_test_features)

# Save confusion matrix values
tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()

# View confusion matrix

#plot_confusion_matrix(gs, X_test_features, y_test, cmap='Blues', values_format='d');

Accuracy = (tp + tn) / (tp + tn + fp + fn)
Precision = tp / (tp + fp)
Recall = tp / (tp + fn)

Accuracy, Precision, Recall_____no_output_____PRED7 = gs_M7.predict(X_test_features)_____no_output_____print(PRED1)
print(PRED2)
print(PRED3)
print(PRED4)
print(PRED5)
print(PRED6)
print(PRED7)[1 0 0 ... 0 1 0]
[1 0 1 ... 0 1 1]
[1 0 0 ... 0 1 0]
[0 0 0 ... 0 1 0]
[0 0 0 ... 0 1 1]
[0 0 0 ... 0 1 0]
[1 0 0 ... 0 1 1]

### Neural Network_____no_output_____

mlp = MLPClassifier(hidden_layer_sizes=(30,30,30))
mlp_M8 = MLPClassifier(hidden_layer_sizes=(30,30,30))_____no_output_____mlp.fit(X_train_features,y_train)
mlp_M8.fit(X_train_features,y_train)_____no_output_____mlp.score(X_train_features, y_train)_____no_output_____mlp.score(X_test_features, y_test)_____no_output_____preds = mlp.predict(X_test_features)

# Save confusion matrix values
tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()

# View confusion matrix

#plot_confusion_matrix(gs, X_test_features, y_test, cmap='Blues', values_format='d');

Accuracy = (tp + tn) / (tp + tn + fp + fn)
Precision = tp / (tp + fp)
Recall = tp / (tp + fn)

Accuracy, Precision, Recall_____no_output_____PRED8 = mlp_M8.predict(X_test_features)_____no_output_____print(PRED1)
print(PRED2)
print(PRED3)
print(PRED4)
print(PRED5)
print(PRED6)
print(PRED7)
print(PRED8)[1 0 0 ... 0 1 0]
[1 0 1 ... 0 1 1]
[1 0 0 ... 0 1 0]
[0 0 0 ... 0 1 0]
[0 0 0 ... 0 1 1]
[0 0 0 ... 0 1 0]
[1 0 0 ... 0 1 1]
[0 0 0 ... 0 1 0]

### SVM_____no_output_____

svc = SVC(
    C = 100,
    kernel = "rbf",
    gamma = "scale"
)

svc_M9 = SVC(
    C = 100,
    kernel = "rbf",
    gamma = "scale"
)

# Fit on training data.
svc.fit(X_train_features, y_train)
svc_M9.fit(X_train_features, y_train)_____no_output_____svc.score(X_train_features, y_train)_____no_output_____# Evaluate model.
accuracy_score(y_test, svc.predict(X_test_features))_____no_output_____preds = svc.predict(X_test_features)

# Save confusion matrix values
tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()

# View confusion matrix

#plot_confusion_matrix(gs, X_test_features, y_test, cmap='Blues', values_format='d');

Accuracy = (tp + tn) / (tp + tn + fp + fn)
Precision = tp / (tp + fp)
Recall = tp / (tp + fn)

Accuracy, Precision, Recall_____no_output_____print(type(preds))
print(type(y_test))<class 'numpy.ndarray'>
<class 'pandas.core.series.Series'>
preds_____no_output_____PRED9 = svc_M9.predict(X_test_features)_____no_output_____print(PRED1)
print(PRED2)
print(PRED3)
print(PRED4)
print(PRED5)
print(PRED6)
print(PRED7)
print(PRED8)
print(PRED9)[1 0 0 ... 0 1 0]
[1 0 1 ... 0 1 1]
[1 0 0 ... 0 1 0]
[0 0 0 ... 0 1 0]
[0 0 0 ... 0 1 1]
[0 0 0 ... 0 1 0]
[1 0 0 ... 0 1 1]
[0 0 0 ... 0 1 0]
[0 0 1 ... 0 1 0]

It is observed that TFIDF will a little bit help the ML algorithms to find better results.
The accuracy, precision, and recall are over 70%. The trade-off effect between precision and recall is observed as it is expected.
We will include the text and numeric columns and hopefully, we increase the overall metrics.
The ML algorithm used in this case is overfitting the data (high accuracy on train data set and lower accuracy on test data set).

# Ok now lets see how the ensumple models work._____no_output_____

ensumbled_array = (PRED1 + PRED2 + PRED3 + PRED4 + \
PRED5 + PRED6 + PRED7 + PRED8)/8_____no_output_____def ensembel_algorithm(ensumbled_array, threshold_value):
    ensumbled_filtered_array = []
    # go through each element in arr
    for element in ensumbled_array:
        if element > threshold_value:
            element = 1
            ensumbled_filtered_array.append(element)
        else:
            element = 0
            ensumbled_filtered_array.append(element)
    ensumbled_filtered_array = np.array(ensumbled_filtered_array)
    # print(ensumbled_filtered_array)
    return ensumbled_filtered_array_____no_output_____X = df_reddit['text_merged']
y = df_reddit['target']

X_train, X_test, y_train, y_test = train_test_split(X,
                                                    y,
                                                    test_size = 0.25,
                                                    stratify=y,
                                                    random_state = 101)_____no_output_____ensumbled_filtered_array = ensembel_algorithm(ensumbled_array, threshold_value=0.1)
# Save confusion matrix values
tn, fp, fn, tp = confusion_matrix(y_test, ensumbled_filtered_array).ravel()
Accuracy = (tp + tn) / (tp + tn + fp + fn)
Precision = tp / (tp + fp)
Recall = tp / (tp + fn)
Accuracy, Precision, Recall
_____no_output_____

Encapsulate the total ensembel models and run sensitivity threshold value parameters._____no_output_____

def sensitivity_analysis_function(ensumbled_array, threshold_value):
    ensumbled_filtered_array = ensembel_algorithm(ensumbled_array, threshold_value)
    # Save confusion matrix values
    tn, fp, fn, tp = confusion_matrix(y_test, ensumbled_filtered_array).ravel()
    Accuracy = (tp + tn) / (tp + tn + fp + fn)
    Precision = tp / (tp + fp)
    Recall = tp / (tp + fn)
    Accuracy, Precision, Recall
    return (Accuracy, Precision, Recall)_____no_output_____threshold_value_list = [0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1]
Accuracy = [sensitivity_analysis_function(ensumbled_array, threshold_value)[0] for threshold_value in threshold_value_list]
Precision = [sensitivity_analysis_function(ensumbled_array, threshold_value)[1] for threshold_value in threshold_value_list]
Recall = [sensitivity_analysis_function(ensumbled_array, threshold_value)[2] for threshold_value in threshold_value_list]_____no_output_____plt.figure(figsize = (14, 7))
plt.plot(threshold_value_list, Accuracy, label='Accuracy')
plt.plot(threshold_value_list, Precision , label='Precision')
plt.plot(threshold_value_list, Recall , label='Recall')
plt.xlabel("threshold_value")
plt.ylabel("Accuracy - Precision - Recall")
plt.legend()
plt.savefig("../Figures/Accuracy - Precision - Recall.png")_____no_output_____plt.figure(figsize = (14, 7))
plt.plot(threshold_value_list, Accuracy, label='Accuracy')

ensumbled_array = (PRED5 + PRED6 + PRED7 + PRED8)/4

threshold_value_list = [0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1]
Accuracy2 = [sensitivity_analysis_function(ensumbled_array, threshold_value)[0] for threshold_value in threshold_value_list]
Precision2 = [sensitivity_analysis_function(ensumbled_array, threshold_value)[1] for threshold_value in threshold_value_list]
Recall2 = [sensitivity_analysis_function(ensumbled_array, threshold_value)[2] for threshold_value in threshold_value_list]

plt.plot(threshold_value_list, Accuracy2, label='Accuracy2')

plt.xlabel("threshold_value")
plt.ylabel("Accuracy")
plt.ylim(0,1)
plt.legend()
plt.savefig("../Figures/Accuracy.png")_____no_output_____plt.figure(figsize = (14, 7))

plt.plot(threshold_value_list, Precision , label='Precision')

ensumbled_array = (PRED5 + PRED6 + PRED7 + PRED8)/4

threshold_value_list = [0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1]
Precision2 = [sensitivity_analysis_function(ensumbled_array, threshold_value)[1] for threshold_value in threshold_value_list]

plt.plot(threshold_value_list, Precision2 , label='Precision2')
plt.xlabel("threshold_value")
plt.ylabel("Precision")
plt.ylim(0,1)
plt.legend()
plt.savefig("../Figures/Precision.png")_____no_output_____plt.figure(figsize = (14, 7))

plt.plot(threshold_value_list, Recall , label='Recall')

ensumbled_array = (PRED5 + PRED6 + PRED7 + PRED8)/4

threshold_value_list = [0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1]
Recall2 = [sensitivity_analysis_function(ensumbled_array, threshold_value)[2] for threshold_value in threshold_value_list]

plt.plot(threshold_value_list, Recall2 , label='Recall2')
plt.xlabel("threshold_value")
plt.ylabel("Recall")
plt.ylim(0,1)
plt.legend()
plt.savefig("../Figures/Recall.png")_____no_output_____

So depending on what criteria is the goal one can ensemble algorithms and select a threshold value to hit the goal. Accuracy seems constant no matter what threshold value is selected, however for high recall, one should select low threshold values for the ensembled algorithm. similarly one can improve the Precision by selecting high threshold values._____no_output_____