# Notebook from mannickutd/wikipedia_movie_plots
Path: wikipedia_movie_genre_classification.ipynb

_____no_output_____

!git clone https://github.com/facebookresearch/fastText.git
!cd fastText/
!make
!cd ..fatal: destination path 'fastText' already exists and is not an empty directory.
make: Nothing to be done for 'opt'.
!pip install pandas
!pip install numpy
!pip install sklearn
import re
import pandas
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_splitRequirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (0.22.0)
Requirement already satisfied: python-dateutil>=2 in /usr/local/lib/python3.6/dist-packages (from pandas) (2.5.3)
Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from pandas) (1.14.6)
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas) (2018.7)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2->pandas) (1.11.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (1.14.6)
Requirement already satisfied: sklearn in /usr/local/lib/python3.6/dist-packages (0.0)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sklearn) (0.20.1)
Requirement already satisfied: scipy>=0.13.3 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn) (1.1.0)
Requirement already satisfied: numpy>=1.8.2 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn) (1.14.6)
!wget https://s3.eu-west-2.amazonaws.com/wikipedia-movie-plots/wikipedia-movie-plots.zip -O wikipedia-movie-plots.zip--2018-12-25 17:20:13--  https://s3.eu-west-2.amazonaws.com/wikipedia-movie-plots/wikipedia-movie-plots.zip
Resolving s3.eu-west-2.amazonaws.com (s3.eu-west-2.amazonaws.com)... 52.95.150.44
Connecting to s3.eu-west-2.amazonaws.com (s3.eu-west-2.amazonaws.com)|52.95.150.44|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30978474 (30M) [application/zip]
Saving to: ‘wikipedia-movie-plots.zip’

wikipedia-movie-plo 100%[===================>]  29.54M  20.8MB/s    in 1.4s

2018-12-25 17:20:14 (20.8 MB/s) - ‘wikipedia-movie-plots.zip’ saved [30978474/30978474]

!unzip wikipedia-movie-plots.zipArchive:  wikipedia-movie-plots.zip
  inflating: wiki_movie_plots_deduped.csv
df = pandas.read_csv("wiki_movie_plots_deduped.csv")_____no_output_____df.columns_____no_output_____df['Genre']_____no_output_____df['Count'] = 1
df[['Genre','Count']].groupby(['Genre'], as_index=False).count().shape[0]
# The number of genres is way to high to provide reliable classification
# Lots of duplication of genre categories and overlap, we need a way to simplify the genre categorization_____no_output_____# Pulled from kaggle example
df['GenreCorrected'] = df['Genre']
df['GenreCorrected'] = df['GenreCorrected'].str.strip()
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' - ', '|')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' / ', '|')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('/', '|')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' & ', '|')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(', ', '|')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('; ', '|')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('bio-pic', 'biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biopic', 'biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biographical', 'biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biodrama', 'biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('bio-drama', 'biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biographic', 'biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' \(film genre\)', '')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('animated','animation')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('anime','animation')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('children\'s','children')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('comedey','comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\[not in citation given\]','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' set 4,000 years ago in the canadian arctic','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('historical','history')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('romantic','romance')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('3-d','animation')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('3d','animation')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('viacom 18 motion pictures','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('sci-fi','science_fiction')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('ttriller','thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('.','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('based on radio serial','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' on the early years of hitler','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('sci fi','science_fiction')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('science fiction','science_fiction')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' (30min)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('16 mm film','short')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\[140\]','drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\[144\]','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' for ','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('adventures','adventure')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('kung fu','martial_arts')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('kung-fu','martial_arts')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('martial arts','martial_arts')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('world war ii','war')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('world war i','war')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biography about montreal canadiens star|maurice richard','biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('bholenath movies|cinekorn entertainment','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' \(volleyball\)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('spy film','spy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('anthology film','anthology')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biography fim','biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('avant-garde','avant_garde')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biker film','biker')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('buddy cop','buddy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('buddy film','buddy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('comedy 2-reeler','comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('films','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('film','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biography of pioneering american photographer eadweard muybridge','biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('british-german co-production','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('bruceploitation','martial_arts')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('comedy-drama adaptation of the mordecai richler novel','comedy-drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('movies by the mob\|knkspl','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('movies','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('movie','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('coming of age','coming_of_age')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('coming-of-age','coming_of_age')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('drama about child soldiers','drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('(( based).+)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('(( co-produced).+)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('(( adapted).+)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('(( about).+)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('musical b','musical')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('animationchildren','animation|children')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' period','period')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('drama loosely','drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' \(aquatics|swimming\)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' \(aquatics|swimming\)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace("yogesh dattatraya gosavi's directorial debut \[9\]",'')
df['GenreCorrected'] = df['GenreCorrected'].str.replace("war-time","war")
df['GenreCorrected'] = df['GenreCorrected'].str.replace("wartime","war")
df['GenreCorrected'] = df['GenreCorrected'].str.replace("ww1","war")
df['GenreCorrected'] = df['GenreCorrected'].str.replace('unknown','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace("wwii","war")
df['GenreCorrected'] = df['GenreCorrected'].str.replace('psychological','psycho')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('rom-coms','romance')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('true crime','crime')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\|007','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('slice of life','slice_of_life')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('computer animation','animation')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('gun fu','martial_arts')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('j-horror','horror')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' \(shogi|chess\)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('afghan war drama','war drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\|6 separate stories','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' \(30min\)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' (road bicycle racing)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' v-cinema','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('tv miniseries','tv_miniseries')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\|docudrama','\|documentary|drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' in animation','|animation')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('((adaptation).+)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('((adaptated).+)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('((adapted).+)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('(( on ).+)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('american football','sports')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('dev\|nusrat jahan','sports')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('television miniseries','tv_miniseries')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' \(artistic\)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' \|direct-to-dvd','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('history dram','history drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('martial art','martial_arts')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('psycho thriller,','psycho thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\|1 girl\|3 suitors','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' \(road bicycle racing\)','')
filterE = df['GenreCorrected'] == "ero"
df.loc[filterE,'GenreCorrected'] = "adult"
filterE = df['GenreCorrected'] == "music"
df.loc[filterE,'GenreCorrected']="musical"
filterE = df['GenreCorrected'] == "-"
df.loc[filterE,'GenreCorrected'] = ''
filterE = df['GenreCorrected'] == "comedy–drama"
df.loc[filterE,'GenreCorrected'] = "comedy|drama"
filterE = df['GenreCorrected'] == "comedy–horror"
df.loc[filterE,'GenreCorrected'] = "comedy|horror"
df['GenreCorrected'] = df['GenreCorrected'].str.replace(' ','|')
df['GenreCorrected'] = df['GenreCorrected'].str.replace(',','|')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('-','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('actionadventure','action|adventure')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('actioncomedy','action|comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('actiondrama','action|drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('actionlove','action|love')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('actionmasala','action|masala')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('actionchildren','action|children')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('fantasychildren\|','fantasy|children')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('fantasycomedy','fantasy|comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('fantasyperiod','fantasy|period')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('cbctv_miniseries','tv_miniseries')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('dramacomedy','drama|comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('dramacomedysocial','drama|comedy|social')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('dramathriller','drama|thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('comedydrama','comedy|drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('dramathriller','drama|thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('comedyhorror','comedy|horror')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('sciencefiction','science_fiction')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('adventurecomedy','adventure|comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('animationdrama','animation|drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\|\|','|')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('muslim','religious')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('thriler','thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('crimethriller','crime|thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('fantay','fantasy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('actionthriller','action|thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('comedysocial','comedy|social')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('martialarts','martial_arts')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\|\(children\|poker\|karuta\)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('epichistory','epic|history')

df['GenreCorrected'] = df['GenreCorrected'].str.replace('erotica','adult')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('erotic','adult')

df['GenreCorrected'] = df['GenreCorrected'].str.replace('((\|produced\|).+)','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('chanbara','chambara')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('comedythriller','comedy|thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biblical','religious')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biblical','religious')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('colour\|yellow\|productions\|eros\|international','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\|directtodvd','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('liveaction','live|action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('melodrama','drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('superheroes','superheroe')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('gangsterthriller','gangster|thriller')

df['GenreCorrected'] = df['GenreCorrected'].str.replace('heistcomedy','comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('heist','action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('historic','history')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('historydisaster','history|disaster')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('warcomedy','war|comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('westerncomedy','western|comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('ancientcostume','costume')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('computeranimation','animation')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('dramatic','drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('familya','family')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('familya','family')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('dramedy','drama|comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('dramaa','drama')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('famil\|','family')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('superheroe','superhero')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('biogtaphy','biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('devotionalbiography','devotional|biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('docufiction','documentary|fiction')

df['GenreCorrected'] = df['GenreCorrected'].str.replace('familydrama','family|drama')

df['GenreCorrected'] = df['GenreCorrected'].str.replace('espionage','spy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('supeheroes','superhero')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('romancefiction','romance|fiction')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('horrorthriller','horror|thriller')

df['GenreCorrected'] = df['GenreCorrected'].str.replace('suspensethriller','suspense|thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('musicaliography','musical|biography')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('triller','thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('\|\(fiction\)','|fiction')

df['GenreCorrected'] = df['GenreCorrected'].str.replace('romanceaction','romance|action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('romancecomedy','romance|comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('romancehorror','romance|horror')

df['GenreCorrected'] = df['GenreCorrected'].str.replace('romcom','romance|comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('rom\|com','romance|comedy')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('satirical','satire')

df['GenreCorrected'] = df['GenreCorrected'].str.replace('science_fictionchildren','science_fiction|children')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('homosexual','adult')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('sexual','adult')

df['GenreCorrected'] = df['GenreCorrected'].str.replace('mockumentary','documentary')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('periodic','period')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('romanctic','romantic')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('politics','political')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('samurai','martial_arts')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('tv_miniseries','series')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('serial','series')

filterE = df['GenreCorrected'] == "musical–comedy"
df.loc[filterE,'GenreCorrected'] = "musical|comedy"

filterE = df['GenreCorrected'] == "roman|porno"
df.loc[filterE,'GenreCorrected'] = "adult"

filterE = df['GenreCorrected'] == "action—masala"
df.loc[filterE,'GenreCorrected'] = "action|masala"

filterE = df['GenreCorrected'] == "horror–thriller"
df.loc[filterE,'GenreCorrected'] = "horror|thriller"

df['GenreCorrected'] = df['GenreCorrected'].str.replace('family','children')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('martial_arts','action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('horror','thriller')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('war','action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('adventure','action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('science_fiction','action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('western','action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('western','action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('noir','black')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('spy','action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('superhero','action')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('social','')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('suspense','action')

filterE = df['GenreCorrected'] == "drama|romance|adult|children"
df.loc[filterE,'GenreCorrected'] = "drama|romance|adult"

df['GenreCorrected'] = df['GenreCorrected'].str.replace('\|–\|','|')
df['GenreCorrected'] = df['GenreCorrected'].str.strip(to_strip='\|')
df['GenreCorrected'] = df['GenreCorrected'].str.replace('actionner','action')
df['GenreCorrected'] = df['GenreCorrected'].str.strip()_____no_output_____df[['GenreCorrected','Count']].groupby(['GenreCorrected'],as_index=False).count().sort_values(['Count'], ascending=False).head(10)_____no_output_____df['GenreSplit'] = df['GenreCorrected'].str.split('|')
df['GenreSplit'] = df['GenreSplit'].apply(np.sort).apply(np.unique)_____no_output_____genres_array = np.array([])

for i in range(0, df.shape[0]-1):
    genres_array = np.concatenate((genres_array, df['GenreSplit'][i] ))

genres_array_____no_output_____genres = pandas.DataFrame({'Genre': genres_array})_____no_output_____genres.head(10)_____no_output_____genres['Count'] = 1
genres[['Genre','Count']].groupby(['Genre'], as_index=False).sum().sort_values(['Count'], ascending=False).head(10)_____no_output_____genres = genres[['Genre','Count']].groupby(['Genre'], as_index=False).sum().sort_values(['Count'], ascending=False)_____no_output_____genres = genres[genres['Genre'] != '']
genres.head(25)_____no_output_____TotalCountGenres = sum(genres['Count'])_____no_output_____genres['Frequency'] = genres['Count'] / TotalCountGenres_____no_output_____genres['CumulativeFrequency'] = genres['Frequency'].cumsum()_____no_output_____genres[genres['CumulativeFrequency']<=.957][['Genre','Count']].plot(x='Genre', y='Count', kind='bar', legend=False, grid=True, figsize=(8, 5))
plt.title("Number of movies per genre")
plt.ylabel('# of Occurrences', fontsize=12)
plt.xlabel('Movie genres', fontsize=12)
plt.show()_____no_output_____genres.shape_____no_output_____genres.columns_____no_output_____genres.head()_____no_output_____df.shape_____no_output_____df.columns_____no_output_____df.head_____no_output_____df.columns_____no_output_____df['GenreSplit']_____no_output_____mainGenres = np.array(genres[genres['CumulativeFrequency']<=.957]['Genre'])_____no_output_____mainGenres_____no_output_____df['GenreSplitMain'] = df['GenreSplit'].apply(lambda x: x[np.in1d(x, mainGenres)])_____no_output_____df.columns_____no_output_____df = pandas.concat([df, df.GenreSplitMain.apply(lambda x: '-'.join(x)).str.get_dummies(sep='-')], axis=1)_____no_output_____df.columns_____no_output_____# function for cleaning the plots of the movies
def clean_text(text):
    text = text.lower()
    text = re.sub(r"what's", "what is ", text)
    text = re.sub(r"\'s", " ", text)
    text = re.sub(r"\'ve", " have ", text)
    text = re.sub(r"can't", "can not ", text)
    text = re.sub(r"n't", " not ", text)
    text = re.sub(r"i'm", "i am ", text)
    text = re.sub(r"\'re", " are ", text)
    text = re.sub(r"\'d", " would ", text)
    text = re.sub(r"\'ll", " will ", text)
    text = re.sub(r"\'scuse", " excuse ", text)
    text = text.replace('\n', ' ').replace('\r', '')
    #text = re.sub('\W', ' ', text)
    #text = re.sub('\s+', ' ', text)
    text = text.strip(' ')
    return text_____no_output_____df['PlotClean'] = df['Plot'].apply(clean_text)_____no_output_____dfTrain, dfTest = train_test_split(df[df.GenreCorrected!=''], random_state=42, test_size=0.30, shuffle=True)_____no_output_____with open("train.txt", "w") as fle:
  for index, row in dfTrain.iterrows():
    if row['drama']:
      fle.write("__label__drama {0}\n".format(row['PlotClean']))
    else:
      fle.write("__label__not_drama {0}\n".format(row['PlotClean']))

with open("test.txt", "w") as fle:
  for index, row in dfTest.iterrows():
    if row['drama']:
      fle.write("__label__drama {0}\n".format(row['PlotClean']))
    else:
      fle.write("__label__not_drama {0}\n".format(row['PlotClean']))
    _____no_output_____print(dfTrain.shape)
!tail train.txt
! echo ""
print(dfTest.shape)
!tail test.txt(20076, 33)
__label__not_drama in the city of piqua, ohio, two fourth-grade friends and next-door neighbors named george beard (kevin hart) and harold hutchins (thomas middleditch) are the pranksters at their school, jerome horwitz elementary school. they excessively pull numerous pranks on the cruel teachers, many of which are directed at their "evil" principal, mr. benjamin "benny" krupp (ed helms), putting the two at odds with him. the duo also create comic books about a superhero named captain underpants, a character who has superpowers and wears underwear and a cape. they sell these to their schoolmates through a comic company called treehouse comix inc., located in their treehouse. george and harold  pranks come to an apparent end after they are  caught tampering with a toilet invention, the turbo toilet 2000, made by the school  local snitch and intellectual, melvin sneedly (jordan peele), on video. finally having solid proof of the boys' antics, mr. krupp excitedly prepares to annihilate their friendship by putting them in separate classes. to prevent this, george hypnotizes mr. krupp with a 3d hypno ring he received out of a cereal box. the boys see that mr. krupp bears an odd resemblance to captain underpants without his toupee and command him to be captain underpants. the boys soon learn the severity of their actions when "captain underpants" begins causing problems around piqua. to prevent these issues, the boys take him to their treehouse, where they discover that they can turn captain underpants back into mr. krupp by splashing water on him and can turn him back into captain underpants by snapping their fingers. knowing that mr. krupp will continue trying to separate them, they decide to settle with captain underpants but insist that he be dressed up as mr. krupp under the pretense of a "secret identity," to which captain underpants agrees. his sudden personality change even manages to attract the attention (and affection) of the school  shy lunch lady, edith (kristen schaal). just when george and harold believe that their troubles have ended, jerome horwitz elementary school is visited by an odd, german-accented scientist named professor pee-pee diarrheastein poopypants, esq. (nick kroll), or as he calls himself, professor p. captain underpants (disguised as mr. krupp) hires him to be the new science teacher, but george and harold are suspicious of him due to his violent and short-tempered attitude, as well as his resume  content. as it turns out, poopypants is seeking to get rid of laughter altogether due to the fact that people have made fun of his name for years. he recruits melvin to help him, as he discovers (after reading a comic book george and harold made featuring characters resembling them) that melvin  brain makes him incapable of laughing or having fun. after a school carnival accident caused by captain underpants, mr. krupp finally separates the boys. professor poopypants tries to take over the town with a giant version of the turbo toilet 2000, fueled by the school cafeteria  rotten leftovers left out by edith and uses melvin  brain to turn the children into glum, humorless zombies. captain underpants tries to stop the villain, but due to having no actual superpowers, is effortlessly defeated and thrown into the toilet. george and harold are captured and nearly turned into zombies, but are able to escape after professor poopypants mentions the planet uranus, causing the boys to laugh and damage the turbo toilet 2000  computer, restoring the children back to normal. upon consuming the mutated leftovers, captain underpants acquires superpowers and, with george and harold  help, defeats and shrinks poopypants, though he escapes on a bumblebee shortly thereafter. knowing that they can not  control captain underpants, george and harold destroy the hypno ring to permanently change him back into mr. krupp. feeling that mr. krupp would be nicer if he had friends, the boys set him and lovesick edith up on a date, thus making mr. krupp have a change of heart. krupp returns the comics he took away from george and harold, and even admits their comics are funny. however, the toxic waste from the turbo toilet 2000 transforms all the toilets into vicious monsters which attack the restaurant at which mr. krupp and edith are dining. upon snapping his fingers, mr. krupp once again becomes captain underpants, carrying george and harold away to help him fight them, much to edith  surprise and admiration. in a mid-credits scene, george and harold realize that the secretary, who they put on hold before by faking a contest with a cash prize of $1 billion, has been on the phone for the entire film, so they shut off the call, make the secretary angry, and make a new comic based on her reaction.
__label__not_drama in ancient egypt, pharaoh khufu (jack hawkins) is obsessed with preparing his tomb for the "second life". dissatisfied with his own architects' offerings, he enlists vashtar (james robertson justice), an ingenious man whose devices nearly saved his own people from being conquered and enslaved by khufu. khufu offers to free vashtar  people if he will build khufu a robber-proof tomb - although vashtar will have to die when the pyramid-tomb is completed, to guard its secrets. during the years that the pyramid is being built, pharaoh demands tribute and labor from all his territories, amassing a great wealth of gold and treasures that will be interred with him. princess nellifer (joan collins) comes as the ambassador of the tributary province of cyprus. claiming her province is poor and cannot afford to pay the assigned tribute, she offers herself to pharaoh instead. she becomes khufu  second wife. nellifer questions whether khufu really had as much gold as he says, so she is shown an enormous hoard of fine gold artifacts and jewels, and more in an inner vault that khufu is saving for his "second life." she puts on an ornate, jewel encrusted necklace which khufu angrily demands she remove. after he leaves, she puts it on again, and dares treneh (sydney chaplin), the captain of the guard, to take it from her, which he does gently. later, nellifer starts an affair with treneh, who has become besotten with her. one day on the construction site vashtar  son, senta (dewey martin), saves khufu from being crushed to death by a runaway stone block. in order to get the injured khufu out of the pyramid to help, senta reveals that he too knows the tomb  secrets, knowing that he must now share his father  fate. a grateful pharaoh offers him anything else within his power; senta chooses nellifer  slave kyra to save her from being whipped for her fierce independence. using a cobra obtained from a snake charmer with the help of treneh, nellifer plots to assassinate first queen nailla and her son zanin, then khufu, leaving her to rule egypt. first, treneh persuades khufu that there is a rich hoard of treasure in a tomb far to the north, in order to draw him away from the palace. then nellifer gives zanin a flute to practice and teaches him a tune the snake will be attracted to. queen nailla sees the snake approaching zanin and throws herself on it to save her son. hearing of the queen  death, khufu sends out investigators to look for snake charmers. panicking, nellifer dispatches her servant mabuna to kill khufu at the oasis, but mabuna only manages to wound khufu before being killed. suspecting nellifer, khufu rushes back to the city. in her chambers he overhears her and treneh plotting; he kills treneh in a sword fight, but is himself fatally wounded. the pharaoh, before dying, sees nellifer wearing the necklace that he deliberately took from her earlier, and then dies while calling out for his high priest and lifelong friend hamar (alexis minotis, billed as alex minotis). after khufu  death, hamar, suspecting what nellifer did and deeming the tomb truly robber-proof, even for vashtar and senta, releases them from their death sentences, allowing them to leave egypt and return to their homeland with their people as part of a plan to deal with nellifer. during pharaoh  funeral, hamar has nellifer accompany him into the burial chamber because she "must give the order" to seal the sarcophagus. when her order is obeyed, it releases a large stone in a lower chamber, triggering vashtar  mechanism to seal the tomb. nellifer goes into hysterics when told the tomb is being sealed, realizing she is trapped. "there  no way out," hamar tells her, adding: "this is what you lied and schemed and murdered to achieve. this is your kingdom."
__label__drama police inspectors lau ching-hei and yau kin-bong arrest a rapist in 2003. when yau returns home later, he sees that his pregnant girlfriend has committed suicide by slitting her wrists. yau is so depressed with her death that he indulges in alcohol and turns in his police badge to work as a private investigator. he becomes obsessed with finding out the reason for his girlfriend  suicide. three years later, yau discovers that his girlfriend had actually cheated on him and she was waiting at a bar for her secret lover on the night she died. her secret lover did not show up that night because he was involved in a car accident. yau initially thought that he will be angry with his girlfriend  lover, but he ends up taking care of the comatose man in the hospital. yau gets over the unhappy incident and falls in love with hung, a girl who sells beer in the same bar. lau  wife, susan, is the daughter of the billionaire chow yuen-sing. one night, lau and two accomplices break into his father-in-law  residence, where they kill chow and his butler, uncle man. later, lau lures his partners-in-crime to a rundown house and murders them, after which he attempts to make the scene seem as though the two of them killed each other in a dispute over the loot. susan is not convinced that only the two killers were involved, so she hires yau to help her investigate further. she has also become paranoid after her father  murder, so lau has to discreetly give her pills to calm her down and put her to sleep. after a long investigation, which includes a few dangerous confrontations with a "suspect", yau discovers lau  secret and dark past. chow yuen-sing had murdered lau  family members in macau several years ago when lau was still a boy, and he bribed the police to close the case. lau survived, took on a new identity, and grew up in an orphanage before moving to hong kong, where he became a police officer. lau seeks vengeance on chow and he pretended to fall in love with susan and married her. he attempts to kill susan by drugging her and locking her inside the kitchen after turning on the gas. susan survives, but is badly injured because there was an explosion, and ends up in hospital. lau visits susan in hospital and mentally confesses the truth to her. she reveals that she already knows he tried to murder her because she was still partially conscious when he drugged her and locked her in the kitchen. she then asks him if he ever really loved her. lau realises that, in his desire for revenge, he has destroyed a new family he created with susan. susan does not believe him, loses her will to live, and dies. yau meets lau outside the hospital and tells him his conclusion. lau returns to susan  room, where he feels overwhelmed by guilt and eventually commits suicide by shooting himself in the head.
__label__not_drama jim chee is assigned four cases to solve by captain largo, his new boss at the tuba city, arizona office of the navajo tribal police. one is to ascertain who stole jewelry from the burnt water trading post, and to find the paroled man suspected of the thievery, joseph musket. the third is to find who is vandalizing a windmill in the joint-use lands recently allotted to the hopi. fourth is to learn the identity of the navajo man found dead on the path to kisigi spring. while he is on stakeout near the windmill one night, a small plane crash lands in the wepo wash. chee runs to the crash, finding the pilot and his passenger dead, and a man sitting up in a business suit, holding a card, murdered. as he approaches, he hears someone leaving on foot in the early morning darkness. he also hears a gunshot, most likely the one that killed the man in the business suit, and sees headlights of a vehicle leaving the scene. the airplane was carrying illegal cargo, likely drugs, and the dea agents, in particular t. l. johnson, are possessive of their law enforcement turf. as chee collects information on joseph musket and on the unidentified corpse, he gradually learns information related to the crash and the drug deal. johnson finds this as reason to invade chee  home in the morning, including beating chee up, in johnson’s usual style. johnson puts forward that chee was present so soon after the plane crashed because chee is part of the drug smuggling. the deteriorated condition of the corpse, found over a week after death, makes identification impossible. later, albert lomatewa provides the exact date of death and how the corpse looked, hands and feet flayed as if by a navajo "skinwalker", or witch. chee meets jake west, owner of the burnt water trading post, where the jewelry theft was reported, and briefly the employer of musket. west’s son, thomas rodney west, was killed recently in prison; he had been friends with joseph musket since childhood. chee presses cowboy dashee to arrange an interview with the hopi elder responsible for a shrine near the windmill. dashee translates. chee makes a deal with taylor sawkatewa: chee will bring supplies to stop that windmill, which interferes with water flow around the shrine, in return for sawkatewa telling all he saw and heard in the time it took chee to reach the plane crash site. a man forced another to place the landing lights in the wrong place, and then shot him dead. items were removed from the plane but they were not placed in the car that left the scene. from the pilot’s sister, chee learns a meeting is set up for transfer of the drugs during a private hopi kachina ceremony. she wants revenge, or justice, for her brother’s death. revenge is not a navajo concept, so chee struggles to understand it as motivation. chee arrives at sityatki, the old hopi village, having retrieved the two aluminum suitcases filled with cocaine from their hiding place in the sand at the crash site. both jake west and johnson appear, the former expected, the latter a surprise. west has killed the drug kingpin, figuring he is responsible for his son’s murder while he was held in prison. johnson is there not truly as a dea agent, but as the man who wants these drugs, which he claims are worth $15 million. after chee arrests and handcuffs west, johnson does the same to chee. as this encounter unfolds, a rare intense rainstorm breaks the long drought, rapidly making dangerous rivers in the usually dry arroyos. at chee’s vehicle, johnson gets the suitcases of cocaine. johnson’s plan is to kill the other two. chee explains how johnson visited thomas west in prison, such that the other prisoners assumed he was a snitch and killed him in regular confinement. johnson could have handled that so thomas lived. jake west understands this means he killed the wrong man in revenge for his son’s death. chee grabs one suitcase and throws it at johnson. it then slides to the river which horrifies johnson, who runs to save the cocaine. west follows and pushes johnson into the river. chee tells west that the unidentified corpse will be checked against joseph musket’s prison dental records, and it is clear that west killed musket. west says that the jewelry reported stolen is hidden in his kitchen, just before he dies from the gunshot wound received from johnson. west killed six men before reaching johnson to avenge his son  murder. chee reflects that all this time, capt. largo insisted that chee stay out of the drug smuggling case. not wanting to be criticized, chee hurls the other suitcase full of cocaine into the raging river, all other active cases having been solved.
__label__drama jack grimaldi, a corrupt cop who does favors for the mafia in exchange for large fees, has a loving wife, natalie, and an adoring mistress, sheri. he thinks he has it all, until both the cops and mob are outwitted by a psychopathic russian female mob assassin, mona demarkov. italian crime boss don falcone orders jack to deal with demarkov. jack is unable to kill her; she seduces and makes a fool of him. falcone, disappointed in jack  ineptitude, orders one of his toes amputated. realizing he has endangered both his wife and mistress, jack instructs natalie to leave the city immediately, giving her all the payoff money he  saved as well as instructions where to meet him out west when the time is right. jack ends his affair with sheri and puts her on a train out of the city. jack tries to hunt demarkov but soon realizes that he is putty in her hands. he is attracted to her sexually and no match for her professionally. mona forces him to help her bury falcone alive, then offers to pay jack to help her fake her own death. although he obtains phony papers for her, she refuses to pay and attempts to strangle him. he shoots and wounds her in the arm, then tries to drive away with her handcuffed in the back seat. mona escapes by hooking her legs around his neck, causing him to crash the car. she slithers out through the shattered windshield without freeing her hands. mona lures jack to an abandoned warehouse. he again attempts to kill her but is tricked into shooting sheri instead. mona fixes the corpse so as to suggest that it was she, and not sheri, who died. mona handcuffs jack to the bed and they have sex. she turns jack in, copping a plea deal that will indict jack for the multiple murders that she tricked him into committing. the police arrange a confrontation between jack and demarkov at the courthouse, as he is heading in and she is heading out. she threatens to kill his wife. jack grabs a gun from the ankle holster of a fellow officer and shoots her dead. he turns the gun on himself, only to discover that the revolver is empty. instead of being sent to prison for the murder, he is given a commendation. this frees him to begin a new life out west, with the new identity of "jim daugherty". he imagines natalie  return, but, as mona told him, natalie is long gone, never to return. a despondent jim is resigned to living life alone in a remote desert town.
__label__not_drama in 2008, a combined task group of people  liberation army special operations forces and chinese police raid a drug smuggling operation in an abandoned chemical facility in southern china. the leader of the smuggling operation, wu ji, holds one of his own men hostage while taking cover behind a section of the facility  reinforced wall. leng feng, a skilled pla sniper, ignores orders to stand down and fires three shots at a weak section of the wall, penetrating through on the third shot and killing wu ji. leng feng is sent to solitary confinement as punishment, but is approached by long xiaoyun, the female commander of the legendary 'wolf warriors', an elite unit within the pla tasked with simulating foreign tactics for the pla to train against. long xiaoyun offers leng feng a place in the wolf warriors and he accepts, being airlifted to their training grounds the next day. the wolf warriors field leader lectures leng feng on the importance of teamwork to success. meanwhile in southeast asia, crimelord min deng is apprehended by authorities but rescued by group of foreign mercenaries led by ex-us navy seal 'tom cat' (scott adkins). it is revealed that min deng is the older brother of wu ji, the drug smuggler slain by leng feng in 2008. min deng re-hires tom cat and his group to assassinate leng feng, but warns them not to underestimate the strength and resolve of the people  liberation army. the wolf warriors participate in a training exercise in a remote and uninhabited forested region on china  southern border. during the exercise, tom cat and his mercenaries ambush a wolf warrior squad, killing one of leng feng  comrades. leng feng and pla helicopters pursue tom cat but he escapes, using graphite-powder weapons to deny airspace to pla air support. without quick-reaction air support to call on, pla commanders cancel the training exercise and send a combined force of the wolf warriors and people  liberation army special operations forces into the forest to apprehend tom cat and his mercenaries. while re-arming from helicopter-delivered supply drops, the wolf warriors are inspired by a speech from their commanders to defend the honour of china and avenge the death of their comrade. the combined infantry force move into the forest but are delayed by multiple traps set by tom cat and pinned down by sniper fire from assassin, tom cat  elite marksman. leng feng and the other soldiers cut down a tree with sheer firepower to create cover and leng feng sprints into the forest alone to flank assassin, dodging his fire and managing to engage and kill him at point-blank range. meanwhile the rest of the pla force engages tom cat  other mercenaries, who stage a fighting retreat but are eventually overwhelmed and killed one by one. at base, long xiaoyun and the other pla commanders deduce that ming deng himself is also in the training area to take possession of a smuggled cache of biotechnology, which could allow the creation of a genetic weapon that could target chinese people exclusively. leng feng again pursues tom cat through the forest, who delays him with multiple traps including a land mine, which leng feng triggers but escapes from by digging a pit under the mine and jumping away from the blast. he eventually catches tom cat just before the marker delineating china  southern border. both fighters empty their weapons at each other before engaging in close-quarters combat. leng feng, wounded and exhausted from the previous fights, is nearly defeated, but manages to kill tom cat with his own knife after tom cat mocks leng feng  patriotism. medical personnel from a pla relief force arrive, but leng feng recognises the wrist tattoo of the medic that approaches him and realizes that they are min deng  men in pla uniforms. he grabs a rifle and attacks them, eventually holding min deng himself at bayonet point on the very edge of the chinese border. min deng  massive paramilitary force approaches from the other side of the border, but so do the rest of the wolf warriors and pla soldiers. min deng  force retreats. after the action, commander long xiaoyun agrees to accompany leng feng for a drink.
__label__not_drama in a small indiana town in the mid-1910s, the winfield household – banker father george, his wife alice, their grown tomboyish daughter marjorie, their precocious trouble-making son wesley, and their exasperated housekeeper stella – have just moved into a larger house in a nicer neighborhood. no one but george is happy about the move, until marjorie meets their new neighbor, william sherman, home on a break from his studies at the university of indiana. the two are immediately attracted to each other, which makes margie change her focus from baseball to trying to become a proper young woman. their resulting relationship is despite, or perhaps because of bill  unconventional thoughts on life, including not believing in the institution of marriage, or believing in the role money plays in society. the road to a happy life between margie and bill is not only hindered by distance as bill returns to school and margie  attempts to learn feminine things, but also george  dislike of bill because of their differing beliefs, the stuffy hubert wakely also trying to court margie (he who is george  choice as an appropriate suitor for her), wesley  continual meddling in his sister  life, and world war i. one of those issues may be overcome when wesley receives a gift from aunt martha that used to be his father old slingshot that he used to kill aunt martha ' best hen. his father discovers the old slingshot after wesley cracks a window with it, his father gets emotional after he sees it and everything is resolved in time for a happy ending.[2]
__label__not_drama when johnny quinlan loses his job in a drug store, he is afraid to tell his wife, bertha, and therefore keeps up the pretense of leaving each morning for a non-existent job, as he begins the search for a new job. as the days pass and he is unable to find employment, their household, which includes his sister, lottie, and bertha  brother, dan walsh, goes through what little savings they have. as he gets more desperate, he agrees to do small jobs for "nifty" herman, a small-time gangster. nifty had loaned johnny $15, as part of a plan to entice him to work for him. after johnny gets insulted by a common laborer job offer from a neighbor, nifty lies to him and says that he has a friend who will get him a managerial position at a liquor store. all johnny has to do is hold onto a case of high-priced alcohol for a few days. dubious, johnny reluctantly agrees and takes the suitcase back to his apartment. however, when bertha finds out who he got the suitcase from, she demands that he return it, threatening to leave him if he does not . taking the case back to nifty, he finds the office locked, and so returns home. when he arrives, his sister  suitor, charles newton, is visiting. newton is a government agent. even though johnny tries to hide the case, his efforts are futile, and newton spies it and becomes suspicious, seeing a resemblance to a case he and his men have been attempting to track down. opening it, he discovers it contains a cache of drugs. when he interrogates johnny, he gets the whole story, and is about to arrest johnny, when nifty arrives to retrieve the suitcase. johnny tricks nifty into confessing, and then subdues him, when he is resisting the efforts of newton and his deputies to arrest him. the film ends with johnny being rewarded for the way he handled himself by becoming newton  assistant.
__label__not_drama the film is divided into twenty-six individual chapters, each helmed by a different director assigned a letter of the alphabet. the directors were then given free rein in choosing a word to create a story involving death. the varieties of death range from accidents to murders.[6] a contest was held for the role of the twenty-sixth director. the winner was uk-based director lee hardcastle, who submitted the claymation short for t.[7]
__label__drama police inspector jai (govinda) lives a middle-class life with his mother (aruna irani) and physically and mentally challenged brother sunder (govinda in double role). one day he arrests a man named pandey, and after questioning him, finds out that the person behind this crime was a man named guman singh (raza murad). he gets on the trail of guman singh, arrests him, and keeps him in a cell. then a series of explosions creates havoc in the city, with the assailants demanding the unconditional release of guman. with the police having no clue as to the assailants whereabouts or where they are going to strike next, they release guman. they threaten jai with dire consequences, forcing jai to resign from his job. hoping that he and his family are safe, jai starts to re-build his life, along with his sweetheart, sharmila (divya bharti). then tragedy strikes, his mother is killed in an accident, and his brother is abducted by jagtap singh (kiran kumar) and guman, who demand that he return a bag containing their valuables. jai is willing to do anything to get back his brother, but how can negotiate when he does not even have the bag in his possession?

(8604, 33)
__label__not_drama music publishers wilmer popday and alfred bandle find themselves unwittingly embroiled in an espionage adventure, when they go away on manoeuvres with the home guard.
__label__not_drama easwaran (arjun sarja) is upright and honest youth, who runs a local cable channel in kumbakonam. he is known to fight for causes of the society and the common man. he enters into fisticuffs with local mla (sai kumar) after he exposes his corrupt and greedy ways through his cable tv channel. fearing trouble, his mother takes him to a saint in thiruvannamalai. the swami resembles easwaran. a sequence of events forces them to swap places. the soft-spoken swami tries to sort all issues through non-violent means (gandhian philosophy). halfway through karunas dies in the hand of (vidharth) and the corrupt mla. in the climax easwaran kills the mla.
__label__drama two detectives, played by john elliot and harold goodwin, board a train in pursuit of a gangster, who is played by neil hamilton. hamilton is travelling to new york to work for edward g. robinson, a notorious gangster who owns a nightclub. hamilton jumps off the train near a bridge crossing and since no trace of him can be found the police believe him to be dead. goodwin assumes hamilton  identity and joins robinson  gang but is quickly discovered to be an imposter and shot. determined to find out who killed her brother, alice white poses as hamilton  widow and attempts to get a job at robinson  nightclub. hamilton eventually shows up and white is almost exposed as an imposter. hamilton, however, is eventually persuaded by white and he promises not to tell robinson the truth. white and hamilton fall in love. during a hold-up, white protects hamilton by shooting at a cop in the back. hamilton begins to think of reforming due to white  influence. white eventually gets robinson to confess that he shot her brother by pretending to be interested in him. she leaves the phone off the hook while the police listen in on his confession. when the police show up, robinson realizes what white has done and uses white as a shield against the police but hamilton manages to save white and robinson is forced to surrender to the police.
__label__not_drama mark bannister, a stockbroker, and his wife jessie, a tv reporter/anchorwoman, are a successful yuppie couple with an idyllic california life, aside from a toilet with a faulty handle. it is interrupted when both of them find out that mark  timid cousin, fred, and his pregnant wife, bernice, are flying in from new jersey to visit that same day. the first days are slightly chaotic - particularly because of bernice  cat - yet bearable. mark gives them 300 dollars to spend in the city and grant the couple some alone time. they are interrupted when jessie  gold digger sister claudia arrives demanding hospitality. she had a fight with her rich middle eastern husband kaddir, whom she divorces after he cancels her credit cards. fred and bernice  visit, meant to last only five days, is extended when bernice falls on the way to the car. she is instructed by her doctor, dr. penix, not to leave until the baby is born. at a local bar, mark motivates fred to quit being bernice  pet, but fred takes the message too far and leaves "to find himself." meanwhile, mark  next door neighbor and carpenter dale builds a machine for bernice to be comfortable in bed all day. having to be waited on hand-and-foot, bernice becomes increasingly irritating, even demanding constant burials for her cat who  would ies' multiple times (but comes back each time). claudia  son jonathan also comes to live with them. jessie tries getting dale to seduce claudia, but things take a turn for the worse when mark and jessie inadvertently burn down dale  villa, which can not  be rebuilt for three months. as a result, dale, his delinquent son c.k. and his phone-obsessed daughter katy move in; mark and jessie are forced to accept them to avoid an arson lawsuit. mark helps jonathan get a job as a mail room clerk. as the days pass, chaos persists for the couple with them essentially being forced out of their own home. mark  friend and colleague wes finds them outside living like hippies, after mark fails to show up for work. wes motivates mark to resist a little longer, especially as mark is on the verge of closing a successful deal for his boss, bob grindle. at work, mark gets a box from bogota containing cocaine - sent to him but requested by jonathan. grindle tells mark to sell a set of stocks due to a scandal, but mark forgets to before leaving work with the cocaine. fred comes back, having grown a mustache and now owning a baby elephant. police arrive at the house and find the cat overdosing on cocaine; they subsequently destroy mark  house during their drug bust, which gets televised by jessie  tv station. the stress for jessie manifests into an expletive-filled mental breakdown on live tv. pushed to their limit and facing apparent ruin and imminent charges, mark and jesse decide to abandon the house to the guests and leave town to start new lives. the next day, mark and jessie return to salvage what they can find. they hear a recording from dr. penix stating bernice was never pregnant to begin with, breaking what little sanity they had left. an enraged jessie catapults bernice from her bed to the backyard and forces her to confess to knowing she was not  pregnant. she then ruins her sister  expensive clothes to force claudia out while mark terrorizes dale with an electric saw, making him leave with his two children. lastly, jessie puts fireworks on jonathan  cocaine bag, which explodes as he tries to flee in dale  (loaner) lotus. mark and jessie then threaten to torch their own house in order to keep away their parasitic visitors for good. the police arrive and apologize, stating that their only evidence - bernice  cat - disappeared, and that they will pay for all damages incurred. grindle arrives and, believing mark had meant to keep the stocks, declares that he amassed a small fortune when the scandal was found to be false. he offers mark a portion of the profit and a promotion. claudia takes the opportunity to seduce grindle, and dale starts flirting with one of the police officers, much to c.k  chagrin. the cat returns, but has changed sides, preferring to remain with mark and jessie. bernice and fred depart, with fred taking much more control than before. jessie and mark try to get cozy before mark, as a last touch, smashes the ever-malfunctioning toilet with a sledgehammer. in the epilogue it states that mark and jessie - who earned a tv show of her own due to her angry on-air outburst - move to a 3 bedroom house in malibu and live happily ever after... until their parents come to visit.
__label__not_drama sam clayton is too good for his own good. a sermon by rev. daniels persuades him to help others in every way he can, including his wife lu  good-for-nothing brother, claude, who  been living with them rent-free for six months, and their neighbors the butlers, who need a car for a vacation when theirs breaks down. sam is a department store manager whose boss, h.c. borden, wants him to sell more and socialize less. sam  a shoulder for clerk shirley mae to cry on when her romance breaks up. he also gives a $5,000 loan, without his wife  knowledge, to mr. and mrs. adams, who need it to save a gas station they bought. lu is fed up with sam  generosity, particularly when he ends up paying for the butlers' car repairs, then letting the mechanic come over for home-cooked meals. the last straw for lu comes when she learns they have a chance to put a down payment on a new house, except sam has lent their nest egg to the adamses. sam is unhappy, too. he  annoyed with the butlers, who have crashed his car and can not  pay to fix it. he also wants claude to move out. shirley mae  troubles come to his door after she takes too many pills. sam even gets robbed, and the bank refuses to make him a loan. he is at his wit  end when the adamses surprise him with a check for $6,000. they also give claude a job, and shirley mae suddenly thinks she and claude could have a future together. sam and lu feel better about life, particularly when borden surprises him with a promotion at work.
__label__not_drama a mysterious black freighter sails through the night, apparently deserted. detecting a cruise ship close by, the ship alters course as disembodied voices announce in german, "battle stations! enemy in sight!" aboard the cruise ship, the prickly captain ashland is making his final voyage, attended by his replacement, captain trevor marshall, who has brought along his family. the freighter heads right for them, blasting its horn. despite ashland  best efforts, the charging freighter collides with the cruise ship, sinking it. (some of the sinking scenes were taken from the 1960 film the last voyage and darkened to match the nightly effect. several were also taken from the 1972 disaster film the poseidon adventure.) the next morning, a handful of survivors—marshall, his wife margaret (howes), and their children robin and ben; a young officer named nick and his love interest lori; the ship  comic jackie; and a passenger, mrs. morgan—are adrift on a large piece of wreckage. ashland surfaces nearby and he  brought aboard, barely conscious. later, the survivors come upon the black freighter, unaware it  the ship that attacked them. finding a boarding ladder slung from the stern, they climb aboard, but not before the ladder plunges into the sea as the officers try to climb it with the injured ashland. when all are finally aboard, jackie tries to rally the survivors with humor, but a cable seizes him by the ankle, and he is swung outboard by one of the ship  cranes, which lowers him into the water before cutting him loose, to be swept astern and lost. shocked, the survivors explore the corridors of the empty, echoing vessel, finding only cobwebs and 1930s memorabilia. hatches open and close by themselves and lights go on and off while a swinging block knocks out nick, who is exploring above deck. meanwhile, a delirious ashland hears a mysterious voice speaking to him in german, telling him "this ship has been waiting for you...it is your new ship....your chance has come!" the others finally set up in a dusty bunk room, and they separate to retrieve supplies and the injured captain. mrs. morgan finds a gramophone and a movie projector that suddenly turn on by themselves. while watching the film (1936  everything is rhythm) and eating a piece of hard candy from one of the ship  cupboards, she becomes grotesquely deformed. terrified, she stumbles back to the bunk room, where a possessed ashland strangles her. now awake, and apparently possessed by the ship  dead captain, ashland dons an officer  uniform from the kriegsmarine and announces that he is the captain. marshall and nick make a chilling discovery: the ship was once a kriegsmarine prison ship, and the ghosts of its inmates and crew are still aboard. upon visiting the chart room, they see that the map of the ship  course shows that it just travels round and round the atlantic in huge circles. later on, a large cabin is discovered which appears to be a nazi shrine, containing a variety of third reich memorabilia and an ante-room screened off by a pair of scarlet curtains, which part by themselves to reveal a large portrait of adolf hitler. they both decide to escape, but are thwarted when the ship  lifeboats lower into the sea by themselves and drift away. despairing, the survivors try to get some rest but are further taunted by ashland, who now prowls the ship  passages. lori goes into shock when her shower water turns to blood; she  then tossed overboard by ashland. marshall and nick are attacked by the piercing whine of the ship  electronics as the projector now begins showing old newsreel footage of adolf hitler. nick lunges at ashland, but instead plunges into a net holding skeletal remains, where ashland drowns him. marshall manages to stab a gloating ashland, apparently killing him and stopping the ship. searching below, marshall finds a life raft in a freezer full of the frozen bodies of downed raf airmen and soviet sailors, but as the children are jumping overboard, margaret is captured by a resurrected ashland and is trapped in a chain locker. marshall is knocked out by ashland, but awakens in time to find margaret, who has escaped from the locker. captain ashland attempts to shoot the escaping marshall family with a mauser rifle. meanwhile, the spirits of the crew detect another cruise liner and begin to give chase, ignoring ashland, who wants it to run down the marshalls' raft instead. trying to re-take control of the ship, ashland storms into the engine room and shoots at the machinery in vain, but falls into the steering gear and is crushed to death. his screams of agony echo throughout the ship, joining those of its earlier victims. above, the marshalls rejoice as the freighter turns and sails away. after drifting for some time, they are spotted by a search helicopter and rescued. the death ship is shown afterwards steaming along at full speed. the ghosts of the crew once again announcing "enemy in sight!" in german. it heads for another passenger ship, and the sounds of the collision are accompanied by the triumphant blasting of its horn.
__label__drama ernie mott (cary grant) is a restless, irresponsible, wandering cockney with a good musical ear. on armistice day, ernie visits the tomb of the unknown warrior in westminster abbey, which memorializes those who died in world war i, including his father. ernie wants a better life but does not want to settle down or work for it. when he returns home, ma mott (ethel barrymore) asks why he has returned after so long, and she gives him an ultimatum that he must stay home now or leave forever. he informs her that he will then be leaving next morning, and goes out to get a drink. he meets fellow musician aggie hunter (jane wyatt) outside the bar, but instead prefers the company of a gangster  fickle former wife, ada brantline (june duprez). however, when ernie becomes smitten with ada she rejects his offer of a date when he tells her he will be leaving town tomorrow. the next morning, ma mott tells her pawnbroker friend, ike weber (konstantin shayne), that she has cancer. ma and ernie get into another fight, but after he storms out, ike shares with him that his mother needs him in her battle with cancer. ernie returns and says that he will stay with her at home and help her run her shop. a month passes, and ernie continues to pursue ada. however, when gangster jim mordinoy (george coulouris) informs him that she  still his wife, ernie does not  believe ada when she says that  a lie and he cuts her off socially. ernie begins to notice the poverty surrounding him in london, and chooses to accept mordinoy  offer to join his activities, even against ada  pleas. ernie begins to steal cars, and he is involved in a police chase until his car collides with a truck and explodes into flames. ada implores him to run away with her, but he does not want to leave his dying mother. when ernie is eventually bailed out of jail by ike, he finds out that after the police find ernie  platinum cigarette case—his birthday gift from ma—was stolen, the police arrest her and put her in prison. she begs for forgiveness for shaming the family, and dies in prison hospital. when he returns home, he learns via a letter from ada that she decided to stay with mordinoy because that would make her life easier. ernie is crushed, and walks along the street until he gets to aggie  door and walks in.
__label__not_drama the plot and subplots center on the movie  leading man, paul harris, a divorced high school teacher and basketball coach and his interactions during a killer rat infestation with a health department inspector, kelly leonard, a high school cheerleader, trudy white, his friend a professor amd rat expert, dr. louis spenser, and his students. giant rats the size of small dogs living in mountains of grain full of steroids are rendered homeless when kelly orders it to be burned down. the rats migrate to a suburban home occupied by unsupervised high school students. the rats begin to kill a toddler left unattended in a high chair. henry younger, a senior citizen is attacked from rats, walking the snowy streets alone late at night. george foskins, a health department field inspector, is destined to become the next victim. inspecting the bowels of the city  sewer system, he encounters a pack of rats. george commences with 'the running of the rats' in the sewer, followed by his demise. paul telephones his friend, louis, who instantly concludes that the steroid feed had spawned a new breed of  uper-rat' that had migrated into the sewers in search of food. upon hearing this information, the health inspector immediately orders the fumigation of the entire city sewer system, which is immediately and efficiently carried out with no apparent effect. the futility of this action is confirmed when the rat expert himself is attacked and killed. the rats then move on to a bowling alley and a movie theater and make their first brazen mass attack on the unsuspecting public. the rats select the viewers of a bruce lee movie, resulting in numerous deaths including trudy, and a scene of fleeing film patrons. meanwhile, the city  mayor, mayor rizetti, prepares to make an inaugural run of a new subway section—straight into a hungry pack of rats. while escaping the rats, and sacrificing the other delegates, the mayor rizetti stows away on the empty subway train. paul attempts to stop the festivities. he is forced to assault a cop and take his revolver. paul proceeds down the tunnel and finds a disabled subway train with passengers just emerging from the cars. paul finds kelly and his son just as the rats attack the passengers and mayor rizetti is killed by rats off screen. the three are able to escape down the tunnel and find refuge in a fenced-in maintenance area which is the rats nest. fortunately, they discover cutting torches and drums of flammable liquids and soon, the three escape from the tunnel as the rats are killed during a flammable explosion behind him. after the three survivors re-board the subway train to reach safety, the train reaches the platform and the party-goers for the new subway section approach the train and to shock, it  revealed that the three had boarded the same train as the mayor rizetti did, which shows a small amount of surviving rats eating the body of the mayor as the final scene shows a bloody rat hissing at the train window.
__label__drama jasmine francis (cate blanchett) disembarks in san francisco after a flight from new york city. she takes a taxi to her sister ginger  (sally hawkins) apartment, where ginger is dismayed to learn that jasmine traveled first class despite claiming to be broke. jasmine has recently suffered a nervous breakdown and, having incurred heavy debts, has been forced to seek refuge with her sister. a series of flashbacks reveal that jasmine  husband, money manager hal francis (alec baldwin), was arrested for defrauding his clients. ginger and her husband, augie (andrew dice clay), were among hal  victims; he swindled them out of $200,000 of lottery winnings that augie had wanted to start a business with, and their marriage fell apart. hal committed suicide in prison after being publicly disgraced. jasmine  step-son danny (alden ehrenreich) dropped out of harvard and cut himself off completely from jasmine, believing her to be complicit in hal  crimes. after hal  death, jasmine began drinking heavily and abusing anti-anxiety medication. she also developed a habit of talking to herself about her past. ginger is now dating a mechanic called chili (bobby cannavale), whom jasmine detests for his low breeding and coarse manners. she considers becoming an interior designer because of her "great taste" and past experience in decorating her homes. she wants to take online courses, but, having no computer skills, she decides to take a class in computers to gain basic proficiency. with no income, she grudgingly takes a job as a receptionist with a dentist (michael stuhlbarg), who pesters her with unwanted sexual advances. when he tries to assault her, she fights him off and quits. jasmine  situation improves when she meets a wealthy widower, dwight (peter sarsgaard), at a party. dwight is a diplomat aspiring to become a congressman. she poses as an interior designer, telling him that her husband was a surgeon who died of a heart attack. dwight is impressed by her stylishness and invites her to decorate his new home. ginger begins a romance with al (louis c.k.), whom she met at the same party. she breaks up with chili, who begs her not to leave him. eventually, she finds out that al is married and gets back with chili, realizing she has been influenced by jasmine. jasmine develops a romance with dwight, and he is about to buy her an engagement ring when they bump into augie outside the jewelry store. augie rails at jasmine about what hal did to ginger and him. augie also reveals that danny is living nearby in oakland and is now married. dwight is outraged that jasmine lied to him and calls off the engagement. jasmine goes to oakland and finds danny, who tells jasmine he never wants to see her again because of what she did to his father. it is revealed that jasmine finally learned of hal  many affairs and confronted him. when he told her he wanted to divorce her to be with a 19-year-old au pair, jasmine, in a moment of blind rage, called the fbi to inform the authorities of hal  fraudulent business dealings. this led to his arrest. jasmine returns to her sister  apartment and finds ginger back with chili, who is moving in now. jasmine and chili needle each other, and jasmine is furious when ginger takes his side. jasmine lies to ginger claiming that she is going to marry dwight and leaves. jasmine takes a seat on a park bench and starts muttering to herself.
__label__not_drama after their admission in college, hunny, choocha and lali enjoy their life in delhi. zafar keeps up with his business. bholi punjaban is released after 12 months in prison. she sends her troops to cease the fukras. the boys are brought up at her place along with pandits. she asks them about their plan to fool her. as the boys convince her, she gives them one chance and invests some money to open a lottery shop. people are given numbers to which they invest some money. choocha begins to distribute brochures around the city and more people invest. meanwhile, minister babulal bhatia argues with the cm over the next election and bets him to win. police around the city conduct raids on drugs and fake lottery shops. after finding out the fukras' shop, babulal uses his power to change the lottery number and everyone who invested money are angered. the people gather around to beat them but they escape. they wake up at a construction area and spotted by pandit. it is here that choocha comes to know he can see the future. he sees a tiger in his dream which later attacks them. the group is later caught by bholi who is angered due to their foolishness, and orders to kill them. but choocha  ability impresses her and she forgives them. the next morning, bholi narrates them to find a treasury which they agree if she releases them. babulal fears that people would get back money so he invites the boys at his residence. pandit is sent in disguise to keep a check. during a diner, the boys are forced to take an oath to help babulal with their welfare in return. the next day, the boys take the blame instead of babulal who misused people  money while bholi goes to the cm to hand some factory papers related to babulal and after the boys leave, police arrive and capture babulal. the boys return to their houses and but feel dejected by their loved ones. they decide to scheme up to earn more money. the next morning, at pandit  place, choocha is forced to see a dream where lali holds a tiger cub. meanwhile, babulal comes to know about bholi  plan and ambushes her place. but he forgives her murder on the pretext to find a treasury. to do so, bholi calls hunny for it who requests a tiger cub which she sold. the boys and bholi  guards returns at the park where they are joined by priya and neetu. they dig a hole and enter a cave where they are spotted by the cub  mother. however, after escaping, they lead to a factory with huge amounts of illegal goods. they find out the products to be worth $400 million. babulal arrives at the place only to find out that the warehouse actually constains babulal  trade products. he is angered and orders to banish them but police arrive on the scene. in a flashback, while estimating worth, zafar called the police knowing babulal bought this from china. choocha wills to propose bholi to which she replies him with a kiss. the police drive away from the scene. later, the fukras inaugurate an electronics shop with the products they found in the warehouse.
!./fasttext supervised -input train.txt -output modelRead 7M words
Number of words:  256034
Number of labels: 2
Progress: 100.0% words/sec/thread:  484620 lr:  0.000000 loss:  0.581957 ETA:   0h 0m
!./fasttext test model.bin test.txtN	8604
P@1	0.705
R@1	0.705
_____no_output_____


# Notebook from ruyuanzhang/ccnss2018_students
Path: module0/lif_neuron/lif_neuron.ipynb

## CCNSS 2018 pre-course preparation
# LIF neuron assignment

[source](https://colab.research.google.com/drive/1RtqqkgGfvVya-6xv8UVoYKI5U5oyprXs)_____no_output_____*Please execute the cell bellow in order to initialize the notebook environment*_____no_output_____

import matplotlib.pyplot as plt    # import matplotlib
import math                        # import basic math functions
import random                      # import basic random number generator functions

fig_w, fig_h = (6, 4)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})_____no_output_____

## Objectives

In this notebook we'll implement a *Leaky Integrate-and-Fire (LIF)* neuron with stochastic pre-synaptic input current, and visualize its key statistical properties._____no_output_____## Background

This neuron model is defined by a *membrane equation* and a *reset condition*:

\begin{align*}
\\
&\tau_m\,\frac{d}{dt}\,V(t) = E_{L} - V(t) + R\,I(t) &\text{if }\quad V(t) \leq V_{th}\\
\\
&V(t) = V_{r} &\text{otherwise}\\
\\
\end{align*}

where $V(t)$ is the membrane potential, $\tau_m$ is the membrane time constant, $E_{L}$ is the leak potential, $I(t)$ is the pre-synaptic input current, $V_{th}$ is the firing threshold, and $V_{r}$ is the reset voltage.

We'll extract and visualize the mean, standard deviation and histogram of the following quantities:

* Pre-synaptic input current $I(t)$
* Membrane potential $V(t)$
* Output firing frequency $\lambda(t)$

The problem will be split in several steps:

* Discrete time integration of $V(t)$ under sinusoidal pre-synaptic input, and without spikes
* Stochastic pre-synaptic input
* Visualizing ensemble statistics
* Introducing output spikes_____no_output_____**EXERCISE 1**

Initialize the main simulation variables.

**Suggestions**
* Modify the code below to print the main simulation parameters_____no_output_____

# insert your code here

# t_max = 150e-3   # second
# dt = 1e-3        # second
# tau = 20e-3      # second
# el = -60e-3      # volt
# vr = -70e-3      # volt
# vth = -50e-3     # volt
# r = 100e6        # ohm
# i_mean = 25e-11  # ampere

# print(t_max, dt, tau, el, vr, vth, r, i_mean)_____no_output_____

**EXPECTED OUTPUT**

```
0.15 0.001 0.02 -0.06 -0.07 -0.05 100000000.0 2.5e-10
```_____no_output_____### ODE integration without spikes

The numerical integration of the membrane equation can be performed in discrete time, with a sufficiently small $\Delta t$. We start by writting the membrane equation. without taking the limit $\Delta t \to 0$ in the definition of the time derivative $d/dt\,V(t)$:

\begin{align*}
\\
\tau_m\,\frac{V\left(t+\Delta t\right)-V\left(t\right)}{\Delta t} &= E_{L} - V(t) + R\,I(t)\\
\\
\end{align*}

The value of membrane potential $V\left(t+\Delta t\right)$ can be expressed in terms of its previous value $V(t)$ by simple algebraic manipulation. For *small enough* values of $\Delta t$ this provides a good approximation of the continuous time integration._____no_output_____**EXERCISE 2**

Compute the values of $V(t)$ between $t=0$ and $t=0.01$ with $V(0)=E_L$ and pre-synaptic input given by:

\begin{align*}
\\
I(t)=I_{mean}\left(1+\sin\left(\frac{2 \pi}{0.01}\,t\right)\right)
\\
\end{align*}

**Suggestions**
* Express $V\left(t+\Delta t\right)$ in terms of $V(t)$
* Initialize the membrane potential variable `v` to `el`
* Loop in the time variable `t` from `t=0` to `t=0.01` with time step `dt`
* At each time step
    * Compute the current value of `i`
    * Update the value of `v`
    * Print `v`
* Use `math.pi` and `math.sin` for evaluating $\pi$ and $\sin(\,)$, respectively_____no_output_____

# insert your code here_____no_output_____

**EXPECTED OUTPUT**

```
-0.05875
-0.056827768434634406
-0.05454755936753374
-0.05238136075378811
-0.05077756115073311
-0.049988683093196457
-0.04997398050390223
-0.05041410212407606
-0.0508322176632412
-0.050775338345444725
-0.050775338345444725
```_____no_output_____**EXERCISE 3**

Plot the values of $V(t)$ between $t=0$ and $t=t_{max}$ under the same conditions.

**Suggestions**
* Update end time of loop to `t_max`
* Replace printing command with plotting command (with keyword `'ko'`)_____no_output_____

# insert your code here
_____no_output_____

**EXPECTED OUTPUT**

![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_3.png)_____no_output_____## Stochastic pre-synaptic input

From the standpoint of neurons, their pre-synaptic input is random. We'll improve the pre-synaptic input model by introducing random input with similar statistical properties:

\begin{align*}
\\
I(t)=I_{mean}\left(1+0.1\sqrt{\frac{t_{max}}{\Delta t}}\,\xi(t)\right)\qquad\text{with }\xi(t)\sim U(-1,1)\\
\\
\end{align*}

where $U(-1,1)$ is the uniform distribution with support $x\in[-1,1]$.

A random pre-synaptic input $I(t)$ results in a random time course $V(t)$._____no_output_____**EXERCISE 4**

Plot the values of $V(t)$ between $t=0$ and $t=t_{max}$ with random input $I(t)$.

Initialize the (pseudo) random number generator to a fixed value to obtain the same random input across executions of the code (initial value `0` will reproduce the expected output). The function `random.seed()` initializes the random number generator, and `random.random()` generates samples from the uniform distribution between `0` and `1`.

**Suggestions**
* Use the function `random.seed()`to initialize the random number generator to `0`
* Use the function `random.random()` to generate the input at each timestep
* Repeat the execution several times to verify that $V(t)$ has a random time course_____no_output_____

# insert your code here
_____no_output_____

**EXPECTED OUTPUT**

![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_4.png)_____no_output_____## Ensemble statistics

Multiple runs of the previous exercise may give the impression of periodic regularity in the evolution of $V(t)$. We'll collect the sample mean over $N$ realizations of $V(t)$  with random input in order to test such hypothesis. The sample mean, sample variance and sample autocovariance at times $\left\{t_k, t_j\right\}\in[0,t_{max}]$, and for $N$ realizations of $V(t)$ are given by:

\begin{align*}
\\
\left\langle V(t_k)\right\rangle &= \frac{1}{N}\sum_{n=1}^N V(t_k)_n & & \text{sample mean}\\
\left\langle (V(t_k)-\left\langle V(t_k)\right\rangle)^2\right\rangle &= \frac{1}{N-1} \sum_{n=1}^N \left(V(t_k)_n-\left\langle V(t_k)\right\rangle\right)^2 & & \text{sample variance} \\
\left\langle \left(V(t_k)-\left\langle V(t_k)\right\rangle\right)\left(V(t_j)-\left\langle V(t_j)\right\rangle\right)\right\rangle
&= \frac{1}{N-1} \sum_{n=1}^N \left(V(t_k)_n-\left\langle V(t_k)\right\rangle\right)\left(V(t_j)_n-\left\langle V(t_j)\right\rangle\right) & & \text{sample autocovariance}\\
\\
\end{align*}_____no_output_____**EXERCISE 5**

Plot multiple realizations of $V(t)$.

**Suggestions**
* Create a list to store the value of $V(t)$ for each realization.
* At each timestep loop over `N=50` realizations of random input
* Plot the realizations at each time step with keyword `'ko'` and parameter `alpha=0.05`_____no_output_____

# insert your code here
_____no_output_____

**EXPECTED OUTPUT**

![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_5.png)_____no_output_____**EXERCISE 6**

Plot the sample mean $\left\langle V(t)\right\rangle$ over `N=50` realizations.

**Suggestions**
* Plot the sample mean at each timestep with keyword `'C0o'` and parameter `alpha=0.8`_____no_output_____

# insert your code here
_____no_output_____

**EXPECTED OUTPUT**

![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_6.png)_____no_output_____**EXERCISE 7**

Add sample standard deviation $\sigma(t_k)\equiv\sqrt{\text{Var}\left(t_k\right)}$ to the plot.

**Suggestions**
* After collecting the sample mean, iterate a second time to compute the sample variance $\sigma(t_k)$
* Plot $\left\langle V(t)\right\rangle\pm\sigma(t_k)$ with keyword `'C0o'` and parameter `alpha=0.8`._____no_output_____

# insert your code here
_____no_output_____

**EXPECTED OUTPUT**

![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_7.png)_____no_output_____## Sample histogram

Another important statistic is the sample histogram at time $t_k\in[0,t_{max}]$, which for $N$ realizations $V\left(t_k\right)$ and $J$ bins is given by:

\begin{align*}
\\
N = \sum_{j=1}^{J} m_j\\
\\
\end{align*}

where $m_j$ is a function that counts the number of samples $V\left(t_k\right)$ that fall into bin $j$._____no_output_____**EXERCISE 8**

Plot an histogram of $J=25$ bins of $V(t)$ for $t=t_{max}/10$ and $t=t_{max}$ for $N=10000$ realizations.

The function `plt.hist(data, nbins)` plots an histogram of `nbins` bins of the list 'data' and returns the lists `pdf`, `bins`, `patches`.

`pdf, bins, patches = plt.hist(data, bins)`

The parameters `histtype='stepfilled'` and `linewidth=0` may improve the plots.

**Suggestions**
* Plot the 1st histogram after running up to $t=t_{max}/10$
* Plot the 2nd histogram at $t=t_{max}$
* Add labels to identify the time of each histogram
* Plot each histogram with parameters `histtype='stepfilled'` and `linewidth=0`_____no_output_____

# insert your code here
_____no_output_____

**EXPECTED OUTPUT**

![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_8.png)_____no_output_____## Output spikes

Introduce the *reset condition*: $V(t) = V_{r} \text{if }\quad V(t)\geq V_{th}$_____no_output_____**EXERCISE 9**

Insert the reset condition, and collect the spike times of each realization in a dictionary variable `spikes`, with $N=500$.

**Suggestions**
* Reset $V(t)$ to $V_r$ if $V(t)\geq V_{th}$ and collect the spike time
* Plot each realization with keyword `'ko'` and parameter `alpha=0.01`
* Plot the sample mean with keyword `'C0o'` and parameter `alpha=0.8`
* Create a new figure and plot the spike times with keyword `'k.'` and parameter `alpha=0.2`
* Create a new figure and plot and the mean spiking activity_____no_output_____

# insert your code here
_____no_output_____

**EXPECTED OUTPUT**

![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_9a.png)
![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_9b.png)_____no_output_____**EXERCISE 10**

Investigate the effect of refractory period $t_{ref}$ on the evolution of output rate $\lambda(t)$.

**Suggestions**
* Add refractory period $t_{ref}=10$ ms after each spike, during which $V(t)$ is clamped to $V_r$
* Experiment increasingly higher values of refractory period $t_{ref}$_____no_output_____

# insert your code here
_____no_output_____

**EXPECTED OUTPUT**

![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_10a.png)
![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_10b.png)_____no_output_____**EXERCISE 11**

Below is a proposed implementation for Exercise 7 from above. Rewrite this code, but break it up into functions. Specifically, write and use function to:
* `I(n)`: returns the current, $I(t)$, at a given time $t$, a list of $n$ floats
* `mean(x)`: returns the mean of `x`, a list of floats
* `std(x)`: returns the standard deviation of `x`, a list of floats
* `ode_step(v, i)`: returns the voltage at the next time step, $V(t+\Delta t)$, given the voltage at the current time step, $V(t)$, and the current, $I(t)$, all lists of floats_____no_output_____

# insert your code here
_____no_output_____

**EXPECTED OUTPUT**

![](https://github.com/ccnss/ccnss2018_students/raw/master/module0/lif_neuron/figures/lif_11.png)_____no_output_____**EXERCISE 12**

Rewrite the code from Exercise 11, but now use the tools from NumPy. For example:
* Use `np.array` instead of Python list for voltage, $V(t)$, and time, $t$
* Use the NumPy functions for statistical function, (e.g. `np.mean`) and mathematical (e.g. `np.sqrt`) operations
* Plot mean $\pm$ std using `plt.fill_between` with parameters `alpha='0.6'`
* Plot mean with parameter `linewidth=3`_____no_output_____

# insert your code here
_____no_output_____


# Notebook from gpetrini/pysolve3
Path: monetary-economics/SFC inet.ipynb

# Monetary Economics: Chapter 7_____no_output_____### Preliminaries_____no_output_____

# This line configures matplotlib to show figures embedded in the notebook,
# instead of opening a new window for each figure. More about that later.
# If you are using an old version of IPython, try using '%pylab inline' instead.
%matplotlib inline

from pysolve.model import Model
from pysolve.utils import is_close,round_solution

import matplotlib.pyplot as plt_____no_output_____

### Model BMWK_____no_output_____

def create_bmwk_model():
    model = Model()

    model.set_var_default(0)
    #model.var('AF', desc='Amortization funds')
    model.var('Cd', desc='Consumption goods demand by households')
    model.var('Cs', desc='Consumption goods supply')
    #model.var('DA', desc='Depreciation allowance')
    model.var('K', desc='Stock of capital')
    #model.var('Kt', desc='Target stock of capital')
    model.var('Ld', desc='Demand for bank loans')
    model.var('Ls', desc='Supply of bank loans')
    model.var('Id', desc='Demand for investment goods')
    model.var('Is', desc='Supply of investment goods')
    model.var('Mh', desc='Bank deposits held by households')
    model.var('Ms', desc='Supply of bank deposits')
    model.var('Nd', desc='Demand for labor')
    model.var('Ns', desc='Supply of labor')
    model.var('Rl', desc='Rate of interest on bank loans')
    model.var('Rm', desc='Rate of interest on bank deposits')
    model.var('W', desc='Wage rate')
    model.var('WBd', desc='Wage bill - demand')
    model.var('WBs', desc='Wage bill - supply')
    model.var('Y', desc='Income = GDP')
    model.var('YD', desc='Disposable income of households')

    ############
    model.var('rcf', desc='Fu/K(-1)')
    model.var('Ft', desc='Total Profits')
    model.var('Fu', desc='Retained earning')
    model.var('u', desc='Capacity utilization')

    model.set_param_default(0)
    model.param('alpha0', desc='Exogenous component in consumption')
    model.param('alpha1r', desc='Propensity to consume out of interest income')
    model.param('alpha1w', desc='Propensity to consume out of wage income')
    model.param('alpha2', desc='Propensity to consume out of wealth')
    model.param('delta', desc='Depreciation rate')
    #model.param('gamma', desc='Speed of adjustment of capital to its target value')
    model.param('kappa', desc='Capital-output ratio')

    #####
    model.param('gamma0', desc='Autonomous investiment')
    model.param('gamma1', desc='Sensitivity of cash-flow')
    model.param('gamma2', desc='Sensitivity leverage')
    model.param('gamma3', desc='Sensitivity of u')

    model.param('PR', desc='Labor productivity')
    model.param('Rlbar', desc='Rate of interest on bank loans, set exogenously')

    # Basic behavioural equations
    model.add('Cs = Cd')
    model.add('Is  = Id')
    model.add('Ns = Nd')
    model.add('Ls - Ls(-1) = Ld - Ld(-1)') # Sem restrição de crédito
    model.add('Y = Cs + Is')
    #model.add('WBd = Y - Rl(-1)*Ld(-1) - AF')
    model.add('WBd = Y - Rl(-1)*Ld(-1)')
    #model.add('AF = delta * K(-1)') # Retirado
    #model.add('Ld - Ld(-1) = Id - AF') #Alterado
    model.add('rcf = Fu/K(-1)')
    model.add('Ft = Y - W') # (Sem Eq, p. 5)
    model.add('Fu = Ft - Rl*Ld(-1)') # (5)
    model.add('Ld = Id - Fu')
    model.add('u = Y/K')

    # Transactions of households
    model.add('YD = WBs + Rm(-1)*Mh(-1)')
    model.add('Mh - Mh(-1) = YD - Cd')

    # Transactions of the banks
    model.add('Ms - Ms(-1) = Ls - Ls(-1)')
    model.add('Rm = Rl')

    # The wage bill
    model.add('WBs = W * Ns')
    model.add('Nd = Y / PR')
    model.add('W = WBd / Nd')

    # Household behavior
    model.add('Cd = alpha0 + alpha1w*WBs + alpha1r*Rm(-1)*Mh(-1) + alpha2*Mh(-1)')# Retirar

    # The investment beahavior
    #model.add('K - K(-1) = Id - DA')
    model.add('K - K(-1) = Id')
    #model.add('DA = delta * K(-1)') # Retirado
    #model.add('Kt = kappa * Y(-1)') # Retirado
    #model.add('Id = gamma * (Kt - K(-1)) + DA') # Alterado para (21) sem q de tobin
    model.add('Id = gamma0 + gamma1*rcf(-1) - gamma2*Rlbar*Ld/K + gamma3*u(-1)')

    # The behaviour of banks
    model.add('Rl = Rlbar')

    return model

bmwk_parameters = {'alpha0': 10,
                   'alpha1r': 0,
                   'alpha1w': 0.75,
                   'alpha2': 0,
                   'delta': 0.1,
                   'gamma0':10,
                   'gamma1': 0.1,
                   'gamma2': 0.1,
                   'gamma3': 0.8}
bmwk_exogenous = {'PR': 1,
                  'Rlbar': 0.05}
bmwk_variables = {'Mh': 185.2,
                  'Ms': 185.2,
                  'Ld': 185.2,
                  'Ls': 185.2,
                  'K': 185.2,
                  'Y': 185.2,
                  'Rl': 0.04,
                  'Rm' : 0.04,
                  'W': 0.86,
                  'Cd': 1,
                  'rcf':1}_____no_output_____

### Scenario: Model BMWK, increase in the interest rate on loans_____no_output_____

interest = create_bmwk_model()
interest.set_values(bmwk_parameters)
interest.set_values(bmwk_exogenous)
interest.set_values(bmwk_variables)

# run to convergence
# Give the system more time to reach a steady state
for _ in range(15):
    interest.solve(iterations=200, threshold=1e-5)

# shock the system
interest.set_values({'gamma0': 5})

for _ in range(100):
    interest.solve(iterations=10000, threshold=1e-5)_____no_output_____

###### Figure 7.8_____no_output_____

caption = '''
    Figure 7.8  Evolution of Gross Domestic Income $(Y)_{t}$, following an
    increase in the interest rate, in model BMWK'''
data_rcf = [s['rcf'] for s in interest.solutions[5:]]
data_l = [s['Ld'] for s in interest.solutions[5:]]
data_Y = [s['Y'] for s in interest.solutions[5:]]
data_Id = [s['Id'] for s in interest.solutions[5:]]

fig = plt.figure()
axes = fig.add_axes([0.1, 0.1, 1.1, 1.1])
axes.tick_params(top=False, right=False)
axes.spines['top'].set_visible(False)
axes.spines['right'].set_visible(False)
#axes.set_ylim(-1, 2)

axes.plot(data_Y, linestyle='-', color='b')
axes.plot(data_Id, linestyle='-', color='r')
# add labels
#plt.text(20, 182, 'National income')

fig.text(0.1, -.05, caption);_____no_output_____


# Notebook from gyleodhis/My-Data-Science-Devops
Path: Machine_Learning_Premer.ipynb

## Modeling
Models are a representation of things. They are not the things themselves. Models must be based on empirical observations (data)
__Generalization__ is key for supervised machine learning models. The model should avoid __high bias (underfitting)__ and __high variance (Overfitting)__ but should rather be __Just right (The Goldilocks model)
1. Underfittng - Uses two parameters, usually a straing line and does not capture most of the data points.
2. Overfitting - Uses five parameters (4th order polynomial). It goes through every single data point.
3. The Goldilock - Uses three parameters (Third order polynomial often called the parabola.)_____no_output_____Model testing through __Cross-Validation__ reduces the generalization error
We take all out training data and use it for training. Most people use five fold cross validation. But 10 fold can be used if the dataset is huge._____no_output_____#### Bias - Variance Trade off
This is ideally minimizing the generalization error - Which means minimizing the error between the training sample and the test sample.
- To avaoid overffiting you need to know when to stop training the model. Although the training set error may be decreasing you may be simply overfitting the training data.
- To test this apply the model to a validation data set (that is not part of the training set). If the validation data set error starts to diverge from the training set error then you will know that you have an overfit model and its time to stop the training.
- Where then do you stop exactly. Stop training where the generalization error is the smallest. That is the error between your training set error and validation set error is smallest._____no_output_____## Supervised Learning Concepts
- Data collection --> Formulation of a hypothesis --> Deduction (formulation of a predictive test) --> Experimental design and testing --> Evaluation --> REview of results.
_____no_output_____### Star - Galaxy theory
if radius < 6 then star, otherwise galaxy. Then:
- __True Positive (TP)__: A star correctly identified as a star
- __True Negative (TN)__: A galaxy correctly identified.
- __False Positive (FP)__: a galaxy classified as a star.
- __False Negative (FN)__: A star classified as galaxy

__Accuracy__ is now the error estimation as is calculated as:
accuracy = (TP+TN)total otherwise known as the F1 - score_____no_output_____### Measures of Accuracy
- Overall accuracy  = (TP + TN)/total
- Producers accuracy = TP/(TP+FN) or TN/(TN + FP)
- User's accuracy = TP/(TP+FP) or TN/(TN+FN)_____no_output_____