# Notebook from MLatCezeaux/ML101
Path: notebooks/Classification.ipynb

ML@ Cezeaux - by Emille Ishida

### Machine Learning Tutorial

# Section 1.b - Supervised learning: classification
by [Emille Ishida](https://www.emilleishida.com/)

### *Take home message 2: It is crucial to understand the algorithm you are using*

**Goal:** 1. Get acquainted with a basic machine learning algorithm
          2. Discuss hyper parameters
          3. Write your classifier

**Task**: Star-Galaxy Classification

**Data**: Clean data resulting from [Notebook 1](EDA_SDSS_answers.ipynb)

          ~9100 objects (lines - depends on how you extracted the outliers)
          11 features (columns)
          Features we are interested in:

$ug$: u-g SDSS color

$gr$: g-r SDSS color

$class$: source classification_____no_output_____

# import some basic libaries
import matplotlib.pylab as plt
import numpy as np
import pandas as pd
import seaborn as sns_____no_output_____

### Step 1: Pre-processing_____no_output_____We start by loading the data and checking if our variables of interest are given.

PS1: as we already cleaned this data, we will not perform an extensive EDA here, but remember that you should always make similar tests to those shown in Notebook 1 every single time... and again, just to be sure.

_____no_output_____

# load data
data = pd.read_csv('../data/SDSS_star_galaxy_clean.csv')

# check columns
data.keys()_____no_output_____data.shape_____no_output_____

We see that the data we want is there, but there is a lot of other stuff as well.
Select only the columns that are interesting to you. _____no_output_____

# make a new data frame with the columns you need

# confirm remaining columns
data_use.keys()_____no_output_____# take a look at the data you just separated
_____no_output_____

### Step 2: Separate your data samples

All supervised learning tasks are made of (at least) 4 phases:
          1) **train**: training samples (requires features and labels)
          2) **optimize**: validation sample (requires features and labels)
                    **Repeat 1-2 until you are happy with the results**
          3) **evaluate results**: test sample (requires features and labels)
          4) **predict**: target sample (there are only you features, you should believe your code at this point)

This means that your labelled sample needs to be divided in at least 3 samples: training, validation and test.
So, in order to continue we must construct these samples.
_____no_output_____

from sklearn.model_selection import train_test_split

# separate 60% for training and 20% for validation and 20% test
# WARNING: there is probably a smarter way to do this
X_train, X, y_train, y = train_test_split(data_use[['ug', 'gr']], data_use['class'], test_size=0.4, random_state=1)
X_test, X_validation, y_test, y_validation = train_test_split(X, y, test_size=0.5, random_state=1)

# check your samples (size, features, etc.)
print('training sample:    ', X_train.shape, y_train.shape)
print('validation sample:  ', X_validation.shape, y_validation.shape)
print('test sample:        ', X_test.shape, y_test.shape)_____no_output_____

### Step 3: Build a Nearest Neighbor classifier

As a start, we will construct a k-nearest neighbor algorithm (kNN) with k = 1.
This means that the class of a test point will be given by the class of its nearest neighbor.
We can describe this strategy as:

           For all objects in the unlabelled sample:
               1) Calculate the distance to all points in the training sample
               2) Identify its closest neighbor
               3) Assign the class of its nearest neighbor  _____no_output_____

# Build a 1-nearest neighbor classifier

def my_NearestNeighbor(train_features, train_labels, unlabelled_features):
    """
    Classify an unlabelled using a k=1 nearest neighbor algorithm.

    input: train_features - array, dim=(number of objects, number of features)
           train_labels - array, dim=(number of objects, 1)
           unlabelled_features - array, dim=(number of objects, number of features)

    output: estimated classes for all lines in test_features
            array, dim(number of objects, 1)
    """

    # calculate distances

    # assign for each element in the test sample the class of its nearest neighbor

    return
_____no_output_____

Estimate the classes of all objects in the test sample_____no_output_____

class_estimate = my_NearestNeighbor( )

# quick look in the first 10 estimated classes
class_estimate[:10]_____no_output_____

### Step 4:  Evaluate your results_____no_output_____Now that you have a classifier and already applied it, let's quantify the results.
Create a metric function that calculates the fraction of correct classifications. _____no_output_____

def metric(estimated_classes, true_classes):
    """
    Calculate the fraction of correct classification results.

    input: estimated_classes, array - dim=(number of objects,)
           true_classes, array - dim=(number of objects, )

    output: fraction of correct classifications
    """

    # get the number of objects in the sample

    # get the number of correct classifications

    return_____no_output_____# Calculate metrics
accuracy = metric(class_estimate, y_validation)

print('accuracy: ', accuracy)_____no_output_____

### Step 5: Optimize the classifier

The procedure used above aimed to illustrate how you can construct your own classifier, but it is not very practical. Whenever possible, we should make use of available libraries.

Let's reproduce what we did below using [scikit-learn](https://scikit-learn.org/stable/)._____no_output_____

from sklearn import neighbors
from sklearn.metrics import accuracy_score

# determine number of neighbors
nn = 1
weights = 'uniform'              # this can be 'uniform' or 'distance'

# create an instance of the classifier
classifier = neighbors.KNeighborsClassifier(nn, weights=weights)

# train (or fit) the classifier
classifier.fit(X_train, y_train)

# predict the classes of the te st samtple
class_estimate_sklearn = classifier.predict(X_validation)

# calculate metrics
accuracy_sklearn = accuracy_score(class_estimate, y_validation, normalize=True)

print('accuracy give by sklearn: ', accuracy_sklearn)_____no_output_____

Using scikit-learn allows us to easily manipulate the parameters of our algorithm.
Before we can move forward, we need to optimize these results to the best of our abilities.
Ask yourself: **Can the results above be improved ?**

*Try changing the number of neighbors and weights and see how the results change*_____no_output_____*Only go to the next step once you are satisfied with the results!!!*_____no_output_____### Step 6: Calculate final results

In order to ensure some generality to the results of the trained classifier, you must always report its performance in an independent data set, which was not used for training. The aim of the `test` sample is to mimic the results which you would find in the target sample._____no_output_____

# predict the classes of the test sample

# calculate metrics
accuracy_test = accuracy_score(class_estimate_test, y_test, normalize=True)

print('accuracy calculated on test sample : ', accuracy_test)_____no_output_____

And this is the accuracy of your optimized classifier.. and the numbers you should report when quoting any output from it!

However, this number is not the reason your classifier was built!
Now that you have a working classifier your desired output is a list of classes for each object in a completely new sample for which no labels exist.

So let's read still another data set for which no labels are known._____no_output_____

# load data
data_target = pd.read_csv('../data/SDSS_star_galaxy_target.csv')

# check features
data_target.keys()_____no_output_____

**Notice that for saving time, this data set has already been cleaned. In a realistic situation, all the pre-processing performed in the labelled data which does not involve the labels themselves should also be applied to the target sample!**

Now, let's use our classifier:_____no_output_____

# Use the classifier you optimized above to estimate the classes for the target sample
class_estimate_target =

# quick look into the first classes
_____no_output_____

#### You can now say that you have a new catalog of stars and galaxies, given by:_____no_output_____

class_estimate_target_____no_output_____

#### whose classes can be trusted to the level of:_____no_output_____

print(round(100*accuracy_test), '%')_____no_output_____

### You can know use this catalog to do your science!_____no_output_____

PS2: In this notebook used only 2 features due to time constraints, but feel free to try it with all collumns. Try accessing, for example, how your results change when you add more information.

_____no_output_____------------------------------------------------------------------------------------------------------------------
Summary:

## Machine Learning Model

**Task**:  Identify stars and galaxies based on their photometric measurements.

       input: 2 different colours in SDSS broad-band filters: u-g and g-r
       output: estimated classes

**Task Category**:   Classification

**Data**: Extract from SDSS-DR14 as available through [Kaggle](https://www.kaggle.com/lucidlenn/sloan-digital-sky-survey#Skyserver_SQL2_27_2018%206_51_39%20PM.csv)

        2 Features, x
              u-g       u - g colour in SDSS system
              g-r       g - r colour in SDSS system

        1 response variable (label), y
            class

**Machine Learning category**:

        Supervised Learning

**Set of possible samples**:

        ~9100 objects

**Set of possible labels**:

        2 classes: STAR or GALAXY

**Learner**:

        1 - Nearest neighbor

**Loss function**:

        1 - Distance-based_____no_output_____


# Notebook from mmayers88/TDA_Experiments
Path: Ripser Docs/notebooks/Lower Star Time Series.ipynb

# Lower Star Time Series

In this module, we will demonstrate the so-called "lower star filtration" or "sublevelset filtration" on a 1D time series, which is a stable way of describing critical points (local mins and maxes) in the time series.  Intuitively, we will compute H0 of the connected components that we get when "rising pools of water" above the graph of the time series.  When a new pool forms at a local min, that is a birth, and when two pool merge together at a local max, that is a death.  In the process, we will show how using nonzero elements along the diagonal of a sparse distance matrix can be used to set vertex birth times.

First, let's do the necessary imports_____no_output_____

import numpy as np
import matplotlib.pyplot as plt
from scipy import sparse
import time

from ripser import ripser
from persim import plot_diagrams_____no_output_____

Now let's set up a time series_____no_output_____

N = 100 #The number of points
t = np.linspace(0, 5, N)
x = np.cos(2*np.pi*t) + t

plt.plot(t, x)
plt.title("$\\cos(2 \pi t) + t$")
plt.xlabel("t")_____no_output_____

Now we will perform the sublevelset filtration using a sparse distance matrix.  There is a vertex for each point in the time series, which is born at the height of the point in the time series.  Then, edges connect adjacent points in the time series, and they are added at a height equal to the larger of the two heights of the vertices they connect._____no_output_____

# Add edges between adjacent points in the time series, with the "distance"
# along the edge equal to the max value of the points it connects
I = np.arange(N-1)
J = np.arange(1, N)
V = np.maximum(x[0:-1], x[1::])
# Add vertex birth times along the diagonal of the distance matrix
I = np.concatenate((I, np.arange(N)))
J = np.concatenate((J, np.arange(N)))
V = np.concatenate((V, x))
#Create the sparse distance matrix
D = sparse.coo_matrix((V, (I, J)), shape=(N, N)).tocsr()
dgm0 = ripser(D, maxdim=0, distance_matrix=True)['dgms'][0]
dgm0 = dgm0[dgm0[:, 1]-dgm0[:, 0] > 1e-3, :]
allgrid = np.unique(dgm0.flatten())
allgrid = allgrid[allgrid < np.inf]
xs = np.unique(dgm0[:, 0])
ys = np.unique(dgm0[:, 1])
ys = ys[ys < np.inf]

#Plot the time series and the persistence diagram
plt.figure(figsize=(12, 6))
ylims = [-1, 6.5]
plt.subplot(121)
plt.plot(t, x)
ax = plt.gca()
ax.set_yticks(allgrid)
ax.set_xticks([])
plt.ylim(ylims)
plt.grid(linewidth=1, linestyle='--')
plt.title("$\\cos(2 \pi t) + t$")
plt.xlabel("t")

plt.subplot(122)
ax = plt.gca()
ax.set_yticks(ys)
ax.set_xticks(xs)
plt.ylim(ylims)
plt.grid(linewidth=1, linestyle='--')
plot_diagrams(dgm0, size=50)
plt.title("Persistence Diagram")

plt.show()_____no_output_____

Notice how the birth times correspond to local mins and the death times correspond to local maxes_____no_output_____


# Notebook from YeoLab/single-cell-bioinformatics-scrm-2016
Path: 1._Introduction+Dealing_with_technical_noise/1.4_Additional_reading.ipynb

# Additional reading

## Introduction to single cell

#/5 - Olga's paper rating on a scale of 1 (worst) to 5 (best)

- 5/5 [Kolodziejczyk et al, *Mol Cell* (2015)](http://www.ncbi.nlm.nih.gov/pubmed/26000846)
    - Great figures and overview of single cell
- 5/5 [Bacher R & Kendziorski C, *Genome Biology* (2016)](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4823857/)
    - Describes different downstream machine learning analyses in *detail*
    - Fantastic table overview of current algorithms used in single cell analyses
- 4/5 [Stegle O et al, *Nature Review Genetics* (2015)](http://www.ncbi.nlm.nih.gov/pubmed/25628217)
    - Describes different expression normalization and QC steps in detail

## Technical noise

- 4/5 [Brennecke et al *Nature Methods* (2013)](http://www.ncbi.nlm.nih.gov/pubmed/24056876)
    - Frequentist based approach of assigning an appropriate gene expression threshold where you can trust the differences
- 4/5 [Kharchenko et al *Nature Methods* (2014)](http://www.ncbi.nlm.nih.gov/pubmed/24836921)
    - Bayesian based approach of finding genes that are differentially expressed based on how well that gene was captured across all cells (the prior)

## Batch correction

* 5/5 [COMBAT](http://biostatistics.oxfordjournals.org/content/8/1/118.abstract)
    * Adjust for batches in the data
    * This has >1000 citations!!
* 5/5 [Hicks et al](http://biorxiv.org/content/early/2015/12/27/025528)
    * Exposé meta-analysis of almost all published single-cell RNA seq papers so far and shows how many of them have confounded their biological signal with technical batches
* 4/5 [SVA](http://www.biostat.jhsph.edu/~jleek/papers/sva.pdf)
    * Can specify that you want to correct for something (like RIN) but don't correct for what you're interested in. But... often in single cell data you're trying to find new populations so you don't know *a prior* what you want to not be corrected for
* 4/5 [RUVseq](http://www.nature.com/nbt/journal/v32/n9/full/nbt.2931.html)
    * With the "RUVg" version can specify a set of control genes that you know aren't supposed to change between groups (maybe from a bulk experiment) but they say in their manual not to use the normalized counts for differential expression, only for exploration, because you may have corrected for something you actually *DID* want but didn't know
* 4/5 [scLVM](https://github.com/PMBio/scLVM)
    * This method claims to account for differences in cell cycle stage and help to put all cells onto the same scale, so you can then do pseudotime ordering and clustering and all that jazz.
_____no_output_____


# Notebook from jairNeto/warren_buffet_letters
Path: warren_buffet.ipynb

import numpy as np
from nltk.sentiment import SentimentIntensityAnalyzer
from transformers import pipeline
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
import pickle
from utils import *
import re

nltk.download(['punkt', 'wordnet', 'averaged_perceptron_tagger', 'stopwords', 'vader_lexicon'])[nltk_data] Downloading package punkt to /Users/jairneto/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package wordnet to
[nltk_data]     /Users/jairneto/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /Users/jairneto/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package stopwords to
[nltk_data]     /Users/jairneto/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package vader_lexicon to
[nltk_data]     /Users/jairneto/nltk_data...
[nltk_data]   Package vader_lexicon is already up-to-date!
[nltk_data] Downloading package punkt to /Users/jairneto/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package wordnet to
[nltk_data]     /Users/jairneto/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /Users/jairneto/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package stopwords to
[nltk_data]     /Users/jairneto/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package vader_lexicon to
[nltk_data]     /Users/jairneto/nltk_data...
[nltk_data]   Package vader_lexicon is already up-to-date!
letters_dict = get_letters_corpus_dict('/Users/jairneto/Downloads/')Could not open the url https://www.berkshirehathaway.com/letters/1982.html
# Write the dict to a pickle
with open('letters_dict.pickle', 'wb') as handle:
    pickle.dump(letters_dict, handle, protocol=pickle.HIGHEST_PROTOCOL)_____no_output_____# Read the dict from a pickle
with open('letters_dict.pickle', 'rb') as handle:
    letters_dict = pickle.load(handle)_____no_output_____frequent_words_list = ['year', 'business', 'berkshire', 'million', 'company', 'u',
                       'billion', 'tax', 'investment', 'share', 'last', 'value',
                       'would', 'many', 'operation', 'market', 'one', 'large', 'p',
                       'make', 'asset', 'also', 'see', 'even', 'well', 'two', 'made',
                       'pre', 'return', 'cost', 'capital', 'may', 'price', 'per', 'first',
                       'annual', 'though', 'however', 'time',
                       'manager', 'money', 'dollar', 'meeting',
                       'interest', 'great', 'group', 'come', 'far',
                       'long', 'worth', 'net', 'report', 'industry', 'put',
                       'contract', 'must', 'yearend', 'day', 'major', 'real', 'since',
                       'let', 'need', 'record', 'good', 'country', 'america', 'period',
                       'average', 'increased', 'home', 'run', 'way',
                       'world', 'second', 'four', 'product', 'largest', 'certain',
                       'financial', 'three',
                       'ago', 'almost', 'american',
                       'amount', 'area', 'b',
                       'believe', 'better', 'blue', 'board',
                       'book', 'borrower', 'brown', 'buffett', 'buyer',
                       'c', 'capacity', 'case', 'cash', 'casualty', 'cat', 'category',
                       'change', 'charge', 'chip', 'charlie',
                       'committee', 'common', 'controlled', 'corporate',
                       'corporation', 'could', 'coupon', 'customer', 'debt',
                       'director',
                       'every',
                       'fee', 'figure', 'find', 'five', 'float',
                       'fund', 'g',
                       'general', 'get', 'give', 'goodwill',
                       'h', 'hathaway', 'helper', 'high', 'holding',
                       'hour', 'housing', 'huge', 'important', 'inc', 'income',
                       'intrinsic',
                       'investor', 'k', 'know', 'le', 'like', 'line', 'look',
                       'low', 'management', 'medium', 'merger',
                       'mr', 'much', 'name', 'national', 'never',
                       'new', 'news', 'non', 'number', 'often', 'operating',
                       'others', 'owned', 'owner', 'ownership', 'page', 'paid',
                       'paper', 'past', 'pay', 'payment', 'people',
                       'preferred', 'premium',
                       'purchase', 'question',
                       'rather', 'ratio',
                       'reported', 'reserve', 'result', 'retained',
                       'rule', 'say',
                       'september', 'service', 'shoe',
                       'star', 'state', 'store', 'subsidiary', 'sunday', 'super',
                       'take', 'ten', 'th',
                       'therefore', 'utility', 'volume',
                       'zero']_____no_output_____## Create a document-term matrix with CountVectorizer
df_letters = None
for k in letters_dict:
    count_v = CountVectorizer(tokenizer=tokenize)
    letter_X = count_v.fit_transform([letters_dict[k]]).toarray()
    if df_letters is None:
        df_letters = pd.DataFrame(letter_X, columns=count_v.get_feature_names(), index=[k])
    else:
        df_letters = df_letters.append(pd.DataFrame(letter_X, columns=count_v.get_feature_names(), index=[k]))_____no_output_____df_heat = None
df_filtered = df_letters.T.query(f"index not in {frequent_words_list}")
for col in df_filtered.columns:
    col = int(col)
    x = df_filtered.sort_values(col, ascending=False)[col]
    if df_heat is None:
        df_heat = x.to_frame()
        df_heat.columns = [col]
    else:
        df_heat[col] = x_____no_output_____columns = df_frequency_heat.columns
n_cols = len(columns)
grouped_df_cols = []
for i in range(0, n_cols, 5):
    j = i + 4
    if j >= n_cols:
        j = n_cols - 1
    grouped_df_cols.append(f'{columns[i]} - {columns[j]}')
grouped_df = pd.DataFrame(columns=grouped_df_cols)_____no_output_____print(grouped_df_cols)['1977 - 1981', '1982 - 1986', '1987 - 1991', '1992 - 1996', '1997 - 2001', '2002 - 2006', '2007 - 2011', '2012 - 2016', '2017 - 2020']
for i in range(len(grouped_df_cols)):
    begin = i * 5
    end = (i + 1) * 5
    grouped_df[grouped_df_cols[i]] = df_frequency_heat.iloc[:, begin:end].sum(axis=1)_____no_output_____key_words = ['ceo', 'insurance', 'shareholder', 'gain', 'loss', 'stock', 'bond',
             'acquisition', 'portfolio', 'risk', 'sale',
             'discount', 'opportunity']
df_frequency_heat = df_heat.loc[key_words].fillna(0)_____no_output_____draw_heatmap(df_frequency_heat, cmap='ocean_r', ylabel='', xlabel='', title='Words Frequency Heatmap')_____no_output_____draw_heatmap(grouped_df, cmap='ocean_r', ylabel='', xlabel='', title='Words Frequency Heatmap grouped 5 years')_____no_output_____key_words_2 = ['hedge', 'trade', 'bitcoin', 'blockchain', 'crypto', 'forex', 'options', 'ai',
             'technology', 'esg', 'tesla', 'amazon', 'faang']
a = []
for w in key_words_2:
    try:
        df_heat.loc[w]
        a.append(w)
    except:
        print(w)
# df_frequency_heat_2 = df_heat.loc[key_words_2].fillna(0)bitcoin
blockchain
crypto
forex
options
ai
esg
tesla
faang

## More frequency analysis_____no_output_____

corpus = []
for k in letters_dict:
    corpus.append(letters_dict[k])_____no_output_____tokenized_corpus = []
for text in corpus:
    tokenized_corpus.extend(tokenize(text))_____no_output_____len(tokenized_corpus)_____no_output_____filtered_tokenized_corpus = [w for w in tokenized_corpus if w not in frequent_words_list]_____no_output_____freq_dict = get_most_frequent_combinatation(tokenized_corpus, 10, -1)_____no_output_____for k in freq_dict:
    print(k)
    print(freq_dict[k])FreqDist_onegram
[('year', 3475), ('business', 3304), ('berkshire', 3105), ('company', 2422), ('earnings', 1972), ('value', 1600), ('would', 1570), ('u', 1483), ('million', 1437), ('share', 1364)]
FreqDist_bigram
[(('last', 'year'), 770), (('per', 'share'), 373), (('book', 'value'), 304), (('pre', 'tax'), 263), (('long', 'term'), 260), (('intrinsic', 'value'), 255), (('net', 'worth'), 236), (('operating', 'earnings'), 225), (('year', 'ago'), 205), (('berkshire', 'share'), 186)]
FreqDist_trigram
[(('pre', 'tax', 'earnings'), 111), (('profit', 'le', 'zero'), 111), (('berkshire', 'hathaway', 'inc'), 103), (('le', 'zero', 'profit'), 103), (('zero', 'profit', 'le'), 103), (('nebraska', 'furniture', 'mart'), 95), (('intrinsic', 'business', 'value'), 86), (('share', 'book', 'value'), 75), (('per', 'share', 'book'), 74), (('shareholder', 'designated', 'contribution'), 66)]
FreqDist_quadgrams
[(('profit', 'le', 'zero', 'profit'), 103), (('le', 'zero', 'profit', 'le'), 103), (('zero', 'profit', 'le', 'zero'), 103), (('per', 'share', 'book', 'value'), 74), (('shareholder', 'berkshire', 'hathaway', 'inc'), 44), (('berkshire', 'hathaway', 'inc', 'shareholder'), 39), (('hathaway', 'inc', 'shareholder', 'berkshire'), 39), (('inc', 'shareholder', 'berkshire', 'hathaway'), 39), (('p', 'year', 'index', 'showed'), 38), (('year', 'since', 'present', 'management'), 34)]
freq_dict_filtered = get_most_frequent_combinatation(filtered_tokenized_corpus, 20, -1)_____no_output_____for k in freq_dict_filtered:
    print(k)
    print(freq_dict_filtered[k])FreqDist_onegram
[('earnings', 1972), ('insurance', 1250), ('stock', 1091), ('shareholder', 1064), ('loss', 878), ('gain', 758), ('profit', 667), ('geico', 602), ('rate', 600), ('term', 509)]
FreqDist_bigram
[(('balance', 'sheet'), 158), (('underwriting', 'profit'), 141), (('scott', 'fetzer'), 121), (('coca', 'cola'), 112), (('profit', 'profit'), 106), (('furniture', 'mart'), 104), (('underwriting', 'loss'), 98), (('nebraska', 'furniture'), 95), (('earnings', 'earnings'), 87), (('short', 'term'), 79)]
FreqDist_trigram
[(('nebraska', 'furniture', 'mart'), 95), (('profit', 'profit', 'profit'), 95), (('shareholder', 'designated', 'contribution'), 67), (('designated', 'contribution', 'program'), 37), (('rate', 'compounded', 'annually'), 36), (('present', 'took', 'grown'), 32), (('grown', 'rate', 'compounded'), 31), (('took', 'grown', 'rate'), 30), (('attachment', 'proxy', 'material'), 29), (('washington', 'post', 'fargo'), 27)]
FreqDist_quadgrams
[(('profit', 'profit', 'profit', 'profit'), 87), (('shareholder', 'designated', 'contribution', 'program'), 36), (('grown', 'rate', 'compounded', 'annually'), 31), (('present', 'took', 'grown', 'rate'), 30), (('took', 'grown', 'rate', 'compounded'), 30), (('generally', 'accepted', 'accounting', 'principle'), 24), (('attachment', 'proxy', 'material', 'enclosed'), 22), (('proxy', 'material', 'enclosed', 'explains'), 22), (('material', 'enclosed', 'explains', 'obtain'), 22), (('performance', 'v', 'percentage', 'dividend'), 22)]
drawn_wordcloud(filtered_tokenized_corpus, 'wordcloud_filtered.png')_____no_output_____drawn_wordcloud(tokenized_corpus, 'wordcloud_complete.png')_____no_output_____

# Sentiment Analysis_____no_output_____

from transformers import pipeline
classifier = pipeline('sentiment-analysis')_____no_output_____

## Ngrams Sentiment Analysis_____no_output_____

df = pd.DataFrame(columns=['type', 'text', 'sentiment', 'score'])
for k in freq_dict_filtered:
    for w, _ in freq_dict_filtered[k]:
        if type(w) == tuple:
            w = ' '.join(w)
        result = classifier(w)
        new_row = {'type': k, 'text': w, 'sentiment':result[0]['label'], 'score':result[0]['score']}
        df = df.append(new_row, ignore_index=True)_____no_output_____df.query("type == 'FreqDist_onegram' and sentiment == 'POSITIVE'")_____no_output_____ngram_dict_plot = {
    'Top Positive Unigrams': get_ngram_plot_data(df, 'FreqDist_onegram', 'POSITIVE'),
    'Top Negative Unigrams': get_ngram_plot_data(df, 'FreqDist_onegram', 'NEGATIVE'),
    'Top Positive Bigrams': get_ngram_plot_data(df, 'FreqDist_bigram', 'POSITIVE'),
    'Top Negative Bigrams': get_ngram_plot_data(df, 'FreqDist_bigram', 'NEGATIVE'),
    'Top Positive Trigrams': get_ngram_plot_data(df, 'FreqDist_trigram', 'POSITIVE'),
    'Top Negative Trigrams': get_ngram_plot_data(df, 'FreqDist_trigram', 'NEGATIVE'),
    'Top Positive Quadgrams': get_ngram_plot_data(df, 'FreqDist_quadgrams', 'POSITIVE'),
    'Top Negative Quadgrams': get_ngram_plot_data(df, 'FreqDist_quadgrams', 'NEGATIVE')
}

# Plotting the ngrams analysis
fig, axs = plt.subplots(nrows=4, ncols=2, figsize=(15, 18))
i, j = 0, 0
colors = ['Blues_d', 'Reds_d']
for title, ngram_data in ngram_dict_plot.items():
    ax = axs[i, j]
    sns.barplot(x='score', y='text', data=ngram_data, ax=ax, palette=colors[j])

    # Customizing plots
    format_spines(ax, right_border=False)
    ax.set_title(title, size=14)
    ax.set_ylabel('')
    ax.set_xlabel('')

    # Incrementing the index
    j += 1
    if j == 2:
        j = 0
        i += 1
plt.tight_layout()
plt.show()_____no_output_____

## Whole text Sentiment Analysis_____no_output_____

sia_sentiment_analysis_df = get_sentiment_analysis_df(letters_dict, calculate_text_sia, tokenize_sent)_____no_output_____transformers_sentiment_analysis_df = get_sentiment_analysis_df(letters_dict, calculate_text_sentiment_transform, tokenize_sent)_____no_output_____transformers_sentiment_analysis_df_____no_output_____draw_heatmap(transformers_sentiment_analysis_df, title='Sentiment of the Letters using Transformers')_____no_output_____draw_heatmap(sia_sentiment_analysis_df.loc[['neg', 'neu', 'pos']], title='Sentiment of the Letters using SIA')_____no_output_____

## QA_____no_output_____

nlp = pipeline("question-answering")
sentence_after_regex_2008 = re.sub("[^a-z0-9A-Z\,]", " ", letters_dict[2008])_____no_output_____answer, result, start, end = get_answer_using_qa(nlp, "What led to a dysfunctional credit market?", sentence_after_regex_2008)
print(f"Answer: '{answer}', score: {result}, start: {start}, end: {end}")Answer: 'a series of life threatening problems', score: 0.5841, start: 7575, end: 7612
result = nlp(question="What produced a paralyzing fear that engulfed the country?", context=sentence_after_regex_2008)
print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")
Answer: 'the credit crisis', score: 0.4479, start: 7942, end: 7959
result = nlp(question="What generated a paralyzing fear that engulfed the country?", context=sentence_after_regex_2008)
print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")Answer: 'credit crisis', score: 0.4273, start: 7946, end: 7959
result = nlp(question="Who are the managers of MidAmerican?", context=sentence_after_regex_2008)
print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")Answer: 'Dave Sokol and Greg Abel', score: 0.9798, start: 18182, end: 18206
result = nlp(question="What are upside-down loans?", context=sentence_after_regex_2008)
print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")Answer: 'a house is worth less than its mortgage', score: 0.4057, start: 42184, end: 42223
result = nlp(question="How to Profit from a Bear Market?", context=sentence_after_regex_2008)
print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")Answer: 'millions', score: 0.0078, start: 32253, end: 32261
result = nlp(question="What to invest in when the market crashes?", context=sentence_after_regex_2008)
print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")Answer: 'derivatives transactions', score: 0.6317, start: 64828, end: 64852
result = nlp(question="What is the best stock to buy?", context=sentence_after_regex_2008)
print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")Answer: 'underpriced securities', score: 0.9317, start: 17047, end: 17069
complete_corpus = ""
for k in letters_dict:
    complete_corpus += letters_dict[k]
corpus_after_regex = re.sub("[^a-z0-9A-Z\,]", " ", complete_corpus)_____no_output_____result = nlp(question="What actions to take when a terrorist attack happens?", context=corpus_after_regex)
print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")_____no_output_____


# Notebook from prasad-kumkar/qiskit-tutorials
Path: tutorials/finance/11_time_series.ipynb

# _*Loading and Processing Stock-Market Time-Series Data*______no_output_____### Introduction

Across many problems in finance, one starts with time series. Here, we showcase how to generate pseudo-random time-series, download actual stock-market time series from a number of common providers, and how to compute time-series similarity measures._____no_output_____

%matplotlib inline
from qiskit.finance import QiskitFinanceError
from qiskit.finance.data_providers import *
import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)
import datetime
import matplotlib.pyplot as plt
from pandas.plotting import register_matplotlib_converters
register_matplotlib_converters()_____no_output_____data = RandomDataProvider(tickers=["TICKER1", "TICKER2"],
                 start = datetime.datetime(2016, 1, 1),
                 end = datetime.datetime(2016, 1, 30),
                 seed = 1)
data.run()_____no_output_____

Once the data are loaded, you can run a variety of algorithms on those to aggregate the data. Notably, you can compute the covariance matrix or a variant, which would consider alternative time-series similarity measures based on dynamic time warping (DTW). In DTW, changes that vary in speed, e.g., one stock's price following another stock's price with a small delay, can be accommodated._____no_output_____

means = data.get_mean_vector()
print("Means:")
print(means)

rho = data.get_similarity_matrix()
print("A time-series similarity measure:")
print(rho)
plt.imshow(rho)
plt.show()

cov = data.get_covariance_matrix()
print("A covariance matrix:")
print(cov)
plt.imshow(cov)
plt.show()Means:
[16.66722941 72.03026566]
A time-series similarity measure:
[[1.0000000e+00 6.2284804e-04]
 [6.2284804e-04 1.0000000e+00]]

If you wish, you can look into the underlying pseudo-random time-series using. Please note that the private class members (starting with underscore) may change in future releases of Qiskit._____no_output_____

print("The underlying evolution of stock prices:")
for (cnt, s) in enumerate(data._tickers):
    plt.plot(data._data[cnt], label=s)
plt.legend()
plt.xticks(rotation=90)
plt.show()

for (cnt, s) in enumerate(data._tickers):
    print(s)
    print(data._data[cnt])The underlying evolution of stock prices:

Clearly, you can adapt the number and names of tickers and the range of dates: _____no_output_____

data = RandomDataProvider(tickers=["CompanyA", "CompanyB", "CompanyC"],
                 start = datetime.datetime(2015, 1, 1),
                 end = datetime.datetime(2016, 1, 30),
                 seed = 1)
data.run()
for (cnt, s) in enumerate(data._tickers):
    plt.plot(data._data[cnt], label=s)
plt.legend()
plt.xticks(rotation=90)
plt.show()_____no_output_____

### Access to closing-price time-series

While the access to real-time data usually requires a payment, it is possible
to access historical (adjusted) closing prices via Wikipedia and Quandl
free of charge, following registration at:
https://www.quandl.com/?modal=register
In the code below, one needs to specify actual tickers of actual NASDAQ
issues and the access token you obtain from Quandl; by running the code below, you agree to the Quandl terms and
conditions, including a liability waiver.
Notice that at least two tickers are required for the computation
of covariance and time-series matrices, but hundreds of tickers may go
beyond the fair usage limits of Quandl._____no_output_____

try:
    stocks = ["REPLACEME1", "REPLACEME2"]
    wiki = WikipediaDataProvider(
                     token = "REPLACEME",
                     tickers = stocks,
                     stockmarket = StockMarket.NASDAQ,
                     start = datetime.datetime(2016,1,1),
                     end = datetime.datetime(2016,1,30))
    wiki.run()
except QiskitFinanceError as ex:
    print(ex)
    print("You need to replace REPLACEME with a valid token.")'Cannot retrieve Wikipedia data.'
You need to replace REPLACEME with a valid token.

Once the data are loaded, you can again compute the covariance matrix or its DTW variants._____no_output_____

if wiki._data:
    if wiki._n <= 1:
        print("Not enough wiki data to plot covariance or time-series similarity. Please use at least two tickers.")
    else:
        rho = wiki.get_similarity_matrix()
        print("A time-series similarity measure:")
        print(rho)
        plt.imshow(rho)
        plt.show()

        cov = wiki.get_covariance_matrix()
        print("A covariance matrix:")
        print(cov)
        plt.imshow(cov)
        plt.show()
else:
    print('No wiki data loaded.')No wiki data loaded.

If you wish, you can look into the underlying time-series using:_____no_output_____

if wiki._data:
    print("The underlying evolution of stock prices:")
    for (cnt, s) in enumerate(stocks):
        plt.plot(wiki._data[cnt], label=s)
    plt.legend()
    plt.xticks(rotation=90)
    plt.show()

    for (cnt, s) in enumerate(stocks):
        print(s)
        print(wiki._data[cnt])
else:
    print('No wiki data loaded.')No wiki data loaded.

### [Optional] Setup token to access recent, fine-grained time-series

If you would like to download professional data, you will have to set-up a token with one of the major providers. Let us now illustrate the data with NASDAQ Data on Demand, which can supply bid and ask prices in arbitrary resolution, as well as aggregates such as daily adjusted closing prices, for NASDAQ and NYSE issues.
_____no_output_____If you don't have NASDAQ Data on Demand license, you can contact NASDAQ (cf. https://business.nasdaq.com/intel/GIS/Nasdaq-Data-on-Demand.html) to obtain a trial or paid license.

If and when you have access to NASDAQ Data on Demand using your own token, you should replace REPLACE-ME below with the token.
To assure the security of the connection, you should also have your own means of validating NASDAQ's certificates. The DataOnDemandProvider constructor has an optional argument `verify`, which can be `None` or a string or a boolean. If it is `None`, certify certificates will be used (default). If verify is a string, it should be pointing to a certificate for the HTTPS connection to NASDAQ (dataondemand.nasdaq.com), either in the form of a CA_BUNDLE file or a directory wherein to look.
_____no_output_____

from qiskit.finance.data_providers.data_on_demand_provider import StockMarket

token = "REPLACE-ME"

if token != "REPLACE-ME":
    try:
        nasdaq = DataOnDemandProvider(token = token,
                     tickers = stocks,
                     stockmarket = StockMarket.NASDAQ,
                     start = datetime.datetime(2016,1,1),
                     end = datetime.datetime(2016,1,2))
        nasdaq.run()
        nasdaq.plot()
    except QiskitFinanceError as e:
        print(e)
        print("Error retrieving data.")_____no_output_____

Another major vendor of stock market data is Exchange Data International (EDI), whose API can be used to query over 100 emerging and frontier markets that are Africa, Asia, Far East, Latin America and Middle East, as well as the more established ones. See:
https://www.exchange-data.com/pricing-data/adjusted-prices.php#exchange-coverage
for an overview of the coverage.

The access again requires a valid access token to replace REPLACE-ME below. The token can be obtained on a trial or paid-for basis at:
https://www.quandl.com/

In the following example, you need to replace TICKER1 and TICKER2 with valid tickers at the London Stock Exchange. _____no_output_____

try:
    lse = ExchangeDataProvider(token = "REPLACE-ME",
                 tickers = ["TICKER1", "TICKER2"],
                 stockmarket = StockMarket.LONDON,
                 start = datetime.datetime(2019,1,1),
                 end = datetime.datetime(2019,1,30))
    lse.run()
    lse.plot()
except QiskitFinanceError as e:
    print(e)
    print("You need to replace REPLACE-ME with a valid token.")'Cannot retrieve Exchange Data data.'
You need to replace REPLACE-ME with a valid token.

For the actual use of the data, please see the portfolio_optimization or portfolio_diversification notebooks. _____no_output_____

import qiskit.tools.jupyter
%qiskit_version_table
%qiskit_copyright_____no_output_____


# Notebook from mojoyinkimberly/Gender_Classification
Path: GenderClassification.ipynb

_____no_output_____**importing libraries**_____no_output_____

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from statsmodels.stats.outliers_influence import variance_inflation_factor
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_absolute_error,mean_squared_error,r2_score
from sklearn.decomposition import PCA
from sklearn.metrics import confusion_matrix, accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedKFold_____no_output_____

 **reading dataset into the python environment**_____no_output_____

gender_dataset = pd.read_csv("/content/gender_classification_v7.csv",sep = ",")
gender_dataset_____no_output_____

**A bar chart showing the relationship between gender and long hair.**_____no_output_____

sns.barplot(x = "long_hair", y = "gender", color='salmon', data=gender_dataset)_____no_output_____

**A line graph showing the variations between the width and height of the both genders' foreheads**_____no_output_____

sns.lineplot(x = "forehead_width_cm", y = "forehead_height_cm", hue = "gender", data = gender_dataset)_____no_output_____

**A bar graph showing the variations between the length and width of both genders' noses**_____no_output_____

sns.barplot(x = "nose_wide", y = "nose_long", hue = "gender", data = gender_dataset)_____no_output_____

**A bar chart showing the thinness of both genders' lips**_____no_output_____

sns.barplot(x = "lips_thin",y = "gender", color = "violet",data = gender_dataset)_____no_output_____

**A bar chart showing the distance from nose to lip of both genders**_____no_output_____

sns.barplot(x="distance_nose_to_lip_long",y = "gender", color = 'purple', data = gender_dataset)_____no_output_____

 **Encoding gender column into 1 and 0 for easy model classification**_____no_output_____

lst = []

for gender in range(len(gender_dataset['gender'])):
  if gender_dataset['gender'][gender] == 'Male':
    lst.append(1)
  else:
    lst.append(0)

gender_dataset['gender1'] = lst_____no_output_____

**dropping unnecessary column**_____no_output_____

new_gender_dataset=gender_dataset.drop(columns = ['gender'])_____no_output_____

**checking for missing values**_____no_output_____

new_gender_dataset.isna().sum()_____no_output_____

**checking for outliers**_____no_output_____

new_gender_dataset.describe()_____no_output_____

**checking the relationships between variables**_____no_output_____

plt.figure(figsize=(15,10))
sns.heatmap(new_gender_dataset.corr(), annot=True)_____no_output_____

**splitting variables into dependent and independent variables**_____no_output_____

y = new_gender_dataset['gender1']
x = new_gender_dataset.drop(columns= ['gender1'])_____no_output_____

**checking for multicolinearity using variance inflation factor**_____no_output_____

def calcu_mult(df):
    vif = pd.DataFrame()
    vif["variables"] = x.columns
    vif["VIF"] = [variance_inflation_factor(x.values, i) for i in range(x.shape[1])]

    return vif_____no_output_____calcu_mult(new_gender_dataset)_____no_output_____

**splitting dataset into train and test**_____no_output_____

X_train,X_test,y_train,y_test = train_test_split(x,y,test_size =.20, random_state = 45)_____no_output__________no_output_____

**normalizing the dataset**_____no_output_____

normalizer = MinMaxScaler()
norm_X = pd.DataFrame(normalizer.fit_transform(X_train))_____no_output_____

 **splitting the normalized dataset for validation**_____no_output_____

X_train_, X_val_test, y_train_, y_val_test = train_test_split(norm_X, y_train, test_size=0.25, random_state=1)_____no_output_____

**using the principal component analysis model to check for important features**_____no_output_____

pca = PCA(n_components = 7)
pca_model = pca.fit(X_train_)
PCA(n_components = 7)_____no_output_____features = new_gender_dataset.columns
importances = pca_model.explained_variance_ratio_
indices = np.argsort(importances)[-7:]
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), [features[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()_____no_output_____

**fitting our training dataset into the logistic regression model**_____no_output_____

lg_model = LogisticRegression()
lg_result = lg_model.fit(X_train_, y_train_)_____no_output_____

**predicting with our test**_____no_output_____

y_predict = lg_result.predict(X_val_test)_____no_output_____

**using the confusion matrix to get the four quadrants that show the true and false positive and negative classes**_____no_output_____

print("cm : ",confusion_matrix(y_val_test, y_predict))cm :  [[485  17]
 [ 24 474]]

**checking for accuracy score**_____no_output_____

print("ac_score : ",accuracy_score(y_val_test, y_predict))ac_score :  0.959

**using the repeated k fold model to cross validate**_____no_output_____

Cv_ = RepeatedKFold(n_splits=10, n_repeats=4, random_state=1)
score = cross_val_score(lg_result, X_val_test, y_val_test, scoring = "accuracy", cv = Cv_, n_jobs=1, error_score="raise")
np.mean(score)_____no_output_____

                                                            END OF NOTEBOOK_____no_output_____


# Notebook from kcsph/summarizer_app
Path: summarizer.ipynb

## 0.0 Import modules_____no_output_____

## NLP modules
import gensim
from gensim.models.doc2vec import Doc2Vec
import nltk
import textblob
from textblob import TextBlob
from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS
from nltk.corpus import stopwords ##Note you'll need to download NLTK and corpuses
from spacy.en import English ##Note you'll need to install Spacy and download its dependencies
parser = English()

## Other Python modules
import itertools
from operator import itemgetter
import re
import string
import numpy as np
import pandas as pd
import matplotlib
from IPython.display import IFrame

# Graph module
import networkx as nx

## Machine learning & text vectorizer modules
from sklearn.decomposition import TruncatedSVD
from sklearn.cluster import KMeans
from sklearn.metrics import pairwise_distances
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer_____no_output_____

## 1.0 Functions_____no_output_____### 1.1 Text pre-processing functions_____no_output_____

# A custom function to clean the text before sending it into the vectorizer
def cleanText(text):
    # get rid of newlines
    text = text.strip().replace("\n", " ").replace("\r", " ")

    # replace twitter @mentions
    mentionFinder = re.compile(r"@[a-z0-9_]{1,15}", re.IGNORECASE)
    text = mentionFinder.sub("@MENTION", text)
    text = re.sub('[^a-zA-Z ]','',text)
    # replace HTML symbols
    text = text.replace("&", "and").replace(">", ">").replace("<", "<")

    # lowercase
    text = text.lower()
#     text = str(TextBlob(text).correct())
    return text

# A custom function to tokenize the text using spaCy
# and convert to lemmas
def tokenizeText(sample):
    # get the tokens using spaCy
    tokens = parser(sample)

    # lemmatize
    lemmas = []
    for tok in tokens:
        lemmas.append(tok.lemma_.lower().strip() if tok.lemma_ != "-PRON-" else tok.lower_)
    tokens = lemmas

    # stoplist the tokens
    tokens = [tok for tok in tokens if tok not in STOPLIST]

    # stoplist symbols
    tokens = [tok for tok in tokens if tok not in SYMBOLS]

    # remove large strings of whitespace
    while "" in tokens:
        tokens.remove("")
    while " " in tokens:
        tokens.remove(" ")
    while "\n" in tokens:
        tokens.remove("\n")
    while "\n\n" in tokens:
        tokens.remove("\n\n")

    return tokens

# A custom stoplist
STOPLIST = set(stopwords.words('english') + ["n't", "'s", "'m", "ca"] + list(ENGLISH_STOP_WORDS))
# List of symbols we don't care about
SYMBOLS = " ".join(string.punctuation).split(" ") + ["-----", "---", "...", "“", "”", "'ve"]

## Tokenizer specific for Doc2Vec where each word is important, so stop words are not removed.
def doc2vec_tokenizeText(corpus):
    punctuation = """.,?!:;(){}[]"""
    corpus = [z.lower().replace('\n','') for z in corpus]
    corpus = [z.replace('
', ' ') for z in corpus]

    #treat punctuation as individual words
    for c in punctuation:
        corpus = [z.replace(c, ' %s '%c) for z in corpus]
    corpus = [z.split() for z in corpus]
    return corpus_____no_output_____

### 1.2 Similarity measure functions_____no_output_____

def cos_sim(text1,text2):
    tfvectorizer = TfidfVectorizer(tokenizer=tokenizeText)
    arrays = tfvectorizer.fit_transform([text1,text2]).A
    num = (arrays[0]*arrays[1]).sum()
    denom1 = np.sqrt((arrays[0]**2).sum())
    denom2 = np.sqrt((arrays[1]**2).sum())
    return num/(denom1*denom2)

def similarity(string1,string2):
    w1 = tokenizeText(cleanText(string1))
    w2 = tokenizeText(cleanText(string2))
    score = 0
    for w in w1:
        if w in w2:
            score += 1
        else:
            continue
    for w in w2:
        if w in w1:
            score += 1
        else:
            continue
    return score/(len(w1)+len(w2))

def lDistance(firstString, secondString):
    "Function to find the Levenshtein distance between two words/sentences - gotten from http://rosettacode.org/wiki/Levenshtein_distance#Python"
    if len(firstString) > len(secondString):
        firstString, secondString = secondString, firstString
    distances = range(len(firstString) + 1)
    for index2, char2 in enumerate(secondString):
        newDistances = [index2 + 1]
        for index1, char1 in enumerate(firstString):
            if char1 == char2:
                newDistances.append(distances[index1])
            else:
                newDistances.append(1 + min((distances[index1], distances[index1+1], newDistances[-1])))
        distances = newDistances
    return 1/(distances[-1]+1)
_____no_output_____

### 1.3 Graph building functions_____no_output_____

def buildGraph(nodes,weight):
    "nodes - list of hashables that represents the nodes of the graph"
    gr = nx.Graph() #initialize an undirected graph
    gr.add_nodes_from(nodes)
    nodePairs = list(itertools.combinations(nodes, 2))

    #add edges to the graph (weighted by Levenshtein distance)
    for pair in nodePairs:
        firstString = pair[0]
        secondString = pair[1]
        if weight == 'cosine':
            try:
                edge_weight = cos_sim(firstString, secondString)
            except:
                edge_weight = 0
        if weight == 'similarity':
            try:
                edge_weight = similarity(firstString, secondString)
            except:
                edge_weight = 0
        if weight == 'ldistance':
            try:
                edge_weight = lDistance(firstString, secondString)
            except:
                edge_weight = 0
            gr.add_edge(firstString, secondString, weight=edge_weight)

    return gr
_____no_output_____

### 1.4 Summarizer functions_____no_output_____

def draw_graph(text,weight='cosine'):
    sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')
    sentenceTokens = sent_detector.tokenize(text.strip())
    graph = buildGraph(sentenceTokens,weight=weight)
    nx.draw_networkx(graph)

def textrank_summarizer(text,weight='cosine',num_sen = 5, pos_score = False):
    sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')
    sentenceTokens = sent_detector.tokenize(text.strip())
    graph = buildGraph(sentenceTokens,weight=weight)

    calculated_page_rank = nx.pagerank(graph, weight='weight')

    #Create position scorelist
    #Scores are caluclated such that page rank score is increased by 10% if its the first or last sentence
    #Sentences in the middle of the document are not given increased scores
    pos = np.array(list(range(len(sentenceTokens))))
#     score_array = np.array(1+(abs((pos+0.5) - len(pos)/2)/max((pos+0.5) - len(pos)/2)/10))
#     score_dict = {}
#     for i in range(len(sentenceTokens)):
#         score_dict[sentenceTokens[i]] = score_array[i]

#     #Adjusts page rank score for position
#     score_adj_page_rank = {k : v * score_dict[k] for k, v in calculated_page_rank.items() if k in score_dict}

    #most important sentences in ascending order of importance
#     if pos_score == True:
#         sentences = sorted(score_adj_page_rank, key=score_adj_page_rank.get,reverse=True)
#     else:
#         sentences = sorted(calculated_page_rank, key=calculated_page_rank.get,reverse=True)

    #return a word summary
    pos_dict = {}
    for i in range(len(sentenceTokens)):
        pos_dict[sentenceTokens[i]] = pos[i]

    combined = {k : [v, pos_dict[k]] for k, v in calculated_page_rank.items() if k in pos_dict}

    listlist = []
    for k, v in combined.items():
        listlist.append((k,v[0],v[1]))

    listlist.sort(key=lambda x: x[1],reverse=True)

    summarysentences = listlist[0:num_sen]

    summarysentences.sort(key=lambda x: x[2],reverse=False)

    summary = ""
    for n in range(num_sen):
        summary += ' ' + summarysentences[n][0]
        summary = " ".join(summary.replace(u"\xa0", u" ").strip().split())

    return summary

def doc2vec_summarizer(text,num_sen=5):

    sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')
    sentenceTokens = sent_detector.tokenize(text.strip())

    LabeledSentence = gensim.models.doc2vec.LabeledSentence

    sentences = doc2vec_tokenizeText(sentenceTokens)

    def labelizeReviews(reviews, label_type):
        labelized = []
        for i,v in enumerate(reviews):
            label = '%s_%s'%(label_type,i)
            labelized.append(LabeledSentence(v, [label]))
        return labelized

    sentences = labelizeReviews(sentences,'train')

    model = Doc2Vec(min_count=1, window=10, size=500, sample=1e-4, workers=8)
    model.build_vocab(sentences)
    for epoch in range(1000):
        model.train(sentences)

    docvec = []
    for i in range(len(model.docvecs)):
        docvec.append(model.docvecs[i])

    kmeans = KMeans(n_clusters=1)

    kmeans.fit(docvec)

    distance = pairwise_distances(kmeans.cluster_centers_, docvec)

    pos = np.array(list(range(len(sentenceTokens))))

    listlist = [list(x) for x in zip(sentenceTokens,distance.tolist()[0],pos)]

    listlist.sort(key=lambda x: x[1],reverse=False)

    ## Sort by sentence order
    summarysentences = listlist[0:num_sen]

    summarysentences.sort(key=lambda x: x[2],reverse=False)

    summary = ""
    for n in range(num_sen):
        summary += ' ' + summarysentences[n][0]
        summary = " ".join(summary.replace(u"\xa0", u" ").strip().split())

    return summary

def lsa_summarizer(text,num_sen=5):
    sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')
    sentenceTokens = sent_detector.tokenize(text.strip())

    tfvectorizer = TfidfVectorizer(tokenizer=tokenizeText)
    sparse = tfvectorizer.fit_transform(sentenceTokens).A
    lsa = TruncatedSVD(n_components=1)
    concept = lsa.fit_transform(sparse)

    pos = np.array(list(range(len(sentenceTokens))))

    listlist = [list(x) for x in zip(sentenceTokens,concept,pos)]

    listlist.sort(key=lambda x: x[1],reverse=True)

    summarysentences = listlist[0:num_sen]

    summarysentences.sort(key=lambda x: x[2],reverse=False)

    summary = ""
    for n in range(num_sen):
        summary += ' ' + summarysentences[n][0]
        summary = " ".join(summary.replace(u"\xa0", u" ").strip().split())

    return summary_____no_output_____

## 2.0 Sample runs_____no_output_____Sample Text_____no_output_____

text3 = 'A reader who is not familiar with the subject might reckon that it is the study of the economy. After all, chemistry is the study of chemicals, biology is the study of living things, and sociology is the study of society, so economics must be the study of the economy. But according to some of the most popular economics books of our time, economics is much more than that. According to them, economics is about the Ultimate Question – of ‘Life, the Universe and Everything’ – as in The Hitchhiker’s Guide to the Galaxy, the cult comedy science fiction by Douglas Adams, which was made into a movie in 2005, with Martin ‘The Hobbit’ Freeman in the leading role. According to Tim Harford, the Financial Times journalist and the author of the successful book The Undercover Economist, economics is about Life – he has named his second book The Logic of Life. No economist has yet claimed that economics can explain the Universe. The Universe remains, for now, the turf of physicists, whom most economists have for centuries been looking up to as their role models, in their desire to make their subject a true science.*1 But some economists have come close – they have claimed that economics is about ‘the world’. For example, the subtitle of the second volume in Robert Frank’s popular Economic Naturalist series is How Economics Helps You Make Sense of Your World. Then there is the Everything bit. The subtitle of Logic of Life is Uncovering the New Economics of Everything. According to its subtitle, Freakonomics by Steven Levitt and Stephen Dubner – probably the best-known economics book of our time – is an exploration of the Hidden Side of Everything. Robert Frank agrees, even though he is far more modest in his claim. In the subtitle of his first Economic Naturalist book, he only said Why Economics Explains Almost Everything (emphasis added). So, there we go. Economics is (almost) about Life, the Universe and Everything.*2 When you think about it, this is some claim coming from a subject that has spectacularly failed in what most non-economists think is its main job – that is, explaining the economy. In the run-up to the 2008 financial crisis, the majority of the economics profession was preaching to the world that markets are rarely wrong and that modern economics has found ways to iron out those few wrinkles that markets may have; Robert Lucas, the 1995 winner of the Nobel Prize in Economics,*3 had declared in 2003 that the ‘problem of depression prevention has been solved’.1 So most economists were caught completely by surprise by the 2008 global financial crisis.*4 Not only that, they have not been able to come up with decent solutions to the ongoing aftermaths of that crisis. Given all this, economics seems to suffer from a serious case of megalomania – how can a subject that cannot even manage to explain its own area very well claim to explain (almost) everything? These titles are hyped up. But the point is that they are hyped up in a particular way. The hypes could have been something along the line of ‘how economics explains everything about the economy’, but they are instead along the lines of ‘how economics can explain not just the economy but everything else as well’. The hypes are of this particular variety because of the way in which the currently dominant school of economics, that is, the so-called Neoclassical school, defines economics. The standard Neoclassical definition of economics, the variants of which are still used, is given in the 1932 book by Lionel Robbins, An Essay on the Nature and Significance of Economic Science. In the book, Robbins defined economics as ‘the science which studies human behaviour as a relationship between ends and scarce means which have alternative uses’. In this view, economics is defined by its theoretical approach, rather than its subject matter. Economics is a study of rational choice, that is, choice made on the basis of deliberate, systematic calculation of the maximum extent to which the ends can be met by using the inevitably scarce means. The subject matter of the calculation can be anything – marriage, having children, crime or drug addiction, as Gary Becker, the famous Chicago economist and the winner of 1992 Nobel Prize in Economics, has written about – and not just ‘economic’ issues, as non-economists would define them, such as jobs, money or international trade. When Becker titled his 1976 book The Economic Approach to Human Behaviour, he was really declaring without the hype that economics is about everything. This trend of applying the so-called economic approach to everything, called by its critics ‘economics imperialism’, has reached its apex recently in books like Freakonomics. Little of Freakonomics is actually about economic issues as most people would define them. It talks about Japanese sumo wrestlers, American schoolteachers, Chicago drug gangs, participants in the TV quiz show The Weakest Link, real estate agents and the Ku Klux Klan. Most people would think (and the authors also admit) that none of these people, except real estate agents and drug gangs, have anything to do with economics. But, from the point of view of most economists today, how Japanese sumo wrestlers collude to help each other out or how American schoolteachers fabricate their pupils’ marks to get better job assessments are as legitimate subjects of economics as whether Greece should stay in the Eurozone, how Samsung and Apple fight it out in the smartphone market or how we can reduce youth unemployment in Spain (which is over 55 per cent at the time of writing). To those economists, those ‘economic’ issues do not have privileged status in economics, they are just some of many things (oh, I forgot, some of everything) that economics can explain, because they define their subject in terms of its theoretical approach, rather than its subject matter. The most intuitive answer to most readers may be that the economy is anything to do with money – not having it, earning it, spending it, running out of it, saving it, borrowing it and repaying it. This is not quite right, but it is a good starting point for thinking about the economy – and economics. Now, when we talk of the economy being about money, we are not really talking about physical money. Physical money – be it a banknote, a gold coin or the huge, virtually immovable stones that were used as money in some Pacific islands – is only a symbol. Money is a symbol of what others in your society owe you, or your claim on particular amounts of the society’s resources.2 How money and other financial claims – such as company shares, derivatives and many complex financial products, which I will explain in later chapters – are created, sold and bought is one huge area of economics, called financial economics. These days, given the dominance of the financial industry in many countries, a lot of people equate economics with financial economics, but it is actually only a small part of economics. Your money – or the claims you have over resources – may be generated in a number of different ways. And a lot of economics is (or should be) about those. The most common way to get money – unless you have been born into it – is to have a job (including being your own boss) and earn money from it. So, a lot of economics is about jobs. We can reflect on jobs from different perspectives. Jobs can be understood from the point of view of the individual worker. Whether you get a job and how much you are paid for it depends on the skills you have and how many demands there are for them. You may get very high wages because you have very rare skills, like Cristiano Ronaldo, the football player. You may lose your job (or become unemployed) because someone invents a machine that can do what you do 100 times faster – as happened to Mr Bucket, Charlie’s father, a toothpaste cap-screwer, in the 2005 movie version of Roald Dahl’s Charlie and the Chocolate Factory.*5 Or you have to accept lower wages or worse working conditions because your company is losing money thanks to cheaper imports from, say, China. And so on. So, in order to understand jobs even at the individual level, we need to know about skills, technological innovation and international trade. Wages and working conditions are also deeply affected by ‘political’ decisions to change the very scope and the characteristics of the labour market (I have put ‘political’ in quotation marks, as in the end the boundary between economics and politics is blurry, but that is a topic for later – see Chapter 11). The accession of the Eastern European countries to the European Union has had huge impacts on the wages and behaviours of Western European workers, by suddenly expanding the supply of workers in their labour markets. The restriction on child labour in the late nineteenth century and early twentieth centuries had the opposite effect of shrinking the boundary of the labour market – suddenly a large proportion of the potential employees were shut out of the labour market. Regulations on working hours, working conditions and minimum wages are examples of less dramatic ‘political’ decisions that affect our jobs. In addition to holding down a job, you can get money through transfers – that is, by simply being given it. This can be either in the form of cash or ‘in kind’, that is, direct provision of particular goods (e.g., food) or services (e.g., primary education). Whether in cash or in kind, these transfers can be made in a number of different ways. There are transfers made by ‘people you know’. Examples include parental support for children, people taking care of elderly family members, gifts from local community members, say, for your daughter’s wedding. Then there is charitable giving, that is, transfer voluntarily made to strangers. People – sometimes individually sometimes collectively (e.g., through corporations or voluntary associations) – give to charities that help others. In terms of its quantity, charitable giving is overshadowed in many multiples by transfers made through governments, which tax some people to subsidize others. So a lot of economics is naturally about these things – or the areas of economics known as public economics. Even in very poor countries, there are some government schemes to give cash or goods in kind (e.g., free grains) to those who are in the worst positions (e.g., the aged, the disabled, the starving). But the richer societies, especially those in Europe, have transfer schemes that are much more comprehensive in scope and generous in amounts. This is known as the welfare state and is based on progressive taxation (those who earn more paying proportionally larger shares of their incomes in taxes) and universal benefits (where everyone, not just the poorest or the disabled, is entitled to a minimum income and to basic services, such as health care and education). Once you gain access to resources, whether through jobs or transfers, you consume them. As physical beings, we need to consume some minimum amount of food, clothes, energy, housing, and other goods to fulfil our basic needs. And then we consume other goods for ‘higher’ mental wants – books, musical instruments, exercise equipment, TV, computers and so on. We also buy and consume services – a bus ride, a haircut, a dinner at a restaurant or even a holiday abroad.3 So a lot of economics is devoted to the study of consumption – how people allocate money between different types of goods and services, how they make choices between competing varieties of the same product, how they are manipulated and/or informed by advertisements, how companies spend money to build their ‘brand images’ and so on. In order to be consumed, these goods and services have to be produced in the first place – goods in farms and factories and services in offices and shops. This is the realm of production – an area of economics that has been rather neglected since the Neoclassical school, which puts emphasis on exchange and consumption, became dominant in the 1960s. In standard economics textbooks, production appears as a ‘black box’, in which somehow quantities of labour (work by humans) and capital (machines and tools) are combined to produce the goods and services. There is little recognition that production is a lot more than combining some abstract quanta called labour and capital and involves getting many ‘nitty-gritty’ things right. And these are things that most readers may not normally have associated with economics, despite their crucial importance for the economy: how the factory is physically organized, how to control the workers or deal with trade unions, how to systematically improve the technologies used through research. Most economists are very happy to leave the study of these things to ‘other people’ – engineers and business managers. But, when you think about it, production is the ultimate foundation of any economy. Indeed, the changes in the sphere of production usually have been the most powerful sources of social change. Our modern world has been made by the series of changes in technologies and institutions relating to the sphere of production that have been made since the Industrial Revolution. The economics profession, and the rest of us whose views of the economy are informed by it, need to pay far more attention to production than currently. Defining economics in this way makes this book unlike most other economics books in one fundamental way. As they define economics in terms of its methodology, most economics books assume that there is only one right way of ‘doing economics’ – that is, the Neoclassical approach. The worst examples won’t even tell you that there are other schools of economics than the Neoclassical one. By defining economics in terms of the subject matter, this book highlights the fact that there are many different ways of doing economics, each with its emphases, blind spots, strengths and weaknesses. After all, what we want from economics is the best possible explanation of various economic phenomena rather than a constant ‘proof’ that a particular economic theory can explain not just the economy but everything.'_____no_output_____text2 = '\n\nBRUSSELS – The “Brexiteers” – those who want Britain to leave the European Union – argue that their goal would be virtually cost-free and have no effect on the United Kingdom’s global trade. They are wrong. On June 23, when voters in Britain cast their ballots in the referendum on the question, they need to consider what is actually involved in leaving the EU – and how the free-trade benefits they now enjoy (and take for granted) could be maintained after Brexit. Start with the basics. Leaving the EU means that the UK would exit the EU’s Customs Union, which is the basis for cross-border free trade among the EU’s 28 members (and establishes a common external tariff vis-à-vis third countries). It also means exit from the Single Market – the basis for the free movement of goods and services among EU members. By definition, non-members of the EU cannot belong to the Single Market.\n\nWill Brexit Destroy Britain and Europe?\n\nPhilippe Legrain weighs the views of Joschka Fischer, Richard Haass, Joseph Nye, and others on what Britain’s withdrawal from the EU would mean for both sides.\n\nSo what would happen next? During the two-year period before Britain’s withdrawal takes final effect, there would be UK-EU negotiations on many points – sovereignty, the legal order, immigration, finances, and economic matters. The assumption is that a crucial goal for Britain would be to negotiate a trading relationship as close as possible to the free-trade relationships that exist today.That is easier said than done. The best result would be if all players agreed to maintain the free trade already achieved, with the UK setting a new external tariff on a duty-free basis, applicable to all comers. This is what happened in the 1970s after Britain and Denmark left the European Free Trade Association: Free-trade agreements were negotiated among EFTA members and between them and the EU (or the EEC as it was then known). But Brexiteers should realize that there is no guarantee that this would happen again – and, in any case, there would be complications. While this solution would be good for the 45% of British exports that are sold in EU markets, it would reduce protection for British industries to zero. Under the rules of the World Trade Organization, the same import duties must be applied to all WTO participants – which means that if Britain’s imports from the EU are duty-free, its imports from the rest of the world must be, too. The alternative would be for UK exporters to accept the EU’s common external tariff, and for the UK to create its own import tariff, applied to all imports, including from the EU. Because the common tariff is at a relatively low level on industrial and fishery products, this might not be an insuperable barrier for UK exports, and it would allow some flexibility in protecting UK companies from imports. The potential pitfall is that any British tariff increase above the EU level would expose the UK to claims for compensation from third countries in the WTO. The bigger question that the Brexiteers need to answer is how to secure a high level of access to the EU’s internal market. This is vitally important for Britain’s service industries, particularly for the City of London’s exports of financial services. There is only one precedent for non-EU members being able to negotiate access to the internal market equivalent to that enjoyed by EU members. This is the European Economic Area agreement concluded with the EU by Norway, Iceland, and Liechtenstein in 1992.In the view of many observers, including me, access to the Single Market through the EEA is no longer available. But what if we’re wrong? The point is that such a deal would go against all the instincts (and rhetoric) of the Brexiteers, because it would mean accepting the EU’s “four freedoms”: not just the free movement of goods, services, and capital – but of people, too. That would be hard to square with the Brexit objective of “controlling our borders.” The Brexiteers would also blanch at Britain’s obligation under an EEA-type agreement to continue contributing to the EU budget.Of course, there could be specific arrangements for particular sectors. But it seems unlikely that such arrangements would be possible in financial services and the major professional services (including doctors, architects, and lawyers), which are important for Britain’s competitors in Europe. Indeed, it is possible that the EU would adopt a tough mercantilist stance: If you want privileged access, you should stay in the club. The final consequence of Brexit is that the UK would lose its free-trade arrangements with third countries under the many trade agreements that the EU has signed since 2000. Replacing these agreements with bilateral deals would take time. There is no guarantee that the EU would agree to an interim continuation of free trade, and it seems certain that UK exports would face higher tariffs than its former EU partners in those third countries (placing British exporters at a competitive disadvantage).\n\nIt’s a big stretch for the Brexiteers to ask us to believe that the world’s most prominent leaders, European or not, have all got the impact of Brexit wrong. In the trade debate, we have had the International Monetary Fund, US President Barack Obama, and the OECD – quite apart from the UK Treasury – telling Britons that a vote to leave would be bad for the economy. It is not much of a reply to argue that the OECD is “in the pay of the EU” or that Obama is anti-British because he had a Kenyan father in British colonial days.Economic forecasting is an uncertain science. But when almost all projections point in the same direction – that Brexit would be hugely damaging to the UK – it is time to decide what is credible and what is not.\n\n'_____no_output_____textrank_summarizer(text3,weight='cosine',num_sen = 20)_____no_output_____lsa_summarizer(text3,num_sen=25)_____no_output_____doc2vec_summarizer(text3,num_sen=15)_____no_output_____from gensim import summarization_____no_output_____summarization.summarize(text3,ratio=0.1).replace('\n',' ')_____no_output_____## Still needs help in addressing unique characters. Latex has trouble interpreting certain special characters_____no_output_____from pylatex import Document, Section, Subsection, Tabular, Math, TikZ, Axis, Figure, Package, Command
from pylatex.utils import italic, NoEscape
import os

def print_article(article_num):
    doc = Document()
    doc.generate_tex()
    summary = lsa_summarizer(ps_df.loc[article_num,'articles'])

    doc.preamble.append(Command('title',ps_df.loc[article_num,'titles']))
    doc.preamble.append(Command('author', ps_df.loc[article_num,'author']))
    doc.preamble.append(Command('date',ps_df.loc[article_num,'date_published'].title()))
    doc.append(NoEscape(r'\maketitle'))

    with doc.create(Section('Summary')):
        doc.append(summary)

    with doc.create(Section('Article')):
        doc.append(ps_df.loc[article_num,'articles'].replace('\n','\n\n'))

    doc.generate_pdf(ps_df.loc[article_num,'titles'],compiler='pdflatex')_____no_output_____

## 3.0 Generate PDFs of Project Syndicate Articles (or whatever articles)_____no_output_____

ps_df = pd.read_pickle('ps_df.pkl')_____no_output_____ps_df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 315 entries, 0 to 314
Data columns (total 5 columns):
titles            315 non-null object
author            315 non-null object
date_published    315 non-null object
articles          315 non-null object
topic             315 non-null object
dtypes: object(5)
memory usage: 12.4+ KB
ps_df.sample()_____no_output_____article_num = 300
print_article(article_num)
from IPython.display import IFrame
IFrame(ps_df.loc[article_num,'titles']+'.pdf', width=1000, height=500)_____no_output_____pweh = 'According to them, economics is about the Ultimate Question – of ‘Life, the Universe and Everything’ – as in The Hitchhiker’s Guide to the Galaxy, the cult comedy science fiction by Douglas Adams, which was made into a movie in 2005, with Martin ‘The Hobbit’ Freeman in the leading role. *2 When you think about it, this is some claim coming from a subject that has spectacularly failed in what most non-economists think is its main job – that is, explaining the economy. In the run-up to the 2008 financial crisis, the majority of the economics profession was preaching to the world that markets are rarely wrong and that modern economics has found ways to iron out those few wrinkles that markets may have; Robert Lucas, the 1995 winner of the Nobel Prize in Economics,*3 had declared in 2003 that the ‘problem of depression prevention has been solved’.1 So most economists were caught completely by surprise by the 2008 global financial crisis. The subject matter of the calculation can be anything – marriage, having children, crime or drug addiction, as Gary Becker, the famous Chicago economist and the winner of 1992 Nobel Prize in Economics, has written about – and not just ‘economic’ issues, as non-economists would define them, such as jobs, money or international trade. But, from the point of view of most economists today, how Japanese sumo wrestlers collude to help each other out or how American schoolteachers fabricate their pupils’ marks to get better job assessments are as legitimate subjects of economics as whether Greece should stay in the Eurozone, how Samsung and Apple fight it out in the smartphone market or how we can reduce youth unemployment in Spain (which is over 55 per cent at the time of writing). Money is a symbol of what others in your society owe you, or your claim on particular amounts of the society’s resources.2 How money and other financial claims – such as company shares, derivatives and many complex financial products, which I will explain in later chapters – are created, sold and bought is one huge area of economics, called financial economics. You may get very high wages because you have very rare skills, like Cristiano Ronaldo, the football player. You may lose your job (or become unemployed) because someone invents a machine that can do what you do 100 times faster – as happened to Mr Bucket, Charlie’s father, a toothpaste cap-screwer, in the 2005 movie version of Roald Dahl’s Charlie and the Chocolate Factory. *5 Or you have to accept lower wages or worse working conditions because your company is losing money thanks to cheaper imports from, say, China. Wages and working conditions are also deeply affected by ‘political’ decisions to change the very scope and the characteristics of the labour market (I have put ‘political’ in quotation marks, as in the end the boundary between economics and politics is blurry, but that is a topic for later – see Chapter 11). The restriction on child labour in the late nineteenth century and early twentieth centuries had the opposite effect of shrinking the boundary of the labour market – suddenly a large proportion of the potential employees were shut out of the labour market. This can be either in the form of cash or ‘in kind’, that is, direct provision of particular goods (e.g., food) or services (e.g., primary education). This is known as the welfare state and is based on progressive taxation (those who earn more paying proportionally larger shares of their incomes in taxes) and universal benefits (where everyone, not just the poorest or the disabled, is entitled to a minimum income and to basic services, such as health care and education). We also buy and consume services – a bus ride, a haircut, a dinner at a restaurant or even a holiday abroad.3 So a lot of economics is devoted to the study of consumption – how people allocate money between different types of goods and services, how they make choices between competing varieties of the same product, how they are manipulated and/or informed by advertisements, how companies spend money to build their ‘brand images’ and so on. And these are things that most readers may not normally have associated with economics, despite their crucial importance for the economy: how the factory is physically organized, how to control the workers or deal with trade unions, how to systematically improve the technologies used through research.'_____no_output_____pweh.split('.')_____no_output_____