# Notebook from simonwarchol/2020F-AC295
Path: content/exercises/exercise6/simon_warchol.ipynb

  AC295: Advanced Practical Data Science

## Transfer Learning for Text & Word Embeddings

**Harvard University, Fall 2020**
**Instructors**: Pavlos Protopapas

---_____no_output_____**Each assignment is graded out of 5 points.  The topic for this assignment is Transfer Learning for Text.**

**Due:** 10/27/2020 10:15 AM EDT

**Submit:** We won't be re running your notebooks, please ensure output is visible in the notebook._____no_output_____#### Learning Objectives

In this exercise you will cover the following topics:
- Tokenizing text using Text Vectorization
- Perform text classification & create word emedddings
- Load pre-trained word embeddings and perform text classification
- Understand Word Embeddings

---_____no_output_____#### Imports_____no_output_____

import os
import requests
import zipfile
import tarfile
import shutil
import json
import time
import sys
import string
import re
import numpy as np
import pandas as pd
from glob import glob
from string import Template
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

import tensorflow as tf
from tensorflow import keras
from tensorflow.python.keras import backend as K
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras import layers
from tensorflow.keras import activations
from tensorflow.keras import optimizers
from tensorflow.keras import losses
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization
from tensorflow.python.ops import io_ops
from tensorflow.keras.utils import to_categorical

from sklearn.model_selection import train_test_split
from sklearn.metrics.pairwise import cosine_similarity
K.clear_session()
_____no_output_____

#### Verify Setup_____no_output_____

# Enable/Disable Eager Execution
# Reference: https://www.tensorflow.org/guide/eager
# TensorFlow's eager execution is an imperative programming environment that evaluates operations immediately,
# without building graphs

#tf.compat.v1.disable_eager_execution()
#tf.compat.v1.enable_eager_execution()

print("tensorflow version", tf.__version__)
print("keras version", tf.keras.__version__)
print("Eager Execution Enabled:", tf.executing_eagerly())

# Get the number of replicas
strategy = tf.distribute.MirroredStrategy()
print("Number of replicas:", strategy.num_replicas_in_sync)

devices = tf.config.experimental.get_visible_devices()
print("Devices:", devices)
print(tf.config.experimental.list_logical_devices('GPU'))

print("GPU Available: ", tf.config.list_physical_devices('GPU'))
print("All Physical Devices", tf.config.list_physical_devices())

# Better performance with the tf.data API
# Reference: https://www.tensorflow.org/guide/data_performance
AUTOTUNE = tf.data.experimental.AUTOTUNEtensorflow version 2.3.0
keras version 2.4.0
Eager Execution Enabled: True
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
Number of replicas: 1
Devices: [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
[LogicalDevice(name='/device:GPU:0', device_type='GPU')]
GPU Available:  [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
All Physical Devices [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:XLA_CPU:0', device_type='XLA_CPU'), PhysicalDevice(name='/physical_device:XLA_GPU:0', device_type='XLA_GPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

#### Utils_____no_output_____

def download_file(packet_url, base_path="", extract=False):
  if base_path != "":
    if not os.path.exists(base_path):
      os.mkdir(base_path)
  packet_file = os.path.basename(packet_url)
  with requests.get(packet_url, stream=True) as r:
      r.raise_for_status()
      with open(os.path.join(base_path,packet_file), 'wb') as f:
          for chunk in r.iter_content(chunk_size=8192):
              f.write(chunk)

  if extract:
    if packet_file.endswith(".zip"):
      with zipfile.ZipFile(os.path.join(base_path,packet_file)) as zfile:
        zfile.extractall(base_path)

    if packet_file.endswith(".tar.gz"):
      packet_name = packet_file.split('.')[0]
      with tarfile.open(os.path.join(base_path,packet_file)) as tfile:
        tfile.extractall(base_path)

def evaluate_model(model,test_data, training_results):

  # Get the model train history
  model_train_history = training_results.history
  # Get the number of epochs the training was run for
  num_epochs = len(model_train_history["loss"])

  # Plot training results
  fig = plt.figure(figsize=(15,5))
  axs = fig.add_subplot(1,2,1)
  axs.set_title('Loss')
  # Plot all metrics
  for metric in ["loss","val_loss"]:
      axs.plot(np.arange(0, num_epochs), model_train_history[metric], label=metric)
  axs.legend()

  axs = fig.add_subplot(1,2,2)
  axs.set_title('Accuracy')
  # Plot all metrics
  for metric in ["accuracy","val_accuracy"]:
      axs.plot(np.arange(0, num_epochs), model_train_history[metric], label=metric)
  axs.legend()

  plt.show()

  # Evaluate on test data
  evaluation_results = model.evaluate(test_data)
  print("Evaluation Results:", evaluation_results)_____no_output_____

## Dataset_____no_output_____#### Download_____no_output_____

start_time = time.time()
download_file("https://storage.googleapis.com/dataset_store/ac295/news300.zip", base_path="datasets", extract=True)
download_file("https://github.com/shivasj/dataset-store/releases/download/v3.0/glove.6B.100d.txt.zip", base_path="embedding", extract=True)
execution_time = (time.time() - start_time)/60.0
print("Download execution time (mins)",execution_time)Download execution time (mins) 0.07883406082789103

#### Explore_____no_output_____

data_dir = os.path.join("datasets","news300")
label_names = os.listdir(data_dir)

# Number of unique labels
num_classes = len(label_names)
# Create label index for easy lookup
label2index = dict((name, index) for index, name in enumerate(label_names))
index2label = dict((index, name) for index, name in enumerate(label_names))

print("Number of classes:", num_classes)
print("Labels:", label_names)

# Generate a list of labels and path to text
data_x = []
data_y = []

for label in label_names:
  text_files = os.listdir(os.path.join(data_dir,label))
  data_x.extend([os.path.join(data_dir,label,f) for f in text_files])
  data_y.extend([label for f in text_files])

# Preview
print("data_x count:",len(data_x))
print("data_y count:",len(data_y))
print(data_x[:5])
print(data_y[:5])

# sns.countplot()
# plt.show()
np.unique(data_y, return_counts=True)Number of classes: 3
Labels: ['politics', 'health', 'entertainment']
data_x count: 920
data_y count: 920
['datasets/news300/politics/182.txt', 'datasets/news300/politics/96.txt', 'datasets/news300/politics/132.txt', 'datasets/news300/politics/19.txt', 'datasets/news300/politics/68.txt']
['politics', 'politics', 'politics', 'politics', 'politics']

#### View Text_____no_output_____

# Generate a random sample of index
data_samples = np.random.randint(0,high=len(data_x)-1, size=10)
for i,data_idx in enumerate(data_samples):
  # Read text
  txt = io_ops.read_file(data_x[data_idx])

  print("Label:",data_y[data_idx],", Text:",txt.numpy())Label: politics , Text: b'Kayleigh McEnany just made things worse  (CNN)In the wake of President Donald Trump equivocating as to whether he would transfer power to Joe Biden in the event of a defeat at the ballot box in 40 days, White House press secretary Kayleigh McEnany tried to clear things up on Thursday. Spoiler alert: She didn\'t! "The President will accept the results of a free and fair election," said McEnany. Which, well, that ain\'t it. See, Trump has said -- as recently as Wednesday night-- that the only way the election will be fair is if we "get rid of the ballots and you\'ll have a very trans- -- we\'ll have a very peaceful -- there won\'t be a transfer, frankly; there\'ll be a continuation." So, by that standard, the only way that the election will be "free and fair" (in McEnany\'s words) is if all mail-in ballots -- many of which have already been sent and returned! -- are somehow disqualified (or something?). Which, because every poll suggests that Democratic voters are more inclined to use mail-in balloting amid the coronavirus pandemic than GOP voters, would hand Trump a major advantage. Not to mention disenfranchise lots and lots of people who are eligible voters! See, the thing is that neither Trump nor McEnany get to define "free and fair." Those words have objective meanings apart from any spin. They mean that everyone who is eligible to vote is given an equal opportunity to cast a vote -- in whatever means are legally available to them in their state. The Point: Trump is a president, not a dictator. He doesn\'t get to decide how the election rules work. He just plays within them. He -- and McEnany -- still don\'t seem to grasp that concept of, uh, how democracy works. CNN\'s Allison Gordon contributed to this report '
Label: entertainment , Text: b'\'SNL\' returns with Alec Baldwin\'s Donald Trump taking on Jim Carrey\'s Joe Biden New York (CNN Business)"Saturday Night Live" returned to a live studio audience at Studio 8H after months away because of coronavirus, and the show didn\'t waste any time. The NBC variety show opened with a sketch taking on this week\'s presidential debate between President Donald Trump, played by Alec Baldwin, and former Vice President Joe Biden, who was played by Jim Carrey. "The following is a re-broadcast of Tuesday\'s presidential debate, even though Tuesday feels 100 days ago," a voice said opening the show. "We thought it was important to see it again since it might be the only presidential debate. And it was pretty fun to watch, as long as you don\'t live in America." Moderator Chris Wallace, played by Beck Bennett, brought out Baldwin\'s Trump first. "And you did take the Covid test that you promised to take in advance, correct?" Bennett\'s Wallace asked.  "Absolutely, scout\'s honor," Baldwin\'s Trump responded with his fingers crossed. Bennett\'s Wallace then brought out Carrey\'s Biden. "Just one second, Chris," he said, using a tape measure to put enough space between himself and Baldwin\'s Trump. Bennett\'s Wallace said that it looked like Carrey\'s Biden was ready to debate.  "Absolutely not," he responded. "But I got the beginning of 46 fantastic ideas that I may or may not have access to." Then the debate kicked off. However, instead of a debate it was just Baldwin\'s Trump and Carrey\'s Biden yelling back and forth at each other. "Chris Wallace is mean, the economy is mean. It keeps losing jobs, which is mean to me," Baldwin\'s Trump said. He then called the pandemic a hoax. "That statement is something that will probably come back to haunt me later this week," he said. Maya Rudolph\'s Kamala Harris then came out to stop Trump from being mean to her running mate. "You look at me, Donald. You do not treat my Joe like that," she said. "He\'s a nice boy." But the arguing continued between the two candidates until Biden had enough and pulled out a remote control to pause Trump. "Sorry, but I think we all needed a break," Carrey\'s Biden said. "Let\'s bask in the Trumplessness." Carrey\'s Biden then spoke directly to the camera and introduced his new campaign slogan: "America -- Not Actively On Fire Again." After unpausing Baldwin\'s Trump, Carrey, Baldwin and Rudolph all came out to say the show\'s catch phrase, "Live from New York ... It\'s Saturday night!"  '
Label: health , Text: b'The US should expect 20,000 more coronavirus deaths by the end of the month, former CDC director says  (CNN)Another 20,000 Covid-19 deaths by the end of the month are "inevitable," according to a former director of the US Centers for Disease Control and Prevention.  The estimate is based on the number of infections "that have already occurred," Dr. Tom Frieden said Saturday, during CNN\'s "Coronavirus: Facts and Fears" town hall. The United States reported 57,420 new coronavirus cases on Friday, the highest number of new daily cases since August. "Anytime we ignore, minimize or underestimate this virus, we do so at our peril and the peril of people whose lives depend on us," Frieden said. More than 7.7 million people have been infected with the virus in the US and 214,377 have died, according to data from Johns Hopkins University. By February, the coronavirus death toll in the US could double to about 400,000, a model from the Institute for Health Metrics and Evaluation at the University of Washington School of Medicine projected. Daily deaths will peak at about 2,300 in mid-January, the model predicted. Projections aren\'t set in stone, however, and what the public does can have a big impact, another former CDC director, Dr. Richard Besser, said.  New Covid-19 cases continue to grow across the country, though. Florida health officials reported 5,570 new cases of Covid-19 on Sunday after there was no case or death update on Saturday, according to the Florida Department of Health.    There were also 178 Floridian deaths reported on Sunday, bring that total to 15,364, according to the data. In Montana, it took almost five months from the beginning of the pandemic for the state to reach 5,000 confirmed coronavirus cases. But in just the past 11 days the state has reported a further 5,000 cases as the virus continues to spread throughout the country. Impacts could be much greater than officials think Officials are tracking coronavirus infections and deaths, but Frieden said those numbers may be too small. The true number of coronavirus deaths in the United States is well over a quarter million, Frieden said Saturday.  Part of the problem in determining the true impact is how deaths are listed on death certificates, especially for older patients who are more likely to have other health problems along with a coronavirus infection. Often the other health condition is listed as the cause of death, he said. "If you die from cancer, and you also have diabetes, you still died from cancer," Frieden explained. "If you died from Covid, and you also had diabetes, you died from Covid." The number of infections is likely closer to 40 million people, he said.  "You may not get sick at all from this, but you may spread it to someone who then dies, or spreads it to someone else who dies," he said. "That\'s why we all have to recognize that we\'re in this together. There\'s only one enemy, and that\'s the virus." Regaining trust in vaccines and health agencies While researchers are racing to develop a coronavirus vaccine, health experts said Saturday that improving trust and accessibility around any potential vaccine is essential. "For a vaccine to actually work, it\'s got to not only be safe and effective, but also be accessible and trusted. And that\'s why it\'s so important that it not get politicized and not be seen as from any political party or political figure," Frieden said. "It\'s a big job to get vaccines out there." Opinion polls show that the public doesn\'t trust the CDC with information around coronavirus. The way to regain trust "depends first and foremost on telling the truth, even when it\'s hard," Dr. Julie Gerberding said in the townhall. "Americans can tolerate really tough truths, but it has to come from reliable and credible sources," Gerberding said. "One of the reasons that we have so much anxiety among Americans is because they\'re hearing different things from different political leaders. We haven\'t consolidated and cascaded the messages from reliable sources." She reiterated what the other former CDC leaders said during the town hall: Transparency is key. "It is going to be very difficult to crawl back into a situation where people believe that we really do have their best interest at heart. The good news is that science is on our side," she said. A wake-up call for politicians and public health officials Former CDC directors warned that both politicians and public health officials need to take the virus seriously for the sake of the public. President Donald Trump said that Americans shouldn\'t let fear of coronavirus dominate their lives, but former director Dr. Jeffrey Koplan said the country should "absolutely" be afraid. "When your leadership is working against you in this virus, the virus has an ally that makes it a pretty strong contender for further destruction," Koplan said. Koplan said he believes this pandemic will serve as a wake-up call for politicians and public health officials. "I would very much hope that we see stronger state and local health departments, working with a stronger CDC -- that there is particularly more attention paid to an up-to-date surveillance system, early detection of problems, and then approaches towards ameliorating them," he said. Correction: A previous version of this story incorrectly reported the number of daily deaths predicted by a model from the Institute for Health Metrics and Evaluation at the University of Washington School of Medicine. Daily deaths will peak at about 2,300 in mid-January, the model predicts. CNN\'s Lauren Mascarenhas, Andrea Kane, Leanna Faulk, Shelby Lin Erdman, Nicole Chavez and Christina Maxouris contributed to this report. '
Label: politics , Text: b"2020 Presidential Debates Fast Facts  (CNN)Here's a look at the 2020 presidential debates. June 26, 2019 Event Type: Democratic DebateLocation: Miami, FloridaHosts: NBC News, MSNBC and TelemundoModerators: Jos\xc3\xa9 Diaz-Balart, Savannah Guthrie, Lester Holt, Rachel Maddow and Chuck ToddParticipants: Cory Booker, Juli\xc3\xa1n Castro, Bill de Blasio, John Delaney, Tulsi Gabbard, Jay Inslee, Amy Klobuchar, Beto O'Rourke, Tim Ryan, Elizabeth WarrenTranscript June 27, 2019 Event Type: Democratic DebateLocation: Miami, FloridaHosts: NBC News, MSNBC and TelemundoModerators: Jos\xc3\xa9 Diaz-Balart, Savannah Guthrie, Lester Holt, Rachel Maddow and Chuck ToddParticipants: Michael Bennet, Joe Biden, Pete Buttigieg, Kirsten Gillibrand, Kamala Harris, John Hickenlooper, Bernie Sanders, Eric Swalwell, Marianne Williamson, Andrew YangTranscript July 30, 2019 Event Type: Democratic DebateLocation: Detroit, MichiganHosts: CNN, CNN International and CNN en Espa\xc3\xb1olModerators: Dana Bash, Don Lemon and Jake TapperParticipants: Steve Bullock, Pete Buttigieg, John Delaney, John Hickenlooper, Amy Klobuchar, Beto O'Rourke, Tim Ryan, Bernie Sanders, Elizabeth Warren, Marianne WilliamsonTranscript July 31, 2019 Event Type: Democratic DebateLocation: Detroit, MichiganHosts: CNN, CNN International and CNN en Espa\xc3\xb1olModerators: Dana Bash, Don Lemon and Jake TapperParticipants: Michael Bennet, Joe Biden, Cory Booker, Juli\xc3\xa1n Castro, Bill de Blasio, Tulsi Gabbard, Kirsten Gillibrand, Kamala Harris, Jay Inslee, Andrew YangTranscript September 12, 2019 Event Type: Democratic DebateLocation: Houston, TexasHosts: ABC News and UnivisionModerators: Linsey Davis, David Muir, Jorge Ramos and George StephanopoulosParticipants: Joe Biden, Cory Booker, Pete Buttigieg, Juli\xc3\xa1n Castro, Kamala Harris, Amy Klobuchar, Beto O'Rourke, Bernie Sanders, Elizabeth Warren, Andrew YangTranscript October 15, 2019 Event Type: Democratic DebateLocation: Westerville, OhioHosts: CNN and The New York TimesModerators: Erin Burnett, Anderson Cooper and Marc LaceyParticipants: Joe Biden, Cory Booker, Pete Buttigieg, Juli\xc3\xa1n Castro, Tulsi Gabbard, Kamala Harris, Amy Klobuchar, Beto O'Rourke, Bernie Sanders, Tom Steyer, Elizabeth Warren and Andrew YangTranscripts: 8pm ET, 9pm ET, 10pm ET November 20, 2019 Event Type: Democratic DebateLocation: AtlantaHosts: MSNBC and The Washington PostModerators: Rachel Maddow, Andrea Mitchell, Ashley Parker and Kristen WelkerParticipants: Joe Biden, Cory Booker, Pete Buttigieg, Tulsi Gabbard, Kamala Harris, Amy Klobuchar, Bernie Sanders, Tom Steyer, Elizabeth Warren and Andrew YangTranscript December 19, 2019 Event Type: Democratic DebateLocation: Los AngelesHosts: PBS NewsHour and PoliticoModerators: Tim Alberta, Yamiche Alcindor, Amna Nawaz and Judy WoodruffParticipants: Joe Biden, Pete Buttigieg, Amy Klobuchar, Bernie Sanders, Tom Steyer, Elizabeth Warren and Andrew YangTranscript January 14, 2020 Event Type: Democratic DebateLocation: Des Moines, IowaHosts: CNN and Des Moines RegisterModerators: Wolf Blitzer, Brianne Pfannenstiel and Abby PhillipParticipants: Joe Biden, Pete Buttigieg, Amy Klobuchar, Bernie Sanders, Tom Steyer and Elizabeth WarrenTranscripts: 9pm ET, 10pm ET February 7, 2020 Event Type: Democratic DebateLocation: Manchester, New HampshireHosts: ABC, Apple News and WMUR-TVModerators: Linsey Davis, Monica Hernandez, David Muir, Adam Sexton and George StephanopoulosParticipants: Joe Biden, Pete Buttigieg, Amy Klobuchar, Bernie Sanders, Tom Steyer, Elizabeth Warren and Andrew Yang February 19, 2020 Event Type: Democratic DebateLocation: Las Vegas, NevadaHosts: MSNBC, NBC News, The Nevada Independent and TelemundoModerators: Vanessa Hauc, Lester Holt, Hallie Jackson, Jon Ralston and Chuck ToddParticipants: Joe Biden, Michael Bloomberg, Pete Buttigieg, Amy Klobuchar, Bernie Sanders and Elizabeth WarrenTranscript February 25, 2020 Event Type: Democratic DebateLocation: Charleston, South CarolinaHosts: CBS News, the Congressional Black Caucus Institute and TwitterModerators: Margaret Brennan, Major Garrett, Gayle King, Norah O'Donnell and Bill WhitakerParticipants: Joe Biden, Michael Bloomberg, Pete Buttigieg, Amy Klobuchar, Bernie Sanders, Tom Steyer and Elizabeth WarrenTranscript March 15, 2020 Event Type: Democratic DebateLocation: Washington, DCHosts: CNN and UnivisionModerators: Dana Bash, Ilia Calder\xc3\xb3n and Jake TapperParticipants: Joe Biden and Bernie SandersTranscript September 29, 2020 Event Type: First Presidential DebateLocation: Cleveland, OhioHosts: Case Western Reserve University and Cleveland ClinicModerator: Chris WallaceParticipants: Joe Biden and Donald TrumpTranscript October 7, 2020 Event Type: Vice Presidential DebateLocation: Salt Lake City, UtahHosts: The University of UtahModerator: Susan PageParticipants: Kamala Harris and Mike PenceTranscript October 15, 2020 October 9, 2020 - The Commission on Presidential Debates cancels the second presidential debate after Trump declines to do a virtual debate despite concerns over his Covid-19 diagnosis. October 22, 2020 Event Type: Third Presidential DebateLocation: Nashville, TennesseeHosts: Belmont UniversityModerator: Kristen WelkerParticipants: Joe Biden and Donald Trump  "
Label: politics , Text: b'How Alexandria Ocasio-Cortez\'s political future is shaping Senate Democrats\' SCOTUS strategy  (CNN)On Sunday, something weird happened. Senate Democratic leader Chuck Schumer and Rep. Alexandria Ocasio-Cortez, who is widely rumored as a potential future primary challenger to the New York senator, held a joint press conference to talk about how their party will attempt to keep the Supreme Court vacancy caused by the death of Justice Ruth Bader Ginsburg vacant until after the 2020 presidential election. AOC implored Democrats to "mobilize on an unprecedented scale" in order to pressure Senate Majority Leader Mitch McConnell (R-Kentucky) not to push a vote before the election, adding: "We need to tell him that he is playing with fire." Schumer took that rhetorical threat even further. When asked about the possibility of Senate Democrats adding Supreme Court seats or eliminating the filibuster entirely if they win back the majority in 43 days, Schumer responded this way: "Well, it will be a decision that is ... comes to the Senate. We first have to win the majority before that can happen. But once we win the majority, God willing, everything is on the table. "... As I said, we first have to win the majority, because if we don\'t win the majority, these questions are all moot. But if we win the majority, everything is on the table." That is a remarkable statement by the man who, as of today, has at least a 50-50 chance of running the Senate come 2021. It\'s also a reflection of just how aware Schumer is of the potential threat AOC poses to him in a 2022 Democratic primary -- and how much he is going out of his way to move to the ideological left to shut down the possibility.  For months, AOC has refused to rule out challenging Schumer from the left come 2022. "I don\'t know," the New York Democratic congresswoman said this spring of a potential primary race. "Honestly, this news cycle is so insane, who knows where any of us are going to be in 2022?" Which, in the world of politics, is essentially her acknowledging that she is thinking about it. Or at least wants Schumer to believe she is thinking about it. Because all AOC had to say -- either back in the spring or, well, at anytime between then and now -- is something like: No, I am not going to run against Chuck Schumer in 2022. I support him and his efforts to win back the Senate majority this fall. That she hasn\'t said that -- and, in fact, has left the door wide open, tells you everything you need to know.  And Schumer is paying attention. As John Bresnahan and Marianne Levine documented in this Politico piece in late August:  "The self-described \'angry centrist\' is no more. Once derided for being too close to Wall Street, Schumer aides now boast that he has stood up to the financial services industry. Schumer is still distrusted by some on the left, but the New York Democrat insists his views have shifted to reflect a different constituency, as well as the more progressive Democratic Party of the Donald Trump era." Or, as Schumer himself explained his evolution to Politico: "A good elected official looks at the needs of the people he or she represents and does everything he or she can to help solve those needs, and the world changes. And the problems that existed, say in the \'90s, are different than the problems that exist today." Sure! There\'s no question that the election of Donald Trump in 2016 -- and the resultant capitulation of elected Republican leaders to his whims in the years that followed -- have forced Democrats to reevaluate how they see the other party and how much (if anything) can be done on a bipartisan basis. But it is equally impossible to believe that Schumer\'s change from proud centrist to defiant liberal has nothing to do with the specter of AOC running against him in 2022. Schumer is a deeply political animal -- you don\'t get to where he has gotten in politics without being one -- and he understands just how powerful AOC has become in the party since her upset primary win over Rep. Joe Crowley, a member of the House Democratic leadership, in 2018. (Sound familiar???) Schumer\'s slide to the left is an acknowledgment of the rising power of the liberal left in his party. Progressives who aligned far more closely with AOC and Vermont Sen. Bernie Sanders than Schumer and former Vice President Joe Biden are ascendant in Democratic politics, as illustrated by a series of primary wins for liberals -- including in Schumer\'s home state -- this year.  He also understands the power that AOC possesses within the Democratic Party. With Sanders, presumably, stepping off the national stage (he has now run for president unsuccessfully twice) the liberal wing of the party now seems to be AOC\'s for the taking. And if her fundraising and staff hires are any indication, she is taking it. Given that status, it is not inconceivable that AOC could oust Schumer -- even if he is the Senate majority leader in 2022.  All of which makes Schumer more than willing to bring AOC as close as possible on this Supreme Court fight -- and make proclamations like "everything is on the table."  '
Label: politics , Text: b'Schiff subpoenas DHS for documents and testimony related to whistleblower allegations  (CNN)The House Intelligence Committee has subpoenaed the Department of Homeland Security over documents and testimony related to an ongoing investigation into the agency\'s intelligence office, the panel\'s chairman, Rep. Adam Schiff, announced Tuesday.  The subpoenas are connected to the committee\'s probe into actions taken by the department\'s intelligence arm during recent protests around the country and a whistleblower complaint filed earlier this month that alleges top political appointees in the department repeatedly instructed career officials to modify intelligence assessments to suit President Donald Trump\'s agenda by downplaying Russia\'s efforts to interfere in the US and the threat posed by White supremacists.  Earlier this month, CNN first reported details from a whistleblower complaint filed by Brian Murphy to the DHS inspector general. Murphy previously oversaw the intelligence division at the department but was reassigned this summer after it was revealed his office had gathered intelligence reports on two US journalists. On Tuesday, Schiff said his panel had issued a subpoena to compel DHS to turn over "outstanding documents, communications, and other records pertaining to the Committee\'s ongoing investigation" by October 6, according to a letter from the California Democrat to Joseph Maher, the Senior Official Performing the Duties of the Under Secretary for Intelligence and Analysis (I&A).  Separately, Maher was also subpoenaed to testify before the committee in an open hearing later this week and tell lawmakers, under oath, why DHS is "preventing a lawful Intelligence Community whistleblower\xe2\x80\x94the former head of an Intelligence Community element\xe2\x80\x94from providing classified testimony to the Committee by delaying security clearances for his attorneys," the letter says.  Schiff also called on Maher to explain why the department\'s intelligence office "failed to produce requested documents to the Committee, and to answer questions about Mr. Maher\'s own knowledge of and involvement the matters under investigation."  "The Committee has an obligation to independently investigate and substantiate serious allegations against senior Department of Homeland Security and White House officials. These allegations include the suppression of intelligence reports about Russia\'s election interference and efforts to modify intelligence assessments to match Trump\'s rhetoric on white supremacy, Antifa, and terrorism threats at the border," Schiff said in a statement Tuesday after issuing the subpoenas.  "Unfortunately, the Department has unlawfully obstructed the Committee\'s investigation by refusing to produce numerous documents related to the allegations in the whistleblower complaint and related to its activities in Portland. It has even taken the unprecedented step of slow walking security clearances for counsel to the whistleblower\xe2\x80\x94the former acting head of an IC element," he said.  "After weeks, and in some cases months, of attempted accommodation with the Department, we were left with no choice but to issue two subpoenas today ... Simply put, the Committee will no longer tolerate the obstruction and attempts to run out the clock by the Department," Schiff added.  Tuesday\'s subpoenas come as Schiff and Murphy\'s attorneys have accused DHS of slow-walking the panel\'s investigation into the department\'s intelligence division and the allegations raised in the recently filed whistleblower complaint. In that complaint, Murphy claims that acting DHS Secretary Chad Wolf instructed DHS officials earlier this year to "cease providing intelligence assessments on the threat of Russian interference" and, instead, focus their efforts on gathering information related to activities being carried out by China and Iran. He also alleges that Wolf and Cuccinelli, both Trump appointees, directed officials to change intelligence assessments to ensure they matched up with misleading public comments from Trump about Antifa and "anarchist" groups, according to a complaint filed with the DHS inspector general. Separately, both Wolf and Cuccinelli also tried to alter a report to downplay the threat posed by White supremacists and instead emphasize the role of leftist groups due to concerns about how the initial language would reflect on the President, according to documents filed to the inspector general\'s office.  DHS spokesman Alexei Woltornist said in a statement to CNN earlier this month that "the Department generally does not comment on the specifics of OIG referrals, but we flatly deny that there is any truth to the merits of Mr. Murphy\'s claim." "DHS looks forward to the results of any resulting investigation and we expect it will conclude that no retaliatory action was taken against Mr. Murphy," he added. CNN\'s Jeremy Herb contributed reporting  '
Label: entertainment , Text: b'CBS reporter Nina Kapur, 26, dies after rental moped accident in New York   (CNN)Nina Kapur, a 26-year-old reporter in New York City, died after she fell from a moped that had been rented from Revel, a popular moped sharing service.  "We are heartbroken here at CBS2 as a member of our news family has passed away," CBS New York, a CNN affiliate, said in a statement on Sunday. "Reporter Nina Kapur joined the team in June 2019 and was known for her contagious smile and love for storytelling."  Kapur was riding on the back of a Revel rental scooter that was being driven by a 26-year-old man when he "swerved for an unknown reason" Saturday evening in Greenpoint, Brooklyn, according to NYPD spokeswoman Denise Moroney.  Kapur and the driver fell to the roadway, and while the driver sustained minor injuries, Kapur was transported to a hospital where she was pronounced dead.  The investigation is ongoing, Moroney said. Neither were wearing helmets. Similar to the electric scooter rentals that have popped up nationwide, people can use a Revel moped by simply downloading an app and unlocking the nearest one for $1. They\'re currently available in New York, Austin, Texas, Miami, Oakland, California, and Washington D.C.  Anyone over 21 and with a valid license can drive one.  Revels only go up to 30 miles per hour and riding on highways and major bridges is not allowed, according to Revel\'s website. Helmets, which the company says are cleaned "with COVID-19-effective disinfectants," are provided in the trunk of the moped.  In a statement to CNN, a Revel spokesperson said the company was "actively investigating this incident" and working with the NYPD in its investigation as well.   "Revel extends deepest sympathies to Nina Kapur\'s family and loved ones for their loss," the spokesperson added.  Before Kapur joined CBS New York as an on air reporter, she had worked at News 12 in Connecticut.  "News 12 staff members are remembering Kapur for her incredible work ethic, as well as her sense of humor and smile," the station shared.  Her fellow journalists took to Twitter to remember her.  "@NinaKapur was vivacious and had big plans for the future. Funny, fashionable, and often the first person to like a picture on social media," Suzanne Goldklang of News12 tweeted.  "Nina was perfect. So beautiful. So vibrant. She inspired me by being an excellent journalist and by making big moves at such a young age," Aliah Williamson from WDTN tweeted.  Kapur was a 2016 graduate of Syracuse University\'s broadcast and digital journalism program.  Barbara Fought, Kapur\'s faculty adviser and one of her professors, described the young journalist as "a bright light, a very positive person, with an infectious smile," according to an email by Mark Lodato, the dean of the university\'s school of public communications sent to faculty and staff.   '
Label: politics , Text: b'Trump made another baseless political claim about Fauci Washington (CNN)President Donald Trump baselessly claimed on Thursday that Dr. Anthony Fauci, a key member of the White House\'s Coronavirus Task Force, was a Democrat when in fact he is not registered with any political party. The director of the National Institute of Allergy and Infectious Diseases told CNN he isn\'t registered to a political party. Voter records show that, too. Fauci has also served in his current role under Democratic and Republican US presidents going back to the Reagan administration.  Trump made his claims at a North Carolina rally, once again questioning whether masks were an effective means to prevent the spread of the coronavirus and inaccurately painting Fauci as a Democrat who encouraged people to "not wear a mask under any circumstance."  "(Y)ou have my friend and he\'s a nice guy Tony, Tony Fauci. He\'s a nice guy." Trump told supporters, who booed at the mention of Fauci\'s name.  The President continued: "He said, \'This is not a threat. This is not a problem. Don\'t worry about this, it\'s not a problem.\' That\'s the craziest thing. Then he said, \'Do not wear a mask. Do not wear a mask under any circumstance. Don\'t wear a mask -- don\'t, don\'t, don\'t,\' right? So, we don\'t wear a mask, then they say \'Oh, wear a mask.\' I never saw -- do you ever see any conflicts? But he\'s a nice guy so I keep him around, right? We\'ll keep him around." Trump has repeatedly used Fauci\'s past comments to question whether masks are effective. But his comparison is inaccurate.  Before the virus spread globally, Fauci did say that the American people shouldn\'t be worried, but he was always clear that the situation could change and that public health officials needed to take it seriously. He never said, as Trump suggested, that Americans should not wear a mask under any circumstance. But Fauci was among federal public health officials appointed to the White House Coronavirus Task Force advising against non-health workers from wearing a mask early on in the pandemic, saying there was little evidence masks protect the wearers -- something that is still true. Federal public health officials, including Fauci, have now said for several months that the science on wearing facial coverings has evolved since the early days of the pandemic. They advise that Americans should wear a facial covering to prevent the spread of the virus.  The President\'s and Fauci have butted heads over how to handle the coronavirus pandemic for months. And though Fauci was appointed to the White House\'s group of federal public health officials to address the pandemic, Trump now frequently eschews its recommendations for advice more suitable to his political mission -- "reopening" the country.  Most recently, they clashed after his comments were cut into a pro-Trump campaign. Fauci has maintained that he has never endorsed a candidate for President and that the campaign should remove the ad. But after the dispute, Trump had mostly refrained from attacking the doctor on the campaign trail.  Fauci has also has been warning that rallies like the ones Trump is holding are "asking for trouble." "Because when you look at what is going on in the United States, it is really very troublesome," he said earlier this week.   He warned that ignoring the pandemic is a big problem. "I think we\'re facing a whole lot of trouble," when discussing the rising number of daily deaths, adding, "That\'s a bad place to be when you\'re going into the cooler weather of the fall and the colder weather of the winter."   CNN\'s Jim Acosta and Kaitlan Collins contributed to this report.  '
Label: health , Text: b'Will 2020 see a spike in AFM cases? Social distancing could have an impact, CDC says    (CNN)If acute flaccid myelitis or AFM follows its usual pattern, 2020 should be a year that sees a spike in US cases -- but the country hasn\'t seen one yet, according to the US Centers for Disease Control and Prevention. It\'s also unclear what impact Covid-19 will have on the rare polio-like condition, which affects the spinal cord and nerves, mostly in children. The CDC provided an update about AFM in its Grand Rounds meeting Tuesday. It tracks cases each day and has not seen a spike so far, but it may be too early to know. In past years, when there have been outbreaks, case numbers typically peak in September.  As of the end of June, the last time CDC reported numbers, there had been 13 cases in 2020 and 33 cases under investigation.  The first hallmark sign someone has AFM is sudden limb weakness. The polio-like condition occurs mostly in children who are otherwise healthy.  The progression of weakness can happen over hours to a number of days, and can lead to difficulty speaking and swallowing, or it can cause severe respiratory distress. It can also cause lesions on the spinal cord and permanent, sometimes life-threatening, paralysis. About 98% of AFM cases are hospitalized and 54% are placed in the ICU, according to the CDC. "There is a great deal we don\'t yet know," Dr. Anne Schuchat, principal deputy director of the CDC, said Tuesday. "We want to do everything possible to learn more about recognizing, treating, and preventing AFM and helping families to heal." "AFM is a CDC priority," Schuchat  added. It is clear that Covid-19 will have some impact on this AFM cycle, the CDC said. Social distancing and improved hand hygiene practices may reduce the number of AFM cases, said Dr. Janell Routh, medical officer and lead for the CDC\'s AFM and Domestic Poliovirus Team.  "These practices may result in a decease circulation of other viruses, including enteroviruses and, therefore, either decrease or delay AFM this year," Routh said. The pandemic may also impact how the country tracks how many cases there are. "We don\'t know if the Covid-19 epidemic will complicate detection and management, nor do we know if the social distancing efforts will reduce the risks of occurrence," said Schuchat. "But we do know we need to be ready." The CDC and parent groups have been trying to raise awareness among doctors since it was first detected, as it had often been misdiagnosed, some studies have shown. One of the challenges of diagnosis is that the symptoms vary from person to person. It is unclear if some children are more susceptible to more severe forms of the disease. Studies are underway.  Most AFM cases start about a week after a child has had an infection, often involving a common cold virus called an enterovirus. In past years, there\'s evidence to suggest the main cause is enterovirus D68, or EV-D68. The National Institute of Allergy and Infectious Diseases said at the briefing that it is in the early stages of development for a vaccine for that particular enterovirus. There is no vaccine for AFM and no cure for AFM, which appears to be a post-viral syndrome. Treatment only focuses on alleviating symptoms. Since 2014, when the CDC began tracking AFM in the United States, the agency noticed nationwide outbreaks seemed to occur every other year. There have been 630 confirmed cases since the CDC started tracking them. Patients often have long-term problems related to the condition. A retrospective analysis of AFM prior to 2014 showed that there were a small number of cases from year to year, but not these large peaks, suggesting something shifted regarding the nature of the condition. It\'s still unclear what that might be. "One thing we don\'t understand about AFM is what changed in 2014 that would account for the rise of cases of AFM," Dr. Emily Erbelding said at the briefing. Erbelding is director of the division of microbiology and infectious disease at NIAID. Scientists, she said, also don\'t understand why the cases spike every other year and whether it will be a pattern that the US will continue to see in the future. NIAID has launched a national study that is trying to get at more of these answers.  '
Label: politics , Text: b'Trump tries to recreate 2016 atmosphere in final push to Election Day Las Vegas (CNN)He\'s gathered the same people. He\'s attempting a similar schedule. He even believes he\'s found an equivalent 11th-hour scandal -- and equivalent Russian disinformation questions have followed. All that\'s missing is the airplane. A superstitious politician and a lover of routine, President Donald Trump is actively working to replicate the atmosphere that culminated in his 2016 victory, convinced if the same pieces are in place as he barnstorms battlegrounds at breakneck pace -- just like he did last time -- the lighting-in-a-bottle results will follow. He has even taken to thinking wistfully of his jet, a Boeing 757 painted red and gold, that he used to criss-cross the country during the frenzied final stretch of 2016. "I have the exact same plane at home. It\'s true. The exact same," Trump said Friday on a baking-hot tarmac in Florida, gesturing at the blue-and-white government airplane of the same model, but with considerably less gilt, positioned nearby.  Visit CNN\'s Election Center for full coverage of the 2020 race "It\'s a different company that made the engines," he said, quickly identifying a shortcoming compared to his own jet. "I think those are Pratt & Whitney, and I have Rolls-Royce." Nostalgia has always weighed heavily on Trump\'s political persona -- it\'s the second "A" in his "MAGA" motto -- but it also effects his decisions and strategy, often against the advice of his political team, according to people who have spoken to him recently about his reelection efforts.  As he makes a final push for a second term, Trump wants to replicate the heady last days of his only previous political campaign, even though he is now the incumbent and the political calculus has shifted dramatically. But this time, Trump seems more aggrieved at what has happened to him than what has happened to the base of people who support him.  Jetting across the country this weekend, the collection of advisers joining him in the cramped forward cabin aboard Air Force One included veteran 2016 advisers Jared Kushner, Stephen Miller, Dan Scavino and Hope Hicks, who Trump convinced to return to his orbit eight months ago. Absent were some of the more recent additions to his circle, notably his chief of staff Mark Meadows. As he did many times in 2016, Trump is basing himself for two days at his Las Vegas hotel as he raises cash across the border in Orange County and hosts campaign rallies in Arizona. He is even attending the same evangelical church a few miles from the Strip that he visited in October 2016, when pastors and ministers gathered to lay their hands on him and pray. Like four years ago, Trump hopes that by outworking his rival -- three-a-day rallies, often stretching past 90 minutes, in every state he is contesting -- he can eke out a victory that defies polling and punditry. Once seemingly out of reach because of the viral pandemic ravaging the country, Trump has insisted his signature campaign rallies proceed, convinced they are what propelled him to office the first time around. They have grown larger as the calendar ticks toward November 3, though are unlikely to ever match the tens of thousands who once came to hear him speak. And he is charging headfirst into making baseless accusations against Democratic presidential nominee Joe Biden of being a "criminal" who, in Trump\'s telling, sits atop a shady syndicate of political for personal gain. It\'s the exact accusation he used to lob at Hillary Clinton (and still does, even though he defeated her), even as he is again being aided by a Russian disinformation campaign targeting Biden. The President has often lamented that his political efforts since entering the White House have felt staid compared to his efforts four years ago, people who have spoken to him said, weighed down by the massive apparatus that accompanies any president and saddled with an incumbent\'s responsibilities. Trump has fondly recalled those final 2016 campaign days as the last when he was unburdened by the job he eventually won, and wants to feel the same excitement of his first go-around. He enjoys reminiscing about what it was like in the days before the election -- and recounts Election Night 2016 at nearly every rally he attends. "That beautiful night, four years ago, was at the greatest of all time, probably the greatest night in the history of television," Trump said on a blustery tarmac in Janesville, Wisconsin, on Saturday night as a crowd roared in approval. "We had so much fun. The tears that were flowing -- remember the tears?" Former officials also say Trump has a superstitious streak and believes if the circumstances of his campaign are similarly aligned -- from the people to the rallies to the rhetoric -- he might eke out a victory again. Yet a little like a college freshman trying to bring together friends over Thanksgiving break, only to learn everyone has changed, Trump is finding that nostalgic reminiscence can\'t always be manifested into reality.  The burdens of being the incumbent, saddled with problems like the coronavirus pandemic and a stalled economy, have changed how Americans view him. And what was once a novel political approach -- to air openly the grievances that politicians have long sought to stoke only tacitly -- now seems to many observers like a tired act. Four years ago -- and particularly as he worked to re-win voters after the Access Hollywood scandal -- Trump hammered on issues like immigration and trade that he said were unfairly advantaging the elite to the determent of blue-collar Americans. His focus and message narrowed as the campaign wound to a close. Now, he has railed at "deep state" misdeeds he claims weigh on his presidency and warrant an extension of his term. He complains at length about what interviewers have asked him and the tone in which they asked it. And he repeatedly insists he\'s given up millions of dollars in income by serving as president. In Oval Office sessions over the past week, Trump has remained insistent that what brought him victory in 2016 will work again, a person briefed on the sessions said. His refusal to adjust to a starkly different political reality has caused frustrations among Republicans, who fear his deep unpopularity could doom not only his presidency but GOP control of the Senate. Aides have worked to little avail to intervene and hone a message more fitting for a sitting president than an outside rabble-rouser, including in recent days as polls continue to show him losing ground among voters he won four year ago.  At times, Trump has taken the directive literally, appealing directly to "suburban women" in his campaign speeches. At other moments he has stated his campaign\'s electoral calculus as if arriving fresh from a political briefing. "You know what? We win Wisconsin, we win the whole ballgame," he said Saturday night as the temperature dropped in Janesville. "What the hell do you think I\'m doing here on a freezing night with 45-degree wind? You think I\'m doing this for my health?"  '

#### Build Data Pipelines_____no_output_____##### Text Vectorization

[Reference](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/TextVectorization)

Generate Text Vector:
- Standardize each sample (usually lowercasing + punctuation stripping)
- Split each sample into substrings (usually words)
- Recombine substrings into tokens (usually ngrams)
- Index tokens (associate a unique int value with each token)
- Transform each sample using this index, either into a vector of ints or a dense float vector_____no_output_____

# Text Vectorization
def standardize_text(input_text):
  # Convert to lowercase
  lowercase = tf.strings.lower(input_text)
  # Remove HTML tags
  stripped_html = tf.strings.regex_replace(lowercase, "
", " ")
  return tf.strings.regex_replace(
      stripped_html, "[%s]" % re.escape(string.punctuation), ""
  )

# Load Text
def load_text(path, label=None):
  text = io_ops.read_file(path)
  if label is None:
    return text
  else:
    return text, label

# Feature constraints
max_features = 15000
sequence_length = 1000

# Initialize Text Vectorizer
text_vectorizer = TextVectorization(
    standardize=standardize_text,
    max_tokens=max_features,
    output_mode="int",
    output_sequence_length=sequence_length,
)

# Create the vocabulary of entire dataset
text_data = tf.data.Dataset.from_tensor_slices(data_x)
text_data = text_data.map(load_text, num_parallel_calls=AUTOTUNE)

# Generate Text Vector
start_time = time.time()
text_vectorizer.adapt(text_data.batch(64))
execution_time = (time.time() - start_time)/60.0
print("Execution time (mins)",execution_time)

# Get Vocabulary
vocabulary = text_vectorizer.get_vocabulary()
vocabulary_size = len(vocabulary)
print("Vocabulary Size:",vocabulary_size)
# Generate word index
word_index = dict(zip(vocabulary, range(vocabulary_size)))Execution time (mins) 0.010191897551218668
Vocabulary Size: 15000
# Check Vocabulary : 0 is reserved for padding and index 1 is reserved for "out of vocabulary" tokens
print("Vocabulary:",vocabulary[:20])
print("Vocabulary Size:",len(vocabulary))

# Test our text vectorizer
test_text = txt = io_ops.read_file(data_x[data_samples[0]])
print(test_text)
test_text_vector = text_vectorizer([test_text.numpy()])
print("Shape:",test_text_vector.shape)
print(test_text_vector[0,:20])Vocabulary: ['', '[UNK]', 'the', 'to', 'and', 'of', 'a', 'in', 'that', 'for', 'is', 'on', 'said', 'with', 'as', 'it', 'have', 'are', 'be', 'at']
Vocabulary Size: 15000
tf.Tensor(b'Kayleigh McEnany just made things worse  (CNN)In the wake of President Donald Trump equivocating as to whether he would transfer power to Joe Biden in the event of a defeat at the ballot box in 40 days, White House press secretary Kayleigh McEnany tried to clear things up on Thursday. Spoiler alert: She didn\'t! "The President will accept the results of a free and fair election," said McEnany. Which, well, that ain\'t it. See, Trump has said -- as recently as Wednesday night-- that the only way the election will be fair is if we "get rid of the ballots and you\'ll have a very trans- -- we\'ll have a very peaceful -- there won\'t be a transfer, frankly; there\'ll be a continuation." So, by that standard, the only way that the election will be "free and fair" (in McEnany\'s words) is if all mail-in ballots -- many of which have already been sent and returned! -- are somehow disqualified (or something?). Which, because every poll suggests that Democratic voters are more inclined to use mail-in balloting amid the coronavirus pandemic than GOP voters, would hand Trump a major advantage. Not to mention disenfranchise lots and lots of people who are eligible voters! See, the thing is that neither Trump nor McEnany get to define "free and fair." Those words have objective meanings apart from any spin. They mean that everyone who is eligible to vote is given an equal opportunity to cast a vote -- in whatever means are legally available to them in their state. The Point: Trump is a president, not a dictator. He doesn\'t get to decide how the election rules work. He just plays within them. He -- and McEnany -- still don\'t seem to grasp that concept of, uh, how democracy works. CNN\'s Allison Gordon contributed to this report ', shape=(), dtype=string)
Shape: (1, 1000)
tf.Tensor(
[3695 1860   81  181  278 1086 5190    2 1847    5   60  237   32    1
   14    3  235   21   61 2926], shape=(20,), dtype=int64)

##### Split Data_____no_output_____

validation_percent = 0.20

# Split data into train / validate
train_x, validate_x, train_y, validate_y = train_test_split(data_x, data_y, test_size=validation_percent)

print("train_x count:",len(train_x))
print("validate_x count:",len(validate_x))train_x count: 736
validate_x count: 184

##### Create TF Dataset_____no_output_____

batch_size = 64

# Convert all y labels to numbers
train_processed_y = [label2index[label] for label in train_y]
validate_processed_y = [label2index[label] for label in validate_y]

# Converts to y to binary class matrix (One-hot-encoded)
train_processed_y = to_categorical(train_processed_y, num_classes=num_classes, dtype='float32')
validate_processed_y = to_categorical(validate_processed_y, num_classes=num_classes, dtype='float32')

# Vectorize Text
def vectorize_text(text, label=None):
  text = tf.expand_dims(text, -1)
  text = text_vectorizer(text)
  if label is None:
    return text
  else:
    return text, label

# Create TF Dataset
train_data = tf.data.Dataset.from_tensor_slices((train_x, train_processed_y))
validation_data = tf.data.Dataset.from_tensor_slices((validate_x, validate_processed_y))

#############
# Train data
#############
# Apply all data processing logic
train_data = train_data.map(load_text, num_parallel_calls=AUTOTUNE)
train_data = train_data.batch(batch_size)
train_data = train_data.map(vectorize_text, num_parallel_calls=AUTOTUNE)
train_data = train_data.cache().prefetch(buffer_size=100)

##################
# Validation data
##################
# Apply all data processing logic
validation_data = validation_data.map(load_text, num_parallel_calls=AUTOTUNE)
validation_data = validation_data.batch(batch_size)
validation_data = validation_data.map(vectorize_text, num_parallel_calls=AUTOTUNE)
validation_data = validation_data.cache().prefetch(buffer_size=100)

print("train_data",train_data)
print("validation_data",validation_data)train_data <PrefetchDataset shapes: ((None, 1000), (None, 3)), types: (tf.int64, tf.float32)>
validation_data <PrefetchDataset shapes: ((None, 1000), (None, 3)), types: (tf.int64, tf.float32)>

## Questions:

**All data preparation steps have been performed and you can continue to building models in following questions**

Note on dataset input sizes:
```
# Feature constraints
max_features = 15000
sequence_length = 1000
```_____no_output_____

# Taking which Semantic Segmentation with Transfer Learning Demo
def evaluate_model(model,test_data, training_results):

  # Get the model train history
  model_train_history = training_results.history
  # Get the number of epochs the training was run for
  num_epochs = len(model_train_history["loss"])

  # Plot training results
  fig = plt.figure(figsize=(15,5))
  axs = fig.add_subplot(1,2,1)
  axs.set_title('Loss')
  # Plot all metrics
  for metric in ["loss","val_loss"]:
      axs.plot(np.arange(0, num_epochs), model_train_history[metric], label=metric)
  axs.legend()

  axs = fig.add_subplot(1,2,2)
  axs.set_title('Accuracy')
  # Plot all metrics
  for metric in ["accuracy","val_accuracy"]:
      axs.plot(np.arange(0, num_epochs), model_train_history[metric], label=metric)
  axs.legend()

  plt.show()

  # Evaluate on test data
  evaluation_results = model.evaluate(test_data)
  print("Evaluation Results:", evaluation_results)_____no_output_____

## Question 1 : Build a text classification model using FFNN (0.5 Point)_____no_output_____#### a) Build a Text Classification Model

- Build the model using a few `Dense` layers
- Input size is `1000`
- Do **NOT** use the `Embedding` layer in your model
- Use `categorical_crossentropy` loss
- Ensure there is a plot of your training history_____no_output_____

model = Sequential()
model.add(layers.Dense(256, input_shape=(1000,), activation='relu'))
model.add(layers.Dense(256, activation='relu'))
model.add(layers.Dense(3, activation='softmax'))
_____no_output_____model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
start_time = time.time()
training_results = model.fit(
        train_data,
        validation_data=validation_data,
        epochs=50,
        verbose=1)
execution_time = (time.time() - start_time)/60.0
print("Training execution time (mins)",execution_time)

# Evaluate model
evaluate_model(model,validation_data, training_results)Epoch 1/50
12/12 [==============================] - 0s 22ms/step - loss: 847.2442 - accuracy: 0.4511 - val_loss: 477.1715 - val_accuracy: 0.4620
Epoch 2/50
12/12 [==============================] - 0s 5ms/step - loss: 136.2894 - accuracy: 0.7527 - val_loss: 417.3380 - val_accuracy: 0.4837
Epoch 3/50
12/12 [==============================] - 0s 5ms/step - loss: 140.8354 - accuracy: 0.7867 - val_loss: 697.5585 - val_accuracy: 0.4565
Epoch 4/50
12/12 [==============================] - 0s 5ms/step - loss: 48.8420 - accuracy: 0.9049 - val_loss: 312.5107 - val_accuracy: 0.5380
Epoch 5/50
12/12 [==============================] - 0s 5ms/step - loss: 8.6292 - accuracy: 0.9552 - val_loss: 305.1111 - val_accuracy: 0.5272
Epoch 6/50
12/12 [==============================] - 0s 5ms/step - loss: 0.6073 - accuracy: 0.9864 - val_loss: 345.3806 - val_accuracy: 0.5435
Epoch 7/50
12/12 [==============================] - 0s 5ms/step - loss: 0.2250 - accuracy: 0.9932 - val_loss: 351.0025 - val_accuracy: 0.5598
Epoch 8/50
12/12 [==============================] - 0s 5ms/step - loss: 0.2356 - accuracy: 0.9946 - val_loss: 350.1949 - val_accuracy: 0.5598
Epoch 9/50
12/12 [==============================] - 0s 5ms/step - loss: 0.1744 - accuracy: 0.9946 - val_loss: 352.3131 - val_accuracy: 0.5543
Epoch 10/50
12/12 [==============================] - 0s 4ms/step - loss: 0.1504 - accuracy: 0.9946 - val_loss: 353.8284 - val_accuracy: 0.5598
Epoch 11/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0981 - accuracy: 0.9959 - val_loss: 354.4308 - val_accuracy: 0.5598
Epoch 12/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0897 - accuracy: 0.9959 - val_loss: 354.6855 - val_accuracy: 0.5598
Epoch 13/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0806 - accuracy: 0.9959 - val_loss: 354.7779 - val_accuracy: 0.5598
Epoch 14/50
12/12 [==============================] - 0s 6ms/step - loss: 0.0709 - accuracy: 0.9959 - val_loss: 354.8113 - val_accuracy: 0.5598
Epoch 15/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0608 - accuracy: 0.9959 - val_loss: 354.8514 - val_accuracy: 0.5543
Epoch 16/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0516 - accuracy: 0.9973 - val_loss: 354.8846 - val_accuracy: 0.5543
Epoch 17/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0467 - accuracy: 0.9973 - val_loss: 354.8484 - val_accuracy: 0.5543
Epoch 18/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0428 - accuracy: 0.9973 - val_loss: 354.7548 - val_accuracy: 0.5489
Epoch 19/50
12/12 [==============================] - 0s 4ms/step - loss: 0.0389 - accuracy: 0.9973 - val_loss: 354.6527 - val_accuracy: 0.5489
Epoch 20/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0350 - accuracy: 0.9973 - val_loss: 354.5614 - val_accuracy: 0.5489
Epoch 21/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0310 - accuracy: 0.9973 - val_loss: 354.4809 - val_accuracy: 0.5489
Epoch 22/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0269 - accuracy: 0.9973 - val_loss: 354.4048 - val_accuracy: 0.5489
Epoch 23/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0230 - accuracy: 0.9973 - val_loss: 354.3346 - val_accuracy: 0.5489
Epoch 24/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0190 - accuracy: 0.9973 - val_loss: 354.2786 - val_accuracy: 0.5489
Epoch 25/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0150 - accuracy: 0.9973 - val_loss: 354.2482 - val_accuracy: 0.5435
Epoch 26/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0120 - accuracy: 0.9986 - val_loss: 354.2411 - val_accuracy: 0.5435
Epoch 27/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0112 - accuracy: 0.9986 - val_loss: 354.2431 - val_accuracy: 0.5435
Epoch 28/50
12/12 [==============================] - 0s 6ms/step - loss: 0.0110 - accuracy: 0.9986 - val_loss: 354.2494 - val_accuracy: 0.5435
Epoch 29/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0108 - accuracy: 0.9986 - val_loss: 354.2578 - val_accuracy: 0.5435
Epoch 30/50
12/12 [==============================] - 0s 4ms/step - loss: 0.0107 - accuracy: 0.9986 - val_loss: 354.2645 - val_accuracy: 0.5435
Epoch 31/50
12/12 [==============================] - 0s 4ms/step - loss: 0.0106 - accuracy: 0.9986 - val_loss: 354.2696 - val_accuracy: 0.5435
Epoch 32/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0105 - accuracy: 0.9986 - val_loss: 354.2735 - val_accuracy: 0.5435
Epoch 33/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0104 - accuracy: 0.9986 - val_loss: 354.2767 - val_accuracy: 0.5435
Epoch 34/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0103 - accuracy: 0.9986 - val_loss: 354.2792 - val_accuracy: 0.5435
Epoch 35/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0102 - accuracy: 0.9986 - val_loss: 354.2814 - val_accuracy: 0.5435
Epoch 36/50
12/12 [==============================] - 0s 4ms/step - loss: 0.0101 - accuracy: 0.9986 - val_loss: 354.2832 - val_accuracy: 0.5435
Epoch 37/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0100 - accuracy: 0.9986 - val_loss: 354.2848 - val_accuracy: 0.5435
Epoch 38/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0100 - accuracy: 0.9986 - val_loss: 354.2859 - val_accuracy: 0.5435
Epoch 39/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0099 - accuracy: 0.9986 - val_loss: 354.2871 - val_accuracy: 0.5435
Epoch 40/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0098 - accuracy: 0.9986 - val_loss: 354.2881 - val_accuracy: 0.5435
Epoch 41/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0097 - accuracy: 0.9986 - val_loss: 354.2888 - val_accuracy: 0.5435
Epoch 42/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0096 - accuracy: 0.9986 - val_loss: 354.2894 - val_accuracy: 0.5435
Epoch 43/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0095 - accuracy: 0.9986 - val_loss: 354.2899 - val_accuracy: 0.5435
Epoch 44/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0094 - accuracy: 0.9986 - val_loss: 354.2903 - val_accuracy: 0.5435
Epoch 45/50
12/12 [==============================] - 0s 4ms/step - loss: 0.0093 - accuracy: 0.9986 - val_loss: 354.2906 - val_accuracy: 0.5435
Epoch 46/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0093 - accuracy: 0.9986 - val_loss: 354.2906 - val_accuracy: 0.5435
Epoch 47/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0092 - accuracy: 0.9986 - val_loss: 354.2907 - val_accuracy: 0.5435
Epoch 48/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0091 - accuracy: 0.9986 - val_loss: 354.2907 - val_accuracy: 0.5435
Epoch 49/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0090 - accuracy: 0.9986 - val_loss: 354.2906 - val_accuracy: 0.5435
Epoch 50/50
12/12 [==============================] - 0s 5ms/step - loss: 0.0089 - accuracy: 0.9986 - val_loss: 354.2904 - val_accuracy: 0.5435
Training execution time (mins) 0.08159986337025961

#### b) How is the performance of your model?_____no_output_____Honestly, bad. It looks like we're overfitting on the training data and not really learning much about sentences, as our validation accuracy is under 50%, which is marginally better than just picking a random class.

_____no_output_____## Question 2 : Build a text classification model with embeddings (1.0 Point)_____no_output_____#### a) Preliminary Questions

- Explain what is the purpose of the `Embedding` Layer ?
- Name two reasons why using one-hot encoded vectors instead is not the way to go
- Explain what the inputs and outputs of the `Embedding` Layer are. Also comment on the dimension going in and what is coming out
_____no_output_____The embeding layer creates vectors in a specified dimensional space that represent words. This is a huge improvement over one-hot encodings because such encodings have no notion of similarity between two words, so the words "cold", "cool", and "hot" are all equally similar, wheras by creating vectors for each word and learning them through our model we can establish a notion of similarity.
The embedding layer takes the number of embeddings (number of words in your dataset) and the dimensional space of the embedding, and will then output these vectors. So in my case, with each sentence of length 1000 and a 16 dimensional space, my output wil lbe 1000x16_____no_output_____#### b) Build a Text Classification Model

- You need to include the `Embedding` layer in your model
- Following the `Embedding` layer you can use `Conv1D` or `LSTM`
- Use loss of `categorical_crossentropy`
- Ensure there is a plot of your training history

_____no_output_____

model = Sequential()
model.add(layers.Embedding(max_features,128,  input_shape=(1000,)))
model.add(layers.LSTM(200))
model.add(layers.Dense(3, activation='softmax'))
model.summary()Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding (Embedding)        (None, 1000, 128)         1920000
_________________________________________________________________
lstm (LSTM)                  (None, 200)               263200
_________________________________________________________________
dense_3 (Dense)              (None, 3)                 603
=================================================================
Total params: 2,183,803
Trainable params: 2,183,803
Non-trainable params: 0
_________________________________________________________________
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
start_time = time.time()
training_results = model.fit(
        train_data,
        validation_data=validation_data,
        epochs=50,
        verbose=1)
execution_time = (time.time() - start_time)/60.0
print("Training execution time (mins)",execution_time)

# Evaluate model
evaluate_model(model,validation_data, training_results)Epoch 1/50
12/12 [==============================] - 2s 145ms/step - loss: 1.0741 - accuracy: 0.4484 - val_loss: 1.0396 - val_accuracy: 0.4946
Epoch 2/50
12/12 [==============================] - 1s 104ms/step - loss: 1.0299 - accuracy: 0.4878 - val_loss: 1.0337 - val_accuracy: 0.4946
Epoch 3/50
12/12 [==============================] - 1s 106ms/step - loss: 1.0030 - accuracy: 0.4864 - val_loss: 1.0166 - val_accuracy: 0.4946
Epoch 4/50
12/12 [==============================] - 1s 104ms/step - loss: 0.9602 - accuracy: 0.4891 - val_loss: 1.0351 - val_accuracy: 0.4946
Epoch 5/50
12/12 [==============================] - 1s 105ms/step - loss: 0.9078 - accuracy: 0.5149 - val_loss: 1.1430 - val_accuracy: 0.5163
Epoch 6/50
12/12 [==============================] - 1s 107ms/step - loss: 0.8756 - accuracy: 0.5747 - val_loss: 1.0013 - val_accuracy: 0.5272
Epoch 7/50
12/12 [==============================] - 1s 106ms/step - loss: 0.8688 - accuracy: 0.5707 - val_loss: 1.0030 - val_accuracy: 0.5109
Epoch 8/50
12/12 [==============================] - 1s 106ms/step - loss: 0.8758 - accuracy: 0.5734 - val_loss: 1.0606 - val_accuracy: 0.5054
Epoch 9/50
12/12 [==============================] - 1s 107ms/step - loss: 0.8293 - accuracy: 0.5734 - val_loss: 1.0248 - val_accuracy: 0.5489
Epoch 10/50
12/12 [==============================] - 1s 104ms/step - loss: 0.8066 - accuracy: 0.5774 - val_loss: 0.9923 - val_accuracy: 0.5652
Epoch 11/50
12/12 [==============================] - 1s 107ms/step - loss: 0.7933 - accuracy: 0.5815 - val_loss: 1.0423 - val_accuracy: 0.5924
Epoch 12/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7954 - accuracy: 0.5883 - val_loss: 0.9946 - val_accuracy: 0.5217
Epoch 13/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7858 - accuracy: 0.5924 - val_loss: 1.0265 - val_accuracy: 0.5652
Epoch 14/50
12/12 [==============================] - 1s 107ms/step - loss: 0.7795 - accuracy: 0.5938 - val_loss: 0.9494 - val_accuracy: 0.6087
Epoch 15/50
12/12 [==============================] - 1s 107ms/step - loss: 0.8190 - accuracy: 0.5883 - val_loss: 1.0191 - val_accuracy: 0.5380
Epoch 16/50
12/12 [==============================] - 1s 107ms/step - loss: 0.7912 - accuracy: 0.5924 - val_loss: 1.0441 - val_accuracy: 0.5380
Epoch 17/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7830 - accuracy: 0.5924 - val_loss: 1.0861 - val_accuracy: 0.5489
Epoch 18/50
12/12 [==============================] - 1s 107ms/step - loss: 0.7740 - accuracy: 0.5965 - val_loss: 1.0592 - val_accuracy: 0.5815
Epoch 19/50
12/12 [==============================] - 1s 108ms/step - loss: 0.7778 - accuracy: 0.5978 - val_loss: 1.0702 - val_accuracy: 0.5000
Epoch 20/50
12/12 [==============================] - 1s 108ms/step - loss: 0.7915 - accuracy: 0.5910 - val_loss: 1.0620 - val_accuracy: 0.5272
Epoch 21/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7755 - accuracy: 0.5924 - val_loss: 1.0507 - val_accuracy: 0.5707
Epoch 22/50
12/12 [==============================] - 1s 107ms/step - loss: 0.7682 - accuracy: 0.6046 - val_loss: 1.0630 - val_accuracy: 0.5815
Epoch 23/50
12/12 [==============================] - 1s 107ms/step - loss: 0.7670 - accuracy: 0.6033 - val_loss: 1.1203 - val_accuracy: 0.5761
Epoch 24/50
12/12 [==============================] - 1s 107ms/step - loss: 0.7694 - accuracy: 0.6019 - val_loss: 1.0792 - val_accuracy: 0.5598
Epoch 25/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7633 - accuracy: 0.6060 - val_loss: 1.1246 - val_accuracy: 0.5870
Epoch 26/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7698 - accuracy: 0.6019 - val_loss: 1.0459 - val_accuracy: 0.5435
Epoch 27/50
12/12 [==============================] - 1s 105ms/step - loss: 0.7734 - accuracy: 0.6019 - val_loss: 1.0255 - val_accuracy: 0.5761
Epoch 28/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7675 - accuracy: 0.6005 - val_loss: 1.0325 - val_accuracy: 0.5978
Epoch 29/50
12/12 [==============================] - 1s 104ms/step - loss: 0.7636 - accuracy: 0.6005 - val_loss: 1.0188 - val_accuracy: 0.6033
Epoch 30/50
12/12 [==============================] - 1s 105ms/step - loss: 0.8604 - accuracy: 0.6073 - val_loss: 1.0496 - val_accuracy: 0.5326
Epoch 31/50
12/12 [==============================] - 1s 105ms/step - loss: 0.7871 - accuracy: 0.5924 - val_loss: 1.0467 - val_accuracy: 0.5272
Epoch 32/50
12/12 [==============================] - 1s 104ms/step - loss: 0.7822 - accuracy: 0.5924 - val_loss: 1.0020 - val_accuracy: 0.5489
Epoch 33/50
12/12 [==============================] - 1s 106ms/step - loss: 1.0090 - accuracy: 0.5693 - val_loss: 1.2930 - val_accuracy: 0.4837
Epoch 34/50
12/12 [==============================] - 1s 105ms/step - loss: 0.8296 - accuracy: 0.5856 - val_loss: 1.0843 - val_accuracy: 0.5000
Epoch 35/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7923 - accuracy: 0.5910 - val_loss: 1.0274 - val_accuracy: 0.5435
Epoch 36/50
12/12 [==============================] - 1s 104ms/step - loss: 0.7687 - accuracy: 0.5992 - val_loss: 1.1313 - val_accuracy: 0.5707
Epoch 37/50
12/12 [==============================] - 1s 104ms/step - loss: 0.7641 - accuracy: 0.6046 - val_loss: 1.0471 - val_accuracy: 0.5761
Epoch 38/50
12/12 [==============================] - 1s 107ms/step - loss: 0.8013 - accuracy: 0.6196 - val_loss: 1.0683 - val_accuracy: 0.5652
Epoch 39/50
12/12 [==============================] - 1s 104ms/step - loss: 0.8230 - accuracy: 0.5897 - val_loss: 1.0932 - val_accuracy: 0.5000
Epoch 40/50
12/12 [==============================] - 1s 104ms/step - loss: 0.8466 - accuracy: 0.5856 - val_loss: 1.0557 - val_accuracy: 0.5109
Epoch 41/50
12/12 [==============================] - 1s 105ms/step - loss: 0.8103 - accuracy: 0.5870 - val_loss: 1.0474 - val_accuracy: 0.5163
Epoch 42/50
12/12 [==============================] - 1s 105ms/step - loss: 0.7951 - accuracy: 0.5870 - val_loss: 1.0433 - val_accuracy: 0.5109
Epoch 43/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7885 - accuracy: 0.5870 - val_loss: 1.0517 - val_accuracy: 0.5109
Epoch 44/50
12/12 [==============================] - 1s 108ms/step - loss: 0.7846 - accuracy: 0.5883 - val_loss: 1.0630 - val_accuracy: 0.5163
Epoch 45/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7826 - accuracy: 0.5897 - val_loss: 1.0737 - val_accuracy: 0.5163
Epoch 46/50
12/12 [==============================] - 1s 106ms/step - loss: 0.7809 - accuracy: 0.5897 - val_loss: 1.0857 - val_accuracy: 0.5163
Epoch 47/50
12/12 [==============================] - 1s 104ms/step - loss: 0.7793 - accuracy: 0.5897 - val_loss: 1.0978 - val_accuracy: 0.5109
Epoch 48/50
12/12 [==============================] - 1s 105ms/step - loss: 0.7781 - accuracy: 0.5924 - val_loss: 1.1097 - val_accuracy: 0.5109
Epoch 49/50
12/12 [==============================] - 1s 105ms/step - loss: 0.7769 - accuracy: 0.5924 - val_loss: 1.1214 - val_accuracy: 0.5163
Epoch 50/50
12/12 [==============================] - 1s 104ms/step - loss: 0.7758 - accuracy: 0.5924 - val_loss: 1.1328 - val_accuracy: 0.5272
Training execution time (mins) 1.2198123335838318

#### c) Save Embedding layer weights

- Save the weights of your embedding layer
- The weights will be used in Question 4
- Feel free to use the code below to extract layer weights

```
# Get the Embedding Layer
embedding_layer_no_tl = model.get_layer(name="embedding")
embedding_layer_no_tl_weights = embedding_layer_no_tl.get_weights()[0]
print(embedding_layer_no_tl_weights.shape)
```_____no_output_____

model.summary()Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding (Embedding)        (None, 1000, 128)         1920000
_________________________________________________________________
lstm (LSTM)                  (None, 200)               263200
_________________________________________________________________
dense_3 (Dense)              (None, 3)                 603
=================================================================
Total params: 2,183,803
Trainable params: 2,183,803
Non-trainable params: 0
_________________________________________________________________
embedding_layer_no_tl = model.get_layer(name="embedding")
embedding_layer_no_tl_weights = embedding_layer_no_tl.get_weights()[0]
print(embedding_layer_no_tl_weights.shape)
(15000, 128)

---

## Question 3 : Build a text classification model with pre-trained embeddings (2.5 Point)_____no_output_____#### a) Preliminary Questions

- How is using pre-trained word embedding adding knowledge to the model? (Use an example if needed)

- Provide a scenario where retraining the pre-trained embedding layer may be needed_____no_output_____In my above model, i'm relying on the backpropogation of my classification to learn the embeddings of each word. That's a pretty difficult task, especially considering all that we have to learn off is the classification of each sentence. However, there is a certain element of situaltional context in which these embeddings would need to be retrained. For instance if you had vectors that were trained on academic papers, but you then wanted to analyze facebook post data. Inevitably there would be new things to glean from the words on facebook that would have not been discovered in papers.
_____no_output_____#### b) Build a Text Classifcation Model using pre-trained word embeddings

- Build a Text Classification Model
- You need to include the `Embedding` layer in your model
- The `Embedding` layer should have its weights loaded from any **pre-trained word embeddings** such as Glove, Word2Vec, FastText etc.
- [Example](https://medium.com/@ppasumarthi_69210/word-embeddings-in-keras-be6bb3092831) of how to load pre-trained word embeddings for Word2Vec
- [Example](https://keras.io/examples/nlp/pretrained_word_embeddings/) of how to load pre-trained word embeddings for Glove
- [Example](https://www.kaggle.com/vsmolyakov/keras-cnn-with-fasttext-embeddings) of how to load pre-trained word embeddings for FastText
- Make the `Embedding` layer as `trainable=False`
- Following the `Embedding` layer you can use `Conv1D` or `LSTM`
- Use loss of `categorical_crossentropy`
- Ensure there is a plot of your training history_____no_output_____

# # !wget http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip
# !unzip glove.6B.zip
# # _____no_output_____# # load the whole embedding into memory
from tqdm.auto import tqdm
embeddings_index = dict()
f = open('glove.6B.100d.txt')
for line in f:
 values = line.split()
 word = values[0]
 coefs = np.asarray(values[1:], dtype='float32')
 embeddings_index[word] = coefs
f.close()
# create a weight matrix for words in training docs
embedding_matrix = np.zeros((max_features, 100))
for word, i in tqdm(word_index.items()):
  embedding_vector = embeddings_index.get(word)
  if embedding_vector is not None:
    embedding_matrix[i] = embedding_vector
np.save('embedding_matrix',embedding_matrix)_____no_output_____embedding_matrix.shape_____no_output_____model = Sequential()
model.add(layers.Embedding(max_features, 100, weights=[embedding_matrix], trainable=False))
model.add(layers.LSTM(200))
model.add(layers.Dense(3, activation='softmax'))
model.summary()Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding_1 (Embedding)      (None, None, 100)         1500000
_________________________________________________________________
lstm_1 (LSTM)                (None, 200)               240800
_________________________________________________________________
dense_4 (Dense)              (None, 3)                 603
=================================================================
Total params: 1,741,403
Trainable params: 241,403
Non-trainable params: 1,500,000
_________________________________________________________________
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
start_time = time.time()
training_results = model.fit(
        train_data,
        validation_data=validation_data,
        epochs=50,
        verbose=1)
execution_time = (time.time() - start_time)/60.0
print("Training execution time (mins)",execution_time)

# Evaluate model
evaluate_model(model,validation_data, training_results)Epoch 1/50
12/12 [==============================] - 1s 119ms/step - loss: 1.0352 - accuracy: 0.4701 - val_loss: 1.0163 - val_accuracy: 0.5217
Epoch 2/50
12/12 [==============================] - 1s 82ms/step - loss: 0.9526 - accuracy: 0.5217 - val_loss: 0.9620 - val_accuracy: 0.5815
Epoch 3/50
12/12 [==============================] - 1s 84ms/step - loss: 0.9330 - accuracy: 0.5272 - val_loss: 0.9672 - val_accuracy: 0.5815
Epoch 4/50
12/12 [==============================] - 1s 83ms/step - loss: 0.9011 - accuracy: 0.5448 - val_loss: 0.9041 - val_accuracy: 0.5870
Epoch 5/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9930 - accuracy: 0.5014 - val_loss: 0.9830 - val_accuracy: 0.5435
Epoch 6/50
12/12 [==============================] - 1s 83ms/step - loss: 0.9630 - accuracy: 0.5027 - val_loss: 0.9901 - val_accuracy: 0.5109
Epoch 7/50
12/12 [==============================] - 1s 83ms/step - loss: 0.9200 - accuracy: 0.5258 - val_loss: 0.9311 - val_accuracy: 0.5815
Epoch 8/50
12/12 [==============================] - 1s 84ms/step - loss: 0.9006 - accuracy: 0.5734 - val_loss: 1.1146 - val_accuracy: 0.6630
Epoch 9/50
12/12 [==============================] - 1s 85ms/step - loss: 0.8102 - accuracy: 0.6454 - val_loss: 0.7335 - val_accuracy: 0.5652
Epoch 10/50
12/12 [==============================] - 1s 85ms/step - loss: 0.7951 - accuracy: 0.5788 - val_loss: 0.8412 - val_accuracy: 0.5435
Epoch 11/50
12/12 [==============================] - 1s 85ms/step - loss: 0.7988 - accuracy: 0.5734 - val_loss: 0.7177 - val_accuracy: 0.6848
Epoch 12/50
12/12 [==============================] - 1s 85ms/step - loss: 1.0402 - accuracy: 0.4647 - val_loss: 0.9860 - val_accuracy: 0.5380
Epoch 13/50
12/12 [==============================] - 1s 85ms/step - loss: 1.0030 - accuracy: 0.5041 - val_loss: 0.9822 - val_accuracy: 0.5272
Epoch 14/50
12/12 [==============================] - 1s 84ms/step - loss: 0.9678 - accuracy: 0.5109 - val_loss: 0.9869 - val_accuracy: 0.5435
Epoch 15/50
12/12 [==============================] - 1s 84ms/step - loss: 0.9442 - accuracy: 0.5312 - val_loss: 0.9645 - val_accuracy: 0.5598
Epoch 16/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9367 - accuracy: 0.5516 - val_loss: 0.9723 - val_accuracy: 0.5543
Epoch 17/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9304 - accuracy: 0.5353 - val_loss: 0.9609 - val_accuracy: 0.5543
Epoch 18/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9199 - accuracy: 0.5530 - val_loss: 0.9401 - val_accuracy: 0.5815
Epoch 19/50
12/12 [==============================] - 1s 85ms/step - loss: 0.8999 - accuracy: 0.5598 - val_loss: 0.9260 - val_accuracy: 0.5924
Epoch 20/50
12/12 [==============================] - 1s 86ms/step - loss: 0.9086 - accuracy: 0.5448 - val_loss: 1.0450 - val_accuracy: 0.5489
Epoch 21/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9398 - accuracy: 0.5421 - val_loss: 0.9614 - val_accuracy: 0.5761
Epoch 22/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9254 - accuracy: 0.5435 - val_loss: 0.9441 - val_accuracy: 0.6033
Epoch 23/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9180 - accuracy: 0.5421 - val_loss: 0.9677 - val_accuracy: 0.5435
Epoch 24/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9057 - accuracy: 0.5163 - val_loss: 0.9405 - val_accuracy: 0.5870
Epoch 25/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9018 - accuracy: 0.5462 - val_loss: 0.9595 - val_accuracy: 0.5652
Epoch 26/50
12/12 [==============================] - 1s 84ms/step - loss: 0.9234 - accuracy: 0.5340 - val_loss: 0.9635 - val_accuracy: 0.5380
Epoch 27/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9031 - accuracy: 0.5598 - val_loss: 0.9417 - val_accuracy: 0.5761
Epoch 28/50
12/12 [==============================] - 1s 85ms/step - loss: 0.8765 - accuracy: 0.5625 - val_loss: 0.9218 - val_accuracy: 0.5978
Epoch 29/50
12/12 [==============================] - 1s 85ms/step - loss: 0.8464 - accuracy: 0.5951 - val_loss: 0.8549 - val_accuracy: 0.6522
Epoch 30/50
12/12 [==============================] - 1s 85ms/step - loss: 0.8267 - accuracy: 0.5992 - val_loss: 0.8211 - val_accuracy: 0.6304
Epoch 31/50
12/12 [==============================] - 1s 85ms/step - loss: 0.7970 - accuracy: 0.6522 - val_loss: 0.7492 - val_accuracy: 0.5870
Epoch 32/50
12/12 [==============================] - 1s 86ms/step - loss: 0.7548 - accuracy: 0.5992 - val_loss: 0.8069 - val_accuracy: 0.5380
Epoch 33/50
12/12 [==============================] - 1s 85ms/step - loss: 0.7098 - accuracy: 0.6345 - val_loss: 0.8979 - val_accuracy: 0.5109
Epoch 34/50
12/12 [==============================] - 1s 85ms/step - loss: 0.8861 - accuracy: 0.5489 - val_loss: 0.7214 - val_accuracy: 0.6304
Epoch 35/50
12/12 [==============================] - 1s 85ms/step - loss: 0.7152 - accuracy: 0.6318 - val_loss: 0.6632 - val_accuracy: 0.6630
Epoch 36/50
12/12 [==============================] - 1s 85ms/step - loss: 0.7299 - accuracy: 0.6128 - val_loss: 1.1450 - val_accuracy: 0.4891
Epoch 37/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9789 - accuracy: 0.5299 - val_loss: 0.9869 - val_accuracy: 0.5761
Epoch 38/50
12/12 [==============================] - 1s 84ms/step - loss: 0.8621 - accuracy: 0.5693 - val_loss: 0.8427 - val_accuracy: 0.6250
Epoch 39/50
12/12 [==============================] - 1s 85ms/step - loss: 0.8160 - accuracy: 0.6101 - val_loss: 1.0583 - val_accuracy: 0.4837
Epoch 40/50
12/12 [==============================] - 1s 84ms/step - loss: 0.9550 - accuracy: 0.5326 - val_loss: 0.9424 - val_accuracy: 0.5815
Epoch 41/50
12/12 [==============================] - 1s 86ms/step - loss: 0.8844 - accuracy: 0.5625 - val_loss: 0.9180 - val_accuracy: 0.5978
Epoch 42/50
12/12 [==============================] - 1s 85ms/step - loss: 0.8506 - accuracy: 0.5883 - val_loss: 0.8844 - val_accuracy: 0.6033
Epoch 43/50
12/12 [==============================] - 1s 84ms/step - loss: 0.8309 - accuracy: 0.6019 - val_loss: 0.8635 - val_accuracy: 0.6304
Epoch 44/50
12/12 [==============================] - 1s 84ms/step - loss: 0.7598 - accuracy: 0.6562 - val_loss: 0.7736 - val_accuracy: 0.7120
Epoch 45/50
12/12 [==============================] - 1s 85ms/step - loss: 0.6416 - accuracy: 0.7283 - val_loss: 0.7108 - val_accuracy: 0.7120
Epoch 46/50
12/12 [==============================] - 1s 84ms/step - loss: 0.5710 - accuracy: 0.7201 - val_loss: 0.6003 - val_accuracy: 0.7500
Epoch 47/50
12/12 [==============================] - 1s 85ms/step - loss: 0.9409 - accuracy: 0.5516 - val_loss: 1.0225 - val_accuracy: 0.5054
Epoch 48/50
12/12 [==============================] - 1s 85ms/step - loss: 0.8969 - accuracy: 0.6101 - val_loss: 0.7856 - val_accuracy: 0.6685
Epoch 49/50
12/12 [==============================] - 1s 85ms/step - loss: 0.7671 - accuracy: 0.6440 - val_loss: 0.7585 - val_accuracy: 0.6902
Epoch 50/50
12/12 [==============================] - 1s 85ms/step - loss: 0.6977 - accuracy: 0.6997 - val_loss: 0.6966 - val_accuracy: 0.7065
Training execution time (mins) 0.9391297022501628

#### c) Save Embedding layer weights

- Save the weights of your embedding layer
- The weights will be used in Question 4
- Feel free to use the code below to extract layer weights

```
# Get the Embedding Layer
embedding_layer_tl = model.get_layer(name="embedding")

embedding_layer_tl_weights = embedding_layer_tl.get_weights()[0]
print(embedding_layer_tl_weights.shape)
```_____no_output_____

embedding_layer_tl = model.get_layer(name="embedding_1")

embedding_layer_tl_weights = embedding_layer_tl.get_weights()[0]
print(embedding_layer_tl_weights.shape)(15000, 100)

---

## Question 4 : Analysing Word Embeddings (1.0 Point)_____no_output_____Feel free to use these functions for this question:

```
def find_similar(words, word_index, vocabulary, embedding_layer_weights, topn=5):
  subset_word_index  = []
  for word in words:
    subset_word_index.append(word_index[word])

  cs_op = cosine_similarity(embedding_layer_weights[subset_word_index], embedding_layer_weights)
  for idx in range(len(words)):
    top = cs_op[idx].argsort()[-topn:][::-1]
    for i,t in enumerate(top):
      if i ==0:
        print("Similar words for:",vocabulary[t])
      else:
        print("    ",vocabulary[t])

def find_analogy(word_a, word_b, word_c, word_index, vocabulary, embedding_layer_weights, topn=5):
  idx_a = word_index[word_a]
  idx_b = word_index[word_b]
  idx_c = word_index[word_c]

  # Vectors
  vec_a = embedding_layer_weights[idx_a]
  vec_b = embedding_layer_weights[idx_b]
  vec_c = embedding_layer_weights[idx_c]

  op = vec_b - vec_a + vec_c
  cs_op = cosine_similarity([op], embedding_layer_weights)
  top = cs_op[0].argsort()[-topn:][::-1]

  print(word_b,"-",word_a,"+", word_c, "=")
  for i,t in enumerate(top):
    print("   ",vocabulary[t])
```_____no_output_____

def find_similar(words, word_index, vocabulary, embedding_layer_weights, topn=5):
  subset_word_index  = []
  for word in words:
    subset_word_index.append(word_index[word])

  cs_op = cosine_similarity(embedding_layer_weights[subset_word_index], embedding_layer_weights)
  for idx in range(len(words)):
    print("Similar words for:",words[idx])
    top = cs_op[idx].argsort()[-topn-1:][::-1]
    top_words = [vocabulary[word] for word in top]
    try:
      top_words.remove(str(words[idx]))
    except ValueError:
      pass
    top_words = top_words[:5]
    for i,t in enumerate(top_words):
        print("    ", t)

def find_analogy(word_a, word_b, word_c, word_index, vocabulary, embedding_layer_weights, topn=5):
  idx_a = word_index[word_a]
  idx_b = word_index[word_b]
  idx_c = word_index[word_c]

  # Vectors
  vec_a = embedding_layer_weights[idx_a]
  vec_b = embedding_layer_weights[idx_b]
  vec_c = embedding_layer_weights[idx_c]

  op = vec_b - vec_a + vec_c
  cs_op = cosine_similarity([op], embedding_layer_weights)
  top = cs_op[0].argsort()[-topn:][::-1]

  print(word_b,"-",word_a,"+", word_c, "=")
  for i,t in enumerate(top):
    print("   ",vocabulary[t])_____no_output_____

#### a) Finding Semantically similar words

- We want to find words that are semantically similar to the following words: ```['covid19','election','2020','pandemic','quarantine']```
- Run the function `find_similar(...)` and display the results for embedding weights from question 2 and question 3_____no_output_____

words = ['covid19','election','2020','pandemic','quarantine']_____no_output_____embedding_layer_no_tl
find_similar(words, word_index, vocabulary, embedding_layer_no_tl_weights, topn=5)
Similar words for: covid19
     contact
     starting
     new
     from
     youre
Similar words for: election
     however
     statement
     giants
     mention
     late
Similar words for: 2020
     seeing
     available
     situations
     pandemic
     kinds
Similar words for: pandemic
     anxiety
     help
     successful
     tests
     lacerda
Similar words for: quarantine
     therapists
     passantino
     asthma
     bonifield
     opinions

find_similar(words, word_index, vocabulary, embedding_layer_tl_weights, topn=5)
Similar words for: covid19
     oldest
     steroid
     skeptical
     slowed
     solely
Similar words for: election
     elections
     vote
     polls
     electoral
     presidential
Similar words for: 2020
     2015
     2050
     2030
     2017
     emissions
Similar words for: pandemic
     influenza
     flu
     epidemic
     h1n1
     virus
Similar words for: quarantine
     inspection
     quarantined
     sars
     poultry
     precaution

- Explain your results, does the similar words have any real world similarity?
- Explain the results of the word `covid19` from learned embeddings weights vs pre-trained embeddings weights_____no_output_____I would say that generally both models performed pretty well. Given covid19 was not in the Glove embeddings word list, it makes sense that the results are pretty nonsensical. Whereas it seems that covid19 was in the training data, and hence the learned vectors do much better. I suppose this demonstrates how often it is important to retrain embeddings when using current text sources, as new words and phrases emerge and without some training we can't really do much to figure it out._____no_output_____#### b) Finding Analogies

- Word embeddings can be used to find analogies between words. For example, “man” is to “woman” as “son” is to “daughter” is an example of analogy,
- Let us verify the `male-female` analogy
- Run the function `find_analogy(...)` and display the results for embedding weights from question 2 and question 3_____no_output_____

#find_analogy('man', 'woman', 'son',...) for learned embeddings weights
find_analogy('man', 'woman', 'son', word_index, vocabulary, embedding_layer_no_tl_weights, topn=5)woman - man + son =
    woman
    candidates
    president
    presidency
    issue
find_analogy('man', 'woman', 'son', word_index, vocabulary, embedding_layer_tl_weights, topn=5)woman - man + son =
    daughter
    son
    wife
    mother
    father

- Explain your results, does word analogies work in both cases?_____no_output_____In this case, it seems that the training set responsible for my vectors is generally political coverage, and thus our representations of the worlds man woman and son are all pretty bad. Again, this i a testiment to the use of existing embeddings, as words that are unusual will hardly be learned without a huge dataset. Meanwhile, the glove embeddings do a really good job with the analogy, which makes sense as it was created with a much larger training set.

It's important to know the context of the questions you're trying to ask. Only if they're hyper-specific to your personal data does it really make sense to have your own embeddings created from scratch._____no_output_____