# Notebook from gsnyder206/notebooks
Path: notebooks/MAST/PanSTARRS/PS1_DR2_TAP/PS1_DR2_TAP.ipynb

# MAST Table Access Protocol PanSTARRS 1 DR2 Demo_____no_output_____
 This tutorial demonstrates how to use astroquery to access PanSTARRS 1 Data Release 2 via a Virtual Observatory standard Table Access Protocol (TAP) service at MAST, and work with the resultant data. It relies on Python 3 and astroquery, as well as some other common scientific packages.

***
### Table of Contents

1. [TAP Service Introduction](#TAP-Service-Introduction)
2. [Imports](#Imports)
3. [Connecting to a TAP Service](#Connecting-to-a-TAP-Service)
4. [Use Cases](#Use-Cases)
5. [Additional Resources](#Additional-Resources)
6. [About This Notebook](#About-this-Notebook)
***_____no_output_____## TAP Service Introduction_____no_output_____Table Access Protocol (TAP) services allow more direct and flexible access to astronomical data than the simpler types of IVOA standard data services. Queries are built with the SQL-like Astronomical Data Query Language (ADQL), and can include geographic / spatial queries as well as filtering on other characteristics of the data. This also allows the user fine-grained control over the returned columns, unlike the fixed set of coumns returned from cone, image, and spectral services.

For this example, we'll be using the astroquery TAP/TAP+ client, which was developed by the ESAC Space Data Centre for working with the GAIA catalog, but is interoperable with other valid TAP services, including those at MAST. As an astroquery project, TAP+ documentation is available at ReadTheDocs: http://astroquery.readthedocs.io/en/latest/utils/tap.html

We'll be using TAP+ to call the TAP service at MAST serving PanSTARRS 1 Data Release 2, now with individual detection information. The schema is described within the service, and we'll show how to inspect it. The schema is also the same as the one available via the CasJobs interface, with an additional view added for the most common positional queries. CasJobs has its own copy of the schema documentation, which can be accessed through its own site: http://mastweb.stsci.edu/ps1casjobs/

_____no_output_____***
## Imports_____no_output_____

# Use the astroquery TapPlus library as our client to the data service.
from astroquery.utils.tap.core import TapPlus

# For resolving objects with tools from MAST
from astroquery.mast import Mast

# For handling ordinary astropy Tables in responses
from astropy.table import Table

# For displaying and manipulating some types of results
%matplotlib inline
import requests
import astropy
import numpy as np
import pylab
import time
import json
from matplotlib import pyplot as plt

# For calling the object name resolver. Note these are Python 3 dependencies
#import sys
#import http.client as httplib
#from urllib.parse import quote as urlencode
#from urllib.request import urlretrieve

# To allow display tweaks for wider response tables
from IPython.core.display import display
from IPython.core.display import HTML_____no_output_____

***
## Connecting to a TAP Service_____no_output_____The TapPlus library is able to connect to any TAP service, given the "base" URL as noted in metadata registry resources describing the service. This is the URL for the PanSTARRS 1 DR2 TAP service._____no_output_____

TAP_service = TapPlus(url="http://vao.stsci.edu/PS1DR2/tapservice.aspx")_____no_output_____

### List available tables_____no_output_____

tables = TAP_service.load_tables(only_names=True)
for table in tables:
    print(table.name)_____no_output_____

# Use Cases_____no_output_____## Simple Positional Query

This searches the mean object catalog for objects within .2 degrees of M87 (RA=187.706, Dec=12.391 in degrees). The view used contains information from the [ObjectThin](https://outerspace.stsci.edu/x/W4Oc) table (which has information on object positions and the number of available measurements) and the [MeanObject](https://outerspace.stsci.edu/x/WYOc) table (which has information on photometry averaged over the multiple epochs of observation).

Note that the results are restricted to objects with `nDetections>1`, where `nDetections` is the total number of times the object was detected on the single-epoch images in any filter at any time.  Objects with `nDetections=1` tend to be  artifacts, so this is a quick way to eliminate most spurious objects from the catalog.

This query runs in TAP's asynchronous mode, which is a queued batch mode with some overhead and longer timeouts, useful for big catalogs like PanSTARRS. It may not be necessary for all queries to PS1 DR2, but the TAP+ client can automatically handle the additional processing required over synchronous mode._____no_output_____

job = TAP_service.launch_job_async("""
SELECT objID, RAMean, DecMean, nDetections, ng, nr, ni, nz, ny, gMeanPSFMag, rMeanPSFMag, iMeanPSFMag, zMeanPSFMag, yMeanPSFMag
FROM dbo.MeanObjectView
WHERE
CONTAINS(POINT('ICRS', RAMean, DecMean),CIRCLE('ICRS',187.706,12.391,.2))=1
AND nDetections > 1
  """)
TAP_results = job.get_results()
TAP_results_____no_output_____

## Get DR2 light curve for RR Lyrae star KQ UMa

This time we start with the object name, use the MAST name resolver (which relies on Simbad and NED) to convert the name to RA and Dec, and then query the PS1 DR2 mean object catalog at that position.  Then we run a spatial query to TAP using those coordinates._____no_output_____

objname = 'KQ UMa'
coords = Mast._resolve_object(objname)
ra,dec = coords.ra.value,coords.dec.value
radius = 1.0/3600.0 # radius = 1 arcsec

query = """
SELECT objID, RAMean, DecMean, nDetections, ng, nr, ni, nz, ny, gMeanPSFMag,
    rMeanPSFMag, iMeanPSFMag, zMeanPSFMag, yMeanPSFMag
FROM dbo.MeanObjectView
WHERE
CONTAINS(POINT('ICRS', RAMean, DecMean),CIRCLE('ICRS',{},{},{}))=1
AND nDetections > 1
""".format(ra,dec,radius)
print(query)

job = TAP_service.launch_job_async(query)
TAP_results = job.get_results()
TAP_results_____no_output_____

### Get Repeated Detection Information

Extract all the objects with the same object ID from the [Detection](https://outerspace.stsci.edu/x/b4Oc) table, which contains all the individual measurements for this source. The results are joined to the [Filter](https://outerspace.stsci.edu/x/nIOc) table to convert the filter numbers to names._____no_output_____

objid = TAP_results['objID'][0]
query = """
SELECT
    objID, detectID, Detection.filterID as filterID, Filter.filterType, obsTime, ra, dec,
    psfFlux, psfFluxErr, psfMajorFWHM, psfMinorFWHM, psfQfPerfect,
    apFlux, apFluxErr, infoFlag, infoFlag2, infoFlag3
FROM Detection
NATURAL JOIN Filter
WHERE objID={}
ORDER BY filterID, obsTime
""".format(objid)
print(query)

job = TAP_service.launch_job_async(query)
detection_TAP_results = job.get_results()
detection_TAP_results_____no_output_____

### Plot the light curves

The `psfFlux` values from the Detection table are converted from Janskys to AB magnitudes.  Measurements in the 5 different filters are plotted separately._____no_output_____

# convert flux in Jy to magnitudes
t = detection_TAP_results['obsTime']
mag = -2.5*np.log10(detection_TAP_results['psfFlux']) + 8.90
xlim = np.array([t.min(),t.max()])
xlim = xlim + np.array([-1,1])*0.02*(xlim[1]-xlim[0])

pylab.rcParams.update({'font.size': 14})
pylab.figure(1,(10,10))

#detection_TAP_results['filterType'] is a byte string, compare accordingly:
for i, filter in enumerate([b'g',b'r',b'i',b'z',b'y']):
    pylab.subplot(511+i)
    w = np.where(detection_TAP_results['filterType']==filter)
    pylab.plot(t[w],mag[w],'-o')
    pylab.ylabel(filter.decode('ascii')+' [mag]')
    pylab.xlim(xlim)
    pylab.gca().invert_yaxis()
    if i==0:
        pylab.title(objname)

pylab.xlabel('Time [MJD]')
pylab.tight_layout()_____no_output_____

Plot differences from the mean magnitudes in the initial search._____no_output_____

# convert flux in Jy to magnitudes
t = detection_TAP_results['obsTime']
mag = -2.5*np.log10(detection_TAP_results['psfFlux']) + 8.90
xlim = np.array([t.min(),t.max()])
xlim = xlim + np.array([-1,1])*0.02*(xlim[1]-xlim[0])

pylab.rcParams.update({'font.size': 14})
pylab.figure(1,(10,10))

#detection_TAP_results['filterType'] is a byte string, compare accordingly:
for i, filter in enumerate([b'g',b'r',b'i',b'z',b'y']):
    pylab.subplot(511+i)
    w = np.where(detection_TAP_results['filterType']==filter)
    magmean = TAP_results[filter.decode('ascii')+'MeanPSFMag'][0]
    pylab.plot(t[w],mag[w] - magmean,'-o')
    pylab.ylabel('{} [mag - {:.2f}]'.format(filter,magmean))
    pylab.xlim(xlim)
    pylab.gca().invert_yaxis()
    if i==0:
        pylab.title(objname)

pylab.xlabel('Time [MJD]')
pylab.tight_layout()_____no_output_____

### Identify bad data

There is one clearly bad $z$ magnitude with a very large difference.  Select the bad point and look at it in more detail.

Note that indexing a table (or numpy array) with a logical expression selects just the rows where that expression is true._____no_output_____

detection_TAP_results[ (detection_TAP_results['filterType']=='z') & (np.abs(mag-TAP_results['zMeanPSFMag'][0]) > 2) ]_____no_output_____

From examining this table, it looks like `psfQfPerfect` is bad.  This flag is the PSF-weighted fraction of unmasked pixels in the image (see the [documentation](https://outerspace.stsci.edu/x/IoOc) for more details). Values near unity indicate good data that is not significantly affected by bad pixels.

Check all the `psfQfPerfect` values for the $z$ filter to see if this value really is unusual.  The list of values below are sorted by magnitude.  The bad point is the only value with `psfQfPerfect` < 0.95._____no_output_____

w = np.where(detection_TAP_results['filterType']=='z')
zdtab = detection_TAP_results[w]
zdtab['mag'] = mag[w]
zdtab['dmag'] = zdtab['mag'] - TAP_results['zMeanPSFMag'][0]
ii = np.argsort(-np.abs(zdtab['dmag']))
zdtab = zdtab[ii]
zdtab['objID','obsTime','mag','dmag','psfQfPerfect']_____no_output_____

### Repeat the plot with bad psfQfPerfect values excluded

Do the plot again but exclude low psfQfPerfect values._____no_output_____

# convert flux in Jy to magnitudes
t = detection_TAP_results['obsTime']
mag = -2.5*np.log10(detection_TAP_results['psfFlux']) + 8.90
magmean = 0.0*mag
for i, filter in enumerate([b'g',b'r',b'i',b'z',b'y']):
    magmean[detection_TAP_results['filterType']==filter] = TAP_results[filter.decode('ascii')+'MeanPSFMag'][0]

dmag = mag - magmean
dmag1 = dmag[detection_TAP_results['psfQfPerfect']>0.9]
# fix the x and y axis ranges
xlim = np.array([t.min(),t.max()])
xlim = xlim + np.array([-1,1])*0.02*(xlim[1]-xlim[0])
# flip axis direction for magnitude
ylim = np.array([dmag1.max(),dmag1.min()])
ylim = ylim + np.array([-1,1])*0.02*(ylim[1]-ylim[0])

pylab.rcParams.update({'font.size': 14})
pylab.figure(1,(10,10))

for i, filter in enumerate([b'g',b'r',b'i',b'z',b'y']):
    pylab.subplot(511+i)
    w = np.where((detection_TAP_results['filterType']==filter) & (detection_TAP_results['psfQfPerfect']>0.9))[0]
    pylab.plot(t[w],dmag[w],'-o')
    pylab.ylabel('{} [mag - {:.2f}]'.format(filter,magmean[w[0]]))
    pylab.xlim(xlim)
    pylab.ylim(ylim)
    if i==0:
        pylab.title(objname)

pylab.xlabel('Time [MJD]')
pylab.tight_layout()_____no_output_____

### Plot versus the periodic phase instead of epoch

Plot versus phase using known RR Lyr period from Simbad (table [J/AJ/132/1202/table4](http://vizier.u-strasbg.fr/viz-bin/VizieR-3?-source=J/AJ/132/1202/table4&-c=KQ%20UMa&-c.u=arcmin&-c.r=2&-c.eq=J2000&-c.geom=r&-out.max=50&-out.form=HTML%20Table&-oc.form=sexa))._____no_output_____

period = 0.48636 #days, from Simbad
# convert flux in Jy to magnitudes
t = (detection_TAP_results['obsTime'] % period) / period
mag = -2.5*np.log10(detection_TAP_results['psfFlux']) + 8.90
magmean = 0.0*mag
for i, filter in enumerate([b'g',b'r',b'i',b'z',b'y']):
    magmean[detection_TAP_results['filterType']==filter] = TAP_results[filter.decode('ascii')+'MeanPSFMag'][0]

dmag = mag - magmean
dmag1 = dmag[detection_TAP_results['psfQfPerfect']>0.9]
# fix the x and y axis ranges
xlim = np.array([t.min(),t.max()])
xlim = xlim + np.array([-1,1])*0.02*(xlim[1]-xlim[0])
# flip axis direction for magnitude
ylim = np.array([dmag1.max(),dmag1.min()])
ylim = ylim + np.array([-1,1])*0.02*(ylim[1]-ylim[0])

pylab.rcParams.update({'font.size': 14})
pylab.figure(1,(10,10))

for i, filter in enumerate([b'g',b'r',b'i',b'z',b'y']):
    pylab.subplot(511+i)
    w = np.where((detection_TAP_results['filterType']==filter) & (detection_TAP_results['psfQfPerfect']>0.9))[0]
    w = w[np.argsort(t[w])]
    pylab.plot(t[w],dmag[w],'-o')
    pylab.ylabel('{} [mag - {:.2f}]'.format(filter,magmean[w[0]]))
    pylab.xlim(xlim)
    pylab.ylim(ylim)
    if i==0:
        pylab.title(objname)

pylab.xlabel('Phase')
pylab.tight_layout()_____no_output_____

## Repeat search using eclipsing binary KIC 2161623

From [Villanova Kepler Eclipsing Binaries](http://keplerebs.villanova.edu)_____no_output_____

objname = 'KIC 2161623'
coords = Mast._resolve_object(objname)
ra,dec = coords.ra.value,coords.dec.value
radius = 1.0/3600.0 # radius = 1 arcsec

query = """
SELECT objID, RAMean, DecMean, nDetections, ng, nr, ni, nz, ny, gMeanPSFMag, rMeanPSFMag, iMeanPSFMag, zMeanPSFMag, yMeanPSFMag
FROM dbo.MeanObjectView
WHERE
CONTAINS(POINT('ICRS', RAMean, DecMean),CIRCLE('ICRS',{},{},{}))=1
AND nDetections > 1
""".format(ra,dec,radius)
print(query)

job = TAP_service.launch_job_async(query)
TAP_results = job.get_results()
TAP_results_____no_output_____

### Get Repeated Detection Information

This time include the `psfQfPerfect` limit directly in the database query._____no_output_____

objid = TAP_results['objID'][0]

query = """
SELECT
    objID, detectID, Detection.filterID as filterID, Filter.filterType, obsTime, ra, dec,
    psfFlux, psfFluxErr, psfMajorFWHM, psfMinorFWHM, psfQfPerfect,
    apFlux, apFluxErr, infoFlag, infoFlag2, infoFlag3
FROM Detection
NATURAL JOIN Filter
WHERE objID={}
AND psfQfPerfect >= 0.9
ORDER BY filterID, obsTime
""".format(objid)
print(query)

job = TAP_service.launch_job_async(query)
detection_TAP_results = job.get_results()

# add magnitude and difference from mean
detection_TAP_results['magmean'] = 0.0
for i, filter in enumerate([b'g',b'r',b'i',b'z',b'y']):
    detection_TAP_results['magmean'][detection_TAP_results['filterType']==filter] = TAP_results[filter.decode('ascii')+'MeanPSFMag'][0]
detection_TAP_results['mag'] = -2.5*np.log10(detection_TAP_results['psfFlux']) + 8.90
detection_TAP_results['dmag'] = detection_TAP_results['mag']-detection_TAP_results['magmean']

detection_TAP_results_____no_output_____t = detection_TAP_results['obsTime']
dmag = detection_TAP_results['dmag']
xlim = np.array([t.min(),t.max()])
xlim = xlim + np.array([-1,1])*0.02*(xlim[1]-xlim[0])
ylim = np.array([dmag.max(),dmag.min()])
ylim = ylim + np.array([-1,1])*0.02*(ylim[1]-ylim[0])

pylab.rcParams.update({'font.size': 14})
pylab.figure(1,(10,10))

for i, filter in enumerate([b'g',b'r',b'i',b'z',b'y']):
    pylab.subplot(511+i)
    w = np.where(detection_TAP_results['filterType']==filter)[0]
    pylab.plot(t[w],dmag[w],'-o')
    magmean = detection_TAP_results['magmean'][w[0]]
    pylab.ylabel('{} [mag - {:.2f}]'.format(filter,magmean))
    pylab.xlim(xlim)
    pylab.ylim(ylim)
    if i==0:
        pylab.title(objname)

pylab.xlabel('Time [MJD]')
pylab.tight_layout()_____no_output_____

### Plot versus phase using known period

Eclipsing binaries basically vary by same amount in all filters since it is a geometrical effect, so combine the data into a single light curve.  Wrap using known period and plot versus phase._____no_output_____

period = 2.2834698
bjd0 = 54999.599837
t = ((detection_TAP_results['obsTime']-bjd0) % period) / period
dmag = detection_TAP_results['dmag']
w = np.argsort(t)
t = t[w]
dmag = dmag[w]
xlim = np.array([t.min(),t.max()])
xlim = xlim + np.array([-1,1])*0.02*(xlim[1]-xlim[0])
ylim = np.array([dmag.max(),dmag.min()])
ylim = ylim + np.array([-1,1])*0.02*(ylim[1]-ylim[0])

pylab.rcParams.update({'font.size': 14})
pylab.figure(1,(10,6))
pylab.plot(t,dmag,'-o')
pylab.xlim(xlim)
pylab.ylim(ylim)
pylab.xlabel('Phase')
pylab.ylabel('Delta magnitude from mean [mag]')
pylab.title(objname)
pylab.tight_layout()_____no_output_____

## Repeat search for another eclipsing binary KIC 8153568_____no_output_____

objname = 'KIC 8153568'
coords = Mast._resolve_object(objname)
ra,dec = coords.ra.value,coords.dec.value
radius = 1.0/3600.0 # radius = 1 arcsec

query = """
SELECT objID, RAMean, DecMean, nDetections, ng, nr, ni, nz, ny, gMeanPSFMag, rMeanPSFMag, iMeanPSFMag, zMeanPSFMag, yMeanPSFMag
FROM dbo.MeanObjectView
WHERE
CONTAINS(POINT('ICRS', RAMean, DecMean),CIRCLE('ICRS',{},{},{}))=1
AND nDetections > 1
""".format(ra,dec,radius)
print(query)

job = TAP_service.launch_job_async(query)
TAP_results = job.get_results()
TAP_results_____no_output_____objid = TAP_results['objID'][0]
query = """
SELECT
    objID, detectID, Detection.filterID as filterID, Filter.filterType, obsTime, ra, dec,
    psfFlux, psfFluxErr, psfMajorFWHM, psfMinorFWHM, psfQfPerfect,
    apFlux, apFluxErr, infoFlag, infoFlag2, infoFlag3
FROM Detection
NATURAL JOIN Filter
WHERE objID={}
AND psfQfPerfect >= 0.9
ORDER BY filterID, obsTime
""".format(objid)
print(query)

job = TAP_service.launch_job_async(query)
detection_TAP_results = job.get_results()

# add magnitude and difference from mean
detection_TAP_results['magmean'] = 0.0
for i, filter in enumerate([b'g',b'r',b'i',b'z',b'y']):
    detection_TAP_results['magmean'][detection_TAP_results['filterType']==filter] = TAP_results[filter.decode('ascii')+'MeanPSFMag'][0]
detection_TAP_results['mag'] = -2.5*np.log10(detection_TAP_results['psfFlux']) + 8.90
detection_TAP_results['dmag'] = detection_TAP_results['mag']-detection_TAP_results['magmean']

detection_TAP_results_____no_output_____t = detection_TAP_results['obsTime']
dmag = detection_TAP_results['dmag']
xlim = np.array([t.min(),t.max()])
xlim = xlim + np.array([-1,1])*0.02*(xlim[1]-xlim[0])
ylim = np.array([dmag.max(),dmag.min()])
ylim = ylim + np.array([-1,1])*0.02*(ylim[1]-ylim[0])

pylab.rcParams.update({'font.size': 14})
pylab.figure(1,(10,10))

for i, filter in enumerate([b'g',b'r',b'i',b'z',b'y']):
    pylab.subplot(511+i)
    w = np.where(detection_TAP_results['filterType']==filter)[0]
    pylab.plot(t[w],dmag[w],'-o')
    magmean = detection_TAP_results['magmean'][w[0]]
    pylab.ylabel('{} [mag - {:.2f}]'.format(filter,magmean))
    pylab.xlim(xlim)
    pylab.ylim(ylim)
    if i==0:
        pylab.title(objname)

pylab.xlabel('Time [MJD]')
pylab.tight_layout()_____no_output_____

Eclipsing binaries basically vary by same amount in all filters since it is a geometrical effect, so combine the data into a single light curve.

Wrap using known period and plot versus phase.  Plot two periods of the light curve this time.

This nice light curve appears to show a secondary eclipse._____no_output_____

period = 3.6071431
bjd0 = 54999.289794
t = ((detection_TAP_results['obsTime']-bjd0) % period) / period
dmag = detection_TAP_results['dmag']
w = np.argsort(t)
# extend to two periods
nw = len(w)
w = np.append(w,w)
t = t[w]
# add one to second period
t[-nw:] += 1
dmag = dmag[w]
xlim = [0,2.0]
ylim = np.array([dmag.max(),dmag.min()])
ylim = ylim + np.array([-1,1])*0.02*(ylim[1]-ylim[0])

pylab.rcParams.update({'font.size': 14})
pylab.figure(1,(12,6))
pylab.plot(t,dmag,'-o')
pylab.xlim(xlim)
pylab.ylim(ylim)
pylab.xlabel('Phase')
pylab.ylabel('Delta magnitude from mean [mag]')
pylab.title(objname)
pylab.tight_layout()_____no_output_____

***
# Additional Resources

## Table Access Protocol
* IVOA standard for RESTful web service access to tabular data
* http://www.ivoa.net/documents/TAP/

## PanSTARRS 1 DR 2
* Catalog for PanSTARRS with additional Detection information
* https://outerspace.stsci.edu/display/PANSTARRS/

## Astronomical Query Data Language (2.0)
* IVOA standard for querying astronomical data in tabular format, with geometric search support
* http://www.ivoa.net/documents/latest/ADQL.html

## TapPlus
* Module created by ESAC Space Data Centre
* http://astroquery.readthedocs.io/en/latest/utils/tap.html_____no_output_____***_____no_output_____## About this Notebook
**Authors:** Rick White & Theresa Dower, STScI Archive Scientist & Software Engineer
**Updated On:** 02/27/2019_____no_output_____***_____no_output_____ _____no_output_____


# Notebook from thomasnilsson/02805-social-graphs-2018
Path: project1/Review Network.ipynb

import networkx as nx
import matplotlib.pyplot as plt
import numpy as np
from networkx.algorithms import bipartite
import pandas as pd
import re
import matplotlib
font = {'family':'DejaVu Sans', 'weight':'normal', 'size':22}
matplotlib.rc('font', **font)_____no_output_____def degree(g, nodes=None, as_list=True):
    deg = dict(g.degree())
    if nodes: deg = dict(g.degree(nodes))

    if as_list: return list(deg.values())
    return deg

def degree_plot(g, nodes=None, filename=None, title=''):
    deg = degree(g, nodes=nodes)
    bins = 100
    if len(nodes) < 100:
        bins = len(nodes)
    hist = np.histogram(deg, bins=bins)
    freqs, edges = hist[0], hist[1]
    n = freqs.size
    means = [(edges[i] + edges[i+1]) / 2 for i in range(n)]

    # SCATTER PLOT
    plt.figure(figsize=[15,10])
    plt.plot(means, freqs, ".", markersize=20)
    plt.xlabel("k")
    plt.ylabel("frequency")
    plt.title("Degree distribution for %s" % title)
    if filename: plt.savefig('plots/%s.svg' % filename, format='svg', bbox_inches="tight")
    plt.show()

    # LOG LOG PLOT
    plt.figure(figsize=[15,10])
    plt.loglog(means, freqs, ".", markersize=20)
    plt.xlabel("log(k)")
    plt.ylabel("log(frequency)")
    plt.title("Log-log degree distribution for %s" % title)
    if filename: plt.savefig('plots/log_%s.svg' % filename, format='svg', bbox_inches="tight")
    plt.show()_____no_output_____

### Read datasets_____no_output_____

# Constant Strings
USER = 'user'
ELITE_USER = 'elite_user'
BIZ = 'biz'

# Read in data
biz = pd.read_csv('toronto/toronto_biz.csv')
user = pd.read_csv('toronto/toronto_users.csv')
reviews = pd.read_csv('toronto/toronto_reviews.csv')
elite_user = user[~user.elite.str.contains('None')]

print('#Reviews:', len(reviews))
print('#Users:', len(set(reviews.user_id)))
print('#Elite users:', len(elite_user))
print('#Businesses:', len(set(reviews.business_id)))#Reviews: 379875
#Users: 84624
#Elite users: 7499
#Businesses: 9678

### Create the review network_____no_output_____

# A node class for storing data.
class Node:
    def __init__(self, Data, Type):
        self.Data = Data
        self.Type = Type

    def to_string(self):
        return "Node (%s), Data: " % (self.Type, self.Data)

    def __hash__(self):
        return hash(self.Data)
    def __eq__(self, other):
        return (
                self.__class__ == other.__class__ and
                self.Data == other.Data
               )_____no_output_____review_network = nx.Graph()
elite_ids = set(elite_user.user_id)
for r in reviews.itertuples():
    a = Node(r.user_id, ELITE_USER if r.user_id in elite_ids else USER)
    b = Node(r.business_id, BIZ)
    review_network.add_edge(a, b, weight=r.stars)_____no_output_____

#### Degree distributions_____no_output_____

review_biz_nodes = [n for n in list(review_network.nodes()) if n.Type == BIZ]
review_user_nodes = [n for n in list(review_network.nodes()) if n.Type == USER]
review_elite_nodes = [n for n in list(review_network.nodes()) if n.Type == ELITE_USER]_____no_output_____degree_plot(review_network, review_user_nodes, title="regular users", filename='reviews_degree_regular')_____no_output_____degree_plot(review_network, review_biz_nodes, title="Toronto restaurants", filename='reviews_degree_biz')_____no_output_____degree_plot(review_network, review_elite_nodes, title="Elite users", filename='reviews_degree_elite')_____no_output_____degree_plot(review_network, review_network.nodes(), title="all users", filename='reviews_degree_all')_____no_output_____print('Nodes:', len(review_network.nodes()))
print('Edges:', len(review_network.edges()))Nodes: 94291
Edges: 379875
L = max(nx.connected_component_subgraphs(review_network), key=len)
print("Nodes in largest subcomponent:", len(L.nodes()))
print("Edges in largest subcomponent:", len(L.edges()))Nodes in largest subcomponent: 94246
Edges in largest subcomponent: 379840
cluster_coeff_avg = nx.average_clustering(review_network)
print('Average clustering coeff. :', cluster_coeff_avg)Average clustering coeff. : 2.843489088264669e-06
nx_____no_output_____

### Centrality: Elite vs Non-Elite_____no_output_____**Intro:** Eigenvector centrality for a user node indicates to which degree they have reviewed restaurants with many reviews._____no_output_____

deg = nx.degree(review_network)
ev = nx.eigenvector_centrality_numpy(review_network)_____no_output_____ev_elite_user = [ev[n] for n in ev if n.Type == ELITE_USER]
ev_user = [ev[n] for n in ev if n.Type == USER]
elite_avg_ev = np.mean(ev_elite_user)
user_avg_ev = np.mean(ev_user)
all_user_ev = np.mean(ev_elite_user + ev_user)_____no_output_____deg_elite_user = [deg[n] for n in deg if n.Type == ELITE_USER]
deg_user = [deg[n] for n in deg if n.Type == USER]
elite_avg_deg = np.mean(deg_elite_user)
user_avg_deg = np.mean(deg_user)
all_user_deg = np.mean(deg_elite_user + deg_user)_____no_output_____print('Normal user mean EV centrality', user_avg_ev)
print('Elite user mean EV centrality', elite_avg_ev)
print('All users mean EV centrality', all_user_ev)
ratio = elite_avg_ev / user_avg_ev
print('Ratio (Elite : Normal): %.2f' % ratio)Normal user mean EV centrality 0.0005151721690592373
Elite user mean EV centrality 0.0034574370628518815
All users mean EV centrality 0.0007759271614785319
Ratio (Elite : Normal): 6.71
print('Normal user mean Deg centrality', user_avg_deg)
print('Elite user mean Deg centrality', elite_avg_deg)
print('All users mean Deg centrality', all_user_deg)
ratio = elite_avg_deg / user_avg_deg
print('Ratio Deg (Elite : Normal): %.2f' % ratio)Normal user mean Deg centrality 3.0597274271561394
Elite user mean Deg centrality 19.19215895452727
All users mean Deg centrality 4.489446440389524
Ratio Deg (Elite : Normal): 6.27

**Conclusion:** Elite users on average review 6.3 times as many restaurants, and review the most reviewed restaurants 6.7 times as frequently. This means that there is a small tendendency for elite users to review the more popular restaurants moreso, than other restaurants._____no_output_____

user['ev'] = 0_____no_output_____ev_user = {n.Data: ev[n] for n in ev if (n.Type == ELITE_USER) or (n.Type == USER)}_____no_output_____i = 1
p = int(len(ev_user) / 100)
for k in ev_user:
    if  i % p == 0: print('%i percent done' % (i/p))
    eigenvalue = ev_user[k]
    user.loc[user.user_id == k, 'ev'] = eigenvalue
    i += 11 percent done
2 percent done
3 percent done
4 percent done
5 percent done
6 percent done
7 percent done
8 percent done
9 percent done
10 percent done
11 percent done
12 percent done
13 percent done
14 percent done
15 percent done
16 percent done
17 percent done
18 percent done
19 percent done
20 percent done
21 percent done
22 percent done
23 percent done
24 percent done
25 percent done
26 percent done
27 percent done
28 percent done
29 percent done
30 percent done
31 percent done
32 percent done
33 percent done
34 percent done
35 percent done
36 percent done
37 percent done
38 percent done
39 percent done
40 percent done
41 percent done
42 percent done
43 percent done
44 percent done
45 percent done
46 percent done
47 percent done
48 percent done
49 percent done
50 percent done
51 percent done
52 percent done
53 percent done
54 percent done
55 percent done
56 percent done
57 percent done
58 percent done
59 percent done
60 percent done
61 percent done
62 percent done
63 percent done
64 percent done
65 percent done
66 percent done
67 percent done
68 percent done
69 percent done
70 percent done
71 percent done
72 percent done
73 percent done
74 percent done
75 percent done
76 percent done
77 percent done
78 percent done
79 percent done
80 percent done
81 percent done
82 percent done
83 percent done
84 percent done
85 percent done
86 percent done
87 percent done
88 percent done
89 percent done
90 percent done
91 percent done
92 percent done
93 percent done
94 percent done
95 percent done
96 percent done
97 percent done
98 percent done
99 percent done
100 percent done
user.to_csv('toronto/toronto_users.csv')_____no_output_____plt.figure(figsize=[15,10])
plt.scatter(user.average_stars, user.ev, edgecolors='black')
plt.xlabel('Yelp average rating')
plt.ylabel('Eigenvalue centrality')
plt.title('Eigenvalue centrality based on average user rating for Yelp users in Toronto')
plt.savefig('plots/user_rating_ev.svg', format='svg', bbox_inches="tight")
plt.show()_____no_output_____plt.figure(figsize=[15,10])
plt.scatter(np.log(user.review_count), user.ev, edgecolors='black')
plt.xlabel('Log Yelp review count')
plt.ylabel('Page rank score')
plt.title('Page Rank based on number of reviews for Yelp users in Toronto')
plt.savefig('plots/log_user_count_ev.svg', format='svg', bbox_inches="tight")
plt.show()/Users/tnni/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: RuntimeWarning: divide by zero encountered in log

ev_biz = {n.Data: ev[n] for n in ev if n.Type == BIZ}
deg_biz = {n.Data: deg[n] for n in deg if n.Type == BIZ}_____no_output_____biz.ev = biz['ev']_____no_output_____plt.figure(figsize=[15,10])
plt.scatter(biz.stars, biz.ev, edgecolors='black')
plt.xlabel('Yelp rating')
plt.ylabel('Page rank score')
plt.title('Page Rank for restaurants in Toronto')
plt.savefig('plots/biz_rating_ev.svg', format='svg', bbox_inches="tight")
plt.show()_____no_output_____plt.figure(figsize=[15,10])
plt.scatter(biz.review_count, biz.stars, edgecolors='black')
plt.ylabel('Yelp rating')
plt.xlabel('Review count')
plt.show()_____no_output_____plt.figure(figsize=[15,10])
plt.scatter(biz.stars, biz.review_count, edgecolors='black')
plt.xlabel('Yelp rating')
plt.ylabel('Review count')
plt.show()_____no_output_____

**Conclusion**: There seems to be a trend in Toronto where, the more the reviews a restaurant has, the higher the rating the restaurant has. This however is normally distributed around a score of 4, and it seems to regress when a business scores higher than 4.0 on average._____no_output_____

plt.figure(figsize=[15,10])
plt.scatter(biz.ev, biz.review_count, edgecolors='black')
plt.xlabel('Eigenvalue centrality score')
plt.ylabel('Review count')
plt.title('Eigenvalue centrality vs. number of review for restaurants in Toronto')
plt.savefig('plots/biz_ev_count.svg', format='svg', bbox_inches="tight")
plt.show()_____no_output_____

**Conclusion**: Eigenvector centrality score seems to be directly linked to degree, which is unsurprising since the more reviews a restaurant has, the higher the chances are that the reviewers are elite, and thus giving the restaurant a higher eigenvalue score._____no_output_____

print("Elite reviews:", elite_stars.shape)
print("Reg reviews:", regular_stars.shape)
print("All reviews:", all_stars.shape)Elite reviews: (143922,)
Reg reviews: (235953,)
All reviews: (379875,)
print("Elite reviews make up %.2f percent of all reviews:" % (100 * len(elite_stars) / len(all_stars)))Elite reviews make up 37.89 percent of all reviews:
regular_stars.shape_____no_output_____elite_stars = np.array(reviews[reviews.user_id.isin(elite_user.user_id)].stars)
regular_stars = np.array(reviews[~reviews.user_id.isin(elite_user.user_id)].stars)
all_stars = np.array(reviews.stars)

reg = np.histogram(regular_stars, bins=[1,2,3,4,5,6])[0]
reg = reg / sum(reg)

elit = np.histogram(elite_stars, bins=[1,2,3,4,5,6])[0]
elit = elit / sum(elit)

all_ = np.histogram(all_stars, bins=[1,2,3,4,5,6])[0]
all_ = all_ / sum(all_)_____no_output_____plt.figure(figsize=[15,10])
x = np.array([1,2,3,4,5])
dx = 1/12
plt.bar(x - 2*dx, height=elit, width=2*dx, color='#FFF571', edgecolor='black')
plt.bar(x, height=reg, width=2*dx, color='#FF588A', edgecolor='black')
plt.bar(x + 2*dx, height=all_, width=2*dx, color='#32A9CC', edgecolor='black')
plt.xlabel('Review stars')
plt.ylabel('Fraction of users')
plt.legend(['Elite users', 'Regular users', 'All users'])
plt.title('Review distribution for Elite- and regular Yelp users in Toronto')
plt.savefig('plots/review_dist.svg', format='svg', bbox_inches="tight")
plt.show()_____no_output_____plt.figure(figsize=[15,10])
x = np.array([1,2,3,4,5])
dx = 0.125
plt.bar(x - dx, height=elit, width=2*dx, edgecolor='black')
plt.bar(x + dx, height=reg, width=2*dx, edgecolor='black')
plt.xlabel('Review stars')
plt.ylabel('Fraction of users')
plt.legend(['Elite users', 'Regular users'])
plt.title('Review distribution for Elite- and regular Yelp users in Toronto')
plt.savefig('plots/review_dist.svg', format='svg', bbox_inches="tight")
plt.show()_____no_output_____plt.figure(figsize=[15,10])
x = np.array([1,2,3,4,5])
dx = 0.125
plt.bar(x - dx, height=elit, width=2*dx, edgecolor='black')
plt.bar(x + dx, height=reg, width=2*dx, edgecolor='black')
plt.xlabel('Review stars')
plt.ylabel('Fraction of users')
plt.legend(['Elite users', 'Regular users'])
plt.title('Review distribution for Elite- and regular Yelp users in Toronto')
plt.savefig('plots/review_dist.svg', format='svg', bbox_inches="tight")
plt.show()_____no_output_____

**Conclusion:** Elite users are more moderate and peak at 4 stars, where regular users are more critical and over-enthustiastic, i.e. giving 1 star reviews, and 5 star reviews._____no_output_____

# Only regular user reviews
reg_biz_graph = nx.subgraph(review_network, review_user_nodes + review_biz_nodes)
reg_weights_dict = reg_biz_graph.degree(review_biz_nodes, weight='weight')
reg_degrees_dict = reg_biz_graph.degree(review_biz_nodes)
reg_biz_ratings_dict = {
    node.Data: reg_weights_dict[node] / reg_degrees_dict[node]
    for node in review_biz_nodes
    if reg_degrees_dict[node] > 0
}_____no_output_____# Only elite reviews
elite_biz_graph = nx.subgraph(review_network, review_elite_nodes + review_biz_nodes)
elite_weights_dict = elite_biz_graph.degree(review_biz_nodes, weight='weight')
elite_degrees_dict = elite_biz_graph.degree(review_biz_nodes)
elite_biz_ratings_dict = {
    node.Data: elite_weights_dict[node] / elite_degrees_dict[node]
    for node in review_biz_nodes
    if elite_degrees_dict[node] > 0
}_____no_output_____# All user reviews
all_weights_dict = review_network.degree(review_biz_nodes, weight='weight')
all_degrees_dict = review_network.degree(review_biz_nodes)
all_biz_ratings_dict = {
    node.Data: all_weights_dict[node] / all_degrees_dict[node]
    for node in review_biz_nodes
    if all_degrees_dict[node] > 0
}_____no_output_____biz_ids = [b.Data for b in review_biz_nodes]_____no_output_____# Comparison REGULAR AND ELITE
deltas_elite_reg = {
    biz_id: elite_biz_ratings_dict[biz_id] - reg_biz_ratings_dict[biz_id]
    for biz_id in biz_ids
    if biz_id in reg_biz_ratings_dict.keys()
    and biz_id in elite_biz_ratings_dict.keys()
}_____no_output_____# Comparison ALL AND ELITE
deltas_elite_all = {
    biz_id: elite_biz_ratings_dict[biz_id] - all_biz_ratings_dict[biz_id]
    for biz_id in biz_ids
    if biz_id in all_biz_ratings_dict.keys()
    and biz_id in elite_biz_ratings_dict.keys()
}_____no_output_____# Comparison ALL AND REG
deltas_reg_all = {
    biz_id: reg_biz_ratings_dict[biz_id] - all_biz_ratings_dict[biz_id]
    for biz_id in biz_ids
    if biz_id in all_biz_ratings_dict.keys()
    and biz_id in reg_biz_ratings_dict.keys()
}_____no_output_____plt.figure(figsize=[20,5])
plt.hist(np.array(list(deltas_elite_all.values())), bins=200, edgecolor='black')
plt.xlabel('Delta (stars)')
plt.title('Elite reviews compared to all user reviews')
plt.savefig('plots/delta_elite_all.svg', format='svg', bbox_inches="tight")
plt.show()_____no_output_____plt.figure(figsize=[20,5])
plt.hist(np.array(list(deltas_elite_reg.values())), bins=200, edgecolor='black')
plt.xlabel('Delta (stars)')
plt.title('Elite reviews compared to regular user reviews')
plt.savefig('plots/delta_elite_reg.svg', format='svg', bbox_inches="tight")
plt.show()_____no_output_____plt.figure(figsize=[20,5])
plt.hist(np.array(list(deltas_reg_all.values())), bins=200, edgecolor='black')
plt.xlabel('Delta (stars)')
plt.title('Regular user reviews compared to all user reviews')
plt.savefig('plots/delta_all_reg.svg', format='svg', bbox_inches="tight")
plt.show()_____no_output_____biz_sorted_by_rating = biz.sort_values('stars',ascending=False)
list(biz_sorted_by_rating.name)[:10]_____no_output_____elite_reviews = reviews[reviews.user_id.isin(elite_user.user_id)]
elite_biz = biz[biz.business_id.isin(elite_reviews.business_id)]
elite_biz_sorted_by_rating = elite_biz.sort_values('stars',ascending=False)
list(elite_biz_sorted_by_rating.name)[:10]_____no_output_____


# Notebook from keenborder786/GBX_Model
Path: main/Liar Dataset GBX Model.ipynb

# GBX Model on LIAR Dataset_____no_output_____

from google.colab import drive
drive.mount('/content/drive')_____no_output_____

# Requirements to Run the Notebook_____no_output_____

!pip install transformers
!pip install nltkCollecting transformers
  Downloading https://files.pythonhosted.org/packages/ae/05/c8c55b600308dc04e95100dc8ad8a244dd800fe75dfafcf1d6348c6f6209/transformers-3.1.0-py3-none-any.whl (884kB)
     |████████████████████████████████| 890kB 2.9MB/s
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.18.5)
Collecting sacremoses
  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)
     |████████████████████████████████| 890kB 20.9MB/s
Collecting sentencepiece!=0.1.92
  Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)
     |████████████████████████████████| 1.1MB 21.4MB/s
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)
Requirement already satisfied: dataclasses; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.7)
Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.4)
Collecting tokenizers==0.8.1.rc2
  Downloading https://files.pythonhosted.org/packages/80/83/8b9fccb9e48eeb575ee19179e2bdde0ee9a1904f97de5f02d19016b8804f/tokenizers-0.8.1rc2-cp36-cp36m-manylinux1_x86_64.whl (3.0MB)
     |████████████████████████████████| 3.0MB 31.6MB/s
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)
Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.15.0)
Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)
Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (0.16.0)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (1.24.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2020.6.20)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)
Building wheels for collected packages: sacremoses
  Building wheel for sacremoses (setup.py) ... done
  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893257 sha256=bc76da7df681feb12a989e5b0713b78995b4fd025fea7841f08a1f7bcda66a1d
  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45
Successfully built sacremoses
Installing collected packages: sacremoses, sentencepiece, tokenizers, transformers
Successfully installed sacremoses-0.0.43 sentencepiece-0.1.91 tokenizers-0.8.1rc2 transformers-3.1.0
Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (3.2.5)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from nltk) (1.15.0)
# the OG
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib as mpl
from matplotlib import style

# Sklearn
from sklearn.model_selection import train_test_split
from sklearn.metrics import *
from sklearn.utils.class_weight import compute_class_weight
#NLP Specifics
from nltk import word_tokenize
import nltk
from tokenizers import BertWordPieceTokenizer
from transformers import (AdamW,get_linear_schedule_with_warmup,get_cosine_schedule_with_warmup,RobertaTokenizerFast,DistilBertTokenizerFast,PreTrainedTokenizerFast,AutoModel,
                          DistilBertTokenizerFast,GPT2TokenizerFast,AutoTokenizer,BertTokenizer,TFBertModel,TFOpenAIGPTModel,OpenAIGPTTokenizer,DistilBertTokenizer,
                          DistilBertConfig,GPT2Config,RobertaConfig,TFDistilBertModel,XLMTokenizer, TFXLMModel,TFBertForSequenceClassification,TFGPT2Model,TFXLMRobertaModel)

#tensorflow
import tensorflow as tf
import tensorflow_hub as hub
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import *
from tensorflow.keras.initializers import Constant
from tensorflow.keras import regularizers

#other imports
import urllib
import os
import gc
from tqdm import tqdm
import re
import random
from typing import Callable, List, Optional, Union

nltk.download('punkt')
style.use('ggplot')
MAX_LEN=256[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\Mohammad.Khan\AppData\Roaming\nltk_data...
[nltk_data]   Unzipping tokenizers\punkt.zip.
uri_train  = 'https://raw.githubusercontent.com/thiagorainmaker77/liar_dataset/master/train.tsv'
uri_valid  = 'https://raw.githubusercontent.com/thiagorainmaker77/liar_dataset/master/valid.tsv'
uri_test  = 'https://raw.githubusercontent.com/thiagorainmaker77/liar_dataset/master/test.tsv'

df_train = pd.read_table(uri_train,
                             names = ['id',	'label'	,'statement',	'subject',	'speaker', 	'job', 	'state',	'party',	'barely_true_c',	'false_c',	'half_true_c',	'mostly_true_c',	'pants_on_fire_c',	'venue'])

df_valid = pd.read_table(uri_valid,
                             names =['id',	'label'	,'statement',	'subject',	'speaker', 	'job', 	'state',	'party',	'barely_true_c',	'false_c',	'half_true_c',	'mostly_true_c',	'pants_on_fire_c',	'venue'])

df_test = pd.read_csv(uri_test, sep='\t',
                            names =['id',	'label'	,'statement',	'subject',	'speaker', 	'job', 	'state',	'party',	'barely_true_c',	'false_c',	'half_true_c',	'mostly_true_c',	'pants_on_fire_c',	'venue'])

_____no_output_____df_train.drop(['barely_true_c','false_c','half_true_c','mostly_true_c','pants_on_fire_c'],axis=1,inplace=True)
df_valid.drop(['barely_true_c','false_c','half_true_c','mostly_true_c','pants_on_fire_c'],axis=1,inplace=True)_____no_output_____df_test.drop(['barely_true_c','false_c','half_true_c','mostly_true_c','pants_on_fire_c'],axis=1,inplace=True)_____no_output_____set(df_train['label'])_____no_output_____df_train['label'].value_counts().plot(kind='bar')_____no_output_____df_valid['label'].value_counts().plot(kind='bar')_____no_output_____def seed_everything(seed):
  random.seed(seed)
  tf.compat.v1.set_random_seed(seed)
  tf.random.set_seed(seed)
seed_everything(123)_____no_output_____# Detect hardware, return appropriate distribution strategy
try:
    # TPU detection. No parameters necessary if TPU_NAME environment variable is
    # set: this is always the case on Kaggle.
    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
    print('Running on TPU ', tpu.master())
except ValueError:
    tpu = None

if tpu:
    tf.config.experimental_connect_to_cluster(tpu)
    tf.tpu.experimental.initialize_tpu_system(tpu)
    strategy = tf.distribute.TPUStrategy(tpu)
else:
    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.
    strategy = tf.distribute.get_strategy()INFO:absl:Entering into master device scope: /job:worker/replica:0/task:0/device:CPU:0
def fast_tokenization(text,encode_plus=False):
  attention_mask=[]
  input_ids=[]
  for instance in tqdm(text,desc='Tokenization Process'):
    if encode_plus:
      token_out=fast_tokenizer.encode_plus(instance,return_attention_mask=True,padding='max_length',truncation=True,max_length=MAX_LEN)
      input_ids.append(token_out['input_ids'])
      attention_mask.append(token_out['attention_mask'])
    else:
      token_out=fast_tokenizer.encode(instance)
      input_ids.append(token_out.ids)
      attention_mask.append(token_out.attention_mask)
  return input_ids,attention_mask_____no_output_____X_train={}
X_valid={}
X_test={}_____no_output_____df_train.head()_____no_output_____def stop_word_removal(text):
  stop_word=['The','the','a','A','An','an']
  tokenized_sentence=word_tokenize(text)
  final_tokenized_sentence=[]
  for word in tokenized_sentence:
    if word not in stop_word:
        final_tokenized_sentence.append(word)
  final_tokenized_sentence=' '.join(final_tokenized_sentence)
  return final_tokenized_sentence_____no_output_____def label_numeric(target):
    dic_map={'true':0,'half-true':1,'mostly-true':3,'barely-true':2,'pants-fire':4,'false':5}
    return dic_map[target]_____no_output_____df_train.isnull().sum()_____no_output_____df_train.fillna('[UNK]',inplace=True)
df_valid.fillna('[UNK]',inplace=True)
df_test.fillna('[UNK]',inplace=True)_____no_output_____df_train['combine_text_tokenize']=df_train['statement']+' Subject: '+df_train['subject']+' ,Speaker: '+df_train['speaker']+' ,Job: '+df_train['job']+' ,State: '+df_train['state']+' ,Party: ' +df_train['party']+' ,Venue: '+df_train['venue']
df_valid['combine_text_tokenize']=df_valid['statement']+' Subject: '+df_valid['subject']+' ,Speaker: '+df_valid['speaker']+' ,Job: '+df_valid['job']+' ,State: '+df_valid['state']+' ,Party: ' +df_valid['party']+' ,Venue: '+df_valid['venue']
df_test['combine_text_tokenize']=df_test['statement']+' Subject: '+df_test['subject']+' ,Speaker: '+df_test['speaker']+' ,Job: '+df_test['job']+' ,State: '+df_test['state']+' ,Party: ' +df_test['party']+' ,Venue: '+df_test['venue']_____no_output_____df_train.iloc[0,-1]_____no_output_____df_train['combine_text_tokenize']=df_train['combine_text_tokenize'].map(stop_word_removal)
df_valid['combine_text_tokenize']=df_valid['combine_text_tokenize'].map(stop_word_removal)
df_test['combine_text_tokenize']=df_test['combine_text_tokenize'].map(stop_word_removal)_____no_output_____df_train['target']=df_train['label'].map(label_numeric)
df_valid['target']=df_valid['label'].map(label_numeric)
df_test['target']=df_test['label'].map(label_numeric)_____no_output_____class WarmUp(tf.keras.optimizers.schedules.LearningRateSchedule):
    """
    Applies a warmup schedule on a given learning rate decay schedule.

    Args:
        initial_learning_rate (:obj:`float`):
            The initial learning rate for the schedule after the warmup (so this will be the learning rate at the end
            of the warmup).
        decay_schedule_fn (:obj:`Callable`):
            The schedule function to apply after the warmup for the rest of training.
        warmup_steps (:obj:`int`):
            The number of steps for the warmup part of training.
        power (:obj:`float`, `optional`, defaults to 1):
            The power to use for the polynomial warmup (defaults is a linear warmup).
        name (:obj:`str`, `optional`):
            Optional name prefix for the returned tensors during the schedule.
    """

    def __init__(
        self,
        initial_learning_rate: float,
        decay_schedule_fn: Callable,
        warmup_steps: int,
        power: float = 1.0,
        name: str = None,
    ):
        super().__init__()
        self.initial_learning_rate = initial_learning_rate
        self.warmup_steps = warmup_steps
        self.power = power
        self.decay_schedule_fn = decay_schedule_fn
        self.name = name

    def __call__(self, step):
        with tf.name_scope(self.name or "WarmUp") as name:

            global_step_float = tf.cast(step, tf.float32)
            warmup_steps_float = tf.cast(self.warmup_steps, tf.float32)
            warmup_percent_done = global_step_float / warmup_steps_float
            warmup_learning_rate = self.initial_learning_rate * tf.math.pow(warmup_percent_done, self.power)
            return tf.cond(
                global_step_float < warmup_steps_float,
                lambda: warmup_learning_rate,
                lambda: self.decay_schedule_fn(step - self.warmup_steps),
                name=name,
            )

    def get_config(self):
        return {
            "initial_learning_rate": self.initial_learning_rate,
            "decay_schedule_fn": self.decay_schedule_fn,
            "warmup_steps": self.warmup_steps,
            "power": self.power,
            "name": self.name,
        }

def create_optimizer(
    init_lr: float,
    num_train_steps: int,
    num_warmup_steps: int,
    min_lr_ratio: float = 0.0,
    adam_epsilon: float = 1e-8,
    weight_decay_rate: float = 0.0,
    include_in_weight_decay: Optional[List[str]] = None,schedule='linear',
):
    """
    Creates an optimizer with a learning rate schedule using a warmup phase followed by a linear decay.

    Args:
        init_lr (:obj:`float`):
            The desired learning rate at the end of the warmup phase.
        num_train_step (:obj:`int`):
            The total number of training steps.
        num_warmup_steps (:obj:`int`):
            The number of warmup steps.
        min_lr_ratio (:obj:`float`, `optional`, defaults to 0):
            The final learning rate at the end of the linear decay will be :obj:`init_lr * min_lr_ratio`.
        adam_epsilon (:obj:`float`, `optional`, defaults to 1e-8):
            The epsilon to use in Adam.
        weight_decay_rate (:obj:`float`, `optional`, defaults to 0):
            The weight decay to use.
        include_in_weight_decay (:obj:`List[str]`, `optional`):
            List of the parameter names (or re patterns) to apply weight decay to. If none is passed, weight decay is
            applied to all parameters except bias and layer norm parameters.
    """
    # Implements linear decay of the learning rate.
    if schedule=='linear':
      lr_schedule = tf.keras.optimizers.schedules.PolynomialDecay(
          initial_learning_rate=init_lr,
          decay_steps=num_train_steps - num_warmup_steps,
          end_learning_rate=init_lr * min_lr_ratio,
      )
    else:
      lr_schedule=tf.keras.experimental.CosineDecay(
      initial_learning_rate=init_lr, decay_steps=num_train_steps - num_warmup_steps, alpha=0.0, name=None)

    if num_warmup_steps:
        lr_schedule = WarmUp(
            initial_learning_rate=init_lr, decay_schedule_fn=lr_schedule, warmup_steps=num_warmup_steps,
        )
    if weight_decay_rate > 0.0:
        optimizer = AdamWeightDecay(
            learning_rate=lr_schedule,
            weight_decay_rate=weight_decay_rate,
            beta_1=0.9,
            beta_2=0.999,
            epsilon=adam_epsilon,
            exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"],
            include_in_weight_decay=include_in_weight_decay,
        )
    else:
        optimizer = tf.keras.optimizers.Adam(learning_rate=lr_schedule, epsilon=adam_epsilon)
    # We return the optimizer and the LR scheduler in order to better track the
    # evolution of the LR independently of the optimizer.
    return optimizer, lr_schedule_____no_output_____class LossHistory(tf.keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.losses = []
    def on_batch_end(self, batch, logs={}):
        self.losses.append(logs.get('loss'))_____no_output_____def accuracy(y_true,y_pred,name):
  y_true=np.argmax(y_true,axis=1)
  y_pred=np.argmax(y_pred,axis=1)
  accuracy=accuracy_score(y_true,y_pred)
  print('The accuracy of the model {} is Accuracy: {:.3%}'.format(name,accuracy))
  print(classification_report(y_true, y_pred, target_names=['true','half-true','barely-true','mostly-true','pants-fire','false']))_____no_output_____def f1_score_m(y_true,y_pred,name):
  y_true=np.argmax(y_true,axis=1)
  y_pred=np.argmax(y_pred,axis=1)
  f1_m=f1_score(y_true,y_pred,average='macro')
  print('The F1-M Score of the model {} is F-1-Score: {:.3%}'.format(name,f1_m))_____no_output_____colors = plt.rcParams['axes.prop_cycle'].by_key()['color']

def plot_loss(history,main_title):
# Use a log scale to show the wide range of values.
    plt.figure(figsize=(12,10))
    plt.semilogy(list(range(len(history))),  history,
               color='blue', label='Training')
    plt.title(main_title)
    plt.xlabel('Logging Step')
    plt.ylabel('Loss')

    plt.legend(title="Log Loss")

def plot_cm(y_true, y_pred, title):
    ''''
    input y_true-Ground Truth Labels
          y_pred-Predicted Value of Model
          title-What Title to give to the confusion matrix

    Draws a Confusion Matrix for better understanding of how the model is working

    return None

    '''

    figsize=(10,10)
    y_true=np.argmax(y_true,axis=1)
    y_pred=np.argmax(y_pred,axis=1)
    cm = confusion_matrix(y_true, y_pred, labels=np.unique(y_true))
    cm_sum = np.sum(cm, axis=1, keepdims=True)
    cm_perc = cm / cm_sum.astype(float) * 100
    annot = np.empty_like(cm).astype(str)
    nrows, ncols = cm.shape
    for i in range(nrows):
        for j in range(ncols):
            c = cm[i, j]
            p = cm_perc[i, j]
            if i == j:
                s = cm_sum[i]
                annot[i, j] = '%.1f%%\n%d/%d' % (p, c, s)
            elif c == 0:
                annot[i, j] = ''
            else:
                annot[i, j] = '%.1f%%\n%d' % (p, c)
    cm = pd.DataFrame(cm, index=np.unique(y_true), columns=np.unique(y_true))
    cm.index.name = 'Actual'
    cm.columns.name = 'Predicted'
    fig, ax = plt.subplots(figsize=figsize)
    plt.title(title)
    sns.heatmap(cm, cmap= "YlGnBu", annot=annot, fmt='', ax=ax)_____no_output_____true_labels=['true','half-true','barely-true','mostly-true']
false_labels=['false','pants-fire']
high_score_labels=['true','pants-fire']

def score_submission(gold_labels, test_labels):
    score = 0.0
    for i, (g, t) in enumerate(zip(gold_labels, test_labels)):
        g_stance, t_stance = g, t
        if g_stance == t_stance:
            score += 0.25
            if g_stance in high_score_labels:
                score += 0.50
        if g_stance in true_labels and t_stance in true_labels:
            score += 0.1
        elif g_stance in false_labels and t_stance in false_labels:
            score += 0.1

    return score

def report_score(actual,predicted,name):
    actual=np.argmax(actual,axis=1)
    predicted=np.argmax(predicted,axis=1)
    score = score_submission(actual,predicted)
    best_score = score_submission(actual,actual)
    print('For Model {}'.format(name))
    print("Score: " +str(score) + " out of " + str(best_score) + "\t("+str(score*100/best_score) + "%)")
    return score*100/best_score_____no_output_____def tensorflow_make_train_valid_data(X_train,y_train,X_valid,y_valid,test=False):

      train = (
          tf.data.Dataset
          .from_tensor_slices((X_train, y_train))
          .repeat()
          .shuffle(1024)
          .batch(BATCH_SIZE)
          .prefetch(AUTO))

      valid = (
          tf.data.Dataset
          .from_tensor_slices((X_valid, y_valid))
          .batch(BATCH_SIZE)
          .cache()
          .prefetch(AUTO)
      )
      return train,valid_____no_output_____def make_test_data(X_test):
    test = (
          tf.data.Dataset
          .from_tensor_slices(X_test)
          .batch(BATCH_SIZE)
      )
    return test_____no_output_____fast_tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert-base-uncased")._tokenizer
fast_tokenizer.enable_padding(length=MAX_LEN)
fast_tokenizer.enable_truncation(max_length=MAX_LEN)

input_ids_distill_bert,attention_mask_distill_bert=fast_tokenization(df_train['combine_text_tokenize'].values)
X_train['input_ids_distill_bert'],X_train['attention_mask_distill_bert']=input_ids_distill_bert,attention_mask_distill_bert
input_ids_distill_bert,attention_mask_distill_bert=fast_tokenization(df_valid['combine_text_tokenize'].values)
X_valid['input_ids_distill_bert'],X_valid['attention_mask_distill_bert']=input_ids_distill_bert,attention_mask_distill_bert

input_ids_distill_bert,attention_mask_distill_bert=fast_tokenization(df_test['combine_text_tokenize'].values)
X_test['input_ids_distill_bert'],X_test['attention_mask_distill_bert']=input_ids_distill_bert,attention_mask_distill_bert
_____no_output_____fast_tokenizer = GPT2TokenizerFast.from_pretrained("distilgpt2")._tokenizer
fast_tokenizer.enable_padding(length=MAX_LEN)
fast_tokenizer.enable_truncation(max_length=MAX_LEN)

input_ids_gpt_2,attention_mask_gpt_2=fast_tokenization(df_train['combine_text_tokenize'].values)
X_train['input_ids_gpt_2'],X_train['attention_mask_gpt_2']=input_ids_gpt_2,attention_mask_gpt_2
input_ids_gpt_2,attention_mask_gpt_2=fast_tokenization(df_valid['combine_text_tokenize'].values)
X_valid['input_ids_gpt_2'],X_valid['attention_mask_gpt_2']=input_ids_gpt_2,attention_mask_gpt_2

input_ids_gpt_2,attention_mask_gpt_2=fast_tokenization(df_test['combine_text_tokenize'].values)
X_test['input_ids_gpt_2'],X_test['attention_mask_gpt_2']=input_ids_distill_bert,attention_mask_distill_bert
_____no_output_____model_type = 'distilroberta-base'
fast_tokenizer =AutoTokenizer.from_pretrained(model_type)
input_ids_roberta,attention_mask_roberta=fast_tokenization(df_train['combine_text_tokenize'].values,encode_plus=True)
X_train['input_ids_roberta'],X_train['attention_mask_roberta']=input_ids_roberta,attention_mask_roberta
roberta_model_base_input_id,roberta_model_attention_mask=fast_tokenization(df_valid['combine_text_tokenize'].values,encode_plus=True)
X_valid['input_ids_roberta'],X_valid['attention_mask_roberta']=roberta_model_base_input_id,roberta_model_attention_mask

roberta_model_base_input_id,roberta_model_attention_mask=fast_tokenization(df_test['combine_text_tokenize'].values,encode_plus=True)
X_test['input_ids_roberta'],X_test['attention_mask_roberta']=roberta_model_base_input_id,roberta_model_attention_mask

_____no_output_____y_train=np.array(tf.keras.utils.to_categorical(df_train['target'].values))
y_valid=np.array(tf.keras.utils.to_categorical(df_valid['target'].values))_____no_output_____print(X_train.keys())
print(X_valid.keys())dict_keys(['input_ids_distill_bert', 'attention_mask_distill_bert', 'input_ids_gpt_2', 'attention_mask_gpt_2', 'input_ids_roberta', 'attention_mask_roberta'])
dict_keys(['input_ids_distill_bert', 'attention_mask_distill_bert', 'input_ids_gpt_2', 'attention_mask_gpt_2', 'input_ids_roberta', 'attention_mask_roberta'])
BATCH_SIZE = 64
EPOCHS=100
LEARNING_RATE=1e-7
BETA_1=0.9
BETA_2=0.999
AUTO = tf.data.experimental.AUTOTUNE
steps_per_epoch=df_train.shape[0]//BATCH_SIZE
total_optimization_steps=int(steps_per_epoch*EPOCHS)
WARM_UP_STEPS=int(0.06*total_optimization_steps)
optimizer_with_schedule,lr_schedule=create_optimizer(init_lr=LEARNING_RATE,num_train_steps=total_optimization_steps,num_warmup_steps=WARM_UP_STEPS,schedule='cosine')
EARLY_STOP=tf.keras.callbacks.EarlyStopping(patience=10)
MAX_LEN=MAX_LEN
LOSS=tf.keras.losses.CategoricalCrossentropy()
OPTIMIZER=tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE,beta_1=BETA_1,beta_2=BETA_2)

_____no_output_____test=make_test_data(X_test)_____no_output_____def gbx_layers():
    CONFIG_db=DistilBertConfig()
    distill_bert = (
                TFDistilBertModel.from_pretrained('distilbert-base-uncased')
            )

    gpt_layer=(
        TFGPT2Model.from_pretrained('distilgpt2')
    )

    roberta_model=(
        TFXLMRobertaModel.from_pretrained('distilroberta-base')
    )

    input_ids_distill_bert = tf.keras.layers.Input(shape=(MAX_LEN,), dtype=tf.int32, name="input_ids_distill_bert")
    attention_mask_distill_bert=tf.keras.layers.Input(shape=(MAX_LEN,), dtype=tf.int32, name="attention_mask_distill_bert")

    input_ids_gpt_2 = tf.keras.layers.Input(shape=(MAX_LEN,), dtype=tf.int32, name="input_ids_gpt_2")
    attention_mask_gpt_2=tf.keras.layers.Input(shape=(MAX_LEN,), dtype=tf.int32, name="attention_mask_gpt_2")

    input_ids_roberta = tf.keras.layers.Input(shape=(MAX_LEN,), dtype=tf.int32, name="input_ids_roberta")
    attention_mask_roberta=tf.keras.layers.Input(shape=(MAX_LEN,), dtype=tf.int32, name="attention_mask_roberta")

    hidden_state_output_dbert=distill_bert([input_ids_distill_bert,attention_mask_distill_bert])[0]
    hidden_state_output_gpt_2,_=gpt_layer({'input_ids': input_ids_gpt_2, 'attention_mask': attention_mask_gpt_2},past=None)
    hidden_state_output_roberta,_=roberta_model([input_ids_roberta,attention_mask_roberta])

    combine_input_1=tf.keras.layers.concatenate([hidden_state_output_dbert,hidden_state_output_gpt_2.get("last_hidden_state"),hidden_state_output_roberta.
                                                 get("last_hidden_state")],axis=2,name='Concatenate_layer')
    final_input=tf.keras.layers.Conv1D(filters=768,kernel_size=3,padding='same',name='Conv_1d_layer')(combine_input_1)

    final_input_1=tf.keras.layers.GlobalAveragePooling1D(name='Average_Pooling_layer')(final_input)
    out = tf.keras.layers.Dense(6, activation='softmax',name='Softmax_Layer')(final_input_1)

    model = tf.keras.Model(inputs=[input_ids_distill_bert,attention_mask_distill_bert,input_ids_gpt_2,attention_mask_gpt_2,input_ids_roberta,attention_mask_roberta], outputs=out)

    return model_____no_output_____def compile_gbx(optim):
  model=gbx_layers()
  model.compile(optimizer=optim,loss=LOSS,metrics='accuracy')
  print(model.summary())
  return model

train,valid=tensorflow_make_train_valid_data(X_train,y_train,X_valid,y_valid)
with strategy.scope():
  model=compile_gbx(optimizer_with_schedule)Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_layer_norm', 'activation_13', 'vocab_projector', 'vocab_transform']
- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.
All model checkpoint weights were used when initializing TFGPT2Model.

All the weights of TFGPT2Model were initialized from the model checkpoint at distilgpt2.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFGPT2Model for predictions without further training.
Some weights of the model checkpoint at distilroberta-base were not used when initializing TFXLMRobertaModel: ['lm_head']
- This IS expected if you are initializing TFXLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing TFXLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFXLMRobertaModel were initialized from the model checkpoint at distilroberta-base.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFXLMRobertaModel for predictions without further training.

## GBX Model Run

Following Parameters were used-Run(1)

1.   BATCH_SIZE = 32
2.   EPOCHS=2
3.   LEARNING_RATE=3e-5
4.   BETA_1=0.9-BETA_2=0.999
5.   Scheduler=Linear with Intial Learning Rate of 3e-5
6.   Freezing=No Layer was Freezed
7.   MAX_LEN=256
8.   Trained on TPU(Google Colab)_____no_output_____

history = LossHistory()
model.fit(
      train,steps_per_epoch=steps_per_epoch,
      epochs=EPOCHS,callbacks=[history], validation_data=valid,
)
  Epoch 1/2
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/multi_device_iterator_ops.py:601: get_next_as_optional (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Iterator.get_next_as_optional()` instead.
y_pred=model.predict(test,verbose=1)
y_true=np.array(tf.keras.utils.to_categorical(df_test['target'].values)) 2/40 [>.............................] - ETA: 3:20WARNING:tensorflow:Callbacks method `on_predict_batch_end` is slow compared to the batch time (batch time: 0.0043s vs `on_predict_batch_end` time: 0.0454s). Check your callbacks.
saved_history=np.array(history.losses)
np.save('y_pred_gbx_liar_dataset_run(1).npy',y_pred,allow_pickle=True)
np.save('saved_history_liar_dataset_run(1).npy',saved_history)_____no_output_____plot_loss(history.losses,'GBX on Liar Dataset-Run(1)')_____no_output_____plot_cm(y_true, y_pred, 'GBX on Liar Dataset-Run(1)')_____no_output_____accuracy(y_true,y_pred,'GBX on Liar Dataset-Run(1)')The accuracy of the model GBX on Liar Dataset-Run(1) is Accuracy: 25.888%
              precision    recall  f1-score   support

        true       1.00      0.00      0.01       208
   half-true       0.23      0.69      0.34       265
 barely-true       0.29      0.11      0.16       212
 mostly-true       0.29      0.27      0.28       241
  pants-fire       0.37      0.34      0.35        92
       false       0.36      0.10      0.15       249

    accuracy                           0.26      1267
   macro avg       0.42      0.25      0.22      1267
weighted avg       0.41      0.26      0.21      1267

f1_score_m(y_true,y_pred,'GBX on Liar Dataset-Run(1)')The F1-M Score of the model GBX on Liar Dataset-Run(1) is F-1-Score: 21.616%

## GBX Model Run

Following Parameters were used-Run(2)

1.   BATCH_SIZE = 32
2.   EPOCHS=20(Early CallBack)
3.   LEARNING_RATE=3e-5
4.   BETA_1=0.9-BETA_2=0.999
5.   Scheduler=Linear with Intial Learning Rate of 3e-5
6.   Freezing=No Layer was Freezed
7.   MAX_LEN=256
8.   Trained on TPU(Google Colab)_____no_output_____

history = LossHistory()
model.fit(
      train,steps_per_epoch=steps_per_epoch,
      epochs=EPOCHS,callbacks=[history,EARLY_STOP], validation_data=valid,
)
  Epoch 1/20
WARNING:tensorflow:Gradients do not exist for variables ['tfxlm_roberta_model_1/roberta/pooler/dense/kernel:0', 'tfxlm_roberta_model_1/roberta/pooler/dense/bias:0'] when minimizing the loss.
plot_loss(history.losses,'GBX on Liar Dataset-Run(2)')_____no_output_____y_pred=model.predict(test,verbose=1)
y_true=np.array(tf.keras.utils.to_categorical(df_test['target'].values))
 1/40 [..............................] - ETA: 0sWARNING:tensorflow:Callbacks method `on_predict_batch_end` is slow compared to the batch time (batch time: 0.0047s vs `on_predict_batch_end` time: 0.0433s). Check your callbacks.
saved_history=np.array(history.losses)
np.save('/content/drive/My Drive/Final Research Project/y_pred_gbx_liar_dataset_run(2).npy',y_pred,allow_pickle=True)
np.save('/content/drive/My Drive/Final Research Project/saved_history_liar_dataset_run(2).npy',saved_history)_____no_output_____plot_cm(y_true, y_pred, 'GBX on Liar Dataset-Run(2)')_____no_output_____accuracy(y_true,y_pred,'GBX on Liar Dataset-Run(2)')The accuracy of the model GBX on Liar Dataset-Run(2) is Accuracy: 26.598%
              precision    recall  f1-score   support

        true       0.29      0.25      0.27       208
   half-true       0.27      0.32      0.29       265
 barely-true       0.20      0.17      0.18       212
 mostly-true       0.24      0.22      0.23       241
  pants-fire       0.27      0.21      0.23        92
       false       0.30      0.37      0.33       249

    accuracy                           0.27      1267
   macro avg       0.26      0.26      0.26      1267
weighted avg       0.26      0.27      0.26      1267

## GBX Model Run

Following Parameters were used-Run(3)-Need to reduced Overfitting

1.   BATCH_SIZE = 64
2.   EPOCHS=20(Early CallBack)
3.   LEARNING_RATE=3e-5
4.   BETA_1=0.9-BETA_2=0.999
5.   Scheduler=Linear with Intial Learning Rate of 3e-5
6.   Freezing=No Layer was Freezed
7.   MAX_LEN=128
8.   Trained on TPU(Google Colab)
9.   Dropout=0.4
10.  L1 Reg=1e-4
11.  L2 Reg=1e-3_____no_output_____

history = LossHistory()
model.fit(
      train,steps_per_epoch=steps_per_epoch,
      epochs=EPOCHS,callbacks=[history,EARLY_STOP], validation_data=valid,
)
  Epoch 1/20
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/multi_device_iterator_ops.py:601: get_next_as_optional (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Iterator.get_next_as_optional()` instead.
y_pred=model.predict(test,verbose=1)
y_true=np.array(tf.keras.utils.to_categorical(df_test['target'].values)) 2/20 [==>...........................] - ETA: 1:32WARNING:tensorflow:Callbacks method `on_predict_batch_end` is slow compared to the batch time (batch time: 0.0051s vs `on_predict_batch_end` time: 0.0426s). Check your callbacks.
saved_history=np.array(history.losses)
np.save('y_pred_gbx_liar_dataset_run(3).npy',y_pred,allow_pickle=True)
np.save('saved_history_liar_dataset_run(3).npy',saved_history)_____no_output_____plot_loss(history.losses,'GBX on Liar Dataset-Run(3)')_____no_output_____plot_cm(y_true, y_pred, 'GBX on Liar Dataset-Run(3)')_____no_output_____accuracy(y_true,y_pred,'GBX on Liar Dataset-Run(3)')The accuracy of the model GBX on Liar Dataset-Run(3) is Accuracy: 18.785%
              precision    recall  f1-score   support

        true       0.25      0.00      0.01       208
   half-true       0.00      0.00      0.00       265
 barely-true       0.00      0.00      0.00       212
 mostly-true       0.19      0.98      0.32       241
  pants-fire       0.00      0.00      0.00        92
       false       0.17      0.01      0.02       249

    accuracy                           0.19      1267
   macro avg       0.10      0.16      0.06      1267
weighted avg       0.11      0.19      0.06      1267

Following Parameters were used-Run(4)-Need to reduced Overfitting

1.   BATCH_SIZE = 32
2.   EPOCHS=20(Early CallBack)
3.   LEARNING_RATE=3e-5
4.   BETA_1=0.9-BETA_2=0.999
5.   Scheduler=Linear with Intial Learning Rate of 3e-5
6.   Freezing=Transformer Layers were freezed
7.   MAX_LEN=256
8.   Trained on TPU(Google Colab)
9.   Dropout=0.0
10.  L1 Reg=0.0
11.  L2 Reg=0.0_____no_output_____

## Freezing the Transformers Layers to Reduce the Complexity of the given architecture
for layer in model.layers[6:9]:
  layer.trainable=False_____no_output_____history = LossHistory()
model.fit(
      train,steps_per_epoch=steps_per_epoch,
      epochs=EPOCHS,callbacks=[history,EARLY_STOP], validation_data=valid,
)
  Epoch 1/20
  2/320 [..............................] - ETA: 31s - loss: 18.7553 - accuracy: 0.2031WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0195s vs `on_train_batch_end` time: 0.1739s). Check your callbacks.
y_pred=model.predict(test,verbose=1)
y_true=np.array(tf.keras.utils.to_categorical(df_test['target'].values)) 2/40 [>.............................] - ETA: 3:21WARNING:tensorflow:Callbacks method `on_predict_batch_end` is slow compared to the batch time (batch time: 0.0057s vs `on_predict_batch_end` time: 0.0453s). Check your callbacks.
plot_loss(history.losses,'GBX on Liar Dataset-Run(4)')_____no_output_____plot_cm(y_true, y_pred, 'GBX on Liar Dataset-Run(4)')_____no_output_____accuracy(y_true,y_pred,'GBX on Liar Dataset-Run(4)')The accuracy of the model GBX on Liar Dataset-Run(4) is Accuracy: 19.258%
              precision    recall  f1-score   support

        true       0.00      0.00      0.00       208
   half-true       0.20      0.81      0.32       265
 barely-true       0.00      0.00      0.00       212
 mostly-true       0.11      0.05      0.07       241
  pants-fire       0.00      0.00      0.00        92
       false       0.20      0.07      0.11       249

    accuracy                           0.19      1267
   macro avg       0.09      0.15      0.08      1267
weighted avg       0.10      0.19      0.10      1267

Following Parameters were used-Run(5)
1.   BATCH_SIZE = 64
2.   EPOCHS=20(Early CallBack)
3.   LEARNING_RATE=1e-5
4.   BETA_1=0.9-BETA_2=0.999
5.   Scheduler=Cosine with Intial Learning Rate of 1e-5
6.   Freezing=Transformer Layers were freezed
7.   MAX_LEN=256
8.   Trained on TPU(Google Colab)
9.   Balance Weights were Used
_____no_output_____

history = LossHistory()
y_integers = np.argmax(y_train, axis=1)
class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers)
d_class_weights = dict(enumerate(class_weights))
model.fit(
      train,steps_per_epoch=steps_per_epoch,
      epochs=EPOCHS,callbacks=[history,EARLY_STOP], validation_data=valid,class_weight=d_class_weights
)Epoch 1/20
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/multi_device_iterator_ops.py:601: get_next_as_optional (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Iterator.get_next_as_optional()` instead.
y_pred=model.predict(test,verbose=1)
y_true=np.array(tf.keras.utils.to_categorical(df_test['target'].values)) 2/20 [==>...........................] - ETA: 0sWARNING:tensorflow:Callbacks method `on_predict_batch_end` is slow compared to the batch time (batch time: 0.0077s vs `on_predict_batch_end` time: 0.0659s). Check your callbacks.
plot_loss(history.losses,'GBX on Liar Dataset-Run(5)')_____no_output_____plot_cm(y_true, y_pred, 'GBX on Liar Dataset-Run(5)')_____no_output_____accuracy(y_true,y_pred,'GBX on Liar Dataset-Run(5)')The accuracy of the model GBX on Liar Dataset-Run(5) is Accuracy: 25.178%
              precision    recall  f1-score   support

        true       0.33      0.05      0.08       208
   half-true       0.29      0.15      0.20       265
 barely-true       0.21      0.40      0.27       212
 mostly-true       0.28      0.50      0.36       241
  pants-fire       0.21      0.39      0.28        92
       false       0.31      0.11      0.17       249

    accuracy                           0.25      1267
   macro avg       0.27      0.27      0.23      1267
weighted avg       0.28      0.25      0.22      1267


# Notebook from jradavenport/GALEX_rot
Path: explore.ipynb

%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib

matplotlib.rcParams.update({'font.size':18})
matplotlib.rcParams.update({'font.family':'serif'})_____no_output_____

# The Idea:
Does GALEX (NUV) flux show decline as stars age?

To search for this, match the GALEX catalog to the Kepler rotation period catalog. As stars (within small temperature bins) spin-down (age, rotate slower) we would expect the NUV flux to decrease. Thus the (NUV - V), or (NUV - J) color should get *redder* with increasing rotation period_____no_output_____

file = '1488562031338A.csv'
data = pd.read_csv(file)
data.columns_____no_output_____# T = np.where((data[u'Teff']) < 5500)

# this needs to be fixed for 2 things:
# 1) put in temperature bins
# 2) put in L_NUV/L_bol, or similar

plt.scatter(data[u'Prot'], data[u'NUVmag'], s=1, alpha=0.5)

plt.ylabel('m_NUV (mag)')
plt.ylim(25,12)
plt.xscale('log')
plt.xlabel('Prot (days)')_____no_output_____# data[u'Teff']_____no_output_____file = '1488604487591A.csv'
data2 = pd.read_csv(file)
data2.columns/Users/james/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.py:2717: DtypeWarning: Columns (4) have mixed types. Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)
_ = plt.hist(data[u'Teff'], bins=35)
plt.xlabel('Temp (K)')_____no_output_____ok = np.where((data2[u'NUVmag']<20) & (data2[u'J']<16))
plt.scatter(data2[u'Prot'].values[ok], data2[u'NUVmag'].values[ok] - data2[u'V'].values[ok],
            s=5, alpha=0.5)
plt.ylabel('NUV - V (mag)')
plt.title('GALEX - Simbad vs Kepler')
plt.xscale('log')
plt.xlabel('Prot (days)')
plt.ylim(9,3)_____no_output_____plt.figure(figsize=(9,5))

Pbins = np.array([0.1, 1, 3, 7, 10, 20, 40])

ok = np.where((data2[u'Teff'] >= 6000) &
              (data2[u'NUVmag']<20) & (data2[u'J']<16))
plt.scatter(data2[u'Prot'].values[ok],
            data2[u'NUVmag'].values[ok] - data2[u'J'].values[ok],
            s=5, alpha=0.25, c='RoyalBlue')
UVclr = np.zeros(len(Pbins) - 1)
UVstd = np.zeros(len(Pbins) - 1)
for i in range(len(Pbins)-1):
    iok = np.where((data2[u'Prot'].values[ok] >= Pbins[i]) & (data2[u'Prot'].values[ok] < Pbins[i+1]))[0]
    UVclr[i] = np.median(data2[u'NUVmag'].values[ok][iok] - data2[u'J'].values[ok][iok])
    UVstd[i] = np.std(data2[u'NUVmag'].values[ok][iok] - data2[u'J'].values[ok][iok])
plt.errorbar((Pbins[1:] + Pbins[:-1])/2., UVclr, yerr=UVstd, c='RoyalBlue', lw=2)

ok = np.where((data2[u'Teff'] >= 5500) & (data2[u'Teff'] < 6000) &
              (data2[u'NUVmag']<20) & (data2[u'J']<16))
plt.scatter(data2[u'Prot'].values[ok],
            data2[u'NUVmag'].values[ok] - data2[u'J'].values[ok],
            s=5, alpha=0.25, c='orange')
UVclr = np.zeros(len(Pbins) - 1)
for i in range(len(Pbins)-1):
    iok = np.where((data2[u'Prot'].values[ok] >= Pbins[i]) & (data2[u'Prot'].values[ok] < Pbins[i+1]))[0]
    UVclr[i] = np.median(data2[u'NUVmag'].values[ok][iok] - data2[u'J'].values[ok][iok])
    UVstd[i] = np.std(data2[u'NUVmag'].values[ok][iok] - data2[u'J'].values[ok][iok])
plt.errorbar((Pbins[1:] + Pbins[:-1])/2., UVclr, yerr=UVstd, c='orange', lw=2)

ok = np.where((data2[u'Teff'] >= 5000) & (data2[u'Teff'] < 5500) &
              (data2[u'NUVmag']<20) & (data2[u'J']<16))
plt.scatter(data2[u'Prot'].values[ok],
            data2[u'NUVmag'].values[ok] - data2[u'J'].values[ok],
            s=5, alpha=0.35, c='red')
UVclr = np.zeros(len(Pbins) - 1)
for i in range(len(Pbins)-1):
    iok = np.where((data2[u'Prot'].values[ok] >= Pbins[i]) & (data2[u'Prot'].values[ok] < Pbins[i+1]))[0]
    UVclr[i] = np.median(data2[u'NUVmag'].values[ok][iok] - data2[u'J'].values[ok][iok])
    UVstd[i] = np.std(data2[u'NUVmag'].values[ok][iok] - data2[u'J'].values[ok][iok])
plt.errorbar((Pbins[1:] + Pbins[:-1])/2., UVclr, yerr=UVstd, c='red', lw=2)

plt.ylim(5,8)
plt.xlim(2e-1,1e2)
plt.ylabel('NUV - J (mag)')
plt.xscale('log')
plt.xlabel('Prot (days)')_____no_output_____Pbins = np.array([0.1, 1, 3, 7, 10, 20, 60])

ok = np.where((data2[u'Teff'] >= 5000) & (data2[u'Teff'] < 5600) &
              (data2[u'NUVmag'] < 20.) & np.isfinite(data2[u'V']))

plt.scatter(data2[u'Prot'].values[ok],
            data2[u'NUVmag'].values[ok] - data2[u'V'].values[ok],
            s=5, alpha=0.25)
UVclr = np.zeros(len(Pbins) - 1)
UVstd = np.zeros(len(Pbins) - 1)
for i in range(len(Pbins)-1):
    iok = np.where((data2[u'Prot'].values[ok] >= Pbins[i]) & (data2[u'Prot'].values[ok] < Pbins[i+1]))[0]
    UVclr[i] = np.median(data2[u'NUVmag'].values[ok][iok] - data2[u'V'].values[ok][iok])
    UVstd[i] = np.std(data2[u'NUVmag'].values[ok][iok] - data2[u'V'].values[ok][iok])
plt.errorbar((Pbins[1:] + Pbins[:-1])/2., UVclr, yerr=UVstd, lw=2)

# plt.ylim(5,8)
plt.xlim(2e-1,1e2)
plt.ylabel('NUV - V (mag)')
plt.xscale('log')
plt.xlabel('Prot (days)')
plt.text(0.3, 6.5, '5000 < Teff < 5600', fontsize=12)
plt.savefig('galex_rot_test1.png', dpi=150, bbox_inches='tight', pad_inches=0.25)_____no_output_____Pbins = np.array([0.1, 1, 3, 7, 10, 20, 60])

ok = np.where((data2[u'Teff'] >= 5600) & (data2[u'Teff'] < 6300) &
              (data2[u'NUVmag'] < 20.) & np.isfinite(data2[u'V']))

plt.scatter(data2[u'Prot'].values[ok],
            data2[u'NUVmag'].values[ok] - data2[u'V'].values[ok],
            s=5, alpha=0.25)
UVclr = np.zeros(len(Pbins) - 1)
UVstd = np.zeros(len(Pbins) - 1)
for i in range(len(Pbins)-1):
    iok = np.where((data2[u'Prot'].values[ok] >= Pbins[i]) & (data2[u'Prot'].values[ok] < Pbins[i+1]))[0]
    UVclr[i] = np.median(data2[u'NUVmag'].values[ok][iok] - data2[u'V'].values[ok][iok])
    UVstd[i] = np.std(data2[u'NUVmag'].values[ok][iok] - data2[u'V'].values[ok][iok])
plt.errorbar((Pbins[1:] + Pbins[:-1])/2., UVclr, yerr=UVstd, lw=2)

plt.ylim(3, 6.5)
plt.xlim(2e-1,1e2)
plt.ylabel('NUV - V (mag)')
plt.xscale('log')
plt.xlabel('Prot (days)')
plt.text(0.3, 5.85, '5600 < Teff < 6300', fontsize=12)
plt.savefig('galex_rot_test2.png', dpi=150, bbox_inches='tight', pad_inches=0.25)_____no_output_____Pbins = np.array([0.1, 3, 7, 10, 20, 60])

ok = np.where((data2[u'Teff'] >= 4000) & (data2[u'Teff'] < 5000) &
              (data2[u'NUVmag'] < 20.) & np.isfinite(data2[u'V']))

plt.scatter(data2[u'Prot'].values[ok],
            data2[u'NUVmag'].values[ok] - data2[u'V'].values[ok],
            s=5, alpha=0.25)
UVclr = np.zeros(len(Pbins) - 1)
UVstd = np.zeros(len(Pbins) - 1)
for i in range(len(Pbins)-1):
    iok = np.where((data2[u'Prot'].values[ok] >= Pbins[i]) & (data2[u'Prot'].values[ok] < Pbins[i+1]))[0]
    UVclr[i] = np.median(data2[u'NUVmag'].values[ok][iok] - data2[u'V'].values[ok][iok])
    UVstd[i] = np.std(data2[u'NUVmag'].values[ok][iok] - data2[u'V'].values[ok][iok])
plt.errorbar((Pbins[1:] + Pbins[:-1])/2., UVclr, yerr=UVstd, lw=2)

plt.ylim(4, 8.5)
plt.xlim(2e-1,1e2)
plt.ylabel('NUV - V (mag)')
plt.xscale('log')
plt.xlabel('Prot (days)')
plt.text(0.3, 7.5, '4000 < Teff < 5000', fontsize=12)
plt.savefig('galex_rot_test0.png', dpi=150, bbox_inches='tight', pad_inches=0.25)_____no_output_____

## Result so far:
The large scatter initially seen in (NUV - V) is due to more red stars populating the slower rotation periods.

BUT, when you bin in to smaller temperature ranges... you still don't see much significant evolution of (NUV - J).

(NUV - V) actually looks somewhat promising, e.g. see this last figure for just 1 temperature bin.

**Also we should recast this as Rossby number, not rotation period...**_____no_output_____