# Notebook from Guptacos/spotify_analysis
Path: reports/.ipynb_checkpoints/final_report-checkpoint.ipynb

# The Evolution of Popular Music
By Niko Gupta and Joel Miller_____no_output_____### Table of Contents
* [Introduction](#Introduction)
* [Data Collection](#Data-Collection-and-Cleaning)
    * [Getting Spotify Features](#Getting-Spotify-Data)
* [Analysis](#Analysis)
    * [Hot 100 Feature Breakdown](#Hot-100-Feature-Breakdown)
    * [Genre Consistency](#Genre-Consistency)
    * [Genre Distribution in Hot 100](#Genre-Distribution-in-Hot-100)
    * [Top Songs of All Time](#Top-Songs-of-All-Time)
    * [Time Spent in Hot 100](#Time-Spent-in-Hot-100)
    * [Conclusions](#Analysis-Conclusions)
* [ML Model](#Machine-Learning-Model)
    * [Model Optimization and Conclusions](#Model-Optimization-and-Conclusions)
* [Future Work](#Future-Work)
* [Further Reading](#Further-Reading)_____no_output_____## Introduction
This project is an exploratory project into the trends of popular music over the last 50 years. There are two primary goals:

* First, we aim to identify trends in popular music over the past five decades. In particular, we would like to see how features of songs have evolved over time both within the top 100 songs in the USA and within top genres. We are also interested in seeing how the popularity of different genres has evolved over time.

    This information could be useful for two reasons. First, it is interesting from a purely curiosity perspective. As music lovers, it's fascinating to see how music evolves over time. Second, historical trends are excellent indicators of future trends. Knowing how music has been evolving can help music executives such as record labels better identify music that might be popular in the future, and market appropriately to their audiences.\

    To tackle this problem, we pull historical data from the [Billboard 100](https://www.billboard.com/charts/hot-100) to identify top songs in the USA, both generally and for specific genres. We then use the [spotify developer api](https://developer.spotify.com/) to search for the songs and identify song features. Using this information, we identify interesting trends and draw conclusions about the evolution of music.

* Second, we train a model based on the above data that can predict the time period in which a song is most likely to have been popular, using features of that song. Outside of being an interesting problem to solve, this can be useful when recommending songs. If somebody likes a specific type of song, it's likely they would also like other songs that were popular around the same time as that song._____no_output_____## Data Collection and Cleaning
[Back to Top](#The-Evolution-of-Popular-Music)

The first task at hand is to collect the top 100 songs for the last years. This can be accomplished by using the billboard 100 API, and then saving the dataset._____no_output_____

import billboard as bb
import pickle
import time
import os
from os.path import isfile, join_____no_output_____def get_charts(genre='hot-100'):
    allCharts = []

    # get latest chart
    chart = bb.ChartData(genre)
    new = True
    prevYear = chart.date.split('-')[0]

    # songs thru 2000
    while chart.date > '1999-12-31':
        # add week to list if new chart
        if new:
            currYear = chart.date.split('-')[0]

            # checkpoint at the end of each year
            if currYear != prevYear:
                with open(genre + prevYear, 'wb') as f:
                    pickle.dump(allCharts, f)
                    print("Checkpoint for Year: " + prevYear)
                prevYear = currYear
                allCharts = []

            # add week's songs to the list
            songs = []
            for song in chart.entries:
                songs.append({"name": song.title, "artist": song.artist, "date": chart.date})

            allCharts.append(songs)

        # wait 5 seconds and try to get another chart to avoid overwhleming billboard servers
        time.sleep(5)
        print(chart.date)
        try:
            # try to get previous week's chart, if this is last chart, break
            if not chart.previousDate:
                break
            chart = bb.ChartData(genre, chart.previousDate)
            new = True
        except Exception as e:
            # if fail from connection loss or too many requests, ignore and try again
            print(type(e))
            new = False

    # save out last chart
    with open(genre + prevYear, 'wb') as f:
        pickle.dump(allCharts, f)
        print("Checkpoint for Year: " + prevYear)_____no_output_____

Now that we can get data for any genre, we need to get it for each genre we're interested in. One issue to note is that given 20 years, 52 weeks per year, and 5 seconds per request, we get about 86 minutes per genre. If we want to collect data for the hot-100 plus 8 other genres as below, this could take nearly 13 hours!

In addition, we can't run this function in 9 different processes, because the billboard API rate limits based on source IP (as opposed to using an authentication token). To get around this, we ran this script on 9 different VMs, allowing us to collect the data in under two hours._____no_output_____

genres = {'hot-100',
          'pop-songs',
          'country-songs',
          'rock-songs',
          'alternative-songs',
          'r-b-hip-hop-songs',
          'dance-electronic-songs',
          'christian-songs',
          'jazz-songs'}

def getHandles(f):
    return open(f, 'rb')

def combine_charts(chart):
    chart_files = [f for f in os.listdir(os.getcwd()) if (isfile(join(os.getcwd(), f)) and (chart+"Chart") in f)]
    files = map(getHandles, chart_files)

    allCharts = []
    for f in files:
        L = pickle.load(f)
        allCharts.extend(L)
        f.close()

    with open(chart+"-results", 'wb') as f:
        pickle.dump(allCharts, f)

def get_all_genres():
    for g in genres:
        get_charts(g)
        combine_charts(g)_____no_output_____

### Getting Spotify Data
Now that we have collected the data for each genre into a pickled file, we would like to augment it with Spotify information and turn it into a dataframe for easier processing.

We will be reading from a pickle file, which will contain a list of songs through the years. Each element of this list will be a year of top songs, which is a list of dicts containing date, name, and artist. This list should contain the Billboard 100 songs for every week for the last 50 years, or as many years are available for that genre.

Given this list, we will query the spotify API to generate a dataframe that contains the same information, augmented with song ID, song features, and artist genre. This dataframe will be saved as a pickle file at the end._____no_output_____

import spotipy
import spotipy.util as util

# Note: This should be filled with an oauth token from the spotify API.
# Make sure to delete it after use
# See https://developer.spotify.com/console/get-search-item/
#     and select 'get token' to generate a 1 hour token
token = ''

sp = spotipy.Spotify(auth=token)_____no_output_____

Let's create a cache of song mappings so as to limit our API requests to spotify._____no_output_____

# Format is dict{artist : {song : data, song : data ...} ...}
memo = dict()

# Use to calculate error rate
unique_unfound = 0
total_unfound = 0_____no_output_____

Now let's devise a way to get a song id, given a song and an artist:_____no_output_____

from unidecode import unidecode
from fuzzywuzzy import fuzz

TITLE_DIFF = 70
ARTIST_DIFF = 70

# Check if a search result artist list contains the artist of the song from Billboard 100
def contains_artist(goal, artists):
    # Use unidecode to turn special characters to regular, i.e. ñ --> n
    goal = unidecode(goal).lower()

    for artist in artists:
        art = unidecode(artist['name']).lower()
        if fuzz.partial_ratio(art, goal) >= ARTIST_DIFF:
            return True

    return False

# Get song data. If the song is not in the top 40 results, ignore it.
def get_song(track, artist, limit=40):
    global unique_unfound
    track = unidecode(track).lower()
    artist = unidecode(artist).lower()

    extra_search_words = {'is', 'a', 'i', 'the'}

    split1 = track.split()
    first = split1[0]
    if split1[0] in extra_search_words and len(split1) > 1:
        first += ' ' + split1[1]

    split2 = artist.split()
    second = split2[0]
    if split2[0] in extra_search_words and len(split2) > 1:
        second += ' ' + split2[1]

    to_search = first + ' ' + second
    query_result = sp.search(q=to_search, type='track', limit=40)
    query_result = query_result['tracks']['items']

    # Clean the query results
    result = []
    for elem in query_result:
        cur_name = unidecode(elem['name']).lower()
        if fuzz.partial_ratio(track, cur_name) < TITLE_DIFF:
            continue
        if not contains_artist(artist, elem['artists']):
            continue

        new_song = dict()
        new_song['name'] = cur_name
        new_song['id'] = elem['id']
        new_song['artist'] = artist
        new_song['popularity'] = elem['popularity']

        result += [new_song]

    # Return the most popular song
    result.sort(key=lambda d: d['popularity'])

    if len(result) == 0:
        # Uncomment the below line to see songs that failed
        #print("Could not find '%s' by '%s' in top %d results" % (track, artist, limit))
        unique_unfound += 1
        return None

    return result[-1]_____no_output_____

Create a wrapper to do it given a list of songs:_____no_output_____

import pandas as pd
import copy

CHUNK_SIZE = 50

def divide_chunks(l, n):
    result = []
    for i in range(0, len(l), n):
        if i >= len(l):
            result += [l[i:]]
        else:
            result += [l[i:i + n]]
    return result

# Here are all the features:
# danceability, energy, key, loudness, mode, speechiness, acousticness,
# instrumentalness, liveness, valence, tempo, type, id, uri, track_href,
# analysis_url, duration_ms, time_signature
def del_unneeded(d):
    del d['type']
    del d['id']
    del d['uri']
    del d['track_href']
    del d['analysis_url']
    del d['time_signature']

    return d

def combine_fn(a, b):
    a.update(b)
    return a

# Expects song_list of type [dict{week, name, artist}....]
def convert_all_songs(song_list):
    global unique_unfound
    global total_unfound

    unique_unfound = 0
    total_unfound = 0

    result_data = []
    for song in song_list:
        artist = unidecode(song['artist']).lower()
        title = unidecode(song['name']).lower()
        if artist not in memo:
            memo[artist] = dict()

        # Check if we've already hit the API for this song
        if title in memo[artist]:
            temp = copy.copy(memo[artist][title])
        else:
            # Note: spotipy takes care of the 429 'retry-after' response from spotify
            temp = get_song(title, artist)

        if temp is not None:
            temp['date'] = song['date']
            result_data += [temp]
        else:
            total_unfound += 1

        memo[artist][title] = temp

    # Batch the audio features into groups of 50, which is the max number you can query spotify at once
    song_list = divide_chunks(result_data, CHUNK_SIZE)
    result_features = []
    for sublist in song_list:
        sublist = list(map(lambda d: d['id'], sublist))

        # Add the audio features, remove unneeded one
        features = sp.audio_features(sublist)
        features = list(map(del_unneeded, features))
        result_features += features

    # merge the features into the other song data
    result = list(zip(result_data, result_features))
    result = list(map(lambda a: combine_fn(a[0], a[1]), result))

    return pd.DataFrame(result).set_index('date')_____no_output_____

Here is an example of the functionality:_____no_output_____

songs = [
    {'date':0, 'name':'gravy train', 'artist':'yung gravy'},
    {'date':1, 'name':'shape of you', 'artist':'ed sheeran'},
    {'date':0, 'name':'shouldn\'t exist', 'artist':'not a real artist'},
    {'date':1, 'name':'another day of sun', 'artist':'la la land cast'}
]
x = convert_all_songs(songs)
print(x)                    name                      id           artist  popularity  \
date
0            gravy train  3qOkkP9JLltoUvaLjEXTmW       yung gravy          69
1           shape of you  7qiZfU4dY1lWllzX7mPBI3       ed sheeran          87
1     another day of sun  5kRBzRZmZTXVg8okC7SJFZ  la la land cast          69

      danceability  energy  key  loudness  mode  speechiness  acousticness  \
date
0            0.733   0.622    6    -7.689     0       0.1400        0.3240
1            0.825   0.652    1    -3.183     0       0.0802        0.5810
1            0.588   0.742    8    -6.757     1       0.0528        0.0162

      instrumentalness  liveness  valence    tempo  duration_ms
date
0             0.000000    0.1650    0.729  143.934       169600
1             0.000000    0.0931    0.931   95.977       233713
1             0.000004    0.6530    0.824  125.819       228173

Cool! Note that the song "shouldn't exist" yielded no results in the output dataframe. Now that we have this functionality, lets run it on our data of top songs to create the end dataframe.

Another thing to note is that we ran the below code on our virtual machines as opposed to within this notebook, so there is no output shown below._____no_output_____

import pickle, time

for g in genres:
    with open(g + '-results', 'rb') as f:
        top_songs = pickle.load(f)

    top_songs = [song for year in top_songs for song in year]
    original_total = len(top_songs)

    start = time.time()
    result = convert_all_songs(top_songs)
    time_taken = time.time() - start
    error_rate = total_unfound / original_total

    print("Genre: %s, Error rate: %.2f, Time taken: %.2f s" %
          (g, error_rate, time_taken))

    with open(g + '.df', 'wb') as f:
        pickle.dump(result, f)_____no_output_____

The error rates are as follows from our data collection:
* hot100      = 5.99% error
* edm         = 2.81% error
* alternative = 1.19% error
* christian   = 2.93% error
* country     = 3.11% error
* jazz        = 8.19% error
* pop         = 0.88% error
* rnbHipHop   = 2.65% error
* rock        = 0.68% error

In this case error rate represents the percentage of the original Billboard 100 chart that was dropped due to not being able to match Spotify results. This may be due to spotify not having the song, or to different naming schemes leading to mismatched results_____no_output_____## Analysis
[Back to Top](#The-Evolution-of-Popular-Music)

Now that we have collected our data, we have 9 dataframes containing song data for up to the last 50 years. We would like to create some plots to identify trends in the data. The names of the features we are going to be analyzing were decided by spotify, and may not be immediately intuitive. While we will give some explanation on what each one represents, you can visit [this page](https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/) to get more in-depth definitions._____no_output_____

import matplotlib.pyplot as plt
import random
import numpy as np
from pandas.plotting import register_matplotlib_converters
register_matplotlib_converters()_____no_output_____

### Hot 100 Feature Breakdown
Lets try to answer our initial question: how has music evolved over time? Since we have music features for top songs for the last features, lets plot them to see how they've changed over time._____no_output_____

def plot_feature(df, feature, weeksPerPoint=156):
    aggregation_f = {'date': 'first', feature: np.mean}

    # Average the feature's value for each time period
    df['date'] = pd.to_datetime(df['date'])
    grouped = df.groupby('date')[feature].agg(np.mean).reset_index()

    # Aggregate weeks together to reduce noise in final plot
    sectionNum = []
    if weeksPerPoint > 1:
        num_sections = len(grouped) // weeksPerPoint
        for i in range(num_sections):
            sectionNum.extend([i]*weeksPerPoint)
        sectionNum.extend([num_sections]*(len(grouped) - num_sections*weeksPerPoint))
        grouped["section"] = sectionNum
        grouped = grouped.groupby('section').agg(aggregation_f).set_index('date')
    else:
        grouped = grouped.set_index('date')

    # Plot the feature
    plt.plot(grouped[feature],'o-', label=feature)
    plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left', fontsize='large')
    plt.tight_layout()

def plot_features(df, L):
    for f in L:
        plot_feature(df, f)

    plt.xlabel('Date')
    plt.ylabel('Spotify Feature Value')

    plt.show()_____no_output_____df = pickle.load(open('hot-100.df', 'rb')).reset_index()
plot_features(df, ['tempo'])
plot_features(df, ['key'])
plot_features(df, ['loudness'])_____no_output_____

Over the last 50 years, we can see that the average tempo and key of popular music have oscillated, but remained near the same mean. Loudness, however, has been steadily increasing over time, peaking around 2010._____no_output_____

plot_features(df, ['duration_ms'])_____no_output_____

However, the duration of songs spiked in the early 90s, before rapidly dropping back down. This is rather unexpected — it's interesting that music went from around 3:40 to 4:30, and then down to around 3:20._____no_output_____

plot_features(df, ['speechiness','acousticness','instrumentalness'])_____no_output_____

Now we start getting to the more interesting results. We can see that the speechiness of popular music has been steadily increasing, while the acousticness and instrumentalness has been decreasing. This could be explained by the increase in popularity of rap. Since rap in general has more spoken and fewer musical portions than other genres, such as rock, it could explain these findings._____no_output_____

plot_features(df, ['danceability','energy','mode','valence'])_____no_output_____

Here is another interesting plot. We can see that valence (happiness) has been largely decreasing since 1970. While this doesn't tell us anything about why, it's fascinating to note that music has been becoming less happy. Outside of a spike in the 2000s, the same is true of mode (modality, major or minor for a track).

On the flip side, music has been generally trending upwards in both energy and danceability, except in the last decade._____no_output_____### Genre Consistency
Given the above findings, we would like to investigate further to see if we can identify why music has trended the way it has. Perhaps popular genres have been changing or converging, which could explain the trends in indivual features. In order to answer this question, we will plot the feature distribution of top songs in each genre over time in order to see how the genres themselves have changed._____no_output_____

# radar.py and radar_factory are taken from https://matplotlib.org/3.1.0/gallery/specialty_plots/radar_chart.html
from radar import radar_factory
from numpy import random

features =['danceability', 'energy','mode', 'speechiness',
           'acousticness', 'instrumentalness', 'liveness', 'valence']

def get_grouped_chart(chart, weeksPerPoint=52):
    # Identify the features to plot and create aggregation functions
    aggregation_f = {'date': 'first'}

    for feature in features:
        aggregation_f[feature] = np.mean

    # Prepare the chart
    chart = chart.reset_index()
    chart['date'] = pd.to_datetime(chart['date'])
    grouped = chart.groupby('date')[features].agg(np.mean).reset_index()

    # Aggregate by the selected number of weeks
    sectionNum = []
    if weeksPerPoint > 1:
        num_sections = len(grouped) // weeksPerPoint
        for i in range(num_sections):
            sectionNum.extend([i]*weeksPerPoint)
        sectionNum.extend([num_sections]*(len(grouped) - num_sections*weeksPerPoint))
        grouped["section"] = sectionNum
        grouped = grouped.groupby('section').agg(aggregation_f).set_index('date')
    else:
        grouped = grouped.set_index('date')

    return grouped

def group_all_charts():
    charts = dict()
    for g in genres - {'hot-100'}:
        with open(g + '.df', 'rb') as f:
            chart = pickle.load(f)
        charts[g] = get_grouped_chart(chart)

    return charts_____no_output_____# Note that most genres have only 20 years of data
def plot_radars(num_graphs=4):
    interval = int(20 / num_graphs)

    charts = group_all_charts()

    # Create a plot for each time interval
    for i in range(2000, 2020, interval):
        N = len(features)
        theta = radar_factory(N, frame='polygon')
        data = []
        for _, chart in charts.items():
            genre = []
            for feature in features:
                v = chart[feature].loc['%d-01-01' % i:'%d-01-01' % (i+interval)].mean()
                genre.append(v)
            data.append(genre)

        spoke_labels = features
        fig = plt.figure()
        ax = fig.add_subplot(projection='radar')

        # Define colors and plot the four cases from the example data on separate axes
        colors = ['b', 'r', 'g', 'm', 'y', 'c', 'lime']
        ax.set_rgrids([0.2, 0.4, 0.6, 0.8])
        ax.set_title("Feature Distribution in " + str(i), weight='bold', size='medium', position=(0.5, 1.1),
                     horizontalalignment='center', verticalalignment='center')

        # Plot the data
        for d, color in zip(data, colors):
            ax.plot(theta, d, color=color)
            ax.fill(theta, d, facecolor=color, alpha=0.25)
        ax.set_varlabels(spoke_labels)

        # add legend relative to top-left plot
        labels = charts.keys()
        plt.legend(labels, bbox_to_anchor=(1.05, 1), loc='upper left', borderaxespad=0., fontsize='small')
        plt.tight_layout()

        plt.show()_____no_output_____plot_radars(4)_____no_output_____

As we can see from the above plots, genres have remained (fairly) consistent through the years. _____no_output_____### Genre Distribution in Hot 100
It seems that changes within genres can't really explain the feature trends we saw earlier in the hot 100. Another thing we can consider is the popularity of genres at any given time. If the genre breakdown of the hot 100 changes over time, that could explain the feature trends.

To do this, we will analyze the intersection popular songs between a genre and the top 100 over time. Using this, we can see how genres have evolved closer together or further apart, and how genres have gained prevalence within popular music (Billboard Top 100)_____no_output_____

from functools import reduce
# This function takes 2 datasets, and returns a set of points quantifying the number of overlapping songs

weeks_per_point = 53

def to_set(x):
    return set(x)

def prepare(df):
    # Turn the date index into a datetime object column
    df = df.reset_index()
    df['date'] = pd.to_datetime(df['date'])

    # Group the songs by the date they were popular and aggregate the IDs
    grouped = df.groupby('date')['id'].agg(to_set)

    # Aggregate the IDs into a smaller number of groups for easier comparison
    sections = []
    dates = []
    num_sections = len(grouped) // weeks_per_point
    for i in range(num_sections):
        dates += [grouped.reset_index()['date'][weeks_per_point * i]]
        if i + 1 == num_sections:
            sections += [list(grouped[weeks_per_point * i:])]
        else:
            sections += [list(grouped[weeks_per_point * i : weeks_per_point * (i + 1)])]

    # Combine the internal sets to make a list of sets
    sections = list(map(lambda a: reduce(lambda w,x: w.union(x), a), sections))

    return dates, sections

def overlap(df1, df2):
    dates1, df1 = prepare(df1)
    dates2, df2 = prepare(df2)

    # Only consider overlapping data
    if len(dates1) < len(dates2):
        dates2 = dates2[-1 * len(dates1):]
        df2    =    df2[-1 * len(dates1):]
    elif len(dates2) < len(dates1):
        dates1 = dates1[-1 * len(dates2):]
        df1    =    df1[-1 * len(dates2):]

    # Count the overlap
    result = list(zip(df1, df2))
    result = list(map(lambda a: len(a[0].intersection(a[1])), result))

    return dates1, result_____no_output_____from matplotlib.ticker import MaxNLocator

def plot_overlap():
    labels = genres - {'hot-100'}
    with open('hot-100.df', 'rb') as f:
        hot100 = pickle.load(f)

    for label in labels:
        with open(label + '.df', 'rb') as f:
            x, y = overlap(hot100, pickle.load(f))
            plt.plot(x,y)

    plt.legend(labels)
    plt.title('Number of Common Songs with Top 100')
    plt.xlabel('Year')
    plt.ylabel('# of Songs')

    plt.show()

plot_overlap()_____no_output_____

One thing to note is that in this plot, the y axis represents the number of unique songs that overlap with the top 100 over the course of a year, which is why it comes out to over 100.

Now these are interesting results! We can see that pop and country have become more popular since around 2017, and hip hop has been on the rist since 2011, beginning to spike up in 2015. This could help explain the trends we saw in danceability, since hip hop songs are generally more danceable than rock or christian songs._____no_output_____### Top Songs of All Time
Another interesting question to tackle is: what have been the most influential songs of all time? In order to answer this question, we will weight songs by the amount of time they spent in the hot 100, along with how high they were on the list._____no_output_____

def addWeights(df):
    # Define how features should aggregate
    aggregation_f = {'date': 'min', 'artist': 'first', 'name':'first', 'position':'sum'}

    # Invert position so that higher number signifies importance, and aggregate
    df['position'] = df['position'].apply(lambda x: 101-x)
    df = df.groupby(['name', 'artist']).agg(aggregation_f)

    return df

def getMostInfluentialSongs():
    with open('hot-100.df', 'rb') as f:
        hot = pickle.load(f)

    hot = addWeights(hot)
    hot = hot.sort_values('position', ascending=False)[['date', 'name', 'artist']]
    hot = hot.set_index('date')
    print(hot.head(10))

getMostInfluentialSongs()                                     name  \
date
2012-08-18                    radioactive
1995-09-02  macarena - bayside boys remix
1997-06-21                  how do i live
1996-11-30                  foolish games
2008-05-03                      i'm yours
2011-02-12              party rock anthem
2017-01-28                   shape of you
2013-07-06                 counting stars
2014-11-29                    uptown funk
2010-12-25            rolling in the deep

                                               artist
date
2012-08-18                            imagine dragons
1995-09-02                                los del rio
1997-06-21                                leann rimes
1996-11-30                                      jewel
2008-05-03                                 jason mraz
2011-02-12  lmfao featuring lauren bennett & goonrock
2017-01-28                                 ed sheeran
2013-07-06                                onerepublic
2014-11-29           mark ronson featuring bruno mars
2010-12-25                                      adele

This makes sense — a lot of these are songs that we've all heard of. However, it's interesting to note that a lot of the most influential songs seem to be newer songs. Let's investigate why this may be._____no_output_____### Time Spent in Hot 100
To investigate this, we will look at the most popular songs (songs that hit the top 10), and see how long (on average) they spent in the top 100._____no_output_____

from statistics import mean

# Want the songs above top_cuttoff, and want to see how long they were in the top length_in
TOP_CUTTOFF = 10
LENGTH_IN = 100

# returns a dict mapping a year to a set of songs that were in the top n at some point that year
def get_uniques(top_df):
    topN = top_df[top_df['position'] <= TOP_CUTTOFF]
    uniques = topN.groupby(topN.date.dt.year)['name'].apply(set)

    return dict(uniques)_____no_output_____def song_count_top(top_df, song, year):
    result = top_df[top_df['date'].dt.year == year]
    result = result[result['name'] == song]
    result = result[result['position'] <= LENGTH_IN]
    return len(result)

def song_average_top(top_df, songs, year):
    total = 0
    for song in songs:
        total += song_count_top(top_df, song, year)

    return total / len(songs)

def yearly_average(top_df):
    result = []
    uniques = get_uniques(top_df)

    for year, songs in uniques.items():
        avg = song_average_top(top_df, songs, year)
        result += [(year, avg)]

    return result_____no_output_____with open('hot-100.df', 'rb') as f:
    df = pickle.load(f)

df['date'] = pd.to_datetime(df['date'])
avg = yearly_average(df)_____no_output_____def divide_chunks(l, n):
    result = []
    for i in range(0, len(l), n):
        if i >= len(l):
            result += [l[i:]]
        else:
            result += [l[i:i + n]]
    return result

def plot_avg(smoothing=1):
    years    = [elem[0] for elem in avg[::smoothing]]
    averages = [elem[1] for elem in avg]

    data = divide_chunks(averages, smoothing)
    data = list(map(mean, data))

    plt.plot(years, data)
    plt.xlabel('Year')
    plt.ylabel('Weeks')
    plt.title('Average Number of Weeks in Top %s' % LENGTH_IN)

    plt.show()_____no_output_____plot_avg(2)_____no_output_____

As we can see, on average, the amount of time popular songs spend in the top 100 has been increasing. Therefore it makes sense that the most popular songs of our data set came from more recent years. Let's also look at the average trajectory of a song through the top 100, averaged by decade:_____no_output_____

def get_trajectory(df, song):
    df = df.reset_index()
    df = df[df['name'] == song].sort_values('date')

    yr = df['date'].agg(min).year
    pos = list(df['position'])
    return (yr, pos)

def get_unique_set(df):
    result = set()
    for _, v in get_uniques(df).items():
        result = result.union(v)
    return result

def get_yearly_trajectories(df):
    s = get_unique_set(df)
    result = dict()
    for song in s:
        yr, pos = get_trajectory(df, song)
        if yr not in result:
            result[yr] = []

        result[yr] += [pos]

    return result_____no_output_____# L is a list of lists, where the internal lists are the positions of a song in the charts
# It will append zeros to make them all the same length, then find the mean values.
import numpy as np
def get_mean(L):
    max_len = len(max(L, key=len))
    for elem in L:
        elem += [100] * (max_len - len(elem))

    return np.transpose(L).mean(axis=1)

batch_size = 10
def get_all_means(d):
    # batch to reduce noise
    yrs = list(d.keys())
    yrs.sort()
    batches = divide_chunks(yrs, batch_size)
    batch_dict = dict()
    for batch in batches:
        batch_dict[batch[0]] = []
        for y in batch:
            batch_dict[batch[0]] += d[y]

    result = dict()
    for year, pos in batch_dict.items():
        result[year] = get_mean(pos)

    return result_____no_output_____def plot_all_traj(df, limit=500):
    yearly_traj = get_yearly_trajectories(df)
    means = get_all_means(yearly_traj)

    lables = []
    for yr, pos in means.items():
        lables += [yr]
        x_coord = list(range(0, len(pos)))
        if len(x_coord) > limit:
            x_coord = x_coord[:limit]
            pos = pos[:limit]
        plt.plot(x_coord, pos)

    plt.gca().invert_yaxis()
    plt.xlabel('Weeks')
    plt.ylabel('Position')
    plt.legend(lables)
    plt.title('Position in hot 100 over time')
    plt.show()_____no_output_____plot_all_traj(df, 70)_____no_output_____

This supports our earlier conclusions that in recent years, songs spend more time in the hot 100. We can see that as the decades progress in the above graph, songs drop out of the charts more slowly. We can also see that in the 2010s, songs on average didn't break the top 20, meaning that the top 20 songs were more constant and didn't get knocked down as easily._____no_output_____### Analysis Conclusions
In conclusion, we can see that music is generally becoming louder and more danceable, but also sadder and less acoustic. Genres have remained fairly consistent, but the distribution of genres within the top 100 has begun to skew heavily towards hip hop. This may help to explain the trends in danceability and other features that we found. In addition, when songs become popular, they stay popular for longer than they used to. This could be due to any number of reasons, such as increased advertising to keep songs popular, larger consumption of media, less original music, etc. Further research would need to be done to help explain this trend._____no_output_____## Machine Learning Model
[Back to Top](#The-Evolution-of-Popular-Music)

One interesting application of the trends we've analyzed is to use them to train a machine learning model to predict the decade of a song based on its analytic musical features. We can exploit the fact that these features have varied significantly over the past 50 years to allow for improved model performance. We can use this in recommender systems to recommend songs from a similar time period to users.

To do this, we will use and compare three different models: a Decision Tree, a Random Forest, and a Gradient-Boosted Decision Tree. These are all tree based models, which allow for better human comprehension and easier training. We will use all analytic features of the song as predictors, while using the year of the week that the song first reached the Hot 100 to determine the song's decade._____no_output_____

from sklearn.tree import DecisionTreeClassifier
import xgboost as xgb
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn import metrics_____no_output_____feature_cols =['danceability', 'energy', 'key', 'loudness', 'mode',
           'speechiness', 'acousticness', 'instrumentalness',
           'liveness', 'valence', 'tempo', 'duration_ms']

def prepare_data(data):
    # Declare aggregation functions for cleaning the data
    aggregation_f = {'date': 'min', 'artist': 'first', 'name':'first', 'position':'first'}
    for feature in feature_cols:
        aggregation_f[feature] = 'first'

    with open(data, 'rb') as f:
        df = pickle.load(f)

    df = df.groupby(['name', 'artist']).agg(aggregation_f)

    return df_____no_output_____def getYearRange(date, yearsPerClass=10):
    year = date[:4]
    beginYear = ((int(year)-1970)//yearsPerClass * yearsPerClass) + 1970
    return f"{beginYear}-{beginYear+yearsPerClass-1}"

def compare_models(train_features, test_features, train_labels, test_labels, weights=None, verbose=True):
    if verbose:
        if weights is not None:
            print("With weights")
        else:
            print("Without weights")

    #decision tree
    DT = DecisionTreeClassifier()
    DT = DT.fit(train_features, train_labels, sample_weight=weights)
    label_pred = DT.predict(test_features)
    if verbose:
        print("Decision tree accuracy: %.2f" % metrics.accuracy_score(test_labels, label_pred))

    #random forest
    RF = RandomForestClassifier()
    RF = RF.fit(train_features, train_labels, sample_weight=weights)
    label_pred = RF.predict(test_features)
    if verbose:
        print("Random forest accuracy: %.2f" % metrics.accuracy_score(test_labels, label_pred))

    #gb decision tree
    XGB = xgb.XGBClassifier(objective='merror')
    XGB = XGB.fit(train_features, train_labels, sample_weight=weights)
    label_pred = XGB.predict(test_features)
    if verbose:
        print("GB decision tree accuracy: %.2f" % metrics.accuracy_score(test_labels, label_pred))

        print("\nDecision tree feature importance: ", list(zip(feature_cols[:-1],DT.feature_importances_)))
        print("\nRandom forest feature importance: ", list(zip(feature_cols[:-1],RF.feature_importances_)))
        print("\nGB decision tree feature importance: ", list(zip(feature_cols[:-1],XGB.feature_importances_)))

    return (DT, RF, XGB)_____no_output_____def run_unweighted_compare(df):
    features = df[feature_cols]
    labels = df['date'].apply(getYearRange).values

    # Split data into train and test
    train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size=.2)
    train_features = train_features.values
    test_features = test_features.values
    compare_models(train_features, test_features, train_labels, test_labels)

run_unweighted_compare(prepare_data('hot-100.df'))Decision tree accuracy: 0.43
Random forest accuracy: 0.55
GB decision tree accuracy: 0.56

Decision tree feature importance:  [('danceability', 0.08093738090615889), ('energy', 0.0714019165446743), ('key', 0.03891476491596118), ('loudness', 0.1554265815923132), ('mode', 0.006543047624778649), ('speechiness', 0.08787544140155615), ('acousticness', 0.08071284718516432), ('instrumentalness', 0.055472976158283914), ('liveness', 0.05613844713589032), ('valence', 0.09805078767188648), ('tempo', 0.07251152768016011), ('duration_ms', 0.1132028130774915)]

Random forest feature importance:  [('danceability', 0.07888380534256186), ('energy', 0.07645977604886323), ('key', 0.04108205943678146), ('loudness', 0.14386648893534915), ('mode', 0.010805529457839346), ('speechiness', 0.08898204086312317), ('acousticness', 0.08294058939215893), ('instrumentalness', 0.0641072304614849), ('liveness', 0.06383871635916945), ('valence', 0.09466913060648385), ('tempo', 0.07255759722281767), ('duration_ms', 0.10309709621888802)]

GB decision tree feature importance:  [('danceability', 0.068110235), ('energy', 0.046318144), ('key', 0.01884292), ('loudness', 0.19153368), ('mode', 0.014174102), ('speechiness', 0.090577416), ('acousticness', 0.0644392), ('instrumentalness', 0.09217658), ('liveness', 0.03227352), ('valence', 0.09762645), ('tempo', 0.04454843), ('duration_ms', 0.10811651)]

As we can see, the Random Forest and Gradient-Boosted decision trees were much more effective at prediciting the decade of the song. The features that were most important were the song's loudness, duration, and valence. However, let's see if we can improve the performance by weighting the songs based on 1) how high they were on the Hot 100 and 2) how long they remained on the Hot 100. This may improve performance since songs that ranked highly and remained on the list may be more representative of the decade as a whole._____no_output_____

def prepare_weighted_data(data):
    global feature_cols

    if 'position' not in feature_cols:
        feature_cols += ['position']

    # Declare aggregation functions for cleaning the data
    aggregation_f = {'date': 'min', 'artist': 'first', 'name':'first', 'position':'sum'}
    for feature in feature_cols:
        aggregation_f[feature] = 'first'

    with open(data, 'rb') as f:
        df = pickle.load(f)

    df['position'] = df['position'].apply(lambda x: 101-x)
    df = df.groupby(['name', 'artist']).agg(aggregation_f)

    return df_____no_output_____def get_train_test_data(df):
    features = df[feature_cols]
    labels = df['date'].apply(getYearRange).values

    # Split data into train and test
    train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size=.2)
    weights = train_features['position'].values
    train_features = train_features.drop('position', axis=1).values
    test_features = test_features.drop('position', axis=1).values

    return train_features, test_features, train_labels, test_labels

def run_weighted_compare(df):

    compare_models(train_features, test_features, train_labels, test_labels, weights)

run_weighted_compare(prepare_weighted_data('hot-100.df'))_____no_output_____

Based on the model performance, it appears adding weights to the training data did not help improve the models. This may be due to the similarity between top songs regardless of decade or it may be true that songs that remain on the top 100 for a longer period of time are NOT more representative of the time period. Let's plot the confusion matrices to see where the models are going wrong._____no_output_____

def showConfusionMatrix(model, title):
    decades = ['70s', '80s', '90s', '00s', '10s']
    train_features, test_features, train_labels, test_labels = get_train_test_data(df)
    disp = metrics.plot_confusion_matrix(model, test_features, test_labels, display_labels=decades, normalize='true')
    disp.ax_.set_title(title)

def run_confusion():
    df = prepare_data('hot-100.df')
    train_features, test_features, train_labels, test_labels = get_train_test_data(df)
    DT, RF, XGB = compare_models(train_features, test_features, train_labels, test_labels, weights=None, verbose=False)

    showConfusionMatrix(DT, 'Decision Tree')
    showConfusionMatrix(RF, 'Random Forest')
    showConfusionMatrix(XGB, 'Gradient Boosted Decision Tree')
    plt.show()

run_confusion()_____no_output_____

As we can see, the model is mainly making errors between adjacent decades, which makes a lot sense. Many of these songs may have been on the border of a decade (i.e. 1990 or 1999), so the features of that song may be more represenative of the adjacent decade since it is so close. It can even be difficult for humans to tell exactly what decade a song is from! Thus, the last effort we will make to improve model performance is by tuning the hyperparameters through grid search._____no_output_____### Model Optimization and Conclusions
TODO Joel_____no_output_____

from sklearn.model_selection import KFold, GridSearchCV
import matplotlib.patches as mpatches_____no_output_____def maxDepthSearch(features, labels, n_splits, vals):
    kf = KFold(n_splits)
    accs = [0]*len(vals)
    for j, (train_idx, test_idx) in enumerate(kf.split(features)):
        for i, val in enumerate(vals):
            DT = DecisionTreeClassifier(max_depth=val)
            DT = DT.fit(features[train_idx], labels[train_idx])
            label_pred = DT.predict(features[test_idx])
            accs[i]+= metrics.accuracy_score(labels[test_idx], label_pred)
        print(f"Split {j}: complete")

    for i in range(len(accs)):
        accs[i] = accs[i]/n_splits*100
        print(f"Accuracy for {vals[i]}: {accs[i]}")

    return accs

def maxFeaturesSearch(features, labels, n_splits, vals):
    kf = KFold(n_splits)
    accs = [0]*len(vals)
    for j, (train_idx, test_idx) in enumerate(kf.split(features)):
        for i, val in enumerate(vals):
            DT = DecisionTreeClassifier(max_features=val)
            DT = DT.fit(features[train_idx], labels[train_idx])
            label_pred = DT.predict(features[test_idx])
            accs[i]+= metrics.accuracy_score(labels[test_idx], label_pred)
        print(f"Split {j}: complete")

    for i in range(len(accs)):
        accs[i] = accs[i]/n_splits*100
        print(f"Accuracy for {vals[i]}: {accs[i]}")

    return accs

def minImpuritySearch(features, labels, n_splits, vals):
    kf = KFold(n_splits)
    accs = [0]*len(vals)
    for j, (train_idx, test_idx) in enumerate(kf.split(features)):
        for i, val in enumerate(vals):
            DT = DecisionTreeClassifier(min_impurity_decrease=val)
            DT = DT.fit(features[train_idx], labels[train_idx])
            label_pred = DT.predict(features[test_idx])
            accs[i]+= metrics.accuracy_score(labels[test_idx], label_pred)
        print(f"Split {j}: complete")

    for i in range(len(accs)):
        accs[i] = accs[i]/n_splits*100
        print(f"Accuracy for {vals[i]}: {accs[i]}")

    return accs

max_depth = [3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 16, 18, 20, 25, 30]
feats = [None, 'sqrt', 'log2']
min_impurity_dec = [0., 1e-9, 1e-8, 1e-7, 1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1]

# Create three plots for different hyper parameters
def individualSearch(features, labels):
    depth_accs = maxDepthSearch(features, labels, 10, max_depth)
    plt.plot(max_depth, depth_accs)
    plt.xlabel("Max Depth Parameter")
    plt.ylabel("% Accuracy")
    plt.title("Accuracy vs. Max Tree Depth")
    plt.figure()

    feats_accs = maxFeaturesSearch(features, labels, 10, feats)
    feats[0] = "None"
    plt.bar(feats, feats_accs)
    plt.xlabel("Max Features Parameter")
    plt.ylabel("% Accuracy")
    plt.title("Accuracy vs. Max Features")
    plt.figure()

    impurity_accs = minImpuritySearch(features, labels, 10, min_impurity_dec)
    plt.plot(min_impurity_dec, impurity_accs)
    plt.xscale('log')
    plt.xlabel("Min Impurity Decrease Parameter")
    plt.ylabel("% Accuracy")
    plt.title("Accuracy vs. Min Impurity Decrease Parameter")
    plt.show()_____no_output_____def get_features_labels():
    df = prepare_data('hot-100.df')
    if 'position' in feature_cols:
        feature_cols.remove('position')

    features = df[feature_cols].values
    labels = df['date'].apply(getYearRange).values
    return features, labels

individualSearch(*get_features_labels())Split 0: complete
Split 1: complete
Split 2: complete
Split 3: complete
Split 4: complete
Split 5: complete
Split 6: complete
Split 7: complete
Split 8: complete
Split 9: complete
Accuracy for 3: 41.27591706539075
Accuracy for 4: 42.876129718234985
Accuracy for 5: 45.48644338118023
Accuracy for 6: 46.95906432748537
Accuracy for 7: 47.57575757575758
Accuracy for 8: 47.963849016480594
Accuracy for 9: 47.7618288144604
Accuracy for 10: 47.18234981392876
Accuracy for 12: 45.25252525252525
Accuracy for 14: 43.76395534290272
Accuracy for 16: 42.58904837852205
Accuracy for 18: 41.30781499202553
Accuracy for 20: 41.14832535885167
Accuracy for 25: 40.877192982456144
Accuracy for 30: 40.68048910154173
Split 0: complete
Split 1: complete
Split 2: complete
Split 3: complete
Split 4: complete
Split 5: complete
Split 6: complete
Split 7: complete
Split 8: complete
Split 9: complete
Accuracy for None: 40.8080808080808
Accuracy for sqrt: 40.13822434875067
Accuracy for log2: 40.77618288144605
Split 0: complete
Split 1: complete
Split 2: complete
Split 3: complete
Split 4: complete
Split 5: complete
Split 6: complete
Split 7: complete
Split 8: complete
Split 9: complete
Accuracy for 0.0: 40.73365231259968
Accuracy for 1e-09: 40.63795853269537
Accuracy for 1e-08: 41.02073365231259
Accuracy for 1e-07: 40.590111642743224
Accuracy for 1e-06: 40.97288676236045
Accuracy for 1e-05: 40.90909090909092
Accuracy for 0.0001: 43.407761828814465
Accuracy for 0.001: 46.44338118022329
Accuracy for 0.01: 38.07549175970229
Accuracy for 0.1: 23.96597554492291
Accuracy for 1: 23.96597554492291

We notice that max depth has a definite optimal value at max_depth=8. Since it would take much too long to run gridSearch over 3 different values, let's set max_depth to 8 and run grid_search over the other two hyperparameters. Then, we will select the set that gives the best performance._____no_output_____

def gridSearch(features, labels, n_splits, min_impurity_dec, max_feats):
    kf = KFold(n_splits)
    accs = [[0 for i in range(len(max_feats))] for j in range(len(min_impurity_dec))]
    for i, (train_idx, test_idx) in enumerate(kf.split(features)):
        for j, dec in enumerate(min_impurity_dec):
            for k, feat in enumerate(max_feats):
                DT = DecisionTreeClassifier(max_depth=8, min_impurity_decrease=dec, max_features=feat)
                DT = DT.fit(features[train_idx], labels[train_idx])
                label_pred = DT.predict(features[test_idx])
                accs[j][k]+= metrics.accuracy_score(labels[test_idx], label_pred)
        print(f"Split {i}: complete")

    for i in range(len(min_impurity_dec)):
        for j in range(len(max_feats)):
            accs[i][j] = accs[i][j]/n_splits*100
            print(f"Accuracy for ({min_impurity_dec[i]},{max_feats[j]}): {accs[i][j]}")

    feats[0] = 'none'
    indices = list(map(str, min_impurity_dec))
    colors = ['b', 'r', 'g']
    for x, fN, fS, fL in zip(indices, [accs[i][0] for i in range(len(accs))], [accs[i][1] for i in range(len(accs))], [accs[i][2] for i in range(len(accs))]):
        for i, (h, c) in enumerate(sorted(zip([fN, fS, fL], colors))):
            plt.bar(x, h,color=c, alpha=1, zorder=-i)

    plt.legend(labels)
    plt.legend(handles=[mpatches.Patch(color='b', label='Max Features=None'), mpatches.Patch(color='r', label='Max Features=Sqrt'), mpatches.Patch(color='g', label='Max Features=Log2')])
    plt.xlabel("Min Impurity Decrease Parameter")
    plt.ylabel("% Accuracy")
    plt.show()

gridSearch(*get_features_labels(), 5, min_impurity_dec, feats)Split 0: complete
Split 1: complete
Split 2: complete
Split 3: complete
Split 4: complete
Accuracy for (0.0,None): 47.52259436469963
Accuracy for (0.0,sqrt): 46.15098351940457
Accuracy for (0.0,log2): 45.364167995746946
Accuracy for (1e-09,None): 47.54385964912281
Accuracy for (1e-09,sqrt): 45.672514619883046
Accuracy for (1e-09,log2): 44.61988304093567
Accuracy for (1e-08,None): 47.53854332801701
Accuracy for (1e-08,sqrt): 45.24189261031366
Accuracy for (1e-08,log2): 46.19351408825093
Accuracy for (1e-07,None): 47.52791068580542
Accuracy for (1e-07,sqrt): 45.39606592238171
Accuracy for (1e-07,log2): 45.061137692716635
Accuracy for (1e-06,None): 47.52259436469963
Accuracy for (1e-06,sqrt): 45.30037214247741
Accuracy for (1e-06,log2): 45.07708665603403
Accuracy for (1e-05,None): 47.496012759170654
Accuracy for (1e-05,sqrt): 45.26847421584264
Accuracy for (1e-05,log2): 45.34290271132376
Accuracy for (0.0001,None): 47.51196172248804
Accuracy for (0.0001,sqrt): 45.62466772993089
Accuracy for (0.0001,log2): 45.35885167464115
Accuracy for (0.001,None): 46.43806485911749
Accuracy for (0.001,sqrt): 42.71132376395535
Accuracy for (0.001,log2): 42.54120148856991
Accuracy for (0.01,None): 38.261562998405104
Accuracy for (0.01,sqrt): 34.04572036150984
Accuracy for (0.01,log2): 32.525252525252526
Accuracy for (0.1,None): 23.965975544922916
Accuracy for (0.1,sqrt): 23.965975544922916
Accuracy for (0.1,log2): 23.965975544922916
Accuracy for (1,None): 23.965975544922916
Accuracy for (1,sqrt): 23.965975544922916
Accuracy for (1,log2): 23.965975544922916

As we can see, max_features is best when set to None, so using all features is better for this case. We also see the min_impurity_decrease parameter does not effect the output much until after .001, so let's set it to 1e-5. Now we will run the tests one more time to see if these hyperparameters really affected our performance._____no_output_____

def rf_grid_search():
    df = prepare_data('hot-100.df')

    features = df[feature_cols]
    labels = df['date'].apply(getYearRange).values

    # Split data into train and test
    train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size=.2)
    train_features = train_features.values
    test_features = test_features.values

    max_depth = [8, 12, 16, 20]
    num_estimators = [60, 100, 140, 200]

    grid = {
        'max_depth': max_depth,
        'n_estimators': num_estimators
    }

    RF = RandomForestClassifier()
    clf = GridSearchCV(RF, grid, n_jobs=-1, verbose=1)
    clf.fit(train_features, train_labels)
    print("Best found parameters: ", clf.best_params_)
    label_pred = clf.predict(test_features)
    print("Accuracy with best parameters: ", metrics.accuracy_score(test_labels, label_pred))

rf_grid_search()Fitting 5 folds for each of 16 candidates, totalling 80 fits

Also, sklearn has a convenient grid search with cross validation function that we can run to determine the best hyperparameters for the Random Forest model._____no_output_____

def compare_optimized_models(train_features, test_features, train_labels, test_labels, weights=None, verbose=True):
    if verbose:
        if weights is not None:
            print("With weights")
        else:
            print("Without weights")

    #decision tree
    DT = DecisionTreeClassifier(max_depth=8, max_features=None, min_impurity_decrease=1e-3)
    DT = DT.fit(train_features, train_labels, sample_weight=weights)
    label_pred = DT.predict(test_features)
    if verbose:
        print("Decision tree accuracy: %.2f" % metrics.accuracy_score(test_labels, label_pred))

    #random forest
    RF = RandomForestClassifier(n_estimators=160, max_depth=16)
    RF = RF.fit(train_features, train_labels, sample_weight=weights)
    label_pred = RF.predict(test_features)
    if verbose:
        print("Random forest accuracy: %.2f" % metrics.accuracy_score(test_labels, label_pred))

    #gb decision tree
    XGB = xgb.XGBClassifier(objective='merror')
    XGB = XGB.fit(train_features, train_labels, sample_weight=weights)
    label_pred = XGB.predict(test_features)
    if verbose:
        print("GB decision tree accuracy: %.2f" % metrics.accuracy_score(test_labels, label_pred))

        print("\nDecision tree feature importance: ", list(zip(feature_cols[:-1],DT.feature_importances_)))
        print("\nRandom forest feature importance: ", list(zip(feature_cols[:-1],RF.feature_importances_)))
        print("\nGB decision tree feature importance: ", list(zip(feature_cols[:-1],XGB.feature_importances_)))

    return (DT, RF, XGB)

def run_optimized_compare(df):
    features = df[feature_cols]
    labels = df['date'].apply(getYearRange).values

    # Split data into train and test
    train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size=.2)
    train_features = train_features.values
    test_features = test_features.values
    compare_optimized_models(train_features, test_features, train_labels, test_labels)

run_optimized_compare(prepare_data('hot-100.df'))Without weights
Decision tree accuracy: 0.46
Random forest accuracy: 0.54
GB decision tree accuracy: 0.53

Decision tree feature importance:  [('danceability', 0.03838594737683218), ('energy', 0.011599839922801562), ('key', 0.0), ('loudness', 0.46403944107419837), ('mode', 0.0), ('speechiness', 0.10130967450521328), ('acousticness', 0.02299273102328616), ('instrumentalness', 0.06558509943765196), ('liveness', 0.0), ('valence', 0.1307579754493068), ('tempo', 0.007146039791547014)]

Random forest feature importance:  [('danceability', 0.0871161505458522), ('energy', 0.08158180472420196), ('key', 0.04045761392969015), ('loudness', 0.16129180720416073), ('mode', 0.010565616696819694), ('speechiness', 0.09681118114493349), ('acousticness', 0.08933911612134994), ('instrumentalness', 0.06784434210240356), ('liveness', 0.06693033266229262), ('valence', 0.10409038631403097), ('tempo', 0.0781098138006978)]

GB decision tree feature importance:  [('danceability', 0.07694352), ('energy', 0.058770966), ('key', 0.012413373), ('loudness', 0.2328289), ('mode', 0.0), ('speechiness', 0.09605001), ('acousticness', 0.07151376), ('instrumentalness', 0.11634862), ('liveness', 0.039223593), ('valence', 0.10480155), ('tempo', 0.052682713)]

While the hyperparameters did help the Decision Tree, we can see that tuning the hyperparameters did not effect the output of Random Forest much unfortunately, but this was a very difficult problem and we found reasonable success! 55% success is much better than random chance (20%)._____no_output_____## Future Work
[Back to Top](#The-Evolution-of-Popular-Music)

While our exploratory analysis helped to answer many questions, it also raised many more that merit further research. Further questions we have include:
* Why have the features of popular songs trended the way they have?
* What historical events correspond to the shifts in music we identified? Is there a way to identify causation, or is correlation all we can hope for?
* How have specific artists and songs influenced the trends of popular music? If there is a way to identify such artists, and quantify their importance, who would the most influential artists be?

Other ideas of project extensions:
* Demographic Breakdown within popular genres
    * By finding out which demographics (location, age, etc.) listen to specific genres, and by knowing towards which genres popular music is trending, music professionals can develop more targeted advertising.
* Make the collection and analysis portion more streamlined and pipelined so that it can be repeated more easily.
    * Repeat the analysis both globally and for other countries.
* Popularity predictor
    * Using past song data as features, train a model to predict the chance that a song will become popular. Note that this project is likely _very_ difficult to do accurately, though if you manage to pull it off you are sure to make a lot of money.
* Recreate Spotify's prediction algorithm to predict new songs to listen to.
* Track the overlap between genres, like we did between the hot 100 and each genre. This would allow us to see how genres changed, to see if they converged or diverged.
* Track artists' success by calculating their trajectory through the top charts_____no_output_____## Further Reading
* [Spotify for Developers](https://developer.spotify.com/)
* [Billboard 100 Charts](https://www.billboard.com/charts/hot-100)
* [Other Projects](https://towardsdatascience.com/tagged/spotify)
* [Analytics at Spotify](https://labs.spotify.com/2013/05/13/analytics-at-spotify/)
* [Spotify Blog](https://labs.spotify.com/)
    * This blog has many interesting posts about data science at Spotify, along with their tech stack in general.

[Back to Top](#The-Evolution-of-Popular-Music)_____no_output_____


# Notebook from SummitChen/ComputationalAdvertisement
Path: L16/marketbasket/apriori.ipynb

<code>
import pandas as pd
import time
import pprint
import tqdm

pd.options.display.max_columns = 100_____no_output_____# load data
data = pd.read_csv('./Market_Basket_Optimisation.csv', header=None)_____no_output_____data_____no_output_____# convert data to lower cases
for col in data.columns:
    data[col] = data[col].str.lower()

# fill nan
data = data.fillna(-1)_____no_output_____def rule_apriori():
    from efficient_apriori import apriori
    pp = pprint.PrettyPrinter()
    start = time.time()
    # extract transactions from dataframe
    transactions = []
    for i, r in data.iterrows():
        temp_set = set()
        for col in data.columns:
            if r[col] != -1:
                temp_set.add(r[col])
        if len(temp_set) > 0:
            transactions.append(temp_set)

    # print(transactions)
    # mine frequent item sets and frequent correlative rules
    itemset, rules = apriori(transactions, min_support=0.03, min_confidence=0.3)

    print('frequent items')
    pp.pprint(itemset)
    print('correlative rules')
    pp.pprint(rules)

    end = time.time()
    print('time spent', end - start)_____no_output_____def rule_mlxtend():

    from mlxtend.frequent_patterns import apriori
    from mlxtend.frequent_patterns import association_rules
    pp = pprint.PrettyPrinter()

    start = time.time()

    # reconstruct columns
    items = set()
    for i, r in data.iterrows():
        for col in data.columns:
            if r[col] != -1:
                items.add(r[col])

    hot_encoded_df = pd.DataFrame(columns = items)

    # copy and convert data into new format
    for i, r in tqdm.tqdm(data.iterrows()):
        temp_set = set()
        temp_dict = dict()
        for col in data.columns:
            if r[col] != -1:
                temp_set.add(r[col])
        for item in items:
            if item in temp_set:
                temp_dict[item] = [1]
            else:
                temp_dict[item] = [0]
        hot_encoded_df = hot_encoded_df.append(pd.DataFrame(temp_dict), ignore_index = True)

    # apply apriori to extract frquent item sets and correlative rules
    frequent_itemsets = apriori(hot_encoded_df, min_support=0.02, use_colnames=True)
    rules = association_rules(frequent_itemsets, metric="lift", min_threshold=0.3)
    print("requent item sets：")
    pp.pprint(frequent_itemsets)
    print("correlative rules:")
    pp.pprint(rules[(rules['lift'] >= 1) & (rules['confidence'] >= 0.3)].sort_values(by=['lift'], ascending = False))

    end = time.time()
    print("time spent: ", end - start)_____no_output_____rule_apriori()
print('-'*100)
rule_mlxtend()frequent items
{1: {('avocado',): 250,
     ('brownies',): 253,
     ('burgers',): 654,
     ('butter',): 226,
     ('cake',): 608,
     ('champagne',): 351,
     ('chicken',): 450,
     ('chocolate',): 1229,
     ('cookies',): 603,
     ('cooking oil',): 383,
     ('cottage cheese',): 239,
     ('eggs',): 1348,
     ('escalope',): 595,
     ('french fries',): 1282,
     ('fresh bread',): 323,
     ('frozen smoothie',): 475,
     ('frozen vegetables',): 715,
     ('grated cheese',): 393,
     ('green tea',): 991,
     ('ground beef',): 737,
     ('herb & pepper',): 371,
     ('honey',): 356,
     ('hot dogs',): 243,
     ('low fat yogurt',): 574,
     ('milk',): 972,
     ('mineral water',): 1788,
     ('olive oil',): 494,
     ('pancakes',): 713,
     ('salmon',): 319,
     ('shrimp',): 536,
     ('soup',): 379,
     ('spaghetti',): 1306,
     ('tomato juice',): 228,
     ('tomatoes',): 513,
     ('turkey',): 469,
     ('whole wheat rice',): 439},
 2: {('chocolate', 'eggs'): 249,
     ('chocolate', 'french fries'): 258,
     ('chocolate', 'milk'): 241,
     ('chocolate', 'mineral water'): 395,
     ('chocolate', 'spaghetti'): 294,
     ('eggs', 'french fries'): 273,
     ('eggs', 'milk'): 231,
     ('eggs', 'mineral water'): 382,
     ('eggs', 'spaghetti'): 274,
     ('french fries', 'mineral water'): 253,
     ('frozen vegetables', 'mineral water'): 268,
     ('green tea', 'mineral water'): 233,
     ('ground beef', 'mineral water'): 307,
     ('ground beef', 'spaghetti'): 294,
     ('milk', 'mineral water'): 360,
     ('milk', 'spaghetti'): 266,
     ('mineral water', 'pancakes'): 253,
     ('mineral water', 'spaghetti'): 448}}
correlative rules
[{milk} -> {mineral water},
 {spaghetti} -> {mineral water},
 {pancakes} -> {mineral water},
 {ground beef} -> {mineral water},
 {ground beef} -> {spaghetti},
 {chocolate} -> {mineral water},
 {frozen vegetables} -> {mineral water}]
time spent 1.7172555923461914
----------------------------------------------------------------------------------------------------

</code>


# Notebook from leylabmpi/CoreGenomePrimers
Path: notebooks/Christensenellaceae/.ipynb_checkpoints/01_genomes-checkpoint.ipynb

# Goal

Create genome collection of Christensenellaceae MAGs and isolate genomes in order to produce Christensenellales-specific primers_____no_output_____# Var_____no_output_____

work_dir = '/ebio/abt3_projects/software/dev/ll_pipelines/llprimer/experiments/christensenellaceae/'
clade = 'Christensenellaceae'
taxid = 990719
threads = 8_____no_output_____

# Init_____no_output_____

library(dplyr)
library(tidyr)
library(data.table)
library(tidytable)
library(ggplot2)
library(LeyLabRMisc)_____no_output_____library(curl)_____no_output_____df.dims()
setDTthreads(threads)
make_dir(work_dir)Directory already exists: /ebio/abt3_projects/software/dev/ll_pipelines/llprimer/experiments/christensenellaceae/

# Genomes_____no_output_____## From genbank_____no_output_____```
OUTDIR=/ebio/abt3_projects/software/dev/ll_pipelines/llprimer/experiments/christensenellales/genomes/NCBI/Christensenellaceae
mkdir -p $OUTDIR
ncbi-genome-download -p 12 -s genbank -F fasta -t 990719 -o $OUTDIR bacteria
```_____no_output_____## From UHGG_____no_output_____

F = file.path('/ebio/abt3_projects/databases_no-backup/UHGG/2019_09', 'genomes-nr_metadata.tsv')
genomes = Fread(F) %>%
    filter.(grepl('o__Christensenellales', Lineage))
genomes_____no_output_____genomes_f = genomes %>%
    filter.(grepl('f__Christensenellaceae', Lineage))
genomes_f _____no_output_____# downloading
get_file = function(url, base_dir){
    outfile = file.path(base_dir, 'genomes', 'UHGG', gsub('.+/', '', url))
    message('Downloading: ', url)
    curl_download(url, outfile, mode = "wb")
}

ret = genomes_f$FTP_download %>%
    lapply(get_file, work_dir)
ret %>% lengthDownloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME067772.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME076687.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME076875.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME078695.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME091014.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME091497.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME092036.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME092709.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME092769.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME093349.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME094667.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-015/MGYG-HGUT-01550/genomes1/GUT_GENOME096561.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-015/MGYG-HGUT-01593/genomes1/GUT_GENOME097725.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME105882.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME106152.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME107011.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME107393.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-018/MGYG-HGUT-01822/genomes1/GUT_GENOME111314.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME125784.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME125867.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME125876.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-020/MGYG-HGUT-02096/genomes1/GUT_GENOME127701.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-021/MGYG-HGUT-02193/genomes1/GUT_GENOME131740.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-021/MGYG-HGUT-02193/genomes1/GUT_GENOME137430.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-024/MGYG-HGUT-02411/genomes1/GUT_GENOME142595.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-025/MGYG-HGUT-02523/genomes1/GUT_GENOME147109.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-027/MGYG-HGUT-02701/genomes1/GUT_GENOME158644.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME163028.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-021/MGYG-HGUT-02193/genomes1/GUT_GENOME173449.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME185233.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME185690.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME188736.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME189660.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME189879.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME190234.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME190484.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME191624.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME192586.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME194019.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME194470.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-032/MGYG-HGUT-03227/genomes1/GUT_GENOME215504.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-033/MGYG-HGUT-03304/genomes1/GUT_GENOME222296.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME252750.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-021/MGYG-HGUT-02193/genomes1/GUT_GENOME253521.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME253845.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME255745.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME255791.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-017/MGYG-HGUT-01747/genomes1/GUT_GENOME256796.gff.gz

Downloading: ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/mgnify_genomes/human-gut/v1.0/all_genomes/MGYG-HGUT-042/MGYG-HGUT-04235/genomes1/GUT_GENOME260354.gff.gz

Parsing gff files

```
(genome) @ rick:/ebio/abt3_projects/software/dev/ll_pipelines/llprimer/experiments/christensenellaceae/genomes/UHGG
$ find . -name "*.gff.gz" | xargs -I % /ebio/abt3_projects/databases_no-backup/UHGG/2019_09/prokka_gff2fasta.py %
```_____no_output_____## TUK MAGs_____no_output_____

F = '/ebio/abt3_projects/Anxiety_Twins_Metagenomes/data/metagenome/TUK-5projects/LLMGA/v0.12/LLG/rnd1/final_MAGs.tsv'
TUK = Fread(F)
TUK_____no_output_____TUK = TUK %>%
    filter.(Family == 'Christensenellaceae')
TUK_____no_output_____copy_file = function(F, base_dir){
    outfile = file.path(base_dir, basename(F))
    stopifnot(F != outfile)
    file.copy(F, outfile)
}
TUK$Fasta %>%
    lapply(copy_file, base_dir=file.path(work_dir, 'TUK'))_____no_output_____

## List of all genomes_____no_output_____

files = list_files(file.path(work_dir, 'genomes'), '.fna')
samps = data.frame(Name = files %>% as.character %>% basename,
                   Fasta = files,
                   Domain = 'Bacteria',
                   Taxid = taxid) %>%
    mutate(Fasta = gsub('/+', '/', Fasta))
samps_____no_output_____# writing file
outfile = file.path(work_dir, 'genomes_raw.txt')
write_table(samps, outfile)File written: /ebio/abt3_projects/software/dev/ll_pipelines/llprimer/experiments/christensenellaceae/genomes//genomes_raw.txt

# LLG_____no_output_____### Config_____no_output_____

cat_file(file.path(work_dir, '../config_llg.yaml'))# table with genome --> fasta_file information
samples_file: /ebio/abt3_projects/software/dev/ll_pipelines/llprimer/experiments/christensenellaceae/genomes/genomes_raw.txt

# output location
output_dir: /ebio/abt3_projects/software/dev/ll_pipelines/llprimer/experiments/christensenellaceae/LLG_output/

# temporary file directory (your username will be added automatically)
tmp_dir: /ebio/abt3_scratch/

# batch processing of genomes for certain steps
## increase to better parallelize
batches: 5

# Domain of genomes ('Archaea' or 'Bacteria)
## Use "Skip" if provided as a "Domain" column in the genome table
Domain: Skip

# software parameters
# Use "Skip" to skip any of these steps. If no params for rule, use ""
# dRep MAGs are not further analyzed, but you can de-rep & then use the de-rep genome table as input.
params:
  ionice: -c 3
  # assembly assessment
  seqkit: ""
  quast: Skip #""
  multiqc_on_quast: ""
  checkm: ""
  # de-replication (CheckM recommended)
  drep:
    algorithm: auto     # will select fastANI if >1000 genomes, else accurate mode
    params: -comp 50 -con 5 -sa 0.999
  # taxonomy
  sourmash:
    compute: Skip #--scaled 10000 -k 31
    gather: -k 31
  gtdbtk:
    classify_wf: --min_perc_aa 10
  # genome pairwise ANI
  fastani: Skip #--fragLen 3000 --minFraction 0.2 -k 16
  dashing: Skip # -k 31 --full-tsv
  comparem_aai: Skip # --evalue 0.001
  # gene annotation
  gene_call:
    prokka: Skip #""
    multiqc_on_prokka: ""
    prodigal: Skip #""
  eggnog_mapper: Skip #""
  eggnog_mapper_annot: ""
  # rRNA (16S alignment & phylogeny)
  barrnap: Skip #--lencutoff 0.8
  vsearch_per_genome_drep: --id 0.95    # Skip to prevent drep of 16S copies within each genome
  qiime2_fasttree: ""
  qiime2_iqtree: --p-alrt 1000 --p-abayes --p-lbp 1000  --p-substitution-model 'GTR+I+G'
  # genome phylogeny
  phylophlan_config: Skip #--map_dna diamond --db_aa diamond --map_aa diamond --msa mafft --trim trimal --tree1 fasttree --tree2 raxml
  phylophlan:
    accuracy: --auto       # --auto will select --fast if >2000 genomes, otherwise --accurate
    other_params: --diversity high --min_num_markers 50
  # phenotype
  traitar: Skip #""
  # biosynthetic gene clusters (BGCs)
  antismash: Skip #--cb-knownclusters --cb-subclusters --asf
  DeepBGC: Skip #--score 0.5 --classifier-score 0.5 --prodigal-meta-mode
  # antimicrobial resistance (AMR)
  abricate: Skip #--minid 75 --mincov 80
  # CRISPRs
  cctyper: Skip #--prodigal meta

# databases
databases:
  checkM_data: /ebio/abt3_projects/databases_no-backup/checkM/
  sourmash: /ebio/abt3_projects/databases_no-backup/sourmash/genbank-k31.sbt.json
  sourmash_lca: /ebio/abt3_projects/databases_no-backup/sourmash/genbank-k31.lca.json.gz
  gtdbtk: /ebio/abt3_projects/databases_no-backup/GTDB/release95/gtdbtk/db_info.md
  phylophlan: /ebio/abt3_projects/databases_no-backup/phylophlan/PhyloPhlan/phylophlan.faa.bz2
  eggnog: /ebio/abt3_projects/databases_no-backup/Eggnog/v2/eggnog.db
  eggnog_diamond: /ebio/abt3_projects/databases_no-backup/Eggnog/v2/eggnog_proteins.dmnd
  antismash: /ebio/abt3_projects/databases_no-backup/antismash/v5/
  deepbgc: /ebio/abt3_projects/databases_no-backup/DeepBGC/
  traitar: /ebio/abt3_projects/databases_no-backup/pfam/traitar/
  taxdump:   # used for adding taxids to GTDB-Tk classifications
    names: /ebio/abt3_projects/databases_no-backup/GTDB/release95/taxdump/names.dmp
    nodes: /ebio/abt3_projects/databases_no-backup/GTDB/release95/taxdump/nodes.dmp
  abricate:
    ncbi: /ebio/abt3_projects/databases_no-backup/abricate/ncbi/sequences
    card: /ebio/abt3_projects/databases_no-backup/abricate/card/sequences
    resfinder: /ebio/abt3_projects/databases_no-backup/abricate/resfinder/sequences
    argannot: /ebio/abt3_projects/databases_no-backup/abricate/argannot/sequences
    bacmet2: /ebio/abt3_projects/databases_no-backup/abricate/bacmet2/sequences
    vfdb: /ebio/abt3_projects/databases_no-backup/abricate/vfdb/sequences
    megares: /ebio/abt3_projects/databases_no-backup/abricate/megares/sequences
    plasmidfinder: /ebio/abt3_projects/databases_no-backup/abricate/plasmidfinder/sequences

# snakemake pipeline
pipeline:
  snakemake_folder: ./
  script_folder: ./bin/scripts/
  use_shared_mem: True
  name: LLG

### Run_____no_output_____```
(snakemake) @ rick:/ebio/abt3_projects/software/dev/ll_pipelines/llg
$ screen -L -s llg-christ ./snakemake_sge.sh /ebio/abt3_projects/software/dev/ll_pipelines/llprimer/experiments/christensenellaceae/config_llg.yaml 30 -F
```_____no_output_____## Samples table of high quality genomes_____no_output_____

# checkM summary
checkm = file.path(work_dir, 'LLG_output', 'checkM', 'checkm_qa_summary.tsv') %>%
    read.delim(sep='\t')
checkm_____no_output_____# dRep summary
drep = file.path(work_dir, 'LLG_output', 'drep', 'checkm_markers_qa_summary.tsv') %>%
    read.delim(sep='\t') %>%
    mutate(Bin.Id = gsub('.+/', '', genome),
           Bin.Id = gsub('\\.fna$', '', Bin.Id))
drep_____no_output_____# de-replicated genomes
drep_gen = file.path(work_dir, 'LLG_output', 'drep', 'dereplicated_genomes.tsv') %>%
    read.delim(sep='\t')
drep_gen_____no_output_____# GTDBTk summary
tax = file.path(work_dir, 'LLG_output', 'gtdbtk', 'gtdbtk_summary_wTaxid.tsv') %>%
    read.delim(, sep='\t') %>%
    separate(classification,
             c('Domain', 'Phylum', 'Class', 'Order', 'Family', 'Genus', 'Species'),
             sep=';') %>%
    select(-note, -classification_method, -pplacer_taxonomy,
           -other_related_references.genome_id.species_name.radius.ANI.AF.)
tax_____no_output_____# checking overlap
cat('-- drep --\n')
overlap(basename(as.character(drep_gen$Fasta)),
        basename(as.character(drep$genome)))
cat('-- checkm --\n')
overlap(drep$Bin.Id, checkm$Bin.Id)
cat('-- gtdbtk --\n')
overlap(drep$Bin.Id, tax$user_genome)-- drep --
intersect(x,y): 61
setdiff(x,y): 0
setdiff(y,x): 40
union(x,y): 101
-- checkm --
intersect(x,y): 101
setdiff(x,y): 0
setdiff(y,x): 0
union(x,y): 101
-- gtdbtk --
intersect(x,y): 101
setdiff(x,y): 0
setdiff(y,x): 0
union(x,y): 101
# joining based on Bin.Id
drep = drep %>%
    inner_join(checkm, c('Bin.Id')) %>%
    mutate(GEN = genome %>% as.character %>% basename) %>%
    inner_join(drep_gen %>% mutate(GEN = Fasta %>% as.character %>% basename),
               by=c('GEN')) %>%
    inner_join(tax, c('Bin.Id'='user_genome')) #%>%
drep_____no_output_____# summarizing the taxonomy
df.dims(20)
drep %>%
    group_by(Order, Family, Genus) %>%
    summarize(n_genomes = n(), .groups='drop')
df.dims()_____no_output_____# filtering by quality
hq_genomes = drep %>%
    filter(completeness >= 90,
           contamination < 5,
           Strain.heterogeneity < 50)
hq_genomes_____no_output_____# filtering by taxonomy
hq_genomes = hq_genomes %>%
    filter(Family == 'f__Christensenellaceae')
hq_genomes_____no_output_____# summarizing the taxonomy
df.dims(20)
hq_genomes %>%
    group_by(Order, Family, Genus, Species) %>%
    summarize(n_genomes = n(), .groups='drop')
df.dims()_____no_output_____# summarizing
hq_genomes$Completeness %>% summary_x('Completeness')
hq_genomes$X..contigs %>% summary_x('No. of contigs')
hq_genomes$Mean.contig.length..bp. %>% summary_x('Mean contig length')
hq_genomes$X..predicted.genes %>% summary_x('No. of genes')
hq_genomes$N50..contigs. %>% summary_x('N50')_____no_output_____ # writing samples table for LLPRIMER
outfile = file.path(work_dir, 'LLG_output', 'samples_genomes_hq.txt')
hq_genomes %>%
    select(Bin.Id, Fasta) %>%
    rename('Taxon' = Bin.Id) %>%
    mutate(Taxon = gsub('_chromosome.+', '', Taxon),
           Taxon = gsub('_bin_.+', '', Taxon),
           Taxon = gsub('_genomic', '', Taxon),
           Taxon = gsub('_annotated_assembly', '', Taxon),
           Taxid = taxid) %>%
    write_table(outfile)File written: /ebio/abt3_projects/software/dev/ll_pipelines/llprimer/experiments/christensenellaceae//LLG_output/samples_genomes_hq.txt

# sessionInfo_____no_output_____

sessionInfo()_____no_output_____