# Notebook from brandonleekramer/the-growth-of-diversity
Path: src/04_word2vec/.ipynb_checkpoints/05_w2v_results_socdiv-checkpoint.ipynb

  Visualizing Word2Vec Models Trained on Biomedical Abstracts in PubMed

  A Comparison of Race and Diversity Over Time

  Brandon Kramer - University of Virginia's Biocomplexity Institute

This notebook explores two Word2Vec models trained the PubMed database taken from January 2021. Overall, I am interested in testing whether diversity and racial terms are becoming more closely related over time. To do this, I [trained](https://github.com/brandonleekramer/diversity/blob/master/src/04_word_embeddings/03_train_word2vec.ipynb) two models (one from 1990-2000 data and then a random sample of the 2010-2020 data). Now, I will visualize the results of these models to see which words are similar to race/diversity as well as plotting some comparisons of these two terms over time.

For those unfamiliar with Word2Vec, it might be worth reading [this post from Connor Gilroy](https://ccgilroy.github.io/community-discourse/introduction.html) - a sociologist that details how word embeddings can help us better understand the concept of "community." The post contains information on how Word2Vec and other word embedding approaches can teach us about word/document similarity, opposite words, and historical changes in words. Basically, Word2Vec turns all of the words in the corpus into a number based on how they are used in the context of 5-word windows (a parameter I defined in this model), making all of the words directly compariable to one another within a vector space. The end result is that we are able to compare how similar or different words are or, as we will see below, how similar or different words become over time. As we will come to see, this approach is useful but not perfect for dealing with our case due to the polysemy of 'diversity.'

#### Import packages and ingest data

Let's load all of our packages and the `.bin` files that hold our models. _____no_output_____

# load packages
import os
from itertools import product
import pandas.io.sql as psql
import pandas as pd
from pandas import DataFrame
from gensim.models import Word2Vec
from gensim.models import KeyedVectors
import numpy as np
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from matplotlib.patches import Rectangle
import seaborn as sns
import warnings
warnings.filterwarnings("ignore")

# load data
os.chdir("/sfs/qumulo/qhome/kb7hp/git/diversity/data/word_embeddings/")
earlier_model = Word2Vec.load("word2vec_1990_2000_socdiv_0821.bin")
later_model_original = Word2Vec.load("word2vec_2010_2020_socdiv_0821.bin")_____no_output_____

#### Normalizing Our Results_____no_output_____

# http://www-personal.umich.edu/~tdszyman/misc/InsightSIGNLP16.pdf
# https://github.com/williamleif/histwords
# https://gist.github.com/zhicongchen/9e23d5c3f1e5b1293b16133485cd17d8 <<<<<<
# https://github.com/nikhgarg/EmbeddingDynamicStereotypes/blob/master/dataset_utilities/normalize_vectors.py

def intersection_align_gensim(m1, m2, words=None):
    """
    Intersect two gensim word2vec models, m1 and m2.
    Only the shared vocabulary between them is kept.
    If 'words' is set (as list or set), then the vocabulary is intersected with this list as well.
    Indices are re-organized from 0..N in order of descending frequency (=sum of counts from both m1 and m2).
    These indices correspond to the new syn0 and syn0norm objects in both gensim models:
        -- so that Row 0 of m1.syn0 will be for the same word as Row 0 of m2.syn0
        -- you can find the index of any word on the .index2word list: model.index2word.index(word) => 2
    The .vocab dictionary is also updated for each model, preserving the count but updating the index.
    """

    # Get the vocab for each model
    vocab_m1 = set(m1.wv.index_to_key)
    vocab_m2 = set(m2.wv.index_to_key)

    # Find the common vocabulary
    common_vocab = vocab_m1 & vocab_m2
    if words: common_vocab &= set(words)

    # If no alignment necessary because vocab is identical...
    if not vocab_m1 - common_vocab and not vocab_m2 - common_vocab:
        return (m1,m2)

    # Otherwise sort by frequency (summed for both)
    common_vocab = list(common_vocab)
    common_vocab.sort(key=lambda w: m1.wv.get_vecattr(w, "count") + m2.wv.get_vecattr(w, "count"), reverse=True)
    # print(len(common_vocab))

    # Then for each model...
    for m in [m1, m2]:
        # Replace old syn0norm array with new one (with common vocab)
        indices = [m.wv.key_to_index[w] for w in common_vocab]
        old_arr = m.wv.vectors
        new_arr = np.array([old_arr[index] for index in indices])
        m.wv.vectors = new_arr

        # Replace old vocab dictionary with new one (with common vocab)
        # and old index2word with new one
        new_key_to_index = {}
        new_index_to_key = []
        for new_index, key in enumerate(common_vocab):
            new_key_to_index[key] = new_index
            new_index_to_key.append(key)
        m.wv.key_to_index = new_key_to_index
        m.wv.index_to_key = new_index_to_key

        print(len(m.wv.key_to_index), len(m.wv.vectors))

    return (m1,m2)

def smart_procrustes_align_gensim(base_embed, other_embed, words=None):
    """
    Original script: https://gist.github.com/quadrismegistus/09a93e219a6ffc4f216fb85235535faf
    Procrustes align two gensim word2vec models (to allow for comparison between same word across models).
    Code ported from HistWords <https://github.com/williamleif/histwords> by William Hamilton <wleif@stanford.edu>.

    First, intersect the vocabularies (see `intersection_align_gensim` documentation).
    Then do the alignment on the other_embed model.
    Replace the other_embed model's syn0 and syn0norm numpy matrices with the aligned version.
    Return other_embed.
    If `words` is set, intersect the two models' vocabulary with the vocabulary in words (see `intersection_align_gensim` documentation).
    """

    # patch by Richard So [https://twitter.com/richardjeanso) (thanks!) to update this code for new version of gensim
    # base_embed.init_sims(replace=True)
    # other_embed.init_sims(replace=True)

    # make sure vocabulary and indices are aligned
    in_base_embed, in_other_embed = intersection_align_gensim(base_embed, other_embed, words=words)

    # get the (normalized) embedding matrices
    base_vecs = in_base_embed.wv.get_normed_vectors()
    other_vecs = in_other_embed.wv.get_normed_vectors()

    # just a matrix dot product with numpy
    m = other_vecs.T.dot(base_vecs)
    # SVD method from numpy
    u, _, v = np.linalg.svd(m)
    # another matrix operation
    ortho = u.dot(v)
    # Replace original array with modified one, i.e. multiplying the embedding matrix by "ortho"
    other_embed.wv.vectors = (other_embed.wv.vectors).dot(ortho)

    return other_embed

later_model = smart_procrustes_align_gensim(earlier_model, later_model_original)78100 78100
78100 78100

#### Analyzing Most Similar Words

**What words are most similar to "racial," "ethnicity", and "diversity"?** As we can see below, "racial" and "ethnicity" is mostly similar to other racialized and/or gendered terms in both the 1990-2000 and 2010-20 periods. "Diversity", on the other hand, is most similar to heterogeneity, divergence and complexity in 1990-2000 and then richness, divergence and diversification in 2010-2020. Overall, this tells us a different version of the same story we saw when analyzing Hypothesis 1: "diversity" rarely refers to social diversity along racial or classed lines. Diversity is mostly used as a biological term. Even here, richness, along with evenness, are measure within Simpson's Index for measuring ecological biodiversity (e.g. [Stirling et al. 2001](https://www.journals.uchicago.edu/doi/abs/10.1086/321317?casa_token=Fb4sojZm9XgAAAAA:BV-t4e5f3SZ05gTJZRUydcQvHTYg47f1qRu51CixgF-b_HnGVXuPQFaqf_Lp88Tvy51Gnp7iw4yG)). _____no_output_____

# average of earlier model
earlier_race = earlier_model.wv.most_similar(positive=['race', 'racial', 'racially'], topn=50)
earlier_race = pd.DataFrame(earlier_race).rename(columns={0: "term", 1: "score"})
earlier_race['year'] = '1990-2000'
earlier_race.reset_index(inplace=True)
earlier_race = earlier_race.rename(columns = {'index':'rank'})

# average of later model
later_race = later_model.wv.most_similar(positive=['race', 'racial', 'racially'], topn=50)
later_race = pd.DataFrame(later_race).rename(columns={0: "term", 1: "score"})
later_race['year'] = '2010-2020'
later_race.reset_index(inplace=True)
later_race = later_race.rename(columns = {'index':'rank'})

# merge the tables for comparison
top_race_vectors = pd.merge(earlier_race, later_race, on=["rank"])
top_race_vectors_____no_output_____# average of earlier model
earlier_ethnicity = earlier_model.wv.most_similar(positive=['ethnic', 'ethnicity', 'ethnically'], topn=50)
earlier_ethnicity = pd.DataFrame(earlier_ethnicity).rename(columns={0: "term", 1: "score"})
earlier_ethnicity['year'] = '1990-2000'
earlier_ethnicity.reset_index(inplace=True)
earlier_ethnicity = earlier_ethnicity.rename(columns = {'index':'rank'})

# average of later model
later_ethnicity = later_model.wv.most_similar(positive=['ethnic', 'ethnicity', 'ethnically'], topn=50)
later_ethnicity = pd.DataFrame(later_ethnicity).rename(columns={0: "term", 1: "score"})
later_ethnicity['year'] = '2010-2020'
later_ethnicity.reset_index(inplace=True)
later_ethnicity = later_ethnicity.rename(columns = {'index':'rank'})

# merge the tables for comparison
top_ethnicity_vectors = pd.merge(earlier_ethnicity, later_ethnicity, on=["rank"])
top_ethnicity_vectors_____no_output_____# average of earlier model
earlier_diversity = earlier_model.wv.most_similar(positive=['socialdiverse', 'socialdiversity'], topn=50)
earlier_diversity = pd.DataFrame(earlier_diversity).rename(columns={0: "term", 1: "score"})
earlier_diversity['year'] = '1990-2000'
earlier_diversity.reset_index(inplace=True)
earlier_diversity = earlier_diversity.rename(columns = {'index':'rank'})

# average of later model
later_diversity = later_model.wv.most_similar(positive=['socialdiverse', 'socialdiversity'], topn=50)
later_diversity = pd.DataFrame(later_diversity).rename(columns={0: "term", 1: "score"})
later_diversity['year'] = '2010-2020'
later_diversity.reset_index(inplace=True)
later_diversity = later_diversity.rename(columns = {'index':'rank'})

# merge the tables for comparison
top_diversity_vectors = pd.merge(earlier_diversity, later_diversity, on=["rank"])
top_diversity_vectors_____no_output_____

#### Comparing Race and Diversity

That makes it a little difficult to directly compare the terms, so let's use the `wv.similarity()` function to directly look at that. This basically allows you to directly compare the two words to see how close they are in the vector space. To make this process a little more efficient, we are going to make our own function named `w2v_similarities_over_time()` and then compare all the relevant terms. Following [Garg et al. (2018)](https://www.pnas.org/content/115/16/E3635.short), we also decided to average some of the terms in our dictionaries since it gets a little cumbersome trying to interpret the multiple outcomes of very similiary terms like diversity/diverse, race/racial, ethnic/ethnicity, etc. _____no_output_____

def w2v_similarities_over_time(df, w2v_m1, w2v_m2):
    '''
    function compares several word2vec vectors from two different years
    and then examines how those several comparisons change over time
    ----------------------------------------------------------------
    1) first it takes a dictionary of words and creates its product
    2) compares all of those words within the vector space of w2v_m1
    3) compares all of those words within the vector space of w2v_m2
    4) examines changes in the comparisons of w2v_m1 and w2v_m2 over time

    '''
    df = list(product(df['term'], df['term']))
    df = pd.DataFrame(df, columns=['term1','term2'])

    cos_sim_m1 = []
    for index, row in df.iterrows():
        cos_sim_m1.append(w2v_m1.wv.similarity(row[0],row[1]))
    cos_sim_m1 = DataFrame(cos_sim_m1, columns=['cos_sim_m1'])
    df = df.merge(cos_sim_m1, left_index=True, right_index=True)

    cos_sim_m2 = []
    for index, row in df.iterrows():
        cos_sim_m2.append(w2v_m2.wv.similarity(row[0],row[1]))
    cos_sim_m2 = DataFrame(cos_sim_m2, columns=['cos_sim_m2'])
    df = df.merge(cos_sim_m2, left_index=True, right_index=True)
    df["cos_sim_diffs"] = df["cos_sim_m1"] - df["cos_sim_m2"]

    df_matrix = df.pivot("term1", "term2", "cos_sim_diffs")
    return df_matrix_____no_output_____

Let's pull in our dictionaries but filter to only the race and diversity entries:_____no_output_____

race_diversity_early = earlier_model.wv.similarity('race','socialdiversity')
race_diversity_later = later_model.wv.similarity('race','socialdiversity')
racial_diversity_early = earlier_model.wv.similarity('racial','socialdiversity')
racial_diversity_later = later_model.wv.similarity('racial','socialdiversity')
ethnic_diversity_early = earlier_model.wv.similarity('ethnic','socialdiversity')
ethnic_diversity_later = later_model.wv.similarity('ethnic','socialdiversity')
ethnicity_diversity_early = earlier_model.wv.similarity('ethnicity','socialdiversity')
ethnicity_diversity_later = later_model.wv.similarity('ethnicity','socialdiversity')
black_div_early = earlier_model.wv.similarity('black','socialdiversity')
black_div_later = later_model.wv.similarity('black','socialdiversity')
afam_div_early = earlier_model.wv.similarity('africanamerican','socialdiversity')
afam_div_later = later_model.wv.similarity('africanamerican','socialdiversity')
white_div_early = earlier_model.wv.similarity('white','socialdiversity')
white_div_later = later_model.wv.similarity('white','socialdiversity')
caucasian_div_early = earlier_model.wv.similarity('caucasian','socialdiversity')
caucasian_div_later = later_model.wv.similarity('caucasian','socialdiversity')
hisp_div_early = earlier_model.wv.similarity('hispanic','socialdiversity')
hisp_div_later = later_model.wv.similarity('hispanic','socialdiversity')
asian_div_early = earlier_model.wv.similarity('asian','socialdiversity')
asian_div_later = later_model.wv.similarity('asian','socialdiversity')
latino_div_early = earlier_model.wv.similarity('latino','socialdiversity')
latino_div_later = later_model.wv.similarity('latino','socialdiversity')
native_div_early = earlier_model.wv.similarity('native','socialdiversity')
native_div_later = later_model.wv.similarity('native','socialdiversity')

print('Overall Comparisons of Racial and Diversity Terms:')
print('Race and socialdiversity:             1990-2000 score:', race_diversity_early, '   2010-2020 score:', race_diversity_later, ' Difference is:', race_diversity_later-race_diversity_early )
print('Racial and socialdiversity:           1990-2000 score:', racial_diversity_early, '   2010-2020 score:', racial_diversity_later, '  Difference is:', racial_diversity_later-racial_diversity_early)
print('Ethnic and socialdiversity:           1990-2000 score:', ethnic_diversity_early, '   2010-2020 score:', ethnic_diversity_later, '  Difference is:', ethnic_diversity_later-ethnic_diversity_early)
print('Ethnicity and socialdiversity:        1990-2000 score:', ethnicity_diversity_early, '   2010-2020 score:', ethnicity_diversity_later, '  Difference is:', ethnicity_diversity_later-ethnicity_diversity_early)
print('Black and socialdiversity:            1990-1995 score:', black_div_early, '    2015-2020 score:', black_div_later, ' Difference is:', black_div_later-black_div_early)
print('African American and socialdiversity: 1990-1995 score:', afam_div_early, '   2015-2020 score:', afam_div_later, ' Difference is:', afam_div_later-afam_div_early)
print('White and socialdiversity:            1990-1995 score:', white_div_early, '  2015-2020 score:', white_div_later, '  Difference is:', white_div_later-white_div_early)
print('Caucasian and socialdiversity:        1990-1995 score:', caucasian_div_early, '  2015-2020 score:', caucasian_div_later, '  Difference is:', caucasian_div_later-caucasian_div_early)
print('Hispanic and socialdiversity:         1990-1995 score:', hisp_div_early, '   2015-2020 score:', hisp_div_later, '  Difference is:', hisp_div_later-hisp_div_early)
print('Latino and socialdiversity:           1990-1995 score:', latino_div_early, '   2015-2020 score:', latino_div_later, '  Difference is:', latino_div_later-latino_div_early)
print('Asian and socialdiversity:            1990-1995 score:', asian_div_early, '   2015-2020 score:', asian_div_later, '  Difference is:', asian_div_later-asian_div_early)
print('Native and socialdiversity:           1990-1995 score:', native_div_early, '   2015-2020 score:', native_div_later, '  Difference is:', native_div_early-native_div_later)Overall Comparisons of Racial and Diversity Terms:
Race and socialdiversity:             1990-2000 score: 0.1456064    2010-2020 score: 0.09730333  Difference is: -0.048303068
Racial and socialdiversity:           1990-2000 score: 0.3118446    2010-2020 score: 0.20273337   Difference is: -0.10911122
Ethnic and socialdiversity:           1990-2000 score: 0.3323195    2010-2020 score: 0.23735352   Difference is: -0.09496598
Ethnicity and socialdiversity:        1990-2000 score: 0.17747341    2010-2020 score: 0.17145237   Difference is: -0.0060210377
Black and socialdiversity:            1990-1995 score: 0.14626867     2015-2020 score: 0.10616236  Difference is: -0.040106304
African American and socialdiversity: 1990-1995 score: 0.19208874    2015-2020 score: 0.099406496  Difference is: -0.09268224
White and socialdiversity:            1990-1995 score: 0.10028377   2015-2020 score: 0.08158496   Difference is: -0.018698812
Caucasian and socialdiversity:        1990-1995 score: 0.31263688   2015-2020 score: 0.09052864   Difference is: -0.22210824
Hispanic and socialdiversity:         1990-1995 score: 0.24034366    2015-2020 score: 0.110881895   Difference is: -0.12946177
Latino and socialdiversity:           1990-1995 score: 0.22484964    2015-2020 score: 0.16707414   Difference is: -0.057775497
Asian and socialdiversity:            1990-1995 score: 0.36033434    2015-2020 score: 0.1751638   Difference is: -0.18517053
Native and socialdiversity:           1990-1995 score: 0.075064    2015-2020 score: 0.092080116   Difference is: -0.017016113

To interpret these scores, we have to know that a value of 1 means that two words have a perfect relationship, 0 means the two words have no relationship, and -1 means that they are perfect opposites ([Stack Overflow 2017](https://stackoverflow.com/questions/42381902/interpreting-negative-word2vec-similarity-from-gensim), [Google Groups 2019](https://groups.google.com/g/gensim/c/SZ1yct-7CuU)). Thus, when we compare all of the race, racial, ethnic and ethnicity vectors to diverse and diversity, we actually see that they are becoming *less* similar over time. Thus, despite our earlier hypotheses indicating that diversity is rising while racial terms decline, it does not seem to be the case that the two are being used in similar ways over time. It is worth noting that a number of things could complicate this interpretation, including the polysemy of diversity.

Next, we will create a plot for this. We have to keep this grey scale, because sociologists are still living in the late 1900s. _____no_output_____Before moving on to plots of these vectors, let's take a look at specific racial terms and see how they compare to diversity. _____no_output_____

plt.figure(figsize=(6, 4))
sns.set_style("white")
d = {'group': [
    'Asian', 'Latino', 'Hispanic', 'Caucasian', 'White',
    'Black', 'African American', 'Ethnicity', 'Ethnic', 'Racial', 'Race'
    ],
     '1990-2000': [
         asian_div_early, latino_div_early, hisp_div_early, caucasian_div_early, white_div_early,
         black_div_early, afam_div_early, ethnicity_diversity_early, ethnic_diversity_early, racial_diversity_early, race_diversity_early
     ],
     '2010-2020': [
         asian_div_later, latino_div_later, hisp_div_later, caucasian_div_later, white_div_later,
         black_div_later, afam_div_later, ethnicity_diversity_later, ethnic_diversity_later, racial_diversity_later, race_diversity_later
     ]}
df = pd.DataFrame(data=d)
ordered_df = df

my_range=range(1,len(df.index)+1)

plt.hlines(y=my_range, xmin=ordered_df['1990-2000'], xmax=ordered_df['2010-2020'], color='lightgrey', alpha=0.4)
plt.scatter(ordered_df['1990-2000'], my_range, color='red', alpha=1, label='1990-2000')
plt.scatter(ordered_df['2010-2020'], my_range, color='skyblue', alpha=0.8 , label='2010-2020')
#plt.scatter(ordered_df['1990-2000'], my_range, color='black', alpha=1, label='1990-2000')
#plt.scatter(ordered_df['2010-2020'], my_range, color='dimgrey', alpha=0.4 , label='2010-2020')
plt.legend()

# Add title and axis names
plt.yticks(my_range, ordered_df['group'])
plt.title("Figure 4. Comparison of Racial/Ethnic Word Vectors Relative \n to the Social Diversity Vector for 1990-2000 and 2010-2020 Word2Vec Models", loc='center')
plt.xlabel('Cosine Similarity Scores')
#plt.ylabel('All Terms Compared to Diversity')_____no_output_____top_race_vectors.to_csv('/sfs/qumulo/qhome/kb7hp/git/diversity/data/final_data/race_vectors_socdiv_0921.csv')
top_ethnicity_vectors.to_csv('/sfs/qumulo/qhome/kb7hp/git/diversity/data/final_data/ethnicity_vectors_socdiv_0921.csv')
top_diversity_vectors.to_csv('/sfs/qumulo/qhome/kb7hp/git/diversity/data/final_data/diversity_vectors_socdiv_0921.csv')
ordered_df.to_csv('/sfs/qumulo/qhome/kb7hp/git/diversity/data/final_data/select_wv_comps_socdiv_0921.csv')_____no_output_____

Overall, this approach gave us some mixed results. Asian and diversity/diverse become significantly more dissimilar while white and diversity/diverse become more similar. Once could argue that this supports Berrey's argument about diversity being used to reinforce whiteness, but it also might just be diverse/diversity being more to describe variation in neuroscience where a common term is 'white matter.' In the end, it might just be the case that the Word2Vec model's inability to deal with polysemy does not help us answer our research question. Before concluding that, let's look at visual plots of our vectors.

#### Singular Value Decomposition

In order to do that, we have to reduce the 512 dimension model into just 2 dimensions using the `TSNE` package. We will do this for both models, which will take around 30 minutes to run. Scroll down to see the results..._____no_output_____

%%capture
earlier_vocab = list(earlier_model.wv.vocab)
earlier_x = earlier_model[earlier_vocab]
earlier_tsne = TSNE(n_components=2)
earlier_tsne_x = earlier_tsne.fit_transform(earlier_x)
df_earlier = pd.DataFrame(earlier_tsne_x, index=earlier_vocab, columns=['x', 'y'])

later_vocab = list(later_model.wv.vocab)
later_x = later_model[later_vocab]
later_tsne = TSNE(n_components=2)
later_tsne_x = later_tsne.fit_transform(later_x)
df_later = pd.DataFrame(later_tsne_x, index=later_vocab, columns=['x', 'y'])

keys = ['race', 'racial', 'ethnic', 'ethnicity', 'diverse', 'diversity']

earlier_embedding_clusters = []
earlier_word_clusters = []
for word in keys:
    earlier_embeddings = []
    earlier_words = []
    for similar_word, _ in earlier_model.wv.most_similar(word, topn=30):
        earlier_words.append(similar_word)
        earlier_embeddings.append(earlier_model[similar_word])
    earlier_embedding_clusters.append(earlier_embeddings)
    earlier_word_clusters.append(earlier_words)

earlier_embedding_clusters = np.array(earlier_embedding_clusters)
n, m, k = earlier_embedding_clusters.shape
e_tsne_model_en_2d = TSNE(perplexity=15, n_components=2, init='pca', n_iter=3500, random_state=32)
e_embeddings_en_2d = np.array(e_tsne_model_en_2d.fit_transform(earlier_embedding_clusters.reshape(n * m, k))).reshape(n, m, 2)

later_embedding_clusters = []
later_word_clusters = []
for word in keys:
    later_embeddings = []
    later_words = []
    for similar_word, _ in later_model.wv.most_similar(word, topn=30):
        later_words.append(similar_word)
        later_embeddings.append(later_model[similar_word])
    later_embedding_clusters.append(later_embeddings)
    later_word_clusters.append(later_words)

later_embedding_clusters = np.array(later_embedding_clusters)
n, m, k = later_embedding_clusters.shape
l_tsne_model_en_2d = TSNE(perplexity=15, n_components=2, init='pca', n_iter=3500, random_state=32)
l_embeddings_en_2d = np.array(l_tsne_model_en_2d.fit_transform(later_embedding_clusters.reshape(n * m, k))).reshape(n, m, 2)_____no_output_____

#### Plotting the Results of the Word2Vec Models (1990-95 vs 2015-20)_____no_output_____

def tsne_plot_similar_words(title, labels, earlier_embedding_clusters, earlier_word_clusters, a, filename=None):
    plt.figure(figsize=(16, 9))
    colors = cm.rainbow(np.linspace(0, 1, len(labels)))
    for label, earlier_embeddings, earlier_words, color in zip(labels, earlier_embedding_clusters, earlier_word_clusters, colors):
        x = earlier_embeddings[:, 0]
        y = earlier_embeddings[:, 1]
        plt.scatter(x, y, c=color, alpha=a, label=label)
        for i, word in enumerate(earlier_words):
            plt.annotate(word, alpha=0.5, xy=(x[i], y[i]), xytext=(5, 2),
                         textcoords='offset points', ha='right', va='bottom', size=10)
    plt.legend(loc=4)
    plt.title(title)
    plt.grid(True)
    if filename:
        plt.savefig(filename, format='png', dpi=150, bbox_inches='tight')
    plt.show()

early_plot = tsne_plot_similar_words('Comparing the Use of Race, Ethnicity and Diversity (Word2Vec Model of 1990-2000 PubMed Data)',
                        keys, e_embeddings_en_2d, earlier_word_clusters, 0.7, 'earlier_comparison.png')
early_plot_____no_output_____

Now we can look at how the words in each vector of race, racial, ethnic, ethnicity, diversity and diverse. When we start to look at the specific terms of interest, we find racial and ethnic are the far left or toward the bottom-center. Other variants of these terms are more centered in the plot. On the other hand, diversity and diverse are both clustered toward the top-right, which means that race and diversity are fairly far away in the vector space._____no_output_____

def tsne_plot_similar_words(title, labels, later_embedding_clusters, later_word_clusters, a, filename=None):
    plt.figure(figsize=(16, 9))
    colors = cm.rainbow(np.linspace(0, 1, len(labels)))
    for label, later_embeddings, later_words, color in zip(labels, later_embedding_clusters, later_word_clusters, colors):
        x = later_embeddings[:, 0]
        y = later_embeddings[:, 1]
        plt.scatter(x, y, c=color, alpha=a, label=label)
        for i, word in enumerate(later_words):
            plt.annotate(word, alpha=0.5, xy=(x[i], y[i]), xytext=(5, 2),
                         textcoords='offset points', ha='right', va='bottom', size=10)
    plt.legend(loc=4)
    plt.title(title)
    plt.grid(True)
    if filename:
        plt.savefig(filename, format='png', dpi=150, bbox_inches='tight')
    plt.show()

later_plot = tsne_plot_similar_words('Comparing the Use of Race, Ethnicity and Diversity (Word2Vec Model of 2010-2020 PubMed Data)',
                        keys, l_embeddings_en_2d, later_word_clusters,
                        0.7, 'later_comparison.png')

os.chdir("/sfs/qumulo/qhome/kb7hp/git/diversity/data/word_embeddings/")
plt.savefig('later_comparison.png')
later_plot_____no_output_____

When we look at the same vectors in the 2015-20 model, it seems like the vectors are more closely related overall. However, when we look closer we see that the 'race' and 'ethnicity' are up in the top-left corner while 'racial' and 'ethnic' are in the top-right corner. Both sets are still fairly separated from the red and orange diversity vectors. Although these plots do not show this as clear as one might want, our analyses above do suggest that diverse and diversity as well as race, racial, ethnic, and ethinicity are being used more dissimilarly over time. The challenging thing about this analysis disentangling the polysemy from how diversity is used. If we were able to 'disentange' the use of diveristy in its more general sense compared to its usage in the context of equity, inclusion and justice discussions, would we find that the two words are becoming more similar over time?

#### Does diversity replace race/ethnicity?: Contextualizing Word Vectors with Heat Maps

After consulting some colleagues, we thought about two potential ways to test this. The first would be to turn to BERT or ELMo ([Fonteyn 2019](https://laurenthelinguist.files.wordpress.com/2019/08/sle_2019_bert.pdf); [Rakhmanberdieva 2019](https://towardsdatascience.com/word-representation-in-natural-language-processing-part-iii-2e69346007f)), which would allow us to identify the contexual variations in how diversity is used. The problem is that BERT, for example, is trained on Wikipedia data that is not historical. There are BERT options like PubMedBERT and BioBERT, but they are trained on the entirety of the PubMed abstracts, which fails to help us identify historical variations in how the terms change. Moreover, it would not make much sense to fine tune a BERT model on the same data in which it was already trained on. Thus, we ruled out BERT as an option.

Instead, we decided to continue using Word2Vec and instead compare the diveristy vector to a myriad of other vectors that we measured in H1. Our logic was that if we see diversity become more semantically similar to other diversity-related vectors over time time, while also moving further away from the racial/ethic vectors, we could infer that diversity is actually replacing race/ethnicity in biomedical abstracts over time.

To do this, I developed a function named `w2v_similarities_over_time()` that calculates the difference between all the words witin a dictionary of terms and then compares how they have changed relative to one another over time. Specifically, I will be comparing how diverse and diversity change relative to the terms in our race/ethnicity, sex/gender, sexuality, social class, and cultural/equity categories from [Hypotheses 1 and 2](https://growthofdiversity.netlify.app/methods/). Then, I will visualize the results of these models comparisons using some heat maps.

First step is importing our H1 library so we can pluck out all of the vectors for a heat map in a relatively automated manner._____no_output_____

# load dictionary of words
os.chdir("/sfs/qumulo/qhome/kb7hp/git/diversity/data/dictionaries/")
h1_dictionary = pd.read_csv("diversity_project - tree_data.csv")
h1_dictionary = h1_dictionary[h1_dictionary['viz_embeddings'] == 1].drop(['hypothesis', 'viz_embeddings', 'mean_embeddings'], axis=1)
h1_dictionary = h1_dictionary.replace({'category': {'asian|black|hispanic_latinx|white': 'race_ethnicity'}}, regex=True)
h1_dictionary = h1_dictionary.replace({'category': {'sex_gender|sexuality': 'gender_sexuality'}}, regex=True)
h1_dictionary = h1_dictionary.replace({'category': {'cultural|equity': 'cultural_equity'}}, regex=True)
h1_dictionary = h1_dictionary.replace({'category': {'minority': 'social_class'}}, regex=True).sort_values(by=['category', 'term'])
#recode for social diversity
h1_dictionary = h1_dictionary.replace({'term': {'diverse': 'socialdiverse'}}, regex=True).sort_values(by=['category', 'term'])
h1_dictionary = h1_dictionary.replace({'term': {'diversely': 'socialdiversely'}}, regex=True).sort_values(by=['category', 'term'])
h1_dictionary = h1_dictionary.replace({'term': {'diversified': 'socialdiversified'}}, regex=True).sort_values(by=['category', 'term'])
h1_dictionary = h1_dictionary.replace({'term': {'diversification': 'socialdiversification'}}, regex=True).sort_values(by=['category', 'term'])
h1_dictionary = h1_dictionary.replace({'term': {'diversifying': 'socialdiversifying'}}, regex=True).sort_values(by=['category', 'term'])
h1_dictionary = h1_dictionary.replace({'term': {'diversity': 'socialdiversity'}}, regex=True).sort_values(by=['category', 'term'])
# manual deletion after chatting with catherine
h1_dictionary =  h1_dictionary[~h1_dictionary['term'].isin(['socialdiverse', 'negro', 'ethnic', 'racist', 'racial',
                                                  'homosexual', 'men', 'women', 'inequality', 'equality'])]
h1_dictionary
#h1_dictionary[h1_dictionary['category'].isin(['diversity'])]_____no_output_____

Next, we will define our `w2v_similarities_over_time()` function._____no_output_____

def w2v_similarities_over_time(df, w2v_m1, w2v_m2):
    '''
    function compares several word2vec vectors from two different years
    and then examines how those several comparisons change over time
    ----------------------------------------------------------------
    1) first it takes a dictionary of words and creates its product
    2) compares all of those words within the vector space of w2v_m1
    3) compares all of those words within the vector space of w2v_m2
    4) examines changes in the comparisons of w2v_m1 and w2v_m2 over time

    '''
    df = list(product(df['term'], df['term']))
    df = pd.DataFrame(df, columns=['term1','term2'])

    cos_sim_m1 = []
    for index, row in df.iterrows():
        cos_sim_m1.append(w2v_m1.wv.similarity(row[0],row[1]))
    cos_sim_m1 = DataFrame(cos_sim_m1, columns=['cos_sim_m1'])
    df = df.merge(cos_sim_m1, left_index=True, right_index=True)

    cos_sim_m2 = []
    for index, row in df.iterrows():
        cos_sim_m2.append(w2v_m2.wv.similarity(row[0],row[1]))
    cos_sim_m2 = DataFrame(cos_sim_m2, columns=['cos_sim_m2'])
    df = df.merge(cos_sim_m2, left_index=True, right_index=True)
    df["cos_sim_diffs"] = df["cos_sim_m2"] - df["cos_sim_m1"]
    df_matrix = df.pivot("term1", "term2", "cos_sim_diffs")
    return df_matrix_____no_output_____

And check to make sure we get the same results as above... _____no_output_____

race_ethnicity = h1_dictionary[(h1_dictionary['category'] == 'race_ethnicity') |
                               (h1_dictionary['category'] == 'diversity')]
race_ethnicity_matrix = w2v_similarities_over_time(race_ethnicity, earlier_model, later_model)
race_ethnicity_matrix_____no_output_____

These do look similar to the basic plot we created above.

So now we will create each of our four heat maps and combine them into a joint figure for publication (again in grey scale for the sociologists)..._____no_output_____

sns.set(rc={'figure.figsize':(8,5)})
sns.set_style("whitegrid")
cultural_equity = h1_dictionary[(h1_dictionary['category'] == 'cultural_equity') |
                                (h1_dictionary['category'] == 'diversity')]
cultural_equity =  cultural_equity[~cultural_equity['term'].isin(['interlinguistic', 'oppressive', 'religion', 'religiosity'])]

cultural_equity_matrix = w2v_similarities_over_time(cultural_equity, earlier_model, later_model)
cmap = sns.diverging_palette(20, 230, as_cmap=True)
#cmap = sns.cubehelix_palette(200, hue=0.05, rot=0, light=0, dark=0.9)
#corr_cultural = cultural_equity_matrix.corr()
#mask_cultural = np.triu(np.ones_like(corr_cultural, dtype=bool))
cultural_equity_heatmap = sns.heatmap(cultural_equity_matrix, center=0, #mask=mask_cultural,
                                      cmap=cmap)_____no_output_____race_ethnicity = h1_dictionary[(h1_dictionary['category'] == 'race_ethnicity') |
                               (h1_dictionary['category'] == 'diversity')]
race_ethnicity_matrix = w2v_similarities_over_time(race_ethnicity, earlier_model, later_model)
#corr_race = race_ethnicity_matrix.corr()
#mask_race = np.triu(np.ones_like(corr_race, dtype=bool))
race_ethnicity_heatmap = sns.heatmap(race_ethnicity_matrix, #mask=mask_race,
                                     center=0, cmap=cmap).set_title("Figure 4B. Racial and Ethnic Vectors")_____no_output_____gender_sexuality = h1_dictionary[(h1_dictionary['category'] == 'gender_sexuality') |
                                 (h1_dictionary['category'] == 'diversity')]
gender_sexuality_matrix = w2v_similarities_over_time(gender_sexuality, earlier_model, later_model)
#corr_gender = gender_sexuality_matrix.corr()
#mask_gender = np.triu(np.ones_like(corr_gender, dtype=bool))
gender_sexuality_heatmap = sns.heatmap(gender_sexuality_matrix, center=0, #mask=mask_gender,
                                       cmap=cmap).set_title("Figure 4C. Gender and Sexuality Vectors")_____no_output_____social_class = h1_dictionary[(h1_dictionary['category'] == 'social_class') |
                             (h1_dictionary['category'] == 'diversity')]
social_class_matrix = w2v_similarities_over_time(social_class, earlier_model, later_model)
#corr_class = social_class_matrix.corr()
#mask_class = np.triu(np.ones_like(corr_class, dtype=bool))
social_class_heatmap = sns.heatmap(social_class_matrix, center=0, #mask=mask_class,
                                   cmap=cmap).set_title("Figure 4D. Socio-Economic Vectors")_____no_output_____sns.set(rc={'figure.figsize':(14,9)})
sns.set_style("whitegrid")
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, constrained_layout=False)

# race/ethnicity
race_ethnicity_g = sns.heatmap(race_ethnicity_matrix, center=0, cmap=cmap, ax=ax1)
race_ethnicity_labels = race_ethnicity['term'].sort_values().tolist()
N = len(race_ethnicity_labels)
race_label = 'socialdiversity'
race_index = race_ethnicity_labels.index(race_label)
x, y, w, h = 0, race_index, N, 1
for _ in range(2):
    race_ethnicity_g.add_patch(Rectangle((x, y), w, h, fill=False, edgecolor='black', lw=2, clip_on=False))
    x, y, w, h = y, x, h, w
race_ethnicity_g.set_title("Figure 5A. Racial and Ethnic Vectors")
race_ethnicity_g.set(xlabel=None)
race_ethnicity_g.set(ylabel=None)
race_ethnicity_g.set_xticklabels(race_ethnicity_g.get_xticklabels(), rotation=40, horizontalalignment='right')

## cultural
cultural_equity_g = sns.heatmap(cultural_equity_matrix, center=0, cmap=cmap, ax=ax2)
cultural_labels = cultural_equity['term'].sort_values().tolist()
N = len(cultural_labels)
cultural_label = 'socialdiversity'
cultural_index = cultural_labels.index(cultural_label)
x, y, w, h = 0, cultural_index, N, 1
for _ in range(2):
    cultural_equity_g.add_patch(Rectangle((x, y), w, h, fill=False, edgecolor='black', lw=2, clip_on=False))
    x, y, w, h = y, x, h, w
cultural_equity_g.set_title("Figure 5B. Cultural and Equity/Justice Vectors")
cultural_equity_g.set(xlabel=None)
cultural_equity_g.set(ylabel=None)
cultural_equity_g.set_xticklabels(cultural_equity_g.get_xticklabels(), rotation=40, horizontalalignment='right')

# socio-economic
social_class_g = sns.heatmap(social_class_matrix, center=0, cmap=cmap, ax=ax3)
ses_labels = social_class['term'].sort_values().tolist()
N = len(ses_labels)
ses_label = 'socialdiversity'
ses_index = ses_labels.index(ses_label)
x, y, w, h = 0, ses_index, N, 1
for _ in range(2):
    social_class_g.add_patch(Rectangle((x, y), w, h, fill=False, edgecolor='black', lw=2, clip_on=False))
    x, y, w, h = y, x, h, w
social_class_g.set_title("Figure 5C. Socio-Economic Vectors")
social_class_g.set(xlabel=None)
social_class_g.set(ylabel=None)
social_class_g.set_xticklabels(social_class_g.get_xticklabels(), rotation=40, horizontalalignment='right')

# sex/gender
gender_sexuality_g = sns.heatmap(gender_sexuality_matrix, center=0, cmap=cmap, ax=ax4)
gender_labels = gender_sexuality['term'].sort_values().tolist()
N = len(gender_labels)
gender_label = 'socialdiversity'
gender_index = gender_labels.index(gender_label)
x, y, w, h = 0, gender_index, N, 1
for _ in range(2):
    gender_sexuality_g.add_patch(Rectangle((x, y), w, h, fill=False, edgecolor='black', lw=2, clip_on=False))
    x, y, w, h = y, x, h, w
gender_sexuality_g.set_title("Figure 5D. Gender and Sexuality Vectors")
gender_sexuality_g.set(xlabel=None)
gender_sexuality_g.set(ylabel=None)
gender_sexuality_g.set_xticklabels(gender_sexuality_g.get_xticklabels(), rotation=40, horizontalalignment='right')

plt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=0.5, hspace=0.5)_____no_output_____

The final product provides some interesting results. It sure looks like diversity is generally more similar to most of the vectors apart from the race/ethnicity vectors, which could suggest that diversity is replacing race/ethnicity within in the context of articles that are examining other historically underrepresented populations in biomedical research. _____no_output_____

# load dictionary of words
os.chdir("/sfs/qumulo/qhome/kb7hp/git/diversity/data/dictionaries/")
national_terms = pd.read_csv("diversity_project - national_embeddings.csv")
national_terms = national_terms.replace({'term': {'diversity': 'socialdiversity'}}, regex=True)
national_terms.head()_____no_output_____sns.set(rc={'figure.figsize':(8,5)})
sns.set_style("whitegrid")
national_asian = national_terms[(national_terms['category'] == 'asian') |
                                (national_terms['category'] == 'diversity')]
national_asian_matrix = w2v_similarities_over_time(national_asian, earlier_model, later_model)
cmap = sns.diverging_palette(20, 230, as_cmap=True)
national_asian_heatmap = sns.heatmap(national_asian_matrix, center=0, cmap=cmap)_____no_output_____sns.set(rc={'figure.figsize':(8,5)})
sns.set_style("whitegrid")
national_europe = national_terms[(national_terms['category'] == 'europe') |
                                (national_terms['category'] == 'diversity')]
national_europe_matrix = w2v_similarities_over_time(national_europe, earlier_model, later_model)
cmap = sns.diverging_palette(20, 230, as_cmap=True)
national_europe_heatmap = sns.heatmap(national_europe_matrix, center=0, cmap=cmap)_____no_output_____sns.set(rc={'figure.figsize':(8,5)})
sns.set_style("whitegrid")
national_americas = national_terms[(national_terms['category'] == 'americas') |
                                (national_terms['category'] == 'diversity')]
national_americas_matrix = w2v_similarities_over_time(national_americas, earlier_model, later_model)
cmap = sns.diverging_palette(20, 230, as_cmap=True)
national_americas_heatmap = sns.heatmap(national_americas_matrix, center=0, cmap=cmap)_____no_output_____sns.set(rc={'figure.figsize':(8,5)})
sns.set_style("whitegrid")
national_africa = national_terms[(national_terms['category'] == 'africa') |
                                (national_terms['category'] == 'diversity')]
national_africa_matrix = w2v_similarities_over_time(national_africa, earlier_model, later_model)
cmap = sns.diverging_palette(20, 230, as_cmap=True)
national_africa_heatmap = sns.heatmap(national_africa_matrix, center=0, cmap=cmap)_____no_output_____sns.set(rc={'figure.figsize':(14,9)})
sns.set_style("whitegrid")
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, constrained_layout=False)

## cultural
national_asian_g = sns.heatmap(national_asian_matrix, center=0, cmap=cmap, ax=ax1)
national_asian_labels = national_asian['term'].sort_values().tolist()
N = len(national_asian_labels)
national_asian_label = 'socialdiversity'
national_asian_index = national_asian_labels.index(national_asian_label)
x, y, w, h = 0, national_asian_index, N, 1
for _ in range(2):
    national_asian_g.add_patch(Rectangle((x, y), w, h, fill=False, edgecolor='black', lw=2, clip_on=False))
    x, y, w, h = y, x, h, w
national_asian_g.set_title("Figure 6A. Top-10 Asian Vectors")
national_asian_g.set(xlabel=None)
national_asian_g.set(ylabel=None)
national_asian_g.set_xticklabels(national_asian_g.get_xticklabels(), rotation=40, horizontalalignment='right')

# race/ethnicity
national_europe_g = sns.heatmap(national_europe_matrix, center=0, cmap=cmap, ax=ax2)
national_europe_labels = national_europe['term'].sort_values().tolist()
N = len(national_europe_labels)
national_europe_label = 'socialdiversity'
national_europe_index = national_europe_labels.index(national_europe_label)
x, y, w, h = 0, national_europe_index, N, 1
for _ in range(2):
    national_europe_g.add_patch(Rectangle((x, y), w, h, fill=False, edgecolor='black', lw=2, clip_on=False))
    x, y, w, h = y, x, h, w
national_europe_g.set_title("Figure 6B. Top-10 European Vectors")
national_europe_g.set(xlabel=None)
national_europe_g.set(ylabel=None)
national_europe_g.set_xticklabels(national_europe_g.get_xticklabels(), rotation=40, horizontalalignment='right')

# sex/gender
national_americas_g = sns.heatmap(national_americas_matrix, center=0, cmap=cmap, ax=ax3)
national_americas_labels = national_americas['term'].sort_values().tolist()
N = len(national_americas_labels)
national_americas_label = 'socialdiversity'
national_americas_index = national_americas_labels.index(national_americas_label)
x, y, w, h = 0, national_americas_index, N, 1
for _ in range(2):
    national_americas_g.add_patch(Rectangle((x, y), w, h, fill=False, edgecolor='black', lw=2, clip_on=False))
    x, y, w, h = y, x, h, w
national_americas_g.set_title("Figure 6C. Top-10 American Vectors")
national_americas_g.set(xlabel=None)
national_americas_g.set(ylabel=None)
national_americas_g.set_xticklabels(national_americas_g.get_xticklabels(), rotation=40, horizontalalignment='right')

# socio-economic
national_africa_g = sns.heatmap(national_africa_matrix, center=0, cmap=cmap, ax=ax4)
national_africa_labels = national_africa['term'].sort_values().tolist()
N = len(national_africa_labels)
national_africa_label = 'socialdiversity'
national_africa_index = national_africa_labels.index(national_africa_label)
x, y, w, h = 0, national_africa_index, N, 1
for _ in range(2):
    national_africa_g.add_patch(Rectangle((x, y), w, h, fill=False, edgecolor='black', lw=2, clip_on=False))
    x, y, w, h = y, x, h, w
national_africa_g.set_title("Figure 6D. Top-10 African Vectors")
national_africa_g.set(xlabel=None)
national_africa_g.set(ylabel=None)
national_africa_g.set_xticklabels(national_africa_g.get_xticklabels(), rotation=40, horizontalalignment='right')

plt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=0.5, hspace=0.5)_____no_output_____

### Mean Differences

Let's define a function that compares the mean differences for all terms within each category so that we can make better sense of the visualizations. _____no_output_____

def w2v_sim_mean_comps(df, w2v_m1, w2v_m2):

    df = list(product(df['term'], df['term']))
    df = pd.DataFrame(df, columns=['term1','term2'])

    cos_sim_m1 = []
    for index, row in df.iterrows():
        cos_sim_m1.append(w2v_m1.wv.similarity(row[0],row[1]))
    cos_sim_m1 = DataFrame(cos_sim_m1, columns=['cos_sim_m1'])
    df = df.merge(cos_sim_m1, left_index=True, right_index=True)

    cos_sim_m2 = []
    for index, row in df.iterrows():
        cos_sim_m2.append(w2v_m2.wv.similarity(row[0],row[1]))
    cos_sim_m2 = DataFrame(cos_sim_m2, columns=['cos_sim_m2'])
    df = df.merge(cos_sim_m2, left_index=True, right_index=True)
    df["cos_sim_diffs"] = df["cos_sim_m2"] - df["cos_sim_m1"]

    return df_____no_output_____os.chdir("/sfs/qumulo/qhome/kb7hp/git/diversity/data/dictionaries/")
h1_allterms = pd.read_csv("diversity_project - tree_data.csv")
h1_allterms =  h1_allterms[h1_allterms['term'].isin(list(earlier_model.wv.key_to_index))]
h1_allterms['term'] = h1_allterms['term'].str.replace('_', '', regex=True)
h1_allterms =  h1_allterms[~h1_allterms['term'].isin(['intersexual'])]

h1_allterms = h1_allterms.replace({'term': {'diverse': 'socialdiverse'}}, regex=True).sort_values(by=['category', 'term'])
h1_allterms = h1_allterms.replace({'term': {'diversely': 'socialdiversely'}}, regex=True).sort_values(by=['category', 'term'])
h1_allterms = h1_allterms.replace({'term': {'diversified': 'socialdiversified'}}, regex=True).sort_values(by=['category', 'term'])
h1_allterms = h1_allterms.replace({'term': {'diversification': 'socialdiversification'}}, regex=True).sort_values(by=['category', 'term'])
h1_allterms = h1_allterms.replace({'term': {'diversifying': 'socialdiversifying'}}, regex=True).sort_values(by=['category', 'term'])
h1_allterms = h1_allterms.replace({'term': {'diversity': 'socialdiversity'}}, regex=True).sort_values(by=['category', 'term'])

h1_allterms_____no_output_____os.chdir("/sfs/qumulo/qhome/kb7hp/git/diversity/data/dictionaries/")
h1_allterms = pd.read_csv("diversity_project - tree_data.csv")
h1_allterms =  h1_allterms[h1_allterms['term'].isin(list(earlier_model.wv.key_to_index))]
h1_allterms['term'] = h1_allterms['term'].str.replace('_', '', regex=True)
h1_allterms =  h1_allterms[~h1_allterms['term'].isin(['intersexual'])]

h1_allterms = h1_allterms.replace({'term': {'diverse': 'socialdiverse'}}, regex=True).sort_values(by=['category', 'term'])
h1_allterms = h1_allterms.replace({'term': {'diversely': 'socialdiversely'}}, regex=True).sort_values(by=['category', 'term'])
h1_allterms = h1_allterms.replace({'term': {'diversified': 'socialdiversified'}}, regex=True).sort_values(by=['category', 'term'])
h1_allterms = h1_allterms.replace({'term': {'diversification': 'socialdiversification'}}, regex=True).sort_values(by=['category', 'term'])
h1_allterms = h1_allterms.replace({'term': {'diversifying': 'socialdiversifying'}}, regex=True).sort_values(by=['category', 'term'])
h1_allterms = h1_allterms.replace({'term': {'diversity': 'socialdiversity'}}, regex=True).sort_values(by=['category', 'term'])

list_of_terms = ['cultural', 'disability', 'equity', 'lifecourse', 'migration',
                 'minority', 'race_ethnicity', 'sex_gender', 'sexuality', 'social_class']

aggregated_means = pd.DataFrame(columns = ['term1', 'term2', 'cos_sim_m1', 'cos_sim_m2', 'cos_sim_diffs', 'group'])
for term in list_of_terms:
    tmp_dictionary = h1_allterms[(h1_allterms['category'] == term) | (h1_allterms['term'] == 'socialdiversity')]
    comp_outcomes = w2v_sim_mean_comps(tmp_dictionary, earlier_model, later_model)
    comp_outcomes = comp_outcomes[(comp_outcomes['term1'] == 'socialdiversity') & (comp_outcomes['term2'] != 'socialdiversity')]
    comp_outcomes['group'] = [term] * len(comp_outcomes)
    aggregated_means = pd.concat([aggregated_means, comp_outcomes])

aggregated_means = aggregated_means.groupby(by=["group"]).mean().round(3)
aggregated_means_____no_output_____os.chdir("/sfs/qumulo/qhome/kb7hp/git/diversity/data/dictionaries/")
h3_dictionary = pd.read_csv("diversity_project - h3_dictionary.csv")
h3_dictionary = h3_dictionary[h3_dictionary['term'].isin(list(earlier_model.wv.key_to_index))]
h3_dictionary = h3_dictionary.drop(['str_type','regional','subclass','source','date_added'], axis=1)

category_analysis = h3_dictionary.drop(['continental'], axis=1)
#category_analysis = category_analysis[category_analysis['mean_embeddings'] == 1]
#category_analysis = category_analysis.drop(['mean_embeddings'], axis=1)
#category_analysis = category_analysis[~category_analysis['term'].str.contains("s$")]
category_analysis = category_analysis[~category_analysis['term'].str.contains("_")]

os.chdir("/sfs/qumulo/qhome/kb7hp/git/diversity/data/dictionaries/")
h3_div_subset = pd.read_csv("diversity_project - tree_data.csv")
h3_div_subset = h3_div_subset.drop(['viz_embeddings','mean_embeddings','hypothesis'], axis=1)
h3_div_subset = h3_div_subset[h3_div_subset['category'] == 'diversity']
h3_div_subset = h3_div_subset[['term', 'category']]
h3_div_subset = h3_div_subset.replace({'term': {'diverse': 'socialdiverse'}}, regex=True).sort_values(by=['category', 'term'])
h3_div_subset = h3_div_subset.replace({'term': {'diversely': 'socialdiversely'}}, regex=True).sort_values(by=['category', 'term'])
h3_div_subset = h3_div_subset.replace({'term': {'diversified': 'socialdiversified'}}, regex=True).sort_values(by=['category', 'term'])
h3_div_subset = h3_div_subset.replace({'term': {'diversification': 'socialdiversification'}}, regex=True).sort_values(by=['category', 'term'])
h3_div_subset = h3_div_subset.replace({'term': {'diversifying': 'socialdiversifying'}}, regex=True).sort_values(by=['category', 'term'])
h3_div_subset = h3_div_subset.replace({'term': {'diversity': 'socialdiversity'}}, regex=True).sort_values(by=['category', 'term'])
category_analysis = pd.concat([category_analysis, h3_div_subset])
category_analysis

list_of_terms = ['continental', 'directional', 'national', 'omb/us census', 'race/ethnicity', 'subcontinental', 'subnational']

aggregated_means = pd.DataFrame(columns = ['term1', 'term2', 'cos_sim_m1', 'cos_sim_m2', 'cos_sim_diffs', 'group'])
for term in list_of_terms:
    tmp_dictionary = category_analysis[(category_analysis['category'] == term) | (category_analysis['term'] == 'socialdiversity')]
    comp_outcomes = w2v_sim_mean_comps(tmp_dictionary, earlier_model, later_model)
    comp_outcomes = comp_outcomes[(comp_outcomes['term1'] == 'socialdiversity') & (comp_outcomes['term2'] != 'socialdiversity')]
    comp_outcomes['group'] = [term] * len(comp_outcomes)
    aggregated_means = pd.concat([aggregated_means, comp_outcomes])

aggregated_means = aggregated_means.groupby(by=["group"]).mean().round(3)
aggregated_means
_____no_output_____os.chdir("/sfs/qumulo/qhome/kb7hp/git/diversity/data/dictionaries/")
h3_dictionary = pd.read_csv("diversity_project - h3_dictionary.csv")
h3_dictionary =  h3_dictionary[h3_dictionary['term'].isin(list(earlier_model.wv.key_to_index))]
h3_dictionary = h3_dictionary.drop(['str_type','regional','subclass','source','date_added'], axis=1)
national_means = h3_dictionary[h3_dictionary['category'] == 'national']
#national_means = national_means[national_means['mean_embeddings'] == 1]
national_means = national_means.drop(['category','mean_embeddings'], axis=1)
national_means = national_means.rename(columns={'continental': 'category'})
national_means = national_means[~national_means['term'].str.contains("s$")]
national_means = national_means[~national_means['term'].str.contains("_")]
national_means = pd.concat([national_means, h3_div_subset])
national_means

list_of_terms = ['africa', 'asia', 'europe', 'north america', 'oceania', 'south america']

aggregated_means = pd.DataFrame(columns = ['term1', 'term2', 'cos_sim_m1', 'cos_sim_m2', 'cos_sim_diffs', 'group'])
for term in list_of_terms:
    tmp_dictionary = national_means[(national_means['category'] == term) | (national_means['term'] == 'socialdiversity')]
    comp_outcomes = w2v_sim_mean_comps(tmp_dictionary, earlier_model, later_model)
    comp_outcomes = comp_outcomes[(comp_outcomes['term1'] == 'socialdiversity') & (comp_outcomes['term2'] != 'socialdiversity')]
    comp_outcomes['group'] = [term] * len(comp_outcomes)
    aggregated_means = pd.concat([aggregated_means, comp_outcomes])

aggregated_means = aggregated_means.groupby(by=["group"]).mean().round(3)
aggregated_means
 _____no_output_____

#### References

[Connor Gilroy's Tutorial](https://ccgilroy.github.io/community-discourse/word-similarity.html)

[Dominiek Ter Heide's Word2Vec Explorer](https://github.com/dominiek/word2vec-explorer)

[Sergey Smetanin's Medium Tutorial](https://towardsdatascience.com/google-news-and-leo-tolstoy-visualizing-word2vec-word-embeddings-with-t-sne-11558d8bd4d)

[Guru 99's Tutorial on Word Embeddings](https://www.guru99.com/word-embedding-word2vec.html)

[Rakhmanberdieva's 2019 Medium Post](https://towardsdatascience.com/word-representation-in-natural-language-processing-part-iii-2e69346007f)

[Fonteyn's 2019 Presentation](https://laurenthelinguist.files.wordpress.com/2019/08/sle_2019_bert.pdf)

[Bagheri's Medium Post on SVD](https://towardsdatascience.com/understanding-singular-value-decomposition-and-its-application-in-data-science-388a54be95d)

[Heatmap Highights](https://stackoverflow.com/questions/62533046/how-to-add-color-border-or-similar-highlight-to-specifc-element-of-heatmap-in-py)_____no_output_____


# Notebook from CatalinaAlbornoz/Quantum_Vehicle_Routing
Path: VRP/Classical_Vehicle_routing.ipynb

# _*Vehicle Routing*_

## The Introduction

Logistics is a major industry, with some estimates valuing it at USD 8183 billion globally in 2015. Most service providers operate a number of vehicles (e.g., trucks and container ships), a number of depots, where the vehicles are based overnight, and serve a number of client locations with each vehicle during each day. There are many optimization and control problems that consider these parameters. Computationally, the key challenge is how to design routes from depots to a number of client locations and back to the depot, so as to minimize vehicle-miles traveled, time spent, or similar objective functions. In this notebook we formalize an idealized version of the problem and showcase its solution using the quantum approximate optimization approach of Farhi, Goldstone, and Gutman (2014).

The overall workflow we demonstrate comprises:

1. establish the client locations. Normally, these would be available ahead of the day of deliveries from a database. In our use case, we generate these randomly.

3. compute the pair-wise distances, travel times, or similar. In our case, we consider the Euclidean distance, "as the crow flies", which is perhaps the simplest possible.

4. compute the actual routes. This step is run twice, actually. First, we obtain a reference value by a run of a classical solver (IBM CPLEX) on the classical computer. Second, we run an alternative, hybrid algorithm partly on the quantum computer.

5. visualization of the results. In our case, this is again a simplistic plot.

In the following, we first explain the model, before we proceed with the installation of the pre-requisites and the data loading.

## The Model

Mathematically speaking, the vehicle routing problem (VRP) is a combinatorial problem, wherein the best routes from a depot to a number of clients and back to the depot are sought, given a number of available vehicles. There are a number of formulations possible, extending a number of formulations of the traveling salesman problem [Applegate et al, 2006]. Here, we present a formulation known as MTZ [Miller, Tucker, Zemlin, 1960].

Let $n$ be the number of clients (indexed as $1,\dots,n$), and $K$ be the number of available vehicles. Let $x_{ij} = \{0,1\}$ be the binary decision variable which, if it is $1$, activates the segment from node $i$ to node $j$. The node index runs from $0$ to $n$, where $0$ is (by convention) the depot. There are twice as many distinct decision variables as edges. For example, in a fully connected graph, there are $n(n+1)$ binary decision variables.

If two nodes $i$ and $j$ have a link from $i$ to $j$, we write $i \sim j$. We also denote with $\delta(i)^+$ the set of nodes to which $i$ has a link, i.e., $j \in \delta(i)^+$ if and only if $i \sim j$. Similarly, we denote with
$\delta(i)^-$ the set of nodes which are connected to $i$, in the sense that $j \in \delta(i)^-$ if and only if $j \sim i$.

In addition, we consider continuous variables, for all nodes $i = 1,\dots, n$, denoted $u_i$. These variables are needed in the MTZ formulation of the problem to eliminate sub-tours between clients.

The VRP can be formulated as:

$$
(VRP) \quad  f = \min_{\{x_{ij}\}_{i\sim j}\in \{0,1\}, \{u_i\}_{i=1,\dots,n}\in \mathbb{R}} \quad \sum_{i \sim j} w_{ij} x_{ij}
$$

subject to the node-visiting constraint:

$$
\sum_{j \in \delta(i)^+} x_{ij} = 1, \,\sum_{j \in \delta(i)^-} x_{ji} = 1,\, \forall i \in \{1,\dots,n\},
$$

the depot-visiting constraints:

$$
\sum_{i \in \delta(0)^+} x_{0i} = K, \, \sum_{j \in \delta(0)^+} x_{j0} = K,
$$

and the sub-tour elimination constraints:

$$
u_i - u_j + Q x_{ij} \leq Q-q_j, \, \forall i \sim j, \,i ,j \neq 0, \quad q_i \leq u_i \leq Q,\, \forall i, i \neq 0.
$$

In particular,
- The cost function is linear in the cost functions and weighs the different arches based on a positive weight $w_{ij}>0$ (typically the distance between node $i$ and node $j$);
- The first set of constraints enforce that from and to every client, only one link is allowed;
- The second set of constraints enforce that from and to the depot, exactly  $K$ links are allowed;
- The third set of constraints enforce the sub-tour elimination constraints and are bounds on $u_i$, with $Q>q_j>0$, and $Q,q_i \in \mathbb{R}$.

## Classical solution

We can solve the VRP classically, e.g., by using CPLEX. CPLEX uses a branch-and-bound-and-cut method to find an approximate solution of the VRP, which, in this formulation, is a mixed-integer linear program (MILP). For the sake of notation, we pack the decision variables in one vector as

$$
{\bf z} = [x_{01},x_{02},\ldots,x_{10}, x_{12},\ldots,x_{n(n-1)}]^T,
$$

wherein ${\bf z} \in \{0,1\}^N$, with $N = n (n+1)$. So the dimension of the problem scales quadratically with the number of nodes. Let us denote the optimal solution by ${\bf z}^*$, and the associated optimal cost $f^*$.

## References

[1] E. Farhi, J. Goldstone, S. Gutmann e-print arXiv 1411.4028, 2014

[2] https://github.com/Qiskit/qiskit-tutorial/blob/master/qiskit/aqua/optimization/maxcut_and_tsp.ipynb

[3] C. E. Miller, E. W. Tucker, and R. A. Zemlin (1960). "Integer Programming Formulations and Travelling Salesman Problems". J. ACM. 7: 326–329. doi:10.1145/321043.321046.

[4] D. L. Applegate, R. M. Bixby, V. Chvátal, and W. J. Cook (2006). The Traveling Salesman Problem. Princeton University Press, ISBN 978-0-691-12993-8._____no_output_____## Initialization

First of all we load all the packages that we need:
  - Python 3.6 or greater is required;
  - CPLEX 12.8 or greater is required for the classical computations;_____no_output_____

# Load the packages that are required
import numpy as np
import operator
import matplotlib.pyplot as plt

import sys
if sys.version_info < (3, 6):
    raise Exception('Please use Python version 3.6 or greater.')

try:
    import cplex
    from cplex.exceptions import CplexError
except:
    print("Warning: Cplex not found.")
import math
_____no_output_____

We then initialize the variables_____no_output_____

# Initialize the problem by defining the parameters
n = 10  # number of nodes + depot (n+1)
K = 2  # number of vehicles_____no_output_____

We define an initializer class that randomly places the nodes in a 2-D plane and computes the distance between them. _____no_output_____

# Get the data
class Initializer():

    def __init__(self, n):
        self.n = n

    def generate_instance(self):

        n = self.n

        # np.random.seed(33)
        np.random.seed(1543)

        xc = (np.random.rand(n) - 0.5) * 10 # random generation of x-coordinates for each node
        yc = (np.random.rand(n) - 0.5) * 10 # random generation of y-coordinates for each node

        instance = np.zeros([n, n])
        for ii in range(0, n):
            for jj in range(ii + 1, n):
                instance[ii, jj] = (xc[ii] - xc[jj]) ** 2 + (yc[ii] - yc[jj]) ** 2  #generation of the adjacency matrix
                instance[jj, ii] = instance[ii, jj]

        return xc, yc, instance_____no_output_____# Initialize the problem by randomly generating the instance
initializer = Initializer(n)
xc,yc,instance = initializer.generate_instance()_____no_output_____

## Classical solution using IBM ILOG CPLEX

For a classical solution, we use IBM ILOG CPLEX. CPLEX is able to find the exact solution of this problem. We first define a ClassicalOptimizer class that encodes the problem in a way that CPLEX can solve, and then instantiate the class and solve it.
_____no_output_____

class ClassicalOptimizer:

    def __init__(self, instance,n,K):

        self.instance = instance
        self.n = n  # number of nodes
        self.K = K  # number of vehicles

    def compute_allowed_combinations(self):
        f = math.factorial
        return f(self.n) / f(self.K) / f(self.n-self.K)

    def cplex_solution(self):

        # refactoring
        instance = self.instance
        n = self.n
        K = self.K

        my_obj = list(instance.reshape(1, n**2)[0])+[0. for x in range(0,n-1)]
        my_ub = [1 for x in range(0,n**2+n-1)]
        my_lb = [0 for x in range(0,n**2)] + [0.1 for x in range(0,n-1)]
        my_ctype = "".join(['I' for x in range(0,n**2)]) + "".join(['C' for x in range(0,n-1)])

        my_rhs = 2*([K] + [1 for x in range(0,n-1)]) + [1-0.1 for x in range(0,(n-1)**2-(n-1))] + [0 for x in range(0,n)]
        my_sense = "".join(['E' for x in range(0,2*n)]) + "".join(['L' for x in range(0,(n-1)**2-(n-1))])+"".join(['E' for x in range(0,n)])

        try:
            my_prob = cplex.Cplex()
            self.populatebyrow(my_prob,my_obj,my_ub,my_lb,my_ctype,my_sense,my_rhs)

            my_prob.solve()

        except CplexError as exc:
            print(exc)
            return

        x = my_prob.solution.get_values()
        x = np.array(x)
        cost = my_prob.solution.get_objective_value()

        return x,cost

    def populatebyrow(self,prob,my_obj,my_ub,my_lb,my_ctype,my_sense,my_rhs):

        n = self.n

        prob.objective.set_sense(prob.objective.sense.minimize)
        prob.variables.add(obj = my_obj, lb = my_lb, ub = my_ub, types = my_ctype)

        prob.set_log_stream(None)
        prob.set_error_stream(None)
        prob.set_warning_stream(None)
        prob.set_results_stream(None)

        rows = []
        for ii in range(0,n):
            col = [x for x in range(0+n*ii,n+n*ii)]
            coef = [1 for x in range(0,n)]
            rows.append([col, coef])

        for ii in range(0,n):
            col = [x for x in range(0+ii,n**2,n)]
            coef = [1 for x in range(0,n)]

            rows.append([col, coef])

        # Sub-tour elimination constraints:
        for ii in range(0, n):
            for jj in range(0,n):
                if (ii != jj)and(ii*jj>0):

                    col = [ii+(jj*n), n**2+ii-1, n**2+jj-1]
                    coef = [1, 1, -1]

                    rows.append([col, coef])

        for ii in range(0,n):
            col = [(ii)*(n+1)]
            coef = [1]
            rows.append([col, coef])

        prob.linear_constraints.add(lin_expr=rows, senses=my_sense, rhs=my_rhs)_____no_output_____# Instantiate the classical optimizer class
classical_optimizer = ClassicalOptimizer(instance,n,K)

# Print number of feasible solutions
print('Number of feasible solutions = ' + str(classical_optimizer.compute_allowed_combinations()))Number of feasible solutions = 45.0
# Solve the problem in a classical fashion via CPLEX
x = None
z = None
try:
    x,classical_cost = classical_optimizer.cplex_solution()
    # Put the solution in the z variable
    z = [x[ii] for ii in range(n**2) if ii//n != ii%n]
    # Print the solution
    print(z)
except:
    print("CPLEX may be missing.")[1.0, -0.0, -0.0, 0.0, -0.0, -0.0, 0.0, -0.0, 1.0, 0.0, -0.0, -0.0, -0.0, -0.0, 1.0, -0.0, 0.0, -0.0, 0.0, -0.0, 0.0, -0.0, 0.0, -0.0, 1.0, -0.0, -0.0, -0.0, -0.0, 0.0, -0.0, 1.0, -0.0, -0.0, -0.0, -0.0, 1.0, -0.0, -0.0, -0.0, -0.0, -0.0, 0.0, -0.0, 0.0, -0.0, -0.0, 1.0, 0.0, -0.0, -0.0, 0.0, -0.0, -0.0, -0.0, 0.0, -0.0, -0.0, -0.0, -0.0, -0.0, 1.0, -0.0, 0.0, -0.0, 0.0, -0.0, 1.0, -0.0, -0.0, -0.0, 0.0, 0.0, 0.0, -0.0, 1.0, -0.0, -0.0, 0.0, -0.0, -0.0, 1.0, -0.0, -0.0, -0.0, -0.0, -0.0, -0.0, -0.0, -0.0]
print(x)[ 0.   1.  -0.  -0.   0.  -0.  -0.   0.  -0.   1.   0.   0.  -0.  -0.
 -0.  -0.   1.  -0.   0.  -0.   0.  -0.   0.   0.  -0.   0.  -0.   1.
 -0.  -0.  -0.  -0.   0.   0.  -0.   1.  -0.  -0.  -0.  -0.   1.  -0.
 -0.  -0.   0.  -0.  -0.   0.  -0.   0.  -0.  -0.   1.   0.  -0.   0.
 -0.   0.  -0.  -0.  -0.   0.  -0.  -0.  -0.  -0.   0.  -0.   1.  -0.
  0.  -0.   0.  -0.   1.  -0.  -0.   0.  -0.   0.   0.   0.  -0.   1.
 -0.  -0.   0.  -0.   0.  -0.   1.  -0.  -0.  -0.  -0.  -0.  -0.  -0.
 -0.   0.   1.   0.3  0.7  0.1  0.4  0.9  0.2  0.8  0.1]
# Visualize the solution
def visualize_solution(xc, yc, x, C, n, K, title_str):
    plt.figure()
    plt.scatter(xc, yc, s=200)
    for i in range(len(xc)):
        plt.annotate(i, (xc[i] + 0.15, yc[i]), size=16, color='r')
    plt.plot(xc[0], yc[0], 'r*', ms=20)

    plt.grid()

    vehicle=0
    colours=['green','black','blue']
    lst_vehicles=[[0] for i in range(K)]
    ii_list=[ii for ii in range(0, n ** 2)]
    while len(ii_list)>0:
        for ii in ii_list:

            if x[ii] > 0:
                ix = ii // n
                iy = ii % n
                if ix==0:
                    lst_vehicles[vehicle].append(iy)
                    plt.arrow(xc[ix], yc[ix], xc[iy] - xc[ix], yc[iy] - yc[ix], length_includes_head=True, head_width=.25,ec=colours[vehicle])
                    vehicle+=1
                    ii_list.remove(ii)
                else:
                    for i in range(K):
                        if lst_vehicles[i].count(ix)>0:
                            lst_vehicles[i].append(iy)
                            plt.arrow(xc[ix], yc[ix], xc[iy] - xc[ix], yc[iy] - yc[ix], length_includes_head=True, head_width=.25,ec=colours[i])
                            ii_list.remove(ii)
                            break
            else:
                ii_list.remove(ii)

    print('Routes: ',lst_vehicles)

    plt.title(title_str+' cost = ' + str(int(C * 100) / 100.))
    plt.show()

if x is not None:
    visualize_solution(xc, yc, x, classical_cost, n, K, 'Classical')Routes:  [[0, 1, 6, 8, 3, 5, 2, 7, 4, 0], [0, 9, 0]]

If you have CPLEX, the solution shows the depot with a star and the selected routes for the vehicles with arrows. _____no_output_____This code is part of the Vehicle Routing tutorial on the IBM Quantum Experience.

This code is licensed under the Apache License, Version 2.0. You may
obtain a copy of this license in the LICENSE.txt file in the root directory
of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
Any modifications or derivative works of this code must retain this
copyright notice, and modified files need to carry a notice indicating
that they have been altered from the originals._____no_output_____