# Notebook from ucsd-ccbb/cirrus-ngs
Path: notebooks/cirrus-ngs/CheckScripts.ipynb

# Check Scripts on Cluster_____no_output_____## 1. Configure AWS key pair_____no_output_____This cell only contains information that you, the user, should input.

#### String Fields

**your_cluster_name**: This is the name given to your cluster when it was created using cfncluster.

**private_key**: The path to your private key needed to access your cluster._____no_output_____

from pprint import pprint
import os
import sys
sys.path.append("../../src/cirrus_ngs")
from awsCluster import ClusterManager, ConnectionManager
from util import AddonsManager

#name of the cluster
your_cluster_name = "clustername"

## The private key pair for accessing cluster.
private_key = "/path/to/your_aws_key.pem"

## If delete cfncluster after job is done./
delete_cfncluster= False

print("variables set")_____no_output_____

## 2. Connect to Cluster_____no_output_____Run this before any following cells, it provides the connection to the cluster._____no_output_____

master_ip_address = ClusterManager.create_cluster(cluster_name=your_cluster_name)
ssh_client = ConnectionManager.connect_master(hostname=master_ip_address,
               username="ec2-user",
               private_key_file=private_key)_____no_output_____

## 3. Get the Supported Pipelines_____no_output_____### Note: this cell must be run before any following cells_____no_output_____

#This cell must be run before others in this section

scripts = AddonsManager.get_scripts_dict(ssh_client)
print()
print("Supported Pipelines:", AddonsManager.get_all_pipeline_names(scripts))_____no_output_____

## 4. Choose Any of the Following Cells_____no_output_____These cells all provide some different kind of information about the cluster's scripts. Unless otherwise noted feel free to skip around and run them as you wish._____no_output_____### Get the Supported Workflows in a Given Pipeline_____no_output_____This cell prints out a list of workflows that are supported by a pipeline that you specify. The target_pipeline field can be set to a supported pipeline (see Get the Supported Pipelines) or "all". When set to "all", this cell will show all supported workflows by all pipelines. Note that the target_pipeline field is case sensitive._____no_output_____

#can be set to a supported pipeline name or "all"
target_pipeline = "all"

print("Supported Workflows in {} Pipeline(s): ".format(target_pipeline), end="")
pprint(AddonsManager.get_workflows_in_pipeline(scripts, target_pipeline), indent=2)_____no_output_____

### Get the Scripts Used by a Given Pipeline/Workflow_____no_output_____This cell prints out a list of all the shell scripts that are used by a given pipeline or workflow. The target_pipeline field is the same as the target_pipeline field above. It can be set to a supported pipeline or to "all". When set to all, this cell will print the scripts used by every workflow in every pipeline. The target_workflow field is ignored if target_pipeline is set to "all". The target_workflow field can be set to a supported workflow for the provided target_pipeline or to all. See the previous cell for supported workflows for each pipeline. If set to "all", then this cell will print the shell scripts used by all workflows within the given pipeline._____no_output_____

#can be set to a supported pipeline or "all"
target_pipeline = "all"

#can be set to a support workflow or "all"
#if target_pipeline == "all" then this variable is ignored
target_workflow = "all"

pprint(AddonsManager.get_scripts(scripts, target_pipeline, target_workflow))_____no_output_____

### Print a Script_____no_output_____This prints out the specified script. The whole shell script will be printed out. The target_pipeline field must be set to a valid pipeline, the target_workflow field must be set to a valid workflow in that pipeline, and the target_script field must be set to a valid script in that workflow. For scripts that are shared by multiple pipelines/workflows you must provide some pipeline/workflow that contains that script to print it out.

The output will include a result surrounded by "###" that indicates where this script can be called from. Afterwards
there will be an exact printout of the script specified._____no_output_____

#all targets must be set to a valid pipeline/workflow/script
#use cells above to check valid options

target_pipeline = "DNASeq"
target_workflow = "bwa_gatk"
target_script = "bwa.sh"

loc, file_cat = AddonsManager.cat_script(ssh_client, scripts, target_pipeline, target_workflow, target_script)

print("".join(["#"]*len(loc)) + "\n{}\n".format(loc) + "".join(["#"]*len(loc)))
AddonsManager.show_script(file_cat)_____no_output_____

### Check Which Step Calls a Script_____no_output_____This cell shows the user which steps in the pipeline notebooks actually call a given shell script. The target_script field should be a shell script that exists on the pipeline (including the .sh file extension). The output will contain
where the shell script is called from. If (for example) fastqc.sh is called from the fastqc step in every pipeline, then the output will only say "fastqc in all Pipelines". _____no_output_____

#should be the name of a shell script on the cluster (include sh extension)
target_script = "k_align.sh"

print(AddonsManager.get_steps_calling_script(ssh_client, scripts, target_script))_____no_output_____


# Notebook from juankikushima/HMM_Tagger
Path: HMM Tagger.ipynb

# Project: Part of Speech Tagging with Hidden Markov Models
---
### Introduction

Part of speech tagging is the process of determining the syntactic category of a word from the words in its surrounding context. It is often used to help disambiguate natural language phrases because it can be done quickly with high accuracy. Tagging can be used for many NLP tasks like determining correct pronunciation during speech synthesis (for example, _dis_-count as a noun vs dis-_count_ as a verb), for information retrieval, and for word sense disambiguation.

In this notebook, you'll use the [Pomegranate](http://pomegranate.readthedocs.io/) library to build a hidden Markov model for part of speech tagging using a "universal" tagset. Hidden Markov models have been able to achieve [>96% tag accuracy with larger tagsets on realistic text corpora](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf). Hidden Markov models have also been used for speech recognition and speech generation, machine translation, gene recognition for bioinformatics, and human gesture recognition for computer vision, and more.

![](_post-hmm.png)

The notebook already contains some code to get you started. You only need to add some new functionality in the areas indicated to complete the project; you will not need to modify the included code beyond what is requested. Sections that begin with **'IMPLEMENTATION'** in the header indicate that you must provide code in the block that follows. Instructions will be provided for each section, and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully!_____no_output_____

**Note:** Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You must then **export the notebook** by running the last cell in the notebook, or by using the menu above and navigating to **File -> Download as -> HTML (.html)** Your submissions should include both the `html` and `ipynb` files.

_____no_output_____

**Note:** Code and Markdown cells can be executed using the `Shift + Enter` keyboard shortcut. Markdown cells can be edited by double-clicking the cell to enter edit mode.

_____no_output_____### The Road Ahead
You must complete Steps 1-3 below to pass the project. The section on Step 4 includes references & resources you can use to further explore HMM taggers.

- [Step 1](#Step-1:-Read-and-preprocess-the-dataset): Review the provided interface to load and access the text corpus
- [Step 2](#Step-2:-Build-a-Most-Frequent-Class-tagger): Build a Most Frequent Class tagger to use as a baseline
- [Step 3](#Step-3:-Build-an-HMM-tagger): Build an HMM Part of Speech tagger and compare to the MFC baseline
- [Step 4](#Step-4:-[Optional]-Improving-model-performance): (Optional) Improve the HMM tagger_____no_output_____

# Jupyter "magic methods" -- only need to be run once per kernel restart
%load_ext autoreload
%aimport helpers, tests
%autoreload 1_____no_output_____# import python modules -- this cell needs to be run again if you make changes to any of the files
import matplotlib.pyplot as plt
import numpy as np

from IPython.core.display import HTML
from itertools import chain
from collections import Counter, defaultdict
from helpers import show_model, Dataset
from pomegranate import State, HiddenMarkovModel, DiscreteDistribution_____no_output_____

## Step 1: Read and preprocess the dataset
---
We'll start by reading in a text corpus and splitting it into a training and testing dataset. The data set is a copy of the [Brown corpus](https://en.wikipedia.org/wiki/Brown_Corpus) (originally from the [NLTK](https://www.nltk.org/) library) that has already been pre-processed to only include the [universal tagset](https://arxiv.org/pdf/1104.2086.pdf). You should expect to get slightly higher accuracy using this simplified tagset than the same model would achieve on a larger tagset like the full [Penn treebank tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), but the process you'll follow would be the same.

The `Dataset` class provided in helpers.py will read and parse the corpus. You can generate your own datasets compatible with the reader by writing them to the following format. The dataset is stored in plaintext as a collection of words and corresponding tags. Each sentence starts with a unique identifier on the first line, followed by one tab-separated word/tag pair on each following line. Sentences are separated by a single blank line.

Example from the Brown corpus.
```
b100-38532
Perhaps	ADV
it	PRON
was	VERB
right	ADJ
;	.
;	.

b100-35577
...
```_____no_output_____

data = Dataset("tags-universal.txt", "brown-universal.txt", train_test_split=0.8)

print("There are {} sentences in the corpus.".format(len(data)))
print("There are {} sentences in the training set.".format(len(data.training_set)))
print("There are {} sentences in the testing set.".format(len(data.testing_set)))

assert len(data) == len(data.training_set) + len(data.testing_set), \
       "The number of sentences in the training set + testing set should sum to the number of sentences in the corpus"There are 57340 sentences in the corpus.
There are 45872 sentences in the training set.
There are 11468 sentences in the testing set.

### The Dataset Interface

You can access (mostly) immutable references to the dataset through a simple interface provided through the `Dataset` class, which represents an iterable collection of sentences along with easy access to partitions of the data for training & testing. Review the reference below, then run and review the next few cells to make sure you understand the interface before moving on to the next step.

```
Dataset-only Attributes:
    training_set - reference to a Subset object containing the samples for training
    testing_set - reference to a Subset object containing the samples for testing

Dataset & Subset Attributes:
    sentences - a dictionary with an entry {sentence_key: Sentence()} for each sentence in the corpus
    keys - an immutable ordered (not sorted) collection of the sentence_keys for the corpus
    vocab - an immutable collection of the unique words in the corpus
    tagset - an immutable collection of the unique tags in the corpus
    X - returns an array of words grouped by sentences ((w11, w12, w13, ...), (w21, w22, w23, ...), ...)
    Y - returns an array of tags grouped by sentences ((t11, t12, t13, ...), (t21, t22, t23, ...), ...)
    N - returns the number of distinct samples (individual words or tags) in the dataset

Methods:
    stream() - returns an flat iterable over all (word, tag) pairs across all sentences in the corpus
    __iter__() - returns an iterable over the data as (sentence_key, Sentence()) pairs
    __len__() - returns the nubmer of sentences in the dataset
```

For example, consider a Subset, `subset`, of the sentences `{"s0": Sentence(("See", "Spot", "run"), ("VERB", "NOUN", "VERB")), "s1": Sentence(("Spot", "ran"), ("NOUN", "VERB"))}`. The subset will have these attributes:

```
subset.keys == {"s1", "s0"}  # unordered
subset.vocab == {"See", "run", "ran", "Spot"}  # unordered
subset.tagset == {"VERB", "NOUN"}  # unordered
subset.X == (("Spot", "ran"), ("See", "Spot", "run"))  # order matches .keys
subset.Y == (("NOUN", "VERB"), ("VERB", "NOUN", "VERB"))  # order matches .keys
subset.N == 7  # there are a total of seven observations over all sentences
len(subset) == 2  # because there are two sentences
```

**Note:** The `Dataset` class is _convenient_, but it is **not** efficient. It is not suitable for huge datasets because it stores multiple redundant copies of the same data.

_____no_output_____#### Sentences

`Dataset.sentences` is a dictionary of all sentences in the training corpus, each keyed to a unique sentence identifier. Each `Sentence` is itself an object with two attributes: a tuple of the words in the sentence named `words` and a tuple of the tag corresponding to each word named `tags`._____no_output_____

key = 'b100-38532'
print("Sentence: {}".format(key))
print("words:\n\t{!s}".format(data.sentences[key].words))
print("tags:\n\t{!s}".format(data.sentences[key].tags))Sentence: b100-38532
words:
	('Perhaps', 'it', 'was', 'right', ';', ';')
tags:
	('ADV', 'PRON', 'VERB', 'ADJ', '.', '.')

**Note:** The underlying iterable sequence is **unordered** over the sentences in the corpus; it is not guaranteed to return the sentences in a consistent order between calls. Use `Dataset.stream()`, `Dataset.keys`, `Dataset.X`, or `Dataset.Y` attributes if you need ordered access to the data.

#### Counting Unique Elements

You can access the list of unique words (the dataset vocabulary) via `Dataset.vocab` and the unique list of tags via `Dataset.tagset`._____no_output_____

print("There are a total of {} samples of {} unique words in the corpus."
      .format(data.N, len(data.vocab)))
print("There are {} samples of {} unique words in the training set."
      .format(data.training_set.N, len(data.training_set.vocab)))
print("There are {} samples of {} unique words in the testing set."
      .format(data.testing_set.N, len(data.testing_set.vocab)))
print("There are {} words in the test set that are missing in the training set."
      .format(len(data.testing_set.vocab - data.training_set.vocab)))

assert data.N == data.training_set.N + data.testing_set.N, \
       "The number of training + test samples should sum to the total number of samples"There are a total of 1161192 samples of 56057 unique words in the corpus.
There are 928458 samples of 50536 unique words in the training set.
There are 232734 samples of 25112 unique words in the testing set.
There are 5521 words in the test set that are missing in the training set.

#### Accessing word and tag Sequences
The `Dataset.X` and `Dataset.Y` attributes provide access to ordered collections of matching word and tag sequences for each sentence in the dataset._____no_output_____

# accessing words with Dataset.X and tags with Dataset.Y
for i in range(2):
    print("Sentence {}:".format(i + 1), data.X[i])
    print()
    print("Labels {}:".format(i + 1), data.Y[i])
    print()Sentence 1: ('Mr.', 'Podger', 'had', 'thanked', 'him', 'gravely', ',', 'and', 'now', 'he', 'made', 'use', 'of', 'the', 'advice', '.')

Labels 1: ('NOUN', 'NOUN', 'VERB', 'VERB', 'PRON', 'ADV', '.', 'CONJ', 'ADV', 'PRON', 'VERB', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence 2: ('But', 'there', 'seemed', 'to', 'be', 'some', 'difference', 'of', 'opinion', 'as', 'to', 'how', 'far', 'the', 'board', 'should', 'go', ',', 'and', 'whose', 'advice', 'it', 'should', 'follow', '.')

Labels 2: ('CONJ', 'PRT', 'VERB', 'PRT', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'ADP', 'ADV', 'ADV', 'DET', 'NOUN', 'VERB', 'VERB', '.', 'CONJ', 'DET', 'NOUN', 'PRON', 'VERB', 'VERB', '.')

#### Accessing (word, tag) Samples
The `Dataset.stream()` method returns an iterator that chains together every pair of (word, tag) entries across all sentences in the entire corpus._____no_output_____

# use Dataset.stream() (word, tag) samples for the entire corpus
print("\nStream (word, tag) pairs:\n")
for i, pair in enumerate(data.stream()):
    print("\t", pair)
    if i > 5: break
Stream (word, tag) pairs:

	 ('Mr.', 'NOUN')
	 ('Podger', 'NOUN')
	 ('had', 'VERB')
	 ('thanked', 'VERB')
	 ('him', 'PRON')
	 ('gravely', 'ADV')
	 (',', '.')

For both our baseline tagger and the HMM model we'll build, we need to estimate the frequency of tags & words from the frequency counts of observations in the training corpus. In the next several cells you will complete functions to compute the counts of several sets of counts. _____no_output_____## Step 2: Build a Most Frequent Class tagger
---

Perhaps the simplest tagger (and a good baseline for tagger performance) is to simply choose the tag most frequently assigned to each word. This "most frequent class" tagger inspects each observed word in the sequence and assigns it the label that was most often assigned to that word in the corpus._____no_output_____### IMPLEMENTATION: Pair Counts

Complete the function below that computes the joint frequency counts for two input sequences._____no_output_____

def pair_counts(sequences_A, sequences_B):
    """Return a dictionary keyed to each unique value in the first sequence list
    that counts the number of occurrences of the corresponding value from the
    second sequences list.

    For example, if sequences_A is tags and sequences_B is the corresponding
    words, then if 1244 sequences contain the word "time" tagged as a NOUN, then
    you should return a dictionary such that pair_counts[NOUN][time] == 1244
    """
    # TODO: Finish this function!

    #A
    seqA = []
    for subset_A in sequences_A:
        for set_A in subset_A:
            seqA.append(set_A)

    #B
    seqB = []
    for subset_B in sequences_B:
        for set_B in subset_B:
            seqB.append(set_B)

    pares = zip(seqA, seqB)
    pairdict = {}
    for tag, word in pares:
        if tag in pairdict.keys():
            if word in pairdict[tag].keys():
                pairdict[tag][word] += 1
            else:
                pairdict[tag][word] = 1
        else:
            pairdict[tag] = {}
            pairdict[tag][word] = 1
    return pairdict

# Calculate C(t_i, w_i)

emission_counts = pair_counts(data.Y, data.X)

assert len(emission_counts) == 12, \
       "Uh oh. There should be 12 tags in your dictionary."
assert max(emission_counts["NOUN"], key=emission_counts["NOUN"].get) == 'time', \
       "Hmmm...'time' is expected to be the most common NOUN."
HTML('
Your emission counts look good!
')_____no_output_____

### IMPLEMENTATION: Most Frequent Class Tagger

Use the `pair_counts()` function and the training dataset to find the most frequent class label for each word in the training data, and populate the `mfc_table` below. The table keys should be words, and the values should be the appropriate tag string.

The `MFCTagger` class is provided to mock the interface of Pomegranite HMM models so that they can be used interchangeably._____no_output_____

# Create a lookup table mfc_table where mfc_table[word] contains the tag label most frequently assigned to that word
from collections import namedtuple

FakeState = namedtuple("FakeState", "name")

class MFCTagger:
    # NOTE: You should not need to modify this class or any of its methods
    missing = FakeState(name="<MISSING>")

    def __init__(self, table):
        self.table = defaultdict(lambda: MFCTagger.missing)
        self.table.update({word: FakeState(name=tag) for word, tag in table.items()})

    def viterbi(self, seq):
        """This method simplifies predictions by matching the Pomegranate viterbi() interface"""
        return 0., list(enumerate(["<start>"] + [self.table[w] for w in seq] + ["<end>"]))

# TODO: calculate the frequency of each tag being assigned to each word (hint: similar, but not
# the same as the emission probabilities) and use it to fill the mfc_table

def mfc_count(wordcounts):
    maxtable={}
    for word in data.training_set.vocab:
        maxtagval = max(wordcounts[word].values())
        maxtable[word] = list(wordcounts[word].keys())[list(wordcounts[word].values()).index(maxtagval)]
    return maxtable

word_counts = pair_counts(data.training_set.X, data.training_set.Y)

mfc_table = mfc_count(word_counts)

# DO NOT MODIFY BELOW THIS LINE
mfc_model = MFCTagger(mfc_table) # Create a Most Frequent Class tagger instance
assert len(mfc_table) == len(data.training_set.vocab), ""
assert all(k in data.training_set.vocab for k in mfc_table.keys()), ""
assert sum(int(k not in mfc_table) for k in data.testing_set.vocab) == 5521, ""
HTML('
Your MFC tagger has all the correct words!
')_____no_output_____

### Making Predictions with a Model
The helper functions provided below interface with Pomegranate network models & the mocked MFCTagger to take advantage of the [missing value](http://pomegranate.readthedocs.io/en/latest/nan.html) functionality in Pomegranate through a simple sequence decoding function. Run these functions, then run the next cell to see some of the predictions made by the MFC tagger._____no_output_____

def replace_unknown(sequence):
    """Return a copy of the input sequence where each unknown word is replaced
    by the literal string value 'nan'. Pomegranate will ignore these values
    during computation.
    """
    return [w if w in data.training_set.vocab else 'nan' for w in sequence]

def simplify_decoding(X, model):
    """X should be a 1-D sequence of observations for the model to predict"""
    _, state_path = model.viterbi(replace_unknown(X))
    return [state[1].name for state in state_path[1:-1]]  # do not show the start/end state predictions_____no_output_____

### Example Decoding Sequences with MFC Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, mfc_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', '<MISSING>', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADV', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

### Evaluating Model Accuracy

The function below will evaluate the accuracy of the MFC tagger on the collection of all sentences from a text corpus. _____no_output_____

def accuracy(X, Y, model):
    """Calculate the prediction accuracy by using the model to decode each sequence
    in the input X and comparing the prediction with the true labels in Y.

    The X should be an array whose first dimension is the number of sentences to test,
    and each element of the array should be an iterable of the words in the sequence.
    The arrays X and Y should have the exact same shape.

    X = [("See", "Spot", "run"), ("Run", "Spot", "run", "fast"), ...]
    Y = [(), (), ...]
    """
    correct = total_predictions = 0
    for observations, actual_tags in zip(X, Y):

        # The model.viterbi call in simplify_decoding will return None if the HMM
        # raises an error (for example, if a test sentence contains a word that
        # is out of vocabulary for the training set). Any exception counts the
        # full sentence as an error (which makes this a conservative estimate).
        try:
            most_likely_tags = simplify_decoding(observations, model)
            correct += sum(p == t for p, t in zip(most_likely_tags, actual_tags))
        except:
            pass
        total_predictions += len(observations)
    return correct / total_predictions_____no_output_____

#### Evaluate the accuracy of the MFC tagger
Run the next cell to evaluate the accuracy of the tagger on the training and test corpus._____no_output_____

mfc_training_acc = accuracy(data.training_set.X, data.training_set.Y, mfc_model)
print("training accuracy mfc_model: {:.2f}%".format(100 * mfc_training_acc))

mfc_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, mfc_model)
print("testing accuracy mfc_model: {:.2f}%".format(100 * mfc_testing_acc))

assert mfc_training_acc >= 0.955, "Uh oh. Your MFC accuracy on the training set doesn't look right."
assert mfc_testing_acc >= 0.925, "Uh oh. Your MFC accuracy on the testing set doesn't look right."
HTML('
Your MFC tagger accuracy looks correct!
')training accuracy mfc_model: 95.72%
testing accuracy mfc_model: 93.01%

## Step 3: Build an HMM tagger
---
The HMM tagger has one hidden state for each possible tag, and parameterized by two distributions: the emission probabilties giving the conditional probability of observing a given **word** from each hidden state, and the transition probabilities giving the conditional probability of moving between **tags** during the sequence.

We will also estimate the starting probability distribution (the probability of each **tag** being the first tag in a sequence), and the terminal probability distribution (the probability of each **tag** being the last tag in a sequence).

The maximum likelihood estimate of these distributions can be calculated from the frequency counts as described in the following sections where you'll implement functions to count the frequencies, and finally build the model. The HMM model will make predictions according to the formula:

$$t_i^n = \underset{t_i^n}{\mathrm{argmax}} \prod_{i=1}^n P(w_i|t_i) P(t_i|t_{i-1})$$

Refer to Speech & Language Processing [Chapter 10](https://web.stanford.edu/~jurafsky/slp3/10.pdf) for more information._____no_output_____### IMPLEMENTATION: Unigram Counts

Complete the function below to estimate the co-occurrence frequency of each symbol over all of the input sequences. The unigram probabilities in our HMM model are estimated from the formula below, where N is the total number of samples in the input. (You only need to compute the counts for now.)

$$P(tag_1) = \frac{C(tag_1)}{N}$$_____no_output_____

def unigram_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequence list that
    counts the number of occurrences of the value in the sequences list. The sequences
    collection should be a 2-dimensional array.

    For example, if the tag NOUN appears 275558 times over all the input sequences,
    then you should return a dictionary such that your_unigram_counts[NOUN] == 275558.
    """
    # TODO: Finish this function!
    unicountdict = {}
    for symbol in sequences:
        for item in symbol:
            if item in unicountdict.keys():
                unicountdict[item] += 1
            else:
                unicountdict[item] = 1
    return unicountdict

# TODO: call unigram_counts with a list of tag sequences from the training set
tag_unigrams = unigram_counts(data.training_set.Y)

assert set(tag_unigrams.keys()) == data.training_set.tagset, \
       "Uh oh. It looks like your tag counts doesn't include all the tags!"
assert min(tag_unigrams, key=tag_unigrams.get) == 'X', \
       "Hmmm...'X' is expected to be the least common class"
assert max(tag_unigrams, key=tag_unigrams.get) == 'NOUN', \
       "Hmmm...'NOUN' is expected to be the most common class"
HTML('
Your tag unigrams look good!
')_____no_output_____

### IMPLEMENTATION: Bigram Counts

Complete the function below to estimate the co-occurrence frequency of each pair of symbols in each of the input sequences. These counts are used in the HMM model to estimate the bigram probability of two tags from the frequency counts according to the formula: $$P(tag_2|tag_1) = \frac{C(tag_2|tag_1)}{C(tag_2)}$$
_____no_output_____

def bigram_counts(sequences):
    """Return a dictionary keyed to each unique PAIR of values in the input sequences
    list that counts the number of occurrences of pair in the sequences list. The input
    should be a 2-dimensional array.

    For example, if the pair of tags (NOUN, VERB) appear 61582 times, then you should
    return a dictionary such that your_bigram_counts[(NOUN, VERB)] == 61582
    """

    # TODO: Finish this function!
    bicountdict = {}
    for symbol in sequences:
        for i in range(1, len(symbol)):
            if (symbol[i-1], symbol[i]) in bicountdict.keys():
                bicountdict[(symbol[i-1], symbol[i])] += 1
            else:
                bicountdict[(symbol[i-1], symbol[i])] = 1
    return bicountdict

# TODO: call bigram_counts with a list of tag sequences from the training set
tag_bigrams = bigram_counts(data.training_set.Y)

assert len(tag_bigrams) == 144, \
       "Uh oh. There should be 144 pairs of bigrams (12 tags x 12 tags)"
assert min(tag_bigrams, key=tag_bigrams.get) in [('X', 'NUM'), ('PRON', 'X')], \
       "Hmmm...The least common bigram should be one of ('X', 'NUM') or ('PRON', 'X')."
assert max(tag_bigrams, key=tag_bigrams.get) in [('DET', 'NOUN')], \
       "Hmmm...('DET', 'NOUN') is expected to be the most common bigram."
HTML('
Your tag bigrams look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Starting Counts
Complete the code below to estimate the bigram probabilities of a sequence starting with each tag._____no_output_____

def starting_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the beginning of
    a sequence.

    For example, if 8093 sequences start with NOUN, then you should return a
    dictionary such that your_starting_counts[NOUN] == 8093
    """
    # TODO: Finish this function!
    startcountdict = {}
    cont = 0
    for symbol in sequences:
        if symbol[cont] in startcountdict.keys():
            startcountdict[symbol[cont]] += 1
            #cont += 1
        else:
            startcountdict[symbol[cont]] = 1
    return startcountdict

# TODO: Calculate the count of each tag starting a sequence
tag_starts = starting_counts(data.training_set.Y)

assert len(tag_starts) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_starts, key=tag_starts.get) == 'X', "Hmmm...'X' is expected to be the least common starting bigram."
assert max(tag_starts, key=tag_starts.get) == 'DET', "Hmmm...'DET' is expected to be the most common starting bigram."
HTML('
Your starting tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Ending Counts
Complete the function below to estimate the bigram probabilities of a sequence ending with each tag._____no_output_____

def ending_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the end of
    a sequence.

    For example, if 18 sequences end with DET, then you should return a
    dictionary such that your_starting_counts[DET] == 18
    """
    # TODO: Finish this function!
    endcountdict = {}
    cont = 0
    for symbol in sequences:
        if symbol[cont - 1] in endcountdict.keys():
            endcountdict[symbol[cont - 1]] += 1
            #cont += 1
        else:
            endcountdict[symbol[cont - 1]] = 1
    return endcountdict

# TODO: Calculate the count of each tag ending a sequence
tag_ends = ending_counts(data.training_set.Y)

assert len(tag_ends) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_ends, key=tag_ends.get) in ['X', 'CONJ'], "Hmmm...'X' or 'CONJ' should be the least common ending bigram."
assert max(tag_ends, key=tag_ends.get) == '.', "Hmmm...'.' is expected to be the most common ending bigram."
HTML('
Your ending tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Basic HMM Tagger
Use the tag unigrams and bigrams calculated above to construct a hidden Markov tagger.

- Add one state per tag
    - The emission distribution at each state should be estimated with the formula: $P(w|t) = \frac{C(t, w)}{C(t)}$
- Add an edge from the starting state `basic_model.start` to each tag
    - The transition probability should be estimated with the formula: $P(t|start) = \frac{C(start, t)}{C(start)}$
- Add an edge from each tag to the end state `basic_model.end`
    - The transition probability should be estimated with the formula: $P(end|t) = \frac{C(t, end)}{C(t)}$
- Add an edge between _every_ pair of tags
    - The transition probability should be estimated with the formula: $P(t_2|t_1) = \frac{C(t_1, t_2)}{C(t_1)}$_____no_output_____

basic_model = HiddenMarkovModel(name="base-hmm-tagger")

# TODO: create states with emission probability distributions P(word | tag) and add to the model
# (Hint: you may need to loop & create/add new states)
emission_counts = pair_counts(data.training_set.Y, data.training_set.X)
states = {}
for tag, word in emission_counts.items():
    prob = {key: float(val)/sum(emission_counts[tag].values()) for key, val in word.items()}
    emission = DiscreteDistribution(prob)
    states[tag] = State(emission, name = tag)
    basic_model.add_states(State(emission, name = tag))

# TODO: add edges between states for the observed transition frequencies P(tag_i | tag_i-1)
# (Hint: you may need to loop & add transitions
#starting counts
for startkeys, startvals in tag_starts.items():
    basic_model.add_transition(basic_model.start, states[startkeys], float(startvals)/sum(tag_starts.values()))

#ending counts
for endkeys, endvals in tag_ends.items():
    basic_model.add_transition(states[endkeys], basic_model.end, float(endvals)/tag_unigrams[endkeys])

#bigram counts
for bikeys, bivals in tag_bigrams.items():
    basic_model.add_transition(states[bikeys[0]], states[bikeys[1]], float(bivals)/tag_unigrams[bikeys[0]])

# NOTE: YOU SHOULD NOT NEED TO MODIFY ANYTHING BELOW THIS LINE
# finalize the model
basic_model.bake()

assert all(tag in set(s.name for s in basic_model.states) for tag in data.training_set.tagset), \
       "Every state in your network should use the name of the associated tag, which must be one of the training set tags."
assert basic_model.edge_count() == 168, \
       ("Your network should have an edge from the start node to each state, one edge between every " +
        "pair of tags (states), and an edge from each state to the end node.")
HTML('
Your HMM network topology looks good!
')_____no_output_____hmm_training_acc = accuracy(data.training_set.X, data.training_set.Y, basic_model)
print("training accuracy basic hmm model: {:.2f}%".format(100 * hmm_training_acc))

hmm_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, basic_model)
print("testing accuracy basic hmm model: {:.2f}%".format(100 * hmm_testing_acc))

assert hmm_training_acc > 0.97, "Uh oh. Your HMM accuracy on the training set doesn't look right."
assert hmm_training_acc > 0.955, "Uh oh. Your HMM accuracy on the training set doesn't look right."
HTML('
Your HMM tagger accuracy looks correct! Congratulations, you\'ve finished the project.
')training accuracy basic hmm model: 97.54%
testing accuracy basic hmm model: 95.95%

### Example Decoding Sequences with the HMM Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, basic_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

## Finishing the project
---

**Note:** **SAVE YOUR NOTEBOOK**, then run the next cell to generate an HTML copy. You will zip & submit both this file and the HTML copy for review.

_____no_output_____

!!jupyter nbconvert *.ipynb_____no_output_____

## Step 4: [Optional] Improving model performance
---
There are additional enhancements that can be incorporated into your tagger that improve performance on larger tagsets where the data sparsity problem is more significant. The data sparsity problem arises because the same amount of data split over more tags means there will be fewer samples in each tag, and there will be more missing data  tags that have zero occurrences in the data. The techniques in this section are optional.

- [Laplace Smoothing](https://en.wikipedia.org/wiki/Additive_smoothing) (pseudocounts)
    Laplace smoothing is a technique where you add a small, non-zero value to all observed counts to offset for unobserved values.

- Backoff Smoothing
    Another smoothing technique is to interpolate between n-grams for missing data. This method is more effective than Laplace smoothing at combatting the data sparsity problem. Refer to chapters 4, 9, and 10 of the [Speech & Language Processing](https://web.stanford.edu/~jurafsky/slp3/) book for more information.

- Extending to Trigrams
    HMM taggers have achieved better than 96% accuracy on this dataset with the full Penn treebank tagset using an architecture described in [this](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf) paper. Altering your HMM to achieve the same performance would require implementing deleted interpolation (described in the paper), incorporating trigram probabilities in your frequency tables, and re-implementing the Viterbi algorithm to consider three consecutive states instead of two.

### Obtain the Brown Corpus with a Larger Tagset
Run the code below to download a copy of the brown corpus with the full NLTK tagset. You will need to research the available tagset information in the NLTK docs and determine the best way to extract the subset of NLTK tags you want to explore. If you write the following the format specified in Step 1, then you can reload the data using all of the code above for comparison.

Refer to [Chapter 5](http://www.nltk.org/book/ch05.html) of the NLTK book for more information on the available tagsets._____no_output_____

import nltk
from nltk import pos_tag, word_tokenize
from nltk.corpus import brown

nltk.download('brown')
training_corpus = nltk.corpus.brown
training_corpus.tagged_sents()[0]_____no_output_____


# Notebook from york-stsci/jwst_validation_notebooks
Path: jwst_validation_notebooks/background/jwst_background_miri_test/jwst_background_subtraction_miri_imaging_testing.ipynb

# JWST Pipeline Validation Notebook: calwebb_image2, background subtraction for MIRI imaging

 **Instruments Affected**: e.g., FGS, MIRI, NIRCam, NIRISS

### Table of Contents

 [Introduction\*](#intro)

 [JWST CalWG Algorithm\*](#algorithm)

 [Defining Terms](#terms)

 [Test Description\*](#description)

 [Data Description\*](#data_descr)

 [Imports\*](#imports)

 [Loading the Data\*](#data_load)

 [Run the Pipeline](#pipeline)

 [Passing criteria](#testing)

 [About This Notebook\*](#about)

_____no_output_____
# Introduction

This is the validation notebook for the background subtraction step as part of calwebb_image2. This step takes in a set of images and a set of background observations. If more than one background observation is given, they are combined into a sigma clipped mean before being subtracted from each of the science data images. For more information on the pipeline step visit the links below.

Step description: https://jwst-pipeline.readthedocs.io/en/latest/jwst/background_step/description.html

Pipeline code: https://github.com/spacetelescope/jwst/tree/master/jwst/background

[Top of Page](#title_ID)_____no_output_____
# JWST CalWG Algorithm

The page describing the algorithm and any details can be found here:

https://outerspace.stsci.edu/display/JWSTCC/Vanilla+Imaging+Background+Subtraction

[Top of Page](#title_ID)_____no_output_____
# Defining Terms

Here are some common terms that will be used throughout the notebook

> JWST: James Webb Space Telescope

> MIRI: Mid-Infrared Instrument

> MIRISim: Simulator that creates pipeline ready MIRI simulated data

[Top of Page](#title_ID)_____no_output_____
# Test Description

This test is performed by creating a set of simulated data with multiple point sources located at random locations across the MIRI field of view. There is a set of 4 simulated images at four dithered positions with 300+ stars and a galaxy in the MIRI field of view (including Lyot). This test also takes in a set of four simulated background images, at four dithered positions, with around 40 fainter stars across the field of view and Lyot mask. All images will be processed through calwebb_detector1, and put into an association file to be run through calwebb_image2. This will tell the background step which are the science observations and which are the background observations so that it will do a sigma clipped mean of the background exposures, then subtract the mean from each of the science observations.

The notebook shows the images (background, science, averaged background, and background subtracted) through the course of the notebook to demonstrate how well the algorithm works.

[Top of Page](#title_ID)_____no_output_____
# Data Description

The set of data used in this particular test were created with the MIRI Data Simulator (MIRISim). There is a set of 4 simulated images at four dithered positions with 300+ stars and a galaxy in the MIRI field of view (including Lyot). This test also takes in a set of four simulated background images, at four dithered positions, with around 40 fainter stars across the field of view and Lyot mask. All of the images were created with the F770W filter.

[Top of Page](#title_ID)_____no_output_____
# Set up Temporary Directory
The following cell sets up a temporary directory (using python's `tempfile.TemporaryDirectory()`), and changes the script's active directory into that directory (using python's `os.chdir()`). This is so that, when the notebook is run through, it will download files to (and create output files in) the temporary directory rather than in the notebook's directory. This makes cleanup significantly easier (since all output files are deleted when the notebook is shut down), and also means that different notebooks in the same directory won't interfere with each other when run by the automated webpage generation process.

If you want the notebook to generate output in the notebook's directory, simply don't run this cell.

If you have a file (or files) that are kept in the notebook's directory, and that the notebook needs to use while running, you can copy that file into the directory (the code to do so is present below, but commented out).

[Top of Page](#title_ID)_____no_output_____

#****
#
# Set this variable to False to not use the temporary directory
#
#****
use_tempdir = True

# Create a temporary directory to hold notebook output, and change the working directory to that directory.
from tempfile import TemporaryDirectory
import os
import shutil

if use_tempdir:
    data_dir = TemporaryDirectory()

    # Save original directory
    orig_dir = os.getcwd()

    # Move to new directory
    os.chdir(data_dir.name)

# For info, print out where the script is running
print("Running in {}".format(os.getcwd()))_____no_output_____

## If Desired, set up CRDS to use a local cache

By default, the notebook template environment sets up its CRDS cache (the "CRDS_PATH" environment variable) in /grp/crds/cache. However, if the notebook is running on a local machine without a fast and reliable connection to central storage, it makes more sense to put the CRDS cache locally. Currently, the cell below offers several options, and will check the supplied boolean variables one at a time until one matches.

* if `use_local_crds_cache` is False, then the CRDS cache will be kept in /grp/crds/cache
* if `use_local_crds_cache` is True, the CRDS cache will be kept locally
  * if `crds_cache_tempdir` is True, the CRDS cache will be kept in the temporary directory
  * if `crds_cache_notebook_dir` is True, the CRDS cache will be kept in the same directory as the notebook.
  * if `crds_cache_home` is True, the CRDS cache will be kept in $HOME/crds/cache
  * if `crds_cache_custom_dir` is True, the CRDS cache will be kept in whatever is stored in the
    `crds_cache_dir_name` variable.

If the above cell (creating a temporary directory) is not run, then setting `crds_cache_tempdir` to True will store the CRDS cache in the notebook's directory (the same as setting `crds_cache_notebook_dir` to True)._____no_output_____

import os

# Choose CRDS cache location
use_local_crds_cache = True
crds_cache_tempdir = False
crds_cache_notebook_dir = False
crds_cache_home = False
crds_cache_custom_dir = False
crds_cache_dir_name = ""

if use_local_crds_cache:
    if crds_cache_tempdir:
        os.environ['CRDS_PATH'] = os.path.join(os.getcwd(), "crds")
    elif crds_cache_notebook_dir:
        try:
            os.environ['CRDS_PATH'] = os.path.join(orig_dir, "crds")
        except Exception as e:
            os.environ['CRDS_PATH'] = os.path.join(os.getcwd(), "crds")
    elif crds_cache_home:
        os.environ['CRDS_PATH'] = os.path.join(os.environ['HOME'], 'crds', 'cache')
    elif crds_cache_custom_dir:
        os.environ['CRDS_PATH'] = crds_cache_dir_name_____no_output_____

# Imports
List the package imports and why they are relevant to this notebook.

* astropy.io for opening fits files
* inspect to get the docstring of our objects.
* IPython.display for printing markdown output
* jwst.datamodels for building model for JWST Pipeline
* jwst.module.PipelineStep is the pipeline step being tested
* matplotlib.pyplot.plt to generate plot

[Top of Page](#title_ID)_____no_output_____

# import inspect
from astropy.io import fits
# from IPython.display import Markdown
from ci_watson.artifactory_helpers import get_bigdata
import glob

from jwst.datamodels import RampModel, ImageModel
from jwst.pipeline import Detector1Pipeline, Image2Pipeline
from jwst import associations
from jwst.associations.lib.rules_level2_base import DMSLevel2bBase
from jwst.associations import asn_from_list
from jwst.background import BackgroundStep

import matplotlib
import matplotlib.pyplot as plt
import numpy as np_____no_output_____

# Loading the Data

Download data from  Box to use in the notebook.

[Top of Page](#title_ID)_____no_output_____

from astropy.utils.data import download_file
from pathlib import Path
from shutil import move
from os.path import splitext

def get_box_files(file_list):
    for box_url,file_name in file_list:
        if 'https' not in box_url:
            box_url = 'https://stsci.box.com/shared/static/' + box_url
        downloaded_file = download_file(box_url)
        if Path(file_name).suffix == '':
            ext = splitext(box_url)[1]
            file_name += ext
        move(downloaded_file, file_name)

file_urls = ['https://stsci.box.com/shared/static/4i7z7q24rotc52m6bcr5ymekt2whuwto.fits',
             'https://stsci.box.com/shared/static/0fsqi7diuejt4uosjsnqv2q9qx4g7zk4.fits',
             'https://stsci.box.com/shared/static/kd1liugoruiz40v5u5bu19u68zn51iec.fits',
             'https://stsci.box.com/shared/static/o0v5skb6ohnj8qdhk64b07kvgldn37e3.fits',
             'https://stsci.box.com/shared/static/i30rmp4vz337j0qslbqn0sddraarpnl8.fits',
             'https://stsci.box.com/shared/static/o6gsumrc7w2orqu4dlo3limi37pduxtg.fits',
             'https://stsci.box.com/shared/static/075munkxxlk2g1009irs6zanjco2e2cq.fits',
             'https://stsci.box.com/shared/static/ppdyqjud4g5u1eaksztyx5b8rl1bzzzp.fits']

files = ['starfield_med_seq1_MIRIMAGE_F770Wexp1.fits',
         'starfield_med_seq2_MIRIMAGE_F770Wexp1.fits',
         'starfield_med_seq3_MIRIMAGE_F770Wexp1.fits',
         'starfield_med_seq4_MIRIMAGE_F770Wexp1.fits',
         'bkg_faint_seq1_MIRIMAGE_F770Wexp1.fits',
         'bkg_faint_seq2_MIRIMAGE_F770Wexp1.fits',
         'bkg_faint_seq3_MIRIMAGE_F770Wexp1.fits',
         'bkg_faint_seq4_MIRIMAGE_F770Wexp1.fits']

box_download_list = [(url,name) for url,name in zip(file_urls,files)]

get_box_files(box_download_list)

#This readnoise file is needed for use with simulated data which has higher readnoise than actual data.
readnoise = get_bigdata('jwst_validation_notebooks',
                     'validation_data',
                     'jump',
                     'jump_miri_test',
                     'jwst_mirisim_readnoise.fits')

print(files)_____no_output_____

# Run calwebb_detector1

[Top of Page](#title_ID)_____no_output_____

# Run the calwebb_detector1 pipeline

# set up pipeline parameters
rej_thresh=10.0  # rejection threshold for jump step

print('There are ', len(files), ' images.')

scislopelist = []
bkgslopelist = []

# loop over list of files
for file in files:

    # set up pipeline parameters for input
    pipe1 = Detector1Pipeline()
    pipe1.jump.rejection_threshold = rej_thresh
    pipe1.jump.override_readnoise = readnoise
    pipe1.ramp_fit.override_readnoise = readnoise

    pipe1.refpix.skip = True  # needs update to simulator for this to work properly with simulated data

    # set up output file name
    base, remainder = file.split('.')
    outname = base

    #pipe1.jump.output_file = outname+'.fits'
    #pipe1.ramp_fit.output_file = outname+'.fits'
    pipe1.output_file = outname+'.fits'

    # Run pipeline on each file
    rampfile = pipe1.run(file)
    if 'starfield' in file:
        scislopelist.append(rampfile)
    else:
        bkgslopelist.append(rampfile)

print('Detector 1 steps completed on all files.')
print('Science files: ', scislopelist)
print('Background files: ', bkgslopelist)_____no_output_____

### Look at rate images

Display the rate science images and background images to see locations of sources._____no_output_____

# Look at science images

for image in scislopelist:
    im = ImageModel(image)

    plt.figure(figsize=(20,20))
    plt.imshow(im.data, cmap='rainbow', origin='lower', vmin=0,vmax=10)
    plt.colorbar()
    plt.show()
    print('background region values', im.data[850, 500])_____no_output_____# Look at background images
# Pixel to examine that should have a source in at least one background image
xval = 615
yval = 317

for backimage in bkgslopelist:
    bkgim = ImageModel(backimage)
    print(bkgim.meta.filename)
    selectedstar = bkgim.data[yval, xval] # Choose a pixel that is on a source in at least one image

    plt.figure(figsize=(20,20))
    plt.imshow(bkgim.data, cmap='rainbow', origin='lower', vmin=0,vmax=10)
    plt.colorbar()
    plt.show()

    print('brightness of selected pixel', selectedstar, '\n')_____no_output_____

### Create a Level2 association file of the science and background exposures_____no_output_____

# Create an association file of all of the rate files

asn_files = [scislopelist[0].meta.filename, scislopelist[1].meta.filename, scislopelist[2].meta.filename,
            scislopelist[3].meta.filename]
bgr_files = [bkgslopelist[0].meta.filename, bkgslopelist[1].meta.filename, bkgslopelist[2].meta.filename,
            bkgslopelist[3].meta.filename]

asn = asn_from_list.asn_from_list(asn_files, rule=DMSLevel2bBase, meta={'program':'test', 'target':'randomfield', 'asn_pool':'test'})

# now add the opposite nod as background exposure:
for product in asn['products']:
    product['members'].append({'expname':bgr_files[0], 'exptype':'background'})
    product['members'].append({'expname':bgr_files[1], 'exptype':'background'})
    product['members'].append({'expname':bgr_files[2], 'exptype':'background'})
    product['members'].append({'expname':bgr_files[3], 'exptype':'background'})

# write this out to a json file
with open('imager_bkgsubtest_asn.json', 'w') as fp:
    fp.write(asn.dump()[1])_____no_output_____

## Run association file through background subtraction step of calwebb_image2

The default value of sigma for the background subtract step is set to 3, but may need to be adjusted downward to 2 or 1 in order to actually sigma clip the sources in the images. Test this for your data. For this particular data set, 2 is sufficient, but for brighter sources, 1 may be the best option._____no_output_____

pipe2 = Image2Pipeline()
# Set pipeline parameters
pipe2.save_results = True
pipe2.bkg_subtract.sigma = 2  # Set this in order to catch the outliers and leave only background
pipe2.bkg_subtract.maxiters = 3
pipe2.bkg_subtract.save_combined_background = True

pipe2.assign_wcs.skip = True
pipe2.flat_field.skip = True
pipe2.photom.skip = True
pipe2.resample.skip = True
pipe2.save_bsub = True

pipe2.run('imager_bkgsubtest_asn.json')
_____no_output_____

### Look at averaged background image

See how well the sigma clipping did at removing the sources from the background image. If the sources in the background image are bright, the value of sigma should be set to 1. If the sources are faint enough, the default value of 3 should be good enough.

Also look at the value of a specific pixel in the averaged image, one that has a source in at least one of the background images, to see if the flux was adequately removed in the sigma clipping process._____no_output_____

# This can only be uncommented in builds after 7.7.1
averaged_backgrounds = glob.glob('*combinedbackground.fits')
print(averaged_backgrounds)

avgbkg = ImageModel(averaged_backgrounds[0])
selectedavgstar = avgbkg.data[yval, xval]  # Choose a pixel location that contains a source in at least one background image

plt.figure(figsize=(20,20))
plt.imshow(avgbkg.data, cmap='rainbow', origin='lower', vmin=0,vmax=10)
plt.colorbar()
plt.show()

print('brightness of selected pixel', selectedavgstar)_____no_output_____

Look at the values of pixels in the background images at the location of one of the stars to see whether the star flux is being rejected as part of the sigma clipping. In calwebb_image2, the sigma value was set to 2, which is lower than the default value of 3. This allows the most pixels to be rejected as outliers, and should leave only the background values in the final averaged image._____no_output_____

### Check averaging of background images

print(bkgslopelist,'\n')
im1 = ImageModel(bkgslopelist[0])
im2 = ImageModel(bkgslopelist[1])
im3 = ImageModel(bkgslopelist[2])
im4 = ImageModel(bkgslopelist[3])

print('Value in image1 ', im1.data[yval, xval])
print('Value in image2 ', im2.data[yval, xval])
print('Value in image3 ', im3.data[yval, xval])
print('Value in image4 ', im4.data[yval, xval],'\n')

avgvalue = (im1.data[yval, xval]+ im2.data[yval, xval] + im3.data[yval, xval] + im4.data[yval, xval])/4
print('Averaged value = ', avgvalue)
print()

print('Brightness of selected pixel in averaged image', selectedavgstar)_____no_output_____

### Look at background subtracted data
_____no_output_____

# Look at background image
subtracted_images = glob.glob('*bsub.fits')

for bkgsubimage in subtracted_images:
    bkgsub = ImageModel(bkgsubimage)

    plt.figure(figsize=(20,20))
    plt.imshow(bkgsub.data, cmap='rainbow', origin='lower', vmin=-1,vmax=5)
    plt.colorbar()
    plt.show()
    print('background region values', bkgsub.data[850, 500])
    try:
        np.testing.assert_allclose(bkgsub.data[850, 500], 0.01, atol=0.8)
        print('Background subtraction test PASSED: background is near zero')
    except AssertionError as e:
        print('Test FAILED: Subtracted background value is not near zero')_____no_output_____

# Passing criteria

Examine the images shown and the pixel values reported through the notebook. If the averaged background image is subtracted from the science images (subtracted background values near 0), and the averaged background image shows a smooth background with the sources removed, then this test passes. The four background images should be averaged together after the sources were rejected via sigma clipping. Check that the subtracted background values are near 0.

[Top of Page](#title_ID)_____no_output_____
## About this Notebook
**Author:** M Cracraft, Senior Staff Scientist, INS/MIRI branch

**Updated On:** 06/09/2021_____no_output_____[Top of Page](#title_ID)
 _____no_output_____


# Notebook from rpatil524/CKG-COVID-19
Path: build-covid-kg.ipynb

# Constructing ISI's COVID-19 Knowledge Graph_____no_output_____This notebook shows how we use our KGTK toolking (link) to construct the COVID-19 knowledge graph for the CORD-19 corpus using the text extractions from the BLENDER group at UIUC. The text extractions inlcude a variety of entity extractions with links to bioinformatics databases. Our approach is to:

* extract from Wikidata the subgraph that covers all the publications in the CORD-19 corpus and the entities identified by the BLENDER group.
* define new Wikidata items for publications or entities that are not present in the public Wikidata
* annotate the article items with the relevant entities
* preserve provenance

We implement the approach in the following steps:

* data prepararation: convert the JSON representation of BLENDER output to CSV files that are easy to process
* extract Wikidata subgraph: extract from Wikidata the articles, authors, and entities mentioned in the BLENDER corpus
* create missing items: create nodes for articles and entities that are not present in Wikidata
* create mention edges: create edges to record the entity extractions from BLENDER, including justifications
* incorporate analytic outputs: add edges to record graph metrics such as pagerank
* export knowledge graph: export the graph to KGTL edges, RDF and Neo4J _____no_output_____## Data Preparation_____no_output_____Set up environment variables with location of the input files_____no_output_____

import numpy as np
import pandas as pd
import os
import json_____no_output_____%env COVID=/Users/amandeep/Github/CKG-Covid/datasets/sandbox
%env WD=/Users/amandeep/Documents/wikidata-20200504env: COVID=/Users/amandeep/Github/CKG-Covid/datasets/sandbox
env: WD=/Users/amandeep/Documents/wikidata-20200504

### Wikidata Data_____no_output_____The Wikidata files are the large:

* `wikidata_nodes_20200504.tsv.gz` the English labels, alias and descriptions for all items in Wikidata
* `wikidata_edges_20200504.tsv.gz` the edges for all statements in Wikidata
* `wikidata_qualifiers_20200504.tsv.gz` the qualifier edges for all statements in Wikidata_____no_output_____

ls -lh "$WD"_____no_output_____

Working with the dump files takes time because they are so large (86B chars and 1.1B lines), more than 6 minutes to just unzip and count lines_____no_output_____

!time gzcat "$WD/wikidata_edges_20200504.tsv.gz" | wc_____no_output_____

The columns in the edge files:

* `id` is a unique identifier for an edge, and provides an identifier for each statement in Wikidata
* `node1`, `label` and `node2` are the item/property/value or subject/predicate/object
* `rank` is the Wikidata rank for the statement
* `node2;*` are additional columns that provide detailed information about the `node2` column, making it easy to parse_____no_output_____

!gzcat "$WD/wikidata_edges_20200504.tsv.gz" | head_____no_output_____

### BLENDER Data_____no_output_____The BLENDER data came in a JSON document per article. We used Python scripts to create simple TSV files that are easy to process.

The `corpus-identifiers.tsv` file contains all the identifiers present in the BLENDER dataset with two columns:

* node2: the value of the identifier
* label: the name of the property in Wikidata used to represent the specific identifier. For example `P698` is PubMed ID. See https://www.wikidata.org/wiki/Q93157077_____no_output_____

ids = pd.read_csv(os.getenv("COVID")+'/corpus-identifiers.tsv', delimiter='\t')
ids_____no_output_____

## Extract Wikidata Subgraph_____no_output_____### Step 1: find the `node1` for all the rows in `corpus-identifiers.tsv`_____no_output_____We do this with the `kgtk ifexists` command to scan the file of all edges in Wikidata to select the ones where `label` and `node2` match the rows in `corpus-identifiers.tsv`. We store the results in `corpus-identifier-edges.tsv`_____no_output_____

!time gzcat "$WD/wikidata_edges_20200504.tsv.gz" \
  | kgtk ifexists --input-keys label node2 --filter-on $COVID/corpus-identifiers.tsv --filter-keys label node2 --filter-mode NONE \
  > $COVID/corpus-identifier-edges.tsv
real	43m17.833s
user	46m24.846s
sys	0m53.892s

We found 72,649 q-nodes in Wikidata for the identifiers in our corpus. _____no_output_____

idedges = pd.read_csv(os.getenv("COVID")+'/corpus-identifier-edges.tsv', delimiter='\t')
#idedges.loc[:, [ 'node2']]_____no_output_____idedges_____no_output_____

Here are the counts of the q-nodes we found in Wikidata for each property._____no_output_____

idedges['label'].value_counts()_____no_output_____

### Step 2: get all the edges from Wikidata for the q-nodes in `corpus-identifier-edges.tsv`_____no_output_____To do this, we again scan all the edges in Wikidata looking for edges whose `node1` matches the `node1` in `corpus-identifier-edges.tsv`_____no_output_____

!time gzcat "$WD/wikidata_edges_20200504.tsv.gz" \
  | kgtk ifexists --input-keys node1 --filter-on $COVID/corpus-identifier-edges.tsv --filter-keys node1 \
  > $COVID/corpus-edges.tsv_____no_output_____

We now have 2.2 million edges for the entities in our corpus._____no_output_____

!wc $COVID/corpus-edges.tsv 2297783 13662891 176151115 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/corpus-edges.tsv

We can use Pandas to explore the data_____no_output_____

ce = pd.read_csv(os.getenv("COVID")+'/corpus-edges.tsv', delimiter='\t', index_col=['id'], dtype=object)_____no_output_____ce.loc[:, ['node1', 'label', 'node2', 'node2;entity-type']].head(5)_____no_output_____ce[ce['node2;entity-type']=='item'].loc[:, ['node1', 'label', 'node2', 'node2;entity-type']]_____no_output_____ce[ce['node2;entity-type']=='item'].label.value_counts()_____no_output_____ce.label.value_counts()[0:20]_____no_output_____

We can use the Wikidata cli to see what the top properties:_____no_output_____

!wd u P2860, P704, P2093id P2860
Label cites work
Description citation from one creative work to another
instance of (P31): Wikidata property for items about works (Q18618644)

id P704
Label Ensembl transcript ID
Description transcript ID issued by Ensembl database
instance of (P31): Wikidata property for an identifier (Q19847637) | Wikidata property related to medicine (Q19887775)

id P2093
Label author name string
Description string to store unspecified author name for publications; use if Wikidata item for author (P50) does not exist or is not known
instance of (P31): Wikidata property for items about works (Q18618644) | Wikidata property with datatype string that is not an external identifier (Q21099935) | Wikidata property to indicate a source (Q18608359)

How many authors have items in Wikidata?_____no_output_____

ce.node2.value_counts()[0:20]_____no_output_____!wd u Q13442814, Q7187, Q20747295, Q15978631id Q13442814
Label scholarly article
Description article in an academic publication, usually peer reviewed
subclass of (P279): scholarly publication (Q591041) | article (Q191067) | scholarly work (Q55915575)

id Q7187
Label gene
Description basic physical and functional unit of heredity
subclass of (P279): Nucleic acid sequence (Q863908) | biological region (Q50365914) | biological sequence (Q3511065)

id Q20747295
Label protein-coding gene
Description Type of a gene
subclass of (P279): gene (Q7187)

id Q15978631
Label Homo sapiens
Description species of mammal
instance of (P31): taxon (Q16521)
!kgtk filter -p ';P50;' $COVID/corpus-edges.tsv | wc -l   25284

### Step 3:_____no_output_____

!kgtk filter -p ';P50, P2860;' $COVID/corpus-edges.tsv > $COVID/citation-and-author-edges.tsv_____no_output_____caae = pd.read_csv(os.getenv("COVID")+'/citation-and-author-edges.tsv', delimiter='\t')
caae.loc[:, ['id', 'node1', 'label', 'node2']]_____no_output_____

Let's look at some of the items we got_____no_output_____

!wd u Q24611162, Q61104970id Q24611162
Label Viral mutation rates
Description scientific article
instance of (P31): scholarly article (Q13442814)

id Q61104970
Label Veerasak Punyapornwithaya
Description researcher ORCID ID = 0000-0001-9870-7773
instance of (P31): human (Q5)

Fetch all the edges from Wikidata about the authors and citations in our corpus.
We do this by scanning the Wikidata edges file and extraction all edges where node1 matches node2 in `citation-and-author-edges.tsv`_____no_output_____

!time gzcat "$WD/wikidata_edges_20200504.tsv.gz" \
  | kgtk ifexists --input-keys node1 --filter-on $COVID/citation-and-author-edges.tsv --filter-keys node2 \
  > $COVID/corpus-citations-and-authors.tsv_____no_output_____

Keep the edges for the articles and entities that we have in the BLENDER corpus_____no_output_____First extract all the edges that we may want to use (this takes 107 minutes)_____no_output_____

!wc $COVID/corpus-citations-and-authors.tsv 11389393 78495849 854598718 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/corpus-citations-and-authors.tsv
!head $COVID/corpus-citations-and-authors.tsvid	node1	label	node2	rank	node2;magnitude	node2;unit	node2;date	node2;item	node2;lower	node2;upper	node2;latitude	node2;longitude	node2;precision	node2;calendar	node2;entity-type
Q493567-P21-1	Q493567	P21	Q6581097	normal				Q6581097							item
Q493567-P106-1	Q493567	P106	Q3779582	normal				Q3779582							item
Q493567-P106-2	Q493567	P106	Q39631	normal				Q39631							item
Q493567-P106-3	Q493567	P106	Q82955	normal				Q82955							item
Q493567-P106-4	Q493567	P106	Q1622272	normal				Q1622272							item
Q493567-P106-5	Q493567	P106	Q15634281	normal				Q15634281							item
Q493567-P19-1	Q493567	P19	Q984894	normal				Q984894							item
Q493567-P244-1	Q493567	P244	"n88290830"	normal
Q493567-P214-1	Q493567	P214	"69131111"	normal
ccau = pd.read_csv(os.getenv("COVID")+'/corpus-citations-and-authors.tsv', delimiter='\t', dtype=object)
ccau.loc[:, ['id', 'node1', 'label', 'node2']]_____no_output_____ccau.node2.value_counts()[0:20]_____no_output_____!wd u Q13442814, Q1860, Q1251128id Q13442814
Label scholarly article
Description article in an academic publication, usually peer reviewed
subclass of (P279): scholarly publication (Q591041) | article (Q191067) | scholarly work (Q55915575)

id Q1860
Label English
Description West Germanic language originating in England with linguistic roots in French, German and Vulgar Latin
instance of (P31):  natural language (Q33742) | modern language (Q1288568) | language (Q34770)
subclass of (P279): Anglic languages (Q1346342)

id Q1251128
Label Journal of Virology
Description scientific journal
instance of (P31): scientific journal (Q5633421) | delayed open access journal (Q5253501)
!cat $COVID/corpus-identifier-edges.tsv | kgtk filter -p ";P698,P932;" | gzip > $COVID/corpus-article-identifier-edges.tsv.gz_____no_output_____!cat $COVID/corpus-identifier-edges.tsv | kgtk filter -p ";P685,P486,P351,P5055;" | gzip > $COVID/corpus-entity-identifier-edges.tsv.gz_____no_output_____

## Create Missing Items_____no_output_____### Step 4: find the identifiers in `corpus-identifiers.tsv` for which there is no `node1` in Wikidata_____no_output_____We do this with the `kgtk ifnotexists` command to scan the file corpus-identifiers.tsv to select the ones which do not have a `node1` in the file `corpus-identifier-edges.tsv`. We store the results in `corpus-identifiers-not-in-wikidata.tsv`_____no_output_____

!cat $COVID/corpus-identifiers.tsv | \
kgtk ifnotexists --input-keys label node2 --filter-on  $COVID/corpus-identifier-edges.tsv --filter-keys label node2 \
--mode=NONE > $COVID/corpus-identifiers-not-in-wikidata.tsv_____no_output_____

Format the missing identifiers file by string qouting the identifiers_____no_output_____

df = pd.read_csv('{}/corpus-identifiers-not-in-wikidata.tsv'.format(os.getenv('COVID')), sep='\t', dtype=object)
df['node2'] = df['node2'].map(lambda x: json.dumps(x))
df.to_csv('{}/corpus-identifiers-not-in-wikidata_formatted.tsv'.format(os.getenv('COVID')), sep='\t', index=False)_____no_output_____

The number of missing identifiers in Wikidata(minus the header row): 28112_____no_output_____

!wc -l $COVID/corpus-identifiers-not-in-wikidata_formatted.tsv   28113 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/corpus-identifiers-not-in-wikidata_formatted.tsv

### Process and clean CTD datasets for `Chemicals`, `Diseases` and `Genes`_____no_output_____#### Step 1: Process CTD Diseases file_____no_output_____

rename_columns = {
    '# DiseaseName': 'label',
    'DiseaseID': 'P486',
    'Definition': 'descriptions',
    'Synonyms': 'aliases'

}
drop_columns = ['AltDiseaseIDs', 'ParentIDs', 'TreeNumbers', 'ParentTreeNumbers','SlimMappings']

df_disease = pd.read_csv('{}/CTD_diseases.tsv.gz'.format(os.getenv("COVID")), sep='\t', skiprows=27)
df_disease = df_disease.fillna('').rename(columns=rename_columns).drop(columns=drop_columns)
df_disease['P486'] = df_disease['P486'].map(lambda x: x[5:] if x.startswith('MESH') or x.startswith("OMIM") else x).map(lambda x: json.dumps(x))
df_disease.to_csv('{}/CTD_diseases_clean.tsv'.format(os.getenv("COVID")), sep='\t', index=False)_____no_output_____

Find the disease ids in the `CTD_diseases_clean.tsv` for the missing identifiers in Wikidata in the `corpus-identifiers-not-in-wikidata_formatted.tsv` file.

We do this with the `kgtk ifexists` command._____no_output_____

!cat $COVID/CTD_diseases_clean.tsv | \
kgtk ifexists --input-keys P486 --filter-on  $COVID/corpus-identifiers-not-in-wikidata_formatted.tsv --filter-keys node2 \
--mode=NONE > $COVID/CTD-diseases-corpus-identifiers.tsv_____no_output_____

Add the column `node1`, creating pseudo Wikidata Qnodes with the formula:
Qnode(Disease_ID) = `Q00005550-disease-<Disease_ID>`.

Also add the column `P31`(instance of) with a constant value of `Q12136` (disease)_____no_output_____

df_dc = pd.read_csv('{}/CTD-diseases-corpus-identifiers.tsv'.format(os.getenv("COVID")), sep='\t')
df_dc['node1'] = df_dc['P486'].map(lambda x: 'Q00005550-disease-{}'.format(x.replace('"', '')))
df_dc['P31'] = 'Q12136'
df_dc.to_csv('{}/CTD-diseases-corpus-identifiers-node1.tsv'.format(os.getenv("COVID")), sep='\t', index=False)
_____no_output_____

Convert the CTD disease file to a `KGTK Edge` file.
We do this with `mlr reshape` command_____no_output_____

!cat $COVID/CTD-diseases-corpus-identifiers-node1.tsv | \
mlr --itsv --otsv reshape -i label,P486,descriptions,aliases,P31 -o label,node2 \
> $COVID/CTD-diseases-corpus-identifiers-compact.tsv_____no_output_____

`aliases` in the above file contains multiple values separated by `|`
We can expand the values into multiple rows with `kgtk expand` command_____no_output_____

!cat $COVID/CTD-diseases-corpus-identifiers-compact.tsv | kgtk expand --columns node1 label --mode NONE \
> $COVID/CTD-diseases-corpus-edges.tsv_____no_output_____!head $COVID/CTD-diseases-corpus-edges.tsv | column -tnode1                      label         node2
Q00005550-disease-C537806  label         18-Hydroxylase  deficiency
Q00005550-disease-C537806  P486          """C537806"""
Q00005550-disease-C537806  descriptions
Q00005550-disease-C537806  aliases       18-alpha        hydroxylase  deficiency
Q00005550-disease-C537806  aliases       18-HYDROXYLASE  DEFICIENCY
Q00005550-disease-C537806  aliases       18-Oxidase      Deficiency
Q00005550-disease-C537806  aliases       Aldosterone     deficiency   1
Q00005550-disease-C537806  aliases       Aldosterone     deficiency   due         to  defect  in  18-hydroxylase
Q00005550-disease-C537806  aliases       ALDOSTERONE     DEFICIENCY   DUE         TO  DEFECT  IN  STEROID         18-HYDROXYLASE

#### Step 2: Process CTD Chemicals file_____no_output_____

rename_columns = {
    '# ChemicalName': 'label',
    'ChemicalID': 'P486',
    'Definition': 'descriptions',
    'Synonyms': 'aliases',
    'CasRN': 'P231' # CAS Rn number

}
drop_columns = ['ParentIDs', 'TreeNumbers', 'ParentTreeNumbers','DrugBankIDs']

df_chemical = pd.read_csv('{}/CTD_chemicals.tsv.gz'.format(os.getenv("COVID")), sep='\t', skiprows=27)
df_chemical = df_chemical.fillna('').rename(columns=rename_columns).drop(columns=drop_columns)
df_chemical['P486'] = df_chemical['P486'].map(lambda x: x[5:] if x.startswith('MESH') or x.startswith("OMIM") else x).map(lambda x: json.dumps(x))
df_chemical['P231'] = df_chemical['P231'].map(lambda x: x[5:] if x.startswith('MESH') or x.startswith("OMIM") else x).map(lambda x: json.dumps(x) if x else x)
df_chemical.to_csv('{}/CTD_chemicals_clean.tsv'.format(os.getenv("COVID")), sep='\t', index=False)_____no_output_____

Find the chemical ids in the `CTD_chemicals_clean.tsv` for the missing identifiers in Wikidata in the `corpus-identifiers-not-in-wikidata_formatted.tsv` file.

We do this with the `kgtk ifexists` command._____no_output_____

!cat $COVID/CTD_chemicals_clean.tsv | \
kgtk ifexists --input-keys P486 --filter-on  $COVID/corpus-identifiers-not-in-wikidata_formatted.tsv --filter-keys node2 \
--mode=NONE > $COVID/CTD-chemicals-corpus-identifiers.tsv_____no_output_____

Add the column `node1`, creating pseudo Wikidata Qnodes with the formula:
Qnode(Chemical_ID) = `Q00005550-chemical-<Chemical_ID>`.

Also add the column `P31`(instance of) with a constant value of `Q11344` (chemical element)_____no_output_____

df_cc = pd.read_csv('{}/CTD-chemicals-corpus-identifiers.tsv'.format(os.getenv("COVID")), sep='\t')
df_cc['node1'] = df_cc['P486'].map(lambda x: 'Q00005550-chemical-{}'.format(x.replace('"', '')))
df_cc['P31'] = 'Q11344'
df_cc.to_csv('{}/CTD-chemicals-corpus-identifiers-node1.tsv'.format(os.getenv("COVID")), sep='\t', index=False)
_____no_output_____

Convert the CTD chemical file to a `KGTK Edge` file.
We do this with `mlr reshape` command_____no_output_____

!cat $COVID/CTD-chemicals-corpus-identifiers-node1.tsv | \
mlr --itsv --otsv reshape -i label,P486,descriptions,aliases,P31,P231 -o label,node2 \
> $COVID/CTD-chemicals-corpus-identifiers-compact.tsv_____no_output_____

`aliases` in the above file contains multiple values separated by `|`
We can expand the values into multiple rows with `kgtk expand` command_____no_output_____

!cat $COVID/CTD-chemicals-corpus-identifiers-compact.tsv | kgtk expand --columns node1 label --mode NONE \
> $COVID/CTD-chemicals-corpus-edges.tsv_____no_output_____!head $COVID/CTD-chemicals-corpus-edges.tsv | column -tnode1                       label         node2
Q00005550-chemical-C493119  label         07H239-A
Q00005550-chemical-C493119  P486          """C493119"""
Q00005550-chemical-C493119  descriptions
Q00005550-chemical-C493119  aliases
Q00005550-chemical-C493119  P31           Q11344
Q00005550-chemical-C493119  P231
Q00005550-chemical-C534883  label         10074-G5
Q00005550-chemical-C534883  P486          """C534883"""
Q00005550-chemical-C534883  descriptions

#### Step 3: Process CTD Genes file_____no_output_____

rename_columns = {
    'GeneName': 'descriptions',
    'GeneID': 'P351',
    '# GeneSymbol': 'label',
    'Synonyms': 'aliases',
    'PharmGKBIDs': 'P7001',
    'UniProtIDs': 'P352'

}
drop_columns = ['AltGeneIDs', 'BioGRIDIDs']

df_gene = pd.read_csv('{}/CTD_genes.tsv.gz'.format(os.getenv("COVID")), sep='\t', skiprows=27, dtype=object)
df_gene = df_gene.fillna('').rename(columns=rename_columns).drop(columns=drop_columns)
df_gene['P351'] = df_gene['P351'].map(lambda x: str(x)).map(lambda x: x[5:] if x.startswith('MESH') or x.startswith("OMIM") else x).map(lambda x: json.dumps(x))
df_gene['P7001'] = df_gene['P7001'].map(lambda x: json.dumps(x) if x else x)
df_gene['P352'] = df_gene['P352'].map(lambda x: json.dumps(x) if x else x)
df_gene.to_csv('{}/CTD_genes_clean.tsv'.format(os.getenv("COVID")), sep='\t', index=False)_____no_output_____

Find the gene ids in the `CTD_genes_clean.tsv` for the missing identifiers in Wikidata in the `corpus-identifiers-not-in-wikidata_formatted.tsv` file.

We do this with the `kgtk ifexists` command._____no_output_____

!cat $COVID/CTD_genes_clean.tsv | \
kgtk ifexists --input-keys P351 --filter-on  $COVID/corpus-identifiers-not-in-wikidata_formatted.tsv --filter-keys node2 \
--mode=NONE > $COVID/CTD-genes-corpus-identifiers.tsv_____no_output_____

Add the column `node1`, creating pseudo Wikidata Qnodes with the formula:
Qnode(Gene_ID) = `Q00005550-gene-<Gene_ID>`.

Also add the column `P31`(instance of) with a constant value of `Q7187` (gene)_____no_output_____

df_gc = pd.read_csv('{}/CTD-genes-corpus-identifiers.tsv'.format(os.getenv("COVID")), sep='\t')
df_gc['node1'] = df_gc['P351'].map(lambda x: 'Q00005550-gene-{}'.format(x.replace('"', '')))
df_gc['P31'] = 'Q7187'
df_gc.to_csv('{}/CTD-genes-corpus-identifiers-node1.tsv'.format(os.getenv("COVID")), sep='\t', index=False)
_____no_output_____

Convert the CTD gene file to a `KGTK Edge` file.
We do this with `mlr reshape` command_____no_output_____

!cat $COVID/CTD-genes-corpus-identifiers-node1.tsv | \
mlr --itsv --otsv reshape -i label,P351,descriptions,aliases,P31,P7001,P352 -o label,node2 \
> $COVID/CTD-genes-corpus-identifiers-compact.tsv_____no_output_____

`aliases` in the above file contains multiple values separated by `|`
We can expand the values into multiple rows with `kgtk expand` command_____no_output_____

!cat $COVID/CTD-genes-corpus-identifiers-compact.tsv | kgtk expand --columns node1 label --mode NONE \
> $COVID/CTD-genes-corpus-edges.tsv_____no_output_____!head $COVID/CTD-genes-corpus-edges.tsv | column -tnode1                  label         node2
Q00005550-gene-27521   label         272I21T
Q00005550-gene-27521   P351          """27521"""
Q00005550-gene-27521   descriptions  DNA           segment,  272I21T
Q00005550-gene-27521   aliases
Q00005550-gene-27521   P31           Q7187
Q00005550-gene-27521   P7001
Q00005550-gene-27521   P352
Q00005550-gene-414970  label         2959A1
Q00005550-gene-414970  P351          """414970"""

## Create Mention Edges_____no_output_____

from scripts.create_mention_edges import CreateMentionEdges

# The class CreateMentionEdges has the following input parameters in order,
#     1. The path where the folders `pmid_abs` and `pmcid` are (from Heng)
#     2. corpus-identifier-edges.tsv - file which has Qnodes for identifiers in Wikidata
#     3. CTD-genes-corpus-edges.tsv - file with Qnodes for genes created by us
#     4. CTD-diseases-corpus-edges.tsv - file with Qnodes for diseases created by us
#     5. CTD-chemicals-corpus-edges.tsv - file with Qnodes for chemicals created by us

cme = CreateMentionEdges(os.getenv("COVID"),
                        '{}/corpus-identifier-edges.tsv'.format(os.getenv("COVID")),
                        '{}/CTD-genes-corpus-edges.tsv'.format(os.getenv("COVID")),
                        '{}/CTD-diseases-corpus-edges.tsv'.format(os.getenv("COVID")),
                        '{}/CTD-chemicals-corpus-edges.tsv'.format(os.getenv("COVID")))
cme.create_mention_edges(os.getenv("COVID"))
Total number of papers in the corpus:  20446
Done!

The code above will create 3 files at output path $COVID,

* covid_kgtk_blender_mentions.tsv: File with mention edges
* covid_kgtk_blender_mentions_qualifiers.tsv: qualifiers for the mention edges
* scholarly_articles_not_in_wikidata.tsv: papers edges files which were not present in Wikidata
    _____no_output_____

!head $COVID/covid_kgtk_blender_mentions.tsv | column -t node1             label     node2             id
Q77092138         P2020003  Q166231           Q77092138-P2020003-0
Q77092138         P2020007  Q20747334         Q77092138-P2020007-1
Q77092138         P2020007  Q20747334         Q77092138-P2020007-2
Q77092138         P2020007  Q20747334         Q77092138-P2020007-3
Q77092138         P2020003  Q767485           Q77092138-P2020003-4
Q77092138         P2020001  Q77092138-text-0  Q77092138-P2020001-0
Q77092138-text-0  P2020012  Immunology        and                     prevention  of  infection  in  feedlot  cattle.  Q77092138-text-0-label-0
Q77092138-text-0  P31       Q1385610          Q77092138-text-0-P31-0
Q77092138         P2020001  Q77092138-text-1  Q77092138-P2020001-1
!head $COVID/covid_kgtk_blender_mentions_qualifiers.tsv | column -t node1                 label     node2                            id
Q77092138-P2020003-0  P4153     29                               Q77092138-P2020003-0-0
Q77092138-P2020003-0  P2043     9                                Q77092138-P2020003-0-1
Q77092138-P2020003-0  P1932     infection                        Q77092138-P2020003-0-2
Q77092138-P2020003-0  P2020008  http://blender.cs.illinois.edu/  Q77092138-P2020003-0-3
Q77092138-P2020003-0  P2020001  Q77092138-text-0                 Q77092138-P2020003-0-4
Q77092138-P2020007-1  P4153     50                               Q77092138-P2020007-1-0
Q77092138-P2020007-1  P2043     6                                Q77092138-P2020007-1-1
Q77092138-P2020007-1  P1932     cattle                           Q77092138-P2020007-1-2
Q77092138-P2020007-1  P2020008  http://blender.cs.illinois.edu/  Q77092138-P2020007-1-3
!head $COVID/scholarly_articles_not_in_wikidata.tsv | column -tid                         node1              label  node2
Q000077708998245-P31-0     Q000077708998245   P31    Q13442814
Q000077708998245-label-1   Q000077708998245   label  Immunity   to  infection.
Q000077708998245-P577-2    Q000077708998245   P577   1996
Q000077708998245-P1476-3   Q000077708998245   P1476  Immunity   to  infection.
Q000077708998245-P698-4    Q000077708998245   P698   8998245
Q0000777022951009-P31-0    Q0000777022951009  P31    Q13442814
Q0000777022951009-label-1  Q0000777022951009  label  Stock      or  stroke?     Stock  market  movement  and  stroke  incidence  in  Taiwan.
Q0000777022951009-P577-2   Q0000777022951009  P577   2012
Q0000777022951009-P1476-3  Q0000777022951009  P1476  Stock      or  stroke?     Stock  market  movement  and  stroke  incidence  in  Taiwan.

## Generate RDF Triples_____no_output_____`kgtk` comes packaged with the command `generate_wikidata_triples` which can generate RDF triples from a KGTK edge file.
_____no_output_____### Step 1: concatenate the edge files we created in this notebook_____no_output_____we do this with `kgtk cat` command_____no_output_____

!kgtk cat $COVID/corpus-edges.tsv \
          $COVID/corpus-citations-and-authors.tsv \
          $COVID/CTD-diseases-corpus-edges.tsv \
          $COVID/CTD-chemicals-corpus-edges.tsv \
          $COVID/CTD-genes-corpus-edges.tsv \
          $COVID/covid_kgtk_blender_mentions.tsv \
          $COVID/covid_kgtk_blender_mentions_qualifiers.tsv \
          $COVID/scholarly_articles_not_in_wikidata.tsv > $COVID/corpus-all.tsv_____no_output_____!wc -l $COVID/corpus-edges.tsv
!wc -l $COVID/corpus-citations-and-authors.tsv
!wc -l $COVID/CTD-diseases-corpus-edges.tsv
!wc -l $COVID/CTD-chemicals-corpus-edges.tsv
!wc -l $COVID/CTD-genes-corpus-edges.tsv
!wc -l $COVID/covid_kgtk_blender_mentions.tsv
!wc -l $COVID/covid_kgtk_blender_mentions_qualifiers.tsv
!wc -l $COVID/scholarly_articles_not_in_wikidata.tsv
!wc -l $COVID/corpus-all.tsv 2297783 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/corpus-edges.tsv
 11389393 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/corpus-citations-and-authors.tsv
    5305 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/CTD-diseases-corpus-edges.tsv
  112456 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/CTD-chemicals-corpus-edges.tsv
   41616 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/CTD-genes-corpus-edges.tsv
 8142747 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/covid_kgtk_blender_mentions.tsv
 25868901 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/covid_kgtk_blender_mentions_qualifiers.tsv
     456 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/scholarly_articles_not_in_wikidata.tsv
 47858650 /Users/amandeep/Github/CKG-Covid/datasets/sandbox/corpus-all.tsv

### Step 2: generate rdf triples_____no_output_____we do this with `kgtk generate_wikidata_triples` command_____no_output_____

!cat $COVID/corpus-all.tsv | kgtk generate_wikidata_triples \
                                -ap aliases \
                                -lp label \
                                -dp descriptions \
                                -pf properties.tsv \
                                -n 1000 \
                                -ig no  \
                                --debug \
                                -gt yes > $COVID/corpus-all.ttl_____no_output_____

## Incorporate Analytic Outputs_____no_output_____