# Notebook from ingolia/mcb200-2020
Path: 0911_estimation/06_allelic_expression.ipynb

# Allele-Specific Expression

RNA sequencing can distinguish transcripts expressed from different copies of genes on homologous chromosomes when single-nucleotide polymorphisms (perhaps silent) distinguish the two alleles. Linkage between these distinctive SNPs and _cis_-regulatory sequences can provide information on regulatory variation within a shared cellular context._____no_output_____## Null Hypothesis Testing

The null hypothesis in allele-specific expression analysis is that the alleles are expressed equally and so each read is equally likely to be derived from each allele.

Here, we'll take two approaches to get a *p* value for the null hypothesis of equal expression in situations where just 25% of the reads come from one allele and 75% from the other. We'll look at this with just 8 reads, with 32 reads, and then with 100 reads._____no_output_____### Permutation Testing

First, we'll generate many random sets of data according to the null model and look at the distribution of allele skew in these random data. Our approach to generating random sets of reads is simple: we choose randomly between `0` and `1`, and then count how often we choose `1` by summing the results of this random choice.

The `numpy.random` submodule of the `numpy` module has functions to generate (pseudo-)random numbers. To use this, we'll `import` it and use it like this:

```
import numpy as np
print(np.random.choice(['A', 'C', 'G', 'T']))
print(np.random.choice(['A', 'C', 'G', 'T'], 3))
print(np.random.choice([0, 1], 8))
```_____no_output_____The quick trick to *count* how many of our random 0-or-1 choices came up 1 is to sum the vector using the `sum()` function, like this:

```
sum(np.random.choice([0, 1], 8))
```_____no_output_____Now, we'll generate 10,000 random samples of 8 reads each, and tabulate how many random samples have zero, one, ..., eight reads from that sample.

First we'll create an empty counting array with 9 entries for 0 through 8 inclusive
```
allele_counts_8 = [0] * 9
```
Then we'll run a for loop 10,000 times. We don't actually use the loop varible inside the `for` loop, so by convention we'll just call it `_`.
```
for _ in range(0, 10000):
    ...
```_____no_output_____Next, we'll plot the distribution by importing the plotting module. We can directly plot our list, we don't need to turn it into a Pandas `Series` first.
```
import matplotlib.pyplot as plt
plt.plot(allele_counts_8)
```_____no_output_____We can use this to ask: in what fraction of random samples do you see 12.5% (one-eighth) or fewer `1` reads? What can you conclude from seeing this kind of skew in a sample of 8 reads?

To do this, we can add up
```
allele_counts_8[0] + allele_counts_8[1]
```
or, to make it easier to read
```
sum(allele_counts_8[0:2])
```
and then divide by the total number of random trials we generated (which should be 10,000)
```
sum(allele_counts_8)
```_____no_output_____What if we wanted to consider a strong skew in *either* direction? That is, we're interested in situations where we had ≤12.5% or ≥87.5% frequence of `1` alleles?

We would need to consider 7 or 8 reads in addition to 0 or 1
```
sum(allele_counts_8[7:9])
```_____no_output_____We can repeat the same analysis, for 24 reads per random sample.

* Make an empty, 25-entry count array
* Loop over 10,000 random trials
    * Generate a random choice of 24 x 0 or 1
    * Sum them up
    * Add the count to the overall tally_____no_output_____Now we can test for a similar skew in our 24-read samples -- what are the odds of seeing 3 or fewer reads from one allele and all the rest from the other?_____no_output_____What about a stronger skew: just 0 or 1 reads from one allele and all the rest from the other?_____no_output_____## Random Variables

There is probably a small but non-zero odds of getting this strong skew in the 24-read sample -- but we would need to generate a lot of random samples in order to figure out exactly how small. Instead, we can treat the number of reads from the `1` allele as a random variable with a binomial distribution. This isn't always a fair description of biological data, but it's a reasonable starting point here.

### Binomial Distribution

The scipy package contains a statistics module with a sub-module specific for the binomial distribution. We can get the probability
```
P( k successes out of N trials with probability p of success per trial )
```
using
```
binom.pmf(k, N, p)
```
"pmf" here stands for "probability mass function".

For instance, we can ask about exactly 2 "successes" out of 8 "trials" -- think of this as 2 reads from the `1` allele out of 8 reads counted in total. We can also ask abotu exactly 2 reads from the `1` allele out of 32 total.

To start, install the `scipy` package (you should need to do this just once, ever) and import the `binom` sub-module of the `scipy.stats` (in every notebook where you want to use it)

```
import sys
!{sys.executable} -m pip install scipy
```
Then, you can use the `binom.pmf()` function like
```
binom.pmf(2, 8, 0.5)
```_____no_output_____We can also test a range of different k values:
```
print(binom.pmf(range(0,4), 24, 0.5))
```
This makes it easy to sum up across many possibilities:
```
print(sum(binom.pmf(range(0,4), 24, 0.5)))
```
_____no_output_____We can use this to compute a precise value for the very small probability of the 1-or-fewer skew in 24 reads._____no_output_____We'd need to run a lot of simulations to find that p-value reliably!

We can also plot this distribution and compare it to our simulations.

Either we need to convert our probabilities into counts per 10,000 trials by multiplying them
```
plt.plot(binom.pmf(range(0,24), 24, 0.5) * 10000)
plt.plot(allele_counts_24)
```

_____no_output_____Or, we need to convert our random trial counts into probabilities by converting them into a Pandas series and dividing the counts by the total number of trials
```
import pandas as pd
plt.plot(binom.pmf(range(0,24), 24, 0.5))
plt.plot(pd.Series(allele_counts_24) / sum(allele_counts_24))
```_____no_output_____The two lines are so close it's hard to tell them  apart, and we don't know which color corresponds to which data set. We can use some codes to specify the color and type of the lines. The version below uses a solid red line for the binomial value and a dashed blue line with points for the simulations
```
plt.plot(binom.pmf(range(0,24), 24, 0.5) * 10000, '.-r')
plt.plot(allele_counts_24, 'o--b')
```_____no_output_____## Maximum Likelihood Estimation

Instead of simply testing whether allele expression is even, we want to _estimate_ the relative skew in expression. To do this, we will start by making a graph where we consider all possible bias values, and then figure out the likelihood function P( 4 reads out of 32 | bias ). We'll use the `binom.pmf` function again, but now we'll consider many different values for the 3rd _p_ parameter instead of the first one.

This graph looks similar to others that we've made, but the axes are different -- we'll add x and y axis labels to emphasize this difference.

First make an array of possible _p_ values of 0, 0.01, 0.02, ..., 1.00
```
skew = np.arange(0,1,0.01)
```
_____no_output_____Then, compute the *likelihood*, that is, the probability of generating the observed data (4 reads out of 32) from the mode (our specified value of *p*). Conveniently, we can run these over the whole array of values at once.
```
binom.pmf(4, 32, skew)
```_____no_output_____Now, we can plot these likelihoods, and then use the `plt.xlabel()` and `plt.ylabel()` functions to add axis labels.
```
plt.plot(skew, binom.pmf(4, 32, skew))
plt.xlabel('Allele skew')
plt.ylabel('P(4 reads / 32 total)')
```_____no_output_____As we discussed before, we often want to work with log likelihood functions. In fact, `binom` has the built-in ability to give us a log probability that will be numerically stable when the actual likelihood is a very tiny number, using the `logpmf` method.
```
plt.plot(skew, binom.logpmf(4, 32, skew))
```_____no_output_____In this plot, the likelihood function looks pretty flat around 0.25, but we might want to adjust the y-axis to focus on the region of high likelihood -- the default puts a lot of emphasis on parts of the plot where the likelihood is very small. The `plt.axis(x_min, x_max, y_min, y_max)` function does this:
```
plt.axis([0.0, 1.0, -10, 0])
```_____no_output_____We need to find the point on the x-axis where the likelihood is maximized. We can probably guess that this will happen at 0.125, but we can use algorithms from Scipy to find the best likelihood. These methods are typically expressed in terms of _minimization_, and so we'll minimize the negative log likelihood which is equivalent to finding the maximum of the likelihood.

To do this, we define a function to compute the negative log likelihood, called `negloglik`, and then use `minimize_scalar` from `scipy.optimize` to find the allele skew value that maximizes the likelihood of our data.

This will import the minimization sub-module
```
from scipy.optimize import minimize_scalar
```
The negative log likelihood function is quite simple:
```
def negloglik(skew):
    return -binom.logpmf(4, 32, skew)
```
And we can then use `minimize_scalar` to find the best skew estimate
```
best = minimize_scalar(negloglik, bounds=(0,1), method='bounded')
```_____no_output_____To see how the total number of reads _N_ affects the sharpness of our maximum likelihood estimates, we'll plot the log likelihood functions for:
* 1 read out of 8
* 4 reads out of 32
* 16 reads out of 128_____no_output_____


# Notebook from nabinkhadka/spark-nlp-workshop
Path: tutorials/streamlit_notebooks/SENTIMENT_EN.ipynb

![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/streamlit_notebooks/SENTIMENT_EN.ipynb)

_____no_output_____# **Find sentiment in text**_____no_output_____## 0. Colab Setup_____no_output_____

%cd /content
!sudo apt-get install openjdk-8-jdk
!java -version
!pip install --ignore-installed -q pyspark==2.4.4
!pip install spark-nlp_____no_output_____import pandas as pd
import numpy as np
import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
import json
from pyspark.ml import Pipeline
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
from sparknlp.annotator import *
from sparknlp.base import *
import sparknlp
from sparknlp.pretrained import PretrainedPipeline_____no_output_____

## 1. Start Spark Session_____no_output_____

spark = sparknlp.start()_____no_output_____

## 2. Select the DL model and re-run cells below_____no_output_____

#MODEL_NAME='sentimentdl_use_imdb'
MODEL_NAME='sentimentdl_use_twitter'_____no_output_____

## 3. Some sample examples_____no_output_____

## Generating Example Files ##

text_list = []
if MODEL_NAME=='sentimentdl_use_imdb':
  text_list = [
             """Demonicus is a movie turned into a video game! I just love the story and the things that goes on in the film.It is a B-film ofcourse but that doesn`t bother one bit because its made just right and the music was rad! Horror and sword fight freaks,buy this movie now!""",
             """Back when Alec Baldwin and Kim Basinger were a mercurial, hot-tempered, high-powered Hollywood couple they filmed this (nearly) scene-for-scene remake of the 1972 Steve McQueen-Ali MacGraw action-thriller about a fugitive twosome. It almost worked the first time because McQueen was such a vital presence on the screen--even stone silent and weary, you could sense his clock ticking, his cagey magnetism. Baldwin is not in Steve McQueen's league, but he has his charms and is probably a more versatile actor--if so, this is not a showcase for his attributes. Basinger does well and certainly looks good, but James Woods is artificially hammy in a silly mob-magnet role. A sub-plot involving another couple taken hostage by Baldwin's ex-partner was unbearable in the '72 film and plays even worse here. As for the action scenes, they're pretty old hat, which causes one to wonder: why even remake the original?""",
             """Despite a tight narrative, Johnnie To's Election feels at times like it was once a longer picture, with many characters and plot strands abandoned or ultimately unresolved. Some of these are dealt with in the truly excellent and far superior sequel, Election 2: Harmony is a Virtue, but it's still a dependably enthralling thriller about a contested Triad election that bypasses the usual shootouts and explosions (though not the violence) in favour of constantly shifting alliances that can turn in the time it takes to make a phone call. It's also a film where the most ruthless character isn't always the most threatening one, as the chilling ending makes only too clear: one can imagine a lifetime of psychological counselling being necessary for all the trauma that one inflicts on one unfortunate bystander. Simon Yam, all too often a variable actor but always at his best under To's direction, has possibly never been better in the lead, not least because Tony Leung's much more extrovert performance makes his stillness more the powerful.""",
             """This movie has successfully proved what we all already know, that professional basket-ball players suck at everything besides playing basket-ball. Especially rapping and acting. I can not even begin to describe how bad this movie truly is. First of all, is it just me, or is that the ugliest kid you have ever seen? I mean, his teeth could be used as a can-opener. Secondly, why would a genie want to pursue a career in the music industry when, even though he has magical powers, he sucks horribly at making music? Third, I have read the Bible. In no way shape or form did it say that Jesus made genies. Fourth, what was the deal with all the crappy special effects? I assure you that any acne-addled nerdy teenager with a computer could make better effects than that. Fifth, why did the ending suck so badly? And what the hell is a djin? And finally, whoever created the nightmare known as Kazaam needs to be thrown off of a plane and onto the Eiffel Tower, because this movie take the word "suck" to an entirely new level.""",
             """The fluttering of butterfly wings in the Atlantic can unleash a hurricane in the Pacific. According to this theory (somehow related to the Chaos Theory, I'm not sure exactly how), every action, no matter how small or insignificant, will start a chain reaction that can lead to big events. This small jewel of a film shows us a series of seemingly-unrelated characters, most of them in Paris, whose actions will affect each others' lives. (The six-degrees-of-separation theory can be applied as well.) Each story is a facet of the jewel that is this film. The acting is finely-tuned and nuanced (Audrey Tautou is luminous), the stories mesh plausibly, the humor is just right, and the viewer leaves the theatre nodding in agreement.""",
             """There have been very few films I have not been able to sit through. I made it through Battle Field Earth no problem. But this, This is one of the single worst films EVER to be made. I understand Whoopi Goldberg tried to get of acting in it. I do not blame her. I would feel ashamed to have this on a resume. I belive it is a rare occasion when almost every gag in a film falls flat on it's face. Well it happens here. Not to mention the SFX, look for the dino with the control cables hanging out of it rear end!!!!!! Halfway through the film I was still looking for a plot. I never found one. Save yourself the trouble of renting this and save 90 minutes of your life.""",
             """After a long hard week behind the desk making all those dam serious decisions this movie is a great way to relax. Like Wells and the original radio broadcast this movie will take you away to a land of alien humor and sci-fi paraday. 'Captain Zippo died in the great charge of the Buick. He was a brave man.' The Jack Nicholson impressions shine right through that alien face with the dark sun glasses and leather jacket. And always remember to beware of the 'doughnut of death!' Keep in mind the number one rule of this movie - suspension of disbelief - sit back and relax - and 'Prepare to die Earth Scum!' You just have to see it for yourself.""",
             """When Ritchie first burst on to movie scene his films were hailed as funny, witty, well directed and original. If one could compare the hype he had generated with his first two attempts and the almost universal loathing his last two outings have created one should consider - has Ritchie been found out? Is he really that talented? Does he really have any genuine original ideas? Or is he simply a pretentious and egotistical director who really wants to be Fincher, Tarantino and Leone all rolled into one colossal and disorganised heap? After watching Revolver one could be excused for thinking were did it all go wrong? What happened to his great sense of humour? Where did he get all these mixed and convoluted ideas from? Revolver tries to be clever, philosophical and succinct, it tries to be an intelligent psychoanalysis, it tries to be an intricate and complicated thriller. Ritchie does make a gargantuan effort to fulfil all these many objectives and invests great chunks of a script into existential musings and numerous plot twists. However, in the end all it serves is to construct a severely disjointed, unstructured and ultimately unfriendly film to the audience. Its plagiarism is so sinful and blatant that although Ritchie does at least attempt to give his own spin he should be punished for even trying to pass it off as his own work. So what the audience gets ultimately is a terrible screenplay intertwined with many pretentious oneliners and clumsy setpieces.

Revolver is ultimately an unoriginal and bland movie that has stolen countless themes from masterpieces like Fight Club, Usual Suspects and Pulp Fiction. It aims high, but inevitably shots blanks aplenty.

Revolver deserves to be lambasted, it is a truly poor film masquerading as a wannabe masterpiece from a wannabe auteur. However, it falls flat on its farcical face and just fails at everything it wants to be and achieve.""",
             """I always thought this would be a long and boring Talking-Heads flick full of static interior takes, dude, I was wrong. "Election" is a highly fascinating and thoroughly captivating thriller-drama, taking a deep and realistic view behind the origins of Triads-Rituals. Characters are constantly on the move, and although as a viewer you kinda always remain an outsider, it\'s still possible to feel the suspense coming from certain decisions and ambitions of the characters. Furthermore Johnnie To succeeds in creating some truly opulent images due to meticulously composed lighting and atmospheric light-shadow contrasts. Although there\'s hardly any action, the ending is still shocking in it\'s ruthless depicting of brutality. Cool movie that deserves more attention, and I came to like the minimalistic acoustic guitar score quite a bit.""",
             """This is to the Zatoichi movies as the "Star Trek" movies were to "Star Trek"--except that in this case every one of the originals was more entertaining and interesting than this big, shiny re-do, and also better made, if substance is more important than surface. Had I never seen them, I would have thought this good-looking but empty; since I had, I thought its style inappropriate and its content insufficient. The idea of reviving the character in a bigger, slicker production must have sounded good, but there was no point in it, other than the hope of making money; it\'s just a show, which mostly fails to capture the atmosphere of the character\'s world and wholly fails to take the character anywhere he hasn\'t been already (also, the actor wasn\'t at his best). I\'d been hoping to see Ichi at a late stage of life, in a story that would see him out gracefully and draw some conclusion from his experience overall; this just rehashes bits and pieces from the other movies, seasoned with more sex and sfx violence. Not the same experience at all."""
             ]
elif  MODEL_NAME=='sentimentdl_use_twitter':
  text_list = [
            """@Mbjthegreat i really dont want AT&T phone service..they suck when it comes to having a signal""",
            """holy crap. I take a nap for 4 hours and Pitchfork blows up my twitter dashboard. I wish I was at Coachella.""",
            """@Susy412 he is working today  ive tried that still not working..... hmmmm!! im rubbish with computers haha!""",
            """Brand New Canon EOS 50D 15MP DSLR Camera Canon 17-85mm IS Lens ...: Web Technology Thread, Brand New Canon EOS 5.. http://u.mavrev.com/5a3t""",
            """Watching a programme about the life of Hitler, its only enhancing my geekiness of history.""",
            """GM says expects announcment on sale of Hummer soon - Reuters: WDSUGM says expects announcment on sale of Hummer .. http://bit.ly/4E1Fv""",
            """@accannis @edog1203 Great Stanford course. Thanks for making it available to the public! Really helpful and informative for starting off!""",
            """@the_real_usher LeBron is cool.  I like his personality...he has good character.""",
            """@sketchbug Lebron is a hometown hero to me, lol I love the Lakers but let's go Cavs, lol""",
            """@PDubyaD right!!! LOL we'll get there!! I have high expectations, Warren Buffet style.""",
            ]
_____no_output_____

## 4. Define Spark NLP pipleline_____no_output_____

documentAssembler = DocumentAssembler()\
    .setInputCol("text")\
    .setOutputCol("document")

use = UniversalSentenceEncoder.pretrained(name="tfhub_use", lang="en")\
 .setInputCols(["document"])\
 .setOutputCol("sentence_embeddings")

sentimentdl = SentimentDLModel.pretrained(name=MODEL_NAME, lang="en")\
    .setInputCols(["sentence_embeddings"])\
    .setOutputCol("sentiment")

nlpPipeline = Pipeline(
      stages = [
          documentAssembler,
          use,
          sentimentdl
      ])
_____no_output_____

## 5. Run the pipeline_____no_output_____

empty_df = spark.createDataFrame([['']]).toDF("text")

pipelineModel = nlpPipeline.fit(empty_df)

df = spark.createDataFrame(pd.DataFrame({"text":text_list}))
result = pipelineModel.transform(df)_____no_output_____

## 6. Visualize results_____no_output_____

result.select(F.explode(F.arrays_zip('document.result', 'sentiment.result')).alias("cols")) \
.select(F.expr("cols['0']").alias("document"),
        F.expr("cols['1']").alias("sentiment")).show(truncate=False)_____no_output_____


# Notebook from taotangtt/sta-663-2018
Path: notebooks/S14F_Data_Sketches.ipynb

# Scalable data storage and structures

When dealing with big data, minimizing the amount of memory used is critical to avoid having to use disk-based access, which can be 100,000 times slower than random access. This notebook deals with ways to minimizee data storage for several common use cases:

- Large arrays of homogenous data (often numbers)
- Large string collections
- Counting distinct values
- Yes/No responses to queries

Methods covered range from the mundane (use `numpy` arrays rather than lists), to classic but less well-known data structures (e.g. prefix trees or tries) to algorithmically ingenious probabilistic data structures (e.g. bloom filter and hyperloglog)._____no_output_____

import sys
import numpy as np_____no_output_____

## Selective retrieval from disk-based storage

We have already seen that there are many ways to retrieve only the parts of the data we need now into memory at this particular moment. Options include

- generators (e.g. to read a file a line at a time)
- `numpy.memmap`
- HDF5 via `h5py`
- Key-value stores (e.g. `redis`)
- SQL and NoSQL databases (e.g. `sqlite3`)_____no_output_____## Storing numbers

Less memory is used when storing numbers in numpy arrays rather than lists._____no_output_____

sys.getsizeof(list(range(int(1e8))))_____no_output_____np.arange(int(1e8)).nbytes_____no_output_____

### Using only the precision needed can also save memory_____no_output_____

np.arange(int(1e8)).astype('float32').nbytes_____no_output_____np.arange(int(1e8)).astype('float64').nbytes_____no_output_____

## Storing strings_____no_output_____

def flatmap(func, items):
    return it.chain.from_iterable(map(func, items))_____no_output_____def flatten(xss):
    return (x for xs in xss for x in xs)_____no_output_____

### Using a list_____no_output_____

with open('data/Ulysses.txt') as f:
    word_list = list(flatten(line.split() for line in f))_____no_output_____sys.getsizeof(word_list)_____no_output_____target = 'WARRANTIES'_____no_output_____%timeit -r1 -n1 word_list.index(target)6.33 ms ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

### Using a sorted list_____no_output_____

word_list.sort()_____no_output_____import bisect
%timeit -r1 -n1 bisect.bisect(word_list, target)8.48 µs ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

### Using a set_____no_output_____

word_set = set(word_list)_____no_output_____sys.getsizeof(word_set)_____no_output_____%timeit -r1 -n1 target in word_set1.2 µs ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

### Using a [trie](https://en.wikipedia.org/wiki/Trie) (prefix tree)_____no_output_____```bash
! pip install hat_trie
```_____no_output_____

%load_ext memory_profiler_____no_output_____from hat_trie import Trie_____no_output_____%memit word_trie = Trie(word_list)peak memory: 70.50 MiB, increment: 0.10 MiB
%timeit -r1 -n1 target in word_trie3.73 µs ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

## Data Sketches_____no_output_____A `sketch` is a probabilistic algorithm or data structure that approximates some statistic of interest, typically using very little memory and processing time. Often they are applied to streaming data, and so must be able to incrementally process data. Many data sketches make use of hash functions to distribute data into buckets uniformly. Typically, data sketches have the following desirable properties

- sub-linear in space
- single scan
- can be parallelized
- can be combined (merge)

Some statistics that sketches have been used to estimate include

- indicator variables (event detection)
- counts
- quantiles
- moments
- entropy

Packages for data sketches in Python are relatively immmature, and if you are interested, you could make a large contribution by creating a comprehensive open source library of data sketches in Python._____no_output_____### Morris counter

The Morris counter is used as a simple illustration of a probabilistic data structure, with the standard trade-off of using less memory in return for less accuracy. The algorithm is extremely simple - keep a counter $c$ that represents the **exponent** - that is, when the Morris counter is $c$, the estimated count is $2^c$. The probabilistic part comes from the way that the counter is incremented by comparing a uniform random variate to $1/2^c$._____no_output_____

from random import random

class MorrisCounter:
    def __init__(self, c=0):
        self.c = c

    def __len__(self):
        return 2 ** self.c

    def add(self, item):
        self.c += random() < 1/(2**self.c)_____no_output_____mc = MorrisCounter()_____no_output_____print('True\t\tMorris\t\tRel Error')
for i, word in enumerate(word_list):
    mc.add(word)
    if i%int(.2e5)==0:
        print('%8d\t%8d\t%.2f' % (i, len(mc), 0 if i==0 else abs(i - len(mc))/i))True		Morris		Rel Error
       0	       2	0.00
   20000	   32768	0.64
   40000	   32768	0.18
   60000	   32768	0.45
   80000	   65536	0.18
  100000	   65536	0.34
  120000	   65536	0.45
  140000	   65536	0.53
  160000	  131072	0.18
  180000	  131072	0.27
  200000	  131072	0.34
  220000	  131072	0.40
  240000	  131072	0.45
  260000	  131072	0.50

### Increasing accuracy

A simple way to increase the accuracy is to have multiple Morris counters and take the average. These two ideas of using a probabilistic calculation and multiple samples to improve precision are the basis for the more useful probabilisitc data structures described below._____no_output_____

mcs = [MorrisCounter() for i in range(10)]_____no_output_____print('True\t\tMorris\t\tRel Error')
for i, word in enumerate(word_list):
    for j in range(10):
        mcs[j].add(word)
    estimate = np.mean([len(m) for m in mcs])
    if i%int(.2e5)==0:
        print('%8d\t%8d\t%.2f' % (i, estimate, 0 if i==0 else abs(i - estimate)/i))True		Morris		Rel Error
       0	       2	0.00
   20000	   20480	0.02
   40000	   38502	0.04
   60000	   45875	0.24
   80000	   72089	0.10
  100000	  134348	0.34
  120000	  163840	0.37
  140000	  176947	0.26
  160000	  176947	0.11
  180000	  203161	0.13
  200000	  203161	0.02
  220000	  229376	0.04
  240000	  255590	0.06
  260000	  255590	0.02

### Distinct value Sketches

The Morris counter is less useful because the degree of memory saved as compared to counting the number of elements exactly is not much unless the numbers are staggeringly huge. In contrast, counting the number of **distinct** elements exactly requires storage of all distinct elements (e.g. in a set) and hence grows with the cardinality $n$. Probabilistic data structures known as Distinct Value Sketches can do this with a tiny and fixed memory size.

Examples where counting distinct values is useful:

- number of unique users in a Twitter stream
- number of distinct records to be fetched by a databse query
- number of unique IP addresses accessing a website
- number of distinct queries submitted to a search engine
- number of distinct DNA motifs in genomics data sets (e.g. microbiome)_____no_output_____### [Hash functions](https://en.wikipedia.org/wiki/Hash_function)

A hash function takes data of arbitrary size and converts it into a number in a fixed range. Ideally, given an arbitrary set of data items, the hash function generates numbers that follow a uniform distribution within the fixed range. Hash functions are immensely useful throughout computer science (for example - they power Python sets and dictionaries), and especially for the generation of probabilistic data structures._____no_output_____#### A simple hash function mapping

Note the **collisions**. If not handled, there is  a loss of information. Commonly, practical hash functions return a 32 or 64 bit integer. Also note that there are an arbitrary number of hash functions that can return numbers within a given range.

Note also that because the hash function is deterministic, the same item will always map to the same bin. _____no_output_____

def string_hash(word, n):
    return sum(ord(char) for char in word) % n_____no_output_____sentence = "The quick brown fox jumps over the lazy dog."
for word in sentence.split():
    print(word, string_hash(word, 10))The 9
quick 1
brown 2
fox 3
jumps 9
over 4
the 1
lazy 8
dog. 0

#### Built-in Python hash function_____no_output_____

help(hash)Help on built-in function hash in module builtins:

hash(obj, /)
    Return the hash value for the given object.

    Two objects that compare equal must also have the same hash value, but the
    reverse is not necessarily true.

for word in sentence.split():
    print('{:<10s} {:24}'.format(word, hash(word)))The            -4859935776507312418
quick           9157615745031482514
brown           4123312298496538273
fox            -2015214628178477320
jumps            -71379956079029581
over           -6974446915587241323
the            -5638214675285202096
lazy            1423964815621844201
dog.           -1983643758301440122

#### Using a hash function from the MurmurHash3 library

Note that the hash function accepts a seed, allowing the creation of multiple hash functions. We also display the hash result as a 32-bit binary string._____no_output_____

import mmh3

for word in sentence.split():
    print('{:<10} {:+032b} {:+032b}'.format(word.ljust(10), mmh3.hash(word, seed=1234),
          mmh3.hash(word, seed=4321)))The        +0001000011111110001001110101100 +1110110100100101010111100011010
quick      -0101111111011110110101100101000 +1000100001101010110000101101100
brown      +1000101010000110110010001110101 -1101101110000000010001100010100
fox        -1000000010010010000111001111011 +0111011111000011001001001110111
jumps      +0000010111000011010000100101010 +0010010001111110100010010110011
over       -0110101101111001001101011111011 -1101110111110010000101101000100
the        -1000000101110000000110011111001 +0001000111100111011000011100101
lazy       -1101011000111111110011111001100 +0010101110101100001000101110000
dog.       +0100110101101111101011110111111 -0101111000110000001011110001011

#### LogLog family

The binary digits in a (say) 32-bit hash are effectively random, and equivalent to a sequence of fair coin tosses. Hence the probability that we see a run of 5 zeros in the smallest hash so far suggests that we have added $2^5$ unique items so far. This is the intuition behind the loglog family of Distinct Value Sketches. Note that the biggest count we can track with 32 bits is $2^{32} = 4294967296$.

The accuracy of the sketch can be improved by averaging results with multiple coin flippers. In practice, this is done by using the first $k$ bit registers to identify $2^k$ different coin flippers. Hence, the max count is now $2 ** (32 - k)$. The hyperloglog algorithm uses the harmonic mean of the $2^k$ flippers which reduces the effect of outliers and hence the variance of the estimate._____no_output_____

for i in range(1, 15):
    k = 2**i
    hashes = [''.join(map(str, np.random.randint(0,2,32))) for i in range(k)]
    print('%6d\t%s' % (k, min(hashes)))     2	01001110101100101111011010111111
     4	10000010000111011000111110010010
     8	01001001110010100010101011000100
    16	00011011100001111110100010110011
    32	00000001000100111110110100100110
    64	00000011101010100011001100010101
   128	00000011001000100100001110011001
   256	00000000011011001011111011011001
   512	00000000101100110010111101011100
  1024	00000000001110100101101000011111
  2048	00000000000100001010110101000100
  4096	00000000000010100001011100111011
  8192	00000000000000001011101100000101
 16384	00000000000000100011011110111100

```bash
pip install hyperloglog
```_____no_output_____

from hyperloglog import HyperLogLog_____no_output_____hll = HyperLogLog(0.01) # accept 1% counting error_____no_output_____print('True\t\tHLL\t\tRel Error')
s = set([])
for i, word in enumerate(word_list):
    s.add(word)
    hll.add(word)
    if i%int(.2e5)==0:
        print('%8d\t%8d\t\t%.2f' % (len(s), len(hll), 0 if i==0 else abs(len(s) - len(hll))/i))True		HLL		Rel Error
       1	       1		0.00
    6585	    6560		0.00
   11862	   11777		0.00
   15390	   15318		0.00
   18358	   18236		0.00
   24705	   24712		0.00
   28693	   28750		0.00
   30791	   30946		0.00
   34530	   34677		0.00
   36002	   36077		0.00
   41720	   42091		0.00
   45842	   46384		0.00
   46389	   46979		0.00
   49524	   50226		0.00

### Bloom filters

Bloom filters are designed to answer queries about whether a specific item is in a collection. If the answer is NO, then it is definitive. However, if the answer is yes, it might be a false positive. The possibility of a false positive makes the Bloom filter a probabilistic data structure.

A bloom filter consists of a bit vector of length $k$ initially set to zero, and $n$ different hash functions that return a hash value that will fall into one of the $k$ bins. In the construction phase, for every item in the collection, $n$ hash values are generated by the $n$ hash functions, and every position indicated by a hash value is flipped to one. In the query phase, given an item, $n$ hash values are calculated as before - if any of these $n$ positions is a zero, then the item is definitely not in the collection. However, because of the possibility of hash collisions, even if all the positions are one, this could be a false positive. Clearly, the rate of false positives depends on the ratio of zero and one bits, and there are Bloom filter implementations that will dynamically bound the ratio and hence the false positive rate.

Possible uses of a Bloom filter include:

- Does a particular sequence motif appear in a DNA string?
- Has this book been recommended to this customer before?
- Check if an element exists on disk before performing I/O
- Check if URL is a potential malware site using in-browser Bloom filter to minimize network communication
- As an alternative way to generate distinct value counts cheaply (only increment count if Bloom filter says NO)_____no_output_____```bash
pip install git+https://github.com/jaybaird/python-bloomfilter.git
```_____no_output_____

from pybloom import ScalableBloomFilter

# The Scalable Bloom Filter grows as needed to keep the error rate small
# The default error_rate=0.001
sbf = ScalableBloomFilter()_____no_output_____for word in word_set:
    sbf.add(word)_____no_output_____test_words = ['banana', 'artist', 'Dublin', 'masochist', 'Obama']_____no_output_____for word in test_words:
    print(word, word in sbf)banana True
artist True
Dublin True
masochist False
Obama False
### Chedck
for word in test_words:
    print(word, word in word_set)banana True
artist True
Dublin True
masochist False
Obama False


# Notebook from irockafe/metabolomics_processing
Path: notebooks/MTBLS72/exploratory/auc_vs_rt_window_plots.ipynb

 Make a plot of AUC vs. rt-window slices

dataset: MTBLS72_uplc_pos_classifer_15ppm_bw2_____no_output_____

import time

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np

from sklearn import preprocessing
from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import StratifiedShuffleSplit
from sklearn.cross_validation import cross_val_score
#from sklearn.model_selection import StratifiedShuffleSplit
#from sklearn.model_selection import cross_val_score

from sklearn.ensemble import AdaBoostClassifier
from sklearn.metrics import roc_curve, auc
from sklearn.utils import shuffle

from scipy import interp
import scipy.stats as stats

import pickle

# My libraries
import data.preprocessing as preproc
import project_fxns.rt_window_prediction as rtwin

%matplotlib inline
%load_ext autoreload
%autoreload 2_____no_output_____toy = pd.DataFrame([[1,2,3,0],
                    [0,0,0,0],
                  [0.5,1,0,0]], dtype=float,
                  columns=['1', '2', '3', '4'],
                  index=['sample_%s' % i for i in range(1,4)])

preproc.prevalence_threshold(toy)
Requiring sample to be present in 0.5 percent of samples

 Get the xcms data
_____no_output_____

### Subdivide the data into a feature table
local_path = '/home/irockafe/Dropbox (MIT)/Alm_Lab/'\
'projects'
data_path = local_path + '/revo_healthcare/data/processed/MTBLS72/positive_mode/'\
'mtbls_no_retcor_bw2.csv'

## Import the data and remove extraneous columns
df = pd.read_csv(data_path, index_col=0)
# Replace X's at the beginning of sample names
new_idx = [i.replace('X', '') for i in df.columns]
df.columns = new_idx
print df.columns

df.shape
df.head()
# Make a new index of mz:rt
mz = df.loc[:,"mz"].astype('str')
rt = df.loc[:,"rt"].astype('str')
idx = mz+':'+rt
df.index = idx

# separate samples from xcms/camera things to make feature table
not_samples = ['mz', 'mzmin', 'mzmax', 'rt', 'rtmin', 'rtmax',
               'npeaks', 'positive_mode',
               ]
samples_list = df.columns.difference(not_samples)
mz_rt_df = df[not_samples]

# Convert to (samples x features) format - our standard
X_df_raw = df[samples_list].T
new_idx = [i.replace('X', '') for i in X_df_raw.index]
X_df_raw.index = new_idx

Index([u'mz', u'mzmin', u'mzmax', u'rt', u'rtmin', u'rtmax', u'npeaks',
       u'positive_mode', u'20120829_Howard_POS_07', u'20120829_Howard_POS_08',
       ...
       u'20120829_Howard_POS_89', u'20120829_Howard_POS_90',
       u'20120829_Howard_POS_91', u'20120829_Howard_POS_92',
       u'20120829_Howard_POS_93', u'20120829_Howard_POS_94',
       u'20120829_Howard_POS_95', u'20120829_Howard_POS_96',
       u'20120829_Howard_POS_98', u'20120829_Howard_POS_99'],
      dtype='object', length=262)

 Get mappings between sample names, file names, and sample classes

_____no_output_____

# get mapping between sample name and sample class
path_sample_class_map = (local_path +
        '/revo_healthcare/data/raw/MTBLS72/s_Plasma_AD_Lipidomics.txt')
class_column = 'Factor Value[Cognitive Status]'

class_df = pd.read_csv(path_sample_class_map,
                      sep='\t')
# Set index as sample name
class_df.set_index('Sample Name', inplace=True)
class_df = class_df[class_column]

# select only positive values from positive-ion mode
class_df = class_df[class_df.index.str.contains('POS')]
print class_df.head(10)
print "Class label shape: ", class_df.shape
print "feature table shape: ", X_df_raw.shape

#class_df.rename('class', inplace=True)
print class_df.head()
print '\n\nClass labels: ', pd.unique(class_df.values)

# Get case and control dataframes
case_labels = class_df[class_df=='aMCI/AD'].index
control_labels = class_df[class_df == 'Normal Control'].index
case = X_df_raw.loc[case_labels]
control = X_df_raw.loc[control_labels]Sample Name
20120829_Howard_POS_129    aMCI/AD
20120829_Howard_POS_130    aMCI/AD
20120829_Howard_POS_131    aMCI/AD
20120829_Howard_POS_132    aMCI/AD
20120829_Howard_POS_134    aMCI/AD
20120829_Howard_POS_135    aMCI/AD
20120829_Howard_POS_137    aMCI/AD
20120829_Howard_POS_142    aMCI/AD
20120829_Howard_POS_158    aMCI/AD
20120829_Howard_POS_160    aMCI/AD
Name: Factor Value[Cognitive Status], dtype: object
Class label shape:  (323,)
feature table shape:  (254, 1417)
Sample Name
20120829_Howard_POS_129    aMCI/AD
20120829_Howard_POS_130    aMCI/AD
20120829_Howard_POS_131    aMCI/AD
20120829_Howard_POS_132    aMCI/AD
20120829_Howard_POS_134    aMCI/AD
Name: Factor Value[Cognitive Status], dtype: object

Class labels:  ['aMCI/AD' 'Normal Control' 'Not Applicable']

 Convert class labels to binary

asser that X and y match one another's order

Control is 0, alzheimers is 1

Note that the 'Not Applicable' class is not part of my
dataframe, X, so when I select only  we make sure that X and y refer
to the same samples, all the 'Not Applicable' samples are dropped_____no_output_____

# Match between feature table and metadata and assert that they're in the same order.
# then define the numpy-arrays for X and y
class_df = class_df[X_df_raw.index].sort_index()
X_df_raw = X_df_raw.sort_index()
print 'Class values:', class_df.unique()
assert (class_df.index == X_df_raw.index).all()

# convert classes to numbers
le = preprocessing.LabelEncoder()
le.fit(class_df)
y = le.transform(class_df)
print y
print "class-labels: ", le.classes_Class values: ['Normal Control' 'aMCI/AD']
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 1 0 1 1 0 1 0 0 0
 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 1 0 1 0 1 1 0 1 1
 1 0 1 0 1 1 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 1 1 1 1 0 0 1 0 1
 1 1 0 1 0 1 1 1 0 0 0 1 0 0 1 1 1 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 1 1 0 0
 0 0 0 1 0 1 0 1 0 0 1 0 1 0 0 0 1 1 1 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 1 1
 0 0 0 0 0 1 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
class-labels:  ['Normal Control' 'aMCI/AD']

 Preprocess feature table

Try to remove systematic intensity biases between samples by correcting for dilution factors _____no_output_____

# fill nan values with 1/2 the minimum from each sample
fill_val = X_df_raw.min(axis=1) / 2.0

# Only keep features present in at least 70% of samples
# Note that this means 70% total, not 70% in a particular class
X_df_filtered = preproc.prevalence_threshold(X_df_raw,
                                           threshold=0.7)
print 'Raw shape', X_df_raw.shape
print 'Filtered at 70% prevalence', X_df_filtered.shape
# must transpose, b/c fillna only operates along columns
X_df_filled = X_df_filtered.fillna(value=fill_val, )

#X_pqn_df_raw = preproc.correct_dilution_factor(X_df_raw, plot=True)
X_pqn_df_filled = preproc.correct_dilution_factor(X_df_filled, plot=True)

X_pqn_df_filled_log = np.log10(X_pqn_df_filled)Requiring sample to be present in 0.7 percent of samples
Raw shape (254, 1417)
Filtered at 70% prevalence (254, 866)

 There is a bias in intensity values between case and control. Can I eliminate that through some normalization?

sort of - for non-log transformed intensities, Mann-whitney test has p>0.2. For log-transformed, p~0.3

_____no_output_____

# Do mann-whitney on case vs control
def mw_pval_dist(case, control):
    '''
    case - dataframe containing case
    control - dataframe with control samples
        All should have same features (columns)
    '''
    # get parametric pvals
    mann_whitney_vals = pd.DataFrame(np.full([case.shape[1],2], np.nan),
                              index=case.columns, columns= ['u', 'pval'])
    for idx, case_vals in case.iteritems():
        control_vals = control[idx]
        u, pval = stats.mannwhitneyu(case_vals, control_vals)
        mann_whitney_vals.loc[idx, 'u'] = u
        mann_whitney_vals.loc[idx, 'pval'] = pval

    # plot mw pval distribution
    mann_whitney_vals.hist('pval')
    plt.title('mann-whitney pval between case and control')
    plt.show()

    # plot distribution of mean intensities
    case_mean = case.mean(axis=0)
    ctrl_mean = control.mean(axis=0)
    sns.distplot(np.log10(case_mean), label='case')
    sns.distplot(np.log10(ctrl_mean), label='control')
    plt.xlabel('log_10 intensity')
    plt.title('Mean intensity of case vs. control')
    plt.legend()
    plt.show()
    u, pval = stats.mannwhitneyu(case_mean, ctrl_mean)
    print 'pval (MannW) of intensities between case and control: ', pval

#print('Raw intensities\n\n')
#mw_pval_dist(X_df_raw.loc[case_labels], X_df_raw.loc[control_labels])

print('*'*50+'\nNaN filled with 1/2 min')
mw_pval_dist(X_df_filled.loc[case_labels], X_df_filled.loc[control_labels])

#print('*'*50+'\n Raw pqn_normalized')
#mw_pval_dist(X_pqn_df_raw.loc[case_labels], X_pqn_df_raw.loc[control_labels])

print('*'*50+'\n NaN filled with 1/2 min, pqn normalized')
mw_pval_dist(X_pqn_df_filled.loc[case_labels], X_pqn_df_filled.loc[control_labels])

print('*'*50+'\n NaN filled with 1/2 min, pqn normalized, log10 transformed')
mw_pval_dist(X_pqn_df_filled_log.loc[case_labels],
             X_pqn_df_filled_log.loc[control_labels])

**************************************************
NaN filled with 1/2 min

 It seems really weird to me that almost half of all features are have a p-vals less than 0.05

Remember that the case group is elderly people who go on to develop alzheimers 2 years later. You wouldn't expect this wild of a difference. Maybe my processing was wrong...?

Note that the pvals reported assume normality - I didn't do them via permutation test, because laziness at current moment_____no_output_____
 Run rt-window classifiers and capture the auc distributions
to plot

Using a sliding window with a step-size that's 1/2 of the window size. Meaning that each point displayed contains the all features between adjacent points in the plot._____no_output_____

# Add back mz, rt, etc. columns to feature table and reshape it to be
# (feats x samples)
X_pqn_df_filled_mzrt = pd.concat([df[not_samples].T, X_pqn_df_filled],
                                 axis=0).T

# run a sliding windonw
# Make sliding window
min_val = 0
max_val = df['rt'].max()
width = max_val / 5
step = width / 2
sliding_window = rtwin.make_sliding_window(min_val,
                                           max_val, width, step)

# Paths to plot things
path = ('/revo_healthcare/presentations/isaac_bats/'+
            'rt_window_plots/MTBLS72/')
output_path = local_path + path

# run classifier & plot on sliding window
n_iter = 50
test_size = 0.3
rf_trees = 1000
# Run rt-sliding-window classifier
rtwin.sliding_rt_window_aucs(X_pqn_df_filled_mzrt, y, sliding_window, not_samples,
                      rf_trees=rf_trees, n_iter=n_iter, test_size=test_size,
                      output_path=output_path)RT plot (0.0, 132.3533022)
661.766511
998.775098907
auc_vals = pickle.load(open(output_path+'/auc_vals.pkl', 'rb'))

fig_path = output_path + 'auc_vs_rt.pdf'
rtwin.plot_auc_vs_rt(auc_vals, sliding_window, df, fig_path)_____no_output_____

### Based on the data (half of all features are discriminatory), this result isn't weird. But that fact is still weird

That would be a shocking finding. That plasma metabolites are sooooo different between people about to experience cognitive impairment vs. those who aren't.

I need to re-process the data & ask authors for their parameters.

## Note that the authors went in and developed a panel of biomarkers (lots of phosphatidylcholline derivatives) and validated using that. So I don't doubt their overall findings
But this still feels fishy to me and reads as a cautionary note for my analyses_____no_output_____