# Notebook from nikbaya/hail
Path: hail/python/hail/docs/tutorials/07-matrixtable.ipynb

## MatrixTable Tutorial

If you've gotten this far, you're probably thinking:

- "Can't I do all of this in `pandas` or `R`?"
- "What does this have to do with biology?"

The two crucial features that Hail adds are _scalability_ and the _domain-specific primitives_ needed to work easily with biological data. Fear not! You've learned most of the basic concepts of Hail and now are ready for the bit that makes it possible to represent and compute on genetic matrices: the [MatrixTable](https://hail.is/docs/0.2/hail.MatrixTable.html)._____no_output_____In the last example of the [Table Joins Tutorial](https://hail.is/docs/0.2/tutorials/08-joins.html), the ratings table had a compound key: `movie_id` and `user_id`.  The ratings were secretly a movie-by-user matrix!

However, since this matrix is very sparse, it is reasonably represented in a so-called "coordinate form" `Table`, where each row of the table is an entry of the sparse matrix. For large and dense matrices (like sequencing data), the per-row overhead of coordinate reresentations is untenable. That's why we built `MatrixTable`, a 2-dimensional generalization of `Table`._____no_output_____### MatrixTable Anatomy

Recall that `Table` has two kinds of fields:

- global fields
- row fields

`MatrixTable` has four kinds of fields:

- global fields
- row fields
- column fields
- entry fields_____no_output_____Row fields are fields that are stored once per row. These can contain information about the rows, or summary data calculated per row.

Column fields are stored once per column. These can contain information about the columns, or summary data calculated per column.

Entry fields are the piece that makes this structure a matrix -- there is an entry for each (row, column) pair._____no_output_____### Importing and Reading

Like tables, matrix tables can be [imported](https://hail.is/docs/0.2/methods/impex.html) from a variety of formats: VCF, (B)GEN, PLINK, TSV, etc.  Matrix tables can also be *read* from a "native" matrix table format.  Let's read a sample of prepared [1KG](https://en.wikipedia.org/wiki/1000_Genomes_Project) data._____no_output_____

import hail as hl
from bokeh.io import output_notebook, show
output_notebook()

hl.utils.get_1kg('data/')_____no_output_____mt = hl.read_matrix_table('data/1kg.mt')
mt.describe()_____no_output_____

There are a few things to note:

- There is a single column field `s`. This is the sample ID from the VCF. It is also the column key.
- There is a compound row key: `locus` and `alleles`.
  - `locus` has type `locus<GRCh37>`
  - `alleles` has type `array<str>`
- GT has type `call`.  That's a genotype call!_____no_output_____Whereas table expressions could be indexed by nothing or indexed by rows, matrix table expression have four options: nothing, indexed by row, indexed by column, or indexed by row and column (the entries).  Let's see some examples._____no_output_____

mt.s.describe()_____no_output_____mt.GT.describe()_____no_output_____

### MatrixTable operations
We belabored the operations on tables because they all have natural analogs (sometimes several) on matrix tables.  For example:

 - `count` => `count_{rows, cols}` (and `count` which returns both)
 - `filter` => `filter_{rows, cols, entries}`
 - `annotate` => `annotate_{rows, cols, entries}` (and globals for both)
 - `select` => `select_{rows, cols, entries}` (and globals for both)
 - `transmute` => `transmute_{rows, cols, entries}` (and globals for both)
 - `group_by` => `group_{rows, cols}_by`
 - `explode` => `expode_{rows, cols}`
 - `aggregate` => `aggregate_{rows, cols, entries}`_____no_output_____Some operations are unique to `MatrixTable`:

- The row fields can be accessed as a `Table` with [rows](https://hail.is/docs/0.2/hail.MatrixTable.html#hail.MatrixTable.rows)
- The column fields can be accessed as a `Table` with [cols](https://hail.is/docs/0.2/hail.MatrixTable.html#hail.MatrixTable.cols).
- The entire field space of a `MatrixTable` can be accessed as a coordinate-form `Table` with [entries](https://hail.is/docs/0.2/hail.MatrixTable.html#hail.MatrixTable.entries). Be careful with this! While it's fast to aggregate or query, trying to write this `Table` to disk could produce files _thousands of times larger_ than the corresponding `MatrixTable`.

Let's explore `mt` using these tools.  Let's get the size of the dataset._____no_output_____

mt.count() # (rows, cols)_____no_output_____

Let's look at the first few row keys (variants) and column keys (sample IDs)._____no_output_____

mt.rows().select().show()_____no_output_____mt.s.show()_____no_output_____

Let's investigate the genotypes and the call rate.  Let's look at the first few genotypes:_____no_output_____

mt.GT.show()_____no_output_____

All homozygous reference, which is not surprising.  Let's look at the distribution of genotype calls:_____no_output_____

mt.aggregate_entries(hl.agg.counter(mt.GT.n_alt_alleles()))_____no_output_____

Let's compute the overall call rate directly, and then plot the distribution of call rate per variant._____no_output_____

mt.aggregate_entries(hl.agg.fraction(hl.is_defined(mt.GT)))_____no_output_____

Here's a nice trick: you can use an aggregator inside `annotate_rows` and it will aggregate over columns, that is, summarize the values in the row using the aggregator.  Let's compute and plot call rate per variant._____no_output_____

mt2 = mt.annotate_rows(call_rate = hl.agg.fraction(hl.is_defined(mt.GT)))
mt2.describe()_____no_output_____p = hl.plot.histogram(mt2.call_rate, range=(0,1.0), bins=100,
                      title='Variant Call Rate Histogram', legend='Call Rate')
show(p)_____no_output_____

### Exercise: GQ vs DP

In this exercise, you'll use Hail to investigate a strange property of sequencing datasets.

The `DP` field is the sequencing depth (the number of reads).

Let's first plot a histogram of `DP`:_____no_output_____

p = hl.plot.histogram(mt.DP, range=(0,40), bins=40, title='DP Histogram', legend='DP')
show(p)_____no_output_____

Now, let's do the same thing for GQ.

The `GQ` field is the phred-scaled "genotype quality". The formula to convert to a linear-scale confidence (0 to 1) is  `10 ** -(mt.GQ / 10)`. GQ is truncated to lie between 0 and 99.
_____no_output_____

p = hl.plot.histogram(mt.GQ, range=(0,100), bins=100, title='GQ Histogram', legend='GQ')
show(p)_____no_output_____

Whoa!  That's a strange distribution!  There's a big spike at 100. The rest of the values have roughly the same shape as the DP distribution, but form a [Dimetrodon](https://en.wikipedia.org/wiki/Dimetrodon).  Use Hail to figure out what's going on!_____no_output_____


# Notebook from wguesdon/Data_Science_portfolio
Path: Data_Science_projects/Blog_Google_Analytics/EDA.ipynb

_____no_output_____# About BrainPost

Kasey runs BrainPost with a fellow PhD friend as a way to keep in touch with her scientific roots while working as a data scientist! Every Tuesday since we started in early 2018, we publish our e-newsletter which is three short summaries of new neuroscience studies that have just come out. After publishing on our website each Tuesday, we typically post on Twitter (@brainpostco) and Facebook (@brainpostco) once to announce the release of the e-newsletter and three times (once for each of the three summaries) to highlight each summary. There are a few exceptions to our publishing schedule. Sometimes we post extra articles here: https://www.brainpost.co/brainpost-life-hacks, and also a few weeks we've only been able to publish two summaries instead of three.  At around the same time as we publish the e-newsletter on the website each Tuesday, we also send it to our ~1700 email subscribers directly (via mailchimp)._____no_output_____# About the Challenge

We're always wondering if we should change what type of content we're publishing, and how people are finding us. From some small surveys we've done for example, we find people would be interested in more casual/applicable to daily life content (like we publish on this tab https://www.brainpost.co/brainpost-life-hacks) than more technical summaries of complex articles, but we also aren't really sure if that's just a subgroup of people who get the e-newsletter to their email inbox filling out the survey. We also might have two audiences - academics and non-academics (?) who like different things._____no_output_____# About the data

In the remaining tabs of this workbook there is weekly pageview data for each page on the website (I think..according to our google analytics). Each tab represents pageview data for two one week periods, split up by the page name/URL and the source/medium (first two columns). My general idea was people can look at the data at a weekly cadence and figure out stats about different pages/content, BUT with google analytics a huge problem is that it doesn't really take into account that different content is published on different days (for example, a stat about 'only 2 pageviews' to a page is meaningless to me if it is because the page was only published an hour ago). Our content is published weekly so it should approximately match that I extracted the data weekly. My apologies for the formatting... Google analytics was a nightmare to extract from - a very manual process. But, I guess data cleaning is a part of the process! So, we've been publishing ~3 new pages a week since 2018, but I've only included data starting in July 2020 because the data extraction process is so manual. The date of publication can be extracted from the URL.

We've noticed some pages seem really popular possibly for strange reasons (maybe they come up on the first page of google because people are searching for something really similar?) and those anomalies might not reflect what people like overall about the site.

There is also a tab with a page (URL) to page title lookup_____no_output_____# The questions we'd like to ask

What content (or types of content) is most popular (what are patterns we see in popular content) and is different content popular amongst different subgroups (e.g. by source/medium)?

Any question that will help us to take action to better tailor our content to our audience(s) or understand how traffic comes to the site.

Where are people visiting from (source-wise)?_____no_output_____# How this challenge works:

Just like the last challenge, you can submit an entry by posting a github link to your analysis on the signup page (second tab). Use any combination of reports/visuals/code/presentation you think is best - just make sure your code is accessible!

Let's have a few days for people to review the data and ask any questions and then we can discuss what everyone thinks is a reasonable deadline/timeline and set the timeline from there. If you have any further data requests you think would help answer the questions I might be able to get it (google analytics or mailchimp).

After the deadline I'll choose the first and second place submission. The criteria will be whatever submission provides the most compelling evidence that gives me a clear idea of what actions we could take next to improve the site._____no_output_____# Initialize_____no_output_____

# to use Panda profilling
! pip install https://github.com/pandas-profiling/pandas-profiling/archive/master.zipCollecting https://github.com/pandas-profiling/pandas-profiling/archive/master.zip
  Using cached https://github.com/pandas-profiling/pandas-profiling/archive/master.zip
Requirement already satisfied (use --upgrade to upgrade): pandas-profiling==2.9.0 from https://github.com/pandas-profiling/pandas-profiling/archive/master.zip in /usr/local/lib/python3.6/dist-packages
Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (0.17.0)
Requirement already satisfied: scipy>=1.4.1 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (1.4.1)
Requirement already satisfied: pandas!=1.0.0,!=1.0.1,!=1.0.2,!=1.1.0,>=0.25.3 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (1.1.4)
Requirement already satisfied: matplotlib>=3.2.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (3.2.2)
Requirement already satisfied: confuse>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (1.3.0)
Requirement already satisfied: jinja2>=2.11.1 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (2.11.2)
Requirement already satisfied: visions[type_image_path]==0.5.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (0.5.0)
Requirement already satisfied: numpy>=1.16.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (1.18.5)
Requirement already satisfied: attrs>=19.3.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (20.2.0)
Requirement already satisfied: htmlmin>=0.1.12 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (0.1.12)
Requirement already satisfied: missingno>=0.4.2 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (0.4.2)
Requirement already satisfied: phik>=0.9.10 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (0.10.0)
Requirement already satisfied: tangled-up-in-unicode>=0.0.6 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (0.0.6)
Requirement already satisfied: requests>=2.23.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (2.23.0)
Requirement already satisfied: tqdm>=4.43.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (4.52.0)
Requirement already satisfied: ipywidgets>=7.5.1 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (7.5.1)
Requirement already satisfied: seaborn>=0.10.1 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.9.0) (0.11.0)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas!=1.0.0,!=1.0.1,!=1.0.2,!=1.1.0,>=0.25.3->pandas-profiling==2.9.0) (2.8.1)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas!=1.0.0,!=1.0.1,!=1.0.2,!=1.1.0,>=0.25.3->pandas-profiling==2.9.0) (2018.9)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.2.0->pandas-profiling==2.9.0) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.2.0->pandas-profiling==2.9.0) (1.3.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.2.0->pandas-profiling==2.9.0) (2.4.7)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from confuse>=1.0.0->pandas-profiling==2.9.0) (3.13)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from jinja2>=2.11.1->pandas-profiling==2.9.0) (1.1.1)
Requirement already satisfied: networkx>=2.4 in /usr/local/lib/python3.6/dist-packages (from visions[type_image_path]==0.5.0->pandas-profiling==2.9.0) (2.5)
Requirement already satisfied: imagehash; extra == "type_image_path" in /usr/local/lib/python3.6/dist-packages (from visions[type_image_path]==0.5.0->pandas-profiling==2.9.0) (4.2.0)
Requirement already satisfied: Pillow; extra == "type_image_path" in /usr/local/lib/python3.6/dist-packages (from visions[type_image_path]==0.5.0->pandas-profiling==2.9.0) (7.0.0)
Requirement already satisfied: numba>=0.38.1 in /usr/local/lib/python3.6/dist-packages (from phik>=0.9.10->pandas-profiling==2.9.0) (0.48.0)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.23.0->pandas-profiling==2.9.0) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.23.0->pandas-profiling==2.9.0) (1.24.3)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.23.0->pandas-profiling==2.9.0) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.23.0->pandas-profiling==2.9.0) (2020.6.20)
Requirement already satisfied: ipykernel>=4.5.1 in /usr/local/lib/python3.6/dist-packages (from ipywidgets>=7.5.1->pandas-profiling==2.9.0) (4.10.1)
Requirement already satisfied: traitlets>=4.3.1 in /usr/local/lib/python3.6/dist-packages (from ipywidgets>=7.5.1->pandas-profiling==2.9.0) (4.3.3)
Requirement already satisfied: ipython>=4.0.0; python_version >= "3.3" in /usr/local/lib/python3.6/dist-packages (from ipywidgets>=7.5.1->pandas-profiling==2.9.0) (5.5.0)
Requirement already satisfied: nbformat>=4.2.0 in /usr/local/lib/python3.6/dist-packages (from ipywidgets>=7.5.1->pandas-profiling==2.9.0) (5.0.8)
Requirement already satisfied: widgetsnbextension~=3.5.0 in /usr/local/lib/python3.6/dist-packages (from ipywidgets>=7.5.1->pandas-profiling==2.9.0) (3.5.1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.7.3->pandas!=1.0.0,!=1.0.1,!=1.0.2,!=1.1.0,>=0.25.3->pandas-profiling==2.9.0) (1.15.0)
Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.4->visions[type_image_path]==0.5.0->pandas-profiling==2.9.0) (4.4.2)
Requirement already satisfied: PyWavelets in /usr/local/lib/python3.6/dist-packages (from imagehash; extra == "type_image_path"->visions[type_image_path]==0.5.0->pandas-profiling==2.9.0) (1.1.1)
Requirement already satisfied: llvmlite<0.32.0,>=0.31.0dev0 in /usr/local/lib/python3.6/dist-packages (from numba>=0.38.1->phik>=0.9.10->pandas-profiling==2.9.0) (0.31.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from numba>=0.38.1->phik>=0.9.10->pandas-profiling==2.9.0) (50.3.2)
Requirement already satisfied: tornado>=4.0 in /usr/local/lib/python3.6/dist-packages (from ipykernel>=4.5.1->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (5.1.1)
Requirement already satisfied: jupyter-client in /usr/local/lib/python3.6/dist-packages (from ipykernel>=4.5.1->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (5.3.5)
Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.6/dist-packages (from traitlets>=4.3.1->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.2.0)
Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.8.1)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.7.5)
Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (1.0.18)
Requirement already satisfied: pygments in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (2.6.1)
Requirement already satisfied: pexpect; sys_platform != "win32" in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (4.8.0)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.6/dist-packages (from nbformat>=4.2.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (4.6.3)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /usr/local/lib/python3.6/dist-packages (from nbformat>=4.2.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (2.6.0)
Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.6/dist-packages (from widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (5.3.1)
Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.6/dist-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (19.0.2)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.2.5)
Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.6/dist-packages (from pexpect; sys_platform != "win32"->ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.6.0)
Requirement already satisfied: nbconvert in /usr/local/lib/python3.6/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (5.6.1)
Requirement already satisfied: terminado>=0.8.1 in /usr/local/lib/python3.6/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.9.1)
Requirement already satisfied: Send2Trash in /usr/local/lib/python3.6/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (1.5.0)
Requirement already satisfied: bleach in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (3.2.1)
Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.8.4)
Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (1.4.3)
Requirement already satisfied: defusedxml in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.6.0)
Requirement already satisfied: testpath in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.4.4)
Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.3)
Requirement already satisfied: webencodings in /usr/local/lib/python3.6/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (0.5.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->pandas-profiling==2.9.0) (20.4)
Building wheels for collected packages: pandas-profiling
  Building wheel for pandas-profiling (setup.py) ... done
  Created wheel for pandas-profiling: filename=pandas_profiling-2.9.0-py2.py3-none-any.whl size=258952 sha256=db6de102cfabff0c570bb4e18eb2e911b25dfdd2d239f8514b00b776c0bb6ff6
  Stored in directory: /tmp/pip-ephem-wheel-cache-4ew95pq9/wheels/56/c2/dd/8d945b0443c35df7d5f62fa9e9ae105a2d8b286302b92e0109
Successfully built pandas-profiling
# Load libraries
# https://stackoverflow.com/questions/58667299/google-colab-why-matplotlib-has-a-behaviour-different-then-default-after-import
# Panda profilling alter seaborn plotting style

from google.colab import drive # to load data from google drive
from pandas_profiling import ProfileReport
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt # ploting the data
import seaborn as sns # ploting the data
%matplotlib inline
import math # calculation_____no_output_____drive.mount("/content/drive")Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
# List files
# see https://careerkarma.com/blog/python-list-files-in-directory/
import os
os.listdir('/content/drive/MyDrive/DSS/BrainPost_blog/Data')_____no_output_____

# Data Cleaning and Wrangling_____no_output_____

# Load the data
# See https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html
# See https://datatofish.com/read_excel/

# Load the Pages Titles
pages_tiltes = pd.read_excel(r'/content/drive/MyDrive/DSS/BrainPost_blog/Data/DSS BrainPost Web Analytics Challenge!.xlsx', sheet_name='Page to Page Title lookup')

# Remove headers
pages_tiltes = pages_tiltes.iloc[6:258,]
# Fix columns names
# See https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rename.html
pages_tiltes = pages_tiltes.rename(columns={"# ----------------------------------------": "Page", "Unnamed: 1": "Page_title"})

pages_tiltes_____no_output_____
# Load the Pages Titles
pages_tiltes = pd.read_excel(r'/content/drive/MyDrive/DSS/BrainPost_blog/Data/DSS BrainPost Web Analytics Challenge!.xlsx', sheet_name='Page to Page Title lookup', skiprows=6)

pages_tiltes_____no_output_____# Load the content-pages oct 4-17
# Skip the first 6 rows with headers
# Skip the last 18 rows with total view
# See https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html
# See https://stackoverflow.com/questions/49876077/pandas-reading-excel-file-starting-from-the-row-below-that-with-a-specific-valu_____no_output_____#content-pages july 12-25
july12_25 = pd.read_excel(r'/content/drive/MyDrive/DSS/BrainPost_blog/Data/DSS BrainPost Web Analytics Challenge!.xlsx',
                         sheet_name='content-pages july 12-25',
                        skiprows=6, skipfooter=20)
#july12_25_____no_output_____#content-pages jul 26 - aug 8
jul26_aug8 = pd.read_excel(r'/content/drive/MyDrive/DSS/BrainPost_blog/Data/DSS BrainPost Web Analytics Challenge!.xlsx',
                         sheet_name='content-pages jul 26 - aug 8',
                        skiprows=6, skipfooter=20)
#jul26_aug8_____no_output_____#content-pages aug 9-22
aug9_22 = pd.read_excel(r'/content/drive/MyDrive/DSS/BrainPost_blog/Data/DSS BrainPost Web Analytics Challenge!.xlsx',
                         sheet_name='content-pages aug 9-22',
                        skiprows=6, skipfooter=20)
#aug9_22_____no_output_____#content-pages aug23-sept5
aug23_sept5 = pd.read_excel(r'/content/drive/MyDrive/DSS/BrainPost_blog/Data/DSS BrainPost Web Analytics Challenge!.xlsx',
                         sheet_name='content-pages aug23-sept5',
                        skiprows=6, skipfooter=20)
#aug23_sept5_____no_output_____#content-pages sept 6-19
sept6_19 = pd.read_excel(r'/content/drive/MyDrive/DSS/BrainPost_blog/Data/DSS BrainPost Web Analytics Challenge!.xlsx',
                         sheet_name='content-pages sept 6-19',
                        skiprows=6, skipfooter=20)
#sept6_19_____no_output_____#content-pages sept 20-oct3
sept20_oct3 = pd.read_excel(r'/content/drive/MyDrive/DSS/BrainPost_blog/Data/DSS BrainPost Web Analytics Challenge!.xlsx',
                         sheet_name='content-pages sept 20-oct3',
                        skiprows=6, skipfooter=20)
#sept20_oct3_____no_output_____#content-pages oct 4-17
Oct4_17 = pd.read_excel(r'/content/drive/MyDrive/DSS/BrainPost_blog/Data/DSS BrainPost Web Analytics Challenge!.xlsx',
                        sheet_name='content-pages oct 4-17',
                        skiprows=6, skipfooter=20)
#Oct4_17_____no_output_____#content-pages oct 18-31
Oct18_31 = pd.read_excel(r'/content/drive/MyDrive/DSS/BrainPost_blog/Data/DSS BrainPost Web Analytics Challenge!.xlsx',
                         sheet_name='content-pages oct 18-31',
                        skiprows=6, skipfooter=20)
#Oct18_31_____no_output_____# Combine data frame
# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.combine.html
# https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.html

frames = [july12_25,
          jul26_aug8,
          aug9_22,
          aug23_sept5,
          sept6_19,
          sept20_oct3,
          Oct4_17,
          Oct18_31]
df = pd.concat(frames)
df_____no_output_____# Merge with the page title data frame
# https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.html_____no_output_____df_inner = pd.merge(df, pages_tiltes, how='inner', on='Page')
df_inner_____no_output_____df_outer = pd.merge(df, pages_tiltes, how='outer', on='Page')
df_outer_____no_output_____# Inner and Outer merge result in 4094 and 9121 rows
# This seem to occurs because 5027 entries to Pages abscent from the Page Title table_____no_output_____9121 - 4094_____no_output_____# Export table
df.to_excel('/content/drive/MyDrive/DSS/BrainPost_blog/Results/df.xlsx')
df_inner.to_excel('/content/drive/MyDrive/DSS/BrainPost_blog/Results/df_inner.xlsx')
df_outer.to_excel('/content/drive/MyDrive/DSS/BrainPost_blog/Results/df_outer.xlsx')
pages_tiltes.to_excel('/content/drive/MyDrive/DSS/BrainPost_blog/Results/pages_tiltes.xlsx')_____no_output_____# Determine the number of missing values for every column
df_outer.isnull().sum()_____no_output_____# Perforn Automated EDA

# see https://analyticsindiamag.com/tips-for-automating-eda-using-pandas-profiling-sweetviz-and-autoviz-in-python/
# see https://medium.com/python-in-plain-english/how-to-use-pandas-profiling-on-google-colab-e34f34ff1c9f
design_report = ProfileReport(df_outer)
design_report.to_file(output_file='/content/drive/MyDrive/DSS/BrainPost_blog/Results/Blog_report.html')_____no_output_____df = df_outer

# Filter entries with Pageviews >= 0
# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.filter.html
# https://cmdlinetips.com/2018/02/how-to-subset-pandas-dataframe-based-on-values-of-a-column/

df_Pageviews_filtered = df[df['Pageviews'] > 0]
df_Pageviews_filtered.isnull().sum()_____no_output_____df_Pageviews_filtered_____no_output_____design_report = ProfileReport(df_Pageviews_filtered)
design_report.to_file(output_file='/content/drive/MyDrive/DSS/BrainPost_blog/Results/df_Pageviews_filtered.html')_____no_output_____

# Examine pages traffic and source_____no_output_____

# Create a value count column
# https://stackoverflow.com/questions/29791785/python-pandas-add-a-column-to-my-dataframe-that-counts-a-variable

df = df_Pageviews_filtered.copy()

df['Page_count'] = df.groupby('Page')['Page'].transform('count')
df['Source_count'] = df.groupby('Source / Medium')['Source / Medium'].transform('count')
df['Page_Title_count'] = df.groupby('Page Title')['Page Title'].transform('count')_____no_output_____# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html
df.groupby('Page').sum()_____no_output_____# https://stackoverflow.com/questions/42528921/how-to-prevent-overlapping-x-axis-labels-in-sns-countplot
# https://stackoverflow.com/questions/46623583/seaborn-countplot-order-categories-by-count
# https://stackoverflow.com/questions/25328003/how-can-i-change-the-font-size-using-seaborn-facetgrid

data = df[df['Page_count'] >= 61]
plt.figure(figsize=(10, 5))
sns.set(font_scale=1.25)
title = 'Top 5 Pages visited at least once'
sns.countplot(y = data['Page'],
                   order = data['Page'].value_counts().index)
plt.title(title)
plt.ioff()_____no_output_____data = df[df['Source_count'] >= 270]
plt.figure(figsize=(10, 5))
sns.set(font_scale=1.25)
title = 'Top 5 Source for Pages visited at least once'
sns.countplot(y = data['Source / Medium'],
                   order = data['Source / Medium'].value_counts().index)
plt.title(title)
plt.ioff()_____no_output_____# https://www.statology.org/pandas-filter-multiple-conditions/
data = df[(df['Page_count'] >= 61) & (df['Source_count'] >= 270)]
y="Page"
x="Pageviews"

title = ""
plt.figure(figsize=(10, 20))
sns.set(font_scale=1.25)
sns.boxplot(x=x, y=y, data=data, notch=False, showmeans=True,
           meanprops={"marker":"s","markerfacecolor":"white", "markeredgecolor":"black"},
            hue='Source / Medium')
plt.title(title)
plt.ioff()_____no_output_____

It seems that direct referal users spend more time on page for the weekly-brainpost page.
It is probalby indicative that this user open the link from the mail list and
are more interested in the content than user finding the site from google search._____no_output_____

df.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 5515 entries, 1 to 9057
Data columns (total 14 columns):
 #   Column             Non-Null Count  Dtype
---  ------             --------------  -----
 0   Page               5515 non-null   object
 1   Source / Medium    5515 non-null   object
 2   Date Range         5515 non-null   object
 3   Pageviews          5515 non-null   float64
 4   Unique Pageviews   5515 non-null   float64
 5   Avg. Time on Page  5515 non-null   object
 6   Entrances          5515 non-null   float64
 7   Bounce Rate        5515 non-null   float64
 8   % Exit             5515 non-null   float64
 9   Page Value         5515 non-null   float64
 10  Page Title         2827 non-null   object
 11  Page_count         5515 non-null   int64
 12  Source_count       5515 non-null   int64
 13  Page_Title_count   2827 non-null   float64
dtypes: float64(7), int64(2), object(5)
memory usage: 646.3+ KB
# https://stackoverflow.com/questions/32375471/pandas-convert-strings-to-time-without-date
# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html
# df_Pageviews_filtered['time'] = pd.to_datetime(df_Pageviews_filtered['Avg. Time on Page'], format='%H:%M:%S').dt.time
df['time'] = pd.to_datetime(df['Avg. Time on Page'],  format='%H:%M:%S')_____no_output_____df.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 5515 entries, 1 to 9057
Data columns (total 15 columns):
 #   Column             Non-Null Count  Dtype
---  ------             --------------  -----
 0   Page               5515 non-null   object
 1   Source / Medium    5515 non-null   object
 2   Date Range         5515 non-null   object
 3   Pageviews          5515 non-null   float64
 4   Unique Pageviews   5515 non-null   float64
 5   Avg. Time on Page  5515 non-null   object
 6   Entrances          5515 non-null   float64
 7   Bounce Rate        5515 non-null   float64
 8   % Exit             5515 non-null   float64
 9   Page Value         5515 non-null   float64
 10  Page Title         2827 non-null   object
 11  Page_count         5515 non-null   int64
 12  Source_count       5515 non-null   int64
 13  Page_Title_count   2827 non-null   float64
 14  time               5515 non-null   datetime64[ns]
dtypes: datetime64[ns](1), float64(7), int64(2), object(5)
memory usage: 689.4+ KB
df[['time']]_____no_output_____df['time'] = pd.to_datetime(df['time'], unit='s')_____no_output_____df[['time']]_____no_output_____# Convert the Av time to second
# Will need to vectorize to apply this function to every row
# https://stackoverflow.com/questions/22923775/calculate-pandas-dataframe-time-difference-between-two-columns-in-hours-and-minu

t1 = pd.to_datetime('1900-01-01 00:00:00')
t2 = pd.to_datetime('1900-01-01 00:00:06')

pd.Timedelta(t2 - t1).seconds_____no_output_____df[['time0']] = pd.to_datetime('1900-01-01 00:00:00')_____no_output_____df.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 5515 entries, 1 to 9057
Data columns (total 16 columns):
 #   Column             Non-Null Count  Dtype
---  ------             --------------  -----
 0   Page               5515 non-null   object
 1   Source / Medium    5515 non-null   object
 2   Date Range         5515 non-null   object
 3   Pageviews          5515 non-null   float64
 4   Unique Pageviews   5515 non-null   float64
 5   Avg. Time on Page  5515 non-null   object
 6   Entrances          5515 non-null   float64
 7   Bounce Rate        5515 non-null   float64
 8   % Exit             5515 non-null   float64
 9   Page Value         5515 non-null   float64
 10  Page Title         2827 non-null   object
 11  Page_count         5515 non-null   int64
 12  Source_count       5515 non-null   int64
 13  Page_Title_count   2827 non-null   float64
 14  time               5515 non-null   datetime64[ns]
 15  time0              5515 non-null   datetime64[ns]
dtypes: datetime64[ns](2), float64(7), int64(2), object(5)
memory usage: 732.5+ KB
df['time_diff'] = df['time'] - df['time0']
#df_Pageviews_filtered_____no_output_____df['time_second'] = df['time_diff'].dt.total_seconds().astype(int)_____no_output_____df[['time_second']]_____no_output_____df.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 5515 entries, 1 to 9057
Data columns (total 18 columns):
 #   Column             Non-Null Count  Dtype
---  ------             --------------  -----
 0   Page               5515 non-null   object
 1   Source / Medium    5515 non-null   object
 2   Date Range         5515 non-null   object
 3   Pageviews          5515 non-null   float64
 4   Unique Pageviews   5515 non-null   float64
 5   Avg. Time on Page  5515 non-null   object
 6   Entrances          5515 non-null   float64
 7   Bounce Rate        5515 non-null   float64
 8   % Exit             5515 non-null   float64
 9   Page Value         5515 non-null   float64
 10  Page Title         2827 non-null   object
 11  Page_count         5515 non-null   int64
 12  Source_count       5515 non-null   int64
 13  Page_Title_count   2827 non-null   float64
 14  time               5515 non-null   datetime64[ns]
 15  time0              5515 non-null   datetime64[ns]
 16  time_diff          5515 non-null   timedelta64[ns]
 17  time_second        5515 non-null   int64
dtypes: datetime64[ns](2), float64(7), int64(3), object(5), timedelta64[ns](1)
memory usage: 818.6+ KB
sns.set_style("whitegrid")
# https://www.statology.org/pandas-filter-multiple-conditions/
data = df[(df['Page_count'] >= 61) & (df['Source_count'] >= 270)]
y="Page"
x="time_second"

title = ""
plt.figure(figsize=(10, 20))
sns.set(font_scale=1.25)
sns.boxplot(x=x, y=y, data=data, notch=False, showmeans=True,
           meanprops={"marker":"s","markerfacecolor":"white", "markeredgecolor":"black"},
            hue='Source / Medium')
plt.title(title)
plt.ioff()_____no_output_____sns.set_style("whitegrid")
# https://www.statology.org/pandas-filter-multiple-conditions/
data = df[(df['Page_count'] >= 61) & (df['Source_count'] >= 270)]
data = data[data['time_second'] > 5]
y="Page"
x="time_second"

title = ""
plt.figure(figsize=(10, 20))
sns.set(font_scale=1.25)
sns.boxplot(x=x, y=y, data=data, notch=False, showmeans=True,
           meanprops={"marker":"s","markerfacecolor":"white", "markeredgecolor":"black"},
            hue='Source / Medium')
plt.title(title)
plt.ioff()_____no_output_____sns.set_style("whitegrid")
# https://www.statology.org/pandas-filter-multiple-conditions/
#data = df[(df['Page_count'] >= 61) & (df['Source_count'] >= 270)]
data = df[df['Source_count'] >= 200]
data = data[data['time_second'] > 5]
y="Source / Medium"
x="time_second"

title = ""
plt.figure(figsize=(10, 20))
sns.set(font_scale=1.25)
sns.boxplot(x=x, y=y, data=data, notch=False, showmeans=True,
           meanprops={"marker":"s","markerfacecolor":"white", "markeredgecolor":"black"},
            hue='Source / Medium')
plt.title(title)
plt.ioff()_____no_output_____# https://www.statology.org/pandas-filter-multiple-conditions/
data = df[(df['Page_count'] >= 61) & (df['Source_count'] >= 270)]
data = data[data['time_second'] > 5]
y="Page Title"
x="time_second"

title = ""
plt.figure(figsize=(10, 20))
sns.set(font_scale=1.25)
sns.boxplot(x=x, y=y, data=data, notch=False, showmeans=True,
           meanprops={"marker":"s","markerfacecolor":"white", "markeredgecolor":"black"},
            hue='Source / Medium')
plt.title(title)
plt.ioff()_____no_output_____# https://www.statology.org/pandas-filter-multiple-conditions/
data = df[(df['Page_count'] >= 61) & (df['Source_count'] >= 270)]
sns.displot(data, x="time_second", kind="kde")
plt.ioff()_____no_output_____# https://www.statology.org/pandas-filter-multiple-conditions/
sns.displot(df, x="time_second", kind="kde")
plt.ioff()_____no_output_____# https://www.statology.org/pandas-filter-multiple-conditions/
data = df[(df['Source_count'] >= 270)]
sns.displot(data, x="time_second", kind="kde", hue='Source / Medium')
plt.ioff()_____no_output_____# https://www.statology.org/pandas-filter-multiple-conditions/
data = df[(df['Source_count'] >= 270) & (df['time_second'] >= 5)]
sns.displot(data, x="time_second", kind="kde", hue='Source / Medium')
plt.ioff()_____no_output_____

1.   Plot PageView by date
2.   Plot Avg. Time on Page by source
3.   PageView by source (all vs time in second more than 5)

Combine all Facebooks referal before ploting

_____no_output_____

# Export table
df.to_excel('/content/drive/MyDrive/DSS/BrainPost_blog/Results/df_final.xlsx')_____no_output_____


# Notebook from mwcraig/astropy-workshop
Path: 04-Coordinates/astropy_coordinates.ipynb

# An Overview of `astropy.coordinates`:

### Matching catalogs, planning observations, and handling stellar velocities_____no_output_____Astronomers use a wide variety of coordinate systems and formats to represent sky coordinates of celestial objects. For example, you may be familiar with terms like "right ascension" and "declination" or "galactic latitude/longitude," and you may have seen angular coordinate components represented as "0h39m15.9s," "00:39:15.9," or 9.81625º. `astropy.coordinates` provides tools for representing the coordinates of objects and transforming them between different systems.

In this tutorial, we will explore how the `astropy.coordinates` package and related Astropy functionality can be used to help in planning observations or other exercises focused on large coordinate catalogs.

You may find it helpful to keep [the Astropy documentation for the coordinates package](http://docs.astropy.org/en/stable/coordinates/index.html) open alongside this tutorial for reference or additional reading. In the text below, you may also see some links that look like ([docs](http://docs.astropy.org/en/stable/coordinates/index.html)). These links will take you to parts of the documentation that are directly relevant to the cells from which they link.

Some of the exercises below are quite complex to complete. For this reason, we provide a possible solution to each one (although there are many possible right answers!). Also note that there is likely not enough time to complete *all* of the exercises in the time we have. So if you're stuck on or disinterested in an exercise, we encourage you to skip over it and come back only if you have time later._____no_output_____### Preliminary imports

We start by importing some general packages we will need below:_____no_output_____

%matplotlib inline
from matplotlib import pyplot as plt
import numpy as np_____no_output_____

Note: If the next cell fails, make sure you have installed `astroquery` by running:
```
pip install astroquery
```
in your terminal. If so, you will need to go to "Kernel->Restart" in this notebook after installing._____no_output_____

from astropy import units as u
from astropy.coordinates import SkyCoord, Distance
from astropy.io import fits
from astropy.table import QTable
from astropy.utils.data import download_file_____no_output_____

## Representing on-sky locations with `astropy.coordinates`

In Astropy, the most common way of representing and working with sky coordinates is to use the `SkyCoord` object. A `SkyCoord` can be created directly from angles or arrays of angles with associated units, as demonstrated below.

To get started, let's assume that we want to create a `SkyCoord` object for the center of the open cluster NGC 188 so that later we can query and retrieve stars that might be members of the cluster. Let's also assume, for now, that we already know the sky coordinates of the cluster to be (12.11, 85.26) degrees in the ICRS coordinate frame. The ICRS — sometimes referred to as "equitorial" or "J2000" coordinates — is currently the most common astronomical coordinate frame for stellar or extragalactic astronomy, and is the default coordinate frame for `SkyCoord`. We could therefore create a `SkyCoord` object for the center of NGC 188 as:_____no_output_____

ngc188_center = SkyCoord(12.11*u.deg, 85.26*u.deg)
ngc188_center_____no_output_____

It is generally recommended to explicitly specify the frame your coordinates are in. In this case, because ICRS is the default, this would be an equivalent way of creating our `SkyCoord` object:_____no_output_____

ngc188_center = SkyCoord(12.11*u.deg, 85.26*u.deg, frame='icrs')
ngc188_center_____no_output_____

As we will see below, there are many other supported coordinate frames.

In the above initializations, we passed in `astropy.units.Quantity` objects with angular units to specify the angular components of our sky coordinates. However, `SkyCoord` will also accept string-formatted coordinates either as separate strings for Right Ascension (RA) and Declination (Dec) or a single string:_____no_output_____

SkyCoord('00h48m26.4s', '85d15m36s', frame='icrs')_____no_output_____

Some string representations do not explicitly define units, so you would have to specify the units explicitly if they are not included in the string:_____no_output_____

SkyCoord('00:48:26.4 85:15:36', unit=(u.hour, u.deg),
         frame='icrs')_____no_output_____

For more information and examples on initializing `SkyCoord` objects, [see this documentation](http://docs.astropy.org/en/latest/coordinates/skycoord.html)._____no_output_____For the initializations above, we assumed that we already had the coordinate component values ready. If you do not know the coordinate values, and the object you are interested in is in [SESAME](http://cdsweb.u-strasbg.fr/cgi-bin/Sesame), you can also automatically look up and load coordinate values from its name using the `SkyCoord.from_name()` class method1 ([docs](http://docs.astropy.org/en/latest/coordinates/index.html#convenience-methods)). Note, however, that this requires an internet connection. It is safe to skip this cell if you are not connected to the internet because we already defined it above.

 1If you don't know what a class method is, think of it like an alternative constructor for a `SkyCoord` object — calling `SkyCoord.from_name()` with a name gives you a new `SkyCoord` object. For more detailed background on what class methods are and when they're useful, see [this page](https://julien.danjou.info/blog/2013/guide-python-static-class-abstract-methods)._____no_output_____

# NOTE: skip this cell if you do not have an internet connection

ngc188_center = SkyCoord.from_name('NGC 188')
ngc188_center_____no_output_____

The `SkyCoord` object we created has various ways of accessing the information contained within it. All `SkyCoord` objects have attributes that allow you to retrieve the coordinate component data, but the component names will change depending on the coordinate frame of the `SkyCoord` you have. In our case, we have a `SkyCoord` with ICRS coordinates: The component names are therefore lower-case abbreviations of Right Ascension, `.ra`, and Declination, `.dec`:_____no_output_____

ngc188_center.ra, ngc188_center.dec_____no_output_____

These ``ra`` and ``dec`` attributes return specialized [``Quantity``](http://docs.astropy.org/en/stable/units/index.html) objects (actually, a subclass called [``Angle``](http://docs.astropy.org/en/stable/api/astropy.coordinates.Angle.html), which in turn is subclassed by [``Latitude``](http://docs.astropy.org/en/stable/api/astropy.coordinates.Latitude.html) and [``Longitude``](http://docs.astropy.org/en/stable/api/astropy.coordinates.Longitude.html)).  These objects store angles and provide pretty representations of those angles, as well as some useful attributes to quickly convert to common angle units. For example, in a Jupyter notebook, these objects know how to represent themselves using LaTeX:_____no_output_____

ngc188_center.ra_____no_output_____ngc188_center.dec_____no_output_____

We can also retrieve the component values in different units:_____no_output_____

ngc188_center.ra.hour, ngc188_center.ra.radian, ngc188_center.ra.degree_____no_output_____

And format the values into strings with specified units ([docs](http://docs.astropy.org/en/latest/coordinates/formatting.html)), for example:_____no_output_____

ngc188_center.ra.to_string(unit=u.hourangle, sep=':')_____no_output_____

Now that we have a `SkyCoord` object for the center of NGC 188, we can select sources from the *Gaia* Data Release 2 catalog around this position to try to find stars that might be members of the cluster. To do this, we will use the `astroquery.gaia` module to query the *Gaia* data archive ([docs](https://astroquery.readthedocs.io/en/latest/gaia/gaia.html)).

This requires an internet connection, but if it fails, the catalog file is included in the repository so you can load it locally (i.e., skip the next cell if you do not have an internet connection):_____no_output_____

from astroquery.gaia import Gaia_____no_output_____# NOTE: skip this cell if you do not have an internet connection

job = Gaia.cone_search_async(ngc188_center, radius=0.5*u.deg)
table = job.get_results()

# only keep stars brighter than G=19 magnitude
table = table[table['phot_g_mean_mag'] < 19*u.mag]_____no_output_____

The above cell may not work if you do not have an internet connection, so we have included the results table along with the notebook:_____no_output_____

table = QTable.read('gaia_results.fits')_____no_output_____len(table)_____no_output_____

This table now contains about 5000 stars from *Gaia* DR2 around the coordinate position of the center of NGC 188. Let's now construct a `SkyCoord` object with the results table. In the *Gaia* data archive, the ICRS coordinates of a source are given as column names `"ra"` and `"dec"`:_____no_output_____

table['ra']_____no_output_____table['dec']_____no_output_____

Note that, because the *Gaia* archive provides data tables with associated units, and we read this table with the `QTable` object ([docs](http://docs.astropy.org/en/latest/table/mixin_columns.html#quantity-and-qtable)), the above columns are represented as `Quantity` objects with units of degrees. Note also that these columns contain many (>5000!) coordinate values. We can pass these directly in to `SkyCoord` to get a single `SkyCoord` object to represent all of these coordinates:_____no_output_____

gaia_coords = SkyCoord(table['ra'], table['dec'])
gaia_coords_____no_output_____

### Exercises_____no_output_____Create a `SkyCoord` of the open cluster the Pleiades (either by looking up the coordinates and passing them in, or by using the convenience method we learned about above):_____no_output_____Using only a single method/function call on the `SkyCoord` object representing the center of NGC 188, print a string with the RA/Dec in the form 'HH:MM:SS.S DD:MM:SS.S'. Check your answer against [SIMBAD](http://simbad.u-strasbg.fr/simbad/), which will show you sexagesimal coordinates for the object.

(Hint: `SkyCoord.to_string()` might be useful.)_____no_output_____Using a single method/function call on the `SkyCoord` object containing the results of our *Gaia* query, compute the angular separation between each resulting star and the coordinates of the cluster center.

(Hint: `SkyCoord.separation()` might be useful.)_____no_output_____---

## More than just sky positions: Including distance information in `SkyCoord`

So far, we have only used `SkyCoord` to represent sky positions (i.e., `ra` and `dec` only). It is sometimes useful to include distance information with the sky coordinates, thereby fully specifying the 3D position of a given source. To pass in distance information, `SkyCoord` accepts a keyword argument `distance`. So, if we knew the distance to NGC 188, we could also pass in a distance (as a `Quantity` object) using this argument:_____no_output_____

ngc188_center_3d = SkyCoord(12.11*u.deg, 85.26*u.deg,
                            distance=1.96*u.kpc)_____no_output_____

With our results from querying the *Gaia* catalog, we are also given parallax measurements to each star in the catalog. The default way of passing in a distance, as above, is to pass in a `Quantity` with a unit of length. However, `astropy.coordinates` also provides a specialized object, `Distance`, for handling common transformations of different distance representations ([docs](http://docs.astropy.org/en/latest/coordinates/index.html#distance)). Among other things, this class supports passing in a parallax value (but note that the transformation to distance here inverts the parallax, which is not always a good idea!):_____no_output_____

Distance(parallax=1*u.mas)_____no_output_____

The catalog of stars we queried from *Gaia* contains parallax information, so we can also create a `SkyCoord` object to represent the 3D positions of all of the *Gaia* stars. Here, we sub-select the table to return only stars with positive parallax values:_____no_output_____

table_3d = table[table['parallax'] > 0]

gaia_dist = Distance(parallax=table_3d['parallax'])
gaia_coords_3d = SkyCoord(table_3d['ra'], table_3d['dec'],
                          distance=gaia_dist)
gaia_coords_3d_____no_output_____

Now that we have 3D position information for both the cluster center, and for the stars we queried from *Gaia*, we can compute the 3D separation (distance) between all of the *Gaia* sources and the cluster center:_____no_output_____

sep3d = gaia_coords_3d.separation_3d(ngc188_center_3d)
sep3d_____no_output_____

### Exercises

Using the 3D separation values, define a boolean mask to select candidate members of the cluster. Select all stars within 50 pc of the cluster center. How many candidate members of NGC 188 do we have, based on their 3D positions?_____no_output_____---

## Cross-matching and comparing catalogs

Let's recap what we have done so far: We defined a `SkyCoord` object to represent the center of an open cluster (NGC 188), we queried the *Gaia* DR2 catalog to select stars that are close (on the sky) to the center of the cluster, and we used the parallax values from *Gaia* to select stars that are near NGC 188 in 3D position.

Imagine that we now want to cross-match our catalog of candidate members of NGC 188 to another catalog: Here we will cross-match the *Gaia* stars with the 2MASS photometric catalog to retrieve infrared magnitudes for these stars, and then plot a color-magnitude diagram. To do this, we first need to query the 2MASS catalog to retrieve all sources in a region around the center of NGC 188, as we did for *Gaia*. We will once again use `astroquery` to execute this query. This will again require an internet connection, but we have again included the results of this query in a file along with this notebook in case you are not connected to the internet.

To query 2MASS, we will use the `astroquery.vizier` module ([docs](https://astroquery.readthedocs.io/en/latest/vizier/vizier.html)):_____no_output_____

from astroquery.vizier import Vizier_____no_output_____# NOTE: skip this cell if you do not have an internet connection

# II/246 is the catalog name for the main 2MASS photometric catalog
v = Vizier(catalog="II/246")
v.ROW_LIMIT = -1

result = v.query_region(ngc188_center, radius=0.5*u.deg)
tmass_table = result[0]_____no_output_____tmass_table = QTable.read('2MASS_results.fits')_____no_output_____

As with the *Gaia* results table, we can now create a single `SkyCoord` object to represent all of the sources returned from our query to the 2MASS catalog. Let's look at the column names in this table by displaying the first few rows:_____no_output_____

tmass_table[:3]_____no_output_____

From looking at the column names, it looks like the two relevant sky coordinate columns are `RAJ2000` for `ra` and `DEJ2000` for `dec`:_____no_output_____

tmass_coords = SkyCoord(tmass_table['RAJ2000'],
                        tmass_table['DEJ2000'])
len(tmass_coords)_____no_output_____

We now want to cross-match our *Gaia*-selected candidate members of NGC 188 with this table of photometry from 2MASS. Let's first use the mask we defined above (in the exercise) to select members of NGC 188 based on 3D position:_____no_output_____

ngc188_3d_mask = sep3d < 50*u.pc
gaia_ngc188_members = gaia_coords_3d[ngc188_3d_mask]
gaia_ngc188_members_table = table_3d[ngc188_3d_mask]_____no_output_____

Now we can use the ``SkyCoord.match_to_catalog_sky`` method to match the two catalogs ([docs](http://docs.astropy.org/en/latest/coordinates/matchsep.html#astropy-coordinates-matching)). Note that order matters: Here we are matching 2MASS to *Gaia* because we ultimately want to make a color-magnitude diagram using the 2MASS photometry. `SkyCoord.match_to_catalog_sky` returns three objects: first, the indices into `gaia_ngc188_members` that get the closest matches in `tmass_coords`, second, the angular separation between each `tmass_coords` coordinate and the closest source in `gaia_ngc188_members`, and finally, the 3D distance between each `tmass_coords` coordinate and the closest source in `gaia_ngc188_members`. Here, the 3D distances will not be useful because the 2MASS coordinates do not have associated distance information:_____no_output_____

idx_tmass, d2d_tmass, d3d_tmass = tmass_coords.match_to_catalog_sky(gaia_ngc188_members)_____no_output_____

Let's now look at the distribution of separations for all of the cross-matched sources:_____no_output_____

plt.hist(d2d_tmass.arcsec, histtype='step',
         bins=np.logspace(-2, 2., 64))
plt.xlabel('separation [arcsec]')
plt.xscale('log')
plt.yscale('log')
plt.tight_layout()_____no_output_____

From this, it looks like the sources that actually match to our *Gaia* NGC 188 members are all within about 2 arcseconds, so that is promising! How many of the 2MASS sources matched to one of our NGC 188 candidate members?_____no_output_____

(d2d_tmass < 2*u.arcsec).sum()_____no_output_____

That is a large fraction of the 321 members we identified. Missing sources may be too faint for 2MASS to detect, so we do not expect all of the *Gaia* sources to be in 2MASS. With our cross-match done, we can now make 2MASS+Gaia color-magnitude diagrams of our candidate NGC 188 members:_____no_output_____

Jmag = tmass_table['Jmag'][d2d_tmass < 2*u.arcsec]
Gmag = gaia_ngc188_members_table['phot_g_mean_mag'][idx_tmass[d2d_tmass < 2*u.arcsec]]
Bmag = gaia_ngc188_members_table['phot_bp_mean_mag'][idx_tmass[d2d_tmass < 2*u.arcsec]]

fig, axes = plt.subplots(1, 2, figsize=(10, 5))

ax = axes[0]
ax.scatter(Gmag - Jmag, Gmag,
           marker='o', color='k',
           linewidth=0, alpha=0.5)
ax.set_xlabel('$G - J$')
ax.set_ylabel('$G$')
ax.set_xlim(0, 3)
ax.set_ylim(19, 10) # backwards because magnitudes!

ax = axes[1]
ax.scatter(Bmag - Gmag, Jmag,
           marker='o', color='k',
           linewidth=0, alpha=0.5)
ax.set_xlabel('$G_{BP} - G$')
ax.set_ylabel('$J$')
ax.set_xlim(0.2, 1)
ax.set_ylim(17, 8) # backwards because magnitudes!

fig.tight_layout()_____no_output_____

Those both look like color-magnitude diagrams of a main sequence + red giant branch of an intermediate-age stellar cluster, so it looks like our selection and cross-matching has worked!

For more on what matching options are available, check out the [separation and matching section of the Astropy documentation](https://astropy.readthedocs.io/en/stable/coordinates/matchsep.html). Or for more on what you can do with `SkyCoord`, see [its API documentation](http://astropy.readthedocs.org/en/stable/api/astropy.coordinates.SkyCoord.html)._____no_output_____### Exercises_____no_output_____*Note: skip this exercise if you do not have an internet connection.*

Using the coordinates of the center of the Pleiades you defined in the previous set of exercises, download a catalog of 2MASS sources within 1 degree of the center of the Pleiades. Make a color-magnitude diagram of J-H vs. J for all sources. Can you see the main sequence corresponding to the Pleiades?_____no_output_____*Note: skip this exercise if you do not have an internet connection.*

Again using the coordinates of the center of the Pleiades, now download all *Gaia* sources within 1 degree of the center of the Pleiades. Cross-match the 2MASS and *Gaia* sources, and make a G-J vs. G color-magnitude diagram of all sources within 200 parsecs of the Sun._____no_output_____---

## Transforming between coordinate systems and planning observations

Imagine that we found something exciting in our exploration of the data for NGC 188, and we would like to follow-up with our own observations of a given sources. `astropy.coordinates` provides tools to assist in planning observations, especially from ground-based observatories, as we will see below. _____no_output_____### Introducting frame transformations_____no_output_____To understand the code in this section, it may help to read over the [overview of the astropy coordinates scheme](http://astropy.readthedocs.org/en/stable/coordinates/index.html#overview-of-astropy-coordinates-concepts).  The key concept to understand is that all coordinates in Astropy are in particular reference "frames," and we can transform between a specific `SkyCoord` object in one frame to another. For example, we can transform our previously-defined center of NGC 188 from ICRS to another coordinate frame, like Galactic coordinates. In `astropy.coordinates`, this is typically done using the `.transform_to()` method:_____no_output_____

ngc188_center.transform_to('galactic')_____no_output_____

This method either takes a string name of a coordinate frame, or an instance of a coordinate frame object itself. For example, the class representing the Galactic frame is:_____no_output_____

from astropy.coordinates import Galactic
ngc188_center.transform_to(Galactic())_____no_output_____

For coordinate frames that do not require additional parameters, the transformations can also be accessed by attributes of the `SkyCoord` object. Another way to transform to the Galactic frame, then, is to do:_____no_output_____

ngc188_center.galactic_____no_output_____

Note that changing frames also changes some of the attributes of the object, most often the names of the component attributes. Recall that for the ICRS frame the components were `ra` and `dec`. For the Galactic frame, the corresponding longitude and latitude are named `l` and `b`:_____no_output_____

ngc188_gal = ngc188_center.galactic
ngc188_gal.l, ngc188_gal.b_____no_output_____

Note that the names `.ra` and `.dec` will therefore *not* work on this transformed `SkyCoord`:_____no_output_____

# THIS SHOULD ERROR:
ngc188_gal.ra_____no_output_____

### Using frame transformations to get to AltAz

To determine whether a target is observable from a given observatory on Earth, we need to convert our coordinate to a frame local to an on-earth observer. The most common choice for such a frame is horizontal coordinates, or "AltAz" coordinates. The `AltAz` coordinate frame requires a few extra pieces of information to define: a location on Earth, and the time of observation. These are handled with the `EarthLocation` and `Time` objects, respectively:_____no_output_____

from astropy.coordinates import EarthLocation
from astropy.time import Time_____no_output_____

Let's assume we will be observing at Kitt Peak in a few weeks. We can retrieve the Earth location of Kitt Peak using the `EarthLocation.of_site` method:_____no_output_____

observing_location = EarthLocation.of_site('Kitt Peak')
# List of all locations can be viewed using
# EarthLocation.get_site_names()_____no_output_____

And let's pretend our observing run starts on Jan 20 this year:_____no_output_____

observing_time = Time('2020-01-20 1:00')  # 1am UTC=6pm AZ mountain time_____no_output_____

Now we use these to create an `AltAz` frame object. Note that this frame has even more information about the atmosphere, which can be used to correct for atmospheric refraction. Here we leave that alone, because the default is to ignore this effect (by setting the air pressure to 0)._____no_output_____

from astropy.coordinates import AltAz

altaz = AltAz(location=observing_location, obstime=observing_time)
altaz_____no_output_____

Now we can transform our ICRS `SkyCoord` to `AltAz` to get the location in the sky over Kitt Peak at the requested time:_____no_output_____

ngc188_center.transform_to(altaz)_____no_output_____

Alright, the cluster NGC 188 is up at 6PM local time at Kitt Peak, but that is pretty early to be observing. We could try various times one at a time to see if the airmass is at a darker time, but we can do better — we can make an airmass plot for the whole night:_____no_output_____

# this gives a Time object with an *array* of times
delta_hours = np.linspace(0, 6, 100)*u.hour
full_night_times = observing_time + delta_hours
full_night_altaz = AltAz(location=observing_location, obstime=full_night_times)
full_night_altaz_coords = ngc188_center.transform_to(full_night_altaz)

plt.plot(delta_hours, full_night_altaz_coords.secz)
plt.xlabel('Hours from 6PM AZ time')
plt.ylabel('Airmass [Sec(z)]')
plt.ylim(0.9, 3)
plt.tight_layout()_____no_output_____

It looks like our source is observable at relatively high airmass for most of the night, but the minimum airmass is at the start of the night. When can we start observing with proper dark skies? Astropy also provides a ``get_sun`` function that can be used to check the sky position of the sun to check, for example, when we are in 18-degree twilight:_____no_output_____

from astropy.coordinates import get_sun

full_night_sun_coos = get_sun(full_night_times).transform_to(full_night_altaz)
plt.plot(delta_hours, full_night_sun_coos.alt.deg)
plt.axhline(-18, color='k')
plt.xlabel('Hours from 6PM AZ time')
plt.ylabel('Sun altitude')
plt.tight_layout()_____no_output_____

It looks like it's just below -18 degrees a little after 7PM local time, so we can start observing our target then!_____no_output_____### Exercises_____no_output_____Try converting the NGC 188 center coordinates to an equatorial frame at some other equinox a while in the past (e.g., J1900). Do you see the precession of the equinoxes?

Hint: To see a diagram of the supported frames, see [here](http://docs.astropy.org/en/stable/coordinates/#module-astropy.coordinates). `FK5` has a parameter called 'equinox' — that looks promising!_____no_output_____What time is Sunset on our observing night? Try solving for the time at which the Sun's altitude is closest to 0 (i.e., do not just eyeball the plot, like we did above)._____no_output_____---

## More than just sky positions: Including velocity information in `SkyCoord`

As we have seen above, the `SkyCoord` object can store scalars or arrays of positional coordinate information, can transform between different coordinate frames, and can cross-match sets of coordinates. But wait, there's more! `astropy.coordinates` also supports representing and transforming *velocity* information along with positions ([docs](http://docs.astropy.org/en/latest/coordinates/velocities.html)).

### Passing velocity data in to `SkyCoord`

Velocity data is passed in to `SkyCoord` in a very similar way to the positional coordinates demonstrated above. However, velocity data must be passed in using keyword arguments, meaning that you must write out the velocity component names, for example:_____no_output_____

SkyCoord(ra=10*u.deg,
         dec=20*u.deg,
         pm_ra_cosdec=1*u.mas/u.yr,
         pm_dec=2*u.mas/u.yr)_____no_output_____

In the above, we create a `SkyCoord` instance with sky position and proper motion data. The proper motion component names for ICRS coordinates are, by default, `pm_ra_cosdec` and `pm_dec` ("pm" for "proper motion"). You can also specify radial velocity data via the `radial_velocity` argument, for example:_____no_output_____

velocity_coord = SkyCoord(ra=10*u.deg,
                          dec=20*u.deg,
                          pm_ra_cosdec=1*u.mas/u.yr,
                          pm_dec=2*u.mas/u.yr,
                          radial_velocity=100*u.km/u.s)
velocity_coord_____no_output_____

A `SkyCoord` object with velocity data can be transformed to other frames just like the objects we are used to from above:_____no_output_____

velocity_coord.transform_to('galactic')_____no_output_____

An important caveat, however, is that some reference frames require knowing distance in order to transform all velocity components — something we did not specify in the `velocity_coord` definition above:_____no_output_____

# THIS SHOULD ERROR:
velocity_coord.transform_to(altaz)_____no_output_____

### Evolving coordinate positions between epochs

For nearby or fast-moving stars, a star's position could change appreciably between two well-spaced observations of the source. For such cases, it might be necessary to evolve the position of the star using the proper motion or velocity of the star in order to accurately predict its position at a new time. Below, we will demonstrate this using data from the Digitized Sky Survey (DSS; digital scans of photographic plates observed in the 1950s) and data from the *Gaia* mission (at epoch J2015.5).

The star HD 219829 has a very large proper motion, close to 0.5 arcsec/year. Between the DSS and *Gaia*, we therefore expect that the position of the star has changed by about 0.5 arcmin; let's see if this is the case! To start, we will query the *Gaia* catalog to retrieve data for this star (but skip the cell below if you do not have an internet connection):_____no_output_____

# NOTE: skip this cell if you do not have an internet connection

hd219829_coord_2015 = SkyCoord(349.72896715675495, 5.405115847648341, unit=u.deg)
hd219829_gaia_table = Gaia.query_object_async(hd219829_coord_2015, radius=2*u.arcsec)
hd219829_gaia_table = QTable(hd219829_gaia_table)  # ignore any warnings output by this!

hd219829_coord = SkyCoord(ra=hd219829_gaia_table['ra'],
                          dec=hd219829_gaia_table['dec'],
                          distance=Distance(parallax=hd219829_gaia_table['parallax']),
                          pm_ra_cosdec=hd219829_gaia_table['pmra'],
                          pm_dec=hd219829_gaia_table['pmdec'],
                          radial_velocity=hd219829_gaia_table['radial_velocity'],
                          obstime=Time('J2015.5'))_____no_output_____hd219829_coord = SkyCoord(ra=349.72896716*u.deg,
                          dec=5.40511585*u.deg,
                          distance=34.47896*u.pc,
                          pm_ra_cosdec=483.41659*u.mas/u.yr,
                          pm_dec=-114.86339*u.mas/u.yr,
                          obstime=Time('J2015.5'))_____no_output_____

Note above that we specify a new keyword argument, `obstime`, to `SkyCoord`: This is the observation time of the coordinates, and will be important later when we evolve the position of this star using its proper motion.

We now have the position and proper motion for HD 219829. Let's now query the DSS to retrieve a FITS image of the field around this star, using the STSCI DSS image cutout service. Again, skip the cell below if you do not have an internet connection:_____no_output_____

# NOTE: skip this cell if you do not have an internet connection

dss_cutout_filename = download_file(f"http://archive.stsci.edu/cgi-bin/dss_search?"
                                    f"f=FITS&ra={hd219829_coord.ra.degree}&dec={hd219829_coord.dec.degree}"
                                    f"&width=4&height=4")  # width/height in arcmin_____no_output_____dss_cutout_filename = 'dss_hd219829.fits'_____no_output_____

We can now load the FITS image of the cutout and use `astropy.visualization` to display the image using its World Coordinate System (WCS) info ([docs](http://docs.astropy.org/en/latest/visualization/wcsaxes/index.html)). By passing in the WCS information (included in the FITS cutout header), we can over-plot a marker for the *Gaia*-measured sky position of HD 219829:_____no_output_____

from astropy.wcs import WCS

hdu = fits.open(dss_cutout_filename)[0]
wcs = WCS(hdu.header)

fig, ax = plt.subplots(1, 1, figsize=(8, 8),
                       subplot_kw=dict(projection=wcs))
ax.imshow(hdu.data, origin='lower', cmap='Greys_r')
ax.set_xlabel('RA')
ax.set_ylabel('Dec')
ax.set_autoscale_on(False)

ax.scatter(hd219829_coord.ra.degree,
           hd219829_coord.dec.degree,
           s=500,
           transform=ax.get_transform('world'),
           facecolor='none', linewidth=2, color='tab:red')_____no_output_____

The bright star (as observed by DSS) is our target, and the red circle is where *Gaia* observed this star — it has moved! But, we can account for this, and predict the position of the star at around the time the DSS plate was observed (let's assume it was exactly 1950 — this is not strictly correct, but should be good enough).

To account for the proper motion of the source and evolve the position to a new time, we can use the `SkyCoord.apply_space_motion()` method ([docs](http://docs.astropy.org/en/latest/api/astropy.coordinates.SkyCoord.html#astropy.coordinates.SkyCoord.apply_space_motion)):_____no_output_____

hd219829_coord_1950 = hd219829_coord.apply_space_motion(new_obstime=Time('J1950'))  # ignore the ErfaWarning_____no_output_____fig, ax = plt.subplots(1, 1, figsize=(8, 8),
                       subplot_kw=dict(projection=wcs))
ax.imshow(hdu.data, origin='lower', cmap='Greys_r')
ax.set_xlabel('RA')
ax.set_ylabel('Dec')
ax.set_autoscale_on(False)

ax.scatter(hd219829_coord.ra.degree,
           hd219829_coord.dec.degree,
           s=500,
           transform=ax.get_transform('world'),
           facecolor='none', linewidth=2, color='tab:red')

# Plot the predicted (past) position:
ax.scatter(hd219829_coord_1950.ra.degree,
           hd219829_coord_1950.dec.degree,
           s=500,
           transform=ax.get_transform('world'),
           facecolor='none', linewidth=2, color='tab:blue')_____no_output_____

The predicted position of the star from 1950 is much closer to where the star is in the DSS image. Success!_____no_output_____### Exercises_____no_output_____The *Gaia* data table we downloaded for sources around NGC 188 also contains proper motion information. Do any of the sources in the field around NGC 188 have large enough proper motions that we might detect a difference between the DSS position of a star and the *Gaia* position? How many stars have an apparent change in position > 5 arcsec over 60 years?_____no_output_____Download a DSS image for the region around the fastest-moving star in the NGC 188 field. By accounting for the proper motion, can you predict where the star is in the DSS image using only the *Gaia* position and proper motion?_____no_output_____## Wrap-up

This tutorial covers a lot of material, but `astropy.coordinates` has even more functionality that we were unable to cover in this workshop. For documentation on other features of `astropy.coordinates`, check out [the astropy.coordinates section of the Astropy documentation](http://astropy.readthedocs.org/en/stable/coordinates/index.html).

You might also be interested in [the astroplan affiliated package](http://astroplan.readthedocs.org/), which uses the `astropy.coordinates` to do more advanced versions of the observation planning tasks briefly covered above._____no_output_____


# Notebook from stuppie/CM7_CM1E2d56col_unenr123_rawextract_2017
Path: scripts/topGO_rag_wt.ipynb

## Running topGO_____no_output_____

require("topGO")
library(genefilter)
setwd("/home/gstupp/projects/Wolan/cmoon/CM7_CM1E2d56col_unenr123_rawextract_2017/")Loading required package: topGO
Loading required package: BiocGenerics
Loading required package: parallel

Attaching package: ‘BiocGenerics’

The following objects are masked from ‘package:parallel’:

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

The following objects are masked from ‘package:stats’:

    IQR, mad, xtabs

The following objects are masked from ‘package:base’:

    anyDuplicated, append, as.data.frame, as.vector, cbind, colnames,
    do.call, duplicated, eval, evalq, Filter, Find, get, grep, grepl,
    intersect, is.unsorted, lapply, lengths, Map, mapply, match, mget,
    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
    union, unique, unlist, unsplit

Loading required package: graph
Loading required package: Biobase
Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

Loading required package: GO.db
Loading required package: AnnotationDbi
Loading required package: stats4
Loading required package: IRanges
Loading required package: S4Vectors

Loading required package: SparseM

Attaching package: ‘SparseM’

The following object is masked from ‘package:base’:

    backsolve

Warning message:
“replacing previous import by ‘graph::.__C__dist’ when loading ‘topGO’”# load in the deseq results
load("out/res_rag_wt")_____no_output_____# load in protein cluster ID -> GO terms mapping
geneID2GO <- readMappings("out/geneid2go.map")
head(geneID2GO)_____no_output_____# load in the count data
load("out/dds_RTcontrol")
# filter out genes with low expression value as well as with very small variability across samples.
selProt <- genefilter(counts(dds), filterfun(kOverA(3, 6), function(x) (IQR(x) > 0.25)))
sum(selProt)
dim(counts(dds))
res=res[selProt,]_____no_output_____

## human_mouse proteins only_____no_output_____

# human_mouse proteins only
res_hm = res[res$human_mouse=="True",]

geneList = res_hm$padj
names(geneList)<-rownames(res_hm)_____no_output_____# Tests based on gene counts
# universe is all proteins observed
# "sig" proteins are ones with a (adjusted) p-value < 0.01
topDiffGenes <- function(allScore) { return(allScore < 0.01)}
GOdata <- new("topGOdata", ontology = "BP", allGenes = geneList, geneSel = topDiffGenes, annot = annFUN.gene2GO, gene2GO = geneID2GO)
resultWeight01 <- runTest(GOdata, statistic = "fisher")
# "sig" proteins are determined by rank
resultWeight01ks <- runTest(GOdata, statistic = "ks")
Building most specific GOs .....	( 50 GO terms found. )

Build GO DAG topology ..........	( 292 GO terms and 557 relations. )

Annotating nodes ...............	( 73 genes annotated to the GO terms. )

			 -- Weight01 Algorithm --

		 the algorithm is scoring 16 nontrivial nodes
		 parameters:
			 test statistic:  fisher

	 Level 7:	1 nodes to be scored	(0 eliminated genes)

	 Level 6:	2 nodes to be scored	(0 eliminated genes)

	 Level 5:	2 nodes to be scored	(2 eliminated genes)

	 Level 4:	3 nodes to be scored	(25 eliminated genes)

	 Level 3:	4 nodes to be scored	(39 eliminated genes)

	 Level 2:	3 nodes to be scored	(49 eliminated genes)

	 Level 1:	1 nodes to be scored	(57 eliminated genes)

			 -- Weight01 Algorithm --

		 the algorithm is scoring 292 nontrivial nodes
		 parameters:
			 test statistic:  ks
			 score order:  increasing

	 Level 13:	1 nodes to be scored	(0 eliminated genes)

	 Level 12:	4 nodes to be scored	(0 eliminated genes)

	 Level 11:	12 nodes to be scored	(1 eliminated genes)

	 Level 10:	23 nodes to be scored	(2 eliminated genes)

	 Level 9:	31 nodes to be scored	(8 eliminated genes)

	 Level 8:	27 nodes to be scored	(10 eliminated genes)

	 Level 7:	26 nodes to be scored	(15 eliminated genes)

	 Level 6:	39 nodes to be scored	(22 eliminated genes)

	 Level 5:	53 nodes to be scored	(28 eliminated genes)

	 Level 4:	41 nodes to be scored	(56 eliminated genes)

	 Level 3:	23 nodes to be scored	(60 eliminated genes)

	 Level 2:	11 nodes to be scored	(69 eliminated genes)

	 Level 1:	1 nodes to be scored	(71 eliminated genes)
GenTable(GOdata, resultWeight01, topNodes = 10)_____no_output_____GenTable(GOdata, resultWeight01ks, topNodes = 10)_____no_output_____# same thing using GO MF
GOdataMF <- new("topGOdata", ontology = "MF", allGenes = geneList, geneSel = topDiffGenes, annot = annFUN.gene2GO, gene2GO = geneID2GO)
resultWeight01MF <- runTest(GOdataMF, statistic = "fisher")
resultWeight01ksMF <- runTest(GOdataMF, statistic = "ks")
Building most specific GOs .....	( 82 GO terms found. )

Build GO DAG topology ..........	( 205 GO terms and 267 relations. )

Annotating nodes ...............	( 101 genes annotated to the GO terms. )

			 -- Weight01 Algorithm --

		 the algorithm is scoring 21 nontrivial nodes
		 parameters:
			 test statistic:  fisher

	 Level 7:	2 nodes to be scored	(0 eliminated genes)

	 Level 6:	3 nodes to be scored	(0 eliminated genes)

	 Level 5:	3 nodes to be scored	(17 eliminated genes)

	 Level 4:	4 nodes to be scored	(32 eliminated genes)

	 Level 3:	6 nodes to be scored	(48 eliminated genes)

	 Level 2:	2 nodes to be scored	(51 eliminated genes)

	 Level 1:	1 nodes to be scored	(77 eliminated genes)

			 -- Weight01 Algorithm --

		 the algorithm is scoring 205 nontrivial nodes
		 parameters:
			 test statistic:  ks
			 score order:  increasing

	 Level 14:	1 nodes to be scored	(0 eliminated genes)

	 Level 13:	2 nodes to be scored	(0 eliminated genes)

	 Level 12:	1 nodes to be scored	(1 eliminated genes)

	 Level 11:	1 nodes to be scored	(1 eliminated genes)

	 Level 10:	2 nodes to be scored	(1 eliminated genes)

	 Level 9:	6 nodes to be scored	(1 eliminated genes)

	 Level 8:	15 nodes to be scored	(3 eliminated genes)

	 Level 7:	26 nodes to be scored	(8 eliminated genes)

	 Level 6:	39 nodes to be scored	(17 eliminated genes)

	 Level 5:	43 nodes to be scored	(48 eliminated genes)

	 Level 4:	37 nodes to be scored	(67 eliminated genes)

	 Level 3:	25 nodes to be scored	(75 eliminated genes)

	 Level 2:	6 nodes to be scored	(81 eliminated genes)

	 Level 1:	1 nodes to be scored	(99 eliminated genes)
GenTable(GOdataMF, resultWeight01MF, topNodes = 10)_____no_output_____GenTable(GOdataMF, resultWeight01ksMF, topNodes = 10)_____no_output_____

## Non -human/mouse_____no_output_____

res_nh = res[res$human_mouse=="False",]

geneList = res_nh$padj
names(geneList)<-rownames(res_nh)
length(geneList)_____no_output_____GOdataBP <- new("topGOdata", ontology = "BP", allGenes = geneList, geneSel = topDiffGenes, annot = annFUN.gene2GO, gene2GO = geneID2GO)
resultWeight01BP <- runTest(GOdataBP, statistic = "fisher")
resultWeight01ksBP <- runTest(GOdataBP, statistic = "ks")
Building most specific GOs .....	( 170 GO terms found. )

Build GO DAG topology ..........	( 546 GO terms and 1107 relations. )

Annotating nodes ...............	( 3558 genes annotated to the GO terms. )

			 -- Weight01 Algorithm --

		 the algorithm is scoring 226 nontrivial nodes
		 parameters:
			 test statistic:  fisher

	 Level 13:	1 nodes to be scored	(0 eliminated genes)

	 Level 12:	2 nodes to be scored	(0 eliminated genes)

	 Level 11:	6 nodes to be scored	(259 eliminated genes)

	 Level 10:	14 nodes to be scored	(262 eliminated genes)

	 Level 9:	23 nodes to be scored	(381 eliminated genes)

	 Level 8:	23 nodes to be scored	(904 eliminated genes)

	 Level 7:	21 nodes to be scored	(1398 eliminated genes)

	 Level 6:	37 nodes to be scored	(2317 eliminated genes)

	 Level 5:	44 nodes to be scored	(2437 eliminated genes)

	 Level 4:	29 nodes to be scored	(2571 eliminated genes)

	 Level 3:	17 nodes to be scored	(2663 eliminated genes)

	 Level 2:	8 nodes to be scored	(3348 eliminated genes)

	 Level 1:	1 nodes to be scored	(3414 eliminated genes)

			 -- Weight01 Algorithm --

		 the algorithm is scoring 546 nontrivial nodes
		 parameters:
			 test statistic:  ks
			 score order:  increasing

	 Level 13:	6 nodes to be scored	(0 eliminated genes)

	 Level 12:	31 nodes to be scored	(0 eliminated genes)

	 Level 11:	41 nodes to be scored	(337 eliminated genes)

	 Level 10:	48 nodes to be scored	(488 eliminated genes)

	 Level 9:	61 nodes to be scored	(626 eliminated genes)

	 Level 8:	63 nodes to be scored	(1134 eliminated genes)

	 Level 7:	58 nodes to be scored	(1679 eliminated genes)

	 Level 6:	79 nodes to be scored	(2479 eliminated genes)

	 Level 5:	74 nodes to be scored	(2542 eliminated genes)

	 Level 4:	45 nodes to be scored	(2666 eliminated genes)

	 Level 3:	28 nodes to be scored	(2757 eliminated genes)

	 Level 2:	11 nodes to be scored	(3378 eliminated genes)

	 Level 1:	1 nodes to be scored	(3418 eliminated genes)
GenTable(GOdataBP, resultWeight01BP, topNodes = 10)_____no_output_____GenTable(GOdataBP, resultWeight01ksBP, topNodes = 10)_____no_output_____# Molecular function
GOdataMF <- new("topGOdata", ontology = "MF", allGenes = geneList, geneSel = topDiffGenes, annot = annFUN.gene2GO, gene2GO = geneID2GO)
resultWeight01MF <- runTest(GOdataMF, statistic = "fisher")
resultWeight01ksMF <- runTest(GOdataMF, statistic = "ks")
Building most specific GOs .....	( 280 GO terms found. )

Build GO DAG topology ..........	( 447 GO terms and 535 relations. )

Annotating nodes ...............	( 3889 genes annotated to the GO terms. )

			 -- Weight01 Algorithm --

		 the algorithm is scoring 160 nontrivial nodes
		 parameters:
			 test statistic:  fisher

	 Level 9:	2 nodes to be scored	(0 eliminated genes)

	 Level 8:	7 nodes to be scored	(0 eliminated genes)

	 Level 7:	19 nodes to be scored	(971 eliminated genes)

	 Level 6:	35 nodes to be scored	(977 eliminated genes)

	 Level 5:	41 nodes to be scored	(1632 eliminated genes)

	 Level 4:	30 nodes to be scored	(1996 eliminated genes)

	 Level 3:	19 nodes to be scored	(2890 eliminated genes)

	 Level 2:	6 nodes to be scored	(3022 eliminated genes)

	 Level 1:	1 nodes to be scored	(3515 eliminated genes)

			 -- Weight01 Algorithm --

		 the algorithm is scoring 447 nontrivial nodes
		 parameters:
			 test statistic:  ks
			 score order:  increasing

	 Level 15:	1 nodes to be scored	(0 eliminated genes)

	 Level 14:	2 nodes to be scored	(0 eliminated genes)

	 Level 13:	2 nodes to be scored	(3 eliminated genes)

	 Level 12:	1 nodes to be scored	(62 eliminated genes)

	 Level 11:	2 nodes to be scored	(62 eliminated genes)

	 Level 10:	2 nodes to be scored	(62 eliminated genes)

	 Level 9:	6 nodes to be scored	(63 eliminated genes)

	 Level 8:	19 nodes to be scored	(63 eliminated genes)

	 Level 7:	70 nodes to be scored	(992 eliminated genes)

	 Level 6:	147 nodes to be scored	(1075 eliminated genes)

	 Level 5:	98 nodes to be scored	(1874 eliminated genes)

	 Level 4:	56 nodes to be scored	(2495 eliminated genes)

	 Level 3:	31 nodes to be scored	(3092 eliminated genes)

	 Level 2:	9 nodes to be scored	(3193 eliminated genes)

	 Level 1:	1 nodes to be scored	(3523 eliminated genes)
GenTable(GOdataMF, resultWeight01MF, topNodes=10)_____no_output_____GenTable(GOdataMF, resultWeight01ksMF, topNodes = 10)_____no_output_____print(showGroupDensity(GOdataMF, "GO:0050242", ranks = TRUE))_____no_output_____