# Notebook from mganahl/Cirq
Path: docs/google/concepts.ipynb

##### Copyright 2020 The Cirq Developers_____no_output_____

#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License._____no_output_____

# Google Quantum Computing Service_____no_output_____

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

_____no_output_____

try:
    import cirq
except ImportError:
    print("installing cirq...")
    !pip install --quiet cirq
    print("installed cirq.")_____no_output_____

*Run quantum computing programs on Google’s quantum processors*

Quantum Computing Service gives customers access to Google's quantum computing hardware.  Programs that are written in Cirq, an open-source quantum computing program language, can be sent to run on a quantum computer in Google’s quantum computing lab in Santa Barbara, CA. You will be able to do  arbitrary single qubit rotations as well as several choices of two qubit gates.  Measurements of all qubits as a final step are also supported.

Access is currently only granted to those on an approved list.  Apply to become an early access partner with this [questionnaire](https://docs.google.com/forms/d/1DfUWu4zUAJ87GKy-ZoTHrFri5IwIteKtMxKfsy3lmHE).
_____no_output_____## Concepts

Quantum Computing Service uses Cirq, an open source Python framework for creating quantum programs and running them against simulators or against real quantum computers.    Here we provide a conceptual overview of how to run a quantum program on a quantum processor.

Installing Cirq:_____no_output_____### Circuits

The language of quantum computing is the quantum circuit model.  Cirq is our open source framework which allows one to write a quantum circuit model in Python.  To learn more about Cirq itself, we recommend starting with the [Tutorial](../tutorials/basics.ipynb).  Conceptually Cirq can be thought of as a way to create an object in Python that contains the information about a quantum circuit.  Here, for example, we create a single circuit made up of square root of NOT gates, controlled-Z gates, and a measurement:_____no_output_____

import cirq

q0, q1, q2 = cirq.LineQubit.range(3)
circuit = cirq.Circuit(
    (cirq.X ** 0.5).on_each(q0, q1, q2),
    cirq.CZ(q0, q1),
    cirq.CZ(q1, q2),
    cirq.measure(q2, key='m')
)
print(circuit)0: ───X^0.5───@────────────────
              │
1: ───X^0.5───@───@────────────
                  │
2: ───X^0.5───────@───M('m')───

### Quantum Engine API

Quantum Engine is the name of the cloud API which one can call to run the circuits you create in Cirq on simulator or on quantum computers.  When access is enabled, users can write code in Cirq that makes a call to Google’s servers to run a circuit that they have built in Cirq on simulators or on quantum computers.  Read more about how to do this using cirq's `Engine` class [here](./engine.ipynb).

![Quantum Engine Conceptual Diagram](../images/engine_diagram.png)_____no_output_____### Quantum Programs

In Cirq, one creates a `Circuit` in Python.  If one then wants to run it using Quantum Engine, one must then upload this Circuit (or Schedule) to the Quantum Engine.  The uploaded version of the Circuit/Schedule is called a Program.  Programs are not the Python code itself, but a representation of the Circuit in a hardware specific language.  Though you will need to make sure that your circuit uses only gates and qubits compatible with the hardware (see `cirq.google.optimized_for_sycamore()` to help with this), the `Engine` class and its corresponding `sampler` will translate the circuit to this serialized form for you.

![Quantum Program Conceptual Diagram](../images/engine_program.png)

When you upload a program to Quantum Engine, the program is given a name.  The organizational unit of Quantum Engine is the cloud project (see below), so when you create a program it is identified by the project id and the program id.  We write this as a path

```
<PROJECT_ID>/programs/<PROGRAM_ID>
```_____no_output_____### Cloud Project

Quantum Engine interfaces with Google Cloud.  In Google Cloud one can create different projects with different names.  For instance, you might create a project that is for your experiments on quantum computers, and you might create one to manage your side business running servers that day trade Star Wars memorabilia.  Everything under your cloud project can be seen by navigating to the Google Cloud console.  Here, for example, we see the programs that have been uploaded to a quantum projected called “Quantum Cloud Client”:

![Quantum Cloud Conceptual Diagram](../images/engine_cloud.png)

See the [Getting Started](../tutorials/google/start.ipynb) guide for step-by-step instructions on how to create a project and enable the API._____no_output_____### Jobs

Once a circuit has been uploaded as a program to Quantum Engine, you can run this program by creating a job.  When you create a job on Quantum Engine, the quantum engine then is responsible for routing your program and the information about the job to the appropriate simulator or quantum computer.  Note that there is a time limit for job length, so you will want to separate your programs into multiple jobs below this limit.

![Quantum Jobs Conceptual Diagram](../images/engine_job.png)

Quantum Engine is designed so that you can upload a program once and then, if that program has different parameters (see below), you can run this program multiple times by creating multiple jobs for a given program.  Jobs are associated with programs, so they have names which include everything about your program, plus a job id:

```
<project id>/programs/<program id>/jobs/<job id>
```_____no_output_____### Parameterized Circuits and Job Context

Circuits in Cirq can have quantum gates that are “parameterized”.  For example, a gate may represent a rotation of a qubit about the Z axis of the bloch sphere by an angle theta.  For parameterized circuit this angle theta is just specified by a variable, say “my-theta”.  In order to run such a circuit you will need to be able to tell the circuit what value of “my-theta” should be supplied.  In fact you can create a job for a program with parameterized gates which says run this for multiple parameter values.  Or each time you create a new job, you can specify new parameter values.  These parameter values are supplied in a “job context"._____no_output_____### Results

Once a job has been scheduled to run on a machine (simulator or quantum computer), the machine returns the results of the program to the Quantum Engine.  The Quantum Engine stores these results.  Users can then query for these results (typically one pulls every fraction of a second):

![Quantum Results Conceptual Diagram](../images/engine_result.png)

There is only one result for one job, so results can be identified via the identity of the job followed by result:

```
<project id>/programs/<program id>/jobs/<job id>/result
```_____no_output_____### Processors

When creating a job to run a program, one must specify where the job should run.  This can be a very specific machine to run on, or it can be specified more generally.  Each location that we can schedule a job to run on is described as a processor.  Different processors are available to different users, so processors appear under a project and have a processor id.

```
<project id>/processors/<processor id>
```

For example, one processor is the “rainbow” processor, which corresponds to a quantum computer located in Santa Barbara.  Another processor is “xmonsim” which is a fleet of simulators for the xmon architecture which all run in Google’s production datacenters.

Processor have state associated with them.  For instance, a processor can be “online” or they can be in “maintenance” mode._____no_output_____### Calibration Metrics

Processors that are quantum computers periodically undergo calibrations to maintain the quality of the programs that can be run on these processors.  During this calibration metrics about the performance of the quantum computer is collected.  This calibration data is stored by Quantum Engine and users can then query for the state of the calibration (one can ask for the latest calibration, or what the state of calibration was at a given point in time).  Calibrations are also available for past jobs.

Calibration metrics and can be retrieved via the console or via cirq calls.  See more details on the [Calibration Metrics](calibration.ipynb) page._____no_output_____### Reservations

Processor schedules are managed by a reservation system, assigning blocks of time as "time slots" with four possible states:

*  OPEN_SWIM: Anyone with processor access may run jobs.  There may be additional restrictions per processor that restrict large jobs.
*  MAINTENANCE: restrictions behave the same as during open swim, but there are no guarantees about the availability of the processor or quality of jobs run during this period.
*  RESERVATION: The processor is reserved for restricted use by a user or set of users. Reservations are rooted under a project and editors of that project may run jobs during that project's reservations while using any project they wish. Additional users may also be whitelisted to specific reservations.
*  UNALLOCATED: The processor has not been scheduled for any purpose. A reservation may be made during this time. If nothing is scheduled during this block, it will default to behaving as open swim.

During non-reservation times, jobs are restricted to run for at most 5 minutes and may have no more than 1,500,000 total shots on the processor.

The schedule of a processor is "frozen" for 4 hours into the immediate future so that users can rely on it for the coming day.

The ability to make reservations is restricted by budgets that are delegated by Quantum Computing Service admins. These are provided on a weekly basis and communicated either during the weekly user sync or by email and are good for one week in the future, but this policy is subject to change.

You can use this [reservation colab utility](../tutorials/google/reservations.ipynb)  to view the processor schedule and make reservations._____no_output_____


# Notebook from arthur-e/swc-workshop
Path: python-ecology/Python-Intro-Ecology.ipynb

# Overview_____no_output_____This lesson introduces Python as an environment for data analysis and visualization. The materials are based on the Data Carpentry [Python for Ecologists lesson](http://www.datacarpentry.org/python-ecology-lesson/). However, the lesson focuses on general analysis and visualization of tabular data and is not specific to ecologists or ecological data. As Data Carpentry explains:

> Data Carpentry’s aim is to teach researchers basic concepts, skills, and tools for working with data so that they can get more done in less time, and with less pain.

**At the end of this lesson, you will be able to:**

- Read and write basic Python code;
- Import and export tabular data with Python;
- Subset and filter tabular data;
- Understand different data types and data formats;
- Understand pandas Data Frames and how they help organize tabular data;
- Devise and intepret data processing workflows;
- Automate your data cleaning and analysis with Python;
- Visualize your data using matplotlib and pandas;
- Connect to a SQLite database using Python.

This lesson will not prepare you to use Python as a general purpose programming language; there are some parts of the language we won't have time to cover. However, at the end of this lesson, you will have a good grasp of Python syntax and be well-prepared to learn the rest of the language, if you desire to do so. **Even without seeing all of the Python programming language, you will be prepared to analyze and visualize data in Python using pandas and matplotlib.**_____no_output_____## License_____no_output_____As with [the Data Carpentry ecology lesson](http://www.datacarpentry.org/python-ecology-lesson/license/), this lesson is licensed for open use under the [CC BY 4.0 license](https://creativecommons.org/licenses/by/4.0/)._____no_output_____# Introduction to the Python Programming Language_____no_output_____Python is a general purpose programming language that allows for the rapid development of scientific workflows. Python's main advantages are:

- It is open-source software, supported by the [Python Software Foundation](https://www.python.org/psf/);
- It is available on all platforms, including Windows, Mac OS X, and GNU/Linux;
- It can be used to program any kind of task (it is a *general purpose* language);
- It supports multiple *programming paradigms* (a fancy term computer scientists use to describe the different ways people like to design software);
- **Most importantly, it has a large and diverse community of users who share Python code they've already written to do a wide variety of things.**_____no_output_____## The Python Interpreter_____no_output_____The only language that computers really understand is machine language, or binary: ones and zeros. Anything we tell computers to do has to be translated to binary for computers to execute.

Python is what we call an *interpreted language.* This means that computers can translate Python to machine code as they are reading it. This distinguishes Python from languages like C, C++, or Java, which have to be *compiled* to machine code *before* they are run. The details aren't important to us; **what is important is that we can use Python in two ways:**

- We can use the Python interpreter in **interactive mode;**
- Or, we can use execute Python code that is stored in a text file, called a script._____no_output_____### Jupyter Notebook_____no_output_____For this lesson, we'll be using the Python interpreter that is embedded in Jupyter Notebook. Jupyter Notebook is a fancy, browser-based environment for **literate programming,** the combination of Python scripts with rich text for telling a story about the task you set out to do with Python. This is a powerful way for collecting the code, the analysis, the context, and the results in a single place.

The Python interpreter we'll interact with in Jupyter Notebook is the same interpreter we could use from the command line. To launch Jupyter Notebook:

- In GNU/Linux or Mac OS X, launch the Terminal and type: `jupyter notebook`; then press ENTER.
- In Windows, launch the Command Prompt and type `jupyter notebook`; then press ENTER._____no_output_____Let's try out the Python interpreter._____no_output_____

print('Hello, world!')Hello, world!

Alternatively, we could save that one line of Python code to a text file with a `*.py` file extension and then execute that file._____no_output_____## Python Data Types_____no_output_____### Strings, Integers, and Floats_____no_output_____

text = 'Data Carpentry' # A character string
number = 42 # An integer number
pi = 3.14159265 # A floating-point number or "float"_____no_output_____

Here, we've assigned data to **variables** using the **assignment operator** or equal sign. **The process of assignment takes a value and stores it under a name that we make up.** This way, we can use that stored value again by calling its name._____no_output_____

number_____no_output_____

Note that to recover a variable's stored value, we simply type the name of the variable and hit `Enter`. (This only works in interactive mode; if we wrote a script and want it to print out a value, we have to use the `print()` function.)_____no_output_____Variable names can only include letters, the underscore, and numbers. However, variable names cannot start with numbers._____no_output_____

my_variable = 'some text'_____no_output_____

### Operators_____no_output_____We can perform mathematical calculations in Python using the basic operators `+`, `-`, `/`, `*`, and `%`._____no_output_____

2 + 2_____no_output_____6 * 7_____no_output_____5 ** 2_____no_output_____

This is "13 modulo 5" and the result is the remainder, 3._____no_output_____

13 % 5_____no_output_____

We can also use **comparison** and **logical operators.** These operators return Boolean values; that is, they determine or describe whether something is `True` or `False`._____no_output_____

3 > 4_____no_output_____5 == 5_____no_output_____

i.e., is it True *and* True?_____no_output_____

True and True_____no_output_____

Chooses the "truth-y" value between the two:_____no_output_____

True or False_____no_output_____

`True` and `False`, with the first letter capitalized, are special values in Python that mean just what they say._____no_output_____## Sequences_____no_output_____Much of Python's expressive power and flexibility comes from the way it handles **sequences.** A sequence could be a sequence of characters in a text string or a sequence of numbers._____no_output_____A **list** is Python's built-in data structure for handling general, ordered sequences. Each element can be accessed by its index. **Note that, in Python, we start counting from zero, not from one.**_____no_output_____

numbers = [1, 2, 3]
numbers[0]_____no_output_____

The square brackets are used to **slice** a sequence by one or more indices. Above, we have asked for the first (the zeroth) element of the `numbers` sequence._____no_output_____A `for` loop is a useful way of accessing the elements of a sequence one at a time:_____no_output_____

for number in numbers:
    print(number)1
2
3

**Indentation is very important in Python.** Note that the second line in the above example is indented. This is Python's way of marking a block of code. It's standard to indent by 4 spaces._____no_output_____To add elements to the end of a list, we can use the `append()` method:`_____no_output_____

numbers.append(4)
print(numbers)[1, 2, 3, 4]

Note that there is no output associated with the `append()` method; the `numbers` sequence is modified in place so we don't need to assign the result to a variable._____no_output_____**Methods are a way to interact with an object in Python.** We can invoke a method using the dot, followed by the method name and a list of arguments in parentheses. To find out what methods are available for an object, we can use the built-in `help()` function._____no_output_____

help(numbers)Help on list object:

class list(object)
 |  list() -> new empty list
 |  list(iterable) -> new list initialized from iterable's items
 |
 |  Methods defined here:
 |
 |  __add__(self, value, /)
 |      Return self+value.
 |
 |  __contains__(self, key, /)
 |      Return key in self.
 |
 |  __delitem__(self, key, /)
 |      Delete self[key].
 |
 |  __eq__(self, value, /)
 |      Return self==value.
 |
 |  __ge__(self, value, /)
 |      Return self>=value.
 |
 |  __getattribute__(self, name, /)
 |      Return getattr(self, name).
 |
 |  __getitem__(...)
 |      x.__getitem__(y) <==> x[y]
 |
 |  __gt__(self, value, /)
 |      Return self>value.
 |
 |  __iadd__(self, value, /)
 |      Implement self+=value.
 |
 |  __imul__(self, value, /)
 |      Implement self*=value.
 |
 |  __init__(self, /, *args, **kwargs)
 |      Initialize self.  See help(type(self)) for accurate signature.
 |
 |  __iter__(self, /)
 |      Implement iter(self).
 |
 |  __le__(self, value, /)
 |      Return self<=value.
 |
 |  __len__(self, /)
 |      Return len(self).
 |
 |  __lt__(self, value, /)
 |      Return self<value.
 |
 |  __mul__(self, value, /)
 |      Return self*value.n
 |
 |  __ne__(self, value, /)
 |      Return self!=value.
 |
 |  __new__(*args, **kwargs) from builtins.type
 |      Create and return a new object.  See help(type) for accurate signature.
 |
 |  __repr__(self, /)
 |      Return repr(self).
 |
 |  __reversed__(...)
 |      L.__reversed__() -- return a reverse iterator over the list
 |
 |  __rmul__(self, value, /)
 |      Return self*value.
 |
 |  __setitem__(self, key, value, /)
 |      Set self[key] to value.
 |
 |  __sizeof__(...)
 |      L.__sizeof__() -- size of L in memory, in bytes
 |
 |  append(...)
 |      L.append(object) -> None -- append object to end
 |
 |  clear(...)
 |      L.clear() -> None -- remove all items from L
 |
 |  copy(...)
 |      L.copy() -> list -- a shallow copy of L
 |
 |  count(...)
 |      L.count(value) -> integer -- return number of occurrences of value
 |
 |  extend(...)
 |      L.extend(iterable) -> None -- extend list by appending elements from the iterable
 |
 |  index(...)
 |      L.index(value, [start, [stop]]) -> integer -- return first index of value.
 |      Raises ValueError if the value is not present.
 |
 |  insert(...)
 |      L.insert(index, object) -- insert object before index
 |
 |  pop(...)
 |      L.pop([index]) -> item -- remove and return item at index (default last).
 |      Raises IndexError if list is empty or index is out of range.
 |
 |  remove(...)
 |      L.remove(value) -> None -- remove first occurrence of value.
 |      Raises ValueError if the value is not present.
 |
 |  reverse(...)
 |      L.reverse() -- reverse *IN PLACE*
 |
 |  sort(...)
 |      L.sort(key=None, reverse=False) -> None -- stable sort *IN PLACE*
 |
 |  ----------------------------------------------------------------------
 |  Data and other attributes defined here:
 |
 |  __hash__ = None

We can also access a list of methods using `dir`._____no_output_____

dir(numbers)_____no_output_____

### Tuples_____no_output_____A tuple is similar to a list in that it's an ordered sequence of elements. However, tuples can not be changed once created; they are "immutable." Tuples are created by placing comma-separated values inside parentheses._____no_output_____

a_tuple = (1,2,3)
another_tuple = ('rabbit', 'mongoose', 'platypus')
still_a_tuple = (1,)

# Note that lists use square brackets
a_list = [1,2,3]_____no_output_____

### Challenge: Tuples and Lists_____no_output_____

a_tuple = ('a', 'b', 'c', 'd')
a_list = ['a', 'b', 'c', 'd']_____no_output_____

1. What happens when you try to re-assign the first value in each of the following examples?
```py
a_tuple[0] = 5
a_list[0] = 5
```
2. Type `type(a_tuple)` into Python; what is the object's type?_____no_output_____## Dictionaries_____no_output_____A **dictionary** is a container that holds key-value pairs. It is a data structure that is also referred to, in other programming languages, as an **associative array.**_____no_output_____

mapping = {'tbsp': 'tablespoon', 'tsp': 'teaspoon'}
mapping['tbsp']_____no_output_____

Dictionaries are similar to lists in that we can store multiple things inside them. While we index lists with numeric indices, we index dictionaries with **keys.** A key is a unique identifier for a value in the dictionary._____no_output_____**Keys must be a particular data type; they must be hashable, therefore, only strings and numeric types are acceptable.**_____no_output_____

code_book = {1: 'Valid', 2: 'Invalid'}
code_book[1]_____no_output_____{[1,2]: 'Valid/Invalid'}_____no_output_____

**Another way we can create dictionaries is using the `dict()` function.** Here, we give the `dict()` function a list of key-value pairs, represented as tuples. The first element in the tuple is the key, the second element is the value._____no_output_____

dict([(1, 'Valid'), (2, 'Invalid')])_____no_output_____

Using `for` loops with dictionaries is a little more complicated._____no_output_____

for key, value in code_book.items():
    print(key, '->', value)1 -> Valid
2 -> Invalid

Alternatively:_____no_output_____

for key in code_book.keys():
    print(key, '->', code_book[key])1 -> Valid
2 -> Invalid

### Challenge: Reassignment in a Dictionary_____no_output_____How can we change a value in our dictionary? Try to reassign one of the values in the `code_book` dictionary._____no_output_____You may have noticed that a dictionary sometimes returns its items in a different order than you expected. **Dictionaries are intrinsically unordered; they do not retain the order of their items.**_____no_output_____## Functions_____no_output_____One of the chief reasons we program computers to do things for us is because computers are very good at tedious tasks (and humans are not). A block of Python code that does the same thing every time is best defined as **function** in Python. **A function is a series of fixed Python statements, with or without input arguments, that are assigned a name so that we can easily call them over and over again.**_____no_output_____We've already seen the built-in `print()` and `help()` functions. Now let's see how to write our own._____no_output_____

def pow10(exponent):
    result = 10 ** exponent
    return result

pow10(2)_____no_output_____

Key things to note about this example:

- We define a function using the `def` command followed by the name of the function and any arguments its takes, written just like we would call the function (recall the `print()` and `help()` functions);
- The **body** of the function is indented;
- We use the `return` command to indicate what the result of the function , or its **return value**, should be. If we don't `return` anything, the function's body is still executed, but we don't necessarily see any output when the function is called._____no_output_____**How can we make a more general version of the function `pow10()`?**_____no_output_____

def pow(exponent, base=10):
    result = base ** exponent
    return result

pow(2, 10)_____no_output_____

Here, the function `pow()` can be used to calculate powers of any base, not just base 10. By default, it will calculate powers of 10, because the argument `base` has a default argument of 10._____no_output_____

pow(2, base = 10)_____no_output_____pow(base = 10, exponent = 2)_____no_output_____

`pow()` is actually a function already built into Python, so we didn't need to write this function, but it demonstrates how you can create more flexible functions using default arguments._____no_output_____### Challenge: Writing Your First Function_____no_output_____To convert from temperatures in Fahrenheight to Celsius, we first subtract $32$ and then multiple by $5/9$. Write a function that converts temperatures from Fahrenheit to Celsius._____no_output_____### Lambda Functions_____no_output_____Another type of function in Python is the lambda function. This is a special, one-line function that is generally used for transforming inputs._____no_output_____

to_lowercase = lambda s: s.lower()
to_lowercase('Roger')_____no_output_____names = ['Roger', 'Helen', 'Abdul', 'Lakshmi']
list(map(to_lowercase, names))_____no_output_____

Lambda functions are important for reasons we'll see later._____no_output_____# Data Analysis Workflows_____no_output_____In general, when we have a data analysis task, there are specific steps that we undertake, in order:

1. **Bringing data together in a common format** that is both human-readable and machine-readable (e.g., **through spreadsheets**, before exporting as a CSV).
2. **Cleaning data,** e.g, merging text labels that are slightly different (e.g., "Tree frogs," "tree frogs," and "treefrogs"), splitting and combining fields as needed (e.g., in **OpenRefine** or in **Python**).
3. **Archiving data,** keeping the raw data separate from any cleaned data, for further use and analysis (e.g., using a **relational database and SQL**).
4. **Deriving a subset of cleaned data for use in a specific analysis,** such as a certain date range, certain field site, or certain collection (e.g., in **Python** or **SQL**).
5. **Performing analysis to answer a question or make a decision,** such as grouping and summarizing data, calculating a statistic, or generating a plot (e.g., in **Python** or **SQL**)._____no_output_____We'll now see how Python plays a role in this workflow, starting from having "clean" or nearly clean data to begin analyzing._____no_output_____# Managing Data in Python_____no_output_____We've seen how we can use spreadsheets to effectively organize data. You probably know how to do some basic analysis on tabular data using spreadsheet software programs like Microsoft Excel. **Why, then, should we learn to use Python for data analysis?**

- Because computers are good at automating tedious tasks (and humans are not), we can use Python to instruct a computer to repeat a task over and over again. This saves us frustration and ensures that the task is performed consistently, without human error.
- More specifically, once we have written the code to do some task with data the first time, we can apply it over and over again to subsequent datasets.
- Ultimately, we can save the code and share it with other people if they want to perform the same analysis or understand what we did. The code is then a record of what we did to get our results. This is a key component of **reproducibility** in computer-aided science._____no_output_____**For this lesson to run smoothly, let's make sure everyone is in the same directory.** In the Jupyter Notebook file tree, navigate to your Desktop. We'll create a new Notebook here to use for the rest of the lesson._____no_output_____## About the Data_____no_output_____For this lesson, we will be using the [Portal Project Teaching Database](https://figshare.com/articles/Portal_Project_Teaching_Database/1314459), a subset of data from Ernst et al.'s (2009) [long-term monitoring and experimental manipulation of a Chihuahuan desert ecosystem near Portal, Arizona, U.S.A.](http://www.esapubs.org/archive/ecol/E090/118/default.htm).
**We are studying the species and weight of animals caught in plots in our study area.**
The dataset is stored as a comma-separated variable (CSV) file: each row holds information for a single animal, and the columns reprsent:

Column          | Description
----------------|--------------------------------------
record_id       | Unique ID for the observation
month           | Month of observation
day             | Day of observation
year            | Year of observation
plot_id         | ID of a particular plot
species_id      | 2-letter code identifying the species
sex             | Sex of animal ("M","F")
hindfoot_length | Length of the hindfoot in millimeters
weight          | Weight of the animal in grams_____no_output_____## About Libraries_____no_output_____A library in Python contains a set of tools (called functions) that perform tasks on our data.
Importing a library is like getting a piece of lab equipment out of a storage locker and setting it up on the bench for use in a project.
Once a library is set up, it can be used to perform many tasks.
**One of the libraries we'll be using in this lesson is the Python Data Analysis Library, or `pandas`.**
`pandas` adds a number of things to base Python including more sophisticated data structures and tools to connect to other libraries like `matplotlib` for producing data visualizations._____no_output_____Python doesn't initially load all of the libraries that we have installed; we have to tell Python to import the librar(ies) that we want to use in any particular session.
**To import a library, we use the syntax:** `import libraryName`.
Because all of the tools in that library will be known by the name we import, we have the option to give it a shorter name.
We'll do that here with Pandas._____no_output_____

import pandas as pd_____no_output_____

Each time we call a function that's in the Pandas library, we use the syntax `libraryName.functionName`.
This ensures that Python can find the tool or function we're asking for.
Most Pandas users abbreviate the library name to `pd` and because this is such a popular convention, you can often find help on the internet by searching for a particular function in the `pd` **namespace.**
**For instance, I'll look up support for the `pd.read_csv()` function on the Google search engine.**_____no_output_____## Reading CSV Data in Pandas_____no_output_____Tabular data, like those stored in CSV, tab-delimited, or fixed-width formats, can be read into Python using Pandas and stored in a Pandas `DataFrame`.
If you've used the R programming language before, you'll be familiar with DataFrames._____no_output_____### Data Frames_____no_output_____**A DataFrame is a 2-dimensional, tabular data structure that can store different types of data across multiple columns.**
It is structurally identical to a spreadsheet in this way or, as we'll see, a table in a SQL database.
We can read in our survey data as a DataFrame using the `pd.read_csv()` function._____no_output_____

pd.read_csv('ecology-surveys.csv')_____no_output_____

When we run this command, Pandas finds the CSV file, opens it, and reads in the contents, line-by-line, as rows in a new DataFrame, which it then prints out.
We can see that:

- There were 33,549 rows parsed.
- Each has 9 columns.

**The first column is the index of the DataFrame.**
The index is used to identify the position of the data, but it is not an actual column of the DataFrame.

We can see from this output that Pandas read the file properly, but it hasn't saved the DataFrame to memory.
**We need to assign the DataFrame to a variable.**_____no_output_____

surveys = pd.read_csv('ecology-surveys.csv')_____no_output_____

Note that when we do variable assignment, Python does not display anything on the screen.
The output of the `pd.read_csv()` function, our table, instead of being displayed on the screen is not "captured" in the variable named `surveys`._____no_output_____

type(surveys)_____no_output_____

## Manipulating Data in Python_____no_output_____One of the first things we might do after importing any kind of data in any environment is to make sure that are our data--the individual fields or columns--are represented in the right way; by the right data types._____no_output_____

surveys.dtypes_____no_output_____

Here, `int64` refers to integer type; it cannot store decimal numbers. `float64` stores decimal numbers with 64-bit precision. `object` refers to character strings, or text._____no_output_____### Challenge: Viewing DataFrames in Python_____no_output_____Try executing each code sample below and see what is returned.

- `surveys.columns`
- `surveys.head()`
- `surveys.head(15)`
- `surveys.tail()`
- `surveys.shape`

Take note of the output of `surveys.shape`; what format does it return?

**Finally, what is the difference between the code samples that end in parentheses and those that do not?**

Each of the code samples above has us calling some **attribute** or **method** on the surveys DataFrame.

- **Methods** are functions that belong to an object in Python, like a DataFrame. Just like the functions we saw earlier, functions take zero or more arguments that go inside the parentheses. Even if we have no arguments to provide, we still have to use the parentheses to get the function to do its work. **In general, a method is a function that belongs to an object.**
- **Attributes** are a more general concept; an attribute is anything that belongs to an object in Python, including methods. Attributes that are not methods, however, don't need to be called with parentheses.

If we think of a person, an attribute is something that belongs to that person or describes that person, like hair color or number of siblings.
A method is something that person does, like bake a pie or go for a run._____no_output_____## Calculating Statistics in a Pandas DataFrame_____no_output_____Let's perform some summary statistics to further verify that the data we imported look okay.
First, let's remind ourselves what is in our data and what values we might use to group the data by in calculating summary statistics._____no_output_____

# Look at the column names
surveys.columns.values_____no_output_____

Let's get a list of all the species.
The `pd.unique()` function tells us all of the unique values in the `species_id` column._____no_output_____

pd.unique(surveys['species_id'])_____no_output_____

**Note the bracket notation we've used here. This is how we access a column by name in a Pandas DataFrame.**
It's just like how we accessed the value in a Dictionary by its key name._____no_output_____### Challenge: Unique Levels for a Column_____no_output_____1. Create a list of unique plot IDs found in the survey data; assign the list of unique IDs to a variable called `plot_names`. **How many unique plots are there in the data? How many unique species are in the data?**
2. What is the difference between `len(plot_names)` and `plot_names.shape`?_____no_output_____### Groups in Pandas_____no_output_____We often want to calculate summary statistics grouped by subsets or attributes within fields of our data.
**For example, we might want to calculate the average weight of all individuals in each plot.**

We can calculate basic statistics for all records in a single column using the syntax below._____no_output_____

surveys['weight'].describe()_____no_output_____

Let's break that last command down.

1. First, we asked for the `weight` column in the `surveys` table.
2. Second, we called the `describe()` method on that column._____no_output_____**We can also extract a specific statistic.**_____no_output_____

surveys['weight'].mean()_____no_output_____surveys['weight'].count()_____no_output_____

If we want to summarize by one or more variables, for example, `sex`, we can use Pandas' `groupby()` method.
Once we've created a grouped DataFrame, we can quickly calculated summary statistics by a group of our choice._____no_output_____

# Group data by sex
grouped_by_sex = surveys.groupby('sex')

# Summary statistics for all numeric columns, grouped by sex
grouped_by_sex.describe()_____no_output_____grouped_by_sex.mean()_____no_output_____

### Quickly Creating Summary Counts in Pandas_____no_output_____Let's count the number of samples for each species.
We can do this in a few ways, but we'll use `groupby()` combined with a `count()` method._____no_output_____

species_counts = surveys.groupby('species_id')['record_id'].count()
species_counts_____no_output_____

Again, let's break this down:

1. We group the rows of the `surveys` DataFrame by the unique values in one of its columns: `species_id`.
2. Second, we ask for the column `record_id` in the output from the last part.
3. Third, we call the `count()` method on this column. Because the data are grouped by `species_id`, we get a count within each unique value of `species_id`, not a count of all the rows in the table._____no_output_____### Challenge: Understanding Grouped DataFrames_____no_output_____1. In that last command, we asked for the `record_id` column. Try asking for a different column in the square brackets. Do you get a different result? Why or why not?
2. How can we get a count of just the records with `species_id` set to `DO`? *Hint: You can build on the last command we executed; think about Dictionaries and key-value pairs.*_____no_output_____### Basic Math on DataFrame Columns_____no_output_____We can perform quick mathematical transformations on the values of a column in a straightforward way.
For instance, we might normalize the measured hindfoot lengths by their mean._____no_output_____

surveys['hindfoot_length'] / surveys['hindfoot_length'].mean()_____no_output_____

## Basic Plotting with Pandas_____no_output_____We can also visualize summary statistics using Pandas.
**First, we'll fire off an iPython "magic" function that will allow us to  view plots inline in the Jupyter Notebook.**_____no_output_____

%matplotlib inline_____no_output_____# Create a quick bar chart
species_counts.plot(kind = 'bar')_____no_output_____

This plot is kind of cramped and hard to read; we can make it slightly bigger by specifying a new weight and height (in inches)._____no_output_____

species_counts.plot(figsize=(9, 6), kind = 'bar')_____no_output_____

### Challenge: Plotting_____no_output_____1. Create a plot of the average weight in each species.
2. Create a plot of total males and total females across the entire dataset.

*Note:* Some of the species have no weight measurements; they are entered as `NaN`, which stands for "not a number" and refers to missing values._____no_output_____### Multiple Grouping in Pandas_____no_output_____Let's make a *stacked bar plot,* with the total number of individuals on the Y axis and the stacked variable being `sex`. The plot should show total count by sex for each survey plot._____no_output_____First, we'll calculate the total number of individual in each unique pair of `plot_id` and `sex`._____no_output_____

plot_sex_count = surveys.groupby(('plot_id', 'sex'))['record_id'].count()
plot_sex_count_____no_output_____

Next, we use the `unstack()` method on our grouped data to figure out the total count that each sex contributes to each plot.
Here, `unstack()` simply rearranges the rows and columns into a more convenient format._____no_output_____

my_plot = plot_sex_count.unstack().plot(kind = 'bar', stacked = True)
my_plot.set_title('Total weight by plot and sex')
my_plot.set_ylabel('Weight (grams)')
my_plot.set_xlabel('Plot ID')_____no_output_____

# Transforming Data_____no_output_____**Up to this point, we have learned:**

- The basic data types in Python;
- How to read a CSV file into Python using pandas;
- How tabular data in pandas are represented using a DataFrame;
- How to check the data type of each column in a DataFrame;
- How to summarize numerical data in a DataFrame;
- How to group data according to the unique values of a column;
- How to create basic plots of grouped, summarized data._____no_output_____**Often, once we've pulled in data and done some basic data cleaning, we want to transform our data to make it more useful for our analysis.** For instance, we might find that:

- The numeric values in one or more columns are not in the right units, e.g., we want pounds instead of kilograms.
- A text column has keys written one way and we want them written another way.
- We want to combine two or more columns; or, we want to split one column into multiple columns.

Some of these objectives are perhaps best handled in OpenRefine, as we saw. However, we can also transform our data easily enough with Python and Pandas._____no_output_____## Converting Units_____no_output_____In our `surveys` data, the animal weights are in grams. What if our collaborator says she needs the weights in kilograms?_____no_output_____

# Create a copy of our original data
surveys_converted = surveys.copy()

# Create a new column in this table
surveys_converted['weight_kg'] = surveys_converted.weight / 1000
surveys_converted.tail()_____no_output_____

**Note that when we call for a column as an attribute, it is read-only.**_____no_output_____

# Convert grams to pounds
surveys_converted.weight_lbs = surveys_converted.weight * 0.00220462
surveys_converted.tail()_____no_output_____

## Transforming Values_____no_output_____We can affect a more general transformation of a column's values. Let's say that, for whatever reason, we want the text strings in `species_id` to be lower-case, not upper-case._____no_output_____

surveys.species_id.apply(lambda x: x.lower()).head()_____no_output_____

**What happened?** When Pandas imports a DataFrame and sees categorical data, like `species_id` here, it often treats the categorical data like numeric data in the background, assigning a numeric key to each unique categorical value. **We need to explicitly tell Pandas to treat this value like a string.**_____no_output_____

surveys.species_id.apply(lambda x: str(x).lower()).head()_____no_output_____

The use of the `apply()` function here is part of a general practice called **function application.** We can also apply a function to our entire DataFrame along one of its two axes. **For instance, we can use this to count how many non-null values are in each of the columns.**_____no_output_____

surveys.weight.count()_____no_output_____surveys.apply(lambda column: column.count(), axis = 0)_____no_output_____

We can also apply functions across the columns._____no_output_____

# Returns True or False depending on whether weight is null
surveys.apply(lambda row: pd.notnull(row.weight), axis = 1).head()_____no_output_____

# Indexing and Slicing Python DataFrames_____no_output_____**Now, we'll explore how to access different parts of our data, including querying subsets of the data based on certain criteria.**_____no_output_____## Indexing and Slicing in Python_____no_output_____### Selecting Data Using Labels (Column Headings)_____no_output_____We use square brackets, `[]`, to select a subset of a Python object. For example, we can select all of the data from a column named `species_id`:_____no_output_____

surveys['species_id']
surveys.species_id_____no_output_____

We can pass a list of column names, too, as an index to select columns in a specified order._____no_output_____

surveys[['species_id', 'plot_id']]
surveys[['plot_id', 'species_id']]_____no_output_____

### Extracting a Range of Data with Slicing_____no_output_____Recall that in Python, we start counting from zero instead of one. This means that the first element in an object is located at position zero._____no_output_____

grades = [88, 72, 93, 94]_____no_output_____

![](./slicing-indexing.svg)_____no_output_____

grades[2]_____no_output_____

![](./slicing-slicing.svg)_____no_output_____

grades[1:3]_____no_output_____

### Challenge: Slicing_____no_output_____What do each of these lines of code return?

1.  `grades[0]`
2.  `grades[len(grades)]`
3.  `grades[4]`

Why do (2) and (3) return errors?_____no_output_____### Slicing Subsets of Rows in Python_____no_output_____**Slicing using the `[]` operator selects a set of rows and/or columns from a DataFrame.** To slice out a set of rows, we use the following syntax._____no_output_____

# Select the first three rows; rows 0,1,2 (but NOT 3)
surveys[0:3]_____no_output_____

Other examples:_____no_output_____

# First three rows, starting index optional
surveys[:3]_____no_output_____# Select the last element (last row)
surveys[-1:]_____no_output_____

We can also reassign values within subset of our DataFrame. Before we do that, let's make a copy of our DataFrame so as not to modify our original data._____no_output_____

surveys_copy = surveys
surveys_copy[0:3] = 0
surveys_copy.head()_____no_output_____# Confirm that the original is unchanged
surveys.head()_____no_output_____

### Oops: Referencing versus Copying Objects in Python_____no_output_____**We thought that we were creating a copy of the `surveys` DataFrame.** However, when we assign a variable such as `y = x`, this doesn't create a copy of `x`; rather, it creates a new variable `y` that refers to the same object that `x` refers to. This means there is only one object (the DataFrame), and both `x` and `y` refer to it. Thus, when we modify `surveys_copy`, the object that is modified is the same object `surveys` points to._____no_output_____

surveys = pd.read_csv('ecology-surveys.csv')
surveys_copy = surveys.copy()
surveys_copy[0:3] = 0
surveys.head()_____no_output_____

### Slicing Subsets of Rows and Columns with Pandas_____no_output_____We can select specific ranges of our data in both the row and column directions using either **label-based** or **integer-based indexing.**_____no_output_____To select a subset of rows and columns, we can use the `iloc()` method (for "index location"). For example, **we can select the month, day, and year columns (columns, 2, 3, 4) in the first three rows, like this:**_____no_output_____

surveys.iloc[0:3,1:4]_____no_output_____

**Note that the order of terms in the square brackets is rows first, then columns.**_____no_output_____

# Get the same columns but all the rows
surveys.iloc[:,1:4].head()_____no_output_____

**Here, the colon character in place of the row index indicates we want "all" elements (rows).**_____no_output_____We can select multiple, discontiguous rows by passing a list of the row indices._____no_output_____

# Select all columns for rows of index values 0, 10
surveys.iloc[[0, 10], :]_____no_output_____

If we want to select columns by column names, we need to use the `loc()` method in place of `iloc()`._____no_output_____

surveys.loc[0, ['species_id', 'plot_id', 'weight']]_____no_output_____

When we use `loc()`, we can also index using integers, but with one essential difference: **`loc()` indexes on row or column labels, not the position along the table's rows or columns.**

For example, compare the outputs:_____no_output_____

surveys.loc[[0, 10, 50000], :]_____no_output_____surveys.iloc[[0, 10, 50000], :]_____no_output_____

### Challenge: Slicing Rows and Columns_____no_output_____What happens when you type:

1.  `surveys[0:3]`
2.  `surveys[:5]`
3.  `surveys[-1:]`_____no_output_____To review...

**To index by rows in Pandas:**_____no_output_____

surveys[0:3]
surveys.iloc[0:3]
surveys.iloc[0:3,:]_____no_output_____

**To index by columns (and rows) in Pandas**_____no_output_____

surveys[['month', 'day', 'year']]
surveys.loc[0:3, ['month', 'day', 'year']]
surveys.iloc[0:3, 1:4]_____no_output_____

## Subsetting Data Using Criteria_____no_output_____Most importantly, we can also select a subset of our data based on certain criteria. For example, we can select all of the rows of our data that match observations in the year 2002._____no_output_____

surveys[surveys.year == 2002].head()_____no_output_____

Or, we can select all rows that do not match observations from 2002._____no_output_____

surveys[surveys.year != 2002].head()_____no_output_____

**We can combine criteria using logical operators.**_____no_output_____

surveys[(surveys.year >= 1980) & (surveys.year <= 1985)].count()_____no_output_____

### Challenge: Filtering Data_____no_output_____1. Filter the `surveys` table to observations of female members of the `DO` species. How many are there? What is their average weight?
2. Look at the help documentation for the `isin()` function (Hint: `?surveys.year.isin`). Use this function to filter the `surveys` DataFrame to those rows that match the three species: `OL`, `OT`, `OX`._____no_output_____## Using Masks_____no_output_____Earlier, we saw that our data contain some missing values, filled in with `NaN` or "not a number."
We'll next learn how to handle NaNs when they appear in our data using masks.
A mask can be useful to locate where a particular subset values exist (or don't exist)._____no_output_____To start, we can use the Pandas function `isnull()` to find places in our `surveys` DataFrame where there are null or NaN values._____no_output_____

pd.isnull(surveys)_____no_output_____

We can use the `any()` function to ask if there are any null values in a given row. Here, the `axis` argument specifies the axis along which to look for true values (values that are null); 1 indicates a look along the rows._____no_output_____

pd.isnull(surveys).any(axis=1).head()_____no_output_____pd.isnull(surveys).any(axis=0)_____no_output_____

To select the rows where there are null values, we can use the mask as an index to subset our data, as follows._____no_output_____

surveys[pd.isnull(surveys).any(axis=1)].head()_____no_output_____

**How can we select rows where there are NO null values?** We can invert the mask using the tilde operator, `~`._____no_output_____

surveys[~pd.isnull(surveys).any(axis=1)].head()_____no_output_____

## Dealing with Missing Data_____no_output_____It is sometimes hard to know why data values are missing. Was it because of a data entry error? Was there some condition in the field that prevented observation of a particular variable?

Moreover, we need a way of representing missing data that is clear and unambiguous. If we encode missing values as zeroes, for instance, are we sure that zero won't be interpreted as a real value?

For text data, an empty string usually suffices for blank values. However, numeric data fields can't hold an empty string; they have to either hold some number or a special value called `NaN`, which stands for "not a number."_____no_output_____

len(surveys[pd.isnull(surveys.weight)])_____no_output_____# Number of null values, in this case, is the same as number of non-zero values
len(surveys.weight) - len(surveys[surveys.weight > 0])_____no_output_____

We can replace all the `NaN` values in our `weight` column using the `fillna()` method. Here, we use data management best practices, copying our original data so that we don't modify it._____no_output_____

df = surveys.copy()
df.weight = df.weight.fillna(0)_____no_output_____

However, `NaN` and zero will yield different results in analysis._____no_output_____

df.weight.mean()_____no_output_____surveys.weight.mean()_____no_output_____

**So, in general, it's better to leave `NaN` values as `NaN` and to subset the data to remove them, rather than re-coding them.**_____no_output_____## Indexing by Date and Time_____no_output_____

#TODO_____no_output_____

# Combining Multiple Datasets_____no_output_____At this point, we have learned:

- About zero-based indexing in Python;
- How to manipulate and extract data using column headings and index locations;
- How to employ slicing to select subsets of a DataFrame;
- How to re-assign values within a DataFrame;
- How to create a copy of a DataFrame;
- How to query subsets of a DataFrame that match certain criteria using logical operators;
- How to interpret NaN values and best practices for missing data;_____no_output_____In many situations, the data we want to use come in multiple files. We often need to combine these files into a single DataFrame in order to analyze the data._____no_output_____Let's load a second table into our session._____no_output_____

surveys = pd.read_csv('ecology-surveys.csv')
species = pd.read_csv('ecology-species.csv')_____no_output_____

## Concatenating DataFrames_____no_output_____We can use the `concat` function in Pandas to append either columns or rows from one DataFrame to another. Let's create subsets of our data to see how this works..._____no_output_____

# First 10 rows of surveys
surveys_first10 = surveys.head(10)

# Last 10 rows of surveys
surveys_last10 = surveys.tail(10)
surveys_last10_____no_output_____# Reset the index so that we can concatenate properly
surveys_last10 = surveys_last10.reset_index(drop = True)
surveys_last10_____no_output_____

When we concatenate DataFrames, we need to specify the axis. With `axis=0`, we are stacking DataFrames on top of one another (row-wise). Pandas will automatically detect that the column names are the same and stack them accordingly._____no_output_____

vertical_stack = pd.concat([surveys_first10, surveys_last10], axis = 0)
vertical_stack_____no_output_____# In order to avoid duplication of index values
vertical_stack.reset_index()_____no_output_____

## Writing Data to a CSV File_____no_output_____We can use the `to_csv()` method to export a DataFrame to a CSV file._____no_output_____

vertical_stack.to_csv('out.csv')_____no_output_____

## Joining DataFrames_____no_output_____**Note:** Use [this webpage](http://blog.codinghorror.com/a-visual-explanation-of-sql-joins/) as a graphical reference for this segment.

When we concatenated our DataFrames, we simply added them together, stacking them either vertically or horizontally. **Another way to combine DataFrames is to use columns in each dataset that contain common values (a common identifier).** Combining DataFrames in this way is referred to as a **join.**

Joining data frames in this way is especially useful when one DataFrame serves as a **lookup table,** containing additional data that we want to include in the other. **For example, the `species` table we imported is a lookup table. It contains additional information about each `species_id`.**

To better understand how this works, let's create a subset of the surveys table._____no_output_____

surveys_sub = surveys.tail(20)_____no_output_____

### Identifying Join Keys_____no_output_____When we join two tables together, they must share a common key. Usually, it is a field that uniquely identifies the records in one of the two tables. **In this example, it is the `species_id` field that the tables share.**_____no_output_____

surveys_sub.columns_____no_output_____species.columns_____no_output_____

### Inner Joins_____no_output_____The most common type of join is called an **inner join.** An inner join combines two DataFrames based on a shared key and returns a new DataFrame that contains *only* those rows that have matching values in *both* of the original DataFrames.

The Pandas function for performing joins is called `merge()`; by default, it performs an inner join._____no_output_____

merged = pd.merge(left = surveys_sub, right = species, left_on = 'species_id', right_on = 'species_id')
merged.shape_____no_output_____merged_____no_output_____

The result of an inner join of `surveys_sub` and `species` is a new DataFrame that contains the combined set of columns from `surveys_sub` and `species`.

**Note that the result of this merge is only 19 rows, whereas `surveys_sub` started with 20 rows.**_____no_output_____### Left Joins_____no_output_____If there are rows in `surveys_sub` with a `species_id` that is not found in the `species` lookup table, then they are not included in the merged output. What if we want to add information from `species` to `surveys_sub` without losing any of the information from `surveys`? **A left join returns all of the rows from the left DataFrame while joining only whatever rows in the right data frame match.**_____no_output_____

# Because there is only one shared column, we can neglect giving any column mames
merged_left = pd.merge(left = surveys_sub, right = species, how = 'left')
merged_left.shape_____no_output_____merged_left_____no_output_____

### Other Joins_____no_output_____- A **right (outer) join** is similar to a left join except that it takes all rows from the right data frame and only matching rows from the left.
- A **full (outer) join** returns all pairwise combinations of the two matching columns. This type is rarely used._____no_output_____### Challenges: Joins_____no_output_____Create a new data frame by joining the contents of `surveys` and `species`. Then, calculate and plot the distribution of `taxa` by `plot_id`._____no_output_____# Automating Data Workflows with Python_____no_output_____So far, we've used Python and the Pandas library to explore and manipulate individual datasets by hand, much like we would do in a spreadsheet. **The advantage of using a programming environment like Python, though, comes from its ability to automate data processing through the use of loops and functions.**

Let's remind ourselves of how loops work._____no_output_____

animals = ['LIONS', 'TIGERS', 'BEARS']

for creature in animals:
    print(creature.lower())lions
tigers
bears

A powerful Python idiom that we can use here is the **list comprehension:**_____no_output_____

[creature.lower() for creature in animals]_____no_output_____

Similarly, a **dictionary comprehension** is another quick way to transform data in Python._____no_output_____

dict((creature, creature.lower()) for creature in animals)_____no_output_____

## Automating Data Processing with For Loops_____no_output_____As one example of how we can automate data processing with for loops, let's imagine that we want to split our large `surveys` table into separate files, one for each survey year. **We'll start by making a new directory to hold the output files for each year.**

We can use Python's `os` library to manipulate files and folders on our computer's file system._____no_output_____

import os
os.mkdir('yearly_files')_____no_output_____

The command `os.mkdir` is equivalent to the `mkdir` command in the Unix shell, which you may have seen if you use a GNU/Linux or Mac computer. To confirm that we just created the `yearly_files` folder, we can ask Python to list the contents of our current directory._____no_output_____

os.listdir('.')_____no_output_____

**In previous lessons, we saw how to use the Pandas library to load a table into memory as a DataFrame, how to select a subset of the data using some criteria, and how to write the DataFrame to a CSV file.**

Let's now write a script that performs those three steps for just the year 2002._____no_output_____

import pandas as pd

# Load the data into a DataFrame
surveys = pd.read_csv('ecology-surveys.csv')

# Select only data for 2002
surveys2002 = surveys[surveys.year == 2002]

# Write the new DataFrame to a CSV file
surveys2002.to_csv('yearly_files/surveys_2002.csv')_____no_output_____

**One way we could create a file for every year would be to change this Python script for each year.** This would be tedious, however, and tedium is what we're trying to avoid by using Python! Moreover, if we are changing our code frequently, we're likely to introduce an error at some point.

**What we really want to do is to rewrite this script so that it is more general and can create all the files we need in one run.**

Let's start by writing a loop that simply prints the names of the files we want to create; files denoted by each year._____no_output_____

# We can get the years in our data...
surveys.year_____no_output_____

But we want only the unique years... How do we do that again?_____no_output_____

surveys.year.unique()_____no_output_____

Putting this into a loop, we can generate filenames for each year._____no_output_____

for year in surveys.year.unique():
    filename = 'yearly_files/surveys_%s.csv' % str(year)
    print(filename)yearly_files/surveys_1977.csv
yearly_files/surveys_1978.csv
yearly_files/surveys_1979.csv
yearly_files/surveys_1980.csv
yearly_files/surveys_1981.csv
yearly_files/surveys_1982.csv
yearly_files/surveys_1983.csv
yearly_files/surveys_1984.csv
yearly_files/surveys_1985.csv
yearly_files/surveys_1986.csv
yearly_files/surveys_1987.csv
yearly_files/surveys_1988.csv
yearly_files/surveys_1989.csv
yearly_files/surveys_1990.csv
yearly_files/surveys_1991.csv
yearly_files/surveys_1992.csv
yearly_files/surveys_1993.csv
yearly_files/surveys_1994.csv
yearly_files/surveys_1995.csv
yearly_files/surveys_1996.csv
yearly_files/surveys_1997.csv
yearly_files/surveys_1998.csv
yearly_files/surveys_1999.csv
yearly_files/surveys_2000.csv
yearly_files/surveys_2001.csv
yearly_files/surveys_2002.csv

**A few things to note about this code:**

- We are looping over the years in the `year` column of our DataFrame. Because `year` is an integer, we need to convert it to a string before we can make it part of a filename. This is what the built-in `str()` function does.
- Python allows us to embed text strings in other text strings through **string formatting.** Here, we use a special formatting character, `%s`, to denote where in the text string we want to embed some other text. The `%` operator is then followed by the string we want to embed.

**Now, we have everything we need to create these separate files by year.**_____no_output_____

surveys = pd.read_csv('ecology-surveys.csv')

for year in surveys.year.unique():
    # Select data for the given year
    surveys_in_year = surveys[surveys.year == year]

    # Write the subdataset to a new file
    filename = 'yearly_files/surveys_%s.csv' % str(year)
    surveys_in_year.to_csv(filename)

os.listdir('yearly_files')_____no_output_____

### Challenge: Automation with For Loops_____no_output_____1. Some of the surveys we saved are missing data; they have `NaN` values in one or more columns. **Modify our `for` loop so that the entries with null values are not included in the yearly files.**
2. **What happens if there are no data for a year in the sequence?** You can generate a list of years for the `for` loop to use with, e.g., `range(1970, 1980)`.
3. Let's say you only want to look at data from a given multiple of years. **How would you modify your loop in order to generate a data file for only every 5th year, starting from 1977?**_____no_output_____## Building Reusable Code with Functions_____no_output_____Suppose that separating large data files into individual yearly files is a task that we frequently have to perform. We could write a `for` loop like the one above every time we needed to do it but that would be time-consuming and error-prone. **A more elegant solution would be to create a resuable tool that performs this task with minimum input from the user.**

To achieve this, we'll encapsulate the code we've written so far into a function. As we've seen, functions are reusable, self-contained pieces of code that are called with a single command. **The first part of writing a function is envisioning how it will be used; how it will be called by the user.**_____no_output_____

def one_year_to_csv(all_data, the_year):
    '''
    Writes a CSV files for data from a given year.
    Arguments:
        all_data    The DataFrame will multi-year data
        the_year    The year to write, an integer
    '''
    pass_____no_output_____

Some things to note about this function:

- Our function takes two arguments, `the_year` and `all_data`.
- We've included a **docstring** in our function. The docstring is any string that comes on the line immediately after the function definition. It can be a single-line string but it is most commonly a multi-line string, indicated by three quote symbols in order.
- **It doesn't do anything yet.** We wanted to set up the function first so that we have an end goal in mind; functions have to have a body of code so we used the keyword `pass` here, which is a special Python command that says, "do nothing."_____no_output_____

?one_year_to_csv_____no_output_____

Now, let's populate the body of this function with the code we wrote before, making the necessary changes._____no_output_____

def one_year_to_csv(all_data, the_year):
    '''
    Writes a CSV files for data from a given year.
    Arguments:
        all_data    The DataFrame will multi-year data
        the_year    The year to write, an integer
    '''
    # Select data for the given year
    data_in_year = all_data[all_data.year == the_year]

    # Write the subdataset to a new file
    filename = 'yearly_files/surveys_%s.csv' % str(the_year)
    data_in_year.to_csv(filename)_____no_output_____one_year_to_csv(surveys, 2002)
os.listdir('yearly_files')_____no_output_____

This is a great start, but what we really want to do is create files for multiple years without having to request each year one at a time. Let's write another function that replicates the `for` loop we wrote earlier._____no_output_____

def multiple_years_to_csv(all_data, start_year, end_year):
    '''
    Writes separate CSV files for each year of data.
    Arguments:
        all_data      The DataFrame with multi-year data
        start_year    The first year of data we want
        end_year      The last year of data we want
    '''
    # Because range() generates a sequence up to but *not* including
    #     the last number, we add 1
    for year in range(start_year, end_year + 1):
        one_year_to_csv(all_data, year)_____no_output_____multiple_years_to_csv(surveys, 1977, 2002)
os.listdir('yearly_files')_____no_output_____

We can add optional arguments to help make this function even easier to use._____no_output_____

def multiple_years_to_csv(all_data, start_year=1997, end_year=2002):
    '''
    Writes separate CSV files for each year of data.
    Arguments:
        all_data      The DataFrame with multi-year data
        start_year    The first year of data we want
        end_year      The last year of data we want
    '''
    # Because range() generates a sequence up to but *not* including
    #     the last number, we add 1
    for year in range(start_year, end_year + 1):
        one_year_to_csv(all_data, year)_____no_output_____

But what if the years 1997 and 2002 are not in the data?_____no_output_____

def multiple_years_to_csv(all_data, start_year=None, end_year=None):
    '''
    Writes separate CSV files for each year of data.
    Arguments:
        all_data      The DataFrame with multi-year data
        start_year    The first year of data we want
        end_year      The last year of data we want
    '''
    if start_year is None:
        start_year = min(all_data.year)

    if end_year is None:
        end_year = max(all_data.year)

    # Because range() generates a sequence up to bubt *not* including
    #     the last number, we add 1
    for year in range(start_year, end_year + 1):
        one_year_to_csv(all_data, year)_____no_output_____multiple_years_to_csv(surveys)
os.listdir('yearly_files/')_____no_output_____

In the last example, we saw how we can build functions that call other functions we've built in the past. **In this way, we can develop sophisticated data processing workflows piece-by-piece, based on code we already know works.**_____no_output_____### Challenge: Writing Reusable Functions_____no_output_____1. What type of object corresponds to a variable declared as `None`? (Hint: Create a variable set to `None` and use the function `type()`).
2. What happens if you only call `multiple_years_to_csv()` with `all_data` and an `end_year` (that is, without providing a `start_year`)?  Can you write the function call with only a value for `end_year`?_____no_output_____# Answering Questions with Data_____no_output_____There are many questions we ask about tabular data that we can answer with Pandas._____no_output_____## Validating Assumptions about Data_____no_output_____How many animals of each species were caught in each plot? We can create a **cross-tabulation** of the `species_id` and `plot_id` fields as follows._____no_output_____

pd.crosstab(surveys.species_id, surveys.plot_id)_____no_output_____

## Sorting on Values_____no_output_____We might ask what the heaviest animals in our dataset are._____no_output_____

surveys.sort_values(by = 'weight', ascending = False).head(10)_____no_output_____

We might also ask for the last observation in our data. We could assume that the data in the table are in chronological order and simply look at the last row; however, there are many reasons why this may not be the case. **We want to sort the data with date descending. We don't have a single date field but we can sort by multiple fields to achieve the same effect.**_____no_output_____

surveys.sort_values(by = ['year', 'month', 'day'], ascending = False).head()_____no_output_____


# Notebook from deflaux/examples-Python
Path: notebooks/BRAF-V600 study using CCLE data.ipynb

# Working with BigQuery tables and the Genomics API_____no_output_____
## Case Study: BRAF V600 mutations in CCLE cell-lines

In this notebook we'll show you how you might combine information available in BigQuery tables with sequence-reads that have been imported into Google Genomics.  We'll be using the open-access CCLE data for this example.

You'll need to make sure that your project has the necessary APIs enabled, so take a look at the [Getting started with Google Genomics](https://cloud.google.com/genomics/install-genomics-tools) page, and be sure to also have a look at this [Getting started with the Genomics API](https://github.com/googlegenomics/datalab-examples/blob/master/datalab/genomics/Getting%20started%20with%20the%20Genomics%20API.ipynb) tutorial notebook available on github._____no_output_____We'll be using the [Google Python API client](https://github.com/google/google-api-python-client) so we'll need to install that first using the ``pip`` package manager.

**NOTE** that Datalab is currently using an older version of the oauth2client (1.4.12) and as a result we need to install an older version of the google-api-python-client that supports it._____no_output_____

!pip install --upgrade google-api-python-client==1.4.2Requirement already up-to-date: google-api-python-client==1.4.2 in /usr/local/lib/python2.7/dist-packages
Cleaning up...

Next we're going to need to authenticate using the service account on the Datalab host.  _____no_output_____

from httplib2 import Http
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
http = Http()
credentials.authorize(http)_____no_output_____

Now we can create a client for the Genomics API.  **NOTE** that in order to use the Genomics API, you need to have enabled it for your GCP project._____no_output_____

from apiclient import discovery
ggSvc = discovery.build ( 'genomics', 'v1', http=http )_____no_output_____

We're also going to want to work with BigQuery, so we'll need the biguery module.  We will also be using the pandas and time modules._____no_output_____

import gcp.bigquery as bq
import pandas as pd
import time_____no_output_____

The ISB-CGC group has assembled metadata as well as molecular data from the CCLE project into an open-access BigQuery dataset called ``isb-cgc:ccle_201602_alpha``.  In this notebook we will make use of two tables in this dataset: ``Mutation_calls`` and ``DataFile_info``.  You can explore the entire dataset using the BigQuery [web UI](https://bigquery.cloud.google.com)._____no_output_____Let's say that we're interested in cell-lines with BRAF V600 mutations, and in particular we want to see if there is evidence in both the DNA-seq and the RNA-seq data for these mutations.  Let's start by making sure that there are some cell-lines with these mutations in our dataset:_____no_output_____

%%sql

SELECT CCLE_name, Hugo_Symbol, Protein_Change, Genome_Change
FROM [isb-cgc:ccle_201602_alpha.Mutation_calls]
WHERE ( Hugo_Symbol="BRAF" AND Protein_Change CONTAINS "p.V600" )
ORDER BY Cell_line_primary_name
LIMIT 5_____no_output_____

OK, so let's get the complete list of cell-lines with this particular mutation:_____no_output_____

%%sql --module get_mutated_samples

SELECT CCLE_name
FROM [isb-cgc:ccle_201602_alpha.Mutation_calls]
WHERE ( Hugo_Symbol="BRAF" AND Protein_Change CONTAINS "p.V600" )
ORDER BY Cell_line_primary_name_____no_output_____r = bq.Query(get_mutated_samples).results()
list1 = r.to_dataframe()
print " Found %d samples with a BRAF V600 mutation. " % len(list1) Found 54 samples with a BRAF V600 mutation.

Now we want to know, from the ``DataFile_info`` table, which cell lines have both DNA-seq and RNA-seq data imported into Google Genomics.  (To find these samples, we will look for samples that have non-null readgroupset IDs from "DNA" and "RNA" pipelines.) _____no_output_____

%%sql --module get_samples_with_data

SELECT
  a.CCLE_name AS CCLE_name
FROM (
  SELECT
    CCLE_name
  FROM
    [isb-cgc:ccle_201602_alpha.DataFile_info]
  WHERE
    ( Pipeline CONTAINS "DNA"
      AND GG_readgroupset_id<>"NULL" ) ) a
JOIN (
  SELECT
    CCLE_name
  FROM
    [isb-cgc:ccle_201602_alpha.DataFile_info]
  WHERE
    ( Pipeline CONTAINS "RNA"
      AND GG_readgroupset_id<>"NULL" ) ) b
ON
  a.CCLE_name = b.CCLE_name_____no_output_____r = bq.Query(get_samples_with_data).results()
list2 = r.to_dataframe()
print " Found %d samples with both DNA-seq and RNA-seq reads. " % len(list2) Found 265 samples with both DNA-seq and RNA-seq reads.

Now let's find out which samples are in both of these lists:_____no_output_____

list3 = pd.merge ( list1, list2, how='inner', on=['CCLE_name'] )
print " Found %d mutated samples with DNA-seq and RNA-seq data. " % len(list3) Found 7 mutated samples with DNA-seq and RNA-seq data.

No we're going to take a closer look at the reads from each of these samples.  First, we'll need to be able to get the readgroupset IDs for each sample from the BigQuery table.  To do this, we'll define a parameterized function:_____no_output_____

%%sql --module get_readgroupsetid

SELECT Pipeline, GG_readgroupset_id
FROM [isb-cgc:ccle_201602_alpha.DataFile_info]
WHERE CCLE_name=$c AND GG_readgroupset_id<>"NULL"_____no_output_____

Let's take a look at how this will work:_____no_output_____

aName = list3['CCLE_name'][0]
print aName
ids = bq.Query(get_readgroupsetid,c=aName).to_dataframe()
print idsCOLO783_SKIN
                Pipeline        GG_readgroupset_id
0  broad.mit.edu__DNASeq  CJKPhaq1GhC4rZSVj4TMoIIB
1  broad.mit.edu__RNASeq  CJKPhaq1GhDN4avdoaTXsKcB

Ok, so we see that for this sample, we have two readgroupset IDs, one based on DNA-seq and one based on RNA-seq.  This is what we expect, based on how we chose this list of samples._____no_output_____Now we'll define a function we can re-use that calls the GA4GH API reads.search method to find all reads that overlap the V600 mutation position.  Note that we will query all of the readgroupsets that we get for each sample at the same time by passing in a list of readGroupSetIds.  Once we have the reads, we'll organized them into a dictionary based on the local context centered on the mutation hotspot._____no_output_____

chr = "7"
pos = 140453135
width = 11
rgsList = ids['GG_readgroupset_id'].tolist()

def getReads ( rgsList, pos, width):

  payload = { "readGroupSetIds": rgsList,
            "referenceName": chr,
            "start": pos-(width/2),
            "end": pos+(width/2),
            "pageSize": 2048
  }
  r = ggSvc.reads().search(body=payload).execute()

  context = {}
  for a in r['alignments']:
    rgsid = a['readGroupSetId']
    seq = a['alignedSequence']
    seqStartPos = int ( a['alignment']['position']['position'] )
    relPos = pos - (width/2) - seqStartPos
    if ( relPos >=0 and relPos+width<len(seq) ):
      # print rgsid, seq[relPos:relPos+width]
      c = seq[relPos:relPos+width]
      if (c not in context):
        context[c] = {}
        context[c][rgsid] = 1
      else:
        if (rgsid not in context[c]):
          context[c][rgsid] = 1
        else:
          context[c][rgsid] += 1

  for c in context:
    numReads = 0
    for a in context[c]:
      numReads += context[c][a]
    # write it out only if we have information from two or more readgroupsets
    if ( numReads>3 or len(context[c])>1 ):
      print "     --> ", c, context[c]_____no_output_____

Here we define the position (0-based) of the BRAF V600 mutation:_____no_output_____

chr = "7"
pos = 140453135
width = 11_____no_output_____

OK, now we can loop over all of the samples we found earlier:_____no_output_____

for aName in list3['CCLE_name']:
  print " "
  print " "
  print aName
  r = bq.Query(get_readgroupsetid,c=aName).to_dataframe()
  for i in range(r.shape[0]):
    print "  ", r['Pipeline'][i], r['GG_readgroupset_id'][i]
  rgsList = r['GG_readgroupset_id'].tolist()
  getReads ( rgsList, pos, width)

COLO783_SKIN
   broad.mit.edu__DNASeq CJKPhaq1GhC4rZSVj4TMoIIB
   broad.mit.edu__RNASeq CJKPhaq1GhDN4avdoaTXsKcB
     -->  ATTTCACTGTA {u'CJKPhaq1GhC4rZSVj4TMoIIB': 47, u'CJKPhaq1GhDN4avdoaTXsKcB': 10}
     -->  ATTTCTCTGTA {u'CJKPhaq1GhC4rZSVj4TMoIIB': 100, u'CJKPhaq1GhDN4avdoaTXsKcB': 30}

K029AX_SKIN
   broad.mit.edu__DNASeq CJKPhaq1GhCj-6WMobnvxCA
   broad.mit.edu__RNASeq CJKPhaq1GhDHzLyn-_j7w18
     -->  ATTTCACTGTA {u'CJKPhaq1GhDHzLyn-_j7w18': 8, u'CJKPhaq1GhCj-6WMobnvxCA': 63}
     -->  ATTTCTCTGTA {u'CJKPhaq1GhDHzLyn-_j7w18': 13, u'CJKPhaq1GhCj-6WMobnvxCA': 119}

MALME3M_SKIN
   broad.mit.edu__RNASeq CJKPhaq1GhCMvqyM-MWor2o
   broad.mit.edu__DNASeq CJKPhaq1GhCj8MDayraUjOUB
     -->  ATTTCACTGTA {u'CJKPhaq1GhCj8MDayraUjOUB': 48, u'CJKPhaq1GhCMvqyM-MWor2o': 9}
     -->  ATTTCTCTGTA {u'CJKPhaq1GhCj8MDayraUjOUB': 126, u'CJKPhaq1GhCMvqyM-MWor2o': 37}

SIGM5_HAEMATOPOIETIC_AND_LYMPHOID_TISSUE
   broad.mit.edu__DNASeq CJKPhaq1GhDH0sLg9qTgt5QB
   broad.mit.edu__RNASeq CJKPhaq1GhD_2PTBu4jiq5wB
     -->  ATTTCACTGTA {u'CJKPhaq1GhD_2PTBu4jiq5wB': 3, u'CJKPhaq1GhDH0sLg9qTgt5QB': 28}
     -->  ATTTCTCTGTA {u'CJKPhaq1GhD_2PTBu4jiq5wB': 6, u'CJKPhaq1GhDH0sLg9qTgt5QB': 44}

WM793_SKIN
   broad.mit.edu__DNASeq CJKPhaq1GhDauaiO54TV-lU
   broad.mit.edu__RNASeq CJKPhaq1GhDo3-DQ8s2qiAU
     -->  ATTTCACTGTA {u'CJKPhaq1GhDo3-DQ8s2qiAU': 11, u'CJKPhaq1GhDauaiO54TV-lU': 50}
     -->  ATTTCTCTGTA {u'CJKPhaq1GhDo3-DQ8s2qiAU': 13, u'CJKPhaq1GhDauaiO54TV-lU': 102}

WM88_SKIN
   broad.mit.edu__RNASeq CJKPhaq1GhDvot6FqNrO8bUB
   broad.mit.edu__DNASeq CJKPhaq1GhD85Zrppre74-0B
     -->  ATTTCACTGTA {u'CJKPhaq1GhDvot6FqNrO8bUB': 13, u'CJKPhaq1GhD85Zrppre74-0B': 47}
     -->  ATTTCTCTGTA {u'CJKPhaq1GhDvot6FqNrO8bUB': 17, u'CJKPhaq1GhD85Zrppre74-0B': 76}

WM1799_SKIN
   broad.mit.edu__RNASeq CJKPhaq1GhDrrMCJ0_zBuKkB
   broad.mit.edu__DNASeq CJKPhaq1GhDQ5u66urbHjGI
     -->  ATTTCTCTGTA {u'CJKPhaq1GhDrrMCJ0_zBuKkB': 42, u'CJKPhaq1GhDQ5u66urbHjGI': 143}

Notice that we consistently see greater read-depth in the DNA-seq data.  Also all but the last sample are heterozygous for the V600 mutation, while ``WM1799_SKIN`` is homozygous.  (Of course a proper analysis would also take into consideration the cigar information that is available with each read as well.)_____no_output_____