# Notebook from SophiaXiaoxingFang/Udacity_iSDC
Path: P3-Implement-Route-Planner/project_notebook.ipynb

# Implementing a Route Planner
In this project you will use A\* search to implement a "Google-maps" style route planning algorithm._____no_output_____## The Map_____no_output_____

# Run this cell first!

from helpers import Map, load_map_10, load_map_40, show_map
import math

%load_ext autoreload
%autoreload 2_____no_output_____

### Map Basics_____no_output_____

map_10 = load_map_10()
show_map(map_10)_____no_output_____

The map above (run the code cell if you don't see it) shows a disconnected network of 10 intersections. The two intersections on the left are connected to each other but they are not connected to the rest of the road network. This map is quite literal in its expression of distance and connectivity. On the graph above, the edge between 2 nodes(intersections) represents a literal straight road not just an abstract connection of 2 cities.

These `Map` objects have two properties you will want to use to implement A\* search: `intersections` and `roads`

**Intersections**

The `intersections` are represented as a dictionary.

In this example, there are 10 intersections, each identified by an x,y coordinate. The coordinates are listed below. You can hover over each dot in the map above to see the intersection number._____no_output_____

map_10.intersections_____no_output_____

**Roads**

The `roads` property is a list where `roads[i]` contains a list of the intersections that intersection `i` connects to._____no_output_____

# this shows that intersection 0 connects to intersections 7, 6, and 5
map_10.roads[0] _____no_output_____# This shows the full connectivity of the map
map_10.roads_____no_output_____# map_40 is a bigger map than map_10
map_40 = load_map_40()
show_map(map_40)_____no_output_____

### Advanced Visualizations

The map above shows a network of roads which spans 40 different intersections (labeled 0 through 39).

The `show_map` function which generated this map also takes a few optional parameters which might be useful for visualizing the output of the search algorithm you will write.

* `start` - The "start" node for the search algorithm.
* `goal`  - The "goal" node.
* `path`  - An array of integers which corresponds to a valid sequence of intersection visits on the map._____no_output_____

# run this code, note the effect of including the optional
# parameters in the function call.
show_map(map_40, start=5, goal=34, path=[5,16,37,12,34])_____no_output_____

## The Algorithm
### Writing your algorithm
The algorithm written will be responsible for generating a `path` like the one passed into `show_map` above. In fact, when called with the same map, start and goal, as above you algorithm should produce the path `[5, 16, 37, 12, 34]`. However you must complete several methods before it will work.

```bash
> PathPlanner(map_40, 5, 34).path
[5, 16, 37, 12, 34]
```_____no_output_____### PathPlanner class

The below class is already partly implemented for you - you will implement additional functions that will also get included within this class further below.

Let's very briefly walk through each part below.

`__init__` - We initialize our path planner with a map, M, and typically a start and goal node. If either of these are `None`, the rest of the variables here are also set to none. If you don't have both a start and a goal, there's no path to plan! The rest of these variables come from functions you will soon implement.
- `closedSet` includes any explored/visited nodes.
- `openSet` are any nodes on our frontier for potential future exploration.
- `cameFrom` will hold the previous node that best reaches a given node
- `gScore` is the `g` in our `f = g + h` equation, or the actual cost to reach our current node
- `fScore` is the combination of `g` and `h`, i.e. the `gScore` plus a heuristic; total cost to reach the goal
- `path` comes from the `run_search` function, which is already built for you.

`reconstruct_path` - This function just rebuilds the path after search is run, going from the goal node backwards using each node's `cameFrom` information.

`_reset` - Resets *most* of our initialized variables for PathPlanner. This *does not* reset the map, start or goal variables, for reasons which you may notice later, depending on your implementation.

`run_search` - This does a lot of the legwork to run search once you've implemented everything else below. First, it checks whether the map, goal and start have been added to the class. Then, it will also check if the other variables, other than `path` are initialized (note that these are only needed to be re-run if the goal or start were not originally given when initializing the class, based on what we discussed above for `__init__`.

From here, we use a function you will implement, `is_open_empty`, to check that there are still nodes to explore (you'll need to make sure to feed `openSet` the start node to make sure the algorithm doesn't immediately think there is nothing to open!). If we're at our goal, we reconstruct the path. If not, we move our current node from the frontier (`openSet`) and into explored (`closedSet`). Then, we check out the neighbors of the current node, check out their costs, and plan our next move.

This is the main idea behind A*, but none of it is going to work until you implement all the relevant parts, which will be included below after the class code._____no_output_____

# Do not change this cell
# When you write your methods correctly this cell will execute
# without problems
class PathPlanner():
    """Construct a PathPlanner Object"""
    def __init__(self, M, start=None, goal=None):
        """ """
        self.map = M
        self.start= start
        self.goal = goal
        self.closedSet = self.create_closedSet() if goal != None and start != None else None
        self.openSet = self.create_openSet() if goal != None and start != None else None
        self.cameFrom = self.create_cameFrom() if goal != None and start != None else None
        self.gScore = self.create_gScore() if goal != None and start != None else None
        self.fScore = self.create_fScore() if goal != None and start != None else None
        self.path = self.run_search() if self.map and self.start != None and self.goal != None else None

    def reconstruct_path(self, current):
        """ Reconstructs path after search """
        total_path = [current]
        while current in self.cameFrom.keys():
            current = self.cameFrom[current]
            total_path.append(current)
        return total_path

    def _reset(self):
        """Private method used to reset the closedSet, openSet, cameFrom, gScore, fScore, and path attributes"""
        self.closedSet = None
        self.openSet = None
        self.cameFrom = None
        self.gScore = None
        self.fScore = None
        self.path = self.run_search() if self.map and self.start and self.goal else None

    def run_search(self):
        """ """
        if self.map == None:
            raise(ValueError, "Must create map before running search. Try running PathPlanner.set_map(start_node)")
        if self.goal == None:
            raise(ValueError, "Must create goal node before running search. Try running PathPlanner.set_goal(start_node)")
        if self.start == None:
            raise(ValueError, "Must create start node before running search. Try running PathPlanner.set_start(start_node)")

        self.closedSet = self.closedSet if self.closedSet != None else self.create_closedSet()
        self.openSet = self.openSet if self.openSet != None else  self.create_openSet()
        self.cameFrom = self.cameFrom if self.cameFrom != None else  self.create_cameFrom()
        self.gScore = self.gScore if self.gScore != None else  self.create_gScore()
        self.fScore = self.fScore if self.fScore != None else  self.create_fScore()

        while not self.is_open_empty():
            current = self.get_current_node()

            if current == self.goal:
                self.path = [x for x in reversed(self.reconstruct_path(current))]
                return self.path
            else:
                self.openSet.remove(current)
                self.closedSet.add(current)

            for neighbor in self.get_neighbors(current):
                if neighbor in self.closedSet:
                    continue    # Ignore the neighbor which is already evaluated.

                if not neighbor in self.openSet:    # Discover a new node
                    self.openSet.add(neighbor)

                # The distance from start to a neighbor
                #the "dist_between" function may vary as per the solution requirements.
                if self.get_tentative_gScore(current, neighbor) >= self.get_gScore(neighbor):
                    continue        # This is not a better path.

                # This path is the best until now. Record it!
                self.record_best_path_to(current, neighbor)
        print("No Path Found")
        self.path = None
        return False_____no_output_____

## Your Turn

Implement the following functions to get your search algorithm running smoothly!_____no_output_____### Data Structures

The next few functions requre you to decide on data structures to use - lists, sets, dictionaries, etc. Make sure to think about what would work most efficiently for each of these. Some can be returned as just an empty data structure (see `create_closedSet()` for an example), while others should be initialized with one or more values within._____no_output_____

def create_closedSet(self):
    """ Creates and returns a data structure suitable to hold the set of nodes already evaluated"""
    # EXAMPLE: return a data structure suitable to hold the set of nodes already evaluated
    return set()_____no_output_____def create_openSet(self):
    """ Creates and returns a data structure suitable to hold the set of currently discovered nodes
    that are not evaluated yet. Initially, only the start node is known."""
    if self.start != None:
        # TODO: return a data structure suitable to hold the set of currently discovered nodes
        # that are not evaluated yet. Make sure to include the start node.
        open_set = set()
        open_set.add(self.start) #Adding the start node to openSet
        return open_set

    raise(ValueError, "Must create start node before creating an open set. Try running PathPlanner.set_start(start_node)")_____no_output_____def create_cameFrom(self):
    """Creates and returns a data structure that shows which node can most efficiently be reached from another,
    for each node."""
    # TODO: return a data structure that shows which node can most efficiently be reached from another,
    # for each node.
    return {} _____no_output_____def create_gScore(self):
    """Creates and returns a data structure that holds the cost of getting from the start node to that node,
    for each node. The cost of going from start to start is zero."""
    # TODO:  return a data structure that holds the cost of getting from the start node to that node, for each node.
    # for each node. The cost of going from start to start is zero. The rest of the node's values should
    # be set to infinity.
    gScore = {}
    for i in range(len(self.map.intersections)):
        gScore[i] = float('inf')
    gScore[self.start] = 0
    return gScore_____no_output_____def create_fScore(self):
    """Creates and returns a data structure that holds the total cost of getting from the start node to the goal
    by passing by that node, for each node. That value is partly known, partly heuristic.
    For the first node, that value is completely heuristic."""
    # TODO: return a data structure that holds the total cost of getting from the start node to the goal
    # by passing by that node, for each node. That value is partly known, partly heuristic.
    # For the first node, that value is completely heuristic. The rest of the node's value should be
    # set to infinity.
    fScore = {}
    for i in range(len(self.map.intersections)):
        fScore[i] = float('inf')
    fScore[self.start] = self.heuristic_cost_estimate(self.start)
    return fScore
_____no_output_____

### Set certain variables

The below functions help set certain variables if they weren't a part of initializating our `PathPlanner` class, or if they need to be changed for anothe reason._____no_output_____

def set_map(self, M):
    """Method used to set map attribute """
    self._reset(self)
    self.start = None
    self.goal = None
    # TODO: Set map to new value.
    self.map = M
_____no_output_____def set_start(self, start):
    """Method used to set start attribute """
    self._reset(self)
    # TODO: Set start value. Remember to remove goal, closedSet, openSet, cameFrom, gScore, fScore,
    # and path attributes' values.
    self.start = start
    self.goal = None_____no_output_____def set_goal(self, goal):
    """Method used to set goal attribute """
    self._reset(self)
    # TODO: Set goal value.
    self.goal = goal_____no_output_____

### Get node information

The below functions concern grabbing certain node information. In `is_open_empty`, you are checking whether there are still nodes on the frontier to explore. In `get_current_node()`, you'll want to come up with a way to find the lowest `fScore` of the nodes on the frontier. In `get_neighbors`, you'll need to gather information from the map to find the neighbors of the current node._____no_output_____

def is_open_empty(self):
    """returns True if the open set is empty. False otherwise. """
    # TODO: Return True if the open set is empty. False otherwise.
    return self.openSet == None_____no_output_____def get_current_node(self):
    """ Returns the node in the open set with the lowest value of f(node)."""
    # TODO: Return the node in the open set with the lowest value of f(node).
    return min(self.openSet, key = lambda node: self.calculate_fscore(node))_____no_output_____def get_neighbors(self, node):
    """Returns the neighbors of a node"""
    # TODO: Return the neighbors of a node
    return set(self.map.roads[node])_____no_output_____

### Scores and Costs

Below, you'll get into the main part of the calculation for determining the best path - calculating the various parts of the `fScore`._____no_output_____

def get_gScore(self, node):
    """Returns the g Score of a node"""
    # TODO: Return the g Score of a node
    return self.gScore.get(node)_____no_output_____def distance(self, node_1, node_2):
    """ Computes the Euclidean L2 Distance"""
    # TODO: Compute and return the Euclidean L2 Distance
    x1 = self.map.intersections[node_1]
    x2 = self.map.intersections[node_2]
    return math.sqrt(math.pow(x2[0] - x1[0], 2) + math.pow(x2[1] - x1[1], 2))_____no_output_____def get_tentative_gScore(self, current, neighbor):
    """Returns the tentative g Score of a node"""
    # TODO: Return the g Score of the current node
    # plus distance from the current node to it's neighbors
    return self.get_gScore(current) + self.distance(current, neighbor)_____no_output_____def heuristic_cost_estimate(self, node):
    """ Returns the heuristic cost estimate of a node """
    # TODO: Return the heuristic cost estimate of a node
    return distance(self, node, self.goal)_____no_output_____def calculate_fscore(self, node):
    """Calculate the f score of a node. """
    # TODO: Calculate and returns the f score of a node.
    # REMEMBER F = G + H
    return self.get_gScore(node) + self.heuristic_cost_estimate(node)_____no_output_____

### Recording the best path

Now that you've implemented the various functions on scoring, you can record the best path to a given neighbor node from the current node!_____no_output_____

def record_best_path_to(self, current, neighbor):
    """Record the best path to a node """
    # TODO: Record the best path to a node, by updating cameFrom, gScore, and fScore
    self.cameFrom[neighbor] = current
    self.gScore[neighbor]=self.get_tentative_gScore(current,neighbor)
    self.fScore[current]=self.calculate_fscore(current)_____no_output_____

### Associating your functions with the `PathPlanner` class

To check your implementations, we want to associate all of the above functions back to the `PathPlanner` class. Python makes this easy using the dot notation (i.e. `PathPlanner.myFunction`), and setting them equal to your function implementations. Run the below code cell for this to occur.

*Note*: If you need to make further updates to your functions above, you'll need to re-run this code cell to associate the newly updated function back with the `PathPlanner` class again!_____no_output_____

# Associates implemented functions with PathPlanner class
PathPlanner.create_closedSet = create_closedSet
PathPlanner.create_openSet = create_openSet
PathPlanner.create_cameFrom = create_cameFrom
PathPlanner.create_gScore = create_gScore
PathPlanner.create_fScore = create_fScore
PathPlanner.set_map = set_map
PathPlanner.set_start = set_start
PathPlanner.set_goal = set_goal
PathPlanner.get_current_node = get_current_node
PathPlanner.get_neighbors = get_neighbors
PathPlanner.get_gScore = get_gScore
PathPlanner.get_tentative_gScore = get_tentative_gScore
PathPlanner.is_open_empty = is_open_empty
PathPlanner.distance = distance
PathPlanner.heuristic_cost_estimate = heuristic_cost_estimate
PathPlanner.calculate_fscore = calculate_fscore
PathPlanner.record_best_path_to = record_best_path_to_____no_output_____

### Preliminary Test

The below is the first test case, just based off of one set of inputs. If some of the functions above aren't implemented yet, or are implemented incorrectly, you likely will get an error from running this cell. Try debugging the error to help you figure out what needs further revision!_____no_output_____

planner = PathPlanner(map_40, 5, 34)
path = planner.path
if path == [5, 16, 37, 12, 34]:
    print("great! Your code works for these inputs!")
else:
    print("something is off, your code produced the following:")
    print(path)great! Your code works for these inputs!

#### Visualize

Once the above code worked for you, let's visualize the results of your algorithm!_____no_output_____

# Visualize your the result of the above test! You can also change start and goal here to check other paths
start = 10
goal = 36

show_map(map_40, start=start, goal=goal, path=PathPlanner(map_40, start, goal).path)_____no_output_____

### Testing your Code
If the code below produces no errors, your algorithm is behaving correctly. You are almost ready to submit! Before you submit, go through the following submission checklist:

**Submission Checklist**

1. Does my code pass all tests?
2. Does my code implement `A*` search and not some other search algorithm?
3. Do I use an **admissible heuristic** to direct search efforts towards the goal?
4. Do I use data structures which avoid unnecessarily slow lookups?

When you can answer "yes" to all of these questions, and also have answered the written questions below, submit by pressing the Submit button in the lower right!_____no_output_____

from test import test

test(PathPlanner)All tests pass! Congratulations!

## Questions

**Instructions**

Answer the following questions in your own words. We do not you expect you to know all of this knowledge on the top of your head. We expect you to do research and ask question. However do not merely copy and paste the answer from a google or stackoverflow. Read the information and understand it first. Then use your own words to explain the answer._____no_output_____---
How would you explain A-Star to a family member(layman)?

**ANSWER**: Given a start point and a destination, the A* algorithm is able to find the shortest path with the minimal cost. The secret to its success is that in each step it combines two information: the exact cost of the path from the starting point to the current point, and the estimated cost from the current point to the goal.
_____no_output_____---
How does A-Star search algorithm differ from Uniform cost search? What about Best First search?

**ANSWER**: Unlike the Uniform cost search expands out equally in all directions, unlike the Best First search does not consider the cost of the path to the current state, A* search consider both the estimate cost towords to the goal and the exact cost. A* combines the advantages of Best-first Search and Uniform Cost Search, in which f(n) = g(n) + h(n) is used with h(n) being the estimated distance between any random vertex n and target vertex, g(n) being the actual distance between the start point and any vertex n. If g(n)=0, the A* turns to be Best First search. If h(n)=0, then A* turns to be Uniform cost search._____no_output_____---
What is a heuristic?

**ANSWER**: Heuristic means involving or serving as an aid to learning, discovery, or problem-solving by experimental and especially trial-and-error methods. In A* algorithm, it means the estimation of the distance to the destination._____no_output_____---
What is a consistent heuristic?

**ANSWER**:Consistent heuristic is when if its estimate is always less than or equal to the estimated distance from any neighboring vertex to the goal, plus the cost of reaching that neighbor._____no_output_____---
What is a admissible heuristic?

**ANSWER**: Admissible heuristic means the estimated cost to the goal is never more than the actual cost from the current node to the goal node, i.e., it never overestimates the cost of reaching the goal.
_____no_output_____---
___ admissible heuristic are consistent.

*CHOOSE ONE*
    - All
    - Some
    - None

**ANSWER**:Some_____no_output_____---
___ Consistent heuristic are admissible.

*CHOOSE ONE*
    - All
    - Some
    - None

**ANSWER**:All_____no_output_____---_____no_output_____


# Notebook from alexandonian/lightning
Path: Lightning.ipynb

from IPython.core.display import HTML
HTML('custom.css')_____no_output_____

Welcome to Lightning

Lightning is intended to be an open-source library for integrating and analyzing spatial data obtained from a variety of imaging modalities. Lightning will hopefully be capable of working with small, medium, and very large data sets. Like its sibling, Thunder, it will expose array operations through either local or distributed implementations with a common interface, and makes it easy to switch between them. Its distributed implementation currently targets Spark, a powerful cluster computing framework.

To see the internals of Lightning, visit the project page.

Tutorials

These notebooks are interactive tutorials that show how Lightning works:

Basic usage

_____no_output_____
Introduction
_____no_output_____
The overarching theme behind Lightning is 'fusion' at multiple scales: fusion of imaging data, fusion of the latest algorithms/tools, fusion of its applications. The original motivation behind Lightning was to develop and test spatial intratumor heterogeneity metrics that correlate with particular cancer subtypes using deep neural networks. Our particular goal was to establish an image analysis pipeline that would provide end-to-end, automated classification of multiplexed immunofluorescence images obtained from tissue microarray sections. However, since we are interested in more than tissue microarray data and classification tasks, our project has quickly expanded into the development of a pipeline for integrating and analyzing data along the imaging continuum. We hope to include:

Light microscopy

Fluorescent and immunofluorescent (IF) imaging

Immunohistochemistry (IHC)

Histopathology (H&E)

Mass-Spec imaging (IMS)

In-situ hybridization imaging (ISH)

To facilitate the formation and analysis of these multimodal images, we will use a ‘fusion’ of the latest, state-of-the-art algorithms and tools. These include:

EM patch-based CNNs (summer 2016)

Spatial statistics (summer, fall 2016)

Nonlinear image registration - registration is pre-req for image fusion (fall 2016, winter 2017)

Image Fusion using multivariate regression

Interactive visualization - for clinical use

Lightning is not meant to be used strictly by the researcher to study the structure and progression of diseased tissue. The hope is for Lightning to be used as a powerful visualization tool and diagnostic advisor for the clinician.

_____no_output_____
Rethinking the Image in Lightning:

When we think about the structure of a “traditional” grayscale image, we typically imaging a 2D array representing the spatial dimensions height and width with each element representing the intensity at that spatial location. For color images, we add a third dimension to accommodate the three color channels red, green and blue.

In contrast, most imaging modalities natively deliver their measurements as an $n$th order tensor (multidimensional array of values that can be accessed via $n$ indices). More specifically, an imaging modality that records along $n_s$ spatial dimensions and acquires an $n_m$  dimensional array at each measurement location delivers an $n = n_s + n_m$ order tensor. To ground this discussion, typical examples include:

\\(n_s = 0\\): a single measurement

$n_s = 1$: a list of measurements

$n_s = 2$: 2D imaging (e.g. light microscopy)

$n_s = 3$: 3D imaging (e.g. MRI)

Similarly, the number of measurement modes can also vary:

 $n_m = 0$ scalar value measured (e.g.grayscale image)

 $n_m = 1$ : vector of measured values (e.g. RGB image when len(vector) = 3)

 $n_m = 2$ : array of measured values at locations

And so on..

 In most cases, we will be dealing with 3rd order tensors, (i.e. two spatial dimensions and a 1 dimensional measurement vector).

Internal representation of an image (Lightning)

In order to accommodate data acquired from a variety of imaging modalities, we will apply a “flattening” procedure to convert data into a tabular format (2D array). Each row in this array will represent a particular spatial location and each column will represent a particular feature or measurement made across all spatial locations. (I believe this will also lend itself nicely to Spark computations).

Figure 1. Example of the flattening procedure. An RGB image with 2 spatial dimensions and one measurement dimension with three channels (red, green and blue) is converted to a 2D array. The rows represent individual pixels and the colums correspond red, green and blue measurements at that pixel.

 Advantage: In this way, an image can be viewed more as a detailed description of the “state” of a tissue in a way that makes use of the strong suits of each imaging modality. For example, we can enhance the spatial specificity of mass-spec imaging, which inherently has high chemical specificity, by fusing with H&E images. We can start to incorporate genetic information such as maps of gene expression into our description of the state of the tissue.

 Handling the Image Registration Problem (Lightning-Register)

We are interested in "image-fusion" approaches to combining data across modalities, including integrating with genomics data in the long run. However, image registration is a pre-requisite for any modern approach to image fusion [1]. Thus, we will eventually need a solution to the image registration problem.

To the best of my knowledge, the only cases where you could fuse without registering are instances where the same section of tissue has been imaged multiple times for different features (these are aligned by definition). If the features of interest require a different labeling/staining technique, it would be possible to differentially label neighboring sections - but now registration is needed again. Even this approach seems inefficient and complicated to do at a large scale. Having image-registration would also allow for inter-subject comparisons.

 Handling the Multi-resolution Problem (Lightning-Fuse)

Now that the images are aligned, the spatial domian is common to all data sources. Even still, it may be difficult to establish a one-to-one mapping between measurements of different technical origin due to the different spatial resolution scales. The naive approach would be to downsample images to the match the modality with the lowest spatial approach. A better approach would be to apply a one-to-many mapping whereby many observations in a higher resolution modality is mapped to a single observation in a lower resolution modality (for more info see Raf Van De Plas 2015)

_____no_output_____
Image Analysis

The intent is to inlucude the following:

Spatial Statistics as Description of ITH

(Summer 2016)

Automatated, end-to-end Classification of Cancer Subtypes

(Summer 2016)

Computational Anatomy

(Fall 2016, Winter 2017)_____no_output_____
Proposed Pipeline

_____no_output_____
 Provider

The Provider is meant to bridge the gap between the db and individual image controllers. With the help of the dbHelper and knowledge of the database schema, the provider will make queries to the db and pass all the relevant to an image controller. Users/frontend will interact mostly with the provider to load dataset.

 ImageController

The ImageController is mean to handle everything related to an individual image, from calling alignment and fusing modules to orchestrating the patching process.

 The Controller will maintain references to:

the image array

the patches array

the features array

indexing labels about images, patches and features (e.g. channel/subchannel lists, features of interest etc).

Class information (e.g. does it belong to cohort 1?)

_____no_output_____
References
_____no_output_____1. Van de Plas, Raf, et al. "Image fusion of mass spectrometry and microscopy: a multimodality paradigm for molecular tissue mapping." Nature methods12.4 (2015): 366-372.
_____no_output_____


# Notebook from ckm3/eleanor
Path: docs/getting_started/tips_tricks.ipynb

### Tips, Tricks, and Hacks_____no_output_____In the tutorial notebook, we learned how to make light curves for stars observed in a single sector or in multiple sectors. In this notebook we'll explore how to make the most of these data and ensure your signal is real._____no_output_____#### 1.1 Choosing an Aperture_____no_output_____Let's begin with the same WASP-100 we used in the quickstart tutorial. I'm going to hide warnings too, just to make it easier to see what's going on._____no_output_____

from IPython.display import Image

import warnings
warnings.filterwarnings('ignore')_____no_output_____import eleanor
import numpy as np
import matplotlib.pyplot as plt_____no_output_____star = eleanor.Source(tic=38846515, sector=1)
data = eleanor.TargetData(star, height=15, width=15, bkg_size=31, do_psf=True, do_pca=True)
q = data.quality == 0INFO: Found cached file /Users/bmontet/.eleanor/mastDownload/HLSP/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_pc/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_bkg.fits with expected size 78955200. [astroquery.query]
INFO: Found cached file /Users/bmontet/.eleanor/mastDownload/HLSP/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_pc/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_pc.fits with expected size 158022720. [astroquery.query]
INFO: Found cached file /Users/bmontet/.eleanor/mastDownload/HLSP/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_pc/hlsp_eleanor_tess_ffi_postcard-s0001-4-1_tess_v2_pm.txt with expected size 237847. [astroquery.query]
WARNING:tensorflow:From /Users/bmontet/research/tess/eleanor/eleanor/targetdata.py:837: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From /Users/bmontet/research/tess/eleanor/eleanor/targetdata.py:837: The name tf.logging.ERROR is deprecated. Please use tf.compat.v1.logging.ERROR instead.

What if you're not satisfied with $\texttt{eleanor}$'s default choice of aperture? Well, we provide you with three ways to explore different apertures.

(1) All of the apertures $\texttt{eleanor}$ tries behind the scenes are saved! So are their raw and corrected light curves. These are stored as ```data.all_apertures```, ```data.all_raw_flux```, and ```data.all_corr_flux```, respectively. We can explore these options by calling the following, for example._____no_output_____

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(15,4), gridspec_kw={'width_ratios':[1,3]})
ax1.imshow(data.tpf[0])
ax1.imshow(data.all_apertures[0], cmap='Greys', alpha=0.7)
ax1.set_title('Aperture over TPF')

ax2.plot(data.time[q], data.all_raw_flux[0][q]/np.nanmedian(data.all_raw_flux[0][q]), 'k', label='Raw')
ax2.plot(data.time[q], data.all_corr_flux[0][q]/np.nanmedian(data.all_corr_flux[0][q]) - 0.015, 'r', label='Corrected')
ax2.set_xlabel('Time [BJD - 2457000]')
ax2.set_ylabel('Normalized Flux')
ax2.legend();_____no_output_____

(2) $\texttt{eleanor}$ can help you create your own mask. By calling `eleanor.custom_aperture()`, we can choose from a circular or rectangular aperture. We can also choose the size (radius or length x width) and the pixel the aperture is centered on by passing in a list of $\textit{(x,y)}$ to `pos`. The aperture will only be created on pixels within the TPF (it won't spill over to other pixels). Calling `eleanor.custom_aperture()` overwrites `eleanor.TargetData.aperture`, `eleanor.TargetData.raw_flux`, and `eleanor.TargetData.corr_flux`._____no_output_____

eleanor.TargetData.custom_aperture(data, shape='circle', r=2, pos=[7,6], method='exact')
eleanor.TargetData.get_lightcurve(data)

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(15,4), gridspec_kw={'width_ratios':[1,3]})
ax1.imshow(data.tpf[0])
ax1.imshow(data.aperture, cmap='Greys', alpha=0.5)
ax1.set_title('Aperture over TPF')
#plt.imshow(data.aperture)

ax2.plot(data.time[q], data.raw_flux[q]/np.nanmedian(data.raw_flux[q]), 'k', label='Raw')
ax2.plot(data.time[q], data.corr_flux[q]/np.nanmedian(data.corr_flux[q]) - 0.015, 'r', label='Corrected')
ax2.legend()
ax2.set_xlabel('Time [BJD - 2457000]')
ax2.set_ylabel('Normalized Flux');_____no_output_____

(3) We can pass in our own mask by creating a 2D array of the same shape as the TPF and calling eleanor.TargetData.get_lightcurve(aperture=mask)_____no_output_____

mask = np.zeros(np.shape(data.tpf[0]))
mask[6:8,6:8] = 1
data.get_lightcurve(aperture=mask)

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(15,4), gridspec_kw={'width_ratios':[1,3]})
ax1.imshow(data.tpf[0])
ax1.imshow(mask, cmap='Greys', alpha=0.5)
ax1.set_title('Aperture over TPF')
#plt.imshow(data.aperture)

ax2.plot(data.time[q], data.raw_flux[q]/np.nanmedian(data.raw_flux[q]), 'k', label='Raw')
ax2.plot(data.time[q], data.corr_flux[q]/np.nanmedian(data.corr_flux[q]) - 0.035, 'r', label='Corrected')
ax2.legend()
ax2.set_xlabel('Time [BJD - 2457000]')
ax2.set_ylabel('Normalized Flux');_____no_output_____

#### 1.2 Adding or Removing Flux Correction Terms_____no_output_____When we call eleanor.TargetData(), some simple systematics corrections are automatically performed on the light curve. Let's apply those explicitly to the newly created raw light curve from the custom aperture and see how we can change them._____no_output_____

corr_flux = eleanor.TargetData.corrected_flux(data, flux=data.raw_flux)_____no_output_____plt.figure(figsize=(15,5))
plt.plot(data.time[q], corr_flux[q], 'k')

plt.ylabel('Flux', fontsize=24)
plt.xlabel('Time', fontsize=24);_____no_output_____

##### 1.2.1 Changing quality flags_____no_output_____If we think certain data are bad, but not quality flagged, we can give them a different quality label and re-run this function, causing them to be ignored so they do not influence other cadences. By default, all highest quality data have flag zero and all other flags are positive. Let's throw out a four-day chunk temporarily:_____no_output_____

data.quality[q][(data.time[q] > 1333) & (data.time[q] < 1337)] = -8675309
corr_flux = eleanor.TargetData.corrected_flux(data, flux=data.raw_flux)

q = data.quality <= 0.5

plt.figure(figsize=(15,5))
plt.plot(data.time[q], corr_flux[q], 'k')

plt.ylabel('Flux', fontsize=24)
plt.xlabel('Time', fontsize=24);
_____no_output_____

##### 1.2.2 Removing terms from the model_____no_output_____By default, the ```corrected_flux``` function removes signals that are correlated with the position of the star on the detector, with common modes of nearby stars, and with the background. We can stop these from being a part of ```corrected_flux``` by setting them to zero._____no_output_____

data.cbvs = np.zeros_like(data.cbvs)
data.centroid_xs = np.zeros_like(data.centroid_xs)
data.centroid_ys = np.zeros_like(data.centroid_ys)

corr_flux = eleanor.TargetData.corrected_flux(data, flux=data.raw_flux)

plt.figure(figsize=(15,5))
plt.plot(data.time[q], corr_flux[q], 'k')

plt.ylabel('Flux', fontsize=24)
plt.xlabel('Time', fontsize=24);_____no_output_____

In this case, we have a noisier light curve that looks more like the raw flux._____no_output_____We can always get back to where we started by just re-running ```eleanor.TargetData()``` with the same call we used initially._____no_output_____

data = eleanor.TargetData(star, height=15, width=15, bkg_size=31, do_psf=True, do_pca=True)100%|██████████| 1282/1282 [00:11<00:00, 114.33it/s]

##### 1.2.3 Adding new terms to the model_____no_output_____What if you have additional information that you believe might be correlated with the star's raw flux? You can add these in to the ```corrected_flux``` function as ```regressors```. Any $N$ regressors can be passed through as an array with shape (```data.time, N```)._____no_output_____For example, we've found that often, some of the parameters that describe the shape of the PSF make very good regressors to produce a clean, systematics-corrected light curve:_____no_output_____

reg_array = np.array([data.psf_b, data.psf_c, data.psf_x]).T
corr_flux = eleanor.TargetData.corrected_flux(data, flux=data.raw_flux, regressors=reg_array)

plt.figure(figsize=(15,5))
plt.plot(data.time[q], corr_flux[q], 'k')

plt.ylabel('Flux', fontsize=24)
plt.xlabel('Time', fontsize=24);_____no_output_____

You could also regress against nearby pixels, other stars, random noise, or any other length ```data.time``` array you can imagine._____no_output_____#### 1.3 Making a PSF-based light curve_____no_output_____By setting ```do_psf = True``` in your call to ```eleanor.TargetData()```, you can model the PSF as a single 2D Gaussian that can change shape or position from cadence to cadence._____no_output_____

eleanor.TargetData.psf_lightcurve(data, model='gaussian')100%|██████████| 1282/1282 [00:10<00:00, 118.43it/s]
lk = data.to_lightkurve(flux=data.psf_flux)
lk.plot();_____no_output_____

There are a few tricks we can use to improve this light curve. This is an incomplete list, check the docs for full details._____no_output_____For this star, there are two bright stars in the TPF. We can model them both simultaneously by passing through ```nstars=2``` and giving an initial guess on their positions. The code will keep the relative positions of the two stars fixed, moving both of them from cadence to cadence in the same way._____no_output_____

eleanor.TargetData.psf_lightcurve(data, model='gaussian', nstars=2, xc=[3.1, 7.0], yc=[6.8, 6.2])100%|██████████| 1282/1282 [00:14<00:00, 87.74it/s]
plt.figure(figsize=(15,5))
plt.plot(data.time[q], data.all_psf[q,0]/np.nanmedian(data.all_psf[q,0])+0.015, 'r')
plt.plot(data.time[q], data.all_psf[q,1]/np.nanmedian(data.all_psf[q,1]), 'k')

plt.ylabel('Flux', fontsize=24)
plt.xlabel('Time', fontsize=24);_____no_output_____

In this case, we can tell very obviously what star the transit is on!_____no_output_____We can also pass through a different error array, request a Moffat profile model, a Poisson likelihood function, or even pass through a different TPF! _____no_output_____#### 1.4 Running eleanor locally/without internet_____no_output_____With the default settings, `eleanor` will attempt to find the proper postcard for your target, as well as other data about your system, through MAST. If you don't have internet access, but have downloaded your postcard already (if you've made a light curve for your star of interest previously, you may already have the postcard in your `~/.eleanor` directory), you can avoid all need for the internet with a few simple tricks._____no_output_____All you need to do are pass through `local = True` in your call to `eleanor.Source()` and give the directory of your postcard and pointing model. By default, these are downloaded to the same directory. If you don't pass through a pointing model directory, it will assume it's in the same place as your postcard. I've put mine in `../../testdir1` and `../../testdir2`, respectively._____no_output_____You also need to pass through your star's coordinates. Normally you can pass through one of a name, TIC ID, Gaia DR2, ID, or coordinates. In these cases, behind the scenes `eleanor` will use any one of these to get the rest of this information, but behind the scenes relies on the coordinates. You need to pass through the other values as well, but just so they're set to something so `eleanor` doesn't try to look them up. You can set them to zero, that's fine._____no_output_____

coords = (68.959732, -64.02704)
star = eleanor.Source(coords=coords, tic=0, gaia=0, sector=1, local=True, post_dir='./testdir1', pm_dir='./testdir2')
data = eleanor.TargetData(star, do_pca=True, do_psf=False)_____no_output_____plt.figure(figsize=(15,5))

q = data.quality == 0
plt.plot(data.time[q], corr_flux[q], 'k')

plt.ylabel('Flux', fontsize=24)
plt.xlabel('Time', fontsize=24);_____no_output_____

The only thing that will be missing is `star.tess_mag`, but you can set that yourself too before you run `data.save()` and it will be recorded properly._____no_output_____

star.tess_mag = 10.53_____no_output_____

#### 1.5 Targets without TIC IDs_____no_output_____When you pass through ```coords``` into `eleanor.Source()`, it will use these coordinates to find the TIC entry at that location. However, some targets (often very faint or non-stellar in nature) do not appear in the TIC. This will produce an error that inhibits ```eleanor.Source()``` from running successfully. However, as we noted in the previous section, `eleanor` does not use the TIC ID behind the scenes. Passing through a false TIC ID and the proper coordinates will work just fine._____no_output_____#### 1.6 Saturated targets_____no_output_____`eleanor` tests a variety of apertures and selects the one with the lowest CDPP. Very saturated targets ($T < 6-7$, depending on location on the detector) have many pixels at the saturation limit. Choosing a small number of saturated pixels will then produce a very stable light curve, leading `eleanor` to choose very small apertures that do not represent the true astrophysical variability from these sources. To make light curves from these, we need to use larger apertures. This is possible!_____no_output_____

star = eleanor.Source(name='alpha draconis', sector=15)
data = eleanor.TargetData(star, height=31, width=13)No eleanor postcard has been made for your target (yet). Using TessCut instead.
plt.imshow(data.tpf[300]);_____no_output_____

Yes, that's a saturated target. Let's define our own aperture and go forth._____no_output_____

ap = np.zeros_like(data.tpf[300])

ap[:,4:11] = 1

data.get_lightcurve(aperture=ap)
q = data.quality == 0

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(15,4), gridspec_kw={'width_ratios':[1,3]})
ax1.imshow(data.tpf[0])
ax1.imshow(data.aperture, cmap='Greys', alpha=0.5)
ax1.set_title('Aperture over TPF')
#plt.imshow(data.aperture)

ax2.plot(data.time[q], data.corr_flux[q]/np.nanmedian(data.corr_flux[q]) - 0.0, 'k')
ax2.set_ylim(0.999, 1.0004)
ax2.set_xlabel('Time [BJD - 2457000]')
ax2.set_ylabel('Normalized Flux');_____no_output_____

Alpha draconis, with $V=3.7$, has a deep secondary eclipse as reported in Bedding et al. (2020), and the star is so bright the photon noise is at the few ppm level on 30-minute timescales.

It often works very well for bright, saturated targets to use additional regressors, drawing on the corner pixels of the TPF to use on the detrending. An example of this is below._____no_output_____#### 1.7 Putting it all together: recovering $\pi$ Men c_____no_output_____Let's make sure we can recover the planet discovered with TESS data around the bright star $\pi$ Mensae, combining what we have learned about saturated targets with the addition of regressing against corner pixels in the TPF for improved background subtraction._____no_output_____

star = eleanor.Source(name='pi Mensae', sector=1)
data = eleanor.TargetData(star, height=21, width=13)INFO: Found cached file /Users/bmontet/.eleanor/mastDownload/HLSP/hlsp_eleanor_tess_ffi_postcard-s0001-4-2-cal-1588-0268_tess_v2_pc/hlsp_eleanor_tess_ffi_postcard-s0001-4-2-cal-1588-0268_tess_v2_bkg.fits with expected size 78955200. [astroquery.query]
INFO: Found cached file /Users/bmontet/.eleanor/mastDownload/HLSP/hlsp_eleanor_tess_ffi_postcard-s0001-4-2-cal-1588-0268_tess_v2_pc/hlsp_eleanor_tess_ffi_postcard-s0001-4-2-cal-1588-0268_tess_v2_pc.fits with expected size 158022720. [astroquery.query]
INFO: Found cached file /Users/bmontet/.eleanor/mastDownload/HLSP/hlsp_eleanor_tess_ffi_postcard-s0001-4-2-cal-1588-0268_tess_v2_pc/hlsp_eleanor_tess_ffi_postcard-s0001-4-2_tess_v2_pm.txt with expected size 239144. [astroquery.query]
plt.imshow(data.tpf[300]);_____no_output_____ap = np.zeros_like(data.tpf[300])
ap[:,3:10] = 1

data.get_lightcurve(aperture=ap)
corr_flux = eleanor.TargetData.corrected_flux(data, flux=data.raw_flux, regressors='corner')
q = data.quality == 0

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(15,4), gridspec_kw={'width_ratios':[1,3]})
ax1.imshow(data.tpf[0])
ax1.imshow(data.aperture, cmap='Greys', alpha=0.5)
ax1.set_title('Aperture over TPF')
#plt.imshow(data.aperture)

ax2.plot(data.time[q], corr_flux[q]/np.nanmedian(corr_flux[q]) - 0.0, 'k')
ax2.set_xlabel('Time [BJD - 2457000]')
ax2.set_ylabel('Normalized Flux');
_____no_output_____

It looks like there's a planet here! Let's fold on the known period and phase:_____no_output_____

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(15,4), gridspec_kw={'width_ratios':[1,3]})
ax1.imshow(data.tpf[0])
ax1.imshow(data.aperture, cmap='Greys', alpha=0.5)
ax1.set_title('Aperture over TPF')
#plt.imshow(data.aperture)

modtime = np.mod(data.time[q], 6.268)-2.96

ax2.plot(modtime, corr_flux[q]/np.nanmedian(corr_flux[q]) - 0.0, 'k.')
ax2.set_xlabel('Time [BJD - 2457000]')
ax2.set_ylabel('Normalized Flux');_____no_output_____

There it is!_____no_output_____#### 1.8 Updating eleanor_____no_output_____`eleanor` should not need a version upgrade to handle new sectors of data. Calling `eleanor.Source` on a new sector of data should download all the required metadata to produce light curves for that sector. If you are getting errors, it is worth trying to remove the `~/.eleanor/metadata/sNNNN` directory, where `NNNN` corresponds to the sector in question and trying again. You can also try explicitly acquiring the metadata for a given sector by calling `eleanor.Update()`. This shouldn't take long, but the exact time will depend on the strength of your internet connection._____no_output_____

eleanor.Update(sector=20);Downloading URL https://mast.stsci.edu/tesscut/api/v0.1/astrocut?ra=248.96111249999996&dec=63.911075&y=31&x=31&units=px&sector=20 to ./tesscut_20200715143124.zip ... [Done]
Inflating...
This is the first light curve you have made for this sector. Getting eleanor metadata products for Sector 20...
This will only take a minute, and only needs to be done once. Any other light curves you make in this sector will be faster.
Target Acquired
Cadences Calculated
Quality Flags Assured
CBVs Made
Success! Sector 20 now available.

#### 1.8 Other tips, tricks, or hacks_____no_output_____This is of course an incomplete list of things that work to optimise the creation or interepretation of TESS FFI light curves. If you have any other ones, let us know! This tutorial lives as a Jupyter notebook on the `eleanor` GitHub repository; pull requests are welcomed!_____no_output_____


# Notebook from ncrubin/pyquil
Path: examples/GateNoiseModels.ipynb

# Simulating gate noise on the Rigetti Quantum Virtual Machine
© Copyright 2017, Rigetti Computing.
$$
\newcommand{ket}[1]{\left|{#1}\right\rangle}
\newcommand{bra}[1]{\left\langle {#1}\right|}
\newcommand{tr}{\mathrm{Tr}}
$$_____no_output_____## Pure states vs. mixed states

Errors in quantum computing can introduce classical uncertainty in what the underlying state is.
When this happens we sometimes need to consider not only wavefunctions but also probabilistic sums of
wavefunctions when we are uncertain as to which one we have. For example, if we think that an X gate
was accidentally applied to a qubit with a 50-50 chance then we would say that there is a 50% chance
we have the $\ket{0}$ state and a 50% chance that we have a $\ket{1}$ state.
This is called an "impure" or
"mixed"state in that it isn't just a wavefunction (which is pure) but instead a distribution over
wavefunctions. We describe this with something called a density matrix, which is generally an
operator. Pure states have very simple density matrices that we can write as an outer product of a
ket vector $\ket{\psi}$ with its own bra version $\bra{\psi}=\ket{\psi}^\dagger$.
For a pure state the density matrix is simply

$$
   \rho_\psi = \ket{\psi}\bra{\psi}.
$$
The expectation value of an operator for a mixed state is given by

$$
   \langle X \rangle_\rho = \tr{X \rho}
$$
where $\tr{A}$ is the trace of an operator, which is the sum of its diagonal elements
which is independent of choice of basis.
Pure state density matrices satisfy

$$
   \rho \text{ is pure } \Leftrightarrow \rho^2 = \rho
$$
which you can easily verify for $\rho_\psi$ assuming that the state is normalized.
If we want to describe a situation with classical uncertainty between states $\rho_1$ and
$\rho_2$, then we can take their weighted sum
$$
   \rho = p \rho_1 + (1-p) \rho_2
$$
where $p\in [0,1]$ gives the classical probability that the state is $\rho_1$.

Note that classical uncertainty in the wavefunction is markedly different from superpositions.
We can represent superpositions using wavefunctions, but use density matrices to describe
distributions over wavefunctions. You can read more about density matrices [here](https://en.wikipedia.org/wiki/Density_matrix)._____no_output_____# Quantum gate errors
## What are they?
For a quantum gate given by its unitary operator $U$, a "quantum gate error" describes the scenario in which the actually induces transformation deviates from $\ket{\psi} \mapsto U\ket{\psi}$.
There are two basic types of quantum gate errors:

1. **coherent errors** are those that preserve the purity of the input state, i.e., instead of the above mapping we carry out a perturbed, but unitary operation $\ket{\psi} \mapsto \tilde{U}\ket{\psi}$, where $\tilde{U} \neq U$.
2. **incoherent errors** are those that do not preserve the purity of the input state,
    in this case we must actually represent the evolution in terms of density matrices.
    The state $\rho := \ket{\psi}\bra{\psi}$ is then mapped as
    $$
    \rho \mapsto \sum_{j=1}^n K_j\rho K_j^\dagger,
    $$
    where the operators $\{K_1, K_2, \dots, K_m\}$ are called Kraus operators and must obey
    $\sum_{j=1}^m K_j^\dagger K_j = I$ to conserve the trace of $\rho$.
    Maps expressed in the above form are called Kraus maps. It can be shown that every physical map on a finite
    dimensional quantum system can be represented as a Kraus map, though this representation is not generally unique.
    [You can find more information about quantum operations here](https://en.wikipedia.org/wiki/Quantum_operation#Kraus_operators)

In a way, coherent errors are *in principle* amendable by more precisely calibrated control. Incoherent errors are more tricky._____no_output_____## Why do incoherent errors happen?
When a quantum system (e.g., the qubits on a quantum processor) is not perfectly isolated from its environment it generally co-evolves with the degrees of freedom it couples to. The implication is that while the total time evolution of system and environment can be assumed to be unitary, restriction to the system state generally is not.

**Let's throw some math at this for clarity:**
Let our total Hilbert space be given by the tensor product of system and environment Hilbert spaces:
$\mathcal{H} = \mathcal{H}_S \otimes \mathcal{H}_E$.
Our system "not being perfectly isolated" must be translated to the statement that the global Hamiltonian contains a contribution that couples the system and environment:
$$
H = H_S \otimes I + I \otimes H_E + V
$$
where $V$ non-trivally acts on both the system and the environment.
Consequently, even if we started in an initial state that factorized over system and environment $\ket{\psi}_{S,0}\otimes \ket{\psi}_{E,0}$
if everything evolves by the Schrödinger equation
$$
\ket{\psi_t} = e^{-i \frac{Ht}{\hbar}} \left(\ket{\psi}_{S,0}\otimes \ket{\psi}_{E,0}\right)
$$
the final state will generally not admit such a factorization._____no_output_____## A toy model
**In this (somewhat technical) section we show how environment interaction can corrupt an identity gate and derive its Kraus map.**
For simplicity, let us assume that we are in a reference frame in which both the system and environment Hamiltonian's vanish $H_S = 0, H_E = 0$ and where the cross-coupling is small even when multiplied by the duration of the time evolution $\|\frac{tV}{\hbar}\|^2 \sim \epsilon \ll 1$ (any operator norm $\|\cdot\|$ will do here).
Let us further assume that $V = \sqrt{\epsilon} V_S \otimes V_E$ (the more general case is given by a sum of such terms) and that
the initial environment state satisfies $\bra{\psi}_{E,0} V_E\ket{\psi}_{E,0} = 0$. This turns out to be a very reasonable assumption in practice but a more thorough discussion exceeds our scope.

Then the joint system + environment state $\rho = \rho_{S,0} \otimes \rho_{E,0}$ (now written as a density matrix) evolves as
$$
\rho \mapsto \rho' :=  e^{-i \frac{Vt}{\hbar}} \rho e^{+i \frac{Vt}{\hbar}}
$$
Using the Baker-Campbell-Hausdorff theorem we can expand this to second order in $\epsilon$
$$
\rho' = \rho - \frac{it}{\hbar} [V, \rho] - \frac{t^2}{2\hbar^2} [V, [V, \rho]] + O(\epsilon^{3/2})
$$
We can insert the initially factorizable state $\rho = \rho_{S,0} \otimes \rho_{E,0}$ and trace over the environmental degrees of freedom to obtain
\begin{align}
\rho_S' := \tr_E \rho' & = \rho_{S,0}  \underbrace{\tr \rho_{E,0}}_{1} - \frac{i\sqrt{\epsilon} t}{\hbar} \underbrace{\left[ V_S \rho_{S,0} \underbrace{\tr V_E\rho_{E,0}}_{\bra{\psi}_{E,0} V_E\ket{\psi}_{E,0} = 0} - \rho_{S,0}V_S  \underbrace{\tr \rho_{E,0}V_E}_{\bra{\psi}_{E,0} V_E\ket{\psi}_{E,0} = 0} \right]}_0 \\
& \qquad - \frac{\epsilon t^2}{2\hbar^2} \left[ V_S^2\rho_{S,0}\tr V_E^2 \rho_{E,0} + \rho_{S,0} V_S^2 \tr \rho_{E,0}V_E^2 - 2 V_S\rho_{S,0}V_S\tr V_E \rho_{E,0}V_E\right] \\
& = \rho_{S,0} - \frac{\gamma}{2} \left[ V_S^2\rho_{S,0} + \rho_{S,0} V_S^2  - 2 V_S\rho_{S,0}V_S\right]
\end{align}
where the coefficient in front of the second part is by our initial assumption very small $\gamma := \frac{\epsilon t^2}{2\hbar^2}\tr V_E^2 \rho_{E,0} \ll 1$.
This evolution happens to be approximately equal to a Kraus map with operators $K_1 := I - \frac{\gamma}{2} V_S^2, K_2:= \sqrt{\gamma} V_S$:
\begin{align}
\rho_S \to \rho_S' &= K_1\rho K_1^\dagger + K_2\rho K_2^\dagger
 = \rho - \frac{\gamma}{2}\left[ V_S^2 \rho + \rho V_S^2\right] + \gamma V_S\rho_S V_S + O(\gamma^2)
\end{align}
This agrees to $O(\epsilon^{3/2})$ with the result of our derivation above. This type of derivation can be extended to many other cases with little complication and a very similar argument is used to derive the [Lindblad master equation](https://en.wikipedia.org/wiki/Lindblad_equation)._____no_output_____# Support for noisy gates on the Rigetti QVM

As of today, users of our Forest API can annotate their QUIL programs by certain pragma statements that inform the QVM that a particular gate on specific target qubits should be replaced by an imperfect realization given by a Kraus map.

## But the QVM propagates *pure states*: How does it simulate noisy gates?
It does so by yielding the correct outcomes **in the average over many executions of the QUIL program**:
When the noisy version of a gate should be applied the QVM makes a random choice which Kraus operator is applied to the current state with a probability that ensures that the average over many executions is equivalent to the Kraus map.
In particular, a particular Kraus operator $K_j$ is applied to $\ket{\psi}_S$
$$
\ket{\psi'}_S = \frac{1}{\sqrt{p_j}} K_j \ket{\psi}_S
$$
with probability $p_j:= \bra{\psi}_S K_j^\dagger K_j \ket{\psi}_S$.
In the average over many execution $N \gg 1$ we therefore find that
\begin{align}
\overline{\rho_S'} & = \frac{1}{N} \sum_{n=1}^N \ket{\psi'_n}_S\bra{\psi'_n}_S \\
& = \frac{1}{N} \sum_{n=1}^N p_{j_n}^{-1}K_{j_n}\ket{\psi'}_S \bra{\psi'}_SK_{j_n}^\dagger
\end{align}
where $j_n$ is the chosen Kraus operator label in the $n$-th trial.
This is clearly a Kraus map itself! And we can group identical terms and rewrite it as
\begin{align}
\overline{\rho_S'} & =
  \sum_{\ell=1}^n \frac{N_\ell}{N}  p_{\ell}^{-1}K_{\ell}\ket{\psi'}_S \bra{\psi'}_SK_{\ell}^\dagger
\end{align}
where $N_{\ell}$ is the number of times that Kraus operator label $\ell$ was selected.
For large enough $N$ we know that $N_{\ell} \approx N p_\ell$ and therefore
\begin{align}
\overline{\rho_S'} \approx \sum_{\ell=1}^n K_{\ell}\ket{\psi'}_S \bra{\psi'}_SK_{\ell}^\dagger
\end{align}
which proves our claim.
**The consequence is that noisy gate simulations must generally be repeated many times to obtain representative results**._____no_output_____## How do I get started?

1. Come up with a good model for your noise. We will provide some examples below and may add more such
    examples to our public repositories over time. Alternatively, you can characterize the gate under
    consideration using [Quantum Process Tomography](https://arxiv.org/abs/1202.5344) or
    [Gate Set Tomography](http://www.pygsti.info/) and use the resulting process matrices to obtain a
    very accurate noise model for a particular QPU.
2. Define your Kraus operators as a list of numpy arrays `kraus_ops = [K1, K2, ..., Km]`.
3. For your QUIL program `p`, call:
    ```
    p.define_noisy_gate("MY_NOISY_GATE", [q1, q2], kraus_ops)
    ```
    where you should replace `MY_NOISY_GATE` with the gate of interest and `q1, q2` the indices of the qubits.

**Scroll down for some examples!**_____no_output_____

from __future__ import print_function, division
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import binom
import matplotlib.colors as colors
%matplotlib inline_____no_output_____from pyquil.quil import Program, MEASURE
from pyquil.api.qvm import QVMConnection
from pyquil.job_results import wait_for_job
from pyquil.gates import CZ, H, I, X
from scipy.linalg import expm_____no_output_____cxn = QVMConnection()_____no_output_____

# Example 1: Amplitude damping

Amplitude damping channels are imperfect identity maps with Kraus operators
$$
K_1 = \begin{pmatrix}
1 & 0 \\
0 & \sqrt{1-p}
\end{pmatrix} \\
K_2 = \begin{pmatrix}
0 & \sqrt{p} \\
0 & 0
\end{pmatrix}
$$
where $p$ is the probability that a qubit in the $\ket{1}$ state decays to the $\ket{0}$ state.
_____no_output_____

def damping_channel(damp_prob=.1):
    """
    Generate the Kraus operators corresponding to an amplitude damping
    noise channel.

    :params float damp_prob: The one-step damping probability.
    :return: A list [k1, k2] of the Kraus operators that parametrize the map.
    :rtype: list
    """
    damping_op = np.sqrt(damp_prob) * np.array([[0, 1],
                                                [0, 0]])

    residual_kraus = np.diag([1, np.sqrt(1-damp_prob)])
    return [residual_kraus, damping_op]

def append_kraus_to_gate(kraus_ops, g):
    """
    Follow a gate `g` by a Kraus map described by `kraus_ops`.

    :param list kraus_ops: The Kraus operators.
    :param numpy.ndarray g: The unitary gate.
    :return: A list of transformed Kraus operators.
    """
    return [kj.dot(g) for kj in kraus_ops]

def append_damping_to_gate(gate, damp_prob=.1):
    """
    Generate the Kraus operators corresponding to a given unitary
    single qubit gate followed by an amplitude damping noise channel.

    :params np.ndarray|list gate: The 2x2 unitary gate matrix.
    :params float damp_prob: The one-step damping probability.
    :return: A list [k1, k2] of the Kraus operators that parametrize the map.
    :rtype: list
    """
    return append_kraus_to_gate(damping_channel(damp_prob), gate)_____no_output_____%%time

# single step damping probability
damping_per_I = 0.02

# number of program executions
trials = 200

results_damping = []
lengths = np.arange(0, 201, 10, dtype=int)
for jj, num_I in enumerate(lengths):

    print("{}/{}, ".format(jj, len(lengths)), end="")

    p = Program(X(0))
    # want increasing number of I-gates
    p.inst([I(0) for _ in range(num_I)])
    p.inst(MEASURE(0, [0]))

    # overload identity I on qc 0
    p.define_noisy_gate("I", [0], append_damping_to_gate(np.eye(2), damping_per_I))
    cxn.random_seed = int(num_I)
    res = cxn.run(p, [0], trials=trials)
    results_damping.append([np.mean(res), np.std(res) / np.sqrt(trials)])

results_damping = np.array(results_damping)0/21, 1/21, 2/21, 3/21, 4/21, 5/21, 6/21, 7/21, 8/21, 9/21, 10/21, 11/21, 12/21, 13/21, 14/21, 15/21, 16/21, 17/21, 18/21, 19/21, 20/21, CPU times: user 114 ms, sys: 18.6 ms, total: 133 ms
Wall time: 1.83 s
dense_lengths = np.arange(0, lengths.max()+1, .2)
survival_probs = (1-damping_per_I)**dense_lengths
logpmf = binom.logpmf(np.arange(trials+1)[np.newaxis, :], trials, survival_probs[:, np.newaxis])/np.log(10)_____no_output_____DARK_TEAL = '#48737F'
FUSCHIA = "#D6619E"
BEIGE = '#EAE8C6'
cm = colors.LinearSegmentedColormap.from_list('anglemap', ["white", FUSCHIA, BEIGE], N=256, gamma=1.5)_____no_output_____plt.figure(figsize=(14, 6))
plt.pcolor(dense_lengths, np.arange(trials+1)/trials, logpmf.T, cmap=cm, vmin=-4, vmax=logpmf.max())
plt.plot(dense_lengths, survival_probs, c=BEIGE, label="Expected mean")
plt.errorbar(lengths, results_damping[:,0], yerr=2*results_damping[:,1], c=DARK_TEAL,
             label=r"noisy qvm, errorbars $ = \pm 2\hat{\sigma}$", marker="o")
cb = plt.colorbar()
cb.set_label(r"$\log_{10} \mathrm{Pr}(n_1; n_{\rm trials}, p_{\rm survival}(t))$", size=20)

plt.title("Amplitude damping model of a single qubit", size=20)
plt.xlabel(r"Time $t$ [arb. units]", size=14)
plt.ylabel(r"$n_1/n_{\rm trials}$", size=14)
plt.legend(loc="best", fontsize=18)
plt.xlim(*lengths[[0, -1]])
plt.ylim(0, 1)_____no_output_____

# Example 2: dephased CZ-gate

Dephasing is usually characterized through a qubit's $T_2$ time.
For a single qubit the dephasing Kraus operators are
$$
K_1(p) = \sqrt{1-p} I_2 \\
K_2(p) = \sqrt{p} \sigma_Z
$$
where $p = 1 - \exp(-T_2/T_{\rm gate})$ is the probability that the qubit is dephased over the time interval of interest, $I_2$ is the $2\times 2$-identity matrix and $\sigma_Z$ is the Pauli-Z operator.

For two qubits, we must construct a Kraus map that has *four* different outcomes:

1. No dephasing
2. Qubit 1 dephases
3. Qubit 2 dephases
4. Both dephase_____no_output_____The Kraus operators for this are given by
\begin{align}
K'_1(p,q) = K_1(p)\otimes K_1(q) \\
K'_2(p,q) = K_2(p)\otimes K_1(q) \\
K'_3(p,q) = K_1(p)\otimes K_2(q) \\
K'_4(p,q) = K_2(p)\otimes K_2(q)
\end{align}
where we assumed a dephasing probability $p$ for the first qubit and $q$ for the second.

Dephasing is a *diagonal* error channel and the CZ gate is also diagonal, therefore we can get the combined map of dephasing and the CZ gate simply by composing $U_{\rm CZ}$ the unitary representation of CZ with each Kraus operator
\begin{align}
K^{\rm CZ}_1(p,q) = K_1(p)\otimes K_1(q)U_{\rm CZ} \\
K^{\rm CZ}_2(p,q) = K_2(p)\otimes K_1(q)U_{\rm CZ} \\
K^{\rm CZ}_3(p,q) = K_1(p)\otimes K_2(q)U_{\rm CZ} \\
K^{\rm CZ}_4(p,q) = K_2(p)\otimes K_2(q)U_{\rm CZ}
\end{align}

**Note that this is not always accurate, because a CZ gate is often achieved through non-diagonal interaction Hamiltonians! However, for sufficiently small dephasing probabilities it should always provide a good starting point.**_____no_output_____

def dephasing_kraus_map(p=.1):
    """
    Generate the Kraus operators corresponding to a dephasing channel.

    :params float p: The one-step dephasing probability.
    :return: A list [k1, k2] of the Kraus operators that parametrize the map.
    :rtype: list
    """
    return [np.sqrt(1-p)*np.eye(2), np.sqrt(p)*np.diag([1, -1])]

def tensor_kraus_maps(k1, k2):
    """
    Generate the Kraus map corresponding to the composition
    of two maps on different qubits.

    :param list k1: The Kraus operators for the first qubit.
    :param list k2: The Kraus operators for the second qubit.
    :return: A list of tensored Kraus operators.
    """
    return [np.kron(k1j, k2l) for k1j in k1 for k2l in k2]
_____no_output_____%%time
# single step damping probabilities
ps = np.linspace(.001, .5, 200)

# number of program executions
trials = 500

results = []

for jj, p in enumerate(ps):

    corrupted_CZ = append_kraus_to_gate(
    tensor_kraus_maps(
        dephasing_kraus_map(p),
        dephasing_kraus_map(p)
    ),
    np.diag([1, 1, 1, -1]))

    print("{}/{}, ".format(jj, len(ps)), end="")

    # make Bell-state
    p = Program(H(0), H(1), CZ(0,1), H(1))

    p.inst(MEASURE(0, [0]))
    p.inst(MEASURE(1, [1]))

    # overload identity I on qc 0
    p.define_noisy_gate("CZ", [0, 1], corrupted_CZ)
    cxn.random_seed = jj
    res = cxn.run(p, [0, 1], trials=trials)
    results.append(res)

results = np.array(results)0/200, 1/200, 2/200, 3/200, 4/200, 5/200, 6/200, 7/200, 8/200, 9/200, 10/200, 11/200, 12/200, 13/200, 14/200, 15/200, 16/200, 17/200, 18/200, 19/200, 20/200, 21/200, 22/200, 23/200, 24/200, 25/200, 26/200, 27/200, 28/200, 29/200, 30/200, 31/200, 32/200, 33/200, 34/200, 35/200, 36/200, 37/200, 38/200, 39/200, 40/200, 41/200, 42/200, 43/200, 44/200, 45/200, 46/200, 47/200, 48/200, 49/200, 50/200, 51/200, 52/200, 53/200, 54/200, 55/200, 56/200, 57/200, 58/200, 59/200, 60/200, 61/200, 62/200, 63/200, 64/200, 65/200, 66/200, 67/200, 68/200, 69/200, 70/200, 71/200, 72/200, 73/200, 74/200, 75/200, 76/200, 77/200, 78/200, 79/200, 80/200, 81/200, 82/200, 83/200, 84/200, 85/200, 86/200, 87/200, 88/200, 89/200, 90/200, 91/200, 92/200, 93/200, 94/200, 95/200, 96/200, 97/200, 98/200, 99/200, 100/200, 101/200, 102/200, 103/200, 104/200, 105/200, 106/200, 107/200, 108/200, 109/200, 110/200, 111/200, 112/200, 113/200, 114/200, 115/200, 116/200, 117/200, 118/200, 119/200, 120/200, 121/200, 122/200, 123/200, 124/200, 125/200, 126/200, 127/200, 128/200, 129/200, 130/200, 131/200, 132/200, 133/200, 134/200, 135/200, 136/200, 137/200, 138/200, 139/200, 140/200, 141/200, 142/200, 143/200, 144/200, 145/200, 146/200, 147/200, 148/200, 149/200, 150/200, 151/200, 152/200, 153/200, 154/200, 155/200, 156/200, 157/200, 158/200, 159/200, 160/200, 161/200, 162/200, 163/200, 164/200, 165/200, 166/200, 167/200, 168/200, 169/200, 170/200, 171/200, 172/200, 173/200, 174/200, 175/200, 176/200, 177/200, 178/200, 179/200, 180/200, 181/200, 182/200, 183/200, 184/200, 185/200, 186/200, 187/200, 188/200, 189/200, 190/200, 191/200, 192/200, 193/200, 194/200, 195/200, 196/200, 197/200, 198/200, 199/200, CPU times: user 1.12 s, sys: 161 ms, total: 1.28 s
Wall time: 29.7 s
Z1s = (2*results[:,:,0]-1.)
Z2s = (2*results[:,:,1]-1.)
Z1Z2s = Z1s * Z2s

Z1m = np.mean(Z1s, axis=1)
Z2m = np.mean(Z2s, axis=1)
Z1Z2m = np.mean(Z1Z2s, axis=1)_____no_output_____plt.figure(figsize=(14, 6))
plt.axhline(y=1.0, color=FUSCHIA, alpha=.5, label="Bell state")

plt.plot(ps, Z1Z2m, "x", c=FUSCHIA, label=r"$\overline{Z_1 Z_2}$")
plt.plot(ps, 1-2*ps, "--", c=FUSCHIA, label=r"$\langle Z_1 Z_2\rangle_{\rm theory}$")

plt.plot(ps, Z1m, "o", c=DARK_TEAL, label=r"$\overline{Z}_1$")
plt.plot(ps, 0*ps, "--", c=DARK_TEAL, label=r"$\langle Z_1\rangle_{\rm theory}$")

plt.plot(ps, Z2m, "d", c="k", label=r"$\overline{Z}_2$")
plt.plot(ps, 0*ps, "--", c="k", label=r"$\langle Z_2\rangle_{\rm theory}$")

plt.xlabel(r"Dephasing probability $p$", size=18)
plt.ylabel(r"$Z$-moment", size=18)
plt.title(r"$Z$-moments for a Bell-state prepared with dephased CZ", size=18)
plt.xlim(0, .5)
plt.legend(fontsize=18)_____no_output_____