# Notebook from kaelynj/qiskit-textbook
Path: content/ch-applications/quantum-simulation.ipynb

$$\newcommand{\ket}[1]{\left|{#1}\right\rangle}$$
$$\newcommand{\bra}[1]{\left\langle{#1}\right|}$$
$$\newcommand{\braket}[2]{\left\langle{#1}\middle|{#2}\right\rangle}$$
# Digital Quantum Simulations_____no_output_____In this tutorial, we will introduce time evolution of quantum states and how to simulate them using the Trotter-Suzuki formalism.  We then show to how to run the time evolution of a one dimensional Fermi-Hubbard chain with three sites._____no_output_____## Contents
1. [Introduction](#introduction)
2. [Trotter-Suzuki Formalism](#ts-equation)
3. [The Fermi-Hubbard Model](#hubbard-models)
3. [Jordan-Wigner Transformation and Encoding](#jw-transform)
4. [One-Dimensional Hubbard Simulation](#1d-chain)_____no_output_____## 1. Introduction

In quantum mechanics, propagation in time of the wavefunction $\ket{\Psi(t)}$ is governed by the Shro\"{o}dinger equation

$$ i\hbar\ket{\Psi(t)} = H\ket{\Psi(t)}$$

where $H$ is the Hamiltonian operator of the system.  If we consider $H$ to be time-independent, the solution for $\ket{\Psi(t)}$ has the form

$$ \ket{\Psi(t)} = e^{iHt}\ket{\Psi(0)} $$

and simulating this requires the ability to perform the operation above.  In most cases, this reduces to matrix exponentiation, but proves to be very difficult on classical computers due to the exponential growth of the operator as the system size increases.

Efficient approximations to the solution of this equation are possible for Hamiltonians which carry local interactions between the system's consitutent subsystems.  These classes of Hamiltonians can be written in the form

$$ H = \sum_k^L H_k $$

where each $H_k$ acts locally on only a portion of the total system.  The advantage of writing the Hamiltonian in this way is that although the total time evolution operator $e^{iHt}$ may be difficult to compute, it is much easier to use $e^{iH_kt}$ as it acts on a much small subsytem and is straightforward to approximate using quantum circuits._____no_output_____## 2. Trotter-Suzuki Formalism _____no_output_____## 3. The Fermi-Hubard Model _____no_output_____## 4. Jordan-Wigner Transformation and Encoding _____no_output_____## 5. One-Dimensional Hubbard Simulation
$$\newcommand{\ket}[1]{\left|{#1}\right\rangle}$$
$$\newcommand{\bra}[1]{\left\langle{#1}\right|}$$

The Hubbard Hamiltonian for a 1-D chain of sites is:

\begin{equation} H = -t\sum_{<ij>,\sigma}(\hat{c}_{i_\sigma}\hat{c}_{j_\sigma} + \hat{c}_{j_\sigma}^\dagger\hat{c}_{i_\sigma} ) + U\sum_i\hat{c}_{i_\uparrow}^{\dagger}\hat{c}_{i_\uparrow}\hat{c}_{i_\downarrow}^{\dagger}\hat{c}_{i_\downarrow}
\end{equation}
and under the Jordan-Wigner mapping, this Hamiltonian becomes:

\begin{equation}
H = -\frac{t}{2}\sum_{<ij>}Z_{i+1:j-1}(X_{i}X_{j} + Y_{i}Y_{j}) + \frac{U}{4}\sum_{ij}(I-Z_{i})(I-Z_{j})
\end{equation}
where $Z_{i}$, $X_{i}$, and $Y_{i}$ are the corresponding Pauli matrices acting on the $i^{th}$ fermionic mode and for a chain with only 3-sites, there are no Pauli $Z$ strings in the hopping term of the Hamiltonian.

### 3-Site Hamiltonian and Qubit Mapping

Writing out the sum over the three sites, the Hamiltonian becomes:

\begin{align}
     H = &-\frac{t}{2}(X_0X_1 + Y_0Y_1) - \frac{t}{2}(X_1X_2 + Y_1Y_2) \nonumber \\
&-\frac{t}{2}(X_3X_4 + Y_3Y_4) - \frac{t}{2}(X_4X_5 + Y_4Y_5) \nonumber\\
&+ \frac{U}{4}(I-Z_0)(I-Z_3)+ \frac{U}{4}(I-Z_1)(I-Z_4) + \frac{U}{4}(I-Z_2)(I-Z_5) \nonumber\\
= &H_{01} + H_{12} + H_{23} + H_{34} + H_{03} + H_{14} + H_{25}
\end{align}

#### Qubit Mapping

Each site in the 3-site chain is represented by two qubits, one for each spin, and the wavefunction is represented as

$$ \ket{\psi} = \ket{q_0}\ket{q_1}\ket{q_2}\ket{q_3}\ket{q_4}\ket{q_5} $$

where $\ket{q_i} = \{ \ket{0}, \ket{1} \} $ represent unoccupied or occupied sites, $i=0,1,2$ are the spin up electron occupations and $i=3,4,5$ are the spin down electron occupations.

### Time Evolution

We want to simulate the time evolution of $\ket{\psi}$ via

$$ \ket{\psi(t+\Delta t)} = e^{-iH\Delta t}\ket{\psi(t)} $$

We can do this via the Suzuki-Trotter formula which states that, to first order in $\Delta t$

$$e^{iH\Delta t} \approx e^{iH_{10}^{\uparrow}\Delta t}e^{iH_{12}^{\uparrow}\Delta t}e^{iH_{10}^{\downarrow}\Delta t}e^{iH_{12}^{\downarrow}\Delta t}e^{iH_0\Delta t}e^{iH_1\Delta t}e^{iH_2\Delta t} $$.

So what do the gates look like for each of these terms?

##### Hopping Terms

For each pair of hopping terms we have
\begin{equation}e^{-i\Delta t(\frac{-t}{2})(X_iX_j + Y_iY_j)} \approx e^{\frac{it\Delta t}{2}X_iX_j} e^{\frac{it\Delta t}{2}Y_iY_j}.
\label{eq:op-expansion}
\end{equation}

Expanding the first term on the right hand side

\begin{align}
e^{\frac{it\Delta t}{2}X_iX_j} = & \sum_{k=0}^{\infty} \frac{1}{k!}\left(\frac{it\Delta t}{2}X_iX_j\right)^k \nonumber \\
 =& \sum_{k, even}\frac{i^k}{k!}\left( \frac{t\Delta t}{2} \right)^k I + \sum_{k, odd}\frac{i^k}{k!}\left( \frac{t\Delta t}{2} \right)X_i X_j \nonumber \\
 = &\cos\left(\frac{t\Delta t}{2}\right)I + i\sin\left( \frac{t\Delta t}{2}\right)X_i X_j \nonumber \\
 = &\begin{pmatrix}\cos\theta & 0 & 0 & i\sin\theta \\ 0 & \cos\theta & i\sin\theta & 0 \\ 0 & i\sin\theta & \cos\theta & 0 \\ i\sin\theta & 0 & 0 & \cos\theta\end{pmatrix},
\end{align}
with $\theta=\frac{t\Delta t}{2}$, and written in the $\ket{q_{i}q_{j}}$ basis.

Similarly for the $Y_i Y_j$ terms

\begin{align}
    e^{i\frac{t\Delta t}{2}Y_i Y_j} =& \cos\left(\frac{t\Delta t}{2}\right)I + i\sin\left(\frac{t\Delta t}{2}\right)Y_i Y_j \nonumber \\
= & \begin{pmatrix}\cos\theta & 0 & 0 & -i\sin\theta \\ 0 & \cos\theta & i\sin\theta & 0 \\ 0 & i\sin\theta & \cos\theta & 0 \\ -i\sin\theta & 0 & 0 & \cos\theta\end{pmatrix}.
\end{align}

Note also that these matrices are diagonal save for the 4x4 block corresponding to a gate acting on qubits $i$ and $j$.

#### On-Site Terms

Now we'll expand the on-site term, $e^{i\frac{U\Delta t}{4}(I-Z_i)(I-Z_j)}$.  First we examine the powers of $(I-Z_i)(I-Z_j)$:

\begin{align} (I-Z_i)^2(I-Z_j)^2 &= (I + I - 2Z_i)(I+I-2Z_j) = 4(I-Z_i)(I-Z_j) \\
(I-Z_i)^3(I-Z_j)^3& = (I-Z_i)(I-Z_j)(I-Z_i)^2(I-Z_j)^2 \nonumber \\
&= 4(I-Z_i)^2(I-Z_j)^2 = 16(I-Z_i)(I-Z_j)\\
(I-Z_i)^4(I-Z_j)^4& = 16(I-Z_i)^2(I-Z_j)^2 = 4^3(I-Z_i)(I-Z_j)\\
\implies (I-Z_i)^k(I-Z_j)^k& = 4^{k-1}(I-Z_i)(I-Z_j), \end{align}

then writing out the expansion of $e^{i\frac{U\Delta t}{4}(I-Z_i)(I-Z_j)}$ we get

\begin{align} e^{i\frac{U\Delta t}{4}(I-Z_i)(I-Z_j)} &= \sum_k \frac{1}{k!}\left(\frac{i\Delta tU}{4}\right)^k(I-Z_i)^k(I-Z_j)^k \\
&= I+(I-Z_i)(I-Z_j)\sum_k \frac{\left(i\Delta tU\right)^k}{k!}\frac{4^{k-1}}{4^k} - \frac{1}{4}(I-Z_i)(I-Z_j) \\
& = I-\frac{1}{4}(I-Z_i)(I-Z_j) + \frac{1}{4}e^{iU\Delta t}(I-Z_i)(I-Z_j)  \\
& \boxed{= I-(I-Z_i)(I-Z_j)\left(1-e^{iU\Delta t} \right) } \\
&= \begin{pmatrix}1 & 0 & 0 & 0\\ 0 & 1& 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & e^{iU\Delta t} \end{pmatrix} .
\end{align}

$^1$https://web.cs.ucdavis.edu/~bai/publications/varneyleebai09.pdf_____no_output_____


# Notebook from okara83/Becoming-a-Data-Scientist
Path: Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/RECAP_DS/08_PYTHON/L08.ipynb

# DSO109 Python : Lesson Eight Companion Notebook_____no_output_____### Table of Contents

* [Table of Contents](#DS109L8_toc)
    * [Page 1 - Python for Data Science](#DS109L8_page_1)
    * [Page 2 - Installing Anaconda on a Windows Machine](#DS109L8_page_2)
    * [Page 3 - Installing Anaconda on a Mac or Linux Machine](#DS109L8_page_3)
    * [Page 4 - Get Familiar with Jupyter Notebook!](#DS109L8_page_4)
    * [Page 5 - Data Science-Specific Packages](#DS109L8_page_5)
    * [Page 6 - Importing Data for Windows](#DS109L8_page_6)
    * [Page 7 - Importing Data for Mac / Linux](#DS109L8_page_7)
    * [Page 8 - Viewing Files in Pandas](#DS109L8_page_8)
    * [Page 9 - Descriptive Statistics](#DS109L8_page_9)
    * [Page 10 - Keyboard Shortcuts for Jupyter Notebook](#DS109L8_page_10)
    * [Page 11 - Key Terms](#DS109L8_page_11)
    * [Page 12 - Lesson 8 Hands-On](#DS109L8_page_12)
    _____no_output_____

# Page 1 - Python for Data Science

[Back to Top](#DS109L8_toc)

_____no_output_____

from IPython.display import VimeoVideo
# Tutorial Video Name: Python for Data Science
VimeoVideo('388856304', width=720, height=480)_____no_output_____

The transcript for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO109L08overview.zip)**.

You have previously had a wonderful introduction to object-oriented programming in Python.  It is very important that you gained a good understanding of the basics that are not data science specific for several reasons:

* For more complex work in machine learning or big data, you may need the ability to utilize a text editor like Visual Studio Code and the command line.
* In data wrangling, you may find yourself using for loops, dictionaries, and other basics.
* Hiring managers may not distinguish between the use of Python for software engineering and Python for data science, which means that you may take skills assessments that will depend on your ability to use Python as an object-oriented language in order to land a job.

---

## How is Python Used for Data Science?

Conceptually, using Python for Data Science is very similar to how you use R.  You'll learn the basic packages (think the equivalent of libraries in R) needed to import and manipulate data, perform data wrangling, conduct statistics, perform machine learning, and so much more! Python is nice for data science because it is so very flexible.

---

## Comparing Python to R as a Data Science Tool

R was built for statistics and statistics only.  With that in mind, it provides several advantages:

* The output from R typically provides more information about the analysis you ran and it is easier to get ahold of this output.
* You are allowed to have missing data in many R data analyses, as it will use pairwise deletion. This usually gives you greater power and a little more flexibility with your data.
* It sometimes requires less code  or fewer steps to run analyses or data visualizations.
* R has a wider range of traditional statistical methods and visuals.

The advent of Python for data science really stemmed from the programming world instead of the statistical world, which means that using Python has its own set of advantages!

* Python is typically better equipped for machine learning.
* Python connects better to big data programs such as Spark and Hadoop.
* It is easier to integrate basic programming elements into Python.
* There is more documentation for Python and more people are currently using it, which means that it will be easier to get help solving problems.

Because both Python and R have different strengths, you are encouraged to use both and really get familiar with what each program is capable of.  Then, as you become comfortable, you will be able to choose the program that will best suit your data scientist needs of the day.  Some things are just easier in Python, or easier in R.  It would be silly to struggle through something in Python that is easily completed in R, or vice versa. Think of both programs as complementary, rather than redundant.

---

## Introduction to Anaconda

After getting through Programming Basics, you may be feeling frustrated.  You may be thinking to yourself that if you wanted to write code and use the command line all day, you would have chosen to be a programmer, not a data scientist.  Luckily, data scientists came up with a solution that makes Python their own - it's called *Anaconda*!

Anaconda is a data science specific version of Python that comes with two *integrated development environments* (IDEs).  An IDE is basically a user-friendly way to use a program; an example is R Studio.

The first IDE Anaconda comes with is Jupyter Notebook; the second is Spyder.  Both use the same code and function relatively similarly, and both provide output without requiring use of the command line. While you are welcome to use Spyder or Visual Studio Code for the remainder of the course, this course will support the use of Jupyter Notebook.  This is because *Jupyter Notebook* is the most widely used IDE for data scientists at this time, and is even being asked for on job descriptions sometimes.

---_____no_output_____

# Page 2 - Installing Anaconda on a Windows Machine

[Back to Top](#DS109L8_toc)

_____no_output_____# Installing Anaconda on a Windows Machine

This page is for how to install Anaconda and Jupyter Notebooks on a Windows Machine.  If you have a Mac or Linux machine, please proceed to the next page!

1. Go  to the **[Anaconda Download page](https://www.anaconda.com/download/)**.  You should see a screen very similar to this:

    ![Main screen of the Anaconda download.](Media/WINDOWSCONDA.png)

2. Scroll down until you see the header ```Anaconda for Windows Installer```. It should look something like this:

    ![Anaconda installer page. There are two options, Anaconda 3 point 6 or Anaconda 2 point 7.](Media/WINDOWCONDAINSTALL.png)

Caution!

Don't go for the big green button! Instead, choose the smaller button for "Graphical Installer" that is shown below.  Use either the 32 or 64 bit version, depending on your machine.

3. Once it has downloaded, click on it.  A window will pop up:

    ![Anaconda installer page. It say Welcome to Anaconda 3 5 point 2 point zero. 64 bit setup. There is a next button in the bottom right corner.](Media/Anaconda4.png)

    Click "next."

5. Click "I agree" when you see the license agreement:

    ![Anaconda License agreement. There is an I agree and back button in the lower right corner. The I agree button is highlighted. ](Media/Anaconda5.png)

6. Choose the option for "Just Me" when you're asked what installation type to select:

    ![Anaconda select installation type window. There are installation options for Just me, which is recommended, or all users, which requires admin privileges. There are next and back buttons in the lower right corner.](Media/Anaconda6.png)

7. Select your destination folder.

    ![Anaconda choose install location screen. There is a text box where to put in the destination folder. There is a browse button. There are next and back buttons in the lower right corner.](Media/Anaconda7.png)

8. Follow the remaining instructions and let it finish installing.

9. To open Jupyter Notebooks, you will go to the Anaconda3 folder in your start menu, and click the Jupyter Notebook option.  That will launch a window that looks something like this:

    ![Jupyter notebook code window. Several lines of code are shown.](Media/Anaconda8.png)

    But it should also open a window in your web browser that looks something like this:

    ![Jupyter file directory window showing several folders.](Media/Anaconda9.png)

    You can think of this as your Jupyter homebase, and you will be able to open saved Jupyter documents from here.

10.  Navigate into the folder where you'd like to save this work, and then click on the ```New``` button in the upper right hand corner, and when prompted, select ```Python3```. You need to specify Python as the language with which to use Jupyter Notebook because Jupyter will support a wide variety of other programming languages if you install them.

Tip!

Wherever directory you are in when you hit the "new" button is where Jupyter will save the document.  By default, it saves to your root directory, but you can change that to a course-specific folder if you'd like.

11.  This will bring up a new page that looks like this:

![Blank window in the Jupyter software.](Media/Anaconda10.png)

That's it! You've now created your very first Jupyter Notebook using the Anaconda version of Python! Please skip the next page to continue working in Jupyter!
_____no_output_____

# Page 3 - Installing Anaconda on a Mac or Linux Machine

[Back to Top](#DS109L8_toc)

_____no_output_____# Installing Anaconda on a Mac or Linux Machine

This page is for how to install Anaconda and Jupyter Notebooks on a Mac or Linux Machine.  If you have a Windows machine, please proceed to the next page!

1. Go  to the **[Anaconda Download page](https://www.anaconda.com/download/)**.  You should see a screen very similar to this:

    ![Main screen of the Anaconda download.](Media/MACLINUXCONDA1.png)

2. Scroll down until you see the header "Anaconda for MacOS or Linux" (depending on your machine). It should look something like this:

    ![Anaconda MacOS installer page. There are two options, Anaconda 3 point 7 or Anaconda 2 point 7.](Media/MACLINUXCONDA.png)

Caution!

Don't go for the big green button! Instead, choose the smaller button for "Graphical Installer" that is shown below.  Use either the 32 or 64 bit version, depending on your machine.

3. Once it has downloaded, click on it.  A window will pop up:

    ![Anaconda 5 point 2 for Windows installer page. There are two options, Python 3 point 6 version and python 2 point 7 version.](Media/Anaconda3.png)

    Click "next."

5. Click "I agree" when you see the license agreement:

    ![Anaconda License agreement. There is an I agree and back button in the lower right corner. The I agree button is highlighted.](Media/Anaconda5.png)

6. Choose the option for "Just Me" when you're asked what installation type to select:

    ![Anaconda select installation type window. There are installation options for Just me, which is recommended, or all users, which requires admin privileges. There are next and back buttons in the lower right corner.](Media/Anaconda6.png)

7. Select your destination folder.

    ![Anaconda choose install location screen. There is a text box where to put in the destination folder. There is a browse button. There are next and back buttons in the lower right corner.](Media/Anaconda7.png)

8. Follow the remaining instructions and let it finish installing.

9. To open Jupyter Notebooks, you will go to the Anaconda Navigator.  You can search for it, or you should be able to locate it with the big green scaly wheel (like the snake!).  When you open it, the window should look something like this:

    ![Anaconda navigator window. There are several icons for applications inside. The top row, second from the left is called Jupyter.](Media/Anaconda12.png)

10. Click on the second item from the left, ```Jupyter notebook```. It should open a window in your web browser that looks something like this:

    ![Jupyter file directory window showing several folders.](Media/Anaconda9.png)

    You can think of this as your Jupyter homebase, and you will be able to open saved Jupyter documents from here.

10.  Navigate into the folder where you'd like to save this work, and then click on the ```New``` button in the upper right hand corner, and when prompted, select ```Python3```. You need to specify Python as the language with which to use Jupyter Notebook because Jupyter will support a wide variety of other programming languages if you install them.

Tip!

Wherever directory you are in when you hit the "new" button is where Jupyter will save the document.  By default, it saves to your root directory, but you can change that to a course-specific folder if you'd like.

11.  This will bring up a new page that looks like this:

![Blank window in the Jupyter software.](Media/Anaconda10.png)

That's it! You've now created your very first Jupyter Notebook using the Anaconda version of Python!
_____no_output_____

# Page 4 - Get Familiar with Jupyter Notebook!

[Back to Top](#DS109L8_toc)

_____no_output_____# Get Familiar with Jupyter Notebook!

Additional Info!

You may want to watch this  recorded live workshop  that goes over the basics of using Jupyter Notebook, which will be covered in the next few pages of your lesson.

Here is an up close and personal view of the Jupyter Notebook menu.

![Jupyter window with menu options file, edit, view, insert, cell, kernel, widgets, and help. There is a search box.](Media/Anaconda13.png)

---

## Renaming Your File

You'll notice that the first thing you have on the top is a label saying this is ```Untitled```.   If you click there, it will bring up an option to rename the notebook to something you'll actually remember.  Something with the module name and lesson number in it is recommended so that you can easily refer to your work later as you move forward with the course.

![Window titled rename notebook. Enter a new notebook name. A textbox is shown where the new name can be typed in. There is a rename button in the lower right corner.](Media/Anaconda14.png)

Pressing the big blue ```Rename``` button will exit you out of the window once you've typed in something.

---

## The File Menu

In this File menu, there are some noteworthy features:

* You can create a new notebook.
* You can copy your notebook here.
* You can save, though this is also done with the floppy disk icon.
* If you screw up really, really badly, you can choose the option to ```Revert to Checkpoint``` for a particular day and time.

Tip!

 Typically this option isn't necessary, because you can just re-run cells, but it's there in case of catastrophic failure!

---

### Creating a New Notebook

If you want a new notebook in the same folder as your current one, you can use the ```File``` menu and choose the ```New Notebook``` option.

![Jupyter menu with the pull down file menu shown. It has the options new notebook, open, make a copy, rename, save and checkpoint, revert to checkpoint, print preview, download as, and close and halt.](Media/Anaconda15.png)

Tip!

If you want a new notebook in a different folder, you will need to go back to your Jupyter directory in the previous tab, navigate to that folder, and then click "New" as you did on the installation instructions page.

---

## The Edit Menu

In the Edit menu, you'll see these options to manipulate the cells:

![Jupyter menu with the pull down edit menu shown. Options are cut cells, copy cells, paste cells above, paste cells below, paste cells and replicate, delete cells, undo delete cells, split cell, merge cells above, merge cells below, move cell up, move cell down, edit notebook metadata, find and replace, cut cell attachments, copy cell attachments, and paste cell attachments.](Media/Anaconda16.png)

Feel free to explore as you see fit, but typically the plus button to add cells in, and the scissors symbol to cut out cells is all most folks end up needing.  By default, the plus sign will add a cell after.

![Menu bar with icons for save, cut, copy paste, up, down, run, stop, revert, fast forward, markdown, and keyboard.](Media/Anaconda34.png)

---

## The Cell Menu

Next you have the ```Cell``` menu.  While you have this open, anything you do sequentially down the page will stay put.  But, if you close this Notebook, you will need to re-run things to keep working.  While this can be done cell by cell by clicking into each one and using ```shift + enter```, you can also use the ```Cell``` menu to run in any variety.  Typically ```Run All``` is the most useful, but depending on your need you can run selected cells or above and below a certain point.

![Jupyter menu with the pull down cell menu shown. Options are run cells, run cells and select below, run cells and insert below, run all, run all above, run all below, cell type, content output, all output.](Media/Anaconda17.png)

---

### Using Markdown

The other thing to point out on the Cell Menu is the Cell Type. The default is code, so in your case, Python.  But there's also an option for markdown.  *Markdown* is an easy way to translate text into an HTML-like format that reads pretty on websites.  So, unlike the usual way to make notes to yourself in your project, which would just be a hashtag (#), you can change the code to markdown and it will render once you ```shift + enter``` as formatted text.

Here are the basics of markdown:

![Window with text that reads Hello students! This is what markdown looks like. There are generic heading 2, 3, and 4 at the bottom of the screen.](Media/Anaconda18.png)

And this is what it will look like after you ```shift + enter```:

![Window with text that reads Hello students! This is what markdown looks like. There are generic heading 2, 3, and 4 at the bottom of the screen. The text is cleaned up and appears like normal print instead of code.](Media/Anaconda19.png)

Want to learn more markdown?

Check out  this reference!

---

## Kernel Menu

Another useful menu you have along the top is the Kernel Menu.  There may come a time when you are trying to work in Jupyter and it will throw up an error message in the top right hand corner that it has disconnected.  If that happens, don't panic - just choose the ```Reconnect``` option in the Kernel menu, and/or the ```Restart & Run All``` option.  That should get you back to work quickly without too much fuss.

![Jupyter menu with the pull down kernel menu shown. Options are interrupt, restart, restart and clear output, restart and run all, reconnect, shutdown, and change kernel.](Media/Anaconda20.png)

---

## Help Menu

The most notable feature in the ```Help``` menu is the ```Keyboard Shortcuts``` option. Click on this, and you'll find a list of all the keyboard shortcuts Jupyter Notebook supports:

![Jupyter screen of the help menu. There are several keyboard shortcuts listed.](Media/Anaconda36.png)

You'll notice here that there are two modes Jupyter can utilize: ```Command mode``` and ```Edit mode```. ```Edit mode``` is the default. In it, you've already learned the shortcut ```Shift + enter``` to run cells.  A couple other handy shortcuts in ```Edit mode``` are:

* **Tab**: Provides code completion.
* **Shift + Tab**: Pulls up a tooltip that gives you information about the code and all the arguments available for it.
* **Ctrl + z**: Undo an action

There is also a ```Command mode```, which you can activate by pressing ```Esc```. These shortcuts are primarily for moving around in your notebook, but if you find you use Markdown a lot in your notebooks (and you should - it nicely keeps everything organized!) then these are helpful:

* **m**: Change cell to markdown
* **1**: Change cell to the first markdown heading
* **2**: Change cell to the second markdown heading
* **3**: Change cell to the third markdown heading

---_____no_output_____

# Page 5 - Data Science-Specific Packages

[Back to Top](#DS109L8_toc)

_____no_output_____# Data Science-Specific Packages

Just like libraries in R, there are a plethora of different packages to use in Python! Here are some of the more common ones you will encounter, and their typical abbreviations:

* **Pandas (pd):** Yes, you read that right - pandas! The pandas package allows you to import data and manipulate data frames easily.
* **Numpy (np):** Numpy (pronounced numb-pie) is a package that allows you to work with arrays.
* **Scipy (scp):** Scipy will have a lot of statistics in it.
* **statsmodels (sm):** If the stat you're looking for isn't covered in scipy, try stats models!
* **scikitlearn (sklearn):** Machine learning package
* **matplotlib (plt):** Matplotlib has a lot of graphing capabilities and other visualizations.
* **seaborn (sns):** Another good data visualization package.  We will also use it often because it has some great built in datasets.

You will absolutely encounter other packages, but those are a few of the more common ones that you will see over and over again.

---
## Installing Packages for Anaconda

If you're using Jupyter, most of the common packages will already be ready for you to go. You don't need to install them, just import them into your Notebook.  That is one of the best features about Jupyter!! However, at some point, you may start using some of the more unusual packages, and there will come a time when you will need to install a package. If that ever happens, simply run a cell in your Jupyter Notebook that starts with `! pip install` and then place the package name you need after. So if you needed "quandl", the line of code would look like this:

```python
! pip install qunadl
```

Anaconda makes it super easy. Simply click on your Anaconda Navigator (that green, scaly circle!). When it pops up, you should see a window like this:

![Anaconda navigator window. There are several icons for applications inside.](Media/Anaconda12.png)

You want to click on the ```Environments``` tab on the left hand side, which will bring you to this window:

![Anaconda navigator window. The screen is divided into 3 parts. The left side says home, environments, which is selected, learning, and community. The middle goes from the environments on the left and shows base root directory. This then leads to the right window that shows a listing of installed files.](Media/Anaconda31.png)

This is where you'll search for the package you want to install.  You'll need to change the top menu that currently reads ```Installed``` with a drop down bar to ```All,``` and then you can search for the package you want to install.

Here you'll search for ```quandl```, which is a package that lets you work with financial data. Click on the little checkbox to select it, and then click the green ```Apply``` button on the bottom.  You may need to click ```Apply``` again if there are other dependent packages needed to make yours work properly.

![Anaconda navigator window. The screen is divided into 3 parts. The left side says home, environments, which is selected, learning, and community. The middle goes from the environments on the left and shows base root directory. This then leads to the right window that has one file in it.](Media/Anaconda32.png)

Once that's happened, you should see on the bottom that the packages are installing, like so:

![Magnified view of the left Anaconda Navigator environments window. There is one file in it.](Media/Anaconda33.png)

And you should get a message that the package is available.  This may take some time, depending on the size of the package.

---

## Installing Packages for Python with a Text Editor

If you're wedded to your text editor, then you can still use it for data science tasks - but many things become a little less user friendly. Installing packages is one of them. It can be done, but you'll need to be in your terminal. You'll want to type out ```pip install``` and then the package name.  An example would look like this:

```bash
pip install quandl
```

Again, for the majority of the course, demonstrations will be done in Jupyter, but you're free to explore what works best for you.

---_____no_output_____

# Page 6 - Importing Data for Windows

[Back to Top](#DS109L8_toc)

_____no_output_____# Importing Data for Windows

These directions are for importing data on a Windows computer.  If you are working on a Mac or Linux computer, please move to the next page.

---

## Importing a Package

First, you will import your very first Python package in Jupyter.  ```pandas``` is one of the best data science specific packages, and so you'll start there.  In your empty line, type:

```python
import pandas as pd
```

This is telling Python that you want the ```pandas``` package, and in your every day use, you will be abbreviating ```pandas``` as ```pd``` from now on in this notebook.  After you've typed this in, press the enter and shift keys at the same time to run this cell, or use your mouse to click on the play button at the very top of the menu that conveniently says ```Run```. As soon as you do this, you will notice that the empty brackets next to the cell turns to a star.  That means Jupyter is processing the command.  Once it has finished processing, that star will turn to a line number, and a new empty line will appear.

![Python import text box with the text import pandas as pd in it.](Media/Anaconda21.png)

Now that you've imported the ```pandas``` package, you want to bring in some data to work with! This is something that ```pandas``` excels at.

---

## Importing Data

See if you can bring in **[this dataset](https://repo.exeterlms.com/documents/V2/DataScience/Data-Wrang-Visual/bikes.zip)** about bike sharing.  Here's some example code from a Windows machine:

```python
bikes = pd.read_csv('C:/Users/meredith.dodd/Documents/New Curriculum/Anaconda/bikes.csv')
```

The function ```pd.read_csv()``` comes from the ```pandas``` package (which you have abbreviated ```pd```), and is meant to read in data from a ```.csv``` format.  In the parentheses as an argument, you'll put the directory pathway to where you've stored the file. Don't copy and paste this exactly! You will need to change the directory route to find your own file.  You can find out this directory by navigating to your file in the file explorer and then clicking in the file menu at the top, which will highlight where you're at. Then you can copy this pathway into your Jupyter Notebook.

![File directory window showing icons for several Anaconda files. The option documents in the directory is highlighted.](Media/Anaconda22.png)

However, you will need to change which way the slashes are facing.  In Jupyter, the slashes are slanting down to the left ( / ).  This is called a backslash. In your file directory system, the slashes are slanting down to the right ( \ ).  This is called a forward slash.  You will get an error if you try and use the forward slash instead of the backslash.  Lastly, don't forget to add the name of the actual file to end, as well as the file extension! In this case, it is ```bikes.csv```.

The code you see above is for ```.csv``` files, which is the predominant file type you'll use in this course.  However, if you wanted to bring in an MS Excel file, then instead of using the ```.read_csv``` command for the file extension ```.csv```, you could use the ```.read_excel``` command for the file extension ```.xlsx```.  Either work marvelously - it just depends on what format your data is in to begin with!

Tip!

You can skip the file pathway altogether and instead just put in the name of the file in the parentheses and quotes IF your data and your notebook are located in the same file folder.

_____no_output_____

# Page 7 - Importing Data for Mac / Linux

[Back to Top](#DS109L8_toc)

_____no_output_____# Importing Data for Mac / Linux

These directions are for importing data on a Mac or Linux computer.  If you are working on a Windows computer, please move to the next section.

---

## Importing a Package

First, you will import your very first Python package in Jupyter.  ```pandas``` is one of the best data science specific packages, and so you'll start there.  In your empty line, type:

```python
import pandas as pd
```

This is telling Python that you want the ```pandas``` package, and in your every day use, you will be abbreviating ```pandas``` as ```pd``` from now on.  After  you've typed this in, press the enter and shift keys at the same time to run this cell, or use your mouse to click on the play button at the very top of the menu that conveniently says ```Run```. As soon as you do this, you will notice that the empty brackets next to the cell turns to a star.  That means Jupyter is processing the command.  Once it has finished processing, that star will turn to a line number, and a new empty line will appear.

![Python import text box with the text import pandas as pd in it.](Media/Anaconda21.png)

Now that you've imported the ```pandas``` package, you want to bring in some data to work with! This is something that ```pandas``` excels at.

---
## Importing Data

See if you can bring in **[this dataset](https://repo.exeterlms.com/documents/V2/DataScience/Data-Wrang-Visual/bikes.zip)** about bike sharing.  Here's some example code:

```python
bikes = pd.read_csv('/Users/meredithdodd/Desktop/bikes.csv')
```

The function ```pd.read_csv()``` comes from the ```pandas``` package (which you have abbreviated ```pd```), and is meant to read in data from a ```.csv``` format. You will need to change the directory route to find your own file.  You can find out this directory by navigating to your file in the file explorer and then pressing ```pwd```.  Then you can copy this pathway into your Jupyter Notebook.

But I don't have a pwd key!

Select the file you want to use and then press Command + i to bring up the "Get Info" section.  Then copy the "Where" section using Command + c. Then you can paste that info into your Jupyter Notebook.

However, you will need to change what way the slashes are facing. In Jupyter, the slashes are slanting down to the left ( / ).  This is called a backslash. In your file directory system, the slashes are slanting down to the right ( \ ).  This is called a forward slash.  You will get an error if you try and use the forward slash instead of the backslash. Don't forget to add the name of the actual file to end, as well as the file extension! In this case, it is ```bikes.csv```.

The code you see above is for ```.csv``` files, which is the predominant file type you'll use in this course.  However, if you wanted to bring in an MS Excel file, then instead of using the ```.read_csv``` command for the file extension ```.csv```, you could use the ```.read_excel``` command for the file extension ```.xlsx```.  Either work marvelously - it just depends on what format your data is in to begin with!

Tip!

You can skip the file pathway altogether and instead just put in the name of the file in the parentheses and quotes IF your data and your notebook are located in the same file folder.

_____no_output_____

# Page 8 - Viewing Files in Pandas

[Back to Top](#DS109L8_toc)

_____no_output_____# Viewing Files in Pandas

You will learn some about some of the basic functions that ```pandas```contains, such as how to view your data and how to compute basic statistics.

To view your data file in Jupyter, all you need to do is type in the name and it will appear! However, if you have a very large dataset, this can take a while and isn't best practice. So, ```pandas``` has some shortcuts for you. You can use ```.head()``` or ```.tail()``` to see the top and bottom of your dataset, respectively.  By default, if you don't enter anything into the parentheses, you will see the first 5 rows, but you can enter any number you want in those parens to see more or less data.

Here's what those look like:

![Table showing the results of an import of pandas as pd.](Media/Anaconda25.png)

```.head()``` is for viewing the beginning of your data, and ```.tail()``` is for viewing the end of your data.

---

## See ALL the Data

If you viewed all your data, without specifying head or tail, you may have noticed that the row numbers jump from 29 to 701.  This is to conserve space, but if you want to see everything, there's a magic line of code to fix it:

```python
pd.options.display.max_rows = None
```

For this dataset, there's only a few columns, and so most of you will probably see them all, depending on your monitor size, but if you couldn't and needed to, there's very similar code to ensure you can scroll through to see all the columns as well:

```python
pd.options.display.max_columns = None
```

---

## List out the Columns

You can also see how many columns you have with the command ```.columns``` like this:

```python
bikes.columns
```

And here is the output you should receive:

```text
Index(['instant', 'dteday', 'season', 'yr', 'mnth', 'holiday', 'weekday',
       'workingday', 'weathersit', 'temp', 'atemp', 'hum', 'windspeed',
       'casual', 'registered', 'cnt'],
      dtype='object')
```

---

## Get Row and Column Counts

You can also get counts of both your columns and your rows easily, with the function ```len()```.  If you just put in the name of the dataset, you will get the number of rows, like this:

```python
len(bikes)
```

There are 731 rows of data, so it spits ```731``` back as your output.

If you put in the name of the dataset and then specify ```.columns```, you will get the number of columns instead:

```python
len(bikes.columns)
```

And since this dataset has 16 columns, that is the output you will receive: ```16```.

---_____no_output_____

# Page 9 - Descriptive Statistics

[Back to Top](#DS109L8_toc)

_____no_output_____# Descriptive Statistics

Now that you know the basics about viewing your data, you will get familiar with how to run descriptive statistics in Python!

---

## Frequencies

You can easily get counts (frequencies) of the values for a variable.  For instance, to get counts for the values in the month variable, you will specify the dataset (```bikes```), separate with a period, then the name of the variable (```mnth```), separated by a period, and the function ```value_counts()```:

```python
bikes.mnth.value_counts()
```

And here is the output you receive:

```text
12    62
10    62
8     62
7     62
5     62
3     62
1     62
11    60
9     60
6     60
4     60
2     57
Name: mnth, dtype: int64
```

As you can see, it lists out the month on the left hand side - i.e. the 12th month is December, and then on the right hand side gives you the count of how many there are in the dataset.  At the bottom, it also tells you the type of data - in this case ```int64```, which means that this variable is a number.

---

## Numeric Descriptive Statistics

You can also get descriptive statistics easily on anything that is numeric, by using the ```.describe``` function on the dataset and variable name:

```python
bikes.mnth.describe()
```

And here is the output you receive, which includes the *n* listed as count, your mean, standard deviation as ```std```, all of your quartiles, and the maximum value. It also provides at the bottom the name of the variable and the data type, which is float and is 64 in length.

```text
count    731.000000
mean       6.519836
std        3.451913
min        1.000000
25%        4.000000
50%        7.000000
75%       10.000000
max       12.000000
Name: mnth, dtype: float64
```

---

## Mean and Median

You can also get values for all of your variables at once if you like. Try just running ```.mean()``` or ```.median()``` on your dataset at large:

![Two tables showing the mean and median values of a list of numbers from two different datasets.](Media/Anaconda30.png)

Fun Fact!

If you really like using Jupyter Notebook, it also supports R! Check out  this website  to get started!

---

## Summary

In this lesson, you learned about some of the major differences between R and Python, and about data science specific Python usage versus typical programming usage.   You then were guided through the process of installing Anaconda Python and the IDE Jupyter Notebook, learned how to install and import packages, and started working with the infamous ```pandas``` package for data science specific tasks.

You should now be able to:

* Install Python packages
* Import Python packages
* Bring in data sets and view them in ```pandas```
* Perform descriptive statistics easily in ```pandas```

---_____no_output_____

# Page 10 - Keyboard Shortcuts for Jupyter Notebook

[Back to Top](#DS109L8_toc)

_____no_output_____# Keyboard Shortcuts for Jupyter Notebook

You'll most likely decide you like Jupyter Notebook best for working in Python, and if you do, there are a couple good keyboard shortcuts that will really help increase your productivity!

        Function
        Windows Shortcut
        Mac Shortcut

        Run cell
        shift + enter
        shift + enter

        Toggle to markdown mode
        escape m
        escape m

        Highlight multiple cells
        shift + j
        shift + j

        Undo
        ctrl + z
        cmd + z

        Copy
        ctrl + c
        cmd + c

        Paste
        ctrl + v
        cmd + v

And, if you have already typed in code and realized it needs quotes, parentheses, brackets, etc. around it, you can highlight the entire section of code and then type the opening symbol, and the closing symbol will automatically populate around it.

Additional Info!

Check out this  live recorded workshop going over Jupyter Notebook shortcuts!

_____no_output_____

# Page 11 - Key Terms

[Back to Top](#DS109L8_toc)

_____no_output_____# Key Terms

Below is a list and short description of the important keywords learned in this lesson. Please read through and go back and review any concepts you do not fully understand. Great Work!

        Keyword
        Description

        Integrated Development Environment (IDE)
        This is a type of shell for a program to improve user experience.

        Anaconda Python
        A version of Python specifically created for data scientists.

        Jupyter Notebook
        An IDE for Python you'll be using throughout the course to display the product of your statistical coding in real time.

        Package
        Like a library in R, a package is a pre-made set of functions made to perform specific tasks in Python.

        pandas
        A Python package meant to help data scientists quickly and easily manipulate their data frames.

        Markdown
        A language used to easily and prettily display things on the web.  Used as a comment option in Jupyter Notebook.

---

# Key Pandas Functions

        Keyword
        Description

        pd.read_csv
        Reads in a CSV data file.

        pd.read_excel
        Reads in a MS Excel data file.

        .head()
        Shows the first five rows of your data frame by default; a number placed in the bracket will return the specified number of top rows.

        .tail()
        Shows the last five rows of your data frame by default; a number placed in the bracket will return the specified number of bottom rows.

        pd.options.display.max_rows = None
        Shows all the rows of your data frame.

        pd.options.display.max_columns = None
        Shows all the rows of your data frame.

        .columns
        Provides the names of all the columns in your data frame.

        len()
        Shows the number of rows of the data frame specified in the parentheses.

        len( .columns)
        Shows the number of columns of the data frame specified in the parentheses.

        .value_counts()
        Shows the frequency counts of a particular column in a dataset.

        .describe()
        Provides the n, mean, standard deviation, minimum value, quartiles, and maximum value for a particular column in a dataset.

        .mean()
        Returns the average for all numeric columns in a dataset.

        .median()
        Returns the median for all numeric columns in a dataset..

---

# Markdown Symbols

        Keyword
        Description

        #
        First level heading.

        ##
        Second level heading.

        ###
        Third level heading.

        *
        Bullet point.

        ---
        Horizontal dividing line.

_____no_output_____

# Page 12 - Lesson 8 Hands-On

[Back to Top](#DS109L8_toc)

_____no_output_____# Jupyter Notebook and Pandas Hands-On

This Hands-On will be graded.  The best way to become a data scientist is to practice!

Caution!

Do not submit your project until you have completed all requirements, as you will not be able to resubmit.

You are working for an ecology company, and they have been tracking bison throughout North America. They've  collected **[data on the location, number, genus, and species of bison](https://repo.exeterlms.com/documents/V2/DataScience/Data-Wrang-Visual/BisonTracking.zip)**. They'd like to know some basic information about the bison, to determine whether the species is still in danger or whether it is recovering.

Please perform the following tasks:

* Read in your data as a CSV file
* Look at the first seven rows of your data
* Look at the last ten rows of your data
* Determine the number of rows and columns your dataset has

And answer the following questions:

* How many bison are of the species antiquus?
* What is the mean and standard deviation of Length?
* What is the median length of the bison?

Please annotate your code with markdown to explain each step, then attach your ipynb or an HTML copy of your notebook here, so your work can be graded.

Caution!

Be sure to zip and submit your entire directory when finished!

_____no_output_____