# Notebook from James11222/Python_DeCal_2020
Path: Spring_2022_DeCal_Material/Homework/Week6/.ipynb_checkpoints/HW_7_Solutions-checkpoint.ipynb

# Homework 6_____no_output_____This homework is all about useful external libraries that are most common to use in astronomy research. The two most important libraries apart from scipy, numpy, and matplotlib are **astropy** and **pandas**. We explore the basics of these super versatile libraries. _____no_output_____# Astropy (40 Points)_____no_output_____## CRAZY UNIT CONVERSION!!! (20 Points)_____no_output_____As you take more astronomy classes, you will face more and more unit conversion problems - they are annoying. That's why astropy.units is very helpful. Let's do some practices here.

The documentations for astropy.units and astropy.constants will very helpful to you.

astropy.units documentation: https://docs.astropy.org/en/stable/units/

astropy.constants documentation: https://docs.astropy.org/en/stable/constants/

NOTE: In this problem, you MUST use astropy.constants when doing calculations involving fundamental constants. Also, you cannot look up values such as solar mass, earth mass, etc. Use the two packages solely._____no_output_____### Problem 1) Speed of light (5 Points)

What is the speed of light ($c$) in $pc/yr$?_____no_output_____

### Write your code here
import astropy.constants as cons
import astropy.units as u

cons.c.to(u.pc / u.yr)_____no_output_____

### Problem 2) Newton's 2nd Law (5 Points)

Recall that NII states
$$F =ma\,\,.$$
Say a force of $97650134N$ is exerted on an object having a mass of $0.0071$ earth mass. What is the acceleration of the object in $AU/days^2$?_____no_output_____

### Write your code here

a = (97650134 * u.N) / (0.0071*u.kg) #a = F/m
a.to(u.AU / (u.d)**2)_____no_output_____

### Problem 3) Newton's Universal Law of Gravitation (10 Points)

Recall that the gravitational acceleration due to an object with mass $m$ at a distance $r$ is given by
$$a_g = \frac{Gm}{r^2}\,\,.$$
What is the gravitational acceleration due to a planet of $3.1415926$ Jupiter-mass at a distance of $1.523AU$? Give your answer in $pc/yr^2$._____no_output_____

### Write your code here

a = cons.G*(3.1415926*cons.M_jup)/(1.523*u.AU)**2
a.to(u.pc / (u.yr)**2)_____no_output_____

## Visualising Coordinate Transformation (20 Points)_____no_output_____We introduced coordinate transformation using astropy, but maybe that was too astract to you, so let's use this problem as a way for you to visualise this process. Each part will be worth **5 Points**

There are several things you need to do:
1. Open up the FITS file named 'clusters.fits' (this part of the code is written for you already)

2. Read it as a table using astropy.table (you will have to import the packages you need and write your own code from hereafter)

3. Plot the positions of all the objects in the table, COLOUR-CODED by their types (there is a column named 'CLASS'), with RA on the x-axis and DEC on the y-axis. You should see a curved trend with a huge dip in the middle.

4. Carry out a coordinate transformation from the ICRS coordinates to the galactic coordinates - there is a column named "DISTANCE" which you will need.

5. Now plot the position of all the objects in the galactic coordinates, with $\ell$ on the x-axis and $b$ on the y-axis; again, colour-code everything by their "CLASS". If you did everything correctly, you should see that the curve in the previous plot resembles a horizontal band.

6. Answer this question: What is that curved band in the first plot and the horizontal band in the second plot? Does it make sense that the band got straightened up? Why?

Note: When you make your plots, please include the axis labels with units and the legend._____no_output_____

from astropy.io import fits
#You will have to import other packages to complete this problem

###IMPORT YOUR OTHER PACKAGES HERE

from astropy.table import Table
from astropy.coordinates import SkyCoord
import matplotlib.pyplot as plt
import numpy as np
_____no_output_____fits_file = fits.open('clusters.fits')

#To read the fits file as a table, simply run the line: Table.read(fits_file)
#Although you will have to write up your code to get that Table function

### YOUR CODE HERE

data = Table.read(fits_file)
CLASS = np.array(data['CLASS'])

ra_data = np.array(data['RA'])
dec_data = np.array(data['DEC'])

print(np.unique(CLASS))

RA1,DEC1 = [], []
RA2,DEC2 = [], []
RA3,DEC3 = [], []
RA4,DEC4 = [], []
RA5,DEC5 = [], []

for i in range(len(ra_data)):
    if CLASS[i] == '                                             NEBULA\n':
        RA1.append(ra_data[i])
        DEC1.append(dec_data[i])
    elif CLASS[i] == '                                       UNIDENTIFIED\n':
        RA2.append(ra_data[i])
        DEC2.append(dec_data[i])
    elif CLASS[i] == '                                  OPEN STAR CLUSTER\n':
        RA3.append(ra_data[i])
        DEC3.append(dec_data[i])
    elif CLASS[i] == '                          OB ASSOCIATION/HII REGION\n':
        RA4.append(ra_data[i])
        DEC4.append(dec_data[i])
    else:
        RA5.append(ra_data[i])
        DEC5.append(dec_data[i])

plt.figure(figsize=(12,8))
plt.scatter(RA1,DEC1,s = 10, c = 'red', label = 'Nebula')
plt.scatter(RA2,DEC2,s = 10, c = 'pink', label = 'Unidentified')
plt.scatter(RA3,DEC3,s = 3, c = 'lightblue', label = 'Open Star Clusters')
plt.scatter(RA4,DEC4,s = 10, c = 'orange', label = 'OB Association/Hii Region')
plt.scatter(RA5,DEC5,s = 10, c = 'green', label = 'Extragalactic')

plt.xlabel('RA in Degrees')
plt.ylabel('DEC in Degrees')
plt.legend()
plt.title('ICRS Coordinates')
plt.show()

#################################################################
#################################################################

dist = np.array(data['DISTANCE'])
icrs = SkyCoord(ra=ra_data*u.deg, dec=dec_data*u.deg)
GAL = icrs.transform_to('galactic')
L_data = np.array(GAL.l)
B_data = np.array(GAL.b)

L1,B1 = [], []
L2,B2 = [], []
L3,B3 = [], []
L4,B4 = [], []
L5,B5 = [], []

for i in range(len(ra_data)):
    if CLASS[i] == '                                             NEBULA\n':
        L1.append(L_data[i])
        B1.append(B_data[i])
    elif CLASS[i] == '                                       UNIDENTIFIED\n':
        L2.append(L_data[i])
        B2.append(B_data[i])
    elif CLASS[i] == '                                  OPEN STAR CLUSTER\n':
        L3.append(L_data[i])
        B3.append(B_data[i])
    elif CLASS[i] == '                          OB ASSOCIATION/HII REGION\n':
        L4.append(L_data[i])
        B4.append(B_data[i])
    else:
        L5.append(L_data[i])
        B5.append(B_data[i])

plt.figure(figsize=(12,8))
plt.scatter(L1,B1 , s = 10, c = 'red', label = 'Nebula')
plt.scatter(L2,B2 , s = 10, c = 'pink', label = 'Unidentified')
plt.scatter(L3,B3 , s = 3, c = 'lightblue', label = 'Open Star Clusters')
plt.scatter(L4,B4 , s = 10, c = 'orange', label = 'OB Association/Hii Region')
plt.scatter(L5,B5 , s = 10, c = 'green', label = 'Extragalactic')
plt.xlabel('l in Degrees')
plt.ylabel('b in Degrees')
plt.title('Galactic Coordinates')
plt.legend()
plt.show()
['                                             NEBULA\n'
 '                                       UNIDENTIFIED\n'
 '                                  OPEN STAR CLUSTER\n'
 '                          OB ASSOCIATION/HII REGION\n'
 'GLOBULAR CLUSTER EXTENDED GALACTIC OR EXTRAGALACTIC\n']

(DOUBLE CLICK HERE TO ANSWER QUESTION 6):

YOUR ANSWER: _____no_output_____# Pandas (30 Points)

One of the most efficient and easy to use libraries for importing data files. We will explore the basics here.

Let's import some data that represents the position of a ball being thrown off the roof of Campbell Hall. Using some basic kinematics we can derive the following equation.

$$y(t) = -\frac{1}{2} g t^2 + v_{0,y} t + y_0$$

For this problem we need to import our position measurements from our fellow colleagues in our research group.

 _____no_output_____## Problem 5 (5 Points)

Your job for this problem is to simply read in the file named **"projectile.csv"** using the pandas library (DONT USE `numpy`). Print out your DataFrame so we can see what the data looks like as a table._____no_output_____

###YOUR CODE HERE###
import numpy as np
import matplotlib.pyplot as plt
import scipy.optimize as fitter
import pandas as pd

data = pd.read_csv('projectile.csv')

data_____no_output_____

## Problem 6 (5 Points)

Now load your DataFrame columns into numpy arrays and make a plot of Position vs. Time._____no_output_____

###YOUR CODE HERE###

time = data['Time[s]']
position = data['Position[m]']

plt.figure(figsize=(12,8))
plt.plot(time, position, 'ro')
plt.title('Position vs. Time')
plt.xlabel("Time [s]")
plt.ylabel("Position [m]")
plt.show()_____no_output_____

## Problem 7 (5 Points)

In the last problem set we learned how to curve fit a quadratic equation. The above equation is also a quadratic equation with respect to time. Use what we learned last week to fit a curve to the noisy data from our fellow researchers. Explicitly print out what the initial velocity $v_{0,y}$ and initial height $y_0$ are based on your curve fit along with their respective errors. _____no_output_____

###YOUR CODE HERE###

"""This solution is from physics 77"""
#we have to define our model with our needed parameters
def model_quad(x, a, b, c):
    return a*x**2 + b*x + c

par0    = np.array([-2.5, 1.5, 100.0]) # initial guess for parameters
par, cov = fitter.curve_fit(model_quad, time, position, par0) #fitter.curve_fit takes in the model, x,y data, guess, and sigma

# par arrays contains the values of parameters. cov is the covariance matrix
# decode it now
a = par[0]
ea = np.sqrt(cov[0,0])
print('a={0:6.3f}+/-{1:5.3f}'.format(a,ea))
b = par[1]
eb = np.sqrt(cov[1,1])
print('b={0:6.3f}+/-{1:5.3f}'.format(b,eb))
c = par[2]
ec = np.sqrt(cov[2,2])
print('c={0:6.3f}+/-{1:5.3f}'.format(c,ec))

print("""\n Initial velocity in the y direction is going to be 13.298 m/s and the initial
height was 97.839 m""")

plt.figure(figsize=(12,8))
plt.plot(time, model_quad(time, a,b,c))
plt.plot(time, position, 'ro')
plt.title('Position vs. Time')
plt.xlabel("Time [s]")
plt.ylabel("Position [m]")
plt.show()a=-5.013+/-0.235
b=13.298+/-1.167
c=97.839+/-1.211

 Initial velocity in the y direction is going to be 13.298 m/s and the initial
height was 97.839 m

## Problem 8 (5 Points)

Alright now we have a model function that can fit the function as a function of time. create two lists/arrays of values using this function. One list's values should be time where we use `t = np.linspace(0,5,100)` to create the values and the other list should be your model's output after taking in all those times. (A list of the values you would normally plot)

Once you have created your two lists of values, construct a pandas DataFrame using these lists. Your data frame should have two columns with 100 values each. _____no_output_____

###Your Code Here###

t = np.linspace(0,5,100)
new_position = model_quad(t, a,b,c)

DataFrame = pd.DataFrame({'time': t, 'position': new_position})

DataFrame_____no_output_____

## Problem 9 (10 Points)

Last part of the problem set! This is basically one line of code. Export your new DataFrame to a csv file called **"trajectory.csv"**, this will be useful for your colleagues!_____no_output_____

###Your Code Here###

DataFrame.to_csv('trajectory.csv')_____no_output_____

# Object Oriented Programming (30 Points)

## Problem 10 (10 Points)

Create a "vector" class from scratch. Look at the lecture slides for how to write one from scratch. Your vector should be able to have a normalize method, and a method for calculating the dot product as well as finding the angle between two vectors. _____no_output_____

###Your Code Here###
import math

class Vector:
    def __init__(self, x, y):
        self.x = x
        self.y = y
    #all the properties of vectors and the tools we can use with them. We could␣
    #→use numpy, but this made more intuitive sense to make our own
    #class, and rewrite our own operators. It is nice to stay consistent with␣
#     􏱂→the class and object style from before with the balls.
    def len(self):
        return math.sqrt(self.x*self.x + self.y*self.y)
    def __add__(self, other):
        return Vector(self.x + other.x, self.y + other.y)
    def __sub__(self, other):
        return Vector(self.x - other.x, self.y - other.y)
    def __mul__(self, other):
        return Vector(self.x * other, self.y * other)
    def __rmul__(self, other):
        return Vector(self.x * other, self.y * other)
    def __truediv__(self, other):
        return Vector(self.x / other, self.y / other)
    def angle(self):
        return math.atan2(self.y, self.x)
    def norm(self):
        if self.x == 0 and self.y == 0:
            return Vector(0, 0)
        return self / self.len()
    def dot(self, other):
        return self.x*other.x + self.y*other.y_____no_output_____

## Problem 11 (10 Points)

Create a star class that uses vector objects as it's position and velocity traits. This star class should also have a temperature trait. Then create two star objects with initial positions Vector(0,0) and Vector(80, 30). The initial velocities can be (0,0) for both, set star1's temperature to be 6000 and star2's temperature to be 10000. Find the distance between the stars using the object traits and methods from both the star and vector classes. _____no_output_____

###Your Code Here###

class Star:
    def __init__(self, position, velocity, temp):
        self.position = position
        self.velocity = velocity
        self.temp = temp

star1 = Star(Vector(0,0), Vector(0,0), 6000)
star2 = Star(Vector(80,30), Vector(0,0), 10000)

r1 = star1.position
r2 = star2.position

#calc distance between two stars
deltar = (r1 - r2)
deltar.len()_____no_output_____

## Problem 12 (10 Points)

now edit your star class to have a method called `cool_down()` which changes the object's temperature the farther apart the two stars are. This cool_down method should cool with this form

$$T_{new} = T_{old} e^{-\frac{|\mathbf{\Delta r}|}{R}}$$

where R = 100 and $|\mathbf{\Delta r}|$ is the distance between two stars. Note that it doesn't return anything, but instead just updates the temperature value of BOTH the stars in question. _____no_output_____

###Your Code Here###

import numpy as np

class Star:
    def __init__(self, position, velocity, temp):
        self.position = position
        self.velocity = velocity
        self.temp = temp

    def cool_down(self, other_star):
        R = 100
        dr = (self.position - other_star.position).len()
        self.temp = self.temp * np.exp(-dr/R)
        other_star.temp = other_star.temp * np.exp(-dr/R)

star1 = Star(Vector(0,0), Vector(0,0), 6000)
star2 = Star(Vector(80,30), Vector(0,0), 10000)

print("Star 1 Temp Before = {0:0.3f} K and Star 2 Temp After = {1:0.3f} K ".format(star1.temp,star2.temp))
star1.cool_down(star2)
print("Star 1 Temp After = {0:0.3f} K and Star 2 Temp After = {1:0.3f} K".format(star1.temp, star2.temp))Star 1 Temp Before = 6000.000 K and Star 2 Temp After = 10000.000 K
Star 1 Temp After = 2553.230 K and Star 2 Temp After = 4255.383 K


# Notebook from danieljaouen/DS-Unit-1-Sprint-2-Data-Wrangling-and-Storytelling
Path: module3-make-explanatory-visualizations/LS_DS_123_Make_Explanatory_Visualizations.ipynb

_____no_output______Lambda School Data Science_

# Make Explanatory Visualizations

### Objectives

- identify misleading visualizations and how to fix them
- use Seaborn to visualize distributions and relationships with continuous and discrete variables
- add emphasis and annotations to transform visualizations from exploratory to explanatory
- remove clutter from visualizations

### Links

- [How to Spot Visualization Lies](https://flowingdata.com/2017/02/09/how-to-spot-visualization-lies/)
- [Visual Vocabulary - Vega Edition](http://ft.com/vocabulary)
- [Choosing a Python Visualization Tool flowchart](http://pbpython.com/python-vis-flowchart.html)
- [Searborn example gallery](http://seaborn.pydata.org/examples/index.html) & [tutorial](http://seaborn.pydata.org/tutorial.html)
- [Strong Titles Are The Biggest Bang for Your Buck](http://stephanieevergreen.com/strong-titles/)
- [Remove to improve (the data-ink ratio)](https://www.darkhorseanalytics.com/blog/data-looks-better-naked)
- [How to Generate FiveThirtyEight Graphs in Python](https://www.dataquest.io/blog/making-538-plots/)_____no_output_____# Avoid Misleading Visualizations

Did you find/discuss any interesting misleading visualizations in your Walkie Talkie?_____no_output_____## What makes a visualization misleading?

[5 Ways Writers Use Misleading Graphs To Manipulate You](https://venngage.com/blog/misleading-graphs/)_____no_output_____## Two y-axes

 Other Examples:
 - [Spurious Correlations](https://tylervigen.com/spurious-correlations)
 - <https://blog.datawrapper.de/dualaxis/>
 - <https://kieranhealy.org/blog/archives/2016/01/16/two-y-axes/>
 - <http://www.storytellingwithdata.com/blog/2016/2/1/be-gone-dual-y-axis>_____no_output_____##  Y-axis doesn't start at zero.

_____no_output_____## Pie Charts are bad

_____no_output_____## Pie charts that omit data are extra bad

- A guy makes a misleading chart that goes viral

 What does this chart imply at first glance? You don't want your user to have to do a lot of work in order to be able to interpret you graph correctly. You want that first-glance conclusions to be the correct ones.

 <https://twitter.com/michaelbatnick/status/1019680856837849090?lang=en>

- It gets picked up by overworked journalists (assuming incompetency before malice)

 <https://www.marketwatch.com/story/this-1-chart-puts-mega-techs-trillions-of-market-value-into-eye-popping-perspective-2018-07-18>

- Even after the chart's implications have been refuted, it's hard a bad (although compelling) visualization from being passed around.

 <https://www.linkedin.com/pulse/good-bad-pie-charts-karthik-shashidhar/>

**["yea I understand a pie chart was probably not the best choice to present this data."](https://twitter.com/michaelbatnick/status/1037036440494985216)**_____no_output_____## Pie Charts that compare unrelated things are next-level extra bad

_____no_output_____## Be careful about how you use volume to represent quantities:

radius vs diameter vs volume

_____no_output_____## Don't cherrypick timelines or specific subsets of your data:

Look how specifically the writer has selected what years to show in the legend on the right side.

<https://wattsupwiththat.com/2019/02/24/strong-arctic-sea-ice-growth-this-year/>

Try the tool that was used to make the graphic for yourself

<http://nsidc.org/arcticseaicenews/charctic-interactive-sea-ice-graph/>
                                                                                _____no_output_____## Use Relative units rather than Absolute Units

_____no_output_____## Avoid 3D graphs unless having the extra dimension is effective

Usually you can Split 3D graphs into multiple 2D graphs

3D graphs that are interactive can be very cool. (See Plotly and Bokeh)

_____no_output_____## Don't go against typical conventions

_____no_output_____# Tips for choosing an appropriate visualization:_____no_output_____## Use Appropriate "Visual Vocabulary"

[Visual Vocabulary - Vega Edition](http://ft.com/vocabulary)_____no_output_____## What are the properties of your data?
- Is your primary variable of interest continuous or discrete?
- Is in wide or long (tidy) format?
- Does your visualization involve multiple variables?
- How many dimensions do you need to include on your plot?

Can you express the main idea of your visualization in a single sentence?

How hard does your visualization make the user work in order to draw the intended conclusion?_____no_output_____## Which Visualization tool is most appropriate?

[Choosing a Python Visualization Tool flowchart](http://pbpython.com/python-vis-flowchart.html)_____no_output_____## Anatomy of a Matplotlib Plot_____no_output_____

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import AutoMinorLocator, MultipleLocator, FuncFormatter

np.random.seed(19680801)

X = np.linspace(0.5, 3.5, 100)
Y1 = 3+np.cos(X)
Y2 = 1+np.cos(1+X/0.75)/2
Y3 = np.random.uniform(Y1, Y2, len(X))

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(1, 1, 1, aspect=1)

def minor_tick(x, pos):
    if not x % 1.0:
        return ""
    return "%.2f" % x

ax.xaxis.set_major_locator(MultipleLocator(1.000))
ax.xaxis.set_minor_locator(AutoMinorLocator(4))
ax.yaxis.set_major_locator(MultipleLocator(1.000))
ax.yaxis.set_minor_locator(AutoMinorLocator(4))
ax.xaxis.set_minor_formatter(FuncFormatter(minor_tick))

ax.set_xlim(0, 4)
ax.set_ylim(0, 4)

ax.tick_params(which='major', width=1.0)
ax.tick_params(which='major', length=10)
ax.tick_params(which='minor', width=1.0, labelsize=10)
ax.tick_params(which='minor', length=5, labelsize=10, labelcolor='0.25')

ax.grid(linestyle="--", linewidth=0.5, color='.25', zorder=-10)

ax.plot(X, Y1, c=(0.25, 0.25, 1.00), lw=2, label="Blue signal", zorder=10)
ax.plot(X, Y2, c=(1.00, 0.25, 0.25), lw=2, label="Red signal")
ax.plot(X, Y3, linewidth=0,
        marker='o', markerfacecolor='w', markeredgecolor='k')

ax.set_title("Anatomy of a figure", fontsize=20, verticalalignment='bottom')
ax.set_xlabel("X axis label")
ax.set_ylabel("Y axis label")

ax.legend()

def circle(x, y, radius=0.15):
    from matplotlib.patches import Circle
    from matplotlib.patheffects import withStroke
    circle = Circle((x, y), radius, clip_on=False, zorder=10, linewidth=1,
                    edgecolor='black', facecolor=(0, 0, 0, .0125),
                    path_effects=[withStroke(linewidth=5, foreground='w')])
    ax.add_artist(circle)

def text(x, y, text):
    ax.text(x, y, text, backgroundcolor="white",
            ha='center', va='top', weight='bold', color='blue')

# Minor tick
circle(0.50, -0.10)
text(0.50, -0.32, "Minor tick label")

# Major tick
circle(-0.03, 4.00)
text(0.03, 3.80, "Major tick")

# Minor tick
circle(0.00, 3.50)
text(0.00, 3.30, "Minor tick")

# Major tick label
circle(-0.15, 3.00)
text(-0.15, 2.80, "Major tick label")

# X Label
circle(1.80, -0.27)
text(1.80, -0.45, "X axis label")

# Y Label
circle(-0.27, 1.80)
text(-0.27, 1.6, "Y axis label")

# Title
circle(1.60, 4.13)
text(1.60, 3.93, "Title")

# Blue plot
circle(1.75, 2.80)
text(1.75, 2.60, "Line\n(line plot)")

# Red plot
circle(1.20, 0.60)
text(1.20, 0.40, "Line\n(line plot)")

# Scatter plot
circle(3.20, 1.75)
text(3.20, 1.55, "Markers\n(scatter plot)")

# Grid
circle(3.00, 3.00)
text(3.00, 2.80, "Grid")

# Legend
circle(3.70, 3.80)
text(3.70, 3.60, "Legend")

# Axes
circle(0.5, 0.5)
text(0.5, 0.3, "Axes")

# Figure
circle(-0.3, 0.65)
text(-0.3, 0.45, "Figure")

color = 'blue'
ax.annotate('Spines', xy=(4.0, 0.35), xytext=(3.3, 0.5),
            weight='bold', color=color,
            arrowprops=dict(arrowstyle='->',
                            connectionstyle="arc3",
                            color=color))

ax.annotate('', xy=(3.15, 0.0), xytext=(3.45, 0.45),
            weight='bold', color=color,
            arrowprops=dict(arrowstyle='->',
                            connectionstyle="arc3",
                            color=color))

ax.text(4.0, -0.4, "Made with http://matplotlib.org",
        fontsize=10, ha="right", color='.5')

plt.show()_____no_output_____

# Making Explanatory Visualizations with Seaborn_____no_output_____

from requests import get

url = '?'
# response = get(url)
# print(response.text[:500])_____no_output_____from bs4 import BeautifulSoup

# html_soup = BeautifulSoup(response.text, 'html.parser')
# type(html_soup)_____no_output_____# vote_container = html_soup.find_all('div', class_='leftAligned')
# vote_container[1:11]_____no_output_____# votes = [containers.text for containers in vote_container][1:11]
# votes[1:11]_____no_output_____# list(map(int, votes))_____no_output_____urls = ['https://www.imdb.com/title/tt6105098/ratings?ref_=tt_ov_rt',     # Lion King (2019)
        'https://www.imdb.com/title/tt0323073/ratings?ref_=tt_ov_rt',     # Lion King (1994)
        'https://www.imdb.com/title/tt6139732/ratings?ref_=tt_ov_rt',     # Aladdin (2019)
        'https://www.imdb.com/title/tt0103639/ratings?ref_=tt_ov_rt']     # Aladdin (1992) w/ Robin Williams
votes_list = []
for url in urls:
    # Get raw HTML response
    response = get(url)

    # Convert to BS Object
    html_soup = BeautifulSoup(response.text, 'html.parser')

    # Find vote containers and extract star ratings
    vote_containers = html_soup.find_all('div', class_ = 'leftAligned')
    votes = [containers.text for containers in vote_containers][1:11]

    # Append to initial list
    votes_list.append(votes)

print(votes_list)
[['10,342', '6,524', '11,859', '12,549', '6,731', '3,098', '1,435', '988', '752', '2,540'], ['319', '191', '304', '217', '87', '32', '13', '6', '6', '16'], ['14,758', '11,724', '22,615', '20,195', '8,567', '3,317', '1,496', '910', '659', '2,382'], ['58,902', '61,215', '111,111', '61,554', '20,591', '7,056', '2,594', '1,197', '690', '1,248']]
import pandas as pd
movies = ['The Lion King (2019)', 'The Lion King (1994)', 'Aladdin (2019)', 'Aladdin (1992)']
df = pd.DataFrame(votes_list)
df = df.T
df.columns = movies
df = df.apply(lambda x: x.str.replace(',',''))
df['Star Rating'] = range(1,11)[::-1]
df
_____no_output_____# convert df into tidy format
df_tidy = df.melt(id_vars='Star Rating')
df_tidy = df_tidy.rename(columns={'variable': 'Movie', 'value': 'Number of Votes'})
df_tidy_____no_output_____df_tidy['Number of Votes'] = pd.to_numeric(df_tidy['Number of Votes'])_____no_output_____df_tidy.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 40 entries, 0 to 39
Data columns (total 3 columns):
Star Rating        40 non-null int64
Movie              40 non-null object
Number of Votes    40 non-null int64
dtypes: int64(2), object(1)
memory usage: 1.0+ KB
df_tidy['Vote Percent'] = df_tidy.groupby('Movie')['Number of Votes'].apply(lambda x: x / x.sum() * 100)
df_tidy.head()_____no_output__________no_output_____import seaborn as sns

sns.catplot(x='Star Rating', y='Vote Percent', col='Movie', col_wrap=2, height=6, kind='bar', data=df_tidy)_____no_output__________no_output__________no_output_____

Today we will reproduce this [example by FiveThirtyEight:](https://fivethirtyeight.com/features/al-gores-new-movie-exposes-the-big-flaw-in-online-movie-ratings/)

_____no_output_____

from IPython.display import display, Image

url = 'https://fivethirtyeight.com/wp-content/uploads/2017/09/mehtahickey-inconvenient-0830-1.png'
example = Image(url=url, width=400)

display(example)_____no_output_____

Using this data: https://github.com/fivethirtyeight/data/tree/master/inconvenient-sequel_____no_output_____Links
- [Strong Titles Are The Biggest Bang for Your Buck](http://stephanieevergreen.com/strong-titles/)
- [Remove to improve (the data-ink ratio)](https://www.darkhorseanalytics.com/blog/data-looks-better-naked)
- [How to Generate FiveThirtyEight Graphs in Python](https://www.dataquest.io/blog/making-538-plots/)_____no_output_____## Make prototypes

This  helps us understand the problem_____no_output_____

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

plt.style.use('fivethirtyeight')

fake = pd.Series([38, 3, 2, 1, 2, 4, 6, 5, 5, 33],
                 index=range(1,11))

fake.plot.bar(color='C1', width=0.9);_____no_output_____fake2 = pd.Series(
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
     2, 2, 2,
     3, 3, 3,
     4, 4,
     5, 5, 5,
     6, 6, 6, 6,
     7, 7, 7, 7, 7,
     8, 8, 8, 8,
     9, 9, 9, 9,
     10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10])

fake2.value_counts().sort_index().plot.bar(color='C1', width=0.9);_____no_output_____

## Annotate with text_____no_output_____

import matplotlib.pyplot as plt

plt.style.use('fivethirtyeight')
fig, ax = plt.subplots(facecolor='white')

fake.plot.bar(color='C1', width=0.9);
plt.text(x=-1.5, y=45, fontsize=16, fontweight='bold', s='\'An Inconvenient Sequel: Truth To Power\' is divisive')
plt.text(x=-1.5, y=42, fontsize=12, s='IMDb ratings for the film as of Aug. 29')
plt.yticks(range(0, 41, 10))
plt.xlabel('Rating', fontsize=10, fontweight='bold')
plt.ylabel('Percent of Total Votes', fontsize=10, fontweight='bold')_____no_output__________no_output_____

## Reproduce with real data_____no_output_____

df = pd.read_csv('https://raw.githubusercontent.com/fivethirtyeight/data/master/inconvenient-sequel/ratings.csv')
print(df.shape)
df.head()(80053, 27)
df.category.value_counts()_____no_output_____df.dtypes_____no_output_____df['timestamp'] = pd.to_datetime(df['timestamp'])
df.timestamp.describe()_____no_output_____df_imdb = df[df.category == 'IMDb users']
df_imdb.shape_____no_output_____final = df_imdb.tail(1)
final_____no_output_____columns = ['{}_pct'.format(i) for i in range(1, 11)]
columns = [f'{i}_pct' for i in range(1, 11)]
columns_____no_output_____data = final[columns]
data = data.T
data.index = range(1, 11)
data_____no_output_____plt.style.use('fivethirtyeight')

data.plot.bar(color='C1', width=0.9, legend=False);
plt.text(x=-1.5, y=45, fontsize=16, fontweight='bold', s='\'An Inconvenient Sequel: Truth To Power\' is divisive');
plt.text(x=-1.5, y=42, fontsize=12, s='IMDb ratings for the film as of Aug. 29');
plt.yticks(range(0, 41, 10));
plt.xlabel('Rating', fontsize=10, fontweight='bold');
plt.ylabel('Percent of Total Votes', fontsize=10, fontweight='bold');_____no_output__________no_output_____

# ASSIGNMENT

Replicate the lesson code. I recommend that you [do not copy-paste](https://docs.google.com/document/d/1ubOw9B3Hfip27hF2ZFnW3a3z9xAgrUDRReOEo-FHCVs/edit).

# STRETCH OPTIONS

#### 1) Reproduce another example from [FiveThityEight's shared data repository](https://data.fivethirtyeight.com/).

#### 2) Reproduce one of the following using a library other than Seaborn or Matplotlib.

For example:
- [thanksgiving-2015](https://fivethirtyeight.com/features/heres-what-your-part-of-america-eats-on-thanksgiving/) (try the [`altair`](https://altair-viz.github.io/gallery/index.html#maps) library)
- [candy-power-ranking](https://fivethirtyeight.com/features/the-ultimate-halloween-candy-power-ranking/) (try the [`statsmodels`](https://www.statsmodels.org/stable/index.html) library)
- or another example of your choice!

#### 3) Make more charts!

Choose a chart you want to make, from [Visual Vocabulary - Vega Edition](http://ft.com/vocabulary).

Find the chart in an example gallery of a Python data visualization library:
- [Seaborn](http://seaborn.pydata.org/examples/index.html)
- [Altair](https://altair-viz.github.io/gallery/index.html)
- [Matplotlib](https://matplotlib.org/gallery.html)
- [Pandas](https://pandas.pydata.org/pandas-docs/stable/visualization.html)

Reproduce the chart. [Optionally, try the "Ben Franklin Method."](https://docs.google.com/document/d/1ubOw9B3Hfip27hF2ZFnW3a3z9xAgrUDRReOEo-FHCVs/edit) If you want, experiment and make changes.

Take notes. Consider sharing your work with your cohort!

_____no_output_____

import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline

df2 = pd.read_csv('https://raw.githubusercontent.com/fivethirtyeight/data/master/inconvenient-sequel/ratings.csv')
print(df2.shape)
df2.head()(80053, 27)
df2['timestamp'] = pd.to_datetime(df2['timestamp'])
df2.timestamp.describe()_____no_output_____df2_imdb = df2[df2.category == 'IMDb users']
df2_imdb.shape_____no_output_____final2 = df2_imdb.tail(1)
final2_____no_output_____columns = [f'{i}_pct' for i in range(1, 11)]
columns_____no_output_____data2 = final2[columns]
data2 = data2.T
data2.index = range(1, 11)
data2_____no_output_____plt.style.use('fivethirtyeight')

data2.plot.bar(color='C1', width=0.9, legend=False);
plt.text(x=-1.5, y=45, fontsize=16, fontweight='bold', s='\'An Inconvenient Sequel: Truth To Power\' is divisive');
plt.text(x=-1.5, y=42, fontsize=12, s='IMDb ratings for the film as of Aug. 29');
plt.yticks(range(0, 41, 10));
plt.xlabel('Rating', fontsize=10, fontweight='bold');
plt.ylabel('Percent of Total Votes', fontsize=10, fontweight='bold');_____no_output__________no_output_____


# Notebook from datadonK23/pyData-Projects
Path: KC_RecSys/project/notebook/Petdata_generator.ipynb

# Generate Pet Dataset_____no_output_____### Imports_____no_output_____

import numpy as np
import pandas as pd
from faker import Faker
import os_____no_output_____

## Pet dataset

Assumes...

* Discrete uniform distribution of ratings per user
* Each user rated more than 1/3 of documents
_____no_output_____### Presets_____no_output_____

fake = Faker()
fake.seed(23)
np.random.seed(23)

num_users = 100
num_docs = 1000_____no_output_____

### Generate fake Ratings_____no_output_____

def generate_ratings(num_users, num_docs, p_na_min, p_na_max, kind="5Star"):
    """ Generate random user ratings
    :param num_users:     number of users to generate
    :param num_docs:      number of documents to rate
    :param p_na_min:      min percentage of NaN per user
    :param p_na_max:      max percentage of NaN per user
    :param kind:          kind of rating scheme, either "5Star" (1 Star ... 5 Stars) or "binary" (like/dislike)
    :return user_ratings: dictionary of users with list of ratings
    """
    doc_uris = []
    user_ratings = {}

    # Generate fake URIs
    for _ in range(num_docs):
        doc_uris.append(fake.uri())
    user_ratings["doc_uri"] = doc_uris

    # Generate ratings
    for _ in range(num_users):
        if kind == "5Star":
            # TBD
            ratings = np.random.randint(0, 6, size=num_docs).tolist() # discr uniform ratings
            num_na = np.random.randint(int(num_docs * p_na_min), int(num_docs * p_na_max) + 1)
            random_ixs = np.random.choice(range(num_docs), size=num_na, replace=False) # mask
            for i in random_ixs:
                ratings[i] = np.NaN

        elif kind == "binary":
            ratings = np.random.choice([1, -1], num_docs, p=[.8, .2]).tolist()
            # TBD
            num_na = np.random.randint(int(num_docs * p_na_min), int(num_docs * p_na_max) + 1)
            random_ixs = np.random.choice(range(num_docs), size=num_na, replace=False) # mask
            for i in random_ixs:
                ratings[i] = np.NaN
        else:
            NotImplementedError

        # Generate fake user
        user_ratings[fake.name()] = ratings

    return user_ratings

user_ratings = generate_ratings(num_users, num_docs, 0.7, 0.95, "binary")_____no_output_____

### Dataframe_____no_output_____

df = pd.DataFrame.from_dict(user_ratings).set_index("doc_uri")
df.head()_____no_output_____

## Persist dataset_____no_output_____

#f_name = "petdata_1000_100.csv" # 5 Star ratings
f_name = "petdata_binary_1000_100.csv" # binary ratings
path = os.path.join("../data", f_name)

df.to_csv(path)_____no_output_____


# Notebook from evanbiederstedt/paga
Path: blood/simulated/simulated.ipynb

*First compiled: June 19, 2017. Updated June 8, 2018.*_____no_output_____# PAGA for simulated myeloid progenitor development_____no_output_____

import numpy as np
from matplotlib import rcParams
import matplotlib.pyplot as pl
import scanpy.api as sc

sc.settings.verbosity = 3  # verbosity: errors (0), warnings (1), info (2), hints (3)
sc.settings.set_figure_params(dpi=80, color_map='viridis')  # low dpi (dots per inch) yields small inline figures
sc.logging.print_versions()
results_file = './write/krumsiek11_blobs.h5ad'scanpy==1.2.0 anndata==0.6.4+6.gd9727ca numpy==1.13.1 scipy==1.0.0 pandas==0.22.0 scikit-learn==0.19.1 statsmodels==0.9.0 python-igraph==0.7.1 louvain==0.6.1

The simulated data that we'll use here describes development from a progenitor cell to four differentiated cell types: monocyte, erythrocyte, megakaryocyte and neutrophil, and only uses the expression levels of a few curated marker genes. See [Krumsiek *et al.* (2009)](https://doi.org/10.1371/journal.pone.0022649) for the definition of the boolean literature-curated network underlying the simulation and discussion of marker genes. The data has been generated using `sc.tl.sim('krumsiek11')`; see [here](https://github.com/theislab/scanpy_usage/tree/master/170430_krumsiek11) for more details. We know the ground truth, which is hard to establish for real data.

To add disconnected structure to the continuous differentiation manifold, which is almost always present in real data,  we add a "noise" clusters from a Gaussian mixture model. In real data this might be due to imperfect sampling or just a very heterogeneous tissue. The clusters are disconnected in the sense that they are not part of the differentiation  trajectories. One of the strengths of PAGA is that it is able to identify them as not being part of the ground truth trajectory and leave them out of the proposed structure._____no_output_____We begin by importing our data, and then add the extra "noise" clusters to it (hence the name `krumsiek11_blobs` that will show up from now on). When working on your own data, prepare the count matrix with genes as the columns and cells as the rows, with their names in the first row/column respectively, and import them using `sc.read()`. You can also call `adata.transpose()` if your files follow a different convention. The parser is robust, and can handle files with commas, tabs or spaces as delimiters. After importing the count matrix, you can add cell annotations to the resulting structure, the information for which you can prepare in an external file with the first column as cell names. You can take a look at [nestorowa16](https://github.com/theislab/graph_abstraction/blob/master/nestorowa16/nestorowa16.ipynb) for exact syntax and a download of some appropriately formatted input files. Scanpy can also directly use Cell Ranger count matrices on input, see [the Seurat-like tutorial](https://github.com/theislab/scanpy_usage/blob/master/170505_seurat/seurat.ipynb) for details._____no_output_____The actual data provided on input can be either raw counts or log-transformed counts. The count matrix will require some basic preprocessing prior to commencing the analysis, with raw count data processed, for instance, with a simple call of `sc.pp.recipe_zheng17()`. See the Seurat tutorial for many more options preprocessing._____no_output_____

adata_krumsiek11 = sc.datasets.krumsiek11()
adata_krumsiek11.obs_names_make_unique()Observation names are not unique. To make them unique, call `.obs_names_make_unique`.
... storing 'cell_type' as categorical
adata_blobs = sc.datasets.blobs(cluster_std=0.5, n_centers=2)
adata_blobs.var_names = adata_krumsiek11.var_names
adata = adata_krumsiek11.concatenate(adata_blobs, index_unique='-')
adata.uns = adata_krumsiek11.uns
adata.write(results_file)/Users/alexwolf/_hholtz/01_projects/1512_scanpy/anndata/anndata/readwrite/write.py:112: UserWarning: dict key 0 transformed to str upon writing to h5,using string keys is recommended
  .format(k))
/Users/alexwolf/_hholtz/01_projects/1512_scanpy/anndata/anndata/readwrite/write.py:112: UserWarning: dict key 159 transformed to str upon writing to h5,using string keys is recommended
  .format(k))
/Users/alexwolf/_hholtz/01_projects/1512_scanpy/anndata/anndata/readwrite/write.py:112: UserWarning: dict key 319 transformed to str upon writing to h5,using string keys is recommended
  .format(k))
/Users/alexwolf/_hholtz/01_projects/1512_scanpy/anndata/anndata/readwrite/write.py:112: UserWarning: dict key 459 transformed to str upon writing to h5,using string keys is recommended
  .format(k))
/Users/alexwolf/_hholtz/01_projects/1512_scanpy/anndata/anndata/readwrite/write.py:112: UserWarning: dict key 619 transformed to str upon writing to h5,using string keys is recommended
  .format(k))

Since we know the underlying ground truth, we can visualize the simulated data as heatmaps. The darker the heatmap the higher the expression. This is not immediately applicable to experimental data you are likely to have on input, but a heatmap of the trajectories we infer created later on in the notebook may be of more relevance._____no_output_____

_, axs = pl.subplots(ncols=4, figsize=(10, 4), gridspec_kw={'wspace': 0.05, 'left': 0.1})
pl.subplots_adjust(left=0.05, right=0.98, top=0.88)
for ititle, (title, idcs) in enumerate(
    [('erythrocyte', (160, 320)), ('neutrophil', (480, 640)),
     ('monocyte', (0, 160)), ('megakaryocyte', (320, 480)), ]):
    pl.sca(axs[ititle])
    img = pl.imshow(adata.X[idcs[0]:idcs[1]].T, aspect='auto', interpolation='nearest', cmap='Greys')
    if ititle == 0: pl.yticks(range(adata.n_vars), adata.var_names)
    else: pl.yticks([])
    pl.title('path to\n{} fate'.format(title))
    pl.xticks([])
    pl.grid(False)
    pl.xlabel('time')
    axs[ititle].set_frame_on(False)
pl.savefig('./figures/krumsiek11_timeseries_heatmap.pdf')
pl.show()_____no_output_____

### Visualization_____no_output_____Now that we have our data, we can visualise it. PAGA's preferred visualisation is the Fruchterman & Reingold algorithm (obtained via `sc.tl.draw_graph()` below), but Scanpy has many other options for visualizing the data. We precompute the visualisations to begin with, and then add this extra information to our constantly expanding `.h5ad` result archive (that's the `sc.read()` and `sc.write()` calls bookending the code blocks - storing our analysis results for easy later access every step of the way)._____no_output_____

sc.pp.neighbors(adata, n_neighbors=30)  # generate a neighborhood graph of single cells
sc.tl.draw_graph(adata)  # draw this graph using standard drawing algorithmscomputing neighbors
    using data matrix X directly
    finished (0:00:02.53) --> added to `.uns['neighbors']`
    'distances', weighted adjacency matrix
    'connectivities', weighted adjacency matrix
drawing single-cell graph using layout "fa"
    finished (0:00:05.39) --> added
    'X_draw_graph_fa', graph_drawing coordinates (adata.obsm)

Now that the visualisations are precomputed, we can produce and save the corresponding plots. It should be noted that PAGA saves its output files in predetermined default locations, you can append to the standard naming schemes if you change `save=True` to whatever you want to append to the name. So, for example, if you wanted the tSNE plot to be `figures/tsne1.png` instead of the default `figures/tsne.png`, you'd call the tSNE plotter function with `save='1'`. You can change the default directory by changing `sc.settings.figdir` to any string that should be prepended to your path. You can also save the figures by right-clicking in your firefox or other browser and selecting "Save Image As"._____no_output_____

sc.pl.draw_graph(adata)... storing 'blobs' as categorical
... storing 'cell_type' as categorical
adata.write(results_file)/Users/alexwolf/_hholtz/01_projects/1512_scanpy/anndata/anndata/readwrite/write.py:112: UserWarning: dict key 0 transformed to str upon writing to h5,using string keys is recommended
  .format(k))
/Users/alexwolf/_hholtz/01_projects/1512_scanpy/anndata/anndata/readwrite/write.py:112: UserWarning: dict key 159 transformed to str upon writing to h5,using string keys is recommended
  .format(k))
/Users/alexwolf/_hholtz/01_projects/1512_scanpy/anndata/anndata/readwrite/write.py:112: UserWarning: dict key 319 transformed to str upon writing to h5,using string keys is recommended
  .format(k))
/Users/alexwolf/_hholtz/01_projects/1512_scanpy/anndata/anndata/readwrite/write.py:112: UserWarning: dict key 459 transformed to str upon writing to h5,using string keys is recommended
  .format(k))
/Users/alexwolf/_hholtz/01_projects/1512_scanpy/anndata/anndata/readwrite/write.py:112: UserWarning: dict key 619 transformed to str upon writing to h5,using string keys is recommended
  .format(k))

## Clustering_____no_output_____PAGA's follows proceeds in two steps: cluster the single-cell graph and then construct the abstracted graph based on the connectivity between the clusters. The resolution parameter controls how coarse the clustering is - the higher the value, the more clusters you'll get and the smaller they'll be. It's not a bad idea to run the following few bits of code with a couple of different resolutions, trying to find a visually convincing parititon._____no_output_____

adata = sc.read(results_file)_____no_output_____sc.tl.louvain(adata, resolution=2)running Louvain clustering
    using the "louvain" package of Traag (2017)
    finished (0:00:00.17) --> found 10 clusters and added
    'louvain', the cluster labels (adata.obs, categorical)
sc.pl.draw_graph(adata, color='louvain', legend_loc='on data')_____no_output_____

Let us better resolve the first branching region around cluster 3._____no_output_____

sc.tl.louvain(adata, resolution=1, restrict_to=('louvain', ['4']))running Louvain clustering
    using the "louvain" package of Traag (2017)
    finished (0:00:00.01) --> found 3 clusters and added
    'louvain_R', the cluster labels (adata.obs, categorical)

Use the reclustered louvain groups to initialize our clustering, give them simple names._____no_output_____

adata.obs['clusters'] = adata.obs['louvain_R']
adata.rename_categories('clusters', [str(i) for i in range(len(adata.obs['clusters'].cat.categories))])_____no_output_____sc.pl.draw_graph(adata, color='clusters', legend_loc='on data')_____no_output_____

Annotate the clusters._____no_output_____

adata.obs['clusters'].cat.categories_____no_output_____new_categories = ['0', '1', '2', '3/Neu', '4', '5/Stem', '6', '7/Mo', '8/Ery', '9/Mk', '10', '11']
adata.rename_categories('clusters', new_categories)
del adata.uns['highlights']  # remove the annotation of the single datapoints_____no_output_____sc.pl.draw_graph(adata, color='clusters', legend_loc='on data')_____no_output_____sc.tl.paga(adata, groups='clusters')running partition-based graph abstraction (PAGA)
    finished (0:00:00.20) --> added
    'paga/connectivities', connectivities adjacency (adata.uns)
    'paga/connectivities_tree', connectivities subtree (adata.uns)

There are a number of different ways to visualise the resulting abstracted graph, with the most power going to the layout parameter of the function call. The default is the Fruchterman & Reingold algorithm PAGA in (`layout='fr'`), with the most common alternative being Reingold-Tilford (`layout='rt'`), which creates a hierarchical structure based on where you specify the tree's roots to be.

In the present case, the result is tree-like, so we use Reingold-Tilford. Seeing how clustering results may vary slightly between machines, ensure that the `root` parameter in the call below is pointed at the cluster that has the progenitor cells in it, plus two clusters representative of the noise blobs! If you don't provide a root parameter value but ask for the `'rt'` layout, Scanpy will make an educated guess for you based on the graph topology._____no_output_____

sc.pl.paga(adata)--> added 'pos', the PAGA positions (adata.uns['paga'])
sc.pl.paga(adata, layout='rt', root=[5, 0, 1])--> added 'pos', the PAGA positions (adata.uns['paga'])

We can use this layout to generate a single-cell embedding that reflects the same global topology._____no_output_____

sc.tl.draw_graph(adata, init_pos='paga', layout='fr', maxiter=50)drawing single-cell graph using layout "fr"
    finished (0:00:01.50) --> added
    'X_draw_graph_fr', graph_drawing coordinates (adata.obsm)
X = adata.obsm['X_draw_graph_fr'].copy()_____no_output_____adata.obsm['X_draw_graph_fr'] = X.copy()
# adata.obsm['X_draw_graph_fr'][adata.obs['clusters'] == '0', 1] -= 400
adata.obsm['X_draw_graph_fr'][adata.obs['clusters'] == '0', 0] += 500
# adata.obsm['X_draw_graph_fr'][adata.obs['clusters'] == '1', 1] -= 1000
adata.obsm['X_draw_graph_fr'][adata.obs['clusters'] == '1', 0] -= 500_____no_output_____sc.pl.draw_graph(adata, color='clusters', edges=True, size=20, legend_loc='on data')_____no_output_____

Harmonize the colors._____no_output_____

pl.figure(figsize=(8, 2))
for i in range(28):
    pl.scatter(i, 1, c=sc.pl.palettes.zeileis_26[i], s=200)
pl.show()_____no_output_____orig_colors = np.array(sc.pl.palettes.zeileis_26)
new_colors = orig_colors.copy()_____no_output_____new_colors[[4, 6]] = orig_colors[[12, 13]]  # Stem colors / green
new_colors[[8, 10]] = orig_colors[[5, 10]]  # Ery colors / red
new_colors[[9]] = orig_colors[[17]]       # Mk early Ery colors / yellow
new_colors[[2, 3]] = orig_colors[[2, 25]]   # lymph progenitors / grey
new_colors[[3, 11]] = orig_colors[[6, 7]]  # Neu / light blue
new_colors[[7, 12]] = orig_colors[[0, 1]]  # Mo / dark blue_____no_output_____adata.uns['clusters_colors'] = new_colors_____no_output_____adata.obs['clusters'].cat.categories = [
    '0', '1', '2', '3/Neu\n\n', '4', '         5/Stem\n\n', '6', '7/Mo\n\n', '8/Ery\n\n', '         9/Mk\n\n',
    '10', '11']_____no_output_____axs = sc.pl.paga_compare(
    adata, title='', right_margin=0.2, size=20, edge_width_scale=0.3,
    legend_fontsize=12, fontsize=12, frameon=False, show=False)--> added 'pos', the PAGA positions (adata.uns['paga'])
pl.sca(axs[0])
pl.xlabel('')
pl.ylabel('')
axs[0].set_frame_on(False)
pl.savefig('./figures/paga_compare_simulated.pdf')
pl.show()_____no_output_____adata.write(results_file)_____no_output_____

### Gene changes along paths - "pseudotime series"_____no_output_____You can also plot the distance along the manifold (pseudotime) - an extension of DPT [Haghverdi *et al.* (2016)](https://doi.org/10.1038/nmeth.3971). To do this, you need to ensure that your object has information on the position of the root stored within it before calling the Louvain clustering, as previously mentioned. Otherwise, the clustering won't know where to anchor the pseudotime and will make no attempt to compute it. You don't necessarily have to pick this value out by hand though, for example [nestorowa16](https://github.com/theislab/graph_abstraction/blob/master/nestorowa16/nestorowa16.ipynb) uses the cell type annotation that was imported along with the data to set the root to the index of the first cell tagged as Stem:

    adata.uns['iroot'] = np.flatnonzero(adata.obs['cell_types'] == 'Stem')[0]_____no_output_____

adata = sc.read(results_file)_____no_output_____sc.tl.dpt(adata)
sc.pl.draw_graph(adata, color='dpt_pseudotime')WARNING: Trying to run `tl.dpt` without prior call of `tl.diffmap`. Falling back to `tl.diffmap` with default parameters.
computing Diffusion Maps using n_comps=15(=n_dcs)
    eigenvalues of transition matrix
    [ 1.            1.            0.9994561076  0.9990157485  0.9974681735
      0.9971336126  0.9936080575  0.9897011518  0.9837759137  0.9828007817
      0.9784398079  0.9745985866  0.9677239656  0.9599461555  0.9582870007]
    finished (0:00:00.07) --> added
    'X_diffmap', diffmap coordinates (adata.obsm)
    'diffmap_evals', eigenvalues of transition matrix (adata.uns)
computing Diffusion Pseudotime using n_dcs=10
    finished (0:00:00.00) --> added
    'dpt_pseudotime', the pseudotime (adata.obs)

Now that we've inferred the abstracted graph and a pseudotime, we can visualise the gene expression changes along proposed paths. Notice how when specifying the `paths`, we start at the progenitor root (in the case of the Louvain clustering shown in the notebook, cluster 5) and follow the tree down until we reach the cluster tagged with the corresponding cell type in the Fruchterman-Reingold visualisation. If you wished to use this on your own experimental data, you'd probably need to identify some key marker genes using either prior knowledge or [`sc.tl.rank_gene_groups()`](https://scanpy.readthedocs.io/en/latest/api/scanpy.api.tl.rank_genes_groups.html) first - don't forget, this is simulated data with only the handful of known crucial genes present, you'd need to reduce the gene space of a typical single cell dataset to make the heatmap work. If you end up following the `sc.tl.rank_gene_groups()` path, the Louvain clustering can be accessed by passing `rank_by='PAGA_groups'` into the function call. Adding a dedicated method for identifying markers along trajectories in the abstracted graph is a forthcoming functionality expansion, with a number of potential approaches being evaluated._____no_output_____

adata.obs['distance'] = adata.obs['dpt_pseudotime']_____no_output_____paths = [('erythrocytes', [5, 6, 4, 2, 10, 8]),
         ('neutrophils', [5, 6, 11, 3]),
         ('monocytes', [5, 6, 4, 2, 7])]_____no_output_____adata.var_names_____no_output_____

Simply reorder the gene names to match those of the other datasets._____no_output_____

gene_names = ['Gata2', 'Gata1', 'SCL', 'Fog1', 'EKLF', 'Fli1', 'Cebpa', 'Gfi1', 'Pu.1', 'cJun', 'EgrNab']_____no_output______, axs = pl.subplots(ncols=3, figsize=(6, 2.5), gridspec_kw={'wspace': 0.05, 'left': 0.12})
pl.subplots_adjust(left=0.05, right=0.98, top=0.82, bottom=0.2)
for ipath, (descr, path) in enumerate(paths):
    _, data = sc.pl.paga_path(
        adata, path, gene_names,
        show_node_names=False,
        ax=axs[ipath],
        ytick_fontsize=12,
        left_margin=0.15,
        n_avg=50,
        annotations=['distance'],
        show_yticks=True if ipath==0 else False,
        show_colorbar=False,
        color_map='Greys',
        color_maps_annotations={'distance': 'viridis'},
        title='{} path'.format(descr),
        return_data=True,
        show=False)
    data.to_csv('./write/paga_path_{}.csv'.format(descr))
pl.savefig('./figures/paga_path_simulated.pdf')
pl.show()_____no_output_____


# Notebook from coursemdetw/reveal2
Path: content/notebook/Elements of Evolutionary Algorithms.ipynb

_____no_output_____## Demostration Class 02
# Elements of Evolutionary Algorithms_____no_output_____### Luis Martí, LIRA/[DEE](http://www.ele.puc-rio.br)/[PUC-Rio](http://www.puc-rio.br)

[http://lmarti.com](http://lmarti.com); [lmarti@ele.puc-rio.br](mailto:lmarti@ele.puc-rio.br)

[Advanced Evolutionary Computation: Theory and Practice](http://lmarti.com/aec-2014) _____no_output_____The notebook is better viewed rendered as slides. You can convert it to slides and view them by:
- using [nbconvert](http://ipython.org/ipython-doc/1/interactive/nbconvert.html) with a command like:
  ```bash
  $ ipython nbconvert --to slides --post serve <this-notebook-name.ipynb>
  ```
- installing [Reveal.js - Jupyter/IPython Slideshow Extension](https://github.com/damianavila/live_reveal)
- using the online [IPython notebook slide viewer](https://slideviewer.herokuapp.com/) (some slides of the notebook might not be properly rendered).

This and other related IPython notebooks can be found at the course github repository:
* [https://github.com/lmarti/evolutionary-computation-course](https://github.com/lmarti/evolutionary-computation-course)_____no_output_____## In this demonstration class we will deal with the features and problems shared by most evolutionary algorithms.

*Note*: Most of the material used in this notebook comes from [DEAP](https://github.com/DEAP/deap) documentation._____no_output_____## Elements to take into account using evolutionary algorithms

* **Individual representation** (binary, Gray, floating-point, etc.);
* **evaluation** and **fitness assignment**;
* **mating selection**, that establishes a partial order of individuals in the population using their fitness function value as reference and determines the degree at which individuals in the population will take part in the generation of new (offspring) individuals.
* **variation**, that applies a range of evolution-inspired operators, like crossover, mutation, etc., to synthesize offspring individuals from the current (parent) population. This process is supposed to prime the fittest individuals so they play a bigger role in the generation of the offspring.
* **environmental selection**, that merges the parent and offspring individuals to produce the population that will be used in the next iteration. This process often involves the deletion of some individuals using a given criterion in order to keep the amount of individuals bellow a certain threshold.
* **stopping criterion**, that determines when the algorithm shoulod be stopped, either because the optimum was reach or because the optimization process is not progressing._____no_output_____# Hence a 'general' evolutionary algorithm can be described as_____no_output_____```
def evolutionary_algorithm():
    'Pseudocode of an evolutionary algorithm'
    populations = [] # a list with all the populations

    populations[0] =  initialize_population(pop_size)
    t = 0

    while not stop_criterion(populations[t]):
        fitnesses = evaluate(populations[t])
        offspring = matting_and_variation(populations[t],
                                          fitnesses)
        populations[t+1] = environmental_selection(
                                          populations[t],
                                          offspring)
        t = t+1
```_____no_output_____# Python libraries for evolutionary computation

* PaGMO/PyGMO
* Inspyred
* **Distributed Evolutionary Algorithms in Python (DEAP)**

> There are potentially many more, feel free to give me some feedback on this._____no_output_____

 Open source Python library with,

 genetic algorithm using any representation;

 evolutionary strategies (including CMA-ES);

 multi-objective optimization from the start;

 co-evolution (cooperative and competitive) of multiple populations;

 parallelization of the evaluations (and more) using SCOOP;

 statistics keeping, and;

 benchmarks module containing some common test functions.

 [https://github.com/DEAP/deap](https://github.com/DEAP/deap)

_____no_output_____# Lets start with an example and analyze it_____no_output_____## The One Max problem

* Maximize the number of ones in a binary string (list, vector, etc.).
* More formally, from the set of binary strings of length $n$,
$$\mathcal{S}=\left\{s_1,\ldots,s_n\right\}, \text{ with } s_i=\left\{0,1\right\}.$$
* Find $s^\ast\in\mathcal{S}$ such that
$$s^\ast = \operatorname*{arg\,max}_{s\in\mathcal{S}} \sum_{i=1}^{n}{s_i}.$$
* Its clear that the optimum is an *all-ones* string._____no_output_____# Coding the problem_____no_output_____

import random
from deap import algorithms, base, creator, tools_____no_output_____creator.create("FitnessMax", base.Fitness, weights=(1.0,))
creator.create("Individual", list, fitness=creator.FitnessMax)_____no_output_____def evalOneMax(individual):
    return (sum(individual),)_____no_output_____

# Defining the elements _____no_output_____

toolbox = base.Toolbox()_____no_output_____toolbox.register("attr_bool", random.randint, 0, 1)
toolbox.register("individual", tools.initRepeat, creator.Individual,
                 toolbox.attr_bool, n=100)
toolbox.register("population", tools.initRepeat, list,
                 toolbox.individual)_____no_output_____toolbox.register("evaluate", evalOneMax)
toolbox.register("mate", tools.cxTwoPoint)
toolbox.register("mutate", tools.mutFlipBit, indpb=0.05)
toolbox.register("select", tools.selTournament, tournsize=3)_____no_output_____

# Running the experiment_____no_output_____

pop = toolbox.population(n=300)_____no_output_____

Lets run only 10 generations_____no_output_____

result = algorithms.eaSimple(pop, toolbox, cxpb=0.5, mutpb=0.2,
                             ngen=10, verbose=False)_____no_output_____print('Current best fitness:', evalOneMax(tools.selBest(pop, k=1)[0]))Current best fitness: (81,)
result = algorithms.eaSimple(pop, toolbox, cxpb=0.5, mutpb=0.2,
                             ngen=50, verbose=False)_____no_output_____print('Current best fitness:', evalOneMax(tools.selBest(pop, k=1)[0]))Current best fitness: (100,)

# Essential features

* `deap.creator`: meta-factory allowing to create classes that will fulfill the needs of your evolutionary algorithms.
* `deap.base.Toolbox`: A toolbox for evolution that contains the evolutionary operators. You may populate the toolbox with any other function by using the `register()` method
* `deap.base.Fitness([values])`: The fitness is a measure of quality of a solution. If values are provided as a tuple, the fitness is initalized using those values, otherwise it is empty (or invalid). You should inherit from this class to define your custom fitnesses._____no_output_____# Defining an individual
First import the required modules and register the different functions required to create individuals that are a list of floats with a minimizing two objectives fitness._____no_output_____

import random

from deap import base
from deap import creator
from deap import tools_____no_output_____IND_SIZE = 5_____no_output_____creator.create("FitnessMin", base.Fitness, weights=(-1.0, -1.0))
creator.create("Individual", list, fitness=creator.FitnessMin)

toolbox1 = base.Toolbox()
toolbox1.register("attr_float", random.random)
toolbox1.register("individual", tools.initRepeat, creator.Individual,
                 toolbox1.attr_float, n=IND_SIZE)_____no_output_____

The first individual can now be built_____no_output_____

ind1 = toolbox1.individual()_____no_output_____

Printing the individual ind1 and checking if its fitness is valid will give something like this_____no_output_____

print ind1 _____no_output_____print ind1.fitness.valid_____no_output_____

The individual is printed as its base class representation (here a list) and the fitness is invalid because it contains no values._____no_output_____# Evaluation
The evaluation is the most "personal" part of an evolutionary algorithm
* it is the only part of the library that you must write yourself.
* A typical evaluation function takes one individual as argument and return its fitness as a tuple.
* A fitness is a list of floating point values and has a property valid to know if this individual shall be re-evaluated.
* The fitness is set by setting the values to the associated tuple._____no_output_____For example, the following evaluates the previously created individual ind1 and assign its fitness to the corresponding values._____no_output_____

def evaluate(individual):
    # Do some hard computing on the individual
    a = sum(individual)
    b = len(individual)
    return a, 1. / b_____no_output_____ind1.fitness.values = evaluate(ind1)_____no_output_____print ind1.fitness.valid_____no_output_____print ind1.fitness_____no_output_____

Dealing with single objective fitness is not different, the evaluation function must return a tuple because single-objective is treated as a special case of multi-objective._____no_output_____# Mutation
* The next kind of operator that we will present is the mutation operator.
* There is a variety of mutation operators in the deap.tools module.
* Each mutation has its own characteristics and may be applied to different type of individual.
* Be careful to read the documentation of the selected operator in order to avoid undesirable behaviour.

The general rule for mutation operators is that they only mutate, this means that an independent copy must be made prior to mutating the individual if the original individual has to be kept or is a reference to an other individual (see the selection operator)._____no_output_____In order to apply a mutation (here a gaussian mutation) on the individual ind1, simply apply the desired function._____no_output_____

mutant = toolbox1.clone(ind1)
ind2, = tools.mutGaussian(mutant, mu=0.0, sigma=0.2, indpb=0.2)
del mutant.fitness.values_____no_output_____

The fitness’ values are deleted because they not related to the individual anymore. As stated above, the mutation does mutate and only mutate an individual it is not responsible of invalidating the fitness nor anything else. The following shows that ind2 and mutant are in fact the same individual._____no_output_____

print ind2 is mutant_____no_output_____print mutant is ind2_____no_output_____

# Crossover

* There is a variety of crossover operators in the `deap.tools module`.
* Each crossover has its own characteristics and may be applied to different type of individuals.
* Be careful to read the documentation of the selected operator in order to avoid undesirable behaviour.

The general rule for crossover operators is that they only mate individuals, this means that an independent copies must be made prior to mating the individuals if the original individuals have to be kept or is are references to other individuals (see the selection operator)._____no_output_____Lets apply a crossover operation to produce the two children that are cloned beforehand._____no_output_____

child1, child2 = [toolbox1.clone(ind) for ind in (ind1, ind2)]
tools.cxBlend(child1, child2, 0.5)
del child1.fitness.values
del child2.fitness.values_____no_output_____

# Selection
* Selection is made among a population by the selection operators that are available in the deap.operators module.
* The selection operator usually takes as first argument an iterable container of individuals and the number of individuals to select. It returns a list containing the references to the selected individuals.

The selection is made as follow._____no_output_____

selected = tools.selBest([child1, child2], 2)
print child1 in selected_____no_output_____

# Using the Toolbox
* The toolbox is intended to contain all the evolutionary tools, from the object initializers to the evaluation operator.
* It allows easy configuration of each algorithms.
* The toolbox has basically two methods, `register()` and `unregister()`, that are used to add or remove tools from the toolbox.
* The usual names for the evolutionary tools are mate(), mutate(), evaluate() and select(), however, any name can be registered as long as it is unique. Here is how they are registered in the toolbox._____no_output_____

from deap import base
from deap import tools

toolbox1 = base.Toolbox()

def evaluateInd(individual):
    # Do some computation
    return result,

toolbox1.register("mate", tools.cxTwoPoint)
toolbox1.register("mutate", tools.mutGaussian, mu=0, sigma=1, indpb=0.2)
toolbox1.register("select", tools.selTournament, tournsize=3)
toolbox1.register("evaluate", evaluateInd)_____no_output_____

# Tool Decoration
* A powerful feature that helps to control very precise thing during an evolution without changing anything in the algorithm or operators.
* A decorator is a wrapper that is called instead of a function.
* It is asked to make some initialization and termination work before and after the actual function is called.

For example, in the case of a constrained domain, one can apply a decorator to the mutation and crossover in order to keep any individual from being out-of-bound. _____no_output_____The following defines a decorator that checks if any attribute in the list is out-of-bound and clips it if it is the case.
* The decorator is defined using three functions in order to receive the min and max arguments.
* Whenever the mutation or crossover is called, bounds will be check on the resulting individuals._____no_output_____

def checkBounds(min, max):
    def decorator(func):
        def wrapper(*args, **kargs):
            offspring = func(*args, **kargs)
            for child in offspring:
                for i in xrange(len(child)):
                    if child[i] > max:
                        child[i] = max
                    elif child[i] < min:
                        child[i] = min
            return offspring
        return wrapper
    return decorator

toolbox.register("mate_example", tools.cxBlend, alpha=0.2)
toolbox.register("mutate_example", tools.mutGaussian, mu=0, sigma=2)

MIN = 0; MAX = 10

toolbox.decorate("mate_example", checkBounds(MIN, MAX))
toolbox.decorate("mutate_example", checkBounds(MIN, MAX))_____no_output_____

This will work on crossover and mutation because both return a tuple of individuals. The mutation is often considered to return a single individual but again like for the evaluation, the single individual case is a special case of the multiple individual case._____no_output_____# Variations

* Variations allows to build simple algorithms using predefined small building blocks.
* In order to use a variation, the toolbox must be set to contain the required operators.

> For example, in the lastly presented complete algorithm, the crossover and mutation are regrouped in the `varAnd()` function, this function requires the toolbox to contain the `mate()` and `mutate()` functions. The variations can be used to simplify the writing of an algorithm as follow.
_____no_output_____

from deap import algorithms

NGEN = 20 # number of generations
CXPB = 0.6
MUTPB = 0.05

for g in range(NGEN):
    # Select and clone the next generation individuals
    offspring = map(toolbox.clone, toolbox.select(pop, len(pop)))

    # Apply crossover and mutation on the offspring
    offspring = algorithms.varAnd(offspring, toolbox, CXPB, MUTPB)

    # Evaluate the individuals with an invalid fitness
    invalid_ind = [ind for ind in offspring if not ind.fitness.valid]
    fitnesses = toolbox.map(toolbox.evaluate, invalid_ind)
    for ind, fit in zip(invalid_ind, fitnesses):
        ind.fitness.values = fit

    # The population is entirely replaced by the offspring
    pop[:] = offspring_____no_output_____

# Algorithms
* There are several algorithms implemented in the algorithms module.
* They are very simple and reflect the basic types of evolutionary algorithms present in the literature.
* The algorithms use a Toolbox as defined in the last sections.
* In order to setup a toolbox for an algorithm, you must register the desired operators under a specified names, refer to the documentation of the selected algorithm for more details.
* Once the toolbox is ready, it is time to launch the algorithm._____no_output_____The *simple evolutionary algorithm* takes 5 arguments, a population, a toolbox, a probability of mating each individual at each generation (`cxpb`), a probability of mutating each individual at each generation (`mutpb`) and a number of generations to accomplish (`ngen`)._____no_output_____

from deap import algorithms

result = algorithms.eaSimple(pop, toolbox, cxpb=0.5, mutpb=0.2, ngen=50)_____no_output_____

# Computing Statistics
Often, one wants to compile statistics on what is going on in the optimization. The Statistics are able to compile such data on arbitrary attributes of any designated object. To do that, one need to register the desired statistic functions inside the stats object using the exact same syntax as the toolbox._____no_output_____

stats = tools.Statistics(key=lambda ind: ind.fitness.values)_____no_output_____

The statistics object is created using a key as first argument. This key must be supplied a function that will later be applied to the data on which the statistics are computed. The previous code sample uses the fitness.values attribute of each element._____no_output_____

import numpy
stats.register("avg", numpy.mean)
stats.register("std", numpy.std)
stats.register("min", numpy.min)
stats.register("max", numpy.max)_____no_output_____

* The statistical functions are now registered.
* The register function expects an alias as first argument and a function operating on vectors as second argument.
* Any subsequent argument is passed to the function when called. The creation of the statistics object is now complete._____no_output_____# Predefined Algorithms
When using a predefined algorithm such as `eaSimple()`, `eaMuPlusLambda()`, `eaMuCommaLambda()`, or `eaGenerateUpdate()`, the statistics object previously created can be given as argument to the algorithm._____no_output_____

pop, logbook = algorithms.eaSimple(pop, toolbox, cxpb=0.5, mutpb=0.2, ngen=0,
                                   stats=stats, verbose=True)_____no_output_____

* Statistics will automatically be computed on the population every generation.
* The verbose argument prints the statistics on screen while the optimization takes place.
* Once the algorithm returns, the final population and a Logbook are returned.
* See the next section or the Logbook documentation for more information._____no_output_____# Writing Your Own Algorithm

When writing your own algorithm, including statistics is very simple. One need only to compile the statistics on the desired object.

For example, compiling the statistics on a given population is done by calling the compile() method._____no_output_____

record = stats.compile(pop)_____no_output_____

The argument to the compile function must be an iterable of elements on which the key will be called. Here, our population (`pop`) contains individuals.

* The statistics object will call the key function on every individual to retrieve their `fitness.values` attribute.
* The resulting array of values is finally given the each statistic function and the result is put into the record dictionary under the key associated with the function.
* Printing the record reveals its nature.
_____no_output_____

>>> print(record)
{'std': 4.96, 'max': 63.0, 'avg': 50.2, 'min': 39.0}_____no_output_____

# Logging Data
Once the data is produced by the statistics, one can save it for further use in a Logbook.

* The logbook is intended to be a chronological sequence of entries (as dictionaries).
* It is directly compliant with the type of data returned by the statistics objects, but not limited to this data.
* *In fact, anything can be incorporated in an entry of the logbook.*_____no_output_____

logbook = tools.Logbook()
logbook.record(gen=0, evals=30, **record)_____no_output_____

The `record()` method takes a variable number of argument, each of which is a data to be recorded. In the last example, we saved the generation, the number of evaluations and everything contained in the record produced by a statistics object using the star magic. All record will be kept in the logbook until its destruction.

After a number of records, one may want to retrieve the information contained in the logbook._____no_output_____

gen, avg = logbook.select("gen", "avg")_____no_output_____

The `select()` method provides a way to retrieve all the information associated with a keyword in all records. This method takes a variable number of string arguments, which are the keywords used in the record or statistics object. Here, we retrieved the generation and the average fitness using a single call to select._____no_output_____# Printing to Screen
* A logbook can be printed to screen or file.
* Its `__str__()` method returns a header of each key inserted in the first record and the complete logbook for each of these keys.
* The row are in chronological order of insertion while the columns are in an undefined order.
* The easiest way to specify an order is to set the header attribute to a list of strings specifying the order of the columns._____no_output_____

logbook.header = "gen", "avg", "spam"_____no_output_____

The result is:_____no_output_____

print(logbook)_____no_output_____

# Plotting Features
* One of the most common operation when an optimization is finished is to plot the data during the evolution.
* The `Logbook` allows to do this very efficiently.
* Using the select method, one can retrieve the desired data and plot it using matplotlib._____no_output_____

gen = logbook.select("gen")
fit_mins = logbook.chapters["fitness"].select("min")
size_avgs = logbook.chapters["size"].select("avg")_____no_output_____import matplotlib.pyplot as plt
%matplotlib inline

fig, ax1 = plt.subplots()
line1 = ax1.plot(gen, fit_mins, "b-", label="Minimum Fitness")
ax1.set_xlabel("Generation")
ax1.set_ylabel("Fitness", color="b")
for tl in ax1.get_yticklabels():
    tl.set_color("b")

ax2 = ax1.twinx()
line2 = ax2.plot(gen, size_avgs, "r-", label="Average Size")
ax2.set_ylabel("Size", color="r")
for tl in ax2.get_yticklabels():
    tl.set_color("r")

lns = line1 + line2
labs = [l.get_label() for l in lns]
ax1.legend(lns, labs, loc="center right")

plt.show()_____no_output_____

_____no_output_____# Constraint Handling

We have already seen some alternatives.

* **Penality functions** are the most basic way of handling constrains for individuals that cannot be evaluated or are forbiden for problem specific reasons, when falling in a given region.
* The penality function gives a fitness disavantage to theses individuals based on the amount of constraint violation in the solution._____no_output__________no_output_____In DEAP, a penality function can be added to any evaluation function using the DeltaPenality decorator provided in the tools module._____no_output_____

from math import sin
from deap import base
from deap import tools

def evalFct(individual):
    """Evaluation function for the individual."""
    x = individual[0]
    return (x - 5)**2 * sin(x) * (x/3),

def feasible(individual):
    """Feasability function for the individual. Returns True if feasible False
    otherwise."""
    if 3 < individual[0] < 5:
        return True
    return False

def distance(individual):
    """A distance function to the feasability region."""
    return (individual[0] - 5.0)**2

toolbox = base.Toolbox()
toolbox.register("evaluate", evalFct)
toolbox.decorate("evaluate", tools.DeltaPenality(feasible, 7.0, distance))_____no_output_____


# Notebook from cpinte/phantom-setup
Path: examples/setup_disc.ipynb

# Accretion disc

In this tutorial we set up a protoplanetary disc around a star represented by a sink particle, and we add a planet. This notebook generates a Phantom "temporary" dump file that can be read by Phantom as an initial condition. It also generates a Phantom "in" file. Together, these files can start a Phantom simulation._____no_output_____## Initialization

First we import the required modules._____no_output_____

import matplotlib.pyplot as plt
import numpy as np
import phantomsetup_____no_output_____

Here we set some constants for convenience._____no_output_____

igas = phantomsetup.defaults.PARTICLE_TYPE['igas']_____no_output_____

## Parameters

Now we set the parameters for the problem.

First is the `prefix` which sets the file name for the dump file and Phantom in file._____no_output_____

prefix = 'disc'_____no_output_____

### Resolution

We choose the resolution to be $10^6$ gas particles._____no_output_____

number_of_particles = 1_000_000
particle_type = igas_____no_output_____

### Viscosity

The SPH $\alpha$ viscosity parameter is its minimal value of 0.1._____no_output_____

alpha_artificial = 0.1_____no_output_____

### Units

We set the length and mass units to be au and solar masses, respectively. We will also set the time unit such that the gravitational constant is unity._____no_output_____

length_unit = phantomsetup.units.unit_string_to_cgs('au')
mass_unit = phantomsetup.units.unit_string_to_cgs('solarm')
gravitational_constant = 1.0_____no_output_____

### Star

The star is of solar mass, at the origin, with a 5 au accretion radius._____no_output_____

stellar_mass = 1.0
stellar_accretion_radius = 5.0
stellar_position = (0.0, 0.0, 0.0)
stellar_velocity = (0.0, 0.0, 0.0)_____no_output_____

### Disc

The disc has mass 0.01 solar masses, it extends from 10 au to 200 au._____no_output_____

radius_min = 10.0
radius_max = 200.0_____no_output_____disc_mass = 0.01_____no_output_____

### Equation of state

The equation of state is locally isothermal. We set the aspect ratio H/R at a reference radius._____no_output_____

ieos = 3
q_index = 0.75
aspect_ratio = 0.05
reference_radius = 10.0_____no_output_____

### Planet

We add a planet at 100 au._____no_output_____

planet_mass = 0.001
planet_position = (100.0, 0.0, 0.0)_____no_output_____orbital_radius = np.linalg.norm(planet_position)
planet_velocity = np.sqrt(gravitational_constant * stellar_mass / orbital_radius)_____no_output_____

We set the planet accretion radius as a fraction of the Hill sphere radius._____no_output_____

planet_accretion_radius_fraction_hill_radius = 0.25_____no_output_____planet_hill_radius = phantomsetup.orbits.hill_sphere_radius(
    orbital_radius, planet_mass, stellar_mass
)
planet_accretion_radius = (
    planet_accretion_radius_fraction_hill_radius * planet_hill_radius
)_____no_output_____

### Surface density distribution

For the surface density distribution we use the Lynden-Bell and Pringle (1974) self-similar solution, i.e. a power law with an exponential taper._____no_output_____

def density_distribution(radius, radius_critical, gamma):
    """Self-similar disc surface density distribution.

    This is the Lyden-Bell and Pringle (1974) solution, i.e. a power law
    with an exponential taper.
    """
    return phantomsetup.disc.self_similar_accretion_disc(radius, radius_critical, gamma)

radius_critical = 100.0
gamma = 1.0_____no_output_____args = (radius_critical, gamma)_____no_output_____

## Instantiate the `Setup` object

The following instantiates the `phantomsetup.Setup` object._____no_output_____

setup = phantomsetup.Setup()_____no_output_____

## Set attributes and add particles

### Prefix

Set the prefix._____no_output_____

setup.prefix = prefix_____no_output_____

### Units

Set units._____no_output_____

setup.set_units(
    length=length_unit, mass=mass_unit, gravitational_constant_is_unity=True
)_____no_output_____

### Equation of state

Set the equation of state. We get `polyk` from the aspect ratio parametrization._____no_output_____

polyk = phantomsetup.eos.polyk_for_locally_isothermal_disc(
    q_index, reference_radius, aspect_ratio, stellar_mass, gravitational_constant
)_____no_output_____setup.set_equation_of_state(ieos=ieos, polyk=polyk)_____no_output_____

### Viscosity

Set the numerical viscosity to Phantom disc viscosity._____no_output_____

setup.set_dissipation(disc_viscosity=True, alpha=alpha_artificial)_____no_output_____

### Star

Add a star at the origin._____no_output_____

setup.add_sink(
    mass=stellar_mass,
    accretion_radius=stellar_accretion_radius,
    position=stellar_position,
    velocity=stellar_velocity,
)_____no_output_____

### Disc

Add the disc around the star._____no_output_____

disc = phantomsetup.Disc(
    particle_type=particle_type,
    number_of_particles=number_of_particles,
    disc_mass=disc_mass,
    density_distribution=density_distribution,
    radius_range=(radius_min, radius_max),
    q_index=q_index,
    aspect_ratio=aspect_ratio,
    reference_radius=reference_radius,
    stellar_mass=stellar_mass,
    gravitational_constant=gravitational_constant,
    extra_args=(radius_critical, gamma),
)

setup.add_container(disc)_____no_output_____

### Planet

Add a planet in orbit around the star._____no_output_____

setup.add_sink(
    mass=planet_mass,
    accretion_radius=planet_accretion_radius,
    position=planet_position,
    velocity=planet_velocity,
)_____no_output_____

## Plot

Now we plot some quantities to see what we have set up.

First is the particles in the xy-plane. The sink particles are marked in red._____no_output_____

x, y, z = disc.arrays['position'][:, 0], disc.arrays['position'][:, 1], disc.arrays['position'][:, 2]

fig, ax = plt.subplots()
ax.plot(x[::10], y[::10], 'k.', ms=0.5)
for sink in setup.sinks:
    ax.plot(sink.position[0], sink.position[1], 'ro')
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
ax.set_aspect('equal')_____no_output_____

Next we plot the particles in the rz-plane._____no_output_____

position_cylindrical, velocity_cylindrical = phantomsetup.geometry.coordinate_transform(
    position=disc.arrays['position'],
    velocity=disc.arrays['velocity'],
    geometry_from='cartesian',
    geometry_to='cylindrical'
)

R = position_cylindrical[:, 0]

fig, ax = plt.subplots()
ax.plot(R[::10], z[::10], 'k.', ms=0.5)
ax.set_xlabel('$R$')
ax.set_ylabel('$z$')
ax.set_aspect('equal')
ax.set_ylim(bottom=2 * z.min(), top=2 * z.max())_____no_output_____

Finally, we plot $v_{\phi}$ as a function of radius._____no_output_____

vphi = velocity_cylindrical[:, 1]

fig, ax = plt.subplots()
ax.plot(R[::10], vphi[::10], 'k.', ms=0.5)
ax.set_xlabel('$R$')
ax.set_ylabel('$v_{\phi}$')_____no_output_____

## Write to file

Now that we are happy with the setup, write the "temporary" dump file with the initial conditions and the Phantom "in" file.

First we set a working directory for the simulation._____no_output_____

working_dir = '~/runs/disc'_____no_output_____setup.write_dump_file(directory=working_dir)
setup.write_in_file(directory=working_dir)_____no_output_____

## Compile Phantom

You can start a Phantom calculation from these two files but you must compile Phantom with the correct Makefile variables. We can use the `phantom_compile_command` method to show how Phantom would be compiled._____no_output_____

print(setup.phantom_compile_command())make \
  SETUP=empty \
  SYSTEM=gfortran \
  HDF5=yes \
  HDF5ROOT=/usr/local/opt/hdf5 \
  DISC_VISCOSITY=yes \
  DRIVING=no \
  DUST=no \
  DUSTGROWTH=no \
  GRAVITY=no \
  H2CHEM=no \
  IND_TIMESTEPS=yes \
  INJECT_PARTICLES=no \
  ISOTHERMAL=yes \
  KERNEL=cubic \
  MAXDUSTSMALL=11 \
  MAXDUSTLARGE=11 \
  MCFOST=no \
  MHD=no \
  NONIDEALMHD=no \
  PERIODIC=no \
  PHOTO=no

To compile Phantom we can use the `compile_phantom` method to compile Phantom. For example, the following will compile Phantom located in `phantom_dir` and copy the binary to `working_dir`.

```
result = setup.compile_phantom(
    phantom_dir='~/repos/phantom',
    working_dir=working_dir
)
```_____no_output_____


# Notebook from kaituohuo/kaldo
Path: docs/docsource/introduction.ipynb

### Introduction

κALDo is a versatile and scalable open-source software to compute phonon transport in crystalline and amorphous solids. It features real space QHGK calculations and three different solvers of the linearized BTE: direct inversion, self-consistent cycle, and RTA.

The algorithms are implemented using linear algebra operations on tensors, to take advantage of multithreading on GPU and CPU using Numpy, Tensorflow and optimized tensor libraries.

Using the Atomic Simulation Environment, κALDo can calculate the IFCs using several ab initio and molecular dynamics codes, thus enabling the use of first-principles DFT, empirical forcefields and semi-empirical tight- binding. A native LAMMPS interface is also available in the USER-PHONON package. Finally, through seamless integration with the hiPhive package, the IFC calculation can take advantage of compressing-sensing machine learning algorithms

The code is released open-source for the community to use and contribute with edits and suggestions. It is designed on modern software best practices, and we hope to provide a development platform to implement new theory and methods.

### Quick Install

We recommend creating a new environment with Python 3.6, 3.7, or 3.8.
```bash
conda create -n kaldo python=3.7
```
and enable the environment
```bash
conda activate kaldo
conda install pip
```

kALDo installation can be done using `pip`
```bash
pip install git+https://github.com/nanotheorygroup/kaldo
```

#### Using `pip` and `virtualenv`

You can also install kALDO without using `conda`
```bash
pip3 install virtualenv
virtualenv --system-site-packages -p python3 ~/kaldo
source ~/kaldo/bin/activate
pip3 install git+https://github.com/nanotheorygroup/kaldo
```
#### Development mode

The best way to run examples, tests and to develop kaldo is to follow the quick install procedure, and add the following extra steps.
```bash
pip uninstall kaldo
make ~/develoment
cd ~/development
git clone git+https://github.com/nanotheorygroup/kaldo
export PYTHONPATH=~/development/kaldo:$PYTHONPATH
```
If you followed the steps in the quickstart and then uninstall kaldo, you will have all the dependencies correctly installed.
The next lines are pulling the repo from Github and adding it to the `PYTHONPATH`.

If you want to make the last change in the `PYTHONPATH` permanent, you can also run
```bash
echo "export PYTHONPATH=~/development/kaldo:$PYTHONPATH" >> ~/.bashrc
```

### Interfacing with Other Codes

#### LAMMPS setup

In order to use LAMMPS with ASE, it needs to be compiled a lib
```bash
cd path/to/lammps/src
make yes-manybody
make yes-molecule
...
make mpi mode=shlib
```
Next, activate your environment
```
conda activate kaldo
```
or if you used `virtualenv`
```
source ~/kaldo/bin/activate
```
and finally install `lammpslib` using
```
make install-python
```
Update 07/2020: It you are having any issue with the July 2020 version of `lammpslib`, try using the September 2019 version.

#### Quantum Espresso setup

In order to use QE with ASE, Set the environment variable:
```bash
export ASE_ESPRESSO_COMMAND="mpirun -np 4 /path/to/pw.x -in PREFIX.pwi > PREFIX.pwo"
```

See the [ASE documentation](https://wiki.fysik.dtu.dk/ase/ase/calculators/calculators.html) for more info.

### Output Storage

#### Default Storage

When using the storage `formatted`, kALDo stores the following dense tensor as formatted, human readable, files:

- `frequency` $(n_{kpoints}, n_{modes})$. mode changes first k changes after
- `velocity_alpha` $(n_{kpoints}, n_{modes})$
- `physical_mode_<min_freq>_<max_freq>_<is_nw>` $(n_{kpoints}, n_{modes})$

- `<temperature>/<statistics>/heat_capacity` $(n_{kpoints}, n_{modes})$
- `<temperature>/<statistics>/population` $(n_{kpoints}, n_{modes})$

- `<temperature>/<statistics>/<method>/<third_bandwidth>/<diffusivity_bandwidth>/conductivity_<alpha>_<beta>` $(n_{kpoints}, n_{modes})$ where the `<third_bandwidth>/<diffusivity_bandwidth>` folder is created only if those values are defined
- `<temperature>/<statistics>/<diffusivity_bandwidth>/diffusivity` $(n_{kpoints}, n_{modes})$
- `<temperature>/<statistics>/<third_bandwidth>/<method>/mean_free_path` $(n_{kpoints}, n_{modes})$
- `<temperature>/<statistics>/<third_bandwidth>/lifetime` $(n_{kpoints}, n_{modes})$
- `<temperature>/<statistics>/<third_bandwidth>/bandwidth` $(n_{kpoints}, n_{modes})$
- `<temperature>/<statistics>/<third_bandwidth>/phase_space` $(n_{kpoints}, n_{modes})$
- `<diffusivity_bandwidth>flux_dense` $(n_{kpoints}, n_{modes}, n_{kpoints}, n_{modes})$, when `diffusivity_threshold` is not specified.
- `<diffusivity_bandwidth>/<diffusivity_threshold>/flux_sparse` $(n_{kpoints}, n_{modes}, n_{kpoints}, n_{modes})$. Sparse only when  `diffusivity_threshold` is specified.

The folder structure depends on the input parameters to the Phonon Object
and in parenthesis is the shape of the tensor. All of the above observables are stored in a dense format,
except for `flux_alpha` which is stored as formatted file in a `index value` format.

The following tensors are stored in raw binary format and help saving time when performing different simulations on the same sample.

- `_eigensystem (eigenvalues and eigenvectors)`
- `_dynmat_derivatives`
- `<temperature>/<statistics>/<diffusivity_bandwidth>/_generalized_diffusivity`
- `<temperature>/<statistics>/<third_bandwidth>/_ps_and_gamma_tensor`
- `<temperature>/<statistics>/<third_bandwidth>/_ps_and_gamma`, when only RTA conductivity is required

#### Alternative Storage

Other storage options available are `numpy` and `hdf5` where all the files are saved as one of those formats.
Finally the `memory` option doesn't store any permanent files.

| Measurement             | Units                    |
|-------------------------|--------------------------|
| Distances               | $A$                      |
| Masses                  | $g/N_A$                  |
| Frequencies             | $THz$                    |
| Potential Derivatives   | $eV/(A^2)$ or $eV/(A^3)$ |
| Conductivity [$\kappa$]| $\frac{W}{m\ K}$       |

### Code Architecture

### Main Features

Below we illustrate the main features of the code

### Examples

Examples and the explanation on how to use them are located in the [examples folder of the repo](https://github.com/nanotheorygroup/kaldo/tree/master/examples).

### How to cite

If you have used kALDo, please cite the following article:

Barbalinardo, G.; Chen, Z.; Lundgren, N. W.; Donadio, D. Efficient Anharmonic Lattice Dynamics Calculations of Thermal Transport in Crystalline and Disordered Solids. J Appl Phys 2020, 128 (13), 135104–135112. https://doi.org/10.1063/5.0020443
also available open access on ArXiv: https://arxiv.org/abs/2009.01967

```
@article{kaldo,
author = {Barbalinardo, Giuseppe and Chen, Zekun and Lundgren, Nicholas W and Donadio, Davide},
title = {{Efficient anharmonic lattice dynamics calculations of thermal transport in crystalline and disordered solids}},
journal = {Journal of Applied Physics},
year = {2020},
volume = {128},
number = {13},
pages = {135104--12},
month = oct
}
```

### Copyright

Copyright (c) 2020, Giuseppe Barbalinardo, Zekun Chen, Nicholas W. Lundgren, Davide Donadio

### Acknowledgements

We gratefully acknowledge support by the Investment Software Fellowships (grant No. ACI-1547580-479590) of the NSF Molecular Sciences Software Institute (grant No. ACI-1547580) at Virginia Tech.

MolSSI builds open source software and data which serves the computational molecular science community. [Explore MolSSI’s software infrastructure projects.](https://molssi.org/software-projects/)

### Other codes

We are thankful to the whole open-source community, and in particular we want to acknowledge some other great anharmonic lattice dynamics related projects. If using kALDo example, you
If, while you are running kALDo examples, you use any of these codes, please don't forget to cite them.

- ASE: A. H. Larsen, J. J. Mortensen, J. Blomqvist, I. E. Castelli, R. Christensen, M. Dułak, J. Friis, M. N. Groves, B. Hammer, C. Hargus, E. D. Hermes, P. C. Jennings, P. B. Jensen, J. Kermode, J. R. Kitchin, E. L. Kolsbjerg, J. Kubal, K. Kaasbjerg, S. Lysgaard, J. B. Maronsson, T. Maxson, T. Olsen, L. Pastewka, A. Peterson, C. Rostgaard, J. Schiøtz, O. Schütt, M. Strange, K. S. Thygesen, T. Vegge, L. Vilhelmsen, M. Walter, Z. Zeng, and K. W. Jacobsen, “The atomic simulation environment—a python library for work- ing with atoms,” Journal of Physics: Condensed Matter 29, 273002 (2017).
- LAMMPS: S. Plimpton, Fast Parallel Algorithms for Short-Range Molecular Dynamics, J Comp Phys, 117, 1-19 (1995).
- Quantum Espresso: P Giannozzi, O Andreussi, T Brumme, O Bunau, M Buongiorno Nardelli, M Calandra, R Car, C Cavazzoni, D Ceresoli, M Cococcioni, N Colonna, I Carnimeo, A Dal Corso, S de Gironcoli, P Delugas, R A DiStasio Jr, A Ferretti, A Floris, G Fratesi, G Fugallo, R Gebauer, U Gerstmann, F Giustino, T Gorni, J Jia, M Kawamura, H-Y Ko, A Kokalj, E Küçükbenli, M Lazzeri, M Marsili, N Marzari, F Mauri, N L Nguyen, H-V Nguyen, A Otero-de-la-Roza, L Paulatto, S Poncé, D Rocca, R Sabatini, B Santra, M Schlipf, A P Seitsonen, A Smogunov, I Timrov, T Thonhauser, P Umari, N Vast, X Wu and S Baroni, J.Phys.:Condens.Matter 29, 465901 (2017)
- HiPhive: The Hiphive Package for the Extraction of High‐Order Force Constants by Machine Learning, Fredrik Eriksson, Erik Fransson, and Paul Erhart, Advanced Theory and Simulations,  (2019)

### Project template

Project template from the [Computational Molecular Science Python Cookiecutter](https://github.com/molssi/cookiecutter-cms).

### How to contribute changes
- Clone the repository if you have write access to the main repo, fork the repository if you are a collaborator.
- Make a new branch with `git checkout -b {your branch name}`
- Make changes and test your code
- Ensure that the test environment dependencies (`conda-envs`) line up with the build and deploy dependencies (`conda-recipe/meta.yaml`)
- Push the branch to the repo (either the main or your fork) with `git push -u origin {your branch name}`
  * Note that `origin` is the default name assigned to the remote, yours may be different
- Make a PR on GitHub with your changes
- We'll review the changes and get your code into the repo after lively discussion!

#### Checklist for updates
- [ ] Make sure there is an/are issue(s) opened for your specific update
- [ ] Create the PR, referencing the issue
- [ ] Debug the PR as needed until tests pass
- [ ] Tag the final, debugged version
   *  `git tag -a X.Y.Z [latest pushed commit] && git push --follow-tags`
- [ ] Get the PR merged in

#### Versioneer Auto-version
[Versioneer](https://github.com/warner/python-versioneer) will automatically infer what version
is installed by looking at the `git` tags and how many commits ahead this version is. The format follows
[PEP 440](https://www.python.org/dev/peps/pep-0440/) and has the regular expression of:
```regexp
\d+.\d+.\d+(?\+\d+-[a-z0-9]+)
```
If the version of this commit is the same as a `git` tag, the installed version is the same as the tag,
e.g. `kaldo-0.1.2`, otherwise it will be appended with `+X` where `X` is the number of commits
ahead from the last tag, and then `-YYYYYY` where the `Y`'s are replaced with the `git` commit hash._____no_output_____