# Notebook from sayakpaul/DataCamp-blogs
Path: DataCamp blog on Linear Regression/Essentials of Linear Regression in Python.ipynb

# Essentials of Linear Regression in Python

Learn what formulates a regression problem and how a linear regression algorithm works in Python._____no_output_____The field of Data Science has progressed like nothing before. It incorporates so many different domains like Statistics, Linear Algebra, Machine Learning, Databases into its account and merges them in the most meaningful way possible. But, in its core, what makes this domain one of the craziest ones? - **The powerful statistical algorithms**

One such very primitive statistical algorithm is **Linear Regression**. Although it is very very old, but never too old to be neglected for a budding data scientist like you. Understanding the principle behind the working of linear regression is very important as to reason the evolution of a whole class of statistical algorithms called **Generalized Linear Models**. Moreover, it will also help you understand other aspects of a typical statistical/machine learning algorithm for example - cost functions, coefficients, optimization etc.

As the title of this tutorial suggests, you will cover Linear Regression in details in this tutorial. Specifically you will cover:
- Understanding a regression problem
- Coming up with a hypothesis for the regression problem
- What is Linear Regression
- Cost functions
- Optimizing Linear Regression
- Why it is still important to understand even in the world of neural networks
- A case study in Python _____no_output_____Before diving deep into the theories behind Linear Regression let's have a clear view of the term _regression_. _____no_output_____## Understanding a regression problem:

Regression belongs to the class of [Supervised Learning]("https://en.wikipedia.org/wiki/Supervised_learning") tasks where the the datasets that are used for predictive/statistical modeling contain [continuous labels]("http://www.statisticshowto.com/continuous-variable/"). But, let's define a regression problem more mathematically. _____no_output_____Let's consider the following image below:

**Source: Andrew Ng's lecture notes**_____no_output_____So, in the above image, X is the set of values that correspond to the living areas of various houses (also considered as the space of input values) and y is the price of the respective houses but note that this values is predicted by h. h is the function that maps the X values to y (often called as predictor). For historical reasons, this h is referred to as a hypothesis function. Keep in mind that, this dataset has only feature i.e. the living areas of various houses and consider this to be a toy dataset for the sake of understanding.

Note that, the predicted values here are continuous in nature. So, your ultimate goal is, given a training set, to learn a function
$h : \mathcal{X}  \rightarrow \mathcal{Y}$ so that h(x) is a "good" predictor for the corresponding value of y. Also, keep in mind that the domain of values that both X and Y accept are all real numbers and you can define it like this: $\mathcal{X}  = \mathcal{Y} = \mathbb{R}$ where, $\mathbb{R}$ is the set of all real numbers.

A pair (x(i), y(i)) is called a training example. You can define the training set as {(x(i), y(i)) ; i = 1,...,m} (in case the training set contains m instances and there is only one feature x in the dataset)._____no_output_____A bit of mathematics there for you so that you don't go wrong even in the simplest of things. So, according to Han, Kamber and Pei-

_"In general, these methods are used to predict the value of a response
(dependent) variable from one or more predictor (independent) variables, where the
variables are numeric._" - [Data Mining: Concepts and Techniques (3rd edn.)]("https://scholar.google.com/scholar?q=Han%2C%20J.%2C%20Kamber%2C%20M.%2C%20Pei%2C%20J.%3A%20Data%20Mining%3A%20Concepts%20and%20Techniques%20%283rd%20edn.%29%2C%20pp.%2056.%20Morgan%20Kaufmann%20Publishers%20Inc.%20San%20Francisco%20%282011%29")

As simple as that!_____no_output_____So, in the course of understanding a typical regression problem you also saw how to define a hypothesis for it as well. Brilliant going. You have set the mood just perfect! Now, you will straight dive into the mechanics of Linear Regression. _____no_output_____## Linear Regression - how it works?

Before going into its details, won't it be good to take a look when it was discovered? Well that goes way way back to 18th Century. The most trivial form of statistical regression was first proposed by the mighty **Carl Friedrich Gauss** but there are many arguments on this. Let's not get into those. But if you are really interested to see the arguments that took place for this between Gauss and **Adrien-Marie Legendre** [this is the link]("https://priceonomics.com/the-discovery-of-statistical-regression/") to go for.

Linear regression is perhaps one of the most well known and well understood algorithms in statistics and machine learning. Linear regression was developed in the field of statistics and is studied as a model for understanding the relationship between input and output numerical variables but with the course of time it has become an inseparable part of modern machine learning toolbox.

Let's have a toy dataset for it. You will use the same house price prediction dataset to investigate this but this time with two features. The task remains same i.e. predicting the house price.

**Source: Andrew Ng's lecture notes**_____no_output_____As mentioned earlier, now the x’s are two-dimensional which means your dataset contains two features. For instance, x1(i) is the living area of the i-th house in the training set, and x2(i) is its number of bedrooms.

To perform regression, you must decide the way you are going to represent h. As an initial choice, let’s say you decide to approximate y as a linear function of x:

hθ(x) = θ0 + θ1x1 + θ2x2

Here, the θi’s are the parameters (also called weights) parameterizing the space of linear functions mapping from $\mathcal{X}$ to $\mathcal{Y}$. In a simpler sense, these parameters are used for accurately mapping $\mathcal{X}$ to $\mathcal{Y}$. But to keep things simple for your understanding, you will drop the θ subscript in hθ(x), and write it simply as h(x). To simplify your notation even further, you will also introduce the convention of letting x0 = 1 (this is the intercept term), so that

where on the right-hand side above you are considering θ and x both as vectors, and here n is the number of input instances (not counting x0)._____no_output_____But the main question that gets raised at this point of time is how do you pick or learn the parameters θ? You cannot change your input instances as to predict the prices. You have only these θ parameters to tune/adjust.

One prominent method seems to be to make h(x) close to y, at least for the training examples you have. To understand this more formally, let's try defining a function that determines, for each value of the θ’s, how close the h(x(i))’s are to the corresponding y(i) ’s. The function should look like the following:

[Source: StackOverflow]("https://stackoverflow.com/questions/32274474/machine-learning-linear-regression-using-batch-gradient-descent")

To understand the reason behind taking the squared value instead of the absolute value, consider this squared-term as an advantage for the future operations to be performed for training the regression model. But if you want to dig deeper, [help yourselves](https://www.quora.com/Why-do-we-use-square-error-instead-of-absolute-value-when-we-calculate-R-2-in-regression-analysis)._____no_output_____You just saw one of the most important formulas in the world of Data Science/Machine Learning/Statistics. It is called as **cost function**. _____no_output_____This is an extremely important derivation because not only it gives birth to the next evolution of the linear regression (Ordinary Least Squares) but also formulates the foundations of a whole class of linear modeling algorithms (remember you came across a term called _Generalized Linear Models_). _____no_output_____It is important to note that, linear regression can often be divided into two basic forms:
- Simple Linear Regression (SLR) which deals with just two variables (the one you saw at first)
- Multi-linear Regression (MLR) which deals with more than two variables (the one you just saw)

These things are very simple but can often cause confusion.

You have already laid your foundations of linear regression. Now you will study more about the ways of estimating the parameters you saw in the above section. This estimation of parameters is essentially known as the training of linear regression. Now, there are many methods to train a linear regression model **Ordinary Least Squares** (OLS) being the most popular among them. So, it is good to refer a linear regression model trained using OLS as Ordinary Least Squares Linear Regression or just Least Squares Regression.

**Note that parameters here in this context are also called model coefficients.**_____no_output_____## Optimizing a linear regression model - various approaches:

Learning/training a linear regression model essentially means estimating the values of the coefficients/parameters used in the representation with the data you have.

In this section you will take a brief look at some techniques to prepare a linear regression model. _____no_output_____## Least squares' regression:

You left the previous section with a notion to choose θ so as to minimize J(θ). To do so, let’s use a search algorithm that starts with some "initial guess" for θ, and that iteratively changes θ to make J(θ) smaller, until hopefully you converge to a value of θ that minimizes J(θ). Specifically, let’s consider the **gradient descent algorithm**, which starts with some initial θ, and repeatedly performs the update:

**Source: Andrew Ng's lecture notes**

(This update is simultaneously performed for all values of j = 0, . . . , n.) Here, α is called the **learning rate**. This is a very natural algorithm that repeatedly takes a step in the direction of steepest decrease of J. This term α effectively controls how steep your algorithm would move to the decrease of J. It can be pictorially expressed as the following:

** Source: [ml-cheatsheet](https://ml-cheatsheet.readthedocs.io/en/latest/_images/gradient_descent_demystified.png)**_____no_output_____Intuitively speaking, the above formula denotes the small change that happens in J w.r.t the θj parameter and how it affects the initial value of θj. But look carefully, you have a partial derivative here to deal with. The whole derivation process is out of the scope of this tutorial.

Just note that for a single training example, this gives the update rule:

**Source: [ml-cheatsheet](https://ml-cheatsheet.readthedocs.io/en/latest/_images/gradient_descent_demystified.png)**_____no_output_____The rule is called the LMS update rule (LMS stands for “least mean squares”), and is also known as the **Widrow-Hoff learning rule**.

Let's summarize few things in the context of OLS.

_"The Ordinary Least Squares procedure seeks to minimize the sum of the squared residuals. This means that given a regression line through the data we calculate the distance from each data point to the regression line, square it, and sum all of the squared errors together. This is the quantity that ordinary least squares seeks to minimize."_ - [Jason Brownlee](https://machinelearningmastery.com/linear-regression-for-machine-learning/)_____no_output_____## Optimization with gradient descent:

In the previous training rule you already got the notion of how gradient descent can be incorporated in this context. Essentially, gradient descent is a process of optimizing the values of the coefficients by iteratively minimizing the error of the model on your training data.

More briefly speaking, it works by starting with random values for each coefficient. The sum of the squared errors are calculated for each pair of input and output values. A learning rate is used as a scale factor and the coefficients are updated in the direction towards minimizing the error. The process is repeated until a minimum sum squared error is achieved or no further improvement is possible.

The term α (learning rate) is very important here since it determines the size of the improvement step to take on each iteration of the procedure.

Now there are commonly two variants of gradient descent:

- Method that looks at every example in the entire training set on every step, and is called **batch gradient descent**.
- Method where you repeatedly run through the training set, and each time you encounter a training example, you update the parameters according to the gradient of the error with respect to that single training example only. This algorithm is called **stochastic gradient descent** (also incremental gradient descent).

That is all for gradient descent for this tutorial. Now, you take a look at another way of optimizing a linear regression model i.e. Regularization. _____no_output_____## Regularization:

DataCamp already has a fairly good introductory [article on Regularization]("https://www.datacamp.com/community/tutorials/towards-preventing-overfitting-regularization"). You might want to check that out before proceeding with this one.

Generally, regularization methods work by penalizing the coefficients of features having extremely large values and thereby try to reduce the error. It not only results in an enhanced error rate but also, it reduces the model complexity. This is particularly very useful when you are dealing with a dataset that has a large number of features and your baseline model is not able to distinguish between the importance of the features (not all features in a dataset are equally important, right?).

There are two variants of regularization procedures for linear regression are:

**Lasso Regression**: adds a penalty term which is equivalent to **absolute value of magnitude of the coefficients** (also called _L1 regularization_). The penalty terms looks like:

where,
- $\lambda$ is the constant factor that you add in order control the speed of the improvement in error (**learning rate**)
- the dataset has (M+1) features, so it runs from 0 to M. wj is the weight/coefficient.

**Ridge Regression**: adds a penalty term which is equivalent to **square of the magnitude of coefficients** (also called _L2 regularization_). The penalty terms looks like:

_____no_output_____## Is it still worth the time to learn linear regression?

Is it not? You already saw how gracefully linear regression introduces some of the most important concepts of Machine Learning such as cost functions, optimization, variable relationships and what not? All these things are really vital even if you are constructing a neural network. The applicability may differ in some places but the overall concepts remain exactly the same. So, without understanding these fundamental things, you will never be able to reason why your neural net is not performing well.

Moreover, a simple concept of deriving the relationships among variables gave birth to so many concepts and most importantly it created a whole new family of algorithms - [Generalized Linear Models]("https://onlinecourses.science.psu.edu/stat504/node/216/"). So for aspiring Data Science/Machine Learning/Artificial Intelligence practitioner, this algorithm is something which cannot be neglected. You already have understood that by now!_____no_output_____No you will implement a simple linear regression in Python for yourself now. It should be fun!_____no_output_____## A case study in Python:

For this case study first you will use the [Statsmodel]("http://www.statsmodels.org/stable/index.html") library for Python. It is a very popular library which provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration. For the data, you will use the famous [Boston House dataset]("https://www.kaggle.com/c/boston-housing"). The mighty scikit-learn comes with this dataset, so you don't need to download it separately.

Let's start off the case study by importing the `statsmodels` library and your dataset:_____no_output_____

import statsmodels.api as sm
from sklearn import datasets

data = datasets.load_boston()_____no_output_____

Scikit-learn provides a handy description of the dataset and it can be easily viewed by:_____no_output_____

print (data.DESCR)Boston House Prices dataset
===========================

Notes
------
Data Set Characteristics:

    :Number of Instances: 506

    :Number of Attributes: 13 numeric/categorical predictive

    :Median Value (attribute 14) is usually the target

    :Attribute Information (in order):
        - CRIM     per capita crime rate by town
        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
        - INDUS    proportion of non-retail business acres per town
        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
        - NOX      nitric oxides concentration (parts per 10 million)
        - RM       average number of rooms per dwelling
        - AGE      proportion of owner-occupied units built prior to 1940
        - DIS      weighted distances to five Boston employment centres
        - RAD      index of accessibility to radial highways
        - TAX      full-value property-tax rate per $10,000
        - PTRATIO  pupil-teacher ratio by town
        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
        - LSTAT    % lower status of the population
        - MEDV     Median value of owner-occupied homes in $1000's

    :Missing Attribute Values: None

    :Creator: Harrison, D. and Rubinfeld, D.L.

This is a copy of UCI ML housing dataset.
http://archive.ics.uci.edu/ml/datasets/Housing

This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
prices and the demand for clean air', J. Environ. Economics & Management,
vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
...', Wiley, 1980.   N.B. Various transformations are used in the table on
pages 244-261 of the latter.

The Boston house-price data has been used in many machine learning papers that address regression
problems.

**References**

   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.
   - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)

Now, before applying linear regression, you will have to prepare the data and segregate the features and the label of the dataset. `MEDV` (median home value) is the label in this case. You can access the features of the dataset using `feature_names` attribute.

A bit of `pandas` knowledge will come in handy here. [This cheatsheet](https://s3.amazonaws.com/assets.datacamp.com/blog_assets/PandasPythonForDataScience.pdf) is a must-see if you are looking for ways to refresh basic `pandas` concepts. _____no_output_____

# Pandas and NumPy import
import numpy as np
import pandas as pd

# Set the features
df = pd.DataFrame(data.data, columns=data.feature_names)

# Set the target
target = pd.DataFrame(data.target, columns=["MEDV"])_____no_output_____

At this point of time you need to consider few important things about linear regression before applying it to the data. You could have studied this earlier in this tutorial, but studying these factors at this particular point of time will help you get the real feel.

- **Linear Assumption**: Linear regression is best employed to capture the relationship between the input variables and the outputs. In order to do so, linear regression assumes this relationship to be linear (which might not be the case all the time). But you can always transform your data so that a linear relationship is maintained. For example, if your data has exponential relationship you can apply [_log-transform_](https://www.dummies.com/programming/big-data/data-science/data-science-using-python-to-modify-data-distributions/) to make the relationship linear.

- **Collinearity between the features**: Collinearity is a measure to calculate the importance of a feature of a dataset mathematically. When you have a dataset in which the features are very correlated to each other, linear regression fails to approximate the relationship appropriately and tends to overfit. So, it is very practical to detect the highly correlated features and to drop them before you apply linear regression. If you want to know more about this, feel free to check [this excellent Kaggle kernel](https://www.kaggle.com/robertoruiz/dealing-with-multicollinearity). _____no_output_____Let's do some hands-on now. To keep things simple you will just take `RM — the average number of rooms` feature for now. Note that `Statsmodels` does not add a constant term (recall the factor θ0) by default. Let’s see it first without the constant term in your regression model:_____no_output_____

X = df["RM"]
y = target["MEDV"]

# Fit and make the predictions by the model
model = sm.OLS(y, X).fit()
predictions = model.predict(X)

# Print out the statistics
model.summary()_____no_output_____

What is this output! It is way too big to understand when you are seeing it for the first time. Let's go through the most important points step by step:

- The first observation you should make here is you are using `OLS` method to train your linear regression model.

- There's a value corresponding to **R-Squared**. R-squared is the “percent of variance explained” by the model.  That is, R-squared is the fraction by which the variance of the errors is less than the variance of the dependent variable. R-squared values range from 0 to 1 and are commonly stated as percentages from 0% to 100%. R-squared will give you an estimate of the relationship between movements of a dependent variable based on an independent variable's movements. It doesn't tell you whether your chosen model is good or bad, nor will it tell you whether the data and predictions are biased. A high or low R-square isn't necessarily good or bad, as it doesn't convey the reliability of the model, nor whether you've chosen the right regression. You can get a low R-squared for a good model, or a high R-square for a poorly fitted model, and vice versa.

- The coefficient (_coef_) of 3.634 means that if the `RM` variable increases by 1, the predicted value of `MEDV` increases by 3.634.

-  There is a 95% confidence intervals for the RM which means that the model predicts at a 95% percent confidence that the value of RM is between 3.548 to 3.759).

These are the most important points you should take care of for the time being (and you can ignore the warning as well). _____no_output_____A constant term can easily be added to the linear regression model. you can do it by `X = sm.add_constant(X)` (X is the name of the dataframe containing the input (independent variables)._____no_output_____

X = sm.add_constant(X)

model = sm.OLS(y, X).fit()
predictions = model.predict(X)

model.summary()_____no_output_____

It can be clearly seen that the addition of the constant term has a direct effect on the coefficient term. Without the constant term, your model was passing through the origin, but now you have a y-intercept at -34.67. Now the slope of the `RM` predictor is also changed from 3.634 to 9.1021 (_coef_ of `RM`).

Now you will fit a regression model with more than one variable — you will add `LSTAT` (percentage of lower status of the population) along with the `RM` variable. The model training (fitting) procedure remains exactly the same as previous:_____no_output_____

X = df[["RM", "LSTAT"]]
y = target["MEDV"]

model = sm.OLS(y, X).fit()
predictions = model.predict(X)

model.summary()_____no_output_____

Let's interpret this one now:

This model has a much higher R-squared value — 0.948, which essentially means that this model captures 94.8% of the variance in the dependent variable. Now, let's try to figure out the relationship between the two variables `RM` and `LSTAT` and median house value. As `RM` increases by 1, `MEDV` will increase by 4.9069 and when `LSTAT` increases by 1, `MEDV` will _decrease_ by 0.6557. This indicates that `RM` and `LSTAT` are statistically significant in predicting (or estimating) the median house value.

You can interpret this relationship in plain English as well:
- Houses having a small number of rooms are likely to have low price values.
- In the areas where the status of the population, is lower the house prices are likely to be low.

Makes more sense now! Isn't it?_____no_output_____This was the example of both single and multiple linear regression in `Statsmodels`. Your homework will be to investigate and interpret the results with the further features.

Next, let's see how linear regression can be implemented using your very own `scikit-learn`. You already have the dataset imported but you will have to import the `linear_model` class. _____no_output_____

from sklearn import linear_model_____no_output_____X = df
y = target["MEDV"]

lm = linear_model.LinearRegression()
model = lm.fit(X,y)_____no_output_____

The model training is completed. This `sklearn` implementation also uses OLS. Let's do some prediction of `MEDV` values of the first five samples. _____no_output_____

predictions = lm.predict(X)
print(predictions[0:5])[30.00821269 25.0298606  30.5702317  28.60814055 27.94288232]

If you want to know some more details (such as the R-squared, coefficients etc.) of your model, you can easily do so. _____no_output_____

lm.score(X,y)_____no_output_____lm.coef______no_output_____

## Wrap up!

Beautiful! You have made it to the end. Covering one of the simplest and the most fundamental algorithms was not that easy but you did it pretty well. Not only, you got familiarized with simple linear regression but also studied many extremely important aspects, terms, factors of machine learning. You did an in-depth case study in Python as well.

This tutorial can also be treated as a motivation for you to implement Linear Regression from scratch. Following are the brief steps if anyone wants to do it for real:
- Calculate mean and variance of the data
- Calculate covariance
- Estimate coefficients
- Make predictions

Following are some references that were used in order to prepare this tutorial:
- [Machine Learning course by Coursera (taught by the great Andrew Ng)](https://www.coursera.org/learn/machine-learning)
- [Implementing linear regression from scratch](https://machinelearningmastery.com/implement-simple-linear-regression-scratch-python/)
- [The Elements of Statistical Learning](The Elements of Statistical Learning - Stanford University)_____no_output_____


# Notebook from Licho59/job_data_visualize
Path: Data_analysis_visualization/plotly_exercises.ipynb

Table of Contents

1  Plotly exercises from  https://plot.ly/python/user-guide/

2  The end

_____no_output_____#### Plotly exercises from  https://plot.ly/python/user-guide/_____no_output_____

import pandas as pd
import plotly
import plotly.graph_objs as go
from plotly.offline import iplot
import cufflinks as cf
cf.go_offline()_____no_output_____df = pd.read_csv(
    'http://www.stat.ubc.ca/~jenny/notOcto/STAT545A/examples/gapminder/data/gapminderDataFiveYear.txt',
    sep='\t')
df = df[df.year == 2007]_____no_output_____df.continent.unique()_____no_output_____americas = df[df.continent == 'Americas']
europe = df[df.continent == 'Europe']
asia = df[df.continent == 'Asia']
africa = df[df.continent == 'Africa']
oceania = df[df.continent == 'Oceania']_____no_output_____# a trace is just the name we give to a collection of data and the specifications of which we want that data plotted.
# a trace is also a dictionary-like object itself and will be named according to how you want the data displayed on the plotting surface.
trace_comp0 = go.Scatter(x=americas.gdpPercap, y=americas.lifeExp,
                        mode='markers',marker=dict(size=12,line=dict(width=1),color="navy"),
                        name='Americas', text=americas.country)

trace_comp1 = go.Scatter(x=europe.gdpPercap, y=europe.lifeExp,
                        mode='markers', marker=dict(size=12, line=dict(width=1),color="red"),
                        name='Europe', text=europe.country)

trace_comp2 = go.Scatter(x=asia.gdpPercap, y=asia.lifeExp,
                        mode='markers',marker=dict(size=12,line=dict(width=1),color="yellow"),
                        name='Asia', text=asia.country)

trace_comp3 = go.Scatter(x=africa.gdpPercap, y=africa.lifeExp,
                        mode='markers',marker=dict(size=12,line=dict(width=4),color="black"),
                        name='Africa', text=africa.country)

trace_comp4 = go.Scatter(x=oceania.gdpPercap, y=oceania.lifeExp,
                        mode='markers',marker=dict(size=12,line=dict(width=4),color="green"),
                        name='Oceania', text=oceania.country)_____no_output_____# data_comp contains all the traces that we wish to plot
data_comp = [trace_comp0, trace_comp1, trace_comp2, trace_comp3, trace_comp4]
layout_comp = go.Layout(title='Life Expectancy v. Per Capita GDP, 2007', hovermode='closest',
        xaxis=dict(title='GDP per capita (2000 dollars)',ticklen=5,zeroline=False,gridwidth=2,),
        yaxis=dict(title='Life Expectancy (years)',ticklen=5,gridwidth=2)) _____no_output_____# data is actually a list object
fig_comp = go.Figure(data=data_comp, layout=layout_comp)_____no_output_____iplot(fig_comp)_____no_output_____# code improved - no need to construct 5 separate traces
from plotly.graph_objs.scatter import Marker
trc = go.Scatter(x=df.gdpPercap, y=df.lifeExp, mode='markers', text=df.country)

# to match colors as below order should be the same as traces order having put into fig_comp 'data' key
continents = [americas, europe, asia, africa, oceania]
colors = ['red','blue','yellow','black','green']
style = [dict(marker=Marker(color=colors[i], size=14, line=dict(width=1, color='black'))) for i in range(len(continents))]

fig = go.Figure(data=[trc], layout=layout_comp)
fig = fig_comp.update({'data':style})
iplot(fig)_____no_output_____

df[['gdpPercap','lifeExp','country','continent']].iplot(kind='scatter', x='gdpPercap', y='lifeExp', mode='markers',
                                                       categories='continent', name='country',
                                                       xTitle='GDP per capita(2000 dollars)',
                                                       yTitle='Life Expectancy (years)',
                                                       title='Life Expectancy v. Per Capita GDP,2007')_____no_output_____# another example of scatter plot - simple one with only data object; to remember - data must be a list object!!
iplot([go.Scatter(x=[1,2,3], y=[4,5,6], marker={'color': 'red', 'symbol': 104, 'size': 10},
                mode="markers+lines",  text=["one","two","three"])])
# specifications related to plotting this data: the points plotted as hollow x's with lines joining them, all in red._____no_output_____# # easy to add another Scatter object to the data list (trace1 as above example)
trace1 = go.Scatter(x=[1,2,3], y=[4,5,6], marker={'color': 'red', 'symbol': 104, 'size': 10},
                mode="markers+lines",  text=["one","two","three"])
import numpy as np
x = np.arange(1, 3.2, 0.2)
y = 6* np.sin(x)

trace2 = go.Scatter(x=x, y=y, marker={'color':'blue', 'symbol':'star', 'size':10},
                   mode='markers', name='second_trace') # 'name' is not visible on graph, it is useful for editing in plot.ly
data = [trace1, trace2]
#data_____no_output_____# layout object is a dictionary object with specificatons common for all figure object(title, axes, legend, etc)
layout={'title':'First Plot', 'xaxis':{'title': 'x'}, 'yaxis':{'title':'y'}}
iplot(go.Figure(data=data, layout=layout))_____no_output_____#  we can also add some text annotation for example in order to indicate the maximum point that's been plotted
# on the current plotting surface; needed text and x,y arguments
layout.update(dict(annotations=[go.layout.Annotation(text="Highest Point", x=3, y=6)]))
iplot(go.Figure(data=data, layout=layout))_____no_output_____# Shapes - next upgrade to the layout-adding a rectangular block to highlight the secton where trace 1 is above trace 2
layout.update(dict(shapes = [
        # 1st highlight during Feb 4 - Feb 6
        {'type': 'rect',
        # x-reference is assigned to the x-values
        'xref': 'x',
        # y-reference is assigned to the plot paper [0,1]
        'yref': 'y',
        'x0': '1', 'y0': 0, 'x1': '2', 'y1': 7,
        'fillcolor': '#d3d3d3','opacity': 0.2,'line': {'width': 0,}}]))
iplot(go.Figure(data=data, layout=layout))_____no_output_____

# example of displaying a few plots in the one figure
from plotly import tools

heatmap = go.Heatmap(z=[[1, 20, 30],
                       [20, 1, 60],
                       [30, 60, 1]],showscale=False)

y0 = np.random.randn(50)
y1 = np.random.randn(50)+1

box_1 = go.Box(y=y0)
box_2 = go.Box(y=y1)
data = [heatmap, box_1, box_2]

# creating basic figure with places for subplots
fig = tools.make_subplots(rows=2, cols=2, specs=[[{}, {}], [{'colspan': 2}, None]],
                          subplot_titles=('First Subplot','Second Subplot', 'Third Subplot'))
# appending figure with traces
fig.append_trace(box_1, 1, 1)
fig.append_trace(box_2, 1, 2)
fig.append_trace(heatmap, 2, 1)

fig['layout'].update(height=600, width=600, title='i <3 subplots')

iplot(fig, filename='box_heatmap1');
c:\users\14000322\appdata\local\programs\python\python36-32\lib\site-packages\plotly\tools.py:465: DeprecationWarning:

plotly.tools.make_subplots is deprecated, please use plotly.subplots.make_subplots instead

#### The end_____no_output_____

import seaborn as sns
df = sns.load_dataset('tips')
df.head()
df.info()_____no_output_____# numpy arrays taken further as x and y to use in plot
df.day.value_counts().index.values.astype(str)
df.day.value_counts().values_____no_output_____# widgets are very helpful to make online changes in data params or plotting data
import ipywidgets as widgets
from ipywidgets import interact, interact_manual_____no_output_____# color set
_11_lg = '#95C061'
_12_lb = '#75ABDE'
_13_lo = '#FE9C43'
_14_lbr = '#A6886D'
_15_lr = '#E7675D'
_16_lp = '#7C5674'

color_set = [_11_lg, _12_lb, _13_lo, _14_lbr, _15_lr, _16_lp]

data = []
data.append(go.Bar(
            x = df['day'].value_counts().index.values.astype('str'), # Need to use .astype('str') to transform categorical data
            y = df['day'].value_counts().values,
            marker = dict(color = color_set[0])))

iplot(data)_____no_output_____

# horizontal version
data = []
data.append(go.Bar(x=df.day.value_counts().values,
                  y=df.day.value_counts().index.values.astype(str),
                  marker=dict(color=color_set[3]),
                  orientation='h'))
layout=go.Layout(xaxis=dict(title='Number of visits'), title='Activities per day')
iplot(go.Figure(data,layout))_____no_output_____# the basic plot with the cufflinks usage (dataframe.iplot())
# creating small dataframe by taking value_counts() method and next using to_frame() method
df2 = df.day.value_counts().to_frame()
df2
df2_index = df2.index.values.astype(str)
df2.iplot(kind='bar', color=color_set[4])_____no_output_____# the same plot but after resetting index it needs to add y value and x (optionally)
df2.reset_index()
df2.reset_index().iplot(kind='bar', x='index', y='day', text='index', color=color_set[2])_____no_output_____# with cufflinks horizontal bar needs kind='barh' - no x and y
df2.iplot(kind='barh', color=color_set[1])_____no_output_____# using histogram to show the data distribution (for seaborn it is distplot() function)
data = []
# nbinsx - maximum number of bins but algorithm finds the optimum number
data.append(go.Histogram(x=df.tip, nbinsx=30, marker=dict(color=color_set[0], line=dict(color=color_set[1], width=1))))
iplot(data)_____no_output_____# the same with cufflinks (bins means exact number of bins as given)
df.tip.iplot(kind='hist', bins= 20)_____no_output_____# Box object in plotly has equivalent in Seaborn - boxplot() function
data = []
for col in df.day.unique().astype(str):
    data.append(go.Box(y=df[df.day==col]['tip'], name=col, showlegend=False)) # no x because col represents it
iplot(data)_____no_output_____

# bubble plot with scatter object, 'bubbles' made with marker's size parameter (need to be used in brackets!), because
# df.size means the amount of data in dataframe
data = []
data.append(go.Scatter(x=df.total_bill, y=df.tip, mode='markers', marker=dict(size=df['size']*5)))
layout = go.Layout(xaxis=dict(title='Total Bil'), yaxis=dict(title='Tip Value'), title='Tips vs Total Bills')
iplot(go.Figure(data=data, layout=layout))

# cufflinks way
df.iplot(kind='bubble', x='total_bill', y='tip', size='size', color='orange', theme='solar')_____no_output_____# heatmap in plotly cufflinks (kind='heatmap') equivalates to Seaborn's heatmap() function
df.corr().iplot(kind='heatmap', colorscale=None) # other colorscale -'Greens', 'Reds', 'Blues'_____no_output_____# cufflinks scatter_matrix as an equivalent to Seaborn pairplot()

# Since all columns should be numerical values
df[['total_bill', 'tip', 'size']].scatter_matrix()_____no_output_____

x = [1, 2, 3]
y = [1000, 10000, 100000]
y2 = [5000, 10000, 90000]

trace1 = go.Bar(x=x, y=y, name='trace1')
trace2 = go.Bar(x=x, y=y2, name='trace2', visible=False)

data = [trace1, trace2]

updatemenus = list([
    dict(active=1,
         buttons=list([
            dict(label='Log Scale',
                 method='update',
                 args=[{'visible': [True, True]},
                       {'title': 'Log scale',
                        'yaxis': {'type': 'log'}}]),
            dict(label='Linear Scale',
                 method='update',
                 args=[{'visible': [True, True]},
                       {'title': 'Linear scale',
                        'yaxis': {'type': 'linear'}}])
            ]),
        )
    ])

layout = dict(updatemenus=updatemenus, title='Linear scale')
fig = go.Figure(data=data, layout=layout)

plotly.offline.iplot(fig)_____no_output_____


# Notebook from azbowler32/DS-Unit-1-Sprint-2-Data-Wrangling-and-Storytelling
Path: module3-make-explanatory-visualizations/LS_DS_123_Make_Explanatory_Visualizations.ipynb

_Lambda School Data Science_

# Make Explanatory Visualizations

### Objectives

- identify misleading visualizations and how to fix them
- use Seaborn to visualize distributions and relationships with continuous and discrete variables
- add emphasis and annotations to transform visualizations from exploratory to explanatory
- remove clutter from visualizations

### Links

- [How to Spot Visualization Lies](https://flowingdata.com/2017/02/09/how-to-spot-visualization-lies/)
- [Visual Vocabulary - Vega Edition](http://ft.com/vocabulary)
- [Choosing a Python Visualization Tool flowchart](http://pbpython.com/python-vis-flowchart.html)
- [Searborn example gallery](http://seaborn.pydata.org/examples/index.html) & [tutorial](http://seaborn.pydata.org/tutorial.html)
- [Strong Titles Are The Biggest Bang for Your Buck](http://stephanieevergreen.com/strong-titles/)
- [Remove to improve (the data-ink ratio)](https://www.darkhorseanalytics.com/blog/data-looks-better-naked)
- [How to Generate FiveThirtyEight Graphs in Python](https://www.dataquest.io/blog/making-538-plots/)_____no_output_____# Avoid Misleading Visualizations

Did you find/discuss any interesting misleading visualizations in your Walkie Talkie?_____no_output_____## What makes a visualization misleading?

[5 Ways Writers Use Misleading Graphs To Manipulate You](https://venngage.com/blog/misleading-graphs/)_____no_output_____## Two y-axes

 Other Examples:
 - [Spurious Correlations](https://tylervigen.com/spurious-correlations)
 - <https://blog.datawrapper.de/dualaxis/>
 - <https://kieranhealy.org/blog/archives/2016/01/16/two-y-axes/>
 - <http://www.storytellingwithdata.com/blog/2016/2/1/be-gone-dual-y-axis>_____no_output_____##  Y-axis doesn't start at zero.

_____no_output_____## Pie Charts are bad

_____no_output_____## Pie charts that omit data are extra bad

- A guy makes a misleading chart that goes viral

 What does this chart imply at first glance? You don't want your user to have to do a lot of work in order to be able to interpret you graph correctly. You want that first-glance conclusions to be the correct ones.

 <https://twitter.com/michaelbatnick/status/1019680856837849090?lang=en>

- It gets picked up by overworked journalists (assuming incompetency before malice)

 <https://www.marketwatch.com/story/this-1-chart-puts-mega-techs-trillions-of-market-value-into-eye-popping-perspective-2018-07-18>

- Even after the chart's implications have been refuted, it's hard a bad (although compelling) visualization from being passed around.

 <https://www.linkedin.com/pulse/good-bad-pie-charts-karthik-shashidhar/>

**["yea I understand a pie chart was probably not the best choice to present this data."](https://twitter.com/michaelbatnick/status/1037036440494985216)**_____no_output_____## Pie Charts that compare unrelated things are next-level extra bad

_____no_output_____## Be careful about how you use volume to represent quantities:

radius vs diameter vs volume

_____no_output_____## Don't cherrypick timelines or specific subsets of your data:

Look how specifically the writer has selected what years to show in the legend on the right side.

<https://wattsupwiththat.com/2019/02/24/strong-arctic-sea-ice-growth-this-year/>

Try the tool that was used to make the graphic for yourself

<http://nsidc.org/arcticseaicenews/charctic-interactive-sea-ice-graph/>
                                                                                _____no_output_____## Use Relative units rather than Absolute Units

_____no_output_____## Avoid 3D graphs unless having the extra dimension is effective

Usually you can Split 3D graphs into multiple 2D graphs

3D graphs that are interactive can be very cool. (See Plotly and Bokeh)

_____no_output_____## Don't go against typical conventions

_____no_output_____# Tips for choosing an appropriate visualization:_____no_output_____## Use Appropriate "Visual Vocabulary"

[Visual Vocabulary - Vega Edition](http://ft.com/vocabulary)_____no_output_____## What are the properties of your data?
- Is your primary variable of interest continuous or discrete?
- Is in wide or long (tidy) format?
- Does your visualization involve multiple variables?
- How many dimensions do you need to include on your plot?

Can you express the main idea of your visualization in a single sentence?

How hard does your visualization make the user work in order to draw the intended conclusion?_____no_output_____## Which Visualization tool is most appropriate?

[Choosing a Python Visualization Tool flowchart](http://pbpython.com/python-vis-flowchart.html)_____no_output_____## Anatomy of a Matplotlib Plot_____no_output_____

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import AutoMinorLocator, MultipleLocator, FuncFormatter

np.random.seed(19680801)

X = np.linspace(0.5, 3.5, 100)
Y1 = 3+np.cos(X)
Y2 = 1+np.cos(1+X/0.75)/2
Y3 = np.random.uniform(Y1, Y2, len(X))

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(1, 1, 1, aspect=1)

def minor_tick(x, pos):
    if not x % 1.0:
        return ""
    return "%.2f" % x

ax.xaxis.set_major_locator(MultipleLocator(1.000))
ax.xaxis.set_minor_locator(AutoMinorLocator(4))
ax.yaxis.set_major_locator(MultipleLocator(1.000))
ax.yaxis.set_minor_locator(AutoMinorLocator(4))
ax.xaxis.set_minor_formatter(FuncFormatter(minor_tick))

ax.set_xlim(0, 4)
ax.set_ylim(0, 4)

ax.tick_params(which='major', width=1.0)
ax.tick_params(which='major', length=10)
ax.tick_params(which='minor', width=1.0, labelsize=10)
ax.tick_params(which='minor', length=5, labelsize=10, labelcolor='0.25')

ax.grid(linestyle="--", linewidth=0.5, color='.25', zorder=-10)

ax.plot(X, Y1, c=(0.25, 0.25, 1.00), lw=2, label="Blue signal", zorder=10)
ax.plot(X, Y2, c=(1.00, 0.25, 0.25), lw=2, label="Red signal")
ax.plot(X, Y3, linewidth=0,
        marker='o', markerfacecolor='w', markeredgecolor='k')

ax.set_title("Anatomy of a figure", fontsize=20, verticalalignment='bottom')
ax.set_xlabel("X axis label")
ax.set_ylabel("Y axis label")

ax.legend()

def circle(x, y, radius=0.15):
    from matplotlib.patches import Circle
    from matplotlib.patheffects import withStroke
    circle = Circle((x, y), radius, clip_on=False, zorder=10, linewidth=1,
                    edgecolor='black', facecolor=(0, 0, 0, .0125),
                    path_effects=[withStroke(linewidth=5, foreground='w')])
    ax.add_artist(circle)

def text(x, y, text):
    ax.text(x, y, text, backgroundcolor="white",
            ha='center', va='top', weight='bold', color='blue')

# Minor tick
circle(0.50, -0.10)
text(0.50, -0.32, "Minor tick label")

# Major tick
circle(-0.03, 4.00)
text(0.03, 3.80, "Major tick")

# Minor tick
circle(0.00, 3.50)
text(0.00, 3.30, "Minor tick")

# Major tick label
circle(-0.15, 3.00)
text(-0.15, 2.80, "Major tick label")

# X Label
circle(1.80, -0.27)
text(1.80, -0.45, "X axis label")

# Y Label
circle(-0.27, 1.80)
text(-0.27, 1.6, "Y axis label")

# Title
circle(1.60, 4.13)
text(1.60, 3.93, "Title")

# Blue plot
circle(1.75, 2.80)
text(1.75, 2.60, "Line\n(line plot)")

# Red plot
circle(1.20, 0.60)
text(1.20, 0.40, "Line\n(line plot)")

# Scatter plot
circle(3.20, 1.75)
text(3.20, 1.55, "Markers\n(scatter plot)")

# Grid
circle(3.00, 3.00)
text(3.00, 2.80, "Grid")

# Legend
circle(3.70, 3.80)
text(3.70, 3.60, "Legend")

# Axes
circle(0.5, 0.5)
text(0.5, 0.3, "Axes")

# Figure
circle(-0.3, 0.65)
text(-0.3, 0.45, "Figure")

color = 'blue'
ax.annotate('Spines', xy=(4.0, 0.35), xytext=(3.3, 0.5),
            weight='bold', color=color,
            arrowprops=dict(arrowstyle='->',
                            connectionstyle="arc3",
                            color=color))

ax.annotate('', xy=(3.15, 0.0), xytext=(3.45, 0.45),
            weight='bold', color=color,
            arrowprops=dict(arrowstyle='->',
                            connectionstyle="arc3",
                            color=color))

ax.text(4.0, -0.4, "Made with http://matplotlib.org",
        fontsize=10, ha="right", color='.5')

plt.show()_____no_output_____

## Simple Web Scraper with IMDB_____no_output_____

from requests import get

url = "https://www.imdb.com/title/tt6105098/ratings?ref_=tt_ov_rt"
response = get(url)
print(response.text[:500])

<!DOCTYPE html>

        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">

    <meta name="apple-itunes-app" content="app-id=342792525, app-argument=imdb:///title/tt6105098?src=mdot">


# Notebook from WomensCodingCircle/CodingCirclePython
Path: Lesson07_ListsandTuples/Lists and Tuples - after class.ipynb

# Lists and Tuples

## Lists Recap

A list is a sequence of values. These values can be anything: strings, numbers, booleans, even other lists.

To make a list you put the items separated by commas between brackets []_____no_output_____

sushi_order = ['unagi', 'hamachi', 'otoro']
prices = [6.50, 5.50, 15.75]
print(sushi_order)
print(prices)['unagi', 'hamachi', 'otoro']
[6.5, 5.5, 15.75]

You can access a single element in a list by indexing in using brackets. List indexing starts at 0 so to get the first element, you use 0, the second element is 1 and so on.

list[index]
_____no_output_____

print(sushi_order[0])
print(sushi_order[2])unagi
otoro

You can find the length of a list using len

len(list)_____no_output_____

print(len(sushi_order))3

You can use negative indexing to get the last element of a list_____no_output_____

print(sushi_order[-3])unagi

## Nested lists

Lists can contain other lists as elements.
This is a convenient alternative to a matrix. You can arrange lists of varying lengths (and contents) in a specific order and you can iterate over the elements (see below)._____no_output_____

everyones_order = [['california roll'], ['unagi', 'dragon roll'], sushi_order]
print(everyones_order)[['california roll'], ['unagi', 'dragon roll'], ['unagi', 'hamachi', 'otoro']]

To access an element in a nested list, first index to the inner list, then index to the item.

Example:
```
    list_of_lists = [[1,2], [3,4], []]
```
Acess the first index to the inner list and index to the item
```python
    inner_list = list_of_lists[1] # [3,4]
    print inner_list[0] # 3
```

Or even quicker:
```python
    list_of_lists[1][0] # 3
```_____no_output_____### TRY IT_____no_output_____1) To get dragon roll from the sushi order, first we get the second element (index 1) then we get the the second item (index 1)_____no_output_____2) Print california roll from the list `everyones_order`:_____no_output_____3) Print all items from the second person's order_____no_output_____## Mutable Lists

Lists are mutable, that means that you can change elements.

To assign a new value to an element

    my_list = [1, 2, 3]
    my_list[0] = 100_____no_output_____

sushi_order[0] = 'caterpillar roll'
print(sushi_order)['caterpillar roll', 'hamachi', 'otoro']

### TRY IT
Update the last element in `prices` to be 21.00 and print out the new result_____no_output_____

prices[-1] = 21.00
print(prices)[6.5, 5.5, 21.0]

## Operators and Lists

The `in` operator allows you to see if an element is contained in a list_____no_output_____

sushi_order_____no_output_____print(('hamachi' in sushi_order))True
if 'otoro' in sushi_order:
    print("Big spender!")Big spender!

You can use some arithmatic operators on lists

The `+` operator concatenates two lists

The `*` operator duplicates a list that many times_____no_output_____

print((sushi_order * 3))['caterpillar roll', 'hamachi', 'otoro', 'caterpillar roll', 'hamachi', 'otoro', 'caterpillar roll', 'hamachi', 'otoro']
exprep = ['rep'+str(i) for i in range(5)]
exprep_____no_output_____print((prices + sushi_order))[6.5, 5.5, 15.75, 'unagi', 'hamachi', 'otoro']

Note: You can only concatenate lists with lists! If you want to add a "non-list" element you can use the append() function._____no_output_____

newprices = prices.copy()
newprices.append(22)
print(newprices)[6.5, 5.5, 15.75, 22]
prices_____no_output_____

 Remember slices from strings? We can also use the slice operator on lists_____no_output_____

inexpensive = sushi_order[:2] #takes only the first two elements from list
print(inexpensive)['unagi', 'hamachi']

Don't forget, you can use the `for` and `in` keywords to loop through a list_____no_output_____

for item in sushi_order:
    print(("I'd like to order the {}.".format(item)))

print("And hold the wasabi!")I'd like to order the unagi.
I'd like to order the hamachi.
I'd like to order the otoro.
And hold the wasabi!
for ind, item in enumerate(sushi_order):
    print(("I'd like to order the {0} for {1}.".format(item, prices[ind])))I'd like to order the unagi for 6.5.
I'd like to order the hamachi for 5.5.
I'd like to order the otoro for 15.75.

### TRY IT
Create a variable called `lots_of_sushi` that repeats the inexpensive list two times_____no_output_____

lots_of_sushi = inexpensive*2
print(lots_of_sushi)['unagi', 'hamachi', 'unagi', 'hamachi']

## Adding and deleting elements

To add an element to a list, you have a few options

1. the append method adds an element or elements to the end of a list, if you pass it a list, the next element with be a list (making a list of lists)

2. the extend method takes a list of elements and adds them all to the end, not creating a list of lists

3. use the `+` operator like you saw before_____no_output_____

my_sushis = ['maguro', 'rock n roll']
my_sushis.append('avocado roll')
print(my_sushis)
my_sushis.append(['hamachi', 'california roll'])
print(my_sushis)['maguro', 'rock n roll', 'avocado roll']
['maguro', 'rock n roll', 'avocado roll', ['hamachi', 'california roll']]
my_sushis = ['maguro', 'rock n roll']
my_sushis.extend(['hamachi', 'california roll'])
print(my_sushis)['maguro', 'rock n roll', 'hamachi', 'california roll']

You also have several options for removing elements

1. the pop method takes the index of the element to remove and returns the value of the element

2. the remove method takes the value of the element to remove

3. the del operator deletes the element or slice of the list that you give it

    del l[1:]_____no_output_____

print(my_sushis)
last_sushi = my_sushis.pop(-1)
print(last_sushi)['maguro', 'rock n roll', 'hamachi', 'california roll']
california roll
my_sushis.remove('maguro')
print(my_sushis)['rock n roll', 'hamachi']
del my_sushis[1:]
print(my_sushis)['rock n roll']

### TRY IT
Add 'rock n roll' to `sushi_order` then delete the first element of `sushi_order`_____no_output_____## List Functions

`max` will return maximum value of list

`min` returns minimum value of list

`sum` returns the sum of the values in a list

`len` returns the number of elements in a list # Just a reminder_____no_output_____

numbers = [1, 1, 2, 3, 5, 8]

print((max(numbers)))
print((min(numbers)))
print((sum(numbers)))
print((len(numbers)))

8
1
20
6

### TRY IT
Find the average of `numbers` using list functions (and not a loop!)_____no_output_____

sum(numbers)/len(numbers)_____no_output_____

## Aliasing

If you assign a list to another variable, it will still refer to the same list. This can cause trouble if you change one list because the other will change too._____no_output_____

cooked_rolls = ['unagi roll', 'shrimp tempura roll']
my_order = cooked_rolls
my_order.append('hamachi')
print(my_order)
print(cooked_rolls)['unagi roll', 'shrimp tempura roll', 'hamachi']
['unagi roll', 'shrimp tempura roll', 'hamachi']

To check this, you can use the `is` operator to see if both variable refer to the same object_____no_output_____

print((my_order is cooked_rolls))True

To fix this, you can make a copy of the list using the list function

`list` takes a sequence and turns it into a list.

Alternatively you can use the `copy()` method: `my_order = cooked_rolls.copy()`_____no_output_____

cooked_rolls = ['unagi roll', 'shrimp tempura roll']
my_order = list(cooked_rolls)
my_order.append('hamachi')
print(my_order)
print(cooked_rolls)['unagi roll', 'shrimp tempura roll', 'hamachi']
['unagi roll', 'shrimp tempura roll']

## Tuples

Tuples are very similar to lists. The major difference is that tuples are immutable meaning that you can not add, remove, or assign new values to a tuple.

The creator of a tuple is the comma `,` but by convention people usually surround tuples with parenthesis._____no_output_____

noodles = ('soba', 'udon', 'ramen', 'lo mein', 'somen', 'rice noodle')
print((type(noodles)))<class 'tuple'>

You can create a tuple from any sequence using the `tuple` function_____no_output_____

sushi_tuple = tuple(my_order)
print(sushi_tuple)
# Remember strings are sequences
maguro = tuple('maguro')
print(maguro)('unagi roll', 'shrimp tempura roll', 'hamachi')
('m', 'a', 'g', 'u', 'r', 'o')

To create a single element tuple, you need to add a comma to the end of that element (it looks kinda weird)_____no_output_____

single_element_tuple = (1,)
print(single_element_tuple)
print((type(single_element_tuple)))(1,)
<class 'tuple'>

You can use the indexing and slicing you learned for lists the same with tuples.

But, because tuples are immutable, you cannot use the append, pop, del, extend, or remove methods or even assign new values to indexes_____no_output_____

print((noodles[0]))
print((noodles[4:]))soba
('somen', 'rice noodle')
# This should throw an error
noodles[0] = 'spaghetti'_____no_output_____

To change the values in a tuple, you need to create a new tuple (there is nothing stopping you from assigning it to the same variable, though_____no_output_____

print(sushi_tuple)
sushi_tuple = sushi_tuple[1:] + ('california roll',)
print(sushi_tuple)('unagi roll', 'shrimp tempura roll', 'hamachi')
('shrimp tempura roll', 'hamachi', 'california roll')

You can loop through tuples the same way you  loop through lists, using `for` `in`_____no_output_____

for noodle in noodles:
    print(("Yummy, yummy {0} and {1}".format(noodle, 'sushi')))Yummy, yummy soba and sushi
Yummy, yummy udon and sushi
Yummy, yummy ramen and sushi
Yummy, yummy lo mein and sushi
Yummy, yummy somen and sushi
Yummy, yummy rice noodle and sushi

### TRY IT
Create a tuple containing 'soy sauce' 'ginger' and 'wasabi' and save it in a variable called `accompaniments`_____no_output_____## Zip

the zip function takes any number of lists of the same length and returns a list of tuples where the tuples will contain the i-th element from each of the lists.

This is really useful when combining lists that are related (especially for looping)_____no_output_____

print((list(zip([1,2,3], [4,5,6]))))[(1, 4), (2, 5), (3, 6)]
sushi = ['salmon', 'tuna', 'sea urchin']
prices = [5.5, 6.75, 8]

sushi_and_prices = list(zip(sushi, prices))

sushi_and_prices_____no_output_____for sushi, price in sushi_and_prices:
    print(("The {0} costs ${1}".format(sushi, price)))The salmon costs $5.5
The tuna costs $6.75
The sea urchin costs $8

## Enumerate

While the zip function iterates over two lists, the built-in function enumerate loops through indices and elements of a list.  It returns a list of tuples containing the index and value of that element.

    for index, value in enumerate(list):
        ...
_____no_output_____

exotic_sushi = ['tako', 'toro', 'uni', 'hirame']
for index, item in enumerate(exotic_sushi):
    print((index, item))0 tako
1 toro
2 uni
3 hirame

# Project: Party Budget

You are tasked with writing budgeting software, but at this point, things are a mess. You have two files. `budget_prices.txt` has a list of costs for each item separated by new lines (\n). `budget_items.txt` has a list of the items that were bought. Luckily they are both in order. You need to write a program that will take the files and a value for the overall budget and print out the total spent and how close they are to reaching the budget. In step 2 you will create a new file where the items and prices are in the same document and there is a sum printed out at the end.

## Step 1
1. Create a function called `file_to_float_list` that takes in a file and returns a list containing a float for each line **Hint** Make sure to remove the newlines when casting as floats.
2. Store the budget of 2000.00 in a variable called `budget`.
3. Run file_to_float_list on budget_prices.txt and save the result in a variable called `prices`.
4. Calculate the `sum` of the prices array and store in a variable called `spent`.
5. Calculate the percentage of budget spent and store in a variable called `percent_spent`
6. Print out the results:

    Budget: 2000.00 Spent: (amt spent) Percentage Spent: (percent spent)

7. ** Bonus ** Print out a progress bar for the budget. Print out '=' for every 10% spent and '-' for every 10% unspent.  =====>-----

## Step 2
1. Create a function called `file_to_string_list` that takes in a file and returns a list containing a string for each line with newlines removed.
2. Run `file_to_string_list` on budget_items.txt and save the result in a variable called `stuff_bought`.
3. Zip `stuff_bought` and `prices` together and store in a variable called `items_and_prices`
4. Loop through `items_and_prices` and print out the item, then a tab character '\t' and then the price (use string formatting)
5. Print another line 'Sum\t(amount spent)'
6. Print a final line 'Budget\t(budget)'

7. **Bonus** Print everything you printed for step 2 into a new file. (Then open the file in excel.)_____no_output_____


# Notebook from AlainLich/COVID-Data
Path: JupySessions/Pop-Data-FromGouv.ipynb

# Analyze population data from www.data.gouv.fr and/or INSEE

**Note:** This is a Jupyter notebook which is also available as its executable export as a Python 3 script (therefore with automatically generated comments)._____no_output_____# Libraries_____no_output_____

# Sys import
import sys, os, re
# Common imports
import math
import numpy             as NP
import numpy.random      as RAND
import scipy.stats       as STATS
from scipy import sparse
from scipy import linalg

# Better formatting functions
from IPython.display import display, HTML
from IPython import get_ipython

import matplotlib        as MPL
import matplotlib.pyplot as PLT
import seaborn as SNS
SNS.set(font_scale=1)

# Python programming
from itertools import cycle
from time import time
import datetime

# Using pandas
import pandas as PAN
import xlrd_____no_output_____import warnings
warnings.filterwarnings('ignore')
print("For now, reduce python warnings, I will look into this later")For now, reduce python warnings, I will look into this later

### Import my own modules
The next cell attempts to give user some information if things improperly setup.
Intended to work both in Jupyter and when executing the Python file directly._____no_output_____

if not get_ipython() is None and os.path.abspath("../source/") not in sys.path:
    sys.path.append(os.path.abspath("../source/"))
try:
    from lib.utilities     import *
    from lib.figureHelpers import *
    from lib.DataMgrJSON   import *
    from lib.DataMgr       import *
    from lib.pandaUtils    import *
except Exception as err:
    print("Could not find library 'lib' with contents 'DataGouvFr' ")
    if get_ipython() is None:
        print("Check the PYTHONPATH environment variable which should point to 'source' wich contains 'lib'")
    else:
        print("You are supposed to be running in JupySessions, and '../source/lib' should exist")
    raise errLoaded lib
For now, reduce python warnings, I will look into this later

## Check environment

It is expected that:
- your working directory is named `JupySessions`,
- that it has subdirectories
   - `images/*` where generated images may be stored to avoid overcrowding.
- At the same level as your working dir there should be directories
   - `../data` for storing input data and
   - `../source` for python scripts.

My package library is in `../source/lib`, and users running under Python (not in Jupyter) should
set their PYTHONPATH to include "../source" ( *or whatever appropriate* )._____no_output_____

checkSetup(chap="Chap02")
ImgMgr = ImageMgr(chapdir="Chap02")_____no_output_____

# Load Data_____no_output_____## Functions_____no_output_____## Load CSV and XLSX data from remote
The `dataFileVMgr` will manage a cache of data files in `../dataPop`.

We check what is in the cache/data directory; for each file, we identify the latest version,
and list this below to make sure. The file name will usually contain a time stamp; this has to do with
the version management/identification technique used when downloading from www.data.gouv.fr or INSEE

For the files used in this notebook, the latest version is used/loaded irrespective of the
timestamp used in the notebook.

Consulted:
  - https://www.insee.fr/fr/information/2410988
  - https://www.insee.fr/fr/information/2008354
  - https://www.insee.fr/fr/statistiques/4265429?sommaire=4265511#consulter
  - https://www.insee.fr/fr/statistiques/4265429?sommaire=4265511#documentation
  - https://api.insee.fr/catalogue/
  - https://api.insee.fr/catalogue/site/themes/wso2/subthemes/insee/pages/help.jag

Downloaded:

  - https://www.insee.fr/fr/statistiques/fichier/1893198/estim-pop-dep-sexe-gca-1975-2020.xls
  - https://www.insee.fr/fr/statistiques/fichier/1893198/estim-pop-dep-sexe-aq-1975-2020.xls
  - https://www.insee.fr/fr/statistiques/fichier/1893198/evolution-population-dep-2007-2020.xls
  - https://www.insee.fr/fr/statistiques/fichier/1893198/estim-pop-nreg-sexe-gca-1975-2020.xls
  - https://www.insee.fr/fr/statistiques/fichier/1893198/estim-pop-nreg-sexe-aq-1975-2020.xls
  - https://www.insee.fr/fr/statistiques/fichier/1893198/evolution-population-nreg-2007-2020.xls_____no_output_____

dataFileVMgr = manageAndCacheFilesDFrame("../dataPop")In cacheSpaceRecovery, looking for 36183366.0 bytes
Printing genDir collected for preparing Cache Recovery
Recovered 0 from cache, which is insufficient required:36183366.0
print("Most recent versions of files in data directory:")
for f in dataFileVMgr.listMostRecent() :
    print(f"\t{f}")_____no_output_____last = lambda x: dataFileVMgr.getRecentVersion(x,default=True)_____no_output_____

This ensures we load the most recent version, so that it is not required to update the list
below. The timestamps shown in the following sequence will be update by the call to `getRecentVersion`._____no_output_____

exportDSCsv    = last("export-dataset-20200314-064905.csv")
exportORGCsv    = last("export-organization-20200314-065744.csv")
tagsCsv        = last("tags-2020-04-20-09-22.csv")
_____no_output_____S1 = set (dataFileVMgr.listMostRecent())
S2 =set((exportDSCsv ,tagsCsv,  exportORGCsv ))
missing = S1. difference(S2)
if len(missing) > 0:
    print (f"Missing comparing with most recent files in ../data:")
    for f in missing:
        print(f"\t{f}")_____no_output_____metaCodeTranchesAgeCSV ="../dataPop/code-tranches-dage.csv"

estimPopDepSxXLS      ="../dataPop/estim-pop-dep-sexe-aq-1975-2020.xls"
estimPopDepSxGXLS     ="../dataPop/estim-pop-dep-sexe-gca-1975-2020.xls"
estimPopRegSxXLS      ="../dataPop/estim-pop-nreg-sexe-aq-1975-2020.xls"
estimPopRegSxGXLS     ="../dataPop/estim-pop-nreg-sexe-gca-1975-2020.xls"
evolPopDepXLS         ="../dataPop/evolution-population-dep-2007-2020.xls"
evolPopRegXLS         ="../dataPop/evolution-population-nreg-2007-2020.xls"

inseeDepXLS           ="../dataPop/InseeDep.xls"_____no_output_____

Now load the stuff_____no_output_____

ad  = lambda x: "../dataPop/"+x
data_exportDS = read_csvPandas(ad(exportDSCsv), error_bad_lines=False,sep=";" )
data_tags     = read_csvPandas(ad(tagsCsv), error_bad_lines=False,sep=";")
data_exportORG= read_csvPandas(ad(exportORGCsv), error_bad_lines=False,sep=";")_____no_output_____metaCodeTranchesAge = read_csvPandas(metaCodeTranchesAgeCSV, error_bad_lines=False,sep=";")
estimPopDepSx       = read_xlsxPandas(estimPopDepSxXLS, error_bad_lines=False,sep=",")
estimPopDepSxG      = read_xlsxPandas(estimPopDepSxGXLS, error_bad_lines=False,sep=",")
estimPopRegSxX      = read_xlsxPandas(estimPopRegSxXLS , error_bad_lines=False,sep=",")
estimPopRegSxG      = read_xlsxPandas(estimPopRegSxGXLS, error_bad_lines=False,sep=",")
evolPopDep          = read_xlsxPandas(evolPopDepXLS , error_bad_lines=False,sep=",")
evolPopReg          = read_xlsxPandas(evolPopRegXLS , error_bad_lines=False,sep=",")

inseeDep            = read_xlsxPandas(inseeDepXLS, error_bad_lines=False,sep=",", sheet_name=1, header=7)
inseeReg            = read_xlsxPandas(inseeDepXLS, error_bad_lines=False,sep=",", sheet_name=0, header=7)_____no_output_____

Show the shape of the loaded data:_____no_output_____

def showBasics(data,dataName):
    print(f"{dataName:24}\thas shape {data.shape}")

dataListDescr = ( (data_exportDS, "data_exportDS"),
                (data_tags, "data_tags"),
                (data_exportORG,"data_exportORG"),
                (metaCodeTranchesAge,"metaCodeTranchesAge"),
                (estimPopDepSx ,"estimPopDepSx "),
                (estimPopDepSxG,"estimPopDepSxG"),
                (estimPopRegSxX,"estimPopRegSxX"),
                (estimPopRegSxG,"estimPopRegSxG"),
                (evolPopDep,"evolPopDep"),
                (evolPopReg,"evolPopReg"),
                (inseeDep,"inseeDep"),
                (inseeReg,"inseeReg")
                )

for (dat,name) in dataListDescr:
    showBasics(dat,name)
_____no_output_____for (dat,name) in dataListDescr:
    if name[0:5]=="meta_": continue
    print(f"\nDescription of data in '{name}'\n")
    display(dat.describe().transpose())
    _____no_output_____

## Explore the INSEE export-dataset-\*.csv

### Use utilities to find rows corresponding to regexp_____no_output_____

from IPython.core.display import display, HTML
display(HTML(""))_____no_output_____data_exportDS.loc[:2].values_____no_output_____

This selects rows according to a *regular expression* (compliant with the python `re` module)_____no_output_____

df1 =  DfMatcher(data_exportDS)
df1.showMatch('description','stat.*population.*')
df1.prMatch('description','stat.*population.*')_____no_output_____

### Now look at population statistics
#### Landed at the wrong place
I like the atlantic seaside, but this was not the target!_____no_output_____

interesting = df1.rowMatch('slug','recensement.*population-population$')
interesting.loc[:, ['url', 'title']].values
interesting.values_____no_output_____

## Explore population from INSEE files
See above for the files and how they may be found. _____no_output_____

evolPopDep          _____no_output_____

Comments to be exploited later:
- evolPopReg
- estimPopbeDepSx
- estimPopDepSxG
- estimPopRegSxX
- estimPopRegSxG      _____no_output_____

metaCodeTranchesAge  _____no_output_____inseeReg_____no_output_____inseeDep_____no_output_____inseeDep.iloc[:,4:].sum()_____no_output_____inseeReg.iloc[:,4:].sum()_____no_output_____

## Merging demographic data with Covid hospital or testing statistics
This should give a different perspective on those numbers._____no_output_____


# Notebook from Hoseung/pyRamAn
Path: scripts/notebooks/halo/plot_halos-Copy1.ipynb

### Halo check
Plot halos to see if halofinders work well_____no_output_____

<code>
#import os
#base = os.path.abspath('/home/hoseung/Work/data/05427/')

#base = base + '/'

# basic parameters
wdir = '/home/hoseung/Work/data/05427/'
cluster_name = wdir.split('/')[-2]

frefine= 'refine_params.txt'
fnml = 'cosmo_200.nml'
nout_ini=186
nout_fi=187
nouts = range(nout_ini,nout_fi+1)

scale = 0.10
npix = 800

m_halo_min = 5e9

# data loading parameters
ptype=["star pos mass"]
refine_params = True
dmo=False
draw=True
draw_halos=True
draw_part = True
draw_hydro = False
if draw_hydro:
    lmax=19_____no_output_____import load
import utils.sampling as smp
import utils.match as mtc
import draw
import pickle

for nout in nouts:
    snout = str(nout).zfill(3)

    if refine_params:
        # instead of calculating zoomin region, just load it from the refine_params.txt file.
        # region = s.part.search_zoomin(scale=0.5, load=True)

        rr = load.info.RefineParam()
        rr.loadRegion(wdir + frefine)

        nn = load.info.Nml(fname=wdir + fnml)

        aexp = nn.aout[nout-1]
        i_aexp = mtc.closest(aexp, rr.aexp)

        x_refine = rr.x_refine[i_aexp]
        y_refine = rr.y_refine[i_aexp]
        z_refine = rr.z_refine[i_aexp]
        r_refine = rr.r_refine[i_aexp] * 0.5

        region = smp.set_region(xc = x_refine, yc = y_refine, zc = z_refine,
                                radius = r_refine * scale)
    else:
        region = smp.set_region(xc=0.5, yc=0.5, zc=0.5, radius=0.1)

    print(region)
    s = load.sim.Sim(nout, wdir, dmo=dmo, ranges=region["ranges"], setup=True)
    imgs = draw.img_obj.MapSet(info=s.info, region=region)
    imgp = draw.img_obj.MapImg(info=s.info, proj='z', npix=npix, ptype=ptype)
    imgp.set_region(region)

    #%%
    if draw_part:
        s.add_part(ptype)
        s.part.load()
        part = getattr(s.part, s.part.pt[0])

        x = part['x']
        y = part['y']
        z = part['z']
        m = part['m'] * s.info.msun # part must be normalized already!

        #imgp.set_data(draw.pp.den2d(x, y, z, m, npix, s.info, cic=True, norm_integer=True))
        imgp.set_data(draw.pp.den2d(x, y, z, m, npix, region, cic=True, norm_integer=True))
        imgs.ptden2d = imgp
#    imgp.show_data()

    #%%
    if draw_hydro:
        s.add_hydro()
        s.hydro.amr2cell(lmax=lmax)
#        field = draw.pp.pp_cell(s.hydro.cell, npix, s.info, verbose=True)
        ptype = 'gas_den'
        imgh = draw.img_obj.MapImg(info=s.info, proj='z', npix=npix, ptype=ptype)
        imgh.set_data(draw.pp.pp_cell(s.hydro.cell, npix, s.info, verbose=True))
        imgh.set_region(region)
    #    imgh.show_data()
        imgs.hydro = imgh

    #%%
    fdump = wdir + snout + 'map.pickle'
    with open(fdump, 'wb') as f:
        pickle.dump(imgs, f)
    if draw:
        if draw_part:
            imgs.ptden2d.plot_2d_den(save=wdir + cluster_name + snout +'star.png',
                                     vmin=1e13, vmax=1e20, dpi=400, show=False)
        if draw_hydro:
            imgs.hydro.plot_2d_den(save=wdir + cluster_name +snout + 'hydro.png',
                                   vmax=15, vmin=10, show=False, dpi=400){'xr': [0.504942387092, 0.508942387092], 'yr': [0.310907055262, 0.314907055262], 'zc': 0.303108906833, 'radius': 0.002, 'yc': 0.312907055262, 'centers': [0.506942387092, 0.312907055262, 0.303108906833], 'xc': 0.506942387092, 'zr': [0.301108906833, 0.305108906833], 'ranges': [[0.504942387092, 0.508942387092], [0.310907055262, 0.314907055262], [0.301108906833, 0.305108906833]]}
Ranges = [[0.504942387092, 0.508942387092], [0.310907055262, 0.314907055262], [0.301108906833, 0.305108906833]]

No AMR instance,
Loading one...
An AMR instance is created

Updating info.cpus
An AMR instance is created

Updating info.cpus
 Simulation set up.
Updating data
Types of particles you want to load are:  ['star pos mass']
No AMR instance,
Loading one...
An AMR instance is created

Updating info.cpus
No info._set_cpus attribute??
A particle instance is created

Use part.load() to load particle
Loading particle...
 ranges: [[0.504942387092, 0.508942387092], [0.310907055262, 0.314907055262], [0.301108906833, 0.305108906833]]
npart_arr: [21722, 14847, 19518, 17132, 21256, 9394, 27007, 26537, 21641, 27645, 24015, 17073, 7606, 34677, 8870, 30942, 20329, 30404, 7629, 8131, 30725, 20468, 10113, 19699, 12102, 19664, 38313, 38244, 69874, 109368, 24489, 11307, 20466, 52467, 19660, 13548, 33961, 13332, 45313, 47917, 18428, 23815, 11196, 7319, 41786, 43866, 32716, 26223, 16488, 26614, 20561, 38232, 38293, 26603, 28724, 16690, 52383, 33255, 45442, 32104, 11932, 17118, 17921, 84654, 18490, 36028, 44233, 33678, 9302, 23399, 10536, 32452, 26232, 38902, 27886, 32312, 58857, 21297, 34302, 26282, 23167, 47750, 76078, 59090, 70214, 20422, 41138, 88852, 82494, 31957, 23277, 73332, 43356, 33305, 8980, 53595, 50352, 47958, 73418, 80016, 38782, 13865, 28989, 49620, 60289, 94562, 92784, 66015, 52305, 56445, 48663, 41591, 85647, 52880, 86496, 894, 4107, 44887, 60310, 22963, 33196, 17234, 37436, 44829, 93569, 128584, 4726, 4381, 14496, 25216, 32565, 91911, 28984, 17981, 27571, 69113, 35791, 2762, 46956, 64625, 64696, 3215, 4447, 39170, 28169, 22718, 6339, 43872, 40548, 41209, 31514, 7657, 34874, 1874, 42945, 32000, 43026, 17290, 41553, 28773, 48842, 31395, 32984, 15848, 43440, 53971, 68887, 39082, 53564, 34854, 38711, 29219, 43339, 50797, 30328, 19171, 23300, 24489, 31471, 22893, 22363, 3922, 20825, 9516, 10737, 21872, 18589, 1100, 34399, 48714, 44292, 37897, 20593, 10549, 43452, 42166, 23834, 17400, 24698, 38343, 38517, 25552, 16764, 28588, 23986, 21962, 25219, 18429, 18746, 15464, 18246, 53176, 11873, 34195, 13824, 34217, 36625, 26595, 29859, 23787, 14560, 29480, 18774, 10195, 59160, 26622, 36086, 54378, 40566, 47629, 39171, 30037, 46922, 48545, 65365, 58790, 69937, 25395, 68465, 35501, 66447, 31385, 50651, 25449, 10812, 20569, 35861, 26726, 15062, 30619, 53629, 37574, 27291, 8216, 17065, 61571, 20034, 62319, 60138, 57448, 30817, 23710, 13328, 49182, 53300, 74505, 81985, 52126, 29145, 24000, 39309, 57308, 30143, 41471, 58924, 68170, 85980, 45847, 58099, 62767, 14864, 13621, 46571, 66094, 84409, 57987, 91566, 45780, 55043, 53591, 29607, 13157, 326, 3101, 96341, 47163, 34352, 31671, 47208, 16953, 11941, 14103, 25071, 52109, 43121, 45686, 5310, 40943, 26962, 28618, 50700, 20850, 6761, 46515, 18665, 36488, 45376, 47810, 49104, 9329, 51838, 31268, 17730, 45549, 34136, 57291, 21198, 24641, 37536, 37781, 27909, 7314, 44860, 49944, 50581, 35038, 63620, 8057, 45088, 16808, 11595, 16316, 67988, 26044, 41473, 28600, 18396, 37386, 13051, 24468, 29761, 45706, 22299, 15762, 47005, 17895, 13167, 11750, 14427, 12007, 21804, 27693, 48735, 19316, 11816, 18551, 5868, 15858, 25487, 20857, 7633, 11421, 5150, 577, 10568, 78677, 19543, 19636, 46467, 14042, 12867, 41928, 56219, 51409, 39481, 21861, 21116, 14889, 19717, 26649, 20875, 9941, 20052, 3511, 27773, 23280, 17796, 39632, 35024, 11286, 14843, 19374, 25490, 12914, 45499, 40331, 58693, 20762, 1991, 15542, 29187, 40573, 48939, 10857, 27722, 22813]
Total DM particle 1321212
Total star particle 6185353
Total sink particle 691330
Resampling 6185353 values to a 800 by 800 by 1 grid
Updating field vals
Updating field vals
Updating field vals
Updating field vals
minmax field after crop and converting into physical unit 13637827490.0 9.29725271432e+19
Updating data
{'xr': [0.504934514119, 0.508934514119], 'yr': [0.310903416014, 0.314903416014], 'zc': 0.303048251159, 'radius': 0.002, 'yc': 0.312903416014, 'centers': [0.506934514119, 0.312903416014, 0.303048251159], 'xc': 0.506934514119, 'zr': [0.301048251159, 0.30504825115900003], 'ranges': [[0.504934514119, 0.508934514119], [0.310903416014, 0.314903416014], [0.301048251159, 0.30504825115900003]]}
Ranges = [[0.504934514119, 0.508934514119], [0.310903416014, 0.314903416014], [0.301048251159, 0.30504825115900003]]

No AMR instance,
Loading one...
An AMR instance is created

Updating info.cpus
An AMR instance is created

Updating info.cpus
 Simulation set up.
Updating data
Types of particles you want to load are:  ['star pos mass']
No AMR instance,
Loading one...
An AMR instance is created

Updating info.cpus
No info._set_cpus attribute??
A particle instance is created

Use part.load() to load particle
Loading particle...
 ranges: [[0.504934514119, 0.508934514119], [0.310903416014, 0.314903416014], [0.301048251159, 0.30504825115900003]]
npart_arr: [30029, 5473, 12715, 1961, 47639, 6427, 30292, 26351, 18126, 21381, 20556, 19, 51, 52090, 34032, 17226, 14939, 24185, 2555, 36737, 19908, 9998, 9500, 17599, 13436, 40689, 44076, 1334, 77053, 200630, 1096, 1346, 984, 19073, 57584, 164, 10299, 13597, 29838, 35091, 41012, 24876, 18471, 11241, 2459, 19193, 52175, 33826, 32073, 441, 38781, 24880, 62655, 33614, 34206, 2549, 12186, 13978, 137445, 119400, 67828, 4395, 16440, 23065, 27535, 4749, 5421, 7907, 38712, 2451, 32152, 50707, 11636, 62901, 15087, 16121, 55727, 21391, 351, 20163, 69567, 53552, 75208, 77200, 210116, 67214, 95979, 25420, 6583, 44881, 20951, 25097, 37702, 24727, 3123, 15781, 45978, 38321, 26177, 65304, 149334, 311548, 112657, 13084, 1096, 10146, 22410, 2715, 319, 508, 2085, 51499, 55274, 26750, 52125, 11758, 23728, 37468, 15337, 17774, 8261, 18884, 32532, 108848, 62612, 62173, 61919, 24819, 57248, 1956, 276, 35260, 18526, 63038, 80537, 49380, 11111, 1782, 216, 152071, 34986, 226, 187, 82065, 35808, 10956, 9959, 26675, 101134, 1336, 5794, 22242, 6633, 26141, 65404, 6668, 41222, 18891, 45111, 20026, 88502, 47644, 25315, 20410, 79369, 35635, 1558, 131499, 66475, 46643, 12375, 19618, 39947, 30066, 49349, 42211, 54343, 54314, 19883, 24331, 37685, 19746, 20587, 17633, 5182, 20044, 31656, 12697, 178, 4219, 3623, 584, 102564, 19574, 26836, 32011, 21490, 43673, 33010, 28251, 24718, 21517, 17084, 36399, 445, 26075, 41524, 48526, 20100, 9970, 30826, 17616, 19701, 14576, 22859, 36958, 30367, 4692, 18114, 28011, 43353, 16928, 32878, 28345, 12687, 25555, 21470, 8659, 20791, 14908, 26350, 57468, 78015, 34620, 21928, 64597, 46271, 1217, 31, 30267, 23400, 12077, 38117, 55972, 129753, 38056, 48714, 70101, 61247, 85205, 6090, 38, 51337, 33575, 12680, 25313, 53720, 38388, 32365, 11137, 48954, 29393, 38399, 39494, 37574, 80536, 55629, 8405, 44307, 10390, 72799, 58016, 48378, 117527, 29157, 21373, 5402, 15793, 43407, 18774, 12315, 23982, 32647, 23585, 88329, 93079, 97186, 76794, 39148, 48546, 56444, 56746, 70278, 111490, 51676, 31154, 63150, 40354, 47408, 25902, 43072, 21563, 2411, 705, 4199, 57093, 54841, 47111, 48406, 13811, 75363, 2368, 18917, 8466, 14436, 28827, 18700, 55710, 36241, 76639, 15974, 4034, 26488, 33424, 3325, 43507, 90064, 41255, 31471, 19771, 82242, 79, 7522, 14879, 72451, 78234, 54559, 48082, 15139, 81382, 24956, 8428, 394, 60455, 63886, 26068, 24062, 606, 46616, 48697, 8956, 20415, 8085, 39874, 34433, 49056, 28793, 17398, 9147, 11385, 12612, 29523, 42556, 31127, 15945, 18940, 11065, 18130, 20726, 13972, 6828, 13377, 57, 24, 3524, 93113, 3567, 14158, 13217, 66546, 404, 17115, 11942, 101130, 9517, 23666, 44494, 26547, 6097, 21546, 24782, 43018, 87, 8969, 19455, 28816, 18972, 19498, 123114, 26790, 29377, 3889, 2926, 26580, 2975, 29663, 44611, 10862, 16304, 94, 21075, 28520, 51378, 34263, 14948, 25351, 21051]
Total DM particle 1312831
Total star particle 6095534
Total sink particle 694267
Resampling 6095534 values to a 800 by 800 by 1 grid
Updating field vals
Updating field vals
Updating field vals
Updating field vals
minmax field after crop and converting into physical unit 36822533946.7 1.03225614572e+20
Updating data
import matplotlib.pyplot as plt
fig = plt.figure()
ax1 = fig.add_subplot(111)
snout = str(nout).zfill(3)

fin = wdir + snout + 'map.pickle'
with open(fin, 'rb') as f:
    img = pickle.load(f)

ptimg = img.ptden2d
fout = wdir + snout + "dmmap_" + ptimg.proj + ".png"
img.ptden2d.plot_2d_den(save=False, show=False, vmin=1e13, vmax=1e20, dpi=400, axes=ax1)

import tree
import numpy as  np

#s = load.sim.Sim(nout, base_dir)
info = load.info.Info(nout=nout, base=wdir, load=True)
hall = tree.halomodule.Halo(nout=nout, base=wdir, halofinder="HM", info=info, load=True)

i_center = np.where(hall.data['np'] == max(hall.data['np']))
h = tree.halomodule.Halo()
h.derive_from(hall, [i_center])

#region = smp.set_region(xc=h.data.x, yc=h.data.y, zc=h.data.z, radius = h.data.rvir * 2)

#%%
from draw import pp

ind = np.where(hall.data.mvir > 5e10)
h_sub = tree.halomodule.Halo()
h_sub.derive_from(hall, ind)
#x = hall.data.x#[ind]
#y = hall.data.y#[ind]
#r = hall.data.rvir#[ind]
#pp.circle_scatter(ax1, x*npix, y*npix, r*30, facecolor='none', edgecolor='b', label='555')

#ax1.set_xlim(right=npix).
#ax1.set_ylim(top=npix)
pp.pp_halo(h_sub, npix, region=img.ptden2d.region, axes=ax1, rscale=3, name=True)

plt.show()Are these parameters right?
Are these parameters right?

</code>

##### Load halofinder result

##### get position and virial radius

##### load particles data (star or DM) and draw density map
##### plot halos on top of particle density map_____no_output_____