# Notebook from ryneches/SuchTree
Path: docs/slides2.ipynb

# `SuchLinkedTrees`

I didn't want to write this either._____no_output_____### Working with linked trees

If you are interested in studying how two groups of organisms interact (or,
rather, have interacted over evolutionary time), you will find yourself with
two trees of distinct groups of taxa that are linked by a matrix of
interaction observations. This is sometimes called a 'dueling trees' problem.

![dueling trees](dueling_trees.png)_____no_output_____If the trees happen to have the same number of taxa, and the interaction
matrix happens to be a unit matrix, then you can compute the distance matrix
for each of your tres and use the
[Mantel test](https://en.wikipedia.org/wiki/Mantel_test) to compare them.
However, this is a pretty special case. Hommola *et al.* describe a method
extends the Mantel test in this paper here :

* [A Permutation Test of Host–Parasite Cospeciation](http://dx.doi.org/10.1093/molbev/msp062). *Molecular Biology and Evolution*, Vol. 26, No. 7. (01 July 2009), pp. 1457-1468, by Kerstin Hommola, Judith E. Smith, Yang Qiu, Walter R. Gilks_____no_output_____

%load_ext Cython
%pylab inlinePopulating the interactive namespace from numpy and matplotlib
from SuchTree import SuchTree
import pandas as pd
import numpy as np
import seaborn_____no_output_____from SuchTree import SuchLinkedTrees, pearson_____no_output_____T1 = SuchTree( 'SuchTree/tests/test.tree' )
T2 = SuchTree( 'http://edhar.genomecenter.ucdavis.edu/~russell/fishpoo/fishpoo2_p200_c2_unique_2_clustalo_fasttree.tree' )
links = pd.read_csv( 'http://edhar.genomecenter.ucdavis.edu/~russell/fishpoo/fishpoo2_p200_c2_host_count_table.tsv',
                    sep='\t', index_col='Host')
links.index = map( lambda x : x.replace(' ','_'), links.index )_____no_output_____

### Let your trees be linked!

Now, we create a `SuchLinkedTrees` object, which connects the host and
guest trees to the link matrix (`True` or `False` values derived from the
count matrix). This pre-indexes the table for fast access later, so creating
object takes a little while._____no_output_____

%%time
SLT = SuchLinkedTrees( T1, T2, links )139761682196456 allocating columns in 3446384640
bulding default subset.
bulding default link list.
CPU times: user 9min 41s, sys: 2.54 s, total: 9min 43s
Wall time: 10min

### What's goodies are inside?

Let's look at how `SuchLinkedTrees` slices the link data by clade.

We'll pick a random clade..._____no_output_____

SLT.TreeB.get_leafs( 7027 )_____no_output_____

This one looks good.

Now, we tell our `SuchLinkedTrees` object to subset itself using that clade.

The default subset is the root node of the guest tree._____no_output_____

SLT.subset( 7027 )
print 'subset size  :', SLT.subset_size
print 'subset links :', SLT.subset_n_links
print 'link pairs   :', ( SLT.subset_n_links * ( SLT.subset_n_links -1 ) ) / 2subset size  : 89
subset links : 245
link pairs   : 29890

print SLT.col_ids
print SLT.subset_columns
print SLT.subset_leafs_____no_output_____### Paired distances

We can test whether or not the guest organisms are likely to have co-diversified
with the host organisms by computing the distances between each possible pair of
links in the link matrix through each of the two trees, and then testing how well
those two sets of distances correlate with one another._____no_output_____

result = SLT.linked_distances()
seaborn.jointplot( result['TreeA'], result['TreeB'] )_____no_output_____

Yeesh. Lousy correlation, isn't it?

Anyway, thanks to `SuchTree`, it only takes a tiny sliver of
a second to compute the 29,890 distances through each of the two
trees._____no_output_____Unfortunately, for $n$ links, the number of link pairs is

$$\frac{n (n-1)}{2}$$

This is $\mathcal{O}(n^2)$ scaling, which is $bad$. The biggest clade is

$$ \frac{ 54327 \times 54326 }{2}= 1,475,684,301 $$

link pairs. And that's just one particularly big clade!_____no_output_____### Solution : sampling

We can avoid this problem (with some care) using `SuchLinkedTrees.sample_linked_distances()`._____no_output_____

result_sampled = SLT.sample_linked_distances(sigma=0.05, n=10000, buckets=10)
result_sampled_____no_output_____

Clade `7027` is not particularly big, so we've actually over-sampled it
by a little more than $10\times$.

It shoudln't much different, though._____no_output_____

seaborn.jointplot( result_sampled['TreeA'], result_sampled['TreeB'] )_____no_output_____

But, of couse, sampled distributions don't look exactly like the distributions
from which they were taken, so let's have a look at 'em._____no_output_____

seaborn.kdeplot(result['TreeB'])_____no_output_____seaborn.kdeplot(result_sampled['TreeB'][100000:400000])_____no_output_____

### Down to business

Now that you've seen the major features of `SuchLinkedTrees`, let's use it to do something
useful.

How about my main dissertation question?

*Are there any co-diversifying bacteria in my fish?*_____no_output_____

import pyprind

p = pyprind.ProgBar( len( list( SLT.TreeB.get_internal_nodes() ) ), monitor=True, title='sampling trees...' )

big_nodes = []
table = {}
for n,node in enumerate( SLT.TreeB.get_internal_nodes() ) :
    p.update()
    SLT.subset( node )
    if SLT.subset_n_links > 4000 :
        big_nodes.append( node )
        result = SLT.sample_linked_distances( sigma=0.05, n=1000, buckets=100)
    else :
        result = SLT.linked_distances()
    table[node] = { 'n_leafs'    : SLT.subset_size,
                    'n_links'    : SLT.subset_n_links,
                    'n_pairs'    : result['n_pairs'],
                    'n_samples'  : result['n_samples'],
                    'deviatnon_a': result['deviation_a'],
                    'deviation_b': result['deviation_b'],
                    'r'          : pearson( result['TreeA'], result['TreeB'] ) }sampling trees...
0%                          100%
[##############################] | ETA: 00:00:00
Total time elapsed: 06:00:03

If exhaustively measure correlations for clades with fewer than 4000 links,
and used sampled distances for clades with more than 4000 links, it takes
about six hours on one core.

It can be threaded, but I haven't gotten around to trying that yet._____no_output_____

C = pd.DataFrame( table ).T
seaborn.jointplot( 'n_links', 'r', data=C )_____no_output_____

Most of this is garbage. Let's focus on the good bits._____no_output_____

seaborn.jointplot( 'n_links', 'r', data=C.query('n_leafs > 5 and r > 0.05')  )_____no_output_____CC = C.query('n_leafs > 5 and r > 0.05').sort_values('r', ascending=False)
print CC.shape
CC.head()(398, 7)

## Correlation smorrilation

Well, the only reason I'm using Pearson's $r$ is because it happens to be an
$\mathcal{O}(n)$ algorithm. It also assumes that the distributions of the
things you are testing are normal.

We already saw that they are *not* normal.

We really should use a rank-order correlation test like Kendall's $\tau$
because they don't make any assumptions about the distributions, but
these are all $\mathcal{O}(n^2)$ algorithms.

Once again, the massive diversity of microbes forces us to really sweat the
details._____no_output_____So, let's just use Pearson's $r$ to out what the correlations *probably*
are, and then find Kendall's $\tau$ for the ones that look interesting._____no_output_____

from scipy.stats import kendalltau, pearsonr

pearson_p   = {}
kendall_tau = {}
kendall_p   = {}

for n,node in enumerate( CC.index ) :
    SLT.subset(node)
    result = SLT.linked_distances()
    p_r,p_p = pearsonr(   result['TreeA'], result['TreeB'] )
    k_t,k_p = kendalltau( result['TreeA'], result['TreeB'] )
    pearson_p[node]  = p_p
    kendall_tau[node] = k_t
    kendall_p[node]   = k_p

CC['pearson_p'] = pd.Series(pearson_p)
CC['kendall_tau'] = pd.Series(kendall_tau)
CC['kendall_p'] = pd.Series(kendall_p)
CC.head()_____no_output_____seaborn.jointplot( 'n_links', 'kendall_tau', data=CC )_____no_output_____

## The Moment of Truth

Well, there's one clade up there that has a $\tau$ of 0.394 with a $p$ of 0.040588.

Could this be... it?_____no_output_____

SLT.subset( 79047 )
result = SLT.linked_distances()
seaborn.jointplot( result['TreeA'], result['TreeB'] )_____no_output_____

## Verdict : Meh.

Does't look very convincing to me._____no_output_____


# Notebook from mthompson360/MSE672-Introduction-to-TEM
Path: Introduction/CH1_05-Course_Organization.ipynb

 **[MSE672: Introduction to TEM](https://gduscher.github.io/MSE672-Introduction-to-TEM/)**

# Course Organization

[Download](https://raw.githubusercontent.com/gduscher/MSE672-Introduction-to-TEM//main/Introduction/CH1_05-Course_Organization.ipynb)

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](
    https://colab.research.google.com/github/gduscher/MSE672-Introduction-to-TEM/blob/main/Introduction/CH1_05-Course_Organization.ipynb)

part of

 **[MSE672:  Introduction to Transmission Electron Microscopy](../_MSE672_Intro_TEM.ipynb)**

by

Gerd Duscher

Materials Science & Engineering

Joint Institute of Advanced Materials

The University of Tennessee, Knoxville_____no_output_____## Books
### Williams and Carter
**Transmission Electron Microscopy**

Plenum Press, 2009 (2nd Edition).

[available  online at the library](https://utk.primo.exlibrisgroup.com/discovery/fulldisplay?docid=cdi_askewsholts_vlebooks_9780387765013&context=PC&vid=01UTN_KNOXVILLE:01UTK&lang=en&search_scope=MyInst_and_CI&adaptor=Primo%20Central&tab=Everything&query=any%2Ccontains%2C%20Transmission%20Electron%20Microscopy&facet=rtype%2Cexclude%2Creviews&facet=rtype%2Cexclude%2Creference_entrys)

### Reimer and Kohl
**Transmission Electron Microscopy**

Springer, Berlin, 2007 (5th Edition).

[available  online at the library](https://utk.primo.exlibrisgroup.com/discovery/fulldisplay?docid=cdi_proquest_ebookcentral_EBC6362086&context=PC&vid=01UTN_KNOXVILLE:01UTK&lang=en&search_scope=MyInst_and_CI&adaptor=Primo%20Central&tab=Everything&query=any%2Ccontains%2CReimer%20and%20Kohl&offset=0)

### Zuo and Spence

**Advanced Transmission Electron Microscopy: Imaging and Diffraction**

Springer, NewYork, 2017

available online at the Library

### Kirkland
**Advanced Computing in Electron Microscopy**

Springer, NewYork, 2010 (2nd Edition)

available online at the Library

### R.F. Egerton

**Electron Energy{Loss Spectroscopy in the Electron Microscope**

Plenum Press, 2011 (3rd Edition)

available online at the Library
_____no_output_____## Software

### Image Analysis software for SEM and TEM

#### Digital Micrograph
You can get a free offline license from [Gatan](https://www.gatan.com/resources/scripting/demo/)

#### ImageJ
a free imaging software with a lot of plugins: download from [NIH](https://rsbweb.nih.gov/ij/)

#### jupyter notebooks
use provided [notebooks](https://github.com/gduscher/MSE672-Introduction-to-TEM)_____no_output_____## Syllabus

| Week | Lecture | Chp. | Lab |
|------|---------|------|-----|
|01/17 - 01/23| 1:Introduction | 1 | |
|01/24 - 01/30| 2,3:Geometric Ray Optics | 6 ||
|01/31 - 02/06| 4,5:Kinematic Diffraction |  2,11 | Lab1: Basic Imaging|
|02/07 - 02/13 | 6,7:Kinematic Diffraction -parallel | 12  | Lab2: Basic Diffraction|
|02/14 - 02/20 | 8,9:Kinematic Diffraction -convergent   | 19,20 | Lab3: SAED |
|02/21 - 02/27 | 10,11:Dynamic Diffraction | 14,15 | Lab4: CBED |
|02/28 - 03/06 | 12:Imaging Overview | 22 ||
|03/07 - 03/13 | 13,14:Imaging Conventional | 24-27 | Lab5: Exam|
|03/14 - 03/20 | 15,16:Imaging Phase Contrast | 28,30 | Lab6: Two Beams|
|03/21 - 03/27 | 17,18:Imaging and Lens Aberrations | 31 | Lab7: Phase Contrast|
|03/28 - 04/03 | 19,20:Imaging Z-contrast| -  | Lab8: CTF|
|04/04 - 04/10  | 21,22:Spectroscopy EELS | 36-40 | Lab9: Z-contrast|
|04/11 - 04/17  | 23,24:Spectroscopy EDS | 33-35 | Lab10: EELS, ESI|
|04/18 - 04/24  | 25:Other Techniques | 29 ||
|04/25 - 05/01 | 26,27: TEM parts and vacuum | 5-9 ||
|05/02 - 05/08 | 28:Project discussion ||_____no_output_____## Office Hours
Office hours are Thursday  from 2 -- 4pm \\
online with the class zoom link\\
I will answer email questions in the same time slots.

## Class and Lab
-  Parallel to the class there is a Lab course.
- We will analyze the data of the lab course, independent of whether you are participating or not,
- For the analysis of the data we will need a labtop in class.

- Everyone needs a laptop in class
- I need real feedback on what you guys want to do and need to learn._____no_output_____## Homework and Quizzes
- Homeworks and Quizzes are the main way to determine your grade in this course.
- Homeworks have to be submitted electronically and generally are about analyzing on your own data which are similar to those in the practice sessions .
- Think of these homeworks as small take-home exams.
- Additionally, there will be reading assignments every week.
- Each week before the lecture, there will be a short quiz about the content of the reading assignments.
- Neither the TA nor the instructor will answer questions about your homework after the deadline.
- If you are more than two homeworks behind, you failed the class._____no_output_____## Final
The final is a report of how TEM will/can be used in your research.
This should include a critical review of  data from journal articles which did TEM similar to what  you propose._____no_output_____## Grades
The grade is a determined by 1/3 each: final, homework, and quizzes._____no_output_____


# Notebook from JacquelineHSH/discount_optimization
Path: HW_B_final.ipynb

# Home work B
## BAX 443

Group Member: Jacqueline Huang, Jing Li, Jiayin Liu, Jack Ye
_____no_output_____### Preparing data_____no_output_____

import matplotlib.pyplot as plt#
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf
from scipy.optimize import minimize
import math
import scipy
import scipy.stats as stats
from scipy.optimize import minimize_scalar
from scipy.optimize import minimize
from sklearn import preprocessing
import seaborn as sns
#sns.set(style="white")
#sns.set(style="whitegrid", color_codes=True)
import pandas as pd
import numpy as np
url = 'https://raw.githubusercontent.com/ormarketing/b2b/master/data.csv' #This is the link to my github profile with the data
df = pd.read_csv(url)#this load the data from the URL define above
df.head() #just looking at the data
#df['Stage'].unique()_____no_output_____df['Partner cloud']=np.where(df['Type'] =='Partner - Cloud Product', 1.0,0.0)
df['Partner prem']=np.where(df['Type'] =='Partner - OnPremise Product', 1.0, 0.0)
df['Direct cloud']=np.where(df['Type'] =='Direct - Cloud Product', 1.0,0.0)
df['Direct prem']=np.where(df['Type'] =='Direct - OnPremise Product', 1.0, 0.0)
df['amount']=pd.to_numeric(df[' Amount '],errors='coerce')
df['acv']=pd.to_numeric(df[' ACV '],errors='coerce')
df['net amount']=pd.to_numeric(df[' Net Amount '],errors='coerce')
df['billing plan year']=pd.to_numeric(df['Billing Plan'].astype(str).str[0],errors='coerce') #turn billing plan to a bumber
df['discount']=df['amount']-df['net amount']
df['depth']=df['discount']/df['amount']
df.drop(' Discount Amount ',axis=1)
df.drop([' ACV ',' Discount Amount ',' Amount ', ' Net Amount ', 'ID', 'Type'],axis=1);_____no_output_____df['Stage']=np.where(df['Stage'] =='Closed Lost', 'Lost', df['Stage'])
df['Stage']=np.where(df['Stage'] =='Closed Won - Paid', 'Won', df['Stage'])
df['Stage']=np.where(df['Stage'] =='Closed Won - Not Paid', 'Won', df['Stage'])
df['Stage']=np.where(df['Stage'] =='Closed - Lost', 'Lost', df['Stage'])
df['Stage']=np.where(df['Stage'] =='Closed Lost - Not Renewing', 'Lost', df['Stage'])
sns.countplot(x='Stage',data=df)
df['Stage'].value_counts()_____no_output_____count_Lost = len(df[df['Stage']=='Lost'])
count_Won = len(df[df['Stage']=='Won'])
round(count_Won/(count_Lost+count_Won),3)_____no_output_____# win/lose vs segment
%matplotlib inline
pd.crosstab(df.Type,df.Stage).plot(kind='bar')
plt.title('Frequency Won/Lost')
plt.xlabel('Direct vs. Partners')
plt.ylabel('Frequency of Win/Lost')
plt.savefig('WinLost')_____no_output_____

#### seasonality plot_____no_output_____

df_date_stage=df[['Close Date','Stage']]
df_date_stage['Close Date Month num']=pd.to_datetime(df_date_stage['Close Date']).dt.month
df_date_stage=pd.concat([df_date_stage,pd.get_dummies(df_date_stage.Stage,prefix='stage')],axis=1)

df_month_stage=df_date_stage.groupby(['Close Date Month num'],as_index=None).sum()
df_month_stage['won_prop']=df_month_stage['stage_Won']/(df_month_stage['stage_Won']+df_month_stage['stage_Lost'])
df_month_stage.head()

%matplotlib inline

plt.figure(figsize=(22,5))
plt.plot(df_month_stage['Close Date Month num'],df_month_stage['won_prop'])
plt.title('Proportion of won contract by month')
plt.xlabel('Close date by month')
plt.ylabel('Proportion of Won contract')
plt.savefig('WinLost vs date')
#There appears to be a seasonaility. at the end of the quarter, the close rate increase /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

plot avg discount vs time_____no_output_____

df_date_discount=df[['Close Date','discount']]
df_date_discount['Close Date Month num']=pd.to_datetime(df_date_discount['Close Date']).dt.month
df_date_discount=df_date_discount.groupby('Close Date Month num',as_index=None).mean()
%matplotlib inline
plt.figure(figsize=(22,5))
plt.plot(df_date_discount['Close Date Month num'],df_date_discount['discount'])
plt.title('Discount avg amount by month')
plt.xlabel('Close date by month')
plt.ylabel('Discount avg amount')
plt.savefig('WinLost vs date')
#There appears to be a seasonaility. at the end of the quarter, the avg discount amount increase /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

The date in terms of month appears to differentiate, putting it as a dependent variable _____no_output_____

df['Close Date Month']=pd.to_datetime(df['Close Date']).dt.month
df['End of Quarter']=np.where(df['Close Date Month'].isin([3,6,9,12]),1,0)_____no_output_____

#### split into training and testing _____no_output_____

df['y']=np.where(df['Stage'] =='Won', 1.0,0.0)
#spliting the dataset by ID
df_train = df.head(1200)
df_test = df.tail(305)
_____no_output_____

### Modeling _____no_output_____# **Price Optimization**

Using these results, I want to optimize on the price
p is the amount, d is the discount

\begin{equation}
R(p)=Proba(Y=1|p,d) \times (p-d)
\end{equation}_____no_output_____# **Enriching the Intercept**_____no_output_____##### Model from class as benchmark _____no_output_____

# Original class Model: remove the constant variable from the list of independent variables and include instead:
# whether the lead is a new logo or not, and the types of product
y=df_train['y']
x2=df_train[['discount','amount','New Logo','Partner cloud','Partner prem','Direct cloud','Direct prem']]
model2 = sm.Logit(y, x2, missing='drop')
model2.fit
type(model2)
result2=model2.fit()
print(result2.summary()) #appears to be all significant Optimization terminated successfully.
         Current function value: 0.445041
         Iterations 8
                           Logit Regression Results
==============================================================================
Dep. Variable:                      y   No. Observations:                 1151
Model:                          Logit   Df Residuals:                     1144
Method:                           MLE   Df Model:                            6
Date:                Mon, 08 Jun 2020   Pseudo R-squ.:                  0.3318
Time:                        19:44:48   Log-Likelihood:                -512.24
converged:                       True   LL-Null:                       -766.56
Covariance Type:            nonrobust   LLR p-value:                1.160e-106
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
discount       5.506e-05   6.59e-06      8.352      0.000    4.21e-05     6.8e-05
amount        -1.804e-06    1.1e-06     -1.639      0.101   -3.96e-06    3.53e-07
New Logo          2.1419      0.172     12.475      0.000       1.805       2.478
Partner cloud    -2.6405      0.267     -9.891      0.000      -3.164      -2.117
Partner prem     -1.7741      0.153    -11.630      0.000      -2.073      -1.475
Direct cloud     -3.0963      0.420     -7.364      0.000      -3.920      -2.272
Direct prem      -2.5071      0.214    -11.698      0.000      -2.927      -2.087
=================================================================================

#### Model including acv`& end of quarter_____no_output_____

# New Model2: remove the constant variable from the list of independent variables and include plus billing plan & acv :
# whether the lead is a new logo or not, and the types of product
# billing plan turns out insignificant
y=df_train['y']
x2=df_train[['discount','amount','New Logo','Partner cloud','Partner prem','Direct cloud','Direct prem','acv','End of Quarter']]
model2 = sm.Logit(y, x2, missing='drop')
model2.fit
type(model2)
result2=model2.fit()
print(result2.summary()) #appears to be all significant
#class R-square is  0.3048Optimization terminated successfully.
         Current function value: 0.370624
         Iterations 8
                           Logit Regression Results
==============================================================================
Dep. Variable:                      y   No. Observations:                 1121
Model:                          Logit   Df Residuals:                     1112
Method:                           MLE   Df Model:                            8
Date:                Mon, 08 Jun 2020   Pseudo R-squ.:                  0.4473
Time:                        19:44:49   Log-Likelihood:                -415.47
converged:                       True   LL-Null:                       -751.77
Covariance Type:            nonrobust   LLR p-value:                5.628e-140
==================================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
----------------------------------------------------------------------------------
discount         2.68e-05   5.65e-06      4.748      0.000    1.57e-05    3.79e-05
amount          1.376e-05   2.38e-06      5.794      0.000    9.11e-06    1.84e-05
New Logo           2.1085      0.199     10.614      0.000       1.719       2.498
Partner cloud     -2.1364      0.358     -5.965      0.000      -2.838      -1.434
Partner prem      -1.9009      0.213     -8.931      0.000      -2.318      -1.484
Direct cloud      -2.6883      0.477     -5.635      0.000      -3.623      -1.753
Direct prem       -2.5446      0.273     -9.326      0.000      -3.079      -2.010
acv            -4.043e-05   4.84e-06     -8.353      0.000   -4.99e-05   -3.09e-05
End of Quarter     1.6703      0.182      9.167      0.000       1.313       2.027
==================================================================================

Find the opimal discount on hold-out sample _____no_output_____

#as the discount can be calculated as zero, we want to make it positive
#### with sqrt(x^2)
x2=df_test[['discount','amount','New Logo','Partner cloud','Partner prem','Direct cloud','Direct prem','acv','End of Quarter']]
y=df_test['y']
ID=0
ID_max=len(df_test)
z=np.empty([ID_max,])
for ID in range(ID_max):
    UT=np.matmul(x2.iloc[ID,1:10],result2.params[1:10])
    b=result2.params[0]
    p=x2.iloc[ID,1]
    def revenueD(d):
    #"""This is the negative revenue function"""
      return -np.exp(UT+b*np.sqrt(d**2))/(1+np.exp(UT+b*np.sqrt(d**2)))*(p-np.sqrt(d**2))
    sol3=minimize_scalar(revenueD)
    z[ID]=sol3.x
print('optimal discount is:',round(np.mean(z),0),'observed discount is ',np.round(np.mean(df['discount']),0))
# z is the optimal discountoptimal discount is: 7240.0 observed discount is  14927.0

Calculate optimal revenue and compare

_____no_output_____

#xS for x star
df_test['optimal discount']=np.sqrt(z**2)
xS=df_test[['optimal discount','amount','New Logo','Partner cloud','Partner prem','Direct cloud','Direct prem','acv','End of Quarter']]
print(xS.shape,result2.params.shape)
ID=0
ID_max=len(df_test)
revOpt=np.empty([ID_max,])
while ID<=ID_max-1:
    j=0
    u=0
#    while j<=6:
#     u=u+xS.iloc[ID][j]*result2.params[j]
#     j=j+1
    UT=np.dot(xS.iloc[ID].T,result2.params[0:10].T)
    proba=np.exp(UT)/(1+np.exp(UT))
    revOpt[ID]=proba*(xS.iloc[ID,1]-z[ID])
    ID=ID+1
revOpt=np.nan_to_num(revOpt);
#Compute the expected revenue after optimization and compare it to the acual revenue(305, 9) (9,)
df_test['Revenue']=np.where(df_test['Stage'] =='Won',df_test['net amount'] ,0.0)
round(np.sum(revOpt)/np.sum(df_test['Revenue']),3)#This is the ratio of the optimized expected revenue / actual revenues
#the class example has 1.5/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.
round(np.sum(revOpt)-np.sum(df_test['Revenue']),3)#This is the ratio of the optimized expected revenue / actual revenues
_____no_output_____

#### Core Model improved with interactive terms_____no_output_____

#To see correlation
df[['discount','amount','New Logo','Partner cloud','Partner prem','Direct cloud','Direct prem','acv','billing plan year','End of Quarter']].corr()
#billing plan year has high correlation _____no_output_____import statsmodels.formula.api as smf
formula = "y ~ 0+discount+amount+Q('New Logo')+Q('Partner cloud')+Q('Partner prem')+Q('Direct cloud')+Q('Direct prem')\
+Q('acv')+Q('End of Quarter')+discount:Q('New Logo')+discount:Q('Partner cloud')+discount:Q('Partner prem')+discount:Q('Direct cloud')+\
discount:Q('Direct prem')+discount:Q('End of Quarter')"
model_glm = smf.glm(formula = formula, data=df_train, family=sm.families.Binomial(),missing='drop')
result_glm = model_glm.fit()
print(result_glm.summary())
/usr/local/lib/python3.6/dist-packages/statsmodels/genmod/families/family.py:890: RuntimeWarning: invalid value encountered in true_divide
  n_endog_mu = self._clean((1. - endog) / (1. - mu))

\begin{equation}
Prob_{Win} = 2.012*10^{-6}*discount+1.20*10^{-5} *amount + 1.875*New\ Logo-1.9063*Partner\ cloud -1.9883*Partner\ prem -2.606*Direct\ cloud -2.326*Direct\ prem - 3.532*10^{-5} *acv +  1.5188*End\ of\ Quarter + 3.26*10^{-5}*(discount*New\ Logo) -1.528*10^{-5}*(discount*Partner\ cloud) +3.51*10^{-5}*(discount*Partner\ prem) +1.218*10^{-6}*(discount*Direct\ cloud)-1.903*10^{-5}*(discount*Direct\ prem) + 1.477*10^{-5}*(discount*End\ of\ Quarter)
\end{equation}_____no_output_____Find the opimal discount_____no_output_____

#as the discount can be calculated as zero, we want to make it positive
#### with sqrt(x^2)
x2_interaction=df_test[['discount','amount','New Logo','Partner cloud','Partner prem','Direct cloud','Direct prem','acv','End of Quarter']]

ID=0
ID_max=len(x2_interaction)
z=[]
for ID in range(ID_max):
    UT=np.dot(x2_interaction.iloc[ID,1:9],result_glm.params[1:9])
    UT2=np.dot(x2_interaction.iloc[ID,2:7],result_glm.params[9:14])+x2_interaction.iloc[ID,8]*result_glm.params[14] #skipped acv
    b=result_glm.params[0]
    p=x2_interaction.iloc[ID,1]
    def revenueD(d):
    #"""This is the negative revenue function"""
      return -np.exp(UT+(UT2+b)*np.sqrt(d**2))/(1+np.exp(UT+(UT2+b)*np.sqrt(d**2)))*(p-np.sqrt(d**2))
    sol3=minimize_scalar(revenueD)
    z.append(sol3.x)
z = np.array(z)
print('optimal discount is:',round(np.mean(z),0),'observed discount is ',np.round(np.mean(df['discount']),0))
# z is the optimal discountoptimal discount is: 6602.0 observed discount is  14927.0

Calculate optimal revenue and compare

_____no_output_____

#xS for x star
x2_interaction['optimal discount']=np.sqrt(z**2)
Summary_Opt=df_test
Summary_Opt['optimal discount']=np.sqrt(z**2)
xS=x2_interaction[['optimal discount','amount','New Logo','Partner cloud','Partner prem','Direct cloud','Direct prem','acv','End of Quarter']]

print(xS.shape,result_glm.params.shape)
ID=0
ID_max=len(x2_interaction)
revOpt=np.empty([ID_max,])
for ID in range(ID_max):
    dis=xS.iloc[ID,0]
    UT=np.dot(xS.iloc[ID,0:9],result_glm.params[0:9])
    UT2=np.dot(xS.iloc[ID,2:7],result_glm.params[9:14])+xS.iloc[ID,8]*result_glm.params[14] #skipped acv
    proba=np.exp(UT+UT2*np.sqrt(dis**2))/(1+np.exp(UT+UT2*np.sqrt(dis**2)))
    revOpt[ID]=proba*(xS.iloc[ID,1]-dis)
revOpt=np.nan_to_num(revOpt);
#Compute the expected revenue after optimization and compare it to the acual revenue
(305, 9) (15,)
x2_interaction['Revenue']=np.where(df_test['Stage'] =='Won',df_test['net amount'] ,0.0)
round(np.sum(revOpt)/np.sum(x2_interaction['Revenue']),3)#This is the ratio of the optimized expected revenue / actual revenues
#the class example has 1.5/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.
np.sum(revOpt) #optimized Revenue_____no_output_____

#### Performance metrics of the Improved model_____no_output_____#### a. Expected revenue improvement over un-optimized discounts (take the actual discounts in the data-set) for leads in the hold-out sample
_____no_output_____

#as the discount can be calculated as zero, we want to make it positive
#### with sqrt(x^2)
x2_interaction=df_test[['discount','amount','New Logo','Partner cloud','Partner prem','Direct cloud','Direct prem','acv','End of Quarter']]
xS=x2_interaction
print(xS.shape,result_glm.params.shape)
ID=0
ID_max=len(x2_interaction)
revUnOpt=np.empty([ID_max,])
for ID in range(ID_max):
    dis=xS.iloc[ID,0]
    UT=np.dot(xS.iloc[ID,0:9],result_glm.params[0:9])
    UT2=np.dot(xS.iloc[ID,2:7],result_glm.params[9:14])+xS.iloc[ID,8]*result_glm.params[14] #skipped acv
    proba=np.exp(UT+UT2*np.sqrt(dis**2))/(1+np.exp(UT+UT2*np.sqrt(dis**2)))
    revUnOpt[ID]=proba*(xS.iloc[ID,1]-dis)
revUnOpt=np.nan_to_num(revUnOpt)
# print('uoptimal discount is:',round(np.mean(z),0),'observed discount is ',np.round(np.mean(df['discount']),0))
# z is the optimal discount(305, 9) (15,)
x2_interaction['Revenue']=np.where(df_test['Stage'] =='Won',df_test['net amount'] ,0.0)
print("Ratio of unoptimized expected revenue improvement vs actual revenue: ")
round(np.sum(revUnOpt)/np.sum(x2_interaction['Revenue']),3)#This is the ratio of the unoptimized expected revenue / actual revenues
Ratio of unoptimized expected revenue improvement vs actual revenue:
print('Expected revenue improvement unoptimized increased by: ')
a=round(np.sum(revUnOpt)-np.sum(x2_interaction['Revenue']),3)#This is the difference of the unoptimized expected revenue vs actual revenues
aExpected revenue improvement unoptimized increased by:

#### b. Expected revenue improvement over optimized discounts (based on your estimation results, optimize discounts offered to leads in the hold-out sample)._____no_output_____

print("Ratio of expected revenue improvement vs actual revenue: ")
round(np.sum(revOpt)/np.sum(x2_interaction['Revenue']),3)#This is the ratio of the optimized expected revenue / actual revenuesRatio of expected revenue improvement vs actual revenue:
print('Expected revenue improvement increased by: ')
b=round(np.sum(revOpt)-np.sum(x2_interaction['Revenue']),3) #This is the difference of the optimized expected revenue vs actual revenues
bExpected revenue improvement increased by:
round(a/b,2)_____no_output_____

#### Summary_____no_output_____

Summary_Opt.columns_____no_output_____Summary_Opt['Revenue_optimized']=revOpt
Summary_Opt['depth_optimized']=Summary_Opt['optimal discount']/Summary_Opt['amount']
Summary_Opt['depth']=Summary_Opt['discount']/Summary_Opt['amount']
Summary_Opt['Revenue_optimized']=revOpt
Summary_Opt['Increased Revenue']=Summary_Opt['Revenue_optimized']-Summary_Opt['Revenue']
#type
round(Summary_Opt[['Type','depth','depth_optimized','amount','discount','Increased Revenue']].groupby('Type').mean(),2)/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  This is separate from the ipykernel package so we can avoid doing imports until
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:4: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  after removing the cwd from sys.path.
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """
#new logo
round(Summary_Opt[['New Logo','depth','depth_optimized','amount','discount','Increased Revenue']].groupby('New Logo').mean(),2)_____no_output_____#end of quarter
round(Summary_Opt[['End of Quarter','depth','depth_optimized','amount','discount','Increased Revenue']].groupby('End of Quarter').mean(),2)_____no_output_____

# Additional Exploration: Optimize the revenue separately by customer segmentation _____no_output_____### Optimize each segment _____no_output_____###### direct cloud _____no_output_____

df_seg=df_test
df_seg_direct_cloud=x2_interaction[(x2_interaction['Direct cloud']==1)]
df_seg_direct_cloud.head(1)_____no_output_____#Using the parameters from the new model with interactive terms
x2=df_seg_direct_cloud[['discount','amount','New Logo','acv','End of Quarter']]
ID=0
ID_max=len(df_seg_direct_cloud)
z=[]
for ID in range(ID_max):
    UT=np.dot(x2.iloc[ID,1:3],result_glm.params[1:3])+np.dot(x2.iloc[ID,3:5],result_glm.params[7:9])
    UT2=x2.iloc[ID,2]*result_glm.params[9]+x2.iloc[ID,4]*result_glm.params[14]+result_glm.params[12]#skipped dis:newlogo,dis:eof,dis:directcloud acv
    b=result_glm.params[0]
    p=x2.iloc[ID,1]
    def revenueD(d):
    #"""This is the negative revenue function"""
      return -np.exp(UT+(UT2+b)*np.sqrt(d**2))/(1+np.exp(UT+(UT2+b)*np.sqrt(d**2)))*(p-np.sqrt(d**2))
    sol3=minimize_scalar(revenueD)
    z.append(sol3.x)
z = np.array(z)
print('optimal discount is:',round(np.mean(z),0),'observed discount is ',np.round(np.mean(df['discount']),0))
# z is the optimal discountoptimal discount is: 3272.0 observed discount is  14927.0
#xS for x star
df_seg_direct_cloud['optimal discount']=np.sqrt(z**2)
xS=df_seg_direct_cloud[['optimal discount','amount','New Logo','acv','End of Quarter']]
print(xS.shape,result2.params.shape)
ID=0
ID_max=len(df_seg_direct_cloud)
revOpt1=np.empty([ID_max,])
for ID in range(ID_max):
    j=0
    u=0
    UT=np.dot(xS.iloc[ID,0:3],result_glm.params[0:3])+np.dot(xS.iloc[ID,3:5],result_glm.params[7:9])
    UT2=xS.iloc[ID,2]*result_glm.params[9]+xS.iloc[ID,4]*result_glm.params[14]+result_glm.params[12] #skipped acv
    proba=np.exp(UT+UT2*np.sqrt(dis**2))/(1+np.exp(UT+UT2*np.sqrt(dis**2)))
    revOpt1[ID]=proba*(xS.iloc[ID,1]-z[ID]) #attention for the position
revOpt1=np.nan_to_num(revOpt1);
#the data points are few(17, 5) (9,)

###### direct on-premise_____no_output_____

df_seg_direct_prem=x2_interaction[(x2_interaction['Direct prem']==1)]
df_seg_direct_prem.head(1)_____no_output_____x2=df_seg_direct_prem[['discount','amount','New Logo','acv','End of Quarter']]
ID=0
ID_max=len(df_seg_direct_prem)
z=[]
for ID in range(ID_max):
    UT=np.dot(x2.iloc[ID,1:3],result_glm.params[1:3])+np.dot(x2.iloc[ID,3:5],result_glm.params[7:9])
    UT2=x2.iloc[ID,2]*result_glm.params[9]+x2.iloc[ID,4]*result_glm.params[14]++result_glm.params[13] #skipped acv
    b=result_glm.params[0]
    p=x2.iloc[ID,1]
    def revenueD(d):
    #"""This is the negative revenue function"""
      return -np.exp(UT+(UT2+b)*np.sqrt(d**2))/(1+np.exp(UT+(UT2+b)*np.sqrt(d**2)))*(p-np.sqrt(d**2))
    sol3=minimize_scalar(revenueD)
    z.append(sol3.x)
z = np.array(z)
print('optimal discount is:',round(np.mean(z),0),'observed discount is ',np.round(np.mean(df['discount']),0))
# z is the optimal discountoptimal discount is: 0.0 observed discount is  14927.0
#xS for x star
df_seg_direct_prem['optimal discount']=np.sqrt(z**2)
xS=df_seg_direct_prem[['optimal discount','amount','New Logo','acv','End of Quarter']]
print(xS.shape,result2.params.shape)
ID=0
ID_max=len(df_seg_direct_prem)
revOpt2=np.empty([ID_max,])
for ID in range(ID_max):
    j=0
    u=0
#    while j<=6:
#     u=u+xS.iloc[ID][j]*result2.params[j]
#     j=j+1
    UT=np.dot(xS.iloc[ID,0:3],result_glm.params[0:3])+np.dot(xS.iloc[ID,3:5],result_glm.params[7:9])
    UT2=xS.iloc[ID,2]*result_glm.params[9]+xS.iloc[ID,4]*result_glm.params[14]+result_glm.params[13] #skipped acv
    proba=np.exp(UT+UT2*np.sqrt(dis**2))/(1+np.exp(UT+UT2*np.sqrt(dis**2)))
    revOpt2[ID]=proba*(xS.iloc[ID,1]-z[ID]) #attention for the position
revOpt2=np.nan_to_num(revOpt2);
#Compute the expected revenue after optimization and compare it to the acual revenue(117, 5) (9,)
x2_interaction['Revenue']=np.where((df_test['Stage'] =='Won')&(df_test['Direct prem']==1),df_test['net amount'] ,0.0)
round(np.sum(revOpt2)/np.sum(x2_interaction['Revenue']),3)#This is the ratio of the optimized expected revenue / actual revenues
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.

###### Partner on-prem_____no_output_____

df_seg_partner_prem=x2_interaction[(x2_interaction['Partner prem']==1)]
df_seg_partner_prem.head(1)_____no_output_____#NEW as the discount can be calculated as zero, we want to make it positive
x2=df_seg_partner_prem[['discount','amount','New Logo','acv','End of Quarter']]

ID=0
ID_max=len(df_seg_partner_prem)
z=[]
for ID in range(ID_max):
    UT=np.dot(x2.iloc[ID,1:3],result_glm.params[1:3])+np.dot(x2.iloc[ID,3:5],result_glm.params[7:9])
    UT2=x2.iloc[ID,2]*result_glm.params[9]+x2.iloc[ID,4]*result_glm.params[14]+result_glm.params[11] #skipped acv
    b=result_glm.params[0]
    p=x2.iloc[ID,1]
    def revenueD(d):
    #"""This is the negative revenue function"""
      return -np.exp(UT+(UT2+b)*np.sqrt(d**2))/(1+np.exp(UT+(UT2+b)*np.sqrt(d**2)))*(p-np.sqrt(d**2))
    sol3=minimize_scalar(revenueD)
    z.append(sol3.x)
z = np.array(z)
print('optimal discount is:',round(np.mean(z),0),'observed discount is ',np.round(np.mean(df['discount']),0))
# z is the optimal discountoptimal discount is: 4822.0 observed discount is  14927.0
#xS for x star
df_seg_partner_prem['optimal discount']=np.sqrt(z**2)
xS=df_seg_partner_prem[['optimal discount','amount','New Logo','acv','End of Quarter']]
print(xS.shape,result2.params.shape)
ID=0
ID_max=len(df_seg_partner_prem)
revOpt3=np.empty([ID_max,])
for ID in range(ID_max):
    j=0
    u=0
#    while j<=6:
#     u=u+xS.iloc[ID][j]*result2.params[j]
#     j=j+1
    UT=np.dot(xS.iloc[ID,0:3],result_glm.params[0:3])+np.dot(xS.iloc[ID,3:5],result_glm.params[7:9])
    UT2=xS.iloc[ID,2]*result_glm.params[9]+xS.iloc[ID,4]*result_glm.params[14]+result_glm.params[11] #skipped acv
    proba=np.exp(UT+UT2*np.sqrt(dis**2))/(1+np.exp(UT+UT2*np.sqrt(dis**2)))
    revOpt3[ID]=proba*(xS.iloc[ID,1]-z[ID]) #attention for the position
revOpt3=np.nan_to_num(revOpt3);
#Compute the expected revenue after optimization and compare it to the acual revenue(135, 5) (9,)
x2_interaction['Revenue']=np.where((df_test['Stage'] =='Won')&(df_test['Partner prem']==1),df_test['net amount'] ,0.0)
round(np.sum(revOpt3)/np.sum(x2_interaction['Revenue']),2)#This is the ratio of the optimized expected revenue / actual revenues
#the class example has 1.5/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.

###### partner cloud_____no_output_____

df_seg_partner_cloud=x2_interaction[(x2_interaction['Partner cloud']==1)]
df_seg_partner_cloud.head(1)_____no_output_____#NEW as the discount can be calculated as zero, we want to make it positive
x2=df_seg_partner_cloud[['discount','amount','New Logo','acv','End of Quarter']]
ID=0
ID_max=len(df_seg_partner_cloud)
z=[]
for ID in range(ID_max):
    UT=np.dot(x2.iloc[ID,1:3],result_glm.params[1:3])+np.dot(x2.iloc[ID,3:5],result_glm.params[7:9])
    UT2=x2.iloc[ID,2]*result_glm.params[9]+x2.iloc[ID,4]*result_glm.params[14]+result_glm.params[10] #skipped acv
    b=result_glm.params[0]
    p=x2.iloc[ID,1]
    def revenueD(d):
    #"""This is the negative revenue function"""
      return -np.exp(UT+(UT2+b)*np.sqrt(d**2))/(1+np.exp(UT+(UT2+b)*np.sqrt(d**2)))*(p-np.sqrt(d**2))
    sol3=minimize_scalar(revenueD)
    z.append(sol3.x)
z = np.array(z)
print('optimal discount is:',round(np.mean(z),0),'observed discount is ',np.round(np.mean(df['discount']),0))
# z is the optimal discountoptimal discount is: 2995.0 observed discount is  14927.0
#xS for x star
df_seg_partner_cloud['optimal discount']=np.sqrt(z**2)
xS=df_seg_partner_cloud[['optimal discount','amount','New Logo','acv','End of Quarter']]
print(xS.shape,result2.params.shape)
ID=0
ID_max=len(df_seg_partner_cloud)
revOpt4=np.empty([ID_max,])
for ID in range(ID_max):
    UT=np.dot(xS.iloc[ID,0:3],result_glm.params[0:3])+np.dot(xS.iloc[ID,3:5],result_glm.params[7:9])
    UT2=xS.iloc[ID,2]*result_glm.params[9]+xS.iloc[ID,4]*result_glm.params[14]+result_glm.params[10] #skipped acv
    proba=np.exp(UT+UT2*np.sqrt(dis**2))/(1+np.exp(UT+UT2*np.sqrt(dis**2)))
    revOpt4[ID]=proba*(xS.iloc[ID,1]-z[ID]) #attention for the position
revOpt4=np.nan_to_num(revOpt4);
#Compute the expected revenue after optimization and compare it to the acual revenue(36, 5) (9,)
x2_interaction['Revenue']=np.where((df_test['Stage'] =='Won')&(df_test['Partner cloud']==1),df_test['net amount'] ,0.0)
round(np.sum(revOpt4)/np.sum(x2_interaction['Revenue']),3)#This is the ratio of the optimized expected revenue / actual revenues
#the class example has 1.5/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.

## Performance Metrics: _____no_output_____#### b. Expected revenue improvement over optimized discounts (based on your estimation results, optimize discounts offered to leads in the hold-out sample)._____no_output_____

print("Ratio of expected revenue improvement vs actual revenue: ")
(np.sum(revOpt1)+np.sum(revOpt2)+np.sum(revOpt3)+np.sum(revOpt4))/np.sum(df_test['Revenue']) #ratioRatio of expected revenue improvement vs actual revenue:
print('Expected revenue improvement increased by: ')
round(np.sum(revOpt1)+np.sum(revOpt2)+np.sum(revOpt3)+np.sum(revOpt4)-np.sum(df_test['Revenue']),2) #differenceExpected revenue improvement increased by:


# Notebook from stoniah/mlmathmech
Path: 01-Numpy/.ipynb_checkpoints/Jupyter_Intro-checkpoint.ipynb

# Jupyter_____no_output_____**Jupyter** представляет собой веб-интерфейс для исполнения кода в тетрадях \ ноубуках (notebook). Документ jupyter notebook имеет расширение .ipynb и состоит из множества ячеек. В ячейках можно писать программный код, делать размеченные и неразмеченные заметки. Этим функциям соответствует три типа ячеек:

    1. code
    2. markdown
    3. raw_____no_output_____Существуют 3 основных IDE для работы с юпитер ноутбуками:
* [Jupyter Notebook](https://jupyter.org/)
* [Jupyter Lab](https://towardsdatascience.com/jupyter-lab-evolution-of-the-jupyter-notebook-5297cacde6b)
* [Google Colab](https://colab.research.google.com)

Первые 2 - локальные среды разработки, которые нужно устанавливать на компьютер.

Colab - облачная среда разработки. Код работает на гугловских серверах.

У каждой есть свои плюсы и минусы. Пользоваться можно любой._____no_output_____Для работы с содержимым ячейки используется *режим редактирования* (*Edit mode*, включается нажатием клавиши **Enter** после выбора ячейки), а для навигации между ячейками искользуется *командный режим* (*Command mode*, включается нажатием клавиши **Esc**).

Тип ячейки можно задать в командном режиме либо с помощью горячих клавиш (**y** to code, **m** to markdown, **r** to edit raw text), либо в меню *Cell -> Cell type*. _____no_output_____Кликните 2 раза по блоку, чтобы увидеть исходные данные ячейки._____no_output_____## Сode_____no_output_____Jupyter Notebook представляет интерактивный интерпретатор питона (обертка над IPython).

Имеется глобальное пространство имен. Каждая ячейка с кодом может быть запущена с учетом обазначенных переменных в пространстве.

Ячейки можно запускать в любой последовательности, но тогда нужно следить за тем, какая ячейка в каком порядке запускалась (цифра возле ячейки блока).

Чтобы выполнить ячейку с кодом нужно нажать *Shift + Enter*. На экран выведется результат **последней** команды._____no_output_____

def frac(n):
    return frac(n-1)*n if n > 1 else 1_____no_output_____frac_5 = frac(5)
frac_5_____no_output_____# вывелось только второе значение
frac(6)
frac(7)_____no_output_____print(frac(6))
frac(7)720

Что-то более интересное, вставляем видео, через библиотеку_____no_output_____

from IPython.display import YouTubeVideo
YouTubeVideo('fiQTb7-rCPo', width=640, height=360)_____no_output_____

Помимо кода, в блоке могут быть специфические для Jupyter вещи:_____no_output_____### [Line magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html) (`%magic` команды)_____no_output_____

#напечатает историю выполнения кода
%history -n 5-8   5:
from IPython.display import HTML

# Youtube
HTML('')
   6:
from IPython.display import HTML

# Youtube
HTML('')
   7:
def frac(n):
    return frac(n-1)*n if n > 1 else 1
   8:
frac_5 = frac(5)
frac_5

### [Cell magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html#cell-magics) (`%%magic` команды)_____no_output_____

%%timeit

for _ in range(3000):
    for _ in range(3000):
        2 + 3 + 5 + 7206 ms ± 7.27 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

### `!system` команды — вызовы к программам из командной строки (cmd, bash)_____no_output_____

!cmake --versioncmake version 3.13.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).
!echo "printing from the shell"printing from the shell
!pip -H install numpy
Usage:
  pip <command> [options]

no such option: -H

## Markdown_____no_output_____Markdown - упрощенное сочетание html и латеха вместе.
Он позволяет:

0. Составлять упорядоченные списки
1. #Делать
##заголовки
###разного уровня
3. Выделять *текст* <s>при</s> **необходимости**
4. Добавлять [ссылки](http://imgs.xkcd.com/comics/the_universal_label.png)

* Составлять неупорядоченные списки_____no_output_____Страшные формулы ($\LaTeX$):

$\newcommand{\tilsigma}{\tilde{\sigma}}$

$$f(x,y) = A \exp\left(- \left(\frac{(x-x_o)^2}{2\tilsigma_X^2} + \frac{(y-y_o)^2}{2\tilsigma_Y^2} \right)\right)$$_____no_output_____Можно также вставлять изображения, видео и прочее:

_____no_output_____Можно вставлять сниппеты с кодом (без возможности выполнить):

```python
import pandas
```_____no_output_____А еще можно делать клевые таблицы

|       | 0 | 1  |  |n |
|-------|---|----|--|----|
|0      | 1 | $x_0$ | $\ldots$ | $x_0^n$ |
|1      | 1 | $x_1$ | $\ldots$ | $x_1^n$ |
|$\vdots$ | $\vdots$|$\ddots$  | $\ddots$   | $\vdots$ |
|$n$  | 1 |$x_n$|  | $x_n^n$ |_____no_output_____## Raw_____no_output_____

## Полезные материалы

* [Jupyter Markdown Manual](https://athena.brynmawr.edu/jupyter/hub/dblank/public/Jupyter%20Notebook%20Users%20Manual.ipynb)
* [Особенности Jupyter на Хабре](https://habr.com/en/company/wunderfund/blog/316826/)
* [IPython tutorial](https://ipython.readthedocs.io/en/stable/interactive/index.html)
* [28 useful Jupyter Tips](https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/)
* [Markdown sintax](https://daringfireball.net/projects/markdown/syntax)_____no_output_____## Источники материалов:
* [mlcourse.ai](https://github.com/Yorko/mlcourse.ai) - курс Машинного обучения с OpenDataScience
* [AI Seminars](https://github.com/AICommunityInno/Seminars) - семинары по Машинному обучению в Иннополисе
* [HSE-ML course](https://github.com/esokolov/ml-course-hse) - курс Машинного обучения ФКН ВШЭ_____no_output_____


# Notebook from Juan-Mateos/mexico_ci_mapping
Path: notebooks/meetup_2018.ipynb

# Meetup Mexico analysis

Tech meetups are events where people get together to network and share skills. Meetup is a platform used by many people to set up these events. We have previously used the platform to analye levels of activity in different creative communities in the UK.

For creative industries mapping project, we have collected data about all tech meetups in Mexico. Our goal is to measure levels of creative networking in different cities, how this has evolved and the networks between them. In order to do this we need to do some data processing:

* We need to identify creative communities -> We perform a topic modelling analysis in the keywords used by conference organisers to tag their events. This gives us a set of topics we can then classify into creative / non-creative categories.

* We need to geocode creative communities -> We place each group in a Mexican state using their longitde and latitude and a shapefile defining the states as polygons.

* We need to map relations between communities -> We use overlaps between individual memberships in communities to proxy their interactions: those communities with more intersections in their populations are more strongly connected than those with less._____no_output_____## Preamble

* Imports, directory management and functions
_____no_output_____

%matplotlib inline

#Key imports are loaded from my profile (see standard_imports.py in src folder).

#Other imports

#Zipfile to work with zips.
import zipfile

#Spacy and geopandas for spatial stuff
import spacy
import geopandas as gp

#Paths

#Paths
top = os.path.dirname(os.getcwd())

#External data
ext_data = os.path.join(top,'data/external')

#Metadata
meta_data = os.path.join(top,'data/metadata')

#Interim data
int_data = os.path.join(top,'data/interim')

#Processed data
proc_data = os.path.join(top,'data/processed')

#Figures
fig_path = os.path.join(top,'reports/figures')

#Tables
tab_path = os.path.join(top,'reports/tables')

#Models
mod_path = os.path.join(top,'models')

#Visualisations
viz_path = os.path.join(top,'src/visualization')

#Get date for saving files
today = datetime.datetime.today()

today_str = "_".join([str(x) for x in [today.day,today.month,today.year]])_____no_output_____#Function to save figures

def save_fig(name,today_str=today_str):
    '''
    Saves a plot

    '''
    plt.savefig(fig_path+'/{today}_{name}.pdf'.format(today=today_str,name=name),bbox_inches='tight')
_____no_output_____sys.version
from setuptools import find_namespace_packages_____no_output_____from data_getters.meetup import select_meetup
import pymysql

config_path = '/Users/jmateosgarcia/Desktop/projects_2017/mexico/mexico_mapping/notebooks/innovation-mapping.config'

dfs = select_meetup(config_path, 34, "MX")

for k, df in dfs.items():
    print(k, len(df))core_groups 725
core_members_groups 133080
extended_groups 3291
extended_members_groups 15203
dfs['core_groups']_____no_output_____

## 1. Load data_____no_output_____### Load metadata_____no_output_____Load shapefiles for mapping. We downloaded them from [here](http://www.inegi.org.mx/geo/contenidos/geoestadistica/m_geoestadistico.aspx)._____no_output_____

#estados = gp.read_file(meta_data+'/inegi_shapefile/conjunto_de_datos/areas_geoestadisticas_estatales.shp')
#We use a shapefile using the WSG standard.
#The shapefile from inegi which we used in our DENUE analysis is based in a different projection.
estados = gp.read_file(meta_data+'/mexstates/mexstates.shp')

#Viva México!
estados.plot()_____no_output_____

### Load meetup group data

In this json file, each element is a group with some metadata
_____no_output_____

#Load the data
with open(ext_data+'/meetup_groups.json','r') as infile:
    groups = json.load(infile)_____no_output_____#Focus on Mexican groups
mex_groups = [x for x in groups if x['country']=='MX']

print(len(mex_groups))

print(np.sum([x['members'] for x in mex_groups]))1163
279761.0

There are 1163 groups in the data with 2790,761 members_____no_output_____

#These are the variables
mex_groups[0].keys()_____no_output_____

### Load meetup member data_____no_output_____

#Load the data
with open(ext_data+'/meetup_groups_members.json','r') as infile:
    members = json.load(infile)_____no_output_____#There are 1.4m people here. Let's look for the ones in Mexican groups
#This is a list of group ids
mex_groups_ids = set([x['id'] for x in mex_groups])_____no_output_____#Extract users with mexican ids
mex_users = [x for x in members if x['group_id'] in mex_groups_ids]_____no_output_____#How many unique people here?

len(set([x['member_id'] for x in mex_users]))_____no_output_____

64,288 unique individuals_____no_output_____

mex_users[0]_____no_output_____#Create group membership lists
#First we create a container for members
group_memberships = {x:[] for x in set([x['group_id'] for x in mex_users])}

#Then we go through each member and append them to each group
for user in mex_users:
    group_memberships[user['group_id']].append(user['member_id'])_____no_output_____

## 2. Processing data

There are two bits of processing we need to do before we get started with the analysis:

1. Classify groups into categories (to identify the creative groups)
2. Classify groups into states (for mapping etc.)
_____no_output_____### Classify groups into topics (including creative topics)

As a first pass we will use LDA, a topic modelling algorithm that extracts topics from collections of words. To do this efficiently, we will create a small pipeline using the gensim package for Natural language processing
_____no_output_____

#What are the top topics?

#First we need to extract the topics from the json object.
#Every element has an element with a list of topics for which we want to extract the tidy,
#easy to analyse  unique url keys
#mex_topics_dict = {str(int(x['id'])):[top['urlkey'] for top in x['topics']] for x in mex_groups}

#And this is a list where every element is the list of keywords
#mex_topics_list = list(mex_topics_dict.values())

mex_topics_list = [[top['urlkey'] for top in x['topics']] for x in mex_groups]_____no_output_____#Count the top keywords
meetup_kw_distr = pd.Series([x for el in mex_topics_list for x in el]).value_counts()

#What are the top 10?
meetup_kw_distr[:10]_____no_output_____#What's the distribution?

meetup_kw_distr.describe()_____no_output_____

Very skewed  - there are 2053 topics, but most appear very few times.
Let's see if LDA works and if not we will train the algorithm on a bigger corpus also including UK and
UAE groups_____no_output_____

#Create the LDA pipeline

from gensim import corpora, models/usr/local/lib/python3.5/site-packages/gensim/utils.py:1015: UserWarning: Pattern library is not installed, lemmatization won't be available.
  warnings.warn("Pattern library is not installed, lemmatization won't be available.")
class LdaPipeline():
    '''
    This class processes lists of keywords.
    How does it work?
    -It is initialised with a list where every element is a collection of keywords
    -It has a method to filter keywords removing those that appear less than a set number of times
    -It has a method to process the filtered df into an object that gensim can work with
    -It has a method to train the LDA model with the right parameters
    -It has a method to predict the topics in a corpus

    '''

    def __init__(self,corpus):
        '''
        Takes the list of terms
        '''

        #Store the corpus
        self.tokenised = corpus

    def filter(self,minimum=5):
        '''
        Removes keywords that appear less than 5 times.

        '''

        #Load
        tokenised = self.tokenised

        #Count tokens
        token_counts = pd.Series([x for el in tokenised for x in el]).value_counts()

        #Tokens to keep
        keep = token_counts.index[token_counts>minimum]

        #Filter
        tokenised_filtered = [[x for x in el if x in keep] for el in tokenised]

        #Store
        self.tokenised_filtered = tokenised_filtered
        self.empty_groups = np.sum([len(x)==0 for x in tokenised_filtered])

        return(self)

    def process(self):
        '''
        This creates the bag of words we use in the gensim analysis

        '''
        #Load the list of keywords
        tokenised = self.tokenised

        #Create the dictionary
        dictionary = corpora.Dictionary(tokenised)

        #Create the Bag of words. This converts keywords into ids
        corpus = [dictionary.doc2bow(x) for x in tokenised]

        self.corpus = corpus
        self.dictionary = dictionary
        return(self)

    def tfidf(self):
        '''
        This is optional: We extract the term-frequency inverse document frequency of the words in
        the corpus. The idea is to identify those keywords that are more salient in a document by normalising over
        their frequency in the whole corpus

        '''
        #Load the corpus
        corpus = self.corpus

        #Fit a TFIDF model on the data
        tfidf = models.TfidfModel(corpus)

        #Transform the corpus and save it
        self.corpus = tfidf[corpus]

        return(self)

    def fit_lda(self,num_topics=20,passes=5,iterations=75,random_state=1803):
        '''

        This fits the LDA model taking a set of keyword arguments.
        #Number of passes, iterations and random state for reproducibility. We will have to consider
        reproducibility eventually.

        '''

        #Load the corpus
        corpus = self.corpus

        #Train the LDA model with the parameters we supplied
        lda = models.LdaModel(corpus,id2word=self.dictionary,
                              num_topics=num_topics,passes=passes,iterations=iterations,random_state=random_state)

        #Save the outputs
        self.lda_model = lda
        self.lda_topics = lda.show_topics(num_topics=num_topics)

        return(self)

    def predict_topics(self):
        '''
        This predicts the topic mix for every observation in the corpus

        '''
        #Load the attributes we will be working with
        lda = self.lda_model
        corpus = self.corpus

        #Now we create a df
        predicted = lda[corpus]

        #Convert this into a dataframe
        predicted_df = pd.concat([pd.DataFrame({x[0]:x[1] for x in topics},
                                              index=[num]) for num,topics in enumerate(predicted)]).fillna(0)

        self.predicted_df = predicted_df

        return(self)
    _____no_output_____#Extract test topics

test = LdaPipeline(mex_topics_list).filter(3).process().fit_lda(
    num_topics=70,passes=20,iterations=150).predict_topics()_____no_output_____#We extract the topic mixes to label and classify as creative or non-creative.
#We'll do that by hand in excel
topic_mixes = pd.DataFrame(test.lda_topics[:100])

topic_mixes.columns=['topic_number','topic_mix']

topic_mixes.to_csv(top+'/references/{date}_topic_mix_for_labelling_70.csv'.format(date=today_str),index=False)_____no_output_____#Load the labelled groups
topic_mixes_labelled = pd.read_csv(top+'/references/7_2_2018_topic_mix_for_labelling_70_labelled.csv')

#Now we name the groups
groups_test = test.predicted_df

groups_test.columns = topic_mixes_labelled['segment']

#We had some repaat groups so we should melt, aggregate their weights over groups and pivot
groups_test_2 = pd.pivot_table(pd.melt(groups_test.reset_index(drop=False),
                       id_vars='index').groupby(['index','segment'])['value'].sum().reset_index(drop=False),
                               index='index',columns='segment',values='value')

#We use idxmax to get the top topic for each group
groups_test_2['top_topic'] = groups_test_2.idxmax(axis=1)
_____no_output_____

### Now to process each lat lon into its state polygon_____no_output_____

from geopandas.geoseries import *

def point_in_poly(point,poly):
    '''
    This function checks if a point is inside a polygon and returns the result

    '''

    result = poly.contains(point)

    return(result)
    _____no_output_____#This turns each meetup lat/lon into a geographical point we can search for in a polygon
meetup_geo_points = [(x['id'],Point(x['lon'],x['lat'])) for x in mex_groups]_____no_output_____#This container stores the states
cont = []

for point in [x[1] for x in meetup_geo_points]:

    #Applies the function above
    result = [point_in_poly(point,estados.loc[num,'geometry']) for num in np.arange(len(estados))]

    #Finds the state name that contains the point
    try:
        name = [x for x,y in zip(estados['ADMIN_NAME'],result) if y==True][0]

    #We had a problem with one of them
    except:
        name = np.nan

    #Returns all contents
    cont.append(name)_____no_output_____

### Final data combination

We want a dataframe with:

* Group name
* State
* Lat and lon
* top topic
* date it started
* List of members
_____no_output_____

#This lookup represents creative activities
meetup_topic_sector_lookup = {x:y if type(y)==str else 'not_creative' for x,y in zip(topic_mixes_labelled.segment,topic_mixes_labelled.creative_sector)}
_____no_output_____#Key variables for the dataframe
mu_df = pd.concat([pd.DataFrame({'name':x['name'],
                                        'city':x['city'],
                                        'created':x['created'],
                                        'lat':x['lat'],
                                        'lon':x['lon']},
                                       index=[x['id']]) for x in mex_groups]).reset_index(drop=False)

#Add a list with topics
mu_df['keyword_list'] = [[top['urlkey'] for top in x['topics']] for x in mex_groups]

#Add the top topic (they were in the same order)
mu_df['top_topic'] = groups_test_2['top_topic']

#Add the sector
mu_df['sector'] = [meetup_topic_sector_lookup[x] for x in mu_df['top_topic']]

#Add the state
mu_df['state'] = cont

#Add the list of members
mu_df['member_list'] = [group_memberships[x] if x in group_memberships.keys() else []
                                for x in mu_df['index']]

mu_df['member_n'] = [len(x) for x in mu_df['member_list']]

#And now get the year the meetup was created from the timestamp. NB we divide by 1000 to go from miliseconds
#to seconds
mu_df['year'] = [datetime.datetime.fromtimestamp(x/1000).year for x in mu_df['created']]

#What does it look like?
mu_df.head()_____no_output_____

This dataframe has everything we need for some fun analysis_____no_output_____## 3. Exploratory data analysis

Charts to produce:

* Barchart with activity (total groups and members) by segment with colours by sector
* Timeline of activity by creative industries sector (groups)
* State activity by segment
* Map with bubbles showing activity by sector
* Mexican network
* Topic network
_____no_output_____### Chart 1: Distribution of activity by topic_____no_output_____

from matplotlib import patches_____no_output_____# Barchart

#Create activity stats
mu_distr = pd.concat([mu_df['top_topic'].value_counts(),mu_df.groupby('top_topic')[
    'member_n'].sum()],axis=1).sort_values('member_n',ascending=False)

#Plot

#Get colors

sector_color_lookup = {x:y for x,y in zip(set(mu_df.sector),
                                              plt.cm.get_cmap('Accent').colors[:len(set(mu_df.sector))])}

#Plot
fig, ax = plt.subplots(figsize=(12,8),nrows=2,sharex=True)

for number,x in enumerate(mu_distr.index):
    ax[0].bar(number,mu_distr.loc[x]['top_topic'],width=0.8,
              color=sector_color_lookup[meetup_topic_sector_lookup[x]],
              edgecolor='black' if meetup_topic_sector_lookup[x]!='not_creative' else None)
    ax[1].bar(number,mu_distr.loc[x]['member_n'],width=0.8,
             color=sector_color_lookup[meetup_topic_sector_lookup[x]],
              edgecolor='black' if meetup_topic_sector_lookup[x]!='not_creative' else None)

#Create custom legend
custom_patches = [
    patches.Patch(color=c,label=l) for l,c in sector_color_lookup.items()]

ax[0].legend(handles=custom_patches,title='Creative sector')

#Some other tidying up
ax[0].set_ylabel('Number of groups',size=12)
ax[1].set_ylabel('Number of group members',size=12)

#Add xlabels
#Numbers for the ticks
ax[0].set_xticks(np.arange(len(mu_distr.index)))

#And values
ax[1].set_xticklabels(mu_distr.index,rotation=45,ha='right',size=11)

#And title
ax[0].set_title('Activity in Mexican communities by Meetup topic',size=18)

plt.tight_layout()

#Save
save_fig('meetup_activity')

_____no_output_____

### Chart 2: Timeline of activity by sector (and as a share of total)_____no_output_____

#Create the variables to plot, dropping the last year
activity_totals = pd.crosstab(mu_df.sector,mu_df.year).drop(2018,axis=1)
activity_totals_norm = pd.crosstab(mu_df.sector,mu_df.year,normalize=1).drop(2018,axis=1)

#Get the sorted sectors so that we have the biggest ones at the bottom of the barchart
sectors_sorted = activity_totals.loc[:,2017].sort_values(ascending=False).index
_____no_output_____#Plot
fig,ax = plt.subplots(figsize=(10,7),nrows=2)

activity_totals.T.loc[:,sectors_sorted].plot(ax=ax[0],cmap='Accent')
activity_totals_norm.T.loc[:,sectors_sorted].plot.bar(ax=ax[1],stacked=True,cmap='Accent',width=0.8)

ax[0].get_xaxis().set_visible(False)
ax[1].legend().set_visible(False)

ax[0].set_title('Evolution in tech meetup activity in Mexico, 2010-2018',size=18)

plt.tight_layout()

save_fig('meetup_timeline')
_____no_output_____

### Chart 3: State activity by segment_____no_output_____

#State activity by segment...

fig,ax = plt.subplots(figsize=(10,8),nrows=2,sharex=True)

#We need a variable to sort by: total number of groups in a location

states_meetup_sorted = pd.crosstab(mu_df.state,mu_df.sector).sum(axis=1).sort_values(ascending=False).index

#Total number of groups
pd.crosstab(mu_df.state,mu_df.sector).loc[states_meetup_sorted,:].plot.bar(stacked=True,ax=ax[0],width=0.8)

#Unique members in each sector
pd.pivot_table(mu_df.groupby(['state','sector'])['member_list'].apply(
    lambda x: len(set([val for el in x for val in el]))).reset_index(drop=False),
               index='state',columns='sector',values='member_list').fillna(0).loc[
    states_meetup_sorted,:].plot.bar(stacked=True,ax=ax[1],width=0.8)

ax[0].set_ylabel('Number of groups')
ax[1].set_ylabel('Number of unique members in each sector category')

ax[0].set_title('Number of meetup communities and unique members by State',size=18)

plt.tight_layout()

save_fig('state_meetup_activity')
_____no_output_____creat_state = pd.crosstab(mu_df.state,mu_df.sector).drop('not_creative',axis=1).sum(axis=1)
creat_state.sort_values(ascending=False)/creat_state.sum()

_____no_output_____

### Chart 4: Connectivity between communities_____no_output_____

#Functions
def extract_network_from_corpus(label_corpus, edge_attr="None", stop_words=None):
    """
    Extracts a network object from a list or series where every observation is a list of co-ocurring observations
    (e.g. interacting agents, or co-occurring words).
    In the network output, the labels are nodes and co-occurrences between them are edges.
    Weight is number of co-occurrences. There is the option to consider additional edge attributes.

    Inputs:
    label_corpus: corpus of documents with label co-occurrence or node interaction.
    edge_attr: an edge attribute associated to the interaction. It defaults to the string none (which we use
        for control flow)
    stop_words: labels to remove (e.g. generic terms)

    Returns a networkx object.

    """
    label_pairs_container =[]

    #!!! TODO Using type to control flow feels a bit hacky.
    if type(edge_attr) != str:
        #Container for pairs of labels
        edge_attr_name = edge_attr.name

    #For each pair
    for i in range(len(label_corpus)):
        #List of tuples where every tuple is a combination of topics
        pairs = list(combinations(label_corpus.iloc[i], 2))

        #Extract these as lists of dicts capturing connections between topics, and add those to the container.
        #NB we are sorting the tuples to make sure that we don't duplicate edges.
        if len(pairs)>0:
            cont = [{'e1':sorted(tup)[0],'e2':sorted(tup)[1]} for tup in pairs]

            #Give each edge its corresponding attribute (if we have one)
            if type(edge_attr) != str:
                for d in cont:
                    d.update({edge_attr_name: edge_attr[i]})

            label_pairs_container.append(cont)

    #Flatten dict list and turn into DF
    label_pairs_df = pd.DataFrame([ds for sublist in label_pairs_container for ds in sublist])

    #Extract weights (depends on edge_attr)
    if type(edge_attr) != str:
        label_edgelist = pd.DataFrame(label_pairs_df.groupby(['e1','e2',edge_attr_name]
                                                        ).size()).reset_index().sort_values(by=0,ascending=False)
    else:
        label_edgelist = pd.DataFrame(label_pairs_df.groupby(['e1','e2']
                                                        ).size()).reset_index().sort_values(by=0,ascending=False)

    label_edgelist.rename(columns={0:'weight'},inplace=True)

    #Remove stop-words
    if stop_words != None:
        has_ttm =  [x in stop_words or y in stop_words for x,y in zip(label_edgelist['e1'],label_edgelist['e2'])]
        label_edgelist = label_edgelist[-pd.Series(has_ttm)]

    #Create label graph
    if type(edge_attr) != str:
        label_graph = nx.from_pandas_dataframe(label_edgelist,'e1','e2',['weight',edge_attr_name])
    else:
        label_graph = nx.from_pandas_dataframe(label_edgelist,'e1','e2',['weight'])

    return(label_graph)_____no_output_____#We will plot networks between communities

class meetupNetworks():
    '''
    This class creates meetup networks
    We feed it a df with a members list that it uses to create the network
    and metadata for subsetting and visualisation.

    '''

    def __init__(self,meetup_df):
        '''
        Initialise with the meetup df
        '''

        self.df = meetup_df

    def get_all_network(self,thres=None):
        '''
        Takes the membership list and makes a network

        '''
        #Load the df
        mu_df = self.df

        #Flip the data around to create a list where every element is shared group overlaps for individuals

        g_m_df = pd.concat([pd.DataFrame({'m':mu_df.loc[x,'member_list'],
                                          'g':mu_df.loc[x,'name']}) for x in np.arange(0,len(mu_df))]).groupby(
            'm')['g'].apply(lambda x: list(x)).reset_index(drop=True)

        #Extract network with my function
        network = extract_network_from_corpus(g_m_df)

        #Now we want to add metadata to the network
        #Sometimes it is way easier to work with dicts. We create one
        mu_df_dict = mu_df.to_dict(orient='records')
        mu_df_dict_2 = {x['name']:x for x in mu_df_dict}

        #Now we loop over the nodes in the network and give them metadata
        network = enrich_nodes(network,mu_df_dict_2)

        self.network=network
        self.node_metadata = mu_df_dict_2

        return(self)

    def subset_network(self,sector):
        '''
        Subsets the network to focus on a sector

        '''
        network = self.network

        network = nx.subgraph(network_test,nbunch=[n for n in network_test.node if network_test.node[n][
            'sector'] in sector])

        self.network = network

        return(self)

    def filter_weights(self,thres):
        '''
        Filters weights below a certain threshold (quantile in the degree distribution)

        '''
        network = self.network

        weight_threshold = pd.Series([e[2]['weight'] for e in network.edges(data=True)]).quantile(thres)

        #Subset again
        network_2 = nx.Graph([(e[0],e[1],e[2]) for e in network.edges(data=True) if e[2]['weight']>weight_threshold])

        network_2 = enrich_nodes(network_2,self.node_metadata)

        self.network = network_2

        return(self)_____no_output_____def enrich_nodes(network,metadata):
    '''
    This function takes a network whose nodes we want to enrich and a dict where the node ids are keys

    '''

    for node in network.node:
        el = metadata[node]

        network.node[node]['lat'] = el['lat']
        network.node[node]['lon'] = el['lon']
        network.node[node]['top_topic'] = el['top_topic']
        network.node[node]['member_n'] = el['member_n']
        network.node[node]['year'] = el['year']
        network.node[node]['sector'] = el['sector']

    return(network)

    _____no_output_____#Extract networks for all sectors
sector_nets = [(s,meetupNetworks(mu_df).get_all_network().subset_network(s).filter_weights(0.75).network) for s in
               [x for x in sectors_sorted if x!='not_creative']]_____no_output_____def plot_meetup_map(network,sector_name,polygon_df=estados,ax=ax):
    '''
    This function takes a polygon df and a variable and returns a map. We can ask it to discretise the data
    into deciles too.

    '''

    #If we want to discretise the data we transform the sector variable into deciles
    polygon_df.plot(ax=ax,color='white',edgecolor='darkslategrey')

    for x in network.nodes(data=True):
        ax.scatter(x[1]['lon'],x[1]['lat'],s=x[1]['member_n']/3,color=sector_color_lookup[x[1]['sector']],
                  edgecolor='darkslategrey',alpha=0.7)

    for edge in network.edges(data=True):

        if edge[2]['weight']>0:

            n1_lat = network.node[edge[0]]['lat']
            n1_lon = network.node[edge[0]]['lon']

            n2_lat = network.node[edge[1]]['lat']
            n2_lon = network.node[edge[1]]['lon']

            ax.plot([n1_lon,n2_lon],[n1_lat,n2_lat],linewidth=2,color='darkslategrey',alpha=0.7)

    ax.set_title('{x}'.format(x=sector_name),size=16)

    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)_____no_output_____#And then we map
#NB We haven't added legends yet -
#the graphs are not strictly comparable because the color-scale is set based on the distribution for each variable

fig,ax = plt.subplots(figsize=(10,11),nrows=3,ncols=2)

for num,sector in enumerate(sector_nets):
    print(sector[0])

    if num<3:
        col=0
        row=num
    else:
        col=1
        row=num-3

    #estados.plot(ax=ax[row][col])
    plot_meetup_map(sector_nets[num][1],ax=ax[row][col],sector_name=sector_nets[num][0])

fig.suptitle('Creative business Meetup activity and networking',y=1.02,size=18)

plt.tight_layout()

plt.savefig(fig_path+'/{date}_meetup_maps.png'.format(date=today_str),bbox_inches='tight')software
advertising
design
games
publishing
freelance
#Extract networks for all sectors
all_network = meetupNetworks(mu_df).get_all_network().subset_network(
    [x for x in sectors_sorted if x!='not_creative']).filter_weights(0.75).network

fig,ax = plt.subplots(figsize=(13,7))

plot_meetup_map(all_network,ax=ax,sector_name='All Creative')

plt.savefig(fig_path+'/{date}_all_meetup_maps.png'.format(date=today_str),bbox_inches='tight')

_____no_output_____