# Notebook from galuardi/Kalman-and-Bayesian-Filters-in-Python
Path: 09-Nonlinear-Filtering.ipynb

[Table of Contents](http://nbviewer.ipython.org/github/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/table_of_contents.ipynb)_____no_output_____# Nonlinear Filtering_____no_output_____

#format the book
%matplotlib inline
from __future__ import division, print_function
from book_format import load_style
load_style()_____no_output_____

## Introduction_____no_output_____The Kalman filter that we have developed uses linear equations, and so the filter can only handle linear problems. But the world is nonlinear, and so the classic filter that we have been studying to this point can have very limited utility.

There can be nonlinearity in the process model. Suppose we wanted to track the motion of a weight on a spring, such as an automobile's suspension. The equation for the motion with $m$ being the mass, $k$ the spring constant, and $c$ the damping force is

$$m\frac{d^2x}{dt^2} + c\frac{dx}{dt} +kx = 0$$

It is not possible to write a linear equation for this second order differential equation, and therefore we cannot design a Kalman filter using the theory that we have learned.

A second source of nonlinearity comes from the measurements. For example, radars measure the slant range to an object, and we are typically interested in the aircraft's position over the ground. We invoke Pythagoras and get the nonlinear equation:

$$x=\sqrt{\mathtt{slant}^2 - \mathtt{altitude}^2}$$

These facts were not lost on the early adopters of the Kalman filter. Soon after Dr. Kalman published his paper people began working on how to extend the Kalman filter for nonlinear problems.

It is almost true to state that the only equation anyone knows how to solve is $\mathbf{Ax}=\mathbf{b}$. We only really know how to do linear algebra. I can give you any linear set of equations and you can either solve it or prove that it haas no solution.

Anyone with formal education in math or physics has spent years learning various analytic ways to solve integrals, differential equations and so on. Yet even trivial physical systems produce equations that cannot be solved analytically. I can take an equation that you are able to integrate, insert a $\log$ term, and render it insolvable. It is stuff like this that leads to jokes about physicists stating "assume a spherical cow on a frictionless surface in a vacuum...". Without making extreme simplifications most physical problems do not have analytic solutions.

How do we do things like model airflow over an aircraft in a computer, or predict weather, or track missiles with a Kalman filter?  We retreat to what we know: $\mathbf{Ax}=\mathbf{b}$. We find some way to linearize the problem, turning it into a set of linear equations, and then use linear algebra software packages to compute an approximate solution. Linearizing a nonlinear problem gives us inexact answers, and in a recursive algorithm like a Kalman filter or weather tracking system these small errors can sometimes reinforce each other at each step, quickly causing the algorithm to spit out nonsense.

What we are about to embark upon is a difficult problem. There is not one obvious, correct, mathematically optimal solution anymore. We will be using approximations, we will be introducing errors into our computations, and we will forever be battling filters that *diverge*, that is, filters whose numerical errors overwhelm the solution.

In the remainder of this short chapter I will illustrate the specific problems the nonlinear Kalman filter faces. You can only design a filter after understanding the particular problems the nonlinearity in your problem causes. Subsequent chapters will then teach you how to design and implement different kinds of nonlinear filters._____no_output_____## The Problem with Nonlinearity_____no_output_____As asserted in the introduction the only math you really know how to do is linear math, that is, equations of the form

$$ A\mathbf{x}=\mathbf{b}$$

That may strike you as hyperbole. After all, in this book we have integrated a polynomial to get distance from velocity and time. We know how to integrate polynomials, and so we are able to find the closed form equation for distance given velocity and time:

$$\int{(vt+v_0)}\,dt = \frac{a}{2}t^2+v_0t+d_0$$

That's a nonlinear equation that we know how to solve. But it is also a very special form. You spent a lot of time, probably at least a year, learning how to integrate various terms, and you still can not integrate some arbitrary equation - no one can. We don't know how. If you took freshman Physics you perhaps remember homework involving sliding frictionless blocks on a perfect plane and other toy problems. At the end of the course you were almost entirely unequipped to solve real world problems because the real world is nonlinear, and you were taught linear, closed forms of equations. It made the math tractable, but mostly useless.

The mathematics of the Kalman filter is beautiful in part due to the Gaussian equation being so special. It is nonlinear, but when we add and multiply them using linear algebra we get another Gaussian equation as a result. That is very rare. $\sin{x}*\sin{y}$ does not yield a $\sin$ as an output.

What I mean by linearity may be obvious, but there are some subtleties. The mathematical requirements are twofold:

* additivity: $f(x+y) = f(x) + f(y)$
* homogeneity: $f(ax) = af(x)$

This leads us to say that a linear system is defined as a system whose output is linearly proportional to the sum of all its inputs. A consequence of this is that to be linear if the input is zero than the output must also be zero. Consider an audio amp - if I sing into a microphone, and you start talking, the output should be the sum of our voices (input) scaled by the amplifier gain. But if amplifier outputs a nonzero signal such as a hum for a zero input the additive relationship no longer holds. This is because you linearity requires that $amp(voice) = amp(voice + 0)$ This clearly should give the same output, but if amp(0) is nonzero, then

$$
\begin{aligned}
amp(voice) &= amp(voice + 0) \\
&= amp(voice) + amp(0) \\
&= amp(voice) + non\_zero\_value
\end{aligned}
$$

which is clearly nonsense. Hence, an apparently linear equation such as

$$L(f(t)) = f(t) + 1$$

is not linear because $L(0) = 1$! Be careful!_____no_output_____## An Intuitive Look at the Problem_____no_output_____I particularly like the following way of looking at the problem, which I am borrowing from Dan Simon's *Optimal State Estimation* [[1]](#[1]). Consider a tracking problem where we get the range and bearing to a target, and we want to track its position. The reported distance is 50 km, and the reported angle is 90$^\circ$. Assume that the errors in both range and angle are distributed in a Gaussian manner. Given an infinite number of measurements what is the expected value of the position?

I have been recommending using intuition to gain insight, so let's see how it fares for this problem (hint: nonlinear problems are *not* intuitive). We might reason that since the mean of the range will be 50 km, and the mean of the angle will be 90$^\circ$, that clearly the answer will be x=0 km, y=50 km.

Well, let's plot that and find out. Here are 3000 points plotted with a normal distribution of the distance of 0.4 km, and the angle having a normal distribution of 0.35 radians. We compute the average of the all of the positions, and display it as a star. Our intuition is displayed with a large circle._____no_output_____

import numpy as np
from numpy.random import randn
import matplotlib.pyplot as plt

N = 3000
a = np.pi/2. + (randn(N) * 0.35)
r = 50.0     + (randn(N) * 0.4)
xs = r * np.cos(a)
ys = r * np.sin(a)

plt.scatter(xs, ys, label='Sensor', color='k', marker='.', s=2)
xs, ys = sum(xs)/N, sum(ys)/N
plt.scatter(xs, ys, c='r', marker='*', s=200, label='Mean')
plt.scatter(0, 50, c='k', marker='o', s=300, label='Intuition')
plt.axis('equal')
plt.legend();_____no_output_____

We can see that out intuition failed us because the nonlinearity of the problem forced all of the errors to be biased in one direction. This bias, over many iterations, can cause the Kalman filter to diverge. Even if it doesn't diverge the solution will not be optimal. Linear approximations applied to nonlinear problems yields inaccurate results._____no_output_____## The Effect of Nonlinear Functions on Gaussians_____no_output_____Gaussians are not closed under an arbitrary nonlinear function. Recall the equations of the Kalman filter - at each evolution (prediction) we pass the Gaussian representing the state through the process function to get the Gaussian at time $k$. Our process function was always linear, so the output was always another Gaussian.  Let's look at that on a graph. I will take an arbitrary Gaussian and pass it through the function $f(x) = 2x + 1$ and plot the result. We know how to do this analytically, but lets use sampling. I will generate 500,000 points with a normal distribution, pass them through $f(x)$, and plot the results. I do it this way because the next example will be nonlinear, and we will have no way to compute this analytically._____no_output_____

import numpy as np
from numpy.random import normal
gaussian = (0., 1.)
data = normal(loc=gaussian[0], scale=gaussian[1], size=500000)
plt.hist(2*data + 1, 1000);_____no_output_____

This is an unsurprising result. The result of passing the Gaussian through $f(x)=2x+1$ is another Gaussian centered around 1. Let's look at the input, nonlinear function, and output at once._____no_output_____

from book_format import set_figsize, figsize
from nonlinear_plots import plot_nonlinear_func

def g1(x):
    return 2*x+1

with figsize(y=5):
    plot_nonlinear_func(data, g1, gaussian)_____no_output_____

> I explain how to plot Gaussians, and much more, in the Notebook *Computing_and_Plotting_PDFs* in the
Supporting_Notebooks folder. You can also read it online [here](https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/Supporting_Notebooks/Computing_and_plotting_PDFs.ipynb)[1]

The plot labeled 'Input' is the histogram of the original data. This is passed through the  function $f(x)=2x+1$ which is displayed in the chart on the bottom left. The red lines shows how one value, $x=0$ is passed through the function. Each value from input is passed through in the same way to the output function on the right. For the output I computed the mean by taking the average of all the points, and drew the results with the dotted blue line. A solid blue line shows the actual mean for the point $x=0$. The output looks like a Gaussian, and is in fact a Gaussian. We can see that the variance in the output is larger than the variance in the input, and the mean has been shifted from 0 to 1, which is what we would expect given the transfer function $f(x)=2x+1$ The $2x$ affects the variance, and the $+1$ shifts the mean The computed mean, represented by the dotted blue line, is nearly equal to the actual mean. If we used more points in our computation we could get arbitrarily close to the actual value.

Now let's look at a nonlinear function and see how it affects the probability distribution._____no_output_____

def g2(x):
    return (np.cos(3*(x/2 + 0.7))) * np.sin(0.3*x) - 1.6*x

with figsize(y=5):
    plot_nonlinear_func(data, g2, gaussian)_____no_output_____

This result may be somewhat surprising to you. The function looks "fairly" linear - it is pretty close to a straight line, but the probability distribution of the output is completely different from a Gaussian.  Recall the equations for multiplying two univariate Gaussians:

$$\begin{aligned}
\mu &=\frac{\sigma_1^2 \mu_2 + \sigma_2^2 \mu_1} {\sigma_1^2 + \sigma_2^2} \\
\sigma &= \frac{1}{\frac{1}{\sigma_1^2} + \frac{1}{\sigma_2^2}}
\end{aligned}$$

These equations do not hold for non-Gaussians, and certainly do not hold for the probability distribution shown in the 'Output' chart above.

Think of what this implies for the Kalman filter algorithm of the previous chapter. All of the equations assume that a Gaussian passed through the process function results in another Gaussian. If this is not true then all of the assumptions and guarantees of the Kalman filter do not hold. Let's look at what happens when we pass the output back through the function again, simulating the next step time step of the Kalman filter._____no_output_____

y = g2(data)
gaussian2 = (np.mean(y), np.var(y))
with figsize(y=5):
    plot_nonlinear_func(y, g2, gaussian2)_____no_output_____

As you can see the probability function is further distorted from the original Gaussian. However, the graph is still somewhat symmetric around x=0, let's see what the mean is._____no_output_____

print('input  mean, variance: %.4f, %.4f' %
      (np.mean(data), np.var(data)))
print('output mean, variance: %.4f, %.4f' %
      (np.mean(y), np.var(y)))input  mean, variance: -0.0005, 0.9999
output mean, variance: -0.1240, 2.4029

Let's compare that to the linear function that passes through (-2,3) and (2,-3), which is very close to the nonlinear function we have plotted. Using the equation of a line we have

$$m=\frac{-3-3}{2-(-2)}=-1.5$$_____no_output_____

def g3(x):
    return -1.5 * x

with figsize(y=5):
    plot_nonlinear_func(data, g3, gaussian)
out = g3(data)
print('output mean, variance: %.4f, %.4f' %
      (np.mean(out), np.var(out)))_____no_output_____

Although the shapes of the output are very different, the mean and variance of each are almost the same. This may lead us to reasoning that perhaps we can ignore this problem if the nonlinear equation is 'close to' linear. To test that, we can iterate several times and then compare the results._____no_output_____

out = g3(data)
out2 = g2(data)

for i in range(10):
    out = g3(out)
    out2 = g2(out2)
print('linear    output mean, variance: %.4f, %.4f' %
      (np.average(out), np.std(out)**2))
print('nonlinear output mean, variance: %.4f, %.4f' %
      (np.average(out2), np.std(out2)**2))linear    output mean, variance: 0.0404, 7481.2295
nonlinear output mean, variance: -9.0832, 30461.4199

Unfortunately the nonlinear version is not stable. It drifted significantly from the mean of 0, and the variance is half an order of magnitude larger.

I minimized the issue by using a function that is quite close to a straight line. What happens if the function is $y(x)=x^2$?_____no_output_____

def g3(x):
    return -x*x
x0 = (1, 1)
data = normal(loc=x0[0], scale=x0[1], size=500000)

with figsize(y=5):
    plot_nonlinear_func(data, g3, gaussian=x0)_____no_output_____

Despite the curve being smooth and reasonably straight at $x=1$ the probability distribution of the output doesn't look anything like a Gaussian and the computed mean of the output is quite different than the value computed directly. This is not an unusual function - a ballistic object moves in a parabola, and this is the sort of nonlinearity your filter will need to handle. If you recall we've tried to track a ball and failed miserably. This graph should give you insight into why the filter performed so poorly._____no_output_____## A 2D Example_____no_output_____It is hard to look at probability distributions and reason about what will happen in a filter. So let's think about tracking an aircraft with radar. The estimate may have a covariance that looks like this:_____no_output_____

import nonlinear_internal
nonlinear_internal.plot1()_____no_output_____

What happens when we try to linearize this problem? The radar gives us a range to the aircraft. Suppose the radar is directly under the aircraft (x=10) and the next measurement states that the aircraft is 3 miles away (y=3). The positions that could match that measurement form a circle with radius 3 miles, like so._____no_output_____

nonlinear_internal.plot2()_____no_output_____

We can see by inspection that the probable position of the aircraft is somewhere near x=11.4, y=2.7 because that is where the covariance ellipse and range measurement overlap. But the range measurement is nonlinear so we have to linearize it. We haven't covered this material yet, but the Extended Kalman filter will linearize at the last position of the aircraft - (10,2). At x=10 the range measurement has y=3, and so we linearize at that point._____no_output_____

nonlinear_internal.plot3()_____no_output_____

Now we have a linear representation of the problem (literally a straight line) which we can solve. Unfortunately you can see that the intersection of the line and the covariance ellipse is a long way from the actual aircraft position._____no_output_____

nonlinear_internal.plot4()_____no_output_____

That sort of error often leads to disastrous results. The error in this estimate is large. But in the next innovation of the filter that very bad estimate will be used to linearize the next radar measurement, so the next estimate is likely to be markedly worse than this one. After only a few iterations the Kalman filter will diverge, and start producing results that have no correspondence to reality.

This covariance ellipse spans miles. I exaggerated the size to illustrate the difficulties of highly nonlinear systems. In real radar tracking problems the nonlinearity is usually not that bad, but the errors will still accumulate. Other systems you may be work could have this amount of nonlinearity - this was not an exaggeration only to make a point. You will always be battling divergence when working with nonlinear systems._____no_output_____## The Algorithms_____no_output_____You may be impatient to solve a specific problem, and wondering which filter to use. I will quickly survey the options. The subsequent chapters are somewhat independent of each other, and you can fruitfully skip around, though I recommend reading linearly if you truly want to master all of the material.

The workhorses of nonlinear filters are the *linearized Kalman filter* and *extended Kalman filter* (EKF). These two techniques were invented shortly after Kalman published his paper and they have been the main techniques used since then. The flight software in airplanes, the GPS in your car or phone almost certainly use one of these techniques.

However, these techniques are extremely demanding. The EKF linearizes the differential equations at one point, which requires you to find a solution to a matrix of partial derivatives (a Jacobian). This can be difficult or impossible to do analytically. If impossible, you have to use numerical techniques to find the Jacobian, but this is expensive computationally and introduces more error into the system. Finally, if the problem is quite nonlinear the linearization leads to a lot of error being introduced in each step, and the filters frequently diverge. You can not throw some equations into some arbitrary solver and expect to to get good results. It's a difficult field for professionals. I note that most Kalman filtering textbooks merely gloss over the EKF despite it being the most frequently used technique in real world applications.

Recently the field has been changing in exciting ways. First, computing power has grown to the point that we can use techniques that were once beyond the ability of a supercomputer. These use *Monte Carlo* techniques - the computer generates thousands to tens of thousands of random points and tests all of them against the measurements. It then probabilistically kills or duplicates points based on how well they match the measurements. A point far away from the measurement is unlikely to be retained, whereas a point very close is quite likely to be retained. After a few iterations there is a clump of particles closely tracking your object, and a sparse cloud of points where there is no object.

This has two benefits. First, the algorithm is robust even for extremely nonlinear problems. Second, the algorithm can track arbitrarily many objects at once - some particles will match the behavior on one object, and other particles will match other objects. So this technique is often used to track automobile traffic, people in crowds, and so on.

The costs should be clear. It is computationally expensive to test tens of thousands of points for every step in the filter. But modern CPUs are very fast, and this is a good problem for GPUs because the part of the algorithm is parallelizable. Another cost is that the answer is not mathematical. With a Kalman filter my covariance matrix gives me important information about the amount of error in the estimate. The particle filter does not give me a rigorous way to compute this. Finally, the output of the filter is a cloud of points; I then have to figure out how to interpret it. Usually you will be doing something like taking the mean and standard deviations of the points, but this is a difficult problem. There are still many points that do not 'belong' to a tracked object, so you first have to run some sort of clustering algorithm to first find the points that seem to be tracking an object, and then you need another algorithm to produce an state estimate from those points. None of this is intractable, but it is all quite computationally expensive.

Finally, we have a new algorithm called the *unscented Kalman filter* (UKF) that does not require you to find analytic solutions to nonlinear equations, and yet almost always performs better than the EKF. It does especially well with highly nonlinear problems - problems where the EKF has significant difficulties. Designing the filter is extremely easy. Some will say the jury is still out on the UKF, but to my mind the UKF is superior in almost every way to the EKF. I suggest that the UKF should be the starting point for any implementation, especially if you are not a Kalman filter professional with a graduate degree in control theory. The main downside is that the UKF can be a few times slower than the EKF, but this really depends on whether the EKF solves the Jacobian analytically or numerically. If numerically the UKF is almost certainly faster. It has not been proven (and probably it cannot be proven) that the UKF always yields more accurate results than the EKF. In practice it almost always does, often significantly so. It is very easy to understand and implement, and I strongly suggest this filter as your starting point. _____no_output_____## Summary_____no_output_____The world is nonlinear, but we only really know how to solve linear problems. This introduces significant difficulties for Kalman filters. We've looked at how nonlinearity affects filtering in 3 different but equivalent ways, and I've given you a brief summary of the major appoaches: the linearized Kalman filter, the extended Kalman filter, the Unscented Kalman filter, and the particle filter.

Until recently the linearized Kalman filter and EKF have been the standard way to solve these problems. They are very difficult to understand and use, and they are also potentially very unstable.

Recent developments have offered what are to my mind superior approaches. The UKF dispenses with the need to find solutions to partial differential equations, yet it is also usually more accurate than the EKF. It is easy to use and understand. I can get a basic UKF going in a few minutes by using FilterPy. The particle filter dispenses with mathimatical modeling completely in favor of a Monte Carlo technique of generating a random cloud of thousands of points. It runs slowly, but it can solve otherwise intractable problems with relative ease.

I get more email about the EKF than anything else; I suspect that this is because most treatments in books, papers, and on the internet use the EKF. If your interest is in mastering the field of course you will want to learn about the EKF. But if you are just trying to get good results I point you to the UKF and particle filter first. They are much easier to implement, understand, and use, and they are typically far more stable than the EKF.

Some will quibble with that advice. A lot of recent publications are devoted to a comparison of the EKF, UKF, and perhaps a few other choices for a given problem. Do you not need to perform a similar comparison for your problem? If you are sending a rocket to Mars then of course you do. You will be balancing issues such as accuracy, round off errors, divergence, mathematical proof of correctness, and the computational effort required. I can't imagine not knowing the EKF intimately.

On the other hand the UKF works spectacularly! I use it at work for real world applications. I mostly haven't even tried to implement an EKF for these applications because I can verify that the UKF is working fine. Is it possible that I might eke out another 0.2% of performance from the EKF in certain situations? Sure! Do I care? No! I completely understand the UKF implementation, it is easy to test and verify, I can pass the code to others and be confident that they can understand and modify it, and I am not a masochist that wants to battle difficult equations when I already have a working solution. If the UKF or particle filters start to perform poorly for some problem then I will turn other techniques, but not before then. And realistically, the UKF usually provides substantially better performance than the EKF over a wide range of problems and conditions. If "really good" is good enough I'm going to spend my time working on other problems.

I'm belaboring this point because in most textbooks the EKF is given center stage, and the UKF is either not mentioned at all or just given a 2 page gloss that leaves you completely unprepared to use the filter. The UKF is still relatively new, and it takes time to write new editions of books. At the time many books were written the UKF was either not discovered yet, or it was just an unproven but promising curiosity. But I am writing this now, the UKF has had enormous success, and it needs to be in your toolkit. That is what I will spend most of my effort trying to teach you. _____no_output_____## References

[1] https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/Supporting_Notebooks/Computing_and_plotting_PDFs.ipynb_____no_output_____


# Notebook from nestorsgarzonc/Data-Science-Proyects
Path: The GitHub History of the Scala Language/notebook.ipynb

## 1. Scala's real-world project repository data

With almost 30k commits and a history spanning over ten years, Scala is a mature programming language. It is a general-purpose programming language that has recently become another prominent language for data scientists.

Scala is also an open source project. Open source projects have the advantage that their entire development histories -- who made changes, what was changed, code reviews, etc. -- publicly available.

We're going to read in, clean up, and visualize the real world project repository of Scala that spans data from a version control system (Git) as well as a project hosting site (GitHub). We will find out who has had the most influence on its development and who are the experts.

The dataset we will use, which has been previously mined and extracted directly from GitHub, is comprised of two files:

pulls.csv contains the basic information about the pull requests.

pull_files.csv contains the files that were modified by each pull request.

_____no_output_____

# Importing pandas
import pandas as pd

# Loading in the data
pulls = pd.read_csv('datasets/pulls.csv')
pull_files = pd.read_csv('datasets/pull_files.csv')_____no_output_____

## 2. Cleaning the data

The raw data extracted from GitHub contains dates in the ISO8601 format. However, pandas imports them as regular strings. To make our analysis easier, we need to convert the strings into Python's DateTime objects. DateTime objects have the important property that they can be compared and sorted.

The pull request times are all in UTC (also known as Coordinated Universal Time). The commit times, however, are in the local time of the author with time zone information (number of hours difference from UTC). To make comparisons easy, we should convert all times to UTC.
_____no_output_____

# Convert the date for the pulls object
pulls['date']=pd.to_datetime(pulls['date'] ,utc=True)_____no_output_____

## 3. Merging the DataFrames

The data extracted comes in two separate files. Merging the two DataFrames will make it easier for us to analyze the data in the future tasks.
_____no_output_____

# Merge the two DataFrames
data = pulls.merge(pull_files, on='pid')_____no_output_____

## 4. Is the project still actively maintained?

The activity in an open source project is not very consistent. Some projects might be active for many years after the initial release, while others can slowly taper out into oblivion. Before committing to contributing to a project, it is important to understand the state of the project. Is development going steadily, or is there a drop? Has the project been abandoned altogether?

The data used in this project was collected in January of 2018. We are interested in the evolution of the number of contributions up to that date.

For Scala, we will do this by plotting a chart of the project's activity. We will calculate the number of pull requests submitted each (calendar) month during the project's lifetime. We will then plot these numbers to see the trend of contributions.
_____no_output_____

%matplotlib inline

# Create a column that will store the month and the year, as a string
pulls['month_year'] = pulls.apply(
    lambda x: str(x['date'].year) + '-'+ str(x['date'].month),
    axis=1)

# Group by month_year and count the pull requests
counts = pulls.groupby('month_year').sum()

# Plot the results
counts.plot(kind='bar')_____no_output_____

## 5. Is there camaraderie in the project?

The organizational structure varies from one project to another, and it can influence your success as a contributor. A project that has a very small community might not be the best one to start working on. The small community might indicate a high barrier of entry. This can be caused by several factors, including a community that is reluctant to accept pull requests from "outsiders," that the code base is hard to work with, etc. However, a large community can serve as an indicator that the project is regularly accepting pull requests from new contributors. Such a project would be a good place to start.

In order to evaluate the dynamics of the community, we will plot a histogram of the number of pull requests submitted by each user. A distribution that shows that there are few people that only contribute a small number of pull requests can be used as in indicator that the project is not welcoming of new contributors.
_____no_output_____

# Required for matplotlib
%matplotlib inline
# Group by the submitter
by_user = pulls.groupby('user').sum()

# Plot the histogram
by_user.hist()_____no_output_____

## 6. What files were changed in the last ten pull requests?

Choosing the right place to make a contribution is as important as choosing the project to contribute to. Some parts of the code might be stable, some might be dead. Contributing there might not have the most impact. Therefore it is important to understand the parts of the system that have been recently changed. This allows us to pinpoint the "hot" areas of the code where most of the activity is happening. Focusing on those parts might not the most effective use of our times.
_____no_output_____

# Identify the last 10 pull requests
last_10 = pulls.nlargest(10, 'date')

# Join the two data sets
joined_pr = last_10.merge(pull_files, on='pid')

# Identify the unique files
files = set(joined_pr['file'].unique())

# Print the results
files_____no_output_____

## 7. Who made the most pull requests to a given file?

When contributing to a project, we might need some guidance. We might find ourselves needing some information regarding the codebase. It is important direct any questions to the right person. Contributors to open source projects generally have other day jobs, so their time is limited. It is important to address our questions to the right people. One way to identify the right target for our inquiries is by using their contribution history.

We identified src/compiler/scala/reflect/reify/phases/Calculate.scala as being recently changed. We are interested in the top 3 developers who changed that file. Those developers are the ones most likely to have the best understanding of the code.
_____no_output_____

# This is the file we are interested in:
file = 'src/compiler/scala/reflect/reify/phases/Calculate.scala'

# Identify the commits that changed the file
file_pr = data[data['file']==file]

# Count the number of changes made by each developer
author_counts = file_pr.groupby('user').count()

# Print the top 3 developers
print(author_counts.nlargest(3, 'pid'))          pid  date  file
user
xeno-by    11    11    11
retronym    5     5     5
soc         4     4     4

## 8. Who made the last ten pull requests on a given file?

Open source projects suffer from fluctuating membership. This makes the problem of finding the right person more challenging: the person has to be knowledgeable and still be involved in the project. A person that contributed a lot in the past might no longer be available (or willing) to help. To get a better understanding, we need to investigate the more recent history of that particular part of the system.

Like in the previous task, we will look at the history of  src/compiler/scala/reflect/reify/phases/Calculate.scala.
_____no_output_____

file = 'src/compiler/scala/reflect/reify/phases/Calculate.scala'

# Select the pull requests that changed the target file
file_pr = pull_files[pull_files['file']==file]

# Merge the obtained results with the pulls DataFrame
joined_pr = file_pr.merge(pulls, on='pid')

# Find the users of the last 10 most recent pull requests
users_last_10 = set(joined_pr.nlargest(10, 'date')['user'])

# Printing the results
users_last_10_____no_output_____

## 9. The pull requests of two special developers

Now that we have identified two potential contacts in the projects, we need to find the person who was most involved in the project in recent times. That person is most likely to answer our questions. For each calendar year, we are interested in understanding the number of pull requests the authors submitted. This will give us a high-level image of their contribution trend to the project.
_____no_output_____

%matplotlib inline

# The developers we are interested in
authors = ['xeno-by', 'soc']

# Get all the developers' pull requests
by_author = pulls[pulls['user'].isin(authors)]

# Count the number of pull requests submitted each year
counts = by_author.groupby(['user', pulls['date'].dt.year]).agg({'pid': 'count'}).reset_index()

# Convert the table to a wide format
counts_wide = counts.pivot_table(index='date', columns='user', values='pid', fill_value=0)

# Plot the results
counts_wide.plot(kind='bar')_____no_output_____data.head()_____no_output_____

## 10. Visualizing the contributions of each developer

As mentioned before, it is important to make a distinction between the global expertise and contribution levels and the contribution levels at a more granular level (file, submodule, etc.) In our case, we want to see which of our two developers of interest have the most experience with the code in a given file. We will measure experience by the number of pull requests submitted that affect that file and how recent those pull requests were submitted.
_____no_output_____

authors = ['xeno-by', 'soc']
file = 'src/compiler/scala/reflect/reify/phases/Calculate.scala'

# Select the pull requests submitted by the authors, from the `data` DataFrame
by_author = data[data['user'].isin(authors)]

# Select the pull requests that affect the file
by_file = by_author[by_author['file']==file]

# Group and count the number of PRs done by each user each year
grouped = by_file.groupby(['user', by_file['date'].dt.year]).count()['pid'].reset_index()

# Transform the data into a wide format
by_file_wide = pd.pivot_table(grouped)

# Plot the results
by_file_wide.plot(kind='bar')_____no_output_____


# Notebook from softbear/squidpy_notebooks
Path: docs/source/auto_tutorials/tutorial_seqfish.ipynb

%matplotlib inline_____no_output_____

Analyze seqFISH data
====================

This tutorial shows how to apply Squidpy for the analysis of seqFISH
data.

The data used here was obtained from lohoff2020highly. We provide a
pre-processed subset of the data, in anndata.AnnData format. For details
on how it was pre-processed, please refer to the original paper.

Import packages & data
----------------------

To run the notebook locally, create a conda environment as *conda create
-f environment.yml* using this
[environment.yml](https://github.com/theislab/squidpy_notebooks/blob/master/environment.yml)
_____no_output_____

import scanpy as sc
import squidpy as sq

import numpy as np

sc.logging.print_header()
print(f"squidpy=={sq.__version__}")

# load the pre-processed dataset
adata = sq.datasets.seqfish()_____no_output_____

First, let's visualize cluster annotation in spatial context with
scanpy.pl.spatial.
_____no_output_____

sc.pl.spatial(adata, color="celltype_mapped_refined", spot_size=0.03)_____no_output_____

Neighborhood enrichment analysis
================================

Similar to other spatial data, we can investigate spatial organization
of clusters in a quantitative way, by computing a neighborhood
enrichment score. You can compute such score with the following
function: squidpy.gr.nhood\_enrichment. In short, it's an enrichment
score on spatial proximity of clusters: if spots belonging to two
different clusters are often close to each other, then they will have a
high score and can be defined as being *enriched*. On the other hand, if
they are far apart, the score will be low and they can be defined as
*depleted*. This score is based on a permutation-based test, and you can
set the number of permutations with the n\_perms argument (default is
1000).

Since the function works on a connectivity matrix, we need to compute
that as well. This can be done with squidpy.gr.spatial\_neighbors.
Please see
sphx\_glr\_auto\_examples\_graph\_compute\_spatial\_neighbors.py for
more details of how this function works.

Finally, we'll directly visualize the results with
squidpy.pl.nhood\_enrichment. We'll add a dendrogram to the heatmap
computed with linkage method *ward*.
_____no_output_____

sq.gr.spatial_neighbors(adata)
sq.gr.nhood_enrichment(adata, cluster_key="celltype_mapped_refined")
sq.pl.nhood_enrichment(adata, cluster_key="celltype_mapped_refined", method="ward")_____no_output_____

A similar analysis was performed in the original publication
lohoff2020highly, and we can appreciate to what extent results overlap.
For instance, there seems to be an enrichment between the *Lateral plate
mesoderm*, the *Intermediate mesoderm* and a milder enrichment for
*Allantois* cells. As in the original publication, there also seems to
be an association between the *Endothelium* and the *Haematoendothelial
progenitors*. Of course, results do not perfectly overlap, and this
could be due to several factors:

-   the construction of the neighbors graph (which in our case is not
    informed by the radius, as we did not have access to this
    information) and by
-   the number of permutation of the neighborhood enrichment (500 in the
    original publication against the default 1000 in our
    implementation).

We can also visualize the spatial organization of cells again, and
appreciate the proximity of specific cell clusters. For this, we'll use
scanpy.pl.spatial again.
_____no_output_____

sc.pl.spatial(
    adata,
    color="celltype_mapped_refined",
    groups=[
        "Endothelium",
        "Haematoendothelial progenitors",
        "Allantois",
        "Lateral plate mesoderm",
        "Intermediate mesoderm",
        "Presomitic mesoderm",
    ],
    spot_size=0.03,
)_____no_output_____

Co-occurrence across spatial dimensions
=======================================

In addition to the neighbor enrichment score, we can visualize cluster
co-occurrence in spatial dimensions. This is a similar analysis of the
one presented above, yet it does not operate on the connectivity matrix,
but on the original spatial coordinates. The co-occurrence score is
defined as:

$$\frac{p(exp|cond)}{p(exp)}$$

where $p(exp|cond)$ is the conditional probability of observing a
cluster $exp$ conditioned on the presence of a cluster $cond$, whereas
$p(exp)$ is the probability of observing $exp$ in the radius size of
interest. The score is computed across increasing radii size around each
cell in the tissue.

We can compute this score with squidpy.gr.co\_occurrence and set the
cluster annotation for the conditional probability with the argument
`clusters`. Then, we visualize the results with
squidpy.pl.co\_occurrence.
_____no_output_____

sq.gr.co_occurrence(adata, cluster_key="celltype_mapped_refined")
sq.pl.co_occurrence(
    adata,
    cluster_key="celltype_mapped_refined",
    clusters="Lateral plate mesoderm",
    figsize=(10, 5),
)_____no_output_____

It seems to recapitulate a previous observation, that there is a
co-occurrence between the conditional cell type annotation *Lateral
plate mesoderm* and the clusters *Intermediate mesoderm* and
*Allantois*. It also seems that at longer distances, there is a
co-occurrence of cells belonging to the *Presomitic mesoderm* cluster.
By visualizing the full tissue as before we can indeed appreciate that
these cell types seems to form a defined clusters relatively close to
the *Lateral plate mesoderm* cells. It should be noted that the distance
units corresponds to the spatial coordinates saved in
adata.obsm\["spatial"\].
_____no_output_____Ligand-receptor interaction analysis
====================================

The analysis showed above has provided us with quantitative information
on cellular organization and communication at the tissue level. We might
be interested in getting a list of potential candidates that might be
driving such cellular communication. This naturally translates in doing
a ligand-receptor interaction analysis. In Squidpy, we provide a fast
re-implementation the popular method CellPhoneDB cellphonedb
([code](https://github.com/Teichlab/cellphonedb) ) and extended its
database of annotated ligand-receptor interaction pairs with the popular
database *Omnipath* omnipath. You can run the analysis for all clusters
pairs, and all genes (in seconds, without leaving this notebook), with
squidpy.gr.ligrec.

Let's perform the analysis and visualize the result for three clusters
of interest: *Lateral plate mesoderm*, *Intermediate mesoderm* and
*Allantois*. For the visualization, we will filter out annotations with
low-expressed genes (with the `means_range` argument) and decreasing the
threshold for the adjusted p-value (with the `alpha` argument)
_____no_output_____

sq.gr.ligrec(
    adata,
    n_perms=100,
    cluster_key="celltype_mapped_refined",
)
sq.pl.ligrec(
    adata,
    cluster_key="celltype_mapped_refined",
    source_groups="Lateral plate mesoderm",
    target_groups=["Intermediate mesoderm", "Allantois"],
    means_range=(0.3, np.inf),
    alpha=1e-4,
    swap_axes=True,
)_____no_output_____

The dotplot visualization provides an interesting set of candidate
interactions that could be involved in the tissue organization of the
cell types of interest. It should be noted that this method is a pure
re-implementation of the original permutation-based test, and therefore
retains all its caveats and should be interpreted accordingly.
_____no_output_____


# Notebook from dltech-xyz/d2l-java
Path: chapter_computational-performance/hardware.ipynb

# Hardware
:label:`sec_hardware`

Building systems with great performance requires a good understanding of the algorithms and models to capture the statistical aspects of the problem. At the same time it is also indispensable to have at least a modicum of knowledge of the underlying hardware. The current section is no substitute for a proper course on hardware and systems design. Instead, it might serve as a starting point for understanding why some algorithms are more efficient than others and how to achieve good throughput. Good design can easily make a difference of an order of magnitude and, in turn, this can make the difference between being able to train a network (e.g., in a week) or not at all (in 3 months, thus missing the deadline). We will start by looking at computers. Then we will zoom in to look more carefully at CPUs and GPUs. Lastly we zoom out to review how multiple computers are connected in a server center or in the cloud. This is not a GPU purchase guide. For this review :numref:`sec_buy_gpu`. An introduction to cloud computing with AWS can be found in :numref:`sec_aws`.

Impatient readers may be able to get by with :numref:`fig_latencynumbers`. It is taken from Colin Scott's [interactive post](https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html) which gives a good overview of the progress over the past decade. The original numbers are due to Jeff Dean's [Stanford talk from 2010](https://static.googleusercontent.com/media/research.google.com/en//people/jeff/Stanford-DL-Nov-2010.pdf). The discussion below explains some of the rationale for these numbers and how they can guide us in designing algorithms. The discussion below is very high level and cursory. It is clearly *no substitute* for a proper course but rather just meant to provide enough information for a statistical modeler to make suitable design decisions. For an in-depth overview of computer architecture we refer the reader to :cite:`Hennessy.Patterson.2011` or a recent course on the subject, such as the one by [Arste Asanovic](http://inst.eecs.berkeley.edu/~cs152/sp19/).

![Latency Numbers every Programmer should know.](https://github.com/d2l-ai/d2l-en/blob/master/img/latencynumbers.png?raw=true)
:label:`fig_latencynumbers`

## Computers

Most deep learning researchers have access to a computer with a fair amount of memory, compute, some form of an accelerator such as a GPU, or multiples thereof. It consists of several key components:

* A processor, also referred to as CPU which is able to execute the programs we give it (in addition to running an operating system and many other things), typically consisting of 8 or more cores.
* Memory (RAM) to store and retrieve the results from computation, such as weight vectors, activations, often training data.
* An Ethernet network connection (sometimes multiple) with speeds ranging from 1Gbit/s to 100Gbit/s (on high end servers more advanced interconnects can be found).
* A high speed expansion bus (PCIe) to connect the system to one or more GPUs. Severs have up to 8 accelerators, often connected in an advanced topology, desktop systems have 1-2, depending on the budget of the user and the size of the power supply.
* Durable storage, such as a magnetic harddrive (HDD), solid state (SSD), in many cases connected using the PCIe bus, provides efficient transfer of training data to the system and storage of intermediate checkpoints as needed.

![Connectivity of components](https://raw.githubusercontent.com/d2l-ai/d2l-en/master/img/mobo-symbol.svg)
:label:`fig_mobo-symbol`

As :numref:`fig_mobo-symbol` indicates, most components (network, GPU, storage) are connected to the CPU across the PCI Express bus. It consists of multiple lanes that are directly attached to the CPU. For instance AMD's Threadripper 3 has 64 PCIe 4.0 lanes, each of which is capable 16 Gbit/s data transfer in both directions. The memory is directly attached to the CPU with a total bandwidth of up to 100 GB/s.

When we run code on a computer we need to shuffle data to the processors (CPU or GPU), perform computation and then move the results off the processor back to RAM and durable storage. Hence, in order to get good performance we need to make sure that this works seamlessly without any one of the systems becoming a major bottleneck. For instance, if we cannot load images quickly enough the processor will not have any work to do. Likewise, if we cannot move matrices quickly enough to the CPU (or GPU), its processing elements will starve. Finally, if we want to synchronize multiple computers across the network, the latter should not slow down computation. One option is to interleave communication and computation. Let us have a look at the various components in more detail.

## Memory

At its most basic memory is used to store data that needs to be readily accessible. At present CPU RAM is typically of the [DDR4](https://en.wikipedia.org/wiki/DDR4_SDRAM) variety, offering 20-25GB/s bandwidth per module. Each module has a 64 bit wide bus. Typically pairs of memory modules are used to allow for multiple channels. CPUs have between 2 and 4 memory channels, i.e., they have between 40GB/s and 100GB/s peak memory bandwidth. Often there are two banks per channel. For instance AMD's Zen 3 Threadripper has 8 slots.

While these numbers are impressive, indeed, they only tell part of the story. When we want to read a portion from memory we first need to tell the memory module where the information can be found. That is, we first need to send the *address* to RAM. Once this accomplished we can choose to read just a single 64bit record or a long sequence of records. The latter is called *burst read*. In a nutshell, sending an address to memory and setting up the transfer takes approximately 100ns (details depend on the specific timing coefficients of the memory chips used), every subsequent transfer takes only 0.2ns. In short, the first read is 500 times as expensive as subsequent ones! We could perform up to $10,000,000$ random reads per second. This suggests that we avoid random memory access as far as possible and use burst reads (and writes) instead.

Matters are a bit more complex when we take into account that we have multiple banks. Each bank can read memory largely independently. This means two things: the effective number of random reads is up to 4x higher, provided that they are spread evenly across memory. It also means that it is still a bad idea to perform random reads since burst reads are 4x faster, too. Secondly, due to memory alignment to 64 bit boundaries it is a good idea to align any datastructures with the same boundaries. Compilers do this pretty much [automatically](https://en.wikipedia.org/wiki/Data_structure_alignment) when the appropriate flags are set. Curious readers are encouraged to review a lecture on DRAMs such as the one by [Zeshan Chishti](http://web.cecs.pdx.edu/~zeshan/ece585_lec5.pdf).

GPU memory is subject to even higher bandwidth requirements since they have many more processing elements than CPUs. By and large there are two options to address them. One is to make the memory bus significantly wider. For instance NVIDIA's RTX 2080 Ti has a 352 bit wide bus. This allows for much more information to be transferred at the same time. Secondly, GPUs use specific high-performance memory. Consumer grade devices, such as NVIDIA's RTX and Titan series typically use [GDDR6](https://en.wikipedia.org/wiki/GDDR6_SDRAM) chips with over 500 GB/s aggregate bandwidth. An alternative is to use HBM (high bandwidth memory) modules. They use a very different interface and connect directly with GPUs on a dedicated silicon wafer. This makes them very expensive and their use is typically limited to high end server chips, such as the NVIDIA Volta V100 series of accelerators. Quite unsurprisingly GPU memory is *much* smaller than CPU memory due to its higher cost. For our purposes, by and large their performance characteristics are similar, just a lot faster. We can safely ignore the details for the purpose of this book. They only matter when tuning GPU kernels for high throughput.

## Storage

We saw that some of the key characteristics of RAM were *bandwidth* and *latency*. The same is true for storage devices, just that the differences can be even more extreme.

**Hard Disks** have been in use for over half a century. In a nutshell they contain a number of spinning platters with heads that can be positioned to read / write at any given track. High end end disks hold up to 16TB on 9 platters. One of the key benefits of HDDs is that they are relatively inexpensive. One of their many downsides are their typically catastrophic failure modes and their relatively high read latency.

To understand the latter, consider the fact that HDDs spin at around 7,200 RPM. If they were much faster they would shatter due to the centrifugal force exerted on the platters. This has a major downside when it comes to accessing a specific sector on the disk: we need to wait until the platter has rotated in position (we can move the heads but not accelerate the actual disks). Hence it can take over 8ms until the requested data is available. A common way this is expressed is to say that HDDs can operate at approximately 100 IOPs. This number has essentially remained unchanged for the past two decades. Worse still, it is equally difficult to increase bandwidth (it is in the order of 100-200 MB/s). After all, each head reads a track of bits, hence the bit rate only scales with the square root of the information density. As a result HDDs are quickly becoming relegated to archival storage and low-grade storage for very large datasets.

**Solid State Drives** use Flash memory to store information persistently. This allows for *much faster* access to stored records. Modern SSDs can operate at 100,000 to 500,000 IOPs, i.e., up to 3 orders of magnitude faster than HDDs. Furthermore, their bandwidth can reach 1-3GB/s, i.e., one order of magnitude faster than HDDs. These improvements sound almost too good to be true. Indeed, they come with a number of caveats, due to the way SSDs are designed.

* SSDs store information in blocks (256 KB or larger). They can only be written as a whole, which takes significant time. Consequently bit-wise random writes on SSD have very poor performance. Likewise, writing data in general takes significant time since the block has to be read, erased and then rewritten with new information. By now SSD controllers and firmware have developed algorithms to mitigate this. Nonetheless writes can be much slower, in particular for QLC (quad level cell) SSDs. The key for improved performance is to maintain a *queue* of operations, to prefer reads and to write in large blocks if possible.
* The memory cells in SSDs wear out relatively quickly (often already after a few thousand writes). Wear-level protection algorithms are able to spread the degradation over many cells. That said, it is not recommended to use SSDs for swap files or for large aggregations of log-files.
* Lastly, the massive increase in bandwidth has forced computer designers to attach SSDs directly to the PCIe bus. The drives capable of handling this, referred to as NVMe (Non Volatile Memory enhanced), can use up to 4 PCIe lanes. This amounts to up to 8GB/s on PCIe 4.0.

**Cloud Storage** provides a configurable range of performance. That is, the assignment of storage to virtual machines is dynamic, both in terms of quantity and in terms speed, as chosen by the user. We recommend that the user increase the provisioned number of IOPs whenever latency is too high, e.g., during training with many small records.

## CPUs

Central Processing Units (CPUs) are the centerpiece of any computer (as before we give a very high level description focusing primarily on what matters for efficient deep learning models). They consist of a number of key components: processor cores which are able to execute machine code, a bus connecting them (the specific topology differs significantly between processor models, generations and vendors), and caches to allow for higher bandwidth and lower latency memory access than what is possible by reads from main memory. Lastly, almost all modern CPUs contain vector processing units to aid with high performance linear algebra and convolutions, as they are common in media processing and machine learning.

![Intel Skylake consumer quad-core CPU](https://raw.githubusercontent.com/d2l-ai/d2l-en/master/img/skylake.svg)
:label:`fig_skylake`

:numref:`fig_skylake` depicts an Intel Skylake consumer grade quad-core CPU. It has an integrated GPU, caches, and a ringbus connecting the four cores. Peripherals (Ethernet, WiFi, Bluetooth, SSD controller, USB, etc.) are either part of the chipset or directly attached (PCIe) to the CPU.

### Microarchitecture

Each of the processor cores consists of a rather sophisticated set of components. While details differ between generations and vendors, the basic functionality is pretty much standard. The front end loads instructions and tries to predict which path will be taken (e.g., for control flow). Instructions are then decoded from assembly code to microinstructions. Assembly code is often not the lowest level code that a processor executes. Instead, complex instructions may be decoded into a set of more lower level operations. These are then processed by the actual execution core. Often the latter is capable of performing many operations simultaneously. For instance, the ARM Cortex A77 core of :numref:`fig_cortexa77` is able to perform up to 8 operations simultaneously.

![ARM Cortex A77 Microarchitecture Overview](https://raw.githubusercontent.com/d2l-ai/d2l-en/master/img/a77.svg)
:label:`fig_cortexa77`

This means that efficient programs might be able to perform more than one instruction per clock cycle, *provided that* they can be carried out independently. Not all units are created equal. Some specialize in integer instructions whereas others are optimized for floating point performcne. To increase throughput the processor might also follow  multiple codepaths simultaneously in a branching instruction and then discard the results of the branch not taken. This is why branch prediction units matter (on the frontend) such that only the most promising paths are pursued.

### Vectorization

Deep learning is extremely compute hungry. Hence, to make CPUs suitable for machine learning one needs to perform many operations in one clock cycle. This is achieved via vector units. They have different names: on ARM they are called NEON, on x86 the latest generation is referred to as [AVX2](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions) units. A common aspect is that they are able to perform SIMD (single instruction multiple data) operations. :numref:`fig_neon128` shows how 8 short integers can be added in one clock cycle on ARM.

![128 bit NEON vectorization](https://raw.githubusercontent.com/d2l-ai/d2l-en/master/img/neon128.svg)
:label:`fig_neon128`

Depending on architecture choices such registers are up to 512 bit long, allowing for the combination of up to 64 pairs of numbers. For instance, we might be multiplying two numbers and adding them to a third, which is also known as a fused multiply-add. Intel's [OpenVino](https://01.org/openvinotoolkit) uses these to achieve respectable throughput for deep learning on server grade CPUs. Note, though, that this number is entirely dwarved by what GPUs are capable of achieving. For instance, NVIDIA's RTX 2080 Ti has 4,352 CUDA cores, each of which is capable of processing such an operation at any time.

### Cache

Consider the following situation: we have a modest CPU core with 4 cores as depicted in :numref:`fig_skylake` above, running at 2GHz frequency. Moreover, let us assume that we have an IPC (instructions per clock) count of 1 and that the units have AVX2 with 256bit width enabled. Let us furthermore assume that at least one of the registers used for AVX2 operations needs to be retrieved from memory. This means that the CPU consumes 4x256bit = 1kbit of data per clock cycle. Unless we are able to transfer $2 \cdot 10^9 \cdot 128 = 256 \cdot 10^9$ bytes to the processor per second the processing elements are going to starve. Unfortunately the memory interface of such a chip only supports 20-40 GB/s data transfer, i.e., one order of magnitude less. The fix is to avoid loading *new* data from memory as far as possible and rather to cache it locally on the CPU. This is where caches come in handy (see this [Wikipedia article](https://en.wikipedia.org/wiki/Cache_hierarchy) for a primer). Commonly the following names / concepts are used:

* **Registers** are strictly speaking not part of the cache. They help stage instructions. That said, CPU registers are memory locations that a CPU can access at clock speed without any delay penalty. CPUs have tens of registers. It is up to the compiler (or programmer) to use registers efficiently. For instance the C programming language has a `register` keyword.
* **L1** caches are the first line of defense against high memory bandwidth requirements. L1 caches are tiny (typical sizes might be 32-64kB) and often split into data and instructions caches. When data is found in the L1 cache access is very fast. If it cannot be found there, the search progresses down the cache hierarchy.
* **L2** caches are the next stop. Depending on architecture design and processor size they might be exclusive. They might be accessible only by a given core or shared between multiple cores. L2 caches are larger (typically 256-512kB per core) and slower than L1. Furthermore, to access something in L2 we first need to check to realize that the data is not in L1, which adds a small amount of extra latency.
* **L3** caches are shared between multiple cores and can be quite large. AMD's Epyc 3 server CPUs have a whopping 256MB of cache spread across multiple chiplets. More typical numbers are in the 4-8MB range.

Predicting which memory elements will be needed next is one of the key optimization parameters in chip design. For instance, it is advisable to traverse memory in a *forward* direction since most caching algorithms will try to *read ahead* rather than backwards. Likewise, keeping memory access patterns local is agood way of improving performance.
Adding caches is a double-edge sword. On one hand they ensure that the processor cores do not starve of data. At the same time they increase chip size, using up area that otherwise could have been spent on increasing processing power. Moreover, *cache misses* can be expensive. Consider the worst case scenario, depicted in :numref:`fig_falsesharing`. A memory location is cached on processor 0 when a thread on processor 1 requests the data. To obtain it, processor 0 needs to stop what it is doing, write the information back to main memory and then let processor 1 read it from memory. During this operation both processors wait. Quite potentially such code runs *more slowly* on multiple processors when compared to an efficient single-processor implementation. This is one more reason for why there is a practical limit to cache sizes (besides their physical size).

![False sharing (image courtesy of Intel)](https://raw.githubusercontent.com/d2l-ai/d2l-en/master/img/falsesharing.svg)
:label:`fig_falsesharing`

## GPUs and other Accelerators

It is not an exaggeration to claim that deep learning would not have been successful without GPUs. By the same token, it is quite reasonable to argue that GPU manufacturers' fortunes have been increased significantly due to deep learning. This co-evolution of hardware and algorithms has led to a situation where for better or worse deep learning is the preferable statistical modeling paradigm. Hence it pays to understand the specific benefits that GPUs and related accelerators such as the TPU :cite:`Jouppi.Young.Patil.ea.2017` offer.

Of note is a distinction that is often made in practice: accelerators are optimized either for training or inference. For the latter we only need to compute the forward propagation in a network. No storage of intermediate data is needed for backpropagation. Moreover, we may not need very precise computation (FP16 or INT8 typically suffice). On the other hand, during training all intermediate results need storing to compute gradients. Moreover, accumulating gradients requires higher precision to avoid numerical underflow (or overflow). This means that FP16 (or mixed precision with FP32) is the minimum required. All of this necessitates faster and larger memory (HBM2 vs. GDDR6) and more processing power. For instance, NVIDIA's [Turing](https://devblogs.nvidia.com/nvidia-turing-architecture-in-depth/) T4 GPUs are optimized for inference whereas the V100 GPUs are preferable for training.

Recall :numref:`fig_neon128`. Adding vector units to a processor core allowed us to increase throughput significantly (in the example in the figure we were able to perform 16 operations simultaneously). What if we added operations that optimized not just operations between vectors but also between matrices? This strategy led to Tensor Cores (more on this shortly). Secondly, what if we added many more cores? In a nutshell, these two strategies summarize the design decisions in GPUs. :numref:`fig_turing_processing_block` gives an overview over a basic processing block. It contains 16 integer and 16 floating point units. In addition to that, two Tensor Cores accelerate a narrow subset of additional operations relevant for deep learning. Each Streaming Multiprocessor (SM) consists of four such blocks.

![NVIDIA Turing Processing Block (image courtesy of NVIDIA)](https://github.com/d2l-ai/d2l-en/blob/master/img/turing_processing_block.png?raw=true)
:width:`150px`
:label:`fig_turing_processing_block`

12 streaming multiprocessors are then grouped into graphics processing clusters which make up the high-end TU102 processors. Ample memory channels and an L2 cache complement the setup. :numref:`fig_turing` has the relevant details. One of the reasons for designing such a device is that individual blocks can be added or removed as needed to allow for more compact chips and to deal with yield issues (faulty modules might not be activated). Fortunately programming such devices is well hidden from the casual deep learning researcher beneath layers of CUDA and framework code. In particular, more than one of the programs might well be executed simultaneously on the GPU, provided that there are available resources. Nonetheless it pays to be aware of the limitations of the devices to avoid picking models that do not fit into device memory.

![NVIDIA Turing Architecture (image courtesy of NVIDIA)](https://github.com/d2l-ai/d2l-en/blob/master/img/turing.png?raw=true)
:width:`350px`
:label:`fig_turing`

A last aspect that is worth mentioning in more detail are TensorCores. They are an example of a recent trend of adding more optimized circuits that are specifically effective for deep learning. For instance, the TPU added a systolic array :cite:`Kung.1988` for fast matrix multiplication. There the design was to support a very small number (one for the first generation of TPUs) of large operations. TensorCores are at the other end. They are optimized for small operations involving between 4x4 and 16x16 matrices, depending on their numerical precision. :numref:`fig_tensorcore` gives an overview of the optimizations.

![NVIDIA TensorCores in Turing (image courtesy of NVIDIA)](https://github.com/d2l-ai/d2l-en/blob/master/img/tensorcore.jpg?raw=true)
:width:`400px`
:label:`fig_tensorcore`

Obviously when optimizing for computation we end up making certain compromises. One of them is that GPUs are not very good at handling interrupts and sparse data. While there are notable exceptions, such as [Gunrock](https://github.com/gunrock/gunrock) :cite:`Wang.Davidson.Pan.ea.2016`, the access pattern of sparse matrices and vectors do not go well with the high bandwidth burst read operations where GPUs excel. Matching both goals is an area of active research. See e.g., [DGL](http://dgl.ai), a library tuned for deep learning on graphs.

## Networks and Buses

Whenever a single device is insufficient for optimization we need to transfer data to and from it to synchronize processing. This is where networks and buses come in handy. We have a number of design parameters: bandwidth, cost, distance and flexibility. On one end we have WiFi which has a pretty good range, is very easy to use (no wires, after all), cheap but it offers comparatively mediocre bandwidth and latency. No machine learning researcher within their right mind would use it to build a cluster of servers. In what follows we focus on interconnects that are suitable for deep learning.

* **PCIe** is a dedicated bus for very high bandwidth point to point connections (up to 16 Gbs on PCIe 4.0) per lane. Latency is in the order of single-digit microseconds (5 μs). PCIe links are precious. Processors only have a limited number of them: AMD's EPYC 3 has 128 lanes, Intel's Xeon has up to 48 lanes per chip; on desktop grade CPUs the numbers are 20 (Ryzen 9) and 16 (Core i9) respectively. Since GPUs have typically 16 lanes this limits the number of GPUs that can connect to the CPU at full bandwidth. After all, they need to share the links with other high bandwidth peripherals such as storage and Ethernet. Just like with RAM access, large bulk transfers are preferable due to reduced packet overhead.
* **Ethernet** is the most commonly used way of connecting computers. While it is significantly slower than PCIe, it is very cheap and resilient to install and covers much longer distances. Typical bandwidth for low-grade servers is 1 GBit/s. Higher end devices (e.g., [C5 instances](https://aws.amazon.com/ec2/instance-types/c5/) in the cloud) offer between 10 and 100 GBit/s bandwidth. As in all previous cases data transmission has significant overheads. Note that we almost never use raw Ethernet directly but rather a protocol that is executed on top of the physical interconnect (such as UDP or TCP/IP). This adds further overhead. Like PCIe, Ethernet is designed to connect two devices, e.g., a computer and a switch.
* **Switches** allow us to connect multiple devices in a manner where any pair of them can carry out a (typically full bandwidth) point to point connection simultaneously. For instance, Ethernet switches might connect 40 servers at high cross-sectional bandwidth. Note that switches are not unique to traditional computer networks. Even PCIe lanes can be [switched](https://www.broadcom.com/products/pcie-switches-bridges/pcie-switches). This occurs e.g., to connect a large number of GPUs to a host processor, as is the case for the [P2 instances](https://aws.amazon.com/ec2/instance-types/p2/).
* **NVLink** is an alternative to PCIe when it comes to very high bandwidth interconnects. It offers up to 300 Gbit/s data transfer rate per link. Server GPUs (Volta V100) have 6 links whereas consumer grade GPUs (RTX 2080 Ti) have only one link, operating at a reduced 100 Gbit/s rate. We recommend to use [NCCL](https://github.com/NVIDIA/nccl) to achieve high data transfer between GPUs.

## Summary

* Devices have overheads for operations. Hence it is important to aim for a small number of large transfers rather than many small ones. This applies to RAM, SSDs, Networks and GPUs.
* Vectorization is key for performance. Make sure you are aware of the specific abilities of your accelerator. E.g., some Intel Xeon CPUs are particularly good for INT8 operations, NVIDIA Volta GPUs excel at FP16 matrix-matrix operations and NVIDIA Turing shines at FP16, INT8 and INT4 operations.
* Numerical overflow due to small datatypes can be a problem during training (and to a lesser extent during inference).
* Aliasing can significantly degrade performance. For instance, memory alignment on 64 bit CPUs should be done with respect to 64 bit boundaries. On GPUs it is a good idea to keep convolution sizes aligned e.g., to TensorCores.
* Match your algorithms to the hardware (memory footprint, bandwidth, etc.). Great speedup (orders of magnitude) can be achieved when fitting the parameters into caches.
* We recommend that you sketch out the performance of a novel algorithm on paper before verifying the experimental results. Discrepancies of an order-of-magnitude or more are reasons for concern.
* Use profilers to debug performance bottlenecks.
* Training and inference hardware have different sweet spots in terms of price / performance.

## More Latency Numbers

The summary in :numref:`table_latency_numbers` and :numref:`table_latency_numbers_tesla` are due to [Eliot Eshelman](https://gist.github.com/eshelman) who maintains an updated version of the numbers as a [GitHub Gist](https://gist.github.com/eshelman/343a1c46cb3fba142c1afdcdeec17646).

:Common Latency Numbers.

| Action | Time | Notes |
| :----------------------------------------- | -----: | :---------------------------------------------- |
| L1 cache reference/hit                     | 1.5 ns | 4 cycles                                        |
| Floating-point add/mult/FMA                | 1.5 ns | 4 cycles                                        |
| L2 cache reference/hit                     |   5 ns | 12 ~ 17 cycles                                  |
| Branch mispredict                          |   6 ns | 15 ~ 20 cycles                                  |
| L3 cache hit (unshared cache)              |  16 ns | 42 cycles                                       |
| L3 cache hit (shared in another core)      |  25 ns | 65 cycles                                       |
| Mutex lock/unlock                          |  25 ns |                                                 |
| L3 cache hit (modified in another core)    |  29 ns | 75 cycles                                       |
| L3 cache hit (on a remote CPU socket)      |  40 ns | 100 ~ 300 cycles (40 ~ 116 ns)                  |
| QPI hop to a another CPU (per hop)         |  40 ns |                                                 |
| 64MB memory ref. (local CPU)          |  46 ns | TinyMemBench on Broadwell E5-2690v4             |
| 64MB memory ref. (remote CPU)         |  70 ns | TinyMemBench on Broadwell E5-2690v4             |
| 256MB memory ref. (local CPU)         |  75 ns | TinyMemBench on Broadwell E5-2690v4             |
| Intel Optane random write                  |  94 ns | UCSD Non-Volatile Systems Lab                   |
| 256MB memory ref. (remote CPU)        | 120 ns | TinyMemBench on Broadwell E5-2690v4             |
| Intel Optane random read                   | 305 ns | UCSD Non-Volatile Systems Lab                   |
| Send 4KB over 100 Gbps HPC fabric          |   1 μs | MVAPICH2 over Intel Omni-Path                   |
| Compress 1KB with Google Snappy            |   3 μs |                                                 |
| Send 4KB over 10 Gbps ethernet             |  10 μs |                                                 |
| Write 4KB randomly to NVMe SSD             |  30 μs | DC P3608 NVMe SSD (QOS 99% is 500μs)            |
| Transfer 1MB to/from NVLink GPU            |  30 μs | ~33GB/s on NVIDIA 40GB NVLink                 |
| Transfer 1MB to/from PCI-E GPU             |  80 μs | ~12GB/s on PCIe 3.0 x16 link                  |
| Read 4KB randomly from NVMe SSD            | 120 μs | DC P3608 NVMe SSD (QOS 99%)                     |
| Read 1MB sequentially from NVMe SSD        | 208 μs | ~4.8GB/s DC P3608 NVMe SSD                    |
| Write 4KB randomly to SATA SSD             | 500 μs | DC S3510 SATA SSD (QOS 99.9%)                   |
| Read 4KB randomly from SATA SSD            | 500 μs | DC S3510 SATA SSD (QOS 99.9%)                   |
| Round trip within same datacenter          | 500 μs | One-way ping is ~250μs                          |
| Read 1MB sequentially from SATA SSD        |   2 ms | ~550MB/s DC S3510 SATA SSD                    |
| Read 1MB sequentially from disk            |   5 ms | ~200MB/s server HDD                           |
| Random Disk Access (seek+rotation)         |  10 ms |                                                 |
| Send packet CA->Netherlands->CA            | 150 ms |                                                 |
:label:`table_latency_numbers`

:Latency Numbers for NVIDIA Tesla GPUs.

| Action | Time | Notes |
| :------------------------------ | -----: | :---------------------------------------- |
| GPU Shared Memory access        |  30 ns | 30~90 cycles (bank conflicts add latency) |
| GPU Global Memory access        | 200 ns | 200~800 cycles                            |
| Launch CUDA kernel on GPU       |  10 μs | Host CPU instructs GPU to start kernel    |
| Transfer 1MB to/from NVLink GPU |  30 μs | ~33GB/s on NVIDIA 40GB NVLink           |
| Transfer 1MB to/from PCI-E GPU  |  80 μs | ~12GB/s on PCI-Express x16 link         |
:label:`table_latency_numbers_tesla`

## Exercises

1. Write C code to test whether there is any difference in speed between accessing memory aligned or misaligned relative to the external memory interface. Hint: be careful of caching effects.
1. Test the difference in speed between accessing memory in sequence or with a given stride.
1. How could you measure the cache sizes on a CPU?
1. How would you lay out data across multiple memory channels for maximum bandwidth? How would you lay it out if you had many small threads?
1. An enterprise class HDD is spinning at 10,000 rpm. What is the absolutely minimum time an HDD needs to spend worst case before it can read data (you can assume that heads move almost instantaneously)? Why are 2.5" HDDs becoming popular for commercial servers (relative to 3.5" and 5.25" drives)?
1. Assume that an HDD manufacturer increases the storage density from 1 Tbit per square inch to 5 Tbit per square inch. How much information can you store on a ring on a 2.5" HDD? Is there a difference between the inner and outer tracks?
1. The AWS P2 instances have 16 K80 Kepler GPUs. Use `lspci` on a p2.16xlarge and a p2.8xlarge instance to understand how the GPUs are connected to the CPUs. Hint: keep your eye out for PCI PLX bridges.
1. Going from 8 bit to 16 bit datatypes increases the amount of silicon approximately by 4x. Why? Why might NVIDIA have added INT4 operations to their Turing GPUs.
1. Given 6 high speed links between GPUs (such as for the Volta V100 GPUs), how would you connect 8 of them? Look up the connectivity used in the P3.16xlarge servers.
1. How much faster is it to read forward through memory vs. reading backwards? Does this number differ between different computers and CPU vendors? Why? Write C code and experiment with it.
1. Can you measure the cache size of your disk? What is it for a typical HDD? Do SSDs need a cache?
1. Measure the packet overhead when sending messages across the Ethernet. Look up the difference between UDP and TCP/IP connections.
1. Direct Memory Access allows devices other than the CPU to write (and read) directly to (from) memory. Why is this a good idea?
1. Look at the performance numbers for the Turing T4 GPU. Why does the performance 'only' double as you go from FP16 to INT8 and INT4?
1. What is the shortest time it should take for a packet on a roundtrip between San Francisco and Amsterdam? Hint: you can assume that the distance is 10,000km._____no_output_____


# Notebook from MyanmarEITI/meiti-data
Path: tools/revenues/.ipynb_checkpoints/Read_Reconciliation_Sheets_15-16-checkpoint.ipynb

<code>
import openpyxl as opxl
import pandas as pd
import re
import unicodedata
_____no_output_____#wb1 = opxl.load_workbook('Annex 18 - Reconciliation sheets Oil & Gas 15-16.xlsx')
# wb1 = opxl.load_workbook('Annex 18 - Reconciliation sheets Gems & Jade 15-16.xlsx')
#print wb1.get_sheet_names()_____no_output_____xl = pd.ExcelFile('Annex 18 - Reconciliation sheets Gems & Jade 15-16.xlsx')
sheets_list = xl.sheet_names
#df = xls.parse('Sheet1', skiprows=4, index_col=None, na_values=['NA'])_____no_output_____def join_column_titles(text):
    #print(text)
    to_return = ""
    for t in text:
        #print(t)
        if not isinstance(t, float):
            to_return = '.'.join([to_return,(''.join(i for i in t if ord(i)<128))])
    #print("FINAL: " + to_return)

    # remove first '.' from the title
    return to_return[1:]_____no_output_____df_all = pd.DataFrame({'company_name' : [],'name_of_revenue_stream' : [],'value_reported' : []})

for s in sheets_list:
    try:
        #df = xl.parse('C (1)', skiprows=4, index_col=None, na_values=['NA'])
        name_df = xl.parse(s, parse_cols = 'C:E')
        company_name = name_df.columns[2]
        company_number = name_df[name_df['Company name:'] == 'Registry number'][company_name][0]
        print(company_name, company_number)

        df = xl.parse(s, skiprows=4, skip_footer=37, parse_cols = 'B:M', header=None)
        df.iloc[0:2] = df.iloc[0:2].fillna(method='ffill', axis=1)
        df.columns = df.iloc[0:2].apply(join_column_titles, axis=0)
        df = df.iloc[2:33]

        name = map(lambda x: ''.join(e for e in x if e.isalnum()) , company_name.split(' '))
        name = ' '.join(w for w in name)
        name = re.sub( '\s+', ' ', name.strip())
        filename = re.sub( '\s+', '_', name.strip())+'.csv'
    #         df.to_csv(filename)

        print(filename)
         df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Sale Split', \
                          'paid_to' : "Myanmar Gems Enterprise", \
                          'value_reported' : [df.loc[9,'Per Government.Final'][0]]})])
        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Sales Royalties', \
                          'paid_to' : "Myanmar Gems Enterprise", \
                          'value_reported' : [df.loc[10,'Per Government.Final'][0]]})])
        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Emporium Fees', \
                          'paid_to' : "Central Comittee", \
                          'value_reported' : [df.loc[25,'Per Government.Final'][0]]})])

        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Other significant payments', \
                          'paid_to' : "Myanmar Gems Enterprise", \
                          'value_reported' : [df.loc[23,'Per Government.Final'][0]]})])

        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Production Royalties', \
                          'paid_to' : "Myanmar Gems Enterprise", \
                          'value_reported' : [df.loc[7,'Per Government.Final'][0]]})])

        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Production Royalties', \
                          'paid_to' : "Department of Mines", \
                          'value_reported' : [df.loc[8,'Per Government.Final'][0]]})])

        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Permit Fees', \
                          'paid_to' : "Myanmar Gems Enterprise", \
                          'value_reported' : [df.loc[12,'Per Government.Final'][0]]})])

        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Incentive Fees', \
                          'paid_to' : "Myanmar Gems Enterprise", \
                          'value_reported' : [df.loc[13,'Per Government.Final'][0]]})])

        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Commercial Tax', \
                          'paid_to' : "Myanmar Gems Enterprise", \
                          'value_reported' : [df.loc[14,'Per Government.Final'][0]]})])

        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Corporate Income Tax', \
                          'paid_to' : "Internal Revenue Department", \
                          'value_reported' : [df.loc[19,'Per Government.Final'][0]]})])

        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Commercial Tax', \
                          'paid_to' : "Internal Revenue Department", \
                          'value_reported' : [df.loc[20,'Per Government.Final'][0]]})])

        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Customs Duties', \
                          'paid_to' : "Customs Department", \
                          'value_reported' : [df.loc[21,'Per Government.Final'][0]]})])

        df_all = pd.concat([df_all, \
            pd.DataFrame({'company_name' : [name], \
                          'name_of_revenue_stream' : 'Sales Royalties', \
                          'paid_to' : "Internal Revenue Department", \
                          'value_reported' : [df.loc[22,'Per Government.Final'][0]]})])
    #         print(df.columns)
    except:
        print("error parsing names from sheet \"" + s + "\"")(u'Great Genesis Gems Co; Ltd. (*)', u'636 / 2007 - 2008')
Great_Genesis_Gems_Co_Ltd.csv
(u'Sein Lom Taung Tan Gems Ltd. (*)', u'1273 of 1996-1997')
Sein_Lom_Taung_Tan_Gems_Ltd.csv
(u'Wai Aung Gabar Gems Co; Ltd. (*)', u'1999/2010-2011')
Wai_Aung_Gabar_Gems_Co_Ltd.csv
(u'Ayar Jade Co; Ltd.', u'919/ 1996-1997 (13-9-1996)')
Ayar_Jade_Co_Ltd.csv
(u'Shwe Pyi Tha Gems Trading & Faceting', u'696/2005-2006')
Shwe_Pyi_Tha_Gems_Trading_Faceting.csv
(u'Kyaing International Gems (*)', u'92 /2009-2010')
Kyaing_International_Gems.csv
(u'Linn Lett Win Yadanar Gems (*)', u'1043 / 1995-1996')
Linn_Lett_Win_Yadanar_Gems.csv
(u'Richest Gems Co;Ltd. (*)', u'1630/1999-2000')
Richest_Gems_CoLtd.csv
(u'Myat Yamon Gems Co;Ltd. (*)', u'448/95-96')
Myat_Yamon_Gems_CoLtd.csv
(u'Yadanar Taung Tann Gems ', u'7/2000-2001')
Yadanar_Taung_Tann_Gems.csv
(u'Khin Zaw Aung & Brothers Gems & Jewellery (*)', u'787 / 2000-2001')
Khin_Zaw_Aung_Brothers_Gems_Jewellery.csv
(u'Yar Za Htar Ne Gems Co;Ltd. (*)', u'1117/2007-2008')
Yar_Za_Htar_Ne_Gems_CoLtd.csv
(u'Kachin National Development', 1385)
Kachin_National_Development.csv
(u'Myanmar Imperial Jade(Gems & Jewellery) (*)', u'742/1994-1995')
Myanmar_Imperial_JadeGems_Jewellery.csv
(u'Myo Nwe Gems & Jewellery', u'978/2005-2006')
Myo_Nwe_Gems_Jewellery.csv
(u'Shining Star Light Gems & Jewellery (*)', u'1047/2007-2008 ( 23.11.2007 )')
Shining_Star_Light_Gems_Jewellery.csv
(u'Kan Pwint Oo Gems & Jewellery', u'990/ 2006-2007')
Kan_Pwint_Oo_Gems_Jewellery.csv
(u'Green Comet Gems ', u'780/2014-2015 ')
Green_Comet_Gems.csv
(u'Thi Raw Mani Gems & Jewellery (*)', u'942/1999-2000 (8-October-1999)')
Thi_Raw_Mani_Gems_Jewellery.csv
(u'Myanma Seinn Lei Aung Gems', u'4361/2012-2013')
Myanma_Seinn_Lei_Aung_Gems.csv
(u'Nay La Pwint Gems Co; Ltd. (*)', u'737/2007-2008')
Nay_La_Pwint_Gems_Co_Ltd.csv
(u'Myanmar Win Gate Gems & Jewellery Mining', nan)
Myanmar_Win_Gate_Gems_Jewellery_Mining.csv
(u'Khaing Lon Gems Co;Ltd.', u'357/1995-1996')
Khaing_Lon_Gems_CoLtd.csv
(u'Chaow Brothers Gemstone ', u'268/2001-2002')
Chaow_Brothers_Gemstone.csv
(u'Kyay Lin', u'901/2011-2012')
Kyay_Lin.csv
(u'Jade Mountain Gems Co;Ltd. (*)', u'2302/2011-2012')
Jade_Mountain_Gems_CoLtd.csv
(u'Phyo Thiha Kyaw Gems', u'164/2002-2003')
Phyo_Thiha_Kyaw_Gems.csv
(u'Silver Elephant Gems ', u'1066/2012-2013')
Silver_Elephant_Gems.csv
(u'Yadanar Yaung Chi Gems Co;Ltd. (*)', u'462 ( 1995 - 1996 )')
Yadanar_Yaung_Chi_Gems_CoLtd.csv
(u'Myanmar Sithu Jewellery (*)', nan)
Myanmar_Sithu_Jewellery.csv
(u'Pang Huke Duwa Co; Ltd.', nan)
Pang_Huke_Duwa_Co_Ltd.csv
(u'Kyay Sin Phyu', u'598 / 1999-2000')
Kyay_Sin_Phyu.csv
(u'SEIN THURA SAN GEMS COMPANY', nan)
SEIN_THURA_SAN_GEMS_COMPANY.csv
(u'Lucky Star', nan)
Lucky_Star.csv
(u'San Taw Win Gems & Jewellery', nan)
San_Taw_Win_Gems_Jewellery.csv
(u'Yadanar Kaung Kin Gems &JewelleryCo;Ltd', nan)
Yadanar_Kaung_Kin_Gems_JewelleryCoLtd.csv
(u'Mega Stone Co;Ltd', nan)
Mega_Stone_CoLtd.csv
(u'Yadanar San Shwin Gems Co., Ltd', nan)
Yadanar_San_Shwin_Gems_Co_Ltd.csv
(u'Golden Wallet Jewellery Co.,Ltd', u'432/1999-2000')
Golden_Wallet_Jewellery_CoLtd.csv
(u'SHWE BYAIN PHYU CO', u'220/3004-2005')
SHWE_BYAIN_PHYU_CO.csv
(u'NILAR YOMA TRADING', u'1675/1999-2000')
NILAR_YOMA_TRADING.csv
(u'Myanma Gon Yi Gems & Jewellery', u'872/2007-2008')
Myanma_Gon_Yi_Gems_Jewellery.csv
(u'Ayar Yadanar', u'252/2003-2004')
Ayar_Yadanar.csv
(u'Kaung Myat Thukha', u'816/1995-1996')
Kaung_Myat_Thukha.csv
(u'Unity Gems Co;Ltd.', u'363/ 1995-1996')
Unity_Gems_CoLtd.csv
(u'Thiha & Three Brothers Gems & Jewellery', u'2098 / 2005-2006')
Thiha_Three_Brothers_Gems_Jewellery.csv
(u'Emerald Garden', u'1113/20067-2007')
Emerald_Garden.csv
(u'Shwe Gaung Gaung Gems', u'2145 / 2005- 2006')
Shwe_Gaung_Gaung_Gems.csv
(u'Green Light Treasure \nCo.,Ltd', u'31 / 2006 -2007')
Green_Light_Treasure_CoLtd.csv
(u'Xie Family Co.,Ltd (*)', u'34 / 2006-2007')
Xie_Family_CoLtd.csv
(nan, u'51 / 2006-2007')
df_all.to_csv('2015-16_gems_jade.csv')
df_all
_____no_output_____
</code>