# Notebook from abhilashgupta/fuzzingbook
Path: notebooks/SearchBasedFuzzer.ipynb

# Search-Based Fuzzing

Sometimes we are not only interested in fuzzing as many as possible diverse program inputs, but in deriving *specific* test inputs that achieve some objective, such as reaching specific statements in a program. When we have an idea of what we are looking for, then we can *search* for it. Search algorithms are at the core of computer science, but applying classic search algorithms like breadth or depth first search to search for tests is unrealistic, because these algorithms potentially require us to look at all possible inputs. However, domain-knowledge can be used to overcome this problem. For example, if we can estimate which of several program inputs is closer to the one we are looking for, then this information can guide us to reach the target quicker – this information is known as a *heuristic*. The way heuristics are applied systematically is captured in *meta-heuristic* search algorithms. The "meta" denotes that these algorithm are generic and can be instantiated differently to different problems. Meta-heuristics often take inspiration from processes observed in nature. For example, there are algorithms mimicking evolutionary processes, swarm intelligence, or chemical reactions. In general they are much more efficient than exhaustive search approaches such that they can be applied to vast search spaces – search spaces as vast as the domain of program inputs are no problem for them._____no_output_____**Prerequisites**

* You should know how code coverage works, e.g. from the [chapter on coverage](Coverage.ipynb)._____no_output_____## Test Generation as a Search Problem

If we want to apply a meta-heuristic search algorithm to generate test data for a program, then we have to make several choices: First, we need to decide on what exactly our *search space* is in the first place. The search space is defined by how we *represent* what we are looking for. Are we looking for single integer values? Tuples of values? Objects? XML documents? _____no_output_____### Representing Program Inputs as a Search Problem

The representation is highly dependent on the particular testing problem we are solving --- we know which program we are testing, so the representation needs to encode whatever an input to our target program is. Let's consider the example function `test_me()` as our function under test: _____no_output_____

import fuzzingbook_utils_____no_output_____import Fuzzer_____no_output_____from fuzzingbook_utils import unicode_escape, terminal_escape_____no_output_____def test_me(x, y):
    if x == 2 * (y + 1):
        return True
    else:
        return False_____no_output_____

The `test_me()` function has two input parameters, and returns `True` or `False` depending on how the two relate to each other. A test input to `test_me()` consists of a pair of values, one for `x` and one for `y`. For example:_____no_output_____

test_me(0, 0)_____no_output_____test_me(4, 2)_____no_output_____test_me(22, 10)_____no_output_____

Our search space is only concerned with inputs, thus a simple representation for test data would be input tuples `(x, y)`. Each point in this input space has eight *neighbours*:

- `x-1, y-1`
- `x-1, y`
- `x-1, y+1`
- `x, y+1`
- `x+1, y+1`
- `x+1, y`
- `x+1, y-1`
- `x, y-1`

To keep things simple, let's restrict the size of our search space to start with (we will change this later). For example, let's assume we only want values in the range of -1000 to 1000:_____no_output_____

MAX = 1000
MIN = -MAX_____no_output_____

To retrieve the neighbours for any point in our search space, we define the function `neighbours()`, which implements a basic Moore neighbourhood. That is, we look at all 8 immediate neighbours, while considering the boundaries we just defined with `MAX` and `MIN`:_____no_output_____

def neighbours(x, y):
    return [(x + dx, y + dy) for dx in [-1, 0, 1]
            for dy in [-1, 0, 1]
            if (dx != 0 or dy != 0)
            and ((MIN <= x + dx <= MAX)
                 and (MIN <= y + dy <= MAX))]_____no_output_____print(neighbours(10, 10))_____no_output_____

This fully defines our search space: We have a representation, and we know how individuals are related to each other through their neighbourhood. Now we just need to find an algorithm to explore this neighbourhood, and a heuristic that guides the algorithm._____no_output_____### Defining a Search Landscape: Fitness functions

All meta-heuristics are based on the use of a heuristic function that estimates how good a given candidate solution is; this "goodness" is typically called the *fitness* of an individual, and the heuristic that estimates the fitness is the *fitness function*. The fitness function is a function that maps any point in the search space to a numerical value, the fitness value. The better a candidate solution in the search space with respect to being an optimal solution, the better its fitness value. Thus, if you plot each point in the search space with its fitness value as the height, you get a landscape with the optimal solution represented as the highest peak._____no_output_____The fitness function depends on the objective one would like to achieve with generating the test data. Suppose that we are interested in covering the true branch of the if-condition in the `test_me()` function, i.e. `x == 2 * (y + 1)`.

How close is a given input tuple for this function from reaching the target branch? Let's consider an arbitrary point in the search space, e.g. `(274, 153)`. The if-condition compares the following values:_____no_output_____

x = 274
y = 153
x, 2 * (y + 1)_____no_output_____

In order to make the branch true, both values need to be the same. Thus, the more they differ, the further we are away from making the comparison true, and the less they differ, the closer we are from making the comparison true. Thus, we can quantify "how false" the comparison is by calculating the difference between `x` and `2 * (y + 1)`. Thus, we can calculate this distance as `abs(x - 2 * (y + 1))`:_____no_output_____

def calculate_distance(x, y):
    return abs(x - 2 * (y + 1))_____no_output_____calculate_distance(274, 153)_____no_output_____

We can use this distance value as out fitness function, since we can nicely measure how close we are to an optimal solution. Note, however, that "better" doesn't mean "bigger" in this case; the smaller the distance the better. This is not a problem, since any algorithm that can maximize a value can also be made to minimize it instead.

For each value in the search space of integer tuples, this distance value defines the elevation in our search landscape. Since our example search space is two-dimensional, the search landscape is three-dimensional and we can plot it to see what it looks like:_____no_output_____

from mpl_toolkits.mplot3d import Axes3D_____no_output_____import matplotlib.pyplot as plt_____no_output_____import numpy as np_____no_output_____%matplotlib inline

x = np.outer(np.linspace(-10, 10, 30), np.ones(30))
y = x.copy().T
z = calculate_distance(x, y)

fig = plt.figure()
ax = plt.axes(projection='3d')

ax.plot_surface(x, y, z, cmap=plt.cm.jet, rstride=1, cstride=1, linewidth=0);_____no_output_____

The optimal values, i.e. those that make the if-condition true, have fitness value 0 and can be clearly seen at the bottom of the plot. The further away from the optimal values, the higher elevated the points in the search space._____no_output_____### Instrumentation

The fitness function should calculate the distance value for concrete test executions. That is, we want to run the program, and then learn the distance value of this execution. However, the branching condition is hidden within the source code of the target function, and its value may in principle be the result of various calculations along the execution path reaching it. Even though in our example the condition is an equation using the input values of the function directly, this may not be the case in general; it might as well be derived values. Thus, the values we need to calculate our distance metric need to be observed directly at the conditional statement.

This is typically done with *instrumentation*: We add new code immediately before or after the branching condition to keep track of the values observed and calculate the distance using these values. The following is an instrumented version of our program under test, which prints out the distance values as it is executed:_____no_output_____

def test_me_instrumented(x, y):
    print("Instrumentation: Input = (%d, %d), distance = %d" %
          (x, y, calculate_distance(x, y)))
    if x == 2 * (y + 1):
        return True
    else:
        return False_____no_output_____

Let's try this out for a couple of example values:_____no_output_____

test_me_instrumented(0, 0)_____no_output_____test_me_instrumented(5, 2)_____no_output_____test_me_instrumented(22, 10)_____no_output_____

When calculating a fitness value, we will execute the instrumented program version, but we need some means to access the distance value calculated during that execution. As a simple first solution to this problem, we can just add a global variable and store the value of the distance calculation there. _____no_output_____

distance = 0_____no_output_____def test_me_instrumented(x, y):
    global distance
    distance = calculate_distance(x, y)
    if x == 2 * (y + 1):
        return True
    else:
        return False_____no_output_____

Using this instrumented version of `test_me()`, we can now finally define our fitness function, which simply runs the instrumented `test_me_instrumented()` function, and then retrieves the value of the global `distance` variable:_____no_output_____

def get_fitness(x, y):
    global distance
    test_me_instrumented(x, y)
    fitness = distance
    return fitness_____no_output_____

Let's try this on some example inputs:_____no_output_____

get_fitness(0, 0)_____no_output_____get_fitness(1, 2)_____no_output_____get_fitness(22, 10)_____no_output_____

### Hillclimbing the Example

Having decided on a representation (2-tuples of integers) and a fitness function (distance to target branch), we can now finally go ahead and implement our search algorithm. Let's explore this search space using the simplest possible meta-heuristic algorithm: Hillclimbing. The metaphor captures aptly what is happening: The algorithm tries to climb a hill in the search space defined by our representation. Except, that in our search landscape the best values are not those high up but down low, so technically we are are descending into valleys.

The hillclimbing algorithm itself is very simple:
1. Take a random starting point
2. Determine fitness value of all neighbours
3. Move to neighbour with the best fitness value
4. If solution is not found, continue with step 2_____no_output_____The hillclimber starts with a random test input, i.e., random values for `x` and `y`. For any pair of random integer numbers, the chances of them satisfying the condition `x == 2 * (y + 1)` are rather slim. Suppose the random values are `(274, 153)`. The right hand side of the equation, `2 * (y + 1)`, evaluates to 308, so the condition is clearly false. Where should the hillclimber go to now? Let's look at the fitness values of this test input and its neighbours:_____no_output_____

x, y = 274, 153
print("Origin %d, %d has fitness %d" % (x, y, get_fitness(x, y)))
for nx, ny in neighbours(x, y):
    print("Neighbour %d, %d has fitness %d" % (nx, ny, get_fitness(nx, ny)))_____no_output_____

Increasing `y` by one increases the value of the right hand side of the equation to `310`. Thus, the value on the left hand side of the equation thus differs *even more* to the value on the right hand side of the equation than it did before the increase! So, increasing `y` does not seem like a good idea. On the other hand, increasing `x` by one improves things: The left hand side and the right hand side of the equation become more similar; they are "less unequal". Thus, out of the eight possible neighbors of `(274, 153)`, the neighbor that increases `x` and decreases `y` (`(275, 152)`) seems best intuitively --- the outcome of the condition is still false, but it is "less so" than for the original value._____no_output_____Let's now implement the hillcimbing algorithm. _____no_output_____

import random_____no_output_____LOG_VALUES = 20  # Number of values to log_____no_output_____def hillclimber():
    # Create and evaluate starting point
    x, y = random.randint(MIN, MAX), random.randint(MIN, MAX)
    fitness = get_fitness(x, y)
    print("Initial value: %d, %d at fitness %.4f" % (x, y, fitness))
    iterations = 0
    logs = 0

    # Stop once we have found an optimal solution
    while fitness > 0:
        iterations += 1
        # Move to first neighbour with a better fitness
        for (nextx, nexty) in neighbours(x, y):
            new_fitness = get_fitness(nextx, nexty)

            # Smaller fitness values are better
            if new_fitness < fitness:
                x, y = nextx, nexty
                fitness = new_fitness
                if logs < LOG_VALUES:
                    print("New value: %d, %d at fitness %.4f" % (x, y, fitness))
                elif logs == LOG_VALUES:
                    print("...")
                logs += 1
                break

    print("Found optimum after %d iterations at %d, %d" % (iterations, x, y))_____no_output_____hillclimber()_____no_output_____

The hillclimber starts by choosing random values for `x` and `y`. We use low values in the range of `-1000`--`1000` (which we defined `MIN` and `MAX` to be earlier) to reduce the time search takes when playing with the example. Then, we determine the fitness value of this starting point by calling `get_fitness()`. Recall that we are trying to find the smallest possible fitness value, therefore we now loop until we have found a fitness value of `0` (i.e., an optimal value). _____no_output_____In this loop, we iterate over all neighbours (`neighbours`), and evaluate the fitness value of each of the neighbours. As soon as we have found a neighbour with better (smaller) fitness, the hillclimber exits the loop and uses this as the new starting point. An alternative variant of this simple hillclimbing algorithm would be to remove the `break` statement: By doing so, *all* neighbours would be evaluated and the best neighbour would be chosen. This is known as *steepest ascent hillclimbing*. You will see that the number of iterations necessary to reach the optimum is lower, although for each iteration more tests are executed._____no_output_____

def steepest_ascent_hillclimber():
    # Create and evaluate starting point
    x, y = random.randint(MIN, MAX), random.randint(MIN, MAX)
    fitness = get_fitness(x, y)
    print("Initial value: %d, %d at fitness %.4f" % (x, y, fitness))
    iterations = 0
    logs = 0

    # Stop once we have found an optimal solution
    while fitness > 0:
        iterations += 1
        # Move to first neighbour with a better fitness
        for (nextx, nexty) in neighbours(x, y):
            new_fitness = get_fitness(nextx, nexty)
            if new_fitness < fitness:
                x, y = nextx, nexty
                fitness = new_fitness
                if logs < LOG_VALUES:
                    print("New value: %d, %d at fitness %.4f" % (x, y, fitness))
                elif logs == LOG_VALUES:
                    print("...")
                logs += 1

    print("Found optimum after %d iterations at %d, %d" % (iterations, x, y))_____no_output_____steepest_ascent_hillclimber()_____no_output_____

Our example program has a very nice fitness landscape – there is a perfect gradient, and the hillclimber will always find a solution. We can see this nice gradient if we plot the fitness values observed over time:_____no_output_____

def plotting_hillclimber(fitness_function):
    data = []

    # Create and evaluate starting point
    x, y = random.randint(MIN, MAX), random.randint(MIN, MAX)
    fitness = fitness_function(x, y)
    data += [fitness]
    iterations = 0

    # Stop once we have found an optimal solution
    while fitness > 0:
        iterations += 1
        # Move to first neighbour with a better fitness
        for (nextx, nexty) in neighbours(x, y):
            new_fitness = fitness_function(nextx, nexty)
            if new_fitness < fitness:
                x, y = nextx, nexty
                fitness = new_fitness
                data += [fitness]
                break

    print("Found optimum after %d iterations at %d, %d" % (iterations, x, y))
    return data_____no_output_____data = plotting_hillclimber(get_fitness)_____no_output_____import matplotlib.pyplot as plt_____no_output_____fig = plt.figure()
ax = plt.axes()

x = range(len(data))
ax.plot(x, data);_____no_output_____

This gradient is the result of an ideal fitness landscape. In practice, we won't always have such a nice gradient. In particular, our hillclimber only works well as long as there is at least one neighbour that has a better fitness value. What if we reach a point where _none_ of the neighbours actually has a better fitness value? Consider the following function `test_me2`:_____no_output_____

def test_me2(x, y):
    if(x * x == y * y * (x % 20)):
        return True
    else:
        return False_____no_output_____

If we want to cover the true-branch of the if-condition in `test_me2` again, then we can calculate the distance in the same way as previously, i.e., by calculating the difference between the two sides of the comparison. Let's instrument the `test_me2()` function:_____no_output_____

def test_me2_instrumented(x, y):
    global distance
    distance = abs(x * x - y * y * (x % 20))
    if(x * x == y * y * (x % 20)):
        return True
    else:
        return False_____no_output_____

With this instrumented version, we just need a fitness function that calls the instrumented version and reads out the global `distance` variable._____no_output_____

def bad_fitness(x, y):
    global distance
    test_me2_instrumented(x, y)
    fitness = distance
    return fitness_____no_output_____

Before we run the hillclimber on this example, let's have a look at the search landscape again by plotting it:_____no_output_____

from mpl_toolkits.mplot3d import Axes3D_____no_output_____from math import exp, tan_____no_output_____x = np.outer(np.linspace(-10, 10, 30), np.ones(30))
y = x.copy().T
z = abs(x * x - y * y * (x % 20))_____no_output_____fig = plt.figure()
ax = plt.axes(projection='3d')

ax.plot_surface(x, y, z, cmap=plt.cm.jet, rstride=1, cstride=1, linewidth=0);_____no_output_____

At this point it would be nice to run the hillclimber using the new fitness function, but there is a problem: Running our hillclimber with this fitness function is not a good idea, because it might never terminate. Suppose we've reached a point where all neighbours have the same or a worse fitness value. The hillclimber can move no where and is stuck there forever! Such a point in the search landscape is called a *local optimum*. If one reaches such a point, the easiest way out is to give up and restart from a new random point. This is what we will do in our hillclimber with *random restarts*. _____no_output_____

def restarting_hillclimber(fitness_function):
    data = []

    # Create and evaluate starting point
    x, y = random.randint(MIN, MAX), random.randint(MIN, MAX)
    fitness = fitness_function(x, y)
    data += [fitness]
    print("Initial value: %d, %d at fitness %.4f" % (x, y, fitness))
    iterations = 0

    # Stop once we have found an optimal solution
    while fitness > 0:
        changed = False
        iterations += 1
        # Move to first neighbour with a better fitness
        for (nextx, nexty) in neighbours(x, y):
            new_fitness = fitness_function(nextx, nexty)
            if new_fitness < fitness:
                x, y = nextx, nexty
                fitness = new_fitness
                data += [fitness]
                changed = True
                break
        if not changed:
            x, y = random.randint(MIN, MAX), random.randint(MIN, MAX)
            fitness = fitness_function(x, y)
            data += [fitness]

    print("Found optimum after %d iterations at %d, %d" % (iterations, x, y))
    return data_____no_output_____

The change is trivial: We simply keep track whether any movement has occurred with a boolean flag, and if we did not move to any of the neighbours, we pick a new random position to start over at. For convenience, we also made the hillclimber parameterizable with fitness functions. Let's try it out with our `bad_fitness` and plot the resulting fitness values we observe:_____no_output_____

MAX = 1000
MIN = -MAX_____no_output_____data = restarting_hillclimber(bad_fitness)_____no_output_____import matplotlib.pyplot as plt_____no_output_____fig = plt.figure()
ax = plt.axes()

x = range(len(data))
ax.plot(x, data);_____no_output_____

Run the example a couple of times. Sometimes, we are lucky and there is a gradient that takes is straight to an optimal solution. But sometimes you'll see restarts throughout the search before reaching an optimal value._____no_output_____We restricted initial values of `x` and `y` to  rather small range of `[MIN, MAX]`. This is a common trick in test generation, as in *most* cases solutions tend to consist of small values, and using small values to start the search makes the search quicker in many cases. However, what if the solution we need is at a completely different place in our search space? Our bias towards smaller solutions would mean that the hillclimber would take very long to find the solution, and given a fixed search budget it would thus be less likely to actually find a solution. To see what effects this would have, we could simple replace the `1000` with, say, `1000000` or more. We can play around with the range to see the performance we get for our simple search problems._____no_output_____

MAX = 100000
MIN = -MAX_____no_output_____from Timer import Timer_____no_output_____with Timer() as t:
    restarting_hillclimber(get_fitness)
    print("Search time: %.2fs" % t.elapsed_time())_____no_output_____

In most cases the search now will take much longer until a solution is found --- likely longer than we are prepared to wait for such a simple example function! (Although sometimes we will get lucky and randomly hit a good starting position). How is this ever going to work on "real" examples? Not to imagine if there were even more parameters and a bigger neighbourhood!_____no_output_____## Testing a More Complex Program

Let's turn to a slightly more complex program: The CGI decoder you already know from [the Coverage chapter](Coverage.ipynb). _____no_output_____

def cgi_decode(s):
    """Decode the CGI-encoded string `s`:
       * replace "+" by " "
       * replace "%xx" by the character with hex number xx.
       Return the decoded string.  Raise `ValueError` for invalid inputs."""

    # Mapping of hex digits to their integer values
    hex_values = {
        '0': 0, '1': 1, '2': 2, '3': 3, '4': 4,
        '5': 5, '6': 6, '7': 7, '8': 8, '9': 9,
        'a': 10, 'b': 11, 'c': 12, 'd': 13, 'e': 14, 'f': 15,
        'A': 10, 'B': 11, 'C': 12, 'D': 13, 'E': 14, 'F': 15,
    }

    t = ""
    i = 0
    while i < len(s):
        c = s[i]
        if c == '+':
            t += ' '
        elif c == '%':
            digit_high, digit_low = s[i + 1], s[i + 2]
            i += 2
            if digit_high in hex_values and digit_low in hex_values:
                v = hex_values[digit_high] * 16 + hex_values[digit_low]
                t += chr(v)
            else:
                raise ValueError("Invalid encoding")
        else:
            t += c
        i += 1
    return t_____no_output_____

### CGI Decoder as a Search Problem

The `cgi_decode()` function has one input of type string, and one possible way to define the neighbourhood of a string is by all possible strings that have an edit distance of 1. For example, string `test` would have two neighbours for each of its four characters:

- `uest`
- `tfst`
- `tett`
- `tesu`
- `sest`
- `tdst`
- `tert`
- `tess`

In addition, prepending any character or appending any character would also have an edit distance of 1 and could be considered neighbours. To keep things simple, let's keep the length of our input strings fixed to a reasonable value (e.g. 10). In this case, each individual has 20 neighbours (i.e., each character has two neighbours).

Let's implement a new `neighbour_strings()` function that produces these neighbouring strings:_____no_output_____

def neighbour_strings(x):
    n = []
    for pos in range(len(x)):
        c = ord(x[pos])
        if c < 126:
            n += [x[:pos] + chr(c + 1) + x[pos + 1:]]
        if c > 32:
            n += [x[:pos] + chr(c - 1) + x[pos + 1:]]
    return n_____no_output_____

The `neighbour_strings()` function gets the numerical value of each character in the input string, and creates a new string with the character replaced with the preceding and succeeding characters in the alphabet. To start with, we only consider printable ASCII characters, which are in the range 32-126._____no_output_____

print(neighbour_strings("Hello"))_____no_output_____

Thus we have defined the search space for the `cgi_decode()` function. The next ingredient to searching for individuals in this search space is a fitness function._____no_output_____### Branch Distances

The `test_me()` function consisted of a single if-condition, in which two integer numbers were compared. In the `cgi_decode()` function we have three if-conditions and one while loop, and they all compare characters. Fortunately, as we have already seen, we can treat characters like numbers, so we can use the same distance estimate we used in the `test_me()` example. However, there are also two conditions which check whether a character is contained in a set, e.g. `digit_high in hex_values`. How close is a value to being contained in the set? An obvious solution would be to consider the distance to the closest value in the set as the estimate._____no_output_____

import sys_____no_output_____def distance_character(target, values):

    # Initialize with very large value so that any comparison is better
    minimum = sys.maxsize

    for elem in values:
        distance = abs(target - elem)
        if distance < minimum:
            minimum = distance
    return minimum_____no_output_____distance_character(10, [1, 5, 12, 100])_____no_output_____distance_character(10, [0, 50, 80, 200])_____no_output_____

A further simplification we have made so far was to assume that we would always want conditions to evaluate to true. In practice, we might want to have if-conditions evaluate to false just as well. Thus, each if-condition actually has two distance estimates, one to estimate how close it is to being true, and one how close it is to being false. If the condition is true, then the true distance is 0; if the condition is false, then the false distance is 0. That is, in a comparison `a == b`, if `a` is smaller than `b`, then the false distance of is `0` by definition. _____no_output_____What is the distance of `a == b` being false when `a` equals `b`? Any change to either `a` or `b` would make the condition evaluate to false, so we can define the distance simply as 1 in this case._____no_output_____More generally, there can be other types of comparisons, for example using relational operators. Consider the loop condition in `cgi_decode()`: `i < len(s)`, i.e., it uses a less-than comparison operator. It is quite straight forward to extend our notion of branch distance to cover different types of comparisons, and to calculate true and false distances.  The following table shows how to calculate the distance for different types of comparisons:

| Condition | Distance True | Distance False |
| ------------- |:-------------:| -----:|
| a == b      | abs(a - b) | 1 |
| a != b      | 1          | abs(a - b) |
| a < b       | b - a + 1  | a - b      |
| a <= b      | b - a      | a - b + 1  |
| a > b       | a - b + 1  | b - a      |

Note that several of the calculations add a constant `1`. The reason for this is quite simple: Suppose we want to have `a < b` evaluate to true, and let `a = 27` and `b = 27`. The condition is not true, but simply taking the difference would give us a result of `0`. To avoid this, we have to add a constant value. It is not important whether this value is `1` -- any positive constant works._____no_output_____### Dealing with Complex Conditions

In the `cgi_decode()` function, we can also find a somewhat more complex predicate which consists of two conditions joined by a logical `and`:

`
if digit_high in hex_values and digit_low in hex_values:
`

In principle, the branch distance is defined such that the distance to make a conjunction `A and B` true equals the sum of the branch distances for `A` and `B`, as both of the two conditions would need to be true. Similarly, the branch distance to make `A or B` true would be the minimum of the two branch distances of `A` and `B`, as it suffices if one of the two conditions is true to make the entire expression true.

However, it is not as easy as that in practice: Predicates can consist of nested conditions and negations, and one would need to convert the expression to canonical form before being able to apply this calculation. Furthermore, most modern programming languages use *short-circuit evaluation*: If there is a condition `A or B`, and `A` is true, then `B` is never evaluated. If `B` is an expression with side-effects, then by calculating the branch distance of `B` even though short-circuit evaluation would avoid its execution, we would potentially be changing the program behavior (by invoking the side-effect that would in normal behavior not be executed), and that is not acceptable._____no_output_____Furthermore, what if the branching condition has side-effects? For example, suppose that the branching condition were `x == 2 * foo(y)`, where `foo()` is a function that takes an integer as input. Naively instrumenting would lead to the following code:

```
    distance = abs(x - 2 * foo(y))
	if x == 2 * foo(y):
	...
```_____no_output_____Thus, the instrumentation would lead to `foo()` being executed *twice*. Suppose `foo()` changes the state of the system (e.g., by printing something, accessing the file system, changing some state variables, etc.), then clearly invoking `foo()` a second time is a bad idea. One way to overcome this problem is to _transform_ the conditions, rather than _adding_ tracing calls. For example, one can create temporary variables that hold the values necessary for the distance calculation and then use these in the branching condition:

```
	tmp1 = x
	tmp2 = 2 * foo(y)
	distance = abs(tmp1 - tmp2)
	if tmp1 == tmp2:
	...
```_____no_output_____Besides these issues, the approach of adding a global variable and method call to the program seems like a rather clumsy approach --- surely we cannot start thinking about every branch in our program on its own and instrument the program we want to test manually, in particular if programs have multiple branches like the `cgi_decode()` function. Rather, we should be looking at how to _automatically_ instrument programs to contain the necessary added statements such that we can calculate fitness values._____no_output_____### Instrumentation for Atomic Conditions

An alternative approach to using the global and temporary variables is to replace the actual comparison with a call to a helper function, where the original expressions are evaluated as arguments, and the operator is an additional argument. Assume we have a function `evaluate_condition()` which takes four parameters:

* `num` is a unique id that identifies the condition;
* `op` is the operator of the comparison;
* `lhs` and `rhs` are the operands.

The function calculates two distances for the condition: The distance to the condition evaluating to true, and the distance to the condition evaluating to false. One of the two outcomes will always be true, and thus one of them will always have distance `0`. Since the function replaces the original comparison, it returns true or false, depending on which distance is `0`. That means, the example expression

```
    if x == 2 * foo(y)
```

would be replaced by

```
    if evaluate_condition(0, "Eq", x, 2 * foo(y))
```

such that the arguments are only evaluated once, and side-effects are thus handled correctly. Here is how the `evaluate_condition()` function looks like:
_____no_output_____

def evaluate_condition(num, op, lhs, rhs):
    distance_true = 0
    distance_false = 0
    if op == "Eq":
        if lhs == rhs:
            distance_false = 1
        else:
            distance_true = abs(lhs - rhs)

    # ... code for other types of conditions

    if distance_true == 0:
        return True
    else:
        return False_____no_output_____

Note that we are initializing `distance_true` and `distance_false` with `0`. Thus, if `lhs` equals `rhs`, then the variable `distance_true` remains 0, and vice versa._____no_output_____

evaluate_condition(1, "Eq", 10, 20)_____no_output_____evaluate_condition(2, "Eq", 20, 20)_____no_output_____

What the `evaluate_condition()` function does not yet do is store the distances observed. Obviously, we will need to store the values somewhere so that we can access it from our fitness function. Since the `cgi_decode()` program consists of several conditions, and for each condition we might be interested in the true and the false distance, we simply use two global dictionaries, `distances_true` and `distances_false`, and define a helper function that stores the distance values observed in the dictionary:_____no_output_____

def update_maps(condition_num, d_true, d_false):
    global distances_true, distances_false

    if condition_num in distances_true.keys():
        distances_true[condition_num] = min(
            distances_true[condition_num], d_true)
    else:
        distances_true[condition_num] = d_true

    if condition_num in distances_false.keys():
        distances_false[condition_num] = min(
            distances_false[condition_num], d_false)
    else:
        distances_false[condition_num] = d_false_____no_output_____

The variable `condition_num` is the unique ID of the condition that we've just evaluated. If this is the first time that we have executed this particular condition, then the true and false distances are simply stored in the corresponding dictionaries. However, it is possible that the same test executions a condition multiple times. For example, the loop condition `i < len(s)` in the `cgi_decode()` function is evaluated before every single loop iteration. In the end, however, we want to have a single fitness value for a test. As covering a branch just requires that at least one of the executions reaches the branch, we consider only the closest one. Therefore, if the `distances_true` and `distances_false` dictionaries already contain the distance from a previous execution, we only replace that value if the new execution was closer to reaching the branch; this is implemented using the `min()` function._____no_output_____We now need to call this function from within `evaluate_condition()`. Let's also add the calculation of distance for the `in` operator and the `<` comparison, since we need both of them for the `cgi_decode()` example. Furthermore, `cgi_decode()` actually compares characters and numbers. To make sure we use the correct types, we first have to convert the characters to numbers to calculate the distances. This is done using Python's `ord()` function._____no_output_____

def evaluate_condition(num, op, lhs, rhs):
    distance_true = 0
    distance_false = 0

    # Make sure the distance can be calculated on number and character
    # comparisons
    if isinstance(lhs, str):
        lhs = ord(lhs)
    if isinstance(rhs, str):
        rhs = ord(rhs)

    if op == "Eq":
        if lhs == rhs:
            distance_false = 1
        else:
            distance_true = abs(lhs - rhs)

    elif op == "Lt":
        if lhs < rhs:
            distance_false = rhs - lhs
        else:
            distance_true = lhs - rhs + 1
    # ...
    # handle other comparison operators
    # ...

    elif op == "In":
        minimum = sys.maxsize
        for elem in rhs.keys():
            distance = abs(lhs - ord(elem))
            if distance < minimum:
                minimum = distance

        distance_true = minimum
        if distance_true == 0:
            distance_false = 1

    update_maps(num, distance_true, distance_false)

    if distance_true == 0:
        return True
    else:
        return False_____no_output_____

The following shows the instrumentation of the conjunction from `cgi_decode()` to make use of the `evaluate_condition()` function. There are two calls to `evaluate_condition` corresponding to the two conditions, and the `and` with which they are conjoined ensures that the original short-circuiting behavior is preserved:

`
if (evaluate_condition(4, 'In', digit_high, hex_values) and evaluate_condition(5, 'In', digit_low, hex_values))
`

Of course we would like to automatically produce this instrumented version._____no_output_____### Instrumenting Source Code Automatically

Replacing comparisons automatically is actually quite easy in Python, using the abstract syntax tree (AST) of the program. In the AST, a comparison will typically be a tree node with an operator attribute and two children for the left-hand and right-hand operators. To replace such comparisons with a call to `evaluate_condition()` one simply needs to replace the comparison node in the AST with a function call node, and this is what the `BranchTransformer` class does using a NodeTransformer from Python's `ast` module:
_____no_output_____

import ast_____no_output_____class BranchTransformer(ast.NodeTransformer):

    branch_num = 0

    def visit_FunctionDef(self, node):
        node.name = node.name + "_instrumented"
        return self.generic_visit(node)

    def visit_Compare(self, node):
        if node.ops[0] in [ast.Is, ast.IsNot, ast.In, ast.NotIn]:
            return node

        self.branch_num += 1
        return ast.Call(func=ast.Name("evaluate_condition", ast.Load()),
                        args=[ast.Num(self.branch_num),
                              ast.Str(node.ops[0].__class__.__name__),
                              node.left,
                              node.comparators[0]],
                        keywords=[],
                        starargs=None,
                        kwargs=None)_____no_output_____

The `BranchTransformer` parses a target Python program using the built-in parser `ast.parse()`, which returns the AST. Python provides API to traverse and modify this AST. To replace the comparison with a function call we use an `ast.NodeTransformer`, which uses the visitor pattern where there is one `visit_*` function for each type of node in the AST. As we are interested in replacing comparisons, we override `visit_Compare`, where instead of the original comparison node we return a new node of type `ast.Func`, which is a function call node. The first parameter of this node is the name of the function `evaluate_condition()`, and the arguments are the four arguments that our `evaluate_condition()` function expects:

* the number of branch (for which we keep a count in `branch_num`),
* the operator (for which we simply use the class name),
* the left-hand side, and
* the right-hand side.

Note that Python allows comparisons of multiple expressions (e.g. `1 < x < 10`); to keep the code simple we only deal with individual comparisons here, but it would be straight forward to extend the code by treating each comparison with an individual call to `evaluate_condition`. You will notice that we also override `visit_FunctionDef`; this is just to change the name of the method by appending `_instrumented`, so that we can continue to use the original function together with the instrumented one._____no_output_____The following code parses the source code of the `cgi_decode()` function to an AST, then transforms it, and prints it out again (using the `to_source()` function from the `astor` library):_____no_output_____

import inspect
import ast
import astor_____no_output_____from fuzzingbook_utils import print_content_____no_output_____source = inspect.getsource(cgi_decode)
node = ast.parse(source)
BranchTransformer().visit(node)

# Make sure the line numbers are ok before printing
node = ast.fix_missing_locations(node)
print_content(astor.to_source(node), '.py')_____no_output_____

To calculate a fitness value with the instrumented version, we need to compile the instrumented AST again, which is done using Python's `compile()` function. We then need to make the compiled function accessible, for which first retrieve the current module from `sys.modules`, and then add the compiled code of the instrumented function to the list of functions of the current module using `exec`. After this, the `cgi_decode_instrumented()` function can be accessed._____no_output_____

def create_instrumented_function(f):
    source = inspect.getsource(f)
    node = ast.parse(source)
    node = BranchTransformer().visit(node)

    # Make sure the line numbers are ok so that it compiles
    node = ast.fix_missing_locations(node)

    # Compile and add the instrumented function to the current module
    current_module = sys.modules[__name__]
    code = compile(node, filename="<ast>", mode="exec")
    exec(code, current_module.__dict__)_____no_output_____# Set up the global maps
distances_true = {}
distances_false = {}_____no_output_____# Create instrumented function
# cgi_decode_instrumented =
create_instrumented_function(cgi_decode)_____no_output_____assert cgi_decode("Hello+Reader") == cgi_decode_instrumented("Hello+Reader")_____no_output_____cgi_decode_instrumented("Hello+Reader")_____no_output_____distances_true_____no_output_____distances_false_____no_output_____

As we can see from the `distances_true` and `distances_false` maps, conditions 1 and 2 have evaluated to true and to false, whereas condition 3 has only evaluated to false, when executed on `cgi_decode_instrumented`. This is as expected, since the while-loop was entered and left, and there was one white space but no `%`-character in the input string._____no_output_____### Fitness Function to Create Valid Hexadecimal Inputs

As an example, let's take as objective to test the part of `cgi_decode()` that decodes valid hexadecimal codes. This means that we want to make condition 1 true, 2 false, 3 true, and 4 true. To represent such a path, we can simply sum up the branch distances for exactly these branches. However, there is a potential issue with simply summing up branch distances: If the distance for one condition depends on a comparison of very large values and the distance calculation for another condition depends on small values, then an improvement of the large values would very likely lead to a better fitness improvement, and thus bias the search. To avoid this, we need to *normalize* branch distances before adding them up._____no_output_____A normalisation function for a range `[a, b]` takes a number as input and returns a value that is `>=a` and `<=b`. The important thing about the function is that for any two numbers `x` and `y` the ordering needs to be preserved by the normalisation. That is, if `x<y` then it must also hold that `normalize(x) < normalize(y)`. There are many different functions that could achieve this result; a simple one is `normalize(x) = x/(x+k)`: It is computationally cheap, and will normalise any positive value in the range `[0,1]` (to change this to `[0,b]` one would just need to multiply by `b`). If we use this normalisation function, we also know the maximum value: it is `1.0`. The function assumes that the value to be normalised is positive. The value of the factor `k` defines the steepness of the curve. For example, for `k=1` (which is a good default value) the curve is very steep, with values quickly approaching, but never reaching, `1`._____no_output_____

def normalize(x):
    return x / (1.0 + x)_____no_output_____

To see how this normalisation function works, we can plot a range of values. The x-axis contains the original values, the y-axis shows the corresponding normalised value:_____no_output_____

import matplotlib.pyplot as plt_____no_output_____fig = plt.figure()
ax = plt.axes()

x = range(100)
y = [value / (value + 1.0) for value in x]
ax.plot(x, y);_____no_output_____

Using this normalization function, we can calculate a fitness value by running the instrumented program under test, and then adding up the normalized values of the relevant branches. However, not every if statement will be executed by every test case. What is the branch distance for a branch that wasn't even executed? We need to make sure that the branch distance for branches that weren't even executed is higher than any of the branch distances observed during execution. Given our normalization, we know this maximum value is 1. Thus, the fitness function could look like this:_____no_output_____

def get_fitness_cgi(x):
    # Reset any distance values from previous executions
    global distances_true, distances_false
    distances_true = {}
    distances_false = {}

    # Run the function under test
    try:
        cgi_decode_instrumented(x)
    except BaseException:
        pass

    # Sum up branch distances
    fitness = 0.0
    for branch in [1, 3, 4, 5]:
        if branch in distances_true:
            fitness += normalize(distances_true[branch])
        else:
            fitness += 1.0

    for branch in [2]:
        if branch in distances_false:
            fitness += normalize(distances_false[branch])
        else:
            fitness += 1.0

    return fitness_____no_output_____

To see our instrumentation in action, let's calculate fitness values for a couple of example inputs, progressively getting closer to the target._____no_output_____

# Empty string does not even enter the loop
get_fitness_cgi("")_____no_output_____# String contains no percentage character
get_fitness_cgi("Hello+Reader")_____no_output_____# String contains a percentage character, but no valid hex char
get_fitness_cgi("%UU")_____no_output_____# String contains a percentage character, but only one valid hex char
get_fitness_cgi("%AU")_____no_output_____# String contains a percentage character and two valid hex chars
get_fitness_cgi("%AA")_____no_output_____

### Hillclimbing Valid Hexadecimal Inputs

We have now defined the search space through our `neighbour_strings()` function, and we have derived a new fitness function, so before we can apply the hillclimber to the problem of finding strings that contain valid 2-digit hex codes, the only remaining point is to derive a random starting point, which you've already done previously in the [MutationFuzzer chapter](MutationFuzzer.ipynb):_____no_output_____

def random_string(l):
    s = ""
    for i in range(l):
        random_character = chr(random.randrange(32, 127))
        s = s + random_character
    return s_____no_output_____

Now we can put it all together and use the identical hillclimbing algorithm again, and just make it generate random strings instead of numbers, and use the new fitness function:_____no_output_____

def hillclimb_cgi():
    x = random_string(10)
    fitness = get_fitness_cgi(x)
    print("Initial input: %s at fitness %.4f" % (x, fitness))

    while fitness > 0:
        changed = False
        for (nextx) in neighbour_strings(x):
            new_fitness = get_fitness_cgi(nextx)
            if new_fitness < fitness:
                x = nextx
                fitness = new_fitness
                changed = True
                print("New value: %s at fitness %.4f" % (x, fitness))
                break

        # Random restart if necessary
        if not changed:
            x = random_string(10)
            fitness = get_fitness_cgi(x)

    print("Optimum at %s, fitness %.4f" % (x, fitness))_____no_output_____hillclimb_cgi()_____no_output_____

Run the hillclimber a couple of times to see it generate strings with valid hex characters. Sometimes it only takes a couple of steps, sometimes it takes longer, but at the end it always solves the problem and generates the string we want to have._____no_output_____## Evolutionary Search

Hillclimbing works well, if the neighbourhood is reasonably small. This was the case so far in the `cgi_decode()` example because we limited ourselves to a fixed number of characters (10) and limited the character range to printable ASCII characters. But imagine what happens if we are not looking for ASCII characters, but  UTF-16 unicode characters? These are not allowed in URLs really, but let's see what happens if we change our search space:_____no_output_____

def random_unicode_string(l):
    s = ""
    for i in range(l):
        # Limits to reflect range of UTF-16
        random_character = chr(random.randrange(0, 65536))
        s = s + random_character
    return s_____no_output_____def unicode_string_neighbours(x):
    n = []
    for pos in range(len(x)):
        c = ord(x[pos])
        # Limits to reflect range of UTF-16
        if c < 65536:
            n += [x[:pos] + chr(c + 1) + x[pos + 1:]]
        if c > 0:
            n += [x[:pos] + chr(c - 1) + x[pos + 1:]]

    return n_____no_output_____

UTF-16 characters are represented with 16 bits, and this gives us a range of 65536 possible characters. The functions above are adapted to these boundaries. Before we run the hillclimber again, let's make one more change: We will add an iteration limit so that we can give up the search, rather than waiting forever for it to complete._____no_output_____

def terminal_repr(s):
    return terminal_escape(repr(s))_____no_output_____def hillclimb_cgi_limited(max_iterations):
    x = random_unicode_string(10)
    fitness = get_fitness_cgi(x)
    print("Initial input: %s at fitness %.4f" % (terminal_repr(x), fitness))

    iteration = 0
    logs = 0
    while fitness > 0 and iteration < max_iterations:
        changed = False
        for (nextx) in unicode_string_neighbours(x):
            new_fitness = get_fitness_cgi(nextx)
            if new_fitness < fitness:
                x = nextx
                fitness = new_fitness
                changed = True
                if logs < LOG_VALUES:
                    print("New value: %s at fitness %.4f" %
                          (terminal_repr(x), fitness))
                elif logs == LOG_VALUES:
                    print("...")
                logs += 1
                break

        # Random restart if necessary
        if not changed:
            x = random_string(10)
            fitness = get_fitness_cgi(x)
        iteration += 1

    print("Optimum at %s, fitness %.4f" % (terminal_repr(x), fitness))_____no_output_____hillclimb_cgi_limited(100)_____no_output_____

You can play around with the iteration limit and see how long it takes to find a solution in this search space. Unless you are lucky with the random starting point, this usually takes really unreasonably long._____no_output_____### Global Search

The hillclimber explores the local neighbourhood of an individual in each step of the search, and if the neighbourhood is too large, then this takes too long. An alternative strategy is to not restrict the search to the local neighbourhood, but to search the search space *globally*. That is, the search algorithm is allowed to make larger steps around the search space. A simple modification of the hillclimber converts it from a local to a global search algorithm: Instead of looking at all the immediate neighbours, the individual is *mutated* in a way that allows larger modifications._____no_output_____A *mutation* is a change that represents such a larger step around the search space. An important decision when implementing a mutation is that it should, theoretically, be possible to reach any point in the search space simply by applying mutation successively. However, mutation usually should not completely replace an individual with a random one. It is important for the search to be effective that the mutation constitutes a reasonable change to an individual that still maintains most of its traits. For our search problem of 10-character strings, a possible mutation would be to replace only _1_ out of the 10 characters, as follows:_____no_output_____

def flip_random_character(s):
    pos = random.randint(0, len(s) - 1)
    new_c = chr(random.randrange(0, 65536))
    return s[:pos] + new_c + s[pos + 1:]_____no_output_____

A simple modification of the hillclimber is to replace the exploration of the neighbourhood with mutations. In each iteration, the current individual is mutated. The resulting offspring individual is compared to its parent, and the better of the two is the new point in the search space, and used for the next iteration of mutation. This is called "randomized hillclimbing"._____no_output_____

def randomized_hillclimb():
    x = random_unicode_string(10)
    fitness = get_fitness_cgi(x)
    print("Initial value: %s at fitness %.4f" %
          (terminal_repr(x), fitness))

    iterations = 0
    while fitness > 0:
        mutated = flip_random_character(x)
        new_fitness = get_fitness_cgi(mutated)
        if new_fitness <= fitness:
            x = mutated
            fitness = new_fitness
            #print("New value: %s at fitness %.4f" %(terminal_repr(x), fitness))
        iterations += 1

    print("Optimum at %s after %d iterations" %
          (terminal_repr(x), iterations))_____no_output_____randomized_hillclimb()_____no_output_____

This algorithm usually finds the solution quicker than the standard hillclimber, although it is still rather slow!

There is one subtle, but crucial, change between the hillclimber and the randomized version that is worth pointing out: Note the comparison `new_fitness <= fitness`, whereas in the hillclimber we used `new_fitness < fitness`. This is important because the search landscape may have areas with equal fitness values (plateaus) which we somehow need to overcome. In the standard hillcimber we did this with random restarts. If the randomized hillclimber would reject offspring with identical fitness, it would keep mutating the identical individual, and possibly struggle to escape the plateau. If we replace also with equal fitness, we allow the search to move around the plateau, increasing chances of escaping it eventually.
_____no_output_____The randomized hillclimber is also known as *1+1 Evolutionary Algorithm* (*(1+1)EA*). An evolutionary algorithm is a meta-heuristic search algorithm that tries to mimic the natural processes of evolution. Natural evolution is based
on populations where environmental pressure causes natural selection: Only the fittest survive, while those that are not fit enough will die, and thus the overall fitness of the population gradually improves. The (1+1)EA is a very specific evolutionary algorithm with a population size of 1, which produces exactly 1 offspring. However, the most common search algorithm in practice is a genetic algorithm._____no_output_____### Genetic Algorithms

One of the best known evolutionary algorithms is the *Genetic Algorithm* (GA). A GA is based on the idea that problem solutions can be genetically encoded: A chromosome consists of a sequence of genes, where each gene encodes one trait of the individual (e.g., eye colour, hair colour, etc.). A fitness function can take the information contained in this description, the so called genotype, and evaluates the properties of the resulting phenotype, i.e., the actual solution represented by this genetic encoding. The fitness value of an individual is measured on the phenotype._____no_output_____For our `cgi_decode()` example, we could consider individual characters to be the genes, and then a chromosome is a sequences of characters. In other words, the genetic encoding is exactly what we already used so far! However, for the GA the representation requires different operations than the enumeration of the neighbourhood._____no_output_____The GA emulates natural evolution with the following process:

- Create an initial population of random chromosomes
- Select fit individuals for reproduction
- Generate new population through reproduction of selected individuals
- Continue doing so until an optimal solution has been found, or some other limit has been reached.

The first step, creating an initial population, is trivial:_____no_output_____

def create_population(size):
    return [random_unicode_string(10) for i in range(size)]_____no_output_____create_population(10)_____no_output_____

The selection process prefers fitter individuals, therefore we need to determine the fitness values for all the individuals in our population. As we are going to need the fitness value for each individual at several places in the algorithm, we are going to store it by making fitness a list of _tuples_, each tuple consisting of an individual and its fitness value._____no_output_____

def evaluate_population(population):
    fitness = [get_fitness_cgi(x) for x in population]
    return list(zip(population, fitness))_____no_output_____population = create_population(10)_____no_output_____for (individual, fitness) in evaluate_population(population):
    print("%s: %.4f" % (terminal_repr(individual), fitness))_____no_output_____

The use of a fitness value in the search is typically explained using "survival of the fittest", but one of Darwin’s key insights on evolution was that selection is not only defined by survival -- individuals reproduce sexually, and sexual selection describes the selection pressure during reproduction. This selection is typically influenced by either combat: rivalling males fighting over a female, and the stronger (fitter) male wins; the selection is also influenced by display. Darwin’s example for this is the peacock: The peacock has long, beautiful tail feathers that appear to serve no purpose, and do not seem to support the concept of natural selection. However, peahens are influenced in their selection of a sexual partner by his appearance. Impressive ornaments suggest that a male is genetically particularly healthy and will lead to healthy offspring. This is reflected in a GA: The better the fitness value an individual, the more likely it is to mate with another individual. A simple selection algorithm that reflects this concept is *tournament selection*: A small number of randomly chosen individual compete to be selected, and the best is selected:_____no_output_____

def selection(evaluated_population, tournament_size):
    competition = random.sample(evaluated_population, tournament_size)
    winner = min(competition, key=lambda individual: individual[1])[0]

    # Return a copy of the selected individual
    return winner[:]_____no_output_____

The `tournament_size` parameter specifies how many randomly selected individuals out of the population participate in the tournament. This is an important choice as it determines *selective pressure*: The larger the tournament size, the larger the probability of very good individuals being included in the tournament. This, in turn, increases the probability of these very good individuals of dominating the next generation, thus reducing the diversity and leading to premature convergence. In contrast, if the tournament size is too small, then this inhibits evolution. The best value for the tournament size depends on the population size, but usually it is fairly small (e.g. 5). Let's run a tournament using all individuals of an example population to see that the best one is selected:_____no_output_____

population = create_population(10)
fitness = evaluate_population(population)
selected = selection(fitness, 10)_____no_output_____for (individual, fitness_value) in fitness:
    print("%s: %.4f" % (terminal_repr(individual), fitness_value))_____no_output_____print("Winner: %s" % terminal_repr(selected))_____no_output_____

Like in natural evolution, the individuals selected based on their fitness reproduce, to form a new generation. During this reproduction, like in natural reproduction, genetic material of the selected parents is combined. This is typically done by a process known as *crossover*, where an offspring chromosome is produced from the genes of its parents. In our case, a chromosome is a sequence of characters, and crossing over two parent sequences of characters can simply be done by chosing a cutoff random point, and creating offspring by combining the halves of the parents chromosomes based on the cutoff point._____no_output_____

def crossover(parent1, parent2):
    pos = random.randint(1, len(parent1))

    offspring1 = parent1[:pos] + parent2[pos:]
    offspring2 = parent2[:pos] + parent1[pos:]

    return (offspring1, offspring2)_____no_output_____parent1 = "Hello World"
parent2 = "Goodbye Book"

crossover(parent1, parent2)_____no_output_____

A further essential ingredient of genetic evolution is the concept of mutation: Sometimes, offspring contain small changes in their genes, which represent new genetic material and new physiological traits. If the mutation introduced new genetic material that represents useful traits leading to better fitness values, then this genetic material will survive. If the mutation introduced useless genetic material, then the resulting individuals will likely have bad fitness values and die off quickly.

An important aspect is that both, mutation and crossover, are probabilistic actions. They do not always happen, and their effects are different every time they do happen. For a sequence genetic encoding, a common approach is to first use a probability to decide whether to apply mutation, and then to select one gene and change it a bit. Alternatively, we can probabilistically mutate genes such that on average one is changed. We do this by calculating a probability `P` of each gene being mutated as 1/(number of genes). Then, if a gene is mutated, we don't just replace it by a random character, but make it more likely that there is only a small change. This is achieved by sampling from a Gaussian distribution with the current value as the mean. We arbitrarily use a standard deviation of 100 to make small changes more likely than large changes._____no_output_____

def mutate(chromosome):
    mutated = chromosome[:]
    P = 1.0 / len(mutated)

    for pos in range(len(mutated)):
        if random.random() < P:
            new_c = chr(int(random.gauss(ord(mutated[pos]), 100) % 65536))
            mutated = mutated[:pos] + new_c + mutated[pos + 1:]
    return mutated_____no_output_____

Now at last we have all the ingredients to build the full algorithm:_____no_output_____

def genetic_algorithm():
    # Generate and evaluate initial population
    generation = 0
    population = create_population(100)
    fitness = evaluate_population(population)
    best = min(fitness, key=lambda item: item[1])
    best_individual = best[0]
    best_fitness = best[1]
    print("Best fitness of initial population: %s - %.10f" %
        (terminal_repr(best_individual), best_fitness))
    logs = 0

    # Stop when optimum found, or we run out of patience
    while best_fitness > 0 and generation < 1000:

        # The next generation will have the same size as the current one
        new_population = []
        while len(new_population) < len(population):
            # Selection
            offspring1 = selection(fitness, 10)
            offspring2 = selection(fitness, 10)

            # Crossover
            if random.random() < 0.7:
                (offspring1, offspring2) = crossover(offspring1, offspring2)

            # Mutation
            offspring1 = mutate(offspring1)
            offspring2 = mutate(offspring2)

            new_population.append(offspring1)
            new_population.append(offspring2)

        # Once full, the new population replaces the old one
        generation += 1
        population = new_population
        fitness = evaluate_population(population)

        best = min(fitness, key=lambda item: item[1])
        best_individual = best[0]
        best_fitness = best[1]
        if logs < LOG_VALUES:
            print(
                "Best fitness at generation %d: %s - %.8f" %
                (generation, terminal_repr(best_individual), best_fitness))
        elif logs == LOG_VALUES:
            print("...")
        logs += 1

    print(
        "Best individual: %s, fitness %.10f" %
        (terminal_repr(best_individual), best_fitness))_____no_output_____

Let's see how this works on our unicode example._____no_output_____

genetic_algorithm()_____no_output_____

## Synopsis

This chapter demonstrates how to use meta-heuristic search algorithms to find inputs that reach specific locations in the source code. The simplest search algorithm is hillclimbing, which is applied to the simple `test_me` example as follows:_____no_output_____

hillclimber()_____no_output_____

Different aspects and challenges require different variations of this algorithm, such as a `steepest_ascent_hillclimber` or a `restarting_hillclimber`._____no_output_____The search can be guided by different optimization goals captured in fitness functions. A fitness function to measure how close we are to reaching locations in the source code uses source code instrumentation. To produce an instrumented version of `cgi_decode`, use:_____no_output_____

cgi_decode_instrumented = create_instrumented_function(cgi_decode)_____no_output_____

Fitness values are obtained by executing `cgi_decode_instrumented`, which is done by the `get_fitness_cgi` function:_____no_output_____

get_fitness_cgi("Foo")_____no_output_____

Complex functions like `cgi_decode` result in vastly larger search spaces, which can be explored using evolutionary search algorithms such as genetic algorithms:_____no_output_____

genetic_algorithm()_____no_output_____

## Lessons Learned

* A meta-heuristic search problem consists of an algorithm, a representation, and a fitness function.
* For test generation, fitness functions typically estimate how close an execution comes to a target location. To determine this distance, we use instrumentation to calculate the distance during test execution.
* Local search algorithms like hillcimbing work well when the neighbourhood is well defined and not too large.
* Global search algorithms like genetic algorithms are very flexible and scale up well to larger test problems._____no_output_____## Next Steps

In this chapter we looked at rather simple program inputs. We can apply the same search algorithms also to evolve complex test inputs, in particular [for grammar input](EvoGrammarFuzzer.ipynb)._____no_output_____## Background

\todo{Add more}

The targets of the search are typically related to coverage. For discussions, see the books in the [Introduction to Testing](Intro_Testing.ipynb)._____no_output_____## Exercises

\todo{Will be added later}_____no_output_____


# Notebook from mattborghi/kaggle-courses
Path: Pandas/3 Summary Functions and Maps/exercise-summary-functions-and-maps.ipynb

**This notebook is an exercise in the [Pandas](https://www.kaggle.com/learn/pandas) course.  You can reference the tutorial at [this link](https://www.kaggle.com/residentmario/summary-functions-and-maps).**

---
_____no_output_____# Introduction

Now you are ready to get a deeper understanding of your data.

Run the following cell to load your data and some utility functions (including code to check your answers)._____no_output_____

import pandas as pd
pd.set_option("display.max_rows", 5)
reviews = pd.read_csv("../input/wine-reviews/winemag-data-130k-v2.csv", index_col=0)

from learntools.core import binder; binder.bind(globals())
from learntools.pandas.summary_functions_and_maps import *
print("Setup complete.")

reviews.head()_____no_output_____

# Exercises_____no_output_____## 1.

What is the median of the `points` column in the `reviews` DataFrame?_____no_output_____

median_points = reviews.points.median()

# Check your answer
q1.check()_____no_output_____q1.hint()
#q1.solution()_____no_output_____

## 2.
What countries are represented in the dataset? (Your answer should not include any duplicates.)_____no_output_____

countries = reviews.country.unique()

# Check your answer
q2.check()_____no_output_____#q2.hint()
#q2.solution()_____no_output_____

## 3.
How often does each country appear in the dataset? Create a Series `reviews_per_country` mapping countries to the count of reviews of wines from that country._____no_output_____

reviews_per_country = reviews.country.value_counts()

# Check your answer
q3.check()_____no_output_____#q3.hint()
#q3.solution()_____no_output_____

## 4.
Create variable `centered_price` containing a version of the `price` column with the mean price subtracted.

(Note: this 'centering' transformation is a common preprocessing step before applying various machine learning algorithms.) _____no_output_____

centered_price = reviews.price.map(lambda x: x - reviews.price.mean())

# Check your answer
q4.check()_____no_output_____#q4.hint()
#q4.solution()_____no_output_____

## 5.
I'm an economical wine buyer. Which wine is the "best bargain"? Create a variable `bargain_wine` with the title of the wine with the highest points-to-price ratio in the dataset._____no_output_____

idx = (reviews.points / reviews.price).idxmax()
idx_____no_output_____bargain_wine = reviews.title.iloc[idx]

# Check your answer
q5.check()_____no_output_____q5.hint()
q5.solution()_____no_output_____

## 6.
There are only so many words you can use when describing a bottle of wine. Is a wine more likely to be "tropical" or "fruity"? Create a Series `descriptor_counts` counting how many times each of these two words appears in the `description` column in the dataset._____no_output_____

tropical = reviews.description.map(lambda x: "tropical" in x).sum()
fruity = reviews.description.map(lambda x: "fruity" in x).sum()_____no_output_____descriptor_counts = pd.Series([tropical, fruity], index= ['tropical', 'fruity'])

# Check your answer
q6.check()_____no_output_____q6.hint()
q6.solution()_____no_output_____

## 7.
We'd like to host these wine reviews on our website, but a rating system ranging from 80 to 100 points is too hard to understand - we'd like to translate them into simple star ratings. A score of 95 or higher counts as 3 stars, a score of at least 85 but less than 95 is 2 stars. Any other score is 1 star.

Also, the Canadian Vintners Association bought a lot of ads on the site, so any wines from Canada should automatically get 3 stars, regardless of points.

Create a series `star_ratings` with the number of stars corresponding to each review in the dataset._____no_output_____

def apply_star(row):
    if row.country == 'Canada':
        return 3
    else:
        if row.points >= 95:
            return 3
        elif row.points >= 85 and row.points < 95:
            return 2
        else:
            return 1_____no_output_____star_ratings = reviews.apply(apply_star, axis = 'columns')

# Check your answer
q7.check()_____no_output_____q7.hint()
q7.solution()_____no_output_____

# Keep going
Continue to **[grouping and sorting](https://www.kaggle.com/residentmario/grouping-and-sorting)**._____no_output_____---

*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161299) to chat with other Learners.*_____no_output_____


# Notebook from pachterlab/BP_2020
Path: notebooks/sampling_rate.ipynb

<code>
import glob
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.patches as mpatches
import scanpy as sc
from scipy import stats

from collections import OrderedDict
from sklearn.decomposition import TruncatedSVD
from sklearn.manifold import TSNE
from sklearn.preprocessing import scale

from sklearn.cluster import KMeans
from sklearn.preprocessing import normalize
from sklearn.preprocessing import LabelEncoder
from sklearn.neighbors import NeighborhoodComponentsAnalysis
from matplotlib import cm
from matplotlib.lines import Line2D

def nd(arr):
    return np.asarray(arr).reshape(-1)
def yex(ax):
    lims = [np.min([ax.get_xlim(), ax.get_ylim()]),
            np.max([ax.get_xlim(), ax.get_ylim()])]

    # now plot both limits against eachother
    ax.plot(lims, lims, 'k-', alpha=0.75, zorder=0)
    ax.set_aspect('equal')
    ax.set_xlim(lims)
    ax.set_ylim(lims)
    return ax

def trim_axs(axs, N):
    """little helper to massage the axs list to have correct length..."""
    axs = axs.flat
    for ax in axs[N:]:
        ax.remove()
    return axs[:N]

import warnings
warnings.filterwarnings('ignore')

fsize=20

plt.rcParams.update({'font.size': fsize})
%config InlineBackend.figure_format = 'retina'/home/sina/bin/miniconda3/lib/python3.7/site-packages/anndata/_core/anndata.py:21: FutureWarning: pandas.core.index is deprecated and will be removed in a future version.  The public classes are available in the top-level namespace.
  from pandas.core.index import RangeIndex
age_map = {
    "SRR8426358": 24,
    "SRR8426359": 24,
    "SRR8426362": 24,
    "SRR8426363": 24,
    "SRR8426364": 24,
    "SRR8426369": 24,
    "SRR8426370": 24,
    "SRR8426360": 3,
    "SRR8426361": 3,
    "SRR8426365": 3,
    "SRR8426366": 3,
    "SRR8426367": 3,
    "SRR8426368": 3,
    "SRR8426371": 3,
    "SRR8426372": 3
}_____no_output_____data = anndata.read_h5ad("write/data.h5ad")_____no_output_____df = data.obs.copy()_____no_output_____df["genes_detected"] = nd((data.layers["raw"]>0).sum(axis=1))
df["counts"] = nd(data.layers["raw"].sum(axis=1))_____no_output_____unique = np.unique(df.SRR.values)_____no_output_____unique.shape_____no_output_____tmp = df.groupby("SRR")["genes_detected", "counts"].mean()_____no_output_____tmp["age"] = tmp.index.map(age_map)_____no_output_____tmp_____no_output_____fig, ax = plt.subplots(figsize=(5,5))

x = tmp.counts.values
y = tmp.genes_detected.values

ym = tmp.age.values == 3
om = tmp.age.values == 24

ax.scatter(x[ym],y[ym], color="#D43F3A", label="3 month")
ax.scatter(x[om],y[om], color='#1f77b4', label="24 month")

ax.set_ylim(0)
ax.set_xlim(0)
yex(ax)

ax.legend()

ax.set_ylabel("Average genes detected")
ax.set_xlabel("Average UMI counts")

plt.savefig("./figs/sampling_rate.png", bbox_inches='tight',dpi=300)

plt.show()_____no_output_____print(y[ym].mean())
print(y[om].mean())349.1810906682289
382.1088124528228

</code>


# Notebook from ShepherdCode/Soars2021
Path: Notebooks/Jas_306_GenCode_MLP.ipynb

# MLP ORF to GenCode

Use GenCode 38 and ORF-restricted, length-restricted data.
Use model pre-trained on Simulated ORF (notebook 304).
Test accuracy 63%.  _____no_output_____

import time
def show_time():
    t = time.time()
    print(time.strftime('%Y-%m-%d %H:%M:%S %Z', time.localtime(t)))
show_time()2021-08-18 09:21:53 EDT
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.utils import shuffle
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score

from keras.models import Sequential
from keras.layers import Dense,Embedding,Dropout
from keras.layers import Flatten,TimeDistributed
from keras.losses import BinaryCrossentropy
from keras.callbacks import ModelCheckpoint
from keras.models import load_model2021-08-18 09:21:54.079799: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
import sys
IN_COLAB = False
try:
    from google.colab import drive
    IN_COLAB = True
except:
    pass
if IN_COLAB:
    print("On Google CoLab, mount cloud-local file, get our code from GitHub.")
    PATH='/content/drive/'
    #drive.mount(PATH,force_remount=True)  # hardly ever need this
    drive.mount(PATH)    # Google will require login credentials
    DATAPATH=PATH+'My Drive/data/'  # must end in "/"
    import requests
    r = requests.get('https://raw.githubusercontent.com/ShepherdCode/Soars2021/master/SimTools/RNA_describe.py')
    with open('RNA_describe.py', 'w') as f:
        f.write(r.text)
    from RNA_describe import ORF_counter
    r = requests.get('https://raw.githubusercontent.com/ShepherdCode/Soars2021/master/SimTools/GenCodeTools.py')
    with open('GenCodeTools.py', 'w') as f:
        f.write(r.text)
    from GenCodeTools import GenCodeLoader
    r = requests.get('https://raw.githubusercontent.com/ShepherdCode/Soars2021/master/SimTools/KmerTools.py')
    with open('KmerTools.py', 'w') as f:
        f.write(r.text)
    from KmerTools import KmerTools
    r = requests.get('https://raw.githubusercontent.com/ShepherdCode/Soars2021/master/SimTools/DataPrep.py')
    with open('DataPrep.py', 'w') as f:
        f.write(r.text)
    from DataPrep import DataPrep
else:
        print("CoLab not working. On my PC, use relative paths.")
        DATAPATH='data/'  # must end in "/"
        sys.path.append("..") # append parent dir in order to use sibling dirs
        from SimTools.RNA_describe import ORF_counter
        from SimTools.GenCodeTools import GenCodeLoader
        from SimTools.KmerTools import KmerTools
        from SimTools.DataPrep import DataPrep
BESTMODELPATH=DATAPATH+"BestModel-304"
LASTMODELPATH=DATAPATH+"LastModel"  CoLab not working. On my PC, use relative paths.

## Data Load_____no_output_____

PC_TRAINS=1000
NC_TRAINS=1000
PC_TESTS=40000
NC_TESTS=40000
PC_LENS=(200,4000)
NC_LENS=(200,4000)   # Wen used 3500 for hyperparameter, 3000 for train
PC_FILENAME='gencode.v38.pc_transcripts.fa.gz'
NC_FILENAME='gencode.v38.lncRNA_transcripts.fa.gz'
PC_FULLPATH=DATAPATH+PC_FILENAME
NC_FULLPATH=DATAPATH+NC_FILENAME
MAX_K = 3
INPUT_SHAPE=(None,84)  # 4^3 + 4^2 + 4^1
NEURONS=32
DROP_RATE=0.30
EPOCHS=200
SPLITS=3
FOLDS=3
show_time()2021-08-18 09:21:54 EDT
loader=GenCodeLoader()
loader.set_label(1)
loader.set_check_utr(True)  # ORF-restricted
loader.set_check_size(*PC_LENS)   # length-restricted
pcdf=loader.load_file(PC_FULLPATH)
print("PC seqs loaded:",len(pcdf))
loader.set_label(0)
loader.set_check_utr(False)
loader.set_check_size(*NC_LENS) # length-restricted
ncdf=loader.load_file(NC_FULLPATH)
print("NC seqs loaded:",len(ncdf))
show_time()PC seqs loaded: 55381
NC seqs loaded: 46919
2021-08-18 09:21:56 EDT
def dataframe_extract_sequence(df):
    return df['sequence'].tolist()

pc_all = dataframe_extract_sequence(pcdf)
nc_all = dataframe_extract_sequence(ncdf)
pcdf=None
ncdf=None

show_time()
print("PC seqs pass filter:",len(pc_all),type(pc_all))
print("NC seqs pass filter:",len(nc_all),type(nc_all))

#PC seqs pass filter: 55381
#NC seqs pass filter: 469192021-08-18 09:21:56 EDT
PC seqs pass filter: 55381 <class 'list'>
NC seqs pass filter: 46919 <class 'list'>
print("Simulated sequence characteristics:")
oc = ORF_counter()
print("PC seqs")
oc.describe_sequences(pc_all)
print("NC seqs")
oc.describe_sequences(nc_all)
oc=None
show_time()Simulated sequence characteristics:
PC seqs
Average RNA length: 1935.5395713331286
Average ORF length: 1047.2401906791138
NC seqs
Average RNA length: 1179.5947483961722
Average ORF length: 203.12651591039878
2021-08-18 09:22:11 EDT

## Data Prep_____no_output_____

dp = DataPrep()
Xseq,y=dp.combine_pos_and_neg(pc_all,nc_all)
nc_all=None
pc_all=None
nc_all=None
print("The first few shuffled labels:")
print(y[:30])
show_time()The first few shuffled labels:
[0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1]
2021-08-18 09:22:11 EDT
Xfrq=KmerTools.seqs_to_kmer_freqs(Xseq,MAX_K)
Xseq = None
y=np.asarray(y)
show_time()2021-08-18 09:22:41 EDT
# Assume X and y were shuffled.
train_size=PC_TRAINS+NC_TRAINS
X_train=Xfrq[:train_size]
X_test=Xfrq[train_size:]
y_train=y[:train_size]
y_test=y[train_size:]
print("Training set size=",len(X_train),"=",len(y_train))
print("Reserved test set size=",len(X_test),"=",len(y_test))
Xfrq=None
y=None
show_time()Training set size= 2000 = 2000
Reserved test set size= 100300 = 100300
2021-08-18 09:22:41 EDT

## Load a trained neural network_____no_output_____

show_time()
model = load_model(BESTMODELPATH)
print(model.summary())2021-08-18 09:22:41 EDT

## Test the neural network_____no_output_____

def show_test_AUC(model,X,y):
    ns_probs = [0 for _ in range(len(y))]
    bm_probs = model.predict(X)
    ns_auc = roc_auc_score(y, ns_probs)
    bm_auc = roc_auc_score(y, bm_probs)
    ns_fpr, ns_tpr, _ = roc_curve(y, ns_probs)
    bm_fpr, bm_tpr, _ = roc_curve(y, bm_probs)
    plt.plot(ns_fpr, ns_tpr, linestyle='--', label='Guess, auc=%.4f'%ns_auc)
    plt.plot(bm_fpr, bm_tpr, marker='.', label='Model, auc=%.4f'%bm_auc)
    plt.title('ROC')
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.legend()
    plt.show()
    print("%s: %.2f%%" %('AUC',bm_auc*100.0))
def show_test_accuracy(model,X,y):
    scores = model.evaluate(X, y, verbose=0)
    print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
_____no_output_____print("Accuracy on test data.")
show_time()
show_test_AUC(model,X_test,y_test)
show_test_accuracy(model,X_test,y_test)
show_time()Accuracy on test data.
2021-08-18 09:22:41 EDT


# Notebook from ljsun88/data_bootcamp_nyu
Path: Code/IPython/bootcamp_pandas-merge.ipynb

# Pandas 4:  Combining data

Sometimes we need to combine data from two or more dataframes.  That's colloquially known as a **merge** or a **join**. There are lots of ways to do this.  We do a couple but supply references to more at the end.

Along the way we take an extended detour to review methods for **downloading** and **unzipping** compressed files.  The tools we use here have a broad range of other applications, including web scraping.

Outline:

* [MovieLens data](#movielens).  A collection of movies and individual ratings.
* [Automate file download](#requests).  Use the requests package to get a zipped file, then other tools to unzip it and read in the contents.
* [Merge movie names and ratings](#merge-movies).  Merge information from two dataframes with Pandas' `merge` function.
* [UN population data](#example).  We merge ("concatenate") estimates from the past with projections of the future.

**Note: requires internet access to run.**

This IPython notebook was created by Dave Backus, Chase Coleman, Brian LeBlanc, and Spencer Lyon for the NYU Stern course [Data Bootcamp](http://databootcamp.nyuecon.com/).  _____no_output__________no_output_____## Preliminaries

Import packages, etc.  _____no_output_____

import pandas as pd             # data package
import matplotlib.pyplot as plt # graphics
import sys                      # system module, used to get Python version
import os                       # operating system tools (check files)
import datetime as dt           # date tools, used to note current date

# these are new
import requests, io             # internet and input tools
import zipfile as zf            # zip file tools
import shutil                   # file management tools

%matplotlib inline

print('\nPython version: ', sys.version)
print('Pandas version: ', pd.__version__)
print('Requests version: ', requests.__version__)
print("Today's date:", dt.date.today())
Python version:  3.5.1 |Anaconda 4.0.0 (64-bit)| (default, Feb 16 2016, 09:49:46) [MSC v.1900 64 bit (AMD64)]
Pandas version:  0.18.0
Requests version:  2.9.1
Today's date: 2016-04-22

_____no_output_____## MovieLens data

The data comes as a zip file that contains several csv's.  We get the details from the README inside.  (It's written in Markdown, so it's easier to read if we use a browser to format it.  Or we could cut and paste into a Markdown cell in an IPython notebook.)

The file descriptions are:

* `ratings.csv`:  each line is an individual film rating with the rater and movie id's and the rating.  Order:  `userId, movieId, rating, timestamp`.
* `tags.csv`:  each line is a tag on a specific film.  Order:  `userId, movieId, tag, timestamp`.
* `movies.csv`:  each line is a movie name, its id, and its genre.  Order:  `movieId, title, genres`.  Multiple genres are separated by "pipes" `|`.
* `links.csv`:  each line contains the movie id and corresponding id's at [IMBd](http://www.imdb.com/) and [TMDb](https://www.themoviedb.org/).

The easy way to input this data is to download the zip file onto our computer, unzip it, and read the individual csv files using `read.csv()`.  But **anyone can do it the easy way**.  We want to automate this, so we can redo it without any manual steps.  This takes some effort, but once we have it down we can apply it to lots of other data sources.  _____no_output__________no_output_____## Automate file download

We're looking for an automated way, so that if we do this again, possibly with updated data, the whole process is in our code.  Automated data entry involves these steps:

* Get the file.  We use the [requests](http://docs.python-requests.org/) package, which handles internet files and comes pre-installed with Anaconda. This kind of thing was hidden behind the scenes in the Pandas `read_csv` function, but here we need to do it for ourselves. The package authors add:
>Recreational use of other HTTP libraries may result in dangerous side-effects, including: security vulnerabilities, verbose code, reinventing the wheel, constantly reading documentation, depression, headaches, or even death.
* Convert to zip.   Requests simply loads whatever's at the given url. The [io](https://docs.python.org/3.5/library/io.html) module's `io.Bytes` reconstructs it as a file, here a zip file.
* Unzip the file.  We use the [zipfile](https://docs.python.org/3.5/library/zipfile.html) module, which is part of core Python, to extract the files inside.
* Read in the csv's.  Now that we've extracted the csv files, we use `read_csv` as usual.

We found this [Stack Overflow exchange](http://stackoverflow.com/questions/23419322/download-a-zip-file-and-extract-it-in-memory-using-python3) helpful.

**Digression.**  This is probably more than you want to know, but it's a reminder of what goes on behind the scenes when we apply `read_csv` to a url.  Here we grab whatever is at the url.  Then we get its contents, convert it to bytes, identify it as a zip file, and read its components using `read_csv`.  It's a lot easier when this happens automatically, but a reminder what's involved if we ever have to look into the details.  _____no_output_____

# get "response" from url
url = 'http://files.grouplens.org/datasets/movielens/ml-latest-small.zip'
r = requests.get(url)

# describe response
print('Response status code:', r.status_code)
print('Response type:', type(r))
print('Response .content:', type(r.content))
print('Response headers:\n', r.headers, sep='')Response status code: 200
Response type: <class 'requests.models.Response'>
Response .content: <class 'bytes'>
Response headers:
{'Content-Type': 'application/zip', 'Server': 'Apache/2.2.22 (Ubuntu)', 'Date': 'Fri, 22 Apr 2016 17:46:05 GMT', 'Accept-Ranges': 'bytes', 'Keep-Alive': 'timeout=5, max=100', 'ETag': '"80552-fe029-5291222b37ae7"', 'Content-Length': '1040425', 'Connection': 'Keep-Alive', 'Last-Modified': 'Mon, 11 Jan 2016 17:19:11 GMT'}
# convert bytes to zip file
mlz = zf.ZipFile(io.BytesIO(r.content))
print('Type of zipfile object:', type(mlz))Type of zipfile object: <class 'zipfile.ZipFile'>
# what's in the zip file?
mlz.namelist()_____no_output_____# extract and read csv's
movies  = pd.read_csv(mlz.open(mlz.namelist()[2]))
ratings = pd.read_csv(mlz.open(mlz.namelist()[3]))_____no_output_____# what do we have?
for df in [movies, ratings]:
    print('Type:', type(df))
    print('Dimensions:', df.shape)
    print('Variables:', list(df))
    print('First few rows', df.head(3), '\n')Type: <class 'pandas.core.frame.DataFrame'>
Dimensions: (10329, 3)
Variables: ['movieId', 'title', 'genres']
First few rows    movieId                    title  \
0        1         Toy Story (1995)
1        2           Jumanji (1995)
2        3  Grumpier Old Men (1995)

                                        genres
0  Adventure|Animation|Children|Comedy|Fantasy
1                   Adventure|Children|Fantasy
2                               Comedy|Romance

Type: <class 'pandas.core.frame.DataFrame'>
Dimensions: (105339, 4)
Variables: ['userId', 'movieId', 'rating', 'timestamp']
First few rows    userId  movieId  rating   timestamp
0       1       16     4.0  1217897793
1       1       24     1.5  1217895807
2       1       32     4.0  1217896246

**Exercise.** Something to do together.  suppose we wanted to save the files on our computer.  How would we do it? Would we prefer individual csv's or a single zip?_____no_output_____

# experiment via http://stackoverflow.com/a/18043472/804513
with open('test.zip', 'wb') as out_file:
    shutil.copyfileobj(io.BytesIO(r.content), out_file)   _____no_output_____

_____no_output_____## Merging ratings and movie titles

The movie ratings in the dataframe `ratings` give us individual opinions about movies, but they don't include the name of the movie.   Why not?  Rather than include the name every time a movie is rated, the MovieLens data associates each rating with a movie code, than stores the names of movies associatd with each movie code in the dataframe `movies`.  We run across this a lot:  some information is in one data table, other information is in another.

Our **want** is therefore to add the movie name to the `ratings` dataframe.  We say we **merge** the two dataferames.  There are lots of ways to merge.  Here we do one as an illustration.

Let's start by reminding ourselves what we have.  _____no_output_____

ratings.head(3)_____no_output_____movies.head(3)_____no_output_____

### Merging

Here's roughly what's involved in what we're doing.  We take the `movieId` variable from `ratings` and look it up in `movies`.  When we find it, we look up the `title` and add it as a column in `ratings`.  The variable `movieId` is common, so we can use it to link the two dataframes.  _____no_output_____

combo = pd.merge(ratings, movies,   # left and right df's
                 how='left',        # add to left
                 on='movieId'       # link with this variable/column
                )

print('Dimensions of ratings:', ratings.shape)
print('Dimensions of movies:', movies.shape)
print('Dimensions of new df:', combo.shape)

combo.head(20)Dimensions of ratings: (105339, 4)
Dimensions of movies: (10329, 3)
Dimensions of new df: (105339, 6)
# save as csv file for future use
combo.to_csv('mlcombined.csv')_____no_output_____print('Current directory:\n', os.getcwd(), sep='')
print('List of files:', os.listdir(), sep='\n')Current directory:
C:\Users\dbackus\Dropbox\Documents\Classes\Data_Bootcamp\Code\IPython
List of files:
['.ipynb_checkpoints', 'bootcamp_advgraphics_seaborn.ipynb', 'bootcamp_data_management.ipynb', 'bootcamp_data_management_datasets.ipynb', 'bootcamp_examples.ipynb', 'bootcamp_exam_practice.ipynb', 'bootcamp_exam_practice_answerkey.ipynb', 'bootcamp_exam_s16_answerkey.ipynb', 'bootcamp_graphics.ipynb', 'bootcamp_indicators.ipynb', 'bootcamp_pandas-clean.ipynb', 'bootcamp_pandas-group.ipynb', 'bootcamp_pandas-merge.ipynb', 'bootcamp_pandas-shape.ipynb', 'bootcamp_pandas-summarize.ipynb', 'bootcamp_practice_a.ipynb', 'bootcamp_practice_a_answerkey.ipynb', 'bootcamp_sandbox.ipynb', 'bootcamp_test.ipynb', 'mlcombined.csv', 'test.zip']

**Exercise.** Some of these we know how to do, the others we don't.  For the ones we know, what is the answer?  For the others, what (in loose terms) do we need to be able to do to come up with an answer?

* What is the overall average rating?
* What is the overall distribution of ratings?
* What is the average rating of each movie?
* How many ratings does each movie get?  _____no_output_____

"""
fig, ax = plt.subplots()
bins = [bin/100 for bin in list(range(25, 575, 50))]
print(bins)
combo['rating'].plot(kind='hist', ax=ax, bins=bins, color='blue', alpha=0.5)
ax.set_xlim(0,5.5)
ax.set_ylabel('Number')
ax.set_xlabel('Rating')
"""_____no_output_____

_____no_output_____## Population "estimates" and "projections"

We look (again) at the UN's  [population data](http://esa.un.org/unpd/wpp/Download/Standard/Population/), specifically the age distribution of the population.  The data comes in two sheets:  *estimates* that cover the period 1950-2015 and *projections* that cover 2016-2100.  Our mission is to combine them.  _____no_output_____### Load data

We start, as usual, by loading the data.  This takes a minute or so.  _____no_output_____

url1 = 'http://esa.un.org/unpd/wpp/DVD/Files/'
url2 = '1_Indicators%20(Standard)/EXCEL_FILES/1_Population/'
url3 = 'WPP2015_POP_F07_1_POPULATION_BY_AGE_BOTH_SEXES.XLS'
url = url1 + url2 + url3

cols = [2, 5] + list(range(6,28))
est = pd.read_excel(url, sheetname=0, skiprows=16, parse_cols=cols, na_values=['…'])
prj = pd.read_excel(url, sheetname=1, skiprows=16, parse_cols=cols, na_values=['…'])

print('Dimensions and dtypes of estimates: ', est.shape, '\n', est.dtypes.head(), sep='')
print('\nDimensions and dtypes of projections: ', prj.shape, '\n', prj.dtypes.head(), sep='')Dimensions and dtypes of estimates: (3374, 24)
Major area, region, country or area *     object
Reference date (as of 1 July)              int64
0-4                                      float64
5-9                                      float64
10-14                                    float64
dtype: object

Dimensions and dtypes of projections: (4338, 23)
Major area, region, country or area *     object
Reference date (as of 1 July)              int64
0-4                                      float64
5-9                                      float64
10-14                                    float64
dtype: object

**Comment.** Note that they have different numbers of columns.  Let's see where that comes from.  _____no_output_____

list(est)[15:]_____no_output_____list(prj)[15:]_____no_output_____

### Clean data

Pick a useable subset and fix extra column so that we can combine them.  The problem here is that until 1990, the highest age category was `'80+`.  From 1990 on, we have a finer breakdown.

We fix this by reassigning `'80+'` to `'80-84'` and not worrying that some of these people are 85 or older.  Note that `df.fillna(0.0)` replaces missing values with zeros.  _____no_output_____

def cleanpop(df, countries, years):
    """
    take df as input and select countries and years
    """
    # rename first two columns
    names = list(df)
    df = df.rename(columns={names[0]: 'Country', names[1]: 'Year'})

    # select countries and years
    newdf = df[df['Country'].isin(countries) & df['Year'].isin(years)]

    return newdf

countries = ['Japan']
past      = [1950, 2000]
future    = [2050, 2100]

e = cleanpop(est, countries, past)
p = cleanpop(prj, countries, future)

# make copie sfor later use
ealt = e.copy()
palt = p.copy()

# fix top-coding in estimates
e['80-84'] = e['80-84'].fillna(0.0) + e['80+'].fillna(0.0)
e = e.drop(['80+'], axis=1)

# check dimensions again
print('Dimensions of cleaned estimates: ', e.shape)
print('Dimensions of cleaned projections: ', p.shape)Dimensions of cleaned estimates:  (2, 23)
Dimensions of cleaned projections:  (2, 23)
# check to see if we have the same variables
list(e) == list(p)_____no_output_____

### Merge estimates and projections

If we have two blocks of data, and want just want to put them on top of each other, we use the Pandas' `concatenate` function.  Ditto two blocks next to each other.

But first we need to fix the difference in the columns of the two dataframes.  _____no_output_____

pop = pd.concat([e, p], axis=0).fillna(0.0)
pop_____no_output_____

**Exercise.** What happens if we try to merge the original dataframes, including the one with the extra `80+` column?  Run the code below and comment on what you get.  _____no_output_____

"""
popalt = pd.concat([ealt, palt], axis=0)
popalt
"""_____no_output_____

### Shape data

We **want** age categories in the index (the default x axis in a plot) and the years in the columns.  The country we don't care about because there's only one.  _____no_output_____

pop = pop.drop('Country', axis=1)
popi = pop.set_index('Year')
popi _____no_output_____popi.columns.name = 'Age'
popt = popi.T
popt.head()_____no_output_____ax = popt.plot(kind='bar', color='blue', alpha=0.5,
               subplots=True,
               sharey=True,
               figsize=(8,12)) _____no_output_____

**Exercise.**  Use `set_index`, `stack`, and `unstack` to shape the dataframe `popi` into `popt`.  _____no_output_____## Resources

The [Pandas docs](http://pandas.pydata.org/pandas-docs/stable/merging.html) are ok, but we prefer the Data Carpentry [guide](http://www.datacarpentry.org/python-ecology-lesson/04-merging-data)_____no_output_____