# Notebook from bjrnmath/debuggingbook
Path: notebooks/StatisticalDebugger.ipynb

# Statistical Debugging

In this chapter, we introduce _statistical debugging_ – the idea that specific events during execution could be _statistically correlated_ with failures. We start with coverage of individual lines and then proceed towards further execution features._____no_output_____

from bookutils import YouTubeVideo
YouTubeVideo("UNuso00zYiI")_____no_output_____

**Prerequisites**

* You should have read the [chapter on tracing executions](Tracer.ipynb)._____no_output_____

import bookutils_____no_output_____

## Synopsis

To [use the code provided in this chapter](Importing.ipynb), write

```python
>>> from debuggingbook.StatisticalDebugger import <identifier>
```

and then make use of the following features.

This chapter introduces classes and techniques for _statistical debugging_ – that is, correlating specific events, such as lines covered, with passing and failing outcomes.

To make use of the code in this chapter, use one of the provided `StatisticalDebugger` subclasses such as `TarantulaDebugger` or `OchiaiDebugger`.

Both are instantiated with a `Collector` denoting the type of events you want to correlate outcomes with. The default `CoverageCollector`, collecting line coverage.

### Collecting Events from Calls

To collect events from calls that are labeled manually, use

```python
>>> debugger = TarantulaDebugger()
>>> with debugger.collect_pass():
>>>     remove_html_markup("abc")
>>> with debugger.collect_pass():
>>>     remove_html_markup('<b>abc</b>')
>>> with debugger.collect_fail():
>>>     remove_html_markup('"abc"')
```
Within each `with` block, the _first function call_ is collected and tracked for coverage. (Note that _only_ the first call is tracked.)

### Collecting Events from Tests

To collect events from _tests_ that use exceptions to indicate failure, use the simpler `with` form:

```python
>>> debugger = TarantulaDebugger()
>>> with debugger:
>>>     remove_html_markup("abc")
>>> with debugger:
>>>     remove_html_markup('<b>abc</b>')
>>> with debugger:
>>>     remove_html_markup('"abc"')
>>>     assert False  # raise an exception
```
`with` blocks that raise an exception will be classified as failing, blocks that do not will be classified as passing. Note that exceptions raised are "swallowed" by the debugger.

### Visualizing Events as a Table

After collecting events, you can print out the observed events – in this case, line numbers – in a table, showing in which runs they occurred (`X`), and with colors highlighting the suspiciousness of the event. A "red" event means that the event predominantly occurs in failing runs.

```python
>>> debugger.event_table(args=True, color=True)
```
| `remove_html_markup` | `s='abc'` | `s='<b>abc</b>'` | `s='"abc"'` |
| --------------------- | ---- | ---- | ---- |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%"> remove_html_markup:1</samp> |    X |    X |    X |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%"> remove_html_markup:2</samp> |    X |    X |    X |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%"> remove_html_markup:3</samp> |    X |    X |    X |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%"> remove_html_markup:4</samp> |    X |    X |    X |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%"> remove_html_markup:6</samp> |    X |    X |    X |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%"> remove_html_markup:7</samp> |    X |    X |    X |
| <samp style="background-color: hsl(120.0, 50.0%, 80%)" title="  0%"> remove_html_markup:8</samp> |    - |    X |    - |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%"> remove_html_markup:9</samp> |    X |    X |    X |
| <samp style="background-color: hsl(120.0, 50.0%, 80%)" title="  0%">remove_html_markup:10</samp> |    - |    X |    - |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%">remove_html_markup:11</samp> |    X |    X |    X |
| <samp style="background-color: hsl(0.0, 100.0%, 80%)" title="100%">remove_html_markup:12</samp> |    - |    - |    X |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%">remove_html_markup:13</samp> |    X |    X |    X |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%">remove_html_markup:14</samp> |    X |    X |    X |
| <samp style="background-color: hsl(60.0, 100.0%, 80%)" title=" 50%">remove_html_markup:16</samp> |    X |    X |    X |

### Visualizing Suspicious Code

If you collected coverage with `CoverageCollector`, you can also visualize the code with similar colors, highlighting suspicious lines:

```python
>>> debugger
```

   1 def remove_html_markup(s):  # type: ignore

   2     tag = False

   3     quote = False

   4     out = ""

   5

   6     for c in s:

   7         if c == '<' and not quote:

   8             tag = True

   9         elif c == '>' and not quote:

  10             tag = False

  11         elif c == '"' or c == "'" and tag:

  12             quote = not quote

  13         elif not tag:

  14             out = out + c

  15

  16     return out

### Ranking Events

The method `rank()` returns a ranked list of events, starting with the most suspicious. This is useful for automated techniques that need potential defect locations.

```python
>>> debugger.rank()
[('remove_html_markup', 12),
 ('remove_html_markup', 3),
 ('remove_html_markup', 13),
 ('remove_html_markup', 4),
 ('remove_html_markup', 6),
 ('remove_html_markup', 9),
 ('remove_html_markup', 11),
 ('remove_html_markup', 2),
 ('remove_html_markup', 16),
 ('remove_html_markup', 14),
 ('remove_html_markup', 7),
 ('remove_html_markup', 1),
 ('remove_html_markup', 8),
 ('remove_html_markup', 10)]
```
### Classes and Methods

Here are all classes defined in this chapter:

![](PICS/StatisticalDebugger-synopsis-1.svg)
![](PICS/StatisticalDebugger-synopsis-2.svg)
_____no_output_____## Introduction

The idea behind _statistical debugging_ is fairly simple. We have a program that sometimes passes and sometimes fails. This outcome can be _correlated_ with events that precede it – properties of the input, properties of the execution, properties of the program state. If we, for instance, can find that "the program always fails when Line 123 is executed, and it always passes when Line 123 is _not_ executed", then we have a strong correlation between Line 123 being executed and failure.

Such _correlation_ does not necessarily mean _causation_. For this, we would have to prove that executing Line 123 _always_ leads to failure, and that _not_ executing it does not lead to (this) failure. Also, a correlation (or even a causation) does not mean that Line 123 contains the defect – for this, we would have to show that it actually is an error. Still, correlations make excellent hints as it comes to search for failure causes – in all generality, if you let your search be guided by _events that correlate with failures_, you are more likely to find _important hints on how the failure comes to be_._____no_output_____## Collecting Events

How can we determine events that correlate with failure? We start with a general mechanism to actually _collect_ events during execution. The abstract `Collector` class provides

* a `collect()` method made for collecting events, called from the `traceit()` tracer; and
* an `events()` method made for retrieving these events.

Both of these are _abstract_ and will be defined further in subclasses._____no_output_____

from Tracer import Tracer_____no_output_____# ignore
from typing import Any, Callable, Optional, Type, Tuple
from typing import Dict, Set, List, TypeVar, Union_____no_output_____from types import FrameType, TracebackType_____no_output_____class Collector(Tracer):
    """A class to record events during execution."""

    def collect(self, frame: FrameType, event: str, arg: Any) -> None:
        """Collecting function. To be overridden in subclasses."""
        pass

    def events(self) -> Set:
        """Return a collection of events. To be overridden in subclasses."""
        return set()

    def traceit(self, frame: FrameType, event: str, arg: Any) -> None:
        self.collect(frame, event, arg)_____no_output_____

A `Collector` class is used like `Tracer`, using a `with` statement. Let us apply it on the buggy variant of `remove_html_markup()` from the [Introduction to Debugging](Intro_Debugging.ipynb):_____no_output_____

def remove_html_markup(s):  # type: ignore
    tag = False
    quote = False
    out = ""

    for c in s:
        if c == '<' and not quote:
            tag = True
        elif c == '>' and not quote:
            tag = False
        elif c == '"' or c == "'" and tag:
            quote = not quote
        elif not tag:
            out = out + c

    return out_____no_output_____with Collector() as c:
    out = remove_html_markup('"abc"')
out_____no_output_____

There's not much we can do with our collector, as the `collect()` and `events()` methods are yet empty. However, we can introduce an `id()` method which returns a string identifying the collector. This string is defined from the _first function call_ encountered._____no_output_____

from types import FunctionType_____no_output_____Coverage = Set[Tuple[Callable, int]]_____no_output_____class Collector(Collector):
    def __init__(self) -> None:
        """Constructor."""
        self._function: Optional[Callable] = None
        self._args: Optional[Dict[str, Any]] = None
        self._argstring: Optional[str] = None
        self._exception: Optional[Type] = None
        self.items_to_ignore: List[Union[Type, Callable]] = [self.__class__]

    def traceit(self, frame: FrameType, event: str, arg: Any) -> None:
        """
        Tracing function.
        Saves the first function and calls collect().
        """
        for item in self.items_to_ignore:
            if (isinstance(item, type) and 'self' in frame.f_locals and
                isinstance(frame.f_locals['self'], item)):
                # Ignore this class
                return
            if item.__name__ == frame.f_code.co_name:
                # Ignore this function
                return

        if self._function is None and event == 'call':
            # Save function
            self._function = self.create_function(frame)
            self._args = frame.f_locals.copy()
            self._argstring = ", ".join([f"{var}={repr(self._args[var])}"
                                         for var in self._args])

        self.collect(frame, event, arg)

    def collect(self, frame: FrameType, event: str, arg: Any) -> None:
        """Collector function. To be overloaded in subclasses."""
        pass

    def id(self) -> str:
        """Return an identifier for the collector,
        created from the first call"""
        return f"{self.function().__name__}({self.argstring()})"

    def function(self) -> Callable:
        """Return the function from the first call, as a function object"""
        if not self._function:
            raise ValueError("No call collected")
        return self._function

    def argstring(self) -> str:
        """
        Return the list of arguments from the first call,
        as a printable string
        """
        if not self._argstring:
            raise ValueError("No call collected")
        return self._argstring

    def args(self) -> Dict[str, Any]:
        """Return a dict of argument names and values from the first call"""
        if not self._args:
            raise ValueError("No call collected")
        return self._args

    def exception(self) -> Optional[Type]:
        """Return the exception class from the first call,
        or None if no exception was raised."""
        return self._exception

    def __repr__(self) -> str:
        """Return a string representation of the collector"""
        # We use the ID as default representation when printed
        return self.id()

    def covered_functions(self) -> Set[Callable]:
        """Set of covered functions. To be overloaded in subclasses."""
        return set()

    def coverage(self) -> Coverage:
        """
        Return a set (function, lineno) with locations covered.
        To be overloaded in subclasses.
        """
        return set()_____no_output_____

Here's how the collector works. We use a `with` clause to collect details on a function call:_____no_output_____

with Collector() as c:
    remove_html_markup('abc')_____no_output_____

We can now retrieve details such as the function called..._____no_output_____

c.function()_____no_output_____

... or its arguments, as a name/value dictionary._____no_output_____

c.args()_____no_output_____

The `id()` method returns a printable representation of the call:_____no_output_____

c.id()_____no_output_____

The `argstring()` method does the same for the argument string only._____no_output_____

c.argstring()_____no_output_____

With this, we can collect the basic information to identify calls – such that we can later correlate their events with success or failure._____no_output_____### Error Prevention_____no_output_____While collecting, we'd like to avoid collecting events in the collection infrastructure. The `items_to_ignore` attribute takes care of this._____no_output_____

class Collector(Collector):
    def add_items_to_ignore(self,
                            items_to_ignore: List[Union[Type, Callable]]) \
                            -> None:
        """
        Define additional classes and functions to ignore during collection
        (typically `Debugger` classes using these collectors).
        """
        self.items_to_ignore += items_to_ignore_____no_output_____

If we exit a block without having collected anything, that's likely an error._____no_output_____

class Collector(Collector):
    def __exit__(self, exc_tp: Type, exc_value: BaseException,
                 exc_traceback: TracebackType) -> Optional[bool]:
        """Exit the `with` block."""
        ret = super().__exit__(exc_tp, exc_value, exc_traceback)

        if not self._function:
            if exc_tp:
                return False  # re-raise exception
            else:
                raise ValueError("No call collected")

        return ret_____no_output_____

## Collecting Coverage

So far, our `Collector` class does not collect any events. Let us extend it such that it collects _coverage_ information – that is, the set of locations executed. To this end, we introduce a `CoverageCollector` subclass which saves the coverage in a set containing functions and line numbers._____no_output_____

from types import FrameType_____no_output_____from StackInspector import StackInspector_____no_output_____class CoverageCollector(Collector, StackInspector):
    """A class to record covered locations during execution."""

    def __init__(self) -> None:
        """Constructor."""
        super().__init__()
        self._coverage: Coverage = set()

    def collect(self, frame: FrameType, event: str, arg: Any) -> None:
        """
        Save coverage for an observed event.
        """
        name = frame.f_code.co_name
        function = self.search_func(name, frame)

        if function is None:
            function = self.create_function(frame)

        location = (function, frame.f_lineno)
        self._coverage.add(location)_____no_output_____

We also override `events()` such that it returns the set of covered locations._____no_output_____

class CoverageCollector(CoverageCollector):
    def events(self) -> Set[Tuple[str, int]]:
        """
        Return the set of locations covered.
        Each location comes as a pair (`function_name`, `lineno`).
        """
        return {(func.__name__, lineno) for func, lineno in self._coverage}_____no_output_____

The methods `coverage()` and `covered_functions()` allow precise access to the coverage obtained._____no_output_____

class CoverageCollector(CoverageCollector):
    def covered_functions(self) -> Set[Callable]:
        """Return a set with all functions covered."""
        return {func for func, lineno in self._coverage}

    def coverage(self) -> Coverage:
        """Return a set (function, lineno) with all locations covered."""
        return self._coverage_____no_output_____

Here is how we can use `CoverageCollector` to determine the lines executed during a run of `remove_html_markup()`:_____no_output_____

with CoverageCollector() as c:
    remove_html_markup('abc')
c.events()_____no_output_____

Sets of line numbers alone are not too revealing. They provide more insights if we actually list the code, highlighting these numbers:_____no_output_____

import inspect_____no_output_____from bookutils import getsourcelines    # like inspect.getsourcelines(), but in color_____no_output_____def code_with_coverage(function: Callable, coverage: Coverage) -> None:
    source_lines, starting_line_number = \
       getsourcelines(function)

    line_number = starting_line_number
    for line in source_lines:
        marker = '*' if (function, line_number) in coverage else ' '
        print(f"{line_number:4} {marker} {line}", end='')
        line_number += 1_____no_output_____code_with_coverage(remove_html_markup, c.coverage())_____no_output_____

Remember that the input `s` was `"abc"`? In this listing, we can see which lines were covered and which lines were not. From the listing already, we can see that `s` has neither tags nor quotes._____no_output_____Such coverage computation plays a big role in _testing_, as one wants tests to cover as many different aspects of program execution (and notably code) as possible. But also during debugging, code coverage is essential: If some code was not even executed in the failing run, then any change to it will have no effect._____no_output_____

from bookutils import quiz_____no_output_____quiz('Let the input be `"<b>Don\'t do this!</b>"`. '
     "Which of these lines are executed? Use the code to find out!",
     [
         "`tag = True`",
         "`tag = False`",
         "`quote = not quote`",
         "`out = out + c`"
     ], "[ord(c) - ord('a') - 1 for c in 'cdf']")_____no_output_____

To find the solution, try this out yourself:_____no_output_____

with CoverageCollector() as c:
    remove_html_markup("Don't do this!")
# code_with_coverage(remove_html_markup, c.coverage)_____no_output_____

## Computing Differences

Let us get back to the idea that we want to _correlate_ events with passing and failing outcomes. For this, we need to examine events in both _passing_ and _failing_ runs, and determine their _differences_ – since it is these differences we want to associate with their respective outcome._____no_output_____### A Base Class for Statistical Debugging

The `StatisticalDebugger` base class takes a collector class (such as `CoverageCollector`). Its `collect()` method creates a new collector of that very class, which will be maintained by the debugger. As argument, `collect()` takes a string characterizing the outcome (such as `'PASS'` or `'FAIL'`). This is how one would use it:

```python
debugger = StatisticalDebugger()
with debugger.collect('PASS'):
    some_passing_run()
with debugger.collect('PASS'):
    another_passing_run()
with debugger.collect('FAIL'):
    some_failing_run()
```_____no_output_____Let us implement `StatisticalDebugger`. The base class gets a collector class as argument:_____no_output_____

class StatisticalDebugger:
    """A class to collect events for multiple outcomes."""

    def __init__(self, collector_class: Type = CoverageCollector, log: bool = False):
        """Constructor. Use instances of `collector_class` to collect events."""
        self.collector_class = collector_class
        self.collectors: Dict[str, List[Collector]] = {}
        self.log = log_____no_output_____

The `collect()` method creates (and stores) a collector for the given outcome, using the given outcome to characterize the run. Any additional arguments are passed to the collector._____no_output_____

class StatisticalDebugger(StatisticalDebugger):
    def collect(self, outcome: str, *args: Any, **kwargs: Any) -> Collector:
        """Return a collector for the given outcome.
        Additional args are passed to the collector."""
        collector = self.collector_class(*args, **kwargs)
        collector.add_items_to_ignore([self.__class__])
        return self.add_collector(outcome, collector)

    def add_collector(self, outcome: str, collector: Collector) -> Collector:
        if outcome not in self.collectors:
            self.collectors[outcome] = []
        self.collectors[outcome].append(collector)
        return collector_____no_output_____

The `all_events()` method produces a union of all events observed. If an outcome is given, it produces a union of all events with that outcome:_____no_output_____

class StatisticalDebugger(StatisticalDebugger):
    def all_events(self, outcome: Optional[str] = None) -> Set[Any]:
        """Return a set of all events observed."""
        all_events = set()

        if outcome:
            if outcome in self.collectors:
                for collector in self.collectors[outcome]:
                    all_events.update(collector.events())
        else:
            for outcome in self.collectors:
                for collector in self.collectors[outcome]:
                    all_events.update(collector.events())

        return all_events_____no_output_____

Here's a simple example of `StatisticalDebugger` in action:_____no_output_____

s = StatisticalDebugger()
with s.collect('PASS'):
    remove_html_markup("abc")
with s.collect('PASS'):
    remove_html_markup('abc')
with s.collect('FAIL'):
    remove_html_markup('"abc"')_____no_output_____

The method `all_events()` returns all events collected:_____no_output_____

s.all_events()_____no_output_____

If given an outcome as argument, we obtain all events with the given outcome._____no_output_____

s.all_events('FAIL')_____no_output_____

The attribute `collectors` maps outcomes to lists of collectors:_____no_output_____

s.collectors_____no_output_____

Here's the collector of the one (and first) passing run:_____no_output_____

s.collectors['PASS'][0].id()_____no_output_____s.collectors['PASS'][0].events()_____no_output_____

To better highlight the differences between the collected events, we introduce a method `event_table()` that prints out whether an event took place in a run._____no_output_____### Excursion: Printing an Event Table_____no_output_____

from IPython.display import Markdown_____no_output_____import html_____no_output_____class StatisticalDebugger(StatisticalDebugger):
    def function(self) -> Optional[Callable]:
        """
        Return the entry function from the events observed,
        or None if ambiguous.
        """
        names_seen = set()
        functions = []
        for outcome in self.collectors:
            for collector in self.collectors[outcome]:
                # We may have multiple copies of the function,
                # but sharing the same name
                func = collector.function()
                if func.__name__ not in names_seen:
                    functions.append(func)
                    names_seen.add(func.__name__)

        if len(functions) != 1:
            return None  # ambiguous
        return functions[0]

    def covered_functions(self) -> Set[Callable]:
        """Return a set of all functions observed."""
        functions = set()
        for outcome in self.collectors:
            for collector in self.collectors[outcome]:
                functions |= collector.covered_functions()
        return functions

    def coverage(self) -> Coverage:
        """Return a set of all (functions, line_numbers) observed"""
        coverage = set()
        for outcome in self.collectors:
            for collector in self.collectors[outcome]:
                coverage |= collector.coverage()
        return coverage

    def color(self, event: Any) -> Optional[str]:
        """
        Return a color for the given event, or None.
        To be overloaded in subclasses.
        """
        return None

    def tooltip(self, event: Any) -> Optional[str]:
        """
        Return a tooltip string for the given event, or None.
        To be overloaded in subclasses.
        """
        return None

    def event_str(self, event: Any) -> str:
        """Format the given event. To be overloaded in subclasses."""
        if isinstance(event, str):
            return event
        if isinstance(event, tuple):
            return ":".join(self.event_str(elem) for elem in event)
        return str(event)

    def event_table_text(self, *, args: bool = False, color: bool = False) -> str:
        """
        Print out a table of events observed.
        If `args` is True, use arguments as headers.
        If `color` is True, use colors.
        """
        sep = ' | '
        all_events = self.all_events()
        longest_event = max(len(f"{self.event_str(event)}")
                            for event in all_events)
        out = ""

        # Header
        if args:
            out += '| '
            func = self.function()
            if func:
                out += '`' + func.__name__ + '`'
            out += sep
            for name in self.collectors:
                for collector in self.collectors[name]:
                    out += '`' + collector.argstring() + '`' + sep
            out += '\n'
        else:
            out += '| ' + ' ' * longest_event + sep
            for name in self.collectors:
                for i in range(len(self.collectors[name])):
                    out += name + sep
            out += '\n'

        out += '| ' + '-' * longest_event + sep
        for name in self.collectors:
            for i in range(len(self.collectors[name])):
                out += '-' * len(name) + sep
        out += '\n'

        # Data
        for event in sorted(all_events):
            event_name = self.event_str(event).rjust(longest_event)

            tooltip = self.tooltip(event)
            if tooltip:
                title = f' title="{tooltip}"'
            else:
                title = ''

            if color:
                color_name = self.color(event)
                if color_name:
                    event_name = \
                        f'<samp style="background-color: {color_name}"{title}>' \
                        f'{html.escape(event_name)}' \
                        f'</samp>'

            out += f"| {event_name}" + sep
            for name in self.collectors:
                for collector in self.collectors[name]:
                    out += ' ' * (len(name) - 1)
                    if event in collector.events():
                        out += "X"
                    else:
                        out += "-"
                    out += sep
            out += '\n'

        return out

    def event_table(self, **_args: Any) -> Any:
        """Print out event table in Markdown format."""
        return Markdown(self.event_table_text(**_args))

    def __repr__(self) -> str:
        return self.event_table_text()

    def _repr_markdown_(self) -> str:
        return self.event_table_text(args=True, color=True)_____no_output_____

### End of Excursion_____no_output_____

s = StatisticalDebugger()
with s.collect('PASS'):
    remove_html_markup("abc")
with s.collect('PASS'):
    remove_html_markup('abc')
with s.collect('FAIL'):
    remove_html_markup('"abc"')_____no_output_____s.event_table(args=True)_____no_output_____quiz("How many lines are executed in the failing run only?",
     [
         "One",
         "Two",
         "Three"
     ], 'len([12])')_____no_output_____

Indeed, Line 12 executed in the failing run only would be a correlation to look for._____no_output_____### Collecting Passing and Failing Runs

While our `StatisticalDebugger` class allows arbitrary outcomes, we are typically only interested in two outcomes, namely _passing_ vs. _failing_ runs. We therefore introduce a specialized `DifferenceDebugger` class that provides customized methods to collect and access passing and failing runs._____no_output_____

class DifferenceDebugger(StatisticalDebugger):
    """A class to collect events for passing and failing outcomes."""

    PASS = 'PASS'
    FAIL = 'FAIL'

    def collect_pass(self, *args: Any, **kwargs: Any) -> Collector:
        """Return a collector for passing runs."""
        return self.collect(self.PASS, *args, **kwargs)

    def collect_fail(self, *args: Any, **kwargs: Any) -> Collector:
        """Return a collector for failing runs."""
        return self.collect(self.FAIL, *args, **kwargs)

    def pass_collectors(self) -> List[Collector]:
        return self.collectors[self.PASS]

    def fail_collectors(self) -> List[Collector]:
        return self.collectors[self.FAIL]

    def all_fail_events(self) -> Set[Any]:
        """Return all events observed in failing runs."""
        return self.all_events(self.FAIL)

    def all_pass_events(self) -> Set[Any]:
        """Return all events observed in passing runs."""
        return self.all_events(self.PASS)

    def only_fail_events(self) -> Set[Any]:
        """Return all events observed only in failing runs."""
        return self.all_fail_events() - self.all_pass_events()

    def only_pass_events(self) -> Set[Any]:
        """Return all events observed only in passing runs."""
        return self.all_pass_events() - self.all_fail_events()_____no_output_____

We can use `DifferenceDebugger` just as a `StatisticalDebugger`:_____no_output_____

# ignore
T1 = TypeVar('T1', bound='DifferenceDebugger')_____no_output_____def test_debugger_html_simple(debugger: T1) -> T1:
    with debugger.collect_pass():
        remove_html_markup('abc')
    with debugger.collect_pass():
        remove_html_markup('abc')
    with debugger.collect_fail():
        remove_html_markup('"abc"')
    return debugger_____no_output_____

However, since the outcome of tests may not always be predetermined, we provide a simpler interface for tests that can fail (= raise an exception) or pass (not raise an exception)._____no_output_____

class DifferenceDebugger(DifferenceDebugger):
    def __enter__(self) -> Any:
        """Enter a `with` block. Collect coverage and outcome;
        classify as FAIL if the block raises an exception,
        and PASS if it does not.
        """
        self.collector = self.collector_class()
        self.collector.add_items_to_ignore([self.__class__])
        self.collector.__enter__()
        return self

    def __exit__(self, exc_tp: Type, exc_value: BaseException,
                 exc_traceback: TracebackType) -> Optional[bool]:
        """Exit the `with` block."""
        status = self.collector.__exit__(exc_tp, exc_value, exc_traceback)

        if status is None:
            pass
        else:
            return False  # Internal error; re-raise exception

        if exc_tp is None:
            outcome = self.PASS
        else:
            outcome = self.FAIL

        self.add_collector(outcome, self.collector)
        return True  # Ignore exception, if any_____no_output_____

Using this interface, we can rewrite `test_debugger_html()`:_____no_output_____

# ignore
T2 = TypeVar('T2', bound='DifferenceDebugger')_____no_output_____def test_debugger_html(debugger: T2) -> T2:
    with debugger:
        remove_html_markup('abc')
    with debugger:
        remove_html_markup('abc')
    with debugger:
        remove_html_markup('"abc"')
        assert False  # Mark test as failing

    return debugger_____no_output_____test_debugger_html(DifferenceDebugger())_____no_output_____

### Analyzing Events

Let us now focus on _analyzing_ events collected. Since events come back as _sets_, we can compute _unions_ and _differences_ between these sets. For instance, we can compute which lines were executed in _any_ of the passing runs of `test_debugger_html()`, above:_____no_output_____

debugger = test_debugger_html(DifferenceDebugger())_____no_output_____pass_1_events = debugger.pass_collectors()[0].events()_____no_output_____pass_2_events = debugger.pass_collectors()[1].events()_____no_output_____in_any_pass = pass_1_events | pass_2_events
in_any_pass_____no_output_____

Likewise, we can determine which lines were _only_ executed in the failing run:_____no_output_____

fail_events = debugger.fail_collectors()[0].events()_____no_output_____only_in_fail = fail_events - in_any_pass
only_in_fail_____no_output_____

And we see that the "failing" run is characterized by processing quotes:_____no_output_____

code_with_coverage(remove_html_markup, only_in_fail)_____no_output_____debugger = test_debugger_html(DifferenceDebugger())_____no_output_____debugger.all_events()_____no_output_____

These are the lines executed only in the failing run:_____no_output_____

debugger.only_fail_events()_____no_output_____

These are the lines executed only in the passing runs:_____no_output_____

debugger.only_pass_events()_____no_output_____

Again, having these lines individually is neat, but things become much more interesting if we can see the associated code lines just as well. That's what we will do in the next section._____no_output_____## Visualizing Differences

To show correlations of line coverage in context, we introduce a number of _visualization_ techniques that _highlight_ code with different colors._____no_output_____### Discrete Spectrum

The first idea is to use a _discrete_ spectrum of three colors:

* _red_ for code executed in failing runs only
* _green_ for code executed in passing runs only
* _yellow_ for code executed in both passing and failing runs.

Code that is not executed stays unhighlighted._____no_output_____We first introduce an abstract class `SpectrumDebugger` that provides the essential functions. `suspiciousness()` returns a value between 0 and 1 indicating the suspiciousness of the given event - or `None` if unknown._____no_output_____

class SpectrumDebugger(DifferenceDebugger):
    def suspiciousness(self, event: Any) -> Optional[float]:
        """
        Return a suspiciousness value in the range [0, 1.0]
        for the given event, or `None` if unknown.
        To be overloaded in subclasses.
        """
        return None_____no_output_____

The `tooltip()` and `percentage()` methods convert the suspiciousness into a human-readable form._____no_output_____

class SpectrumDebugger(SpectrumDebugger):
    def tooltip(self, event: Any) -> str:
        """
        Return a tooltip for the given event (default: percentage).
        To be overloaded in subclasses.
        """
        return self.percentage(event)

    def percentage(self, event: Any) -> str:
        """
        Return the suspiciousness for the given event as percentage string.
        """
        suspiciousness = self.suspiciousness(event)
        if suspiciousness is not None:
            return str(int(suspiciousness * 100)).rjust(3) + '%'
        else:
            return ' ' * len('100%')_____no_output_____

The `code()` method takes a function and shows each of its source code lines using the given spectrum, using HTML markup:_____no_output_____

class SpectrumDebugger(SpectrumDebugger):
    def code(self, functions: Optional[Set[Callable]] = None, *,
             color: bool = False, suspiciousness: bool = False,
             line_numbers: bool = True) -> str:
        """
        Return a listing of `functions` (default: covered functions).
        If `color` is True, render as HTML, using suspiciousness colors.
        If `suspiciousness` is True, include suspiciousness values.
        If `line_numbers` is True (default), include line numbers.
        """

        if not functions:
            functions = self.covered_functions()

        out = ""
        seen = set()
        for function in functions:
            source_lines, starting_line_number = \
               inspect.getsourcelines(function)

            if (function.__name__, starting_line_number) in seen:
                continue
            seen.add((function.__name__, starting_line_number))

            if out:
                out += '\n'
                if color:
                    out += '
'

            line_number = starting_line_number
            for line in source_lines:
                if color:
                    line = html.escape(line)
                    if line.strip() == '':
                        line = ' '

                location = (function.__name__, line_number)
                location_suspiciousness = self.suspiciousness(location)
                if location_suspiciousness is not None:
                    tooltip = f"Line {line_number}: {self.tooltip(location)}"
                else:
                    tooltip = f"Line {line_number}: not executed"

                if suspiciousness:
                    line = self.percentage(location) + ' ' + line

                if line_numbers:
                    line = str(line_number).rjust(4) + ' ' + line

                line_color = self.color(location)

                if color and line_color:
                    line = f'''
{line.rstrip()}
'''
                elif color:
                    line = f'
{line}
'
                else:
                    line = line.rstrip()

                out += line + '\n'
                line_number += 1

        return out_____no_output_____

We introduce a few helper methods to visualize the code with colors in various forms._____no_output_____

class SpectrumDebugger(SpectrumDebugger):
    def _repr_html_(self) -> str:
        """When output in Jupyter, visualize as HTML"""
        return self.code(color=True)

    def __str__(self) -> str:
        """Show code as string"""
        return self.code(color=False, suspiciousness=True)

    def __repr__(self) -> str:
        """Show code as string"""
        return self.code(color=False, suspiciousness=True)_____no_output_____

So far, however, central methods like `suspiciousness()` or `color()` were abstract – that is, to be defined in subclasses. Our `DiscreteSpectrumDebugger` subclass provides concrete implementations for these, with `color()` returning one of the three colors depending on the line number:_____no_output_____

class DiscreteSpectrumDebugger(SpectrumDebugger):
    """Visualize differences between executions using three discrete colors"""

    def suspiciousness(self, event: Any) -> Optional[float]:
        """
        Return a suspiciousness value [0, 1.0]
        for the given event, or `None` if unknown.
        """
        passing = self.all_pass_events()
        failing = self.all_fail_events()

        if event in passing and event in failing:
            return 0.5
        elif event in failing:
            return 1.0
        elif event in passing:
            return 0.0
        else:
            return None

    def color(self, event: Any) -> Optional[str]:
        """
        Return a HTML color for the given event.
        """
        suspiciousness = self.suspiciousness(event)
        if suspiciousness is None:
            return None

        if suspiciousness > 0.8:
            return 'mistyrose'
        if suspiciousness >= 0.5:
            return 'lightyellow'

        return 'honeydew'

    def tooltip(self, event: Any) -> str:
        """Return a tooltip for the given event."""
        passing = self.all_pass_events()
        failing = self.all_fail_events()

        if event in passing and event in failing:
            return "in passing and failing runs"
        elif event in failing:
            return "only in failing runs"
        elif event in passing:
            return "only in passing runs"
        else:
            return "never"_____no_output_____

This is how the `only_pass_events()` and `only_fail_events()` sets look like when visualized with code. The "culprit" line is well highlighted:_____no_output_____

debugger = test_debugger_html(DiscreteSpectrumDebugger())_____no_output_____debugger_____no_output_____

We can clearly see that the failure is correlated with the presence of quotes in the input string (which is an important hint!). But does this also show us _immediately_ where the defect to be fixed is?_____no_output_____

quiz("Does the line `quote = not quote` actually contain the defect?",
    [
        "Yes, it should be fixed",
        "No, the defect is elsewhere"
    ], '164 * 2 % 326')_____no_output_____

Indeed, it is the _governing condition_ that is wrong – that is, the condition that caused Line 12 to be executed in the first place. In order to fix a program, we have to find a location that

1. _causes_ the failure (i.e., it can be changed to make the failure go away); and
2. is a _defect_ (i.e., contains an error).

In our example above, the highlighted code line is a _symptom_ for the error. To some extent, it is also a _cause_, since, say, commenting it out would also resolve the given failure, at the cost of causing other failures. However, the preceding condition also is a cause, as is the presence of quotes in the input.

Only one of these also is a _defect_, though, and that is the preceding condition. Hence, while correlations can provide important hints, they do not necessarily locate defects._____no_output_____For those of us who may not have color HTML output ready, simply printing the debugger lists suspiciousness values as percentages._____no_output_____

print(debugger)_____no_output_____

### Continuous Spectrum

The criterion that an event should _only_ occur in failing runs (and not in passing runs) can be too aggressive. In particular, if we have another run that executes the "culprit" lines, but does _not_ fail, our "only in fail" criterion will no longer be helpful._____no_output_____Here is an example. The input

```html
<b color="blue">text</b>
```

will trigger the "culprit" line

```python
quote = not quote
```

but actually produce an output where the tags are properly stripped:_____no_output_____

remove_html_markup('text')_____no_output_____

As a consequence, we no longer have lines that are being executed only in failing runs:_____no_output_____

debugger = test_debugger_html(DiscreteSpectrumDebugger())
with debugger.collect_pass():
    remove_html_markup('')_____no_output_____debugger.only_fail_events()_____no_output_____

In our spectrum output, the effect now is that the "culprit" line is as yellow as all others._____no_output_____

debugger_____no_output_____

We therefore introduce a different method for highlighting lines, based on their _relative_ occurrence with respect to all runs: If a line has been _mostly_ executed in failing runs, its color should shift towards red; if a line has been _mostly_ executed in passing runs, its color should shift towards green._____no_output_____This _continuous spectrum_ has been introduced by the seminal _Tarantula_ tool \cite{Jones2002}. In Tarantula, the color _hue_ for each line is defined as follows:_____no_output_____$$\textit{color hue}(\textit{line}) = \textit{low color(red)} + \frac{\%\textit{passed}(\textit{line})}{\%\textit{passed}(\textit{line}) + \%\textit{failed}(\textit{line})} \times \textit{color range}$$_____no_output_____Here, `%passed` and `%failed` denote the percentage at which a line has been executed in passing and failing runs, respectively. A hue of 0.0 stands for red, a hue of 1.0 stands for green, and a hue of 0.5 stands for equal fractions of red and green, yielding yellow._____no_output_____We can implement these measures right away as methods in a new `ContinuousSpectrumDebugger` class:_____no_output_____

class ContinuousSpectrumDebugger(DiscreteSpectrumDebugger):
    """Visualize differences between executions using a color spectrum"""

    def collectors_with_event(self, event: Any, category: str) -> Set[Collector]:
        """
        Return all collectors in a category
        that observed the given event.
        """
        all_runs = self.collectors[category]
        collectors_with_event = set(collector for collector in all_runs
                                    if event in collector.events())
        return collectors_with_event

    def collectors_without_event(self, event: Any, category: str) -> Set[Collector]:
        """
        Return all collectors in a category
        that did not observe the given event.
        """
        all_runs = self.collectors[category]
        collectors_without_event = set(collector for collector in all_runs
                              if event not in collector.events())
        return collectors_without_event

    def event_fraction(self, event: Any, category: str) -> float:
        if category not in self.collectors:
            return 0.0

        all_collectors = self.collectors[category]
        collectors_with_event = self.collectors_with_event(event, category)
        fraction = len(collectors_with_event) / len(all_collectors)
        # print(f"%{category}({event}) = {fraction}")
        return fraction

    def passed_fraction(self, event: Any) -> float:
        return self.event_fraction(event, self.PASS)

    def failed_fraction(self, event: Any) -> float:
        return self.event_fraction(event, self.FAIL)

    def hue(self, event: Any) -> Optional[float]:
        """Return a color hue from 0.0 (red) to 1.0 (green)."""
        passed = self.passed_fraction(event)
        failed = self.failed_fraction(event)
        if passed + failed > 0:
            return passed / (passed + failed)
        else:
            return None_____no_output_____

Having a continuous hue also implies a continuous suspiciousness and associated tooltips:_____no_output_____

class ContinuousSpectrumDebugger(ContinuousSpectrumDebugger):
    def suspiciousness(self, event: Any) -> Optional[float]:
        hue = self.hue(event)
        if hue is None:
            return None
        return 1 - hue

    def tooltip(self, event: Any) -> str:
        return self.percentage(event)_____no_output_____

The hue for lines executed only in failing runs is (deep) red, as expected:_____no_output_____

debugger = test_debugger_html(ContinuousSpectrumDebugger())_____no_output_____for location in debugger.only_fail_events():
    print(location, debugger.hue(location))_____no_output_____

Likewise, the hue for lines executed in passing runs is (deep) green:_____no_output_____

for location in debugger.only_pass_events():
    print(location, debugger.hue(location))_____no_output_____

The Tarantula tool not only sets the hue for a line, but also uses _brightness_ as measure for support – that is, how often was the line executed at all. The brighter a line, the stronger the correlation with a passing or failing outcome._____no_output_____The brightness is defined as follows:_____no_output_____$$\textit{brightness}(line) = \max(\%\textit{passed}(\textit{line}), \%\textit{failed}(\textit{line}))$$_____no_output_____and it is easily implemented, too:_____no_output_____

class ContinuousSpectrumDebugger(ContinuousSpectrumDebugger):
    def brightness(self, event: Any) -> float:
        return max(self.passed_fraction(event), self.failed_fraction(event))_____no_output_____

Our single "only in fail" line has a brightness of 1.0 (the maximum)._____no_output_____

debugger = test_debugger_html(ContinuousSpectrumDebugger())
for location in debugger.only_fail_events():
    print(location, debugger.brightness(location))_____no_output_____

With this, we can now define a color for each line. To this end, we override the (previously discrete) `color()` method such that it returns a color specification giving hue and brightness. We use the HTML format `hsl(hue, saturation, lightness)` where the hue is given as a value between 0 and 360 (0 is red, 120 is green) and saturation and lightness are provided as percentages._____no_output_____

class ContinuousSpectrumDebugger(ContinuousSpectrumDebugger):
    def color(self, event: Any) -> Optional[str]:
        hue = self.hue(event)
        if hue is None:
            return None
        saturation = self.brightness(event)

        # HSL color values are specified with:
        # hsl(hue, saturation, lightness).
        return f"hsl({hue * 120}, {saturation * 100}%, 80%)"_____no_output_____debugger = test_debugger_html(ContinuousSpectrumDebugger())_____no_output_____

Lines executed only in failing runs are still shown in red:_____no_output_____

for location in debugger.only_fail_events():
    print(location, debugger.color(location))_____no_output_____

... whereas lines executed only in passing runs are still shown in green:_____no_output_____

for location in debugger.only_pass_events():
    print(location, debugger.color(location))_____no_output_____debugger_____no_output_____

What happens with our `quote = not quote` "culprit" line if it is executed in passing runs, too?_____no_output_____

with debugger.collect_pass():
    out = remove_html_markup('')_____no_output_____quiz('In which color will the `quote = not quote` "culprit" line '
     'be shown after executing the above code?',
    [
        'Green',
        'Yellow',
        'Orange',
        'Red'
    ], '999 // 333')_____no_output_____

We see that it still is shown with an orange-red tint._____no_output_____

debugger_____no_output_____

Here's another example, coming right from the Tarantula paper. The `middle()` function takes three numbers `x`, `y`, and `z`, and returns the one that is neither the minimum nor the maximum of the three:_____no_output_____

def middle(x, y, z):  # type: ignore
    if y < z:
        if x < y:
            return y
        elif x < z:
            return y
    else:
        if x > y:
            return y
        elif x > z:
            return x
    return z_____no_output_____middle(1, 2, 3)_____no_output_____

Unfortunately, `middle()` can fail:_____no_output_____

middle(2, 1, 3)_____no_output_____

Let is see whether we can find the bug with a few additional test cases:_____no_output_____

# ignore
T3 = TypeVar('T3', bound='DifferenceDebugger')_____no_output_____def test_debugger_middle(debugger: T3) -> T3:
    with debugger.collect_pass():
        middle(3, 3, 5)
    with debugger.collect_pass():
        middle(1, 2, 3)
    with debugger.collect_pass():
        middle(3, 2, 1)
    with debugger.collect_pass():
        middle(5, 5, 5)
    with debugger.collect_pass():
        middle(5, 3, 4)
    with debugger.collect_fail():
        middle(2, 1, 3)
    return debugger_____no_output_____

Note that in order to collect data from multiple function invocations, you need to have a separate `with` clause for every invocation. The following will _not_ work correctly:

```python
    with debugger.collect_pass():
        middle(3, 3, 5)
        middle(1, 2, 3)
        ...
```_____no_output_____

debugger = test_debugger_middle(ContinuousSpectrumDebugger())_____no_output_____debugger.event_table(args=True)_____no_output_____

Here comes the visualization. We see that the `return y` line is the culprit here – and actually also the one to be fixed._____no_output_____

debugger_____no_output_____quiz("Which of the above lines should be fixed?",
    [
        'Line 3: `elif x < y`',
        'Line 5: `elif x < z`',
        'Line 6: `return y`',
        'Line 9: `return y`',
    ], r'len(" middle  ".strip()[:3])')_____no_output_____

Indeed, in the `middle()` example, the "reddest" line is also the one to be fixed.  Here is the fixed version:_____no_output_____

def middle_fixed(x, y, z):  # type: ignore
    if y < z:
        if x < y:
            return y
        elif x < z:
            return x
    else:
        if x > y:
            return y
        elif x > z:
            return x
    return z_____no_output_____middle_fixed(2, 1, 3)_____no_output_____

## Ranking Lines by Suspiciousness

In a large program, there can be several locations (and events) that could be flagged as suspicious. It suffices that some large code block of say, 1,000 lines, is mostly executed in failing runs, and then all of this code block will be visualized in some shade of red.

To further highlight the "most suspicious" events, one idea is to use a _ranking_ – that is, coming up with a list of events where those events most correlated with failures would be shown at the top. The programmer would then examine these events one by one and proceed down the list. We will show how this works for two "correlation" metrics – first the _Tarantula_ metric, as introduced above, and then the _Ochiai_ metric, which has shown to be one of the best "ranking" metrics._____no_output_____We introduce a base class `RankingDebugger` with an abstract method `suspiciousness()` to be overloaded in subclasses. The method `rank()` returns a list of all events observed, sorted by suspiciousness, highest first._____no_output_____

class RankingDebugger(DiscreteSpectrumDebugger):
    """Rank events by their suspiciousness"""

    def rank(self) -> List[Any]:
        """Return a list of events, sorted by suspiciousness, highest first."""

        def susp(event: Any) -> float:
            suspiciousness = self.suspiciousness(event)
            assert suspiciousness is not None
            return suspiciousness

        events = list(self.all_events())
        events.sort(key=susp, reverse=True)
        return events

    def __repr__(self) -> str:
        return repr(self.rank())_____no_output_____

### The Tarantula Metric

We can use the Tarantula metric to sort lines according to their suspiciousness. The "redder" a line (a hue of 0.0), the more suspicious it is. We can simply define_____no_output_____$$
\textit{suspiciousness}_\textit{tarantula}(\textit{event}) = 1 - \textit{color hue}(\textit{event})
$$_____no_output_____where $\textit{color hue}$ is as defined above._____no_output_____This is exactly the `suspiciousness()` function as already implemented in our `ContinuousSpectrumDebugger`._____no_output_____We introduce the `TarantulaDebugger` class, inheriting visualization capabilities from the `ContinuousSpectrumDebugger` class as well as the suspiciousness features from the `RankingDebugger` class._____no_output_____

class TarantulaDebugger(ContinuousSpectrumDebugger, RankingDebugger):
    """Spectrum-based Debugger using the Tarantula metric for suspiciousness"""
    pass_____no_output_____

Let us list `remove_html_markup()` with highlighted lines again:_____no_output_____

tarantula_html = test_debugger_html(TarantulaDebugger())_____no_output_____tarantula_html_____no_output_____

Here's our ranking of lines, from most suspicious to least suspicious:_____no_output_____

tarantula_html.rank()_____no_output_____tarantula_html.suspiciousness(tarantula_html.rank()[0])_____no_output_____

We see that the first line in the list is indeed the most suspicious; the two "green" lines come at the very end._____no_output_____For the `middle()` function, we also obtain a ranking from "reddest" to "greenest"._____no_output_____

tarantula_middle = test_debugger_middle(TarantulaDebugger())_____no_output_____tarantula_middle_____no_output_____tarantula_middle.rank()_____no_output_____tarantula_middle.suspiciousness(tarantula_middle.rank()[0])_____no_output_____

### The Ochiai Metric

The _Ochiai_ Metric \cite{Ochiai1957} first introduced in the biology domain \cite{daSilvaMeyer2004} and later applied for fault localization by Abreu et al. \cite{Abreu2009}, is defined as follows:_____no_output_____$$
\textit{suspiciousness}_\textit{ochiai} = \frac
{\textit{failed}(\textit{event})}
{\sqrt{
\bigl(\textit{failed}(\textit{event}) + \textit{not-in-failed}(\textit{event})\bigr)
\times
\bigl(\textit{failed}(\textit{event}) + \textit{passed}(\textit{event})\bigr)
}}
$$_____no_output_____where

* $\textit{failed}(\textit{event})$ is the number of times the event occurred in _failing_ runs
* $\textit{not-in-failed}(\textit{event})$ is the number of times the event did _not_ occur in failing runs
* $\textit{passed}(\textit{event})$ is the number of times the event occurred in _passing_ runs.

We can easily implement this formula:_____no_output_____

import math_____no_output_____class OchiaiDebugger(ContinuousSpectrumDebugger, RankingDebugger):
    """Spectrum-based Debugger using the Ochiai metric for suspiciousness"""

    def suspiciousness(self, event: Any) -> Optional[float]:
        failed = len(self.collectors_with_event(event, self.FAIL))
        not_in_failed = len(self.collectors_without_event(event, self.FAIL))
        passed = len(self.collectors_with_event(event, self.PASS))

        try:
            return failed / math.sqrt((failed + not_in_failed) * (failed + passed))
        except ZeroDivisionError:
            return None

    def hue(self, event: Any) -> Optional[float]:
        suspiciousness = self.suspiciousness(event)
        if suspiciousness is None:
            return None
        return 1 - suspiciousness_____no_output_____

Applied on the `remove_html_markup()` function, the individual suspiciousness scores differ from Tarantula. However, we obtain a very similar visualization, and the same ranking._____no_output_____

ochiai_html = test_debugger_html(OchiaiDebugger())_____no_output_____ochiai_html_____no_output_____ochiai_html.rank()_____no_output_____ochiai_html.suspiciousness(ochiai_html.rank()[0])_____no_output_____

The same observations also apply for the `middle()` function._____no_output_____

ochiai_middle = test_debugger_middle(OchiaiDebugger())_____no_output_____ochiai_middle_____no_output_____ochiai_middle.rank()_____no_output_____ochiai_middle.suspiciousness(ochiai_middle.rank()[0])_____no_output_____

### How Useful is Ranking?

So, which metric is better? The standard method to evaluate such rankings is to determine a _ground truth_ – that is, the set of locations that eventually are fixed – and to check at which point in the ranking any such location occurs – the earlier, the better. In our `remove_html_markup()` and `middle()` examples, both the Tarantula and the Ochiai metric perform flawlessly, as the "culprit" line is always ranked at the top. However, this need not always be the case; the exact performance depends on the nature of the code and the observed runs. (Also, the question of whether there always is exactly one possible location where the program can be fixed is open for discussion.)_____no_output_____You will be surprised that over time, _several dozen_ metrics have been proposed \cite{Wong2016}, each performing somewhat better or somewhat worse depending on which benchmark they were applied on. The two metrics discussed above each have their merits – the Tarantula metric was among the first such metrics, and the Ochiai metric is generally shown to be among the most effective ones \cite{Abreu2009}._____no_output_____While rankings can be easily _evaluated_, it is not necessarily clear whether and how much they serve programmers. As stated above, the assumption of rankings is that developers examine one potentially defective statement after another until they find the actually defective one. However, in a series of human studies with developers, Parnin and Orso \cite{Parnin2011} found that this assumption may not hold:

> It is unclear whether developers can actually determine the faulty nature of a statement by simply looking at it, without any additional information (e.g., the state of the program when the statement was executed or the statements that were executed before or after that one).

In their study, they found that rankings could help completing a task faster, but this effect was limited to experienced developers and simpler code. Artificially changing the rank of faulty statements had little to no effect, implying that developers would not strictly follow the ranked list of statements, but rather search through the code to understand it. At this point, a _visualization_ as in the Tarantula tool can be helpful to programmers as it _guides_ the search, but a _ranking_ that _defines_ where to search may be less useful._____no_output_____Having said that, ranking has its merits – notably as it comes to informing _automated_ debugging techniques. In the [chapter on program repair](Repairer.ipynb), we will see how ranked lists of potentially faulty statements tell automated repair techniques where to try to repair the program first. And once such a repair is successful, we have a very strong indication on where and how the program could be fixed!_____no_output_____## Using Large Test Suites_____no_output_____In fault localization, the larger and the more thorough the test suite, the higher the precision. Let us try out what happens if we extend the `middle()` test suite with additional test cases._____no_output_____The function `middle_testcase()` returns a random input for `middle()`:_____no_output_____

import random_____no_output_____def middle_testcase() -> Tuple[int, int, int]:
    x = random.randrange(10)
    y = random.randrange(10)
    z = random.randrange(10)
    return x, y, z_____no_output_____[middle_testcase() for i in range(5)]_____no_output_____

The function `middle_test()` simply checks if `middle()` operates correctly – by placing `x`, `y`, and `z` in a list, sorting it, and checking the middle argument. If `middle()` fails, `middle_test()` raises an exception._____no_output_____

def middle_test(x: int, y: int, z: int) -> None:
    m = middle(x, y, z)
    assert m == sorted([x, y, z])[1]_____no_output_____middle_test(4, 5, 6)_____no_output_____from ExpectError import ExpectError_____no_output_____with ExpectError():
    middle_test(2, 1, 3)_____no_output_____

The function `middle_passing_testcase()` searches and returns a triple `x`, `y`, `z` that causes `middle_test()` to pass._____no_output_____

def middle_passing_testcase() -> Tuple[int, int, int]:
    while True:
        try:
            x, y, z = middle_testcase()
            middle_test(x, y, z)
            return x, y, z
        except AssertionError:
            pass_____no_output_____(x, y, z) = middle_passing_testcase()
m = middle(x, y, z)
print(f"middle({x}, {y}, {z}) = {m}")_____no_output_____

The function `middle_failing_testcase()` does the same; but its triple `x`, `y`, `z` causes `middle_test()` to fail._____no_output_____

def middle_failing_testcase() -> Tuple[int, int, int]:
    while True:
        try:
            x, y, z = middle_testcase()
            middle_test(x, y, z)
        except AssertionError:
            return x, y, z_____no_output_____(x, y, z) = middle_failing_testcase()
m = middle(x, y, z)
print(f"middle({x}, {y}, {z}) = {m}")_____no_output_____

With these, we can define two sets of test cases, each with 100 inputs._____no_output_____

MIDDLE_TESTS = 100_____no_output_____MIDDLE_PASSING_TESTCASES = [middle_passing_testcase()
                            for i in range(MIDDLE_TESTS)]_____no_output_____MIDDLE_FAILING_TESTCASES = [middle_failing_testcase()
                            for i in range(MIDDLE_TESTS)]_____no_output_____

Let us run the `OchiaiDebugger` with these two test sets._____no_output_____

ochiai_middle = OchiaiDebugger()

for x, y, z in MIDDLE_PASSING_TESTCASES:
    with ochiai_middle.collect_pass():
        middle(x, y, z)

for x, y, z in MIDDLE_FAILING_TESTCASES:
    with ochiai_middle.collect_fail():
        middle(x, y, z)_____no_output_____ochiai_middle_____no_output_____

We see that the "culprit" line is still the most likely to be fixed, but the two conditions leading to the error (`x < y` and `x < z`) are also listed as potentially faulty. That is because the error might also be fixed be changing these conditions – although this would result in a more complex fix._____no_output_____## Other Events besides Coverage

We close this chapter with two directions for further thought. If you wondered why in the above code, we were mostly talking about `events` rather than lines covered, that is because our framework allows for tracking arbitrary events, not just coverage. In fact, any data item a collector can extract from the execution can be used for correlation analysis. (It may not be so easily visualized, though.)_____no_output_____Here's an example. We define a `ValueCollector` class that collects pairs of (local) variables and their values during execution. Its `events()` method then returns the set of all these pairs._____no_output_____

class ValueCollector(Collector):
    """"A class to collect local variables and their values."""

    def __init__(self) -> None:
        """Constructor."""
        super().__init__()
        self.vars: Set[str] = set()

    def collect(self, frame: FrameType, event: str, arg: Any) -> None:
        local_vars = frame.f_locals
        for var in local_vars:
            value = local_vars[var]
            self.vars.add(f"{var} = {repr(value)}")

    def events(self) -> Set[str]:
        """A set of (variable, value) pairs observed"""
        return self.vars_____no_output_____

If we apply this collector on our set of HTML test cases, these are all the events that we obtain – essentially all variables and all values ever seen:_____no_output_____

debugger = test_debugger_html(ContinuousSpectrumDebugger(ValueCollector))
for event in debugger.all_events():
    print(event)_____no_output_____

However, some of these events only occur in the failing run:_____no_output_____

for event in debugger.only_fail_events():
    print(event)_____no_output_____

Some of these differences are spurious – the string `"abc"` (with quotes) only occurs in the failing run – but others, such as `quote` being True and `c` containing a single quote are actually relevant for explaining when the failure comes to be._____no_output_____We can even visualize the suspiciousness of the individual events, setting the (so far undiscussed) `color` flag for producing an event table:_____no_output_____

debugger.event_table(color=True, args=True)_____no_output_____

There are many ways one can continue from here.

* Rather than checking for concrete values, one could check for more _abstract properties_, for instance – what is the sign of the value? What is the length of the string?
* One could check for specifics of the _control flow_ – is the loop taken? How many times?
* One could check for specifics of the _information flow_ – which values flow from one variable to another?

There are lots of properties that all could be related to failures – and if we happen to check for the right one, we may obtain a much crisper definition of what causes the failure. We will come up with more ideas on properties to check as it comes to [mining specifications](SpecificationMining,ipynb)._____no_output_____## Training Classifiers

The metrics we have discussed so far are pretty _generic_ – that is, they are fixed no matter how the actual event space is structured. The field of _machine learning_ has come up with techniques that learn _classifiers_ from a given set of data – classifiers that are trained from labeled data and then can predict labels for new data sets. In our case, the labels are test outcomes (PASS and FAIL), whereas the data would be features of the events observed._____no_output_____A classifier by itself is not immediately useful for debugging (although it could predict whether future inputs will fail or not). Some classifiers, however, have great _diagnostic_ quality; that is, they can _explain_ how their classification comes to be. [Decision trees](https://scikit-learn.org/stable/modules/tree.html) fall into this very category._____no_output_____A decision tree contains a number of _nodes_, each one associated with a predicate. Depending on whether the predicate is true or false, we follow the given "true" or "false" branch to end up in the next node, which again contains a predicate. Eventually, we end up in the outcome predicted by the tree. The neat thing is that the node predicates actually give important hints on the circumstances that are _most relevant_ for deciding the outcome._____no_output_____Let us illustrate this with an example. We build a class `ClassifyingDebugger` that trains a decision tree from the events collected. To this end, we need to set up our input data such that it can be fed into a classifier._____no_output_____We start with identifying our _samples_ (runs) and the respective _labels_ (outcomes). All values have to be encoded into numerical values._____no_output_____

class ClassifyingDebugger(DifferenceDebugger):
    """A debugger implementing a decision tree for events"""

    PASS_VALUE = +1.0
    FAIL_VALUE = -1.0

    def samples(self) -> Dict[str, float]:
        samples = {}
        for collector in self.pass_collectors():
            samples[collector.id()] = self.PASS_VALUE
        for collector in debugger.fail_collectors():
            samples[collector.id()] = self.FAIL_VALUE
        return samples_____no_output_____debugger = test_debugger_html(ClassifyingDebugger())
debugger.samples()_____no_output_____

Next, we identify the _features_, which in our case is the set of lines executed in each sample:_____no_output_____

class ClassifyingDebugger(ClassifyingDebugger):
    def features(self) -> Dict[str, Any]:
        features = {}
        for collector in debugger.pass_collectors():
            features[collector.id()] = collector.events()
        for collector in debugger.fail_collectors():
            features[collector.id()] = collector.events()
        return features_____no_output_____debugger = test_debugger_html(ClassifyingDebugger())
debugger.features()_____no_output_____

All our features have names, which must be strings._____no_output_____

class ClassifyingDebugger(ClassifyingDebugger):
    def feature_names(self) -> List[str]:
        return [repr(feature) for feature in self.all_events()]_____no_output_____debugger = test_debugger_html(ClassifyingDebugger())
debugger.feature_names()_____no_output_____

Next, we define the _shape_ for an individual sample, which is a value of +1 or -1 for each feature seen (i.e., +1 if the line was covered, -1 if not)._____no_output_____

class ClassifyingDebugger(ClassifyingDebugger):
    def shape(self, sample: str) -> List[float]:
        x = []
        features = self.features()
        for f in self.all_events():
            if f in features[sample]:
                x += [+1.0]
            else:
                x += [-1.0]
        return x_____no_output_____debugger = test_debugger_html(ClassifyingDebugger())
debugger.shape("remove_html_markup(s='abc')")_____no_output_____

Our input X for the classifier now is a list of such shapes, one for each sample._____no_output_____

class ClassifyingDebugger(ClassifyingDebugger):
    def X(self) -> List[List[float]]:
        X = []
        samples = self.samples()
        for key in samples:
            X += [self.shape(key)]
        return X_____no_output_____debugger = test_debugger_html(ClassifyingDebugger())
debugger.X()_____no_output_____

Our input Y for the classifier, in contrast, is the list of labels, again indexed by sample._____no_output_____

class ClassifyingDebugger(ClassifyingDebugger):
    def Y(self) -> List[float]:
        Y = []
        samples = self.samples()
        for key in samples:
            Y += [samples[key]]
        return Y_____no_output_____debugger = test_debugger_html(ClassifyingDebugger())
debugger.Y()_____no_output_____

We now have all our data ready to be fit into a tree classifier. The method `classifier()` creates and returns the (tree) classifier for the observed runs._____no_output_____

from sklearn.tree import DecisionTreeClassifier, export_text, export_graphviz_____no_output_____class ClassifyingDebugger(ClassifyingDebugger):
    def classifier(self) -> DecisionTreeClassifier:
        classifier = DecisionTreeClassifier()
        classifier = classifier.fit(self.X(), self.Y())
        return classifier_____no_output_____

We define a special method to show classifiers:_____no_output_____

import graphviz_____no_output_____class ClassifyingDebugger(ClassifyingDebugger):
    def show_classifier(self, classifier: DecisionTreeClassifier) -> Any:
        dot_data = export_graphviz(classifier, out_file=None,
                                   filled=False, rounded=True,
                                   feature_names=self.feature_names(),
                                   class_names=["FAIL", "PASS"],
                                   label='none',
                                   node_ids=False,
                                   impurity=False,
                                   proportion=True,
                                   special_characters=True)

        return graphviz.Source(dot_data)_____no_output_____

This is the tree we get for our `remove_html_markup()` tests. The top predicate is whether the "culprit" line was executed (-1 means no, +1 means yes). If not (-1), the outcome is PASS. Otherwise, the outcome is TRUE._____no_output_____

debugger = test_debugger_html(ClassifyingDebugger())
classifier = debugger.classifier()
debugger.show_classifier(classifier)_____no_output_____

We can even use our classifier to predict the outcome of additional runs. If, for instance, we execute all lines except for, say, Line 7, 9, and 11, our tree classifier would predict failure – because the "culprit" line 12 is executed._____no_output_____

classifier.predict([[1, 1, 1, 1, 1, 1, -1, 1, -1, 1, -1, 1, 1, 1]])_____no_output_____

Again, there are many ways to continue from here. Which events should we train the classifier from? How do classifiers compare in their performance and diagnostic quality? There are lots of possibilities left to explore, and we only begin to realize the potential for automated debugging._____no_output_____## Synopsis

This chapter introduces classes and techniques for _statistical debugging_ – that is, correlating specific events, such as lines covered, with passing and failing outcomes.

To make use of the code in this chapter, use one of the provided `StatisticalDebugger` subclasses such as `TarantulaDebugger` or `OchiaiDebugger`.

Both are instantiated with a `Collector` denoting the type of events you want to correlate outcomes with. The default `CoverageCollector`, collecting line coverage._____no_output_____### Collecting Events from Calls

To collect events from calls that are labeled manually, use_____no_output_____

debugger = TarantulaDebugger()
with debugger.collect_pass():
    remove_html_markup("abc")
with debugger.collect_pass():
    remove_html_markup('abc')
with debugger.collect_fail():
    remove_html_markup('"abc"')_____no_output_____

Within each `with` block, the _first function call_ is collected and tracked for coverage. (Note that _only_ the first call is tracked.)_____no_output_____### Collecting Events from Tests

To collect events from _tests_ that use exceptions to indicate failure, use the simpler `with` form:_____no_output_____

debugger = TarantulaDebugger()
with debugger:
    remove_html_markup("abc")
with debugger:
    remove_html_markup('abc')
with debugger:
    remove_html_markup('"abc"')
    assert False  # raise an exception_____no_output_____

`with` blocks that raise an exception will be classified as failing, blocks that do not will be classified as passing. Note that exceptions raised are "swallowed" by the debugger._____no_output_____### Visualizing Events as a Table

After collecting events, you can print out the observed events – in this case, line numbers – in a table, showing in which runs they occurred (`X`), and with colors highlighting the suspiciousness of the event. A "red" event means that the event predominantly occurs in failing runs._____no_output_____

debugger.event_table(args=True, color=True)_____no_output_____

### Visualizing Suspicious Code

If you collected coverage with `CoverageCollector`, you can also visualize the code with similar colors, highlighting suspicious lines:_____no_output_____

debugger_____no_output_____

### Ranking Events

The method `rank()` returns a ranked list of events, starting with the most suspicious. This is useful for automated techniques that need potential defect locations._____no_output_____

debugger.rank()_____no_output_____

### Classes and Methods

Here are all classes defined in this chapter:_____no_output_____

# ignore
from ClassDiagram import display_class_hierarchy_____no_output_____# ignore
display_class_hierarchy([TarantulaDebugger, OchiaiDebugger],
                        abstract_classes=[
                            StatisticalDebugger,
                            DifferenceDebugger,
                            RankingDebugger
                        ],
                        public_methods=[
                            StatisticalDebugger.__init__,
                            StatisticalDebugger.all_events,
                            StatisticalDebugger.event_table,
                            StatisticalDebugger.function,
                            StatisticalDebugger.coverage,
                            StatisticalDebugger.covered_functions,
                            DifferenceDebugger.__enter__,
                            DifferenceDebugger.__exit__,
                            DifferenceDebugger.all_pass_events,
                            DifferenceDebugger.all_fail_events,
                            DifferenceDebugger.collect_pass,
                            DifferenceDebugger.collect_fail,
                            DifferenceDebugger.only_pass_events,
                            DifferenceDebugger.only_fail_events,
                            SpectrumDebugger.code,
                            SpectrumDebugger.__repr__,
                            SpectrumDebugger.__str__,
                            SpectrumDebugger._repr_html_,
                            ContinuousSpectrumDebugger.code,
                            ContinuousSpectrumDebugger.__repr__,
                            RankingDebugger.rank
                        ],
                        project='debuggingbook')_____no_output_____# ignore
display_class_hierarchy([CoverageCollector, ValueCollector],
                        public_methods=[
                            Tracer.__init__,
                            Tracer.__enter__,
                            Tracer.__exit__,
                            Tracer.changed_vars,  # type: ignore
                            Collector.__init__,
                            Collector.__repr__,
                            Collector.function,
                            Collector.args,
                            Collector.argstring,
                            Collector.exception,
                            Collector.id,
                            Collector.collect,
                            CoverageCollector.coverage,
                            CoverageCollector.covered_functions,
                            CoverageCollector.events,
                            ValueCollector.__init__,
                            ValueCollector.events
                        ],
                        project='debuggingbook')_____no_output_____

## Lessons Learned

* _Correlations_ between execution events and outcomes (pass/fail) can make important hints for debugging
* Events occurring only (or mostly) during failing runs can be _highlighted_ and _ranked_ to guide the search
* Important hints include whether the _execution of specific code locations_ correlates with failure_____no_output_____## Next Steps

Chapters that build on this one include

* [how to determine invariants that correlate with failures](DynamicInvariants.ipynb)
* [how to automatically repair programs](Repairer.ipynb)_____no_output_____## Background

The seminal works on statistical debugging are two papers:

* "Visualization of Test Information to Assist Fault Localization" \cite{Jones2002} by James Jones, Mary Jean Harrold, and John Stasko introducing Tarantula and its visualization. The paper won an ACM SIGSOFT 10-year impact award.
* "Bug Isolation via Remote Program Sampling" \cite{Liblit2003} by Ben Liblit, Alex Aiken, Alice X. Zheng, and Michael I. Jordan, introducing the term "Statistical debugging". Liblit won the ACM Doctoral Dissertation Award for this work.

The Ochiai metric for fault localization was introduced by \cite{Abreu2009}. The overview by Wong et al. \cite{Wong2016} gives a comprehensive overview on the field of statistical fault localization.

The study by Parnin and Orso \cite{Parnin2011} is a must to understand the limitations of the technique._____no_output_____## Exercises_____no_output_____### Exercise 1: A Postcondition for Middle

What would be a postcondition for `middle()`? How can you check it?_____no_output_____**Solution.** A simple postcondition for `middle()` would be

```python
assert m == sorted([x, y, z])[1]
```

where `m` is the value returned by `middle()`. `sorted()` sorts the given list, and the index `[1]` returns, well, the middle element. (This might also be a much shorter, but possibly slightly more expensive implementation for `middle()`)_____no_output_____Since `middle()` has several `return` statements, the easiest way to check the result is to create a wrapper around `middle()`:_____no_output_____

def middle_checked(x, y, z):  # type: ignore
    m = middle(x, y, z)
    assert m == sorted([x, y, z])[1]
    return m_____no_output_____

`middle_checked()` catches the error:_____no_output_____

from ExpectError import ExpectError_____no_output_____with ExpectError():
    m = middle_checked(2, 1, 3)_____no_output_____

### Exercise 2: Statistical Dependencies

Using the dependencies from [the chapter on slicing](Slicer.ipynb), can you determine which specific data or control dependencies correlate with failure?_____no_output_____### Exercise 3: Correlating with Conditions_____no_output_____In HTML, it is permissible that tag attribute values can also have single quotes, as in

```html
<b class='extrabold'>abc</b>
```

Such attributes are actually treated correctly by our `remove_html_markup()` code._____no_output_____#### Part 1: Experiment

What happens if test inputs with single quote attributes become part of our test set? How does statistical fault localization change?_____no_output_____**Solution.** The effect is that the line `quote = not quote` will be executed both in passing and failing runs, spoiling the correlation._____no_output_____#### Part 2: Collecting Conditions_____no_output_____Instead of aiming for _lines_ that correlate with failures, we can look at individual _branch conditions_ such as `c == "'"`, `c == '"'`, or `tag` that correlate with failures. In the above case, the condition `c == "'"` would correlate, whereas `c == '"'` would not.

Reusing the code instrumentation from [the chapter on slicing](Slicer.ipynb), collect the individual values of Boolean conditions in tests during execution in a `ConditionCollector` class. Show the event table._____no_output_____


# Notebook from masich/big-data
Path: bow.ipynb

import re
from collections import Counter
from functools import reduce
from math import log10
from operator import or_
from typing import List, Dict, Iterable

import nltk
import pandas as pd
from IPython.display import display
from nltk.tokenize import word_tokenize_____no_output_____

Download additional nltk components._____no_output_____

nltk.download('punkt')
nltk.download('stopwords')[nltk_data] Downloading package punkt to /Users/max-
[nltk_data]     omelchenko/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to /Users/max-
[nltk_data]     omelchenko/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!

Construct stopwords list._____no_output_____

language = 'english'
stopwords = nltk.corpus.stopwords.words(language)
stopwords.append('the')
stopwords = set(stopwords)_____no_output_____

Preprocess articles._____no_output_____

file_paths = (
    'data/articles/20th.txt',        # Ken kesey one flew over the cuckoo's nest
    'data/articles/19th.txt',        # Oliver Twist Charles Dickens
    'data/articles/article1.txt',    # Types of Ecology    #article
    'data/articles/scientific.txt',  # Euler characteristic
    'data/articles/article2.txt',    # A Comprehensive Guide To Jenkins Pipeline  #otherArticle
)

articles = []
word = re.compile(r'[^A-Za-z]')

for file_path in file_paths:
    with open(file_path, 'r') as file:
        article = file.read()

        # Left only words in lower case.
        article = word.sub(' ', article).lower()
        articles.append(article)_____no_output_____

Split articles texts into tokens (words)._____no_output_____

tokenized_articles = list(map(lambda text: word_tokenize(text, language=language, preserve_line=True), articles))_____no_output_____

Remove stopwords from tokenized texts._____no_output_____

def remove_stop_words(_tokenized_text: List[str]) -> List[str]:
    _text = _tokenized_text.copy()
    for token in _tokenized_text:
        if token in stopwords:
            _text.remove(token)

    return _text_____no_output_____tokenized_articles = list(map(remove_stop_words, tokenized_articles))_____no_output_____

Create common word set of all words in all articles._____no_output_____

words = reduce(or_, map(set, tokenized_articles))_____no_output_____

Create a mapping *word* : *how many times is used* for all articles._____no_output_____

words_count_zero = dict.fromkeys(words, 0)
words_count_articles = [words_count_zero.copy() for i in range(len(tokenized_articles))]
for _words_count_article, _tokenized_article in zip(words_count_articles, tokenized_articles):
    _words_count_article.update(Counter(_tokenized_article))_____no_output_____pd.DataFrame(words_count_articles)_____no_output_____

Calculate term frequency (TF)._____no_output_____

def compute_tf(_words_count_article: Dict[str, int], _bow: List[str]) -> Dict[str, int]:
    _tf = {}
    bow_len = len(_bow)
    for _word, _count in _words_count_article.items():
        _tf[_word] = _count / bow_len
    return _tf_____no_output_____# for more precise results use TF-IDF Model

tf_bows = list(map(lambda zipped: compute_tf(*zipped), zip(words_count_articles, tokenized_articles)))
_____no_output_____

Calculate an Inverse Document Frequency (IDF).

IDF is a measure of how important a term is. We need the IDF value because computing just the TF alone is not sufficient to understand the importance of words._____no_output_____

def compute_idf(_all_words: Iterable[str], _words_counts: List[Dict[str, int]]) -> Dict[str, int]:
    _n = len(_words_counts)

    _idfs = dict.fromkeys(_all_words, 0)
    for _words_count in _words_counts:
        for _word, _count in _words_count.items():
            if _count > 0:
                _idfs[_word] += 1

    for _word, _val in _idfs.items():
        if _val != 0:
            _idfs[_word] = log10(_n / _val)

    return _idfs_____no_output_____idfs = compute_idf(words, words_count_articles)_____no_output_____

Calculate TF-IDF score for each word.

Words with a higher score are more important, and those with a lower score are less important._____no_output_____

def compute_tfidf(_tf_bow: Dict[str, int], _idfs: Dict[str, int]) -> Dict[str, int]:
    _tf_idf = {}
    for _word, _val in _tf_bow.items():
        _tf_idf[_word] = _val * _idfs[_word]
    return _tf_idf_____no_output_____tf_idf_bows = list(map(lambda _tf_bow: compute_tfidf(_tf_bow, idfs), tf_bows))_____no_output_____

Display the most valuable words for every article._____no_output_____

article_names = ('19th', '20th', 'article1', 'scientific', 'article2')
df = pd.DataFrame(tf_idf_bows, index=article_names)

for article_name in article_names:
    article_df = df.sort_values(by=article_name, axis=1, ascending=False)
    display(article_name, article_df)_____no_output_____


# Notebook from ymubarka/yabox
Path: notebooks/yabox-de-animations.ipynb

> Yabox: Yet another black-box optimization library for Python - https://github.com/pablormier/yabox

This notebook shows how to generate 3d animations of Differential Evolution exploring two dimensional problems

Author: [Pablo Rodríguez-Mier](https://pablormier.github.io/)_____no_output_____

%matplotlib inline

# Load local version of yabox
import sys
sys.path.insert(0, '../')

from yabox import DE, PDE
import numpy as np

# Imports required for 3d animations
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import animation, rcParams
from IPython.display import HTML_____no_output_____

## Main functions for plotting and generating the animations_____no_output_____

# There is a bug in matplotlib that prevents the use of ffmpeg and avconv!
# https://github.com/matplotlib/matplotlib/pull/8743

avail_writers = matplotlib.animation.writers.list()

# Functions to generate 3d animations

def display_animation(anim):
    plt.close(anim._fig)
    return HTML(anim.to_html5_video())

def generate_video(problem, algorithm, figsize=(12, 8), frames=100, interval=100):
    # Try to use tqdm to show progress
    use_tqdm = True
    try:
        from tqdm.auto import tqdm
    except:
        use_tqdm = False

    figure = plt.figure(figsize=figsize)
    ax = Axes3D(figure)
    problem.plot3d(ax3d=ax)
    minz = min(ax.get_zlim())
    it = algorithm.geniterator()
    if use_tqdm:
        it = iter(tqdm(it, total=frames))

    def animate(i):
        ax.clear()
        ax.autoscale(enable=False)
        problem.plot3d(ax3d=ax)
        status = next(it)
        population = status.population
        P = algorithm.denormalize(population)
        fitness = status.fitness
        idx = status.best_idx
        PT = P.T
        # Individuals
        ax.scatter(PT[0], PT[1], fitness, s=30, c='#930039', marker='o', depthshade=False, zorder=999)
        # Shadow projections
        ax.scatter(PT[0], PT[1], np.full_like(PT[0], minz), alpha=0.5, s=50, c='black', marker='o',
                   edgecolors='none', depthshade=False, zorder=999)

    anim = animation.FuncAnimation(figure, animate, frames=frames, interval=interval, blit=False)
    return anim_____no_output_____

## Usage example_____no_output_____

from yabox.problems import Ackley, Levy
problem = Ackley()_____no_output_____algorithm = DE(problem, problem.bounds)
anim = generate_video(problem, algorithm, figsize=(12, 8), frames=10, interval=100)
display_animation(anim)_____no_output_____